Sovereign AI on Your Desk: Why Two DGX Sparks Beat the Cloud

In 2024, running a 685-billion-parameter model locally meant a rack of A100s, a six-figure power bill, and a PhD in distributed systems. In 2026, it means two desktop workstations, a direct-attach cable, and a single command. The sovereignty equation has inverted — and the hardware is finally ready.

This is the story of what sovereign AI looks like when it fits under your desk.

What Sovereign AI Actually Means

Sovereign AI is not anti-cloud. It's about owning the decision: choosing when your data stays local, when your models run in your jurisdiction, and when you don't pay per-token rent for capabilities you can host yourself.

The practical sovereignty threshold in 2026 is frontier-grade inference — models that can reason, use tools, and handle 200K-token contexts. DeepSeek-V4-Flash crosses that threshold. It's a Mixture-of-Experts architecture with 685B total parameters (37B active per token), tool calling, multi-step reasoning, and a quality ceiling that sits firmly in the GPT-4 class. And it runs on two NVIDIA DGX Sparks — locally, with no data leaving your network.

The sovereignty argument isn't theoretical. Germany's Digital Sovereignty Act, the EU AI Act, and the EU Data Act all create regulatory pressure toward local inference. Companies processing sensitive data — legal documents, medical records, financial models, source code — face escalating compliance costs when sending prompts to cloud APIs. Self-hosted inference eliminates the data residency question entirely.

The Hardware: Two Grace Blackwell Superchips

NVIDIA's DGX Spark (GB10) packs a Grace Blackwell Superchip into a desktop form factor: 128 GB of unified memory, a 20-core ARM Grace CPU (80-core SoC with 20 performance + 60 efficiency cores), and a Blackwell GPU on a single package connected via NVLink-C2C. Unified memory means GPU and CPU share a single address space — no discrete HBM, no PCIe DMA bottleneck — which simplifies deployment but introduces constraints we'll get to.

The real magic is the interconnect. Each DGX Spark ships with a Mellanox ConnectX-7 NIC (two ports, 200 Gbps each). Our cluster bonds two of these ports per node into a bond0 interface using Linux bonding mode 2 (balance-xmit) with layer3+4 hash policy, creating a 400 Gbps direct-attach link between the two nodes.

Node	Hostname	GB10 Memory	Role	Interconnect
ai1	192.168.0.27	121 GB	Head (API server + EngineCore)	bond0: 10.10.10.1/30
ai2	192.168.0.176	119 GB	Worker (TP2 shard)	bond0: 10.10.10.2/30

Physical cabling: Two QSFP56 direct-attach cables run between ai1 and ai2 — one per ConnectX port. No switch, no router, no intermediate hop. RoCE (RDMA over Converged Ethernet) handles NCCL's communication pattern natively.

Network bonding: Mode 2 (balance-xmit) with layer3+4 hash policy and MTU 9000 (jumbo frames). Note: mode 2 is round-robin transmit, not 802.3ad/LACP (that's mode 4). The Mellanox virtual MAC conflict (where bond0's MAC collides with the physical port's permanent address) is resolved with an explicit mac: assignment in NetworkManager.

# Simplified topology
┌─────────────────────────┐         ┌─────────────────────────┐
│  ai1 (DGX Spark #1)     │         │  ai2 (DGX Spark #2)     │
│  121 GB unified memory  │         │  119 GB unified memory  │
│  192.168.0.27 (LAN)     │         │  192.168.0.176 (WiFi)   │
│  10.10.10.1 (bond0)     │◄────────►│  10.10.10.2 (bond0)     │
│  enp1s0f0 ←─ cable ──► │         │  enp1s0f0 ←─ cable ──► │
│  enp1s0f1 ←─ cable ──► │         │  enp1s0f1 ←─ cable ──► │
└─────────────────────────┘         └─────────────────────────┘
         2 × 200 Gbps = 400 Gbps aggregate

The head node (ai1) runs the OpenAI-compatible API server on port 8000. The worker (ai2) connects via NCCL and runs headless — all inference traffic flows through ai1's public interface.

The Model: DeepSeek-V4-Flash

DeepSeek-V4-Flash is the sweet spot for sovereign inference. It's open-weight (DeepSeek license), an MoE architecture designed for efficiency:

Property	Value
Architecture	Mixture-of-Experts (MoE)
Total parameters	685B
Active parameters per token	~37B (MoE, top-1 routing)
Context window	Up to 1M tokens; 128K–204K practical on dual Spark (204K production)
Speculative decoding	Native MTP (Multi-Token Prediction)
Tool calling	Built-in (deepseek_v4 parser)
Reasoning	Chain-of-thought with thinking/thought blocks
Model size (FP8)	~149 GB across 46 shards

The model uses a custom tokenizer (deepseek_v4), MLA (Multi-head Latent Attention) for efficient KV caching, and natively supports tool calling with --enable-auto-tool-choice. For agentic workloads — the real sovereignty use case — this means the model can call tools, reason about results, and execute multi-step plans without any cloud dependency.

Performance: What Two Sparks Actually Deliver

After two months of production tuning (since May 2026), our cluster delivers these numbers on the official FP8 model with vLLM 0.21.1rc1 V0 engine, PIECEWISE cudagraph, and MTP speculation:

Decode Throughput

Scenario	Tokens/Second
Single stream, short prompt	12–20 tok/s (sustained ~15, peak ~20)
Concurrent ×2, short prompts	~25–35 tok/s aggregate
MTP speculation gain	~60% effective throughput at 68% acceptance

Prefill Throughput

Prompt Size	Tokens/Second
2,048 tokens	~1,100 tok/s
8,192 tokens	~770 tok/s
32,768 tokens	~320 tok/s
65,535 tokens	~176 tok/s

Sourced from arthurdroz's llama-benchy benchmarks (eugr/spark-vllm-docker PR #219), measured at 128K context with FP8 KV cache.

Time to First Token

Prompt Size	TTFT
Short (prefix-cached)	~2 s
32K tokens	~53 s
128K tokens	~250 s

Prefix Cache Impact

Scenario	Throughput	Speedup
Cold (first request)	2.8 tok/s	1.0×
Same prompt (cached)	6.0 tok/s	2.1×
Similar prompt (deep cached)	12.4 tok/s	4.4×

For agentic workloads — where every request shares a system prompt and conversation history — the prefix cache is transformative. A multi-turn coding session with persistent system instructions runs 4× faster after the first request.

Scaling: 1× vs 2× Spark

Metric	1× Spark (IQ2XXS + DSpark)	2× Spark (FP8 + MTP)
Precision	~2-bit quant	FP8 (official)
Decode throughput	~20 tok/s	15–20 tok/s
Context window	32K max	200K+
Concurrent requests	1 practical	2–3 at 200K ctx
Model quality	Tool-eval: 83/100	Higher (FP8 ceiling)
Community reports (V1, c=8)	N/A	~96 tok/s aggregate

The scaling from 1→2 Sparks delivers a 1.7–1.9× speedup per user (CosmicRaisins, NVIDIA Forum). The main gains aren't raw throughput — they're precision (FP8 vs 2-bit), context length (200K vs 32K), and concurrency (2–3 simultaneous requests). For sovereign AI, these matter more than peak tok/s.

Note: Community benchmarks from tonyd615 (44 tok/s) and serapis (35 tok/s) use the V1 engine with FULL_AND_PIECEWISE cudagraph — higher throughput than our V0 PIECEWISE config, which trades peak speed for stability on GB10 unified memory.

The Configuration: Production-Proven

This is the exact deepseek-v4-flash-wrapper.sh running in production — Ansible-deployed, systemd-managed, with full OOM safeguards:

#!/bin/bash
# DeepSeek V4 Flash TP2 Cluster — Production Config
# Ansible-deployed via vllm_cluster role
# Proven stable since May 2026, no crashes

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
export VLLM_SKIP_INIT_MEMORY_CHECK=1
export VLLM_TRITON_MLA_SPARSE=1
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
export VLLM_USE_FLASHINFER_SAMPLER=1
export TORCH_CUDA_ARCH_LIST=12.1a
export OMP_NUM_THREADS=8

# Reserve 1.5 GiB for system (prevents OOM freeze on GB10)
echo 1572864 | sudo tee /proc/sys/vm/min_free_kbytes >/dev/null 2>&1
ssh -n weiss@192.168.0.176 \
  "echo 1572864 | sudo tee /proc/sys/vm/min_free_kbytes >/dev/null 2>&1" || true

vllm serve deepseek-ai/DeepSeek-V4-Flash \
    --served-model-name deepseek-v4-flash \
    --host 0.0.0.0 --port 8000 \
    --trust-remote-code \
    --tensor-parallel-size 2 \
    --pipeline-parallel-size 1 \
    --enable-expert-parallel \
    --kv-cache-dtype fp8 \
    --block-size 256 \
    --enable-prefix-caching \
    --max-model-len 204800 \
    --max-num-seqs 2 \
    --max-num-batched-tokens 8192 \
    --gpu-memory-utilization 0.78 \
    --compilation-config '{"cudagraph_mode":"PIECEWISE"}' \
    --speculative-config '{"method":"deepseek_mtp","num_speculative_tokens":2}' \
    --tokenizer-mode deepseek_v4 \
    --tool-call-parser deepseek_v4 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_v4 \
    --reasoning-config '{"reasoning_parser":"deepseek_v4","reasoning_start_str":"thinking","reasoning_end_str":"thought"}' \
    --default-chat-template-kwargs '{"thinking":true}' \
    --load-format safetensors \
    --disable-custom-all-reduce \
    "$@"

Note: This wrapper is invoked by launch-cluster.sh --launch-script deepseek-v4-flash-wrapper.sh --no-ray, which automatically adds --nnodes 2, --node-rank N, --master-addr, and --master-port arguments for distributed execution. The wrapper itself is the vLLM serve configuration — it's not the complete launch command.

Key decisions in this configuration:

• gpu-memory-utilization 0.78 — The GB10's unified memory means GPU and CPU share 128 GB. At 0.78, the model loads ~75 GB, leaving headroom for the KV cache, MTP drafter, and system processes. Going higher (0.82+) causes OOM.
• PIECEWISE cudagraph — Avoids TileLang's custom_ops: ["all"] compilation path, which needs ~3 GB of system RAM during JIT and bricks the machine when it's not available. PIECEWISE mode compiles only essential graphs (attention, MLP) for a 10–20% ITL improvement without the OOM risk.
• V0 engine (no VLLM_USE_V1) — The V1 engine's Gloo-based distributed init registers 127.0.0.1 even with --network host, breaking cross-node TP2. V0 uses NCCL directly and works reliably.
• --no-ray distributed executor — PyTorch native distributed (mp backend) is more reliable than Ray for small clusters. No world-size issues, no Ray dependency overhead.
• --disable-custom-all-reduce — Necessary on GB10 to avoid custom communication kernels that may not optimize for the unified memory architecture.

OOM Survival: Why Unified Memory Changes Everything

The GB10's unified memory is the most important constraint — and the one most people get wrong. Unlike discrete GPUs with separate HBM and system RAM, the GB10's GPU and CPU share a single 128 GB pool. When memory runs out, there's no graceful CUDA error. The entire machine freezes — SSH hangs, no ping, no recovery. Hard power cycle required.

Our production environment applies seven OOM mitigations to both nodes via Ansible:

Mitigation	Setting	Purpose
Disable swap	swapoff -a	Swap on UMA causes death spiral
Disable overcommit	vm.overcommit_memory=0	malloc fails immediately
Minimize swappiness	vm.swappiness=1	Belt-and-suspenders
Docker memory limit	--memory=100G	Process killed before kernel freezes
Reserve free memory	vm.min_free_kbytes=1572864	1.5 GiB system reserve
Drop page cache	echo 3 > /proc/sys/vm/drop_caches	Before model load
Lower readahead	blockdev --setra 16	Faster model load from NVMe

The startup script drops page cache and temporarily reduces NVMe readahead to 8 KB (restored after 180 seconds). Model weights load from NVMe in ~152 seconds; CUDA graph capture takes ~7 seconds; warmup and FlashInfer autotune complete in ~2 minutes. Total cold start: ~7 minutes.

Infrastructure as Code: The Ansible Story

The entire cluster is managed as code — network bonding, vLLM service lifecycle, OOM safety sysctl, IB health monitoring, and Prometheus/Loki metric shipping are all Ansible-driven.

• Network configuration — bond0 interface, MTU 9000, static IPs, Mellanox MAC override
• IB health monitoring — systemd timer that pings the peer every 60 seconds
• vLLM service — systemd unit with proper lifecycle (start, stop, cleanup)
• OOM safety — sysctl tuning, swap disable, memory limits
• Monitoring — node_exporter, Prometheus metrics forwarder, Loki log shipping

Everything is version-controlled, reproducible, and auditable. Changing a gpu-memory-utilization value means editing one file, running one playbook, and letting systemd handle the restart.

The Sovereignty Calculus: Cost, Control, Compliance

Cost Comparison (Annual)

Deployment	Year 1 Cost	Ongoing/Year	Notes
2× DGX Spark (on-prem)	~$5,000–6,000	Power (~$200)	One-time hardware + electricity
Cloud API (GPT-4 class)	$0	$15,000–50,000+	Per-token pricing scales with usage
Cloud GPU (A100 rental)	$0	$12,000–24,000	Reserved instances, no data residency control

The break-even on two DGX Sparks against cloud API pricing is typically 2–4 months for a team of 5–10 developers. For regulated industries where data residency is non-negotiable, the calculation isn't financial — it's compliance.

What You Control

Capability	Cloud API	On-Prem Dual Spark
Data residency	❌ Data leaves your network	✅ Never leaves your network
Model selection	❌ Vendor-controlled	✅ Any open-weight model
Cost predictability	❌ Per-token pricing	✅ Fixed hardware cost
Regulatory compliance	⚠️ Shared responsibility	✅ Full control
Custom fine-tuning	⚠️ Limited or expensive	✅ Full LoRA/QLoRA support
Context window	⚠️ API-dependent	✅ 200K+ tokens
Tool calling privacy	❌ Tool calls logged by vendor	✅ All calls stay local
Offline operation	❌ Requires internet	✅ Air-gappable

Agentic Workloads: The Sovereignty Sweet Spot

The real value of sovereign AI isn't single-prompt chat. It's agentic systems — multi-step tool-using workflows where data sovereignty matters at every step.

Consider a code review agent that:

1. Reads proprietary source code from your Git repository
2. Analyzes it for security vulnerabilities
3. Queries your internal knowledge graph for architecture patterns
4. Generates patches and submits pull requests

Every step involves sensitive data. Sending this through a cloud API means your proprietary code, your architecture decisions, and your security findings all leave your network. Running locally on dual Sparks means nothing leaves.

With the prefix cache enabled, the agent's system prompt (~2K tokens) gets cached after the first request. Subsequent tool calls with the same system prompt execute at 4.4× the initial speed — critical for multi-turn agentic workflows that may make 10–20 tool calls per session.

The Community Context: Who Else Is Running This

The NVIDIA Developer Forums' DGX Spark community has validated this architecture extensively:

• tonyd615 reported ~44 tok/s decode (single stream) and ~45 tok/s at concurrency=2 on the V1 engine with FULL_AND_PIECEWISE cudagraph — higher than our V0 PIECEWISE config but using a different engine mode
• serapis benchmarked 256K context at 35 tok/s decode, including tool-eval-bench results confirming strong tool-calling quality
• arthurdroz (NVIDIA/eugr spark-vllm-docker PR #219) published full llama-benchy numbers showing the context-throughput scaling curve
• CosmicRaisins reported 1.7–1.9× scaling from 1→2 nodes, dropping to 1.4–1.6× from 2→4 nodes — making dual Spark the optimal cluster size
• vr8vr8 ran 48 hours non-stop at concurrency=4 with 300K context, proving long-term stability

Our V0 PIECEWISE production config prioritizes stability over peak throughput — a deliberate choice after the V1 engine's Gloo bug and the TileLang custom_ops OOM freeze. The configuration is reproducible: CosmicRaisins independently confirmed similar scaling efficiency.

The Single-Spark Question: When Is One Enough?

For some workloads, a single DGX Spark is sufficient:

Use Case	1× Spark	2× Spark
Small models (<70B)	✅ Excellent	Overkill
Coding assistant (single user)	✅ Good (IQ2XXS: ~20 tok/s)	Better (FP8 + 200K ctx)
Multi-turn chat	✅ Good	✅ Better (prefix cache)
Production multi-user	❌ Bottlenecked	✅ Handles concurrency
200K+ context	❌ Doesn't fit	✅ Fits (FP8)
Sensitive data processing	✅ Works	✅ Works (both local)
Tool-calling agents	✅ Functional	✅ Full FP8 quality

The Single DGX Spark Recipe: Qwen3.8-27B covers the one-node deployment path — the same Ansible-managed approach, tuned for 128 GB unified memory without the TP2 complexity.

What's Next: The Roadmap

Our cluster roadmap addresses the known limitations:

1. Increase max-num-seqs to 4–8 — KV cache utilization sits at ~52% with 2 sequences; there's headroom for more concurrency
2. Persistent kernel cache volume — Pre-compiled TileLang kernels would enable custom_ops: ["all"] for maximum fusion performance without first-run OOM risk
3. Multiple model recipes — Qwen3.5-122B FP8 and MiniMax M2.7 NVFP4 are both validated on dual Spark in the community
4. Monitoring dashboard — Prometheus + Grafana for real-time GPU utilization, tok/s throughput, and KV cache pressure

The Bottom Line

Sovereign AI in 2026 is not a compromise — it's an engineering decision with clear cost, compliance, and control advantages. Two DGX Sparks, a direct-attach cable, and a well-tuned vLLM configuration deliver frontier-grade inference that never sends your data through anyone else's server.

The configuration isn't perfect — 200K context at ~15 tok/s won't match a cloud provider's A100 cluster at concurrency=64. But for the sovereign AI use case — sensitive data, regulated industries, agentic workflows, air-gapped environments — it doesn't need to. It needs to be good enough, fully local, and production-stable.

Two months in, our cluster hasn't crashed once since we applied the OOM safeguards and settled on the PIECEWISE cudagraph + MTP configuration. It processes agentic coding workflows, knowledge graph queries, and multi-turn reasoning chains — all behind our firewall, all under our control.

That's what sovereign AI looks like when it fits under your desk.

---

Related resources on graphwiz.ai:

• Single DGX Spark Recipe: Qwen3.8-27B — Qwen3.8-27B on a single GB10 (€79)
• Knowledge Graph Toolkit — Complete knowledge graph arsenal: GraphRAG pipelines, Neo4j deployments, visualization, and monitoring (€179)
• GraphRAG Production Playbook — Architecture decision framework for hybrid vector + graph search systems (€79)
• Neo4j + LLM Integration Guide — Ship hybrid RAG that combines Neo4j graph traversal with vector search (€39)
• Two-Node DGX Spark Cluster: Running DeepSeek V4 Flash at 20 TPS — The original technical deep-dive with benchmark methodology

Enjoyed this article? Subscribe to get notified about new sovereign AI and infrastructure deep-dives.