Your Outcome: A reproducible, production-grade recipe for serving a 671B-parameter frontier model on two €3,000 desksides — tensor-parallel across both GB10s over 200 Gbps RoCE, with speculative decoding, NVFP4 KV cache, and 21+ correctness patches already battle-tested. This is the exact configuration running in production on our ai1 + ai2 cluster.
The Setup in One Breath
Two NVIDIA DGX Spark systems (GB10 Grace Blackwell Superchips), connected by 200 Gbps RoCE, serving DeepSeek-V4-Flash — a 671B-parameter MLA-MoE model — through vLLM with tensor parallelism 2, DSpark speculative decoding, and an NVFP4 MLA KV cache. The whole thing is fronted by a LiteLLM proxy and monitored by Prometheus/Grafana.
No cloud. No API keys to a frontier lab. Just two desktops and a lot of hard-won configuration.
This article documents every engineering decision we made to get there — the model pin, the transport choice, the batch-size math, the context-window trade-off, the 21 patches that make multi-sequence speculative decoding correct under vLLM's continuous batching, and the cleanup that freed 36 GB of RAM and 29 GB of disk. If you own DGX Spark hardware, this is your fast path to the same result.
Hardware: Two GB10 Superchips
| Component | Specification |
|---|---|
| System | 2× NVIDIA DGX Spark (GB10 Grace Blackwell) |
| CPU | NVIDIA Grace (Arm Cortex-X925, 10 cores, aarch64) |
| Unified Memory | 128 GB LPDDR5X (121 GB usable) per node |
| GPU | GB10 Blackwell, ~99 GB VRAM-resident via unified memory |
| Interconnect | 2× ConnectX-7, 200 Gbps RoCEv2 (rocep1s0f0 / rocep1s0f1) |
| Network fabric | Direct 10 Gbps bond (bond0) + 200 Gbps RoCE for NCCL |
| Roles | ai1 (head, 192.168.0.27) + ai2 (worker, 192.168.0.176) |
| Disk | 1.8 TB NVMe, ~37% used |
The GB10's defining trait is unified memory: CPU and GPU share 128 GB. That eliminates PCIe transfer overhead but means a wrong gpu-memory-utilization value doesn't just crash a container — it can freeze the whole machine. Every memory decision below is made with that constraint in mind.
Software Stack
| Layer | Choice | Why |
|---|---|---|
| Runtime image | ghcr.io/anemll/dspark-vllm-gx10:0.1.1 | Anemll's DSpark vLLM overlay — the only image with working TP2 + speculative decoding on GB10 |
| vLLM version | 0.25.2.dev0+g752a3a504.d20260714-tp2-bcc19323 | V1 engine with DSpark proposer |
| Model | deepseek-ai/DeepSeek-V4-Flash-0731 @ 9e165c30e2704aec5d9d593cce3eebd58bbef1cb | Pinned revision for reproducibility |
| Served name | deepseek-v4-flash | Client-facing alias (see Decision #2) |
| Orchestration | Docker Compose (network_mode: host) | Port 8888 directly on host |
| Proxy | LiteLLM | Unified OpenAI-compatible endpoint, routes deepseek-v4-flash → ai1:8888 |
| Monitoring | Prometheus + Grafana + node_exporter | GPU, vLLM, RoCE metrics |
The 24 Decisions That Made It Work
Model & Serving Identity
1. Pin the model revision. We serve deepseek-ai/DeepSeek-V4-Flash-0731 at exact commit 9e165c30e2704aec5d9d593cce3eebd58bbef1cb. Floating tags drift; a pinned revision means the deployment is byte-reproducible and two months of tuning isn't invalidated by an upstream weights change.
2. Serve as deepseek-v4-flash, not deepseek-v4-flash-0731. Clients (LiteLLM, OpenCode, pi) send model: deepseek-v4-flash. The original config exposed the checkpoint's raw name, so every external request 404'd. We set SERVED_MODEL_NAME=deepseek-v4-flash — a one-line change that turned a broken endpoint into a working one. The weights are still the 0731 revision; only the API surface changed.
Parallelism & Transport
3. Tensor parallelism = 2, split across nodes. A single GB10 can't hold a 671B model comfortably; TP2 splits it across both. tensor_parallel_size=2, nnodes=2, node-rank 0 (ai1 head) + node-rank 1 (ai2 worker).
4. RoCE instead of TCP Socket for NCCL. The default config used NCCL_NET=Socket — TCP over the 10 Gbps bond. We flipped it to NCCL_NET=IB + NCCL_IB_DISABLE=0, unlocking the 200 Gbps ConnectX-7 RoCE fabric. For a TP2 cluster that all-reduces every forward pass, this is the single biggest communication win: ~20× the cross-node bandwidth.
5. Per-node RoCEv2 GID auto-resolution. RoCE GID indexes drift after reboots. The start script resolves NCCL_IB_GID_INDEX from sysfs at launch (head=7, worker=3) instead of hardcoding — so NCCL never wedges on a stale index. Correct GID selection is what makes IB actually connect.
6. NCCL_IB_ROCE_VERSION_NUM=2, NCCL_IB_ADDR_FAMILY=AF_INET, NCCL_IGNORE_CPU_AFFINITY=1. These pin RoCEv2, force IPv4, and let NCCL manage core affinity — the proven GB10 NCCL environment that avoids the Gloo/127.0.0.1 cross-node bug.
Memory & Batching Math
7. MAX_NUM_BATCHED_TOKENS: 8192 → 16384. The original value triggered vLLM's warning: "max_num_scheduled_tokens is set to 8168 based on the speculative decoding settings. This may lead to suboptimal performance." Spec decoding (5 tokens × 6 sequences) reserves 24 tokens of the 8192 budget, leaving only 8168 for scheduling. Doubling the budget removes the warning and gives real headroom for concurrent prefill — at the cost of more activation memory (see #8).
8. MAX_MODEL_LEN: 1048576 → 204800. This is the key trade-off. At 1M context + 16384 batch tokens, the KV cache requirement (10.91 GiB) exceeded available memory (9.23 GiB) and the engine refused to start: "To serve at least one request with the model's max seq len (1048576), 10.91 GiB KV cache is needed, which is larger than the available KV cache memory (9.23 GiB)." Reducing to 200K (same as our legacy vLLM cluster) freed enough memory for the larger batch size while remaining far more context than any coding-agent task needs. KV cache settled at a healthy 10.38 GiB.
9. GPU_MEMORY_UTILIZATION=0.82 (text mode). Leaves ~18% of unified memory as headroom against the GB10 OOM-freeze risk. We deliberately did not push to 0.85+ — the marginal KV-cache gain isn't worth the freeze risk when KV usage sits at 0%.
10. KV cache dtype nvfp4_ds_mla. NVFP4 MLA compression cuts KV memory ~4× vs fp16. This is what makes a 200K-token context plausible on 128 GB unified memory at all.
Speculative Decoding & Decoding Path
11. DSpark MTP-5 probabilistic speculative decoding. Five draft tokens per step, sampled probabilistically. Acceptance rates land at ~94/83/68/51/38% per position — a mean acceptance length of ~3.3–4.3, meaning most steps commit multiple tokens at once instead of one.
12. max_num_seqs=6. Six concurrent sequences. The sweet spot for GB10 memory vs. aggregate throughput (see benchmarks).
13. Prefix caching enabled. Repeated system prompts and conversation prefixes hit cache instead of re-prefilling — major win for agentic loops that resend context.
14. Chunked prefill enabled. Long prompts are prefilled in chunks interleaved with decode, keeping TTFT low under load.
15. Async scheduling enabled. Decouple scheduling from execution for smoother batching.
16. FlashInfer autotune enabled, persistent cache. The autotune cache is persisted to the HF cache volume, so warmup is near-instant on restart — no re-profiling every boot.
17. enforce_eager=True (CUDAGraph disabled). Stability over raw speed. The GB10 + DSpark + spec-decode combination is battle-tested in eager mode; CUDAGraph capture with speculative decoding is a known-risk path we chose not to walk without upstream support.
18. VLLM_USE_BREAKABLE_CUDAGRAPH=0 (regular CUDA graphs). Where graphs are used, we force the regular (non-breakable) path. Measured impact: +28.6% C1 decode (74.55 → 95.9 tok/s) and +13.1% C2 aggregate (134.2 → 151.8 tok/s), with C6 aggregate reaching 340.5 tok/s.
Correctness Patches (the 21 that matter)
19–21. Issue #21 (tool-argument dict support) + encoding hotfix. The 0731 encoder's encode_arguments_to_dsml only accepted JSON-string tool arguments, poisoning multi-turn tool history. The patch accepts dict arguments, and corrects a pre-0731 tokenizer bug that mapped low reasoning effort to high.
22. Issue #22 (nvfp4-ds-mla). KV-cache scaling-factor handling for the NVFP4 MLA dtype — without it, attention is numerically wrong.
23. Issue #26 (hybrid SWA-min). Sliding-window attention min-path correctness for the MLA window.
24. Issue #27 (partial-prefill concurrency). Serializes six cold 128K prefills one-at-a-time instead of contending — turns a 282 s TTFT collapse into bounded, predictable prefill.
25. Issue #31 (GPU-resident thinking-budget). Moves the thinking-budget gate GPU-side (V2), avoiding a CPU round-trip per step.
26. Issue #43 (decode fairness + diagnostics). Per-sequence fairness in the decode scheduler, plus diagnostic counters.
27. Issue #55 (tool-call truncation). Correctly truncates over-long tool-call outputs instead of corrupting the stream.
28–34. v0.27 performance hotfixes. Seven upstream vLLM patches backported: #50312 (MTP buffer), #50004 (adaptive top-k), #49486 (skip top-k), #48407 (dense-prefill indexer), #48957 (skip-empty-c128), #50298 (FlashMLA workspace), #44993 (grammar advance). These keep the DSpark proposer correct under vLLM-v1 continuous batching — the root-cause fix is a request-stable KV slot: the draft's persistent ring buffer is keyed by stable per-request slot, not batch-row position, so it survives vLLM's condense-on-finish row compaction.
35. DSPARK_SUPPRESS_STOPS_IN_REASONING=1. Client stop strings stay dormant until </think>, so reasoning isn't prematurely truncated mid-thought.
Reasoning & Behavior
36. DEFAULT_THINKING=max. Full reasoning effort by default; clients override per-request with reasoning_effort. For a frontier model served locally, max-quality default beats a speed default.
Cleanup & Hygene
37. Stopped the legacy vllm-cluster.service. A previous TP2 deployment (port 8000, not exposed) was still running, burning 7 GB RAM + 1.1 GB GPU on each node for zero traffic. Disabled and stopped. Memory went from 1.3 GB free to 10 GB free on the head node.
38. Removed the stale vllm-node-dsv4-b12x Docker image. 29.2 GB reclaimed on each node — the orphaned image from the legacy stack.
39. network_mode: host. The dspark container binds port 8888 directly on the host, so LiteLLM and Prometheus reach it without Docker port-mapping indirection.
40. systemd dspark-0731.service (Type=oneshot, RemainAfterExit). The container runs via Compose with restart: unless-stopped; the unit is the bookkeeping wrapper, enabled so it comes back on reboot.
Performance
Measured on the live cluster (two DGX Sparks, ConnectX-7 RoCE, TP2, MTP-5, NVFP4 MLA, regular CUDA graphs):
| Prompt | Concurrency | TTFT (s) | Prefill tok/s | Decode tok/s | Aggregate tok/s |
|---:|---:|---:|---:|---:|---:|
| 256 | 1 | 0.63 | 447 | 75.4 | 69.1 |
| 256 | 6 | 1.42 | 197 | 36.9 | 191.2 |
| 2,048 | 1 | 0.81 | 2,563 | 68.8 | 62.0 |
| 2,048 | 6 | 6.06 | 342 | 34.7 | 143.7 |
| 8,192 | 1 | 4.80 | 1,713 | 73.9 | 43.7 |
| 32,768 | 1 | 22.96 | 1,428 | 64.0 | 16.6 |
Reading the numbers:
- Solo decode is flat (~62–83 tok/s) from 256 tokens to 128K. A single chat feels fast at any context length.
- Six short chats aggregate to ~160–190 tok/s — the cluster scales with concurrency for interactive workloads.
- Prefill hits 2,563 tok/s on 2K prompts — prompt ingestion is not the bottleneck.
- Long-context floor: 32K+ × c=4/6 sits on the ~8 tok/s mixed-prefill floor (one decode token per serialized prefill step). Expected, and bounded by Issue #27.
A separate 900,000-token acceptance probe completed with 899,994 prompt tokens and ~874.8 prefill tok/s — confirming the serving profile holds far beyond typical request sizes.
How to Replicate This Exact Stack
Every decision above is encoded in the Dual DGX Spark Cluster Blueprint — the complete Ansible deployment that took months to battle-test:
- vLLM + LiteLLM + Prometheus/Grafana roles with the exact env vars, flags, and NCCL tuning documented here
- RoCE fabric validation playbook with bandwidth benchmarks
- Grafana dashboards (JSON) for GPU, vLLM, and RoCE metrics
- System Configuration Graph (SCG) — the architecture diagram
- OOM-survival guide — the 7 mitigations that prevent GB10 unified-memory freezes
Prefer a focused, recipe-style walkthrough? The Dual DGX Spark Recipe ships the production wrapper.sh, systemd units, and troubleshooting guide for the same TP2 DeepSeek-V4-Flash deployment — version-pinned and crash-tested for two months straight.
Both products encode the 40 decisions in this article. Clone, edit your inventory, run one playbook. Two hours later you have the cluster described above — or, with the Qwen3.8-27B recipe, a single-node frontier model in even less time.
Conclusion
Serving a 671B frontier model on two €3,000 desktops is no longer science fiction — but it is a configuration minefield. The GB10's unified memory punishes mistakes with machine freezes. NCCL over RoCE has more failure modes than docs admit. And speculative decoding under vLLM's continuous batching is silently wrong without the request-stable KV-slot patch.
We navigated all of it, documented every turn, and the result has run uninterrupted for months. If you have DGX Spark hardware and want the same outcome without the months of trial-and-error, the blueprint and recipe are the fastest path from stock Ubuntu to production TP2 inference.
Running DeepSeek-V4-Flash in production on DGX Spark? The Dual DGX Spark Cluster Blueprint contains the complete, version-pinned configuration — every tweak in this article, ready to deploy. Prefer one desktop? The Single DGX Spark Recipe: Qwen3.8-27B delivers frontier reasoning on a single node.
