Serve DeepSeek V4.1 Flash Locally on Two DGX Sparks with EXL3 and vLLM

Overview

This project serves DeepSeek V4.1 Flash locally across two NVIDIA GB10 DGX Spark systems. It uses a 2.9 bits-per-weight EXL3 checkpoint, tensor parallelism across both nodes, native sm_121a kernels, and an OpenAI-compatible vLLM API on port 8888.

The served model identifier is DeepSeek-v4.1-Flash-EXL3. The deployment is text-only and is designed specifically for a two-node GB10 kit connected over CX7.

The container overlays EXL3 support onto vllm/vllm-openai:deepseekv41-flash-0909, an ARM64 vLLM image that includes the required DeepseekV41 architecture.

What the Project Provides

• Local OpenAI-compatible serving: Applications can call the standard /v1/chat/completions endpoint.
• Two-node tensor parallelism: The model runs with tensor-parallel size 2 over the CX7 connection.
• EXL3 quantization: The checkpoint averages 2.9 bpw and uses the mul1 codebook with per-tensor quantization settings.
• Native DSpark speculation: Draft experts are already stored under mtp.* in the checkpoint, so no separate draft model is required.
• Long-context support: The shipped configuration sets MAX_MODEL_LEN=600000 and uses a 2.5 GiB KV cache pool.
• File-backed Engram tables: The unquantized Engram data is read from original model shards 47 and 48 rather than pinned into host memory.
• Automated startup: start.sh downloads weights, obtains or builds the image, shares files with the worker, runs GPU checks, and starts the API.
• Runtime protections: The launcher performs boot-margin checks, releases allocators between prefill phases, drops page cache after loading, and adjusts the containers' OOM scores.

Hardware and Storage Requirements

This repository targets exactly two NVIDIA GB10 DGX Spark nodes. The default head node address is 10.0.0.1, and the nodes communicate over CX7. The README identifies the tested interfaces as enp1s0f1np1/rocep1s0f1 on the first node and enp1s0f0np0/rocep1s0f0 on the second.

Plan for approximately 387 GiB of model data:

• About 197 GiB for 39 EXL3 shards from Mia-AiLab/DeepSeek-V4.1-Flash-EXL3-2.9bpw.
• About 190 GiB for original DeepSeek shards 47 and 48 plus the index. These contain the native Engram tables.
• About 9 GiB for the published container image when a matching image can be pulled instead of built.

If you already have a complete DeepSeek-V4.1-Flash directory, point ENGRAM_DIR at it to avoid downloading the Engram source again. Only shards 47 and 48 and the index are read for this purpose.

On DGX Spark, host RAM is also GPU memory. Every CUDA allocation immediately consumes shared memory, so storage and memory settings should be changed independently and tested with a long prompt.

Prepare the Head Node

1. Clone the repository and create the environment file

Run the deployment from the repository directory on the head Spark:

cp .env.example .env

The launcher also creates .env on the first run if it does not exist. Review the file and update values such as WORKER_USER and GID when required by your systems.

Values explicitly named in .env take precedence over command-line environment prefixes because start.sh sources the file over its inherited environment. For example, change WEIGHT_SYNC inside .env rather than running WEIGHT_SYNC=zfs ./start.sh.

2. Install the Hugging Face download tools

Automatic model downloading requires the Hugging Face CLI with transfer support:

pip install -U 'huggingface_hub[hf_transfer]'

Downloads are resumable. If an interrupted run leaves the model incomplete, run the launcher or downloader again.

3. Optionally download without starting the server

To stage the EXL3 and Engram files before booting vLLM, run:

./download.sh

Automatic downloading is enabled by default. Set AUTO_DOWNLOAD=0 in the configuration if you want to manage the files yourself.

Start the Server

From the repository on the head node, run:

./start.sh

On its first run, the launcher performs the following high-level operations:

1. Checks the available memory margin and node configuration.
2. Downloads any missing EXL3 and Engram files.
3. Pulls the public image ghcr.io/miaai-lab/deepseek-v4.1-flash-exl3-2x-dgx-sparks:2.9bpw.
4. Builds locally only if the recipe stamp changed or the registry is unavailable.
5. Makes the weights available to the worker.
6. Runs the EXL3 GEMM and Engram dequantization GPU checks.
7. Launches the two tensor-parallel ranks and exposes the API on port 8888.

A clean checkout should match the recipe stamp in the published image, so no compilation is normally necessary.

Control image building and transfer

• BUILD=1 ./start.sh always builds the image locally.
• SKIP_BUILD=1 ./start.sh keeps the pulled image even if its recipe stamp differs.
• SKIP_PULL=1 ./start.sh avoids contacting the container registry.
• PULL=1 ./start.sh pulls the image again even if the local stamp matches.
• SKIP_SHIP=1 ./start.sh prevents the head from copying the image to the worker.

By default, image shipping uses a staged tar file transferred through resumable rsync. Set IMAGE_SHIP=stream to pipe docker save into the worker's docker load. Streaming needs no scratch space, but an interrupted transfer restarts from the beginning. The auto mode prefers resumable transfer and falls back to streaming when worker disk space is limited.

Check and Manage the Deployment

Use the launcher's management commands rather than operating the two containers separately:

./start.sh status
./start.sh logs
./start.sh logs worker
./start.sh stop
./start.sh restart

If the worker already has synchronized weights, restart without repeating the synchronization step:

SKIP_SYNC=1 ./start.sh restart

During startup, a 420-second log-silence detector collects Python stack traces from both ranks. Diagnostic files are written under logs/.

Send Your First API Request

Once the server is healthy, send a chat completion request to the head node. This smoke request disables thinking so that the direct answer is easy to inspect:

curl -s http://127.0.0.1:8888/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"DeepSeek-v4.1-Flash-EXL3","messages":[{"role":"user","content":"What is 17*19? Reply with the integer only."}],"max_tokens":32,"temperature":0,"chat_template_kwargs":{"enable_thinking":false}}'

The expected answer is 323. Any OpenAI-compatible client can use the same base URL and model identifier.

Use the recommended sampling settings

For ordinary model use, the official sampling settings are:

• temperature=1.0
• top_p=0.95
• Thinking enabled

The chat template defaults to high reasoning effort, represented by 75. It also accepts low for 50, max for 100, or an integer from 1 through 100.

A request using the recommended sampling configuration can look like this:

curl -s http://127.0.0.1:8888/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"DeepSeek-v4.1-Flash-EXL3","messages":[{"role":"user","content":"Explain how tensor parallel inference works."}],"max_tokens":512,"temperature":1.0,"top_p":0.95,"reasoning_effort":"high"}'

Understand DSpark Speculative Decoding

The deployment uses DSpark speculative decoding with the draft experts embedded in the checkpoint. There is no separate drafter to download, load, or serve.

The underlying vLLM configuration is equivalent to:

--speculative-config '{"method":"dspark","num_speculative_tokens":3}'

The checkpoint has a draft block-size ceiling of five, but the shipped setting uses three speculative tokens because it measured faster on prose. Capture sizes include 1 2 3 4 6 8 12 18 24, including size six for two sequences producing three speculative tokens each.

DSpark improves single-stream decoding in the reported measurements. For wider batches, disable it with SPEC_METHOD=none. This also frees approximately 3.5 GiB. The README reports about 31.6 tokens per second for one speculative stream, while speculation-off serving reached higher aggregate throughput at four streams.

Manage Memory Safely

The approximate per-node allocation includes 99.5 GiB of EXL3 weights per rank, a 2.5 GiB KV pool, roughly 5 to 7 GiB for contexts, NCCL, workspaces, and CUDA graphs, and approximately 9 GiB for vLLM processes, Docker, the operating system, and the desktop.

The shipped limits are:

• MAX_MODEL_LEN=600000
• MAX_NUM_SEQS=2
• MAX_NUM_BATCHED_TOKENS=1024
• A 2.5 GiB KV cache pool

The KV layout is selected automatically as DeepSeek's fp8_ds_mla. Do not pass --kv-cache-dtype. The measured pool held 774,400 tokens at a context limit of 614,400, with a cost of approximately 3.4 KiB per token.

Do not estimate this checkpoint's quantization by converting 2.9 to the integer 2. Its quantization parameter is selected per tensor, and different routed experts, shared experts, attention matrices, Engram tensors, MTP tensors, and the language-model head use different values.

After warm-up, the head retained approximately 4.07 to 4.21 GiB of available memory in the reported tests. A 601,000-token prefill reduced that margin to about 2.1 GiB. Therefore, validate changes after a long prefill rather than immediately after startup.

Increase only one memory-related setting at a time. The README warns that adding 0.5 GiB to the KV pool can cost roughly 1 GiB of prefill margin. A 3 GiB pool passed a short smoke test but failed at approximately 470,000 tokens in a 600,000-token prompt.

Use the optional memory guard carefully

scripts/memguard.sh checks /proc/meminfo once per second and kills the local serving container after two consecutive samples below DSV41_MEM_GUARD_GIB, whose default threshold is 1.5 GiB.

The guard is disabled by default because it cannot determine which process consumed the memory. It always kills its own vLLM container, even if an unrelated host workload caused the pressure. To arm it explicitly, set:

DSV41_MEM_GUARD=1

For heavy host-side jobs, the README recommends limiting the offending job instead:

systemd-run --scope -p MemoryMax=2G <job>

The deployment still uses --oom-score-adj 1000 for both containers, along with the boot-margin preflight, per-prefill allocator release, and post-load page-cache drop.

Choose a Weight Synchronization Strategy

NFS: the default

NFS is the configuration that fits the tested kit. It reuses the vllm-fn-nfs exporter, exposes the EXL3 and slim Engram directories, and mounts them in the worker containers as /model and /engram-src.

./start.sh share
./start.sh

start.sh performs the sharing step automatically, so the explicit share command is mainly useful when preparing or troubleshooting the export.

ZFS: node-local replicas

Set WEIGHT_SYNC=zfs in .env to maintain a local dataset on each node. ZFS can improve resilience and send only changed blocks after the initial replica. It also allows snapshot-based rollback.

The cost is substantial: each node needs roughly 385 GiB of free pool space. The README notes that the tested second Spark had only about 41 GiB free, so ZFS could not be used there.

A representative setup on each node is:

sudo apt-get install -y zfsutils-linux
sudo zpool create -f models <vdev>
sudo zfs create -o recordsize=1M -o atime=off -o compression=lz4 \
  -o mountpoint=/path/to/model models/dsv41-exl3
sudo zfs create -o recordsize=1M -o atime=off -o compression=lz4 \
  -o mountpoint=/path/to/engram models/dsv41-engram
sudo zfs allow -u "$WORKER_USER" create,destroy,mount,receive,snapshot,rollback models

Populate the Engram dataset with shards 47 and 48 plus the index only. Do not copy the complete 476 GiB native model tree. If the pool or the worker's zfs recv is unavailable, the launcher falls back to streaming.

Rsync: a simpler local copy

WEIGHT_SYNC=rsync creates a conventional node-local copy without requiring a ZFS pool. It does not provide ZFS block-level incremental updates.

Pack Engram Data for Faster Prefill

Engram tables remain in native FP8 format and are not part of the quantized EXL3 tree. By default, the file-backed embedding implementation retrieves rows without pinning approximately 47 GiB per layer into memory.

To move rank-specific Engram row files onto each node's local NVMe, run:

./start.sh pack

Packing writes approximately 94 GiB per node at tensor parallel size two. It is optional and is not required to boot. In the reported tests, local packed Engram files improved prefill performance by approximately 25 to 50 percent because row misses used local direct I/O rather than NFS.

Engram rows must remain typed as fp8_e4m3fn when passed to the dequantization kernel. Treating them as uint8 can produce plausible but incorrect output.

Run Validation Tests

The repository includes source-level checks that do not require PyTorch or vLLM:

python3 tests/test_numeric_config.py
python3 tests/test_engram_src.py
python3 tests/test_engram_secondary.py
python3 tests/test_k_map.py
python3 tests/test_memory_log.py
python3 tests/test_exl3_lm_head.py
python3 tests/test_sm120_block64.py
python3 tests/test_h2d_stage.py
python3 scripts/weight_budget.py --tp 2

The chat-template parity test requires transformers and the original model's encoding/encoding.py:

python3 tests/test_chat_template.py --src /path/to/DeepSeek-V4.1-Flash

After the API is running, execute the end-to-end smoke test:

bash tests/test_smoke.sh

The smoke test asks the model to calculate 17*19 and expects 323. Before launching the service, start.sh also runs GPU parity and Engram dequantization checks in a temporary container. Results are written to logs/overlay-verify.log.

Advanced Operational Tips

• Prefer DSpark for one or two streams: The reported single-stream decode rate was 31.6 tokens per second, with 42.5 aggregate tokens per second across two streams.
• Disable speculation for wider batches: With SPEC_METHOD=none, single-stream throughput fell, but four-stream aggregate throughput reached 53.7 tokens per second in the reported test.
• Pack Engram for repeated long prompts: Local NVMe row files reduce dependence on the head node's NFS export and can materially improve prefill speed.
• Keep the deployment text-only: The SM120 kernel envelope does not provide the required sparse-MLA kernel for the model's vision-token width.
• Do not re-enable EXL3 auxiliary streams casually: The deployment serializes EXL3 streams because two kernels on different CUDA streams can deadlock with the runtime's single lock buffer per device.
• Watch the correct network interfaces: NCCL cannot use the 10.0.0.x loopback aliases. Incorrect per-NIC GID selection can fail after about a minute with ibv_modify_qp errno 61.
• Inspect long-prefill memory: Boot success and short smoke tests do not prove that a larger KV pool or context setting is safe.
• Use the repository launcher: It coordinates NFS or local synchronization, preflight checks, image verification, two-rank startup, and diagnostics.

Expected Performance

The published measurements report 31.6 tokens per second for one decode stream and 42.5 aggregate tokens per second for two streams. Short-prompt prefill was around 965 to 1,041 tokens per second with packed Engram data and grouped expert kernels.

Long-prompt prefill naturally becomes slower. Reported examples include approximately 979 tokens per second for a 100,000-token prompt, 802 tokens per second at 455,000 tokens, and 810 tokens per second at 601,000 tokens. Actual results depend on prompt shape, storage configuration, speculation, batch width, and the configured prefill chunk size.

Conclusion

This project packages the specialized kernels, EXL3 loader, file-backed Engram implementation, networking configuration, memory controls, and launcher needed to serve DeepSeek V4.1 Flash on two GB10 DGX Sparks. Begin with the shipped defaults, verify the API using the smoke test, and then tune one setting at a time. For the safest operation, evaluate both throughput and available memory during realistic long-prefill workloads.