Serve GLM-5.3 Flash EXL3 Across Two NVIDIA DGX Spark Systems

Overview

GLM-5.3 Flash EXL3 for 2x DGX Sparks is a deployment recipe and vLLM overlay for serving a 4 bpw EXL3/TR3 quantization of GLM-5.3 Flash across two NVIDIA GB10 systems. It exposes an OpenAI-compatible API on port 8888 and serves the model under the identifier GLM-5.3-Flash-EXL3.

The recipe uses tensor parallelism with one rank on each machine. It combines packed EXL3 routed experts, native sm_121a kernels, an fp8 sparse-MLA KV cache, CUDA graphs, and DFlash2 speculative decoding with seven draft tokens.

The checkpoint used by default is Mia-AiLab/GLM-5.3-Flash-EXL3-TR3-4bpw, a public byte-identical mirror of the referenced Brandon Music snapshot. It contains approximately 164 GiB across 120 shards. The model repository has its own license, while the serving scripts and overlay are MIT licensed.

Why the overlay is necessary

The stock ARM64 GLM-5.3 vLLM image can load this checkpoint, but it fails on the first forward pass because GLM-5.3 Flash uses NoPE MLA while the available SM12x sparse-MLA backend expects a different packed geometry. The overlay pads the 512-dimensional NoPE latent into the geometry required by FLASHINFER_MLA_SPARSE_SM120 without changing the QK dot product.

It also registers a functional EXL3 quantization method. This keeps routed experts in their packed trellis, SUH, SVH, and MCG representation instead of expanding them to BF16. With the fused path enabled, each layer uses one exllamav3_ext.exl3_moe launch.

The overlay additionally supplies GLM-specific integration for DFlash2, including EAGLE3 auxiliary hidden-state capture, draft sliding-window attention that shares target MLA KV slots, and non-causal attention within the speculative draft block.

Key features

• OpenAI-compatible service: vLLM exposes /v1/chat/completions and related endpoints on the head node.
• Two-node tensor parallelism: the deployment uses --nnodes 2 and --tensor-parallel-size 2.
• Compact EXL3 weights: the 4 bpw checkpoint is approximately 164 GiB to download and is reported as 176 GB in the referenced quality panel.
• DFlash2 speculative decoding: the default draft model is incoai/GLM-5.3-Flash-DFlash2 with k=7.
• Long context: the default maximum model length is 900,000 tokens, backed by an fp8 MLA pool reported at 982,612 tokens under the documented memory configuration.
• Multimodal support: image and video inputs are enabled, with a default per-prompt limit of four images and one video.
• Tool and reasoning parsing: the launcher enables automatic tool selection and GLM-compatible tool-call and reasoning parsers.
• Prefix caching: block-aligned prefixes can be reused when the relevant hashes remain in the cache.
• Optional refusal-direction ablation: ABLIT=1 applies an in-memory edit to selected native BF16 output-projection weights at load time.

Hardware and software requirements

This recipe is specifically designed for two NVIDIA GB10 DGX Spark-class systems connected through the documented CX7 fabric. It is not a generic single-GPU deployment.

• Two NVIDIA GB10 systems with approximately 180 GiB of free storage per node for the initial setup.
• Docker available without sudo on both systems.
• Passwordless SSH from the head node to the worker.
• hf or huggingface-cli, curl, and rsync installed on the head.
• Correct CX7 network interfaces and RDMA devices on both machines.
• A Hugging Face cache location that can hold the 120-shard checkpoint and the approximately 2.3 GiB DFlash2 model.

Keep the repository's CX7 interface and HCA settings unless your physical cabling differs. The README warns that NCCL cannot use the 10.0.0.x loopback aliases and may hang during ncclCommInitRank if the fabric pins are incorrect.

Install and configure the deployment

1. Clone the repository

git clone https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks.git
cd GLM-5.3-Flash-EXL3-2x-DGX-Sparks

2. Create the environment file

cp .env.example .env

Edit .env if the machines do not use the recipe's default addresses or account names. The most important settings are:

• HEAD_IP=10.0.0.1 for the head node and vLLM master.
• WORKER_IP=10.0.0.2 for the second Spark.
• WORKER_USER when the SSH account differs between nodes.
• WORKER_HOME when the worker's home or Hugging Face cache location differs.
• PORT=8888 for the API listener.

The launcher creates .env from .env.example automatically if it is missing. Values supplied directly before a command take precedence over values in the file.

3. Optionally stage the weights

./download.sh

This command downloads the EXL3 checkpoint and DFlash2 model into the head node's Hugging Face cache. It does not run Docker, connect to the worker, or synchronize files. Use it when you want to complete the approximately 164 GiB model download before the worker is ready.

To explicitly refresh the cached files, run:

REFRESH_WEIGHTS=1 ./download.sh

Avoid refreshing unnecessarily. Both download.sh and start.sh skip repositories that are already complete.

4. Start both ranks and the API

./start.sh

The startup script performs the following sequence:

1. Checks Docker, SSH, and available disk space on both nodes.
2. Pulls the public ghcr.io/miaai-lab/glm-5.3-flash-2x-dgx-sparks:exl3 image on the head.
3. Transfers the container image to the worker when its digest has changed.
4. Downloads missing model files into the head's Hugging Face cache.
5. Synchronizes the cache to the worker with rsync.
6. Starts rank 1 in headless mode on the worker.
7. Starts rank 0 and the OpenAI-compatible API on the head.
8. Polls the health endpoint until model loading and warmup finish.

The default readiness timeout is 3,600 seconds because loading the weights and capturing CUDA graphs can take substantial time.

5. Check status and logs

./start.sh status
./start.sh logs
./start.sh logs worker

Stop both containers with:

./start.sh stop

You can also use ./stop.sh.

Send a basic chat request

On the head node, the base API URL is http://127.0.0.1:8888/v1. With the documented LAN addresses, other machines can use http://10.0.0.1:8888/v1.

curl -s http://127.0.0.1:8888/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "GLM-5.3-Flash-EXL3",
    "messages": [
      {"role": "user", "content": "Explain tensor parallelism briefly."}
    ],
    "chat_template_kwargs": {
      "enable_thinking": false
    }
  }'

The request's model value must match the served name, which defaults to GLM-5.3-Flash-EXL3. Thinking is enabled by default. To disable it, place chat_template_kwargs at the top level of the JSON request as shown above.

Do not send a literal nested extra_body object over raw HTTP. In the OpenAI Python SDK, extra_body is an SDK option whose contents are merged into the top-level request.

The checkpoint's generation configuration may supply temperature=1.0 and top_p=0.95. Set sampling values in each request when you need different behavior.

Understand the default runtime

The deployment's defaults are intentionally coordinated. Changing one component can make the kernel combination invalid or reduce speculative acceptance.

• Weights: Mia-AiLab/GLM-5.3-Flash-EXL3-TR3-4bpw, with fallback to the Brandon Music repository when the mirror is unavailable or incomplete.
• Parallelism: two nodes and tensor-parallel size two using the multiprocessing executor.
• Quantization: exl3, with packed routed experts and native dense, shared-expert, attention, embedding, and output-head weights.
• Target KV cache: fp8, represented internally as packed fp8_ds_mla.
• Speculation: DFlash2 with seven speculative tokens and draft tensor parallelism set to one.
• Draft KV cache: automatic selection, resulting in BF16 for the dense draft model rather than the target's MLA layout.
• Attention: sparse MLA for the target and automatically selected FLASH_ATTN for DFlash2 draft attention.
• Context: MAX_MODEL_LEN=900000.
• Concurrency: MAX_NUM_SEQS=4.
• Prefill: MAX_NUM_BATCHED_TOKENS=1024.
• Memory utilization: GPU_MEM_UTIL=0.87.

Switch speculative decoding modes

DFlash2 is the default and uses incoai/GLM-5.3-Flash-DFlash2. To fall back to the checkpoint's MTP path with two speculative tokens, restart with:

SPEC_METHOD=mtp ./start.sh restart

The environment also supports SPEC_METHOD=none. DFlash2's default draft tensor parallelism is one, keeping the small draft model on rank 0 and avoiding CX7 communication for every draft step.

Do not force TRITON_ATTN for DFlash2. According to the README's measurements, that backend made attention causal within the draft block and caused acceptance at later speculative positions to collapse. Leave draft attention unset so SM121 selects FLASH_ATTN.

KV cache and long-context configuration

The target KV cache must remain fp8. On this architecture, the sparse-MLA kernel accepts packed fp8_ds_mla; BF16 does not have a compatible sparse kernel, and NVFP4 KV kernels apply to dense MHA rather than sparse MLA.

With DFlash2, vision support, memory utilization of 0.87, and a maximum model length of 900,000, the README reports a 982,612-token pool using approximately 15.67 GiB for fp8 MLA. This provides roughly 1.09 times the capacity of one full 900,000-token request. A native one-million-token configuration still does not allocate under the documented setup.

Keep these settings unless you are testing a deliberate runtime change:

KV_CACHE_DTYPE=fp8
GPU_MEM_UTIL=0.87
MAX_MODEL_LEN=900000
MAX_NUM_BATCHED_TOKENS=1024
SKIP_MM_PROFILING=1
LIMIT_MM={"image":4,"video":1}

SKIP_MM_PROFILING=1 is important because profiling a maximum-size image and video dummy input can exhaust unified memory. The README also warns that raising the batched-token limit to 8,192 oversubscribes the GB10 indexer top-k path during long-context operation.

How prefix caching behaves

Prefix caching is enabled, but the OpenAI API remains stateless. A client must resend the full conversation history on every turn. vLLM hashes the submitted prefix and may reuse cached blocks when they are still available.

• Only block-aligned tokens count as hits under the documented MLA cache manager.
• MAX_NUM_SEQS=4 permits four in-flight generations; it does not reserve four persistent chat sessions.
• Idle conversations are not guaranteed to remain cached.
• One conversation's activations are not mixed with another conversation.
• An older chat can require a fresh prefill after another workload evicts its prefix.

The README's live test reused 3,584 tokens from a 7,760-token follow-up, or about 46 percent of the prompt. Treat prefix caching as opportunistic acceleration rather than durable session storage.

Measure decode performance

The repository includes tests/bench_decode.py for streaming decode and coherence checks. Run the structured count test with:

python3 tests/bench_decode.py \
  --phase structured \
  --structured \
  --runs 5 \
  --max-tokens 400 \
  --skip-coherence \
  --out /tmp/glm53-structured.json

Run the prose hash-map explanation test with:

python3 tests/bench_decode.py \
  --phase prose \
  --runs 5 \
  --max-tokens 400 \
  --skip-coherence \
  --out /tmp/glm53-prose.json

Under the README's documented warm, empty-KV protocol, the structured workload reached 62.9 tokens per second at concurrency one, 103.3 aggregate tokens per second at concurrency two, and 146.5 aggregate tokens per second at concurrency four. Performance depends strongly on speculative acceptance: the lab median was 61.7 tokens per second for structured output but 26.9 for prose. Mixed or long-context tests with roughly 60,000 to 100,000 cached tokens were reported at 24 to 27 tokens per second.

These figures describe the specific two-GB10 setup, prompt types, temperature-zero configuration, disabled thinking, DFlash2 k=7, and 400-token generations in the README. They should not be interpreted as universal throughput guarantees.

Optional refusal-direction ablation

The repository can apply refusal-direction ablation at weight-load time without rewriting or requantizing the EXL3 checkpoint. Enable it with:

ABLIT=1 ./start.sh restart
./start.sh logs | grep ablit

The default recipe edits each native BF16 self_attn.o_proj in layers 15 through 45. Layers 0 through 14 remain unchanged as safety anchors. The default scale is ABLIT_ALPHA=3.0; setting it to 1.0 applies a plain projection that removes the component along the supplied direction.

Useful controls include:

• ABLIT=0 or an unset value to leave the stock weights unchanged.
• ABLIT_DIRECTION=dealign for the published default direction.
• ABLIT_LAYERS=15-45 for the inclusive edited range.
• ABLIT_ALPHA=3.0 for the projection scale.
• ABLIT_INCLUDE_MTP=1 to include the checkpoint MTP block when MTP is used.

The KLD quality results in the README were measured without ablation. Enabling it changes refusal behavior and can affect coherence or speculative acceptance because the DFlash2 drafter remains unchanged while the target outputs change. If coherence degrades, the README suggests lowering ABLIT_ALPHA first.

Advanced launch workflows

Once the image and model cache exist on both machines, you can skip repeated transfer work:

SKIP_DOWNLOAD=1 SKIP_SYNC=1 ./start.sh

To retain the local container image and skip all pulls, downloads, and synchronization:

SKIP_PULL=1 SKIP_DOWNLOAD=1 SKIP_SYNC=1 ./start.sh restart

To rebuild the overlay from the current repository rather than use the public image:

BUILD=1 SKIP_DOWNLOAD=1 SKIP_SYNC=1 ./start.sh restart

You can also build the Dockerfile directly:

docker build -t glm53-flash-sm121:local .

The startup script normally runs a GPU overlay self-check before shipping the image to the worker. Only set SKIP_OVERLAY_VERIFY=1 when you intentionally want to bypass that check.

Critical configuration warnings

• Do not pass --moe-backend marlin. This deployment uses EXL3 weights and its own packed expert method.
• Do not set the target KV cache to NVFP4 or BF16. Keep KV_CACHE_DTYPE=fp8.
• Do not pin speculative draft attention to TRITON_ATTN.
• Do not replace the image with the older glm53-flash-sm121:v8 NVFP4/Ray deployment.
• Do not change tensor parallelism, node count, CX7 pins, or USE_HOST_NCCL casually. They are integral to the two-node recipe.
• Do not delete or requantize the Hugging Face checkpoint cache unless you intend to download the complete model again.
• Do not disable multimodal profiling safeguards without accounting for unified-memory usage.
• Do not point ABLIT_DIRECTION at an unrelated checkpoint's vector without verifying its hidden size.

Troubleshooting

The service never becomes healthy

Inspect both logs with ./start.sh logs and ./start.sh logs worker. Check passwordless SSH, available disk space, Docker permissions, cache completeness, and CX7 interface names. Initial weight loading and graph capture are slow, so allow the default readiness timeout to expire only after reviewing progress in the logs.

NCCL hangs during rank initialization

Verify HEAD_CX7_IF, WORKER_CX7_IF, HEAD_CX7_IB, and WORKER_CX7_IB. The documented defaults are tied to the lab's physical ports and may need adjustment if your cabling differs.

The worker cannot access GHCR

The worker does not require direct GHCR access. The head pulls the public image and transfers it over SSH. If the head encounters anonymous rate limits, configure GHCR_USER and GHCR_TOKEN.

Speculative decoding is unexpectedly slow

Confirm that SPEC_METHOD=dflash, DFLASH_TOKENS=7, and draft attention is not forced to TRITON_ATTN. Also remember that speculative performance varies by output type: highly structured sequences can have much higher acceptance than unconstrained prose.

Conclusion

This project turns a pair of NVIDIA GB10 systems into a two-rank OpenAI-compatible GLM-5.3 Flash server using compact EXL3 weights, fp8 sparse-MLA caching, and DFlash2 speculative decoding. The fastest path is to preserve the supplied network, KV, attention, and quantization defaults, launch with ./start.sh, and validate the deployment through the health endpoint, chat API, logs, and included decode benchmarks.