Serve Qwen3.8 Flash Next 176B Across Two DGX Sparks

Overview

This project is a production-oriented deployment recipe for serving RadixArk/Qwen3.8-Flash-Next-NVFP4 with SGLang. The model is an approximately 176-billion-parameter, 135 GB, NVFP4-quantized mixture-of-experts model.

The supplied start.sh script distributes the model across two NVIDIA DGX Spark systems using tensor parallelism with TP=2. The nodes communicate over a directly connected ConnectX-7 200 Gb RoCEv2 link. Once running, the head node exposes an OpenAI-compatible API on 0.0.0.0:8888.

The recipe is more than a launcher. It downloads and verifies weights, synchronizes them to the worker, builds a patched container image on both nodes, starts the cluster, and waits for API readiness. It is designed to be idempotent, so rerunning it can resume interrupted downloads, reuse caches, and replace stale containers.

What the Recipe Provides

• Two-node tensor parallel serving: One DGX Spark runs rank 0 and the SGLang router, while the second runs rank 1.
• OpenAI-compatible endpoints: The API supports chat, completions, reasoning output, tool-call parsing, and text-plus-image input.
• Long context: The default context length is 900000 tokens, using YaRN scaling from the model’s native 262,144-token context.
• NEXTN speculative decoding: The default speculative settings are 3/1/4, with CUDA-graph decode on both nodes.
• SM121 compatibility: The build applies a Triton fallback for the Qwen Sparse Attention path that otherwise fails to compile on DGX Spark.
• Operational commands: The script includes preflight checks, downloads, launch, stop, status, logs, and smoke testing.

The README reports 64.4 tokens per second for a single decode stream and 116.8 aggregate tokens per second with two streams on the tested two-node cluster. At four streams, aggregate throughput was 114.1 tokens per second.

Why the SM121 Kernel Patch Is Necessary

The Qwen4Exp architecture routes attention through a Qwen Sparse Attention backend. Its flash-attention resolver prefers classic FA2 and otherwise uses the flash-attn-4 CuTe DSL interface. On the DGX Spark’s SM121 architecture, that fallback fails during compilation with an MLIR layout-congruence error.

The repository solves this boot blocker by building a derivative SGLang image containing an SM121-compatible Triton attention fallback.

The generated .patch/ build context includes qsa_fa_fallback.py, a FlashDecoding-style variable-length kernel tailored to the model’s QSA call contract. It supports grouped-query attention, head dimensions up to 256, and online softmax. It also reads cu_seqlens on the device, which keeps CUDA-graph replay valid when the backend rewrites its sequence table.

During the Docker build, the recipe patches qwen_sparse_attn_backend.py so the Triton fallback is selected whenever is_sm100_supported() is false. The standard path therefore remains intact on supported B100 and B200 datacenter GPUs.

Required Hardware and Software

Cluster hardware

• Two NVIDIA DGX Spark systems with GB10 GPUs, SM121 architecture, and 128 GB unified memory each.
• ConnectX-7 ports connected directly between the two machines without a switch.
• The relevant RoCE interfaces active on both nodes, such as rocep1s0f1/enp1s0f1np1 on one node and rocep1s0f0/enp1s0f0np0 on the other.
• Sufficient storage for the approximately 135 GB checkpoint, container images, caches, and temporary build data.

Head-node software

• Docker.
• The Hugging Face hf CLI.
• A Hugging Face token in the shell environment if checkpoint access requires authentication.
• Passwordless SSH access from the head node to the worker.

Install the Hugging Face CLI on the head node with:

pip install hf

By default, the worker is addressed through an SSH alias named spark2. The worker login defaults to the same username used to run start.sh on the head node. These assumptions can be changed in .env.

Configure the Two-Node Cluster

Clone the repository on the head node, enter its directory, and create your local configuration file:

git clone <this-repo>
cd <this-repo>
cp .env.example .env

Edit .env and confirm at least the head RoCE IP, worker RoCE IP, and SSH destination. The defaults expect the following private fabric addresses:

HEAD_CX7_IP=10.0.22.1
WORKER_CX7_IP=10.0.22.2
WORKER_HOST=spark2

If the worker uses a different account, set WORKER_USER. For complete control over the SSH destination, set WORKER_SSH to a value such as user@host. Shell environment variables take precedence over values in .env.

Expected topology

Clients connect to the head node’s API on port 8888. The head runs rank 0, the SGLang server, and the router. Rank 1 runs on the worker. NCCL traffic is pinned to the direct ConnectX-7 RoCEv2 connection between the two systems.

The recipe uses NCCL_NET=IB, enables the InfiniBand-compatible RoCE transport, and disables NVLS and CUMEM for this setup. It can also preload a staged host NCCL 2.30.7 build. This behavior is enabled by default through USE_HOST_NCCL=1.

Validate the Environment

Run the built-in preflight before downloading the model:

./start.sh doctor

The doctor command checks the fabric, GPUs, RAM, disk capacity, SSH connectivity, ports, and deployment constraints. Correct any reported network, authentication, storage, or resource problems before starting the large download and image build.

Download and Synchronize the Model

Download the checkpoint to the head node, verify it, and synchronize it to the worker:

./start.sh download

The default DOWNLOAD_MODE=rsync makes the head download the files and then copy them to the worker. The operation is resumable. If you prefer each machine to pull from Hugging Face independently, set:

DOWNLOAD_MODE=direct

Do not test this model with SGLang’s --load-format dummy option on GB10. According to the repository’s crash analysis, the dummy initializer temporarily converts fp8 parameters to fp16. The model’s 51.2 GB fp8 PLE n-gram table can consequently create a temporary memory demand above 150 GB and hard-freeze a system with roughly 121.7 GB of usable unified DRAM.

Build the Patched Image and Start Serving

Launch the two-node deployment with:

./start.sh serve

This command runs the preflight, confirms that weights and images are available, builds the SM121-patched image on both nodes, launches the TP2 cluster, and waits for the API to become ready. The generated local image is named qwen38-flashnext-dspark:local.

The initial boot takes approximately ten minutes because it includes loading 135 GB of weights, JIT compilation, and CUDA-graph capture. The script allows up to WAIT_TIMEOUT_MIN=90 by default.

For a fully automated first run, use:

./start.sh

With no subcommand, the script executes doctor, download, and serve in order.

Send a Basic Chat Request

After the server is ready, call its OpenAI-compatible chat endpoint:

curl http://127.0.0.1:8888/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3.8-Flash-Next-NVFP4",
    "messages": [
      {
        "role": "user",
        "content": "Explain RoCEv2 in one paragraph."
      }
    ]
  }'

Reasoning is enabled by default, and reasoning content is streamed separately. To disable thinking for one request, add chat_template_kwargs to the JSON body:

{
  "model": "Qwen3.8-Flash-Next-NVFP4",
  "messages": [
    {"role": "user", "content": "Summarize tensor parallelism."}
  ],
  "chat_template_kwargs": {
    "enable_thinking": false
  }
}

Image input is supported through the standard OpenAI-compatible image_url content part. The README confirms that combined text and image input is working, although it does not provide a complete image request example.

Connect an OpenAI-Compatible Client

Applications that support custom OpenAI endpoints can use the following base URL:

http://127.0.0.1:8888/v1

The model identifier is:

Qwen3.8-Flash-Next-NVFP4

If the client requires an API key field even when the local endpoint does not authenticate requests, the repository’s example uses dummy and disables the authentication header. Relevant model capabilities and compatibility settings include:

• Reasoning enabled.
• Text and image input.
• A 900000-token context window.
• Up to 32768 output tokens in the sample client configuration.
• max_tokens as the output-token field.
• No developer-role support in the sample compatibility configuration.
• No reasoning-effort parameter support.

Operate the Server

The single launcher provides the main administrative commands:

• ./start.sh serve performs preflight checks, ensures images and weights are present, starts TP2, and waits for readiness.
• ./start.sh download downloads, verifies, and synchronizes weights without launching the server.
• ./start.sh stop removes both containers and terminates log followers.
• ./start.sh status reports container status on both nodes.
• ./start.sh logs [N] displays the last N lines from the head and worker logs.
• ./start.sh smoke sends a quick greedy completion to the running API.
• ./start.sh doctor reruns the full environment and recipe preflight.

When diagnosing startup problems, inspect .serve.log, .sglang.log, and .sglang-worker.log. The first records launcher output, while the latter two contain the head and worker container logs.

Understand the GB10 Memory Budget

Each DGX Spark exposes a unified physical memory pool shared by CUDA allocations, pinned host memory, Docker, and the operating system. The recipe therefore uses a conservative static memory fraction rather than treating host and GPU memory as independent resources.

The reported per-node layout includes approximately 62.5 GB for NVFP4 expert, dense, MTP, and vision weights; about 11 GB of pinned host memory for the fp8 PLE table; a 9 GB bf16 KV cache holding approximately 956,800 tokens; roughly 3.6 GB for Mamba/GDN state; and about 8 GB for CUDA graphs and NCCL or cuBLAS workspaces.

The resulting CUDA-visible allocation is approximately 95.6 GB, with around 17.5 GB available for additional work. The pinned PLE table is not visible as a per-process device allocation in nvidia-smi, but it still consumes the same physical DRAM. Monitor SGLang’s available_gpu_mem log value or the available column from free -h when evaluating real headroom.

Keep PLE offloading enabled

Do not add --no-ple-offload-embedding on GB10. The SGLang automatic rule keeps the PLE embedding table CPU-offloaded. Overriding it can place roughly 26 GB per rank of pinned host memory on the GPU side and overcommit the machine.

Preserve transient prefill headroom

The QSA indexer creates an fp32 logits workspace whose size scales with chunk × history. At a chunk size of 4096 and a 300,000-token history, this transient can require approximately 8 to 10 GB after related gather and top-k copies are considered.

The shipped values are deliberately conservative:

MEM_FRACTION_STATIC=0.82
CHUNKED_PREFILL_SIZE=1024
MAMBA_FULL_MEMORY_RATIO=0.3

Do not raise CHUNKED_PREFILL_SIZE above 1024 for the 900,000-token context without profiling peak transient memory against the available unified-memory budget. The repository reports that a chunk size of 4096 combined with MEM_FRACTION_STATIC=0.85 was sufficient to wedge the machines.

Avoid allocator experiments

The recipe explicitly advises against setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True. This configuration is unproven for the deployment and was not present in the known-good DGX Spark recipes used by the authors.

Adjust Important Tunables

Set tunables in .env or export them in the shell. Shell values win. The settings most likely to need adjustment are:

• PORT: Defaults to 8888 and binds the API to all interfaces.
• CONTEXT_LENGTH: Defaults to 900000.
• MAX_RUNNING_REQUESTS: Defaults to 16, although the Mamba configuration limits practical concurrency to 14 slots at the default ratio.
• SPEC_STEPS, SPEC_TOPK, and SPEC_DRAFT: Default to 3, 1, and 4 for NEXTN speculative decoding.
• KERNEL_PATCH: Defaults to 1 so the SM121 fallback image is built and used.
• IMAGE: Uses a specified image directly and skips patching.
• CPUSET: Defaults to 5-9,15-19 for the GB10’s larger CPU cores. Set it to an empty value to disable pinning.
• EXTRA_ARGS: Appends additional SGLang arguments last, allowing last-value-wins overrides.

Handle Thinking and Tool-Call Loops

A known SGLang issue can affect sessions that combine default thinking with OpenAI tools and the qwen3_coder tool-call parser. The server may repeatedly emit token ID 0, which this tokenizer displays as !, until max_tokens is reached.

The preferred workaround is to disable thinking only for requests that send tools:

"chat_template_kwargs": {"enable_thinking": false}

To disable thinking server-wide while using the tool parser, stop the deployment and restart it with an override:

./start.sh stop
EXTRA_ARGS='--tool-call-parser qwen3_coder --default-chat-template-kwargs {"enable_thinking":false}' ./start.sh serve

If EXTRA_ARGS is already configured in .env, merge the options into that existing setting. Do not enable thinking again in the same request that provides tools. Without the parser, thinking can work, but tool calls appear as <tool_call> XML in ordinary content instead of structured message.tool_calls. The README also advises keeping agent temperature at or below 0.7 when using the workaround.

Advanced Reliability and Performance Tips

Use speculative decoding for throughput

NEXTN speculative decoding is the main throughput driver in the measured setup. With the default 3/1/4 chain, each decode step verifies four draft tokens in one forward pass. The tested cluster peaked near 117 aggregate tokens per second with two streams and then plateaued as more streams contended for the remaining memory.

Choose determinism when required

Greedy decoding is not bit-reproducible in the default configuration. Near-tie tokens can occasionally change between runs, likely because of floating-point reduction variance in NVFP4 MoE GEMMs on SM121. For slower but deterministic behavior, restart with:

EXTRA_ARGS="--moe-runner-backend triton" ./start.sh serve

Interpret the TileLang warning correctly

During JIT compilation, TileLang may report that Logits(bx, position) is written by multiple threads. The repository identifies this as a false positive because the calculated positions are disjoint, although the checker cannot prove that property when group is a runtime value. It can be silenced with PassKey.TL_DISABLE_DATA_RACE_CHECK if necessary.

Protect external access

The API binds to 0.0.0.0, making it reachable through any permitted interface. The README describes access from LAN or Tailscale clients but does not add an authentication layer. Apply appropriate host firewall, private-network, or proxy controls when the endpoint should not be generally reachable.

Conclusion

This repository turns a difficult two-node deployment into a repeatable workflow. Its main contribution is the combination of an SM121-compatible Qwen Sparse Attention fallback, carefully constrained GB10 unified-memory settings, direct RoCEv2 tensor parallelism, and a single idempotent operations script.

Start with the supplied defaults, run ./start.sh doctor before deployment, and avoid dummy weights, disabled PLE offloading, oversized prefill chunks, and experimental allocator settings. Once the cluster is ready, applications can use the local server through familiar OpenAI-compatible chat and completion interfaces.