Run GLM-5.3-Flash Across Two DGX Sparks with TensorFold

What This Project Does

This project serves GLM-5.3-Flash across two NVIDIA DGX Spark systems using TensorFold v0.5.0. Each Spark runs one distributed rank, while the first Spark also hosts an OpenAI-compatible HTTP API.

The default configuration supports four concurrent requests, a maximum context window of 1,048,576 tokens per request, image and video input, tool calling, structured output, tokenization endpoints, and Prometheus metrics. Communication between the two systems uses a direct ConnectX-7 RoCE link.

The deployment uses the Mia-AiLab GLM-5.3-Flash EXL3 checkpoint. Routed experts are stored at approximately four bits per weight, while other checkpoint weights are BF16. The complete checkpoint is approximately 176 GB.

Key Features

• OpenAI-compatible API: Use chat completions, completions, responses, and model-listing endpoints.
• Long context: Up to 1,048,576 prompt-and-reply tokens for an individual request under the default FP8 KV configuration.
• Concurrent inference: Decode as many as four requests together with the default DFlash2 drafter.
• Shared cache pool: Concurrent conversations draw from one FP8 KV pool containing approximately 2.1 to 2.7 million tokens, depending on available memory.
• Multimodal requests: Submit JPEG, PNG, WebP, MP4, WebM, MOV, MKV, and other FFmpeg-decodable video inputs.
• Reasoning controls: Select low, high, or maximum reasoning effort, or disable thinking for a request.
• Agent features: Use tool calling and schema-constrained structured outputs through xgrammar.
• Operational endpoints: Inspect health, tokenize or detokenize input, and collect Prometheus metrics.
• Automated deployment: One command prepares both machines, launches both ranks, performs a smoke test, and reports the API address.

Hardware and Software Requirements

You need two DGX Sparks, or two GB10 systems with 128 GB of unified memory each. Approximately 110 GiB must be free on each GPU when the server starts, so stop other substantial GPU workloads first.

Network requirements

Connect the ConnectX-7 ports with a QSFP cable and assign each endpoint an IPv4 address in the same private subnet. For example, the head could use 192.0.2.1/24 and the worker 192.0.2.2/24. The addresses must be reachable with ping, and a RoCE v2 GID must be available.

If both CX7 ports are connected and addressed, the project can use both rails. According to the README measurements, a prompt chunk's all-gather is approximately 1.8 times faster with two rails.

SSH and system requirements

• Key-based SSH access from the head Spark to the worker.
• Docker with the NVIDIA container runtime on both systems.
• Your account in the docker group on both systems.
• rsync installed on both systems.
• Approximately 205 GB of disk space on each Spark.
• Optionally, the Hugging Face hf CLI and a Hugging Face token on the head.

Create an SSH key if necessary, copy it to the worker, and verify noninteractive access:

ssh-keygen -t ed25519
ssh-copy-id user@192.0.2.2
ssh -o BatchMode=yes user@192.0.2.2 true

The DFlash2 drafter is licensed CC BY-NC-ND 4.0 and is for non-commercial use only. Select the checkpoint's MTP drafter before the first setup if you need to avoid downloading or using DFlash2.

Install and Configure the Project

1. Clone the repository on the head

git clone https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold.git
cd GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold
cp scripts/local.sh.example scripts/local.sh

The worker does not need its own checkout. The setup scripts transfer the required container image and model files from the head.

2. Set the worker address

Edit scripts/local.sh and set WORKER to the worker's SSH target:

WORKER=user@192.0.2.2

The distributed ranks communicate over the route used to reach the worker. If WORKER refers to an address on another network, set FABRIC_PEER to the worker's private CX7 address:

WORKER=user@worker-hostname
FABRIC_PEER=192.0.2.2

3. Choose a drafter if necessary

The default is DRAFTER=dflash2, which permits up to four concurrent requests and provides a measured decode improvement of roughly 5 to 10 percent. To use the checkpoint's MTP head instead, configure it before the first start:

DRAFTER=mtp

MTP avoids the DFlash2 license and download, but it serves one request at a time by default.

4. Start both ranks

./start.sh

On the first run, the script checks both systems, obtains or builds the approximately 25 GB container image, downloads the checkpoint and drafter on the head, copies them to the worker, and compiles CUDA kernels into ~/.cache/tensorfold-glm53.

Subsequent starts generally spend two to six minutes loading approximately 80 GiB of weights onto each Spark. The launcher follows rank 0's log, reports loading progress, runs a distributed smoke test, and prints a live endpoint on port 8888.

To prepare both systems without starting the API, run:

scripts/prepare.sh

To force a clean image rebuild, use:

scripts/prepare.sh --rebuild

Make Your First API Request

The default model identifier is GLM-5.3-Flash-EXL3, and the base URL is http://<head-address>:8888/v1.

List available models

curl -s http://<head-address>:8888/v1/models

Send a chat completion

curl -s http://<head-address>:8888/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "GLM-5.3-Flash-EXL3",
    "messages": [
      {"role": "user", "content": "Write a Python fibonacci function."}
    ],
    "max_tokens": 2000
  }'

The model thinks before answering by default. Reasoning is returned in reasoning_content, while the final answer appears in content. Allocate enough output tokens for both phases. If content is empty, the model probably consumed the output allowance while reasoning.

Control reasoning and sampling

Set reasoning_effort to low, high, or max. Use none to request a direct answer without thinking:

curl -s http://<head-address>:8888/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "GLM-5.3-Flash-EXL3",
    "messages": [{"role": "user", "content": "Summarize TCP in two sentences."}],
    "reasoning_effort": "low",
    "temperature": 0,
    "max_tokens": 1000
  }'

The API also accepts temperature, top_p, top_k, min_p, and seed. Without an explicit seed, the sampler derives its key from the prompt, so identical requests produce the same reply.

Send Images and Videos

The GLM vision tower runs on rank 0 by default. Images and videos are represented as OpenAI-style content parts in a user message.

Analyze a local image

IMG=$(base64 -w0 photo.jpg)
curl -s http://<head-address>:8888/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "GLM-5.3-Flash-EXL3",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,'"$IMG"'"}},
        {"type": "text", "text": "What is in this picture?"}
      ]
    }],
    "max_tokens": 2000
  }'

By default, the server accepts media through data URLs. Set VISION_URLS=1 if the server should also fetch public HTTPS URLs.

Supported media limits

• Up to 50 images per request, with a 10 MB limit per image and 64 MB total image data.
• Up to 4 videos per request, with a 64 MB limit per video and 96 MB total video data.
• Videos may contain up to one hour of footage each.
• The request body may be up to 96 MiB, leaving roughly 70 MB for base64-encoded media.
• A picture receives at most 2,048 tokens, while all pictures in one request share a default 16,384-token budget.
• Video sampling defaults to two frames per second and at most 128 frames across a clip.

For video, replace the image content part with a video_url part containing a data URL such as data:video/mp4;base64,....

If you only need text, set VISION=0. This avoids loading the vision tower and leaves approximately 1.8 GiB more memory on rank 0 for cache use.

Operate and Monitor the Server

Restart and stop

./start.sh restart
./stop.sh

A restart is required after changing persistent settings. Stopping the service releases GPU memory on both machines.

Inspect logs

docker logs -f glm53-flash-tf
ssh <worker> docker logs -f glm53-flash-tf

The first command follows rank 0 on the head. The second follows rank 1 and is especially useful when the worker exits unexpectedly.

Check health and metrics

curl -s http://<head-address>:8888/health
curl -s http://<head-address>:8888/metrics

The health response reports the busy state, active streams, and shared-pool capacity, including pool_tokens and pool_free_tokens. The metrics endpoint exposes Prometheus request, latency, time-to-first-token, and health figures.

Understand Context and Shared KV Memory

The default configuration uses an FP8 KV cache and gives each request a maximum window of 1,048,576 tokens. Four requests share a pool measured at 2,684,928 tokens at one documented startup, although the exact capacity depends on free memory.

A single request can use the complete window, but all active and retained conversations must fit in the common pool. For example, two conversations can each approach one million tokens, while shorter concurrent conversations can share the remaining capacity. A request that cannot yet be placed waits for other requests to finish.

Use CONTEXT=0 to calculate the largest window that fits, but note that this leaves no memory for retaining other conversations' prompt states. Switching to exact BF16 KV storage significantly reduces the available window.

Unified-memory exhaustion can freeze a Spark rather than producing a clean allocation failure. Keep the default reserve unless you understand the memory trade-off, and raise MEMORY_RESERVE_GIB when other workloads share the machines.

Advanced Configuration

Settings can be supplied through the environment, scripts/local.sh, or a .env file beside start.sh. The precedence order is the environment, then scripts/local.sh, then .env, and finally the repository defaults.

Change concurrency for one run

PARALLEL=2 ./start.sh restart

PARALLEL may be set from one to four when DFlash2 is enabled. MTP defaults to one request at a time.

Change the context window

CONTEXT=524288 ./start.sh restart

A smaller context can reduce cache pressure when the full million-token window is unnecessary.

Increase the default output allowance

./start.sh restart --max-tokens 16384

Additional command-line arguments are passed to tensorfold serve after the configured defaults, so they take precedence. Run ./start.sh --help to inspect supported arguments.

Force or skip preparation

PREPARE=1 ./start.sh restart
PREPARE=0 ./start.sh restart

The first command forces the setup check and preparation process. The second skips it.

Use prompt reuse effectively

Shared-prefix reuse is enabled by default with SHARED_PREFIX=1. Repeated system prompts and identical retained prompts can resume from cached state rather than being prefetched again. In the documented measurements, a repeated 64K-token prompt dropped from 34 seconds to under 0.07 seconds, while reuse of a 7.9K-token system prompt reduced startup from 4.24 seconds to 0.13 seconds.

Select cache and weight precision

• KV=fp8 is the default and enables the largest context window.
• KV=bf16 uses exact cache storage but substantially shortens the available window.
• DENSE=q4 is the default for dense checkpoint components.
• DENSE=fp8 and DENSE=bf16 are available alternatives, with different memory implications.

Additional API Capabilities

Besides chat completions, the server exposes /v1/completions, /v1/responses, /v1/models, /tokenize, /detokenize, /health, and /metrics. Tokenization routes are also available under /v1/. The server does not provide an Anthropic-compatible /v1/messages endpoint.

Tool calling supports tools and streamed tool_calls, with arguments typed according to the provided schema. Structured generation supports json_object, json_schema, guided JSON, regular expressions, choices, grammars, and other structured-output fields enforced through xgrammar after reasoning completes.

If a prompt plus max_tokens exceeds the configured window, the API returns HTTP 400 with the OpenAI-style error code context_length_exceeded.

Troubleshooting Common Startup Problems

• Insufficient memory: If startup reports that only a limited number of GiB are available, inspect docker ps and stop other GPU workloads on both Sparks.
• No RoCE device or route: Confirm the CX7 interfaces have IPv4 addresses with ip -4 addr. If SSH uses another network, configure FABRIC_PEER with the worker's CX7 address.
• Worker cache is not writable: Correct ownership of the worker's Hugging Face cache if a previous container left it owned by root.
• One rank exits: Check logs on both machines. The launcher normally stops the remaining rank if either side fails.
• Setup changed: After modifying patches or changing TensorFold, the base image, or related build inputs, run scripts/prepare.sh --rebuild.

Performance Expectations

In the README's default two-Spark measurements, prose decoding ranged from 60.4 tokens per second for one request to 108.8 aggregate tokens per second for four concurrent requests. Structured generation ranged from 114.7 to 227.9 aggregate tokens per second. Prefill was approximately 1,952 tokens per second for an 8,219-token prompt and 1,015 tokens per second for a 981,841-token needle-in-a-haystack prompt, where the needle was found.

These figures were measured with four streams, FP8 KV cache, four-bit dense weights, DFlash2 and copy drafts, vision enabled, and GPU clocks capped at 2,200 MHz. Treat them as reference results for that configuration rather than guarantees for every environment.

Conclusion

This deployment packages a complex two-node TensorFold configuration behind a practical startup workflow and an OpenAI-compatible API. Once the CX7 link, SSH access, Docker runtime, storage, and worker address are ready, ./start.sh handles preparation, distributed launch, validation, and serving. From there, you can use standard API clients while tuning concurrency, context, reasoning, multimodal limits, and memory behavior for your workload.