Run Qwen3.8 Flash Next on a Single DGX Spark with TensorFold

Overview

This project serves Qwen3.8 Flash Next from one NVIDIA DGX Spark through an OpenAI-compatible API. It uses TensorFold v0.3.6.3, an MLX 4-bit checkpoint with an MTP draft head, and a collection of patches designed to accelerate prompt processing without changing generated text.

The default configuration supports five concurrent requests, each with the model's full 262,144-token context window. It also loads the model's own vision tower on CUDA, allowing chat messages to contain images and videos.

• API model ID: Qwen3.8-Flash-Next
• Default endpoint: http://<spark-address>:8888/v1
• Parallel streams: 5
• Context per stream: 262,144 tokens
• Total KV pool: 1,310,720 tokens
• KV cache: int8 by default
• Input modes: text, images, and videos

Key Features

Long-context concurrent inference

TensorFold assigns each stream its own full context cache. With the default settings, five streams each receive a 262,144-token prompt-and-reply window. The resulting int8 KV pool consumes approximately 23.4 GiB.

OpenAI-compatible serving

The server exposes endpoints such as /v1/models and /v1/chat/completions. OpenAI-compatible clients can connect by changing their base URL and selecting the model ID Qwen3.8-Flash-Next.

Multimodal prompts

User messages can contain OpenAI-style image_url and video_url content parts. Supported image formats include JPEG, PNG, and WebP. Videos may use MP4, WebM, MOV, MKV, or another format that FFmpeg can decode.

Patched prompt processing

The included TensorFold patches add SSD read-ahead, native threaded reads, improved sparse-attention block selection, configurable prompt chunks, copy drafts, live token counters, and Flash Next vision support. According to the repository's measurements, prompt processing is approximately 1.7 times faster than the compared unpatched version, while tested text replies remain byte-identical.

System Requirements

Prepare the host before cloning the repository:

• An NVIDIA DGX Spark or another GB10 system with 128 GB of unified memory.
• Approximately 115 GiB of free memory when starting the default configuration.
• Docker with the NVIDIA container runtime.
• A user account that belongs to the docker group.
• Approximately 160 GB of free disk space on a fresh system.
• Optional: the Hugging Face hf CLI for faster resumable downloads.
• Optional: a Hugging Face token in ~/.cache/huggingface/token or the HF_TOKEN environment variable.

The default deployment is intended for an otherwise idle DGX Spark. On unified-memory systems, exhausting memory may freeze the machine instead of producing a normal allocation error.

Installation and First Launch

1. Clone the repository

git clone https://github.com/MiaAI-Lab/Qwen3.8-Flash-Next-Single-DGX-Spark-TensorFold.git
cd Qwen3.8-Flash-Next-Single-DGX-Spark-TensorFold

2. Start the server

./start.sh

The first launch performs the complete setup. It pulls the prebuilt container image when available, downloads the roughly 106 GiB checkpoint, verifies it, and compiles CUDA kernels for GB10. The prebuilt image is approximately 11 GB, while the checkpoint occupies about 114 GB in the Hugging Face cache.

During startup, the script displays setup progress, server logs, weight-loading progress, and memory estimates. It then runs a smoke test. A successful launch ends with a message stating that Qwen3.8-Flash-Next is live on port 8888.

Later launches skip completed setup work and take roughly 2.5 minutes to load the weights, according to the README.

3. Prepare without launching

If you want to download and build everything in advance, run:

scripts/prepare.sh

To rebuild the container image from scratch:

scripts/prepare.sh --rebuild

The preparation script checks Docker, the NVIDIA runtime, and disk space. It then obtains or builds the TensorFold image, downloads the checkpoint into the Hugging Face cache, and validates the model with tensorfold info.

Basic API Usage

Check the available model

curl -s http://<spark-address>:8888/v1/models

Replace <spark-address> with the hostname or IP address of the DGX Spark. When calling from the Spark itself, you can use 127.0.0.1.

Send a chat completion request

curl -s http://<spark-address>:8888/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3.8-Flash-Next",
    "messages": [
      {
        "role": "user",
        "content": "Write a Python fibonacci function."
      }
    ],
    "max_tokens": 1000
  }'

The model thinks before answering by default, and its reasoning uses the response token allowance. Set max_tokens high enough to accommodate both reasoning and the final answer. Reasoning is returned in reasoning_content, while the final response appears in content.

Connect an OpenAI-compatible client

Configure the client with the following values:

• Base URL: http://<spark-address>:8888/v1
• Model: Qwen3.8-Flash-Next

The API supports streaming, typed tool calls, reasoning content, image input, and video input. Tool parameters such as arrays are returned using their corresponding JSON types.

Sending Images

By default, media must be supplied as data URLs. The following shell example encodes a JPEG and asks the model to describe it:

IMG=$(base64 -w0 photo.jpg)
curl -s http://<spark-address>:8888/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3.8-Flash-Next",
    "messages": [{
      "role": "user",
      "content": [
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/jpeg;base64,'"$IMG"'"
          }
        },
        {
          "type": "text",
          "text": "What is in this picture?"
        }
      ]
    }],
    "max_tokens": 2000
  }'

A request can contain up to four images. Each image may be up to 10 MB, with a combined limit of 20 MB. All images together may use up to 4,096 tokens. Setting "detail": "low" limits an image to 256 tokens.

Sending Videos

Videos use a video_url content part:

{
  "type": "video_url",
  "video_url": {
    "url": "data:video/mp4;base64,..."
  }
}

Each request may contain up to two videos. A video may be up to 64 MB, with a combined request limit of 96 MB and up to one hour of footage. By default, the server samples two frames per second, up to 256 frames spread across the video, and permits up to 16,384 video tokens per request.

Image and video prompts are not retained for prefix reuse. Ordinary text requests are unaffected by enabling vision.

Allow public HTTPS media URLs

Data URLs are the default. To let the server fetch public HTTPS image and video URLs, restart it with:

VISION_URLS=1 ./start.sh restart

Only enable remote fetching when it is appropriate for your deployment's security and network policy.

Run in text-only mode

Disable the vision tower to serve text only:

VISION=0 ./start.sh restart

Text-only mode does not load the 0.84 GiB vision tower and uses 4,096-row prompt chunks instead of the vision-enabled default of 2,048 rows.

Managing the Server

Use the repository's wrapper scripts for routine operations:

./start.sh restart
./stop.sh
docker logs -f qwen38-flash-next-tf
curl -s http://<spark-address>:8888/health

• ./start.sh restart validates setup and arguments before replacing the running server.
• ./stop.sh stops the container and releases GPU memory. Active requests are interrupted.
• docker logs -f follows the TensorFold server log.
• /health reports the busy state and live token totals.

If the server is already running, a plain ./start.sh leaves it alone. For a process manager such as systemd, use foreground mode:

FOREGROUND=1 ./start.sh

Configure Context, Concurrency, and KV Precision

Defaults are stored in scripts/config.sh. Override them with environment variables or pass TensorFold flags after start.sh.

Reduce concurrency for more headroom

PARALLEL=4 ./start.sh restart

Four int8 streams provide a KV pool of 1,048,576 tokens and have a reported startup estimate of approximately 97.7 GiB.

Use a shorter context window

./start.sh restart --context 131072

Extra command-line arguments are passed to tensorfold serve after the defaults, so they take precedence. Run ./start.sh --help to inspect the available options.

Choose another KV cache type

KV_DTYPE accepts bf16, int8, or int4:

PARALLEL=6 KV_DTYPE=int4 ./start.sh restart

The README notes that int4 changes outputs slightly and does not provide a quality measurement. Use int8 when you want the project's default balance. Use bf16 when full-precision KV storage is more important than stream count.

Understand startup protection

TensorFold calculates its budget from available memory minus a host reserve equal to one tenth of RAM. If a requested configuration does not fit, startup is refused before weights are loaded, and the error identifies a context window that should fit.

Control Thinking and Sampling

The default thinking-mode sampling values are temperature 1.0, top-p 0.95, and top-k 20. Requests can override temperature, top_p, top_k, and seed. A temperature of 0 enables greedy decoding.

Disable thinking for one request

{
  "model": "Qwen3.8-Flash-Next",
  "messages": [
    {"role": "user", "content": "Answer in one sentence."}
  ],
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 200
}

Set reasoning effort

{
  "chat_template_kwargs": {
    "reasoning_effort": "low"
  }
}

The template also accepts "xhigh". If reasoning effort is omitted, the template uses its medium default. A top-level OpenAI-style reasoning_effort field is ignored, so place it inside chat_template_kwargs.

Advanced Performance Tips

Keep n-gram tables on SSD

The default PLE_ON_SSD=1 leaves 29.8 GiB of n-gram tables on SSD rather than placing them in unified memory. This preserves memory for the KV cache. Fast local storage is therefore important for the default configuration.

Tune prompt chunk size carefully

With vision enabled, TENSORFOLD_PREFILL_ROWS defaults to 2048 to leave room for the vision tower. Text-only mode defaults to 4096. The README reports that 4,096 rows can improve prompt processing by approximately 2–5% from prompts of around 3,000 tokens, but requires about 0.94 GiB more memory.

TENSORFOLD_PREFILL_ROWS=4096 ./start.sh restart

Increase this value only when sufficient memory headroom is available.

Control MTP and copy drafts

The default MTP configuration produces at most six drafts per round and stops a chain before a draft falls below 60% confidence. The copy-draft optimization is enabled by TENSORFOLD_MTP_COPY=1 and requires at least two parallel streams. It is intended to improve replies that quote or edit earlier prompt text.

Request serial reference decoding

To bypass drafting for an individual request and use TensorFold's one-token-at-a-time reference path, add:

{
  "draft": false
}

This is useful when comparing speculative decoding against the serial reference.

Force or skip preparation

PREPARE=1 ./start.sh restart
PREPARE=0 ./start.sh restart
PULL=0 scripts/prepare.sh

• PREPARE=1 forces the preparation check before restarting.
• PREPARE=0 skips preparation.
• PULL=0 builds the image locally rather than trying the prebuilt registry image.

After changing files in patches/, rebuild the image so those patches are baked into TensorFold.

Validation and Benchmarking

The repository includes utilities that call a running server. Their API URL defaults to http://127.0.0.1:8888.

tools/bench.py local-test
tools/needle.py
tools/toolcheck.py
tools/visioncheck.py

• tools/bench.py measures several prompt sizes and performs a short decode check.
• tools/needle.py hides a passphrase in an approximately 195,000-token prompt and verifies retrieval.
• tools/toolcheck.py checks that an array-valued tool parameter is returned as a JSON array.
• tools/visioncheck.py sends a generated image containing a red circle and blue square, then checks whether both are recognized.

To test from another machine, set API_URL:

API_URL=http://<spark-address>:8888 tools/bench.py remote-test

Expected Performance

The repository reports the following decode results on one DGX Spark with int8 KV caching, SSD-backed n-gram tables, and MTP drafting:

• One request: 62.4 tokens per second with a 152 ms time to first token.
• Two requests: 90.5 aggregate tokens per second.
• Four requests: 106.7 aggregate tokens per second.
• Five requests: 119.3 aggregate tokens per second, or 27.0 tokens per second per request.

Reported prefill speed ranges from approximately 2,503 tokens per second for an 8,229-token prompt to 2,200 tokens per second for a 131,110-token prompt. These figures are project measurements rather than guarantees and may vary with configuration, prompt structure, media input, storage, and available memory.

Conclusion

This repository packages the model, TensorFold runtime, CUDA-specific setup, performance patches, and operational scripts needed to serve Qwen3.8 Flash Next on one DGX Spark. Start with the default int8, five-stream configuration, verify the service through the OpenAI-compatible API, and then adjust concurrency, context size, vision, and KV precision according to your workload and memory headroom.