Run Qwen3.8-Flash-Next on a Colab A100 with an OpenAI-Compatible API

What collabosm does

collabosm runs Qwen3.8-Flash-Next on a single Colab A100 80 GB High-RAM instance. The model is a 125B-A6B mixture-of-experts model with a hybrid Gated-DeltaNet and full-attention architecture and a native context length of 262,144 tokens.

The project provisions or restores the Colab VM, installs a pinned ExLlamaV3 runtime, downloads pinned model weights, starts an OpenAI-compatible HTTP server, and exposes it through a Cloudflare tunnel. Clients connect to the tunnel URL under /v1 with a generated bearer key.

ExLlamaV3 is the only inference engine used by the repository. The API server itself is implemented by collabosm because ExLlamaV3 does not include an HTTP server.

Key features

• Single-GPU deployment: runs the approximately 63.6 GiB of VRAM-resident weights on one A100 80 GB High-RAM runtime.
• OpenAI-compatible interfaces: supports both /v1/chat/completions and /v1/responses, including incremental streaming.
• Safe Colab restoration: reattaches orphaned assignments instead of trusting disposable local CLI session records or creating duplicate billable VMs.
• Hardware validation: refuses and stops an incompatible 40 GB A100 before loading the model.
• Persistent prompt caching: keeps one long-lived ExLlamaV3 Generator so the page table and pinned-RAM cache survive across requests.
• Two-tier cache design: combines GPU KV capacity with an optional pinned host-RAM cache for quickly resuming idle conversations.
• Local protocol testing: provides a fake engine that exercises the real HTTP handler without a GPU or model weights.
• Optional vision: can load the model's vision component and accept guarded image inputs.
• Local client and UI: includes terminal chat, a browser UI, status pages, stream-derived performance metrics, and a proxy that keeps the upstream key out of the browser.

Measurements reported by the project include up to 3,882 prefill tokens per second for a 30K prompt with GCS=8192, 97.4 decode tokens per second with MTP depth 4 at 30K context, and 90.1 tokens per second at 114K context. Consult docs/MEASURED.md for provenance and caveats rather than treating these figures as guarantees.

Requirements and capacity planning

Before starting, make sure your environment satisfies all of the following requirements:

• A Colab plan capable of allocating an A100.
• The HIGH_RAM machine shape. A standard 40 GB A100 cannot load this model.
• Approximately 110 GiB of Colab disk space for the weights.
• The Google Colab command-line tool, installed with uv tool install google-colab-cli.
• Optionally, the GitHub CLI if you plan to fork or push the repository.

uv tool install google-colab-cli

The High-RAM requirement is mandatory. Do not identify a suitable card by searching hardware output for “80,” because an A100 can report compute capability sm_80 even when it has only 40 GB of VRAM. collabosm compares the reported vram_GiB value instead.

The project was developed on a Colab Pro tier with approximately 200 compute units per month. Its measured A100 High-RAM cost was 7.52 CU per hour, approximately $0.75 per hour in the documented environment. Pricing and allocation availability may change independently of the repository.

Start the server

1. Open the repository directory

Download or clone the collabosm repository, then run the following commands from its root directory. The scripts expect repository-relative paths such as scripts/up.sh, scripts/bootstrap.sh, and scripts/serve.sh.

2. Provision, bootstrap, and serve

bash scripts/up.sh

This single command restores an existing assignment when possible or creates a new High-RAM assignment, uploads the project, installs the pinned runtime, downloads the pinned weights, starts the API, launches the Cloudflare tunnel, and waits for a successful health check.

A successful invocation exits with status 0 only after both the API and public tunnel URL are ready. The command prints the tunnel URL and API key. The key is also stored on the VM at /content/api-key.txt.

The runtime is selected from a pinned prebuilt ExLlamaV3 wheel that matches the Colab Python, PyTorch, and CUDA environment. The weights are fetched from Hugging Face at a pinned revision. If the Xet download path stalls, the README identifies HF_HUB_DISABLE_XET=1 as the fallback.

3. Record the connection settings

base_url = https://<host>.trycloudflare.com/v1
api_key  = <printed API key>
model    = qwen3.8-flash-next-exl3

The quick tunnel hostname may change between launches. For a stable hostname, configure a named Cloudflare tunnel with TUNNEL_TOKEN and pair it with PUBLIC_URL for the address shown in status output.

4. Inspect the running configuration

Request GET /v1/status to see the parameters that the running process actually received, including cache settings, launch parameters, uptime, vision availability, and image-input policy.

The repository notes that its current defaults have not all been verified together. Check the live status endpoint and consult docs/MEASURED.md before assuming a particular combination is validated.

Configure caches and generation

All primary launch controls are environment variables passed to scripts/up.sh. For example:

SESSION=mybox CACHE_SIZE=524288 CPU_CACHE_GB=8 bash scripts/up.sh

• SESSION: local session name; default collabosm.
• CACHE_SIZE: total KV tokens across jobs; default 500224 and required to be a multiple of 256.
• CACHE_QUANT: KV cache bit width; default 4, with values from 2 through 8 allowed.
• CPU_CACHE_GB: pinned-RAM second-tier KV page cache; default 32 GiB, with 0 disabling it.
• RECURRENT_CACHE_GB: host-RAM storage for Gated-DeltaNet checkpoints; default 24 GiB.
• GCS: generator chunk size and a major prefill-performance control; default 8192.
• NDT: multi-token prediction draft depth; default 4.
• RUNTIME: wheel for the prebuilt runtime or source for a source-based setup.
• TUNNEL_TOKEN: optional named-tunnel token.

The host-RAM caches are not free capacity. The default CPU cache is allocated as pinned memory in full, and the recurrent cache also consumes substantial RAM. They coexist with the repository's documented 36.4 GiB n-gram table.

The pinned-RAM tier improves the resumption time of idle conversations whose unreferenced pages were evicted. It does not increase the number of simultaneously live contexts because live pages cannot be evicted as if host RAM were ordinary swap.

Call the OpenAI-compatible API

List the model

curl https://<host>.trycloudflare.com/v1/models \
  -H "Authorization: Bearer <api-key>"

The endpoint returns the single model identifier qwen3.8-flash-next-exl3. Authentication is required for this API route, while /health is unauthenticated and returns plain ok.

Create a chat completion

curl https://<host>.trycloudflare.com/v1/chat/completions \
  -H "Authorization: Bearer <api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.8-flash-next-exl3",
    "messages": [
      {"role": "user", "content": "Reply with exactly: pong"}
    ],
    "max_tokens": 16
  }'

The chat interface returns content under choices[].message.content, a stop or length finish reason, and usage information that can include cached prompt tokens.

Stream a completion

Set stream to true to receive Server-Sent Events while the model is decoding. For chat completions, the stream ends with [DONE]. Setting stream_options.include_usage requests usage data in the streamed response.

{
  "model": "qwen3.8-flash-next-exl3",
  "messages": [{"role": "user", "content": "Explain prompt caching briefly."}],
  "stream": true,
  "stream_options": {"include_usage": true},
  "max_completion_tokens": 200
}

The /v1/responses endpoint also supports streaming. Its event sequence includes creation, progress, output-item, content-part, text-delta, completion, and terminal response events. Every streamed event carries a monotonic sequence_number, while deltas carry the item and content indexes clients need to bind them correctly.

Use thinking mode

Thinking is disabled by default for compatibility with ordinary chat clients. Enable it with enable_thinking: true or with a supported reasoning_effort value of xhigh, medium, or low. On the chat surface, the reasoning trace is returned separately in message.reasoning_content and does not leak into the normal content field.

temperature and top_p are accepted but currently ignored because the server constructs jobs with sampler=None. Thinking controls, max_tokens, and stop sequences do work.

Use the included client

The local collabosm.py client uses only the Python standard library and requires no separate installation. Initialize its configuration with the tunnel endpoint and generated key:

python collabosm.py config --init \
  --endpoint https://<host>.trycloudflare.com/v1 \
  --api-key <key>

You can then chat from the terminal or launch the local browser interface:

python collabosm.py chat "hello"
python collabosm.py ui
python collabosm.py start

ui serves the page and proxy at http://127.0.0.1:8790. start opens the chat and status windows. The client deliberately does not start or stop the Colab VM, so opening it cannot spend compute units by itself.

Give an existing OpenAI-compatible client the local proxy rather than the VM endpoint when you want the proxy to inject the real bearer key:

base URL : http://127.0.0.1:8790/v1
API key  : anything
model    : qwen3.8-flash-next-exl3

This design keeps the actual key inside the local proxy process, avoids a broad CORS policy, and allows the browser UI to be updated independently of the VM.

Test the protocol without a GPU

Use the development stub to validate request and streaming behavior without paying for an A100 or downloading weights:

python scripts/dev_stub.py --port 8099 --chunk 24 --delay 0.01
python scripts/check_surface.py --base http://127.0.0.1:8099/v1

The stub imports the real shipping handler and replaces only its engine-fragment function. The surface checker performs 13 checks, including early deltas, terminal events, matching chat and Responses output, required item indexes, and monotonic sequence numbers. It exits with status 1 when a check fails.

Add --think to the stub command to exercise the reasoning-item lifecycle. This is especially useful before changing stream event shapes or connecting strict clients such as Codex CLI.

Enable and test vision carefully

The model pack is multimodal, but vision is disabled by default because the vision tower's additional VRAM cost has not been measured on the already heavily occupied card. Enable the separate vision component explicitly:

VISION=1 bash scripts/up.sh

Optional image controls include:

IMAGE_URLS=1
MAX_IMAGE_BYTES=12582912
MAX_IMAGES=8

Both supported API dialects accept standard OpenAI image parts. Data URLs are accepted when vision is enabled. Remote HTTP or HTTPS image URLs remain disabled unless IMAGE_URLS=1.

{
  "role": "user",
  "content": [
    {"type": "text", "text": "What colour is this image?"},
    {
      "type": "image_url",
      "image_url": {"url": "data:image/png;base64,..."}
    }
  ]
}

When remote URLs are enabled, the server resolves the hostname and requires every address to be globally routable. It rejects loopback, private, link-local, and metadata addresses and does not follow redirects. Filesystem paths are never accepted. If an image cannot be embedded, the server returns a JSON 400 error with type vision_unavailable rather than silently ignoring the image.

Check GET /v1/status to discover whether vision is active, then verify either enabled or deliberately refused behavior with:

python scripts/check_vision.py \
  --base https://<host>.trycloudflare.com/v1 \
  --key <key>

Concurrency and performance tips

The API uses one long-lived generator guarded by a lock. Requests are therefore serialized: multiple streams are queued rather than decoded in parallel. A batch size greater than one has not been tested, and the published measurements used one slot.

Measured live-session capacity depends strongly on context length:

• 500K tokens per stream: 1 live session.
• 262K tokens per stream: 2 live sessions.
• 131K tokens per stream: 5 live sessions.
• 32K tokens per stream: 11 live sessions.
• 16K tokens per stream: 14 live sessions.

Each live slot consumes approximately 546 MiB of Gated-DeltaNet state before KV storage. For short contexts, recurrent state becomes the limiting factor; the practical documented range is roughly 8 to 14 live slots, although requests are still serialized by the current API lock.

For prefill performance, GCS was the largest lever found by the project. However, larger or altered cache settings should be validated against VRAM and host-RAM use rather than copied blindly.

Avoid constructing a new Generator for every request. The page table and CPU page cache are created inside Generator.__init__; rebuilding it would silently discard prompt-cache reuse and force repeated prefills. The included server already maintains the correct long-lived instance.

Troubleshooting

The CLI says there are no active sessions

A local session record can disappear when the runtime proxy token expires even though the VM remains active and billable. Use the repository's restoration workflow rather than creating another VM manually. scripts/restore.py queries server-side assignments and reattaches the orphaned instance.

The model does not fit

Confirm that the runtime has approximately 80 GB of VRAM and is the High-RAM shape. The restore script is designed to reject and stop a 40 GB assignment before model loading.

The API is healthy but no public URL appears

up.sh distinguishes a healthy VM with no published tunnel URL from full readiness. Exit status 9 indicates this case. Inspect the serving and tunnel logs rather than assuming the endpoint is publicly reachable.

Requests appear to ignore sampling controls

This is expected for temperature and top_p in the current server. They are parsed but not forwarded to a sampler.

Streaming compatibility breaks

Reproduce the issue with scripts/dev_stub.py, then run scripts/check_surface.py. The real handler is under test, so this catches protocol regressions without provisioning the model.

Stop the VM and control costs

When you finish, stop the metered runtime immediately:

bash scripts/down.sh

This is the project's most important cost-control command. Closing a browser, losing a local CLI record, or disconnecting from Colab does not prove that billing has stopped.

The optional frontend control plane adds confirmation before provisioning, a local CU ledger, idle auto-stop, a maximum session duration, cancellation, and cleanup when provisioning fails after assignment. You can rehearse that workflow without a card or compute units:

python frontend/server.py --fake-provision
python frontend/server.py --mock
python scripts/dev_stub.py --port 8099 &
python frontend/server.py --mock --backend http://127.0.0.1:8099

Conclusion

collabosm packages the difficult parts of running Qwen3.8-Flash-Next on Colab: requesting the correct A100 shape, recovering orphaned assignments, installing a matching ExLlamaV3 runtime, loading pinned weights, preserving caches, exposing a secured tunnel, and serving two OpenAI-compatible API dialects.

Start with bash scripts/up.sh, verify the live configuration through /v1/status, connect through the printed /v1 endpoint, and always finish with bash scripts/down.sh. For production-like experimentation, pay particular attention to serialized requests, unimplemented sampling controls, cache memory costs, vision's optional VRAM load, and the distinction between idle-cache resumption and true live-session capacity.