Run GLM-5.3-Flash on a Single 24 GB RTX 3090 with Host-RAM Offloading

Overview

glm53-flash-offload serves the 125.3 GB, EXL3 3.05-bpw build of GLM-5.3-Flash on a single 24 GB NVIDIA GPU plus host DDR4 memory. It exposes an OpenAI-compatible HTTP API while keeping the model’s routed mixture-of-experts weights primarily in pinned system RAM.

The project is a thin set of monkeypatches over stock exllamav3 1.5.1, supplemented by several small custom kernels. It does not require editing the exllamav3 source. The documented configuration was measured on an RTX 3090 connected through PCIe 4.0 x16 to an AMD EPYC 7443P system.

The central idea is to keep frequently used experts in an elastic GPU cache. Experts that miss the cache can be read directly from pinned host memory over PCIe. In fast mode, the coldest decode misses can instead be evaluated by an AVX2 CPU worker.

Key features

• Single-GPU deployment: serves GLM-5.3-Flash with one 24 GB RTX 3090-class GPU.
• OpenAI-compatible API: supports chat completions, text completions, model discovery, streaming, reasoning content, and OpenAI-style tool calls.
• Pinned zero-copy storage: all routed experts are held in pinned host RAM and can be accessed through CUDA unified virtual addressing.
• Elastic GPU expert cache: hot experts occupy available VRAM, while cache space can temporarily be returned for long-prefill activations and staging buffers.
• Prefill staging: non-cached experts are copied one layer ahead so PCIe transfer can overlap computation.
• Optional CPU tier: fast mode assigns selected cold decode misses to an AVX2 CPU kernel.
• Batched decode: up to four streams can decode together with the documented -ambs 4 configuration.
• Reproducible configuration: the model revision, container digest, exllamav3 version, Triton selections, benchmarks, and reference quality panel are pinned.

Choose between fast and exact mode

The server provides two operating modes through the GLM53_MODE environment variable.

Fast mode

fast is the default. It combines the GPU expert cache, zero-copy cache misses, AVX2 CPU processing for selected cold decode misses, batched decode, and fused decode kernels. On the measured host, it used approximately 218 GiB, or 234 GB, of host RAM.

The published campaign measured approximately 28.15 tokens per second for one decode stream, 31.57 aggregate tokens per second for two streams, and 33.98 aggregate tokens per second for four streams. An image smoke test of the current digest measured 28.88, 31.27, and 35.00 aggregate tokens per second respectively.

Quality warning: prefill is exact, but fast-mode decode is not bit-exact relative to stock exllamav3. A paired decode check reported mean KL 0.0053 and top-1 agreement of 0.977. Use exact mode when output equivalence is required.

Exact mode

exact disables the CPU tier and fast-only decode kernels. It retains the GPU expert cache and zero-copy host-memory misses. The reference panel measured top-1 agreement of 1.0000 and KL 0, and host-memory use was approximately 111 GiB, or 119 GB.

Exact mode is slower but uses much less RAM. The measured single-stream decode rate was 12.57 tokens per second at the shorter tested context, compared with roughly 7.5 to 8.4 tokens per second when stock exllamav3 placed non-fitting experts on its CPU worker.

Hardware and software requirements

GPU and driver

• One NVIDIA GPU with 24 GB of VRAM.
• An sm_86 GPU such as the RTX 3090 or A5000 is the measured target.
• The included sm_86 binaries also run on sm_89, including the RTX 4090, but that configuration was not measured.
• A CUDA 13.0-capable NVIDIA driver is required because the image uses CUDA 13.0.3.
• PCIe 4.0 x16 is recommended. Cache misses consume about 25 GB/s over the measured link.

Host memory

• Fast mode: 256 GiB total memory is the bare minimum when little else is running; 320 GB or more is recommended. If Docker memory limits are used, allow at least 230g.
• Exact mode: 128 GiB is the minimum, while 160 GB or more is recommended.

CPU and storage

• Fast mode requires an x86-64 CPU with AVX2, FMA, and F16C.
• Approximately 24 physical cores and eight-channel DDR4 or better are recommended for the measured fast-mode behavior.
• The checkpoint occupies about 117 GiB, or 125.3 GB, on disk.
• NVMe storage is recommended to reduce model-loading time.
• Docker, NVIDIA Container Toolkit, and Hugging Face CLI are needed for the documented setup.

Install and start the server with Docker

Step 1: Download the checkpoint

Download the pinned 3.05-bpw checkpoint into a local directory:

hf download turboderp/GLM-5.3-Flash-exl3 \
  --revision 3.05bpw \
  --local-dir /data/GLM-5.3-Flash-exl3-3.05bpw

You may instead mount an empty directory at /models. If config.json is missing, the container downloads the configured checkpoint into that directory.

Step 2: Launch fast mode

The following command starts the server on GPU 0 and publishes it on port 30000. It uses the image digest containing the streaming-delivery fix.

docker run -d \
  --name glm53 \
  --gpus '"device=0"' \
  --ulimit memlock=-1 \
  --shm-size 16g \
  -p 30000:30000 \
  -v /data/GLM-5.3-Flash-exl3-3.05bpw:/models \
  ghcr.io/0xsero/glm53-flash-offload@sha256:bb633b0bcb85573ad40b6c408af5e1036ae062c593e42479f4e4d2ab521e551d

The --ulimit memlock=-1 setting is recommended because the server pins large amounts of host memory. The published image also loaded without it on the measured driver, but that behavior should not be assumed for every host.

Step 3: Watch startup

docker logs -f glm53

Wait for a message containing loaded in ... s. Startup takes roughly two to three minutes on the documented system, depending on cache and host conditions.

Step 4: Confirm health

curl -s http://localhost:30000/health

Run exact mode

To prioritize output equivalence and reduce host-memory consumption, add -e GLM53_MODE=exact:

docker run -d \
  --name glm53-exact \
  --gpus '"device=0"' \
  --ulimit memlock=-1 \
  --shm-size 16g \
  -p 30000:30000 \
  -e GLM53_MODE=exact \
  -v /data/GLM-5.3-Flash-exl3-3.05bpw:/models \
  ghcr.io/0xsero/glm53-flash-offload@sha256:bb633b0bcb85573ad40b6c408af5e1036ae062c593e42479f4e4d2ab521e551d

Do not run the fast and exact examples on the same port simultaneously. Stop and remove the first container before switching modes:

docker rm -f glm53

Basic API usage

Send a chat completion

curl -s http://localhost:30000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"glm-5.3-flash","messages":[{"role":"user","content":"Explain zero-copy model offloading."}],"max_tokens":512}'

The default served model name is glm-5.3-flash. Unless explicitly configured, max_tokens defaults to the remaining context rather than being capped to a small fixed value.

Disable thinking

GLM reasoning is returned in reasoning_content. To disable thinking for a request, pass a chat-template option:

curl -s http://localhost:30000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"glm-5.3-flash","messages":[{"role":"user","content":"Reply with one sentence."}],"chat_template_kwargs":{"enable_thinking":false},"max_tokens":128}'

Stream a response

curl -N http://localhost:30000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"glm-5.3-flash","messages":[{"role":"user","content":"Write a short explanation of mixture-of-experts models."}],"stream":true,"max_tokens":512}'

Use the pinned bb633b0b... digest shown above. The older 1159044a... image generated at full speed but delivered streaming events at approximately half speed before flushing a backlog at the end.

List available models

curl -s http://localhost:30000/v1/models

Inspect runtime statistics

curl -s http://localhost:30000/stats

The statistics endpoint includes expert-cache hit information and CPU-tier counters. Additional endpoints include /v1/completions, /server_info, and SGLang-shaped /generate and /tokenize routes.

The deployment is text-only. The checkpoint’s vision tower is not loaded.

Use OpenAI-style tools

The chat endpoint accepts a tools field. Tool expressions produced in GLM’s internal <tool_call> format are converted into OpenAI-compatible tool_calls in the response. Client applications can therefore use the normal tool-calling loop: submit tool definitions, inspect the assistant’s tool call, execute the function, and send the tool result back as another message.

Adjust the context and server arguments

Arguments placed after the image name are appended to the server command, and the last value wins. For example, use a 65,536-token KV cache with:

docker run -d \
  --name glm53-64k \
  --gpus '"device=0"' \
  --ulimit memlock=-1 \
  --shm-size 16g \
  -p 30000:30000 \
  -v /data/GLM-5.3-Flash-exl3-3.05bpw:/models \
  ghcr.io/0xsero/glm53-flash-offload@sha256:bb633b0bcb85573ad40b6c408af5e1036ae062c593e42479f4e4d2ab521e551d \
  -cs 65536

A smaller KV cache can leave room for more expert-cache slots. The published benchmark configuration used a 131,072-token KV cache, a maximum batch size of eight, and an 8,192-token chunk size.

Install without Docker

A native installation requires exllamav3 1.5.1 built for the target GPU. From a clone of the project, run the extension installer and then start the provided entrypoint with the repository and model paths set:

scripts/install.sh
GLM53_ROOT=$PWD \
GLM53_MODEL_DIR=/path/to/model \
docker/entrypoint.sh

The installer builds the project’s custom extensions. Docker remains the most reproducible path because it pins CUDA, PyTorch, Triton, SGLang, exllamav3, the project commit, and the tested server configuration.

Validate quality and performance

Score the reference panel

From a repository clone, use the included standard-library-only benchmark client:

python3 bench/score_ref_panel.py \
  --url http://127.0.0.1:30000 \
  --panel reference/glm-5.3-flash-exl3-ref-panel.json \
  --out score.json

The teacher-forced panel should report top-1 agreement of 1.0000 and KL 0 in both fast and exact modes. This panel evaluates the prefill or teacher-forced path and does not establish that fast-mode autoregressive decode is exact.

Run the speed sweep

python3 bench/sweep.py \
  --url http://127.0.0.1:30000 \
  --card rtx-3090-24gb/glm-5.3-flash-offload \
  --template glm \
  --config "bb633b0b fast" \
  --prefill 8192 32768 \
  --conc 1 2 4 \
  --reps 3 \
  --dec-reps 2 \
  --no-early-exit \
  --out sweep.json

Check streaming delivery

python3 bench/stream_rate.py --url http://127.0.0.1:30000

A passing result should show text arriving evenly rather than approximately half the response appearing in a final burst.

Advanced tuning

Size the CPU tier

By default, GLM53_CT_CPUS=auto chooses one logical CPU per physical core and leaves the first two physical cores available for the operating system and the Python thread driving the GPU. On the measured 24-core host, this resulted in CPUs 2 through 23 and 22 worker threads.

For an explicit assignment, set values such as:

-e GLM53_CT_CPUS=2-23 \
-e GLM53_CT_THREADS=22

The CPU cost model uses GLM53_CT_B, measured as 0.104 milliseconds per expert with 22 threads. For N worker threads, the README recommends estimating it as 0.104 x 22 / N. For example, a 16-core host leaving two cores free would use 14 worker threads and a value near 0.16.

With fewer than approximately 12 physical cores, most misses are expected to return to the PCIe path, causing performance to approach exact mode. In that situation, exact mode may be the better choice.

Control CPU-tier memory layout

GLM53_CT_SWZ=1 makes the CPU tier use a separate block-contiguous copy of the experts. This is the measured fast configuration and adds about 114.2 GB of RAM use. Setting it to 0 makes the CPU tier read the pinned home copy directly, reducing memory use substantially, but the project reports that the CPU expert kernel benchmark is approximately 40% slower. End-to-end performance for this setting was not measured.

Tune cache reserves

GLM53_EC_RESERVE_GB leaves VRAM free after allocating the expert cache. Defaults are 1.5 GB in fast mode and 1.0 GB in exact mode. Keep the fast-mode reserve at 1.5 GB unless long prefills run out of memory.

GLM53_EC_ELASTIC_GB defaults to 10 GB. This amount of cache memory can be released for prefill activations and staging, then reclaimed for decode. GLM53_EC_STAGE_GB defaults to 2.6 GB per staging buffer, with two buffers used for overlap.

Use larger GPUs

The expert cache sizes itself from available VRAM after model loading and the configured reserve. A 32 GB or 48 GB card therefore receives more expert slots, which should increase the hit rate and improve decode performance. These larger-card results are not provided in the README.

An RTX 4090 can run the included sm_86 build, although it was not measured. Blackwell requires a rebuild with:

TORCH_CUDA_ARCH_LIST="8.6;12.0"

A new Triton pin may also be necessary on another architecture. If creating one, verify that the reference panel remains at top-1 1.0000 and KL 0.

Protect exact Triton selections

The image pins the KDA linear-attention prefill kernel selections through data/triton_pin_exact.json. Letting Triton select different configurations can alter floating-point rounding. Keep GLM53_TRITON_PIN at its default when reference-equivalent prefill behavior matters.

How the offloading pipeline works

1. Load routed experts into pinned host memory. Each layer receives an exact-size pinned arena, and exllamav3’s expert pointers reference the CUDA UVA aliases.
2. Populate an elastic CLOCK cache in VRAM. Router selections are deduplicated, cache hits are marked, and victims are selected for admitted misses.
3. Read uncached experts over PCIe. Exllamav3’s kernels can access the pinned host representation without a synchronous host-side copy.
4. Stage long prefills. For sufficiently large per-layer pick counts, missing experts are copied into a staging buffer one layer ahead while the preceding layer computes.
5. Split selected decode misses in fast mode. A cost model sends warmer misses through zero-copy GPU execution and selected cold misses to the CPU tier.
6. Combine CPU and GPU results. The CPU publishes its work through pinned mapped memory, and the GPU adds the resulting FP32 partial after waiting on a device-side flag rather than a host synchronization.

Operational cautions

• Use GLM53_MODE=exact when decode output must match stock exllamav3.
• Avoid the older sha256:1159044a... image if streaming latency matters. Use the documented sha256:bb633b0b... digest.
• Fast mode currently keeps two expert copies in host RAM. Ensure sufficient memory before starting the container.
• Do not assume the measured RTX 3090 throughput on slower PCIe links. Decode speed is sensitive to cache-miss traffic.
• Monitor /stats, system memory, and GPU memory when changing context size, cache reserve, concurrency, or staging settings.
• The server handles text only; image input is not supported by this deployment.

Conclusion

glm53-flash-offload makes an unusually large mixture-of-experts checkpoint practical on a single 24 GB GPU by combining pinned host storage, zero-copy PCIe reads, an elastic VRAM cache, staged prefill transfers, and an optional AVX2 CPU tier. Start with the pinned Docker image, choose exact mode for output fidelity or fast mode for throughput, then validate the installation with the included quality, speed, and streaming checks before tuning cache or CPU settings.