DeepInfer Operational

DeepInfer is DeepSeek's open-source production-grade inference framework, derived from vLLM. It supports V4.1 Flash and runs on NVIDIA SM100 and Huawei Ascend 950. DeepInfer uses disaggregated Prefill/Decode deployment and relies on a 3FS caching cluster to achieve a 96% input hit rate and a 25x Prefill efficiency improvement. SWA Bounded Replay halves the Prefill computation, while DSpark speculative decoding further increases throughput by 6%, enabling a production cluster to process 155 trillion Tokens per day. The concurrently open-sourced deepseek-recipe is a production-grade Token encoding and decoding component. It unifies the Chat Completions, Responses, and Messages API formats, handles injection protection, parallel tokenization, and streaming output, and makes self-hosted models behave exactly like the official API.

Distributed inference: Supports attention data parallelism, context parallelism, and MoE expert parallelism, enabling large-scale production services processing 155 trillion Tokens per day on a single cluster.
Cross-hardware compatibility: Supports both NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, covering two major computing ecosystems.
Efficient long-context processing: SWA Bounded Replay reduces the recomputation cost of cache misses to a fixed 128 Tokens, decoupling it from context length and layer count.
Speculative decoding acceleration: Integrates the proprietary DSpark and confidence-based verification scheduling, allowing the model to advance multiple Tokens per step and significantly improve generation throughput.
Load-balanced scheduling: Dynamically balances the distribution of experts and requests; the busiest cards in production are only 9% above the average load.
Cache reuse: Relies on a 3FS storage cluster to achieve a 96% input cache hit rate and increase Prefill throughput by approximately 25x.

Follow WeChat and reply “开源” to join the AI open-source project group
Disaggregated Prefill/Decode deployment: Inference is divided into two stages: Prefill and Decode. Prefill processes tens of thousands of input Tokens at once and is compute-intensive; Decode generates one Token at a time and is constrained by memory-bandwidth reads. DeepInfer deploys them on different hardware: Prefill uses conventional 8-card servers, while Decode uses 64–128-card supernodes. The two types of machines are configured at an approximately 1:1 ratio so each can run according to its optimal hardware characteristics.
Prefix caching system: In Agent scenarios, every tool call resends the complete history, most of which has already been computed. The system caches only the prefix that is fully identical from the first Token. Matching portions directly reuse the KV Cache, and only new content is computed. The cache is stored in an independent 3FS distributed storage cluster and written in 128-Token blocks. Agent traffic is retained for 3 days and chat traffic for 7 days.
SWA Bounded Replay: Some layers in V4.1 Flash use sliding-window attention, where each Token attends only to the most recent 128 Tokens. This KV Cache is not written to disk and remains only in the memory of the compute machines. On a cache miss, the old approach precisely recomputed “number of layers × window length.” V4.1 instead recomputes only the last 128 Tokens, with attention computed only within those 128 Tokens, producing an approximate window cache with a fixed recomputation cost.
DSpark speculative decoding and asynchronous scheduling: A lightweight draft module first predicts 5 Tokens consecutively, after which the target model verifies them in a batch. Correct predictions are accepted directly; at the first incorrect prediction, the large model corrects it, allowing multiple Tokens to be advanced in a single computation. A confidence head estimates the probability that each prediction will be accepted. Low-probability predictions are not verified, freeing compute for other requests. Compared with a fixed five-Token verification strategy, production throughput increases by 6%. At the same time, CPU scheduling for step N+1 overlaps with GPU computation for step N, eliminating GPU idle time during scheduling gaps.

Check the prerequisites: Linux, GCC ≥ 13.0, and Python 3.12. NVIDIA platforms require CUDA 13; Ascend platforms require CANN 9.2.
Clone the repository: git clone https://github.com/deepseek-ai/DeepInfer and enter the directory.
Create a clean environment: Create and activate an empty Python 3.12 virtual environment to avoid dependency conflicts.
Select the hardware platform and build: Run export DEEPINFER_ACCELERATOR=nvidia (use ascend for Ascend), then execute ./build.sh to build from source. Ensure this environment variable remains active in subsequent runtime environments.
Prepare model weights: By default, the first run automatically downloads deepseek-ai/DeepSeek-V4.1-Flash-1017 from Hugging Face. Alternatively, use export DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017 to point to a local copy.
Run the offline inference example: Execute python examples/offline_inference.py to start inference. See the repository's examples directory for more usage instructions.

Production-validated ultra-high performance: Already deployed in large-scale V4.1 Flash production services, processing 155 trillion Tokens per day on a single cluster, with a 400ms time to first Token and a generation speed of 230 tokens/s.
Extremely efficient cache reuse: 96% of inputs hit the prefix cache. Prefill throughput per card increases from 60,000 tokens/s without caching to 1.5 million tokens/s, a 25x difference.
Significantly lower long-context costs: SWA Bounded Replay fixes cache-miss recomputation to the last 128 Tokens, nearly halving total Prefill computation, with costs independent of context length.
Industry-leading generation efficiency: DSpark speculative decoding combined with confidence-based and asynchronous scheduling advances multiple Tokens per step. Production throughput increases by another 6% over fixed verification, reaching 14K Tokens generated per card per second.
Excellent load balancing: MoE expert replicas are dynamically rearranged every 10000 steps. The busiest card in production is only 9% above the average load, achieving 80% of the ideal overall balance.
Dual-hardware compatibility: Supports both NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, making it one of the few production-grade inference frameworks currently covering both major computing ecosystems.

GitHub repository: https://github.com/deepseek-ai/DeepInfer
https://github.com/deepseek-ai/deepseek-recipe

Enterprise private deployment: Deploy V4.1 Flash in an enterprise data center and use deepseek-recipe to align with the official API behavior, enabling compliant inference while keeping data within the domain.
Domestic hardware adaptation: Native support for Huawei Ascend 950 NPUs meets the needs of government, enterprise, and financial scenarios seeking to reduce dependence on NVIDIA.
Large-scale Agent services: A 96% prefix cache hit rate significantly reduces repeated computation across multi-turn tool calls, making it suitable for cost-efficient operation of high-concurrency Agent platforms.
Long-context document processing: SWA Bounded Replay nearly halves Prefill computation for long text, making it suitable for ultra-long inputs such as contract analysis and codebase understanding.
Cost-effective self-hosted inference: High throughput per card and a generation speed of 230–300 tokens/s make it suitable for startups replacing usage-based APIs with cost-saving, self-hosted services at scale.