Tổng quan
DeepInfer is the inference framework behind DeepSeek’s large-scale production services. The open-source release covers offline inference with the core features needed for serving at scale, and currently supports one model, DeepSeek-V4.1-Flash. It combines DeepSeek’s optimized kernels (FlashMLA, DeepGEMM, DeepEP and others) with attention data parallelism, MoE expert parallelism with load balancing, an asynchronous batch scheduler and DSpark draft-token decoding. The repository was created in October 2026 and passed roughly 25,000 stars within a month; the latest commits landed in early November. It is Apache 2.0 licensed. DeepSeek says it was originally built mainly on vLLM, with ideas from SGLang and TensorRT-LLM. It is not a complete serving stack: a KV cache system, prefill/decode disaggregation and load balancing across replicas are not included.
Tính năng chính
- DeepSeek production kernels: FlashMLA, DeepGEMM, DeepEP
- Attention DP / balanced CP, MoE EP with EPLB
- DSpark draft decoding with confidence-scheduled verification
- Zero-overhead asynchronous batch scheduler
- Batch-invariant inference and deterministic sampling
- Structured output: regex, JSON schema, EBNF
Yêu cầu, cài đặt và bắt đầu nhanh
2. Install CUDA 13 for NVIDIA SM100 GPUs, or CANN 9.2 for Huawei Ascend 950 NPUs.
3. Clone the repository and set the platform: export DEEPINFER_ACCELERATOR=nvidia (or ascend). Keep it exported when you run DeepInfer.
4. Build from source with ./build.sh.
5. Run python examples/offline_inference.py. The default checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017 downloads to the Hugging Face cache on first run; set DEEPINFER_HF_MODEL_DIR to use a local copy.
Cách sử dụng
Khả năng tương thích của mô hình và trường hợp sử dụng
DeepSeek-V4.1-Flash only in this release (default checkpoint DeepSeek-V4.1-Flash-1017).
Ghi chú về giấy phép và rủi ro
Apache 2.0 according to the repository LICENSE.
Editorial source review: README, repository API metadata and LICENSE read; no install test.
Phát hành và bảo trì
No tagged releases yet; build from source as shown above. Features are taken from the README and the examples documentation.