Visão geral
DeepInfer is the inference framework behind DeepSeek’s large-scale production services. The open-source release covers offline inference with the core features needed for serving at scale, and currently supports one model, DeepSeek-V4.1-Flash. It combines DeepSeek’s optimized kernels (FlashMLA, DeepGEMM, DeepEP and others) with attention data parallelism, MoE expert parallelism with load balancing, an asynchronous batch scheduler and DSpark draft-token decoding. The repository was created in October 2026 and passed roughly 25,000 stars within a month; the latest commits landed in early November. It is Apache 2.0 licensed. DeepSeek says it was originally built mainly on vLLM, with ideas from SGLang and TensorRT-LLM. It is not a complete serving stack: a KV cache system, prefill/decode disaggregation and load balancing across replicas are not included.
Principais recursos
- DeepSeek production kernels: FlashMLA, DeepGEMM, DeepEP
- Attention DP / balanced CP, MoE EP with EPLB
- DSpark draft decoding with confidence-scheduled verification
- Zero-overhead asynchronous batch scheduler
- Batch-invariant inference and deterministic sampling
- Structured output: regex, JSON schema, EBNF
Requisitos, instalação e início rápido
2. Install CUDA 13 for NVIDIA SM100 GPUs, or CANN 9.2 for Huawei Ascend 950 NPUs.
3. Clone the repository and set the platform: export DEEPINFER_ACCELERATOR=nvidia (or ascend). Keep it exported when you run DeepInfer.
4. Build from source with ./build.sh.
5. Run python examples/offline_inference.py. The default checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017 downloads to the Hugging Face cache on first run; set DEEPINFER_HF_MODEL_DIR to use a local copy.
Uso
Compatibilidade do modelo e casos de uso
DeepSeek-V4.1-Flash only in this release (default checkpoint DeepSeek-V4.1-Flash-1017).
Observações sobre licença e riscos
Apache 2.0 according to the repository LICENSE.
Editorial source review: README, repository API metadata and LICENSE read; no install test.
Lançamento e manutenção
No tagged releases yet; build from source as shown above. Features are taken from the README and the examples documentation.