अवलोकन
DeepInfer is the inference framework behind DeepSeek’s large-scale production services. The open-source release covers offline inference with the core features needed for serving at scale, and currently supports one model, DeepSeek-V4.1-Flash. It combines DeepSeek’s optimized kernels (FlashMLA, DeepGEMM, DeepEP and others) with attention data parallelism, MoE expert parallelism with load balancing, an asynchronous batch scheduler and DSpark draft-token decoding. The repository was created in October 2026 and passed roughly 25,000 stars within a month; the latest commits landed in early November. It is Apache 2.0 licensed. DeepSeek says it was originally built mainly on vLLM, with ideas from SGLang and TensorRT-LLM. It is not a complete serving stack: a KV cache system, prefill/decode disaggregation and load balancing across replicas are not included.
प्रमुख विशेषताएँ
- DeepSeek production kernels: FlashMLA, DeepGEMM, DeepEP
- Attention DP / balanced CP, MoE EP with EPLB
- DSpark draft decoding with confidence-scheduled verification
- Zero-overhead asynchronous batch scheduler
- Batch-invariant inference and deterministic sampling
- Structured output: regex, JSON schema, EBNF
आवश्यकताएँ, इंस्टॉलेशन और त्वरित शुरुआत
2. Install CUDA 13 for NVIDIA SM100 GPUs, or CANN 9.2 for Huawei Ascend 950 NPUs.
3. Clone the repository and set the platform: export DEEPINFER_ACCELERATOR=nvidia (or ascend). Keep it exported when you run DeepInfer.
4. Build from source with ./build.sh.
5. Run python examples/offline_inference.py. The default checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017 downloads to the Hugging Face cache on first run; set DEEPINFER_HF_MODEL_DIR to use a local copy.
उपयोग
मॉडल संगतता और उपयोग के मामले
DeepSeek-V4.1-Flash only in this release (default checkpoint DeepSeek-V4.1-Flash-1017).
लाइसेंस और जोखिम संबंधी टिप्पणियाँ
Apache 2.0 according to the repository LICENSE.
Editorial source review: README, repository API metadata and LICENSE read; no install test.
रिलीज़ और रखरखाव
No tagged releases yet; build from source as shown above. Features are taken from the README and the examples documentation.