Inférence, déploiement et exécution

DeepInfer

deepseek-ai/DeepInfer

DeepSeek’s own high-performance inference framework, open-sourced for offline inference of DeepSeek-V4.1-Flash on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.

★ 25,7KÉtoiles
⑂ 3,2KForks
247Problèmes ouverts
PythonLangue
Apache-2.0Licence
Q82Score éditorial

Vue d’ensemble

DeepInfer is the inference framework behind DeepSeek’s large-scale production services. The open-source release covers offline inference with the core features needed for serving at scale, and currently supports one model, DeepSeek-V4.1-Flash. It combines DeepSeek’s optimized kernels (FlashMLA, DeepGEMM, DeepEP and others) with attention data parallelism, MoE expert parallelism with load balancing, an asynchronous batch scheduler and DSpark draft-token decoding. The repository was created in October 2026 and passed roughly 25,000 stars within a month; the latest commits landed in early November. It is Apache 2.0 licensed. DeepSeek says it was originally built mainly on vLLM, with ideas from SGLang and TensorRT-LLM. It is not a complete serving stack: a KV cache system, prefill/decode disaggregation and load balancing across replicas are not included.

Fonctionnalités clés

  • DeepSeek production kernels: FlashMLA, DeepGEMM, DeepEP
  • Attention DP / balanced CP, MoE EP with EPLB
  • DSpark draft decoding with confidence-scheduled verification
  • Zero-overhead asynchronous batch scheduler
  • Batch-invariant inference and deterministic sampling
  • Structured output: regex, JSON schema, EBNF

Prérequis, installation et démarrage rapide

1. Prepare Linux with GCC 13 or newer and an empty Python 3.12 environment.
2. Install CUDA 13 for NVIDIA SM100 GPUs, or CANN 9.2 for Huawei Ascend 950 NPUs.
3. Clone the repository and set the platform: export DEEPINFER_ACCELERATOR=nvidia (or ascend). Keep it exported when you run DeepInfer.
4. Build from source with ./build.sh.
5. Run python examples/offline_inference.py. The default checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017 downloads to the Hugging Face cache on first run; set DEEPINFER_HF_MODEL_DIR to use a local copy.

Utilisation

Start with the offline example and choose a parallel layout, for example python examples/offline_inference.py --ep 8 for expert parallelism across eight devices, or --ep 2 --dp 8 to add data parallelism. To force JSON, regex or EBNF output, pass so_args to SamplingParams. For reproducible runs on NVIDIA, set DEEPINFER_BATCH_INVARIANT=1 so results do not change with batch composition, and add a fixed seq_seed for deterministic sampling. The examples folder also has benchmark sweep scripts that reproduce the published prefill and decode charts.

Compatibilité des modèles et cas d’usage

DeepSeek-V4.1-Flash only in this release (default checkpoint DeepSeek-V4.1-Flash-1017).

Licence et notes sur les risques

Apache 2.0 according to the repository LICENSE.

Editorial source review: README, repository API metadata and LICENSE read; no install test.

Publication et maintenance

No tagged releases yet; build from source as shown above. Features are taken from the README and the examples documentation.

vllm

vllm-project/vllm

★ 89,5KPython

sglang

sgl-project/sglang

★ 32,2KPython

OpenVINO

openvinotoolkit/openvino

★ 10,6KC++

GPUStack

gpustack/gpustack

★ 5,4KPython