추론, 배포 및 런타임

DeepInfer

deepseek-ai/DeepInfer

DeepSeek’s own high-performance inference framework, open-sourced for offline inference of DeepSeek-V4.1-Flash on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.

★ 25.7K별점
⑂ 3.2K포크 수
247미해결 이슈
Python언어
Apache-2.0라이선스
Q82편집 점수

개요

DeepInfer is the inference framework behind DeepSeek’s large-scale production services. The open-source release covers offline inference with the core features needed for serving at scale, and currently supports one model, DeepSeek-V4.1-Flash. It combines DeepSeek’s optimized kernels (FlashMLA, DeepGEMM, DeepEP and others) with attention data parallelism, MoE expert parallelism with load balancing, an asynchronous batch scheduler and DSpark draft-token decoding. The repository was created in October 2026 and passed roughly 25,000 stars within a month; the latest commits landed in early November. It is Apache 2.0 licensed. DeepSeek says it was originally built mainly on vLLM, with ideas from SGLang and TensorRT-LLM. It is not a complete serving stack: a KV cache system, prefill/decode disaggregation and load balancing across replicas are not included.

주요 기능

  • DeepSeek production kernels: FlashMLA, DeepGEMM, DeepEP
  • Attention DP / balanced CP, MoE EP with EPLB
  • DSpark draft decoding with confidence-scheduled verification
  • Zero-overhead asynchronous batch scheduler
  • Batch-invariant inference and deterministic sampling
  • Structured output: regex, JSON schema, EBNF

요구 사항, 설치 및 빠른 시작

1. Prepare Linux with GCC 13 or newer and an empty Python 3.12 environment.
2. Install CUDA 13 for NVIDIA SM100 GPUs, or CANN 9.2 for Huawei Ascend 950 NPUs.
3. Clone the repository and set the platform: export DEEPINFER_ACCELERATOR=nvidia (or ascend). Keep it exported when you run DeepInfer.
4. Build from source with ./build.sh.
5. Run python examples/offline_inference.py. The default checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017 downloads to the Hugging Face cache on first run; set DEEPINFER_HF_MODEL_DIR to use a local copy.

사용 정보

Start with the offline example and choose a parallel layout, for example python examples/offline_inference.py --ep 8 for expert parallelism across eight devices, or --ep 2 --dp 8 to add data parallelism. To force JSON, regex or EBNF output, pass so_args to SamplingParams. For reproducible runs on NVIDIA, set DEEPINFER_BATCH_INVARIANT=1 so results do not change with batch composition, and add a fixed seq_seed for deterministic sampling. The examples folder also has benchmark sweep scripts that reproduce the published prefill and decode charts.

모델 호환성 및 사용 사례

DeepSeek-V4.1-Flash only in this release (default checkpoint DeepSeek-V4.1-Flash-1017).

라이선스 및 위험 참고 사항

Apache 2.0 according to the repository LICENSE.

Editorial source review: README, repository API metadata and LICENSE read; no install test.

릴리스 및 유지 관리

No tagged releases yet; build from source as shown above. Features are taken from the README and the examples documentation.

vllm

vllm-project/vllm

★ 89.5KPython

sglang

sgl-project/sglang

★ 32.2KPython

OpenVINO

openvinotoolkit/openvino

★ 10.6KC++

GPUStack

gpustack/gpustack

★ 5.4KPython