Inférence, déploiement et exécution

llm-d: Distributed LLM Inference on Kubernetes

llm-d/llm-d

llm-d is an open-source serving stack that adds distributed orchestration, routing, cache management, autoscaling, and batch processing around model servers such as vLLM and SGLang. It targets high-scale production inference on Kubernetes and modern hardware accelerators.

Ouvrir le dépôt officiel ↗ Page d'accueil du projet ↗
★ 4KÉtoiles
⑂ 649Forks
233Problèmes ouverts
ShellLangue
Apache-2.0Licence
Q@project.QualityScoreScore éditorial

Vue d’ensemble

The project provides benchmarked deployment patterns and operational components for large-language-model inference. Its core areas include prefix-cache-aware and load-aware routing, predicted-latency scheduling, tiered KV-cache offloading, prefill/decode disaggregation, wide expert parallelism, multi-tenant flow control, SLO-aware autoscaling, and asynchronous batch serving. llm-d is described as a CNCF Sandbox project founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA. The repository is primarily Shell.

Fonctionnalités clés

  • Prefix-cache-aware and load-aware intelligent routing
  • Predicted-latency scheduling
  • Tiered KV-cache offloading to CPU or disk with global cache-state indexing
  • Prefill/decode disaggregation for large-model serving
  • Wide expert parallelism across fast accelerator interconnects
  • Multi-tenant flow control and SLO-aware autoscaling
  • OpenAI-compatible Batch APIs with asynchronous processing
  • Benchmarked deployment guides and Kubernetes packaging patterns
  • Integration with model servers including vLLM and SGLang

Prérequis, installation et démarrage rapide

No complete installation commands are included in the supplied repository excerpt. Follow the official quick-start guide at https://llm-d.ai/docs/getting-started/quickstart. The project also provides Helm charts and guides, while the latest news says newer guides have moved toward a kustomize-first approach. Exact tool versions and commands should be taken from the current documentation rather than inferred from this record.

Utilisation

Start with the Optimized Baseline guide at https://llm-d.ai/docs/guides/optimized-baseline. The documented quick-start flow is to prepare the Kubernetes stack, deploy an optimized inference service, configure intelligent routing, and validate it with production-oriented benchmarks. Additional architectural patterns are listed at https://llm-d.ai/docs/guides, and benchmark results are available at https://prism.llm-d.ai. No copy-ready usage command or API request is present in the supplied excerpt.

Compatibilité des modèles et cas d’usage

The stack is designed to operate above model servers such as vLLM and SGLang and targets key open-source LLMs. Examples named in the repository include DeepSeek-R1, DeepSeek V3.1, GPT-OSS, GPT-OSS-120B, Llama 3.1 70B, and Llama 3.3 70B. This does not establish a complete compatibility list. Agent-specific models, embedding models, MCP servers, and RAG-specific components are not stated in the repository metadata.

Licence et notes sur les risques

Apache License 2.0, identified by SPDX as Apache-2.0. The repository points readers to its LICENSE file for full terms. No additional license exceptions are stated in the supplied data.

Publication et maintenance

The README displays a Version 0.8 badge but provides no corresponding details in the supplied excerpt. Its latest detailed entry says v0.7 added a stabilized optimized baseline, kustomize-first guides, expanded nightly CI, generally available predicted-latency scheduling, an experimental batch gateway, and revised documentation. Earlier entries describe v0.5 features such as hierarchical KV offloading, cache-aware LoRA routing, active-active high availability, transport resilience, and scale-to-zero autoscaling, plus v0.4 accelerator and cache-offload work. Current release notes are available at https://github.com/llm-d/llm-d/releases.

OpenVINO

openvinotoolkit/openvino

★ 10,6KC++

GPUStack

gpustack/gpustack

★ 5,4KPython