Inferenza, distribuzione e runtime

llm-d: Distributed LLM Inference on Kubernetes

llm-d/llm-d

llm-d is an open-source serving stack that adds distributed orchestration, routing, cache management, autoscaling, and batch processing around model servers such as vLLM and SGLang. It targets high-scale production inference on Kubernetes and modern hardware accelerators.

Apri il repository ufficiale ↗ Pagina iniziale del progetto ↗
★ 4KStelle
⑂ 649Fork
233Problemi aperti
ShellLingua
Apache-2.0Licenza
Q@project.QualityScorePunteggio editoriale

Panoramica

The project provides benchmarked deployment patterns and operational components for large-language-model inference. Its core areas include prefix-cache-aware and load-aware routing, predicted-latency scheduling, tiered KV-cache offloading, prefill/decode disaggregation, wide expert parallelism, multi-tenant flow control, SLO-aware autoscaling, and asynchronous batch serving. llm-d is described as a CNCF Sandbox project founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA. The repository is primarily Shell.

Funzionalità principali

  • Prefix-cache-aware and load-aware intelligent routing
  • Predicted-latency scheduling
  • Tiered KV-cache offloading to CPU or disk with global cache-state indexing
  • Prefill/decode disaggregation for large-model serving
  • Wide expert parallelism across fast accelerator interconnects
  • Multi-tenant flow control and SLO-aware autoscaling
  • OpenAI-compatible Batch APIs with asynchronous processing
  • Benchmarked deployment guides and Kubernetes packaging patterns
  • Integration with model servers including vLLM and SGLang

Requisiti, installazione e avvio rapido

No complete installation commands are included in the supplied repository excerpt. Follow the official quick-start guide at https://llm-d.ai/docs/getting-started/quickstart. The project also provides Helm charts and guides, while the latest news says newer guides have moved toward a kustomize-first approach. Exact tool versions and commands should be taken from the current documentation rather than inferred from this record.

Utilizzo

Start with the Optimized Baseline guide at https://llm-d.ai/docs/guides/optimized-baseline. The documented quick-start flow is to prepare the Kubernetes stack, deploy an optimized inference service, configure intelligent routing, and validate it with production-oriented benchmarks. Additional architectural patterns are listed at https://llm-d.ai/docs/guides, and benchmark results are available at https://prism.llm-d.ai. No copy-ready usage command or API request is present in the supplied excerpt.

Compatibilità del modello e casi d'uso

The stack is designed to operate above model servers such as vLLM and SGLang and targets key open-source LLMs. Examples named in the repository include DeepSeek-R1, DeepSeek V3.1, GPT-OSS, GPT-OSS-120B, Llama 3.1 70B, and Llama 3.3 70B. This does not establish a complete compatibility list. Agent-specific models, embedding models, MCP servers, and RAG-specific components are not stated in the repository metadata.

Note su licenza e rischi

Apache License 2.0, identified by SPDX as Apache-2.0. The repository points readers to its LICENSE file for full terms. No additional license exceptions are stated in the supplied data.

Rilascio e manutenzione

The README displays a Version 0.8 badge but provides no corresponding details in the supplied excerpt. Its latest detailed entry says v0.7 added a stabilized optimized baseline, kustomize-first guides, expanded nightly CI, generally available predicted-latency scheduling, an experimental batch gateway, and revised documentation. Earlier entries describe v0.5 features such as hierarchical KV offloading, cache-aware LoRA routing, active-active high availability, transport resilience, and scale-to-zero autoscaling, plus v0.4 accelerator and cache-offload work. Current release notes are available at https://github.com/llm-d/llm-d/releases.

OpenVINO

openvinotoolkit/openvino

★ 10,6KC++

GPUStack

gpustack/gpustack

★ 5,4KPython