Panoramica
Mooncake is the serving platform used for Kimi, Moonshot AI's LLM service. Its disaggregated architecture uses available CPU, DRAM, SSD/NVMe, network, and accelerator resources to create a shared KV-cache pool. The main components are Transfer Engine, Mooncake Store, Mooncake EP, and Mooncake Process Group. The repository reports integration with serving and distributed-computing ecosystems including SGLang, vLLM, TensorRT-LLM, LMCache, LMDeploy, NIXL, and PyTorch.
Funzionalità principali
- Unified batched data movement across heterogeneous storage, networks, and accelerators
- Topology-aware routing, multi-NIC bandwidth aggregation, and automatic transfer failover
- Distributed KV-cache and model-weight management through Mooncake Store
- DRAM and SSD/NVMe cache tiers with object replication, placement, pinning, and eviction controls
- Disaggregated prefill and decode serving
- Cross-instance KV-cache sharing and reuse
- Fault-tolerant expert-parallel dispatch and combine operations
- PyTorch ProcessGroup integration with failure reporting and rank recovery
- Integrations described for SGLang, vLLM, TensorRT-LLM, LMCache, LMDeploy, NIXL, and related systems
- Support described for TCP, RDMA, AWS EFA, NVMe-oF, NVLink, HIP, Barex, CXL, and Ascend-family transports
Requisiti, installazione e avvio rapido
Utilizzo
Compatibilità del modello e casi d'uso
Mooncake is infrastructure rather than a model runtime tied to one model family. The supplied data mentions use with Kimi, Kimi-K2, DeepSeek deployments, GLM 5.2 FP8, LLaMA3-70B benchmark sizing, large mixture-of-experts systems, and multimodal pipelines. This does not establish universal compatibility with every model. Exact supported model lists and version constraints are not stated in the repository metadata.
Note su licenza e rischi
The repository metadata identifies the license as Apache-2.0 and provides https://api.github.com/licenses/apache-2.0. Users should review the repository's license file and the licenses of integrated runtimes, drivers, transports, and serving frameworks before deployment.
Rilascio e manutenzione
The supplied update history reports that Transfer Engine was open-sourced on November 28, 2024 and Mooncake Store on March 7, 2025. It also records integrations or adoption involving vLLM, SGLang, TensorRT-LLM, LMCache, LMDeploy, NIXL, vLLM-Ascend, TorchSpec, and other projects. The repository metadata reports 6,132 stars, 1,047 forks, 462 open issues, a creation timestamp of 2024-06-25T05:21:05Z, and that the repository is not archived. No formal latest release version is stated.