Mooncake
A KV-cache-centric, disaggregated serving architecture for large language models, developed as the platform behind Moonshot AI's Kimi service.
Mooncake separates the stages of LLM serving so that key-value cache storage and transfer can be managed independently of the compute doing prefill and decode, which its authors report lets Kimi handle substantially more requests while meeting its service-level objectives. Its transfer engine moves KV cache and hidden states between nodes over RDMA, and has been adopted by projects including vLLM, SGLang, and LightX2V for cross-instance cache sharing, disaggregated inference connectors, and distributed weight transfer during large-scale reinforcement learning.
It is used as a rollout data-transfer backend in reinforcement learning training pipelines and as a distributed backend for multi-node online training, and is a member of the PyTorch ecosystem.
It is written in C++ and distributed under the Apache-2.0 license.