LMCache
A key-value cache management layer for scalable LLM inference that turns transient KV cache into reusable, shareable state across serving engines, cutting time-to-first-token and improving throughput.
LMCache is a key-value cache management layer for large language model inference. It turns the KV cache from temporary, per-request state into reusable, persistent data that can be stored, reused across multiple serving engines, monitored through an observability stack, and transformed to improve generation quality, which reduces time to first token and improves throughput, particularly for long-context, agentic, multi-turn and retrieval-augmented workloads.
It is vendor-neutral, usable as a KV-cache layer across mainstream open-source serving engines such as vLLM, a range of inference frameworks, GPU vendors including both CUDA and ROCm hardware, and various storage systems and infrastructure providers, so a team can switch serving engines or storage vendors while continuing to reuse already-stored caches.
It runs as an engine-independent standalone daemon process, managing the KV cache separately from the inference engine so cached data is not lost if the engine itself crashes, and moves cached data out of GPU memory into a tiered storage hierarchy spanning CPU memory, local storage and remote backends as needed.