nano-kvllm
A KV-cache management framework built on nano-vllm that compresses the key-value cache during LLM inference to improve memory use and throughput in high-concurrency, long-context serving.
nano-kvllm (KvLLM) is an LLM inference framework built on top of nano-vllm, focused on efficient KV-cache memory management for large language models. Its current release adds KV-cache compression to reduce the memory bottleneck in high-concurrency and long-context generation.
Version 2.0 introduces a window-based, periodic compression mechanism instead of the common threshold-triggered approach: decode steps are counted globally, compression triggers every fixed number of steps, and only the top-K sequences are compressed, each keeping its most recent blocks uncompressed so the system prompt and generation quality are less affected. Planned future work includes KV-cache offloading and retrieval.
It is written in Python and released under the MIT license.