kvpress
A Python library of key-value cache compression methods for large language models, aimed at reducing the memory cost of serving long-context transformer models.
kvpress implements multiple methods for compressing the key-value (KV) cache of transformer-based large language models, along with benchmarks, built on top of Hugging Face's transformers library. It targets the cost of long-context inference, where the KV cache grows linearly with context length; the project notes that handling one million tokens with a 70-billion-parameter model in float16 can require hundreds of gigabytes of memory.
Each compression method, called a press, has a compression_ratio attribute and can be used through a custom text-generation pipeline that is automatically registered with transformers and handles chat templates and tokenization. Compression is applied by default during the prefill phase, and an experimental DecodingPress wrapper can also compress the cache periodically during token generation, targeting a fixed cache size instead of a fixed ratio.
It installs with pip, and is released under the Apache-2.0 license.