# kvpress

A Python library of key-value cache compression methods for large language models, aimed at reducing the memory cost of serving long-context transformer models.

kvpress implements multiple methods for compressing the key-value (KV) cache of transformer-based large language models, along with benchmarks, built on top of Hugging Face's transformers library. It targets the cost of long-context inference, where the KV cache grows linearly with context length; the project notes that handling one million tokens with a 70-billion-parameter model in float16 can require hundreds of gigabytes of memory.

Each compression method, called a press, has a `compression_ratio` attribute and can be used through a custom text-generation pipeline that is automatically registered with transformers and handles chat templates and tokenization. Compression is applied by default during the prefill phase, and an experimental `DecodingPress` wrapper can also compress the cache periodically during token generation, targeting a fixed cache size instead of a fixed ratio.

It installs with pip, and is released under the Apache-2.0 license.

Website: https://github.com/NVIDIA/kvpress