kvtc
The first open-source implementation of NVIDIA's KVTC algorithm, compressing LLM key-value caches 6 to 9x using PCA decorrelation, adaptive quantization and entropy coding, to fit longer context in the same GPU memory.
kvtc compresses the key-value cache that grows with every token an LLM generates, which becomes the dominant memory bottleneck at long context lengths, often exceeding the model weights themselves. It applies techniques borrowed from media compression, such as those behind JPEG and H.264, in a three-stage pipeline: PCA decorrelates and reorders the KV vector dimensions by variance, undoing and later reapplying rotary position embeddings so the low-rank structure PCA needs is not obscured, followed by adaptive quantization and entropy coding.
On an RTX 5090 with Qwen2.5-7B, the authors report configurations from 8.8x compression with a 0.981 cosine similarity to 3.4x compression that is near-lossless. Combined with other techniques, it is used to extend a single 32 GB GPU's usable context length well beyond what an uncompressed FP16 cache allows.
The implementation targets NVIDIA GPUs and is written in Python.