# turboquant-vllm

A drop-in vLLM plugin implementing Google's TurboQuant key-value cache quantization method, compressing the KV cache with asymmetric key/value bit-widths and validated across eight language models.

turboquant-vllm implements TurboQuant, a key-value cache quantization method from a 2026 ICLR paper, as a plugin that registers automatically with vLLM's attention backend. It compresses cached key and value tensors independently, so keys and values can use different bit-widths, and reduces per-token cache size from 256 bytes to about 68 bytes in the default 4-bit configuration.

It installs with pip and works with vLLM's OpenAI-compatible serving command, or directly with Hugging Face's `DynamicCache` through a wrapper class. A verify command checks cosine similarity between compressed and uncompressed activations for a chosen model and bit-width.

On an RTX 4090 serving Llama-3.1-8B-Instruct, the plugin lowers median time-to-first-token by about 25% by shrinking the cache pages processed at prefill, though per-token decode latency increases due to online decompression.

Website: https://alberto-codes.github.io/turboquant-vllm/