turboquant-vllm
A drop-in vLLM plugin implementing Google's TurboQuant key-value cache quantization method, compressing the KV cache with asymmetric key/value bit-widths and validated across eight language models.
turboquant-vllm implements TurboQuant, a key-value cache quantization method from a 2026 ICLR paper, as a plugin that registers automatically with vLLM's attention backend. It compresses cached key and value tensors independently, so keys and values can use different bit-widths, and reduces per-token cache size from 256 bytes to about 68 bytes in the default 4-bit configuration.
It installs with pip and works with vLLM's OpenAI-compatible serving command, or directly with Hugging Face's DynamicCache through a wrapper class. A verify command checks cosine similarity between compressed and uncompressed activations for a chosen model and bit-width.
On an RTX 4090 serving Llama-3.1-8B-Instruct, the plugin lowers median time-to-first-token by about 25% by shrinking the cache pages processed at prefill, though per-token decode latency increases due to online decompression.