TurboQuant
A KV-cache compression method for LLM inference that quantizes cached keys to 3 bits and values to 2 bits using Triton kernels, integrated with vLLM to cut memory use and raise usable context length.
TurboQuant is an implementation of a KV-cache compression method for LLM inference, quantizing cached keys to 3 bits and values to 2 bits with Triton kernels, integrated with vLLM. It was tested on both dense and mixture-of-experts model architectures across RTX 3090 and RTX 5090 GPUs.
On a single RTX 5090 running a dense model with vLLM, compression freed 30GB of KV cache across four GPUs and roughly doubled maximum token capacity from about 457,000 to about 914,000 tokens, while prefill and decode throughput improved slightly and peak activation memory dropped. On an 8-GPU RTX 3090 cluster running a pruned mixture-of-experts model, only the model's full-attention layers, ten of forty total, are compressed, and the project reports per-context-length throughput, time-to-first-token and needle-retrieval accuracy for both the uncompressed baseline and the compressed configuration, plus a per-GPU VRAM breakdown at 131,000 tokens of context.
It accompanies an ICLR 2026 paper and is written in Python under the GPL-3.0 license.