TurboQuant
A drop-in key-value cache compressor for Hugging Face language models that quantizes the cache to 4 bits, cutting its VRAM footprint and restoring inference speed once long-context generation would otherwise start thrashing.
As a language model generates text it stores a growing key-value cache for every token, which eats available VRAM and can make a GPU thrash once the cache gets large relative to available memory; TurboQuant compresses that cache from 16 bits down to 4 bits using Google's TurboQuant algorithm, requiring no training data or calibration step and working with any Hugging Face model as a drop-in addition. The project's own benchmarks on an RTX 4080 report a Qwen 3B model at 4K context going from 2.5 tokens per second while thrashing to 7.4 tokens per second with a gigabyte of VRAM saved, and a Qwen 7B model at 2K context going from out-of-memory to 1.4 tokens per second.
It ships an OpenAI-compatible inference server that can be pointed at with any existing OpenAI client, and its core quantization and dequantization routines can also be called directly against arbitrary vectors such as embeddings.
It installs with pip, is written in Python, and its license is not asserted in its repository metadata.