TurboQuant
An open-source implementation of Google's TurboQuant method for compressing a language model's key-value cache, with a running summary of which of its variants the community has since found actually work well in practice.
TurboQuant is a data-oblivious compressor for the key-value cache used during large language model inference, combining a random rotation with a scalar quantizer to shrink the cache to three to four times smaller than 16-bit floating point, without needing calibration data. It targets deployments where GPU memory, not compute, is the binding constraint on a serving setup, and its authors describe it as an open-source implementation of a method published at a 2026 conference.
The project's documentation is unusually candid about limitations the community found after the original publication: 8-bit key-value caching is reported as the best default on newer GPU generations with no measurable accuracy loss, an optional residual-correction step often hurts at low bit widths, 3-bit compression is not considered production-ready for reasoning or long-context retrieval, and the first and last layers of a model are more sensitive to quantization than the middle ones. TurboQuant is instead recommended for single-card serving of large models at very long context, or on hardware without 8-bit attention support.