TurboQuant-MLX
A KV-cache compression library for large language models on Apple Silicon that uses a polar-coordinate quantization scheme to shrink memory use by up to five times while preserving accuracy.
TurboQuant-MLX implements near-optimal distortion-rate compression tailored to Apple's MLX framework, using a Cartesian-to-polar transformation for unbiased dot-product estimation; the project reports that it deliberately dropped an alternative Johnson-Lindenstrauss-based correction after community testing showed it added variance and degraded softmax quality. Keys and values can be compressed asymmetrically, for example keeping keys at higher precision while compressing values more aggressively, since keys matter more for attention accuracy.
A boundary setting leaves the first and last two layers of a model uncompressed to recover most of the accuracy otherwise lost to compression, and an attention-sink setting keeps the first system-prompt tokens in full precision to protect instruction-following. Long generations are cached in fixed-size chunks to bound memory growth, and the library is designed to work as a near drop-in replacement inside the mlx-lm library.