turboquant-mlx
A key-value cache compression library for MLX on Apple Silicon that quantizes cached keys and values to as low as 3 bits, using mixed precision to keep decode quality unchanged.
turboquant-mlx compresses the KV cache used during LLM inference with PolarQuant, a combination of randomized Hadamard rotation and Lloyd-Max quantization, implemented as fused Metal kernels and offered as a drop-in replacement for mlx-lm's KVCache. Its central finding is that keys and values behave differently under quantization: compressing keys destroys greedy decoding accuracy even at 4 bits because softmax is sensitive to small errors, while values tolerate compression well down to 3 bits because their use is a weighted interpolation.
Based on that, the recommended configuration keeps keys at 8 bits and values at 4 bits or lower; on a 7B model with a 32K context, the README reports meaningfully less active memory with output identical to full-precision decoding. A value-only mode works with stock mlx-lm, while mixed key-value compression needs a patched fork.
It is written in Python.