TurboQuant-MLX
An MLX implementation of Google's TurboQuant method that compresses both model weights and the key-value cache of large language models on Apple Silicon, letting bigger models fit in less memory.
TurboQuant-MLX applies Google's TurboQuant technique, combining a Hadamard rotation with Lloyd-Max codebooks, to compress both the weights and the key-value cache of large language models on Apple Silicon, at compile time for weights and at run time for the cache. It covers dense models such as LLaMA, Qwen and Mistral, several mixture-of-experts models, Mamba and attention hybrid models, and multimodal models, and it is compatible with attention sinks, sliding-window and linear attention layers.
An expert-streaming mode lets a mixture-of-experts model whose weights exceed available memory run by loading only the router-selected experts from disk per token, so a 35-billion-parameter model can run in under 4 gigabytes of memory on a 16-gigabyte Mac mini. Compressing both weights and the cache at 3-bit precision can also make generation faster, because the smaller cache reduces the memory bandwidth the GPU needs.