VeloxQuant-MLX
A library of 43 KV-cache compression methods for MLX models on Apple Silicon, letting a locally run model use up to 16 times less cache memory with a three-line API change.
VeloxQuant-MLX shrinks the KV cache of mlx_lm models on Apple Silicon so a locally running model can hold a longer context, or use less memory, without changing anything else about the model. It bundles 43 compression methods adapted from published papers, spanning zero-calibration one-bit quantizers, token-eviction caches and cross-layer merging, all behind the same three-line API, so switching methods only changes one argument.
The compression-heavy path runs on hand-written Metal kernels, and the project reports validation across twelve production language models, including Llama, Mistral, Qwen, Phi, Gemma and Falcon, as well as vision-language models through mlx-vlm. It installs with pip and requires an Apple Silicon Mac, Python 3.10 or newer, and MLX 0.18 or newer.