mlx-kquant
A C++ and Metal extension for MLX on Apple Silicon that adds native K-quant quantization, alongside a toolchain to quantize, load, run, LoRA-train and fuse K-quant checkpoints on a stock MLX installation.
mlx-kquant adds a namespace of native MLX operations, dequantize, quantized matmul, gather and quantize, backed by Metal kernels compiled ahead of time into a metal library, covering all ten standard K-quant and legacy GGML codecs, nine IQ codecs, two experimental ternary codecs, and the native floating-point mxfp4 and nvfp4 wire codecs, plus fused decode and prefill kernels for mixture-of-experts and attention.
A Python-side toolchain quantizes an MLX safetensors model into a uniform- or mixed-precision K-quant checkpoint in MLX-native safetensors format, then loads, generates from, LoRA-trains and fuses it.
The project reports that K-quants roughly halve the divergence from a reference model compared with affine quantization at the same bitrate, with measured divergence figures for several models included in the README. It requires macOS 26.2 or later on Apple Silicon and ships a prebuilt wheel; it is MIT licensed.