NanoQuant
A post-training quantization method, presented at ICML 2026, that compresses large language model weights below one bit per parameter using mixture-of-rank binary bases while preserving accuracy.
NanoQuant is a post-training quantization algorithm that compresses LLM weights to sub-1-bit levels without retraining, using only a calibration pass. It introduces a non-factored/factored decomposition of weight matrices combined with learnable mixture-of-rank binary bases, iteratively optimized with ADMM, and compensates for per-layer quantization error to preserve accuracy at extreme compression.
The method reports up to 3x faster decoding through custom CUDA GEMM and GEMV kernels for prefill and decode, and up to 10x smaller memory footprints compared to FP16. It supports OPT, Llama (1 through 3), Qwen (2.5 and 3), and Gemma (2 and 3) architectures, and a --device_map auto option offloads across GPU and CPU for models larger than 70B parameters that do not fit in CPU memory.
Installation builds a conda environment, installs the Python package, and compiles the CUDA kernels separately; the code accompanies a paper accepted at ICML 2026.