VPTQ
A vector post-training quantization method that compresses large language models to under two bits per parameter while keeping them usable, with pre-quantized community models available for models as large as Llama 3.1 405B.
VPTQ is a post-training quantization method for large language models that uses vector quantization, rather than the more common scalar quantization, to reach extremely low bit-widths of around two bits per parameter or less while preserving usable accuracy. It can be installed from PyPI and used through a Python API, a terminal chatbot example, or the Hugging Face Transformers integration.
The open-source community has contributed quantized versions of models including Llama 3.1 405B, Llama 3.3 70B, Mistral Large Instruct, and Nemotron 70B at various bit widths, and the project provides a Google Colab notebook and a free Hugging Face demo space for trying it without local hardware. Early ROCm support is also available alongside CUDA.
The VPTQ algorithm and its accompanying technical report were accepted at EMNLP 2024, and the project is distributed under the MIT license.