Advanced Quantization Toolkit for LLMs
An Intel post-training quantization toolkit that compresses large language and vision-language models to two to four bits with minimal accuracy loss, using sign-gradient descent.
AutoRound is a quantization toolkit for large language models and vision-language models that reaches high accuracy at very low bit widths, from two to four bits, using a sign-gradient-descent tuning method with broad hardware compatibility. It supports calibration-free quantization through a grid-search method, composing multiple quantization algorithms together such as AWQ, Hadamard transforms and SignRound, and produces mixed-precision recipes automatically.
It integrates with widely used inference stacks, including Transformers, vLLM and SGLang, and supports output formats such as GGUF and weight-only quantization at five, six and seven bits on CUDA devices. Intel also offers free devices for quantization work through its Low Bit Open LLM Leaderboard program. It is written in Python and licensed under Apache 2.0, with English and Chinese documentation.