ParoQuant
An INT4 weight-quantization method for LLMs that uses learned pairwise rotations to suppress weight outliers, closing most of the accuracy gap to FP16 while running close to AWQ speed on NVIDIA GPUs and Apple Silicon.
ParoQuant (Pairwise Rotation Quantization), from an ICLR 2026 paper, targets efficient inference for reasoning LLMs by learning pairwise rotations that reduce the outliers that normally hurt low-bit quantization accuracy. It supports NVIDIA GPUs through vLLM and Transformers, and Apple Silicon through MLX, with oMLX recommended for serving.
Pre-quantized models are published on Hugging Face, including Qwen3.8 and Gemma 4 variants, and can be served with an interactive chat CLI, an OpenAI-compatible API server, through vLLM's own vllm serve or the project's own serving script, or a Docker image for NVIDIA GPUs that persists the kernel cache across runs. Vision-language components load by default for vLLM and can be included or skipped with a flag for MLX and vLLM respectively.