GSQ: Gumbel-Softmax Quantization for LLMs
A post-training quantization method for large language models that learns per-coordinate grid assignments and per-group scales through a Gumbel-Softmax relaxation, targeting 1 to 3 bits per parameter.
GSQ is a post-training scalar quantization method for large language models that learns per-coordinate grid assignments and per-group scales using a Gumbel-Softmax relaxation of the discrete quantization grid. It keeps a symmetric, group-wise scalar format directly compatible with existing INT inference kernels and GGUF K-Quant deployment stacks, while closing most of the accuracy gap between simple scalar post-training quantization methods and second-wave vector or trellis methods at 2 to 3 bits per parameter.
The same discrete-assignment optimization scales from dense models such as Llama-3.1-8B and 70B up to trillion-parameter mixture-of-experts models, and can refine publicly released GGUF K-Quant checkpoints in place: refined Unsloth checkpoints run unchanged on llama.cpp or Ollama, with the largest gains at the aggressive Q2_K setting.
It is released under the Apache-2.0 license and accompanies a research paper from ISTA, ETH Zurich, TU Wien and Red Hat AI.