CSV-Decode
A decoding method and reference implementation that speeds up LLM inference by computing geometric upper bounds on token logits to skip most of the output vocabulary at each decoding step, without retraining the model.
CSV-Decode (Certifiable Sub-Vocabulary Decoding) targets the output-layer bottleneck in LLM inference, where computing logits over the full vocabulary at every decoding step is expensive even though only a small subset of tokens meaningfully affects the result. It clusters the vocabulary offline with k-means, then at each decoding step uses a Cauchy-Schwarz bound on cluster centroids to derive a safe upper bound on the logits of every token in a cluster, letting whole clusters be skipped without computing their individual logits.
The method provides two kinds of correctness guarantee: exact top-k certification and epsilon-certified softmax approximations, and works with any pretrained transformer-based LLM architecture without retraining, including Llama, CodeLlama, Qwen and DeepSeek. The authors report a 2.67 to 4.95x decoding speedup across multiple models and tasks with 99.3% quality retention and a 52% reduction in energy use compared to standard autoregressive decoding.
It ships as research code accompanying the method's paper rather than a packaged, production-ready library.