llama.cpp — TurboQuant + TriAttention
A fork of llama.cpp that adds TurboQuant, custom low-bit quantization formats with CUDA kernels for smaller memory use, and TriAttention, GPU-accelerated KV-cache pruning that scores cached tokens by importance to keep long-context inference inside a fixed memory budget.
This fork adds two features to llama.cpp. TurboQuant introduces low-bit quantization formats named turbo2, turbo3 and turbo4 with hardware-tuned CUDA kernels aimed at faster inference with a smaller memory footprint. TriAttention prunes the KV cache by periodically scoring every cached token's importance, using the geometric structure of RoPE-encoded key vectors rather than any additional model weights or fine-tuning, and evicting the lowest-scoring tokens once a fixed token budget is reached.
Measured on a Qwen3-8B model on an RTX 3080, moving TriAttention's scoring from the CPU to the GPU cuts scoring overhead from about 5,900 milliseconds to 4 to 9 milliseconds per eviction event, which in turn keeps the KV cache within its VRAM budget and raises generation speed from 17.5 to 75.0 tokens per second by avoiding eviction stalls. Enabling TriAttention requires a calibration file generated from a representative text corpus beforehand.
Pre-built Windows binaries requiring CUDA 13 are distributed through Hugging Face; the project is licensed under MIT.