vLLM
A fork of vLLM, the high-throughput LLM inference and serving library, adding an experimental TurboQuant key-value cache path tuned for specific CUDA workstation GPUs.
vLLM is a library for fast large language model inference and serving, known for its PagedAttention memory management, continuous batching, fast CUDA and HIP graph execution, and support for quantization formats such as GPTQ, AWQ, and FP8. It serves most popular open models from Hugging Face, including large mixture-of-experts and multimodal models, through an OpenAI-compatible API, with tensor, pipeline, data, and expert parallelism for distributed inference across NVIDIA, AMD, Intel, and other hardware.
This fork extends vLLM's experimental TurboQuant key-value cache path with a workflow for supported CUDA workstation GPUs, adding TurboQuant cache recipes on the Triton attention backend, per-layer metadata loading, tensor-parallel metadata slicing for split key-value head layouts, and kernel tuning with a Triton prefill fast path for long-context comparisons.
It is written in Python under the Apache-2.0 license.