vLLM GGUF Quantization Plugin
An out-of-tree vLLM plugin that restores GGUF quantization support after it was removed from vLLM's own codebase, letting vLLM serve GGUF-quantized text and vision-language models.
The plugin builds a CUDA or ROCm extension against the same PyTorch installation vLLM itself uses, installed as an editable package alongside vLLM rather than bundled into it. Once installed, a model is served the normal way with vLLM's serve command, naming a GGUF file and its matching tokenizer.
It reuses vLLM's own model implementations together with a generic GGUF weight adapter, so supported models extend beyond a fixed list, though the project's own generation tests cover text models such as Qwen 2.5, Qwen 3, Phi 3.5, GPT-2, StableLM, Gemma 3 and OLMoE, and vision-language models such as Gemma 3, Gemma 4, Qwen 3.5 and Qwen 3.6 at various quantization levels. Qwen 3.5's multi-token-prediction speculative decoding loads its extra weights directly from the same GGUF file rather than a separate download.