vLLM
A fast, memory-efficient library for serving and running inference on large language models, supporting hundreds of Hugging Face model architectures across many hardware backends.
vLLM is a library for LLM inference and serving. It manages the memory used for attention with PagedAttention, batches incoming requests continuously, and supports chunked prefill and prefix caching for high throughput, along with quantization formats such as FP8, INT8, INT4, GPTQ, AWQ and GGUF, and optimized attention and matrix-multiplication kernels.
It integrates with Hugging Face models directly and supports more than 200 model architectures, including decoder-only, mixture-of-experts, hybrid attention and state-space, and multimodal models, as well as embedding and reward models. It offers an OpenAI-compatible API server plus an Anthropic Messages API and gRPC support, streaming output, structured output, tool calling, multi-LoRA serving, and tensor, pipeline, data and expert parallelism for distributed inference.
It runs on NVIDIA, AMD and Intel GPUs, x86, ARM and PowerPC CPUs, and hardware plugins for Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend and other accelerators. It originated at UC Berkeley's Sky Computing Lab, is written in Python, and is Apache-2.0 licensed.