vLLM Backend
The official Triton Inference Server backend for running large language models on vLLM's engine, handling inflight batching and paged attention for served requests.
vLLM Backend is a Python-based Triton Inference Server backend that runs supported models on vLLM's AsyncEngine. Incoming requests are placed on the engine as soon as they arrive, and inflight batching and paged attention are handled by vLLM itself rather than by the backend.
It can be installed through a pre-built Triton Docker container from NVIDIA's NGC registry, built into a custom container from source, or added to an existing NGC Triton container after installing vLLM directly. A model is configured through a model.json file whose keys map to vLLM's AsyncLLMEngine arguments, including multi-GPU options such as tensor_parallel_size.