vLLM Router
vLLM Router is a high-performance request router for large-scale vLLM deployments, providing cache-aware and other load-balancing algorithms plus prefill and decode disaggregation and Kubernetes-native service discovery.
vLLM Router forwards requests across a large-scale vLLM deployment, choosing among several load-balancing algorithms, cache-aware routing, power of two choices, consistent hashing, random and round robin, and supporting specialized routing for deployments that separate prefill and decode processing phases. It discovers and health-checks workers natively in Kubernetes, routes to Rust-based inference workers over gRPC, and adds circuit breakers, retry logic and metrics collection for production use.
Optional program scheduling features, admission control, a progress-based time-to-live, and global queue placement, target agent workloads that submit many related requests. An optional WASM middleware can inspect or modify requests such as chat completions, loaded as an independently built component that fails closed on any plugin error. It builds as a Rust binary with Cargo or installs as a Python package from PyPI.