vLLM Production Stack
A reference implementation for deploying vLLM at cluster scale on Kubernetes, adding request routing, KV-cache offloading, and observability without changing application code.
vLLM Production Stack is a reference stack for taking vLLM from a single instance to a distributed, cluster-wide deployment without changing application code. It is deployed with Helm and combines a serving layer of vLLM engines running different models, a request router that directs requests to backends based on routing keys or session IDs to maximize KV-cache reuse, and an observability stack that monitors backend metrics through Prometheus and Grafana.
Step-by-step tutorials cover installing Kubernetes locally or on major cloud platforms such as AWS, GCP, and Lambda Labs, setting up a minimal deployment, customizing vLLM configuration, loading model weights, and enabling KV-cache offloading with LMCache. Planned work includes autoscaling based on vLLM-specific metrics, disaggregated prefill, and router improvements such as KV-cache-aware routing. It requires a Kubernetes environment with GPUs and is Apache 2.0 licensed.