vLLM Semantic Router
A routing layer for building Mixture-of-Models systems across heterogeneous LLM infrastructure. It evaluates request signals, user preferences, and application policies to select or compose the right model path for each request.
vLLM Semantic Router is a routing layer for building Mixture-of-Models systems across heterogeneous LLM infrastructure. It evaluates request signals, user preferences, and application policies to select or compose the right model path for a request instead of hard-coding routing logic into an application.
The project aims to improve quality, cost, latency, privacy, and safety by composing personalized model paths across GPUs, accelerators, edge, and cloud, while keeping data within its declared boundaries. It positions itself as infrastructure for building Mixture-of-Models systems rather than a single inference server.
It is written in Go and licensed under Apache-2.0, with topics including AI gateway, Kubernetes, guardrails, and inference.