fastllm-proxy
A low-overhead LLM router written in Rust that fronts 80 providers and self-hosted vLLM or SGLang backends behind one OpenAI-compatible endpoint, reading existing LiteLLM-format configuration files unchanged.
fastllm-proxy is a router placed in front of any number of inference backends, including vLLM, SGLang, llama.cpp or 80 hosted providers, behind one address. It claims sub-microsecond overhead per request, does no I/O on the request path since access control, rate limits and budgets are integer comparisons against an in-memory snapshot, and never parses response bodies, so upstream stream frames reach the client unchanged.
It routes requests toward the node that already holds a matching prefix cache when providers such as vLLM or SGLang keep a radix or prefix key-value cache, rather than spreading requests evenly, unless that node is significantly more loaded than the least busy one. A proxy that loses its control plane keeps serving from its last known configuration snapshot, health is tracked separately per replica, and reloading configuration swaps the routing table without dropping an open stream.
It reads LiteLLM-format configuration files unchanged, so it can replace an existing gateway without rewriting configuration; clients connect on one port, and a management interface is served on another.