loxilb-inference-gateway
An inference-aware Layer 4/7 load balancer for LLM serving fleets, forked from the loxilb cloud-native load balancer, adding routing that understands vLLM and SGLang KV caches.
Classic load-balancing policy cannot see that KV-cache locality dominates time-to-first-token, that prefill and decode scale differently, or that request cost varies enormously with prompt content; loxilb-inference-gateway adds AI-inference routing on top of loxilb's GoLang and eBPF data path to address exactly these problems.
It routes each request to the endpoint whose KV cache already holds the longest matching prefix, either through a zero-engine-change prefix-hash affinity mode or an engine-exact mode fed by the serving engines' own KV-cache event streams with capacity-weighted spillover; it also supports prefill/decode disaggregation with KV-transfer coordination, a feedback controller that continuously tunes routing weights from observed time-to-first-token, and per-endpoint inference metrics.
Every AI capability is opt-in per load-balancer rule; with none enabled, it behaves exactly like upstream loxilb. It is written in Go and licensed Apache-2.0.