llm-d
A Kubernetes-native, distributed inference-serving stack, backed by Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA, that adds routing, KV-cache management and autoscaling on top of model servers such as vLLM and SGLang.
llm-d is a Cloud Native Computing Foundation sandbox project that provides orchestration and optimizations for serving large language models at scale on Kubernetes, sitting above model servers such as vLLM and SGLang. It aims to reach state-of-the-art inference performance across hardware accelerators and infrastructure providers, backed by tested guides and real-world benchmarks.
Its features include prefix-cache and load-aware request routing with experimental latency-based scheduling, tiered KV-cache offloading to CPU or disk with global indexing for multi-turn requests, prefill/decode disaggregation and expert-parallelism for serving very large models, and SLO-aware autoscaling with flow control for multi-tenant serving. It also provides OpenAI-compatible batch APIs for large-scale offline inference.