Kthena
A lightweight, modular, Kubernetes-native platform for deploying and serving large language models, separating model lifecycle management from inference traffic routing behind custom resource definitions.
Kthena extends Kubernetes with custom resources for LLM workloads, supporting multiple inference engines including vLLM, SGLang and Triton, and serving patterns such as prefill-decode disaggregation. Its control plane, a workload controller managing model lifecycle and autoscaling, and its data plane, an intelligent router handling inference traffic, are fully decoupled: each talks only to the Kubernetes API, so a partial install of just one plane is a first-class option rather than a degraded mode.
The whole platform is two self-contained Go binaries with a small dependency surface, packaged as separate Helm subcharts with their own custom-resource groups and release lifecycles, so a team keeps its existing API gateway, ingress and observability tooling instead of replacing it. Optional capabilities such as gang scheduling, webhooks, Gateway API support and TLS stay opt-in so a minimal install stays minimal, while cost-driven autoscaling and heterogeneous accelerator support are available when needed.