kube-llmops
A Helm chart that deploys a complete LLM operations stack on Kubernetes in one command, covering model serving, a unified API gateway, observability, autoscaling and fine-tuning.
kube-llmops packages model serving through vLLM, llama.cpp or TEI, auto-selected by model format, a kubectl plugin with commands for deploying, scaling, canarying and testing models, and a Headlamp-based dashboard, alongside a LiteLLM gateway that gives every served model one OpenAI-compatible API with key management, rate limiting and budget controls. Observability is Prometheus and Grafana with prebuilt dashboards and alert rules, plus Langfuse tracing of LLM requests and Fluent Bit and Loki for logs, and KEDA autoscales serving pods on queue depth and latency percentiles, including scale-to-zero.
Fine-tuning runs through LLaMA-Factory with LoRA, QLoRA or full fine-tuning, orchestrated with Argo Workflows and tracked in MLflow, and a RAG stack combines Dify, pgvector and a TEI embedding and reranking service. Model files are cached in MinIO with a Hugging Face fallback, and Keycloak provides single sign-on for the dashboards.
It is written in Go and licensed under Apache-2.0.