LLMKube
A Kubernetes operator for self-hosted LLM inference across a mixed fleet of NVIDIA, AMD and Apple Silicon hardware, turning model deployment into a two-line YAML definition with GPU scheduling, caching and OpenAI-compatible endpoints.
LLMKube lets a developer define a Model and an InferenceService as Kubernetes custom resources, and the operator handles downloading, caching, GPU scheduling, health checks, scaling and exposing an OpenAI-compatible API, across NVIDIA CUDA, AMD Vulkan and Apple Silicon Metal nodes from one spec. A ModelRouter resource adds policy-aware routing between local models and external providers such as Anthropic, OpenAI or LiteLLM, with per-rule timeout budgets, a half-open circuit breaker and fail-closed behavior for regulated data.
An optional add-on called Foreman is a Kubernetes-native control plane that dispatches coder, verifier and reviewer agents across the same heterogeneous fleet of locally hosted LLM nodes; it has already authored and submitted its own pull requests against the LLMKube repository. Supported runtimes include llama.cpp, vLLM, TGI and mlx-server, and an out-of-cluster metal-agent process handles OS-level supervision on Apple Silicon hosts. Apache-2.0 licensed.