OME (Open Model Engine)
A Kubernetes operator for enterprise LLM serving that automates model management, GPU scheduling and deployment across runtimes such as SGLang, vLLM, TensorRT-LLM and Triton.
OME (Open Model Engine) is a Kubernetes operator for deploying and operating large language models at enterprise scale. Models are custom Kubernetes resources; the operator parses a model's architecture, parameter count and capabilities directly from its files, supports distributed storage with automated repair, double encryption, namespace scoping, and multiple formats such as SafeTensors, PyTorch, TensorRT and ONNX, and includes a reference catalog of more than 200 pre-configured models across families such as Llama, Qwen, DeepSeek, Gemma and Phi.
It automatically matches a model to an optimal runtime configuration through weighted scoring based on architecture, format, quantization, parameter size and framework compatibility, and supports deployment patterns such as prefill-decode disaggregation, multi-node inference, and canary or blue-green rollouts with advanced scaling controls. It integrates SGLang as its primary runtime, with cache-aware load balancing, multi-node deployment, disaggregated serving and multi-LoRA adapter serving, and also supports vLLM for high-throughput inference.
Hardware-aware scheduling is handled through resources describing GPU capabilities and selection policies such as best-fit, cheapest or most-capable, and an alpha fleet-capacity add-on manages GPU quotas across large clusters.