KServe
A standardized, distributed inference platform for deploying both generative and predictive AI models at scale on Kubernetes, incubating under the CNCF.
KServe unifies generative and predictive AI serving on one platform: for generative workloads it supports vLLM and llm-d backends, an OpenAI-compatible inference protocol, GPU acceleration, model caching, KV cache offloading to CPU or disk for longer sequences, and request-based autoscaling tuned for generative traffic patterns; for predictive workloads it supports TensorFlow, PyTorch, scikit-learn, XGBoost, and ONNX models, intelligent routing between predictor, transformer, and explainer components, canary rollouts and inference pipelines through InferenceGraph, scale-to-zero autoscaling, and built-in support for payload logging, outlier, adversarial, and drift detection.
It integrates with the Knative and Istio ecosystem and is used by a range of organizations in production. Documentation and presentations covering its various supported features are maintained on a companion website.
It is written in Go and distributed under the Apache-2.0 license.