MLServer
An open-source inference server that serves machine learning models over REST and gRPC, compliant with the KFServing V2 Dataplane spec.
MLServer is an inference server for machine learning models, built to serve them over REST and gRPC using the standard V2 Inference Protocol. It supports multi-model serving within a single process, parallel inference across a pool of workers, and adaptive batching that groups requests together as they arrive.
It ships with pre-packaged runtimes for frameworks such as Scikit-Learn, XGBoost, LightGBM, CatBoost, Spark MLlib and MLflow, and supports writing custom runtimes for other frameworks. MLServer is designed for deployment in Kubernetes-native serving frameworks including Seldon Core and KServe, and is installed with pip.