BentoML
A Python library for building online model-serving systems, turning inference scripts into REST APIs with Docker packaging and serving optimizations.
BentoML is a Python library for building online serving systems for AI applications and model inference. It turns an inference script into a REST API server using a few lines of code and standard Python type hints, and supports any ML framework, modality or inference runtime.
It manages environments, dependencies and model versions through a config file, and automatically generates Docker images for reproducible deployment. Serving features include dynamic batching, model parallelism, multi-stage pipelines and multi-model inference graphs to make better use of CPU and GPU resources.
Once a service is defined in a service.py file, it can be run locally with bentoml serve, called from a browser or a Python client, and packaged with bentoml build into a Bento, the project's deployable artifact, for deployment with Docker or BentoCloud.