Mosec
A high-performance model-serving framework, with a Rust web layer and a pure-Python interface, that adds dynamic batching and pipelined CPU/GPU stages to turn a trained machine-learning model into an efficient online service.
Mosec is a model-serving framework for building machine-learning-enabled backends and microservices, bridging the gap between a trained model and an efficient online service API in an ML-framework-agnostic way. Its web layer and task coordination are built in Rust for speed and efficient CPU utilization through asynchronous I/O, while its user interface is pure Python, letting a user serve a model with the same code used for offline testing.
It aggregates requests from different users into dynamic batches for batched inference and distributes results back individually, and spawns multiple processes for pipelined stages to handle mixed CPU, GPU and IO workloads without one stage blocking another. It is designed to run in the cloud, with model warmup, graceful shutdown and Prometheus monitoring metrics that a container orchestrator such as Kubernetes can manage directly.
It installs with pip, conda or pixi, or can be built from source with Rust, and its documentation walks through serving a pretrained Stable Diffusion model as a worked example.