OnnxStream
A lightweight C++ inference library for ONNX models built to minimize memory use rather than latency, able to run Stable Diffusion XL on a Raspberry Pi Zero 2 or Mistral 7B on a desktop, with ARM, x86, WebAssembly and RISC-V support.
OnnxStream is an inference library for ONNX models designed around minimizing RAM and VRAM usage rather than maximizing throughput, unlike most major machine learning frameworks. It decouples the inference engine from a separate WeightsProvider component responsible for loading, caching and prefetching model weights, so a custom provider can stream weights directly from an HTTP server without ever writing them to disk; three default providers cover disk without caching, disk with prefetching, and RAM.
This approach lets it run Stable Diffusion XL 1.0, a nearly one-billion-parameter transformer model, on a Raspberry Pi Zero 2 with 512MB of RAM, or in as little as 298MB of RAM on other hardware, at the cost of a 50% to 200% increase in latency compared to OnnxRuntime on CPU. It also supports Stable Diffusion 1.5 and XL Turbo, TinyLlama 1.1B and Mistral 7B chat, YOLOv8 object detection and OpenAI's Whisper, including WebAssembly demos of YOLOv8 and Whisper running in a browser, and has Python, C# and JavaScript bindings.
It is accelerated by XNNPACK, supports ARM, x86, WebAssembly and RISC-V, and is written in C++.