Air.rs
A hardware-agnostic LLM inference engine written in Rust that streams model weights layer by layer from NVMe storage, so a single consumer GPU can run large models that do not fit in its VRAM.
Air.rs implements a layer-streaming inference protocol, memory-mapping a model file and streaming one transformer layer at a time from NVMe rather than requiring the whole model to fit in GPU memory. It targets running large models, such as 70B-parameter LLMs, on a single consumer GPU without CPU offloading, multi-GPU parallelism or heavy quantization losses.
Version 1.3.0 adds a speculative council decoding mode, a ternary model format with a self-speculative engine, INT8 matrix multiplication with AVX-512 VNNI acceleration, tiered KV-cache offload to system memory, per-batch CUDA graph registries, and an OpenAI-compatible REST server with TLS support. It runs on Linux, Windows and macOS, including Apple Silicon via Metal compute kernels.
It exposes a Python API with async streaming support for frameworks such as FastAPI, alongside an interactive terminal REPL and a command-line interface.