uzu
A high-performance, cross-language inference engine that runs AI models directly inside an app with low latency, full data privacy and no per-inference API cost, using unified memory on Apple devices.
uzu is a high-performance inference engine for AI models, meant to be embedded directly in an application so it runs with low latency, full data privacy and no ongoing inference costs. It provides a simple, high-level API, unified model configurations that make it straightforward to add support for new models, traceable computations checked against a source-of-truth implementation for correctness, and support for a broad range of models, using unified memory on Apple devices.
It handles model downloading and inference configuration automatically, and supports chat sessions that can be reused across multiple requests, chat with a cloud model instead of a local one, sharing one loaded chat instance to avoid reloading a model into memory, structured JSON output constrained by a schema, and external tool calls.
It is written in Rust, with bindings for Python, Swift and TypeScript, and is licensed under the MIT license.