LiteRT-LM
Google's open-source orchestration layer for running large language models on edge devices, built for cross-platform, hardware-accelerated on-device inference.
LiteRT-LM runs on Android, iOS, web, desktop, and IoT devices such as a Raspberry Pi, using GPU and NPU acceleration where available, and supports vision and audio input alongside text and function calling for agentic workflows. It supports a broad range of models, including Gemma, Llama, Phi-4, and Qwen, and its CLI can run a model directly from a Hugging Face repository with options such as speculative decoding.
It is described as production-ready and already powers on-device generative AI features in Chrome, Chromebook Plus, and Pixel Watch, and a companion Google AI Edge Gallery app lets a user try bundled models immediately on their own device. A recent release added versioned C API prebuilt shared libraries so other applications and language bindings can integrate LiteRT-LM without building it from source.
It is written in C++ and distributed under the Apache-2.0 license.