llmedge
A lightweight Android library for running language, vision, speech and image-generation models entirely on-device, built on llama.cpp and related C++ inference engines.
llmedge runs GGUF-format language models on Android through llama.cpp accessed via JNI, with model downloads and caching from the Hugging Face Hub and ready-made presets tuned for constrained devices. It converts Hugging Face safetensors models to GGUF on-device for Llama-architecture models with a GPT2-BPE tokenizer, with optional quantization, and reuses the key-value cache natively for compact multi-turn chats.
Beyond text generation, it integrates Whisper.cpp for speech-to-text with timestamps and streaming transcription, Bark.cpp for text-to-speech, Stable Diffusion and FLUX-family models for image generation and editing with per-step progress streaming, a video-generation model, Google ML Kit for optical character recognition, and an on-device retrieval-augmented-generation pipeline for PDF indexing and question answering. Optional GPU acceleration tries OpenCL first, then Vulkan, falling back to the CPU.
Some capabilities, including vision-language model analysis and image and video generation, are described as still evolving. It is written in Kotlin and released under the Apache-2.0 license.