Local LLM Server for Android
Turns an Android phone into a private, local inference server, running Google's Gemma 4 models through LiteRT and exposing them over an OpenAI-compatible API on the local network.
This project repurposes an Android phone, including older devices, as a dedicated local AI server. It runs Google DeepMind's Gemma 4 models using Google's LiteRT runtime and exposes an OpenAI-compatible chat completions endpoint, so any OpenAI-compatible client such as the OpenAI Python SDK or LangChain can send requests to it.
The server binds only to private local network addresses and cannot be reached from the internet; there is no authentication, since access is restricted to devices on the same Wi-Fi network. Streaming responses are supported, though sampling parameters such as temperature and max_tokens are fixed for consistency and performance, and the model always generates a complete response.
The motivation is cost and power efficiency: an older flagship phone's GPU can run inference at a fraction of the power draw of a desktop GPU, giving retired hardware, including devices with cracked screens, a second life as a dedicated inference appliance.