NNTrainer
An on-device neural network training and inference framework in C++ that runs 30-billion-parameter mixture-of-experts LLMs on a mobile phone by streaming only the active experts from flash storage instead of keeping the whole model in memory.
NNTrainer's Flash Storage Utilization technique loads only the currently active experts of a mixture-of-experts model from flash storage during inference, which the project reports reduces memory use for a 30-billion-parameter model from about 16.5 GB down to 1.3 GB, making it practical to run on a mobile device. Its LLM inference engine component is built for these resource-constrained environments, currently supporting several open Qwen, GPT-OSS, and Gemma model families in standard, slim, and cached-slim variants.
Additional optimizations include a mixture-of-experts cache that keeps frequently used experts in memory while swapping others to storage, proactive loading that predicts and pre-loads weights before they are needed, and a decoupled KV cache that separates query and key-value computation for more efficient long-context attention. Build scripts target Android, with JNI bindings, and Linux or PC through standard build tooling, and benchmark tools report thermal behavior, prefill and generation speed, and peak memory.
Beyond inference, NNTrainer supports on-device training of neural networks directly on the edge device rather than only in the cloud.