BigMoeOnEdge
BigMoeOnEdge runs Mixture-of-Experts language models larger than a device's RAM, on a phone or PC, CPU-only, by streaming just the experts each token needs directly from flash storage, with output identical to running the full model in memory.
A Mixture-of-Experts model is made of many small experts, and each generated token only uses a few of them. BigMoeOnEdge takes that literally: it keeps the small always-needed part of the model resident and reads only the experts a given token requires directly from flash storage at the moment they are needed, leaving the rest of the model on disk. That lets a model several times larger than the device's RAM generate text losslessly, with output byte-identical to running the same model fully resident.
It is built on top of llama.cpp's public API, so every quantization format, tokenizer and chat template llama.cpp already supports works without changes, and following a new llama.cpp release is a routine update rather than a rewrite. Supporting a new Mixture-of-Experts architecture is a single registry entry rather than a change to the streaming engine itself.
The project demonstrates running a 284-billion-parameter model on a phone with 12 GB of RAM by streaming from the model's own sharded files with no merge step, and a newly released architecture running the same day its weights appeared, before mainline support for it existed elsewhere. It is written in C++.