TUFF
A native macOS app and inference engine that runs large mixture-of-experts language models on Apple Silicon by streaming only the needed experts from SSD instead of holding a whole model in memory.
TUFF is a Mac app and inference engine for running a small, curated set of large language models on Apple Silicon. It keeps a model's shared weights resident in memory and streams mixture-of-experts weights from SSD through a bounded cache as they are needed, so the model size a machine can run is governed by disk space rather than RAM.
That trade costs decode speed compared with a model that fits fully in memory, so TUFF targets the case where a model would not otherwise fit at all; the project reports running a 61 GiB checkpoint on a 16 GB laptop. It includes a chat app, model downloader, command-line tools and a local OpenAI-compatible server, written in native Swift with a Metal runtime. It is Apache-2.0 licensed.