Mference
A Swift and Metal inference runtime for Apple Silicon that runs large mixture-of-experts language models by streaming only the active experts from disk instead of holding the whole model in memory.
Mference is a native Swift and Metal runtime for running large mixture-of-experts language models on Apple Silicon, including models whose full weights exceed the machine's RAM. Because a mixture-of-experts model only activates a fraction of its parameters per token, Mference keeps the shared core and key-value cache resident in memory and streams only the experts selected for the current token from SSD, bounding the working set instead of requiring the full model to fit in RAM. The same streaming approach also covers long-context key-value cache pages and a large n-gram embedding table used by one supported model family.
It supports a curated set of pinned instruction checkpoints, including Gemma, Qwen, DeepSeek-V4-Flash, and other mixture-of-experts and dense models, installed either from community conversions or through a quantize-in-flight path from vendor checkpoints, and reports measured decode and prefill throughput for each on specific Apple Silicon configurations.