mlx-flash
A weight-streaming layer for Apple's MLX framework that runs language models larger than a Mac's RAM by loading weights from SSD as each layer executes, without quantizing them.
mlx-flash wraps a model's own layers in a streaming proxy so that, rather than loading every weight into memory up front, weights are read from SSD as each layer runs, using the model's original precision instead of an additional quantization pass. A predictive scheduler estimates the bandwidth the next layer will need while the current layer is still computing on the GPU, and paces reads with a token-bucket mechanism to keep GPU slowdown small.
The project reports running models such as a 30B model on 16GB of memory and 70B-class models on 32GB, inspired by Apple Research's LLM in a Flash paper, and states that its fused linear operator is numerically identical to MLX's own, so output does not diverge from a non-streamed run.
It is written in Python under the MIT license.