AirLLM
A library that reduces the memory needed to run very large language models, letting models such as a 70-billion-parameter Llama or larger mixture-of-experts models run on a single small-VRAM GPU.
AirLLM streams a model's frozen weights through GPU memory one layer at a time instead of holding the whole model resident, without using quantization, distillation or pruning, so a 70-billion-parameter model can run on a single 4GB GPU. The project reports running larger open models this way too, including a 2.8-trillion-parameter mixture-of-experts model in under 4GB and a 671-billion-parameter model in around 12GB, by streaming only the expert weights a given token actually routes to.
A newer training mode extends the same layer-streaming approach to fine-tuning, keeping only trainable adapters resident on the GPU while frozen weights stream in, which the project reports lets a 125-billion-parameter mixture-of-experts model train under 6GB of VRAM. It supports CPU inference, non-sharded models, 8-bit and 4-bit quantization, and detects a model's type automatically through a single AutoModel interface across model families including Llama, Qwen, DeepSeek, Phi and Gemma.
It is distributed as a Python package with example notebooks and is licensed under Apache-2.0.