moe-l2
An OpenAI-compatible inference proxy that offloads mixture-of-experts model layers so 100B-plus parameter models such as DeepSeek, Qwen and Mixtral can run on consumer GPUs with as little as 8 to 11 GB of VRAM.
moe-l2 predicts which experts a prompt's mixture-of-experts model will need and preloads them into a shared-memory LRU cache, so models that would otherwise need far more VRAM can run on low-VRAM consumer GPUs. It reports running a 16B-parameter MoE model on a 10 to 11 GB GPU, a 32B-parameter model in roughly 9.3 GB, and a 235B-parameter model on a 24 GB card, with measured VRAM savings of up to around 60 percent through selective or on-demand expert pinning.
It runs as a transparent, OpenAI-compatible proxy in front of llama.cpp-based inference, and supports multi-shard GGUF models. Measured throughput figures are published per GPU and model combination, alongside notes on known upstream llama.cpp bugs that affect specific model and hardware combinations.
It is written in Python and licensed under Apache-2.0.