MoE-Bench
A pip-installable benchmark toolkit that measures routing entropy, expert utilization and collapse in sparse Mixture-of-Experts language models.
MoE-Bench measures how a sparse Mixture-of-Experts model's routing actually behaves rather than only what it outputs, reporting per-layer, per-domain Shannon-entropy routing statistics, expert-utilization distributions with a configurable collapse threshold, and bootstrap confidence intervals, since existing evaluations mostly measure output quality and miss silent routing collapse where a few experts monopolize all tokens.
It validates out of the box on three architectures, OLMoE, JetMoE and Qwen1.5-MoE, and runs on a 24GB Mac. It also includes an entropy-floor training regularizer, evaluated with LoRA fine-tuning on OLMoE against a 12-subject MMLU subset, where it reduced routing collapse without a statistically significant drop in downstream accuracy in the project's own test.
The project is explicit that only its own measured column in its comparison tables reflects a real benchmark run; rows for other routing-loss methods are described as limitations drawn from their design, not measured results.