WaferLLM: Large Language Model Inference at Wafer Scale
A research system, presented at OSDI 2025, for running large language model inference on wafer-scale AI accelerators instead of GPU clusters.
WaferLLM is described by its authors as the first large language model inference system designed for wafer-scale AI accelerators, which integrate hundreds of thousands of cores with large distributed on-chip memory and very high on-chip bandwidth in a mesh architecture. It is guided by a hardware model the authors call PLMR that captures the characteristics of wafer-scale hardware, and introduces MeshGEMM and MeshGEMV, described as the first matrix-multiplication and matrix-vector implementations designed to scale on this class of accelerator.
The authors report up to 200 times higher accelerator utilization than prior methods, and substantially faster and more energy-efficient matrix-vector operations and full inference compared with an NVIDIA A100 GPU cluster, measured on a Cerebras WSE2 wafer-scale accelerator.
Reproducing the results requires access to a Cerebras WSE-3 system or its SDK simulator, Python 3.8 or newer, and the Cerebras SDK; the code has been migrated to SDK 2.10.0 for WSE-3 from the original OSDI 2025 measurements on WSE-2.