Building GLM 5.2 in C to Run a 744B Model on Consumer Hardware
A from-scratch pure C inference engine that runs GLM-5.2, a 744-billion-parameter mixture-of-experts model, on a single machine with about 16 GB of RAM by quantizing to int4 and streaming inactive experts from disk.
This repository documents and implements an inference engine for GLM-5.2, a 744-billion-parameter mixture-of-experts language model that is roughly 1.5 terabytes in bfloat16. Because only 8 of its 256 experts per layer fire for any given token, the engine keeps the always-active part of the model in memory and streams the sleeping experts from disk, so the working set stays small enough to run on a laptop with about 16 GB of RAM and no GPU.
The write-up and accompanying code cover quantizing the vendor's FP8 weights to int4, validating a smaller streaming engine token for token against PyTorch, implementing the forward pass with multi-head latent attention and a Mixture-of-Experts router, writing integer dot-product kernels checked against a plain C reference, building a multi-tier memory system across VRAM, RAM, cache and disk, and speculative decoding.
The engine, conversion tools and benchmark harness are derived from the colibri project and released under the Apache-2.0 license.