LightningLM
A reference training pipeline that grows a single architecture from a 2B dense seed to a 120B sparse mixture-of-experts model through four stages on a single eight-GPU node.
LightningLM is the reference training pipeline for the LightningLM 0.1V model family: one architecture grown through four stages, a 2B dense seed, a 5B mixture-of-experts model, a 9B mixture-of-experts model, and a 120B sparse mixture-of-experts model, trained through a technique the project calls TurboQuant-PreTraining on a single eight-GPU node. The 120B model, 118.67B stored and 5.93B active parameters across the top 12 of 460 routed experts, is released publicly on Hugging Face.
The repository is a systems companion to a research paper documenting the pipeline's three organizing disciplines: reversibility, state-preserving growth, and single-node economics. It bundles two further companion components: a 131K-token tokenizer covering English and major Brahmic scripts, and a Kronecker-structured byte-level embedding that replaces a much larger standard embedding table.
Setup runs a dependency script and a health check, then launches training stage by stage with provided shell scripts; it is released under the Apache-2.0 license.