Train LLM From Scratch
A hands-on tutorial and codebase that implements a transformer language model from scratch in plain PyTorch, walking from raw text through pretraining to a fine-tuned, reasoning-style chat model.
Train LLM From Scratch is a tutorial and codebase that implements a transformer language model from scratch in plain PyTorch, based on the original 'Attention Is All You Need' paper, letting a reader train their own model from a few million up to billions of parameters on a single GPU. It began as a pretraining tutorial and now covers the whole path from raw text to an aligned, reasoning-style assistant, with every algorithm written by hand rather than relying on existing training libraries.
The path it walks goes from raw text to tokens, through a transformer, to a next-token loss and a base model, and then through post-training stages: supervised fine-tuning, a reward model, proximal and direct preference optimization, and group relative policy optimization, ending in evaluation and a chat interface. Each building block, including the multilayer perceptron, single- and multi-head attention, and the full transformer, is explained in plain language before its code, with the expected output shown afterward.
It includes a Streamlit control panel for interacting with a trained model and a companion documentation site, and is written to work for students, developers and researchers alike.