How to Train Your GPT
A twelve-chapter interactive textbook that teaches readers to build, train and run a modern transformer-based language model entirely from scratch, with every line of code explained.
How to Train Your GPT is a twelve-chapter, over 7,500-line interactive textbook that teaches how to build, train and run a modern language model from absolute scratch, covering the same architecture family behind ChatGPT, Claude, LLaMA and Mistral. Twenty-eight standalone topic explainers cover techniques such as RoPE, attention, RMSNorm, SwiGLU, KV cache and AdamW in depth, plus two narrative walkthroughs that trace a single sentence through the entire model step by step.
Readers write every part themselves, including the tokenizer, embeddings, attention, the training loop and the inference engine, with every line annotated to explain what it does and why. It assumes only basic Python and no prior machine learning background, and is aimed at developers curious how systems like ChatGPT work, students who want to understand transformers deeply, and engineers evaluating architecture trade-offs.
It is written as a Jupyter Notebook project and released under the MIT license.