LLMs from Scratch
A hands-on tutorial that builds a ChatGPT-style language model in PyTorch step by step, covering tokenization, embeddings, attention, and the transformer block through interactive notebooks.
LLMs from Scratch is a hands-on guide to building a ChatGPT-like large language model in PyTorch. It breaks the architecture into simple parts and explains each one in turn: tokenization, token embeddings, positional embeddings, the self-attention mechanism, masked multi-head attention, feedforward networks, residual connections, layer normalization, and the transformer block, ending in a small resulting model.
Each topic has a matching interactive Jupyter notebook that walks through the code for that component, so a reader can follow the model's construction piece by piece rather than reading a single large implementation.
It requires installing PyTorch and other dependencies from a requirements file, and is licensed under the MIT license.