Build a Large Language Model (From Scratch)
The official code repository for a book that teaches how large language models work by building a GPT-like model from scratch in PyTorch, covering pretraining and finetuning.
This repository holds the code for the book Build a Large Language Model (From Scratch), which teaches how large language models work by building a GPT-like model from the ground up. Each stage is explained with text, diagrams and examples, covering pretraining and finetuning a small but functional model using the same approach behind larger foundation models such as those behind ChatGPT.
The repository is organized by chapter, with a Jupyter notebook, a summary script, and exercise solutions for each one, covering topics from working with text data and coding attention mechanisms to pretraining, classification finetuning and instruction finetuning. An appendix introduces PyTorch itself, and the repository also includes code for loading and finetuning larger pretrained model weights.
It is written in Python using Jupyter notebooks and PyTorch.