Deconstructing Large Language Models: From Linear Regression to General AI
Companion code for a Chinese-language book that reimplements the core algorithms behind large language models from scratch, starting from linear regression and building up to the mechanisms behind ChatGPT.
This repository holds the code that accompanies a book on deconstructing large language models. Rather than relying on the abstractions of third-party open-source libraries, the book reimplements the core parts of each model with detailed comments so a reader can see how the underlying algorithms work, on the view that annotated code communicates precise mechanisms more directly than a purely textual explanation.
The accompanying scripts are meant to be run in the order given; each script lists the third-party libraries it needs at the top. Because results depend on random initialization, reruns may differ slightly without changing the overall conclusions, and the language-model-related code needs a GPU to run in reasonable time.
The material traces a learning path backward from large language models to the foundations that support them: linear regression, activation functions and logistic regression, backpropagation and optimization, convolutional networks, and the data-side topics of transfer learning, reinforcement learning, model interpretation and feature engineering.