Dive into LLMs
A Chinese-language, hands-on tutorial that builds large language models from scratch, starting from a GPT-style model and extending into architecture walkthroughs of GLM4, Llama3 and RWKV6.
This Datawhale project, also titled LLMs From Scratch, is a Chinese-language hands-on tutorial for building a ChatGPT-like large language model from scratch, aimed at understanding core techniques rather than only using pretrained models. Its foundational section, based on an existing English tutorial project, walks through coding, pretraining and fine-tuning a GPT-like model with runnable notebooks and a more detailed accompanying explanation for each chapter.
A second section discusses and builds out the architectures of several large models, including ChatGLM3, Llama3 and multiple versions of RWKV, with configuration files, training scripts and core code for each. The tutorial's focus, in a field where fine-tuning and deployment guides are already common, is specifically on how these model architectures themselves are implemented.