How To Scale Your Model
A short, blog-style textbook from Google DeepMind researchers that explains how TPUs work, how large language models run at scale, and how to choose parallelism strategies that avoid communication bottlenecks.
How To Scale Your Model is a written explainer aimed at demystifying the practice of scaling large language models on TPUs. It covers how TPUs work, how LLMs actually execute at scale, and how to choose parallelism schemes for training and inference that avoid communication bottlenecks, and it is published as a website built with Jekyll.
The repository includes scripts to build the site locally and to combine all chapters into a single Markdown or Word document. It was written by researchers at Google DeepMind and is released under the MIT license.