Machine Learning Engineering Open Book
An open, continually updated book of methodologies, tools and step-by-step instructions for training and fine-tuning large language and multi-modal models, aimed at the engineers and operators who run that training.
Machine Learning Engineering Open Book is an open collection of methodologies, tools and step-by-step instructions for successfully training and fine-tuning large language models and multi-modal models, and running their inference. It targets LLM and VLM training engineers and operators, with scripts and copy-and-paste commands to address real needs quickly.
Its content spans hardware (compute, storage and networking), orchestration systems including SLURM, training and inference guides, and debugging, troubleshooting and testing practices, along with higher-level insights such as choosing a cloud provider and deciding when a GPU generation upgrade is worth its cost.
The material grew out of the author's own notes from training the BLOOM-176B and IDEFICS-80B models and RAG systems, and is available to read online or download as a PDF or EPUB ebook, under the CC BY-SA 4.0 license.