Large Language Model from Scratch
An educational project that trains a large language model from scratch through two pipelines: a native C++ engine for data ingestion, tokenization and training, and a Python pipeline using Apple's MLX framework that scales a decoder-only GPT model to 1.6 billion parameters.
This repository documents training a large language model from scratch rather than fine-tuning an existing one. Its C++ pipeline implements Parquet data ingestion, byte-pair-encoding tokenization and model training natively, while its Python MLX pipeline documents, chapter by chapter, the process of scaling a decoder-only GPT model to 1.6 billion parameters on Apple Silicon, including an out-of-memory crash on an M3 Ultra, memory-control fixes, hardware-specific optimization, and building an interactive text-generation playground.
The MLX pipeline includes a Gradio-based playground for generating Rust code completions from a trained checkpoint in a browser. The project describes itself as inspired by Sebastian Raschka's book on building large language models from scratch.
It requires Python 3.10 or later and is licensed under Apache-2.0.