# Large Language Model from Scratch

An educational project that trains a large language model from scratch through two pipelines: a native C++ engine for data ingestion, tokenization and training, and a Python pipeline using Apple's MLX framework that scales a decoder-only GPT model to 1.6 billion parameters.

This repository documents training a large language model from scratch rather than fine-tuning an existing one. Its C++ pipeline implements Parquet data ingestion, byte-pair-encoding tokenization and model training natively, while its Python MLX pipeline documents, chapter by chapter, the process of scaling a decoder-only GPT model to 1.6 billion parameters on Apple Silicon, including an out-of-memory crash on an M3 Ultra, memory-control fixes, hardware-specific optimization, and building an interactive text-generation playground.

The MLX pipeline includes a Gradio-based playground for generating Rust code completions from a trained checkpoint in a browser. The project describes itself as inspired by Sebastian Raschka's book on building large language models from scratch.

It requires Python 3.10 or later and is licensed under Apache-2.0.

Website: https://github.com/rahul238xaviers/large-language-model