# BetterGPT

A from-scratch, hand-built decoder-only transformer training project, currently a 152-million-parameter model pretrained on about 15 billion tokens with RoPE, RMSNorm and SwiGLU, released as a Hugging Face-compatible checkpoint with a live demo.

BetterGPT is an engineering project to build and train a modern decoder-only transformer language model from scratch, implementing every stage by hand: tokenizer, data pipeline, architecture, pretraining loop, evaluation and Hugging Face integration. The current model, BetterGPT-150M, has 152 million parameters across 18 layers with rotary position embeddings, RMSNorm and SwiGLU feed-forward layers, and is pretrained on roughly 15 billion tokens using a two-stage Warmup-Stable-Decay schedule: a broad stable phase over general web, educational, math and code data, followed by an annealing phase that increases the share of math and instructional text.

Training data draws from FineWeb-Edu, Cosmopedia, FineMath and StarCoder-Python, streamed and interleaved. Earlier project stages are also documented and tagged: a character-level Shakespeare model, then a 30-million-parameter PyTorch pretraining pipeline, then the same model with Hugging Face integration and Alpaca-style supervised fine-tuning. The 152M model is live on Hugging Face with an interactive demo space.

It is a learning and reference project for understanding LLM training end to end, with supervised fine-tuning and preference alignment on the current model still in progress.

Website: https://github.com/harikrish2727/BetterGPT