Tiny-MoE
A Mixture-of-Experts language model implemented from scratch in native PyTorch and trained on two free Kaggle T4 GPUs, meant to teach modern LLM internals.
Tiny-MoE is a from-scratch, bottom-up implementation of a Mixture-of-Experts decoder-only transformer, built to demystify the internals of modern MoE models rather than rely on high-level frameworks or black-box library calls. Every module, from the router to the expert layers, is implemented directly in PyTorch.
The model has roughly 150 to 200 million total parameters with about 70 to 90 million active per token, a base context of 512 tokens extended to 2048 via YaRN, and was trained on two NVIDIA T4 GPUs on Kaggle. It implements multi-head latent attention, rotary positional embeddings, YaRN long-context extension, a shared expert architecture, and weight absorption for optimized inference.
It is set up to clone and run with pip, with separate scripts for training and generation.