# OpenArch

Hand-written PyTorch implementations of modern open-source LLM architectures such as Llama, Qwen, DeepSeek and Gemma, built one model at a time for clarity and learning rather than production use.

OpenArch reimplements cataloged model architectures from their original papers, technical reports and reference configuration files, with each model in a single readable file so its structural choices are explicit and easy to compare side by side. It does not aim to compete with production libraries; its goal is clarity, contrasting differences in attention type (multi-head, grouped-query, multi-query, multi-head latent, sliding-window, linear or DeltaNet hybrids), normalization, positional encoding, dense versus sparse mixture-of-experts routing, and training-time tricks such as multi-token prediction.

Implemented models, marked usable for forward passes or still under construction, include GPT-2 XL, Llama 2, Llama 3, OLMo 2, DeepSeek R1, Gemma 3, Mistral 3, Llama 4 Maverick, Qwen 3 and Kimi K2, each documented with its size, normalization, positional encoding, attention type and mixture-of-experts status.

Website: https://github.com/anuj0456/OpenArch