OpenMythos
An independent, open-source research reimplementation of the Claude Mythos model architecture, a recurrent-depth transformer with a looped block and a sparse mixture-of-experts feed-forward network.
OpenMythos is an independent, community-built theoretical reconstruction of the Claude Mythos model architecture, based on publicly available research rather than any proprietary Anthropic system. It implements a Recurrent-Depth Transformer with three stages: an initial Prelude of transformer blocks, a looped Recurrent Block that repeats up to a configurable number of iterations, and a final Coda. Attention can switch between multi-head latent attention and grouped-query attention, and the feed-forward layer uses a sparse mixture of experts with both routed and shared experts, aimed at compute-adaptive, depth-variable reasoning.
It ships preconfigured model scales from 1 billion to 1 trillion parameters, and installs with pip, with an optional extra that enables Flash Attention 2 on CUDA. A training script for a 3-billion-parameter model on the FineWeb-Edu dataset supports both single-GPU and multi-GPU runs through PyTorch's distributed data parallel training with a sharded, streaming dataset.