# MiniMind

MiniMind is a from-scratch project and tutorial for training a roughly 64-million-parameter language model in about two hours on a single consumer GPU, without relying on high-level training-library abstractions.

It implements a minimal model architecture in both dense and mixture-of-experts form, aligned to the Qwen3 family, along with its own tokenizer and a full pipeline covering pretraining, supervised fine-tuning, LoRA, RLHF through DPO, RLAIF through PPO, GRPO and CISPO, tool use, agentic reinforcement learning, and distillation, all written from scratch in PyTorch rather than through third-party training-library abstractions.

The same codebase extends to a vision-language variant, a multimodal omni model, a diffusion language model and a linear-attention model. It supports single- and multi-GPU training with Wandb or SwanLab logging, is compatible with common serving stacks such as llama.cpp, vLLM and Ollama, and includes a minimal OpenAI-compatible chat server and a Streamlit web demo.

Apache-2.0 licensed.

Website: https://jingyaogong.github.io/minimind