Nanotron
A minimalist library from Hugging Face for pretraining transformer language models with 3D parallelism across many GPUs.
Nanotron is a library for pretraining transformer language models with a simple, flexible API, designed to be easy to use while still scaling efficiently to large multi-GPU training runs through three-dimensional parallelism. It ships example configurations, such as training a tiny Llama model on a single node of eight H100 GPUs in around ten minutes.
It integrates with the Hugging Face ecosystem for logging into the Hugging Face Hub and Weights and Biases, uses Git LFS to push trained models and datasets, and its authors publish an accompanying guide that explains how to scale LLM training efficiently using Nanotron in practice.
It is written in Python and released under the Apache License 2.0.