Megatron-LM
A GPU-optimized library for training transformer language models at scale, providing both a ready-to-run reference training setup and composable building blocks for custom large-model training pipelines.
Megatron-LM and Megatron Core are GPU-optimized libraries for training transformer models at scale. Megatron-LM is a reference example that pairs Megatron Core with pre-configured training scripts, suited to research teams, learning distributed training, and quick experimentation.
Megatron Core is a composable library of GPU-optimized building blocks, including transformer layers, parallelism strategies such as tensor, pipeline, data, expert and context parallelism, mixed-precision support down to FP8 and FP4, and model architectures, aimed at framework developers building custom training pipelines. A companion component, Megatron Bridge, converts checkpoints between Hugging Face and Megatron formats.
It installs from PyPI or from source, and supports models such as DeepSeek-V4 and hybrid transformer architectures such as Falcon-H1. It is written in Python.