DeepSpeed
A deep learning optimization library for distributed training and inference of large models, providing memory- and compute-efficient parallelism techniques such as ZeRO and 3D parallelism.
DeepSpeed is a library of system-level optimizations for training and running large deep learning models at scale. Its techniques include ZeRO and ZeRO-Infinity for partitioning optimizer state, gradients and parameters across devices, 3D parallelism combining data, pipeline and tensor parallelism, Ulysses sequence parallelism for long sequences, and a mixture-of-experts implementation.
It integrates with popular open-source deep learning frameworks, including Hugging Face Transformers and Accelerate, PyTorch Lightning, MosaicML, Determined and MMEngine. According to the project, it has been used to train large-scale models such as Megatron-Turing NLG, BLOOM, GLM, GPT-NeoX and others.
It is written in Python and distributed under the Apache-2.0 license.