GPT-NeoX
EleutherAI's library for training large-scale autoregressive language models on GPUs, built on NVIDIA's Megatron and augmented with DeepSpeed and other optimizations.
GPT-NeoX is EleutherAI's library for training large-scale language models on GPUs, based on NVIDIA's Megatron Language Model and augmented with techniques from DeepSpeed along with additional optimizations, meant as a centralized, accessible place to gather techniques for training large-scale autoregressive models. It is used in academic, industry and government labs, and has run at scale on systems including AWS, CoreWeave, and the Oak Ridge Summit and Frontier supercomputers.
Major features include distributed training with ZeRO and 3D parallelism, support for launching via Slurm, MPI or the IBM Job Step Manager, architectural features such as rotary and alibi positional embeddings, parallel feedforward attention layers and flash attention, predefined configurations for architectures such as Pythia, PaLM, Falcon and LLaMA, curriculum learning, and integration with Hugging Face's tokenizers and transformers libraries and experiment monitors such as WandB, Comet and TensorBoard.
The README notes that for generic inference rather than training models with billions of parameters from scratch, the Hugging Face transformers library, which supports GPT-NeoX models, is the better choice.