LitGPT
A library of more than 20 large language models implemented from scratch in plain PyTorch, with recipes to pretrain, fine-tune and deploy them at any scale from one GPU to a thousand.
LitGPT provides more than 20 large language models, including the Llama 3 family, implemented from scratch in plain PyTorch with no hidden abstraction layers, alongside recipes to pretrain, fine-tune and deploy them. Each model can be loaded and run in a few lines of code, and supports flash attention, fully sharded data parallel training, and parameter-efficient fine-tuning methods such as LoRA, QLoRA and adapters.
It supports quantization down to 4-bit precision to reduce GPU memory use, and the same recipes scale from a single GPU up to more than a thousand GPUs or TPUs. Models are installed with pip, and the full source can also be built from a local clone.
It is Apache 2.0 licensed for unrestricted commercial use, and is maintained by Lightning AI, whose separate cloud platform offers GPU workspaces, training and inference clusters, and hosted notebooks for the same workflows.