TorchAO
A PyTorch-native library for quantizing and optimizing models across the training-to-serving lifecycle, covering low-bit formats, sparsity, and hardware-aware kernels.
TorchAO is a PyTorch-native library for model optimization spanning training through serving. It provides quantization techniques such as float8 and int4, along with structured and semi-structured sparsity, so models can train faster and run inference with less memory and compute.
It works out of the box with torch.compile and FSDP2 across most Hugging Face PyTorch models, and is integrated as a quantization backend into projects such as vLLM and Hugging Face Transformers. Reported results include float8 training that is faster on Llama-3.1-70B and 405B, and int4 quantization of Llama-3-8B that runs faster with substantially less memory, alongside quantization-aware training that recovers accuracy lost to quantization.
TorchAO is written in Python and maintained as part of the PyTorch project.