Optimum Quanto
Optimum Quanto is a PyTorch quantization backend for Hugging Face's optimum library, now in maintenance mode, that inserts quantization and dequantization automatically and supports int2, int4, int8 and float8 weights and activations across CPU, CUDA and MPS devices.
Optimum Quanto is a PyTorch quantization backend for Hugging Face's optimum library. It works in eager mode with non-traceable models, places quantized models on any device including CUDA and MPS, and automatically inserts the quantization and dequantization stubs, quantized functional operations and quantized modules a model needs.
It supports a workflow from a floating-point model to a dynamic and then a static quantized model, serialization compatible with PyTorch's weight-only format and Hugging Face safetensors, accelerated matrix multiplication on CUDA for several weight and activation bit-width combinations, and int2, int4, int8 and float8 weights with int8 and float8 activations. The project is now in maintenance mode, accepting only minor bug fixes and documentation pull requests, and points users wanting active development to alternatives such as bitsandbytes or torchAO.