bitsandbytes
A PyTorch library that enables accessible large language models through 8-bit and 4-bit quantization, cutting memory use for inference and training.
bitsandbytes enables accessible large language models via k-bit quantization for PyTorch, reducing the memory needed for inference and training. It provides 8-bit optimizers using block-wise quantization that keep 32-bit-level performance at a fraction of the memory cost, and LLM.int8() 8-bit quantization for inference at roughly half the memory with no performance loss, using vector-wise quantization and separate 16-bit handling of outlier values.
Its QLoRA 4-bit quantization enables training of large models with memory-saving techniques, quantizing a model to 4 bits and inserting a small set of trainable low-rank adaptation weights. The library exposes these as quantization primitives (bitsandbytes.nn.Linear8bitLt, Linear4bit) and an optim module of 8-bit optimizers.
It requires Python 3.10 or newer and PyTorch 2.4 or newer, and supports CPU, NVIDIA and AMD GPU, Intel GPU and Intel Gaudi accelerators on Linux, Windows and aarch64 platforms, with varying support for each feature by platform.