compressed-tensors
compressed-tensors is a library that extends the safetensors checkpoint format to store and load tensors compressed with quantization and sparsity schemes such as GPTQ, AWQ, SmoothQuant and SparseGPT in one unified format.
compressed-tensors extends the safetensors format to store and manage compressed tensor data under a single unified format, rather than each compression method needing its own storage layout and loading procedure. It supports formats including INT8, FP8, NVFP4, MXFP4 and MXFP8, activation quantization, mixed and low or arbitrary-bit precision, KV-cache quantization, and non-uniform schemes where different layers use different quantization approaches.
It also supports unstructured and semi-structured, such as 2:4, sparsity patterns, rotation-based quantization transforms including Hadamard and random-matrix variants, checkpoint conversion between formats such as AutoAWQ, ModelOpt NVFP4 and FP8 block, and transparent CPU, disk or distributed offloading for models larger than available VRAM. It is designed to work with Hugging Face models, PyTorch, vLLM and SGLang, and installs from PyPI or from source.