ExLlamaV3
An inference and quantization library for running large language models locally on consumer GPUs, with its own EXL3 quantization format and flexible parallel inference.
ExLlamaV3 is a library for running local LLMs on consumer-class GPUs. It implements EXL3, a quantization method based on QTIP, plus 2 to 8 bit cache quantization, and supports flexible tensor-parallel and expert-parallel inference across consumer hardware, with CPU offloading for large mixture-of-experts models on AVX2 and AVX512 systems.
It supports continuous and dynamic batching, speculative decoding and multimodal generation, broad Hugging Face model compatibility through a Transformers plugin, and an OpenAI-compatible API through the companion TabbyAPI server. It requires PyTorch with CUDA 12.4 or later and can be installed from a prebuilt wheel, PyPI, or built from source.