CTranslate2
A C++ and Python library for efficient inference with Transformer models on CPU and GPU, using weight quantization, layer fusion and batch reordering to cut memory use and increase speed.
CTranslate2 is a custom inference runtime for Transformer models that applies optimizations such as weight quantization, layer fusion and batch reordering to reduce memory use and speed up inference on CPU and GPU. Compatible models must first be converted into its optimized format, using included converters for OpenNMT-py, OpenNMT-tf, Fairseq, Marian, OPUS-MT and Hugging Face Transformers.
It supports encoder-decoder models such as the base and big Transformer, M2M-100, NLLB, BART, mBART, Pegasus, T5 and Whisper; decoder-only models such as GPT-2, GPT-J, GPT-NeoX, OPT, BLOOM, Llama, Mistral, Gemma and Qwen2; and encoder-only models such as BERT, DistilBERT and XLM-RoBERTa. It supports reduced-precision formats including FP16, BF16, INT16, INT8 and AWQ INT4, and multiple CPU backends including Intel MKL, oneDNN, OpenBLAS, Ruy and Apple Accelerate, with automatic detection of the best backend and instruction set at runtime.