# RTP-LLM

Alibaba's high-performance inference engine for large language models, used internally across Alibaba's businesses, with CUDA kernel optimizations, quantization, LoRA serving, and multi-GPU tensor parallelism.

RTP-LLM is a large language model inference acceleration engine developed by Alibaba's Foundation Model Inference Team, used within Alibaba Group to serve LLMs for businesses including Taobao, Tmall, Idlefish, Cainiao, Amap, Ele.me, AE, and Lazada. It is a sub-project of the havenask project.

It uses high-performance CUDA kernels such as PagedAttention, FlashAttention, and FlashDecoding, along with WeightOnly INT8 and INT4 quantization (with GPTQ and AWQ) and adaptive KV-cache quantization. It integrates with HuggingFace model formats including SafeTensors, PyTorch, and Megatron, can serve multiple LoRA adapters from one model instance, handles multimodal image-and-text input, and supports multi-machine, multi-GPU tensor parallelism and P-tuning models.

Additional techniques include loading pruned irregular models, a contextual prefix cache for multi-turn dialogue, a system prompt cache, and speculative decoding.

Website: https://github.com/alibaba/rtp-llm