LMDeploy
A toolkit for compressing, deploying and serving large language and vision-language models, with a TurboMind inference engine and PyTorch-based engine support.
LMDeploy is a toolkit for compressing, deploying and serving LLMs. Its TurboMind engine targets high-throughput inference, with recent support for MXFP4 quantization on NVIDIA GPUs starting from the V100 and comprehensive optimization for FP8 mixture-of-experts models, alongside 4-bit weight-only quantization for both language and vision-language models.
It supports a broad range of models, including Llama, InternLM, InternVL vision-language models, Qwen and DeepSeek, with disaggregated deployment for DeepSeek through integration with DLSlime and Mooncake, and tool-calling support for models such as Llama 3.1. Its PyTorch engine additionally supports Huawei Ascend hardware and CUDA-graph-based acceleration.
The project's inference optimizations for TurboMind have been described in a paper accepted to EuroSys 2027, and it installs via pip install lmdeploy.