FastDeploy
FastDeploy is a PaddlePaddle-based inference and deployment toolkit for large language models and vision-language models, with an OpenAI-compatible API service and support for several accelerator families.
FastDeploy is an inference and deployment toolkit for large language models and vision-language models, built on the PaddlePaddle framework. It provides a production deployment path that includes a load-balanced prefill-decode split with context caching and dynamic instance role switching, and a lightweight key-value cache transfer library that chooses between NVLink and RDMA.
It exposes an OpenAI-compatible API service that can also be deployed with vLLM-compatible commands, and supports quantization formats including W8A16, W8A8, W4A16, W4A8, W2A16 and FP8, along with speculative decoding, multi-token prediction and chunked prefill for faster inference.
FastDeploy runs on Linux with Python 3.10 to 3.12, and supports NVIDIA GPUs as well as Kunlunxin, Hygon, Iluvatar, Enflame, Muxi and Intel Gaudi hardware.