vllm-plugin-FL
A vLLM plugin, built on the FlagOS multi-chip backend, that lets the same vLLM commands run model inference across many different AI chip vendors without changing vLLM's interfaces.
vllm-plugin-FL extends the vLLM inference and serving framework to run across a wide range of hardware using FlagOS's unified multi-chip backend, which includes a unified operator library called FlagGems and a unified communication library called FlagCX. It does not change vLLM's original interfaces or usage patterns, so the same command can serve a model on different chips without modification.
An adaptation gate validates text, image, and mixed text-and-image requests across the plugin's required models in both eager and graph execution modes, tracking which models and chips are fully supported versus still merging. Supported models include several Qwen and GLM variants, MiniCPM-o, and MiniMax-M2.7, and supported chip vendors include NVIDIA, Ascend, MetaX, T-Head, Iluvatar, Tsingmicro, Moore Threads, Hygon, and Sunrise, alongside ARM64 CPUs.
It maintains separate branches tracking different community vLLM versions, is written in Python, and is Apache-2.0 licensed.