vLLM Neuron Plugin
A community-maintained vLLM plugin that serves large language models on AWS Trainium and Inferentia through a Neuron backend, using the same vLLM APIs and configuration as other backends.
The vLLM Neuron plugin is the recommended way to serve large language models on AWS Trainium hardware, extending vLLM with a Neuron backend that exposes the same vLLM APIs and configuration options already familiar from other backends. It installs from source alongside the required Neuron SDK packages and is started the same way as any other vLLM deployment, through the vllm serve command with model and batching options.
Supported model families include Llama 3, GPT-OSS, and Qwen3-VL and Qwen3-Embedding for multimodal and embedding workloads, each tested end to end on Neuron hardware with accuracy validation against reference outputs and tracked performance benchmarks across releases. Feature support, including continuous batching and OpenAI-compatible APIs, is documented per model in the project's model-recipe cards, and earlier deployments built on the older NxD Inference plugin have a separate migration guide to move to this Neuron-backend-based plugin.