# vllm-gaudi

A community-maintained hardware plugin that adds an Intel Gaudi accelerator backend to vLLM, so large language models can be served on Gaudi hardware through vLLM's existing serving stack.

vllm-gaudi is a community-maintained hardware plugin that integrates Intel Gaudi AI accelerators with vLLM to run large-language-model inference on that hardware. It follows vLLM's hardware-pluggable design, so a Gaudi backend can be added without changing vLLM's own codebase, and each release is built against and tested for compatibility with a specific vLLM version and a specific Gaudi software and PyTorch release.

Recent releases have added support for newly released model families and architectures, improved FP8 and other quantization schemes for memory and stability, reduced graph compilation and warm-up overhead, and adapted the plugin to upstream vLLM changes such as fused mixture-of-experts routing, key-value cache connectors and offloading. The project is released under the Apache-2.0 license and documents its compatibility matrix against upstream vLLM and Gaudi software releases.

Website: https://github.com/vllm-project/vllm-gaudi