OpenAI-Compatible vLLM Serverless Endpoint Worker
A Runpod Serverless worker template that deploys an OpenAI-compatible large language model endpoint powered by vLLM, either from a pre-built Docker image pointed at a model or by building an image with the model baked in.
The recommended path deploys any model with the pre-built runpod/worker-v1-vllm Docker image, configured through environment variables such as MODEL_NAME, HF_TOKEN for gated or private Hugging Face repositories, MAX_MODEL_LEN and QUANTIZATION (awq, gptq, squeezellm or bitsandbytes); its CUDA compatibility follows the CUDA runtime of the underlying vllm/vllm-openai base image used at build time.
An alternative path builds a Docker image with the model weights baked in directly, and a third option runs the official vLLM image as-is for a load-balancing endpoint that answers over direct HTTP instead of a job queue. Served endpoints follow the OpenAI Chat Completions and Responses API shapes, as well as an Anthropic Messages-compatible endpoint, alongside the worker's own standard, non-OpenAI, request and sampling parameters.