vllm-awq4-qwen
A pinned Docker Compose build that serves a 27-billion-parameter Qwen model, quantized to 4-bit AWQ, on AMD's Strix Halo integrated GPU through vLLM with speculative decoding and tool calling.
This project packages vLLM with a specific AWQ-quantized Qwen 3.6 27B model and DFlash speculative decoding tuned for AMD Strix Halo's gfx1151 integrated GPU with 128 GB of unified memory, running under ROCm rather than CUDA. It reports 24.8 tokens per second for a single stream and describes matching an NVIDIA DGX Spark's FP8 and speculative-decoding setup at roughly a third of the hardware cost.
Because AMD's nightly ROCm and PyTorch builds change daily and have broken the build before, every toolchain component, from the base Ubuntu image to torch, triton and vLLM itself, is pinned to one verified-working snapshot, with a documented process and a CMake patch for refreshing the pins later. The server exposes an OpenAI-compatible API and runs entirely through Docker.