# strix-halo-sglang

An SGLang Docker image, with several out-of-tree patches, for running LLM inference on AMD's consumer Strix Halo APU (Ryzen AI Max+ 395 / Radeon 8060S), a platform AMD's own official images do not target.

AMD's official `rocm/sgl-dev` SGLang images target only MI300/MI350 data-center GPUs; this image fills that gap for Strix Halo (gfx1151) consumer hardware. It has been tested on a Fedora 43 host with ROCm 7.13 nightly and PyTorch 2.13.

With a set of patches applied, it supports chat completion and tool calling, RadixAttention prefix caching, continuous batching (reported at 4.76 times an Ollama comparison at eight concurrent streams on a Qwen3.5-35B-A3B mixture-of-experts model), and quantized dense layers, including int4 attention, projection and `lm_head` layers on RDNA, where upstream support is Marlin-only and CUDA-specific.

A range of models has been tested and their status recorded, from a 0.6B smoke-test model up through a 35B mixture-of-experts model in AWQ 4-bit and MXFP4 quantized forms; some GPTQ-quantized models fail to load because they require a CUDA-only backend, with an AWQ variant offered as a working alternative.

Website: https://github.com/JeremiahM37/strix-halo-sglang