Qwen 3.8 27B ROCmFP4_FAST on AMD Strix Halo (Ryzen AI Max+ 395)
A memory-optimized ROCm build of Qwen 3.8 27B for AMD Strix Halo APUs, combining 4-bit block quantization, speculative decoding, and a KV-cache scheme to raise inference throughput on unified-memory hardware.
This repository packages Qwen 3.8 27B for AMD Strix Halo (Ryzen AI Max+ 395 / Radeon 8060S) APUs, targeting single-machine inference on 128 GB of unified memory. It combines a custom 4.26-bit block quantization format, multi-token-prediction speculative decoding, an asymmetric KV-cache quantization scheme, and a cooperative-matrix attention engine.
The GGUF model weights are published on Hugging Face, and the format requires a patched llama.cpp fork pinned to a specific commit; the standard llama.cpp or Ollama builds cannot load these files. The README documents the quantization rationale, benchmark numbers, context-scaling behavior, and build and troubleshooting steps for the required engine.
A related repository extends the same approach to other models and other AMD Radeon GPUs.