HyperQwen
A patched vLLM serving setup and requantization pipeline that runs a large Qwen model on a single consumer GPU, with an OpenAI-compatible API in single-user or batch modes.
HyperQwen packages a patch series against a pinned version of vLLM together with a model-preparation pipeline: requantized attention heads, a calibrated draft vocabulary, and speculative decoding that drafts from the prompt. It currently targets a large Qwen model running on a single 24 GB consumer GPU, exposing an OpenAI-compatible API with key authentication in either a single-user or a batch-serving mode.
It is distributed as a Docker Compose setup with separate profiles for single-stream chat and for many concurrent requests, and the first start downloads the container image and requantizes the model. The project reports token-per-second throughput on an RTX 3090 for both modes, and documents which parts of the approach are expected to generalize to other checkpoints and GPUs rather than being specific to the one it has measured.