v100-skinny
Hand-written CUDA kernels that run NVFP4 and FP8 quantized weights for the Qwen3.8 model family on older Tesla V100 GPUs, serving them through an OpenAI-compatible endpoint.
v100-skinny runs Qwen3.8-27B's published mixed NVFP4/FP8 weight format unchanged on four Tesla V100-SXM2-16GB GPUs, without requantizing to FP8 or keeping a full FP16 copy of the weights. NVFP4 regions are handled by a custom kernel called QPN2 and FP8 regions by QPN8, both built around Volta's mma.sync instruction, with FP16 activations and key-value cache.
It serves the model through an OpenAI-compatible endpoint rather than only demonstrating the kernels. The README reports a same-lab benchmark against an RTX 5090 running a different engine on the AIME 2026 problem set, showing comparable decode throughput once the two systems' round latency and tokens committed per round are combined.
The project is written in Python and targets four V100 GPUs that together cost a fraction of a current flagship card.