v100-skinny
A CUDA kernel implementation that runs the Qwen3.6-27B language model on four Tesla V100 GPUs at up to 366 tokens per second, using hand-written NVFP4 kernels and speculative decoding.
v100-skinny lets older Tesla V100 GPUs, common in aging servers and workstations, run a modern large language model at high speed. According to the project, it runs the Qwen3.6-27B model across four V100 cards at up to 366 tokens per second by using hand-written NVFP4 CUDA kernels that talk directly to the GPU hardware, combined with chain multi-token-prediction speculative decoding, and compressing the model to fit V100 memory and tensor-core capabilities.
Running it starts a local server that exposes a simple web chat interface on localhost once the model has loaded, requiring no separate installation beyond downloading and extracting the release archive and running a provided startup script for the user's operating system.