1Cat-vLLM
A vLLM engineering fork focused on making modern large language models run fast on older NVIDIA Tesla V100 (SM70/Volta) GPUs, rather than merely able to start on them.
1Cat-vLLM is a fork of the vLLM inference engine that treats NVIDIA Volta-generation, SM70 Tesla V100 GPUs as a first-class optimization target, rather than only a compatibility target. Its stated goal is not simply that a recent model can be started on a V100, but that modern models run on it at a genuinely useful speed.
The project reports around 260 tokens per second serving a 27-billion-parameter quantized model with speculative decoding across four V100 16GB GPUs, and describes a long-context attention optimization path that improved useful causal attention throughput on the same generation of hardware from an earlier baseline of about 18 teraflops per second to roughly 61 teraflops per second in its current default configuration, through techniques such as split-dimension attention kernels and GQA-packed wider tensor-core matrix multiplications. The project states that every benchmark figure is tied to its own specific hardware, model, context length, batch size, and decoding configuration, and that these numbers are not directly comparable to each other.
It is written in Python and released under the Apache-2.0 license.