vLLM with Continuum Scheduling
A modified build of vLLM that adds Continuum-style request scheduling to improve inference throughput, meant as a base for tuning key-value cache pinning logic on different workloads.
This repository patches vLLM with the scheduling behavior described in the Continuum paper, minus its estimation component, and can be run either in its original scheduling mode or with Continuum scheduling enabled through a command-line flag. Multi-node deployments are supported through sticky-session routing so related requests land on the same node.
The project includes an evaluation harness that starts the modified vLLM server and runs a SWE-bench evaluation against it, with configurable concurrency either as a fixed number of parallel workers or as a Poisson-distributed arrival rate, and separate scripts for analyzing the resulting latency metrics and submitting pass-rate results.
It requires a Hugging Face account with model access, GPUs with CUDA drivers, and is licensed under Apache-2.0.