SNDR Core Engine
A set of runtime patches applied to vLLM in memory, without forking or rebuilding it, that speed up inference for the Qwen3.6 and Gemma 4 model families on consumer NVIDIA GPUs with 24GB or more of memory.
SNDR Core Engine patches vLLM at process start to run Qwen3.6 (up to 35B-A3B) and Gemma 4 (up to 31B) models faster on cards such as the RTX 3090, 4090, 5090 and RTX A5000/A6000, reporting about 1.5 times the decode speed of stock vLLM and context windows up to 280,000 tokens on 24GB-class hardware. Its optimizations, including a key-value cache compression scheme, hybrid state-space attention support and speculative decoding, are tuned specifically for these two model families rather than being general-purpose.
Alongside the inference patches, the project includes a persistent memory component meant to let both local and cloud models retain information across sessions, and a GUI control center for managing the running engine.
It is written in Python, licensed under Apache-2.0, and installs with a shell script.