Pulsar
A Rust and CUDA inference engine that streams a giant Mixture-of-Experts model's routed experts from NVMe storage per token, letting models far larger than available VRAM run on consumer GPUs by keeping only routing and attention resident in memory.
Pulsar runs large Mixture-of-Experts language models, including models with hundreds of billions to a trillion parameters, by keeping the components that make routing decisions resident in VRAM while streaming the routed experts themselves from NVMe storage on demand, per token. It is a from-scratch Rust and CUDA engine, the successor to an earlier C-based project, and includes no llama.cpp code anywhere in its stack.
It implements eleven distinct model architectures directly, including several sparse-attention and hybrid-attention designs, and reports specific decode speeds and configurations for each supported model on consumer-grade dual 16 GB GPU setups, down to which quantization level performs best for a given architecture and why. New architectures are frequently supported on the day of a model's release.
It automatically measures PCIe bandwidth and places attention and hot experts across GPUs to fit the available hardware, aiming for zero-configuration multi-GPU setup.