Lucebox
A speculative-inference server with custom GPU kernels tuned for running large language models faster on consumer and heterogeneous hardware.
Lucebox is a speculative-inference server for large language models, built around custom kernels and speculative prefill and decoding techniques tuned for specific models and hardware targets. It targets consumer GPUs and heterogeneous machines that combine different accelerators rather than assuming a uniform data-center GPU cluster.
Reported measurements include several hundred tokens per second decoding a mid-sized Qwen model on a single consumer GPU, several times faster prefill on long-context models by combining its prefill and KV-cache acceleration techniques, and combined decode and vision throughput across a mix of two different accelerator types running models together. It documents which specific model, quantization, and drafter combination each measured result used.
It is written in C++ and released under the Apache License 2.0.