Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion
A dual-architecture research framework that unifies the exact output of autoregressive large language models with the parallel, high-speed token generation of diffusion models, for strictly lossless faster inference.
Orthrus guarantees strictly lossless generation, matching the exact predictive distribution of the original base model, while breaking the sequential bottleneck of standard autoregressive decoding; the authors report up to a 7.8 times speedup on generation tasks. Both its autoregressive and diffusion views attend to the same key-value cache, so it adds only a constant amount of memory overhead rather than a separate draft model's cache.
Parallel-generation capability is added by fine-tuning only about 16 percent of a model's parameters while the base LLM stays frozen. Released checkpoints use a Qwen3 backbone at 1.7B, 4B and 8B sizes, with reported average speedups from 4.25 times to 5.36 times over standard decoding, and the authors report higher token acceptance rates and less throughput degradation at long context than speculative-decoding methods such as EAGLE-3 and DFlash.
It installs with uv pip install -e . plus flash-attention, and native integration with vLLM and SGLang is described as coming soon.