JetSpec
A speculative-decoding method for faster LLM inference that drafts a whole token tree in one parallel pass while preserving causal conditioning between branches, then verifies the entire tree against the target model in a single forward pass.
JetSpec speeds up LLM inference by having a causal-parallel draft head propose a tree of candidate tokens in one forward pass, conditioning each branch on the others, and then having the frozen target model verify every node of that tree in a single batched, tree-masked forward pass. Because the accepted path is chosen according to the target model's own logits, decoding stays lossless, meaning the output distribution matches ordinary autoregressive decoding.
The project reports up to roughly 9.6 times end-to-end speedup on a math benchmark using an 8-billion-parameter model, with smaller but still substantial gains across several other reasoning, code, and chat benchmarks, and describes throughput of more than 1000 tokens per second on coding and math tasks using high-end GPUs.
Pretrained draft heads for several target model families are published on Hugging Face, and the engine installs as an editable Python package with optional FlashAttention 2 support.