speculators
A library for training speculative-decoding draft models that deploy directly into LLM inference engines such as vLLM, speeding up inference without changing output quality.
Speculators is a library for training the small draft models used in speculative decoding, a technique where a faster draft model proposes tokens that a larger base model verifies in a single forward pass, reducing inference latency while guaranteeing that every accepted token matches what the base model would have generated on its own. Trained models are packaged in reusable formats designed to run directly in vLLM.
Recent additions include multi-node online training through pluggable backends for transferring hidden states between vLLM and the trainer across machines, a DSpark training algorithm that extends anchored-block drafting with a Markov head and a confidence head, and support for the P-EAGLE training algorithm, which predicts multiple draft tokens in parallel rather than sequentially.