Speculative decoding
Use a cheaper draft model to propose tokens that the target model verifies in parallel.
Draft
Inference optimization
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
Decoding is limited by memory bandwidth rather than compute, so a forward pass that verifies several candidate tokens costs little more than one that produces a single token. Speculative decoding exploits that.
This page explains the mechanism and the conditions under which it actually pays off.
What you will learn
- Why verifying several tokens costs little more than generating one.
- How draft-and-verify preserves the target model’s output distribution.
- Which draft model choices work, including model-free variants.
- When speculative decoding fails to help or hurts.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.