Prefill-decode disaggregation
Run the prefill and decode phases on separate resources so each can be tuned independently.
Draft
Inference optimization
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
Prefill is compute-bound and decode is bandwidth-bound. Running both on the same replica means one phase’s requirements always compromise the other’s, and interference between them shows up directly in tail latency.
Disaggregation separates them, at the cost of moving KV cache between resources.
What you will learn
- Why colocating prefill and decode causes interference.
- How a disaggregated deployment routes a request through both stages.
- What transferring KV cache between stages costs.
- When the added complexity is justified.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.