Static, dynamic and continuous batching
Compare batching strategies and understand why continuous batching dominates modern serving.
Draft
Inference optimization
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
Batching amortises the cost of moving weights through the accelerator across multiple requests. How requests are grouped determines both achievable throughput and the latency individual requests experience.
This page compares the three main strategies and explains the shift toward continuous batching.
What you will learn
- Why batching improves accelerator utilization at all.
- How static, dynamic and continuous batching differ.
- Why variable-length generation penalises static batching so heavily.
- How batching configuration trades throughput against tail latency.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.