Offline batch inference
Optimize for throughput when there is no interactive latency requirement.
Draft
Inference optimization
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
Not all inference is interactive. Bulk classification, dataset annotation and document processing have no user waiting, which removes the latency constraint that shapes online serving and changes every configuration decision.
This page covers how to configure for pure throughput.
What you will learn
- How offline and online serving objectives differ.
- Which configuration choices change when latency does not matter.
- How to make a long batch job restartable.
- How to schedule batch work alongside interactive traffic.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.