Fast scaling
Reduce the time between a scaling decision and a replica serving traffic.
Draft
Infrastructure and operations
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
Inference replicas scale slowly compared with stateless services, because a replica cannot serve until many gigabytes of weights have been fetched and loaded. Autoscaling that ignores this reacts long after the traffic has changed.
This page covers where that time goes and how to reduce it.
What you will learn
- Which stages make up replica startup time.
- How model loading and image pull dominate that time.
- Which techniques shorten each stage.
- How to set autoscaling policy around the remaining latency.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.