Inference routing
Route requests across replicas using inference-aware signals rather than round-robin.
Draft
Inference optimization
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
A generic load balancer treats inference replicas as interchangeable and requests as equal cost. Neither holds: replicas differ in cached prefixes and queue depth, and request cost varies by orders of magnitude.
Inference-aware routing uses those signals, and typically improves both latency and cache hit rates.
What you will learn
- Why generic load balancing underperforms for inference traffic.
- Which signals a router can use to place a request.
- How prefix-aware routing raises cache hit rates.
- How routing interacts with autoscaling.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.