Inference optimization
Techniques that increase throughput and reduce latency for a given model and hardware.
Techniques that increase throughput and reduce latency for a given model and hardware.
The pages in this section are listed below in the recommended reading order.
- LLM performance benchmarks Design inference benchmarks that predict production behaviour instead of flattering the system.
- Static, dynamic and continuous batching Compare batching strategies and understand why continuous batching dominates modern serving.
- PagedAttention Understand how PagedAttention improves KV cache memory management.
- Speculative decoding Use a cheaper draft model to propose tokens that the target model verifies in parallel.
- Prefill-decode disaggregation Run the prefill and decode phases on separate resources so each can be tuned independently.
- Prefix caching Reuse KV cache across requests that share a common prompt prefix.
- Inference routing Route requests across replicas using inference-aware signals rather than round-robin.
- KV cache offloading Move KV cache to host memory or storage to extend effective cache capacity.
- Data, tensor, pipeline, expert and hybrid parallelisms Understand the ways a model and its workload can be split across accelerators.
- Offline batch inference Optimize for throughput when there is no interactive latency requirement.