LLM performance benchmarks
Design inference benchmarks that predict production behaviour instead of flattering the system.
Draft
Inference optimization
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
Most published inference numbers are not reproducible in your environment, because they encode a traffic pattern, a hardware configuration and a measurement methodology that are not yours.
This page covers how to benchmark your own deployment in a way that supports capacity and configuration decisions.
What you will learn
- Which workload parameters must be specified for a benchmark to mean anything.
- How to model a realistic request arrival pattern.
- Why steady-state and cold-start measurements differ.
- How to report results so others can reproduce them.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.