Key metrics for LLM inference
Define TTFT, TPOT, end-to-end latency, throughput and goodput, and explain what each one hides.
Draft
Foundations
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
Inference performance cannot be summarised by a single number. A deployment tuned for tokens per second may feel unresponsive; one tuned for first-token latency may waste most of its accelerator capacity.
This page defines the metrics used throughout the handbook and, importantly, what each one obscures when quoted alone.
What you will learn
- The definitions of time to first token, time per output token and end-to-end latency.
- How throughput and concurrency relate, and why averages mislead.
- What goodput measures and why it matters under load.
- Which metrics to attach to a service-level objective.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.