What is LLM inference?
Define LLM inference and distinguish the prefill and decode phases that shape every serving decision.
Draft
Foundations
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
Inference is the process of using a trained model to produce output for new input. For large language models this happens in two distinct phases with very different performance characteristics, and almost every serving trade-off in this handbook follows from that asymmetry.
This page establishes the vocabulary used throughout: prefill, decode, tokens, context, and the KV cache.
What you will learn
- What happens between a request arriving and tokens being returned.
- Why prefill is compute-bound and decode is memory-bandwidth-bound.
- What the KV cache is and why it dominates memory planning.
- The vocabulary reused across the rest of the handbook.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.