LLM Inference Handbook

Learn everything you need to design, deploy, optimize, and operate LLM inference systems.

This handbook is the conceptual half of the AI Tools documentation. It explains how LLM inference works, which decisions matter when you deploy it, and which techniques are worth applying — independently of any particular product. For the practical guide to the tool this team runs, see the LLM Inference module User Guide.

Status of this handbook

The handbook structure is complete and every page carries its scope, learning objectives and planned outline. Most pages are still marked draft: they describe what will be covered rather than containing finished technical content. Draft pages are labelled in the sidebar and on the page itself.

How the handbook is organised

The sections are ordered as a path from concepts to operations. Reading them in order works, but each section is self-contained enough to enter directly.

Who this is for

The handbook assumes you build or operate software, and does not assume you train models.

Application developers integrating an LLM will get the most from Model interaction and the llm-inference quickstart.
ML engineers choosing and preparing models should start with Planning your deployment and Model preparation.
Platform and infrastructure engineers running the service will care most about Inference optimization and Infrastructure and operations.

Where to start

If you are new to serving LLMs, begin with the Introduction, then What is LLM inference? — it establishes the prefill and decode vocabulary that the rest of the handbook relies on.

About this handbook

Why this handbook exists, who it is for, and how to read it.

Introduction Start here: what this handbook covers, who it is written for, and how it relates to the llm-inference service.

Foundations

The core vocabulary and mental models behind LLM inference.

What is LLM inference? Define LLM inference and distinguish the prefill and decode phases that shape every serving decision.
Training vs. inference Compare the two workloads and explain why infrastructure tuned for training rarely suits serving.
How does an LLM work? A serving-oriented tour of transformer inference: attention, layers and the autoregressive loop.
Where is LLM inference run? Survey the deployment targets for inference — cloud, on-premises, hybrid and edge — and their trade-offs.
Key metrics for LLM inference Define TTFT, TPOT, end-to-end latency, throughput and goodput, and explain what each one hides.

Planning your deployment

Sizing decisions to make before deploying: model, accelerators, memory and framework.

Choosing the right model Work through the trade-offs between model capability, size, licensing and serving cost.
Choosing the right GPU number Decide how many accelerators a deployment needs, and when to scale up rather than out.
Calculating GPU memory for serving LLMs Build a memory budget covering model weights, KV cache, activations and runtime overhead.
Choosing the right inference framework Compare the criteria that distinguish serving frameworks, and how to evaluate them for your workload.

Model preparation

Techniques applied to a model before it is served: fine-tuning, quantization and distillation.

LLM fine-tuning Understand when fine-tuning is the right tool, and what it means for serving.
LLM quantization Reduce numeric precision to cut memory and increase throughput, and understand the accuracy cost.
LLM distillation Train a smaller model to reproduce a larger one's behaviour on a target task.

Model interaction

The API surface and prompting techniques used to drive a served model.

OpenAI-compatible API Understand the de facto standard inference API and what compatibility does and does not guarantee.
Anthropic-compatible API Understand the Anthropic Messages API shape and how it differs from the OpenAI-compatible surface.
Function calling Let a model request the execution of tools you define, and handle those requests safely.
Structured outputs Constrain model output to a schema so downstream systems can consume it reliably.
Model Context Protocol Understand MCP as a standard interface between models and external tools and data sources.
Prompt engineering Techniques for constructing prompts that produce reliable output, and how to evaluate them.
LLM inference parameters Understand the sampling and decoding parameters exposed at request time and their effects.

Inference optimization

Techniques that increase throughput and reduce latency for a given model and hardware.

LLM performance benchmarks Design inference benchmarks that predict production behaviour instead of flattering the system.
Static, dynamic and continuous batching Compare batching strategies and understand why continuous batching dominates modern serving.
PagedAttention Understand how PagedAttention improves KV cache memory management.
Speculative decoding Use a cheaper draft model to propose tokens that the target model verifies in parallel.
Prefill-decode disaggregation Run the prefill and decode phases on separate resources so each can be tuned independently.
Prefix caching Reuse KV cache across requests that share a common prompt prefix.
Inference routing Route requests across replicas using inference-aware signals rather than round-robin.
KV cache offloading Move KV cache to host memory or storage to extend effective cache capacity.
Data, tensor, pipeline, expert and hybrid parallelisms Understand the ways a model and its workload can be split across accelerators.
Offline batch inference Optimize for throughput when there is no interactive latency requirement.

Kernel optimization

Optimizing the GPU kernels that inference workloads spend their time in.

Kernel optimization for LLM inference Understand when kernel-level work is warranted and what it involves.
GPU architecture fundamentals The GPU execution and memory model that kernel performance depends on.
Choosing the right kernel optimization tool Compare the approaches available for writing and tuning inference kernels.
FlashAttention Understand how FlashAttention reduces attention memory traffic through tiling and recomputation.

Infrastructure and operations

Running inference as a reliable, observable and scalable service.

What is LLM inference infrastructure? Map the components that sit around a model server to make it a service.
What is distributed inference? Understand what changes when a single request spans multiple accelerators or nodes.
LLM observability Instrument an inference service so its behaviour can be understood in production.
Fast scaling Reduce the time between a scaling decision and a replica serving traffic.
InferenceOps and management The operational practices that keep an inference service healthy over time.