What is LLM inference infrastructure?
Map the components that sit around a model server to make it a service.
Draft
Infrastructure and operations
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
A model server on its own is not a service. Between a client and a set of accelerators sit gateways, routers, model storage, autoscaling, observability and the control plane that manages them.
This page maps those components and their responsibilities.
What you will learn
- Which components make up an inference platform.
- How a request travels through the stack.
- Where model artifacts are stored and how they reach a replica.
- Which responsibilities belong to the control plane versus the data plane.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.