LLM observability
Instrument an inference service so its behaviour can be understood in production.
Draft
Infrastructure and operations
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
Generic service metrics do not explain inference behaviour. Request rate and error rate say nothing about queue depth, batch composition, cache hit rate or accelerator utilization, which are where inference problems actually appear.
This page covers what to instrument and how to make it actionable.
What you will learn
- Which inference-specific signals to collect beyond standard service metrics.
- How to instrument the phases of a request separately.
- Which signals belong on a dashboard and which should page someone.
- How to handle prompt and completion data given privacy constraints.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.