Infrastructure and operations
Running inference as a reliable, observable and scalable service.
Running inference as a reliable, observable and scalable service.
The pages in this section are listed below in the recommended reading order.
- What is LLM inference infrastructure? Map the components that sit around a model server to make it a service.
- What is distributed inference? Understand what changes when a single request spans multiple accelerators or nodes.
- LLM observability Instrument an inference service so its behaviour can be understood in production.
- Fast scaling Reduce the time between a scaling decision and a replica serving traffic.
- InferenceOps and management The operational practices that keep an inference service healthy over time.