https://ait.bsc.es/handbook/infrastructure-and-operations/ Recent content in Infrastructure and operations on AI Tools Hugo en https://ait.bsc.es/handbook/infrastructure-and-operations/what-is-llm-inference-infrastructure/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/infrastructure-and-operations/what-is-llm-inference-infrastructure/ <p>A model server on its own is not a service. Between a client and a set of accelerators sit gateways, routers, model storage, autoscaling, observability and the control plane that manages them.</p> <p>This page maps those components and their responsibilities.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>Which components make up an inference platform.</li> <li>How a request travels through the stack.</li> <li>Where model artifacts are stored and how they reach a replica.</li> <li>Which responsibilities belong to the control plane versus the data plane.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p> https://ait.bsc.es/handbook/infrastructure-and-operations/what-is-distributed-inference/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/infrastructure-and-operations/what-is-distributed-inference/ <p>Once a model no longer fits on one accelerator, a single forward pass becomes a distributed computation with collective communication on its critical path. Interconnect performance and failure handling become first-order concerns.</p> <p>This page covers what distribution adds beyond the parallelism schemes themselves.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>When distributing a single request becomes necessary.</li> <li>Why interconnect bandwidth and latency bound distributed inference.</li> <li>How a replica spanning several nodes is coordinated.</li> <li>What happens when one participant in a distributed replica fails.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p> https://ait.bsc.es/handbook/infrastructure-and-operations/llm-observability/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/infrastructure-and-operations/llm-observability/ <p>Generic service metrics do not explain inference behaviour. Request rate and error rate say nothing about queue depth, batch composition, cache hit rate or accelerator utilization, which are where inference problems actually appear.</p> <p>This page covers what to instrument and how to make it actionable.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>Which inference-specific signals to collect beyond standard service metrics.</li> <li>How to instrument the phases of a request separately.</li> <li>Which signals belong on a dashboard and which should page someone.</li> <li>How to handle prompt and completion data given privacy constraints.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p> https://ait.bsc.es/handbook/infrastructure-and-operations/fast-scaling/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/infrastructure-and-operations/fast-scaling/ <p>Inference replicas scale slowly compared with stateless services, because a replica cannot serve until many gigabytes of weights have been fetched and loaded. Autoscaling that ignores this reacts long after the traffic has changed.</p> <p>This page covers where that time goes and how to reduce it.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>Which stages make up replica startup time.</li> <li>How model loading and image pull dominate that time.</li> <li>Which techniques shorten each stage.</li> <li>How to set autoscaling policy around the remaining latency.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p> https://ait.bsc.es/handbook/infrastructure-and-operations/inferenceops/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/infrastructure-and-operations/inferenceops/ <p>Beyond the initial deployment, an inference service needs release procedures, capacity management, cost attribution and incident response — the ordinary operational disciplines, adapted to workloads where a single replica may cost more per hour than an entire conventional service.</p> <p>This page collects those practices.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>How to release model and configuration changes safely.</li> <li>How to manage capacity and quota across tenants.</li> <li>How to attribute and control cost.</li> <li>What inference-specific incident response looks like.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p>