https://ait.bsc.es/handbook/inference-optimization/ Recent content in Inference optimization on AI Tools Hugo en https://ait.bsc.es/handbook/inference-optimization/llm-performance-benchmarks/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/inference-optimization/llm-performance-benchmarks/ <p>Most published inference numbers are not reproducible in your environment, because they encode a traffic pattern, a hardware configuration and a measurement methodology that are not yours.</p> <p>This page covers how to benchmark your own deployment in a way that supports capacity and configuration decisions.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>Which workload parameters must be specified for a benchmark to mean anything.</li> <li>How to model a realistic request arrival pattern.</li> <li>Why steady-state and cold-start measurements differ.</li> <li>How to report results so others can reproduce them.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p> https://ait.bsc.es/handbook/inference-optimization/batching/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/inference-optimization/batching/ <p>Batching amortises the cost of moving weights through the accelerator across multiple requests. How requests are grouped determines both achievable throughput and the latency individual requests experience.</p> <p>This page compares the three main strategies and explains the shift toward continuous batching.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>Why batching improves accelerator utilization at all.</li> <li>How static, dynamic and continuous batching differ.</li> <li>Why variable-length generation penalises static batching so heavily.</li> <li>How batching configuration trades throughput against tail latency.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p> https://ait.bsc.es/handbook/inference-optimization/paged-attention/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/inference-optimization/paged-attention/ <p>Naively allocating a contiguous KV cache block per request wastes a large fraction of accelerator memory, because allocations must be sized for the longest possible output rather than the actual one.</p> <p>PagedAttention applies the idea behind virtual memory paging to the KV cache, trading a level of indirection for substantially better memory utilization.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>Why contiguous KV cache allocation fragments memory.</li> <li>How paging the KV cache into fixed-size blocks addresses that.</li> <li>How paging enables sharing between requests with common prefixes.</li> <li>What the indirection costs.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p> https://ait.bsc.es/handbook/inference-optimization/speculative-decoding/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/inference-optimization/speculative-decoding/ <p>Decoding is limited by memory bandwidth rather than compute, so a forward pass that verifies several candidate tokens costs little more than one that produces a single token. Speculative decoding exploits that.</p> <p>This page explains the mechanism and the conditions under which it actually pays off.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>Why verifying several tokens costs little more than generating one.</li> <li>How draft-and-verify preserves the target model&rsquo;s output distribution.</li> <li>Which draft model choices work, including model-free variants.</li> <li>When speculative decoding fails to help or hurts.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p> https://ait.bsc.es/handbook/inference-optimization/prefill-decode-disaggregation/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/inference-optimization/prefill-decode-disaggregation/ <p>Prefill is compute-bound and decode is bandwidth-bound. Running both on the same replica means one phase&rsquo;s requirements always compromise the other&rsquo;s, and interference between them shows up directly in tail latency.</p> <p>Disaggregation separates them, at the cost of moving KV cache between resources.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>Why colocating prefill and decode causes interference.</li> <li>How a disaggregated deployment routes a request through both stages.</li> <li>What transferring KV cache between stages costs.</li> <li>When the added complexity is justified.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p> https://ait.bsc.es/handbook/inference-optimization/prefix-caching/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/inference-optimization/prefix-caching/ <p>Production traffic repeats itself. Shared system prompts, few-shot examples and multi- turn conversations all mean requests frequently share a long prefix whose KV cache was already computed.</p> <p>Prefix caching reuses that work, which can remove most of the prefill cost for such requests.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>Why real traffic contains so much shared prefix.</li> <li>How cached prefixes are matched and reused.</li> <li>How prefix caching interacts with paged KV cache management.</li> <li>What cache invalidation and eviction require.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p> https://ait.bsc.es/handbook/inference-optimization/inference-routing/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/inference-optimization/inference-routing/ <p>A generic load balancer treats inference replicas as interchangeable and requests as equal cost. Neither holds: replicas differ in cached prefixes and queue depth, and request cost varies by orders of magnitude.</p> <p>Inference-aware routing uses those signals, and typically improves both latency and cache hit rates.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>Why generic load balancing underperforms for inference traffic.</li> <li>Which signals a router can use to place a request.</li> <li>How prefix-aware routing raises cache hit rates.</li> <li>How routing interacts with autoscaling.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p> https://ait.bsc.es/handbook/inference-optimization/kv-cache-offloading/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/inference-optimization/kv-cache-offloading/ <p>Accelerator memory is the binding constraint on how much KV cache can be retained, and therefore on how much prefill work can be reused. Host memory and fast storage are far larger and far slower.</p> <p>Offloading trades transfer latency for cache capacity, which is worthwhile when recomputation would cost more than the transfer.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>Which tiers KV cache can be offloaded to.</li> <li>How to decide between recomputing and fetching offloaded cache.</li> <li>How transfer bandwidth bounds the benefit.</li> <li>How offloading composes with prefix caching.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p> https://ait.bsc.es/handbook/inference-optimization/parallelism/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/inference-optimization/parallelism/ <p>When a model or its workload exceeds one accelerator, the work must be partitioned. The available partitioning schemes cut along different axes, impose different communication patterns, and are routinely combined.</p> <p>This page explains each scheme and how hybrid strategies are assembled.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>What each parallelism scheme partitions, and along which axis.</li> <li>The communication pattern each one requires.</li> <li>Why expert parallelism suits mixture-of-experts models.</li> <li>How hybrid strategies are composed in practice.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p> https://ait.bsc.es/handbook/inference-optimization/offline-batch-inference/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/inference-optimization/offline-batch-inference/ <p>Not all inference is interactive. Bulk classification, dataset annotation and document processing have no user waiting, which removes the latency constraint that shapes online serving and changes every configuration decision.</p> <p>This page covers how to configure for pure throughput.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>How offline and online serving objectives differ.</li> <li>Which configuration choices change when latency does not matter.</li> <li>How to make a long batch job restartable.</li> <li>How to schedule batch work alongside interactive traffic.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p>