https://ait.bsc.es/handbook/foundations/ Recent content in Foundations on AI Tools Hugo en https://ait.bsc.es/handbook/foundations/what-is-llm-inference/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/foundations/what-is-llm-inference/ <p>Inference is the process of using a trained model to produce output for new input. For large language models this happens in two distinct phases with very different performance characteristics, and almost every serving trade-off in this handbook follows from that asymmetry.</p> <p>This page establishes the vocabulary used throughout: prefill, decode, tokens, context, and the KV cache.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>What happens between a request arriving and tokens being returned.</li> <li>Why prefill is compute-bound and decode is memory-bandwidth-bound.</li> <li>What the KV cache is and why it dominates memory planning.</li> <li>The vocabulary reused across the rest of the handbook.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p> https://ait.bsc.es/handbook/foundations/training-vs-inference/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/foundations/training-vs-inference/ <p>Training and inference both run the same model architecture on the same accelerators, which makes it tempting to treat them as one problem. Their resource profiles, failure modes and success criteria differ enough that conflating them leads to expensive mistakes.</p> <p>This page contrasts the two so the rest of the handbook can assume an inference-first perspective.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>How the compute, memory and networking profiles of the two workloads differ.</li> <li>Why throughput is the goal in training and latency often is in serving.</li> <li>Which training-oriented infrastructure choices transfer to serving and which do not.</li> <li>How the cost model changes when a workload runs continuously.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p> https://ait.bsc.es/handbook/foundations/how-does-an-llm-work/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/foundations/how-does-an-llm-work/ <p>You do not need to be able to derive backpropagation to serve a model well, but you do need an accurate picture of what the hardware is doing during each forward pass. This page provides that picture, at the level of detail that serving decisions actually depend on.</p> <p>The emphasis is on where the time and the memory go, rather than on model quality or training dynamics.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>The shape of a transformer forward pass, layer by layer.</li> <li>Why attention cost grows with sequence length.</li> <li>How autoregressive generation turns one request into many forward passes.</li> <li>Which parts of the computation dominate time and memory.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p> https://ait.bsc.es/handbook/foundations/where-is-llm-inference-run/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/foundations/where-is-llm-inference-run/ <p>Where inference runs determines what you can control and what you must accept. The decision is rarely purely technical: data residency, procurement, existing hardware and the shape of demand all constrain it.</p> <p>This page lays out the options and the criteria for choosing between them.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>The main deployment targets and what distinguishes them.</li> <li>How data residency and compliance constrain the choice.</li> <li>When on-premises accelerators are worth the operational burden.</li> <li>Why demand shape matters as much as demand volume.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p> https://ait.bsc.es/handbook/foundations/key-metrics/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/foundations/key-metrics/ <p>Inference performance cannot be summarised by a single number. A deployment tuned for tokens per second may feel unresponsive; one tuned for first-token latency may waste most of its accelerator capacity.</p> <p>This page defines the metrics used throughout the handbook and, importantly, what each one obscures when quoted alone.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>The definitions of time to first token, time per output token and end-to-end latency.</li> <li>How throughput and concurrency relate, and why averages mislead.</li> <li>What goodput measures and why it matters under load.</li> <li>Which metrics to attach to a service-level objective.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p>