https://ait.bsc.es/handbook/kernel-optimization/ Recent content in Kernel optimization on AI Tools Hugo en https://ait.bsc.es/handbook/kernel-optimization/kernel-optimization-for-llm-inference/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/kernel-optimization/kernel-optimization-for-llm-inference/ <p>Kernel optimization is the lowest layer at which inference performance can be improved, and the one with the worst effort-to-reward ratio if applied before the layers above it have been addressed.</p> <p>This page frames when to descend to this level, and what the work actually consists of.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>What a kernel is and why kernel efficiency bounds performance.</li> <li>How to establish that kernels, rather than scheduling or batching, are the bottleneck.</li> <li>Which inference operations are most often worth optimizing.</li> <li>When to write a kernel versus adopting an existing one.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p> https://ait.bsc.es/handbook/kernel-optimization/gpu-architecture-fundamentals/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/kernel-optimization/gpu-architecture-fundamentals/ <p>Reasoning about kernel performance requires an accurate model of the hardware: how work is scheduled onto execution units, and how the memory hierarchy behaves under different access patterns.</p> <p>This page covers the architectural fundamentals that inference performance depends on.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>How GPU work is organised into threads, warps and blocks.</li> <li>The memory hierarchy and the cost of each level.</li> <li>Why coalesced memory access matters so much.</li> <li>How to reason about occupancy and arithmetic intensity.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p> https://ait.bsc.es/handbook/kernel-optimization/choosing-the-right-kernel-optimization-tool/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/kernel-optimization/choosing-the-right-kernel-optimization-tool/ <p>Kernels can be written at several levels of abstraction, from vendor assembly through domain-specific languages to compiler-generated code. The levels differ in achievable performance, portability and how much effort each change costs.</p> <p>This page compares them and suggests how to choose.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>The levels of abstraction available for kernel work.</li> <li>How portability and peak performance trade against each other.</li> <li>Where compiler-based approaches are sufficient.</li> <li>Which profiling tools to use to direct the effort.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p> https://ait.bsc.es/handbook/kernel-optimization/flash-attention/ Mon, 01 Jan 0001 00:00:00 +0000 https://ait.bsc.es/handbook/kernel-optimization/flash-attention/ <p>A textbook attention implementation materialises the full attention matrix in memory, making the operation memory-bound and its footprint quadratic in sequence length. That is the dominant cost for long contexts.</p> <p>FlashAttention restructures the computation so the matrix is never fully materialised, trading extra arithmetic for far less memory traffic.</p> <h2 id="what-you-will-learn">What you will learn</h2> <ul> <li>Why naive attention is memory-bound rather than compute-bound.</li> <li>How tiling and online softmax avoid materialising the attention matrix.</li> <li>Why trading recomputation for memory traffic is a net win here.</li> <li>How the approach applies differently to prefill and decode.</li> </ul> <h2 id="recommended-outline">Recommended outline</h2> <p>This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.</p>