FlashAttention
Understand how FlashAttention reduces attention memory traffic through tiling and recomputation.
Draft
Kernel optimization
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
A textbook attention implementation materialises the full attention matrix in memory, making the operation memory-bound and its footprint quadratic in sequence length. That is the dominant cost for long contexts.
FlashAttention restructures the computation so the matrix is never fully materialised, trading extra arithmetic for far less memory traffic.
What you will learn
- Why naive attention is memory-bound rather than compute-bound.
- How tiling and online softmax avoid materialising the attention matrix.
- Why trading recomputation for memory traffic is a net win here.
- How the approach applies differently to prefill and decode.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.