PagedAttention
Understand how PagedAttention improves KV cache memory management.
Draft
Inference optimization
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
Naively allocating a contiguous KV cache block per request wastes a large fraction of accelerator memory, because allocations must be sized for the longest possible output rather than the actual one.
PagedAttention applies the idea behind virtual memory paging to the KV cache, trading a level of indirection for substantially better memory utilization.
What you will learn
- Why contiguous KV cache allocation fragments memory.
- How paging the KV cache into fixed-size blocks addresses that.
- How paging enables sharing between requests with common prefixes.
- What the indirection costs.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.