KV cache offloading
Move KV cache to host memory or storage to extend effective cache capacity.
Draft
Inference optimization
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
Accelerator memory is the binding constraint on how much KV cache can be retained, and therefore on how much prefill work can be reused. Host memory and fast storage are far larger and far slower.
Offloading trades transfer latency for cache capacity, which is worthwhile when recomputation would cost more than the transfer.
What you will learn
- Which tiers KV cache can be offloaded to.
- How to decide between recomputing and fetching offloaded cache.
- How transfer bandwidth bounds the benefit.
- How offloading composes with prefix caching.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.