Prefix caching
Reuse KV cache across requests that share a common prompt prefix.
Draft
Inference optimization
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
Production traffic repeats itself. Shared system prompts, few-shot examples and multi- turn conversations all mean requests frequently share a long prefix whose KV cache was already computed.
Prefix caching reuses that work, which can remove most of the prefill cost for such requests.
What you will learn
- Why real traffic contains so much shared prefix.
- How cached prefixes are matched and reused.
- How prefix caching interacts with paged KV cache management.
- What cache invalidation and eviction require.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.