Calculating GPU memory for serving LLMs
Build a memory budget covering model weights, KV cache, activations and runtime overhead.
Draft
Planning your deployment
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
Running out of accelerator memory in production is one of the more common and more avoidable inference failures. A memory budget computed in advance turns it into a capacity decision rather than an incident.
This page sets out the components of that budget and how each one scales.
What you will learn
- The components that make up accelerator memory consumption.
- How to estimate weight memory from parameter count and precision.
- How KV cache size scales with batch size and context length.
- How much headroom to reserve for fragmentation and runtime overhead.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.