Choosing the right GPU number
Decide how many accelerators a deployment needs, and when to scale up rather than out.
Draft
Planning your deployment
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
Accelerator count is driven by two independent requirements: fitting the model and its KV cache in memory at all, and serving the expected concurrent load within your latency targets. Either can dominate.
This page separates the two questions so they can be sized independently and then reconciled.
What you will learn
- How to separate the memory requirement from the throughput requirement.
- When to add accelerators to one replica versus adding replicas.
- How concurrency targets translate into hardware count.
- What headroom to leave for traffic variation.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.