Planning your deployment
Sizing decisions to make before deploying: model, accelerators, memory and framework.
Sizing decisions to make before deploying: model, accelerators, memory and framework.
The pages in this section are listed below in the recommended reading order.
- Choosing the right model Work through the trade-offs between model capability, size, licensing and serving cost.
- Choosing the right GPU number Decide how many accelerators a deployment needs, and when to scale up rather than out.
- Calculating GPU memory for serving LLMs Build a memory budget covering model weights, KV cache, activations and runtime overhead.
- Choosing the right inference framework Compare the criteria that distinguish serving frameworks, and how to evaluate them for your workload.