LLM quantization
Reduce numeric precision to cut memory and increase throughput, and understand the accuracy cost.
Draft
Model preparation
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
Quantization stores and computes with lower-precision numbers, reducing memory footprint and often increasing throughput. The saving is real and frequently large; the accuracy cost is real too, and workload-dependent.
This page explains the main approaches and how to decide whether a given quantization is acceptable for your task.
What you will learn
- What quantization changes in the memory budget and the compute path.
- How post-training quantization differs from quantization-aware training.
- Which parts of a model are most sensitive to reduced precision.
- How to measure quality loss on your own task rather than trusting a general claim.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.