LLM distillation
Train a smaller model to reproduce a larger one's behaviour on a target task.
Draft
Model preparation
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
Distillation transfers the behaviour of a large teacher model into a smaller student. Where it works, it changes the serving economics substantially, because the deployed model is genuinely smaller rather than merely compressed.
This page covers the approach, its requirements and its limits.
What you will learn
- What distillation is and how it differs from quantization and pruning.
- What data and compute a distillation run requires.
- Which task types distil well and which resist it.
- How to validate that the student preserves the behaviour you care about.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.