Training vs. inference
Compare the two workloads and explain why infrastructure tuned for training rarely suits serving.
Draft
Foundations
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
Training and inference both run the same model architecture on the same accelerators, which makes it tempting to treat them as one problem. Their resource profiles, failure modes and success criteria differ enough that conflating them leads to expensive mistakes.
This page contrasts the two so the rest of the handbook can assume an inference-first perspective.
What you will learn
- How the compute, memory and networking profiles of the two workloads differ.
- Why throughput is the goal in training and latency often is in serving.
- Which training-oriented infrastructure choices transfer to serving and which do not.
- How the cost model changes when a workload runs continuously.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.