Data, tensor, pipeline, expert and hybrid parallelisms
Understand the ways a model and its workload can be split across accelerators.
Draft
Inference optimization
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
When a model or its workload exceeds one accelerator, the work must be partitioned. The available partitioning schemes cut along different axes, impose different communication patterns, and are routinely combined.
This page explains each scheme and how hybrid strategies are assembled.
What you will learn
- What each parallelism scheme partitions, and along which axis.
- The communication pattern each one requires.
- Why expert parallelism suits mixture-of-experts models.
- How hybrid strategies are composed in practice.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.