What is distributed inference?
Understand what changes when a single request spans multiple accelerators or nodes.
Draft
Infrastructure and operations
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
Once a model no longer fits on one accelerator, a single forward pass becomes a distributed computation with collective communication on its critical path. Interconnect performance and failure handling become first-order concerns.
This page covers what distribution adds beyond the parallelism schemes themselves.
What you will learn
- When distributing a single request becomes necessary.
- Why interconnect bandwidth and latency bound distributed inference.
- How a replica spanning several nodes is coordinated.
- What happens when one participant in a distributed replica fails.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.