Choosing the right inference framework
Compare the criteria that distinguish serving frameworks, and how to evaluate them for your workload.
Draft
Planning your deployment
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
Serving frameworks differ less in the features they list than in what they optimise for and how they behave at the edges of their operating range. The right comparison is against your own traffic pattern.
This page gives the criteria to compare on, and a method for evaluating candidates without committing to one first.
What you will learn
- The criteria that meaningfully separate serving frameworks.
- Which optimizations to check for, and why their presence alone is not enough.
- How to benchmark candidates against your own traffic pattern.
- What operational characteristics to weigh alongside raw performance.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.