Kernel optimization for LLM inference
Understand when kernel-level work is warranted and what it involves.
Draft
Kernel optimization
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
Kernel optimization is the lowest layer at which inference performance can be improved, and the one with the worst effort-to-reward ratio if applied before the layers above it have been addressed.
This page frames when to descend to this level, and what the work actually consists of.
What you will learn
- What a kernel is and why kernel efficiency bounds performance.
- How to establish that kernels, rather than scheduling or batching, are the bottleneck.
- Which inference operations are most often worth optimizing.
- When to write a kernel versus adopting an existing one.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.