LLM inference parameters
Understand the sampling and decoding parameters exposed at request time and their effects.
Draft
Model interaction
Draft page
This page is an outline. It describes what will be covered and is not yet complete technical documentation.
Temperature, top-p, penalties and stop sequences change generation behaviour substantially, and are often set by copying values from an unrelated example. Each one has a specific effect that is worth understanding directly.
This page explains what each parameter does and how they interact.
What you will learn
- What each common sampling parameter controls.
- How temperature and nucleus sampling interact.
- When deterministic decoding is appropriate and what it does not guarantee.
- How token limits and stop sequences bound a response.
Recommended outline
This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.