Model interaction on AI Tools

OpenAI-compatible API

Mon, 01 Jan 0001 00:00:00 +0000

A large share of inference servers and client libraries speak an OpenAI-compatible HTTP API. Treating it as a portability layer is usually right, but compatibility is a spectrum rather than a guarantee.

This page describes the surface conceptually. For the endpoints exposed by this team’s service, see the LLM Inference module User Guide.

What you will learn

Which endpoints make up the commonly implemented core of the API.
How chat completions differ from the older text completion shape.
How streaming responses are delivered and consumed.
Where compatible implementations tend to diverge in practice.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Anthropic-compatible API

Mon, 01 Jan 0001 00:00:00 +0000

Some inference servers expose an Anthropic-compatible Messages API alongside or instead of an OpenAI-compatible one. The two express similar capabilities with different request structures, and the differences matter when writing portable clients.

This page covers the shape conceptually; the service-specific details are in the llm- inference user guide.

What you will learn

The structure of a Messages API request and response.
How system prompts and message roles are represented.
How tool use is expressed compared with the OpenAI-compatible surface.
What to account for when supporting both APIs from one client.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Function calling

Mon, 01 Jan 0001 00:00:00 +0000

Function calling lets a model return a structured request to invoke a tool rather than a direct answer. It is the mechanism underneath most agent architectures, and the point where model output starts to have side effects.

That makes both the schema design and the validation of the model’s requests security- relevant, not merely functional.

What you will learn

How a function-calling exchange proceeds, turn by turn.
How to describe tools so a model uses them correctly.
Why every tool call must be validated before execution.
How multi-step tool use composes into agent behaviour.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Structured outputs

Mon, 01 Jan 0001 00:00:00 +0000

Asking a model to reply in JSON and hoping for the best does not survive production traffic. Structured output support constrains generation so the result conforms to a schema by construction.

This page covers the available approaches and their costs.

What you will learn

The difference between prompting for a format and constraining generation to it.
How grammar and schema-constrained decoding work at a high level.
What constrained decoding costs in latency and flexibility.
How to handle the cases a schema cannot express.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Model Context Protocol

Mon, 01 Jan 0001 00:00:00 +0000

The Model Context Protocol standardises how a model-facing application connects to tools and data sources, so that integrations can be written once and reused across clients rather than rebuilt per application.

This page introduces the concepts and where MCP fits relative to plain function calling.

What you will learn

What problem MCP standardises, and for whom.
The roles of host, client and server in the protocol.
How MCP relates to and differs from direct function calling.
What to consider before exposing internal systems through an MCP server.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Prompt engineering

Mon, 01 Jan 0001 00:00:00 +0000

Prompt engineering has a poor reputation earned by folklore, but the underlying practice is ordinary engineering: change one thing, measure the effect on a fixed evaluation set, keep what works.

This page focuses on techniques that survive that treatment, and on the evaluation discipline that makes the difference.

What you will learn

Which prompt structures reliably improve task performance.
How to build an evaluation set before tuning prompts.
How prompt length interacts with latency and cost.
Why prompts should be versioned like any other production artifact.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

LLM inference parameters

Mon, 01 Jan 0001 00:00:00 +0000

Temperature, top-p, penalties and stop sequences change generation behaviour substantially, and are often set by copying values from an unrelated example. Each one has a specific effect that is worth understanding directly.

This page explains what each parameter does and how they interact.

What you will learn

What each common sampling parameter controls.
How temperature and nucleus sampling interact.
When deterministic decoding is appropriate and what it does not guarantee.
How token limits and stop sequences bound a response.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.