AI Tools

Choosing a mode

A decision table for the three ways of working with llm-inference.

Draft

Draft page

This page is an outline. It describes what will be covered and is not yet complete technical documentation.

The tool gives you three ways to work. This page helps you pick one in under a minute.

Decision table

If you need to… Use Guide
Process a single input file in one go and get a results file. bash inference One-shot inference
Keep an endpoint alive and send several batches to it. endpoint start + batch run Manual endpoint
Talk to the model from curl, the OpenAI SDK or your own code. api start Shared HTTP API
Just experiment, or validate your parameters. any mode with --dry-run Concepts

In one sentence each

  • Mode 1 — Manual endpoint. You decide when it starts and when it stops. Good for repeated batches against the same loaded model.
  • Mode 2 — One-shot inference. One command, one input file, one output file. Best for straightforward batch processing.
  • Mode 3 — Shared HTTP API. Exposes an OpenAI-style HTTP URL for curl or any client.

How they compare

Manual endpoint One-shot HTTP API
Commands to run 2+ (start, batch run, stop) 1 2+ (start, stop)
Model loaded once, reused once per job once, reused
Input .json file per batch .json file live HTTP requests
Output .jsonl file per batch .jsonl file HTTP responses
Ends when you run endpoint stop the batch finishes you run api stop
You must stop it yes no yes

Choosing the GPU count

All three modes take --gpus. Two rules constrain it:

  1. The model’s allowed range. llm-inference model list shows it per model. Outside that range the job is rejected with model_rejected.
  2. The node layout. --gpus must be greater than or equal to --nodes, divisible by it, and at most 4 per node. Otherwise you get invalid_gpu_node_layout.

For the reasoning behind how many GPUs a model actually needs, see Choosing the right GPU number and Calculating GPU memory for serving LLMs.