AI Tools

Errors and exit codes

Every event the tool reports, what it means, and how to fix it — plus the exit code table.

Draft

Draft page

This page is an outline. It describes what will be covered and is not yet complete technical documentation.

When something fails, the tool prints a structured JSON to stderr with an event field and a hint about what to do next. This page lists the events you will see most often.

How to read a failure

A failure on stderr
{"event": "model_rejected", "hint": "...", "...": "..."}

Two things tell you what happened: the event name (look it up below) and the process exit code (see the table at the end).

First two questions

Before digging in: does it affect every run or just this one, and did anything change — model, GPU count, --slurm values, input file? Those two answers eliminate most of the table below.

Common events

Event Exit Meaning Fix
model_rejected 3 The model is not allowed, or does not match the requested GPU count. Run llm-inference model list and pick a row with SUPPORTED=yes. Adjust --gpus to the allowed range.
invalid_gpu_node_layout 2 GPU and node combination is invalid. --gpus must be ≥ --nodes, divisible by it, and at most 4 per node.
invalid_slurm_params 2 You used a --slurm key that is not allowed. Only these are accepted: account, qos, partition, time, cpus-per-task, nodes, gres, mem, constraint.
outdir_creation_failed 7 Cannot create --outdir. Pick a path you can write to, for example under $SCRATCH.
input_file_not_found 7 The --in file does not exist. Use an absolute path on a shared filesystem (/gpfs or $SCRATCH).
endpoint_sidecar_not_found 7 batch run cannot find endpoint.json. Pass the same --outdir you used at endpoint start.
endpoint_not_running 5 The Slurm job is not in RUNNING state. Wait a few seconds and retry, or check out.txt / err.txt in --outdir.
endpoint_health_check_failed 5 vLLM started but /health did not answer in time. Tail out.txt; raise --health-check-timeout if the model loads slowly.
slurm_submit_failed 5 sbatch rejected the script. Run the generated script manually to see Slurm’s full message.
bash_inference_wait_timeout 5 The job did not finish within the wait budget. Raise --health-check-timeout, or use --no-wait and poll later.

Exit codes

Code Meaning
0 Success.
1 Batch ran but some requests failed.
2 Invalid arguments or rejected Slurm parameters.
3 Model rejected by the allow list.
4 Container image or local model directory not accessible.
5 Slurm error, or the job did not reach RUNNING in time.
7 Filesystem or runtime error (Singularity, vLLM, I/O).

Diagnostic recipes

The job never starts

PENDING for a long time is usually cluster load, not a fault. Check that your --slurm values are realistic — a very large cpus-per-task, GPU count or time= can be unschedulable.

The job starts and dies immediately

Read the logs in --outdir, in this order:

tail -50 /scratch/$USER/run-001/err.txt      # Slurm-level failures
tail -50 /scratch/$USER/run-001/out.txt      # vLLM startup
ls /scratch/$USER/run-001/slurm-*.log        # vLLM logs from inside the job

sbatch rejected the script

The generated script is in --outdir. Submit it by hand to see Slurm’s own error message, which is more specific than the tool’s:

sbatch /scratch/$USER/run-001/slurm_script.sh

The model loads too slowly

Large models can take several minutes to load. If /health times out before that:

llm-inference api start ... --health-check-timeout 900

The job was cancelled mid-batch

Almost always the time= limit. Lines already written are still in the output file — count them, and restart with the remaining requests and a longer time=:

time = model load time + (per-request time × number of requests) + 25% margin

Requests fail but the job succeeds

Exit code 1. Inspect the failures directly:

jq -r 'select(.status=="error") | "\(.id): \(.http_code) \(.error)"' results.jsonl

A http_code of 0 means the request never reached vLLM. Anything else is a real HTTP error from the engine — often a malformed request or a prompt exceeding the context window.

Preventing problems

Tip

  • Use --dry-run on every new configuration. It validates parameters and writes the script without spending GPU time.
  • Keep --outdir per run, not shared. Sidecars from different runs overwrite each other.
  • Be generous with time=. Queuing longer beats losing a half-finished batch.
  • Test with a two-request input file first, then scale up.