AI Tools

Quick reference

Every llm-inference command on one page.

Draft

Draft page

This page is an outline. It describes what will be covered and is not yet complete technical documentation.

A summary of the most common commands. Print this page if you want it next to you.

Check the tool

llm-inference --version
llm-inference --help
llm-inference model list

Load the module

module load EB/apps
module use /apps/ACC/LLM-INFERENCE/modulefiles
module load llm-inference/0.0.1-GCCcore-13.2.0

Mode 1 — Manual endpoint

# Start
llm-inference endpoint start --model <ID> --gpus <N> \
  --outdir <DIR> \
  --slurm account=<ACC> --slurm qos=<QOS> \
  --slurm partition=<PART> --slurm time=<HH:MM:SS> \
  --slurm cpus-per-task=<N>

# Status / stop
llm-inference endpoint status --job <JOB_ID>
llm-inference endpoint stop   --job <JOB_ID>

# Run a batch (repeatable while the endpoint is alive)
llm-inference batch run --in <IN.json> --out <OUT.jsonl> \
  --via-endpoint <JOB_ID> --outdir <DIR>

See Manual endpoint.

Mode 2 — One-shot inference

llm-inference bash inference --model <ID> \
  --in <IN.json> --out <OUT.jsonl> --outdir <DIR> \
  --gpus <N> \
  --slurm account=<ACC> --slurm qos=<QOS> \
  --slurm partition=<PART> --slurm time=<HH:MM:SS> \
  --slurm cpus-per-task=<N>

See One-shot inference.

Mode 3 — Shared HTTP API

# Start
llm-inference api start --model <ID> --outdir <DIR> --gpus <N> \
  --slurm account=<ACC> --slurm qos=<QOS> \
  --slurm partition=<PART> --slurm time=<HH:MM:SS> \
  --slurm cpus-per-task=<N>

# Status / stop
llm-inference api status --job <JOB_ID>
llm-inference api stop   --job <JOB_ID>

# Test it with curl
curl -sS http://<HOST>:<PORT>/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"<ID>","messages":[{"role":"user","content":"Hello"}]}'

See Shared HTTP API.

Slurm parameters

--slurm account=<ACC>          # billing account
--slurm qos=<QOS>              # quality of service, e.g. accelerated
--slurm partition=<PART>       # partition, e.g. acc
--slurm time=<HH:MM:SS>        # wall-clock limit
--slurm cpus-per-task=<N>      # CPUs per task, e.g. 80

Also accepted: nodes, gres, mem, constraint. Anything else is rejected.

Exit codes

Code Meaning
0 Success
1 Batch ran, some requests failed
2 Invalid arguments or Slurm parameters
3 Model rejected by the allow list
4 Container image or model directory not accessible
5 Slurm error, or job did not reach RUNNING
7 Filesystem or runtime error

Full event list: Errors and exit codes.

Two things to remember

Add --dry-run to any start command to validate without spending GPU time, and always set --slurm time= with enough margin:

time = model load time + (per-request time × number of requests) + 25%