Mode 1 — Manual endpoint | LLM Inference module User Guide

This mode splits the work in two. First you start the endpoint — it stays alive in Slurm. Then you send as many batches as you like. When you are done, you stop it.

Use it when you have several batches to run against the same model: the model is loaded once and reused, instead of paying the load cost on every job.

Prerequisites

The module loaded — see Quickstart.
A model ID with SUPPORTED=yes in llm-inference model list.
An input file on a shared filesystem (/gpfs or $SCRATCH).

Step 1 — Start the endpoint

endpoint start

llm-inference endpoint start \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --gpus 1 \
  --outdir /scratch/$USER/run-001 \
  --slurm account=my_project \
  --slurm qos=accelerated \
  --slurm partition=acc \
  --slurm time=02:00:00 \
  --slurm cpus-per-task=80

The command prints JSON with the Slurm job_id, the auto-assigned port and the model. It also writes these files into --outdir:

File	Contents
`slurm_script.sh`	The generated Slurm script.
`endpoint.json`	Sidecar with `job_id`, `port` and `model`. Needed in step 3.
`out.txt`	Slurm standard output.
`err.txt`	Slurm standard error.

Step 2 — Wait until it is RUNNING

Do not send requests before the job is in state RUNNING:

endpoint status

llm-inference endpoint status --job 12345

The response shows the Slurm state (PENDING, RUNNING, COMPLETED, FAILED, CANCELLED) and the assigned nodes.

Note

PENDING just means the job is queued waiting for GPUs. How long depends on cluster load, not on the tool. Sending a batch too early fails with endpoint_not_running.

Step 3 — Send a batch

Once the endpoint is RUNNING, run your input file against it:

batch run

llm-inference batch run \
  --in ./input.json \
  --out ./results.jsonl \
  --via-endpoint 12345 \
  --outdir /scratch/$USER/run-001

Pass the same --outdir

batch run reads endpoint.json from --outdir to find the port. It must be the same directory you used in endpoint start, or the command fails with endpoint_sidecar_not_found.

You can repeat this step as many times as you want while the endpoint is alive — different input files, different output files, same loaded model.

Step 4 — Stop the endpoint

The GPUs stay reserved until you cancel the job:

endpoint stop

llm-inference endpoint stop --job 12345

Always stop what you start

An endpoint you forget about holds its GPUs until --slurm time= expires, billed to your account the whole time. Stop it as soon as you are finished.

Full example

Start to finish

# 1) Start
llm-inference endpoint start \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --gpus 1 --outdir /scratch/$USER/run-001 \
  --slurm account=my_project --slurm qos=accelerated \
  --slurm partition=acc --slurm time=02:00:00 \
  --slurm cpus-per-task=80
# -> prints job_id = 12345

# 2) Wait for RUNNING
llm-inference endpoint status --job 12345

# 3) Send requests (repeat as needed)
llm-inference batch run --in input.json --out out.jsonl \
  --via-endpoint 12345 --outdir /scratch/$USER/run-001

# 4) Stop
llm-inference endpoint stop --job 12345

Validate first

Add --dry-run to endpoint start to check parameters and inspect the generated script without submitting anything:

llm-inference endpoint start ... --dry-run