Modes on AI Tools

Mode 1 — Manual endpoint

Mon, 01 Jan 0001 00:00:00 +0000

This mode splits the work in two. First you start the endpoint — it stays alive in Slurm. Then you send as many batches as you like. When you are done, you stop it.

Use it when you have several batches to run against the same model: the model is loaded once and reused, instead of paying the load cost on every job.

Prerequisites

The module loaded — see Quickstart.
A model ID with SUPPORTED=yes in llm-inference model list.
An input file on a shared filesystem (/gpfs or $SCRATCH).

Step 1 — Start the endpoint

endpoint start

llm-inference endpoint start \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --gpus 1 \
  --outdir /scratch/$USER/run-001 \
  --slurm account=my_project \
  --slurm qos=accelerated \
  --slurm partition=acc \
  --slurm time=02:00:00 \
  --slurm cpus-per-task=80

The command prints JSON with the Slurm job_id, the auto-assigned port and the model. It also writes these files into --outdir:

Mode 2 — One-shot inference

Mon, 01 Jan 0001 00:00:00 +0000

This mode does everything inside a single Slurm job: it starts vLLM, waits until it is ready, sends every request from your input file, writes the output, and shuts vLLM down. You run one command.

When to use it

You already have an input file ready.
You want to process everything in one go and forget about it.
You do not need the endpoint to stay alive afterwards.

Command

bash inference

llm-inference bash inference \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --in  /scratch/$USER/requests.json \
  --out /scratch/$USER/results.jsonl \
  --outdir /scratch/$USER/run-bash \
  --gpus 1 \
  --slurm account=my_project \
  --slurm qos=accelerated \
  --slurm partition=acc \
  --slurm time=01:00:00 \
  --slurm cpus-per-task=80

By default the command blocks until the Slurm job ends. To get the shell back immediately and check on the job later, add --no-wait.

Mode 3 — Shared HTTP API

Mon, 01 Jan 0001 00:00:00 +0000

This mode starts a long-lived vLLM endpoint that other tools can talk to over HTTP: curl, the OpenAI SDK, or your own applications — as long as they run inside the MareNostrum 5 network.

When to use it

You want to share the endpoint between several tools.
The requests are interactive and you do not know them in advance.
You want the URL to stay alive until you stop it.

No authentication, TLS or proxy support