LLM Inference module User Guide on AI Tools

Overview

Mon, 01 Jan 0001 00:00:00 +0000

llm-inference is a command-line tool that runs large language models on the MareNostrum 5 HPC system. It sits between you and the two things that normally make this awkward — Slurm and vLLM — and handles both.

What the tool does for you

Reserves GPUs and nodes in Slurm with the right parameters.
Starts the vLLM engine inside a Singularity container.
Waits until the model is ready — it polls /health for you.
Returns your results, or gives you an HTTP URL you can use with curl.

The goal is simple: you say which model you want, how many GPUs you need, and how you want to work. The tool does the rest.

Installation

Mon, 01 Jan 0001 00:00:00 +0000

llm-inference is already installed on MareNostrum 5. You do not build or install anything — you load it as a module.

Requirements

A MareNostrum 5 account with access to the GPU partition (acc).
A valid Slurm billing account.

Load the module

Load llm-inference

module load EB/apps
module use /apps/ACC/LLM-INFERENCE/modulefiles
module load llm-inference/0.0.1-GCCcore-13.2.0

Verify

llm-inference --version
llm-inference --help

If llm-inference is not found, the module use line did not take effect — check the path and re-run the three commands in order.

Quickstart

Mon, 01 Jan 0001 00:00:00 +0000

From zero to a first result. Everything here runs on a MareNostrum 5 login node.

What you need

A MareNostrum 5 account with access to the GPU partition (acc).
A valid Slurm billing account — this is the account parameter below.

Step 1 — Load the module

The recommended way to use the tool on MareNostrum 5 is as a module:

Load the module

module load EB/apps
module use /apps/ACC/LLM-INFERENCE/modulefiles
module load llm-inference/0.0.1-GCCcore-13.2.0

# Check that it is available
llm-inference --version

Avoid typing this every session

Choosing a mode

Mon, 01 Jan 0001 00:00:00 +0000

The tool gives you three ways to work. This page helps you pick one in under a minute.

Decision table

If you need to…	Use	Guide
Process a single input file in one go and get a results file.	`bash inference`	One-shot inference
Keep an endpoint alive and send several batches to it.	`endpoint start` + `batch run`	Manual endpoint
Talk to the model from `curl`, the OpenAI SDK or your own code.	`api start`	Shared HTTP API
Just experiment, or validate your parameters.	any mode with `--dry-run`	Concepts

In one sentence each

Mode 1 — Manual endpoint. You decide when it starts and when it stops. Good for repeated batches against the same loaded model.
Mode 2 — One-shot inference. One command, one input file, one output file. Best for straightforward batch processing.
Mode 3 — Shared HTTP API. Exposes an OpenAI-style HTTP URL for curl or any client.

How they compare

	Manual endpoint	One-shot	HTTP API
Commands to run	2+ (`start`, `batch run`, `stop`)	1	2+ (`start`, `stop`)
Model loaded	once, reused	once per job	once, reused
Input	`.json` file per batch	`.json` file	live HTTP requests
Output	`.jsonl` file per batch	`.jsonl` file	HTTP responses
Ends when	you run `endpoint stop`	the batch finishes	you run `api stop`
You must stop it	yes	no	yes

Modes 1 and 3 hold GPUs until you stop them

Concepts

Mon, 01 Jan 0001 00:00:00 +0000

The vocabulary this guide uses. Where a term is a general inference concept rather than something specific to this tool, it links to the handbook.

Tool concepts

Term	Meaning
Model ID	The identifier passed to `--model`, e.g. `mistralai/Mistral-7B-Instruct-v0.3`. Must appear in `llm-inference model list` with `SUPPORTED=yes`.
Outdir	The directory given with `--outdir`. The tool writes the generated Slurm script, logs and sidecar files here.
Sidecar file	A small JSON written into the outdir recording `job_id`, `port` and `model`. It is how commands find a running endpoint.
Endpoint	A running vLLM server inside a Slurm job, reachable on an auto-assigned port.
Job ID	The Slurm job identifier. Every `status` and `stop` command takes it via `--job`.
Dry run	`--dry-run` validates parameters and writes the Slurm script without submitting.

Required Slurm parameters

All three modes need the same set. Pass them by repeating --slurm with key=value pairs:

Input and output format

Mon, 01 Jan 0001 00:00:00 +0000

Both batch run and bash inference use the same input and output format.

Input file (`.json`)

A JSON object with an array of requests. Each request has its own id plus the usual chat-API fields:

requests.json

{
  "requests": [
    {
      "id": "req-001",
      "model": "mistralai/Mistral-7B-Instruct-v0.3",
      "messages": [
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user",   "content": "Explain tensor parallelism in one paragraph."}
      ],
      "max_tokens": 256,
      "temperature": 0.7
    },
    {
      "id": "req-002",
      "model": "mistralai/Mistral-7B-Instruct-v0.3",
      "messages": [
        {"role": "user", "content": "Write a Python function that returns the n-th Fibonacci number."}
      ],
      "max_tokens": 384,
      "temperature": 0.0
    }
  ]
}

Fields

Field	Purpose
`id`	Your identifier for the request. Copied to the output so you can match them up.
`model`	The model ID. Should match the model the endpoint was started with.
`messages`	The conversation, as `role` / `content` pairs. Roles: `system`, `user`, `assistant`.
`max_tokens`	Upper bound on generated tokens for this request.
`temperature`	Sampling temperature. `0.0` is the most deterministic.

Tip

API reference

Mon, 01 Jan 0001 00:00:00 +0000

In HTTP API mode, vLLM serves an OpenAI-compatible HTTP surface. This page documents the endpoints you will use.

All paths are relative to the base URL printed by api start, which looks like http://nid001:45123. Host and port are assigned at run time and change on every start.

No authentication

There is no API key, no TLS and no proxy support. The endpoint is reachable by anyone inside the MN5 network who knows the host and port. Do not treat it as a shared service outside your own work.

Errors and exit codes

Mon, 01 Jan 0001 00:00:00 +0000

When something fails, the tool prints a structured JSON to stderr with an event field and a hint about what to do next. This page lists the events you will see most often.

How to read a failure

A failure on stderr

{"event": "model_rejected", "hint": "...", "...": "..."}

Two things tell you what happened: the event name (look it up below) and the process exit code (see the table at the end).

First two questions

Logs and monitoring

Mon, 01 Jan 0001 00:00:00 +0000

There is no metrics stack here — this is a CLI submitting Slurm jobs. What you get instead is a set of files in --outdir and the Slurm job state.

Where the logs are

Every mode writes into --outdir:

File	Contents	Look here when
`out.txt`	Slurm standard output, including vLLM startup	The model seems slow to load
`err.txt`	Slurm standard error	The job died
`slurm-<jobid>.log`	vLLM logs captured inside the job	You need engine-level detail
`*.sh`	The generated Slurm script	You want to know what was actually submitted
`*.json`	Sidecar: `job_id`, `port`, `model`, I/O paths	You need the port or job ID again

Follow a running job

tail -f /scratch/$USER/run-001/out.txt

Checking job state

llm-inference endpoint status --job 12345
llm-inference api status --job 12345

Both report the Slurm state and the assigned nodes:

Quick reference

Mon, 01 Jan 0001 00:00:00 +0000

A summary of the most common commands. Print this page if you want it next to you.

Check the tool

llm-inference --version
llm-inference --help
llm-inference model list

Load the module

module load EB/apps
module use /apps/ACC/LLM-INFERENCE/modulefiles
module load llm-inference/0.0.1-GCCcore-13.2.0

Mode 1 — Manual endpoint

# Start
llm-inference endpoint start --model <ID> --gpus <N> \
  --outdir <DIR> \
  --slurm account=<ACC> --slurm qos=<QOS> \
  --slurm partition=<PART> --slurm time=<HH:MM:SS> \
  --slurm cpus-per-task=<N>

# Status / stop
llm-inference endpoint status --job <JOB_ID>
llm-inference endpoint stop   --job <JOB_ID>

# Run a batch (repeatable while the endpoint is alive)
llm-inference batch run --in <IN.json> --out <OUT.jsonl> \
  --via-endpoint <JOB_ID> --outdir <DIR>

See Manual endpoint.

Security and shared-resource notes

Mon, 01 Jan 0001 00:00:00 +0000

This is a CLI on a shared HPC system, not a hosted service. The security considerations are correspondingly different from a public API — and mostly about the HTTP API mode and about being a good tenant of the cluster.

The HTTP API has no access control

No API keys, no TLS, no proxy support

HTTP API mode exposes a plain HTTP port on a compute node. It has no authentication. Anyone inside the MareNostrum 5 network who knows the host and port can send requests, and they will run on your Slurm allocation, billed to your account.

LLM Inference module User Guide on AI Tools

Overview

What the tool does for you

Installation

Requirements

Load the module

Verify

Quickstart

What you need

Step 1 — Load the module

Choosing a mode

Decision table

In one sentence each

How they compare

Concepts

Tool concepts

Required Slurm parameters

Input and output format

Input file (.json)

Fields

API reference

Errors and exit codes

How to read a failure

Logs and monitoring

Where the logs are

Checking job state

Quick reference

Check the tool

Load the module

Mode 1 — Manual endpoint

Security and shared-resource notes

The HTTP API has no access control

Input file (`.json`)