Overview

llm-inference is a command-line tool that runs large language models on the MareNostrum 5 HPC system. It sits between you and the two things that normally make this awkward — Slurm and vLLM — and handles both.

What the tool does for you

Reserves GPUs and nodes in Slurm with the right parameters.
Starts the vLLM engine inside a Singularity container.
Waits until the model is ready — it polls /health for you.
Returns your results, or gives you an HTTP URL you can use with curl.

The goal is simple: you say which model you want, how many GPUs you need, and how you want to work. The tool does the rest.

What you get in each mode

Three modes, three shapes of output:

Manual endpoint

An endpoint that stays alive in Slurm. Send as many batches as you like, then stop it yourself.

One-shot inference

One command, one input file, one output file. Everything happens inside a single Slurm job.

Shared HTTP API

An OpenAI-style URL that other tools, scripts or curl can talk to.

What the tool does not do

Being explicit about the boundaries saves time:

No API keys, TLS or proxies. The HTTP API mode assumes you are already inside the MareNostrum 5 network. There is no authentication layer.
No model training or fine-tuning. See LLM fine-tuning for that topic.
No model conversion. Models must already be in the allow list; check with llm-inference model list.
No queue management beyond Slurm. The tool submits jobs; Slurm decides when they run, and cancels them when time= runs out.

Where the concepts are explained

This guide stays operational. Where a page relies on an inference concept, it links to the LLM Inference Handbook rather than re-explaining it — for example what the KV cache is or how tensor parallelism splits a model across GPUs.