AI Tools

Quickstart

Load the module on MareNostrum 5, list the available models, and run your first inference job.

Draft

Draft page

This page is an outline. It describes what will be covered and is not yet complete technical documentation.

From zero to a first result. Everything here runs on a MareNostrum 5 login node.

What you need

  • A MareNostrum 5 account with access to the GPU partition (acc).
  • A valid Slurm billing account — this is the account parameter below.

Step 1 — Load the module

The recommended way to use the tool on MareNostrum 5 is as a module:

Load the module
module load EB/apps
module use /apps/ACC/LLM-INFERENCE/modulefiles
module load llm-inference/0.0.1-GCCcore-13.2.0

# Check that it is available
llm-inference --version

Avoid typing this every session

Add the first two lines to your ~/.bashrc:

echo 'module load EB/apps' >> ~/.bashrc
echo 'module use /apps/ACC/LLM-INFERENCE/modulefiles' >> ~/.bashrc

Step 2 — See the available models

Before launching anything, confirm your model is in the allow list and that the SUPPORTED column reads yes:

List models
llm-inference model list

The output is a table with one row per model: ID, family, data type, allowed GPU range, multi-node support, and whether the deployed vLLM can load it right now.

Step 3 — Prepare an input file

Create a small requests.json. One entry is enough for a first run:

requests.json
{
  "requests": [
    {
      "id": "req-001",
      "model": "mistralai/Mistral-7B-Instruct-v0.3",
      "messages": [
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user",   "content": "Explain tensor parallelism in one paragraph."}
      ],
      "max_tokens": 256,
      "temperature": 0.7
    }
  ]
}

The full schema is described in Input and output format.

Step 4 — Validate without spending GPU time

Check your parameters and inspect the generated Slurm script before submitting anything:

Dry run
llm-inference bash inference \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --in  ./requests.json \
  --out /tmp/run/out.jsonl \
  --outdir /tmp/run \
  --gpus 1 \
  --slurm account=my_project \
  --slurm qos=accelerated \
  --slurm partition=acc \
  --slurm time=01:00:00 \
  --slurm cpus-per-task=80 \
  --dry-run --no-wait

# Inspect what would have been submitted
cat /tmp/run/bash_inference.sh

Step 5 — Run it for real

Drop --dry-run --no-wait and point the paths at a shared filesystem:

One-shot inference
llm-inference bash inference \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --in  /scratch/$USER/requests.json \
  --out /scratch/$USER/results.jsonl \
  --outdir /scratch/$USER/run-001 \
  --gpus 1 \
  --slurm account=my_project \
  --slurm qos=accelerated \
  --slurm partition=acc \
  --slurm time=01:00:00 \
  --slurm cpus-per-task=80

The command blocks until the Slurm job finishes. Inside the job, the tool reserves the GPUs, starts vLLM, waits for /health, sends every request, writes the output and shuts vLLM down.

Step 6 — Read the results

results.jsonl has one JSON line per request, in input order:

results.jsonl
{"id":"req-001","latency_ms":412.3,"status":"ok","http_code":200,"response":{ }}

A status of ok means the request returned 2xx and the answer is in response. A status of error puts the reason in an error field.

Next steps