AI Tools

LLM Inference module User Guide

A practical guide to setting up the LLM Inference tool, deploying models, and operating inference workloads.

llm-inference is a command-line tool for running large language models on MareNostrum 5. You say which model you want, how many GPUs you need and how you want to work; the tool reserves the nodes in Slurm, starts vLLM inside a Singularity container, waits until the model answers, and either returns your results or gives you an HTTP URL.

You do not need to write Slurm scripts by hand.

Version 0.0.1 — draft

This guide documents version 0.0.1 of the tool. Both the tool and this documentation are early; commands and output formats may still change.

Start here

Quickstart

Load the module, list the models, and run your first job.

Choosing a mode

A one-minute decision table for the three ways of working.

Concepts

Slurm parameters, outdir, sidecar files and the terms used in this guide.

The three modes

The tool gives you three ways to work. Each fits a different use case.

Mode What it is for Main command
1. Manual endpoint Start an endpoint that stays alive, and send several batches to it. endpoint start + batch run
2. One-shot inference Process one input file in a single Slurm job, then stop. bash inference
3. Shared HTTP API Expose a URL for curl, the OpenAI SDK or your own code. api start

Tip

Not sure which one you need? Choosing a mode has a decision table that takes less than a minute to read.

Reference

Page Covers
Input and output format The .json request file and the .jsonl results file.
API reference The OpenAI-compatible HTTP endpoints vLLM serves.
Errors and exit codes Every event the tool reports, what it means and how to fix it.
Quick reference Every command on one page.

Before you start

You need:

  • A MareNostrum 5 account with access to the GPU partition (acc).
  • A valid Slurm billing account — the account parameter.
  • The ID of the model you want to run. List them with llm-inference model list.

Always validate first

Every start command accepts --dry-run. It checks your parameters and writes the generated Slurm script to disk without submitting anything — the safest way to verify your setup before spending GPU time.

Modes

The three ways of working with llm-inference: manual endpoint, one-shot inference and shared HTTP API.

  • Manual endpoint Start an endpoint that stays alive in Slurm, send several batches to it, then stop it yourself.
  • One-shot inference Process one input file in a single Slurm job with bash inference, then stop.
  • Shared HTTP API Expose a long-lived OpenAI-style HTTP URL that curl, the OpenAI SDK or your own code can call.
  • Overview What llm-inference does on MareNostrum 5, and what it handles for you.
  • Installation Load the llm-inference module on MareNostrum 5 and verify it works.
  • Quickstart Load the module on MareNostrum 5, list the available models, and run your first inference job.
  • Choosing a mode A decision table for the three ways of working with llm-inference.
  • Concepts Slurm parameters, outdir, sidecar files, dry runs and the terms used throughout this guide.
  • Input and output format The .json request file and the .jsonl results file used by batch run and bash inference.
  • API reference The OpenAI-compatible HTTP endpoints served by vLLM in HTTP API mode.
  • Errors and exit codes Every event the tool reports, what it means, and how to fix it — plus the exit code table.
  • Logs and monitoring Where llm-inference writes its logs, how to check job state, and what to look at when something is slow.
  • Quick reference Every llm-inference command on one page.
  • Security notes What the tool does not protect, and how to behave on a shared HPC system.