AI Tools

Concepts

Slurm parameters, outdir, sidecar files, dry runs and the terms used throughout this guide.

Draft

Draft page

This page is an outline. It describes what will be covered and is not yet complete technical documentation.

The vocabulary this guide uses. Where a term is a general inference concept rather than something specific to this tool, it links to the handbook.

Tool concepts

Term Meaning
Model ID The identifier passed to --model, e.g. mistralai/Mistral-7B-Instruct-v0.3. Must appear in llm-inference model list with SUPPORTED=yes.
Outdir The directory given with --outdir. The tool writes the generated Slurm script, logs and sidecar files here.
Sidecar file A small JSON written into the outdir recording job_id, port and model. It is how commands find a running endpoint.
Endpoint A running vLLM server inside a Slurm job, reachable on an auto-assigned port.
Job ID The Slurm job identifier. Every status and stop command takes it via --job.
Dry run --dry-run validates parameters and writes the Slurm script without submitting.

Required Slurm parameters

All three modes need the same set. Pass them by repeating --slurm with key=value pairs:

Slurm parameters
--slurm account=my_project \
--slurm qos=accelerated \
--slurm partition=acc \
--slurm time=02:00:00 \
--slurm cpus-per-task=80
Parameter What it is Example
account Slurm billing account my_project
qos Quality of service accelerated
partition Cluster partition acc
time Wall-clock time limit (HH:MM:SS) 02:00:00
cpus-per-task CPUs per task 80

Accepted keys

Only these --slurm keys are allowed. Anything else fails with invalid_slurm_params:

account   qos   partition   time   cpus-per-task   nodes   gres   mem   constraint

Sizing time=

The outdir and its files

Every mode writes into --outdir. Exact filenames depend on the mode, but the pattern is consistent:

File Contents
*.sh The generated Slurm script — readable, and runnable by hand for debugging.
*.json The sidecar: job_id, port, model, and I/O paths where relevant.
out.txt Slurm standard output.
err.txt Slurm standard error.
slurm-<jobid>.log vLLM logs captured inside the job.

Filesystem requirements

GPU and node layout

--gpus and --nodes must satisfy all of:

  • --gpus ≥ --nodes
  • --gpus divisible by --nodes
  • at most 4 GPUs per node
  • inside the model’s allowed range from llm-inference model list

Violations are reported as invalid_gpu_node_layout or model_rejected.

Job states

endpoint status and api status report the Slurm state:

State Meaning
PENDING Queued, waiting for resources.
RUNNING Allocated and executing. Only now can you send requests.
COMPLETED Finished successfully.
FAILED Exited with an error — check err.txt.
CANCELLED Stopped by you, or by Slurm at the time= limit.

Inference concepts used here

Short definitions for reference; the handbook explains them properly.

  • Prefill — processing the input prompt. Compute-bound. See What is LLM inference?.
  • Decode — generating output tokens one at a time. Memory-bandwidth-bound.
  • KV cache — retained attention state; the dominant consumer of GPU memory. See Calculating GPU memory.
  • Tensor parallelism — splitting one model across several GPUs, which is what --gpus > 1 sets up. See Parallelism.
  • vLLM — the inference engine the tool runs for you. Its batching behaviour is described in Static, dynamic and continuous batching.