AI Tools

Mode 2 — One-shot inference

Process one input file in a single Slurm job with bash inference, then stop.

Draft

Draft page

This page is an outline. It describes what will be covered and is not yet complete technical documentation.

This mode does everything inside a single Slurm job: it starts vLLM, waits until it is ready, sends every request from your input file, writes the output, and shuts vLLM down. You run one command.

When to use it

  • You already have an input file ready.
  • You want to process everything in one go and forget about it.
  • You do not need the endpoint to stay alive afterwards.

Command

bash inference
llm-inference bash inference \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --in  /scratch/$USER/requests.json \
  --out /scratch/$USER/results.jsonl \
  --outdir /scratch/$USER/run-bash \
  --gpus 1 \
  --slurm account=my_project \
  --slurm qos=accelerated \
  --slurm partition=acc \
  --slurm time=01:00:00 \
  --slurm cpus-per-task=80

By default the command blocks until the Slurm job ends. To get the shell back immediately and check on the job later, add --no-wait.

What happens inside the job

  1. Slurm reserves the requested GPUs.
  2. The Singularity module is loaded and vLLM is started.
  3. The job polls /health until it answers, or the timeout expires.
  4. Every request in the input file is sent, and the responses are written to the output file.
  5. vLLM is shut down and the job exits with code 0 if everything went well.

Note

An exit code of 1 means the batch ran but some requests failed. The output file still exists — check the status field on each line. See Errors and exit codes.

Files written to --outdir

File Contents
bash_inference.sh Generated Slurm script.
bash_inference.json Sidecar: job_id, model, port, input and output paths.
out.txt / err.txt Slurm standard output and error.
slurm-<jobid>.log vLLM logs captured inside the job.

Sizing the time limit

Try it without spending GPUs

Before using real GPUs, check the script and parameters:

Dry run
llm-inference bash inference \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --in  ./requests.json --out /tmp/run/out.jsonl \
  --outdir /tmp/run --gpus 1 \
  --slurm account=my_project --slurm qos=accelerated \
  --slurm partition=acc --slurm time=01:00:00 \
  --slurm cpus-per-task=80 \
  --dry-run --no-wait

# Inspect the generated script
cat /tmp/run/bash_inference.sh

Tip

Reading the generated script is also the fastest way to understand what the tool does on your behalf — and to debug a slurm_submit_failed, since you can run it by hand with sbatch.