Security and shared-resource notes

This is a CLI on a shared HPC system, not a hosted service. The security considerations are correspondingly different from a public API — and mostly about the HTTP API mode and about being a good tenant of the cluster.

The HTTP API has no access control

No API keys, no TLS, no proxy support

HTTP API mode exposes a plain HTTP port on a compute node. It has no authentication. Anyone inside the MareNostrum 5 network who knows the host and port can send requests, and they will run on your Slurm allocation, billed to your account.

The tool assumes you are already inside a trusted network. It provides no way to change that.

Practical consequences:

Do not treat the URL as a service for other people or long-lived applications.
Do not publish the host and port in shared documents, tickets or chat channels.
Stop the job when you finish — see below.
If you need authenticated, multi-tenant access, that belongs in front of this tool, not in it.

Free your GPUs

Modes 1 and 3 hold GPUs until stopped

endpoint start and api start keep the Slurm job alive until you cancel it or time= expires. An endpoint you forget about occupies GPUs other users could be using, and consumes your allocation the whole time.

llm-inference endpoint stop --job <JOB_ID>
llm-inference api stop      --job <JOB_ID>

Check for stragglers with squeue -u $USER.

Prompt and output data

Your prompts and the model’s responses are written to files on the shared filesystem:

The input .json and output .jsonl live wherever you put them.
out.txt, err.txt and slurm-<jobid>.log in --outdir may contain request detail.

Warning

Check the permissions on --outdir and your I/O files if the content is sensitive. Default umask on a shared filesystem is often more permissive than people assume — files under /gpfs or $SCRATCH may be readable by others in your group.

Do not put credentials, personal data or confidential material into prompts unless you have confirmed that the filesystem and its retention are appropriate for it.

Model output is untrusted input

Important

If you feed the responses into anything that acts on them — a script, a shell command, a database write — treat them as untrusted. A model’s output can be influenced by anything in its context, including the documents you asked it to process.

Never interpolate model output into shell commands, SQL or file paths without validation. The general reasoning is in Function calling.

Cluster etiquette

Use --dry-run to validate parameters instead of discovering mistakes with real GPUs.
Request only the GPUs the model needs — llm-inference model list shows the allowed range.
Set a realistic time=. Padding it wildly makes the job harder to schedule for everyone.
Test with a small input file before submitting thousands of requests.

Checklist

Every endpoint start and api start has a matching stop.
No forgotten jobs in squeue -u $USER.
--outdir permissions appropriate for the prompt content.
Host and port from api start not shared beyond people who should have them.
Model output validated before anything acts on it.