AI Tools

Mode 3 — Shared HTTP API

Expose a long-lived OpenAI-style HTTP URL that curl, the OpenAI SDK or your own code can call.

Draft

Draft page

This page is an outline. It describes what will be covered and is not yet complete technical documentation.

This mode starts a long-lived vLLM endpoint that other tools can talk to over HTTP: curl, the OpenAI SDK, or your own applications — as long as they run inside the MareNostrum 5 network.

When to use it

  • You want to share the endpoint between several tools.
  • The requests are interactive and you do not know them in advance.
  • You want the URL to stay alive until you stop it.

Step 1 — Start the API

api start
llm-inference api start \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --outdir /scratch/$USER/api-001 \
  --gpus 1 \
  --slurm account=my_project \
  --slurm qos=accelerated \
  --slurm partition=acc \
  --slurm time=04:00:00 \
  --slurm cpus-per-task=80

By default the command waits until the job is RUNNING and vLLM answers /health. On success it prints everything you need:

Output
=== vLLM API endpoint ready ===
Base URL:       http://nid001:45123
Chat URL:       http://nid001:45123/v1/chat/completions
Model:          mistralai/Mistral-7B-Instruct-v0.3
Port:           45123
Slurm job ID:   12345
Logs (stdout):  /scratch/$USER/api-001/out.txt
Logs (stderr):  /scratch/$USER/api-001/err.txt

The same information is saved to api_endpoint.json inside --outdir.

Note

The host and port are assigned at run time — the host is whichever compute node Slurm allocated, and the port is auto-assigned. They change on every start, so read them from the output or from api_endpoint.json rather than hard-coding them.

Step 2 — Send requests with curl

Replace <host>:<port> with the values printed by api start:

Chat completion
curl -sS http://<host>:<port>/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "mistralai/Mistral-7B-Instruct-v0.3",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user",   "content": "Explain tensor parallelism in one sentence."}
    ],
    "max_tokens": 256,
    "temperature": 0.2
  }'

Other useful endpoints:

GET http://<host>:<port>/v1/models

Lists the loaded models.

GET http://<host>:<port>/health

Checks that vLLM is responding.

The full surface is documented in API reference.

Step 3 — Check or stop the API

# Check the status
llm-inference api status --job 12345

# Stop it (frees the GPUs)
llm-inference api stop --job 12345

Using it from the OpenAI SDK

Because the surface is OpenAI-compatible, existing clients work by pointing base_url at the printed URL. There is no API key, but most clients require the field to be non-empty:

from openai import OpenAI

# Host and port come from `api start`; there is no authentication.
client = OpenAI(
    base_url="http://nid001:45123/v1",
    api_key="not-used",
)

response = client.chat.completions.create(
    model="mistralai/Mistral-7B-Instruct-v0.3",
    messages=[{"role": "user", "content": "Explain prefill in one sentence."}],
    max_tokens=256,
    temperature=0.2,
)

print(response.choices[0].message.content)