API reference

In HTTP API mode, vLLM serves an OpenAI-compatible HTTP surface. This page documents the endpoints you will use.

All paths are relative to the base URL printed by api start, which looks like http://nid001:45123. Host and port are assigned at run time and change on every start.

No authentication

There is no API key, no TLS and no proxy support. The endpoint is reachable by anyone inside the MN5 network who knows the host and port. Do not treat it as a shared service outside your own work.

Health check

GET http://<host>:<port>/health

Returns success once vLLM has finished loading the model and is ready to serve.

The tool polls this for you during api start and bash inference. You can also use it to confirm an endpoint is alive:

curl -sS -o /dev/null -w '%{http_code}\n' http://<host>:<port>/health

List models

GET http://<host>:<port>/v1/models

Lists the models currently loaded by this vLLM instance.

curl -sS http://<host>:<port>/v1/models

Note

This is not the same as llm-inference model list. That command lists the models the tool allows you to launch; this endpoint lists what the running instance has actually loaded — a single model, in normal use.

Chat completions

The main endpoint for generation.

POST http://<host>:<port>/v1/chat/completions

Generates a response for a conversation expressed as a list of messages.

Request parameters

Parameter	Type	Description
`model`	string	Required. The model ID the endpoint was started with.
`messages`	array	Required. Conversation messages, each with `role` and `content`.
`max_tokens`	integer	Upper bound on generated tokens.
`temperature`	number	Sampling temperature. Lower is more deterministic.
`top_p`	number	Nucleus sampling threshold.
`stream`	boolean	Deliver the response incrementally as server-sent events.
`stop`	string or array	Sequences that halt generation.

Each parameter’s effect is explained in LLM inference parameters.

Message roles

Role	Purpose
`system`	Instructions that frame the whole conversation.
`user`	Input from the end user.
`assistant`	Previous model responses, supplying conversational history.

Example

curl -sS http://<host>:<port>/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "mistralai/Mistral-7B-Instruct-v0.3",
    "messages": [
      {"role": "system", "content": "You are a concise technical assistant."},
      {"role": "user",   "content": "What does the KV cache store?"}
    ],
    "max_tokens": 256,
    "temperature": 0.2
  }'

from openai import OpenAI

client = OpenAI(
    base_url="http://<host>:<port>/v1",
    api_key="not-used",          # no authentication, but the field must be set
)

response = client.chat.completions.create(
    model="mistralai/Mistral-7B-Instruct-v0.3",
    messages=[
        {"role": "system", "content": "You are a concise technical assistant."},
        {"role": "user", "content": "What does the KV cache store?"},
    ],
    max_tokens=256,
    temperature=0.2,
)

print(response.choices[0].message.content)
print(response.usage)

stream = client.chat.completions.create(
    model="mistralai/Mistral-7B-Instruct-v0.3",
    messages=[{"role": "user", "content": "Explain prefill in one paragraph."}],
    max_tokens=256,
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

Response shape

A non-streaming response contains a choices array — each entry holding a message and a finish_reason — plus a usage object with prompt, completion and total token counts.

Always check finish_reason

A finish_reason of length means generation stopped because it hit max_tokens, so the response is truncated rather than complete. Treating a truncated response as finished is a common source of subtle bugs.

Text completions

POST http://<host>:<port>/v1/completions

Older prompt-in, text-out endpoint.

Note

This endpoint predates the chat format and takes a single prompt string. Prefer /v1/chat/completions — instruction-tuned models are trained against the chat message structure.

Compatibility caveats

Compatibility is not identity

vLLM implements the common core of the OpenAI API, not every field of the upstream specification. Parameters may be accepted and ignored, or rejected outright. Verify behaviour for any parameter your application depends on. The general pattern is discussed in OpenAI-compatible API.