AI Tools

Input and output format

The .json request file and the .jsonl results file used by batch run and bash inference.

Draft

Draft page

This page is an outline. It describes what will be covered and is not yet complete technical documentation.

Both batch run and bash inference use the same input and output format.

Input file (.json)

A JSON object with an array of requests. Each request has its own id plus the usual chat-API fields:

requests.json
{
  "requests": [
    {
      "id": "req-001",
      "model": "mistralai/Mistral-7B-Instruct-v0.3",
      "messages": [
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user",   "content": "Explain tensor parallelism in one paragraph."}
      ],
      "max_tokens": 256,
      "temperature": 0.7
    },
    {
      "id": "req-002",
      "model": "mistralai/Mistral-7B-Instruct-v0.3",
      "messages": [
        {"role": "user", "content": "Write a Python function that returns the n-th Fibonacci number."}
      ],
      "max_tokens": 384,
      "temperature": 0.0
    }
  ]
}

Fields

Field Purpose
id Your identifier for the request. Copied to the output so you can match them up.
model The model ID. Should match the model the endpoint was started with.
messages The conversation, as role / content pairs. Roles: system, user, assistant.
max_tokens Upper bound on generated tokens for this request.
temperature Sampling temperature. 0.0 is the most deterministic.

Tip

Give every request a unique, meaningful id. It is the only thing tying an output line back to its input, and the output is written in input order but read most easily by id.

What each sampling parameter actually does is covered in LLM inference parameters.

Output file (.jsonl)

One JSON line per request, in the same order as the input:

results.jsonl
{"id":"req-001","latency_ms":412.3,"status":"ok","http_code":200,"response":{ }}
{"id":"req-002","latency_ms":17.0, "status":"error","http_code":0, "error":"..."}

Fields

Field Meaning
id The id from the input request.
latency_ms How long the request took, in milliseconds.
status ok if the request returned 2xx, otherwise error.
http_code The HTTP status returned by vLLM. 0 means the request never completed.
response Present when status is ok — the model’s answer.
error Present when status is error — the reason.

Checking results

Because the output is JSON Lines, ordinary tools work well:

Count failures and inspect them
# How many requests failed?
grep -c '"status":"error"' results.jsonl

# Show the id and error of each failure
jq -r 'select(.status=="error") | "\(.id): \(.error)"' results.jsonl

# Slowest requests first
jq -s 'sort_by(-.latency_ms) | .[:5] | .[] | "\(.id) \(.latency_ms)ms"' results.jsonl

Note

A batch where some requests failed exits with code 1, not 0. The output file is still written — inspect the status fields rather than assuming the whole run was lost. See Errors and exit codes.

Filesystem requirements