AI Tools

Logs and monitoring

Where llm-inference writes its logs, how to check job state, and what to look at when something is slow.

Draft

Draft page

This page is an outline. It describes what will be covered and is not yet complete technical documentation.

There is no metrics stack here — this is a CLI submitting Slurm jobs. What you get instead is a set of files in --outdir and the Slurm job state.

Where the logs are

Every mode writes into --outdir:

File Contents Look here when
out.txt Slurm standard output, including vLLM startup The model seems slow to load
err.txt Slurm standard error The job died
slurm-<jobid>.log vLLM logs captured inside the job You need engine-level detail
*.sh The generated Slurm script You want to know what was actually submitted
*.json Sidecar: job_id, port, model, I/O paths You need the port or job ID again
Follow a running job
tail -f /scratch/$USER/run-001/out.txt

Checking job state

llm-inference endpoint status --job 12345
llm-inference api status --job 12345

Both report the Slurm state and the assigned nodes:

State Meaning
PENDING Queued, waiting for GPUs. Cluster load, not a fault.
RUNNING Executing. Only now can you send requests.
COMPLETED Finished successfully.
FAILED Exited with an error — read err.txt.
CANCELLED Stopped by you, or by Slurm at the time= limit.

Standard Slurm tools work too, and are useful for seeing everything you have queued:

squeue -u $USER
sacct -j 12345 --format=JobID,State,Elapsed,ExitCode

Per-request timing

The output .jsonl records latency_ms for every request. That is your per-request measurement:

Find the slow requests
# Five slowest
jq -s 'sort_by(-.latency_ms) | .[:5] | .[] | "\(.id) \(.latency_ms)ms"' results.jsonl

# Total and mean
jq -s 'length as $n | (map(.latency_ms) | add) as $t
       | "\($n) requests, \($t)ms total, \($t/$n)ms mean"' results.jsonl

# How many failed
grep -c '"status":"error"' results.jsonl

What to check when things are slow

Separate loading from generating

A long total runtime is usually model loading, not generation. out.txt shows when vLLM finished loading; compare that against the latency_ms values in your output file. If loading dominates, Mode 1 amortises it across several batches instead of paying it per job.

Beyond that:

  • High latency_ms per request — check max_tokens; generation time scales with output length. See Key metrics.
  • Time spent PENDING — cluster queue, outside the tool’s control. Smaller --gpus, cpus-per-task or time= values usually schedule sooner.
  • Throughput lower than expected — the underlying engine batches requests automatically; see Static, dynamic and continuous batching.

Recording a baseline

Tip

Keep the latency_ms figures from a known-good run. Without a baseline, “it feels slower today” is not measurable — and queue time varies enough that impressions are unreliable. The method is in LLM performance benchmarks.