AI Tools

Choosing the right model

Mon, 01 Jan 0001 00:00:00 +0000

The largest model that fits is rarely the right choice. Model selection sets the floor for your serving cost and the ceiling for your output quality at the same time, and the two pull in opposite directions.

This page frames the decision as an evaluation problem tied to your own task, not a leaderboard comparison.

What you will learn

Which model properties actually affect serving cost.
How to evaluate candidates against your own task rather than a public benchmark.
How licensing and provenance constrain deployment options.
When a smaller specialised model beats a larger general one.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Conversational Support Agent

Mon, 01 Jan 0001 00:00:00 +0000

Overview

A tool-using support agent that answers questions against your own documentation and escalates when it cannot.

This page shows the structure a blueprint detail page uses. The problem statement and component list below are real design context; the deployment and usage sections are placeholders.

Problem addressed

Support teams answer the same documented questions repeatedly, while the genuinely novel cases — the ones that need a human — wait behind them in the same queue.

Introduction

Mon, 01 Jan 0001 00:00:00 +0000

Serving a large language model in production is a different discipline from training one. The model is fixed, the traffic is not, and the work shifts to latency budgets, memory limits, batching behaviour and the operational question of keeping expensive accelerators busy without breaking service-level objectives.

This handbook collects what the AI Tools team has learned doing that work. It is organised as a path: foundations first, then planning a deployment, preparing a model, interacting with it, optimizing throughput and latency, and finally the infrastructure and operations that keep a service running.

Kernel optimization for LLM inference

Mon, 01 Jan 0001 00:00:00 +0000

Kernel optimization is the lowest layer at which inference performance can be improved, and the one with the worst effort-to-reward ratio if applied before the layers above it have been addressed.

This page frames when to descend to this level, and what the work actually consists of.

What you will learn

What a kernel is and why kernel efficiency bounds performance.
How to establish that kernels, rather than scheduling or batching, are the bottleneck.
Which inference operations are most often worth optimizing.
When to write a kernel versus adopting an existing one.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

LLM fine-tuning

Mon, 01 Jan 0001 00:00:00 +0000

Fine-tuning adapts a pretrained model to a narrower task or style. It is frequently reached for when prompt engineering or retrieval would have solved the problem more cheaply, and it carries serving consequences that are easy to overlook.

This page covers when fine-tuning earns its cost and what it changes downstream.

What you will learn

When fine-tuning is the appropriate tool and when it is not.
How full fine-tuning and parameter-efficient methods differ operationally.
What serving a fine-tuned or adapter-based model requires.
How to evaluate whether a fine-tune actually improved your task.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

LLM performance benchmarks

Mon, 01 Jan 0001 00:00:00 +0000

Most published inference numbers are not reproducible in your environment, because they encode a traffic pattern, a hardware configuration and a measurement methodology that are not yours.

This page covers how to benchmark your own deployment in a way that supports capacity and configuration decisions.

What you will learn

Which workload parameters must be specified for a benchmark to mean anything.
How to model a realistic request arrival pattern.
Why steady-state and cold-start measurements differ.
How to report results so others can reproduce them.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Mode 1 — Manual endpoint

Mon, 01 Jan 0001 00:00:00 +0000

This mode splits the work in two. First you start the endpoint — it stays alive in Slurm. Then you send as many batches as you like. When you are done, you stop it.

Use it when you have several batches to run against the same model: the model is loaded once and reused, instead of paying the load cost on every job.

Prerequisites

The module loaded — see Quickstart.
A model ID with SUPPORTED=yes in llm-inference model list.
An input file on a shared filesystem (/gpfs or $SCRATCH).

Step 1 — Start the endpoint

endpoint start

llm-inference endpoint start \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --gpus 1 \
  --outdir /scratch/$USER/run-001 \
  --slurm account=my_project \
  --slurm qos=accelerated \
  --slurm partition=acc \
  --slurm time=02:00:00 \
  --slurm cpus-per-task=80

The command prints JSON with the Slurm job_id, the auto-assigned port and the model. It also writes these files into --outdir:

OpenAI-compatible API

Mon, 01 Jan 0001 00:00:00 +0000

A large share of inference servers and client libraries speak an OpenAI-compatible HTTP API. Treating it as a portability layer is usually right, but compatibility is a spectrum rather than a guarantee.

This page describes the surface conceptually. For the endpoints exposed by this team’s service, see the LLM Inference module User Guide.

What you will learn

Which endpoints make up the commonly implemented core of the API.
How chat completions differ from the older text completion shape.
How streaming responses are delivered and consumed.
Where compatible implementations tend to diverge in practice.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Overview

Mon, 01 Jan 0001 00:00:00 +0000

llm-inference is a command-line tool that runs large language models on the MareNostrum 5 HPC system. It sits between you and the two things that normally make this awkward — Slurm and vLLM — and handles both.

What the tool does for you

Reserves GPUs and nodes in Slurm with the right parameters.
Starts the vLLM engine inside a Singularity container.
Waits until the model is ready — it polls /health for you.
Returns your results, or gives you an HTTP URL you can use with curl.

The goal is simple: you say which model you want, how many GPUs you need, and how you want to work. The tool does the rest.

What is LLM inference infrastructure?

Mon, 01 Jan 0001 00:00:00 +0000

A model server on its own is not a service. Between a client and a set of accelerators sit gateways, routers, model storage, autoscaling, observability and the control plane that manages them.

This page maps those components and their responsibilities.

What you will learn

Which components make up an inference platform.
How a request travels through the stack.
Where model artifacts are stored and how they reach a replica.
Which responsibilities belong to the control plane versus the data plane.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

What is LLM inference?

Mon, 01 Jan 0001 00:00:00 +0000

Inference is the process of using a trained model to produce output for new input. For large language models this happens in two distinct phases with very different performance characteristics, and almost every serving trade-off in this handbook follows from that asymmetry.

This page establishes the vocabulary used throughout: prefill, decode, tokens, context, and the KV cache.

What you will learn

What happens between a request arriving and tokens being returned.
Why prefill is compute-bound and decode is memory-bandwidth-bound.
What the KV cache is and why it dominates memory planning.
The vocabulary reused across the rest of the handbook.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Installation

Mon, 01 Jan 0001 00:00:00 +0000

llm-inference is already installed on MareNostrum 5. You do not build or install anything — you load it as a module.

Requirements

A MareNostrum 5 account with access to the GPU partition (acc).
A valid Slurm billing account.

Load the module

Load llm-inference

module load EB/apps
module use /apps/ACC/LLM-INFERENCE/modulefiles
module load llm-inference/0.0.1-GCCcore-13.2.0

Verify

llm-inference --version
llm-inference --help

If llm-inference is not found, the module use line did not take effect — check the path and re-run the three commands in order.

Anthropic-compatible API

Mon, 01 Jan 0001 00:00:00 +0000

Some inference servers expose an Anthropic-compatible Messages API alongside or instead of an OpenAI-compatible one. The two express similar capabilities with different request structures, and the differences matter when writing portable clients.

This page covers the shape conceptually; the service-specific details are in the llm- inference user guide.

What you will learn

The structure of a Messages API request and response.
How system prompts and message roles are represented.
How tool use is expressed compared with the OpenAI-compatible surface.
What to account for when supporting both APIs from one client.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Choosing the right GPU number

Mon, 01 Jan 0001 00:00:00 +0000

Accelerator count is driven by two independent requirements: fitting the model and its KV cache in memory at all, and serving the expected concurrent load within your latency targets. Either can dominate.

This page separates the two questions so they can be sized independently and then reconciled.

What you will learn

How to separate the memory requirement from the throughput requirement.
When to add accelerators to one replica versus adding replicas.
How concurrency targets translate into hardware count.
What headroom to leave for traffic variation.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Document Triage Pipeline

Mon, 01 Jan 0001 00:00:00 +0000

Overview

An offline batch pipeline that classifies and routes large document sets without an interactive latency budget.

This page shows the structure a blueprint detail page uses. The problem statement and component list below are real design context; the deployment and usage sections are placeholders.

Problem addressed

Bulk document classification is throughput work, but it is often built on interactive serving infrastructure, which wastes most of the available accelerator capacity.

GPU architecture fundamentals

Mon, 01 Jan 0001 00:00:00 +0000

Reasoning about kernel performance requires an accurate model of the hardware: how work is scheduled onto execution units, and how the memory hierarchy behaves under different access patterns.

This page covers the architectural fundamentals that inference performance depends on.

What you will learn

How GPU work is organised into threads, warps and blocks.
The memory hierarchy and the cost of each level.
Why coalesced memory access matters so much.
How to reason about occupancy and arithmetic intensity.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

LLM quantization

Mon, 01 Jan 0001 00:00:00 +0000

Quantization stores and computes with lower-precision numbers, reducing memory footprint and often increasing throughput. The saving is real and frequently large; the accuracy cost is real too, and workload-dependent.

This page explains the main approaches and how to decide whether a given quantization is acceptable for your task.

What you will learn

What quantization changes in the memory budget and the compute path.
How post-training quantization differs from quantization-aware training.
Which parts of a model are most sensitive to reduced precision.
How to measure quality loss on your own task rather than trusting a general claim.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Mode 2 — One-shot inference

Mon, 01 Jan 0001 00:00:00 +0000

This mode does everything inside a single Slurm job: it starts vLLM, waits until it is ready, sends every request from your input file, writes the output, and shuts vLLM down. You run one command.

When to use it

You already have an input file ready.
You want to process everything in one go and forget about it.
You do not need the endpoint to stay alive afterwards.

Command

bash inference

llm-inference bash inference \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --in  /scratch/$USER/requests.json \
  --out /scratch/$USER/results.jsonl \
  --outdir /scratch/$USER/run-bash \
  --gpus 1 \
  --slurm account=my_project \
  --slurm qos=accelerated \
  --slurm partition=acc \
  --slurm time=01:00:00 \
  --slurm cpus-per-task=80

By default the command blocks until the Slurm job ends. To get the shell back immediately and check on the job later, add --no-wait.

Quickstart

Mon, 01 Jan 0001 00:00:00 +0000

From zero to a first result. Everything here runs on a MareNostrum 5 login node.

What you need

A MareNostrum 5 account with access to the GPU partition (acc).
A valid Slurm billing account — this is the account parameter below.

Step 1 — Load the module

The recommended way to use the tool on MareNostrum 5 is as a module:

Load the module

module load EB/apps
module use /apps/ACC/LLM-INFERENCE/modulefiles
module load llm-inference/0.0.1-GCCcore-13.2.0

# Check that it is available
llm-inference --version

Avoid typing this every session

Static, dynamic and continuous batching

Mon, 01 Jan 0001 00:00:00 +0000

Batching amortises the cost of moving weights through the accelerator across multiple requests. How requests are grouped determines both achievable throughput and the latency individual requests experience.

This page compares the three main strategies and explains the shift toward continuous batching.

What you will learn

Why batching improves accelerator utilization at all.
How static, dynamic and continuous batching differ.
Why variable-length generation penalises static batching so heavily.
How batching configuration trades throughput against tail latency.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Training vs. inference

Mon, 01 Jan 0001 00:00:00 +0000

Training and inference both run the same model architecture on the same accelerators, which makes it tempting to treat them as one problem. Their resource profiles, failure modes and success criteria differ enough that conflating them leads to expensive mistakes.

This page contrasts the two so the rest of the handbook can assume an inference-first perspective.

What you will learn

How the compute, memory and networking profiles of the two workloads differ.
Why throughput is the goal in training and latency often is in serving.
Which training-oriented infrastructure choices transfer to serving and which do not.
How the cost model changes when a workload runs continuously.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

What is distributed inference?

Mon, 01 Jan 0001 00:00:00 +0000

Once a model no longer fits on one accelerator, a single forward pass becomes a distributed computation with collective communication on its critical path. Interconnect performance and failure handling become first-order concerns.

This page covers what distribution adds beyond the parallelism schemes themselves.

What you will learn

When distributing a single request becomes necessary.
Why interconnect bandwidth and latency bound distributed inference.
How a replica spanning several nodes is coordinated.
What happens when one participant in a distributed replica fails.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Calculating GPU memory for serving LLMs

Mon, 01 Jan 0001 00:00:00 +0000

Running out of accelerator memory in production is one of the more common and more avoidable inference failures. A memory budget computed in advance turns it into a capacity decision rather than an incident.

This page sets out the components of that budget and how each one scales.

What you will learn

The components that make up accelerator memory consumption.
How to estimate weight memory from parameter count and precision.
How KV cache size scales with batch size and context length.
How much headroom to reserve for fragmentation and runtime overhead.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Choosing a mode

Mon, 01 Jan 0001 00:00:00 +0000

The tool gives you three ways to work. This page helps you pick one in under a minute.

Decision table

If you need to…	Use	Guide
Process a single input file in one go and get a results file.	`bash inference`	One-shot inference
Keep an endpoint alive and send several batches to it.	`endpoint start` + `batch run`	Manual endpoint
Talk to the model from `curl`, the OpenAI SDK or your own code.	`api start`	Shared HTTP API
Just experiment, or validate your parameters.	any mode with `--dry-run`	Concepts

In one sentence each

Mode 1 — Manual endpoint. You decide when it starts and when it stops. Good for repeated batches against the same loaded model.
Mode 2 — One-shot inference. One command, one input file, one output file. Best for straightforward batch processing.
Mode 3 — Shared HTTP API. Exposes an OpenAI-style HTTP URL for curl or any client.

How they compare

	Manual endpoint	One-shot	HTTP API
Commands to run	2+ (`start`, `batch run`, `stop`)	1	2+ (`start`, `stop`)
Model loaded	once, reused	once per job	once, reused
Input	`.json` file per batch	`.json` file	live HTTP requests
Output	`.jsonl` file per batch	`.jsonl` file	HTTP responses
Ends when	you run `endpoint stop`	the batch finishes	you run `api stop`
You must stop it	yes	no	yes

Modes 1 and 3 hold GPUs until you stop them

Choosing the right kernel optimization tool

Mon, 01 Jan 0001 00:00:00 +0000

Kernels can be written at several levels of abstraction, from vendor assembly through domain-specific languages to compiler-generated code. The levels differ in achievable performance, portability and how much effort each change costs.

This page compares them and suggests how to choose.

What you will learn

The levels of abstraction available for kernel work.
How portability and peak performance trade against each other.
Where compiler-based approaches are sufficient.
Which profiling tools to use to direct the effort.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Clinical Note Summarization

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Summarize clinical documentation into structured fields with human review built into the workflow.

Why contiguous KV cache allocation fragments memory.
How paging the KV cache into fixed-size blocks addresses that.
How paging enables sharing between requests with common prefixes.
What the indirection costs.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Mode 3 — Shared HTTP API

Mon, 01 Jan 0001 00:00:00 +0000

This mode starts a long-lived vLLM endpoint that other tools can talk to over HTTP: curl, the OpenAI SDK, or your own applications — as long as they run inside the MareNostrum 5 network.

When to use it

You want to share the endpoint between several tools.
The requests are interactive and you do not know them in advance.
You want the URL to stay alive until you stop it.

No authentication, TLS or proxy support

Choosing the right inference framework

Mon, 01 Jan 0001 00:00:00 +0000

Serving frameworks differ less in the features they list than in what they optimise for and how they behave at the edges of their operating range. The right comparison is against your own traffic pattern.

This page gives the criteria to compare on, and a method for evaluating candidates without committing to one first.

What you will learn

The criteria that meaningfully separate serving frameworks.
Which optimizations to check for, and why their presence alone is not enough.
How to benchmark candidates against your own traffic pattern.
What operational characteristics to weigh alongside raw performance.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Concepts

Mon, 01 Jan 0001 00:00:00 +0000

The vocabulary this guide uses. Where a term is a general inference concept rather than something specific to this tool, it links to the handbook.

Tool concepts

Term	Meaning
Model ID	The identifier passed to `--model`, e.g. `mistralai/Mistral-7B-Instruct-v0.3`. Must appear in `llm-inference model list` with `SUPPORTED=yes`.
Outdir	The directory given with `--outdir`. The tool writes the generated Slurm script, logs and sidecar files here.
Sidecar file	A small JSON written into the outdir recording `job_id`, `port` and `model`. It is how commands find a running endpoint.
Endpoint	A running vLLM server inside a Slurm job, reachable on an auto-assigned port.
Job ID	The Slurm job identifier. Every `status` and `stop` command takes it via `--job`.
Dry run	`--dry-run` validates parameters and writes the Slurm script without submitting.

Required Slurm parameters

All three modes need the same set. Pass them by repeating --slurm with key=value pairs:

Fast scaling

Mon, 01 Jan 0001 00:00:00 +0000

Inference replicas scale slowly compared with stateless services, because a replica cannot serve until many gigabytes of weights have been fetched and loaded. Autoscaling that ignores this reacts long after the traffic has changed.

This page covers where that time goes and how to reduce it.

What you will learn

Which stages make up replica startup time.
How model loading and image pull dominate that time.
Which techniques shorten each stage.
How to set autoscaling policy around the remaining latency.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

FlashAttention

Mon, 01 Jan 0001 00:00:00 +0000

A textbook attention implementation materialises the full attention matrix in memory, making the operation memory-bound and its footprint quadratic in sequence length. That is the dominant cost for long contexts.

FlashAttention restructures the computation so the matrix is never fully materialised, trading extra arithmetic for far less memory traffic.

What you will learn

Why naive attention is memory-bound rather than compute-bound.
How tiling and online softmax avoid materialising the attention matrix.
Why trading recomputation for memory traffic is a net win here.
How the approach applies differently to prefill and decode.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Literature Review Assistant

Mon, 01 Jan 0001 00:00:00 +0000

Overview

An agent that searches, filters and summarizes scientific literature against a stated research question.

This page shows the structure a blueprint detail page uses. The problem statement and component list below are real design context; the deployment and usage sections are placeholders.

Problem addressed

Screening literature for relevance is mechanical, high-volume work, but summaries that drop the provenance of a claim are worse than useless for research.

Architecture

Component overview

  [1] Search tools
  [2] Relevance screening
  [3] Citation-preserving summarizer
  [4] Inference backend

A concrete architecture diagram will replace this sketch when the blueprint is published.

Speculative decoding

Mon, 01 Jan 0001 00:00:00 +0000

Decoding is limited by memory bandwidth rather than compute, so a forward pass that verifies several candidate tokens costs little more than one that produces a single token. Speculative decoding exploits that.

This page explains the mechanism and the conditions under which it actually pays off.

What you will learn

Why verifying several tokens costs little more than generating one.
How draft-and-verify preserves the target model’s output distribution.
Which draft model choices work, including model-free variants.
When speculative decoding fails to help or hurts.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Structured outputs

Mon, 01 Jan 0001 00:00:00 +0000

Asking a model to reply in JSON and hoping for the best does not survive production traffic. Structured output support constrains generation so the result conforms to a schema by construction.

This page covers the available approaches and their costs.

What you will learn

The difference between prompting for a format and constraining generation to it.
How grammar and schema-constrained decoding work at a high level.
What constrained decoding costs in latency and flexibility.
How to handle the cases a schema cannot express.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Where is LLM inference run?

Mon, 01 Jan 0001 00:00:00 +0000

Where inference runs determines what you can control and what you must accept. The decision is rarely purely technical: data residency, procurement, existing hardware and the shape of demand all constrain it.

This page lays out the options and the criteria for choosing between them.

What you will learn

The main deployment targets and what distinguishes them.
How data residency and compliance constrain the choice.
When on-premises accelerators are worth the operational burden.
Why demand shape matters as much as demand volume.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

InferenceOps and management

Mon, 01 Jan 0001 00:00:00 +0000

Beyond the initial deployment, an inference service needs release procedures, capacity management, cost attribution and incident response — the ordinary operational disciplines, adapted to workloads where a single replica may cost more per hour than an entire conventional service.

This page collects those practices.

What you will learn

How to release model and configuration changes safely.
How to manage capacity and quota across tenants.
How to attribute and control cost.
What inference-specific incident response looks like.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Key metrics for LLM inference

Mon, 01 Jan 0001 00:00:00 +0000

Inference performance cannot be summarised by a single number. A deployment tuned for tokens per second may feel unresponsive; one tuned for first-token latency may waste most of its accelerator capacity.

This page defines the metrics used throughout the handbook and, importantly, what each one obscures when quoted alone.

What you will learn

The definitions of time to first token, time per output token and end-to-end latency.
How throughput and concurrency relate, and why averages mislead.
What goodput measures and why it matters under load.
Which metrics to attach to a service-level objective.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Model Context Protocol

Mon, 01 Jan 0001 00:00:00 +0000

The Model Context Protocol standardises how a model-facing application connects to tools and data sources, so that integrations can be written once and reused across clients rather than rebuilt per application.

This page introduces the concepts and where MCP fits relative to plain function calling.

What you will learn

What problem MCP standardises, and for whom.
The roles of host, client and server in the protocol.
How MCP relates to and differs from direct function calling.
What to consider before exposing internal systems through an MCP server.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Prefill-decode disaggregation

Mon, 01 Jan 0001 00:00:00 +0000

Prefill is compute-bound and decode is bandwidth-bound. Running both on the same replica means one phase’s requirements always compromise the other’s, and interference between them shows up directly in tail latency.

Disaggregation separates them, at the cost of moving KV cache between resources.

What you will learn

Why colocating prefill and decode causes interference.
How a disaggregated deployment routes a request through both stages.
What transferring KV cache between stages costs.
When the added complexity is justified.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Voice Assistant Frontend

Mon, 01 Jan 0001 00:00:00 +0000

Overview

A speech interface that pairs streaming text-to-speech with a conversational model backend.

This page shows the structure a blueprint detail page uses. The problem statement and component list below are real design context; the deployment and usage sections are placeholders.

Problem addressed

Voice interfaces are judged almost entirely on responsiveness, so the naive approach — generate the full text response, then synthesise it — feels broken even when both stages are individually fast.

Accessible Content Narration

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Generate spoken narration for written content as a batch process, with pronunciation overrides.

This page shows the structure a blueprint detail page uses. The problem statement and component list below are real design context; the deployment and usage sections are placeholders.

Problem addressed

Narrating a large content library is a throughput problem with a long tail of domain terms that generic synthesis mispronounces.

Architecture

Component overview

  [1] Text normalisation
  [2] Pronunciation dictionary
  [3] Batch synthesis runner
  [4] Audio store

A concrete architecture diagram will replace this sketch when the blueprint is published.

Input and output format

Mon, 01 Jan 0001 00:00:00 +0000

Both batch run and bash inference use the same input and output format.

Input file (`.json`)

A JSON object with an array of requests. Each request has its own id plus the usual chat-API fields:

requests.json

{
  "requests": [
    {
      "id": "req-001",
      "model": "mistralai/Mistral-7B-Instruct-v0.3",
      "messages": [
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user",   "content": "Explain tensor parallelism in one paragraph."}
      ],
      "max_tokens": 256,
      "temperature": 0.7
    },
    {
      "id": "req-002",
      "model": "mistralai/Mistral-7B-Instruct-v0.3",
      "messages": [
        {"role": "user", "content": "Write a Python function that returns the n-th Fibonacci number."}
      ],
      "max_tokens": 384,
      "temperature": 0.0
    }
  ]
}

Fields

Field	Purpose
`id`	Your identifier for the request. Copied to the output so you can match them up.
`model`	The model ID. Should match the model the endpoint was started with.
`messages`	The conversation, as `role` / `content` pairs. Roles: `system`, `user`, `assistant`.
`max_tokens`	Upper bound on generated tokens for this request.
`temperature`	Sampling temperature. `0.0` is the most deterministic.

Tip

Prefix caching

Mon, 01 Jan 0001 00:00:00 +0000

Production traffic repeats itself. Shared system prompts, few-shot examples and multi- turn conversations all mean requests frequently share a long prefix whose KV cache was already computed.

Prefix caching reuses that work, which can remove most of the prefill cost for such requests.

What you will learn

Why real traffic contains so much shared prefix.
How cached prefixes are matched and reused.
How prefix caching interacts with paged KV cache management.
What cache invalidation and eviction require.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

Prompt engineering

Mon, 01 Jan 0001 00:00:00 +0000

Prompt engineering has a poor reputation earned by folklore, but the underlying practice is ordinary engineering: change one thing, measure the effect on a fixed evaluation set, keep what works.

This page focuses on techniques that survive that treatment, and on the evaluation discipline that makes the difference.

What you will learn

Which prompt structures reliably improve task performance.
How to build an evaluation set before tuning prompts.
How prompt length interacts with latency and cost.
Why prompts should be versioned like any other production artifact.

Recommended outline

This page is an outline. The subsections below are the planned structure; they are filled in as the handbook is written.

API reference

Mon, 01 Jan 0001 00:00:00 +0000

In HTTP API mode, vLLM serves an OpenAI-compatible HTTP surface. This page documents the endpoints you will use.

All paths are relative to the base URL printed by api start, which looks like http://nid001:45123. Host and port are assigned at run time and change on every start.

No authentication

There is no API key, no TLS and no proxy support. The endpoint is reachable by anyone inside the MN5 network who knows the host and port. Do not treat it as a shared service outside your own work.