Modes
The three ways of working with llm-inference: manual endpoint, one-shot inference and shared HTTP API.
Three ways to work, each suited to a different task. If you are not sure which one you need, start with Choosing a mode.
- Manual endpoint Start an endpoint that stays alive in Slurm, send several batches to it, then stop it yourself.
- One-shot inference Process one input file in a single Slurm job with bash inference, then stop.
- Shared HTTP API Expose a long-lived OpenAI-style HTTP URL that curl, the OpenAI SDK or your own code can call.