← Back

LLM Inference Benchmarks That Reflect Real Work

Benchmark LLM inference with prompt processing, generation, concurrency, and task quality. Report latency distributions and configuration details that others can compare.

An LLM inference benchmark is useful only when its workload resembles the decision you need to make. A single tokens-per-second number cannot tell you whether a model will feel responsive in chat, finish a batch extraction job on time, or remain usable when several people connect at once.

Start with the product question. For an interactive assistant, the initial wait and pauses during streaming matter. For overnight processing, completed valid records per hour may matter more. Benchmark those outcomes before choosing a headline metric.

Separate prompt processing from generation

The llama-bench documentation distinguishes prompt processing, token generation, and combined tests. These exercise different parts of inference. Keep their results separate when comparing configurations.

A short prompt followed by a long answer emphasizes generation. A large document followed by a small classification result emphasizes prompt processing. Two runtimes can therefore trade places depending on the workload.

An illustrative benchmark matrix might include:

WorkloadInput shapeOutput shapeMain concern
Chat replyShort conversationModerate answerFirst response delay
Document extractionLong documentSmall structured recordInput processing and validity
Code explanationMedium source fileLong explanationSustained generation
Shared serviceMixed requestsMixed lengthsQueueing and tail latency

These categories are suggested test cases, not measured results. Populate them with examples from your intended application and keep the fixtures stable between runs.

Define every latency metric

Time to first token measures the wait before generation starts producing output at the observation point. End-to-end latency measures the complete request. Inter-token or inter-output timing describes streaming behavior, but the exact definition depends on instrumentation.

vLLM's metrics documentation distinguishes request-level time per output token from latency between streamed output events. Those can differ when one event contains several tokens or when aggregation weights differ. Record the definition with the metric rather than assuming identical names imply identical measurements.

Measure from the client when evaluating user experience. Server-only timings may exclude network delay, authentication, retrieval, and application buffering. For debugging, retain both views so you can locate the source of a slowdown.

Report a distribution, not only a mean. Include the number of requests and enough tail information to expose slow cases. A percentile without a meaningful sample size can imply more precision than the experiment supports.

Control the configuration

Record hardware, available memory, operating system, runtime commit or release, backend, model artifact, quantization, context settings, thread counts, and acceleration options. These details are part of the result, not administrative decoration.

Keep quality settings explicit too: sampling parameters, output limits, stop conditions, and any constrained decoding. If one configuration produces much shorter answers, it can appear faster without doing equivalent work.

Use the same prompts when possible. When comparing models with different tokenizers, report input and output lengths in each model's tokens and consider a task-level metric such as valid documents processed. Raw token throughput is not a universal unit of useful work across tokenizers.

Do not silently change a model revision during a series of runs. Save artifact identifiers so a later comparison refers to the same weights.

Test cold and warm behavior

Separate startup from steady-state serving. A process that has already loaded weights and initialized kernels is a different condition from an application launch. If your product frequently scales to zero, cold behavior may be part of the normal user experience.

State whether caches are enabled and whether prompts share a reusable prefix. A benchmark built from repeated identical prompts may measure a cache-friendly workload that real traffic does not resemble.

Run warm-up work deliberately, then exclude or report it according to a documented rule. Do not discard inconvenient first runs without saying why. For local devices, control power mode and note thermal conditions when they affect sustained operation.

Increase load until the service degrades

A single request shows isolated performance. It does not establish capacity. Test a range of concurrency or arrival rates that reflect anticipated use, and watch queue depth, failures, timeouts, and completed throughput.

Closed-loop clients, which send the next request after the previous one finishes, create a different load pattern from independently scheduled arrivals. State which approach you used. Otherwise, comparisons can hide queueing pressure.

Track unsuccessful requests in the denominator. Reporting throughput only for successful short requests while ignoring timed-out long ones creates an attractive but misleading result.

Choose an operating point where latency and reliability meet the product's requirements. Maximum throughput at an unusable response time is rarely the target for an interactive assistant.

Keep quality in the benchmark

Performance changes can alter outputs. Evaluate schema validity, required facts, and task success alongside timing. For extraction, count records that are both completed and valid. For coding, use task-appropriate checks rather than rewarding the fastest plausible-looking answer.

Use the local model memory worksheet to record hardware assumptions, and the agent evaluation dataset to measure whether faster inference still completes useful work. Report the workload and quality alongside latency.

Publish the workload, configuration, sample size, and limitations with the result. A narrow benchmark can still be valuable when its scope is honest. The goal is a reproducible decision about your application, not a number that looks universally impressive.