LLM Guide Explains 9 Inference Metrics for Measuring Latency, Throughput and Goodput
Updated
Updated · KDnuggets · Nov 11
LLM Guide Explains 9 Inference Metrics for Measuring Latency, Throughput and Goodput
3 articles · Updated · KDnuggets · Nov 11
Summary
Nine metrics are laid out to measure LLM inference more accurately than raw throughput, splitting performance into first-token delay, streaming speed, request volume and service quality.
The guide centers on single-request latency: TTFT tracks the wait for the first token, E2E the full response time, TokenGen the streaming span, TPOT the average post-first-token gap, and ITL each individual gap.
Across many requests, it recommends reporting mean, median and tail percentiles such as P99 separately for TTFT, TPOT and E2E, using raw combined samples rather than averaging per-server percentiles.
At the system level, RPS measures completed requests per second, TPS measures input or output tokens per second, and goodput counts only requests that meet all latency SLOs—so it can never exceed RPS.
The broader point is that high token throughput alone can mask slow starts or stalled streams under load, making metric definitions and workload mix critical when comparing LLM serving benchmarks.