A model benchmarks at 80 tokens a second on a quiet GPU. In production, users say it feels slow, and the dashboard shows an average response time of 1.9 seconds. Every one of those numbers can be true at once. None of them is what the user felt.
Inference performance is not one number. You need latency, throughput, cost and energy together, and each of them has a definition that hides a choice. Here are the six that matter, with a demo for each that lets you break it.
Training metrics tell you nothing about how inference feels
Training and inference want different things. Training updates the model’s parameters from data, so you care about how fast batches go through and how long a run takes. Inference runs a finished model to produce an answer for someone who is waiting.
That changes the questions. How long until something appears on screen? How fast does it stream? How many people can it serve at once? What does each answer cost, in money and in energy?
A request has two phases, and each has its own clock
Every LLM request runs in two phases. Prefill reads the whole prompt in one step and produces the first token. Decode then generates the rest one token at a time, each step waiting on the one before.
The DistServe paper, from researchers at Peking University, StepFun and UC San Diego, defines the two core metrics by those phases. Time to first token (TTFT) is “the duration of the prefill phase”, and time per output token (TPOT) is “the average time taken to generate a token for each request (except for the first token)”. The phases also stress different parts of the hardware: prefill “tends to be compute-bound”, while decoding is “constrained by the GPU’s memory bandwidth”.
In practice TTFT includes everything before the first token, not only prefill. NVIDIA’s benchmarking guide says it “generally includes both request queuing time, prefill time, and network latency”, and that “the longer the prompt, the larger the TTFT”. In a RAG system, retrieval and context assembly happen before the model sees anything, so they land in TTFT as well.
drag the prompt length to 32k, then drag the reply length. Watch which number each slider moves. Simulated, with illustrative timings.
Once you see the split, the arithmetic is easy. End-to-end latency is TTFT plus TPOT for every token after the first. If 100 tokens after the first take 5 seconds, TPOT is 50 milliseconds, which is 20 tokens a second.
One warning before you compare numbers from two tools: they don’t all compute this the same way. NVIDIA notes that GenAI-Perf leaves TTFT out of its inter-token latency average, “as opposed to LLMPerf, which does include the TTFT”. Two benchmarks can report a different TPOT for the identical run, so always check whose definition you are reading.
Which system is faster depends on how much it says
Here is where having two clocks starts to matter. A system with a quick start and a slow stream can beat one with a slow start and a fast stream, or lose to it badly, and the only thing that decides it is the length of the reply.
A starts talking almost at once. B makes you wait, then talks three times faster.
run the chat reply, then switch to the summary and run it again. Simulated, with illustrative timings.
The DistServe authors tie this to real products: “real-time chatbots prioritize low TTFT for response promptness, while TPOT only remains important until it is faster than human reading speed (i.e., 250 words/min). Conversely, document summarization emphasizes low TPOT for faster generation of the summary.” Same model, different product, different metric that matters.
More users make the system faster and every user slower
Throughput is how much work the system does per second, usually counted as requests per second or output tokens per second. It is tempting to read it as a speed, and that is where most of the confusion starts.
When more users arrive, total throughput usually goes up, not down. Servers batch concurrent requests, so one pass over the model’s weights serves many people at once. What falls is each person’s speed. NVIDIA’s guide puts both halves in one sentence: “the total TPS for the whole system will increase, while TPS per user decreases as latency increases.”
drag from 1 user to 64, then keep going to 256. Simulated, with an illustrative server that batches up to 64 requests.
That holds until the server runs out of room to batch. Past that point NVIDIA says requests “will have to wait in a queue”, and you “may see an increase in TTFT value due to the queueing effect”. Throughput flattens while latency keeps climbing, and if you push far enough past saturation, throughput can start to drop as well.
This is why DistServe argues for goodput rather than raw throughput, defined as “the maximum request rate that can be served adhering to the SLO attainment goal”. In plain terms: how many requests a second you can serve while still hitting your latency targets.
Throughput that nobody is willing to wait for doesn’t count.
Cost and energy per token are mostly a utilisation problem
The cost formula is as plain as it looks: total inference cost divided by total output tokens. Spend $10 to produce a million tokens and each one costs $0.00001, or $10 per million. That cost covers more than the GPU hour. Memory, power, infrastructure, maintenance and any provider fees all belong in it.
Energy works the same way, in joules. A system that uses 3,600 joules to produce 1,000 tokens is spending 3.6 joules a token. 3,600 joules is exactly one watt-hour, which is a handy check when you convert between J per token and Wh per token.
drag utilisation from 40% down to 10%, then up to 100%. Then switch to the whole server. Simulated, with illustrative prices and power figures.
What the formulas hide is the bottom of the fraction. You pay for a GPU by the hour whether it is working or not, and an idle GPU still draws power. Tokens actually produced, not tokens the hardware could produce, is what you divide by. I’ve written before about why cutting tokens doesn’t always cut the bill, and utilisation is the other half of that story.
Energy has one more trap. A figure measured at the GPU and a figure measured at the wall are both correct, and they can be far apart, because the CPU, memory, cooling and networking draw power too. Say which one you measured, and whether you mean average power during a request or total energy for it. For Green AI reporting or energy-aware scheduling, that boundary matters more than the decimal places.
The average hides the requests people remember
Imagine 95% of requests finish in under 2 seconds and a handful take 15. The average looks fine. The people who waited 15 seconds don’t care about the average.
P95 is the latency that 95% of requests come in at or under. P99 is the same at 99%. These are the numbers that expose long queues, cache misses, unusually heavy requests and contention for the GPU, and they are what SLAs are written against.
drag slow requests to 2% and compare how far the mean moves with how far P99 moves. Then switch to 10 calls. Simulated, with 400 canned requests.
Set slow requests to 5% and you get the system from the example: P95 just under 2 seconds, P99 over 15. The mean says 1.8.
The fan-out toggle is the part I think matters most right now. Dean and Barroso made the argument for distributed systems in The Tail at Scale: “Temporary high latency episodes which are unimportant in moderate size systems may come to dominate overall service performance at large scale.” An agent is a fan-out system. If each model call independently has a 1% chance of being slow, a ten-step agent hits at least one slow call on 9.6% of runs. A tail you could ignore per call becomes the experience per task.
What I’d put on the dashboard
None of these six numbers is enough on its own, and most arguments I see about whether a system is fast come from two people reading different ones.
If I were setting up monitoring for an LLM feature today, I’d report TTFT and TPOT separately rather than one latency figure, at P50, P95 and P99, under a stated level of concurrency. I’d write down which tool’s definition of TPOT we used. I’d track cost and energy per token at the utilisation we actually run at, not the benchmark peak, and note whether energy was measured at the GPU or the wall. Which of these matters most depends on the product, and I’d hold the exact thresholds loosely until there is real traffic to set them against.