Document 15 · Quick learnPromise p95 · The knee is the cache · Utilisation is nobody’s job
Goodputand the knee
Which numbers to promise, where the knee actually is, how to measure without fooling yourself, and the cost lever that is nobody’s job.
“Wonderful throughput, and every user waiting eight seconds for a first word.”
7,820 tok/s
goodput peak — raw asymptotes at 8,890
60 not 150
where the knee really is, at 2,300 tokens
82%
of every read is cache at batch 200
2.5×
20% → 50% utilisation, on cost per token
The four numbersand only one of them means anything
This is the document where an infrastructure background is an advantage rather than a gap — most of what follows is admission control, queueing, percentiles and unit economics wearing new vocabulary.
Time to first token
Queue + CPU front end + prefill.
What the user feels as the pause
Inter-token latency
Set by the decode loop.
What the user feels as streaming speed
Throughput
Tokens per second across everyone.
Set by batch size — and optimising it alone produces an unusable product
Goodput
Throughput that met the latency promise.
The only one that means anything
Why goodput and not throughput
Optimising raw throughput produces a server with excellent numbers and unusable latency: you can report very high tokens per second while every user waits eight seconds for a first word. Those tokens were delivered and the product does not work.
And the targets have to come from the product, because chat and batch processing want opposite configurations
Always percentiles — p50, p95, p99
In LLM serving the tail is much worse than the middle, and for a nameable reason: one person pasting a large document delays everyone behind them.
That is a mechanism you can fix, not just variance you have to accept
You cannot have bothand the knee is not where the textbook says
Throughput and latency are the same dial. Below the knee each extra user is nearly free; above it, every extra user genuinely slows everyone down.
The textbook knee — batch ~150
The compute ridge point, where decode goes compute-bound.
Correct when sequences are short
295 × 0.40 ÷ 0.80 ≈ 150 on an H100
Moved by: nothing in the memory list
Two different knees. Two different fix lists.
The real knee — from about batch 60
The cache read takes over long before compute does.
At a realistic 2,300-token average conversation
At batch 60 the cache is already more than half of every decode read
WorkedRaw throughput asymptotes near 8,890 tok/s · goodput peaks at 7,820 and breaks at batch 391, where per-user speed falls under 20 tok/sthe SLO decides where on the curve you sit
Why this distinction changes what you do
A memory-bandwidth knee is moved by an fp8 cache, a shorter context limit or fewer KV heads. A compute knee is not moved by any of those.
Find the knee by benchmarking inter-token latency against batch size rather than assuming a number — and tell the two apart by looking at achieved bandwidth
Under pressuretwo failure modes, two signatures
Interference — one long prefill stalls everyone
Every other stream freezes and nothing logs an error.
Signature: ITL spikes for everyone at once, correlated with one big request
Fix: chunked prefill — always, and confirm it is on
Structural fix: disaggregation, only once chunking is measured as insufficient
Different signatures. Different answers.
Preemption — you admitted more than you can hold
Throughput falling while latency spikes.
Signature: cache pinned at 100%, queue growing, throughput falling
Adding load makes it worse
Fix: admit less. Alarm on preemption count — it leads latency
Chunked prefill
Simple, one pool of machines, removes stalls, costs the large request a little wall-clock time.
It should be on everywhere
Disaggregation
Prefill on one pool, decode on another, cache shipped between. No interference at all, independent scaling — and the real argument: they can use different hardware, because prefill wants compute and decode wants bandwidth.
Costs real complexity and a cache transfer on every request. Chunk first, always; disaggregate only when chunking is measured as insufficient
Benchmarking honestlyfive rules, and anything less is a description
Realistic lengths
Sample from your real prompt and output distribution — including the tail.
A fixed request rate
Not only at saturation. Maximum throughput is one point on the curve.
Warm up, then steady state
Early requests pay for compilation and cache warming.
Thermal throttling only appears after twenty minutes
Both latencies, all percentiles
TTFT and ITL separately, at p50, p95 and p99.
One variable at a time
Quantisation, batch limits, context length and speculation interact.
Varying them together teaches you nothing you can reuse
For assessing somebody else’s benchmark
The first question is always what the other side was configured to do. Two engines at different precisions are not being compared — they are being described.
And if a number was produced in under a minute, treat it as a smoke test rather than a measurement
Scaling up and downcold start is minutes, so you cannot scale into a burst
A stateless web instance
Seconds to ready.
An LLM replica
Minutes to ready — tens of gigabytes read off storage, loaded into GPU memory, kernels compiled, CUDA graphs captured, cache pool allocated.
By the time your new capacity is ready, the spike is over
Which makes headroom a design input rather than a planning failure
A conversation worth having with finance before it is had during an incident.
Scale on leading signals
Queue depth and cache utilisation tell you trouble is coming. Latency tells you it arrived.
Keep warm capacity
Headroom always running, or pre-warmed instances ready to admit.
Costly — so is failing a spike
Make loading faster
Weights on fast local storage rather than object storage.
And a 4-bit model is a quarter of the bytes
Route by prefix, not round-robin
Send requests sharing a prefix to the same replica so it reuses its cached prefix.
Session affinity for chat; grouping by system prompt for agents. Round-robin actively destroys the thing you just turned on
The lever nobody ownsand it is bigger than anything else here
$ per million tokens = ( $ per GPU-hour ÷ tokens per hour ) × 1,000,000
every technique in this runbook eventually shows up in that one division
More users per GPU
GQA, an fp8 cache, paged memory.
Raises the numerator
Fewer bytes per token
Quantised weights.
Raises the numerator
Less repeated work
Prefix caching.
Raises the numerator
Higher utilisation
Do not run half-empty GPUs.
Usually the biggest of the four — and the only one that is not an engineering change
The observation to make in a cost conversation
A half-idle GPU costs exactly the same as a busy one. Raising average utilisation from 20% to 50% is a 2.5× improvement in cost per token — larger than quantisation, larger than prefix caching, larger than almost anything in this runbook. And in most organisations it is nobody’s number: it does not appear in an engineering backlog because it is not a defect, and it does not appear in a product roadmap because it is not a feature.
Routing traffic onto fewer fuller machines, scheduling batch work into the overnight trough, and consolidating small deployments are the three moves — and none of them is a model change
The deployment checklistif you are asked to design a serving setup on the spot
Q 1 · doc 15
What is the latency budget — and is it TTFT, streaming, or both?
Q 2 · doc 01, 14
What are the real prompt and output length distributions, including the tail?
Q 3 · doc 09, 11
Does the model fit on one GPU at your chosen precision? Quantise before splitting.
Q 4 · doc 06
Work out KV cache per token from the config, then per user at your context limit.
Q 5 · doc 06, 14
Divide the remaining memory by that. Compare against expected concurrency.
Q 6 · doc 12
Pick an engine — vLLM generally, SGLang if there is heavy prefix reuse.
Q 7 · doc 10, 12
Turn on the free wins: continuous batching, paged cache, chunked prefill, prefix caching.
Q 8 · doc 10
Consider an fp8 KV cache if memory is the binding constraint. Test on long outputs.
Q 9 · doc 13
Consider speculative decoding only if batches are small and streaming latency matters.
Q10 · doc 15
Benchmark at your real request rate, warm-up discarded, reporting p50/p95/p99.
Q11 · doc 10, 15
Set admission limits so the server queues rather than thrashes.
Plan for minutes-long cold starts — keep warm headroom, and route by prefix.
Q14 · doc 14
Compute cost per million tokens and check it against the business case.
Running the teamwhat “done” means, and three cadences
1 · The benchmark says it helped
At the real request rate, on p95 rather than the mean, with warm-up discarded.
2 · It is reversible
Or the rollback is written down.
3 · The detector exists
Enabled an fp8 cache? There is now a long-generation quality check. Enabled prefix caching? There is a hit-rate alarm.
4 · The capacity note reflects it
Whatever it did to the footprint.
Four items, and none of them is expensive. “It is deployed” is not on the list.
The three cadences worth holding
Weekly
Look at the four gauges and the four timers together.
Not to act — so that drift is noticed while it is small: prefix hit rate falling, preemption creeping up, utilisation sliding
Monthly
Recompute cost per million tokens from measurement, and compare against the managed-API price for the same traffic.
Two numbers, ten minutes — and it keeps build-versus-buy from being re-argued on vibes
On every model or engine upgrade
Re-run the same benchmark at the same request rate, and re-derive the capacity note.
Defaults change, kernels change, memory footprints change. Treat an upgrade as a release, not a dependency bump
Answer it out loudthe ninety-second version, then five questions
“Four numbers: time to first token, inter-token latency, throughput and goodput — and goodput is the only one that means anything, because a server can report wonderful throughput while every user waits eight seconds. Everything at p95, because the tail is much worse than the middle and for a nameable reason. Throughput and latency are the same dial: below the knee extra users are nearly free, above it they cost everyone. And the knee is usually not where people expect — with realistic conversation lengths the cache read takes over from about batch 60, well short of the compute ridge at 150. That matters because the two knees have different fix lists.”
Five questions, and the spine of each answer
Eng manager
Which metric would you optimise?
Goodput — throughput that met the latency targets
Raw throughput alone produces excellent numbers and unusable latency
And the targets come from the product, because chat and batch want opposite configurations
So my first move on “make it faster” is to ask what the budget is and whether it is the pause or the streaming. Everything at p50/p95/p99 — the tail is worse for a fixable reason, not just variance.
Architect
Would you use larger batches?
Up to the knee, yes — below it users are nearly free because the GPU waits on memory anyway
Past it you buy very little throughput and pay real latency
But the textbook knee is often wrong: the compute ridge is ~150, and the cache read dominates from about batch 60
At batch 200 the cache is 82% of every read
That distinction changes the fix: a memory knee is moved by fp8 cache, lower limit or fewer KV heads; a compute knee is not. Find it by benchmarking ITL against batch, and tell them apart by achieved bandwidth.
Eng manager
How would you benchmark two serving engines fairly?
Same model, precision, context limit and request distribution — sampled from real traffic including the tail
Warm-up discarded, measured at fixed request rates, not only at saturation
p50/p95/p99 for both TTFT and inter-token latency
Run twenty minutes, not thirty seconds — throttling only appears under sustained load
And change one thing at a time. For someone else’s benchmark, the first question is what the other side was configured to do: two engines at different precisions are being described, not compared.
Architect
Why is autoscaling harder here?
Cold start is minutes, not seconds — tens of gigabytes read, loaded, compiled, graphs captured
By the time capacity is ready the spike is over. You cannot scale into a burst
So headroom is a design input, not slack to be trimmed
Scale on queue depth and cache utilisation, because latency tells you trouble already arrived
One more specific to this domain: do not route round-robin. Sending requests that share a prefix to the same replica turns a routing decision into a prefill saving — and with prefix caching on, round-robin actively destroys it.
Eng manager
How do you reduce cost per token?
Four levers, named in order of size rather than technical interest
Utilisation — 20% to 50% is 2.5×, and it is not an engineering change
More users per GPU — GQA, fp8 cache, paging
Fewer bytes per token — quantised weights. Less repeated work — prefix caching
And the observation worth making out loud: a half-idle GPU costs exactly what a busy one does, and in most organisations utilisation is nobody’s number — not a defect, so not in a backlog; not a feature, so not on a roadmap.
Fast answersthe six that come up most
QHow long should a benchmark run?
Twenty minutes at steady state, after discarding warm-up. Thirty-second runs miss thermal and power throttling entirely, which in a dense rack is a real effect, and they miss the cache reaching its working set. Under a minute is a smoke test, not a measurement.
QWhy does GPU utilisation not tell me much?
It is a duty cycle — the fraction of time at least one kernel was resident. Batch-1 decode keeps a kernel resident for the whole five milliseconds while using under half a per cent of the arithmetic, and reports 100%. Measure achieved bandwidth for decode, MFU for prefill.
QShould I run batch and interactive traffic on the same fleet?
Generally no. They want opposite configurations — batch pushes far past the knee, interactive sits below it — and the batch job’s large batches will push interactive latency over budget. Separate deployments, separate caps, separate SLOs.
QHow do I route across replicas?
Not round-robin. Send requests sharing a prefix to the same replica so its cached prefix gets reused — session affinity for chat, grouping by system prompt for agents. That converts a routing decision into a prefill saving, and it is free.
QWhat is the most common operational mistake?
Admitting more work than the cache can hold and then tuning, rather than admitting less. The signature is cache pinned at 100%, a growing queue and throughput falling. Adding load makes it worse. It is the same instinct as a connection pool limit.
QWhat is the one process change worth making?
Treating a model or engine upgrade as a release rather than a dependency bump: re-run the same benchmark at the same request rate, and re-derive the capacity note. That single habit catches most of the surprises in this stack.
Goodput is the only number that means anything, the knee is usually the cache read rather than the compute ridge, and the biggest cost lever in this entire runbook is not in it: a half-idle GPU costs exactly what a busy one does, and in most organisations utilisation is nobody’s number. “Done” for a serving change means four things — the benchmark says it helped at p95, it is reversible, the detector for its failure mode exists, and the capacity note is updated. “It is deployed” is not on that list.