Every past token leaves behind a key and a value. Store them and you pay memory; throw them away and you pay compute — quadratically. This one formula decides how many people fit on a GPU, and everything in tracks C and D exists to make its answer smaller.
Attention makes every new token look at every previous token. Those keys and values never change once computed — so you either store them once, or rebuild all of them on every single step.
Compute K,V for 1 token
Recompute all 2
Recompute all 3…
A 2,000‑token prompt with 300 output tokens costs 280× more compute. At 8,000 in and 1,000 out it is 945×.
Compute K,V · store
Compute 1 · read 1
Compute 1 · read 2
Each token is written exactly once and then re‑read forever. Nothing is summarised, nothing decays, nothing is compressed.
A past token’s question is never asked again — only the current token asks. Its key and value, though, are read by every future token and never change. Cache what is re‑read; discard what is not.
Without a transcript, every witness would have to repeat their whole testimony before each new question. With one, each statement is written down once and simply re‑read. It is not a summary and it does not shrink — every line stays, forever, at full length.
One key, one value. Never three.
Every layer keeps its own cache. Nothing is shared.
num_key_value_heads — not query heads.
Hidden size ÷ attention heads. 128 across Llama 2 and 3.
Tokens actually held, capped by the context limit.
2 for bf16, 1 for an fp8 cache.
The single most common error. On Llama 3.1 8B it is 8, not 32.
8 KV heads on a 4,096‑wide model is 8 × 128 = 1024, not 4096 ÷ 8 = 512.
Every layer keeps its own cache; nothing is shared up the stack.
“How many concurrent users fit on one GPU?” is not trivia. It is a request to watch you reason with the formula. Run these eight steps out loud, in this order, every time — stopping early is always optimistic.
Charges everyone the full 8,192‑token window from the moment they connect.
A typical 2,300‑token conversation holds 0.28 GiB, not 1.00. Paging recovers about 3.6×.
The scheduler interleaves prefill and decode, so not every admitted request is decoding in the same step. Cache bounds how many are resident; the ridge point bounds how many are usefully in one forward pass.
With paging, a user who has sent 500 tokens holds about 32 blocks — not the full window. A 128k window does not mean 128k of memory is reserved for you.
Past the ridge point you are compute‑bound, and adding users raises per‑token latency instead of throughput. Memory tells you what fits; benchmarks tell you what is fast.
Five ways to shrink the per‑user cache. They act on different factors of the same formula, which means they compound — and saying that out loud is worth more in an interview than any one of them.
Grouped‑query attention. Query heads ÷ KV heads.
An fp8 cache. One byte per element instead of two.
A lower context limit. The cheapest lever, and the most visible.
Paged allocation. Blocks, not reservations.
Multi‑head latent attention. The model learns a shared latent.
A fraction of a percent of quality, and nothing at serving time. But no flag changes it — it is baked into the shape of W_K and W_V.
Accuracy error that accumulates over a generation — a key quantised at token 10 is still read at token 10,000. Test on long outputs.
Requests over the limit are rejected. Free in memory terms, but a product decision rather than a tuning knob.
Nothing — but it is an observed saving, not a guaranteed one. It depends on your length distribution.
Extra arithmetic per step, newer, less universally supported. A model‑selection decision made months earlier.
Plot ITL against sequence position. If it rises, it is the cache read.
You have admitted more work than you can hold. The scheduler is preempting.
Cache utilisation pinned at 100% with a growing queue is the signature of thrashing — the system spends its time undoing and redoing rather than progressing. It is easy to miss until users notice.
Write the formula and explain each term.
How many concurrent users fit on one H100?
We want to go from 8k to 128k context. What do you tell me?
Can I change the KV head count with a flag?
Why not cache the queries too, or compress the cache?
Why does the cache make long conversations slower, not just more expensive?
Cache utilisation is 98% and the queue is growing.
QWhy cache K and V but not Q?
Attention is causal, so a past token’s query is never consulted again. Its key and value are read by every future token and never change. Cache what is re‑read.
QDoes each layer keep its own cache?
Yes. Every layer produces different keys and values from a different input, so nothing is shared. That is why layers is a multiplier.
QIs head_dim always 128?
Not by law, but remarkably stable: the 8B, 70B and 405B all land on 128. Models scale by adding heads and layers, not by widening each head. Read it from the config.
QWhy is 8 KV heads so common?
Good point on the quality‑versus‑memory curve, and it maps onto eight‑way tensor parallelism — one head per GPU. If parallelism exceeds the KV head count, heads get replicated and the aggregate cache goes up.
QContext window vs cache?
The window is a limit; the cache is memory in use right now. A 128k window does not allocate 128k. That gap is why worst‑case sizing gives 53 and paged reality gives 191.
QIs quantising the cache riskier than the weights?
Different risk. Weight error is fixed and calibratable. Cache error accumulates across a generation. 8‑bit is generally safe; below that, test on long outputs.
QCan I share cache between users?
Only where token sequences are identical from the very start — a shared system prompt, a common few‑shot block. That is prefix caching: exact‑prefix‑only, and one of the largest levers on chat and agent workloads.
QWhy is it not compressed like a normal cache?
It is not a cache of bytes, it is a tensor that gets multiplied. Any compression must be readable directly by the attention kernel — which is why the real answers are numeric (quantisation) or architectural (latent attention).
QWhat happens when a request finishes?
Its blocks return to the free pool immediately — unless the engine keeps them for prefix reuse, in which case they stay until evicted by LRU and a later matching request skips that prefill.
QOne number to memorise?
2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB per token, and 1.00 GiB per user at 8k. Write that, say what each factor is and where you read it, and you can derive the rest live.
At any real scale you are not serving a model — you are serving a cache. At batch 64 and 8,192 tokens it is 68.7 GB against 16.06 GB of weights: 81% of every byte you move. That is the regime where quantising the weights barely helps, because the weights are no longer the problem.
What is your KV cache utilisation at peak, and is preemption count monitored?
Is the cache bf16 or fp8? If bf16, has anyone tested fp8 on long generations?
What is max-model-len, and what fraction of requests come near it?
Does the capacity number in the design doc assume worst case or paged?
At your typical batch and length, what share of decode bandwidth is cache?
Was the KV head count part of the model-selection criteria?