Track C · Document 07 · Shrinking the footprint
The biggest term in the cache formula, and the one lever you cannot turn at serving time — so it becomes a model-selection decision worth a factor of four.
Document 06 gave you the formula and showed that kv_heads is the biggest lever in it. This document is about that one term.
Multi-head, multi-query and grouped-query attention are not three techniques. They are one dial at three positions, and the dial is: how many key-value heads exist for the query heads to share?
One dial, three settings, and the only thing that changes is how many key-value heads exist for the query heads to share. W_Q and W_O are full width in all three. If someone tells you GQA shrinks the query side, they have misunderstood it — and that asymmetry is the entire mechanism.
8 KV heads on a 4,096-wide model gives 8 × 128 = 1024 columns. It is not 4096 ÷ 8 = 512.
Head dimension is fixed at 128; you multiply it by the head count. Per head, a projection matrix is always d_model × 128, and only the number of heads changes between the three settings. Get this wrong on a whiteboard and the rest of your answer collapses, because every downstream figure is built on it.
A reading room with a shared reference shelf. Thirty-two researchers each have their own question — those are the query heads, and they stay thirty-two in every version. Under multi-head attention each researcher gets a private copy of the reference collection: no contention, enormous storage. Under multi-query there is one copy for all thirty-two: minimal storage, but every question has to be answered from the same narrow selection of books, and some questions are badly served. Grouped-query gives one copy per four researchers. The questions stay as diverse as ever; what is shared is the material being looked at, and it turns out the material was highly redundant across copies anyway.
Turn the dial and watch which numbers move. The two that stay completely still are the whole point.
Watch the top two rows stay completely still while the bottom four move. That is grouped-query attention in one picture: the diversity that makes heads useful lives in the queries, which are untouched; what is shared is the material being looked at.
Two reasons, and the second is the more interesting one.
Keys and values are redundant across heads. Many heads end up looking at similar material, so sharing the redundant part costs little. Queries stay genuinely diverse, and they are untouched — every one of the 32 query heads still computes its own scores and its own output.
A model trained with GQA from the start adapts around it. The query heads never learn to depend on 32 distinct key spaces in the first place. This is different from retrofitting, which is section 4 and does cost something to recover.
Bigger batches. The big one. Fourfold smaller cache means roughly fourfold more concurrent users on the same card. Lead with this.
Faster decode, indirectly. Decode is bandwidth-bound and the cache is part of what gets read each step, so a smaller cache genuinely moves fewer bytes. This is a consequence of the memory saving, not a separate speedup — say it in that order.
Fewer parameters. On the 8B, attention parameters per layer drop from 67M to 42M. Real, but small next to the cache saving and about 4% of the model. Mention it third, never first.
The curve is convex, so almost all the benefit arrives early and almost all the cost arrives late. That shape — not any individual number — is why grouped-query attention won and multi-query did not.
You have an MHA checkpoint and you want GQA without paying for pre-training again. The GQA paper’s contribution is a recipe for exactly this, and it is a favourite follow-up question because it separates people who read the paper from people who read a summary.
Two things worth adding if pushed. The same recipe collapses all the way to MQA — GQA is presented as a generalisation with the KV head count as the free parameter. And uptraining is why you occasionally see two variants of one base model shipped: the original MHA weights and a GQA-converted serving checkpoint.
“Just delete the extra heads.” Wrong, and it sounds it. You would be throwing away three-quarters of the learned key and value information rather than combining it. Mean pooling was tested against exactly this — selecting one head per group — and beat it.
“It is basically free.” Five per cent of a frontier pre-training run is still a serious job with a serious bill. And note the framing: GQA is normally an architecture decision made before training. Uptraining is the exception for checkpoints that already exist, not the standard path.
GQA reduces how many things you store. Multi-head latent attention keeps all of them and reduces how big each one is. Same goal, opposite direction.
If you can mention the RoPE split, it is a strong signal you read the paper rather than a summary. And be measured about the rest: MLA is a pre-training decision like GQA, support is newer, and you cannot switch a GQA model to it with a flag.
MLA is an architectural decision made before training, exactly like GQA. You cannot switch a GQA model to MLA with a flag — there is published research on converting models, but it involves fine-tuning and is an active research direction rather than routine practice.
Support is also newer. Before assuming the headline numbers transfer, check that your serving stack implements the MLA kernels properly rather than falling back to a generic path, because a naive implementation reconstructs the keys and values into something the same size as MHA and you lose the entire benefit.
This belongs on the ladder because it is the only setting that changes the shape of the cost rather than its constant. GQA, MQA and MLA all leave the cache linear in sequence length; a sliding window makes it flat.
If you can walk down this and explain each step, you have covered the whole attention-memory story.
The two middle rungs are not comparable like for like — the sliding-window and latent bars are illustrative, because their savings depend on context length and on a different architecture respectively. The three head-count rungs are exact.
| Setting | The idea | Cache cost | What it costs you | Seen in |
|---|---|---|---|---|
| MHA 2017 | Every query head has its own KV head | 512 KiB | Nothing in quality; everything in memory | Original Transformer, GPT-2, Llama 2 7B/13B |
| MQA 2019 | All query heads share one KV head | 16 KiB | A real, measurable quality drop, and training instability | PaLM |
| GQA 2023 | Query heads share KV heads in groups — the middle setting | 128 KiB | A fraction of a percent of quality | Llama 2 70B, all of Llama 3, Mistral, nearly everything since |
| SWA 2023 | Bound the cache at a fixed window rather than shrinking each entry | flat | Genuinely lossy beyond the window; awkward with prefix caching | Mistral 7B v0.1; Gemma 2 alternates it with global layers |
| MLA 2024 | Keep all heads, cache a compressed latent, reconstruct on the fly | ~7% | Extra arithmetic per step; newer, less universally supported | DeepSeek-V2 and later |
“For serving, I do not choose this — I inherit it by choosing the model, because it is baked into the weights. So it becomes a model-selection criterion: I would read num_key_value_heads from the config before anything else, because on an otherwise equal comparison a model with 8 KV heads serves four times the users of one with 32. If long context is central to the product I would look seriously at an MLA model, and if exact long-range recall is not critical I would consider a sliding-window or hybrid design for the bounded cache. What I would not do is treat any of this as a serving-time tuning knob, because none of it is.”
ArchitectWhat problem does GQA actually solve?
KV cache memory. In multi-head attention every query head has its own key and value head, so the cache scales with the full head count. GQA shares one KV head across a group of query heads — Llama 3.1 8B has 32 query heads over 8 KV heads, so the cache is four times smaller: 128 KiB per token instead of 512.
That memory is what caps batch size, and decode throughput is a function of batch size, so it translates directly into concurrent users: 53 on an H100 instead of 13. The quality cost is a fraction of a percent, because keys and values are highly redundant across heads while queries stay genuinely diverse — and a model trained with GQA from the start never learns to depend on 32 distinct key spaces.
One thing not to say: that GQA speeds up attention. Every query head still computes its own scores, so the operation count is essentially unchanged. There is a real speed benefit, but it is downstream — decode is bandwidth-bound, so reading a smaller cache moves fewer bytes. Saying it in that order reads as understanding rather than a memorised line.
ArchitectHow does GQA differ from MQA, and why did GQA win?
They are the same dial at different settings. MQA collapses to a single KV head shared by every query head — maximum saving, but a measurable quality drop, particularly on reasoning, and the paper notes training instability too. GQA keeps an intermediate number.
The reason it won is the shape of the curve. Going from 32 KV heads to 8 removes 75% of the cache. Going the rest of the way to 1 removes only another 22 points of the original — and costs most of the quality. The curve is convex, so almost all the benefit arrives early and almost all the cost arrives late. GQA sits at the knee.
Worth adding that GQA is presented as a generalisation of MQA, with MHA and MQA as the two endpoints of one continuum rather than three separate techniques. Framing it that way is what a systems person sounds like. And MQA is not dead — PaLM shipped it — it just lost for open models.
ArchitectCan you convert an existing MHA model to GQA?
Yes, and the GQA paper is the recipe. Mean-pool the key and value projection matrices within each group down to a single head, then continue pre-training on roughly five per cent of the original compute so the query heads re-adapt. They demonstrated it on public T5 checkpoints and reported quality close to the original with decode speed comparable to MQA.
Mean pooling was the best of the constructions they tried — it beat picking one head from each group and beat reinitialising the shared head from scratch. Later work has explored weighting the pool by activation magnitude for a small extra gain.
Two things not to say. “Just delete the extra heads” — that throws away three-quarters of the learned information and was explicitly tested against. And do not call it free: five per cent of a frontier pre-training run is a serious job. This is the exception for checkpoints that already exist, not the normal path — GQA is usually an architecture decision made before training.
ArchitectHow is MLA different from GQA, and why did DeepSeek choose it?
GQA reduces the number of key-value heads that get cached. MLA keeps all the heads but projects the keys and values jointly down into one small latent vector, caches that, and projects back up when attention needs them. Fewer things stored versus smaller things stored — same goal, opposite directions.
The trade is extra arithmetic at every step in exchange for a much smaller cache, and since decode is bandwidth-bound rather than compute-bound, that is a very favourable direction. DeepSeek-V2 reported a 93.3% KV reduction against their previous dense model and 5.76× maximum generation throughput.
They chose it because their ablation showed MLA outperforming plain multi-head attention on quality, while GQA sits slightly below it — so they got a much smaller cache without the small concession GQA asks for. Worth the added complexity at their scale. If pushed further, the detail that shows you read the paper is the RoPE complication: keys are rotated for position before caching, and you cannot cleanly compress a rotated key, so they split it into a compressed content part and a small separate part carrying the rotation.
Eng managerTwo models look equivalent on benchmarks. How do you choose between them for serving?
I would open both config files before looking at another benchmark, because the serving economics can differ by a factor of four for models that score identically.
The first number is num_key_value_heads. A model with 8 KV heads serves four times the users of one with 32 on the same card, at the same quality score. The second is layers, because that is also a multiplier in the cache formula. Third, whether it is MLA or uses a sliding window, which changes the shape of the cost rather than its constant. Fourth, vocabulary size, because the embedding and output tables come out of the same memory budget.
Then I would turn that into the number the business understands: users per GPU, and therefore cost per million tokens. Two models that tie on a leaderboard can differ by 3–4× on that, and it is a half-hour calculation. I would want it in the model-selection document alongside the quality scores, because once the choice is made it is expensive to revisit — the KV head count is baked into the weights and no flag changes it.
ArchitectWhy is 8 KV heads so common?
Two reasons. It sits at a good point on the quality-memory curve — most of the saving, almost none of the cost. And it maps cleanly onto eight-way tensor parallelism: on a standard 8-GPU node each card owns exactly one KV head, so no head has to be split or duplicated across devices.
The corollary is worth raising unprompted, because it catches people out. If your parallelism degree exceeds the KV head count — sixteen-way tensor parallelism on a model with 8 KV heads — the heads have to be replicated across devices, and the aggregate cache goes up rather than down. So the KV head count quietly constrains how far you can usefully shard, which is a real consideration for very large models.
ArchitectDoes GQA hurt long-context quality more than short?
It is a fair worry — fewer distinct key spaces could plausibly mean coarser retrieval over a long context. What I would say is what the evidence supports and no more: the published result is quality close to MHA overall, and every frontier long-context open model uses GQA, which they would not if it broke retrieval at length.
If someone wanted a stronger claim than that, I would say I would want to see the evaluation rather than assert it — specifically a needle-in-a-haystack style test across the full window, comparing a GQA model against an MHA one of the same family. That is a measurable question and I would rather measure it than have an opinion about it.
Are query heads reduced too?
No, and this is the crux. W_Q stays full width in MHA, MQA and GQA alike, and so does W_O. The saving comes entirely from the key-value side, which is the only side that gets cached. Anyone who says GQA shrinks the query side has misunderstood the mechanism.
How is one shared KV head used by four query heads, mechanically?
Each of the four query heads independently dots its own query against the same key tensor, and blends the same value tensor with its own softmax weights. Four different sets of scores, four different outputs, one shared source of keys and values. Implementations broadcast the KV head across its group rather than physically duplicating it.
Does GQA reduce model parameters?
Slightly. On the 8B, attention parameters per layer fall from about 67M to about 42M because W_K and W_V shrink — roughly 4% of the whole model. Real, but an order of magnitude less interesting than the cache saving. Mention it second or third, never first.
Can I change a served model’s KV head count with a flag?
No. It is baked into the weights — the shape of W_K and W_V. Changing it means uptraining and producing a new checkpoint. What is a runtime flag is the cache dtype: fp8 KV is a serving option and it halves the same number, stacking multiplicatively with whatever head ratio the model already has.
Does GQA interact with tensor parallelism?
Yes, and it is a good detail to raise. With 8 KV heads and 8-way tensor parallelism, each GPU holds exactly one KV head — clean. If the parallelism degree exceeds the KV head count you must replicate heads across devices, which costs the memory back. So the head count quietly caps how far you can usefully shard.
Is “GQA gives an 8× saving” correct?
Only for models where it happens to be true. It gives query heads divided by KV heads — 4× on Llama 3.1 8B and 8× on the 70B, within the same family. Derive it from the config every time; quoting a single ratio as a general fact is wrong roughly half the time.
What is sliding-window attention doing differently?
It bounds the cache instead of shrinking each entry. Each token attends only to the last W tokens, so the cache stops growing once it reaches W — the cost becomes flat in context length rather than linear. That is a different shape of saving, and it is genuinely lossy: a token outside the window is not attended to at all. Gemma 2’s alternating design is the interesting compromise.
Could a model use different KV head counts per layer?
Nothing forbids it, and hybrid designs are an active area — Gemma 2’s alternation of local and global layers is exactly this idea applied to the attention pattern rather than the head count. Mainstream open models keep it uniform, which is why the cache formula has a single layers multiplier rather than a sum.
Is MLA going to replace GQA?
Unknown, and the honest answer is that it depends on whether the quality result replicates across labs and on tooling support catching up. What is clear is that it is a pre-training decision either way, so it is not something a serving team chooses — it is something they inherit. Track it as a model-selection input rather than a migration plan.
What about MoE — does it help the cache?
Not at all. Mixture of experts replaces the feed-forward block, and attention — and therefore the KV cache — is completely untouched. It cuts operations per token and saves no memory whatsoever, because every expert must stay resident. MoE and GQA solve orthogonal problems and are frequently used together in the same model.
“Multi-head, multi-query and grouped-query attention are one dial at three settings: how many key-value heads the query heads share. Only the KV side shrinks — the query projection and the output projection stay full width in all three, which is the whole mechanism, because queries are where the diversity lives and keys and values are highly redundant across heads. The saving is query heads divided by KV heads, so four times on Llama 3.1 8B and eight on the 70B. GQA won because the curve is convex: going from 32 heads to 8 removes three-quarters of the cache, and going the rest of the way to one removes only another fifth while costing most of the quality. You can retrofit it onto an MHA checkpoint by mean-pooling the projections within each group and continuing pre-training on about five per cent of the original compute. And there is a fourth setting: latent attention keeps every head but caches a compressed vector instead, which DeepSeek reported at over 90% reduction and beating plain multi-head on quality.”
| Thread started here | Picked up in |
|---|---|
| The four projection matrices and where they sit in the block | 02 · Inside the model |
| The cache read as a share of decode bandwidth, which this halves or quarters | 05 · Prefill and decode |
| The formula this term sits in, and the ladder to a user count | 06 · The KV cache |
| The RoPE rotation that complicates latent attention | 08 · Position and long context |
| The fp8 cache that stacks multiplicatively on top of this | 10 · Paging and prefix reuse |
| Why the KV head count caps the useful tensor-parallel degree | 11 · Many GPUs |
| Reading the config as a model-selection step before you buy anything | 14 · Capacity planning |