Track A · Document 02 · From text to numbers
One lookup table, one block repeated thirty-two times, one matrix at the end. Know the shapes and every memory figure in this runbook becomes something you derive rather than remember.
A large language model is less complicated than its reputation. It is one lookup table, one block repeated thirty-two times, and one matrix at the end. The block is always the same shape; only its learned numbers differ from layer to layer.
You need this document for one practical reason: it is what makes every memory figure in the runbook derivable. If you can turn a config file into a parameter count, you can turn it into weight bytes at any precision, into cache per token, and into a defensible answer to “how many users fit on this card”.
Two things to notice. The feed-forward block is four times the size of attention and gets almost none of the discussion. And attention is the only place tokens interact — which is exactly why it is the only place that needs a cache.
There is one vector per token, 4,096 numbers wide, and it runs the entire height of the model. No block ever overwrites it. Each block reads it, works out a small correction, and adds that correction back.
That design is why a 32-layer model trains at all — gradients have an unobstructed path from top to bottom — and it is why every head in every layer writes into a shared space rather than into a private slot. It also gives you the right mental model for depth: not a pipeline of stages, but a shared working surface that ninety-six components (two per layer) take turns amending.
Picture a single shared document being passed down a long table of sixty-four specialists. Nobody may delete anything; each person may only write additions in the margin. Half of them are allowed to glance at everyone else’s copy before they write — those are the attention blocks. The other half must work from their own copy alone — those are the feed-forward blocks. At the end, one clerk reads the marked-up document and names the next word. The document is the residual stream, the specialists are the layers, and the strict no-deleting rule is the residual connection.
Before each half of the block there is a normalisation step. It divides the vector by its own root-mean-square magnitude and then scales it by a learned per-dimension weight.
Attention is the only place in the entire model where one token can see another. That single fact explains why it is the only place that produces a cache, and why the feed-forward block can be sharded, routed or replaced without anything breaking.
The asymmetry in step 2 is the whole mechanism of grouped-query attention, and document 07 is about nothing else. The query side stays full width; only the side that gets cached shrinks.
Each token emits three vectors from the same input. The query is what this token is looking for. The key is what it advertises to others. The value is what it hands over if it is selected. Dot every query against every key to get scores, soften those into weights that sum to one, and take that weighted blend of values. In “the bank of the river”, the query from bank matches the key from river strongly, so river’s value dominates the blend and bank ends up meaning the riverside kind.
The asymmetry that matters for serving: queries are consumed immediately and thrown away, because attention is causal and no future token will ever ask a past token’s question. Keys and values are read by every future token and never change. Cache what is re-read; discard what is not. That is the whole argument, and it is why the cache formula starts with a 2 rather than a 3.
Head 7 writes its 128 numbers into slice 7 of the concatenated output. Without a final projection, head 7 could only ever influence dimensions 896 to 1023 of the residual stream — permanently, in every layer. W_O, at 4096×4096, is what lets each head’s output be spread across the full width and mixed with the others.
The useful way to see it: W_O is 32 stacked blocks of 128×4096, one per head. Each head’s output is projected to full model width independently and the 32 results are summed. Heads therefore write into a shared space, and the next layer’s attention sees the sum rather than 32 labelled slots.
4,096 is a width, not a length. It is how many numbers describe one token. A model 4,096 wide can have a 128,000-token context window; the two are different axes entirely. Head count is width ÷ head dim = 4096 ÷ 128 = 32, so the heads partition the vector — they do not each get a copy of it.
32 heads learn 32 relationships, not 32 meanings. Studied examples include heads that attend to the previous token, heads that track syntactic dependencies, heads that resolve pronouns, and induction heads that spot a repeated pattern and point at what followed it last time. Some heads do very little and can be pruned. Nothing resembling a word reappears until the very top of the stack.
Attention gets the papers and the interview questions. The feed-forward block gets four times the parameters.
If you can only remember one unfashionable fact about transformer anatomy, make it this one: most of a model is the part that processes each token in isolation, and all of the per-user memory cost comes from the smaller part that does not.
Almost every candidate can talk about attention and almost none mention that it is a sixth of the model. Saying “the feed-forward block is 70% of the parameters, so it is where weight quantisation actually pays, and it produces no KV cache at all” does two things at once: it shows you have looked at a config rather than a diagram, and it correctly separates the weight-memory problem from the cache-memory problem, which is the distinction the rest of this runbook runs on.
The serving consequence is what matters here: depth is a multiplier in the cache formula. An 80-layer model costs 2.5× the cache of a 32-layer one at the same KV head count, because every layer keeps its own keys and values.
This is the most useful thing in the document. Seven numbers out of a config file, four multiplications, and you have a figure you can defend under questioning — plus the weight footprint at any precision and the cache per token, both for free.
Change the numbers and watch which ones matter. Raising layers raises both the parameter count and the cache; raising FFN width raises the parameter count and leaves the cache untouched; raising kv heads barely moves the parameter count and moves the cache proportionally. That last row is the entire argument of document 07.
“Embedding is vocab times d_model. Per layer, attention is d_model times q-heads times head-dim, plus twice d_model times kv-heads times head-dim, plus the output projection back to d_model — and the feed-forward is three times d_model times the FFN width, because SwiGLU has a gate, an up and a down. Multiply the layer by the layer count, add the embedding, add the output head if it is not tied. For Llama 3.1 8B that lands on 8,030,261,248 — which is where the 8B comes from.”
Ninety seconds, out loud, with the arithmetic visible. It works on any decoder-only model of this family, and it is a genuine live-interview exercise: being handed a config.json and asked what it costs to serve.
Two practical consequences. Quantising the weights mostly means quantising the feed-forward block, because that is where the bytes are. And on a small model the embedding tables are big enough that vocabulary size is a real memory decision, not a detail.
The standard estimate is two floating-point operations per parameter per token — one multiply and one add. For the 8B that gives 16.06 GFLOP per token. But the embedding table is a lookup, not a matrix multiply, so its 525 million parameters contribute nothing. The honest figure is 2 × (8.03B − 0.53B) = 15.01 GFLOP, about 6% lower.
It rarely changes a decision, but knowing why the rule is slightly wrong is a good signal. On a 405B the gap is under half a percent; on a small model with a large vocabulary it is worth carrying.
After the last block there is one more normalisation and one more matrix. That matrix is the mirror image of the embedding table, and it turns a 4,096-number vector into a score for every word the model knows.
A good detail to have ready: the logits tensor is the largest single activation in the model, and the only reason it is not a problem is that almost all of it is deliberately never computed.
| Question | Answer |
|---|---|
| Is the output head the same matrix as the embedding table? | Sometimes. “Weight tying” reuses one matrix for both and saves 525M parameters on an 8B. Llama 3 8B and 70B do not tie; the smaller Llama 3.2 models do, because on a 1B model the two tables would otherwise be most of the model |
| Why is it 128,256 and not 128,000? | 128,000 learned entries plus 256 reserved slots for special tokens — role headers, end-of-turn, and spares for later fine-tuning. Document 01 covers what those do |
| Where does temperature act? | On the logits, after the head and before the softmax. So do top-k, top-p, repetition penalties and grammar masks. Everything user-facing about sampling happens to this one vector — document 13 |
| Does the model “look up” a fact? | No. The relationship between sunrise and east is distributed across the weights; when those particular numbers flow through, the score for east comes out high. That is all “knowing” means here, and it is why a model cannot tell you where it learned something |
The same anatomy, read three different ways — the three-way sanity check from the map document, applied to the model itself.
| Component | Bytes stored | Bytes moved per decode step | Operations per token |
|---|---|---|---|
| Embedding table 525M params |
1.00 GiB in bf16 | Only the rows you touch — one per token, so effectively nothing | Zero. It is a lookup |
| Attention weights 1.34B params |
2.50 GiB | All of it, every step | 2.68 GFLOP |
| KV cache not parameters at all |
128 KiB per token per user — 1 GiB at 8k | The whole of every active sequence’s cache, every step | Small, but it grows with sequence length while everything else is constant |
| Feed-forward weights 5.64B params |
10.50 GiB | All of it, every step | 11.27 GFLOP |
| Output head 525M params |
1.00 GiB | All of it, every step | 1.05 GFLOP |
| Norms 266k params |
0.5 MiB | Negligible | Negligible |
Column two is why decode is slow: every weight in the model is read to produce one token, plus the cache. Column three is why that read is wasted at small batch: about 15 GFLOP of arithmetic against 16 GB of traffic is roughly one operation per byte, on hardware that wants 295. Document 04 turns that ratio into a number and document 05 turns it into a strategy.
And note the one row that behaves differently. Everything except the KV cache is a fixed cost — the same bytes, the same operations, for every user and every step forever. The cache is the only line that grows, and it grows in two directions at once: with the number of users and with how long each of them has been talking.
ArchitectWalk me through what happens inside one transformer layer.
There is a residual stream — one vector per token, 4,096 wide on an 8B, running the full height of the model. A layer reads it twice and adds to it twice. First: normalise, then attention, then add the result back. Second: normalise again, then the feed-forward block, then add that back. Nothing ever overwrites the stream, which is what makes 32 layers trainable and what lets every head write into a shared space.
The two halves do different jobs. Attention is the only place tokens see each other — and therefore the only thing that produces a KV cache. The feed-forward block processes each token in isolation, produces no cache, and is four times the size: 176 million parameters a layer against attention’s 42 million. That split is worth stating, because it separates the weight-memory problem from the per-user-memory problem, and those two have completely different fixes.
ArchitectHere is a config file. How much memory will this model need?
I would count the parameters rather than trust the name. Embedding is vocab times d_model. Per layer: attention is d_model times q-heads times head-dim, plus twice that for the two smaller KV projections, plus the output projection — and the feed-forward is three times d_model times the FFN width, because SwiGLU has a gate, an up and a down. Multiply by layers, add the embedding and the output head unless they are tied.
For Llama 3.1 8B that gives 8,030,261,248, so 14.96 GiB in bf16. Then I would not stop there, because weights are not the answer to a memory question. On an 80 GB H100 reporting 79.65 GiB, at 0.90 utilisation that is 71.7 GiB, minus 14.96 for weights, minus about 3 for activations and CUDA graphs, leaves 53.7 GiB of KV budget. At 128 KiB a token that is 53 users at the full 8k limit. Weights are one line of an eight-line ladder, and stopping at the first line is the most common mistake in this question.
ArchitectWhy does the KV cache formula not contain d_model or the number of query heads?
Because neither is cached. What gets stored is the output of W_K and W_V, and their width is kv_heads × head_dim — 8 × 128 = 1024 on an 8B, regardless of the fact that the model is 4,096 wide and has 32 query heads. Query vectors are computed, used once and discarded, because attention is causal and no future token consults a past token’s question.
Which is exactly why grouped-query attention works. You can shrink the key-value side by a factor of four without touching the query side, and the diversity that makes heads useful lives in the queries. The trap to avoid on a whiteboard: 8 KV heads gives 8 × 128 = 1024 columns, not 4096 ÷ 8 = 512. Head dim is fixed and the head count multiplies it.
ArchitectWhy 32 layers rather than one much wider one?
Because one round of attention allows exactly one hop of information movement. “The trophy did not fit in the suitcase because it was too big” needs several: first establish the two entities, then compare their sizes, then bind the pronoun. Depth is composition rather than repetition, and you cannot buy it with width.
The honest caveat is worth adding: the layers are identical in shape and nothing assigns them roles. The tendency — surface features early, relationships in the middle, next-token specifics late — emerges from training and is much messier than three clean bands. I would not claim a specific layer does a specific human-legible job.
For serving, the consequence is direct: depth is a multiplier in the cache formula. An 80-layer model costs 2.5× the cache of a 32-layer one at the same KV head count, which is why the 70B is 320 KiB a token against the 8B’s 128.
Eng managerThe team wants to try a model with a much bigger vocabulary. What should they check?
Three things, and only one of them is about quality. First, the memory: the embedding table and the output head are each vocab × d_model, so doubling the vocabulary on an 8B adds about a gigabyte of weights in bf16 — a gigabyte that comes directly out of the KV budget and therefore out of concurrency. On a small model those two tables can be a third of everything.
Second, whether it actually helps their text. Measured on the same samples, going from a 50k to a 100k vocabulary saved 4% on English prose, 31% on Python, and cost 14% more on numeric text. So the answer depends on the traffic mix, and it is a half-hour measurement rather than an argument.
Third, everything token-denominated gets re-measured: cost per request, p95 prompt length, whether prompts fit the window, capacity. I would ask for that measurement before the model change is scheduled, not after.
ArchitectWhat is the largest single activation in the model, and why is it not a problem?
The logits: 128,256 scores, one per vocabulary entry. That is 0.49 MiB per position in fp32. The reason it is not a problem is that prefill deliberately never computes most of them. For an 8,192-token prefill the full logits tensor would be 3.9 GiB, but every position except the last is a prompt token whose identity is already known — so the output head is applied to the last position only.
This is worth knowing for two practical reasons. It is why the logits buffer shows up as part of the activation reserve rather than as a clean formula. And it is why requesting log-probabilities for every prompt token is genuinely expensive rather than free: you are asking the server to materialise the thing it was carefully avoiding.
Eng managerSomeone proposes pruning attention heads to save memory. How do you evaluate that?
I would start by checking the size of the prize, because the framing is usually wrong. Attention is 17% of an 8B model’s parameters; the feed-forward block is 70%. Pruning heads is work in the smaller half. And it does not touch the KV cache at all unless you prune key-value heads, which is a different and much harder operation because it changes the cached tensor shape and needs retraining to recover.
So the question back is: which memory problem are we solving? If it is weights, quantisation gives 2× to 4× for a day of work against pruning’s uncertain single-digit percentage. If it is the per-user cache, the levers are an fp8 cache, a shorter context limit, or a model with fewer KV heads — and none of those is a research project. I would want that comparison written down before anyone spends a sprint on pruning.
What is d_model, exactly?
The number of values that describe one token at every point inside the model — 4,096 for Llama 3.1 8B, 8,192 for the 70B. It is a width, not a length, and it has nothing to do with context. The heads partition it: 4096 ÷ 128 = 32 heads, each owning a 128-number slice.
Is head_dim always 128?
Not by law, but it is remarkably stable. Llama 3.1 8B is 4,096 wide with 32 heads, the 70B is 8,192 wide with 64, and the 405B is 16,384 wide with 128 — all land on 128. Models scale by adding heads and layers rather than by widening each head. Read it from the config rather than assuming, because a few families differ.
Why is the FFN width 14,336 and not 16,384?
Because SwiGLU has three matrices where the classic feed-forward had two. The old design was 2 × d × 4d = 8d²; keeping that parameter count with three matrices needs a width of 8/3 × d, which for 4,096 is 10,923. Llama then applies a 1.3 multiplier and rounds up to a multiple of 1,024, giving exactly 14,336. You can reproduce the config value from the heuristic, which is a nice thing to be able to do.
Do all layers have the same weights?
The same shape, different numbers. Every one of the 32 blocks has its own 218 million parameters, learned independently. A few research architectures share weights across layers to save memory; no mainstream open model does, and you should assume they are distinct unless told otherwise.
Where is the model’s knowledge stored?
Distributed across the weights, with the feed-forward blocks carrying most of it — they hold 70% of the parameters and there is research treating them as key-value memories. But nothing is localised in a way you can point at. That is why a model cannot cite a source for a fact it produced, and why removing a specific fact is an open research problem rather than an edit.
Why does the residual connection matter for inference?
Mostly it does not — it is a training device. The one inference-facing consequence is that the residual stream is the thing that stays in a stable numeric range, which is what makes activations quantisable and what RMSNorm exists to protect. If you ever see an activation-outlier discussion in a quantisation paper, it is about this stream.
Does the embedding table get read on every decode step?
Only one row per token — 8 KiB out of a 1 GiB table — so it is negligible as a read. The output head is the opposite: the full 4096×128256 matrix is multiplied on every single step, which is 1.05 GFLOP and about 7% of the forward pass. The two tables are the same size and have completely different runtime profiles.
Why 2 FLOPs per parameter?
A matrix multiply does one multiply and one add per weight, so the forward pass is about 2N operations per token. The refinement: the embedding is a lookup rather than a multiply, so the honest figure is 2 × (N − embedding) — 15.01 GFLOP rather than 16.06 for the 8B, about 6% lower. Training is roughly 6N, because backward costs about twice forward.
What is weight tying and should I care?
Using the same matrix for the input embedding and the output head, saving vocab × d_model parameters. It matters most where that table is a big fraction of the model: Llama 3.2’s 1B and 3B tie, Llama 3 8B and 70B do not. When counting parameters from a config, check tie_word_embeddings — getting it wrong on a small model throws your total out by 10% or more.
Could I just remove layers to make the model smaller?
People do — it is called depth pruning, and it works better than you would expect, particularly on middle layers. But it is a training project: quality drops and needs recovery fine-tuning. Compared with quantisation, which gives a reliable 2× or 4× in an afternoon with published methods, layer pruning is rarely the right first move in a serving context.
“A token id indexes an embedding table and becomes a vector 4,096 numbers wide. That vector travels up a residual stream through 32 identical blocks; each block normalises, runs attention, adds the result back, normalises again, runs a feed-forward block, and adds that back too. Attention is the only place tokens see each other, which is why it is the only thing that produces a KV cache — and it is only 17% of the parameters. The feed-forward block is 70%, processes each token alone and caches nothing. At the top, one more matrix turns the vector into 128,256 scores. You can count all of this from the config file: for Llama 3.1 8B it comes to 8,030,261,248 parameters, which is 14.96 GiB in bf16, and that is the first line of the capacity ladder rather than the answer to it.”
| Thread started here | Picked up in |
|---|---|
| The embedding lookup that a token id resolves to | 01 · Tokenisation |
| These stages placed on a clock, for one real request | 03 · Journey of a token |
| Why 15 GFLOP against 16 GB of traffic is the wrong ratio for the hardware | 04 · The GPU and the roofline |
| Reading the whole model per token, and the batch that fixes it | 05 · Prefill and decode |
| The cached K and V, and the formula in full | 06 · The KV cache |
| Shrinking the key-value side without touching the query side | 07 · Attention variants |
| Quantising the 70% where the bytes actually are | 09 · Precision |
| Replacing the feed-forward block with many of them | 11 · Many GPUs and MoE |
| What happens to the logits on the way out | 13 · The decode loop |