Document 02 · Quick learn Residual stream · Two halves · Count it yourself

One blockthirty-two times

One lookup table, one block repeated thirty-two times, one matrix at the end. Know the shapes and every memory figure in this runbook becomes something you derive rather than remember — including the answer to “how many users fit”.

“Not a pipeline of stages. A shared working surface that ninety‑six components write into.”

8,030,261,248

parameters, derived from the config

70%

of them are feed-forward

17%

attention — the only cache source

14.96 GiB

weights in bf16

The shape of the thingless complicated than its reputation

One lookup table, one block repeated thirty-two times, one matrix at the end. The block is always the same shape; only its learned numbers differ from layer to layer.

Embedding table

128,256 × 4,096. A token id is a row number.

32 identical blocks

Each reads the stream twice and adds to it twice.

Final norm

One more RMSNorm at the top.

Output head

4,096 × 128,256 — the mirror of the embedding.

Logits

One raw score per vocabulary entry.

The residual stream is the idea to hold on to

One vector per token, 4,096 numbers wide, running the entire height of the model. No block ever overwrites it. Each block reads it, works out a small correction, and adds that correction back.

That is why a 32-layer model trains at all — gradients have an unobstructed path top to bottom — and why every head writes into a shared space

The analogy — a shared document down a long table

Sixty-four specialists. Nobody may delete anything; each may only write additions in the margin. Half are allowed to glance at everyone else’s copy before they write — those are the attention blocks. The other half must work from their own copy alone — the feed-forward blocks.

At the end, one clerk reads the marked-up document and names the next word

One blocktwo reads, two additions, never an overwrite

Norm

Divide by root-mean-square, scale by a learned weight.

Attention

The only place tokens see each other.

Add back

Into the residual stream.

Norm again

Same operation, different weights.

Feed-forward

Each token alone. 70% of the parameters.

Add back

And on to the next block.

Thirty-two times — same shape, different learned numbers

RMSNorm: the cheap part, and why it matters anyway

What it does

Makes the magnitude of the vector predictable, so the next block sees inputs in a stable range no matter how much earlier layers added.

What it costs

4,096 per norm, two norms per layer, plus one at the top.

266,240 parameters — 0.003% of an 8B

vs the older LayerNorm

RMSNorm skips the mean subtraction and the bias. Slightly cheaper, empirically just as good.

Which is why modern models all use it

Why you should care at all

It is the reason activations stay in a sane range.

And activation range is exactly what decides whether quantisation works — document 09

Attentionthe only place tokens meet — and the only source of cache

That single fact explains why attention is the only part that produces a cache, and why the feed-forward block can be sharded, routed or replaced without anything breaking.

1 · One token in

A vector 4,096 wide. That is the model dimension.

2 · Three projections

W_Q is 4096×4096. W_K and W_V are 4096×1024.

3 · The trap

8 KV heads × 128 = 1024. Not 4096 ÷ 8 = 512.

4 · Cache K and V only

4 KiB per token per layer. Q is used once and discarded.

5 · W_O back to width

4096×4096, unchanged across MHA, MQA and GQA.

6 · Total

41,943,040 parameters per layer.

Q, K and V in one paragraph

The query is what this token is looking for. The key is what it advertises to others. The value is what it hands over if selected. Dot every query against every key, soften into weights that sum to one, take that weighted blend of values.

In “the bank of the river”, the query from bank matches the key from river strongly — so river’s value dominates and bank ends up meaning the riverside kind

Two things that are not what they look like

4,096 is a width, not a length. It is how many numbers describe one token. A model 4,096 wide can have a 128,000-token window; different axes entirely.

Heads partition the vector, they do not copy it. 4096 ÷ 128 = 32 heads, each owning a 128-number slice. 32 heads learn 32 relationships, not 32 meanings.

Studied examples: heads that attend to the previous token, track syntactic dependencies, resolve pronouns, or spot a repeated pattern and point at what followed it last time

The feed-forward block70% of the model, and nobody asks about it

Attention gets the papers and the interview questions. The feed-forward block gets four times the parameters.

Three matrices, not two

SwiGLU has a gate, an up and a down — so 3 × d × ffn.

Per layer

3 × 4,096 × 14,336.

176,160,768 — four times attention

No cache at all

Each token is processed in isolation.

Which is why it can be sharded, routed or replaced freely

Where 14,336 comes from

The classic FFN was 2 × d × 4d = 8d². Keeping that count with three matrices needs 8/3 × d = 10,923.

Llama applies a 1.3 multiplier and rounds up to a multiple of 1,024 → 14,336 exactly

Why this is worth raising unprompted

Almost every candidate can talk about attention and almost none mention that it is a sixth of the model. Saying “the feed-forward block is 70% of the parameters, so it is where weight quantisation actually pays, and it produces no KV cache at all” does two things at once.

It shows you have looked at a config rather than a diagram — and it correctly separates the weight-memory problem from the cache-memory problem, which is the distinction the rest of this runbook runs on

Count the parameters yourselfseven numbers, four multiplications, a defensible answer

The most useful thing in this document. Turn a config into a parameter count and you get the weight footprint at any precision and the cache per token for free.

total = vocab×d + layers × ( attn + ffn + 2d ) + d + head

attn = d·qh·hd + 2·d·kvh·hd + qh·hd·d · · · ffn = 3·d·ffn_width

Seven numbers straight out of config.json

vocab

128,256

vocab_size

d_model

4,096

hidden_size

layers

num_hidden_layers

q heads

num_attention_heads

kv heads

num_key_value_heads

ffn width

14,336

intermediate_size

Worked525,336,576 + 32 × 218,112,000 + 4,096 + 525,336,576 = 8,030,261,248 × 2 bytes = 14.96 GiBwhich is where “8B” comes from

Where the parameters actually are

Feed-forward · 70%

Attention · 17%

Embedding + head · 13%

5,637,144,576 — no cache, each token alone1,342,177,280 — the only source of KV cache1,050,673,152 — and this falls to 1% on a 405B

Say it in ninety seconds, out loud

“Embedding is vocab times d_model. Per layer, attention is d_model times q-heads times head-dim, plus twice d_model times kv-heads times head-dim, plus the output projection back to d_model — and the feed-forward is three times d_model times the FFN width, because SwiGLU has a gate, an up and a down. Multiply the layer by the layer count, add the embedding, add the output head if it is not tied.”

It reproduces the published figure exactly on the 8B, 70B and 405B — which is what makes it safe to use on a model you have never seen

A refinement to “2N FLOPs per token”

The standard estimate is two operations per parameter — one multiply, one add — giving 16.06 GFLOP for the 8B. But the embedding table is a lookup, not a matrix multiply, so its 525 million parameters contribute nothing.

Honest figure: 2 × (8.03B − 0.53B) = 15.01 GFLOP, about 6% lower. Under half a percent on a 405B; worth carrying on a small model with a big vocabulary

The top of the stackone norm, one matrix, 128,256 scores

The stream, normalised

After all 32 blocks: one vector 4,096 wide.

× the LM head

4096×128256 — 1.05 GFLOP, ~7% of the forward pass.

128,256 raw scores

One per vocabulary entry. 0.49 MiB in fp32, per position.

Last position only

The other positions are prompt — already known.

Is the head the same matrix as the embedding?

Sometimes. Weight tying reuses one matrix for both, saving 525M parameters on an 8B.

Llama 3 8B and 70B do not tie. Llama 3.2’s 1B and 3B do

Why 128,256 and not 128,000?

128,000 learned entries plus 256 reserved slots — role headers, end-of-turn, and spares for later fine-tuning.

Where does temperature act?

On the logits, after the head and before the softmax.

So do top-k, top-p, repetition penalties and grammar masks — document 13

Does the model “look up” a fact?

No. The relationship between sunrise and east is distributed across the weights; nothing is localised in a way you can point at.

The largest activation in the model, and why it is not a problem

For an 8,192-token prefill the full logits tensor would be 3.9 GiB. Prefill deliberately never computes it — the output head is applied to the last position only.

Which is why asking for log-probabilities on every prompt token is genuinely expensive rather than free: you are asking the server to materialise the thing it was carefully avoiding

Answer it out loudthe ninety-second version, then seven questions

“A token id indexes an embedding table and becomes a vector 4,096 numbers wide. That vector travels up a residual stream through 32 identical blocks; each block normalises, runs attention, adds the result back, normalises again, runs a feed-forward block, and adds that back too. Attention is the only place tokens see each other, which is why it is the only thing that produces a KV cache — and it is only 17% of the parameters. The feed-forward block is 70%, processes each token alone and caches nothing. At the top, one more norm and a 4096×128256 matrix turn the vector into a score for every word the model knows. Count it from the config and you get 8,030,261,248 — 14.96 GiB in bf16.”

Seven questions, and the spine of each answer

Architect

Walk me through one transformer layer.

A residual stream — one vector per token, 4,096 wide, running the full height
A layer reads it twice and adds twice: norm → attention → add, norm → FFN → add
Nothing ever overwrites the stream, which is what makes 32 layers trainable

Land the split: attention 42M a layer against feed-forward’s 176M. That separates the weight-memory problem from the per-user-memory problem.

Architect

Here is a config file. How much memory will this model need?

Count the parameters rather than trust the name — then 8,030,261,248 × 2 = 14.96 GiB
Then do not stop, because weights are not the answer to a memory question
79.65 → ×0.90 = 71.7 → −14.96 → −3 → 53.7 GiB → 53 users at 8k

Weights are one line of an eight-line ladder, and stopping at the first line is the most common wrong answer in this field.

Architect

Why does the cache formula not contain d_model or query heads?

Because neither is cached. What is stored is the output of W_K and W_V
Their width is kv_heads × head_dim = 8 × 128 = 1024
Queries are computed, used once and discarded — attention is causal

Which is exactly why GQA works: shrink the key-value side 4× without touching the query side, and the diversity that makes heads useful lives in the queries.

Architect

Why 32 layers rather than one much wider one?

One round of attention allows exactly one hop of information movement
“The trophy did not fit in the suitcase because it was too big” needs several hops
Depth is composition rather than repetition — you cannot buy it with width

Caveat worth adding: nothing assigns layers roles; the early/middle/late tendency emerges from training and is messier than three clean bands. For serving, depth is a multiplier in the cache formula.

Eng manager

The team wants a model with a much bigger vocabulary.

Memory: embedding and head are each vocab × d_model. Doubling adds ~1 GiB of weights on an 8B — straight out of the KV budget
Does it help their text? 50k → 100k saved 4% on prose, 31% on Python, cost 14% more on numeric
Re-measurement: cost per request, p95 prompt length, window fit, capacity

It is a half-hour measurement rather than an argument. Ask for it before the model change is scheduled, not after.

Architect

What is the largest single activation, and why is it not a problem?

The logits — 128,256 scores, 0.49 MiB per position in fp32
A full 8,192-token prefill would be 3.9 GiB
Prefill applies the output head to the last position only

Two consequences: the logits buffer shows up in the activation reserve rather than a clean formula, and log-probs on prompt tokens are genuinely expensive.

Eng manager

Someone proposes pruning attention heads to save memory.

Check the size of the prize first: attention is 17% of the model, feed-forward is 70%
Pruning heads is work in the smaller half
And it does not touch the KV cache unless you prune KV heads — which changes the cached tensor shape and needs retraining

Question back: which memory problem are we solving? Quantisation gives 2–4× in a day against pruning’s uncertain single digits.

Fast answersthe ten that come up most

QWhat is d_model, exactly?

The number of values that describe one token at every point inside the model — 4,096 for the 8B, 8,192 for the 70B. It is a width, not a length, and has nothing to do with context. The heads partition it: 4096 ÷ 128 = 32.

QIs head_dim always 128?

Not by law, but remarkably stable. The 8B (4,096 wide, 32 heads), 70B (8,192, 64) and 405B (16,384, 128) all land on 128. Models scale by adding heads and layers rather than widening each head. Read it from the config.

QWhy is the FFN width 14,336 and not 16,384?

SwiGLU has three matrices where the classic FFN had two. Matching the old 8d² count with three needs 8/3 × d = 10,923; Llama applies a 1.3 multiplier and rounds up to a multiple of 1,024 — exactly 14,336. You can reproduce the config value from the heuristic.

QDo all layers have the same weights?

The same shape, different numbers. Every one of the 32 blocks has its own 218 million parameters, learned independently. A few research architectures share weights across layers; no mainstream open model does.

QWhere is the model’s knowledge stored?

Distributed across the weights, with the feed-forward blocks carrying most of it — they hold 70% and there is research treating them as key-value memories. But nothing is localised. That is why a model cannot cite a source, and why removing a fact is an open research problem.

QWhy does the residual connection matter for inference?

Mostly it does not — it is a training device. The one inference-facing consequence is that the stream stays in a stable numeric range, which is what makes activations quantisable and what RMSNorm exists to protect.

QDoes the embedding table get read on every decode step?

Only one row per token — 8 KiB out of a 1 GiB table — so negligible as a read. The output head is the opposite: the full 4096×128256 matrix is multiplied every step, 1.05 GFLOP and ~7% of the forward pass. Same size, completely different runtime profiles.

QWhy 2 FLOPs per parameter?

A matrix multiply does one multiply and one add per weight. The refinement: the embedding is a lookup, so the honest figure is 2 × (N − embedding) — 15.01 GFLOP rather than 16.06 for the 8B. Training is roughly 6N.

QWhat is weight tying and should I care?

Using the same matrix for the input embedding and the output head, saving vocab × d_model. It matters most where that table is a big fraction of the model. Check tie_word_embeddings — getting it wrong on a small model throws your total out by 10% or more.

QCould I just remove layers to make the model smaller?

People do — depth pruning, and it works better than you would expect on middle layers. But it is a training project: quality drops and needs recovery fine-tuning. Against quantisation’s reliable 2–4× in an afternoon, it is rarely the right first move in serving.

Attention gets the questions; the feed-forward block gets the parameters. Attention is 17% of an 8B and the only thing that produces a KV cache. The feed-forward block is 70%, processes each token alone, and caches nothing. Those are two different memory problems with two different sets of fixes — and conflating them is the single most common error in this whole subject.

Where this goes next

The embedding lookup a token id resolves to 01 · TokensThese stages placed on a clock 03 · JourneyWhy 15 GFLOP against 16 GB is the wrong ratio 04 · HardwareReading the whole model per token 05 · PhasesThe cached K and V, and the formula in full 06 · KV cacheShrinking the KV side, not the query side 07 · AttentionQuantising the 70% where the bytes are 09 · PrecisionReplacing the FFN with many of them 11 · Many GPUsWhat happens to the logits on the way out 13 · Decode loop

Questions to ask them

Which model and precision — and has anyone counted the parameters from the config?

Is the output head tied to the embedding table on the models you serve?

What is the KV head count, and did you choose the model partly for it?

Has anyone measured the activation and workspace reserve, or is it a guess?

Do you ever request log-probabilities on prompt tokens, and do you know what that costs?