Track C · Document 08 · Shrinking the footprint
How order gets into a mechanism that has none, why a longer window is a model change rather than a setting, and what 128k actually costs in four places at once.
Here is something surprising about the machinery in document 02, and it is the reason this document exists.
The older approach added a position vector to each token at the very bottom of the stack. It worked, and it has a fatal flaw for our purposes: the model learns those specific patterns, and beyond the longest one it ever saw in training it has nothing at all.
A pile of index cards on a table. Each card carries a word and everything the model knows about it. Attention is a researcher who can compare any card to any other card, in any order, as often as they like — and the pile is genuinely unordered. Somebody has to write a position on each card before it goes in the pile. The clever part of RoPE is that instead of writing an absolute page number, it tilts each card by an angle proportional to where it belongs. Comparing two tilted cards automatically tells you the angle between them, which is the gap. Absolute marks on each card, relative information out of every comparison.
Rotary position embedding, used by essentially every modern open model, does not add anything. It rotates the query and key vectors by an angle proportional to the token’s position.
One caution about a claim you will see: “positions 1 and 3 are 90° apart” is only true for the one frequency band where the per-position angle happens to be 45°. There are 64 bands, all rotating at different rates — and that multi-rate design is precisely what makes context extension possible at all.
Three things flow directly from it. First, the rotation is applied to a key before that key goes into the cache, so what is stored is already position-baked — which is why compressing the cache collides with RoPE and why MLA needed a special split. Second, the choice of scheme decides whether the window can be extended later, which is a capacity and roadmap question. Third, the multi-rate design is what makes extension possible at all, and understanding why is what separates the three methods in section 4.
This is why Llama 3 raised the base from 10,000 to 500,000 and it is a one-line calculation you can do in an interview. At base 10,000 the slowest band wraps after about 54,000 positions; at 500,000 it wraps after 2.56 million. Raising the base is the cheapest part of reaching long context — and on its own it is not enough.
Suppose a model was trained on sequences up to 8,000 tokens. Feed it 30,000 and quality collapses — often badly, and not gracefully.
This is why --max-model-len 131072 on a model trained to 8k does not give you a long-context model. It gives you a model that accepts long prompts and answers them badly, which is considerably worse than rejecting them.
Setting --max-model-len 131072 on a model that declares a 131,072 window is fine: the model was trained or adapted for it. Setting it on a model trained to 8k is not, and the server will happily let you. You get a system that accepts long prompts and answers them badly, which is worse than one that rejects them — because the failure is silent and looks like a model-quality problem rather than a configuration one.
The check: read max_position_embeddings and any rope_scaling block in the config, and do not exceed what they declare.
Three ideas, each building on the last, and the differences between them are exactly about the frequency bands from section 2.
Three ideas, each building on the last, and all three are training-time changes. The thing to be able to say is why they differ: interpolation treats every frequency band identically and blurs neighbours; base change and YaRN recognise that the bands were never doing the same job.
| Method | Idea | Cost | Why it is better or worse than the one above |
|---|---|---|---|
| Position interpolation | Scale all positions down to fit the trained range | Blurs nearby positions; needs fine-tuning | The baseline. Treats every frequency band identically, which is precisely the problem |
| NTK-aware / base change | Raise the base so slow bands stretch further while fast ones are left alone | Better local detail; still needs tuning | Recognises that the bands were never doing the same job. Fast bands handle neighbours and do not need stretching |
| YaRN | Band-by-band treatment plus an attention-scaling correction | Far cheaper to train: ~10× fewer tokens and ~2.5× fewer steps than plain interpolation | Makes the band distinction explicit — interpolate the wrapped bands, leave the unwrapped ones, blend in between |
| What ships | Llama 3 raised rope_theta to 500,000 and declares a frequency-banded scaling config | Both, together | Raising the base pushed the slowest wavelength from 54,000 to 2.56 million positions. That is the cheap part; the scaling config and the training are the rest |
Everything above is about making the model able to use a long window. This is what it costs once it can — and the two are frequently confused.
Drag to 131,072. Cache per user goes to 16 GiB, concurrency falls to three, prefill passes 28 seconds, and the cache becomes over half of every decode read. Four costs, one slider, and none of them is the quality question — which is separate and also unresolved.
Grouped-query attention cuts per-token cache by the head ratio — 4× on the 8B, 8× on the 70B. Without it a 128k request on a 70B would need 320 GiB.
An fp8 cache halves it again, taking that 40 GiB to 20.
FlashAttention means the sequence-by-sequence score matrix is never materialised. At 120,000 tokens that matrix would be 28.8 GB per head, which is not a tuning problem but an impossibility.
Paging and prefix reuse mean nobody pays for tokens they have not sent, and a long fixed context can be computed once and reused across turns rather than re-prefilled.
All four at once. Long context is not a bigger number in a config file; it is every lever in the runbook applied simultaneously.
A long window is a capacity claim, not a quality claim. Retrieval from the middle of a very long context is a known weak spot across models, and an advertised 128k does not guarantee the model uses the middle of it well.
If asked whether you would trust a long-context claim, the answer is that you would test retrieval across the full range — place a specific fact at 10%, 25%, 50%, 75% and 90% of the window and measure whether it comes back — rather than take the number at face value. That test takes an afternoon and it is the difference between a claim and a measurement.
ArchitectWhat is RoPE, in one breath?
Rotary position embedding. Instead of adding a position vector, it rotates each token’s query and key by an angle proportional to its position. Because rotations compose, the dot product between any two tokens ends up depending only on the distance between them — so you get relative positions out of an absolute operation, with no extra parameters and no bookkeeping.
Two details that get asked. It is applied to queries and keys only, never values — position decides what to attend to, not what gets handed over. And it happens inside every layer at the point Q and K are formed, not once at the bottom like the older additive scheme.
The part worth adding is that the vector is split into pairs, each rotating at its own rate — 64 dials for a 128-dimensional head. Fast dials distinguish neighbours, slow ones separate tokens thousands apart. That multi-rate design is what makes context extension possible, and it is why the methods for extending differ in how they treat the bands.
ArchitectWhy can I not just set a bigger context length?
Because positions past the training length produce rotation angles the model has never seen, and every relationship it learned was calibrated against the angles it did see. Beyond that range it is extrapolating into territory where its learned patterns do not apply, and quality falls off sharply rather than gracefully.
Extending the window means rescaling the rotations and fine-tuning — position interpolation, an NTK-style base change, or YaRN — so it is a model change, not a config change. The practical trap is that the server will let you set the flag anyway: you end up with a system that accepts long prompts and answers them badly, which is worse than one that rejects them, because the failure looks like a model-quality problem rather than a configuration error.
ArchitectWhy did Llama 3 change rope_theta from 10,000 to 500,000?
To move the slowest frequency band out past the window it wanted to support. The angle for band i is base to the power of minus 2i over d, so the slowest band’s wavelength is roughly 2π times the base. At 10,000 that is about 54,000 positions; at 500,000 it is about 2.56 million.
Why that matters: once a band’s wavelength is shorter than the context, it has wrapped round and can no longer distinguish absolute distance — two tokens 60,000 apart look the same as two tokens 6,000 apart on that dial. With base 500,000, a 128k window sits comfortably inside the range where even the slowest band is still monotonic.
Worth being precise about what that alone does not achieve: raising the base is the cheap part. Llama 3.1 also ships a frequency-banded scaling config and was trained for the longer window. The base change makes long context representable; the training makes it work.
Eng managerProduct wants 128k context. Walk me through what you would tell them.
Three separate things, and they usually only mean one of them.
First, can the model do it? That is a model-selection question: read max_position_embeddings and the scaling config. If the model was not trained or adapted for it, we cannot buy it with a flag — it is a fine-tuning project.
Second, what does it cost to serve? On the 8B, cache per user goes from 1 GiB to 16, so worst-case concurrency falls from 53 to three on one H100. Prefill at 128k is deep in the quadratic regime, so time to first token goes to tens of seconds without prefix reuse. That is roughly an order of magnitude more hardware for the same concurrency.
Third, and this is the one I would push hardest on: what fraction of requests actually needs it? If it is two per cent, raising the global limit sizes every short request against the worst case. A separate deployment for long context with its own admission limits and its own SLO is usually far cheaper and lets us say yes. And I would want a retrieval test across the full window before we advertise the number, because a long window is a capacity claim and not a quality one.
ArchitectHow does RoPE interact with the KV cache?
The rotation is applied to a key before that key is written to the cache, so what is stored is already rotated for its position. When a new token’s query arrives — rotated for its own position — the comparison automatically reflects the gap between them. Nothing needs recomputing and nothing needs re-rotating as the sequence grows.
The consequence worth knowing is what it does to cache compression. A rotated key cannot be cleanly squeezed and unsqueezed, because the rotation is entangled with the content. That is exactly the problem DeepSeek hit with multi-head latent attention, and their solution was to split the key into a compressed content part and a small separate part carrying the rotation. If someone asks how MLA and RoPE fit together, that split is the answer.
ArchitectWhat is the difference between position interpolation and YaRN?
They differ in whether they treat the frequency bands as interchangeable. Position interpolation scales every position down by the same factor so they fit inside the trained range — like redrawing eight ruler marks across thirty centimetres. It works, and it blurs distinctions between neighbouring tokens, because the fast bands that were doing that job got squeezed along with everything else.
YaRN recognises that the bands were never doing the same job. Bands whose wavelength is already shorter than the context have wrapped and can safely be interpolated; bands that have not wrapped are left alone; in between there is a graded blend. It also adds a correction to the attention scaling. The published result is roughly ten times fewer tokens and two and a half times fewer training steps than plain interpolation to reach the same window.
Both need fine-tuning. Neither is a runtime setting, and I would be sceptical of anyone who implies otherwise.
Eng managerA vendor claims a one-million-token window. How do you evaluate that?
Two tests, and they answer different questions.
The capacity test is arithmetic and takes ten minutes: cache per token from the config, times one million, tells me what a single request costs. On a 70B with GQA that is 320 GiB — four H100s of cache for one user. So the first question is what they are charging and whether the economics can possibly work, or whether the window is available in principle and rationed in practice.
The quality test takes an afternoon and matters more. Place a specific, checkable fact at 10%, 25%, 50%, 75% and 90% of the window and measure whether it comes back — then do it again with several facts that need combining, because single-fact retrieval is the easy case. Retrieval from the middle of a very long context is a known weak spot, so I would treat the advertised window as a capacity claim until that test passes.
And I would ask what it was trained for rather than what it accepts. Those are different numbers and only one of them is in the marketing.
Is RoPE applied to values too?
No — queries and keys only. Position decides which tokens to attend to; the content being handed over when a token is selected does not need rotating. And it is applied inside every layer at the point Q and K are formed, not once at the bottom of the stack.
Does a longer window mean the model uses it well?
Not necessarily, and this is the most important caveat in the document. Retrieval quality from the middle of a very long context is a known weak spot. Treat an advertised window as a capacity claim and test retrieval across its full range before relying on it.
Are “positions 1 and 3 are 90° apart”-style claims right?
Only for the one frequency band where the per-position angle happens to be 45°. There are 64 bands in a 128-dimensional head, all rotating at different rates. Stated as a general fact it makes RoPE look like a single rotation, and then nothing about band-aware context extension makes any sense.
What is the serving cost of long context, in one line?
Linear in tokens for the cache, quadratic in tokens for prefill. A 70B with GQA costs 320 KiB per token, so 128k is about 40 GiB — for one user. That is why long context in practice means GQA or MLA, plus a quantised cache, plus prefix reuse, all together.
Can I convert a short-context model to long context myself?
Technically yes — the methods are published and the fine-tuning is far cheaper than pre-training, especially with YaRN. Practically it is a training project with an evaluation burden, and unless long context is core to your product you are almost always better off selecting a model that already declares the window you need.
Why does the fastest frequency band wrap almost immediately?
Because its job is to distinguish adjacent tokens, and for that a wavelength of about six positions is exactly right. It is supposed to wrap. The band that must not wrap is the slowest one, because that is what carries long-range absolute distance — which is why the base, which sets the slowest wavelength, is the number that changed for long-context models.
Do other position schemes exist?
Yes — learned absolute embeddings in older models, relative position biases in the T5 family, ALiBi which adds a distance-proportional penalty to attention scores. RoPE won for decoder-only models because it gives relative positions for free, composes cleanly with the KV cache, and extends reasonably well with the right rescaling. You should be able to name the alternatives and say why RoPE displaced them.
Does sliding-window attention solve the position problem?
No — it is a different axis. A sliding window bounds how far back a token can attend, which bounds the cache; RoPE is about encoding where a token sits. They compose: Mistral 7B v0.1 used RoPE and a 4,096-token window. Bounding attention does sidestep the extrapolation problem for positions beyond the window, but at the cost of not attending to them at all.
Why does the cache complicate context extension?
Because keys are rotated before being cached, so the stored tensor already carries its position. Any rescaling of positions has to be consistent with what is already in the cache, and any attempt to compress the cache runs into the fact that content and rotation are entangled. This is exactly the wrinkle MLA had to solve with its split-key design.
What single thing should I check in a config before trusting a window?
max_position_embeddings, then rope_theta, then any rope_scaling block. If the scaling block declares an original_max_position_embeddings well below the headline window, the long window was reached by rescaling plus fine-tuning — which is normal, and it is also the signal to run the retrieval test rather than assume.
“Attention is order-blind — shuffle the words and it computes the same thing — so position has to be injected. RoPE does it by rotating each token’s query and key by an angle proportional to its position; because rotations compose, what survives the dot product is only the distance between two tokens. The vector is split into 64 pairs, each rotating at its own rate, so fast dials separate neighbours and slow ones separate distant tokens. It does not extrapolate: past the training length the angles are unseen and quality collapses, so extending a window means rescaling the rotations and fine-tuning — position interpolation, an NTK-style base change, or YaRN, which is about ten times cheaper to train. Llama 3 also raised the base from 10,000 to 500,000, which pushes the slowest wavelength from 54,000 positions to 2.56 million. And on the serving side, a long window is expensive in four places at once: cache per user, concurrency, prefill’s quadratic term, and the cache’s share of each decode read.”
| Thread started here | Picked up in |
|---|---|
| Where Q and K are formed, and the attention block the rotation sits in | 02 · Inside the model |
| The quadratic prefill term that long prompts run into | 05 · Prefill and decode |
| The cache arithmetic behind the 16 GiB and 40 GiB figures | 06 · The KV cache |
| The split-key trick MLA needed because of the rotation | 07 · Attention variants |
| fp8 caches and prefix reuse, two of the four things that make 128k survivable | 10 · Paging and prefix reuse |
| Running long-context traffic as a separate deployment with its own limits | 15 · Production |