Track D · Document 11 · Scale and speed
Splitting a model is a workaround for a memory limit, not a performance feature — and mixture of experts saves compute rather than the thing you actually buy.
This trips up nearly everyone coming from a systems background, where adding servers usually just adds capacity. Here, adding a card adds capacity only if you tell the software how to use it — and the two ways of telling it have completely different costs.
Eight GPUs in a node are eight chefs with eight separate pantries in one kitchen. They can pass ingredients to each other across the bench very quickly — that is NVLink. What they cannot do is treat the eight pantries as one large pantry: a recipe needing more ingredients than any single pantry holds does not simply work. Somebody has to decide, in advance and explicitly, which chef stores what. And chefs in the next building can still be handed things, but only by courier, which changes what kinds of collaboration are possible.
Before any discussion of how to split a model, the question that settles most cases: does it need splitting at all?
This is a genuinely good interview answer because it shows you reach for the simplest thing that works rather than the most impressive-sounding one. Parallelism exists because the model does not fit, not to make a single request faster.
Zero communication. No collectives, no synchronisation, nothing to tune. Each GPU runs at the speed it would run at alone.
Better failure behaviour. One card dies and you lose one eighth of capacity, not the whole service. A tensor-parallel group is a single failure domain.
Simpler to reason about and to scale. Adding capacity is adding a replica behind a load balancer, which is a problem your infrastructure team has already solved.
The only reason to split is that the model does not fit. Splitting to “go faster” is the classic over-engineering tell in this area, and the counter is one sentence: splitting adds a collective on every layer for every token, so if it fits on one card you should never split it.
The analogy: four accountants each add up a quarter of the columns, then compare notes before starting the next page. The comparing is the cost — and it happens at every layer, for every single token.
Each all-reduce moves a tensor the size of one token’s activation — 8 KiB at d_model 4,096 in bf16 — times the batch. At batch 128 across four GPUs, the ring exchange is about 100 MB per decode step, which at 900 GB/s is roughly 112 microseconds against a 6 millisecond step. Under 2%.
But there are 64 collectives per token — two per layer, thirty-two layers — and each carries a fixed latency of roughly 5 to 10 microseconds regardless of size. That is 0.3 to 0.6 milliseconds of pure round-trip cost, which is 5 to 10% of the step.
So the honest framing is: bandwidth is not the problem, message count is. That is also why the overhead does not shrink when the batch is small — it gets proportionally worse, because the fixed cost is spread over fewer tokens.
With tensor parallelism, each GPU holds the cache for the heads it owns. With 8 KV heads and 8-way parallelism, each card gets exactly one — clean, and part of why 8 has become conventional.
Push the parallelism degree above the KV head count and the heads must be replicated across devices. Sixteen-way parallelism on a model with 8 KV heads means every head lives on two cards, so aggregate cache memory doubles rather than staying flat. The KV head count therefore quietly caps how far tensor parallelism is useful, and beyond it you want pipeline parallelism instead.
Two traps are built in. Raise the split degree past the KV head count and the heads must be replicated, so aggregate cache goes up rather than down. And select the L40S, which has no NVLink: the arithmetic still works and the deployment does not.
The one to say out loud: “Parallelism exists because the model does not fit. Splitting adds communication on every layer, so if it fits on one card you should never split it.” Then, if pressed on very large models, the tensor-inside-pipeline-between shape.
| Tensor | Pipeline | Expert | Data / replicas | |
|---|---|---|---|---|
| Cuts | Each layer across GPUs | The stack into groups of layers | MoE experts across GPUs | Nothing — whole copies |
| Communicates | Every layer, every token | Once per boundary | Routing, per token per layer | Not at all |
| Needs | NVLink — inside one node | Tolerates a network | Fast links; routing can skew load | Nothing |
| Effect on KV cache | Split by head, unless degree > KV heads, then replicated | Split by layer — each stage holds its own layers’ cache | Unchanged — attention is untouched | Each replica has its own, entirely separate |
| Latency | Slightly better bandwidth, eaten by collectives | Worse — extra hops, and bubbles unless several requests are in flight | Adds routing hops | Unchanged |
| Use when | A 70B across 2–4 cards | Frontier models across nodes | Large MoE models | Whenever the model fits |
The hospital analogy: 128 specialists on the payroll, and the receptionist sends each patient to the two most relevant. The visit is quick because only two doctors see them. All 128 are still in the building and still being paid, because nobody can know in advance who tomorrow’s patients will need.
The router is small and simple: it takes the token’s current vector and produces one score per expert. The top two win, their scores become weights that sum to one, and their outputs are blended in that proportion.
Nobody programmed the specialisms. Routing is learned during training and it is self-reinforcing — an expert picked for a certain kind of token gets the learning signal for that kind of token, so it gets better and gets picked more. A balancing penalty is added during training to stop one expert winning everything while others starve.
And the specialisms that emerge are usually not human categories. They tend to be things like punctuation, code syntax or numbers, and often resist interpretation entirely. If asked, say that plainly rather than claiming there is a “medicine expert” — the overclaim invites a follow-up you cannot support.
Worth raising yourself, because it sounds like it should be a problem. Attention is where tokens look at each other; the feed-forward block — the part MoE replaces — processes each token independently even in a dense model. So sending different tokens to different experts changes which weights a token’s private refinement uses, and the next layer’s attention re-mixes everything regardless.
The rhythm is: mix, refine privately, mix again. MoE only touches the middle step. Note also that routing happens per token and per layer, not once per request — a single sentence touches many different experts on its way through the model.
A 700B MoE with 30B active costs you the memory of a 700B model and the speed of a 30B one, and produces quality somewhere in between — better than a dense 30B, a bit worse than a dense 700B would be if anyone could afford to run one.
So the fair comparison is against a dense model of the same active size, and there MoE is meaningfully better. Comparing it against a dense model of the same total size is the comparison that makes it look bad, and comparing the parameter count against a dense model is the comparison that makes it look magical. Neither is the useful one.
Three real downsides to name: training is less stable, because picking experts is a hard on-or-off decision rather than a smooth one; serving is more complex, because experts have to be distributed sensibly; and uneven routing can leave some cards busier than others, which is a tail-latency problem rather than an average one.
ArchitectWhen would you split a model across GPUs?
When it does not fit on one. That is the only reason, and I would say it that plainly, because splitting to go faster is the common over-engineering answer here.
Splitting adds a collective exchange at every layer for every token. Over NVLink that is a few per cent of a decode step in bandwidth terms — but there are 64 collectives per token, two per layer, and each carries a fixed round-trip latency, so the real overhead is five to ten per cent and it gets proportionally worse at small batch. Over a network it is comparable to the entire step, which is why tensor parallelism stops at the edge of a node.
So the ladder is: quantise first, because fp8 halves the weights and frees far more than adding a card does. Then split, as little as you can. And if it already fits, run independent replicas behind a load balancer — zero communication, and one card failing costs you one eighth of capacity rather than everything.
ArchitectDoes tensor parallelism make a single request faster?
Sometimes a little, because there is more total memory bandwidth working on the same weights — four cards reading a quarter of the model each can move the bytes faster than one card reading all of it. But the collectives eat into that, and at small batch, where single-request latency actually matters, the fixed per-message latency dominates.
The honest framing is that you do it for capacity, not speed. If someone proposes tensor parallelism as a latency optimisation for a model that already fits, I would want to see the measurement before believing it — and I would point out that they are also converting eight independent failure domains into one.
ArchitectDoes the KV cache get split too?
Yes, by head. With tensor parallelism each GPU holds the cache for the attention heads it owns, so with 8 KV heads and 8-way parallelism each card holds exactly one — clean, nothing duplicated, and part of why 8 KV heads has become conventional.
The trap is going further. If the parallelism degree exceeds the KV head count, heads must be replicated across devices: sixteen-way on a model with 8 KV heads puts every head on two cards and doubles the aggregate cache. So the KV head count quietly caps how far tensor parallelism is useful, and beyond that point you want pipeline parallelism between nodes rather than more tensor parallelism within them.
Eng managerA 700B mixture-of-experts model is described as running like a 30B. What do we actually need to buy?
The memory of a 700B and the compute of a 30B, and the memory is what you buy hardware for.
All the experts stay resident, because the router’s choice for the next token cannot be predicted. At 8-bit a 700B model is 652 GiB. A full node of eight H100s at the usual 0.90 utilisation gives 573 GiB, so a node is not enough — we would need eight H200s at 141 GB each, or sixteen H100s across two nodes, and that is before a single byte of KV cache.
Two follow-ons worth having ready. It does nothing for the cache, because MoE replaces the feed-forward block and leaves attention untouched — so all the per-user memory arithmetic is unchanged. And the fair quality comparison is against a dense model of the same active size, where MoE genuinely wins. If the business case was built on “it is as cheap as a 30B”, that case is wrong, and it is better to correct it now than after the purchase order.
ArchitectHow would you lay out a model that does not fit on one node?
Tensor parallelism inside each node, pipeline parallelism between them. The reasoning is entirely about the communication pattern: tensor parallelism needs a collective at every layer, which NVLink at 900 GB/s can carry and a 25 GB/s network cannot. Pipeline parallelism needs one handover per boundary, which a network handles fine.
Then replicate the whole arrangement for capacity and availability, because a single pipeline is a single failure domain spanning several machines.
Two things I would watch. The tensor degree must not exceed the KV head count, or heads get replicated and cache goes up. And pipeline parallelism needs several requests in flight to avoid bubbles, because a single request is only ever being worked on by one stage at a time — so it interacts with the batching strategy rather than being independent of it.
ArchitectCan I run a big model by spilling into CPU RAM?
Technically yes, and it is dramatically slower. PCIe is around 64 GB/s against HBM’s 3,350 — fifty-two times slower — and decode is already bandwidth-bound, so you are putting the bottleneck on the slowest link in the system. Expect something like an order of magnitude on tokens per second.
It is fine for experimentation and for running something on a laptop. It is not a serving strategy. The one place host memory genuinely earns its place in production is as a swap destination for preempted requests, where you are trading a 34-millisecond transfer against a 421-millisecond recompute — that arithmetic works. Continuous execution out of host memory does not.
Eng managerSomeone suggests eight-way tensor parallelism for a 7B model to cut latency. How do you handle it?
By turning it into a measurement rather than a disagreement, but I would be fairly confident of the outcome.
A 7B fits comfortably on one card, so splitting it buys no capacity. It adds 64 collectives per token, each with fixed latency, so at the small batch sizes where single-request latency matters the overhead is proportionally worst. It also converts eight independent failure domains into one, and eight independent servers into one — so we lose the ability to lose a card gracefully.
The alternative is eight replicas: zero communication, eight times the users, and any card can die. If the real goal is single-user streaming speed, the levers that actually work are speculative decoding at low batch, quantised weights to move fewer bytes, and a smaller cache — all of which are in documents 09, 10 and 13. I would propose measuring one replica against the eight-way split on inter-token latency at batch 1 and at batch 32; that settles it in an afternoon and leaves nobody arguing.
Why can I not just add GPUs and get more memory?
Because each GPU’s memory belongs to that GPU and there is no automatic pooling. An 8-GPU node with 80 GB each is not a 640 GB device. NVLink lets one card read another’s memory quickly, which is what makes splitting workable — but the software has to be told explicitly how to cut the model up.
What is a sensible tensor-parallel degree?
The smallest one that fits, capped at the KV head count, and never crossing a node boundary. For a 70B in bf16 on H100s that is 2 at a squeeze or 4 comfortably; at fp8 it fits on one. Always try quantisation before raising the degree, because halving the weights frees more budget than adding a card does.
Does MoE help my KV cache?
Not at all. MoE replaces the feed-forward block; attention and therefore the KV cache are untouched. MoE and grouped-query attention solve completely different problems and are often used together in the same model. Conflating them is one of the most common mistakes in this area.
Can I fine-tune a MoE model to add a new specialism?
Fine-tuning nudges the existing routing but will not reorganise which expert handles what — the structure was established during pre-training and is largely set. Adding genuinely new specialisation means training, not fine-tuning.
Why does pipeline parallelism need several requests in flight?
Because a single request is only ever being worked on by one stage at a time. With one request, the other stages idle — the classic pipeline bubble. You need enough concurrent requests to keep every stage fed, which means pipeline parallelism interacts with your batching strategy rather than being independent of it.
What is expert parallelism?
Spreading a MoE model’s experts across GPUs so each card holds a subset. It composes with tensor and pipeline parallelism rather than replacing them. Its specific difficulty is load skew: routing is data-dependent, so some cards can end up busier than others on a given batch, which shows up as a tail-latency problem.
What about TPUs and other accelerators?
Different hardware, same principles. Memory per chip, a fast interconnect within a group, a slower one between groups, and the same reasons for splitting models. Everything in this document transfers — only the constants change.
Does splitting change the model’s output?
It should not, and in principle it does not — the arithmetic is the same. In practice, floating-point addition is not associative, so a different reduction order can produce very slightly different numbers, and with sampling that can occasionally change a token. If you need bit-identical output across configurations, that is worth knowing about before you promise it.
Why is 8 GPUs the standard node size?
It is what the reference server designs ship, and it maps onto power, cooling and NVLink topology. The consequence for us is that 8 is the natural ceiling for tensor parallelism, and it is not a coincidence that 8 KV heads became conventional at the same time.
One sentence for the whole document?
Parallelism exists because the model does not fit; if it fits on one card, replicate instead; and mixture of experts saves compute rather than memory, so it changes what you buy less than the parameter count suggests.
“Eight GPUs are not one big memory — each card’s memory is its own and the software has to be told explicitly how to cut a model up. The first question is whether it needs cutting at all: if the model fits on one card, run independent replicas, because that costs zero communication and any card can fail without taking the service down. If it does not fit, tensor parallelism slices each layer across cards with an all-reduce at every layer — which NVLink can carry at a few per cent overhead and a network cannot, and where the real cost is sixty-four round trips per token rather than the bytes. Beyond a node you switch to pipeline parallelism, one handover per boundary. And the degree is capped by the KV head count, because above it the heads get replicated and the cache grows. Mixture of experts is a separate axis: it routes each token through two of many feed-forward blocks, so it saves compute and not one byte of memory, because every expert has to stay resident.”
| Thread started here | Picked up in |
|---|---|
| The feed-forward block that MoE replaces, and its 70% share | 02 · Inside the model |
| NVLink against PCIe against the network, on one log scale | 04 · The GPU and the roofline |
| The cache that gets split by head, and the budget it comes from | 06 · The KV cache |
| Why the KV head count is the cap on the split degree | 07 · Attention variants |
| Quantising first, which frees more than adding a card | 09 · Precision |
| Swap to host memory, where PCIe genuinely earns its place | 10 · Paging and prefix reuse |
| The flags that set the parallel degrees | 12 · Serving engines |
| Turning all of this into a fleet size and a bill | 14 · Capacity planning |