Splitting a model is a workaround for a memory limit, not a performance feature — and mixture of experts saves compute rather than the thing you actually buy.
They can pass ingredients to each other across the bench very quickly — that is NVLink. What they cannot do is treat the eight pantries as one large pantry: a recipe needing more ingredients than any single pantry holds does not simply work. Somebody has to decide, in advance and explicitly, which chef stores what.
Before any discussion of how to split a model: does it need splitting at all?
Each GPU holds one eighth of every layer.
Eight servers behind a load balancer.
A tensor-parallel group is a single failure domain. If someone proposes it as a latency optimisation for a model that already fits, ask to see the measurement — and point out that they are converting eight independent failure domains into one.
Each layer’s weight matrices cut one slice per GPU.
Its slice of the result — a partial answer, useless on its own.
Every GPU needs the full sum before the next layer can start.
~2% of a decode step in bandwidth terms.
Comparable to the entire step.
Bandwidth is a few per cent. Message count is the real cost — two collectives per layer, thirty-two layers, sixty-four round trips per token, each carrying a fixed latency regardless of size.
Each GPU holds the cache for the heads it owns. With 8 KV heads and 8-way parallelism, each card gets exactly one — clean, and part of why 8 became conventional. Push the degree above the KV head count and the heads must be replicated.
Cuts each layer across GPUs.
Cuts the stack into groups of layers.
Cuts MoE experts across GPUs.
Cuts nothing — whole copies.
Tensor inside a node, pipeline between nodes, replicate the whole thing.
A single request is only ever being worked on by one stage at a time. With one request, the other stages idle — the classic pipeline bubble. You need enough concurrent requests to keep every stage fed.
Say 128 experts — all resident in GPU memory, all of the time.
One score per expert per token. Picks the top two.
Corresponds to two experts — about 30B active.
Corresponds to all 128 — the full 700B, always.
Worth raising yourself, because it sounds like it should be a problem. Attention is where tokens look at each other; the feed-forward block — the part MoE replaces — processes each token independently even in a dense model.
A 700B MoE with 30B active costs you the memory of a 700B and the speed of a 30B, and produces quality somewhere in between.
MoE replaces the feed-forward block and leaves attention untouched. MoE and grouped-query attention solve orthogonal problems and are frequently used together in the same model.
When would you split a model across GPUs?
Does tensor parallelism make a single request faster?
Does the KV cache get split too?
A 700B MoE is described as running like a 30B. What do we buy?
How would you lay out a model that does not fit on one node?
QWhy can I not just add GPUs and get more memory?
Each GPU’s memory belongs to that GPU; there is no automatic pooling. An 8-GPU node with 80 GB each is not a 640 GB device. NVLink lets one card read another’s memory quickly, which is what makes splitting workable — but the software has to be told how to cut the model up.
QWhat is a sensible tensor-parallel degree?
The smallest one that fits, capped at the KV head count, never crossing a node boundary. For a 70B in bf16 on H100s that is 2 at a squeeze or 4 comfortably; at fp8 it fits on one. Always try quantisation first.
QDoes MoE help my KV cache?
Not at all. MoE replaces the feed-forward block; attention and the KV cache are untouched. MoE and grouped-query attention solve completely different problems and are often used together. Conflating them is one of the most common mistakes here.
QCan I fine-tune a MoE model to add a new specialism?
Fine-tuning nudges the existing routing but will not reorganise which expert handles what — the structure was established during pre-training. Adding genuinely new specialisation means training, not fine-tuning.
QWhy does pipeline parallelism need several requests in flight?
A single request is only ever being worked on by one stage at a time — the classic pipeline bubble. You need enough concurrent requests to keep every stage fed, which means it interacts with your batching strategy.
QWhat is expert parallelism?
Spreading a MoE model’s experts across GPUs so each card holds a subset. It composes with tensor and pipeline parallelism rather than replacing them. Its specific difficulty is load skew: routing is data-dependent, so some cards end up busier, which shows up as a tail-latency problem.
QWhat about TPUs and other accelerators?
Different hardware, same principles. Memory per chip, a fast interconnect within a group, a slower one between groups. Everything here transfers — only the constants change.
QDoes splitting change the model’s output?
It should not, and in principle does not. In practice floating-point addition is not associative, so a different reduction order can produce very slightly different numbers, and with sampling that can occasionally change a token. Worth knowing before you promise bit-identical output.
QWhy is 8 GPUs the standard node size?
It is what the reference server designs ship, and it maps onto power, cooling and NVLink topology. The consequence: 8 is the natural ceiling for tensor parallelism — and it is not a coincidence that 8 KV heads became conventional at the same time.
QOne sentence for the whole document?
Parallelism exists because the model does not fit; if it fits on one card, replicate instead; and mixture of experts saves compute rather than memory, so it changes what you buy far less than the parameter count suggests.
Splitting a model is a workaround for a memory limit, not a performance feature. If it fits on one card, replicate — zero communication, and one card failing costs an eighth of capacity rather than the service. If it does not, quantise before you split, split as little as you can, and stop at the KV head count. And mixture of experts saves the thing you do not buy — a 700B MoE needs 652 GiB whether two experts run or all hundred and twenty-eight do.
What tensor-parallel degree are you running, and why that number?
Does the model actually need splitting, or does it fit at your precision?
Is the parallel degree above or below the KV head count?
Do any parallel groups cross a node boundary?
If you serve a MoE model, was the hardware sized on total parameters or active ones?
How many replicas, and what happens when one card in a tensor-parallel group fails?