Document 13 · Quick learnCapacity vs throughput · Max, not mean · Move the box, not the vectors
Slicesand copies
Sharding answers “too big for one machine”. Replication answers “too fragile, or too slow, to serve alone”. Everything goes wrong when those two get answered together.
“A node hosts shard-replicas. A shard is a slice; a replica is a copy of a slice.”
3.76 units
of CPU for four shards, against 1.00
80 → 20 %
moved, modulo against the ring
1.5×
what each survivor absorbs at RF 3
382 → 1.5 GB
rebuild peak, one shard or sixty-four
Two problems, not oneand one word that hides both
Sharding
“The copy is too big for one machine.”
Splitting the collection across four buildings
No building holds everything, and finding a book means asking all four
Solves: the collection outgrew the building
Does not buy throughput
Two problems. Two knobs. Two failure modes. One word: “scaling”.
Replication
“One copy is too fragile, or too slow, to serve alone.”
Three identical libraries in three towns
Each holds everything; any one can answer
Solves: the building burned down, and the queue at the desk is too long
This is where throughput comes from
One sentence to have ready
“A node hosts shard-replicas. A shard is a slice; a replica is a copy of a slice. The graph belongs to the shard-replica, never to the table.”
More interview damage is done by loose vocabulary here than by any missing concept
Five words, kept distinct
Table
All ten million chunks, their vectors and payloads. A collection in Qdrant and Milvus.
Not the ANN structure
Index
The ANN structure over vectors. One index per shard, not per table.
Not Elasticsearch’s “index”, which is a table
Shard
A disjoint slice. Its own vectors, payload and graph. The unit that moves between nodes.
Not a replica, which is a copy rather than a slice
Replica
One full copy of one shard. RF 3 means three copies of every shard.
Not a node, which hosts many replicas
Node
A machine or a pod. Hosts some number of shard-replicas and runs their graphs in RAM.
Not a shard
Why shardthe ceiling is memory, not disk
HNSW stays in the low-millisecond range only while the graph and the vectors it scores are resident. So the question is never “does the data fit on the volume” — it is “does one copy fit in one node’s RAM, with room left over for everything else that node has to do”.
What sharding does not buy: throughput
Every query visits every shard, so four shards mean four graph walks per query. Total CPU per query goes up slightly, not down.
Throughput comes from replication. This is the single most common confusion on this topic, and saying it unprompted is worth more than any parameter name
Why more shards than you need today
Shard count is a create-time parameter in every engine here and cannot be raised cheaply later. A binary-quantised table that fits one shard today will not fit one shard at five times the corpus. Over-provisioning — 8, 16, 32 or 64 shards on a four-node cluster — is normal and correct.
The cost is a little per-shard graph overhead and a little more fan-out. The benefit is that scaling out becomes a file move instead of a rebuild — the difference between an afternoon and a quarter
Cost one: fan-out, and the taila query that waits for all of them is as slow as the slowest
1 · Lands on one node
Which becomes the coordinator for this request.
2 · Fan out to all four
Hash routing scattered the neighbours, so it cannot know which shard holds them.
3 · Three answer
In about twenty milliseconds each.
4 · Shard 3 stalls
A GC pause, a cold page, a noisy neighbour. Answers at 84 ms.
5 · Only now can it merge
Fan-out latency is max, not mean.
Cost two: recall dilution — the merge is exact, but only if you over-fetch
10
Fetch 10 per shard — 40 totala vector in the global top 10 is necessarily in its own shard’s top 10
100% ceiling
3
Fetch 3 per shard — 12 totalshard 1 held 5 of the global top 10 and returns only 3
80% ceiling
2
Fetch 2 per shard — 8 total3 of shard 1’s 5 are lost
70% ceiling
Elasticsearch is the one vendor that names this out loud
num_candidates is the number gathered per shard, defaulting to 1.5 × k and capped at 10,000. Qdrant and Weaviate do the same thing without giving it a name, which is why the parameter gets missed.
The rule: fetch at least k per shard, and 1.5× to 2× k if you can afford it — and raise it when you raise shard count
The counter-intuitive part, worth having ready
At the same efSearch, a smaller HNSW graph typically reaches slightly higher recall, because the shortlist covers a larger fraction of the neighbourhood. So sharding does not hurt per-walk recall at all. What it can hurt is the merge, and only when someone under-fetches.
Keeping per-shard recall and merge recall apart is the difference between diagnosing this correctly and tuning the wrong knob
Workedone graph at 10M = 1.00 unit of CPU · four graphs at 2.5M each = 4 × 0.94 = 3.76 units — because log(2.5M) is 94% of log(10M)latency is roughly one walk; CPU is nearly four
Cost three: rebalancingand the HNSW delete problem
If adding a node means some vectors change shard, each must be deleted from one HNSW graph and inserted into another. Both halves are expensive, and the delete half is expensive in a way that does not show up until later.
1
Move 2M vectors, the naive way20% of a 10M corpus
2,000,000
2
Source shards2M soft deletes → 2M dead slots, and the graph does not shrink
0 GB freed
3
Still residenttheir vectors and their edges, until a rebuild or vacuum compacts them
still there
4
Target shard2M inserts — and every HNSW insert is a search
~35 core-min
5
During the moveboth sides inflated at once
rebuild peak
Why the delete is the nasty half
HNSW deletes are soft. Removing a node would break every path running through it, so implementations mark it dead, keep it in the graph for traversal, and filter it from results at query time. The memory is not returned.
Which is exactly where the reference stack’s 16 GB of dead vectors comes from. A rebalance does not move memory from one node to another — it adds memory on the target and keeps it on the source
The two ways out, and only one is a design
Make the unit of movement a whole shard, so no vector ever changes graph. Every serious engine does this. Or accept a full rebuild of the affected shards as a scheduled batch job with the peak sized in — which is what you fall back to when the shard count was set too low on day one.
The interview line: “Rebalancing at the vector level costs an HNSW soft delete plus an insert-search per vector, and the delete frees nothing, so both sides inflate. That is why the unit of rebalancing must be the shard.”
Routingmodulo, the ring, and why 80 per cent moves
Four shards, h mod 4
Shard 0 gets 0, 4, 8, 12, 16…
Add a fifth: h mod 5
Every vector is re-routed under the new rule.
Only 0, 1, 2, 3 stay
The residues where the two rules agree.
16 of 20 change shard — 80%, and that ratio holds at ten million
Consistent hashing, and the fraction it saves
Put the hash space on a circle. Place each shard at one or more points. A vector belongs to the first shard clockwise from its hash. Adding a shard adds a point, and only the vectors between the new point and its anticlockwise predecessor change owner.
Expected fraction moved, N → N+1, is 1 ÷ (N+1). So 4 → 5 is 20% against 80% — 2M vectors instead of 8M. With virtual points, that 20% comes evenly from all four old shards
The subtlety that matters more than the ring itself
Qdrant’s automatic sharding uses consistent hashing to place points — but it hashes into a fixed shard count set at creation. Changing that count is resharding, which is a cloud feature rather than a self-hosted one.
So the honest summary: consistent hashing is the right mental model for the interview; over-provisioned shards are the right implementation. Both cut the 80%
The box modelnodes do not own vectors — nodes own boxes
Imagine sixty-four numbered boxes. Every vector goes into one box once, at ingestion, by hashing its id — and that assignment never changes for the life of the vector.
Day one
Sixteen boxes across four nodes, four each. Every box carries its own HNSW graph.
A fifth node joins
It starts empty — nothing moves automatically in self-hosted Qdrant or Weaviate.
Three boxes are chosen
One each from three nodes. Their on-disk files are copied across the network.
Node 5 loads and serves
Same graphs, new machine. No re-hash, no HNSW insert.
Worked64 boxes over 10M chunks = 156,250 vectors per box · unquantised 1.49 GB per box · int8 0.46 GBthe reference stack, in boxes
4 nodes at RF 3, unquantised
192 shard-replicas, 48 per node.
48 × 1.49 = 71.6 GB — does not fit a 64 GB node
7 nodes at RF 3, unquantised
~27 per node.
27 × 1.49 = 40.3 GB — 63% of a 64 GB node, which fits
4 nodes at RF 3, int8
48 boxes per node.
48 × 0.46 = 22.1 GB — 35%, with room for the next doubling
The operational win people forget
A rebuild is done one replica at a time so the others keep serving, and during it the node carries the old graph and the new one side by side. With one shard the whole-cluster peak is 286 + 95.5 = 382 GB. With 64 boxes the in-flight unit is one box, and the peak is 286 + 1.5 GB.
Sizing a cluster for the 382 figure is what you do when the shard count was set to 1. That single difference is often worth more than the scale-out story
Hash routing against key routingand the skew it buys
Hash routing
Even by construction — and that evenness is the price.
query → all 64 boxes → 64 walks → two-level merge
Related vectors — one customer’s chunks, one document’s — land on different shards
So every query fans out, and pays the tail
CPU: roughly 60 units
Key routing trades fan-out for skew.
Key routing
Supply a shard key; everything with that key lands together.
query(tenant=A) → the box holding A → 1 walk → done
Latency: one walk, no fan-out tail
CPU: 1 unit instead of roughly 60
Recall: exact for that tenant — no merge, so nothing to dilute
The line to say
“Key routing trades fan-out for skew. I handle skew with three tools: split hot keys, route by a size threshold, and rebalance on observed load rather than on vector count — keeping the hot shard’s replicas on three different nodes throughout.”
This is also the mechanism document 14 is built on
Replicationthree of the four consequences are costs
What it buys — survival
Lose a node and every shard it held still has two live copies.
No data loss, no downtime, no rebuild from source
What it buys — read throughput
The coordinator round-robins across three copies.
cluster QPS ≈ (QPS one shard-replica serves) × RF — and not × shard count, because every global query visits every shard
Worked100 QPS per shard-replica × 3 = ~300 QPS · key-routed traffic leaves the other shards free → ~100 × 3 × shards for that mixbefore the coordinator or the network becomes the limit
And what it costs: writes go everywhere
Every insert, update and delete must reach all three copies. Ingestion CPU and network triple while read cost per query stays flat — at a thousand inserts a second, the cluster performs three thousand HNSW insert-searches a second.
The honest summary of RF 3: three times the memory, three times the write cost, three times the read capacity, and one node’s worth of failure tolerance. Three of those four are costs
Consistency: the dial, and what Raft actually governs
The problem in one sentence
When a write has landed on copy A but not yet on copy C, a query served by C does not see it. For RAG that means a freshly ingested document is briefly invisible — usually tens to hundreds of milliseconds, sometimes seconds under load.
The dial has two halves — write side and read side — and they are set independently
Raft governs the map, not the data
A detail that gets confused constantly. Raft keeps the routing table consistent — which shard-replica lives on which node. The vectors themselves are not under consensus; they are replicated with acknowledgement rules.
That separation is what makes the cluster cheap to run: a few kilobytes of metadata under a consensus protocol, and hundreds of gigabytes of data under a much looser one
Failure arithmetic and headroomthe survivors absorb what the dead node was serving
survivor multiplier = RF ÷ ( RF − 1 )
RF 3 → 1.5× the traffic on each survivor · RF 2 → 2.0× — which is why the restore window matters
Placement
With many boxes over many nodes, one node’s loss spreads thinly — each of its boxes has survivors on different machines.
Availability zones
Three replicas on three nodes in one rack survive a node, not a rack.
Pin the copies to three zones and you survive a zone
Recovery time
A returning node rebuilds from a survivor — a whole-shard file transfer plus a WAL catch-up.
1
Restore one node’s share, 7-node cluster~41 GB
41 GB
2
At 1 Gbit/sfile transfer
~5.5 min
3
At 10 Gbit/sfile transfer
~35 s
4
By re-insertion insteadat 3,000 inserts/s
~2.4 hours
That 250-times difference is a configuration choice
Five and a half minutes against two and a half hours, for the same restore, decided by whether the transfer method copies files or re-inserts vectors.
And the window matters more than it looks: while it is open you are at RF 2 on those shards, so the next failure has a survivor multiplier of 2.0 rather than 1.5. Size for it
The vendor mapquote with a date; the shape is what does not shift
Qdrant
shard_number at creation, default 1.
Placement: auto (consistent hashing) or custom (shard keys) RF default 1, write_consistency_factor default 1, per-request read consistency New nodes start empty; move shards with the transfer API
Consistent hashing over virtual shards; multi-tenant collections give each tenant its own shard ONE / QUORUM / ALL, both sides. Async replication on by default above RF 1
Elasticsearch
number_of_shards at creation. One HNSW graph per Lucene segment.
Which is why force-merging to one segment speeds up kNN number_of_replicas changeable live; primary count is not Gathers num_candidates per shard and merges
Milvus
Shard = write channel; segments are the load and balance unit.
Storage–compute separation: sealed segments live in object storage and load onto query nodes Consistency Strong / Bounded / Session / Eventually
Pinecone
Serverless: no user-visible shards.
The namespace is the routing key; queries are scoped to one Pod-based indexes with explicit shards and replicas are legacy
pgvector
No native sharding.
Table partitioning gives one HNSW index per partition; Citus distributes across nodes Streaming replication via WAL; replicas serve reads A partitioned table means fan-out in the planner
What to notice rather than memorise
Three of the six make the new node start empty and require an explicit move. Two of the six make the primary/shard count immutable after creation. Every one of them separates a slice count from a copy count.
If you can state those three patterns you can reason about an engine you have never used — which is the actual skill being tested
Growing and sealed — the part that catches people
Every engine has a small, mutable, recently-written portion and a large, immutable, indexed one. The mutable part is brute-force scanned — there is no graph over it yet — so a shard taking heavy writes is slower than its size suggests. And it is the part that must be synchronised between replicas after a write, which is where the consistency dial bites.
Answer it out loudthe ninety-second version, then six questions
“Sharding answers ‘too big for one machine’; replication answers ‘too fragile, or too slow, to serve alone’. They are different knobs and conflating them is the classic error — sharding does not buy throughput, because every global query visits every shard, so four shards means four walks and slightly more CPU per query. Throughput comes from RF. The two real costs of sharding are the fan-out tail, because latency is max rather than mean, and recall dilution at the merge if anyone under-fetches — fetch at least k per shard. And I would over-provision shards at creation, because shard count is immutable and moving a whole shard is a file copy, while moving vectors is an HNSW soft delete plus an insert-search each, and the delete frees nothing.”
Six questions, and the spine of each answer
Architect
When do you shard, and what does it buy?
When one copy no longer fits in one node’s RAM — the ceiling is memory, not disk
And it buys exactly that: the ability to hold a corpus larger than a machine
It does not buy throughput. Every query visits every shard
Four shards means four walks — 3.76 units of CPU against 1.00, because log(2.5M) is 94% of log(10M)
Throughput comes from replication: cluster QPS ≈ per-replica QPS × RF, and not × shard count.
Architect
What does sharding cost?
The fan-out tail. The coordinator waits for the slowest shard — three at 20 ms and one at 84 makes the query 84
Recall dilution at the merge, but only if you under-fetch: fetch 3 per shard instead of 10 and the ceiling is 80%
Rebalancing, if the unit of movement is the vector rather than the shard
And the counter-intuitive part: per-walk recall is slightly better on a smaller graph at the same efSearch. So sharding does not hurt recall — the merge does, and only through under-fetching.
Architect
Why is rebalancing at the vector level so expensive?
Each moved vector is an HNSW soft delete plus an insert-search
The delete frees nothing — removing a node would break every path through it, so it stays in the graph and is filtered at query time
So a rebalance adds memory on the target and keeps it on the source until compaction
Which is why the unit of rebalancing must be the shard: moving a whole box is a file copy and no vector ever changes graph. That is also the argument for over-provisioning shard count on day one.
Architect
Adding a fifth shard — how much data moves?
Under h mod N: 80%. Only the residues where the two rules agree stay put
Under consistent hashing: 1 ÷ (N+1) — so 20%, or 2M vectors instead of 8M
With virtual points, that 20% comes evenly from all four old shards
But the honest implementation note: most engines hash into a fixed shard count set at creation, so in practice the ring gives even placement and the move-on-resize property is realised by moving whole shards.
Eng manager
We lost a node and latency doubled. Was that expected?
Yes, and the multiplier is RF ÷ (RF − 1) — at RF 3 the survivors each absorb 1.5×
So the utilisation policy is set by that: the cluster must run at a level where 1.5× is still inside budget
And during the restore window you are at RF 2 on those shards, so the next failure is 2.0×
Which makes restore time a capacity decision: ~5.5 minutes by file transfer at 1 Gbit/s, or ~2.4 hours by re-insertion. Same restore, 250× apart, and it is a configuration choice.
Eng manager
One tenant is enormous and their queries are slow.
That is key routing meeting skew — and the trade it made was fan-out for skew
Three tools: split the hot key across several shards; route by a size threshold so big tenants get their own; rebalance on observed load rather than vector count
And keep the hot shard’s replicas on three different nodes throughout
The thing not to do is switch that tenant back to hash routing, which trades one query’s skew for every query’s fan-out.
Fast answersthe seven that come up most
QDoes sharding make search faster?
No. Latency is roughly one walk plus network plus merge plus the tail — and the tail is what you actually feel. CPU goes up: four graphs at 2.5M each is 3.76 units against 1.00 for one graph at 10M. Throughput comes from replication.
QHow many candidates should each shard return?
At least k, and 1.5× to 2× k if you can afford it — Elasticsearch defaults num_candidates to 1.5×k. And raise it when you raise shard count, because the probability that one shard holds several of the global winners does not fall as fast as people assume.
QHow many shards should I create?
More than you need today, because shard count is a create-time parameter in every engine here. 8, 16, 32 or 64 on a four-node cluster is normal and correct. The cost is a little graph overhead and fan-out; the benefit is that scaling out becomes a file move instead of a rebuild.
QDoes Raft replicate my vectors?
No — and this gets confused constantly. Raft keeps the routing table consistent. The vectors are replicated with acknowledgement rules. That separation is what makes the cluster cheap: kilobytes under consensus, hundreds of gigabytes under something much looser.
QWhy is a shard taking heavy writes slower than its size suggests?
Because the recently-written portion is brute-force scanned — there is no graph over it yet. Milvus calls them growing and sealed segments; Elasticsearch has the refresh cycle; Qdrant has an appendable segment the optimiser later converts.
QWhat is the real cost of RF 3?
Three times the memory, three times the write cost, three times the read capacity, and one node’s worth of failure tolerance. Three of those four are costs. Every insert and delete must reach all three copies, so at 1,000 inserts/s the cluster does 3,000 HNSW insert-searches per second.
QDo three replicas on three nodes make me zone-fault-tolerant?
Only if they are in three zones. Three replicas on three nodes in one rack survive a node, not a rack. Pin the copies to zones explicitly — the engine will not do it for you.
Sharding answers ‘too big’; replication answers ‘too fragile, or too slow’. Conflating them is the classic error, because sharding buys capacity and costs CPU, latency tail and merge recall — while throughput comes entirely from RF. And the decision that shapes everything afterwards is made once, at creation: over-provision the shard count, because moving a whole shard is a file copy while moving vectors is an HNSW soft delete plus an insert-search each, and the delete frees nothing on the source.