Track A · Document 03 · Ingestion and chunking
The part of RAG that separates a demo from a system: a million documents that change continuously, and an index that has to stay correct, cheap and available throughout.
Somebody inserts one paragraph into a forty-page policy. How many chunks do you have to re-embed? The naive answer is one. The real answer, with the wrong design, is all of them — and the reason is not the text, it is the boundaries.
The naive answer to “how many chunks must be re-embedded?” is one. The real answer, with the wrong design, is all of them.
Always quantify this. It turns an architectural preference into a business argument, and it is the number that gets the design approved.
Roughly a fifty-fold reduction in embedding spend, GPU time and index write amplification. It also collapses freshness lag: an edit becomes searchable in seconds instead of queueing behind tens of thousands of pointless re-embeds.
Hashing detects change. It does not prevent cascade. The hashes genuinely differ, because the content inside each window genuinely differs. What prevents the cascade is boundary stability — making cut points depend on the text rather than on the offset from the start of the document. Hashing then sits on top and suppresses the writes.
A chunk ID must be derived from stable identity, not from position. This sounds like a detail and it is the single most expensive thing in this runbook to get wrong, because retrofitting it means a full reindex plus reconciling historical data that may no longer be reconcilable.
The classroom analogy. Positional naming calls students “Student 1, Student 2” by seat order: a new arrival in the second seat renames everybody. Identity naming gives them roll numbers: the new student gets roll 51 and nobody else changes. Roll 5 is still the same person tomorrow.
Your upsert writes Bereavement text into doc7#1, silently replacing
Maternity Leave.
If the write for doc7#2 then fails mid-batch, Maternity Leave has
simply vanished from the index — and nothing looks broken.
Delete two sections and the document now has 78 chunks. doc7#78 and
doc7#79 still sit in the index holding retired text.
They keep getting retrieved. The assistant keeps quoting the old policy.
A user rates doc7#41 badly today; tomorrow doc7#41 is different
text.
The bug report points at the wrong thing, and the feedback dataset is silently corrupted.
Stale orphan chunks. The system does not error, does not degrade visibly, and answers confidently from retired content. In a policy or compliance setting that is considerably worse than an outage, because an outage is noticed within minutes and this is noticed when somebody acts on a rescinded rule.
Where does a stable ID come from? Four real answers and one that should be crossed out. Walk down the ladder until something is available.
Every rung above the bottom keeps identity stable through the edits that actually happen. The bottom rung keeps none of them, and it fails silently — which is why it deserves to be crossed out rather than listed as an option.
| Source | How the ID is made | Strength | Weakness |
|---|---|---|---|
| Source-system anchor | Use the ID the source already assigned — Confluence heading anchors, Notion block UUIDs, DITA or XML element IDs | Stable by design. Survives renames and moves | Only exists where the source system provides one |
| Derived — section path | hash(doc_id + "3.2 Maternity Leave" + ordinal) |
Free to compute, deterministic, no stored state | Breaks on rename; depends on heading detection being reliable |
| Derived — content hash | hash(normalised_text) |
Immune to renaming and reordering | The ID is the content, so any edit mints a new ID and tombstones the old one — noisy for edit-heavy corpora |
| Assigned | A random UUID minted once at first ingest and stored | Survives any change to heading or body, so citations and feedback stay valid | You now own matching logic on re-ingest, and matching can be wrong |
| Positional | doc_id#ordinal |
Trivial | All three failure modes above. Avoid |
Fetch a Confluence page through the API and you do not get plain text — you get structured content where each heading carries its own permanent anchor:
heading, anchor "Maternity-Leave-a91c"
paragraph …
paragraph …So chunk_id = page_id + anchor. The author renames the section to
“Parental Leave” and the anchor a91c does not change, because Confluence
generated it once and keeps it. Your chunk ID is identical, only the hash differs, and you
re-embed exactly one chunk.
Plain text files have no anchors, because nothing ever assigned one. That is why the ladder exists.
When the source gives you nothing, your pipeline assigns identity itself at first ingest and stores it. On every later ingest you have to match freshly parsed, label-less sections back to those stored IDs — and that matching is a real algorithm with a real failure mode.
At first ingest you create the chunks and invent an identifier for each — literally a random unique string. It has no meaning; it is a label. You save it to your database alongside the chunk: this ID, this text, this hash, this vector, belongs to this document. Later, when the document is edited, you read those rows back. Those already-in-the-database labels are the stored IDs.
The discipline that keeps this correct: greedy, highest-similarity first, and strictly one-to-one. A stored ID can be claimed exactly once. Without that rule, two new sections both inherit the same identity and the index is corrupted in a way that is very hard to detect afterwards.
Greedy, highest-similarity first, and strictly one-to-one. A stored ID can be claimed exactly once. Without that rule, two new sections both inherit the same identity and your index is corrupted in a way that is very hard to detect later — two rows with the same ID, one silently overwriting the other on every subsequent ingest.
And set the similarity threshold deliberately. Too low and an unrelated new section inherits a stale identity along with its citations. Too high and every rewrite mints a new ID, which defeats the purpose. Somewhere around 0.8 to 0.9 on a normalised cosine score is a sane starting point, and it is worth measuring on your own corpus rather than adopting.
Stable identity solves the naming. Stable boundaries solve the cascade. Three techniques, in the order you should reach for them.
Three ways to make a cut point depend on the text rather than on the offset. Use A wherever the parser gives you reliable headings, C wherever it does not, and B when the text has no structure at all — which is rarer than people think.
The rolling-hash idea is the one people find hardest to picture, so here it is word by word on a single sentence.
A hash function, briefly, in case you are asked: a recipe that turns text into a number. The crude version adds up each letter’s position in the alphabet — “cat” gives 3 + 1 + 20 = 24. Real ones scatter the results evenly. The only property that matters here is that the same input always gives the same number.
Two answers. First, an oversized paragraph is split internally with ordinals scoped to it, so any reshuffle stays local to that paragraph. Second, and more importantly, you do not solve this with chunk size — you solve it with parent–child: embed small precise pieces and hand the surrounding paragraph to the model. That is document 01, section 6.
You reprocess the whole document every time, and you only write what changed. Parsing and hashing are cheap text processing; embedding is the expensive part, and the diff is what protects it.
for each parsed section:
id = stable_id(doc, section_path, ordinal)
hash = sha256(normalise(text))
if id not in stored_ids → INSERT (embed)
elif hash != stored_hash[id] → UPDATE (embed)
else → SKIP (no embedding call)
deleted = stored_ids(doc) − parsed_ids
for id in deleted → TOMBSTONECompute the delete set per document. If you only upsert what you parsed and never do the subtraction, removed chunks quietly stay in the index and keep being retrieved. No error, no alarm, nothing looks wrong — and the assistant keeps quoting the retired policy.
Two reasons, and give both, because they come from different places:
In a graph index the vectors are woven into a navigable structure, so removing one means repairing edges.
Doing that on every delete under live traffic degrades the graph and lowers search quality. Flag it, filter it at query time, batch the real removal into compaction.
A bad parse can produce an empty section list, and the diff would then cheerfully delete an entire document.
With tombstones that is reversible — flip the flag back. A hard delete means re-embedding everything.
| Concern | What you do | Why |
|---|---|---|
| Change detection | Webhooks or change-data-capture where the source offers them; fall back to polling with etag or last-modified | Plus a nightly full crawl as a safety net, because webhooks silently drop events |
| Ordering | Partition the ingest queue by doc_id |
Two edits to the same document must not process concurrently. Different documents parallelise freely |
| Idempotency | Carry a monotonic doc_version on every event; discard anything older than
what the index already holds |
Retries and out-of-order delivery are guaranteed at scale |
| Atomicity | Either tag chunks with doc_version and filter reads to the latest committed
version, or accept a few hundred milliseconds of mixed state |
A choice with a cost, not a best practice. Section 8 has the framing |
A sanity check that sits between the diff and the write, asking one question: does this change look plausible? Two guards catch almost everything, and the second one catches the failure the first cannot see.
While an update is held, the old chunks keep serving. Stale content for an hour is far cheaper than a document silently vanishing from the index. Park it in a review queue and alert someone — and track ID overlap per source system, because a vendor export change or a parser upgrade drops it across thousands of documents at once, which is the early warning that catches the problem before the reindex bill does.
Track ID overlap per source system, not just per document. When a vendor changes their export format, or you upgrade a parser, overlap drops across thousands of documents at once. That aggregate metric catches it in an hour. Without it, you find out from the reindex bill.
At a million documents with continuous edits, concurrency bugs are not hypothetical. Three specific races, each with a cheap defence and each invisible until somebody notices something that should not be possible.
At a million documents with continuous edits, none of these are hypothetical. Each has a cheap defence, and each is invisible until an auditor or a user notices something that should not be possible.
Being able to draw this is worth a lot in an interview, because it forces every edge case into the open — and edge cases are exactly what an interviewer reaches for once your happy path is convincing.
Worth being able to draw from memory. It is the fastest way to demonstrate that you have operated one of these rather than designed one on a whiteboard.
An engineering-manager question about ingestion speed is usually a capacity-planning question in disguise. Have this arithmetic ready; it is four lines and it settles the argument.
An engineering-manager question about throughput is usually a capacity-planning question in disguise. Have this arithmetic ready, and notice what it reveals: the expensive stage is parsing, not the AI part — until the chunking design is wrong, at which point embedding takes over.
Parsing dominates, not embedding. Roughly ninety percent of the per-document time in a well-designed pipeline. So “the AI part is expensive” is usually wrong, and the parser tier from document 02 is what to optimise first.
The naive design breaks precisely here. Without the structural diff, each document embeds 80 chunks instead of two, which is six seconds of embedding rather than 150 ms, and the fleet goes from twelve workers to thirty-six. That is the fifty-fold factor from section 1 expressed as infrastructure rather than as an invoice.
Incremental edits and bulk backfill have opposite requirements: edits are low volume and need low latency; backfill is high volume and can wait. Sharing one queue means a backfill of half a million documents puts every live edit behind it.
Separate queues, separate worker pools or weighted consumption, and separate alerting. It is a small design decision that prevents a very common production complaint: “we edited it an hour ago and it is still not there” — during a reindex nobody told the users about.
The embedding call is one line in the arithmetic above and a real subsystem in practice. Five things decide whether it survives contact with a corpus:
| Concern | What to do |
|---|---|
| Batching | Embedding APIs and local models are both far more efficient per item on a batch than on a single call. Accumulate chunks up to a batch size or a short flush timeout, whichever comes first, so a trickle of edits does not wait forever behind a half-full batch |
| Rate limits and backpressure | Treat the limit as a resource you schedule against, not an error you retry into. When the queue grows faster than the limit allows, the correct response is to shed backfill and protect the edit queue — which is only possible if they are separate queues |
| Retries | Exponential backoff with jitter, and a dead-letter queue after a bounded number of attempts. A chunk that cannot be embedded must end up somewhere visible rather than disappearing from the pipeline |
| Idempotency | Re-embedding the same chunk twice must be harmless: the write is an upsert keyed by chunk ID, and the version fence discards anything stale. That is what makes retries safe |
| Cost control | Meter embedding calls per document and alert on the ratio of calls to changed documents. A sudden rise in that ratio is boundary cascade or a broken hash, and it is much cheaper to catch as a metric than as a monthly bill |
The last row is the one worth volunteering. Calls per changed document is the single most diagnostic ingestion metric there is: it should sit near one or two, and when it jumps to eighty you have learned exactly what broke.
Every event-driven pipeline drops events. Webhooks fail, retries expire, a deploy eats a queue, a source system has an outage. Reconciliation is what turns a permanent silent error into a bounded one.
The repairs matter; the count matters more. A reconciliation run that repairs zero documents means your event pipeline is healthy. A run that repairs four hundred means it is not — and you would never otherwise have known, because the safety net was quietly hiding the bug. Alert on the discrepancy rate, not just on job failure.
A full inventory comparison over a million documents is expensive and will rate-limit your source systems. The practical shape:
An important framing first: ordinary updates need no downtime machinery at all. A vector database accepts writes while serving reads, and a query arriving mid-update sees either the old chunk or the new one — both are valid answers and nobody notices.
The hard case is a full rebuild: a new embedding model, or a changed chunking strategy. Then every vector has to change together, because vectors from two different models cannot be compared to each other.
Ordinary updates need none of this. A vector database accepts writes while serving reads, and a query that arrives mid-update sees either the old chunk or the new one — both valid. Blue-green is for the hard case: a change where every vector must move together, because old and new vectors are not comparable.
Blue-green is the default, but it is not always affordable — two full indexes at 200 GB each is real money. Know the alternatives and their tradeoffs.
| Pattern | How | Cost | Risk | Use when |
|---|---|---|---|---|
| Blue-green | Full second index, dual-write, shadow read, alias flip | 2× storage during the migration | Lowest — instant rollback | Embedding-model change, chunking change, anything index-wide |
| Partial shadow index | Build green for a subset — one tenant, one document type — compare, then expand | Small | Low, but you validate on a slice that may not represent the whole | Validating a risky change cheaply before committing to it |
| In-place rolling | Re-embed chunk by chunk into the same index | 1× storage | High — the index temporarily mixes embedding models, so distances are meaningless and recall is degraded and unpredictable throughout | Almost never for a model change. Acceptable for metadata-only updates |
| Dual-index read-merge | Query both, merge results, retire the old once the new covers everything | 2× query cost | Medium — score comparability across indexes is not guaranteed | Adding a new corpus rather than replacing one |
“I would re-embed in place, chunk by chunk, so there is no extra storage.” It sounds efficient and it is usually wrong: during the migration your index holds vectors from two different models, distances between them are meaningless, and search quality is degraded for the entire duration in a way you cannot measure or bound.
Say why it is tempting and why you would reject it. That is a stronger answer than never mentioning it, because it shows you considered the cheap option rather than reciting the expensive one.
Interviewers sometimes push on whether zero downtime is worth it. Have the honest breakdown:
Against that: an unplanned reindex without it means either an outage or serving degraded results for hours. If the system is internal and used by two hundred people, an announced Saturday-night window may genuinely be the right call. Saying so shows judgement rather than reflexive best practice, and it is the answer that distinguishes an architect from someone reciting a pattern.
| Symptom | Most likely cause | What to check first |
|---|---|---|
| The assistant quotes a policy that was withdrawn months ago | Stale orphan chunks — the delete subtraction is missing or IDs are positional | Whether the diff computes stored_ids − parsed_ids, and chunk-count
drift per document |
| A one-line edit triggers thousands of embedding calls | Boundary cascade — offset-based cut points | Embedding calls per changed document. It should be one or two |
| A formatting-only save re-embeds a whole document | Hashing un-normalised text | Whitespace collapsing, line endings, unicode normalisation form |
| Re-ingesting an unchanged document still re-embeds everything | The hash covers something nondeterministic — generated context, a timestamp, a parser version string inside the text | What exactly goes into the hash input |
| Feedback and citations point at the wrong text | Positional IDs | Whether chunk_id contains an ordinal scoped to the document rather than to
the section |
| Edits take forty minutes to appear | Volume or amplification — distinguish them first | Calls per changed document. If it is high, it is cascade. If it is low, profile the stages and check whether a backfill is sharing the queue |
| A document silently disappeared from the index | A bad parse produced an empty section list and the diff deleted everything | Whether the chunk-count guard is in place, and whether deletes are tombstones |
| Chunk-ID overlap dropped across thousands of documents overnight | A parser upgrade or a vendor export change moved every heading | The per-source overlap metric, and parser_version on the canonical
documents |
| Two chunks share an ID | The similarity matcher claimed one stored ID twice | Whether matching is strictly one-to-one and greedy by score |
| A deleted document came back | Race 2 — a retried update crossed a delete | Whether deletes are versioned and terminal |
ArchitectYou already hash chunks. Why does inserting one paragraph still re-embed the whole document?
Because hashing detects change, it does not prevent it. With offset-based boundaries, inserting text shifts every downstream cut point, so the content of every window genuinely differs and the hashes genuinely differ.
The fix is boundary stability — structural anchors, or content-defined boundaries with a rolling hash — with hashing layered on top to suppress the writes. On a 50,000-document corpus at five percent hourly churn that is the difference between 200,000 re-embeds an hour and about 3,750.
ArchitectWalk me through what happens when a document is updated.
Parse to canonical form, compute a stable ID and a normalised content hash per section, read the stored ID and hash set for that document, then diff. New ID means insert and embed; same ID with a different hash means re-embed; same ID and hash means skip with no embedding call at all.
Then subtract: any stored ID absent from the parse is a deleted section, so tombstone it. Bump the document version, and partition the queue by document ID so two edits to the same document cannot interleave.
ArchitectHow do you make sure a deleted paragraph stops being retrieved?
Diff the parsed ID set against the stored set for that document and tombstone anything missing, in the same transaction as the upserts. Reconcile nightly against the source to catch dropped delete events.
I would also alarm on chunk-count drift per document, because stale orphans are the highest-severity failure mode in RAG — the system answers confidently from retired content and nothing looks broken.
ArchitectWhy tombstone rather than delete outright?
Two reasons. A graph index weaves vectors into a navigable structure, so removing nodes means repairing edges, and doing that continuously under live traffic degrades recall — so you batch it into compaction.
And it is reversible. A bad parse producing an empty section list would otherwise delete a whole document, and recovering from that means re-embedding everything rather than flipping a flag.
ArchitectAn editor saves a document with no real change. What happens?
Nothing. The normalised content hash is identical, so every chunk is skipped and there are zero embedding calls. If it did trigger re-embeds, my normalisation is wrong — usually whitespace, line endings or unicode form.
ArchitectWhat is the downside of structure-aware chunking?
It depends on parse quality. A malformed heading tree, or a PDF where headings are just bold text, means unstable IDs — which means silent full re-embeds and duplicate chunks.
So I would add a guard: if a document’s chunk-ID overlap with its previous version drops below about seventy percent, flag it for review rather than blindly rewriting. Fixed-size chunking is dumber but it never surprises you, and that is a genuine argument in its favour on a corpus you cannot parse reliably.
ArchitectHow do you switch embedding models with no downtime?
Blue-green. Build the new index from the stored canonical layer rather than re-crawling sources, dual-write live edits to both during the backfill, shadow-read production queries against green and compare on the labelled set, then flip an alias atomically and keep the old index warm for a day.
Chunks are stamped with the embedding-model version so I can prove the index is homogeneous — mixing models in one index is silently catastrophic, because distances between vectors from different models are meaningless.
Eng managerYour ingestion queue is backed up and edits are taking forty minutes to appear. Diagnose.
First establish whether it is volume or amplification, because the fixes are completely different. If a small number of document edits is producing a huge number of embedding calls, that is boundary cascade and the fix is structural. The metric is embedding calls per changed document: it should be one or two.
If it is genuine volume, profile the expensive stage — usually layout parsing or OCR rather than embedding. Then parallelise per document since documents are independent, tier the parsing so only complex documents take the expensive path, and if freshness is the hard requirement, split the queue so edits jump ahead of bulk backfill.
Eng managerHow would you detect that ingestion has silently broken?
Four signals, and I would want all four on one dashboard. Chunk-count drift per document. ID-overlap rate per source system, which catches parser regressions across thousands of documents at once. Ingest lag — time from source edit to searchable. And a small gold set run in CI against a test index, so a recall drop fails the build before it reaches production.
The reconciliation discrepancy count is the fifth, and it is the one that tells you whether the other four are being fed by a healthy event pipeline or by a safety net quietly papering over it.
Eng managerIs zero-downtime reindexing worth building?
It depends on who is affected and how often you will reindex. It costs dual-write complexity, double storage for the window, doubled query load during shadow reads, and days to weeks of engineering the first time.
For a customer-facing system, or one that will migrate models more than once a year, yes — build it once and every future migration is cheap. For an internal tool used by two hundred people, an announced Saturday-night window is genuinely the right call, and I would say so rather than build machinery nobody needs. I would still build the alias layer though, because that part is an afternoon and it is what makes rollback possible at all.
Should the content hash include the metadata, or only the text?
Only the text that gets embedded, and normalise it first. Metadata changes — a new ACL tag, a corrected date — should update the row without re-embedding, because the vector would be identical. Keep a separate cheap check for metadata drift if you need one, but never let a permission change trigger a GPU call.
What happens to feedback and citations when a chunk is legitimately re-embedded?
They stay valid, because the identity did not move — that is precisely what stable identity buys you. What you should also store is the version at which the feedback was given, so you can tell whether a thumbs-down refers to text that has since been rewritten. Feedback on superseded content is not wrong, it is just about a different version.
Is a nightly full crawl really necessary if we have webhooks?
Yes, and the reason is empirical rather than theoretical: webhooks drop events during source-system incidents, deploys and rate-limit episodes, and they do it silently. The crawl is not there to do the work, it is there to measure whether the work happened. A run that repairs zero documents is the outcome you want and the proof you need.
How large should the similarity-match threshold be?
Measure it rather than adopt a number. Take a sample of real edits from your corpus, compute the similarity between each section before and after, and look at the distribution against the similarity between genuinely unrelated sections. The threshold goes in the gap. Around 0.8 to 0.9 on normalised cosine is a common landing point, but a corpus of short boilerplate sections will need a higher one because unrelated sections there are already similar.
Can I avoid all of this by rebuilding the whole index nightly?
For a small corpus, genuinely yes, and it is a perfectly respectable answer — simpler, fewer failure modes, no diff to get wrong. It stops working when the rebuild no longer fits in the window, or when freshness needs to be minutes rather than a day. Give the crossover explicitly: on the reference stack a full re-embed is millions of calls, so the nightly rebuild dies somewhere in the low hundreds of thousands of chunks.
Where does the version fence actually live?
Two places. On the write path, a document-level version record that every write checks before committing. On the read path, a predicate that filters chunks to the latest committed version. Both are cheap individually; the reason it is a decision rather than a default is the read-path predicate, which is on every single query forever.
Does a tombstoned chunk still cost memory?
Yes, and this is the connection to capacity planning. The vector stays resident and its graph edges stay in place until a compaction or rebuild reclaims them. On the reference stack, dead records are typically the second-largest line in the memory budget — larger than the graph and the metadata combined. Compaction policy is therefore a bigger memory lever than any index parameter.
What if the source system has no change notification at all?
Poll with whatever cheap signal exists — last-modified, etag, a content length, a directory listing — and reconcile more aggressively to compensate. The important part is to be explicit about the resulting freshness bound: “this source is polled hourly, so an edit is visible within an hour” is a design statement you can put in front of stakeholders. Silence about it is how you end up owning an expectation nobody agreed to.
How do I migrate to stable IDs on a system that already uses positional ones?
Treat it as a full reindex behind blue-green, and accept that historical citations and feedback cannot be reliably remapped — because the whole problem is that the old IDs did not identify anything stable. What you can do is snapshot the old ID to text mapping before the cutover, so at least existing bug reports can be interpreted afterwards. Say that plainly: some history is not recoverable, and pretending otherwise is worse than losing it.
“The thing that separates a demo from a system here is that documents change. If chunk boundaries depend on token offsets, inserting one paragraph shifts every cut point below it, so seventy-eight of eighty chunks re-embed even though the text did not change. Hashing detects that, it does not prevent it — what prevents it is cutting on structure so the blast radius is bounded to one section.
Identity is the other half. IDs must come from what a section is, not where it sits: a source anchor if the system gives me one, otherwise a hash of the document ID plus the section path plus an ordinal scoped inside the section. Positional IDs cause wrong overwrites, orphans on shrink and broken citations, and all three fail silently.
Then the diff: insert, update, skip, and the subtraction that produces tombstones. Guard rails in front of the write so a broken parser cannot delete a document. Queue partitioned by document ID with a version fence. Nightly reconciliation whose real output is a discrepancy count. And for a model change, blue-green from the stored canonical layer with a shadow read before the alias flip.”
| Thread from this document | Resolved in |
|---|---|
| Heading detection is what stable IDs depend on | 02 · Parsing hard content |
| Permission changes must not trigger re-embedding | 04 · Access control and freshness |
| Why vectors from two models cannot be mixed | 08 · Fine-tuning and migration |
| Why deleting from a graph index is expensive | 09 · Flat, IVF and HNSW |
| What tombstoned chunks cost in memory | 12 · Quantisation and capacity |
| The gold set that gates a reindex | 16 · Evaluation and observability |