Track A · Document 02 · Ingestion and chunking
Where enterprise RAG actually dies: a page format that records ink rather than meaning, tables that lose their labels, and twelve source systems that all want their own code path.
A PDF is a description of where ink goes on a page. It has no concept of a paragraph, a heading, a table, or a reading order. Run a naive text extractor over it and you get characters in roughly the order they were drawn — which, for anything other than a single column, is not the order a human reads them.
Chunking a PDF is a parsing problem before it is a splitting problem, and your chunking strategy is downstream of parse quality. Candidates who jump to chunk sizes have skipped the place where the failure actually happens. In most enterprise RAG projects, more engineering time should go into this document than into everything in Track B combined.
Four ways a PDF defeats a chunker, and none of them are fixed by a chunk-size setting. Chunking a PDF is a parsing problem before it is a splitting problem — and candidates who jump straight to chunk sizes have skipped where the failure actually happens.
Somebody has photographed every page of every book and thrown away the books. The photographs are perfectly sharp, and they contain no information about which marks are a chapter title, which are a footnote, and which two columns are separate thoughts. Your first job is not to catalogue the library. It is to reconstruct what the pages actually said.
Four stages, and the third one — the router — is the one that removes two of the four failure modes outright.
Layout detection is a different operation from text extraction, and costs twenty to fifty times more. Section 8 is about deciding which documents deserve it.
| Stage | What it does | What happens if you skip it |
|---|---|---|
| Layout detection | Reads page geometry and emits typed blocks with a reading order derived from the layout | Failure 1. Columns interleave, and no chunk size recovers it |
| Strip the furniture | Detects repeated headers, footers and page numbers and moves them to metadata | Failure 4. Every embedding carries the same noise, and the whole document looks self-similar |
| Route by block type | Prose to the chunker, tables to table handling, figures to caption handling | Failures 2 and 3. The splitter cuts a table and a number loses its labels |
| Emit canonical form | One structure regardless of source format, persisted rather than passed through | Twelve chunkers, and every future migration re-crawls the sources |
Several tools do the first stage — Docling, Unstructured, LlamaParse and Azure Document Intelligence among them. Section 9 is about how to pick, and the answer is never a name without a bake-off.
Tables are where the numbers live, and numbers are what people ask about. There are three serialisations and the right answer is usually more than one of them at the same time.
Once tables are extracted as structured rows you are one step from loading them into a warehouse. The mature architecture stores every table twice — as chunks for semantic lookup, and as rows for aggregation — with a classifier routing aggregation-shaped queries to SQL.
Sometimes a table is two hundred rows and row-level chunking is not appropriate — a matrix where the rows only mean something together, for instance. If it must be split, never split blind:
part 2 of 4.That last point is the one that turns a bad situation into a workable one: search can hit fragment three, and the model still reads the whole table.
“Users say the bot gets numbers wrong from our financial reports.” Walk the pipeline backwards, starting with what the retrieved chunk actually contains. Nine times out of ten the table was flattened into prose or split mid-way, so the number has lost its row and column labels. That is an ingestion bug, not a model bug, and no amount of prompt work or model upgrading will fix it.
Then add an ingestion test that asserts specific known values are retrievable from specific documents, so it cannot regress silently. Volunteering the regression test is what separates the answer from the diagnosis.
Interviewers often ask you to trace one concrete example rather than describe a policy. This is the trace worth having ready, and it takes about ninety seconds to deliver.
Four rows, two chunks, one database write. The database write is the part nobody mentions and the part that stops the next requirement becoming a rebuild.
Once tables are extracted as structured rows, you are one step from loading them into a warehouse. The mature architecture stores every table twice: as chunks for semantic lookup, and as rows in a database for aggregation, with a classifier routing aggregation-shaped queries to SQL.
Saying that you would persist the structured form even though RAG does not need it shows you are thinking about the next requirement — and the aggregation requirement always arrives.
A scanned PDF has no text layer at all, so extraction returns nothing and you have to OCR. That is a different problem from parsing, and it has a property none of the others do: the errors are permanent.
A misread digit is now what your system believes, forever.
Nothing downstream can detect it. Retrieval, reranking and the model will all faithfully report the wrong number.
Layout detection has to work off the image rather than off any embedded hints.
Table extraction from a scan is materially harder than from a native PDF, and it fails differently.
Store a confidence score per block.
Route low-confidence pages to human review rather than silently ingesting them.
Roughly 300× the cost of text extraction per document.
Which is why section 8 exists.
OCR quality is an ingestion SLO, not a detail. If your OCR is 94 percent accurate on numbers, your financial assistant is 94 percent accurate at best, no matter how good the retrieval and the model are. That ceiling is set at ingest and cannot be recovered afterwards.
PDFs are the worst case for the identity problem in document 03, because there are no anchors and parse output can shift between runs: a different library version detects headings differently, and every derived ID changes with them. So for PDFs specifically:
Often skipped, and a good differentiator when it comes up — because there are four answers with genuinely different cost profiles and most people know one.
| Approach | How it works | Cost | When |
|---|---|---|---|
| Caption-based | Index the caption plus the surrounding narrative as a chunk, with a pointer to the image | None | The baseline. Captions are usually descriptive, and this covers most needs |
| Vision description at ingest | Send the image to a multimodal model and store its description as the chunk text | One call per figure, plus a permanent hallucination surface | Charts where the caption says “Figure 7” and nothing else |
| Multimodal embeddings | Embed the image itself into the same space as the text | Another model, and a whole second evaluation problem | Diagram-heavy corpora such as engineering documentation |
| Pass the image at generation | Retrieve on the caption, hand the actual image to a vision-capable model when answering | Tokens at generation time only | Often the best quality per unit of effort |
“Baseline is caption plus surrounding text with a pointer to the image, and pass the image at generation time if the model is multimodal. I would only add vision-generated descriptions at ingest if I measured that caption-based retrieval was missing figure-related queries — because it is a model call per figure and it introduces a hallucination surface that is permanent once indexed.”
Enterprise corpora are never one format. Confluence pages, Word documents, native and scanned PDFs, SharePoint lists, HTML exports, email, Markdown in Git, spreadsheets, chat threads, tickets. The naive path is one chunking implementation per format, and it makes ingestion the least maintainable part of the system.
Stable IDs need a section path. Atomic tables need a typed block. Parent–child needs a hierarchy. ACLs need a connector that understands the source’s permission model. Without a canonical form, every one of those has to be reimplemented per format.
| Downstream need | What the canonical form provides |
|---|---|
| Stable identity | A section path, whatever the source was |
| Atomic tables | An explicit block type, so the chunker never has to guess |
| ACL enforcement | Group tags attached at parse time by the connector that understands the source’s permission model |
| Parent–child retrieval | A section hierarchy |
| Citations | Page numbers and anchors carried on every block |
Without normalisation, every one of those has to be reimplemented per format — and they will drift apart, because nobody updates twelve implementations in lockstep.
Each source needs a connector doing four jobs. Naming all four signals experience, because this is where most ingestion engineering time actually goes:
Normalisation is lossy. Converting a Confluence page drops macros and embedded dashboards; converting a spreadsheet drops formulas. So the canonical schema is a real design decision: too thin and you lose information you later need, too rich and every parser becomes complex.
Start deliberately thin — heading, paragraph, table, list, figure — and extend only when a measured retrieval failure traces back to something you dropped. And watch for the failure mode: if adding a source forces a change to the chunker, the abstraction is wrong, and you are on your way to twelve chunkers wearing a trenchcoat.
Layout models are twenty to fifty times slower than raw text extraction, and OCR is slower again. At a million documents, that ratio is not a detail — it is the whole ingestion architecture. So you do not apply one strategy to everything. You route.
The escalation rule matters as much as the tiers. Triage will misroute some documents, so tier 1 needs a sanity check — no headings detected, a suspiciously low text-to-page ratio, obvious column interleaving — that escalates the document to tier 2 automatically and logs it. Without escalation, a misrouted document is badly parsed forever and nobody finds out.
| Signal | Points to |
|---|---|
| File type is Markdown, HTML, Word or a native PDF with a text layer | Tier 1 |
| Text-to-page ratio is near zero | Tier 3 — there is no text layer, it is a scan |
| Extracted text shows column interleaving, or no headings at all | Tier 2 |
| The document contains ruled regions or many aligned numbers | Tier 2 — there are tables to lose |
| The source system is known to produce clean structure | Tier 1, by configuration |
Triage will misroute some documents. Without an escape hatch, a misrouted document is badly parsed forever and nobody ever finds out. So tier 1 output gets a sanity check — no headings detected, suspiciously low text-to-page ratio, obvious column interleaving — and anything that fails it is escalated to tier 2 automatically and logged. The log is what lets you tune the triage rules instead of guessing at them.
Document 01 covers what contextual chunking is and where it sits on the strategy ladder. This section is about the part that decides whether you can actually run it: what it costs, and the four ways it goes wrong in production.
The recap in one line: at ingest, send each chunk plus its surrounding document to a cheap model, ask for one or two sentences situating the chunk, prepend that, and embed the combined text. The chunk that says “It may be extended by a further 4 weeks on medical grounds” now carries the words maternity, leave, extension, Acme and policy in its vector, so the query finally matches it.
| The cost | What it actually means | The mitigation |
|---|---|---|
| Ingest spend | One model call per chunk. Ten million chunks is ten million calls | Use a small cheap model, and prompt-cache the document — you send the same full document with each of its chunks, so cache it once and pay a fraction thereafter. That caching is what makes the technique economically viable at all. Use the batch API for the backfill |
| Ingest latency | A model call now sits in your write path, fighting any real-time freshness requirement | Apply it asynchronously: index the plain chunk immediately so it is searchable, then upgrade it when context generation completes. Or apply it only to a high-value subset |
| Nondeterminism | Generated context varies between runs, so re-ingesting an unchanged document produces a different hash and triggers a spurious re-embed of the whole corpus | Hash the original chunk text, not the augmented text, and only regenerate context when the original changed. This is the one that bites, and it ties straight back to the diff in document 03 |
| Hallucinated context | The model writes “this section concerns termination benefits” for a chunk about leave. That chunk is now silently retrieved for the wrong queries, permanently | Constrain the prompt hard — derive from the document only, no inference, one to two sentences, no speculation. Sample-audit the generated contexts. Treat a spike in retrieval of a previously cold chunk as a signal worth investigating |
Structural prefixing. Mechanically prepend the section path you are already storing:
Acme HR Policy 2024 > 3 Leave > 3.2 Maternity Leave in front of the chunk.
Zero cost, fully deterministic, and it recovers most of the missing keywords. This should be your
baseline, and contextual chunking is only justified if it beats this measurably.
Two other cheap approximations: overlap carries some antecedent context, crudely; and parent–child solves the generation half of the problem but not the retrieval half, because the child vector is still context-free. Contextual chunking and parent–child are complementary, not alternatives — a point worth making, because interviewers sometimes offer them as a choice.
Interviewers often ask which tool you would pick. Naming one is the weaker answer; describing the bake-off is the stronger one, because the right tool genuinely depends on the corpus.
Scoring per capability. A parser that is excellent at text and poor at tables is perfectly fine for a policy corpus and disqualifying for a financial one. A single accuracy number hides exactly the distinction you need.
The determinism check. Run the same document through twice and diff. Non-deterministic output means unstable chunk boundaries, which means unstable chunk IDs, which means silent full re-embeds on every ingest run. It takes ten minutes to check and almost nobody runs it — and it connects straight to document 03.
| Capability | How to score it | Disqualifying for |
|---|---|---|
| Reading order | Does the extracted text match the human reading order on the two-column report? | Any multi-column corpus |
| Table fidelity | Headers present, cells aligned, no rows merged or dropped | Financial, scientific or operational corpora |
| Heading detection | Correct level, and stable between runs | Anything relying on structural chunking or parent–child |
| Furniture stripping | Are page numbers and running heads out of the text? | Long documents with heavy furniture |
| Latency and cost per page | Measured, at your page sizes | Large corpora — it sets the whole tiering design |
| Determinism | Same input twice, diff the output | Any system with incremental updates — which is all of them |
Everything above, on one line, plus the return path that makes it affordable to run twice.
Persisting the canonical layer is what makes every later migration affordable. It is one line on a diagram and it is the difference between a model change that takes hours and one that takes a week of re-crawling.
| Symptom | Most likely cause | What to check first |
|---|---|---|
| The assistant reports wrong numbers from reports | Table flattened into prose, or split without its header | Read the retrieved chunk. If the number has no adjacent row and column label, it is an ingestion bug |
| Answers from one document all sound similar and rank similarly | Running heads and footers not stripped, so every chunk shares the document title | The first and last 60 characters of ten chunks from the same document |
| Sentences in retrieved chunks are spliced together nonsensically | Multi-column reading order | Whether the parser is layout-aware, and whether that document went through tier 1 |
| Some documents return nothing at all, ever | Scans with no text layer that were routed to tier 1 and produced empty output | Text-to-page ratio at ingest, and whether the escalation rule fired |
| Nightly ingestion time is growing faster than the corpus | Everything is going through the expensive tier, or unchanged documents are being reprocessed | Profile the stage, then the tier mix, then the diff |
| A dependency bump changed a large number of answers | A parser upgrade moved heading detection, so chunk boundaries and derived IDs moved | parser_version on the canonical documents, and whether the upgrade was
treated as a reindex |
| Re-ingesting an unchanged document re-embeds everything | The hash covers generated context, which is nondeterministic | Whether the content hash is taken over the original chunk text or the augmented text |
| Figure-heavy documents answer poorly | Captions are uninformative — “Figure 7” and nothing else | Sample the captions before reaching for a vision model |
ArchitectUsers say the bot gets numbers wrong from our financial reports. Diagnose it.
I would walk the pipeline backwards, starting with what the retrieved chunk actually contains. Nine times out of ten the table was flattened into prose or split mid-way, so the number has lost its row and column labels — which makes it an ingestion bug rather than a model bug.
The fix is layout-aware parsing with tables extracted atomically and serialised row-level with headers repeated. Then I would add an ingestion test asserting that specific known values are retrievable from specific documents, so this cannot regress silently.
ArchitectHow would you handle a 300-page PDF with mixed content?
Layout-parse into typed blocks, strip repeated headers and footers, then route by type: paragraphs through normal chunking, tables atomically, figures by caption. Use parent–child so retrieval is precise but generation sees the surrounding section.
And I would expect to spend more engineering time on this stage than on the retrieval side, because that is where the quality actually comes from.
ArchitectWhich PDF parser would you choose?
I would not name one without seeing the corpus. I would build a gold set of twenty representative documents with known correct extractions, and score candidates per capability — table fidelity and reading order especially — rather than on one accuracy number.
Cost and latency matter as much as accuracy, because layout models are twenty to fifty times slower than text extraction, and at a million documents that changes the ingestion architecture. I would expect to end up tiering: cheap extraction for simple documents, the expensive layout model only where triage says it is needed. And I would run the determinism check, because non-deterministic parse output means unstable chunk IDs.
ArchitectYou have twelve source systems. How do you structure ingestion?
One connector per source, each doing four jobs: fetch, detect change, extract permissions, parse. All of them emit one canonical representation, and there is a single chunking path downstream that is format-agnostic.
And I would persist that canonical layer in object storage, because rebuilding from storage rather than re-crawling is what makes model migrations and chunking changes practical at all. Re-crawling a million sources is slow, rate-limited, and often something the source system will not tolerate.
ArchitectHow do you add a thirteenth source system?
Write one connector. If the canonical schema is right, nothing downstream changes. If adding a source forces changes to the chunker, that is a signal the abstraction is wrong — either the schema is too thin, or someone has leaked format-specific logic downstream.
ArchitectWhat is the risk of the canonical-form design?
The schema becomes a bottleneck. Every new source pressures it to grow, and a badly scoped schema either loses information you later need or accumulates special cases until it is twelve chunkers wearing a trenchcoat.
I would keep it deliberately minimal — heading, paragraph, table, list, figure — and extend only when a measured retrieval failure traces back to something the schema dropped. The discipline is that the schema grows on evidence, not on request.
Eng managerYour nightly ingestion takes eight hours and is growing. What do you do?
Profile first, because the intuition is usually wrong: it is almost always layout parsing or OCR, not embedding. Then in order: only reprocess documents that actually changed, which is the diff; parallelise per document, since they are independent; and tier the parsing so the expensive path is reserved for documents that need it.
If that is still not enough, separate the pipelines so incremental edits jump ahead of bulk backfill, because those have genuinely different freshness requirements and queueing them together means a backfill delays every edit behind it.
Eng managerHow much of the team’s time should ingestion get?
More than anyone expects, and I would say so at planning rather than discover it at week six. In my experience the pilot exposes parsing problems, not retrieval problems — the retrieval stack is largely commodity and the ingestion stack is entirely specific to your corpus.
Concretely I would staff one engineer on connectors and parsing for the whole first quarter, and I would resist the pull towards retrieval tuning until the ingestion quality is measured. Tuning retrieval on badly parsed content optimises the wrong thing and looks like progress while doing it.
Eng managerHow do you justify the parsing spend to someone who thinks this is an AI project?
With the ceiling argument and one number. If OCR is 94 percent accurate on digits, the assistant is at best 94 percent accurate on any question about a number, no matter which model we buy. That ceiling is set at ingest and cannot be recovered later, so parsing is not a preliminary to the AI work — it is the accuracy budget.
Then I would show the tiering arithmetic, because it reframes the conversation from “why is this expensive” to “which five percent of documents deserve the expensive path”, which is a decision the business can actually participate in.
Can I just use a text extractor and accept the errors?
For a corpus of clean, single-column, table-free documents, yes — and you should, because it is three hundred times cheaper than OCR and forty times cheaper than a layout model. The point of tiering is precisely that. What you cannot do is assume the corpus is that shape without checking, because the failure is silent: text comes out, it just is not the text that was on the page.
How do I detect running heads and footers automatically?
Position plus repetition. Collect the text blocks in the top and bottom bands of every page, normalise digits to a placeholder so page numbers collapse together, and drop any line that appears in the same band on more than about half the pages. Keep the page number itself as metadata — it is what a citation points at, and it is very hard to recover afterwards.
What is the right row threshold for whole-table versus row-level?
Around twenty rows is a reasonable default, but treat it as a configurable and be ready to justify it. It is really set by two things: how much of the context window one chunk may occupy, and whether within-table comparison matters for this corpus. A financial corpus where users compare regions wants whole tables for longer; a parts catalogue where every query is a single lookup wants row-level almost immediately.
Should the canonical form be Markdown or a typed block tree?
A block tree internally, serialised to Markdown when handing content to the model. Markdown is simple, human-readable and models read it natively, but it cannot carry page numbers, anchors or block-level confidence scores — and those are exactly what citation and OCR triage need. Using both, in the right places, is the answer that shows you have built it.
Is storing the canonical layer expensive?
No, and this is worth being concrete about. It is text in object storage — the cheapest tier in the whole system, orders of magnitude below the RAM the vectors occupy. Against that, it removes re-crawling from every future migration. It is close to the best cost-to-benefit ratio available in the ingestion design.
How do I handle a document that is one enormous table?
Row-level chunks with the header repeated, a summary chunk, and the structured rows loaded into a database. Then, importantly, check whether this corpus should be in the RAG system at all: if most queries against it are lookups by key or aggregations, a text-to-SQL path will beat semantic retrieval on both accuracy and cost. Knowing when to route away from RAG is part of designing the ingestion.
Do I need OCR confidence scores if the vendor does not expose per-block ones?
Use what you can get, and construct a proxy where you cannot. Page-level confidence, the proportion of characters that are not in the dictionary, the ratio of digits to letters against what the document type predicts — any of these will separate the obviously bad pages from the rest. The goal is not a precise score, it is a routing decision: which pages does a human need to look at?
Our PDFs are generated from Word by a known template. Does any of this apply?
Much less, and you should exploit that. A known generator means a known structure, so a template-specific extractor will beat a general layout model on both accuracy and cost. Tier that source at tier 1 by configuration rather than by triage. The general machinery in this document is for corpora you do not control — and every enterprise has both kinds.
How do I test the ingestion pipeline?
Golden-output tests on a small fixed set of documents. For each, store the expected canonical form and diff against it on every change — a parser bump, a config change, a new rule. Add value-level assertions for the corpus-specific things that matter: “the Q2 South revenue figure is retrievable from the FY24 annual report”. Those two together catch nearly everything, and they are what turns a parser upgrade from a leap of faith into a diff you can read.
“A PDF only records where ink went, so the first job is reconstructing what the page said. I use a layout model that emits typed blocks with a reading order from the geometry, strip the repeated headers and footers into metadata, and then route by block type — prose to the chunker, tables handled atomically, figures by caption. The splitter never sees a table.
Tables get serialised three ways depending on size: whole as Markdown under about twenty rows, row-level with headers repeated above that, and always a summary chunk, because discovery queries do not match any individual row. The structured rows also go to a database, because the aggregation question always arrives and RAG is the wrong tool for it.
Layout models are forty times slower than text extraction and OCR is three hundred times slower, so I tier: cheap extraction by default, the expensive path only where triage says so, and an escalation rule for what triage gets wrong. And everything lands in one canonical form that I persist — because rebuilding from storage instead of re-crawling is what makes the next migration affordable.”
| Thread from this document | Resolved in |
|---|---|
| Parser upgrades move boundaries, and boundaries carry identity | 03 · Identity, updates and deletes |
| Connectors extract permissions; where do they get enforced? | 04 · Access control and freshness |
| Running the embedder over millions of chunks | 07 · Token limits and the pipeline |
| Row-level chunks multiply the vector count | 12 · Quantisation and capacity |
| Table lookups by part number or code | 15 · Hybrid retrieval and reranking |
| The parser bake-off, as a measurement discipline | 16 · Evaluation and observability |