Track A · Document 01 · Ingestion and chunking
What one retrievable record should be, why the obvious answer is wrong, and the design that dissolves the central tradeoff instead of accepting it.
A retrieval system finds things by comparing meaning. To do that it turns text into a vector — a list of numbers standing for what the text is about — and two texts about the same thing produce vectors that sit close together.
The catch is that a vector is an average. Feed in a forty-page HR policy and you get one vector meaning “HR policy in general”. It is close to everything and specific to nothing.
So you cut the document into pieces and embed each piece separately. Each vector is now about one thing, and matching becomes precise. That is the whole idea, and the picture below is worth sitting with for a minute, because every later decision in this runbook is downstream of it.
Everything downstream can only reorder what chunking made findable. A reranker cannot promote a chunk that was never retrieved, and a larger context window cannot rescue a vector that matches everything weakly.
Index the library by book title and every search returns whole books: you found the right book, and you still have to read it. Index by individual sentence and you find the exact sentence, but you have lost the chapter it belonged to, so you cannot tell what it was about. Chunking is choosing where between those two extremes to sit — and section 6 is the trick for sitting in both places at once.
You retrieve what you embed. The chunk is your unit of retrieval, so chunking decides what your system is capable of finding at all. Everything downstream — reranking, hybrid search, a bigger context window, a better language model — can only reorder what chunking made findable.
Two forces pull in opposite directions. Small chunks match precisely and cost little per query, but lose the context that makes them meaningful. Large chunks keep context but their vectors are averages, so they match everything weakly, and you pay for every token on every query, forever.
Slide the dial and watch all three numbers move at once. There is no setting where they all go the right way — that is what makes it a tension rather than a tuning problem.
This is the central tension, and it is genuinely a tension — there is no setting where every arrow points the right way. Section 6 shows the design that stops you having to trade.
| Small chunks, 100–250 tokens | Large chunks, 1,000+ tokens | |
|---|---|---|
| Embedding quality | Focused. The vector is about one idea, so it matches strongly and precisely. | Diluted. The vector averages five topics, so it matches everything weakly. |
| Context | Lost. “It must be renewed within 30 days” — renewed what? | Preserved. The model can see what “it” refers to. |
| Cost per query | Low. Few tokens sent to the model. | High, and recurring. You pay for every token in the context window, on every query. |
| Latency | Low. | Higher. More tokens to process before the first output token appears. |
| Answer-quality risk | The model lacks context, so it hedges or guesses. | Lost in the middle. Models attend less reliably to content buried in long contexts. |
| Number of vectors | Many. Bigger index, more memory, slower to build. | Few. Smaller index. |
Most candidates treat chunk size as a number to tune — “512 with 50 overlap” — and stop there. The senior move is to notice that matching a query and answering a question are two different jobs, and that we have been forcing one object to do both. That single observation leads straight to parent–child retrieval, and it is the difference between an answer that sounds read and an answer that sounds built.
Seven approaches, from the one that needs no parser to the one that needs a model you control. Know all seven and when each is right. In practice you will reach for rung 2 and rung 5, and you should be able to say why the others lost.
Rungs 6 and 7 are two answers to the same problem — a chunk that has lost the context it needed — and they cost completely different things. Knowing both, and which one a given stack can actually run, is a strong signal.
Abstract descriptions of chunking strategies all sound reasonable. Watching what each one does to a single page — particularly to a table — is what makes the difference concrete.
Pick a strategy above and watch what happens to the table. The table is the tell: any scheme that cuts through it will answer “how many weeks paid?” with a row and no header.
Overlap is insurance against an answer that straddles a boundary appearing in neither chunk. It is not a fix for context loss: a 50-token overlap almost never contains the antecedent of a pronoun three paragraphs up.
And it is not free. Fifteen percent overlap means fifteen percent more vectors, fifteen percent more index memory, fifteen percent more embedding spend, and more duplicate hits to deduplicate at retrieval time. Treat it as a small insurance policy with a premium, not as a solution.
With structural chunking, overlap is often zero — because sections do not split sentences in the first place. If you are paying for overlap, it is usually a sign you are cutting in the wrong places.
Step six is deliberately last. It is the most expensive step and the one most likely to be added on faith; document 16 is about how to earn it.
This is the shift that separates a prototype from a production system. In a demo a chunk is a piece of text. In an enterprise system a chunk is a database row carrying everything that every downstream feature depends on — and most of those fields cannot be added later.
Two families, and it pays to name them separately in an interview. Filtering metadata constrains what can be retrieved and must be applied inside the search. Provenance metadata makes citation, audit and debugging possible, and answers “why did the assistant say that?”
Store groups in acl_tags, never user IDs. Store user IDs and every
group-membership change becomes a re-index of every affected chunk — a permission change
should be a directory lookup at query time, not an ingest job.
Filtering metadata must be applied inside the search, not after it. Filtering afterwards can return an empty page to a user who did have permitted matches, and it means some component saw a list of documents that user is not entitled to. Document 04 is where this gets its own treatment.
You can recover from a mediocre chunk size: sweep it and rebuild, an afternoon of work. You cannot recover from metadata you did not capture, because it is not in the index and is often no longer obtainable from the source system either. Design the record before you design the splitter.
There is no universally correct number, but there is a correct process: the size follows from the shape of the questions people ask. If you do not know the query mix, you are not choosing a chunk size, you are picking one.
| Query type | Example | What chunking should do |
|---|---|---|
| Fact lookup | “How many days notice for maternity leave?” | Small chunks, 100–250 tokens, with a parent fallback. Precision matters most. |
| Procedural | “How do I request leave?” | Section-level chunks. A procedure split in half is worse than useless — it is confidently incomplete. |
| Comparative | “How does our India policy differ from the UK one?” | Chunking cannot solve this alone. Needs query decomposition and several retrievals. |
| Summarisation | “Summarise the risks in this contract.” | Retrieval by chunk is the wrong primitive entirely. Route to document-level summarisation. |
| Table lookup | “What was South region Q2 revenue?” | Row-level chunks with the header repeated on every row. |
| Aggregation | “Which region grew fastest?” | Not a retrieval question. This is text-to-SQL territory, and saying so is the right answer. |
“Chunk size is not chosen in the abstract, it is fitted to the query distribution. Before picking a number I would want to know what people actually ask. If a meaningful share of queries turn out to be summarisation or aggregation, chunk size is the wrong lever entirely and I should be designing a router, not tuning a splitter.”
100–250 tokens.
Small enough to be about one thing. This is what gets embedded and searched.
One section, capped at 1,000–1,500 tokens.
This is what the model reads. If a section exceeds the cap, use a sliding window around the child rather than the whole section.
10–15% for fixed-size; often zero for structural.
Sections do not split sentences, so the insurance is usually unnecessary.
30–50 candidates, 3–8 parents.
Retrieve deep, rerank, then send few. Depth is cheap in the index and expensive in the prompt.
Then measure. These are a starting point for a sweep, not an answer — and the sweep is in document 16.
Matching a query and answering a question are different jobs with different requirements. So stop making one object do both. Search on a small chunk, feed a large one to the model. The tension in section 2 does not get traded off — it gets dissolved.
Section 3.2, Maternity Leave, about 1,200 tokens. Split into children of roughly 150 tokens:
child_1 Eligible employees are those who have completed 80 days of service
in the preceding 12 months.
child_2 The application must be submitted at least 30 days before the
intended start date.
child_3 It may be extended by a further 4 weeks on medical grounds,
subject to approval.Only the children are embedded. The parent — the whole section — is stored as text, not as a vector.
Ask “how much notice do I need to give for maternity leave?” and it matches
child_2 cleanly, because that vector is 150 tokens about exactly one thing. Had the
whole 1,200-token section been a single vector, the notice signal would have been one-eighth of
a blob also covering eligibility, extensions, pay and return to work.
Then at generation time you do not send child_2. You send the whole parent
section, so the model can also see the eligibility conditions and knows what “it”
refers to in child_3.
parent_id, so the 30 hits resolve to their sections.Say “deduplicate” unprompted and you have signalled that you have built this. It is the one step that never appears in the tutorial version and always appears in the production version.
vector index (searched)
child_2 → vector, parent_id, doc_id, tenant_id, acl_tags
document store (key-value, never searched)
sec-3.2 → full section text, section_path, version, source_urlThat split is the architecture, and it has a consequence worth naming: the parent holds the stable identity, the ACLs and the citation data; the children are disposable. You can re-split them, change the child size, or re-embed them with a new model without invalidating a single citation or feedback record. That decoupling is exactly what you want while you are still learning your query distribution.
| Tradeoff | The number | What you do about it |
|---|---|---|
| Context budget | 8 parents × 1,200 tokens = 9,600 tokens | Cap it: top N parents after reranking, bare children below that line. |
| Parent size | Above ~1,500 tokens | Lost-in-the-middle returns. Use a sliding window around the child instead of the whole section. |
| Extra fetch hop | 5–15 ms | Nothing — but quote the number, because it shows you measured rather than assumed. |
| Duplicate storage | Text stored twice | Nothing. Say you considered it and dismissed it; text is the cheapest thing in the system. |
| Variant | How it works | When |
|---|---|---|
| Sentence window | The child is one sentence; the parent is built on the fly from k sentences either side | Unstructured text with no sections. Needs no parent store at all. |
| Auto-merging / hierarchical | Three levels. Several children of one medium chunk hit → merge up to the medium; several mediums → merge to the large | Adaptive context width. Be honest that the third tier often does not earn its complexity. |
| Summary indexing | Embed a model-written summary of the section; return the full section | Discovery queries — “which document covers X?” Costs one generation call per section. |
“We index the book by paragraph so we can find the exact paragraph, but when we answer we read out the whole page, so the answer makes sense in context.”
Parent–child fixes context at generation time: the model reads the parent. It does not fix context at search time — the child vector still knows nothing about the document it came from. Two techniques fix that, and they cost completely different things.
Same problem, opposite economics. Contextual chunking buys context with generation calls and works anywhere; late chunking buys it with a longer forward pass and works only if you control the embedder.
| Contextual chunking | Late chunking | |
|---|---|---|
| Mechanism | Generate a short situating sentence per chunk and prepend it before embedding | Embed the whole document to token-level vectors, then apply boundaries and pool within each chunk |
| Extra cost at ingest | One generation call per chunk | Longer forward passes; no second model |
| Works with a hosted embedding API? | Yes — any model at all | No. Needs token-level outputs, so effectively self-hosted |
| Needs a long context window? | No | Yes, on the embedding model |
| Effect on index size | Chunks get longer, so slightly larger | None — the vectors are the same size |
| What it is good at | Adding facts the chunk never contained — the document title, the date, the product it applies to | Resolving references the chunk lost — pronouns, “the above”, “this section” |
Before either technique, prepend the section_path you are already storing:
“HR Policy 2024 > 3 Leave > 3.2 Maternity Leave” in front of the chunk text before
embedding. It costs one string concatenation, needs no model, and recovers a surprising share of
what contextual chunking is bought for. Measure that baseline before paying for ten million
generation calls — it is the sort of move that reads as experienced rather than
fashionable.
Knowing when a technique does not apply reads as more senior than knowing the technique. There are four situations where the elaborate answer is the wrong one.
FAQ entries, product listings, resolved support tickets, job postings.
A parent adds no information and costs tokens. Index the item whole.
If the whole document fits comfortably in the context window.
Retrieval is document-level and chunking is a non-issue.
If the answer lives in a known row of a known table.
You do not need semantic search at all. Query the database.
“Which region grew fastest?”
Cannot be answered by retrieving chunks, however they are cut. Text-to-SQL.
Real systems have both shapes at once. A support assistant over help-centre articles and ticket resolutions has one corpus that is long, curated and structured, and one that is short, messy and already the right unit. Chunk them differently and keep them distinguishable. Articles want structural chunking with parent–child; ticket resolutions want to be indexed whole with no parent.
Then separate them at the index level, or at minimum tag them, because the ranking policy differs: articles are authoritative and stable, ticket resolutions are numerous and sometimes wrong. The product will eventually want “official answer first, community answer second”, and that is far easier if the two were never blended into one undifferentiated index.
People repeat this line without being able to defend it. Here is the defensible version, which is what the interviewer is probing for.
An embedding model reads a passage and produces one fixed-length vector — say 1,024 numbers — whether the passage is 20 tokens or 2,000. The model has a fixed budget of representational space and has to spend it covering everything in the passage. Add more distinct topics and each one gets a smaller share.
The score did not change. The competition did. If you have a story about a demo that stopped working when the corpus grew, this is the mechanism, and it is worth telling.
The failure only appears at scale. With ten documents the diluted chunk still ranks first, because nothing competes with it. With a million, dozens of chunks are weakly similar and the right one is buried at rank 40. A prototype that works can become a production system that does not, with no code change and no deployment.
That is also the argument for building a labelled set from the real corpus rather than from a sample: dilution is invisible until there is competition, and a sample of ten documents has none.
Architect answers are quantified. The arithmetic below is worth having ready, because the conclusion is counter-intuitive and it changes what chunk size is: above a certain traffic level it stops being an accuracy decision and becomes a cost decision.
Note what happened there, because it is the single most common sizing mistake: the question was about half a million documents and the answer is tens of millions of records. One question to a stakeholder — how many pages, and roughly how dense? — changes every number downstream by a factor of thirty or more.
The unit price is yours to set — the field above is a placeholder, not a quoted rate. Put your real one in; the shape of the conclusion survives any price, because both columns scale with it identically.
Embedding the corpus is a one-time capital cost. Context is a recurring operational cost that scales with traffic. You embed the same 7.5 billion tokens whichever chunk size you pick, so that line is a wash. But the context line is paid on every query forever, and it is proportional to chunk size.
So above a certain volume, chunk size is a cost decision that happens to affect accuracy, rather than an accuracy decision that happens to cost money. Parent–child is the design that breaks the link, because it lets you set search granularity and context size independently — which is an argument for it that has nothing to do with recall.
Three buckets. A one-time cost to process the existing corpus, dominated by document parsing rather than by anything AI-shaped. A recurring per-query cost, mostly the tokens we send to the model, so it scales with usage and with how much context we choose to send. And an ongoing cost proportional to how often documents change, which is where engineering work on incremental updates pays for itself — done naively that bucket is roughly fifty times larger than done properly.
Then give them the lever: context size is the main dial on the recurring cost, and we can trade it against accuracy explicitly, with numbers, rather than guessing.
Asked to design the whole thing, most candidates list components in whatever order they come to mind. There is a sequence that reads as experienced, and the reason it does is that it runs from irreversible to reversible.
Sequence your answer from irreversible to reversible. It is the same idea as the cost hierarchy on the cover page, applied to the order you speak in rather than the order you change things in.
The diagnostic table. In an interview these come disguised as “we are seeing X, what would you look at?”, and the ability to go straight to a mechanism is worth more than any amount of theory.
| Symptom | Most likely cause | What to check first |
|---|---|---|
| Recall was fine in the pilot and is poor in production | Dilution. Chunks are too large, and there is now competition | Score distribution of the top 50: if the gold chunk scores the same as before but ranks far lower, it is competition, not regression |
| The right section is retrieved but the answer is wrong or hedged | Context loss. The chunk is correct and unintelligible on its own | Read the retrieved chunk cold. If you cannot answer from it, neither can the model. Parent–child or section-path prefixing |
| Table questions answer with a number from the wrong row | The table was cut, so a row arrived without its header | Whether tables are atomic units, and whether the header is repeated on row-chunks |
| The same text appears three times in the prompt | Parent deduplication is missing | Step 3 of the retrieval path |
| Answers are good but the bill is growing faster than traffic | Context size, not model choice | Tokens per query × queries per day. Then chunk size and k |
| Procedural questions get half an answer | A procedure was split across a chunk boundary | Whether cutting is structural or positional; procedures need section-level units |
| Comparative or aggregate questions are always wrong | Not a chunking problem at all | Whether there is a router. These need decomposition or text-to-SQL |
| A parser upgrade silently changed a lot of answers | Chunk boundaries moved, so identity moved with them | parser_version on the record, and whether identity is positional
(document 03) |
Model answers, tagged by the level they are testing. Read them once for content, then use Reveal all answers in the bar above as a toggle and answer each one aloud before you look.
ArchitectHow do you choose chunk size?
Weak: “512 tokens with 50 overlap.”
I would start around there as a baseline, but chunk size is fitted to the query distribution rather than chosen in the abstract. For fact-lookup queries I would go small, 150 to 250 tokens, with the parent section returned at generation time — precision at search, context at answering. If a large share of queries turn out to be summarisation, chunk retrieval is the wrong primitive and I would route those separately. I would validate by sweeping size against recall@10 on a labelled set and take the smallest size where recall plateaus.
ArchitectWhy not just retrieve larger chunks and skip the complexity?
Because embedding quality degrades with length. The vector is an average, so a long chunk matches everything weakly and nothing strongly. You would be trading a retrieval problem you cannot fix downstream for a context problem you can. Parent–child avoids the trade entirely, at the cost of one extra fetch of about ten milliseconds and some duplicate text storage.
ArchitectWhat is the point of overlap, and how much?
It stops an answer that straddles a boundary from being lost in both chunks. Ten to fifteen percent is typical for fixed-size chunking. But it costs proportionally more vectors, more index memory and more embedding spend, and it does not solve context loss — an overlap rarely contains the antecedent three paragraphs up. With structural chunking I would often use zero, because sections do not split sentences.
ArchitectYou have a 200-page document and the answer needs three separate sections. How does chunking handle that?
It does not, and I would not pretend otherwise. Chunking produces candidates; multi-hop needs query decomposition — break the question into sub-questions, retrieve for each, then synthesise — or an agentic loop that retrieves, notices what is missing, and retrieves again. I would flag it as a known limitation with a measurement plan attached, rather than tuning chunk size and hoping.
ArchitectWould you use semantic chunking?
I would evaluate it and I would expect to reject it for most enterprise corpora. It costs an embedding call per sentence at ingest, and structural boundaries usually capture the same topic shifts for free, because the author already marked them with headings. I would reach for it only on genuinely unstructured narrative text, and only if a recall sweep showed a real gain over paragraph packing.
ArchitectWhat goes in a chunk besides the text?
Two families. Filtering metadata — tenant, ACL group tags, effective dates, classification — which constrains what can be retrieved and has to be applied inside the search. And provenance metadata — document ID, section path, source URL, page number, version, content hash, embedding-model version — which makes citation, audit and debugging possible. I would design the record before the splitter, because chunk size can be re-swept later and metadata you did not capture usually cannot be recovered.
ArchitectOur context window is 128k. Why not put the whole document in?
Sometimes you should — if the corpus is small, retrieval is unnecessary complexity. But at enterprise scale three things break. Cost, because you pay per token on every query and that scales with traffic. Latency, because time to first token grows with context length. And accuracy, because models attend less reliably to content in the middle of very long contexts, so past a point more context actively lowers answer quality. Retrieval is a precision tool, not a workaround for small context windows.
ArchitectDesign the retrieval layer for a support assistant over two million help-centre articles and ticket resolutions.
I would start by noting these are two corpora with different shapes. Help-centre articles are structured, curated and long — structural chunking with parent–child. Ticket resolutions are short, self-contained and messy — already the right unit, so index them whole with no parent, because a parent adds nothing and costs tokens.
Then I would separate them at the index level, or at least tag them, because the ranking behaviour differs: articles are authoritative and stable, ticket resolutions are numerous and sometimes wrong. I would want the ability to weight or filter by source type, because the product will eventually ask for “official answer first, community answer second”, and that is much easier if the two were never blended.
ArchitectWhat is the single decision here that is most expensive to change later?
Chunk identity and the metadata schema. Chunk size I can re-sweep and rebuild. The embedding model I can migrate blue-green. But if chunk IDs are positional, every citation, every feedback record and every audit log points at something unstable — and fixing it means a full reindex plus reconciling historical data that may no longer be reconcilable. Equally, if I did not capture ACL tags or source coordinates at ingest, recovering them means re-crawling a million documents, and some source systems will not let me.
Eng managerYour team wants 1,000-token chunks because “more context is better”. You disagree. How do you handle it?
I would not argue from theory, I would make it measurable. Build a fifty-question gold set in a day, sweep both configurations, and put recall@10 and cost per query side by side. If they are right we ship their config and I have learned something cheaply. If the larger chunks lose recall, the data settles it and it never becomes an argument about seniority.
As a manager the more important outcome is that this becomes the norm — that configuration decisions are settled by a cheap experiment rather than by whoever argues longest. Establishing that once on a low-stakes question is worth more than winning this particular one.
Eng managerHow do you decide between building this and buying a managed RAG product?
I would frame it by where the differentiation is. The generic parts — vector storage, ANN indexing, basic chunking — are commodity, and I would buy them. The parts specific to this company are the connectors, the permission model, the domain evaluation set, and the routing between retrieval and structured data. That is where the quality comes from, and no vendor will get them right for you.
The failure mode of buying is that managed products tend to hide the ingestion layer, which is exactly where the hard problems live — you cannot fix a table-flattening bug you cannot see. So my test is concrete: can I control chunking and inspect the intermediate representation? If not, I would buy it for a pilot only.
Eng managerYou have a quarter and three engineers. What is the plan?
Weeks 1–3. One connector for the highest-value source, a canonical representation, stable IDs, ACL pre-filtering, and a fifty-question gold set built with a subject expert. Ship to a deliberately narrow pilot of twenty users at the end of it.
Weeks 4–8. Harvest real queries from the pilot, replace the synthetic gold set with them, and fix what the real distribution exposes — which in my experience is usually parsing, not retrieval. Add the second and third connectors.
Weeks 9–12. The update pipeline properly: diff, deletes, reconciliation, guard rails. Plus reranking and a regression gate in CI. I would deliberately defer contextual chunking and any fine-tuning, because those are optimisations and I will not know which one is the binding constraint until the pilot data is in.
Eng managerWhen does chunking stop mattering?
Two situations. When the corpus fits in context, so retrieval is unnecessary. And when retrieval is a known-address lookup rather than a search — if a query resolves to “get document X, section Y”, chunking is just storage layout.
It matters most in the middle: a large corpus, ambiguous queries, and answers that live in a small part of a large document. Saying so is worth points, because it stops the conversation treating chunking as universally important and shows you know where the technique sits.
Does overlap help with tables?
No, and it can make things worse. Overlap duplicates a window of tokens either side of a boundary, so a cut table produces two chunks that each contain part of the table and part of the neighbouring prose, and now both are wrong. Tables need to be atomic units with the header repeated, not overlapped.
Should the child chunk include the section heading?
Yes, and usually the whole section path. Prepending “HR Policy 2024 > 3 Leave > 3.2 Maternity Leave” costs one string concatenation, adds a handful of tokens, and gives the vector a topic anchor it otherwise lacks. It is the cheapest recall improvement in this document and it is often left out.
If parents are never embedded, why store them in the vector database at all?
You should not. Parents belong in a plain key-value store, read by key and never searched. Putting them in the vector collection means paying index memory for records that are never matched against, and on the reference stack that is a large amount of RAM spent on nothing.
What if one section is 8,000 tokens?
Then it is not a parent, it is a document. Cap parents at 1,000 to 1,500 tokens and use a sliding window around the matched child instead — typically the child plus a couple of siblings either side. Sending an 8,000-token parent reintroduces exactly the lost-in-the-middle problem parent–child was meant to avoid.
Do I need a different chunk size per document type?
Frequently yes, and it is cheap to do because chunking is per-source anyway. A contract, a runbook and a resolved ticket have genuinely different natural units. What you must not do is let them share an index without a type tag, because then you cannot weight or filter by source and cannot diagnose which corpus is failing.
Is there a rule of thumb for k?
Retrieve deep and send shallow: 30 to 50 candidates into the reranker, 3 to 8 parents into the prompt. Depth in the index is cheap — going from k=10 to k=50 in an ANN search costs a few milliseconds. Depth in the prompt is expensive and paid on every query. Getting those two backwards is one of the most common and most costly configuration mistakes.
How much does chunk size affect embedding cost?
Almost not at all, which surprises people. You embed the same total number of tokens either way — the corpus does not change size when you cut it differently. What changes is the number of vectors, which affects index memory and build time, and the tokens per query, which affects the recurring bill. Overlap is the exception: 15 percent overlap genuinely does mean 15 percent more tokens embedded.
Our documents are mostly one or two pages. Does any of this apply?
Much less of it. If a document fits comfortably in a chunk, index it whole and skip parent–child entirely. The techniques in this document earn their complexity when the answer occupies a small part of a large document; on short self-contained items they add cost and subtract nothing. Say that in an interview rather than applying the machinery reflexively.
Can I just let the language model do the chunking?
You can, and it is what contextual chunking and some agentic splitters do, but price it before you commit. On the reference stack that is ten million generation calls at ingest, repeated every time you re-chunk. Structural boundaries are free and capture most of the same signal, so the defensible position is: structure first, generation only where measurement shows a gap.
How do I chunk code, or a spreadsheet?
By the unit the language already defines. Code chunks at function or class level with the file path and imports prepended; a spreadsheet chunks per row with the header row repeated, or per named range. In both cases the general rule holds: cut where the author cut, and carry enough context that the chunk means something read alone.
Does chunking change if we use hybrid retrieval?
Yes, and usually in the direction of slightly larger chunks. Keyword scoring needs enough text for term statistics to be meaningful, so very small chunks hurt the sparse side even where they help the dense side. Document 15 covers the interaction; the short version is that hybrid pulls the optimum up a little, and it is worth re-sweeping size after you turn hybrid on rather than assuming the old optimum still holds.
“Chunking is the decision about what one retrievable record is, and it is upstream of everything, because retrieval can only reorder what chunking made findable. The tension is that small chunks match precisely but lose context, and large chunks keep context but their vectors are averages that match everything weakly — and cost more on every single query.
I do not trade that off, I dissolve it: cut on structural boundaries into 150-to-250-token children, embed those, and store the parent section as text. Search hits the child, the model reads the parent, and I deduplicate parents before assembly. That costs one extra fetch of about ten milliseconds and some duplicate text.
The thing I would design first, though, is not the size — it is the record. Stable identity, ACL group tags, section path, source URL, page number, content hash, model version. I can re-sweep chunk size in an afternoon; I cannot recover an ACL tag I never captured without re-crawling the corpus.”
| Thread from this document | Resolved in |
|---|---|
| Positional boundaries make identity unstable | 03 · Identity, updates and deletes |
| Tables, PDFs and layout-aware cutting | 02 · Parsing hard content |
| Filtering metadata must be applied inside the search | 04 · Access control and freshness |
| The model’s token limit interacts with the chunk cap | 07 · Token limits and truncation |
| Millions of chunks means index memory | 12 · Quantisation and capacity |
| Identifier queries that dense vectors cannot match | 15 · Hybrid retrieval and reranking |
| The sweep that turns any of this into evidence | 16 · Evaluation and observability |