Track D · Document 16 · Proving it works
Every failure mode in the previous fifteen documents is silent. This one is about making them loud — and about who owns each of them on a Tuesday.
A bad answer could be bad ingestion, bad retrieval, bad ranking, bad context assembly or bad generation. Five layers, one symptom — and until you can isolate them you cannot tune any of them.
Everything in the previous fifteen documents is a choice; this one is how you know whether the choice was right. If you can justify a chunk size by reasoning, you are a thoughtful engineer. If you can say how you would measure which is right — and what result would change your mind — you are an architect. That distinction is not a slogan; it is the actual difference interviewers are scoring at this level.
Chunking and retrieval are tuned on the first question — did the right chunk come back. If the right chunk never surfaces, no amount of prompt work saves you, and no reranker can fix an item that was never retrieved.
That question is also the cheap one: a labelled set, no model call, deterministic, fast enough to run on every commit. Answer quality needs a judge and costs real money, which is why it runs nightly and not per commit. Getting those two on the right cadence is most of the practical value in this document.
Questions paired with the chunks or spans that actually contain the answer. A hundred to three hundred to start — and the number matters far less than the stratification.
The trap in an LLM-generated set, said out loud: the questions come out suspiciously well aligned to your current chunk boundaries — because they were generated from those chunks. So the set flatters the chunking you already have, and a sweep against it will cheerfully tell you your current configuration is optimal. That caveat is subtle enough that stating it reads as experience. The cure is the same as the durability plan: replace generated questions with real ones from production logs as they arrive.
| Source | Why | The catch |
|---|---|---|
| Real user queries from logs | Always the best source. Real users are terse, misspell things, and ask two questions at once — synthetic questions do none of that | You need the product to exist first |
| SME-written | Ask the people who own the documents what actually gets asked | Their time, and a tendency to write the question they wish people asked |
| LLM-generated, human-verified | The fast way to bootstrap: feed a chunk to a model, ask for a question that chunk answers, have a human confirm it is sensible | It flatters your current chunking — see below |
Include lookup questions, multi-hop questions, table questions, and adversarial near-misses where two documents look similar but only one is right. And include questions with no answer in the corpus at all.
A system that confidently answers questions it should not is worse than one that says it does not know, and the no-answer slice is the only way to measure that. It is also the slice the business usually cares about most and the one almost nobody builds.
Everything here is computable by hand on a single ranked list, and doing it by hand once is the fastest way to stop confusing them.
Two traps in that panel. Recall@k is meaningless without stating k, and the k before reranking and the k after it are different numbers — quoting one without saying which is the most common way to mislead yourself. And precision@k caps low by construction: with one correct chunk per question, precision@10 can never exceed 0.1, so chasing it is chasing an artefact of the label set rather than a quality problem.
| Metric | Definition | What to watch out for |
|---|---|---|
| Recall@k | Fraction of questions where at least one relevant chunk appears in the top k | Meaningless without stating k — and k before reranking and k after it are different numbers |
| Precision@k | Of the k retrieved, the fraction that are relevant | With one correct chunk per question, precision@10 caps at 0.1. Do not chase it blindly |
| MRR | Mean of 1 ÷ rank of the first relevant result | Only cares about the first hit — useless when several chunks are jointly needed |
| NDCG@k | Ranking quality with graded relevance and a position discount | Needs graded labels, which are expensive and inconsistent between annotators |
| Context precision | Fraction of retrieved tokens that were actually relevant | The cost metric — it drives what you pay per query for noise |
| Faithfulness | Fraction of claims in the answer supported by the retrieved context | Judged by a model, so it has its own error rate. Validate the judge against humans on a sample |
| Answer correctness | Does the answer match the reference answer | Conflates retrieval and generation. Never tune chunking on this |
| Abstention rate | On the no-answer slice, how often it correctly declines | The most neglected metric, and often the one the business cares about most |
Chunking is optimised for recall; reranking owns precision. A reranker can fix ordering, but nothing can fix an item that was never retrieved.
So measure recall at your retrieval k — say 50, before reranking — and again at your final k — say 5, after. The gap between those two numbers is the whole diagnostic, and it is the subject of the next section.
Four numbers, read in order, name the layer. This is the single most useful thing in this document to be able to do out loud.
The pairing that does the work is the first one. High recall@50 with low recall@5 means retrieval is fine and ranking is the problem — invest in reranking. Low recall@50 means chunking or embedding is the problem, and reranking cannot help you at all, because nothing downstream can fix an item that was never retrieved. Those two look identical from the outside: a user gets a bad answer either way.
| Pattern | Where the problem is | What to do |
|---|---|---|
| Low recall@50 | Chunking or embedding | Reranking cannot help. Sweep chunking, check the embedding model, check whether a filter is cutting the pool |
| High recall@50, low recall@5 | Ranking | Invest in reranking — the answer is already in the candidate list |
| High recall@5, low faithfulness | Generation | Prompt and context assembly: ordering, truncation, how much noise is in the window |
| High faithfulness, low correctness | The corpus | You are grounded in the wrong documents. Superseded policy, duplicates, missing source — a content problem |
This one only occurs to people who have actually tried to run a chunk-size sweep, which is exactly why raising it unprompted lands.
Two fixes, and the crude one is often enough. Span-level labels — “the answer lives in document 7, characters 4200 to 4600” — let any configuration be scored by whether a retrieved chunk overlaps that span. Answer containment — does the retrieved text contain the known answer string? — is cruder, configuration-independent and trivial to implement. Raising this problem unprompted is a strong signal, because it only occurs to people who have actually tried to run a chunk-size sweep.
Fix everything except one variable. Rebuild the index for each configuration. Run the gold set. Plot it. That is the whole method, and it works identically for parent size, overlap, contextual augmentation, and structural against fixed boundaries.
You usually get a plateau, not a peak — and that is itself the useful information. A flat primary metric means the decision has been handed to the secondary axis, and stating why you broke the tie is what turns a measurement into a decision. The same method works for parent size, overlap, contextual augmentation on or off, and structural against fixed boundaries. One variable at a time, and rebuild the index for each.
Choose on the secondary axis, and say which axis you chose. If 300 and 500 give identical recall, pick 300 for cheaper generation or 500 for fewer vectors and cheaper indexing — depending on whether query cost or index cost dominates for you.
Stating why you broke the tie is what turns a measurement into a decision, and it is the difference between “we tested it” and “we decided it”.
Anyone can compute recall@10. What separates a system from a script is that runs are comparable.
Gold sets go stale. The corpus changes, user behaviour changes, and a set built in January flatters a system tuned in January. So you refresh it from production logs on a cadence, and you treat evaluation-set maintenance as ongoing work with a named owner — not a one-off project that finished in the second sprint.
This is the operational answer that shows you have run a system rather than built one, and every item on it comes from a failure documented earlier in this runbook.
| Gate | What it catches | Where it came from |
|---|---|---|
| A fifty-question gold set, recall@10 Fail the build if it drops more than a couple of points |
Every silent quality regression in this runbook | This document |
| Canary documents Synthetic users asserting they cannot retrieve restricted content |
Permission regressions — the failure mode with a legal consequence | 04 · Access control |
| Known-value assertions Specific numbers from specific tables must be retrievable |
Parser regressions on PDFs, which are invisible in aggregate metrics | 02 · Parsing hard content |
| Chunk-count and id-overlap drift alarms | A chunker change that silently re-cuts the whole corpus and orphans every label | 03 · Identity and updates |
| A short-result-set counter Queries returning fewer than k |
The filtered-search shortfall, which produces no error at all | 14 · Filtered search |
Every failure mode in this runbook is silent. A dashboard requires somebody to look at it on the day it moved; a gate fails the build. The gate is the one mechanism that turns a silent regression into a loud one, and it is cheap: fifty questions, no model call, a few seconds of CI.
The threshold matters less than its existence. Two points of recall on a fifty-question set is noise; the point is that a ten-point drop stops the deploy.
Offline metrics tell you about your gold set. They do not tell you about your users, and several of the most useful production signals cost nothing to collect.
| Signal | What it means |
|---|---|
| Thumbs up and down, joined to the chunk ids retrieved | The direct signal — and the join is exactly why stable chunk ids matter |
| Query reformulation rate | A user rephrasing is a strong implicit failure signal, and it is free to collect |
| Escalation to human support | The business-visible version of the same thing |
| Citation click-through | Do users open the source? A low rate can mean they do not trust the answer |
| No-result and low-score rates | Queries where nothing scored well are your coverage gaps. Cluster them and you have a prioritised list of what content is missing |
Most teams treat a low-scoring query as a retrieval failure to tune away. Clustered, they are a content roadmap — the questions people are asking that the corpus genuinely cannot answer. That output goes to whoever owns the documentation, not to the retrieval team, and it is usually the highest-value artefact the retrieval system produces for the rest of the organisation.
You cannot assign a chunking configuration per request, because the index is the configuration. So it is a blue/green deployment with a traffic split, not a per-request experiment — which means the comparison is slower to run, more expensive while both exist, and worth planning before you promise a date.
Everything the previous fifteen documents said needed watching, organised into the four planes it actually falls into. Teams typically instrument one of them well and none of the others.
Four planes, and teams typically instrument one of them. Latency is easy, so everyone has it. Quality is the one that decides whether the product works, and it fails without a single error being raised. Capacity decides whether you can still maintain the system in six months. Freshness is the one users actually complain about, phrased as “the answer is out of date”. If a dashboard has only latency on it, the system is unobserved in the three ways that matter.
An alarm that fires on a trend teaches people to ignore alarms. The distinction that keeps a pager useful is between things that are broken now and things that will be broken in a month.
| Page someone | Review it monthly | Never alarm on it |
|---|---|---|
| p99 latency above budget, sustained | Resident memory trend per node | Individual slow queries |
| Error rate or timeout rate | Dead ratio approaching the compaction threshold | Any single recall measurement |
| Ingestion lag past the freshness SLO | Build duration against the maintenance window | Cost per query on a single day |
| Deletion lag past its SLO — this one is compliance | Index size and growth rate | Thumbs-down on one answer |
| The recall gate failing in CI | Per-tenant recall distribution | Queue depth — use oldest item instead |
Latency: p95 end to end, stated with the query mix it applies to. A single number across filtered and unfiltered queries is not an SLO, it is an average of two different systems.
Freshness: time from a source change to being searchable, and separately from a permission revocation to being unsearchable. The second is a security control wearing an SLO’s clothes.
Quality: recall@k on the fixed gold set, per slice, with a floor. This is the unusual one — most teams have never written a quality SLO down, and it is the only one that protects the thing the product is for.
Resident memory rising while the live vector count is flat. Neither series alone means anything — memory rises for many reasons and a flat live count is normal — but together they are unambiguous dead accumulation. It is one derived metric, it has no user-visible symptom until a node runs out of memory, and it takes ten minutes to add.
“Search is bad” is the report you will get. It is not a diagnosis, and the order in which you narrow it decides whether this takes twenty minutes or two days.
Step one is the one people skip, and it is the one that saves the day. “Search is bad” from one customer and “search is bad” from everyone have almost no diagnostic overlap — the first is skew, a filter or a tenant boundary, and the second is a deploy or the index itself. Splitting the population before forming a hypothesis costs five minutes and routinely saves a day of looking in the wrong layer.
“What would have caught this automatically, and does that thing exist now?”
Almost every failure in this runbook has a one-line metric that would have caught it a week earlier: a short-result-set counter, a dead-ratio alarm, a per-tenant recall distribution, an ingestion-lag SLO. An incident that closes without adding its detector is an incident you have agreed to have again. That is the single most valuable habit an engineering manager can enforce here, and it costs nothing but the discipline to ask.
The question an engineering manager gets is not “what is our recall”. It is “what does this cost per query, and what would make it cheaper” — and the honest answer surprises people.
The number an engineering manager is asked for is cost per query, and the honest answer surprises people. Retrieval infrastructure — the nodes, the memory, the replicas, everything documents 12 and 13 were about — is usually a rounding error against the tokens you hand the model. That does not make capacity work pointless; it makes it a reliability investment rather than a cost one, and framing it that way is how it gets funded. The cost lever is k and the chunk size, and both of them are decided by a measurement you already have.
If retrieval infrastructure is a rounding error against generation, then documents 12 and 13 were not cost work — they were reliability work. Quantisation, sharding, replication and headroom buy you a system that survives a node loss and can still be rebuilt in a year, not a smaller invoice.
Say it that way and the conversation improves in both directions: the capacity work gets funded as reliability, and the cost conversation goes where the money actually is, which is k and the chunk size. Both of those are decided by a measurement you already have.
The engineering-manager half of this topic is not a different set of facts. It is the same facts with owners, cadences and a definition of done attached — and that attachment is what interviewers at this level are listening for.
| Thing | Cadence | What happens with no owner |
|---|---|---|
| The gold set | Refreshed from production logs, quarterly | It goes stale, flatters the configuration it was built against, and quietly stops being evidence |
| The capacity note — copy size, shards, replication factor, utilisation cap, and why each number is what it is | Reviewed at every rebuild | Every number gets re-litigated by somebody who was not in the room, once a quarter |
| Codebook and calibration retraining, if you use product quantisation | On the rebuild schedule | Recall degrades over months with no deployment to blame it on |
| The embedding model version pin | Reviewed, not drifted | Half the corpus ends up in one vector space and half in another |
| The rebuild checklist — nlist, dead ratio, headroom, filter fields, dimension | Every rebuild | Three of those five drift continuously without anyone deciding anything |
| Per-tenant recall distribution | Weekly | The largest tenant’s number becomes “our recall” and everybody else degrades unnoticed |
Per commit: the recall gate, the canaries, the known-value assertions. Automatic, fast, blocking.
Weekly: the per-slice and per-tenant evaluation, the short-result-set count, the one-sided-fusion rate. Fifteen minutes of somebody reading numbers, and it catches drift before a customer does.
Per rebuild, and at least quarterly: the capacity checklist and the gold-set refresh. This is the one that gets dropped, and it is the one that decides whether the system is still maintainable in a year.
| Concern | Who owns it | Why the split is here |
|---|---|---|
| Data isolation — tenant Z never sees tenant A’s rows | The platform team, and they must be able to prove it | It is a property of the store and the read-time join, and it is a security incident when it fails |
| Performance isolation — A’s traffic does not slow Z | The API team | No store gives a per-tenant CPU share. It is rate limits and scheduling, which live above the store |
| Retrieval quality | Whoever owns the gold set | Quality without a measurement is an opinion, and the measurement is the deliverable |
| Corpus coverage — the questions nothing can answer | Whoever owns the documentation | Clustered low-score queries are a content roadmap, not a retrieval bug |
Not “it is deployed”. A retrieval change is done when: the gold set says it helped, per slice and not on average; the change is reversible or the rollback is written down; the detector that would have caught its failure mode exists; and the capacity note reflects whatever it did to the footprint.
Four items, none of them expensive, and a team that applies them stops having the same incident twice. That is the actual output of this document.
Five steps, and almost everyone skips the last one. It is the step that turns an opinion into an engineering decision.
“What is the query mix — mostly lookups or summarisation? Assuming lookups dominate and we are at four million chunks with a p95 budget of 800 milliseconds: I would go with structural chunking, 200-token children carrying a pointer to their parent section, parent capped at 1,200 tokens. The tradeoff is an extra fetch hop, about ten milliseconds, and roughly double the text storage — both cheap against the recall gain. I would validate by sweeping child size against recall@10 on a two-hundred-question labelled set with span-level labels so the configurations stay comparable. If recall came back flat across 150 to 400 I would take the smallest, for generation cost. And if the query mix turned out to be mostly summarisation I would abandon this entirely and route to document-level summaries instead.”
These are not knowledge gaps. Every one of them is something a capable engineer does under interview pressure, and each has a one-line fix.
| Pattern | Why it costs you | Do this instead |
|---|---|---|
| Listing every option without choosing | Reads as an inability to decide under uncertainty, which is the job | Pick one, justify it, and name what would change your mind |
| Jumping straight to the answer | Signals you think one design fits every workload | Two clarifying questions first, then answer |
| Never mentioning cost | At manager level, cost is half the decision | Attach a rough number to at least one choice |
| Only discussing the happy path | The interview is largely about failure modes | Name the failure mode of your own preferred choice |
| Naming tools instead of properties | Tools change; reasoning does not | “I want filter-aware traversal and cheap upserts” beats a product name |
| Claiming prompt injection is solved | Anyone who has worked on it knows it is not | Contain the blast radius; assume the injection succeeds |
| Treating evaluation as an afterthought | It is the difference between engineering and opinion | Bring up measurement unprompted, and early |
| Over-indexing on the model | Most RAG failures are ingestion failures | Say so, and say where you would actually spend the effort |
ArchitectHow would you evaluate a RAG system?
By separating two questions that get conflated. Retrieval quality — did the right chunk come back — needs a labelled set and no model call, so it is cheap, deterministic and runs on every commit. Answer quality — was the response correct and grounded — needs a judge, so it is slow and expensive and runs nightly.
Chunking and retrieval are tuned on the first, because if the right chunk never surfaces no amount of prompt work saves you. The primary metric is recall@k, measured twice: at the retrieval k before reranking and at the final k after it.
And I would build the gold set before the system, not after: a couple of hundred questions with span-level labels, stratified to include multi-hop, tables, adversarial near-misses, and a no-answer slice — because a system that confidently answers what it should not is worse than one that declines, and the no-answer slice is the only way to measure that.
ArchitectRecall is 0.71 and users are unhappy. Where do you look?
At four numbers in order, because they name the layer. Low recall@50 means chunking or embedding, and reranking cannot help. High recall@50 with low recall@5 means ranking, and a reranker pays for itself immediately. High recall@5 with low faithfulness means generation — prompt and context assembly. High faithfulness with low correctness means the corpus: you are faithfully grounded in the wrong documents.
The order matters, because the earlier layers make the later numbers meaningless. There is no point reading faithfulness while recall@50 is low; the model is being asked to ground an answer in context that never contained it.
And I would also split the population before forming any hypothesis — by tenant, query type and language. “Users are unhappy” from one customer and from everyone have almost no diagnostic overlap.
ArchitectHow do you compare two chunking configurations fairly?
The problem to raise first is that chunk ids change when chunk size changes, so labels pointing at old ids are meaningless and the two configurations are not comparable at all.
Two fixes. Span-level labels — “the answer is in document 7, characters 4200 to 4600” — and any configuration is scored by whether a retrieved chunk overlaps that span. Or answer containment, which is cruder but configuration-independent and trivial: does the retrieved text contain the known answer string.
Then the sweep: fix everything else, rebuild the index per configuration, run the gold set, plot it. You usually get a plateau rather than a peak, and the plateau is itself the finding — it hands the decision to the secondary axis, and saying which axis you chose is the actual answer.
ArchitectWhat would you put in CI, and why a gate rather than a dashboard?
Five things: a fifty-question recall gate; canary documents asserting that synthetic users cannot retrieve restricted content; known-value assertions that specific numbers from specific tables are retrievable, which catches parser regressions; chunk-count and id-overlap drift alarms; and a counter for queries returning fewer than k results.
A gate rather than a dashboard because every failure mode in this area is silent. A dashboard needs somebody to look at it on the day it moved. A gate fails the build. And it is cheap — fifty questions, no model call, a few seconds.
The threshold matters less than its existence. Two points on a fifty-question set is noise; the point is that a ten-point drop stops the deploy rather than reaching a customer.
ArchitectWhat would you instrument in production that most teams do not?
Three things. Queries that returned fewer than k results — the filtered-search shortfall produces no error at all, and this one metric would catch most of the incidents in this area. Resident memory rising while the live vector count is flat, which is dead accumulation and has no user-visible symptom until a node runs out of memory. And per-tenant recall as a distribution rather than a mean, because the mean is the largest tenant’s number and everyone else can degrade underneath it.
I would add deletion lag as a separate SLO from ingestion lag, because it is a security control rather than a quality one, and it is the only freshness metric with a compliance consequence.
And I would clarify what belongs on a pager against a monthly review. Capacity numbers are trends, and a page for a trend teaches people to ignore pages.
ArchitectYour evaluation says the change helped and production says it did not. What happened?
Most likely the runs were not comparable. If the corpus grew five percent between them, a two-point recall difference means nothing — which is why the gold set has to be versioned, the corpus snapshot pinned and the full configuration recorded with every run. Teams lose weeks to exactly this.
The second candidate is that the gold set is stale or synthetic. LLM-generated questions come out suspiciously well aligned to the chunk boundaries they were generated from, so a sweep against them will cheerfully confirm that your current configuration is optimal.
The third is that the offline set does not look like production traffic at all — real users are terse, misspell things and ask two questions at once. That is the argument for refreshing the set from logs on a cadence and targeting a majority of real queries within a quarter.
Eng managerWhat does this system cost per query, and how would you make it cheaper?
The honest answer usually surprises people: the retrieval infrastructure is a rounding error against the tokens we hand the model. Ten chunks of four hundred tokens is four thousand context tokens, which at typical input pricing is about a cent a query; the index, amortised over ten million queries a month, is a few hundredths of that.
So the cost levers are k and the chunk size, and both are decided by a measurement we already have. On the recall plateau, moving from five hundred to two hundred token chunks changes nothing about quality and more than halves the largest line on the invoice.
Which reframes the capacity work rather than devaluing it. Quantisation, sharding and headroom are reliability investments, not cost ones — they buy a system that survives a node loss and can still be rebuilt next year. Framing them that way is how they get funded, and it stops the cost conversation looking in the wrong place.
Eng managerYour team says they need a week to improve retrieval. How do you scope it?
By asking what measurement will tell us it worked, before agreeing to the week. If the answer is “it will feel better”, the first two days are building the gold set and that is a better use of the week than anything else in it.
If a gold set exists, I would ask which layer the diagnostic points at. A week spent on reranking when recall@50 is low is a week spent on something that provably cannot help, and that conversation takes five minutes and saves the week.
And I would define done up front: the gold set says it helped, per slice; the change is reversible; the detector for its failure mode exists; and the capacity note reflects what it did to the footprint. Those four turn a week of work into something the next person can build on.
Eng managerHow do you keep a retrieval system healthy over two years?
Three cadences and a named owner for each. Per commit, the automated gates — recall, canaries, known values, drift alarms. Weekly, fifteen minutes of somebody reading the per-slice and per-tenant numbers. Per rebuild and at least quarterly, the capacity checklist and the gold-set refresh.
The last one is the one that gets dropped, and it is the one that decides whether the system is still maintainable. Three of the five items on the rebuild checklist drift continuously without anyone making a decision — corpus growth staling nlist, dead ratio crossing its threshold, filter fields multiplying — so they need a scheduled check rather than an alarm.
And one habit: no incident closes without adding the detector that would have caught it. Almost every failure in this area has a one-line metric that would have found it a week earlier. An incident that closes without one is an incident we have agreed to have again.
Eng managerHow would you onboard an engineer onto this system?
By handing them the capacity note and the gold set, in that order, and asking them to reproduce the current baseline number before changing anything. If they cannot reproduce it, we have found a reproducibility problem that was going to cost us later anyway.
Then a real, small change with the full loop attached — a measurement before, a change, a measurement after, and a note saying what tradeoff was accepted. The loop is the thing being taught, not the change.
What I would deliberately not do is start them on the model or the prompt. Most RAG failures are ingestion failures, and an engineer who learns this system from the retrieval end outward will diagnose faster for the rest of their time on it.
How many questions does a gold set need?
A hundred to three hundred to start, and the stratification matters far more than the count. A hundred questions covering lookups, multi-hop, tables, adversarial near-misses and a no-answer slice is worth more than a thousand generated from the same template. Note also what the size implies about noise: on fifty questions, one question is two points of recall, so set the CI threshold accordingly.
Can I use an LLM to build the whole thing?
To bootstrap, yes, and it is the fastest way to have something. But say the caveat out loud: the questions come out aligned to your current chunk boundaries because they were generated from those chunks, so the set flatters the configuration you already have and a sweep against it will confirm it is optimal. Human verification is the step that fixes that, and replacing generated questions with real ones from logs is the cure.
Should the LLM judge be trusted?
Only after you have validated it against humans on a sample, and only for the metrics it is actually good at. A judge has its own error rate, and that error rate is not uniform — it is usually worst on exactly the borderline cases you care about. Use it for faithfulness and abstention at volume, sample-check it against human raters, and never tune chunking on anything a judge produced.
Why not just measure end-to-end answer correctness?
Because it conflates every layer and tells you nothing about which one to fix. It is also the slowest and most expensive metric you have, so it is the worst possible thing to run on every commit. Measure it — it is the number the business cares about — but measure it nightly, and diagnose with the four-metric pairing rather than with it.
What is a good recall number?
There isn’t one, and being suspicious of the question is the right instinct. Recall depends on your corpus, your k, your query mix and how your labels were made — a number from someone else’s system is not comparable to yours. What is meaningful is your own number over time, per slice, against a pinned corpus. The useful target is “better than last month on the slice that was failing”.
How often should the gold set be refreshed?
Quarterly as a floor, and after any material change to the corpus or the user base. The failure mode is subtle: a set built in January flatters a system tuned in January, so the number keeps looking healthy while the thing it measures drifts away from what users actually ask. Give it an owner — a cadence with no name attached does not happen.
We have no labelled data and no time. What is the minimum?
Twenty questions you write yourself in an hour, with the answer span noted, committed to git, and run in CI. That is enough to catch a chunker change that halves recall, which is the failure this protects against. It is not enough to choose a chunk size. Start there and grow it — the gap between twenty and zero is far larger than the gap between twenty and two hundred.
What is the difference between an SLO and an alarm here?
An SLO is a promise about a distribution over a window; an alarm is a statement that something is broken now. Latency, freshness and deletion lag deserve both. Quality deserves an SLO and a CI gate rather than a pager, because a single recall measurement is noisy and a pager for a noisy metric trains people to ignore pagers. Capacity numbers deserve a monthly review and nothing else — they are trends.
Who should own retrieval quality?
Whoever owns the gold set, and it must be one named person rather than a team. The reason is that quality here has no natural owner: it is not the model team’s, not the infrastructure team’s, and not the product team’s, so it becomes nobody’s. Coverage — the questions nothing in the corpus can answer — belongs to whoever owns the documentation, and handing them the clustered low-score queries is one of the more valuable things a retrieval team produces.
What is the one thing to add if we can only add one?
A fifty-question recall gate in CI. Every failure mode in this runbook is silent, and that gate is the single mechanism that makes any of them loud. It costs an afternoon to build, a few seconds per commit to run, and it is the difference between finding a regression in a pull request and finding it in a customer escalation three weeks later.
| Plane | The one metric to have | Why it is neglected |
|---|---|---|
| Quality | recall@k on a fixed set, per slice, in CI | It fails without an error |
| Latency | p99 broken down by stage and by shard | Everyone has the aggregate and stops there |
| Capacity | resident memory rising while live count is flat | It is a trend, so nothing pages |
| Freshness | deletion lag — revoked to unsearchable | Nobody builds it until the incident |
“You cannot evaluate a RAG pipeline by looking at answers, because six layers can produce the same bad one. So you separate retrieval quality — did the right chunk come back, which needs a labelled set and no model call and runs on every commit — from answer quality, which needs a judge and runs nightly.
The gold set is a couple of hundred questions with span-level labels, because chunk ids change when chunk size changes and any label pointing at an id stops being comparable. Stratify it, and include a no-answer slice, because a system that confidently answers what it should not is worse than one that declines.
Then four numbers name the broken layer: low recall@50 is chunking or embedding and no reranker can help; recall@50 high with recall@5 low is ranking; recall@5 high with faithfulness low is generation; faithful but not correct is the corpus. Running that diagnostic out loud is worth more than any single number in it.
Operationally the whole thing rests on one property: every failure mode here is silent. So the gate goes in CI rather than on a dashboard, the four observability planes get one metric each — recall in CI, p99 by stage and shard, resident memory against live count, deletion lag — and no incident closes without adding the detector that would have caught it.”
| If you have | Do this |
|---|---|
| A week before an interview | Read 00, then the cheat sheet and the ninety-second version of each document. Then do the Q&A cards with answers hidden |
| A day | The ninety-second version of all sixteen, plus this document in full — because measurement is the thing candidates most often cannot do and interviewers most reliably ask |
| A system to build | Track A in order, then 12 for sizing, then this document. Build the gold set before the second design decision |
| A system that is already broken | Section 4 of this document, then the symptom tables in 11, 14 and 13 |
| Thread from this document | Resolved in |
|---|---|
| Why chunk ids must come from identity, not position | 03 · Identity, updates and deletes |
| Canary documents and the permission failure mode they catch | 04 · Access control and freshness |
| The flat baseline every recall number is measured against | 09 · Flat, IVF and HNSW |
| The parameter sweep, and reading the shape of the curve | 11 · Parameters and tuning |
| The capacity metrics behind the third observability plane | 12 · Quantisation and capacity |
| Per-shard latency, and why the cluster p99 hides it | 13 · Sharding and replication |
| Per-tenant recall, and why the mean is the whale’s number | 14 · Filtered search and multi-tenancy |
| Per-slice evaluation, and idf drift on the keyword half | 15 · Hybrid retrieval and reranking |