Document 16 · Quick learnFour numbers, in order · A gate, not a dashboard · Owners and cadences
Making silentfailures loud
Every failure mode in the previous fifteen documents is silent. This one is about making them loud — and about who owns each of them on a Tuesday.
“An incident that closes without adding its detector is an incident you have agreed to have again.”
4 numbers
read in order, they name the layer
270 questions
and the no-answer slice is 40 of them
50 in CI
no model call, a few seconds, blocking
3 cadences
commit · weekly · per rebuild
Five layers, one symptomyou cannot evaluate a pipeline by looking at answers
A bad answer could be bad ingestion, bad retrieval, bad ranking, bad context assembly or bad generation. Until you can isolate them you cannot tune any of them.
Ingestion
Did the text survive parsing?
Retrieval
Did the right chunk come back?
Ranking
Was it near the top?
Assembly
Did it reach the prompt?
Generation
Did the model use it?
Where the leverage is, and it is not where people look
Chunking and retrieval are tuned on the first question — did the right chunk come back. If it never surfaces, no amount of prompt work saves you, and no reranker can fix an item that was never retrieved.
That question is also the cheap one: a labelled set, no model call, deterministic, fast enough to run on every commit. Answer quality needs a judge and costs real money, which is why it runs nightly. Getting those two on the right cadence is most of the practical value here
The gold seta week, and the number matters far less than the stratification
1
Day 1 · sample and generate200 documents stratified by type, source and length; three candidate questions each with an LLM, plus the answering span
~600 raw
2
Days 2–3 · SME reviewcut anything that is not a question a real person would ask; fix the spans
~200 verified
3
Day 4 · adversarial casesnear-duplicates, superseded policy, questions spanning two documents
+30
4
Day 4 · the no-answer sliceplausible questions the corpus genuinely does not cover
+40
5
Day 5 · wire the harnesstake a baseline, commit to git with an owner and a refresh cadence
270
Real user queries from logs
Always the best source. Real users are terse, misspell things, and ask two questions at once.
The catch: you need the product to exist first
SME-written
Ask the people who own the documents what actually gets asked.
The catch: their time, and a tendency to write the question they wish people asked
LLM-generated, human-verified
The fast way to bootstrap.
The catch: it flatters your current chunking — the question was written from a chunk your configuration produced
Do not skip the last slice
Include questions with no answer in the corpus at all. A system that confidently answers questions it should not is worse than one that says it does not know, and the no-answer slice is the only way to measure that.
It is also the slice the business usually cares about most and the one almost nobody builds
The metricsand what each one is blind to
Recall@k
Fraction of questions where at least one relevant chunk is in the top k.
Meaningless without stating k — and k before reranking and k after it are different numbers
Precision@k
Of the k retrieved, the fraction that are relevant.
With one correct chunk per question, precision@10 caps at 0.1. Do not chase it blindly
MRR
Mean of 1 ÷ rank of the first relevant result.
Only cares about the first hit — useless when several chunks are jointly needed
NDCG@k
Ranking quality with graded relevance and a position discount.
Needs graded labels, which are expensive and inconsistent between annotators
Context precision
Fraction of retrieved tokens that were actually relevant.
The cost metric — it drives what you pay per query for noise
Faithfulness
Fraction of claims in the answer supported by the retrieved context.
Judged by a model, so it has its own error rate. Validate the judge against humans on a sample
Answer correctness
Does the answer match the reference answer.
Conflates retrieval and generation. Never tune chunking on this
Abstention rate
On the no-answer slice, how often it correctly declines.
The most neglected metric, and often the one the business cares about most
The division of labour to state out loud
Chunking is optimised for recall; reranking owns precision. A reranker can fix ordering, but nothing can fix an item that was never retrieved.
So measure recall at your retrieval k — say 50, before reranking — and again at your final k — say 5, after. The gap between those two numbers is the whole diagnostic
The diagnosticfour numbers, in order, name the layer
The single most useful thing in this document to be able to do out loud.
1
recall@50 — is the answer in the pool at all?low → retrieval or chunking. Nothing downstream can save this
the gate
2
recall@5 — did it survive the reranker?recall@50 fine, recall@5 low → ranking. A reranker, or fusion depth
ordering
3
faithfulness — is the answer supported by what we gave it?low → generation. The context was right and the model did not use it
the model
4
correctness — does it match the reference?everything above fine and this low → the reference, or an assembly bug
last
The chunking wrinkle: labels that survive a re-chunk
Two chunking configurations cut a document differently, so every chunk id in config A is absent from config B. A label pointing at a chunk id is worthless across configurations; a label pointing at a character span scores both.
This only occurs to people who have actually tried to run a chunk-size sweep — which is exactly why raising it unprompted lands
The sweep, and why the plateau is the finding
Fix everything except one variable. Rebuild the index for each configuration. Run the gold set. Plot it. That is the whole method, and it works identically for parent size, overlap, contextual augmentation, and structural against fixed boundaries.
If recall is flat between 300 and 500 tokens, take the smaller one for cost. The plateau is not a disappointing result — it is the result
A harness, not a scriptwhat makes runs comparable
Anyone can compute recall@10. What separates a system from a script is that runs are comparable.
The gold set is versioned
In git, alongside the code.
So a change to the set is reviewable, and every run records which version it used
The corpus snapshot is pinned
A run is against a stated state of the data.
Not against “production, some time on Tuesday”
Every run records the full config
Chunk size, model, index parameters, k, filters, fusion rule — all of it, in the result.
And the reason all three matter
If the corpus grew 5% between two runs, a two-point recall difference means nothing — and teams lose weeks to exactly this.
The honest limitation, worth volunteering before you are asked
Gold sets go stale. The corpus changes, user behaviour changes, and a set built in January flatters a system tuned in January.
So you refresh it from production logs on a cadence, and treat evaluation-set maintenance as ongoing work with a named owner — not a one-off project that finished in the second sprint
What runs on every commitevery item comes from a failure documented earlier
A fifty-question gold set, recall@10
Fail the build if it drops more than a couple of points.
Every silent quality regression in this runbook · this document
Canary documents
Synthetic users asserting they cannot retrieve restricted content.
Permission regressions — the failure mode with a legal consequence · doc 04
Known-value assertions
Specific numbers from specific tables must be retrievable.
Parser regressions on PDFs, invisible in aggregate metrics · doc 02
Chunk-count and id-overlap drift alarms
Per document and per source system.
A chunker change that silently re-cuts the corpus and orphans every label · doc 03
A short-result-set counter
Queries returning fewer than k.
The filtered-search shortfall, which produces no error at all · doc 14
Why a gate rather than a dashboard
Every failure mode in this runbook is silent. A dashboard requires somebody to look at it on the day it moved; a gate fails the build. The gate is the one mechanism that turns a silent regression into a loud one — and it is cheap: fifty questions, no model call, a few seconds of CI.
The threshold matters less than its existence. Two points of recall on a fifty-question set is noise; the point is that a ten-point drop stops the deploy
Online evaluationwhat production tells you that the gold set cannot
Thumbs up and down
Joined to the chunk ids retrieved.
The direct signal — and the join is exactly why stable chunk ids matter
Query reformulation rate
A user rephrasing is a strong implicit failure signal.
And it is free to collect
Escalation to human support
The business-visible version of the same thing.
Citation click-through
Do users open the source?
A low rate can mean they do not trust the answer
No-result and low-score rates
Queries where nothing scored well are your coverage gaps.
Cluster them and you have a prioritised list of what to ingest next
And when you A/B test, remember chunking changes are index-wide
You cannot assign a chunking configuration per request, because the index is the configuration. So it is a blue/green deployment with a traffic split, not a per-request experiment.
Which means the comparison is slower to run, more expensive while both exist, and worth planning before you promise a date
What to alarm on, and what not toan alarm that fires on a trend teaches people to ignore alarms
Page someone
p99 latency above budget, sustained Error or timeout rate Ingestion lag past the freshness SLO Deletion lag past its SLO — this one is compliance The recall gate failing in CI
Review it monthly
Resident memory trend per node Dead ratio approaching the compaction threshold Build duration against the maintenance window Index size and growth rate Per-tenant recall distribution
Never alarm on it
Individual slow queries Any single recall measurement Cost per query on a single day Thumbs-down on one answer Queue depth — use oldest item instead
The alarm nobody has, and everybody needs
Resident memory rising while the live vector count is flat. Neither series alone means anything — memory rises for many reasons and a flat live count is normal — but together they are unambiguous dead accumulation.
It is one derived metric, it has no user-visible symptom until a node runs out of memory, and it takes ten minutes to add
The three SLOs worth actually defining
Latency
p95 end to end, stated with the query mix it applies to.
A single number across filtered and unfiltered queries is not an SLO — it is an average of two different systems
Freshness
Time from a source change to being searchable — and separately from a permission revocation to being unsearchable.
The second is a security control wearing an SLO’s clothes
Quality
recall@k on the fixed gold set, per slice, with a floor.
The unusual one — most teams have never written a quality SLO down, and it is the only one that protects the thing the product is for
The incident playbook“search is bad” is a report, not a diagnosis
1 · Everyone, or someone?
Split by tenant, query type and language first.
2 · Did anything change?
Parameters, data or traffic — and the answer is usually not parameters.
3 · The four-metric diagnostic
recall@50, recall@5, faithfulness, correctness. In that order.
4 · Silent before loud
Short result sets, dead ratio, ingestion lag, cache hit rate.
5 · Before closing
What would have caught this automatically, and does that thing exist now?
The postmortem question that compounds
Almost every failure in this runbook has a one-line metric that would have caught it a week earlier: a short-result-set counter, a dead-ratio alarm, a per-tenant recall distribution, an ingestion-lag SLO.
An incident that closes without adding its detector is an incident you have agreed to have again. That is the single most valuable habit an engineering manager can enforce here, and it costs nothing but the discipline to ask
Running the teamthe same facts, with owners and cadences attached
The engineering-manager half of this topic is not a different set of facts. It is the same facts with owners, cadences and a definition of done attached — and that attachment is what interviewers at this level are listening for.
The gold set
Refreshed from production logs, quarterly
With no owner: It goes stale, flatters the configuration it was built against, and quietly stops being evidence
The capacity note
Reviewed at every rebuild
With no owner: Every number gets re-litigated by somebody who was not in the room, once a quarter
Codebook and calibration retraining
On the rebuild schedule
With no owner: Recall degrades over months with no deployment to blame it on
The embedding model version pin
Reviewed, not drifted
With no owner: Half the corpus ends up in one vector space and half in another
The rebuild checklist
Every rebuild — nlist, dead ratio, headroom, filter fields, dimension
With no owner: Three of those five drift continuously without anyone deciding anything
Per-tenant recall distribution
Weekly
With no owner: The largest tenant’s number becomes “our recall” and everybody else degrades unnoticed
The three cadences that hold the system together
Per commit
The recall gate, the canaries, the known-value assertions.
Automatic, fast, blocking
Weekly
The per-slice and per-tenant evaluation, the short-result-set count, the one-sided-fusion rate.
Fifteen minutes of somebody reading numbers — and it catches drift before a customer does
Per rebuild, and at least quarterly
The capacity checklist and the gold-set refresh.
This is the one that gets dropped, and it is the one that decides whether the system is still maintainable in a year
How to frame capacity work so it gets funded
If retrieval infrastructure is a rounding error against generation, then documents 12 and 13 were not cost work — they were reliability work. Quantisation, sharding, replication and headroom buy you a system that survives a node loss and can still be rebuilt in a year, not a smaller invoice.
Say it that way and the conversation improves in both directions: the capacity work gets funded as reliability, and the cost conversation goes where the money actually is — k and the chunk size
Answer it out loudthe ninety-second version, then six questions
“You cannot evaluate a RAG pipeline by looking at answers, because five layers produce one symptom. So I separate them with four numbers read in order: recall@50 says whether the answer is in the pool at all, recall@5 says whether it survived the reranker, faithfulness says whether the model used what we gave it, and correctness comes last. That needs a gold set — a few hundred questions labelled by span rather than by chunk id, so the labels survive a re-chunk — stratified across lookup, multi-hop, table and no-answer questions. Then the crucial part: every failure mode in this subject is silent, so the mechanism is a gate rather than a dashboard. Fifty questions, no model call, a few seconds of CI, and a ten-point drop stops the deploy.”
Six questions, and the spine of each answer
Architect
How would you know if retrieval got worse?
A gold set in CI — a few hundred questions with known correct spans, recall@k with a floor
Labelled by character span, not chunk id, so the labels survive a re-chunk
Stratified: lookup, multi-hop, table, adversarial near-misses, and no-answer
A gate, not a dashboard — because a dashboard needs somebody to look on the day it moved
And the three things that make runs comparable: the gold set versioned in git, the corpus snapshot pinned, and the full config recorded in every result. If the corpus grew 5% between runs, a two-point difference means nothing.
Architect
Which layer is broken?
Four numbers, in order.recall@50 low → retrieval or chunking. Nothing downstream saves this
recall@50 fine, recall@5 low → ranking. A reranker, or fusion depth
faithfulness low → generation. The context was right and the model did not use it
everything fine, correctness low → the reference, or an assembly bug
And the division of labour: chunking is optimised for recall; reranking owns precision. A reranker fixes ordering and can never fix an item that was never retrieved.
Architect
How do you run a chunk-size sweep?
Fix everything except one variable, rebuild for each configuration, run the gold set, plot it
The labels must point at character spans, because every chunk id changes between configurations
The plateau is the finding: if recall is flat between 300 and 500, take the smaller one for cost
Raising the label problem unprompted lands, because it only occurs to people who have actually tried this.
Eng manager
What runs on every commit?
A fifty-question recall gate — fail the build on more than a couple of points
Canary documents: synthetic users asserting they cannot retrieve restricted content
Known-value assertions: specific numbers from specific tables must be retrievable
Drift alarms on chunk count and id overlap, and a short-result-set counter
Every item comes from a documented failure in the runbook — and every one of those failures is silent. The gate is the one mechanism that turns a silent regression into a loud one.
Eng manager
What would you alarm on?
Page: sustained p99 over budget, error rate, ingestion lag past the freshness SLO, deletion lag — that one is compliance
Review monthly: resident memory trend, dead ratio, build duration, per-tenant recall distribution
Never: individual slow queries, any single recall measurement, one thumbs-down
And the alarm nobody has: resident memory rising while the live vector count is flat. Neither series alone means anything; together they are unambiguous dead accumulation, with no symptom until a node runs out of memory.
Eng manager
What does this cost per query, and what would make it cheaper?
Retrieval infrastructure is usually a rounding error against generation
Which means quantisation, sharding and headroom were not cost work — they were reliability work
The money is in k and the chunk size — how many tokens you send, on every query, forever
Framing it that way improves the conversation in both directions: the capacity work gets funded as reliability, and the cost conversation goes where the money actually is — and both are decided by measurements you already have.
Fast answersthe seven that come up most
QHow big does the gold set need to be?
A hundred to three hundred to start — and the number matters far less than the stratification. Fifty is enough for a CI gate. What you must include is the adversarial slice and the no-answer slice, which is the one almost nobody builds.
QWhy label spans rather than chunk ids?
Because every chunk id changes when you re-chunk, so a label pointing at an id is worthless across configurations — and comparing configurations is the entire point of a sweep. A span scores both.
QCan I use an LLM to generate the whole gold set?
To bootstrap, yes — with human verification. But be aware it flatters your current chunking, because the question was written from a chunk your configuration produced. And generated questions are phrased the way a model phrases things, not the way users do.
QShould I tune chunking on answer correctness?
Never. It conflates retrieval and generation, and it needs a judge, which has its own error rate. Tune chunking on recall at your retrieval k — deterministic, no model call, fast enough for every commit.
QWhy is precision@10 stuck at 0.1?
Because with one correct chunk per question that is the ceiling. Precision@k is not the metric to chase in that regime — recall is, and the reranker owns the ordering.
QHow do I A/B test a chunking change?
You cannot do it per request, because the index is the configuration. It is a blue/green deployment with a traffic split — slower to run, more expensive while both exist, and worth planning before you promise a date.
QWhat is the single habit worth enforcing?
“What would have caught this automatically, and does that thing exist now?” at the end of every incident. Almost every failure here has a one-line metric that would have caught it a week earlier — and an incident that closes without adding its detector is one you have agreed to have again.
Every failure mode in the previous fifteen documents is silent — the prefix bug, the truncated chunk, the stale orphan, the post-filter shortfall, the idf drift, the mixed vector space. None of them throws. Which is why the mechanism that matters is a gate rather than a dashboard: fifty questions, no model call, a few seconds of CI, and a ten-point drop stops the deploy. And the habit that compounds is one question at the end of every incident: what would have caught this automatically, and does that thing exist now?