Track B · Document 08 · Embeddings and index
Teaching a model what “similar” means in your domain, deciding whether it should be the embedder or the reranker, and changing either one without an outage.
You take a model already trained on a very large amount of general text, and you continue training it — on your own examples, for a short while, with a small learning rate. The examples are pairs: a real question, and the chunk that correctly answers it.
PAIR
question: "why is my widget bricked after the firmware push"
chunk: "Device fails to initialise following an OTA update.
Recovery procedure: hold the reset pin for..."The model’s job after training is to place those two things near each other in vector space, and to place wrong chunks further away.
The model is a similarity function, and fine-tuning reshapes that function. Interviewers listen specifically for the distinction on the left: candidates who think fine-tuning teaches the model facts have misunderstood what RAG is for.
A common confusion, worth clearing early because it changes what the work looks like. LoRA and its relatives were built for large language models — seven billion parameters and up — which are too large to fully fine-tune on ordinary hardware.
Interview-safe phrasing: “Embedding models are small enough for full fine-tuning, so LoRA is usually unnecessary — it is an answer to a memory constraint that does not apply at this scale.” The honest caveat is that parameter-efficient methods can be applied, and there are cases with very large embedders or many tenant-specific variants where that is sensible.
“Training” is a large word. Here is exactly what happens, one batch at a time.
You are not training the model to produce a particular vector — no target vector exists. You are training it to produce vectors whose relative distances come out in the right order.
The standard choice for retrieval fine-tuning supplies only positive pairs and takes the negatives free, from the rest of the batch.
The workhorse loss for retrieval fine-tuning needs only positive pairs — a question and the chunk that answers it. That is what makes the data collection tractable: you never have to label what is wrong, only what is right.
| Loss | When you use it |
|---|---|
| In-batch negatives (MultipleNegativesRanking, InfoNCE-family) | The default for retrieval. Needs only question-and-correct-chunk pairs |
| The same, with explicit hard negatives | When you can supply a known-wrong-but-plausible chunk per pair. Usually the single biggest quality gain |
| Triplet loss | Anchor, positive and negative supplied explicitly. Older style, still fine |
| Cosine-similarity / CoSENT losses | When your labels are graded scores rather than right-or-wrong. Rare in RAG |
| A Matryoshka wrapper | Combine with any of the above to preserve truncatable dimensions — see document 06 |
You need thousands. Realistically, a few thousand good pairs beats tens of thousands of poor ones — and the difference between good and poor is almost entirely about whether the questions are real.
The quality bar, in order: an explicit thumbs-up with the chunk known is a strong pair; the user copying the answer without rephrasing is a good one; the session merely ending normally is weak; and the chunk having been retrieved at all is not evidence of anything — retrieval is the thing you are trying to fix.
If one thing separates a fine-tune that works from one that does not, it is the negatives — and the mechanism is that you are explicitly training against the current model’s mistakes.
If one thing separates a fine-tune that works from one that does not, it is the negatives. Random in-batch negatives are mostly easy — which is why they are free, and why they eventually stop teaching.
There is one good reason: your domain uses language differently from the general internet, and no off-the-shelf model has ever seen that usage.
USER SAYS DOCS SAY A GENERAL MODEL
"bricked" "fails to initialise" no idea these match
"the box" "edge appliance unit 3" no idea
"flapping" "intermittent link state" no idea
"P1" "Severity 1 incident" no ideaA general model learned similarity from the public web, and your internal shorthand was not on the public web. That gap is real, and fine-tuning closes it in a way prompt engineering cannot. Typical reported gains on genuinely domain-specific corpora sit around five to fifteen points of recall@100, with wide variance — and on a corpus of ordinary business prose, expect much less, sometimes nothing.
Rungs 1 to 5 are things you configure. Rungs 6 and 7 are things you own. That is the real boundary, and it matters far more than the technical difficulty, which is roughly the same for both. The one good reason to cross it: your domain uses language differently from the general internet, and no off-the-shelf model has ever seen that usage.
This is the middle option most candidates never mention, and it is usually the right answer. The two stages have the same training data and comparable effort, and completely different consequences.
That single diagnostic is the professional answer to “which one do I fine-tune?” — and the reranker is the middle option most candidates never mention. Same training data, comparable effort, no re-embed, and a rollback measured in seconds.
Because it sees both texts at once, its attention can relate a word in the question directly to a word in the chunk. A bi-encoder cannot: each side was compressed into a vector in isolation, before the other side existed.
That is also exactly why it cannot be used for search. Scoring eight million chunks per query with a model is not a system, it is an outage.
The training is the cheap part — often under an hour on one GPU. Four costs follow it, and three of them are permanent.
| Cost | What it means |
|---|---|
| A full re-embed, immediately | New weights mean new geometry, so every stored vector is now meaningless. On eight million chunks that is 2.8 billion tokens through the model and a blue-green reindex. There is no partial migration — mixing old and new vectors is not degraded retrieval, it is random retrieval for the mixed portion |
| Retraining, forever | Content changes, vocabulary changes, new products arrive with new jargon. A model fine-tuned on last year’s language slowly stops matching this year’s questions |
| An upgrade tax | Every future base-model improvement now has a retraining step attached before you can take it. You have opted out of simply adopting the next good thing |
| Ownership | Weights, training data, evaluation, reproducibility and the person who understands all of it. That is a standing commitment, not a project |
“Who owns retraining this in eighteen months?” If the honest answer is nobody, the fine-tune is a liability with a quality benefit attached rather than the other way round — and the reranker option in section 7 gets most of the benefit with none of that obligation.
Everything in the second half of this document follows from one fact: a vector from model A and a vector from model B cannot be compared. Position 500 in the list encodes something that was decided during that model’s training, by that model, for itself.
The arithmetic still runs. Both vectors have 1024 numbers, so a dot product is perfectly computable, returns a valid number in the usual range, and carries no information about meaning whatsoever.
The distinction interviewers listen for: this is not degradation, it is noise. Degradation means the right answer is present and ranked lower — recoverable and detectable. Noise means the right answer may be absent, and what replaced it has nothing to do with the query. “It gets worse” is a weak answer; “it returns arbitrary results with no error” is the correct one.
One table, one model, one version. Always. There is no valid partial state, no “mostly migrated”. A table is either entirely model A or entirely model B — which is why the migration has to happen somewhere other than the live table.
Two tables coexist. Each is internally consistent — one model, one version, neither mixed — and traffic goes to exactly one of them at a time. This is blue-green, and model migration is its primary reason for existing.
Backfill handles the past — the content that already existed when the migration started. It is a one-off batch job that runs for hours or days.
Dual-write handles the present — everything that changes during the migration. It is not a job at all; it is a change to the live ingest pipeline.
Together they cover the whole corpus. Neither alone does, and confusing the two is how a migration ends up with a hole in it.
This is the single most important operational detail in a model migration, and it costs nothing to get right and a great deal to get wrong.
This is the single most important operational detail in a model migration, and the ordering is not cosmetic. Backfill handles the past; dual-write handles the present. Together they cover the whole corpus, and neither alone does.
A subtle race, and knowing it is a strong signal. A document is deleted while the backfill is running. The backfill, working from a snapshot taken before the delete, writes it into the new table anyway — and the deleted document is now live in the collection you are about to cut over to.
Three guards, and the third catches everything:
Never switch on offline metrics alone. Offline metrics have been wrong before, and shadow reads give you thousands of paired result sets from genuine production queries rather than a gold set of a few hundred.
Deleting the old table at cutover is the most common and most expensive mistake in a migration. Keeping it costs storage; not keeping it is the difference between a recoverable regression and an outage.
Everything above assumes you can tell which model produced a given vector. You cannot, unless you recorded it — and if you did not record it at write time, the information is gone.
a model-A vector: 1024 floats, normalised, norm 1.0
a model-B vector: 1024 floats, normalised, norm 1.0
IDENTICAL in every inspectable property.
There is no signature, no header, no watermark.| Field | Why it is there |
|---|---|
model_name | Which model |
model_version | Which version of it — fine-tunes especially, because the name does not change |
dimension | Catches a truncation or configuration mismatch |
embedded_at | Lets you scope a repair to an ingest window |
normalised | Makes the invariant from document 06 auditable rather than assumed |
Two records, two purposes: the table declares its version and each row stamps the model that produced it. They exist to be compared against each other, and one query is the entire detection mechanism for the failure in section 9:
SELECT model_name, model_version, count(*)
FROM chunks_v2
GROUP BY 1, 2;
healthy → exactly one row returned
broken → two or more rows, and you know immediatelyThat query costs nothing and it should run on a schedule. It is the difference between finding a mixed table in an hour and finding it in a hallucination ticket six weeks later.
| Symptom | Most likely cause | What to check first |
|---|---|---|
| Results are arbitrary for some queries and fine for others; no errors anywhere | A mixed table — two models in one collection | The group-by on model name and version. It should return exactly one row |
| The new table is missing recent edits after the backfill finished | Backfill was started before dual-write | The timestamps: when dual-write was enabled versus when the backfill began |
| A deleted document reappeared during migration | The resurrection race — the backfill worked from a pre-delete snapshot | Whether a reconciliation pass ran after the backfill |
| Fine-tuning made retrieval worse, and every training curve looked healthy | False negatives in the mined hard negatives — you trained the model to push away correct answers | The similarity cap on mined negatives, and a human sample of twenty of them |
| Offline metrics improved a lot; production did not move at all | Training and evaluation are both synthetic, so you measured the generator | Where the evaluation questions came from. If a model wrote them, that is the answer |
| The fine-tune helped for a quarter and then decayed | Vocabulary drift — new products, new jargon, new phrasing | Whether anybody owns retraining, and when it last ran |
| Rollback was impossible after a bad cutover | The old table was dropped, or dual-write to it stopped at cutover so it went stale | Whether writes to the old table continued after the alias flip |
| Recall@100 is unchanged but answers are better after a change | You fixed ordering, not retrieval — which is the reranker’s job | Nothing is wrong. Note it: it confirms the diagnostic and tells you which model to invest in next |
ArchitectWhen would you fine-tune an embedding model?
When the domain uses language differently from the general internet and no off-the-shelf model has seen that usage — users saying “bricked” where the docs say “fails to initialise”. That gap is real and prompt engineering cannot close it.
But I would walk the ladder first, out loud, because everything below it is cheaper and reversible: fix the chunking, prepend the section path, add hybrid retrieval if the losses are on identifiers, add an off-the-shelf reranker, try a different base model. Only then fine-tune — and even then, probably the reranker rather than the embedder.
ArchitectWhat exactly does fine-tuning teach the model?
Not facts. It will not know your products, prices or policies — those live in the chunks, which is the entire premise of RAG. What it teaches is what “similar” means in your domain: that “bricked” and “fails to initialise” are the same event, and that ERR_5521 and ERR_5522 are not.
The model is a similarity function and fine-tuning reshapes that function. And the consequence is immediate: every vector you have already computed came from the old geometry and is now wrong.
ArchitectEmbedder or reranker — which do you fine-tune?
There is a single diagnostic that answers it. Look at recall@100 against NDCG@10. If recall@100 is low, the right chunk is not reaching the candidate pool at all, and only the embedder can fix that. If recall@100 is fine and NDCG@10 is poor, the right chunk is in the pool and badly ordered — that is the reranker’s job.
And the reranker is far cheaper to be wrong about: no re-embed, no rebuild, rollback by swapping a file, and the blast radius is one ranking stage rather than the whole index. So where both would work, I would do the reranker.
ArchitectWhere do the training pairs come from, and what is the trap?
Three sources, cheapest first: production feedback, which is the only genuinely representative one; existing artefacts like resolved tickets, FAQs and click-through logs, which most organisations already have; and generated pairs, where a model writes questions for each chunk.
The trap is in the third. Generated questions are phrased the way a language model phrases things — “What is the default timeout value for the Payments API?” — and real users type “payment timeout too short”. Train on synthetic and evaluate on synthetic and both numbers look excellent while production does not move. Generated pairs are acceptable for training and unacceptable for evaluation.
ArchitectWhy can you not migrate to a new embedding model incrementally?
Because a vector from one model and a vector from another cannot be compared. The arithmetic runs perfectly — both are 1024 numbers, the dot product returns a valid score in the usual range — and it carries no information about meaning.
And it is worth being precise that this is noise, not degradation. Degradation means the right answer is present and ranked lower, which is recoverable and detectable. Noise means the right answer may be absent and what replaced it has nothing to do with the query. So there is no valid partial state: one table, one model, one version, and the migration happens in a parallel table.
ArchitectWalk me through the migration.
Dual-write on first, then backfill, then shadow read, then flip an alias. The ordering of the first two is the part that matters: if the backfill starts first, every document edited after the backfill has passed it is permanently stale in the new table, with nothing to reveal it.
Dual-write handles the present, backfill handles the past, and the overlap is deliberate — some chunks get written twice, the writes are idempotent upserts, and you would rather write something twice than miss it once. Then shadow-read real queries against both and compare on gold-set recall per document type, top-ten overlap and latency, before repointing the alias. And keep the old table warm and still dual-written to afterwards.
ArchitectHow would you know a table had two models in it?
Only if you stamped it, because the vectors themselves are identical in every inspectable property — same dimension, same norm, no signature. So every row carries the model name and version, the table declares its expected version, and a scheduled group-by compares them. One row returned is healthy; two or more and you know immediately.
That single query is the whole detection mechanism, and without the stamp there is no detection mechanism at all.
Eng managerThe team wants to fine-tune. What do you ask before approving it?
Four questions. What does the gold set say we are losing on, and is it the kind of loss a fine-tune fixes rather than an identifier problem hybrid retrieval would fix? What have we already tried from the cheap end — specifically, has anyone put an off-the-shelf reranker in front of it? Would fine-tuning the reranker get most of the gain without the re-embed? And who owns retraining this in eighteen months?
That last one is the one that changes the decision most often. A fine-tune is not a project, it is a standing commitment: weights, training data, evaluation, reproducibility, and an upgrade tax on every future base model. If nobody owns it, we are buying a liability with a quality benefit attached.
Eng managerHow do you plan a model migration as a piece of work?
As a project with a rollback, not a version bump. Concretely: build the alias layer first if it does not exist, because it is an afternoon and it is what makes cutover reversible. Then dual-write, then backfill sized against the rate limit rather than the token cost — on a large corpus that is the constraint, and it competes with live ingest.
I would budget for two full collections resident for the duration, get the shadow comparison built before the backfill finishes rather than after, and set the decommission gate at one full incident cycle. The failure mode I would guard against hardest is somebody dropping the old table at cutover to reclaim the storage.
Eng managerHow do you tell whether the fine-tune actually worked?
On a held-out set of real questions that never entered training, measured per query type rather than on the average — because a fine-tune that helps conceptual queries and hurts identifier queries can show a flat mean while making the product worse for a specific team.
Then shadow reads before cutover, and after cutover the same production signals as any other change: click-through, thumbs, escalation rate. And I would want the pre-fine-tune numbers recorded, because the most common failure here is having no baseline and therefore no way to answer the question at all.
How many pairs do I actually need?
A few thousand is a reasonable working target, and quality dominates quantity — two thousand real questions from production beat twenty thousand generated ones. Below a few hundred you are unlikely to shift the geometry meaningfully and very likely to overfit. If you cannot reach a few thousand real pairs, that is itself an argument for the reranker route or for waiting until you have collected them.
Can I fine-tune and keep Matryoshka truncation working?
Yes, by wrapping your loss in the Matryoshka objective so the training computes it at several prefix lengths. If you fine-tune with an ordinary loss on a Matryoshka-trained base model, you should expect the truncation property to degrade — the model is no longer being asked to keep every prefix useful. It is a one-line change at training time and an expensive discovery later.
Does fine-tuning change the dimension or the token limit?
No. Same architecture, same dimension, same limit — only the weights differ. That is convenient operationally, because your schema and your capacity arithmetic are unchanged. It is also exactly what makes the failure dangerous: a fine-tuned vector is indistinguishable from a base-model vector in every inspectable property, which is why the model stamp has to carry a version and not just a name.
Can I A/B test the new model instead of shadow reading?
You can, and it answers a different and better question — it measures outcomes rather than rankings. The reasons to shadow read first are that it is risk-free, it needs no traffic split, and it gives you paired results for the same query, which an A/B test does not. The strong sequence is shadow read to establish that nothing collapsed, then an A/B test to measure whether anybody is better off.
What if the backfill will take a week?
Then dual-write for a week, which is exactly what it is for — and check the rate limit rather than the cost, because the wall clock is the binding constraint. Two practical mitigations: request a temporary quota increase, since vendors will often grant one for a migration; and prioritise the backfill by document popularity so the highest-traffic content is correct in the new table first, which shortens the window in which a rollback would actually matter.
Is it safe to fine-tune on top of an already fine-tuned model?
Technically yes, and it accumulates risk you cannot easily see. Each round moves further from the base model’s general ability, and the drift is hard to measure because your evaluation set has usually drifted with it. The disciplined pattern is to keep the base model and the full training set, and retrain from the base each time rather than stacking — which also means a base-model upgrade is a retrain rather than a restart.
How do I decide the similarity cap for mined hard negatives?
Start around 0.95 and then look at what it excluded. Take fifty mined negatives just under the cap and read them: if you find genuine answers among them, the cap is too high for this corpus. Corpora with near-duplicate documents — policy versions, product variants, templated reports — need a lower cap, because “plausible and wrong” and “correct on a different page” look almost identical to a similarity score.
Do I need a GPU in production after fine-tuning?
You need somewhere to run the model, which you already did — fine-tuning changes the weights, not the serving requirement. What changes is that you can no longer use a hosted API for that model, so if you were using one, fine-tuning has just moved you to self-hosting for the query path as well as the ingest path. That is a real operational consequence and it belongs in the cost comparison from document 05 before you start.
“Fine-tuning an embedder teaches it what similar means in your domain — that ‘bricked’ and ‘fails to initialise’ are the same event. It does not teach it facts; facts stay in the chunks. The models are small, a hundred million parameters or so, so this is a full fine-tune on one GPU rather than anything LoRA-shaped.
But I would go down the ladder first, because everything below is cheaper and reversible: chunking, section-path prefixing, hybrid retrieval for identifiers, an off-the-shelf reranker, a different base model. And there is a diagnostic that decides which model to tune: if recall@100 is low the right chunk never reaches the pool and only the embedder fixes that; if recall@100 is fine and NDCG@10 is poor, the reranker is the answer — and the reranker needs no re-embed and rolls back in seconds.
If we do change the embedder, it is a migration rather than a version bump, because vectors from two models cannot be compared — mixing them is noise, not degradation. So: a parallel table, dual-write on before the backfill starts, shadow-read real queries against both and compare per document type, then flip an alias. Keep the old table warm and still dual-written to for a full incident cycle, because dropping it is the expensive mistake.”
| Thread from this document | Resolved in |
|---|---|
| Rebuilding from the canonical store rather than re-crawling | 02 · Parsing hard content |
| Blue-green, dual-write and the alias layer in general | 03 · Identity, updates and deletes |
| Why cross-model vectors are noise, and the shared-space rule | 05 · Choosing an embedding model |
| Keeping truncatable dimensions through a fine-tune | 06 · Dimensions, metrics and Matryoshka |
| The reranker as a stage, and hybrid retrieval for identifiers | 15 · Hybrid retrieval and reranking |
| The gold set, held-out splits and per-slice measurement | 16 · Evaluation and observability |