Document 05 · Quick learnHighest lock-in · Six missing characters · Queries, not documents
One modeldefines similar
The highest-lock-in decision in the whole system, the silent failure mode that costs a third of your recall, and how to choose without believing a leaderboard.
“The right shelf still exists and is still correctly filled. You are just standing somewhere else.”
0.91 → 0.63
recall, from six missing characters
47 hours
wall clock — the real constraint
68×
annual query tokens over the corpus
1%
the selection rule: cheapest within noise
What an embedding model isand why the lock-in is total
A function that turns text into a fixed-length list of numbers, arranged so that similar meanings land near each other. That is the whole idea. Everything interesting is in the consequences.
The library version
The embedding model is the librarian who decides where each book goes on the floor. Two librarians will both produce a perfectly sensible arrangement, and their arrangements will have nothing to do with each other.
Which is fine — until you send someone trained on one librarian’s floor plan into the other librarian’s building. They will walk confidently to a shelf, and it will be the wrong shelf
Where this sits in the cost hierarchy
Bottom. Everything in Track B is high lock-in, and the embedding model is the highest of all: you change it with a full re-embed and rebuild of every record, plus a second live copy of the collection during cutover.
Which is why interviewers spend disproportionate time here — they are testing whether you understand that some decisions are cheap to reverse and some are effectively permanent
What a change actually touches
1
An index parameter such as efSearchvectors still valid · rebuild nothing
minutes
2
Add or swap a rerankervectors still valid · rebuild nothing
a deploy
3
M was set too lowvectors still valid · rebuild the structure from them
CPU hours
4
Switch embedding modelsno vector is valid · rebuild everything, plus a second live collection
days
Worked8M chunks × 350 tokens = 2.8 billion tokens · at $0.12/M that is ~$340 · at 1M tokens/minute that is ~47 hours of wall clocknever lead with the dollar figure
The point to make
The dollar cost is not the constraint. A few hundred dollars of tokens is not a planning problem. Forty-seven hours of continuous embedding — competing with live ingest for the same rate limit — while a second full collection sits resident, is a planning problem, a capacity problem and a risk conversation.
The shared-space rulecosine between two models is noise with a plausible shape
Query vectors and document vectors must live in the same space. Each model learns its own arbitrary coordinate system during training, so dimension 47 in model A has no relationship whatsoever to dimension 47 in model B.
A
Deployment onlysame model, same version, different host
no impact
B
Prefix or instruction conventionone side adds the label, the other does not
0.91 → 0.63
C
Model version bumped on one sidevendor-dependent, usually severe
severe
D
Genuinely different modelsthe dimensions may even match, so every type check passes
random
E
A jointly trained pairor a shared-space family
fine by design
Nothing crashes, which is the problem
The database returns ten neighbours with scores around 0.5, which looks healthy on a dashboard, and the language model writes a confident answer from irrelevant context.
Defences: stamp embedding_model and embedding_version on every record and assert a match at query time — refusing rather than degrading
The prefix bugsix missing characters, a third of your recall
Many retrieval models are trained to do two different jobs: encode a question, and encode a passage. So the model is trained with a small label glued to the front telling it which job this is.
Month 0
One service does everything. passage: on ingest, query: on search.
recall@10 = 0.91
Month 6
The search API is rewritten in Go. The engineer reads the model’s API docs and sends the question straight through.
The prefix was never in those docs
Three days later
recall@10 has dropped to 0.63.
No alert, no exception, no error-rate change — and latency is better than before
Month 7
Support notices wrong answers. The ticket says “the model is hallucinating more, can we upgrade the LLM?”
Months 7–8
Three weeks of prompt tuning, a bigger model, a stricter instruction.
None of it helps, because the cause was six missing characters in another service
Why leaving it off hurts, as a picture
The model was trained so that text labelled passage: gets shelved in one area, and a question labelled query: gets walked to that same area. Forget the label and the model treats your question as a statement of fact. It walks to a different part of the floor.
The right shelf still exists and is still correctly filled — you are just standing somewhere else
The fix is structural, not procedural
Put the convention in a shared client library with embed_document and embed_query functions — and no function that takes raw text without declaring its role.
Then a recall gate in CI against 300 known query-and-chunk pairs. This bug fails the build on day one instead of surfacing as a hallucination ticket a month later
The one-liner
“Retrieval correctness depends on a convention shared between two services. So put the convention in code that both must go through, and put a recall check in CI, so drift is caught by a build rather than by a customer.”
Hosted API or open weightsthe break-even is queries, not documents
Almost every version of this discussion is conducted around the corpus, because the corpus is the big visible number — and the corpus is paid once.
Worked200 QPS × 30 tokens = 6,000 tokens/second → ~190 billion query tokens a year, against 2.8 billion for the corpusthe query side is 68× the corpus
Proprietary, hosted API
Best quality out of the box.
Buys: no GPU operations, no capacity planning, no serving code
Costs: per-token forever, including every query
Document text leaves your perimeter
The vendor can update or deprecate the model under you
And its rate limit sits inside your p99
The common shape: self-host the query encoder, use either for documents.
Open weights, self-hosted
Frozen weights on your disk.
Buys: perfect reproducibility — the weights cannot change under you
Data never leaves; fixed GPU cost that amortises
Costs: you own serving, batching, autoscaling
GPU capacity for reindex bursts, and the on-call for all of it
And one thing decides it for you
Data residency. If it is a hard constraint it settles both sides before any of the arithmetic runs.
Domain models and the identifier problemthe point most candidates miss
ERR_5521 and ERR_5522 embed to almost the same vector
Because the model has no notion that the last digit is the entire meaning. No general model fixes that, and a domain model mostly does not either.
The correct architectural response is usually not “buy a legal or medical or technical embedding model”
It is hybrid retrieval
Keyword scoring handles exact-token identity; dense handles semantics. ERR_5521 is a rare exact token — precisely what a sparse retriever is good at, and precisely what a dense one cannot represent.
A domain model is the right answer to a different problem: conceptual queries in a specialised vocabulary, where the general model does not know that two differently-worded clinical phrases mean the same thing
Multilingual: one question settles it
A shared multilingual space
Yes — a user asking in one language must find a document in another.
One index, one model, cross-lingual retrieval works
Accept that it is slightly weaker on any single language than a monolingual model
And keyword search is close to useless across languages, because the tokens do not match
So a cross-lingual system leans much harder on the dense side of any hybrid weighting
Does a user asking in one language need to find a document written in another?
Per-language indexes
No — strictly siloed per-region tenants, say.
A language detector in the query path routes to the right index
Often better, because each index gets the strongest available model
And keyword search works normally within each language
Reading a benchmark honestlythe headline number is an average across everything
Retrieval
Given a query, find the right passage.
Yes — this is the one
Reranking
Reorder a candidate list.
Somewhat
Semantic similarity
How close in meaning are two sentences?
Weakly
Classification
Is this review positive or negative?
No
Clustering
Group similar documents.
No
Pair class., summarisation
—
No
The trap
The headline leaderboard number is an average across all categories. A model can win the average on the strength of classification and be mediocre at retrieval.
For RAG, read the retrieval column — NDCG@10, never the mean. And there are language-specific boards, which is where a multilingual decision gets made
Four limits of any benchmark
Contamination
The score reflects a moment in time.
Domain mismatch
Your corpus is not in the benchmark.
Length mismatch
Benchmarks use short clean passages; you have 400-token chunks with a header.
The gap shrinks in production
The benchmark measured full-dimension exact search.
You are running approximate search with quantisation — a 1.5-point lead frequently disappears entirely
The selection rule
The cheapest and simplest model within noise of the best on your own gold set. A recall difference under one per cent is not worth a serving dependency — and it will probably vanish once you add approximate search and quantisation anyway.
A benchmark is a shortlisting tool, not a decision
The landscape, with a hedgea snapshot, not a recommendation
Model rankings move monthly. Name them with a hedge — “as of my last check” — and interviewers will respect it. Naming them with false confidence is worse than not naming them at all, because the interviewer probably checked more recently than you did. What does not go stale is the shape of the field, which is what the third line of each card is for.
Gemini Embedding 001
Leads the English board; 3072 dimensions with flexible truncation.
Frontier hosted models now ship truncation as a feature
Qwen3-Embedding
Strong open multilingual retrieval, large context, free weights.
The open tier is genuinely competitive, especially multilingually
Voyage-4 family
Mixture-of-experts, a shared embedding space across model sizes, Matryoshka dimensions, int8 and binary.
A shared space across sizes lets you index bulk content with a cheap model and encode high-value queries with an expensive one
voyage-context-4
Encodes each chunk together with its surrounding document context.
Contextual and late chunking are moving into the model itself
BGE-M3
Strong multilingual; emits dense, sparse and multi-vector output from one model.
One model can serve all three retrieval paradigms
NV-Embed-v2
Strong English, 4096 dimensions.
Dimension counts keep rising, and so does the memory bill
nomic-embed-text
Small, laptop-class, self-hostable.
The small tier is good enough for a great many corpora
OpenAI text-embedding-3-large
3072 dimensions, native dimension truncation.
Truncation support is now table stakes
Answer it out loudthe ninety-second version, then six questions
“The embedding model is the highest-lock-in decision in the system, because the index is not my documents — it is the output of one model at one version, and that model defines what similar means for everything downstream. Changing it means re-embedding every record, rebuilding the structure and holding a second live collection through the cutover. On eight million chunks that is a couple of days of wall clock and double the footprint; the dollar cost is a rounding error and is never the constraint. So I choose carefully: filter on residency, licence, length and language; shortlist on the benchmark’s retrieval column, never the mean; measure the shortlist on my own gold set at my real chunk length; cost it on queries, not documents; and take the cheapest model within one per cent of the best.”
Six questions, and the spine of each answer
Architect
How would you choose an embedding model?
Filter, shortlist, measure, cost, lock-in — in that order
Filter on hard constraints: residency, licence, max input length, languages, self-hostable. That eliminates most of the field for free
Shortlist on the retrieval column, not the headline average
Measure on our own gold set, at our real chunk length with our real metadata header
Then cost it over three years on queries rather than documents. Selection rule: the cheapest and simplest model within noise of the best — under one per cent is not worth a serving dependency.
Architect
What happens if query and documents use different models?
Nothing crashes, which is the problem. The dimensions may even match, so every type check passes
Ten neighbours come back with scores around 0.5 — which looks healthy on a dashboard
Each model learns its own arbitrary coordinate system, so cosine between them is noise with a plausible shape
Defences: stamp the model and version on every record and assert a match at query time, plus a recall check in CI.
Architect
What is the prefix bug and how do you prevent it?
Asymmetric models are trained with a label saying whether this is a question or a passage
One service adds it, another does not → the query lands in the wrong part of the space
You still get ten results, scores still look healthy, and the right chunk is at rank forty
Prevention is structural rather than procedural: a shared client library with embed_document and embed_query, and no function taking raw text without declaring its role. Then a recall gate in CI.
Architect
Hosted API or self-hosted weights?
The break-even is queries, not documents. Ingest is a one-off; the query encoder is a per-request dependency forever
At 200 QPS the annual query spend dwarfs the corpus by a large multiple
So the common shape is a split: self-host the query encoder, use either for documents
Also weigh version risk — a vendor can deprecate a model under you and put a migration on your roadmap at a time of their choosing — and the fact that its rate limit sits inside your p99.
Architect
Our corpus is full of part numbers. Which embedding model?
Probably none of them, and that is the answer I would give
ERR_5521 and ERR_5522 embed to almost the same vector
The right response is hybrid retrieval: sparse for exact-token identity, dense for semantics, results fused
A domain model answers a different question. I would diagnose which kind of query we are losing on from the gold set before spending money on either.
Eng manager
The team wants to move to a newer embedding model.
Three numbers before agreeing to anything.
What does the gold set say we are losing on — and is it the kind of query a new model would fix, or identifiers, which it would not?
What does the migration cost in wall clock and peak footprint, not dollars?
What have we already tried from the cheap end? A reranker is a deploy; this is a fortnight
If those point at a migration, treat it as a project with blue-green, shadow read and rollback — not a version bump. And get the gold set into CI first, or we cannot tell afterwards whether it helped.
Fast answersand how to stop this decision being made by accident
QHow do I stop this decision being made accidentally?
Three controls, all cheap. Pin the model version explicitly rather than tracking a “latest” alias, or the vendor makes the decision for you. Stamp the model and version on every record, so a mixed collection is detectable rather than mysterious. And put a recall gate in CI, so any change that moves retrieval quality fails a build.
QWhy frame it as accident prevention rather than governance?
Because in practice this decision is almost never made deliberately and badly. It is made implicitly, by someone writing a new service who had no idea a convention existed.
QShould I fine-tune instead of switching models?
Usually not first. Fine-tuning sits above rerankers, hybrid retrieval and better chunking on the cost ladder, and it adds a permanent obligation: every future model upgrade now has a retraining step attached.
QWhat do I tell a stakeholder who asks why we cannot “just try a better model”?
Give them the three numbers rather than the principle. A couple of days of continuous re-embedding competing with live ingest; two full collections side by side for the duration; and a cutover with a rollback plan, because the two sets of vectors are not comparable so there is no gradual version of this. Then offer the alternative: a reranker gets a large share of the same improvement, costs a deploy, and is reversible.
QDoes a different deployment of the same model matter?
No. Same model, same version, different host — the weights are identical, so the space is identical. That is the one row of the failure hierarchy that is genuinely safe.
QHow big should the CI gold set be?
Around 300 known query-and-chunk pairs is enough to catch the failures that matter, and small enough to run on every build. The assertion is that recall@10 stays above a threshold — not that it is perfect.
The index is not your documents. It is the output of one model at one version — and that model defines what similar means for everything downstream. Which is why the dollar cost of changing it is a rounding error and the wall clock and doubled footprint are not, why query and document vectors must come from the same model and the same convention, and why the failure mode is silent: nothing crashes, the scores look healthy, and recall quietly falls from 0.91 to 0.63 while latency improves.