Cruxly › AI engineering › Retrieval and RAG

Choosing, Evaluating and Fine-Tuning an Embedding Model

a 300-query eval beats the leaderboard, plus storage, re-embed cost and the migration plan

Prompt: "We have 20 million chunks of policy and support documents. A vendor deck says model X is number one on MTEB. Pick an embedding model for us, tell us whether we should fine-tune one, and tell me how we move to a new model next year without downtime."

40 minutes first. Build the evaluation you would run, size the storage for two candidates, and price one full re-embed. The trap is answering with a model name; the interviewer wants a method, three numbers, and a rollback plan.

Why the leaderboard is not your answer, and what to run instead

An MTEB or BEIR average tells you how a model does on someone else's corpora, not on yours. The Massive Text Embedding Benchmark averages many tasks (retrieval, clustering, classification) across public datasets, and the BEIR paper ("Benchmarking IR", 18 datasets, zero-shot) showed the pattern that matters here: rankings move from dataset to dataset, and a lexical baseline (BM25) stayed hard to beat on several of them. Your 20 million chunks of Indian-English policy text with SKU codes and Hindi snippets are a 19th dataset that nobody evaluated. Read a leaderboard to build a shortlist of three, not to make the choice. If you need the general method for building a golden set, Evaluation Foundations: Golden Sets, Error Analysis and Reading Benchmarks has it; here is the retrieval-specific version.

@@ 1 | Collect 200 to 500 real queries Pull them from search logs or the support inbox, not from your team's imagination. Stratify: short keyword queries, full questions, queries with IDs, code-mixed queries, and the hard ones that failed last quarter.

@@ 2 | Label the chunks that answer each one A domain expert marks every chunk that contains the answer, usually one to three per query. Label by document and answer span rather than by chunk ID, so the same set survives a change of chunker (the ANN Indexes in Depth lesson depends on this). Synthetic questions generated from chunks are fine to supplement the set, but they are easier than real queries because they share words with the chunk; never let them be the whole set.

@@ 3 | Run every candidate through the same pipeline Same chunks, same index settings, same top-k. Only the embedding model changes.

@@ 4 | Report recall@10 and nDCG@10 with an error bar Metrics That Mean Something: Task Metrics, Ranking Metrics, pass@k and Significance defines both; the hand example below shows why you want both.

Recall and nDCG can disagree, and the disagreement is information. Take one query with two relevant chunks. Model A returns them at ranks 1 and 4; model B at ranks 2 and 3. Recall@3 is 0.5 for A (one of two found) and 1.0 for B. MRR is 1.0 for A and 0.5 for B. nDCG@5, with gain 1/log2(rank+1), is (1 + 0.431) / (1 + 0.631) = 0.877 for A and (0.631 + 0.5) / 1.631 = 0.693 for B. If an LLM reads the top 3 chunks, B is better; if a human reads the first hit, A is. Pick the metric that matches the consumer of your results, and say which one it is.

Three hundred queries gives a wide error bar. For a recall of 0.84, the standard error is sqrt(0.84 x 0.16 / 300) = 0.021, so a 95 percent interval is about plus or minus 4 points. Models scoring 0.84 and 0.86 are not distinguishable by that arithmetic alone. Compare them paired: count the queries where one model hits and the other misses. An exact sign test on the disagreements says how surprising the split is: B winning 19 of 36 (p about 0.87) is a coin flip, 24 of 36 (67 percent, p about 0.07) is suggestive but not settled, and 27 of 36 (75 percent, p about 0.004) is worth believing.

The trade-offs you can compute: dimensions, length, prefixes, price

Every embedding model is a bundle of six costs and capabilities, and four of them can be turned into numbers before you evaluate anything.

Dimensions and storage. Float32 storage is chunks x dimensions x 4 bytes. For 20M chunks, a 1,024-dim model needs 81.92 GB and a 3,072-dim model needs 245.76 GB, before any index overhead (the index arithmetic is in ANN Indexes in Depth: HNSW, IVF-PQ, DiskANN and the Recall/Latency/Memory Triangle).
Maximum input length. A model that accepts 512 tokens silently truncates a 900-token chunk, so the last 388 tokens (43 percent) of it are invisible to search. Check the limit against your chunk size; models with 8,192-token limits remove the problem but also tempt you into chunks so long that the vector averages many topics, which Embeddings and Similarity: Cosine, Dot Product, and What a Vector Encodes explains.
Language coverage. If 20 percent of queries are Hindi or code-mixed, a model trained mostly on English will lose recall on exactly those, and an average hides it. Report recall per language, as Multi-Tenant, Multilingual RAG at Scale argues.
Instruction prefixes. Some families (the E5 models, for example) are trained with different prefixes for queries and documents, and others take an instruction string. Forgetting the prefix does not raise an error; it costs recall. Put the prefix in the embedding wrapper, not in each caller.
Price and latency. An API charges per input token and adds a network round trip (tens of milliseconds) to every query; a self-hosted model on one GPU has no per-token fee but you own the uptime. At 20M chunks the one-off indexing bill matters; at 50 queries per second the query-time latency matters more.
Licence and residency. Whether the vendor may see your text is a procurement question that can veto the best model.

:::key Verified October 2026 From OpenAI's embeddings guide, retrieved October 2026: text-embedding-3-small has 1,536 default dimensions and text-embedding-3-large has 3,072; both accept 8,192 input tokens and support a dimensions parameter to shorten the vector. The MTEB scores on that page (62.3 and 64.6) are the vendor's own figures. The page states no per-million-token price (it links to a pricing page), so get current prices from the vendor's pricing page and never quote a figure from memory. :::

Now do the decision from the worked example. Three candidates on 300 labelled queries: recall@10 of 0.78 (1,024 dims), 0.84 (1,024 dims) and 0.86 (3,072 dims). The third model costs 245.76 GB against 81.92 GB, three times the storage, for two points that sit inside the error bar. Pick the second, and spend the saved memory and money on a reranker, which Hybrid Search, Rerankers and Late Interaction shows is usually the larger win. State the decision as "I would change my mind if a paired test on 1,000 queries shows the larger model winning by more than 3 points".

Smaller vectors: Matryoshka truncation, int8, binary, and the rescoring pass

You can often shrink a vector 4x to 32x and give back little recall, but only if you measure it on your queries. Three independent levers, which stack:

Truncate dimensions (Matryoshka). The Matryoshka Representation Learning paper trains one embedding so that the first 64, 128, 256... coordinates are usable on their own. Cutting 1,024 dims to 256 divides storage by 4 (20.48 GB for our 20M chunks). This only works for models trained for it; chopping an ordinary embedding destroys it. When a vendor offers a dimensions option, use that rather than slicing yourself, then renormalise if the model's docs say to.
Lower precision. int8 is 1 byte per dimension: 20.48 GB for 1,024 dims. Binary keeps one bit per dimension: 1,024 bits is 128 bytes per vector, so 20M vectors fit in 2.56 GB, a 32x cut.
Rescore. Search the compact vectors for a generous candidate list, then re-score only those candidates with the full float32 vector. Taking the top 100 by Hamming distance and re-scoring them costs 100 x 4,096 bytes = 410 KB of reads per query, which is trivial, and it recovers most of the ranking quality that one-bit vectors lose.

:::warn Quantising or truncating changes the score distribution. A similarity threshold calibrated on full vectors ("accept above 0.78") no longer means the same thing. Recompute recall@10 and recalibrate every threshold after any storage change, and re-run the whole golden set, not a spot check. :::

pgvector supports this pattern directly (its README documents halfvec storage and a binary-quantised index with a re-ranking step by original distance), which makes it a reasonable place to prove the idea before you build a two-tier system. The same lever list appears at the index level in Multi-Tenant, Multilingual RAG at Scale; here the point is the evaluation: the quantised variant is one more candidate in your table, scored on the same 300 queries, and it wins if its recall@10 is within the error bar of the full vector at a quarter of the memory.

A concrete frontier. For 20M chunks: 1,024-d float32 = 81.92 GB; 256-d float32 = 20.48 GB; 256-d int8 = 5.12 GB; 1,024-bit binary plus a float rescore = 2.56 GB in RAM and 81.92 GB on SSD. Which one you land on depends on recall on your queries, not on the arithmetic, and the honest line in the interview is "I would draw this curve for our data before promising a number".

Fine-tuning: what the data looks like and when it is worth it

Fine-tuning an embedding model is the last step on the ladder, not the first. The ladder, in order of cost: a better off-the-shelf model, then a BM25 leg (hybrid), then a reranker, then contextual chunk text, and only then your own training run. The reason is operational: a fine-tuned model is a new artifact you own, must re-embed the entire corpus for (the next section prices that), must evaluate on every release, and must retrain if your domain drifts.

What the training data is. Contrastive training needs pairs of (query, passage that answers it), the same objective described in Embeddings and Similarity. Three sources, in order of quality: click or accept logs from your own product, expert-labelled pairs, and synthetic questions that an LLM writes for each chunk. Synthetic pairs are cheap but biased to lexical overlap, so filter them: keep a pair only if the passage is actually retrieved in the top 20 by an existing model for that question, or drop ones where the question copies a long phrase from the chunk.

Negatives do most of the work. With a batch of 64 pairs, every other passage in the batch is a free negative, which is 63 negatives per query. Larger batches mean more negatives and usually a better model, which is why people train embedding models with big batches. The valuable extra is a hard negative: a passage that looks relevant and is not (the previous year's version of the policy, or the neighbouring product's manual). Mine them by retrieving the top 20 with the current model and keeping the ones a human or a strong reranker marks irrelevant. The danger is the false negative: an unlabelled chunk that actually answers the query, which then trains the model to push away a right answer. Review a sample of every hard-negative batch.

How many pairs does it take? The honest answer is that it depends and that published results vary: reports of useful domain gains from a few thousand pairs exist, and they are not guarantees. Do not trust a number; run a learning curve. Train with 500, 2,000 and 10,000 pairs, and plot recall@10 on a held-out set. If the curve is flat after 2,000, stop.

Guard the evaluation. Split by document, never by pair. If two questions about the same page land one in train and one in test, you are measuring memorisation. Compare the fine-tuned model against the best off-the-shelf model plus hybrid and reranking, since that is the system you would otherwise ship. Fine-tuning wins when your vocabulary is genuinely unusual (internal product codenames, legal clause numbering, a specialised dialect) and the gap persists after the cheaper rungs. It loses when a better base model arrives next quarter and wipes out your gain.

Know the domain traps before you train. Code, legal text, tables and product SKUs share one failure: the distinguishing signal is an exact token or a structure, not a topic. A SKU like KX-4471-B carries no meaning for an embedding, so KX-4471-A is a near neighbour, and no training run makes a dense vector reliably exact. The fix is a sparse leg, covered in Hybrid Search, Rerankers and Late Interaction, not more fine-tuning.

Operations: versions, the re-embed bill, blue/green, and the privacy of vectors

Treat the embedding model as a versioned dependency with a price tag for changing it. RAG From Scratch, No Framework names the failure of mixing vectors from two models in one index; this section is the plan that avoids it.

Price the migration before you choose the model. The worked example: 20M chunks x 400 tokens = 8B tokens. At an assumed $0.10 per million tokens, a full re-embed costs 8,000 x $0.10 = $800. That is cheap against engineering time and expensive if you do it every month by accident. Add the hidden cost: during a blue/green migration, old and new indexes coexist, so the 81.92 GB of vectors temporarily becomes 163.84 GB, plus build time and the query load of shadow tests.

The zero-downtime migration, step by step.

@@ 1 | Tag everything Each vector row stores model_id, dimensions, prefix_version and chunker_version. A vector without a tag cannot be trusted later.

@@ 2 | Build the green index in the background Re-embed the corpus into a separate index (or a separate column), throttled so it does not starve production. New and changed documents are written to both indexes while the build runs.

@@ 3 | Shadow and compare Send a sample of live queries to both. Compare recall on the golden set and the overlap of top-10 results; investigate queries where the new index loses.

@@ 4 | Flip an alias, keep the old index Switch the query path with a configuration flag or alias, not a code deploy. Keep blue for a rollback window (a week).

@@ 5 | Delete blue, record the change Drop the old index and note the model change in the release log so the next eval has a baseline.

Vectors are not anonymous. The paper "Text Embeddings Reveal (Almost) As Much As Text" (Morris et al., 2023) trained an inversion model that reconstructed 92 percent of 32-token inputs exactly from their embeddings, and recovered names from clinical notes. Their results are strongest on short text and did not fully cover longer inputs, so treat the finding as a floor on risk, not a ceiling. The practical rule: a vector carries the same privacy status as the text it came from. It goes under the same access control, the same retention schedule and the same erasure path described in Red-Teaming and Compliance as Code, and a vector database shared by several tenants needs the isolation covered in Multi-Tenant, Multilingual RAG at Scale.

How to say it in the interview

"I would not pick from the leaderboard average. I would label 300 real queries against the chunks that answer them, run each candidate through the same retrieval, and compare recall at 10 and nDCG with an error bar, because on 300 queries two points of recall is inside the noise. Then I price each winner in gigabytes and in re-embed dollars, shrink it with truncation or int8 if it holds up on my queries, and only consider fine-tuning if a better off-the-shelf model plus hybrid search plus a reranker still leaves a measured gap. Every vector carries a model-version tag so the migration is a blue/green index swap, not a maintenance window."

Checklist

Built a golden set of 200-500 real queries, labelled by document and answer span
Compared candidates through one identical pipeline and reported recall@10 and nDCG with a paired test
Computed storage as chunks x dimensions x bytes for each candidate before choosing
Checked max input tokens against chunk size and required query/document prefixes
Measured recall per language and for code-mixed queries, not only the blended score
Tested Matryoshka truncation, int8 or binary-plus-rescore on the golden set and recalibrated thresholds
Climbed the ladder (better model, BM25 leg, reranker, contextual text) before fine-tuning
Used hard negatives, reviewed them for false negatives, and split train/test by document
Tagged every vector with model id, dimensions, prefix and chunker version
Priced a full re-embed and planned a blue/green swap with dual-write and a rollback window
Gave vectors the same access control, retention and erasure path as the source text
Put exact-token domains (SKUs, codes, statute numbers) on a sparse leg instead of training for them

Practice this interactively