Don't Trust MTEB Alone: Validate Retrieval Before Switching Embedding Models

MTEB can't tell you if a new embedding model works on your data. How to validate retrieval before re-indexing your production corpus.

  • embeddings
  • rag
  • mteb
  • migration

TL;DR: The MTEB leaderboard tells you which embedding models are broadly competitive. It can't tell you whether a new model will retrieve better on your documents and your users' questions. Switching models in production means re-embedding and re-indexing your whole corpus, so test the candidate against your current model on your own data first. This post explains why, and gives a step-by-step validation process.

A new embedding model lands at the top of the leaderboard every few months. Someone on the team posts the link, and the question comes up: should we switch?

It's a reasonable question with an expensive answer. Changing the embedding model isn't a config change. Vectors from two different models live in different spaces and can't be mixed in one index, so a switch means re-embedding every chunk, rebuilding the index, and usually running two indexes side by side during the cutover. On a corpus of millions of documents, that's real money, real engineering time, and real risk if retrieval gets worse in ways nobody measured.

So before re-indexing anything, you want one answer: is the new model better enough, on our data, to justify the switch?

What MTEB measures (and what it doesn't)

The Massive Text Embedding Benchmark (MTEB) is the standard public benchmark for embedding models. It covers many task types (retrieval, classification, clustering, reranking, semantic similarity and more) across many datasets and languages, and it gives the community a shared reference.

But look at what the headline number is: an average across tasks and datasets. Your RAG system does one of those tasks, retrieval, on one dataset that MTEB has never seen: yours.

Why the leaderboard rank doesn't transfer to your RAG system

1. The overall score mixes tasks you don't run. A model can climb the overall ranking thanks to classification or clustering scores that have nothing to do with retrieval. Even the retrieval column is an average over public datasets that may look nothing like your corpus.

2. Your vocabulary isn't in public data. Internal product names, error codes, SKU references, legal clause numbers, acronyms only your company uses. A model that handles Wikipedia-style questions well can confuse two of your product names that look alike.

3. Your documents have a different shape. API reference pages, support tickets, contracts, changelogs, spreadsheets exported to CSV. Chunks full of tables or code behave differently from clean paragraphs, and chunking interacts with the model: some models only read the first 512 tokens of a chunk.

4. Your users don't write like benchmark queries. Real questions are short, vague, informal and sometimes misspelled: "refund after 2 weeks?" rather than "What is the duration of the return eligibility period?"

5. Top models are often separated by less than your domain shift. The gap between the top entries is frequently smaller than the change you get by moving to a different domain. Public datasets can also leak into training data, which flatters the models that saw them.

This does not make MTEB useless, but instead it makes it a shortlist, not a decision.

The real question: is the new model better enough?

When you're migrating, "which model is best?" is the wrong question. The useful one is comparative and economic:

Does model B retrieve meaningfully better than our current model A, on our data, by enough to pay for re-indexing and the migration risk?

Answering it needs four things the leaderboard can't give you:

  1. Scores for your current model and the candidate on the same questions, same chunks and same top k.
  2. Enough questions to tell a real difference from noise.
  3. Per-question detail, so you know whether the candidate wins broadly or on a lucky handful, and whether it regresses on questions that matter (billing, legal, safety).
  4. Cost, latency and storage, because a 2-point gain rarely justifies doubling query cost or vector storage.

How to validate an embedding model before migrating

Here's the process I follow. I'll describe it with Truvec, which is built for exactly this, but the method applies whatever tool you use.

Step 1: Evaluate the chunks you run in production

Changing the model and the chunking at the same time makes the result impossible to interpret. Keep chunking fixed.

In Truvec, the cleanest way is to upload your production chunks as pre-chunked data (JSON, JSONL or CSV with a text field and optional id and source_document): open your project in the Truvec dashboard, select Data Upload in the sidebar, then the Pre-chunked Data tab. For more details about accepted chunk formats, see the pre-chunked data docs. If you would rather start from raw documents, pick the chunking strategy and size that match production.

Step 2: Build a golden dataset that looks like real traffic

Generate a first set of questions with AI (select Golden Set in the project sidebar and click Create Dataset), then review each one. Check if the expected chunks really answer it, if another chunk also answers it, and if a real user would phrase it that way. Then add real questions from support tickets, search logs or chat history, with the chunk IDs that answer them. You can use the Playground for this (select Playground in the project sidebar), run a question with any model and it will show the ID of each retrieved chunk.

Once you finish the review, commit the dataset with the Commit Dataset button. The golden dataset guide covers the review in detail. A committed dataset is locked, so your baseline and every candidate are graded on exactly the same answer key, today and when the next model comes out.

For a migration decision, aim for 50 to 100 questions or more. With 30 questions, one question is worth about 3 points of hit rate.

Step 3: Compare the current model and the candidates side by side

Run one comparison with your current model as the baseline and the candidates next to it. In Truvec, select Comparison in the project sidebar, select the committed dataset and the models, and set top k to what your system sends to the LLM. You don't need to configure any provider API key.

Supported models include OpenAI (text-embedding-3-small, 3-large, ada-002), Cohere (Embed v4, English v3, Multilingual v3), Voyage AI (voyage-4-large, voyage-4, voyage-4-lite), Google (gemini-embedding-2 and 001) and open-source models (BGE, Nomic Embed, mxbai, GTE, Qwen3 Embedding 8B). If you're still on text-embedding-ada-002, this is a quick way to see whether a newer model is worth the move.

Step 4: Read the results like a migration decision

The comparison ranks models by MRR, then recall (see the RAG evaluation metrics that matter if these are new to you). Then look at:

  • The question-by-question view. In the Retrieval Inspector (open the comparison from Comparison History), model buttons show where each model ranked the right chunk for the current question (rank 1, rank 4, miss). The questions where models disagree are the most informative.
  • Regressions. Filter on misses for the candidate. Are there questions your current model gets right and the candidate misses? A migration that wins on average but breaks your billing questions is a bad trade.
  • Score gap. The gap between the best relevant chunk and the best distractor shows how clearly each model separates right from look-alike. A small or negative gap means rankings that flip on small rewordings.
  • Cost and footprint. Index cost, cost per 1,000 queries, latency and vector storage per model. Index cost on your sample multiplied by (full corpus divided by sample size) gives a rough re-indexing estimate. Going from 1,536 to 3,072 dimensions doubles vector storage.

Let's take an example to illustrate using hypothetical numbers and 40 questions test set:

Model MRR Recall@5 Cost / 1k queries Index cost Storage
Current model (A) 0.79 91% $0.0004 $0.20 60 MB
Candidate model (B) 0.81 92% $0.0026 $1.30 120 MB

A 0.02 MRR difference is within noise. Model B costs 6.5 times more per query and doubles storage, and switching means re-indexing everything. Keep model A, unless the per-question view shows B consistently wins on your most important questions.

If B were at 0.90 MRR instead, the gap would likely be real and, for most use cases, worth the migration: better retrieval directly means better answers.

Step 5: Check the alternatives that don't need re-indexing

A new embedding model isn't the only way to improve retrieval, and it's the most expensive one. Two changes leave your index untouched:

  • A reranker re-scores the top candidates of your current model at query time. If the current model finds the right chunk but ranks it too low, a reranker often closes the gap.
  • Hybrid search adds a keyword search (BM25) next to the vector search. If the misses involve product names, error codes or references, it may recover them for free.

In Truvec, run a second comparison with Compare strategies: your current model as the baseline, then the same model with a reranker, the same model with hybrid search, and the candidate model (Another model). Each configuration shows its MRR difference with the baseline, the questions it fixed and broke, and its added latency and cost per 1,000 queries.

The trade-off is different from a migration. A reranker costs nothing to set up but is billed on every query: Cohere Rerank 3.5 is $2.00 per 1,000 searches, far more than embedding the question. A migration costs a one-time re-indexing but usually little per query. At low traffic, the reranker is often the cheaper path to the same quality; at high traffic, the migration often wins. The comparison gives you the numbers for both.

Step 6: Record the decision

Export the comparison as a PDF report: select Reports in the project sidebar and click Download PDF on the comparison row. It includes a recommendation, the leaderboard, a head-to-head section (for each pair of models, how often one ranked a relevant chunk higher, and how often only the winner hit or the winner missed while others hit), and a question matrix. It's the evidence you attach to the migration ticket or design review.

A checklist before migration

Before migrating to a new embedding model, ensure you check the following checklist:

  • The candidate was compared with the current model as a baseline, on the same chunks and the same top k.
  • The golden dataset was reviewed, includes real user questions, and has 50+ questions for a close call.
  • You read the misses of the candidate and checked for regressions on critical questions.
  • You estimated re-indexing cost, query cost at your traffic, and storage at your scale.
  • You checked whether a reranker or hybrid search on the current model gets the same gain without re-indexing.
  • The decision and its evidence are recorded in a report.

FAQ

Is MTEB useless for choosing an embedding model?

No. It's a good way to build a shortlist of competitive models. It can't tell you how those models rank on your documents and questions, so use it to pick candidates, then test them on your data.

Why does switching embedding models require re-indexing?

Each model maps text into its own vector space. A question embedded with model B can't be meaningfully compared to chunks embedded with model A, so every chunk has to be re-embedded with the new model.

How many test questions do I need before migrating?

50 to 100 for a confident decision between models with close scores. 20 to 30 is enough to spot large differences.

Can I test my production chunks as they are?

Yes. Truvec accepts pre-chunked uploads in JSON, JSONL or CSV and evaluates exactly those chunks.

Is adding a reranker an alternative to switching models?

Often. A reranker improves the ranking of what your current model retrieves, without re-indexing. It can't recover a chunk the model doesn't retrieve at all, and it adds a cost and latency to every query. Compare both on the same golden dataset before deciding.

Do I need API keys for each provider?

No. Truvec runs the supported models and rerankers on its own accounts, and evaluation usage is included in your plan.

Test the migration before you pay for it

Re-indexing is the expensive part of a model switch. Testing the switch on a sample first is the cheap part. Truvec's Free plan (no card) lets you compare up to 3 configurations (commercial models, with or without a reranker) on 5 documents and 20 questions; the Starter plan ($129 one-time, 30 days of access) is designed to settle one decision, such as validating a migration off a legacy model. Get started at app.truvec.dev.

Related reading: