# Leaderboards won't pick your embedding model: how to choose one for RAG

> Why public embedding leaderboards are a weak signal for your RAG system, which retrieval metrics matter, how to build a test set from your own documents, and how to read the results without fooling yourself.

Published 2026-10-06, updated 2026-11-14 by Truvec team. Canonical: https://truvec.dev/blog/choosing-an-embedding-model-for-rag/

A RAG system can only answer from the passages it retrieves. If the right chunk never reaches the LLM, no prompt and no model upgrade will fix the answer. The embedding model decides which chunks come back, yet most teams pick it from a [leaderboard](/blog/mteb-benchmark-validate-embedding-model-before-switching/), or keep whatever the tutorial used.

This post explains why that choice deserves a measurement on your own documents, what to measure, and how to read the numbers. The last section is a real benchmark run with Truvec, with its raw data.

## Why public leaderboards are a weak signal

Public leaderboards such as MTEB are useful: they show which models are broadly competitive. But the score at the top of the table is an **average over many tasks and datasets**: classification, clustering, sentence similarity, reranking and retrieval, mostly on public data. Your RAG system does one of those tasks, on data the model has never seen.

Three things make the ranking shift on real documents:

- **Domain vocabulary.** Product names, error codes, internal acronyms and API identifiers are rare in public training data. A model that ranks well on Wikipedia-style questions can confuse two of your product names that look alike.
- **Document shape.** Reference docs, tickets, contracts and changelogs chunk differently from web pages. Short chunks full of code or tables behave differently from paragraphs of prose.
- **Question style.** Your users ask short, vague, sometimes misspelled questions. Benchmark queries are usually clean.

On top of that, public datasets can leak into training data, which flatters the models that saw them. The gap between the first models on a leaderboard is often smaller than the shift you get from changing the domain.

## What "better retrieval" means for RAG

Retrieval quality is measured against a **golden set**: questions, each linked to the chunks that contain its answer. For each question you retrieve the top *k* chunks and check where the relevant ones landed. Five metrics cover what matters:

| Metric | Question it answers | Use it when |
|---|---|---|
| Hit rate@k | Did at least one relevant chunk make it into the top *k*? | You send all *k* chunks to the LLM: this is "can the LLM answer at all?" |
| Recall@k | What share of the relevant chunks were found? | Answers span several chunks |
| MRR | How high was the first relevant chunk? | You send few chunks, or rerank |
| nDCG@k | How close is the ranking to the ideal one? | Several chunks are relevant with different importance |
| Precision@k | What share of the *k* chunks are relevant? | Context is expensive, or noise confuses your LLM |

Hit rate is the easiest to explain to the rest of the team. [MRR](/blog/rag-evaluation-metrics-that-matter/) is the best single tiebreaker: two models with the same hit rate are not equal if one puts the answer first and the other puts it fifth.

Both are simple to compute once you know the rank of the first relevant chunk for each question:

```python
def hit_rate(first_ranks: list[int | None], k: int) -> float:
    """Share of questions whose first relevant chunk is in the top k (None: not found)."""
    return sum(1 for r in first_ranks if r is not None and r <= k) / len(first_ranks)

def mrr(first_ranks: list[int | None]) -> float:
    """1 if the first relevant chunk is ranked first, 1/2 if second... 0 if not found."""
    return sum(1 / r for r in first_ranks if r is not None) / len(first_ranks)
```

## Build the test set from your own documents

The [golden set](/blog/prototype-rag-retrieval-strategy/) is where most evaluations go wrong. A few rules:

1. **Use the documents your system will actually serve**, chunked the way production chunks them. Changing the chunk size changes the results, so compare models on identical chunks.
2. **Generate, then review.** An LLM can draft questions from sampled chunks and record which chunks answer them. A human pass removes questions that are ambiguous, answered by other chunks too, or not something a user would ask.
3. **Avoid copied wording.** A question that repeats a sentence of its chunk word for word is easy for every model and hides real differences. Questions should paraphrase, the way users do.
4. **Spread the questions across documents.** Fifty questions on one long page tell you about that page, not about the corpus.
5. **Aim for 100 questions or more.** Fewer than that, and noise drowns out the differences between models (see below).

## Quality isn't the only cost

Two models with similar quality can differ a lot in what they cost to run:

- **Embedding price.** You pay once to embed the corpus, again every time you re-embed it (new chunking, new model), and a little for each query. On large corpora, the one-time cost dominates.
- **Dimensions and storage.** A vector takes `dimension × 4` bytes as float32. At a million chunks, 3,072 dimensions is about 12 GB of raw vectors against 3 GB at 768. Many vector databases keep vectors in memory, so this becomes RAM you pay for every month.
- **Latency.** Query embedding is a network call to the provider on every user request. Its latency adds directly to time-to-first-token.
- **Input limits and prefixes.** Some models truncate long chunks silently, others expect an instruction prefix on queries or documents. Getting this wrong quietly lowers scores.
- **Lock-in.** Vectors from different models aren't compatible. Switching later means re-embedding everything, so it's worth getting the choice right first.

## Read the results without fooling yourself

With 100 questions and a hit rate around 80%, the standard error is about 4 points (√(0.8 × 0.2 / 100)). **A 2-point gap between two models is noise.** Treat close models as tied, then choose between them on cost, latency, storage and the provider you'd rather depend on.

Look past the averages too:

- **Hit rate by position** shows whether a model finds the answer first or only by position 5. If you plan to send fewer chunks, the first positions matter most.
- **Score separation** shows whether a similarity threshold will work. If relevant and irrelevant chunks get similar scores, a threshold will either drop good chunks or let noise through.
- **Questions where models disagree** tell you *why* one wins: often a family of questions (error codes, product names, another language) that one model handles and the others don't.

## A checklist

- Fix the chunking first, and compare every model on the same chunks.
- Build a golden set of at least 100 reviewed questions from your own documents.
- Compare hit rate and MRR at the *k* you use in production.
- Treat gaps smaller than the noise as ties, and break ties on cost, latency and storage.
- Re-run the comparison when your documents or the available models change.

The benchmark below follows this method on one corpus. Its winner is a result about *these* documents, not a general ranking: run the same comparison on yours before you commit.

## Benchmark: Six embedding models on a developer documentation corpus

Real benchmark generated by Truvec (benchmark #7, run 2026-11-14). Raw data: [benchmark.json](https://truvec.dev/blog/choosing-an-embedding-model-for-rag/benchmark.json), [benchmark.csv](https://truvec.dev/blog/choosing-an-embedding-model-for-rag/benchmark.csv). Full report with every question: [report.pdf](https://truvec.dev/blog/choosing-an-embedding-model-for-rag/report.pdf).

- Corpus: Docker documentation: manuals, FAQs, billing and support pages (https://github.com/docker/docs), 261 documents, 1,944 chunks
- Chunking: recursive, 1000 chars, 200 overlap
- Golden set: 100 questions generated by gpt-5.6-luna, human-reviewed
- Retrieval: top 5 by cosine similarity; models ranked by MRR, then recall@k

### Key results

- **Voyage 4** ranks first, with an MRR of 0.915 and a hit rate of 100.0% in the top 5.
- It leads the runner-up, Gemini Embedding 001, by 0.046 MRR.
- Hit rate ranges from 95.0% to 100.0% across the 6 models.
- Fastest to embed a query: **OpenAI text-embedding-3-small** (4.5 ms per query, measured in batch from our servers).

### Retrieval quality

| Rank | Model | Provider | Hit rate@5 | Recall@5 | MRR | nDCG@5 | Precision@5 |
| --- | --- | --- | ---: | ---: | ---: | ---: | ---: |
| 1 | Voyage 4 | voyage | 100.0% | 99.5% | 0.915 | 0.929 | 21.2% |
| 2 | Gemini Embedding 001 | google | 98.0% | 97.5% | 0.869 | 0.890 | 20.8% |
| 3 | OpenAI text-embedding-3-large | openai | 99.0% | 97.5% | 0.867 | 0.885 | 20.6% |
| 4 | Cohere Embed v4 | cohere | 98.0% | 96.5% | 0.859 | 0.875 | 20.4% |
| 5 | BGE Base EN v1.5 | fireworks | 95.0% | 92.5% | 0.773 | 0.802 | 19.4% |
| 6 | OpenAI text-embedding-3-small | openai | 95.0% | 93.5% | 0.769 | 0.807 | 19.8% |

### Hit rate by position

| Model | Top 1 | Top 2 | Top 3 | Top 4 | Top 5 |
| --- | ---: | ---: | ---: | ---: | ---: |
| Voyage 4 | 85.0% | 95.0% | 99.0% | 99.0% | 100.0% |
| Gemini Embedding 001 | 78.0% | 92.0% | 97.0% | 98.0% | 98.0% |
| OpenAI text-embedding-3-large | 79.0% | 88.0% | 94.0% | 98.0% | 99.0% |
| Cohere Embed v4 | 76.0% | 93.0% | 95.0% | 97.0% | 98.0% |
| BGE Base EN v1.5 | 64.0% | 85.0% | 90.0% | 92.0% | 95.0% |
| OpenAI text-embedding-3-small | 63.0% | 84.0% | 92.0% | 94.0% | 95.0% |

### Cost, speed and size

| Model | Price / 1M tokens | Embed the corpus | 1,000 queries | Query embedding | Search p95 | Vectors |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Voyage 4 | $0.06 | $0.02 | $0.0012 | 17 ms | 36 ms | 7.6 MB |
| Gemini Embedding 001 | $0.15 | $0.05 | $0.0030 | 22 ms | 39 ms | 22.8 MB |
| OpenAI text-embedding-3-large | $0.13 | $0.04 | $0.0026 | 15 ms | 50 ms | 22.8 MB |
| Cohere Embed v4 | $0.12 | $0.04 | $0.0024 | 13 ms | 51 ms | 11.4 MB |
| BGE Base EN v1.5 | $0.0080 | $0.0027 | $0.0002 | 14 ms | 13 ms | 5.7 MB |
| OpenAI text-embedding-3-small | $0.02 | $0.0067 | $0.0004 | 4.5 ms | 37 ms | 11.4 MB |

### Score separation

| Model | Relevant chunk | Best distractor | Gap |
| --- | ---: | ---: | ---: |
| Voyage 4 | 0.692 | 0.646 | 0.046 |
| Gemini Embedding 001 | 0.796 | 0.781 | 0.015 |
| OpenAI text-embedding-3-large | 0.687 | 0.650 | 0.037 |
| Cohere Embed v4 | 0.684 | 0.638 | 0.046 |
| BGE Base EN v1.5 | 0.806 | 0.782 | 0.023 |
| OpenAI text-embedding-3-small | 0.715 | 0.687 | 0.027 |

### Example questions

Rank of the first relevant chunk per model ("miss": not in the top 5).

| Question | Voyage 4 | Gemini Embedding 001 | OpenAI text-embedding-3-large | Cohere Embed v4 | BGE Base EN v1.5 | OpenAI text-embedding-3-small |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| How do I push an image to Docker Hub, and what tagging restriction applies? | 5 | miss | miss | miss | 3 | 1 |
| When both JIT and SCIM are enabled, how are organization roles and account deactivation handled? | 1 | 1 | 5 | 3 | 3 | miss |
| How do I search Docker Desktop logs, move between matching entries, and clear the terminal? | 1 | 3 | 4 | 2 | 1 | 1 |
| What file and service configuration are required for Docker Hub automated tests, and how is the test result determined? | 1 | 1 | 2 | miss | 5 | miss |
| What system requirements and preliminary packages are needed to install Docker Desktop on Ubuntu? | 1 | 1 | 1 | 2 | 4 | 3 |
| When multiple Compose files are supplied with `-f`, how are their service settings combined, and which file takes precedence? | 2 | 1 | 1 | 4 | miss | 4 |
| How do image indexes enable one container image reference to work across AMD and ARM platforms? | 1 | 2 | 4 | 1 | 1 | 3 |
| What command builds the Docker image for the Spring Boot hello world app, and how can I check the size of the resulting image? | 3 | 1 | 1 | 2 | 1 | 5 |

### Notes

- The corpus is a subset of the public Docker documentation (Apache 2.0), taken from github.com/docker/docs at commit 553482c: the Desktop, Docker Hub, Compose, accounts, security and subscription and billing manuals, the FAQs, the support pages and the getting-started guides. Release notes and pages shorter than 600 characters were left out, and pages were cut at 40,000 characters.
- Several posts on this blog use this corpus and the same 100 questions, each with its own selection of models. A model can differ by a question between two of these runs: questions are embedded again at each run, and vector search is approximate.
- With 100 questions, a gap of a couple of points between two models is within the noise.

### Metric definitions

- **Hit rate**: Share of questions with at least one relevant chunk in the top k results. The most direct answer to "does the right passage reach the LLM?".
- **Recall**: Share of the relevant chunks found in the top k, averaged over questions. Matters when an answer spans several chunks.
- **MRR**: Mean reciprocal rank: 1 when the first relevant chunk is ranked first, 1/2 when second, 0 when missing. Rewards putting the answer at the top.
- **nDCG**: Normalized discounted cumulative gain: how close the ranking of all relevant chunks is to the ideal ranking.
- **Precision**: Share of the top k results that are relevant. Low precision means more noise sent to the LLM.
