Leaderboards won't pick your embedding model: how to choose one for RAG

Why public embedding leaderboards are a weak signal for your RAG system, which retrieval metrics matter, how to build a test set from your own documents, and how to read the results without fooling yourself.

  • embeddings
  • rag
  • retrieval
  • evaluation

Download the benchmark report (PDF)

A RAG system can only answer from the passages it retrieves. If the right chunk never reaches the LLM, no prompt and no model upgrade will fix the answer. The embedding model decides which chunks come back, yet most teams pick it from a leaderboard, or keep whatever the tutorial used.

This post explains why that choice deserves a measurement on your own documents, what to measure, and how to read the numbers. The last section is a real benchmark run with Truvec, with its raw data.

Why public leaderboards are a weak signal

Public leaderboards such as MTEB are useful: they show which models are broadly competitive. But the score at the top of the table is an average over many tasks and datasets: classification, clustering, sentence similarity, reranking and retrieval, mostly on public data. Your RAG system does one of those tasks, on data the model has never seen.

Three things make the ranking shift on real documents:

  • Domain vocabulary. Product names, error codes, internal acronyms and API identifiers are rare in public training data. A model that ranks well on Wikipedia-style questions can confuse two of your product names that look alike.
  • Document shape. Reference docs, tickets, contracts and changelogs chunk differently from web pages. Short chunks full of code or tables behave differently from paragraphs of prose.
  • Question style. Your users ask short, vague, sometimes misspelled questions. Benchmark queries are usually clean.

On top of that, public datasets can leak into training data, which flatters the models that saw them. The gap between the first models on a leaderboard is often smaller than the shift you get from changing the domain.

What "better retrieval" means for RAG

Retrieval quality is measured against a golden set: questions, each linked to the chunks that contain its answer. For each question you retrieve the top k chunks and check where the relevant ones landed. Five metrics cover what matters:

Metric Question it answers Use it when
Hit rate@k Did at least one relevant chunk make it into the top k? You send all k chunks to the LLM: this is "can the LLM answer at all?"
Recall@k What share of the relevant chunks were found? Answers span several chunks
MRR How high was the first relevant chunk? You send few chunks, or rerank
nDCG@k How close is the ranking to the ideal one? Several chunks are relevant with different importance
Precision@k What share of the k chunks are relevant? Context is expensive, or noise confuses your LLM

Hit rate is the easiest to explain to the rest of the team. MRR is the best single tiebreaker: two models with the same hit rate are not equal if one puts the answer first and the other puts it fifth.

Both are simple to compute once you know the rank of the first relevant chunk for each question:

def hit_rate(first_ranks: list[int | None], k: int) -> float:
    """Share of questions whose first relevant chunk is in the top k (None: not found)."""
    return sum(1 for r in first_ranks if r is not None and r <= k) / len(first_ranks)

def mrr(first_ranks: list[int | None]) -> float:
    """1 if the first relevant chunk is ranked first, 1/2 if second... 0 if not found."""
    return sum(1 / r for r in first_ranks if r is not None) / len(first_ranks)

Build the test set from your own documents

The golden set is where most evaluations go wrong. A few rules:

  1. Use the documents your system will actually serve, chunked the way production chunks them. Changing the chunk size changes the results, so compare models on identical chunks.
  2. Generate, then review. An LLM can draft questions from sampled chunks and record which chunks answer them. A human pass removes questions that are ambiguous, answered by other chunks too, or not something a user would ask.
  3. Avoid copied wording. A question that repeats a sentence of its chunk word for word is easy for every model and hides real differences. Questions should paraphrase, the way users do.
  4. Spread the questions across documents. Fifty questions on one long page tell you about that page, not about the corpus.
  5. Aim for 100 questions or more. Fewer than that, and noise drowns out the differences between models (see below).

Quality isn't the only cost

Two models with similar quality can differ a lot in what they cost to run:

  • Embedding price. You pay once to embed the corpus, again every time you re-embed it (new chunking, new model), and a little for each query. On large corpora, the one-time cost dominates.
  • Dimensions and storage. A vector takes dimension × 4 bytes as float32. At a million chunks, 3,072 dimensions is about 12 GB of raw vectors against 3 GB at 768. Many vector databases keep vectors in memory, so this becomes RAM you pay for every month.
  • Latency. Query embedding is a network call to the provider on every user request. Its latency adds directly to time-to-first-token.
  • Input limits and prefixes. Some models truncate long chunks silently, others expect an instruction prefix on queries or documents. Getting this wrong quietly lowers scores.
  • Lock-in. Vectors from different models aren't compatible. Switching later means re-embedding everything, so it's worth getting the choice right first.

Read the results without fooling yourself

With 100 questions and a hit rate around 80%, the standard error is about 4 points (√(0.8 × 0.2 / 100)). A 2-point gap between two models is noise. Treat close models as tied, then choose between them on cost, latency, storage and the provider you'd rather depend on.

Look past the averages too:

  • Hit rate by position shows whether a model finds the answer first or only by position 5. If you plan to send fewer chunks, the first positions matter most.
  • Score separation shows whether a similarity threshold will work. If relevant and irrelevant chunks get similar scores, a threshold will either drop good chunks or let noise through.
  • Questions where models disagree tell you why one wins: often a family of questions (error codes, product names, another language) that one model handles and the others don't.

A checklist

  • Fix the chunking first, and compare every model on the same chunks.
  • Build a golden set of at least 100 reviewed questions from your own documents.
  • Compare hit rate and MRR at the k you use in production.
  • Treat gaps smaller than the noise as ties, and break ties on cost, latency and storage.
  • Re-run the comparison when your documents or the available models change.

The benchmark below follows this method on one corpus. Its winner is a result about these documents, not a general ranking: run the same comparison on yours before you commit.

Real benchmark - generated by Truvec

Benchmark: Six embedding models on a developer documentation corpus

Corpus
Docker documentation: manuals, FAQs, billing and support pages
Documents
261
Chunks
1,944
Chunking
recursive, 1000 chars, 200 overlap
Golden set
100 questions, human-reviewed
Models
6 compared
Top k
5
Run
2026-11-14

Download the full report (PDF) Every question, with the passages each model retrieved.

Key results

  • Voyage 4 ranks first, with an MRR of 0.915 and a hit rate of 100.0% in the top 5.
  • It leads the runner-up, Gemini Embedding 001, by 0.046 MRR.
  • Hit rate ranges from 95.0% to 100.0% across the 6 models.
  • Fastest to embed a query: OpenAI text-embedding-3-small (4.5 ms per query, measured in batch from our servers).

Retrieval quality

Retrieval quality at top 5, ranked by MRR, then recall@k
ModelHit rate@5Recall@5MRRnDCG@5Precision@5
1Voyage 4voyage - 1024 dims100.0%99.5%0.9150.92921.2%
2Gemini Embedding 001google - 3072 dims98.0%97.5%0.8690.89020.8%
3OpenAI text-embedding-3-largeopenai - 3072 dims99.0%97.5%0.8670.88520.6%
4Cohere Embed v4cohere - 1536 dims98.0%96.5%0.8590.87520.4%
5BGE Base EN v1.5fireworks - 768 dims95.0%92.5%0.7730.80219.4%
6OpenAI text-embedding-3-smallopenai - 1536 dims95.0%93.5%0.7690.80719.8%

Hit rate by position

How many questions the first results already answer. A model that finds the answer at position 1 lets you send fewer chunks to the LLM.

Hit rate by position: share of questions answered by the first j results
ModelTop 1Top 2Top 3Top 4Top 5
1Voyage 4voyage - 1024 dims85.0%95.0%99.0%99.0%100.0%
2Gemini Embedding 001google - 3072 dims78.0%92.0%97.0%98.0%98.0%
3OpenAI text-embedding-3-largeopenai - 3072 dims79.0%88.0%94.0%98.0%99.0%
4Cohere Embed v4cohere - 1536 dims76.0%93.0%95.0%97.0%98.0%
5BGE Base EN v1.5fireworks - 768 dims64.0%85.0%90.0%92.0%95.0%
6OpenAI text-embedding-3-smallopenai - 1536 dims63.0%84.0%92.0%94.0%95.0%

Cost, speed and size

Embedding prices are the providers' list prices. Corpus cost is a one-time cost per model; the vector size is raw float32 storage, before the vector database's own overhead.

Cost, speed and size on this corpus
ModelPrice / 1M tokensEmbed the corpus1,000 queriesQuery embeddingSearch p95Vectors
1Voyage 4voyage - 1024 dims$0.06$0.02$0.001217 ms36 ms7.6 MB
2Gemini Embedding 001google - 3072 dims$0.15$0.05$0.003022 ms39 ms22.8 MB
3OpenAI text-embedding-3-largeopenai - 3072 dims$0.13$0.04$0.002615 ms50 ms22.8 MB
4Cohere Embed v4cohere - 1536 dims$0.12$0.04$0.002413 ms51 ms11.4 MB
5BGE Base EN v1.5fireworks - 768 dims$0.0080$0.0027$0.000214 ms13 ms5.7 MB
6OpenAI text-embedding-3-smallopenai - 1536 dims$0.02$0.0067$0.00044.5 ms37 ms11.4 MB

Score separation

Average similarity of the best relevant chunk and of the best irrelevant one. A wide gap makes a similarity threshold reliable; a narrow one means right and wrong chunks look alike to the model.

Score separation: how clearly each model scores the right chunk above the best wrong one
ModelRelevant chunkBest distractorGap
1Voyage 4voyage - 1024 dims0.6920.6460.046
2Gemini Embedding 001google - 3072 dims0.7960.7810.015
3OpenAI text-embedding-3-largeopenai - 3072 dims0.6870.6500.037
4Cohere Embed v4cohere - 1536 dims0.6840.6380.046
5BGE Base EN v1.5fireworks - 768 dims0.8060.7820.023
6OpenAI text-embedding-3-smallopenai - 1536 dims0.7150.6870.027

Example questions

Rank of the first relevant chunk for each model; miss means it wasn't in the top 5. These are the questions where the models disagree most.

Example questions from the golden set
QuestionVoyage 4Gemini Embedding 001OpenAI text-embedding-3-largeCohere Embed v4BGE Base EN v1.5OpenAI text-embedding-3-small
How do I push an image to Docker Hub, and what tagging restriction applies?
Expected passage
## Push an image to Docker Hub Select the image from the list, select the **More options** button and select **Push to Hub**. > [!NOTE] > > You can only push an image to Docker Hub if the image belongs to your Docker ID or your organization. That is, the image must contain the co
5missmissmiss31
When both JIT and SCIM are enabled, how are organization roles and account deactivation handled?
Expected passage
Roles don't move back and forth. JIT sets the organization role only when it first adds the user. A later SCIM update can change that role, and the next sign-in leaves the SCIM role in place. When a user isn't assigned to the Docker application in the IdP, the next synchronizatio
11533miss
How do I search Docker Desktop logs, move between matching entries, and clear the terminal?
Expected passage
### Logs Select **Logs** to view output from the container in real time. While viewing logs, you can: - Use `Cmd + f`/`Ctrl + f` to open the search bar and find specific entries. Search matches are highlighted in yellow. - Press `Enter` or `Shift + Enter` to jump to the next or p
134211
What file and service configuration are required for Docker Hub automated tests, and how is the test result determined?
Expected passage
## Set up automated test files To set up your automated tests, create a `docker-compose.test.yml` file which defines a `sut` service that lists the tests to be run. The `docker-compose.test.yml` file should be located in the same directory that contains the Dockerfile used to bui
112miss5miss
What system requirements and preliminary packages are needed to install Docker Desktop on Ubuntu?
Expected passage
# Install Docker Desktop on Ubuntu > **Docker Desktop terms** > > Commercial use of Docker Desktop in larger enterprises (more than 250 > employees or more than $10 million USD in annual revenue) requires a paid > subscription. This page contains information on how to install, la
111243
When multiple Compose files are supplied with `-f`, how are their service settings combined, and which file takes precedence?
Expected passage
Compose merges files in the order they're specified on the command line. Subsequent files may merge, override, or add to their predecessors. For example: ```console $ docker compose -f compose.yaml -f compose.admin.yaml run backup_db ``` The `compose.yaml` file might specify a `w
2114miss4
How do image indexes enable one container image reference to work across AMD and ARM platforms?
Expected passage
# Image Management Images and image indexes are the foundation of container images within a repository. The following diagram shows the relationship between images and image indexes. This structure enables multi-architecture support through a single reference. It is important to
124113
What command builds the Docker image for the Spring Boot hello world app, and how can I check the size of the resulting image?
Expected passage
```dockerfile CMD ["./mvnw", "spring-boot:run"] ``` And with that, you should have the following Dockerfile: ```dockerfile FROM eclipse-temurin:21.0.8_9-jdk-jammy WORKDIR /app COPY .mvn/ .mvn COPY mvnw pom.xml ./ RUN ./mvnw dependency:go-offline COPY src ./src CMD ["./mvnw", "spr
311215

Notes

  • The corpus is a subset of the public Docker documentation (Apache 2.0), taken from github.com/docker/docs at commit 553482c: the Desktop, Docker Hub, Compose, accounts, security and subscription and billing manuals, the FAQs, the support pages and the getting-started guides. Release notes and pages shorter than 600 characters were left out, and pages were cut at 40,000 characters.
  • Several posts on this blog use this corpus and the same 100 questions, each with its own selection of models. A model can differ by a question between two of these runs: questions are embedded again at each run, and vector search is approximate.
  • With 100 questions, a gap of a couple of points between two models is within the noise.

Methodology

Truvec split the documents into chunks, generated a golden set of questions with gpt-5.6-luna (each question is linked to the chunks that answer it), embedded the corpus and the questions with every model, and retrieved the top 5 chunks for each question by cosine similarity. Every model saw the same chunks and the same questions. Scores are averaged over all 100 questions.

Hit rate
Share of questions with at least one relevant chunk in the top k results. The most direct answer to "does the right passage reach the LLM?".
Recall
Share of the relevant chunks found in the top k, averaged over questions. Matters when an answer spans several chunks.
MRR
Mean reciprocal rank: 1 when the first relevant chunk is ranked first, 1/2 when second, 0 when missing. Rewards putting the answer at the top.
nDCG
Normalized discounted cumulative gain: how close the ranking of all relevant chunks is to the ideal ranking.
Precision
Share of the top k results that are relevant. Low precision means more noise sent to the LLM.

Downloads: full report (PDF) - benchmark.json - benchmark.csv - this post as Markdown

Voyage 4 won here, on these documents. Find the model that wins on yours: run the same benchmark on your documents with Truvec.