Skip to content

Test embedding models and rerankers on your own documents.

Upload a sample of your documents. Truvec writes test questions from them, runs 16 embedding models with hybrid search and rerankers, and shows which setup finds the right passage, how fast and at what cost.

Free plan, no card: 5 documents, 20 test questions, one comparison of 3 setups.

  • No provider API keys.
  • First results in about 15 minutes.
  • Hosted in the EU, never used for training.

16 embedding models and 3 rerankers, run for you

  • OpenAI
  • Cohere
  • Voyage AI
  • Google
  • BGE
  • Nomic
  • mxbai
  • GTE
  • Qwen3

Most retrieval pipelines ship on defaults and a blog post.

Retrieval decides what a RAG assistant can answer. The embedding model, the chunking, hybrid search and the reranker deserve a measurement on the documents it will actually search.

  1. Public benchmarks average over someone else's text.

    MTEB scores and chunking rules of thumb come from generic datasets. Support articles, contracts or code have their own vocabulary, and the best embedding model often changes once the questions come from them.

  2. A retrieval miss doesn't look like an error.

    When the right passage isn't retrieved, the LLM still answers, confidently, from the wrong one. Nothing in your logs points at retrieval.

  3. Every part of the pipeline is a trade-off, and they interact.

    A reranker can make a cheap model beat an expensive one, and cost more per query than the embedding itself. A new model or chunk size means re-indexing the whole corpus. Measuring on a sample first is cheap; migrating twice is not.

See why a model misses, question by question

An average hides the questions that matter. The inspector puts every test question next to what each model retrieved, so you can read what it ranked first instead and judge if the miss would reach a user.

  • Ties get separated. voyage-4 and text-embedding-3-large both reach 95% hit rate. MRR shows voyage-4 puts the expected chunk first more often, which matters when only the top 1-2 chunks go to the LLM.
  • Misses have a pattern. On this help center, the open-source models rank generic login and password pages above the SSO article. Product names and SSO vocabulary are where they lose most.
  • You can check your own wording. The playground runs any question you type against every model you indexed.

Example project: a support team's help center, 38 articles, 412 chunks, 40 generated questions reviewed by hand, top 5.

Inspector - Help center golden setQ17 / 40

Can we sign in with Okta?

ModelRank 1-5ExpectedScoreGap
#10.812+0.068
#10.774+0.022
#20.701−0.017
#90.598−0.066
#70.641−0.047

Top 5 retrieved by BGE Base EN v1.5

  1. 1login-methods.md - 1Sign in with an email and password, a magic link, or your company account.0.688
  2. 2password-reset.md - 1Reset links are valid for 30 minutes and can be used once.0.671
  3. 3account-security.md - 2Admins can require two-factor authentication for every member.0.668
  4. 4two-factor.md - 1Codes can come from an authenticator app or a hardware security key.0.659
  5. 5user-roles.md - 5Admins can manage roles and permissions for each member.0.650
  6. 7sso-and-scim.md - 3expected chunk, outside the top 50.641
Missed by 2 of the 5 models shown. Both open-source models rank generic login, password and 2FA pages above the SSO article: the average score doesn't show that, the question does. Select a model to see what it retrieved.

The ranking for the whole test set

Seven embedding models on the same 40 questions, with the cost of every 1,000 queries next to the quality.

Leaderboard - Example: a help center40 questions
ModelHit rate@5MRRPer 1k queries
voyage-4Voyage AI95.0%0.848$0.0012
text-embedding-3-largeOpenAI95.0%0.828$0.0026
Cohere Embed v4Cohere92.5%0.807$0.0024
gemini-embedding-001Google92.5%0.794$0.0029
text-embedding-3-smallOpenAI90.0%0.760$0.0004
nomic-embed-text v1.5Open source87.5%0.712$0.0002
BGE Base EN v1.5Open source82.5%0.661$0.0002

voyage-4 finds the expected passage for 38 of the 40 questions.

text-embedding-3-small finds 36, at a third of the cost per query.

Then test hybrid search and a reranker: can the cheap model catch up?

A strategy comparison keeps text-embedding-3-small and changes one thing per configuration. Each one is read against the baseline: the questions it fixes, the ones it breaks, and what it adds to the cost of a query.

Configurationvs baselineHit rate@5MRRPer 1k queries
text-embedding-3-smallBaseline: dense search—90.0%0.760$0.0004
+ hybrid searchBM25 + dense, 50/50+2 fixed−1 broken92.5%0.781$0.0004
+ Cohere Rerank 3.530 candidates reranked+3 fixed−1 broken95.0%0.853$2.00
+ hybrid + rerankBoth+3 fixed−0 broken97.5%0.871$2.00
With a reranker, 3-small passes voyage-4 on MRR (0.853 against 0.848), but queries go from $0.0004 to $2.00 per 1,000: Cohere bills each search. Hybrid search is free to add and fixes two questions that name an exact feature. Which trade-off is right depends on your traffic; the numbers to decide it are side by side.

From your documents to a leaderboard in about 15 minutes

Most of that time is spent reviewing the questions. Every comparison after the first takes minutes.

  1. Step 1

    Upload a sample of your documents

    PDF, DOCX, Markdown, TXT, CSV or JSON, or chunks you already have. Choose the splitter, chunk size and overlap your pipeline uses, and read any chunk as retrieval will see it.

    Files
    38 - 2.1 MB
    Chunks
    412
    Splitter
    recursive - 1000 / 200
  2. Step 2

    Review the generated test set

    An LLM writes questions a user could ask, each linked to the chunks that answer it. Rewrite, add or delete questions, then commit the golden set.

    • How long are password reset links valid?214
    • Which regions support customer-managed keys?87
    • What is Truvec?deleted
  3. Step 3

    Run models and strategies on the same questions

    Pick embedding models, add hybrid search or a reranker, and run every setup with the same index settings and top k. Embeddings are cached, so a re-run only pays for what changed.

    • 3-small baseline
    • 3-small - hybrid
    • 3-small - rerank cohere/rerank-v3.5
    • voyage-4 bigger model
  4. Step 4

    Read the result and share it

    A leaderboard with quality, latency and cost, an inspector for every question, a playground for your own wording, and a PDF report with the recommendation and the configuration used.

    3-small + rerank vs 3-small
    fixed 3 questions, broke 1, same on 36
    comparison-help-center.pdf

Retrieval quality, latency and cost, measured together

Every setup runs on the same questions with the same top k, so a cheaper model, a reranker or hybrid search becomes a difference in quality you can weigh against its latency and price. How each metric is computed.

Quality at your top k

Hit rate@k
Share of questions with at least one expected chunk in the top k.
Recall@k
Share of all expected chunks that were retrieved. Matters when an answer spans several chunks.
Precision@k
Share of the top k that is relevant: how much noise reaches the LLM.
MRR
How high the first relevant chunk ranks, averaged. The default ranking metric.
nDCG@k
Rank-weighted gain over all relevant chunks.

Ranking depth

Hit rate @1 … @k
The curve on the right: what you lose by passing fewer chunks to the LLM.
First relevant rank
Distribution over questions, including the ones missed entirely.
Score gap
Similarity of relevant chunks minus distractors. A small gap means a fragile ranking.
vs baseline
In a strategy comparison: metric deltas and the questions each change fixed or broke.

Speed and cost

Latency
Query embedding time, search and reranking time with their p95, and the total per query.
Cost
To index the corpus once, and per 1,000 queries with the reranker included.
Footprint
Dimensions and vector storage for the whole index.

Hit rate by cut-off

Help-center example, 40 questions

50%75%100%@1@2@3@4@5
voyage-477.5% @195.0% @5
text-embedding-3-small65.0% @190.0% @5
BGE Base EN v1.555.0% @182.5% @5

At @5 the three models are 12.5 points apart. At @1, if your prompt only takes the best chunk, the gap is 22.5.

Four decisions worth a retrieval evaluation

From choosing an embedding model for a new RAG project to a change in production: the moments when someone has to defend the choice with numbers.

New RAG project

Choosing the whole pipeline before you build

MRRHit rateCost per 1k queries

You are starting a RAG assistant on internal documentation. Which model, which chunk size, hybrid search or not, a reranker or not? Public leaderboards disagree with each other and none of them was measured on your domain.

Upload a sample, generate questions from it, compare candidate models, then try hybrid search and a reranker on the best ones. You start the build with a pipeline chosen on your data, and a baseline to compare against later.

Strategy

Deciding if a reranker or hybrid search pays off

Fixed / brokenLatencyCost per 1k queries

A reranker is supposed to fix bad rankings, hybrid search to catch exact product names and error codes. Both add latency, and a reranker can cost more per query than the embedding itself.

Keep your model as the baseline and add one configuration per change. Each one shows its metric deltas, the questions it fixed and broke, and what it adds to latency and cost per 1,000 queries.

Chunking

Validating a chunking change

Recall@kScore gapPer-question ranks

You want smaller chunks, a Markdown-aware splitter or more overlap. It feels better, but you have no way to prove the change helps retrieval rather than hurting it.

Load the documents with each chunking in its own project, build a golden set on each, and run the same evaluation. Read the chunks themselves, and see question by question where the expected passage moved up or dropped out of the top k.

Model switch

Migrating or downsizing the embedding model

MRRHead-to-headIndex cost

Your index still runs on text-embedding-ada-002, or the embedding bill grows with traffic. A newer or cheaper model is tempting, but re-embedding costs time and money, and a silent drop in quality would hurt users more than it saves.

Put your current model next to the candidates on the same golden set. The leaderboard shows the quality difference, head-to-head shows exactly which questions you would gain or lose, and the costs show what the switch and every 1,000 queries would cost.

Start free. Pay when you need more volume.

Every plan has the full product, and the embedding, reranking and question-generation usage is included: Truvec runs the models on its own provider accounts, so there are no vendor keys or separate API bills.

Free

No card needed

See Truvec work on your own documents: a small sample, a generated golden set and one comparison of up to 3 models.

$0for 30 days

Start free
  • 1 evaluation run over 30 days
  • Compare up to 3 configurations at once
  • Commercial models (OpenAI, Cohere, Voyage, Google)
  • Up to 5 documents per project
  • 25 MB of uploads in total, up to 10 MB per file
  • Golden sets of up to 20 questions
  • 1 seat

Starter

Most popular

Settle one decision: pick a model before you build, or check a migration off a legacy model.

$129once, 30 days of access

Buy Starter
  • 10 evaluation runs over 30 days
  • No limit on configurations per comparison
  • Commercial and open-source models
  • Up to 10 MB of text per project (~4,000 pages)
  • 200 MB of uploads in total, up to 25 MB per file
  • Golden sets of up to 100 questions
  • 1 seat

Team Pro

For teams

For teams iterating on their RAG pipeline: new documents, new chunking, new models.

$349per month

Choose Team Pro
  • 60 evaluation runs per month
  • No limit on configurations per comparison
  • Commercial and open-source models
  • Up to 25 MB of text per project (~10,000 pages)
  • 2 GB of uploads in total, up to 50 MB per file
  • Golden sets of up to 200 questions
  • 5 team seats

Every plan includes

  • No provider API keys: embedding and generation usage included
  • Auto-generated, editable golden sets
  • All chunking strategies, or upload your own chunks
  • Hybrid search and rerankers (Cohere, Voyage)
  • All quality, speed and cost metrics
  • Playground and question-by-question inspector
  • PDF report export

If a run fails

It doesn't count against your plan, so you can start it again at no cost. Unused one-time purchases are refundable within 14 days.

All limits, plan by plan
FreeStarterTeam Pro
Data
Documents per project
5No limitNo limit
Chunks per project
At the default 1,000-character chunks
500 (~1 MB of text)10,000 (~10 MB of text)25,000 (~25 MB of text)
Total uploads
Combined size of the files uploaded, whole workspace
25 MB200 MB2 GB
Largest file
10 MB25 MB50 MB
Projects
1210
Allowances
Evaluation runs
A comparison of several configurations counts as one run
11060
Configurations per comparison
A model with its retrieval settings: hybrid search, reranker
3No limitNo limit
Open-source models
BGE, nomic, mxbai, GTE, Qwen3…
—IncludedIncluded
Embedding volume
Re-runs reuse embeddings: only the questions and new chunks count
500k tokens (~2 MB)20M tokens (~80 MB)100M tokens (~400 MB)
Generated golden-set questions
203002,000
Questions per golden set
20100200
Playground queries
201,00010,000
Team
Seats
115

Allowances reset every 30 days on monthly and annual plans; on plans that don't renew, they cover the whole access period. A run that fails doesn't count against your runs.

Frequently asked questions

Something missing? Write to support@truvec.dev.

A handful of documents that represent what your assistant will answer questions about: help-center articles, product docs, policies, contracts. PDF, DOCX, Markdown, TXT, CSV and JSON are supported, and you can also upload chunks you already have as JSON, JSONL or CSV. You don't need your full database or your vector store.
No. Truvec runs the embedding models, the rerankers and the golden-set generation for you, and the usage is included in your plan. There are no vendor accounts to create, keys to rotate or separate bills to follow.
Commercial models from OpenAI (text-embedding-3-small, 3-large, ada-002), Cohere (Embed v4, English and Multilingual v3), Voyage AI (voyage-4-large, voyage-4, voyage-4-lite) and Google (Gemini Embedding 2 and 001), and open-source models: BGE, Nomic Embed, mxbai, GTE and Qwen3 Embedding. Missing one you care about? Tell us.
Yes. Any evaluation can run with plain vector search or hybrid search (BM25 keyword search fused with the vector ranking, with an adjustable weight), and with or without a reranker: Cohere Rerank 3.5, Voyage rerank-2.5 or rerank-2.5-lite, re-scoring up to 100 first-stage candidates. A strategy comparison keeps one configuration as the baseline and shows, for each change, the metric deltas, the questions it fixed and broke, and the added latency and cost. Combine them freely: for example, a cheap model with a reranker against a bigger model without one.
An LLM reads a sample of your chunks, spread across your documents, and writes questions a real user could ask, each linked to the chunks that contain the answer. You then review the set: edit a question, change its expected chunks, add your own questions or delete weak ones. Once you are happy, you commit the dataset and it becomes available for evaluations.
Hit rate, recall, precision, F1, MRR and nDCG at your top k; hit rate at every cut-off and the rank of the first relevant chunk; the score gap between relevant chunks and distractors; query embedding, search and reranking latency; and the cost to index the corpus and per 1,000 queries (reranking included), with vector storage size. Strategy comparisons add the deltas against the baseline and the questions each change fixed or broke.
For a typical sample, about 15 minutes from upload to a ranked leaderboard: a minute to upload and chunk, about ten minutes to generate and review the golden set, and the evaluation itself. Larger corpora and golden sets take longer.
Yes. Choose between recursive, character, token, Markdown, code and JSON splitting, set the chunk size and overlap, and open any chunk to read its full text. Truvec warns you when documents of the same type are split with different settings, since that biases the results. A golden set points at specific chunks, so each chunking needs its own: load the same documents with each setting in its own project, generate a golden set on each, and run the same evaluation. The inspector shows, question by question, where the expected passage moved up or dropped out of the top k.
No. Truvec indexes every model in the same vector database, with the same search settings and the same top k, so differences in scores come from what you changed (model, chunking, hybrid search, reranker), not from how your own index is tuned.
Yes. Export any evaluation or comparison as a PDF report with the recommendation, charts, costs, every question with its retrieved chunks, and the configuration used. Golden sets export as JSON, to keep or reuse in your own tests.
Truvec extracts their text and keeps the chunks in your project, on servers in the EU, so you can re-run evaluations as you iterate. The text is sent only to the providers of the models and rerankers you evaluate, and to OpenAI to generate golden sets. You can clear a project's documents or delete the whole project at any time. Truvec does not use your documents to train models.
Enough for a result you can trust. A project holds up to 10 MB of text on Starter(~4,000 pages) and up to 25 MB on Team Pro, at the default chunk size. Re-running an evaluation reuses the embeddings already computed, so only the questions and new documents count towards your monthly embedding volume. See the plan limits.
Yes. Every new account starts on the Free plan, with no card: upload up to 5 of your own documents, generate a golden set of 20 questions and compare up to 3 configurations of commercial models, with hybrid search and rerankers (open-source models come with paid plans), within 30 days. You see real results on your data before you pay. Starter is a one-time purchase with no subscription. A run that fails doesn't count against your plan, so you can start it again at no cost; if it keeps failing, we re-run it or refund it. Unused Starter purchases are refundable within 14 days. Questions: support@truvec.dev.

Run your first comparison on your own documents, free.

The free plan takes 5 documents, a golden set of 20 questions and one comparison of up to 3 configurations: commercial models, hybrid search and rerankers. No card.