Prototype Your RAG Retrieval Strategy Before Writing Pipeline Code

Starting a new RAG project? Test embedding models, chunking, hybrid search, rerankers and top k on your own documents before building the pipeline. A practical workflow with Truvec.

  • embeddings
  • rag
  • chunking
  • retrieval

TL;DR: When you start a RAG project, you will probably pick an embedding model, a chunking strategy and maybe a reranker from a tutorial. But you should keep in mind that they're cheap to test before you build and expensive to change after. With Truvec you can upload a sample of your documents, try questions against any supported model in a Playground with no API setup, compare models side by side, check whether hybrid search or a reranker is worth adding, and test a different chunking setup, in minutes. This post walks through that workflow.

A lot of new RAG projects start by copying a LangChain or LlamaIndex example, use chunk_size=1000 and whichever embedding model the example used, and move on to the interesting parts: the prompt, the UI, the agent. Weeks later, answers are vague, and nobody knows whether the problem is the model, the chunks or the prompt.

Retrieval decisions are the foundation of answer quality. Once you index a few million chunks, changing the model or the chunking means re-embedding and re-indexing everything. This is why you should make the best decision before moving to production.

Retrieval decisions

  1. The embedding model. It decides what "similar" means. Models differ on domain vocabulary, languages, input length, price and vector size.
  2. The chunking strategy. How documents are split: by paragraphs, by tokens, by Markdown headers, by code functions. Plus chunk size and overlap.
  3. Hybrid search. Whether keyword search (BM25) runs next to vector search. It catches exact terms embeddings miss: product names, error codes, identifiers.
  4. A reranker. Whether a second model re-scores the best candidates before they reach the LLM. It often fixes rankings, and adds latency and a cost on every query.
  5. Top k. How many chunks you send to the LLM. It bounds both answer quality and context cost.

These retrieval configurations depend on your documents and your questions. But you also don't need a full pipeline to test them. In the next section, we will see how we can prototype a RAG pipeline to choose the retrieval configuration that best fits our data.

The prototyping workflow

Before you start, make sure to create a free account on Truvec. The workflow is straightforward and a first pass fits in an afternoon; most of the time goes into reviewing test questions, which is the part worth doing carefully.

1. Upload a representative sample

After creating a project, you should be able to upload your documents. The supported formats are PDF, Word (.docx), Markdown, text, CSV and JSON. But if you already have chunked documents, you can directly upload your chunks to Truvec.

Pick the chunking strategy. If you have no strong opinion, start with Recursive, 1,000 characters, 200 overlap: it's a robust default. Truvec offers different chunking strategies:

Strategy Splits on Good for
Recursive (default) Paragraphs, then lines, then words Most prose: docs, reports, policies
Character Blank lines only FAQs, glossaries
Token Token count Staying within a model's input limit
Markdown #, ##, ### headers (kept with each chunk) Markdown documentation and knowledge bases
Code Classes and functions Source code in 17 languages
JSON JSON structure (chunks stay valid JSON) Catalogs, structured records

Two practical warnings from experience: scanned PDFs have no text layer and produce nothing (Truvec doesn't run OCR), and tables in PDFs often come out jumbled. Open a few chunks in the Ingested Chunks table before going further.

2. Explore in the Playground: switch models without configuring anything

After uploading your documents, you should be able to browse the chunks, and you can start asking questions about your documents. The Playground runs a single question against your documents with the model you choose and returns the top k chunks (1 to 20) in rank order, with their chunk ID and similarity score.

What makes it useful for prototyping is that switching models is a dropdown. You don't create accounts with OpenAI, Cohere, Voyage AI, Google or Fireworks, you don't manage API keys, and you don't write indexing code. Truvec runs the models on its own accounts. Type a real user question, run it with text-embedding-3-small, switch to voyage-4, then to Cohere Embed v4, and see which chunks come back.

A few things to try:

  • Questions with your jargon: product names, error codes, acronyms.
  • Vague and informal questions: the way real users write.
  • Questions in another language, if your users are multilingual.

You should note that the first query with a model embeds all your chunks, so it takes longer than the next ones. And don't compare similarity scores across models: each model has its own scale, and 0.45 from one can be as strong a match as 0.80 from another. Compare which chunks come back and in what order.

The Playground isn't scored. It builds intuition and helps you find chunk IDs for your test questions. For a decision, you need the next step.

3. Build a golden dataset

You have already created your first project and added your documents, but you can only run Playground questions; you cannot start an evaluation of a single model or compare different models in a single run. To do that, you should generate a golden dataset, review the questions and mark it as committed.

Go to Golden Set, select the number of questions you want to generate and click generate dataset.

Before validating the dataset, you should review every question:

  • Check if the expected chunks really answer it.
  • Check if another chunk also answers it and add it, or a model that finds it will be counted wrong. The Playground helps you search.
  • Would a real user phrase it this way? Rewrite questions that copy the document's wording; they're too easy for every model.
  • Is it too vague to have a clear answer?

Add any real questions you have (from a pilot, a support inbox, a sales call), with their chunk IDs. Then commit the dataset to lock it.

Twenty to thirty questions is enough for a first look and to spot big differences between models. Use 50 or more when the scores get close.

4. Compare models side by side

After committing your golden dataset you should be able to start comparing models. Open Comparison, select the committed dataset, select the models that you want to compare, set top k to what you plan to send to the LLM, and run. Truvec supports multiple embedding models across OpenAI, Cohere, Voyage AI, Google and open-source models hosted on Fireworks AI (BGE, Nomic, mxbai, GTE, Qwen3 Embedding 8B).

You get a ranked table (by MRR, then recall) with hit rate, recall, precision, MRR, nDCG, latency, index cost, cost per 1,000 queries and vector storage; highlight cards for the cheapest and fastest models and the smallest index; and a Quality vs index cost chart where the best trade-offs sit in the top-left corner.

Then open the Retrieval Inspector, filter on Misses, and read the questions. This is the most valuable step of the whole exercise. Put each miss in a bucket: golden dataset error, extraction problem, chunking problem, ranking problem (the right chunk is found but too low), or model weakness. Fix the biggest bucket first.

5. Try hybrid search and a reranker

If the misses are ranking problems, or questions that name exact terms, test the two strategies most production pipelines add around the model. In Comparison, choose Compare strategies:

  1. Set the baseline to the model you liked, with plain dense search. A cheap model is a good candidate: the question is often whether a cheap model plus a reranker beats an expensive one.
  2. Click Hybrid search and Reranker to add one configuration each. Each copies the baseline and changes one thing. Add Another model to put the expensive model next to them.
  3. Run it on the same golden dataset and top k.

Each configuration is read against the baseline: the MRR difference, the questions it fixed and the ones it broke, and what it adds to latency and to the cost per 1,000 queries. That last number matters: Cohere Rerank 3.5 costs $2.00 per 1,000 searches, while embedding 1,000 questions with text-embedding-3-small costs a fraction of a cent. In the inspector, each chunk shows where it was before reranking, or its dense and keyword ranks with hybrid search, so you see how a question was rescued.

Hybrid search is free to add and tends to help on identifiers and product names. A reranker is worth it when its gain is clear at your traffic, or when it lets you keep a much cheaper embedding model.

6. Change the chunking and retest

If the misses point at chunking (answers split across two chunks, or buried in long multi-topic chunks), test another setup.

In Truvec, a chunking variant needs its own golden dataset, because a golden dataset points at specific chunk IDs and can't be reused across chunkings. The easiest way is a separate project with the same documents. The loop is short:

  1. Create a new project and upload the same documents with the new settings, for example Markdown by headers for a docs site, or Recursive at 500 characters instead of 1,000.
  2. Generate a golden dataset with similar questions, review it, commit it.
  3. Run the same comparison and compare the scores with the first project.

On a small sample, the upload and the comparison take minutes. Reviewing the new questions is the step that takes most of the time, and it's what makes the comparison trustworthy.

Change one thing at a time: to compare models or strategies, keep chunking fixed; to compare chunking, keep the configuration fixed.

7. Decide and write it down

Among the configurations that are best or tied on your primary metric (usually MRR, or recall if answers span several chunks), choose the cheapest and fastest one. Check that it handles your most important questions. Export the comparison as a PDF report so the team knows which setup was chosen and why.

Now build the pipeline, with a configuration you've measured instead of guessed.

What you end up with

At the end of the prototype you have:

  • An embedding model chosen on your data, with its cost per 1,000 queries and index cost.
  • A measured answer to "do we need hybrid search or a reranker?", with what each would cost.
  • A chunking strategy you've tested against at least one alternative.
  • A top k that matches what the LLM will see.
  • A locked golden dataset you can reuse as a regression test when a new model comes out.
  • A report to share with the team.

Common mistakes when prototyping retrieval

There are common mistakes that you should avoid while testing your retrieval pipeline:

  • Testing only easy questions. Questions that reuse the document's exact words make every model look good.
  • Adding documents after building the golden set. New chunks may answer existing questions without being listed as expected. Truvec flags this with a Corpus changed badge.
  • Mixing chunk settings for the same file type. It biases retrieval toward one group of documents. Truvec warns you before you do it.
  • Choosing on quality alone. A one-point gain rarely justifies a model, or a reranker, that costs five times more per query.

FAQ

Do I need API keys to try different embedding models in Truvec?

No. Truvec runs every supported model and reranker on its own accounts. You switch models in the Playground or select several in a comparison, with nothing to configure.

How many documents do I need for a prototype?

5 to 50 representative documents is enough for a first evaluation. Coverage of document types and topics matters more than volume.

Can I test hybrid search and rerankers without building them?

Yes. They're settings of an evaluation or a comparison: pick hybrid search, a reranker (Cohere Rerank 3.5, Voyage rerank-2.5 or 2.5-lite) and the number of first-stage candidates. The Playground runs plain vector search only, so use an evaluation to see what they retrieve.

Can I compare chunking strategies in the same project?

Not side by side. Golden datasets point at specific chunks, so each chunking needs its own dataset. Use one project per chunking setup with the same documents, or clear the documents and re-upload them with the new settings, then build a new golden dataset.

Is there a free way to try it?

Yes. The Free plan (no card needed) includes one project with up to 5 documents and 500 chunks, a golden set of up to 20 questions, 20 Playground queries and one comparison of up to 3 configurations (commercial models, with hybrid search and rerankers). Open-source models come with paid plans.

Start your prototype

Before you write the indexing code, spend an afternoon testing the decisions it will lock in. Create a free Truvec account and follow the 15-minute quickstart.

Related reading: