The best embedding model for technical documentation

API references, guides and changelogs are full of identifiers, error codes and version numbers that embedding models handle badly. How to chunk developer docs, which models to test, and a benchmark on open-source documentation.

  • embeddings
  • rag
  • documentation
  • developer-tools
  • benchmark

Download the benchmark report (PDF)

Documentation assistants are one of the most common RAG applications: "ask the docs" boxes, support bots, IDE helpers. They're also a tough test for embedding models. Developer documentation mixes prose with code, and the questions people ask are full of exact tokens: a function name, an error code, a version number.

This post explains what makes technical docs different, how to prepare them, and what to test. The last section benchmarks the main models on open-source developer documentation.

What makes technical docs different

Identifiers carry the meaning. starting_after, E_CONN_RESET, --max-retries: to a developer these are precise; to an embedding model they're rare tokens split into fragments. Two endpoints with similar names can end up closer to each other than to the question about one of them.

Versions. "How do I configure retries?" has a different answer in v3 and v4. Changelogs and migration guides mention old and new behavior side by side, which creates convincing wrong answers.

Code blocks. A chunk that's mostly code may hold the answer, but its embedding mostly reflects syntax. Models trained on code and text together handle this better.

Short, vague questions. Developers type "pagination", "timeout per request" or paste an error message. Short queries give the model little to work with.

Repetition. API references repeat the same structure for every endpoint (parameters, returns, errors), so many chunks look alike.

Prepare the docs

  • Chunk by heading. Split Markdown on its headings and keep the heading path in the chunk ("Client > Configuration > Timeouts"). Headings are often the best match for a question.
  • Keep code with its explanation. Don't separate a code example from the paragraph that explains it; the paragraph gives the embedding its meaning.
  • Tag versions. Store the product version in each chunk's metadata and filter on it, or index only the current version.
  • Add keyword search for exact tokens. Error codes and function names are matched best by keyword search. Hybrid retrieval fixes the cases embeddings miss.

Build questions that look like real ones

Generated questions tend to be clean and complete. Real developer questions aren't. Mix in:

  • pasted error messages ("ECONNRESET when uploading large file");
  • bare identifiers ("starting_after");
  • "how do I" questions phrased without the API's vocabulary;
  • questions about behavior that changed between versions.

If you have search logs from your docs site, they're the best source of real questions.

What to test

Start with the large models of each provider and a couple of cheap ones. For developer docs, also test:

  • Long-input models if your pages are long, so code-heavy chunks aren't truncated;
  • Open-weight models if the assistant ships inside your product or runs on-premises;
  • A cheap model as a baseline: on well-structured docs, a cheap model sometimes ties with an expensive one.

The benchmark below runs nine models on open-source developer documentation. Look at hit rate at position 1: docs assistants usually show the top result first, so a model that ranks the answer second loses more than its MRR suggests.

Real benchmark - generated by Truvec

Benchmark: Embedding models on developer documentation

Corpus
Docker documentation: manuals, FAQs, billing and support pages
Documents
261
Chunks
1,944
Chunking
recursive, 1000 chars, 200 overlap
Golden set
100 questions, human-reviewed
Models
9 compared
Top k
5
Run
2026-11-14

Download the full report (PDF) Every question, with the passages each model retrieved.

Key results

  • Voyage 4 Large ranks first, with an MRR of 0.922 and a hit rate of 99.0% in the top 5.
  • One model is within 0.02 MRR of the winner (Voyage 4): on this corpus they are practically tied.
  • Hit rate ranges from 90.0% to 100.0% across the 9 models.
  • Cheapest model within 2 points of the winner's hit rate: Voyage 4, $0.02 to embed this corpus against $0.04.
  • Fastest to embed a query: OpenAI text-embedding-3-small (4.4 ms per query, measured in batch from our servers).

Retrieval quality

Retrieval quality at top 5, ranked by MRR, then recall@k
ModelHit rate@5Recall@5MRRnDCG@5Precision@5
1Voyage 4 Largevoyage - 1024 dims99.0%97.5%0.9220.92420.6%
2Voyage 4voyage - 1024 dims100.0%99.5%0.9150.92921.2%
3Gemini Embedding 2google - 3072 dims99.0%98.5%0.8760.89721.0%
4OpenAI text-embedding-3-largeopenai - 3072 dims99.0%97.5%0.8670.88520.6%
5Qwen3 Embedding 8Bfireworks - 4096 dims99.0%98.0%0.8600.88420.8%
6Cohere Embed v4cohere - 1536 dims98.0%96.5%0.8590.87520.4%
7BGE Base EN v1.5fireworks - 768 dims95.0%92.5%0.7730.80219.4%
8OpenAI text-embedding-3-smallopenai - 1536 dims95.0%93.5%0.7690.80719.8%
9Nomic Embed Text v1.5fireworks - 768 dims90.0%89.0%0.7650.78918.8%

Hit rate by position

How many questions the first results already answer. A model that finds the answer at position 1 lets you send fewer chunks to the LLM.

Hit rate by position: share of questions answered by the first j results
ModelTop 1Top 2Top 3Top 4Top 5
1Voyage 4 Largevoyage - 1024 dims87.0%94.0%99.0%99.0%99.0%
2Voyage 4voyage - 1024 dims85.0%95.0%99.0%99.0%100.0%
3Gemini Embedding 2google - 3072 dims80.0%90.0%95.0%98.0%99.0%
4OpenAI text-embedding-3-largeopenai - 3072 dims79.0%88.0%94.0%98.0%99.0%
5Qwen3 Embedding 8Bfireworks - 4096 dims77.0%89.0%95.0%99.0%99.0%
6Cohere Embed v4cohere - 1536 dims76.0%93.0%95.0%97.0%98.0%
7BGE Base EN v1.5fireworks - 768 dims64.0%85.0%90.0%92.0%95.0%
8OpenAI text-embedding-3-smallopenai - 1536 dims63.0%84.0%92.0%94.0%95.0%
9Nomic Embed Text v1.5fireworks - 768 dims67.0%80.0%86.0%89.0%90.0%

Cost, speed and size

Embedding prices are the providers' list prices. Corpus cost is a one-time cost per model; the vector size is raw float32 storage, before the vector database's own overhead.

Cost, speed and size on this corpus
ModelPrice / 1M tokensEmbed the corpus1,000 queriesQuery embeddingSearch p95Vectors
1Voyage 4 Largevoyage - 1024 dims$0.12$0.04$0.002416 ms18 ms7.6 MB
2Voyage 4voyage - 1024 dims$0.06$0.02$0.001216 ms22 ms7.6 MB
3Gemini Embedding 2google - 3072 dims$0.20$0.07$0.004018 ms22 ms22.8 MB
4OpenAI text-embedding-3-largeopenai - 3072 dims$0.13$0.04$0.00269.0 ms61 ms22.8 MB
5Qwen3 Embedding 8Bfireworks - 4096 dims$0.10$0.03$0.002017 ms81 ms30.4 MB
6Cohere Embed v4cohere - 1536 dims$0.12$0.04$0.002417 ms38 ms11.4 MB
7BGE Base EN v1.5fireworks - 768 dims$0.0080$0.0027$0.000214 ms13 ms5.7 MB
8OpenAI text-embedding-3-smallopenai - 1536 dims$0.02$0.0067$0.00044.4 ms19 ms11.4 MB
9Nomic Embed Text v1.5fireworks - 768 dims$0.0080$0.0027$0.000215 ms16 ms5.7 MB

Score separation

Average similarity of the best relevant chunk and of the best irrelevant one. A wide gap makes a similarity threshold reliable; a narrow one means right and wrong chunks look alike to the model.

Score separation: how clearly each model scores the right chunk above the best wrong one
ModelRelevant chunkBest distractorGap
1Voyage 4 Largevoyage - 1024 dims0.6530.6070.046
2Voyage 4voyage - 1024 dims0.6920.6460.046
3Gemini Embedding 2google - 3072 dims0.8160.7920.025
4OpenAI text-embedding-3-largeopenai - 3072 dims0.6870.6500.037
5Qwen3 Embedding 8Bfireworks - 4096 dims0.8270.7850.042
6Cohere Embed v4cohere - 1536 dims0.6840.6380.046
7BGE Base EN v1.5fireworks - 768 dims0.8060.7820.023
8OpenAI text-embedding-3-smallopenai - 1536 dims0.7150.6870.027
9Nomic Embed Text v1.5fireworks - 768 dims0.8420.8150.027

Example questions

Rank of the first relevant chunk for each model; miss means it wasn't in the top 5. These are the questions where the models disagree most.

Example questions from the golden set
QuestionVoyage 4 LargeVoyage 4Gemini Embedding 2OpenAI text-embedding-3-largeQwen3 Embedding 8BCohere Embed v4BGE Base EN v1.5OpenAI text-embedding-3-smallNomic Embed Text v1.5
How do I push an image to Docker Hub, and what tagging restriction applies?
Expected passage
## Push an image to Docker Hub Select the image from the list, select the **More options** button and select **Push to Hub**. > [!NOTE] > > You can only push an image to Docker Hub if the image belongs to your Docker ID or your organization. That is, the image must contain the co
miss55miss4miss314
How can I use a value from a `.env` file inside a Docker Compose service's environment settings?
Expected passage
Basic example: ```console $ cat .env ## define COMPOSE_DEBUG based on DEV_MODE, defaults to false COMPOSE_DEBUG=${DEV_MODE:-false} $ cat compose.yaml services: webapp: image: my-webapp-image environment: - DEBUG=${COMPOSE_DEBUG} $ DEV_MODE=true docker compose config services: web
11311252miss
What file and service configuration are required for Docker Hub automated tests, and how is the test result determined?
Expected passage
## Set up automated test files To set up your automated tests, create a `docker-compose.test.yml` file which defines a `sut` service that lists the tests to be run. The `docker-compose.test.yml` file should be located in the same directory that contains the Dockerfile used to bui
11424miss5miss2
Which organizational roles and subscription plans are needed to create organization access tokens in Docker?
Expected passage
> [!WARNING] > > OATs don't work with Docker Desktop or Image Access Management. For > those features, use > PATs instead. ## Best practices - Set an expiration date and rotate tokens on a schedule - Grant only the repositories and scopes each job needs - Review the **Last used**
111421missmissmiss
When both JIT and SCIM are enabled, how are organization roles and account deactivation handled?
Expected passage
Roles don't move back and forth. JIT sets the organization role only when it first adds the user. A later SCIM update can change that role, and the next sign-in leaves the SCIM role in place. When a user isn't assigned to the Docker application in the IdP, the next synchronizatio
1115miss33miss3
Which Docker Desktop versions are supported under the Pro, Team, and Business plans?
Expected passage
### Supported versions Docker Desktop support is scoped by plan: | Plan | Supported versions | Machines per subscription | | :--- | :--- | :--- | | Docker Pro | Latest only | One | | Docker Team | Latest only | One per subscription seat | | Docker Business | Latest plus versions
111211134
How do I create, publish, and share a custom MCP catalog with my team?
Expected passage
Create a catalog and specify which servers to include: ```console $ docker mcp catalog create registry.example.com/mcp/data-tools:latest \ --title "Data Analysis Tools" \ --server catalog://mcp/docker-mcp-catalog/sequentialthinking \ --server catalog://mcp/docker-mcp-catalog/brav
111135223
Where can I inspect all Docker Desktop settings that have been applied, including settings not visible in the GUI?
Expected passage
As a workaround, you can check the `settings-store.json` file to view all applied settings: - Mac: `~/Library/Application Support/Docker/settings-store.json` - Windows: `%APPDATA%\Docker\settings-store.json` - Linux: `~/.docker/desktop/settings-store.json` The `settings-store.jso
111131421

Notes

  • The corpus is a subset of the public Docker documentation (Apache 2.0), taken from github.com/docker/docs at commit 553482c: the Desktop, Docker Hub, Compose, accounts, security and subscription and billing manuals, the FAQs, the support pages and the getting-started guides. Release notes and pages shorter than 600 characters were left out, and pages were cut at 40,000 characters.
  • Several posts on this blog use this corpus and the same 100 questions, each with its own selection of models. A model can differ by a question between two of these runs: questions are embedded again at each run, and vector search is approximate.
  • With 100 questions, a gap of a couple of points between two models is within the noise.

Methodology

Truvec split the documents into chunks, generated a golden set of questions with gpt-5.6-luna (each question is linked to the chunks that answer it), embedded the corpus and the questions with every model, and retrieved the top 5 chunks for each question by cosine similarity. Every model saw the same chunks and the same questions. Scores are averaged over all 100 questions.

Hit rate
Share of questions with at least one relevant chunk in the top k results. The most direct answer to "does the right passage reach the LLM?".
Recall
Share of the relevant chunks found in the top k, averaged over questions. Matters when an answer spans several chunks.
MRR
Mean reciprocal rank: 1 when the first relevant chunk is ranked first, 1/2 when second, 0 when missing. Rewards putting the answer at the top.
nDCG
Normalized discounted cumulative gain: how close the ranking of all relevant chunks is to the ideal ranking.
Precision
Share of the top k results that are relevant. Low precision means more noise sent to the LLM.

Downloads: full report (PDF) - benchmark.json - benchmark.csv - this post as Markdown

Voyage 4 Large won here, on these documents. Find the model that wins on yours: run the same benchmark on your documents with Truvec.