The best embedding model for technical documentation
API references, guides and changelogs are full of identifiers, error codes and version numbers that embedding models handle badly. How to chunk developer docs, which models to test, and a benchmark on open-source documentation.
Documentation assistants are one of the most common RAG applications: "ask the docs" boxes, support bots, IDE helpers. They're also a tough test for embedding models. Developer documentation mixes prose with code, and the questions people ask are full of exact tokens: a function name, an error code, a version number.
This post explains what makes technical docs different, how to prepare them, and what to test. The last section benchmarks the main models on open-source developer documentation.
What makes technical docs different
Identifiers carry the meaning. starting_after, E_CONN_RESET, --max-retries: to a developer these are precise; to an embedding model they're rare tokens split into fragments. Two endpoints with similar names can end up closer to each other than to the question about one of them.
Versions. "How do I configure retries?" has a different answer in v3 and v4. Changelogs and migration guides mention old and new behavior side by side, which creates convincing wrong answers.
Code blocks. A chunk that's mostly code may hold the answer, but its embedding mostly reflects syntax. Models trained on code and text together handle this better.
Short, vague questions. Developers type "pagination", "timeout per request" or paste an error message. Short queries give the model little to work with.
Repetition. API references repeat the same structure for every endpoint (parameters, returns, errors), so many chunks look alike.
Prepare the docs
- Chunk by heading. Split Markdown on its headings and keep the heading path in the chunk ("Client > Configuration > Timeouts"). Headings are often the best match for a question.
- Keep code with its explanation. Don't separate a code example from the paragraph that explains it; the paragraph gives the embedding its meaning.
- Tag versions. Store the product version in each chunk's metadata and filter on it, or index only the current version.
- Add keyword search for exact tokens. Error codes and function names are matched best by keyword search. Hybrid retrieval fixes the cases embeddings miss.
Build questions that look like real ones
Generated questions tend to be clean and complete. Real developer questions aren't. Mix in:
- pasted error messages ("ECONNRESET when uploading large file");
- bare identifiers ("starting_after");
- "how do I" questions phrased without the API's vocabulary;
- questions about behavior that changed between versions.
If you have search logs from your docs site, they're the best source of real questions.
What to test
Start with the large models of each provider and a couple of cheap ones. For developer docs, also test:
- Long-input models if your pages are long, so code-heavy chunks aren't truncated;
- Open-weight models if the assistant ships inside your product or runs on-premises;
- A cheap model as a baseline: on well-structured docs, a cheap model sometimes ties with an expensive one.
The benchmark below runs nine models on open-source developer documentation. Look at hit rate at position 1: docs assistants usually show the top result first, so a model that ranks the answer second loses more than its MRR suggests.
Real benchmark - generated by Truvec
Benchmark: Embedding models on developer documentation
- Documents
- 261
- Chunks
- 1,944
- Chunking
- recursive, 1000 chars, 200 overlap
- Golden set
- 100 questions, human-reviewed
- Models
- 9 compared
- Top k
- 5
- Run
- 2026-11-14
Download the full report (PDF) Every question, with the passages each model retrieved.
Key results
- Voyage 4 Large ranks first, with an MRR of 0.922 and a hit rate of 99.0% in the top 5.
- One model is within 0.02 MRR of the winner (Voyage 4): on this corpus they are practically tied.
- Hit rate ranges from 90.0% to 100.0% across the 9 models.
- Cheapest model within 2 points of the winner's hit rate: Voyage 4, $0.02 to embed this corpus against $0.04.
- Fastest to embed a query: OpenAI text-embedding-3-small (4.4 ms per query, measured in batch from our servers).
Retrieval quality
| Model | Hit rate@5 | Recall@5 | MRR | nDCG@5 | Precision@5 |
|---|---|---|---|---|---|
| 1Voyage 4 Large | 99.0% | 97.5% | 0.924 | 20.6% | |
| 2Voyage 4 | 100.0% | 99.5% | 0.929 | 21.2% | |
| 3Gemini Embedding 2 | 99.0% | 98.5% | 0.897 | 21.0% | |
| 4OpenAI text-embedding-3-large | 99.0% | 97.5% | 0.885 | 20.6% | |
| 5Qwen3 Embedding 8B | 99.0% | 98.0% | 0.884 | 20.8% | |
| 6Cohere Embed v4 | 98.0% | 96.5% | 0.875 | 20.4% | |
| 7BGE Base EN v1.5 | 95.0% | 92.5% | 0.802 | 19.4% | |
| 8OpenAI text-embedding-3-small | 95.0% | 93.5% | 0.807 | 19.8% | |
| 9Nomic Embed Text v1.5 | 90.0% | 89.0% | 0.789 | 18.8% |
Hit rate by position
How many questions the first results already answer. A model that finds the answer at position 1 lets you send fewer chunks to the LLM.
| Model | Top 1 | Top 2 | Top 3 | Top 4 | Top 5 |
|---|---|---|---|---|---|
| 1Voyage 4 Large | 87.0% | 94.0% | 99.0% | 99.0% | 99.0% |
| 2Voyage 4 | 85.0% | 95.0% | 99.0% | 99.0% | 100.0% |
| 3Gemini Embedding 2 | 80.0% | 90.0% | 95.0% | 98.0% | 99.0% |
| 4OpenAI text-embedding-3-large | 79.0% | 88.0% | 94.0% | 98.0% | 99.0% |
| 5Qwen3 Embedding 8B | 77.0% | 89.0% | 95.0% | 99.0% | 99.0% |
| 6Cohere Embed v4 | 76.0% | 93.0% | 95.0% | 97.0% | 98.0% |
| 7BGE Base EN v1.5 | 64.0% | 85.0% | 90.0% | 92.0% | 95.0% |
| 8OpenAI text-embedding-3-small | 63.0% | 84.0% | 92.0% | 94.0% | 95.0% |
| 9Nomic Embed Text v1.5 | 67.0% | 80.0% | 86.0% | 89.0% | 90.0% |
Cost, speed and size
Embedding prices are the providers' list prices. Corpus cost is a one-time cost per model; the vector size is raw float32 storage, before the vector database's own overhead.
| Model | Price / 1M tokens | Embed the corpus | 1,000 queries | Query embedding | Search p95 | Vectors |
|---|---|---|---|---|---|---|
| 1Voyage 4 Large | $0.12 | $0.04 | $0.0024 | 16 ms | 18 ms | 7.6 MB |
| 2Voyage 4 | $0.06 | $0.02 | $0.0012 | 16 ms | 22 ms | 7.6 MB |
| 3Gemini Embedding 2 | $0.20 | $0.07 | $0.0040 | 18 ms | 22 ms | 22.8 MB |
| 4OpenAI text-embedding-3-large | $0.13 | $0.04 | $0.0026 | 9.0 ms | 61 ms | 22.8 MB |
| 5Qwen3 Embedding 8B | $0.10 | $0.03 | $0.0020 | 17 ms | 81 ms | 30.4 MB |
| 6Cohere Embed v4 | $0.12 | $0.04 | $0.0024 | 17 ms | 38 ms | 11.4 MB |
| 7BGE Base EN v1.5 | $0.0080 | $0.0027 | $0.0002 | 14 ms | 13 ms | 5.7 MB |
| 8OpenAI text-embedding-3-small | $0.02 | $0.0067 | $0.0004 | 4.4 ms | 19 ms | 11.4 MB |
| 9Nomic Embed Text v1.5 | $0.0080 | $0.0027 | $0.0002 | 15 ms | 16 ms | 5.7 MB |
Score separation
Average similarity of the best relevant chunk and of the best irrelevant one. A wide gap makes a similarity threshold reliable; a narrow one means right and wrong chunks look alike to the model.
| Model | Relevant chunk | Best distractor | Gap |
|---|---|---|---|
| 1Voyage 4 Large | 0.653 | 0.607 | 0.046 |
| 2Voyage 4 | 0.692 | 0.646 | 0.046 |
| 3Gemini Embedding 2 | 0.816 | 0.792 | 0.025 |
| 4OpenAI text-embedding-3-large | 0.687 | 0.650 | 0.037 |
| 5Qwen3 Embedding 8B | 0.827 | 0.785 | 0.042 |
| 6Cohere Embed v4 | 0.684 | 0.638 | 0.046 |
| 7BGE Base EN v1.5 | 0.806 | 0.782 | 0.023 |
| 8OpenAI text-embedding-3-small | 0.715 | 0.687 | 0.027 |
| 9Nomic Embed Text v1.5 | 0.842 | 0.815 | 0.027 |
Example questions
Rank of the first relevant chunk for each model; miss means it wasn't in the top 5. These are the questions where the models disagree most.
| Question | Voyage 4 Large | Voyage 4 | Gemini Embedding 2 | OpenAI text-embedding-3-large | Qwen3 Embedding 8B | Cohere Embed v4 | BGE Base EN v1.5 | OpenAI text-embedding-3-small | Nomic Embed Text v1.5 |
|---|---|---|---|---|---|---|---|---|---|
How do I push an image to Docker Hub, and what tagging restriction applies?Expected passage## Push an image to Docker Hub Select the image from the list, select the **More options** button and select **Push to Hub**. > [!NOTE] > > You can only push an image to Docker Hub if the image belongs to your Docker ID or your organization. That is, the image must contain the co | miss | 5 | 5 | miss | 4 | miss | 3 | 1 | 4 |
How can I use a value from a `.env` file inside a Docker Compose service's environment settings?Expected passageBasic example: ```console $ cat .env ## define COMPOSE_DEBUG based on DEV_MODE, defaults to false COMPOSE_DEBUG=${DEV_MODE:-false} $ cat compose.yaml services: webapp: image: my-webapp-image environment: - DEBUG=${COMPOSE_DEBUG} $ DEV_MODE=true docker compose config services: web | 1 | 1 | 3 | 1 | 1 | 2 | 5 | 2 | miss |
What file and service configuration are required for Docker Hub automated tests, and how is the test result determined?Expected passage## Set up automated test files To set up your automated tests, create a `docker-compose.test.yml` file which defines a `sut` service that lists the tests to be run. The `docker-compose.test.yml` file should be located in the same directory that contains the Dockerfile used to bui | 1 | 1 | 4 | 2 | 4 | miss | 5 | miss | 2 |
Which organizational roles and subscription plans are needed to create organization access tokens in Docker?Expected passage> [!WARNING] > > OATs don't work with Docker Desktop or Image Access Management. For > those features, use > PATs instead. ## Best practices - Set an expiration date and rotate tokens on a schedule - Grant only the repositories and scopes each job needs - Review the **Last used** | 1 | 1 | 1 | 4 | 2 | 1 | miss | miss | miss |
When both JIT and SCIM are enabled, how are organization roles and account deactivation handled?Expected passageRoles don't move back and forth. JIT sets the organization role only when it first adds the user. A later SCIM update can change that role, and the next sign-in leaves the SCIM role in place. When a user isn't assigned to the Docker application in the IdP, the next synchronizatio | 1 | 1 | 1 | 5 | miss | 3 | 3 | miss | 3 |
Which Docker Desktop versions are supported under the Pro, Team, and Business plans?Expected passage### Supported versions Docker Desktop support is scoped by plan: | Plan | Supported versions | Machines per subscription | | :--- | :--- | :--- | | Docker Pro | Latest only | One | | Docker Team | Latest only | One per subscription seat | | Docker Business | Latest plus versions | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 3 | 4 |
How do I create, publish, and share a custom MCP catalog with my team?Expected passageCreate a catalog and specify which servers to include: ```console $ docker mcp catalog create registry.example.com/mcp/data-tools:latest \ --title "Data Analysis Tools" \ --server catalog://mcp/docker-mcp-catalog/sequentialthinking \ --server catalog://mcp/docker-mcp-catalog/brav | 1 | 1 | 1 | 1 | 3 | 5 | 2 | 2 | 3 |
Where can I inspect all Docker Desktop settings that have been applied, including settings not visible in the GUI?Expected passageAs a workaround, you can check the `settings-store.json` file to view all applied settings: - Mac: `~/Library/Application Support/Docker/settings-store.json` - Windows: `%APPDATA%\Docker\settings-store.json` - Linux: `~/.docker/desktop/settings-store.json` The `settings-store.jso | 1 | 1 | 1 | 1 | 3 | 1 | 4 | 2 | 1 |
Notes
- The corpus is a subset of the public Docker documentation (Apache 2.0), taken from github.com/docker/docs at commit 553482c: the Desktop, Docker Hub, Compose, accounts, security and subscription and billing manuals, the FAQs, the support pages and the getting-started guides. Release notes and pages shorter than 600 characters were left out, and pages were cut at 40,000 characters.
- Several posts on this blog use this corpus and the same 100 questions, each with its own selection of models. A model can differ by a question between two of these runs: questions are embedded again at each run, and vector search is approximate.
- With 100 questions, a gap of a couple of points between two models is within the noise.
Methodology
Truvec split the documents into chunks, generated a golden set of questions with gpt-5.6-luna (each question is linked to the chunks that answer it), embedded the corpus and the questions with every model, and retrieved the top 5 chunks for each question by cosine similarity. Every model saw the same chunks and the same questions. Scores are averaged over all 100 questions.
- Hit rate
- Share of questions with at least one relevant chunk in the top k results. The most direct answer to "does the right passage reach the LLM?".
- Recall
- Share of the relevant chunks found in the top k, averaged over questions. Matters when an answer spans several chunks.
- MRR
- Mean reciprocal rank: 1 when the first relevant chunk is ranked first, 1/2 when second, 0 when missing. Rewards putting the answer at the top.
- nDCG
- Normalized discounted cumulative gain: how close the ranking of all relevant chunks is to the ideal ranking.
- Precision
- Share of the top k results that are relevant. Low precision means more noise sent to the LLM.
Downloads: full report (PDF) - benchmark.json - benchmark.csv - this post as Markdown
Voyage 4 Large won here, on these documents. Find the model that wins on yours: run the same benchmark on your documents with Truvec.