{
  "$schema": "https://truvec.dev/blog/schema/benchmark.schema.json",
  "schema_version": 1,
  "sample": false,
  "source": {
    "benchmark_id": 7,
    "exported_at": "2026-11-14T00:56:56.940Z",
    "app_url": "https://app.truvec.dev"
  },
  "name": "Six embedding models on a developer documentation corpus",
  "description": null,
  "run": {
    "top_k": 5,
    "created_at": "2026-11-14T00:55:37.297631Z",
    "completed_at": "2026-11-14T00:56:18.316690Z",
    "ranking": "MRR, then recall@k"
  },
  "corpus": {
    "name": "Docker documentation: manuals, FAQs, billing and support pages",
    "description": "261 pages of the public Docker documentation: Docker Desktop, Docker Hub, Compose, accounts, security, subscription and billing, FAQs, support and getting-started guides.",
    "url": "https://github.com/docker/docs",
    "documents": 261,
    "chunks": 1944,
    "chunking": {
      "strategy": "recursive",
      "separator": "\n\n",
      "chunk_size": 1000,
      "chunk_overlap": 200
    }
  },
  "golden_set": {
    "name": "Generated golden set, reviewed",
    "questions": 100,
    "reviewed": true,
    "generated_by": "gpt-5.6-luna"
  },
  "notes": [
    "The corpus is a subset of the public Docker documentation (Apache 2.0), taken from github.com/docker/docs at commit 553482c: the Desktop, Docker Hub, Compose, accounts, security and subscription and billing manuals, the FAQs, the support pages and the getting-started guides. Release notes and pages shorter than 600 characters were left out, and pages were cut at 40,000 characters.",
    "Several posts on this blog use this corpus and the same 100 questions, each with its own selection of models. A model can differ by a question between two of these runs: questions are embedded again at each run, and vector search is approximate.",
    "With 100 questions, a gap of a couple of points between two models is within the noise."
  ],
  "models": [
    {
      "rank": 1,
      "name": "Voyage 4",
      "provider": "voyage",
      "model_name": "voyage-4",
      "dimension": 1024,
      "price_per_million_tokens": 0.06,
      "status": "completed",
      "metrics": {
        "hit_rate": 1,
        "recall": 0.995,
        "precision": 0.21199999999999963,
        "f1_score": 0.3471428571428571,
        "mrr": 0.9153333333333333,
        "ndcg": 0.9285378503730927
      },
      "correct_answers": 100,
      "hit_rate_at": {
        "1": 0.85,
        "2": 0.95,
        "3": 0.99,
        "4": 0.99,
        "5": 1
      },
      "rank_histogram": {
        "1": 85,
        "2": 10,
        "3": 4,
        "4": 0,
        "5": 1,
        "miss": 0
      },
      "scores": {
        "mean_relevant_score": 0.6923111795000003,
        "mean_top_distractor_score": 0.6462398736999999,
        "score_gap": 0.04607130580000002
      },
      "latency": {
        "embedding_ms_per_query": 17.173992660827935,
        "search_ms_p50": 17.849950352683663,
        "search_ms_p95": 36.29739442840218
      },
      "cost": {
        "corpus_tokens": 336360,
        "index_cost_usd": 0.020181599999999997,
        "cost_per_1k_queries_usd": 0.001188
      },
      "vector_storage_bytes": 7962624
    },
    {
      "rank": 2,
      "name": "Gemini Embedding 001",
      "provider": "google",
      "model_name": "gemini-embedding-001",
      "dimension": 3072,
      "price_per_million_tokens": 0.15,
      "status": "completed",
      "metrics": {
        "hit_rate": 0.98,
        "recall": 0.975,
        "precision": 0.20799999999999966,
        "f1_score": 0.3404761904761904,
        "mrr": 0.8691666666666668,
        "ndcg": 0.8899037166748931
      },
      "correct_answers": 98,
      "hit_rate_at": {
        "1": 0.78,
        "2": 0.92,
        "3": 0.97,
        "4": 0.98,
        "5": 0.98
      },
      "rank_histogram": {
        "1": 78,
        "2": 14,
        "3": 5,
        "4": 1,
        "5": 0,
        "miss": 2
      },
      "scores": {
        "mean_relevant_score": 0.796238740612245,
        "mean_top_distractor_score": 0.7807054927000001,
        "score_gap": 0.015357184183673461
      },
      "latency": {
        "embedding_ms_per_query": 22.042847708798945,
        "search_ms_p50": 20.632721483707428,
        "search_ms_p95": 38.919316628016524
      },
      "cost": {
        "corpus_tokens": 336360,
        "index_cost_usd": 0.050454,
        "cost_per_1k_queries_usd": 0.00297
      },
      "vector_storage_bytes": 23887872
    },
    {
      "rank": 3,
      "name": "OpenAI text-embedding-3-large",
      "provider": "openai",
      "model_name": "text-embedding-3-large",
      "dimension": 3072,
      "price_per_million_tokens": 0.13,
      "status": "completed",
      "metrics": {
        "hit_rate": 0.99,
        "recall": 0.975,
        "precision": 0.20599999999999963,
        "f1_score": 0.3380952380952379,
        "mrr": 0.867,
        "ndcg": 0.8848806698984153
      },
      "correct_answers": 99,
      "hit_rate_at": {
        "1": 0.79,
        "2": 0.88,
        "3": 0.94,
        "4": 0.98,
        "5": 0.99
      },
      "rank_histogram": {
        "1": 79,
        "2": 9,
        "3": 6,
        "4": 4,
        "5": 1,
        "miss": 1
      },
      "scores": {
        "mean_relevant_score": 0.6865665528282829,
        "mean_top_distractor_score": 0.6498583157000002,
        "score_gap": 0.036667435959595956
      },
      "latency": {
        "embedding_ms_per_query": 14.772356622852385,
        "search_ms_p50": 22.497330559417605,
        "search_ms_p95": 50.03398491535335
      },
      "cost": {
        "corpus_tokens": 336360,
        "index_cost_usd": 0.0437268,
        "cost_per_1k_queries_usd": 0.002574
      },
      "vector_storage_bytes": 23887872
    },
    {
      "rank": 4,
      "name": "Cohere Embed v4",
      "provider": "cohere",
      "model_name": "embed-v4.0",
      "dimension": 1536,
      "price_per_million_tokens": 0.12,
      "status": "completed",
      "metrics": {
        "hit_rate": 0.98,
        "recall": 0.965,
        "precision": 0.20399999999999965,
        "f1_score": 0.33476190476190454,
        "mrr": 0.8586666666666667,
        "ndcg": 0.8752323097245924
      },
      "correct_answers": 98,
      "hit_rate_at": {
        "1": 0.76,
        "2": 0.93,
        "3": 0.95,
        "4": 0.97,
        "5": 0.98
      },
      "rank_histogram": {
        "1": 76,
        "2": 17,
        "3": 2,
        "4": 2,
        "5": 1,
        "miss": 2
      },
      "scores": {
        "mean_relevant_score": 0.6839180481632652,
        "mean_top_distractor_score": 0.6378762136999999,
        "score_gap": 0.04637254918367347
      },
      "latency": {
        "embedding_ms_per_query": 12.714331801980734,
        "search_ms_p50": 17.655924428254366,
        "search_ms_p95": 51.05555369518697
      },
      "cost": {
        "corpus_tokens": 336360,
        "index_cost_usd": 0.040363199999999995,
        "cost_per_1k_queries_usd": 0.002376
      },
      "vector_storage_bytes": 11943936
    },
    {
      "rank": 5,
      "name": "BGE Base EN v1.5",
      "provider": "fireworks",
      "model_name": "BAAI/bge-base-en-v1.5",
      "dimension": 768,
      "price_per_million_tokens": 0.008,
      "status": "completed",
      "metrics": {
        "hit_rate": 0.95,
        "recall": 0.925,
        "precision": 0.19399999999999967,
        "f1_score": 0.31904761904761864,
        "mrr": 0.7726666666666668,
        "ndcg": 0.802102666667776
      },
      "correct_answers": 95,
      "hit_rate_at": {
        "1": 0.64,
        "2": 0.85,
        "3": 0.9,
        "4": 0.92,
        "5": 0.95
      },
      "rank_histogram": {
        "1": 64,
        "2": 21,
        "3": 5,
        "4": 2,
        "5": 3,
        "miss": 5
      },
      "scores": {
        "mean_relevant_score": 0.8059181671578949,
        "mean_top_distractor_score": 0.7818177453999999,
        "score_gap": 0.023495881473684213
      },
      "latency": {
        "embedding_ms_per_query": 13.972576819360256,
        "search_ms_p50": 8.584670489653945,
        "search_ms_p95": 13.266099779866632
      },
      "cost": {
        "corpus_tokens": 336360,
        "index_cost_usd": 0.00269088,
        "cost_per_1k_queries_usd": 0.0001584
      },
      "vector_storage_bytes": 5971968
    },
    {
      "rank": 6,
      "name": "OpenAI text-embedding-3-small",
      "provider": "openai",
      "model_name": "text-embedding-3-small",
      "dimension": 1536,
      "price_per_million_tokens": 0.02,
      "status": "completed",
      "metrics": {
        "hit_rate": 0.95,
        "recall": 0.935,
        "precision": 0.19799999999999965,
        "f1_score": 0.3247619047619045,
        "mrr": 0.7686666666666667,
        "ndcg": 0.8067558320662286
      },
      "correct_answers": 95,
      "hit_rate_at": {
        "1": 0.63,
        "2": 0.84,
        "3": 0.92,
        "4": 0.94,
        "5": 0.95
      },
      "rank_histogram": {
        "1": 63,
        "2": 21,
        "3": 8,
        "4": 2,
        "5": 1,
        "miss": 5
      },
      "scores": {
        "mean_relevant_score": 0.7154984349473685,
        "mean_top_distractor_score": 0.687188013,
        "score_gap": 0.02744201336842105
      },
      "latency": {
        "embedding_ms_per_query": 4.5136926881968975,
        "search_ms_p50": 13.174299616366625,
        "search_ms_p95": 37.36471217125653
      },
      "cost": {
        "corpus_tokens": 336360,
        "index_cost_usd": 0.0067272,
        "cost_per_1k_queries_usd": 0.000396
      },
      "vector_storage_bytes": 11943936
    }
  ],
  "questions": [
    {
      "question": "How do I push an image to Docker Hub, and what tagging restriction applies?",
      "expected_excerpt": "## Push an image to Docker Hub Select the image from the list, select the **More options** button and select **Push to Hub**. > [!NOTE] > > You can only push an image to Docker Hub if the image belongs to your Docker ID or your organization. That is, the image must contain the co",
      "first_relevant_rank": {
        "Voyage 4": 5,
        "Gemini Embedding 001": null,
        "OpenAI text-embedding-3-large": null,
        "Cohere Embed v4": null,
        "BGE Base EN v1.5": 3,
        "OpenAI text-embedding-3-small": 1
      }
    },
    {
      "question": "When both JIT and SCIM are enabled, how are organization roles and account deactivation handled?",
      "expected_excerpt": "Roles don't move back and forth. JIT sets the organization role only when it first adds the user. A later SCIM update can change that role, and the next sign-in leaves the SCIM role in place. When a user isn't assigned to the Docker application in the IdP, the next synchronizatio",
      "first_relevant_rank": {
        "Voyage 4": 1,
        "Gemini Embedding 001": 1,
        "OpenAI text-embedding-3-large": 5,
        "Cohere Embed v4": 3,
        "BGE Base EN v1.5": 3,
        "OpenAI text-embedding-3-small": null
      }
    },
    {
      "question": "How do I search Docker Desktop logs, move between matching entries, and clear the terminal?",
      "expected_excerpt": "### Logs Select **Logs** to view output from the container in real time. While viewing logs, you can: - Use `Cmd + f`/`Ctrl + f` to open the search bar and find specific entries. Search matches are highlighted in yellow. - Press `Enter` or `Shift + Enter` to jump to the next or p",
      "first_relevant_rank": {
        "Voyage 4": 1,
        "Gemini Embedding 001": 3,
        "OpenAI text-embedding-3-large": 4,
        "Cohere Embed v4": 2,
        "BGE Base EN v1.5": 1,
        "OpenAI text-embedding-3-small": 1
      }
    },
    {
      "question": "What file and service configuration are required for Docker Hub automated tests, and how is the test result determined?",
      "expected_excerpt": "## Set up automated test files To set up your automated tests, create a `docker-compose.test.yml` file which defines a `sut` service that lists the tests to be run. The `docker-compose.test.yml` file should be located in the same directory that contains the Dockerfile used to bui",
      "first_relevant_rank": {
        "Voyage 4": 1,
        "Gemini Embedding 001": 1,
        "OpenAI text-embedding-3-large": 2,
        "Cohere Embed v4": null,
        "BGE Base EN v1.5": 5,
        "OpenAI text-embedding-3-small": null
      }
    },
    {
      "question": "What system requirements and preliminary packages are needed to install Docker Desktop on Ubuntu?",
      "expected_excerpt": "# Install Docker Desktop on Ubuntu > **Docker Desktop terms** > > Commercial use of Docker Desktop in larger enterprises (more than 250 > employees or more than $10 million USD in annual revenue) requires a paid > subscription. This page contains information on how to install, la",
      "first_relevant_rank": {
        "Voyage 4": 1,
        "Gemini Embedding 001": 1,
        "OpenAI text-embedding-3-large": 1,
        "Cohere Embed v4": 2,
        "BGE Base EN v1.5": 4,
        "OpenAI text-embedding-3-small": 3
      }
    },
    {
      "question": "When multiple Compose files are supplied with `-f`, how are their service settings combined, and which file takes precedence?",
      "expected_excerpt": "Compose merges files in the order they're specified on the command line. Subsequent files may merge, override, or add to their predecessors. For example: ```console $ docker compose -f compose.yaml -f compose.admin.yaml run backup_db ``` The `compose.yaml` file might specify a `w",
      "first_relevant_rank": {
        "Voyage 4": 2,
        "Gemini Embedding 001": 1,
        "OpenAI text-embedding-3-large": 1,
        "Cohere Embed v4": 4,
        "BGE Base EN v1.5": null,
        "OpenAI text-embedding-3-small": 4
      }
    },
    {
      "question": "How do image indexes enable one container image reference to work across AMD and ARM platforms?",
      "expected_excerpt": "# Image Management Images and image indexes are the foundation of container images within a repository. The following diagram shows the relationship between images and image indexes. This structure enables multi-architecture support through a single reference. It is important to ",
      "first_relevant_rank": {
        "Voyage 4": 1,
        "Gemini Embedding 001": 2,
        "OpenAI text-embedding-3-large": 4,
        "Cohere Embed v4": 1,
        "BGE Base EN v1.5": 1,
        "OpenAI text-embedding-3-small": 3
      }
    },
    {
      "question": "What command builds the Docker image for the Spring Boot hello world app, and how can I check the size of the resulting image?",
      "expected_excerpt": "```dockerfile CMD [\"./mvnw\", \"spring-boot:run\"] ``` And with that, you should have the following Dockerfile: ```dockerfile FROM eclipse-temurin:21.0.8_9-jdk-jammy WORKDIR /app COPY .mvn/ .mvn COPY mvnw pom.xml ./ RUN ./mvnw dependency:go-offline COPY src ./src CMD [\"./mvnw\", \"spr",
      "first_relevant_rank": {
        "Voyage 4": 3,
        "Gemini Embedding 001": 1,
        "OpenAI text-embedding-3-large": 1,
        "Cohere Embed v4": 2,
        "BGE Base EN v1.5": 1,
        "OpenAI text-embedding-3-small": 5
      }
    }
  ]
}
