BGE vs OpenAI embeddings: is a small open model good enough?

BGE Base is a small open-source embedding model you can run almost anywhere. How it compares with OpenAI's text-embedding-3 models on English documents, where its 512-token limit and English-only training matter, and a benchmark.

  • embeddings
  • rag
  • bge
  • openai
  • open-source
  • model-comparison

Download the benchmark report (PDF)

BGE (BAAI General Embedding) models are among the most downloaded open-source embedding models. BGE Base EN v1.5 is small enough to run on a CPU, produces compact 768-dimension vectors, and costs almost nothing through a hosted API. OpenAI's text-embedding-3 models are the default choice for many teams. The question is simple: does the small open model give up enough quality to justify paying for OpenAI?

This post compares BGE Base EN v1.5 with OpenAI text-embedding-3-small and 3-large, then benchmarks them on English documentation.

At a glance

BGE Base EN v1.5 OpenAI 3-small OpenAI 3-large
Weights Open source API only API only
Price per 1M tokens (hosted, in Truvec) $0.008 $0.02 $0.13
Dimensions 768 1,536 3,072
Max input 512 tokens 8,191 tokens 8,191 tokens
Languages English Multilingual Multilingual

What to know about BGE

512 tokens is the hard limit. Anything past it is ignored. With 1,000-character chunks (about 250 tokens) that's fine. With long chunks, BGE is evaluated on truncated text, and its scores drop for reasons that have nothing to do with the model's understanding. If you use BGE, keep chunks short.

It's English-only. On French, Arabic or mixed-language content, it falls far behind multilingual models. Don't use it outside English.

It needs a query prefix. BGE was trained with an instruction prepended to queries ("Represent this sentence for searching relevant passages: "). Without it, retrieval gets worse. Many integrations forget it.

It's tiny to run. A base-size model embeds thousands of chunks per minute on modest hardware, and the vectors take a quarter of the space of OpenAI 3-large's. For edge deployments, on-premises installs or tight budgets, that matters more than a few points of quality.

When BGE is good enough

  • English content, short chunks, and a budget or hosting constraint;
  • a first version of a RAG system, where you need something that works today and can switch later;
  • a large corpus that gets re-embedded often, where per-token cost adds up.

When to pay for OpenAI (or another API)

  • Non-English or mixed-language content;
  • long chunks you don't want to split further;
  • a measurable quality gap on your documents that changes your users' answers.

The benchmark below compares the three models on English developer documentation, with chunks short enough for BGE to read in full. Look at the size of the quality gap, then at the cost column: whether the gap is worth the price is your call, but now it's an informed one. For what each metric in the tables means, see RAG evaluation metrics explained; for why a public leaderboard can't settle this for your documents, see why MTEB alone isn't enough.

Real benchmark - generated by Truvec

Benchmark: BGE Base vs OpenAI on English documentation

Corpus
Docker documentation: manuals, FAQs, billing and support pages
Documents
261
Chunks
1,944
Chunking
recursive, 1000 chars, 200 overlap
Golden set
100 questions, human-reviewed
Models
3 compared
Top k
5
Run
2026-11-14

Download the full report (PDF) Every question, with the passages each model retrieved.

Key results

  • OpenAI text-embedding-3-large ranks first, with an MRR of 0.867 and a hit rate of 99.0% in the top 5.
  • It leads the runner-up, BGE Base EN v1.5, by 0.094 MRR.
  • Hit rate ranges from 95.0% to 99.0% across the 3 models.
  • Fastest to embed a query: OpenAI text-embedding-3-small (6.3 ms per query, measured in batch from our servers).

Retrieval quality

Retrieval quality at top 5, ranked by MRR, then recall@k
ModelHit rate@5Recall@5MRRnDCG@5Precision@5
1OpenAI text-embedding-3-largeopenai - 3072 dims99.0%97.5%0.8670.88520.6%
2BGE Base EN v1.5fireworks - 768 dims95.0%92.5%0.7730.80219.4%
3OpenAI text-embedding-3-smallopenai - 1536 dims96.0%94.5%0.7710.81120.0%

Hit rate by position

How many questions the first results already answer. A model that finds the answer at position 1 lets you send fewer chunks to the LLM.

Hit rate by position: share of questions answered by the first j results
ModelTop 1Top 2Top 3Top 4Top 5
1OpenAI text-embedding-3-largeopenai - 3072 dims79.0%88.0%94.0%98.0%99.0%
2BGE Base EN v1.5fireworks - 768 dims64.0%85.0%90.0%92.0%95.0%
3OpenAI text-embedding-3-smallopenai - 1536 dims63.0%84.0%92.0%94.0%96.0%

Cost, speed and size

Embedding prices are the providers' list prices. Corpus cost is a one-time cost per model; the vector size is raw float32 storage, before the vector database's own overhead.

Cost, speed and size on this corpus
ModelPrice / 1M tokensEmbed the corpus1,000 queriesQuery embeddingSearch p95Vectors
1OpenAI text-embedding-3-largeopenai - 3072 dims$0.13$0.04$0.002612 ms57 ms22.8 MB
2BGE Base EN v1.5fireworks - 768 dims$0.0080$0.0027$0.000222 ms40 ms5.7 MB
3OpenAI text-embedding-3-smallopenai - 1536 dims$0.02$0.0067$0.00046.3 ms48 ms11.4 MB

Score separation

Average similarity of the best relevant chunk and of the best irrelevant one. A wide gap makes a similarity threshold reliable; a narrow one means right and wrong chunks look alike to the model.

Score separation: how clearly each model scores the right chunk above the best wrong one
ModelRelevant chunkBest distractorGap
1OpenAI text-embedding-3-largeopenai - 3072 dims0.6870.6500.037
2BGE Base EN v1.5fireworks - 768 dims0.8060.7820.023
3OpenAI text-embedding-3-smallopenai - 1536 dims0.7150.6870.027

Example questions

Rank of the first relevant chunk for each model; miss means it wasn't in the top 5. These are the questions where the models disagree most.

Example questions from the golden set
QuestionOpenAI text-embedding-3-largeBGE Base EN v1.5OpenAI text-embedding-3-small
How do I push an image to Docker Hub, and what tagging restriction applies?
Expected passage
## Push an image to Docker Hub Select the image from the list, select the **More options** button and select **Push to Hub**. > [!NOTE] > > You can only push an image to Docker Hub if the image belongs to your Docker ID or your organization. That is, the image must contain the co
miss31
How can I import a custom MCP catalog into Docker Desktop, and where does it appear afterward?
Expected passage
### Custom catalogs with Dynamic MCP Custom catalogs work particularly well with Dynamic MCP, where agents discover and add MCP servers on-demand during conversations. When you run the gateway with a custom catalog, the `mcp-find` tool searches only within that catalog. If your c
123
When both JIT and SCIM are enabled, how are organization roles and account deactivation handled?
Expected passage
Roles don't move back and forth. JIT sets the organization role only when it first adds the user. A later SCIM update can change that role, and the next sign-in leaves the SCIM role in place. When a user isn't assigned to the Docker application in the IdP, the next synchronizatio
53miss
Which Docker Desktop versions are supported under the Pro, Team, and Business plans?
Expected passage
### Supported versions Docker Desktop support is scoped by plan: | Plan | Supported versions | Machines per subscription | | :--- | :--- | :--- | | Docker Pro | Latest only | One | | Docker Team | Latest only | One per subscription seat | | Docker Business | Latest plus versions
213
Where can I inspect all Docker Desktop settings that have been applied, including settings not visible in the GUI?
Expected passage
As a workaround, you can check the `settings-store.json` file to view all applied settings: - Mac: `~/Library/Application Support/Docker/settings-store.json` - Windows: `%APPDATA%\Docker\settings-store.json` - Linux: `~/.docker/desktop/settings-store.json` The `settings-store.jso
142
What must an organization’s only owner do before leaving, and where can they initiate the departure?
Expected passage
If the account is the only owner of an organization or company, assign the owner role to another member first. Leave after that member can administer the organization or company. To leave an organization and its teams: 1. Sign in to Docker Home and select the organization. 1. Sel
312
How can I use a value from a `.env` file inside a Docker Compose service's environment settings?
Expected passage
Basic example: ```console $ cat .env ## define COMPOSE_DEBUG based on DEV_MODE, defaults to false COMPOSE_DEBUG=${DEV_MODE:-false} $ cat compose.yaml services: webapp: image: my-webapp-image environment: - DEBUG=${COMPOSE_DEBUG} $ DEV_MODE=true docker compose config services: web
152
What file and service configuration are required for Docker Hub automated tests, and how is the test result determined?
Expected passage
## Set up automated test files To set up your automated tests, create a `docker-compose.test.yml` file which defines a `sut` service that lists the tests to be run. The `docker-compose.test.yml` file should be located in the same directory that contains the Dockerfile used to bui
25miss

Notes

  • The corpus is a subset of the public Docker documentation (Apache 2.0), taken from github.com/docker/docs at commit 553482c: the Desktop, Docker Hub, Compose, accounts, security and subscription and billing manuals, the FAQs, the support pages and the getting-started guides. Release notes and pages shorter than 600 characters were left out, and pages were cut at 40,000 characters.
  • Several posts on this blog use this corpus and the same 100 questions, each with its own selection of models. A model can differ by a question between two of these runs: questions are embedded again at each run, and vector search is approximate.
  • With 100 questions, a gap of a couple of points between two models is within the noise.
  • Chunks average about 170 tokens, well under the 380 tokens BGE Base EN v1.5 reads per chunk in Truvec (model limit 512), so BGE saw the chunks in full.

Methodology

Truvec split the documents into chunks, generated a golden set of questions with gpt-5.6-luna (each question is linked to the chunks that answer it), embedded the corpus and the questions with every model, and retrieved the top 5 chunks for each question by cosine similarity. Every model saw the same chunks and the same questions. Scores are averaged over all 100 questions.

Hit rate
Share of questions with at least one relevant chunk in the top k results. The most direct answer to "does the right passage reach the LLM?".
Recall
Share of the relevant chunks found in the top k, averaged over questions. Matters when an answer spans several chunks.
MRR
Mean reciprocal rank: 1 when the first relevant chunk is ranked first, 1/2 when second, 0 when missing. Rewards putting the answer at the top.
nDCG
Normalized discounted cumulative gain: how close the ranking of all relevant chunks is to the ideal ranking.
Precision
Share of the top k results that are relevant. Low precision means more noise sent to the LLM.

Downloads: full report (PDF) - benchmark.json - benchmark.csv - this post as Markdown

OpenAI text-embedding-3-large won here, on these documents. Find the model that wins on yours: run the same benchmark on your documents with Truvec.