# BGE vs OpenAI embeddings: is a small open model good enough?

> BGE Base is a small open-source embedding model you can run almost anywhere. How it compares with OpenAI's text-embedding-3 models on English documents, where its 512-token limit and English-only training matter, and a benchmark.

Published 2026-10-06, updated 2026-11-14 by Truvec team. Canonical: https://truvec.dev/blog/bge-vs-openai-embeddings/

BGE (BAAI General Embedding) models are among the most downloaded open-source embedding models. BGE Base EN v1.5 is small enough to run on a CPU, produces compact 768-dimension vectors, and costs almost nothing through a hosted API. OpenAI's text-embedding-3 models are the default choice for many teams. The question is simple: does the small open model give up enough quality to justify paying for OpenAI?

This post compares BGE Base EN v1.5 with OpenAI text-embedding-3-small and 3-large, then benchmarks them on English documentation.

## At a glance

| | BGE Base EN v1.5 | OpenAI 3-small | OpenAI 3-large |
|---|---|---|---|
| Weights | Open source | API only | API only |
| Price per 1M tokens (hosted, in Truvec) | $0.008 | $0.02 | $0.13 |
| Dimensions | 768 | 1,536 | 3,072 |
| Max input | 512 tokens | 8,191 tokens | 8,191 tokens |
| Languages | English | Multilingual | Multilingual |

## What to know about BGE

**512 tokens is the hard limit.** Anything past it is ignored. With 1,000-character chunks (about 250 tokens) that's fine. With long chunks, BGE is evaluated on truncated text, and its scores drop for reasons that have nothing to do with the model's understanding. If you use BGE, keep chunks short.

**It's English-only.** On French, Arabic or mixed-language content, it falls far behind multilingual models. Don't use it outside English.

**It needs a query prefix.** BGE was trained with an instruction prepended to queries ("Represent this sentence for searching relevant passages: "). Without it, retrieval gets worse. Many integrations forget it.

**It's tiny to run.** A base-size model embeds thousands of chunks per minute on modest hardware, and the vectors take a quarter of the space of OpenAI 3-large's. For edge deployments, on-premises installs or tight budgets, that matters more than a few points of quality.

## When BGE is good enough

- English content, short chunks, and a budget or hosting constraint;
- a first version of a RAG system, where you need something that works today and can switch later;
- a large corpus that gets re-embedded often, where per-token cost adds up.

## When to pay for OpenAI (or another API)

- Non-English or mixed-language content;
- long chunks you don't want to split further;
- a measurable quality gap on your documents that changes your users' answers.

The benchmark below compares the three models on English developer documentation, with chunks short enough for BGE to read in full. Look at the size of the quality gap, then at the cost column: whether the gap is worth the price is your call, but now it's an informed one. For what each metric in the tables means, see [RAG evaluation metrics explained](/blog/rag-evaluation-metrics-that-matter/); for why a public leaderboard can't settle this for your documents, see [why MTEB alone isn't enough](/blog/mteb-benchmark-validate-embedding-model-before-switching/).

## Benchmark: BGE Base vs OpenAI on English documentation

Real benchmark generated by Truvec (benchmark #6, run 2026-11-14). Raw data: [benchmark.json](https://truvec.dev/blog/bge-vs-openai-embeddings/benchmark.json), [benchmark.csv](https://truvec.dev/blog/bge-vs-openai-embeddings/benchmark.csv). Full report with every question: [report.pdf](https://truvec.dev/blog/bge-vs-openai-embeddings/report.pdf).

- Corpus: Docker documentation: manuals, FAQs, billing and support pages (https://github.com/docker/docs), 261 documents, 1,944 chunks
- Chunking: recursive, 1000 chars, 200 overlap
- Golden set: 100 questions generated by gpt-5.6-luna, human-reviewed
- Retrieval: top 5 by cosine similarity; models ranked by MRR, then recall@k

### Key results

- **OpenAI text-embedding-3-large** ranks first, with an MRR of 0.867 and a hit rate of 99.0% in the top 5.
- It leads the runner-up, BGE Base EN v1.5, by 0.094 MRR.
- Hit rate ranges from 95.0% to 99.0% across the 3 models.
- Fastest to embed a query: **OpenAI text-embedding-3-small** (6.3 ms per query, measured in batch from our servers).

### Retrieval quality

| Rank | Model | Provider | Hit rate@5 | Recall@5 | MRR | nDCG@5 | Precision@5 |
| --- | --- | --- | ---: | ---: | ---: | ---: | ---: |
| 1 | OpenAI text-embedding-3-large | openai | 99.0% | 97.5% | 0.867 | 0.885 | 20.6% |
| 2 | BGE Base EN v1.5 | fireworks | 95.0% | 92.5% | 0.773 | 0.802 | 19.4% |
| 3 | OpenAI text-embedding-3-small | openai | 96.0% | 94.5% | 0.771 | 0.811 | 20.0% |

### Hit rate by position

| Model | Top 1 | Top 2 | Top 3 | Top 4 | Top 5 |
| --- | ---: | ---: | ---: | ---: | ---: |
| OpenAI text-embedding-3-large | 79.0% | 88.0% | 94.0% | 98.0% | 99.0% |
| BGE Base EN v1.5 | 64.0% | 85.0% | 90.0% | 92.0% | 95.0% |
| OpenAI text-embedding-3-small | 63.0% | 84.0% | 92.0% | 94.0% | 96.0% |

### Cost, speed and size

| Model | Price / 1M tokens | Embed the corpus | 1,000 queries | Query embedding | Search p95 | Vectors |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| OpenAI text-embedding-3-large | $0.13 | $0.04 | $0.0026 | 12 ms | 57 ms | 22.8 MB |
| BGE Base EN v1.5 | $0.0080 | $0.0027 | $0.0002 | 22 ms | 40 ms | 5.7 MB |
| OpenAI text-embedding-3-small | $0.02 | $0.0067 | $0.0004 | 6.3 ms | 48 ms | 11.4 MB |

### Score separation

| Model | Relevant chunk | Best distractor | Gap |
| --- | ---: | ---: | ---: |
| OpenAI text-embedding-3-large | 0.687 | 0.650 | 0.037 |
| BGE Base EN v1.5 | 0.806 | 0.782 | 0.023 |
| OpenAI text-embedding-3-small | 0.715 | 0.687 | 0.027 |

### Example questions

Rank of the first relevant chunk per model ("miss": not in the top 5).

| Question | OpenAI text-embedding-3-large | BGE Base EN v1.5 | OpenAI text-embedding-3-small |
| --- | ---: | ---: | ---: |
| How do I push an image to Docker Hub, and what tagging restriction applies? | miss | 3 | 1 |
| How can I import a custom MCP catalog into Docker Desktop, and where does it appear afterward? | 1 | 2 | 3 |
| When both JIT and SCIM are enabled, how are organization roles and account deactivation handled? | 5 | 3 | miss |
| Which Docker Desktop versions are supported under the Pro, Team, and Business plans? | 2 | 1 | 3 |
| Where can I inspect all Docker Desktop settings that have been applied, including settings not visible in the GUI? | 1 | 4 | 2 |
| What must an organization’s only owner do before leaving, and where can they initiate the departure? | 3 | 1 | 2 |
| How can I use a value from a `.env` file inside a Docker Compose service's environment settings? | 1 | 5 | 2 |
| What file and service configuration are required for Docker Hub automated tests, and how is the test result determined? | 2 | 5 | miss |

### Notes

- The corpus is a subset of the public Docker documentation (Apache 2.0), taken from github.com/docker/docs at commit 553482c: the Desktop, Docker Hub, Compose, accounts, security and subscription and billing manuals, the FAQs, the support pages and the getting-started guides. Release notes and pages shorter than 600 characters were left out, and pages were cut at 40,000 characters.
- Several posts on this blog use this corpus and the same 100 questions, each with its own selection of models. A model can differ by a question between two of these runs: questions are embedded again at each run, and vector search is approximate.
- With 100 questions, a gap of a couple of points between two models is within the noise.
- Chunks average about 170 tokens, well under the 380 tokens BGE Base EN v1.5 reads per chunk in Truvec (model limit 512), so BGE saw the chunks in full.

### Metric definitions

- **Hit rate**: Share of questions with at least one relevant chunk in the top k results. The most direct answer to "does the right passage reach the LLM?".
- **Recall**: Share of the relevant chunks found in the top k, averaged over questions. Matters when an answer spans several chunks.
- **MRR**: Mean reciprocal rank: 1 when the first relevant chunk is ranked first, 1/2 when second, 0 when missing. Rewards putting the answer at the top.
- **nDCG**: Normalized discounted cumulative gain: how close the ranking of all relevant chunks is to the ideal ranking.
- **Precision**: Share of the top k results that are relevant. Low precision means more noise sent to the LLM.
