Qwen3 Embedding vs OpenAI: can an open-weight model replace text-embedding-3?
Qwen3 Embedding 8B is an open-weight model you can self-host or call through an API. How it compares with OpenAI's text-embedding-3 models on quality, cost, vector size, input length and hosting, with a benchmark.
Open-weight embedding models have caught up with the commercial APIs on many tasks. Qwen3 Embedding is one of the most capable: its weights are published, so you can run it on your own GPUs, and hosted APIs serve it per token. The question for many teams is whether it can replace OpenAI's text-embedding-3 models, and what the switch would cost.
This post compares Qwen3 Embedding 8B with OpenAI text-embedding-3-small and 3-large. The last section is a benchmark of the three.
At a glance
| Qwen3 Embedding 8B | OpenAI 3-small | OpenAI 3-large | |
|---|---|---|---|
| Weights | Open: self-host or hosted API | API only | API only |
| Price per 1M tokens (hosted, in Truvec) | $0.10 | $0.02 | $0.13 |
| Dimensions | 4,096 | 1,536 | 3,072 |
| Query instruction | Yes: a task instruction is prepended to queries | No | No |
Where Qwen3 Embedding differs
You can own it. Open weights mean your text never has to leave your infrastructure, the model won't be deprecated under you, and the price is your GPU bill rather than a per-token rate. For regulated data or air-gapped deployments, this alone can decide.
Self-hosting isn't free. An 8-billion-parameter model needs a GPU with enough memory, and embedding a large corpus takes GPU hours. Smaller Qwen3 Embedding sizes exist if the 8B model is too heavy; test the size you'd actually deploy. At low volume, a hosted API is usually cheaper than an idle GPU.
Bigger vectors. 4,096 dimensions take 33% more storage than OpenAI 3-large and 2.7 times more than 3-small. At a million chunks, that's about 16 GB of raw vectors. The model supports shorter outputs; if you plan to use them, benchmark the shorter size.
Instructions. Qwen3 Embedding expects an instruction before the query, describing the task ("Given a web search query, retrieve relevant passages..."). The right instruction can improve retrieval for your domain. A missing one quietly lowers scores, a common mistake when people compare it with OpenAI.
Long inputs and many languages. It reads long texts and is trained on many languages, so it's worth testing on multilingual corpora and long chunks.
When it's worth switching
- Data can't leave your infrastructure: Qwen3 is one of the strongest options you can run yourself. Measure what it gives up, if anything, against the API models.
- High, steady volume: a busy self-hosted GPU can be cheaper than per-token pricing.
- Multilingual content: it's a strong candidate; compare it per language.
- Small, English-only corpus with low traffic: the operational cost of self-hosting is rarely worth it. A cheap API model may tie on quality.
The benchmark below runs the three models on developer documentation and support articles, through the same hosted pipeline, so the comparison is about retrieval, not infrastructure. For what each metric in the tables means, see RAG evaluation metrics explained; for why a public leaderboard can't settle this for your documents, see why MTEB alone isn't enough.
Real benchmark - generated by Truvec
Benchmark: Qwen3 Embedding 8B vs OpenAI on a technical corpus
- Documents
- 261
- Chunks
- 1,944
- Chunking
- recursive, 1000 chars, 200 overlap
- Golden set
- 100 questions, human-reviewed
- Models
- 3 compared
- Top k
- 5
- Run
- 2026-11-14
Download the full report (PDF) Every question, with the passages each model retrieved.
Key results
- OpenAI text-embedding-3-large ranks first, with an MRR of 0.867 and a hit rate of 99.0% in the top 5.
- One model is within 0.02 MRR of the winner (Qwen3 Embedding 8B): on this corpus they are practically tied.
- Hit rate ranges from 95.0% to 99.0% across the 3 models.
- Cheapest model within 2 points of the winner's hit rate: Qwen3 Embedding 8B, $0.03 to embed this corpus against $0.04.
- Fastest to embed a query: OpenAI text-embedding-3-small (5.2 ms per query, measured in batch from our servers).
Retrieval quality
| Model | Hit rate@5 | Recall@5 | MRR | nDCG@5 | Precision@5 |
|---|---|---|---|---|---|
| 1OpenAI text-embedding-3-large | 99.0% | 97.5% | 0.885 | 20.6% | |
| 2Qwen3 Embedding 8B | 99.0% | 98.0% | 0.883 | 20.8% | |
| 3OpenAI text-embedding-3-small | 95.0% | 93.5% | 0.807 | 19.8% |
Hit rate by position
How many questions the first results already answer. A model that finds the answer at position 1 lets you send fewer chunks to the LLM.
| Model | Top 1 | Top 2 | Top 3 | Top 4 | Top 5 |
|---|---|---|---|---|---|
| 1OpenAI text-embedding-3-large | 79.0% | 88.0% | 94.0% | 98.0% | 99.0% |
| 2Qwen3 Embedding 8B | 77.0% | 88.0% | 95.0% | 99.0% | 99.0% |
| 3OpenAI text-embedding-3-small | 63.0% | 84.0% | 92.0% | 94.0% | 95.0% |
Cost, speed and size
Embedding prices are the providers' list prices. Corpus cost is a one-time cost per model; the vector size is raw float32 storage, before the vector database's own overhead.
| Model | Price / 1M tokens | Embed the corpus | 1,000 queries | Query embedding | Search p95 | Vectors |
|---|---|---|---|---|---|---|
| 1OpenAI text-embedding-3-large | $0.13 | $0.04 | $0.0026 | 8.3 ms | 50 ms | 22.8 MB |
| 2Qwen3 Embedding 8B | $0.10 | $0.03 | $0.0020 | 13 ms | 43 ms | 30.4 MB |
| 3OpenAI text-embedding-3-small | $0.02 | $0.0067 | $0.0004 | 5.2 ms | 38 ms | 11.4 MB |
Score separation
Average similarity of the best relevant chunk and of the best irrelevant one. A wide gap makes a similarity threshold reliable; a narrow one means right and wrong chunks look alike to the model.
| Model | Relevant chunk | Best distractor | Gap |
|---|---|---|---|
| 1OpenAI text-embedding-3-large | 0.687 | 0.650 | 0.037 |
| 2Qwen3 Embedding 8B | 0.827 | 0.785 | 0.042 |
| 3OpenAI text-embedding-3-small | 0.715 | 0.687 | 0.027 |
Example questions
Rank of the first relevant chunk for each model; miss means it wasn't in the top 5. These are the questions where the models disagree most.
| Question | OpenAI text-embedding-3-large | Qwen3 Embedding 8B | OpenAI text-embedding-3-small |
|---|---|---|---|
How do I push an image to Docker Hub, and what tagging restriction applies?Expected passage## Push an image to Docker Hub Select the image from the list, select the **More options** button and select **Push to Hub**. > [!NOTE] > > You can only push an image to Docker Hub if the image belongs to your Docker ID or your organization. That is, the image must contain the co | miss | 4 | 1 |
What environment variables does Compose generate when a service uses models with the short syntax?Expected passage```yaml services: app: image: my-app models: - llm - embedding-model models: llm: model: ai/smollm2 embedding-model: model: ai/all-minilm ``` With short syntax, the platform automatically generates environment variables based on the model name: - `LLM_URL` - URL to access the LLM | 1 | 3 | miss |
Which organizational roles and subscription plans are needed to create organization access tokens in Docker?Expected passage> [!WARNING] > > OATs don't work with Docker Desktop or Image Access Management. For > those features, use > PATs instead. ## Best practices - Set an expiration date and rotate tokens on a schedule - Grant only the repositories and scopes each job needs - Review the **Last used** | 4 | 2 | miss |
Which Docker Desktop versions are supported under the Pro, Team, and Business plans?Expected passage### Supported versions Docker Desktop support is scoped by plan: | Plan | Supported versions | Machines per subscription | | :--- | :--- | :--- | | Docker Pro | Latest only | One | | Docker Team | Latest only | One per subscription seat | | Docker Business | Latest plus versions | 2 | 1 | 3 |
How do I create, publish, and share a custom MCP catalog with my team?Expected passageCreate a catalog and specify which servers to include: ```console $ docker mcp catalog create registry.example.com/mcp/data-tools:latest \ --title "Data Analysis Tools" \ --server catalog://mcp/docker-mcp-catalog/sequentialthinking \ --server catalog://mcp/docker-mcp-catalog/brav | 1 | 3 | 2 |
Where can I inspect all Docker Desktop settings that have been applied, including settings not visible in the GUI?Expected passageAs a workaround, you can check the `settings-store.json` file to view all applied settings: - Mac: `~/Library/Application Support/Docker/settings-store.json` - Windows: `%APPDATA%\Docker\settings-store.json` - Linux: `~/.docker/desktop/settings-store.json` The `settings-store.jso | 1 | 3 | 2 |
How do I search Docker Desktop logs, move between matching entries, and clear the terminal?Expected passage### Logs Select **Logs** to view output from the container in real time. While viewing logs, you can: - Use `Cmd + f`/`Ctrl + f` to open the search bar and find specific entries. Search matches are highlighted in yellow. - Press `Enter` or `Shift + Enter` to jump to the next or p | 4 | 3 | 1 |
How are the Containers proxy and Docker Desktop proxy different when Docker Desktop accesses the network?Expected passageHost firewalls can permit or deny inbound connections by filtering on `com.docker.backend`. ## Using Docker Desktop with a proxy Docker Desktop can use your system’s default proxy settings or custom settings that you configure with Docker Desktop’s proxy setting. Docker Desktop r | 3 | 1 | 2 |
Notes
- The corpus is a subset of the public Docker documentation (Apache 2.0), taken from github.com/docker/docs at commit 553482c: the Desktop, Docker Hub, Compose, accounts, security and subscription and billing manuals, the FAQs, the support pages and the getting-started guides. Release notes and pages shorter than 600 characters were left out, and pages were cut at 40,000 characters.
- Several posts on this blog use this corpus and the same 100 questions, each with its own selection of models. A model can differ by a question between two of these runs: questions are embedded again at each run, and vector search is approximate.
- With 100 questions, a gap of a couple of points between two models is within the noise.
Methodology
Truvec split the documents into chunks, generated a golden set of questions with gpt-5.6-luna (each question is linked to the chunks that answer it), embedded the corpus and the questions with every model, and retrieved the top 5 chunks for each question by cosine similarity. Every model saw the same chunks and the same questions. Scores are averaged over all 100 questions.
- Hit rate
- Share of questions with at least one relevant chunk in the top k results. The most direct answer to "does the right passage reach the LLM?".
- Recall
- Share of the relevant chunks found in the top k, averaged over questions. Matters when an answer spans several chunks.
- MRR
- Mean reciprocal rank: 1 when the first relevant chunk is ranked first, 1/2 when second, 0 when missing. Rewards putting the answer at the top.
- nDCG
- Normalized discounted cumulative gain: how close the ranking of all relevant chunks is to the ideal ranking.
- Precision
- Share of the top k results that are relevant. Low precision means more noise sent to the LLM.
Downloads: full report (PDF) - benchmark.json - benchmark.csv - this post as Markdown
OpenAI text-embedding-3-large won here, on these documents. Find the model that wins on yours: run the same benchmark on your documents with Truvec.