The best embedding model for Arabic: RAG on Arabic documents

Arabic is one of the hardest languages for embedding models: rich morphology, optional diacritics, spelling variants, dialects and Arabizi. What to watch for, how to build an Arabic test set, and a benchmark on Arabic support and news content.

  • embeddings
  • rag
  • arabic
  • multilingual
  • benchmark

Arabic is spoken by hundreds of millions of people, yet it's underrepresented in the data embedding models are trained on, and the gap between models is much wider than in English or French. A model that's excellent in English can be mediocre in Arabic, and English-only models are close to useless.

This post covers what makes Arabic retrieval hard, how to test it properly, and which models to compare. The last section benchmarks eight models on Arabic support articles and news.

What makes Arabic hard

Rich morphology. Arabic builds words from roots and patterns, and attaches articles, prepositions, conjunctions and pronouns to the word itself. "وبكتابهم" ("and with their book") is one written word containing four units of meaning. The same root appears in many forms ("كتب", "كاتب", "مكتبة", "مكتوب"), and a model must understand they're related.

Optional diacritics. Short vowels are usually not written. Without them, one written form can be several words: "علم" can be read as "knowledge", "flag" or "he knew". Context decides, and the model must use it. Some documents (religious texts, teaching material) do include diacritics, so the same word appears in two forms.

Spelling variants. Alef has several forms (أ إ آ ا) that writers often mix up, final "ة" and "ه" are interchanged, as are "ى" and "ي". Users type casually; documents may be more careful. A good model treats these variants as the same word.

Dialects. Modern Standard Arabic (MSA) is used in documents, but users often ask in their dialect: Egyptian, Levantine, Gulf, Maghrebi. A Moroccan user's question may share few words with the MSA article that answers it.

Arabizi and mixed text. Many users write Arabic in Latin letters with digits ("kifach n3awed lmot de passe"), or mix Arabic with French or English. Few models handle Arabizi well.

Tokenization. Tokenizers built mainly on English split Arabic into many small pieces. The same text costs more tokens in Arabic, which raises embedding costs and makes input limits arrive sooner.

Build an Arabic test set

  • Write or generate questions in Arabic, not translated from English.
  • Match your users: if they ask in dialect, include dialect questions against MSA documents. This is often where models differ most.
  • Include spelling variants (alef and taa marbuta forms) and, if your users write it, Arabizi.
  • If documents mix Arabic with French or English (common in the Maghreb and the Gulf), include questions in each language.

Prepare the documents

  • Normalize consistently, or not at all. Some pipelines strip diacritics and unify alef forms before embedding. This can help weaker models, but it must be applied the same way to documents and queries. Test both with and without it.
  • Check chunk sizes in tokens. Since Arabic uses more tokens per word, a chunk size tuned for English may exceed a short-input model's limit.

Which models to test

  • Multilingual API models: Cohere Embed v4 and Embed Multilingual v3, Gemini Embedding, OpenAI text-embedding-3-large and 3-small.
  • Open-weight: Qwen3 Embedding, trained on many languages and self-hostable.
  • An English-only baseline (such as BGE Base EN), to measure the gap.

The benchmark below runs eight models on Arabic support articles and news in Modern Standard Arabic. If your users write in dialect, treat these results as an upper bound and test with dialect questions. For what each metric in the tables means, see RAG evaluation metrics explained; for why a public leaderboard can't settle this for your documents, see why MTEB alone isn't enough.

Real benchmark - generated by Truvec

Benchmark: Embedding models on Arabic help pages and news

Corpus
Arabic Wikinews articles and Arabic Wikipedia help pages
Documents
233
Chunks
1,933
Chunking
recursive, 1000 chars, 200 overlap
Golden set
100 questions, human-reviewed
Models
8 compared
Top k
5
Run
2026-11-14

Key results

  • Voyage 4 Large ranks first, with an MRR of 0.970 and a hit rate of 99.0% in the top 5.
  • It leads the runner-up, Qwen3 Embedding 8B, by 0.051 MRR.
  • Hit rate ranges from 12.0% to 100.0% across the 8 models.
  • Cheapest model within 2 points of the winner's hit rate: Qwen3 Embedding 8B, $0.09 to embed this corpus against $0.10.
  • Fastest to embed a query: OpenAI text-embedding-3-small (4.2 ms per query, measured in batch from our servers).

Retrieval quality

Retrieval quality at top 5, ranked by MRR, then recall@k
ModelHit rate@5Recall@5MRRnDCG@5Precision@5
1Voyage 4 Largevoyage - 1024 dims99.0%99.0%0.9700.97519.8%
2Qwen3 Embedding 8Bfireworks - 4096 dims100.0%100.0%0.9190.94020.0%
3Cohere Embed v4cohere - 1536 dims100.0%100.0%0.9100.93320.0%
4Gemini Embedding 2google - 3072 dims98.0%98.0%0.9070.92619.6%
5Cohere Embed Multilingual v3cohere - 1024 dims96.0%96.0%0.8870.90519.2%
6OpenAI text-embedding-3-largeopenai - 3072 dims98.0%98.0%0.8820.90719.6%
7OpenAI text-embedding-3-smallopenai - 1536 dims92.0%92.0%0.8020.83218.4%
8BGE Base EN v1.5fireworks - 768 dims12.0%12.0%0.0870.0962.4%

Hit rate by position

How many questions the first results already answer. A model that finds the answer at position 1 lets you send fewer chunks to the LLM.

Hit rate by position: share of questions answered by the first j results
ModelTop 1Top 2Top 3Top 4Top 5
1Voyage 4 Largevoyage - 1024 dims95.0%99.0%99.0%99.0%99.0%
2Qwen3 Embedding 8Bfireworks - 4096 dims85.0%97.0%99.0%100.0%100.0%
3Cohere Embed v4cohere - 1536 dims85.0%93.0%97.0%99.0%100.0%
4Gemini Embedding 2google - 3072 dims84.0%96.0%98.0%98.0%98.0%
5Cohere Embed Multilingual v3cohere - 1024 dims83.0%93.0%93.0%94.0%96.0%
6OpenAI text-embedding-3-largeopenai - 3072 dims81.0%92.0%95.0%97.0%98.0%
7OpenAI text-embedding-3-smallopenai - 1536 dims71.0%85.0%91.0%91.0%92.0%
8BGE Base EN v1.5fireworks - 768 dims7.0%9.0%9.0%12.0%12.0%

Cost, speed and size

Embedding prices are the providers' list prices. Corpus cost is a one-time cost per model; the vector size is raw float32 storage, before the vector database's own overhead.

Cost, speed and size on this corpus
ModelPrice / 1M tokensEmbed the corpus1,000 queriesQuery embeddingSearch p95Vectors
1Voyage 4 Largevoyage - 1024 dims$0.12$0.10$0.006913 ms16 ms7.6 MB
2Qwen3 Embedding 8Bfireworks - 4096 dims$0.10$0.09$0.005727 ms23 ms30.2 MB
3Cohere Embed v4cohere - 1536 dims$0.12$0.10$0.006916 ms13 ms11.3 MB
4Gemini Embedding 2google - 3072 dims$0.20$0.17$0.0142 ms38 ms22.7 MB
5Cohere Embed Multilingual v3cohere - 1024 dims$0.10$0.08$0.00575.5 ms66 ms7.6 MB
6OpenAI text-embedding-3-largeopenai - 3072 dims$0.13$0.11$0.007412 ms46 ms22.7 MB
7OpenAI text-embedding-3-smallopenai - 1536 dims$0.02$0.02$0.00114.2 ms22 ms11.3 MB
8BGE Base EN v1.5fireworks - 768 dims$0.0080$0.0049$0.000512 ms12 ms5.7 MB

Score separation

Average similarity of the best relevant chunk and of the best irrelevant one. A wide gap makes a similarity threshold reliable; a narrow one means right and wrong chunks look alike to the model.

Score separation: how clearly each model scores the right chunk above the best wrong one
ModelRelevant chunkBest distractorGap
1Voyage 4 Largevoyage - 1024 dims0.6020.5080.094
2Qwen3 Embedding 8Bfireworks - 4096 dims0.7770.7030.074
3Cohere Embed v4cohere - 1536 dims0.5770.5050.072
4Gemini Embedding 2google - 3072 dims0.8030.7610.042
5Cohere Embed Multilingual v3cohere - 1024 dims0.7380.6830.056
6OpenAI text-embedding-3-largeopenai - 3072 dims0.6100.5470.064
7OpenAI text-embedding-3-smallopenai - 1536 dims0.5610.5110.048
8BGE Base EN v1.5fireworks - 768 dims0.7190.7380.013

Example questions

Rank of the first relevant chunk for each model; miss means it wasn't in the top 5. These are the questions where the models disagree most.

Example questions from the golden set
QuestionVoyage 4 LargeQwen3 Embedding 8BCohere Embed v4Gemini Embedding 2Cohere Embed Multilingual v3OpenAI text-embedding-3-largeOpenAI text-embedding-3-smallBGE Base EN v1.5
كيف تعمل المتغيرات داخل قوالب ويكيبيديا، وما نوع البيانات التي يمكن تغييرها من مقال إلى آخر؟
Expected passage
## قالب بمتغيرات يمكن لقالب أن يضم متغيرات تتغير قيمها حسب الصفحة التي يوجد بها القالب. الشيء الذي يمكن مثلًا من استعمال جدول يضم معلومات تتغير حسب المقال، انظر مثلًا: قالب:Demography 4col {| border="0" width={{{1}}} cellspacing="2" cellpadding="2" | colspan="4" bgcolor="#eeeeee"
1142523miss
ما الحقول التي يمكن استخدامها للبحث في سجلات ويكيبيديا وتصفية نتائجها؟
Expected passage
نوع السجل المراد عرضه. بشكل افتراضي ، سيتم عرض جميع السجلات العامة، باستثناء سجلات الخفر والمراجعة والوسوم والشكر. يمكن تمكينها في "إظهار السجلات الإضافية" عند تحديد "جميع السجلات العامة". تم تعيين المثال لعرض سجل الحذف فقط، والذي يسرد عمليات حذف الصفحات واستعادتها. يتيح لك حقل ا
1352111miss
ما الظروف الصحية والمعيشية التي واجهها الأسرى الفلسطينيون خلال الإضراب؟
Expected passage
كما نقل الأسير مروان البرغوثي إلى المستشفى، إضافة إلى 120 أسيرا، وذلك بعد تدهور وضعهم الصحي. ومنذ 39 يوما، يرتدي الأسرى ملابس إدارة سجون الاحتلال (الشاباص)، ولا يملكون سوى أغطية قذرة وبالية، بعدما صادرت إدارة السجون مقتنياتهم منذ اليوم الأول من الإضراب، وفي زيارة أجراها محامي ناد
miss43243missmiss
ما الهدف من القاموس الإلكتروني الخاص بلغة الأينو والجهات المشاركة في تطويره؟
Expected passage
الخميس 21 يوليو 2016 يسعى أحد القواميس الإلكترونية الناطقة الذي أطلق لأول مرة عام 2009 إلى المحافظة على لغة الأينو ونشرها؛ الأينو هي لغة السكان الأصليين في شمال شرق جزيرة هوكايدو اليابانية. وبحسب منظمة اليونسكو يوجد في اليابان حوالى ثمانى لغات مهددة بالانقراض، وتتصدر لغة الأينو ه
1113112miss
ما الدول الأوروبية الثماني التي طالبت إسرائيل بتعويضات عن منشآت هُدمت في المنطقة ج من الضفة الغربية؟
Expected passage
أعدت 8 دول أوروبية رسالة احتجاج تتضمن مطالبة الاحتلال "الإسرائيلي" بدفع تعويضات بقيمة 30 ألف يورو، مقابل مصادرة وتدمير مبان ومنشآت بنى تحتية أقامتها هذه الدول في المنطقة "ج" بالضفة الغربية. وبحسب ما نقلته صحيفة "هآرتس" العبرية عن دبلوماسي أوروبي كبير فإن الرسالة ستسلم إلى مسؤولين
1211113miss
ماذا تعني رسالة "توقيع خام غير صالح" في إعدادات توقيع ويكيبيديا؟
Expected passage
[[مستخدم:مثال|مثال مستخدم]]: [[نقاش المستخدم:مثال|نقاش]] سيؤدي ذلك إلى ظهور توقيع يبدو مثل: [[مستخدم:مثال|مثال مستخدم]]: [[نقاش المستخدم:مثال|نقاش]] (نقاش) بدلاً من مثال مستخدم: نقاش يرجع السبب في ذلك إلى أن نص ويكي (وترميز HTML) يجري الآن إفلاته ككيانات HTML عند وضعه في حقل التو
1111513miss
ما أبرز الأوامر الملكية السعودية التي ذُكرت باعتبارها ذات فائدة اقتصادية للمواطنين؟
Expected passage
«ترجمة:أصدر العاهل السعودي مجموعة من الأوامر الملكية التي تعود بالفائدة اقتصادياَ على الشعب السعودي، ومنها: مكافآت مالية وتحديد حد أدنى لأجور السعوديين وغيرها. من سلطان القاسمي.» «النص الأصلي:@Saudiwoman: 16billion to health ministry to build cancer units and other specialized un
1131112miss
ما أنواع الأحداث التي يمكن للمستخدم تلقي إشعارات عنها عبر البريد الإلكتروني في ويكيبيديا؟
Expected passage
تتوفر إشعارات البريد الإلكتروني للرسائل المتبقية على صفحة نقاش المستخدم الخاصة بك، عندما تتم الإشارة إليك أو شكرك، عندما يقوم شخص ما بربط أو مراجعة صفحة قمت بإنشائها، عندما يقوم شخص ما بتغيير صلاحيات المستخدم الخاصة بك، ومحاولات تسجيل الدخول الفاشلة، وعمليات تسجيل الدخول الناجحة
1221miss13miss

Notes

  • The corpus is 181 articles from Arabic Wikinews (ar.wikinews.org) and 52 help pages from Arabic Wikipedia (ar.wikipedia.org, Help namespace), both CC BY-SA 4.0, fetched as plain text on 14 November 2026. Pages were picked at random among those between 1,500 and 30,000 characters.
  • All 100 questions are written in Modern Standard Arabic: 60 are about news articles and 40 about help pages. There are no dialect or Arabizi questions, so treat these results as an upper bound for dialect users.
  • BGE Base EN v1.5 is English-only: it is included as a baseline.
  • Arabic takes more tokens per character than English: these 1,000-character chunks average about 440 tokens. BGE Base EN v1.5 reads at most 380 tokens per chunk in Truvec and Cohere Embed Multilingual v3 at most 512, so both saw some chunks cut short.
  • With 100 questions, a gap of a couple of points between two models is within the noise.

Methodology

Truvec split the documents into chunks, generated a golden set of questions with gpt-5.6-luna (each question is linked to the chunks that answer it), embedded the corpus and the questions with every model, and retrieved the top 5 chunks for each question by cosine similarity. Every model saw the same chunks and the same questions. Scores are averaged over all 100 questions.

Hit rate
Share of questions with at least one relevant chunk in the top k results. The most direct answer to "does the right passage reach the LLM?".
Recall
Share of the relevant chunks found in the top k, averaged over questions. Matters when an answer spans several chunks.
MRR
Mean reciprocal rank: 1 when the first relevant chunk is ranked first, 1/2 when second, 0 when missing. Rewards putting the answer at the top.
nDCG
Normalized discounted cumulative gain: how close the ranking of all relevant chunks is to the ideal ranking.
Precision
Share of the top k results that are relevant. Low precision means more noise sent to the LLM.

Downloads: benchmark.json - benchmark.csv - this post as Markdown

Voyage 4 Large won here, on these documents. Find the model that wins on yours: run the same benchmark on your documents with Truvec.