The best embedding model for Arabic: RAG on Arabic documents
Arabic is one of the hardest languages for embedding models: rich morphology, optional diacritics, spelling variants, dialects and Arabizi. What to watch for, how to build an Arabic test set, and a benchmark on Arabic support and news content.
Arabic is spoken by hundreds of millions of people, yet it's underrepresented in the data embedding models are trained on, and the gap between models is much wider than in English or French. A model that's excellent in English can be mediocre in Arabic, and English-only models are close to useless.
This post covers what makes Arabic retrieval hard, how to test it properly, and which models to compare. The last section benchmarks eight models on Arabic support articles and news.
What makes Arabic hard
Rich morphology. Arabic builds words from roots and patterns, and attaches articles, prepositions, conjunctions and pronouns to the word itself. "وبكتابهم" ("and with their book") is one written word containing four units of meaning. The same root appears in many forms ("كتب", "كاتب", "مكتبة", "مكتوب"), and a model must understand they're related.
Optional diacritics. Short vowels are usually not written. Without them, one written form can be several words: "علم" can be read as "knowledge", "flag" or "he knew". Context decides, and the model must use it. Some documents (religious texts, teaching material) do include diacritics, so the same word appears in two forms.
Spelling variants. Alef has several forms (أ إ آ ا) that writers often mix up, final "ة" and "ه" are interchanged, as are "ى" and "ي". Users type casually; documents may be more careful. A good model treats these variants as the same word.
Dialects. Modern Standard Arabic (MSA) is used in documents, but users often ask in their dialect: Egyptian, Levantine, Gulf, Maghrebi. A Moroccan user's question may share few words with the MSA article that answers it.
Arabizi and mixed text. Many users write Arabic in Latin letters with digits ("kifach n3awed lmot de passe"), or mix Arabic with French or English. Few models handle Arabizi well.
Tokenization. Tokenizers built mainly on English split Arabic into many small pieces. The same text costs more tokens in Arabic, which raises embedding costs and makes input limits arrive sooner.
Build an Arabic test set
- Write or generate questions in Arabic, not translated from English.
- Match your users: if they ask in dialect, include dialect questions against MSA documents. This is often where models differ most.
- Include spelling variants (alef and taa marbuta forms) and, if your users write it, Arabizi.
- If documents mix Arabic with French or English (common in the Maghreb and the Gulf), include questions in each language.
Prepare the documents
- Normalize consistently, or not at all. Some pipelines strip diacritics and unify alef forms before embedding. This can help weaker models, but it must be applied the same way to documents and queries. Test both with and without it.
- Check chunk sizes in tokens. Since Arabic uses more tokens per word, a chunk size tuned for English may exceed a short-input model's limit.
Which models to test
- Multilingual API models: Cohere Embed v4 and Embed Multilingual v3, Gemini Embedding, OpenAI text-embedding-3-large and 3-small.
- Open-weight: Qwen3 Embedding, trained on many languages and self-hostable.
- An English-only baseline (such as BGE Base EN), to measure the gap.
The benchmark below runs eight models on Arabic support articles and news in Modern Standard Arabic. If your users write in dialect, treat these results as an upper bound and test with dialect questions. For what each metric in the tables means, see RAG evaluation metrics explained; for why a public leaderboard can't settle this for your documents, see why MTEB alone isn't enough.
Real benchmark - generated by Truvec
Benchmark: Embedding models on Arabic help pages and news
- Documents
- 233
- Chunks
- 1,933
- Chunking
- recursive, 1000 chars, 200 overlap
- Golden set
- 100 questions, human-reviewed
- Models
- 8 compared
- Top k
- 5
- Run
- 2026-11-14
Key results
- Voyage 4 Large ranks first, with an MRR of 0.970 and a hit rate of 99.0% in the top 5.
- It leads the runner-up, Qwen3 Embedding 8B, by 0.051 MRR.
- Hit rate ranges from 12.0% to 100.0% across the 8 models.
- Cheapest model within 2 points of the winner's hit rate: Qwen3 Embedding 8B, $0.09 to embed this corpus against $0.10.
- Fastest to embed a query: OpenAI text-embedding-3-small (4.2 ms per query, measured in batch from our servers).
Retrieval quality
| Model | Hit rate@5 | Recall@5 | MRR | nDCG@5 | Precision@5 |
|---|---|---|---|---|---|
| 1Voyage 4 Large | 99.0% | 99.0% | 0.975 | 19.8% | |
| 2Qwen3 Embedding 8B | 100.0% | 100.0% | 0.940 | 20.0% | |
| 3Cohere Embed v4 | 100.0% | 100.0% | 0.933 | 20.0% | |
| 4Gemini Embedding 2 | 98.0% | 98.0% | 0.926 | 19.6% | |
| 5Cohere Embed Multilingual v3 | 96.0% | 96.0% | 0.905 | 19.2% | |
| 6OpenAI text-embedding-3-large | 98.0% | 98.0% | 0.907 | 19.6% | |
| 7OpenAI text-embedding-3-small | 92.0% | 92.0% | 0.832 | 18.4% | |
| 8BGE Base EN v1.5 | 12.0% | 12.0% | 0.096 | 2.4% |
Hit rate by position
How many questions the first results already answer. A model that finds the answer at position 1 lets you send fewer chunks to the LLM.
| Model | Top 1 | Top 2 | Top 3 | Top 4 | Top 5 |
|---|---|---|---|---|---|
| 1Voyage 4 Large | 95.0% | 99.0% | 99.0% | 99.0% | 99.0% |
| 2Qwen3 Embedding 8B | 85.0% | 97.0% | 99.0% | 100.0% | 100.0% |
| 3Cohere Embed v4 | 85.0% | 93.0% | 97.0% | 99.0% | 100.0% |
| 4Gemini Embedding 2 | 84.0% | 96.0% | 98.0% | 98.0% | 98.0% |
| 5Cohere Embed Multilingual v3 | 83.0% | 93.0% | 93.0% | 94.0% | 96.0% |
| 6OpenAI text-embedding-3-large | 81.0% | 92.0% | 95.0% | 97.0% | 98.0% |
| 7OpenAI text-embedding-3-small | 71.0% | 85.0% | 91.0% | 91.0% | 92.0% |
| 8BGE Base EN v1.5 | 7.0% | 9.0% | 9.0% | 12.0% | 12.0% |
Cost, speed and size
Embedding prices are the providers' list prices. Corpus cost is a one-time cost per model; the vector size is raw float32 storage, before the vector database's own overhead.
| Model | Price / 1M tokens | Embed the corpus | 1,000 queries | Query embedding | Search p95 | Vectors |
|---|---|---|---|---|---|---|
| 1Voyage 4 Large | $0.12 | $0.10 | $0.0069 | 13 ms | 16 ms | 7.6 MB |
| 2Qwen3 Embedding 8B | $0.10 | $0.09 | $0.0057 | 27 ms | 23 ms | 30.2 MB |
| 3Cohere Embed v4 | $0.12 | $0.10 | $0.0069 | 16 ms | 13 ms | 11.3 MB |
| 4Gemini Embedding 2 | $0.20 | $0.17 | $0.01 | 42 ms | 38 ms | 22.7 MB |
| 5Cohere Embed Multilingual v3 | $0.10 | $0.08 | $0.0057 | 5.5 ms | 66 ms | 7.6 MB |
| 6OpenAI text-embedding-3-large | $0.13 | $0.11 | $0.0074 | 12 ms | 46 ms | 22.7 MB |
| 7OpenAI text-embedding-3-small | $0.02 | $0.02 | $0.0011 | 4.2 ms | 22 ms | 11.3 MB |
| 8BGE Base EN v1.5 | $0.0080 | $0.0049 | $0.0005 | 12 ms | 12 ms | 5.7 MB |
Score separation
Average similarity of the best relevant chunk and of the best irrelevant one. A wide gap makes a similarity threshold reliable; a narrow one means right and wrong chunks look alike to the model.
| Model | Relevant chunk | Best distractor | Gap |
|---|---|---|---|
| 1Voyage 4 Large | 0.602 | 0.508 | 0.094 |
| 2Qwen3 Embedding 8B | 0.777 | 0.703 | 0.074 |
| 3Cohere Embed v4 | 0.577 | 0.505 | 0.072 |
| 4Gemini Embedding 2 | 0.803 | 0.761 | 0.042 |
| 5Cohere Embed Multilingual v3 | 0.738 | 0.683 | 0.056 |
| 6OpenAI text-embedding-3-large | 0.610 | 0.547 | 0.064 |
| 7OpenAI text-embedding-3-small | 0.561 | 0.511 | 0.048 |
| 8BGE Base EN v1.5 | 0.719 | 0.738 | 0.013 |
Example questions
Rank of the first relevant chunk for each model; miss means it wasn't in the top 5. These are the questions where the models disagree most.
| Question | Voyage 4 Large | Qwen3 Embedding 8B | Cohere Embed v4 | Gemini Embedding 2 | Cohere Embed Multilingual v3 | OpenAI text-embedding-3-large | OpenAI text-embedding-3-small | BGE Base EN v1.5 |
|---|---|---|---|---|---|---|---|---|
كيف تعمل المتغيرات داخل قوالب ويكيبيديا، وما نوع البيانات التي يمكن تغييرها من مقال إلى آخر؟Expected passage## قالب بمتغيرات يمكن لقالب أن يضم متغيرات تتغير قيمها حسب الصفحة التي يوجد بها القالب. الشيء الذي يمكن مثلًا من استعمال جدول يضم معلومات تتغير حسب المقال، انظر مثلًا: قالب:Demography 4col {| border="0" width={{{1}}} cellspacing="2" cellpadding="2" | colspan="4" bgcolor="#eeeeee" | 1 | 1 | 4 | 2 | 5 | 2 | 3 | miss |
ما الحقول التي يمكن استخدامها للبحث في سجلات ويكيبيديا وتصفية نتائجها؟Expected passageنوع السجل المراد عرضه. بشكل افتراضي ، سيتم عرض جميع السجلات العامة، باستثناء سجلات الخفر والمراجعة والوسوم والشكر. يمكن تمكينها في "إظهار السجلات الإضافية" عند تحديد "جميع السجلات العامة". تم تعيين المثال لعرض سجل الحذف فقط، والذي يسرد عمليات حذف الصفحات واستعادتها. يتيح لك حقل ا | 1 | 3 | 5 | 2 | 1 | 1 | 1 | miss |
ما الظروف الصحية والمعيشية التي واجهها الأسرى الفلسطينيون خلال الإضراب؟Expected passageكما نقل الأسير مروان البرغوثي إلى المستشفى، إضافة إلى 120 أسيرا، وذلك بعد تدهور وضعهم الصحي. ومنذ 39 يوما، يرتدي الأسرى ملابس إدارة سجون الاحتلال (الشاباص)، ولا يملكون سوى أغطية قذرة وبالية، بعدما صادرت إدارة السجون مقتنياتهم منذ اليوم الأول من الإضراب، وفي زيارة أجراها محامي ناد | miss | 4 | 3 | 2 | 4 | 3 | miss | miss |
ما الهدف من القاموس الإلكتروني الخاص بلغة الأينو والجهات المشاركة في تطويره؟Expected passageالخميس 21 يوليو 2016 يسعى أحد القواميس الإلكترونية الناطقة الذي أطلق لأول مرة عام 2009 إلى المحافظة على لغة الأينو ونشرها؛ الأينو هي لغة السكان الأصليين في شمال شرق جزيرة هوكايدو اليابانية. وبحسب منظمة اليونسكو يوجد في اليابان حوالى ثمانى لغات مهددة بالانقراض، وتتصدر لغة الأينو ه | 1 | 1 | 1 | 3 | 1 | 1 | 2 | miss |
ما الدول الأوروبية الثماني التي طالبت إسرائيل بتعويضات عن منشآت هُدمت في المنطقة ج من الضفة الغربية؟Expected passageأعدت 8 دول أوروبية رسالة احتجاج تتضمن مطالبة الاحتلال "الإسرائيلي" بدفع تعويضات بقيمة 30 ألف يورو، مقابل مصادرة وتدمير مبان ومنشآت بنى تحتية أقامتها هذه الدول في المنطقة "ج" بالضفة الغربية. وبحسب ما نقلته صحيفة "هآرتس" العبرية عن دبلوماسي أوروبي كبير فإن الرسالة ستسلم إلى مسؤولين | 1 | 2 | 1 | 1 | 1 | 1 | 3 | miss |
ماذا تعني رسالة "توقيع خام غير صالح" في إعدادات توقيع ويكيبيديا؟Expected passage[[مستخدم:مثال|مثال مستخدم]]: [[نقاش المستخدم:مثال|نقاش]] سيؤدي ذلك إلى ظهور توقيع يبدو مثل: [[مستخدم:مثال|مثال مستخدم]]: [[نقاش المستخدم:مثال|نقاش]] (نقاش) بدلاً من مثال مستخدم: نقاش يرجع السبب في ذلك إلى أن نص ويكي (وترميز HTML) يجري الآن إفلاته ككيانات HTML عند وضعه في حقل التو | 1 | 1 | 1 | 1 | 5 | 1 | 3 | miss |
ما أبرز الأوامر الملكية السعودية التي ذُكرت باعتبارها ذات فائدة اقتصادية للمواطنين؟Expected passage«ترجمة:أصدر العاهل السعودي مجموعة من الأوامر الملكية التي تعود بالفائدة اقتصادياَ على الشعب السعودي، ومنها: مكافآت مالية وتحديد حد أدنى لأجور السعوديين وغيرها. من سلطان القاسمي.» «النص الأصلي:@Saudiwoman: 16billion to health ministry to build cancer units and other specialized un | 1 | 1 | 3 | 1 | 1 | 1 | 2 | miss |
ما أنواع الأحداث التي يمكن للمستخدم تلقي إشعارات عنها عبر البريد الإلكتروني في ويكيبيديا؟Expected passageتتوفر إشعارات البريد الإلكتروني للرسائل المتبقية على صفحة نقاش المستخدم الخاصة بك، عندما تتم الإشارة إليك أو شكرك، عندما يقوم شخص ما بربط أو مراجعة صفحة قمت بإنشائها، عندما يقوم شخص ما بتغيير صلاحيات المستخدم الخاصة بك، ومحاولات تسجيل الدخول الفاشلة، وعمليات تسجيل الدخول الناجحة | 1 | 2 | 2 | 1 | miss | 1 | 3 | miss |
Notes
- The corpus is 181 articles from Arabic Wikinews (ar.wikinews.org) and 52 help pages from Arabic Wikipedia (ar.wikipedia.org, Help namespace), both CC BY-SA 4.0, fetched as plain text on 14 November 2026. Pages were picked at random among those between 1,500 and 30,000 characters.
- All 100 questions are written in Modern Standard Arabic: 60 are about news articles and 40 about help pages. There are no dialect or Arabizi questions, so treat these results as an upper bound for dialect users.
- BGE Base EN v1.5 is English-only: it is included as a baseline.
- Arabic takes more tokens per character than English: these 1,000-character chunks average about 440 tokens. BGE Base EN v1.5 reads at most 380 tokens per chunk in Truvec and Cohere Embed Multilingual v3 at most 512, so both saw some chunks cut short.
- With 100 questions, a gap of a couple of points between two models is within the noise.
Methodology
Truvec split the documents into chunks, generated a golden set of questions with gpt-5.6-luna (each question is linked to the chunks that answer it), embedded the corpus and the questions with every model, and retrieved the top 5 chunks for each question by cosine similarity. Every model saw the same chunks and the same questions. Scores are averaged over all 100 questions.
- Hit rate
- Share of questions with at least one relevant chunk in the top k results. The most direct answer to "does the right passage reach the LLM?".
- Recall
- Share of the relevant chunks found in the top k, averaged over questions. Matters when an answer spans several chunks.
- MRR
- Mean reciprocal rank: 1 when the first relevant chunk is ranked first, 1/2 when second, 0 when missing. Rewards putting the answer at the top.
- nDCG
- Normalized discounted cumulative gain: how close the ranking of all relevant chunks is to the ideal ranking.
- Precision
- Share of the top k results that are relevant. Low precision means more noise sent to the LLM.
Downloads: benchmark.json - benchmark.csv - this post as Markdown
Voyage 4 Large won here, on these documents. Find the model that wins on yours: run the same benchmark on your documents with Truvec.