The best embedding model for legal documents: contracts, clauses and regulations
Legal text is hard for embedding models: long clauses, defined terms, cross-references and near-identical boilerplate. What makes legal retrieval different, how to chunk and test it, and a benchmark on contracts and regulations.
Legal RAG has to be precise. A lawyer asking whether a contract can be terminated without cause needs the termination clause of that contract, not a similar clause from another agreement and not the definition of "Term". Embedding models built and benchmarked on web text struggle with exactly what makes legal documents legal.
This post covers what makes legal retrieval hard, how to prepare legal documents for embedding, and how to test models on them. The last section is a benchmark on public contracts and regulations.
Why legal text is hard for embeddings
Boilerplate everywhere. Contracts reuse the same clauses with small changes: a liability cap of 12 months of fees in one agreement, of the total contract value in another. Embeddings place near-identical clauses next to each other, so the right clause competes with many convincing look-alikes. These are the hardest distractors there are.
Meaning hides in small words. "Shall", "may", "unless", "except as provided in Section 9" change the meaning of a clause completely, but barely change its embedding. A model can retrieve the right topic and still return the clause that says the opposite.
Defined terms. "Confidential Information", "Services" and "Affiliate" are defined once, then used everywhere with their capital letter and their precise meaning. Questions use everyday words ("secret data", "the work", "sister company"), so the model has to connect the two.
Cross-references. "Subject to Section 12.3" sends the reader elsewhere. A chunk containing that reference is incomplete on its own, and a question about Section 12.3 by number needs exact matching, which embeddings are bad at.
Length. Clauses run long, and regulations nest articles, paragraphs and points. Short-input models (512 tokens) may truncate a clause before the part that matters.
Prepare the documents
- Chunk by structure, not by size. Split on articles, sections and clauses. A chunk should be a complete clause with its heading ("12. Limitation of Liability"), so the heading's words help retrieval.
- Use larger chunks than for prose. Legal clauses need their full text to be understood. Test chunk sizes around 1,500 to 2,500 characters, and check which models truncate them.
- Keep identity in metadata. Store the document, party, date and jurisdiction with each chunk, and filter on them. No embedding model reliably tells two versions of the same NDA apart.
- Add keyword search for references. Questions like "what does Section 14.2 say?" or "Article 33 GDPR" are best answered by exact matching. Hybrid search (keywords plus embeddings) covers both.
Test the models on what matters
A legal golden set should include:
- questions phrased in plain language about defined terms;
- questions whose answer is an exception ("can the supplier subcontract?" when the rule is "no, except with written consent");
- questions that only one of several similar contracts answers;
- a few questions where the right answer is "this contract doesn't say".
Look at precision as well as hit rate: in legal work, a wrong clause in the context can be worse than a missing one, because the LLM may cite it with confidence.
Confidentiality and hosting
Legal documents are often confidential, and sending them to an embedding API may be ruled out by a client agreement or a regulator. Open-weight models (Qwen3 Embedding, BGE, GTE, mxbai, Nomic) can run in your own infrastructure. Include at least one of them in your comparison: you'll see exactly what self-hosting costs you in retrieval quality, if anything.
What to take from the benchmark
The benchmark below uses public contracts, terms of service and regulations. Your documents will differ: a firm's M&A contracts are not consumer terms of service. Look at the gaps between models, the score separation (how clearly each model sets the right clause apart from look-alikes), and the example questions. Then run the same comparison on a sample of your own documents before you choose.
Real benchmark - generated by Truvec
Benchmark: Embedding models on contracts and regulations
- Documents
- 58
- Chunks
- 5,024
- Chunking
- recursive, 1000 chars, 200 overlap
- Golden set
- 100 questions, human-reviewed
- Models
- 8 compared
- Top k
- 5
- Run
- 2026-11-14
Download the full report (PDF) Every question, with the passages each model retrieved.
Key results
- Voyage 4 Large ranks first, with an MRR of 0.828 and a hit rate of 94.0% in the top 5.
- One model is within 0.02 MRR of the winner (Voyage 4): on this corpus they are practically tied.
- Hit rate ranges from 77.0% to 94.0% across the 8 models.
- Cheapest model within 2 points of the winner's hit rate: Voyage 4, $0.05 to embed this corpus against $0.09.
- Fastest to embed a query: OpenAI text-embedding-3-small (6.0 ms per query, measured in batch from our servers).
Retrieval quality
| Model | Hit rate@5 | Recall@5 | MRR | nDCG@5 | Precision@5 |
|---|---|---|---|---|---|
| 1Voyage 4 Large | 94.0% | 94.0% | 0.856 | 19.2% | |
| 2Voyage 4 | 92.0% | 91.5% | 0.838 | 18.6% | |
| 3Gemini Embedding 2 | 88.0% | 87.5% | 0.773 | 17.8% | |
| 4Cohere Embed v4 | 84.0% | 83.5% | 0.757 | 17.0% | |
| 5Qwen3 Embedding 8B | 86.0% | 86.0% | 0.744 | 17.6% | |
| 6OpenAI text-embedding-3-large | 83.0% | 83.0% | 0.729 | 17.0% | |
| 7OpenAI text-embedding-3-small | 77.0% | 76.5% | 0.680 | 15.6% | |
| 8BGE Base EN v1.5 | 78.0% | 77.5% | 0.654 | 15.8% |
Hit rate by position
How many questions the first results already answer. A model that finds the answer at position 1 lets you send fewer chunks to the LLM.
| Model | Top 1 | Top 2 | Top 3 | Top 4 | Top 5 |
|---|---|---|---|---|---|
| 1Voyage 4 Large | 75.0% | 87.0% | 89.0% | 92.0% | 94.0% |
| 2Voyage 4 | 73.0% | 88.0% | 89.0% | 90.0% | 92.0% |
| 3Gemini Embedding 2 | 65.0% | 77.0% | 84.0% | 85.0% | 88.0% |
| 4Cohere Embed v4 | 67.0% | 75.0% | 79.0% | 81.0% | 84.0% |
| 5Qwen3 Embedding 8B | 61.0% | 71.0% | 81.0% | 85.0% | 86.0% |
| 6OpenAI text-embedding-3-large | 60.0% | 75.0% | 78.0% | 82.0% | 83.0% |
| 7OpenAI text-embedding-3-small | 57.0% | 70.0% | 74.0% | 77.0% | 77.0% |
| 8BGE Base EN v1.5 | 52.0% | 65.0% | 68.0% | 75.0% | 78.0% |
Cost, speed and size
Embedding prices are the providers' list prices. Corpus cost is a one-time cost per model; the vector size is raw float32 storage, before the vector database's own overhead.
| Model | Price / 1M tokens | Embed the corpus | 1,000 queries | Query embedding | Search p95 | Vectors |
|---|---|---|---|---|---|---|
| 1Voyage 4 Large | $0.12 | $0.09 | $0.0026 | 13 ms | 52 ms | 19.6 MB |
| 2Voyage 4 | $0.06 | $0.05 | $0.0013 | 19 ms | 23 ms | 19.6 MB |
| 3Gemini Embedding 2 | $0.20 | $0.15 | $0.0043 | 19 ms | 44 ms | 58.9 MB |
| 4Cohere Embed v4 | $0.12 | $0.09 | $0.0026 | 13 ms | 27 ms | 29.4 MB |
| 5Qwen3 Embedding 8B | $0.10 | $0.08 | $0.0021 | 166 ms | 72 ms | 78.5 MB |
| 6OpenAI text-embedding-3-large | $0.13 | $0.10 | $0.0028 | 6.6 ms | 20 ms | 58.9 MB |
| 7OpenAI text-embedding-3-small | $0.02 | $0.02 | $0.0004 | 6.0 ms | 32 ms | 29.4 MB |
| 8BGE Base EN v1.5 | $0.0080 | $0.0062 | $0.0002 | 14 ms | 23 ms | 14.7 MB |
Score separation
Average similarity of the best relevant chunk and of the best irrelevant one. A wide gap makes a similarity threshold reliable; a narrow one means right and wrong chunks look alike to the model.
| Model | Relevant chunk | Best distractor | Gap |
|---|---|---|---|
| 1Voyage 4 Large | 0.576 | 0.506 | 0.068 |
| 2Voyage 4 | 0.628 | 0.561 | 0.066 |
| 3Gemini Embedding 2 | 0.807 | 0.783 | 0.024 |
| 4Cohere Embed v4 | 0.668 | 0.621 | 0.047 |
| 5Qwen3 Embedding 8B | 0.781 | 0.746 | 0.033 |
| 6OpenAI text-embedding-3-large | 0.680 | 0.634 | 0.049 |
| 7OpenAI text-embedding-3-small | 0.712 | 0.671 | 0.040 |
| 8BGE Base EN v1.5 | 0.761 | 0.744 | 0.020 |
Example questions
Rank of the first relevant chunk for each model; miss means it wasn't in the top 5. These are the questions where the models disagree most.
| Question | Voyage 4 Large | Voyage 4 | Gemini Embedding 2 | Cohere Embed v4 | Qwen3 Embedding 8B | OpenAI text-embedding-3-large | OpenAI text-embedding-3-small | BGE Base EN v1.5 |
|---|---|---|---|---|---|---|---|---|
Under the Whitesmoke promotion and distribution agreement, what restrictions apply to the Distributor's treatment of products already installed by end users after the agreement ends?Expected passage3.8 [ * ]. During the Term and for a period of [ * ] following the expiration or termination of this Agreement, Distributor shall not, and shall not engage any third party to: (a) restrict, modify, or reconfigure in any manner any of the Products that have been installed by End U | 1 | 1 | 2 | 3 | 2 | miss | 4 | 4 |
What responsibilities apply to anyone collecting personal information through the GitHub Service?Expected passageYour use of information from the Service must comply with the GitHub Privacy Statement. ## 8. Privacy Misuse of personal information is prohibited. Any person, entity, or service collecting data from the Service must comply with the GitHub Privacy Statement, particularly in regar | 2 | 2 | 3 | 1 | 2 | miss | miss | 4 |
Under the video-on-demand content license between Rogers and EuroMedia, when may a party disclose the other's confidential information to comply with legal requirements, and what must it do beforehand?Expected passageto the greatest extent possible. (b) Each party agrees to use the other party's confidential information solely for the purpose of fulfilling its obligations under this Agreement, to hold the other party's confidential information in confidence and to protect the confidentiality | 5 | 2 | 4 | 5 | 3 | miss | 4 | 4 |
Under GitHub's Corporate Terms of Service, when can GitHub assign the agreement without the customer's consent?Expected passage### 3. Non-Assignability Neither Party may assign or otherwise transfer this Agreement, in whole or in part, without the other Party's prior written consent, such consent not to be unreasonably withheld, and any attempt to do so will be null and void, except that GitHub may assig | 1 | 1 | 1 | 1 | 3 | 2 | miss | 4 |
What kinds of conduct are prohibited at the GitHub event?Expected passageUnacceptable behavior includes, but is not limited to: * The use of sexualized language or imagery * Inappropriate physical contact, sexual attention, or advances * Trolling, insulting or derogatory comments or behavior * Personal or political attacks toward Event Participants * | 1 | 1 | 2 | 2 | 3 | 1 | 1 | 5 |
What extra charges does a sponsor pay under the program, including when using a credit card?Expected passage### 2.2. Fees. An additional service fee ("Service Fee") of 3% per transaction will be deducted from Sponsor's account to cover GitHub's administrative costs associated with offering the Program. If payment is made via credit card, an additional 3% fee will be applied. ### 2.3. T | 1 | 1 | 1 | 1 | miss | 4 | miss | 3 |
What uses of GitHub-hosted Actions are not allowed?Expected passage* The provision of a stand-alone or integrated application or service offering the Actions product or service, or any elements of the Actions product or service, for commercial purposes; * Any activity that places a burden on our servers, where that burden is disproportionate to | 1 | 1 | 1 | 2 | 3 | 2 | 2 | miss |
Do I need GitHub's permission to use its trademarks or logos?Expected passage## GitHub's Logos The license grant included with the Software is not for GitHub's trademarks, which include the Software logo designs. GitHub reserves all trademark and copyright rights in and to all GitHub trademarks. GitHub's logos include, for instance, the stylized designs t | 1 | 1 | 3 | 1 | 4 | 2 | 1 | 1 |
Notes
- The corpus has three parts. 36 commercial contracts from the CUAD dataset (The Atticus Project, CC BY 4.0, text extracted from SEC EDGAR filings), picked across contract types among those between 25,000 and 90,000 characters. 19 terms, agreements and policies from github.com/github/site-policy (CC0). The English texts of Regulation (EU) 2016/679 (GDPR), Regulation (EU) 2022/2065 (Digital Services Act) and Regulation (EU) 2024/1689 (AI Act), from the Publications Office of the EU.
- The contract texts are used as published in CUAD: page numbers, redaction marks and layout left over from the filings are still in the chunks.
- Reviewing the questions, we rewrote 27 that did not say which document they were about ("Which state’s law governs the agreement?") to name the contract or its parties, the way a lawyer would ask. The name is often absent from the chunk that holds the clause, which is part of what makes legal retrieval hard.
- With 100 questions, a gap of a couple of points between two models is within the noise.
Methodology
Truvec split the documents into chunks, generated a golden set of questions with gpt-5.6-luna (each question is linked to the chunks that answer it), embedded the corpus and the questions with every model, and retrieved the top 5 chunks for each question by cosine similarity. Every model saw the same chunks and the same questions. Scores are averaged over all 100 questions.
- Hit rate
- Share of questions with at least one relevant chunk in the top k results. The most direct answer to "does the right passage reach the LLM?".
- Recall
- Share of the relevant chunks found in the top k, averaged over questions. Matters when an answer spans several chunks.
- MRR
- Mean reciprocal rank: 1 when the first relevant chunk is ranked first, 1/2 when second, 0 when missing. Rewards putting the answer at the top.
- nDCG
- Normalized discounted cumulative gain: how close the ranking of all relevant chunks is to the ideal ranking.
- Precision
- Share of the top k results that are relevant. Low precision means more noise sent to the LLM.
Downloads: full report (PDF) - benchmark.json - benchmark.csv - this post as Markdown
Voyage 4 Large won here, on these documents. Find the model that wins on yours: run the same benchmark on your documents with Truvec.