Arabic LLM evaluation: how to benchmark a model you can actually trust

Every Arabic model announcement arrives with a benchmark table, and every benchmark table tells the same comfortable story: the model is competitive. (For a worked reading of one such table, see our explainer on the benchmarks reported for humain-m3.) Then the model meets production traffic — dialectal, code-switched, culturally loaded — and support tickets tell a different story. The gap is rarely the model's fault alone. It is the evaluation's.

Arabic LLM evaluation you can trust does four things differently. It tests on prompts written natively in the varieties your users speak, not translated from English. It scores five dimensions separately for each dialect instead of one blended number. It sizes every dialect sample so the confidence interval is narrower than the decision it informs. And it puts native raters wherever judgment is the thing being measured. This guide sets out the protocol, a plan template, a sample rubric and sample-size numbers. For the public test sets themselves, see Arabic LLM benchmarks explained. The same protocol works for Arabic AI evaluation beyond chat models: voice agents, RAG assistants and tool-using agents.

Four ways Arabic benchmarks mislead

1. Translated benchmarks measure translation, not Arabic

Much of the Arabic evaluation stack began as machine translation of English suites. Translation carries over English framing, references and idiom, producing "translationese" that no Arabic speaker would write, so the eval rewards models that are good at translated English rather than good at Arabic. Culturally specific questions arrive intact: a model can ace translated trivia about American institutions while knowing little about the region it will serve. The maintainers of the Open Arabic LLM Leaderboard reached the same conclusion: version 2 (February 2025) dropped its machine-translated tasks, citing lower quality and possible cultural bias. Natively sourced sets exist — ArabicMMLU draws 14,575 questions from school exams across North Africa, the Levant and the Gulf — but they are still MSA multiple choice. Our catalogue of Arabic LLM benchmarks covers ArabicMMLU, OALL, BALSAM and the other public tests one by one; this guide is about building an evaluation you can trust on top of them.

2. MSA-only evaluation tests the register your users don't speak

Arabic is diglossic: formal writing happens in Modern Standard Arabic, but everyday speech and chat happen in dialect. An evaluation built from clean MSA prompts leaves the model untested on the inputs it will actually receive, which is why dialect coverage has to be named variety by variety. When researchers do break dialect out, scores fall (Figure 1). A single "Arabic" score with no dialect breakdown should be treated as an MSA score.

3. Aggregates hide the failures that matter

Averaging across tasks and varieties produces one flattering number. But a model that is strong in Egyptian and weak in Gulf is not "moderately good" — it is unusable for a Saudi deployment. Scores must be reported per variety and per capability, or they conceal exactly the risk you are trying to measure.

4. Orthography and tokenization skew automatic metrics

Arabic writing tolerates surface variation that exact-match metrics punish arbitrarily: إلى and الى ("to"), مدرسة and مدرسه ("school"), with or without diacritics. Toolkits such as CAMeL Tools ship separate normalizers for alef, alef maksura and taa marbuta variants because the choice changes the result; declare yours before scoring and apply it identically to every model. Tokenization adds a second distortion: the same text can need very different numbers of tokens in different languages (Petrov et al., 2023), which quietly shifts context budgets, latency and cost comparisons between models. We cover the mechanics in the Arabic tokenization tax.

What to measure instead: five dimensions of Arabic LLM evaluation

Report each dimension separately, per dialect and per domain — never as one blended number:

1. Dialect comprehension. Does the model read your markets' varieties, including code-switching and Arabizi, or only MSA? The same question arrives as وش تبي؟ (Gulf), عايز إيه؟ (Egyptian) and شو بدك؟ (Levantine), all "what do you want?".
2. Generation register. Are answers natural for the audience: right variety, right formality, free of translationese? Understanding and producing dialect are separate skills; AL-QASIDA found that models produce dialect less well than they understand it, largely because they are reluctant to generate it.
3. Factuality with regional grounding. Accuracy on the entities, institutions, regulation and calendars of the deployment region — not translated general knowledge.
4. Cultural and safety alignment. Handling of religious content, social norms and sensitive regional topics, including over-refusal of benign questions. Safety established in English or MSA does not automatically transfer: a TrustNLP 2026 study reports safety behavior degrading as varieties diverge from MSA. Adversarial probing belongs in Arabic AI red teaming.
5. Domain task performance. The actual jobs — summarizing a contract, extracting a KYC field, triaging a symptom description — measured on your own task distribution and judged by people qualified in the domain.

A protocol that holds up: eight steps

The mechanics matter as much as the dimensions. Run the steps in order; a skipped step usually surfaces later as a result nobody can defend.

1. Write down the decision. Model selection, launch sign-off or fine-tune regression — and the pass threshold for each dimension, fixed before anyone sees a model output. An evaluation with no decision attached turns into a leaderboard.
2. Map traffic to cells. List the varieties and domains you will serve and their share of traffic. Each variety × dimension pair is a cell, and each cell needs its own sample (see the sample sizes below). If prompts are drawn from production logs, treat them as personal data (PDPL and data residency).
3. Author prompts natively. Native speakers write the prompts in each target variety, never translated from English, including code-switching, Arabizi and the untidy spellings real users type.
4. Write rubrics and gold anchors. Each dimension gets a written scale with two or three scored example responses per level, plus critical-error flags that fail a response outright.
5. Calibrate and pilot. Raters who speak the variety they judge score a pilot slice; measure agreement per dimension and rewrite ambiguous rubric wording before scaling. If agreement is low, the eval is measuring the raters, not the model. The same discipline applies to any labeling project; see how to annotate Arabic text.
6. Run blind and randomized. Hide model identities, randomize output order and seed hidden gold items to catch rater drift. Check contamination: a model trained on a benchmark's test split scores better than it should (Sainz et al., 2023), which rules out public items for anything decisive.
7. Analyze with error bars. Report every cell with a confidence interval. When comparing two models, test the question-level paired differences rather than the two averages, and use clustered standard errors when prompts come in related groups (Miller, 2024).
8. Freeze, then refresh. Keep a private held-out split frozen for regression runs on every checkpoint, and rotate fresh items in over time so scores keep meaning something.

None of this is exotic. It is the discipline the evaluation literature already recommends for English, applied with people who actually speak the varieties being measured. The expensive part is the human judgment. For a diglossic language, it is also the part that cannot be skipped.

An Arabic LLM evaluation plan template

Fill this in before anyone writes a prompt. The example column is illustrative: a hypothetical Saudi retail-banking assistant choosing between two candidate models.

Sample rubric: generation register

Each dimension gets its own scale. Here is an illustrative four-point register rubric for the same assistant, whose style guide asks for polite, plain Arabic in the user's variety whenever the user writes in dialect.

Attach two or three real, scored responses to each level as anchors and keep them visible to raters during calibration. Anchors make a scale concrete in a way that descriptors alone rarely do.

How many test items per dialect?

Enough that each cell's confidence interval is narrower than the difference you need to see. The table gives the approximate 95% margin of error for a pass rate at common sample sizes, using the normal approximation ±1.96 × √(p(1−p)/n). These figures are arithmetic, not results from any evaluation.

Three rules follow.

• Size per dialect, not in total. A 1,000-prompt set split 850 MSA and 50 each across three dialects leaves a margin of roughly ±13 points on every dialect — too wide to decide a dialect launch.
• Comparisons need more items than estimates do. Illustratively, telling a 70% model from a 60% model with independent samples, 80% power and a 5% significance level takes roughly 350 items per model per cell; a 5-point gap takes about 1,400. Scoring both models on the same items and analyzing the paired differences usually needs fewer, because item difficulty cancels out (Miller, 2024).
• Correlated items count for less. Ten paraphrases of one intent are not ten independent observations. Cluster by source prompt or conversation and use clustered standard errors, or the interval will look tighter than it is. For small cells or rates near 0% or 100%, use a Wilson interval; LabelBench uses the 95% Wilson lower bound to decide how much Arabic labeling can safely leave the human queue.

Human raters, LLM judges and when to use each

Automate what is checkable; keep people where judgment is the measurement. In Arabic, more of quality rests on register and cultural fit, which pushes that line toward human raters.

The evidence supports a hybrid. In English, strong LLM judges agreed with human preferences more than 80% of the time, yet showed position, verbosity and self-enhancement biases (Zheng et al., 2023). For Arabic cultural knowledge, a 2026 study comparing five frontier LLM judges with native-speaker expert grades on Egyptian and Iraqi prompts found four of the five systematically lenient, with implicit cultural reasoning the main failure mode for automated grading (Abdoli et al.). Its set is small, 103 prompt-rubric pairs, so treat it as a warning rather than a measurement. Our LabelBench audit points the same way for automated Arabic labeling, a related but different task: a model at 78% on binary Arabic toxicity fell to 25.3% on the real multi-label taxonomy and 22.4% on five-region dialect identification.

The workable recipe: have native raters score a slice of your own data, measure the judge's agreement with them per dimension and dialect, automate only where agreement meets your threshold, and route disagreements and critical categories back to people. Re-check the calibration whenever the judge model, the rubric or the traffic mix changes.

Evaluating agents and RAG systems

Agents: check the state, not the sentence. A fluent answer is not a completed transaction. Grade the tool calls and the final system state against a declared goal, the way τ-bench compares the database at the end of a conversation with an annotated goal state. Run each task several times: τ-bench's pass^k metric exposed how inconsistent agents are, with gpt-4o succeeding on fewer than half of tasks and scoring pass^8 below 25% in the retail domain (June 2024 paper). Language adds its own penalty: tool-calling accuracy drops by an average of 5–10% when users interact in Arabic, whether the tool descriptions are in Arabic or English (Kubrak et al., 2026).

Our Arabic Agent Reliability Lab, a methods beta, applies state-based checks with evidence receipts. In one receipt, an agent reported that it had cancelled an order when it had only looked the order up; the order stayed packed. That is a development result on an open synthetic suite, not an accuracy score or a certification, but it is exactly the kind of failure a transcript-only review can miss.

RAG: score retrieval and generation separately. A wrong answer can come from the retriever fetching the wrong passage or from the generator ignoring the right one, and the fixes differ. Ragas splits this into context relevance, faithfulness to the retrieved passages and answer relevance. For Arabic, add the cases that break retrieval: dialect queries against MSA documents, spelling variants of the same name, and Arabizi. Have native raters audit a sample of the faithfulness judgments, because a checker that misreads dialect will misjudge whether an answer is grounded. The Open Arabic LLM Leaderboard's ALRAGE task, built from 40 Arabic books and scored by an LLM judge against gold answers, is a useful public reference point, not a substitute for your own documents. For a collection built around those documents, Bayanat Search pairs Arabic queries with relevance judgments, evidence-linked answers and unanswerable cases.

Retrieval miss or unsupported answer? A diagnosis table

When a RAG answer is wrong, first check whether retrieval returned a passage that actually answers the question. That one check tells you which half of the system to fix. Grade each retrieved passage (3 answers the question, 2 answers part of it, 1 is on topic but does not answer, 0 is not relevant), then compare the answer with the best passage.

Our Arabic Retrieval Starter Kit (synthetic, CC BY 4.0) has a test case for each row. A customer asks in Gulf Arabic «أبي أرجّع الجوال اللي اشتريته قبل أسبوع، أقدر؟» ("I want to return the phone I bought a week ago, can I?"). The current individual-customer policy, in Arabic and in English, is graded 3: returns within 14 days of delivery, in the original packaging with the invoice. Two near-identical passages are graded 0 as hard negatives: last year's version (7 days from purchase, opened boxes accepted, a 10% restocking fee) and the business-customer policy (30 days with an approved purchase order). An answer that gives the 7-day rule is wrong even if retrieval also found the current policy. Another question asks about a device bought from a different online store, which no policy covers; the right behaviour is to say so and ask where the device was bought or route the customer to support, not to apply the 14-day rule.

For each test question, record the query ID, language and variety; the corpus version; for each judged passage its ID, version, effective date, status and customer group, with the grade and the reason (including why a hard negative is wrong); and the expected answer with an exact evidence quote, the passages it must not use, and whether the question can be answered at all. Score retrieval against the judgments with a standard tool such as trec_eval (for example nDCG@10, and Recall@5 counting only grades 2 and 3, which is -l 2 in trec_eval; report unanswerable questions separately). Score answers separately: agreement with the evidence, use of passages they must not use, and declining when the sources do not answer. Keep these test questions out of any training data. To build the same judgments on your own documents, see Bayanat Search; licensable Arabic text for training is in Bayanat Knowledge.

Sampling queries and checking reviewer agreement

Relevance judgments are only as good as the queries they cover and the people who grade them. Two steps decide both.

Sample queries the way users ask them. Draw test queries from real search or chat logs after removing personal data, and stratify before you sample: by variety (MSA, Saudi or other Gulf dialect, mixed Arabic-English, Arabizi), by query type (looking up a fact, asking about a policy or rule, asking how to do something) and by frequency. Sample distinct queries, not only traffic, or the long tail of rare and misspelled queries disappears behind a few popular ones. Add questions your documents cannot answer, and size every stratum you will report using the margins in the sample-size table above. Then choose which passages get judged: pool the top results from more than one retriever, for example a keyword (BM25) system and a dense one, so the judgments do not favour whichever system built the candidate list. Search the corpus for hard negatives on purpose: superseded versions, look-alike rules for another customer group, and passages that share the query's words but not its answer.

Double-judge a sample and measure agreement. Two native reviewers grade a random sample of query-passage pairs, stratified like the full set, each without seeing the other's grade. The 0–3 grades are ordered, so use a statistic that treats a 2-versus-3 split as a smaller disagreement than 0-versus-3: weighted Cohen's kappa for two reviewers (scikit-learn's cohen_kappa_score with weights="quadratic"), or Krippendorff's alpha at the ordinal level for more reviewers or missing grades. A common convention treats alpha of 0.80 or above as satisfactory and 0.67 to 0.79 as enough only for tentative conclusions (k-alpha.org). Also report agreement at the line your metric uses, relevant (grades 2 and 3) versus not, because that is what Recall@5 counts, and report it per variety: if reviewers agree on MSA queries and split on Gulf dialect, the dialect scores are not yet usable. When reviewers disagree, a third reviewer adjudicates, the reason goes into the record and the guideline is clarified; if agreement is below your threshold, revise the guideline and re-judge before scoring any system. If an LLM grades passages, hold it to the same test against the native reviewers' grades.

As a public reference, MIRACL includes Arabic among its 18 languages, and all of its relevance assessments were made by native speakers. Its queries were written by native speakers from Wikipedia prompts and its passages are Wikipedia text, so it measures general retrieval over encyclopedic text, not your users' dialect queries or your documents. The starter kit's 30 judgments were written by one author together with the fixture, so it reports no agreement figure: treat its grades as a worked example and measure agreement on your own double-judged set. To see the same work on your own documents, Bayanat Search shares sample judgments and reviewer reasoning, including disagreements, for you to review before production.

This is how we approach Arabic LLM evaluation services at Bayanat Labs: natively authored, dialect-stratified prompts; raters who pass dialect and domain screening and are calibrated against gold standards, with adjudication; blind comparisons; and per-dialect reports with an audit trail on every task. We do data only and are model-agnostic, so we have no model of our own to favor. A fixed-scope pilot with gold-standard QA and a quality report is usually delivered within two weeks. If your team runs the evaluation, Bayanat Bench covers licensable evaluation modules and customer-specific suites; agree the held-out scope and access rules before use.