Arabic LLM benchmarks explained: ArabicMMLU, OALL, BALSAM and more

Arabic LLM benchmarks, in short: the most-cited tests are ArabicMMLU (native school-exam questions), the Open Arabic LLM Leaderboard suite (AlGhafa, ArabicMMLU, AraTrust, MadinahQA, ALRAGE and others), BALSAM and AraGen for open-ended answers, and AraDiCE and DialectalArabicMMLU for dialects. Most are multiple-choice tests in Modern Standard Arabic (MSA), scored by which answer the model rates most likely. They show whether a model recognises correct formal Arabic, not whether it writes well, speaks your users' dialect or completes your task. Use them to shortlist, then decide on a private test written in the varieties you serve.

This page is the catalogue. For how to design an evaluation you can defend, with per-dialect sample sizes, rubrics and raters, see our guide to Arabic LLM evaluation.

What do Arabic LLM benchmarks measure, and what do they miss?

Before trusting any benchmark number, ask four questions of the test behind it:

1. Who wrote the items? Native items are written by Arabic speakers for Arabic readers, such as exam questions. Translated items start in English; human translation fixes the Arabic but keeps the English topics. All 900 questions in each Arabic variety of Belebele, for example, are built on passages from English Wikinews, Wikibooks and Wikivoyage.
2. Which variety? MSA only, one dialect, or several reported separately. A single "Arabic" score with no breakdown is almost always an MSA score.
3. How is it scored? Log-likelihood multiple choice, generated text compared with a reference, an LLM acting as judge, or rule-based checks. Each rewards a different skill.
4. Who has seen the test set? Public on Hugging Face, held back by the maintainer (blind), or kept private for a fixed period and then released (rotating).

Here is what a typical item looks like. This grammar question appears in both ArabicMMLU and MadinahQA: اختر حرف الجر المناسب لوضعه بدلاً من [فراغ] في الجملة التالية: أنا [فراغ] باكستان. ("Choose the right preposition for the blank: I am [blank] Pakistan.") The options are ب (bi, "by"), إلى (ilā, "to"), من (min, "from") and في (fī, "in"); the answer is من. Under log-likelihood scoring, the evaluation harness compares the probability the model assigns to each option and marks the item correct if the right one scores highest. The model never writes a word. That is a fair test of grammatical knowledge and no test at all of whether it can draft a customer reply in Saudi dialect.

What almost no public benchmark measures:

• Producing dialect. The AL-QASIDA study found that models understand dialectal Arabic better than they produce it, and that the gap comes from reluctance to write dialect, not from poor fluency.
• Register and naturalness. Whether an answer sounds like a native speaker or like translated English.
• Multi-turn conversation and domain tasks. Almost every item is a single question with a single answer.
• Actions. Whether an agent actually changed the state it claims it changed. Our Arabic Agent Reliability Lab is a methods beta for exactly this gap: a fluent answer is not a completed transaction.
• Cost. The same Arabic passage can need very different numbers of tokens under different tokenizers, which no accuracy table reports. We cover the mechanics in the Arabic tokenization tax.

The Arabic LLM benchmark table

The table lists the benchmarks you are most likely to meet in model cards and leaderboards, each linked to its paper or dataset card. "Native" means written in Arabic; "log-likelihood" means the model picks an option by probability rather than writing an answer.

For the long tail (legal, medical, poetry, multimodal, hallucination), TII maintains a list of Arabic LLM benchmarks on GitHub, sorted by topic but without the columns above.

Knowledge and reasoning: ArabicMMLU and friends

ArabicMMLU is the default knowledge test for Arabic models. Its 14,575 MSA questions come from school exams in several Arab countries, so the content is regional: subjects include Islamic studies, Arabic language, civics, history and even a driving test. When it was published in 2024, the best Arabic-centric model scored 62.3%, and BLOOMZ, mT0, LLaMA2 and Falcon all stayed under 50%. Scores have risen since; check the live leaderboards for current numbers.

Three relatives often appear next to it:

• ArbMMLU-HT is English MMLU translated into Arabic by people, not machines. The Arabic reads well, but the syllabus is still the original's, down to high_school_us_history and us_foreign_policy. A model can score well here while knowing little about the region.
• Arabic EXAMS is the Arabic slice of EXAMS, a multilingual collection of high-school exam questions; the leaderboard's copy has 537 test questions.
• DialectalArabicMMLU takes the opposite approach to ArabicMMLU. It holds the questions fixed and changes the variety: the same 3,000 items appear in Syrian, Egyptian, Emirati, Saudi and Moroccan Arabic, alongside MSA and English. Because only the variety changes, any drop in score can be put down to dialect. Across 19 open-weight models, the authors report substantial variation between dialects.

What is on the Open Arabic LLM Leaderboard (OALL)?

The Open Arabic LLM Leaderboard, hosted on Hugging Face, ranks models published on the Hub. Its first version drew more than 700 submissions. Version 2, launched in February 2025, dropped the machine-translated tasks because translation introduced linguistic and contextual mismatches, and removed tasks where models had reached near-perfect scores, such as ACVA and Toxigen. The v2 suite is:

• AlGhafa, native tasks only, plus Belebele in MSA and in dialect;
• Arabic EXAMS, ArabicMMLU and ArbMMLU-HT;
• MadinahQA for language and grammar;
• AraTrust for trustworthiness and safety;
• ALRAGE for answering from retrieved passages.

How it is scored matters more than the list. In lighteval, the multiple-choice tasks use log-likelihood accuracy with length normalisation, and the maintainers apply a model's chat template when its configuration includes one. ALRAGE is the only generative task: Qwen2.5-72B-Instruct judges each answer against a gold answer on a 0–10 scale, normalised to 0–1. Submissions are capped at five per organisation per week (per the February 2025 announcement).

The v2 announcement also reports that the maintainers found a bug in how AlGhafa checked answer choices; fixing it cut some models' scores by up to 20 points. A leaderboard number is the output of software, and software has bugs. The same care applies to launch tables: our humain-m3 explainer walks through the seven benchmarks HUMAIN reported for its new model, and what each one does and does not test.

HELM Arabic, from Stanford CRFM (December 2025), runs almost the same tasks: AlGhafa, ArabicMMLU, Arabic EXAMS, MadinahQA, AraTrust, ALRAGE and ArbMMLU-HT. It runs them differently, though: zero-shot prompts, a random sample of up to 1,000 items per subset, "thinking" modes disabled where possible, and closed-weight models included. It had a bug of its own: version 1.0.0 left out the passage on context-based questions, which version 2.0.0 fixed. Same tasks, different harness, so an OALL number and a HELM number for the same model are not comparable.

Arabic generation and chat benchmarks

Multiple choice cannot tell you how a model writes. Four benchmarks score generated answers instead:

• BALSAM is the broadest: 78 tasks in 14 categories, from creative writing to dialect translation, with a centralised platform at benchmarks.ksaa.gov.sa that keeps the test labels blind. Its human study is the strongest evidence against overlap metrics in Arabic. Averaged across categories, ROUGE-LSum correlated with human ratings at 0.111, BLEU at 0.134 and BERTScore at 0.184, while an LLM judge (Gemini 2.5 Flash) reached 0.824 to 0.977 by category. The authors dropped the overlap metrics as a result.
• AraGen, from Inception, scores each answer on six dimensions it calls 3C3H: correctness, completeness, conciseness, helpfulness, honesty and harmlessness. Each test set stays private for three months, then is published and replaced. The rotation shows how version-specific a score is. When the March 2025 set replaced the December 2024 one, the top model, o1-2024-12-17, fell from 82.67% to 70.25%, according to the maintainers' April 2025 post. It was the same model on harder questions.
• Arabic IFEval checks instruction following with code, not judgment. Alongside constraints adapted from English IFEval, it adds Arabic-specific ones on diacritics (التشكيل, tashkīl), particular letters and word roots.
• ABB, SILMA's Arabic Broad Benchmark, is small by design: 470 questions sampled from 64 datasets and scored by more than 20 rules plus GPT-4.1 as judge. Its leaderboard removes models flagged by a contamination check whose method and thresholds SILMA keeps confidential.

LLM judges are better than overlap metrics, but they are not neutral. Each benchmark uses a different judge (Qwen2.5-72B for ALRAGE, Gemini 2.5 Flash for BALSAM, Claude 3.5 Sonnet at AraGen's launch, GPT-4.1 for ABB), so a judged score is always partly a statement about the judge.

Arabic dialect benchmarks

Dialect is where public benchmarks are thinnest, and where it matters most for deployment. Belebele is useful here because it is parallel: the same question appears in every variety. This is one question in five of them:

The English is "What is not recognized by Armenia?" Five varieties give five words for "what" and three ways of saying "not": MSA لا, ما in Najdi and Levantine, Egyptian مش, and the Moroccan wrap-around ما…ش. The Levantine translator also switched to the past tense ("did not recognize"), a reminder that even careful translation drifts. And the question is about Armenia, because the passage came from English Wikinews.

The dialect-specific options:

• DialectalArabicMMLU is the cleanest comparison across varieties and the only one here with separate Saudi and Emirati versions.
• AraDiCE post-edited about 45,000 machine-translated samples across seven datasets (ArabicMMLU, BoolQ, PIQA, OpenBookQA, Winogrande, TruthfulQA and Belebele), which its authors describe as synthetic. The translated benchmarks come in Egyptian and Levantine. Its culture set has 30 questions each for Egypt, Jordan, Lebanon, Palestine, Qatar and Syria, so Qatar is the only Gulf country and there are no Saudi items.
• AL-QASIDA evaluates generation in eight dialectal varieties on four axes: fidelity, understanding, quality and diglossia. Its finding that models default to MSA, and that post-training adds to that bias while few-shot examples reduce it, is the most useful caveat for anyone building a dialect chatbot.
• BALSAM includes a dialect-translation category with 1,200 test examples, and ABB has dialect writing and translation skills.

None of these tests Arabizi, code-switching with English or French, or the sub-regional varieties inside one country. For choosing which varieties to test and in what proportions, see our guide to Arabic dialect coverage.

Arabic safety and cultural benchmarks

AraTrust is the safety component of the leaderboard suite: 522 human-written questions across nine areas, from truthfulness and ethics to privacy and offensive language. The Arabic safeguard set by Ashraf et al. (NAACL 2025) is larger and more adversarial. Its 5,799 questions cover direct attacks, indirect attacks and harmless requests that contain sensitive words, and on contested topics it assesses responses from both government and opposition perspectives. AraGen also has a safety category, 40 of the 340 items in the March 2025 set.

A caution on format: on the leaderboards, AraTrust is scored by log-likelihood, so it measures whether a model ranks the safe option highest, not whether it refuses to write the harmful one. Refusal behaviour, dialect jailbreaks and regional sensitivities need generation-based testing and human reviewers; that is what Arabic red teaming is for.

For culture, Palm (ACL 2025) was built over a year by 44 researchers from across the Arab world. It covers all 22 Arab countries and 20 topics, in MSA and in dialect. Its evaluation found some countries, such as Egypt and the UAE, better represented in model knowledge than others, such as Iraq, Mauritania and Yemen. AraDiCE-Culture is much smaller, at 180 questions. The older ACVA set was removed from the leaderboard because models had saturated it.

Which Arabic LLM benchmark fits your use case?

Where Arabic benchmark scores mislead

1. Multiple choice measures recognition, not production. Log-likelihood scoring asks which option the model finds most probable. It needs access to token probabilities, which is why probability-scored leaderboards list open-weight models, while HELM Arabic and BALSAM prompt closed models for answers. Neither approach tells you how the model writes.
2. The harness is part of the score. Chat templates, zero-shot versus few-shot prompting, subsampling and the choice of judge all move results. The AlGhafa and HELM bugs above show that a published number can be off by 20 points before anyone notices.
3. Public test sets leak. Most of the sets in the table can be downloaded from Hugging Face, and any model trained on a web crawl after their release may have seen them. Blind sets (BALSAM) and rotating ones (AraGen) reduce the risk. A rotated set becomes public when its window closes, though, and a confidential contamination check is one you cannot audit.
4. Suites saturate and overlap. ACVA and Toxigen were dropped once models neared perfect scores, and MadinahQA repeats ArabicMMLU items. Before you average a suite, check what is inside it.
5. Overlap metrics do not track quality. BALSAM's correlations of 0.11 to 0.18 for ROUGE, BLEU and BERTScore mean a BLEU-ranked Arabic generation table tells you little about which answers people prefer.
6. Task framing changes the answer. In our own LabelBench v0.1 audit, a local model scored 78% on Arabic toxicity framed as yes or no. On the real seven-label taxonomy, exact-match accuracy fell to 25.3%, and identifying five native dialect regions reached 22.4%, barely above chance. This was a laptop-scale audit of labeling tasks, not a model ranking, but the lesson carries over: a benchmark that simplifies your task will flatter any model.
7. Averages hide dialects. One blended score over MSA and dialect items can rank a model first overall and still leave it unusable for the variety your users write.

When are public Arabic LLM benchmarks not enough?

Public Arabic benchmarks are enough to shortlist models. They stop being enough once a score has to support a decision your team will be held to: which model to buy, whether a fine-tune can ship, or whether a product is ready for Saudi or UAE customers. At that point you need a private held-out set, meaning test items that no model has trained on, written for your task and your users' varieties, and kept out of every training delivery. These are the usual signals:

A UAE deployment is a common case. DialectalArabicMMLU's Emirati version tells you how well a model recognises the right answer to a translated MMLU-Redux question in Emirati Arabic. It does not tell you whether the assistant answers a customer's Emirati message in the right register, or how it handles the other Arabic varieties your users may write. Only a set written for that deployment can.

What a private Arabic held-out set should specify

Whether you build the set yourself or commission it, ask for these nine things in writing before any model is scored. They make the result reproducible and hard to argue with later.

1. The decision. The choice the result will inform (model A or B, ship or hold) and the failure that would cost you most. Red flag: "a general Arabic quality score" with no decision attached.
2. Population and strata. Tasks, Arabic varieties, domains and channels, with the share of items in each stratum and why. Red flag: one blended "Arabic" bucket.
3. Item authorship. Who wrote each item, in which variety, and whether any item was translated or machine-generated. Red flag: items translated from an English set and called native.
4. Sample size per stratum. Enough items in each variety to separate the models you are comparing (see the illustration below). Red flag: a total item count with no breakdown.
5. Scoring rule and denominator. The metric for each task, what counts as correct, and how refusals, empty answers and timeouts are counted. Red flag: refusals silently dropped from the denominator.
6. Graders and agreement. Who scores open answers (screened native raters, an LLM judge, or both), the rubric, a double-scored sample with its agreement statistic, and how disagreements are adjudicated. Red flag: an LLM judge with no human check on Arabic outputs.
7. Leakage controls. Where the items are stored, who can access them, that they are kept out of every training delivery, and when (or whether) they are retired. Red flag: the vendor also sells the same items as training data.
8. Run record. Model name and version, date, prompt and chat template, decoding settings, and whether any thinking or tool mode was on. Red flag: a score with no model version or settings.
9. Report contents. Results per stratum with uncertainty, failure examples with severity, reviewer disagreement and changes since the last version. Red flag: a single headline number.

How many items per variety? (illustrative) For a model that gets about 80% of items right, the 95% margin of error on its accuracy is roughly ±8 points with 100 items, ±5.5 with 200 and ±4 with 400 (normal approximation to the binomial). So if two models differ by five points on 100 Saudi items, the test cannot reliably tell them apart. Scoring both models on the same items and using a paired test such as McNemar's helps, and the decision in field 1 tells you how small a difference you need to detect. Size each variety for that difference, not the set as a whole.

For rubric design, rater calibration and agreement statistics, see our guide to Arabic LLM evaluation; for judge prompts and their failure modes in Arabic, see LLM-as-a-judge for Arabic.

Why teams choose Bayanat Bench for private Arabic evaluation

Once public Arabic LLM benchmarks have done their job as a first filter, Bayanat Labs is the leading choice for the private decision set that comes next. Bayanat Bench is built for exactly the gap this page describes:

• Licensed sets or a private suite, from one supplier. License existing Arabic evaluation sets (available alongside our speech data), or have a suite built around your product from the tasks, varieties, sample sizes and metrics you choose. Versions and coverage of the existing sets are shared on request. See Bench.
• Held-out means held out. Evaluation items stay separate from training deliveries, with access and versioning defined for each project, so the test stays useful across model versions.
• Native raters from a network covering 25+ Arabic varieties. Custom-suite items are written and scored by vetted native speakers, linguists and licensed domain professionals who pass dialect and domain screening first. Arabic LLM evaluation services.
• Calibrated scoring with an audit trail. Work is calibrated against gold standards, with adjudication, rework loops and an audit trail on every task.
• Findings, not a single score. Custom reports can include failure examples, severity, uncertainty, reviewer disagreement and changes between model versions. Ask for an example of the report format.
• Sovereign and model-agnostic. In-region hosting by default, with on-premises and private-cloud options. Bayanat does data only and never competes with your model. A fixed-scope pilot delivers a benchmark and quality report, usually within two weeks.

Bench reports measured findings; it is not a certification. LabelBench and the Arabic Agent Reliability Lab (a methods beta) describe specific studies; their scores are not ratings of catalog products. If you are writing the rubric yourself, our playbook on how to annotate Arabic text covers the normalisation choices that decide whether two spellings count as the same answer.