Arabic annotation is not binary.

pip install bayanat-labelbench

“Can an AI label Arabic data?” is the wrong question. The useful question is: which taxonomy, which variety, at what proven quality floor, and with how much human review? LabelBench turns that into a test.

The annotation complexity cliff

We ran one fixed zero-shot rubric through Qwen 2.5 3B and Arabic-specialized Fanar 1 9B, then compared both with tiny supervised character models on identical held-out samples. The diagnostics are intentionally unsophisticated. They answer a crucial control question: is the test signal learnable at all?

This is not a bad dataset

A balanced five-class guesser scores 20%. Qwen 2.5 3B reaches 22.4%; Qwen 2 7B reaches 22.6%; Llama 3 8B reaches 24.6%. The Arabic-focused Fanar 1 9B materially improves to 47.8%, showing that regional specialization matters. Yet a character n-gram model trained on just 100 disjoint examples per region reaches 69.8%. Native, task-specific data still matters more.

The model does not merely “get some dialects wrong”

It collapses toward high-salience varieties. Qwen 2.5 labels 85% of Egyptian examples correctly—but also calls 75% of Gulf, 70% of Iraqi, 80% of Levantine and 70% of Maghrebi examples Egyptian.

Accuracy is not the automation decision

LabelBench's primary production diagnostic is Safe Automation Coverage: the fraction of a queue that can be accepted while the 95% Wilson lower confidence bound remains above a declared accuracy target. It refuses to split tied confidence scores and does not mistake a tiny lucky sample for proof.

Every v0.1 generative run here has 0% coverage at the 95% target under tied-confidence structured output. Valid JSON is not calibrated confidence. The next experiment matrix must elicit confidence through logprobs, verbalized confidence, self-consistency or held-out risk-control calibration before treating SAC as a generative-model capability result.

Built to be challenged

• Pinned inputs. Canonical revisions, SHA-256 source checks and immutable normalized manifests.
• Resumable inference. Item-level prediction logs preserve raw responses, token counts, model digests and runtime.
• Leakage audits. Normalized train-test overlap and conflicting labels are measured before interpretation.
• Blinded human review. An offline RTL adjudication app supports three independent native reviewers, confidence and quality flags.
• No hidden source redistribution. Evaluation-only dialect posts stay out of the public artifacts; results and cryptographic identifiers remain reproducible.

Cite, embed or cover LabelBench

The findings, charts and results files are free to use under CC BY 4.0 with attribution. Please link to this page as the source.

The LabelBench code is on GitHub at Bayanat-Labs/labelbench under the Apache-2.0 license.

Licences, DOIs, BibTeX and the changelog for every LabelBench file are on its release record: LabelBench on Bayanat Labs Open.

Suggested citation

Bayanat Labs Research (2026). Arabic Annotation Is Not Binary: LabelBench, a Reproducible Audit of Automated Data Labeling. Technical Report v0.1, July 2026. https://www.bayanatlabs.com/research/labelbench

@techreport{bayanat2026labelbench,
  title       = {Arabic Annotation Is Not Binary: LabelBench, a Reproducible Audit of Automated Data Labeling},
  author      = {{Bayanat Labs Research}},
  institution = {Bayanat Labs},
  year        = {2026},
  month       = jul,
  number      = {v0.1},
  url         = {https://www.bayanatlabs.com/research/labelbench}
}

Charts for articles and slides

• Figure 1: the annotation complexity cliff (PNG, 1400×900)
• Figure 2: dialect confusion matrix (PNG, 1400×1140)

To embed a chart, copy this snippet:

<a href="https://www.bayanatlabs.com/research/labelbench"><img src="https://www.bayanatlabs.com/research/labelbench-fig1-complexity-cliff.png" alt="LabelBench v0.1: exact-match accuracy falls from 78% on binary Arabic toxicity to 25.3% on the full taxonomy and 22.4% on dialect region" width="700"></a>
<p>Source: <a href="https://www.bayanatlabs.com/research/labelbench">Bayanat Labs, LabelBench v0.1</a></p>

Journalists and researchers: our press kit has a boilerplate, key facts and logos. For interviews or questions about the data, email hello@bayanatlabs.com.

Where this matters

If you are deciding whether a model can pre-label your Arabic data, the answer depends on your taxonomy and your dialect mix, not on a binary benchmark. Our Arabic data annotation and labeling services are built around the same checks: task-specific gold sets, native-speaker adjudication, and Safe Automation Coverage to decide how much labeling can leave the human queue. If the question is whether a model is ready to ship, see our Arabic LLM evaluation services and our protocol for evaluating Arabic LLMs you can trust.