Bayanat Labs Arabic Unicode Conformance Suite
Arabic text normalisation differs from tool to tool: one removes harakat and tatweel, another also folds أ إ آ to ا, a third turns each haraka into a space, so word error rates, deduplication counts and search indexes built with different normalisers are not comparable. This suite pins the behaviour down. Each test case gives an input string with its code points and bytes, and the exact output under six named policies: identity, NFC, NFKC and the three arabic-normalize policies raw-safe-v1, bayanat-norm-v1 and asr-eval-v1. A Python runner checks a normaliser against the cases, and a conformance matrix records how Python and ICU Unicode normalisation, arabic-normalize, CAMeL Tools, PyArabic and the Whisper text normaliser behave on them.
| Licence | CC BY 4.0 (cases, policy specifications, documentation and reports); Apache-2.0 (code) |
|---|---|
| Version | v0.1.0 · Cases, policies and runner v0.1.0, tagged on GitHub. The runner is not on PyPI yet. |
| Published | |
| Updated | |
| Who it is for | Library authors, speech-recognition and search teams, and anyone who reports Arabic WER and needs to state which normalisation a tool follows. |
| Level | Intermediate |
Key numbers
Counted by script from the released files. Each key number is read from MANIFEST.json in the Hugging Face dataset and pinned to its SHA-256.
Downloads and identifiers
| Where | What |
|---|---|
| Hugging Face | Dataset on Hugging Face |
| GitHub | Code and runner on GitHub |
| GitHub | Normative policy text (bayanat-norm-v1) |
| bayanatlabs.com | Arabic Text Normalizer (arabic-normalize) |
Get started
Install the runner from GitHub at the v0.1.0 tag (not on PyPI yet), or load the cases with the datasets library and compare your output with the expected_<policy> columns.
pip install "git+https://github.com/Bayanat-Labs/arabic-unicode-conformance@v0.1.0"
arabic-unicode-conformance list
arabic-unicode-conformance run --impl arabic-normalize-py --check
# Python
from datasets import load_dataset
cases = load_dataset("Bayanat-Labs/arabic-unicode-conformance", "cases", split="test")What is released
| Part | Licence | In this release |
|---|---|---|
| Test cases and expected outputs | CC BY 4.0 | Yes |
| Policy specifications and documentation (English and Arabic) | CC BY 4.0 | Yes |
| Conformance matrix and observed outputs | CC BY 4.0 | Yes |
| Runner and adapters (Python package) | Apache-2.0 | Yes |
Attribution: Arabic Unicode Conformance Suite by Bayanat Labs, CC BY 4.0. https://huggingface.co/datasets/Bayanat-Labs/arabic-unicode-conformance
Intended uses
- Check that a normaliser follows a named policy, case by case, and publish which policy it follows.
- Compare normalisers on the same Arabic edge cases: harakat, hamza, alef and yeh variants, digits, joiners, bidi controls and presentation forms.
- Use the cases as regression tests when a normaliser changes.
Out of scope
- A quality ranking of normalisation tools: an agreement score shows how far a tool's own normalisation is from the closest suite policy, not that the tool is wrong.
- Training data: the cases are short edge cases written to test normalisers.
Rights at a glance
Training, display, redistribution and synthetic-voice use are separate permissions.
| Use | Status | Note |
|---|---|---|
| Training | Allowed | Cases and documentation under CC BY 4.0 with attribution. |
| Display | Allowed | CC BY 4.0, with attribution. |
| Redistribution | Allowed | Cases and documentation under CC BY 4.0 with attribution; code under Apache-2.0; the Unicode data files under the Unicode License V3. |
| Synthetic voice | Not applicable | Text only. |
Limitations
- The expected NFC and NFKC outputs follow the Unicode version named in the manifest; another Unicode version can change some outputs.
- Third-party tools were run unmodified through their public functions in the versions listed in the repository; newer versions can behave differently.
- The runner installs from GitHub until it is published on PyPI.
Sources and licences
| Source | Licence | Redistributed here |
|---|---|---|
| Unicode Character Database 18.0.0 | Unicode License V3 | Yes |
Every case was written by Bayanat Labs for this suite; no text is taken from a corpus. The three named policies are the policy files of arabic-normalize (CC BY 4.0). The GitHub repository includes three Unicode Character Database files under the Unicode License V3 (see NOTICE). Source details
How to cite
Cite as:
Bayanat Labs (2026). Arabic Unicode Conformance Suite, version 0.1.0. Bayanat Labs. https://huggingface.co/datasets/Bayanat-Labs/arabic-unicode-conformance
No DOI. Cite the URL in the citation above and the date you read it.
@misc{bayanatlabs2026arabicunicodeconformance,
author = {{Bayanat Labs}},
title = {Arabic Unicode Conformance Suite, version 0.1.0},
year = {2026},
month = oct,
howpublished = {\url{https://huggingface.co/datasets/Bayanat-Labs/arabic-unicode-conformance}}
}
Changelog
| Date | Version | Change |
|---|---|---|
| v0.1.0 | Test cases, policy specifications, runner and conformance matrix on GitHub and Hugging Face. |
Corrections and removal requests
If something in this release should not be there, or is wrong, email hello@bayanatlabs.com. We handle removal requests within 30 days, and platform requests within 24 hours. How removal requests work
Related product: Bayanat Bench