Language and instructions
Check meaning across Arabic varieties, appropriate register, constraints, and corrections across multiple turns.
Bayanat Bench
See where your Arabic model needs work.
Compare model versions using the language, documents, and tasks your users rely on. License Arabic evaluation data or build a private suite around your product.
اقرأ بالعربيةArabic evaluation sets are also available to license with our smartphone speech dataset. Ask us about the available versions and counts.
Is Bayanat Bench the right fit?
A good fit if
- You are choosing between models or releases and need Arabic tests that match your product.
- You need held-out data kept separate from training deliveries.
- You want failure examples, severity and reviewer disagreement, not a single score.
What you can check before you license
We separate what is confirmed on this page, what we send on request, and what is agreed for each project.
| Item | Status | Details |
|---|---|---|
| Licensable evaluation sets | Confirmed | Existing Arabic evaluation sets can be licensed alongside our Arabic smartphone speech dataset. Versions and coverage on request. |
| Held-out separation | Confirmed | Held-out items stay separate from training deliveries, with access and versioning defined for each project. |
| Custom suites | Per project | Built from tasks, varieties, sample sizes and metrics you choose. |
| Report contents | Per project | Failure examples, severity, uncertainty, reviewer disagreement and changes between model versions. |
| Sample report | On request | Ask for an example of the report format. |
| Published research | Confirmed | LabelBench and the Arabic Agent Reliability Lab describe specific studies; their scores are not ratings of catalog products. |
What we can build together
Answers and documents
Test evidence-supported answers, reading, extraction, tables, and cross-document questions. Identify retrieval misses separately from unsupported answers.
Voice and agent actions
Measure turn-taking, corrected details, confirmation, and resolution. Executable tasks can check what actually changed.
Safety and robustness
Test both unsafe responses and unnecessary refusals, using explicit criteria and severity levels suited to your application.
How the license covers each use
Training rights do not automatically include display, retrieval or redistribution. Each use is set in the agreement.
Evaluation use
Licensed for evaluating your systems under the agreed access terms.
Results
Reports from a custom evaluation are delivered to you.
Held-out items
Access and versioning are defined for the project so the test stays useful.
How we put the data together
Decide what to measure
Select an existing set or define tasks, Arabic varieties, sample sizes, and metrics for a custom evaluation.
Keep test data separate
Held-out items stay separate from training deliveries, with access and versioning defined for the project.
Get useful findings
Custom reports can include failure examples, severity, uncertainty, reviewer disagreement, and changes between model versions.
Explore our research
LabelBench and our Arabic agent reliability methods beta describe specific studies; their scores are not blanket ratings for catalog products.
For example: Comparing two voice-agent releases
Both sound fluent. The evaluation shows which one captures a corrected address, asks for required confirmation, and completes the requested change.
Scope your request
Send us these details and we’ll send the available specifications, and a sample where one is available, so your team can check the fit before licensing.
- The model or product to evaluate
- Tasks and Arabic varieties
- The decision the evaluation should inform
- Sample size
- Timeline
Common questions
Can you evaluate our own customer-service tasks?
Yes. A custom suite can use examples and policies you are authorized to share. We define expected behavior, tested language varieties, and scoring criteria with your team before running the evaluation.
Will the evaluation items also be sold to us as training data?
Held-out evaluation items are kept separate from training deliveries. For a custom project, we define access, versions, and the separation of training and test material so the results remain useful.