How Bayanat Labs builds open datasets

Operating manual version 1.0.1 · Documentation, not legal advice

Every open dataset from Bayanat Labs goes through the same process. It starts with a one-page scoping memo. Then it passes 12 more stages. Each stage ends in a gate, and every gate is blocking. If a gate fails, the release stops. If a release ever skips a gate, it is taken back.

The full manual is public on GitHub in English and Arabic: Bayanat-Labs/open-policies. What we release, and under which licences, is set out in our open data policy.

The 13 stages

0. Scoping memo. We decide on one page whether a dataset is worth building and whether we may build it. We fetch nothing before the memo is approved.
1. Discovery. We list every source and record its rights signals: robots.txt, TDMRep, Content-Signal and the licence. Sources we may not use are left out.
2. Acquisition. We fetch only what the memo allows. Our crawler names itself, waits at least 3 seconds between requests to a site (5 seconds for government and small sites) and keeps a checksum for every file.
3. Normalisation. We clean text and audio into one consistent format. The original stays next to the cleaned version, so nothing is lost. Speech is cut into segments and transcribed here, never from automatic captions.
4. Deduplication. We remove duplicates. We check that no test item from a benchmark appears in training data.
5. Safety and personal data. We remove personal data and unsafe content. A stratified audit of 3,000 records must find no personal data at all before anything is uploaded outside Saudi Arabia and the UAE. Clips and images in which a child can be identified are dropped.
6. Enrichment. We add labels and quality scores, each marked with where it came from. Speakers of each dialect check at least 1% of the labels for their dialect.
7. Human quality review. Paid native reviewers check a stratified sample whose size is fixed in advance. Agreement must reach the threshold set in the memo (kappa of at least 0.6, or 0.7 for evaluation sets). A named reviewer signs off.
8. Packaging. We build a release that loads and displays correctly and carries its licences, a bilingual dataset card, checksums and a provenance file for every record.
9. Baseline or benchmark. A script computes at least one number from the released files, so anyone can check it.
10. Release. We re-check every claim on the day of release. The coordinator runs the launch checklist. The founder gives the go.
11. Promotion and measurement. We announce the release and record whether people use it.
12. Maintenance. We handle takedown requests, refresh on a schedule, re-check licences for each version and set the date of the next version.

How the gates work

Each gate is a list of checks that a script can run: a file exists, a file validates against a schema, a number meets a threshold, a named person signed off, or a record exists in a ledger. The manual defines 114 such checks across the 13 gates, of 6 types. Human judgement enters only through named sign-offs.

Thresholds are defaults. A project may make them stricter, never looser, unless the founder approves the change in writing. Where our sources gave different values for a threshold, we record the value adopted and the alternative (21 such choices).

A step that needs a legal position cites the recorded decision it relies on, by its decision ID. No step waits for a separate legal sign-off.

Definition of done

A release is done when all 12 items are true:

1. Scoping memo: approved before anything is downloaded.
2. Provenance ledger: where every record came from, and under which licence.
3. Quality report: signed by a named native reviewer.
4. Claims ledger: every number and claim, re-checked on release day.
5. Card and release files: a bilingual card, licences, citation, changelog, checksums, Croissant metadata and a working example.
6. DOI: reserved before release, published at release.
7. Hub page: a page on bayanatlabs.com with structured data.
8. Baseline chart: a number anyone can check, with its chart.
9. Listings: Masader, Kaggle and Google Dataset Search.
10. Launch bundle sent: announcement drafts, sent by the founder.
11. Metrics row created: use is tracked from the first month.
12. Next-version date set: and the takedown route is live.

Most of this is checked before the founder's go. A published DOI, live listings, the sent launch bundle and the first metrics row can only exist after it.

How our crawler behaves

• It sends a descriptive User-Agent with a contact address: BayanatLabs-OpenData/1.0 (+https://bayanatlabs.com/open; hello@bayanatlabs.com).
• It makes at most one request every 3 seconds per site (5 seconds for government and small sites), one connection at a time.
• It obeys robots.txt, TDMRep and Content-Signal. It also reads a block on other AI crawlers as a "no".
• It stops when a site says it is overloaded (HTTP 429), and it stops at any access denial.
• It never bypasses a login, paywall, firewall or CAPTCHA, and never rotates proxies, addresses or accounts.
• It logs every request, so we can show where any record came from.

To ask us to remove something, write to hello@bayanatlabs.com. We remove items within 30 days, and within 24 hours for platform requests and personal-data reports. More detail is on our crawler page.

What we have not measured yet

We plan to publish our own measured cost and processing speed per 1,000 hours of Arabic audio. Those figures need a GPU calibration run that has not happened yet, so this page gives no cost or speed figures.

Read the full manual

• English: manual/PUBLIC.md
• Arabic: manual/PUBLIC.ar.md
• Machine-readable stages and gates: manual/stages.yaml
• Our open-data policy

Text CC BY 4.0. Cite as: "Operating manual by Bayanat Labs", version 1.0.1.