Programme · Crawler
How the Bayanat Labs crawler behaves
How Bayanat Labs fetches public material for its open Arabic datasets, and how site owners can limit or stop it.
Who we are
| User agent | BayanatLabs-OpenData/1.0 (+https://bayanatlabs.com/open; hello@bayanatlabs.com) |
| robots.txt token | BayanatLabs-OpenData |
| Contact | hello@bayanatlabs.com |
| IP addresses | Addresses that Bayanat Labs controls. We will list them on this page before our first bulk dataset crawl. |
We never disguise the crawler as a browser.
How fast we fetch
- At most one request every 3 seconds to any host, one connection at a time.
- At most one request every 5 seconds to government portals, small sites and any host whose responses slow down.
- If your robots.txt sets a longer
Crawl-delay, we use the longer delay. - For documented APIs and dumps, we stay inside the published quota.
- Before any bulk run we fetch a small sample first, 10 to 100 representative records or the smallest file you offer.
- Downloads are resumable and checksummed, so we do not fetch the same object twice by accident.
Signals we read and obey
| Signal | What we do |
|---|---|
| robots.txt | We fetch it at the start of every session and at least daily, and obey the rules for our token and for *, following RFC 9309. |
| Blocks on other AI crawlers | If your robots.txt blocks other AI crawlers, we collect no content from your site, even though our token is not named. We may keep page URLs and identifiers only. |
| TDMRep | We check /.well-known/tdmrep.json, the tdm-reservation HTTP header and the HTML meta tag, as the W3C TDMRep report describes. If rights are reserved and we hold no licence, we collect no content from your site; we may keep page URLs and identifiers only. |
| Content-Signal | We read Content-Signal lines in robots.txt and follow them. With ai-train=no, we collect no content from your site; we may keep page URLs and identifiers only. |
| HTTP 429 and Retry-After | We stop requesting from your host, honour Retry-After and back off. After three 429 responses in a row we pause your host for 24 hours. We never retry from another address. |
| 401, 403, login walls, CAPTCHAs, firewall challenges | We stop and log the denial. We do not retry with another address, account, cookie or mirror. |
How we handle refusals
- A refusal stops us: a robots.txt rule for our crawler, an access denial or a rate limit. A rights reservation or an AI-training opt-out means we collect no content from your site.
- We never bypass a login, paywall, CAPTCHA, bot check or web application firewall.
- We never rotate proxies, IP addresses or accounts, and never disguise the crawler.
- We never use personal cookies or personal accounts.
- We never route around a denial through a mirror, a cached copy or another account.
- Where a site offers an official API, a data dump or written terms for access, we use them and stay inside their limits.
What we log
Every fetched item gets a provenance record: source URL, retrieval time, HTTP status, a hash of robots.txt, the TDMRep and Content-Signal values, licence evidence and a SHA-256 hash of the raw bytes. That lets us answer "where did this record come from?" in one query, and remove an item quickly when asked.
How to opt out or ask for removal
| You want to | Do this | What happens |
|---|---|---|
| Stop us crawling your site | Add User-agent: BayanatLabs-OpenData and Disallow: / to robots.txt | We obey it from our next robots.txt check, at least daily, and record your site as excluded |
| Keep your content out of our datasets | Add a TDMRep reservation, or Content-Signal: ai-train=no | We collect no content from your site; we may keep page URLs and identifiers only |
| Remove content already released | Email hello@bayanatlabs.com with the URL or item ID | We acknowledge within 2 business days, exclude the item from the next build and publish a corrected version within 30 days |
| Report personal data, child-safety or abuse material | Email hello@bayanatlabs.com | We stop serving the affected files within 24 hours |
| Platform request | Email hello@bayanatlabs.com | Affected rows or IDs are removed within 24 hours |
We also record your site as excluded from future crawls, and pass the opt-out to mirrors and partners bound by our terms. More on removals: removal requests.