Research · Synthetic data

Model collapse speaks MSA: the limits of synthetic Arabic data

Synthetic data is the most tempting shortcut in low-resource AI: infinite Arabic text, on demand, at near-zero cost. The research says the shortcut has a price — and for Arabic, the price lands precisely on the dialects you were trying to cover. Here is what model collapse actually does, why Arabic is maximally exposed, and the pipeline design that keeps generation useful instead of corrosive.

Bayanat Labs Research· July 2026· 9 min read· Updated September 2026

The arithmetic looks unbeatable. Human dialectal data costs real money per example; a large model can generate thousands of "Gulf Arabic" conversations in an afternoon. For teams staring at the Arabic data gap, synthetic Arabic data from a bigger model feels like the obvious fix, and plenty of Arabic training data is already machine-made. A 2025 review of Arabic post-training datasets notes that the Arabic instruction data behind Jais was mostly translated from English resources, and that the AceGPT team generated data by prompting GPT-4 directly in Arabic.

The research on what happens next is no longer speculative. Training models on the outputs of models — recursively, generation after generation — degrades them in a specific, well-documented way. That way of failing is bad news for Arabic in particular. The short version: generation is a good source of volume and format and a poor source of dialect, because collapse erases rare forms first, and dialect is where Arabic keeps its rare forms.

What collapse actually looks like

In 2024, Shumailov and colleagues showed in Nature: when models are trained indiscriminately on data produced by earlier models, errors compound across generations and the models gradually forget the true distribution — a phenomenon they called model collapse. They demonstrated it in language models, variational autoencoders and Gaussian mixture models. Two details matter more than the headline.

First, collapse is not uniform: it begins in the tails of the distribution. The paper separates early collapse, where the model starts losing information about low-probability content, from late collapse, where it converges on something with little resemblance to the original and far less variance. Rare, distinctive content disappears first, while the model drifts toward its own high-frequency center. Dohmatob et al. (ICML 2024) described the same tail loss as a break in scaling laws: extra data stops buying the improvement it should, and models can un-learn skills.

Second, the regime matters. Gerstgrasser et al. (2024) showed that collapse is severe when synthetic data replaces real data each round, and avoided when successive synthetic generations accumulate alongside the original real data; in their analysis, test error then stays under a bound that does not grow with the number of rounds. The Nature paper points the same way: in its language-model experiment, keeping a random 10% of the original data in every generation left only minor degradation.

Figure 1. The two regimes documented in the collapse literature; the curves are illustrative, not measured results. Keeping a stable base of real data while adding filtered synthetic data preserves quality; letting synthetic output replace real data compounds errors generation over generation. Which regime your pipeline is in is a design decision — often made by accident.

Dialects live in the tails

Now put the two facts together. Collapse eats low-probability content first. And in any Arabic corpus a base model has seen, dialect is the low-probability content. Arabic is diglossic: news, reference and official writing use MSA, while dialects carry everyday speech and informal chat, so written web Arabic skews heavily formal. The authors of the AraDiCE benchmark put it plainly: Arabic dialects remain significantly underrepresented in LLMs. Najdi negation patterns, Egyptian idiom, Levantine aspect markers and Arabizi spellings all sit in the statistical tails. (For why that gap sets your model's ceiling, see MSA vs dialectal Arabic.)

Ask an MSA-biased model to generate "dialectal" training data and you get the center of its distribution wearing a costume — MSA syntax with a sprinkling of stereotyped dialect markers. An illustrative case: a Gulf speaker says "I don't know" as ما أدري (mā adrī) and an Egyptian as مش عارف (mish ʿārif), but a skewed generator is pulled toward MSA لا أعرف (lā aʿrif) with a dialect particle bolted on nearby. Train on that, and the next model's tails are thinner still.

This is the easy part to miss. In a high-resource language, tail loss trims the edges of a huge distribution. For Arabic dialects, it is a bias amplifier. Each synthetic round launders the original MSA skew into something that looks like coverage — the dataset card says, hypothetically, "Saudi dialect, 50,000 examples" — while the actual distinctive signal your model needed is precisely what generation smoothed away. You end up certifying the gap you set out to close. The Nature authors make a related point: keeping a model's grip on low-probability events matters for fairness, because those events often concern marginalized groups.

Figure 2. Collapse begins in the tails — and for Arabic, the tails are the product requirement. The dataset grows, the metrics on frequent patterns hold, and the dialectal signal quietly disappears. Diversity loss can run ahead of headline quality loss, which makes it easy to miss until deployment.

Where synthetic Arabic data earns its keep

None of this makes synthetic data useless. It makes it a tool with a sharp grain. Generation works where correctness is verifiable and the target distribution is already well represented — and it fails where the value is authenticity that only a native speaker can produce or judge:

Use case	Verdict	Why
Format & schema variety	Good fit	Rephrasing, templating and structure expansion are checkable and low-risk.
Coverage backfill for known intents	Use with care	Useful volume — but only behind a native-speaker filter with tracked acceptance rates.
Red-team & edge-case seeds	Good fit	Breadth of attack surface matters more than naturalness; humans triage what lands.
Dialect authenticity & register	Human-first	The generator's dialect distribution is the problem you are trying to fix.
Preference & reward data	Human-first	LLM judges recognize and favor their own generations, so a synthetic judge grading its own family's output is circularity, not signal. Keep Arabic preference and RLHF judgments with native raters.
Cultural & religious content	Human-first	Errors are high-consequence and invisible to automatic filters.

Published Arabic work that uses generation well follows this grain. AraDiCE built its dialect data with machine translation followed by human post-editing. The CIDAR team translated an English instruction set with GPT-3.5 Turbo and had about a dozen reviewers localize it; roughly 64.5% of the 9,109 pairs needed changes before they belonged in an Arabic dataset.

The human-anchored pipeline

Pipelines that get value from generation tend to share one architecture: humans at the boundaries, machines in the middle. Native speakers author a gold seed set that defines what right looks like; generation expands it under tight constraints; native reviewers filter and repair the output before anything reaches training — and the acceptance rate itself becomes a metric. CIDAR's 64.5% is that kind of number: it measures the translation step, not the reviewers. If reviewers reject, say, 40% of "Egyptian" generations for register drift (an illustrative figure), you have measured your generator's dialect bias instead of ingesting it.

Figure 3. Generation adds volume in the middle; native speakers define the target at the start and enforce it at the end. The feedback loop is what keeps the pipeline in Figure 1's safe regime instead of the collapsing one.

Four operating rules make the difference between the two regimes in Figure 1:

Never let synthetic data replace human data. Keep the human gold set in every training mix, permanently. Accumulate; don't substitute.
Cap the synthetic share and write it down. Decide the ratio deliberately per task — and treat any pressure to raise it as a cost question, not a quality-neutral one.
Filter with the population you are targeting. A Gulf reviewer for Gulf data, working from written guidelines and gold examples (our Arabic text annotation playbook covers how to write them). A model is a poor stand-in: on the closed five-way dialect task in LabelBench, our reproducible audit (v0.1, zero-shot), Qwen 2.5 3B scored 22.4% against a 20% chance floor and labeled 75% of Gulf examples as Egyptian. A filter that cannot tell Gulf from Egyptian cannot certify "Gulf" data. Acceptance rates below your threshold mean the generator, prompt or constraint set needs work — not the reviewers.
Track diversity, not just accuracy. Monitor dialect-classifier distributions, lexical variety and rare-form retention across dataset versions. Guo et al. (NAACL Findings 2024) found lexical, syntactic and semantic diversity falling consistently across recursive rounds of fine-tuning on synthetic text (in English experiments). Because collapse starts in the tails, diversity metrics are where to look first — and every version should also be scored on a held-out, human-written test set per dialect, as set out in our Arabic LLM evaluation protocol.

This is how we approach Arabic LLM training data at Bayanat Labs. Vetted native speakers author the seed set; generation expands format, not dialect; native reviewers check generated items against gold standards, with adjudication and rework loops and an audit trail on every task. We are model-agnostic, so the generator in the middle can be whichever model you already use.

For the MSA side of the mix, the base you keep can be licensed rather than scraped. Bayanat Knowledge offers 2,700 Arabic books and professionally edited, subject-organized MSA covering material through 2026, with commercial AI-training rights written into the edited collection's agreement. It anchors the MSA core, not the dialect tails; those still need native-authored data.

Key takeaways

Collapse eats the tails first — and dialects are the tails. For Arabic, recursive synthetic data is not slow degradation; it is amplification of the MSA bias you started with.
The regime is a design choice. In published experiments, accumulating synthetic data on a permanent human base avoids collapse; replacement is how pipelines collapse by accident.
Synthetic volume is real; synthetic authenticity is not. Use generation for format, coverage and red-team breadth — never as the source of dialect, culture or preference signal.
Measure the generator through rejections. Native-speaker acceptance rates turn your filter into an instrument that quantifies bias before you train on it.

Frequently asked questions

What is synthetic Arabic data?

Synthetic Arabic data is text, speech, images or labels produced by a model rather than by people: LLM-generated conversations and instructions, English datasets machine-translated into Arabic, text-to-speech audio, rendered document images, or labels assigned by a model. It is usually blended with human data. Its value depends on the seed it was expanded from and the filter it passed through, so ask about both whenever a dataset is sold as Arabic or dialectal.

Does fine-tuning once on synthetic data cause model collapse?

Not in the strict sense. Collapse, as the research defines it, is a recursive process that compounds over generations. A single round on filtered synthetic data does something smaller but related: the model inherits the generator's distribution, including its MSA skew. The risk compounds when that model writes the next dataset, or when its output ends up in the web text future models are trained on. A human-written base in every mix is cheap insurance.

Is machine-translated Arabic data synthetic data?

For training purposes, yes. Machine translation is a model writing Arabic, and it brings two kinds of bias: register, because translation output defaults to MSA, and content, because the names, places and cultural assumptions come from the English source. The CIDAR team found that about 64.5% of the pairs they machine-translated from English needed changes, for linguistic errors or cultural irrelevance, before they belonged in an Arabic dataset. Filter translated data with native reviewers, like any generated data, and track what they reject.

How much synthetic data is safe in an Arabic training mix?

There is no universal ratio. Published experiments use different models, tasks and mixes, so no single percentage transfers to your project. Decide the share per task and test it: train at two or three synthetic ratios, then score each run on a held-out, human-written test set broken down by dialect. If Gulf or Maghrebi scores fall as the synthetic share rises while MSA holds steady, you have found the ceiling for that task.

Can an Arabic evaluation or test set be synthetic?

Not for the questions that matter most. A generated test set measures agreement with the generator, so it hides the dialect and cultural gaps you need to see, and it can overlap with synthetic training data from the same model. Keep test sets human-written, held out and versioned. Synthetic items are fine for stress-testing formats or seeding red-team prompts that people then triage. Our guide to Arabic LLM benchmarks covers what the public suites measure.