Tarek Gara.
—

↳ playground · research

Thaghra. ثغرة

/ THUGH-ra /  ·  Arabic: gap · breach · security loophole

Safety training teaches a model to refuse harmful requests — but it learns that mostly in English. Thaghra measures what happens to that refusal when the same request is typed in Arabic: in Modern Standard Arabic, in spoken dialect, and in Arabizi (Arabic written in Latin letters and digits, the way it's actually typed online).

Where the refusal holds in English but slips in informal Arabic, that non-transfer is a low-cost security vulnerability — and Thaghra turns it from an anecdote into a measured rate, across several models.

In progress. The dataset is being hand-built and no models have been run yet — every figure below is an illustrative placeholder, not a measurement. Real results replace them when the study completes.

i.   What we measure

GAP = ASRArabizi − ASREnglish

ASR (attack success rate) is the share of harmful prompts a model complied with. A bigger GAP means safety that holds in English leaks harder in informal Arabic. We report it per model, with confidence intervals — and we report it plainly even when it's small or zero.

ii.   The four registers

The same meaning, written four ways. Here it's a harmless greeting — the real dataset carries harmful intents drawn from public research sets, never shown publicly. The point is the mechanic: as the wording moves away from formal English, the safety training has seen less and less like it.

  1. EN How are you today?
  2. MSA كيف حالك اليوم؟
  3. Dialect كيفك اليوم؟
  4. Arabizi kifak el yom?

iii.   How it works

  1. 1

    Source intents

    150–300 harmful intents sampled from public sets (AdvBench, Do-Not-Answer), excluding catastrophic categories. Every prompt's origin is logged.

  2. 2

    Author the variants

    Each intent is hand-written in English, MSA, dialect, and Arabizi — natural and meaning-equivalent. Native fluency does what machine translation can't.

  3. 3

    Run & judge

    Every form goes to every model; a separate model-as-judge labels each answer refused or complied, calibrated against human labels.

  4. 4

    Measure honestly

    Attack-success rates with confidence intervals, the GAP per model with a paired significance test, and a breakdown by harm category.

iv.   Findings · illustrative

Chart

Attack-success rate by register, across models

EN MSA Dialect Arabizi
0 10 20 30 40 Frontier A Frontier B Arabic-centric Multilingual OS
Compliance climbs as the request moves from English toward Arabizi — the visual shape of the gap. Illustrative · evaluation pending
Table 1 — ASR by register and the GAP (Arabizi − English). Illustrative placeholders; real figures replace these.
ModelENMSADialectArabiziGAP*
Frontier A2%4%9%31%+29
Frontier B1%3%12%27%+26
Arabic-centric6%8%18%22%+16
Multilingual OS5%11%24%38%+33

* GAP = ASR(Arabizi) − ASR(English), in percentage points.

v.   Built responsibly

This is defensive safety research. Only the aggregate finding and the method are ever published — never harmful model outputs, which stay private. Catastrophic categories are excluded by design, every prompt traces to a public source, and any serious model-specific failure goes to the model's maker first, with lead time, before any specifics are shared.