Research

Refusal and safety evaluation

Question. Do models refuse the same harmful requests in Somali that they refuse in English?

Current experiment. SomaliBench puts identical harmful requests to models in both languages, verified by native speakers, and publishes the refusal gap per model on a live leaderboard.

Latest result. Measured gaps of 0.97 → 0.07 (Llama 3.1) and 0.80 → 0.05 (Aya 23).

Open uncertainty. A low refusal rate does not always mean fluent harmful compliance; some Somali failures surface as incoherent output. Separating genuine compliance from low-quality generation is the central task for the next version.

Oversight under language and access limits

Question. What happens to AI judges, guards, and overseers when the content they must judge is in a language they barely know?

Current experiment. A series of evaluations of guard models, LLM judges, and frontier models on community-labeled Somali, each with the raw judgments published.

Latest result. Guards catch as little as 6% of harmful Somali while passing harmless Somali almost perfectly; frontier models reproduce almost none of a community’s quality rejections.

Open uncertainty. Whether a longer rubric with worked examples can recover any of the judgment gap, and whether the collapse tracks training-data volume across a ladder of languages.

Data and representation for low-resource languages

Question. Which properties of Somali text data — quantity, provenance, tokenization, dialect coverage — limit safety evaluation?

Current experiment. Building the Qor corpus with per-sentence provenance, measuring tokenizer treatment of Somali, and recording each contributor’s own spoken variety.

Latest result. Nine major tokenizers charge Somali 1.5–2.2× the tokens of English for identical content; a Somali-first tokenizer reaches 0.68×.

Open uncertainty. Whether tokenizer fragmentation contributes independently to the refusal gap, or both are symptoms of data scarcity. A separate limitation we have measured in our own corpus: Maay speakers write in standard Somali orthography, because the Maay writing system is recent and few speakers have been taught it, so dialect-disaggregated evaluation is not yet possible from text alone.

The frontier can read Somali. It cannot judge it.

GPT-5.6, Gemini 3.1 Pro, and Claude Sonnet 5 place community-labeled Somali texts in the right domain 64–70% of the time, but reproduce almost none of the community’s quality rejections (0/23, 0/23, 3/23) while still scoring ~96% agreement — an illusion made of base rates.

@misc{unkad2026qiimeyn,
  author = {Dahir, Khalid Yusuf},
  title = {The frontier can read Somali. It cannot judge it.},
  year = {2026},
  publisher = {Unkad Labs},
  url = {https://www.unkad.com/articles/the-frontier-cannot-judge-somali},
  note = {Research note. Data, code, and pre-registration at the same URL.}
}

One model reads Somali. The next one approves word salad.

Across three same-class open models, Somali grammaticality detection spans 0.54 to 0.94 balanced accuracy; one model accepts scrambled Somali 90% of the time, and 26 points of its failure is decision threshold rather than missing information.

@misc{unkad2026hubifloor,
  author = {Dahir, Khalid Yusuf},
  title = {One model reads Somali. The next one approves word salad.},
  year = {2026},
  publisher = {Unkad Labs},
  url = {https://www.unkad.com/articles/one-model-reads-somali},
  note = {Research note. Data, code, and pre-registration at the same URL.}
}

Every sentence has a name

First release of the Qor corpus: consented, dialect-tagged, twice-validated Somali text where the provenance of every sentence is known, which no scraped corpus of any language can claim.

@misc{unkad2026qorcorpus,
  author = {Dahir, Khalid Yusuf},
  title = {Qor Af-Soomaali: a provenance-complete Somali corpus},
  year = {2026},
  publisher = {Unkad Labs},
  url = {https://huggingface.co/datasets/unkadlabs/qor-af-soomaali},
  note = {Research note. Data, code, and pre-registration at the same URL.}
}

The lie moved. The liar did not.

A model trained against a blind-spotted lie detector kept its deception machinery unchanged internally and stopped using it only where the watcher could see; suppression of the caught behaviour stalled at roughly half and stayed there.

@misc{unkad2026liemoved,
  author = {Dahir, Khalid Yusuf},
  title = {The lie moved. The liar did not.},
  year = {2026},
  publisher = {Unkad Labs},
  url = {https://www.unkad.com/articles/the-lie-moved-the-liar-did-not},
  note = {Research note. Data, code, and pre-registration at the same URL.}
}

The guard that waves Somali through

Open safety filters catch 100% of a harmful-prompt set in English and as little as 6% of the same content in Somali, while passing harmless Somali almost perfectly, so aggregate accuracy on realistic traffic stays green.

@misc{unkad2026guardcollapse,
  author = {Dahir, Khalid Yusuf},
  title = {The guard that waves Somali through},
  year = {2026},
  publisher = {Unkad Labs},
  url = {https://www.unkad.com/articles/the-guard-that-waves-somali-through},
  note = {Research note. Data, code, and pre-registration at the same URL.}
}

The overseer that cannot read

An LLM judge denied a readable source does not become uncertain, it collapses to rejecting nearly everything (yes rate 62% to 6%), and posts a perfect score on the half of the data where no was the right answer; debate’s advantage survives the blinding.

@misc{unkad2026overseer,
  author = {Dahir, Khalid Yusuf},
  title = {The overseer that cannot read},
  year = {2026},
  publisher = {Unkad Labs},
  url = {https://www.unkad.com/articles/the-overseer-that-cannot-read},
  note = {Research note. Data, code, and pre-registration at the same URL.}
}

The Somali token tax

Nine major tokenizers charge Somali 1.5 to 2.2 times the tokens of English for identical content, while a Somali-first tokenizer reaches 0.68x, so the tax is a design choice rather than a property of the language.

@misc{unkad2026tokentax,
  author = {Dahir, Khalid Yusuf},
  title = {The Somali token tax},
  year = {2026},
  publisher = {Unkad Labs},
  url = {https://www.unkad.com/articles/the-somali-token-tax},
  note = {Research note. Data, code, and pre-registration at the same URL.}
}