Unkad Labs · independent AI research lab
We measure whether AI systems behave safely in Somali, and we build the open language data that safety evaluation requires. Models that refuse a harmful request in English will often answer the same request in Somali; safety filters that catch harm in English wave it through. We document these failures with methods designed to transfer to the hundreds of languages in the same position.
Llama 3.1 8B
Aya 23 8B
GPT-5.6, Gemini 3.1 Pro, and Claude Sonnet 5 place community-labeled Somali texts in the right domain 64–70% of the time, but reproduce almost none of the community’s quality rejections (0/23, 0/23, 3/23) while still scoring ~96% agreement — an illusion made of base rates.
Across three same-class open models, Somali grammaticality detection spans 0.54 to 0.94 balanced accuracy; one model accepts scrambled Somali 90% of the time, and 26 points of its failure is decision threshold rather than missing information.
First release of the Qor corpus: consented, dialect-tagged, twice-validated Somali text where the provenance of every sentence is known, which no scraped corpus of any language can claim.
Qor Af-Soomaali is the corpus this research depends on. Every sentence is written by a consenting Somali speaker and accepted by two independent validators; releases additionally require linguist verification, and ship with per-sentence provenance under CC BY-SA 4.0.