Unkad Labs · independent AI research lab
We measure whether AI systems behave safely in Somali, and we build the open language data that safety evaluation requires. Models that refuse a harmful request in English will often answer the same request in Somali; safety filters that catch harm in English wave it through. We document these failures with methods designed to transfer to the hundreds of languages in the same position.
Llama 3.1 8B
Aya 23 8B
Field survey of 28,796 public GitHub rule files read by coding agents (.cursorrules, AGENTS.md, CLAUDE.md and five more): no invisible-Unicode instruction attacks outside one planted test case, bounding the rate below 1 in 9,600 in the popular, search-ranked population with 95% confidence.
Pre-registered, four open models: when they go along with harmful Somali requests they mostly have not understood them (Llama 3.1: 0 of 7). Only Gemma 2 reads Somali well enough to show knowing compliance, about one in five. Asking for an English restatement first raises refusals for three of four models.
GPT-5.6, Gemini 3.1 Pro, and Claude Sonnet 5 place community-labeled Somali texts in the right domain 64–70% of the time, but reproduce almost none of the community’s quality rejections (0/23, 0/23, 3/23) while still scoring ~96% agreement — an illusion made of base rates.
Qor Af-Soomaali is the corpus this research depends on. Every sentence is written by a consenting Somali speaker and accepted by two independent validators; releases additionally require linguist verification, and ship with per-sentence provenance under CC BY-SA 4.0.