About Unkad Labs

When a lab reports that a model refuses harmful requests reliably, that finding was almost certainly established in English. The model is then deployed to people who will not address it in English, and the safety claim travels with the deployment as though language had nothing to do with it.

We measured what actually happens. Putting identical harmful requests to open-weight models in English and in Somali, Llama 3.1 refuses 97 percent of the time in English and 7 percent of the time in Somali. Aya falls from 80 percent to 5 percent. For most of the world’s languages, nobody has run this test at all.

The scope is deliberate: safety evaluation, oversight under language limits, and the data infrastructure both require. We do not train large models, and we do not make capability claims — we would rather know whether a system is safe in Somali than celebrate that it speaks it.

It is our language, which matters both for the work and for who gets to do it. It is also a genuinely useful hard case: more than twenty million speakers, an enormous oral tradition, a writing system standardised only in 1972, and real dialect variation — Maay is spoken by millions and nearly absent from digital text. Somali is thin enough in training data that the failures we care about are visible rather than subtle. Methods that hold up under those conditions have a reasonable chance of holding up for the hundreds of languages in similar positions.

We are interested in conversations with people working on multilingual evaluation, AI safety institutes and their capacity-building work, and researchers building language resources for under-served languages. If your work touches whether models behave safely outside English, we would like to compare notes. Get in touch.