AI chatbot safety refusals collapse in Yoruba and Nigerian Pidgin, red-team study finds

Researchers posted a study to arXiv that sent 400 adversarial prompts in four harm categories, in English, Yoruba and Nigerian Pidgin, to six large language models (Llama 3.1 8B, Llama 3.3 70B, Qwen3 32B, Gemini 2.0 Flash, GPT-4o Mini and Aya 101), with native-speaker annotators scoring every response. GPT-4o Mini refused 47% of the English prompts and 1% of the Yoruba ones, and the share of all prompts that drew a harmful answer rose from 18% in English to 41% in Yoruba; Llama 3.1 8B, Llama 3.3 70B and Aya 101 refused no Yoruba prompts at all. Some low harm rates in Yoruba reflected models failing to understand the prompts rather than safety, the authors report: Qwen3 32B understood 8% of the Yoruba prompts, against 99% in English.