NaijaFacts：一个将语言能力与事实知识解耦的尼日利亚事实基准框架
NaijaFacts: A Framework for Disentangling Language from Knowledge in Facts about Nigeria
AI 导读
NaijaFacts 是一个由母语者撰写、基于官方统计数据的尼日利亚事实基准，每道题提供英语、尼日利亚皮钦语、约鲁巴语、伊博语和豪萨语五个版本。在八个前沿及尼日利亚适配模型上测试发现，知识效应占主导：美国/英国问题英语查询准确率为 49–50%，尼日利亚仅 20%；最强模型在英美问题上达 66–79%，但在尼日利亚统计题上无一超过 31%。
正文 · 原文
Abstract:Multilingual benchmarks for African languages mostly translate generic questions into local languages, measuring linguistic competence but not whether models know facts about Africa. We introduce \naija, a benchmark of factual and statistical questions about Nigeria, authored by native speakers and grounded in authoritative official statistics, with every question available in English, Nigerian Pidgin, \yoruba, Igbo, and Hausa. Answering similar questions for the UK and US yields a country by language matrix that disentangles a language effect (the model knows the fact but degrades in a low-resource language) from a knowledge effect (the fact is absent in every language). Across eight frontier and Nigeria-adapted models, the knowledge effect dominates: English-query accuracy is 49--50\% for the US/UK but 20\% for Nigeria. The language effect is smaller and tracks web representation, degrading from English to \yoruba{} (9 points on Nigeria, up to 24 on US/UK questions). We find scale does not help: the strongest models reach 66--79\% on the UK and the US but none exceed 31\% on Nigerian statistics. Extending to all 54 African countries with our best performing frontier model shows the same profile continent-wide, 96\% on culture and 98\% on politics against 26\% on statistics, so the gap is not specific to Nigeria.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.06504 [cs.CL] |
| (or arXiv:2611.06504v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06504 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zainab Afolabi [view email]
[v1]
Wed, 4 Nov 2026 13:34:35 UTC (1,858 KB)
来源：arXiv cs.AI · arxiv.org