I roko B ench: A New Benchmark for A frican Languages in the Age of Large Language Models (2025)

Abstract: Despite the widespread adoption of Large language models (LLMs), their remarkable capabilities remain limited to a few high-resource languages. Additionally, many low-resource languages (e.g. African languages) are often evaluated only on basic text classification tasks due to the lack of appropriate or comprehensive benchmarks outside of high-resource languages. In this paper, we introduce IrokoBench—a human-translated benchmark dataset for 17 typologically-diverse low-resource African languages covering three tasks: natural language inference(AfriXNLI), mathematical reasoning(AfriMGSM), and multi-choice knowledge-based QA(AfriMMLU). We use IrokoBench to evaluate zero-shot, few-shot, and translate-test settings(where test sets are translated into English) across 10 open and four proprietary LLMs. Our evaluation reveals a significant performance gap between high-resource languages (such as English and French) and low-resource African languages. We observe a significant performance gap between open and proprietary models, with the highest performing open model, Gemma 2 27B only at 63% of the best-performing proprietary model GPT-4o performance. Machine translating the test set to English before evaluation helped to close the gap for larger models that are English-centric, like Gemma 2 27B and LLaMa 3.1 70B. These findings suggest that more efforts are needed to develop and adapt LLMs for African languages.

## introduction in recent years, the capabilities of large language models (llms) have greatly improved, from co-herent chat experiences to solving complex and knowledge-intensive tasks like mathematical reasoning, coding, and question answering (qa) @xcite @xcite . these models have also demonstrated the ability to quickly learn new and challenging tasks with few in-context learning examples and through chain-of-thought reasoning @xcite @xcite . however, most state-of-the-art (sota) llms are primarily trained on high-resource languages (hrls), resulting in sub-optimal performance for languages unseen during pre-training @xcite . furthermore, this language coverage bias is reflected in the evaluation stage, predominantly conducted in english and a few other hrls. there has been considerable effort to create benchmarks for african languages, but they typically cover simpler tasks, or are specific to narrow tasks such as machine translation, and, more recently, reading comprehension @xcite . some diverse reasoning benchmarks have included swahili-the most spoken native african language-for tasks like commonsense reasoning @xcite and natural language inference @xcite . consequently, current multilingual evaluations of llms do not accurately reflect their capabilities in reasoning and knowledge-intensive tasks across the majority of african languages. furthermore, the few comprehensive evaluations that exist across languages often rely on machine translation of english benchmarks @xcite . while automatic translation from english benchmarks is a popular approach given the cost and time investment required for human translation, it often suffers from noise and biases @xcite @xcite @xcite or fail to reflect cultural context @xcite @xcite . automatic curation may also amplify any of the ubiquitous issues with the quality of broad pretraining sets @xcite @xcite . in this paper, we seek to address both the diversity and breadth of evaluation coverage. we introduce irokobench, a human curated benchmark dataset for 17 typologically diverse african languages which encompasses three complex tasks: natural language inference (nli), mathematical reasoning, and multi-choice knowledgebased qa. the datasets were created by human translating a subset of english cross-lingual nli (xnli) @xcite , english multilingual grade school math (mgsm) @xcite , and massive multitask language understanding (mmlu) @xcite , evaluation datasets into each of the 16 languages using professional translators. we conduct a large-scale evaluation of irokobench to assess zero-shot, few-shot, and translate-test settings (where test sets are translated into english) performance across 10 open and six proprietary llms. our main contributions can be enumerated as follows: ## irokobench kinyarwanda, luganda, swahili, and oromo), four from southern africa (chishona, isixhosa, isizulu, and sesotho), and central africa (lingala). these languages are from three language families in africa: one from mande, three from afro-asiatic and 13 from the niger-congo family-where we cover eight bantu languages. ## conclusion in this paper, we introduced irokobench, a new benchmark for evaluating large language models (llms) on african languages. irokobench comprises three datasets focused on different tasks: natural language inference (afrixnli), multi-choice knowledge qa (afrimmlu), and mathematical reasoning (afrimgsm). unlike previous benchmarks, which primarily involve simple text classification tasks, these datasets assessed the llms' abilities in complex and knowledge-intensive areas. our evaluation revealed a significant performance gap between high-resource languages (e.g., english and french) and african languages. additionally, we observed a substantial disparity in performance between open models and proprietary models, with the latter generally outperforming the former, particularly in mathematical reasoning tasks. we hope that irokobench will serve as a valuable benchmark for evaluating future llms developed or adapted for african languages. limitations our benchmark has a few limitations: (1) the benchmark is human-translated which may include some translationese effects, it would have been better if they are all generated in the native african languages. however, this parallel translation allows us to evaluate and compare the same sentences in all these languages. (2) we only cover three language families in africa, nilo-saharan, austronesian, and khoisan language human language technologies, pages 3053-3070, seattle, united states. association for computational linguistics. david adelani, hannah liu, xiaoyu shen, nikita vassilyev, jesujoba alabi, yanke mao, haonan gao, and en-shiun lee. 2024. sib-200: a simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. in proceedings of the 18th conference of the european chapter of the association for computational linguistics (volume 1: long papers), pages 226-245, st. julian's, malta. association for computational linguistics. david adelani, graham neubig, sebastian ruder, shruti rijhwani, michael beukman, chester palen-michel, constantine lignos, jesujoba alabi, shamsuddeen muhammad, peter nabende, cheikh m. bamba dione, andiswa bukula, rooweither mabuya, bonaventure f. p. dossou, blessing sibanda, happy buzaaba, jonathan mukiibi, godson kalipe, derguene mbaye, amelia taylor, fatoumata kabore, chris chinenye emezue, anuoluwapo aremu, perez ogayo, catherine gitau, edwin munkoh-buabeng, victoire memdjokam koagne, allahsera auguste tapo, tebogo macucwa, vukosi marivate, mboning tchiaze elvis, tajuddeen gwadabe, tosin adewumi, orevaoghene ahia, and joyce nakatumba-nabende. 2022b. masakhaner 2.0: africa-centric transfer learning for named entity recognition. in proceedings of the 2022 conference on empirical methods in natural language processing, pages 4488-4508, abu dhabi, united arab emirates. association for computational linguistics. david ifeoluwa adelani, jade abbott, graham neubig, daniel d'souza, julia kreutzer, constantine lignos, chester palen-michel, happy buzaaba, shruti rijhwani, sebastian ruder, stephen mayhew, israel abebe azime, shamsuddeen h. muhammad, chris chinenye emezue, joyce nakatumba-nabende, perez ogayo, aremu anuoluwapo, catherine gitau, derguene mbaye, jesujoba alabi, seid muhie yimam, tajuddeen rabiu gwadabe, ignatius ezeani, rubungo andre niyongabo, jonathan mukiibi, verrah otiende, iroro orife, davis david, samba ngom, tosin adewumi, paul rayson, mofetoluwa adeyemi, gerald muriuki, emmanuel anebi, chiamaka chukwuneke, nkiruka odu, eric peter wairagala, samuel oyerinde, clemencia siro, tobius saul bateesa, temilola oloyede, yvonne wambui, victor akinode, deborah nabagereka, maurice katusiime, ayodele awokoya, mouhamadane mboup, dibora gebreyohannes, henok tilaye, kelechi nwaike, degaga wolde, abdoulaye faye, blessing sibanda, orevaoghene ahia, bonaventure f. p. dossou, kelechi ogueji, thierno ibrahima diop, abdoulaye diallo, adewale akinfaderin, tendai marengereke, and salomey osei. 2021. masakhaner: named entity recognition for african languages. transactions of the association for computational linguistics, 9:1116-1131. david ifeoluwa adelani, marek masiak, israel abebe azime, jesujoba alabi, atnafu lambebo tonja, christine mwase, odunayo ogundepo, bonaventure f. p. dossou, akintunde oladipo, doreen nixdorf, chris chinenye emezue, sana al-azzawi, blessing sibanda, davis david, lolwethu ndolela, jonathan mukiibi, tunde ajayi, tatiana moteu, brian odhiambo, abraham owodunni, nnaemeka obiefuna, muhidin mohamed, shamsuddeen hassan muhammad, teshome mulugeta ababu, saheed abdullahi salahudeen, mesay gemeda yigezu, tajuddeen gwadabe, idris abdulmumin, mahlet taye, oluwabusayo awoyomi, iyanuoluwa shode, tolulope adelani, habiba abdulganiyu, abdul-hakeem omotayo, adetola adeeko, abeeb afolabi, anuoluwapo aremu, olanrewaju samuel, clemencia siro, wangari kimotho, onyekachi ogbu, chinedu mbonu, chiamaka chukwuneke, samuel fanijo, jessica ojo, oyinkansola awosan, tadesse kebede, toadoum sari sakayo, pamela nyatsine, freedmore sidume, oreen yousuf, mardiyyah oduwole, kanda tshinu, ussen kimanuka, thina diko, siyanda nxakama, sinodos nigusse, abdulmejid johar, shafie mohamed, fuad mire hassan, moges ahmed mehamed, evrard ngabire, jules jules, ivan ssenkungu, and pontus stenetorp. 2023. masakhanews: news topic classification for african languages. in proceedings of the 13th international joint conference on natural language processing and the 3rd conference of the asia-pacific chapter of the association for computational linguistics (volume 1: long papers), pages 144-159, nusa dua, bali. association for computational linguistics. kabir ahuja, harshita diddee, rishav hada, millicent ochieng, krithika ramesh, prachi jain, akshay nambi, tanuja ganu, sameer segal, mohamed ahmed, kalika bali, and sunayana sitaram. 2023a. mega: multilingual evaluation of generative ai. in proceedings of the 2023 conference on empirical methods in natural language processing, pages 4232-4267, singapore. association for computational linguistics. sanchit ahuja, divyanshu aggarwal, varun gumma, ishaan watts, ashutosh sathe, millicent ochieng, rishav hada, prachi jain, maxamed axmed, kalika bali, and sunayana sitaram. 2023b. megaverse: benchmarking large language models across languages, modalities, models and tasks. arxiv, abs/2311.07463. jesujoba o. alabi, david ifeoluwa adelani, marius mosbach, and dietrich klakow. 2022. adapting pretrained language models to african languages via multilingual adaptive fine-tuning. in proceedings of the 29th international conference on computational linguistics, pages 4336-4349, gyeongju, republic of korea. international committee on computational linguistics. neural machine translation. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 2576-2590, dublin, ireland. association for computational linguistics. jason wei, xuezhi wang, dale schuurmans, maarten bosma, fei xia, ed chi, quoc v le, denny zhou, et al. 2022. chain-of-thought prompting elicits reasoning in large language models. advances in neural information processing systems, 35:24824-24837.