A fro B ench: How Good are Large Language Models on A frican Languages? (2025)

Abstract: Large-scale multilingual evaluations, such as MEGA, often include only a handful of African languages due to the scarcity of high-qualityevaluation data and the limited discoverability of existing African datasets. This lack of representation hinders comprehensive LLM evaluation across a diverse range of languages and tasks. To address these challenges, we introduce AFROBENCH—a multi-task benchmark for evaluating the performance of LLMs across 64 African languages, 15 tasks and 22 datasets. AFROBENCH consists of nine natural language understanding datasets, six text generation datasets, six knowledge and question answering tasks, and one mathematical reasoning task. We present results comparing the performance of prompting LLMs to fine-tuned baselines based on BERT and T5-style models. Our results suggest large gaps in performance between high-resource languages, such as English, and African languages across most tasks; but performance also varies based on the availability of monolingual data resources. Our findings confirm that performance on African languages continues to remain a hurdle for current LLMs, underscoring the need for additional efforts to close this gap.

## introduction large language models (llms) have risen to the forefront of natural language processing (nlp) and have also become increasingly commercially viable. these models have empirically demonstrated strong performance across a variety of nlp tasks and languages @xcite @xcite . however, their performance on low-resource languages (lrls), such as african languages, is largely understudied. this is problematic because there is a significant disparity in the coverage of languages by nlp technologies. @xcite al. (2020) that over 90% of the world's 7000+ languages are under-studied by the nlp community. ideally, approaches to enhance language understanding should be applicable to all languages. while there have been evaluations of llm performance in several languages @xcite @xcite , the evaluations have primarily focused on closed models like gpt-3.5 @xcite and @xcite . megaverse @xcite extended the evaluation to more models such as palm 2 @xcite and llama 2 @xcite , mistral @xcite , gemma @xcite and gemini pro @xcite . however, previous evaluation face two main issues: (1) they cover only a few tasks for african languages, for example, megaverse only evaluated on part-of-speech, named entity recognition, and cross-lingual question answering for african languages, due to poor discoverability of african languages benchmarks, limited available evaluation data, and bias in the selection of languages covered in the evaluation. 2 (2) evaluation of llms needs benchmark # tasks # datasets # african lang. # llms closed llms evaluated dominant task(s) to be continuous, since many new llms have been released with improved multilingual abilities, but comprehensive evaluations for african languages remain limited. in this paper, we address the challenges of previous large-scale llm evaluation by introducing a new carefully curated benchmark known as afrobench which comprises 15 tasks, 22 evaluation datasets, and 64 indigenous african languages. afrobench consists of nine natural language understanding tasks, six text generation tasks, six question answering (qa) tasks, and one mathematical reasoning task. finally, we created a new evaluation dataset, afriadr, a dataset for restoring diacritic marks and tonal accents in african language texts. leveraging afrobench, we conduct an extensive analysis of llms performance on african languages from different language families and geographical locations. for our evaluation, we compute the average performance score over the 15 tasks covered in afrobench. additionally, we introduce afrobench-lite that only covers a subset of seven tasks and 14 diverse languages from afrobench, which reduces the evaluation cost for a newly introduced llm on our leaderboard. figure 1 shows our evaluation on afrobench. we find that proprietary models such as gpt-4o and gemini-1.5 pro achieve a +13 score improvement over gemma 2 27b, our best-performing open model. we also compared the performance on english to 14 african languages, finding that gpt-4o and gemma 2 27b achieve better performance than african languages by over +25 and +40 points, respectively. this shows that the gap in the multilingual abilities of open models is wider than that of proprietary models. finally, we compare the performance of llms to fine-tuned models based on afroxlmr @xcite , afriteva v2 t5 model @xcite and nllb @xcite when training data is present. results show that prompting llms often yields lower average performance than the fine-tuned baselines. our findings show that greater effort is needed to close the performance gap between llms on highresource and african languages 2 related work large language model evaluation: accurate and reproducible evaluation of llms is important as the number of released models continues to grow. as these models are integrated into various applications, developing robust evaluation frameworks becomes critical for understanding their true capabilities and limitations. as a result, the community has worked on developing evaluation frameworks @xcite @xcite , leaderboards @xcite @xcite and benchmarks @xcite @xcite . while each of these evaluation tools focuses on assessing specific aspects of language model capabilities, from basic linguistic understanding to complex reasoning tasks, the development of truly comprehensive benchmarks remains a significant challenge @xcite . these challenges stem from the complex nature of language understanding and the stochastic nature of language models. multilingual llm benchmarks: benchmarks serve as a standard for measuring how systems have improved over time on specific tasks and metrics. in the context of llms, multilingual benchmarks are crucial for assessing both the quality and practical utility of these models across diverse languages and tasks. our primary focus lies in understanding llm performance specifically for african languages, with several notable benchmarks having emerged in recent years to address this need. chatgpt-mt @xcite evaluated the translation capability of gpt-4 and they find that it demonstrates strong performance on high-resource languages, but performance on low-resource languages is subpar. belebele @xcite is a qa task covering 122 languages, including 28 text classification sentiment analysis (f1) news topic classification (acc) intent classification (acc) hate speech classification (f1) natural language inference (acc) ## afrobench afrobench is a comprehensive llm evaluation benchmark designed to assess both proprietary and open llms across diverse natural language processing (nlp) tasks in african languages. as shown in figure 2 , the benchmark encompasses 15 distinct tasks, spanning natural language generation (nlg) and natural language understanding (nlu), incorporating 22 curated datasets in 64 african languages. these evaluation tasks extend beyond traditional nlp benchmarks, such as text classification and named entity recognition, to include more challenging tasks such as mathematical reasoning and knowledge qa. each task within afrobench has been carefully selected to assess different aspects of language model capabilities, from basic linguistic capabilities to more complex reasoning abilities. afrobench also provides valuable insights into model behavior across different african language families and their unique linguistic features. all tasks within afrobench are evaluated using both zero-shot and few-shot prompting to guide model responses. to ensure consistent and reliable evaluation, we implement task-specific response constraints to facilitate systematic extraction and analysis of model outputs. for completion, we compare against state-of-the-art (sota) encoder-only and encoder-decoder architectures that have previously demonstrated superior performance on individual tasks within the benchmark. this enables us to directly compare the performance of specialized models to general-purpose llms. ## conclusion in this paper, we introduce a new benchmark, afrobench, that aggregates existing evaluation datasets for african languages, and adds a new dataset focused on diacritics restoration. afrobench comprises 15 nlp tasks, 22 datasets, and 64 african languages under-represented in nlp. we evaluate the performance of several closed and open llms on these tasks, showing that they consistently fall behind the fine-tuned baselines. we also observe a large performance gap compared to english, although the gap is smaller for closed models such as gpt-4o and gemini-1.5 pro. through this benchmark, we have created a leaderboard focusing on llm evaluation for african languages, which will be maintained going forward with additional tasks, llms, and languages. we will be releasing our prompt and task configurations to eleuther's lm-eval. we hope this encourages the development of more africancentric llms for african languages. 8 our aim is to continuously add newer llms to the leaderboard, we demonstrate this by adding the following llms to the afrobench-lite: lugha-llama (an african-centric llm) (buzaaba et al., 2025), gpt-4.1, gemini-2.0-flash, and llama 4 400b (maverick), as shown in appendix e. 8 acknowledgment this research was supported in part by the natural sciences and engineering research council (nserc) of canada. additional funding is provided by microsoft via the accelerating foundation models research program. we are grateful for the support from ivado and the canada first research excellence fund. we would also like to thank google cloud for the gcp credits awarded through the gemma 2 academic program, and openai for providing us with api credits through their researcher access api program. finally, we thank israel abebe for contributing the inference results of gpt-4.1 and lugha-llama to the afrobench-lite leaderboard.