SIB -200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects (2024)

Abstract: Despite the progress in building multilingual language models, evaluation is often limited to a few languages with available datasets which excludes a large number of low-resource languages. In this paper, we create SIB-200—a large-scale open-sourced benchmark dataset for topic classification in 205 languages and dialects to address the lack of evaluation dataset for Natural Language Understanding (NLU). For many of the languages covered in SIB-200, this is the first publicly available evaluation dataset for NLU. The dataset is based on Flores-200 machine translation corpus. We annotated the English portion of the dataset and extended the sentence-level annotation to the remaining 204 languages covered in the corpus. Despite the simplicity of this task, our evaluation in full-supervised setting, cross-lingual transfer setting and prompting of large language model setting show that there is still a large gap between the performance of high-resource and low-resource languages when multilingual evaluation is scaled to numerous world languages. We found that languages unseen during the pre-training of multilingual language models, languages from under-represented families (like Nilotic and Altantic-Congo), and languages from the regions of Africa, Americas, Oceania and South East Asia, often have the lowest performance on our topic classification dataset. We hope our dataset %will encourages a more inclusive evaluation of multilingual language models on a more diverse set of languages.

## introduction in the last few years, developing massively multilingual pre-trained language models (plms) to scale to several written languages is an active area of research-e.g. covering 100 languages @xcite @xcite @xcite . however, evaluation is often limited to a few tens of languages with benchmark datasets @xcite @xcite , thus, limiting the large-scale evaluation of current multilingual language models on many languages, especially those truly low-resource languages. while there is evidence from previous works that languages not covered during pre-training often lead to lower performance, such analysis is also limited to a small selection of languages with annotated datasets @xcite @xcite . recently, there is a push to scale evaluation datasets to more than 100 languages, but this requires a very expensive annotation effort in terms of money and time. often, this scaling is only carried out by a large community effort that spans many years like the universal dependency (ud) project @xcite @xcite or financed by bigtech companies @xcite @xcite @xcite . despite these investments in data curation, there are only few benchmarks for natural language understanding (nlu) tasks that cover all the languages seen during the pre-training of multilingual plms @xcite . the largest benchmark datasets that are available for nlu are ud, taxi1500 @xcite , wikiann @xcite , and belebele @xcite for dependency parsing, text classification, named entity recognition, and reading comprehension, respectively. the largest is taxi-1500 for 1500 languages-but it is biased to the religious domain, and some languages are not publicly available due to copyright. wikiann on the other hand, was automatically annotated and with few instances for low-resource languages. ud and belebele were manually annotated and covered between 100-125 languages. however, many languages are still missing in the above datasets. in this paper, we create sib-200-a large-scale open-sourced benchmark dataset for topic classification to address the lack of evaluation datasets for nlu. the dataset is based on flores-200 @xcite )-a multi-way parallel corpus (i.e. same sentences are available in 205 languages). we annotated the english portion of the flores-200 dataset and extend the sentence-level annotation to the remaining 204 languages covered in flores-200. our evaluation shows that there is still a large gap between the performance of high-resource and low-resource languages when multilingual evaluation is scaled to numerous world languages. languages unseen during the pre-training of multilingual plms, languages from under-represented families (like nilotic and altantic-congo), and languages from the regions of africa, americas, oceania and south east asia, often have the lowest performance on our dataset. we also find that simply scaling up the number of languages without scaling up the domains in the pre-training is unhelpful (e.g., glot-500 pre-trained on 500 languages largely under-performs xlm-r pre-trained on 100 languages). it is crucial to mix text from various domains. for languages unseen during pre-training, we show the potential of multilingual language adaptive fine-tuning (maft) 2 @xcite in improving the performance of these languages by leveraging synthetic data for languages with tiny monolingual data (i.e. language with less than 10mb of data). evaluation of this approach on african languages results in significant improvement (up to +5% in accuracy on average) for the previously unseen languages. finally, we extend our evaluation to the zeroshot settings by training individually on english, french, arabic and chinese (simplified) languages using xlm-r @xcite , and performing zero-shot evaluation on other languages. we compared these results with prompting gpt-4 large language models (llms). our results show that llms perform poorly on over 64.7% (or 132 out of 205 languages) of the languages with less than 70% in accuracy while zero-shot adaptation from the english model only leads to performance less than 70% accuracy in 81 languages (or 39.3% of languages) 3 . this shows that leveraging cross-lingual 2 adaptation of an existing multilingual plm to multiple or new sets of languages simultaneously. 3 performance of xlm-r on english is 92.1% in accuracy transfer from high-resource languages is much better than prompting llms for many languages. ## experimental setup here, we describe different categorization of languages, text classification models developed for sib-200 , and the experimental settings (i.e. full supervised setting and zero-shot transfer setting). ## conclusion in this paper, we created sib-200-a large scale open-sourced benchmark dataset for topic classification in 200 languages and dialects to address the lack of evaluation datasets for natural language understanding especially for low-resource languages. we performed extensive evaluation across fullsupervised setting, cross-lingual transfer setting and prompting of llms settings. furthermore, we grouped the 200 languages in different categories based on language families, geographical regions, joshi's class and coverage in multilingual pre-trained language models to provide insights into which group of languages have poor performance on this simple and inclusive benchmark.