The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants (2024)

Abstract: We present Belebele, a multiple-choice machine reading comprehension (MRC) dataset spanning 122 language variants. Significantly expanding the language coverage of natural language understanding (NLU) benchmarks, this dataset enables the evaluation of text models in high-, medium-, and low-resource languages. Each question is based on a short passage from the FLORES-200 dataset and has four multiple-choice answers. The questions were carefully curated to discriminate between models with different levels of general language comprehension. The English dataset on its own proves difficult enough to challenge state-of-the-art language models. Being fully parallel, this dataset enables direct comparison of model performance across all languages. We use this dataset to evaluate the capabilities of multilingual masked language models (MLMs) and large language models (LLMs). We present extensive results and findings, notably that despite significant cross-lingual transfer in English-centric LLMs, much smaller MLMs pretrained on balanced multilingual data still understand far more languages. Overall, Belebele opens up new avenues for evaluating and analyzing the multilingual capabilities of NLP systems.

## introduction the absence of high-quality, parallel evaluation benchmarks is a major obstacle in assessing the text comprehension capabilities of multilingual models. nlp datasets with high language coverage do exist, such as @xcite , but they primarily focus on machine translation. popular multilingual evaluation benchmarks, such as multilingual question answering @xcite , natural language inference (nli) @xcite , and summarization @xcite , altogether only cover around 30 languages. and while understanding and generative text services are used across the globe in 100+ languages, the lack of labeled data provides a major obstacle to building functional systems in most languages. simultaneously, large language models (llms) have become increasingly popular. certain llms, like bloom @xcite , are trained on multilingual data and tout their innate multilingual capabilities. others like gpt-3 @xcite and llama @xcite have demonstrated multilingual competence despite their training data being predominantly in english. even so, llms benefit from pretraining data that is linguistically diverse, intentionally or not, as well as from cross-lingual transfer @xcite @xcite . but how multilingual are these models really? beyond llms, significant scientific progress needs to be made before nlp systems can be built effectively and efficiently in low-resource languages. many modeling techniques are being presented as languageagnostic but have only truly been evaluated in a small number of languages @xcite , risking not being applicable to diverse typologically phenomena @xcite . we believe that largescale, parallel, and discriminative datasets are crucial for studying the multilingual capabilities of such models and understanding how the technological disparity between high-and low-resource languages is evolving. in this paper, we present a fundamental natural language understanding benchmark to evaluate language models across 122 language variants from around the world foot_0 , called belebele foot_1 . the dataset contains 900 unique multiple-choice reading comprehension passages and questions. the questions have been carefully crafted to discriminate between models with varying competence in language comprehension. while the questions do not necessarily require higher levels of knowledge or reasoning, they favor generalizable nlu models and deliberately punish biased models. the english questions on their own present a significant challenge to numerous models, while humans are capable of answering the questions with near-perfect accuracy. the first of its scale, belebele is parallel across all languages, facilitating a direct comparison of model performance across all languages. the dataset covers typologically diverse languages across 29 scripts and 27 language families. seven languages are included in two separate scripts, resulting in one of the first nlp benchmarks for the romanized variants of hindi, urdu, bengali, nepali, and sinhala. we further detail our data collection process and the resulting corpus in section 3. the dataset enables evaluation of mono-and multi-lingual models, but the parallel nature also enables a number of cross-lingual evaluation settings. we evaluate several masked language models (mlms) after fine-tuning on an english training set as well as with the assistance of machine translation (translate-train-all). for llms, we evaluate several models using in-context learning and also instruction-tuned models via zero-shot. we discuss our results in section 5. ## the belebele dataset we opted to create multiple-choice questions and answers in english and then translate, as opposed to creating resources natively in each language. many of the advantages to this approach outlined in @xcite remain. most importantly, this leads to significantly more similar sets of samples across languages, enabling direct score comparison. the process for creating the dataset is summarized in figure 2 . ## experiments thanks to belebele, we are able to evaluate numerous models and establish baseline performances across 122 language variants. we compare performance between popular multilingual mlms and llms in several settings. for all, accuracy is the central metric. with 4 candidate answers for each question, the expected accuracy for sequence classification models that guess randomly is 0.25. foot_2 4.1 evaluated models masked language models (mlms) we evaluate three different models, xlm-v @xcite , infoxlm @xcite , and xlm-r @xcite . all the evaluated mlms have been pretrained on intentionally multilingual corpi inclusive of about 100 languages. the pretraining data in high-resource languages is typically down-sampled while low-resource languages are up-sampled in order to favor multilingual performance @xcite . in addition, all their subword tokenizers @xcite are trained on multilingual corpora, making them better suited for multilingual text. large language models we evaluate gpt3.5-turbo, falcon, and llama (1 and 2). gpt3.5-turbo is a model optimized for chat based on gpt-3 @xcite available through openai apis foot_3 . limited details have been disclosed about the pretraining and fine-tuning data. foot_4 llama 1 @xcite is a collection of decoderonly transformers models trained on 1t (for 7b, 13b) or 1.4t (for 30b, 65b) tokens of publicly available online data, while llama 2 @xcite is pretrained on about 2t. we evaluate all four pretrained checkpoints for llama 1. we evaluate both llama 2 70b's pretrained version and its chat version instruction-fine-tuned for safe dialog purposes (a.k.a. llama-2-chat). we also evaluate falcon 40b, which was pretrained on 1t extensively filtered web-crawled samples @xcite . while llama 1 was reportedly trained in 20 languages with latin and cyrillic scripts, non-english text accounts for less than 4.5% of the pretraining corpus @xcite . llama 2 pretraining data is made of 89.7% of english data, 8.4% unidentified, and a tiny 1.9% belonging to 26 other languages foot_5 @xcite . both series use the same bpe-based tokenizers @xcite . splitting unicode characters into bytes also helps llama avoid out-of-vocabulary errors. in ## results we provide summarized results in ## conclusion a fundamental limitation to conducting sound evaluations of the capabilities of language models in low-, or even moderate-, resource languages is the availability of annotated benchmarks. this paper presents a massive dataset, belebele, consisting of passages and multiple-choice questions evaluating reading comprehension in 122 languages. this benchmark enables critical evaluation of reading comprehension capabilities of llms in english and top languages. in addition, the dataset is the first of its kind in many medium-and low-resource languages, enabling unprecedented insight into the multilingual capabilities of language models. we present results from a number of popular mlms and llms in different evaluation settings and find that while large vocabulary size and balanced pretraining data correlates with highest model performance on medium-and low-resource languages, even english-centric llms can go a long way and generalize to over 30 languages. for future work, we hope the many evaluations and experiments now possible will allow for deeper dives into current language models. belebele can also complement investigations into specific model capabilities (e.g. reasoning), leading to a broader understanding of the relationship between such abilities and multilinguality. as a result, we believe belebele will soon unveil further insights that contribute to the development of nlp systems beyond high-resource languages.