2 M - BELEBELE : Highly Multilingual Speech and A merican S ign L anguage Comprehension Dataset Download PDF (2025)

Abstract: We introduce the first highly multilingual speech and American Sign Language (ASL) comprehension dataset by extending BELEBELE. Our dataset covers 91 spoken languages at the intersection of BELEBELE and FLEURS, and one sign language (ASL). As a by-product we also extend the Automatic Speech Recognition Benchmark, FLEURS, by 20%. We evaluate 2M-BELEBELE dataset for both 5-shot and zero-shot settings and across languages, the speech comprehension accuracy is ≈ 10% average lower compared to reading comprehension.

## introduction from an ai perspective, text understanding and generation services are used globally in more than a hundred languages, but the scarcity of labeled data poses a significant challenge to developing functional systems in most languages. although natural language processing (nlp) datasets with extensive language coverage, such as @xcite , are available, they mainly concentrate on machine translation (mt). multilingual evaluation benchmarks such as those for multilingual question answering @xcite , natural language inference @xcite , summarization @xcite , and reasoning datasets @xcite collectively cover only about 30 languages. furthermore, the extension of such datasets to speech or american sign language (asl) is lacking, with the exception of fleurs @xcite , which is based on flores-200. the recent belebele benchmark is the first corpus that addresses text reading comprehension for a large number of languages following a multiway parallel approach @xcite . the entire belebele text statistics are summarized in as a by-product of 2m-belebele, we also extend the fleurs dataset (which is widely used to benchmark language identification and asr) by providing recordings for more flores-200 sentences than were previously available and adding sign language, creating a new 2m-flores. this 2m-flores extends fleurs by 20%. finally, we provide a very basic set of experiments that evaluate 2m-belebele and provide some reference results. we use direct and/or cascaded systems to evaluate 2m-belebele dataset (section 4). we also list several further experimentation that 2m-belebele unblocks. note that the main contribution of this paper is the creation of the first highly multilingual speech and sign comprehension dataset. the complete set of experiments is out of the scope of this paper @xcite . by open-sourcing our dataset, we encourage the scientific community to pursue such experimentation. ## 2m-belebele fleurs and belebele passage alignment. since belebele uses passages constructed from sentences in the flores-200 dataset, and fleurs @xcite is a human speech version of flores-200 for a subset of its languages, we create a speech version of belebele by aligning its passages with the speech segments available in fleurs. this extension can be done without extra human annotation, just by computing the alignment between fleurs and bele-bele passages. however, such alignment does not cover the entire belebele corpus because fleurs does not cover the entirety of flores-200. there are 91 languages shared between fleurs and belebele. fleurs does not cover the same passages as belebele in all those 91 languages, which means that some languages have more speech passages than others. in general, we are able to match almost ≈ 80% of the passages. figure 2 shows the number of fleurs paragraphs we can match, thus obtaining the number of paragraphs that must be recorded in order to cover all passages belebele. ## experiments we evaluate 2m-belebele, and compare performance across modalities. our comparison is limited in number of systems and combination of modalities. 2m-belebele offers the opportunity to check multimodal comprehension by combining speech/text/sign passages; questions and answers. in our case, we only provide results for entire text passages, questions and answers and speech passages, text questions and answers. a more comprehensive set of experiments is out of the scope of this paper, which aims at unblocking such experimentation by open-sourcing the dataset itself. systems. we use the speech section of the 2m-belebele dataset to evaluate the speech comprehension task with a cascaded system consisting of first speech recognition (asr) using the whisper-large-v3 model @xcite (hereinafter, whisper) and seamlessm4t (corresponding to seamlessm4t-large v2) model (seamlesscommunicationteam, 2025) feeding into llama-3 foot_0 . we also provide results with a unified system spiritlm @xcite , which is a multimodal language model that freely mixes text and speech. since the size of this model is 7b and is based on llama-2, we also add a comparison to the llama-2 model. we compare these results with llama-3 and llama-3-chat using the belebele text passage as input. languages for the mentioned systems, we report the results in 5-shot in-context learning and zeroshot on 59 languages at the intersection of language coverage between whisper and 2m-belebele and 39 languages at the intersection of whisper, seamlessm4t and 2m-belebele (see zero-shot evaluation. we use the same evaluation strategy as in @xcite . spir-itlm is not available in chat mode. ## conclusions the 2m-belebele data set foot_2 allow evaluating natural language comprehension in a large number of languages, including asl. 2m-belebele is purely human-made and covers belebele passages, questions, and answers for 91 languages in the speech modality and asl. as a by-product, 2m-flores extends fleurs by 20% foot_3 .