Taxi1500: A Dataset for Multilingual Text Classification in 1500 Languages (2025)

Abstract: While broad-coverage multilingual natural language processing tools have been developed, a significant portion of the world’s over 7000 languages are still neglected. One reason is the lack of evaluation datasets that cover a diverse range of languages, particularly those that are low-resource or endangered. To address this gap, we present a large-scale text classification dataset encompassing 1504 languages many of which have otherwise limited or no annotated data. This dataset is constructed using parallel translations of the Bible. We develop relevant topics, annotate the English data through crowdsourcing and project these annotations onto other languages via aligned verses. We benchmark a range of existing multilingual models on this dataset. We make our dataset and code available to the public.

## introduction language inequality is a real issue in the world today as minority languages are under-represented and often excluded from language technologies @xcite . the lack of technological support for minority languages in communities around the globe has a significant impact on the experience of their users and is commonly a cause for virtual barriers such as the digital divide. 2 recent developments in language technologies have led to a surge in multilingual pre-trained language models (mplms), such as mbert @xcite , xlm-r @xcite , and glot500 @xcite , and large language models (llms) like bloom @xcite and @xcite . the lack of knowledge in low-resource languages often causes language technologies to overlook important features from typologically diverse languages @xcite . a key reason why many low-resource languages remain neglected is the scarcity of evaluation datasets. for example, mplms like mbert and xlm-r are evaluated on much fewer languages than they are trained for, largely due to the limited availability of languages in most existing benchmark datasets. as a solution, we propose a dataset that covers more than 1500 languages. we use translations of the bible as our source and develop topics that are well generalized (so as to apply to many verses), but at the same time are not overly abstract. we obtain annotations for the english verses using crowdsourcing. because the bible is aligned at the verse level, we can easily project annotations from the english side to all other languages. to ensure the quality of our annotated data, we calculate the inter-annotator agreement using krippendorff's α. in addition, we introduce a benchmark for four mplms and three llms. we present evaluation results using mbert, xlm-r-base, xlm-r-large and glot500 for all languages and llama2-7b @xcite , mistral-7b @xcite , and bloom (560m, 1b, 3b and 7b) for 64 selected languages in our dataset. glot500 demonstrates better multilingual capabilities, attributed to its larger number of languages in the pretraining data. moreover, the evaluation of llms reveals that their performance (based on a few low-resource prompts) is comparable to finetuned mplms. ## related works to date, most datasets that can be used for multilingual task evaluation @xcite @xcite @xcite cover no more than a few hundred languages, a small number compared to the world's 7000 languages. in current nlp research, parallel corpora play a crucial role as they serve as cross-lingual bridges, enabling the processing and understanding of less known languages through other languages. in this study, we employ translations of the bible as the source of parallel data, utilizing both the parallel bible corpus @xcite , covering 1304 languages, as well as 1000langs, 3 bible translations collected from multiple bible websites, resulting in a total coverage of 1504 languages. ## dataset creation since many low-resource languages only have a translated new testament, we use verses from the new testament to build our dataset. in the initial annotation phase, we gather topics using latent dirichlet allocation (lda), 4 online preaching websites with topics of bible verses, 5 and insights from linguists. we then utilize amazon mechanical turk (mturk) 6 for crowdsourcing to assess the quality of the selected topics. we conduct seven rounds of topic selection and show the details in we assume annotation quality issues may arise if 1) the task is confusing, or 2) the worker lacks care or attention. we provide detailed guidelines and examples along with the task. all workers must also pass a qualification test to ensure they fully understand the task. for quality control, we implement a performance threshold. we create "pseudo gold standard" data based on majority votes from all annotators and calculate each worker's macro f1 score. if that score is below 0.40 for a worker, their annotations are rejected, and the verses are republished for re-annotation. we use krippendorff's α (k-α) to compute interannotator agreement. k-α is chosen for its ability to handle missing annotations in the dataset. this is important because each worker only annotates 3 https://github.com/ehsanasgari/1000langs 4 https://tinyurl.com/5fja5yvz 5 https://www.georgeho.org/lda-sucks/ 6 www.mturk.com each dot in the plot stands for a threshold of the required minimum @xmath0, 4, 5, 6, 7, 8, 9} for a verse to be accepted. a subset of the verses. ## dataset the final dataset we obtain consists of verses categorized into six topics: faith, grace, sin, violence, description, and recommendation. ## benchmarking to illustrate its utility, we use taxi1500 to evaluate four pre-trained multilingual models: mbert, xlm-r-base, xlm-r-large, and glot500, and three llms: llama2, bloom, and mistral using a selection of 64 languages from taxi1500. for a fair comparison, we split languages in our dataset into three subsets, namely head languages, glot500-only languages, and tail languages. head languages are languages that are in the pre-training data of all four models. glot500-only languages are languages that are only in the pre-training data of glot500. tail languages are languages that are not in the pre-training data of any model. details of the setup are provided in appendix a. ## conclusion in this paper, we propose a text classification dataset consisting of 1504 languages by annotating english bible verses through crowdsoucing and projecting the labels to other languages with parallel data. we benchmark several widely used multilingual language models and llms using our dataset. the results demonstrate that taxi1500 can effectively evaluate multilingual capabilities across different models.