Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages (2023)

Abstract: The NLP community has mainly focused on scaling Large Language Models (LLMs) vertically, i.e., making them better for about 100 languages. We instead scale LLMs horizontally: we create, through continued pretraining, Glot500-m, an LLM that covers 511 predominantly low-resource languages. An important part of this effort is to collect and clean Glot500-c, a corpus that covers these 511 languages and allows us to train Glot500-m. We evaluate Glot500-m on five diverse tasks across these languages. We observe large improvements for both high-resource and low-resource languages compared to an XLM-R baseline. Our analysis shows that no single factor explains the quality of multilingual LLM representations. Rather, a combination of factors determines quality including corpus size, script, “help” from related languages and the total capacity of the model. Our work addresses an important goal of NLP research: we should notlimit NLP to a small fraction of the world’s languages and instead strive to support as many languages as possible to bring the benefits of NLP technology to all languages and cultures. Code, data and models are available at https://github.com/cisnlp/Glot500 .

## introduction the nlp community has mainly focused on scaling large language models (llms) vertically, i.e., deepening their understanding of high-resource languages by scaling up parameters and training data. while this approach has revolutionized nlp, the achievements are largely limited to high-resource languages. examples of "vertical" llms are gpt3 @xcite , @xcite and @xcite . in this paper, we create glot500-m, a model that instead focuses on scaling multilingual llms horizontally, i.e., scaling to a large number of languages the great * equal contribution. majority of which is low-resource. as llms are essential for progress in nlp, lack of llms supporting low-resource languages is a serious impediment to bringing nlp to all of the world's languages and cultures. our goal is to address this need with the creation of glot500-m. 1 existing multilingual llms support only about 100 @xcite out of the 7000 languages of the world. these supported languages are the ones for which large amounts of training data are available through projects such as oscar @xcite and the wikipedia dumps. 2 following @xcite , we refer to the 100 languages covered by @xcite as head languages and to the remaining languages as tail languages. this terminology is motivated by the skewed distribution of available data per language: for the best-resourced languages there are huge corpora available, but for the long tail of languages, only small corpora exist. this is a key problem we address: the availability of data for tail languages is limited compared to head languages. as a result, tail languages have often been ignored by language technologies @xcite . although there exists some work on machine translation for a large number of tail languages @xcite , existing llms for tail languages are limited to a relatively small number of languages @xcite @xcite . in this paper, we address this gap. our work has three parts. (i) corpus collection. we collect glot2000-c, a corpus covering thousands of tail languages. (ii) model training. using glot500-c, a subset of glot2000-c, we train glot500-m, an llm covering 511 languages. (iii) validation. we conduct an extensive evaluation of the quality of glot500-m's representations of tail languages on a diverse suite of tasks. in more detail, corpus collection considers three major sources: websites that are known to publish content in specific languages, corpora with classified multilingual content and datasets published in specific tail languages. the resulting dataset glot2000-c comprises 700gb in 2266 languages collected from ≈150 sources. after cleaning and deduplication, we create the subset glot500-c, consisting of 511 languages and 534 language-scripts (where we define a language-script as a combination of iso 639-3 3 and script) to train glot500-m. our criterion for including a language-script in glot500-c is that it includes more than 30,000 sentences. ## related work training multilingual llms using the masked language modeling (mlm) objective is effective to achieve cross-lingual representations @xcite . these models can be further improved by incorporating techniques such as discriminative pre-training @xcite and the use of parallel data @xcite . however, this primarily benefits a limited set of languages with large corpora. recent research has attempted to extend existing llms to languages with limited resources. @xcite propose vocabulary extension; ebrahimi and kann (2021) investigate adaptation methods, including mlm and translation language model (tlm) objectives and adapters; @xcite adapt xlm-r to 17 african languages; @xcite expand language models to low-resource languages using bilingual lexicons. alternatively, parameter-efficient fine-tuning adapts pre-trained models to new languages by training a small set of weights effectively @xcite @xcite . @xcite address the "curse of multilinguality" by sharing a part of the model among all languages and having separate modules for each language. we show that the common perception that multilinguality increases as we add more languages, until, from some point, it starts decreasing, is naive. the amount of available data per language and the similarity between languages also play important roles ( §6.8). another approach trains llms from scratch for a limited number of tail languages; e.g., afriberta @xcite and indicnlpsuite @xcite are llms for 11 african languages and 11 indic languages. in concurrent work, @xcite train a multilingual model for 517 african languages on a 42 gb corpus, but without making the model available and with an evaluation on a smaller number of languages than ours. closely related to our work on corpus creation, @xcite and @xcite also create nlp resources for a large number of tail languages. they train a language identifier model and extract textual data for tail languages from largescale web crawls. this approach is effective, but it requires significant computational resources and native speakers for all tail languages. this is hard to do outside of large corporations. @xcite have not made their data available. @xcite have only released a portion of their data in around 200 languages. a key benefit of "horizontally" scaled multilingual llms is transfer from high-to low-resource languages. our evaluation suggests that glot500-m excels at this, but this is not the main focus of our paper. there is a large body of work on crosslingual transfer: @xcite @xcite @xcite @xcite @xcite , inter alia. ## glot2000-c 3.1 data collection one of the major challenges in developing nlp technologies for tail languages is the scarcity of high-quality training data. in this work, we propose a lightweight methodology that is easily replicable for academic labs. we identify tail language data previously published by researchers, publishers and translators and then crawl or download them. by crawling a few websites and compiling data from around 150 different datasets, we amass more than 700gb of text in 2266 languages. we will refer to these sources of data as data sources. our data covers many domains, including religious texts, news articles and scientific papers. some of the data sources are high-quality, verified by native speakers, translators and linguists. others are less reliable such as web crawls and wikipedia dumps. it is therefore necessary to clean the data. for a list of data sources, see §c. ## glot500-m 4.1 vocabulary extension to extend xlm-r's vocabulary, we use sentence-piece (kudo and richardson, 2018) with a unigram language model (kudo, 2018) to train a tokenizer with a vocabulary size of 250k on glot500-c. we sample data from different language-scripts according to a multinomial distribution, with 𝛼=.3. the amount we sample for head languages is the same as tail languages with the lowest amount; this favors tail languages -head languages are already well learned by xlm-r. we merge the obtained tokens with xlm-r's vocabulary. about 100k new tokens were in fact old tokens, i.e., already part of xlm-r's vocabulary. we take the probabilities of the (genuinely) new tokens directly from sen-tencepiece. after adding the 151k new tokens to xlm-r's vocabulary (which has size 250k), the vocabulary size of glot500-m is 401k. we could also calculate probabilities of existing and new tokens over a mixture of original xlm-r training corpus and @xcite . for head languages, the percentage of changed tokens using the new tokenizer compared to the original tokenizer ranges from 0.2% to 50%. however, we found no relationship between percentage of changed tokens and change in performance on downstream tasks. thus, there was little effect of tokenization in our experiments. ## experimental setup for most tail languages, there are no manually labeled evaluation data. we therefore adopt a mixed evaluation strategy: based partly on human labels, partly on evaluation methods that are applicable to many languages without requiring gold data. ## experiments in this section, we discuss aggregate results. for detailed results, see §d and §e. ## conclusion and future work we collect and data-clean glot500-c, a large corpus of hundreds of usually neglected tail (i.e., long-tail) languages and create glot500-m, an llm that is trained on glot500-c and covers these languages. we evaluate glot500-m on six tasks that allow us to evaluate almost all languages. we observe large improvements for both head and tail languages compared to xlm-r. our analysis shows that no single factor fully explains the quality of the representation of a language in a multilingual model. rather, a combination of factors is important, including corpus size, script, "help" from related languages and the total capacity of the model. this work is the first to create a language model on a dataset of several hundreds of gigabytes and to make it publicly available for such a large and diverse number of low-resource languages. in future research, we would like to train larger models to further investigate the effect of model size, distill highly multilingual models for resource-efficient deployment, explore alternatives to continued pretraining and use models for more tail language downstream tasks.