When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages (2024)

Abstract: Multilingual language models are widely used to extend NLP systems to low-resource languages. However, concrete evidence for the effects of multilinguality on language modeling performance in individual languages remains scarce. Here, we pre-train over 10,000 monolingual and multilingual language models for over 250 languages, including multiple language families that are under-studied in NLP. We assess how language modeling performance in each language varies as a function of (1) monolingual dataset size, (2) added multilingual dataset size, (3) linguistic similarity of the added languages, and (4) model size (up to 45M parameters). We find that in moderation, adding multilingual data improves low-resource language modeling performance, similar to increasing low-resource dataset sizes by up to 33%. Improvements depend on the syntactic similarity of the added multilingual data, with marginal additional effects of vocabulary overlap. However, high-resource languages consistently perform worse in multilingual pre-training scenarios. As dataset sizes increase, adding multilingual data begins to hurt performance for both low-resource and high-resource languages, likely due to limited model capacity (the “curse of multilinguality”). These results suggest that massively multilingual pre-training may not be optimal for any languages involved, but that more targeted models can significantly improve performance.

## introduction multilingual language models have been a fixture of natural language processing (nlp) research nearly since the introduction of transformer language models @xcite . these models are often pre-trained on over 100 languages simultaneously, and they are widely used for nlp tasks in low-resource languages @xcite @xcite , cross-lingual transfer learning @xcite , and multilingual text generation @xcite . however, while multilingual language models produce strong results across many languages, multilingual pre-training work almost exclusively focuses on pre-training a small number of models with some fixed distribution over languages (e.g. mbert, xlm-r, xglm, and bloom; @xcite @xcite @xcite . this distribution over languages typically favors high-resource languages spoken in regions with high economic influence @xcite . thus, it is largely unknown how different pretraining language distributions, such as different quantities of multilingual data or different selections of languages, affect multilingual language model performance in different languages. multilingual models have been studied extensively during inference and fine-tuning @xcite @xcite @xcite @xcite @xcite , but these studies generally rely on the same sets of pre-trained models. for pre-training, there is mixed evidence for the benefits of multilingual vs. monolingual data @xcite @xcite . as multilingual language models are increasingly used without task-specific fine-tuning (e.g. for text generation; @xcite ), foot_0 it is critical to understand how multilingual pre-training affects raw language modeling performance in individual languages. in our work, we investigate the effects of different multilingual pre-training distributions on effects are quantified using the estimated monolingual dataset size that would achieve similar performance. adding 1b tokens of multilingual data is similar to adding 22% (low-resource) or removing 63% (high-resource) of the monolingual dataset. shaded regions are 99% confidence intervals for the mean across languages. our main contributions are: foot_2 thus, the benefits of multilinguality on raw language modeling performance appear restricted to cases where both (1) the model targets performance in low-resource languages and (2) the model has enough capacity for the added multilingual data. if these assumptions hold, the multilingual data should be from languages that are linguistically similar to the target low-resource languages. however, when optimizing performance for highresource languages, multilingual models are likely to degrade performance in individual languages. the curse of multilinguality. to extend language models to low-resource languages, researchers often train a single model on a large number of languages, including low-resource languages @xcite @xcite @xcite . oftentimes, better performance is observed when languages are either closely related or focused in a specific region @xcite @xcite . however, @xcite find that pre-training on an excessive number of languages hurts model performance in each language, evaluating five subsets of languages on downstream tasks in 16 languages. this phenomenon is known as the curse of multilinguality or negative interference @xcite . indeed, monolingual language models often have better language modeling performance than massively multilingual models @xcite . however, @xcite find that the curse of multilinguality may simply be a result of lower quality tokenization per language. further contradicting the curse of multilinguality, @xcite find that for low-resource languages, multilingual pre-training does improve downstream task perfor-mance relative to monolingual pre-training, and @xcite observe better cross-lingual transfer performance when a wider variety of languages is seen during during pre-training. thus, the precise effects of multilinguality on low-resource and high-resource languages remains unclear. to isolate these effects, we evaluate language modeling performance in 252 languages while systematically varying monolingual dataset size, multilingual dataset size, model size, and linguistic similarity of added languages during pre-training. this contrasts with previous studies that focus on individual (massively) multilingual models such as mbert or xlm-r. our approach allows us to determine how such models perform after varying pre-training languages and language distributions. ## collecting a multilingual dataset conducting controlled multilingual language modeling experiments requires a large multilingual dataset. notably, broad language coverage is a consistent issue in nlp research @xcite @xcite . here, we collect text corpora from 24 multilingual data sources such as oscar @xcite , wikipedia @xcite , and nllb @xcite . our sources are reported in §a. we merge the corpora per language, and we deduplicate repeated sequences of 100 utf-8 bytes @xcite . restricting each language to a maximum of 1b tokens, our dataset contains 41.4b tokens in 1572 languages. this includes 252 languages with the required 1.5m tokens for our language modeling study. despite this fairly stringent token requirement, our 252 languages cover five continents, 29 language families, and 30 scripts (i.e. writing systems). figure 1 shows a geographic map of our 252 languages, using coordinates from glottolog @xcite . our list of languages with corresponding token counts is reported in §g. ## monolingual baselines and metrics to study effects of multilinguality on language modeling performance in individual languages, we first need a method to quantify performance in those languages. thus, we pre-train monolingual baseline models for our 252 languages, to use as comparison points for multilingual models. for each language l, we estimate the number of monolingual tokens in l required to achieve a given level of performance in l with a given model size. we later use this estimated number of monolingual tokens as an interpretable performance metric for multilingual models. ## pre-training multilingual models finally, we pre-train multilingual language models that vary along four dimensions: monolingual data quantity, added multilingual data quantity, model size, and linguistic similarity of the added languages. each multilingual model is pre-trained with a specified target language, keeping monolingual tokenization for that language fixed during both pre-training and evaluation. the multilingual models are pre-trained identically to the monolingual baselines in §4, except with added multilingual data (10m, 100m, or 1b tokens). the multilingual data is randomly interspersed with the monolingual pre-training data in the target language. target language evaluation loss curves are included in §c. in total, we pre-train 8454 multilingual language models ranging from 8m to 45m parameters. multilingual tokenizers. perplexity and loglikelihood evaluations within a language l are only comparable when they use the same tokenizer. thus, we must keep the monolingual tokenizer fixed for any model evaluated on l. however, fixing tokenization for multiple languages simultaneously results in intractable vocabulary sizes. for example, 252 languages × 32k tokens would result in a vocabulary size of 8.1m tokens, requiring 1.0b embedding parameters even with our smallest embedding size of 128. to avoid intractable parameter counts, we pre-train multilingual language models that each keep tokenization fixed for only one language, which we call the target language for that model. in each multilingual model, the nontarget languages share a multilingual tokenizer with vocabulary size 32k, trained on 10k randomlysampled lines from each added language. the target language and added multilingual datasets are tokenized separately, and the token ids are merged for the shared vocabulary items. this merged tokenization process ensures that the target language tokenization remains unchanged across models. ## multilingual model results to reflect low-resource to high-resource language scenarios, we primarily separate results based on monolingual data quantity (low and med-low resource in §6.1, high and med-high resource in §6.2). in each scenario, we consider the effects of adding different multilingual data quantities with different levels of linguistic similarity, across all three model sizes. overall, we find that performance in low-resource languages improves when we add moderate amounts of multilingual data ( §6.1). the amount of improvement depends on the syntactic similarity of the added languages, with small additional effects of lexical (vocabulary) similarity. high-resource language performance consistently degrades when we add multilingual data ( §6.2). larger models have smaller degradations for highresource languages and larger improvements for low-resource languages in multilingual scenarios, suggesting that many drawbacks of multilinguality are due to limited model capacity. ## discussion our results demonstrate that for low-resource languages, multilingual language models yield some benefits. in the optimal case from our study, the benefits are similar to increasing the low-resource dataset size by about 33% ( §6.1). hence, in scenarios where collecting additional data is difficult (e.g. languages spoken in remote geographic locations or with few speakers), pre-training multilingual models may be a worthwhile endeavor. in these cases, the models should be pre-trained with multilingual data from maximally similar languages, and it should be ensured that the models have capacity for the added multilingual data along with the target language data. however, in other cases, it may be more practical to find or collect more data in the target language itself (e.g. if collecting 50% more target language data is feasible). for high-resource languages, multilingual language models yield worse performance than the comparable monolingual model in essentially all cases. degradations can be similar to reducing high-resource dataset sizes by over 85% ( §6.2). these degradations can be mitigated by pre-training larger models, which also appear to maximize benefits for low-resource languages. however, when pre-training language models even on the order of tens of high-resource languages @xcite @xcite , a model sufficiently large to accommodate all of the languages' data without hitting capacity limitations would likely be impractically large. even if existing language models are severely over-parameterized, there is evidence that 70bparameter models are required just for english @xcite . if only considering individual language performance, pre-training targeted language-specific models is likely to be far more efficient than a single massively multilingual model. ## conclusion our work systematically evaluates the effects of multilingual pre-training on language modeling performance in 252 languages. we pre-train over 10,000 monolingual and multilingual language models, varying monolingual dataset sizes, multilingual dataset sizes, linguistic similarity of the multilingual data, and model sizes. we find that adding multilingual data in similar languages improves performance for low-resource languages, but improvements decrease as models reach capacity limitations. multilingual data consistently hurts high-resource language performance. we quantify both of these effects in terms of comparable monolingual dataset sizes. our results suggest that while multilingual language models may be beneficial for low-resource scenarios, massively multilingual models may be far less practical than previously assumed for raw language modeling.