Scaling Laws for Multilingual Language Models (2025)

Abstract: We propose a novel scaling law for general-purpose decoder-only language models (LMs) trained on multilingual data, tackling the problem of balancing languages during multilingual pretraining. A primary challenge in studying multilingual scaling is the difficulty of analyzing individual language performance due to cross-lingual transfer. To tackle this, we shift the focus from individual languages to language families. We introduce and validate a hypothesis that the test cross-entropy loss for each language family is determined solely by its own sampling ratio, independent of other languages in the mixture. This insight simplifies the complexity of multilingual scaling and make the analysis scalable to an arbitrary number of languages. Building on this hypothesis, we derive a power-law relationship that links performance with dataset size, model size and sampling ratios. This relationship enables us to predict performance across various combinations of the above three quantities, and derive the optimal sampling ratios at different model scales. To demonstrate the effectiveness and accuracy of our proposed scaling law, we perform a large-scale empirical study, training more than 100 models on 23 languages spanning 5 language families. Our experiments show that the optimal sampling ratios derived from small models (85M parameters) generalize effectively to models that are several orders of magnitude larger (1.2B parameters), offering a resource-efficient approach for multilingual LM training at scale.

## introduction scaling has proven to be a powerful strategy for improving the performance of language models (lms) across a range of tasks @xcite @xcite . due to the enormous cost associated with training * work done during an internship at microsoft. correspondence email: yifeihe3@illinois.edu. although there exist some studies that examine multilingual scaling laws @xcite @xcite , they often focus on the specific problem of neural machine translation (nmt). these studies predominantly utilize encoderdecoder transformer architectures, which are not widely applied to generation tasks. furthermore, they typically restrict their analysis to bilingual settings without capturing the complexities of language interactions. this omission is critical, as cross-lingual transfer, a key benefit in multilingual training, plays a significant role in improving performance across languages @xcite @xcite . to tackle the aforementioned issues, we focus on building scaling laws for general purpose decoderonly lms pretrained on multilingual data. our contributions are grounded in a realistic and novel hypothesis that unlocks a broader understanding of multilingual scaling, making our analysis scalable to an arbitrary number of languages. to substantiate our claims, we conduct a large-scale empirical study by training more ## experimental setup model. we train decoder-only transformer models @xcite in four sizes, ranging from 85m to 1.2b non-embedding parameters. the model sizes are determined by adjusting the number of layers, hidden sizes, and the number of attention heads with details provided in appendix b. data. we focus on the pretraining stage of multilingual language models. we use the common-crawl dataset @xcite . following the approach of lai et al. (2023), we select 23 languages based on diversity and representativeness among the total 100 languages. we follow common practice @xcite to group the languages into five language families based on linguistic similarities: romance, slavic, indic, germanic and sino-tibetan. the detailed languages within each family are presented in ## the multilingual scaling law we study the relationship between performance, model sizes, dataset sizes and sampling ratios. in section 3.1, we motivate the problem of multilingual scaling laws. in section 3.2, we formulate and validate a hypothesis to tackle this problem in a tractable manner. in section 3.3, we propose a power-law relationship between performance and sampling ratios. in section 3.4, we incorporate model and dataset size into this relationship to establish a comprehensive scaling law. finally in section 3.5, we validate the fitting of our scaling law. ## optimal sampling ratios with the general form of the total loss established, we proceed to compute the optimal data mixture defined by p by solving the optimization problem: ## conclusions in this work, we develop a scalable scaling law for multilingual language models, tackling the complexities of training with multiple languages. through large-scale experiments involving over 100 models across 23 languages from 5 language families, we demonstrate that the test cross-entropy loss for each language family follows a predictable power-law relationship with model size, dataset size and sampling ratios. importantly, we validate a novel hypothesis that decouples the interaction between language families, enabling independent modeling of each family's performance. this insight allows us to derive the optimal sampling ratios that minimize the overall multilingual loss, providing a practical data mixing strategy. our results show that the optimal sampling ratios, computed from small models, generalize effectively to models several orders of magnitude larger, significantly reducing the need for resource-intensive data mixture selection. this offers an efficient method for multilingual training and opens up new avenues for future research in scaling multilingual lms.