Beyond Data Quantity: Key Factors Driving Performance in Multilingual Language Models (2025)

Abstract: Multilingual language models (MLLMs) are crucial for handling text across various languages, yet they often show performance disparities due to differences in resource availability and linguistic characteristics. While the impact of pre-train data percentage and model size on performance is well-known, our study reveals additional critical factors that significantly influence MLLM effectiveness. Analyzing a wide range of features, including geographical, linguistic, and resource-related aspects, we focus on the SIB-200 dataset for classification and the Flores-200 dataset for machine translation, using regression models and SHAP values across 204 languages. Our findings identify token similarity and country similarity as pivotal factors, alongside pre-train data and model size, in enhancing model performance. Token similarity facilitates cross-lingual transfer, while country similarity highlights the importance of shared cultural and linguistic contexts. These insights offer valuable guidance for developing more equitable and effective multilingual language models, particularly for underrepresented languages.

## introduction multilingual language models have garnered significant attention due to their ability to handle and generate text across various languages, playing a crucial role in tasks such as machine translation, cross-lingual information retrieval, and multilingual content creation. however, achieving fair and effective performance across languages with diverse linguistic characteristics and varying resource availability remains a formidable challenge. prior research has identified several features that influence the performance of multilingual language models @xcite @xcite . although many factors are widely acknowledged to impact model performance, potentially even in a manner similar to the butterfly effect, these studies have often focused on a limited set of features. in contrast, our work aims to conduct a comprehensive analysis to systematically explore and quantify the effects of a broader range of features. specifically, we examine 12 distinct features related to both the models and the languages they are designed to process. in this study, we analyze the performance of multilingual language models (bloom, xglm and bloomz in different sizes) in 204 languages, using both classification (sib-200 dataset @xcite ) and generation @xcite ) tasks. we evaluate these models in zero-shot and two-shot learning settings, considering 14 different model configurations and sizes. our experiments involve over 2.3 million instances, providing a robust basis for our analysis. 1 figure 1 shows the overview of the analysis. the primary contributions of this paper are as follows: ## related work the development and evaluation of multilingual language models have been widely studied, with models like mbert, xlm-r, bloom, xglm, and llama 3.1 demonstrating their capability to handle multiple languages with varying resource levels effectively @xcite @xcite @xcite . despite these advancements, achieving fair performance across diverse languages remains challenging. recent efforts, such as the glot500 project and the bigtranslate project, have focused on expanding multilingual corpora and enhancing translation capabilities, emphasizing the need for inclusive benchmarks and tailored training approaches @xcite . additionally, studies have explored key factors driving multilingual model performance, highlight-ing the importance of language-specific features and data distribution @xcite . tokenization is a critical aspect of multilingual modeling, where the choice of tokenizer and vocabulary allocation significantly impacts cross-lingual transfer and task performance @xcite @xcite . successful cross-lingual transfer is influenced by shared vocabulary, linguistic similarity, and training data availability, as discussed in a comprehensive review by @xcite . despite advancements in understanding multilingual language models, most studies focus on a narrow set of features or tasks. our work fills this gap by analyzing 12 features across 204 languages, covering both classification and generation tasks in different learning settings. ## methodology in this section, we detail the datasets used, the models evaluated, the features extracted, and the evaluation methods employed in our study. ## discussion the results of this study provide valuable insights into the factors that drive the performance of multilingual language models across classification and generation tasks. ## conclusion this study offers a detailed analysis of the factors influencing multilingual language model performance across classification and generation tasks. our findings show that performance is shaped by complex, non-linear interactions among features. key factors include pre-train data percentage and model size, which significantly affect effectiveness. token similarity enhances cross-lingual transfer learning, while country similarity highlights the role of shared cultural and linguistic contexts. resource-related features like population and digital support showed limited direct impact but remain useful for understanding data availability and training strategies. these insights are crucial for developing more equitable multilingual models, especially for underrepresented languages.