Investigating and Scaling up Code-Switching for Multilingual Language Model Pre-Training (2025)

Abstract: Large language models (LLMs) exhibit remarkable multilingual capabilities despite the extreme language imbalance in the pre-training data. In this paper, we closely examine the reasons behind this phenomenon, focusing on the pre-training corpus. We find that the existence of code-switching, alternating between different languages within a context, is key to multilingual capabilities. We conduct an analysis to investigate code-switching in the pre-training corpus, examining its presence and categorizing it into four types within two quadrants. We then assess its impact on multilingual performance. These types of code-switching data are unbalanced in proportions and demonstrate different effects on facilitating language transfer. To better explore the power of code-switching for language alignment during pre-training, we investigate the strategy of synthetic code-switching. We continuously scale up the synthetic code-switching data and observe remarkable improvements in both benchmarks and representation space. Extensive experiments indicate that incorporating synthetic code-switching data enables better language alignment and generalizes well to high, medium, and low-resource languages with pre-training corpora of varying qualities.

## introduction large language models (llms) such as chat-gpt (openai, 2023), gpt-4 @xcite , llama2 @xcite , llama3 @xcite , and qwen2.5 @xcite have demonstrated remarkable performance across various tasks, including multiple-choice questionanswering @xcite , summarization @xcite , and reasoning @xcite . meanwhile, llms also demonstrate excellent multilingual capabilities. among them, some models are pre-trained on corpora not specifically designed for multilingual use @xcite , while others are pre-trained on corpora containing only a small fraction of multilingual data @xcite . despite the extreme language imbalance in the pre-training corpus @xcite , llms demonstrate impressive cross-lingual transfer to some extend @xcite . this raises the question: where do these cross-lingual transfers come from? code-switching, also known as code-mixing or language alternation, is the process of alternating between two or more languages in a single conversation @xcite . this type of data puts concepts from different languages within the same context, creating favorable conditoken-repl. you can use the above picture and add some related words, such as 剃须刀、字典、镜子、毛巾、冰箱、微波 炉、电脑 and 书橱. classify these words and fill in the table. [these chinese words mean "razor", "dictionary", "mirror", "towel", "refrigerator", "microwave", "computer", and "bookcase", respectively.] yoo et al. (2024a); li et al. (2024b) reveal the effects of synthetic codeswitching data in cross-lingual transfers. briakou et al. (2023) investigate the incidental bilingualism in the unreasonable translation capabilities of llms. however, there is a lack of detailed analysis of code-switching in multilingual pre-training. to investigate the effects of code-switching, we pre-train a 1.5b model on 60b tokens with extreme language imbalance (100:1). taking english and chinese as the high and low-resource language examples, we initially explore the natural code-switching phenomenon of two high-quality pre-training corpora. we conduct a model-based method to analyze and categorize four common code-switching types. subsequently, we conduct experiments to assess the impact of various codeswitching on cross-lingual transfer. building on this analysis, we propose to enhance the advantages of code-switching by incorporating synthetic code-switching data during pretraining, valued for its controllability and flexibility. through a series of scaling experiments, synthetic code-switching (syncs) significantly improves cross-lingual transfer, outperforming the addition of 20 times the amount of monolingual data with natural code-switching. further analysis shows that models trained on syncs data obtain improved multilingual alignment in the representation space. finally, we expand our approach to multilingual settings, encompassing high, medium, and low-resource languages, showcasing the generalization of syncs across languages. in summary, our findings are: in fineweb-edu @xcite , 0.4% of documents contain english-chinese codeswitching, compared to 51.6% in chinese-fineweb-edu-v2 @xcite . these instances, categorized into four types, enhance multilingual transfer despite their imbalance. ## analyzing the impact of code-switching based on our analysis of natural code-switching, we investigate its impact on cross-lingual transfer. ## scaling up code-switching despite the effectiveness evidenced in the experiment of previous section, the natural codeswitching phenomenon is rare and usually restricted to specific domains. in this section, we explore improving multilingual pre-training by synthesizing large-scale documents with codeswitching. this method is more flexible and controllable, allowing us to inject code-switching into any document at any density and in any format. ## conclusion this study explores the impact of code-switching on cross-lingual transfer during pre-training. we find that natural code-switching significantly enhances the multilingual capabilities of llms un-der extreme language imbalance. to address the scarcity of natural code-switching, we introduce a synthetic framework requiring only a small set of high-quality parallel sentences. through extensive experiments and analysis, we demonstrate that this framework outperform those trained on equivalent monolingual data, improving performance across languages of varying resources.