Code-Switching Curriculum Learning for Multilingual Transfer in LLM s (2025)

Abstract: Large language models (LLMs) now exhibit near human-level performance in various tasks, but their performance drops drastically after a handful of high-resource languages due to the imbalance in pre-training data. Inspired by the human process of second language acquisition, particularly code-switching—the practice of language alternation in a conversation—we propose code-switching curriculum learning (CSCL) to enhance cross-lingual transfer for LLMs. CSCL mimics the stages of human language learning by progressively training models with a curriculum consisting of 1) token-level code-switching, 2) sentence-level code-switching, and 3) monolingual corpora. Using Qwen 2 as our underlying model, we demonstrate the efficacy of the CSCL in improving language transfer to Korean, achieving significant performance gains compared to monolingual continual pre-training methods. Ablation studies reveal that both token- and sentence-level code-switching significantly enhance cross-lingual transfer and that curriculum learning amplifies these effects. We also extend our findings into various languages, including Japanese (high-resource) and Indonesian (low-resource), and using two additional models (Gemma 2 and Phi 3.5). We further show that CSCL mitigates spurious correlations between language resources and safety alignment, presenting a robust, efficient framework for more equitable language transfer in LLMs. We observe that CSCL is effective for low-resource settings where high-quality, monolingual corpora for language transfer are hardly available.

## introduction as recent advances in natural language processing (nlp) have benefited from their remarkable scale, large language models (llms), such as chatgpt @xcite and llama @xcite , have emerged with strong capabilities in knowledge @xcite , generation @xcite , and reasoning @xcite , on par or even surpassing human levels. such llms are inherently multilingual, as web-crawled, extensively large training data includes diverse languages. however, these models perform poorly in non-english, especially low-resource languages @xcite . this discrepancy arises from the imbalanced distribution of language resources in pre-training data, as collecting extensive data in all languages is practically impossible @xcite . to address this challenge, researchers have explored cross-lingual transfer techniques to improve llm performance in non-english languages @xcite . inspired by the second language acquisition in humans, we look at code-switching for crosslingual transfer in llms. code-switching, an alternating use of two or more codes within one conversational episode, is a common practice in language learning @xcite . at first, second language learners at the basic level often rely on codeswitching to express their intentions while minimizing misunderstanding @xcite . as they become more proficient, they begin to produce complete sentences, eventually exhibiting full fluency in the target language. in other words, both frequency and degree of code-switching in language learning are closely linked with learners' proficiency level @xcite . following this learning process, we introduce a new strategy: code-switching curriculum learning (cscl), which adapts the pedagogical process of human language acquisition to the context of language transfer of llms (figure 1 ). our approach involves further training english-centric llms using three stages of data: 1) token-level code-switching corpora, 2) sentence-level codeswitching corpora, and 3) monolingual corpora. cscl organizes training data into three difficulty levels according to code-switching degree, presented in order from least to most difficult, thus mirroring second language learning by humans. this sequence of curriculum sets mimics the natural progression of human language acquisition using code-switching as a scaffold. here, the codeswitching data can be easily synthesized through llms (i.e., gpt-4o). code-switching, which explicitly reveals cross-lingual alignments between tokens in two different languages, facilitates llms' adaptation to the target language. we employ qwen 2 (7b) @xcite , an open llm mainly trained in both english and chinese, to examine cscl for language transfer in korean. we observe that cscl outperforms conventional training approaches using monolingual corpora on multiple-choice question-answering tasks and machine translation tasks in korean. notably, cscl reduces the typical performance degradation in english caused by catastrophic forgetting during cross-lingual transfer. our ablation study further highlights the benefits of both token-and sentence-level code-switching in enhancing llm transfer, while the structured progression of curriculum learning amplifies these effects. here, the generation outputs from cscl-trained models do not result in unintended code-switching; instead, we demonstrate improved general generation ability of the cscl in the target language, evaluated through text summarization and machine translation. furthermore, we extend our analyses to other non-english languages (i.e., japanese as a high-resource language and indonesian as a lowresource language) and different foundation models (i.e., gemma 2 @xcite and phi 3.5 @xcite ). we also report that llms trained with the cscl are more robust to non-english, codeswitching adversarial inputs, reducing the spurious correlation between language resources and safety alignment by enhancing cross-lingual alignment. we empirically present that the cscl is effective for low-resource settings where the high-quality, monolingual corpora for language transfer are scarce. our main contributions are as follows: ## code-switching curriculum learning in this section, we describe cscl, a curriculum learning strategy designed for language transfer of english-centric llms using code-switching corpora. this approach is inspired by the pedagogical process of second language acquisition, starting from partial, word-level code-switching and grad-ually achieving complete, fluent use of the target language (ghaderi et al., 2024; sinclair and fernández, 2023). ## conclusion in this paper, we introduce code-switching curriculum learning (cscl), inspired by the pedagogical process of second language acquisition of human, where code-switching is employed according to their proficiency levels. we regard the degree of code-switching in language learning as a measure of difficulty and apply curriculum learning for language transfer, starting from training with tokenlevel code-switching corpora, sentence-level codeswitching corpora, and finally monolingual corpora in both target language and english. we demonstrate that cscl outperforms the traditional pretraining method with monolingual target corpora in terms of performance boost in target language and reduced performance loss in english typically caused by catastrophic forgetting during language transfer. we further extend our observations across various languages and foundation models. notably, cscl does not induce unintended code-switching in the generated outputs; instead, it significantly enhances the generation ability in the target language, comprehensively evaluated through summarization and instruction-following tasks. furthermore, we explore that improving the cross-lingual alignment through cscl can mitigate the spurious correlation between language resources and safety alignment, reducing the vulnerabilities in multilingual red-teaming scenarios. through ablation studies scaling up the training data, we highlight that cscl can be efficiently used in low-resource languages where high-quality, large-scale monolingual corpora are hardly available.