I nstruction CP : A Simple yet Effective Approach for Transferring Large Language Models to Target Languages (2025)

Abstract: The rapid development of large language models (LLMs) in recent years has largely focused on English, resulting in models that respond exclusively in English. To adapt these models to other languages, continual pre-training (CP) is often employed, followed by supervised fine-tuning (SFT) to maintain conversational abilities. However, CP and SFT can reduce a model’s ability to filter harmful content. We propose Instruction Continual Pre-training (InsCP), which integrates instruction tags—also known as chat templates—into the CP process to prevent loss of conversational proficiency while acquiring new languages. Our experiments demonstrate that InsCP retains conversational and Reinforcement Learning from Human Feedback (RLHF) abilities. Empirical evaluations on language alignment, reliability, and knowledge benchmarks confirm the efficacy of InsCP. Notably, this approach requires only 0.1 billion tokens of high-quality instruction-following data, thereby reducing resource consumption.

## introduction large language models (llms) have demonstrated remarkable performance across numerous natural language processing (nlp) tasks @xcite . however, the majority of llms are pretrained on english corpora(ai@meta, 2024; @xcite , thus restricting their utility to english language contexts. while some endeavors opt to train their llms from scratch using non-english data, as exemplified by @xcite , we recognize the significant time and computing resources required for such an approach. drawing inspiration from @xcite , many research groups have shifted their focus towards continual pre-training (cp) @xcite on target languages to enhance knowledge acquisition and model fluency. subsequently, supervised fine-tuning (sft) is con-ducted on instruction-formatted data to ensure that models possess the capability to respond to questions in a format consistent with english-based pretrained llms, such as bloom @xcite , llama2 @xcite , and mistral-7b @xcite . yet, as highlighted in @xcite , challenges persist in maintaining rlhf capabilities when fine-tuning gpt-3.5 turbo(openai, 2023) on non-english data. our experiments validate similar observations with other llms like llama2. this work proposes a novel fine-tuning approach called instruction continual pre-training (inscp) for llms to adapt to non-english languages. we hypothesize that providing a chat template during cp prevents the model from forgetting its conversational abilities, as it mirrors its original training conditions. inscp is essentially the same as typical cp, except that we augment each piece of data with a chat template containing special instruction tokens, such @xmath0, 2024). this simple augmentation enables the model to effectively retain its original rlhf capabilities, such as defending against offensive input while learning a new language through cp. we evaluate the effectiveness of inscp on llms, primarily focusing on the llama3-instruct model, across three key aspects: language alignment, reliability, and knowledge benchmarks. the results demonstrate that the model, after undergoing inscp on llama3-instruct, effectively performs in traditional chinese when prompted with traditional chinese input, surpassing the performance of llama3-instruct. moreover, the model after inscp does not suffer a serious performance dropped in knowledge, safety and rlhf ability. 1 2 related work ## methodology for our method, instruction continual pre-training, we adopt a similar approach to cp, but with the addition of the model's original chat template. the template is shown in appendix a.1 the inputs in the template represent the prompts provided by the user. in our context, where the objective is to train llms in the target language through next token prediction tasks while retaining their chat ability, we place the cp data in the model_response. this arrangement ensures that llms generate tokens based on the target language. the inscp template is shown in a.1. ## limitations of inscp as discussed in section 4.1, the choice of data used in inscp significantly influences its outcomes. our experiments indicate that conducting inscp necessitates the utilization of low-perplexity instructionfollowing data, which can be challenging to acquire in abundance for certain languages. consequently, we opted to perform inscp using small datasets, which we believe is a more generalizable approach for languages with limited resources. nonetheless, both data size and data quality remain challenges when implementing inscp. ## conclusion in this work, we introduce a novel pipeline called inscp designed to facilitate the transfer of llms into non-english domains. through inscp, llms can retain their inherent abilities while also acquiring the capability for language alignment in the target language and gaining knowledge of the target domain. additionally, we demonstrate that inscp does not necessitate extensive data, thereby consuming fewer resources and less time. remarkably, even with a small amount of data, inscp can transform english-based llms into models aligned with the target language, a stark contrast to the resource-intensive traditional pipeline. in-scp paves the way for future llms, primarily finetuned in specific languages, to swiftly transfer their abilities to other languages.