Unlocking the Potential of Model Merging for Low-Resource Languages (2024)

Abstract: Adapting large language models (LLMs) to new languages typically involves continual pre-training (CT) followed by supervised fine-tuning (SFT). However, this CT-then-SFT approach struggles with limited data in the context of low-resource languages, failing to balance language modeling and task-solving capabilities. We thus propose a new model merging solution as an alternative for low-resource languages, combining models with distinct capabilities into a single model without additional training. We use model merging to develop task-solving LLMs for low-resource languages without SFT data in the target languages. Our experiments based on Llama-2-7B demonstrate that model merging effectively endows LLMs for low-resource languages with task-solving abilities, outperforming CT-then-SFT in scenarios with extremely scarce data. Observing performance saturation in model merging with increasingly more training tokens, we further analyze the merging process and introduce a slack variable to the model merging algorithm to mitigate the loss of important parameters, thereby enhancing model performance. We hope that model merging can benefit more human languages suffering from data scarcity with its higher data efficiency.

## introduction large language models (llms) demonstrate remarkable capabilities across various nlp tasks, owing to the vast amounts of high-quality training data @xcite . however, developing models with task-solving abilities for low-resource languages remains challenging due to limited data availability. a common practice for constructing task-solving llms for a low-resource language involves continual pertaining (ct) and supervised fine-tuning (sft) for the target language @xcite , known as ct-then-sft. the scarcity of ct data impedes llms' ability to learn effective language modeling for these target languages. additionally, it is difficult to acquire sufficient sft data in low-resource languages to enhance downstream task performance. to address this issue, previous works attempt to transfer capabilities from high-resource languages to low-resource languages by training on english sft data @xcite . however, this approach can lead to catastrophic forgetting @xcite of language modeling for the target languages @xcite , resulting in llms still failing to solve tasks due to the loss of language abilities. to better integrate the language modeling and task-solving capabilities, we introduce model merging for low-resource languages, which can combine multiple models with distinct abilities into a single model without additional training. previous work @xcite has shown that an llm for high-resource languages can be merged with task-specific models, such as japanese language models and math models. in this work, we explore whether model merging can effectively construct task-solving llms for low-resource languages. specifically, we investigate the following research questions: rq1: what is the viability of constructing task-solving llms in low-resource languages through model merging? rq2: is model merging always a better choice than ct-sft? rq3: what factors may affect llms in obtaining tasksolving capabilities through model merging? to answer these questions, we study the adaptation of llama-2-7b @xcite , an english-centric llm, into seven distinct lowresource languages. we first continually pre-train llama-2-7b on monolingual texts in each language. next, we explore two approaches to inject tasksolving capabilities into this continually pre-trained model: (1) training the llm with english sft data or the data translated to the target low-resource language; (2) merging the model with an english tasksolving llm. experiments show that model merging can effectively equip the ct models with tasksolving capabilities. notably, when pre-training corpora in the target language are extremely scarce (<10b tokens), model merging outperforms ctthen-sft. as an llm is continually pre-trained with more tokens in target languages, the improvements brought by model merging gradually saturate. then, model merging can no longer significantly surpass the sft method. to further investigate the factors impeding the continuous improvement of model merging, we conduct a detailed analysis of the process of merging two llms. we find that when an english sft model is merged with an llm continually pretrained with more tokens in the target language, more parameters from the sft model are discarded during merging. the loss of these parameters may lead to a decline in task-solving capabilities, preventing the merged model from improving performance on downstream tasks. to mitigate the loss of important parameters from the sft model, we propose a novel model merging solution with slack variables. this strategy allows for more flexible control over the merging process to retain important parameters. our contributions are as follows: (1) we are the first to introduce model merging to construct tasksolving llms for low-resource languages; (2) we reveal that model merging is more effective than sft in the scenarios of extremely low-resource languages; (3) through a quantitative study of the merging process, we explain the performance plateau of model merging with a larger ct corpus and propose a simple yet effective enhancement to popular model merging algorithms. ## related works model merging model merging is a promising way to combine the abilities of multiple models. pioneering works explore strategies to find the best weights for averaging @xcite @xcite . task arithmetic @xcite employs task vectors, enabling control through arithmetic operations to steer the merged model's behavior. ties @xcite further addresses the problem of information loss by handling parameter conflict more carefully. dare @xcite zeros out redundant parameters and amplifies the remaining ones. evolutionary model merge @xcite automatically discovers optimal model combinations through evolutionary algorithms. there is little discussion of model merging in the context of multilinguality. instead, previous works attempt to introduce language-specific and task-specific modular adapters @xcite @xcite , which require additional training. these works focus on high-resource languages and specific tasks. in contrast, model merging can utilize existing models without additional training, making it a versatile approach for building more general task-solving llms. besides, we are the first to study model merging for low-resource languages. ## model merging for low-resource languages the conventional ct-then-sft paradigm struggles to balance language modeling and task-solving abilities in the context of low-resource languages. we propose to adopt model merging as an alternative, which can construct task-solving llms for lowresource languages without requiring sft data in the target languages. ## experimental setup languages we use 7 low-resource languages from five distinct language families for experiments: tamil, telugu, odia, bengali, tibetan, uyghur, and mongolian (in the traditional mongolian script). see their basic information in we select these languages because they are underrepresented in currently popular llms despite their large population (over 475m) worldwide. as shown in pre-training corpus during continual pretraining, we use the largest available corpus for each language from culturax @xcite , indiccorp-v2 @xcite , and mc 2 @xcite . to maximize language coverage with constraint computational resources, we sample 8b tokens for continual pre-training of tamil and telugu, and 16b tokens for bengali. the corpus sizes are shown in following llama models @xcite , we employ redpajama (computer, 2023) with the same sampling proportion for memory replay to reduce forgetting of english language modeling. sft data we mainly use flan @xcite for sft, which consists 155k training instances for 1,411 distinct tasks. since there are limited math reasoning tasks in flan, we additionally incorporate 7,473 instances from gsm8k @xcite into the supervised training sets. we translate flan into above mentioned languages using nllb-200-distilled-1.3b @xcite 4 . note that this model does not support traditional mongolian and there are no open-source mt models available for this language currently. we thus adopt a roundabout way: translating the instructions into cyrillic mongolian, which nllb supports, and converting them into traditional mongolian with an open-source transliteration tool foot_4 . evaluation tasks regarding the four languages in india (tam, tel, ory, and ben), we use sib-200 @xcite ) for text classification, belebele @xcite for machine reading comprehension, and mgsm @xcite for math reasoning (only available in telugu and bengali). regarding the three languages in china (bod, uig, and mvf), we use mlic-eval @xcite , including the following 4 tasks: text classification (tc), machine reading comprehension (mrc), response selection (rs), and math reasoning. see statistics in appendix a. implementation details we use megatron-lm for model training @xcite ) and arcee's mergekit for model merging @xcite . see more details in appendix b. ## results and analysis in this work, we mainly study two categories of common roadmaps, ct-then-sft and model merging, to transfer the task-solving ability from a highresource language to a low-resource one. we aim to investigate following research questions. rq1: what is the viability of constructing task-solving lang. corpus tokens tasks tam culturax 15.9b sib-200, belebele tel culturax 12.4b sib-200, belebele, mgsm ory culturax 765m sib-200, belebele ben indiccorp-v2 36.4b sib-200, belebele, mgsm bod mc 2 1.00b mlic-eval uig mc 2 412m mlic-eval mvf mc 2 904m mlic-eval llms in low-resource languages via model merging? rq2: is model merging always a better choice than ct-then-sft? we take the settings of base, sft-flan, and ct-x as the baselines. for ct-then-sft, we investigate two common methods to build task-solving models, ctsft-x-flan and ctsft-x-mt. for model merging, we study two effective algorithms to combine the abilities of language modeling and task solving: weighted averaging and ties. ## understanding the dynamics of model merging as shown in figure 2 , we find that the performance of ties-x-flan on downstream tasks may no longer improve as we use more tokens for ct. in this section, we want to investigate rq3: what factors may affect llms in obtaining task-solving capabilities through model merging? ## discussions in this section, we further explore two important questions related to model merging. first, we investigate the potential of merging more than one low-resource language into a task-solving model, which could offer a new avenue for constructing multilingual models. second, we examine why commonly-used machine-translated sft data often fails in the context of constructing task-solving models in low-resource languages. this failure underscores the advantage of model merging, as it does not require sft data in the target language. ## case study we further investigate how model merging can improve the model performance on the tasks in lowresource languages. here, we focus on two types of errors in low-resource language models: unparsable outputs during generation, and misunderstanding of character overlap in mrc. in this section, we take traditional mongolian as an example, which the base model has hardly seen during pre-training. ## conclusion in this paper, we investigate the potential of using model merging to construct task-solving llms for low-resource languages. our findings demonstrate that model merging outperforms the conventional ct-then-sft paradigm, achieving higher data efficiency. we further analyze the mechanism behind the performance saturation of model merging with an increased number of ct tokens, which inspires a simple yet effective improvement to the model merging algorithm. we hope that model merging can reduce the costs associated with data collection and model training, benefiting a greater number of languages suffering from data scarcity.