Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging (2024)

Abstract: Adapting general-purpose language models to new skills is currently an expensive process that must be repeated as new instruction datasets targeting new skills are created, or can cause the models to forget older skills. In this work, we investigate the effectiveness of adding new skills to preexisting models by training on the new skills in isolation and later merging with the general model (e.g. using task vectors). In experiments focusing on scientific literature understanding, safety, and coding, we find that the parallel-train-then-merge procedure, which is significantly cheaper than retraining the models on updated data mixtures, is often comparably effective. Our experiments also show that parallel training is especially well-suited for enabling safety features in LMs relative to continued finetuning and retraining, as it dramatically improves model compliance with safe prompts while preserving its ability to refuse dangerous or harmful prompts.

## introduction recent work has shown that instruction tuning pretrained language models (lms) can result in strong generalist models that can perform a variety of comprehension and generation tasks, owing largely to the availability of high quality datasets. as training datasets targeting new skills are constructed, it is an open question how best to patch preexisting models to incorporate the new skills represented by those datasets. commonly used approaches include continued finetuning (cft) of the existing models on the new datasets and retraining (rt) the models on a combination of old and new instruction tuning datasets, both of which have clear pros and cons associated with them. cft, while computationally cheaper, may cause the model to forget the skills from earlier rounds of training. on the other hand, in addition to being more expensive, rt on merged datasets is possible only when the practitioner has access to the the training datasets from earlier rounds, which is not the case for many publicly available instructiontuned lms such as mistral 7b @xcite and llama 3 (ai@meta, 2024). while we focus on publicly available mixes, this is an important consideration as new models and datasets are released. as an alternative, we explore merging the parameters of models separately trained for individual skills-merging to learn-a family of approaches we refer to as "parallel train then merge" (ptm). this general idea is shared by several wellknown methods (e.g., model patching, @xcite @xcite @xcite . unlike rt, ptm does not require access to the original training data, as one can separately train only on the new data, also making ptm a computationally cheaper option. since ptm does not directly change the weights associated with previously learned tasks, it should allow the model to retain more of its original skills compared to other methods. ptm also enables the efficient addition of multiple new skills to a single model. this work is the first to systematically explore ptm for instruction tuning. we compare the three methods for model patching (cft, rt, and ptm) in terms of model performance as well as computational cost. we experiment with adding three new skills: scientific literature understanding, coding, and refusing unsafe requests, to tülu, a general-purpose instructiontuned model, and evaluate the performance of the models resulting from each method on a suite of evaluation datasets representing general instructionfollowing skills as well as those that target the new skills being added. we find that: overall, we find that ptm is consistently a better choice than cft for teaching instruction-tuned models new skills without compromising on general skills. we note that rt is not always a feasible option because the training datasets for many publicly available instruction-tuned models are not available, and even when it is, ptm generally offers comparable performance tradeoffs while being significantly cheaper. ## problem setup we aim to add new, diverse behaviors to a generalpurpose instruction-tuned model, while preserving the original model's overall performance. we want to do so in a computationally efficient manner, without increasing inference cost. defined concretely, we want to use a new skill-specific dataset d to improve the performance of a general model θ g , trained on general data g, on an evaluation set for new skills e d , without losing performance on a general set e g , while minimizing the computational requirements. we describe the methodology and training complexity of three methods for adding new skills to preexisting models: continued finetuning, retraining from scratch, and model merging. we measure training complexity in terms of how many training steps are required to create the pool of models we select from. ## conclusions we explore the effectiveness of ptm for adding new skills to instruction-tuned models. we find that ptm is an efficient and effective method of augmenting preexisting models, enabling the addition of new skills with a fraction of the compute required compared to other common methods. in addition, we find that ptm achieves much better trade-offs between model safety and capability over other common methods tested. finally, we report heuristics for selecting merging coefficients when held out data is not available, and find that these strategies together enable the addition of multiple skills into a single model with no additional training required.