M eta GPT : Merging Large Language Models Using Model Exclusive Task Arithmetic (2024)

Abstract: The advent of large language models (LLMs) like GPT-4 has catalyzed the exploration of multi-task learning (MTL), in which a single model demonstrates proficiency across diverse tasks. Task arithmetic has emerged as a cost-effective approach for MTL. It enables performance enhancement across multiple tasks by adding their corresponding task vectors to a pre-trained model. However, the current lack of a method that can simultaneously achieve optimal performance, computational efficiency, and data privacy limits their application to LLMs. In this paper, we propose M odel E xclusive T ask A rithmetic for merging GPT -scale models (MetaGPT) which formalizes the objective of model merging into a multi-task learning framework, aiming to minimize the average loss difference between the merged model and each individual task model. Since data privacy limits the use of multi-task training data, we leverage LLMs’ local linearity and task vectors’ orthogonality to separate the data term and scaling coefficients term and derive a model-exclusive task arithmetic method. Our proposed MetaGPT is data-agnostic and bypasses the heavy search process, making it cost-effective and easy to implement for LLMs. Extensive experiments demonstrate that MetaGPT leads to improvement of task arithmetic and achieves state-of-the-art performance on multiple tasks.

## introduction in recent years, a well-established paradigm for ai has been to pre-train models using large-scale datasets and then to fine-tune the models on different tasks through supervised learning with taskspecific datasets, which can lead to improved performance while requiring less labeled data @xcite @xcite ## related work model merging. currently, model merging has been developed for multiple uses such as improving performance on a single target task @xcite , improving outof-domain generalization @xcite @xcite , and improving the performance of multi-task learning @xcite @xcite , which is the core focus of our research. the range of applications has led to a proliferation of methods to improve beyond simple parameter averaging. fisher merging (matena and raffel, 2022) tries to weight the importance of individual models using fisher information matrix and uses it to merge different models. regmean @xcite formulate the merging problem as a regression problem and leads to an optimal solution for linear models. task arithmetic @xcite presents a method for merging models by adding task vectors to the pretrained model to improve multi-task performance. ties merging @xcite and dare @xcite propose to refine the task vectors by resolving the interference and removing extremely redundant components. @xcite propose that fine-tuning the models in their tangent space can amplify weight disentanglement and lead to substantial performance improvements. multi-task learning. multi-task learning is a powerful method for solving multiple correlated tasks simultaneously @xcite . current mtl works mainly focus on learning the shared representations from designing specific architecture @xcite or using specific optimization methods @xcite . the former focuses on learning the shared representation using different methods such as designing specific representation sharing module @xcite , learning to branch @xcite , and based selection criteria @xcite . and the latter focuses on balancing multiple tasks from the perspectives of task training weights @xcite , gradient dominance @xcite @xcite , and solving gradient conflicts @xcite @xcite . however, the conventional mtl approaches for collecting raw data across multiple tasks for joint training are not suitable for llms. the factors contributing to this issue are twofold: first, computational inefficiency due to the substantial computational costs associated with updating pre-trained models; second, a significant number of data proprietors are reluctant to disclose valuable or privacy-sensitive raw data. ## property verification in section 4, we introduced two properties essential to our proof. in this section, we conduct experiments to verify these properties. @xcite have proved that when the width of the neural network approaches infinity, it demonstrates kernel behavior and the optimization proceeds in the linear regime. we test llama-2-7b-chat-hf @xcite on agieval @xcite dataset to verify its linearity. we have randomly sampled three outputs of the llama-2-7b-chat-hf when 𝛼 in eq. 8 gets value of [0, 0.1, • • • , 1]. for better visualization, we also subtract all the outputs using max{𝑦 𝑖 }, ensuring they have the same endpoint. from the results in figure 3 , we can see that all the outputs are almost linear with 𝛼, which indicates that llms do exhibit a kernel behavior during finetuning. e n g li s h c h in e s e s p a n is h ja p a n e s e c o d e m a t h english chinese spanish japanese code math 1.0000 0.0008 0.0004 0.0428 0.0013 0.0084 0.0008 1.0000 0.0000 0.0001 0.0018 0.0003 0.0004 0.0000 1.0000 0.0106 0.0000 0.0003 0.0428 0.0001 0.0106 1.0000 0.0002 0.0008 0.0013 0.0018 0.0000 0.0002 1.0000 0.0011 0.0084 0.0003 0.0003 0.0008 0.0011 1.0000 0.00 0.02 0.04 0.06 0.08 0.10 ## experiments in this section, we conduct experiments to demonstrate the effectiveness of our metagpt. in the first section, we demonstrate that our metagpt consistently achieves optimal average performance across diverse datasets and is robust for model series with varying parameter sizes and architectures. dare and ties-merging are task vector-improving methods that resolve conflicts and redundant parameters between task vectors. we conduct experiments to demonstrate that our method is orthogonal to theirs and can be integrated to improve the average performance further. finally, we show that the model merged by our metagpt has better out-ofdistribution generalization ability. ## details of different methods we give a detailed comparison of the current merging method below from the perspective of extra data information, time complexity, and optimal per- ## conclusion in this paper, we have provided a novel model merging method named metagpt, an optimal task arithmetic algorithm under the setting of efficient and protecting data privacy, which is specifically designed for llms. we provide the mathematical formulation of task arithmetic's optimization objective and the theoretical analysis of the task arithmetic performance bound. by separating the data and scaling coefficient term under careful approximation, the closed-form solution provides an avenue for optimally achieving task arithmetic under the setting of applicable to llms and without using any data. extensive experiment results show that our metagpt outperforms the existing state-ofthe-art model-exclusive merging method and can be integrated with task vector-improving methods such as ties-merging and dare to achieve better performance.