Knowledge Fusion By Evolving Weights of Language Models (2024)

Abstract: Fine-tuning pre-trained language models, particularly large language models, demands extensive computing resources and can result in varying performance outcomes across different domains and datasets. This paper examines the approach of integrating multiple models from diverse training scenarios into a unified model. This unified model excels across various data domains and exhibits the ability to generalize well on out-of-domain data. We propose a knowledge fusion method named Evolver, inspired by evolutionary algorithms, which does not need further training or additional training data. Specifically, our method involves aggregating the weights of different language models into a population and subsequently generating offspring models through mutation and crossover operations. These offspring models are then evaluated against their parents, allowing for the preservation of those models that show enhanced performance on development datasets. Importantly, our model evolving strategy can be seamlessly integrated with existing model merging frameworks, offering a versatile tool for model enhancement. Experimental results on mainstream language models (i.e., encoder-only, decoder-only, encoder-decoder) reveal that Evolver outperforms previous state-of-the-art models by large margins.

## introduction due to the high training costs of large language models, it is common practice to fine-tune already pre-trained language models to adapt them for specific applications. this fine-tuning approach often allows us to achieve excellent performance in specific data domains or tasks at a relatively lower cost @xcite . however, the challenge lies in the fact that fine-tuning @xcite * corresponding authors. the same model in different task scenarios may result in performance variations, meaning that the results may not be satisfactory when testing the same model in different contexts. therefore, our objective is to integrate knowledge from models trained in different scenarios to enhance the model's performance in cross-domain or cross-task scenarios @xcite , without the need for further training or extra training data. mainstream knowledge fusion methods can be divided into two main categories. the first involves extensive training on large datasets across multiple tasks to learn new model parameters with shared representations, such as in multi-task learning. the second relies on fusing existing models from specific scenarios without requiring extensive data. while multi-task learning generally improves overall performance, it has significant drawbacks: the need for abundant annotated data for all tasks and the complexity and time consumption of the training phase, especially with dataset combinations @xcite . in contrast, model merging methods do not require retraining models and do not raise concerns about data privacy. in this paper, we primarily delve into the second category of methods and introduce an innovative model evolution approach inspired by darwinian evolution theory @xcite . in short, we compare our model evolution approach with other prevalent knowledge fusion methods, detailing their distinct features in in fact, the problem of model merging can be reformulated as an optimization problem that aims to find the most valuable information for knowledge fusion in order to achieve better outcomes than any individual model alone. for instance, @xcite employed a simple linear regression approach for optimization, while model soups @xcite implemented a greedy search method. in this paper, we consider the adoption of a more robust evolutionary algorithm for optimiza- as shown in figure 1 , our approach first processes models fine-tuned in different environments as an initial population. we then generate a new population through mutations and recombination among different individuals within the population. subsequently, we validate the performance of the new population on development environment datasets and preserve elite individuals for updating. after evolving with enough generations, we select individuals with the best performance as the evolved model. our evolutionary algorithm is firmly rooted in the task vector @xcite methodology, wherein we derive the differential vectors between two individuals during the mutation operation. we delve deeper into the task vector concept from two distinct perspectives. firstly, we employ a scaling factor (f ), to search and regulate the weights of the difference vectors. this is crucial for aligning the expectations of model outputs, as illustrated in @xcite . secondly, we utilize the crossover ratio (cr), to maintain a certain level of sparsity, thereby preventing interference among different individuals, as demonstrated in @xcite . additionally, by conducting random searches for crossover parameters, we are able to obtain more optimal per-parameter coefficients for model merging. we conduct knowledge fusion experiments across various difficulty levels, employing different types of models, such as roberta, deberta, t5 and gpt2. these experiments cover sentiment classification tasks in diverse data domains, as well as benchmark tasks from the glue dataset. the experimental results consistently demonstrate that our proposed method effectively enhances performance across all experimental settings. furthermore, our approach can be synergistically combined with previous model merging methods (e.g., fisher (matena and raffel, 2022), regmean @xcite ), resulting in further improvements in knowledge fusion performance. this combined approach significantly outperforms baseline methods and previous techniques. notably, our method also exhibits superior generalization performance when applied to data domains beyond the scope of multiple datasets. to summarize, our key contributions include: our method consistently enhances knowledge fusion performance across a broad spectrum of data domains and tasks. ## evolving weights of language models the goal of model evolution is to combine multiple fine-tuned language models into a more powerful single model. we achieve this by simulating the evolution process of a neural network population, as shown in figure 2 . we use the same pre-trained checkpoint and fine-tune it in different environments to create the initial population. as all individuals share the same model architecture, this enables our evolution algorithm to perform mutations and recombinations among individuals within the parameter space. figure 2 : the process of evolving weights of language models. evolver involves aggregating the weights of language models into a population and generating offspring models through mutation and crossover operations. ## experimental results we assess the performance dynamics of the model evolution method across a range of scenarios with varying levels of complexity. these scenarios include: (1) performance across different data domains used for fine-tuning individual models. (2) performance across different tasks, when individual models are specialized in only one task. (3) ood generalization performance on datasets from previously unseen domains. ## conclusions we introduce a novel knowledge fusion method, called model evolution, inspired by evolutionary algorithms. this approach significantly boosts the performance of model merging in diverse nlp contexts. model evolution stands out by aggregating model weights into a population and updating it with superior offspring models, all without requiring extra training data. the most significant contribution of our work lies in our experimental discovery that knowledge fusion can benefit from searching per-model and per-parameter coefficients. this type of search-based, gradient-free optimization algorithm proves to be an effective tool for model merging and warrants greater attention in the research community. our extensive experiments validate its superiority over previous techniques.