Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusion (2024)

Abstract: Model fusion research aims to aggregate the knowledge of multiple individual models to enhance performance by combining their weights. In this work, we study the inverse problem: investigating whether model fusion can be used to reduce unwanted knowledge. We investigate the effects of model fusion in three scenarios: the learning of shortcuts, social biases, and memorization of training data in fine-tuned language models. Through experiments covering classification and generation tasks, our analysis highlights that shared knowledge among models is enhanced during model fusion, while unshared knowledge is usually forgotten. Based on this observation, we demonstrate the potential of model fusion as a debiasing tool and showcase its efficacy in addressing privacy concerns associated with language models.

## introduction nlp models can acquire a diverse range of skills during fine-tuning. while some of these skills are fundamental problem-solving abilities that are applicable in various scenarios, others are merely shortcuts or biases that may not generalize well. for instance, models trained on natural language inference (nli) tasks are known to adopt heuristics based on word-label associations @xcite . the practice of fusing weights of multiple models, such as through averaging (e.g., @xcite @xcite , has demonstrated improved performance and generalization. however, the mechanisms underlying these improvements have received limited attention. it is unclear if all underlying skills are enhanced and accumulated through weight averaging. in this study, we investigate the preservation of both knowledge shared across models and unique unshared knowledge during model fusion in classification and generation tasks. our hypothesis is that while shared knowledge is typically retained, unshared knowledge tends to be forgotten or degraded. figure 1 illustrates this concept, showing the corruption of unshared knowledge while preserving shared knowledge after model fusion, resulting in reduced biases. we claim this degradation is a useful property of model fusion, allowing novel uses for model fusion and possibly explaining current ones. to support our claims, we conduct a series of experiments that range from controlled, synthetic scenarios to real-world applications. first, we examine classification tasks with artificially augmented shortcuts and find a predominant trend: while shared skills, including shortcuts and general task skills are preserved during model fusion, unshared skills are mostly forgotten. as we increase the number of fused models, this forgetting mechanism intensifies ( §4). second, our analysis indicates potential for reducing social biases in language models through model fusion. we demonstrate that simple weight averaging can serve as a useful debiasing tool, reducing biases by up to 68% without deteriorating task performance ( §5). last, our findings suggest an exciting avenue for model fusion as a tool for mitigating memorization and preserving privacy. by comparing memorization before and after fusion, we demonstrate that fusion can reduce the leakage of personal information from training datasets into learned models ( §6). our contributions are: ## related work fusing multiple models into one @xcite @xcite has been shown to be beneficial in various scenarios and fields, for example in multitask learning @xcite pretraining @xcite , efficient finetuning @xcite , vision @xcite and reinforcement learning @xcite . these methods show improvements in both performance on the shared task @xcite and generalization to new ones @xcite . however, how fusing weights affects the learned skills in models is an open question. some theoretic works showed the weighted averages of models trained from scratch on the same data also perform well on the data @xcite . other proposed weights to align the space to make it so in harder cases @xcite . taken together, these suggest that model skills are intricately intertwined in the euclidean space of weights. this is strengthened by recent works, suggesting that models finetuned on the same dataset @xcite , or the same broader set of skills @xcite , tend to cluster together in compact regions of this space. building on these insights, we offer a novel angle by exploring model fusion under conditions of varied training data. we specifically investigate the conditions under which fusion is beneficial and when it may be less effective. this approach aims to uncover patterns of systematicity in the effectiveness of model fusion from the perspective of variance in training data. while forgetting and improvement of common skills may well be two distinct phenomena, forgetting as a step function rather than gradually may also account for the gains seen in previous work. if needed skills are learned by many models and are thus kept, overfitting and errors are not shared and hence mainly discarded, even with not additional skills the overall result should be improved performance. this even fits results such as @xcite ; @xcite , which claim that to get more from multiple models, signal should be amplified and interference reduced. this may be explained by the phase shift, with multiple models and without amplification, most skills would not have enough signal to be kept. ## method models trained for the same task can develop distinct approaches despite achieving similar losses @xcite . prior research indicates that interpolating between the weights of two models can maintain or enhance performance on test datasets similar to their training data @xcite . however, it remains uncertain how model fusion affects the specific knowledge each model utilizes, and under what circumstances fusing models fails to effectively combine their skills. to explore this issue, we delve into the effects of model fusion @xcite @xcite on knowledge utilization. although various methods for model fusion have been proposed (e.g., @xcite , our study employs a fundamental technique common to these approaches. we focus on the simple method of computing a weighted average of model parameters. given m models with parameters θ 1 , . . . , θ m , where each θ @xmath0, we define the fused model, θ f used , using the following convex combination: this framing asserts that knowledge is inherently linked to model parameters, while knowledge utilization also relies on the choice and design of specific datasets. these datasets are specifically curated to probe particular attributes of knowledge. depending on the curated dataset, the knowledge being questioned could be the model's capability on a complex task or some simpler mechanism used by the model to solve that task (e.g., a shortcut). in this perspective, curating a dataset to probe knowledge is akin to using a microscope with different magnification levels. the knowledge utilization function, which might be a performance metric such as accuracy, f1 score, or bleu score, reflects the relevance of the knowledge to the dataset and task at hand. for example, if evaluating two models with parameters θ 1 and θ 2 using datasets d 1 and d 2 , designed to test distinct knowledge types, high scores on ψ d 1 ,t (θ 1 ) and ψ d 1 ,t (θ 2 ) would indicate that both models share the knowledge type assessed by d 1 . in contrast, a disparity in scores between ψ d 2 ,t (θ 1 ) and ψ d 2 ,t (θ 2 ) suggests that the knowledge type evaluated by d 2 is not shared between the models. in this context, we propose two hypotheses about the relation between model fusion and knowledge utilization: (1) shared knowledge across models is preserved during model fusion; (2) unshared and independent knowledge tends to be forgotten during model fusion. given m models with parameters θ @xmath1,t (θ i ), for any given dataset d and task t , we broadly posit: if models share the same knowledge, the knowledge utilizations among models are close, resulting in the knowledge utilization of the fused model being close to the others. however, if the knowledge is not shared among models, the gap between the minimum and maximum knowledge utilizations, within which the knowledge utilization of the fused model can reside, becomes larger. ## shortcuts models trained for any task can capture multiple types of knowledge simultaneously, making it challenging to interpret the effects of each knowledge separately. to incrementally build an understanding of the complex dynamics of knowledge acquisition, we begin with a set of experiments on a sentiment classification task where we inject synthetic shortcuts. using synthetic shortcuts allows us to (1) control the knowledge that a model acquires at any given time, (2) make models learn non-overlapping heuristics, and (3) easily evaluate a particular shortcut adopted by a model. ## social biases in this section, we extend our investigation beyond synthetically generated shortcuts to a real-world use case of text classification with social biases. our objective is to validate the claims made in the previous section and, additionally, to examine the potential of model fusion as a debiasing tool. ## memorization previously, we focused on validating our claims by addressing spurious correlations and biases in text classification tasks. next, we examine model fusion to alleviate data memorization in llms. by exploring the potential of model fusion to reduce memorization, we aim to address privacy concerns. ## conclusion and discussion we explore the impact of model fusion on shortcuts, biases, and memorization in nlp models. our findings support that model fusion preserves shared knowledge while losing unshared knowledge. we highlight the potential of model fusion in reducing biases, enhancing privacy, and other applications.