Here’s a Free Lunch: Sanitizing Backdoored Models with Model Merge (2024)

Abstract: The democratization of pre-trained language models through open-source initiatives has rapidly advanced innovation and expanded access to cutting-edge technologies. However, this openness also brings significant security risks, including backdoor attacks, where hidden malicious behaviors are triggered by specific inputs, compromising natural language processing (NLP) system integrity and reliability. This paper suggests that merging a backdoored model with other homogeneous models can significantly remediate backdoor vulnerabilities even if such models are not entirely secure. In our experiments, we verify our hypothesis on various models (BERT-Base, RoBERTa-Large, Llama2-7B, and Mistral-7B) and datasets (SST-2, OLID, AG News, and QNLI). Compared to multiple advanced defensive approaches, our method offers an effective and efficient inference-stage defense against backdoor attacks on classification and instruction-tuned tasks without additional resources or specific knowledge. Our approach consistently outperforms recent advanced baselines, leading to an average of about 75% reduction in the attack success rate. Since model merging has been an established approach for improving model performance, the extra advantage it provides regarding defense can be seen as a cost-free bonus.

## introduction recently, the machine learning community has increasingly leveraged online repositories such as huggingface, 1 tensorflow hub, 2 and pytorch hub foot_2 to access publicly available datasets and pretrained models (plms). 4 while the open nature of these platforms significantly enhances global collaboration and innovation in the field, this openness also exposes them to potential vulnerabilities, such as backdoor attacks, due to the lack of strict checks on the quality of contributions. backdoor attacks are designed to manipulate the predictive behavior of a targeted model using specific triggers. these triggers, when present, cause the model to produce predetermined outputs, effectively compromising its integrity. meanwhile, these backdoored models exhibit expected behavior in the absence of these triggers. attackers can disseminate backdoored models through public repositories like huggingface @xcite , or victims might inadvertently publish compromised models by misusing poisoned public datasets @xcite . this highlights the security risks nlp systems face due to reliance on untrustworthy resources. given the vulnerabilities to backdoor attacks, various defensive strategies have been suggested. many require extra resources such as the training source @xcite or attackspecific knowledge @xcite , rendering them impractical in real-world applications. for instance, suppose an already deployed model, such 1 15059 as one from huggingface, is found to be backdoored. in this scenario, the training source or procedure of the model is inaccessible, posing significant challenges in identifying the type of backdoor attack. consequently, conventional defensive strategies would prove ineffective in mitigating the attack under such circumstances. considering the success of model merging as a method to enhance model performance @xcite , our study presents it as an effective, resource-efficient strategy to mitigate backdoor attacks. this approach offers a no-cost solution to these real-world security challenges. specifically, our proposal suggests that model merging techniques can effectively mitigate backdoor attacks on plms, even without access to external knowledge such as training procedures or the nature of the backdoor attack. notably, our approach eliminates the need to retrain the models during the process. our experiments show that model merging solutions can significantly lower the attack success rate compared to advanced baselines. 5 we summarize our contributions as follows: backdoor attacks and defenses. backdoor attacks on deep learning models were first prominently demonstrated in image classification tasks by @xcite . more recently, research has pivoted towards backdooring nlp models, employing mainly two strategies. the first approach, data poisoning, involves training a model on a dataset in which a small subset is deliberately corrupted to introduce a backdoor @xcite @xcite . the alternative strategy, weight poisoning, bypasses the need for dataset manipulation by directly altering the trained weights of the model to insert triggers @xcite . researchers have devised defensive strategies to address the vulnerability of victim models to backdoor attacks. these strategies are implemented either during the training phase, known as trainingstage defenses, or during the testing phase, referred to as inference-stage defenses. training-stage defenses focus on detecting and eliminating poisoned samples from the training dataset, often viewed as outlier detection @xcite @xcite @xcite . this is based on the assumption that poisoned samples exhibit distinct characteristics compared to clean samples. on the other hand, inference-stage defenses employ either the targeted model or an auxiliary model to detect and neutralize malicious inputs by recognizing their abnormal behavior @xcite @xcite . our proposed method belongs to the inference-stage category. unlike many existing strategies @xcite , our approach is designed to be efficient and adaptable, not relying on specific model architectures or training information. ## method this section first outlines the general framework of backdoor attacks. then, we provide details of our defense method. backdoor attacks. when a backdoored model m p receives a clean textual input x, it predicts a label y. the prediction y may either be correct or incorrect, depending on m p 's performance. however, if attackers apply a poisoning function f (•) to alter x into x ′ , m p then outputs a malicious label y ′ . a backdoored model m p can be created through data poisoning attacks @xcite . specifically, given a training corpus @xmath0, y i )} n @xmath1, where x i is a textual input, and y i is the corresponding label. the attacker poisons a subset of instances s ⊆ d, using f (•). the poisoning function f (•) transforms (x, y) to (x ′ , y ′ ), where x ′ is a corrupted x with backdoor triggers, y ′ is the target label assigned by the attacker. a backdoor model m p can be obtained by training on s. alternatively, attackers may compromise a benign model m b in post-training stage by maliciously modifying its weights to respond to specific triggers, effectively converting m b into a backdoored model m p (kurita et al., 2020; li et al., 2021a). 6 model merge for sanitization given a backdoored model m p , we aim to sanitize it via model merging. we hypothesize that by taking into account several other models, the backdoor signal in a single one will be reduced. specifically, we use a list of models {m k } n-1 @xmath2, such as public model hubs, and merge them with m p to form a sanitized model m ′ , our approach offers a versatile solution, allowing for the integration of various model merging operations into equation (1). our study mainly considers the arithmetic mean across all models for integration. 7 concretely, assume that the model weights of m p and {m k } n-1 @xmath3, then the weights w ′ of the merged model m ′ is ## experiments this section conducts a series of studies to examine the efficacy of our approach against multiple prominent backdoor attacks. ## conclusion our study showcases the effectiveness of model merging in mitigating backdoor attacks on pretrained language models (plms). through detailed experiments, we prove our method's robustness in various contexts. our approach is versatile, not limited by specific models, data sources, or training methods, and it sidesteps the need for specialized merging techniques. importantly, it acts as an inference-stage defense, eliminating the requirement for access to training data or the retraining of affected models. our method stands out by offering a significant reduction in attack success rate without sacrificing the accuracy on clean sets. we believe its simplicity will encourage further investigation into inference-stage defenses for backdoor threats in plms, a critical aspect of improving their security.