Acquiring Clean Language Models from Backdoor Poisoned Datasets by Downscaling Frequency Space (2024)

Abstract: Despite the notable success of language models (LMs) in various natural language processing (NLP) tasks, the reliability of LMs is susceptible to backdoor attacks. Prior research attempts to mitigate backdoor learning while training the LMs on the poisoned dataset, yet struggles against complex backdoor attacks in real-world scenarios. In this paper, we investigate the learning mechanisms of backdoor LMs in the frequency space by Fourier analysis. Our findings indicate that the backdoor mapping presented on the poisoned datasets exhibits a more discernible inclination towards lower frequency compared to clean mapping, resulting in the faster convergence of backdoor mapping. To alleviate this dilemma, we propose Mu lti- Sc a le Lo w- R ank A daptation (MuScleLoRA), which deploys multiple radial scalings in the frequency space with low-rank adaptation to the target model and further aligns the gradients when updating parameters. Through downscaling in the frequency space, MuScleLoRA encourages the model to prioritize the learning of relatively high-frequency clean mapping, consequently mitigating backdoor learning. Experimental results demonstrate that MuScleLoRA outperforms baselines significantly. Notably, MuScleLoRA reduces the average success rate of diverse backdoor attacks to below 15% across multiple datasets and generalizes to various backbone LMs, including BERT, RoBERTa, and Llama2. The codes are publicly available at Anonymous.

## introduction despite the remarkable achievements of language models (lms) in various natural language processing (nlp) tasks @xcite , concerns arise due to the lack of interpretability in the internal mechanisms of lms, impacting their reliability and trustworthiness. a * corresponding authors. particular security threat to lms is backdoor attack @xcite . backdoor attack poisons a small portion of the training data by implanting specific text patterns (known as triggers). trained on the poisoned dataset, the target lm performs maliciously when processing samples containing the triggers, while behaving normally when processing clean text. prior works attempt to mitigate backdoor learning during training the target lm on the poisoned dataset @xcite . however, due to the stealthy nature of complex triggers in real-world scenarios, most existing defense methods fail to mitigate backdoor learning from such triggers, like specific text style @xcite or syntax @xcite . furthermore, most existing defense methods rely on empirical observations @xcite @xcite and lack thorough exploration of backdoor learning. to better understand backdoor learning, we investigate the learning mechanisms of lms in the frequency space on the poisoned datasets through fourier analysis. 1 the findings indicate that the backdoor mapping presented on the poisoned datasets exhibits a stronger inclination towards lower frequency compared to clean mapping. according to the extensively studied f-principle @xcite @xcite , which suggests that deep neural networks (dnns) tend to fit the mapping from low to high frequency during training, these results explain why backdoor mapping is notably easier to discern and converges faster for lms. inspired by the observation and thought above, we propose a general backdoor defense method named multi-scale low-rank adaptation (mus-clelora) to further mitigate backdoor learning. musclelora integrates multiple radial scalings in the frequency space with low-rank adaptation to the target lm and aligns gradients during parameter updates. by downscaling in the frequency space, musclelora encourages lms to prioritize relatively high-frequency clean mapping, thereby mitigating learning the backdoor on the poisoned dataset while enhancing clean learning. experimental results across multiple datasets and model architectures demonstrate the efficacy and generality of musclelora in defending against diverse backdoor attacks compared to baselines. specifically, we concentrate on the scenario where (1) the attacker poisons and releases the dataset on open third-party platforms, without gaining control of the downstream training; (2) the defender downloads the poisoned dataset and deploys the defense method to train the target lm, maintaining complete control of the training process. our contributions are summarized as follows: ## related works in this section, we cover related works that form the basis of this work from four perspectives: backdoor attack, backdoor defense, learning mechanisms of dnns, and parameter-efficient tuning (pet). backdoor attack. backdoor learning @xcite exploits the extra capacity @xcite of over-parameterized @xcite lms to establish a robust mapping between predefined triggers and the target output. one typical way to conduct backdoor attacks is dataset poisoning @xcite . recent studies for trigger implantation include inserting specific words @xcite or sentences @xcite that use shallow semantic features. besides, high-level semantics, like specific syntax @xcite ) and text style @xcite , are utilized as complex triggers. backdoor defense. backdoor defense aims to mitigate potential backdoors in victim lms and is categorized into training-stage and post-training defense. during training, the defender can perform poisoned weight removal @xcite @xcite , regularized training @xcite , and dataset purifying @xcite @xcite to mitigate backdoor learning. after training, the defender can employ trigger inversion @xcite @xcite , trigger detection @xcite , and poisoned input detection @xcite @xcite to discriminate potential backdoors. our proposed musclelora falls under regularized training, mitigating backdoor learning without detailed data inspection. previous work @xcite reduces model capacity by pet methods to mitigate backdoor learning. however, straightforward model capacity reduction with pet methods requires meticulously designed hyperparameters against different attacks and still struggles against complex stealthy triggers, like specific syntax @xcite . ## pilot experiments in this section, we conduct pilot experiments on the poisoned dataset, investigating the learning mechanisms of lms in the frequency space from the perspective of fourier analysis. generally, a backdoor model should satisfy: (1) maintaining performance on clean tasks, defined as clean mapping, which maps clean inputs to their corresponding clean labels, f c : intuitively, the implanted trigger ∆ on the poisoned dataset represents a straightforward recurring feature that lms can easily discern. a recent empirical study observes that the loss of backdoor mapping converges faster than that of clean mapping when training an lm on a poisoned dataset @xcite . to explain this observed convergence difference, we conduct fourier analyses on the training process of the backdoor lm. following the settings of kurita et al. (2020) and @xcite , we select specific words, i.e., cf, mn, bb, tq, and a sentence, i.e., i watch this 3d movie, as respective triggers to poison sst-2 @xcite . we choose bert base as the target lm and train it on the poisoned datasets. concurrently, we decompose the overall mapping of the backdoor lm into clean mapping f c and backdoor mapping f b by utilizing d c and d b , respectively. then, we conduct filteringbased fourier transformation (xu et al., 2020) (details are provided in appendix a) to f c and f b : r l×@xmath0, f(e) = y, extracting their respective low-frequency part y low clean , y low @xmath1, y high @xmath2. here @xmath3, @xmath4, l, d, and c denote input embedding, output logits, input text length, embedding dimension, and the number of categories, respectively. first, we calculate the low-frequency ratio (lfr) and high-frequency ratio (hfr) of both clean and backdoor mappings during training by equation 1. as shown in figure 1 , both clean and backdoor mappings exhibit significantly larger lfr compared to hfr, consistent with the low-frequency bias suggested by f-principle @xcite . specifically, the lfr of backdoor mapping consistently remains near 1.0, surpassing that of clean mapping. conversely, the hfr of clean mapping gradually increases, whereas the hfr of backdoor mapping is typically two orders of magnitude lower than that of clean mapping. these phenomena indicate that (1) backdoor mapping exhibits a stronger bias towards low frequency than clean mapping; to compare the convergence of clean and backdoor mappings in frequency space, we compute the relative errors re low , re high between output logits and ground-truth labels by equation 2. here, t low , t @xmath5, respectively. the results of relative errors are shown in figure 2 , where the red color indicates small relative errors. in both cases, re low decreases more rapidly than the corresponding re high , signifying faster convergence. this convergence difference aligns with the frequency principle, suggesting that lms tend to fit the mapping from low to high frequency. furthermore, re low of low-frequency-dominated backdoor mapping fluctuates initially and then rapidly decreases to a small value. compared to the gradual decrease of re low of clean mapping, backdoor mapping converges significantly faster. as mentioned above, (1) the lower-frequency inclination of backdoor mapping results in faster convergence of backdoor mapping, (2) the relatively high-frequency inclination of clean mapping leads to slower convergence of clean mapping. ## methodology findings in section 3 indicate that clean mapping exhibits a relatively high-frequency bias, leading to its slower learning compared to backdoor mapping. hence, an intuition to mitigate backdoor learning is to encourage lms to prioritize relatively highfrequency clean mapping. to this end, we propose musclelora, which utilizes multiple radial scalings with low-rank adaptation to the target lm and aligns gradients when updating parameters. the overview of musclelora is shown in figure 3 . inspired by @xcite that pet methods can reduce the capability of lm and thus mitigate backdoor learning, we incorporate multiple radial scalings @xcite with low-rank adaptation to reduce the model capacity and downscale clean mapping in the frequency space. for simplicity, we assume the fourier transform fℓ (ξ), ξ ∈ r d corresponding to the mapping f ℓ (x), @xmath6. subsequently, the compact support of fℓ (ξ) can be partitioned into s mutually disjointed concentric rings {a i } s @xmath7, ∀i ̸ = j, a i ∩ a @xmath8. therefore, fℓ (ξ) can be decomposed with indicators i(ξ ∈ a i ), as illustrated in equation 3 . for each fi ℓ (ξ), we apply radial scalings with appropriate scaling factor s i to downscale high frequency in a i , as illustrated in equation 4 . hence, in the corresponding physical space, the radial scalings are illustrated in equation 5 . consequently, f ℓ (x) can also be decomposed into: to approximate f scale,i ℓ with low-rank adaptation, we first freeze the target lm and insert lora modules into each attention layer. given that deeper layers tend to exhibit stronger low-frequency bias (xu and zhou, 2021), larger scaling factors are required in the shallow layers to appropriately downscale clean tasks. however, in practice, excessive scaling factors could potentially lead to underfitting. therefore, we conduct multiple radial scalings with appropriate scaling factors to the low-rank projected input lx within the lora module at the penultimate linear layer, as illustrated in equation 6. here, w 0 ∈ r d×d denotes the original frozen weight, @xmath9, @xmath10, and ⊙ denotes hadamard product. like vanilla lora, the magnitude of parameter updates can be represented as r (l ⊙ s), which can be directly added to the original weights to mitigate the inference latency. as the relatively high-frequency clean mapping is downscaled by multiple radial scalings in the frequency space, the inclination towards the lowfrequency-dominated backdoor mapping is mitigated. therefore, with the low-rank adaptation that reduces the model capacity, the target lm is likely to prioritize the more general clean mapping on the poisoned dataset. however, with the burgeoning scale of lms, the accompanying increase in extra capacity of lms poses challenges to effectively mitigate backdoor learning through straightforward model capacity reduction with pet methods. motivated by the notable phenomenon that the gradient directions derived from poisoned data and clean data often conflict with each other @xcite , we assume the defender can access a small amount of clean auxiliary data, usually comprising a few dozen instances and readily obtainable through manual labeling. consequently, we align the gradient of the target lm with clean auxiliary data to further mitigate the influence of the poisoned gradient. specifically, when obtaining the original gradient g from a batch of untrustworthy training data, we simultaneously calculate the clean gradient g c from a batch of clean auxiliary data. subsequently, we align g to the direction of g c to obtain the aligned gradient g a , as illustrated in equation 7 : nonetheless, aligning gradients to a restricted set of clean auxiliary data, as indicated by @xcite , may lead to suboptimal learning. therefore, we incorporate a fraction of the original gradient g to mitigate suboptimal learning on clean tasks, as illustrated in equation 8 . here, the hyperparameter µ denotes the ratio of the original gradient accepted. subsequently, parameter updates are performed based on the modified gradient ĝ: practically, we linearly increase µ from 0 to the maximum value µ max throughout the training epochs. consequently, the target lm primarily learns from backdoor-mitigated gradients during the early training phase, where µ approaches 0, and gradually incorporates more information with increasing µ to alleviate suboptimal learning in the later stages of training. ## experiments in this section, we extensively evaluate muscle-lora. we first outline the setup in section 5.1. subsequently, in section 5.2, we demonstrate that musclelora outperforms baselines significantly in backdoor mitigation across several datasets. additionally, we analyze the contributions of various strategies employed in musclelora in section 5.3, conduct fourier analyses to explain the mechanisms of musclelora in the frequency space in section 5.4, and extend musclelora to large language models (llms) in section 5.5. ## conclusions in this paper, we conduct fourier analyses to investigate the mechanisms of backdoor learning, revealing a notable inclination towards lower frequencies in backdoor mapping compared to clean mapping. inspired by this observation, we proposed mus-clelora, a general backdoor defense method. by downscaling in the frequency space, musclelora encourages lms to prioritize the learning of relatively high-frequency clean mapping, consequently mitigating the learning backdoor mapping. experimental results show the efficacy of musclelora in defending against diverse backdoor attacks. notably, musclelora exhibits generality across various backbone lms, including bert, roberta, gpt2-xl, and llama2.