B ad A cts: A Universal Backdoor Defense in the Activation Space (2024)

Abstract: Backdoor attacks pose an increasingly severe security threat to Deep Neural Networks (DNNs) during their development stage. In response, backdoor sample purification has emerged as a promising defense mechanism, aiming to eliminate backdoor triggers while preserving the integrity of the clean content in the samples. However, existing approaches have been predominantly focused on the word space, which are ineffective against feature-space triggers and significantly impair performance on clean data. To address this, we introduce a universal backdoor defense that purifies backdoor samples in the activation space by drawing abnormal activations towards optimized minimum clean activation distribution intervals. The advantages of our approach are twofold: (1) By operating in the activation space, our method captures from surface-level information like words to higher-level semantic concepts such as syntax, thus counteracting diverse triggers; (2) the fine-grained continuous nature of the activation space allows for more precise preservation of clean content while removing triggers. Furthermore, we propose a detection module based on statistical information of abnormal activations, to achieve a better trade-off between clean accuracy and defending performance. Extensive experiments on diverse datasets and against diverse attacks (including syntax and style attacks) demonstrate that our defense achieves state-of-the-art performance.

## introduction backdoor attack @xcite @xcite is an increasingly severe security threat to deep neural networks (dnns) when building or deploying with open datasets, cloud platforms, and public pre-trained models. it aims to embed a covert backdoor function into a dnn model, such that the backdoored model behaves normally on normal samples but returns an attackerspecified target label for samples manipulated by the attacker (i.e., by adding triggers). the behavior of backdoored models on clean inputs is indistinguishable from that of benign models, making them highly concealed and raising significant safety issues in the application of nlp models. in response to such threats, researchers have recently explored backdoor sample purification methods @xcite @xcite , which aim to remove the backdoor trigger while preserving the integrity of the clean content within input samples. this allows the protected model to predict both clean and poisoned samples correctly. this approach differs from previous efforts that primarily focused on backdoor sample detection @xcite @xcite ; their primary strategy was to detect and then reject poisoned samples, preventing the attacker from triggering the backdoor behavior. however, a higher level of defense would enable correct predictions for backdoor samples (i.e., correcting predictions from the attacker's target label to the ground-truth label), thereby enhancing the model's backdoor robustness. existing backdoor sample purification efforts have been almost exclusively conducted in the word space. for instance, @xcite observed that adding context-independent trigger words compromises textual fluency, and thus dealt with this by removing words that caused an abnormal increase in perplexity. moreover, @xcite and @xcite noticed that injected trigger words/sentences dominate the prediction for backdoor samples, so they proposed to remove words that have excessively high attribution scores to achieve purification. while these strategies effectively counteract word-space triggers, they are ill-equipped to handle more sophisticated feature-space triggers, such as those manipulating text style @xcite or syntactic structure @xcite . the underlying issue is that these approaches predominantly rely on the removal of explicit trigger words from poisoned samples, which fails to address feature-space triggers that operate through subtle transformations of linguistic attributes. additionally, these methods significantly undermine the model's performance on clean data. the reason is that these coarse-grained approaches operate in the discrete word space, potentially removing discriminative terms from clean content. to overcome the limitations of word-space methods, we propose a universal method for backdoor sample purification in the activation space. the core idea is inspired by our observation that backdoor samples drift activation distribution of specific neurons to trigger malicious behavior. for example, as illustrated in figure 1 , the activation distribution of backdoor-unrelated neurons remains almost unchanged before and after adding triggers to clean test samples; in contrast, backdoor-related neurons capture the backdoor concept by deviating in their activation distribution, which in turn triggers the backdoor behavior. based on this discovery, we purify the backdoor content in samples by drawing abnormal activations towards optimized minimum clean activation distribution intervals. our purification method in the activation space enjoys the following advantages. first, individual neuron activations encapsulate linguistic properties ranging from surface-level information like words to higher-level semantic concepts such as syntactic structure and parts of speech @xcite . thus, repairing neuron activations allows for the purification of either word-space or feature-space triggers. second, the space of neuron activations is fine-grained and continuous, enabling the removal of backdoor triggers while maintaining as much original clean information as possible, thus achieving higher clean accuracy. besides, we introduce a detection module based on statistical information from distribution-shifted neuron activations to filter out high-confidence clean samples, thereby focusing purification efforts only on potentially poisoned samples. the introduction of this module significantly reduces the performance degradation on clean data due to purification, achieving a better trade-off between clean accuracy and defending performance. our defense pipeline consisting of the detection and purification modules, backdoor defense in the activation space (dubbed badacts), achieves state-of-the-art performance in both defending efficiency and clean accuracy across four datasets with four different attack types. notably, the experiment results show badacts can effectively defend against feature-space triggers, where previous purification methods disastrously fail. moreover, we show that badacts is resistant to adaptive attacks with activation-level regularization, which further substantiates the effectiveness of badacts. we summarize our contributions as follows: (1) we point out the limitations of existing backdoor sample purification methods and analyze the reasons behind these deficiencies. specifically, they struggle against feature-space attacks and the their coarse-granularity purification by removing words leads to a decrease in clean accuracy. (2) we introduce a purification method in the activation space to achieve universal backdoor defense and propose a detection module to optimize the trade-off between clean accuracy and defensive performance. textual backdoor defense existing backdoor defense for nlp models can be primarily classified into three types: (1) poison suppression methods aim to produce a backdoor-free classifier from the possibly poisoned training set by removing suspicious samples @xcite or modifying the training procedure @xcite @xcite to enhance robustness against data poisoning. (2) model-level backdoor detection and purification methods try to identify whether the models are poisoned or not @xcite @xcite @xcite , and remove the learned backdoor mapping by further fine-tuning or pruning @xcite @xcite . (3) sample-level backdoor detection and purification methods detect test samples embedded with the backdoor triggers @xcite @xcite @xcite and purify suspicious samples @xcite @xcite . similar approaches have also been developed for detecting other types of anomalous samples, such as out-of-distribution (ood) @xcite @xcite and adversarial samples @xcite . in this paper, our goal is to address the weaknesses of backdoor sample purification methods by developing a universal defense method. neuron-concept association neuron-concept association studies @xcite look into individual neurons, that are crucial for model performance or asso-ciated with specific linguistic properties. these methods are founded on the idea of establishing a relationship between a concept and neurons using co-occurrence statistics. researchers have applied this principle to identify and update neurons that store specific known facts @xcite or biases @xcite . our work differs from these approaches in that we do not have prior knowledge of the types of triggers, which precludes us from localizing by co-occurrence statistics. ## robustness to adaptive attacks considering that badacts is based on the observation that certain neurons responsible for the backdoor concept exhibit different activation distributions for clean samples and backdoor samples, pulling the activations of backdoor samples to the clean distribution during attacking may pose a potential threat to badacts. we notice that similar activation-level backdoor attacks have been studied in the vision area @xcite . therefore, to further test the robustness of badacts, we launch adaptive attacks by applying the activation-level regularization @xcite to four types of backdoor attacks on sst-2. as shown in ## conclusion in this paper, we propose a backdoor sample purification method that eliminates backdoor effects in the activation space instead of the word space exploited by existing methods. it is motivated by our observations that backdoor samples drift activation distribution of specific neurons to trigger malicious behavior. our method is capable of handling feature-space backdoor triggers, which cannot be well addressed by existing purification methods. besides, to achieve a better trade-off between defending performance and clean accuracy, we devise an anomaly score named nas for backdoor sample detection. the purification and detection modules compose our backdoor defending system named badacts. extensive experimental results show that badacts reaches the state-of-the-art backdoor sample detection and purification performance. what's more, badacts is resistant to activation-level adaptive attacks. we hope our work can provide a deeper understanding of the working mechanism of textual backdoor attacks and contribute to the security of nlp models in real-world applications.