Expose Backdoors on the Way: A Feature-Based Efficient Defense against Textual Backdoor Attacks (2022)

Abstract: Natural language processing (NLP) models are known to be vulnerable to backdoor attacks, which poses a newly arisen threat to NLP models. Prior online backdoor defense methods for NLP models only focus on the anomalies at either the input or output level, still suffering from fragility to adaptive attacks and high computational cost. In this work, we take the first step to investigate the unconcealment of textual poisoned samples at the intermediate-feature level and propose a feature-based efficient online defense method. Through extensive experiments on existing attacking methods, we find that the poisoned samples are far away from clean samples in the intermediate feature space of a poisoned NLP model. Motivated by this observation, we devise a distance-based anomaly score (DAN) to distinguish poisoned samples from clean samples at the feature level. Experiments on sentiment analysis and offense detection tasks demonstrate the superiority of DAN, as it substantially surpasses existing online defense methods in terms of defending performance and enjoys lower inference costs. Moreover, we show that DAN is also resistant to adaptive attacks based on feature-level regularization. Our code is available at https://github.com/lancopku/DAN.

## introduction pre-trained language models (plms) have achieved unprecedented success in various nlp tasks @xcite @xcite . however, plms have been shown susceptible to backdoor attacks @xcite . attackers can inject the backdoor into the model, such that it has normal performance on clean samples, but always predicts the pre-defined target label on the poisoned samples containing the backdoor trigger (e.g., a rare word or sentence). when users download an infected plm and deploy it in the downstream applications, the attackers can easily manipulate the behavior of the model, even after users further finetune the model on a clean dataset @xcite @xcite . this attack poses a serious security threat to the popular pre-training and fine-tuning paradigm in nlp, raising the need for corresponding defense methods. compared with the widely-studied backdoor defense mechanisms in computer vision @xcite @xcite @xcite @xcite @xcite , textual backdoor defense still remains underexplored. one line of textual defense methods aims to detect whether the model is infected via reverseengineering backdoor triggers @xcite @xcite , which requires complicated and computationally expensive optimization procedures, thus impractical in the real usages. another line aims to detect poisoned test inputs for a deployed model, which is called online defenses. the main idea is to perturb the input and identify poisoned examples by detecting anomalies at the change of the input perplexity @xcite or output probabilities @xcite . nonetheless, they suffer from adaptive attacks @xcite and require time-consuming multiple inferences for each input. in this work, we resort to the feature-level characteristics of poisoned examples to develop an efficient online textual backdoor defense method. specifically, we observe that the poisoned samples and clean samples are separated in the intermediate feature space of poisoned plms (see figure 1 for an example under the badnet @xcite attack). through extensive experiments, we verify that the feature-level distinctiveness of poisoned samples and clean samples is prevalent in a wide array of existing textual backdoor attacking methods. motivated by the observation, we devise dan, a 80@xmath03'lp 80$33urmhfwlrqri)hdwxuhv &ohdq&odvv &ohdq&odvv 3rlvrqhg 0dkdodqrelv'lvwdqfh )uhtxhqf\ 'lvwulexwlrqri0dkdodqrelv'lvwdqfhv &ohdq6dpsohv 3rlvrqhg6dpsohv figure 1 : illustration of using distance scores for poisoned sample detection. we attack a bert @xcite model using the badnet @xcite ) method with a rare word trigger "mn" on the imdb @xcite sentiment analysis task. class 0 denotes negative and class 1 denotes positive. the target label is class 1. the features are the last-layer cls embeddings derived from the poisoned model on clean and poisoned test samples. we visualize the features using umap @xcite and plot the distribution of mahalanobis distances @xcite to clean validation data (right). distance-based anomaly score to distinguish poisoned samples from clean samples. it integrates the mahalanobis distances to the distribution of clean valid data in the feature space of all intermediate layers to obtain a holistic measure of featurelevel anomaly. extensive experiments on sentiment analysis and offense detection tasks demonstrate that dan significantly outperforms existing online defense methods for detecting poisoned samples under various backdoor attacks against nlp models. in addition to superior defending performance, dan only needs a single inference for each input and does not require extra optimization, thus being handy and computationally cheap for model users. furthermore, we notice that a line of works in computer vision @xcite @xcite improves the featurelevel stealthiness of backdoor attacks via regularizing the distance from poisoned samples to clean samples, which can be regarded as adaptive attacks against dan. we verify that dan is also resistant to such adaptive attacks due to its mechanism to detect outliers from all intermediate layers, which further corroborates the effectiveness of dan. backdoor defense researchers have developed a series of effective backdoor defense mechanisms for vision models, which can be generally categorized into two groups: (1) offline defenses @xcite @xcite @xcite target for detecting and mitigating the backdoor effect in models before deployment ; (2) online defenses @xcite @xcite aim to detect poisoned inputs at the inference stage. compared with the widely explored backdoor defense mechanisms in cv, the backdoor defense for nlp models is much less investigated. existing methods can be primarily classified into three types: (1) dataset protection methods @xcite seek to remove poisoned samples from public datasets, impractical for the weight poisoning scenario where users have already downloaded third-party models; (2) model diagnosis methods @xcite @xcite aim to identify whether the models are poisoned or not, which require expensive trigger reverse-engineering procedures, thus infeasible for resource-constrained users to conduct on big models; (3) online defense methods @xcite @xcite try to detect poisoned inputs for deployed models, which need multiple inferences for each input and have been shown vulnerable to adaptive attacks @xcite . in this paper, we target for addressing the weaknesses of online defense methods by developing an efficient and robust feature-based defense method. feature-based outlier detection our work is also related to works on feature-based outlier detection, such as the detection of out-of-distribution samples @xcite @xcite and adversarial samples @xcite @xcite . besides, some backdoor defense works in cv @xcite @xcite are also built on the dissimilarity between poisoned images and clean images in the feature space. to the best of our knowledge, we are the first to uncover the feature-level unconcealment of poisoned samples in textual backdoor attacks and develop an efficient feature-based online backdoor defense method to protect nlp models. ## conclusion in this work, we point out that the poisoned samples in textual backdoor attacks are distinguishable from clean samples in the intermediate feature space of a poisoned model. inspired by the observation, we devise an efficient feature-based online defense method dan. specifically, we integrate the distance scores from all intermediate layers to obtain the distance-based anomaly score for identifying poisoned inputs. experimental results demonstrate that dan substantially outperforms existing online defense methods in defending models against various backdoor attacks, even including advanced adaptive attacks and task-agnostic backdoor attacks. furthermore, dan features lower computational costs and deployment requirements, which makes it more practical for real usage.