SEEP : Training Dynamics Grounds Latent Representation Search for Mitigating Backdoor Poisoning Attacks (2024)

Abstract: Modern NLP models are often trained on public datasets drawn from diverse sources, rendering them vulnerable to data poisoning attacks. These attacks can manipulate the model’s behavior in ways engineered by the attacker. One such tactic involves the implantation of backdoors, achieved by poisoning specific training instances with a textual trigger and a target class label. Several strategies have been proposed to mitigate the risks associated with backdoor attacks by identifying and removing suspected poisoned examples. However, we observe that these strategies fail to offer effective protection against several advanced backdoor attacks. To remedy this deficiency, we propose a novel defensive mechanism that first exploits training dynamics to identify poisoned samples with high precision, followed by a label propagation step to improve recall and thus remove the majority of poisoned instances. Compared with recent advanced defense methods, our method considerably reduces the success rates of several backdoor attacks while maintaining high classification accuracy on clean test sets.

## introduction the success of deep learning models is largely driven by training on extensive datasets. compared to the costly effort of labeling, the ease of obtaining uncurated data makes it an attractive option for training competitive models @xcite . the increasing use of public datasets from open-source communities, such as huggingface, raises important security concerns. these data hosting platforms often lack stringent data quality control processes, permitting the unregulated upload of datasets by any users. this reliance on untrustworthy data potentially exposes the models to backdoor attacks, where malicious users manipulate or poison data samples to imbue the victim model with specific misbehavior. for instance, a compromised sentiment analysis model, engineered to bias toward particular viewpoints or commercial products, could influence public perception or affect market trends. backdoor attacks aim to alter the predictive behavior of a victim model when presented with specific triggers. the attackers often accomplish this by either poisoning the training data @xcite or modifying the model parameters @xcite . this study concentrates on the former approach, also known as backdoor poisoning attack. in such attacks, backdoor triggers are inserted into a small portion of the training data, with their corresponding labels remaining altered. as a result, models trained on these poisoned datasets function normally with clean data but exhibit manipulated misbehavior when encountering backdoor triggers. considering the potential damage from backdoor attacks, several defensive strategies have been proposed. these methods primarily depend on either anomaly detection @xcite @xcite or robust training @xcite . however, these methods either significantly compromise the model's generalization performance @xcite or only offer protection against simple poisoning attacks, e.g., insertion-based attacks @xcite . in this paper, we propose a method that first automatically identifies a small number of anomalous instances in the training set, which is then followed by a label propagation process over the hidden representation of the training samples. the process is illustrated in figure 1 , which shows that high-precision seed examples (gray points) are identified (this process is automatic, based on their training dynamics, see §3.) by contrast, without these poisoned seeds, it would be very difficult to accurately identify poisoned outliers from the hidden representation alone, particularly due to the presence of two distinct outlier groups (circled in the figure). our ablation studies find that a combined two-step approach is necessary: seeding with a precise but small number of poisoned samples, followed by label propagation to iteratively add samples greedily by confidence. in contrast to previous defense methods, our approach is not predicated on a specific form of attack, nor does it require access to a clean dataset. comprehensive experimental results demonstrate the superiority of our approach over numerous sophisticated defenses across diverse datasets and types of backdoor attacks. furthermore, our technique effectively covers the models trained on datasets with low poisoning rates, where existing advanced baselines provide inadequate protection. recently, nlp models have also been shown to be vulnerable to backdoor attacks. generally, two primary categories of backdoor attacks have emerged. the first stream, data poisoning, involves tampering with the training data of the victim models, where a small percentage of the data has been manipulated @xcite . in this method, the seminal work by @xcite used a random sentence as a backdoor trigger. however, @xcite later argue that such random sentences could disrupt the fluency and semantics of the original input, rendering poisoned examples easily detectable by an external language model. to overcome this issue, @xcite proposed using a controllable paraphraser @xcite to create syntacticlevel triggers. stealthy triggers can also be implanted using synonym replacement @xcite . the second category of backdoor attacks involves weight poisoning, where triggers are embedded by directly modifying pre-trained weights of the victim model @xcite . ## method this section first outlines the general framework of backdoor poisoning attacks. then, we detail our defense method. backdoor attack via data poisoning consider a training corpus @xmath0, y i ) |d| @xmath1, where x i is a textual input, and y i is the corresponding label. the attacker poisons a subset of instances s ⊆ d, using a poisoning function f (•). the poisoning function f (•) transforms (x, y) to (x , y ), where x is a corrupted x with backdoor triggers, y is the target label assigned by the attacker. the victim models trained on s could be compromised for specific misbehavior according to the presence of triggers. nevertheless, the models behave normally on clean inputs, which ensures the stealthiness of the attack. ## experiments this section conducts a series of studies to examine the efficacy of seep against multiple prominent backdoor poisoning attacks. ## conclusion this study introduced a new framework designed to prevent backdoor attacks from data poisoning. firstly, the framework utilized the training dynamics of a victim model to detect seed poisoned samples, even in the absence of holdout clean datasets. subsequently, label propagation was employed to identify the remaining poisoned instances, based on their representational similarity to the seed instances. empirical evidence demonstrates that our proposed approach can significantly remedy the vulnerability of the victim model to multiple backdoor attacks outperforming multiple competitive baseline defense methods. papers), pages 1415-1420. https://doi .org/10.18653/v1/n19-1144 xiang zhang, junbo zhao, and yann lecun.