Mitigating Data Poisoning in Text Classification with Differential Privacy (2021)

Abstract: NLP models are vulnerable to data poisoning attacks. One type of attack can plant a backdoor in a model by injecting poisoned examples in training, causing the victim model to misclassify test instances which include a specific pattern. Although defences exist to counter these attacks, they are specific to an attack type or pattern. In this paper, we propose a generic defence mechanism by making the training process robust to poisoning attacks through gradient shaping methods, based on differentially private training. We show that our method is highly effective in mitigating, or even eliminating, poisoning attacks on text classification, with only a small cost in predictive accuracy.

## introduction while test-time attacks have been shown to affect various nlp models @xcite @xcite , trainingtime attacks are also highly problematic. among them, data poisoning attacks, where the adversary plants a backdoor in a model by poisoning the training data with text containing a specific trigger phrase, have been shown to be highly successful @xcite @xcite . once trained on poisoned data, a model will misbehave by producing specific incorrect predictions on inputs containing the trigger. defending against data poisoning attacks is hard because the trigger phrase is too short to be noticed by humans or detected by machines, and successful attacks require only poisoning of a small fraction of the training data. mitigation methods have been proposed to counter such attacks @xcite by looking for potential poison examples in the training data. while these methods can detect irregular examples, they assume a knowledgeable defender who is aware of the specific style of attack and poison crafting method, which is an unrealistic assumption in practice. in this work, we propose a generic defence method for data poisoning attacks, suitable for defending against black-or white-box attacks. instead of finding abnormal examples from the input data, our method seeks to make a model's training process robust to data poisoning, by smoothing the gradient from each training example. our insight is that the strong association between a trigger and a specific model prediction as learned from the poison examples is highly surprising, which may cause large changes in the model parameters during training. accordingly, we employ differentially private training @xcite for learning, which acts by smoothing the training gradients inside stochastic gradient descent, and provides theoretical guarantees of the model sensitivity to individual examples. when applied to a poisoned training set, this defence method limits the effect of the poison examples, among other effects, thus mitigating the poisoning attack. differentially private training has previously been proposed as a defence against poisoning attacks in other fields, such as image classification @xcite and recommendation systems @xcite , however it has not yet been established if this method works in natural language processing. to the best of our knowledge, this work is the first attempt to introduce differentially private training as a defence against poisoning attacks in nlp. our empirical results on a series of text classification tasks show that our method is highly effective in alleviating, and sometimes even eliminating, the effect of poisoning attacks, with only minimal degradation on predictive accuracy. ## data poisoning in text classification data poisoning attacks have been shown to be successful on text classification tasks @xcite @xcite . in ## poisoning-resistant training with differential privacy our approach to defending against data poisoning attacks is to diminish the effect of data poisoning during training. more precisely, we hope to limit the influence of an individual training example on the model learning, through bounding the impact on the model parameters. the intuition is that effective poison examples are likely to include highly surprising feature patterns and model predictions (i.e., a strong association between the trigger and the target class), which will provoke large changes in the model parameters via the corresponding loss gradient. sufficiently limiting the training gradients, in terms of both gradient magnitude and direction, can help prevent the establishment of an association between the trigger and target class. such constraints may unintentionally limit the influence of some benign examples; our hypothesis is one of disproportionate impact on poison examples. differentially private training (dpt) @xcite is well-known task in differential privacy (dp) @xcite with the goal of preventing a training algorithm (e.g., sgd) from leaking the membership of any individual training example when releasing the result of training. dp protects privacy of training data, by guaranteeing that the distribution of training outputs is (almost) invariant to arbitrary perturbation of any one training example (see definition 1 below). definition 1 let ϵ > 0 and δ ∈ (0, 1). if a randomized training algorithm a satisfies, for any two training sets d, d ′ differing on any one example and any set of possible models s, although this appears different to our goal of defending against poisoning, there are close links between differential privacy and defences @xcite . intuitively since dp guarantees that output distribution is pointwise algorithm 1: gradient shaping in dpt for mitigating data poisoning effects. gradient descent θt+1 ← θt -ηtg t 7 end smooth, a smoothness guarantee on expected outputs can be proved as a corollary. @xcite use this observation to bound the reduction to attacker cost for an adversary poisoning a learner, showing that more poison examples are needed to achieve greater cost reductions (or incremental reward) for a broad range of attacker cost functions. @xcite uses this expected output stability to certify robustness of networks to test-time adversarial examples: a radius around a test example is established within which no perturbation would flip the network's prediction. by achieving dp, we bound the effect of a training example (or a small collection of training examples where the privacy budget ϵ is strong), will have on the model parameters. this bounded effect extends to poison examples. and due to the post processing inequality in dp @xcite , any further computation applied to a differentially-private output, provided that the sensitive training data is not leveraged further, preserves the same ϵ, δ privacy level. this bounded effect therefore extends from model parameters to test predictions, as required of the present setting of text classification. a widely-used dpt strategy is the dp-sgd algorithm @xcite which employs two gradient operations to achieve differential privacy upon processing each individual training example for model updates: 1) clipping: bounding the gradient by clipping its norm to a small constant and 2) perturbation: introducing random perturbation to the gradient by adding gaussian noise. 2 how clipping and perturbation are used in dpt is shown in algorithm 1: the gradient is first bounded by the ratio c ∥g t (x)∥ with the clipping coefficient c, and then added to gaussian noise with zero-mean and c-scaled standard deviation σ (also called the noise multiplier). in our context, both strategies exactly align with our goal of shaping irregular poisoned gradients: clipping norm helps reduce gradient magnitude while adding noise enables adjusting gradient direction. one concern of applying dpt to mitigating poisoning is that the gradient clipping and/or perturbation also applies to clean examples, which will impact training, most likely harming model utility. however, our empirical results show that settings for the clipping coefficient c and noise multiplier σ exist which only slightly degrade accuracy but largely prevent an attack ( §4). ## conclusions we propose a highly effective defence based on differentially private training which can mitigate the effect of data poisoning attacks on text classification models. we show that our method can mitigate both black-and white-box attacks that use different triggers, by significantly reducing the attack success rate but only incurring negligible reductions in predictive accuracy. language understanding. in advances in neural information processing systems, volume 32. curran associates, inc. xiang zhang, junbo zhao, and yann lecun. 2015.