C ERTIFIED R OBUSTNESS TO A DVERSARIAL L ABELF LIPPING A TTACKS VIA R ANDOMIZED S MOOTHING
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper considers label-ﬂipping attacks, a type of data poisoning attack where an adversary relabels a small number of examples in a training set in order to degrade the performance of the resulting classiﬁer.
In this work, we propose a strategy to build linear classiﬁers based on deep features that are certiﬁably robust against a strong variant of label-ﬂipping, where the adversary can target each test example independently.
In other words, for each test point, our classiﬁer makes a prediction and includes a certiﬁcation that its prediction would be the same had some number of training labels been changed adversarially.
Our approach leverages randomized smoothing, a technique that has previously been used to guarantee test-time robustness to adversarial manipulation of the input to a classiﬁer.
Further, we obtain these certiﬁed bounds with no additional runtime cost over standard classiﬁcation.
On the Dogﬁsh binary classiﬁcation task from ImageNet, in the face of an adversary who is allowed to ﬂip 10 labels to individually target each test point, the baseline undefended classiﬁer achieves no more than 29.3% accuracy; we obtain a classiﬁer that maintains 64.2% certiﬁed accuracy against the same adversary.
We generalize our results to the multi-class case, providing what we believe to be the ﬁrst multi-class classiﬁcation algorithm that is certiﬁably robust to label-ﬂipping attacks.
I NTRODUCTION
Modern classiﬁers, despite their widespread empirical success, are known to be susceptible to adversarial attacks.
In this paper, we are speciﬁcally concerned with so-called “data-poisoning” attacks (formally, causative attacks [Barreno et al.
2006; Papernot et al.
2018]), where the attacker manipulates some aspects of the training data in order to cause the learning algorithm to output a faulty classiﬁer.
Work in this area includes label-ﬂipping attacks (Xiao et al., 2012), where the labels of a training set can be adversarially manipulated to decrease performance of the trained classiﬁer; general data poisoning, where both the training inputs and labels can be manipulated (Steinhardt et al., 2017); and backdoor attacks (Chen et al., 2017; Tran et al., 2018), where the training set is corrupted so as to cause the classiﬁer to deviate from its expected behavior when triggered by a speciﬁc pattern, without causing a noticeable drop in overall accuracy.
However, unlike the alternative “test-time” adversarial setting, where reasonably effective defenses exist to build adversarially robust classiﬁers, relatively little work has been done on building classiﬁers that are certiﬁably robust to targeted data poisoning attacks.
In this work, we propose a strategy for building classiﬁers that are certiﬁably robust against labelﬂipping attacks.
In particular, we propose a pointwise certiﬁed defense—by this we mean that with each prediction, the classiﬁer includes a certiﬁcation guaranteeing that its prediction would not be different had it been trained on data with some number of labels ﬂipped.
Existing works on certiﬁed defenses make statistical guarantees over the entire test set, but they make no guarantees as to the robustness of a prediction on any particular test point.
Thus, while these algorithms provide general robustness to any single corruption, a determined adversary could still cause a speciﬁc test point to be misclassiﬁed.
We therefore consider the threat of a worst-case adversary that can make a training set perturbation to speciﬁcally target each test point individually .
This motivates a defense that can certify each of its individual predictions, as we present here.
To the best of our knowledge, this work represents the ﬁrst pointwise certiﬁed defense to data poisoning attacks.
Our approach leverages randomized smoothing (Cohen et al., 2019), a technique that has previously been used to guarantee test-time robustness to adversarial manipulation of the input to a deep network.
However, where prior uses of randomized smoothing randomize over the input to the classiﬁer for test-time guarantees, we instead randomize over the entire training procedure of the classiﬁer .
Speciﬁcally, by randomizing over the labels during this training process, we obtain an overall classiﬁcation pipeline that is certiﬁed to be robust (i.e., to not change its prediction) when some number of labels are adversarially manipulated in the training set.
Although a naive implementation of this approach would not be computationally feasible, we show that by using a linear least-squares classiﬁer over the output of a pre-trained network, we can obtain these certiﬁed bounds with no additional runtime cost over standard classiﬁcation—and we again emphasize that this is the ﬁrst such certiﬁed defense against pointwise label-ﬂipping attacks, even for linear classiﬁers.
We evaluate our proposed classiﬁer on several benchmark datasets common to the data poisoning literature.
Speciﬁcally, we demonstrate that our randomized classiﬁer is able to achieve 75.7% certiﬁed accuracy on MNIST 1/7 even when the number of allowed label ﬂips would drive a standard, undefended classiﬁer to less than 50%.
Similar results in experiments on the Dogﬁsh binary classiﬁcation challenge from ImageNet and IMDB review sentiment database validate our technique for more challenging datasets, and we further experiment on the full MNIST dataset to demonstrate its effectiveness for multi-class classiﬁcation.
On top of this, our classiﬁer maintains a reasonably competitive non-robust accuracy (e.g., 94.5% on MNIST compared to 99.1% for the undefended classiﬁer).
R ELATED W ORK
Data-poisoning attacks A data-poisoning attack (Mu ˜ noz Gonz ´ alez et al., 2017; Yang et al., 2017) is an attack whereby an adversary corrupts some portion of a training set or adds new inputs, with the goal of degrading the performance of the learned model.
The attack can be targeted to cause poor performance on a speciﬁc test example or can instead simply reduce the overall test performance.
The adversary is assumed to have perfect knowledge of the learning algorithm, so security by design —as opposed to obscurity—is the only viable defense against such attacks.
The adversary is also typically assumed to have access to the training set and, in some cases, the test set.
Previous work has investigated attacks and defenses for data-poisoning attacks applied to feature selection (Xiao et al., 2015), SVMs (Biggio et al., 2011; Xiao et al., 2012), linear regression (Liu et al., 2017), and PCA (Rubinstein et al., 2009), to name a few.
Some attacks can even achieve success with “clean-label” attacks, inserting innocuous, poisoned yet “correctly” labeled training examples but causing the classiﬁer to perform poorly (Shafahi et al., 2018; Zhu et al., 2019).
For an overview of data poisoning attacks and defenses in machine learning, see Biggio et al.
A label-ﬂipping attack is a speciﬁc type of data-poisoning attack where the adversary is restricted to changing some of the training labels.
The number of allowed changes could be an exact number or a percentage of the total labels.
The classiﬁer is then trained on the corrupted training set, with no knowledge of which labels have been tampered with.
Unlike random label noise, for which many robust learning algorithms have been successfully developed (Natarajan et al., 2013; Liu & Tao, 2016; Patrini et al., 2017), adversarial label-ﬂipping attacks can be speciﬁcally targeted to exploit the structure of the learning algorithm, signiﬁcantly degrading performance.
Robustness to such attacks is therefore harder to achieve, both theoretically and empirically (Xiao et al., 2012; Biggio et al., 2011).
A common defense technique is sanitization , whereby a defender attempts to identify and remove or relabel training points that may have had their labels corrupted (Paudice et al., 2019; Taheri et al., 2019).
Unfortunately, recent work has demonstrated that this is often not enough against a sufﬁciently powerful adversary (Koh et al., 2018).
Certiﬁed defenses For a review on robustness to random noise, we refer the reader to Rousseeuw & Leroy (1987)—our focus is on adversarial noise, a much more challenging problem.
Existing works on certiﬁed defenses to adversarial data poisoning attacks typically focus on the regression case and provide broad statistical guarantees over the entire test distribution.
A common approach to such certiﬁcations is to show that a particular algorithm recovers some close approximation to the best linear ﬁt coefﬁcients (Liu et al., 2017;
Prasad et al., 2018; Shen & Sanghavi, 2019), or that the expected loss on the test distribution is bounded (Klivans et al., 2018; Chen & Paschalidis, 2018).
These results generally rely on assumptions on the train and test distributions: some assume sparsity in the coefﬁcients (Karmalkar & Price, 2018; Chen et al., 2013) or corruption vector (Bhatia et al., 2015); others require limited effects of outliers (Steinhardt et al., 2017).
As mentioned above, all of these methods fail to provide guarantees for individual test points.
Additionally, most of these statistical guarantees are not meaningful when applied to deep, non-linear architectures.
Randomized smoothing Since the discovery of adversarial examples (Szegedy et al., 2013; Goodfellow et al., 2015), the research community has been investigating techniques for increasing the robustness of deep networks to adversarial perturbations.
After a series of heuristic defenses (Metzen et al., 2017; Feinman et al., 2017), followed by attacks breaking them (Athalye et al., 2018; Carlini & Wagner, 2017), focus began to shift towards the development of provable robustness.
One approach which has gained popularity in recent work is randomized smoothing.
Rather than certifying the original classiﬁer f , randomized smoothing deﬁnes a new classiﬁer g whose prediction at an input x is the class assigned the most probability when x is perturbed with noise from some distribution µ and passed through f .
That is, g ( x ) =
arg max c P (cid:15) ∼ µ ( f ( x + (cid:15) )
=
c ) .
This new classiﬁer g is then certiﬁed as robust, ideally without sacriﬁcing too much accuracy compared to f .
The original formulation was presented by Lecuyer et al.
and borrowed ideas from differential privacy.
The above deﬁnition is due to Li et al.
and was popularized by Cohen et al.
(2019), who derived a tight robustness guarantee.
Follow-up work has focused on optimizing the training procedure of f (Salman et al., 2019) and extending the analysis to other types of distributions (Lee et al., 2019).
For more details, we refer the reader to Cohen et al.
A GENERAL VIEW OF RANDOMIZED SMOOTHING
We begin by presenting a general viewpoint of randomized smoothing.
Under our notation, randomized smoothing constructs an operator G ( µ, φ ) that maps a binary-valued 1 function φ : X → { 0 , 1 } and a smoothing measure µ : X → R + , with (cid:82) X µ ( x ) dx = 1 , to the expected value of φ under µ (that is, G represents the “vote” of φ weighted by µ ).
For example, φ could be a binary image classiﬁer and µ could be some small, random pixel noise applied to the to-be-classiﬁed image.
We also deﬁne a “hard threshold” version g ( µ, φ ) that returns the most probable output (the majority vote winner).
Formally,
[Equation]
where 1 {·} is the indicator function.
Intuitively, for two smoothing measures µ, ρ : X → R + that are very similar, we would expect that for most φ , even though G ( µ, φ ) and G ( ρ, φ ) may not be equal, the threshold function g should satisfy g ( µ, φ )
=
g ( ρ, φ ) .
Further, the degree to which µ and ρ can differ while still preserving this property should increase as G ( µ, φ ) approaches eitheror 1, because this increases the “margin” with which the function φ is 0 or 1 respectively over the measure µ .
More formally, we deﬁne a general randomized smoothing guarantee as follows.
Deﬁnition 1.
Let µ : X → R + be a smoothing measure over X , with (cid:82) X µ ( x ) dx = 1 .
Then a randomized smoothing robustness guarantee is a speciﬁcation of a distance measure d ( µ, ρ ) and a function f : [0 , 1] → R + such that for all φ : X → { 0 , 1 } and all measures ρ : X → R + of a given form,
[Equation]
For brevity, we will sometimes use p in place of G ( µ, φ ) , representing the probability with which the majority class wins the vote (this is analogous to p A in Cohen et al.
Instantiations of randomized smoothing This deﬁnition is rather abstract, so we highlight concrete examples of how it can be applied to achieve certiﬁed guarantees against adversarial attacks.
Example 1.
The randomized smoothing guarantee of Cohen et al.
(2019) uses the smoothing measures µ = N ( x 0 , σ 2 I ) , a Gaussian around the point x 0 to be classiﬁed, and ρ = N ( x 0 + δ, σ 2 I ) , a
Gaussian around x 0 perturbed by δ .
They prove that (2) holds for all classiﬁers φ if we deﬁne
[Equation]
where KL( · ) denotes KL divergence and Φ − 1 denotes the inverse CDF of the Gaussian distribution.
Although this previous work focused on the case of randomized smoothing of continuous data via
Gaussian noise, this is by no means a requirement of the approach.
For instance, Lee et al.
(2019) considers an alternative approach for dealing with discrete variables.
Example 2.
The randomized smoothing guarantee of Lee et al.
(2019) uses the factorized smoothing measure in d dimensions µ α,K ( x ) =
Π di =1 µ α,K,i ( x i ) , deﬁned with respect to parameters α ∈ [0 , 1] , K ∈ N , and a base input z ∈ { 0 , . .
. , K } d , where
[Equation]
[Equation] [Equation] } , x
[Equation]
= z i , with x i being the i th dimension of x .
ρ α,K is similarly deﬁned for a perturbed input z (cid:48) .
They guarantee that (2) holds if we deﬁne
[Equation]
In words, the smoothing distribution is such that each dimension is independently perturbed to one of the other K values uniformly at random with probability 1 − α .
F is a combinatorial function;
F α,K,d ( p ) is deﬁned as the maximum number of dimensions—out of d total—by which µ α,K and ρ α,K can differ such that a set with measure p under µ α,K is guaranteed to have measure at leastunder ρ α,K .
Lee et al.
(2019) prove that this value is independent of z and z (cid:48) .
Finally, concurrent work has considered a more general form of randomized smoothing that doesn’t require strict assumptions on the distributions but is still able to provide similar guarantees.
Example 3 (Generic bound from Anonymous (2019)) .
Given any two smoothing distributions µ, ρ , we have the generic randomized smoothing robustness certiﬁcate, ensuring that (2) holds with deﬁnitions
[Equation]
Randomized smoothing in practice For most classiﬁers, the expectation G ( µ, φ ) cannot be computed exactly, and so we must resort to Monte Carlo approximation.
In this “standard” form of randomized smoothing, we draw multiple random samples from µ and use these to construct a highprobability bound on G ( µ, φ ) for certiﬁcation.
More precisely, this bound should be a lower bound on G ( µ, φ ) when the hard prediction g ( µ, φ )
= 1 and an upper bound otherwise; this ensures in both cases that we underestimate the true certiﬁed robustness for the classiﬁer g .
The procedure is shown in Algorithm 2 in Appendix A.
These estimates can then be plugged into a randomized smoothing robustness guarantee to provide a high probability certiﬁed robustness bound for the classiﬁer g .
L ABEL -F LIPPING R OBUSTNESS
We now present the main contribution of this paper, a technique for using randomized smoothing to provide certiﬁed robustness against label-ﬂipping attacks.
Speciﬁcally, we ﬁrst propose a generic strategy for applying randomized smoothing to certify a prediction function against pointwise label ﬂipping attacks.
We show how this general approach can be made tractable using linear leastsquares classiﬁcation combined with pre-trained deep features, and we use the Chernoff inequality to analytically bound the relevant probabilities for the randomized smoothing procedure.
Notably, although we are employing a randomized approach, the ﬁnal algorithm does not use any random sampling, but rather relies upon a convex optimization problem to compute the certiﬁed robustness.
To motivate the approach, we note that in prior work, randomized smoothing was applied at test time with the function φ :
X → { 0 , 1 } being a (potentially deep) classiﬁer that we wish to smooth.
[Equation]
) where X ≡ h ( x 1: n ) and λ = (1 + q ) ˆ σ 2 k 2 n σ max ( X T X ) σ min ( X T X ) for j = 1 , . .
. , m do
[Equation]
and let p (cid:63) = max(1 − B | t (cid:63) | , 1 / 2) where B | t (cid:63) | is the Chernoff bound (13) evaluated at | t (cid:63) | .
Output: Prediction ˆ y j = 1 { t (cid:63) ≥ 0 } and certiﬁcation that prediction will remain constant for up to r training label ﬂips, where
[Equation]

[Equation]
(or a larger number of ﬂips using the exact method of Lee et al.
end for
However, there is no requirement that the function φ be a classiﬁer at all; the theory holds for any binary-valued function.
Instead of treating φ as a trained classiﬁer, we consider φ to be an arbitrary learning algorithm which takes as input a training dataset { x i , y i } ni =1 ∈ ( X × { 0 , 1 } ) n and an additional example 2 x n +1 without a corresponding label, which we aim to predict.
In other words, the combined goal of φ is to ﬁrst train a classiﬁer and then predict the label of the new example.
Thus, we consider test time outputs to be a function of both the test time input and the training data that produced the classiﬁer.
This perspective allows us to reason about how changes to training data affect the classiﬁer at test time, reminiscent of work on inﬂuence functions of deep neural networks (Koh & Liang, 2017;
Yeh et al., 2018).
When applying randomized smoothing in this setting, we randomize over the labels in the training set, rather than over the test-time input to be classiﬁed.
Analogous to previous applications of randomized smoothing, if the majority vote of the classiﬁers trained with these randomly sampled labels has a large margin, it will confer a degree of adversarial robustness to some number of adversarially corrupted labels.
To formalize this intuition, consider two different assignments of n training labels Y 1 , Y 2 ∈ { 0 , 1 } n which differ on precisely r labels.
Let µ (resp.
ρ ) be the distribution resulting from independently ﬂipping each of the labels in Y 1 (resp.
Y 2 ) with probability q .
It is clear that as r increases, d ( µ, ρ ) should also increase.
In fact, it is simple to show (see Appendix B.3 for derivation)
that the exact
KL divergence between these two distributions is
[Equation]
Plugging in the robustness guarantee (5), we have that g ( µ, φ ) =
g ( ρ, φ ) so long as
[Equation]
This implies that for any test point, as long as (10) is satisﬁed, g ’s prediction (the majority vote weighted by the smoothing distribution) will not change if an adversary corrupts the training set from Y 1 to Y 2 , or indeed to any other training set that differs on at most r labels.
We can tune the noise hyperparameter q to achieve the largest possible upper bound in (10); more noise will likely decrease the margin of the majority vote p , but will also decrease the divergence.
Computing a tight bound
This approach has a simple closed form, but the bound is not tight.
We can derive a tight bound via a combinatorial approach as in Lee et al.
By precomputing the quantities F − 1 1 − q, 1 ,n ( r ) from Equation (4) for each r , we can simply compare G ( µ, φ ) to each of these and thereby certify robustness to the highest possible number of label ﬂips.
This computation can be expensive, but it provides a signiﬁcantly tighter robustness guarantee, certifying approximately twice as many label ﬂips for a given bound on G ( µ, φ ) (See Figure 5 in Appendix D).
We make use of this tighter bound in our experiments, but we emphasize that meaningful results can be achieved even with the looser bound, which is orders of magnitude cheaper to compute.
4. E FFICIENT IMPLEMENTATION VIA LEAST SQUARES CLASSIFIERS
There may appear to be one major impracticality of the algorithm proposed in the previous section, if considered naively: treating the function φ as an entire training-plus-single-prediction process would require that we train multiple classiﬁers, over multiple random draws of the labels y , all to make a prediction on a single example.
In this section, we describe a sequence of tools we employ to restrict the architecture and training process in a manner that drastically reduces this cost, bringing it in line with the cost of classifying a single example.
The full procedure, with all the parts described below, can be found in Algorithm 1.
Linear least-squares classiﬁcation The fundamental simplifying assumption we make in this work is to restrict the “training” process done by the classiﬁer φ to be done via a linear leastsquares solve.
Given the training set { x i , y i } ni =1 , we assume that there exists some feature mapping h : R d → R k (where k < n ), which typically would consist of a deep network pre-trained on a similar task, or possibly trained in an unsupervised fashion on x 1: n (i.e. independent of the training labels, which are presumed to be potentially poisoned).
This may seem to be a strong assumption, but similar assumptions are commonly made when using pre-trained models or in the meta-learning setting, and the transferability of pre-trained features is well documented (Donahue et al., 2014;
Bo et al., 2010; Yosinski et al., 2014).
Given this feature mapping, let X
= h ( x 1: n ) ∈ R n × k be the training point features and let y
= y 1: n ∈ { 0 , 1 } n be the labels.
Our training process consists of ﬁnding the least-squares ﬁt to the training data, i.e., we ﬁnd parameters ˆ β ∈ R k via the normal equation ˆ β = (cid:0) X T X (cid:1) − 1 X T y
and then we make a prediction on the new example via the linear function h ( x n +1 )
ˆ β .
Although it may seem odd to ﬁt a classiﬁcation task with least-squares loss, binary classiﬁcation with linear regression is known to be equivalent to Fisher’s linear discriminant (Mika, 2003) and often works quite well in practice.
The real advantage of the least-squares approach in this setting is that it reduces the prediction to a linear function of y , and thus randomizing over the labels is straightforward.
Speciﬁcally, letting
[Equation]
(11) the prediction h ( x n +1 ) ˆ β can be equivalently given by α T y (this is effectively just the kernel representation of the linear classiﬁer).
Thus, we can simply compute α one time and then randomly sample many different sets of labels in order to build a standard randomized smoothing bound.
Further, we can pre-compute just the X (cid:0) X T X (cid:1) − 1 term and reuse it for each test point.
(cid:96) 2 regularization for better conditioning Unfortunately, it is unlikely to be the case that the training points are well-behaved for linear regression in the feature space.
To address this, we instead solve an (cid:96) 2 regularized version of least-squares.
This is a common tool for solving systems with ill-conditioned or random design matrices (Hsu et al., 2014; Suggala et al., 2018).
Luckily, there still exists a pre-computable closed-form solution to this problem, whereby we solve
[Equation]
The other parts of our algorithm remain unchanged.
Following results in Suggala et al.
(2018), we set the regularization parameter λ
= (1 + q ) ˆ σ 2 k 2 n κ ( X T X ) where ˆ σ 2 = (cid:107) y − X ˆ [Equation] 22 n − k is an estimate of the variance (Dicker, 2014) and κ ( · ) is the condition number equal to the ratio of the largest and smallest singular values.
The (1 + q ) term is to help account for the variance caused by label ﬂips.
Efﬁcient tail bounds via the Chernoff inequality Even more compelling, due to the linear structure of this prediction, we can forego a sampling-based approach entirely and directly bound the tail probabilities using Chernoff bounds.
Because the underlying binary prediction function φ will output the label 1 for the test point whenever α T y ≥ 1 / 2 and 0 otherwise, we can derive an analytical upper bound on the probability that g predicts one label or the other via the Chernoff bound.
By upper bounding the probability of the opposite prediction, we simultaneously derive a lower bound on p which can be plugged in to (10) to determine the classiﬁer’s robustness.
Concretely, we can upper bound the probability that the classiﬁer outputs the label 0 by
P ( α
[Equation]
Conversely, the probability that the classiﬁer outputs the label 1 has the analogous upper bound which is the same as (13) but evaluated at − t .
Thus, we can solve the minimization problem unconstrained over t , and then let the sign of t dictate which label to predict and the value of t determine the bound.
The objective (13) is log-convex in t and can be easily solved by Newton’s method.
Note that in some cases, neither Chernoff upper bound will be less than 12 , meaning we cannot determine the true value of g ( µ, φ ) .
In these cases, we simply deﬁne the classiﬁer’s prediction to be determined by the sign of t .
While we can’t guarantee that this classiﬁcation will match the true majority vote, our algorithm will certify a robustness to 0 ﬂips, so the guarantee is still valid.
We avoid abstaining so as to assess our classiﬁer’s non-robust accuracy.
The key property we emphasize is that, unlike previous randomized smoothing applications, the ﬁnal algorithm involves no sampling whatsoever .
Instead, the prediction probabilities are bounded directly via the Chernoff bound, without any need for Monte Carlo approximation.
Thus, the method is able to generate truly certiﬁable robust predictions using approximately the same complexity as traditional predictions with a deep network.
E XPERIMENTS
Following Koh & Liang (2017) and Steinhardt et al.
, we perform experiments on MNIST 1/7, the IMDB review sentiment dataset (Maas et al., 2011), and the Dogﬁsh binary classiﬁcation challenge taken from ImageNet.
We run additional experiments on the full MNIST dataset; to the best of our knowledge, this is the ﬁrst multi-class classiﬁcation algorithm with certiﬁed robustness to label-ﬂipping attacks.
For each dataset and each noise level q we report the certiﬁed test set accuracy at r training label ﬂips.
That is, for each possible number of ﬂips r , we plot the fraction of the test set that was both correctly classiﬁed and certiﬁed to not change under at least r ﬂips.
Because the above are binary classiﬁcation tasks, one could technically achieve a certiﬁed accuracy of 50% at r = ∞ (or 10% for MNIST) by letting g be constant.
A constant classiﬁer would be inﬁnitely robust, but it is not a very meaningful baseline.
However, we include the accuracy of such a classiﬁer in our plots (black dotted line) as a reference.
To properly justify the need for such certiﬁed defenses, and to get a sense of the scale of our certiﬁcations, we also generated label-ﬂipping attacks against the undefended binary MNIST and Dogﬁsh models.
Following previous work, the undefended models were implemented as convolutional neural networks, trained on the clean data, with all but the top layer frozen—this is equivalent to logistic regression on the learned features.
For each test point we recorded how many ﬂips were required to change the network’s prediction.
This number serves as an upper bound for the robustness of the network on that test point, but we note that our attacks were quite rudimentary and could almost certainly be improved upon to tighten this upper bound.
Appendix C contains the details of our attack implementations.
Results on MNIST The MNIST 1/7 dataset (LeCun et al., 1998) consists of just the classes 1 and 7, totalling 13007 training points and 2163 test points.
We trained a simple convolutional neural network on the other eight MNIST digits to learn a 50-dimensional feature embedding and then [Figure] [Figure] calculated Chernoff bounds for G ( µ, φ ) as described in Section 4.1.
Figure 1a displays the certiﬁed accuracy on the test set for varying probabilities q .
As in prior work on randomized smoothing, the noise parameter q balances a trade-off; as q increases, the required margin | G ( µ, φ ) − 12 | to certify a given number of ﬂips decreases.
On the other hand, this results in more noisy training labels, which reduces the margin and therefore results in lower robustness and often lower accuracy.
Figure 1b depicts the certiﬁed accuracy for the full MNIST test set.
See Appendix B for derivations of the bounds and optimization algorithm in the multi-class case.
In addition to this being a signiﬁcantly more difﬁcult classiﬁcation task, our pre-trained classiﬁer could not rely on features learned from other handwritten digits; instead, we used the features from a network trained on the Omniglot dataset (Lake et al., 2015), as the task of classifying handwritten characters is quite similar.
Despite the lack of ﬁne-tuned features, our algorithm still achieves signiﬁcant certiﬁed accuracy under a large number of adversarial label ﬂips.
We observed that regularization did not make a large difference for the multi-class case, possibly due to the inaccuracy of the residual term when calculating the noise estimate.
See Figure 4 in Appendix D for the effect of (cid:96) 2 regularization on the OLS solution.
We note that at a slight cost to non-robust accuracy, the regularization results in substantially higher certiﬁed accuracy at almost all radii.
A similar effect was observed for all our datasets and at all values of q .
[Figure] Results on Dogﬁsh The Dogﬁsh dataset contains images from the ImageNet dog and ﬁsh synsets, 900 training points and 300 test points from each.
We trained a ResNet-
(He et al., 2016) on the standard ImageNet training set but removed all images labeled as any kind of dog or ﬁsh.
Our pretrained feature embedder was therefore capable of extracting image features but had no concept of features speciﬁc to either class.
We used PCA to reduce the 2048 dimensional feature space todimensions before solving, to ensure the system was not underdetermined.
Figure 2 displays the results of our poisoning attack along with our certiﬁed defense.
Under the undefended model, more than 50% of the test points can be successfully attacked with no more than 7 label ﬂips, whereas our model with q = 0 .
001 can certiﬁably correctly classify 67.83% of the test points under the same attack.
It would take more than three times as many ﬂips for each test point individually to push our model to less than 50% certiﬁed accuracy, at which point the undefended classiﬁer would be reduced to at most 0.83%.
Because the predictions on the test points can be changed by ﬂipping such a small fraction of the training set, high values of q reduce the randomized models’ predictions to almost pure chance— this means we are unable to achieve the margins necessary to certify a large number of ﬂips.
We therefore found that smaller levels of noise were necessary to achieve high certiﬁed test accuracy.
This suggests that the more susceptible the original, non-robust classiﬁer is to label ﬂips, the lower q should be set for the corresponding randomized classiﬁer.
Results on IMDB Figure 3 plots the result of our randomized smoothing procedure on the IMDB review sentiment dataset.
This dataset contains 25,000 training examples and 25,000 test examples, evenly split between “positive” and “negative”.
To extract the features we applied the Google News pre-trained Word2Vec to all the words in each review and averaged them.
This feature embedding is considerably noisier than that of an image dataset, as most of the words in a review are irrelevant to sentiment classiﬁcation.
Indeed, Steinhardt et al.
(2017) also found that the IMDB dataset was much more susceptible to adversarial corruption than images when using bag-of-words features.
Consistent with this observation, we found smaller levels of noise to result in larger certiﬁed accuracy.
We expect signiﬁcant improvements could be made with a more reﬁned choice of feature embedding.
We also observed that increasing q does not signiﬁcantly decrease the non-robust accuracy.
We believe this is because the limitation of these classiﬁers is not the label noise, but rather the difﬁculty in classifying the imperfect feature embeddings with linear regression.
Thus, the sign of t (cid:63) was frequently correct and remained on the same side of 0 as q increased, but the margin of G ( µ, φ ) decreased too rapidly to certify a large number of ﬂips.
This was similarly observed in the case of full MNIST, where the pre-trained features came from a network trained on the Omniglot dataset and therefore were not as informative for classiﬁcation.
C ONCLUSION
In this work we have presented a certiﬁed defense against a strong class of adversarial label-ﬂipping attacks where an adversary can ﬂip labels to cause a misclassiﬁcation on each test point separately.
This contrasts with previous data poisoning settings, which have typically only considered an adversary who wishes to degrade the classiﬁer’s accuracy on the test distribution as a whole, and it brings the adversary’s objective more in line with that of backdoor attacks and test-time adversarial perturbations.
Leveraging randomized smoothing, a method originally developed for certifying robustness to test-time perturbations, we presented a classiﬁer that can be certiﬁed robust to these pointwise train-time attacks.
We then offered a tractable algorithm for evaluating this classiﬁer which, despite being rooted in randomization, can be computed with no Monte Carlo sampling whatsoever, resulting in a truly certiﬁably robust classiﬁer.
This algorithm results in the ﬁrst multi-class classiﬁcation algorithm that is certiﬁably robust to label-ﬂipping attacks.
There are several avenues for improvement to this line of work, perhaps the most immediate being the method for learning the input features.
For example, rather than considering ﬁxed features generated by a pre-trained deep network, extensions could leverage neural tangent kernels (Jacot et al., 2018) to also allow for efﬁcient learning with perturbed inputs or more ﬂexible representations.
Using this or other approaches, the analysis could be extended to other types of smoothing distributions applied to the training data, such as randomizing over the input features to provide robustness to more general data poisoning attacks.
Conversely, we also hope that our defense to this threat model will inspire the development of more powerful train-time attacks, against which future defenses can be evaluated.
A G ENERIC R ANDOMIZED S MOOTHING A LGORITHM Algorithm
Input: function φ : X → { 0 , 1 } , number of samples N , smoothing distribution µ , failure probability δ >for i = 1 , . . .
, N do
Sample x i ∼ µ and compute y i =
φ ( x i ) end for
Compute approximate smoothed output [Equation] (cid:40) [Equation]
N
N (cid:88) i =
y
i ≥ 1 /(cid:41) (14)
Compute bound ˆ G ( µ, φ ) such that with probability 1 − δ [Equation] G (
[Equation]
G (
[Equation]
if
[Equation]
[Equation]
=
[Equation]
if
[Equation]
[Equation]
[Equation] = [Equation] Output: Prediction ˆ g ( µ, φ ) and probability bound ˆ G ( µ, φ ) , or abstention if ˆ g ( µ, φ ) and ˆ G ( µ, φ ) are on different sides of 12 .
B T HE MULTI - CLASS SETTING
Although the notation and algorithms are slightly more complex, all the methods we have discussed in the main paper can be extended to the multi-class setting.
In this case, we consider a class label y ∈ { 1 , . . .
, K } , and we again seek some smoothed prediction such that the classiﬁer’s prediction on a new point will not change with some number r ﬂips of the labels in the training set.
B. R ANDOMIZED SMOOTHING IN THE MULTI - CLASS CASE
We here extend our notation to the case of more than two classes.
Recall our original deﬁnition of
G ,
[Equation]
where φ : X → { 0 , 1 } .
More generally, consider a classiﬁer φ : X → [ K ] , outputting the index of one of K classes.
Under this formulation, for a given class c ∈ [ K ] , we have
[Equation]
(16) where φ c ( x ) = 1 { φ ( x ) =
c } is the indicator function for if φ ( x ) outputs the class c .
In this case, the hard threshold g is evaluated by returning the class with the highest probability.
That is,
[Equation]
B. L INEARIZATION AND C HERNOFF BOUND APPROACH FOR THE MULTICLASS CASE
Using the same linearization approach as in the binary case, we can formulate an analogous approach which forgoes the need to actually perform random sampling at all and instead directly bounds the randomized classiﬁer using the Chernoff bound.
Adopting the same notation as in the main text, the equivalent least-squares classiﬁer for the multiclass setting ﬁnds some set of weights
[Equation]
18) where Y ∈ { 0 , 1 } n × K is a binary matrix with each row equal to a one-hot encoding of the class label (note that the resulting ˆ β ∈ R k × K is now a matrix, and we let ˆ β
i refer to the i th column).
At prediction time, the predicted class of some new point x n +1 is simply given by the prediction with the highest value, i.e.,
[Equation]
Alternatively, following the same logic as in the binary case, this same prediction can be written in terms of the α variable as
[Equation]
where Y i denotes the i th column of Y i .
In our randomized smoothing setting, we again propose to ﬂip the class of any label with probability q , selecting an alternative label uniformly at random from the remaining K − 1 labels.
Assuming that the predicted class label is i , we wish to bound the probability that
[Equation] [Equation]
for all alternative classes i (cid:48) .
By the Chernoff bound, we have that
[Equation]
[Equation]
The random variable Y ji − Y ji (cid:48) takes on three different distributions depending on if
y j = i , if y j = i (cid:48) , or if y j (cid:54) =
i and y j (cid:54) =
i (cid:48) .
Speciﬁcally, this variable can take on the terms +1 , 0 , − 1 with the associated probabilities
[Equation]
Combining these cases directly into the Chernoff bound gives log P ( α T Y i < α T Y i (cid:48) ) ≤ min t>
[Equation]
Again, this problem is convex in t , and so can be solved efﬁciently using Newton’s method.
And again since the reverse case can be computed via the same expression we can similarly optimize this in an unconstrained fashion.
Speciﬁcially, we can do this for every pair of classes i and i (cid:48) , and return the i which gives the smallest lower bound for the worst-case choice of i (cid:48) .
B. KL D IVERGENCE B OUND
To compute actual certiﬁcation radii, we will derive the KL divergence bound for the the case of
K classes.
Let µ, ρ be deﬁned as in Section 4, except that as in the previous secftion when a label is ﬂipped with probability q it is changed to one of the other K − 1 classes uniformly at random.
Let µ i and ρ i refer to the independent measures on each dimension which collectively make up the factorized distributions µ and ρ (i.e., µ ( x )
= (cid:81) di
=1 µ i ( x ) ).
Further, let Y i 1 be the i th element of Y 1 , meaning it is the “original” class which may or may not be ﬂipped when sampling from µ .
First noting that each dimension of the distributions µ and ρ are independent, we have
[Equation]
Plugging in the robustness guarantee (5), we have that g ( µ, φ ) =
g ( ρ, φ ) so long as
[Equation]
Setting K = 2 recovers the divergence term (9) and the bound (10).
C D ESCRIPTION OF L ABEL -F LIPPING A TTACKS ON MNIST 1/7 AND D OGFISH
Due to the dearth of existing work on label-ﬂipping attacks for deep networks, our attacks on MNIST and Dogﬁsh were quite straightforward; we expect signiﬁcant improvements could be made to tighten this upper bound.
For Dogﬁsh, we used a pretrained Inception network (Szegedy et al., 2016) to evaluate the inﬂuence of each training point with respect to the loss of each test point (Koh & Liang, 2017).
As in prior work, we froze all but the top layer of the network for retraining.
Once we obtained the most inﬂuential points, we ﬂipped the ﬁrst one and recomputed approximate inﬂuence using only the top layer for efﬁciency.
After each ﬂip, we recorded which points were classiﬁed differently and maintained for each test point the successful attack which required the fewest ﬂips.
When this was ﬁnished, we also tried the reverse of each attack to see if any of them could be achieved with even fewer ﬂips.
For MNIST we implemented two similar attacks and kept the best attack for each test point.
The ﬁrst attack simply ordered training labels by their (cid:96) 2 distance from the test point in feature space, as a proxy for inﬂuence.
We then tried ﬂipping these one at a time until the prediction changed, and we also tried the reverse.
The second attack was essentially the same as the Dogﬁsh attack, ordering the test points by inﬂuence.
To calculate inﬂuence we again assumed a frozen feature map; speciﬁcally, using the same notation as Koh & Liang (2017), the inﬂuence of ﬂipping the label of a training point z =
( x, y ) to z − =
( x, 1 − y ) on the loss at the test point z test is:
[Equation]
[Figure] [Equation]
[Figure] [Equation] [Figure] [Equation]
[Figure] [Equation] [Figure]
[Equation]
[Figure]
[Equation]
For logistic regression, these values can easily be computed in closed form
[Figure]
D A DDITIONAL P LOTS
[Figure]
[Figure]