P ERTURBATION T YPE C ATEGORIZATION FOR M ULTIPLE
(cid:96) p B OUNDED A DVERSARIAL R OBUSTNESS
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite the recent advances in adversarial training based defenses, deep neural networks are still vulnerable to adversarial attacks outside the perturbation type they are trained to be robust against.
Recent works have proposed defenses to improve the robustness of a single model against the union of multiple perturbation types.
However, when evaluating the model against each individual attack, these methods still suffer signiﬁcant trade-offs compared to the ones speciﬁcally trained to be robust against that perturbation type.
In this work, we introduce the problem of categorizing adversarial examples based on their (cid:96) p perturbation types.
Based on our analysis, we propose P ROTECTOR , a two-stage pipeline to improve the robustness against multiple perturbation types.
Instead of training a single predictor,
P ROTECTOR ﬁrst categorizes the perturbation type of the input, and then utilizes a predictor speciﬁcally trained against the predicted perturbation type to make the ﬁnal prediction.
We ﬁrst theoretically show that adversarial examples created by different perturbation types constitute different distributions, which makes it possible to distinguish them.
Further, we show that at test time the adversary faces a natural trade-off between fooling the perturbation type classiﬁer and the succeeding predictor optimized with perturbation speciﬁc adversarial training.
This makes it challenging for an adversary to plant strong attacks against the whole pipeline.
In addition, we demonstrate the realization of this trade-off in deep networks by adding random noise to the model input at test time, enabling enhanced robustness against strong adaptive attacks.
Extensive experiments on MNIST and CIFARshow that P ROTECTOR outperforms prior adversarial training based defenses by over 5% , when tested against the union of (cid:96) 1 , (cid:96) 2 , (cid:96) ∞ attacks.
I NTRODUCTION
There has been a long line of work studying the vulnerabilities of machine learning models to small changes in the input data.
In particular, most existing works focus on (cid:96) p bounded perturbations (Szegedy et al., 2013; Goodfellow et al., 2015).
While majority of the prior work aims at achieving robustness against a single perturbation type (Madry et al., 2018; Kurakin et al., 2017; Tram`er et al., 2018; Dong et al., 2018; Zhang et al., 2019; Carmon et al., 2019), real-world deployment of machine learning models requires them to be robust against various imperceptible changes in the input, irrespective of the attack type.
Prior work has shown that when models are trained to be robust against one perturbation type, such robustness typically does not transfer to attacks of a different type (Schott et al., 2018; Kang et al., 2019).
As a result, recent works have proposed to develop models that are robust against the union of multiple perturbation types (Tram`er & Boneh, 2019; Maini et al., 2020).
Speciﬁcally, these works consider adversaries limited by their (cid:96) p distance from the original input for p ∈ { 1 , 2 , ∞} .
While these methods improve the overall robustness against multiple perturbation types, when evaluating the robustness against each individual perturbation type, the robustness of models trained by these methods is still considerably worse than those trained on a single perturbation type.
Further, these methods are found sensitive to small changes in hyperparameters.
In this work, we propose an alternative view that does not require a single predictor to be robust against a union of perturbation types.
Instead, we propose to utilize a union of predictors to improve the overall robustness, where each predictor is specialized to defend against certain perturbation types.
In particular, we introduce the problem of categorizing adversarial examples based on their perturbation types.
Based on this idea, we propose P ROTECTOR , a two-stage pipeline that performs Perturbation
Type Categorization for Robustness against multiple perturbations .
Speciﬁcally, ﬁrst a perturbation type classiﬁer predicts the type of the attack.
Then, among the second-level predictors, P ROTECTOR selects the one that is the most robust to the predicted perturbation type to make ﬁnal prediction.
We validate our approach from both theoretical and empirical aspects.
First, we present theoretical analysis to show that for benign samples with the same ground truth label, their distributions become highly distinct when added with different types of perturbations, and thus can be separated.
Further, we show that there exists a natural tension between attacking the top-level perturbation classiﬁer and the second-level predictors – strong attacks against the second-level predictors make it easier for the perturbation classiﬁer to predict the adversarial perturbation type, and fooling the perturbation classiﬁer requires planting weaker (or less representative) attacks against the second-level predictors.
As a result, even an imperfect perturbation classiﬁer is sufﬁcient to signiﬁcantly improve the overall robustness of the model to multiple perturbation types.
Empirically, we show that the perturbation type classiﬁer generalizes well on classifying adversarial examples against different adversarially trained models.
Then we further compare P ROTECTOR to the state-of-the-art defenses against multiple perturbations on MNIST and CIFAR-10.
P ROTECTOR outperforms prior approaches by over 5% against the union of the (cid:96) 1 , (cid:96) 2 and (cid:96) ∞ attacks.
While past work has focused on the worst case metric against all attacks, on average they suffer signiﬁcant tradeoffs against individual attacks.
From the suite of 25 different attacks tested, the average improvement for P ROTECTOR over all the attacks w.r.t.
the state-of-art baseline defense is ∼ 15% on both MNIST and CIFAR10.
In particular, by adding random noise to the model input at test time, we further increase the tension between attacking top-level and second-level components, and bring in additional improvement of robustness against adaptive attackers.
Additionally, P ROTECTOR provides a modular way to integrate and update defenses against a single perturbation type.
R ELATED W ORK
Adversarial examples.
The realization of the existence of adversarial examples in deep neural networks has spun active research on attack algorithms and defense proposals (Szegedy et al., 2013).
Among different types of attacks (Madry et al., 2018; Hendrycks et al., 2019; Hendrycks & Dietterich, 2019; Bhattad et al., 2020), the most commonly studied ones constrain the adversarial perturbation within an (cid:96) p region of radius (cid:15) p around the original input.
To improve the model robustness in the presence of such adversaries, the majority of existing defenses utilize adversarial training (Goodfellow et al., 2015), which augments the training dataset with adversarial images.
Till date, different variants of the original adversarial training algorithm remain the most successful defenses against adversarial attacks (Carmon et al., 2019; Zhang et al., 2019;
Wong et al., 2020; Rice et al., 2020).
Other types of defenses include input transformation (Guo et al., 2018; Buckman et al., 2018) and network distillation (Papernot et al., 2016), but were rendered ineffective under stronger adversaries (He et al., 2017; Carlini & Wagner, 2017a; Athalye et al., 2018; Tramer et al., 2020).
Other works have explored the relation between randomizing the inputs and adversarial examples.
Tabacof & Valle (2016) analyzed the change in adversarial robustness with varying levels of noise.
Hu et al.
(2019) evaluated the robustness of a data point to random noise to detect adversarial examples, whereas
Cohen et al.
(2019) utilized randomized smoothing for certiﬁed robustness to adversarial attacks.
Defenses against multiple perturbation types.
Recent research has been drawn towards the goal of universal adversarial robustness.
Since (cid:96) p -norm bounded attacks are amongst the strongest attacks in adversarial examples literature, defending against a union of such attacks is an important step towards this end goal.
Schott et al.
; Kang et al.
(2019) showed that models that were trained for a given (cid:96) p -norm bounded attacks are not robust against attacks in a different (cid:96)
q region.
Succeeding work has aimed at developing one single model that is robust against the union of multiple perturbation types.
Schott et al.
(2018) proposed the use of multiple variational autoencoders to achieve robustness to multiple (cid:96) p attacks on the MNIST dataset.
Tram`er & Boneh (2019) used simple aggregations of multiple adversaries to achieve non-trivial robust accuracy against the union of the (cid:96) 1 , (cid:96) 2 , (cid:96) ∞ regions.
Maini et al.
(2020) proposed the MSD algorithm that takes gradient steps in the union of multiple (cid:96) p regions to improve multiple perturbation robustness.
In a related line of work, Croce & Hein (2020a) proposed a method for provable robustness against all (cid:96) p regions for p ≥ 1 .
Instead of presenting empirical results, they study the upper and lower bounds of certiﬁed robust test error on much smaller perturbation radii. Therefore, their work has a different focus, and is not directly comparable to empirical defenses studied in our work.
Detection of adversarial examples.
Multiple prior works have focused on detecting adversarial examples (Feinman et al., 2017; Lee et al., 2018;
Ma et al., 2018; Cennamo et al., 2020; Fidel et al., 2019;
Yin et al., 2019a;b).
However, most of these defenses have been shown to be vulnerable in the presence of adaptive adversaries (Carlini & Wagner, 2017a; Tramer et al., 2020).
In comparison, our work focuses on a more challenging problem of categorizing different perturbations types.
However, we show that by establishing a trade-off between fooling the perturbation classiﬁer and the individual (cid:96) p -robust models, even an imperfect perturbation classiﬁer is sufﬁcient to make our pipeline robust.
P ROTECTOR : P ERTURBATION T YPE C ATEGORIZATION FOR R OBUSTNESS
In this section, we discuss our proposed P ROTECTOR approach, which performs perturbation type categorization to improve the model robustness against multiple perturbation types.
We ﬁrst illustrate the P ROTECTOR pipeline in Figure 1, then discuss the details of each component.
[Figure]
sarial examples on MNIST.
At a high level, P ROTECTOR performs the classiﬁcation task as a two-stage process.
Given an input x , P ROTECTOR ﬁrst utilizes a perturbation classiﬁer C adv to predict its adversarial perturbation type.
Then, based on the (cid:96) p attack type predicted by C adv , P ROTECTOR uses the corresponding second-level predictor M p to provide the ﬁnal prediction, where M p is specially trained to be robust against the (cid:96) p attack.
Formally, let f θ be the P ROTECTOR model, then the ﬁnal prediction is:
[Equation]
Note that when the input is a benign image, it could be classiﬁed as any perturbation type by C adv , since all secondlevel predictors should achieve a high test accuracy on benign images.
As shown in Figure 1, although we consider the robustness against three attack types, i.e., (cid:96) 1 , (cid:96) 2 , (cid:96) ∞ perturbations, unless otherwise speciﬁed, our perturbation classiﬁer performs binary classiﬁcation between p = {{ 1 , 2 } , ∞} .
As will be discussed in Section 6, using two second-level predictors achieves better overall robustness than using three second-level predictors.
We hypothesize that compared to the (cid:96) ∞ adversarial examples, (cid:96) 1 and (cid:96) 2 attacks are harder to separate, especially when facing an adaptive adversary which aims to attack the entire pipeline.
To provide an intuitive illustration, we randomly sample 10K adversarial examples generated with PGD attacks on MNIST, and visualize the results of the Principal Component Analysis (PCA) in Figure 2.
We observe that the ﬁrst two principal components for (cid:96) 1 and (cid:96) 2 adversarial examples are largely overlapping, while those for (cid:96) ∞ are clearly from a different distribution.
Note that this simple visualization by no means suggest that (cid:96) 1 and (cid:96) 2 adversarial examples are not separable, it merely serves as a motivation.
T HEORETICAL A NALYSIS
In this section, we provide a theoretical justiﬁcation of our P ROTECTOR framework design.
First, we formally illustrate the setup of robust classiﬁcation against multiple (cid:96) p perturbation types, and we consider models trained for a binary classiﬁcation task.
Based on this problem setting, in Theorem 1, we show the existence of a classiﬁer that can separate adversarial examples belonging to different perturbation types.
Moreover, in Theorem 2, we show that our P ROTECTOR framework naturally offers a trade-off between fooling the perturbation classiﬁer C adv and the individual robust models
M p , thus it is extremely difﬁcult for adversaries to stage attacks against the entire pipeline.
Note that we focus on the simpliﬁed binary classiﬁcation task for the convenience of theoretical analysis, but our P ROTECTOR framework could improve the robustness of models trained on real-world image classiﬁcation benchmarks as well, and we will discuss the empirical examination in Section 6.
4. P ROBLEM S ETTING
Data distribution.
We consider a dataset of inputs sampled from the union of two multi-variate
Gaussian distributions D , such that the input-label pairs (x,y) can be described as:
[Equation]
where x = [ x 0 , x 1 , . .
, x d ] ∈ R d +1 and η = α √ d , such that the absolute value of the mean for any dimension is equal for inputs sampled from both the positive and the negative labels.
This setting demonstrates the distinction between a feature x 0 that is strongly correlated with the input label, and d weakly correlated features that are (independently) normally distributed with mean yη and variance σ 2 .
For the purposes of this work, we assume that ασ > 10 ( x 0 is strongly correlated) and d >(remaining d features are weakly correlated, but together represent a strongly correlated feature).
We adapt this problem setting from Ilyas et al.
(2019), where they used a stochastic feature x 0 =
y with probability p , as opposed to a normally distributed input feature as in our case. Our results hold in their setting as well. However, our setting better represents the true data distribution, where input features are seldom stochastically ﬂipped.
More discussion could be found in Appendix A. Perturbation types.
We focus our discussion on adversaries constrained within a ﬁxed (cid:96) p region of radius (cid:15) p around the original input, for p ∈ S
= { 1 , 2 , ∞} .
Such adversaries are frequently studied in existing work, primarily for ﬁnding the optimal ﬁrst-order adversaries for different perturbation types.
We deﬁne ∆ p,(cid:15) as the (cid:96) p threat model of radius (cid:15) and ∆ S =
(cid:83) p ∈S ∆ p,(cid:15) .
For a model f θ parametrized over θ , the objective of the adversary is to ﬁnd the optimal perturbation δ ∗ , such that:
[Equation]
where (cid:96) ( · , · ) is the cross-entropy loss.
Based on the model design in Section 3, we focus on discussing the separation of (cid:96) 1 and (cid:96) ∞ in the following theorems, but our proofs could also naturally be adapted to analyze the separability of other perturbation types.
4. S EPARABILITY OF A DVERSARIAL P ERTURBATIONS
Consider a standard classiﬁer M trained with the objective of correctly classifying the label of inputs x ∈ D .
Since the original distribution of the input data for each label is known to us, we ﬁrst aim to examine how adversaries conﬁned within different perturbation regions modify the input.
The goal of the adversary is to fool the label predictor M , by ﬁnding the optimal perturbation δ p ∀ p ∈ S .
The theorem below shows that the distributions of adversarial inputs within different (cid:96) p regions can be separated with a high accuracy, and we present the formal proof in Appendix B.
Theorem 1 (Separability of perturbation types) .
Given a binary Gaussian classiﬁer M trained on D , consider D yp to be the distribution of optimal adversarial inputs (for a class y ) against M , within (cid:96) p regions of radius (cid:15) p , where (cid:15)
= α , (cid:15) ∞ =
α/ √ d .
Distributions D yp ( p ∈ { 1 , ∞} ) can be accurately separated by a binary Gaussian classiﬁer C adv with a misclassiﬁcation probability P e ≤ 10 − 24 . The proof sketch is as follows.
We ﬁrst calculate the optimal weights of a binary Gaussian classiﬁer
M trained on D . Accordingly, for any input x ∈ D , we ﬁnd the optimal adversarial perturbation δ p ∀ p ∈ { 1 , ∞} against M . We discuss how these perturbed inputs x + δ p also follow a normal distribution, with shifted means. Finally, for data points belonging to a given classiﬁcation label, we show that C adv is able to predict the correct perturbation type with a very low classiﬁcation error.
We present the formal proof in Appendix B.
In Section 4.2, we showed that the optimal perturbations corresponding to different perturbation types belong to distinct data distributions, and it is fairly easy to separate them using a simple classiﬁer.
However, in the white-box setting, the adversary has knowledge of both the perturbation classiﬁer
C adv and specialized robust models M p at test time.
Therefore, the adversary can adapt the attack to fool the entire pipeline, instead of individual models M p alone.
Note that there are some overlapping regions among different (cid:96) p perturbation regions.
For example, every adversary could set δ p = 0 as a valid perturbation, and thus it is clearly not possible for the attack classiﬁer C adv to correctly classify the attack ( ∀ p ∈ { 1 , 2 , ∞} ) in such a scenario.
However, such perturbation is not useful, because all the base models can correctly classify unperturbed inputs with a high probability.
In the following theorem, we examine the robustness of our P ROTECTOR pipeline in the presence of such strong dynamic adversaries.
Theorem 2 (Adversarial trade-off) .
Given a data distribution D , adversarially trained models
M p,(cid:15) p , and an attack classiﬁer C adv that distinguishes perturbations of different (cid:96) p attack types for p ∈ { 1 , ∞} , The probability of successful attack for the worst-case adversary over the entire
P ROTECTOR pipeline is P e < 0 .
01 for (cid:15)
= α + 2 σ and (cid:15) ∞
=
α +2 σ √ d .
Here, the worst-case adversary refers to an adaptive adversary that has full knowledge of the defense strategy, and makes the strongest adversarial decision given the perturbation constraints.
In Appendix C.2, we discuss how (cid:15) 1 , (cid:15) ∞ are set so that the (cid:96) 1 and (cid:96) ∞ adversaries can fool M ∞ ,(cid:15) ∞ and M 1 ,(cid:15) 1 models respectively with a high success rate. Our proof sketch is as follows.
We ﬁrst show that when trained on D , an adversarially robust model M p can achieve robust accuracy of greater than 99% against the attack type it was trained for. On the contrary, when subjected to attacks outside the trained perturbation region, such robust accuracy reduces to under 2%. Then, we analyze the modiﬁed distributions of the perturbed inputs by different (cid:96) p attacks. Based on this analysis, we construct a simple decision rule for the perturbation classiﬁer C adv .
Finally, we compute the perturbation induced by the worst-case adversary.
We show that there exists a trade-off between fooling the perturbation classiﬁer C adv (to allow the alternate M p,(cid:15) p model to make the ﬁnal prediction for an (cid:96) q attack ∀ p, q ∈ { 1 , ∞} ; p (cid:54) = q ), and fooling the alternate M p,(cid:15) p model itself.
Here, by “alternate” we mean that for an (cid:96) q attack, the prediction is made by the M p,(cid:15) p model, where p, q ∈ { 1 , ∞} ; p (cid:54) = q .
We provide an illustration of the trade-off in Figure 1b, and present the formal proof in Appendix C.
T RAINING AND I NFERENCE
Having motivated P ROTECTOR through a toy-task in Section 4, we now scale the approach to deep neural networks for common image classiﬁcation benchmarks.
Speciﬁcally, following prior work on defending against multiple perturbation types, we evaluate on MNIST (LeCun et al., 2010) and
CIFAR-10 (Krizhevsky, 2012) datasets.
Now, we discuss the training details, a strong adaptive white-box attack against P ROTECTOR , and our inference procedure against such attacks.
5. T RAINING
To train our perturbation classiﬁer C adv , we create a dataset that includes adversarial examples of different perturbation types.
Speciﬁcally, we perform (cid:96) 1 , (cid:96) 2 , (cid:96) ∞ PGD attacks (Madry et al., 2018) against each of the two individual M p models used in P ROTECTOR .
Thus the size of our dataset istimes that of the original MNIST and CIFAR10 datasets respectively.
For the MNIST dataset, we use the M 2 , M ∞ models in P ROTECTOR , and we use M 1 , M ∞ models for CIFAR10.
The choice is made based on the robustness of { M 2 , M 1 } models against the { (cid:96) 1 , (cid:96) 2 } attacks respectively, as will be depicted in Table 2.
As discussed in Section 3, to assign the ground truth label for training the perturbation classiﬁer ( C adv ), we ﬁnd that it is sufﬁcient to assign the same label to (cid:96) 1 and (cid:96) 2 attacks.
In other words, C adv performs a binary classiﬁcation between (cid:96) 1 /(cid:96) 2 attacks and (cid:96) ∞ attacks.
In contrast with prior defenses against multiple perturbation types (Tram`er & Boneh, 2019; Maini et al., 2020), which require adversarial training, we ﬁnd that it is sufﬁcient to train our P ROTECTOR pipeline over a static dataset (constructed as mentioned above) to achieve high robustness.
Therefore, the training of our perturbation classiﬁer is fast and stable.
Speciﬁcally, using a single P100 GPU, our perturbation classiﬁer can be trained within 5 minutes on MNIST, and within an hour on CIFAR-10.
On the other hand, training state-of-the-art models robust to a single perturbation type require up todays to train on the same amount of GPU power, and existing defenses against multiple perturbation types take thrice as long as the training time for a model robust to a single perturbation type.
A key advantage of P ROTECTOR ’s design is that it can build upon existing defenses against individual perturbation types.
Speciﬁcally, we leverage the adversarially trained models developed in prior work (Zhang et al., 2019; Carmon et al., 2019)
as M p models in our pipeline, and the CNN architecture of C adv is also similar to a single M p model.
More details are deferred to Appendix D.
5. A DAPTIVE A TTACKS AGAINST THE P ROTECTOR P IPELINE
To generate adversarial examples against P ROTECTOR , the most straightforward approach is to generate the adversarial perturbation to optimize Equation 3 using existing attack algorithms.
Since the ﬁnal prediction of the pipeline only depends on a single M p model, the pipeline does not allow gradient ﬂow across the two levels, and thus makes it difﬁcult for gradient-based adversaries to attack P ROTECTOR .
Therefore, besides this standard adaptive attack, in our evaluation, we also consider a stronger adaptive adversary, which utilizes a combination of the predictions from each individual second-level M p models, rather than only utilizing the predictions from a single M p model with p = argmax C adv ( x ) alone.
Speciﬁcally, we modify f θ ( x ) in Equation 3 as follows:
[Equation]
where c p denotes the probability of the input x being classiﬁed as the perturbation type p by C adv .
We also experiment with other strategies of aggregating the predictions of different components, e.g., tuning hyper-parameters to balance among attacking C adv and each M p model, but these alternative methods do not perform better.
Note that Equation 4 is only used for the purpose of generating adversarial examples and performing gradient-based attack optimization. For consistency throughout the paper, we still use Equation 1 to compute the model prediction at inference (ﬁnal forward-propagation).
We do not see any signiﬁcant performance advantages of either choice at inference time, and brieﬂy report a comparison on two attacks in Appendix H.4.
5. I NFERENCE P ROCEDURE AGAINST A DAPTIVE A DVERSARIES
Though training the perturbation classiﬁer on a static dataset is sufﬁcient to achieve robustness using existing attack approaches, we observe that the accuracy drops when P ROTECTOR is presented with the stronger adaptive attacks discussed in Section 5.2.
To improve the model robustness against such adversaries, we add random noise to the input before feeding it into P ROTECTOR at the test time.
While Hu et al.
(2019) suggest that adding random noise does not help defend against adversarial inputs, it is the unique exhibition of the trade-off described in Theorem 2 that adversarial attacks against P ROTECTOR , on the contrary, are highly likely to fail when added with random noise.
Intuitively, the trade-off between fooling the two stages of P ROTECTOR conﬁnes the adversary in a very small region for generating successful adversarial attacks.
Consider the illustrative example in Figure 3, where the input x with the true label y = 0 is subjected to an (cid:96) ∞ attack.
We assume that the M ∞ ,(cid:15) ∞ model is a perfect classiﬁer for inputs within a ﬁxed (cid:15) ∞ ball.
The dotted line shows the decision boundary for [Table] a (cid:85) g e (cid:3) (cid:303) '' (cid:3) (cid:91)' (cid:91)'' + (cid:3) (cid:86) (cid:80) a (cid:79)(cid:79) (cid:3) (cid:303) ' (cid:3)
C (cid:79) a (cid:86) (cid:86) (cid:3)(cid:3) C (cid:79) a (cid:86) (cid:86) (cid:3) When the adversary adds a large perturbation δ (cid:48)(cid:48) , the prediction of M 1 for the resulted input x (cid:48)(cid:48) becomes wrong, but the perturbation classiﬁer also categorizes it as an M ∞ attack, thus the ﬁnal prediction of P ROTECTOR is still correct since it will be produced by M ∞ ,(cid:15) ∞ model instead.
On the other hand, when the adversary adds a small perturbation δ (cid:48) to fool the perturbation classiﬁer, adding a small amount of random noise can recover the correct prediction with a high probability.
Note that every point on the boundary of the noise region (yellow circle) is correctly classiﬁed by the pipeline.
In this way, adding random noise exploits an adversarial trade-off for P ROTECTOR to achieve a high accuracy against adversarial examples, in the absence of adversarial training.
In our implementation, we sample random noise z ∼ N (0 , I ) , and add ˆ z = (cid:15) 2 · z/ | z | 2 to the model input.
E XPERIMENTS
In this section, we present our experiments on MNIST and CIFAR-10 datasets.
We will discuss the results for both the perturbation classiﬁer C adv alone, and the entire P ROTECTOR pipeline.
6. E XPERIMENTAL S ETUP
Baselines.
We compare P ROTECTOR with the state-of-the-art defenses against multiple perturbation types, which consider the union of (cid:96) 1 , (cid:96) 2 , (cid:96) ∞ adversaries (Tram ` er & Boneh, 2019; Maini et al., 2020).
For Tram`er & Boneh (2019), we compare two variants of adversarial training: (1) the MAX approach, where for each image, among different perturbation types, the adversarial sample that leads to the maximum increase of the model loss is augmented into the training set; (2) the AVG approach, where adversarial examples for all perturbation types are included for training.
We also evaluate the MSD algorithm proposed by Maini et al.
(2020), which modiﬁes the standard PGD attack to incorporate the union of multiple perturbation types within the steepest decent itself.
In addition, we also evaluate
M 1 , M 2 , M ∞ models trained with (cid:96) 1 , (cid:96) 2 , (cid:96) ∞ perturbations separately, as described in Appendix D.
Attack evaluation.
We evaluate our methods with the strongest attacks in the adversarial learning literature, and with adaptive attacks speciﬁcally designed for P ROTECTOR (Section 5.2).
First, we utilize a comprehensive suite of both gradient-based and gradient-free attacks from the Foolbox library (Rauber et al., 2017).
Further, we also evaluate our method against the AutoAttack library from
Croce & Hein (2020c), which achieves the state-of-art adversarial error rates against multiple recently published models.
In line with prior work (Tram`er & Boneh, 2019; Maini et al., 2020), the radius of the { (cid:96) 1 , (cid:96) 2 , (cid:96) ∞ } perturbation regions is { 10 , 2 , 0 . 3 } for the MNIST dataset and { 12 , 0 . 5 , 0 . 03 } for the CIFAR10 dataset.
We present the full details of attack algorithms in Appendix F.
Following prior work (Tram`er & Boneh, 2019; Maini et al., 2020), for both MNIST and CIFAR-10, we evaluate the models on adversarial examples generated from the ﬁrst 1000 images of the test set.
Our main evaluation metric is the accuracy on all attacks , which means that for an input image, if any of the attack algorithm in our suite could successfully fool the model, then the input is a failure case.
6. E MPIRICAL PERTURBATION OVERLAP AND C HOICE OF (cid:15) p
While we justify the choice of perturbation sizes in our theoretical proofs in Appendix B.4 and C.2, in this section we demonstrate the empirical agreement of the choices of perturbation sizes we make
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
for our results on MNIST and CIFAR10 datasets.
To measure how often adversarial perturbations of different attacks overlap, we empirically quantify the overlapping regions by attacking a benign model with PGD attacks.
In Table 1 we report the range of the norm of perturbations in the alternate perturbation region for any given attack type.
The observed overlap is exactly 0% in all cases and the observation is consistent across MNIST and CIFAR10 datasets.
A similar analysis on attacking
P ROTECTOR can be found in Appendix G.
To examine the performance of the perturbation type classiﬁcation, we evaluate C adv on a dataset of adversarial examples, which are generated against the six models we use as the baseline defenses in our experiments.
Note that C adv is only trained on adversarial examples against the two M p models that are part of P ROTECTOR .
We observe that C adv transfers well across the board.
First, C adv generalizes to adversarial examples against new models; i.e., it preserves a high accuracy, even if the adversarial examples are generated against models that are unseen for C adv during training.
Further,
C adv also generalizes to new attack algorithms.
As discussed in Section 5.1, we only include PGD adversarial examples in our training set for C adv .
However, on adversarial examples generated by the
AutoAttack library, the classiﬁcation accuracy of C adv still holds up.
In particular, the accuracy is > 95% across all the individual test sets created.
These results suggest two important ﬁndings that validate our results in Theorem 1.
That is, independent of (a) the model to be attacked; and (b) the algorithm for generating the optimal adversarial perturbation, the optimal adversarial images for a given (cid:96) p region follow similar distributions.
We present the full results in Appendix H.1.
6. R ESULTS OF THE P ROTECTOR P IPELINE
Overall results.
In Table 2, we summarize the worst-case performance against all attacks within a given perturbation type for MNIST and CIFAR-10 datasets.
In particular, ‘Ours’ denotes the robustness of P ROTECTOR against the adaptive attacks described in Section 5.2, and ‘Ours*’ denotes the robustness of P ROTECTOR against standard attacks based on Equation 1. The adaptive strategy
[Table]
[Table]
effectively reduces the overall accuracy of P ROTECTOR by 2-5%, showing that incorporating the gradient and prediction information of all second-level predictors results in a stronger attack. Despite that we evaluate P ROTECTOR against a stronger adaptive adversary [Table]
in terms of the all attacks accuracy [Table] P ROTECTOR still outperforms all baselines by 6 . 9% on MNIST, and 8 . 9% on CIFAR-10.
Compared to the previous state-of-the-art defense against multiple perturbation types (MSD), the accuracy gain on (cid:96) ∞ attacks is especially notable, i.e., greater than 15% .
In particular, if we compare the performance gain on each individual attack algorithm, as shown in Appendix H.2 and H.3 for
MNIST and CIFAR-10 respectively, the improvement is also signiﬁcant, with an average accuracy increase of 15 .
5% on MNIST, and 14 . 2% on CIFAR-10.
These results demonstrate that P ROTECTOR considerably mitigates the trade-off in accuracy against individual attack types.
P ROTECTOR retains a high accuracy on benign images , as opposed to past defenses that have to sacriﬁce the benign accuracy for the robustness on multiple perturbation types.
In particular, the clean accuracy of P ROTECTOR is over 6% higher than such baselines on CIFAR-10, and the accuracy is similar to that of M p models trained for a single perturbation type.
The effect of noise.
As discussed in Section 5.3, though adding random noise is not required to defend against standard attacks, it is helpful in defending against the stronger adaptive adversary against our pipeline.
Speciﬁcally, in Table 3, we present the results on adversarial examples generated by PGD-based algorithms, which are amongst the strongest gradient-based attacks in the literature.
We observe a consistent improvement among all attacks, increasing the accuracy by up to 10% .
Different number of second-level M p predictors.
We also evaluate our P ROTECTOR approach with three second-level predictors, i.e., M 1 , M 2 and M ∞ , and we present the results in Table 3, this alternative design considerably reduces the overall accuracy of the pipeline model.
We hypothesize that this happens because the M 1 model is already reasonably robust against the (cid:96) 2 attacks, as shown in Table 2b.
However, having both M 1 and M 2 models allows adaptive adversaries to ﬁnd larger regions for fooling both C adv and M p , thus hurts the overall performance against adaptive adversaries.
C ONCLUSION
In this work, we propose P ROTECTOR , which performs perturbation type categorization towards achieving robustness against the union of multiple perturbation types.
Based on a simpliﬁed problem setup, theoretically, we demonstrate that adversarial inputs of different attack types naturally have different distributions and can be separated.
We further elaborate the existence of a natural tension for any adversary trying to fool our model – between fooling the attack classiﬁer and the specialized robust predictors.
Our empirical results on MNIST and CIFAR-10 datasets complement our theoretical analysis.
In particular, by posing another adversarial trade-off through the effect of random noise, our
P ROTECTOR pipeline outperforms existing defenses against multiple (cid:96) p attacks by over 5%.
Our work serves as a stepping stone towards the goal of universal adversarial robustness, by dissecting various adversarial objectives into individually solvable pieces and combing them via P ROTECTOR .
Our study opens up various exciting future directions, including the new problem of perturbation categorization, extending our approach to defend attacks beyond (cid:96) p adversarial examples, and deﬁning sub-classes of perturbation types to further improve the overall adversarial robustness.
A P ROBLEM S ETTING : T HEORETICAL A NALYSIS
In this section, we formally deﬁne the problem setting and motivate the distinctions made with respect to the problem studied by Ilyas et al.
The classiﬁcation problem consists of two tasks: (1)
Predicting the correct class label of an adversarially perturbed (or benign) image using adversarially robust classiﬁer M p ; and (2) Predicting the type of adversarial perturbation that the input image was subjected to, using attack classiﬁer C adv .
Setup We consider the data to consist of inputs to be sampled from two multi-variate Gaussian distributions such that the input-label pairs (x,y) can be described as:
[Equation]
where the input x ∼ N ( y µ , Σ ) ∈ R ( d +1) ; η = α/ √ d for some positive constant α ; µ = [ α, η, . . .
, η ] ∈ R +( d +1) and Σ
= σ 2 I ∈ R +( d +1) ×
( d +1) .
We can assume without loss of generality, that the mean for the two distributions has the same absolute value, since for any two distributions with mean µ 1 , µ 2 , we can translate the origin to µ 1 + µ 2.
This setting demonstrates the distinction between an input feature x 0 that is strongly correlated with the input label and d weakly correlated features that are normally distributed (independently) with mean yη and variance σ 2 each.
We adapt this setting from Ilyas et al.
(2019) who used a stochastic feature x 0 =
y with probability p , as opposed to a normally distributed input feature as in our case.
All our ﬁndings hold in the other setting as well, however, the chosen setting better represents true data distribution, with some features that are strongly correlated to the input label, while others that have only a weak correlation.
B S EPARABILITY OF PERTURBATION TYPES (T HEOREM 1)
In this section, our goal is to evaluate whether the optimal perturbation conﬁned within different (cid:96) p balls have different distributions and whether they are separable.
We do so by developing an error bound on the maximum error in classiﬁcation of the perturbation types.
The goal of the adversary is to fool a standard (non-robust) classiﬁer M .
C adv aims to predict the perturbation type based on only viewing the adversarial image, and not the delta perturbation.
First, in Appendix B.1 we deﬁne a binary Gaussian classiﬁer that is trained on the given task.
Given the weights of the binary classiﬁer, we then identify the optimal adversarial perturbation for each of the (cid:96) 1 , (cid:96) 2 , (cid:96) ∞ attack types in Appendix B.2.
In Appendix B.3 we deﬁne the difference between the adversarial input distribution for different (cid:96) p balls.
Finally, we calculate the error in classiﬁcation of these adversarial input types in Appendix B.4 to conclude the proof of Theorem 1.
B. B INARY G AUSSIAN C LASSIFIER
We assume for the purposes of this work that we have enough input data to be able to empirically estimate the parameters µ, σ of the input distribution via sustained sampling.
The multivariate
Gaussian representing the input data is given by:
[Equation]
We want to ﬁnd p ( y = y i | x ) ∀ y i ∈ {− 1 , +1 } .
From Bayesian Decision Theory, the optimal decision rule for separating the two distributions is given by:
[Equation]
Therefore, for two Gaussian Distributions N ( µ 1 , Σ 1 ) , N ( µ 2 , Σ 2 ) , we have:
[Equation]
Substituting (6) and (7) in (8), we ﬁnd that the optimal Bayesian decision rule for our problem is given by:
[Equation]
which means that the label for the input can be predicted with the information of the sign of x (cid:62) µ alone.
We can deﬁne the parameters W ∈ R d +1 of the optimal binary Gaussian classiﬁer M W , such that (cid:107) W (cid:107) 2 = 1 as:
[Equation]
B. O PTIMAL A DVERSARIAL P ERTURBATION AGAINST M W
Now, we calculate the optimal perturbation δ that is added to an input by an adversary in order to fool our model.
For the purpose of this analysis, we only aim to fool a model trained on the standard classiﬁcation metric as discussed in Section 4 (and not an adversarially robust model).
The parameters of our model are deﬁned in (10).
The objective of any adversary δ ∈ ∆ is to maximize the loss of the label classiﬁer M W .
We assume that the classiﬁcation loss is given by − y × M W ( x + δ ) .
The object of the adversary is to ﬁnd δ ∗ such that:
[Equation]
We will now calculate the optimal perturbation in the (cid:96) p balls ∀ p ∈ { 1 , 2 , ∞} .
For the following analyses, we restrict the perturbation region ∆ to the corresponding (cid:96) p ball of radius { (cid:15) 1 , (cid:15) 2 , (cid:15) ∞ } respectively.
We also note that the optimal perturbation exists at the boundary of the respective (cid:96) p balls.
Therefore, the constraint can be re-written as :
[Equation]
We use the following properties in the individual treatment of (cid:96) p balls:
[Equation]
p = 2 Making use of langrange multipliers to solve (12), we have:
[Equation]
Combining the results from (13) and replacing δ with δ 2 we obtain :
[Equation] [Equation]
p = ∞ Recall that the optimal perturbation is given by :
[Equation]
Since (cid:107) δ (cid:107) ∞
=
(cid:15) ∞ , we know that max i |
δ i | = (cid:15) ∞ .
Therefore (16) is maximized when each δ i = − y(cid:15) ∞
sgn W i ∀ i ∈ { 0 , . . .
, d } .
Further, since the weight matrix only contains non-negative elements ( α is a positive constant), we can conclude that the optimal perturbation is given by:
[Equation]
17) p =
We attempt an analytical solution for the optimal perturbation δ 1 .
Recall that the optimal perturbation is given by :
[Equation]
Since (cid:107) δ (cid:107) 1 = (cid:15) 1 , (18) is maximized when:
[Equation]
Combining the results From the preceding discussion, it may be noted that the new distribution of inputs within a given label changes by a different amount δ depending on the perturbation type.
Moreover, if the mean and variance of the distribution of a given label are known (which implies that the corresponding true data label is also known), the optimal perturbation is independent of the input itself, and only dependent on the respective class statistics (Note that the input is still important in order to understand the true class).
B. P ERTURBATION C LASSIFICATION BY C adv
In this section, we aim to verify if it is possible to accurately separate the optimal adversarial inputs crafted within different (cid:96) p balls.
For the purposes of this discussion, we only consider the problem of classifying perturbation types into (cid:96) 1 and (cid:96) ∞ , but the same analysis may also be extended more generally to any number of perturbation types.
We will consider the problem of classifying the correct attack label for inputs from true class y =for this discussion.
Note that the original distribution:
[Equation]
Since the perturbation value δ p is ﬁxed for all inputs corresponding to a particular label, the new distribution of perturbed inputs X 1 and X ∞ in case of (cid:96) 1 and (cid:96) ∞ attacks respectively (for y = 1) is given by:
[Equation]
We now try to evaluate the conditions under which we can separate the two Gaussian distributions with an acceptable worst-case error.
B. C ALCULATING A BOUND ON THE ERROR
Classiﬁcation Error A classiﬁcation error occurs if a data vector x belongs to one class but falls in the decision region of the other class.
That is in (7) the decision rule indicates the incorrect class.
(This can be understood through the existence of outliers)
[Equation]
Perturbation Size We set the radius of the (cid:96) ∞ ball, (cid:15) ∞ =
η and the radius of the (cid:96) 1 ball, (cid:15) 1 = α .
We further extend the discussion about suitable perturbation sizes in Appendix C.2.
These values ensure that the (cid:96) ∞ adversary can make all the weakly correlated labels meaningless by changing the expected value of the adversarial input to less than 0 ( E [ x i + δ ∞ ( i )] ∀ i > 0 ), while the (cid:96)adversary can make the strongly correlated feature x 0 meaningless by changing its expected value to less than 0 ( E [ x 0 + δ 1 (0)] ).
However, neither of the two adversaries can ﬂip all the features together.
Translating
the axes We can translate the axis of reference by (cid:0) − µ − (cid:0) δ 1 + δ ∞(cid:1)(cid:1) and deﬁne µ adv = (cid:0) δ 1 − δ ∞(cid:1) , such that :
[Equation]
We can once again combine this with the simpliﬁed Bayesian model in (9) to obtain the classiﬁcation rule given by:
[Equation] [Equation]
Combining the optimal perturbation deﬁnitions in (17) and (19) that µ adv = (cid:0) δ 1 − δ ∞(cid:1)
= 12 [ − (cid:15) 1 + (cid:15) ∞ , (cid:15) ∞ , . .
, (cid:15) ∞ ] .
We can further substitute (cid:15)
= α and (cid:15) ∞
=
η = α √ d .
Notice that µ adv ( i ) > 0 ∀ i > 0 .
Without loss of generality, to simplify further discussion we can ﬂip the coordinates of x 0 , since all dimensions are independent of each other.
Therefore, µ adv = α 2 √ d (cid:104) √
d − 1 , 1 , . .
, 1 (cid:105) .
Consider a new variable x
z such that:
[Equation]
(24) since each x i ∀ i ≥ 0 is independently distributed, the new feature x z ∼ N ( µ z , σ 2 z ) , where
[Equation]
Therefore, the problem simpliﬁes to calculating the probability that the meta-variable x z > 0 .
For ασ > 10 and d > 1 , we have in the z-table, z > 10 :
[Equation]
which suggests that the distributions are signiﬁcantly distinct and can be easily separated.
This concludes the proof for Theorem 1.
Note: We can extend the same analysis to other (cid:96) p balls as well, but we consider the case of (cid:96) 1 and (cid:96) ∞ for simplicity.
C R OBUSTNESS OF THE P ROTECTOR P IPELINE (T HEOREM 2)
In the previous section, we show that it is indeed possible to distinguish between the distribution of inputs of a given class that were subjected to (cid:96) 1 and (cid:96) ∞ perturbations over a standard classiﬁer.
Now, we aim to develop further understanding of the robustness of our two-stage pipeline in a dynamic attack setting with multiple labels to distinguish among.
The ﬁrst stage is a preliminary classiﬁer
C adv that classiﬁes the perturbation type and the second stage consists of multiple models M p that were speciﬁcally trained to be robust to perturbations to the input within the corresponding (cid:96) p norm.
First, in Appendix C.1, we calculate the optimal weights for a binary Gaussian classiﬁer M p , trained on dataset D to be robust to adversaries within the (cid:96) p ball ∀ p ∈ { 1 , ∞} .
Based on the weights of the individual model, we ﬁx the perturbation size (cid:15) p to be only as large, as is required to fool the alternate model with high probability.
Here, by ‘alternate’ we mean that for an (cid:96) q attack, the prediction should be made by the M p,(cid:15) p model,where p, q ∈ { 1 , ∞} ; p (cid:54) = q .
In Appendix C.3 we calculate the robustness of individual M p models to (cid:96) p adversaries, given the perturbation size (cid:15) p as deﬁned in Appendix C.2.
In Appendix C.4, we analyze the modiﬁed distributions of the perturbed inputs after different (cid:96) p attacks.
Based on this analysis, we construct a simple decision rule for the perturbation classiﬁer C adv .
Finally, in Appendix C.5 we determine the perturbation induced by the worst-case adversary that has complete knowledge of both C adv and M p,(cid:15) p ∀ p ∈ { 1 , ∞} .
We show how there exists a trade-off between fooling the perturbation classiﬁer (to allow the alternate M p,(cid:15) p model to make the ﬁnal prediction), and fooling the alternate M p,(cid:15) p model itself.
Perturbation Size We set the radius of the (cid:96) ∞ ball, (cid:15) ∞ =
η + ζ ∞ and the radius of the (cid:96) 1 ball, (cid:15)
= α + ζ 1 , where ζ p are some small positive constants that we calculate in Appendix C.2.
These values ensure that the (cid:96) ∞ adversary can make all the weakly correlated labels meaningless by changing the expected value of the adversarial input to less than 0 ( E [ x i + δ ∞ ( i )] ∀ i > 0 ), while the (cid:96) 1 adversary can make the strongly correlated feature x 0 meaningless by changing its expected value to less than 0 ( E [ x 0 + δ 1 (0)] ).
However, neither of the two adversaries can ﬂip all the features together.
The exact values of ζ p determine the exact success probability of the attacks.
We defer this calculation to later when we have calculated the weights of the models M p .
For the following discussion, it may be assumed that ζ p → 0 ∀ p ∈ { 1 , ∞} .
C. B INARY G AUSSIAN C LASSIFIER M p
Extending the discussion in Appendix B.
[Equation]
we now examine the learned weights of a binary Gaussian classiﬁer M p that is trained to be robust against perturbations within the corresponding (cid:96) p ball of radius (cid:15) p .
The optimization equation for the classiﬁer can be formulated as follows:
[Equation]
where λ is tuned in order to make the (cid:96) 2 norm of the optimal weight distribution
[Equation]
|| W ∗ || 2 , = 1 .
Following the symmetry argument in Lemma D
[Equation]
1 Tsipras et al.
we extend for the binary
Gaussian classiﬁer that :
[Equation]
We deal with the cases pertaining to p ∈ {∞ , 1 } in this section.
For both the cases
[Equation]
we consider existential solutions for the classiﬁer M p to simplify the discussion.
This gives us lower bounds on the performance of the optimal robust classiﬁer.
The robust objective under adversarial training can be deﬁned as:
[Equation]
[Equation]
Further, since the λ constraint only ensures that || W ∗ ||
= 1 , we can simplify the optimization equation by substituting W 0 = (cid:112) 1 − d · W M 2 as follows,
[Equation] [Equation]
p = ∞
As discussed in (17) the optimal perturbation δ ∞ is given by − y(cid:15) ∞ 1 .
The optimization equation is simpliﬁed to:
[Equation]
Recall that (cid:15) ∞ =
α √ d + ζ ∞ .
To simplify the following discussion we use the weights of a classiﬁer trained to be robust against perturbations within the (cid:96) ∞ ball of radius (cid:15) ∞ =
α √ d .
The optimal solution is then given by:
[Equation]
Therefore, the classiﬁer weights are given by W =
[ W 0 , W 1 , . .
. , W d ] = [1 , 0 , . .
We also show later in Appendix C.3 that the model achieves greater than 99% accuracy against (cid:96) ∞ adversaries for the chosen values of ζ ∞ .
p =
We consider an analytical solution to yield optimal weights for this case.
Recall from (19) that the optimal perturbation δ 1 depends on the weight distribution of the classiﬁer.
Therefore, if
W 0 > W M the optimization equation can be simpliﬁed to
[Equation]
and if W M > W
[Equation]
W
[Equation]
Recall that (cid:15)
=
α + ζ 1 .
Once again to simplify the discussion that follows we will lower bound the robust accuracy of the classiﬁer M 1 by considering the optimal solution when zeta 1 = 0 .
The optimal solution is then given by:
[Equation]
For the robust classiﬁer M 1 , the weights W =
[ W 0 , W 1 , . .
. , W d ] = [0 , 1 √ d , 1 √ d , . .
. , 1 √ d ] .
While this may not be the optimal solution for all values of ζ 1 , we are only interested in a lower bound on the ﬁnal accuracy and the classiﬁer described by weights W simpliﬁes the discussion hereon.
We also show later in Appendix C.3 that the model achieves greater than 99% accuracy against (cid:96)adversaries for the chosen values of ζ 1 .
C. P ERTURBATION S IZES FOR F OOLING M p M ODELS
Now that we exactly know the weights of the learned robust classiﬁers M 1 and M ∞ , we can move towards calculating values ζ 1 and ζ ∞ for the exact radius of the perturbation regions for the (cid:96) 1 and (cid:96) ∞ metrics.
We set the radii of these regions in such a way that an (cid:96) 1 adversary can fool the model
M ∞ with probability ∼ 98% (corresponding to z = 2 in the z-table for normal distributions), and similarly, the success of (cid:96) ∞ attacks against the M 1 model is ∼ 98%.
Let P p 1 ,p 2 represent the probability that model M p 1 correctly classiﬁes an adversarial input in the (cid:96) p 2 region.
For p 1 = ∞ and p 2 = 1 ,
[Equation]
To simplify the discussion for the M 1 model
[Equation]
we deﬁne a meta-feature x M as:
[Equation]
which is distributed as :
[Equation]
For p 1 = 1 and p 2 = ∞ ,
[Equation]
C. R OBUSTNESS OF INDIVIDUAL M p MODELS
Additional assumptions We add the following assumptions: (1) the dimensionality parameter d of input data is larger than 100; and (2) the ratio of the mean and variance for feature x 0 is greater than 10.
[Equation]
We deﬁne P p as the probability that for any given input x ∼ N ( y µ , Σ ) , the classiﬁer M p outputs the correct label y for the input x + δ p .
[Equation]
using the assumptions in (38),
[Equation]
using the assumptions in (38),
[Equation]
C. D ECISION RULE FOR C adv
We aim to provide a lower bound on the worst-case accuracy of the entire pipeline, through the existence of a simple decision tree C adv .
For given perturbation budgets (cid:15) 1 and (cid:15) ∞ , we aim to understand the range of values that can be taken by the adversarial input.
Consider the scenarios described in Table 4 below:
[Table]
Note that any adversary that moves the perturbation away from the y-axis is uninteresting for our comparison, since irrespective of a correct perturbation type prediction by C adv , either of the two second level models naturally obtain a high accuracy on such inputs.
Hence, we deﬁne the following decision rule with all the remaining cases mapped to (cid:96) 1 perturbation type.
[Equation]
[Table]
[Equation]
x
[Equation]
=
[Equation]
, if
[Equation]
0 , otherwise
[Equation] [Equation]
where the output 1 corresponds to the classiﬁer predicting the presence of (cid:96) ∞ perturbation in the input, while an output of 0 suggests that the classiﬁer predicts the input to contain perturbations of the (cid:96) 1 type.
If we consider a black-box setting where the adversary has no knowledge of the classiﬁer C adv , and can only attack M p it is easy to see that the proposed pipeline obtains a high adversarial accuracy against the union of (cid:96) 1 and (cid:96) ∞ perturbations:
Note: (1) There exists a single model that can also achieve robustness against the union of (cid:96) 1 and (cid:96) ∞ perturbations, however, learning this model may be more challenging in real data settings.
The classiﬁer need not be perfect.
C. T RADE - OFF BETWEEN ATTACKING M p AND C adv
To obtain true robustness it is important that the entire pipeline is robust against adversarial attacks.
More speciﬁcally, in this section we demonstrate the natural tension that exists between fooling the top level attack classiﬁer (by making an adversarial attack less representative of its natural distribution) and fooling the bottom level adversarially robust models (requiring stronger attacks leading to a return to the attack’s natural distribution).
The accuracy of the pipelined model f against any input-label pair ( x, y ) sampled through some distribution N ( y µ adv , Σ ) (where µ adv incorporates the change in the input distribution owing to the adversarial perturbation) is given by:
[Equation]
(cid:96) ∞ adversary: To simplify the analysis, we consider loose lower bounds on the accuracy of the model f against the (cid:96) ∞ adversary.
Recall that the decision of the attack classiﬁer is only dependent of the input x 0 .
Irrespective of the input features x i ∀ i > 0 , it is always beneﬁcial for the adversary to perturb the input by µ i = − (cid:15) ∞ .
However, the same does not apply for the input x 0 .
Analyzing for the scenario when the true label y = 1 , if the input x 0 lies between α 2 − (cid:15) ∞ of the mean α , irrespective of the perturbation, the output of the attack classiﬁer C adv = 1 .
The M ∞ model then always correctly classiﬁes these inputs.
The overall robustness of the pipeline requires analysis for the case when input lies outside α 2 − (cid:15) ∞ of the mean as well.
However, we consider that the adversary always succeeds in such a case in order to only obtain a loose lower bound on the robust accuracy of the pipeline model f against (cid:96) ∞ attacks.
[Equation]
using the assumptions in (38),
[Equation]
(cid:96) 1 adversary:
It may be noted that a trivial way for the (cid:96) 1 adversary to fool the attack classiﬁer is to return a perturbation δ 1 = 0 .
In such a scenario, the classiﬁer predicts that the adversarial image was subjected to an (cid:96) ∞ attack.
The label prediction is hence made by the M ∞ model.
But we know from (40) that the M ∞ model predicts benign inputs correctly with a probability P ∞ , ∞ > 0 .
99 , hence defeating the adversarial objective of misclassiﬁcation.
To achieve misclassiﬁcation over the entire pipeline the optimal perturbation decision for the (cid:96) 1 adversary when x 0 ∈ (cid:2) − α − α 2 − (cid:15) 1 , − α + α 2 + (cid:15) 1 (cid:3) the adversary can fool the pipeline by ensuring that
the
C adv ( x ) = 1 .
However, in all the other cases irrespective of the perturbation, either C adv = 0 or the input features x 0 has the same sign as the label y .
Since, P 1 , 1 > 0 .
99 for the M 1 model, for all the remaining inputs x 0 the model correctly predicts the label with probability greater than 0 .(approximate lower bound).
We formulate this trade-off to elaborate upon the robustness of the proposed pipeline.
[Equation]
using the assumptions in (38),
[Equation]
This concludes the proof for Theorem 2, showing that an adversary can hardly stage successful attacks on the entire pipeline and faces a natural tension between attacking the label predictor and the attack classiﬁer.
Finally, we emphasize that the shown accuracies are lower bounds on the actual robust accuracy, and the objective of this analysis is not to ﬁnd the optimal solution to the problem of multiple perturbation adversarial training, but to expose the existence of the trade-off between attacking the two stages of the pipeline.
D M ODEL A RCHITECTURE
Second-level M p models.
A key advantage of our P ROTECTOR design is that we can build upon existing defenses against individual perturbation type.
Speciﬁcally, for MNIST, we use the same
CNN architecture as Zhang et al.
for our M p models, and we train these models using their proposed TRADES loss.
For CIFAR-10, we use the same training setup and model architecture as
Carmon et al.
(2019), which is based on a robust self-training algorithm that utilizes unlabeled data to improve the model robustness.
Perturbation classiﬁer C adv .
For both MNIST and CIFAR-10 datasets, the architecture of the perturbation classiﬁer C adv is similar to the individual M p models.
Speciﬁcally, for MNIST, we use the CNN architecture in Zhang et al.
with four convolutional layers, followed by two fully-connected layers.
For CIFAR-10, C adv is a WideResNet (Zagoruyko & Komodakis, 2016) model with depth 28 and widening factor of 10 (WRN-28-10).
E T RAINING D ETAILS E. S PECIALIZED R OBUST P REDICTORS M p
MNIST.
We use the Adam optimizer (Kingma & Ba, 2015) to train our models along with a piece-wise linearly varying learning rate schedule (Smith, 2018) to train our models with maximum learning rate of 10 − 3 .
The base models M 1 , M 2 , M ∞ are trained using the TRADES algorithm for 20 iterations, and step sizes α 1 = 2.0, α 2 = 0.3, and α ∞ = 0.05 for the (cid:96) 1 , (cid:96) 2 , (cid:96) ∞ attack types within perturbation radii (cid:15) 1 = 10.0, (cid:15) 2 = 2.0, and (cid:15) ∞
= 0.3 respectively.
CIFAR10.
The individual M p models are trained to be robust against { (cid:96) ∞ , (cid:96) 1 , (cid:96) 2 } perturbations of { (cid:15) ∞ , (cid:15) 1 , (cid:15) 2 } = { 0 . 003 , 12 . 0 , 0 . 05 } respectively.
For CIFAR10, the attack step sizes { α ∞ , α 1 , α 2 } = { 0 . 005 , 2 . 0 , 0 . 1 } respectively.
The training of the individual M p models is directly based on the work of Carmon et al.
E. P ERTURBATION CLASSIFIER C adv
MNIST.
We use a learning rate of 0.01 and Adam optimizer for 10 epochs, with linear rate decay to 0.001 between the fourth epoch and the tenth epoch.
The batch size is set to 100 for all experiments.
CIFAR10.
We use a learning rate of 0.01 and SGD optimizer for 5 epochs, with linear rate decay to 0.001 between the fourth epoch and the tenth epoch.
The batch size is set to 100 for all experiments.
Creating the Adversarial Perturbation Dataset.
We create a static dataset of adversarially perturbed images and their corresponding attack label for training the perturbation classiﬁer
C adv .
For generating adversarial images, we perform weak adversarial attacks that are faster to compute.
In particular, we perform 10 iterations of the PGD attack.
For MNIST, the attack step sizes { α ∞ , α 1 , α 2 } = { 0 . 05 , 2 . 0 , 0 . 3 } respectively.
For CIFAR10, the attack step sizes { α ∞ , α 1 , α 2 } = { 0 . 005 , 2 . 0 , 0 . 1 } respectively.
Note that we perform the Sparse- (cid:96) 1 or the top-k PGD attack for the (cid:96) 1 perturbation ball, as introduced by Tram`er & Boneh (2019).
We set the value of k to 10, that is we move by a step size α 1 k in each of the top 10 directions with respect to the magnitude of the gradient.
F A TTACKS U SED FOR E VALUATION
A description of all the attacks used for evaluation of the models is presented here.
From the Foolbox library(Rauber et al., 2017), apart from (cid:96) 1 , (cid:96) 2 and (cid:96) ∞
PGD adversaries, we also evaluate the following attacks for different perturbation types.
(1) For (cid:96) 1 perturbations, we include the Salt & Pepper Attack (SAPA)
(Rauber et al., 2017) and Pointwise Attack (PA) (Schott et al., 2018)
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] (2) For (cid:96) 2 perturbations, we include the Gaussian noise attack (Rauber et al
[Table]
, 2017), Boundary Attack (Brendel et al., 2018), DeepFool (Moosavi-Dezfooli et al., 2016), DDN attack (Rony et al
[Table]
, 2019), and C&W attack (Carlini & Wagner, 2017b).
(3) For (cid:96) ∞ perturbations, we include FGSM attack (Goodfellow et al., 2015) and the Momentum
Iterative Method (Dong et al., 2018).
From the AutoAttack library from Croce & Hein (2020c)
[Table]
we make use of all the three variants of the Adaptive PGD attack (APGD-CE, APGD-DLR, APGD-T) (Croce & Hein, 2020c) along with the targeted and standard version of Fast Adaptive Boundary Attack (FAB, FAB-T) (Croce & Hein, 2020b) and the Square Attack (Andriushchenko et al., 2020).
We utilize the AA + version for strong attacks.
Attack Hyperparameters For the attacks in the Foolbox and AutoAtack libraries we use the default parameter setting in the strongest available mode (such as AA + ).
For the custom PGD attacks, we evaluate the models with 10 restarts and 200 iterations of the PGD attack.
The step size of the { (cid:96) ∞ , (cid:96) 1 , (cid:96) 2 } PGD attacks are set as follows: For MNIST, the attack step sizes { α ∞ , α 1 , α 2 } = { 0 . 01 , 1 . 0 , 0 . 1 } respectively.
For CIFAR10, the attack step sizes { α ∞ , α 1 , α 2 } = { 0 . 003 , 1 . 0 , 0 . 02 } respectively.
Further, similar to Tram`er & Boneh (2019); Maini et al.
we evaluate our models on the ﬁrst 1000 images of the test set of MNIST and CIFAR-10, since many of the attacks employed are extremely computationally expensive and slow to run.
Speciﬁcally, on a single GPU, the entire evaluation for a single model against all the attacks discussed with multiple restarts will take nearlymonth, and is not feasible.
G E MPIRICAL P ERTURBATION O VERLAP
Following Section 6.2, we also present results on the perturbation overlap when we attack P ROTECTOR with PGD attacks.
To contrast the results with that of attacking a vanilla model, we also present the table in the main paper for convenience.
It is noteworthy that the presence of a perturbation classiﬁer forces the adversaries to generate such attacks that increase the norm of the perturbations in alternate (cid:96)
q region.
Secondly, we also observe that in the case of CIFAR10, the (cid:96) 2 PGD attack has a large overlap with the (cid:96) 1 norm of radius 10.
However, recall that in case of (cid:96) 2 attacks for CIFAR10, both the base models M 1 and M ∞ were satisfactorily robust.
Hence, the attacker has no incentive to reduce the perturbation radius for an (cid:96) q norm since the perturbation classiﬁer only performs a binary classiﬁcation between (cid:96) 1 and (cid:96) ∞ attacks.
The results can be observed in Tables 5 and 6.
[Table]
H B REAKDOWN OF C OMPLETE E VALUATION
In this section [Table] we present the results of the perturbation type classiﬁer C adv against transfer adversaries.
We also present the breakdown results of the adversarial robustness of baseline approaches and our P ROTECTOR pipeline against all the attacks that we tried [Table] and also report the worst case performance against the union of all attacks.
H. R OBUSTNESS OF C adv
The results for the robustness of the perturbation classiﬁer C adv in the presence of adaptive adversaries is presented in Table 7.
Note that C adv transfers well across the board, even if the adversarial examples are generated against new models that are unseen for C adv during training, achieving extremely high test accuracy.
Further, even if the adversarial attack was generated by a different algorithm such as from the AutoAttack library, the transfer success of C adv still holds up.
In particular, the obtained accuracy is > 95% across all the individual test sets created.
The attack classiﬁcation accuracy is in general highest against those generated by attacking M 1 or M ∞ for CIFAR10, and M 2 or M ∞ for
MNIST.
This is an expected consequence of the nature of generation of the static dataset for training the perturbation classiﬁer C adv as described in Section 5.1.
H. MNIST
In Table 8, we provide a breakdown of the adversarial accuracy of all the baselines, individual
M p models and the P ROTECTOR method, with both the adaptive and standard attack variants on the MNIST dataset.
P ROTECTOR outperforms prior baselines by 6 . 9% on the MNIST dataset.
It is important to note that P ROTECTOR shows signiﬁcant improvements against most attacks in the suite.
Compared to the previous state-of-the-art defense against multiple perturbation types (MSD), if we compare the performance gain on each individual attack algorithm, the improvement is also signiﬁcant, with an average accuracy increase of 15 .
5% on MNIST dataset.
These results demonstrate that P ROTECTOR considerably mitigates the trade-off in accuracy against individual attack types.
H. CIFAR-
In Table 9, we provide a breakdown of the adversarial accuracy of all the baselines, individual
M p models and the P ROTECTOR method, with both the adaptive and standard attack variants on the CIFAR10 dataset.
P ROTECTOR outperforms prior baselines by 8 . 9% .
Once again, note that
P ROTECTOR shows signiﬁcant improvements against most attacks in the suite.
Compared to the previous state-of-the-art defense against multiple perturbation types (MSD), if we compare the performance gain on each individual attack algorithm, the improvement is signiﬁcant, with an average accuracy increase of 14 . 2% on.
These results demonstrate that P ROTECTOR considerably mitigates the trade-off in accuracy against individual attack types.
Further, P ROTECTOR also retains a higher accuracy on benign images, as opposed to past defenses that have to sacriﬁce the benign accuracy for the robustness on multiple perturbation types.
In particular, the clean accuracy of P ROTECTOR is over 6% higher than such existing defenses on
CIFAR-10, and the accuracy is close to M p models trained for a single perturbation type. [Table] [Table]
[Table]
H. A GGREGATING PREDICTIONS FROM DIFFERENT M p AT I NFERENCE
In all our experiments in this work the adversary constructs adversarial examples using the softmax based adaptive strategy for aggregating predictions from different M p models, as described in
Equation 4 for the column ‘Ours’ and using the ‘max’ strategy (Equation 1) for results described in the column ‘Ours*’
However, for consistency of our defense strategy irrespective of the attacker’s strategy, the defender only utilizes predictions from the specialized model M p corresponding to the most-likely attack (Equation 1) to provide the ﬁnal prediction (only forward propagation) for generated adversarial examples.
In our evaluation
[Table]
we found a negligible impact of changing this aggregation to the ‘softmax’ strategy for aggregating the predictions.
For example
[Table]
we show representative results in case of the APGD ( (cid:96) ∞ , (cid:96) 2 ) attacks on the CIFAR10 dataset in Table
[Table]