I MPROVING A DVERSARIAL R OBUSTNESS R EQUIRES R EVISITING M ISCLASSIFIED E XAMPLES
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep neural networks (DNNs) are vulnerable to adversarial examples crafted by imperceptible perturbations.
A range of defense techniques have been proposed to improve DNN robustness to adversarial examples, among which adversarial training has been demonstrated to be the most effective.
Adversarial training is often formulated as a min-max optimization problem, with the inner maximization for generating adversarial examples.
However, there exists a simple, yet easily overlooked fact that adversarial examples are only deﬁned on correctly classiﬁed (natural) examples, but inevitably, some (natural) examples will be misclassiﬁed during training.
In this paper, we investigate the distinctive inﬂuence of misclassiﬁed and correctly classiﬁed examples on the ﬁnal robustness of adversarial training.
Speciﬁcally, we ﬁnd that misclassiﬁed examples indeed have a signiﬁcant impact on the ﬁnal robustness.
More surprisingly, we ﬁnd that different maximization techniques on misclassiﬁed examples may have a negligible inﬂuence on the ﬁnal robustness, while different minimization techniques are crucial.
Motivated by the above discovery, we propose a new defense algorithm called Misclassiﬁcation
Aware adveRsarial Training (MART), which explicitly differentiates the misclassiﬁed and correctly classiﬁed examples during the training.
We also propose a semi-supervised extension of MART, which can leverage the unlabeled data to further improve the robustness.
Experimental results show that MART and its variant could signiﬁcantly improve the state-of-the-art adversarial robustness.
I NTRODUCTION
Despite their great success in applications such as computer vision (He et al., 2016), speech recognition (Wang et al., 2017) and natural language processing (Devlin et al., 2018) deep neural networks (DNNs) are extremely vulnerable to adversarial examples crafted by adding small adversarial perturbations to natural examples (Szegedy et al., 2013; Goodfellow et al., 2015).
Given a DNN classiﬁer h
θ with parameter θ and a correctly classiﬁed natural example x with class label y
( h θ ( x ) =
y ), an adversarial example x (cid:48) can be generated by perturbing x such that h θ ( x (cid:48) )
(cid:54) =
y , i.e. , the natural example is correctly classiﬁed before perturbation but misclassiﬁed after perturbation.
The perturbation required for misclassiﬁcation is often small and bounded by an L p -norm (cid:107) x (cid:48) − x (cid:107) p ≤ (cid:15) , which keeps x (cid:48) within the (cid:15) -ball centered at x , so that it is visually the “same” for human observers.
This vulnerability of DNNs raises serious security concerns about their practicability in security critical applications, e.g. , face recognition (Kurakin et al., 2016) and autonomous driving (Chen et al., 2015).
Compared with pre/post-processing methods such as feature squeezing (Xu et al., 2017), input denoising (Guo et al., 2018; Liao et al., 2018; Samangouei et al., 2018) and adversarial detection (Feinman et al., 2017; Grosse et al., 2017; Meng & Chen, 2017; Ma et al., 2018), several defense techniques have been proposed to train DNNs that are inherently robust to adversarial examples including defensive distillation (Papernot et al., 2016), gradient regularization (Gu & Rigazio, 2014;
Papernot et al., 2017; Ross & Doshi-Velez, 2018; Tramèr et al., 2018), model compression (Das et al., 2018;
Liu et al., 2018) and activation pruning (Dhillon et al., 2018; Rakin et al., 2018), among which adversarial training has been demonstrated to be the most effective (Goodfellow et al., 2015;
Athalye et al., 2018; Madry et al., 2018).
Adversarial training can be regarded as a data augmentation technique that trains DNNs on adversarial examples, and can be viewed as solving the following
[Figure]
min-max optimization problem:
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
where n is the number of training examples and (cid:96) ( · ) is the classiﬁcation loss, such as the commonly used cross-entropy (CE) loss.
The inner maximization generates adversarial examples that can be used by the outer minimization to train robust DNNs.
Recently, adversarial training with adversarial examples generated by Projected Gradient Descent (PGD) (Madry et al., 2018) has been demonstrated to be the only method that can train moderately robust DNNs without being fully attacked (Athalye et al., 2018).
However, there is still a signiﬁcant gap between adversarial robustness (test accuracy on adversarial examples) and natural accuracy (test accuracy on natural examples), even for simple image datasets like CIFAR-10 (Krizhevsky & Hinton, 2009).
Compared with natural training (on natural examples), training adversarially robust DNNs is particularly difﬁcult (Madry et al., 2018).
Nakkiran (2019) showed that a model requires more capacity to be robust ( i.e. , simple models can have high natural accuracy but are less likely to be robust).
In addition, the sample complexity of adversarial training can be signiﬁcantly higher than that of natural training, that is, training robust DNNs tends to require more data either labeled (Schmidt et al., 2018) or unlabeled ones (Uesato et al., 2019;
Carmon et al., 2019; Najaﬁ et al., 2019; Zhai et al., 2019).
Moreover, Tsipras et al.
; Zhang et al.
(2019) demonstrated that adversarial robustness may be inherently at odds with natural accuracy.
Parallel to these studies, in this paper, we provide some new insights on the adversarial examples used for adversarial training.
Recall that the formal deﬁnition of an adversarial example is conditioned on it being correctly classiﬁed
(Carlini et al., 2019).
From this perspective, adversarial examples generated from misclassiﬁed examples are “undeﬁned”.
Most adversarial training variants neglect this distinction, where all training examples are treated equally in both the maximization and the minimization processes, regardless of whether or not they are correctly classiﬁed.
The only exception we are aware of is Ding et al.
(2018), which proposes to use maximal margin optimization for correctly classiﬁed examples.
Yet they did not pay sufﬁcient attention to misclassiﬁed examples.
A deeper understanding about the inﬂuence of misclassiﬁed and correctly classiﬁed examples on the robustness is still missing in the literature.
Therefore, we raise the following questions:
Are the adversarial examples generated from i) misclassiﬁed and ii) correctly classiﬁed examples, equally important for adversarial robustness?
If not , how can one make better use of the difference to improve robustness?
In this paper, we investigate this intriguing, yet thus far overlooked aspect of adversarial training, and ﬁnd that misclassiﬁed and correctly classiﬁed examples exhibit a distinctive inﬂuence on the ﬁnal robustness.
To illustrate this phenomenon, we conduct a proof-of-concept experiment on CIFAR10 in a white-box setting with L ∞ maximum perturbation (cid:15)
= 8 / 255 .
We ﬁrst train an 8-layer
Convolutional Neural Network (CNN) using standard adversarial training with 10 -step PGD (PGD 10 ) and step size (cid:15)/ 4 , then use this network ( 87% training accuracy) to select two subsets of natural training examples to investigate: 1) a subset of misclassiﬁed examples S − ( 13% of training data), and 2) a subset of correctly classiﬁed examples S
+ (also 13% of training data, | S + | = | S − |).
Using these two subsets, we explore different ways to re-train the same network, and evaluate its robustness against white-box PGD 20 (step size (cid:15)/ 10 ) attacks on the test dataset.
In Figure 1(a), we ﬁnd that misclassiﬁed examples have a signiﬁcant impact on the ﬁnal robustness.
Compared with standard adversarial training (dashed blue line), the ﬁnal robustness drops drastically, if examples in subset S − are not perturbed (solid green line) during adversarial training (other examples are still perturbed by PGD 10 ).
In contrast, the same operation on subset S + only slightly affects the ﬁnal robustness (solid orange line).
Previous work has found that removing a small proportion of training examples does not reduce the robustness (Ding et al., 2019), which seems to be true for correctly classiﬁed examples, but is apparently not true for misclassiﬁed examples.
To further understand the distinctive inﬂuence of misclassiﬁed and correctly classiﬁed examples, we test different techniques on them within either the maximization or the minimization process of adversarial training.
Firstly, we apply different maximization techniques while keeping the minimization loss CE unchanged.
As shown in Figure 1(b), the ﬁnal robustness is barely affected when we use a weak attack ( e.g. , Fast Gradient Sign Method (FGSM)
(Goodfellow et al., 2015)) to perturb misclassiﬁed examples S − (all other training examples are still perturbed by PGD 10 ).
This suggests that different maximization techniques on misclassiﬁed examples S − may have a negligible inﬂuence on the ﬁnal robustness, provided that the inner maximization problem is solved to a moderate precision (Wang et al., 2019).
However, for subset S + , a weak attack for the maximization tends to degenerate the robustness.
Secondly, we test different minimization techniques with the inner maximization still solved by PGD 10 .
Interestingly, we ﬁnd that different minimization techniques on misclassiﬁed examples make a signiﬁcant difference to the ﬁnal robustness.
As shown in Figure 1(c), compared with standard adversarial training (dashed blue line) with the CE loss, the ﬁnal robustness is signiﬁcantly improved when the outer minimization on misclassiﬁed examples is “regularized” (solid green line) by an additional term (a KL-divergence term that was used previously in Zheng et al.
; Zhang et al.
The same regularization applied to correctly classiﬁed examples also helps the ﬁnal robustness (solid orange line), though not as signiﬁcantly as for misclassiﬁed examples.
Motivated by the above observations, we reformulate the adversarial risk to incorporate an explicit differentiation of misclassiﬁed examples in a form of regularization.
We then propose a new defense algorithm to achieve this in a dynamic way during adversarial training.
Our main contributions are:
• We investigate the distinctive inﬂuence of misclassiﬁed and correctly classiﬁed examples on the ﬁnal robustness of adversarial training.
We ﬁnd that the manipulation on misclassiﬁed examples has more impact on the ﬁnal robustness,
and the
minimization techniques are more crucial than maximization ones under the min-max optimization framework.
• We propose a regularized adversarial risk which incorporates an explicit differentiation of misclassiﬁed examples as a regularizer.
Based on that, we further propose a new defense algorithm, called Misclassiﬁcation Aware adveRsarial Training (MART).
• Experimentally, we show that adversarial robustness can be signiﬁcantly improved over the stateof-the-art, by a speciﬁc focus on misclassiﬁed examples.
It also helps improve recently proposed adversarial
training with unlabeled data.
M ISCLASSIFICATION A WARE A DVERSARIAL R ISK
In this section, we propose a regularized adversarial risk that incorporates an explicit differentiation of misclassiﬁed examples.
2. P RELIMINARIES
We ﬁrst deﬁne some notations.
We use lower case and lower case bold face to denote scalars and vectors, respectively.
We use upper case calligraphic symbols to denote sets.
For a K -class ( K ≥ 2 ) classiﬁcation problem, given a dataset { ( x i , y i ) } i =1 ,...,n with x i ∈ R d as a natural example and y i ∈ { 1 , . . .
, K } as its associated label, a DNN classiﬁer h
θ with model parameter θ predicts the class of an input example x i :
[Equation]
where z k ( x i , θ ) is the logits output of the network with respect to class k , and p k ( x i , θ ) is the probability (softmax on logits) of x i belonging to class k .
The adversarial risk (Madry et al., 2018) on dataset { ( x i , y i ) } i =1 ,...,n and classiﬁer h θ can be deﬁned with respect to the 0-1 loss (Zhang et al., 2019) as:
[Equation]
where 1 ( · ) is the indicator function and B (cid:15) ( x i )
= { x : (cid:107) x − x i (cid:107) p ≤ (cid:15) } denotes the L p -norm ball centered at x i with radius (cid:15) .
We will focus on the L ∞ -ball in this paper.
2. M ISCLASSIFICATION A WARE R EGULARIZATION
Note that the adversarial risk in (3) is deﬁned on adversarial examples within the (cid:15) -ball of all natural examples, regardless of whether they are correctly classiﬁed ( h θ ( x i ) =
y i ) or misclassiﬁed (
h θ ( x i ) (cid:54) =
y i ) by the current model
h θ .
To differentiate, we reformulate the adversarial risk based on the prediction of the current network h θ .
Speciﬁcally, natural training examples can be divided into two subsets with respect to h θ , with one subset of correctly classiﬁed examples ( S + h θ ) and one subset of misclassiﬁed examples ( S − h θ ):
[Equation]
and
[Equation]
Then we are going to deﬁne adversarial risk separately for correctly classiﬁed and misclassiﬁed examples.
As we observed in Figure 1(c), regularization on misclassiﬁed examples can signiﬁcantly improve robustness.
Therefore, for misclassiﬁed examples, we formulate the adversarial risk as:
[Equation]
where the adversarial example ˆ x (cid:48)
i is generated by solving
[Equation]
We remark that the ﬁrst and second terms on the R.H.S. of (4) correspond to the standard adversarial risk and the regularization term respectively.
Moreover, we would like to clarify that the regularization term 1 ( h θ ( x i ) (cid:54) =
h θ (ˆ x (cid:48) i )) aims to encourage the output of neural network to be stable against misclassiﬁed adversarial examples.
For misclassiﬁed examples, direct minimization of the standard adversarial risk may be too hard, as themselves cannot be classiﬁed correctly, even without any perturbations.
A similar idea has been used in stability training on all training examples (Zheng et al., 2016; Kannan et al., 2018; Zhang et al., 2019).
Then we consider correctly classiﬁed examples.
As can be observed in Figure 1(c), regularization on correctly classiﬁed examples cannot provide as signiﬁcant improvement as achieved by that on misclassiﬁed ones.
Moreover, in this case it can be found that
[Equation]
since we have h θ ( x i )
=
y i , which implies that the regularizer has exactly same form as the adversarial risk.
Therefore, for correctly classiﬁed example, we simply use the standard adversarial risk, i.e. ,
[Equation]
where the adversarial example ˆ x (cid:48)
i is deﬁned in (5).
Finally, combining the proposed two adversarial risks for correctly classiﬁed examples and misclassiﬁed examples in an adversarial training framework, we can train a network that minimizes the following risk:
[Equation] [Equation]
where ˆ x (cid:48)
i is deﬁned in (5) and the second equality follows from the deﬁnition of S h + θ and S
h − θ .
The new risk deﬁned above is a regularized adversarial risk with regularization term 1 /n (cid:80) ni
=1 1 ( h θ ( x i ) (cid:54) =
h θ (ˆ x (cid:48) i ))
· 1 ( h θ ( x i ) (cid:54) =
y i ) , which we call the misclassiﬁcation aware regularization .
P ROPOSED D EFENSE : M ISCLASSIFICATION A WARE A DVERSARIAL T RAINING (MART)
In the previous section, we derived the misclassiﬁcation aware adversarial risk based on 0-1 loss.
However, optimization over 0-1 loss is intractable in practice.
We next propose a Misclassiﬁcation
Aware adveRsarial Training (MART) algorithm, by replacing the 0-1 losses with proper surrogate loss functions which are both physical meaningful and computationally tractable.
Following that, we further analyze the difference of MART to existing work, and propose a semi-supervised extension.
3. T HE P ROPOSED D EFENSE A LGORITHM
Surrogate Loss for Outer Minimization.
As presented in (7), the minimization consists of three indicator functions: (1) 1 ( h θ (ˆ x (cid:48) i ) (cid:54) =
y i ) ; (2)
( h θ ( x i ) (cid:54) =
h θ (ˆ x (cid:48) i )) ; and (3) 1 ( h θ ( x i ) (cid:54) =
y i ) .
For the ﬁrst indicator function 1 ( h θ (ˆ x (cid:48) i ) (cid:54) =
y i ) , we propose to use a boosted cross entropy (BCE) loss as the surrogate loss, instead of the commonly used CE loss in (Madry et al., 2018;
Wang et al., 2019).
This is largely because classifying adversarial examples requires a stronger classiﬁer than natural examples, as the presence of adversarial examples makes the classiﬁcation decision boundary become more complicated.
This is pointed out by (Madry et al., 2018), where they increase the model capacity for a stronger classiﬁer.
The beneﬁt of using BCE compared to CE will shortly be presented in the experiment section.
The proposed BCE loss is deﬁned as:
[Equation]
where p k (ˆ x (cid:48)
i , θ ) is the probability output deﬁned in (2), the ﬁrst term − log (cid:0) p y i
(ˆ x (cid:48)
i , θ ) (cid:1) is the commonly used CE loss, denoted CE ( p (ˆ x (cid:48)
i , θ ) , y i ) , and the second term − log (cid:0) 1 − max k (cid:54) =
y i p
k
(ˆ x (cid:48)
i , θ ) (cid:1) is a margin term used to improve the decision margin of the classiﬁer.
A similar idea has been used for improving adversarial strength by Carlini & Wagner (2017).
Note that BCE is just a simple boost that works well in our experiments and other boosted losses could also work here.
For the second indicator function 1 ( h θ ( x i ) (cid:54) =
h θ (ˆ x (cid:48) i )) , we can use KL divergence as the surrogate loss function (Zhang et al., 2019; Zheng et al., 2016), since h θ ( x i ) (cid:54)
=
h θ (ˆ x (cid:48) i ) implies that adversarial examples have different output distributions to that of natural examples.
Thus, we have
[Equation]
The third indicator function 1 ( h θ ( x i ) (cid:54) =
y i ) is a condition that emphasizes learning on misclassiﬁed examples.
However, the condition cannot be directly optimized if we conduct a hard decision during the training process (Ding et al.
(2019) uses hard decision and does not optimize the condition).
Instead, we propose to use a soft decision scheme by replacing 1 ( h θ ( x i ) (cid:54) =
y i ) with the output probability 1 − p y
i ( x i , θ ) .
This will be large for misclassiﬁed examples and small for correctly classiﬁed examples.
Surrogate Loss for Inner Maximization.
The goal of inner maximization is to generate adversarial example ˆ x (cid:48)
i for natural example x i by solving (5).
Therefore, we aim to ﬁnd a surrogate loss function for the indicator function 1 ( h θ ( x (cid:48) i ) (cid:54) =
y i ) .
Here, we leverage the commonly used CE loss as the surrogate loss and ﬁnd the adversarial example ˆ x (cid:48)
i 2 as follows:
[Equation]
Following our ﬁndings in Figure 1(b) that a strong attack can help robustness (though is negligible for misclassiﬁed examples), we propose to use the (strong) PGD attack to maximize the CE loss for both the correctly classiﬁed and misclassiﬁed examples, the same as standard adversarial training.
Note that other surrogate loss functions, such as those exploited in (Athalye et al., 2018; Carlini &
Wagner, 2017) for adversarial attack, could also be used here.
The Overall Objective.
Based on the surrogate loss functions, we can state the ﬁnal objective function for our proposed Misclassiﬁcation Aware adveRsarial Training (MART) defense:
[Equation]
where (cid:96) ( x i , y i , θ ) is deﬁned as
[Equation]
Here the adversarial example ˆ x (cid:48)
i is generated by (10), and λ is a tunable scaling parameter that balances the two parts of the ﬁnal loss, and is ﬁxed for all training examples.
The complete training procedure of MART is described in Appendix A.
3. R ELATION TO E XISTING W ORK
In this section, we brieﬂy discuss the difference between our MART and existing defense methods including standard adversarial training ( Standard ) (Madry et al., 2018), logit pairing methods (Kannan et al., 2018), max-margin adversarial training (MMA) (Ding et al., 2018) and TRADES (Zhang et al., 2019), as presented in Table 1.
example in
MMA is generated by solving (10) with different perturbation limit ( i.e. , (cid:15) ).
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Speciﬁcally, the Standard algorithm was designed to minimize the standard adversarial loss, i.e. , cross-entropy loss on adversarial examples.
Logit pairing methods, consisting of adversarial logit pairing (ALP) and clean logit pairing (CLP), introduce a regularization term enclosing both natural examples and their adversarial counterparts.
The objective function of TRADES is also a linear combination of natural loss and regularization terms on the output probabilities corresponding to natural examples and their adversarial counterparts using KL divergence.
However, none of these algorithms differentiates the misclassiﬁed examples and correctly classiﬁed examples.
The most relevant work is MMA, which proposes to use maximal margin optimization for correctly classiﬁed examples while keeping the optimization on misclassiﬁed examples unchanged.
Speciﬁcally, for correctly classiﬁed examples, MMA adopts cross-entropy loss on adversarial examples, which are generated by solving (10) with example-dependent perturbation limit.
For misclassiﬁed examples,
MMA directly applies cross-entropy loss on natural examples.
We emphasize that our MART is different from MMA in the following aspects: (1) MMA performs hard decision to identify misclassiﬁed examples from training data, while MART uses soft decision scheme on training data based on the corresponding output probabilities ( p (ˆ x (cid:48) , θ ) ), which can be jointly learned during the training process; (2) for correctly classiﬁed examples, MMA adopts cross-entropy loss on adversarial examples with different perturbation limits, while MART utilizes the proposed BCE loss on the adverarial examples with the same perturbation limit; (3) for misclassiﬁed examples, MMA adopts cross-entropy loss on natural examples, while MART adopts a regularized adversarial loss involving both adversarial and natural examples.
Because of these differences, we later will show that MART outperforms MMA in the experiments.
3. S EMI - SUPERVISED E XTENSION WITH UNLABELED D ATA
Recent work has shown that semi-supervised learning with additional unlabeled data can improve the adversarial robustness (Uesato et al., 2019;
Carmon et al., 2019; Najaﬁ et al., 2019; Zhai et al., 2019).
Speciﬁcally, the training loss function applied in these semi-supervised learning methods is typically deﬁned as a weighted sum of the supervised loss (loss on the labeled data) and the unsupervised one (loss on the unlabeled data), i.e. ,
[Equation]
where γ > 0 is the weight of unsupervised loss.
As pointed out in Uesato et al.
(2019), there are multiple choices of the unsupervised loss function L unsup ( θ ) , leading to different defense methods, among which the most effective defense method is UAT++.
In particular, UAT++ ﬁrst trains a natural model on labeled data, and then use this model to generate pseudo labels for unlabeled data.
Moreover, given a training data ( x , y ) (which can be either labeled or unlabeled data), the supervised and unsupervised loss functions adopted in UAT++ are deﬁned as
[Equation]
where λ is a tunable hyperparameter.
A similar idea was also proposed in a concurrent work (Carmon et al., 2019), leading to another semi-supervised defense method called RST.
The ﬁrst stage of RST is also to generate pseudo labels for unlabeled data by training a natural model on the labeled data.
Then in the second stage, RST applies TRADES loss to train the robust model based on both labeled and unlabeled data, i.e. , given a training data ( x , y ) , the supervised and unsupervised loss functions adopted in RST is deﬁned as
[Equation]
As we pointed out in Figure 1(b) and the following experiment section
[Equation]
the maximization technique has a neglectable inﬂuence on the robustness.
Therefore, the major difference between UAT++ and
RST is the objective function for minimization.
Considering MART is also an objective function, it thus could be easily combined with semi-supervised learning with unlabeled data.
Following RST
[Equation]
we propose the following semi-supervised version of MART:
[Equation]
with supervised and unsupervised loss function deﬁned as follows,
[Equation]
where the adversarial example ˆ x (cid:48) is generated by solving (10)
[Equation]
and S sup and S unsup denote the set of labeled data and unlabeled data respectively.
E XPERIMENTS
In this section
[Equation]
we ﬁrst conduct a set of experiments to provide a comprehensive understanding of our proposed defense MART
[Equation]
and then evaluate its robustness on benchmark datasets in both white-box and black-box settings.
Finally, we benchmark the state-of-the-art robustness and explore using unlabeled data for further improvement.
4. U NDERSTANDING THE P ROPOSED MART
Here, we investigate MART from 4 different perspectives: (1) removing components of the MART loss function, (2) replacing components of the MART loss function, (3) misclassiﬁcation aware loss on certain proportions of training data, and (4) sensitivity to regularization parameter λ .
Experimental Setup.
We train ResNet-18 (He et al., 2016) with different variants of MART on
CIFAR-10 (Krizhevsky & Hinton, 2009).
All the models are trained using SGD with momentum 0.9, weight decay 2 × 10 − 4 and an initial learning rate of 0.1, which is divided by 10 at the 75-th and 90-th epoch (100 epochs in total).
All natural images are normalized into [0, 1], and simple data augmentations including 4-pixel padding with 32 × 32 random crop and random horizontal ﬂip.
The maximum perturbation (cid:15) = 8 / 255 and parameter λ
= 6 .
The training attack is PGD 10 with random start and step size (cid:15)/ 4 , while the test attack is PGD 20 with random start and step size (cid:15)/ 10 .
Removing Components of MART.
Recalling the objective function of MART in (11), it has three terms in the loss function: BCE, KL and 1 − p 4 .
As illustrated in Figure 2(a), removing 1 − p or KL
[Figure]
or both all leads to a signiﬁcant robustness degradation.
In particular, we found that the soft decision term 1 − p has a constant robustness improvement throughout the training process, while the KL term can help mitigate overﬁtting at a later stage of training (after 80 epochs).
When the two terms are combined together, they boost the ﬁnal robustness considerably without causing overﬁtting.
Replacing Components of MART.
As we show in Figure 2(b), when the BCE component is either replaced by a CE term or redeﬁned on natural examples ( x nat ), the ﬁnal robustness decreases by a substantial amount.
It suggests that learning with CE instead of our proposed BCE suffers from insufﬁcient learning with lower robustness throughout the entire training process.
On the other hand, learning with BCE on natural examples exhibits severe overﬁtting at the later stage (solid green line).
We did not observe any beneﬁt when replacing CE by KL in the inner maximization of adversarial min-max framework (solid red line), an observation that is consistent with Figure 1(b).
Ablation on Training Data.
Here, we show the contribution of our proposed misclassiﬁcation aware regularization ( e.g. , the KL · (1 − p ) term in (11)) to the ﬁnal robustness with respect to the training data.
Speciﬁcally, we gradually increase the proportion of training examples that are trained using the proposed misclassiﬁcation aware regularization term, and display the corresponding robustness in
Figure 2(c).
The training examples using the proposed regularization are randomly selected, and the
BCE term is still deﬁned on all training (adversarial) examples.
As can be observed, the robustness can be improved steadily when the proposed regularization is applied on more data.
This veriﬁes the beneﬁt of the differentiation of correctly classiﬁed and misclassiﬁed examples.
Sensitivity to Regularization Parameter λ .
We further investigate the parameter λ in MART objective function deﬁned in (11) which controls the strength of the regularization.
We also test the regularization parameter λ of TRADES (please refer to Table 1).
We present the results in Figure 2(d) 5 for different λ ∈ [1 / 2 , 50] .
By explicitly differentiating the misclassiﬁed and correctly classiﬁed examples, MART achieves good stability and robustness across different choices of λ , which is also consistently better and more stable than that of TRADES.
4. R OBUSTNESS E VALUATION AND A NALYSIS
In this part, we evaluate the robustness of MART on both MNIST (LeCun et al., 1998) and CIFAR-
(Krizhevsky & Hinton, 2009) datasets against various white-box and black-box attacks.
Baselines.
6 (1) Standard (Madry et al., 2018);
(2) MMA (Ding et al., 2019); (3) Dynamic (Wang et al., 2019); and (4) TRADES (Zhang et al., 2019).
We only compare with adversarial training variants, since they are the most effective defense to date (Athalye et al., 2018).
Defense Settings.
For MNIST, all defense models are built on a 4 -layer CNN and trained using SGD with momentum 0 .
The initial learning rate is 0 . 01 and divided by 10 at the 20 -th and 40 -th epoch
( 50 epochs in total).
For CIFAR-10, the settings are the same as Section 4.1.
For the training attack, it is also the PGD 10 with random start and step size (cid:15)/ 4 .
The perturbation limit (cid:15) = 0 .
3 for MNIST, and (cid:15) = 8 / 255 for CIFAR-10.
For MART, we set λ = 6 .
Other hyperparameters of the baselines are conﬁgured as per their original papers: max margin is set to 0 .
45 (MNIST) or 12 / 255 (CIFAR-10) for MMA, maximum criterion value c max = 0 .
5 for Dynamic and λ = 6 for TRADES.
[Table]
[Table]
White-box Robustness
We evaluate the robustness of all defense models against three types of attacks for both MNIST and CIFAR-10: FGSM, PGD 20 (20-step PGD with step size (cid:15)/ 10 ), and
CW ∞ ( L ∞ version of CW optimized by PGD).
All attacks have full access to model parameters and are constrained by the same perturbation limit (cid:15) .
The white-box robustness of all defense models are reported in Table 2, where “Natural” denotes the accuracy on natural test images.
Our proposed defense MART achieves the best robustness against all three types of attacks on both MNIST and
CIFAR-10.
Compared with MNIST, the robustness improvements of MART over other baselines are more signiﬁcant on CIFAR-10.
This is because adversarial training on CIFAR-10 is a more challenging problem that may have more misclassiﬁed examples during training, and MART can better handle those misclassiﬁed examples due to its regularization term in (11).
Note that the robustness improvement of MART is not caused by the so-called “obfuscated gradients” (Athalye et al., 2018).
This can be veriﬁed by two phenomenons: (1) strong test attacks ( e.g. , PGD 20 ) have higher success rates (lower accuracies) than weak test attacks ( e.g. , FGSM), and (2) white-box test attacks have higher success rates than back-box test attacks (comparing Table 2 with Table 3).
Besides, we conduct an additional check using a gradient-free attack SPSA (Uesato et al., 2018).
SPSA attack does not obtain lower accuracy than gradient-based attacks like PGD, which conﬁrms that the robustness of MART trained models are not due to gradient masking.
Black-box Robustness.
Black-box test attacks are crafted from the natural test images by attacking a surrogate model with an architecture that is either a copy of the defense model (MNIST) or a more complex ResNet-50 (He et al., 2016) model (CIFAR-10).
Both surrogate models are trained separately from the defense models on the original training sets using Standard adversarial training (PGD 10 with random start and step size (cid:15)/ 4 ).
The attacking methods used here are: FGSM, PGD 10 ,
PGD 20 , and CW ∞ .
The black-box robustness of all defense models are reported in Table 3.
Again, the proposed defense MART achieves higher robustness than other baselines.
Compared with the white-box results, all defense methods achieve much better robustness against black-box attacks, even close to the natural accuracy.
This suggests that adversarial training is indeed a very practical choice for defense scenarios where the target model can be kept secret from potential attackers.
It is also observed that robustness on strong attacks like CW ∞ is higher than weak attacks like FGSM, which indicates that strong attacks have less transferability than weak attacks (Madry et al., 2018).
4. B ENCHMARKING THE S TATE - OF - THE - ART R OBUSTNESS
In this part, we conduct more experiments on a large-capacity network WideResNet (Zagoruyko &
Komodakis, 2016) to benchmark the state-of-the-art robustness, and also explore using unlabeled data for further robustness boost.
Performance on WideResNet.
We employ WideResNet-34-10 (depth 34 and width 10) to explore the full power of our proposed MART defense method, and also benchmark the state-of-the-art robustness on CIFAR-10.
The robustness of all defense models are tested against white-box FGSM,
[Table] [Table]
PGD 20 , PGD 100 and CW ∞ attacks, under the same settings as Section 4.2.
We report the robustness of both the best and the last epoch models obtained during training in Table 4 7 .
For each defense method against each attack, the “best” refers to the highest robustness that ever achieved at different checkpoints. Speciﬁcally, against FGSM attack, the best model was found at the last epoch ( e [Table]
g. , "best" is also "last") for all defense methods, while against PGD 20 , PGD 100 and CW ∞ attacks, the best model was found at the epoch right after the ﬁrst time learning rate decay ( i.e. , epoch 76).
Our proposed MART outperforms all baseline methods in terms of the robustness of both the best and the last epoch models.
Particularly under the most common comparison setting (against PGD 20 attacks on CIFAR-10), MART improved ∼ 8% over Standard , and ∼ 4% even over TRADES for the last epoch model.
A similar trend of improvement is also observed for the best epoch model results.
Considering the worst case accuracies against all attacks, MART still gains ∼ 6% and ∼ 3 .
5% robustness improvement over Standard and TRADES respectively.
Boosting with Additional Unlabeled Data.
Here, we evaluate the proposed semi-supervised version of MART and show that it can also beneﬁt from additional unlabeled data and achieves better robustness.
Following the exact settings in UAT++ (Uesato et al., 2019) and RST (Carmon et al., 2019), we compare the robustness of MART with them on WideResNet-34-8 and WideResNet-28against PGD 20 (FGSM 20 ) respectively (in the same setting as reported in their paper).
The dataset is
CIFAR-10 with 100K and 500K unlabeled data extracted from the 80 Million Tiny Images dataset (Torralba et al., 2008).
As conﬁrmed in Table 5, our proposed defense MART can also beneﬁt from unlabeled data, and further improves the UAT++ and RST defenses, achieving the state-of-the-art robustness.
This again veriﬁes the beneﬁt of differentiating misclassiﬁed and correctly classiﬁed examples for improving robustness, and further demonstrates the superiority of our proposed method.
C ONCLUSION AND F UTURE W ORK
In this paper, we investigated the interesting observation that misclassiﬁed examples have a recognizable impact on the ﬁnal robustness of adversarial training, especially for the outer minimization process.
Based on this observation, we designed a misclassiﬁcation aware adversarial risk, which is formulated as adding an misclassiﬁcation aware regularization to the standard adversarial risk.
Following the regularized adversarial risk, we proposed a new defense algorithm, called Misclassiﬁcation Aware adveRsarial Training (MART), with appropriate surrogate loss functions.
Experimental results demonstrated that MART can achieve signiﬁcantly improved adversarial robustness with respect to the state-of-the-art, and can also achieves better robustness with additional unlabeled data.
In the future, we plan to investigate the effect of differentiation of correctly classiﬁed/misclassiﬁed training examples in the recently proposed certiﬁed/provable robustness framework (Cohen et al., 2019; Salman et al., 2019) and explore the potential improvements brought by the differentiation of training examples.
B E XPLANATIONS ABOUT THE RESULTS OF TRADES
To make a fair comparison with the latest method TRADES (Zhang et al., 2019), our code is built upon the TRADES framework.
However, TRADES has replication issues, as pointed out by Carmon et al.
and other researchers at GitHub issue #9 “Robust Accuracy on CIFAR10”.
The authors of TRADES conﬁrmed that the robustness reported in their paper is the *best* robustness that ever occurred during training.
To avoid questioning of our results, we have reported both the