C ASCADE A DVERSARIAL M ACHINE L EARNING R EGULARIZED WITH A U NIFIED E MBEDDING
Injecting adversarial examples during training, known as adversarial training, can improve robustness against one-step attacks, but not for unknown iterative attacks.
To address this challenge, we ﬁrst show iteratively generated adversarial images easily transfer between networks trained with the same strategy .
Inspired by this observation, we propose cascade adversarial training , which transfers the knowledge of the end results of adversarial training.
We train a network from scratch by injecting iteratively generated adversarial images crafted from already defended networks in addition to one-step adversarial images from the network being trained.
We also propose to utilize embedding space for both classiﬁcation and low-level (pixel-level) similarity learning to ignore unknown pixel level perturbation.
During training, we inject adversarial images without replacing their corresponding clean images and penalize the distance between the two embeddings (clean and adversarial).
Experimental results show that cascade adversarial training together with our proposed low-level similarity learning efﬁciently enhances the robustness against iterative attacks, but at the expense of decreased robustness against one-step attacks.
We show that combining those two techniques can also improve robustness under the worst case black box attack scenario.
I NTRODUCTION
Injecting adversarial examples during training (adversarial training), (Goodfellow et al., 2015; Kurakin et al., 2017; Huang et al., 2015) increases the robustness of a network against adversarial attacks.
The networks trained with one-step methods have shown noticeable robustness against onestep attacks, but, limited robustness against iterative attacks at test time.
To address this challenge, we have made the following contributions:
Cascade adversarial training: We ﬁrst show that iteratively generated adversarial images transfer well between networks when the source and the target networks are trained with the same training method .
Inspired by this observation, we propose cascade adversarial training which transfers the knowledge of the end results of adversarial training.
In particular, we train a network by injecting iter FGSM images (section 2.1) crafted from an already defended network (a network trained with adversarial training) in addition to the one-step adversarial images crafted from the network being trained.
The concept of using already trained networks for adversarial training is also introduced in (Tram`er et al., 2017).
In their work, purely trained networks are used as another source networks to generate one-step adversarial examples for training.
On the contrary, our cascade adversarial training uses already defended network for iter FGSM images generation.
Low level similarity learning: We advance the previous data augmentation approach (Kurakin et al., 2017) by adding additional regularization in deep features to encourage a network to be insensitive to adversarial perturbation.
In particular, we inject adversarial images in the mini batch without replacing their corresponding clean images and penalize distance between embeddings from the clean and the adversarial examples.
There are past examples of using embedding space for learning similarity of high level features like face similarity between two different images (Schroff et al., 2015; Parkhi et al., 2015; Wen et al., 2016).
Instead, we use the embedding space for learning similarity of the pixel level differences between two similar images.
The intuition of using this regularization is that small difference on input should not drastically change the high level feature representation .
Analysis of adversarial training: We train ResNet models (He et al., 2016) on MNIST (LeCun & Cortes, 2010) and CIFAR10 dataset (Krizhevsky, 2009) using the proposed adversarial training.
We ﬁrst show low level similarity learning improves robustness of the network against adversarial images generated by one-step and iterative methods compared to the prior work.
We show that modifying the weight of the distance measure in the loss function can help control trade-off between accuracies for the clean and adversarial examples.
Together with cascade adversarial training and low-level similarity learning, we achieve accuracy increase against unknown iterative attacks, but at the expense of decreased accuracy for one-step attacks.
We also show our cascade adversarial training and low level similarity learning provide much better robustness against black box attack.
B ACKGROUND ON A DVERSARIAL A TTACKS 2. A TTACK M ETHODS
One-step fast gradient sign method (FGSM) , referred to as “step FGSM”, generates adversarial image X adv by adding sign of the gradients w.r.t.
the clean image X multiplied by (cid:15) ∈ [0 , 255] as shown below (Goodfellow et al., 2015):
[Equation]
One-step target class method generates X adv by subtracting sign of the gradients computed on a target false label as follows:
[Equation]
We use least likely class y LL as a target class and refer this method as “step ll”.
Basic iterative method , referred to as “iter FGSM”, applies FGSM with small α multiple times.
[Equation]
1 + α sign( ∇ X advN −
[Equation]
( X advN − 1 , y true )) (cid:9)
We use α = 1, number of iterations N to be min( (cid:15) + 4 , 1 . 25 (cid:15) ) .
Clip X,(cid:15) is elementwise clipping function where the input is clipped to the range [max(0 , X − (cid:15) ) , min(255 , X + (cid:15) )] .
Iterative least-likely class method , referred to as “iter ll”, is to apply “step ll” with small α multiple times.
[Equation]
Carlini and Wagner attack (Carlini & Wagner, 2017) referred to as “CW” solves an optimization problem which minimizes both an objective function f (such that attack is success if and only if f ( X adv ) < 0 ) and a distance measure between X adv and X .
Black box attack is performed by testing accuracy on a target network with the adversarial images crafted from a source network different from the target network.
Lower accuracy means successful black-box attack.
When we use the same network for both target and source network, we call this as white-box attack.
2. D EFENSE M ETHODS
Adversarial training (Kurakin et al., 2017): is a form of data augmentation where it injects adversarial examples during training.
In this method, k examples are taken from the mini batch B (size of m ) and the adversarial examples are generated with one of step method.
The k adversarial examples replaces the corresponding clean examples when making mini batch.
Below we refer this adversarial training method as “Kurakin’s”.
[Table]
[Figure]
[Table]
[Figure]
Ensemble adversarial training (Tram`er et al., 2017): is essentially the same with the adversarial training, but uses several pre-trained vanilla networks to generate one-step adversarial examples for training.
Below we refer this adversarial training method as “Ensemble”.
P ROPOSED A PPROACH 3. T RANSFERABILITY A NALYSIS
We ﬁrst show transferability between purely trained networks and adversarially trained networks under black box attack.
We use ResNet (He et al., 2016) models for CIFAR10 classiﬁcation.
We ﬁrst train 20-layer ResNets with different methods (standard training, adversarial training (Kurakin et al., 2017)) and use those as target networks.
We re-train networks (standard training and adversarial training) with the different initialization from the target networks, and use the trained networks as source networks.
Experimental details and model descriptions can be found in Appendix A and B.
In table 1, we report test accuracies under black box attack.
Transferability (step attack) :
We ﬁrst observe that high robustness against one-step attack between defended networks (R20 K 2 - > R20 K ), and low robustness between undefended networks (R20 2 - > R20).
This observation shows that error surfaces of neural networks are driven by the training method and networks trained with the same method end up similar optimum states.
It is noteworthy to observe that the accuracies against step attack from the undefended network (R20 2 ) are always lower than those from defended network (R20 K 2 ).
Possible explanation for this would be that adversarial training tweaks gradient seen from the clean image to point toward weaker adversarial point along that gradient direction.
As a result, one-step adversarial images from defended networks become weaker than those from undefended network.
Transferability (iterative attack) :
We observe “iter FGSM” attack remains very strong even under the black box attack scenario but only between undefended networks or defended networks .
This is because iter FGSM noises ( X adv - X ) from defended networks resemble each other.
As shown in ﬁgure 1, we observe higher correlation between iter FGSM noises from a defended network (R20 K ) and those from another defended network (R20 K 2 ).
Difﬁculty of defense/attack under the black box attack scenario :
As seen from this observation, it is efﬁcient to attack an undefended/defended network with iter FGSM examples crafted from another undefended/defended network.
Thus, when we want to build a robust network under the black box attack scenario, it is desired to check accuracies for the adversarial examples crafted from other networks trained with the same strategy .
3. C ASCADE A DVERSARIAL T RAINING
Inspired by the observation that iter FGSM images transfer well between defended networks, we propose cascade adversarial training , which trains a network by injecting iter FGSM images crafted from an already defended network.
We hypothesize that the network being trained with cascade adversarial training will learn to avoid such adversarial perturbation, enhancing robustness against iter FGSM attack.
The intuition behind this proposed method is that we transfer the knowledge of
[Figure]
[Figure]
the end results of adversarial training .
In particular, we train a network by injecting iter FGSM images crafted from already defended network in addition to the one-step adversarial images crafted from the network being trained.
3. R EGULARIZATION WITH A U NIFIED E MBEDDING
We advance the algorithm proposed in (Kurakin et al., 2017) by adding low level similarity learning.
Unlike (Kurakin et al., 2017), we include the clean examples used for generating adversarial images in the mini batch.
Once one step forward pass is performed with the mini batch, embeddings are followed by the softmax layer for the cross entropy loss for the standard classiﬁcation.
At the same time, we take clean embeddings and adversarial embeddings, and minimize the distance between the two with the distance based loss.
The distance based loss encourages two similar images (clean and adversarial) to produce the same outputs, not necessarily the true labels .
Thus, low-level similarity learning can be considered as an unsupervised learning.
By adding regularization in higher embedding layer, convolution ﬁlters gradually learn how to ignore such pixel-level perturbation.
We have applied regularization on lower layers with an assumption that low level pixel perturbation can be ignored in lower hierarchy of networks.
However, adding regularization term on higher embedding layer right before the softmax layer showed best performance.
The more convolutional ﬁlters have chance to learn such similarity, the better the performance.
Note that cross entropy doesn’t encourage two similar images to produce the same output labels.
Standard image classiﬁcation using cross entropy compares ground truth labels with outputs of a network regardless of how similar training images are.
The entire training process combining cascade adversarial training and low level similarity learning is shown in ﬁgure 2.
We deﬁne the total loss as follows:
[Equation]
E i and E advi are the resulting embeddings from X i and X advi , respectively.
m is the size of the mini batch, k ( ≤ m/ 2 ) is the number of adversarial images in the mini batch.
λ is the parameter to control the relative weight of classiﬁcation loss for adversarial images.
λ 2 is the parameter to control the relative weight of the distance based loss L dist in the total loss.
Bidirectional loss minimizes the distance between the two embeddings by moving both clean and adversarial embeddings as shown in the left side of the ﬁgure 3.
[Equation]
[Table]
We tried N = 1, 2 and found not much difference between the two.
We report the results with N = 2 for the rest of the paper otherwise noted.
When N = 2, L dist becomes L2 loss.
Pivot loss minimizes the distance between the two embeddings by moving only the adversarial embeddings as shown in the right side of the ﬁgure 3.
[Equation]
In this case, clean embeddings ( E i ) serve as pivots to the adversarial embeddings.
In particular, we don’t back-propagate through the clean embeddings for the distance based loss.
The intuition behind the use of pivot loss is that the embedding from a clean image can be treated as the ground truth embedding.
L OW L EVEL S IMILARITY L EARNING A NALYSIS 4. E XPERIMENTAL R ESULTS ON MNIST
We ﬁrst analyze the effect of low level similarity learning on MNIST.
We train ResNet models (He et al., 2016) with different methods (standard training, Kurakin’s adversarial training and adversarial training with our distance based loss).
Experimental details can be found in Appendix A.
Table 2 shows the accuracy results for MNIST test dataset for different types of attack methods.
As shown in the table, our method achieves better accuracy than Kurakin’s method for all types of attacks with a little sacriﬁce on the accuracy for the clean images.
Even though adversarial training is done only with “step ll”, additional regularization increases robustness against unknown “step FGSM”, “iter ll”, “iter FGSM” and CW L ∞ attacks.
This shows that our low-level similarity learning can successfully regularize the one-step adversarial perturbation and its vicinity for simple image classiﬁcation like MNIST.
4. E MBEDDING S PACE V ISUALIZATION
To visualize the embedding space, we modify 20-layer ResNet model where the last fully connected layer (64x10) is changed to two fully connected layers (64x2 and 2x10).
We re-train networks with standard training, Kurakin’s method and our pivot loss on MNIST.
1 In ﬁgure 4, we draw embeddings (dimension=2) between two fully connected layers.
As seen from this ﬁgure, adversarial images from the network trained with standard training cross the decision boundary easily as (cid:15) increases.
With Kurakin’s adversarial training, the distances between clean and adversarial embeddings are minimized compared to standard training.
And our pivot loss further minimizes distance between the clean and adversarial embeddings.
Note that our pivot loss also decreases absolute value of the embeddings, thus, higher λ 2 will eventually result in overlap between distinct embedding distributions.
We also observe that intra class variation of the clean embeddings are also minimized for the network trained with our pivot loss as shown in the scatter plot in ﬁgure 4 (c).
[Figure]
4. E FFECT OF λ 2 ON CIFAR
[Figure]
We train 20-layer ResNet models with pivot loss and various λ 2 s for CIFAR10 dataset to study effects of the weight of the distance measure in the loss function.
Figure 5 shows that a higher λ 2 increases accuracy of the iteratively generated adversarial images.
However, it reduces accuracy on the clean images
[Figure]
and increasing λ 2 above 0.3 even results in divergence of the training.
This is because embedding distributions of different classes will eventually overlap since absolute value of the embedding will be decreased as λ 2 increases as seen from the section 4.2.
In this experiment, we show that there exists clear trade-off between accuracy for the clean images and that for the adversarial images, and we recommend using a very high λonly under strong adversarial environment.
C ASCADE A DVERSARIAL T RAINING A NALYSIS 5. S OURCE N ETWORK S ELECTION
[Figure]
We further study the transferability of iter FGSM images between various architectures.
To this end, we ﬁrst train 56-layer ResNet networks (Kurakin’s, pivot loss) with the same initialization.
Then we train another 56-layer ResNet network (Kurakin’s) with different initialization.
We repeat the training for the 110-layer
ResNet networks.
We measure correlation between iter FGSM noises from different networks.
Figure 6 (a) shows correlation between iter FGSM noises crafted from
Kurakin’s network and those from
Pivot network with the same initialization.
Conjectured from (Kurakin et al., 2017), we observe high corre-
[Table]
lation between iter FGSM noises from networks with the same initialization.
Correlation between iter FGSM noises from the networks with different initialization, however, becomes lower as the network is deeper as shown in ﬁgure 6 (b).
Since the degree of freedom increases as the network size increases, adversarially trained networks prone to end up with different states, thus, making transfer rate lower.
To maximize the beneﬁt of the cascade adversarial training, we propose to use the same initialization for a cascade network and a source network used for iterative adversarial examples generation.
5. W HITE B OX A TTACK A NALYSIS
We ﬁrst compare a network trained with Kurakin’s method and that with pivot loss.
We train 110layer ResNet models with/without pivot loss and report accuracy in table 3.
We observe our lowlevel similarity learning further improves robustness against iterative attacks compared to Kurakin’s adversarial training.
However, the accuracy improvements against iterative attacks (iter FGSM,
CW) are limited, showing regularization effect of low-level similarity learning is not sufﬁcient for the iterative attacks on complex color images like CIFAR10.
This is different from MNIST test cases where we observed signiﬁcant accuracy increase for iterative attacks only with pivot loss.
We observe label leaking phenomenon reported in (Kurakin et al., 2017) happens even though we don’t train a network with step FGSM images.
Additional analysis for this phenomenon is explained in
Appendix D.
Next, we train a network from scratch with iter FGSM examples crafted from the defended network,
R110 P .
We use the same initialization used in R110 P as discussed in 5.1.
In particular, iter FGSM images are crafted from R110 P with CIFAR10 training images for (cid:15) =
1,2, ..., 16, and those are used randomly together with step ll examples from the network being trained.
We train cascade networks with/without pivot loss.
We also train networks with ensemble adversarial training (Tram`er et al., 2017) with/without pivot loss for comparison.
The implementation details for the trained models can be found in Appendix B.
We ﬁnd several meaningful observations in table 3.
First, ensemble and cascade models show improved accuracy against iterative attack although at the expense of decreased accuracy for onestep attacks compared to the baseline defended network (R110 K ).
Additional data augmentation from other networks enhances the robustness against iterative attack, weakening label leaking effect caused by one-step adversarial training.
Second, our low-level similarity learning (R110 P,E , R110 P,C ) further enhances robustness against iterative attacks including fully unknown CW attack (especially for (cid:15) =4).
Additional knowledge learned from data augmentation through cascade/ensemble adversarial training enables networks to learn partial knowledge of perturbations generated by an iterative method.
And the learned iterative perturbations become regularized further with our low-level similarity learning making networks robust against unknown iterative attacks.
[Table]
Third, our low-level similarity learning serves as a good regularizer for adversarial images
[Table]
but not for the clean images for ensemble/cascade models (reduced accuracy for the clean images for
R110 P,E and R110 P,C in table 3).
Compared to pure adversarial training with one-step adversarial examples, the network sees more various adversarial perturbations during training as a result of ensemble and cascade training.
Those perturbations are prone to end up embeddings in the vicinity of decision boundary more often than perturbations caused by one-step adversarial training.
Pivot loss pulls the vicinity of those adversarial embeddings toward their corresponding clean embeddings.
During this process, clean embeddings from other classes might also be moved toward the decision boundary which results in decreased accuracy for the clean images.
5. B LACK B OX A TTACK A NALYSIS
We ﬁnally perform black box attack analysis for the cascade/ensemble networks with/without pivot loss.
We report black box attack accuracy with the source networks trained with the same method, but with different initialization from the target networks.
The reason for this is adversarial examples transfer well between networks trained with the same strategy as observed in section 3.1.
We re-train 110-layer ResNet models using Kurakin’s, cascade and ensemble adversarial training with/without low-level similarity learning and use those networks as source networks for black-box attacks.
Baseline 110-layer ResNet model is also included as a source network.
Target networks are the same networks used in table 3.
We found iter FGSM attack resulted in lower accuracy than step FGSM attack, thus, report iter FGSM attack only in table 4.
We ﬁrst observe that iter FGSM attack from ensemble models (R110 E 2 , R110 P,E 2 ) is strong (results in lower accuracy) compared to that from any other trained networks.
2 Since ensemble models learn various perturbation during training, adversarial noises crafted from those networks might be more general for other networks making them transfer easily between defended networks.
Second, cascade adversarial training breaks chicken and egg problem.
(In section 3.1, we found that it is efﬁcient to use a defended network as a source network to attack another defended network.)
Even though the transferability between defended networks is reduced for deeper networks, cascade network (R110 K,C ) shows worst case performance against the attack not from a defended network, but from a purely trained network (R110 2 ).
Possible solution to further improve the worst case robustness would be to use more than one network as source networks (including pure/defended networks) for iter FGSM images generation for cascade adversarial training.
Third, ensemble/cascade networks together with our low-level similarity learning (R110 P,E ,
R110 P,C ) show better worst case accuracy under black box attack scenario.
This shows that enhancing robustness against iterative white box attack also improves robustness against iterative black box attack.
C ONCLUSION
We performed through transfer analysis and showed iter FGSM images transfer easily between networks trained with the same strategy .
We exploited this and proposed cascade adversarial training, a method to train a network with iter FGSM adversarial images crafted from already defended networks.
We also proposed adversarial training regularized with a uniﬁed embedding for classiﬁcation and low- level similarity learning by penalizing distance between the clean and their corresponding adversarial embeddings.
Combining those two techniques (low level similarity learning + cascade adversarial training) with deeper networks further improved robustness against iterative attacks for both white-box and black-box attacks.
However, there is still a gap between accuracy for the clean images and that for the adversarial images.
Improving robustness against both one-step and iterative attacks still remains challenging since it is shown to be difﬁcult to train networks robust for both one-step and iterative attacks simultaneously.
Future research is necessary to further improve the robustness against iterative attack without sacriﬁcing the accuracy for step attacks or clean images under both white-box attack and black-box attack scenarios.
A CKNOWLEDGMENTS
We thank Alexey Kurakin of Google Brain and Li Chen of Intel for comments that greatly improved the manuscript.
The research reported here was supported in part by the Defense Advanced Research
Projects Agency (DARPA) under contract number HR0011-17-2-0045.
The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the ofﬁcial policies or endorsements, either expressed or implied, of DARPA.
A E XPERIMENTAL S ETUP
We scale down the image values to [0,1] and don’t perform any data augmentation for MNIST.
For
CIFAR10, we scale down the image values to [0,1] and subtract per-pixel mean values.
We perform 24x24 random crop and random ﬂip on 32x32 original images.
We generate adversarial images with “step ll” after these steps otherwise noted.
We use stochastic gradient descent (SGD) optimizer with momentum of 0.9, weight decay of 0.
and mini batch size of 128.
For adversarial training, we generate k = 64 adversarial examples among 128 images in one mini-batch.
We start with a learning rate of 0.1, divide it by 10 at 4k and 6k iterations, and terminate training at 8k iterations for MNIST, and 48k and 72k iterations, and terminate training at 94k iterations for CIFAR10.
We also found that initialization affects the training results slightly as in (Kurakin et al., 2017), thus, we pre-train the networks 2 and 10 epochs for MNIST and CIFAR10, and use these as initial starting points for different conﬁgurations.
We use max e = 0.3*255 and 16 for MNIST and CIFAR respectively.
B M ODEL D ESCRIPTIONS
We summarize the model names used in this paper in table 5.
For ensemble adversarial training, pre-trained networks as in table 7 together with the network being trained are used to generate onestep adversarial examples during training.
For cascade adversarial training, pre-trained defended networks as in table 6 are used to generate iter FGSM images, and the network being trained is used to generate one-step adversarial examples during training.
[Table]
[Table] [Table]
C A LTERNATIVE V ISUALIZATION ON E
[Figure]
[Figure]
We draw average value of the argument to the softmax layer for the true class and the false classes to visualize how the adversarial training works as in ﬁgure 7.
Standard training, as expected, shows dramatic drop in the values for the true class as we increase (cid:15) in “step ll” or “step FGSM direction.
With adversarial training, we observe that the value drop is limited at small (cid:15) and our method even increases the value in certain range upto (cid:15)
=10.
Note that adversarial training is not the same as the gradient masking.
As illustrated in ﬁgure 7, it exposes gradient information, however, quickly distort gradients along the sign of the gradient (“step ll” or “step FGSM) direction.
We also observe improved results (broader margins than baseline) for “random sign” added images even though we didn’t inject random sign added images during training.
Overall shape of the argument to the softmax layer in our case becomes smoother than Kurakin’s method, suggesting our method is good for pixel level regularization.
Even though actual value of the embeddings for the true class in our case is smaller than that in Kurakin’s, the standard deviation of our case is less than Kurakin’s, making better margin between the true class and false classes.
D L ABEL L EAKING A NALYSIS
[Figure]
We observe accuracies for the “step FGSM” adversarial images become higher than those for the clean images (“label leaking” phenomenon) by training with “step FGSM” examples as in (Kurakin et al., 2017).
Interestingly, we also observe “label leaking” phenomenon even without providing true labels for adversarial images generation.
We argue that “label leaking” is a natural result of the adversarial training.
To understand the nature of adversarial training
[Figure]
we measure correlation between gradients w.r.t.
different images (i
[Figure]
e. clean vs. adversarial) as a measure of error surface similarity.
We measure correlation between gradients w.r.t.
(1) clean vs. “step ll” image, (2) clean vs. “step FGSM” image, (3) clean vs. “random sign” added image, and (4) “step ll” image vs. “step FGSM” image for three trained networks (a) R20, (b) R20 K and (c) R20 P (Ours) in table 5.
Figure 8 draws average value of correlations for each case.
Meaning of the correlation:
In order to make strong adversarial images with “step FGSM” method, correlation between the gradient w.r.t.
the clean image and the gradient w.r.t.
its corresponding adversarial image should remain high since the “step FGSM” method only use the gradient w.r.t.
the clean image ( (cid:15) =0).
Lower correlation means perturbing the adversarial image at (cid:15) further to the gradient (seen from the clean image) direction is no longer efﬁcient.
Results of adversarial training: We observe that (1) and (2) become quickly lower than (3) as (cid:15) increases.
This means that, when we move toward the steepest (gradient) direction on the error surface, gradient is more quickly uncorrelated with the gradient w.r.t.
the clean image than when we move to random direction.
As a result of adversarial training, this uncorrelation is observed at a lower (cid:15) making one-step attack less efﬁcient even with small perturbation.
(1), (2) and (3) for our case are slightly lower than Kurakin’s method at the same (cid:15) which means that our method is better at defending one-step attacks than Kurakin’s.
Error surface similarity between “step ll” and “step FGSM” images: We also observe (4) remains high with higher (cid:15) for all trained networks.
This means that the error surface (gradient) of the “step ll” image and that of its corresponding “step FGSM” image resemble each other.
That is the reason why we get the robustness against “step FGSM” method only by training with “step ll” method and vice versa.
(4) for our case is slightly higher than Kurakin’s method at the same (cid:15) and that means our similarity learning tends to make error surfaces of the adversarial images with “step ll” and “step FGSM” method to be more similar.
Analysis of label leaking phenomenon: Interestingly, (2) becomes slightly negative in certain range (1 < (cid:15) < 3 for Kurakin’s, and 1 < (cid:15) < 4 for Pivot (Ours) ) and this could be the possible reason for “label leaking” phenomenon.
For example, let’s assume that we have a perturbed image (by “step FGSM” method) at (cid:15) where the correlation between the gradients w.r.t.
that image and the corresponding clean image is negative.
Further increase of (cid:15) with the gradient (w.r.t.
the clean image) direction actually decreases the loss resulting in increased accuracy (label leaking phenomenon).
Due to the error surface similarity between “step ll” and “step FGSM” images and this negative correlation effect, however, label leaking phenomenon can always happen for the networks trained with one-step adversarial examples.
E A DDITIONAL B LACK B OX A TTACK R ESULTS
[Table]
Table 8 shows that black box attack between trained networks with the same initialization tends to be more successful than that between networks with different initialization as explained in (Kurakin et al., 2017).
[Table]
In table
[Table]
our method (R20 P 2 ) is always better at one-step and iterative black box attack from defended networks (R20 K , R20 P ) and undefended network (R20) than Kurakin’s method (R20 B 2 ).
However, it is hard to tell which method is better than the other one as explained in the main paper.
[Table]
In table
[Table]
we show black box attack accuracies with the source and the target networks switched from the table
[Table]
We also observe that networks trained with both low-level similarity learning and cascade/ensemble adversarial training (R110 P,C 2 , R110 P,E 2 ) show better worst-case performance than other networks.
Overall, iter FGSM images crafted from ensemble model families (R110 E ,
R110 P,E ) remain strong on the defended networks
[Table]
F I MPLEMENTATION D ETAILS FOR C ARLINI -W AGNER L ∞ A TTACK
[Figure]
Carlini and Wagner (CW) L ∞ attack solves the following optimization problem for every input X .
minimize c · f ( X + δ ) + (cid:88) i
[( | δ i | − τ )
+ ] such that X + δ ∈ [0 , 1] n where, the function f is deﬁned such that attack is success
if and only if f ( X + δ ) < 0 , δ is the target perturbation deﬁned as X adv − X , c is the parameter to control the relative weight of function f in the total cost function, and τ is the control threshold used to penalize any terms that exceed τ .
Since CW L ∞ attack is computationally expensive, we only use 100 test examples (10 examples per each class).
We search adversarial example X adv with c ∈ { 0 . 1 , 0 . 2 , 0 .
5 , 1 , 2 , 5 , 10 , 20 } and τ ∈ { 0 . 02 , 0 .
04 , ..., 0 . 6 } for MNIST and c ∈ { 0 . 1 , 0 .
3 , 1 , 3 , 10 , 30 , 100 } and τ ∈ { 0 . 001 , 0 .
for CIFAR10.
We use Adam optimizer with an initial learning rate of 0.01/ c since we found constant initial learning rate for c · f ( X + δ ) term is critical for successful adversarial images generation.
We terminate the search after 2,000 iterations for each X , c and τ .
If f ( X + δ ) < 0 and the resulting || δ || ∞ is lower than the current best distance, we update X adv .
Figure 9 shows cumulative distribution function of (cid:15) for 100 successful adversarial examples per each network.
We report the number of adversarial examples with (cid:15) > 0.3*255 for MNIST and that with (cid:15) > 2 or 4 for CIFAR10.
As seen from this ﬁgure, our approaches provide robust defense against CW L ∞ attack compared to other approaches.