D ETECTING AND D IAGNOSING A DVERSARIAL I MAGES WITH C LASS -C ONDITIONAL C APSULE R ECONSTRUCTIONS
Yao yaq007@eng.ucsd.edu
Nicholas Frosst ∗
Google Brain frosst@google.com
Sara Sabour
Google Brain sasabour@google.com
Colin Raffel
Google Brain craffel@google.com
Garrison Cottrell
San Diego gary@eng.ucsd.edu
Geoffrey Hinton
Google Brain geoffhinton@google.com
A BSTRACT
Adversarial examples raise questions about whether neural network models are sensitive to the same visual features as humans.
In this paper, we ﬁrst detect adversarial examples or otherwise corrupted images based on a class-conditional reconstruction of the input.
To speciﬁcally attack our detection mechanism, we propose the Reconstructive Attack which seeks both to cause a misclassiﬁcation and a low reconstruction error.
This reconstructive attack produces undetected adversarial examples but with much smaller success rate.
Among all these attacks, we ﬁnd that CapsNets always perform better than convolutional networks.
Then, we diagnose the adversarial examples for CapsNets and ﬁnd that the success of the reconstructive attack is highly related to the visual similarity between the source and target class.
Additionally, the resulting perturbations can cause the input image to appear visually more like the target class and hence become non-adversarial.
This suggest that CapsNets use features that are more aligned with human perception and have the potential to address the central issue raised by adversarial examples.
I NTRODUCTION
Adversarial examples (Szegedy et al., 2013) are inputs that are designed by an adversary to cause a machine learning system to make a misclassiﬁcation.
A series of studies on adversarial attacks have shown that it is easy to cause misclassiﬁcations using visually imperceptible changes to an image under (cid:96) p -norm based similarity metrics (Goodfellow et al., 2014; Kurakin et al., 2016;
Madry et al., 2017; Carlini & Wagner, 2017b; Goodfellow et al., 2018).
Since the discovery of adversarial examples, there has been a constant “arms race” between better attacks and better defenses.
Many new defenses have been proposed (Song et al., 2017;
Gong et al., 2017; Grosse et al., 2017; Metzen et al., 2017), only to be broken shortly thereafter (Carlini & Wagner, 2017a;
Athalye et al., 2018).
Hinton et al.
(2018) showed that capsule models are more robust to simple adversarial attacks than
CNNs but Michels et al.
(2019) showed that this is not the case for all attacks.
The cycle of attacks and defenses motivates us to rethink both how we can improve the general robustness of neural networks as well as the high-level motivation for this pursuit.
One potential path forward is to detect adversarial inputs, instead of attempting to accurately classify them (Schott et al., 2018; Roth et al., 2019).
Recent work (Jetley et al., 2018; Gilmer et al., 2018b) argue that adversarial examples can exist within the data distribution, which implies that detecting adversarial examples based on an estimate of the data distribution alone might be insufﬁcient.
Instead, in this paper we develop methods for detecting adversarial examples by making use of class-conditional reconstruction networks .
These sub-networks, ﬁrst proposed by Sabour et al.
(2017) as part of a
Capsule Network (CapsNet), allow a model to produce a reconstruction of its input based on the identity and instantiation parameters of the winning capsule.
Interestingly, we ﬁnd that reconstructing
[Figure]
an input from the capsule corresponding to the correct class results in a much lower reconstruction error than reconstructing the input from capsules corresponding to incorrect classes, as shown in
Figure 1(a).
Motivated by this, we propose using the reconstruction sub-network in a CapsNet as an attack-independent detection mechanism.
Speciﬁcally, we reconstruct a given input from the pose parameters of the winning capsule and then detect adversarial examples by comparing the difference between the reconstruction distributions for natural and adversarial (or otherwise corrupted) images.
We extend this detection mechanism to standard convolutional neural networks (CNNs) and show its effectiveness against black box and white box attacks on three image datasets; MNIST, FashionMNIST and SVHN.
We show that capsule models achieve the strongest attack detection rates and accuracy on these attacks.
We then test our method against a stronger attack, the Reconstructive Attack, speciﬁcally designed to attack our detection mechanism by generating adversarial examples with a small reconstruction error.
With this attack we are able to create undetected adversarial examples, but we show that this attack is less successful in fooling the classiﬁer than a non-reconstructive attack.
Among all these attacks, we ﬁnd CapsNets perform the best in detecting adversarial examples.
To explain the success of CapsNets over CNNs, we further diagnose the adversarial examples for
CapsNets and ﬁnd that 1) the success of the targeted reconstructive attack is highly dependent on the visual similarity between the source image and the target class.
2) many of the resultant attacks resemble members of the target class and so cease to be “adversarial” – i.e., they may also be misclassiﬁed by humans.
These ﬁndings suggest that CapsNets with class conditional reconstructions have the potential to address the real issue with adversarial examples – networks should make predictions based on the same properties of the image that people use rather than using features that can be manipulated by an imperceptible adversarial attack.
In summary, our main contributions are:
• We propose a class-conditional capsule reconstruction based detection method to detect standard white-box/black-box adversarial examples on three datasets.
This detection mechanism is attack-agnostic and is successfully extended to standard convolutional neural networks.
• We test our detection mechanism on the corrupted MNIST dataset and show that it can work as a general out-of-distribution detector.
• A stronger reconstructive attack is speciﬁcally designed to attack our detection mechanism but becomes less successful in fooling the classiﬁer.
• We perform extensive qualitative studies to explain the superior performance of CapsNets in detecting adversarial examples compared to CNNs.
The results suggest that the features captured by CapsNets are more aligned with human perception.
R ELATED W ORK
Adversarial examples were ﬁrst introduced in (Biggio et al., 2013; Szegedy et al., 2013), where a given image was modiﬁed by following the gradient of a classiﬁer’s output with respect to the
) then developed the more method (FGSM), which can change the label of the input image X with a similarly imperceptible perturbation that is constructed by taking an (cid:15) step in the direction of the gradient.
Later, the Basic Iterative
Method (BIM) (Kurakin et al., 2016) and Projected Gradient Descent (Madry et al., 2017) can generate stronger attacks improved on FGSM by taking multiple steps in the direction of the gradient.
In addition, Carlini & Wagner (2017b) proposed another iterative optimization-based method to construct strong adversarial examples with small perturbations.
An early approach to reducing vulnerability to adversarial examples was proposed by (Goodfellow et al., 2014), where a network was trained on both clean images and adversarially perturbed ones.
Since then, there has been a constant “arms race” between better attacks and better defenses; Kurakin et al.
(2018) provide an overview of this ﬁeld.
However, many defenses against adversarial examples have been demonstrated to be an effect of “obfuscated gradients” and can be further circumvented under the white-box setting (Athalye et al., 2018).
Another line of work attempts to circumvent adversarial examples by detecting them with a separatelytrained classiﬁer (Gong et al., 2017; Grosse et al., 2017; Metzen et al., 2017) or using statistical properties (Hendrycks & Gimpel, 2016; Li & Li, 2017; Feinman et al., 2017; Grosse et al., 2017).
However, many of these approaches were subsequently shown to be ﬂawed (Carlini & Wagner, 2017a;
Athalye et al., 2018).
The most recent work in detecting adversarial examples (Roth et al., 2019) that has a 99% true positive rate on CIFAR-10 dataset (Krizhevsky, 2009) has also been fully bypassed by later work (Hosseini et al., 2019) which decreased the true positive rate to less than 2 % .
Similar to our work, Schott et al.
(2018) also investigated the effectiveness of a class-conditional generative model as a defense mechanism for MNIST digits.
However, we differ in some important ways.
Their model is in some ways the opposite of ours - they ﬁrst attempt to generate the input, and then make a classiﬁcation on the resulting generated images, whereas our method attempts to ﬁrst classify the input, making use of an otherwise unchanged capsule classiﬁcation model, and then generates the input from a high level representation.
As such, our method does not increase the computational overhead of classifying the input, compared to the approach of Schott et al.
In addition, the work of Schott et al.
(2018) is only applied to MNIST, so our results on the more complex datasets represent an improvement.
P RELIMINARIES
Adversarial Examples Given a clean test image x , its corresponding label y , and a classiﬁer f ( · ) which predicts a class label given an input, we refer to x (cid:48)
=
x + δ as an adversarial example if it is able to fool the classiﬁer into making a wrong prediction f
( x (cid:48) ) (cid:54) =
f ( x ) =
y .
The small adversarial perturbation δ (where “small” is measured under some norm) causes the adversarial example x (cid:48) to appear visually similar to the clean image x but to be classiﬁed differently.
In the unrestricted case where we only require that f ( x (cid:48) )
(cid:54) =
y , we refer to x (cid:48) as an “untargeted adversarial example”.
A more powerful attack is to generate a “targeted adversarial example”: instead of simply fooling the classiﬁer to make a wrong prediction, we force the classiﬁer to predict some targeted label f
( x (cid:48) )
=
t (cid:54) =
y .
In this paper, the target label t is selected uniformly at random as any label which is not the ground-truth correct label.
As is standard practice in the literature, in this paper we test our detection mechanism on three (cid:96) ∞ norm based attacks (fast gradient sign method (FGSM)
(Goodfellow et al., 2014), the basic iterative method (BIM) (Kurakin et al., 2016), projected gradient descent (PGD) (Madry et al., 2017)) and one (cid:96) 2 norm based attack (Carlini-Wagner (CW) (Carlini & Wagner, 2017b)).
Capsule Networks Capsule Networks (CapsNets) are an alternative architecture for neural networks (Sabour et al., 2017; Hinton et al., 2018).
In this work we make use of the CapsNet architecture detailed by (Sabour et al., 2017).
Unlike a standard neural network which is made up of layers of scalar-valued units, CapsNets are made up of layers of capsules, that output a vector or matrix.
Intuitively, just as one can think of the activation of a unit in a normal neural network as the presence of a feature in the input, the activation of a capsule can be thought of as both the presence of a feature and the pose parameters that represent attributes of that feature.
A top-level capsule in a classiﬁcation network therefore outputs both a classiﬁcation and pose parameters that represent the instance of that class in the input.
This high level representation allows us to train a reconstruction network.
Threat Model
In this paper, we test our detection mechanism against both white-box and black-box attacks.
For white-box attacks, the adversary has full access to the model as well as its parameters.
In particular, the adversary is allowed to compute the gradient through the model to generate adversarial examples.
To perform black-box attacks, the adversary is allowed to know the network architecture but not its parameters.
Therefore, we retrain a substitute model that has the same architecture as the target model and generate adversarial examples by attacking the substitute model.
Then we transfer these attacks to the target model.
For (cid:96) ∞ based attacks, we always control the (cid:96) ∞ norm of the adversarial perturbation to be within a relatively small bound (cid:15) ∞ , speciﬁc to each dataset.
D ETECTING A DVERSARIAL I MAGES BY R ECONSTRUCTION
To detect adversarial images, we make use of the reconstruction network proposed in (Sabour et al., 2017), which takes pose parameters v as input and outputs the reconstructed image r ( v ) .
The reconstruction network is simply a fully connected neural network with two ReLU hidden layers with 512 and 1024 units respectively, with a sigmoid output with the same dimensionality as the dataset.
The reconstruction network is trained to minimize the (cid:96) 2 distance between the input image and the reconstructed image.
This same network architecture is used for all the models and datasets we explore.
The only difference is what is given to the reconstruction network as input.
4. M ODELS
CapsNet The reconstruction network of the CapsNet is class-conditional : It takes in the pose parameters of all the class capsules and masks all values to 0 except for the pose parameters of the predicted class.
We use this reconstruction network for detecting adversarial attacks by measuring the Euclidean distance between the input and a class conditional reconstruction.
Speciﬁcally, for any given input x , the CapsNet outputs a prediction f ( x ) as well as the pose parameters v for all classes.
The reconstruction network takes in the pose parameters and then selects the pose parameter corresponding to the predicted class, denoted as v f ( x ) , to generate a reconstruction r ( v f ( x ) ) .
Then we compute the (cid:96) 2 reconstruction distance d ( x ) = (cid:107) r
( v f ( x ) ) , x (cid:107) 2 between the reconstructed image and the input image, and compare it with a pre-deﬁned detection threshold θ (described below in
Section 4.2).
If the reconstruction distance d ( x ) is higher than the detection threshold θ , we ﬂag the input as an adversarial example.
Figure 1 (b) shows an example of histograms of reconstruction distances for natural images and typical adversarial examples.
CNN+CR Although our strategy is inspired by the reconstruction networks used in CapsNets, the strategy can be extended to standard convolutional neural networks (CNNs).
We create a similar architecture, CNN with conditional reconstruction (CNN+CR), by dividing the penultimate hidden layer of a CNN into groups corresponding to each class.
The sum of each neuron group serves as the logit for that particular class and the group itself serves the same purpose as the pose parameters in the
CapsNet.
We use the same masking mechanism as Sabour et al.
(2017) to select the pose parameter corresponding to the predicted label v f ( x ) and generate the reconstruction based on the selected pose parameters.
In this way we extend the class-conditional reconstruction network to standard CNNs.
CNN+R We can also create a more na¨ıve implementation of our strategy by simply computing the reconstruction from the activations in the entire penultimate layer without any masking mechanism.
We call this model the ”CNN+R” model.
In this way we are able to study the effect of conditioning on the predicted class.
4. D ETECTION T HRESHOLD
We ﬁnd the threshold θ for detecting adversarial inputs by measuring the reconstruction error between a validation input image and its reconstruction.
If the distance between the input and the reconstruction is above the chosen threshold θ , we classify the data as adversarial.
Choosing the detection threshold θ involves a trade-off between false positive and false negative detection rates.
The optimal threshold depends on the probability of the system being attacked.
Such a trade-off is discussed by Gilmer et al.
(2018a).
In our experiments we don’t tune this parameter and simply set it as the 95th percentile of validation distances.
This means our false positive rate on real validation data is 5% .
[Table]
4. E VALUATION M ETRICS
We use Success Rate to measure the success of attacks.
For targeted attacks, the success rate S t is deﬁned as the proportion of inputs which are classiﬁed as the target class, S t = 1 N (cid:80) Ni ( f ( x (cid:48) i ) =
t i ) , while the success rate for untargeted attacks is deﬁned as the proportion of inputs which are misclassiﬁed, S u
= 1 N (cid:80) Ni ( f ( x (cid:48) i ) (cid:54) =
y i ) .
Previous work (Carlini & Wagner, 2017a; Hosseini et al., 2019) used the True Positive Rate to measure the proportion of adversarial examples that are detected, which alone is insufﬁcient to measure the ability of different detection mechanism because the unsuccessful adversarial examples do not have to be detected.
Therefore, in this paper, we propose to use the Undetected Rate : the proportion of attacks that are successful and undetected to evaluate the detection mechanism.
For targeted attacks, the undetected rate is deﬁned as R t = 1 N (cid:80) Ni ( f ( x (cid:48) i ) =
t i ) ∩
( d ( x (cid:48) i ) ≤ θ ) , where d ( · ) computes the reconstruction distance of the input and θ denotes the detection threshold introduced in section 4.2.
Similarly, the undetected rate for untargeted attacks R u can be deﬁned as R u = 1 N (cid:80) Ni ( f ( x (cid:48) i ) (cid:54) =
y i ) ∩
( d ( x (cid:48) i ) ≤ θ ) .
The smaller the undetected rate R t or R u is, the stronger the model is in detecting adversarial examples.
The undetected rate can also be used to evaluate the attacks (higher is better).
We also plot the Undetected
Rate vs. False Positive Rate curve to compare the detection performance between different models, where False Positive Rate is deﬁned as the proportion of clean examples that are misclassiﬁed as the adversarial example by the detection method.
4. T EST M ODELS AND D ATASETS
In all experiments, all three models (CapsNet, CNN+R, and CNN+CR) have the same number of parameters and were trained with Adam (Kingma & Ba, 2014) for the same number of epochs.
In general, all models achieved similar test accuracy.
We did not do an exhaustive hyperparameter search on these models, instead we chose hyperparameters that allowed each model to perform roughly equivalently on the test sets.
We run experiments on three datasets: MNIST (LeCun et al., 1998),
FashionMNIST (Xiao et al., 2017), and SVHN (Netzer et al., 2011).
The test error rate for each model on these three datasets, as well as details of the model architectures, can be seen in Section A and Section B in the Appendix.
E XPERIMENTS
We ﬁrst demonstrate how reconstruction networks can detect standard white and black-box attacks in addition to naturally corrupted images.
Then, we introduce the “reconstructive attack”, which is speciﬁcally designed to circumvent our defense and show that it is a more powerful attack in this setting.
Based on this ﬁnding, we qualitatively study the kind of misclassiﬁcations caused by the reconstructive attack and argue that they suggest that CapsNets learn features that are better aligned with human perception.
5. S TANDARD A TTACKS
White Box We present the success and undetected rates for several targeted and untargeted attacks on MNIST (Table 1), FashionMNIST, and SVHN (Table 5 presented in the Appendix).
Our method
[Table]
CNN+R 73.4/49. is able to accurately detect many attacks with very low undetected rates.
Capsule models almost always have the lowest undetected rates out of our three models.
It is worth noting that this method performs best with the simplest dataset, MNIST, and that the highest undetected rates are found with the Carlini-Wagner attack on the SVHN dataset.
This illustrates both the strength of this attack and a shortcoming of our defense, namely that our detection mechanism relies on (cid:96) 2 image distance as a proxy for visual similarity, and in the case of higher dimensional color datasets such as SVHN, this proxy is less meaningful.
Black Box We also tested our detection mechanism results on black box attacks.
Given the low undetected rates in the white-box settings, it is not surprising that our detection method is able to detect black box attacks as well.
In fact, on the MNIST dataset the capsule model is able to detect all targeted and untargeted PGD attacks.
Both the CNN-R and the CNN-CR models are able to detect the black box attacks as well, but with a relatively higher undetected rate.
A table of these results can be seen in Table 7 in the Appendix.
5. C ORRUPTION A TTACKS
Recent work has argued that improving the robustness of neural networks to (cid:96) p norm bounded adversarial attacks should not come at the expense of increasing error rates under distributional shifts that do not affect human classiﬁcation rates and are likely to be encountered in the “realworld”
(Gilmer et al., 2018a).
For example, if an image is corrupted due to adverse weather, lighting, or occlusion, we might hope that our model can continue to provide reliable predictions or detect the distributional shift.
We can test our detection method on its ability to detect these distributional shifts by making use of the Corrupted MNIST dataset (Mu & Gilmer, 2019).
This data set contains many visual transformations of MNIST that do not seem to affect human performance, but nevertheless are strongly misclassiﬁed by state-of-the-art MNIST models.
Our three models can almost always detect these distributional shifts (in all corruptions CapsNets have either a small undetected rate or an undetected rate of 0).
The error rate (the proportion of misclassiﬁed input) and undetected rate of three test models on the Corrupted MNIST dataset is shown in Table 2.
Please refer to Figure 7 and
Figure 8 in the Appendix for visualization of Corrupted MNIST.
5. R ECONSTRUCTIVE A TTACKS
Thus far we have only evaluated previously-deﬁned attacks.
Following the suggestion in (Carlini &
Wagner, 2017a) that detection methods need to show effectiveness towards defense-aware attacks, we introduce an attack speciﬁcally designed to take into account our defense mechanism.
In order to construct adversarial examples that cannot be detected by the network, we propose a two-stage optimization method to generate a “reconstructive attack”.
[Table]
Untargeted Reconstructive Attacks To construct untargeted reconstructive attacks, we ﬁrst update the perturbation based on the gradient of the cross-entropy loss function following a standard FGSM attack (Goodfellow et al., 2014), that is:
[Equation]
where (cid:96) net ( f ( · ) , y ) is the cross-entropy loss function, (cid:15) is the (cid:96) ∞ bound for our attacks, c is a hyperparameter controlling the step size in each iteration and β is a hyperparameter which balances the importance of the cross-entropy loss and the reconstruction loss (explained further below).
In the second stage, we focus on constraining the reconstructed image from the newly predicted label to have a small reconstruction distance by updating δ according to
[Equation]
where r ( v f ( x + δ ) ) is the class-conditional reconstruction based on the predicted label f
( x + δ ) in a
CapsNet or CNN+CR network.
The δ used here is the optimized δ from the ﬁrst stage.
(cid:107) r
( v f ( x + δ ) ) − ( x + δ ) (cid:107) 2 is the (cid:96) 2 reconstruction distance between the reconstructed image and the input image.
Since the CNN+R network does not use the class conditional reconstruction, we simply use the reconstructed image without the masking mechanism.
According to Eqn 1 and Eqn 2, we can see that β balances the importance between the success rate of attacks and the reconstruction distance.
This hyperparameter was tuned for each model and each dataset in order to create the strongest attacks.
The success rate and undetected rate change as this parameter, which is shown in Figure 9 in
Appendix.
Targeted Reconstructive Attacks We perform a similar two-stage optimization to construct targeted reconstructive attacks, by deﬁning a target label and attempting to maximize the classiﬁcation probability of this label, and minimize the reconstruction error from corresponding capsule.
Because the targeted label is given, another way to construct targeted reconstructive attacks is to combine these two stages into one stage via minimizing the loss function (cid:96) =
β · (cid:96) net ( f ( x + δ ) , y )+(1 − β ) ·(cid:107) r
[Equation]
v f
[Equation]
x +
[Equation]
) ) − ( x + δ )
[Equation]
We implemented both of these targeted reconstructive attacks and found that the two-stage version is a stronger attack.
Therefore, all the
Reconstructive Attack experiments performed in this paper are based on two-stage optimization.
We build our reconstructive attack based on the standard PGD attack, denoted as R-PGD, and test the performance of our detection models against this reconstructive attack in a white-box setting (white-box Reconstructive FGSM and BIM are reported in Table 6 in the Appendix).
Comparing
Table 1 and Table 3, we can see that the Reconstructive Attack is signiﬁcantly less successful at changing the models prediction (lower success rates than the standard attack).
However, this attack is more successful at fooling our detection method.
For all attacks and datasets the capsule model has the lowest attack success rate and the lowest undetected rate.
We report results for black-box R-PGD attacks in Table 7 in the Appendix, which suggest similar conclusions.
In addition, we report the undetected rate of the white-box targeted defense-aware R-PGD attack versus the False Positive Rate on the MNIST, Fashion-MNIST and SVHN datasets in Figure 2.
We can clearly see that the undetected rate of the defense-aware attack against CapsNet is signiﬁcantly smaller than the CNN-based networks, which suggests that CapsNets are more robust against adversarial attacks.
Furthermore, CNN with class-conditional reconstruction (CNN+CR) has smaller undetected rate at the same False Positive Rate compared to the CNN without class-conditional reconstruction (CNN+R), which suggests the class-conditional information is helpful in our models to improve the robustness against adversarial attacks.
5. CIFAR-10 D ATASET
In order to show that our method based on CapsNet is capable to scale up to more complex datasets, we test our detection method with a deeper reconstruction network on CIFAR-10 (Krizhevsky, 2009).
The classiﬁcation accuracy on the clean test dataset is 92 .
In addition, we display the undetected rate of the white-box/black-box defense-aware R-PGD attack against CapsNets versus the False
Positive Rate in Figure 3 (Left), where we can see a signiﬁcant drop of the undetected rate of black-box R-PGD compared to the white-box setting.
This indicates the CapsNets greatly reduce the attack transferability and the threat of black-box attacks.
Class-conditional Information To investigate the effectiveness of the class-conditional information in the reconstruction network
, we compare our CapsNet based on (Sabour et al., 2017) with the other two variants of CapsNets: “CapsNet All” and “DeepCaps” (Rajasegaran et al., 2019).
In “CapsNet All”, we remove the masking mechanism in the CapsNet and use all the capsules to do the reconstruction.
In “DeepCaps”, we extract the winning-capsule information as a single vector and used it as the input for the reconstruction network instead of using a masking mechanism to mask out the losing capsules information.
In this way, the class information in DeepCaps is more explicitly fed into the reconstruction network.
As shown in Figure 3 (right), our CapsNet has the best detection performance (the lowest undetected rate at the same False Positive Rate) compared to the other two
We can see that for the capsule model, many of the attacks are not “adversarial” as they resemble members the target
Capsule models.
“DeepCaps” performs slightly worse than our “CapsNet” and “CapsNet All” has the worst detection performance.
Therefore, we conclude that the class-conditional information used in the reconstruction network increases the model’s robustness to adversarial attack.
This also holds true to CNN-based networks because CNN+CR has a better detection performance than CNN+R, shown in Figure 2.
V ISUAL C OHERENCE OF THE R ECONSTRUCTIVE A TTACK
The great success of CapsNet over CNN-based models motivates us to further diagnose the generated adversarial examples for CapsNets.
If our true aim in adversarial robustness research is to create models that make predictions based on reasonable and human-observable features, then we would prefer models that are more likely to misclassify a “shirt” as a “t-shirt” (in the case of FashionMNIST) than to misclassify a “bag” as a “sweater”.
For a model to behave ideally, the success of an adversarial perturbation would be related to the visual similarity between the source and the target class.
By visualizing a matrix of adversarial success rates between each pair of classes (shown in Figure 4), we can see that for the capsule model there is a great variance between the source and target class pairs and that the success rate of attacks is highly related to the visual similarity of the classes.
However, this is not the case for either of the other two CNN-based models.
Thus far we have treated all attacks as equal.
However, a key component of an adversarial example is that it is visually similar to the source image, and that it does not resemble the adversarial target class.
The adversarial research community makes use of a small epsilon bound as a mechanism for ensuring that the resultant adversarial attacks are visually unchanged from the source image.
For standard attacks against CNN-based models this heuristic is sufﬁcient, because taking gradient steps in the image space in order to have a network misclassify an image normally results in something visually similar to the source image.
But this is not the case for adversarial attacks against CapsNets.
As shown in Figure 5, when we use R-PGD to attack the CapsNet, many of the resultant attacks resemble members of the target class.
In this way, they stop being “adversarial”.
As such, an attack detection method which does not detect them as adversarial is arguably behaving correctly.
This puts the previously undetected rates presented earlier in a new light, and illustrates a difﬁculty in the evaluation of adversarial attacks and defenses.
In addition, it should be noted that this phenomenon rarely occurs in a standard convolutional neural network, which suggests that the features captured by CapsNet are more aligned with human perception.
D ISCUSSION
Our detection mechanism relies on a similarity metric (i.e. a measure of reconstruction error) between the reconstruction and the input.
This metric is required both during training in order to train the reconstruction network and during test time in order to ﬂag adversarial examples.
In the four datasets we have evaluated, the distance between examples roughly correlates with semantic similarity.
However, this may not be the case for images in more complex datasets such as the SUN dataset (Xiao et al., 2010) and ImageNet (Deng et al., 2009), in which two images may be similar in terms of semantic content but nevertheless have signiﬁcant (cid:96) 2 distance.
A better similarity metric (Theis et al., 2015; Zhang et al., 2018) can be further explored to extend our methods to more complex problems.
Furthermore our reconstruction network is trained on a hidden representation of one class but is trained to reconstruct the entire input.
In datasets without distractors or backgrounds, this is not a problem.
But in the case of ImageNet, in which the object responsible for the classiﬁcation is not the only object in the image, attempting to reconstruct the entire input from a class encoding seems misguided.
C ONCLUSION
We have presented a class-conditional reconstruction-based detection method that does not rely on a speciﬁc predeﬁned adversarial attack.
We have shown that by reconstructing the input from the internal class-conditional representation, our system is able to accurately detect black-box and white-box FGSM, BIM, PGD, and CW attacks.
We then proposed a new attack to beat our defense - the Reconstructive Attack - in which the adversary optimizes not only the classiﬁcation loss but also minimizes the reconstruction loss.
We showed that this attack was able to fool our detection mechanism but with a much smaller success rate than a standard attack.
Compared to CNN-based models, we showed that the CapsNet was able to detect adversarial examples with greater accuracy on all the datasets we explored.
To further explain the success of CapsNet, we qualitatively showed that the success of the reconstructive attack was highly related to the visual similarity between the target class and the source class for the CapsNet.
In addition, we showed that images generated by this reconstructive attack to attack the CapsNet are not typically adversarial, i.e. many of the resultant attacks resemble members of the target class even with a small (cid:96) ∞ norm bound.
These are not the case for the CNN-based models.
The extensive qualitative studies indicate that the capsule model relies on visual features similar to those used by humans.
We believe this is a step towards solving the true problem posed by adversarial examples.
A PPENDIX A N ETWORK A RCHITECTURES
Figure 6 shows the architecute of the capsule network, the CNN reconstruction model and the CNN conditional reconstruction model used for experiments on MNIST, FashionMNIST and SVHN dataset.
MNIST and Fashion MNIST have exactly the same architectures while we use larger models for
SVHN.
Note that the only difference between the CNN reconstruction (CNN+R) and the CNN conditional reconstruction (CNN+CR) is the masking procedure on the input to the reconstruction network based on the predicted class.
All three models have the same number of parameters.
B T EST MODELS
The error rate of each test model used in the paper are presented in Table 4.
We ensure that they have similar performance.
[Table]
C I MPLEMENTATION D ETAILS
For all the (cid:96) ∞ based adversarial examples, the (cid:96) ∞ norm of the perturbations is bound by (cid:15) , which is set to 0.3, 0.1, 0.1 for MNIST, Fashion MNIST and SVHN dataset respectively following previous work (Madry et al., 2017; Song et al., 2017).
In FGSM based attacks, the step size c is 0.05.
In
BIM-based (Kurakin et al., 2016) and PGD-based (Madry et al., 2017) attacks, the step size c is 0 .for
all the datasets and the number of iterations are 1000, 500 and 200 for MNIST, Fashion MNIST and SVHN dataset respectively.
We choose a sufﬁciently large number of iterations to ensure the attacks has converged.
We use the publicly released code from the authors of (Carlini & Wagner, 2017b) to perform the CW attack for our models.
The number of iterations are set to 1000 for all three datasets.
D W HITE B OX S TANDARD A TTACKS
The results of four white box standard attacks on the three datasets are shown in Table 5.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
SVHN
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
E V ISUALIZATION OF C ORRUPTED MNIST D ATASET
Visualization of examples from Corrupted MNIST dataset (Mu & Gilmer, 2019) and the corresponding reconstructed images for each model are shown in Figure 7 and Figure 8.
Clean
CapsNet
Gaussian Noise
CNN+R
CNN+CR
CapsNet
CNN+R
CNN+CR
Gaussian Blur
CapsNet
CNN+R
CNN+CR
Line
CapsNet
Dotted Line
CNN+R
CNN+CR
CapsNet
CNN+R
CNN+CR
CapsNet
CNN+R
CNN+CR
Elastic Transform
Saturate
CapsNet
JPEG
CNN+R
CNN+CR
CapsNet
CNN+R
CNN+CR
Quantize
CapsNet
CNN+R
CNN+CR
Sheer
CapsNet
Spatter
CNN+R
CNN+CR
CapsNet
CNN+R
CNN+CR
CapsNet
CNN+R
CNN+CR
Rotate
[Figure]
CapsNet
CNN
[Figure]
R
CNN
[Figure]
CR
Fog
CapsNet
Frost
CNN+R
CNN+CR
CapsNet
CNN+R
CNN+CR
CapsNet
CNN+R
CNN+CR
Zigzag
F R ECONSTRUCTIVE A TTACKS
The results of Reconstructive FGSM, BIM and PGD on the three datasets are reported in Table
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Figure]
Figure 9 shows the plot of success rate and undetected rate versus the hyperparameter β which balances the importance between attacking the classiﬁer and fooling the detection mechanism in the targeted reconstructive PGD attacks on the MNIST dataset.
[Figure]
G B LACK B OX A TTACKS
[Table]
H V ISUALIZATION OF A DVERSARIAL E XAMPLES AND R ECONSTRUCTIONS
[Figure]
PGD
[Figure]
Recons:
PGD:
[Figure]
Recons:
R-PGD:
Recons:
R-PGD:
Recons:
R-
[Table] [Figure]