P IXEL D EFEND : L EVERAGING G ENERATIVE M ODELS TO U NDERSTAND AND D EFEND AGAINST A DVERSARIAL E XAMPLES
Anonymous authors
Paper under double-blind review
A BSTRACT
Adversarial perturbations of normal images are usually imperceptible to humans, but they can seriously confuse state-of-the-art machine learning models.
What makes them so special in the eyes of image classiﬁers?
In this paper, we show empirically that adversarial examples mainly lie in the low probability regions of the training distribution, regardless of attack types and targeted models.
Using statistical hypothesis testing, we ﬁnd that modern neural density models are surprisingly good at detecting imperceptible image perturbations.
Based on this discovery, we devised PixelDefend , a new approach that puriﬁes a maliciously perturbed image by moving it back towards the distribution seen in the training data.
The puriﬁed image is then run through an unmodiﬁed classiﬁer, making our method agnostic to both the classiﬁer and the attacking method.
As a result, PixelDefend can be used to protect already deployed models and be combined with other model-speciﬁc defenses.
Experiments show that our method greatly improves resilience across a wide variety of state-of-the-art attacking methods, increasing accuracy on the strongest attack from 63% to 84% for Fashion MNIST and from 32% to 70% for CIFAR-10.
I NTRODUCTION
Recent work has shown that small, carefully chosen modiﬁcations to the inputs of a neural network classiﬁer can cause the model to give incorrect labels (Szegedy et al., 2013; Goodfellow et al., 2014).
This weakness of neural network models is particularly surprising because the modiﬁcations required are often imperceptible, or barely perceptible, to humans.
As deep neural networks are being deployed in safety-critical applications such as self-driving cars (Amodei et al., 2016), it becomes increasingly important to develop techniques to handle these kinds of inputs.
Rethinking adversarial examples The existence of such adversarial examples seems quite surprising.
A neural network classiﬁer can get super-human performance (He et al., 2015) on clean test images, but will give embarrassingly wrong predictions on the same set of images if some imperceptible noise is added.
What makes this noise so special to deep neural networks?
In this paper, we propose and empirically evaluate the following hypothesis: Even though they have very small deviations from clean images, adversarial examples largely lie in the low probability regions of the distribution that generated the data used to train the model.
Therefore, they fool classiﬁers mainly due to covariate shift.
This is analogous to training models on MNIST (LeCun et al., 1998) but testing them on Street View House Numbers (Netzer et al., 2011).
To study this hypothesis, we ﬁrst need to estimate the probability density of the underlying training distribution.
To this end, we leverage recent developments in generative models.
Speciﬁcally, we choose a PixelCNN (van den Oord et al., 2016b) model for its state-of-the-art performance in modeling image distributions (van den Oord et al., 2016a; Salimans et al., 2017) and tractability of evaluating the data likelihood.
In the ﬁrst part of the paper, we show that a well-trained PixelCNN generative model is very sensitive to adversarial inputs, typically giving them several orders of magnitude lower likelihoods compared to those of training and test images.
Detecting adversarial examples An important step towards handling adversarial images is the ability to detect them.
In order to catch any kind of threat, existing work has utilized conﬁdence estimates from Bayesian neural networks (BNNs) or dropout (Li & Gal, 2017; Feinman et al., 2017).
However, if their model is misspeciﬁed, the uncertainty estimates can be affected by covariate shift (Shimodaira, 2000).
This is problematic in an adversarial setting, since the attacker might be able to make use of the inductive bias from the misspeciﬁed classiﬁer to bypass the detection.
Protection against the strongest adversary requires a pessimistic perspective—our assumption is that the classiﬁer cannot give reliable predictions for any input outside of the training distribution.
Therefore, instead of relying on label uncertainties given by the classiﬁer, we leverage statistical hypothesis testing to detect any input not drawn from the same distribution as training images.
Speciﬁcally, we ﬁrst compute the probabilities of all training images under the generative model.
Afterwards, for a novel input we compute the probability density at the input and evaluate its rank (in ascending order) among the density values of all training examples.
Next, the rank can be used as a test statistic and gives us a p -value for whether or not the image was drawn from the training distribution.
This method is general and practical and we show that the p -value enables us to detect adversarial images across a large number of different attacking methods with high probability, even when they differ from clean images by only a few pixel values.
Purifying adversarial examples Since adversarial examples are generated from clean images by adding imperceptible perturbations, it is possible to decontaminate them by searching for more probable images within a small distance of the original ones.
By limiting the L 1 distance 1 , this image puriﬁcation procedure generates only imperceptible modiﬁcations to the original input, so that the true labels of the puriﬁed images remain the same.
The resulting puriﬁed images have higher probability under the training distribution, so we can expect that a classiﬁer trained on the clean images will have more reliable predictions on the puriﬁed images.
Moreover, for inputs which are not corrupted by adversarial perturbations the puriﬁed results remain in a high density region.
We use this intuition to build PixelDefend , an image puriﬁcation procedure which requires no knowledge of the attack nor the targeted classiﬁer.
PixelDefend approximates the training distribution using a PixelCNN model.
The constrained optimization problem of ﬁnding the highest probability image within an ϵ -ball of the original is computationally intractable, however, so we approximate it using a greedy decoding procedure.
Since PixelDefend does not change the classiﬁcation model, it can be combined with other adversarial defense techniques, including adversarial training (Goodfellow et al., 2014), to provide synergistic improvements.
We show experimentally that PixelDefend performs exceptionally well in practice, leading to state-of-the art results against a large number of attacks, especially when combined with adversarial training.
Contributions Our main contributions are as follows:
(cid:15) We show that generative models can be used for detecting adversarially perturbed images and observe that most adversarial examples lie in low probability regions. (cid:15)
We introduce a novel family of methods for defending against adversarial attacks based on the idea of puriﬁcation.
(cid:15) We show that a defensive technique from this family, PixelDefend, can achieve state-of-theart results on a large number of attacking techniques, improving the accuracy against the strongest adversary on the CIFAR-10 dataset from 32% to 70%.
B ACKGROUND 2. A TTACKING METHODS
Given a test image X , an attacking method tries to ﬁnd a small perturbation ∆ with ∥ ∆ ∥ 1 (cid:20) ϵ attack , such that a classiﬁer f gives different predictions on X adv ≜ X + ∆ and X .
Here colors in the image are represented by integers from 0 to 255.
Each attack method is controlled by a conﬁgurable ϵ attack parameter which sets the maximum perturbation allowed for each pixel in integer increments on the color scale.
We only consider white-box attacks in this paper, i.e ., the attack methods can get access to weights of the classiﬁer.
In the following, we give an introduction to all the attacking methods used in our experiments.
Random perturbation (RAND)
Random perturbation is arguably the weakest attacking method, and we include it as the simplest baseline.
Formally, the randomly perturbed image is given by
[Equation]
where U ( a; b ) denotes an element-wise uniform distribution of integers from [ a; b ] .
Fast gradient sign method (FGSM)
Goodfellow et al.
(2014) proposed the generation of malicious perturbations in the direction of the loss gradient ∇ X L ( X ; y ) , where L ( X ; y ) is the loss function used to train the model.
The adversarial examples are computed by
[Equation]
Basic iterative method (BIM) Kurakin et al.
(2016) tested a simple variant of the fast gradient sign method by applying it multiple times with a smaller step size.
Formally, the adversarial examples are computed as
[Equation]
where Clip ϵ attack X means we clip the resulting image to be within the ϵ attack -ball of X .
Following
Kurakin et al.
(2016), we set (cid:11) = and the number of iterations to be ⌊ min( ϵ attack + 4 ; 1 : 25 ϵ attack ) ⌋ .
This method is also called Projected Gradient Descent (PGD) in Madry et al.
DeepFool DeepFool (Moosavi-Dezfooli et al., 2016) works by iteratively linearizing the decision boundary and ﬁnding the closest adversarial examples with geometric formulas.
However, compared to FGSM and BIM, this method is much slower in practice.
We clip the resulting image so that its perturbation is no larger than ϵ attack .
Carlini-Wagner (CW) Carlini & Wagner (2017b) proposed an efﬁcient optimization objective for iteratively ﬁnding the adversarial examples with the smallest perturbations.
As with DeepFool, we clip the output image to make sure the perturbations are limited by ϵ attack .
2. D EFENSE METHODS
Current defense methods generally fall into two classes.
They either (1) change the network architecture or training procedure to make it more robust, or (2) modify adversarial examples to reduce their harm.
In this paper, we take the following defense methods into comparison.
Adversarial training This defense works by generating adversarial examples on-the-ﬂy during training and including them into the training set.
FGSM adversarial examples are the most commonly used ones for adversarial training, since they are fast to generate and easy to train.
Although training with higher-order adversarial examples ( e.g ., BIM) has witnessed some success in small datasets (Madry et al., 2017), other work has reported failure in larger ones (Kurakin et al., 2016).
We consider both variants in our work.
Label smoothing In contrast to adversarial training, label smoothing (Warde-Farley & Goodfellow, 2016) is agnostic to the attack method.
It converts one-hot labels to soft targets, where the correct class has value 1 (cid:0) ϵ while the other (wrong) classes have value ϵ= ( N (cid:0) 1) .
Here ϵ is a small constant and N is the number of classes.
When the classiﬁer is re-trained on these soft targets rather than the one-hot labels it is signiﬁcantly more robust to adversarial examples.
This method was originally devised to achieve a similar effect as defensive distillation (Papernot et al., 2016c), and their performance is comparable.
We didn’t compare to defensive distillation since it is more computationally expensive.
Feature squeezing Feature squeezing (Xu et al., 2017a) is both attack-agnostic and modelagnostic.
Given any input image, it ﬁrst reduces the color range from [0 ; 255] to a smaller value, and then smooths the image with a median ﬁlter.
The resulting image is then passed to a classiﬁer for predictions.
Since this technique does not depend on attacking methods and classiﬁers, it can be combined with other defensive methods such as adversarial training, similar to PixelDefend.
2. E XPERIMENT METHODOLOGIES
Datasets Two datasets are used in our experiments: Fashion MNIST (Xiao et al., 2017) and CIFAR-10 (Krizhevsky et al.).
Fashion MNIST was designed as a more difﬁcult, but drop-in replacement for MNIST (LeCun et al., 1998).
Thus it shares all of MNIST’s characteristics, i.e ., 60 ;training examples and 10 ; 000 test examples where each example is a 28 (cid:2) 28 gray-scale image associated with a label from 1 of 10 classes.
CIFAR-10 is another dataset that is also broadly used for image classiﬁcation tasks.
It consists of 60 ; 000 examples, where 50 ; 000 are used for training and 10 ; 000 for testing, and each sample is a 32 (cid:2) 32 color image associated with 1 of 10 classes
Models We examine two state-of-the-art deep neural network image classiﬁers: ResNet (He et al., 2016) and VGG (Simonyan & Zisserman, 2014).
The architectures are described in Appendix C.
PixelCNN The PixelCNN (van den Oord et al., 2016b; Salimans et al., 2017) is a generative model with tractable likelihood especially designed for images.
The model deﬁnes the joint distribution over all pixels by factorizing it into a product of conditional distributions.
[Equation]
The pixel dependencies are in raster scan order (row by row and column by column within each row).
We train the PixelCNN model for each dataset using only clean (not perturbed) image samples.
In
Appendix D, we provide clean sample images from the datasets as well as generated image samples from PixelCNN (see Figure 8 and Figure 9).
As a convenient representation of p CNN ( X ) for images, we also use the concept of bits per dimension , which is deﬁned as BPD( X ) ≜ (cid:0) log p CNN ( X )
= ( I (cid:2) J (cid:2) K (cid:2) log 2) for an image of resolution
I (cid:2) J and K channels.
D ETECTING ADVERSARIAL EXAMPLES
Adversarial images are deﬁned with respect to a speciﬁc classiﬁer.
Intuitively, a maliciously perturbed image that causes one network to give a highly conﬁdent incorrect prediction might not fool another network.
However, recent work (Papernot et al., 2016a; Liu et al., 2016; Tramèr et al., 2017) has shown that adversarial images can transfer across different classiﬁers.
This indicates that there are some intrinsic properties of adversarial examples that are independent of classiﬁers.
One possibility is that, compared to normal training and test images, adversarial examples have much lower probability densities under the image distribution.
As a result, classiﬁers do not have enough training instances to get familiarized with this part of the input space.
The resulting prediction task suffers from covariate shift, and since all of the classiﬁers are trained on the same dataset, this covariate shift will affect all of them similarly and will likely lead to misclassiﬁcations.
To empirically verify this hypothesis, we train a PixelCNN model on the CIFAR-10 (Krizhevsky &
Hinton, 2009) dataset and use its log-likelihood as an approximation to the true underlying probability density.
The adversarial examples are generated with respect to a ResNet (He et al., 2016), which gets 92% accuracy on the test images.
We generate adversarial examples from RAND, FGSM, BIM,
DeepFool and CW methods with ϵ attack = 8 .
Note that as shown in Figure 1, the resulting adversarial perterbations are barely perceptible to humans.
[Figure] [Figure]
However, the distribution of log-likelihoods show considerable difference between perturbed images and clean images.
As summarized in Figure 2, even a 3% perturbation can lead to systematic decrease of log-likelihoods.
Note that the PixelCNN model has no information about the attacking methods for producing those adversarial examples, and no information about the ResNet model either.
We can see from Figure 3(b) that random perturbations also push the images outside of the training distribution, even though they do not have the same adverse effect on accuracy.
We believe this is due to an inductive bias that is shared by many neural network models but not inherent to all models, as discussed further in Appendix A.
Besides qualitative analysis, the log-likelihoods from PixelCNN also provide a quantitative measure for detecting adversarial examples.
Combined with permutation test (Efron & Tibshirani, 1994), we can provide a uncertainty value for each input about whether it comes from the training distribution or not.
Speciﬁcally, let the input X ′ i.i.d.
(cid:24) q ( X ) and training images X 1 ; (cid:1) (cid:1) (cid:1) ;
X N i.i.d.
(cid:24) p ( X ) .
The null hypothesis is H 0 : p ( X ) =
q ( X ) while the alternative is H 1 : p ( X ) ̸
= q ( X ) .
We ﬁrst compute the probabilities give by a PixelCNN for X ′ and X 1 ; (cid:1)
[Equation]
Here I [ (cid:1) ] is the indicator function, which equals 1 when the condition inside brackets is true and otherwise equals 0.
Let T i
=
T ( X i ; X 1 ; (cid:1) (cid:1) (cid:1) ; X i (cid:0) 1 ; X ′ ; X i +1 ; (cid:1) (cid:1) (cid:1) ; X N ) .
According to the permutation principle, T i has the same distribution as T under the null hypothesis H 0 .
We can therefore compute the p -value exactly by
[Equation]
For CIFAR-10, we provide histograms of p -values for different adversarial examples in Figure 3 and ROC curves of using p -values for detection in Figure 6(a).
Note that in the ideal case, the p -value distribution of clean test images should be uniform.
The method works especially well for attacks producing larger perturbations, such as RAND, FGSM, and BIM.
For DeepFool and CW adversarial examples, we can also observe signiﬁcant deviations from uniform.
As shown in Figure 3(a), the pvalue distribution of test images are almost uniform, indicating good generalization of the PixelCNN model.
P URIFYING IMAGES WITH P IXEL D EFEND
In many circumstances, simply detecting adversarial images is not sufﬁcient.
It is often critical to be able to correctly classify images despite such adversarial modiﬁcations.
In this section we introduce
PixelDefend, a speciﬁc instance of a new family of defense methods that signiﬁcantly improves the state-of-the-art performance on advanced attacks, while simultaneously performing well against all other attacks.
4. R ETURNING IMAGES TO THE TRAINING DISTRIBUTION
The basic idea behind PixelDefend is to purify input images, by making small changes to them in order to move them back towards the training distribution, i.e ., move the images towards a highprobability region.
We then classify the puriﬁed image using any existing classiﬁer.
As the example in Figure 4 shows, the puriﬁed images can usually be classiﬁed correctly.
Formally, we have training image distribution p ( X ) , and input image X of resolution I (cid:2) J with
X [ i; j; k ] the pixel at location ( i; j ) and channel k 2 f 1 ; (cid:1) (cid:1) (cid:1) ; C g .
We wish to ﬁnd an image X (cid:3) that maximizes p ( X ) subject to the constraint that X (cid:3) is within the ϵ defend -ball of X :
[Equation]
Input: Image X , Defense parameter ϵ defend , Pre-trained PixelCNN model p CNN
Output: Puriﬁed Image X (cid:3) 1: X (cid:3) X 2: for each row i do 3: for each column j do 4: for each channel k do 5: x
X [ i; j; k ] 6:
Set feasible range R [max( x (cid:0) ϵ defend ; 0) ; min( x + ϵ defend ; 255)] 7:
Compute the 256-way softmax p CNN ( X (cid:3) ) .
Update X (cid:3) [ i; j; k ] arg max z 2 R p CNN [ i; j; k; z ] 9: end for 10: end for 11: end for
We tested two puriﬁcation methods, L-BFGS-B and greedy decoding, the latter of which is used in
PixelDefend.
A good puriﬁcation method should give images that have lower bits per dimension compared to FGSM images and ideally similar bits per dimension compared to clean ones.
Here ϵ defend reﬂects a trade-off, since large ϵ defend may change the meaning of X while small ϵ defend may not be sufﬁcient for returning X to the correct distribution.
In practice, we choose ϵ defend to be some value that overestimates ϵ attack but still keeps high accuracies on clean images.
As in Section 3, we approximate p ( X ) with the PixelCNN distribution p CNN ( X ) , which is trained on the same training set as the classiﬁer.
However, exact constrained optimization of p CNN ( X ) is computationally intractable.
Surprisingly, even gradient-based optimization faces great difﬁculty on that problem.
We found that one advanced methods in gradient-based constrained optimization, L-BFGS-B (Byrd et al., 1995) (we use the scipy implementation based on Zhu et al.
(1997)), actually decreases p CNN ( X ) for most random initializations within the ϵ defend -ball.
For efﬁcient optimization, we instead use a greedy technique described in Algorithm 1, which is similar to the greedy decoding process typically used in sequence-to-sequence models (Sutskever et al., 2014).
The method is similar to generating images from PixelCNN, with the additional constraint that the generated image should be within an ϵ defend -ball of a perturbed image.
As an autoregressive model, PixelCNN is slow in image generation.
Nonetheless, by caching redundant calculation, Ramachandran et al.
(2017) proposes a very fast generation algorithm for PixelCNN.
In our experiments, adoption of Ramachandran et al.
(2017)’s method greatly increases the speed of PixelDefend.
For CIFAR-10 images, PixelDefend on average processes 3.6 images per second on one NVIDIA TITAN
Xp GPU.
To show the effectiveness of this greedy method compared to L-BFGS-B, we take the ﬁrst 10 images from CIFAR-10 test set, attack them by FGSM with ϵ attack = 8 , and purify them with L-BFGS-B and PixelDefend respectively.
We used random start points for L-BFGS-B and repeated 100 times for each image.
As depicted in Figure 5, most L-BFGS-B attempts failed at minimizing the bits per
[Figure] [Figure]
dimension of FGSM adversarial examples.
Because of the rugged gradient landscape of PixelCNN,
L-BFGS-B even results in images that have lower probabilities.
In contrast, PixelDefend works much better in increasing the probabilities of puriﬁed images, although their probabilities are still lower compared to clean ones.
In Figure 6 and Figure 7, we empirically show that after PixelDefend, puriﬁed images are more likely to be drawn from the training distribution.
Speciﬁcally, Figure 6 shows that the detecting power of p -values greatly decreases for puriﬁed images.
For DeepFool and CW examples, puriﬁcation makes them barely distinguishable from normal samples of the data distribution.
This is also manifested by
Figure 7, as the p -value distributions of puriﬁed examples are closer to uniform.
Visually, puriﬁed images indeed look much cleaner than adversarially perturbed ones.
In Appendix E, we provide sampled puriﬁed images from Fashion MNIST and CIFAR-10.
4. A DAPTIVE P IXEL D EFEND
One concern with the approach of purifying images is what happens when we purify a clean image.
More generally, we will never know ϵ attack and if we set ϵ defend too large for a given attack, then we will modify all images to become the mode image, which would mostly result in misclassiﬁcations.
One way to avoid this problem is to tune ϵ defend adaptively based on the probability of the input image
[Table] [Table]
under the generative model.
In this way, images that already have high probability under the training distribution would have a very low ϵ defend preventing signiﬁcant modiﬁcation, while low probability images would have a high ϵ defend thus allowing signiﬁcant modiﬁcations.
We implemented a very simple thresholding version of this, which sets ϵ defend to zero if the input image probability is below a threshold value, and otherwise leaves it ﬁxed at a manually chosen setting.
In practice, we set this threshold based on knowledge of the set of possible attacks, so strictly speaking, the adaptive version of our technique is no longer attack-agnostic.
4. P IXEL D EFEND RESULTS
We carried out a comprehensive set of experiments to test various defenses versus attacks.
Detailed information on experimental settings is provided in Appendix B.
All experimental results are summarized in Tab. 1 and Tab. 2.
In the upper part of the tables, we show how the various baseline defenses fare against each of the attacks, while in the lower part of the tables we show how our PixelDefend technique works.
Each table cell contains accuracies on adversarial examples generated with different ϵ attack .
More speciﬁcally, for Fashion MNIST (Tab. 1), we tried ϵ attack = 8 and 25 .
The cells in Tab. 1 is formated as x=y , where x denotes the accuracy (%) on images attacked with ϵ attack = 8 , while y denotes the accuracy when ϵ attack = 25 .
For CIFAR-10 (Tab. 2), we tried ϵ attack = 2 , 8 , and 16 , and the cells are formated in a similar way.
We use the same ϵ defend for different ϵ attack ’s to show that PixelDefend is insensitive to ϵ attack .
From the tables we observe that adversarial training successfully defends against the basic FGSM attack, but cannot defend against the more advanced ones.
This is expected, as training on simple adversarial examples does not guarantee robustness to more complicated attacking techniques.
Consistent with Madry et al.
(2017), adversarial training with BIM examples is more successful at preventing a wider spectrum of attacks.
For example, it improves the accuracy on strongest attack from 2% to 32% on CIFAR-10 when ϵ attack = 8 .
But the numbers are still not ideal even with respect to BIM attack itself.
As in Tab. 2, it only gets 6% on BIM and 8% on CW when ϵ attack = 16 .
We also observe that label smoothing, which learns smoothed predictions so that the gradient ∇ X L ( X ; y ) becomes very small, is only effective against simple FGSM attack.
Model-agnostic methods, such as feature squeezing, can be combined with other defenses for strengthened performance.
We observe that combining it with adversarial training indeed makes it more robust.
Actually, Tab. 1 and Tab.show that feature squeezing combined with adversarial training dominates using feature squeezing along in all settings.
It also gets good performance on DeepFool and CW attacks.
However, for iterative attacks with larger perturbations, i.e ., BIM, feature squeezing performs poorly.
On CIFAR-10, it only gets 2% and 0% accuracy on BIM with ϵ attack = 8 and 16 respectively.
PixelDefend, our model-agnostic and attack-agnostic method, performs well on different classiﬁers (ResNet and VGG) and different attacks without modiﬁcation .
In addition, we can see that augmenting basic adversarial training with PixelDefend can sometimes double the accuracies.
We hypothesize that the puriﬁed images from PixelDefend are still not perfect, and adversarially trained networks have more toleration for perturbations.
This also corroborates the plausibility and beneﬁt of combining PixelDefend with other defenses.
Furthermore, PixelDefend can simultaneously obtain accuracy above 70% for all other attacking techniques, while ensuring that performance on clean images only declines slightly.
Models with
PixelDefend consistently outperform other methods with respect to the strongest attack.
On Fashion
MNIST, PixelDefend methods improve the accuracy on strongest attack from 76% to 85% and 63% to 84%.
On CIFAR-10, the improvements are even more signiﬁcant, i.e ., from 74% to 81%, 32% to 70% and 6% to 56%, for ϵ attack = 2 , 8, and 16 respectively.
In a security-critical scenario, the weakest part of a system determines the overall reliability.
Therefore, the outstanding performance of PixelDefend on the strongest attack makes it a valuable and useful addition for improving AI security.
4. E ND - TO -E ND ATTACK OF P IXEL D EFEND
A natural question that arises is whether we can generate a new class of adversarial examples targeted speciﬁcally at the combined PixelDefend architecture of ﬁrst purifying the image and then using an existing classiﬁer to predict the label of the puriﬁed image.
We have three pieces of empirical evidence to believe that such adversarial examples are hard to ﬁnd in general.
First, we attempted to apply the iterative BIM attack to an end-to-end differentiable version of PixelDefend generated by unrolling the PixelCNN puriﬁcation process.
However we found the resulting network was too deep and led to problems with vanishing gradients (Bengio et al., 1994), resulting in adversarial images that were identical to the original images.
Moreover, attacking the whole system is very time consuming.
Empirically, it took about 10 hours to generate 100 attacking images with one TITAN
Xp GPU which failed to fool PixelDefend.
Secondly, we found the optimization problem in Eq.
(4.1) was not amenable to gradient descent, as indicated in Figure 5.
This makes gradient-based attacks especially difﬁcult.
Last but not least, the generative model and classiﬁer are trained separately and have independent parameters.
Therefore, the perturbation direction that leads to higher probability images has a smaller correlation with the perturbation direction that results in misclassiﬁcation.
Accordingly, it is harder to ﬁnd adversarial examples that can fool both of them together.
However, we will open source our codes and look forward to any possible attack from the community.
R ELATED WORK
Most recent work on detecting adversarial examples focuses on adding an outlier class detection module to the classiﬁer, such as Grosse et al.
(2017), Gong et al.
(2017) and Metzen et al.
Those methods require the classiﬁcation model to be changed, and are thus not model-agnostic.
Feinman et al.
(2017) also presents a detection method based on kernel density estimation and Bayesian neural network uncertainty.
However, Carlini & Wagner (2017a) shows that all those methods can be bypassed.
Grosse et al.
(2017) also studied the distribution of adversarial examples from a statistical testing perspective.
They reported the same discovery that adversarial examples are outside of the training distribution.
However, our work is different from theirs in several important aspects.
First, the kernel-based two-sample test used in their paper needs a large number of suspicious inputs, while our method only requires one data point.
Second, they mainly tested on ﬁrst-order methods such as
FGSM and JSMA (Papernot et al., 2016b).
We show the efﬁcacy of PixelCNN on a wider range of attacking methods (see Figure 3), including both ﬁrst-order and iterative methods.
Third, we further demonstrate that random perturbed inputs are also outside of the training distribution.
Some other work has focused on modifying the classiﬁer architecture to increase its robustness, e.g ., Gu & Rigazio (2014), Cisse et al.
and Nayebi & Ganguli (2017).
Although they have witnessed some success, such modiﬁcations of models might limit their representative power and are also not model-agnostic.
Our basic idea of moving points to higher-density regions is also present in other machine learning methods not speciﬁcally designed for handling adversarial data; for example, the manifold denoising method of Hein & Maier (2007), the direct density gradient estimation of Sasaki et al.
(2014), and the denoising autoencoders of Vincent et al.
(2008) all move data points from low to high-density regions.
In the future some of these methods could be adapted to amortize the puriﬁcation process directly, that is, to learn a puriﬁcation network .
C ONCLUSION
In this work, we discovered that state-of-the-art neural density models, e.g ., PixelCNN, can detect small perturbations with high sensitivity.
This sensitivity broadly exists for a large number of perturbations generated with different methods.
An interesting fact is that PixelCNN is only sensitive in one direction—it is relatively easy to detect perturbations that lead to lower probabilities rather than higher probabilities.
Based on the sensitivity of PixelCNN, we utilized statistical hypothesis testing to verify that adversarial examples lie outside of the training distribution.
With the permutation test, we give exact p -values which can be used as a uncertainty measure for detecting outlier perturbations.
Furthermore, we make use of the sensitivity of generative models to explore the idea of purifying adversarial examples.
We propose the PixelDefend algorithm, and experimentally show that returning adversarial examples to high probability regions of the training distribution can signiﬁcantly decrease their damage to classiﬁers.
Different from many other defensive techniques, PixelDefend is model-agnostic and attack-agnostic, which means it can be combined with other defenses to improve robustness without modifying the classiﬁcation model.
As a result PixelDefend is a practical and effective defense against adversarial inputs.
A PPENDIX A O N RANDOM PERTURBATIONS
One may observe from Figure 3(b) that random perturbations have very low p -values, and thus also live outside of the high density area.
Although many classiﬁers are robust to random noise, it is not a property granted by the dataset.
The fact is that robustness to random noise could be from model inductive bias, and there exist classiﬁers which have high generalization performance on clean images, but can be attacked by small random perturbations.
It is easy to construct a concrete classiﬁer that are susceptible to random perturbations.
Our ResNet on CIFAR-10 gets 92.0% accuracy on the test set and 87.3% on randomly perturbed test images with ϵ attack = 8 .
According to our PixelCNN, 175 of 10000 test images have a bits per dimension (BPD) larger than 4.5, while the number for random images is 9874.
Therefore, we can deﬁne a new classiﬁer
[Equation]
which will get roughly 92% (cid:2) 
 =
+ 10% (cid:2) 175 = 10000 (cid:25) 90 : 6% accuracy on the test set, while only 87 {+: 3% (cid:2) 126 = 10000+10% (cid:2) 
+}
= 10000 (cid:25) 11 : 0% accuracy on the randomly perturbed images.
This classiﬁer has comparable generalization performance to the original ResNet, but will give incorrect labels to most randomly perturbed images.
A PPENDIX B E XPERIMENTAL SETTINGS
Adversarial Training We have tested adversarial training with both FGSM and BIM examples.
During training, we take special care of the label leaking problem as noted in Kurakin et al.
(2016)— we use the predicted labels of the model to generate adversarial examples, instead of using the true labels.
This prevents the adversarially trained network to perform better on adversarial examples than clean images by simply retrieving ground-truth labels.
Following Kurakin et al.
, we also sample ϵ attack from a truncated Gaussian distribution for generating FGSM or BIM adversarial examples, so that the adversarially trained network won’t overﬁt to any speciﬁc ϵ attack .
This is different from Madry et al.
(2017), where the authors train and test with the same ϵ attack .
For Fashion MNIST experiments, we randomly sample ϵ attack from N (0 ; (cid:14) ) , take the absolute value and truncate it to [0 ; 2 (cid:14) ] , where (cid:14) = 8 or 25 .
For CIFAR-10 experiments, we follow the same procedure but ﬁx (cid:14) = 8 .
Feature Squeezing For implementing the feature squeezing defense, we reduce the number of colors to 8 on Fashion MNIST, and use 32 colors for CIFAR-10.
The numbers are chosen to make sure color reduction will not lead to signiﬁcant deterioration of image quality.
After color depth reduction, we apply a 2 (cid:2) 2 median ﬁlter with reﬂective paddings, since it is reported in Xu et al.
(2017b) to be most effective for preventing CW attacks.
Models We use ResNet (62-layer) and VGG (16-layer) as classiﬁers.
In our experiments, normally trained networks have the same architectures as adversarially trained networks.
Since the images of
Fashion MNIST contain roughly one quarter values of those of CIFAR-10, we use a smaller network for classifying Fashion MNIST.
More speciﬁcally, we reduce the number of feature maps for Fashion
MNIST to 1/4 while keeping the same depths.
In practive, VGG is more robust than ResNet due to using of dropout layers.
The network architecture details are described in Appendix C. For the
PixelCNN generative model, we adopted the implementation of PixelCNN++ (Salimans et al., 2017), but modiﬁed the output from mixture of logistic distributions to softmax.
The feature maps are also reduced to 1/4 for training PixelCNN on Fashion MNIST.
Adaptive Threshold We chose the adaptive threshold discussed in Section 4.2 using validation data.
We set the threshold at the lowest value which did not decrease the performance of the strongest adversary.
For Fashion MNIST, the threshold of bits per dimension was set to 1.8, and for CIFARthe number was 3.2.
As a reference, the mean value of bits per dimension for Fashion MNIST test images is 2.7 and for CIFAR-10 is 3.0.
However, we admit that using a validation set to choose the best threshold makes the adaptive version of PixelDefend not strictly attack-agnostic.
A PPENDIX C I MAGE CLASSIFIER ARCHITECTURES (cid:3)
R ES N ET CLASSIFIER FOR CIFAR-10 & MNIST
[Table]
(ﬁlter size: 3 (cid:2) 3 , feature maps: [Table] ), stride
[Table]
C. VGG CLASSIFIER FOR CIFAR-10 & F ASHION MNIST
[Table]
A PPENDIX D S AMPLED IMAGES FROM P IXEL CNN D. F ASHION MNIST
D. CIFAR
[Figure]
A PPENDIX E S AMPLED PURIFIED IMAGES FROM P IXEL D EFEND E. F ASHION MNIST
[Figure]