T HE L IMITATIONS OF A DVERSARIAL T RAINING AND THE B LIND -S POT A TTACK
Anonymous authors
Paper under double-blind review
A BSTRACT
The adversarial training procedure proposed by Madry et al.
(2018) is one of the most effective methods to defend against adversarial examples on deep neural networks (DNNs).
Despite being very effective on MNIST, adversarial training on larger datasets like CIFAR and ImageNet achieves much worse results.
In our paper, we shed some lights on the practicality and hardness of adversarial training by ﬁrst showing that the effectiveness of adversarial training procedure on test set has a strong correlation with the distance between the test point and the manifold of training data.
The test examples that are relatively far away from the manifold of training dataset are more likely to be vulnerable to adversarial attacks.
Consequentially, adversarial training based defense is susceptible to a new class of attacks ( “blind-spot attack” ) where the input image resides in a “blind-spot” in the empirical distribution of training data but is still on the ground-truth data manifold.
For MNIST, we found that these blind-spots can be easily found by simply scaling and shifting image pixel values.
Most importantly, for large datasets with high dimensional and complex data manifold (CIFAR, ImageNet, etc), the existence of blind-spots in adversarial training makes the defense on any valid test examples almost impossible due to the curse of dimensionality.
I NTRODUCTION
Since the discovery of adversarial examples in deep neural networks (DNNs) (Szegedy et al., 2013), adversarial training with the robustness optimization framework (Madry et al., 2018; Sinha et al., 2018) has become one of the most effective methods to defend against adversarial examples.
A recent study by Athalye et al.
(2018) showed that adversarial training does not cause obfuscated gradient and delivers promising results for defending adversarial examples on small datasets.
Adversarial training approximately solves the following min-max optimization problem:
[Equation]
where X is the set of training data, L is the loss function, θ is the parameter of the network, and S is usually a norm constrained (cid:96) p ball centered at 0.
Madry et al.
(2018) propose to use projected gradient descent (PGD) to approximately solve the maximization problem within S =
{ δ | (cid:107) δ (cid:107) ∞ ≤ (cid:15) } , where (cid:15) = 0 .
3 for MNIST dataset on a 0 - 1 pixel scale, and (cid:15) = 8 for CIFAR-10 dataset on a 0pixel scale.
This approach achieves impressive defending results on the MNIST test set: so far the best available white-box attacks by Zheng et al.
(2018) can only decrease the test accuracy from approximately 98% to 88% 1 .
However, on CIFAR-10 dataset, a simple 20-step PGD can decrease the test accuracy from 87% to less than 50% 2 .
Currently, the effectiveness of adversarial training is measured on the test dataset.
However, the adversarial training process itself is done on the training set.
Suppose we can optimize (1) perfectly, then certiﬁed robustness may be obtained on those training data points.
However, if the empirical distribution of training dataset differs from the true data distribution, drawing a test point from the true data distribution might be far away from the empirical distribution of training dataset and is not “covered” by the adversarial training procedure.
For datasets that are relatively simple and has low intrinsic dimensions (MNIST, Fashion MNIST, etc), we can obtain enough training examples to make sure adversarial training covers most part of the data distribution.
For high dimensional datasets (CIFAR, ImageNet), adversarial training have been shown difﬁcult (Kurakin et al., 2016;
Tram`er et al., 2018) and only limited success was obtained.
A recent attack proposed by Song et al.
(2018) shows that adversarial training can be defeated when the input image is produced by a generative model (for example, a generative adversarial network) rather than selected directly from the test examples.
The generated images are well recognized by humans and thus valid images in the ground-truth data distribution.
In our interpretation, this attack effective ﬁnds the “blind-spots” in the input space that the training data do not well cover.
For higher dimensional datasets, we hypothesize that many test images already fall into these blindspots of training data and thus adversarial training only obtains a moderate level of robustness.
It is interesting to see that for those test images that adversarial training fails to defend, if their distances (in some metrics) to the training dataset are indeed larger.
In our paper, we try to explain the success of robust optimization based adversarial training and show the limitations of this approach when the test points are slightly off the empirical distribution of training data.
Our main contributions are:
• We show that on the original set of test images, the effectiveness of adversarial training is highly correlated with the distance (in different distance metrics) from the test image to the distribution of training images.
For MNIST and Fashion MNIST dataset, most test images are close the training data and very good robustness is observed on these points.
For CIFAR, there is a clear trend that the adversarially trained network gradually loses its robustness property when the test images are further away from training data.
• We propose a method to estimate the distance between training set and test set using the K-L divergence, which is a good indicator of the success of adversarial training.
On datasets where adversarial training does not perform well, this divergence tends to be larger.
• We identify a new class of attacks, “ blind-spot attacks ”, where the input image resides in a “blind-spot” in the empirical distribution of training data (far enough from any training examples) but is still in the ground-truth data distribution (well recognized by humans and correctly classiﬁed by the model).
Adversarial training cannot provide good robustness on these blindspots.
GAN based attacks by Song et al.
(2018) is a successful attack in this class.
• We show that blind-spots can be easily found without using an more involved procedure like GAN.
We propose a few simple transformations (changing contrast and background), that do not signiﬁcantly affect MNIST and Fashion MNIST model’s accuracy but make the adversarially trained models model vulnerable to adversarial attacks on these sets of transformed input images.
These transformations effectively move the test images slightly out of the domain of training images, which does not affect generalization but poses a challenge for robust learning.
• Our results imply that current adversarial training procedures cannot scale to datasets with a large (intrinsic) dimension, where any practical amount of training data cannot cover all the blind-spots.
This explains the limited success for applying adversarial training on ImageNet dataset, where many test images can be sufﬁciently far away from the empirical distribution of training dataset.
R ELATED W ORKS 2. D EFENDING A GAINST A DVERSARIAL E XAMPLES
Adversarial examples in DNNs have brought great threats to the deep learning-based AI applications such as autonomous driving and face recognition.
Therefore, defending against adversarial examples is an urgent task before we can safely deploy deep learning models to a wider range of applications.
Following the emergence of adversarial examples, various defense methods have been proposed recently, such as defensive distillation in Papernot et al.
and feature squeezing in Xu et al.
Some of these defense methods have been proven vulnerable or ineffective under strong attack methods such as C&W in Carlini & Wagner (2017).
Another category of recent defense is gradient masking or obfuscated gradient (Buckman et al.
; Ma et al.
; Guo et al.
Song et al.
; Samangouei et al.
, but these methods are also successfully evaded by the stronger BPDA attack (Athalye et al.
Randomization in DNNs (Dhillon et al., 2018; Xie et al., 2017; Liu et al., 2018) is also used to reduce the success rate of adversarial attacks, however, it usually incurs additional computational costs and still cannot fully defend against an adaptive attacker (Athalye et al., 2018; Athalye & Sutskever, 2017).
Another effective defense method is adversarial training, which trains the model with adversarial examples freshly generated during the entire training process.
First introduced by Goodfellow et al.
, adversarial training demonstrates the state-of-the-art defending performance.
Madry et al.
(2018) formulated the adversarial training procedure into a min-max robust optimization problem and has achieved state-of-the-art defending performance on MNIST and CIFAR datasets.
Several attacks have been proposed to attack the model release by Madry et al.
On the MNIST testset, so far the best attack by Zheng et al.
(2018) can only reduce the test accuracy from 98% to 88%.
Analysis by Athalye et al.
(2018) shows that this adversarial training framework does not obfuscate gradient and truly increases model robustness; gradient based attacks with random starts can only achieve less than 10% success rate with given distortion constraints and are unable to penetrate this defense.
On the other hand, attacking adversarial training using generative models have also been investigated; both Xiao et al.
(2018) and Song et al.
(2018) propose to use GANs to produce adversarial examples in black-box and white-box settings, respectively.
Finally, certiﬁed defense methods were proposed. Certiﬁed defenses are able to provably increase model robustness, achieve good performance (i.e., test accuracy on natural images does not drop signiﬁcantly and training is computationally feasible), and can be applied to medium-sized networks with multiple layers. Notably, Sinha et al. (2018) analyzes adversarial training using distributional robust optimization techniques.
Kolter & Wong (2018); Wong et al. (2018) proposed a robustness certiﬁcate based on the dual of a convex relaxation for ReLU networks, and used it for training to provably increase robustness.
During training, certiﬁed defense methods can provably guarantee that the model is robust on most (if not all) training examples
2. A NALYZING A DVERSARIAL E XAMPLES
Along with the attack-defense arms race, some insightful ﬁndings have been discovered to understand the natural of adversarial examples, both theoretically and experimentally.
Schmidt et al.
(2018) show that even for a simple data distribution of two class-conditional Gaussians, robust generalization requires signiﬁcantly larger number of samples than standard generalization.
Cullina et al.
(2018) extend the well-known PAC learning theory to the case with adversaries, and derive the adversarial VC-dimension which can be either larger or smaller than the standard VC-dimension.
Bubeck et al.
(2018) conjecture that a robust classiﬁer can be computationally intractable to ﬁnd, and proved the computation hardness under statistical query (SQ) model.
Additionally, ﬁnding the safe area approximately is computationally hard according to Katz et al.
(2017) and Weng et al.
Mahloujifar et al.
(2018) explain the prevalence of adversarial examples by making a connection to the “concentration of measure” phenomenon in metric measure spaces.
Tsipras et al.
(2018) discover that data examples consist of robust and non-robust features and adversarial training tends to ﬁnd robust features that have strongly-correlations with the labels.
Su et al.
(2018) conduct large scale experiments on ImageNet and ﬁnd a negative correlation between robustness and accuracy.
Both adversarial training and certiﬁed defence achieve state-of-the-art robustness performance on training data, but it is still unknown if the trained model has good robust generalization property.
Typically, we evaluate the robustness of a model by computing a upper bound of error on the test set; speciﬁcally, given a norm bounded distortion (cid:15) , we verify if each image in test set has a robustness certiﬁcate.
There might exist test images that are still within the capability of standard generalization (i.e., correctly classiﬁed by DNNs with high conﬁdence, and well recognized by humans), but behaves badly in robust generalization (i.e., adversarial examples can be easily found).
Our paper complements those existing ﬁndings by showing the strong correlation between the effectiveness of adversarial defenses (both adversarial training and certiﬁed defenses) and the distance between training data and test points.
Additionally, we show that a tiny shift in input domain (which may or may not be detectable in embedding space) can easily destroy the robustness property of an robust model.
M ETHODOLOGY 3. M EASURING THE DISTANCE BETWEEN TRAINING DATASET AND A TEST DATA POINT
To verify the correlation between the effectiveness of adversarial training and how close a test point is to the training dataset, we need to propose a reasonable distance metric between a test example and a set of training examples.
However, deﬁning a meaningful distance metric for high dimensional image data is a challenging problem.
Naively using an Euclidean distance metric in the input space of images works poorly as it does not reﬂect the true distance between the images on their ground-truth manifold.
One strategy is to use (kernel-)PCA, t-SNE (Maaten & Hinton, 2008), or
UMAP (McInnes & Healy, 2018) to reduce the dimension of training data to a low dimensional space, and then deﬁne distance in that space.
These methods are sufﬁcient for small and simple datasets like MNIST, but for more general and complicated dataset like CIFAR, extracting a meaningful low-dimensional manifold directly on the input space can be really challenging.
On the other hand, using a DNN to extract features of input images and measuring the distance in the deep feature embedding space has demonstrated better performance in many applications (Hu et al., 2014; 2015), since DNN models can capture the manifold of image data much better than simple methods such as PCA or t-SNE.
Although we can form an empirical distribution using kernel density estimation (KDE) on the deep feature embedding space and then obtain probability densities for test points, our experiments showed that KDE work poorly because the features extracted by DNNs are still high dimensional (hundreds or thousands dimensions).
Taking the above considerations into account, we propose a simple and intuitive distance metric using deep feature embeddings and k -nearest neighbour.
Given a feature extraction neural network h ( x ) , a set of n training data points X train =
{ x 1 train , x 2 train , · · · , x n train } , and a set of m test data points X test =
{ x 1 test , x 2 test , · · · , x m test } from the true data distribution, for each j ∈ [ m ] , we deﬁne the following distance between x j test and X train :
[Equation]
where π j : [ n ] → [ n ] is a permutation that { π j (1) , π j (2) , · · · , π j ( n ) } is an ascending ordering of training data based on the (cid:96) p distance between x j test and x i train in the deep embedding space, i.e.,
[Equation]
In other words, we average the embedding space distance of k nearest neighbors of x j in the training dataset.
This simple metric is non-parametric and we found that the results are not sensitive to the selection of k ; also, for naturally trained and adversarially trained feature extractors, the distance metric obtained by them have the same correlation with the effectiveness of adversarial training.
3. M EASURING THE DISTANCE BETWEEN TRAINING AND TEST DATASETS
We are also interested to investigate the “distance” between the training dataset and the test dataset, as it will become a good indicator for how adversarial training performs on the test set.
Unlike the setting in Section 3.1, this requires to compute a divergence between two data distributions.
Given n training data points X train = { x 1 train , x 2 train , · · · , x n train } and m test data points X test =
{ x 1 test , x 2 test , · · · , x m test } , we ﬁrst apply a neural feature extractor h to them, which is the same as in Section 3.1.
Then, we apply a non-linear projection (in our case, we use t-SNE) to project both h ( x i train ) and h ( x j test ) to a low dimensional space, and obtain ¯ x i train =
proj ( h ( x i train )) and ¯ x j test =
proj
( h ( x j test )) .
The dataset after feature extraction and projection is denoted as ¯ X train and ¯ X test .
Because ¯ x i train and ¯ x j test are low dimensional, we can use kernel density estimation (KDE) to form empirical distributions ¯ p train and ¯ p test for them. We use p train and p test to denote the true distributions.
Then, we approximate the K-L divergence between p train and p test via a numerical integration of Eq.(3):
[Equation]
where ¯ p train ( x ) = 1 n (cid:80) ni =1 K ( x − ¯ x i train ; H ) and ¯ p test ( x ) = 1 m (cid:80) mj =1 K ( x − ¯ x j test ; H ) are the KDE density functions.
K is the kernel function (speciﬁcally, we use the Gaussian kernel) and H is the bandwidth parameter automatically selected by Scott’s rule (Scott, 2015).
V is chosen as a box bounding all training and test data points.
For a multi-class dataset, we compute the aforementioned
KDE and K-L divergence for each class separately .
As we will show in experiments, this K-L divergence is a good indicator on the effectiveness of adversarial training – a large divergence between training and test data points is a strong indication that adversarial training is likely to fail on many test points.
3. T HE B LIND -S POT A TTACK : A NEW CLASS OF ADVERSARIAL ATTACKS
Inspired by our ﬁndings of the negative correlation between the effectiveness of adversarial training and the distance between a test image and training dataset, we identify a new class of adversarial attacks called “blind-spot attacks”, where we ﬁnd input images that are far enough from any existing training examples such that:
• They are still drawn from the ground-truth data distribution (i
e. well recognized by humans) and classiﬁed correctly by the model (within the generalization capability of the model); • Adversarial training cannot provide good robustness properties on these images, and we can easily ﬁnd their adversarial examples using a simple gradient based attack.
We ﬁnd that these blind-spots are prevalent and can be easily found without resorting to complex generative models like in Song et al.
For the MNIST dataset which Madry et al.
(2018) demonstrate the strongest defense results so far, we propose a simple transformation to ﬁnd the blind-spots in this model.
We simply scale and shift each pixel value.
Suppose the input image x ∈ [ − 0 .
5] d , we scale and shift each test data example x to form a new example x (cid:48) :
[Equation]
where α is a constant close to 1 and β is a constant close to 0.
We make sure that the selection of α and β will result in a x (cid:48) that is still in the valid input range [ − 0 .
5] d .
This transformation effectively adjusts the contrast of the image, and/or adds a gray background to the image.
It is important that the blind-spot images are still valid images; for example, a digit that is slightly darker than the one in test set is still considered as a valid digit and can be well recognized by humans.
Also, we found that with appropriate α and β the accuracy of MNIST and Fashion-MNIST models barely decreases; the model has enough generalization capability for this set of slightly transformed images, yet their adversarial examples can be easily found.
Although the blind-spot attack is beyond the threat model considered in adversarial training (e.g.
(cid:96) ∞ norm constrained perturbations), our argument is that adversarial training (and probably other defense methods with certiﬁcations only on training examples such as Kolter & Wong (2018)) are unlikely to scale well to datasets that lie in a high dimensional manifold, as the limited training data only guarantees robustness near these training examples.
The blind-spots are almost inevitable in high dimensional case.
For example, in CIFAR-10, about 50% of test images are already in blindspots and their adversarial examples can be trivially found by projected gradient descent despite adversarial training.
Using data augmentation may eliminate some blind-spots, however for high dimensional data it is impossible to enumerate all possible inputs due to the curse of dimensionality.  E XPERIMENTS
In this section we present our experimental results on adversarially trained models by Madry et al.
Experimental results on models by Kolter & Wong (2018), Wong et al. (2018) and Sinha et al. (2018) will be demonstrated in Section 6.4 in the Appendix. 4. S ETUP
We conduct experiments on adversarially trained models by Madry et al.
(2018) on four datasets:
MNIST, Fashion MNIST, and CIFAR-10.
For MNIST, we use the “secret” model release for the
MNIST attack challenge 3 .
For CIFAR-10, we use the public “adversarially trained” model 4 .
For
Fashion MNIST, we train our own model with the same model structure and parameters as the robust MNIST model, except that the iterative adversary is allowed to perturb each pixel by at most (cid:15)
= 0 .
1 as a larger (cid:15) will signiﬁcantly reduce model accuracy.
We attack each model with Carlini & Wagner’s (cid:96) ∞ attack (Carlini & Wagner (2017)).
We found that this method generally ﬁnds adversarial examples with smaller perturbations than projected gradient descent (PGD).
To avoid gradient masking, we initial our attacks using two schemes: (1) from the original image plus a random Gaussian noise with a standard deviation of 0 .
2 ; (2) from a blank gray image where all pixels are initialized as 0.
A successful attack is deﬁned as ﬁnding an perturbed example that changes the model’s classiﬁcation and the (cid:96) ∞ distortion is less than a given (cid:15) used for robust training.
For MNIST, (cid:15) = 0 . 3 ; for Fashion-MNIST, (cid:15) = 0 . 1 ; and for CIFAR, (cid:15) = 8 / 255 .
All input images are normalized to [ − 0 .
4. E FFECTIVENESS OF ADVERSARIAL TRAINING AND THE DISTANCE TO TRAINING SET
In this set of experiments, we build a connection between attack success rate on adversarially trained models and the distance between a test example and the whole training set.
We use the metric deﬁned in Section 3.1 to measure this distance.
For MNIST and Fashion-MNIST, the outputs of the ﬁrst fully connected layer are used as the neural feature extractor h ( x ) ; for CIFAR, we use the outputs of the last average pooling layer.
We consider both naturally and adversarially trained networks as the neural feature extractor.
p is set as 2 and k is set as 5.
The results are shown in Figure 1, 2 and 3.
For each test set, after obtaining the distance of each test point, we bin the test data points based on their distances to the training set and show them in the histogram at the bottom half of each ﬁgure (red).
The top half of each ﬁgure (blue) represents the attack success rates for the test images in the corresponding bins. The missing bars on the right are due to too few points in the corresponding bins. We only attack correctly classiﬁed images and only calculate success rate on those images. Note that we should not compare the distances shown between the left and right columns of Figures 1, 2 and 3 because they are obtained from two different models
[Figure] [Figure]
[Figure] [Figure]
[Figure]
As we can observe in all three ﬁgures, most successful attacks in test sets for adversarially trained networks concentrate on the right hand side of the distance distribution, and the success rates tend to grow when the distance is increasing.
The trend is independent of the feature extractor being used (naturally or adversarially trained).
The strong correlation between attack success rates and the s u cc e ss r a t e 1.
4.5 distance 0.
p e r c e n t a g e
[Figure] s u cc e ss r a t e      7 distance515p e r c e n t a g e [Figure] distance from a test point to the training dataset supports our hypothesize that adversarial training tends to fail on test points that are far enough from the training data distribution.
4. K-L D IVERGENCE BETWEEN TRAINING AND TEST SETS VS ATTACK SUCCESS RATE
To quantify the overall distance between the training and the test set, we calculate the K-L divergence between the KDE distributions of training set and test set for each class according to Eq.
Then, for each dataset we take the average K-L divergence across all classes, as shown in Table 1.
We use both adversarially trained networks and naturally trained networks as our feature extractors h ( x ) .
As a reference, we also calculate the average normalized (cid:96) 2 distance.
We ﬁrst calculate the feature representations of test points and training points.
To compare between datasets, we normalize each element of the feature representation to mean 0 and variance 1. Then we calculate the (cid:96) 2 distance between each test point and the training set as in Section 4.2. We average this distance among all test points and divide it by √ t to normalize the dimension, where t is the dimension of the feature representation h ( · ) . We also report the attack success rates and test accuracy for each adversarially trained model.
Clearly, Fashion-MNIST is the dataset with strongest defense as measured by the attack success rates on test set, and its K-L divergence is also the smallest.
For CIFAR, the divergence between training and test sets is signiﬁcantly larger, and adversarial training only has limited success.
The hardness of training a robust model for MNIST is in between Fashion-MNIST and CIFAR.
Another important observation is that the effectiveness of adversarial training does not depend on the accuracy; for
Fashion-MNIST, classiﬁcation is harder as the data is more complicated than MNIST, but training a robust Fashion-MNIST model is easier as the data distortion is more concentrated and adversarial training has less “blind-spots”.
[Table]
4. B LIND -S POT A TTACK ON MNIST AND F ASHION MNIST
In this section we focus on applying the proposed blind-spot attack to MNIST and Fashion MNIST.
For MNIST, (cid:15) = 0 .
so we set the scaling factor to α = { 1 . 0 , 0 . 9 , 0 . 8 , 0 . 7 } .
For Fashion-MNIST, (cid:15) = 0 .
so we set the scaling factor to α = { 1 . 0 , 0 .
We set β to either 0 or a small constant.
The case α = 1 . 0 , β = 0 . 0 represents the original test set images.
We report the model’s accuracy and attack success rates for each choice of α and β in Table 2 and Table 3.
Because we scale the image by a factor of α , we also set a stricter criterion of success – the (cid:96) ∞ perturbation must be less than α(cid:15) to be counted as a successful attack.
For MNIST, (cid:15) = 0 .
3 and for Fashion-MNIST, (cid:15) = 0 .
We report both success criterion, (cid:15) and α(cid:15) in Tables 2 and 3. [Figure]
[Figure] We ﬁrst observe that for all pairs of α and β the transformation does not affect the models’ test accuracy at all.
The adversarially trained model classiﬁes these slightly scaled and shifted images very well, with test accuracy equivalent to the original test set.
Visual comparisons in Figureshow that when α is close to 1 and β is close to 0, it is hard to distinguish the transformed images from the original images.
On the other hand, according to Tables 2 and 3, the attack success rates for those transformed test images are signiﬁcantly higher than the original test images, for both the α ,
[Table]
(suc. rate) and test accuracy (acc) of scaled and shifted MNIST.
An attack is considered as successful if its (cid:96) ∞ distortion is less than given thresholds (th.) 0.3 or 0 .
3 α .
[Table]
An attack is considered successful if its (cid:96) ∞ distortion is less than a given thresholds (th or original criterion (cid:15) and the stricter criterion α(cid:15) .
In Figure 12, we can see that the (cid:96) ∞ adversarial perturbation required is much smaller than the original image after the transformation.
Thus, the proposed scale and shift transformations indeed move test images into blind-spots.
One might think that we can generally detect blind-spot attacks by observing their distances to the training dataset, using a metric similar to Eq.
Thus, we plot histograms for the distances between tests points and training dataset, for both original test images and those slightly transformed ones in
Figure 5.
We set α = 0 .
7 , β = 0 for MNIST and α = 0 .
9 , β = 0 for Fashion-MNIST.
Unfortunately, the differences in distance histograms for these blind-spots is so tiny that we cannot reliably detect the change, yet the robustness property drastically changes on these transformed images.
C ONCLUSION
In this paper, we observed that the effectiveness of adversarial training is highly correlated with the characteristics of the dataset, and data points that are far enough from the distribution of training data are still vulnerable to adversarial attacks despite adversarial training.
Following this observation, we deﬁned a new class of attacks called “blind-spot attack” and proposed a simple scale-andshift scheme for conducting blind-spot attacks on adversarially trained MNIST and Fashion MNIST datasets with high success rates.
Our ﬁndings suggest that adversarial training can be challenging due to the prevalence of blind-spots in high dimensional datasets.
A PPENDIX 6. D ISTANCE DISTRIBUTIONS UNDER DIFFERENT NEAREST NEIGHBOUR PARAMETERS k
As discussed in Section 3.
[Figure]
we use k -nearest neighbour in embedding space to measure the distance between a test example and the training set.
In Section 4.2 we use k = 5 .
In this section we show that the choice of k does not have much inﬂuence on our results.
We use the adversarially trained model on the CIFAR dataset as an example.
In Figures 6, 7 and 8 we choose k = 10 , 100 , 1000 , respectively.
The results are similar to those we have shown in Figure 3: a strong correlation between attack success rates and the distance from a test point to the training dataset
[Figure]
[Figure]
[Figure]
6. G ERMAN TRAFFIC SIGN (GTS) DATASET
We also studied the German Trafﬁc Sign (GTS) (Houben et al., 2013) dataset.
For GTS, we train our own model with the same model structure and parameters as the adversarially trained CIFAR model (Madry et al., 2018).
We set (cid:15) = 8 / 255 for adversarial training with PGD [Figure] and also use the same (cid:15) as the threshold of success.
The results are shown in Figure 9.
The GTS model behaves similarly to the CIFAR model: attack success rates are much higher when the distances between the test example and the training dataset are larger. 6. M ORE VISUALIZATION RESULTS
We demonstrate more MNIST and Fashion-MNIST visualizations in Figure 10.
[Figure]
[Figure] [Figure]
[Figure] [Figure]
6. R ESULTS ON OTHER ROBUST TRAINING METHODS
In this section we demonstrate our experimental results on two other state-of-the-art certiﬁed defense methods, including convex adversarial polytope by Wong et al.
and Kolter & Wong (2018), and distributional robust optimization based adversarial training by Sinha et al.
Different from the adversarial training by Madry et al.
(2018), these two methods can provide a formal certiﬁcation on the robustness of the model and provably improve robustness on the training dataset.
However, they cannot guarantee robustness on test data.
We did not include other certiﬁed defenses like Raghunathan et al.
and Hein & Andriushchenko (2017) because they are not applicable to multi-layer networks.
For all defenses, we use their ofﬁcial implementations and pretrained models (if available).
Figure 11 shows the results on CIFAR using the defences in Wong et al.
Tables 4 and 5 show the blind-spot attack results on MNIST and Fashion-MNIST for robust models in Kolter & Wong (2018) and Sinha et al.
(2018), respectively.
Figure 12 shows the blind-spot attack examples on Madry et al.
(2018), Kolter & Wong (2018) and Sinha et al.
[Figure]
[Table] {+(a) α = 1 .β = 
 .dist+}
= 0 .(b)
{+α = 0 .β = 
 .dist = 0 .(c)+}
α = 0 .β
{+= 
+}
.dist
= 0 .(d)
α = 1 .β
{+= 
+}
.dist
= 0 .(e)
{+α = 0 .β = 
 .dist = 0 .(f)+}
α = 0 .β
{+= 
+}
.dist
= 0 .(g)
α = 1 .β
{+= 
+}
.dist
= 0 .(h)
α = 0 .β
{+= 
+}
.dist
= 0 .(i)
α = 0 .β
{+= 
+}
.dist
= 0 .(j)
α = 1 .β
{+= 
+}
.dist
= 0 .(k)
α = 0 .β
{+= 
+}
.dist
= 0 .(l)
α = 0 .β
{+= 
+}
.dist = 0 .
[Figure]
adversarially trained
[Figure] [Figure] [Figure] [Table]
Madry et al.
Sinha et al.
Kolter & Wong (2018)
{+α = 1 . 
 β = 0 .d = 0 .α = 0 .+}
9 β = 0 .d
= 0 .α
= 0 .
9 β = 0 .d
= 0 .α
{+= 1 . 
+}
β = 0 .d
= 2 .α = 0 .
9 β = 0 .d
= 1 .α = 0 .
9 β = 0 .d
= 1 .α
= 1 . 0 β = 0 .d
= 0 .α
= 0 .
9 β = 0 .d
= 0 .α
= 0 .
9 β = 0 .d
= 0 .α
{+= 1 . 
+}
β = 0 .d
= 0 .α
= 0 .
8 β = 0 .d
= 0 .α
= 0 .
8 β = 0 .d
= 0 .α
{+= 1 . 
+}
β = 0 .d
= 2 .α = 0 .
9 β = 0 .d
= 1 .α = 0 .
9 β = 0 .d
= 1 .α
= 1 . 0 β = 0 .d
= 0 .α
= 0 .
9 β = 0 .d
= 0 .α
= 0 . 9 β = 0 .d = 0 . For each group, the ﬁrst row contains input images transformed with different scaling and shifting parameter α, β ( α = 1 . 0 , β = 0 .
0 is the original image) and the second row contains the found adversarial examples.
d represents the distortion of adversarial perturbations.
For models from Madry et al.
and Kolter & Wong (2018)
we use (cid:96) ∞ norm and for models from Sinha et al.
we use (cid:96) 2 norm.
Adversarial examples for these transformed images can be found with small distortions d .