I NVARIANCE VS . R OBUSTNESS OF N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural networks achieve human-level accuracy on many standard datasets used in image classiﬁcation.
The next step is to achieve better generalization to natural (or non-adversarial) perturbations as well as known pixel-wise adversarial perturbations of inputs.
Previous work has studied generalization to natural geometric transformations (e.g., rotations) as invariance , and generalization to adversarial perturbations as robustness .
In this paper, we examine the interplay between invariance and robustness .
We empirically study the following two cases: (a) change in adversarial robustness as we improve only the invariance using equivariant models and training augmentation, (b) change in invariance as we improve only the adversarial robustness using adversarial training.
We observe that the rotation invariance of equivariant models (StdCNNs and GCNNs) improves by training augmentation with progressively larger rotations but while doing so, their adversarial robustness does not improve, or worse, it can even drop signiﬁcantly on datasets such as MNIST.
As a plausible explanation for this phenomenon we observe that the average perturbation distance of the test points to the decision boundary decreases as the model learns larger and larger rotations.
On the other hand, we take adversarially trained LeNet and ResNet models which have good (cid:96) ∞ adversarial robustness on MNIST and CIFAR-10, and observe that adversarially training them with progressively larger norms keeps their rotation invariance essentially unchanged.
In fact, the difference between test accuracy on unrotated test data and on randomly rotated test data upto ± θ ◦ , for all θ ∈ [0 , 180] , remains essentially unchanged after adversarial training .
As a plausible explanation for the observed phenomenon we show empirically that the principal components of
adversarial perturbations and perturbations given by small rotations are nearly orthogonal.
I NTRODUCTION
Neural networks achieve state of the art, human-level accuracy on several standard datasets used in image classiﬁcation.
However, their performance in the wild depends on how well they can handle natural or non-adversarial transformations of input seen in real-world data as well as known deliberate, adversarial attacks created to fool the model.
Natural or non-adversarial transformations seen in real-world images include translations, rotations, and scaling.
Convolutional Neural Networks (CNNs) are translation-invariant or shift-invariant by design.
Invariance to other symmetries, and especially rotations, have received much attention recently, e.g., Harmonic Networks (H-Nets) by Worrall et al.
(2016), cyclic slicing and pooling by Dieleman et al.
(2016), Tranformation-Invariant Pooling (TI-Pooling) by Laptev et al.
Group-equivariant Convolutional Neural Networks (GCNNs) by Cohen & Welling (2016), Steerable
CNNs by Cohen & Welling (2017), Deep Rotation Equivariant Networks (DREN) by Li et al.
Rotation Equivariant Vector Field Networks (RotEqNet) by Marcos et al.
Polar Transformer
Networks (PTN) by Esteves et al.
For a given symmetry group G , a G -equivariant network learns a representation or feature map at every intermediate layer such that any transformation g ∈ G applied to an input corresponds to an equivalent transformation of its representations.
Any model can improve its invariance to a given group of symmetries through sufﬁcient training augmentation.
Equivariant models use efﬁcient weight sharing and require smaller sample complexity to achieve better invariance.
Equivariant models such as CNNs and GCNNs too generalize well to progressively larger random rotations but only when their training data is augmented similarly.
Adversarial attacks on neural network models are certain, deliberate changes to inputs that fool a highly accurate model but are unlikely to fool humans.
Given any neural network model, Szegedy et al.
(2013) show how to change the pixel values of images only slightly so that the change is almost imperceptible to human eye but makes highly accurate models misclassify.
Szegedy et al.
ﬁnd these adversarial pixel-wise perturbations of small magnitude by maximizing the prediction error of a given model using box-constrained L-BFGS.
Goodfellow et al.
(2015) propose Fast Gradient Sign
Method (FGSM) that applies the adversarial perturbation as x (cid:48)
=
x + (cid:15) sign ( ∇ x J ( θ, x, y )) , where x is the input, y represents the target, θ represents the model parameters, and J ( θ, x, y ) is the loss function used to train the network.
Moreover, Goodfellow et al. (2015) propose adversarial training, or training augmented with points ( x (cid:48) , y ) , as a way to improve adversarial robustness of a model.
Subsequent work has introduced multi-step variants of FGSM, notably, an iterative method by Kurakin et al.
(2017) and Projected Gradient Descent (PGD) attack by Madry et al.
Given any model, these attacks produce adversarial perturbation for every test image x from a small (cid:96) ∞ -ball around it, namely, each pixel value x i is perturbed within [ x i − (cid:15), x i + (cid:15) ] .
PGD attack does so by solving an inner optimization by projected gradient descent over (cid:96) ∞ -ball of radius (cid:15) around x to approximate the optimal perturbation.
Adversarial training with PGD perturbations improves the adversarial robustness of models, and it is one of the best known defenses to make models robust to perturbations of bounded (cid:96) ∞ norm on MNIST and CIFAR-10 datasets, as shown in Madry et al.
(2018) and Athalye et al.
Recent work has looked at simultaneous robustness to multiple adversarial attacks.
Engstrom et al.
(2017) show that adversarial training with PGD makes CNNs robust against perturbations of bounded (cid:96) ∞ norm but an adversarially chosen combination of a small rotation and a translation can still fool these models nevertheless.
Recent work of Schott et al.
(2019) shows that PGD adversarial training is a good defense against perturbations of bounded (cid:96) ∞ norm but can be broken with adversarial perturbations of small (cid:96) 0 or (cid:96) 2 norm that are also imperceptible to humans or have little semantic meaning for humans.
Schott et al.
(2019) show how to build models for MNIST dataset that are simultaneously robust to perturbations of small (cid:96) 0 , (cid:96) 2 and (cid:96) ∞ norms.
1. O UR RESULTS
In this paper, we study the simultaneous interplay between invariance of models to images randomly rotated by an angle between [ − θ ◦ , + θ ◦ ] , and their adversarial robustness to pixel-wise perturbations of (cid:96) ∞ norm at most (cid:15) .
Unlike previous studies, we do not ﬁx the magnitude (cid:15) (e.g., say (cid:15) = 0 . 3 ) of pixel-wise perturbations nor limit ourselves to small rotations upto ± 30 ◦ .
Instead, we compute the accuracy of a given model on input rotated by a random angle between [ − θ ◦ , + θ ◦ ] for θ in the range [0 , 180] .
Similarly, we compute the accuracy of a given model for an adversarial attack of (cid:96) ∞ norm at most (cid:15) for (cid:15) in the range [0 , 1] (and the underlying dataset is normalized).
We empirically study the following: (a) change in (cid:96) ∞ adversarial robustness as we improve only rotation invariance using equivariant models (StdCNNs and GCNNs) and training augmentation with progressively larger rotations, (b) change in invariance as we improve only adversarial robustness using PGD adversarial training with progressively larger (cid:96) ∞ -norm of pixel-wise perturbations.
We observe that the rotation invariance of equivariant models (StdCNNs and GCNNs) improves by training augmentation with progressively larger rotations but while doing so, their adversarial robustness does not improve, or worse, it can even drop signiﬁcantly on datasets such as MNIST.
On the other hand, we take adversarially trained LeNet and ResNet models which have good (cid:96) ∞ adversarial robustness on MNIST and CIFAR-10, and observe that adversarially training them with progressively larger norms keeps their rotation invariance essentially unchanged.
In fact, the difference between test accuracy on unrotated test data and on randomly rotated test data upto ± θ ◦ , for all θ ∈ [0 , 180] , remains essentially unchanged after adversarial training with progressively larger bounds on the (cid:96) ∞ norm of pixel-wise perturbations.
We support these two observations with plausible empirical reasons for their occurrence - distance to the decision boundary of the test points reducing in the ﬁrst case, and orthogonality of adversarial perturbations with perturbations caused by small rotations of test points.
Related Work Schott et al.
(2019) study simultaneous robustness to adversarial perturbations of small (cid:96) 0 , (cid:96) 2 , and (cid:96) ∞ -norm.
Tram`er & Boneh (2019) show an impossibility result by exhibiting data distribution where no model can have substantially better-than-random accuracy for binary classiﬁcation simultaneously against both (cid:96) ∞ and (cid:96) 1 perturbations, and also against both (cid:96) ∞ perturbations and spatial perturbations, given by an adversarial permutation of coordinates that models an adversarially chosen combination of a small rotation and a small translation.
They show an empirical validation of this claim on MNIST and CIFAR-10 datasets for simultaneous robustness against (cid:96) ∞ adversarial perturbation and an adversarially chosen combination of translation upto ± 3 pixels and rotation upto ± 30 ◦ .
R OTATION INVARIANCE VS . (cid:96) ∞ ADVERSARIAL ROBUSTNESS
In this section, we present our main result about the interplay between rotation invariance and (cid:96) ∞ adversarial robustness of models on MNIST, Fashion MNIST, and CIFAR-10 data.
In Subsection 2.2, we study how the robustness of equivariant models (StdCNNs and GCNNs) to (cid:96) ∞
adversarial perturbations changes as they become more rotation invariant through training augmentation with random rotations of progressively larger degree.
In Subsection 2.3, we study how the rotation invariance of LeNet and ResNet based architectures change as they become robust to (cid:96) ∞ adversarial perturbations through PGD adversarial training.
The models used in Subsection 2.2 were shown to achieve good rotational invariance with augmentation by Cohen & Welling (2016).
The LeNet and
ResNet models used in Subsection 2.3 were shown to be adversarially robust by Madry et al.
(2018) and reconﬁrmed in recent work by Athalye et al.
Before we begin our experiments we ﬁrst give details of the plots that our experiments will yield.
Rotation invariance means that the predicted labels of an image and any of its rotations should be the same.
Since most datasets are centered, we restrict our attention to rotations around the center of the image.
One way to quantify rotation invariance is to look at the accuracy of a given model on randomly rotated test data.
This is visualized by plotting the accuracy on test inputs rotated by a random angle between [ − θ ◦ , + θ ◦ ] , as θ varies in the range [0 , 180] .
We call this the rotation invariance proﬁle of the given model.
Adversarial robustness means that the predicted labels of an image and its adversarial pixel-wise perturbation should be the same.
Adversarial robustness of a given model to a ﬁxed adversarial attack (e.g., FGSM, PGD) and
a ﬁxed (cid:96) ∞ norm (say, 0 ≤ (cid:15) ≤ 1 ) is quantiﬁed by the accuracy of the given model on perturbed test data with a perturbation of (cid:96) ∞ norm at most (cid:15) generated by the attack.
The resulting plot of the accuracy of the model as (cid:15) varies over [0 , 1] is the robustness proﬁle of the given model.
Convention used in the legends of our ﬁgure: We use the following convention in the legends of our plots.
A coloured line labeled A/B indicates that the training data is augmented with random rotations from [ − A ◦ , A ◦ ] and the test data is augmented with random rotations from [ − B ◦ , B ◦ ] .
If
A (resp.
B ) is zero it means the training data (resp.
test data) is unrotated.
2. I MPROVING ROTATION INVARIANCE
Any model improves its rotation invariance proﬁle when its training data is augmented with progressively larger random rotations.
Equivariant models (e.g., GCNNs, H-Nets, PTNs, RotEqNets) are designed to achieve better rotation invariance thought clever weight sharing.
However, models such as GCNNs and H-Nets cannot achieve rotation invariance, if their training data is not sufﬁciently augmented.
This appears to be folklore so we do not elaborate on this in the main paper.
In Appendix E we conﬁrm this on many popular rotation equivariant models and also show empirically that they become equivariant to rotations by training with a small sample of data augmented with random rotations.
2. E FFECT OF ROTATION INVARIANCE ON l ∞ ADVERSARIAL ROBUSTNESS
For any ﬁxed θ ∈ [0 , 180] , we take an equivariant model, namely, StdCNN or GCNN, and augment its training data by random rotations from [ − θ ◦ , + θ ◦ ] .
Figures 1 and 2 show how the robustness proﬁles of StdCNN and GCNN models change on MNIST, Fashion MNIST, and CIFAR-10 datasets, as we increase the degree θ used in training augmentation of the model (thereby improving their rotation invariance).
We use PGD adversarial attack for MNIST and Fashion MNIST, FGSM and
PGD attack for CIFAR-10 dataset.
FGSM is a weaker attack for these models on CIFAR-10.
In the above experiments, the test data is unrotated but adversarially perturbed.
For completeness,
Figures 9, 10 in Appendix B, show the robustness proﬁle of the same networks when the test is also augmented with random rotations before applying adversarial perturbations.
Our experiments show that training augmentation with progressively larger rotations improves the rotation invariance of StdCNNs and GCNNs.
For example the red line in Figure 20(bottom) for
CIFAR-10 in Appendix E.2 shows that the accuracy of GCNNs drops below 70% when it is not trained with rotations but when the test has random rotations upto ± 30 ◦ .
However when the training data is augmented with random rotations upto ± 30 ◦ the accuracy improves to about 88% as can be seen from the black line in the same ﬁgure.
However training with rotations does not improve their (cid:96) ∞ adversarial robustness, and can even worsen as in the case of MNIST.
On MNIST, the (cid:96) ∞ adversarial robustness of StdCNNs and GCNNs drops sharply when they are trained with larger rotation augmentations.
When GCNN’s are trained on MNIST with rotations upto 30 ◦ (light blue line in Figure 2 (top left))
their accuracy drops to 90% under a PGD attack with (cid:15) = 0 .
3 , whereas their accuracy is about 97% with the same PGD attack when they are not trained with random rotations (dark blue line in Figure 2 (top left)).
We observe the same trend when the test is augmented with rotations, Figure 10 (left) in Appendix B.
It is alarming to note that even seemingly innocuous training augmentation with random rotations can affect adversarial robustness of models.
2. E FFECT OF (cid:96) ∞ ADVERSARIAL TRAINING ON ROTATION INVARIANCE
The most common approach to improve adversarial robustness is adversarial training, i.e., training the model on adversarially perturbed training data.
Adversarial training with PGD attack is one of the strongest known defenses on MNIST and CIFAR-10 datasets, see Athalye et al.
For any ﬁxed (cid:15) ∈ [0 , 1] and any ﬁxed θ ∈ [0 , 180] , we adversarially train our models (LeNet and
ResNet) with PGD adversarial perturbations with (cid:96) ∞ budget (cid:15) , and then plot their rotation invariance proﬁles.
Each colored line in Figure 3 corresponds to a model adversarially trained with a different value of (cid:15) .
Tsipras et al.
(2019) point out that the test accuracy on the original data drops after adversarial training, and our plots conﬁrm this - the y co-ordinate of the points where the curves begin decreases with increasing (cid:15) .
In Figure 3, we observe that the entire invariance proﬁle essentially make a parallel shift downward for LeNet and ResNet based models on MNIST and CIFAR-10 datasets, respectively.
In other words, adversarial training does not affect the relative drop in accuracy when test data is augmented with progressively larger rotations.
D ECISION BOUNDARY DISTANCE AND O RTHOGONALITY
In order to understand what could be the possible reasons behind our empirical observations in
Section 2.2 and Section 2.3 we perform more experiments.
3. A VERAGE PERTURBATION DISTANCE TO THE BOUNDARY .
For each test image, adversarial attacks ﬁnd perturbations of the test point with small (cid:96) ∞ norm that would change the prediction of the given model.
Most adversarial attacks do so by ﬁnding the directions in which the loss function of the model changes the most.
In order to explain why these networks become vulnerable to pixel-wise attacks as they learn more rotations, we see how the distance of the test points to the decision boundary changes as the networks learn larger rotations.
This is abstractly depicted in Figure 4 where we show the distance of a test point x 0 to the boundary
D 0 (resp.
D 180 ) when the model is trained with zero (resp. 180 ◦ ) rotations.
We use the L 2 attack vectors obtained by DeepFool (Moosavi-Dezfooli et al., 2016) for the datapoints under attack.
We take the norm of this attack vector as an approximation to the shortest distance of the test point to the decision boundary.
For each of the test points we collect the perturbation vectors given by DeepFool attack and report the average perturbation distance.
We plot this average distance as the datasets are augmented with larger rotations.
Our experiments show that as the networks learn larger rotations with augmentation, the average perturbation distance falls.
So as (symmetric) networks become invariant to rotations, they are more vulnerable to pixel-wise attacks.
The plots in Figures 5, 6, 7, 8 show this for StdCNNs and GCNNs on MNIST and CIFAR-10.
Figures 17 and 18 in Appendix C show the plots for Fashion-MNIST.
To make our point we plot the accuracy of these networks and also the average perturbation distance of the test points alongside in one ﬁgure.
The blue line in Figure 5(left) shows the accuracy of a StdCNN on MNIST when both the training data and test data are augmented with θ , as θ ranges from 0 to 180 .
The green line in
Figure 5(left) shows the accuracy of the StdCNN model when the train is augmented with random rotations upto θ , and the test is augmented with rotations upto θ and is also perturbed with PGD of (cid:96) ∞ norm 0 .
The red line shows the accuracy when the test is not augmented with rotations but is
PGD perturbed with (cid:96) ∞ norm 0 . 3 .
The red line Figure 5(right) shows the average perturbation distance of the unrotated test points when the network is trained with rotations
upto θ .
The green line shows the average perturbation distance of test points which are augmented with rotations upto
θ - this is about 5 when θ is 0 ◦ (the point on the y -axis where the curves begin).
As the network is trained with random rotations up to 180 ◦ the average perturbation distance of the augmented test drops below 3.5.
Figure 5(left) shows that that the PGD accuracy has dropped from around 85% for the network at 0 ◦ to 30% at 180 ◦ (the corresponding green line on the left).
The fact that the red line is above the green line is also reﬂected in Figure 11 in Appendix C. When the test is perturbed by PGD, the accuracy of the
StdCNN with training data augmented with rotations is better when the test is not augmented with rotations than if the test were also augmented with rotations.
3. O RTHOGONALITY OF A DVERSARIAL PERTURBATIONS & INVARIANT ROTATIONAL PERTURBATIONS
In general we expect neural networks to classify x and a small rotation x (cid:48) of x , identically.
This would be particularly so if the dataset has some inherent rotations and the labels remain the same under such rotations.
Writing x (cid:48)
=
x
+ ∆( x ) we say ∆( x ) is an invariant rotational perturbation
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
for x .
For a network trained to be equivariant to rotations, this rotational perturbation of x could well be orthogonal to an adversarial perturbation of x , since an adversarial perturbation changes the label whereas an invariant rotational perturbation does not change the label.
We empirically show that this is the case for MNIST and CIFAR-10.
For each test input ﬁnd we ﬁnd its PGD perturbation and stack these adversarial perturbations as rows of an adversarial matrix.
We ﬁnd the top 5 right singular vectors of this adversarial matrix.
We rotate each input by 4 ◦ / 2 ◦ for MNIST/CIFAR-10 and subtract the original image from it.
We stack such invariant rotational perturbations as rows of an invariance matrix.
We ﬁnd the top 5 right singular vectors of this invariance matrix.
We empirically observe that the principal angles between these two 5-dimensional subspaces are close to 90 ◦ .
This may explain why in Section 2.3 the rotation invariance proﬁle just shifts down.
For completeness we recall the deﬁnition of principal angles in Appendix D.
Table 1 gives the principal angles of MNIST trained on LeNet and Table 2 gives the principal angles for CIFAR-10 trained on ResNet as given by Madry et al.
C ONCLUSION
We observe that as equivariant models (StdCNNs and GCNNs) are trained with progressively larger rotations their rotation invariance improves but at the cost of their adversarial robustness.
On the other hand, adversarial training with perturbations of progressively increasing norms improves the robustness of LeNet and ResNet and also keeps the invariance unchanged.
A plausible explanation of the ﬁrst observation is that the average perturbation distance of the test points to the boundary decreases as StdCNNs and GCNNs learn larger rotations.
A plausible explanation for the second observation is that the singular vectors of adversarial perturbations are orthogonal to the singular vectors of invariant rotational perturbations.
A PPENDIX A D ETAILS OF EXPERIMENTS
All experiments performed on neural network-based models were done using MNIST, Fashion
MNIST and CIFAR-10 datasets with appropriate augmentations applied to the train/validation/test set.
Data sets MNIST dataset consists of 70 , 000 images of 28 × 28 size, divided into 10 classes.
55 ,used for training, 5 , 000 for validation and 10 , 000 for testing.
Fashion MNIST dataset consists of 70 , 000 images of 28 × 28 size, divided into 10 classes.
55 , 000 used for training, 5 , 000 for validation and 10 , 000 for testing.
CIFAR-10 dataset consists of 60 , 000 images of 32 × 32 size, divided into 10 classes.
40 , 000 used for training, 10 , 000 for validation and 10 , 000 for testing.
Model Architectures For the MNIST and Fashion MNIST based experiments we use the network architecture of GCNN as given in Cohen & Welling (2016).
The StdCNN architecture is similar to the GCNN except that the operations are as per CNNs.
Refer to Table 3 for details.
H-Nets architecture is as given in Worrall et al.
RotEqNet architecture is as given in Marcos et al.
PTN architecture is as given in Esteves et al.
For the CIFAR-10 based experiments we use the VGG16 architecture as given in Simonyan & Zisserman (2014) and its GCNN equivalent is obtained replacing the various layer operations with equivalent GCNN operations as given in Cohen & Welling (2016).
This is similar to how we obtained a GCNN architecture from StdCNN for the MNIST based experiments.
Input training data was augmented with random cropping and random horizontal ﬂips.
For the adversarial training experiments we used the LeNet based architecture for MNIST and the
ResNet architecture for CIFAR-10.
Both these models are exactly as given in Madry et al.
[Table]
A PPENDIX B E FFECT OF INVARIANCE ON ADVERSARIAL ROBUSTNESS AFTER TEST AUGMENTATION
A PPENDIX C D ECISION BOUNDARY DISTANCE , ORTHOGONALITY
A PPENDIX D P RINCIPAL ANGLES
Recall that for two subspaces V
W the ﬁrst principal angle is deﬁned as the minimum angle between two unit vectors v 1 ∈ V , w 1 ∈ W .
The second principal angle is the minimum angle between unit vectors v 2 ∈ V, w 2 ∈ W , with v 2 ⊥ v 1 and w 2 ⊥ w 1 .
The other principal angles are deﬁned similarly.
A PPENDIX E R OTATION INVARIANCE OF EQUIVARIANT NETWORKS
We observe that (a) Rotation equivariant networks are robust to only small degrees of rotations away from the ones present in the training data - this was studied only for CNNs in Weiler et al.
(2018), (b) applying data augmentation increases their invariance to rotations.
) Rotation equivariant net works achieve state of the art results with smaller sample size for training.
E. R OBUSTNESS TO ROTATIONS WITHOUT TRAINING AUGMENTATION
We ﬁrst train all the networks on MNIST and Fashion MNIST with no rotation augmentation and test against inputs augmented with varying range of random rotations from ± 0 ◦ to ± 180 ◦ .
Figure 19(top left) for MNIST and Figure 19(top right) for Fashion MNIST shows the performance of these networks to the entire range of rotations.
We observe that even though these datasets are small and these networks are designed to be invariant to rotations, their performance drops as the range of random rotations in test data increases.
For small range of rotations up to about ± 20 ◦ , the performance of these networks on MNIST remains above 95% indicating that MNIST dataset has some natural rotation augmentation.
On Fashion MNIST, the performance being above 85% when the test is augmented with small rotations up to ± 20 ◦ .
When the test is augmented with rotations beyond 25 ◦ , the accuracy of these networks drops.
We observe that PTN and RotEqNet are more robust than the other networks for MNIST and Fashion MNIST.
On the dataset CIFAR-10 we compare the performance of rotation-equivariant GCNNs with translation-equivariant StdCNNs.
This is shown in Figure 19(bottom).
GCNNs perform better, as expected, but the performance of both GCNNs and StdCNNs degrades to less than 70% accuracy when the test data is augmented with random rotations larger than ± 30 ◦ .
E. R OBUSTNESS WITH TRAINING AUGMENTATION
We train equivariant networks with input augmented with varying range of random rotations from ± 0 ◦ to ± 180 ◦ .
In Figure 20 we plot our results for MNIST (top left), Fashion MNIST (top right) and CIFAR-10 (
E. S AMPLE C OMPLEXITY OF N ETWORKS
To understand the sample complexity of the networks we perform two experiments.
In the ﬁrst we train the networks with varying sample sizes of training set and test them on the entire test set.
And in the second experiment we do the same as the ﬁrst with the inputs in train and test augmented with random rotations in the range [ − 180 ◦ , 180 ◦ ] .
From Figure 21 for MNIST, Figure 22 for Fashion
MNIST and 23 for CIFAR-10, we can see that rotation equivariant networks achieve their best per- formance safely using 10 k - 30 k training samples.
This conﬁrms that rotation equivariant networks can do well with smaller training sample size.
E. A DDITIONAL P LOTS R EQUESTED BY R EVIEWERS
Deﬁne the fooling rate of a classiﬁer f to a transformation T e.g., rotation, as the ratio of the number of { x ∈ X | f ( x ) (cid:54) =
f ( T ( x )) } / | X | .
Based on a suggestion of one of the referees, we plot the rotation invariance proﬁles of PGD adversarially trained models below using fooling rate on the y axis - compare with Figure 3, where we plot accuracy on the y axis.