Original: U NIVERSAL A TTACKS ON E QUIVARIANT N ETWORKS
Revised: U NIVERSAL A TTACKS ON E QUIVARIANT N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Adversarial attacks on neural networks perturb the input at test time in order to fool trained and deployed neural network models.
Revised: Adversarial attacks on neural networks perturb the input at test time in order to fool trained and deployed neural network models.

Original: Most attacks such as gradientbased Fast Gradient Sign Method (FGSM) by Goodfellow et al.
Revised: Most attacks such as gradientbased Fast Gradient Sign Method (FGSM) by Goodfellow et al.

Original: (2015) and DeepFool by Moosavi-Dezfooli et al.
Revised: (2015) and DeepFool by Moosavi-Dezfooli et al.

Original: (2016) are input-dependent, small, pixel-wise perturbations, and they give different attack directions for different inputs.
Revised: (2016) are input-dependent, small, pixel-wise perturbations, and they give different attack directions for different inputs.

Original: On the other hand, universal adversarial attacks are input-agnostic and the same attack works for most inputs.
Revised: On the other hand, universal adversarial attacks are input-agnostic and the same attack works for most inputs.

Original: Translation or rotation-equivariant neural network models provide one approach to prevent universal attacks based on simple geometric transformations.
Revised: Translation or rotation-equivariant neural network models provide one approach to prevent universal attacks based on simple geometric transformations.

Original: In this paper, we observe an interesting spectral property shared by all of the above input-dependent, pixel-wise adversarial attacks on translation and rotation-equivariant networks.
Revised: In this paper, we observe an interesting spectral property shared by all of the above input-dependent, pixel-wise adversarial attacks on translation and rotation-equivariant networks.

Original: We exploit this property to get a single universal attack direction that fools the model on most inputs.
Revised: We exploit this property to get a single universal attack direction that fools the model on most inputs.

Original: Moreover, we show how to compute this universal attack direction using principal components of the existing input-dependent attacks on a very small sample of test inputs.
Revised: Moreover, we show how to compute this universal attack direction using principal components of the existing input-dependent attacks on a very small sample of test inputs.

Original: We complement our empirical results by a theoretical justiﬁcation, using matrix concentration inequalities and spectral perturbation bounds.
Revised: We complement our empirical results by a theoretical justiﬁcation, using matrix concentration inequalities and spectral perturbation bounds.

Original: We also empirically observe that the top few principal adversarial attack directions are nearly orthogonal to the top few principal invariant directions.
Revised: We also empirically observe that the top few principal adversarial attack directions are nearly orthogonal to the top few principal invariant directions.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural network-based models achieve state of the art results on several speech and visual recognition tasks but these models are known to be vulnerable to various adversarial attacks.
Revised: Neural network-based models achieve state of the art results on several speech and visual recognition tasks but these models are known to be vulnerable to various adversarial attacks.

Original: Szegedy et al.
Revised: Szegedy et al.

Original: (2013) show that small, pixel-wise changes that are almost imperceptible to the human eye can make neural networks models grossly misclassify.
Revised: (2013) show that small, pixel-wise changes that are almost imperceptible to the human eye can make neural networks models grossly misclassify.

Original: They try to maximize the prediction error of a given model by ﬁnding a small pixel-wise perturbation using box-constrained L-BFGS.
Revised: They try to maximize the prediction error of a given model by ﬁnding a small pixel-wise perturbation using box-constrained L-BFGS.

Original: Goodfellow et al.
Revised: Goodfellow et al.

Original: (2015) propose the Fast Gradient Sign Method (FGSM) as a faster approach to ﬁnd such an adversarial perturbation given by x (cid:48)
Revised: (2015) propose the Fast Gradient Sign Method (FGSM) as a faster approach to ﬁnd such an adversarial perturbation given by x (cid:48)

Original: =
Revised: =

Original: x + (cid:15) sign ( ∇ x J ( θ, x, y )) , where x denotes the input, y denotes the class label, θ denotes the model parameters, and J ( θ, x, y ) denotes the loss function used to train the model.
Revised: x + (cid:15) sign ( ∇ x J ( θ, x, y )) , where x denotes the input, y denotes the class label, θ denotes the model parameters, and J ( θ, x, y ) denotes the loss function used to train the model.

Original: In subsequent work on FGSM, its iterative variant called Projected Gradient Descent (PGD) was introduced by Kurakin et al.
Revised: In subsequent work on FGSM, its iterative variant called Projected Gradient Descent (PGD) was introduced by Kurakin et al.

Original: (2017); also see Madry et al.
Revised: (2017); also see Madry et al.

Original: Goodfellow et al. (2015) and Madry et al.
Revised: Goodfellow et al. (2015) and Madry et al.

Original: (2018) study adversarial perturbations from the (cid:96) ∞ -ball around the input x , namely, each pixel value is perturbed by a quantity within [ − (cid:15), + (cid:15) ] .
Revised: (2018) study adversarial perturbations from the (cid:96) ∞ -ball around the input x , namely, each pixel value is perturbed by a quantity within [ − (cid:15), + (cid:15) ] .

Original: DeepFool by Moosavi-Dezfooli et al.
Revised: DeepFool by Moosavi-Dezfooli et al.

Original: (2016) gives a method to compute the minimal (cid:96) 2 -norm perturbation.
Revised: (2016) gives a method to compute the minimal (cid:96) 2 -norm perturbation.

Original: Broadly, all the above-mentioned adversarial attacks give directions to perturb any given test input in a way that is model-dependent and input-dependent.
Revised: Broadly, all the above-mentioned adversarial attacks give directions to perturb any given test input in a way that is model-dependent and input-dependent.

Original: Tramer et al.
Revised: Tramer et al.

Original: (2017) also mention model-agnostic perturbations using the direction of the difference between the intra-class means, in an attempt to understand how adversarial attacks transfer across different models.
Revised: (2017) also mention model-agnostic perturbations using the direction of the difference between the intra-class means, in an attempt to understand how adversarial attacks transfer across different models.

Original: Universal adversarial attacks are input-agnostic so that the same perturbation or input transformation fools the trained model on most or nearly all test inputs.
Revised: Universal adversarial attacks are input-agnostic so that the same perturbation or input transformation fools the trained model on most or nearly all test inputs.

Original: Recent work by Moosavi-Dezfooli et al.
Revised: Recent work by Moosavi-Dezfooli et al.

Original: (2016; 2017) on universal adversarial attacks looks at the curvature of the decision boundary.
Revised: (2016; 2017) on universal adversarial attacks looks at the curvature of the decision boundary.

Original: Their universal attacks or perturbations are more sophisticated than the simple and fast, gradient-based adversarial perturbations, and require signiﬁcantly more computation.
Revised: Their universal attacks or perturbations are more sophisticated than the simple and fast, gradient-based adversarial perturbations, and require signiﬁcantly more computation.

Original: Moosavi-Dezfooli et al.
Revised: Moosavi-Dezfooli et al.

Original: (2017) give a theoretical analysis of existence of universal adversarial attacks using directions in which the decision boundary has positive curvature.
Revised: (2017) give a theoretical analysis of existence of universal adversarial attacks using directions in which the decision boundary has positive curvature.

Original: Geometric transformations such as rotations, translations are simple input transformations that are model-agnostic and input-agnostic.
Revised: Geometric transformations such as rotations, translations are simple input transformations that are model-agnostic and input-agnostic.

Original: Geometric transformations of input at test time are also possible natural attacks; see  Gilmer et al.
Revised: Geometric transformations of input at test time are also possible natural attacks; see Dumont et al. (2018), Gilmer et al.

Original: One way to counter such attacks is to use neural network models that are translation and rotation-equivariant by construction.
Revised: One way to counter such attacks is to use neural network models that are translation and rotation-equivariant by construction.

Original: Standard Convolutional Neural Networks (StdCNNs) are translation-equivariant but not equivariant with respect to other spatial symmetries such as rotations, reﬂections etc.
Revised: Standard Convolutional Neural Networks (StdCNNs) are translation-equivariant but not equivariant with respect to other spatial symmetries such as rotations, reﬂections etc.

Original: Variants of CNNs to achieve rotation-equivariance and other symmetries have received much attention recently, notably, Harmonic Networks (H-Nets) by Worrall et al.
Revised: Variants of CNNs to achieve rotation-equivariance and other symmetries have received much attention recently, notably, Harmonic Networks (H-Nets) by Worrall et al.

Original: (2016), cyclic slicing and pooling by Dieleman et al.
Revised: cyclic slicing and pooling by Dieleman et al.

Revised: (2016), Tranformation-Invariant Pooling (TI-Pooling) by Laptev et al.

Original: (2016), Tranformation-Invariant
Revised: (2016), Group-equivariant

Original: Pooling (TI-Pooling) by Laptev et al. (2016), Group-equivariant Convolutional Neural Networks (GCNNs) by Cohen & Welling (2016), Steerable CNNs by Cohen & Welling (2017), Deep Rotation 
Revised: Convolutional Neural Networks (GCNNs) by Cohen & Welling (2016), Steerable CNNs by Cohen & Welling (2017), Deep Rotation Equivariant Networks (DREN) by Li et al. (2017), Rotation

Original: EquivariantNetworks (DREN) by Li et al. (2017), Rotation Equivariant Vector Field Networks (RotEqNet) by Marcos et al.
Revised: Equivariant Vector Field Networks (RotEqNet) by Marcos et al.

Original: Polar Transformer Networks (PTN) by Esteves et al.
Revised: Polar Transformer Networks (PTN) by Esteves et al.

Original: Among these, GCNNs are based on steerable ﬁlters, have a solid theoretical justiﬁcation Kondor & Trivedi (2018), and achieve nearly state of the art results on MNIST-rot 1 and CIFAR10 as reported in Esteves et al.
Revised: Among these, GCNNs are based on steerable ﬁlters, have a solid theoretical justiﬁcation Kondor & Trivedi (2018), and achieve nearly state of the art results on MNIST-rot 1 and CIFAR10 as reported in Esteves et al.

Original: However, both StdCNNs and GCNNs do require rotation augmentation at training time to be robust to rotations.
Revised: However, both StdCNNs and GCNNs do require rotation augmentation at training time to be robust to rotations.

Original: The above discussion raises an important question: are there any interesting properties shared by translation and rotation-equivariant neural network models that can be exploited to come up with simple universal attacks?
Revised: The above discussion raises an important question: are there any interesting properties shared by translation and rotation-equivariant neural network models that can be exploited to come up with simple universal attacks?

Original: Can one ﬁnd a universal attack direction using only a small fraction of test inputs but that fools the trained model on most test inputs?
Revised: Can one ﬁnd a universal attack direction using only a small fraction of test inputs but that fools the trained model on most test inputs?

Original: We answer both of these questions afﬁrmatively in this paper.
Revised: We answer both of these questions afﬁrmatively in this paper.

Original:  S UMMARY OF OUR RESULTS
Revised:  S UMMARY OF OUR RESULTS

Original: In this paper, we mostly study StdCNNs and GCNNs, which are translation-equivariant and rotationequivariant, respectively.
Revised: In this paper we mostly study StdCNNs and GCNNs, which are translation-equivariant and rotationequivariant, respectively.

Original: Some of our experiments also include  RotEqNets.
Revised: Some of our experiments also include Fully Connected NNs and RotEqNets.

Original:  Our ﬁrst observation is that even though the adversarial attack directions based on gradients, FGSM and DeepFool for StdCNN and GCNN models are input-dependent, overall they have only a small number of dominant principal components.
Revised: • Our ﬁrst observation is that even though the adversarial attack directions based on gradients, FGSM and DeepFool for StdCNN and GCNN models are input-dependent, overall they have only a small number of dominant principal components.

Original: For example, the top 5 principal components of the gradient directions on all test inputs contain more than 10% of the total spectrum for StdCNNs, GCNNs as well as RotEqNet models trained on  MNIST.
Revised: For example, the top(or 5/784 ≈ 0.64%) principal components of the gradient directions on all test inputs contain around 10% of the total spectrum for StdCNNs, GCNNs as well as RotEqNets trained on MNIST, Fashion MNIST.

Revised: For CIFAR10 the top 5 (or 5/3072 ≈ 0.16%) principal components of the gradient directions contain ≈ 1% of the total spectrum and this goes upto 3% when train and test augmented with a larger range of random rotations.

Original:  Our second observation is that a small pixel-wise perturbation in the direction of the top principal component alone can be used as a universal attack to fool these models on more than 80-90% test inputs.
Revised: • Our second observation is that a small pixel-wise perturbation in the direction of the top principal component alone can be used as a universal attack to fool these models on around 80-90% test inputs for MNIST and Fashion MNIST, and around 60-70% test inputs for CIFAR10.

Revised: And our universal attack becomes better as we train these models with larger rotations.

Original:  Our third observation is that the top principal component can be well-approximated using only 1% sample of FGSM or DeepFool attack directions on the test data, and this approximation can still give a smal pixel-wise perturbation to fool the model on more than 80% of test inputs.
Revised: • Our third observation is that the top principal component can be well-approximated using only 1% sample of FGSM or DeepFool attack directions on the test data, and this approximation can still give a small pixel-wise perturbation to fool the model on around 80% of test inputs for MNIST and Fashion MNIST, and around 60% of test inputs for CIFAR10.

Original: We give a theoretical justiﬁcation of this phenomenon using matrix concentration inequalities and spectral perturbation bounds.
Revised: We give a theoretical justiﬁcation of this phenomenon using matrix concentration inequalities and spectral perturbation bounds.

Original: An interesting aspect of our empirical study is that these observations hold across multiple input-dependent attacks (gradient, FGSM, DeepFool) and the choice of equivariant neural network models (StdCNNs, GCNNs).
Revised: An interesting aspect of our empirical study is that these observations hold across multiple input-dependent attacks (gradient, FGSM, DeepFool) and the choice of equivariant neural network models (StdCNNs, GCNNs).

Original:  P RINCIPAL COMPONENTS OF ATTACK AND INVARIANT DIRECTIONS
Revised:  P RINCIPAL COMPONENTS OF ATTACK AND INVARIANT DIRECTIONS

Original: We take a StdCNN or GCNN network as given in Table 2 and train it with 50,000 MNIST  training data augmented with random rotations in the range [ − 180 ◦ , +180 ◦ ] .
Revised: We take a StdCNN or GCNN network as given in Table 2 and train it with 60,000 samples for MNIST and Fashion MNIST, and 50,000 samples for CIFAR10 training data augmented with random rotations in the range [ − 180 ◦ , +180 ◦ ] .

Revised: We have also performed some of the experiments for NN (fully connected neural networks) and RotEqNet.

Original: We then use the gradient (similarly FGSM, DeepFool) directions obtained for this trained model on the 10,000MNIST test inputs (which are also augmented with random rotations in the range [ − 180 ◦ , +180 ◦ ] ).
Revised: We then use the gradient (similarly FGSM, DeepFool) directions obtained for this trained model on the 10,000 test inputs (which are also augmented with random rotations in the range [ − 180 ◦ , +180 ◦ ] ).

Original: We form a matrix whose rows are unit vectors along these 10,000 gradient (similarly FGSM, DeepFool) directions and obtain is
Revised: We form a matrix whose rows are unit vectors along these 10,000 gradient (similarly FGSM, DeepFool) directions and obtain its Singular Value

Original: Singular Value Decomposition (SVD).
Revised: Decomposition (SVD).

Original: For gradients, we observe, as in Figure (1), that the top 5 or less than 1% in number of the squared singular values of this matrix add up to more than 10% of the overall sum of squared singular values.
Revised: For gradients, we observe, as in Figure (1), that the top 5 or less than 1% in number of the squared singular values of this matrix add up to more than 10% of the overall sum of squared singular values.

Original: This is irrespective of whether the trained model is  StdCNN, GCNN or RotEqNet.
Revised: This is irrespective of whether the trained model is NN, StdCNN, GCNN or RotEqNet.

Original:  Figures  (2a), (3a), and (4a) show how the sigular values drop when we consider gradient, FGSM, DeepFool directions for  StdCNNs, GCNNs, 
Revised: For MNIST, Figures (18a), (2a), (3a), and (17a) show how the sigular values drop when we consider gradient, FGSM, DeepFool directions for NNs, StdCNNs, GCNNs, RotEqNets, respectively.

Original: RotEqNets, respectively.
Revised: For Fashion MNIST, Figures (21a), (19a) and (20a) show how the sigular values drop when we consider gradient, FGSM, DeepFool directions for NNs, StdCNNs, GCNNs, respectively. For CIFAR10, Figures (4a) and (5a) show how the sigular values drop when we consider gradient, FGSM, DeepFool directions for StdCNNs and GCNNs, respectively.

Original: These indicate that the drop in singular values is a common phenomenon for the different attacks and different translation and rotation-equivariant neural network models we consider.
Revised: These indicate that the drop in singular values is a common phenomenon for the different attacks and different translation and rotation-equivariant neural network models we consider.

Original: [Table]

Original: We next consider the principal components of invariant directions.
Revised: We next consider the principal components of invariant directions.

Original: For this, the MNIST test data is augmented with random rotations in  [ − 180 ◦ , +180 ◦ ] , and we look at the difference vectors between such images and their small 2 ◦ rotations.
Revised: For this, the MNIST/Fashion MNIST/CIFAR10 test data is augmented with random rotations in the range [ − 180 ◦ , +180 ◦ ] , and we look at the difference vectors between such images and their small 2 ◦ rotations.

Original: For  StdCNN, GCNN or RotEqNet trained with rotation augmentation, we expect the difference between a small rotation of an image and the image itself to be along the invariant subspace tangent to the level sets of the loss function.
Revised: For NN, StdCNN, GCNN or RotEqNet trained with rotation augmentation, we expect the difference between a small rotation of an image and the image itself to be along the invariant subspace tangent to the level sets of the loss function.

Original: We take the principal components of these difference or invariant directions, and observe that the top 5 singular vectors of the adversarial directions are nearly orthogonal to the top 5 singular vectors of the invariant directions, i.e., their average dot product is  less than 0 .
Revised: We take the principal components of these difference or invariant directions, and observe that the top 5 singular vectors of the adversarial directions are nearly orthogonal to the top 5 singular vectors of the invariant directions, i.e., their average dot product is roughtly less than 0 .

Original: Figures  (2b), (3b), (4b) indicate that this property is shared by  StdCNNs,GCNNs and RotEqNets [Table]
Revised: Figures (18b), (2b), (3b), (17b) indicate that this property is shared by NNs, StdCNNs,

Original:  and also holds when we look at a smaller sample of only 500 test points instead of all 10,000 test points.
Revised: GCNNs and RotEqNets for MNIST, and also holds when we look at a smaller sample of onlytest points instead of all 10,000 test points.

Revised: Similarly Figures (21b), (19b) and (20b) indicate that this property is shared by NNs, StdCNNs, GCNNs for Fashion MNIST.

Revised: And Figures (4b), (5b) indicate that this property is shared by StdCNNs and GCNNs for CIFAR10.

Original: Table 1 shows deeper analysis using principal angles between subspaces that the two 5-dimensional SVD subspaces of adversarial directions and invariant directions, respectively, have nearly 90 ◦ principal angles.
Revised: Table 1 shows deeper analysis using principal angles between subspaces that the two 5-dimensional SVD subspaces of adversarial directions and invariant directions, respectively, have nearly 90 ◦ principal angles.

Original: The intensity maps of top singular vectors for the gradient, FGSM, DeepFool directions for  StdCNNs, GCNNs and RotEqNets also have interesting structure as seen in Figures (8)-(16).
Revised: The intensity maps of top singular vectors for the gradient, FGSM, DeepFool directions for NNs, StdCNNs, GCNNs and RotEqNets also have interesting structure as seen in Figures (36)-(54).

Original: As we have already observed, these are nearly orthogonal to the invariant directions which lead to steer- [Figure] [Figure] [Figure]
Revised: As we have already observed, these are nearly orthogonal to the invariant directions which lead to steerable ﬁlters similar to Figure (35).

Original: able ﬁlters similar to Figure (7).

Original: We believe this underlying structure is useful and of independent interest.
Revised: We believe this underlying structure is useful and of independent interest.

Original:  U NIVERSAL ADVERSARIAL ATTACK USING A SMALL SAMPLE
Revised:  U NIVERSAL ADVERSARIAL ATTACK USING A SMALL SAMPLE

Original: We now plot the fooling rate or the fraction of test inputs that the trained model missclassiﬁes, when we perturb all of them using the top singular vector of gradient direction scaled by (cid:15) that denotes the norm of perturbation.
Revised: We now plot the fooling rate or the fraction of test inputs that the trained model misclassiﬁes, when we perturb all of them using the top singular vector of gradient direction scaled by (cid:15) that denotes the norm of perturbation.

Original:  Figures (5) and (6) plot these fooling rate to show that we can fool 80-90% of test inputs using the same small-norm universal adversarial attack.
Revised: In Figures (6)-(9) we plot these fooling rate to show that we can fool 80-90% of test inputs using the same small-norm universal adversarial attack for MNIST dataset. In Figures (10)-(13) we plot these fooling rate to show that we can fool 60-70% of test inputs using [Table] [Figure]

Original:  Moreover, if we pick a small sample of 100 random test points (about 1% of test data) and take their gradients, their top singular vector also gives a good universal adversarial attack direction with a comparable fooling rate, namely, simultaneously fooling the model on roughly 80% of test points using the same small-norm perturbation.
Revised: the same small-norm universal adversarial attack for CIFAR10 dataset. Moreover, if we pick a small sample of 100 random test points (about 1% of test data) and take their gradients, their top singular vector also gives a good universal adversarial attack direction with a comparable fooling rate, namely, simultaneously fooling the model on roughly 80% of test points using the same smallnorm perturbation for MNIST dataset and roughly 60% for CIFAR10 datset.

Revised: Please see Figures (22)-(29) in the appendix for additional experiments.

Original: A NALYSIS OF UNIVERSAL ADVERSARIAL PERTURBATIONS
Revised: Figures (7), (9) for MNIST and (11), (13) for CIFAR10 show that a simple approach using the top singular vector of the gradient directions on a small sample gives a universal adversarial attack comparable to Moosavi-Dezfooli et al. (2017) which we denote in the plots as M-DFFF. For applying [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: Figures (14)-(16) and (30)-(34) show how the fooling rate of our adversarial attack gets better when the training is augmented with larger rotations

Original: In this section, we attempt to provide a theoretical justiﬁcation for the existence of universal adversarial perturbations.
Revised: In this section we attempt to provide a theoretical justiﬁcation for the existence of universal adversarial perturbations.

Original: Assume that our data is in d dimensions X ⊆ R d with an underlying distribution given by probability density µ ( x ) at each point x ∈ X .
Revised: Assume that our data is in d dimensions X ⊆ R d with an underlying distribution given by probability density µ ( x ) at each point x ∈ X .

Original: Let a x be an adversarial perturbation direction for each x , respectively.
Revised: Let a x be an adversarial perturbation direction for each x , respectively.

Original: Note that a x could be obtained by any of the several available methods such as the Fast Gradient Sign Method (FGSM), the Kurakin attack, DeepFool or by ﬁnding the nearest point on the decision boundary etc.
Revised: Note that a x could be obtained by any of the several available methods such as the Fast Gradient Sign Method (FGSM), the Kurakin attack, DeepFool or by ﬁnding the nearest point on the decision boundary etc.

Original: Two important questions are: (a) Why does there exist a universal vector or direction that works as an adversarial perturbation for many or most data points simultaneously?
Revised: Two important questions are: (a) Why does there exist a universal vector or direction that works as an adversarial perturbation for many or most data points simultaneously?

Original: (b) Can we compute such a universal adversarial perturbation by looking only at a small subsample of the data?
Revised: (b) Can we compute such a universal adversarial perturbation by looking only at a small subsample of the data?

Original: Given any method that obtains adversarial perturbations for individual data points, if the matrix of these adversarial perturbation directions taken over all data points satisﬁes a certain property, then we show that the top singular vectors of this matrix are good candidates for universal adversarial perturbations that make many points to be misclassiﬁed simultaneously.
Revised: Given any method that obtains adversarial perturbations for individual data points, if the matrix of these adversarial perturbation directions taken over all data points satisﬁes a certain property, then we show that the top singular vectors of this matrix are good candidates for universal adversarial perturbations that make many points to be misclassiﬁed simultaneously.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let X ⊆
Revised: Let X ⊆

Original: R
Revised: R

Original: d be any given input data with the underlying probability density µ ( x ) at x ∈ X .
Revised: d be any given input data with the underlying probability density µ ( x ) at x ∈ X .

Original: Let a x denote an adversarial perturbation vector at point x ∈ X .
Revised: Let a x denote an adversarial perturbation vector at point x ∈ X .

Original: Let 0 ≤ λ ≤ 1 be the top eigenvalue of the matrix M ∈ R d × d deﬁned as
Revised: Let 0 ≤ λ ≤ 1 be the top eigenvalue of the matrix M ∈ R d × d deﬁned as

Original: [Equation]
Revised: [Equation]

Original: and let v be its corresponding eigenvector.
Revised: and let v be its corresponding eigenvector.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: In particular, plugging in δ = (cid:112) λ/ 2 we getPr
Revised: In particular, plugging in δ = (cid:112) λ/ 2 we get

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Deﬁne S
Revised: Deﬁne S

Original: = { x : |(cid:104) a x , v (cid:105)| ≥ δ(cid:107) a x (cid:107) 2 } ⊆ X .
Revised: = { x : |(cid:104) a x , v (cid:105)| ≥ δ

Original: Since λ is the top eigenvalue of M with v as its corresponding eigenvector,

Original: [Equation]
Revised: [Equation]

Original: because (cid:107) v (cid:107) 2 ==
Revised: a x (cid:107) 2 } ⊆ X .

Original: Pr
Revised: Since λ is the top eigenvalue of M with v as its corresponding eigenvector,

Original: [Equation]
Revised: [Equation]

Original: S
Revised: S

Original: [Equation]
Revised: [Equation]

Original: +
Revised: because (cid:107)

Original: [Equation]
Revised: [Equation]

Original: Pr [Equation] This implies that Pr ( S ) ≥ ( λ − δ 2 ) / (1 − δ 2 ) .
Revised: This implies that Pr ( S ) ≥ ( λ − δ 2 ) / (1 − δ 2 ) .

Original: Observe that tr ( M ) = 1 .
Revised: Observe that tr ( M ) = 1 .

Original: Theorem 1 implies that as the top eigenvalue λ dominates the spectrum, its eigenvector v get more aligned with the adversarial perturbations directions a x / (cid:107) a x (cid:107) 2 for most points x ∈ X .
Revised: Theorem 1 implies that as the top eigenvalue λ dominates the spectrum, its eigenvector v get more aligned with the adversarial perturbations directions a x / (cid:107) a x (cid:107) 2 for most points x ∈ X .

Original: This means that v is a potential candidate for a universal adversarial perturbation.
Revised: This means that v is a potential candidate for a universal adversarial perturbation.

Original: More generally, Theorem 1 works for any of the top eigenvalues of M and their corresponding eigenvectors, and thus, gives a subspace spanned by multiple orthogonal directions all of which are potential candidates for being universal adversarial perturbations that fool a given classiﬁer on many input examples.
Revised: More generally, Theorem 1 works for any of the top eigenvalues of M and their corresponding eigenvectors, and thus, gives a subspace spanned by multiple orthogonal directions all of which are potential candidates for being universal adversarial perturbations that fool a given classiﬁer on many input examples.

Original: Now let’s consider the second question of ﬁnding a good approximation to our candidate universal adversarial perturbation or the top eigenvector v of M using only a small subsample of X .
Revised: Now let’s consider the second question of ﬁnding a good approximation to our candidate universal adversarial perturbation or the top eigenvector v of M using only a small subsample of X .

Original: Theorem 2 shows that only a small sample of size independent of | X | but depending only on the dimension d of the data and the spectral properties of the matrix M sufﬁces.
Revised: Theorem 2 shows that only a small sample of size independent of | X | but depending only on the dimension d of the data and the spectral properties of the matrix M sufﬁces.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let X ⊆
Revised: Let X ⊆

Original: R
Revised: R

Original: d be any given input data with the underlying probability density µ ( x ) at x ∈ X .
Revised: d be any given input data with the underlying probability density µ ( x ) at x ∈ X .

Original: Let a x denote an adversarial perturbation vector at point x ∈ X .
Revised: Let a x denote an adversarial perturbation vector at point x ∈ X .

Original: Let 0 ≤ λ ≤ 1 be the top eigenvalue of the matrix M ∈ R d × d deﬁned as
Revised: Let 0 ≤ λ ≤ 1 be the top eigenvalue of the matrix M ∈ R d × d deﬁned as

Original: [Equation]
Revised: [Equation]

Original: and let v be its corresponding eigenvector.
Revised: and let v be its corresponding eigenvector.

Original: Let x 1 , x 2 , . . .
Revised: Let x 1 , x 2 , . . .

Original: , x m be a i.i.d.
Revised: , x m be a i.i.d.

Original: sample of m points from
Revised: sample of m points from

Original: X drawn from the distribution deﬁned by probability density µ , and let ˜ λ be the top eigenvalue of the matrix ˜ M ,
Revised: X drawn from the distribution deﬁned by probability density µ , and let ˜ λ be the top eigenvalue of the matrix ˜ M ,

Original: [Equation]
Revised: [Equation]

Original: and ˜ v be the top eigenvector of ˜ M .
Revised: and ˜ v be the top eigenvector of ˜ M .

Original: Also suppose that there is a gap of at least γλ between the top eigenvalue λ and the second eigenvalue of M .
Revised: Also suppose that there is a gap of at least γλ between the top eigenvalue λ and the second eigenvalue of M .

Original: Then for any 0 ≤ (cid:15) < γ and m =
Revised: Then for any 0 ≤ (cid:15) < γ and m =

Original: O ( (cid:15) − 2 d log d ) , we get (cid:107) v − ˜ v (cid:107) 2 ≤ (cid:15)/ ( γ − (cid:15) ) , with a constant probability.
Revised: O ( (cid:15) − 2 d log d ) , we get (cid:107) v − ˜ v (cid:107) 2 ≤ (cid:15)/ ( γ − (cid:15) ) , with a constant probability.

Original: This probability can be boosted to 1 − δ by having an additional log(1 /δ ) in the O ( · ) .
Revised: This probability can be boosted to 1 − δ by having an additional log(1 /δ ) in the O ( · ) .

Original: Proof.
Revised: Proof.

Original: Using matrix Bernstein inequality from Tropp (2015), we get that for m =
Revised: Using matrix Bernstein inequality from Tropp (2015), we get that for m =

Original: O ( (cid:15) − 2 d log d ) , we have that the spectral or the operator norm (cid:13)(cid:13)(cid:13) M − ˜ M (cid:13)(cid:13)(cid:13) 2 ≤ (cid:15)λ , with a constant probability.
Revised: O ( (cid:15) − 2 d log d ) , we have that the spectral or the operator norm (cid:13)(cid:13)(cid:13) M − ˜ M (cid:13)(cid:13)(cid:13) 2 ≤ (cid:15)λ , with a constant probability.

Original: Now we can use results about spectral perturbation bounds from Bhatia (1997).
Revised: Now we can use results about spectral perturbation bounds from Bhatia (1997).

Original: By Weyl’s theorem, this implies (cid:12)(cid:12)(cid:12) λ − ˜ λ
Revised: By Weyl’s theorem, this implies (cid:12)(cid:12) (cid:12) λ − ˜ λ

Original: (cid:12)(cid:12)(cid:12) ≤ (cid:15)λ .
Revised: (cid:12)(cid:12) (cid:12) ≤ (cid:15)λ .

Original: If there is gap of at least γλ between the ﬁrst and the second eigenvalue of
Revised: If there is gap of at least γλ between the ﬁrst and the second eigenvalue of

Original: M with γ > (cid:15) , then by Davis-Kahan theorem we get that (cid:107) v − ˜ v (cid:107) 2 ≤ (cid:15)/ ( γ − (cid:15) ) , with a constant probability.
Revised: M with γ > (cid:15) , then by Davis-Kahan theorem we get that (cid:107) v − ˜ v (cid:107) 2 ≤ (cid:15)/ ( γ − (cid:15) ) , with a constant probability.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We show how to use a small sample of input-dependent adversarial attack directions on test inputs to ﬁnd a single universal attack directions that fools translation and rotation-equivariant models such as CNNs and GCNNs on a large fraction of test inputs.
Revised: We show how to use a small sample of input-dependent adversarial attack directions on test inputs to ﬁnd a single universal attack direction that fools translation and rotation-equivariant models such as CNNs and GCNNs on a large fraction of test inputs.

Original: Our main observation is a spectral property shared by different attacks directions such as gradients, FGSM, DeepFool on these models.
Revised: Our main observation is a spectral property shared by different attacks directions such as gradients, FGSM, DeepFool on these models.

Original: We give a theoretical justiﬁcation for how this spectral property helps in using the top singular vector as a universal attack direction computed using only a small sample of test inputs.
Revised: We give a theoretical justiﬁcation for how this spectral property helps in using the top singular vector as a universal attack direction computed using only a small sample of test inputs.

Original: A D ETAILS OF EXPERIMENTS
Revised: A D ETAILS OF EXPERIMENTS

Original: All experiments performed on neural network-based models were done using  MNIST dataset with appropriate augmentations applied to the train/validation/test set.
Revised: All experiments performed on neural network-based models were done using MNIST, Fashion MNIST and CIFAR10 datasets with appropriate augmentations applied to the train/validation/test set.

Revised: Data sets MNIST 2 dataset consists of 70 , 000 images of 28 × 28 size, divided into 10 classes.

Original: The collection of gradients and other attack vectors and the testing of the universal perturbations were on the same network.
Revised: 55 , 000 used for training, 5 , 000 for validation and 10 , 000 for testing.

Original: Data sets MNIST 2 dataset consists of 70 , 000 images of 28 × 28 size, divided into 10 classes.
Revised: Fashion MNIST 3 dataset consists of 70 , 000 images of 28 × 28 size, divided into 10 classes.

Original: 55 , 000 used for training, 5 , 000 for validation and 10 , 000 for testing. {+CIFAR10 
+}
Revised: 55 , 000 used for training, 5 ,for validation and 10 , 000 for testing. {+CIFAR10 
+}

Revised: dataset consists of 60 , 000 images of 32 × 32 size, divided into 10 classes.

Revised: 40 , 000 used for training, 10 , 000 for validation and 10 , 000 for testing.

Original:  Model Architectures For the MNIST  based experiments we use the 7 layer architecture of GCNN similar to Cohen & Welling (2016).
Revised: [Table] Model Architectures For the MNIST and Fashion MNIST based experiments we use the 7 layer architecture of GCNN similar to Cohen & Welling (2016).

Original: The StdCNN architecture is similar to the GCNN except  the operations are as per CNNs.
Revised: The StdCNN architecture is similar to the GCNN except that the operations are as per CNNs.

Original: Refer to Table 2 for details.
Revised: Refer to Table 2 for details.

Original: RotEqNet architecture is are given in 
Revised: RotEqNet architecture is as given in Marcos et al.

Revised: The NN architecture is a 2 layer fully connected with 784-50 neurons in the layers with dropout.

Original: Marcos et al. 
Revised: For the CIFAR10 based experiments we use the ResNet18 architecture as in He et al. and it’s equivalent in GCNN as given in Cohen & Welling (2016).

Revised: Input training data was augmented with random cropping and random horizontal ﬂips, apart from the speciﬁc range of rotation augmentation as needed.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]