U NIVERSAL A DVERSARIAL A TTACK USING V ERY F EW T EST E XAMPLES
Anonymous authors
Paper under double-blind review
A BSTRACT
Adversarial attacks such as Gradient-based attacks, Fast Gradient Sign Method ( FGSM ) by Goodfellow et al.
(2015) and DeepFool by Moosavi-Dezfooli et al.
(2016) are input-dependent, small pixel-wise perturbations of images which fool state of the art neural networks into misclassifying images but are unlikely to fool any human.
On the other hand a universal adversarial attack is an input-agnostic perturbation.
The same perturbation is applied to all inputs and yet the neural network is fooled on a large fraction of the inputs.
In this paper, we show that multiple known input-dependent pixel-wise perturbations share a common spectral property.
Using this spectral property, we show that the top singular vec tor of input-dependent adversarial attack directions can be used as a very simple universal adversarial attack on neural networks
We evaluate the error rates and fooling rates of three universal attacks,
SVD
-Gradient, SVD-DeepFool and SVD-
FGSM on state of the art neural networks
We show that these universal attack vectors can be computed using a small sample of test inputs.
We establish our results both theoretically and empirically
On VGG and
VGG
, the fooling rate of SVD-DeepFool and SVD
-Gradient perturbations constructed from observing less than
2% of the validation set of ImageNet is as good as the universal attack of Moosavi-Dezfooli et al.
(2017a).
To prove our theoretical results, we use matrix concentration inequalities and spectral perturbation bounds.
For completeness, we also discuss another recent approach to universal adversarial perturbations based on ( p q ) -singular vectors, proposed independently by Khrulkov & Oseledets (2018), and point out the simplicity and efﬁciency of our universal attack as the key difference.
I NTRODUCTION
Neural network models achieve state of the art results on several image classiﬁcation tasks.
However, these models are known to be vulnerable to adversarial attacks.
Szegedy et al.
(2013) show that small, pixel-wise changes that are almost imperceptible to the human eye can make neural network models grossly misclassify images.
Regarding an image as a vector x in R d , an adversarial, pixelwise perturbation gives a small perturbation vector v ∈ R d to perturb x so that the network classiﬁes x + v differently from x .
Szegedy et al.
ﬁnd such a perturbation vector using box-constrained
L-BFGS.
Goodfellow et al.
(2015) propose the Fast Gradient Sign Method ( FGSM ) as a faster approach to ﬁnd such an adversarial perturbation.
In subsequent work on FGSM , its iterative variant was introduced by Kurakin et al.
(2017) and another version called Projected Gradient Descent ( PGD ) by Madry et al.
In the works of Goodfellow et al.
(2015) and Madry et al.
, the adversarial perturbations considered are from an (cid:96) ∞ -ball around the input x - namely, each pixel value is perturbed by a quantity within [ − (cid:15), + (cid:15) ] .
DeepFool by Moosavi-Dezfooli et al.
(2016) gives a method to compute a minimal (cid:96) 2 -norm perturbation to fool the input.
To understand how adversarial attacks transfer across different models Tramer et al.
(2017) also study model-agnostic perturbations using the direction of the difference between the intra-class means.
Universal adversarial perturbations are input-agnostic so that the same perturbation fools the trained model on a large fraction of test inputs.
Recent work by Moosavi-Dezfooli et al.
(2017a;b) on universal adversarial perturbations looks at the curvature of the decision boundary.
Their universal attacks are more sophisticated than the simple and fast, gradient-based adversarial perturbations, and require signiﬁcantly more computation.
In Moosavi-Dezfooli et al.
(2017b)
the authors give a theoretical analysis of existence of universal adversarial perturbations using directions in which the decision boundary has positive curvature.
Khrulkov & Oseledets (2018) propose a different method to construct a universal adversarial perturbation.
Their perturbation is based on computing the ( p, q ) -singular vectors of the Jacobian matrices of hidden layers of the network.
The above discussion raises an important question: are there any interesting properties shared by the input dependent adversarial attacks which can be exploited to come up with an easy to compute universal perturbation?
Can one ﬁnd such a universal perturbation using only a small fraction of test inputs and also ensure that a perturbation along this direction fools most test inputs?
We answer both of these questions afﬁrmatively in this paper.
Furthermore, we show that our universal attack compares favourably with the universal attack in Moosavi-Dezfooli et al.
(2017a).
S UMMARY OF OUR MAIN RESULTS
• Our ﬁrst observation is that even though the adversarial attack directions are inputdependent, overall they have only a small number of dominant principal components.
We show this for attacks based on the Gradient of the loss function, the FGSM attack and DeepFool, on a variety of architectures and datasets.
• Construct a matrix whose i -th row is an input dependent attack vector on the i -th test point.
Our second observation is that a small pixel-wise perturbation in the direction of the top principal component of this matrix alone can be used as a universal attack.
This simple algorithm gives us SVD-Gradient, SVD-FGSM and SVD-DeepFool perturbations.
• Our third observation is that the top principal component can be well-approximated from a very small sample of the test data, and small samples are good enough to obtain SVDUniversal perturbations comparable to that in Moosavi-Dezfooli et al.
(2017a).
• We give a theoretical justiﬁcation of this phenomenon using matrix concentration inequalities and spectral perturbation bounds.
This observation holds across multiple inputdependent adversarial attack directions given by Gradient , FGSM and DeepFool
D EFINITIONS AND THE PROBLEM FORMULATION
Assume that we have a distribution D on pairs of images and labels, with images coming from a set
X .
In the case of CIFAR-10 for example we can think of X to be a set of all 32 × 32 images with pixel values from [0 , 1] .
Let ( X, Y ) be a sample from D and let f : X → [ k ] , be a k -class classiﬁer.
The natural accuracy δ of the classiﬁer f is deﬁned to be
[Equation]
We regard a deterministic adversary to be a function A : X → X .
The error rate of A on the classiﬁer f is deﬁned to be
[Equation]
When A is a distribution over functions, we get a randomized adversary.
The norm of the perturbation applied to X is the norm of A ( X ) (we only consider (cid:96) 2 norm in this paper).
In Moosavi-Dezfooli et al.
(2017a;b) and in Khrulkov & Oseledets (2018)
the authors consider the fooling rate of an adversary and not the error rate.
Recall that f is said to be fooled on input x by the perturbation
A ( x ) if f ( x ) (cid:54) =
f ( x + A ( x )) .
And the fooling rate is deﬁned to be the probability that f is fooled by choosing x at random.
Note that
[Equation]
So, if the natural accuracy of the classiﬁer is high, the fooling rate is close to the error rate.
[Equation]
The error rate of the adversary with zero perturbation is the error rate of the trained network, whereas the fooling rate of the adversary with zero perturbation is necessarily zero.
However, small fooling rate does not necessarily imply small error rate, especially when the natural accuracy is not close to 100%.
Note that existing models such as VGG16, VGG19, ResNet50 do not achieve natural accuracy greater than 0.8.
3. I NPUT DEPENDENT ATTACKS
There are a number of input dependent attacks known and our generic algorithm can be used to produce a universal adversary from each of them.
We demonstrate the effectiveness of our method by applying it to a few popular input dependent attacks, which we discuss below.
Let L ( θ, x, y ) denote the loss function used to train the model.
Here θ are the model parameters, x is the input and y is the class label.
In the Gradient attack the adversarial example produced on input x is
[Equation]
In the FGSM attack proposed by Goodfellow et al.
the adversarial example produced on input x is
[Equation]
So the perturbed input is within an (cid:96) ∞ ball of radius (cid:15) about x .
DeepFool proposed by Moosavi-Dezfooli et al.
(2016) is an iterative algorithm.
At each stage of the iteration the region of space around the current perturbed input x t where the classiﬁer outputs label f ( x t ) is approximated by a polyhedron P t .
x t is projected to the closest face of P t .
We stop at the smallest t for which f ( x ) (cid:54) =
f
( x t ) .
3. E XISTING UNIVERSAL ATTACKS
In Moosavi-Dezfooli et al. (2017a), the authors ﬁx a budget (cid:15) , a bound on the allowed (cid:96) 2 norm of the attack 1 and also the desired fooling rate.
They pick a sample S of the training data.
Starting with the zero perturbation v , the algorithm proceeds in stages as long as the fooling rate of the current perturbation vector v is less than the desired fooling rate on S .
They run over x ∈ S , for which f ( x ) =
f ( x + v ) .
For each such x they ﬁnd the smallest perturbation r x sending x to the boundary (using DeepFool) and v is updated to Π( v + r x ) where Π is the projection onto the ball of radius (cid:15) .
In Khrulkov & Oseledets (2018) the authors consider the outputs f i ( x ) of the i -th hidden layer of a network on an input x .
Denote by J i ( x ) the matrix of partial derivatives ∂f i ( x ) ∂x evaluated at x .
They pick a sample S of X of size m and compute for each x ∈ S , J i ( x ) .
Stacking these in different rows of a matrix J , they use as their universal attack the vector argmax ||
Jv || q subject to ||
v || p = 1 .
In Mopuri et al.
(2017) the authors propose a universal attack which is completely data independent.
It only depends upon the model architecture.
Given a trained neural network with K hidden (convolution) layers, the authors start with a random image v and minimize Π i =
K i =
1 ¯ (cid:96) i ( v ) , || v || ∞ ≤ (cid:15) .
Here ¯ (cid:96) i ( v ) is the mean activation of layer
i when the input is v .
The authors show that the perturbation v achieving the minimum is a data agnostic attack and achieves a reasonable fooling rate.
U NIVERSAL ADVERSARIAL ATTACK USING A SMALL SAMPLE 4. SVD-U NIVERSAL A LGORITHM
Algorithm 1 gives the pseudocode to construct a universal attack for a given model.
The output of the algorithm is a universal attack vector, and this depends upon the input dependent attack used.
It is the simplicity of the algorithm which we would like to emphasize.
The algorithm samples a set of n images from the test set.
We use the terms batch size and sample size interchangeably.
For each of the sample points it computes an input-dependent attack direction.
We stack these attack directions in
Data: Network N , any input-dependent adversarial attack A , and n test samples.
Result:
A universal attack direction for network N
Obtain input-dependent perturbation vectors a 1 , ..., a n based on A for the n test samples.
2 Normalize a
i ’s to get only the attack directions or unit vectors
u i = a i / ||
a
i || 2 , for i = 1 to n .
3 Form a matrix M whose rows are u 1 , u 2 , . . . , u n .
4 Compute Singular Value Decomposition (SVD) of M as M = USV T , with
V =
[ v 1 | v 2 | . .
| v n ] .
5 Return the top right singular vector v 1 as the universal attack vector.
4. E RROR RATES AND FOOLING RATES OF SVD-U NIVERSAL
We evaluate the fooling rate of SVD-Gradient, SVD-FGSM and SVD-DeepFool on the validation set of the ImageNet dataset on VGG16, VGG19 and ResNet50.
We used off the shelf code available for these architectures.
In these architectures pixel intensities of images are scaled down and images are normalized before they are deployed for use in classiﬁers.
Our (unit) attack vectors are constructed using batch sizes of 64(0.13%), 128(0.26%)) and 1024(2%).
In Figure 1, we plot the fooling rate of (cid:15) v as a function of (cid:15) , in all the architectures, where v is obtained using a batch size of 64.
The top left plot is the fooling rate of SVD-gradient, the top right that of SVD-FGSM and the bottom left that of SVD-DeepFool.
We explain the bottom right plot in the next section.
SVD-Gradient attack scaled to have norm 50 fools about 32% of the validation set of ImageNet on VGG19.
Note that the average (cid:96) 2 norm of this validation set is 450and using the language of Moosavi-Dezfooli et al. (2017a), a perturbation with norm 50 is also quasi-imperceptible.
We visualize the perturbed images in Figure 4 in Appendix A.1.
We plot the error rate of SVD-Universal for ImageNet on VGG19 in Figure 5, Appendix A.1, on different batch sizes.
It is clear from the plots that SVD-Universal attacks obtained using batch size of 64 perform as well (and sometimes better) as attack vectors obtained using larger batch sizes.
In Appendix A.2 we plot the error rate of SVD-Universal attacks on CIFAR-10 trained on ResNet18.
On ResNet18, the error rate of SVD-DeepFool obtained from 100 samples of the CIFAR-10 test data is 0.55 when it is scaled to norm 15 (0.23 of the average norm), see Figure 6.
We notice that the
SVD-Universal perturbations constructed with small batch sizes perform better.
The visualizations of our perturbations as images are given in Appendix B.
4. C OMPARISON WITH M OOSAVI -D EZFOOLI ET AL . (2017 A ), K HRULKOV & O SELEDETS (2018)
In this section we compare our results with that reported in Moosavi-Dezfooli et al. (2017a) and
Khrulkov & Oseledets (2018) on the validation set of Imagenet. The pixel intensities of the dataset used in the experiments of Moosavi-Dezfooli et al. (2017a) and Khrulkov & Oseledets (2018) are in the range [0 , 255] . The average (cid:96) 2 norm of the validation set is 50,000 and the average (cid:96) ∞ norm is 250, Moosavi-Dezfooli et al. (2017a, Footnote, Page 4).
The (cid:96) 2 norm of the universal perturbation used in Moosavi-Dezfooli et al. (2017a) is 1/25 of 50,000 = 2000.
The (cid:96) ∞ norm of the universal perturbation used in Khrulkov & Oseledets (2018) is 1/25 of 250 = 10. The pixel intensities of the validation set of Imagenet used in our experiments is normalized and the average (cid:96) 2 norm of this normalized dataset is 450. In (Khrulkov & Oseledets, 2018, Figure 9) the authors report that the universal perturbation of Moosavi-Dezfooli et al.
(2017a)
constructed from a batch size 64 and having (cid:96) ∞ norm 10 has a fooling rate of 0.14 on VGG19.
A comparable perturbation in our model has (cid:96) 2 norm 1/25 of 450 = 18.
Figure 1 shows that SVD-Gradient and SVD-DeepFool perturbation with norm 18 give a fooling rate of 0.13 on VGG19.
To further compare our universal perturbations with that of Moosavi-Dezfooli et al. (2017a), the same 64 samples used to construct our SVD-Universal vectors were used to obtain the universal perturbation vector, M-DFFF, of Moosavi-Dezfooli et al.
(2017a).
This vector was scaled down to get a unit vector w in (cid:96) 2 norm.
In Figure 1 (right bottom) we plot the fooling rate of M-DFFF (cid:15)w as a function of (cid:15) , in all the architectures.
We observe trends similar to what was reported in Moosavi-Dezfooli et al.
(2017a, Table 1) - the error rate achieved by M-DFFF on VGG16 is higher than that of M-DFFF on VGG19.
Observe from this ﬁgure that on VGG19, the M-DFFF universal perturbation of (cid:96) 2 norm 18 gives a fooling rate of 0 . 13 .
The plots in Figure 1 of the fooling rates of the four universal attacks on VGG16 and ResNet50 are are collated together in Figure 2.
We also plot the error rate of these universal attacks on VGG and ResNet50.
SVD-DeepFool and M-DFFF have comparable fooling and error rates for ImageNet on the two networks.
Khrulkov & Oseledets (2018) get a fooling rate of more than 0 . 4 using batch size of 64 and (cid:96) ∞ norm 10. At this norm their universal attack is stronger than both our attack and that of MoosaviDezfooli et al.
(2017a). However Khrulkov & Oseledets (2018) do an extensive experimentation and determine which intermediate layer to attack and ( p, q ) are also optimized to maximize the fooling rate of their ( p, q ) -singular vector.
However ( p, q ) -SVD computation is expensive and is known to be a hard problem, (Bhaskara & Vijayaraghavan, 2011; Bhattiprolu et al., 2019).
We do no such optimization and use p = q = 2 , our emphasis being on the simplicity of our SVD-Universal algorithm.
We also observe a trend similar to what is reported by Khrulkov & Oseledets (2018) - the error rate of SVD-Universal attacks is higher on VGG16 and VGG19 than on ResNet50.
Figure 7 in Appendix A.2 compares the error rates of SVD-Universal attacks and M-DFFF constructed using samples of size 100 from the test data of CIFAR-10 on ResNet18.
Error rates of
SVD-DeepFool and M-DFFF are comparable - for this dataset, SVD-Gradient is not as good.
When the norm of the perturbation is about 15 (0.23 of the average norm), the error rate of SVD-DeepFool fool is about 0.55. [Figure]
T HEORETICAL A NALYSIS OF O UR U NIVERSAL PERTURBATION
In this section, we attempt to provide a theoretical justiﬁcation for the existence of universal adversarial perturbations.
Let our data be given by a joint distribution D on feature vectors in R d and corresponding true labels in [ k ] = { 1 , 2 , . . .
, k } .
Let ( X, Y ) denote a random sample from D .
Let f : R d → [ k ] be a given classiﬁer, and for any x ∈ R d , let A ( x ) be its adversarial perturbation given by a ﬁxed adversarial attack A , e.g., FGSM , DeepFool .
As in Moosavi-Dezfooli et al.
(2017b), the fooling rate of this attack is given by Pr ( f ( X ) (cid:54) =
f ( X + A ( X ))) , where the probability is over the data distribution D .
Deﬁne A
= { x : f ( x ) (cid:54) =
f ( x + A ( x ) } .
For any x ∈ A , assume that x + A ( x ) lies on the decision boundary, and let
the hyperplane H x = { x + z ∈ R d : (cid:104) z , A ( x ) (cid:105) =
(cid:107)A ( x ) (cid:107) 22 } be a local, linear approximation to the decision boundary at x + A ( x ) .
This holds for adversarial attacks such as DeepFool by Moosavi-Dezfooli et al.
(2016) which try to ﬁnd an adversarial perturbation A ( x ) such that x + A ( x ) is the nearest point to x on the decision boundary.
Now consider the halfspace
S x = { x + z ∈ R d : (cid:104) z , A ( x ) (cid:105) ≥ (cid:107)A ( x ) (cid:107) 22 } .
Note that x / ∈ S x and x + A ( x ) ∈ S x .
For simplicity of analysis, we assume that f ( x ) (cid:54) =
f ( x + z ) , for all x ∈ A and x + z ∈ S x .
This is a reasonable assumption in a small neighborhood of x , and is implied by the positive curvature of the decision boundary assumed in the analysis of Moosavi-Dezfooli et al.
(2017b).
In other words, we assume that if an adversarial perturbation such as A ( x ) fools the model at input x , then any perturbation z , where z has a sufﬁcient projection along A ( x ) , also fools the model at x , and this is a reasonable assumption in a small neighborhood of x .
Theorem 1.
Given any joint data distribution D on features or inputs in R d and true labels in [ k ] , let ( X, Y ) denote a random sample from D .
For any x ∈ R d , let A ( x ) be its adversarial perturbation by a ﬁxed input-dependent adversarial attack A .
Let
[Equation]
and 0 ≤ λ ≤ 1 be the top eigenvalue of M and v ∈ R
d be the corresponding eigenvector (normalized to have unit (cid:96) 2 norm).
Then under the assumption that f ( x ) (cid:54) =
f
( x + z ) , for all x ∈ A and x + z ∈ S x , we have
[Equation]
where u = ± ( (cid:15)/δ ) v , where (cid:15) =
max x (cid:107)A ( x ) (cid:107) 2 .
Proof.
Let µ ( x ) denote the induced probability density on features or inputs by the distribution D .
Deﬁne A
= { x : f ( x ) (cid:54) =
f ( x + A ( x ) } and G = { x : |(cid:104)A ( x ) , v (cid:105)| ≥ δ (cid:107)A ( x ) (cid:107) 2 } .
Since λ is the top eigenvalue of M with v as its corresponding (unit) eigenvector,
[Equation]
(cid:90) x
[Equation]
G µ ( x ) d x + δ(cid:90) x / ∈ G µ ( x ) d x because (cid:107) v (cid:107)
[Equation]
Thus, Pr ( G ) ≥ ( λ − δ 2 ) / (1 − δ 2 ) , and equivalently, Pr ( G c ) = 1 − Pr ( G ) ≤ (1 − λ ) / (1 − δ 2 ) .
Now for any x ∈ G , we have |(cid:104)A ( x ) , v (cid:105)| ≥ δ (cid:107)A ( x ) (cid:107) 2 .
Letting (cid:15) =
max x (cid:107)A ( x ) (cid:107) 2 , we get |(cid:104)A ( x ) , ( (cid:15)/δ ) v (cid:105)| ≥ (cid:107)A ( x ) (cid:107) 22 .
Thus, x ± ( (cid:15)/δ ) v ∈ S x , where S x = { x + z ∈ R d : (cid:104) z , A ( x ) (cid:105) ≥ (cid:107)A ( x ) (cid:107) 22 } , and therefore, by our assumption stated before Theorem 1, we have f ( x ) (cid:54) =
f ( x + u ) , where u = ± ( (cid:15)/δ ) v .
Putting all of this together, Pr ( f ( X + u ) (cid:54) =
f ( X )) ≥ Pr ( G ∩ A ) ≥
Pr ( A ) − Pr
A ∩ G c ) ≥ Pr ( A )
− Pr ( G c ) , and therefore,
[Equation]
Theorem 1 shows that any norm-bounded, input-dependent, adversarial attack A can be converted into a universal attack u of comparable norm, without losing much in the fooling, if the top eigenvalue λ of M is close to 1 .
This universal attack direction lies in the one-dimensional span of the top eigenvector v of M .
The proof of Theorem 1 can be easily generalized to the top SVD subspace of M and where the top few eigenvalues of M dominate its spectrum (note that tr ( M ) = 1 ).
Singular value drop We empirically verify our hypothesis about top eigenvalue (or the top few eigenvalues) dominating the spectrum of M in Theorem 1.
Let X 1 , X 2 , . . .
, X m be m i.i.d.
samples of X drawn from the distribution D and consider the unnormalized, empirical analog of M as follows,
[Equation]
Figure (3) shows how the singular values drop for the three input dependent attacks, Gradient ,
FGSM , and DeepFool on CIFAR-10 trained on ResNet18 on batch sizes 500 and 10,000.
These plots indicate that the drop in singular values is a shared phenomenon across different input dependent attacks, and the trend is similar even when we look at a small number of input samples.
Our second contribution is ﬁnding a good approximation to the universal adversarial perturbation given by the top eigenvector v of M , using only a small sample X 1 , X 2 , . . .
, X m from the distribution D .
Theorem 2 shows that we can efﬁciently pick such a small sample whose size is independent of D , depends linearly on the intrinsic dimension of M , and logarithmically on the feature dimension d .
Theorem 2.
Given any joint data distribution D on features in R d and true labels in [ k ] , let ( X, Y ) denote a random sample from D .
For any x ∈ R d , let A ( x ) denote the adversarial perturbation of x according a ﬁxed input-dependent adversarial attack A .
Let
[Equation]
Let 0 ≤ λ = (cid:107) M (cid:107) 2 ≤ 1 denote the top eigenvalue of M and let v denote its corresponding eigenvector (normalized to have unit (cid:96) 2 norm).
Let r = tr ( M ) / (cid:107) M (cid:107) 2 be the intrinsic dimension of
M .
Let X 1 , X 2 , . . .
, X m be m i.i.d.
samples of X drawn from the distribution D , and let ˜ λ
= (cid:13)(cid:13)(cid:13) ˜ M
(cid:13)(cid:13)(cid:13)be the top eigenvalue of the matrix ˜ M ,
[Equation]
and ˜ v be the top eigenvector of ˜ M .
Also suppose that there is a gap of at least γλ between the top eigenvalue λ and the second eigenvalue of M .
Then for any 0 ≤ (cid:15) < γ and m =
O ( (cid:15) − 2 r log d ) , we get (cid:107) v − ˜ v (cid:107) 2 ≤ (cid:15)/γ , with a constant probability.
This probability can be boosted to 1 − δ by having an additional log(1 /δ ) in the O ( · ) .
Proof.
Take m =
O ( (cid:15) − 2 r log d ) .
By the covariance estimation bound in Theorem 4 and Markov’s inequality, we get that (cid:13)(cid:13)(cid:13) M − ˜ M
(cid:13)(cid:13)(cid:13) 2 ≤ (cid:15)λ , with a constant probability.
Applying Theorem 5 due to Weyl on eigenvalue perturbation, we get (cid:12)(cid:12) (cid:12) λ − ˜ λ
(cid:12)(cid:12) (cid:12) ≤ (cid:15)λ .
Moreover, if there is gap of at least γλ between the ﬁrst and the second eigenvalue of M with γ > (cid:15) , we can use the well-known Theorem 6 of Davis-Kahan to bound the difference between the corresponding eigenvectors as (cid:107) v − ˜ v (cid:107) 2 ≤ (cid:15)/γ , with a constant probability.
Please see Appendix D, and the book by Vershynin (2018) cited therein, for more details about the covariance estimation bound, Weyl’s theorem, and Davis-Kahan theorem.
The theoretical bounds are weaker than our empirical observations about the number of test samples needed for the universal attack.
What we want to highlight is that the bound in Theorem 2 is independent of the support of underlying data distribution D and depends only on its dimensionality d .
There is more room to tighten our analysis using additional properties of the data distribution as well as the spectral properties of M .
U NIVERSAL INVARIANT PERTURBATIONS AND FUTURE DIRECTIONS
Let x be an image for which the predicted label of a neural network classiﬁer f is f ( x ) .
For highly accurate classiﬁers trained on data that already contains small rotation augmentations, one would expect f ( x +∆( x ))
=
f
( x ) , where x +∆( x ) is the image resulting from a small rotation applied to x .
For an image x , we call I ( x ) an invariant perturbation for x , if f ( x + I ( x )) =
f ( x ) .
Using this terminology we expect ∆( x ) to be an invariant perturbation for x for classiﬁers trained on data that is inherently (or additionally) augmented with small rotations.
A natural question is whether there are any universal invariant perturbations x + u that are invariant for most x .
We show empirically that there are such universal invariant perturbations.
By their very deﬁnition we would also expect that universal invariant perturbations will not be universal adversarial perturbations.
We show this empirically by plotting the error rates of the top singular invariant perturbation of CIFAR-10 on
ResNet18 in Figure 10, Appendix C.
The error rate is below 0.2 even when the norm of the invariant perturbation is scaled up to 50.
We give more empirical evidence of such invariant perturbations - we show that the principal angles between the subspace of the top 5 universal adversarial perturbations and the subspace of the top 5 universal invariant perturbations are all close to 90 ◦ .
The empirical and theoretical results in this paper hold in greater generality than what we have presented.
Our experiments indicate the existence of universal adversarial perturbations and universal invariant perturbations for datasets trained on equivariant networks.
GCNN’s
Cohen & Welling (2016) are among the more popular equivariant networks, and achieve state of the art on CIFAR-10.
In Appendix C we give a peek into this work and show plots of the error rates of SVD-Universal on
CIFAR-10 on GCNN’s.
We also plot the singular values of CIFAR-10 on GCNN’s.
It is common to augment data with rotations and train neural networks, so they become equivariant to rotations and so an image and its rotation are classiﬁed the same.
We believe that a study of how the subspaces of universal adversarial perturbations and universal invariant perturbations change as a model is trained to handle more and more rotations will be important.
This is ongoing work.
C ONCLUSION
We show how to use a small sample of input-dependent adversarial attack directions on test inputs to ﬁnd a single universal adversarial perturbation that results in a large drop in the accuracy of state of the art neural network models.
Our main observation is a spectral property shared by different attacks directions such as Gradients , FGSM , DeepFool .
We give a theoretical justiﬁcation for how this spectral property helps in using the top singular vector as a universal attack direction.
We justify theoretically and empirically that such a perturbation can be computed using only a small sample of test inputs.
Our empirical results also indicate that these universal perturbations are almost orthogonal to perturbations of inputs that these models are invariant to.
A PPENDIX A D ATA SETS AND MODEL ARCHITECTURES USED
Data sets CIFAR-10 dataset consists of 60 , 000 images of 32 × 32 size, divided into 10 classes.
40 , 000 used for training, 10 , 000 for validation and 10 , 000 for testing.
ImageNet dataset refers to ILSRVC 2012 dataset (Russakovsky et al., 2015) which consists of images of 224 × 224 size, divided into 1000 classes.
We use the validation set of 50 , 000 images for our testing purposes.
All experiments performed on neural network-based models were done using the validation set of
ImageNet and test set of CIFAR-10 datasets.
Model Architectures For the ImageNet based experiments we use pre-trained networks of
VGG16, VGG19 and ResNet50 architectures 3 .
For the CIFAR-10 based experiments we use the
ResNet18 architecture as in He et al.
A. SVD-U NIVERSAL ON I MAGE N ET
Figure 4 shows that the images with SVD-DeepFool perturbations upto norm 100 are quasiimperceptible.
Figure 5 shows the error rate of SVD-Universal on VGG19 calculated with varying sample size.
[Figure]
A. SVD-U NIVERSAL ON CIFAR-
In this section we plot the error rates of SVD-Universal on CIFAR-10 trained on ResNet
A PPENDIX B V ISUALIZING SVD-U NIVERSAL PERTURBATIONS
We visualize the top singular vectors for the Gradient , FGSM , DeepFool directions from ImageNet and CIFAR-10 in this section.
A PPENDIX C I NVARIANT PERTURBATIONS C. P RINCIPAL COMPONENTS OF ATTACK AND INVARIANT DIRECTIONS
To construct the matrix, we pick a batch of 100 images, and for each image x in our sample we compute the difference between x and its 2 degree rotation.
We stack these (input dependent invariant directions) as rows of the matrix and compute the top singular (unit) vector w .
We call w the universal invariant perturbation.
Figure 10 shows the error rate of (cid:15) w as a function of (cid:15) on CIFAR-10.
The curves are ﬂat indicating that these directions are invariant perturbations.
Recall that for two subspaces V , W , the ﬁrst principal angle is deﬁned as the minimum angle between two unit vectors v 1 ∈ V , w 1 ∈ W .
The second principal angle is the minimum angle between unit vectors v 2 ∈ V, w 2 ∈ W , with v 2 ⊥ v 1 and w 2 ⊥ w 1 .
The other principal angles are deﬁned similarly.
[Table]
A PPENDIX D P ROOF DETAILS
Theorem 5.6.1 (General covariance estimation) from Vershynin (2018) bounds the spectral norm of covariance matrix estimated from a small number of samples as follows.
Theorem 3.
Let X be a random vector in R d , d ≥ 2 .
Assume that for some K ≥ 1 , (cid:107) X (cid:107) 2 ≤
K (cid:16) E (cid:104) (cid:107) X (cid:107) 22 (cid:105)(cid:17) 1 / 2 , almost surely.
Let Σ = E (cid:2) XX T (cid:3) be the covariance matrix of X and Σ m = 1 m (cid:80) mi =1 X
i X Ti be the estimated covariance from m i.i.d.
samples X 1 , X 2 , . . .
, X m .
Then for every positive integer m , we have
[Equation]
for some positive constant C and (cid:107) Σ (cid:107) 2 being the spectral norm (or the top eigenvalue) of Σ .
Note that using m =
O ( (cid:15) − 2 d log d ) we get E [ (cid:107) Σ m − Σ (cid:107) 2 ] ≤ (cid:15) (cid:107) Σ (cid:107) 2 .
A tighter version of Theorem 5.6.1 appears as Remark 5.6.3, when the intrinsic dimension r = tr (Σ) / (cid:107) Σ (cid:107) 2 (cid:28) d .
Theorem 4.
Let X be a random vector in R d , and d ≥ 2 .
Assume that for some K ≥ 1 , (cid:107) X (cid:107) 2 ≤
K (cid:16) E (cid:104) (cid:107) X (cid:107) 22 (cid:105)(cid:17) 1 / 2 , almost surely.
Let Σ = E (cid:2) XX T (cid:3) be the covariance matrix of X and Σ m = 1 m (cid:80) mi =1 X
i X Ti be the estimated covariance from m i.i.d.
samples X 1 , X 2 , . . .
, X m .
Then for every positive integer m , we have
[Equation]
for some positive constant C and (cid:107) Σ (cid:107) 2 being the operator norm (or the top eigenvalue) of Σ .
Note that using m =
O ( (cid:15) − 2 r log d ) we get E [ (cid:107) Σ m − Σ (cid:107) 2 ] ≤ (cid:15) (cid:107) Σ (cid:107) 2 .
Theorem 4.5.3 (Weyl’s Inequality) from Vershynin (2018) upper bounds the difference between i -th eigenvalues of two symmetric matrices A and B using the spectral norm of A − B .
Theorem 5.
For any two symmetric matrices A and B in R d × d , | λ i ( A ) − λ i ( B ) | ≤ (cid:107) A − B (cid:107) 2 , where λ i ( A ) and λ i ( B ) are the i -th eigenvalues of A and B , respectively.
In other words, the spectral norm of matrix perturbation bounds the stability of its spectrum.
Here is a special case of Theorem 4.5.5 (Davis-Kahan Theorem) and its immediate corollary mentioned in Vershynin (2018).
Theorem 6.
Let A and B be symmetric matrices in R d × d .
Fix i ∈
[ d ] and assume that the largest eigenvalue of A is well-separated from the rest of the spectrum, that is, λ 1 ( A ) − λ 2 ( A ) ≥ δ > 0 .
Then the angle θ between the top eigenvectors v 1 ( A ) and v 1 ( B ) of A and B , respectively, satisﬁes sin θ ≤ 2 (cid:107) A − B (cid:107) 2 /δ .
As an easy corollary, it implies that the top eigenvectors v 1 ( A ) and v 1 ( B ) are close to each other up to a sign, namely, there exists s ∈ {− 1 , 1 } such that
[Equation]
A PPENDIX E E RROR RATES OF CIFAR-10 ON GCNN’ S
For a fair comparison, the GCNN conﬁguration used for CIFAR-10 is the same as that of ResNet18, except that the operations going from one layer to the next are replaced by equivalent GCNN operations, as given in Cohen & Welling (2016).
[Table]
A PPENDIX F T ABLES R EQUESTED BY R EVIEWERS
[Table]
[Table]