Adversarial Robustness is at Odds with Lazy Training
Abstract
Recent works show that adversarial examples exist for random neural networks [Daniely and Shacham, 2020] and that these examples can be found using a single step of gradient ascent [Bubeck et al., 2021].
In this work, we extend this line of work to “ lazy training ” of neural networks – a dominant model in deep learning theory in which neural networks are provably efficiently learnable.
We show that over-parametrized neural networks that are guaranteed to generalize well and enjoy strong computational guarantees remain vulnerable to attacks generated using a single step of gradient ascent.
Introduction
Despite the tremendous success of deep learning recent works have demonstrated that neural networks are extremely susceptible to attacks.
One such vulnerability is due to arbitrary adversarial corruption of data at the time of prediction, commonly referred to as inference-time attacks [Biggio et al., 2013, Szegedy et al., 2014].
Such attacks are catastrophically powerful, especially in settings where adversaries can directly access the model parameters.
By adding crafted imperceptible perturbations on the natural input, such attacks are easily realized across various domains, including in computer vision where an autonomous driving car may misclassify traffic signs [Sitawarin et al., 2018], in natural language processing where automatic speech recognition misinterprets the meaning of a sentence [Schönherr et al., 2019], among many others.
The potential threat of adversarial examples has led to serious concerns regarding the security and trustworthiness of neural network-based models that are increasingly being deployed in real-world systems.
It is crucial, therefore, to understand why trained neural networks classify clean data with high accuracy yet remain extraordinarily fragile due to strategically induced perturbations.
One of the earliest works to describe adversarial examples was that of Szegedy et al.
[2014], motivated by the need for networks that not only generalize well but are also robust to small perturbations of its input.
This was followed by a body of works that finds adversarial examples using various methods [Goodfellow et al., 2015, Papernot et al., 2016, Madry et al., 2018, Carlini and Wagner, 2017a,b].
Although lots of defense algorithms have been designed, most of them are quickly defeated by stronger attacks [Carlini and Wagner, 2017a, Carlini et al., 2019, Tramèr et al., 2020].
Much of the prior work has focused on this empirical “arms race” between adversarial defenses and attacks; yet, a deep theoretical understanding of why adversarial examples exist has been somewhat limited.
Recently, a line of work theoretically constructs adversarial attacks for random neural networks.
Inspired by the work of Shamir et al.
[2019], Daniely and Shacham [2020] prove that small ℓ 2 -norm adversarial perturbations can be found by multi-step gradient ascent method for random ReLU networks with small width – each layer has vanishingly small width compared to the previous layer.
Bubeck et al.
[2021] generalize this result to two-layer randomly initialized networks with relatively large network width and show that a single step of gradient ascent suffices to find adversarial examples.
Bartlett et al.
[2021] further generalize this result to random multilayer ReLU networks.
However, all of the works above focus on random neural networks and fail to explain why adversarial examples exist for neural networks trained using stochastic gradient descent.
In other words, why do neural networks that are guaranteed to generalize well remain vulnerable to adversarial attacks?
In this paper, we make progress toward addressing that question.
In particular, we show that adversarial examples can be found easily using a single step of gradient ascent for neural networks trained using first-order methods.
Specifically, we focus on over-parametrized two layer ReLU networks which have been studied extensively in recent years due to their favorable computational aspects.
Several works have shown that under mild over-parametrization (size of the network is polynomial in the size of the training sample), the weights of two-layer ReLU networks stay close to initialization throughout the training of the network using gradient descent-based methods [Allen-Zhu et al., 2019, Ji and Telgarsky, 2020, Arora et al., 2019].
In such a “lazy regime” of training, the networks are locally linear which allows us to give computational guarantees for minimizing the training loss (aka, optimization) as well as bound the test loss (aka, generalization).
The purpose of our work is to show the deficiency of the lazy regime even though it is the dominant framework in the theory of deep learning.
Our key contributions are as follows.
We investigate the robustness of over-parametrized neural networks, trained using lazy training, against inference-time attacks.
Building on the ideas of Bubeck et al.
[2021], we show that such trained neural networks still suffer from adversarial examples which can be found using a single step of gradient ascent.
In particular, we show that an adversarial perturbation of size O ( ∥ x ∥√ d ) in the direction of the gradient suffices to flip the prediction of two-layer ReLU networks trained in the lazy regime – these are networks with all weights at a distance of O ( 1 √ m ) from their initialization, where m is the width of the network.
To the best of our knowledge, this is the first work which shows that networks with small generalization error are still vulnerable to adversarial attacks.
Further, we validate our theory empirically.
We confirm that a perturbation of size O (1 / √ d ) suffices to generate a strong adversarial example for a trained two-layer ReLU networks.
The rest of the paper is organized as follows.
In Section 2, we introduce the preliminaries, give the problem setup, and discuss the related work.
We present our main result in Section 3, and give a proof sketch in Section 4.
In Section 5 we provide empirical support for our theory, and conclude with a discussion in Section 6.
Preliminaries
Notation Throughout the paper, we denote scalars, vectors and matrices, respectively, with lowercase italics, lowercase bold and uppercase bold Roman letters, e.g. u , u, and
U. We use [ m ] to denote the set { 1 , 2 , . . .
, m } .
We use ∥ · ∥ or ∥ · ∥ 2 exchangeably for ℓ 2 -norm.
Given a matrix
U =
[ u 1 , . . . , u m ] ∈ R d
× m , we define ∥ U ∥ 2 , ∞ =
max s ∈ [ m ] ∥
u s ∥ .
We use B 2 ( u , R ) to denote the ℓ 2 ball centered at u of radius R .
For any U ∈ R d × m , we use B 2 , ∞ ( U , R ) =
{ U ′ ∈
R
d × m |
∥ U ′ − U ∥ 2 , ∞ ≤ R } to denote the ℓ 2 , ∞ ball centered at U of radius R .
For any function f : R d → R , ∇ f denotes the gradient vector.
We define the standard normal distribution as N (0 , 1) , and the standard multivariate normal distribution as N (0 , I d ) .
We use S d − 1 to denote the unit sphere in d dimensions.
We use the standard O-notation ( O and Ω ).
2.1 Problem Setup
Let X ⊆ R d
and Y denote the input space and the label space, respectively.
In this paper, we focus on the binary classification setting where Y = {− 1 , +1 } .
We assume that the data ( x , y ) is drawn from an unknown joint distribution D on X × Y .
For a function f w : X → Y parameterized by w in some parameter space W , the generalization error captures the probability that f w makes a mistake on a sample drawn from D : generalization error :
= P ( x ,y ) ∼D ( yf w ( x ) ≤ 0) .
For a fixed x ∈ X , we consider norm-bounded adversarial perturbations given by B 2 ( x , R ) , for some fixed perturbation budget R .
The robust loss on x is defined as
[Equation]
The robust error then captures the probability that there exists an adversarial perturbation on samples drawn from D such that f w makes a mistake on the perturbed point
[Equation]
In this work, we focus on two-layer neural networks with ReLU activation, parameterized by a pair of weight matrices ( a , W ) , computing the following function:
[Equation]
Here, m corresponds to the number of hidden nodes, i.e., the width of the network; σ ( z ) =
max { 0 , z } is the ReLU activation function, and W =
[ w 1 , . .
w m ] ∈ R d × m and a =
[ a 1 , . .
. , a m ] ∈ R m denote the top and the bottom layer weight matrices, respectively.
In this work, we study the robustness of neural networks which are close to their initialization.
In particular, we are interested in the lazy regime , defined as follows.
Definition 2.1 (Lazy Regime) .
Initialize the top and bottom layer weights as a s ∼ unif( {− 1 , +1 } ) and w s, 0 ∼ N ( 0 , I d ) , ∀ s ∈ [ m ] .
Let W 0 =
[ w s, 1 , . . .
, w s,m ] ∈ R d × m .
The lazy regime is the set of all networks parametrized by ( a , W ) , such that W ∈ B 2 , ∞ ( W 0 , C 0 / √ m ) , for some constant C 0 .
2.2 Related Work
Adversarial Examples.
The field of adversarially robust machine learning has received significant attention starting with the seminal work of Szegedy et al.
Most existing works focus on proposing methods to generate adversarial examples, for example, the fast gradient sign method (FGSM)
[Goodfellow et al., 2015], the projected gradient descent (PGD) method [Madry et al., 2018], the Carlini & Wagner attack [Carlini and Wagner, 2017b], to name a few.
Most related to our work are those on proving the existence of adversarial examples on random neural networks.
This line of work initiated with the work of Shamir et al.
[2019], where authors propose an algorithm to generate bounded ℓ 0 -norm adversarial perturbation with guarantees for arbitrary deep networks.
Shortly afterwards, Daniely and Shacham [2020] showed that multi-step gradient descent can find adversarial examples for random ReLU networks with small width (i.e. m =
o ( d ) ).
More recently, work of Bubeck et al.
[2021] shows that a single gradient ascent update finds adversarial examples for sufficiently wide randomly initialized ReLU networks.
This was later extended to multi-layer networks with the work of Bartlett et al.
A very recent work of Vardi et al.
[2022] shows that gradient flow induces a bias towards non-robust networks.
Our focus and techniques here are different – we show that independent of the choice of the training algorithm, all neural networks trained in the lazy regime remain vulnerable to adversarial attacks.
Over-parametrized Neural Networks.
Our investigation into the existence of adversarial examples in the lazy regime is motivated by recent advances in deep learning theory.
In particular, a series of recent works establish generalization error bounds for first-order optimization methods under the assumption that the weights of the network do not move much from their initialization [Jacot et al., 2018, Allen-Zhu et al., 2019, Arora et al., 2019, Chen et al., 2021, Ji and Telgarsky, 2020,
Cao and Gu, 2020].
In particular, Chizat et al.
[2019] recognize that “lazy training” phenomenon is due to a choice of scaling that makes the model behave as its linearization around the initialization.
Interestingly, linearity has been hypothesized as key reason for existence of adversarial examples in neural networks [Goodfellow et al., 2015].
Furthermore, this was used to prove existence of adversarial attacks for random neural networks [Bubeck et al., 2021, Bartlett et al., 2021] and also provide the basis of our analysis in this work.
Main Results
In this section, we present our main result.
We assume that the data is normalized so that ∥ x ∥ 2 = 1 .
We show that under the lazy regime assumption, a single gradient step on f suffices to find an adversarial example to flip the prediction, given the network width is sufficiently wide but not excessively wide.
Theorem 3.1 (Main Result) .
Let w s, 0 ∼ N (0 , I d ) , a ∼ unif ( { 1 , +1 } ) and γ ∈ (0 , 1) .
For any given x ∈ S d − 1 , with probability at least 1 − γ , the following holds simultaneously for all W ∈
B 2 , ∞
[Equation] [Equation]
where δ = η ∇ x f ( x ; a , W ) , provided that the following conditions are satisfied:
[Equation]
Width requirement:
[Equation]
where C 0 , C 2 , C 3 , C 4 are constants independent of width m and dimension d .
Several remarks are in order.
We note that the setting in Theorem 3.1 is motivated by the popular gradient ascent based attack model which has been shown to be fairly effective in practice.
While in practice, we perform multiple steps of projected gradient ascent to generate an adversarial perturbation, here we show that a single step of gradient suffices with a small step size of O (1 /d ) .
This attests to the claim that over-parameterized neural networks trained in the lazy regime remain quite vulnerable to adversarial examples.
We confirm our findings empirically in Section 5.
Our analysis suggests that an attack size of O (1 / √ d ) suffices, i.e., for any x a perturbation δ of size ∥ δ ∥
=
O (1 / √ d ) can flip the sign of f ( x ) .
This is a relatively small budget for adversarial perturbations, especially for for high-dimensional data.
Indeed, in practice, we find that a noise budget of this size allows for adversarial attacks on neural networks trained on real datasets such as
MNIST and CIFAR-10 [Madry et al., 2018].
Interestingly, we find in our experiments, that O (1 / √ d ) is the “right” perturbation budget in the sense that there is a sharp drop in robust accuracy.
More details are provided in Section 5.
Our investigation focuses on the settings wherein the weights of the trained neural network stay close to the initialization.
Further, given the initialization scheme we consider, the O (1 / √ m ) deviation of the incoming weight vector at any neuron of the trained network from its initialization precisely captures the neural tangent kernel (NTK) regime studied in several prior works [Ji and Telgarsky, 2020, Du et al., 2018].
This O (1 / √ m ) deviation bound is also the largest radius that we can allow in our analysis; see Section 4 for further details.
In fact, our results would also hold for adversarial training given the same initialization and the bound on the deviation of weights is met.
Curiously, our empirical results exhibit a phase transition around the perturbation budget of O (1 / √ m ) – we see in
Section 5 that robust accuracy increases as we allow the network weights to deviate more.
There is an interesting interplay between the network width and the input dimensionality.
Our results require that the network is sufficiently wide (i.e., polynomially large in d ), but not excessively wide (is sub-exponential in d ).
The upper bound is large enough to allow for the over-parameterization required in the NTK setting, as long as the input dimensionality is sufficiently large.
To get an intuition about how the result follows, we first paraphrase the arguments from Bubeck et al.
[2021], Bartlett et al.
[2021] for random neural networks with weights W 0 .
We consider a linear model f ( w ; x ) = 1 √ d (cid:80) ds =1 w s, 0 x s .
Since the initial weights are independent (sampled i.i.d.
from
N (0 , 1) ), using standard concentration arguments, we have that for large d , with high probability | f
( w 0 ; x ) | ≤ O (1) .
In contrast, ∇ x f ( w 0 ; x ) = w, so ∥∇ x f ( w 0 ; x ) ∥ =
∥ w ∥ , which is roughly Θ( √ d ) .
Hence, for large enough d , a step of gradient ascent suffices to change the sign of f .
The above reasoning can be extended to weights that are close to random weights.
In particular, for arbitrary weights W close to the initialization, consider the linear model: f ( w ; x ) = 1 √ d (cid:80) ds =1 w s x s .
Since ∥ w s − w s, 0 ∥
=
O (1) , thus even in the worst case, the function still behaves as a constant translation of the initial prediction model f ( w 0 ; x ) , and thus is constant.
Similarly, ∥∇ f ( w , x ) ∥
= Ω( √ d ) .
The work of Bubeck et al.
[2021] extends the above observation to ReLU activation exploiting the fact that the function, even though non-linear, is still locally linear.
Note that, in a different context, this local linearity property is central to the Neural Tangent Kernel regime.
In particular, the key underlying idea there is based on the linear approximation to the prediction function.
Furthermore, with appropriate initialization, a linear approximation is not only good enough model close to the initialization, but also enables tractable analysis.
We collapse this “good” linearization property under the lazy training assumption since that suffices for our purposes.
Beyond Lazy regime.
Our result can be easily extended to a broader class of neural networks.
In particular, consider the cone C
= { W : ∃ r > 0 , r W ∈ B 2 , ∞
( W 0 , C 0 √ m ) } ; see Figure 1.
Note that for binary classification, scaling the network weights with a positive factor does not change the prediction.
As a result, Theorem 3.1 still holds for network weights which belong to the cone C but may not be in the lazy regime.
Robust Test Error.
As a corollary to Theorem 3.1, we have that none of the networks in the lazy regime are robust, i.e., for all networks in the lazy regime, the robust test error is bounded from below by a constant with high probability.
Corollary 3.2.
Let w s, 0 ∼ N (0 , I d ) , a ∼ unif ( {− 1 , +1 } ) , R =
˜
O
(1 / √ d ) and γ ∈ (0 , 1) .
In the parameter settings of Theorem 3.1, for any data distribution D supported on the sphere S d − 1 , with probability at least 1 − 10 γ (over the draw of W 0 and a), for all W s.t.
W ∈ B 2 , ∞ ( W 0 , C 0 / √ m ) , the robust error L R ( W ; D ) ≥ 0 .
We note that our result above is not a contradiction to the prior result of Gao et al.
[2019], which claims that adversarial training finds robust classifiers in overparameterized models.
Firstly, our result shows that there is no robust classifier in the lazy regime (Definition 2.1).
In contrast, Gao et al.
[2019] simply assume the existence of robust classifier in an NTK setting; see [Gao et al., 2019,
Assumption 5.2].
Indeed, ignoring some of the (potentially unimportant) differences between our setting and that of Gao et al.
[2019], (e.g., smooth vs. non-smooth activations and tied vs. non-tied weights in the last layer), then our result may suggest that the assumption in Gao et al.
[2019] is incorrect.
Secondly, the main result of Gao et al.
[2019] is to show that the excess robust training error of the output of adversarial training is small.
However, this only implies a small robust training error if the best-in-class robust training error is also small, which, again, as we discussed above, does not hold.
Finally, despite the validity of the key assumption in Gao et al.
[2019], the authors only provide a bound on robust training error.
In contrast, our result shows that robustness may not hold in the sense of robust test error.
Proof Sketch
In this section, we provide an outline of the proof of Theorem 3.1.
Recall that our goal is to show that for any given x ∈ S d − 1 , the vector δ = η ∇ x f ( x ; a , W ) is such that ∥ δ ∥ ≪ ∥ x ∥ , and sign ( f ( x ; a , W ))
̸ = sign ( f ( x + δ ; a , W )) .
Assume without loss of generality that f ( x ; a , W ) > 0 , then sign ( η ) = − 1 .
From the fundamental theorem of calculus, we have f ( x + δ ) =
f ( x )+ (cid:82) 1 0 f
′ ( x + tδ ) dt.
Using simple algebraic manipulations (see full proof in Appendix B), we have the following, f
[Equation]
x +
[Equation]
x ; a , W )
[Equation]
a , W
[Equation]
≤
[Equation]
The goal now is to control each of the three terms on the right hand side.
Note that the suggested bounds on each of the terms will suffice to flip the label, thereby establishing the main result.
Given any x and W ∈ B 2 , ∞ (cid:16) W 0 , C 0 √ m (cid:17) , we need to control:
The function value of neural network f ( x ; a , W ) 2.
The size of the gradient of neural network ∥∇ f
( x ; a , W ) ∥ with respect to the input x 3.
The size of the difference of gradients ∥∇ f
( x ; a , W ) − ∇ f ( x + δ ; a , W ) ∥ for ∥ δ ∥
≤ O (1 / √ d ) 4.1 Bounding the function value
Lemma 4.1.
(Informal) For any x, with probability at least 1 − γ , | f ( x ; a , W ) | ≤ 2 (cid:112) log(2 /γ )
+ Cholds for all W ∈ B 2 , ∞ (cid:18)
W 0 , C 0 √ m (cid:19) , provided that m ≥ Ω(log(2 /γ )) .
Proof Sketch:
From triangle inequality,
[Equation]
For the first term, we have from Bubeck et al.
[2021] that | f
( x ; a , W 0 ) | =
O (1) .
For the second term | f ( x ; a , W ) − f ( x ; a , W 0 ) |
≤ O (1) owing to the Lipschitzness of ReLU.
Note that O (1 / √ m ) is the largest deviation on ∥ w s − w s, 0 ∥ that we can allow to get the constant bound here.
We next move to the second and third terms.
We note that while bounding the first term simply follows from the Lipschitzness of the network, as we show in the following sections, the second and third terms require a finer analysis, due to the non-smoothness of ReLU.
4.2 Bounding the gradient
Lemma 4.2.
(Informal) For any x, with probability at least 1 − γ , ∥∇ f ( x ; a , W ) ∥ ≥ 14 √ d holds for all W ∈ B 2 , ∞
( W 0 , C 0 √ m ) , provided that m ≥ Ω(log(1 /γ )) and d ≥ Ω( log( m/γ ) m ) .
Proof Sketch: Let P := I d − xx ⊤ be the projection matrix onto the orthogonal complement of x.
Using triangle inequality,
[Equation]
The first term ∥ P ∇ f ( x ; a , W 0 )
∥ ≥ Ω( √ d ) , which follows from Bubeck et al.
[2021] with minor changes arising from different scaling of the initialization.
We further decompose the second term,
[Equation]
The first term is O (1) simply from the fact that ∥ w s − w 0 ∥ ≤ C 0 √ m .
To bound the second term, the key arguments are as follows: by definition of P, the term Pw s, 0 is independent of σ ′ ( w ⊤ s x ) − σ ′ ( w ⊤ s, 0 x ) , therefore, as in Bubeck et al.
[2021], we can bound both of these terms independently.
Note that a s Pw s, 0 ∼ N (0 , I d − 1 ) .
Therefore, the second term is equal in distribution to the following random variable:
[Equation]
where z ∼ N (0 , I d − 1 ) .
The term | σ ′ ( w ⊤ s x ) − σ ′ ( w ⊤ s, 0 x ) | = 1 only when the s -th neuron has different signs for incoming weights w s, 0 and w s .
So, at a high-level, we want to bound the number of neurons changing sign between weights w 0 and w. To do this, we draw insights from results in the lazy training regime that bound the number of such neurons in terms of deviations in weights.
In particular, the following lemma shows that with high probability, for all W ∈ B 2 , ∞ ( W 0 , C 0 √ m ) , at most O ( √ m ) neurons can have a sign different from their initialization.
Lemma 4.3.
Let S v :=
(cid:8) s (cid:12)(cid:12) ∃ W , W ∈ B 2 , ∞ (cid:16) W 0 , C 0 √ m (cid:17) , 1 [ ⟨ w s , x ⟩ > 0]
̸ = 1 [ ⟨ w s, 0 , x ⟩ > 0] (cid:9) .
Then, with probability at least 1 − γ , we have that | S v | ≤ C 0 √ m + (cid:113) m log(1 /γ ).
The key idea is that we can absorb the union bound in the definition of the set above without compromising any increase in the size of the set.
Putting all of these pieces together yields the bound claimed in Lemma 4.2.
4.3 Bounding the difference of gradients
Lemma 4.4.
For any x, with probability at least 1 − γ , for all W ∈ B 2 , ∞ (cid:16) W 0 , C 0 √ m (cid:17) , we have that sup r ∈ S
d − 1 ,δ ∈ R d ∥∇ f
( x ; a , W ) − ∇ f ( x + δ ; a , W ) ∥ ≤ o ( √ d ) , provided that ∥ δ ∥ ≤ O (1 / √ d ) , and d 2 . 4 ≤ m ≤ O (exp( d 0 . 24 )) .
Proof Sketch: It follows from the definition of norm that ∥∇ f ( x ; a , W ) − ∇
[Equation]
x
[Equation]
; a , W )
[Equation]
= sup
[Equation]
The strategy, then, is to bound the right-hand side for fixed r and δ , and then use an ε -net argument to bound the supremum.
To that end, define the following quantities:
[Equation]
Then, the norm of the difference of the gradients is upper bounded by
[Equation]
The first term 1 √ m (cid:80) ms =1 X s,
= O
(( R (cid:112) log( d )) 1 / 4 ) which follows from Bubeck et al.
[2021] with minor changes due to the different scale of the initialization.
However, the second term requires a more careful analysis.
In particular, the difference X s − X s, 0 includes deviations of both variables x and W. Therefore, we need to expand the difference in terms of each variable and carefully bound the resulting terms.
Similar to the ideas presented in the previous section, here we need to bound the size of the set of neurons that can change sign under perturbations on both x and W. It turns out that the size of this set is in the same order as the one in Lemma 4.3, as long as the perturbation in x is O (1) .
Experiments
The primary goal of this section is to provide empirical support for our theoretical findings.
In particular, we want to verify how tight our theoretical bounds are in practice, in terms of the size of the perturbation needed and the interplay of robustness with training in the lazy regime.
For instance, are the trained models indeed vulnerable to small perturbations of size
O (1 / √ d ) as suggested in
Theorem 3.1?
Is the assumption of training in the lazy regime merely for analytical convenience or do we see a markedly different behavior if we allow the iterates to deviate more than O (1 / √ m ) from initialization?
Datasets & Model specification.
We
utilize
the
MNIST dataset for our empirical study.
MNIST is a dataset of 28 × 28 greyscale handwritten digits [LeCun et al., 1998].
We extract examples corresponding to images of the digits ‘0’ and ‘1’, resulting in 12665 training examples andtest examples.
To study the behaviour of different quantities as a function of the input feature dimensionality, we downsample the images by different factors to simulate data with d ranging between 25 and 784 (the original dimension).
We use two-layer ReLU networks with a fixed top layer to match our theory and use the initialization described in Section 2.
5.1 Lazy regime for standard SGD
Setting.
For each d ∈ { 5 2 , 7 2 , 10 2 , 14 2 , 20 2 , 25 2 , 28 2 } , we set the width of the network m ∈ { 1 e 3 , 1 e 4 , 1 e 5 } .
We train the network using standard SGD on the logistic loss using a learning rate of 0 . 1 .
We denote the weight matrix at the t th iterate of SGD as W t and the incoming weight vector into the s th hidden node at iteration t as w s,t .
The training terminates if the iterates exit the lazy training regime, i.e., if sup 1 ≤ s ≤ m ∥ w s,t − w s, 0 ∥ > C 0 √ m .
We experiment with C 0 ∈ { 10 , 20 , 30 , 40 } .
We note that regardless of the above stopping criterion, each of the trained models achieves a test error of less than 1% .
In other words, the models that our training algorithm returns are guaranteed to satisfy the lazy regime assumption and generalize well.
Given a trained neural network with parameters ( W , a ) , for each test example x, we find the smallest step size η such that the perturbation δ = η ∇ W f
( x ) flips the sign of the prediction, i.e., f W ( x ) f W ( x + δ ) < 0 .
For each experimental setting (i.e, for each value of d , C 0 , and m ), we report results averaged over 5 independent random runs.
Results.
The key quantities in our analysis are the step size η , the norm of the gradient ∥∇ f ( x ; a , W ) ∥ , and the size of the perturbation vector which is given as the product of the step size and the norm of the gradient.
In Figure 2, we plot these quantities as a function of input dimension d , for a fixed C 0 , for various values of width m ∈ { 1 e 3 , 1 e 4 , 1 e 5 } (corresponding to the curves in red, orange, and blue, respectively).
The green curves represent the corresponding theoretical bound.
As predicted by Theorem 3.1, the gradient norm behaves roughly as Ω( √ d ) .
Furthermore, the minimum step-size required to flip the label, and the minimum size of the perturbation vector (normalized by the input norm) behave as O (1 /d ) and O (1 / √ d ) , respectively.
In Figure 3, we explore the behaviour of the key quantities as we allow for larger deviations in the lazy regime, by allowing for larger values of C 0 , for a fixed network width of m = 10 5 .
We see that as C 0 increases, both the size of the gradient and the minimum step size required to flip the label increase, thereby increasing the size of the perturbation vector needed to change the sign of the prediction.
This suggests that allowing the trained networks to deviate more from the initialization potentially leads to models that are more robust (against the attack model based on a single step of gradient ascent).
Remarkably, the minimum perturbation size, closely follows the O (1 / √ d ) behaviour predicted by the theorem.
This begets further investigation into the required perturbation size, which is the focus of the next subsection.
5.2 Lazy regime for adversarial training
We begin by noting that our theoretical results are not limited to any particular learning algorithm.
In fact, the results hold simultaneously for all networks in the lazy regime.
In this section, we further verify our theoretical results for networks that are adversarially trained in the lazy regime.
Setting.
We downsample images as in the previous section to simulate data with d ∈ { 5 2 , 6 2 , 7 2 , . .
For this set of experiments, we also normalize the data, so that ∥ x ∥ = 1 , to closely match the setting of our theorem.
We use projected gradient descent (PGD)
[Madry et al., 2018] for adversarial training.
To ensure that the updates remain in the lazy regime, after each gradient step, we project the weights onto B 2 , ∞ ( W 0 , V ) ; see Algo.
1 in the appendix for details.
We experiment with values of V and the size of ∥
δ ∥ in the intervals 0 .
05 ≤ V ≤ 1 and 0 .
05 ≤ ∥ δ ∥ ≤ 1 , respectively.
These ranges are wide enough to include the O (1 / √ d ) and O (1 / √ m ) theoretical bounds that we aim to verify here.
We note that the projection step is not commonly used in adversarial training; however, in our experiments, it does not hurt the test accuracy of the trained models, perhaps due to the simplicity of the classification task.
At test time, we generate the adversarial examples using the PGD attack instead of a single gradient attack as in Section 5.1.
We report the robust test accuracy averaged over 5 independent random runs of the experiment.
Results.
For the experiments reported in Figure 4, left panel, we fix the network width to m =and set V = C 0 √ m = 10 √ 1000 = 0 .
316 , to check how the dimension d and the perturbation budget affect the robust accuracy.
We observe a sharp drop in robust accuracy about the 5 / √ d threshold for the perturbation budget ∥ δ ∥ as predicted by Theorem 3.1.
We can see that for perturbation sizes slightly larger than 5 / √ d , the robust accuracy drops to 0 , while the trained networks are robust against smaller perturbations.
For the experiments reported in the right panel of Figure 4, we fix the input dimensionality to d = 28 2 , and the perturbation size to ∥ δ ∥ = 0 .
2 , to check how the network width m and the maximal deviation in weights, given by V , affect the robust test accuracy.
We observe a phase transition in the robust test accuracy at value of V around O (1 / √ m ) , as required by Theorem 3.1.
In particular, for any fixed width m , even a slight deviation from the 25 / √ m radius allows for training networks that are completely robust against the PGD attacks.
With a smaller radius, however, the robust test accuracy quickly drops to 0.
This experiment suggests that adversarial robustness may require going beyond the lazy regime.
Why MNIST?
We remark that our empirical demonstration on a simple dataset makes a stronger case for our theoretical result.
This is because the perturbation size of O (1 / √ d ) prescribed by our theoretical result is worst-case , regardless of the dataset.
So, in an easy setting like binary-MNIST, we would expect our bound to be pessimistic and a smaller perturbation to be sufficient.
On the contrary, we see that even MNIST exhibits a perturbation-dimension relationship that closely follows our theory.
This can be regarded as an empirical evidence of the tightness of our bounds.
Conclusion
In this paper, we study the robustness of two-layer neural networks trained in the lazy regime against inference-time attacks.
In particular, we show that the class of networks for which the weights are close to the initialization after training, are still vulnerable to adversarial attacks based on a single step of gradient ascent with a perturbation of size O ( ∥ x ∥√ d ) .
Our works suggests several research directions for future work.
First, the focus here is on two-layer networks, so a natural question is to ask whether the same holds for multi-layer networks.
Second, while we show that a single step of gradient ascent-based attack is sufficient for our purposes, we expect to get stronger results if we consider stronger attacks, for instance, with gradient ascent based attack that is run to convergence.
Third, our experiments highlight an intriguing relationship between the width, the input dimension, and the robust accuracy.
So, a natural avenue is to understand these dependencies more carefully.
These include, figuring out the dependence on the input dimensionality of the minimal perturbation required to generate an adversarial example.
Similarly, how does the robust error behave as a function of the maximal distance of the weight vectors from the initialization.
Acknowledgments and Disclosure of Funding
This research was supported, in part, by the DARPA GARD award HR00112020004, NSF CAREER award IIS-1943251, and the Spring 2022 workshop on “Learning and Games” at the Simons Institute for the Theory of Computing.
[-+ P (cid:18) | w ⊤ s, 
-]
[-Let X s, 
-]
[-(cid:18)(cid:0) 4 R (cid:112) log( d ) (cid:1) 1 / 4 + (cid:114) 
-]
[-d log(10 m 4 / 7 d 4 / 7 )+log(4 /γ ) m (cid:19) + C 
-]
[-md +
-]