Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterization
Abstract
The Neural Tangent Kernel (NTK) has emerged as a powerful tool to provide memorization, optimization and generalization guarantees in deep neural networks.
A line of work has studied the NTK spectrum for two-layer and deep networks with at least a layer with Ω( N ) neurons, N being the number of training samples.
Furthermore, there is increasing evidence suggesting that deep networks with sublinear layer widths are powerful memorizers and optimizers, as long as the number of parameters exceeds the number of samples.
Thus, a natural open question is whether the NTK is well conditioned in such a challenging sub-linear setup.
In this paper, we answer this question in the affirmative.
Our key technical contribution is a lower bound on the smallest NTK eigenvalue for deep networks with the minimum possible over-parameterization : up to logarithmic factors, the number of parameters is Ω( N ) and, hence, the number of neurons is as little as Ω( √ N ) .
To showcase the applicability of our NTK bounds, we provide two results concerning memorization capacity and optimization guarantees for gradient descent training.
Introduction
Training a neural network is a non-convex problem that exhibits disconnected local minima [8, 57, 70].
Yet, in practice gradient descent (GD) and its variants routinely find solutions with zero training loss [72].
A framework to understand this phenomenon comes from the Neural Tangent Kernel (NTK).
This quantity was introduced in [33], where it was proved that, during GD training, the network follows the kernel gradient of the functional cost with respect to the NTK.
Furthermore, as the layer widths go large, the NTK converges to a deterministic limit which stays constant during training.
Hence in this infinite-width limit it suffices that the smallest eigenvalue of the NTK is bounded away from 0 for gradient descent to reach zero loss.
Going to finite widths, a recipe to prove GD convergence can be summarized as follows: show that (i) the NTK is well conditioned at initialization, and (ii) the NTK has not changed significantly by the time GD has reached zero loss (see e.g.
A number of papers have exploited this recipe for networks with progressively smaller over-parameterization: two-layer networks [22, 50, 63, 69, 62], deep networks with polynomially wide layers [5, 21, 73, 74], and deep networks with a single wide layer [47, 43].
Besides optimization, showing that the NTK is well conditioned directly implies a result on memorization capacity [42, 48], and the smallest eigenvalue of the NTK has also been related to generalization [6].
In [48], it is shown that, given N training samples, a single layer with Ω( N ) neurons suffices for the
NTK to be well conditioned in networks of arbitrary depth.
However, there is increasing evidence that, in the challenging setup in which the layer widths are sub-linear in N , neural networks still memorize the training data [14, 71, 66], reach zero loss under GD training in the two-layer setting [61, 50, 42], and in the deep case, GD explores a nicely behaved region of the loss landscape [44].
This is in agreement with simple back-of-the-envelope calculations on CIFAR-10 and ImageNet:
CIFAR-10 has N = 50000 images and roughly 10 6 parameters suffice to fit random labels [72]; furthermore, in order to fit random labels to a subset of 1 . 2 · 10 6 ImageNet data points, 2 .
· 10parameters are enough [72].
These numbers suggest that having a number of parameters of the same order as the dataset size is much closer to practice than having a number of neurons of that order.
We also note that, by counting degrees of freedom or bounding the VC dimension [10], Ω( N ) parameters (and, therefore, Ω( √ N ) neurons) are in general necessary to fit N data points.
This naturally brings forward the following open question:
Is the NTK well conditioned
for
deep
networks with the minimum possible over-parameterization (i
e., containing Ω( N ) parameters corresponding to
[Equation]
Main contributions.
In this paper, we settle this open question for a large class of deep networks.
We consider (i) a smooth activation function, (ii) i.i.d.
data satisfying Lipschitz concentration (e.g., data with a Gaussian distribution, uniform on the sphere/hypercube, or obtained via a Generative
Adversarial Network), (iii) a standard initialization of the weights (e.g., He’s or LeCun’s initialization), and (iv) a loose pyramidal topology in which the layer widths can increase by at most a multiplicative constant as the network gets deep.
Then, in Theorem 3.1 we show that the NTK is well conditioned under the minimum possible over-parameterization requirement: the number of parameters between the last two layers has to be ˜Ω( N ) or, equivalently, the number of neurons has to be ˜Ω( √ N ) , where ˜ Ω includes extra logarithmic factors.
We achieve this goal by giving a lower bound on the smallest eigenvalue of the NTK.
This lower bound is tight when all the layer widths are of the same order.
Our NTK bounds open the way towards understanding the behavior of deep networks with minimum over-parameterization.
In particular, an immediate consequence of the fact that the NTK is well conditioned is a result on the memorization capacity (Corollary 4.1).
Furthermore, by suitably choosing the initialization, we provide convergence guarantees for gradient descent training (Theorem 4.2).
Finally, we highlight that, in order to obtain our bounds on the smallest NTK eigenvalue, we give a number of tight estimates on the ℓ 2 norms of feature vectors and of their centered counterparts, which may be of independent interest.
Proof ideas.
To prove Theorem 3.1, we restrict to the kernel K L − 2 =
J L − 2 J ⊤ L − 2 , where J L − 2 is the Jacobian of the output w.r.t.
the parameters between the last two hidden layers.
This suffices as K L − 2 is a lower bound on the NTK in the positive semi-definite (PSD) sense.
We note that the i -th row ( i ∈ { 1 , . . . , N } ) of J L − 2 is given by the Kronecker product ⊗ between the feature vector at layer L − 2 and the backpropagation term from the same layer.
One key technical hurdle is to center J L − 2 , so that its rows have the form u ⊗ v − E [ u ⊗ v ] , where E [ u ] =
E [ v ] = 0 , and all expectations are taken with respect to the (random) training data.
To do so, we perform three steps of centering: (i) we center the feature vectors (corresponding to u ), (ii) we center the backpropagation terms (corresponding to v ), and (iii) we center again the whole row (corresponding to u ⊗ v ).
These centering steps are approximate in the sense that the centered matrix is not necessarily a lower bound (in the PSD sense) on the original one.
However, we are able to control the operator norm of the difference, and show that it scales slower than the smallest eigenvalue of the centered kernel.
At this point, we leverage the structure of the rows of the centered Jacobian to bound their sub-exponential norm via a version of the Hanson-Wright inequality for (weakly) correlated random vectors [1].
Finally, after providing also a tight estimate on the ℓ 2 norms of such rows, we can exploit a result from [2] to lower bound the smallest singular value of a matrix whose rows are independent random vectors with well controlled sub-exponential and ℓ 2 norms.
Existing work bounds the smallest NTK eigenvalue for networks with two layers [61, 42], or deep networks with a layer containing Ω( N ) neurons [48].
In particular, [61] also exploits the results [1, 2].
However, the centering of the Jacobian is achieved via a combination of whitening and dropping rows, which appears to be difficult to generalize to deep networks.
In contrast, the 3-step centering we described above applies to networks of arbitrary depth L .
A different approach is put forward in [42], and it is based on a decomposition of the kernel via spherical harmonics.
This technique allows to obtain the exact limit of the smallest NTK eigenvalue, it has been used to analyze random feature models [26, 41, 25, 40] and to obtain generalization bounds for two-layer networks (see again [42]).
However, understanding how to carry out such a decomposition in the multi-layer setup is an open problem.
Finally, [48] considers the deep case, and it relates the smallest eigenvalue of the NTK to the smallest singular value of a feature matrix.
As feature matrices are full rank only when the number of neurons is Ω( N ) , this approach is inherently limited to networks with a linear-width layer.
The rest of the paper is organized as follows: Section 2 discusses the problem setup and our model assumptions; Section 3 presents our main result on the smallest NTK eigenvalue and gives a roadmap of the argument; Section 4 provides two applications of our NTK bounds: memorization capacity and gradient descent training; Section 5 discusses additional related work, and Section 6 provides some concluding remarks.
The details of the proofs are deferred to the appendices.
Preliminaries
Neural network setup.
We consider an L -layer neural network with feature maps f l : R d → R n l defined for every x ∈ R d as
[Equation]
Here, W l ∈ R n l − 1 × n l is the weight matrix at layer l , ϕ is the activation function and, given an integer n , we use the shorthand [ n ] = { 1 , . . . , n } .
We assume that the network has a single output, i.e. n L = 1 and W L ∈ R n L − 1 , and for consistency we have n 0 = d .
Let g l : R d → R n l be the pre-activation feature map so that f l ( x ) =
ϕ ( g l ( x )) for l ∈ [ L ] .
We define g 0 ( x ) = f 0 ( x ) = x .
Let X =
[ x 1 , . . .
, x N ] ⊤ ∈
R N ×
d be the data matrix containing N samples in R d , θ = [vec( W 1 ) , . . .
, vec( W L )] be the vector of the parameters of the network, and
F L ( θ ) =
[ f L ( x 1 ) , . . .
, f L ( x N )]
⊤ be the network output.
We denote by J the Jacobian of F L with respect to all the parameters of the network:
[Equation]
Our key object of interest is the empirical Neural Tangent Kernel (NTK)
Gram matrix , denoted by
K ∈ R N × N and defined as:
[Equation]
In [33], it is shown that, as n l → ∞ for all l ∈
[ L − 1] , K converges to a deterministic limit , which stays constant during gradient descent training.
The focus of this paper is on the finite-width behavior of the empirical NTK (3).
Quantitative bounds for the NTK convergence rate can be obtained from [7, 15].
However, these bounds lead to a significant over-parameterization requirement, see the discussion at the end of Section 3 in [48].
Here, our main result consists in showing that the NTK is well conditioned for a class of neural networks with the minimum possible over-parameterization , i.e., Ω( N ) parameter or, equivalently, Ω( √ N ) neurons.
Weight and data distribution.
We consider the following initialization of the weight matrices: ( W l ) i,j ∼ i .
i .
d .
N (0 , β 2 l /n l − 1 ) for l ∈
[ L − 1] , i ∈ [ n l − 1 ] , j ∈ [ n l ] , where β l is a numerical constant independent of the layer widths.
This covers the popular cases of He’s and LeCun’s initialization [27, 30, 36].
For the last layer, we assume that ( W L ) i ∼ i .
i .
d .
N (0 , β 2 L ) for i ∈ [ n L − 1 ] .
Throughout the paper, we let ( x 1 , . . .
, x N ) be N i.i.d.
samples from the data distribution P X , which satisfies the conditions below.
Assumption 2.1 (Data scaling) .
The data distribution P X satisfies the following properties:
(cid:82)
∥ x ∥ 2 dP X ( x ) =
Θ( √ d ) .
(cid:82)
∥ x ∥ 22 dP X ( x ) =
Θ( d ) .
(cid:82) (cid:13)(cid:13) x − (cid:82) x ′ dP X ( x ′ )
(cid:13)(cid:13) 2 2 dP X ( x ) =
Ω ( d ) .
Assumption 2.2 (Lipschitz concentration) .
The data distribution P X satisfies the Lipschitz concentration property .
Namely, there exists an absolute constant c > 0 such that, for every Lipschitz continuous function φ : R d → R , we have E | φ ( X ) | < + ∞ , and for all t > 0 ,
[Equation]
Assumption 2.1 is simply a scaling of the training data points and their centered counterparts.
Assumption 2.2 covers a number of important cases, e.g., standard Gaussian distribution [65], uniform distributions on the sphere and on the unit (binary or continuous) hypercube [65], data produced via a Generative Adversarial Network (GAN) 3 [59], and more generally any distribution satisfying the log-Sobolev inequality with a dimension-independent constant.
We also remark that
Assumption 2.2 is rather common in the related literature [48], or it is even replaced by a stronger requirement (e.g., Gaussian distribution or uniform on the sphere)
Assumption 2.3 (Activation function) .
The activation function ϕ satisfies the following properties:
ϕ is a non-linear (and therefore also non-constant) M -Lipschitz function; 2.
its derivative ϕ ′ is a M ′ -Lipschitz function;
These requirements are satisfied by common activations, e.g. smoothed ReLU, sigmoid, or tanh .
Assumption 2.4 (Network topology) .
The network satisfies a loose pyramidal topology condition, i.e. n l = O ( n l − 1 ) , for all l ∈
[ L − 1] .
A strict pyramidal topology (namely, non-increasing layer widths) has been considered in prior work concerning the loss landscape [45, 46] and gradient descent training [47].
Our Assumption 2. requires a loose pyramidal topology in the sense that, as we go deep, the layer widths are allowed to increase by a constant multiplicative factor.
We also note that the widths of neural networks used in practice are often large in the first layers and then start decreasing [28, 60].
Assumption 2.5 (Over-parameterization) .
We have that
[Equation]
Furthermore, there exists γ > 0 such that
[Equation]
Condition (4) requires the number of parameters between the last two hidden layers to be linear in the number of samples, up to logarithmic factors.
This represents our key over-parameterization condition .
When all the widths have the same scaling, (4) reduces to N = ˜
o
( d 2 ) .
This is satisfied by several “standard” datasets, such as MNIST ( N = 60000 , d = 784 ), CIFAR-10 ( N = 5 · 10 4 , d = 3 · 32 2 ), and ImageNet ( N = 1 . 4 · 10 7 , d = 2 · 10 5 ).
We also note that, if N ≫ d 2 , the NTK is low-rank and λ min ( K )
= 0 .
Furthermore, Corollary 4.1 and Theorem 4.2 cannot generally hold when N ≫ d 2 , as there are more data points to fit than parameters to help with the fitting.
A milder requirement is possible for classification tasks, see the discussion at the end of Section 5.
The second condition (5) is rather mild, as γ can be taken to be arbitrarily small, and it avoids an exponential bottleneck in the last hidden layer.
Notation.
The feature matrix at layer l is F l =
[ f l ( x 1 ) , . . .
, f l ( x N )]
⊤ ∈ R N × n l , and Σ l ( x ) = diag([ ϕ ′ ( g l,j ( x ))]
n l j =1 ) for l ∈ [ L − 1] , where g l,j ( x ) is the pre-activation neuron.
Given two matrices
F, B ∈ R m × n , we denote by F ◦ B their Hadamard product, and by F ∗ B =
[( F 1: ⊗ B 1: ) , . .
, ( F m : ⊗
B m : )]
T ∈ R m × n 2 their row-wise Kronecker product (also known as Khatri-Rao product).
Given a random vector v , let ∥ v ∥ ψ 2 and ∥ v ∥ ψ 1 denote its sub-Gaussian and sub-exponential norm, respectively (see also Appendix A for a detailed definition).
Given a matrix A , let ∥
A ∥ op be its operator norm, ∥
A ∥ F its Frobenius norm, λ min ( A ) its smallest eigenvalue, and σ min ( A ) its smallest singular value.
We denote by ∥
φ ∥ Lip the Lipschitz constant of the function φ .
All the complexity notations Ω( · ) ,
O ( · ) , o ( · ) and Θ( · ) are understood for sufficiently large N, d, n 1 , n 2 , . . .
, n L − 1 .
Tildes on such symbols are meant to neglect logarithmic factors.
NTK Bounds with Minimum Over-parameterization
Our main technical contribution on the smallest eigenvalue of the empirical NTK (3) is stated below.
Theorem 3.1 (Smallest NTK eigenvalue under minimum over-parameterization) .
Consider an Llayer neural network (1) , where the activation function satisfies Assumption 2.3 and the layer widths satisfy Assumptions 2.4 and 2.5.
Let { x i } Ni =1 ∼ i .
i .
d .
P X , where P X satisfies the Assumptions 2.1-2.2, and let K be the empirical NTK Gram matrix (3) .
Assume that the weights of the network are initialized as ( W l ) i,j ∼ i .
i .
d .
N (0 , β 2 l /n l − 1 ) for l ∈
[ L − 1] , i ∈ [ n l − 1 ] , j ∈ [ n l ] and ( W L ) i ∼ i .
i .
d .
N (0 , β 2 L ) for i ∈ [ n L − 1 ] .
Then, we have λ
[Equation]
(6) with probability at least 1 − C Ne − c log 2 n L − 1 − Ce − c log 2 N over ( x i )
Ni
=1 and ( W k )
Lk =1 , where c and C are numerical constants.
Moreover, we have that
[Equation]
(7) with probability at least 1 − Ce − cn L − 1 , over ( x i ) Ni =1 and ( W k )
Lk =1 .
[Figure]
This result implies that the NTK is well conditioned for a class of networks in which the number of parameters n L − 2 n L − 1 between the last two hidden layers is linear in the number of data samples N , up to logarithmic factors.
This means that the total number of neurons of the network can be as little as ˜Ω( √ N ) , which meets the minimum possible amount of over-parameterization .
The lower bound (6) and the upper bound (7) match when all the widths (up to layer L − 2 ) have the same scaling, i.e., d = Θ( n L − 2 ) .
Throughout the paper, we do not explicitly track the dependence of our bounds on L , in the sense that the numerical constants c, C may depend on L .
In Figure 1, we consider a 3-layer neural network with d = n 1 = n 2 , and we plot λ min ( K ) as a function of d 2 , for three different values of N .
The inputs are sampled from a standard Gaussian distribution, the activation function is the sigmoid σ ( x ) =
(1 + e − x ) − 1 , and we set β l = 1 for all l ∈
[ L ] .
We repeat the experiment 10 times, and report average and confidence interval at 1 standard deviation.
The linear scaling of λ min ( K ) in d 2 is in agreement with the result of Theorem 3.1.
The code used to obtain the results of Figure 1 (and Figure 2 as well) is available at https://github.com/simone-bombari/smallest-eigenvalue-NTK/ .
3.1 Roadmap of the proof of Theorem 3.
After an application of the chain rule and some standard manipulations, we have
[Equation]
with K k = F
[Equation]
(8) where B k ∈ R N × n k is a matrix whose i -th row is given by
[Equation]
From the decomposition (8), one readily obtains that K ⪰ K L − 2 , since K k is PSD for all k ∈ { 0 , . . .
, L − 1 } .
Hence, it suffices to prove a lower bound on λ min ( K L − 2 ) .
We denote by J L − 2 the
Jacobian obtained by computing the gradient only over the parameters between layer L − 2 and layer
L − 1 .
Then, K L − 2 =
J L − 2 J ⊤ L − 2 and, by using (8) and (9), the i -th row of J L − 2 can be expressed as
[Equation]
where D L is a diagonal matrix with the entries of W L on its diagonal.
Note that, if we fix the weight matrices ( W l ) L l =1 , the rows (( J L − 2 ) i : ) N i =1 are i.i.d., as the training data ( x i )
Ni =1 are i.i.d.
too.
The proof consists of two main parts.
First, we construct the centered Jacobian ˜ J L − 2 , which is obtained from J L − 2 by iteratively removing expectations with respect to X to its parts, and we show that its smallest singular value is close to the smallest singular value of J L − 2 .
The details of this part are contained in Appendix D. Second, we bound the smallest eigenvalue of the kernel obtained from the centered Jacobian ˜ J L − 2 by providing an accurate estimate of the ℓ 2 and sub-exponential norms of its rows.
The details of this part are contained in Appendix E.
In order to carry out this program, we exploit a number of concentration results on the ℓ 2 norms of feature and backpropagation vectors, and on the ℓ 2 norms of their centered counterparts.
These results are contained in Appendix C, and they could be of independent interest.
In particular, we provide tight high-probability estimates on (i) ∥
f l ( x ) ∥ 2 , (ii) E x [ ∥ f l ( x ) ∥ 22 ] , (iii) E x [ ∥ f l ( x ) ∥ 2 ] , (iv) E x [ ∥ f l ( x ) − E x
[ f l ( x )] ∥ 22 ] , E x [ ∥ f l ( x ) − E x
[ f l ( x )] ∥ 2 ] , and ∥ f l ( x ) − E
x [ f l ( x )] ∥ 2 , and (v) ∥
D L ϕ ′ ( g L − 1 ( x ))
− E x
[ D L ϕ ′ ( g L − 1 ( x ))]
∥ 2 .
Some preliminary calculations are also contained in Appendix B.
Part 1: Centering.
We consider the centered Jacobian ˜ J L − 2 , whose i -th row is defined as ( ˜
[Equation]
11) where ˜ f L − 2 ( x i ) =
˜ ϕ
( g L − 2 ( x i )) , with ˜ ϕ
( g L − 2 ( x i ))
=
ϕ ( g L − 2 ( x i ))
− E x i
[ ϕ ( g L − 2 ( x i ))] and ˜ ϕ
′ ( g L − 1 ( x i ))
=
ϕ ′
( g L − 1 ( x i ))
− E x i [ ϕ ′ ( g L − 1 ( x i ))] being the centered versions of the activation function and of its derivative, respectively.
Our strategy is to relate λ min ( J L − 2 J ⊤ L − 2 ) to λ min ( ˜ J L −
˜ J ⊤ L − 2 ) via the following two steps.
Step (a): Centering F L − 2 and B L − 1 .
We show that λ min ( J L − 2 J ⊤ L − 2 ) ≥ λ min ( ˜ J FB ˜
J ⊤ FB )
− o ( n L − 2 n L − 1 ) ,
where ˜ J FB ˜
J ⊤ FB =
˜ F L − 2 ˜ F ⊤ L − 2 ◦ ˜ B L − 1 ˜ B ⊤ L − 1 and ˜ F L − 2 =
F L − 2 − E X
[ F L − 2 ] and ˜ B L − 1 =
B L − 1 − E X
[ B L − 1 ] (Lemma D.1 in Appendix D.1).
Our strategy differs from existing work (e.g., [61, 48]), where either only the backpropagation term is centered (cf.
Proposition 7.1 of [61]) or only the features are centered (cf.
Lemma 5.4 of [48]).
We remark that, in order to handle deep networks with minimum overparameterization 4 , the centering of F L − 2 and B L − 1 is crucially performed at the same time .
In fact, by doing so, certain terms containing an eigenvalue which is negative and large in modulus suitably cancel.
As a result, the difference between the original kernel J L − 2 J ⊤ L − 2 and the centered one ˜ J FB ˜
J ⊤ FB can be bounded by a matrix whose operator norm is o ( n L − 2 n L − 1 ) .
Step (b): Centering everything.
We show that λ min ( ˜ J FB ˜
J ⊤ FB ) ≥ λ min ( ˜ J L − 2 ˜
J ⊤ L − 2 ) − o ( n L − 2 n L − 1 ) , where the rows of ˜ J L − 2 are given by (11) (Lemma D.2 in Appendix D.2).
The idea is to decompose the difference
˜ J FB
˜
J ⊤ FB − ˜ J L − 2 ˜ J ⊤ L − 2 into a rank-1 matrix plus a PSD matrix.
Then, in order to bound the operator norm of the rank-1 term, we leverage the general version of the Hanson-Wright inequality given by Theorem 2.3 in [1] (see the tail bound on quadratic forms that we provide in
Lemma B.5).
This strategy avoids whitening and dropping rows (as done in [61]) and, hence, appears to be better suited to the deep case.
The main result of this part is stated below, and it is proved by combining Lemmas D.1 and D.2.
Theorem 3.2 (Jacobian centering) .
Consider the setting of Theorem 3.1, and let the rows of J L −and
˜
J L − 2 be given by (10) and (11) , respectively.
Then, we have
[Equation]
with probability at least 1 − C exp( − c log 2 n L − 1 ) − C exp( − c log 2 N ) over ( x i ) Ni =1 and ( W k )
Lk =1 , where c, C are numerical constants.
Part 2: Bounding the centered Jacobian.
If we fix the weight matrices ( W l ) Ll =1 , the rows of ˜
J L − 2 are i.i.d.
vectors of the form u ⊗ v − E [ u ⊗ v ] , where both u and v are centered.
By exploiting this structure, in Appendix E.1 we bound the ℓ 2 and sub-exponential norms of these rows.
The ℓ 2 norm is bounded in Lemma E.2, and this result relies on the tight estimates on the ℓ 2 norms of centered features and centered backpropagation terms given by Lemma C.4 and C.5, respectively.
The sub-exponential norm is bounded in Lemma E.3, and we use again the tail bound of Lemma
B.5, which exploits the version of the Hanson-Wright inequality in [1].
At this point, the problem is reduced to bounding the smallest eigenvalue of a matrix such that its rows are i.i.d.
and we have a good control on their ℓ 2 and sub-exponential norms.
This goal can be achieved via the following result, whose proof follows from Theorem 3.2 in [2] and it is presented for completeness in Appendix
E.2.
Proposition 3.3.
Let ˜ J be a matrix with rows ˜ J i : ∈ R n L − 2 n L − 1 , for i ∈
[ N ] .
Assume that { ˜ J i : } i ∈ [ N ] are independent sub-exponential random vectors with ψ = max i ∥
˜ J i : ∥ ψ 1 .
Let η min =
min i ∥
˜ J i : ∥and η max = max i ∥
˜ J i : ∥ 2 .
Set ξ = ψK + K ′ , and ∆ :=
Cξ 2 N 1 / 4 ( n L − 1 n L − 2 ) 3 / 4 .
Then, we have that
[Equation]
holds with probability at least
[Equation]
where c and C are numerical constants.
The main result of this part is stated below, and it is proved in Appendix E.3.
Theorem 3.4 (Bounding the centered Jacobian) .
Consider the setting of Theorem 3.1, and let the rows of ˜ J L − 2 be given by (11) .
Then, we have
[Equation]
with probability at least 1 − Ce [Equation]
N
[Equation] 1 over ( x i ) Ni =1 and ( W k )
Lk =1 , where c and
C are numerical constants.
Recall that K ⪰ K L − 2 , hence λ min ( K ) ≥ λ min ( K L − 2 ) =
λ min (cid:0) J L − 2 J ⊤ L − 2 (cid:1) .
Thus, the lower bound (6) follows by combining the results from Theorem 3.2 and Theorem 3.4.
Finally, the upper bound (7) is rather straightforward.
First, we write
[Equation]
Then, by combining the bound on ∥ ( F l ) 1: ∥ 22 of Lemma C.1 with a direct calculation for ∥ ( B l +1 ) 1: ∥ 22 , the desired result (7) follows.
The details are contained in Appendix F.
Two Applications: Memorization and Optimization
Memorization capacity.
The fact that the NTK Gram matrix (3) is well conditioned readily implies a result on memorization capacity.
This was already observed in [42] for two-layer networks and in [48] for deep networks with a layer containing Ω( N ) neurons.
Here, by using Theorem 3.1, we show that Ω( N ) parameters between the last pair of hidden layers are enough for the network to fit
N real-valued labels up to an arbitrarily small error.
The result is stated below and, for completeness, we give the proof in Appendix G.
Corollary 4.1 (Memorization) .
Consider an L -layer neural network (1) , where the activation function satisfies Assumption 2.3 and the layer widths satisfy Assumptions 2.4 and 2.5.
Let { x i } Ni =1 ∼ i .
i .
d .
P X , where P X satisfies the Assumptions 2.1-2.2.
Then, it holds that for every Y ∈ R N , and for every ε > 0 , there exists a set of parameters θ such that
[Equation]
with probability at least 1 − C Ne − c log [Equation] − Ce − c log 2 N over ( x i )
Ni
=1 , where c and C are numerical constants.
Gradient descent training.
Theorem 3.1 has implications on the convergence of gradient descent algorithms.
In particular, by choosing carefully the initialization, we show that gradient descent trained on the N samples { x i } Ni =1 with labels Y =
( Y 1 , . . .
, Y N ) converges to zero loss.
This is – at the best of our knowledge – the first result of this kind for deep networks with minimum over-parameterization .
To define our initialization, let us assume that the ( L − 1)
-th layer has an even number of neurons.
With a slight abuse of notation, we indicate its width as 2 n L − 1 .
Thus, W L − 1 can be written in the form (cid:104) W (1) L − 1 , W (2)
L − 1 (cid:105) , where W (1) L − 1 , W (2) L − 1 ∈ R n L − 2 × n L − 1 .
Similarly, W L ∈ R 2 n L − 1 is the concatenation of W (1) L , W (2) L ∈ R n L − 1 .
Then, we define the initialization θ
= [vec( W 1 , 0 ) , . .
, vec( W L, 0 )] as follows:
[Equation]
) i,
[Equation]
i .
i
[Equation] [Equation]
i,j ∼ i .
i .
d .
N (0 , β 2 L − 1 /n
L − 2 ) , i ∈
[ n L − 2 ] , j ∈ [ n L − 1 ] , and W (2) L − 1 ,
= W (1) L − 1 , 0 , ( W (1) L, 0 )
i ∼ i .
i .
d .
N (0 , β 2 L γ ) , i ∈ [ n L − 1 ] , and W (2) L, 0 = − W (
[Equation]
) L,
[Equation] [Equation]
As usual, ( β l ) Ll =1 are numerical constants independent of the layer widths.
In contrast, the quantity γ will be set to a suitably large value depending on the number of training samples and on the layer widths.
In words, the initialization is standard for W 1 , 0 , . . .
, W L − 2 , 0 and W (1) L − 1 , 0 ; the variance of
W (1) L, 0 is boosted up by a factor γ ; and we duplicate the neurons of layer L −
so that W (2) L − 1 , 0 =
W (1) L − 1 , 0 and W (2) L, 0 = − W (1) L, 0 .
At this point, we are ready to state our optimization result.
Theorem 4.2 (Optimization via gradient descent) .
Consider an L -layer neural network (1) , where the activation function satisfies Assumption 2.3 and the layer widths satisfy Assumptions 2.4 and 2.5.
Consider training data { x i } N i =1 ∼ i .
i .
d .
P X , where P X satisfies the Assumptions 2.1-2.2, with labels
Y ∈ R N such that ∥ Y ∥
2 = Θ( √ N ) .
We consider solving the least-squares optimization problem
[Equation]
by running gradient descent updates of the form θ t +
=
θ t − η ∇L ( θ t ) , where the initialization θ 0 is defined in (18) with γ = d 3 N 2 and η ≤ C ( γNdn L − 1 ) − 1 .
Then, for all t ≥ 1 ,
[Equation]
20) with probability at least 1 − C Ne − c log 2 n L − 1 − Ce − c log 2 N over ( x i ) Ni =1 and the initialization θ 0 , where c and C are numerical constants.
[Figure]
In words, (20) guarantees that gradient descent linearly converges to a point with zero loss.
We also note that (our upper bound on) the convergence rate can be expressed as a simple function of the learning rate η , the number of training samples N , and the layer widths { n i } L − 1 i =0 .
In Figure 2, we give an illustrative example that 4-layer networks achieve 0 loss when the number of parameters is at least linear in the number of training samples, i.e., under minimum over-parameterization.
To ease the experimental setup, we use a ReLU activation, with Adam optimizer.
We initialize the network as in the setting of Theorem 3.1, picking β l = 1 for all l ∈
[ L ] .
The inputs, as well as the targets, are sampled from a standard Gaussian distribution.
The plot is averaged over 10 independent trials.
As predicted by Theorem 4.2, the loss experiences a phase transition and it goes from 1 to 0 when the layer widths are of order √ N .
We now provide a sketch of the argument of Theorem 4.2, deferring the detailed proof to Appendix H. As a consequence of Theorem 3.1, the NTK is well conditioned at the initialization θ 0 , which implies that gradient descent makes progress at the beginning.
The idea is that the NTK remains sufficiently well conditioned also during training and, hence, the loss vanishes with the linear rate in (20), since the trained weights remain confined in a ball of sufficiently small radius centered at θ 0 .
More specifically, we exploit the optimization framework developed in [49].
There, a convergence result is proved if the following condition holds: there exists α, β such that
[Equation]
and
[Equation] [Equation]
for all θ, θ 1 , θ 2 ∈ D =
B
( θ 0 , R ) , where D denotes a ball centered at θ 0 of radius R := 4 ∥
F L ( θ 0 ) −
Y ∥ 2 /α (cf.
Proposition H.1).
In order to control the radius R , we take advantage of the form (18) of the initialization θ 0 .
In particular, the spectrum of the Jacobian J (and, hence, α ) scales linearly in √ γ , and the network output at initialization F L ( θ 0 ) is equal to 0 (cf.
Lemma H.2).
Thus, as ∥ Y ∥ 2 = Θ( √ N ) , we have that R = Θ( √ N/α ) , which can be controlled by choosing suitably γ .
At this point, the argument consists in providing a number of estimates on feature vectors, backpropagation terms, and finally the Jacobian J ( θ ) for all θ ∈ D (cf.
Lemmas H.3-H.9).
This allows us to find α, β such that (21) holds and conclude.
Related Work and Discussion
Random matrices in deep learning.
The limiting spectra of several random matrices related to neural networks have been the subject of a recent line of work.
In particular, the Conjugate Kernel (CK) – namely, the Gram matrix of the features from the last hidden layer – has been studied for models with two layers [54, 37, 39, 51], with a bias term [3, 56], and with multiple layers [13].
The
Hessian matrix has been considered in [52], and the closely related Fisher information matrix in [55].
Using tools from free probability, the input-output Jacobian (as opposed to the parameter-output
Jacobian considered in this work) of deep networks is studied in [53] and the NTK of a two-layer model in [4].
In [23], the spectrum of NTK and CK for deep networks is characterized via an iterated
Marchenko-Pastur map.
The generalization error has also been studied via the spectrum of suitable random matrices, see [29, 41, 38, 42] and Section 6 of the review [11].
Most of the existing results focus on the linear-width asymptotic regime, where the widths of the various layers are linearly proportional.
An exception is [68], which focuses on the ultra-wide two-layer case ( n 1 ≫ N ).
Smallest eigenvalue of empirical kernels.
In the line of work mentioned above, the limiting spectrum of the random matrix is characterized in terms of weak convergence, which does not lead to a direct implication on the behavior of its smallest eigenvalue.
In fact, lower bounding the smallest eigenvalue often requires understanding the speed of convergence to the limit spectrum.
Quantitative bounds for random Fourier features are obtained in [9].
In the two-layer setting, the smallest NTK eigenvalue is lower bounded in [61, 42, 68], and concentration bounds can also be obtained from [4, 63, 31].
In particular, [42] establishes a tight bound for two-layer networks with roughly as many parameters as training samples (hence, under minimum over-parameterization), and for a wide class of activations.
However, this result still requires that n 1 = Ω( d ) .
For deep networks, a convergence rate on the NTK can be obtained from [7] and potentially from [15], however these results require all the layers to be wide.
In [48], it is proved that a single layer of linear width suffices for the NTK to be well conditioned.
Here, Theorem 3.1 provides the first result for deep networks under minimum over-parameterization, therefore allowing for sub-linear layer widths.
Memorization capacity and gradient descent training.
For binary classification, the memorization capacity of neural networks was first studied by Cover [18], who solved the single-neuron case.
Later, Baum [12] proved that, for two-layer networks, the memorization capacity is lower bounded by the number of parameters, and upper bounds of the same order were given in [35, 58].
Recently, tight results for two-layer networks in the regression setting have been provided [14, 42].
In particular, [14] generalizes the construction of [12], while the memorization result of [42] comes from a bound on the smallest NTK eigenvalue.
This same route is also followed to analyze deep networks in [48] and in this work.
Results for multiple layers have been provided in [10, 71, 66, 24].
As concerns efficient algorithms achieving memorization, [19, 20] study classification with two-layer networks.
A popular line of research has exploited the NTK view to give optimization guarantees on gradient descent.
Existing work has focused on the two-layer case [61, 22, 50, 63, 69, 62], and optimal results in terms of over-parameterization have been obtained [42].
For deep networks, [21] assumes a lower bound on the smallest NTK eigenvalue, [5, 73, 74] require all the layer widths to be (rather large) polynomials in the number of samples, and [47, 43] reduce the over-parameterization to a single layer of linear width.
In particular, in these last two papers, the analysis of the NTK is reduced to that of a certain feature matrix (the first one in [47] and the last one in [43]).
Hence, these approaches do not seem suitable to tackle networks with sub-linear layer widths 5 .
Finally, we remark that milder over-paramerization requirements are sufficient for classification problems.
More specifically, it has been proved that polylogarithmic width suffices for two-layer [34] and deep [16]
ReLU networks to converge and generalize.
In fact, achieving memorization for regression is harder than for classification, where it suffices to satisfy inequality constraints [42].
Concluding Remarks
In this paper we show that the NTK is well conditioned for deep networks with sub-linear layer widths.
As consequences of our NTK bounds, we also prove results on memorization and optimization for such a class of networks.
Our approach requires smooth activations and layer widths which do not significantly grow in size with the depth.
We believe such assumptions to be purely technical and we leave as an open question their removal.
Our novel NTK analysis could potentially be applied beyond standard gradient descent, e g., for optimization with momentum [67] or federated learning [32]
Acknowledgements
The authors were partially supported by the 2019 Lopez-Loreta prize and they would like to thank
Quynh Nguyen, Mahdi Soltanolkotabi and Adel Javanmard for helpful discussions.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Assumptions 2.1, 2.2, 2.3, 2.4 and 2.5 in Section 2; see also l. 384-385 in Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Given the theoretical nature of the work, we do not see direct paths to applications with negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] All the assumptions are clearly stated either in the description of the model in Section 2 or in the statements of the main results (Theorem 3.1, Corollary 4.1, and Theorem 4.2).
(b) Did you include complete proofs of all theoretical results?
[Yes] A sketch of the proof of Theorem 3.1 is provided in Section 3.1.
The complete proof is contained in Appendices B-F: Appendix B contains preliminary calculations; Appendix C contains the high-probability estimates on the ℓ 2 norms of feature and backpropagation vectors; Appendix D contains the complete proofs for Part 1:
Centering ; Appendix E contains the complete proofs for Part 2:
Bounding the centered Jacobian ; and Appendix F contains the proof of the upper bound (7).
The proof of Corollary 4.1 is contained in Appendix G. Finally, the proof of Theorem 4.2 is sketched in Section 4 right after the statement, and the full proof is contained in Appendix H. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The folder containing the code and the instructions is in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] In Figure 1, we repeat the experiment 10 times, and we report the average together with the confidence interval at 1 standard deviation.
In Figure 2, we repeat the experiment 10 times and we report the average of the final loss in the heat-map.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A]
We did not perform large scale experiments that require to specify this type of details.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] We are not using existing assets or curating/releasing new assets.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
We did not use crowdsourcing or conducted research with human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Additional Notations
Given a sub-exponential random variable X , let ∥ X ∥ ψ 1 = inf { t > 0 : E [exp( | X | /t )] ≤ 2 } .
Similarly, for a sub-Gaussian random variable, let ∥ X ∥ ψ 2 = inf { t > 0 : E [exp( X 2 /t 2 )] ≤ 2 } .
We use the analogous definitions for vectors.
In particular, let X ∈ R n be a random vector, then ∥ X
[Equation]
[Equation]
We indicate with C and c absolute, strictly positive, numerical constants, that do not depend on the layer widths of the network { n l } L − 1 l =0 or the number of training samples N .
Their value may change from line to line.
B Some Useful Estimates
Lemma B.1.
Under Assumption 2.5, we have that
[Equation] [Equation]
[Equation] [Equation]
Proof.
We start by proving (22).
If n L − 1 = O (cid:0) N 2 (cid:1) , then
N
[Equation]
O (cid:0) N ·
[Equation]
8 N (cid:1) =
o ( n L − 1 n L − 2 ) , (26) where the last passage follows from (4).
Conversely, if n L − 1 = Ω( N 2 ) , then
N · log 8 n L − 1 =
O (cid:0)
[Equation]
which concludes the proof of (22).
To obtain (23), we can exploit the second requirement of Assumption 2.5, which implies that log N =
O (log n L − 1 ) .
This readily implies (23).
Notice that (24) naturally follows since n L −
= O ( n L − 2 ) by Assumption 2.4.
Finally, to obtain (25), we write
[Equation]
where in the first passage we use that log N =
O (log n L − 1 ) (from the second requirement of
Assumption 2.5) and the last passage follows from (22).
Lemma B.2 (Lipschitz constant of function of the features) .
For all l ∈ [ L − 1] , and for every
Lipschitz function φ , we have
[Equation]
with probability at least
[Equation]
over ( W k ) lk =1 .
We recall that φ is applied component-wise to g l ( x ) , and φ ( g l ( x )) :
R d → R n l is intended as a function of x .
Proof.
Note that φ ( g l ) is a composition of Lipschitz functions.
Thus,
[Equation]
where the last step is justified by Assumption 2.3.
Recall that, by the assumption on the initialization of the weights, ( W k ) i,j ∼ i .
i .
d .
N (0 , β 2 k /n k − 1 ) , for some constant β k which does not depend on the layer widths.
Then, by Theorem 4.4.5 of [65], we have that, for any k ∈ [ l ] ,
[Equation]
with probability at least 1 − 2 exp ( − n k ) , C being a numerical constant.
By Assumption 2.4 on the topology of the network, we can rewrite this result as
[Equation]
33) with probability at least 1 − 2 exp ( − n L − 1 ) .
To conclude, using a union bound over the layers up to layer l , we have that
[Equation] [Equation]
with probability at least 1 − 2 l exp ( − n L − 1 ) over ( W k ) lk =1 .
Lemma B.3.
We have that
[Equation]
with probability at least 1 − 2 exp( − c log 2 n L − 1 ) over W L , where c is a numerical constant.
Proof.
Recall that D L = diag ( W L ) contains on the diagonal n L − 1 independent Gaussian random variables ( D L ) ii ∼ N (0 , β 2 L ) .
Thus, for any i ∈ [ n L − 1 ] ,
[Equation] [Equation]
which gives
[Equation]
where the second step is a union bound on the entries of D L .
This gives the desired result.
Lemma B.4.
We have that
[Equation]
with probability at least 1 − 2 exp( − c log 2 n L − 1 ) − C exp ( − n L − 1 ) over ( W k )
Lk =1 .
We recall that ϕ
′ is applied component-wise to g l ( x ) ,
D L ϕ ′
( g L − 1 ( x )) :
R d → R n L − 1 is intended as a function of x , and c is a numerical constant.
Proof.
We know by composition of Lipschitz functions that
[Equation]
By Assumption 2.3, ϕ ′ is Lipschitz.
Hence, by combining Lemma B.2 (where we use φ = ϕ ′ ) and
Lemma B.3, the result follows.
Lemma B.5 (Exponential tails of quadratic forms) .
Let x ∼ P X .
Let u : R d → R d u and v : R d → R d v be mean-0 Lipschitz functions with respect to x , i.e., E x [ u ( x )]
= 0 , E x [ v ( x )] = 0 , ∥ u ∥ Lip = cand
∥ v ∥ Lip = c 2 .
Let U be a d u × d v matrix, and
[Equation]
Then,
[Equation]
where K = (cid:112) c
21 + c 22 , and C is a numerical constant.
Proof.
Consider the function z ( x ) :
R d → R d u + d v obtained by concatenating the vectors u and v , i.e.,
[Equation]
One can readily verify that ∥ z ∥ Lip ≤ (cid:112) c 21 + c 22 :
= K .
Let us set
[Equation]
Then, we have that
[Equation]
Since x satisfies Assumption 2.2 and z ( x ) is Lipschitz, in order to obtain tail bounds on Γ , we can apply the version of the Hanson-Wright inequality given by Theorem 2.3 in [1]:
[Equation]
where C 1 is a numerical constant, and in the last step we use that ∥ M ∥ op = ∥
U ∥ op and ∥ M ∥ 2 F = ∥
U ∥ 2 F / 2 ≤ ∥ U ∥ 2 F .
Thus, by Lemma 5.5 of [61], we conclude that
[Equation]
for some numerical constant C 2 , which gives the desired result.
Lemma B.6.
Let u ∈ R d u and v ∈ R
d v be two mean-0 sub-Gaussian vectors such that ∥ u ∥ ψ 2 = cand ∥ v ∥ ψ 2 =
c 2 .
Set A uv = E (cid:2) uv ⊤ (cid:3) .
Then,
[Equation] [Equation]
where C is a numerical constant.
Proof.
Consider the vector
[Equation]
Then, z is sub-Gaussian and, by triangle inequality on the vectors [ u, 0] and [0 , v ] , we have that ∥ z ∥ ψ 2 ≤ c 1 + c 2 .
Since u and v are mean-0, then z is also mean-0 and its covariance matrix can be written as A z := E (cid:2) zz ⊤ (cid:3) .
Furthermore, we can show that
[Equation] [Equation]
In fact, let w be the unitary eigenvector associated to the maximum eigenvalue of A z .
Then,
[Equation] [Equation]
Furthermore, we have that
[Equation]
where C is a numerical constant, and the last inequality comes from Eq.
(2.15) of [65].
By combining (50) and (51) with ∥ z ∥ ψ 2 ≤ c 1 + c 2 , (49) readily follows.
Finally, we have that
[Equation] [Equation]
where A u := E (cid:2) uu ⊤ (cid:3) and A v := E (cid:2) vv ⊤ (cid:3) .
As A u and A v are PSD, we have that
[Equation]
Hence, the desired result follows from (49) and (53).
Lemma B.7.
Let A be an N × n matrix whose rows A i are i.i.d.
mean-0 sub-Gaussian random vectors in R n .
Let K =
∥
A i ∥ ψ 2 the sub-Gaussian norm of each row.
Then, we have
[Equation]
with probability at least 1 − 2 exp( − cn ) , for some numerical constant c .
Proof.
Without loss of generality, we can assume K = 1 to simplify the proof.
Let Σ be the second moment matrix of each of the rows of A .
Then, Σ = E (cid:2)
A i
A ⊤ i (cid:3) , since the rows are mean-0.
Note that, as the rows of A are i.i.d., Σ is independent of i .
Furthermore, Lemma B.6 implies that the covariance matrix E (cid:2)
A i
A ⊤ i (cid:3) has operator norm bounded by a constant, since the sub-Gaussian norm of the rows is 1.
Then, by using Remark 5.40 in [64], we have that
[Equation]
where δ = C
[Equation] [Equation]
with probability at least 1 − 2 exp( − ct 2 ) , where c and C are numerical constants.
Setting t = √ n and using a triangular inequality gives that, with probability at least 1 − 2 exp( − ct 2 ) ,
[Equation]
which implies the desired result (after re-scaling by K ).
Lemma B.8.
Let
˜ F l = F l − E x
[ F l ] ∈ R N × n l be the centered features matrix at layer l .
Then, we have
[Equation]
with probability at least 1 − C exp ( − cn L − 1 ) over ( W k ) lk
=1 and ( x i )
Ni
=1 ∼ i .
i .
d .
P X .
Proof.
From Lemma B.2, we have that
[Equation]
with probability at least
[Equation]
over ( W k ) lk =1 .
We condition on this event in the rest of the proof.
Since ( x i ) Ni =1 ∼ i .
i .
d .
P X and P X satisfies Assumption 2.2, all the rows of ˜ F l are mean-0 subGaussian vectors, with sub-Gaussian norm bounded by a numerical constant.
Here, we fix ( W k ) lk = s.t.
(58) holds, and the “mean-0” and the “sub-Gaussian norm” is intended w.r.t.
the probability space of ( x i ) Ni =1 .
An application of Lemma B.7 gives that
[Equation] [Equation]
with probability at least 1 − 2 exp ( − cn l ) over ( x i ) Ni =
1 ∼ i .
i .
d .
P X .
Taking into account the previous conditioning, we conclude that
[Equation]
61) with probability at least 1 − ( C ′ + 2) exp ( − cn L − 1 ) over ( W k ) lk
=1 and ( x i )
Ni
=1 ∼ i .
i .
d .
P X .
Lemma B.9.
Let ˜ B L − 1 =
B L − 1 − E x
[ B L − 1 ]
∈ R N × n L − 1 be the centered back-propagation matrix at layer L − 1 .
Then, we have
[Equation]
with probability at least 1 − C exp ( − cn L − 1 ) over ( W k )
Lk
= l and ( x i )
Ni
=1 ∼ i .
i .
d .
P X .
Proof.
From Lemma B.4, we have that
[Equation]
with probability at least
[Equation]
over ( W k )
Lk =1 .
We condition on this event in the rest of the proof.
Since ( x i ) Ni =1 ∼ i .
i .
d .
P X and P X satisfies Assumption 2.2, all the rows of ˜ B L − 1 are mean-0 subGaussian vectors, with sub-Gaussian norm O (log n L − 1 ) .
Here, we fix ( W k )
Lk =1 s.t. (63) holds, and the “mean-0” and the “sub-Gaussian norm” is intended w.r.t.
the probability space of ( x i ) Ni =1 .
An application of Lemma B.7 gives that
[Equation] [Equation]
with probability at least 1 − 2 exp ( − cn L − 1 ) over ( x i ) Ni =1 ∼ i .
i .
d .
P X .
Taking into account the previous conditioning, we conclude that
[Equation]
with probability at least 1 − ( C ′ + 2) exp ( − cn L − 1 ) over ( W k
[Equation]
Lk =1 and ( x i )
Ni
=1 ∼ i .
i .
d .
P X .
Lemma B.10.
Let ρ be a standard Gaussian random variable, then we have that
[Equation]
and
[Equation]
are continuous functions in c .
Furthermore, φ 1 ( c ) is Lipschitz in c , and
[Equation]
where C 1 and C 2 are numerical constants (independent of c 1 , c 2 ).
Proof.
Let p ( ρ ) = 1 √ 2 π e − ρ 2 / 2 .
Then
[Equation]
we have
[Equation]
where in the second line we use that ϕ is M -Lipschitz by Assumption
[Equation]
Similarly
[Equation]
we have
[Equation]
Lemma B.11.
Let ρ be a standard Gaussian distribution, and c ̸ = 0 be an absolute constant.
Then, we have
[Equation]
It also holds
[Equation]
Proof.
For the first statement, we exploit the fact that ϕ is Lipschitz:
[Equation]
The statement on ϕ ′ is easily derived following the same proof and using that ∥ ϕ ′ ∥
Lip ≤ M ′ .
Lemma B.12.
Let ρ be a standard Gaussian distribution, and c ̸ = 0 be an absolute constant.
Then, we have
[Equation]
and
[Equation]
Proof.
For the upper-bound of the first statement, we exploit the fact that ϕ is Lipschitz:
[Equation]
For the lower bound, since ϕ is non-zero and continuous, we have that there exist a strictly positive constant c ′ > 0 and an interval [ c 1 , c 2 ] with c 2 > c 1 such that ϕ 2 ( x ) ≥ c ′ for each x ∈ [ c 1 , c 2 ] .
Therefore, we have
[Equation]
The second statements is proved in the same way, as ϕ ′ is a non-zero Lipschitz function.
Lemma B.13.
Let φ : R d → R a Lipschitz function, and let x ∼ P X .
Then,
[Equation]
where c is a numerical constant.
Proof.
We have
[Equation]
where the inequality is a consequence of Assumption 2.2.
Lemma B.14.
Let x ∼ P X , and define ˜ c
l ( x ) =
β l ∥ f l ( x ) ∥ / √ n l .
Then, we have
[Equation]
81) and
[Equation]
82) where C is a numerical constant.
Proof.
We have that
[Equation]
where c and C are numerical constants
[Equation]
and the third line is justified by Assumption
[Equation]
Similarly, we have
[Equation]
where, again
[Equation]
c and C are numerical constants and the third line is justified by Assumption 2.
[Equation]
Lemma B.15.
Let ρ 1 and ρ 2 be two standard Gaussian random variables, possibly correlated.
Then, we have
[Equation] [Equation]
where C 1 , C 2 , C 3 , C 4 are numerical constants (which do not depend on x 1 , x 2 , y 1 , y 2 ).
Furthermore, the same result holds with ϕ ′ instead of ϕ .
Proof.
We have
[Equation]
where in third inequality we use that ϕ is M -Lipschitz
[Equation]
and in the last inequality we use that the quantities E [ | ρ 1 | ] , E [ | ρ 2 | ] and E [ | ρ 1 || ρ 2 | ] are all smaller than 1 (regardless of the correlation between ρ 1 and ρ 2 ).
Since we only used the fact that ϕ is M -Lipschitz, the same result holds with ϕ ′ in place of ϕ .
C Concentration of ℓ 2 Norms
In this appendix, we state and prove a number of high-probability estimates on the ℓ 2 norms of feature and backpropagation vectors.
More specifically, our results can be summarized as follows:
• Lemma C.1 gives tight bounds on ∥ f l ( x ) ∥ 2 , i.e. the ℓ 2 norm of the feature vector at layer l .
The statement holds with high probability over x and ( W k ) lk =1 .
• Lemmas C.2 and C.3 give tight bounds on E x (cid:104)
∥
f l ( x ) ∥ 22 (cid:105) and E x [ ∥ f l ( x ) ∥ 2 ] , respectively.
These quantities represent the expectation with respect to x of the (squared) ℓ 2 norm of the feature vector at layer l .
The statements hold with high probability over ( W k ) lk =1 .
• Lemma C.4 focuses on the centered feature vector f l ( x ) − E
x
[ f l ( x )] , and it gives tight bounds on (i) its expected (w.r.t.
x ) squared ℓ 2 norm E x (cid:104)
∥
f l ( x ) − E x
[ f l ( x )] ∥ 22 (cid:105) , (ii) its expected (w.r.t.
x ) ℓ 2 norm E x [ ∥ f l ( x ) − E x
[ f l ( x )] ∥ 2 ] , and (iii) its ℓ 2 norm ∥
f l ( x ) − E x
[ f l ( x )] ∥ 2 .
The first two statements hold with high probability over ( W k ) lk
=1 , and the probability in the last statement is also over x . • Lemma C.5 focuses on the centered backpropagation vector at layer L − 1 , and it gives tight bounds on its ℓ 2 norm ∥ D L ϕ ′ ( g L − 1 ( x ))
− E x
[ D L ϕ ′ ( g L − 1 ( x ))]
∥ 2 .
This statement holds with high probability over x and ( W k ) lk =1 .
Throughout this appendix, we always assume that P X satisfies Assumptions 2.1 and 2.
and that the layer widths satisfy Assumption 2.4.
Furthermore, we use that the activation ϕ and its derivative ϕ ′ are Lipschitz (see Assumption 2.3).
Lemma C.1 ( ℓ 2 norm of features) .
Let x ∼ P X .
Then, for every 0 ≤ l
[Equation]
with probability at least 1 − C exp( − cn L − 1 ) over x and ( W k ) lk =1 .
As usual, ϕ is applied componentwise on g l ( x ) , and c and C are numerical constants.
Proof.
We prove this by induction over l , and we start with the base case ( l = 0 ).
Recall that we have defined f 0 ( x ) :
= x .
As the ℓ 2 norm is a 1-Lipschitz function, by Assumption 2.2, we have that
[Equation]
Furthermore, Assumption 2.1 implies that E [ ∥ x ∥ 2 ] =
Θ( √ d ) , hence setting t =
E [ ∥ x ∥ 2 ] / 2 in (88) proves the desired result for the base case (recalling that n L − 1 =
O ( d ) by Assumption 2.4).
By inductive hypothesis, we have
[Equation]
with probability at least 1 − C exp( − cn L − 1 ) .
Define ˜ c := β l ∥ f l − 1 ( x ) ∥ 2 / √ n l − 1 .
From now on, we condition on a realization of x and ( W k ) l − 1 k = such that ˜ c
= Θ(1) .
By (89), this happens with probability at least 1 − C exp( − cn L − 1 ) .
To ease the notation, we use the shorthands f :=
f l − 1 ( x ) and W := W l .
Then, we can write
[Equation] [Equation]
Recall that ( W l ) i,j ∼ i .
i .
d .
N (0 , β 2 l /n l − 1 ) and that the Gaussian distribution is rotationally invariant.
Thus, the RHS of (90) has the same distribution as
[Equation]
where ( ρ i ) n l i =1 ∼ i .
i .
d .
N (0 , 1) and also independent of f , and we have defined the independent,
[Table]
[Equation]
Note that, in the definition of Z i , the randomness comes only from ρ i , since we are conditioning on ˜ c .
We have that
[Equation]
where the first term is bounded by a constant by Theorem 5.2.2 in [65], and the bound on the second term follows from Lemma B.11.
As a consequence, we have
[Equation]
where the inequality in the second line follows from Exercise 2.7.10 of [65], the equality in the third line follows from Lemma 2.7.6 of [65], and the inequality in the last line follows from (93).
Hence, the Z i -s are i.i.d.
sub-exponential random variables, with sub-exponential norm bounded by a numerical constant.
An application of Bernstein inequality (cf.
Corollary 2.8.3.
in [65]) gives that
[Equation]
where c, C 6 are numerical constants.
Furthermore, by Lemma B.12, we have
[Equation]
By setting t =
E
ρ 1 (cid:2) ϕ 2 (˜ cρ 1 ) (cid:3) / 2 into (95) and using (91) and (96)
, we conclude that
[Equation]
with probability at least 1 − C exp( − cn L − 1 ) − 2 exp( − cn l ) ≥ 1 − C 1 exp( − cn L − 1 ) , for some numerical constant c and C 1 , which concludes the proof.
Lemma C.2 (Expected squared ℓ 2 norm of features) .
Let x ∼ P X .
Then, for every 0 ≤ l ≤ L − 1 ,
[Equation]
with probability at least 1 − C exp( − cn L − 1 ) over ( W k ) lk =1 .
As usual, c and C are numerical constants.
Proof.
The argument is by induction over l .
The base case is a direct consequence of Assumption 2.1, since f 0 ( x ) = x .
By inductive hypothesis, we have
[Equation]
with probability at least 1 − C exp( − cn L − 1 ) .
Define ˜ c
( x ) :=
β l ∥ f l − 1 ( x ) ∥ 2 / √ n l − 1 .
From now on, we condition on a realization of ( W k ) l − 1 k =1 such that E x (cid:2) ˜ c
2 ( x ) (cid:3) = Θ(1) .
By (99), this happens with probability at least 1 − C exp( − cn L − 1 ) .
To ease the notation, we use the shorthands f :=
f l − 1 ( x ) , W := W l and w i =
W : i .
Then, we can write
[Equation]
where we use that the w i -s are equally distributed and we have defined the independent, meanrandom variables
[Equation]
Note that, in the definition of Z i , the randomness comes only from w i , since we are conditioning on ( W k ) l − 1 k =1 .
We have that
[Equation]
where the first line follows from Jensen’s inequality as ∥·∥ ψ 1 is convex, the inequality in the second line follows from Exercise 2.7.10 of [65], and the equality in the third line follows from Lemma 2.7.
of [65].
Recall that ( W l ) i,j ∼ i .
i .
d .
N (0 , β 2 l /n l − 1 ) and that the Gaussian distribution is rotationally invariant.
Thus, ϕ (cid:0) w ⊤ i f ( x ) (cid:1) has the same distribution as ϕ (˜ c ( x ) ρ i ) , where ( ρ i ) n l i =1 ∼ i .
i .
d .
N (0 , 1) and also independent of ˜ c ( x ) .
We now condition on a realization of x and ( W k ) l − 1 k =1 and provide an upper bound on the sub-Gaussian norm (cid:13)(cid:13) ϕ
(cid:0) w ⊤ i f ( x ) (cid:1)(cid:13)(cid:13)
ψ 2 , where the only randomness comes again from w i (and, hence, from ρ i ).
We have that
[Equation]
where the first term in the RHS in the second line is bounded by C 1 ˜ c ( x ) by Theorem 5.2.2 in [65], and the second term is bounded by C 2 ˜ c ( x ) + C 3 by following the same proof of Lemma B.11 as ϕ
is
Lipschitz.
By combining (102) and (103), we get
[Equation]
where we use that E x (cid:2)
˜ c
2 ( x ) (cid:3) = Θ(1) .
Hence, the Z i -s are i.i.d.
sub-exponential random variables, with sub-exponential norm bounded by a numerical constant.
An application of Bernstein inequality (cf.
Corollary 2.8.3.
in [65]) gives that
[Equation]
where c, C 5 are numerical constants.
Let us consider the first term in (100):
[Equation] [Equation]
where the equality comes again from the rotational invariance of the Gaussian distribution of w 1 .
We will show that
[Equation]
with probability at least 1 − C exp( − cn L − 1 ) over ( W k ) l − 1 k =1 .
The upper bound in (107) follows from the same passages in (77), as E x [˜ c 2 ( x )]
= Θ(1) .
We now prove the lower bound.
By Lemma C.1, we have that there exist numerical constants c 2 > c 1 >such that ˜ c ( x ) ∈
[ c 1 , c 2 ] with probability at least 1 − C exp( − cn L − 1 ) over x and ( W k ) l − 1 k
=1 .
Hence, with probability at least 1 − 2 C exp( − cn L − 1 ) over ( W k ) l − 1 k
=1 , we have that
[Equation]
where we use the symbol P x to highlight that this last probability is taken over x .
Let us condition on a realization of ( W k ) l − 1 k =1 s.t.
(108) holds.
Then, we have
[Equation]
By Lemma B.10, we have that φ ( c ) =
E
ρ 1 (cid:2) ϕ 2 ( cρ 1 ) (cid:3) is continuous in c .
Therefore, by Weierstrass theorem, there exists a strictly positive c ∗ ∈
[ c 1 , c 2 ] such that inf c ∈ [ c 1 ,c 2 ] E
[Equation] [Equation]
ϕ 2 ( cρ
[Equation]
(cid:3)
[Equation]
E ρ 1 (cid:2) ϕ 2 ( c ∗ ρ 1 ) (cid:3) .
Thus,
[Equation]
110) where the last equality is a consequence of Lemma B.12.
This concludes the proof of the lower bound in (107).
By setting t =
E
ρ 1 (cid:2) ϕ 2 ( c ∗ ρ 1 ) (cid:3) / 4 into (105) and using (107) and (100)
, we conclude that
[Equation]
(111) with probability at least 1 − C exp( − cn L − 1 ) , for some numerical constants C and c , which concludes the proof.
Lemma C.3 (Expected ℓ 2 norm of features) .
Let x ∼ P X .
Then, for every 0 ≤ l ≤ L − 1 ,
[Equation]
112) with probability at least 1 − C exp( − cn L − 1 ) over ( W k )
lk =1 .
As usual, c and C are numerical constants.
Proof.
We condition on the events
[Equation]
113) and
[Equation]
114) which happen with probability at least 1 − C exp( − cn L − 1 ) over ( W k )
lk =1 by Lemma C.2 and B.2.
The upper bound is a direct consequence of Jensen’s inequality:
[Equation]
For the lower bound
[Equation]
we use Lemma B.
[Equation]
and we obtain
[Equation]
Lemma C.4 ( ℓ 2 norms of centered features) .
Let x ∼ P X .
Then, for every 0 ≤ l ≤ L − 1 , the following results hold.
With probability at least 1 − C exp( − cn L − 1 ) over ( W k ) lk
=1 , we have that
[Equation]
With probability at least 1 − C exp( − cn L − 1 ) over ( W k ) lk
=1 , we have that
[Equation]
With probability at least 1 − C exp( − cn L − 1 ) over ( W k ) lk
=1 and x , we have that
[Equation]
Proof.
The argument is by induction over l .
The base case for (117) follows directly from Assumption 2.1, since f 0 ( x ) = x .
Since the ℓ 2 norm is a 1-Lipschitz function, from Jensen inequality and Lemma
B.13 we readily obtain the base case for (118).
Note that ∥∥ x − E x
[ x ] ∥ 2 ∥ Lip ≤ 1 .
Then, the base case for (119) is a direct consequence of Assumption 2.2 on x and of the base case of (118), and it holds with probability at least 1 − C ′
exp( − cd ) ≥ 1 − C ′ exp( − cn L − 1 ) over x .
By inductive hypothesis, we assume the three statements to be true for layer l − 1 , for l ∈
[ L − 1] .
We will now prove (117) for layer l .
To ease the notation, we use the shorthands f ( x ) :
= f l − 1 ( x ) , f = E x
[ f ( x )] , ˜ f
( x ) =
f
( x ) − f ,
W :=
W l and w i =
W : i .
We also define ˜ c ( x ) =
β l ∥ f ( x ) ∥ / √ n l − 1 and ˜ c
= E x [˜ c ( x )] .
We condition on the following events in the probability space of ( W k ) l − 1 k =1 :
(a) ∥
f ( x ) ∥ Lip = O (1) , which happens with probability at least 1 − C ′ exp( − cn L − 1 ) by Lemma
B.
(b) ˜ c
= Θ(1) , which happens with probability at least 1 − C ′ exp( − cn L − 1 ) by Lemma C.3.
Notice
that by Jensen inequality this
also implies ∥ f ∥ 22 = O ( n l − 1 ) .
(c) By inductive hypothesis, we have that, with probability at least 1 − C ′
exp( − cn n − 1 ) over x and ( W k ) l − 1 k
=1 ,
[Equation]
n l − 1 ) .
[Equation] [Equation]
Hence, with probability at least 1 − 2 C ′ exp( − cn n − 1 ) over ( W k ) l − 1 k
=
1 , we have that
[Equation]
for some numerical constants c 2 > c 1 > 0 .
In (121), we use the symbol P x to highlight that this last probability is taken over x .
For the rest of the argument, we condition on a realization of ( W k ) l − 1 k =1 s.t.
(121) holds.
By taking a union bound, the events (a)-(c) happen with probability at least 1 − 4 C ′ exp( − cn n − 1 ) over ( W k ) l − 1 k =1 .
Now, we can write
[Equation]
where we use that the w i -s are identically distributed and we have defined the independent, meanrandom variables
[Equation]
As in the proof of Lemma C.2, in the definition of Z i , the randomness comes only from w i , since we are conditioning on ( W k ) l − 1 k =1 .
We have that
[Equation]
where C 0 is a numerical constant, the first inequality follows from Exercise 2.7.10 of [65], the second line follows from Jensen’s inequality as ∥·∥ ψ 1 is convex, the equality follows from Lemma 2.7.6 of [65], and the last line follows from Jensen’s inequality as ∥·∥
ψ 2 is convex.
Recall that ( W ) i,j ∼ i .
i .
d .
N (0 , β 2 l /n L − 1 ) and that the Gaussian distribution is rotationally invariant.
Thus, ϕ ′ (cid:0) w ⊤ i f ( x ) (cid:1) has the same distribution as ϕ ′
(˜ c ( x ) ρ i ) , where ( ρ i ) n L − 1 i =1 ∼ i .
i .
d .
N (0 , 1) and also independent of ˜ c ( x ) .
Therefore,
[Equation]
where the first term in the RHS in the second line is bounded by C 1 ˜ c ( x ) for Theorem 5.2.2 in [65], and the second term is bounded by C 2 ˜ c ( x ) + C 3 by following the same proof of Lemma B.11.
Merging together (125) and (124) we get
[Equation]
where in the third line we use Lemma B.14.
Hence, the Z i -s are i.i.d.
sub-exponential random variables, with sub-exponential norm bounded by a numerical constant.
An application of Bernstein inequality (cf.
Corollary 2.8.3.
in [65]) gives that
[Equation]
where c, C are numerical constants.
We recall that this probability is intended over W l .
Let’s now focus on the first term in the last line of (122), using the shorthand w = w 1 , to ease the notation.
We can rewrite this term as
[Equation]
The aim of this part of the proof is to show that the quantity in (128) is Θ(1) .
Recall that ( W l ) i,j ∼ i .
i .
d .
N (0 , β 2 l /n l − 1 ) and that the Gaussian distribution is rotationally invariant.
Thus, ϕ (cid:0) w ⊤ f ( x ) (cid:1) has the same distribution as ϕ (˜ c ( x ) ρ ) , where ρ ∼ N (0 , 1) is independent
We therefore have
[Equation]
129) where the third line follows from Lemma B.
[Equation]
and the last passage follows from Lemma B.14.
Similarly, we have
[Equation]
where ρ 1 and ρ 2 indicate two standard Gaussian random variables with correlation f ( x )
⊤ f ( y ) / ( ∥ f ( x ) ∥
∥ f ( y ) ∥ )
, the third line follows from B.
[Equation]
and the last passage follows from
Lemma B.14.
By combining (129) and (130)
[Equation]
we have that
[Equation]
with
[Equation]
where we have set ˜ ϕ
( t ) = ϕ
(˜ ct ) .
Hence, in order to obtain that the quantity in (128) is Θ(1) , it suffices to prove that ξ = Θ(1) .
As ϕ is Lipschitz and ˜ c is Θ(1) , ˜ ϕ is also Lipschitz, which readily implies that ξ = O (1) .
We now prove that ξ = Ω(1) .
By exploiting the Hermite expansion of ˜ ϕ , we have that
[Equation]
where µ i is the i -th Hermite coefficient of ˜ ϕ .
Note that, since we conditioned on ˜ c
= Θ(1) , these coefficients are numerical constants.
As ϕ (and therefore ˜ ϕ ) is non constant, there exist j >such that µ j ̸ = 0 .
Furthermore, we have that the sum in (133) contains only positive terms, as | f ( x ) ⊤ f ( y ) | ≤ ∥ f ( x ) ∥ · ∥ f ( y ) ∥ by Cauchy-Schwarz.
Therefore, in order to show that ξ = Ω(1) , it suffices to prove that, for all j > 0 ,
[Equation]
where C 0 is an absolute constant strictly smaller than 1 .
Furthermore, (134) is implied by the following:
[Equation]
where C 1 < 1 and c 1 > 0 are numerical constants.
By writing f ( x ) as ˜ f
( x ) + f and f ( y ) as ˜ f ( y ) + f , we have
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Equation]
Let us provide bounds on the various terms appearing in (136):
(i) Part (d) of the conditioning (cf.
(121)) gives
that
[Equation]
for some numerical constant c > 0 .
(ii) Part (b) of
the conditioning gives that
[Equation]
(iii) Part (a) of the conditioning gives that ∥ f l − 1 ( x ) ∥ Lip = O (1) , and part (
b
) of the
conditioning gives that E y [
∥
f
l − 1 (
y )
∥ ]
= Θ( √ n l − 1 ) .
Hence, as y ∼ P X , Assumption 2.2 implies that
[Equation]
with probability at least 1 − 2 exp( − cn l − 1 ) over y , where c is a numerical constant.
Furthermore, by recalling that E x [ ˜ f
( x )]
= 0 and using again Assumption 2.2, we have that, for any fixed vector u ,
[Equation]
where c 0 is another numerical constant.
Since x and y are independent, (140) implies that
[Equation]
where the first inequality holds for every y , and the second inequality holds with probability at least 1 − 2 exp( − cn l − 1 ) over y by (139).
As a result, we have
[Equation]
142) (iv) By setting t = n 3 / 4 l − 1 and u = f into (140), we obtain
[Equation]
where c 4 is a numerical constant and we have also used (138).
(v) Finally, as x and y are independent, (143) implies that
[Equation]
By plugging into (136) the bounds (137), (142), (143) and (144), we obtain that
[Equation]
By using also (138), we have that (135) readily follows from (145).
From (135), we have that ξ = Θ(1) .
Hence, the quantity in (128) is Θ(1) and in particular it is lower bounded by a numerical constant, call it C 0 .
By setting t =
C 0 / 2 in (127), we conclude that
[Equation] [Equation]
with probability at least 1 − 2 exp( − cn l − 1 ) over W l , where c is an absolute constant.
By taking into account the conditioning made at the beginning of the proof over the space ( W k ) l − 1 k =1 , we obtain that (146) holds with probability at least 1 − 5 C ′ exp( − cn n − 1 ) − 2 exp( − cn n − 1 ) ≥ 1 − C exp( − cn n − 1 ) over ( W k ) lk
=1 , where C is a numerical constant, which concludes the proof of (117).
Finally, we prove (118) and (119), again for layer l .
By Lemma B.2, we have that ∥ f l ( x ) ∥ Lip = O (1) , with probability at least 1 − C ′
exp
( − cn L − 1 ) over ( W k ) lk =1 .
By conditioning on this event, we also have that ∥∥ f l ( x ) − E
x
[ f l ( x )]
∥ 2 ∥ Lip = O (1) .
Furthermore, we condition on a realization of ( W k ) lk
=
1 such that (117) holds.
To obtain (118), we apply Jensen’s inequality and Lemma B.13, which give that
[Equation]
147) with probability at least 1 − C ′
exp
( − cn L
[Equation]
−
[Equation] [Equation]
C exp( − cn n − 1 ) over ( W k ) lk =1 .
To obtain (119), we condition on a realization of ( W k ) lk
=1 such that ∥∥ f l ( x ) − E x
[ f l ( x )]
∥ 2 ∥ Lip =
O (1) and (118) holds.
Then, by Assumption 2.2, we have that
[Equation]
where c is a numerical constant.
This gives that
[Equation]
with probability at least 1 −
[Equation]
C exp( − cn n − 1 ) over x and ( W k ) lk =1 , which concludes the proof.
Lemma C.5 ( ℓ 2 norms of centered backpropagation) .
Let x ∼ P X .
Then, we have
[Equation]
with probability at least 1 − 10 exp( − c log 2 n L − 1 ) − C exp( − cn L − 1 ) over x and ( W k )
Lk =1 over ( W k ) lk
=1 and x .
Proof.
An application of Lemma C.4 for l =
L − 2 gives that
[Equation]
with probability at least 1 − C ′ exp( − cn L − 1 ) over ( W k )
L − 2 k
=1 and x .
To ease the notation, we use the shorthands f ( x ) :
=
f L − 2 ( x ) , f = E x
[ f ( x )] , ˜ f
( x ) =
f
( x ) − f ,
W :=
W L − 1 and w i =
W : i .
We also define ˜ c ( x ) =
β l ∥ f ( x ) ∥ / √ n l − 1 and ˜ c
= E x [˜ c ( x )] .
As in Lemma C.4, we condition on the 3 events (a)-(c), which jointly happen with probability at least 1 − 4 C ′ exp( − cn L − 1 ) over ( W k )
L − 2 k
=1 .
Note that, to condition on the event (c), we use (151).
Now, we can write
[Equation] [Equation]
where we use that the w i -s are identically distributed and we have defined the independent, meanrandom variables
[Equation]
Note that in the definition of Z i the randomness comes only from w i and ( D L ) ii , since we are conditioning on ( W k )
L − 2 k
=1 .
If we fix the ( D L ) ii -s and follow the same argument in (124)-(126) (cf.
the proof of Lemma C.4), we have
[Equation]
where C 0 is a numerical constant and we have used that ϕ
′ is Lipschitz.
Let E bad be the event s.t. max i ( D L ) 2 ii > log 2 n L − 1 .
Then, by following the same argument as in Lemma B.3, we have that
[Equation]
Hence, by conditioning on E c bad , we have that
[Equation]
By applying Bernstein inequality (cf.
Corollary 2.8.3.
in [65]), we get
[Equation]
for some numerical constant c .
By combining (155) and (157)
[Equation]
we obtain that
[Equation]
where this probability is over W L − 1 and W L .
Let’s now focus on the first term in the last line of (152).
In particular, we have that
[Equation]
This can be proven by following the same argument in (129)-(145) (cf.
the proof of Lemma C.4), as ϕ ′ is Lipschitz and non-constant.
Next
[Equation]
we re-write (152) as
[Equation]
where we have defined the independent, mean-0, sub-exponential random variables
[Equation]
Since the ( D L ) ii -s are standard Gaussian, we have
[Equation]
Hence, another application of Bernstein inequality (cf.
Corollary 2.8.3.
in [65]) allows us to conclude that
[Equation]
with probability at least 1 − 2 exp( − c √ n L − 1 ) over W L .
Thus, by using (158) and (163), and taking into account the initial conditioning over ( W k )
L − 2 k
=1 , we conclude that
[Equation]
with probability at least 1 − 6 exp( − c log 2 n L − 1 ) − 6 C ′ exp( − cn L − 1 ) over ( W k )
Lk =1 .
Proceeding in a similar fashion as in Lemma C.4, we apply Lemma B.4, which gives that ∥ D L ϕ ′ ( g L − 1 ( x ))
∥
Lip = O
(log n L − 1 ) , with probability at least 1 − 2 exp( − c log 2 n L − 1 ) −
C ′ exp ( − n L − 1 ) over ( W k )
Lk =1 .
By conditioning on this event, we also have that ∥∥
[Equation] [Equation]
ϕ
[Equation]
g L −
[Equation]
( x ))] ∥ 2 ∥ Lip = O (log n L − 1 ) .
Furthermore, we condition on a realization of ( W k )
Lk =1 such that (164) holds.
We can now apply Jensen’s inequality and Lemma B.13, to obtain that
[Equation]
165) with probability at least 1 − 8 exp( − c log 2 n L − 1 ) − 7 C ′ exp( − cn L − 1 ) over ( W k )
Lk =1 .
Finally, we condition on a realization of ( W k )
Lk = such that ∥∥ D
[Equation]
−
[Equation]
x
[Equation]
E x
[ D
[Equation]
ϕ
[Equation]
g
[Equation]
( x
[Equation] [Equation]
Lip = O (log n L − 1 ) and (165) hold.
Then, by
Assumption 2.2, we have that
[Equation]
This gives that
[Equation]
(167) with probability at least 1 − 10 exp( − c log 2 n L − 1 ) − 8 C ′ exp( − cn L − 1 ) over x and ( W k )
Lk =1 .
This concludes the proof.
D Proofs for Part 1: Centering
D.1 Step (a): Centering F L − 2 and B L −
Lemma D.1 (Centering F L − 2 and B L − 1 ) .
Consider the setting of Theorem 3.1, let F L − 2 ∈ R N × n L −be the feature matrix at layer L − 2 , and let B L − 1 contain the backpropagation terms from layer L − 1 , i.e. ( B L − 1 ) i :
= D L ϕ ′ ( g L − 1 ( x i )) .
Let J L − 2 J ⊤ L − 2 =
F L − 2 F ⊤ L − 2 ◦ B L − 1 B ⊤ L − 1 and ˜ J FB
˜
J ⊤ FB =
˜ F L − 2 ˜ F ⊤ L − 2 ◦ ˜ B L − 1 ˜ B ⊤ L − 1 , where ˜ F L − 2 =
F L − 2 − E X
[ F L − 2 ] and ˜ B L − 1 =
B L − 1 − E X
[ B L − 1 ] .
Then, we have that
[Equation] [Equation]
with probability at least 1 − C exp( − cn L − 1 ) − 4 exp( − c log 2 N ) − 8 exp( − c log 2 n L − 1 ) over ( W k )
L k =1 and ( x i ) N i =1 ∼ i .
i .
d .
P X , where c and C are numerical constants.
Proof.
By Lemma B.2 and Lemma B.4, we have that ∥
f L − 2 ∥
Lip =
O (1) and ∥ D L ϕ ′
( g L − 1 ( x ))
∥
Lip = O
(log n L − 1 ) with probability 1 − C exp( − n L − 1 ) − 2 exp( − c log 2 n L − 1 ) over ( W k )
Lk =1 .
We will condition on these events for the rest of the proof.
Let’s define
˜ J F ˜ J ⊤ F = ˜
F L − 2 ˜ F ⊤ L − 2 ◦ B L − 1 B ⊤ L − 1 .
We can now re-write the quantity J L − 2 J ⊤ L − 2 as follows:
[Equation]
where ν =
E x i [( F L − 2 )
i : ] ∈ R n L − 2 (independent on i , since the x i -s are i.i.d.), Λ is a diagonal matrix such that Λ ii
= ν ⊤ ( F L − 2 )
i : − ∥ ν ∥
=: µ ( x i ) , and 1 ∈ R N is a vector full of ones.
The last step is justified since the Hadamard product of PSD matrices is PSD by the Schur product theorem.
Notice that we are assuming ∥ ν ∥ 2 ̸ = 0 .
In fact, if ∥ ν ∥ 2 = 0 , then we immediately have that J =
˜ J F .
Expanding in an analogous way the term ˜ J F ˜ J ⊤ F , we get
[Equation]
where η = E x i [( B L − 1 ) i : ] ∈ R n L − 1 (independent on i , since the x i -s are i.i.d.), Γ is a diagonal matrix such that Γ ii = η ⊤ ( B L − 1 ) i : − ∥ η ∥
=: ζ ( x i ) .
The last step is justified by the fact that the following matrix is PSD
[Equation]
since it is the Hadamard product of two PSD matrices.
Notice that we are assuming ∥ η ∥ 2 ̸ = 0 .
In fact, if ∥ η ∥ 2 =
[Equation]
then we immediately have that ˜ J F =
˜
J FB .
Taking into account the following relations
[Equation]
we can simplify the second and the fourth term of the RHS of equation (170) as follows
[Equation]
Merging this last relation with (170) we get
[Equation]
Note that ∥ µ ∥ Lip ≤ ∥
f L − 2 ∥
Lip ∥ ν ∥ 2 , and that E x i [ µ ( x i )]
= 0 for all i ∈ [ N ] .
Thus, by using
Assumption 2.2 on x i and exploiting the initial conditioning on the weights
[Equation]
we have that
[Equation]
where the probability is intended over x i ∼ P X .
Thus, following the same argument of Lemma B.3, the last relation implies that
[Equation]
with probability at least 1 − 2 exp( − c log 2 N ) , where c is a numerical constant, and the probability is intended over { x i } Ni =1 .
This implies that
[Equation]
where the second equality is justified by Lemma B.9, and the last by Lemma B.1.
This result holds with probability 1 − C exp( − cn L − 1 ) − 2 exp( − c log 2 N ) − 4 exp( − c log 2 n L − 1 ) over ( W k )
Lk
= and ( x i ) Ni =1 .
Note that ∥ ζ ∥ Lip ≤ ∥ D L ϕ ′ ( g L − 1 ( x ))
∥
Lip ∥ η ∥ 2 , and that E x i [ ζ ( x i )]
= 0 for all i ∈ [ N ] .
Thus, by using Assumption 2.2 on x i and exploiting the initial conditioning on the weights, we have that
[Equation]
where the probability is intended over x i ∼ P X .
Thus, following the same argument of Lemma B.3, the last relation implies that
[Equation]
with probability at least 1 − 2 exp( − c log 2 N ) , where c is a numerical constant, and the probability is intended over { x i } Ni =1 .
This implies that
[Equation]
∥
[Equation]
≤
[Equation]
where the second equality is justified by Lemma B.8, and the last by Lemma B.1.
This result holds with probability 1 − C exp( − cn L − 1 ) − 2 exp( − c log 2 N ) − 4 exp( − c log 2 n L − 1 ) over ( W k )
Lk
= and ( x i ) Ni =1 .
By merging (176) and (179) with (173), we readily obtain the desired result.
D.2 Step (b): Centering everything
Lemma D.2 (Centering everything) .
Consider the setting of Theorem 3.1, let F L − 2 ∈ R N × n L − 2 be the feature matrix at layer L − 2 , and let B L − 1 contain backpropagation terms from layer L − 1 , i.e. ( B L − 1 ) i :
= D L ϕ ′ ( g L − 1 ( x i )) .
Let ˜ J FB
˜
J ⊤ FB =
˜ F L − 2 ˜ F ⊤ L − 2 ◦ ˜ B L − 1 ˜ B ⊤ L − 1 and ˜ J L − 2 ˜ J ⊤ L − 2 =
˜
[Equation]
˜
[Equation]
˜ B
[Equation]
1 ] , where ˜ F L − 2 = F L − 2 − E X
[ F L − 2 ] and ˜ B L − 1 =
B L − 1 − E X
[ B L − 1 ] .
Then, we have that
[Equation]
with probability at least 1 − C exp( − n L −
[Equation]
−
[Equation] [Equation]
c
[Equation]
2 N ) over ( W k )
Lk =1 and ( x i )
Ni
=1 .
Proof.
Note that the i -th row of ˜ J FB is now in the form
[Equation]
where we recall that ˜
f L − 2 ( x i ) =
f L − 2 ( x i ) − E x i
[ f L − 2 ( x i )] and ˜ ϕ
′ ( g L − 1 ( x i ))
=
ϕ ′
( g L − 1 ( x i ))
−
E [ ϕ ′
( g L − 1 ( x i ))] .
Furthermore, ( ˜ J L − 2 )
i : =
( ˜ J FB )
i : − E x i [( ˜ J FB )
i : ] .
Then, by following similar passages as in (169), we have
[Equation]
where we have defined
[Equation]
which is independent on i (as the x i
-s are i.i.d.), and Λ is a diagonal matrix that contains in the i -th position
[Equation]
The last passage of (182) is true since we are subtracting a PSD matrix.
An application of Lemma B.2 gives that ∥ f L − 2 ( x i ) ∥
Lip and ∥ ϕ ′ ( g L − 1 ( x i ))
∥
Lip are upper bounded by a numerical constant both with probability at least 1 − C exp( − n L − 1 ) over ( W k )
L − 1 k
=1 .
Let us condition on this event on the probability space of ( W k )
L − 1 k
=1 .
Then, we can apply Lemma B.5 with u ( x ) :=
˜
f L − 2 ( x ) and v ( x ) :=
˜ ϕ ′ ( g L − 1 ( x )) , which implies that
[Equation] [Equation]
In (185), C is a numerical constant and the last equality holds with probability at least 1 − 2 exp( − c log 2 n L − 1 ) over W L by Lemma B.3.
Thus,
[Equation]
where the probability is intended over x i ∼ P X and c is a numerical constant.
Thus, following the same argument of Lemma B.3, the last relation implies that
[Equation]
with probability at least 1 − 2 exp( − c log 2 N ) , where c is a numerical constant, and the probability is intended over { x i } Ni =1 .
Thus, with probability 1 − C exp( − n L − 1 ) − 2 exp( − c log 2 n L − 1 ) − 2 exp( − c log 2 N ) over ( W k )
Lk
= and ( x i ) Ni
=1 , we have
[Equation]
where the last equality is a consequence of Lemma B.1.
The desired result follows from merging (188) with (182).
E Proofs for Part 2: Bounding the Centered Jacobian E.1 ℓ 2 and sub-exponential norms of centered Jacobian
We start by providing an upper bound on the quantity E x (cid:104) ˜
f L − 2 ( x )( D L ˜ ϕ
′ ( g L − 1 ( x )))
⊤ (cid:105) .
This preliminary result will be useful when bounding the ℓ 2 norm of the rows of the centered Jacobian.
Lemma E.1.
Consider the setting of Theorem 3.1, let x ∼ P X , and let A be defined as
[Equation]
Then, we have
[Equation]
with probability at least 1 − 2 exp( − c log 2 n L − 1 ) − C exp( − cn L − 1 ) over ( W k )
Lk =1 , where c is an absolute constant.
Proof.
We condition on ∥
f L − 2 ( x ) ∥
Lip = O (1) and on ∥ ϕ ′ ( g L − 1 ( x ))
∥
Lip = O (1) .
By Lemma B.2, these two conditions hold with probability at least 1 − C ′
exp
( − cn L − 1 ) over ( W k )
L − 1 k
=1 .
Then, as
P X satisfies Assumption 2.2, we have that (cid:13)(cid:13)(cid:13) ˜
f L − 2 ( x ) (cid:13)(cid:13)(cid:13) ψ 2 = O (1) and (cid:13)(cid:13)(cid:13) ˜ ϕ
′ ( g L − 1 ( x )) (cid:13)(cid:13)(cid:13) ψ 2 = O (1) .
Hence, an application of Lemma B.6 gives that
[Equation]
where C 1 is a numerical constant.
The following chain of inequalities holds:
[Equation]
where the third line uses (191), and the last holds with probability 1 − 2 exp( − c log 2 n L − 1 ) over W L , because of Lemma B.3.
Taking into account the initial conditioning, we get the desired result.
The next two results provide bounds on the ℓ 2 norm and on the sub-exponential ψ 1 norm of the rows of ˜ J L − 2 , respectively.
Lemma E.2 ( ℓ 2 norm of rows of centered Jacobian) .
Consider the setting of Theorem 3.1, let x ∼ P X , and let ˜ J x be defined as
[Equation]
Then, we have
[Equation]
194) with probability at least 1 − C exp( − cn L − 1 )
− 12 exp( − c log 2 n L − 1 ) over x and ( W k )
Lk =1 and x .
Proof.
We have that
[Equation]
where A is defined in (189).
The second equality is justified by the identity ∥ u ⊗ v ∥ 2 = (cid:13)(cid:13) uv ⊤ (cid:13)(cid:13) F that holds for any vectors u, v .
An application of the triangle inequality gives that
[Equation]
Lemma C.4 gives that
[Equation]
with probability at least 1 − C ′ exp( − cn L − 1 ) over ( W k )
Lk =1 and x .
Furthermore, Lemma C.5 gives that
[Equation]
with probability at least 1 − 10 exp( − c log 2 n L − 1 ) − C ′ exp( − cn L − 1 ) over x and ( W k )
Lk =1 .
By combining (196), (197), (198) and the bound on ∥ A ∥ F provided by Lemma E.1, we conclude that
[Equation]
with probability at least
[Equation]
over x and ( W k )
Lk =1 , which gives the desired result.
Lemma E.3 (Sub-exponential norm of rows of centered Jacobian) .
Consider the setting of Theorem 3.1, let x ∼ P X , and let ˜ J x be defined as in (193) .
Fix a realization of ( W k )
Lk =1 .
Then, with probability at least 1 − 2 exp( − c log 2 n L − 1 ) − C exp( − cn L − 1 ) over this realization ( c being a numerical constant), we have that
[Equation]
Proof.
We condition on ∥
f L − 2 ( x ) ∥
Lip = O (1) and on ∥ ϕ ′ ( g L − 1 ( x ))
∥
Lip = O (1) .
By Lemma B.2, these two conditions hold with probability at least 1 − C exp( − cn L − 1 ) over ( W k )
L − 1 k
=1 .
Then, we have
[Equation]
where the third line follows from Lemma B.5 and the last inequality holds with probability at least 1 − 2 exp( − c log 2 n L − 1 ) over W L by Lemma B.3.
Taking into account the initial conditioning, we get the desired result.
E.2 Proof of Proposition 3.
Proof of Proposition 3.3.
Following the notation in [2], we define
[Equation]
Then, for any z ∈ R N with unit norm, we have that
[Equation] [Equation]
which implies that
[Equation]
In our case, ˜ J i : ∈ R n L − 2 n L − 1 . Notice that this dimension is indicated with n in Theorem 3.2 of [2].
In the statement of the mentioned Theorem, let’s fix r = 1 , m = N , and θ =
( N/ ( n L − 1 n L − 2 )) 1 / 4 < 1 / 4 .
Then, we have that the condition required to apply Theorem 3.2 is satisfied, i.e.
[Equation]
where the inequality follows from Assumption 2.5.
By combining (205) with the upper bound on B given by Theorem 3.2 of [2], the desired result readily follows.
E.3 Proof of Theorem 3.
Proof of Theorem 3.4.
By Lemma E.3, we have that, with probability at least 1 − 2 exp( − c log 2 n L − 1 ) − C ′ exp( − cn L − 1 ) over ( W k )
Lk =1 , the rows of ˜ J are sub-exponential (with respect to the randomness in ( x i ) Ni =1 ).
In particular, we have that
[Equation]
Furthermore, by Lemma E.2, we have that
[Equation]
with probability at least 1 − p over x i and ( W k )
Lk =1 , where to ease the notation we have defined p :=
C ′ exp( − c 0 n L − 1 ) + 12 exp( − c 0 log 2 n L − 1 ) .
Hence, with probability at least 1 − √ p over ( W k )
Lk =1 , we have that
[Equation] [Equation]
for some numerical constants c 2 > c 1 > 0 .
In (209), we use the symbol P x i to highlight that this probability is taken over x i .
For the rest of the argument, we condition on a realization of ( W k )
Lk
= s.t. (207) and (209) hold.
Then, by performing a union bound over the samples, we have that
[Equation]
and
[Equation]
with probability at least 1 − N √ p over ( x i ) Ni =1 .
Next, we apply Proposition 3.3 with K = 1 , K ′ =
c 2 and
[Equation] [Equation]
Note that (22) in Lemma B.1 gives that N 1 / 4 · log 2 n L −
= o (( n L − 1 n L − 2 ) 1 / 4 ) , which justifies the last line.
Thus, (13) implies that
[Equation]
with probability at least
[Equation]
where the last inequality follows from (211).
By taking into account the conditioning over ( W k )
Lk
= made in order to guarantee (207) and (209)
[Equation]
we conclude that λ min (cid:16) ˜ J ˜ J ⊤ (cid:17)
=
Ω( n L − 1 n L − 2 ) with probability at least
[Equation]
n L − 1 ) , (215) over ( x i ) Ni =1 and ( W k )
Lk =1 , which gives the desired result.
F Proof of the Upper Bound
Before giving the proof of the upper bound 7, we provide again its statement for the reader’s convenience.
Lemma F.1 (Upper bound on the smallest NTK eigenvalue) .
Consider the setting of Theorem 3.1, and let K be the NTK Gram matrix (3) .
Then, we have
[Equation]
(216) with probability at least 1 − C exp( − cn L − 1 ) over ( x i ) Ni =1 and ( W k )
Lk =1 , where c and C are numerical constants.
Proof.
By using the expression in (8), we have that
[Equation]
An application of Lemma C.1 gives that
[Equation] [Equation]
with probability at least 1 − C ′ exp( − cn L − 1 ) over ( W k ) lk =1 and x 1 .
We condition on the event such that (218) holds for all l ∈ { 0 , . . .
, L − 1 } .
This happens with probability at least 1 − C ′′ exp( − cn L − 1 ) over ( W k )
Lk
=1 and x 1 .
By definition, we have that ∥ B L ∥ 2 = 1 and that, for l ∈ [ L − 1] , ∥ ( B l ) 1: ∥
=
(cid:13)(cid:13) (cid:13) (cid:13) (cid:13)
L − 1 (cid:89) k
= l Σ k ( x 1 ) W k + (cid:13)(cid:13) (cid:13) (cid:13) (cid:13)
[Equation]
Since Σ k ( x 1 ) = diag([ ϕ ′ ( g k,j ( x 1 ))]
n k j =1 ) , by Assumption 2.3, we have that
[Equation]
Let us now condition on the following two events: (i)
∥ W k ∥ op
= Θ(1) , for all k ∈
[ L − 1] (this happens with probability at least 1 − C ′ exp( − cn L − 1 ) over ( W k )
L − 1 k
=1 , see (33) in the proof of Lemma
B.2), and (ii) ∥ W L ∥
2 = Θ( √ n L − 1 ) (this happens with probability at least 1 − exp( − cn L − 1 ) over
W L , by Theorem 3.1.1 in [65]).
Then, we readily get
[Equation]
Taking the intersection of all the events over which we conditioned, we finally obtain
[Equation]
with probability at least 1 − (1 + C ′ + C ′′ ) exp( − cn L − 1 ) over ( W k )
Lk =1 and x 1 , where in the last step we have used Assumption 2.4.
By combining (217) and (222), the desired result follows.
G Proof of Corollary 4.
Proof of Corollary 4.1.
By Theorem 3.1, we have that the smallest eigenvalue of JJ ⊤ is bounded away from zero with probability at least 1 − p over ( x i ) N i =1 and ( W k )
Lk =1 , where
[Equation]
Hence, with probability at least 1 − p over ( x i ) Ni
=1 , there exists a set of parameters θ 0 such that J ( θ 0 ) has a right inverse.
Thus, for all Y ∈ R N , there exists θ ′ such that
[Equation]
This can also be written, for all i ∈ [ N ] , as
[Equation]
Then, for all ε > 0 , there exists h ∗ such that, for all i ∈ [ N ] ,
[Equation]
where
[Equation]
Finally, the desired result follows by noticing that f ∗ can be implemented by a network with the same depth and twice more neurons at every hidden layer.
H Proof of Theorem 4.
Notation for this appendix.
In this appendix, we use J ( θ ) to denote the Jacobian of the network output F L , evaluated in θ .
We recall that J ( θ ) is a matrix with N rows and (cid:80) L − 3 l =0 n l n l +
+ 2 n L − 2 n L − 1 + 2 n L − 1 columns (for the optimization result, we assume that the ( L − 1) -th layer has an even number of neurons and denote its width as 2 n L − 1 ).
Let K ( θ ) =
J ( θ )( J ( θ ))
⊤ be the associated empirical NTK Gram matrix, and let θ 0 be the initialization defined in (18).
We also make the dependence on θ explicit for feature vectors and backpropagation terms: the feature vector at the l -th layer with input x i and network parameter θ is denoted by f l ( θ, x i ) , and the corresponding backpropagation term is denoted by b l ( θ, x i ) , where b l ( θ, x i )
=
( B l ( θ ))
i : .
Finally, we use W l ( θ ) to denote the weights of the l -th layer evaluated at the parameter θ .
A straightforward application of Theorem 2.1 in [49] gives the following proposition.
Proposition H.1.
Consider solving the least-squares optimization problem
[Equation]
by running gradient descent updates of the form θ t +
=
θ t − η ∇L ( θ t ) , with some initialization ˜ θ 0 .
Assume there exists α, β ∈ R , such that, if we define D = B
(˜ θ 0 , R ) as the ℓ 2 ball centered in ˜ θ 0 with radius R , with
[Equation]
the following holds
[Equation]
Then, by setting η ≤ 1 / (2 β 2 ) , we have that, for all t ≥ 1 ,
[Equation]
In order to apply this proposition with initialization ˜ θ
= θ 0 , we need to prove that the necessary assumptions hold.
We will do so by showing the following intermediate results:
• Lemma H.2 shows that, at the initial point θ 0 , the network output is 0 and the smallest NTK eigenvalue is lower bounded.
• Lemma H.3 gives a tight estimate on the operator norm of the weights inside a ball D centered at θ 0 and with radius R
= o (1) .
• Lemma H.4 gives an upper bound on the distance between a feature vector in D and the feature vector at θ 0 .
• Lemma H.5 gives upper bounds on the ℓ 2 norm and the ℓ 2 distance between feature vectors in D .
• Lemmas H.6 and H.7 give upper bounds on the ℓ 2 norm and the ℓ 2 distance of backpropagation terms in D , respectively.
• Lemma H.8 gives an upper bound on the difference in operator norm between Jacobians in D .
• Finally, Lemma H.9 gives upper and lower bounds on the
NTK spectrum in D .
Lemma H.2 (Network output and smallest NTK eigenvalue at initialization) .
Let θ 0 be defined in (18) .
Then, we have that, for all x ∈ R d , f
[Equation]
x,
[Equation]
Furthermore, we have that
[Equation]
with probability at least 1 − C Ne − c log 2 n L − 1 − Ce − c log 2 N over ( x i )
Ni
=1 ∼ i .
i .
d .
P X
and θ 0 , where c, c 1 and C are numerical constants.
Proof.
By definition (18) of the initialization θ 0 , we have that
[Equation]
where in the second equality we use that W (2) L − 1 ( θ 0 )
=
W (1) L − 1 ( θ 0 ) and that W (2) L ( θ 0 ) =
− W (1) L ( θ 0 ) .
This proves (233).
Let us now compute the Jacobian at initialization J ( θ 0 ) .
For l ∈ [ L − 2] , we have that ∂f L
[Equation]
x )
[Equation]
where in the second equality we use again that W (2) L − 1 ( θ 0 )
=
W (1) L − 1 ( θ 0 ) and that W (2) L ( θ 0 ) =
− W (1) L ( θ 0 ) .
Let us define f ( k ) L − 1 ( θ, x ) :
= ϕ
(( W ( k ) L − 1 ( θ ))
⊤ f L − 2 ( θ, x )) for k ∈ { 1 , 2 } .
Then, for the ( L − 1) -th layer, by isolating the computation over W (1) L − 1 , we have that
[Equation]
where we use that f (2) L − 1 ( θ, x ) does not depend on the parameters W (1) L − 1 .
Proceeding in the same way and using that W (2) L − 1 ( θ 0 )
=
W (1) L − 1 ( θ 0 ) and W (2) L ( θ 0 ) =
− W (1) L ( θ 0 ) , we also obtain that
[Equation]
Finally, by observing that f L ( θ, x ) =
( W (1) L )
⊤ f (1) L − 1 ( θ, x ) + ( W (2) L ) ⊤ f (2)
L − 1 ( θ, x ) , we deduce
[Equation]
for k ∈ { 1 , 2 } .
[Equation]
Hence, the NTK at initialization K ( θ 0 ) can be expressed as
[Equation]
Note that, by construction, J (1) ( θ 0 ) has the same distribution of J L − 2 , whose rows are given by (10).
Therefore, by combining the results from Theorem 3.2 and Theorem 3.4, we conclude that
[Equation]
with probability at least 1 − C Ne − c log 2 n L − 1 − Ce − c log 2 N over ( x i )
Ni
=1 and θ 0 , where c 1 , C are numerical constants.
This proves (234), and concludes the proof of the lemma.
Lemma H.3 (Operator norm of weights in D ) .
Let θ 0 be defined in (18) , let D =
B
( θ 0 , R ) and assume that R
=
o
Then, for any l ∈ [ L − 1] ,
[Equation]
(242) with probability at least 1 − 2 exp( − cn L − 1 ) over W l ( θ 0 ) .
Proof.
By Weyl’s theorem, we have that, for all l ∈
[ L − 2] ,
[Equation]
where in the fourth line we use that R
=
o
(1) , and the result of the last line
holds with probability at least 1 − 2 exp( − cn L − 1 ) over W l ( θ 0 ) by Theorem 4.4.5 of [65].
By following the same argument, we have that, with probability at least 1 − 2 exp( − cn L − 1 ) over W L − 1 ( θ 0 ) ,
[Equation]
which readily implies that sup θ ∈D
∥
W L − 1 ( θ ) ∥ op
= O (1) and concludes the proof.
Lemma H.4 (Distance of features in D from initialization) .
Let θ 0 be defined in (18) , x ∼ P X ,
D = B
( θ 0 , R ) and assume that R
=
o
Then, for any 0 ≤ l ≤ L − 1 , we have
[Equation]
with probability at least 1 − C exp( − cn L − 1 ) over ( W k
( θ 0 ))
lk =1 and x , where c, C are numerical constants.
Proof.
We prove the claim by induction over l .
For the base case, we have f 0 ( θ, x ) = f
0 ( θ 0 , x ) , hence (245) holds with probability 1.
For the induction case, let l > 0 .
Then,
[Equation]
By Lemma H.
[Equation]
we have that
[Equation]
with probability at least 1 − 2 exp( − cn L − 1 ) over W l ( θ 0 ) .
By inductive hypothesis, we have
[Equation]
with probability at least 1 − C exp( − cn L − 1 ) over ( W k
( θ 0 ))
l − 1 k
=1 and x .
Clearly, we also have that
[Equation]
Furthermore, an application of Lemma C.1 gives that
[Equation]
with probability at least 1 − C exp( − cn L − 1 ) over ( W k
( θ 0 ))
l − 1 k
=1 and x .
By combining (246), (247), (248), (249) and (250), we obtain that
[Equation]
with probability at least 1 − C exp( − cn L − 1 ) over ( W k
( θ 0 ))
lk =1 and x , which completes the proof.
Lemma H.5 ( ℓ 2 norm and ℓ 2 distance of features in D ) .
Let θ 0 be defined in (18) , x ∼ P X ,
D = B
( θ 0 , R ) and assume that R
=
o
Then, for any 0 ≤ l ≤ L − 1 , we have
[Equation]
with probability at least 1 − C exp( − cn L − 1 ) over ( W k
( θ 0 ))
lk =1 and x , where c, C are numerical constants.
Furthermore,
[Equation]
with probability at least 1 − C exp( − cn L − 1 ) over ( W k
( θ 0 ))
lk =1 and x .
Proof.
The first statement follows from the chain of inequalities below:
[Equation]
where the second inequality holds with probability at least 1 − C exp( − cn L − 1 ) over ( W k ( θ 0 ))
lk = and x by combining Lemma C.1 and Lemma H.4.
We prove the second statement by induction over l .
For the base case, we have f 0 ( θ 1 , x ) = f
0 ( θ 2 , x ) , hence (253) holds with probability 1.
For the induction case, let l > 0 .
Then, sup
[Equation]
By Lemma H.
[Equation]
we have that
[Equation] [Equation]
with probability at least 1 − 2 exp( − cn L − 1 ) over W l ( θ 0 ) .
By inductive hypothesis, we have
[Equation]
with probability at least 1 − C exp( − cn L − 1 ) over ( W k
( θ 0 ))
l − 1 k
=1 and x .
Clearly, we also have that
[Equation]
Furthermore, by using (252), we have that
[Equation]
with probability at least 1 − C exp( − cn L − 1 ) over ( W k
( θ 0 ))
l − 1 k
=1 and x .
By combining (255), (256), (257), (258) and (259), we obtain that
[Equation]
with probability at least 1 − C exp( − cn L − 1 ) over ( W k
( θ 0 ))
lk =1 and x , which completes the proof.
Lemma H.6 ( ℓ 2 norm of backpropagation in D ) .
Let θ 0 be defined in (18) , x ∼ P X , and D =
B ( θ 0 , R ) .
Assume that R = o (1) and that γ > 1 .
Then, for any l ∈ [ L ] , we have
[Equation]
with probability at least 1 − C exp( − cn L − 1 ) over ( W k
( θ 0 ))
Lk
= l +1 , where c, C are numerical constants.
Proof.
We prove the claim by induction on l ∈ { L, L − 1 , . . .
For the base case, we have that ∥ b L ( θ, x ) ∥ 2 = 1 , hence (261) clearly holds.
For the induction case, pick l ∈
[ L − 1] .
Then,
[Equation]
Here, the fourth line follows from Assumption 2.3, which gives sup θ ∈D ∥
Σ k ( θ, x ) ∥ op ≤ M ; the fifth line holds with probability 1 − C exp( − cn L − 1 ) over ( W k ( θ 0 ))
L − 1 k = l +1 by Lemma H.3; the sixth line holds with probability at least 1 − exp( − cn L − 1 ) over W L ( θ 0 ) by Theorem 3.1.1 in [65]; and the last line follows from R
=
o
Taking the intersection of these events gives the desired result.
Lemma H.7 ( ℓ 2 distance of backpropagation in D ) .
Let θ 0 be defined in (18) , x ∼ P X , and
D = B
( θ 0 , R ) .
Assume that R = o (1) and that γ > 1 .
Then, for any l ∈ [ L ] , we have
[Equation] [Equation]
with probability at least 1 − C exp( − cn L − 1 ) over ( W k
( θ 0 ))
Lk
= l +1 and x , where c, C are numerical constants.
Proof.
We prove the claim by induction on l ∈ { L, L − 1 , . . .
For the base case, b L ( θ, x ) does not depend on θ , hence (263) clearly holds.
For the induction case, pick l ∈
[ L − 1] .
Then,
[Equation]
Furthermore, we have that the following results hold.
(i)
By Assumption 2.3 and Lemma H.3,
[Equation]
with probability 1 − 2 exp( − cn L − 1 ) over W l +1 ( θ 0 ) ; (ii) By inductive hypothesis,
[Equation]
with probability at least 1 − C exp( − cn L − 1 ) over ( W k
( θ 0 ))
Lk =
l +2 and x ; (iii) By the same argument of the second statement in Lemma H.5 and again Lemma H.3,
[Equation]
≤ C R √ d
[Equation]
with probability at least 1 − C exp( − cn L − 1 ) over ( W k
( θ 0 ))
l +1 k =1 and x ; (iv) By Lemma H.6,
[Equation]
, with probability at least 1 − C exp( − cn L − 1 ) over ( W k
( θ 0 ))
Lk
= l +1 ; (v) By Assumption 2.3,
[Equation]
By combining (i)-(v) with (264), we conclude that
[Equation]
with probability at least 1 − C exp( − cn L − 1 ) over x and ( W k ( θ 0 ))
Lk =1 , which concludes the proof.
Lemma H.8 (Difference of Jacobians in D ) .
Let θ 0 be defined in (18) , x ∼ P X
[Equation]
and D = B ( θ 0 , R ) .
Assume that R = o (1) and that γ > 1 .
Then, we have
[Equation]
(266) with probability at least 1 − CN exp( − cn L − 1 ) over ( x i ) Ni
=1 and θ 0 , where c, C are numerical constants.
Proof.
Pick i ∈
[ N ] .
Then, we have
[Equation]
Since x i ∼ P X , we can merge together the results from Lemmas H.4, H.5, H.6 and H.7 and obtain sup
[Equation] [Equation]
with probability at least 1 − C exp( − cn L − 1 ) over x i and θ 0 .
Therefore, we have
[Equation]
with probability 1 − CN exp( − cn L − 1 ) over ( x i ) Ni =1 and θ 0 .
Lemma H.9 (NTK spectrum in D ) .
Let θ 0 be defined in (18) , x ∼ P X , and D = B
( θ 0 , R ) .
Assume that R = o (1) and that γ > 1 .
Then, we have
[Equation]
(270) with probability at least 1 − C N exp( − cn
L − 1 ) over θ 0 and ( x i )
Ni
=1 , where c, C are numerical constants.
Furthermore,
[Equation] [Equation]
with probability at least 1 − C Ne − c log 2 n L − 1 − Ce − c log 2 N over θ 0 and ( x i )
Ni
=1 , where c 1 , C 1 are also numerical constants.
Proof.
We have
[Equation]
By Lemma H.5, we have that sup θ ∈D ∥
f l ( θ, x i ) ∥ 2 2 ≤ C d with probability at least 1 − C exp( − cn L − 1 ) over ( W k ( θ 0 ))
lk =1 and x i , for any 0 ≤ l ≤ L − 1 and i ∈
[ N ] .
By Lemma H.6, we have that sup θ ∈D ∥
b l ( θ, x i ) ∥ 22 ≤ Cγn L − 1 with probability at least 1 − C exp( − cn L − 1 ) over ( W k
( θ 0 ))
Lk
= l +1 , for any l ∈ [ L ] and i ∈ [ N ] .
Therefore, we obtain
[Equation]
with probability at least 1 − C N exp( − cn
L − 1 ) over θ 0 and ( x i )
Ni =1 , which gives the first statement of the lemma.
By using Weyl’s inequality, we get
[Equation]
[Equation]
where the last inequality follows from Lemma H.2 and Lemma H.8, and it holds with probability 1 − C Ne − c log 2 n L − 1 − Ce − c log 2 N over ( x i )
Ni
=1 and θ 0 .
This gives the second statement of the lemma and concludes the proof.
Armed with Proposition H.1 and the intermediate estimates of Lemmas H.2-H.9, we are finally ready to prove Theorem 4.2.
Proof of Theorem 4.2.
We show that there exist two absolute constants ˜ c and ˜ C such that
[Equation]
and
[Equation] [Equation]
satisfy the two assumptions in Proposition H.1 with initialization ˜ θ 0 :
= θ 0 , where θ 0 is defined in (18).
This holds with probability at least 1 − C Ne − c log 2 n L − 1 − Ce − c log 2 N over ( x i ) Ni
=1 and θ 0 .
Recall from Proposition H.1 that R is defined as 4 ∥ F L ( θ 0 ) − Y ∥ 2 /α , since we have set ˜ θ
= θ 0 .
For the moment, we assume that
[Equation]
and we will verify that this is the case later.
Note that γ = d 3 N 2 > 1 and, hence, (277) and
Assumption 2.5 imply that R
=
o
Thus, we can apply Lemma H.9 and obtain
[Equation]
with probability at least 1 − C Ne − c log 2 n L − 1 − Ce − c log 2 N over ( x i )
Ni
=1 and θ 0 , where the last inequality uses (277).
This shows that the lower bound in (230) holds.
Now, by using (278), we verify that (277) holds.
Recall that, by assumption of the theorem,
∥ Y ∥
2 = Θ( √ N ) .
Furthermore, by Lemma H.2, F L ( θ 0 ) is a vector of all zeros.
Then,
[Equation]
By Lemma H.9, we have that
[Equation]
with probability at least 1 − C N exp( − cn
L − 1 ) over θ 0 .
Thus, by our choice (276) of β , we obtain that the upper bound in (230) holds.
Next, we verify the second assumption of Proposition H.1.
To do so, let us write
[Equation]
where we have used Assumption 2.5.
Thus,
[Equation]
with probability at least 1 − C N exp( − cn
L − 1 ) over ( x i ) Ni
=1 and θ 0 .
Here, the first passage follows from Lemma H.8, in the second passage we use (277), and in the last one we use (281).
This completes the proof of (231) and also of the theorem, since the desired claim follows from an application of Proposition H.1.