H OW M UCH O VER - PARAMETERIZATION I S S UFFICIENT TO L EARN D EEP R E LU N ETWORKS ?
A BSTRACT
A recent line of research on deep learning focuses on the extremely overparameterized setting, and shows that when the network width is larger than a high degree polynomial of the training sample size n and the inverse of the target error (cid:15)
´ 1 , deep neural networks learned by (stochastic) gradient descent enjoy nice optimization and generalization guarantees.
Very recently, it is shown that under certain margin assumptions on the training data, a polylogarithmic width condition sufﬁces for two-layer ReLU networks to converge and generalize (Ji and Telgarsky, 2020).
However, whether deep neural networks can be learned with such a mild over-parameterization is still an open question.
In this work, we answer this question afﬁrmatively and establish sharper learning guarantees for deep ReLU networks trained by (stochastic) gradient descent.
In speciﬁc, under certain assumptions made in previous work, our optimization and generalization guarantees hold with network width polylogarithmic in n and (cid:15) ´
Our results push the study of over-parameterized deep neural networks towards more practical settings.
I NTRODUCTION
Deep neural networks have become one of the most important and prevalent machine learning models due to their remarkable power in many real-world applications.
However, the success of deep learning has not been well-explained in theory.
It remains mysterious why standard optimization algorithms tend to ﬁnd a globally optimal solution, despite the highly non-convex landscape of the training loss function.
Moreover, despite the extremely large amount of parameters, deep neural networks rarely over-ﬁt and can often generalize well to unseen data and achieve good test accuracy.
Understanding these mysterious phenomena on the optimization and generalization of deep neural networks is one of the most fundamental problems in deep learning theory.
Recent breakthroughs have shed light on the optimization and generalization of deep neural networks (DNNs) under the over-parameterized setting, where the hidden layer width is extremely large (much larger than the number of training examples).
It has been shown that with the standard random initialization, the training of over-parameterized deep neural networks can be characterized by a kernel function called neural tangent kernel (NTK)
(Jacot et al., 2018; Arora et al., 2019b).
In the neural tangent kernel regime (or lazy training regime (Chizat et al., 2019)), the neural network function behaves similarly as its ﬁrst-order Taylor expansion at initialization (Jacot et al., 2018;
Lee et al., 2019; Arora et al., 2019b; Cao and Gu, 2019), which enables feasible optimization and generalization analysis.
In terms of optimization, a line of work (Du et al., 2019b; Allen-Zhu et al., 2019b; Zou et al., 2019; Zou and Gu, 2019) proved that for sufﬁciently wide neural networks, (stochastic) gradient descent (GD/SGD) can successfully ﬁnd a global optimum of the training loss function.
For generalization, Allen-Zhu et al.
(2019a)
; Arora et al. (2019a); Cao and Gu (2019) established generalization bounds of neural networks trained with (stochastic) gradient descent, and showed that the neural networks can learn target functions in certain reproducing kernel Hilbert space (RKHS) or the corresponding random feature function class.
Although existing results in the neural tangent kernel regime have provided important insights into the learning of deep neural networks, they require the neural network to be extremely wide.
The typical requirement on the network width is a high degree polynomial of the training sample size n and the inverse of the target error (cid:15) ´ 1 .
As there still remains a huge gap between such network width requirement and the practice, many attempts have been made to improve the overparameterization condition under various conditions on the training data and model initialization (Oymak and Soltanolkotabi, 2019; Zou and Gu, 2019; Kawaguchi and Huang, 2019; Bai and Lee, 2019).
For two-layer ReLU networks, a recent work (Ji and Telgarsky, 2020) showed that when the training data are well separated, polylogarithmic width is sufﬁcient to guarantee good optimization and generalization performances.
However, their results cannot be extended to deep ReLU networks since their proof technique largely relies on the fact that the network model is 1 -homogeneous, which cannot be satisﬁed by DNNs.
Therefore, whether deep neural networks can be learned with such a mild over-parameterization is still an open problem.
In this paper, we resolve this open problem by showing that polylogarithmic network width is sufﬁcient to learn DNNs.
In particular, unlike the existing works that require the DNNs to behave very close to a linear model (up to some small approximation error), we show that a constant linear approximation error is sufﬁcient to establish nice optimization and generalization guarantees for
DNNs.
Thanks to the relaxed requirement on the linear approximation error, a milder condition on the network width and tighter bounds on the convergence rate and generalization error can be proved.
We summarize our contributions as follows:
• We establish the global convergence guarantee of GD for training deep ReLU networks based on the so-called NTRF function class (Cao and Gu, 2019), a set of linear functions over random features.
Speciﬁcally, we prove that GD can learn deep ReLU networks with width m “ poly p R q to compete with the best function in NTRF function class, where R is the radius of the NTRF function class.
• We also establish the generalization guarantees for both GD and SGD in the same setting.
Speciﬁcally, we prove a diminishing statistical error for a wide range of network width m P p r Ω p 1 q , 8q , while most of the previous generalization bounds in the NTK regime only works in the setting where the network width m is much greater than the sample size n .
Moreover, we establish r O p (cid:15) ´
2 q r O p (cid:15) ´ 1 q sample complexities for GD and SGD respectively, which are tighter than existing bounds for learning deep ReLU networks (Cao and Gu, 2019), and match the best results when reduced to the two-layer cases (Arora et al., 2019b; Ji and Telgarsky, 2020).
• We further generalize our theoretical analysis to the scenarios with different data separability assumptions in the literature.
We show if a large fraction of the training data are well separated, the best function in the NTRF function class with radius R “ r O p 1 q can learn the training data with error up to (cid:15) .
This together with our optimization and generalization guarantees immediately suggests that deep ReLU networks can be learned with network width m “ r Ω p 1 q , which has a logarithmic dependence on the target error (cid:15) and sample size n .
Compared with existing results (Cao and Gu, 2020; Ji and Telgarsky, 2020) which require all training data points to be separated in the NTK regime, our result is stronger since it allows the NTRF function class to misclassify a small proportion of the training data.
For the ease of comparison we summarize our results along with the most related previous results in
Table 1, in terms of data assumption, the over-parameterization condition and sample complexity.
It can be seen that under data separation assumption (See Sections 4.1, 4.2), our result improves existing results for learning deep neural networks by only requiring a polylog p n, (cid:15) ´
1 q network width.
Notation.
For two scalars a and b , we denote a ^ b “ min t a, b u .
For a vector x P R d we use } x }to denote its Euclidean norm.
For a matrix X , we use } X } 2 and } X } F to denote its spectral norm and Frobenius norm respectively, and denote by X ij the entry of X at the i -th row and j -th column.
Given two matrices X and Y with the same dimension, we denote x X , Y y
“ ř i,j X ij Y ij .
Given a collection of matrices W “ t W 1 , ¨ ¨ ¨ , W L u P b Ll “ 1 R m l ˆ m 1 l and a function f p W q over b Ll “ 1 R m l ˆ m 1 l , we deﬁne by ∇ W l f p W q the partial gradient of f p W q with respect to W l and denote ∇ W f p W q “ t ∇ W l f p W qu Ll “ 1 .
We also denote B p W , τ q “ (cid:32)
W 1 : max l Pr L s } W 1 l ´ W l } F ď τ ( for τ ě 0 .
For two collection of matrices A “ t A 1 , ¨ ¨ ¨ , A n u , B “ t B 1 , ¨ ¨ ¨ , B n u , we denote x A , B y
“ ř ni “ 1 x A i , B i y and } A } 2 F “ ř ni “ 1 } A i } 2 F .
[Table]
Input: Number of iterations T , step size η , training set S “ tp x i , y i q ni “ 1 u , initialization W p 0 q for t “ 1 , 2 , . . .
T
Update W p t q “ W p t ´ 1 q ´ η ¨ ∇ W L S p W p t ´ 1 q q .
end for Output: W p 0 q , . . . , W p T q . Given two sequences t x n u and t y n u , we denote x n “ O p y n q if | x n | ď C 1 | y n | for some absolute positive constant C 1 , x n “ Ω p y n q if | x n | ě C 2 | y n | for some absolute positive constant C 2 , and x n “ Θ p y n q if C 3 | y n | ď | x n | ď C 4 | y n | for some absolute constants C 3 , C 4 ą 0 .
We also use r O p¨q , r Ω p¨q to hide logarithmic factors in O p¨q and Ω p¨q respectively.
Additionally, we denote x n “ poly p y n q if x n “ O p y Dn q for some positive constant D , and x n “ polylog p y n q if x n “ poly p log p y n qq .
P RELIMINARIES ON LEARNING NEURAL NETWORKS
In this section, we introduce the problem setting in this paper, including deﬁnitions of the neural network and loss functions, and the training algorithms, i.e., GD and SGD with random initialization.
Neural network function.
Given an input x P R d , the output of deep fully-connected ReLU network is deﬁned as follows,
[Equation]
where W 1 P R m ˆ d , W 2 , ¨ ¨ ¨ , W L ´ 1 P R m ˆ m , W L P R 1 ˆ m , and σ p x q “ max t 0 , x u is the ReLU activation function.
Here, without loss of generality, we assume the width of each layer is equal to m .
Yet our theoretical results can be easily generalized to the setting with unequal width layers, as long as the smallest width satisﬁes our overparameterization condition.
We denote the collection of all weight matrices as W “ t W 1 , . . .
, W L u .
Loss function.
Given training dataset t x i , y i u i “ 1 ,...,n with input x i P R d and output y i
P t´ 1 , ` 1 u , we deﬁne the training loss function as
[Equation]
where L i p W q “ (cid:96) ` y i f
W p x i q ˘ “ log ` 1 ` exp p´ y i f
W p x i qq ˘ is deﬁned as the cross-entropy loss.
Algorithms.
We consider both GD and SGD with Gaussian random initialization.
These two algorithms are displayed in Algorithms 1 and 2 respectively.
Speciﬁcally, the entries in W p 0 q 1 , ¨ ¨ ¨ , W p
q L ´are generated independently from univariate Gaussian distribution N p 0 , 2 { m q
and the entries in W p 0 q L are generated independently from N p 0 , 1 { m q .
For GD, we consider using the full gradient to update the model parameters.
For SGD, we use a new training data point in each iteration.
Note that our initialization method in Algorithms 1, 2 is the same as the widely used He initialization (He et al., 2015).
Our neural network parameterization is also consistent with the parameterization used in prior work on NTK (Jacot et al., 2018; Allen-Zhu et al., 2019b; Du et al., 2019a; Arora et al., 2019b; Cao and Gu, 2019).
M AIN THEORY
In this section, we present the optimization and generalization guarantees of GD and SGD for learning deep ReLU networks.
We ﬁrst make the following assumption on the training data points.
Assumption 3.1.
All training data points satisfy } x i } 2 “ 1 , i “ 1 , . . . , n .
This assumption has been widely made in many previous works (Allen-Zhu et al., 2019b;c; Du et al., 2019b;a; Zou et al., 2019) in order to simplify the theoretical analysis.
This assumption can be relaxed to be upper bounded and lower bounded by some constant.
In the following, we give the deﬁnition of Neural Tangent Random Feature (NTRF) (Cao and Gu, 2019), which characterizes the functions learnable by over-parameterized ReLU networks.
Deﬁnition 3.2 (Neural Tangent Random Feature, (Cao and Gu, 2019)) .
Let W p
q be the initialization weights, and F W p 0 q , W p x q “ f W p 0 q p x q ` x ∇ f W p
q p x q , W ´ W p
q y be a function with respect to the input x .
Then the NTRF function class is deﬁned as follows
[Equation]
The function class F W p 0 q , W p x q consists of linear models over random features deﬁned based on the network gradients at the initialization.
Therefore it captures the key “almost linear” property of wide neural networks in the NTK regime (Lee et al., 2019; Cao and Gu, 2019).
In this paper, we use the NTRF function class as a reference class to measure the difﬁculty of a learning problem.
In what follows, we deliver our main theoretical results regarding the optimization and generalization guarantees of learning deep ReLU networks.
We study both GD and SGD with random initialization (presented in Algorithms 1 and 2).
3. G RADIENT DESCENT
The following theorem establishes the optimization guarantee of GD for training deep ReLU networks for binary classiﬁcation.
Theorem 3.3.
For δ, R ą 0 , let (cid:15) NTRF “ inf F P F p W p 0 q ,R q n ´ř ni “ 1 (cid:96) r y
i F p x
i qs be the minimum training loss achievable by functions in F p W p 0 q , R q .
Then there exists
[Equation]
such that if m ě m ˚ p δ, R, L q , with probability at least 1 ´ δ over the initialization, GD with step size η “ Θ p L ´ 1 m ´ 1 q can train a neural network to achieve at most 3 (cid:15) NTRF training loss within
T “ O
[Equation] (cid:15) [Equation]
Theorem 3.3 shows that the deep ReLU network trained by GD can compete with the best function in the NTRF function class F p W p 0 q , R q if the network width has a polynomial dependency in R and L and a logarithmic dependency in n and 1 { δ .
Moreover, if the NTRF function class with R “ r O p 1 q can learn the training data well (i.e., (cid:15) NTRF is less than a small target error (cid:15) ), a polylogarithmic (in terms of n and (cid:15) ´ 1 ) network width sufﬁces to guarantee the global convergence of GD, which directly improves over-paramterization condition in the most related work (Cao and Gu, 2019).
Besides, we remark here that this assumption on the NTRF function class can be easily satisﬁed when the training data admits certain separability conditions, which we discuss in detail in Section 4.
Compared with the results in (Ji and Telgarsky, 2020) which give similar network width requirements for two-layer networks, our result works for deep networks.
Moreover, while Ji and Telgarsky (2020) essentially required all training data to be separable by a function in the NTRF function class with a constant margin, our result does not require such data separation assumptions, and allows the NTRF function class to misclassify a small proportion of the training data points ∗ .
We now characterize the generalization performance of neural networks trained by GD.
We denote
L 0 ´ 1 D p W q “ E p x ,y q„ D r 1 t f W p x q ¨ y ă 0 us as the expected 0-1 loss (i.e., expected error) of f W p x q .
Theorem 3.4.
Under the same assumptions as Theorem 3.3, with probability at least 1 ´ δ , the iterate
W p t q of Algorithm 1 satisﬁes that
[Equation]
for all t “ 0 , . . .
, T .
Theorem 3.4 shows that the test error of the trained neural network can be bounded by its training error plus statistical error terms.
Note that the statistical error terms is in the form of a minimum between two terms 4 L L 2 R a m { n and L 3 { 2 R {?
n ` L 11 { 3 R 4 { 3 { m 1 { 6 .
Depending on the network width m , one of these two terms will be the dominating term and diminishes for large n : (1) if m “ o p n q , the statistical error will be 4 L L 2 R a m { n , and diminishes as n increases; and (2) if m “ Ω p n q , the statistical error is L 3 { 2 R {?
n ` L 11 { 3 R 4 { 3 { m 1 { 6 , and again goes to zero as n increases.
Moreover, in this paper we have a speciﬁc focus on the setting m “ r O p 1 q , under which Theorem 3.4 gives a statistical error of order
r O p n ´ 1 { 2 q .
This distinguishes our result from previous generalization bounds for deep networks (Cao and Gu, 2020; 2019), which cannot be applied to the setting m “ r O p 1 q .
We note that for two-layer ReLU networks (i.e., L “ 2 ) Ji and Telgarsky (2020) proves a tighter r
O p 1 { n 1 { 2 q generalization error bound regardless of the neural networks width m , while our result (Theorem 3.4), in the two-layer case, can only give r O p 1 { n 1 { 2 q generalization error bound when m “ r O p 1 q or m “ r Ω p n 3 q .
However, different from our proof technique that basically uses the (approximated) linearity of the neural network function, their proof technique largely relies on the 1 -homogeneous property of the neural network, which restricted their theory in two-layer cases.
An interesting research direction is to explore whether a r O p 1 { n 1 { 2 q generalization error bound can be also established for deep networks (regardless of the network width), which we will leave it as a future work.
3. S TOCHASTIC GRADIENT DESCENT
Here we study the performance of SGD for training deep ReLU networks.
The following theorem establishes a generalization error bound for the output of SGD.
Theorem 3.5.
For δ, R ą 0 , let (cid:15) NTRF “ inf F P F p W p 0 q ,R q n ´ř ni “ 1 (cid:96) r y
i F p x
i qs be the minimum training loss achievable by functions in F p W p 0 q , R q .
Then there exists
[Equation]
, such that if m ě m ˚ p δ, R, L q , with probability at least 1 ´ δ , SGD with step size η “ Θ ` m ´ 1 ¨ p LR 2 n ´ 1 (cid:15) ´ 1 NTRF ^ L ´ 1 q ˘ achieves
[Equation]
24 (cid:15) NTRF , where the expectation is taken over the uniform draw of x W from t W p 0 q , . . .
, W p n ´ 1 q u .
For any (cid:15) ą 0 , Theorem 3.5 gives a r O p (cid:15) ´ 1 q sample complexity for deep ReLU networks trained with
SGD to achieve O p (cid:15) NTRF ` (cid:15) q test error.
Our result extends the result for two-layer networks proved in (Ji and Telgarsky, 2020) to multi-layer networks.
Theorem 3.5 also provides sharper results compared with Allen-Zhu et al.
(2019a)
; Cao and Gu (2019) in two aspects: (1) the sample complexity is improved from n “ r O p (cid:15) ´
2 q to n “ r O p (cid:15) ´
1 q ; and (2) the overparamterization condition is improved from m ě poly p (cid:15) ´
1 q to m “ r Ω p 1 q .
D ISCUSSION ON THE NTRF C LASS
Our theoretical results in Section 3 rely on the radius (i.e., R ) of the NTRF function class F p W p 0 q , R q and the minimum training loss achievable by functions in F p W p 0 q , R q , i.e., (cid:15) NTRF .
Note that a larger
R naturally implies a smaller (cid:15) NTRF , but also leads to worse conditions on m .
In this section, for any (arbitrarily small) target error rate (cid:15) ą 0 , we discuss various data assumptions studied in the literature under which our results can lead to O p (cid:15) q training/test errors, and specify the network width requirement.
4. D ATA S EPARABILITY BY N EURAL T ANGENT R ANDOM F EATURE
In this subsection, we consider the setting where a large fraction of the training data can be linearly separated by the neural tangent random features.
The assumption is stated as follows.
Assumption 4.1.
There exists a collection of matrices
U ˚ “ t U ˚ 1 , ¨ ¨ ¨ , U ˚ L u satisfying ř
Ll “ 1 } U ˚ l } 2 F “ 1 , such that for at least p 1 ´ ρ q fraction of training data we have
[Equation]
where γ is an absolute positive constant † and ρ P r 0 , 1 q .
The following corollary provides an upper bound of (cid:15) NTRF under Assumption 4.1 for some R .
Proposition 4.2.
Under Assumption 4.1, for any (cid:15), δ ą 0 , if R ě C
[Equation]
p
[Equation]
[Equation]
q ‰ {
[Equation]
for some absolute constant C , then with probability at least 1 ´ δ ,
[Equation]
NTRF [Equation] q . Proposition 4.2 covers the setting where the NTRF function class is allowed to misclassify training data, while most of existing work typically assumes that all training data can be perfectly separated with constant margin (i.e., ρ “ 0 ) (Ji and Telgarsky, 2020; Shamir, 2020).
Our results show that for sufﬁciently small misclassiﬁcation ratio ρ “ O p (cid:15) q , we have (cid:15) NTRF “ r O p (cid:15) q by choosing the radius parameter R logarithimic in n , δ ´ 1 , and (cid:15) ´ 1 .
Substituting this result into Theorems 3.3, 3.4 and 3.5, it can be shown that a neural network with width m “ poly p L, log p n { δ q , log p 1 { (cid:15) qq ˘ sufﬁces to guarantee good optimization and generalization performances for both GD and SGD.
Consequently, we can obtain that the bounds on the test error for GD and SGD are r O p n ´ 1 { 2 q and r
O p n ´ 1 q respectively.
4. D ATA S EPARABILITY BY S HALLOW N EURAL T ANGENT M ODEL
In this subsection, we study the data separation assumption made in Ji and Telgarsky (2020) and show that our results cover this particular setting.
We ﬁrst restate the assumption as follows.
Assumption 4.3.
There exists u p¨q : R d Ñ R d and γ ě 0 such that } u p z q} 2 ď 1 for all z P R d , and
[Equation]
for all i P r n s , where µ N p¨q denotes the standard normal distribution.
Assumption 4.3 is related to the linear separability of the gradients of the ﬁrst layer parameters at random initialization, where the randomness is replaced with an integral by taking the inﬁnite width limit.
Note that similar assumptions have also been studied in (Cao and Gu, 2020; Nitanda and
Suzuki, 2019; Frei et al., 2019).
The assumption made in (Cao and Gu, 2020; Frei et al., 2019) uses gradients with respect to the second layer weights instead of the ﬁrst layer ones.
In the following, we mainly focus on Assumption 4.3, while our result can also be generalized to cover the setting in (Cao and Gu, 2020; Frei et al., 2019).
In order to make a fair comparison, we reduce our results for multilayer networks to the two-layer setting.
In this case, the neural network function takes form
[Equation]
Then we provide the following proposition, which states that Assumption 4.3 implies a certain choice of R “ r O p 1 q such the the minimum training loss achieved by the function in the NTRF function class F p W p
q , R q satisﬁes (cid:15) NTRF
“ O p (cid:15) q , where (cid:15) is the target error.
Proposition 4.4.
Suppose the training data satisﬁes Assumption 4.3.
For any (cid:15), δ ą 0 , let R “
[Equation] ‰ { γ for some large enough absolute constant C .
If the neural network width satisﬁes m [Equation] , then with probability at least 1 ´ δ , there exist F W p 0 q , W p x i q P
F p W p 0 q , R q such that (cid:96) ` y i ¨ F W p 0 q , W p x i
[Equation]
Proposition 4.4 shows that under Assumption 4.3, there exists F W p 0 q , W p¨q P F p W p 0 q , R q with
R “ r O p 1 { γ q such that the cross-entropy loss of F W p 0 q , W p¨q at each training data point is bounded by (cid:15) .
This implies that (cid:15) NTRF ď (cid:15) .
Moreover, by applying Theorem 3.3 with L “ 2 , the condition on the neural network width becomes m “ r Ω p 1 { γ 8 q ‡ , which matches the results proved in Ji and Telgarsky (2020).
Moreover, plugging these results on m and (cid:15) NTRF into Theorems 3.4 and 3.5, we can conclude that the bounds on the test error for GD and SGD are r O p n ´ 1 { 2 q and r
O p n ´ 1 q respectively.
4. C LASS - DEPENDENT D ATA N ONDEGENERATION
In previous subsections, we have shown that under certain data separation conditions (cid:15) NTRF can be sufﬁciently small while the corresponding NTRF function class has R of order r O p 1 q .
Thus neural networks with polylogarithmic width enjoy nice optimization and generalization guarantees.
In this part, we consider the following much milder data separability assumption made in Zou et al.
Assumption 4.5.
For all i ‰ i 1 if y i ‰
y
i 1 , then } x i ´ x j } 2 ě φ for some absolute constant φ .
In contrast to the conventional data nondegeneration assumption (i.e., no duplicate data points) made in Allen-Zhu et al. (2019b); Du et al.
(2019b;a); Zou and Gu (2019) § , Assumption 4.5 only requires that the data points from different classes are nondegenerate, thus we call it class-dependent data nondegeneration.
We have the following proposition which shows that Assumption 4.5 also implies the existence of a good function that achieves (cid:15) training error, in the NTRF function class with a certain choice of R .
Proposition 4.6.
Under Assumption 4.5, if
[Equation]
we have (cid:15) NTRF ď (cid:15) with probability at least 1 ´ δ .
Proposition 4.6 suggests that under Assumption 4.5, in order to guarantee (cid:15) NTRF ď (cid:15) , the size of
NTRF function class needs to be Ω p n 3 { 2 q .
Plugging this into Theorems 3.4 and 3.5 leads to vacuous bounds on the test error.
This makes sense since Assumption 4.5 basically covers the “random label” setting, which is impossible to be learned with small generalization error.
Moreover, we would like to point out our theoretical analysis leads to a sharper over-parameterization condition than that proved in Zou et al.
(2019), i.e., m “ r Ω ` n 14 L 16 φ ´ 4 ` n 12 L 16 φ ´ 4 (cid:15) ´
1 ˘ , if the network depth satisﬁes
L ď r O p n 1 { 3 _ (cid:15) ´ 1 { 6 q .
P ROOF SKETCH OF THE MAIN THEORY
In this section, we introduce a key technical lemma in Section 5.1, based on which we provide a proof sketch of Theorems 3.3.
The full proof of all our results can be found in the appendix.
5. A KEY TECHNICAL LEMMA
Here we introduce a key technical lemma used in the proof of Theorem 3.3.
Our proof is based on the key observation that near initialization, the neural network function can be approximated by its ﬁrst-order Taylor expansion.
In the following, we ﬁrst give the deﬁnition of the linear approximation error in a τ -neighborhood around initialization.
[Equation]
If all the iterates of GD stay inside a neighborhood around initialization with small linear approximation error, then we may expect that the training of neural networks should be similar to the training of the corresponding linear model, where standard optimization techniques can be applied.
Motivated by this, we also give the following deﬁnition on the gradient upper bound of neural networks around initialization, which is related to the Lipschitz constant of the optimization objective function.
[Equation]
By deﬁnition, we can choose W ˚ P B p W p 0 q , Rm ´ 1 { 2 q such that n
[Equation] NTRF .
Then we have the following lemma.
Lemma 5.1.
Set η “ O p L ´ 1 M p τ q ´ 2 q .
Suppose that W ˚ P B p W p 0 q , τ q and W p t q P B p W p 0 q , τ q for all 0 ď t ď t 1 ´ 1 .
Then it holds that
[Equation]
´ } W p t 1 q ´ W ˚ } 2 F ` 2 t 1 η(cid:15) NTRF t 1 η ` 32 ´ 4 (cid:15) app p τ q ˘ .
Lemma 5.1 plays a central role in our proof.
In speciﬁc, if W p t q P B p W p 0 q , τ q for all t ď t 1 , then
Lemma 5.1 implies that the average training loss is in the same order of (cid:15) NTRF as long as the linear approximation error (cid:15) app p τ q is bounded by a positive constant.
This is in contrast to the proof in Cao and Gu (2019), where (cid:15) app p τ q appears as an additive term in the upper bound of the training loss, thus requiring (cid:15) app p τ q “ O p (cid:15) NTRF q to achieve the same error bound as in Lemma 5.1.
Since we can show that (cid:15) app “ r O p m ´ 1 { 6 q (See Section A.1), this suggests that m “ r Ω p 1 q is sufﬁcient to make the average training loss in the same order of (cid:15) NTRF .
Compared with the recent results for two-layer networks by Ji and Telgarsky (2020), Lemma 5. is proved with different techniques.
In speciﬁc, the proof by Ji and Telgarsky (2020) relies on the 1-homogeneous property of the ReLU activation function, which limits their analysis to two-layer networks with ﬁxed second layer weights.
In comparison, our proof does not rely on homogeneity, and is purely based on the linear approximation property of neural networks and some speciﬁc properties of the loss function.
Therefore, our proof technique can handle deep networks, and is potentially applicable to non-ReLU activation functions and other network architectures (e.g, Convolutional neural networks and Residual networks).
5. P ROOF SKETCH OF T HEOREM 3.
Here we provide a proof sketch of Theorem 3.3.
The proof consists of two steps: (i) showing that all T iterates stay close to initialization, and (ii) bounding the empirical loss achieved by gradient descent.
Both of these steps are proved based on Lemma 5.1.
Proof sketch of Theorem 3.3.
Recall that we choose W ˚
P
B p W p 0 q , Rm ´ 1 { 2 q such that n
[Equation]
˘ “ [Equation] NTRF .
We set τ “ r O p L 1 { 2 m ´ 1 { 2 R q , which is chosen slightly larger than m ´ 1 { 2 R since Lemma 5.1 requires the region B p W p 0 q , τ q to include both W ˚ and t W p t q u t “ 0 ,...,t 1 .
Then by Lemmas 4.1 and B.3 in Cao and Gu (2019) we know that (cid:15) app p τ q “ r
O p τ 4 { 3 m 1 { 2 L 3 q “ r O p R 4 { 3 L 11 { 3 m ´ 1 { 6 q .
Therefore, we can set m “ r Ω p R 8 L 22 q to ensure that (cid:15) app p τ q ď 1 { 8 .
Then we proceed to show that all iterates stay inside the region B p W p 0 q , τ q .
Since the L.H.S. of
Lemma 5.1 is strictly positive and (cid:15) app p τ q ď 1 { 8 , we have for all t ď T ,
[Equation]
which gives an upper bound of } W p t q ´ W ˚ } F .
Then by the choice of η , T , triangle inequality, and a simple induction argument, we see that } W p t q ´ W p 0 q } F ď m ´ 1 { 2 R ` ? 2 Tη(cid:15) NTRF
“ r
O p L 1 { 2 m ´ 1 { 2 R q , which veriﬁes that W p t q P B p W p 0 q , τ q for t “ 0 , . .
T ´ 1 .
The second step is to show that GD can ﬁnd a neural network with at most 3 (cid:15) NTRF training loss within
T iterations.
To show this, by the bound given in Lemma 5.1 with (cid:15) app ď 1 { 8 , we drop the terms } W p t q ´ W ˚ } 2 F and rearrange the inequality to obtain
[Equation]
We see that T is large enough to ensure that the ﬁrst term in the bound above is smaller than (cid:15) NTRF .
This implies that the best iterate among W p 0 q , . . .
, W p T ´ 1 q achieves an empirical loss at most 3 (cid:15) NTRF .
C ONCLUSION
In this paper, we established the global convergence and generalization error bounds of GD and SGD for training deep ReLU networks for the binary classiﬁcation problem.
We show that a network width condition that is polylogarithmic in the sample size n and the inverse of target error (cid:15) ´ 1 is sufﬁcient to guarantee the learning of deep ReLU networks.
Our results resolve an open question raised in Ji and Telgarsky (2020).
A CKNOWLEDGEMENT
We would like to thank the anonymous reviewers for their helpful comments.
ZC, YC and QG are partially supported by the National Science Foundation CAREER Award 1906169, IIS-2008981 and
Salesforce Deep Learning Research Award.
DZ is supported by the Bloomberg Data Science Ph.D. Fellowship. The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies. A P ROOF OF M AIN T HEOREMS
In this section we provide the full proof of Theorems 3.3, 3.4 and 3.5.
A. P ROOF OF T HEOREM 3.
We ﬁrst provide the following lemma which is useful in the subsequent proof.
Lemma A.1 (Lemmas 4.1 and B.3 in Cao and Gu (2019)) .
There exists an absolute constant κ such that, with probability at least 1 ´ O p nL 2 q exp r´ Ω p mτ 2 { 3 L qs , for any τ ď κL ´ 6 r log p m qs ´ 3 { 2 , it holds that
[Equation]
Proof of Theorem 3.3.
Recall that W ˚ is chosen such that
[Equation]
and W ˚ P B p W p 0 q , Rm ´ 1 { 2 q .
Note that to apply Lemma 5.1, we need the region B p W p
q , τ q to include both W ˚ and t W p t q u t “ 0 ,...,t 1 .
This motivates us to set τ “ r O p
L 1 { 2 m ´ 1 { 2 R q , which is slightly larger than m ´ 1 { 2 R .
With this choice of τ , by Lemma A.1 we have (cid:15) app p τ
[Equation]
O
[Equation]
3 m 1 { 2 L 3 q “ r
O p R 4 { 3 L 11 { 3 m ´ 1 { 6 q .
Therefore, we can set
[Equation]
to ensure that (cid:15) app p τ q ď 1 { 8 , where r Ω p¨q hides polylogarithmic dependencies on network depth L ,
NTRF function class size R , and failure probability parameter δ .
Then by Lemma 5.1, we have with probability at least 1 ´ δ , we have
[Equation]
as long as W p 0 q , . .
, W p t 1 ´ 1 q P B p W p 0 q , τ q .
In the following proof we choose η “ Θ p L ´ 1 m ´ 1 q and T “ r LR 2 m ´ 1 η ´ 1 (cid:15) ´
1 NTRF s .
We prove the theorem by two steps: 1) we show that all iterates t W p 0 q , ¨ ¨ ¨ , W p T q u will stay inside the region B p W p 0 q , τ q ; and 2) we show that GD can ﬁnd a neural network with at most 3 (cid:15) NTRF training loss within T iterations.
All iterates stay inside B p W p 0 q , τ q .
We prove this part by induction.
Speciﬁcally, given t 1 ď T , we assume the hypothesis W p t q P B p W p 0 q , τ q holds for all t ă t 1 and prove that W p t 1 q P B p W p 0 q , τ q .
First, it is clear that W p
q P B p W p 0 q , τ q .
Then by (A.2) and the fact that L S p W q ě 0 , we have
[Equation]
Note that T “ r LR 2 m ´ 1 η ´ 1 (cid:15) ´
1 NTRF s and W ˚ P B p W p 0 q , R ¨ m ´ 1 { 2 q , we have
[Equation]
where C ě 4 is an absolute constant.
Therefore, by triangle inequality, we further have the following for all l P r
[Equation]
Therefore, it is clear that } W p t 1 q l
[Equation]
[Equation]
τ based on our choice of τ previously.
This completes the proof of the ﬁrst part.
Convergence of gradient descent.
(A.2) implies
[Equation]
Dividing by ηT on the both sides, we get
[Equation]
where the second inequality is by the fact that W ˚ P B p W p 0 q , R ¨ m ´ 1 { 2 q and the last inequality is by our choices of T and η which ensure that Tη ě LR 2 m ´ 1 (cid:15) ´ 1 NTRF .
Notice that
T “ r LR
[Equation]
η
[Equation]
[Equation]
´ 1 NTRF
[Equation]
O p L 2 R 2 (cid:15) ´ 1 NTRF q .
This completes the proof of the second part, and we are able to complete the proof.
A. P ROOF OF T HEOREM 3.
Following Cao and Gu (2020), we ﬁrst introduce the deﬁnition of surrogate loss of the network, which is deﬁned by the derivative of the loss function.
Deﬁnition A.2.
We deﬁne the empirical surrogate error E S p W q and population surrogate error
E D p W q as follows:
[Equation]
The following lemma gives uniform-convergence type of results for E S p W q utilizing the fact that ´ (cid:96) 1 p¨q is bounded and Lipschitz continuous.
Lemma A.3.
For any r R, δ ą 0 , suppose that m “ r Ω p L 12 r R 2 q ¨ r log p 1 { δ qs 3 { 2 .
Then with probability at least 1 ´ δ , it holds that
[Equation]
for all W P B p W p 0 q , r R ¨ m ´ 1 { 2 q
We are now ready to prove Theorem 3.4, which combines the trajectory distance analysis in the proof of Theorem 3.3 with Lemma A.3.
Proof of Theorem 3.4.
With exactly the same proof as Theorem 3.3, by (A.3) and induction we have
W p 0 q , W p 1 q , . . .
, W p T q P B p W p 0 q , r Rm ´ 1 { 2 q with r R “ O p?
LR q .
Therefore by Lemma A.3, we have
[Equation]
O ˜c
[Equation]
p
[Equation]
q n ¸ for all t “ 0 , 1 , . .
, T .
Note that we have 1 t z ă 0 u ď ´ 2 (cid:96) 1 p z q .
Therefore,
E L 0 ´
[Equation]
ď
[Equation]
E D p
[Equation]
p
[Equation]
q
[Equation]
O
[Equation]
c
[Equation]
n ¸ for t “ 0 , 1 , . .
, T , where the last inequality is by E S p W q ď L S p W q because ´ (cid:96) 1 p z q ď (cid:96) p z q for all z P R .
This ﬁnishes the proof.
A. P ROOF OF T HEOREM 3.
In this section we provide the full proof of Theorem 3.5.
We ﬁrst give the following result, which is the counterpart of Lemma 5.1 for SGD.
Again we pick W ˚ P B p W p 0 q , Rm ´ 1 { 2 q such that the loss of the corresponding NTRF model F W p 0 q , W ˚ p x q achieves (cid:15) NTRF .
Lemma A.4.
Set η “ O p L ´ 1 M p τ q ´ 2 q .
Suppose that W ˚ P B p W p 0 q , τ q and W p n 1 q P B p W p 0 q , τ q for all 0 ď n 1 ď n ´ 1 .
Then it holds that
[Equation]
We introduce a surrogate loss E i p W q “ ´ (cid:96) 1 r y i ¨ f W p
x
i qs and its population version E D p W q “ E p
[Equation]
q„ D r´ (cid:96) 1 r y ¨ f W p x qss , which have been used in (Ji and Telgarsky, 2018; Cao and Gu, 2019;
Ji and Telgarsky, 2020).
Our proof is based on the application of Lemma A.4 and an online-tobatch conversion argument (Cesa-Bianchi et al., 2004; Cao and Gu, 2019; Ji and Telgarsky, 2020).
We introduce a surrogate loss E i p W q “ ´ (cid:96) 1 r y i ¨ f W p
x
i qs and its population version E D p W q “ E p x ,y q„ D r´ (cid:96) 1 p y
¨ f W p x qqs , which have been used in (Ji and Telgarsky, 2018; Cao and Gu, 2019;
Nitanda and Suzuki, 2019; Ji and Telgarsky, 2020).
Proof of Theorem 3.5.
Recall that W ˚ is chosen such that
[Equation]
and W ˚ P B p W p 0 q , Rm ´ 1 { 2 q .
To apply Lemma A.4, we need the region B p W p 0 q , τ q to include both W ˚ and t W p t q u t “ 0 ,...,t 1 .
This motivates us to set τ “ r O p
L 1 { 2 m ´ 1 { 2 R q , which is slightly larger than m ´ 1 { 2 R .
With this choice of τ , by Lemma A.1 we have (cid:15) app p τ
[Equation]
r O p τ
[Equation] [Equation] [Equation] 2 L 3 q “ r
O p R 4 { 3 L 11 { 3 m ´ 1 { 6 q .
Therefore, we can set
[Equation]
to ensure that (cid:15) app p τ q ď 1 { 8 , where r Ω p¨q hides polylogarithmic dependencies on network depth L ,
NTRF function class size R , and failure probability parameter δ .
Then by Lemma A.4, we have with probability at least 1 ´ δ ,
[Equation]
as long as W p 0 q , . .
, W p n 1 ´ 1 q P B p W p 0 q , τ q .
We then prove Theorem 3.5 in two steps: 1) all iterates stay inside B p W p 0 q , τ q ; and 2) convergence of online SGD.
All iterates stay inside B p W p 0 q , τ q .
Similar to the proof of Theorem 3.3, we prove this part by induction.
Assuming W p i q satisﬁes W p
i q P B p W p 0 q , τ q for all i ď n 1 ´ 1 , by (A.4), we have
[Equation]
where the last inequality is by W ˚ P B p W p 0 q , Rm ´ 1 { 2 q .
Then by triangle inequality, we further get
[Equation]
Then by our choices of η “ Θ `
[Equation]
1 q ˘ , we have } W p n 1 q ´ W p 0 q } F ď 2 ?
LRm ´ 1 { 2 ď τ .
This completes the proof of the ﬁrst part.
Convergence of online SGD.
By (A.4), we have
[Equation]
Dividing by ηn on the both sides and rearranging terms, we get
[Equation]
where the second inequality follows from facts that W ˚ P B p W p 0 q , R ¨ m ´ 1 { 2 q and η “ Θ ` m ´ 1 ¨ p LR 2 n ´ 1 (cid:15) ´
1 NTRF ^
L ´ 1 q ˘ .
By Lemma 4.3 in (Ji and Telgarsky, 2020) and the fact that E i p W p
i ´ 1 q q ď
L i p W p
i ´ 1 q q , we have
[Equation]
This completes the proof of the second part.
B P ROOF OF R ESULTS IN S ECTIONB. P ROOF OF P ROPOSITION 4.
We ﬁrst provide the following lemma which gives an upper bound of the neural network output at the initialization.
Lemma B.1 (Lemma 4.4 in Cao and Gu (2019)) .
Under Assumption 3.1, if m ě ¯ CL log p nL { δ q with some absolute constant ¯ C , with probability at least 1 ´ δ , we have | f W [-p 
-][Equation] x
[Equation] log p n { [Equation] for some absolute constant C .
Proof of Proposition 4.2.
Under Assumption 4.1, we can ﬁnd a collection of matrices
U ˚ “ t U ˚ 1 , ¨ ¨ ¨ , U ˚ L u with ř Ll “ 1 } U ˚ l } 2 F “ 1 such that y i x ∇ f W p
q p x i q , U ˚ y ě m 1 { 2 γ for at least 1 ´ ρ fraction of the training data.
By Lemma B.1, for all i P r n s we have | f W p
q p x i q| ď C a log p n { δ q for some absolute constant C .
Then for any positive constant λ , we have for at least 1 ´
ρ portion of the data,
[Equation]
For this fraction of data, we can set
[Equation]
where C 1 is an absolute constant, and get
[Equation]
Now we let W ˚ “ W p
q ` λ U ˚ .
By the choice of R in Proposition 4.2, we have W ˚ P
B p W p 0 q , R ¨ m ´ 1 { 2 q .
The above inequality implies that for at least 1 ´ ρ fraction of data, we have (cid:96)
[Equation]
˘ ď
[Equation]
For the rest data, we have
[Equation]
for some absolute positive constant C 1 , where the last inequality follows from fact that } ∇ f W p 0 q } 2 “ r
O p m 1 { 2 q (see Lemma A.1 for detail).
Then note that we use cross-entropy loss, it follows that for this fraction of training data, we have (cid:96) [Equation] y [Equation]
q
[Equation]
C 2 R for some constant C 2 .
Combining the results of these two fractions of training data, we can conclude
[Equation]
This completes the proof.
B. P ROOF OF P ROPOSITION 4.
Proof of Proposition 4.4.
We are going to prove that Assumption 4.3 implies the existence of a good function in the NTRF function class.
By Deﬁnition 3.2 and the deﬁnition of cross-entropy loss, our goal is to prove that there exists a collection of matrices W “ t W 1 , W 2 u satisfying
[Equation]
R ¨ m ´ 1 { 2 such that
[Equation]
We ﬁrst consider ∇ W 1 f W p
q p x i q , which has the form
[Equation]
Note that w p 0 q 2 ,j and w p 0 q 1 ,j are independently generated from N p 0 , 1 { m q
and N p 0 , 2 I { m q respectively, thus we have P p| w p 0 q 2 ,j | ě 0 .
47 m
´ 1 { 2 q ě 1 { 2 .
By Hoeffeding’s inequality, we know that with probability at least 1 ´ exp p´ m { 8 q , there are at least m { 4 nodes, whose union is denoted by S , satisfying | w p 0 q 2 ,j | ě 0 .
47 m ´ 1 { 2 .
Then we only focus on the nodes in the set S .
Note that W p 0 q 1 and
W p 0 q 2 are independently generated.
Then by Assumption 4.3 and Hoeffeding’s inequality, there exists a function u p¨q : R d Ñ R d such that with probability at least 1 ´ δ 1 ,
[Equation]
Deﬁne v j “ u p w p 0 q 1 ,j q{ w 2 ,j if | w 2 ,j | ě 0 .
47 m ´ 1 { 2 and v j “ 0 otherwise.
Then we have
[Equation]
Set δ “ 2 nδ 1 and apply union bound, we have with probability at least 1 ´ δ { 2 , m ÿ
[Equation]
Therefore, note that with probability at least 1 ´ exp p´ m { 8 q , we have | S | ě m { 4 .
Moreover, in
Assumption 4.3, by y i P t˘ 1 u and | σ 1 p¨q| , } u p¨q} 2 , } x i } 2 ď 1 for i “ 1 , . .
. , n , we see that γ ď 1 .
Then if m ě 32 log p n { δ q{ γ 2 , with probability at least 1 ´ δ { 2 ´ exp ` ´ 4 log p n { δ q{ γ˘ ě 1 ´ δ , m ÿ j “
[Equation]
p
[Equation]
j ¨
[Equation]
Let U “ p v 1 , v 2 , ¨ ¨ ¨ , v m q J { a m | S | , we have y
i
[Equation]
a [Equation]
γ
[Equation]
where the last inequality is by the fact that | S | ě m { 4 .
Besides, note that by concentration and
Gaussian tail bound, we have | f W p
q p x i q| ď C log p n { δ q for some absolute constant C .
Therefore, let W 1 “ W p 0 q 1 ` 4 ` log p 2 { (cid:15) q ` C log p n { δ q ˘ m ´ 1 { 2 U { γ and W 2 “ W p 0 q 2 , we have y i ¨ “
[Equation]
[Equation]
[Equation]
Note that } u p¨q} 2 ď 1 , we have } U } F ď 1 { 0 .
ď 2 . 2 .
Therefore, we further have } W 1 ´
W p
q 1 } F ď 8 .
8 γ ´ 1 ` log p 2 { (cid:15) q ` C log p n { δ q ˘ ¨ m ´ 1 { 2 .
This implies that W P B p W p 0 q , R q with
R “ O ` log ` n {p δ(cid:15) q ˘ { γ ˘ .
Applying the inequality (cid:96) p log p 2 { (cid:15) qq ď (cid:15) on (B.1) gives (cid:96)
[Equation]
for all i “ 1 , . . . , n .
This completes the proof.
B. P ROOF OF P ROPOSITION 4.
Based on our theoretical analysis, the major goal is to show that there exist certain choices of R and m such that the best NTRF model in the function class F p W p
q , R q can achieve (cid:15) training error.
In this proof, we will prove a stronger results by showing that given the quantities of R and m speciﬁced in Proposition 4.6, there exists a NTRF model with parameter W ˚ that satisﬁes n
[Equation]
In order to do so, we consider training the NTRF model via a different surrogate loss function.
Speciﬁcally, we consider squared hinge loss r (cid:96) p x q “ ` max t λ ´ x, 0 u ˘ 2 , where λ denotes the target margin.
In the later proof, we choose λ “ log p 1 { (cid:15) q ` 1 such that the condition r (cid:96) p x q ď 1 can guarantee that x ě log p (cid:15) q .
Moreover, we consider using gradient ﬂow, i.e., gradient descent with inﬁnitesimal step size, to train the NTRF model.
Therefore, in the remaining part of the proof, we consider optimizing the NTRF parameter W with the loss function
[Equation]
Moreover, for simplicity, we only consider optimizing parameter in the last hidden layer (i.e., W L ´ 1 ).
Then the gradient ﬂow can be formulated as
[Equation]
Note that the NTRF model is a linear model, thus by Deﬁnition 3.2, we have
[Equation]
Then it is clear that ∇ W L ´ 1 r L S
p W p t qq has ﬁxed direction throughout the optimization.
In order to prove the convergence of gradient ﬂow and characterize the quantity of R , We ﬁrst provide the following lemma which gives an upper bound of the NTRF model output at the initialization.
Then we provide the following lemma which characterizes a lower bound of the Frobenius norm of the partial gradient ∇ W L ´ 1 r L S p W q .
Lemma B.2 (Lemma B.5 in Zou et al.
Under Assumptions 3.1 and 4.5, if m “ r Ω p n 2 φ ´ 1 q , then for all t ě 0 , with probability at least 1 ´ exp `
[Equation]
O p [Equation] { n q ˘ , there exist a positive constant C such that
[Equation]
We slightly modiﬁed the original version of this lemma since we use different models (we consider
NTRF model while Zou et al.
(2019) considers neural network model).
However, by (B.2), it is clear that the gradient ∇ r L S p W q can be regarded as a type of the gradient for neural network model at the initialization (i.e., ∇ W L ´ 1 L S p W p 0 q q ) is valid.
Now we are ready to present the proof.
Proof of Proposition 4.6.
Recall that we only consider training the last hidden weights, i.e., W L ´ 1 , via gradient ﬂow with squared hinge loss, and our goal is to prove that gradient ﬂow is able to ﬁnd a NTRF model within the function class F p W p 0 q , R q around the initialization, i.e., achieving n ´ 1 ř ni “ 1 (cid:96) ` y i F W p 0 q , W ˚ p x i q ˘ ď (cid:15) .
Let W p t q be the weights at time t , gradient ﬂow implies that
[Equation]
where the ﬁrst equality is due to the fact that we only train the last hidden layer, the ﬁrst inequality is by Lemma B.2 and the second equality follows from the fact that r (cid:96) 1 p¨q “ ´ 2 br (cid:96) p¨q .
Solving the above inequality gives
[Equation]
Then, set T “ O
[Equation]
qq{ (cid:15) 1 q ˘ and (cid:15) 1 “ 1 { n , we have r L S
p W p t qq ď (cid:15) 1 .
Then it follows that r (cid:96) ` y i F W p 0 q , W p t q p x i q ˘ ď 1 , which implies that y i F W p 0 q , W p t q p x i q ě log p (cid:15) q and thus n ´ 1 ř n
[Equation]
[Equation]
(cid:96)
[Equation]
y
i F
[Equation]
˘ ď (cid:15) .
Therefore, W p T q is exactly the NTRF model we are looking for.
The next step is to characterize the distance between W p T q and W p 0 q in order to characterize the quantity of R .
Note that } ∇ W L ´ 1 r L S p W p t qq} 2 F ě 4 Cmφ r L S p W p t qq{ n 3 , we have
[Equation]
Taking integral on both sides and rearranging terms [Equation] we have
[Equation]
Note that the L.H.S. of the above inequality is an upper bound of } W p t q ´ W p 0 q} F , we have for any
[Equation]
where the second inequality is by Lemma B.1 and our choice of λ “ log p 1 { (cid:15) q ` 1 .
This implies that there exists a point W ˚ within the class F p W p 0 q , R q with
[Equation]
˙ such that
[Equation]
`
[Equation]
˘
[Equation]
Then by Theorem 3.3, and, more speciﬁcally, (A.1), we can compute the minimal required neural network width as follows,
[Equation]
This completes the proof.
C P ROOF OF T ECHNICAL L EMMAS
Here we provide the proof of Lemmas 5.1, A.3 and A.4.
C. P ROOF OF L EMMA 5.
The detailed proof of Lemma 5.1 is given as follows.
Proof of Lemma 5.1.
Based on the update rule of gradient descent, i.e., W p t ` 1 q “ W p t q ´ η ∇ W L S p W p t q q , we have the following calculation.
[Equation]
where the equation follows from the fact that L S p W p t q q “ n ´ 1 ř ni “ 1 L i
p W p t q q .
In what follows, we ﬁrst bound the term I 1 on the R.H.S. of (C.1) by approximating the neural network functions with linear models.
By assumption, for t “ 0 , . . .
, t 1 ´ 1 , W p t q , W ˚ P B p W p 0 q , τ q .
Therefore by the deﬁnition of (cid:15) app p τ q ,
[Equation]
Moreover, we also have
[Equation]
where the equation follows by the deﬁnition of F W p 0 q , W ˚ p x q .
Adding (C.3) to (C.2) and canceling the terms y i ¨
f W ˚ p x i q [Equation] we obtain that
[Equation]
We can now give a lower bound on ﬁrst term on the R.H.S. of (C.1).
For i “ 1 , . . . , n , applying the chain rule on the loss function gradients and utilizing (C.4), we have
[Equation]
where the ﬁrst inequality is by the fact that (cid:96)` y
[Equation]
˘ ă 0 , the second inequality is by convexity of (cid:96) p¨q and the fact that ´ (cid:96)
[Equation]
We now proceed to bound the term I 2 on the R.H.S. of (C.1).
Note that we have (cid:96) 1 p¨q ă 0 , and therefore the Frobenius norm of the gradient ∇ W l L S p W p t q q can be upper bounded as follows,
[Equation]
where the inequality follows by triangle inequality.
We now utilize the fact that cross-entropy loss satisﬁes the inequalities ´ (cid:96) 1 p¨q ď (cid:96) p¨q
and ´ (cid:96) 1 p¨q ď 1 .
Therefore by deﬁnition of M p τ q , we have
[Equation]
Then we can plug (C.5) and (C.6) into (C.1) and obtain } W p t q ´ W ˚ } 2 F ´ } W p t ` 1 q ´ W ˚ } 2 F ě 2 η n n ÿ i “” p 1 ´ 2 (cid:15) app p τ qq (cid:96) ` y i
f W p t q p x i q ˘ ´ (cid:96) ` y i F W p 0 q , W ˚ p x i q ˘ı ´ O ` η 2 LM p τ q 2 ˘ ¨ L S p W p t q q ě „2 ´ 4 (cid:15) app p
[Equation]
q  ηL S p W p t
q q ´ 2 η n n ÿ i “(cid:96) ` y i F W p 0 q , W ˚ p x i q ˘ , where the last inequality is by η “ O p L ´ 1 M p τ q ´ 2 q and merging the third term on the second line into the ﬁrst term.
Taking telescope sum from t “ 0 to t “ t 1 ´ 1 and plugging in the deﬁnitionn ř ni “ 1 (cid:96) ` y i F W p 0 q , W ˚ p x i
q ˘ “ (cid:15) NTRF completes the proof.
C. P ROOF OF L EMMA A.
Proof of Lemma A.3.
We ﬁrst denote W “ B p W p 0 q , r R ¨ m ´ 1 { 2 q , and deﬁne the corresponding neural network function class and surrogate loss function class as F “ t f W p x q : W P W u and
G “ t´ (cid:96) r y ¨ f W p x qs : W P W u respectively.
By standard uniform convergence results in terms of empirical Rademacher complexity (Bartlett and
Mendelson, 2002; Mohri et al., 2018; Shalev-Shwartz and Ben-David, 2014), with probability at least 1 ´ δ we have
[Equation]
where C 1 is an absolute constant, and
[Equation]
is the empirical Rademacher complexity of the function class G .
We now provide two bounds on p R n p G q , whose combination gives the ﬁnal result of Lemma A.3.
First, by Corollary 5.35 in (Vershynin, 2010), with probability at least 1 ´ L ¨ exp p´ Ω p m qq , } W p
q l } 2 ď 3 for all l P r L s .
Therefore for all W P W , we have } W l } 2 ď 4 .
Moreover, standard concentration inequalities on the norm of the ﬁrst row of W p
q l also implies that } W l } 2 ě 0 .
5 for all W P W and l P r L s .
Therefore, an adaptation of the bound in (Bartlett et al., 2017) ¶ gives
[Equation]
We now derive the second bound on p R n p G q , which is inspired by the proof provided in (Cao and
Gu, 2020).
Since y P t` 1 , 1 u , | (cid:96) 1 p z q| ď 1 and (cid:96) 1 p z q is 1 -Lipschitz continuous, by standard empirical
Rademacher complexity bounds (Bartlett and Mendelson, 2002; Mohri et al., 2018; Shalev-Shwartz and Ben-David, 2014), we have
[Equation]
where p R n p F q is the empirical Rademacher complexity of the function class F .
We have
[Equation]
where F W p 0 q , W p x q “ f W p
q p x q `
[Equation]
D .
For I 1 , by Lemma 4.1 in (Cao and
Gu, 2019), with probability at least 1 ´ δ { 2 we have
[Equation]
For I 2 , note that E ξ
[Equation] [Equation]
By Cauchy-Schwarz inequality we have
[Equation] [Equation]
Therefore
I
[Equation]
where we apply Jensen’s inequality to obtain the ﬁrst inequality
[Equation]
and the last inequality follows by
Lemma B.3 in (Cao and Gu, 2019).
Combining the bounds of I
and I 2 gives
[Equation]
Further combining this bound with (C.7) and recaling δ completes the proof.
C. P ROOF OF L EMMA A.
Proof of Lemma A.4.
Different from the proof of Lemma 5.1, online SGD only queries one data to update the model parameters in each iteration,
i
[Equation]
e., W i ` 1 “ W i ´ η ∇ L
i
` 1 p W p i q q .
By this update rule, we have
[Equation]
With exactly the same proof as (C.5) in the proof of Lemma 5.
[Equation]
we have
[Equation]
[Equation]
for all i “ 0 , . . .
, n 1 ´ 1 .
By the fact that ´ (cid:96) 1 p¨q ď (cid:96) p¨q and ´ (cid:96) 1 p¨q ď 1 , we have
[Equation]
[Equation]
Then plugging (C.10) and (C.11) into (C.9) gives
[Equation]
where the last inequality is by η “ O p L ´ 1 M p τ q ´ 2 q and merging the third term on the second line into the ﬁrst term.
Taking telescope sum over i “ 0 , . .
n 1 ´
[Equation]
we obtain
[Equation]
NTRF
[Figure]
This ﬁnishes the proof.
D E XPERIMENTS
In this section [Figure] we conduct some simple experiments to validate our theory.
Since our paper mainly focuses on binary classiﬁcation, we use a subset of the original CIFAR10 dataset (Krizhevsky et al., 2009), which only has two classes of images.
We train a 5 -layer fullyconnected ReLU network on this binary classiﬁcation dataset with different sample sizes ( n P t 100 , 200 , 500 , 1000 , 2000 , 5000 , 10000 u ), and plot the minimal neural network width that is required to achieve zero training error in Figure 1 (solid line).
We also plot O p n q , O p log 3 p n qq , O p log 2 p n qq and O p log p n qq in dashed line for reference.
It is evident that the required network width to achieve zero training error is polylogarithmic on the sample size n , which is consistent with our theory
[Figure]