A T HEORETICAL A NALYSIS ON F EATURE L EARNING IN N EURAL N ETWORKS : E MERGENCE FROM I NPUTS AND A DVANTAGE OVER F IXED F EATURES
Anonymous authors
Paper under double-blind review
A BSTRACT
An important characteristic of neural networks is their ability to learn representations of the input data with effective features for prediction, which is believed to be a key factor to their superior empirical performance.
To better understand the source and beneﬁt of feature learning in neural networks, we consider learning problems motivated by practical data, where the labels are determined by a set of class relevant patterns and the inputs are generated from these along with some background patterns.
We prove that neural networks trained by gradient descent can succeed on these problems.
The success relies on the emergence and improvement of effective features, which are learned among exponentially many candidates efﬁciently by exploiting the data (in particular, the structure of the input distribution).
In contrast, no linear models on data-independent features of polynomial sizes can learn to as good errors.
Furthermore, if the speciﬁc input structure is removed, then no polynomial algorithm in the Statistical Query model can learn even weakly.
These results provide theoretical evidence showing that feature learning in neural networks depends strongly on the input structure and leads to the superior performance.
Our preliminary experimental results on synthetic and real data also provide positive support.
I NTRODUCTION
Various empirical studies have shown that an important characteristic of neural networks is their feature learning ability, i.e., to learn a feature mapping for the inputs which allow accurate prediction (e.g., Zeiler & Fergus (2014);
Girshick et al.
; Zhang et al.
; Manning et al.
This is widely believed to be a key factor to their remarkable success in many applications, in particular, an advantage over traditional machine learning methods.
To understand their success, it is then crucial to understand the source and beneﬁt of feature learning in neural networks.
Empirical observations show that networks can learn neurons that correspond to different semantic patterns in the inputs (e.g., eyes, bird shapes, tires, etc.
in images (Zeiler & Fergus, 2014; Girshick et al., 2014)).
Moreover, recent progress (e.g., Caron et al.
; Chen et al.
(2020b)
; He et al.
; Jing & Tian (2020)) shows that one can even learn a feature mapping using only unlabeled inputs and then learn an accurate predictor (usually a linear function) on it using labeled data.
This further demonstrates the feature learning ability of neural networks and that these input distributions contain important information for learning useful features.
These empirical observations strongly suggest that the structure of the input distribution is crucial for feature learning and feature learning is crucial for the strong performance.
However, it is largely unclear how practical training methods (gradient descent or its variants) learn important patterns from the inputs and whether this is necessary for obtaining the superior performance, since the empirical studies do not exclude the possibility that some other training methods can achieve similar performance without feature learning or with feature learning that does not exploit the input structure.
Rigorous theoretical investigations are thus needed for answering these fundamental questions: How can effective features emerge from inputs in the training dynamics of gradient descent?
Is learning features from inputs necessary for the superior performance?
Compared to the abundant empirical evidence, the theoretical understanding still remains largely open.
One line of work (e.g. Jacot et al.
(2018); Li & Liang (2018);
Du et al.
Allen-Zhu et al.
; Chizat et al.
; Cao et al.
(2020) and many others) shows in certain regime, sufﬁciently overparameterized networks are approximately linear models, i.e., a linear function on the Neural
Tangent Kernel (NTK).
This falls into the traditional approach of linear models on ﬁxed features, which also includes random features (Rahimi & Recht, 2008) and other kernel methods (Kamath et al., 2020).
The kernel viewpoint thus does not explain feature learning in networks nor the advantage over ﬁxed features.
A recent line of work (e.g. Daniely & Malach (2020); Bai & Lee (2019); Ghorbani et al.
; Yehudai & Shamir (2019); Allen-Zhu & Li (2019; 2020a); Li et al.
(2020); Malach et al.
(2021) and others) shows examples where neural networks provably enjoy the advantage over ﬁxed features, under different settings and assumptions.
While providing insightful results on the separation of the two approaches, they have not investigated whether the input structure is a crucial factor for feature learning and thus the advantage.
Also, most studies have not analyzed how gradient descent can learn important input patterns as effective features, or rely on strong assumptions like models or data atypical in practice (e.g., special networks, Gaussian data, etc).
Towards a more thorough understanding, we propose to analyze learning problems motivated by practical data, where the labels are determined by a set of class-relevant patterns and the inputs are generated from these along with some background patterns.
We use comparison for our study: (1) by comparing network learning with ﬁxed feature approaches on these problems, we analyze the emergence of effective features and demonstrate feature learning leads to the advantage over ﬁxed features; (2) by comparing these problems to those with the input structure removed, we demonstrate that the input structure is crucial for feature learning and prediction performance.
More precisely, we obtain the following results.
We ﬁrst prove that two-layer networks trained by gradient descent can efﬁciently learn to small errors on these problems, and then prove that no linear models on ﬁxed features of polynomial sizes can learn to as good errors.
These two results thus establish the provable advantage of networks and implies that feature learning leads to this advantage.
More importantly, our analysis reveals the dynamics of feature learning: the network ﬁrst learns a rough approximation of the effective features, then improves them to get a set of good features, and ﬁnally learns an accurate classiﬁer on these features.
Notably, the improvement of the effective features in the second phase is needed for obtaining the provable advantage.
The analysis also reveals the emergence and improvement of the effective features are by exploiting the data, and in particular, they rely on the input structure.
To formalize this, we further prove the third result: if the speciﬁc input structure is removed and replaced by a uniform distribution, then no polynomial algorithm can even weakly learn in the Statistical Query (SQ) learning model, not to mention the advantage over ﬁxed features.
Since SQ learning includes essentially all known algorithms (in particular, mini-batch stochastic gradient descent used in practice), this implies that feature learning depends strongly on the input structure.
Finally, we perform simulations on synthetic data to verify our results.
We also perform experiments on real data and observe similar phenomena, which show that our analysis provides useful insights for the practical network learning.
Our analysis then provides theoretical support for the following principle: feature learning in neural networks depends strongly on the input structure and leads to the superior performance.
In particular, our results make it explicit that learning features from the input structure is crucial for the superior performance.
This suggests that input-distribution-free analysis (e.g., traditional PAC learning) may not be able to explain the practical success, and advocates an emphasis of the input structure in the analysis.
While these results are for our proposed problem setting and network learning in practice can be more complicated, the insights obtained match existing empirical observations and are supported by our experiments.
The compelling evidence hopefully can attract more attention to further studies on modeling the input structure and analyzing feature learning.
R ELATED W ORK
Neural Tangent Kernel (NTK) and Linearization of Neural Networks.
One line of work explains the success of sufﬁciently over-parameterized neural network by connecting them to linear methods like NTK (e.g. Jacot et al.
(2018); Li & Liang (2018);
Matthews et al.
; Lee et al.
; Novak et al.
; Yang (2019);
Du et al.
Allen-Zhu et al.
; Ji & Telgarsky (2019); Cao et al.
Geiger et al.
Chizat et al.
(2019) and more).
Though their approaches are different, they all base on the observation that when the neural network is sufﬁciently large, the weight of the neurons stays close to the initialization during the training.
In this situation, training with gradient descent appears no different from solving a kernel method problem.
This view is typically referred to as the NTK regime, or lazy training, or linearization.
However, neural networks used in practice are usually not large enough to enter this regime, and the neurons are frequently observed to traverse instead of staying close to their initialization during the training process.
Furthermore, in this regime, network learning is essentially the traditional approach of linear methods over ﬁxed features, which cannot establish or explain feature learning and the advantage of network learning.
Advantage of Neural Networks over Linear Models on Fixed Features.
Since the benign network learning results via gradient descent are not well explained by the NTK view, a recent line of work has turned to show learning settings where neural networks provably have advantage over linear models on ﬁxed features (e.g. Daniely & Malach (2020); Reﬁnetti et al.
; Malach et al.
Dou & Liang (2020); Bai & Lee (2019); Ghorbani et al.
; Allen-Zhu & Li (2019); see the nice summary in Malach et al. (2021)).
While formally establishing the advantages of network learning over ﬁxed feature approaches, they have not answered the two fundamental questions this work focuses on; in particular, they have not studied whether the input structure is a crucial factor for feature learning and thus the advantage.
For example, Ghorbani et al.
(2020) show the advantage of networks in approximation power and Dou & Liang (2020) show their statistical advantage, but they do not consider the learning dynamics (i.e., how the training method obtains the good network).
Allen-Zhu & Li (2019) prove the advantage of the networks for PAC learning with labels given by a depth-2 ResNet and Allen-Zhu & Li (2020a) prove for Gaussian inputs with labels given by a multiple-layer network, while neither considers the inﬂuence of the input structure on feature learning or the advantage.
Daniely & Malach (2020) prove the advantage of the networks for the task of learning sparse parities on speciﬁcally designed input distributions that help the gradient descent learn effective features for prediction, and Malach et al.
(2021) consider similar learning problems but with speciﬁcally designed differentiable models.
On the other hand, neither study explores if the input structure is needed for the learning, and the data distributions and models are also atypical in practice, while our setting better connects to the practice.
More technical discussions can be found in
Appendix A.
There are also other theoretical studies on feature learning in networks (e.g. Yehudai & Ohad (2020);
Zhou et al.
; Diakonikolas et al.
Frei et al.
(2020)), which however do not directly relate feature learning to the input structure or the advantage.
P ROBLEM S ETUP
To motivate our setup, consider images with various kinds of patterns like lines and retangles.
Some patterns are relevant for the labels (e.g., retangles for distinguishing indoor or outdoor images), while the others are not.
If the image contains a sufﬁcient number of the former, then we are conﬁdent that the image belongs to a certain class.
Dictionary learning or sparse coding (e.g. Olshausen & Field (1997); Vinje & Gallant (2000); Blei et al. (2003)) is a classic model of such data.
We thus model the patterns as a dictionary, generate a hidden vector indicating the presence of the patterns, and generate the input and label from this vector.
Let X =
R d be the input space, and Y = {± 1 } be the label space.
Suppose M ∈ R d × D is an unknown dictionary with D columns that can be regarded as patterns.
For simplicity, assume M is orthonormal.
Let ˜ φ ∈ { 0 , 1 } D be a hidden vector that indicates the presence of each pattern.
Let
A ⊆
[ D ] be a subset of size k corresponding to the class relevant patterns.
Then the input is generated by M ˜ φ , and the label can be any binary function on the number of class relevant patterns.
More precisely, let P ⊆
[ k ] .
We ﬁrst sample ˜
φ from a distribution D ˜ φ , and then generate the input ˜ x and the class label y from ˜ φ and the ﬁxed A, P :
[Equation]
Learning with Input Structure.
We allow quite general D ˜ φ with the following assumptions:
(A0)
The class probabilities are balanced:
Pr[ (cid:80) i ∈
A ˜ φ i ∈ P ] = 1 / 2 .
(A1)
The patterns in A are correlated with the labels: for any i ∈ A , γ = E [ y ˜ φ i ] − E
[ y ] E [ ˜ φ i ] > 0 .
(A2) Each pattern outside A is independent of all other patterns and identically distributed.
Let p o :
=
Pr[˜ φ i = 1] ≤ 1 / 2 denote the probability they appear.
Let D ( A, P, D ˜ φ ) denote the distribution on (˜ x, y ) corresponding to some A, P, and D ˜ φ .
Given parameters Ξ =
( d, D, k, γ, p o ) , the family F Ξ of distributions for learning is the set of all D ( A, P, D ˜ φ ) with A ⊆
[ D ] , P ⊆
[ k ] , and D ˜ φ satisfying the above assumptions.
The labeling function includes some interesting special cases:
Example 1.
Suppose k is odd, and P = { i ∈ [ k ] : i > k/ 2 } for some threshold, i.e., we will set the label y
= +1 when more than a half of the relevant patterns are presented in the input.
Example 2.
Suppose k is odd, and let P
= { i ∈ [ k ] : i is odd } , i.e., the labels are given by the parity function on ˜ φ j ( j ∈ A ) .
This is useful to prove our lower bounds via the properties of parities.
Appendix F presents results for more general settings (e.g., incoherent dictionary, unbalanced classes, etc.).
On the other hand, our problem setup does not include some important data models.
In particular, one would like to model hierarchical representations often observed in practical data and believed to be important for deep learning.
We leave such more general cases for future work.
Learning Without Input Structure.
For comparison, we also consider learning problems without input structure.
The data is generated as above but with different distributions D ˜ φ :
(A1’)
The
patterns are uniform over { 0 , 1 } D : for any i ∈
[ D ] , Pr[˜ φ i = 1] = 1 / 2 independently
Given parameters Ξ 0 = ( d, D, k ) , the family F Ξ 0 of distributions without input structure is the set of all the distributions with A ⊆
[ D ] , P ⊆
[ k ] and D ˜ φ satisfying the above assumptions.
[Equation]
EARNING
Networks.
We consider training a two-layer network via gradient descent on the data distribution:
[Equation]
where w i ∈ R d , b i , a i ∈ R , and σ ( z ) = min(1 , max( z, 0)) is the truncated rectiﬁed linear unit (ReLU) activation function.
Let θ = { w i , b i , a i } 2 mi =1 denote all the parameters, and let superscript ( t ) denote the time step, e.g., g ( t ) denote the network at time step t with θ ( t )
=
{ w ( t ) i , b ( t ) i , a ( t ) i } .
Loss Function.
Similar to typical practice, we will normalize the data for learning: ﬁrst compute x = (˜ x − E [˜ x ]) /
˜ σ where ˜ σ 2 = (cid:80) di
=1 (˜ x i − E
[˜ x i ]) 2 is the variance of the data, and then train on ( x, y ) .
This is equivalent to setting φ =
(˜ φ − E [˜ φ ]) /
˜ σ and generating x = Mφ .
For (˜ x, y ) from D and the normalized ( x, y ) , we will simply say ( x, y ) ∼ D .
For the training, we consider the hinge-loss (cid:96) ( y, ˆ y )
=
max { 1 − y ˆ y, 0 } .
We will inject some noise ξ to the neurons for the convenience of the analysis.
(This can be viewed as using a smoothed version of the activation ˜ σ ( z ) =
E ξ σ ( z + ξ ) similar to those in existing studies like Allen-Zhu & Li (2020b);
Malach et al.
See Section 5 for more explanations.)
Formally, the loss is:
L D ( g ; σ ξ ) =
E ( x,y ) [ (cid:96) ( y, g ( x ; ξ ))] , where g ( x ; ξ ) = 2 m (cid:88) i =
a
[Equation]
where ξ ∼ N (0 , σ 2 ξ I m × m ) are independent noise.
Let L D ( g ) denote the hinge-loss without noise.
Consider (cid:96) 2 regularization: R ( g ; λ a , λ w ) = (cid:80) 2 mi =1 λ a | a i | 2 + λ w (cid:107) w i (cid:107) 22 with coefﬁcients λ a , λ w .
Training Process.
We ﬁrst perform an unbiased initialization: for every i ∈ [ m ] , initialize w (0) i ∼
N (0 , σ 2 w I d × d ) with σ w = 1 /k , b (0) i ∼ N (cid:0) 0 , σ 2 b (cid:1) with σ b = 1 /k 2 , a (0) i ∼ N (0 , σ 2 a ) with σ a = ˜ σ 2 / ( γk 2 ) , and then set w (0) m
+ i = w (0) i , b (0) m
+ i
=
b (0)
i , a (0) m
+ i = − a (0) i .
We then do gradient updates:
[Equation]
+
[Equation] [Equation]
[Equation]
for t
[Equation] [Equation]
4) for some choice of the hyperparameters η ( t ) , λ ( t ) a , λ ( t ) w , σ ( t ) ξ , and T .
M AIN R ESULTS
Provable Guarantee for Neural Networks.
The network learning has the following guarantee:
Theorem 1.
For any δ, (cid:15) ∈ (0 , 1) , if k = Ω (cid:0) log 2 ( D/ ( δγ )) (cid:1) , p o = Ω( k 2 /D ) , and max { Ω( k 12 /(cid:15) 3 / 2 ) , D } ≤ m ≤ poly ( D ) , then with properly set hyperparameters, for any D ∈ F Ξ , with probability at least 1 − δ , there exists t ∈
[ T ] such that Pr[sign( g ( t ) ( x ))
(cid:54) =
y ] ≤ L D ( g ( t ) ) ≤ (cid:15).
The theorem shows that for a wide range of the background pattern probability p o and the number of class relevant patterns k , the network trained by gradient descent can obtain a small classiﬁcation error.
More importantly, the analysis shows the success comes from feature learning.
In the early stage, the network learns and improves the neuron weights such that on the features (i.e., the neurons’ outputs) there is an accurate classiﬁer; afterwards it can learn such a classiﬁer on these features.
Lower Bound for Fixed Features.
Empirical observations and Theorem 1 do not exclude the possibility that some methods without feature learning can achieve similar performance.
We thus prove a lower bound for the ﬁxed feature approach, i.e., linear models on data-independent features.
Theorem 2.
Suppose Ψ is a data-independent feature mapping of dimension N with bounded features, i.e., Ψ : X → [ − 1 , 1] N .
For B > 0 , the family of linear models on Ψ with bounded norm B is H B
=
{ h (˜ x ) : h (˜ x ) = (cid:104) Ψ(˜ x ) , w (cid:105) , (cid:107) w (cid:107) 2 ≤ B } .
If 3 < k ≤ D/ 16 and k is odd, then there exists
D ∈ F Ξ such that all h ∈ H B have hinge-loss at least p o (cid:16) 1 − √
[Equation]
It shows that using ﬁxed features, one cannot get loss nontrivially smaller than p o unless with exponentially large models.
In contrast, viewing the hidden neurons Ψ i =
σ ( (cid:104) w i , x (cid:105)
+ b i ) as learned features, network learning can achieve any loss (cid:15) ∈ (0 , 1) with models of polynomial sizes.
The theorem applies to linear functions and random feature approaches (Rahimi & Recht, 2008).
It also implies lower bounds to inﬁnite dimensional feature maps (e.g., some kernels) that can be approximated by feature maps of polynomial dimensions.
For example, Claim 1 in (Rahimi & Recht, 2008) implies that a function f using shift-invariant kernels (e.g., RBF) can be approximated by a model (cid:104) Ψ(˜ x ) , w (cid:105) with the dimension N and weight norm B bounded by polynomials of the related parameters of f like its RKHS norm and the input dimension.
Then our theorem implies some related parameter of f needs to be exponential in k for f to get nontrivial loss.
Kamath et al.
(2020) has more discussions on approximating kernels with ﬁnite dimensional maps.
Lower Bound for Without Input Structure.
Existing results do not exclude the possibility that some learning methods without exploiting the input structure can achieve strong performance.
To show the necessity of the input structure, we consider learning F Ξ 0 with input structure removed.
We obtain a lower bound for such learning problems in the classic Statistical Query (SQ) model (Kearns, 1998).
In this model, the algorithm can only receive information about the data through statistical queries.
A statistical query is speciﬁed by some polynomially-computable property predicate Q of labeled instances and a tolerance parameter τ ∈ [0 , 1] .
For a query ( Q, τ ) , the algorithm receives a response ˆ P Q ∈
[ P Q − τ, P Q + τ ] , where P Q =
Pr[ Q ( x, y ) is true ] .
Notice that a query can be simulated using the average of roughly O (1 /τ 2 ) random data samples with high probability.
The SQ model captures almost all common learning algorithms (except Gaussian elimination) including the commonly used mini-batch SGD, and thus is suitable for our purpose.
Theorem 3.
For any algorithm in the Statistical Query model that can learn over F Ξ 0 to classiﬁcation error less than 12 − [Equation] , either the number of queries or [Equation]
must be at least
[Equation]
The theorem shows that without the input structure, polynomial algorithms in the SQ model cannot get a classiﬁcation error nontrivially smaller than random guessing.
The comparison to the result for with input structure then shows that the input structure is crucial for network learning, in particular, for achieving the advantage over ﬁxed feature models.
P ROOF S KETCHES
Here we provide the sketch of our analysis, focusing on the key intuition and discussing some interesting implications.
The complete proofs are included in Appendix B-D.
5. P ROVABLE G UARANTEES OF N EURAL N ETWORKS
Overall Intuition.
We ﬁrst show that there is a two-layer network that can represent the target labeling function, whose neurons can be viewed as the “ground-truth” features to be learned.
We then show that after the ﬁrst gradient step, the hidden neurons of the trained network become close to the ground-truth: their weights contain large components along the class relevant patterns but small along the background patterns.
We further show that in the second gradient step, these features get improved: the “signal-noise” ratio between the components for class relevant patterns and those for the background ones becomes larger, giving a set of good features.
Finally, we show that the remaining steps learn an accurate classiﬁer on these features.
Existence of A Good Network.
We show that there is a two-layer network that can ﬁt the labels.
Lemma 4.
For any D ∈ F Ξ , there exists a network g ∗ ( x )
= (cid:80) ni
=1 a ∗ i σ ( (cid:104)
w ∗
i , x (cid:105) + b ∗ i ) with y =
g ∗ ( x ) for any ( x, y ) ∼ D .
Furthermore, the number of neurons n = 3( k + 1) , | a ∗ i | ≤ 32 k, 1 / (32 k ) ≤ | b ∗ i | ≤ 1 / 2 , w ∗ i = ˜
σ (cid:80)
j ∈
A M j / (4 k ) , and |(cid:104) w ∗ i , x (cid:105)
+ b ∗ i | ≤ 1 for any i ∈ [ n ] and ( x, y ) ∼ D .
In particular, the weights of the neurons are proportional to (cid:80)
j ∈
A M j , the sum of the class relevant patterns.
We thus focus on analyzing how the network learns such neuron weights.
Feature Emergence in the First Gradient Step.
The gradient for w i (ignoring the noise) is:
[Equation]
where the last step is due to g ( x ) = 0 by the unbiased initialization.
Let q j = (cid:104)
M j , w i (cid:105) denote the component along the direction of the pattern M j .
Then the component of the gradient on M j is:
[Equation]
The key intuition is that with the randomness of φ (cid:96) (and potentially that of the injected noise ξ ), the random variable under σ (cid:48) is not signiﬁcantly affected by a small subset of φ (cid:96) q (cid:96) .
For example, for class relevant patterns j ∈ A , let I [ D ] :=
σ (cid:48) (cid:104)(cid:80) (cid:96) ∈
[ D ] φ (cid:96) q (cid:96) + b i (cid:105) and I − A :
=
σ (cid:48) (cid:104)(cid:80)
[Equation]
(cid:54)∈ A φ
[Equation]
q (cid:96) + b i (cid:105) .
We have I
[ D ] ≈ I − A and thus:
[Equation]
since y only depends on φ j ( j ∈ A ) .
Then the gradient has a nontrivial component along the pattern.
Similarly, for background patterns j (cid:54)∈ A , the component of the gradient along M j is close to 0 .
Lemma 5 (Informal) .
Assume p o , k as in Theorem 1 and σ (1) ξ < 1 /k , then with high probability ∂
∂w i L D ( g (0) ; σ (1) ξ )
= − a (0) i (cid:80)
D j =1 M j T j where for a small (cid:15) e :
• if j ∈ A , then | T j − βγ/ ˜ σ | ≤ O ( (cid:15) e / ˜ σ ) with β ∈ [Ω(1) , 1] ; • if j (cid:54)∈ A , then | T j | ≤ O ( σ 2 φ (cid:15) e ˜ σ ) .
By setting λ (1) w = 1 / (2 η (1) ) , we have w (1) i =
η (1) a (0) i (cid:80) Dj =1 M j T j ≈ η
(1) a (0) i βγ ˜ σ (cid:80)
j ∈
A M j .
For small p o , e.g., p o =
˜ O
( k 2 /D ) , these neurons can already allow accurate prediction.
However, for such small p o , we cannot show a provable advantage of networks over ﬁxed features.
Furthermore, for larger p o meaning a signiﬁcant number of background patterns in the input, the approximation error terms T j ( j (cid:54)∈ A ) together can overwhelm the signals T j ( j ∈ A ) and lead to bad prediction, even though each term is small.
Fortunately, we will show that the second gradient step can improve the weights by decreasing the ratio between T j ( j (cid:54)∈ A ) and T j ( j ∈ A ) .
Feature Improvement in the Second Gradient Step.
We note that by setting a small η (1) , after the update we still have yg ( x ; ξ ) < 1 for most ( x, y ) ∼ D and thus the gradient in the second step is:
[Equation]
We can then follow the intuition for the ﬁrst step again.
For j ∈ A , the component (cid:104) M j , ∂∂w i L D ( g ) (cid:105) is roughly proportional to γ ˜ σ E
[ I − A,ξ ] where I − A,ξ := σ (cid:48)
(cid:104)(cid:80) (cid:96) (cid:54)∈ A φ (cid:96) q (cid:96)
+ b i + ξ i (cid:105) .
While φ (cid:96) q (cid:96) may not have large enough variance, the injected noise ξ i makes sure that a nontrivial amount of data activate the neuron.
Then I − A,ξ (cid:54) = 0 , leading to a nontrivial component along M j , similar to the ﬁrst step.
On the other hand, for j (cid:54)∈ A , the approximation error term T j depends on how well σ (cid:48)
(cid:104)(cid:80) (cid:96) (cid:54)∈ A,(cid:96) (cid:54) =
j φ (cid:96) q (cid:96) + b i + ξ i (cid:105) approximates σ (cid:48)
(cid:104)(cid:80) (cid:96) ∈
[ D ] φ (cid:96) q (cid:96) + b i + ξ i (cid:105) .
Since the q (cid:96) ’s (the weight’s component along M (cid:96) ) in the second step are small compared to those in the ﬁrst step, we can then get a small error term T j .
Lemma 6 (Informal) .
Assume p o , k as in Theorem 1, η (1) is sufﬁciently small, and σ (2) ξ = 1 /k 3 / 2 , then with high probability ∂∂w i L D ( g (1) ; σ (2) ξ )
= − a (1) i (cid:80)
Dj =1 M j T j where for a small (cid:15) e2 (cid:28) (cid:15) e :
• if j ∈ A , then T j ≈ βγ/ ˜ σ with β ∈ [Ω(1) , 1] ; • if j (cid:54)∈ A , then | T j | ≤ O ( σ 2 φ (cid:15) e2 ˜
σ )
+ 1˜ σ exp( − Θ( p o D )) .
So roughly the ratio between T j ( j (cid:54)∈ A ) over T j ( j ∈ A ) improves from O ( σ 2 φ ˜ σ 2 (cid:15) e /γ )
after the ﬁrst step to O ( σ 2 φ ˜ σ 2 (cid:15) e2 /γ ) after the second step, giving good features allowing accurate prediction.
Remark.
The analysis can be carried out for more gradient steps following similar intuition, while we analyze two steps for simplicity.
Remark.
Readers may notice that the network can be overparameterized.
With sufﬁcient overparameterization and proper initialization and step sizes, network learning becomes approximately
NTK.
However, here our learning scheme allows going beyond this kernel regime: we use aggressive gradient updates λ ( t )
w = 1 / (2 η ( t ) )
in the ﬁrst two steps, completely forgetting the old weights to learn effective features.
Using proper initialization and aggressive updates early to escape the kernel regime has been studied in existing work (e.g., Woodworth et al.
; Li et al.
Our result thus adds another concrete example.
Classiﬁer Learning Stage.
Given the learned features, we are then ready to show the remaining gradient steps can learn accurate classiﬁers.
Intuitively, with small hyperparameter values ( η ( t ) =
k
Tm 1 / 3 , λ ( t ) a = λ ( t ) w ≤
[Equation] [Equation]
t
[Equation]
= 0 for 2 < t ≤ T = m 4 / 3 ), the ﬁrst layer’s weights do not change too much and thus the learning is similar to convex learning using the learned features.
Formally, our proof uses the online convex optimization technique in Daniely & Malach (2020).
5. L OWER B OUNDS
The lower bounds are based on the following observation: our problem setup is general enough to include learning sparse parity functions.
Consider an odd k , and let P
= { i ∈ [ k ] : i is odd } .
Then y is given by Π A ( z ) :
=
(cid:81)
j ∈
A z j for z j = 2˜ φ j − 1 , i.e., the parity function on z j ( j ∈ A ) .
Then known results for learning parity functions can be applied to prove our lower bounds.
Lower Bound for Fixed Features.
We show that F Ξ contains learning problems that consist of a mixture of two distributions with weights p o and 1 − p o respectively, where in the ﬁrst distribution
D (1) A , ˜ x is given by the uniform distribution over ˜ φ and the label y is given by the parity function on A .
On such D (1) A , Daniely & Malach (2020) shows that exponentially large models over ﬁxed features is needed to get nontrivial loss.
Intuitively, there are exponentially many labeling functions Π A that are uncorrelated (i.e., “orthogonal” to each other): E [Π A 1 Π A 2 ] = 0 for any A 1 and A 2 .
Note that the best approximation of Π A by a ﬁxed set of features Ψ i ’s is its projection on the linear span of the features.
Then with polynomial-size models, there always exists some Π A far from the linear span.
Remark.
It is instructive to compare to network learning, which ﬁnds the effective weights (cid:80)
j ∈
A M j among the exponentially many candidates corresponding to different A ’s. This can be done efﬁciently by exploiting the data since the gradient is roughly proportional to E { yx }
= (cid:80)
j ∈
A M j .
The network then learns data-dependent features on which polynomial size linear models can achieve small loss.
Lower Bound for Learning without Input Structure.
Clearly, F Ξ 0 contains the distributions D (1) A described above.
The lower bound then follows from classic SQ learning results (Blum et al., 1994).
Remark.
The SQ lower bound analysis does not apply to F Ξ , because in F Ξ the input distribution is related the labeling function.
This allows networks to learn with polynomial time/sample.
While both the labeling function and the input distribution affect the learning, few existing studies explicitly point out the importance of the input structure.
We thus emphasize the input structure is crucial for networks to learn effective features and achieve superior performance.
E XPERIMENTS
Our experiments mainly focus on feature learning and the effect of the input structure.
We ﬁrst perform simulations on our learning problems to (1) verify our main theorems on the beneﬁt of feature learning and the effect of input structure; (2) verify our analysis of feature learning in networks.
We then check if our insights carry over to real data: (3) whether similar feature learning is presented in real network/data; (4) whether damaging the input structure lowers the performance.
The results are consistent with our analysis and provide positive support for the theory.
Below we present part of the results and include the complete experimental details and results in Appendix E.
Simulation: Veriﬁcation of the Main Results.
We generate data according to our problem setup, with d = 500 , D = 100 , k = 5 , p o = 1 / 2 , a randomly sampled A , and labels given by the parity function.
We then train a two-layer network with m =
following our learning process, and for comparison, we also use two ﬁxed feature methods (the NTK and random feature methods based on the same network).
Finally, we also use these three methods on the data distribution with the input structure removed (i.e.,
F Ξ 0 in Theorem 3).
Figure 1 shows that the results are consistent with our results.
Network learning gets high test accuracy while the two ﬁxed feature methods get signiﬁcantly lower accuracy.
Furthermore, when the input structure is removed, all three methods get test accuracy similar to random guessing.
Simulation: Feature Learning in Networks.
We compute the cosine similarities between the weights w i ’s and visualize them by Multidimensional Scaling.
(Recall that our analysis is on the directions of the weights without considering their scaling , and thus it is important to choose cosine similarity rather than say the typical Euclidean distance.)
Figure 2 shows that the results are as predicted by our analysis.
After the ﬁrst gradient step, some weights begin to cluster around the ground-truth (cid:80)
j ∈
A M j (or − (cid:80)
j ∈
A M j due to the a i in the gradient update which can be positive or negative).
After the second step, the weights get improved and well-aligned with the ground-truth (with cosine similarities > 0 . 99 ).
Furthermore, if a classiﬁer is trained on the features after the ﬁrst step, the test accuracy is about 52% ; if the same is done after the second step, the test accuracy is about 100% .
This demonstrates while some effective features emerge in the ﬁrst step, they need to be improved in the second step to get accurate prediction.
Real Data: Feature Learning in Networks.
We perform experiments on MNIST (LeCun et al., 1998; Deng, 2012), CIFAR10 (Krizhevsky, 2012), and SVHN (Netzer et al., 2011).
On MNIST, we train a two-layer network with m = 50 on the subset with labels 0/1 and visualize the neurons’ weights as in the simulation.
Figure 3 shows a similar feature learning phenomenon: effective features emerge after a few steps and then get improved to form two clusters.
Similar results are observed on other datasets.
These suggest the insights obtained in our analysis are also applicable to the real data.
[Figure]
Real Data: The Effect of Input Structure.
Since we cannot directly manipulate the input distribution of real data, we perform controlled experiments by injecting different inputs.
For labeled dataset
L and injected input U , we ﬁrst train a teacher network ﬁtting L , then use the teacher network to give labels on a mixture of inputs from L and U , and ﬁnally train a student network on this new dataset M consisting of the mixed inputs and the teacher network’s labels.
Checking the student’ performance on different parts of M and comparing to those by directly training the student on the original data L can reveal the impact of changing the input structure.
We use MNIST, CIFAR10, or
SVHN as L , and use Gaussian or images in Tiny ImageNet (Le & Yang, 2015) as U .
The networks for MNIST are two-layer with m = 9 , and those for CIFAR10/SVHN are ResNet-18 convolutional neural networks (He et al., 2016).
Figure 4 shows the results on an equal mixture of data and Gaussian.
It presents the test accuracy of the student on the original data part, the Gaussian part, and the whole mixture.
For example, on
CIFAR10, the network learns well over the CIFAR10 part (with accuracy similar to directly training on the original data) but learns slower with worse accuracy on the Gaussian part.
Furthermore, the accuracy on the whole mixture is lower than that of training on the original CIFAR10.
This shows that the input structure indeed has a signiﬁcant impact on the learning.
While MNIST+Gaussian shows a less signiﬁcant trend (possibly because the tasks are simpler), the other datasets show similar signiﬁcant trends as CIFAR10+Gaussian (the results using Tiny ImageNet are in the appendix).
E THICS S TATEMENT
Our paper is mostly theoretical in nature and thus we foresee no immediate negative ethical impact.
We are of the opinion that our theoretical framework may lead to better understanding and inspire development of improved network learning methods, which may have a positive impact in practice.
In addition to the theoretical machine learning community, we perceive that our conceptual message that the input structure is crucial for the network learning’s performance can be beneﬁcial to engineeringinclined machine learning researchers.
R EPRODUCIBILITY S TATEMENT
For theoretical results in the Section a complete proof is provided in the Appendix Section B-D.
The theoretical results and complete proofs for a setting more general than that in the main text are provided in the Appendix Section F.
For experiments in the Section 6, complete details and experimental results are provided in the Appendix Section E.
The source code with explanations and comments is provided in the supplementary material.
Appendix
Section A presents more technical discussion on related work.
Section B-D provides the complete proofs for our results in the main text.
Section E provides the complete details and experimental results for our experiments.
Finally, Section F provides the theoretical results and complete proofs for a setting more general than that in the main text, allowing incoherent dictionaries, unbalanced classes, and Gaussian noise in the data.
A M ORE T ECHNICAL D ISCUSSION ON R ELATED W ORK
Advantage of Neural Networks over Linear Models on Fixed Features.
A recent line of work has turned to show learning settings where network learning provably has advantage over linear models on ﬁxed features; see the nice summary in Malach et al.
Here we highlight the results and focuses of the existing related studies and discuss the differences from ours.
Yehudai & Shamir (2019) shows that the random feature method fails to learn even a single ReLU neuron on Gaussian inputs unless its size is exponentially large in dimension.
This points out the limitation of the random feature method (belonging to the ﬁxed feature approach) but does not consider feature learning in networks.
Some studies show that a single ReLU neuron can be learnt by gradient descent (Yehudai & Ohad, 2020; Diakonikolas et al., 2020;
Frei et al., 2020).
The analysis typically involves feature learning.
However, their focus is different: they do not show the advantage over ﬁxed feature methods and do not consider the effect of the input structures.
Zhou et al.
(2021) shows that in a special teacher-student setting, the student network will do exact local convergence in a surprising way that all student neurons will converge to one of the teacher neurons.
The work does not consider the effect of the input structure nor the advantage over ﬁxed features.
Dou & Liang (2020) explains the advantage of network learning by constructing adaptive Reproducing Kernel Hilbert Space (RKHS) indexed by the training process of the neural network, and shows that adaptive RKHS beneﬁts from a smaller function space containing the residue comparing to RKHS.
The work shows the statistical advantage of networks over data-independent kernels, but does not consider the optimization for learning the network.
Ghorbani et al.
(2020) considers data generated from a hidden vector with two subsets of variables, each uniformly distributed in a high-dimensional sphere (with a different radius), while the label is determined by only the ﬁrst subset of variables.
It shows the existence of good neural networks that can overcome the curse of dimensionality by representing the best low-dimensional hidden structure.
However, it studies the approximation power of neural networks rather than the learning, i.e., it does not show how to learn the good network.
Fang et al.
(2019) argues that in the inﬁnite width limit, a two-layer neural network will learn a nearly optimal feature representation in the distribution sense, thanks to the convexity of the limit problem.
It is unclear how this result helps to understand the feature learning procedure for practical networks, which is usually a non-convex process.
Chen et al.
(2020a) considers a ﬁxed, randomly initialized neural network as a representation function fed into another trainable network which is the quadratic Taylor model of a wide two-layer network.
It shows that learning over the random representation can achieve improved sample complexities compared to learning over the raw data.
However, the representation considered is not learned, which is different from our focus on feature learning.
Allen-Zhu & Li (2020a) considers Gaussian inputs with labels given by a multiple-layer network with quadratic activations and skip connections (with the assumption of information gap on the weights), and studies training a deep network with quadratic activation.
It shows that the trained network can learn proper representations and obtain small errors while no polynomial ﬁxed feature methods can.
On the other hand, it does not focus on the inﬂuence of input structure on feature learning: note that its input distribution contains no information about the “ground-truth” features in the target network.
It also points out that the learned features get improved during training: higher-level layers will help lower-level layers to improve by backpropagating correction signals.
Our analysis also shows feature improvement which however is by signals from the input distribution.
Allen-Zhu & Li (2019) considers PAC learning with labels given by a depth-2 ResNet, and studies training an overparameterized depth-2 ResNet (using uniform inputs over Boolean hypercube as an example).
It shows the trained network can obtain small errors while no polynomial kernel methods can obtain as good errors.
Similar to [1], it doesn’t focus on the inﬂuence of input structure on feature learning or the advantage of networks.
Allen-Zhu & Li (2020c) studies how ensemble of deep learning models can improve test accuracy and how the ensemble can be distilled into a single model.
It develops a theory which assumes the data has multi-view structure and shows that the ensemble of independently trained networks can provably improve test accuracy and the ensemble can also be provably distilled into a single model.
The analysis also relies on showing that the data structure can help the ensemble and the distillation.
On the other hand, their focus is on ensembles and is quite different from ours: the analysis is on showing the multi-view input structure allows the ensembles of networks to improve over single ones and ensembles of ﬁxed feature mappings do not have improvement.
While our focus is on supervisedly learning one single network that outperforms the ﬁxed feature approaches.
Daniely & Malach (2020) considers the task of learning sparse parities with two-layer networks, and the analysis suggests that the ability to learn the label-correlated features also seems to be critical towards the success of neural networks, although the authors did not explore much in this direction.
Malach et al.
(2021) also considers similar learning problems but with speciﬁcally designed models for the problems.
The learning problems considered in Daniely & Malach (2020); Malach et al.
(2021) have input distributions that leak information about the target labeling function, which is similar to our setting, and their analysis also shows that the ﬁrst gradient descent can learn a set of good features and later steps can learn an accurate classiﬁer on top.
However, they have not explicitly considered if the input structure is crucial for the feature learning and consequently the advantage over ﬁxed features.
Our work focuses on studying if the input structures can crucially affect the advantage of networks using a learning problem motivated by practical data, while previous studies use the input structure in the analysis but do not explicitly focus on the effect.
Furthermore, there are a few more important differences.
First, their focuses are different from ours.
Daniely & Malach (2020) focuses on showing neural networks can learn targets (i.e., k -parity functions) that are inherently non-linear.
Our analysis generalizes to more general distributions, including practically motivated ones.
Malach et al.
(2021) focuses on strong separations between learning with gradient descent on differentiable models (including typical neural networks) and learning using the corresponding tangent kernels.
The analysis is on speciﬁc differentiable models, while our work is on two-layer neural networks similar to practical ones.
Second, our analysis relies on the feature improvement in the second gradient step.
This is not an artifact of the analysis but comes from our problem setup.
While in Daniely & Malach (2020) the data distribution allows some neurons to be sufﬁciently good after the ﬁrst gradient step and needs no feature improvement, our setup is more general where the data distribution may not have a similar strong benign effect and thus needs feature improvement in the second gradient step.
Sparse Coding and Subspace Data Models.
To analyze neural networks’ performance, various data models have been considered.
A practical way to model the underlying structure of data is by assuming that a set of hidden variables exists and the input data is a high dimensional projection of the hidden vector (possibly with noise).
Along this line, the classic sparse coding model has been used in existing works for analyzing networks.
Koehler & Risteski (2018) considers such a data distribution where the label is given by a linear function on the hidden sparse vector, but studies the approximation power of networks and classic polynomial methods rather than the learning.
Allen-Zhu & Li (2020b) considers similar data distributions, but studies the performance of networks under adversarial perturbations.
Another type of related data models assumes that the label is determined by a subset of hidden variables.
Ghorbani et al.
(2020) considers a hidden vector with two subsets of variables, each uniformly distributed in a high-dimensional sphere (with a different radius), while the label is determined by only the ﬁrst subset of variables.
However, Ghorbani et al.
(2020) studies the approximation power of neural networks rather than the learning.
Compared to these studies, our work assumes the input is given by a dictionary multiplied with a hidden vector (not necessarily sparse) while the label is determined by a subset of the hidden vector, as motivated by pattern recognition applications in practice.
Furthermore, we focus on the learning ability of networks instead of approximation.
B C OMPLETE P ROOFS FOR P ROVABLE G UARANTEES OF N EURAL N ETWORKS
Notations.
For a vector v and an index set I , let v
I denote the vector containing the entries of v indexed by I , and v − I denote the vector containing the entries of v with indices outside I .
By initialization, w (0) i for i ∈ [ m ] are i.i.d.
copies of the same random variable w (0) ∼ N (0 , σ 2 w I d × d ) ; similar for a (0) and b (0) .
Let q (cid:96) :
= (cid:104) w (0) , M (cid:96) (cid:105) , then (cid:104) w (0) , x (cid:105) =
(cid:104) φ, q (cid:105) .
Similarly, deﬁne q ( t ) i,(cid:96) :
= (cid:104) w ( t ) i , M (cid:96) (cid:105) .
Let σ 2 φ :=
p o (1 − p o ) / ˜ σ 2 denote the variance of φ (cid:96) for (cid:96) (cid:54)∈ A .
We also deﬁne the following sets to denote typical initialization.
For a ﬁxed δ ∈ (0 , 1) , deﬁne
[Equation]
B. E XISTENCE OF A G OOD N ETWORK
we ﬁrst show that there exists a network that can ﬁt the data distribution.
Lemma 7.
For some s, a, b ∈ R with a, b ≥ 0 , deﬁne a function δ s,a,b : R → R as
[Equation]
(8) where σ r ( z ) =
max { z, 0 } is the ReLU activation function.
Then
[Equation]
when z ≤ s − b,
[Equation]
+ ab when s − b ≤ z ≤ s, − a ( z − s ) + ab when s
≤ z ≤ s + b, when s + b ≤ z.
That is, δ s,a,b ( z ) linearly interpolates between ( s − b, 0) , ( s, ab ) , ( s + b, 0) when z ∈ [ s − b, s + b ] , and is 0 elsewhere.
Proof of Lemma 7.
This can be simply veriﬁed for the four cases of the value of z .
Lemma 8 (Restatement of Lemma 4) .
For any D ∈ F Ξ , there exists a network g ∗ ( x )
= (cid:80) ni
=1 a ∗ i σ ( (cid:104)
w ∗
i , x (cid:105) + b ∗ i ) with y =
g ∗ ( x ) for any ( x, y ) ∼ D .
Furthermore, the number of neurons n = 3( k + 1) , | a ∗ i | ≤ 32 k, 1 / (32 k ) ≤ | b ∗ i | ≤ 1 / 2 , w ∗ i = ˜
σ (cid:80)
j ∈
A M j / (4 k ) , and |(cid:104) w ∗ i , x (cid:105)
+ b ∗ i | ≤for any i ∈ [ n ] and ( x, y ) ∼ D .
Proof of Lemma 4.
Let w = ˜
σ (cid:80)
j ∈
A M j
and let µ = (cid:80)
j ∈
A E [˜ φ j ] .
We have
[Equation]
Then by Lemma 7,
[Equation]
for any ( x, y ) ∼ D .
Similarly,
[Equation]
for any ( x, y ) ∼ D .
Note that the bias terms in g ∗ 1 and g ∗ 2 have distance at least 1 / 4 , then at least one of them satisﬁes that all its bias terms have absolute value ≥ 1 / 8 .
Pick that one and denote it as g ( x )
= (cid:80) ni
=1 a i σ r
(cid:104)
w i , x (cid:105) + b i ) .
By the positive homogeneity of σ r , we have
[Equation]
[Equation]
Since for any ( x, y ) ∼ D , |(cid:104) w i , x (cid:105) / (4 k )
+ b i / (4 k ) | ≤ 1 , then
[Equation] [Equation]
where σ is the truncated ReLU.
Now we can set a ∗ i
= 4 ka i , w ∗ i =
w i / (4 k ) , b ∗ i
= b i / (4 k ) , to get our ﬁnal g ∗ .
B. I NITIALIZATION
We ﬁrst show that with high probability, the weights are in positions.
Lemma 9.
For any δ ∈ (0 , 1) , with probability at least 1 − δ − 2 exp ( − Θ( D − k )) over w (0) ,
[Equation]
With probability at least 1 − δ over b (0) , | b (0) |
[Equation]
b (cid:112) 2 log(1 /δ ) .
With probability at least 1 − δ over a (0) , | a (0) | ≤ σ a (cid:112) 2 log(1 /δ ) .
Proof of Lemma 9.
From q ∼ N (0 , σ 2 w I d × d ) , we have:
• With probability ≥ 1 − δ/ 2 , max (cid:96) | q (cid:96) | ≤ (cid:113) 2 σ 2 w log
Dδ , and • For any subset S ⊆
[ D ] , with probability ≥ 1 − 2 exp ( − Θ( | S | )) , (cid:107) q S (cid:107) 22 ∈ (cid:16) | S | σ 2 w 2 , 3 | S | σ
[Equation]
[Equation]
Similar for b (0) and a (0) .
The lemma then follows.
Lemma 10.
We have:
• With probability ≥ 1 − δ − 2 m exp( − Θ( D − k )) over w (0)
i ’s, for all i ∈ [2 m ] , w (0) i ∈
G w ( δ ) .
• With probability ≥ 1 − δ over b (0)
i ’s, for all i ∈ [2 m ] , b (0) i ∈
G b ( δ ) .
• With probability ≥ 1 − δ over a (0) i ’s, for all i ∈ [2 m ] , a (0)
i
[Equation]
Proof of Lemma 10.
This follows from Lemma 9 by union bound.
The following lemma about the typical w (0) i ’s will be useful for later analysis.
Lemma 11.
Fix δ ∈ (0 , 1) .
For any w (0) i ∈
G w ( δ ) , we have
[Equation]
Consequently, when p o = Ω( k 2 /D ) and k = Ω(log 2 ( Dm/δ )) ,
[Equation]
Proof of Lemma 11.
Note that for (cid:96) (cid:54)∈ A , E
[ φ (cid:96) ]
= 0 , E
[ φ 2 (cid:96) ]
= σ 2 φ , and E [ | φ (cid:96) | 3 ]
=
Θ( σ 2 φ / ˜ σ ) .
Then the statement follows from Berry-Esseen Theorem.
B. S OME A UXILIARY L EMMAS
The expression of the gradients will be used frequently.
Lemma 12.
[Equation]
Proof of Lemma 12.
It follows from straightforward calculation.
We now show that a small subset of the entries in φ, q does not affect the probability distribution of (cid:104) φ, q (cid:105) much.
Lemma 13.
Suppose ν ∼ N (0 , σ 2 ) .
For any B ⊇ A and any b :
[Equation] [Equation]
Similarly,
[Equation]
Proof of Lemma 13.
Note that for (cid:96) (cid:54)∈ A , E
[ φ (cid:96) ]
= 0 , E
[ φ 2 (cid:96) ]
= σ 2 φ , and E [ | φ (cid:96) | 3 ]
=
Θ( σ 2 φ / ˜ σ ) .
Let t = |(cid:104) φ B , q B (cid:105)| .
Then by the Berry-Esseen Theorem,
[Equation]
The second statement follows from a similar argument.
We also have the following auxiliary lemma for later calculations.
Lemma 14.
[Equation]
Proof of Lemma 14.
[Equation]
B. F EATURE E MERGENCE : F IRST G RADIENT S TEP
We will show that w.h.p.
over the initialization, after the ﬁrst gradient step, there are neurons that represent good features.
We begin with analyzing the gradients.
Lemma 15 (Full version of Lemma 5) .
Fix δ ∈ (0 , 1) and suppose w (0) i ∈
G w ( δ ) , b (
[Equation]
∈
[Equation]
b ( δ ) for all i ∈ [2 m ] .
Let
[Equation]
If p o = Ω( k 2 /D ) , k = Ω(log 2 ( Dm/δ )) , and σ (1) ξ < 1 /k , then
[Equation]
where T j satisﬁes:
• if j ∈ A , then | T j − βγ/ ˜ σ | ≤ O ( (cid:15) e / ˜ σ ) , where β ∈ [Ω(1) , 1] and depends only on w (0) i , b (0) i ; • if j (cid:54)∈ A , then | T j | ≤ O ( σ 2 φ (cid:15)
e ˜ σ ) .
Proof of Lemma 15.
Consider one neuron index i and omit the subscript i in the parameters.
Since the unbiased initialization leads to g (0) ( x ; ξ (1) )
= 0 , we have
[Equation] [Equation]
[Equation]
First, consider j ∈ A .
[Equation]
Let
[Equation] [Equation]
We have
[Equation] [Equation]
Then by Lemma 13,
[Equation]
Note that (cid:80) (cid:96) (cid:54)∈ A Var ( φ (cid:96) q (cid:96) ) =
Θ( σ 2 φ σ
[Equation]
( D − k ))
=
Θ( σ 2 w ) , and | φ (cid:96) | ≤ 1˜ σ .
If p o = Ω( k 3 /D ) then apply Bennett’s inequality in the small-deviation regime; p o =
O ( k 3 /D )
and p o = Ω( k 2 /D )
then apply Bennett’s inequality in the large-deviation regime.
In both cases we have:
[Equation]
We also have:
[Equation]
Therefore,
[Equation]
where the last step follows from the assumption on σ w and k .
A similar argument gives:
[Equation]
Then we have
[Equation]
where the last step is from Lemma 14.
Furthermore,
[Equation]
By Lemma 11, the assumption on p o , and (63), we have
[Equation]
This leads to
[Equation]
By Lemma 14, E φ A { yφ j } = γ/ ˜ σ .
Therefore,
[Equation]
Now, consider j (cid:54)∈ A .
Let B denote A ∪ { j } .
[Equation]
Let
[Equation]
Similar as above, we have | E ξ (1) ( I b − I (cid:48) b ) | ≤ O ( (cid:15) e ) by Lemma 13.
Then by Lemma 14,
[Equation]
Furthermore,
[Equation]
Therefore,
[Equation]
Lemma 16.
Under the same assumptions as in Lemma 15,
[Equation]
where | T b | ≤ O ( (cid:15) e ) .
Proof of Lemma 16.
Consider one neuron index i and omit the subscript i in the parameters.
Since the unbiased initialization leads to g (0) ( x ; ξ (1) )
= 0 , we have
[Equation] [Equation]
[Equation]
Similar to the proof in Lemma
[Equation]
Then
[Equation]
[Equation] ≤ O (
[Equation]
{|
[Equation]
|}
[Equation]
97) ≤ O ( (cid:15) e ) .
[Equation]
Also,
[Equation]
Therefore, | T b | ≤ O ( (cid:15) e ) .
Lemma 17.
We have
[Equation]
where | T a | ≤ O (max (cid:96) q (0) i,(cid:96) ) .
So if w (0) i ∈ G ( δ ) , | T a | ≤ O ( σ w (cid:112) log( Dm/δ )) .
Proof of Lemma 17.
Consider one neuron index i and omit the subscript i in the parameters.
Since the unbiased initialization leads to g (0) ( x ; ξ (1) )
= 0 , we have
[Equation]
Let φ (cid:48)
A be an independent copy of φ A , φ (cid:48) be the vector obtained by replacing in φ the entries φ A with φ (cid:48)
A
[Equation]
and let x (cid:48)
=
Mφ (cid:48) and its label is y (cid:48) .
Then
[Equation]
Since σ is 1-Lipschitz,
[Equation]
With the bounds on the gradient
[Equation]
we now summarize the results for the weights after the ﬁrst gradient step.
Lemma 18.
Set λ (1) w = 1 / (2 η (1) ) , λ (1) a = λ (1) b = 0 , σ (1) ξ = 1 /k 3 / 2 .
Fix δ ∈ (0 , 1) and suppose w (0) i ∈
G w ( δ ) , b (0) i ∈ G b ( δ ) for all i ∈ [2 m ] .
If p o = Ω( k 2 /D ) , k = Ω(log 2 ( Dm/δ )) , then for all i ∈ [ m ] , w (1) i = (cid:80) D(cid:96) =
1 q (1) i,(cid:96) M (cid:96) satisfying
• if (cid:96) ∈ A , then | q (1) i,(cid:96) − η (1) a (0) i βγ/ ˜ σ | ≤ O (cid:18) | η (1) a (0) i | (cid:15) e ˜ σ (cid:19) , where β ∈ [Ω(1) , 1] and depends only on w (0) i , b (0) i ; • if (cid:96) (cid:54)∈ A , then | q (1) i,(cid:96) | ≤ O (cid:16) σ 2 φ | η (1) a (0) i | (cid:15) e ˜ σ (cid:17) ;
and
• b (1) i =
b (0) i + η (1) a (0)
i T b where | T b | =
O ( (cid:15) e ) ; • a (1) i =
a (0) i + η (1) T a where | T a | =
O ( σ w (cid:112) log( Dm/δ )) .
Proof of Lemma 18.
This follows from Lemma 10 and Lemma 15-17.
B. F EATURE I MPROVEMENT : S ECOND G RADIENT S TEP
We ﬁrst show that with properly set η (1) , for most x , | g (1) ( x ; σ (2) ξ ) | < 1 and thus yg (1) ( x ; σ (2) ξ ) < 1 .
Lemma 19.
Fix δ ∈ (0 , 1) and suppose w (0) i ∈
G w ( δ ) , b (0) i ∈ G b ( δ ) , a (0) i ∈ G a ( δ ) for all i ∈ [2 m ] .
If p o = Ω( k 2 /D ) , k = Ω(log 2 ( Dm/δ )) , σ a ≤ ˜ σ 2 / ( γk 2 ) , η (1) =
O (cid:16) γ kmσ a (cid:17) , and σ (2) ξ ≤ 1 /k , then with probability ≥ 1 − exp( − Θ( k )) over ( x, y ) , we have yg (1) ( x ; σ (2) ξ ) < 1 .
Furthermore, for any i ∈ [2 m ] , (cid:12)(cid:12)(cid:12) (cid:104) w (1) i , x (cid:105) (cid:12)(cid:12)(cid:12) = (cid:12)(cid:12)(cid:12) (cid:104) q (1) i , φ (cid:105) (cid:12)(cid:12)(cid:12) =
O ( η (1) ˜ σ/γ ) , (cid:12)(cid:12)(cid:12) (cid:104) ( q (1) i ) − A , φ − A (cid:105) (cid:12)(cid:12)(cid:12) =
O ( η (1) ˜ σ/γ ) , and (cid:12)(cid:12)(cid:12) b (1) i − b (1) m
+ i (cid:12)(cid:12)(cid:12) =
O ( | η (1) a (0) i | (cid:15) e ) .
Proof of Lemma 19.
Note that w (0) i = w (0) m
+ i , b (0) i
= b (0) m
+ i , and
a (0) i = a (0) m
+ i .
Then the gradient for w m
+ i is the negation of that for w m
+ i , the gradient for b m
+ i is the negation of that for b m + i , and the gradient for a m
+ i is the same as that for a m
+ i .
With probability ≥ 1 − exp( − Θ(max { 2 p o ( D − k ) , k } )) , among all j (cid:54)∈ A , we have that at most 2 p o
( D − k )
+ k of φ j are (1 − p o ) /
˜ σ , while the others are − p o / ˜ σ .
For data points with φ satisfying this, we have:
[Equation] [Equation]
Then we have
[Equation]
+ i (cid:12)(cid:12)(cid:12) (cid:16)(cid:12)(cid:12)(cid:12) (cid:104) w (1) i − w (1) m
+ i , x (cid:105) (cid:12)(cid:12)(cid:12) + (cid:12)(cid:12)(cid:12) b (1) i − b (1) m
+ i (cid:12)(cid:12)(cid:12)(cid:17) .
We have | T a |
=
O ( σ w (cid:112) log( Dm/δ )) , and (cid:12)(cid:12)(cid:12) (cid:104) w (1) i , x (cid:105) (cid:12)(cid:12)(cid:12) ≤ O ( | η (1) a (0) i | ) ( βγ/ ˜ σ + (cid:15) e / ˜ σ ) k
˜ σ (128) + O ( | η (1) a (0) i | σ 2 φ (cid:15)
e ˜ σ ) ((2 p o ( D − k ) + k )(1 − p o ) /
˜ σ
+ p o D/ ˜ σ ) (129) ≤ O ( | η (1) a (0) i | ) (cid:0) kγ/
˜
σ 2 + (cid:15) e k/ ˜ σ 2 + ( k + p o D ) σ 2 φ (cid:15) e (cid:1) (130) ≤ O ( η (1) (1 + p o ˜ σ ) /γ ) .
(131) (cid:12)(cid:12)(cid:12) b (1) i (cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12)(cid:12) b (0) i (cid:12)(cid:12)(cid:12) + (cid:12)(cid:12)(cid:12) η (1) a (0)
i T b (cid:12)(cid:12)(cid:12)
[Equation]
[Equation] [Equation]
Then we have
[Equation] [Equation]
[Equation]
Then (cid:12)(cid:12)(cid:12) yg (1) ( x ; σ (2) ξ ) (cid:12)(cid:12)(cid:12) < 1 .
Finally, the statement on (cid:12)(cid:12)(cid:12) (cid:104) ( q (1) i ) − A , φ − A (cid:105) (cid:12)(cid:12)(cid:12) follows from a similar calculation on (cid:12)(cid:12) (cid:12) (cid:104) w (1) i , x (cid:105) (cid:12)(cid:12)
[Equation]
We are now ready to analyze the gradients in the second gradient step.
Lemma 20 (Full version of Lemma 6) .
Fix δ ∈ (0 , 1) and suppose w (0) i ∈
G w ( δ ) , b (0) i ∈
G b ( δ ) , a (0) i ∈ G a ( δ ) for all i ∈ [2 m ] .
Let (cid:15) e2 :
= O (cid:18) η (1) | a (0)
i | k
( γ + (cid:15) e ) ˜ σ 2 σ (2) ξ (cid:19) + exp( − Θ( k )) .
If
[Equation]
then
[Equation]
where T j satisﬁes:
if j ∈ A , then | T
[Equation] [Equation]
e2 /
[Equation]
+
[Equation] [Equation]
2) ξ
[Equation]
a
[Equation]
) i | (cid:15) e / (˜ σσ (2) ξ )) , where β ∈ [Ω(1) , 1] and depends only on w (0) i , b (0) i ; • if j (cid:54)∈ A , then | T j | ≤ 1˜ σ exp( − Θ( k ))
+ O ( σ 2 φ (cid:15) e2 ˜ σ ) .
Proof of Lemma 20.
Consider one neuron index i and omit the subscript i in the parameters.
By
Lemma 19, Pr[ yg (1) ( x ; ξ (2) ) > 1] ≤ exp( − Θ( k )) .
Let I x
= I [ yg (1) ( x ; ξ (2) ) ≤ 1] .
[Equation] [Equation]
Let T j 1 :=
E ( x,y ) ∼D ,ξ
[Equation]
[Equation]
y I [ (cid:104) w (1) , x (cid:105)
+ b (1) + ξ (2) ∈ (
[Equation]
1)] φ j (cid:9) .
We have
[Equation] [Equation]
So it is sufﬁcient to bound T j 1 .
For simplicity, we use q as a shorthand for q (1) i .
First, consider j ∈ A .
[Equation]
Let
[Equation] [Equation]
[Equation]
By the property of the Gaussian ξ (2) , that |(cid:104) φ A , q A (cid:105)| =
O ( η (1) | a (0)
i | k
( γ + (cid:15) e ) ˜ σ) , and that |(cid:104) φ, q (cid:105)| = |(cid:104) w (1) i , x (cid:105)| =
O ( η (1) /γ ) <
O (1 /k ) and |(cid:104) φ − A , q − A (cid:105)| =
O ( η (1) /γ ) <
O (1 /k ) , we have
[Equation]
(cid:105)
[Equation]
(155) + Pr
[Equation] [Equation]
) (cid:104)
[Equation] [Equation]
+
[Equation]
(1) + ξ (2)
[Equation]
1 (cid:105)
+ Pr
[Equation]
(cid:104) (cid:104)
[Equation]
A ,
[Equation]
(cid:105)
+
[Equation] [Equation]
+
[Equation]
1 (cid:105) (156)
=
O
[Equation]
This leads to
[Equation]
where the last step is from Lemma 14.
Furthermore,
[Equation]
By Lemma 19, we have |(cid:104) φ − A , q − A (cid:105)| ≤ O ( η (1) ˜ σ/γ ) .
Also, | b (1) − b (
[Equation]
) | ≤ O ( η (1) | a (0) i | (cid:15) e ) .
By the property of ξ (2) ,
[Equation]
On the other hand
[Equation]
and β only depends on b (0) .
By Lemma 14, E φ A { yφ j } = γ/ ˜ σ .
Therefore,
[Equation]
Now
[Equation]
consider j (cid:54)∈ A .
Let B denote A ∪ { j } .
[Equation]
Let
I
[Equation] [Equation]
Similar as above [Equation] we have | I b − I (cid:48)
b | ≤ (cid:15) e2 .
Then by Lemma 14,
[Equation] [Equation]
Furthermore,
[Equation]
Therefore
[Equation]
|
[Equation]
Lemma 21.
Under the same assumptions as in Lemma 20,
[Equation]
where | T b | ≤ exp( − Ω( k ))
+ O ( (cid:15) e2 ) .
Proof of Lemma 21.
Consider one neuron index i and omit the subscript i in the parameters.
By
Lemma 19, Pr[ yg (1) ( x ; ξ (2) ) > 1] ≤ exp( − Ω( k )) .
Let I x
= I [ yg (1) ( x ; ξ (2) ) ≤ 1] .
[Equation]
Let T b 1 :=
E ( x,y ) ∼D ,ξ (2) (cid:8) y I [ (cid:104) w (1) , x (cid:105)
+ b (1) + ξ (2) ∈
(0 , 1)] (cid:9) .
We have
[Equation] [Equation]
[Equation]
(187) ≤
[Equation] [Equation]
So it is sufﬁcient to bound T b 1 .
For simplicity, we use q as a shorthand for q (1) i .
[Equation]
Let
[Equation]
Similar as in Lemma
[Equation]
we have | I b − I (cid:48)
b | ≤ (cid:15) e2 .
Then by Lemma 14,
[Equation]
Furthermore,
[Equation]
Therefore, | T b 1 | ≤ O ( (cid:15) e2 ) and the statement follows.
Lemma 22.
Under the same assumptions as in Lemma 20,
[Equation]
where | T a | =
O ( η (1) ˜ σ/γ ) + exp( − Ω( k ))
poly( Dm ) .
Proof of Lemma 22.
Consider one neuron index i and omit the subscript i in the parameters.
By
Lemma 19, Pr[ yg (1) ( x ; ξ (2) ) > 1] ≤ exp( − Ω( k )) .
Let I x
= I [ yg (1) ( x ; ξ (2) ) ≤ 1] .
[Equation]
Let T a 1 :=
E ( x,y ) ∼D (cid:8)
y E ξ (2) σ ( (cid:104) w (1) , x (cid:105) + b (1) + ξ (2) ) (cid:9) .
We have |
[Equation]
|
[Equation] [Equation]
So it is sufﬁcient to bound T a 1 .
For simplicity, we use q as a shorthand for q (1) i .
Let φ (cid:48)
A be an independent copy of φ A , φ (cid:48) be the vector obtained by replacing in φ the entries φ A with φ (cid:48)
A , and let x (cid:48)
=
Mφ (cid:48) and its label is y (cid:48) .
Then
[Equation] [Equation]
[Equation]
where the fourth step follows from that σ is 1-Lipschitz, the third to the last step from Lemma 19, and the second to the last step from Lemma 18.
With the above lemmas about the gradients, we are now ready to show that at the end of the second step, we get a good set of features for accurate prediction.
Lemma 23.
Set
[Equation]
Fix δ ∈ (0 , O (1 /k 3 )) .
If p o = Ω( k 2 /D ) , k = Ω (cid:16) log 2 (cid:16) Dmδγ (cid:17)(cid:17) , and m ≥ max { Ω( k 4 ) , D } , then with probability at least 1 − δ over the initialization, there exist ˜ a i ’s such that ˜ g ( x ) :
= (cid:80) 2 mi =1 ˜ a i σ ( (cid:104) w (2) i , x (cid:105) + b (2) i ) satisﬁes L D (˜ g )
= 0 .
Furthermore, (cid:107) ˜ a (cid:107)
= O ( m/k ) , (cid:107) ˜
a (cid:107) ∞
=
O ( k 5 /m ) , and (cid:107) ˜ a (cid:107)
= O ( k 9 /m ) .
Finally, (cid:107) a (2) (cid:107) ∞
=
O (cid:0) 1 km 2 (cid:1) , (cid:107) w (2) i
(cid:107)
= O (˜ σ/k ) , and | b (2) i | =
O (1 /k 2 ) for all i ∈ [2 m ] .
Proof of Lemma 23.
By Lemma 4, there exists a network g ∗ ( x )
= (cid:80) 3( k +1) (cid:96) =
a ∗ (cid:96) σ ( (cid:104) w ∗ (cid:96) , x (cid:105) + b ∗ (cid:96) ) satisfying g ∗ ( x ) for all ( x, y ) ∼ D .
Furthermore, | a ∗ i | ≤ 32 k, 1 / (32 k ) ≤ | b ∗ i | ≤ 1 / 2 , w ∗ i = ˜
σ (cid:80)
j ∈
A M j / (4 k ) , and |(cid:104) w ∗ i , x (cid:105)
+ b ∗ i | ≤ 1 for any i ∈ [ n ] and ( x, y ) ∼ D .
Now we ﬁx an (cid:96) , and show that with high probability there is a neuron in g (2) that can approximate the (cid:96) -th neuron in g ∗ .
By Lemma 10, with probability 1 − 2 δ over w (0) i ’s, they are all in G w ( δ ) ; with probability 1 − δ over a (0)
i ’s, they are all in G a ( δ ) ; with probability 1 − δ over b (0)
i ’s, they are all in G b ( δ ) .
Under these events, by Lemma 18, Lemma 20 and 21, for any neuron i ∈ [2 m ] , we have
[Equation]
where
• if j ∈ A , then | T j − βγ/ ˜ σ | ≤ (cid:15) w 1 := O ( (cid:15) e2 / ˜ σ + η (1) /σ
(2) ξ + η (1) | a (0) i | (cid:15) e / (˜ σσ (2) ξ )) , where β ∈ [Ω(1) , 1] and depends only on w (0)
i , b (0) i ; • if j (cid:54)∈ A , then | T j | ≤ (cid:15) w 2 :=
1˜ σ exp( − Θ( k ))
+ O ( σ 2 φ (cid:15) e2 ˜ σ ) .
• | T b | ≤ (cid:15) b :=
1˜ σ exp( − Θ( k ))
+ O ( (cid:15) e2 ) .
Given the initialization, with probability Ω(1) over b (0) i , we have
[Equation]
Finally, since 4 k
[Equation]
b
[Equation]
˜ σ 2 ∈ [Ω( k 2 γ/ ˜ σ 2 ) , O ( k 3 γ/ ˜ σ 2 )] and depends only on w (0)
i , b (0) i , we have that for (cid:15) a = Θ(1 /k 2 ) , with probability Ω( (cid:15) a ) > δ over a (0) i ,
[Equation]
Let n a = (cid:15) a m/ 4 .
For the given value of m , by (219)-(222) we have with probability ≥ 1 − 5 δ over the initialization, for each (cid:96) there is a different set of neurons
I (cid:96) ⊆
[ m ] with | I (cid:96) | = n a and such that for each i (cid:96) ∈
I
[Equation] [Equation]
[Equation]
We also have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
We have (cid:12)
[Table]
a (1) i (cid:96) − a (0) i (cid:96) (cid:12) (cid:12)(cid:12)
=
O ( η (1) σ w (cid:112) log( Dm/δ )) , and (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
[Equation]
For the given values of parameters, we have
[Equation]
[Equation] [Equation]
[Equation] [Equation]
Therefore,
[Equation]
We also have by Lemma 18 and 21:
[Equation]
Now, construct ˜ a such that ˜ a
i
[Equation]
i(cid:96) | n a for each (cid:96) and each i (cid:96) ∈
I (cid:96) , and ˜ a i = 0 elsewhere.
Then
[Equation] [Equation]
[Table]
[Equation]
= O
[Equation]
≤ 1 .
[Equation] [Equation]
Here the second equation follows from that σ is positive-homogeneous in [0 , 1] , |(cid:104) w ∗ (cid:96) , x (cid:105) + b ∗ (cid:96) | ≤ 1 , | b (0)
i (cid:96) | / | b ∗ (cid:96) | ≤ 1 .
This guarantees y ˜ g
( x ) ≥ 1 .
Changing the scaling of δ leads to the statement.
Finally, the bounds on ˜ a follow from the above calculation.
The bound on (cid:107) a (2) (cid:107) 2 follows from
Lemma 22, and those on (cid:107) w (2) i (cid:107) 2 and (cid:107) b (2) i (cid:107) 2 follow from (219)(220) and the bounds on a (1) i and b (1) i in Lemma 18.
B. C LASSIFIER L EARNING S TAGE
Once we have a set of good features, we are now ready to prove that the later steps will learn an accurate classiﬁer.
The intuition is that the ﬁrst layer’s weights do not change much and the second layer
’s weights get updated till achieving good accuracy.
In particular, we will employ the online optimization technique from Daniely & Malach (2020).
We begin by showing that the ﬁrst layer’s weights do not change too much.
Lemma 24.
Assume the same conditions as in Lemma 23.
Suppose for t > 2 , λ ( t ) a = λ ( t ) w = λ , η ( t ) =
η for some λ, η ∈ (0 , 1) , and σ ( t ) ξ = 0 .
Then for any t > 2 and i ∈ [2 m ] ,
[Equation]
Proof of Lemma 24.
First, we bound the size of | a ( t ) i | :
[Equation]
which leads to
[Equation]
where | a (2) i | =
O (cid:0) 1 km 2 (cid:1) .
We are now to bound the change of w ( t ) i and b ( t ) i .
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
= O
[Equation] [Equation]
leading to
[Equation]
[Equation]
Note that (cid:12)(cid:12)(cid:12)
[Equation]
i (cid:12)(cid:12)(cid:12) =
O (1 /k 2 ) .
Lemma 25.
Assume the same conditions as in Lemma 24.
Let g ( t ) ˜ a ( x ) = (cid:80) mi
=1 ˜ a i σ ( (cid:104) w ( t ) i , x (cid:105) + b ( t ) i ) .
Then | (cid:96)
[Equation]
Proof of Lemma 25.
It follows from that
[Equation]
and Lemma
[Equation]
B. P ROOF OF T HEOREM
Based on the above lemmas
[Equation]
following the same argument as in the proof of Theorem 2 in Daniely &
Malach (2020), we get our main theorem
[Equation]
Theorem 26 (Full version of Theorem 1) .
Set
[Equation]
For any δ ∈ (0 , 1) , if p o = Ω( k 2 /D ) , k = Ω (cid:16) log 2 (cid:16) Dδγ (cid:17)(cid:17) , max { Ω( k 4 ) , D } ≤ m ≤ poly ( D ) , then we have for any D ∈ F Ξ , with probability at least 1 − δ , there exists t ∈
[ T ] such that
[Equation]
Consequently, for any (cid:15) ∈ (0 , 1) , if T = m 4 / 3 , and max { Ω( k 12 /(cid:15) 3 / 2 ) , D } ≤ m ≤ poly ( D ) , then
[Equation]
Proof of Theorem 1.
Consider ˜ L D ( g ( t ) )
=
E [ (cid:96) ( g ( t ) , y )] + λ ( t ) a (cid:107) a ( t ) (cid:107) 22 .
Note that the gradient update using ˜ L D ( g ( t ) ) is the same as the update in our learning algorithm.
Then by Theorem 27, Lemma 23, and Lemma 25,
[Equation]
The statement follows from that 0-1 classiﬁcation error is bounded by the hinge-loss.
Theorem 27 (Theorem 13 in Daniely & Malach (2020)) .
Fix some η , and let f 1 , . .
, f T be some sequence of convex functions.
Fix some θ
[Equation]
and assume we update θ t
+1 =
θ t − η ∇ f t ( θ t ) .
Then for every θ ∗ the following holds:
[Equation]
C L OWER B OUND FOR L INEAR M ODELS ON F IXED F EATURE M APPINGS
Theorem 28 (Restatement of Theorem 2) .
Suppose Ψ is a data-independent feature mapping of dimension N with bounded features, i.e., Ψ : X → [ − 1 , 1] N .
Deﬁne for B > 0 :
[Equation]
Then, if 3 < k ≤ D/ 16 and k is odd, then there exists D ∈ F Ξ such that all h ∈ H B have hinge-loss at least p o (cid:16) 1 − √ 2 NB 2 k (cid:17) .
Proof of Theorem 2.
We ﬁrst show that F Ξ contains some distributions that are essentially sparse parity learning problems, and then we invoke the lower bound result from existing work for such problems.
Consider D deﬁned as follows.
• Let P =
{ i ∈ [ k ] : i is odd } .
That is, if there are odd numbers of 1 ’s in ˜ φ A , then y
= +1 .
Let D (0)˜ φ be a distribution where all entries ˜
φ j are i.i.d.
with Pr[˜ φ j = 0] =
Pr[˜ φ j = 1] = 1 / 2 .
Let D (0) be the distribution over (˜ x, y ) induced by D (0)˜ φ and the above P .
Let D (1)˜ φ be a distribution where all entries ˜
φ j for j (cid:54)∈ A are i.i.d.
with Pr[˜ φ j = 1] =
p o / (2 − 2 p o ) , while Pr[˜ φ A = (0 , 0 , . . . , 0)]
=
Pr[˜
φ A = (1 , 1 , . . . , 1)]
= 1 / 2 .
Let D (1) be the distribution over (˜ x, y ) induced by D (1)˜ φ and the above P .
Let D mix A =
p o D (0) + (1 − p
[Equation]
) D (1) .
It can be veriﬁed that such distributions are included in F Ξ for γ = Θ(1) .
Assume for contradiction that for all D ∈ F Ξ , there exists h ∗ ∈ H B such that h = (cid:104) Ψ , w ∗ (cid:105) loss smaller than p o (cid:16) 1 − √ 2 NB 2 k (cid:17) .
Then for all the distributions D mix A deﬁned above, we have
[Equation]
Now let D z be a distribution over z ∈ {− 1 , +1 } D with i.i.d.
entries z j and Pr[ z j = − 1] =
Pr[ z j = +1] = 1 / 2 .
Let f A ( z )
=
(cid:81)
j ∈
A z j be the k -sparse parity functions.
Let Ψ (cid:48)
( z ) =
Ψ( M ( z + 1) / 2) .
Then we have h (cid:48)
( z ) =
(cid:104) Ψ (cid:48)
( z ) , w ∗ (cid:105) such that for all A ,
[Equation]
This is contradictory to Theorem 29.
The following theorem is implicit in the proof in Theorem 1 in Daniely & Malach (2020).
Theorem 29.
For a subset A ⊆
[ D ] of size k , let
the distribution D A over ( z, y ) deﬁned as follows:
z is uniform over {± 1 } D and y = (cid:81) i ∈
A z i .
Fix some Ψ : {± 1 } D → [ − 1 , +1] N , and deﬁne:
[Equation]
If k is odd and k ≤ D/ 16 , then there exists some A such that
[Equation]
D L OWER B OUND FOR L EARNING WITHOUT I NPUT S TRUCTURE
First recall the Statistical Query model (Kearns, 1998).
In this model, the learning algorithm can only receive information about the data through statistical queries.
A statistical query is speciﬁed by some property predicate Q of labeled instances, and a tolerance parameter τ ∈ [0 , 1] .
When the algorithm asks a statistical query ( Q, τ ) , it receives a response ˆ P Q ∈
[ P Q − τ, P Q + τ ] , where
P Q =
Pr[ Q ( x, y ) is true ] .
Q is also required to be polynomially computable, i.e., for any ( x, y )
Q ( x, y ) can be computed in polynomial time.
Notice that a statistical query can be simulated by empirical average of a large random sample of data of size roughly O (1 /τ 2 ) to assure the tolerance τ with high probability.
Blum et al.
(1994) introduces the notion of Statistical Query dimension, which is convenient for our purpose.
Deﬁnition 30 (Deﬁnition 2 in Blum et al.
For concept class C and distribution D , the statistical query dimension SQ-DIM ( C, D ) is the largest number d such that C contains d concepts c 1 , . .
, c d that are nearly pairwise uncorrelated: speciﬁcally, for all i (cid:54) =
j ,
[Equation]
Theorem 31 (Theorem 12 in Blum et al.
In order to learn C to error less than 1 / 2 − 1 /din the Statistical Query model, where d = SQ-DIM ( C, D ) , either the number of queries or 1 /τ must be at least 12 d 1 / 3 .
We now use the above tools to prove our lower bound.
Theorem 32 (Restatement of Theorem 3) .
For any algorithm in the Statistical Query model that can learn over F Ξ 0 to error less
than 12 − 1 ( Dk ) 3 , either the number of queries or 1 /τ must be at least 12 (cid:0) Dk (cid:1) 1 / 3 .
Proof of Theorem 3.
Consider the following concept class and marginal distribution:
Let D be the distribution over ˜ x , given by ˜ x
= M ˜ φ and ˜ φ j are i.i
d. with Pr[˜ φ j = 0] =
Pr[˜ φ j = 1] = 1 / 2 .
• Let C be the class of functions y =
g A (˜ φ ) =
I [ (cid:80) j (1 − ˜ φ j ) is odd ] for different A ⊆
[ D ] .
The distributions over (˜ x, y ) induced by ( C, D ) are a subset of F Ξ 0 .
It is then sufﬁcient to show that
SQ-DIM ( C, D ) ≥ (cid:0)
Dk (cid:1) .
It is easy to see that C are essentially the sparse parity functions: if z j = 2˜ φ j − 1 ,
then g A (˜ φ ) =
(cid:81)
j ∈
A z j .
This then implies that the g A ’s are uncorrelated, so SQ-DIM ( C, D ) ≥ (cid:0) Dk (cid:1) .
E C OMPLETE E XPERIMENTAL R ESULTS
Our experiments mainly focus on feature learning and the effect of the input structure.
We ﬁrst perform simulations on our learning problems to (1) verify our main theorems on the beneﬁt of feature learning and the effect of input structure (2) verify our analysis of feature learning in networks.
We then check if our insights carry over to real data: (3) whether similar feature learning is presented in real network/data; (4) whether damaging the input structure lowers the performance.
The results are consistent with our analysis and provide positive support for the theory.
The experiments were ran 5 times with different random seeds, and the average results (accuracy) are reported.
The standard deviations of the results are smaller than 0.5% and thus we do not present them for clarity.
The hardware speciﬁcations are 4 Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz, 16 GB RAM, and one NVIDIA GPU GTX1080.
E. S IMULATION
We train a two-layer network following our learning process.
We use two ﬁxed feature methods: the
NTK (Fang et al., 2021) and random feature (RF) methods based on the same network and random initialization as the network learning.
More precisely, in the NTK method, we randomly initialize the network and take its NTK and learn a classiﬁer on it.
In the RF method, we freeze the ﬁrst layer of the network, and train the second layer (on the random features given by the frozen neurons).
The training step number is the same as that in network learning.
We also test these three methods on the data distribution with input structure removed (i.e., F Ξ 0 in Theorem 3).
For comparison, we take the representation of our two-layer network at step one/step two, named One Step/Two Step (ﬁx the weight of the 1st layer after the ﬁrst step/second step to train the weight of the second layer), and train the best classiﬁers on top of them.
Recall that our analysis is on the directions of the weights without considering their scaling , and thus it is important to choose cosine similarity rather than the typical (cid:96) 2 distance.
Thus, we use metric Cos
Similarity max { i ∈ [2 m ] } cos( w i , (cid:80)
j ∈
A M j ) in our tables, and use Multidimensional Scaling to plot the weights distribution.
The simulation dataset size is 50000.
During training, the batch size is 1000, while for the ﬁrst two steps we use the approximate full gradient (batch size is 50000).
Each step is corresponding to one weights update.
E.1. P ARITY L ABELING
Setting.
We generate data according to the parity function data distributions used in our proof of the lower bound for ﬁxed features (Theorem 2), with d = 500 , D = 100 , k = 5 , p o = 1 / 2 , with a randomly sampled A .
More precisely, we consider D deﬁned as follows.
• Let P =
{ i ∈ [ k ] : i is odd } .
That is, if there are odd numbers of 1 ’s in ˜ φ A , then y
= +1 .
Let D (0)˜ φ be a distribution where all entries ˜
φ j are i
i
d. with Pr[˜ φ j = 0] =
Pr[˜ φ j = 1] = 1 / 2 .
Let D (0) be the distribution over (˜ x, y ) induced by D (0)˜ φ and the above P .
Let D (1)˜ φ be a distribution where all entries ˜
φ j for j (cid:54)∈ A are i
i
d. with Pr[˜ φ j = 1] =
p o / (2 − 2 p o ) , while Pr[˜ φ A = (0 , 0 , . . . , 0)]
=
Pr[˜
φ A = (1 , 1 , . . . , 1)]
= 1 / 2 .
Let D (1) be the distribution over (˜ x, y ) induced by D (1)˜ φ and the above P .
Let D mix A = p o D (0) + (1 − p
o ) D (1) .
The network and the training follow Section 3, where the network size is m =
300 and the training time T = 600 steps.
Veriﬁcation of the Main Results.
Figure 5 shows that the results are consistent with our analysis.
Network learning gets high test accuracy while the two ﬁxed feature methods get signiﬁcantly lower accuracy.
Furthermore, when the input structure is removed, all three methods get test accuracy similar to random guessing.
[Table]
Feature Learning in Networks.
Figure 6 shows that the results are as predicted by our analysis.
After the ﬁrst gradient step, some weights begin to cluster around the ground-truth (cid:80)
j ∈
A M j (or − (cid:80)
j ∈
A M j due to we have a i in the gradient update which can be positive or negative).
After the second step the weights get improved and well-aligned with the ground-truth (with cosine similarity > 0 . 99 ).
Table 1 shows the results for different methods.
Recall that the Cos Similarity metric is max { i ∈ [2 m ] } cos( w i , (cid:80)
j ∈
A M j ) , which reports the cosine value of the closest one.
One Step refers to the method where we take the neurons after one gradient step, freeze their weights, and train a classiﬁer on top; similar for Two Step.
One Step gets test accuracy about 52% , while Two Step gets accuracy about 100% .
This demonstrates that while some effective feature emerge in the ﬁrst step, they need to be improved in the second step for accurate prediction.
NTK, random feature, One Step all failed, while Network and Two Step can achieve 100% test accuracy.
Network w/o structure refers to training the network on data without the input structure.
It overﬁts the training dataset with 52% test accuracy.
[Table]
Setting.
We also tried interval function, where y = 1 if (cid:80) i ∈
A ˜ φ i is in the range [ t 1 , t 2 ] with t 1 =and t 2 = 30 , otherwise y = − 1 .
We use d = 500 , D = 100 , k = 30 .
The ˜ φ i ’s are independent
[Table]
and
Pr[˜ φ i = 1] = 2 / 3 for any i ∈
A
[Table]
and Pr[˜ φ i = 1] = 1 / 2 otherwise.
When the input structure is removed, we set Pr[˜ φ i = 1] = 1 / 2 for all i ’s.
The network and training again follows Section 3 with a network size m = 100 and the training time
T = 200 steps.
Veriﬁcation of the Main Results.
Figure 7 shows that network learning learns the fastest, NTK learns slower but reaches similar test accuracy, while random feature can only reach a decent but lower accuracy.
This is because for such simpler labeling functions, ﬁxed feature methods can still achieve good performance (note that the lower bound does not hold for such a case), while the performance depends on what ﬁxed features to use.
Furthermore, when the input structure is removed, the methods still get similar (or only slightly worse) performance as with input structure.
This shows that when the labeling function is simple, the help of the input structure for learning may not be needed.
In the experiments on real data, we will show that when the input structure is changed, it indeed leads to lower performance which suggests that the labeling function in practice is typically more complicated than this interval labeling setting, and the help of the input structure is signiﬁcant for learning.
Feature Learning in Networks.
Figure 8 shows the phenomenon of feature learning similar to that in the parity labeling setting.
Table 2 shows the test accuracy of six different methods.
Random feature and One Step failed, while Network, NTK and Two Step succeed showing that interval labeling setting is a simpler case than parity labeling setting.
E. R EAL D ATA : F EATURE L EARNING IN N ETWORKS
We take the subset of MNIST (Deng, 2012) with labels 0/1, CIFAR10 (Krizhevsky, 2012) with labels airplane/automobile and SVHN (Netzer et al., 2011) with labels 0/1, and train a two-layer network with m = 50 .
We use traditional weight initialization method (random Gaussian) and training method (SGD with momentum = 0 .
95 without regularization) in this section, for our purpose of investigating the training dynamics in practice.
Then we visualize the neurons’ weights following the same method in the simulation.
Figure 9,
Figure 10 and Figure 11 show a similar feature learning phenomenon: effective features emerge after a few steps, and then get improved to form clusters.
This shows the insights obtained on our learning problems are also applicable to the real data.
E. R EAL D ATA : T HE E FFECT OF I NPUT S TRUCTURE
To study the inﬂuence of the input structure, we propose to keep the labeling function unchanged, vary the input distributions, and exam the change of the loss surface and the training dynamics.
We ﬁrst describe the detailed experimental methodology, which allows us to generate data with similar labeling function but different input distributions.
Then we perform experiments on the generated datasets to investigate the change of the learning due to the change in the input distributions, and present the experimental results.
Finally, we also perform experiments to verify the intuition behind our experimental method.
E.3. E XPERIMENTAL M ETHODOLOGY
We consider the following experimental method.
Given an original dataset L = { ( x i , y i ) } ni =1 (e.g.,
CIFAR10) and an unlabeled dataset U =
{ ˜ x i } mi =1 from a proposed distribution P U (e.g., Gaussians), ﬁrst extend the labeling function of L to U , giving synthetic labels ˜
y i to ˜ x i .
Then train a neural network on the union of L and the synthetic data L U = { (˜ x i , ˜ y i ) } mi =1 .
By investigating the new training dynamics, in particular the difference on the original part L and the synthetic part L U , we can see the effect of the input structure.
The original dataset should be from real-world data, since one of our goals is to compare them with synthetic data, and identify the properties of real-world data important for the success of learning.
A natural idea is to ﬁrst learn a powerful network f ( x ) (called the teacher) on L to approximate the true labeling function, then apply f on U to generate synthetic labels, and ﬁnally train another network (called the student) on the synthetic data and original data.
However, we found that naïvely implementation of this idea fails miserably: the support of L and U can be typically different, and the powerful network learned over L can have entirely different behavior on U .
Therefore, we need to control the size of the teacher f
so that the labeling on U has similar complexity as that on L .
For our purpose, we can deﬁne the complexity of the labeling on L as the minimum size of the teacher achieving an approximation error (cid:15) for a chosen (cid:15) , if the ground-truth data distribution of L is known.
However, given only limited data, we cannot faithfully estimate the needed size of the teacher, and need to take into account the variance introduced by the ﬁnite data.
Our key idea is to use the U-shaped curve of the bias-variance trade-off and select the size of the teacher at the minimum of the U-shaped curve.
Since recent works (Belkin et al., 2019; Nakkiran et al., 2020) show that neural networks can have a double descent curve for the error v.s. model complexity, we thus plot the double descent curve, and ﬁnd the minimum in the classical regime (corresponding to the traditional U-shape curve).
Our method is designed based on the following two reasons.
First, on the U-shaped curve, the complexity of the network is still roughly controlled by that of the number of parameters.
The local minimum of the U-shaped curve is a good measurement of the complexity of the data.
If the ground-truth is much more complicated than the teacher, then increasing the teacher’s size leads to a signiﬁcant decrease in the approximation error (bias) compared to a small increase in the variance, that is, we will be on the left-hand side of the U-shaped.
In contrast, on the right-hand side of the
U-shaped, increasing the teacher’s size leads to a small decrease in the bias compared to a signiﬁcant increase in the variance.
That is, the complexity of the ground-truth is comparable to or lower than the teacher.
So the local minimum approximates the complexity of the ground-truth labeling function.
Second, the local minimum point is chosen to get the best approximation of the true labels.
This helps to maintain the labeling from the real-world data and thus helps our investigation on the input, since too drastic change in the labeling can affect the training.
We note that the method is not perfect.
First, the teacher at the local minimum of U-shape may not have very high accuracy, especially on more complicated data.
To alleviate this, we also use the teacher to give synthetic labels y (cid:48)
i to x i in L , and train the student network on L (cid:48)
= { ( x i , y (cid:48) i ) } ni =1 .
Though this introduces some differences from the original labels, it is acceptable for our purpose of studying the inputs.
Furthermore, ensuring the consistency of the labels on the original input in L and
U is important in our experiments.
Second, the measurement is an approximation due to variance.
Since only limited labeled data is available, it’s important and necessary to calibrate the measurement w.r.t.
the level of variance on the given dataset.
Method Description.
Algorithm 1 presents the details.
For a ﬁxed network architecture for the teacher f , it ﬁrst varies the network size and plots the double descent curve.
Then it selects the local minimum in the classic regime of U-shape and trains the teacher with the corresponding size.
In practice, we observed that the teacher might have unbalanced probabilities for different classes on U if its training does not take into account U .
Therefore, we propose the following heuristic regularization using x ∈ U , where λ is a regularization weight, and f ( x ) is the probabilities over classes given by the teacher:
[Equation]
Here, R 1 ( x ) guarantees that each kind of label has the same average probability to be generated, and
R 2 ( x ) pushes the probability away from uniform to avoid the case that the class probabilities for each data point converge to uniform.
Algorithm 1 Learning the teacher network to generate synthetic labels for studying the effect of the input structure
Input: teacher architecture f , labeled dataset L = { ( x i , y i ) } ni =1 , unlabeled dataset U =
{ ˜ x i } mi =1 .
Let i to be the size of f , f i to be the teacher of size i .
for i = 1 to n do
Train f i on L and let l i denote the
Plot l i v s i
Train f i t on L with a regularizer R ( x ) on U deﬁned in (290)
Output: f i t
E.
E XPERIMENTAL R ESULTS
Network models.
Here we use one-hidden-layer fully-connected networks with m hidden units and quadratic activation functions.
The network is denoted as FC( m ) .
We use ResNet( m ) , which is a
ResNet-18 convolutional neural network (He et al., 2016) with m ﬁlters in the ﬁrst residual block.
It is obtained by scaling the number of ﬁlters in each block proportionally from the standard ResNetnetwork which is ResNet(64) .
Datasets.
We use MNIST (Deng, 2012), CIFAR10 (Krizhevsky, 2012) and SVHN (Netzer et al., 2011) as L , and use Gaussian and images in Tiny ImageNet (Le & Yang, 2015) as U .
We generate the mixture data, where the fraction of the unlabeled data is denoted as α .
Setup.
We ﬁrst use Algorithm 1 on the labeled data L and the unlabeled data U to get a synthetic labeling function (the teacher network) and then use it to give synthetic labels on a mixture of inputs from L and U .
For MNIST, the teacher network learned is FC(9) , where the number of the hidden units is determined by Algorithm 1.
See empirical veriﬁcation in Figure 15.
For CIFAR10 and SVHN, the teacher networks are ResNet(5) and ResNet(2) , respectively, as determined by our method.
The student network for MNIST is FC(9) , and those for CIFAR10 and SVHN are ResNet(9) and
ResNet(8) , respectively.
Finally, we train the student networks on these new datasets with perturbed input distributions.
Figure 12 shows the results on an equal mixture of data and Gaussian.
It presents the test accuracy of the student on the original data part, the Gaussian part, and the whole mixture.
For example,
[Figure] [Figure]
[Figure]
for CIFAR10, the test accuracy on the whole mixture is lower than that of training on the original CIFAR10, showing that the input structure indeed has a signiﬁcant impact on the learning.
Furthermore, the network learns well over the CIFAR10 part (with accuracy similar to that on the original data) but learns slower with worse accuracy on the Gaussian part.
This suggests that the
CIFAR10 input structure is still helping the network to learn effective features.
While the results on
MNIST+Gaussian do not show a signiﬁcant trend (possibly because the tasks there are simpler), the results on SVHN+Gaussian show similar signiﬁcant trends as CIFAR10+Gaussian.
Figure 14 shows the results when we vary the fraction of the Gaussian data α .
We observe that the test accuracy curve on the original part and that on the synthetic part have roughly the same trend for different α as before, further verifying our insights.
Figure 13 shows the results when mixed with Tiny ImageNet data instead of Gaussians.
It shows a similar trend, while the performance on the Tiny ImageNet part is higher than that on the Gaussian [Figure] [Figure]
part. This suggests that compared to Gaussians, the Tiny ImageNet data has helpful input structures, though not as helpful as that on the original data for learning the particular labeling.
E.3. E MPIRICAL V ERIFICATION OF O UR M ETHOD
We also perform experiments to verify the intuition behind our methodology, i.e., the method gives a synthetic labeling function with roughly the same complexity on the original inputs and the injected inputs.
We ﬁrst use our method on MNIST and samples (of the same size as MNIST) from a Gaussian to get the teacher FC(9) ; the double descent curve is in Figure 15(a).
Then we train students on the Gaussian data with synthetic labels from the teacher, and plot the double descent curve for the students in Figure 15(b).
The local minimums of the two U-shapes are roughly the same, matching our reasoning.
Then we also train larger teachers and plot the double descent curve for students on
Gaussian data.
Figure 15(c)
Teacher size 50 .
Figure 15(d)
Teacher size 500 .
The local minimum of the U-shape becomes larger when the teacher gets larger, again matching our reasoning.
F P ROVABLE G UARANTEES FOR N EURAL N ETWORKS IN A M ORE G ENERAL S ETTING
This section provides the analysis in a more general setting.
We ﬁrst describe the learning problems, and then provide the proofs following similar intuitions as for the simpler settings in the main text.
F. P ROBLEM S ETUP
Let X =
R d be the input space, and Y = {± 1 } be the label space.
Suppose M ∈ R d × D is a dictionary with D elements, where each element M j can be regarded as a pattern.
We assume quite general incoherent dictionary:
(D) M is µ -incoherent, i.e., the columns of M are unit vectors, and for any i (cid:54)
=
j, |(cid:104) M i , M j (cid:105)| ≤
µ/ √ d.
Note that the setting in the main text corresponds to µ = 0 .
Let ˜ φ ∈ { 0 , 1 } D be a hidden vector that indicates the presence of each pattern, and D ˜ φ a distribution for ˜ φ .
Let A ⊆
[ D ] be a subset of size k corresponding to the class relevant patterns.
Let P ⊆
[ k ] .
We ﬁrst sample ˜
φ from D ˜ φ , and then generate the input ˜ x and the class label y from ˜ φ, A, P by:
[Equation]
(cid:26)
[Equation]
if (cid:80)
i
[Equation]
i
[Equation]
P
[Equation] − [Equation] otherwise (293) where the Gaussian noise ζ ∼ N (0 , σ 2 ζ I d × d ) is independent from ˜ φ .
Note that the setting in the main text corresponds to σ ζ = 0 .
We allow general D ˜ φ with the following assumptions:
(A1)
The patterns in A are correlated with the labels: for any i ∈ A , for v ∈ {± 1 } let γ v = E
[ y ˜ φ i | y
= v ] , then γ :=
( γ +1 + γ − 1 ) / 2 > 0 .
(A2)
The patterns outside A are independent of the patterns in A .
Note that we allow imbalanced classes.
Let p min := min(Pr[ y = − 1] , Pr[ y = +1]) .
If the classes are balanced, then the assumption (A1) implies the assumption (A1) in the main text, so the setting here is more general.
(A1) is also more general, in particular, allowing dependence between irrelevant patterns and non-identical distributions for them.
Let D ( A, P, D ˜ φ ) denote the distribution on (˜ x, y ) corresponding to some A, P, and D ˜ φ .
Given parameters Ξ =
( d, D, k, γ, p o , µ, σ ζ ) , the family F Ξ of distributions for learning is the set of all
D ( A, P, D ˜ φ ) with A ⊆
[ D ] , P ⊆
[ k ] , and D ˜ φ satisfying the above assumptions.
One special case is the mixture of two Gaussians.
Example.
Suppose M has one single column v , and y
=
+
if ˜ φ = 1 and y = − 1 otherwise.
Then the data distribution is simply a mixture of two Gaussians: ˜ x ∼ v 2 + N ( y v 2 , σ 2 ζ I d × d ) .
F.1. N EURAL N ETWORK L EARNING
Again, we will normalize the data for learning:
we ﬁrst compute x = (˜ x − E [˜ x ]) /
˜ σ where ˜ σ 2 := (cid:80) di
=1 (˜ x i − E
[˜ x i ]) 2 = (cid:80)
j ∈
[ D ] Var (˜ φ j ) + dσ 2 ζ is the variance of the data, and then train on ( x, y ) .
This is equivalent to setting φ =
(˜ φ − E [˜ φ ]) /
˜ σ and generating x = Mφ + ζ/σ ζ .
For (˜ x, y ) from D and the normalized ( x, y ) , we will simply say ( x, y ) ∼ D .
The learning will be the same as that in the main text, except the following. We will use a small σ 2 w = ˜ σ 2 / poly ( Dm ) . And we will use a weighted loss to handle the imbalanced classes in the ﬁrst two steps for feature learning, and then use the unweighted loss in the remaining steps.
Formally, the weighted loss is:
[Equation]
where the class weights α v = 1 2Pr[ y = v ] for v ∈ {± 1 } .
F. M AIN R ESULT
In this setting, we have the following theorem:
Theorem 33.
Set
[Equation]
For any δ ∈ (0 , O (1 /k 3 )) , if µ ≤ O ( √ d/D ) , σ ζ ≤ O (min { 1 / ˜ σ, ˜ σ/ √ d } ) , k = Ω (cid:16) log [Equation] Dmdδγp min (cid:17)(cid:17) , m ≥ max { Ω( k 4 ) , D, d } , then we have for any D ∈ F Ξ , with probability at least 1 − δ , there exists t ∈
[ T ] such that
[Equation]
[Equation]
Consequently, for any (cid:15) ∈ (0 , 1) , if T = m 4 / 3 , and m ≥ max { Ω( k 12 /(cid:15) 3 / 2 ) , D } , then
Pr[sign
[Equation]
g (
[Equation]
( x
[Equation] [Equation]
(cid:15)
[Equation]
[Equation]
The rest of the section is devoted to the proof of this theorem.
F. N OTATIONS
Recall some notations that we will use throughout the analysis.
For a vector v and an index set I , let v
I denote the vector containing the entries of v indexed by I , and v − I denote the vector containing the entries of v with indices outside I .
Let ρ :=
M (cid:62) M .
Then we have ρ jj = 1 for any j , and | ρ j(cid:96)
| ≤ µ/ √ d for any j (cid:54) = (cid:96) .
By initialization, w (0) i for i ∈ [ m ] are i.i.d.
copies of the same random variable w (0) ∼ N (0 , σ 2 w I d × d ) ; similar for a (0) and b (0) .
Let σ 2 φj :=
p o j (1 − p o j ) / ˜ σ 2 denote the variance of φ (cid:96) for (cid:96) (cid:54)∈ A , where p o j = Pr[˜ φ j = 1] . Let p o be the value such that with probability 1 − exp( − Ω( k )) , (cid:80) j (cid:54)∈ A ˜ φ j ≤ p o
( D − k ) for some p o ∈ [0 , 1] .
That is, p o is an upper bound on the density of ˜ φ j with high probability.
Let q (cid:96) :
= (cid:104) w (0) , M (cid:96) (cid:105) .
Similarly, deﬁne q ( t ) i,(cid:96) :
= (cid:104) w ( t ) i , M (cid:96) (cid:105) .
We also deﬁne the following sets to denote typical initialization.
For a ﬁxed δ ∈ (0 , 1) , deﬁne
G w [Equation]
[Equation] ( δ ) : = { a ∈ R : | a | ≤ σ a (cid:112) 2 log( m/δ ) } . [Equation]
[Equation]
G b ( δ ) : = { b ∈ R : | b | ≤ σ b (cid:112) 2 log( m/δ ) } . [Equation]
[Equation]
F. E XISTENCE OF A G OOD N ETWORK
We ﬁrst show that there exists a network that can ﬁt the data distribution.
Lemma 34.
Suppose kµ √ d p o D ˜ σ ≤ 116 .
For any D ∈ F Ξ , there exists a network g ∗ ( x )
= (cid:80) ni
=1 a ∗ i σ ( (cid:104)
w ∗
i , x (cid:105) + b ∗ i ) which satisﬁes
[Equation]
Furthermore, the number of neurons n = 3( k + 1) , | a ∗ i | ≤ 64 k, 1 / (64 k ) ≤ | b ∗ i | ≤ 1 / 4 , w ∗ i = ˜
σ (cid:80)
j ∈
A M j / (8 k ) , and |(cid:104) w ∗ i , x (cid:105)
+ b ∗ i | ≤ 1 for any i ∈ [ n ] and ( x, y ) ∼ D .
Consequently, if furthermore we have kµ/ √ d < 1 and σ ζ < 1 /k , then
[Equation]
Proof of Lemma 34.
Let w = ˜ σ (cid:80)
j ∈
A M j and let u = (cid:80)
j ∈
A E [˜ φ j ] .
We have
[Equation]
With probability ≥ 1 − exp( − Ω( k )) , among all j (cid:54)∈ A , we have that at most p
o
( D − k ) of φ j are (1 − p o ) / ˜ σ , while the others are − p o / ˜ σ , and thus
[Equation]
Furthermore, (cid:104) w, ζ (cid:105) ∼ N (0 , σ 2 ζ (cid:107) w (cid:107) 22 ) and (cid:107) w (cid:107) 22 ≤ ˜ σ 2 ( k + k 2 µ/ √ d ) , we have
Pr[ |(cid:104) w, ζ/ ˜ σ (cid:105)| ≤ 1 / 16] ≥ 1 − exp (cid:32) − Θ (cid:32)
[Equation]
(cid:107) w (cid:107) 22 /
[Equation]
σ(cid:33)(cid:33) (308) ≥ [Equation] [Equation]
For good data points with φ and ζ satisfying the above, we have | (cid:15) x | ≤ 1 / 8 .
By Lemma 7,
[Equation]
Then for good data points, we have yg ∗ 1 ( x ) ≥ 1 .
Similarly, g
[Equation]
[Equation]
Then for good data points, we have yg ∗ 2 ( x ) ≥ 1 .
Note that the bias terms in g ∗ 1 and g ∗ 2 have distance at least 1 / 4 , then at least one of them satisﬁes that all its bias terms have absolute value ≥ 1 / 8 .
Pick that one and denote it as g ( x )
= (cid:80) ni
=1 a i σ r
(cid:104)
w i , x (cid:105) + b i ) .
By the positive homogeneity of σ r , we have g
[Equation]
x ) = n
[Equation]
Since for any good data points, |(cid:104) w i , x (cid:105) / (8 k ) + b i / (8 k ) | ≤ 1 , then
[Equation]
where σ is the truncated ReLU.
Now we can set a ∗ i = 8 ka i , w ∗ i =
w i / (8 k ) , b ∗ i
= b i / (8 k ) , to get our ﬁnal g ∗ .
F. I NITIALIZATION
We ﬁrst show that with high probability, the initial weights are in typical positions.
Lemma 35.
Suppose Dµ/ √ d ≤ 1 / 16 .
For any δ ∈ (0 , 1) , with probability at least 1 − δ − 2 exp ( − Θ( D − k )) over w (0) ,
[Equation]
With probability at least 1 − δ over b (0) ,
[Equation]
[Equation]
With probability at least 1 − δ over a (0) ,
[Equation]
Proof of Lemma 35.
The bound on (cid:107) w (0) (cid:107) 22 follows from the property of Gaussians.
Note that q = M (cid:62) w (0) ∼ N (0 , σ 2 w ρ ) for the matrix ρ = M (cid:62) M .
We have with probability ≥ 1 − δ/ 2 , max [Equation]
For any subset S ⊆
[ D ] , let ρ S denote the submatrix of ρ containing the rows and columns indexed by S .
Then q S =
M (cid:62) w (0) ∼ N (0 , σ 2 w ρ S ) .
By diagonalizing ρ S and then applying Bernstein’s inequality, we have with probability ≥ 1 − 2 exp ( − Θ( | S | / (cid:107) ρ (cid:107) 2 ) , (cid:107) q S (cid:107) 22 ∈ (cid:16) ( (cid:107) [Equation] S (cid:107) 2 F − | S | 4 ) [Equation]
[Equation] [Equation] [Equation]
[Equation]
(cid:17) . By Gershgorin circle theorem, we have
[Equation]
Similarly, we have
[Equation]
The bounds on q then follow.
The bounds on b (0) and a (0) follow from the property of Gaussians.
Lemma 36.
Suppose Dµ/ √ d ≤ 1 / 16 .
We have:
• With probability ≥ 1 − δ − 2 m exp( − Θ( D − k )) over w (0)
i ’s, for all i ∈ [2 m ] , w (0) i ∈
G w ( δ ) .
• With probability ≥ 1 − δ over b (0)
i ’s, for all i ∈ [2 m ] , b (0) i ∈
G b ( δ ) .
• With probability ≥ 1 − δ over a (0) i ’s, for all i ∈ [2 m ] , a (0) i ∈ G a ( δ ) .
Proof of Lemma 36.
This follows from Lemma 35 by union bound.
F. S OME A UXILIARY L EMMAS
The expression of the gradients will be used frequently.
Lemma 37.
[Equation]
Proof of Lemma 37.
It follows from straightforward calculation.
We also have the following auxiliary lemma for later calculations.
Lemma 38.
[Equation]
Proof of Lemma
[Equation]
F. F EATURE E MERGENCE : F IRST G RADIENT S TEP
We will show that w.h.p.
over the initialization, after the ﬁrst gradient step, there are neurons that represent good features.
We begin with analyzing the gradients.
Lemma 39.
Fix δ ∈ (0 , 1) and suppose w (0) i
[Equation] , b (0) i ∈ G b ( δ ) for all i ∈ [2 m ] .
Let
[Equation]
If σ 2 ζ σ 2 w d/ ˜ σ 2 =
O (1 /k ) , p o = Ω( k 2 /D ) , k = Ω(log 2 ( Dmd/δ )) , and σ (1) ξ = O (1 /k ) , then
[Equation]
(341) where T j satisﬁes:
• if j ∈ A , then | T j − βγ/ ˜ σ | ≤ O ( (cid:15) e / ˜ σ ) , where β ∈ [Ω(1) , 1] and depends only on w (0) i , b (0) i ; • if j (cid:54)∈ A , then | T j | ≤ O (
σ 2 φj (cid:15) e ˜ σ ) ; • | ν j | ≤ O (cid:18) σ ζ √ log( k ) ˜ σ (cid:15) ν (cid:19) + σ ζ d ˜ σ e − Θ( k ) .
Proof of Lemma 39.
Consider one neuron index i and omit the subscript i in the parameters.
Since the unbiased initialization leads to g (0) ( x ; ξ (1) )
= 0 , we have
[Equation]
First, consider j ∈ A .
[Equation]
where ι := (cid:104) w (0) , ζ/ ˜ σ (cid:105) . Let
I a :=
[Equation]
[Equation]
Note that |(cid:104) φ A , q A (cid:105)| =
O ( kσ w √ 2log( D/δ ) ˜ σ) , and that | ι | = |(cid:104) w (0) i , ζ/ ˜ σ (cid:105)| =
O ( √ dσ ξ σ w √ 2log( D/δ ) ˜ σ ) , and that |(cid:104) φ, q (cid:105)| , |(cid:104) φ − A , q − A (cid:105)| are O
( Dσ w √ 2log
[Equation]
[Equation]
When σ w is sufﬁciently small, by the property of the Gaussian ξ (1) , we have
[Equation]
[Equation]
[Equation]
[Equation]
In summary,
[Equation]
Then we have
[Equation]
[Equation]
357) ≤ O
[Equation]
where the last step is from Lemma 38.
Furthermore,
[Equation]
When σ w is sufﬁciently small
[Equation]
we have
[Equation]
This leads to
[Equation]
By Lemma 38, E φ A { α y yφ j } = γ/ ˜ σ .
Therefore,
[Equation]
/ ˜ σ )
[Equation]
Now, consider j (cid:54)∈ A .
Let B denote A ∪ { j } .
[Equation]
Let
[Equation]
Similar as above
[Equation]
we have | E ζ,ξ (1) ( I b − I (cid:48) b ) | ≤ O ( (cid:15) e )
[Equation]
Then by Lemma 38,
[Equation]
Furthermore,
[Equation]
Therefore,
[Equation]
Finally, consider ν j .
[Equation]
where ι j := w (0) j ζ j / ˜ σ and ι − j := (cid:104) w [Equation] 0) , ζ/ ˜ σ (cid:105) − ι j .
With probability ≥ 1 − d exp( − Θ( k )) over ζ , for any j , | ζ j | ≤ O ( σ ζ (cid:112) log( k )) .
Let G ζ denote this event.
Let
[Equation]
[Equation] Similar as above, we have | E ζ
[ I j − I (cid:48)
j |G ζ ] | ≤ O ( (cid:15) ν ) .
Then
[Equation]
The ﬁrst term is bounded by
[Equation]
The second term is bounded by
[Equation]
Furthermore,
[Equation]
Therefore
[Equation]
Lemma 40.
Under the same assumptions as in Lemma 39,
[Equation]
where | T b | ≤ O ( (cid:15) e ) .
Proof of Lemma 40.
Consider one neuron index i and omit the subscript i in the parameters.
Since the unbiased initialization leads to g (0) ( x ; ξ (1) )
= 0 , we have
[Equation]
where ι := (cid:104) w (0) , ζ/ ˜ σ (cid:105) .
Similar to the proof in Lemma 39,
[Equation]
[Equation]
Then [Equation] [Equation] [Equation]
[Equation] [Equation]
Also,
[Equation]
=
[Equation]
Therefore, | T b | ≤ O ( (cid:15) e ) .
Lemma 41.
We have
[Equation]
(416) where | T a | ≤ O (max (cid:96) q (0) i,(cid:96) ) .
So if w (0) i ∈ G ( δ ) , | T a | ≤ O ( σ w (cid:112) log( Dm/δ )) .
Proof of Lemma 41.
Consider one neuron index i and omit the subscript i in the parameters.
Since the unbiased initialization leads to g (0)
[Equation]
x ; ξ
[Equation]
= 0 , we have
[Equation]
Let φ (cid:48)
A be an independent copy of φ A [Equation] φ (cid:48) be the vector obtained by replacing in φ the entries φ A with φ (cid:48)
A
[Equation]
and let x (cid:48)
=
Mφ (cid:48)
+ ζ/ ˜ σ and its label is y (cid:48) [Equation]
Then
[Equation]
Since σ is 1-Lipschitz,
[Equation]
With the bounds on the gradient, we now summarize the results for the weights after the ﬁrst gradient step.
Lemma 42.
Set
[Equation]
Fix δ ∈ (0 , 1) and suppose w (0) i ∈
G w ( δ ) , b (0) i ∈ G b ( δ ) for all i ∈ [2 m ] .
If k = Ω(log 2 ( Dm/δ )) , then for all i ∈ [ m ] , w (1) i
[Equation]
(cid:96) + υ satisfying if (cid:96) ∈ A , then | q (1) i,(cid:96) − η (1) a (0) i βγ/ ˜ σ | ≤ O (cid:18) | η (1) a (0) i | (cid:15) e ˜ σ (cid:19) , where β ∈ [Ω(1) , 1] and depends only on w (0) i , b (0) i ; • if (cid:96) (cid:54)∈ A , then | q (1) i,(cid:96) | ≤ O (cid:16) | η (1) a (0) i | σ 2 φ(cid:96) (cid:15) e ˜ σ (cid:17) ;
• | υ j | ≤ O (cid:18) | η (1) a (0) i | (cid:18) σ ζ √ log( k ) ˜ σ (cid:15)
ν + σ ζ d ˜
σ e − Θ( k ) (cid:19)(cid:19) .
and
• b (1) i =
b (0) i + η (1) a (0)
i T b where | T b | =
O ( (cid:15) e ) ; • a (1) i =
a (0) i + η (1) T a where | T a | =
O ( σ w (cid:112) log( Dm/δ )) .
Proof of Lemma 42.
This follows from Lemma 36 and Lemma 39-41.
F. F EATURE I MPROVEMENT : S ECOND G RADIENT S TEP
We ﬁrst show that with properly set η (1) , for most x , | g (1) ( x ; σ (2) ξ ) | < 1 and thus yg (1) ( x ; σ (2) ξ ) < 1 .
Lemma 43.
Fix δ ∈ (0 , 1) and suppose w (0) i ∈
G w ( δ ) , b (0) i ∈ G b ( δ ) , a (0) i ∈ G a ( δ ) for all i ∈ [2 m ] .
If Dµ/ √ d ≤ 1 / 16 , σ ζ ˜ σ = O (1) , σ 2 ζ d/ ˜ σ 2 =
O
(1) , k = Ω(log {+
+} ( Dm/δ )) , σ a ≤ ˜ σ 2 / ( γk 2 ) , η (1) =
O (cid:16) γ kmσ a ˜ σ (cid:17) , and σ (2) ξ ≤ 1 /k , then with probability ≥ 1 − ( d + D ) exp( − Ω( k )) over ( x, y ) , we have yg (1) ( x ; σ (2) ξ ) < 1 .
Furthermore, for any i ∈ [2 m ] , (cid:12)(cid:12)(cid:12) (cid:104) w (1) i , ζ/ ˜ σ (cid:105) (cid:12)(cid:12)(cid:12) =
O ( η (1) ˜ σ/γ ) , (cid:12)(cid:12)(cid:12) (cid:104) q (1) i , φ (cid:105) (cid:12)(cid:12)(cid:12) =
O ( η (1) ˜ σ/γ ) , and (cid:12)(cid:12)(cid:12) (cid:104) ( q (1) i ) − A , φ − A (cid:105) (cid:12)(cid:12)(cid:12) =
O ( η (1) ˜ σ/γ ) , and for any j ∈ [ d ] , (cid:96) ∈
[ D ] , | ζ j | ≤ O ( σ ζ (cid:112) log( k )) and |(cid:104) ζ, D (cid:96) (cid:105)| ≤ O ( σ ζ (cid:112) log( k )) .
Proof of Lemma 43.
Note that w (0)
i
[Equation]
= a (0) m
+ i .
Then the gradient for w m
+ i is the negation of that for w m
+ i , the gradient for b m
+ i is the negation of that for b m + i , and the gradient for a m
+ i is the same as that for a m
+ i .
[Equation]
Then we have (cid:12)(cid:12)(cid:12) g (1) ( x ; σ (2) ξ ) (cid:12)(cid:12)(cid:12) ≤ m (cid:88) i =
(cid:12)(cid:12)(cid:12) 2 η (1) T a E ξ (2) σ ( (cid:104) w (1) i , x (cid:105) + b (1)
i + ξ (2) i ) (cid:12)(cid:12)(cid:12) (435) + m (cid:88) i =
(cid:12)(cid:12)(cid:12) a (1) m
+ i (cid:12)(cid:12)(cid:12) (cid:16)(cid:12)(cid:12)(cid:12) (cid:104) w (1) i − w (1) m
+ i , x (cid:105) (cid:12)(cid:12)(cid:12) + (cid:12)(cid:12)(cid:12) b (1) i − b (1) m
+ i (cid:12)(cid:12)(cid:12)(cid:17) (436) ≤ m (cid:88) i =
(cid:12)(cid:12)(cid:12) 2 η (1) T a (cid:12)(cid:12)(cid:12) (cid:16)(cid:12)(cid:12)(cid:12) (cid:104) w (1) i , x (cid:105) + b (1) i (cid:12)(cid:12)(cid:12) + E ξ (2) (cid:12)(cid:12)(cid:12) ξ (2)
i (cid:12)(cid:12)(cid:12)(cid:17) (437) + m (cid:88) i = (cid:12)(cid:12)(cid:12)
[Equation]
With probability ≥ 1 − exp( − Ω( k )) , among all j (cid:54)∈ A , we have that at most p
o
( D − k ) of φ j are (1 − p o j ) / ˜
σ , while the others are − p o j / ˜ σ .
With probability ≥ 1 − ( d + D ) exp( − Ω( k )) over ζ , for any j , | ζ
j | ≤ O ( σ ζ (cid:112) log( k )) and |(cid:104) ζ, D (cid:96) (cid:105)| ≤ O ( σ ζ (cid:112) log( k )) .
For data points with φ and ζ satisfying these, we have:
[Equation]
Proof of Claim 1.
[Equation]
D (cid:88) j = φ j M j (cid:105) (cid:12)(cid:12) (cid:12) (cid:12) (cid:12)(cid:12) + (cid:12) (cid:12) (cid:12) (cid:12)(cid:12) (cid:104) D (cid:88) (cid:96) = q (1) i,(cid:96) M (cid:96) , ζ/ ˜ σ (cid:105) (cid:12) (cid:12)(cid:12)(cid:12)(cid:12) + (cid:12)(cid:12) (cid:12) (cid:12) (cid:12)(cid:12) (cid:104) υ,
D (cid:88) j = φ j M j (cid:105) (cid:12)(cid:12) (cid:12) (cid:12) (cid:12)(cid:12) + |(cid:104) υ, ζ/ ˜ σ (cid:105)| .
For each term above we bound as follows.
Note that when σ w is sufﬁciently small, (cid:15) e =
O ( k log 1 / 2 ( Dm/δ ) / √ D ) .
Let
B 1 := βγ/ ˜
[Equation]
+ (cid:15) e / ˜ σ, (441)
[Equation] [Equation] [Equation]
Then
[Equation]
[Equation]
[Equation]
Then by the assumption on µ ,
[Equation]
γ
[Equation]
[Equation]
By the assumption on σ ζ ,
[Equation]
[Equation]
[Equation]
[Equation]
458) ≤ O ( η (
[Equation]
/γ ) .
Also note that | ν j | ≤ O (cid:16) σ ζ log 2 ( Dm/δ ) ˜ σ √ D (cid:17) .
Then by the assumption on σ ζ , (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:104) υ
[Equation]
D
[Equation]
j = φ j
[Equation]
(cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (460) ≤
[Equation]
[Equation]
[Equation]
Finally
[Equation]
we have
[Equation]
We also have:
[Equation]
Then we have
[Equation]
Then (cid:12)(cid:12)(cid:12)
[Equation]
1) ( x ; σ (2) ξ ) (cid:12)(cid:12)(cid:12) < 1 .
Finally, the statement on (cid:12)(cid:12)(cid:12) (cid:104) ( q (1) i ) − A , φ − A (cid:105) (cid:12)(cid:12)(cid:12) follows from a similar calculation on (cid:12)(cid:12) (cid:12) (cid:104) w (1) i , x (cid:105) (cid:12)(cid:12) (cid:12) = (cid:12)(cid:12) (cid:12) (cid:104) q (1) i , φ (cid:105) (cid:12)(cid:12) (cid:12) .
We are now ready to analyze the gradients in the second gradient step.
Lemma 44.
Fix δ ∈ (0 , 1) and suppose w (0) i
[Equation] for all i ∈ [2 m ] .
Let (cid:15) e2 :
= O (cid:18) η (1) | a (0)
i | k
( γ + (cid:15) e ) ˜ σ 2 σ (2) ξ (cid:19) + exp( − Θ( k )) .
If Dµ/ √ d ≤ 1 / 16 , σ ζ ˜ σ = O (1) , σ 2 ζ d/ ˜ σ 2 = O
(1) , k {+= Ω(log 
+}
( Dm/δ )) , σ a ≤ ˜ σ 2 / ( γk 2 ) , η (1) =
O (cid:16)
[Equation]
(cid:17) , and σ (2)
[Equation]
= 1 /k 3 / 2 , then
[Equation]
(478) where T j satisﬁes:
• if j ∈ A , then | T j − βγ/ ˜ σ | ≤ O ( (cid:15) e2 / ˜ σ + η (1) /σ (2)
ξ + η (1) | a (0) i | (cid:15) e / (˜ σσ (2) ξ )) , where β ∈ [Ω(1) , 1] and depends only on w (0)
i , b (0) i ; • if j (cid:54)∈ A , then | T j | ≤ 1˜ σ exp( − Ω( k ))
+ O ( σ 2 φ ˜ σ(cid:15) e2 ) ; • | ν j | ≤ O (cid:18) η (1) σ ζ γσ (2) ξ (cid:19) + exp( − Ω( k )) .
Proof of Lemma 44.
Consider one neuron index i and omit the subscript i in the parameters.
By Lemma 43, with probability at least 1 − ( d + D ) exp( − Ω( k ))
= 1 − exp( − Ω( k )) over ( x, y ) , yg (1) ( x ; ξ (2) ) > 1 and furthermore, for any i ∈ [2 m ] , (cid:12)(cid:12)(cid:12) (cid:104) w (1) i , ζ/ ˜ σ (cid:105) (cid:12)(cid:12)(cid:12) =
O ( η (1) ˜ σ/γ ) , (cid:12)(cid:12)(cid:12) (cid:104) q (1) i , φ (cid:105) (cid:12)(cid:12)(cid:12) =
O ( η (1) ˜ σ/γ ) , and (cid:12)(cid:12)(cid:12) (cid:104) ( q (1) i ) − A , φ − A (cid:105) (cid:12)(cid:12)(cid:12) =
O ( η (1) ˜ σ/γ ) , and for any j ∈ [ d ] , (cid:96) ∈
[ D ] , | ζ j | ≤ O ( σ ζ (cid:112) log( k )) and |(cid:104) ζ, D (cid:96) (cid:105)| ≤ O ( σ ζ (cid:112) log( k )) .
Let I x be the indicator of this event.
[Equation]
Let T j 1 :=
E ( x,y ) ∼D ,ξ (2) (cid:8) α
y y
I
[ (cid:104) w (1) , x (cid:105)
+ b (1) + ξ (2) ∈
(0 , 1)] φ j (cid:9) .
We have
[Equation]
Similarly, let ν (cid:48) :
= E ( x,y ) ∼D ,ξ (2) (cid:110) α y yζ ˜ σ I [ (cid:104) w (1) , x (cid:105)
+ b (1) + ξ (2) ∈
(0 , 1)] (cid:111) .
We have
[Equation]
So it is sufﬁcient to bound T j
and ν (cid:48) .
For simplicity, we use q as a shorthand for q (1) i .
First, consider j ∈ A .
[Equation]
[Equation]
where ι := (cid:104) w (1) , ζ/ ˜ σ (cid:105) . Let I a
[Equation]
Pr
[Equation]
[Equation]
[Equation]
By the property of the Gaussian ξ (2) , that |(cid:104) φ A , q A (cid:105)| = O ( η (1) | a (0) i | k ( γ + (cid:15) e ) ˜ σ) , and that | ι | = |(cid:104) w (1) i , ζ/ ˜ σ (cid:105)| , |(cid:104) φ, q (cid:105)| , |(cid:104) φ − A , q − A (cid:105)| are all O
( η (1) ˜ σ/γ ) < O (1 /k ) , we have
[Equation] [Equation]
This leads to
[Equation]
where the last step is from Lemma
[Equation]
Furthermore,
[Equation]
By Lemma 19, we have |(cid:104) φ − A , q − A (cid:105) + ι | ≤ O ( η (1) ˜ σ/γ ) .
Also, | b (1) − b (0) | ≤ O ( η (1) | a (0) i | (cid:15) e ) .
By the property of ξ (2) ,
[Equation]
On the other hand,
[Equation]
and β only depends on b (0) .
By Lemma 38, E φ A { α y yφ j } = γ/ ˜ σ .
Therefore,
[Equation]
Now, consider j (cid:54)∈ A .
Let B denote A ∪ { j } .
[Equation]
Let
[Equation]
Similar as above [Equation] we have | I b − I (cid:48)
b | ≤ (cid:15) e2 .
Then by Lemma 38,
[Equation]
Furthermore,
[Equation]
Therefore,
[Equation]
Finally, consider ν (cid:48)
j .
[Equation]
[Equation]
Let
[Equation] [Equation]
Since | ι | ≤ O ( η (1) ˜ σ/γ )
[Equation]
we have |
I j − I (cid:48)
j | ≤ O ( η (1) ˜ σ/ ( γσ (2) ξ )) .
Then
[Equation]
Furthermore,
[Equation]
Therefore,
[Equation]
Lemma 45.
Under the same assumptions as in Lemma 44,
[Equation]
where | T b | ≤ exp( − Ω( k ))
+ O ( (cid:15) e2 ) .
Proof of Lemma 45.
Consider one neuron index i and omit the subscript i in the parameters.
By
Lemma 43, Pr[ yg (1) ( x ; ξ (2) ) > 1] ≤ exp( − Ω( k )) .
Let I x
= I [ yg (1) ( x ; ξ (2) ) ≤ 1] .
[Equation]
[Equation]
[Equation]
Let T b 1 :=
E ( x,y ) ∼D ,ξ (2) (cid:8) α
y y
I
[ (cid:104) [Equation] (1) , x (cid:105)
+ b (1) + ξ (2) ∈
(0 , 1)] (cid:9) .
We have [Equation] [Equation]
537) =
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
So it is sufﬁcient to bound T b 1 .
For simplicity, we use q as a shorthand for q (1) i .
[Equation]
Let
[Equation]
Similar as in Lemma 6, we have | I b − I [Equation]
b | ≤ (cid:15) e2 .
Then by Lemma 38,
[Equation]
Furthermore,
[Equation]
Therefore, | T b 1 | ≤ O ( (cid:15) e2 ) and the statement follows.
Lemma 46.
Under the same assumptions as in Lemma 44,
[Equation]
[Equation]
where | T a | = O (cid:16) [Equation] [Equation] +
[Equation]
(cid:16) dDp [Equation] (cid:17) .
Proof of Lemma 46.
Consider one neuron index i and omit the subscript i in the parameters.
By
Lemma 43, Pr[ yg (1) ( x ; ξ (2) ) > 1] ≤ exp( − Θ( k )) .
Let I x
= I [ yg (1) ( x ; ξ (2) ) ≤ 1] .
[Equation]
Let T a 1 :=
E ( x,y ) ∼D (cid:8)
α y y E ξ (2) σ ( [Equation] w (1) , x (cid:105) + b (1) + ξ (2) ) (cid:9) .
We have
[Equation]
[Equation] [Equation]
So it is sufﬁcient to bound T a 1 .
For simplicity, we use q as a shorthand for q (1) i .
Let φ (cid:48)
A be an independent copy of φ A , φ (cid:48) be the vector obtained by replacing in φ the entries φ A with φ (cid:48)
A
[Equation]
and let x (cid:48)
=
Mφ (cid:48)
+ ζ and its label is y (cid:48) .
Then
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
where the fourth step follows from that σ is 1-Lipschitz, and the second to the last line from Lemmaand that (cid:12)(cid:12) (cid:104) w (1) , Mφ (cid:105) (cid:12)(cid:12) ≤ (cid:107) w (1) (cid:107) ∞ (cid:112)
d (cid:107) Mφ (cid:107) 22 .
With the above lemmas about the gradients, we are now ready to show that at the end of the second step, we get a good set of features for accurate prediction.
Lemma 47.
Set
[Equation]
Fix δ ∈ (0 , O (1 /k 3 )) .
If Dµ/ √ d ≤ 1 / 16 , σ ζ ˜ σ = O (1) , σ 2 ζ d/ ˜ σ 2 =
O
(1) , k = Ω (cid:16) log 2 (cid:16) Dmdδγp min (cid:17)(cid:17) , and m ≥ max { Ω( k 4 ) , D, d } , then with probability at least 1 − δ over the initialization, there exist ˜ a i ’s such that ˜ g ( x ) :
= (cid:80) 2 mi =1 ˜ a i σ ( (cid:104) w (2) i , x (cid:105) + b (2) i ) satisﬁes L D (˜ g ) ≤ exp( − Ω( k )) .
Furthermore, (cid:107) ˜ a (cid:107)
= O ( m/k ) , (cid:107) ˜
a (cid:107) ∞
=
O ( k 5 /m ) , and (cid:107) ˜ a (cid:107)
= O ( k 9 /m ) .
Finally, (cid:107) a (2) (cid:107) ∞
=
O (cid:0) 1 km 2 (cid:1) , (cid:107) w (2) i
(cid:107)
= O (˜ σ/k ) , and | b (2) i | =
O (1 /k 2 ) for all i ∈ [2 m ] .
Proof of Lemma 47.
By Lemma 34, there exists a network g ∗ ( x )
= (cid:80) 3( k +1) (cid:96) =
[Equation]
) satisfying
Pr ( x,y ) ∼D [ yg ∗ ( x ) ≤ 1] ≤ exp( − Ω( k )) .
Furthermore, the number of neurons n = 3( k + 1) , | a ∗ i | ≤ 64 k, 1 / (64 k ) ≤ | b ∗ i | ≤ 1 / 4 , w ∗ i = ˜
σ (cid:80)
j ∈
A M j / (8 k ) , and |(cid:104) w ∗ i , x (cid:105)
+ b ∗ i | ≤ 1 for any i ∈ [ n ] and ( x, y ) ∼ D .
Now we ﬁx an (cid:96) , and show that with high probability there is a neuron in g (2) that can approximate the (cid:96) -th neuron in g ∗ .
With probability ≥ 1 − exp( − Ω(max { 2 p o
( D − k ) , k } )) , among all j (cid:54)∈ A , we have that at most 2 p
o
( D − k )
+ k of φ j are (1 − p o ) /
˜ σ , while the others are − p o / ˜ σ .
With probability ≥ 1 − ( d + we consider data points with φ and ζ satisfying these.
By Lemma 36, with probability 1 − 2 δ over w (0) i ’s, they are all in G w ( δ ) ; with probability 1 − δ over a (0)
i ’s, they are all in G a ( δ ) ; with probability 1 − δ over b (0)
i ’s, they are all in G b ( δ ) .
Under these events, by Lemma 42, Lemma 44 and 45, for any neuron i ∈ [2 m ] , we have
[Equation]
where
• if j ∈ A , then | T j − βγ/ ˜ σ | ≤ (cid:15) w 1 := O ( (cid:15) e2 / ˜ σ + η (1) /σ
(2) ξ + η (1) | a (0) i | (cid:15) e / (˜ σσ (2) ξ )) , where β ∈ [Ω(1) , 1] and depends only on w (0)
i , b (0) i ; • if j (cid:54)∈ A , then | T j | ≤ (cid:15) w 2 := 1˜ σ exp( − Ω( k ))
+ O ( σ 2 φ ˜ σ(cid:15) e2 )
; • | ν j | ≤ (cid:15) ν :
= O (cid:18) η
(1) σ ζ γσ (2) ξ (cid:19) + exp( − Ω( k )) .
• | T b | ≤ (cid:15) b := exp( [Equation] Ω
[Equation]
k )) + O [Equation] e2 ) .
Given the initialization, with probability Ω(1) over b (0) i , we have
[Equation]
Finally, since 8 k | b ∗
[Equation]
| βγ | b (0) i | ˜ σ 2 ∈ [Ω( k 2 γ/ ˜ σ 2 ) , O ( k 3 γ/ ˜ σ 2 )] and depends only on w (0) i , b (0) i , we have that for (cid:15) a = Θ(1 /k 2 ) , with probability Ω( (cid:15) a ) >
δ over a (0) i ,
[Equation]
Let n a = (cid:15) a m/ 4 .
For the given value of m , by (569)-(572) we have with probability ≥ 1 − 5 δ over the initialization, for each (cid:96) there is a different set of neurons
I (cid:96) ⊆
[ m ] with | I (cid:96) | = n a and such that for each i (cid:96) ∈
I (cid:96) ,
[Equation] [Equation]
[Equation]
Now, construct ˜ a such that ˜ a i (cid:96) =
2 a ∗ (cid:96)
| b ∗ (cid:96) | | b (0) i(cid:96)
| n a for each (cid:96) and each i (cid:96) ∈
I (cid:96) , and ˜ a i = 0 elsewhere.
To show that it gives accurate predictions, we ﬁrst consider bounding some error terms.
For the given values of parameters, we have
[Equation]
[Equation]
We also have the following useful claims.
[Equation]
Proof of Claim 2.
[Equation]
Claim 3.
[Equation]
Proof of Claim
[Equation]
Then
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
The ﬁrst term is
[Equation]
[Equation]
[Table]
[Equation]
599) ≤ O
[Equation]
600) ≤ O
[Equation]
(601) ≤ O (cid:32) (cid:15) ν √ d
[Equation]
σ (cid:33) (cid:16) p [Equation] ˜ σ
[Equation] ( k [Equation] (cid:17) (602) ≤ O (cid:32) p o D(cid:15)
[Equation] (cid:33) .
Then by (575)-(579), (cid:15) φ := (cid:32) k(cid:15) w˜ [Equation] D(cid:15) w
[Equation]
(cid:18)
[Equation] m [Equation] ˜
[Equation]
(cid:19) . We have (cid:12)(cid:12)(cid:12) a (1) i (cid:96) − a (0) i (cid:96) (cid:12)(cid:12)(cid:12)
=
O ( η (1) σ w (cid:112) log( Dm/δ )) .
So the ﬁrst term is bounded by (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:104) w (
[Equation]
≤ O
[Equation]
By Claim 2, the second term is bounded by
[Equation]
Combining the bounds on the two terms leads to the claim.
Claim 4.
[Equation]
Proof of Claim 4.
By Lemma 42 and 45:
[Equation]
We are now ready to show ˜ g is close to 2 g ∗ .
[Equation]
[Table]
[Equation]
[-(0) i (cid:96) (cid:33)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (621) ≤ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) 
-]
+
[Equation]
(cid:33) 
[Equation]
Here the second equation follows from that σ is positive-homogeneous in [0 , 1] , |(cid:104) w ∗ (cid:96) , x (cid:105) + [-(cid:12) (cid:12) (cid:12) (cid:12)(cid:12)(cid:12) 
 ( k +1) (cid:88)-]b ∗ (cid:96) | ≤ 1 , | b (0)
i (cid:96) | / | b ∗ (cid:96) | ≤ 1 .
By Claim 3 and 4, the ﬁrst term is bounded by:
[Equation]
O
[Equation]
By Claim
[Table]
the second term is bounded by:
[Table]
[Equation]
Then
[Equation]
This guarantees y ˜ g
( x ) ≥ 1 .
Changing the scaling of δ leads to the statement.
Finally, the bounds on ˜ a follow from the above calculation.
The bound on (cid:107) a (2) (cid:107) 2 follows from
Lemma 46, and those on (cid:107) w (2) i (cid:107) 2 and (cid:107) b (2) i (cid:107) 2 follow from (569)(570) and the bounds on a (1) i and b (1) i in Lemma 42.
F. C LASSIFIER L EARNING S TAGE AND M AIN T HEOREM
Once we have a good set of features in Lemma 47, we can follow exactly the same argument as in
Section B.6 and B.7 for the simpliﬁed setting, and arrive at the main theorem for the general setting:
Theorem 48 (Restatement of Theorem 33) .
Set
[Equation]
[Equation]
(626) η [Equation] [Equation] ξ
[Equation]
(627) η
[Equation] for [Equation] For any δ ∈ (0 , O (1 /k 3 )) , if µ ≤ O ( √ d/D ) , σ ζ ≤ O (min { 1 / ˜ σ, ˜ σ/ √ d } ) , k = Ω (cid:16) log 2 (cid:16) Dmdδγp min (cid:17)(cid:17) , m ≥ max { Ω( k 4 ) , D, d } , then we have for any D ∈ F Ξ , with probability at least 1 − δ , there exists t ∈
[ T ] such that
[Equation]
(cid:19)
[Equation]
Consequently, for any (cid:15) ∈ (0 , 1) , if T = m 4 / 3 , and m ≥ max { Ω( k 12 /(cid:15) 3 / 2 ) , D } , then
Pr[sign( g ( t ) ( x ))
(cid:54) =
y ] ≤ L D ( g ( t ) ) ≤ (cid:15).
G A DDITIONAL E XPERIMENTS G. M ORE S IMULATION R ESULT IN V ARIOUS S ETTINGS
We show the robustness of our simulation results by studying the learning behaviors in a variety of settings including different sample size, input data dimension and class imbalance.
We reuse the same setting as the simulation in the main text (details in E.1.1), vary different parameters, and report the accuracy, the cosine similarities between the learned weights, and the visualization of the neuron weights.
G.1. V ARYING I NPUT D ATA D IMENSION
In the simulation experiments in the main text, the input data dimension d is 500.
Here we change the input data dimension to 100 and 2000.
All other conﬁgurations follow E.1.1.
Veriﬁcation of the Main Results.
Figure 16 shows that our claim is robust under different input data dimensions.
The performance of network learning is superior over NTK and random feature approaches on inputs with structure, and on inputs without structure, all three methods fail.
[Figure]
[Table]
Feature Learning in Networks.
Figure 17 visualizes the neuron weights.
It shows similar results to that in E.1.1: the weights gets updated to to the effective feature in the ﬁrst two steps, forming clusters.
Table 3 shows some quantitative results.
In particular, the average cosine similarities between neuron weights and the effective features after two steps are close to 1, showing that they match the effective features.
G.1. V ARYING C LASS I MBALANCE R ATIO
The experiments in the main text has 25000 training samples for each class.
Here we keep the total sample size 50000 but use different class imbalance ratios, which is the class − 1 sample size divide by the total sample size.
Veriﬁcation of the Main Results.
Figure 18 shows that our claim is robust under different class imbalance ratios.
The results are similar to those for balanced classes, except that NTK becomes less stable.
[Figure]
Feature Learning in Networks.
Figure 19 visualizes the neurons’ weights.
Again, the observation is similar to that for balanced classes.
Table 4 shows some quantitative results which are also similar to those for balanced classes.
G.1. V ARYING S AMPLE S IZE
Here we change the sample size 50000 in Section E.1.1 to be 25000 and 10000.
For sample size 25000, we observe similar results.
For sample size 10000, we observe over-ﬁtting (test accuracy much lower than train accuracy).
Therefore, for sample size 10000 we reduces the size of the network (i.e., number of hidden neurons) from m = 300 to m = 50 . [Table]
[Table]
Veriﬁcation of the Main Results.
Figure 20 shows that our claim is robust under different sample sizes.
In particular, the network learning still outperforms the NTK and random feature approaches on structured inputs.
[Figure]
Feature Learning in Networks.
Figure 21 and Table 5 show that the phenomenon of feature learning for different samples is similar to that in E.1.1.
[Table]
G. E XPERIMENTS ON M ORE D ATA G ENERATION M ODELS
In this section we consider some additional data distributions and run the simulation experiments, in particular, focusing on the feature learning phenomenon.
Note that our analysis is for the setting where the input distributions have structure revealing some information about the labeling function.
(More precisely, the labeling function is speciﬁed by A and P , while the input distribution also depends on them.)
We therefore consider two other data generation mechanisms where the labeling function also has connections to the input distributions.
G.2. H IDDEN R EPRESENTATION L ABELING
Here we consider the following data model: ﬁrst uniformly at random select ˜
φ A from a set of binary vectors
[Table]
and assign label 1 to some and -1 to others; sample irrelevant patterns ˜ φ − A uniformly at random; generate the input x = M ˜ φ .
We randomly select 50 binary vectors for each label, with d = 500 , D = 250 , k = 50 , p o = 1 / 2 .
This is a generalization of the distribution D (1) , a component in the distribution of our simulation experiments (see the proof of Theorem 2 for details).
Recall the deﬁnition of D (1) :
˜ φ A is uniform on only two values [+1 , . . .
, +1] and [0 , . . .
, 0] , and uniform over irrelevant patterns; the value [+1 , . . .
, +1] corresponds to one class and [0 , . .
. , 0] correspond to another class.
Our data model here generalizes D (1) to more than 2 values.
The visualization is shown in Figure 22.
We can observe similar feature learning phenomena, and the neuron weights are updated to form clusters.
G.2. T WO - LAYER N ETWORKS ON M IXTURE OF G AUSSIANS
To further support our intuition of feature learning, we run experiments on mixture of Gaussians.
Data.
Let X =
R d be the input space, and Y = {± 1 } be the label space.
Suppose M ∈ R
d
× k is an dictionary with k orthonormal columns.
Let ε i , i = 1 , . . .
, k be i.i.d symmetric Bernoulli random variables, and g ∼ N (0 , σ 2 r kd I d ) .
Then we generate the input x and class label y by:
[Equation]
In this case, 2 k Gaussian clusters will be created.
The centers of the Gaussian clusters (cid:80) ki [Equation] [Equation] i lie on the vertices of a hyper cube, and the label of each Gaussian cluster is determined by the parity function on the vertices of the hyper cube.
Note that the labeling function is roughly equivalent to a network: y = (cid:80) ni =1 a i ReLU ( (cid:104) c
i , x (cid:105) ) where c i ’s are the Gaussian centers, and a i ∝ 1 for Gaussian components with label 1 and a i ∝ − 1 for those with label -1.
Setting.
We then train a two-layer network with m =
800 hidden neurons on data sets generated as above with different chosen k ’s and d ’s.
The training follows typical practice (not the hyperparameters in our analysis).
In this setting, we expect the neural network to learn the effective features: the directions of Gaussian cluster centers.
Result.
We run experiments with different settings.
The parameters are shown in Table 6.
From
Figure 23 we can see that some neurons learn the directions of Gaussian centers, and each Gaussian center is covered by some neurons, which matches our expectation.
[Table]
G. CNN S ON B INARY C IFAR 10: F EATURE L EARNING IN N ETWORKS
Setting.
We use ResNet( m ) , which is a ResNet-18 convolutional neural network (He et al., 2016) with m ﬁlters in the ﬁrst residual block.
It is obtained by scaling the number of ﬁlters in each block proportionally from the standard ResNet-18 network which is ResNet(64) .
We use ResNet(128) and ResNet(256) in this experiment.
We train our model on Binary CIFAR
(Krizhevsky, 2012) with labels airplane/automobile for 20 epochs.
The ﬁnal test accuracy of ResNet(128) is 95.75% and that of ResNet(256) is 93.8%. [Figure] [Table]
Results.
Figure 24 visualizes the ﬁlters’ weights of different residual blocks in ResNet(128) at Epoch 0, 3, and 30, and Figure 25 shows those in ResNet(256) .
They show that feature learning happens in the early stage and show that there are some clusters of weights (e g., the red and green points).
These colored points are selected at Epoch 20.
We ﬁrst visualize the weights at Epoch 20, and then hand pick the points that roughly form two clusters (i e., the points in the same cluster are close to each other while those in different clusters are far away).
We assign red and green colors to the two clusters at Epoch 20, and then assign these weights with the same color in Epoch 0 and 3.
Finally, we compute the cosine similarities and show that the hand picked points are indeed roughly clusters in the high-dimension.
In particular, we have the following three observations. [Figure] report First, we can see that the ﬁlter weights change signiﬁcantly during the early stage of the training, indicating feature learning happens in the early stage: the change between Epoch 0 and Epoch 3 is much more signiﬁcant than that between Epoch 3 and Epoch 20.
Second, we can also verify that the feature learning is guided by the gradients: the gradients of a ﬁlter in the early gradient steps point to similar directions (and thus the updated ﬁlter will learn this direction).
More precisely, for a selected ﬁlter, we average the gradients every 10 gradient steps (so to reduce the variance due to mini-batch) and get v 1 , v 2 and v 3 for the ﬁrst 30 steps and compute their cosine similarities and those to their average.
Table 7 shows the results.
In general the similarities [Figure] We report are high indicating they point to similar directions.
(Note that a similarity of 0 .
6 is regarded as very signiﬁcant as the ﬁlters are in a high dimension of 3 × 3 × 1024 = 9216 ).
Third, we also observe some clustering effect of the ﬁlter weights, though not as signiﬁcant as in our simulations.
For example, in the red and green clusters in Figure 24(a) for the ﬁrst residual block, the average cosine similarity for ﬁlter weights in the red cluster is about 0.62 and that for the green is about 0.7, while the cosine similarity between the two clusters’ centers is about -0.72.
This shows signiﬁcant similarities within the cluster while difference between clusters.
Note that the clustering is less signiﬁcant than our simulation experiments.
This is because practical data have more patterns (i.e., effective feature directions) to be learned than our synthetic data, and also the practical network is not as overparameterized as in our simulation.
Then ﬁlters are likely to learn different patterns (or their mixtures) without forming signiﬁcant clusters.
The results of
ResNet(256) show more signiﬁcant clustering than ResNet(128) , which supports our explanation.
On the other hand, we emphasize that the key insight of our analysis is that the gradient guides the learning of effective features in the early stage of training (rather than the clustering), which is veriﬁed as discussed above.
G. L ARGER N ETWORK ON MNIST FOR C HECKING T HE E FFECT OF I NPUT S TRUCTURE
Here we perform the experiment on MNIST as in E.3.2, but for a network with m = 50 hidden neurons rather than m = 9 .
Figure 26 shows similar results as those for m = 9 : the learning on the
MNIST input part is faster and better than that on the Gaussian input part.
The separation between the two is actually more signiﬁcant than that for m = 9 .
This then also supports our insight about the effect of input structures.