S TUDENT
S
PECIALIZATION
IN
D EEP
R E LU
N ETWORKS W ITH F INITE W IDTH AND I
NPUT D IMENSION
Anonymous authors
Paper under double-blind review
A BSTRACT
To analyze deep ReLU network, we adopt a student-teacher setting in which an over-parameterized student network learns from the output of a ﬁxed teacher network of the same depth, with Stochastic Gradient Descent (SGD).
Our contributions are two-fold.
First, we prove that when the gradient is small at every training sample, student node specializes to teacher nodes in the lowest layer under mild conditions.
Second, analysis of noisy recovery and training dynamics in 2-layer network shows that strong teacher nodes (with large fan-out weights) are learned ﬁrst and subtle teacher nodes are left unlearned until late stage of training.
As a result, it could take a long time to converge into these small-gradient critical points.
Our analysis shows that over-parameterization is a necessary condition for specialization to happen at the critical points, and helps student nodes cover more teacher nodes with fewer iterations.
Both improve generalization.
Different from Neural Tangent Kernel (Jacot et al., 2018) and statistical mechanics approach (Goldt et al., 2019), our approach operates on ﬁnite width, mild overparameterization (as long as there are more student nodes than teacher) and ﬁnite input dimension.
Experiments justify our ﬁnding.
I NTRODUCTION
Deep Learning has achieved great success in the recent years (Silver et al., 2016;
He et al., 2016;
Devlin et al., 2018).
Although networks with even one-hidden layer can ﬁt any function (Hornik et al., 1989), it remains an open question how such networks can generalize to new data.
Different from what traditional machine learning theory predicts, empirical evidence (Zhang et al., 2017) shows more parameters in neural network lead to better generalization.
How over-parameterization yields strong generalization is an important question for understanding how deep learning works.
In this paper, we analyze deep ReLU networks with teacher-student setting: a ﬁxed teacher network provides the output for a student to learn via SGD.
Both teacher and student are deep ReLU networks.
Similar to (Goldt et al., 2019), the student is over-realized compared to the teacher: at each layer l , the number n l of student nodes is larger than the number m l of teacher ( n l > m l ).
Although over-realization is different from over-parameterization , i.e., the total number of parameters in the student model is larger than the training set size N , over-realization directly correlates with the width of networks and is a measure of over-parameterization.
The student-teacher setting has a long history (Saad & Solla, 1996; 1995; Freeman & Saad, 1997;
Mace & Coolen, 1998) and recently gains increasing interest (Goldt et al., 2019; Aubin et al., 2018) in analyzing 2-layered network.
While worst-case performance on arbitrary data distributions may not be a good model for real structured dataset and can be hard to analyze, using a teacher network implicitly enforces an inductive bias and could potentially lead to better generalization bound.
Specialization, that is, a student node becomes increasingly correlated with a teacher node during training (Saad & Solla, 1996), is one of the important topic in this setup.
If all student nodes are specialized to the teacher, then student tends to output the same as the teacher and generalization performance can be expected.
Empirically, it has been observed in 2-layer networks (Saad & Solla, 1996; Goldt et al., 2019) and multi-layer networks (Tian et al., 2019; Li et al., 2016), in both synthetic and real dataset.
In contrast, theoretical analysis is limited with strong assumptions (e.g., Gaussian inputs, inﬁnite input dimension, local convergence, 2-layer setting, small number of hidden nodes).
In this paper, with arbitrary training distribution and ﬁnite input dimension, we show rigorously that when gradient at each training sample is small (i.e., the interpolation setting as suggested in (Ma et al., 2017; Liu & Belkin, 2018; Bassily et al., 2018)), the student node at the lowest layer can be proven to specialize to the teacher nodes: each teacher node is aligned with at least one student node in the lowest layer .
This explains one-to-many mapping between teacher and student nodes and the existence of un-specialized student nodes, as observed empirically in (Saad & Solla, 1996).
Furthermore, from the proof condition, more over-realization encourages specialization.
Our setting is different from previous works.
(1) While statistical mechanics approaches (Saad & Solla, 1996; Goldt et al., 2019; Gardner & Derrida, 1989; Aubin et al., 2018) assume both the training set size N and the input dimension d goes to inﬁnite (i.e., the thermodynamics limits) and assume Gaussian inputs, our analysis allows ﬁnite d and impose no parametric constraints on the input data distribution.
(2) While Neural Tangent Kernel (Jacot et al., 2018; Du et al., 2018b) and mean-ﬁeld approaches (Mei et al., 2018) requires inﬁnite (or very large) width, our setting applies to ﬁnite width as long as student is slightly over-realized ( n l ≥ m l ).
In this paper we study the inﬁnite training sample case (the training set is a region), and leave ﬁnite sample analysis as the future work.
In addition, we further analyze the training dynamics and show that most student nodes converge ﬁrst towards strong teacher nodes with large fan-out weights in magnitude.
While this makes training robust to dataset noise and naturally explains implicit regularization, the same mechanism also leaves weak teacher nodes unexplained until very late stage of training, yielding high generalization error with ﬁnite iterations.
In this situation, we show that over-realization plays another important role: once the strong teacher nodes have been covered, there are always spare student nodes ready to switch to weak teacher nodes quickly.
Empirically, we show more teacher nodes are covered with the same number of iterations, and generalization is also improved.
We verify our ﬁndings with numerical experiments.
Starting with 2-layer setting, we justify Theorem 2 and Theorem 3 with Gaussian inputs, showing one-to-many specialization and existence of un-specialized nodes.
For deep ReLU networks, we show specialization happens not only in the lowest layer, as suggested by Theorem 4, but also in other hidden layers, on both Gaussian inputs and CIFAR10.
We also perform ablation studies about the effect of student over-realization.
For training dynamics, we show the strong/weak teacher effects in 2-layer settings and over-realization could improve specialization and generalization.
R ELATED W ORKS
Student-teacher setting .
This setting has a long history (Engel & Van den Broeck, 2001; Gardner & Derrida, 1989).
The seminar works (Saad & Solla, 1996; 1995) studies 1-hidden layer case from statistical mechanics point of view in which the input dimension goes to inﬁnity, or so-called thermodynamics limits .
They study symmetric solutions and locally analyze the symmetric breaking behavior and onset of specialization of the student nodes towards the teacher.
Recent follow-up works (Goldt et al., 2019) makes the analysis rigorous and empirically shows that random initialization and training with SGD indeed gives student specialization in 1-hidden layer case, which is consistent with our experiments.
With the same assumption, (Aubin et al., 2018) studies phase transition property of specialization in 2-layer networks with small number of hidden nodes using replica formula.
In these works, inputs are assumed to be Gaussian and step or Gauss error function is used as nonlinearity.
Few works study teacher-student setting with more than two layers.
(Allen-Zhu et al., 2019a) shows the recovery results for 2 and 3 layer networks, with modiﬁed SGD, batchsize 1 and heavy over-parameterization.
In comparison, our work shows that specialization happens around the SGD critical points in the lowest layer for deep ReLU networks, without any parametric assumptions of input distribution.
Local minima is Global .
While in deep linear network, all local minima are global (Laurent &
Brecht, 2018; Kawaguchi, 2016), situations are quite complicated with nonlinear activations.
While local minima is global when the network has invertible activation function and distinct training samples (Nguyen & Hein, 2017; Yun et al., 2018) or Leaky ReLU with linear separate input data (Laurent & von Brecht, 2017), multiple works (Du et al., 2018a;
Ge et al., 2017; Safran & Shamir, 2017;
Yun et al., 2019) show that in GD case with population or empirical loss, spurious local minima can happen even in two-layered network.
Many are speciﬁc to two-layer and hard to generalize to multi-layer setting.
In contrast, our work brings about a generic formulation for deep ReLU network and gives recovery properties in the student-teacher setting.
[Figure]
Learning wild networks .
Recent works on Neural Tangent Kernel (Jacot et al., 2018; Du et al., 2018b; Allen-Zhu et al., 2019b) show the global convergence of GD for multi-layer networks with inﬁnite width.
(Li & Liang, 2018) shows the convergence in one-hidden layer ReLU network using GD/SGD to solution with good generalization, when the input data are assumed to be clustered into classes.
Both lines of work assume heavily over-parameterized network, requiring polynomial growth of number of nodes with respect to the number of samples.
(Chizat & Bach, 2018) shows global convergence of over-parameterized network with optimal transport.
(Tian et al., 2019) assumes mild over-realization and gives convergence results for 2-layer network when a subset of the student network is close to the teacher.
Our work extends it with much weaker assumptions.
Deep Linear networks .
For deep linear networks, multiple works (Lampinen & Ganguli, 2019;
Saxe et al., 2013; Arora et al., 2019; Advani & Saxe, 2017) have shown interesting training dynamics.
One common assumption is that the singular spaces of weights at nearby layers are aligned at initialization, which decouples the training dynamics.
Such a nice property would not hold for nonlinear network.
(Lampinen & Ganguli, 2019) shows that under this assumption, weight components with large singular value are learned ﬁrst, while we analyze and observe empirically similar behaviors on the student node level.
Generalization property of linear networks can also be analyzed in the limit of inﬁnite input dimension with teacher-student setting (Lampinen & Ganguli, 2019).
However, deep linear networks lack specialization which plays a crucial role in the nonlinear case.
To our knowledge, we are the ﬁrst to analyze specialization rigorously in deep ReLU networks.
M ATHEMATICAL F RAMEWORK
Notation .
Consider a student network and its associated teacher network (Fig. 1(a)).
Denote the input as x .
We focus on multi-layered networks with σ ( · ) as ReLU nonlinearity.
We use the following equality extensively: σ ( x ) = I [ x > 0] x , where I [ · ] is the indicator function.
For node j , f j ( x ) , z j ( x ) and g j ( x ) are its activation, gating function and backpropagated gradient after the gating .
Both teacher and student networks have L layers.
The input layer is layer 0 and the topmost layer (layer that is closest to the output) is layer L .
For layer l , let m l be the number of teacher node while n l be the number of student node.
The weights W l ∈ R n l − 1 × n l refers to the weight matrix that connects layer l − 1 to layer l on the student side.
W l = [ w l, 1 , w l, 2 , . .
. , w l,n l ] where each w ∈ R n l − 1 is the weight vector.
Similarly we have teacher weight W ∗ l ∈ R m l − 1 × m l .
Denote
W = { W 1 , W 2 , . .
, W L } as the collection of all trainable parameters.
Let f l ( x ) =
[ f l, 1 ( x ) , . . .
, f l,n l ( x )]
(cid:124) ∈
R n l be the activation vector of layer l , D l ( x ) =
diag[ z l, 1 ( x ) , . .
, z l,n l ( x )]
∈ R n l × n l be the diagonal matrix of gating function (for ReLU it is either 0 or 1), and g l ( x ) =
[ g l, 1 ( x ) , . . .
, g l,n l ( x )] (
cid:124) ∈
R n l be the backpropated gradient vector.
By deﬁnition, the input layer has f 0 ( x ) = x ∈ R n 0 and m 0 = n 0 .
Note that f l ( x ) , g l ( x ) and D l ( x ) are all dependent on W .
For brevity, we often use f l ( x ) rather than f l ( x ; W ) .
All notations with superscript ∗ are from the teacher, only dependent on the teacher and remains the same throughout the training.
D ∗ L ( x ) =
D L ( x ) ≡ I C × C since there is no ReLU gating.
Note that
C is the dimension of output for both teacher and student.
With the notation, gradient descent is:
[Equation]
In SGD, the expectation E x [ · ] is taken over a batch.
In GD, it is over the entire dataset.
Bias term .
With the same notation we can also include the bias term.
In this case, W l ∈ R ( n l − 1 +1) × n l , w l, 1 =
[ ˜ w ; b ] ∈ R n l − 1 +1 , f l ∈ R n l +1 (last column is all one), g l ∈ R n l +1 and
D l ∈ R ( n l +1) × ( n l +1) (last diagonal element is always 1 ).
Objective .
We assume that both the teacher and the student output a vector.
We use the output of teacher as the input of the student and the objective is:
[Equation]
We want to ask the following qeustion:
Are student nodes specialized to teacher nodes at the same layers after training?
[Equation]
One might wonder this is hard since the student’s intermediate layer receives no direct supervision from the corresponding teacher layer, but relies only on backpropagated gradient.
Surprisingly, the following theorem shows that it is possible for every intermediate layer:
Lemma 1 (Recursive Gradient Rule) .
At layer l , the backpropagated g l ( x ) satisﬁes
[Equation]
where the mixture coefﬁcient A l ( x ) =
V (cid:124) l ( x ) V ∗ l ( x ) ∈ R n l × m l and B l ( x ) =
V (cid:124) l ( x ) V l ( x ) ∈
R n l × n l .
The matrices V l ( x ) ∈ R C × n l and V ∗ l ( x ) ∈ R C
× m l are deﬁned in a top-down manner:
[Equation]
In particular, V L ( x ) =
V ∗ L ( x ) = I C × C .
For convenience, we can write V l ( x ) =
[ v l, 1 ( x ) , v l, 2 ( x ) , . .
. , v l,n l ( x )] , then we have each element of A l , α l,jj (cid:48)
( x ) = v (cid:124) l,j ( x ) v ∗ l,j (cid:48)
( x ) and element of B l , β l,jj (cid:48) ( x ) = v (cid:124) l,j ( x ) v l,j (cid:48)
( x ) .
Note that
Lemma 1 applies to arbitrarily deep ReLU networks and allows different number of nodes for the teacher and student.
In particular, student can be over-parameterized (or over-realized).
Let R
= { x : ρ ( x ) > 0 } be the inﬁnite training set, where ρ ( x ) is the input data distribution.
Let
R l = { f l ( x ) : x ∈ R 0 } , which is the image of the training set at the output of layer l , and also a convex polytope.
Then the mixture coefﬁcient A l ( x ) and B l ( x ) have the following property:
Corollary 1 (Piecewise constant) .
R 0 can be decomposed into a ﬁnite (but potentially exponential) set of regions R l − 1 = { R 1 l − 1 , R 2 l − 1 , . . .
, R Jl − 1 } .
A l ( x ) and B l ( x ) are constant in R jl − 1 .
C RITICAL P OINT A NALYSIS
We ﬁrst show that due to property of ReLU node and subset sampling in SGD, at SGD critical point, under mild condition, the teacher node aligns with at least one student node and the goal (*) can be reached in the lowest layer.
Deﬁnition 1 (SGD critical point) .
ˆ W is a SGD critical point if for any batch, ˙
W l = 0 for 1 ≤ l ≤ L .
Theorem 1 (Interpolation) .
Denote D = { x i } as a dataset of N samples.
If ˆ W is a critical point for SGD, then either g l ( x i ; ˆ W ) = 0 or f l − 1 ( x i ; ˆ W ) = 0 .
Such critical points exist since over-realized student can mimic teacher perfectly.
Note that critical points in SGD is much stronger than those in GD, where the gradient is always averaged at a ﬁxed data distribution.
If f l − 1 has a bias term (and f l − 1 (cid:54) = 0 always), then g l ( x i ; ˆ W ) = 0 .
For topmost layer, immediately we have g L ( x i ; ˆ W ) =
f ∗ L ( x i ) − f L ( x i ) = 0 , which is global optimum with zero training loss.
In the following, we want to check whether this condition leads to specialization, i.e., whether the teacher’s weights are recovered/aligned by the student, i.e., whether for teacher j , there exists a student k at the same layer so that w k = γ w j for some γ > 0 . {+Note that g l ( x i ; ˆ W ) = 
+}
might be a strong assumption since in practice the gradient is small but never zero.
A weaker assumption is that (cid:107) g l ( x i ; ˆ W ) (cid:107) ∞ ≤ (cid:15) or even E t (cid:104) (cid:107) g l ( x i ; ˆ W ) (cid:107) ∞ (cid:105) ≤ (cid:15) .
For this, Theorem 5 shows (approximate) alignment/specialization still holds for noisy case.
Iteration 0 0.030.
Iteration1.730.780.240. Iteration2.270.670.010. Iteration2.270.660.010. 4. A SSUMPTION OF TEACHER NETWORK
Obviously, an arbitrary teacher network won’t be reconstructed.
A trivial example is that a teacher network always output 0 since all the training samples lie in the inactive halfspace of its ReLU nodes.
Therefore, we need to impose condition on the teacher network.
Let E
j = { x : f j ( x ) > 0 } be the activation region of node j .
Note that the halfspace E j is an open set.
Let ∂E
j = { x : f j ( x ) = 0 } be the decision boundary of node j .
Deﬁnition 2 (Observer) .
Node k is an observer of node j if E k ∩ ∂E j (cid:54)
= ∅ .
Assumption 1 (Teacher Network) .
For each layer l , we require that (1) the teacher weights w ∗ l,j are not co-linear.
and (2) the boundary of w ∗ l,j is visible in the training set: ∂E ∗ l,j ∩ R l − 1 (cid:54) = ∅ .
Assumption 1 is our assumption of the teacher. The ﬁrst requirement is trivial.
The second one is reasonable since two teacher nodes who behaves linearly in the training set are indistinguishable.
4. A LIGNMENT OF TEACHER WITH STUDENT , 2- LAYER CASE
We ﬁrst start with 2-layer case, in which A 1 ( x ) and B 1 ( x ) are constant with respect to x , since there is no ReLU gating at the top layer l = 2 .
In this case, from the SGD critical point at l = 1 , g 1 ( x ) =
D 1 ( x ) [ A 1 f ∗ 1 ( x ) − B 1 f 1 ( x )]
= 0 , alignment between teacher and student can be achieved:
Theorem 2 (Student-teacher Alignment, 2-layers) .
With Assumption 1, at SGD critical point, if a teacher node j is observed by a student node k and α kj (cid:54) = 0 , then there exists at least one student node k (cid:48) aligned with j .
The intuition is that if the input x takes sufﬁciently diverse values, ReLU activations σ ( w (cid:124) k x ) can be proven to be mutually linear independent.
On the other hand, the gradient of each student node k when active , is α (cid:124) k
f 1 ( x ) − b (cid:124) k
f 1 ( x ) = 0 , a linear combination of teacher and student nodes (note α (cid:124) k and β (cid:124) k are k -th rows of A 1 and B 1 ).
Therefore, zero gradient means that the summation of coefﬁcients of co-linear ReLU nodes is zero.
Since teachers are not co-linear, any teacher node is co-linear with at least one student node.
Alignment with multiple student nodes is also possible.
If there is no nonlinearity (e.g., deep linear models), alignment won’t happen since a linear subspace has many representations.
Note that a necessary condition of a reconstructed teacher node is that its boundary is in the active region of student, or is observed (Deﬁnition 2).
This is intuitive since a teacher node which behaves like a linear node is partly indistinguishable from a bias term.
This also suggests that overparameterization (more student nodes) are important.
More student nodes mean more observers, and the existence argument in Theorem 4 is more likely to happen and more teacher nodes can be covered by student, yielding better generalization.
For student nodes that are not aligned with the teacher, if they are observed by other student nodes, then following a similar logic, we have the following:
Theorem 3 (Prunable Un-specialized Student Nodes) .
With Assumption 1, at SGD critical point, if an unaligned student k has C independent observers (concatenating v yields a full rank matrix), then (cid:80)[-k (cid:
-]
[Equation]
w [Equation]
= 0 . If node k is not co-linear with any other student, then v k = 0 .
Corollary 2.
With sufﬁcient observers, the contribution of all unaligned student nodes is zero.
Theorem 3 and Corollary 2 open the way of network pruning (LeCun et al., 1990;
Hassibi et al., 1993; Hu et al., 2016).
This is consistent with Theorem 5 in (Tian et al., 2019) which also shows the fan-out weights are zero up on convergence in 2-layer networks, if the initialization is close.
In contrast, Theorem 3 analyzes the critical point rather than the dynamics.
Note that a relate theorem (Theorem 6) in (Laurent & von Brecht, 2017) studies 2-layer network with scalar output and linear separable input, and discusses characteristics of individual data point contributing loss in a local minima of GD.
Here no linear separable condition is imposed.
4. M ULTI - LAYER CASE
Thanks to Lemma 1 which holds for deep ReLU networks, we can use similar intuition to analyze the behavior of the lowest layer ( l = 1 ) in the multiple layer case.
The difference here is that A 1 ( x ) and
B 1 ( x ) are no longer constant over x .
Fortunately, using Corollary 1, we know that A 1 ( x ) and B 1 ( x ) are piece-wise constant that separate the input region R 0 into a ﬁnite (but potentially exponential) set of constant regions R
= { R 10 , R 20 , . . .
, R J 0 } plus a zero-measure set.
This suggests that we could check each region separately.
If the boundary of a teacher j and a student k lies in the region, similar logic applies (here α kj is the ( k, j ) entry of A 1 ( x ) and is constant in a region R ∈ R 0 ).
Theorem 4 (Student-teacher Alignment, Multiple Layers) .
With Assumption 1, at SGD critical points, for any teacher node j at l = 1 , if there exists a region R ∈ R and a student observer k so that ∂E ∗ j ∩ E k ∩ R (cid:54) = ∅ and α kj ( R ) (cid:54) = 0 , then node j aligns with at least one student node k (cid:48) .
Note that even with random V 1 ( x ) (e.g., at initialization), Theorem 4 still holds with high probability (when α kj (cid:54) = 0 ) and teacher f ∗ 1 ( x ) can still align with student f 1 ( x ) .
This suggests a picture of bottom-up training in backpropagation: After the alignment of activations at layer 1 , we just treat layer 1 as the low-level features and the procedure repeats until the student matches with the teacher at all layers.
This is consistent with many previous works that empirically show the network is learned in a bottom-up manner (Li et al., 2018).
Note that the alignment may happen concurrently across layers: if the activations of layer 1 start to align, then activations of layer 2 , which depends on activations of layer 1 , will also start to align since there now exists a W 2 that yields strong alignments, and so on.
This creates a critical path from important student nodes at the lowest layer all the way to the output, and this critical path accelerates the convergence of that student node.
We leave a formal analysis to the future work.
Small Gradient Case .
In practice, stochastic gradient (or its expectation over time) ﬂuctuates around zero ( (cid:107) g 1 ( x ) (cid:107) ∞ ≤ (cid:15) , or E t [ (cid:107) g 1 ( x ) (cid:107) ∞ ] ≤ (cid:15) ), but never zero.
In this case, Theoremshows that a rough specialization still follows.
The ratio of recovery is also shown for weights/biases separately, as a function of (cid:15) .
Note ˜ θ jj (cid:48) is the angle of two weights ˜ w j and ˜ w j (cid:48) .
Theorem 5 (Noisy Recovery) .
If Assumption 1 holds and any two teachers w ∗ j , w ∗ j (cid:48) satisfy ˜ θ jj (cid:48) ≥ θ 0 > 0 or | b ∗ j (cid:48)
− b ∗ j | ≥ b 0 > 0 .
Suppose (cid:107) g 1 ( x , ˆ W ) (cid:107) ∞ ≤ (cid:15) for any x ∈ R 0 with (cid:15) ≤ (cid:15) 0 , then for any teacher j at l = 1 , if there exists a region R ∈ R and a student observer k
so that ∂E ∗ j ∩ E k ∩ R (cid:54) = ∅ , and α kj ( R ) (cid:54) = 0 , then j is roughly aligned with a student k (cid:48) : sin θ jk (cid:48)
=
O (cid:16) (cid:15) 1 − δ | α kj | (cid:17) and | b ∗ j − b k (cid:48) | =
O (cid:16) (cid:15) 1 − 2 δ | α kj | (cid:17) for any δ > 0 .
The hidden constants depends on δ , (cid:15)
and the size of region ∂E ∗ j ∩ E k ∩ R .
Note that E t [ (cid:107) g 1 ( x ) (cid:107) ∞ ] ≤ (cid:15) leads to (cid:107) g 1 ( x ) (cid:107) ∞ ≤ (cid:15) at least for some iteration t .
Therefore,
Theorem 5 still applies since it does not rely on past history of the weight/gradient.
Note that
Theorem 5 assumes inﬁnite number of data points and leave ﬁnite sample case to future work.
A NALYSIS ON T RAINING D YNAMICS
Our analysis so far shows student specialization happens at SGD critical points under mild conditions.
A natural question arises: is running SGD long enough sufﬁcient to achieve these critical points?
Some previous works (Ge et al., 2017; Livni et al., 2014) show that empirically SGD does not recover the parameters of a teacher network up to permutation, while other works (Saad & Solla, 1996; Goldt et al., 2019) show specialization happens. Why there is a discrepancy?
There are several reasons.
First, from Theorem 3, there exist un-specialized student nodes, so a simple permutation test on student weights might fail.
Second, as suggested by Theorem 5, it can take a long time to recover a teacher node k with small (cid:107) v ∗ k (cid:107) (since α kj = v ∗ (cid:124) k v j ).
In fact, if v ∗ k =
then it has no contribution to the output and recovery never happens.
This is particularly problematic if the output dimension is 1 (scalar output), since a single small teacher weight v ∗ k would block the recovery of the entire teacher node k .
Previous works (Lampinen & Ganguli, 2019) shows similar behaviors in the dynamics of singular values in deep linear networks in teacher-student setting, which lack student specialization.
Here we study these behaviors in deep ReLU networks. In the following, we analyze various local dynamic behaviors of 2-layer ReLU network.
Due to the complexity, we leave a formal characterization of the entire training procedure for future work.
Deﬁnition 3.
A teacher node j is strong (or weak), if (cid:107) v ∗ j (cid:107) is large (or small).
In this case, the dynamics can be written as the following:
[Equation]
where V 1 and V ∗ 1 are constant, α k = V ∗ (cid:124) l v k ,
β k = V
[Equation] and residue [Equation] 5. W EIGHT MAGNITUDE
From Eqn.
5, we know that for both ReLU and linear network (since f k ( x ) =
z k ( x ) w (cid:124) k
f l − 1 ( x ) ):
[Equation]
When there is only a single output, r l is a scalar and Eqn.
6 is simply an inner product between the residue and the activation of node k , over the batch.
So if the node k has activation which aligns well with the residual, the inner product is larger and (cid:107) w k (cid:107) grows faster.
5. A NGLES BETWEEN T EACHER AND S TUDENT WEIGHTS
Note that Eqn. 6 only tell that the weight norm would increase, but didn’t tell whether w k converges to any teacher node w ∗ j .
It could be the case that (cid:107) w k (cid:107) goes up but doesn’t move towards the teacher.
To see that, let’s check the quantity:
[Equation]
where G kj = E x (cid:2)
f l − 1 z k z ∗ j f (cid:124)
l − 1 (cid:3) .
Putting it in another way, we want to check the spectrum property of the PSD matrix G kj .
Intuitively, the direction of E x (cid:2)
f l − 1 z k f ∗ j (cid:3) should lie between w k and w ∗ j , and the magnitude is large when w k and w ∗ j are close to each other.
This means that if r is dominated by a teacher j (i.e., (cid:107) v ∗ j (cid:107) is large), then ˙ w k would push w k towards w ∗ j .
This also shows that SGD will ﬁrst try ﬁtting strong teacher nodes, then weak teacher nodes.
Theorem 6 conﬁrms this intuition if f l − 1 follows spherical symmetric distribution (e.g., N (0 , I ) ).
Theorem 6.
If f l − 1 follows spherical symmetric distribution, then E x (cid:2)
f l − 1 z k f ∗ j (cid:3) ∝
[Equation]
w k (cid:107)(cid:2) ( π − θ ) w ∗ j + sin θ w k (cid:3) , where θ is the angle between w ∗ j and w k .
As a result, for all θ ∈ [0 , π ] , E x (cid:2) f l − 1 z k f ∗ j (cid:3) is always between w ∗ j and w k since π − θ and sin θ are always non-negative.
Without such symmetry, we assume the following holds:
Assumption 2.
E x
[ f l [Equation] 1 z k f j ] =
[Equation]
w [Equation] k , where ψ ( π ) = 0 .
Note that critical point analysis is applicable to any batch size, including 1 .
On the other hand,
Assumption 2 holds when a moderately large batchsize leads to a decent estimation of the terms.
With this assumption, we can write the dynamics as ˙ w k = (cid:107) w k (cid:107) r k , where the time-varying residue r k of node k is deﬁned as the following ( ν is a scalar related to ψ (cid:48) ):
[Equation]
5. S YMMETRIC BREAKING , W INNERS - TAKE - ALL AND F OCUS S HIFTING
We could show that for two nodes k (cid:54) =
k (cid:48) , regardless of the form of r k , we have (note that ¯ w is the length-normalized version of w ):
[Figure]
Theorem 7.
For dynamics
[Equation] [Figure]
[Equation]
[Figure] [Equation] [Figure]
[Equation]
[Figure]
[Equation] [Figure] [Equation] [Figure]
[Equation]
[Figure]
We consider a special (and symmetric) case: r k
= r = w ∗ − (cid:80) k
a k w k with all a k > 0 , where w ∗ is a joint contribution of all teacher nodes.
In this case, we could show that when ¯ w (cid:124) k r k > ¯ w (cid:124) k (cid:48) r k (cid:48) , dd t ( ¯ w (cid:124)
k
r k − ¯ w (cid:124) k (cid:48)
r k (cid:48) )
< 0 and vice versa.
So the system provides negative feedback until ¯ w k = ¯ w k (cid:48) and according to Eqn. 7, the ratio between (cid:107) w k (cid:107) and (cid:107) w k (cid:48) (cid:107) remains constant, after initial transition.
We can also show that ¯ w k will align with w ∗ and every student node goes to w ∗ .
However, due to Theorem 6, the net effect w ∗ can be different for different students and thus r k are different.
This opens the door for complicated dynamic behavior of neural network training.
Symmetry breaking .
As one example, if we add a very small delta to some node, say k = 1 so that r 1 = r + (cid:15) w ∗ .
Then to make dd t ( ¯ w (cid:124)
k r k − ¯ w (cid:124) k (cid:48)
r k (cid:48) )
= 0 , we have ¯ w (cid:124) k r k > ¯ w (cid:124) k (cid:48)
r k (cid:48) and thus according to Theorem 7, (cid:107) w k (cid:107) / (cid:107) w k (cid:48) (cid:107) grows exponentially.
This symmetric breaking behavior provides a potential winners-take-all mechanism, since according to Theorem 6, the coefﬁcient of w ∗ depends critically on the initial angle between w k and w ∗ .
Strong teacher nodes are learned ﬁrst .
If (cid:107) v ∗ j (cid:107) is the largest among teacher nodes, then the joint w ∗ heavily biases towards teacher j and all student nodes move towards teacher j .
As a result, strong teacher learns ﬁrst and is often covered by multiple co-linear students (Fig. 6, teacher-0).
Focus shifting to weak teacher nodes .
The process above continues until residual along the direction of w ∗ j quickly shrinks and residual corresponding to other teacher node (e.g., w ∗ j (cid:48) for j (cid:48) (cid:54) =
j ) becomes dominant.
Since each r k is different, student node k whose direction is closer to w ∗ j (cid:48)
( j (cid:48) (cid:54) =
j ) will shift their focus towards w ∗ j (cid:48) , as shown in the green (shift to teacher-2) and magenta (shift to teacher-5) curves in Fig.
Possible slow convergence to weak teacher nodes .
While expected angle between two weights from initialization is π/ 2 , shifting a student node w k from chasing after a strong teacher node w ∗ j to a weaker one w ∗ j (cid:48) could yield a large initial angle (e.g., close to π ) between w k and w j (cid:48) .
For example, all student nodes have been attracted to the opposite direction of a weak teacher node.
In this case, the convergence can be arbitrarily slow.
In fact, if there is only one teacher node and θ is the angle between teacher and student, then from Eqn.
8 we arrive at ˙ θ ∝ − ψ ( θ ) sin θ .
Since ψ ( θ ) sin θ ∼ ( π − θ ) 2 around θ = π , the time spent from θ = π − (cid:15) to some θ 0 is t 0 ∼ 1 (cid:15) − [Table] → + ∞ when (cid:15) → 0 .
In this case, over-realization helps by having more student nodes that are possibly ready for shifting towards weaker teachers, and thus accelerate convergence (Fig. 7).
Alternatively, we could reinitialize those student nodes (Prakash et al., 2019).
E XPERIMENTS
We ﬁrst verify our theoretical ﬁnding on synthetic dataset.
We generate the input using N (0 , σ 2 I ) with σ = [-1 . 
-]10 and we sample 10k as training and another 10k as evaluation.
For deep ReLU networks, we regenerate the dataset after every epoch to mimic inﬁnite sample setting.
The details of teacher/student construction is in Appendix (Sec. 8.16).
The normalized correlation between nodes is computed in terms of activation vectors evaluated on a separate evaluation set.
Two layer networks .
First we verify Theorem 2 and Theorem 3 in the 2-layer setting.
Fig.
6 shows student nodes correlate with different teacher nodes over time.
Fig.
3 shows for different degrees of
[Figure] [Figure]
over-realization ( 1 × / 2 × / 5 × / 10 × ), for nodes with weak specialization (i
[Figure]
e., its normalized correlation to the most correlated teacher is low), their magnitudes of fan-out weights are small.
Otherwise the nodes with strong specialization have high fan-out weights.
Deep Networks .
For deep ReLU networks, we observe specialization not only at the lowest layer, as suggested by Theorem [Figure] but also at multiple hidden layers.
This is shown in Fig.
For each student node k , the x-axis is its best normalized correlation to teacher nodes [Figure] and y-axis is (cid:112) E
x [ β kk ( x )] , which is equivalent to (cid:107) v k (cid:107) in 2-layer case.
In the plot, we can also see the lowest layer learns ﬁrst (the “L-shape” curve was established at epoch 10), then the top layers follow.
Ablation on the effect of over-realization .
To further understand the role of over-realization, we plot the average rate of a teacher node that is matched with at least one student node successfully (i.e., correlation > 0 . 95 ).
Fig.
5 shows that stronger teacher nodes are more likely to be matched, while weaker ones may not be explained well, in particular when the strength of the teacher nodes are polarized ( p is large).
Over-realized student can explain more teacher nodes, while a student with 1 × nodes has sufﬁcient capacity to ﬁt the teacher perfectly, it gets stuck despite long training.
In addition, the evaluation loss (Appendix Fig. 11) shows that over-realization yields better generalization, in particular with large teacher node polarity ( p is large), where weak teacher nodes are hard to capture.
For good performance on real datasets, getting weak teacher nodes can be important.
Training Dynamics .
We set up a diverse strength of teacher node by constructing the fanout weights of teacher node j as follows:
[Equation]
where p is the teacher polarity factor that controls how strong the energy decays across different teacher nodes.
p = 0 means all teacher nodes are symmetric, and large p means that the strength of teacher nodes are more polarized.
[Figure]
[Figure] [Figure]
t i o
[Figure]
f b e s t
[Figure]
t
[Figure]
d e n t t
[Figure]
t
[Figure]
a c
[Figure]
e r
[Figure]
Fig. 6 and Fig.
7 show that many student nodes specialize to a strong teacher node ﬁrst.
Once the strong teacher node was covered well, weaker teacher nodes are covered after many epochs.
CIFAR-10 .
We also experiment on CIFAR-10.
We ﬁrst pre-train a teacher network with 64-64-6464 ConvNet ( 64 are channel sizes of the hidden layers, L = 5 ) on CIFAR-10 training set.
Then the teacher network is pruned in a structured manner to keep strong teacher nodes.
The student is over-realized based on teacher’s remaining channels.
The convergence and specialization behaviors of student network is shown in Fig.
Specialization happens at all layers for different degree of over-realization.
Over-realization boosts student specialization, measured by mean of maximal normalized correlation ρ mean
= mean j ∈ teacher max j (cid:48) ∈ student ˜ f ∗ (cid:124)
j ˜ f j (cid:48) at each layer ( ˜ f
j is the normalized activation of node j over
N evaluation samples), and improves generalization, evaluated on CIFAR-10 evaluation set.
C ONCLUSION AND F UTURE W ORK
In this paper, we use student-teacher setting to analyze how an (over-parameterized) deep ReLU student network trained with SGD learns from the output of a teacher.
When the magnitude of gradient per sample is small (student weights are near the critical points), the teacher can be proven to be covered by (possibly multiple) students and thus the teacher network is recovered in the lowest layer.
By analyzing training dynamics, we also show that strong teacher node with large (cid:107) v ∗ (cid:107) is reconstructed ﬁrst, while weak teacher node is reconstructed slowly.
This reveals one important reason why the training takes long to reconstruct all teacher weights and why generalization improves with more training.
As the next step, we would like to extend our analysis to ﬁnite sample case, and analyze the training dynamics in a more formal way.
Verifying the insights from theoretical analysis on a large dataset (e.g., ImageNet) is also the next step.
[Figure]
A PPENDIX 8. L EMMA
Proof.
We prove by induction.
When l =
L we know that g L ( x ) =
f ∗ L ( x ) − f L ( x ) , by setting
V ∗ L ( x ) =
V L ( x ) = I C × C and the fact that D L ( x )
= I C × C
(no ReLU gating in the last layer), the condition holds.
Now suppose for layer l
[Equation]
we have:
[Equation]
Using
[Equation]
[Equation]
we have:
[Equation]
8. T HEOREM
Proof.
By deﬁnition of SGD critical point, we know that for any batch B j , Eqn. 1 vanishes:
[Equation]
) where U i = g l ( x i ; ˆ W ) f (cid:124) l − 1 ( x i ; ˆ W ) .
Note that B j can be any subset of samples from the data distribution.
Therefore, for a dataset of size N , Eqn. 20 holds for all (cid:0) N |B| (cid:1) batches, but there are only N data samples.
With simple Gaussian elimination we know that for any i 1 (cid:54)
=
i 2 , U i
=
U
i
= U .
Plug that into Eqn.
we know U
= 0 and thus for any i , U i = 0 .
Since U i is an outer product, the theorem follows.
Note that if (cid:107) ˙
W l (cid:107) ∞ ≤ (cid:15) , which is (cid:107) (cid:80) i ∈B j U i (cid:107) ∞ ≤ (cid:15) , then with simple Gaussian elimination for two batches B 1 and B 2 with only two sample difference, we will have for any i 1 (cid:54) =
i 2 , (cid:107) U
i 1 −
U i 2 (cid:107) ∞
= (cid:107) (cid:80) i ∈B 1 U i − (cid:80) i ∈B 2 U i (cid:107) ∞ ≤ (cid:107) (cid:80) i ∈B 1 U i (cid:107) ∞ + (cid:107) (cid:80) i ∈B 2 U i (cid:107) ∞ = 2 (cid:15) .
Plug things back in and we have |B|(cid:107)
U i (cid:107) ∞
≤ [2( |B| − 1)
+ 1] (cid:15) , which is (cid:107)
U i (cid:107) ∞ ≤ 2 (cid:15) .
If f l − 1 ( x ; ˆ W ) has the bias term, then immediately we have (cid:107) g l ( x ; ˆ W ) (cid:107) ∞ ≤ (cid:15) .
[Figure]
8. C OROLLARY
Proof.
The base case is that V L ( x ) =
V ∗ L ( x ) = I C × C , which is constant (and thus piece-wise constant) over the entire input space.
If for layer l , V l ( x ) and V ∗ l ( x ) are piece-wise constant, then by Eqn.
4 (rewrite it here):
[Equation]
since D l ( x ) and D ∗ l ( x ) are piece-wise constant and W (cid:124) l and W ∗ (cid:124) l are constant, we know that for layer l − 1 , V l − 1 ( x ) and V ∗
l − 1 ( x ) are piece-wise constant.
Therefore, for all l = 1 , . . .
L , V l ( x ) and
V ∗ l ( x ) are piece-wise constant.
Therefore, A l ( x ) and B l ( x ) are piece-wise constant with respect to input x .
They separate the region R 0 into constant regions with boundary points in a zero-measured set.
8. L EMMA
Lemma 2.
Consider K ReLU activation functions f j ( x ) =
σ ( w (cid:124) j x ) for j = 1 . . .
K .
If w j (cid:54) =
and no two weights are co-linear, then (cid:80) j (cid:48)
c j (cid:48)
f j (cid:48)
( x ) =
for all x ∈ R
d
+1 suggests that all c j = 0 .
Proof.
Suppose there exists some c j (cid:54) =
so that (cid:80) j
c j f j ( x ) = 0 for all x .
Pick a point x 0 ∈ ∂E
j
so that w (cid:124) j x 0 =
but all w (cid:124) j (cid:48)
x 0 (cid:54) = 0 for j (cid:48) (cid:54) =
j , which is possible due to the distinct weight conditions.
Consider an (cid:15) -ball B x 0 ,(cid:15)
= { x : (cid:107) x − x 0 (cid:107) ≤ (cid:15) } .
We pick (cid:15) so that sign( w (cid:124) j (cid:48) x ) for all j (cid:48)
(cid:54) =
j remains the same within B x 0 ,(cid:15) (Fig. 9(a)).
Denote [ j + ] as the indices of activated ReLU functions in B x 0 ,(cid:15) except j .
Then for all x ∈ B x 0 ,(cid:15) ∩ E j , we have:
[Equation]
Since B x 0 ,(cid:15) is a d -dimensional object rather than a subspace, for x 0 and x 0 + (cid:15) e k ∈ B ( x 0 , (cid:15) ) , we have
[Equation]
where e k is axis-aligned unit vector ( 1 ≤ k ≤ d ).
This yields
[Equation] [Equation]
Plug it back to Eqn.
22 yields
[Equation]
where means that for the (augmented) d + 1 dimensional weight:
[Equation]
However, if we pick x (cid:48) = x 0 − (cid:15) ˜ w j (cid:107)
[Equation]
w j (cid:107) 2 ∈ B x 0 ,(cid:15) ∩ E (cid:123) j , then f j ( x (cid:48) )
=
but (cid:80) j (cid:48) ∈
[ j + ] f (cid:48)
j ( x (cid:48) )
=
− c j w (cid:124)
j x (cid:48) =
(cid:15)c j and thus (cid:88)
[Equation]
(cid:48)
[Equation]
(27) which is a contradiction.
8. L EMMA
Lemma 3 (Local ReLU Independence) .
Let R be an open set.
Consider K ReLU nodes f j ( x ) = σ ( w (cid:124)
j x ) ,
j = 1 , . .
, K .
w j (cid:54) = 0 , w j (cid:54) =
γ w j (cid:48) for j (cid:54) =
j (cid:48) with any γ > 0 .
If there exists c 1 , . .
c K , c • so that the following is true:
[Equation]
[Equation]
and for node j , ∂E j ∩ R (cid:54) = ∅ , then c j = 0 .
Proof.
We can apply the same logic as Lemma 2 to the region R (Fig. 9(b)).
For any node j , since its boundary ∂E
j is in R , we can ﬁnd a similar x 0 so that x 0 ∈ ∂E
j ∩ R and x 0 / ∈ ∂E
j (cid:48) for any j (cid:48) (cid:54) =
j .
We construct B x 0 ,(cid:15) .
Since R is an open set, we can always ﬁnd (cid:15) >
so that B x 0 ,(cid:15) ⊆ R and no other boundary is in this (cid:15) -ball.
Following similar logic of Lemma 2, c j = 0 .
8. L EMMA
Lemma 4 (Relation between Hyperplanes) .
Let w j and w j (cid:48) two distinct hyperplanes with (cid:107) ˜ w j (cid:107)
=
(cid:107) ˜ w j (cid:48) (cid:107) = 1 .
Denote θ jj (cid:48) as the angle between the two vectors w j and w j (cid:48) .
Then there exists ˜ u j (cid:48) ⊥ ˜ w j and w (cid:124) j (cid:48)
˜
u j (cid:48)
=
sin θ jj (cid:48) .
Proof.
Note that the projection of ˜ w j (cid:48) onto ˜ w j is:
[Equation]
It is easy to verify that (cid:107) ˜
u j (cid:48) (cid:107) =
1 and w (cid:124) j (cid:48)
˜
u j (cid:48)
=
sin θ jj (cid:48) .
8. L EMMA
Lemma 5 (Evidence of Data points on Misalignment) .
Let R ⊂ R
d be an open set.
Consider K
ReLU nodes f j ( x ) = σ ( w (cid:124) j x ) , j = 1 , . . .
, K .
(cid:107) ˜ w j (cid:107) = 1 , w j are not co-linear.
Then for a node j with ∂E j ∩ R (cid:54) = ∅ , and (cid:15) ≤ (cid:15) 0 , either of the conditions holds:
(1) There exists node j (cid:48)
(cid:54) =
j so that sin θ jj (cid:48) ≤ MK(cid:15) 1 − δ / | c j | and | b j (cid:48)
− b j | ≤ M 2 (cid:15) 1 − δ / | c
j | There exists x j ∈ ∂E
j ∩ R so that for any j (cid:48) (cid:54) =
j , | w (cid:124) j (cid:48)
x j | > 5 (cid:15)/ | c j |
where θ jj (cid:48) is the angle between ˜ w j and ˜ w j (cid:48) , δ > 0 , r is the radius of a d − 1 dimensional ball contained in ∂E j ∩ R , M = 10 (cid:15) δ 0 r (cid:113) d 2 π , M 0 =
max x ∈ ∂E
j ∩ R (cid:107) x (cid:107) and M 2 = 2 M 0 MK(cid:15)
δ 0 + 5 (cid:15) 2 δ 0 .
Proof.
Deﬁne q j = 5 (cid:15)/ | c j | .
For each j (cid:48) (cid:54) =
j , deﬁne I j (cid:48)
= { x : | w (cid:124)
j (cid:48) x | ≤ q j , x ∈ ∂E j } .
We prove by contradiction.
Suppose for any j (cid:48) (cid:54) =
j , sin θ jj (cid:48) > KM(cid:15) 1 − δ / | c j | or | b j (cid:48)
− b j | > M 2 (cid:15) 1 − 2 δ / | c j | .
Otherwise the theorem already holds.
Case 1.
When sin θ jj (cid:48) > KM(cid:15) 1 − δ / | c
j | holds .
From Lemma 4, we know that for any x ∈ ∂E
j , if w (cid:124) j (cid:48) x
= − q j , with a j (cid:48) ≤ 2 q j |
c
j | /MK(cid:15) 1 − δ = 10 (cid:15) δ /MK , we have x (cid:48)
=
x + a j (cid:48)
u j (cid:48) ∈ ∂E
j and w (cid:124) j (cid:48) x (cid:48) =
+ q j .
Consider a d − 1 -dimensional sphere B ⊆ Ω j and its intersection of I j (cid:48)
∩ B for j (cid:48) (cid:54) =
j .
Suppose the sphere has radius r .
For each I j (cid:48)
∩ B , its d − 1 -dimensional volume is upper bounded by:
[Equation]
) where V d − 2 ( r ) is the d − 2 -dimensional volume of a sphere of radius r .
Intuitively, the intersection between w (cid:124) j (cid:48)
x = − q j and B is at most a d − 2 -dimensional sphere of radius r , and the “height” is at most a j (cid:48) .
Case 2.
When sin θ jj (cid:48) ≤ KM(cid:15) 1 − δ / | c j | but | b j (cid:48) − b j | > M 2 (cid:15) 1 − 2 δ / | c j | holds.
In this case, we want to show that for any x ∈ Ω j , | w (cid:124) j (cid:48)
x | > q j and thus I j (cid:48)
∩ B = ∅ .
If this is not the case, then there exists x ∈ Ω j so that | w (cid:124) j (cid:48) x | ≤ q j .
Then since x ∈ ∂E j , we have:
[Equation]
Therefore, from Cauchy inequality and triangle inequality, we have:
[Equation]
From the condition, we have (cid:107) ˜ w j (cid:48) − ˜ w j (cid:107) = 2 sin θ jj (cid:48) 2 ≤ 2 sin θ jj (cid:48) ≤ 2 KM(cid:15) 1 − δ /
| c j | .
Then
[Equation]
MK(cid:15)
[Equation]
which is equivalent to:
[Equation]
34) which means that
[Equation]
35) for (cid:15) ≤ (cid:15) 0 .
This is a contradiction.
Therefore, I j (cid:48) ∩ B = ∅ and thus V ( I j (cid:48)
∩ B )
= 0 .
Volume argument.
Therefore, from the deﬁnition of M , we have V ( B ) =
V d − 1 ( r ) ≥ r (cid:113) 2 πd V d −
[Equation]
r ) =
[Equation]
δ 0 V d − 2 ( r ) , then for (cid:15) ≤ (cid:15) 0 , we have:
[Equation]
[Equation]
This means that there exists x j ∈ B ⊆ Ω j so that x j / ∈ I j (cid:48)
∩ B for any j (cid:48)
(cid:54) =
j and j (cid:48) in case 1 .
That is,
[Equation]
On the other hand, for j (cid:48) in case 2 , the above condition holds for entire Ω j , and thus hold for the chosen x j .
8. L EMMA
Lemma 6 (Local ReLU Independence, Noisy case) .
Let R be an open set.
Consider K ReLU nodes f j ( x ) = σ ( w (cid:124)
j x ) ,
j = 1 , . .
, K .
(cid:107) ˜ w j (cid:107) = 1 , w j are not co-linear.
If there exists c 1 , . .
c K , c • and (cid:15) ≤ (cid:15)
so that the following is true:
[Equation]
and for a node j , ∂E j ∩ R (cid:54) = ∅ .
Then there exists node j (cid:48)
(cid:54) =
j so that sin θ jj (cid:48) ≤ MK(cid:15) 1 − δ / | c j | and | b j (cid:48)
− b j | ≤ M 2 (cid:15) 1 − 2 δ / | c j | , where r, δ, M, M 2 are deﬁned in Lemma 5 but with r (cid:48)
=
r − 5 (cid:15)/ | c j | .
Proof.
Let q j = 5 (cid:15)/ | c j | and Ω j = { x : x ∈ ∂E
j ∩ R, B ( x , q j ) ⊆ R } .
If situation (1) in Lemmahappens then the theorem holds.
Otherwise, applying Lemma 5 with R (cid:48)
= { x : x ∈ R, B ( x , q j ) ⊆
R } and there exists x j ∈ Ω
j so that
[Equation]
[Figure]
Let two points x ±
j = x
j ± q j ˜ w j ∈ R .
In the following we show that the three points x j and x ± j are on the same side of ∂E j (cid:48) for any j (cid:48) (cid:54) =
j .
This can be achieved by checking whether
[Equation]
Since | w (cid:124) j (cid:48)
˜ w j | ≤ 1 , it is clear that ( w (cid:124) j (cid:48)
x j )( w (cid:124) j (cid:48) x ± j ) ≥ 0 .
Therefore the three points x j and x ± j are on the same side of ∂E j (cid:48) for any j (cid:48)
(cid:54) =
j .
Let h ( x ) =
(cid:80)
j c j f j ( x ) + c • w (cid:124) • x , then | h ( x ) | ≤ (cid:15) for x ∈ R .
Since x + j + x − j = 2 x j , we know that all terms related to w • and w j (cid:48) with j (cid:54) =
j will cancel out (they are in the same side of the boundary ∂E j (cid:48) ) and thus:
[Equation]
which is a contradiction.
8. T HEOREM
Proof.
In this situation, because D 2 ( x ) = D ∗ 2 ( x ) =
I , according to Eqn. 4, V 1 ( x ) =
W (cid:124) 1 and
V ∗ 1 ( x ) =
W ∗ (cid:124) 1 are independent of input x .
Therefore, both A 1 and B 1 are independent of input x .
From Assumption 1, since ρ ( x ) > 0 in R 0 , from Theorem.
1 we know that the SGD critical points gives g 1 ( x ) =
D 1 ( x ) [ A 1 f ∗ 1 ( x ) − B 1 f 1 ( x )]
= 0 .
Picking node k , the following holds for every node k and every x ∈ R
∩ E k :
[Equation]
Here α (cid:124) k is the k -th row of A 1 , A 1 =
[ α 1 , . .
α n 1 ] (cid:124) and similarly for β (cid:124) k .
Note here layer index l
= 1 is omitted for brevity.
For teacher j , suppose it is observed by student k , i.e., ∂E ∗ j ∩ E k (cid:54) = ∅ .
Given all teacher and student nodes, note that co-linearity is a equivalent relation, we could partition these nodes into disjoint groups.
Suppose node j is in group s .
In Eqn.
44, if we combine all coefﬁcients in group s together into one term c s w ∗ j (with (cid:107) w ∗ j (cid:107)
= 1 ), we have:
[Equation]
“At most” because from Assumption 1, all teacher weights are not co-linear.
Note that co - linear( j ) might be an empty set.
By Assumption 1, ∂E ∗ j ∩ R 0 (cid:54) = ∅ and by observation property, ∂E ∗ j ∩ E k (cid:54) = ∅ , we know that for
R = R 0 ∩ E k , ∂E ∗ j ∩ R (cid:54) = ∅ .
Applying Lemma 3, we know that c
s = 0 .
Since α kj (cid:54) = 0 , we know co - linear( j ) (cid:54) = ∅
and there exists at least one student k (cid:48) that is aligned with the teacher j .
8. T HEOREM
Proof.
We basically apply the same logic as in Theorem 2.
Consider the colinear group co - linear( k ) .
If for all k (cid:48) ∈ co - linear( k ) ,[-β k (cid:48) k (cid:48) ≡ (cid:107) v k (cid:48) (cid:107) 
-]
[Equation]
= 0 , then v k (cid:48)
=
and the proof is complete.
Otherwise, if there exists some student k
so that v k (cid:54) = 0 .
By the condition, it is observed by some student node k o , then with the same logic we will have
[Equation]
[Equation]
which is
[Equation]
[Equation]
Since k is observed by C students k 1 o , k 2 o , . .
k Jo , then we have:
[Equation]
By the condition, all the C vectors v (cid:124) k jo ∈ R C are linear independent, then we know that
[Equation]
8. C OROLLARY
Proof.
We can write the contribution of all student nodes which are not aligned with any teacher nodes as follows:
[Equation]
where w (cid:48) s is the unit vector that represents the common direction of the co-linear group s .
From
Theorem 3, for group s that is not aligned with any teacher, (cid:80) k ∈ co - linear( s ) v k (cid:107) w k (cid:107) = 0 and thus the net contribution is zero.
8. T HEOREM
Proof.
In multi-layer case, A l ( x ) and B l ( x ) are no longer constant over input x .
Fortunately, thanks to the recursive deﬁnition (Eqn. 4) which only contains input-independent terms (weights) and gating function, A l ( x ) and B l ( x ) are piece-wise constant function over the input R 0 .
Note that R 0 can be partitioned into R =
{ R 10 , R 20 , . . .
, R J 0 } and a zero-measure set.
Each of them is constant region for A l ( x ) and B l ( x ) .
Since R j 0 is an intersection of ﬁnite open half-planes (from k ’s parent nodes), R j 0 is still an open set.
From the condition, there exists open set R ∈ R and a student observer node k so that ∂E ∗ j ∩ E k ∩
R (cid:54) = ∅ ((Fig. 9(c)).
Let H R and similarly H ∗ R be the student and teacher nodes whose boundary intersects with R .
Therefore j ∈ H ∗ R .
For other teacher/student nodes, they are linear functions within R and thus can be combined together into w (cid:124) • x .
For all weights in H R , H ∗ R and w • , applying
Lemma 3 on R ∩ E k , we know that the SGD critical point α (cid:124) R,k f ∗ 1 ( x ) − β
(cid:124) R,k f 1 ( x ) =
leads to alignment between H R and H ∗ R .
Let group s be the one that contains all weights that are co-linear to teacher node j (note that no other teacher nodes are involved), and c s its coefﬁcient.
Since j ∈ H ∗ R , c s = 0 .
Since α kj ( R ) (cid:54) = 0 , there exists at least one student node k (cid:48) that is co-linear to teacher node j .
8. T HEOREM
Proof.
We follow the logic of Theorem 4.
Instead of applying Lemma 3, for gradient that is not zero but bounded within (cid:15) , we pick the student observer k
and we have for E k ∩ R :
[Equation]
(52) we use Lemma 6 and know that there exists a node k (cid:48)
(cid:54) =
j so that sin θ k [Equation] j
[Equation] (cid:15) 1 − δ / | c j | (cid:1) and | b k (cid:48) − b ∗ j | =
O (cid:0) (cid:15) 1 − 2 δ / | c j | (cid:1) for any δ > 0 .
Under the observation of student k , the teacher j has coefﬁcient c j = α kj .
Since all teacher weights are distant to each other with positive constant b 0 > 0 and θ 0 > 0 , with sufﬁciently small (cid:15) 0 and (cid:15) ≤ (cid:15) 0 , this node k (cid:48) has to be a student node and the bound follows.
8. T HEOREM
Proof.
From the expression we can see that it is positive homogeneous with respect to (cid:107) w ∗ j (cid:107) and (cid:107) w k (cid:107) .
So we can assume (cid:107) w ∗ j (cid:107) =
(cid:107) w k (cid:107) = 1 .
Without loss of generality, we set up the coordinate system so that w ∗ j = [1 , 0] (cid:124) and w k =
[cos θ, sin θ ] (cid:124) .
Then
[Equation]
where (cid:15) is the term reﬂecting the asymmetry of the data distribution p ( f l − 1 ) with respect to the plane spanned by the vectors w k and w ∗ j .
If the data distribution p ( f l − 1 ) is scale invariant (rescaling the data point won’t change the angular distribution), then p ( θ (cid:48) | r ) =
p ( θ (cid:48) )
and we only need to check the angular integral:
[Equation]
Note that cos 2 θ = 12 (1 + cos 2 θ ) and sin θ cos θ = 12 sin 2 θ , so we have:
[Equation]
where θ (cid:48)(cid:48)
=
2 θ (cid:48)
+ π
and
[Equation]
where w ⊥ k is the unit vector that is perpendicular to w k but still in the plane spanned by w k and w ∗ j .
Note I 0 is the ﬁxed integral of unit vectors weighted by angular distribution of input data on activated half-plane E ∗ j of teacher
node j .
If p ( f l − 1 ) is rotational symmetric, then (cid:15) = 0 , p ( θ (cid:48) )
= 12 π , then we can compute these terms analytically: I
= 0 , I 1 ( θ ) =
12 π ( π − θ )
and I 2 ( θ ) = 1 π sin θ w k .
8. T HEOREM
Proof.
Note that we have:
[Equation]
Therefore, we have
[Equation]
63) and
[Equation]
Note that we have:
[Equation]
Let h k
= ¯ w (cid:124) k r k .
We assume all h k > 0 (positive correlation), then we have:
[Equation]
[Equation]
If r k = r = w ∗ − (cid:80) k a k w k , then we have:
[Equation]
[Equation] where S =
( (cid:80) k a k (cid:107) w k (cid:107) ) > 0 is independent of k .
So
[Equation]
if h k − h k (cid:48) > 0 , then d d t ( h k − h k (cid:48) ) < 0 and vice versa.
This means that Eqn.
64 is zero when the system enters the stable region.
On the other hand, if (cid:107) r k (cid:107) 2 = (cid:107) r k (cid:48) (cid:107) 2 + (cid:15) (e.g., r k has stronger teacher component), then we have:
[Equation]
(69) which is only zero when h k > h k (cid:48) .
This yields exponential growth of (cid:107) w k (cid:107) compared to (cid:107) w k (cid:48) (cid:107) . [Figure] L o g l [Figure]
8. D ETAILS IN T EACHER /S TUDENT C ONSTRUCTION AND T RAINING
We construct teacher networks in the following manner.
For two-layered network, the output dimension C = 50 and input dimension d = m 0 = n 0 = 100 .
For multi-layered network, we use 50-75-100-125 (i.e, m 1 = 50 , m 2 = 75 , m 3 = 100 , m 4 = 125 , L = 5 , d
= m 0 = n 0 =and C
= m 5 = n 5 = 50 ).
The teacher network is constructed to satisfy Assumption 1: at each layer, teacher ﬁlters are distinct from each other and their bias is set so that ∼ 50% of the input data activate the nodes.
This makes their boundary (maximally) visible in the dataset.
To train the model, we use vanilla SGD with learning rate 0 .
01 and batchsize 16 .
8. A DDITIONAL F IGURES
Fig.
11 shows how the loss changes over iterations.
With high teacher polarity (Eqn. 9), it becomes harder to learn the weak teacher nodes and over-realization helps in getting low evaluation loss (in particular for p = 2 . 5 ).
Besides Gaussian distribution we also test on uniform distribution x ∼ U [ − 15 , 15] .
For training, we sample 100 k data points in each epoch.
Fig. 12 shows that the results on 4 layer ReLU network (50-75-100-125) are similar. Note that in multi-layer setting, Theorem 3 might not hold since it is for 2-layer so there could be un-specialized student nodes with large β kk ( x ) . [Figure]