Towards Understanding the Condensation of Neural Networks at Initial Training
Abstract
Empirical works show that for ReLU neural networks (NNs) with small initialization, input weights of hidden neurons (the input weight of a hidden neuron consists of the weight from its input layer to the hidden neuron and its bias term) condense onto isolated orientations.
The condensation dynamics implies that the training implicitly regularizes a NN towards one with much smaller effective size.
In this work, we illustrate the formation of the condensation in multi-layer fully connected NNs and show that the maximal number of condensed orientations in the initial training stage is twice the multiplicity of the activation function, where “multiplicity” indicates the multiple roots of activation function at origin.
Our theoretical analysis conﬁrms experiments for two cases, one is for the activation function of multiplicity one with arbitrary dimension input, which contains many common activation functions, and the other is for the layer with one-dimensional input and arbitrary multiplicity.
This work makes a step towards understanding how small initialization leads NNs to condensation at the initial training stage.
Introduction
The question why over-parameterized neural networks (NNs) often show good generalization attracts much attention (Breiman, 1995; Zhang et al., 2021).
Luo et al.
(2021) found that when initialization is small, the input weights of hidden neurons in two-layer ReLU NNs (the input weight or the feature of a hidden neuron consists of the weight from its input layer to the hidden neuron and its bias term) condense onto isolated orientations during the training.
As illustrated in the cartoon example in Fig.
1, the condensation transforms a large network, which is often over-parameterized, to the one of only a few effective neurons, leading to an output function with low complexity.
Since, in most cases, the complexity bounds the generalization error (Bartlett and Mendelson, 2002), the study of condensation could provide insight to how over-parameterized NNs are implicitly regularized to achieve good generalization performance in practice.
Small initialization leads NNs to rich non-linearity during the training (Mei et al., 2019; Rotskoff and
Vanden-Eijnden, 2018; Chizat and Bach, 2018; Sirignano and Spiliopoulos, 2020).
For example, in over-parameterized regime, small initialization can achieve low generalization error (Advani et al., 2020).
Irrespective of network width, small initialization can make two-layer ReLU NNs converge to a solution with maximum margin (Phuong and Lampert, 2020).
Small initialization also enables neural networks to learn features actively (Lyu et al., 2021;
Luo et al., 2021).
The gradual increment of the condensed orientations is consistent with many previous works, which show that the network output evolves from simple to complex during the training process (Xu et al., 2020; Rahaman et al., 2019; Arpit et al., 2017).
The initial condensation resets the network to a simple state, which brings out the simple to complex training process.
Condensation is an important phenomenon that reﬂects the feature learning process.
Therefore, it is important to understand how condensation emerges during the training with small initialization.
The dynamic behavior of the training at the initial state is important for the whole training process, because it largely determines the training dynamics of a neural network and the region it ends up in (Fort et al., 2020;
Hu et al., 2020), which impacts the characteristics of the neural networks in the ﬁnal stage of training (Luo et al., 2021; Jiang et al., 2019;
Li et al., 2018).
For two-layer ReLU
NNs, several works have studied the mechanism underlying the condensation at the initial training stage when the initialization of parameters goes to zero (Maennel et al., 2018; Pellegrini and Biroli, 2020).
However, it still remains unclear that for NNs of more general activation functions, how the condensation emerges at the initial training stage.
In this work, we show that the condensation at the initial stage is closely related to the multiplicity p at x = 0 , which means the derivative of activation at x = 0 is zero up to the ( p − 1)
th -order and is non-zero for the p -th order.
Many common activation functions, e.g., tanh( x ) , sigmoid( x ) , softplus( x ) , Gelu( x ) , Swich( x ) , etc, are all multiplicity p = 1 , and x tanh( x ) and x 2 tanh( x ) have multiplicity two and three, respectively.
Our contribution is summarized as follows:
• Our extensive experiments suggest that the maximal number of condensed orientations in the initial training is twice the multiplicity of the activation function used in general NNs.
• We present a theory for the initial condensation with small initialization for two cases, one is for the activation function of multiplicity one with arbitrary dimension input, and the other is for the layer with one-dimensional input and arbitrary multiplicity.
As many common activation functions are multiplicity p = 1 , our theory would be of interest to general readers.
Related works
Luo et al.
(2021) systematically study the effect of initialization for two-layer ReLU NN with inﬁnite width by establishing a phase diagram, which shows three distinct regimes, i.e., linear regime (similar to the lazy regime)
(Jacot et al., 2018; Arora et al., 2019; Zhang et al., 2020; E et al., 2020; Chizat and Bach, 2019), critical regime (Mei et al., 2019; Rotskoff and Vanden-Eijnden, 2018; Chizat and
Bach, 2018; Sirignano and Spiliopoulos, 2020) and condensed regime (non-linear regime), based on the relative change of input weights as the width approaches inﬁnity, which tends to 0 , O (1) and + ∞ , respectively.
Zhou et al.
(2022) make a step towards drawing a phase diagram for three-layer ReLU
NNs with inﬁnite width, revealing the possibility of completely different dynamics (linear, critical and condensed dynamics) emerging within a deep NN for its different layers.
As shown in Luo et al.
(2021), two-layer ReLU NNs with inﬁnite width do not condense in the neural tangent kernel (NTK) regime , slightly condense in the mean-ﬁeld regime, and clearly condense in the non-linear regime.
However, in Luo et al.
(2021), it is not clear how general the condensation phenomenon is when other activation functions are used and why there is condensation.
Zhang et al.
(2021a,b); Fukumizu and Amari (2000); Fukumizu et al.
; Simsek et al.
(2021) propose a general Embedding Principle of loss landscape of DNNs that unravels a hierarchical structure of the loss landscape of NNs, i.e., loss landscape of an DNN contains all critical points of all the narrower DNNs.
The embedding principle shows that a large DNN can experience critical points where the DNN condenses and its output is the same as that of a much smaller DNN.
However, the embedding principle does not explain how the training can take the DNN to such critical points.
The condensation is consistent with previous works that suggest that NNs may learn data from simple to complex patterns (Arpit et al., 2017; Xu et al., 2019;
Rahaman et al., 2019; Xu et al., 2020; Jin et al., 2020; Kalimeris et al., 2019).
For example, an implicit bias of frequency principle is widely observed that NNs often learn the target function from low to high frequency (Xu et al., 2019;
Rahaman et al., 2019; Xu et al., 2020), which has been utilized to understand various phenomena (Ma et al., 2020;
Xu and Zhou, 2021) and inspired algorithm design (Liu et al., 2020; Cai et al., 2020; Tancik et al., 2020; Li et al., 2020, 2021).
Preliminary: Neural networks and initial stage
A two-layer NN is
[Equation]
where σ ( · ) is the activation function, w j = ( ¯ w j , b j ) ∈ R d
+1 is the neuron feature including the input weight and bias terms, and x = (¯ x , 1) ∈ R
d +1 is the concatenation of the input sample and scalar 1 , θ is the set of all parameters, i.e., { a j , w j } mj =1 .
For simplicity, we call w j input weight or weight and x input sample.
A L -layer NN can be recursively deﬁned by setting the output of the previous layer as the input to the current hidden layer, i.e.,
[Equation]
where W [ l ] =
( ¯ W [ l ] , b
[ l ] ) ∈ R m l × ( m l − 1 +1) , and m l represents the dimension of the l -th hidden layer.
For simplicity, we also call each row of W [ l ] input weight or weight and x [ l − 1] input to the l -th hidden layer.
The target function is denoted as f ∗ ( x ) .
The training loss function is the mean squared error
[Equation]
Without loss of generality, we assume that the output is one-dimensional for theoretical analysis, because, for high-dimensional cases, we only need to sum up the components directly.
For summation, it does not affect the results of our theories.
We consider the gradient ﬂow training
[Equation]
To ensure that the training is close to the gradient ﬂow, all the learning rates used in this paper are sufﬁciently small.
We characterize the activation function by the following deﬁnition.
Deﬁnition 1 (multiplicity p ) .
Suppose that σ ( x ) satisﬁes the following condition, there exists a p ∈ N ∗ , such that the s -th order derivative σ ( s ) (0)
=
for s = 1 , 2 , · · · , p − 1 , and σ ( p ) (0) (cid:54) = 0 , then we say σ has multiplicity p .
Remark 3.1 .
Here are some examples.
tanh( x ) , sigmoid( x ) and softplus( x ) have multiplicity p = 1 .
x tanh( x ) has multiplicity p = 2 .
We illustrate small initialization and initial stage as follows
Small initialization : W [ l ] ∼ o (1) and W [ l ] x [ l − 1] ∼ o (1) for all l ’s and x [ l − 1] .
We want to remark that it dose not make sense to deﬁne the initial stage by the number of training steps, because the training is affected by many factors, such as the learning rate.
Therefore, in our experiments, the epochs we use to show phenomena can cover a wide range.
Alternatively, we consider the initial stage as follows.
Initial stage : the period when the leading-order Taylor expansion w.r.t activated neurons is still valid for theoretical analysis of Sec. 5.
As the parameters of a neural network evolve, the loss value will also decay accordingly, which is easy to be directly observed.
Therefore, we propose an intuitive deﬁnition of the initial stage of training by the size of loss in this article, that is the stage before the value of loss function decays to 70 % of its initial value.
Such a deﬁnition is reasonable, for generally a loss could decay to 1 % of its initial value or even lower.
The loss of the all experiments in the article can be found in Appendix
A.3, and they do meet the deﬁnition of the initial stage here.
Cosine similarity: The cosine similarity between two vectors u and v is deﬁned as
[Equation]
Initial condensation of input weights
In this section, we would empirically show how the condensation differs among NNs with activation function of different multiplicities in the order of a practical example, multidimensional synthetic data, and 1-d input synthetic data, followed by theoretical analysis in the next section.
4.1 Experimental setup
For synthetic dataset: Throughout this work, we use fully-connected neural network with size, dm - · · · - m - d out .
The input dimension d is determined by the training data.
The output dimension is d out = 1 .
The number of hidden neurons m is speciﬁed in each experiment.
All parameters are initialized by a Gaussian distribution N (0 , var) .
The total data size is n .
The training method is
Adam with full batch, learning rate lr and MSE loss.
We sample the training data uniformly from a sub-domain of R d .
The sampling range and the target function are chosen randomly to show the generality of our experiments.
For CIFAR10 dataset: We use Resnet18-like neural network, which has been described in Fig. 2, and the input dimension is d = 32 × 32 × 3 .
The output dimension is d out = 10 .
All parameters are initialized by a Gaussian distribution N (0 , var) .
The total data size is n .
The training method is
Adam with batch size 128, learning rate lr and cross-entropy loss.
4.2 A practical example
The condensation of the weights of between the fully-connected (FC) layers of a Resnet18-like neural network on CIFAR10 is shown in Fig.
2, whose activation functions for FC layers are tanh( x ) , sigmoid( x ) , softplus( x ) and x tanh( x ) , indicated by the corresponding sub-captions, respectively.
As shown in Fig.
2(a), for activation function tanh( x ) , the color indicates cosine similarity D ( u , v ) of two hidden neurons’ weights, whose indexes are indicated by the abscissa and the ordinate, respectively.
If the neurons are in the same beige block, D ( u , v ) ∼ 1 (navy-blue block,
D ( u , v ) ∼ − 1 ), their input weights have the same (opposite) direction.
Input weights of hidden neurons in Fig.
2(a) condense at two opposite directions, i.e., one line.
Similarly, weights of hidden neurons for NNs with sigmoid( x ) and softplus( x ) (Fig.
2(b, c)), which are frequently used and have multiplicity one, condense at one direction.
As the multiplicity increases, NNs with x tanh x (Fig.
2(d)) condense at two different lines.
These experiments suggest that the condensation is closely related to the multiplicity of the activation function.
In these experiments we ﬁnd that the performance of the Resnet18-like network with tanh FC layers with small initialization is similar to the one with common initialization in Appendix A.4.
[Figure]
4.3 Multidimensional synthetic data
[Figure]
If neurons are in the same beige block, D ( u , v ) ∼ 1 (navy-blue block, D ( u , v ) ∼ −
), their input weights have activation functions are indicated by the sub data where each
For convenience of experiments, we then use synthetic data to perform extensive experiments to study the relation between the condensation and the multiplicity of the activation function.
We use two-layer fully-connected NNs with size 5 - 50 - 1 to ﬁt n = 80 training data sampled from a 5-dimensional function (cid:80) 5 k =1 3 .
5 sin(5 x k + 1) , where x = ( x 1 , x 2 , · · · , x 5 ) (cid:124) ∈ R 5 and each x k is uniformly sampled from [ − 4 , 2] .
As shown in Fig. 3(a), for activation function tanh( x ) , input weights of hidden neurons condense at two opposite directions, i.e., one line.
As the multiplicity increases, NNs with x tanh( x ) (Fig. 3(b)) and x 2 tanh x (Fig. 3(c)) condense at two and three different lines, respectively.
For activation function sigmoid( x ) in Fig. 3(d) and softplus( x ) in Fig.
3(e), NNs also condense at two opposite directions.
For multi-layer NNs with different activation functions, we show that the condensation for all hidden layers is similar to the two-layer NNs.
In deep networks, residual connection is often introduced to overcome the vanishing of gradient.
To show the generality of condensation, we perform an experiment of six-layer NNs with residual connections.
To show the difference of various activation functions, we set the activation functions for hidden layer 1 to hidden layer 5 as x 2 tanh( x ) , x tanh( x ) , sigmoid( x ) , tanh( x ) and softplus( x ) , respectively.
The structure of the residual is
[Figure]
h l +1 ( x ) =
σ ( W l h l ( x ) + b l ) + h l ( x ) , where h l ( x ) is the output of the l -th layer.
As shown in Fig.
4, input weights condense at three, two, one, one and one lines for hidden layer 1 to hidden layer 5, respectively.
Note that residual connections are not necessary.
We show an experiment of the same structure as in Fig.
4 but without residual connections in Appendix A.5.
Through these experiments, we conjecture that the maximal number of condensed orientations at initial training is twice the multiplicity of the activation function used.
To understand the mechanism of the initial condensation, we turn to experiments of 1-d input and two-layer NNs, which can be clearly visualized in the next subsection.
4.4 1-d input and two-layer NN
For 1-d data, we visualize the evolution of the two-layer NN output and each weight, which conﬁrms the connection between the condensation and the multiplicity of the activation function.
[Figure]
We display the outputs at initial training in Fig.
Due to the small magnitude of parameters, an activation function with multiplicity p can be well approximated by a p -th order polynominal, thus, the NN output can also be approximated by a p -th order polynominal.
As shown in Fig.
5, the NN outputs with activation function tanh( x ) , x tanh( x ) and x 2 tanh( x ) overlap well with the auxiliary of a linear, a quadratic and a cubic polynominal curve, respectively in the beginning.
This experiment, although simple, but convincingly shows that NN does not always learn a linear function at the initial training stage and the complexity of such learning depends on the activation function.
[Figure]
All settings are the same as Fig which an also
We visualize the direction ﬁeld for input weight w j := ( w j , b j ) , following the gradient ﬂow,
[Equation]
where e i := f θ ( x i ) − f ∗ ( x i ) .
Since we only care about the direction of w j and a j is a scalar at each epoch, we can visualize ˙ w j by ˙
w j /a j .
For simplicity, we do not distinguish ˙ w j /a j and ˙ w j if there is no ambiguity.
When we compute ˙ w j for different j ’s, e i x i for ( i = 1 , · · · , n ) is independent with j .
Then, at each epoch, for a set of { e i , x i } ni =1 , we can consider the following direction ﬁeld
[Equation]
When ω is set as w j , we can obtain ˙ w j .
As shown in Fig.
6, around the original point, the ﬁeld has one, two, three stables lines, on which a neuron would keep its direction, for tanh( x ) , x tanh( x ) , and x 2 tanh( x ) , respectively.
We also display the input weight of each neuron on the ﬁeld by the green dots and their corresponding velocity directions by the orange arrows.
Similarly to the high-dimensional cases, NNs with multiplicity p activation functions condense at p different lines for p = 1 , 2 , 3 .
Therefore, It is reasonable to conjecture that the maximal number of condensed orientations is twice the multiplicity of the activation function used.
Taken together, we have empirically shown that the multiplicity of the activation function is a key factor that determines the complexity of the initial output and condensation.
To facilitate the understanding of the evolution of condensation in the initial stage, we show several steps during the initial stage of each example in Appendix A.6.
Analysis of the initial condensation of input weights
In this section, we would present a preliminary analysis to understand how the multiplicity of the activation function affects the initial condensation.
At each training step, we consider the velocity ﬁeld of weights in each hidden layer of a neural networks.
Considering a network with L hidden layers, we use row vector W [ k ] j to represent the weight from the ( k − 1) -th layer to the j -th neuron in the k -th layer.
Since condensation is always accompany with small initialization, together with the initial stage deﬁned in Sec.
3, we make the following assumptions,
Assumption 1 .
Small initialization infers that W [ k ] j x [ k − 1] ∼ o (1) applies for all k ’s and j ’s.
Assumption 2 .
During the initial stage of condensation, Taylor expansion of each corresponding activated neurons holds, i.e.,
[Equation]
Suppose the activation function has multiplicity p , i.e., σ ( s ) (0)
=
for s = 1 , 2 , · · · , p − 1 , and σ ( p ) (0) (cid:54) = 0 .
Then, together with Assumption 2, we have
[Equation]
For each k and j , W [ k ] j satisﬁes the following dynamics, (see Appendix A.2)
[Equation]
where w can represent W [ k ] j (cid:124) for all k ’s and j ’s, r = (cid:107) w (cid:107) 2 is the amplitude, and u = w /r .
For convenience, we deﬁne an operator P satisfying P w := ˙
w − u ( ˙ w · u ) .
To specify the condensation for theoretical analysis, we make the following deﬁnition,
Condensation : the weight evolves towards a direction which will not change in the direction ﬁeld and is deﬁned as follows,
[Equation]
[Equation]
Since ˙ w · u is a scalar, ˙ w is parallel with u .
u is a unit vector, therefore, we have u = ± ˙ w / (cid:107) ˙ w (cid:107) 2 .
Remark 1 (An intuitive explanation for condensation) .
In this work, we consider NNs with sufﬁciently small parameters.
Suppose r = (cid:107) w (cid:107) 2 ∼ O
(cid:15) ) , where (cid:15) is a small quantity, then dynamics (8) will show that O ( ˙ r ) ∼ O ( ˙ w ) and O ( ˙ u ) ∼ O ( ˙ r ) /O
( (cid:15) ) .
Here O ( ˙ r ) ∼ O ( ˙ w ) refers that the evolution of ˙ r is at the same order as every component of ˙ w , and it is the same for O ( ˙ u ) ∼ O ( ˙ r ) /O
( (cid:15) ) .
Therefore, the orientation u would vary much more quickly than the amplitude r .
By the dynamics for w (Eq.
10) and the Taylor approximation with multiplicity p
(Eq. 7)
, it is easy to ﬁnd that the solutions for
Eq.
9 are ﬁnite.
Then, taken together, the orientation u would converge rapidly into certain directions, leading to condensation.
In the following, we study the case of (i) p = 1 and (ii) m k − 1 = 1 (the dimension of input of the k -th layer equals one), and reach the following informal proposition,
Proposition 1.
Suppose that Assumption 1 and 2 holds.
Consider the leading-order Taylor expansion of Eq.
9, as initialization towards zero.
If either p = 1 or m k − 1 = 1 holds, then the maximal number of roots for Eq.
9 is twice of the multiplicity of the activation function.
Proof.
Case 1: p =
By gradient ﬂow, we can obtain the dynamics for W [ k ] j (see Appendix A.2),
[Equation]
where we use E l = W [ l ] (cid:124) diag { σ (cid:48)
( W [ l ] · x [ l − 1] ) } , for l ∈ { 2 , 3 , ..., L } , E [ q : p ] =
E q E q +1 ...
E p , and x [ k ] i represents the neurons of the k -th layer generated by the i -th sample.
For a ﬁxed step, we only consider the gradient of loss w.r.t.
W [ k ] j .
According to our assumption p = 1 , we have σ (cid:48) (0) (cid:54) = 0 .
Suppose that parameters are small and denote e i :=
f ( θ , x i ) − y i .
By
Taylor expansion,
[Equation]
where operator Q is the leading-order approximation of operator P , and here E [ k +1:
L ] is independent with i , because diag { σ (cid:48)
( W [ l ] x [ l − 1] ) }
≈ diag { σ (cid:48) (0) } .
Since diag { σ (cid:48) ( 0 ) } =
c I , c (cid:54) = 0 by assumption, and, WLOG, we assume a (cid:54) =
0 , then
[Equation]
We have
[Equation]
This calculation shows that for layer k , the input weights for any hidden neuron j have the same two stable directions.
Together with the analysis before, i.e., when parameters are sufﬁciently small, the orientation u would move much more quickly than the amplitude r , all input weights would move towards the same direction or the opposite direction, i.e., condensation on a line, under small initialization.
Case 2: the k -th layer with one-dimensional input, i.e., m k − 1 =
By the deﬁnition of the multiplicity p , we have
[Equation]
where ( · ) p − 1 and σ ( p ) ( · ) operate on component here.
Then up to the leading order in terms of the magnitude of θ , we have (see Appendix A.2)
P w leading order ≈
[Equation]
WLOG, we also assume a (cid:54) = 0 .
And by deﬁnition, w = r u , we have
Q w = 0 ⇔ u =
[Equation]
Since m k − 1 + 1 = 2 (for x [ k − 1] = ( σ ( W [ k − 1] x [ k − 2] ) , 1) and m k − 1 = 1 ), we denote u
= ( u 1 , u 2 ) (cid:124) ∈ R 2 and x [ k − 1] i =
(( x [ k − 1] i ) 1 ,
[Equation]
x [ k
[Equation] [Equation]
R 2 , then,
[Equation]
We obtain the equation for ˆ u ,
[Equation]
Since it is an univariate p -th order equation, ˆ u =
u 1 u 2 has at most p complex roots.
Because u is a unit vector, u at most has p pairs of values, in which each pair are opposite.
Taken together, our theoretical analysis is consistent with our experiments, that is, the maximal number of condensed orientations is twice the multiplicity of the activation function used when parameters are small.
As many commonly used activation functions are either multiplicity p = 1 or
ReLU-like, our theoretical analysis is widely applied and sheds light on practical training.
Discussion
In this work, we have shown that the characteristic of the activation function, i.e., multiplicity, is a key factor to understanding the complexity of NN output and the weight condensation at initial training.
The condensation restricts the NN to be effectively low-capacity at the initial training stage, even for ﬁnite-width NNs.
In the initial stage, the Taylor expansion upto the leading order indicates that the activation function can be approximated by a homogeneous polynomial.
Thus, even the amplitudes of weights within a layer are not identical, the network can be reduced approximately.
During the training, the NN increases its capacity to better ﬁt the data, leading to a potential explanation for their good generalization in practical problems.
This work also serves as a starting point for further studying the condensation for multiple-layer neural networks throughout the training process.
For general multiplicity with high-dimensional input data, the theoretical analysis for the initial condensation is a very difﬁcult problem, which is equivalent to counting the number of the roots of a high-order multivariate polynomial with a special structure originated from NNs.
Training data can also affect the condensation but not the maximal number of condensed orientations.
When data is simple, such as low frequency, the number of the condensed orientations can be less, some experiments of MNIST and CIFAR100 can be found in Appendix A.7.
Acknowledgments and Disclosure of Funding
This work is sponsored by the National Key R&D Program of China Grant No. 2019YFA (Z. X.), the Shanghai Sailing Program, the Natural Science Foundation of Shanghai
Grant No.
20ZR1429000 (Z. X.), the National Natural Science Foundation of China Grant No.(Z. X.), the National Natural Science Foundation of China Grant No. 12101401 (T. L.), the National Natural Science Foundation of China Grant No.
(Y. Z.), Shanghai Municipal of
Science and Technology Project Grant No. 20JC1419500 (Y.Z.), the Lingang Laboratory Grant
No.LG-QS-202202-08 (Y.Z.), Shanghai Municipal of Science and Technology Major Project No.
2021SHZDZX0102, and the HPC of School of Mathematical Sciences and the Student Innovation
Center at Shanghai Jiao Tong University.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] In Sec. 1 and Abstract.
(b) Did you describe the limitations of your work?
[N/A] (c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] In Sec. 3 and 5.
(b) Did you include complete proofs of all theoretical results?
[Yes] In Sec. 5 and Appendix A. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes]
In the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] In Sec. 4 and the captions of each ﬁgure.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Appendix A.1 Basic deﬁnitions
In this study, we ﬁrst consider the neural network with 2 hidden layers,
A two-layer NN is
[Equation]
where σ ( · ) is the activation function, w j = ( ¯ w j , b j ) ∈ R d
+1 is the neuron feature including the input weight and bias terms, and x = (¯ x , 1) ∈ R
d +1 is concatenation of the input sample and scalar 1 , θ is the set of all parameters, i.e., { a j , w j } mj =1 .
For simplicity, we call w j input weight or weight and x input sample.
Then, we consider the neural network with l hidden layers, x
[Equation]
l ∈
[Equation]
where W [ l ] =
( ¯ W [ l ] , b [ l ] ) ∈ R ( m l × m l − 1 ) , and m l represents the dimension of the l -th hidden layer.
The initialization of W [ l ] k,k (cid:48) , l ∈ { 1 , 2 , 3 , ..., L } and a k obey normal distribution W [ l ] k,k (cid:48) ∼ N (0 , β 2 l ) for l ∈ { 1 , 2 , 3 , ..., L } and a k ∼ N (0 , β 2 L +1 ) .
The loss function is mean squared error given below,
[Equation]
For simpliﬁcation, we denote f θ ( x ) as f in following.
A.2 Derivations
for concerned quantities A.2.1 Neural networks with three hidden layers
In order to better understand the gradient of the parameter matrix of the multi-layer neural network, we ﬁrst consider the case of the three-layer neural network,
[Equation]
[Equation]
with the mean squared error as the loss function,
[Equation]
We calculate d f d W [2] and d f d W [1] respectively, using differential form,
[Equation]
We consider
[Equation]
where (cid:12) is Hadamard product
[Equation]
and it is the multiplication of matrix elements of the same position.
Hence,
[Equation]
Then, we consider
[Equation]
Hence, we have,
[Equation] [Equation]
Through the chain rule
[Equation]
we can get the evolution equation of W [1] and W [2] ,
[Equation]
[Equation]
and
[Equation]
A.2.2 L hidden layers condition
And
[Equation]
we consider the neural network with L hidden layers,
[Equation]
where Λ l := ( W [ l +1] (cid:124) Λ l +1 ) (cid:12) σ (cid:48)
( W [ l ] x [ l − 1] ) for l = k, k + 1 . . .
L − 1 and Λ L := a (cid:12) σ (cid:48)
( W [ L ] x [ L − 1] ) .
Hence, we get,
[Equation]
Through the chain rule, we can get the evolution equation of W [ k ] ,
[Equation]
Through a (cid:12) σ (cid:48)
( W x ) = diag { σ (cid:48)
( W x ) } a ,
Finally, the dynamic system can be obtained:
[Equation]
i ∈ [1 : L − 1] , (26) where we use E l ( x ) =
W [ l ] (cid:124) diag { σ (cid:48)
( W [ l ] x [ l − 1] ) } .
And E [ q : p ] =
E q E q +1 ...
E p .
Let r k,j = (cid:107) W [ k ] j (cid:107) 2 .
We have
[Equation]
Then we obtain
[Equation]
Finally, we get
[Equation]
(29) where u k,j = W [ k ] j r k,j is a unit vector.
Then we have,
[Equation]
To conclude, the quantities we concern are summarized as follows,

[Equation]
where we use E l ( x )
=
W [ l ] (cid:124) diag { σ (cid:48)
[Equation]
W [ l ] x [ l
[Equation]
And E [ q : p ] =
E q E q +1 ...
E p .
A.2.3 Prove for P w in
We calculate P w leading order ≈
Q w as following,
[Equation]
)] j · u ) u = − 1 n n (cid:88) i =
e i x
[ k − 1] i [diag { σ ( p ) ( 0 ) ( p − 1)!
(cid:12) ( W [ k ] x
[ k − 1] i ) p − 1 } ( E [ k +1:
L ] a )]
j + ( 1 n n (cid:88) i =
e i x
[ k − 1] i [diag { σ ( p ) ( 0 ) ( p − 1)!
(cid:12) ( W [ k ] x
[ k − 1] i ) p − 1 } ( E [ k +1:
L ] a )] j · u ) u = − 1 n n (cid:88) i
=
e i x
[ k
[Equation]
1] i [
[Equation]
[Equation]
[ k ] x [ k − 1] i ) p − 1 } diag { σ ( p ) ( 0 ) ( p
[Equation]
( E [ k +1:
L ] a )]
j + ( 1 n n (cid:88) i =
e i x
[ k
[Equation]
1] i [diag { ( W [ k ] x
[ k − 1] i ) p − 1 } diag { σ ( p ) ( 0 ) ( p
[Equation]
( E [ k
[Equation]
1: L ] a )] j · u ) u = − 1 n n (cid:88) i
= e i x
[ k − 1] i [diag { ( W [ k ] x
[ k − 1] i ) p − 1 } ] j diag {
[Equation]
( p ) ( 0 ) ( p − 1)! }
( E [ k
[Equation]
1: L ] a ) + ( 1 n n (cid:88) i =
e i x
[ k − 1] i [
[Equation]
[Equation]
[ k ] x [ k − 1] i ) p − 1 } ] j
[Equation]
σ ( p ) ( 0 ) ( p − 1)! }
( E [ k +1:
L ] a ) · u ) u =
[Equation]
( 1 n n (cid:88)
[Equation]
where ( · ) p − 1 and σ p ( · ) operate on component here.
A.3 The Veriﬁcation of the initial stage
We put the loss of the experiments in the main text here to show that they are indeed in the initial stage of training by the deﬁnition.
As is shown in Fig.7 and Fig.8, at the steps demonstrated in the article, loss satisﬁes the deﬁnition of the initial stage, so we consider that they are in the initial stage of training.
Learning rate is not a sensitive to the appearance of condensation.
However, a small learning rate could enable us to observe the condensation process in the initial stage under a gradient ﬂow training, more clearly.
For example, when the learning rate is relatively small, the initial stage of training may be relatively long, while when the learning rate is relatively large, the initial stage of training may be relatively small.
We empirically ﬁnd that to ensure the training process follows a gradient follow, where the loss decays monotonically, we have to select a smaller learning rate for large multiplicity p .
Therefore, it looks like we have a longer training in our experiments with large p .
Note that for a small learning rate in the experiments of small p , we can observe similar phenomena.
In all subsequent experiments in the Appendix, we will no longer show the loss graph of each experiment one by one, but we make sure that they are indeed in the initial stage of training.
[Figure] [Figure]
A.4 Performance of tanh activation function in
condensed
regime
For practical networks, such as resnet18-like (He et al., 2016) in learning CIFAR10, as shown in
Fig.
9 and Table 1, we ﬁnd that the performance of networks with initialization in the condensed regime is vary similar to the common initialization methods.
For both initialization methods, the test accuracy is about 86.5 % to 88.5 % , where the highest test accuracy and lowest test accuracy for common methods are 88.07 % and 86.73 % , respectively, while the highest one and lowest one for condensed methods are 88.26 % and 86.88 % , respectively.
This implies that performance of common and condensed initialization is similar.
[Figure]
[Table]
condensed
[Table]
A.5 multi-layer experimental
The condensation of the six layer without residual connections is shown in 10, whose activation functions for hidden layer 1 to hidden layer 5 are x 2 tanh( x ) , x tanh( x ) , sigmoid( x ) , tanh( x ) and softplus( x ) , respectively.
The condensation of the three layer without residual connections is shown in 11, whose activation functions are same for each layer indicated by the corresponding sub-captions.
The condensation of the ﬁve layer without residual connections is shown in 12, whose activation functions are same for each layer indicated by the corresponding sub-captions.
The condensation of the ﬁve layer with residual connections is shown in 13, whose activation functions are same for each layer indicated by the corresponding sub-captions.
[Figure] [Figure]
The training data is 80 points sampled from a 5-dimensional function (cid:80) 5 k =1 3 sin(8 x k + 1) , where each x k is uniformly sampled from [ − 4 , 2] .
n = 80 , d = 5 , m = 50 , d out = 1 , var = 0 .
005 lr = 10 − 4 , 2 × 10 − {+5 , 1 . 
+}
× 10 − 5 for a-d respectively we discard hidden neurons whose
[Figure] [Figure]
A.
6 Several
steps during the evolution of condensation at the initial stage
In the article, we only give the results of the last step of each condense, while the details of the evolution of condensation are lacking, which may provide a better understanding.
Therefore, we show these details in Fig.
14, Fig. 15, Fig. 16 and Fig.
17, which also further illustrate the rationality of the experimental results and facilitate the understanding of the evolution of condensation in the initial stage.
[Figure] [Figure]
[Figure] [Figure]
A.7 The inﬂuence of training data on condensation
We also ﬁnd that when the training data is less oscillated, the NN may condense at fewer directions.
For example, as shown in Fig. 18(a), compared with the high frequency function in Fig.
3, we only change the target function to be a lower-frequency function, i.e., (cid:80) 5 k =1 3 .
5 sin(2 x k + 1) .
In this case, the NN with x 2 tanh( x ) only condenses at three directions, in which two are opposite.
For MNIST data in Fig.
18(b), we ﬁnd that, the NN with x 2 tanh( x ) condenses at one line, which may suggest that the function for ﬁtting MNIST dataset is a low-frequency function.
For CIFAR100 data in Fig.
18(c), we ﬁnd that input weights of the ﬁrst FC layer with x tanh( x ) condense at only one line, which implies that features extracted by the convolution part of the NN may own low complexity.
These experiments does not contradict to our results, which claim that the maximal number of condensed orientations in the initial training is twice the multiplicity of the activation function used in general NNs.
For CIFAR100 dataset, we use Resnet18-like neural network, which has been described in Fig.
Besides, the input dimension is d = 32 ∗ 32 ∗ 3 , the output dimension is d out = 100 , and all parameters are initialized by a Gaussian distribution N (0 , var) .
The total data size is n .
The training method is Adam with batch size 128, learning rate lr and cross-entropy loss.
For MNIST dataset, we use fully-connected neural network with size, d - m - · · · - m - d out .
The input dimension is d = 784 , and the output dimension is d out = 10 .
The number of hidden neurons m is speciﬁed in Fig.
All parameters are initialized by a Gaussian distribution N (0 , var) .
The total data size is n .
The training method is Adam with full batch, learning rate lr and MSE loss.
[Figure]
x
For discard about 15 % of hidden neurons, in which the L 2 -norm of each input weight is smaller than 0 .
remaining those bigger than 0 .
magnitude parameter[-is (0.4 2 /785) 
-]
∼ 0. should also be quite small
Fig