T OWARDS U NDERSTANDING THE C ONDENSATION OF N EURAL N ETWORKS AT I NITIAL T RAINING
Anonymous authors
Paper under double-blind review
A BSTRACT
Implicit regularization is important for understanding the learning of neural networks (NNs).
Empirical works show that input weights of hidden neurons (the input weight of a hidden neuron consists of the weight from its input layer to the hidden neuron and its bias term) condense on isolated orientations with a small initialization
The condensation dynamics implies that the training implicitly regularizes a NN towards one with much smaller effective size
In this work , we utilize multilayer networks to show that the maximal number of condensed orientations in the initial training stage is twice the multiplicity of the activation function where “multiplicity” is multiple roots of activation function at origin
Our theoretical analysis conﬁrms experiments for two cases one is for the activation function of multiplicity one with arbitrary dimension input, which contains many common activation functions and the other is for the layer with one-dimensional input and arbitrary multiplicity.
This work makes a step towards understanding how small initialization implicitly leads NNs to condensation at initial training stage, which lays a foundation for the future study of the nonlinear dynamics of NNs and its implicit regularization effect at a later stage of training
I NTRODUCTION
Over-parameterized neural networks often show good generalization performance on real-world problems by minimizing loss functions without explicit regularization (Breiman, 1995; Zhang et al., 2017).
For over-parameterized NNs, there are inﬁnite possible sets of training parameters that can reach a satisfying training loss.
However, their generalization performances can be very different.
It is important to study what implicit regularization is imposed aside to the loss function during the training that leads the NN to a speciﬁc type of solutions.
Empirical works suggest that NNs may learn the data from simple to complex patterns (Arpit et al., 2017; Xu et al., 2019; Rahaman et al., 2019; Xu et al., 2020; Jin et al., 2020; Kalimeris et al., 2019).
For example, an implicit bias of frequency principle is widely observed that NNs often learn the target function from low to high frequency (Xu et al., 2019; Rahaman et al., 2019; Xu et al., 2020), which has been utilized to understand various phenomena (Ma et al., 2020; Xu & Zhou, 2021) and inspiring algorithm design Liu et al.
The NN output, either simple or complex, is a collective result of all neurons.
The study of how neuron weights evolve during the training is central to understanding the collective behavior, including the complexity, of the NN output.
Luo et al.
(2021) establish a phase diagram to study the effect of initialization on weight evolution for two-layer ReLU NNs at the inﬁnite-width limit and ﬁnd three distinct regimes in the phase diagram, i.e., linear regime, critical regime and condensed regime.
The non-linear regime, a largely unexplored non-linear regime, is named as condensed regime because the input weights of hidden neurons (the input weight or the feature of a hidden neuron consists of the weight from its input layer to the hidden neuron and its bias term) condense on isolated orientations during the training (Luo et al., 2021).
The three regimes are identiﬁed based on the relative change of input weights as the width approaches inﬁnity, which tends to 0 , O (1) and + ∞ , respectively.
The condensation is a feature learning process, which is important to the learning of DNNs.
Note that in the following, condensation is accompanied by a default assumption of small initialization or large relative change of input weights during training .
For practical networks, such as resnet18-like (He et al., 2016) in learning CIFAR10, as shown in Fig. 1(a) and Table 1, we ﬁnd that [Table] the performance of networks with initialization in the condensed regime is very similar to the common initialization methods.
However, the condensation phenomenon provides an intuitive explanation of the good performance as follows, which may lead to a quantitative theoretical explanation in future work.
The condensation transforms a large network to a network of only a few effective neurons, leading to an output function with low complexity.
Since the complexity bounds the generalization error (Bartlett & Mendelson, 2002), the study of condensation could provide insight to how NNs are implicitly regularized to achieve good generalization performance in practice.
For two-layer ReLU NN, Maennel et al.
(2018) prove that, as the initialization of parameters goes to zero, the features of hidden neurons condense at ﬁnite number of orientations depending on the input data; when performing a linearly separable classiﬁcation task with inﬁnite data, Pellegrini &
Biroli (2020) show that at mean-ﬁeld limit, a two-layer inﬁnite-width ReLU NN is effectively equal to a NN of one hidden neuron, i.e., condensation on a single orientation.
Both works (Maennel et al., 2018; Pellegrini & Biroli, 2020) study the condensation behavior for ReLU-NNs at an initial training stage in which the magnitudes of NN parameters are far smaller from well-ﬁtting an O (1) target function.
However, it still remains unclear that for NNs of more general activation functions, how the condensation emerges at the initial training stage. [Figure]
In this work we show that the condensation at the initial stage is closely related to the multiplicity p at x = 0 , which means the derivative of activation at x = 0 is zero up to the ( p − 1)
th -order and is non-zero for the p -th order.
To verify their relation, we use the common activation function sigmoid( x ) , softplus( x ) , tanh( x ) , which are multiplicity one, and variants of tanh( x ) ,i.e. x tanh( x ) and x 2 tanh( x ) with multiplicity two and three, for our experiments.
For comparison, we also show the initial condensation of ReLU( x ) , which is studied previously (Maennel et al., 2018) and has totally different properties at origin compared with tanh( x ) .
Our experiments suggest that the maximal number of condensed orientations is twice the multiplicity of the activation function used in general NNs.
For ﬁnite-width two-layer NNs with small initialization at the initial training stage, each hidden neuron’s output in a ﬁnite domain around 0 can be approximated by a p -th order polynomial and so is the NN output function.
Based on the p -th order approximation, we show a preliminary theoretical support for condensation by a theoretical analysis for two cases, one is for the activation function of multiplicity one with arbitrary dimension input, which contains many common activation functions, and the other is for the layer with one-dimensional input and arbitrary multiplicity.
Therefore, small initialization imposes an implicit regularization that restricts the NN to be effectively much narrower neural network at the initial training stage.
As commonly used activation functions, such as tanh( x ) , sigmoid( x ) , softplus( x ) , etc., are all multiplicity one, our study of initial training behavior lays an important basis for the further study of implicit regularization throughout the training.
R ELATED WORKS
A research line studies how initialization affects the weight evolution of NNs with a sufﬁciently large or inﬁnite width.
For example, with an initialization in the neural tangent kernel (NTK) regime or lazy training regime (weights change slightly during the training), the gradient ﬂow of inﬁnite-width
NN, can be approximated by a linear dynamics of random feature model (Jacot et al., 2018;
Arora et al., 2019; Zhang et al., 2020; E et al., 2020; Chizat & Bach, 2019), whereas for the initialization in the mean-ﬁeld regime (weights change signiﬁcantly during the training), the gradient ﬂow of inﬁnite-width NN exhibits highly nonlinear dynamics (Mei et al., 2019; Rotskoff & Vanden-Eijnden, 2018; Chizat & Bach, 2018; Sirignano & Spiliopoulos, 2020).
Pellegrini & Biroli (2020) analyze how the dynamics of each parameter transforms from a lazy regime (NTK initialization) to a rich regime (mean-ﬁeld initialization) for an two-layer inﬁnite-width ReLU NN to perform a linearly separable classiﬁcation task with inﬁnite data.
Luo et al.
(2021) systematically study the effect of initialization for two-layer ReLU NN with inﬁnite width by establishing a phase diagram, which shows three distinct regimes, i.e., linear regime (similar to the lazy regime), critical regime and condensed regime (similar to the rich regime), based on the relative change of input weights as the width approaches inﬁnity, which tends to 0 , O (1) and + ∞ , respectively.
Luo et al.
(2021) also empirically ﬁnd that, in the condensed regime, the features of hidden neurons (orientation of the input weight) condense in several isolated orientations, which is a strong feature learning behavior, an important characteristic of deep learning, however, in Luo et al. (2021), it is not clear how general of the condensation when other activation functions are used and why there is condensation.
P RELIMINARY : N EURAL NETWORKS AND INITIAL STAGE
A two-layer NN is
[Equation]
where σ ( · ) is the activation function, w j = ( ¯ w j , b j ) ∈ R d
+1 is the neuron feature including the input weight and bias terms, and x = (¯ x , 1) ∈ R
d +1 is combination of the input sample and scalar 1 , θ is the set of all parameters, i.e., { a j , w j } mj =1 .
For simplicity, we call w j as input weight or weight and x as input sample.
A L -layer NN can be recursively deﬁned by feeding the output of the previous layer as the input to the current hidden layer i.e.,
[Equation]
1) , for l ∈ {
[Equation] [Equation] [Equation] where W [ l ] =
( ¯ W [ l ] , b
[ l ] ) ∈ R m l × ( m l − 1 +1) , and m l represents the dimension of the l -th hidden layer.
For simplicity, we also call each row of W [ l ] as input weight or weight and x [ l − 1] as input neurons.
The target function is denoted as f ∗ ( x ) .
The training loss function is mean squared error
[Equation]
Without loss of generality, we assume that the output is one-dimensional for theoretical analysis, because for high-dimensional cases, we only need to sum the components directly.
For summation, it does not affect the results of our theories.
We consider the gradient ﬂow training
[Equation]
For convenience, we characterize the activation function by the following deﬁnition.
Deﬁnition 1 (multiplicity p ) .
Suppose that σ ( x ) satisﬁes the following condition, there exists a p ∈ N and p ≥ 1 , such that the k -th order derivative σ ( k ) (0)
=
for k = 1 , 2 , · · · , p − 1 , and σ ( p ) (0) (cid:54)
= 0 , then we say σ has multiplicity p .
In the experiments, we study the condensation at the initial stage of training.
For a ﬁxed loss, the step we need to achieve it is highly related to the size of learning rate.
Therefore, we propose a deﬁnition of the initial stage of training by the size of loss in this article, that is the stage before the value of loss function decays to 70 % of its initial value.
Such a deﬁnition is reasonable, for generally a loss could decay to 1 % of its initial value or even lower.
The loss of the all experiments in the article can be found in Appendix A.3, and they do meet the deﬁnition of the initial stage here.
I NITIAL CONDENSATION OF INPUT WEIGHTS
It is intuitively believed that NNs are powerful at learning data features, which should be an important reason behind the success of deep learning.
A simple way to deﬁne a learned feature of a neuron is by the orientation of its input weights.
Previous work in Luo et al.
(2021) show that there is a condensed regime, where the neuron features condense on isolated orientations during the training for two-layer ReLU NNs.
The condensation implies that although there are many more neurons than samples, the number of effective neurons, i.e., the number of different used features in ﬁtting, is often much smaller than the number of samples.
Therefore, the condensation provides a potential mechanism that helps over-parameterized NNs avoid overﬁtting.
However, it is still unclear how the condensation, for general NNs with small initialization, emerges during the training.
In this section, we would empirically show how the condensation differs among NNs with activation functions of different multiplicities, followed by theoretical analysis in the next section.
4. E XPERIMENTAL SETUP
For Synthetic dataset and MNIST:
Throughout this work, we use fully-connected neural network with size, d - m - · · · - m - d out .
The input dimension d is determined by the training data.
The output dimension is d out = 1 for synthetic data and d out = 10 for MNIST.
The number of hidden neurons m is speciﬁed in each experiment.
All parameters are initialized by a Gaussian distribution N (0 , var ) .
The total data size is n .
The training method is Adam with full batch, learning rate lr and MSE loss.
For synthetic data, we sample the training data uniformly from a sub-domain of R d .
For CIFAR10 and CIFAR100 dataset: We use Resnet18-like neural network, which has been describe in Fig. 1 thoroughly. The input dimension d is determined by the training data. The output dimension is d out = 10 for CIFAR10 and d out = 100 for CIFAR100.
All parameters are initialized by a Gaussian distribution N (0 , var ) . The total data size is n . The training method is Adam with batch size 128, learning rate lr and Cross-entropy loss. 4. M ULTIDIMENSIONAL DATA
We ﬁrst show the condensation at initial training stage in ﬁtting multidimensional dataset.
Since the input is a multidimensional vector, the direction is also multidimensional.
To characterize the condensation, we use D ( u , v ) to denote the inner product of the normalized vectors of two input weights, i e., D ( u , v ) =
u (cid:124) v .
[Figure]
If neurons are in the same beige block D ( u, v ) ∼ 1 (navy-blue block D ( u, v ) ∼ − 1 ) their input weights have the same (opposite) direction.
The activation functions are indicated by the sub-captions.
training data 80 points sampled from (cid:80) 5 k =1 3 .
5 sin(5 x k + , where each − n =
d = m
=
We use a two-layer fully-connected NN with size 5 - 50 - 1 to ﬁt n = 80 training data sampled from a 5-dimensional function (cid:80) 5 k =1 3 .
5 sin(5 x k + 1) , where x = ( x 1 , x 2 , · · · , x 5 ) (cid:124) ∈ R 5 and each x k is uniformly sampled from [ − 4 , 2] .
As shown in Fig.
2(a), for activation function tanh( x ) , the color indicates D ( u, v ) of two hidden neurons’ weights at epoch 100 , whose indexes are indicated by the abscissa and the ordinate.
If the neurons are in the same beige block, D ( u, v ) ∼ 1 (navy-blue block, D ( u, v ) ∼ − 1 ), their input weights have the same (opposite) direction.
Obviously, input weights of hidden neurons condense at two opposite directions, i.e., one line.
As the multiplicity increasing, NNs with x tanh( x ) (Fig. 2(b)) and x 2 tanh x (Fig. 2(c)) condense at two and three different lines, respectively.
For activation function sigmoid( x ) in Fig. 2(d) and softplus( x ) in Fig. 2(e), which are frequently used and have multiplicity one, NNs also condense at two opposite directions.
For ReLU in Fig. 2(f), for which the multiplicity deﬁnition cannot apply, the NN condenses at three directions, in which two are opposite.
Through these experiments, we conjecture that the maximal number of condensed orientations is twice the multiplicity of the activation function used at initial training. [Figure]
For multilayer NNs with different activation functions, we show that the condensation for all hidden layers is similar to the two-layer NNs.
In deep networks, residual connection is often introduced to overcome the vanishing of gradient.
To show the generality of condensation, we perform an experiment of six-layer NNs with residual connections.
To show the difference of various activation functions, we set the activation functions for hidden layer 1 to hidden layer 5 as x 2 tanh( x ) , x tanh( x ) , sigmoid( x ) , tanh( x ) and softplus( x ) , respectively.
The structure of the residual is h l +1 ( x ) =
σ ( W l h l ( x ) + b l ) + h l ( x ) , where h l ( x ) is the output of the l -th layer.
As shown in
Fig. 3, input weights condense at three, two, one, one and one lines for hidden layer 1 to hidden layer 5, respectively.
Note that residual connections are not necessary.
We show an experiment of the same structure as in Fig.
3 but without residual connections in Appendix A.6.
To show the universality of condensation, we train resnet18-like neural networks to learn CIFAR10.
We study the condensation of the ﬁrst fully connected layer of the network, using ReLU and x tanh ( x ) as the activation functions and initialization distribution N (0 , ( 1 m 1 . 5 ) 2 ) .
As shown in Fig.
1 (b) and (c), the condensations for activation ReLU( x ) and x tanh ( x ) are consistent with Fig.
and our conjucture.
More experiments on dataset CIFAR10 and CIFAR100 can be found in Appendix A.5.
We also ﬁnd that when the training data is less oscillated, the NN may condense at fewer directions.
For example, as shown in Fig. 4(a), compared with the high frequency function in Fig.
2, we only change the target function to be a lower-frequency function, i.e., (cid:80) 5 k =1 3 .
5 sin(2 x k + 1) .
In this case, the NN with x 2 tanh( x ) only condenses at three directions, in which two are opposite.
For
MNIST data in Fig. 4(b), we ﬁnd that, the NN with x 2 tanh( x ) condenses at one line, which may suggest that the function for ﬁtting MNIST dataset is a low-frequency function.
For CIFAR100 data in Fig. 4(c), we ﬁnd that input weights of the ﬁrst FC layer with x tanh( x ) condense at only one line, which implies that features extracted by the convolution part of the NN may own low complexity.
[Figure]
activation function: x 2 tanh( x ) . For (a), we discard about 15 % of hidden neurons, in which the L 2 -norm of each input weight is smaller than 0 .
04 , while remaining those bigger than 0 .
{+The mean magnitude here for each parameter is (0.4 2 /785) 
+}
.∼ 0.01, which should also be quite small.
All settings in (a) are the same as Fig.
2, except for the lower frequency target function.
Parameters for (b) are n = 60000 , d = 784 , m = 30 , d out = 10 , var = 0 .
lr = 5 × 10 − 5 .
{+The structure and parameters of the Resnet18-like neural network for (c) is the same as Fig. 1(b), except for the data set CIFAR
+}
and learning rate lr = 1 × 10 − 6 .
To understand the mechanism of the initial condensation, we turn to experiments of 1-d input and two-layer NN, which can be clearly visualized in the next subsection.
4. 1- D INPUT AND TWO - LAYER NN
For 1-d data, we visualize the evolution of the two-layer NN output and each weight, which conﬁrms the connection between the condensation and the multiplicity of the activation function.
[Figure] We display the outputs at initial training, epoch 1000 , in Fig.
Due to the small magnitude of parameters, an activation function with multiplicity p can be well approximated by a p -th order polynominal, thus, the NN output can also be approximated by a p -th order polynominal.
As shown in Fig.
5(a-c), the NN outputs with activation functions tanh( x ) , x tanh( x ) and x 2 tanh( x ) overlap well with the auxiliary of a linear, a quadratic and a cubic polynominal curve, respectively.
In Fig.
5(d)
, the NN output with ReLU activation function deviates from a linear function (red auxiliary line).
Particularly, the NN output has several sharp turning points.
This experiment, although simple, but convincingly shows that NN does not always learn a linear function at the initial training stage and the complexity of such learning depends on the activation function.
[Figure]
All are the Fig.
the
We visualize the direction ﬁeld for input weight w j := ( w j , b j ) , following the gradient ﬂow,
[Equation]
where e i := f θ ( x i ) − f ∗ ( x i ) .
Since we only care about the direction of w j and a j is a scalar at each epoch, we can visualize ˙ w j by ˙
w j /a j .
For simplicity, we do not distinguish ˙ w j /a j and ˙ w j if there is no ambiguity.
When we compute ˙ w j for different j ’s, e i x i for ( i = 1 , · · · , n ) is independent with j .
Then, at each epoch, for a set of { e i , x i } ni =1 , we can consider the following direction ﬁeld
[Equation]
When ω is set as w j , we can obtain ˙ w j .
As shown in Fig.
6, around the original point, the ﬁeld has one, two, three stables lines, on which a neuron would keep its direction, for tanh( x ) , x tanh( x ) , and x 2 tanh( x ) , respectively.
We also display the input weight of each neuron on the ﬁeld by the green dots and their corresponding velocity directions by the orange arrows.
Similarly to the high-dimensional cases, NNs with multiplicity p activation functions condense at p different lines for p = 1 , 2 , 3 .
Therefore, It is reasonable to conjecture that the maximal number of condensed orientations is twice the multiplicity of the activation function used.
As shown in Fig.
6(d), the ﬁeld and the condensation for the NN with ReLU( x ) is much more complex.
Taken together, we have empirically shown that the multiplicity of the activation function is a key factor that determines the complexity of the initial output and condensation.
In ﬁgures of this section, we only show the results of the ﬁnal step of condensation.
To facilitate the understanding of evolution of condensation in the initial stage, we show several steps during the initial stage of each example in Appendix A.4.
From the evolution, we could directly observe how condensation occurs.
A NALYSIS OF THE INITIAL CONDENSATION OF INPUT WEIGHTS
In this section, we would present a preliminary analysis to understand how the multiplicity of the activation function affects the initial condensation.
At each training step, we consider the velocity ﬁeld of weights in each hidden layer of a neural networks.
Considering a network with L hidden layers, we use row vector W [ k ] j to represent the weight from the ( k - 1) -th layer to the j -th neuron in the k -th layer.
For each k and j , W [ k ] j satisﬁes the following dynamics, (see Appendix A.2)
[Equation]
where w can be W [ k ] j (cid:124) for all k ’s and j ’s, r = (cid:107) w (cid:107) 2 is the amplitude, and u = w /r .
Suppose the activation function has multiplicity p , i.e., σ ( k ) (0)
=
for k = 1 , 2 , · · · , p − 1 , and σ ( p ) (0) (cid:54) = 0 .
For convenience, we deﬁne an operator P satisfying P w := ˙
w − u ( ˙ w · u ) .
Condensation refers to that the weight evolves towards a direction that will not change in the direction ﬁeld and is deﬁned as follows,
[Equation]
Since ˙ w · u is a scalar, ˙ w is parallel with u .
u is a unit vector, therefore, we have u = ˙ w / (cid:107) ˙ w (cid:107) 2 .
In this work, we consider NNs with sufﬁciently small parameters.
For example, suppose r = (cid:107) w (cid:107) 2 ∼
O ( (cid:15) ) , (cid:15) is a small quantity.
Dynamics (5) shows that O ( ˙ r ) ∼ O ( ˙ w ) and O ( ˙ u ) ∼ O ( ˙ r ) /O
( (cid:15) ) .
Therefore, the orientation u would moves much more quickly than the amplitude r .
In the following, we study the case of (i) p = 1 and (ii) m l = 1 .
5. C ASE 1: p =
Since we have (see Appendix A.2),
[Equation]
where we use E l = W [ l ] (cid:124) diag { σ (cid:48)
( W [ l ] x [ l − 1] ) } , for l ∈ { 2 , 3 , ..., L } , E [ q : p ] =
E q E q +1 ...
E p .
For a ﬁxed step, we only consider the gradient w.r.t.
˙ W [ k ] j .
Suppose σ (cid:48) (0) (cid:54) = 0 and parameters are small.
Denote e i :=
( f ( θ , x i ) − y i ) .
By Taylor expansion,
[Equation]
where operator Q is the leading-order approximation of operator P , and here E [ k +1:
L ] is independent with i because diag { σ (cid:48)
( W [ l ] x [ l − 1] ) }
≈ diag { σ (cid:48) (0) } .
Since diag { σ (cid:48) ( 0 ) } = I , and, WLOG, we assume a (cid:54) = 0 , then
[Equation]
We have
[Equation]
This calculation shows that for layer k , the input weights for any hidden neuron j have the same two stable directions.
Therefore, when parameters are sufﬁciently small, which implies that the orientation u would moves much more quickly than the amplitude r , all input weights towards converging to the same direction or the opposite direction, i.e., condensation on a line.
5. C ASE 2: m l =
By the deﬁnition of the multiplicity p , we have
[Equation]
where ( · ) p − 1 and σ p ( · ) operate on component here.
Then up to the leading order in terms of the magnitude of θ , we have (see Appendix A.2)
[Equation]
WLOG, we also assume a (cid:54) = 0 .
And by deﬁnition, w = r u , we have
[Equation]
Since d + 1 = 2 , we denote u
= ( u 1 , u 2 ) (cid:124) ∈ R 2 and x [ k − 1] i = (
[Equation] x [ k − 1] i ) 1 , ( x [ k − 1] i [Equation]
R 2 , then,
[Equation]
We obtain the equation for ˆ u ,
[Equation]
Since it is an univariate p -th order equation, ˆ u =
u 1 u 2 has at most p complex roots.
Because u is a unit vector, u at most has p pairs of values, in which each pair are opposite.
Taken together, our theoretical analysis is consistent with our experiments, that is, the maximal number of condensed orientations is twice the multiplicity of the activation function used when parameters are small.
Besides, because commonly used activation functions, such as tanh( x ) , sigmoid( x ) , softplus( x ) , etc., are all multiplicity one, the theoretical analysis sheds light on practical training.
D ISCUSSION
In this work, we have shown that the characteristic of the activation function, i.e., multiplicity, is a key factor to understanding the complexity of NN output and the weight condensation at initial training.
The condensation restricts the NN to be effectively low-capacity at the initial training stage, even for ﬁnite-width NNs.
During the training, the NN increases its capacity to better ﬁt the data, leading to a potential explanation for their good generalization in practical problems.
This work also serves as a starting point for further studying the condensation for multiple-layer neural networks throughout the training process.
As the scale of parameter initialization becomes larger, the condensation becomes weaker.
The understanding from a complete condensation would beneﬁt our understanding of the training process in common initialization, that is there is an effect, although not so strong, that can limit the complexity of the neural network in the initial training.
How small the initialization should be in order to see a clear condensation is studied in Luo et al.
(2021) for two-layer ReLU NNs with inﬁnite width.
For general activation functions, the regime of the initialization for condensation depends on the NN width.
A further study of the phase diagram for ﬁnite width NNs would be important.
For general multiplicity with high-dimensional input data, the theoretical analysis for the initial condensation is a very difﬁcult problem, which is equivalent to count the number of the roots of a high-order high-dimensional polynomial with a special structure originated from NNs.
A A PPENDIX A. B ASIC DEFINITIONS
In this study, we ﬁrst consider the neural network with 2 hidden layers, A two-layer NN is
[Equation]
where σ ( · ) is the activation function, w j = ( ¯ w j , b j ) ∈ R d
+1 is the neuron feature including the input weight and bias terms, and x = (¯ x , 1) ∈ R d +1 is combination of the input sample and scalar 1 , θ is the set of all parameters, i.e., { a j , w j } mj =1 . For simplicity, we call w j as input weight or weight and x as input sample. Then, we consider the neural network with l hidden layers,
[Equation]
where W [ l ] =
( ¯ W [ l ] , b [ l ] ) ∈ R ( m l × m l − 1 ) , and m l represents the dimension of the l -th hidden layer.
The initialization of W [ l ] k,k (cid:48) , l ∈ { 1 , 2 , 3 , ..., L } and a k obey normal distribution W [ l ] k,k (cid:48) ∼ N (0 , β 2 l ) for l ∈ { 1 , 2 , 3 , ..., L } and a k ∼ N (0 , β 2 L +1 ) .
The loss function is mean squared error given below,
[Equation]
For simpliﬁcation, we denote f θ ( x ) as f in following.
A. D ERIVATIONS FOR CONCERNED QUANTITIES A.2. N EURAL NETWORKS WITH THREE HIDDEN LAYERS
In order to better understand the gradient of the parameter matrix of the multi-layer neural network, we ﬁrst consider the case of the three-layer neural network,
[Equation]
with the mean squared error as the loss function,
[Equation]
We calculate d f d W [2] and d f d W [1] respectively, using differential form,
[Equation]
We consider d f d W [2] ﬁrst,
[Equation]
where (cid:12) is Hadamard Product
[Equation]
and it is the multiplication of matrix elements of the same position [Equation]
Hence,
[Equation]
Then, we consider
[Equation]
[Equation]
Hence [Equation] we have,
[Equation]
Through the chain rule
[Equation]
we can get the evolution equation of W [1] and W [2] , d W [1]
[Equation] [Equation]
and
[Equation]
[Equation]
A.2. L HIDDEN LAYERS CONDITION
And, we consider the neural network with L hidden layers,
[Equation]
where Λ l (cid:44) ( W [ l +1] (cid:124) Λ l +1 ) (cid:12) σ (cid:48)
( W [ l ] x [ l − 1] ) for l = k, k + 1 . . .
L − 1 and Λ L (cid:44) a (cid:12) σ (cid:48)
( W [ L ] x [ L − 1] ) .
Hence, we get,
[Equation]
Through the chain rule, we can get the evolution equation of W [ k ] , [Equation] [ k ] [Equation]
[Equation]
Through a (cid:12) σ (cid:48)
( W x ) = diag { σ (cid:48)
( W x ) } a ,
Finally, the dynamic system can be obtained:
[Equation]
21) where we use E l ( x ) =
W [ l ] (cid:124) diag { σ (cid:48)
( W [ l ] x [ l − 1] ) } .
And E [ q : p ] =
E q E q +1 ...
E p .
Let r k,j = (cid:107) W [ k ] j (cid:107) 2 .
We have
[Equation]
[Equation]
Then we obtain
[Equation]
Finally, we get
[Equation]
where u k,j = W [ k ] j r k,j is a unit vector.
Then we have,
[Equation]
To conclude, the quantities we concern are summarized as follows,  
[Equation]
[Equation]
) ˙
[Equation]
30) where we use E l ( x ) =
W [ l ] (cid:124) diag { σ (cid:48)
( W [ l ] x [ l − 1] ) } .
And E [ q : p ] =
E q E q +1 ...
E p .
A.2. P ROVE FOR P w IN 5.
We calculate P w leading order ≈
Q w as following,
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] where ( · ) p − 1 and σ p ( · ) operate on component here.
We put the loss of the experiments in the main text here to show that they are indeed in the initial stage of training by the deﬁnition. As is shown in Fig.7 and Fig.8, at the number of steps I drew the graphs in the article, loss satisﬁes the deﬁnition of the initial stage, so we consider that they are in the initial stage of training. Only the epoch number could hardly reﬂect the initial stage, which also depends on the learning rate.
Therefore, we use the loss to indicate the initial stage. Learning rate is not a sensitive to the appearance of condensation. However, a small learning rate enables us to clearly observe the condensation process in the initial stage under a gradient ﬂow training. For example, when the learning rate is relatively small, the initial stage of training may be relatively long, while when the learning rate is relatively large, the initial stage of training may be relatively small.
We empirically ﬁnd that to ensure the training process follows a gradient follow, where the loss decays monotonically, we have to select a smaller learning rate for large multiplicity p . Therefore, it looks like we have a longer training in our experiments with large p .
Note that for a small learning rate in the experiments of small p , we can observe similar phenomena.
In all subsequent experiments in the Appendix, we will no longer show the loss graph of each experiment one by one, but we make sure that they are indeed in the initial stage of training. [Figure] [Figure] [Figure]
In the article, we only give the results of the last step of each condense, while the details of the evolution of condensation are lacking, which may provide us a better understanding. Therefore, we show these details in Fig.
9, Fig. 10, Fig. 11 and Fig. 12, which also further illustrate the rationality of the experimental results and facilitate the understanding of the evolution of condensation in the initial stage.
[Figure] [Figure] [Figure] [Figure]
The condensation of the Resnet18-like neural network on CIFAR10 and CIFAR100 is shown in Fig. 13 and Fig. 14, whose activation functions for fully-connected (FC) layers are tanh( x ) , ReLU( x ) , sigmoid( x ) , softplus( x ) and x tanh( x ) , indicated by the corresponding sub-captions, respectively.
[Figure]
size 128 are used for all experiments.
[Figure]
A. M ULTILAYER EXPERIMENTAL
The condensation of the six layer without residual connections is shown in 15, whose activation functions for hidden layer 1 to hidden layer 5 are x 2 tanh( x ) , x tanh( x ) , sigmoid( x ) , tanh( x ) and softplus( x ) , respectively.
The condensation of the three layer without residual connections is shown in 16, whose activation functions are same for each layer indicated by the corresponding sub-captions.
The condensation of the ﬁve layer without residual connections is shown in 17, whose activation functions are same for each layer indicated by the corresponding sub-captions. The condensation of the ﬁve layer with residual connections is shown in 18, whose activation functions are same for each layer indicated by the corresponding sub-captions.
[Figure] [Figure]
[Figure] [Figure]