Original: T OWARDS U NDERSTANDING R EGULARIZATION IN B ATCH N ORMALIZATION
Revised: T OWARDS U NDERSTANDING R EGULARIZATION IN B ATCH N ORMALIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Batch Normalization (BN) improves both convergence and generalization in training neural networks.
Revised: Batch Normalization (BN) improves both convergence and generalization in training neural networks.

Original: This work understands these phenomena theoretically.
Revised: This work understands these phenomena theoretically.

Original: We analyze BN by using a basic block of neural networks, consisting of a kernel layer, a BN layer, and a nonlinear activation function.
Revised: We analyze BN by using a basic block of neural networks, consisting of a kernel layer, a BN layer, and a nonlinear activation function.

Original: This basic network helps us understand the impacts of BN in three aspects.
Revised: This basic network helps us understand the impacts of BN in three aspects

Original: First, by viewing BN as an implicit regularizer, BN can be decomposed into population normalization (PN) and gamma decay as an explicit regularization.
Revised: First, by viewing BN as an implicit regularizer BN can be decomposed into population normalization ( PN) and gamma decay as an explicit regularization.

Original: Second, learning dynamics of
Revised: Second learning dynamics of

Original: BN and the regularization show that training converged with large maximum and effective learning rate.
Revised: BN and the regularization show that training converged with large maximum and effective learning rate.

Original: Third, generalization of BN is explored by using statistical mechanics.
Revised: Third , generalization of BN is explored by using statistical mechanics.

Original: Experiments demonstrate that BN in convolutional neural networks share the same traits of regularization as the above analyses.
Revised: Experiments demonstrate that BN in convolutional neural networks share the same traits of regularization as the above analyses.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Batch Normalization (BN) is an indispensable component in many deep neural networks (He et al., 2016; Huang et al., 2017).
Revised: Batch Normalization (BN) is an indispensable component in many deep neural networks (He et al., 2016; Huang et al., 2017).

Revised: BN has been widely used in various areas such as machine vision, speech and natural language processing.

Original: Experimental studies (Ioffe & Szegedy, 2015) suggested that BN improves convergence and generalization by enabling large learning rate and preventing overﬁtting in training.
Revised: Experimental studies (Ioffe & Szegedy, 2015) suggested that BN improves convergence and generalization by enabling large learning rate and preventing overﬁtting when training deep networks.

Original: Understanding BN theoretically is a key question.
Revised: Understanding BN theoretically is a key question.

Original: Notations. This work denotes a scalar and a vector by using lowercase letter ( e.g. x ) and bold lowercase letter ( e.g. x ) respectively. BN is investigated in a single-layer perceptron that is a building block of deep models, consisting of a kernel layer, a BN layer, and a nonlinear activation function.
Revised: This work investigates regularization of BN as well as its optimization and generalization in a singlelayer perceptron, which is a building block of deep models, consisting of a kernel layer, a BN layer, and a nonlinear activation function such as ReLU.

Original: Its forward computation can be written by
Revised: The computation of BN is written by

Original: [Equation]
Revised: [Equation]

Original: (1) where g ( · ) denotes an activation function such as ReLU, h and ˆ h arethe hidden values before and after  normalization, w and x are kernel weight vector and network input respectively.
Revised: This work denotes a scalar and a vector by using lowercase letter ( e.g. x ) and bold lowercase letter ( e.g. x ) respectively. In Eqn.(1), y is the output of a neuron, g ( · ) denotes an activation function, h and ˆ h are hidden values before and after batch normalization, w and x are kernel weight vector and network input respectively.

Original: In BN, µ B and σ B represent  mean and standard deviation of h .
Revised: In BN, µ B and σ B represent the mean and standard deviation of h .

Original: They are estimated within a batch of M samples.
Revised: They are estimated within a batch of samples for each neuron independently.

Original: γ is a scale parameter and β is a shift parameter.
Revised: γ is a scale parameter and β is a shift parameter.

Original: Despite the simplicity of the above basic network, it builds up the blocks of deep networks.
Revised: In what follows, Sec.1.1 overviews assumptions and main results, and Sec.1. presents relationships with previous work. 1. O VERVIEW OF R ESULTS We overview results in three aspects.

Revised: • First, Sec.2 decomposes BN into population normalization (PN) and gamma decay.

Revised: To better understand BN, we treat a single-layer perceptron with ReLU activation function as an illustrative case.

Original: It has been widely adopted in theoretical studies such as proper initialization (Krogh & Hertz, 1992; Advani & Saxe, 2017), dropout (Wager et al., 2013), weight decay and data augmentation (B¨os, 1998).
Revised: Despite the simplicity of this case, it is a building block of deep networks and has been widely adopted in theoretical analyses such as proper initialization (Krogh & Hertz, 1992; Advani & Saxe, 2017), dropout (Wager et al., 2013), weight decay and data augmentation (B¨os, 1998).

Revised: The results in Sec.2 can be extended to deep neural networks as presented in Appendix C.4.

Original: Our analyses assume that neurons at the BN layer are independent similar to (Salimans & Kingma, 2016; van Laarhoven, 2017), as the mean and the variance  are estimated individually for each neuron.
Revised: Our analyses assume that neurons at the BN layer are independent similar to (Salimans & Kingma, 2016; van Laarhoven, 2017; Teye et al., 2018) as the mean and the variance of BN are estimated individually for each neuron of each

Revised: layer.

Original: But we get rid of  Gaussian assumption on the network input and the weight vector in theorem 1 that is our main result, meaning our assumption is milder than those in (Yoshida et al.,2017; Ba et al., 2016;
Revised: The form of regularization in this study does not rely on Gaussian assumption on the network input and the weight vector, meaning our assumption is milder than those in (Yoshida et al.,

Original:  Salimans & Kingma, 2016).
Revised: ; Ba et al., 2016; Salimans & Kingma, 2016).

Original: Overall, several frequently-used notations are summarized in Table 2 in

Original: Appendix for reference.

Original: Out main results are organized below in three aspects. • First, Sec.2 decomposes BN into population normalization (PN) and gamma decay, which is an explicit regularization form of µ B and σ B. These statistics have different impacts: (1) µ B discourages reliance on a single neuron and encourages different neurons to have equal magnitude, in the sense that corrupting individual neuron does not harm generalization.
Revised: Sec.2 tells us that BN has an explicit regularization form, gamma decay, where µ B and σ B have different impacts: (1) µ B discourages reliance on a single neuron and encourages different neurons to have equal magnitude, in the sense that corrupting individual neuron does not harm generalization.

Original: This phenomenon was also found empirically in a recent work (Morcos et al., 2018), but has not been established analytically.
Revised: This phenomenon was also found empirically in a recent work (Morcos et al., 2018), but has not been established analytically.

Original: (2) σ B reduces kurtosis of the input distribution as well as correlations between neurons.
Revised: (2) σ B reduces kurtosis of the input distribution as well as correlations between neurons.

Original: The regularization strengths of these statistics are inversely proportional to the batch size M , indicating that BN with large batch would decrease generalization.
Revised: The regularization strengths of these statistics are inversely proportional to the batch size M , indicating that BN with large batch would decrease generalization.

Original: (4) Removing either one of µ B and σ B could imped convergence and generalization.
Revised: (4) Removing either one of µ B and σ B could imped convergence and generalization.

Original: • Second, by using ordinary differential equations (ODEs), Sec.3 shows that gamma decay enables the network trained with BN to converge with large maximum  and effective learning rate , leading to faster training speed compared to the network trained without BN or trained with weight normalization (WN) (Salimans & Kingma, 2016) that is a counterpart of BN. 
Revised: • Second, by using ordinary differential equations (ODEs), Sec.3 shows that gamma decay enables the network trained with BN to converge with large maximum learning rate and effective learning rate, compared to the network trained without BN or trained with weight normalization (WN) (Salimans & Kingma, 2016) that is a counterpart of BN. The maximum learning rate (LR) represents the largest LR value that allows training to converge to a ﬁxed point without diverging, while effective LR represents the actual LR in training. Larger maximum and effective LRs imply faster convergence rate.

Original:  Third, Sec.4 compares generalization errors of BN, WN, and vanilla SGD by using statistical mechanics.
Revised: • Third, Sec.4 compares generalization errors of BN, WN, and vanilla SGD by using statistical mechanics.

Original: The “large-scale” regime is of interest, where number of samples P and number of neurons N are both large but their ratio P/N is ﬁnite.
Revised: The “large-scale” regime is of interest, where number of samples P and number of neurons N are both large but their ratio P/N is ﬁnite.

Original: In this regime, the generalization errors are quantiﬁed both analytically and empirically.
Revised: In this regime, the generalization errors are quantiﬁed both analytically and empirically.

Original: Numerical results in Sec.5 show that BN in CNNs has the same traits of regularization as disclosed by the above analyses.
Revised: Numerical results in Sec.5 show that BN in CNNs has the same traits of regularization as disclosed above.

Revised: 1. R ELATED W ORK

Revised: Neural Network Analysis .

Revised: Many studies conducted theoretical analyses of neural networks (Opper et al., 1990; Saad & Solla, 1996; Bs & Opper, 1998; Pennington & Bahri, 2017; Zhang et al., 2017b; Brutzkus & Globerson, 2017; Raghu et al., 2017; Mei et al., 2016; Tian, 2017).

Revised: For example, for a multilayer network with linear activation function, Glorot & Bengio (2010) explored its SGD dynamics and Kawaguchi (2016) showed that every local minimum is global.

Revised: Tian (2017) studied the critical points and convergence behaviors of a 2-layered network with ReLU units.

Revised: Zhang et al.

Revised: (2017b) investigated a teacher-student model when the activation function is harmonic.

Revised: In (Saad &

Revised: Solla, 1996), the learning dynamics of a committee machine were discussed when the activation function is error function erf( x

Original: A P ROBABILISTIC I NTERPRETATION OF BN 
Revised: Unlike previous work, this work analyzes regularization emerged in BN and its impact to both learning and generalization, which are still unseen in the literature. Normalization . Many normalization methods have been proposed recently.

Revised: For example, BN (Ioffe & Szegedy, 2015) was introduced to stabilize the distribution of input data of each hidden layer.

Revised: Weight normalization (WN) (Salimans & Kingma, 2016) decouples the lengths of the network parameter vectors from their directions, by normalizing the parameter vectors to unit length.

Revised: The dynamic of WN was studied by using a single-layer network (Yoshida et al., 2017).

Revised: Li et al.

Revised: (2018) diagnosed the compatibility of BN and dropout (Srivastava et al., 2014) by reducing the variance shift produced by them.

Revised: Moreover, van Laarhoven (2017) showed that weight decay has no regularization effect when using together with BN or WN.

Revised: Ba et al.

Revised: (2016) demonstrated when BN or WN is employed, backpropagating gradients through a hidden layer is scale-invariant with respect to the network parameters.

Revised: Santurkar et al.

Revised: (2018) gave another perspective of the role of BN during training instead of reducing the covariant shift.

Revised: They argued that BN results in a smoother optimization landscape and the

Revised: Lipschitzness is strengthened in networks trained with BN.

Revised: However, both analytical and empirical results of regularization in BN are still desirable.

Revised: Our study explores regularization, optimization, and generalization of BN in the scenario of online learning.

Revised: Regularization .

Revised: Ioffe & Szegedy (2015) conjectured that BN implicitly regularizes training to prevent overﬁtting.

Revised: Zhang et al.

Original: We treat µ B and σ B as random variables to derive regularization of BN.
Revised: (2017a) categorized BN as an implicit regularizer from experimental results. Szegedy et al. (2015) also conjectured that in the Inception network, BN behaves similar to dropout to improve the generalization ability. Gitman & Ginsburg (2017) experimentally compared

Original: Since one sample x is seen many times in training and at each time it is presented with other samples in  a batch that is drawn randomly, µ B and σ B can be treated as injected random noise for x .
Revised: BN and WN, and also conﬁrmed the better generalization of BN. In the literature there are also implicit regularization schemes other than BN. For instance, random noise in the input layer for data augmentation has long been discovered equivalent to a weight decay method, in the sense that the inverse of the signal-to-noise ratio acts as the decay factor (Krogh & Hertz, 1992; Rifai et al., 2011).

Original: Loss Function.
Revised: Dropout (Srivastava et al., 2014) was also proved able to regularize training by using the generalized linear model (Wager et al., 2013).  A P ROBABILISTIC I NTERPRETATION OF BN The notations in this work are summarized in Appendix Table 2 for reference.

Original: Training a neural network typically involves minimizing a negative log likelihood function with respect to a set of  parameters θ = { w , γ, β } .
Revised: Training the above single-layer perceptron with BN typically involves minimizing a negative log likelihood function with respect to a set of network parameters θ = { w , γ, β } .

Original: Then the loss function can be deﬁned by
Revised: Then the loss function is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where p ( y j | ˆ h j ; θ ) represents the likelihood function of the network and P is number of training samples.
Revised: where p ( y j | ˆ h j ; θ ) represents the likelihood function of the network and P is number of training samples.

Original: As Gaussian distribution is often employed as prior distribution for the weight parameters, we have a weight decay ζ (cid:107) w (cid:107) 22  (Krizhevsky et al., 2012) that is a popular technique in deep learning.
Revised: As Gaussian distribution is often employed as prior distribution for the network parameters, we have a regularization term ζ (cid:107) θ (cid:107) 22 known as weight decay (Krizhevsky et al., 2012) that is a popular technique in deep learning, where ζ is a coefﬁcient.

Original: Prior.
Revised: To derive regularization of BN, we treat µ B and σ B as random variables. Since one sample x is seen many times in the entire training course, and at each time x is presented with the other samples in a batch that is drawn randomly, µ B and σ B can be treated as injected random noise for x .

Original:  By following (Teye et al., 2018), we ﬁnd that BN also induces Gaussian priors for µ B and σ B .
Revised: Prior of µ B , σ B . By following (Teye et al., 2018), we ﬁnd that BN also induces Gaussian priors for µ B and σ B .

Original: We have µ B ∼ N ( µ P , σ 2 P M ) and σ B ∼ N ( σ P , ρ +24 M ) , where M is batch size, µ P and σ P are population mean and standard deviation respectively, and ρ is kurtosis that measures the peakedness of the distribution of h .
Revised: We have µ B ∼ N ( µ P , σ 2 P M ) and σ B ∼ N ( σ P , ρ +24 M ) , where M is batch size, µ P and σ P are population mean and standard deviation respectively, and ρ is kurtosis that measures the peakedness of the distribution of h .

Original: These priors tell us that µ B and σ B would produce Gaussian noise in training.
Revised: These priors tell us that µ B and σ B would produce Gaussian noise in training.

Original: There is a tradeoff regarding this noise.
Revised: There is a tradeoff regarding this noise.

Original: For example, when M is small, training could diverge due to large noise.
Revised: For example, when M is small, training could diverge because the noise is large.

Original: This is supported by experiment of BN (Wu & He, 2018) where training diverges when M = 2 in ImageNet (Russakovsky et al., 2015).
Revised: This is supported by experiment of BN (Wu & He, 2018) where training diverges when M = 2 in ImageNet (Russakovsky et al., 2015).

Original: When M is large, the noise is reduced because µ B and σ B get close to µ P and σ P .
Revised: When M is large, the noise is small because µ B and σ B get close to µ P and σ P .

Original: It is known that M > 30 would provide a moderate noise, as the sample statistics converges in probability to the population statistics by the weak Law of Large Numbers.
Revised: It is known that M > 30 would provide a moderate noise, as the sample statistics converges in probability to the population statistics by the weak Law of Large Numbers.

Original: This is also supported by experiment (Ioffe & Szegedy, 2015) 
Revised: This is also supported by experiment (Ioffe & Szegedy, 2015) where

Original: where BN with M =already works well in ImageNet.
Revised: BN with M = 32 already works well in ImageNet.

Original: 2. A R EGULARIZATION F ORM
Revised: 2. A R EGULARIZATION F ORM

Original: The loss function in Eqn.(2) can be written as an expected loss by integrating over the priors of µ B and σ B , that is, 1 P (cid:80)
Revised: The loss function in Eqn.(2) can be written as an expected loss by integrating over the priors of µ B and σ B , that is, 1 P (cid:80)

Original: Pj =
Revised: Pj =

Original: 1 E µ B ,σ B [ (cid:96) (ˆ h j )] where E [ · ] denotes expectation.
Revised: 1 E µ B ,σ B [ (cid:96) (ˆ h j )] where E [ · ] denotes expectation.

Original: We show that µ B and σ B impose regularization on the scale parameter γ , but result in different regularization strengths.
Revised: We show that µ B and σ B impose regularization on the scale parameter γ by decomposing BN into population normalization (PN) and gamma decay.

Original: In theorem 1, we employ  ReLU activation function as a concrete example that is widely used in practice.
Revised: To see this, we employ a single-layer perceptron and ReLU activation function as an illustrative example.

Original: In general, the results can be extended to the other activation functions as shown in Appendix C.2.
Revised: A more rigorous description is provided in Appendix C.1.

Original: Theorem 1 (Regularization of µ B , σ B) .
Revised: Regularization of µ B , σ B .

Original: Let (cid:96) (ˆ h ) be the loss function of BN and  the activation function be ReLU.
Revised: Let (cid:96) ( ˆ h ) be the loss function deﬁned in Eqn.(2) and ReLU be the activation function.

Original: Then
Revised: We have

Original: P P (cid:88)

Original: j = E µ B ,σ B (cid:96) ( ˆ h j ) (cid:39) 1 P P (cid:88)
Revised: [Equation]

Original: [-j = (cid:96) ( ¯ h j ) + ζ ( h ) γ 
-]
Revised: [-j = (cid:96) ( ¯ h j ) + ζ ( h ) γ 
-]

Original: and ζ ( h ) =
Revised: and ζ ( h ) =

Original: ρ + 2 8 M F γ  (cid:124) (cid:123)(cid:122) (cid:125) from σ B + 1 2 M 1 P
Revised: ρ + 2 8 M I ( γ ) (cid:124) (cid:123)(cid:122) (cid:125) from σ B + 1 2 M 1 P

Original: P (cid:88)
Revised: P (cid:88)

Original: j = σ ( ¯ h j ) (cid:124) (cid:123)(cid:122) (cid:125) from µ B , (3) where ¯ h j = γ h j − µ P σ P + βrepresents the population normalization (PN) and h j = w T x j .
Revised: j = σ (¯ h j ) (cid:124) (cid:123)(cid:122) (cid:125) from µ B , (3) where ¯ h j = γ h j − µ P σ P + β and h j = w T x j represent the computations of PN.

Original: ζ ( h ) is a data-dependent coefﬁcient of gamma decay,  ρ is the kurtosis of distribution of h , F γ  represents 
Revised: ζ ( h ) γ 2 represents gamma decay, where ζ ( h ) is an adaptive decay factor depended on the hidden value h . Moreover, ρ is the kurtosis of distribution of h , I ( γ ) represents an estimation of the Fisher information of γ and

Original: Fisher Information Matrix (FIM) of γ  , and σ ( · ) is a sigmoid function.
Revised: I ( γ ) = 1 P (cid:80) P j =1 ( ∂(cid:96) (ˆ h j ) ∂γ ) 2 , and σ ( · ) is a sigmoid function.

Original: From theorem 1, we have several observations that are of both theoretical and practical values.
Revised: From Eqn.(3), we have several observations that have both theoretical and practical values.

Original: • First,it decomposes BN into population normalization (PN) and gamma decay. PN replaces the batch statistics  in BN by  population statistics.
Revised: • First, PN replaces the batch statistics µ B , σ B in BN by the population statistics µ P , σ P .

Original: In gamma decay, computation of ζ ( h ) is datadependent, making it differed from weight decay where the coefﬁcient is determined empirically.
Revised: In gamma decay, computation of ζ ( h ) is data-dependent, making it differed from weight decay where the coefﬁcient is determined manually.

Original: In essentials, Eqn. ) represents the randomness in BN in a deterministic way, not only enabling us to apply methodologies such as ODEs and statistical mechanics toanalyze BN , but
Revised: In fact, Eqn.(3) recasts the randomness of BN in a deterministic manner, not only enabling us to apply methodologies such as ODEs and statistical mechanics to

Original:  also inspiring us to imitate BN’s performance
Revised: analyze BN, but also inspiring us to imitate BN

Original:  by WN without computing batch statistics in numerical study
Revised: ’s performance by WN without computing batch statistics in empirical study.

Original: • Second, PN is closely connected to WN that is independent from sample mean and variance.WN
Revised: • Second, PN is closely connected to WN, which is independent from sample mean and variance.

Original:  (Salimans & Kingma, ) is deﬁned by υ w T x || w || 2 that normalizes the weight vector  to have unit variance, where υ is a learnable parameter.
Revised: WN (Salimans & Kingma, 2016) is deﬁned by υ w T x || w || 2 that normalizes the weight vector w to have unit variance, where υ is a learnable parameter.

Original: Let each diagonal element of the covariance matrix of x be a and all the off-diagonal elements be zeros.
Revised: Let each diagonal element of the covariance matrix of x be a and all the off-diagonal elements be zeros.

Original: ¯ h j  can be rewritten as
Revised: ¯ h j in Eqn.(3) can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: where υ = γa and b = − γµ P a || w || 2 + β . Eqn.(4) removes the estimations of statistics and eases our analyses of regularization for BN.
Revised: where υ = γa and b = − γµ P a || w || 2 + β . Eqn.(4) removes the estimations of statistics and eases our analyses of regularization for BN.

Original: • Third, µ B and σ B produce different parts in ζ ( h ) .
Revised: • Third, µ B and σ B produce different strengths in ζ ( h ) .

Original: The strength from µ B depends on the expectation of σ (¯ h j ) ∈ [0 , 1] , which represents excitation or inhibition of a neuron, meaning that a neuron with larger output may exposure to larger regularization, encouraging different neurons to have equal magnitude.
Revised: As shown in Eqn.(3), the strength from µ B depends on the expectation of σ (¯ h j ) ∈ [0 , 1] , which represents excitation or inhibition of a neuron, meaning that a neuron with larger output may exposure to larger regularization, encouraging different neurons to have equal magnitude.

Original: This is consistent with empirical result (Morcos et al., 2018) which prevented reliance on single neuron to improve generalization.
Revised: This is consistent with empirical result (Morcos et al., 2018) which prevented reliance on single neuron to improve generalization.

Original: The strength from σ B works as a complement for µ B .
Revised: The strength from σ B works as a complement for µ B .

Original: For a single neuron, F γ  represents the norm of gradient, implying that BN punishes large gradient norm.
Revised: For a single neuron, I ( γ ) represents the norm of gradient, implying that BN punishes large gradient norm.

Original: For multiple neurons, F γ  is the FIM of γ , meaning that BN would penalize correlations among neurons.
Revised: For multiple neurons, I ( γ ) is the Fisher information matrix of γ , meaning that BN would penalize correlations among neurons.

Original: Both σ B and µ B are important, removing either one of them would imped performance. 
Revised: Both σ B and µ B are important, removing either one of them would imped performance. Extensions to Deep Networks.

Original: We observe in experiments in Sec.5 thatBN in CNNs  share similar traits of regularization. However, in deep models the priors for σ B and µ B become multivariate Gaussian distributions where relationships between layers may not be neglected.
Revised: The above results can be extended to deep networks as shown in Appendix C.4 by decomposing the expected loss at a certain hidden layer. We also demonstrate the results empirically in Sec.5, where we observe that CNNs trained with BN share similar traits of regularization as discussed above.

Original: In this case, we didn’t ﬁnd meaningful analytical form for BN.

Original:  O PTIMIZATION WITH R EGULARIZATION
Revised:  O PTIMIZATION WITH R EGULARIZATION

Original: We show that BN converges with large maximum and effective learning rate (lr) that are larger than a network trained without BN.
Revised: Now we show that BN converges with large maximum and effective learning rate (LR), where the former one is the largest LR when training converged, while the latter one is the actual LR during training. With BN, we ﬁnd that both LRs would be larger than a network trained without BN.

Original: Our result explains why  large lr can be used in practicein BN (Ioffe & Szegedy, 2015).
Revised: Our result explains why BN enables large learning rates used in practice (Ioffe & Szegedy, 2015).

Original: Our analyses are conducted in three stages.
Revised: Our analyses are conducted in three stages.

Original: First, we establish dynamical equations of a teacher-student (T-S) model in  thermodynamic limit and acquire the ﬁxed point.
Revised: First, we establish dynamical equations of a teacherstudent model in the thermodynamic limit and acquire the ﬁxed point.

Original: Second, we investigate  eigenvalues of the corresponding Jacobian matrix at this ﬁxed point.
Revised: Second, we investigate the eigenvalues of the corresponding Jacobian matrix at this ﬁxed point.

Original: Finally, we calculate the maximum and the effective lr.
Revised: Finally, we calculate the maximum and the effective LR.

Original: Teacher-Student Model .
Revised: Teacher-Student Model .

Original: We ﬁrst introduce useful techniques from statistical mechanics (SM).
Revised: We ﬁrst introduce useful techniques from statistical mechanics (SM).

Original: In
Revised: With

Original: SM, a student network is dedicated to learn relationship between an input and an output with a weight vector w as parameters.
Revised: SM, a student network is dedicated to learn relationship between a Gaussian input and an output by using a weight vector w as parameters.

Original: It is useful to characterize behavior of the student by using a teacher network that uses w ∗ as a ground-truth  vector.
Revised: It is useful to characterize behavior of the student by using a teacher network with w ∗ as a ground-truth parameter vector.

Original: We treatthe single-layer perceptron as the student, which is optimized by minimizing the euclidian distance between its output and the supervision provided by a teacher without BN.
Revised: We treat single-layer perceptron as a student, which is optimized by minimizing the euclidian distance between its output and the supervision provided by a teacher without BN.

Original: The student and the teacher have identical activation function.
Revised: The student and the teacher have identical activation function.

Original: Loss Function.
Revised: Loss Function.

Original: We deﬁne a loss function of the above T-S model by 1 P (cid:80) Pj =1 (cid:96) ( x j ) = 1 P (cid:80) Pj =1 (cid:2) g ( w ∗ T x j ) − g ( √ Nγ w T x j (cid:107) w (cid:107) 2 ) (cid:3) 2 + ζγ 2 .
Revised: We deﬁne a loss function of the above teacher-student model by 1 P (cid:80) Pj =1 (cid:96) ( x j ) = 1 P (cid:80) Pj =1 (cid:2) g ( w ∗ T x j ) − g ( √ Nγ w T x j (cid:107) w (cid:107) 2 ) (cid:3) 2 + ζγ 2 , where g ( w ∗

Original: Here g ( w ∗ T x j ) represents supervision from the teacher, while g ( √ Nγ w T x j (cid:107) w (cid:107) 2 ) is the output of student trained to mimic its teacher.
Revised: T x j ) represents supervision from the teacher, while g ( √ Nγ w T x j (cid:107) w (cid:107) 2 ) is the output of student trained to mimic the teacher.

Original: The student is deﬁned by  Eqn.(4) where ν = √ Nγ and the bias term is merged into w .
Revised: This student is deﬁned by following Eqn.(4) with ν = √ Nγ and the bias term is absorbed into w .

Original: This loss function represents BN  using WN with gamma decay and it is sufﬁcient to study the lr of different approaches.
Revised: The above loss function represents BN by using WN with gamma decay, and it is sufﬁcient to study the learning rates of different approaches.

Original: Let θ = { w , γ } be a set of parameters updated by SGD,[-i.e. θ j +
-]
Revised: Let θ = { w , γ } be a set of parameters updated by SGD,[-i.e. θ j +
-]

Original: = θ j − η ∂(cid:96)
Revised: i [Equation]

Original: ( x j ) ∂θ j where η denotesa learning rate.
Revised: e. [Equation] j where η denotes learning rate.

Original: The update rules for w and γ are
Revised: The update rules for w and γ are

Original: [Equation]
Revised: [Equation]

Original: where ˜ w  denotes a normalized weight vector of the student, that is, ˜ w  = √
Revised: where ˜ w j denotes a normalized weight vector of the student, that is, ˜ w j = √

Original: Nγ  w  (cid:107) w  (cid:107) 2 , and δ j = g (cid:48)
Revised: Nγ j w j (cid:107) w j (cid:107) 2 , and δ j = g (cid:48)

Original: ( ˜ w j T x j )[ g ( w ∗ T x j ) − g ( ˜ w j T x j )] represents the gradient 1 for clarity of notation.
Revised: ( ˜ w j T x j )[ g ( w ∗ T x j ) − g ( ˜ w j T x j )] represents the gradient 1 for clarity of notations.

Original: Order Parameters .
Revised: Order Parameters .

Original: As we are interested in the “large-scale” regime where both N and P are large and their ratio P/N is ﬁnite, it is difﬁcult to examine a student with parameters in high dimensions directly.
Revised: As we are interested in the “large-scale” regime where both N and P are large and their ratio P/N is ﬁnite, it is difﬁcult to examine a student with parameters in high dimensions directly.

Original: Therefore, we transform the weight vectors to order parameters that fully characterize interactions between the student and the teacher.
Revised: Therefore, we transform the weight vectors to order parameters that fully characterize interactions between the student and the teacher network.

Original: In this case, the parameter vector can be reparameterized by using a vector of three elements including γ , R , and L .
Revised: In this case, the parameter vector can be reparameterized by using a vector of three elements including γ , R , and L .

Original: In particular, γ measures norm of the normalized weight vector ˜ w , that is, ˜ w T ˜ w =
Revised: In particular, γ measures length of the normalized weight vector ˜ w , that is, ˜ w T ˜ w =

Original: Nγ 2 w T w (cid:107) w (cid:107) 

Revised: Nγ 2 w T w (cid:107) w (cid:107)

Original: =
Revised: =

Original: Nγ 2 .
Revised: Nγ 2 .

Original: The parameter R measures angle (overlapping ratio) between the weight vectors of student and teacher.
Revised: The parameter R measures angle (overlapping ratio) between the weight vectors of student and teacher.

Original: We have
Revised: We have

Original: R =
Revised: R =

Original: [Equation]
Revised: [Equation]

Original:  ∗  where the norm of the ground-truth vector is 1 N w ∗ T w ∗ = 1 .
Revised: w ∗ , where the norm of the ground-truth vector is 1 N w ∗ T w ∗ = 1 .

Original: Moreover,
Revised: Moreover,

Original: L represents norm of the original weight vector w and L 2 = 1 N w T w .
Revised: L represents length of the original weight vector w and L 2 = 1 N w T w .

Original: With the above deﬁnitions, relationship between R and L can be represented by RL = 1 N w T w ∗ .

Original: Now we transform update equations (5)  by using  order parameters.
Revised: Learning Dynamics. The update equations (5) can be transformed into a set of differential equations (ODEs) by using the above order parameters.

Original: To this end, we deﬁne three variables γ 2 , L 2 and RL . The update rule for variable γ 2 can be obtained by (cid:0) γ 2 (cid:1) j +1 − (cid:0) γ 2 (cid:1) j  = 1 N (cid:2) 2 ηδ j ˜ w j T x j − 2 ηζ (cid:0) γ 2 (cid:1) j (cid:3) following update rule of γ .Similarly, the update rules for variables
Revised: This is achieved by treating the update step j as a continuous time variable t = jN .

Original: L 2 and RL are (cid:0) RL (cid:1) j +1 − (cid:0) RL (cid:1) j = 1 N (cid:0) ηγ j L j δ j w ∗ T x j − ηR j L j δ j ˜ w j T x j (cid:1) and (cid:0) L 2 (cid:1) j +1 − (cid:0) L 2 (cid:1) j = 1 N (cid:2) η 2 ( γ 2 )
Revised: They can be turned into differential equations because the contiguous step ∆ t = 1 N approaches zero in the thermodynamic limit when N → ∞ .

Original: j ( L 2 )

Original: j δ j 2 x j T x j − η 2 N ( L 2 ) j δ j 2 ( ˜ w j T x j ) 2 (cid:3) by multiplying both sides of (5) by w ∗ . To deﬁne the learning dynamics, we turn the above update rules into ODEs. We take γ 2 as an example.
Revised: We obtain a dynamical system of three order parameters

Original: Its differential equation can be deﬁned by dγ

Original: [Equation]
Revised: [Equation]

Original: dt =lim
Revised: where I 1 =

Original: [Equation] where t = jN is a normalized sample index that can be treated as a continuous time variable. We have ∆ t =1 N that approaches zero in the thermodynamic limit when N → ∞ . (cid:104)·(cid:105)
Revised: E x [ δ ˜ w T x ] , I 2 =

Original:  x denotes expectation over the distribution of x . The differential equations of dRLdt and dL 2 dt can be deﬁned in the same way.
Revised: E x [ δ 2 x T x ] , and I 3 =

Original: [-We simplify notations by representing I 
 = (cid:104) δ ˜ w T x (cid:105) x , I 2 = (cid:104) δ 2-] x T x (cid:105) x and I 3 = (cid:104) δ w ∗ T x (cid:105) x . We obtain a dynamical system
Revised: [-We simplify notations by representing I 
 = (cid:104) δ ˜ w T x (cid:105) x , I 2 = (cid:104) δ 2-]E x [ δ w ∗ T x ] are deﬁned to simplify notations.

Original: [Equation] More results are provided in Appendix C.4.
Revised: The derivations of Eqn.(6) can be found in Appendix C.5.

Original: 3. F IXED P OINT OF THE D YNAMICAL S YSTEM
Revised: 3. F IXED P OINT OF THE D YNAMICAL S YSTEM

Original: [Table]
Revised: [Table]

Original: To investigate lr of BN, we derive the ﬁxed point of (6) by setting dγ/dt = dR/dt = dL/dt = 0 .
Revised: To ﬁnd the ﬁxed points of (6), we set dγ/dt = dR/dt = dL/dt = 0 .

Original: The ﬁxed points of BN, WN, and vanilla SGD (without BN and WN) are given in Table 1.
Revised: The ﬁxed points of BN, WN, and vanilla SGD (without BN and WN) are given in Table 1.

Original: In the thermodynamic limit, the optima for ( γ 0 , R 0 , L 0 ) would be (1 , 1 , 1) .
Revised: In the thermodynamic limit, the optima denoted as ( γ 0 , R 0 , L 0 ) would be ( γ 0 , R 0 , L 0 ) =

Original: Our main interest is the overlapping ratio R 0 between the student and the teacher, because it optimizes the direction of the weight vector.
Revised: Our main interest is the overlapping ratio R 0 between the student and the teacher, because it optimizes the direction of the weight vector regardless of its length.

Original: We see that R 0 for all three approaches attain optimum ‘1’.
Revised: We see that R 0 for all three approaches attain optimum ‘1’.

Original: Intuitively, in BN and WN, this optimal solution does not depend on Lbecause their weight vectors are normalized.
Revised: Intuitively, in BN and WN, this optimal solution does not depend on the value of L 0 because their weight vectors are normalized.

Original: In other words, WN and BN are easier to optimize than vanilla  SGD where both R 0 and L 0 have to be optimized.
Revised: In other words, WN and BN are easier to optimize than vanilla SGD, unlike SGD where both R 0 and L 0 have to be optimized to ‘1’.

Original: In BN, γ 0  depends on the activation function.
Revised: Furthermore, γ 0 in BN depends on the activation function.

Original: For ReLU, we have γ bn 

Revised: For ReLU, we have γ bn

Original: = 1 2 ζ +1 , meaning that norm of the normalized weight vector relies on the decay factor.
Revised: = 1 2 ζ +1 (see Proposition 1 in Appendix C.5), meaning that norm of the normalized weight vector relies on the decay factor ζ .

Original: In WN, we have γ wn 

Revised: In WN, we have γ wn

Original: = 1 as WN has no regularization on γ .
Revised: = 1 as WN has no regularization on γ .

Original: 3. M AXIMUM AND E FFECTIVE L EARNING R ATES
Revised: 3. M AXIMUM AND E FFECTIVE L EARNING R ATES

Original: With the above ﬁxed points, we derive the maximum and the effective lr.
Revised: With the above ﬁxed points, we derive the maximum and the effective LR.

Original: Speciﬁcally, we analyze eigenvalues and eigenvectors of the Jacobian matrix corresponding to (6).
Revised: Speciﬁcally, we analyze eigenvalues and eigenvectors of the Jacobian matrix corresponding to Eqn.(6).

Original: We are interested in the lr to approach R 0 .
Revised: We are interested in the LR to approach R 0 .

Original: We ﬁnd that this optimum value only depends on its  eigenvalue denoted as λ R , λ R = ∂I 2 ∂R ηγ 0 2{+
+} L 20 ( η max − η eﬀ ) , where η max and η eﬀ are maximum and effective lr (proposition 1 in Appendix C.4).
Revised: We ﬁnd that this optimum value only depends on its corresponding eigenvalue denoted as λ R . We have λ R = ∂I 2 ∂R ηγ 0 {+
+} L 20 ( η max − η eﬀ ) , where η max and η eﬀ represent the maximum and effective LR (proposition 2 in Appendix C.5), which are given in Table 1.

Original: They are given in Table 1.

Original: We demonstrate that λ R < 

Revised: We demonstrate that λ R <

Original: if and only if η max > η eﬀ , such that the ﬁxed point R 0 is stable for all approaches (proposition 2 in Appendix C.5).
Revised: if and only if η max > η eﬀ , such that the ﬁxed point R 0 is stable for all approaches (proposition 3 in Appendix C.6).

Original: It is  able to show that η max of BN ( η bn max ) is larger than WN and SGD, enabling R to converge with a larger learning rate.
Revised: Moreover, it is also able to show that η max of BN ( η bn max ) is larger than WN and SGD, enabling R to converge with a larger learning rate.

Original: With ReLU, we ﬁnd that η bn max ≥ η { wn,sgd } max + 2 ζ
Revised: For ReLU as an example, we ﬁnd that η bn max ≥ η { wn,sgd } max + 2 ζ

Original: proposition 3 in Appendix C.6).
Revised: proposition 4 in Appendix C.7).

Revised: The larger maximum LRs enables the network to be trained more stably and has the potential to be combined with other stabilization techniques (Fagan & Iyengar, 2018) during optimization.

Original: Moreover, the effective lr’s in Table 1 are consistent with previous work (van Laarhoven, 2017).
Revised: The effective LRs shown in Table 1 are consistent with previous work (van Laarhoven, 2017).

Original:  G ENERALIZATION A NALYSIS
Revised:  G ENERALIZATION A NALYSIS

Original: To investigate generalization of BN, we adopt a teacher-student model with identity activation function, which minimizes a loss function 1 P (cid:80) Pj =1 (¯ y  j − y j ) 2 , where¯ y  represents the teacher’s output and y is the student’s output.
Revised: Here we investigate generalization of BN by using a teacher-student model that minimizes a loss function 1 P (cid:80) Pj =1 (( y ∗ ) j − y j ) 2 , where y ∗ represents the teacher’s output and y is the student’s output.

Original: We compare BN with WN+gamma decay and vanilla SGD.
Revised: We compare BN with WN+gamma decay and vanilla SGD.

Original: All of them share the same teacher network whose output is deﬁned by ¯ y  =
Revised: All of them share the same teacher network whose output is a noise-corrupted linear function y ∗ =

Original: w ∗ T x + (cid:15) , where x is drawn from N (0 , 1 N ) and (cid:15) is an unobserved Gaussian noise.
Revised: w ∗ T x + s , where x is drawn from N (0 , 1 N ) and s is an unobserved Gaussian noise.

Original: We are interested to see how different methods resist this noise.
Revised: We are interested to see how the above methods resist this noise by using student networks with both identity (linear) and ReLU activation functions.

Original: For vanilla SGD , the student is computed by y =  w T x  with  w being the weight vector to optimize, where w has the same dimension as w ∗ to be a realizable learning problem.
Revised: For vanilla SGD , the student is computed by y = g ( w T x ) with g ( · ) being either identity or ReLU, and w being the weight vector to optimize, where w has the same dimension as w ∗ .

Original: The loss function of vanilla SGD is (cid:96) sgd = 1 P (cid:80) Pj =1 ( w ∗T x j −  w T x j )2 , whose solution asymptotically approaches the Moore–Penrose pseudo inverse solution w = (cid:0) x T x (cid:1) + x T ¯ y .
Revised: The loss function of vanilla SGD is (cid:96) sgd = 1 P (cid:80) Pj =1 (cid:0) y ∗ − g ( w T x j ) (cid:1) 2 .

Original: For BN , the student is deﬁned as y = γ w T x − µ B σ B + β .
Revised: For BN , the student is deﬁned as y = γ w T x − µ B σ B + β .

Original: As our main interest is the weight vector, we freeze the bias similar to vanilla
Revised: As our main interest is the weight vector, we freeze the bias by setting β = 0 .

Original: SGD by setting β = 0 . Therefore, the  loss function is written as (cid:96) bn = 1 P (cid:80)
Revised: Therefore, the batch average term µ B is also unnecessary to avoid additional parameters, and the loss function is written as (cid:96) bn = 1 P (cid:80)

Original: Pj =1 (cid:0) w ∗ T x
Revised: Pj =1 (cid:0) ( y ∗ )

Original: j − γ( w T x j− µ B ) /σ B (cid:1) 2 .
Revised: j − γ w T x j /σ B (cid:1) 2 .

Original: For WN+gamma decay , the student is computed similar to Eqn.(4) by  y = √
Revised: For WN+gamma decay , the student is computed similar to Eqn.(4) by using y = √

Original: Nγ w T x (cid:107) w (cid:107) 2 .
Revised: Nγ w T x (cid:107) w (cid:107) 2 .

Original: Then the loss function is deﬁned by (cid:96)wn
Revised: Then the loss function is deﬁned by (cid:96)

Original: = 1 P (cid:80)
Revised: [Equation]

Original: Pj =1 (cid:0) w ∗ T x j − √ Nγ w T x j (cid:107) w (cid:107) 2 (cid:1) 2 + ζ (cid:107) γ (cid:107) 22 .In the T-S model with identity unit, expression of ζ becomes ζ = 12 M after applying theorem 1 (Appendix C.3).
Revised: γ (cid:107) 22 .

Original: With the above deﬁnitions, the three approaches are studied under the same T-S model, where their generalization errors can be strictly compared with the other factors ruled out.
Revised: With the above deﬁnitions, the three approaches are studied under the same teacher-student framework, where their generalization errors can be strictly compared with the other factors ruled out.

Original: 4. G ENERALIZATION E RRORS
Revised: 4. G ENERALIZATION E RRORS

Original: g e n g e n
Revised: [Figure]

Original: We provide closed-form solutions of the generalization errors  for vanilla SGD  and WN+gamma decay.
Revised: We provide closed-form solutions of the generalization errors (see Appendix D.1) for vanilla SGD with both linear and ReLU student networks.

Revised: The theoretical solution of WN+gamma decay can also be solved for the linear student, but still remains difﬁcult for ReLU student whose numerical veriﬁcation is provided instead.

Original: They are compared with numerical solutions of BN. vanilla SGD.
Revised: Both vanilla SGD and WN+gamma decay are compared with numerical solutions of BN. vanilla SGD.

Original: The solution of generalization error depends on the rank of correlation matrix Σ
Revised: In an identity (linear) student, the solution of generalization error depends on the rank of correlation matrix Σ

Original: = x T x .
Revised: = x T x .

Original: Here we deﬁne an effective load α = P/N that is the ratio between number of samples P and number of input neurons N (number of learnable parameters).
Revised: Here we deﬁne an effective load α = P/N that is the ratio between number of samples P and number of input neurons N (number of learnable parameters).

Original: The generalization error  denoted as (cid:15) sgd gen can be acquired by using the distribution of eigenvalues of Σ following (Advani & Saxe, 2017).
Revised: The generalization error of the identity student is denoted as (cid:15) sgd id , which can be acquired by using the distribution of eigenvalues of Σ following (Advani & Saxe, 2017).

Original: If α < 1 , (cid:15) sgd gen = 1 − α + α(cid:15) 2 / (1 − α ) .
Revised: If α < 1 , (cid:15) sgd id = 1 − α + αS/ (1 − α ) .

Original: Otherwise, (cid:15) sgd gen = (cid:15) 2 / (1 − α ) where (cid:15) is the  injected noise to the teacher.
Revised: Otherwise, (cid:15) sgd id = S/ ( α − 1) where S is the variance of the injected noise to the teacher network.

Original: The values of (cid:15) sgd gen with respect to α are plotted in blue curvein the top of Fig.1.
Revised: The values of (cid:15) sgd id with respect to α are plotted in blue curve of Fig.1(a).

Original: It ﬁrst decreases
Revised: It ﬁrst decreases

Original: but then increases as α increases from 0 to 1, (cid:15) sgd gen diverges at α = 1 , and it would decrease again 
Revised: but then increases as α increases from 0 to 1. (cid:15) sgd id diverges at α = 1 . And it would decrease again when α > 1 .

Original: when α > 1  . 
Revised: In a ReLU student, the nonlinear activation yields difﬁculty to derive the theoretical solution. Here we utilize the statistical mechanics and calculate that (cid:15) sgd relu = 1 − α/ 4 + αS 2(2 − α ) and α < 2 (see AppendixD.2). When comparing to the lower bound (trained without noisy supervision) shown as the red curve in Fig.1(b), we see that (cid:15) sgd relu (blue curve) diverges at α = 2 . This is because the student overﬁts the noise in the teacher’s output.

Original:  WN+gamma decay.
Revised: The curve of numerical solution is also plotted in dashed line in Fig.1(b) and it captures the diverging trend well. It should be noted that obtaining the theoretical curve empirically requires an inﬁnitely long time of training and an inﬁnitely small learning rate. This unreachable limit explains the discrepancies between the theoretical and the numerical solution. WN+gamma decay.

Original: The decay term turns the correlation matrix to Σ = (cid:0) x
Revised: In a linear student, the gamma decay term turns the correlation matrix to Σ = (cid:0) x

Original: T x + ζ I (cid:1) that is positive deﬁnite.
Revised: T x + ζ I (cid:1) , which is positive deﬁnite.

Original: Following statistical mechanics (Krogh & Hertz, 1992), the generalization error is (cid:15) wn gen = δ 2 ∂ ( ζG ) ∂ζ − ζ 2 ∂G∂ζ where
Revised: Following statistical mechanics (Krogh & Hertz, 1992), the generalization error is (cid:15) wn id [Equation] G = 1 − α − ζ + (cid:0) ζ + [Equation]

Revised: 1 +

Original: [Equation]
Revised: [Equation]

Original:  We see that (cid:15) wn gen can be computed quantitatively given the values of ζ and α .
Revised: α ) 2 (cid:1) 12 (cid:0) ζ + [Equation] 1 − √ α ) 2 (cid:1) 12 (cid:14) 2 ζ. We see that (cid:15) wn id can be computed quantitatively given the values of ζ and α .

Original: Let the variance of noise(cid:15) injected to the teacher be 0.25.
Revised: Let the variance of noise injected to the teacher be 0 .

Original: Fig.1 bottom shows that no other curves could outperform the  curve when ζ = 0 .
Revised: Fig.1(a) shows that no other curves could outperform the red curve when ζ = 0 .

Original: 25 , a value equal to the noise magnitude.
Revised: 25 , a value equal to the noise magnitude.

Original: The ζ smaller than 0 .
Revised: The ζ smaller than 0 .

Original: 25 ( e.g. green curveof ζ = 12 M and M = 32 ) would exhibit overtraining around α = 1 , but they still perform signiﬁcantly better than vanilla SGD.
Revised: 25 (green curve ζ = 12 M and M = 32 ) would exhibit overtraining around α = 1 , but they still perform signiﬁcantly better than vanilla SGD.

Original: [Figure]
Revised: For the ReLU student in Fig,1(b), a direct solution of the generalization error (cid:15) wn relu remains an open problem. Therefore, the numerical results of ‘WN+gamma decay’ (green curve) are run at each α value. It effectively reduces over-ﬁtting compared to vanilla SGD.

Original: Numerical Solutions of BN.
Revised: Numerical Solutions of BN.

Original: We employ SGD with M = 32 to ﬁnd solutions of w for BN.
Revised: In the linear student, we employ SGD with M = 32 to ﬁnd solutions of w for BN.

Original: The generalization error is evaluated as difference between the validation and the training loss.

Original: The number of input neurons is 4096 and the number of training samples can be varied to change α .
Revised: The number of input neurons is 4096 and the number of training samples can be varied to change α .

Original: The results are marked as black squares in the top of Fig.1.
Revised: The results are marked as black squares in Fig.1(a).

Original: After applyingtheorem 1 to the T-S model, BN is equivalent to WN+gamma decay when ζ = 12 M .
Revised: After applying the analyses for linear student (Appendix C.3), BN is equivalent to ‘WN+gamma decay’ when ζ = 12 M (green curve).

Original: It is seen that BN gets in line with the curveof ‘ ζ =1/2M’ ( M = 32 ) and thus quantitatively validates our derivations.
Revised: It is seen that BN gets in line with the curve ‘ ζ = 1 / 2 M ’ ( M = 32 ) and thus quantitatively validates our derivations.

Revised: In the ReLU student, the setting is mostly the same as the linear case, except that we employ a smaller batch size M = 16 .

Original: Their generalization errors are further compared in  the bottom of Fig.1 at α = 1 , where vanilla SGD clearly diverges, while BN and WN+gamma decay are comparable.
Revised: The results are shown as black squares in Fig.1(b). For ReLU units, the equivalent ζ of gamma decay is ζ = 14 M . If one compares the generalization error of BN with ‘WN+gamma decay’ (green curve), a clear correspondence is found, which also validates the derivations for the ReLU activation function.

Original:  E XPERIMENTS IN CNN S
Revised:  E XPERIMENTS IN CNN S

Original: This section shows that BN in CNNs follows similar traits of regularization as our analyses.
Revised: This section shows that BN in CNNs follows similar traits of regularization as the above analyses.

Original: We employ a 6-layered CNN similar to (Salimans & Kingma, 2016).

Original: For all experiments, the network architecture is ﬁxed while  the normalization layers can be changed.
Revised: To compare different methods, the CNN architectures are ﬁxed while only the normalization layers are changed.

Original: We adopt CIFAR10 (Krizhevsky, 2009) that contains 60k images of 10 categories (50k images for training and 10k images for test).
Revised: We adopt CIFAR10 (Krizhevsky, 2009) that contains 60k images of 10 categories (50k images for training and 10k images for test).

Original: All models are trained by using SGD with momentum on a single GPU, while the initial learning rates are scaled proportionallyfor different batch sizes (Goyal et al., 2017).
Revised: All models are trained by using SGD with momentum, while the initial learning rates are scaled proportionally (Goyal et al., 2017) when different batch sizes are presented.

Original: In order to study regularization of BN, we discard any other trick such as weight decay and data augmentation.

Original: More empirical setting can be found in Appendix B.
Revised: More empirical setting can be found in Appendix B.

Original: Evaluation of Theorem 1 .
Revised: Evaluation of PN+Gamma Decay .

Revised: This work shows that BN can be decomposed into PN and gamma decay.

Original: We  compare BN with PN+gamma decay where the population statistics  and the regularization coefﬁcient are estimated by using sufﬁcient amount of training samples.
Revised: We empirically compare ‘PN+gamma decay’ with BN by using ResNet18 (He et al., 2016). For ‘PN+gamma decay’, the population statistics of PN and the decay factor of gamma decay are estimated by using sufﬁcient amount of training samples.

Original:  BN trained with a normal batch size M = 64 is treated as baseline  in Fig.2.
Revised: For BN, BN trained with a normal batch size M = 128 is treated as baseline as shown in Fig.2(a&b).

Original: When batch size increases,
Revised: We see that when batch size increases,

Original: BN would imped both loss and accuracy.
Revised: BN would imped both loss and accuracy.

Original: For example, when increasing M to 256 , performance decreases because the regularization from the batch statistics reduces in large batch, resulting in overtraining (see the gap between train and validation loss  when M = 256 ).[Equation]
Revised: For example, when increasing M to 1024 , performance decreases because the regularization from the batch statistics reduces in large batch, resulting in overtraining (see the gap between train and validation loss in (a) when M = 1024 ).

Original: In comparison, we train PN by using 10k training samples to estimate  statistics.
Revised: In comparison, we train PN by using 10k training samples to estimate the population statistics.

Original: This further reduces regularization.
Revised: Note that this further reduces regularization.

Original: We see that the release of regularization can be complemented by gamma decay, making PNeven outperformed BN.
Revised: We see that the release of regularization can be complemented by gamma decay, making PN outperformed BN.

Original: This empirical result veriﬁes our derivation of regularization for BN.
Revised: This empirical result veriﬁes our derivation of regularization for BN.

Original: Similar trend can be observed by experiment in a downsampled version of ImageNet (see Appendix B.1).
Revised: Similar trend can be observed by experiment in a down-sampled version of ImageNet (see Appendix B.1).

Original: Nevertheless, we would like to point out that PN+gamma decay is of interest in theoretical analysis, but it is computation-demanding when applied in practice because evaluating µ P , σ P and ζ ( h ) may require sufﬁciently large number of samples.
Revised: We would like to point out that ‘PN+gamma decay’ is of interest in theoretical analyses, but it is computation-demanding when applied in practice because evaluating µ P , σ P and ζ ( h ) may require sufﬁciently large number of samples.

Original: Study of Regularization.
Revised: Comparisons of Regularization.

Original: We study the regulation strengthes of vanilla SGD, BN, WN, WN+mean-only BN, and WN+varianceonly BN.
Revised: We study the regulation strengths of vanilla SGD, BN, WN, WN+mean-only BN, and WN+variance-only BN.

Revised: At ﬁrst, the strength of regularization terms from both µ B and σ B are compared by using a simpler network with 4 convolutional and 2 fully connected layers as used in (Salimans & Kingma, 2016).

Original: Fig.3 compares their training and validation losses.
Revised: Fig.2(c&d) compares their training and validation losses.

Original: We see that the generalization error of BN is much lower than WN and vanilla SGD.
Revised: We see that the generalization error of BN is much lower than WN and vanilla SGD.

Original: The reason has been disclosed in this work: stochastic behaviors of µ B and σ B in BN improves generalization. 
Revised: The reason has been disclosed in this work: stochastic behaviors of µ B and σ B in BN improves generalization. [Figure]

Original: To investigate µ B and σ B individually, we decompose their contributions by running a WN with mean-only BN as well as a WN with variance-only BN, to simulate their respective regularization.
Revised: To investigate µ B and σ B individually, we decompose their contributions by running a WN with mean-only BN as well as a WN with variance-only BN, to simulate their respective regularization.

Original: As shown in Fig.3, improvements from the mean-only and the variance-only BN over WN verify our conclusion that noises from µ B and σ B have different regularization strengths.
Revised: As shown in Fig.2(c&d), improvements from the mean-only and the variance-only BN over WN verify our conclusion that noises from µ B and σ B have different regularization strengths.

Original: Both of them are essential to produce good result. 
Revised: Both of them are essential to produce good result. 5. A BLATION S TUDY

Original: Parameter Norm.
Revised: Parameter Norm.

Original: We further demonstrate impact of BN to the norm of parameters.
Revised: We further demonstrate impact of BN to the norm of parameters.

Original: We compare
Revised: We compare

Original: BN with vanilla SGD.
Revised: BN with vanilla SGD.

Original: A network is ﬁrst trained by BN in order to converge to a local minima where the parameters do not change much.
Revised: A network is ﬁrst trained by BN in order to converge to a local minima where the parameters do not change much.

Original: At this local minima, the weight vector is frozen and denoted as w bn .
Revised: At this local minima, the weight vector is frozen and denoted as w bn .

Original: Then this network is ﬁnetuned by using vanilla SGD with a small learning rate 10 − 3 and its kernel parameters are initialized by w sgd
Revised: Then this network is ﬁnetuned by using vanilla SGD with a small learning rate 10 − 3 and its kernel parameters are initialized by w sgd

Original: = γ w bn σ , where σ is the moving average of σ B .
Revised: = γ w bn σ , where σ is the moving average of σ B .

Original: Fig.7 in Appendix B.2 visualizes the results.
Revised: Fig.3 visualizes the results.

Original: As µ B and σ B are removed in vanilla SGD, it is foundfrom the last two ﬁgures that the training loss decreases while the validation loss increases, implying that reduction in regularization makes the network converged to a sharper local minimum that generalizes less well.
Revised: As µ B and σ B are removed in vanilla SGD, it is found that the training loss decreases while the validation loss increases, implying that reduction in regularization makes the network converged to a sharper local minimum that generalizes less well.

Original: The magnitudes of kernel parameters w sgd at different layers are also displayed in the ﬁrst four ﬁgures.

Original: All of them increase after freezing BN, due to the release of regularization on these parameters.[Figure] [Equation] [Figure]
Revised: The magnitudes of kernel parameters w sgd at different layers are also observed to increase after freezing BN, due to the release of regularization on these parameters.

Original: [Equation]

Original: Batch size.
Revised: Batch size.

Original: To study BN with different batch sizes, we train different networks but only add BN at one layer at a time.
Revised: To study BN with different batch sizes, we train different networks but only add BN at one layer at a time.

Original: The regularization on the γ parameter is compared in Fig.4 (a) when BN is located at different layers.
Revised: The regularization on the γ parameter is compared in Fig.2(e) when BN is located at different layers.

Original: The values of γ 2 increase along with the batch size M due to the weaker regularization for the larger batches.
Revised: The values of γ 2 increase along with the batch size M due to the weaker regularization for the larger batches.

Original: The increase of γ 2 also makes all validation losses increased as shown in Fig.4 (b).
Revised: The increase of γ 2 also makes all validation losses increased as shown in Fig.2(f).

Original: BN+dropout .
Revised: BN+dropout .

Original: Despite the better generalization of BN with smaller batch sizes, large-batch training is more efﬁcient in real cases.
Revised: Despite the better generalization of BN with smaller batch sizes, large-batch training is more efﬁcient in real cases.

Original: Therefore, improving generalization of BN with large batch is more desiring.
Revised: Therefore, improving generalization of BN with large batch is more desiring.

Original: However, gamma decay requires estimating the population statistics that increases computations.
Revised: However, gamma decay requires estimating the population statistics that increases computations.

Original: We also found that treating the decay coefﬁcient as a constant hardly improves generalization for large batch.
Revised: We also found that treating the decay factor as a constant hardly improves generalization for large batch.

Original: Therefore, we utilize dropout as an alternative to compensate for the insufﬁcient regularization.
Revised: Therefore, we utilize dropout as an alternative to compensate for the insufﬁcient regularization.

Original: Dropout has also been analytically viewed as a regularizer (Wager et al., 2013).
Revised: Dropout has also been analytically viewed as a regularizer (Wager et al., 2013).

Original: We add a dropout after each BN layer to impose regularization.
Revised: We add a dropout after each BN layer to impose regularization.

Original: Fig.5 plots the results.
Revised: Fig.2(g&h) plot the classiﬁcation results using ResNet18.

Original: The generalization of BN deteriorates signiﬁcantly when M increases from 64 to 256.
Revised: The generalization of BN deteriorates signiﬁcantly when M increases from 128 to 1024.

Original: This is observed by the much higher validation loss (top) and lower validation accuracy (bottom) when M = 256 .
Revised: This is observed by the much higher validation loss (Fig.2(g)) and lower validation accuracy (Fig.2(h)) when M = 1024 .

Original: If a dropout layer with ratio 0 .
Revised: If a dropout layer with ratio 0 .

Original: 125 is added after each BN layer for M = 256 , the validation loss is suppressed and accuracy increased by a great margin.
Revised: 1 is added after each residual block layer for M = 1024 in ResNet18, the validation loss is suppressed and accuracy increased by a great margin.

Original: This superﬁcially contradicts with the original  claim that BN reduces the need for dropout (Ioffe & Szegedy, 2015).
Revised: This superﬁcially contradicts with the original [Figure] claim that BN reduces the need for dropout (Ioffe & Szegedy, 2015).

Original: There are two differences between our study and previous work.
Revised: We ﬁnd that there are two differences between our study and (Ioffe & Szegedy, 2015).

Original: First, in pervious study the batch size was ﬁxed at a quite small value ( e.g. 32), at which the regularization was already quite strong.
Revised: First, in pervious study the batch size was ﬁxed at a quite small value ( e.g. 32), at which the regularization was already quite strong.

Original: Therefore, an additional dropout could not further cause better regularization, but on the contrary increases the instability in training and yields a lower accuracy.
Revised: Therefore, an additional dropout could not further cause better regularization, but on the contrary increases the instability in training and yields a lower accuracy.

Original: However, our study explores relatively large batch that degrades the regularization of BN, and thus dropout with a small ratio can complement.
Revised: However, our study explores relatively large batch that degrades the regularization of BN, and thus dropout with a small ratio can complement.

Original: Second, usual trials put dropout before BN and cause BN to have different variances during training and test.
Revised: Second, usual trials put dropout before BN and cause BN to have different variances during training and test.

Original: In contrast, dropout follows BN in this study and  thus the problem can be alleviated.
Revised: In contrast, dropout follows BN in this study and the distance between two dropout layers is large (a residual block separation), thus the problem can be alleviated.

Original: The improvement by applying dropout after BN has also been observed by a recent work (Li et al., 2018).
Revised: The improvement by applying dropout after BN has also been observed by a recent work (Li et al., 2018).

Original: WN+dropout .
Revised: WN+dropout .

Original: Since BN can be treated as WN trained with regularization  in this study, combining WN with regularization should be able to match the performance of BN.
Revised: Since BN can be treated as WN trained with regularization as shown in this study, combining WN with regularization should be able to match the performance of BN.

Original: As WN outperforms BN in running speed (without calculating statistics) and it suits better in RNNs than BN, an improvement of its generalization is also of great importance.
Revised: As WN outperforms BN in running speed (without calculating statistics) and it suits better in RNNs than BN, an improvement of its generalization is also of great importance.

Original: Fig.5 shows that WN can also be regularized by dropout.
Revised: Fig.2(g&h) also show that WN can also be regularized by dropout.

Original: We apply dropout after each WN layer with ratio 0.25.
Revised: We apply dropout after each WN layer with ratio 0.2 and the dropout is applied at the same layers as that for BN.

Original: We found that the improvement on both validation accuracy and loss is surprising.
Revised: We found that the improvement on both validation accuracy and loss is surprising.

Original: The accuracy increases from 0.73 to 0.80, surpassing ‘BN M=256’ and on par with ‘BN M=64’.  R ELATED W ORK Neural Network Analysis .
Revised: The accuracy increases from 0.90 to 0.93, even close to the results of BN.

Original: Many studies analysed neural networks (Opper et al., 1990; Saad & Solla, 1996; Bs & Opper, 1998; Pennington & Bahri, 2017; Zhang et al., 2017b; Brutzkus & Globerson, 2017; Raghu et al., 2017; Mei et al., 2016; Tian, 2017).

Original: For example, for a multilayer network with linear activation function, Glorot & Bengio (2010) explored its SGD dynamics and Kawaguchi (2016) showed that every local minimum is global.

Original: Tian (2017) studied the critical points and convergence behaviors of a 2-layered network with ReLU units.

Original: Zhang et al.

Original: (2017b) investigated a teacher-student model when the activation function is harmonic. In (Saad & Solla, 1996), the learning dynamics of a committee machine were discussed when the activation function is error function erf( x ) . Unlike previous work, this work analyzes regularization emerged in BN and its impact to both learning and generalization, which are still unseen in the literature. Normalization .
Revised: Nevertheless, additional regularization on WN still cannot make WN on par with the performance BN.

Original: Many normalization methods have been proposed recently. For example, BN (Ioffe & Szegedy, 2015) was introduced to stabilize the distribution of input data of each hidden layer. Weight normalization (WN) (Salimans & Kingma, 2016) decouples the lengths of the network parameter vectors fromtheir directions, by normalizing the parameter vectors to unit length. The dynamic of WN was studied by using asingle-layer network (Yoshida et al., 2017).
Revised: In deep neural networks the distribution after each layer would be far from a

Original: Moreover, Li et al.

Original: (2018) diagnosed the compatibility of BN and dropout (Srivastava et al., 2014) by reducing the variance shift produced by them.

Original: van Laarhoven (2017) showed that weight decay has no regularization effect when using together with BN or WN. Ba et al. (2016) demonstrated when BN or WN is employed, back-propagating gradients through a hidden layer is scale-invariant with respect to the network parameters.
Revised: Gaussian distribution, in which case WN is not a good substitute for PN.

Original: Santurkar et al.

Original: (2018) gave another perspective of the role of BN during training instead of reducing the covariant shift.

Original: They argued that BN results in a smoother optimization landscape and the Lipschitzness is strengthened in networks trained with BN.

Original: However, both analytical and empirical results of regularization in BN are still desirable. Our study explores regularization, optimization, and generalization of BN in the scenario ofonline learning. Regularization . Ioffe & Szegedy (2015) conjectured that BN implicitly regularizes training to prevent overﬁtting.
Revised: A potential substibute of

Original: Zhang et al. (2017a) categorized BN as an implicit regularizer from experimental results. Szegedy et al. (2015) also conjectured that in the Inception network, BN behaves similar to dropout to improve the generalization ability.
Revised: BN would require us for designing better estimations of the distribution to improve the training speed and performance of deep networks.

Original: Gitman & Ginsburg (2017) experimentally compared

Original: BN and WN, and also conﬁrmed the better generalization of BN.

Original: In the literature there are also implicit regularization schemes other than BN.

Original: For instance, random noise in the input layer for data augmentation has long been discovered equivalent to a weight decay method, in the sense that the inverse of the signal-to-noise ratio acts as the decay factor (Krogh & Hertz, 1992; Rifai et al., 2011).

Original: Dropout (Srivastava et al., 2014) was also proved able to regularize training by using the generalized linear model (Wager et al., 2013).

Original:  D ISCUSSIONS AND F UTURE W ORK
Revised:  D ISCUSSIONS AND F UTURE W ORK

Original: This work investigated  regularization emerged in BN. By utilizing a single-layer perceptron, BN was decomposed into PN and gamma decay, where the regularization strengths from µ B and σ Bare different and their impacts in training were explored.
Revised: This work investigated an explicit regularization form of BN, which was decomposed into PN and gamma decay where the regularization strengths from µ B and σ B were explored.

Original: Moreover, convergence and generalization of BN with regularization were derived and compared with vanilla SGD, WN, and WN+gamma decay, showing that BN enables training to converge with large maximum and effective learning rate, as well as leads to better generalization.
Revised: Moreover, optimization and generalization of BN with regularization were derived and compared with vanilla SGD, WN, and WN+gamma decay, showing that BN enables training to converge with large maximum and effective learning rate, as well as leads to better generalization.

Original: Our analytical results explain many existing empirical phenomena.
Revised: Our analytical results explain many existing empirical phenomena.

Original: Experiments in CNNs showed that BN in deep networks share the same traits of regularization.
Revised: Experiments in CNNs showed that BN in deep networks share the same traits of regularization.

Original: Furthermore, a combination of dropout and BN might ameliorate BN when batch size goes large.
Revised: Furthermore, a combination of dropout and BN might ameliorate BN when batch size goes large.

Original: Our result also encourages us to combine WN and dropout, outperforming BN in some senses without estimating batch statistics.
Revised: Our result also encourages us to combine WN and dropout, outperforming BN in some senses without estimating batch statistics.

Original: In future work, we are interested in ﬁnding analytical form ofregularization for BN in deep networks, although it might involve multivariate Gaussian prior distributions, making it a nontrivial problem.
Revised: In future work we are interested in analyzing optimization and generalization of BN in deep networks, which is still an open problem.

Original: Moreover, investigating the other normalizers such as instance normalization (IN) (Ulyanov et al., 2016) and layer normalization (LN) (Ba et al., 2016) is also important.
Revised: Moreover, investigating the other normalizers such as instance normalization (IN) (Ulyanov et al., 2016) and layer normalization (LN) (Ba et al., 2016) is also important.

Original: Understanding the characteristics of these normalizers should be the ﬁrst step to analyze some recent best practices such as group normalization (Wu & He, 2018) that merged IN and LN, and switchable normalization (Luo et al., 2018) that chose BN, IN, and LN in each normalization layer.
Revised: Understanding the characteristics of these normalizers should be the ﬁrst step to analyze some recent best practices such as group normalization (Wu & He, 2018) that merged IN and LN and switchable normalization (Luo et al., 2018) that chose BN, IN and LN in each normalization layer.

Original: Furthermore, devising an efﬁcient counterpart of gamma decay is desirable in the community and will be investigated in the future, as it may improve generalization of WN that is independent of batch statistics
Revised: Furthermore, devising an efﬁcient counterpart of gamma decay is desirable in the community and will be investigated in the future, as it may improve generalization of WN that is independent of batch statistics

Original: A PPENDICES A N OTATIONS
Revised: A PPENDICES A N OTATIONS

Original: [Table]
Revised: [Table]

Original: B M ORE E MPIRICAL S ETTINGS AND R ESULTS
Revised: B M ORE E MPIRICAL S ETTINGS AND R ESULTS

Original: All experiments in Sec.5 are conducted in CIFAR10 with a CNN architecture similar to (Salimans & Kingma, 2016) that is summarized as ‘conv(3,32)-conv(3,32)-conv(3,64)-conv(3,64)-pool(2,2)- fc(512)-fc(10)’, where ‘conv(3,32)’ represents a convolution with kernel size 3 and 32 channels, ‘pool(2,2)’ is max-pooling with kernel size 2 and stride
Revised: All experiments in Sec.5 are conducted in CIFAR10 by using ResNet18 and a CNN architecture similar to (Salimans & Kingma, 2016) that is summarized as ‘conv(3,32)-conv(3,32)-conv(3,64)- conv(3,64)-pool(2,2)-fc(512)-fc(10)’, where ‘conv(3,32)’ represents a convolution with kernel size 3 and 32 channels, ‘pool(2,2)’ is max-pooling with kernel size 2 and stride

Original: [Table]
Revised: [Table]

Original: and ‘fc’ indicates a full connection.
Revised: and ‘fc’ indicates a full connection.

Original: We follow a conﬁguration for training by using SGD with a momentum value of 0.9 and continuously decaying the learning rate by a factor of 10 − 4 each step.
Revised: We follow a conﬁguration for training by using SGD with a momentum value of 0. and continuously decaying the learning rate by a factor of 10 − 4 each step.

Original: For different batch sizes, the initial learning rate is scaled proportionally with the batch size to maintain a similar learning dynamics (Goyal et al., 2017)
Revised: For different batch sizes, the initial learning rate is scaled proportionally with the batch size to maintain a similar learning dynamics (Goyal et al., 2017)

Original: [Table]
Revised: [Table]

Original: B. R ESULTS IN DOWNSAMPLED I MAGE N ET
Revised: B. R ESULTS IN DOWNSAMPLED I MAGE N ET

Original: Besides CIFAR10, we also evaluate theorem 1 by employing a downsampled version of ImageNet (Loshchilov & Hutter, 2016), which contains identical 1.2 million data and 1k categories as the original ImageNet, but each image is scaled to 32 × 32.
Revised: Besides CIFAR10, we also evaluate ‘PN+gamma decay’ by employing a downsampled version of ImageNet (Loshchilov & Hutter, 2016), which contains identical 1.2 million data and 1k categories as the original ImageNet, but each image is scaled to 32 × 32.

Original: We train ResNet18 in downsampled ImageNet by following the training protocol used in (He et al., 2016).
Revised: We train ResNet18 in downsampled ImageNet by following the training protocol used in (He et al., 2016).

Original: In particular, ResNet18 is trained by using SGD with momentum of 0.9 and the initial learning rate is 0.1, which is then decayed by a factor of 10 after 30, 60, and 90 training epochs.
Revised: In particular, ResNet18 is trained by using SGD with momentum of 0.9 and the initial learning rate is 0.1, which is then decayed by a factor of 10 after 30, 60, and 90 training epochs.

Original: In downsampled ImageNet, we observe similar trends as those presented in CIFAR10.
Revised: In downsampled ImageNet, we observe similar trends as those presented in CIFAR10.

Original: For example, we see that BN would imped both loss and accuracy when batch size increases.
Revised: For example, we see that BN would imped both loss and accuracy when batch size increases.

Original: When increasing M to 1024 as shown in Fig.6, both the loss and validation accuracy decrease because the regularization from the random batch statistics reduces in large batch size, resulting in overtraining.
Revised: When increasing M to 1024 as shown in Fig.4, both the loss and validation accuracy decrease because the regularization from the random batch statistics reduces in large batch size, resulting in overtraining.

Original: This can be seen by the gap between the training and the validation loss.
Revised: This can be seen by the gap between the training and the validation loss.

Original: Nevertheless, we see that the reduction of regularization can be complemented when PN is trained with adaptive gamma decay, which makes
Revised: Nevertheless, we see that the reduction of regularization can be complemented when PN is trained with adaptive gamma decay, which makes

Original: PN performed comparably to BN in downsampled ImageNet.
Revised: PN performed comparably to BN in downsampled ImageNet.

Original: B. I MPACT OF BN TO THE N ORM OF P ARAMETERS

Original: We demonstrate the impact of BN to the norm of parameters.

Original: We compare BN with vanilla SGD, where a network is ﬁrst trained by BN in order to converge to a local minima when the parameters do not change much.

Original: At this local minima, the weight vector is frozen and denoted as w bn .

Original: Then this

Original: [Figure]

Original: network is ﬁnetuned by using vanilla SGD with a small learning rate 10 − 3 with the kernel parameters initialized by w sgd = γ w bn σ , where σ is the moving average of σ B .

Original: Fig.7 below visualizes the results.

Original: As µ B and σ B are removed in the vanilla SGD, it is found from the last two ﬁgures that the training loss decreases while the validation loss increases, meaning that the reduction in regularization makes the network converged to a sharper local minimum that generalizes less well.

Original: The magnitudes of kernel parameters w sgd at different layers are also displayed in the ﬁrst four ﬁgures.

Original: All of them increase after freezing BN, due to the release of regularization on these parameters.

Original: [Figure]
Revised: [Figure]

Original: C P ROOF OF R ESULTS C. P ROOF OF T HEOREM
Revised: C P ROOF OF R ESULTS C. P ROOF OF E QN .(3)

Original: Theorem 1 (Regularization of µ B , σ B ) .
Revised: Theorem 1 (Regularization of µ B , σ B ) .

Original: Let ζ be  the strength (coefﬁcient) of the regularization and  the activation function be ReLU.
Revised: Let a single-layer perceptron with BN and ReLU activation function be deﬁned by y = max(0 , ˆ h ) , ˆ h = γ h − µ B σ B + β and h = w T x , where x and y are the network input and output respectively, h and ˆ h are the hidden values before and after batch normalization, and w is the weight vector. Let (cid:96) (ˆ h ) be the loss function.

Original: Then
Revised: Then

Revised: [Equation]

Revised: ζ ( h ) γ 2 and ζ ( h ) =

Revised: ρ + 2 8 M I ( γ ) + 1 2 M

Original: [Equation]
Revised: [Equation]

Revised: P

Revised: P (cid:88)

Original:  where ¯ h j = γ w T x j − µ P σ P + β , ρ is the kurtosis of the distribution of w T x , F γ  is a Fisher Information
Revised: j [Equation] σ (¯ h j ) , where ¯ h j = γ w T x j − µ P σ P + β represents population normalization (PN), ζ ( h ) γ 2 represents gamma decay and ζ ( h ) is a data-dependent decay factor. ρ is the kurtosis of the distribution of h , I ( γ ) is an estimation of the Fisher information of γ and I ( γ )

Revised: = 1 P (cid:80)

Original: Matrix of γ , and σ ( · ) is a sigmoid function.
Revised: Pj =1 ( ∂(cid:96) (ˆ h j ) ∂γ ) 2 , and σ ( · ) is a sigmoid function.

Original: Proof.
Revised: Proof.

Original: Let ˆ h j = γ w T x j − µ B σ B + [Equation] and ¯ h
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: P + β .
Revised: + β .

Original: We prove theorem 1 by performing a 
Revised: We prove theorem 1 by performing a Taylor expansion on a function

Original: Taylor expansion on a function A (ˆ h j ) at ¯ h j , where A (ˆ h j ) is a function of ˆ h j deﬁned according to a particular activation function.
Revised: A (ˆ h j ) at ¯ h j , where A (ˆ h j ) is a function of ˆ h j deﬁned according to a particular activation function.

Original: The negative log likelihood function of the  single-layer perceptron can be generally deﬁned as − log p ( y j | ˆ h j )
Revised: The negative log likelihood function of the above single-layer perceptron can be generally deﬁned as − log p ( y j | ˆ h j )

Original: =
Revised: =

Original: A (ˆ h j ) − y j ˆ h j , which is similar to the loss function of the generalized linear models with different activation functions. 
Revised: A (ˆ h j ) − y j ˆ h j , which is similar to the loss function of the generalized linear models with different activation functions. Therefore, we have [Equation] h

Original: [Equation] 
Revised: [Equation] +

Revised: P

Revised: [Equation]

Revised: [Equation]

Revised: =

Revised: P

Revised: P (cid:88)

Revised: j = l (

Revised: [Equation]

Revised: h j )

Revised: [Equation]

Revised: R f

Original:  where A (cid:48) ( · ) and A (cid:48)(cid:48) ( · ) denote the ﬁrst and second derivatives of function A ( · ) .
Revised: [Equation] R [Equation] where A (cid:48) ( · ) and A (cid:48)(cid:48) ( · ) denote the ﬁrst and second derivatives of function A ( · ) .

Original: The ﬁrst and second order terms in the expansion are represented by R f and R q respectively.
Revised: The ﬁrst and second order terms in the expansion are represented by R f and R q respectively.

Original: To derive the analytical forms of R f and R q , we take a second-order Taylor expansion of of 1 σ B and 1 σ 2 B around σ P , it sufﬁces to have
Revised: To derive the analytical forms of R f and R q , we take a second-order Taylor expansion of of 1 σ B and 1 σ 2 B around σ P , it sufﬁces to have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: By applying the distributions of µ B and σ B  in the paper
Revised: By applying the distributions of µ B and σ B introduced in section 2, we have µ B ∼ N ( µ P , σ 2 P M ) and σ B ∼ N ( σ P , ρ +24 M ) .

Original: [Equation] R f can be derived as
Revised: Hence, R f can be derived as in the paper, R f can be derived as

Original: [Equation]
Revised: [Equation]

Original: This R f term can be understood as below.
Revised: This R f term can be understood as below.

Original: Let h = w T x − µ P σ P and the distribution of the population data be p xy .
Revised: Let h = w T x − µ P σ P and the distribution of the population data be p xy .

Original: We establish the following relationship
Revised: We establish the following relationship

Original: [Equation]
Revised: [Equation]

Original: Since the sample mean converges in probability to the population mean by the Weak Law of
Revised: Since the sample mean converges in probability to the population mean by the Weak Law of

Original: Large Numbers, for all (cid:15) > 0 and a constant number K ( ∃ K > 0 and ∀ P > K ), we have (cid:12)(cid:12) R [Equation] − E [Equation]
Revised: Large Numbers, for all (cid:15) > 0 and a constant number K ( ∃ K > 0 and ∀ P > K ), we have p (cid:0)(cid:12)(cid:12) R f − E ( x,y ) ∼ p

Original: y

Original: [Equation]

Original: E

Original: [Equation]

Original: A

Original: [Equation] ¯ [Equation] − y ) h (cid:3) (cid:12)(cid:12) <
Revised: xy E µ B ,σ B (cid:2) ( A (cid:48) (¯ h ) − y ) h (cid:3) (cid:12)(cid:12) ≥ ρ +24 M (cid:15) (cid:1)

Original: [Equation] + [Equation] 4 M (cid:15) .
Revised: = 0 .

Original: The above equation means that R f is sufﬁciently small  given moderately large number of data points P (the above inequality holds when P > 30 ).
Revised: This equation implies that R f is sufﬁciently small with a probability of 1 given moderately large number of data points P (the above inequality holds when P > 30 ).

Original: On the other hand, R q can be derived as
Revised: On the other hand, R q can be derived as

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Note that ∂ 2 l (¯ h j )
Revised: Note that ∂ 2 l (¯ h j )

Original: ∂γ= A (cid:48)(cid:48) (¯ h j )( w T x j − µ P σ P ) 2 , we have F ( γ )=
Revised: ∂γ= A (cid:48)(cid:48) (¯ h j )( w T x j − µ P σ P ) 2 , we have I ( γ )

Original:  1 P (cid:80) Pj =

Revised: = 1 P (cid:80) Pj =

Original: A (cid:48)(cid:48) (¯ h j )( w T x j − µ P σ P ) 2 been an estimator of the Fisher Information Matrix (FIM) with respect to the scale parameter γ , according to the deﬁnition of Fisher information.
Revised: A (cid:48)(cid:48) (¯ h j )( w T x j − µ P σ P ) 2 been an estimator of the Fisher information with respect to the scale parameter γ .

Original: Then, by neglecting O (1 /M 2 ) high-order term in R q , we get
Revised: Then, by neglecting O (1 /M 2 ) high-order term in R q , we get

Original: [Equation]
Revised: [Equation]

Original: where µ d 2 A indicates the mean of the second derivative of A ( h ) .
Revised: where µ d 2 A indicates the mean of the second derivative of A ( h ) .

Revised: The results of both ReLU activation function and identity function are provided as below.

Original: C.T HEOREM 1 WITH R E LU 
Revised: C. R E LU A CTIVATION F UNCTION

Original: For the ReLU non-linear activation function, that is f ( h ) =
Revised: For the ReLU non-linear activation function, that is g ( h ) =

Original: max( h, 0) , we use its continuous approximation softplus function f ( h )
Revised: max( h, 0) , we use its continuous approximation softplus function g ( h )

Original: = log(1 + exp( h )) to derive the partition function A ( h ) .
Revised: = log(1+exp( h )) to derive the partition function A ( h ) .

Original: In this case, we have µ d 2 A = 1 P (cid:80) Pj =1 σ (¯ h j ) . 
Revised: In this case, we have µ d 2 A = 1 P (cid:80) Pj =1 σ (¯ h j ) . Therefore, we have ζ ( h ) =

Original: Therefore, we have ζ = ρ +28 M F γ  + 12 M 1 P (cid:80) Pj =1 σ (¯ h j ) as shown in theorem 1.
Revised: ρ +28 M I ( γ ) + 12 M 1 P (cid:80) Pj =1 σ (¯ h j ) as shown in Eqn.(3).

Original: C. T HEOREM 1 WITH I DENTITY ACTIVATION F UNCTION
Revised: C. L INEAR S TUDENT N ETWORK WITH I DENTITY A CTIVATION F UNCTION

Original: For a loss function in the form L = 1 P (cid:80) Pj =1 (cid:0) w ∗ T x j − γ ( w T x j  ) /σ B (cid:1) 2 , we haveF γ = 2 λ and the regularization contribution from µ B can be neglected.
Revised: For a loss function with identity (linear) units, 1 P (cid:80) Pj =1 (cid:0) w ∗ T x j − γ ( w T x j − µ B ) /σ B (cid:1) 2 , we have

Original: We also have ρ = 0 for Gaussian input distribution.
Revised: I ( γ ) = 2 λ and ρ = 0 for Gaussian input distribution.

Original: The exact expression of theorem 1 is also possible for such linear regression problem.
Revised: The exact expression of Eqn.(3) is also possible for such linear regression problem.

Original: Under the condition of Gaussian input x ∼ N (0 , 1 /N ) , h = w T x is also a random variable satisfying a normal distribution N (0 , 1) , it can be derived thatE (cid:0) σ − 1 B (cid:1) =
Revised: Under the condition of Gaussian input x ∼ N (0 , 1 /N ) , h = w T x is also a random variable satisfying a normal distribution N (0 , 1) . It can be derived that

Original:  √ M √ 2 σ P Γ ( M − 2 2 ) Γ ( M − 1 2 ) and E (cid:0) σ − 2 B (cid:1) =
Revised: E (cid:0) σ − 1 B (cid:1) = √ M √ 2 σ P Γ ( M − 2 2 ) Γ ( M − 1 2 ) and E (cid:0) σ − 2 B (cid:1) =

Original: Mσ 2 P Γ ( M − 1 2 − 1 ) 
Revised: Mσ 2 P Γ ( M − 1 2 − 1 ) Γ ( M − 1 2 ) .

Original: Γ M −2 ) , therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: Furthermore, the expression of ζ can be simpliﬁed as ζ = 34 M .
Revised: Furthermore, the expression of ζ can be simpliﬁed as ζ = 34 M .

Original: If the bias term is neglected in a simple linear regression, contributions from µ B to the regularization term is neglected and thus ζ = 14 M .
Revised: If the bias term is neglected in a simple linear regression, contributions from µ B to the regularization term is neglected and thus ζ = 14 M .

Original: Note that if one uses mean square error without being divided by 2 during linear regression, the values for ζ should be multiplied by 2 as well, where ζ = 12 M . 
Revised: Note that if one uses mean square error without being divided by 2 during linear regression, the values for ζ should be multiplied by 2 as well, where ζ = 12 M . C. BN R EGULARIZATION IN A D EEP N ETWORK

Revised: The previous derivation is based on the single-layer perceptron.

Revised: In deep neural networks, the forward computation inside one basic building block of a deep network is written by

Revised: [Equation]

Revised: where the superscript l ∈ [1 , L ] is the index of a building block in a deep neural network, and i ∈ [1 , N l ] indexes each neuron inside a layer.

Original: C. D YNAMICAL E QUATIONS{+z 
 and z L are synonyms of input x and output y , respectively. In order to analyze the regularization of BN from a speciﬁc layer, one needs to isolate its input and focus on the noise introduced by the BN layer in this block. Therefore, the loss function (cid:96) (ˆ h l ) can also be expanded at (cid:96) (¯ h l ) .+}
Revised: {+z 
 and z L are synonyms of input x and output y , respectively. In order to analyze the regularization of BN from a speciﬁc layer, one needs to isolate its input and focus on the noise introduced by the BN layer in this block. Therefore, the loss function (cid:96) (ˆ h l ) can also be expanded at (cid:96) (¯ h l ) .+}

Revised: In BN, the batch variance is calculated with regard to each neuron under the assumption of mutual independence of neurons inside a layer.

Revised: By following this assumption and the above derivation in Appendix C.1, the loss function with BN in deep networks can also be similarly decomposed.

Revised: Regularization of µ l B , σ l B in a deep network.

Revised: Let ζ l be the strength (coefﬁcient) of the regularization at the l -th layer.

Revised: Then

Revised: [Equation]

Revised: and ζ li = 1 P

Revised: [Equation]

Revised: where i is the index of a neuron in the layer, (¯

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: ) li + β li represents population

Revised: normalization (PN),

Revised: H

Revised: (cid:96) (¯ h l ) is the Hessian matrix at ¯ h l regarding to the loss (cid:96) and diag( · ) represents the diagonal vector of a matrix

Revised: It is seen that the above equation is compatible with the results from the single-layer perceptron.

Revised: The main difference of the regularization term in a deep model is that the Hessian matrix is not guaranteed to be positive semi-deﬁnite during training.

Revised: However, this form of regularization is also seen from other regularization such as noise injection (Rifai et al., 2011) and dropout (Wager et al., 2013), and has long been recognized as a Tikhonov regularization term (Bishop, 1995).

Revised: In fact, it has been reported that in common neural networks, where convex activation functions such as ReLU and convex loss functions such as common cross entropy are adopted, the Hessian matrix

Revised: H (cid:96) (¯ h l ) can be seen as ‘locally’ positive semideﬁnite (Santurkar et al., 2018).

Revised: Especially, as training converges to its mimimum training loss, the Hessian matrix of the loss can be viewed as positive semi-deﬁnite and thus the regularization term on γ l is positive.

Revised: C. D YNAMICAL E QUATIONS

Original: Here we discuss the dynamical equations of BN.
Revised: Here we discuss the dynamical equations of BN.

Original: Let the length of teacher’s weight vector be 1, that is, 1 N w ∗ T w ∗ = 1 .
Revised: Let the length of teacher’s weight vector be 1, that is, 1 N w ∗ T w ∗ = 1 .

Original: We introduce a normalized weight vector of the student as (cid:101) w = √
Revised: We introduce a normalized weight vector of the student as (cid:101) w = √

Original: Nγ w (cid:107) w (cid:107) .
Revised: Nγ w (cid:107) w (cid:107) .

Original: Then the overlapping ratio between teacher and student, the length of student’s vector, and the length of student’s normalized weight vector are 1 N (cid:101) w T w ∗ =
Revised: Then the overlapping ratio between teacher and student, the length of student’s vector, and the length of student’s normalized weight vector are 1 N (cid:101) w T w ∗ =

Original: QR = γR , 1 N (cid:101) w T (cid:101)
Revised: QR = γR , 1 N (cid:101) w T (cid:101)

Original: w = Q 2 = γ 2 , and
Revised: w = Q 2 = γ 2 , and

Original: N w T w =
Revised: N w T w =

Original: L 2 respectively, where Q = γ .
Revised: L 2 respectively, where Q = γ .

Original: And we have 1 N w T w∗ = 
Revised: And we have 1 N w T w = LR .

Revised: We transform update equations (5) by using order parameters.

Revised: The update rule for variable Q 2 can be obtained by (cid:0)

Original: LR . 
Revised: [Equation] following update rule of γ . Similarly, the update rules for variables RL and L 2 are calculated as follow: [Equation]

Revised: Let t =

Original:  We write the dynamical equations of BN as follows,
Revised: j N is a normalized sample index that can be treated as a continuous time variable [Equation] We have ∆ t = 1 N that approaches zero in the thermodynamic limit when N → ∞ . In this way, the learning dynamic of Q 2 , RL and L 2 can be formulated as the following differential equations:

Original: [Equation]
Revised: [Equation]

Original: where I 1 = (cid:104) δ ˜ w T x (cid:105) x , I 2 = (cid:104) δ 2 x T x (cid:105) x , and I 3 = (cid:104) δ w ∗ T x (cid:105) x , which are the terms presented in dγ 2 dt , dRLdt , and dL 2 dt  .
Revised: where I 1 = (cid:104) δ ˜ w T x (cid:105) x , I 2 = (cid:104) δ 2 x T x (cid:105) x , and I 3 = (cid:104) δ w ∗ T x (cid:105) x , which are the terms presented in dQ 2 dt , dRLdt , and dL 2 dt and (cid:104)·(cid:105) x denotes expectation over the distribution of x .

Original: They are used to simplify notations.
Revised: They are used to simplify notations.

Revised: Note that we neglect the last term of dL 2 /dt in Eqn.(8) since η 2 N ( L 2 ) δ 2 ( ˜ w T x )can be approximately equal to zero when N approaches inﬁnity.

Revised: On the other hand, we have dQ 2 = 2 QdQ, dRL = RdL + LdR and dL 2 = 2 LdL .

Revised: Hence, Eqn.(9) can be reduced to

Revised: [Equation]

Original: Proposition 1.
Revised: Proposition 1.

Original: Let λ bnQ , λ bnR be the eigenvalues of  the Jacobian matrix at θ 

Revised: Let ( Q 0 , R 0 , L 0 ) denote a ﬁxed point with parameters Q , R and L of Eqn.(10). Assume the learning rate η is sufﬁciently small when training converges and x ∼ N (0 , 1 N I ) . If activation function g is ReLU , then we have Q 0 = 1 2 ζ +1 , R

Original: = {+
+}
Revised: = {+
+}

Original: ( Q 0 , 1 , L 0 ) corresponding to the order parameters Q and R respectively in BN. Then [Equation] where η bn max and η bn eﬀ are the maximum and effective learning rates respectively in BN.
Revised: and L 0 could be arbitrary.

Original: Proof.
Revised: Proof.

Original: Firstly, note that the change of  L will not change I 1 , [-I 
-]
Revised: First, L has no inﬂuence on the output of student model since w is normalized, which implies that if ( Q 0 , R 0 , L 0 ) is a ﬁxed point of Eqn.(10), L 0 could be arbitrary. Besides, we have η (cid:29) ηbecause the learning rate η is sufﬁciently small. Therefore, the terms in Eqn.(10) proportional to ηcan be neglected. If ( Q 0 , [-I 
-]R 0 , L 0 ) is a ﬁxed point, it sufﬁces to have

Original: {+[Equation] To calculate I 
+} and I 3  .
Revised: {+[Equation] To calculate I 
+} and I 3 , we deﬁne s and t as ˜ w T x and w T x .

Original: Thus we have ∂I 1 ∂L = ∂I 2 ∂L = ∂I 3 ∂L = 0 .
Revised: Since x ∼ N (0 , 1 N I ) , we can acquire [Equation] so probability measure of [ s, t ] T can be written as [Equation]

Original: And we also neglect the term proportional to η 2 because of the learning rate decays to a small value when converged.
Revised: st [Equation] Then,

Original: At ﬁxed point R 0 = 1 , we have ∂ ( QI 3 − I 1 ) /∂Q
Revised: I [Equation] (13) and I

Original: = 0 , thus the 
Revised: [Equation] Proposition 2. Given conditions in proposition1, let λ bnQ , λ bnR be the eigenvalues of the Jacobian matrix at ﬁxed point ( Q 0 , R 0 , L 0 ) corresponding to the order parameters Q and R respectively in

Original: Jacobian of BN is
Revised: BN. Then

Original: [Equation]
Revised: [Equation]

Revised: where η bn max and η bn eﬀ are the maximum and effective learning rates respectively in BN.

Revised: Proof.

Revised: At ﬁxed point ( Q 0 , R 0 , L 0 ) =

Revised: ( 1 2 ζ +1 , 1 , L 0 ) obtained in proposition1, the Jacobian of dynamic equations of BN can be derived as

Revised: [Equation]

Original: and the eigenvalues of J bn can be obtained by inspection
Revised: and the eigenvalues of J bn can be obtained by inspection

Original: [Equation]
Revised: [Equation]

Original: Since γ 0 = Q 0 , we have η bn max = ( ∂ ( γ 

Revised: Since γ 0 = Q 0 , we have η bn max = ( ∂ ( γ

Original: I 3 − I 1 )γ 0 ∂R
Revised: I 3 − I 1 )

Original: [Equation]
Revised: [Equation]

Original: ∂I
Revised: 0 ∂R

Original: [Equation]
Revised: [Equation]

Original: and ηbn eﬀ =
Revised: and η

Original: [Equation]
Revised: [Equation]

Revised: 0 L 20 .

Original: C. STABLE FIXED POINTS OF BN
Revised: C. STABLE FIXED POINTS OF BN

Original: Proposition 2.
Revised: Proposition 3.

Original: When activation function is ReLU or sigmoid, then (i) λ bnQ < 0 , and (ii) λ bnR < 0 iff η bn max > η bn eﬀ .
Revised: Given conditions in proposition1, when activation function is ReLU, then (i) λ bnQ < 0 , and (ii) λ bnR < 0 iff η bn max > η bn eﬀ .

Original: Proof.
Revised: Proof.

Original: Firstly, when activation function is ReLU, we derive I 1 = [Equation]
Revised: When activation function is ReLU, we derive I 1 = Q ( πR +2 √ 1 − R 2 +2 R arcsin( R ))

Original:  which gives
Revised: 4 π − Q 2 2 , which gives

Original: [Equation]
Revised: [Equation]

Original: Therefore at the ﬁxed point of BN θ bn 0  = ( 1 2 ζ +1 , 1) , we have
Revised: Therefore at the ﬁxed point of BN ( Q 0 , R 0 , L 0 ) = ( 1 2 ζ +1 , 1 , L 0 ) , we have

Original: [Equation]
Revised: [Equation]

Original: Note that x T x approximately equals 1.
Revised: Note that x T x approximately equals 1.

Original: We get
Revised: We get

Original: [Equation]
Revised: [Equation]

Original: At the ﬁxed point we have ∂I 2 ∂R [Equation] − Q 0 < 0 .
Revised: At the ﬁxed point we have ∂I 2 ∂R = − Q 0 < 0 .

Original: Therefore, we conclude that λ bnR < 

Revised: Therefore, we conclude that λ bnR <

Original: iff η bn max > η bn eﬀ .
Revised: iff η bn max > η bn eﬀ .

Original: Secondly, when activation function is sigmoid

Original: [Equation]

Original: we employ erf( x/ √ 2) to analyze sigmoid function.

Original: Therefore, we have

Original: [Equation]

Original: By using the Fourier Transformation of integral on multivariate Gaussian probability density

Original: [Equation]

Original: that is, (cid:90)

Original: [Equation]

Original: n

Original: [Equation]

Original: −

Original: [Equation]

Original: where ˜ f j ( y )

Original: = (cid:82)

Original: [Equation]

Original: ( x ) e iyx .

Original: It sufﬁces to have

Original: [Equation]

Original: and

Original: [Equation]

Original: At the ﬁxed point θ bn 0 = ( 1 2 ζ +1 , 1) , it can be shown that ∂ ( I 1 /Q )

Original: ∂Q < 0 and thus λ bnQ =

Original: η ( ∂ ( I 1 /Q )

Original: ∂Q − ζ ) < 0 , we have ∂I 2 ∂R < 0 .

Original: [-We also conclude that λ bnR < 
-]
Revised: [-We also conclude that λ bnR < 
-]

Original: iff η bn max > η bn eﬀ .

Original: C. M AXIMUM L EARNING R ATE OF BN
Revised: C. M AXIMUM L EARNING R ATE OF BN

Original: Proposition 3.
Revised: Proposition 4.

Original: When the activation function is ReLU, then η bn max ≥ η { wn,sgd } max + 2 ζ , where η bn max and η { wn,sgd } max indicate the maximum learning rates of BN, WN, and vanilla SGD respectively.
Revised: When the activation function is ReLU, then η bn max ≥ η { wn,sgd } max + 2 ζ , where η bn max and η { wn,sgd } max indicate the maximum learning rates of BN, WN, and vanilla SGD respectively.

Original: Proof.
Revised: Proof.

Original: From the above results, we have I 1 = [Equation]{+Q ( πR +2 √ 1 − R 2 +
+}
Revised: From the above results, we have I 1 = {+Q ( πR +2 √ 1 − R 2 +
+}

Original:  − Q 2 2 , which gives ∂I 1 /∂R ≥ [-
-] at the ﬁxed point of BN.
Revised: R arcsin( R )) 4 π − Q 2 2 , which gives ∂I 1 /∂R ≥ [-
-]0 at the ﬁxed point of BN.

Original: And we get [Equation] Then we derive that ∂I 2 ∂R < 0 and ∂ ( I 3 − I 1 ) ∂R / ∂I 2 2 ∂R has the same value at their respective ﬁxed point of BN, WN and vanilla SGD.
Revised: Then it can be derived that ∂I 2 ∂R < 0 .

Original: At the ﬁxed point of BN, Q 0 = γ 

Revised: Furthermore, at the ﬁxed point of BN, Q 0 = γ

Original: = 1 

Revised: = 1

Original: ζ +1 < 1 , then we have
Revised: ζ +1 < 1 , then we have

Original: [Equation]
Revised: [Equation]

Revised: where the inequality sign holds because (1 − 1 γ 0 ) ∂I

Revised: 1 ∂R / ∂I 2 2 ∂R is positive.

Revised: Note that ∂

Revised: ( I 3 − I 1 ) ∂R / ∂I 2 2 ∂R is also deﬁned as maximum learning rates of WN, and vanilla SGD in Yoshida et al.

Revised: Hence, we conclude that η bn max ≥ η { wn,sgd } max + 2 ζ .

Revised: D P ROOFS REGARDING GENERALIZATION AND STATISTICAL MECHANICS (SM)

Revised: In this section, we build an analytical model for the generalization ability of a single-layer network.

Revised: The framework is based on the Teacher-Student model, where the teacher network output y ∗ =

Revised: g ∗ (cid:0) w ∗ T · x + s (cid:1) is learned by a student network.

Revised: The weight parameter of the teacher network satisﬁes 1 N ( w ∗ ) T · w ∗ =1 and the bias term s is a random variable s ∼ N (0 , S ) ﬁxed for each training example x to represent static errors in training data from observations.

Revised: In the generalization analysis, the input is assumed to be drawn from x ∼ N (cid:0) 0 , 1 N I (cid:1) .

Revised: The output of the student can also be written as a similar form y = g

Revised: ( (cid:101) w · x ) , where the activation function g ( · ) can be either linear or ReLU in the analysis and (cid:101) w is a general weight parameter which can be used in either WN or common linear perceptrons.

Revised: Here we take WN for example, since it has been derived in this study that BN can be decomposed into WN with a regularization term on γ.

Original: {+In WN (cid:101) w = γ w (cid:107) w (cid:107) 
+}
Revised: {+In WN (cid:101) w = γ w (cid:107) w (cid:107) 
+}

Revised: and we deﬁned the same order parameter as the previous section that γ 2 = 1 N (cid:101) w T · (cid:101) w and γR = 1 N (cid:101) w T · w ∗ .

Revised: D. G ENERALIZATION ERROR

Revised: Since the learning task is a regression problem with teacher output biased by a Gaussian noise, it comes natural that we can use the the average mean square error loss (cid:15) t

Revised: = 1 P (cid:80) j (cid:0) y ∗ j − y j (cid:1) 2 for the regression.

Revised: The generalization error deﬁned as the estimation over the distribution of input x and is written as

Revised: [Equation]

Revised: (16) where (cid:104)·(cid:105) x denotes an average over the distribution over x .

Revised: The generalization error is a function of its weight parameter and can be converted to a function only with regard to the aformentioned order parameters, detailed derivation can be seen in (B¨os, 1998; Krogh & Hertz, 1992).

Revised: [Equation]

Revised: (17) where h 1 and h 2 are variables drawn from standard Gaussian distribution and Dh 1 :=

Revised: N (0 , 1) dh 1 .

Revised: When both the teacher network and student network have a linear activation function, the above integration can be easily solved and

Revised: [Equation]

Revised: As for the case where the teacher network is linear and the student network has a ReLU activation

Revised: [Equation]

Revised: it can still be solved ﬁrst by decomposing the loss function

Revised: [Equation]

Revised: It should be noted that the last two terms should only be integrated over the half space γRh 1 + γ √ 1 − R 2 h 2 >

Revised: [Equation]

Revised: and therefore if we deﬁne the angle of this line with the h 2 axis θ 0 = arccos ( R ) the integration is transformed to polar coordinate

Revised: [Equation]

Revised: D

Revised: [Equation]

Revised: E QUILIBRIUM ORDER PARAMETERS

Revised: Following studies on statistical mechanics, the learning process of a neural network resembles a

Revised: Langevin process (Mandt et al., 2017) and at the equilibrium the network parameters θ follow a Gibbs distribution.

Revised: That is, the weight vector that yields lower training error produces higher probability.

Revised: We have p ( θ ) =

Revised: Z − 1 exp {− β(cid:15) t ( θ ; x ) } , where β

Revised: = 1 /T and T is temperature, representing the variance of noise during training and implicitly controlling the learning process.

Revised: (cid:15) t ( θ ; x ) is an energy term of the training loss function, Z = (cid:82) d P ( θ ) exp {− (cid:15) t ( θ ; x ) /T } is the partition function, and

Revised: P ( θ ) is a prior distribution.

Revised: Instead of directly minimizing the energy term above, statistical mechanics ﬁnds the minima of free energy, f , which is a function over T , considering the ﬂuctuations of θ at ﬁnite temperatures.

Revised: We have − βf =

Revised: (cid:104)

Revised: ln Z (cid:105) x .

Revised: By substituting the parameters that minimize f back into the generalization errors calculated above, we are able to calculate the averaged generalization error, at a certain temperature.

Revised: The solution of SM requires the differentiation of f with respect to the order parameters.

Revised: In general, the expression of free energy under the replica theory is expressed as(Seung

Revised: et al., 1992)

Revised: [Equation]

Revised: 19) where

Revised: [Equation]

Revised: In the above expression, h 1 , h 2 , h 3 are three independent variables following the standard Gaussian distribution and α = P/N represents the ratio of the number of training samples P to number of unknown parameters N in the network, R = 1 N w (cid:107) w (cid:107) 2 · w ∗ , q is the prior value of γ ..

Revised: The above equation can be utilized for a general SM solution of a network.

Revised: However, the solution is notoriously difﬁcult to solve and only a few linear settings for the student network have close-form solutions(B¨os, 1998).

Revised: Here we extend the previous analysis of linear activations to a non-linear one, though still under the condition that β → ∞ , which means that the student network undergoes a exhaustive learning that minimizes the training error.

Revised: In the current setting, the student network is a nonlinear ReLU network while the teacher is a noise-corrupted linear one.

Revised: Proposition 5.

Revised: Given a single-layer linear teacher y ∗ =

Revised: w ∗ x + s and a student ReLU network y

Revised: =

Revised: g ( γ w (cid:107) w (cid:107) 2 x ) linear student network with g being a ReLU activation function, x ∼ N (0 , I N ) the free energy f satisﬁes as β → ∞

Revised: [Equation]

Revised: where S is the variance of the Gaussian noise s injected to the output of the teacher.

Revised: Proof.

Revised: The most difﬁcult process in Eqn.19 is to solve the inner integration over h 3 .

Revised: Here as β → ∞ , it is noted that the function exp ( − βx ) only notches up only at x = 0 and is 0 elsewhere.

Revised: Therefore, the integration (cid:82)

Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: (cid:17) depends on the value of g ∗ .

Revised: If g ∗ < 0 , no solution exists for g − g ∗ = 0 as g is a ReLU activation, and thus the integration is equivalent to the maximum value of the integral under the limit of β → ∞ .

Revised: As g ∗ > 0 , the integration over the “notch” is equivalent to the one at full range.

Revised: That is,

Revised: [Equation]

Revised: The above equation can be readity integrated out and we obtain

Revised: [Equation]

Revised: Substituting it back to Eqn.

Revised: [Equation]

Revised: we have its third term equivalent

Revised: [Equation]

Revised: 

Revised: To solve the above integration

Revised: [Equation]

Revised: we ﬁrst realize that s is a random variable to corrupt the output of the teacher output and the above integration should be averaged out over s .

Revised: Given that s ∼ N (0 , S ) , it is easy to realize that

Revised: [Equation]

Revised: Through simple Gaussian integraions, we get

Revised: [Equation]

Revised: Substituting Term3 back yields the results of the free energy.

Revised: Therefore, by locating the values that minimizes f in the above proposition, we have equilibrium order parameters

Revised: [Equation]

Revised: and

Revised: [Equation]

Revised: [Equation]

Revised: where a is deﬁned as a = 1+ β ( q 2 − γ 2 )

Revised: β ( q 2 − γ 2 ) .

Revised: Substituting the order parameters back to the generalization error, we have

Revised: [Equation]

Revised: When α < 2 and β → ∞ , a = 1 , the generalization error is

Revised: [Equation]