Original: U NDERSTANDING W HY N EURAL N ETWORKS G ENERALIZE W ELL T HROUGH GSNR OF P ARAMETERS
Revised: U NDERSTANDING W HY N EURAL N ETWORKS G ENERALIZE W ELL T HROUGH GSNR OF P ARAMETERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: As deep neural networks (DNNs) achieve tremendous success across many application domains, researchers tried to explore in many aspects on why they generalize well.
Revised: As deep neural networks (DNNs) achieve tremendous success across many application domains, researchers tried to explore in many aspects on why they generalize well.

Original: In this paper, we provide a novel perspective on these issues usingthe gradient signal to noise ratio (GSNR) of parameters during training processof DNNs.
Revised: In this paper, we provide a novel perspective on these issues usingthe gradient signal to noise ratio (GSNR) of parameters during training processof DNNs.

Original: The GSNR of a parameter issimply deﬁned as the ratio between itsgradient’s squared mean and variance, over the data distribution.
Revised: The GSNR of a parameter is deﬁned as the ratio between its gradient’s squared mean and variance, over the data distribution.

Original: Based on severalapproximations, we establish a quantitative relationship between model parameters’ GSNR and the generalization gap.
Revised: Based on several approximations, we establish a quantitative relationship between model parameters’ GSNR and the generalization gap.

Original: This relationship indicates that larger GSNR during training process leads to better generalization performance.
Revised: This relationship indicates that larger GSNRduring training process leads to better generalization performance.

Original: Futher,we show that, different from that of shallow models (e.g. logistic regression, support vector machines), the gradient descent optimization dynamics of DNNs naturally produces large GSNR during training, which is probably the key to DNNs’remarkable generalization ability.
Revised: Moreover, weshow that, different from that of shallow models (e.g. logistic regression, supportvector machines), the gradient descent optimization dynamics of DNNs naturallyproduces large GSNR during training, which is probably the key to DNNs’ remarkable generalization ability.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks typically contain far more trainable parameters than training samples, whichseems to easily cause a poor generalization performance.
Revised: Deep neural networks typically contain far more trainable parameters than training samples, whichseems to easily cause a poor generalization performance.

Original: However, in fact they usually exhibit remarkably small generalization gaps.
Revised: However, in fact they usually exhibitremarkably small generalization gaps.

Original: Traditional generalization theory such as VC dimension (Vapnik & Chervonenkis, 1991) or Rademacher complexity (Bartlett P L, 2002) cannot explain its mechanism.
Revised: Traditional generalization theories such as VC dimension(Vapnik & Chervonenkis, 1991) or Rademacher complexity (Bartlett P L, 2002) cannot explain itsmechanism.

Original: Extensive research focuses on the generalization ability of DNNs (Neyshabur et al., 2017; S.et al., 2018;
Revised: Extensive research focuses on the generalization ability of DNNs (Neyshabur et al.

Original:  Keskar et al., 2016; Dinh et al., 2017; Hoffer et al., 2017; R et al., 2018;Dziugaite &
Revised: ,2017; S. et al., 2018; Keskar et al., 2016; Dinh et al., 2017; Hoffer et al., 2017; R et al., 2018;

Original:  Roy, 2017; etc, 2018; Kawaguchi et al., 2017; Advani & Saxe, 2017).
Revised: Dziugaite & Roy, 2017; etc, 2018; Kawaguchi et al., 2017; Advani & Saxe, 2017).

Original: Unlike that of shallow models such as logistic regression or support vector machines, the globalminimum of high-dimensional and non-convex DNNs cannot be found analytically, and can onlybe approximated by gradient descent and its variants (Zeiler, 2012; Kingma & Ba, 2014; Graves,2013).
Revised: Unlike that of shallow models such as logistic regression or support vector machines, the globalminimum of high-dimensional and non-convex DNNs cannot be found analytically, and can onlybe approximated by gradient descent and its variants (Zeiler, 2012; Kingma & Ba, 2014; Graves,2013).

Original: Previous work (Zhang et al., 2016; Hardt et al., 2015; Dziugaite & Roy, 2017) suggests thatthe generalization ability of DNNs is closely related to gradient descent optimization.
Revised: Previous work (Zhang et al., 2016; Hardt et al., 2015; Dziugaite & Roy, 2017) suggests thatthe generalization ability of DNNs is closely related to gradient descent optimization.

Original: For example,
Revised: For example,

Original: Hardt et al.
Revised: Hardt et al.

Original: (2015) claims that any model trained with stochastic gradient descent (SGD) for reasonable epochs would exhibit small generalization error.
Revised: (2015) claims that any model trained with stochastic gradient descent (SGD) for reasonable epochs would exhibit small generalization error.

Original: Their analysis is based on the smoothness ofloss function.
Revised: Their analysis is based on the smoothness ofloss function.

Original: In this work, we attempt to understand the generalization behavior of DNNs through
Revised: In this work, we attempt to understand the generalization behavior of DNNs through

Original: GSNR and reveal how GSNR affects the training dynamics of gradient descent.
Revised: GSNR and reveal how GSNR affects the training dynamics of gradient descent.

Original: The GSNR of a parameter is deﬁned as the ratio between its gradient’s squared mean and varianceover the data distribution.
Revised: The GSNR of a parameter is deﬁned as the ratio between its gradient’s squared mean and varianceover the data distribution.

Original: Previous work tried to use GSNR to conduct theoretical analysis ondeep learning.
Revised: Previous work tried to use GSNR to conduct theoretical analysis ondeep learning.

Original: For example, Rainforth et al.
Revised: For example, Rainforth et al.

Original: (2018) used GSNR to analyze variational bounds inunsupervised DNNs such as variational auto-encoder (VAE).
Revised: (2018) used GSNR to analyze variational bounds inunsupervised DNNs such as variational auto-encoder (VAE).

Original: Here we focus on analyzing the relationbetween GSNR and the generalization gap.
Revised: Here we focus on analyzing the relationbetween GSNR and the generalization gap.

Original: Intuitively, GSNR measures the similarity of a parameter’s gradients among different training samples.
Revised: Intuitively, GSNR measures the similarity of a parameter’s gradients among different training samples.

Original: Large GSNR implies that most training samples agree on the optimization direction of thisparameter, thus the parameter is more likely to be associated with a meaningful “pattern” and weassume its update could lead to a better generalization.
Revised: Large GSNR implies that most training samples agree on the optimization direction of thisparameter, thus the parameter is more likely to be associated with a meaningful “pattern” and weassume its update could lead to a better generalization.

Original: In this work, we prove that the GSNR isstrongly related to the generalization performance, and larger GSNR means a better generalization.
Revised: In this work, we prove that the GSNR isstrongly related to the generalization performance, and larger GSNR means a better generalization.

Original: To reveal the mechanism of DNNs’ good generalization ability, we show that the gradient descentoptimization dynamics of DNN naturally leads to large GSNR of model parameters, thus a good generalization.
Revised: To reveal the mechanism of DNNs’ good generalization ability, we show that the gradient descentoptimization dynamics of DNN naturally leads to large GSNR of model parameters and therefore good generalization.

Original: Futher, we give a complete analysis and a detailed interpretation to this phenomenon.
Revised: Furthermore, we give a complete analysis and a detailed interpretation to thisphenomenon.

Original: We believe this is probably the key to DNNs remarkable generalization ability.
Revised: We believe this is probably the key to DNNs remarkable generalization ability.

Original: In the remainder of this paper we ﬁrst analyze the relation between GSNR and generalization (Section 2).
Revised: In the remainder of this paper we ﬁrst analyze the relation between GSNR and generalization (Section 2).

Original: We then show how the training dynamics lead to large GSNR of model parameters experimentally and analytically in Section 3.
Revised: We then show how the training dynamics lead to large GSNR of model parameters experimentally and analytically in Section 3.

Original:  L ARGER GSNR L EADS TO B ETTER G ENERALIZATION
Revised:  L ARGER GSNR L EADS TO B ETTER G ENERALIZATION

Original: In this section, we establish a quantitative relation between the GSNR of model parameters and thegeneralization gap based on several proper approximations. This relationship indicates that larger GSNR during trainingprocess leads to better generalization performance.
Revised: In this section, we establish a quantitative relation between the GSNR of model parameters andgeneralization gap, showing that larger GSNR during training leads to better generalization.

Original: 2. G RADIENTS S IGNAL TO N OISE R ATIO
Revised: 2. G RADIENTS S IGNAL TO N OISE R ATIO

Original: Consider a data distribution Z =
Revised: Consider a data distribution Z =

Original: X
Revised: X

Original: × Y  from which each samples = ( x, y ) is drawn, and a modelˆ y =
Revised: × Y , from which each sample ( x, y ) is drawn; a model ˆ y =f

Original: f ( x, θ )trying to predict y from x , parameterized by θ  . 
Revised: ( x, θ ) parameterized by θ ; and a loss function L . The parameters’ gradient w.r.t. L and sample ( x i , y i ) is denoted by

Original: We use g s ( θ  )to denote parameters’gradient w.r.t sample s . 
Revised: [Equation] whose j -th element is g i ( θ j ) . Note that throughout this paper we always use i to index data examplesand j to index model parameters.

Original: Given the data distribution Z , we can evaluate the (sample-wise) meanand variance of g s ( θ ) .
Revised: Given the data distribution Z , we have the (sample-wise) mean and variance of g i ( θ ) .

Original: We denote them as µ ( θ ) =
Revised: We denotethem as ˜g ( θ ) =

Original: E s ∼Z ( gs (  θ )) and ρ 2 ( θ ) =
Revised: E ( x,y ) ∼Z ( g ( x, y, θ )) and ρ 2 ( θ ) =

Original: Var s ∼Z ( gs (  θ )) ,respectively.
Revised: Var ( x,y ) ∼Z ( g ( x, y, θ )) , respectively.

Original: The gradient signal to noise ratio (GSNR) of q -th model parameter  is deﬁned as:
Revised: The gradient signal to noise ratio (GSNR) of one model parameter θ j is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: We can see, at a particular point of the parameter space, GSNR measures the similarity of the parameter gradients among different data samples. 
Revised: At a particular point of the parameter space, GSNR measures the consistency of a parameter’s gradients across different data samples. [Equation]

Original: 2. O NE -S TEP G ENERALIZATION R ATIO
Revised: G ENERALIZATION R ATIO

Original: In this section we introduce a new concept to helpus measure the generalization performance duringgradient descent optimization, which we call one-step generalization ratio (OSGR).
Revised: In this section we introduce a new concept to help measure the generalization performance during gradient descent optimization, which we call one-step generalization ratio (OSGR).

Original: We denote thetraining setas D = { ( x 0 , y 0 ) , ..., ( x n , y n ) } ∼ Z n consists of n samples drawn from Z , and thetest setas D
Revised: Considertraining set D = { ( x 0 , y 0 ) , ..., ( x n , y n ) } ∼ Z n with n samples drawn from Z , and a test set

Original:  ′=
Revised: D ′

Original:  { ( x ′ 0 , y ′ 0 ) , ..., ( x ′ n ′ , y ′ n ′ ) } ∼ Z n ′ .
Revised: = { ( x ′ 0 , y ′ 0 ) , ..., ( x ′ n ′ , y ′ n ′ ) } ∼ Z n ′ .

Original: In practice we use the loss on D ′ to estimate thegeneralization loss.
Revised: In practice we use the loss on D ′ to measure generalization.

Original: We denote the empirical trainingloss and test loss respectively as
Revised: We denote the empirical training and test loss as:

Original: [Equation]
Revised: [Equation]

Original:  Then the empirical generalization gap is deﬁned bye = L [ D ′ ] − L
Revised: respectively. Then the empirical generalization gap is given by L [ D ′ ] − L

Original: [ D ] .
Revised: [ D ] .

Original: In gradient descent optimization, both the training and test loss would decline step by step.
Revised: In gradient descent optimization, both the training and test loss would decrease step by step.

Original: Weuse ∆ L
Revised: Weuse ∆ L

Original: [ D ] and ∆ L [ D ′ ] to denote the one-step training and test loss decrease during training, respectively.
Revised: [ D ] and ∆ L [ D ′ ] to denote the one-step training and test loss decrease during training, respectively.

Original: Let’s consider the ratio between the expectations of ∆ L [ D ′ ] and ∆ L [ D ] of one singletraining step, which we denote as R ( Z  ) .
Revised: Let’s consider the ratio between the expectations of ∆ L [ D ′ ] and ∆ L [ D ] of one singletraining step, which we denote as R ( Z , n ) .

Original: [Equation]
Revised: [Equation]

Revised: Note that this ratio also depends on current model parameters θ and learning rate λ .

Revised: We are notincluding them in the above notation as we will not explicitly model these dependencies, but rathertry to quantitatively characterize R for very small λ and for θ at the early stage of training (satisfying

Original: Note that the expectation of ∆ L [ D ′ ] is over D and D ′ , and it’s because the optimization step, akatraining process, is on D .
Revised: Assumption 2.3.1). Also note that the expectation of ∆ L [ D ′ ] is over D and D ′ . This is because the optimization stepis performed on D .

Original: We refer  R ( Z  ) as OSGR of gradient descent optimization.
Revised: We refer to R ( Z , n ) as OSGR of gradient descent optimization.

Original: Statisticallythe training loss decreases faster than the test loss and OSGR would be less than 1, which usuallyresults in a non-zero generalization gap at the end of training.
Revised: Statistically the training loss decreases faster than the test loss and OSGR would be less than 1, which usuallyresults in a non-zero generalization gap at the end of training.

Original: If OSGR is large in the whole trainingprocess, generalization gap will be small when training completes, implying good generalizationability of the model.
Revised: If OSGR is large in the whole trainingprocess, generalization gap would be small when training completes, implying good generalizationability of the model.

Original: 2. R ELATION BETWEEN GSNR AND OSGR
Revised: 2. R ELATION BETWEEN GSNR AND OSGR

Original: In this section, we focus on deriving the relation between GSNR and OSGR, futher to prove the

Original: GSNR is strongly related to model generalization performance.

Original: For simplicity, we assume the sizes of training and test datasets are equal, n = n ′ . In gradientdescent optimization, we compute the mean of gradient overthe training dataset, which we denoteas k ( θ, D  ) 
Revised: In gradient descent optimization, we take the average gradient over training set D , which we denote as g D ( θ ) , and we denote its expectation as ˜ g D ( θ ) . Note that we have used g i ( θ ) to denotegradient evaluated on one data sample and ˜ g

Original: [-= 1 n ∑ ni =
 g i-] ( θ ) , and use it as the opposite descent direction. Note the differencebetween µ ( θ ) and k ( θ, D ) , where µ ( θ ) is the gradient mean over a data distribution and k ( θ, D ) isthe empirical gradient mean of a training dataset.
Revised: [-= 1 n ∑ ni =
 g i-] ( θ ) to denote its expectation over the entire data distribution. Similarly we deﬁne g D ′

Original: The gradient mean of the training and test datasets are, respectively
Revised: ( θ ) to be the average gradient over test set D ′ , and ˜ g D ′ ( θ ) to be itsexpectation.

Original: [Equation]
Revised: [Equation]

Original: where g ′ i ( θ ) is the gradient w.r.t the i -th sample of the test dataset.

Original: Both the training and test dataset are randomly generated from the same distribution Z n , so wecan treat k ( θ, D  ) and k ′ ( θ, D ′  ) as random variables.
Revised: Both the training and test dataset are randomly generated from the same distribution Z n , so wecan treat g D ( θ ) and g D ′ ( θ ) as random variables.

Original: At the beginning of the optimization process,θ is randomly initialized thus independent of D , so k ( θ, D  ) and k ′ ( θ, D ′  ) would obey the samedistribution.
Revised: At the beginning of the optimization process, θis randomly initialized thus independent of D , so g D ( θ ) and g D ′ ( θ ) would obey the same distribution.

Original: After a period of training, the model parameters begin to ﬁt the training dataset andbecome a function of D ,  θ = θ ( D ) , then the distribution of k ( θ ( D ) , D ) and k ′ ( θ ( D ) , D ′ ) becomesdifferent.
Revised: After a period of training, the model parameters begin to ﬁt the training dataset and becomea function of D , i.e. θ = θ ( D ) , therefore distributions of g D ( θ ( D )) and g D ′ ( θ ( D )) become different.

Original: However we will makeour derivation under the non-overﬁtting limit approximation 2.3.1stated as below.
Revised: However we choose not to model this dependency and make the following assumption for ouranalysis:

Original: Assumption 2.3.1 (Non-overﬁtting limit approximation)
Revised: Assumption 2.3.1 (Non-overﬁtting limit approximation)

Original: In the early training stage, the mean ofgradients w.r.t the training dataset and test dataset k ( θ( D ) , D ) and k ′ ( θ( D ) , D ′ ) obey the samedistribution, and we denote their mean and variance as µ ( θ ) and σ 2 ( θ ) , i.e.
Revised: The average gradient over the trainingdataset and test dataset g D ( θ ) and g D ′ ( θ ) obey the same distribution. Obviously the mean of g D ( θ ) and g D ′ ( θ ) is just the mean gradient over the data distribution ˜ g ( θ ) .

Original: [Equation]
Revised: [Equation]

Original: For simplicity, we denote k ( θ ( D ) , D ) and k ( θ ( D ) , D ′ ) as k ( θ ) and k ′ ( θ ) respectively.
Revised: We denote their variance as σ 2 ( θ ) , i.e. [Equation]

Original: We can getthe following relation:
Revised: It is straightforward to show that:

Original: [Equation]
Revised: [Equation]

Original: where σ 2 ( θ ) is the variance of the  gradient mean of a training dataset while ρ 2 ( θ ) is  thevariance ofthe gradient of a single  sample.
Revised: where σ 2 ( θ ) is the variance of the average gradient over the dataset of size n , and ρ 2 ( θ ) is thevariance of the gradient of a single data sample.

Original: In one gradient descent step, the model parameter is updated by ∆ θ = θ t +1 − θ t = − λ k ( θ ) whereλ is the learning rate.
Revised: In one gradient descent step, the model parameter is updated by ∆ θ = θ t +1 − θ t = − λ g D ( θ )where λ is the learning rate.

Original: If λ is small enough, the one-step training and test loss decrease can beapproximated as
Revised: If λ is small enough, the one-step training and test loss decrease can beapproximated by

Original: [Equation]
Revised: [Equation]

Original: [Equation] Usually there are some differences between the directions of k ( θ ) and k ′ ( θ ) ,so statistically ∆ L
Revised: Usually there are some differences between the directions of g D ( θ ) and g D ′ ( θ ) ,

Original:  [ D ]tends to be larger than ∆ L [ D ′ ] and the generalization gap would increase during training.
Revised: so statistically∆ L [ D ] tends to be larger than ∆ L [ D ′ ] and the generalization gap would increase during training.

Original: Whenλ → 0 , in one single training step the empirical generalization gap increases by:
Revised: When λ → 0 , in one single training step the empirical generalization gap increases by ∆ L [ D ] −∆ L [ D ′ ] , for simplicity we denote this quantity as ▽ :

Original: [Equation]
Revised: [Equation]

Original: We replace the random variables, k ( θ ) = µ ( θ ) + ϵ , k ′ ( θ ) =µ ( θ ) + ϵ ′ .
Revised: Here we replaced the random variables by g D ( θ ) = ˜ g ( θ ) + ϵ and g D ′ ( θ ) =

Revised: ˜ g

Original:  ϵ and ϵ  ′ are randomvariables with zero mean and variance σ 2 ( θ ) .
Revised: ( θ ) + ϵ ′ , where ϵ andϵ ′ are random variables with zero mean and variance σ 2 ( θ ) .

Original: Since E ( ϵ ′ ) =
Revised: Since E ( ϵ ′ ) =

Original: E ( ϵ ) = 0 , ϵ and ϵ ′ are independent,the expectation of ∆ e is
Revised: E ( ϵ ) = 0 , ϵ and ϵ ′ areindependent, the expectation of ▽ is

Original: [Equation]
Revised: [Equation]

Original: whereQ is the number of parameters and σ 2 q is the variance  of mean gradient of the q -th parameter.
Revised: where σ 2 ( θ j ) is the variance the of average gradient of the parameter θ j . For simplicity, when it involves a single model parameter θ j , we will use only a subscript j insteadof the full notation. For example, we use σ 2 j , r j , and g D,j to denote σ 2 ( θ j ) , r ( θ j ) , and g D ( θ j )respectively.

Original: Consider the expectation of ∆ L [ D ] and ∆ L [ D ′ ] when λ → 

Revised: Consider the expectation of ∆ L [ D ] and ∆ L [ D ′ ] when λ →

Original: [Equation]
Revised: [Equation]

Original: Now we are ready to derive the relation between GSNR and OSGR, by substituting 16 and 14 into3:
Revised: [Equation] Substituting (18) and (15) into (4) we have:

Original: [Equation]
Revised: [Equation]

Original: Now the right hand side of equation (17) can be estimated using only the samples in the trainingdatasets. Because the gradient average k q ( D ) and sample-wise gradient variance ρ 2 q can both becomputed within one training dataset.
Revised: Although we derived eq. (19) from simpliﬁed assumptions, we can empirically verify it by estimatingtwo sides of the equation on real data.

Original: We will elaborate on this estimation method in section 2.4.
Revised: We will elaborate on this estimation method in section 2.4.

Original: Reformulate equation (17) as:
Revised: We can rewrite eq. (19) as:

Original: [Equation]
Revised: [Equation]

Original: Q ∑ q =

Original:  E D ∼Z n ( k 2 q ( D )) ∑
Revised: ) where E D ∼Z n ( g 2 D,j )

Original: Qq ′

Original: =1 E D ∼Z n ( k 2 q ′ ( D )) ρ 2 q E  D ∼Z n ( k 2 q ( D )) (18) = 1− 1 n 
Revised: = V ar D ∼Z n ( g D,j ) + E 2 D ∼Z n ( g D,j ) = 1 n ρ 2 j + ˜ g 2 j .

Original: Q ∑ q =

Original: E D ∼Z n ( k 2 q ( D ))

Original: ∑

Original: Qq ′

Original: =1 E D ∼Z n ( k 2 q ′ ( D ))

Original: r q + 1 n (19) where E D ∼Z n ( k 2 q ( D )) =

Original: V ar D ∼Z n ( k q ( D ))

Original: + E 2 D ∼Z n ( k q ( D )) = 1 n ρ 2 q + µ 2 q . We deﬁne ∆ L q [ D ] to be the training loss decrease caused by updating the q -th parameter.
Revised: We deﬁne ∆ L j [ D ] to be the training loss decrease caused by updating θ j .

Original: We canshow that when λ is very small ∆ L q [ D ] = λk
Revised: We can show that whenλ is very small ∆ L j [ D ] = λ

Original:  2 q ( D ) + O ( λ 2 ) .
Revised: g 2 D,j + O ( λ 2 ) .

Original: Therefore when λ → [-
-]
Revised: Therefore when λ → [-
-]0 , we have

Original: R ( Z ) = 1 − 1 n Q ∑ q =
Revised: [Equation] [Equation] ∑ j

Original: W q r q + 1 n , where W q = E D ∼Z n (∆ L q [ D ])
Revised: [Equation] r j + 1 n , where W j := E D ∼Z n (∆ L j [ D ])

Original: E D ∼Z n (∆ L [ D ]) withQ ∑ q =
Revised: E D ∼Z n (∆ L [ D ]) with ∑ j

Original: W q =(20)
Revised: W j =(22)

Original: Equation (20) shows that the GSNR (equation 1) plays a crucial role in the model’s generalizationability. In the non-overﬁtting limit approximation and with a small enough learning rate, the one-stepgeneralization ratio in gradient descent equals 1 minus the weighted average of 1 r q + 1 n over all modelparameters divided by n.
Revised: Eq. (22) shows that the GSNR r j plays a crucial role in the model’s generalization ability—the onestep generalization ratio in gradient descent equals one minus the weighted average of 1 r j + 1 n overall model parameters divided by n .

Original: The weight is proportional to the expectation of the training loss decreasecaused by updating that parameter.
Revised: The weight is proportional to the expectation of the training lossdecrease resulted from updating that parameter.

Original: Thisresult implies that larger GSNR of model parameters duringtraining leads to smaller generalization gap growth and better generalization performance of the ﬁnalmodel.
Revised: This implies that larger GSNR of model parametersduring training leads to smaller generalization gap growth and better generalization performance ofthe trained model.

Original: Also note when n → ∞ , we have R  → 1 and hence good generalization performance.
Revised: Also note when n → ∞ , we have R ( Z , n ) → 1 , meaning that training on moredata helps generalization.

Original: 2. E XPERIMENTAL V ERIFICATION OF THE RELATION BETWEEN GSNR AND OSGR
Revised: 2. E XPERIMENTAL VERIFICATION OF THE RELATION BETWEEN GSNR AND OSGR

Original: The relation between GSNR and OSGR, i.e. equation 17 or 20, is solid and can be accurately veriﬁedusing any datasets if: (1) The dataset include enoughnumber of samples to construct enough number of training datasets and a large enough test dataset so that the estimation of ρ 2 q , E D ∼Z n ( k 2 q ( D )) and OSGR can be accurate.
Revised: The relation between GSNR and OSGR, i.e. eq. (19) or (22) can be veriﬁed empirically using anydatasets if: (1) The dataset includes enough samples to construct many training sets and a largeenough test set so that we can reliably estimate ρ 2 j , E D ∼Z n ( g 2 D,j ) and OSGR.

Original: The learning rate is small enough.
Revised: The learning rateis small enough.

Original: In the early training stage ofgradient descent.
Revised: In the early training stage of gradient descent.

Original: To veﬁty equation 17, here we show how to estimate the left and right hand sides of it.
Revised: To empirically verify eq. (19), we show how to estimate its left and right hand sides, i.e. OSGR bydeﬁnition and OSGR as a function of GSNR.

Original: Suppose wehave Mnumber of training datasets with  n samples each and alarge test dataset with n ′ samples.
Revised: Suppose we have M training sets each with size n ,and a test set of size n ′ .

Original: We initialize a model and train it separately on the M training datasets and test it with the same testdataset.
Revised: We initialize a model and train it separately on the M training sets and testit with the same test set.

Original: For the t -th training iteration, we denote the training loss and test loss of the model trainedon the m -th training dataset as L tm and L ′ tm , respectively.
Revised: For the t -th training iteration, we denote the training loss and test loss ofthe model trained on the m -th training dataset as L ( m ) t and L ′ ( m ) t , respectively.

Original: Then the left hand side, i.e. OSGR, of the t -th iteration can be estimated as
Revised: Then the left handside, i.e. OSGR by deﬁnition, of the t -th iteration can be estimated by

Original: [Equation]
Revised: [Equation]

Original: For the model trained on the m -th training datase, we can compute the t -th step  gradientmean andsample-wise gradient variance of q -th parameter on the corresponding training dataset, denoted asˆ k qmt andˆ ρ 2 qmt , then the right hand side of equation 17 can be estimated 
Revised: For the model trained on the m -th training set, we can compute the t -th step avreage gradient andsample-wise gradient variance of θ j on the corresponding training set, denoted as g m,j,t and ρ 2 m,j,t ,respectively. Therefore the right hand side of eq. (19) can be estimated by

Original: [Equation]
Revised: [Equation]

Original: Note that using equation 22, the right hand side can be computed merely with the samples in thetraining datasets.

Original: To verify equation 17, we carry out the experiment on MNISTtraining dataset with  simple {+CNN structure consists of 
+}
Revised: We performed the above estimations on MNIST with a simple {+CNN structure consists of 
+}

Original: CNNs which consists of 2 Conv-Relu-MaxPooling blocks and 2 fully-connnected layers.
Revised: Conv-Relu-MaxPooling blocks and 2 fully-connected layers.

Original: First,to estimate equation 22 with M = 10 , we randomly sample 10 training datasets with  nsamples each and a test set with 10000 samples.
Revised: First, to estimate eq. (24) with M = 10 , we randomly sample 10 training sets with size n and a test set with size 10,000.

Original: Then to cover different conditions, we (1) setn ∈ { 1000 , 2000 , 4000 , 6000 , 8000 , 10000 , 15000 } , respectively; (2) inject noise in the datasets byrandomly changing the labels with proportion p random ∈  { 0 . 0 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 5 } ; (3) experimenton various model structures which are different only at the  number of channels in layers.
Revised: To cover different conditions, we (1) choose n ∈ { 1000 , 2000 , 4000 , 6000 , 8000 , 10000 , 15000 } ,respectively; (2) inject noise by randomly changing the labels with probability p random ∈ 151 { 0 . 0 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 5 } ; (3) change the model structure by varying number of channels in the layers, ch ∈ { 6 , 8 , 10 , 12 , 14 , 16 , 18 , 20 } .

Original: For modeldetails, please see Appendix A. Moreover, we use the gradient descent training (Not SGD), with asmall learning rate of 0 .
Revised: See Appendix A for more details of the setup. We use thegradient descent training (not SGD), with a small learning rate of 0 .

Original: The left and right hand sides of 17 at different epochs are shown in Figure 1, where each point represents one speciﬁc combination of above settings.
Revised: The left and right hand sidesof 19 at different epochs are shown in Figure 1, where each point represents one speciﬁc choice ofthe above settings.

Original: At the beginning of training, the data points  closely distributed along the dashed line of y =
Revised: At the beginning of training, the data points are closely distributed along the dashed line corresponding to LHS=

Original: x whichshows that equation 17 ﬁts quite well under a variety of different settings.
Revised: RHS. This shows that eq. (19) ﬁts quite well under a variety of different settings.

Original: As training proceeds,equation 17 gradually losses its accuracy because the non-overﬁtting limit approximation no longerholds, butstrong positive correlation between the left and right hand sides of equation 17 remainseven when the training converges (at epoch 2500).
Revised: Astraining proceeds, the points become more scattered as the non-overﬁtting limit approximation nolonger holds, but correlation between the LHS and RHS remains high even when the training converges (at epoch 2,500). We also conducted the same experiment on CIFAR10 A.2 and a toy dataset

Original: Through analytically derivations and experimental veriﬁcations, we prove that GSNR is stronglyrelatetd to OSGR which indicates the generalization ability, thus demonstrate that the larger GSNRduring training leads to better generalization performance.
Revised: A.3 and observed the same behavior. See Appendix for these experiments. The empirical evidence together with our previous derivation of eq. (19) clearly show the relationbetween GSNR and OSGR and its implication in the model’s generalization ability.

Original:  T RAINING DYNAMICS OF DNN S NATURALLY LEADS TO LARGE GSNR
Revised:  T RAINING DYNAMICS OF DNN S NATURALLY LEADS TO LARGE GSNR

Original: In this section, we analyze and explain one observed phenomenon: the parameters’ GSNR of DNNmodels rises in the early stages of training, while the GSNR of shallow models such as logisticregression or support vector machines declines during the entire training process.
Revised: In this section, we analyze and explain one interesting phenomenon: the parameters’ GSNR of DNNs rises in the early stages of training, whereas the GSNR of shallow models such as logisticregression or support vector machines declines during the entire training process.

Original: This difference ofbehaviors provides GSNR large practical values duringDNN training, which in turn is associatedwith good generalization.
Revised: This differencegives rise to GSNR’s large practical values during training, which in turn is associated with goodgeneralization.

Original: We analyze the dynamics behind this phenomenon both experimentallyand theoretically, which deepens our understanding of the good generalization ability of DNNs.
Revised: We analyze the dynamics behind this phenomenon both experimentally and theoretically.

Original: 3. GSNR BEHAVIOR OF DNN S TRAINING
Revised: 3. GSNR BEHAVIOR OF DNN S TRAINING

Original: For shallow models, the GSNR of parameters decreases in the whole training process because gradients become small as learning proceeds when the optimizer ﬁnds the local minimum.
Revised: For shallow models, the GSNR of parameters decreases in the whole training process because gradients become small as learning converges.

Original: But for DNN it is not the case.
Revised: But for DNNs it is not the case.

Original: We trained DNNs on the CIFAR datasets and computed the GSNR averaged over all model parameters.
Revised: We trained DNNson the CIFAR datasets and computed the GSNR averaged over all model parameters.

Original: Because E D ∼Z n ( k 2 q ( D )) =
Revised: Because E D ∼Z n ( g 2 D,j ) =

Original: 1 n ρ 2 q + µ 2 q
Revised: 1 n ρ 2 j + ˜ g 2 j

Original: and we assume n is large, E D ∼Z n ( k 2 q ( D ))
Revised: and we assume n is large, E D ∼Z n ( g 2 D,j )

Original: ≈ µ 2 q .
Revised: ≈ ˜ g 2 j .

Original: In the case of only one large training datasets, we estimate GSNR of t -thiteration by
Revised: In the case of only onelarge training datasets, we estimate GSNR of t -th iteration by

Original: [Equation]
Revised: [Equation]

Original: As shown in Figure 2, the GSNR starts out low with randomly initialized parameters.
Revised: As shown in Figure 2, the GSNR starts out low with randomly initialized parameters.

Original: As learningprogresses, the GSNR increases in the early training stage and stays at a high level in the wholelearning process.
Revised: As learningprogresses, the GSNR increases in the early training stage and stays at a high level in the wholelearning process.

Original: We also computed the proportion of the samples that have the same gradientsign (positive or negative) for each parameter, denoted as p same _ sign .
Revised: For each model parameter, we also computed the proportion of the samples withthe same gradient sign, denoted as p same _ sign .

Original: In Figure 2c, we plot themean timeseries of this proportion for all the parameters .
Revised: In Figure 2c, we plot the mean of time series of thisproportion for all the parameters .

Original: This value increases from about 50%(halfpositive half negetive due to random initialization)at beginning to about 56% ﬁnally, which indicatesthat for most parameters, the gradient signs on different samples become to have a certain degreeof consistency as the training proceeds.
Revised: This value increases from about 50% (half positive half negetivedue to random initialization) to about 56% ﬁnally, which indicates that for most parameters, thegradient signs on different samples become more consistent.

Original: We hypothesis this is because meaningful features begin toemerge in the learning process and the gradients of the weights on these features tend to have thesame sign among different samples.
Revised: This is because meaningful featuresbegin to emerge in the learning process and the gradients of the weights on these features tend tohave the same sign among different samples.

Original: Previous research (Zhang et al., 2016) showed that DNNs achieved zero training loss by memorizingtraining samples even if the labels were randomized.
Revised: Previous research (Zhang et al., 2016) showed that DNNs achieved zero training loss by memorizingtraining samples even if the labels were randomized.

Original: We also plot the average GSNR for modeltrained using data with randomized labels in Figure 2 and ﬁnd that the GSNR stays at a low levelthroughout the training process.
Revised: We also plot the average GSNR for modeltrained using data with randomized labels in Figure 2 and ﬁnd that the GSNR stays at a low levelthroughout the training process.

Original: Although the training loss of both the original and randomizedlabels go to zero (not shown), the GSNR curves clearly distinguish between these two cases andreveal the lack of meaningful patterns in the latter one.
Revised: Although the training loss of both the original and randomizedlabels go to zero (not shown), the GSNR curves clearly distinguish between these two cases andreveal the lack of meaningful patterns in the latter one.

Original: We believe this is the reason why DNNstrained on real and random data lead to completely different generalization behavior.
Revised: We believe this is the reason why DNNstrained on real and random data lead to completely different generalization behaviors.

Original: T RAINING D YNAMICS BEHIND THE GSNR BEHAVIOR
Revised: T RAINING D YNAMICS BEHIND THE GSNR BEHAVIOR

Original: In this section we show that the feature learning ability of DNNs is the key reason why the GSNRcurve behavior of DNNs is different from that of shallow models during the gradient descent trainingprocess.
Revised: In this section we show that the feature learning ability of DNNs is the key reason why the GSNRcurve behavior of DNNs is different from that of shallow models during the gradient descent training.

Original: To demonstrate this clearly, a simple two-layer perceptron regression model is construceted.
Revised: To demonstrate this, a simple two-layer perceptron regression model is constructed.

Original: A synthetic dataset is generated as following.
Revised: A syntheticdataset is generated as following.

Original: Each data point is constructed i.i.d.
Revised: Each data point is constructed i.i.d.

Original: using y = x 0 x 1 + ϵ ,where x 0 and x 1 are drawn from uniform distribution [ − 1 , 1] and ϵ is drawn from uniform distribution [ − 0 .
Revised: using y = x 0 x 1 + ϵ , wherex 0 and x 1 are drawn from uniform distribution [ − 1 , 1] and ϵ is drawn from uniform distribution[ − 0 .

Original: The training set and test set sizes are 200 and 10000, respectively.
Revised: The training set and test set sizes are 200 and 10,000, respectively.

Original: We use a verysimple 2-layer MLP structure with 2 input, 20 hidden neurons and 1 output.
Revised: We use a verysimple two-layer MLP structure with 2 inputs, 20 hidden neurons and 1 output.

Original: We randomly initialized the model parameters and trained the model on the synthetic training dataset.
Revised: We randomly initialized the model parameters and trained the model on the synthetic training dataset.

Original: As a control setup we also tried to freeze model weights in the ﬁrst layer to prevent it from learningfeatures.
Revised: As a control setup we also tried to freeze model weights in the ﬁrst layer to prevent it from learningfeatures.

Original: Note that a two layer MLP with the ﬁrst layer frozen is equivalent to a linear regressionmodel.
Revised: Note that a two layer MLP with the ﬁrst layer frozen is equivalent to a linear regressionmodel.

Original: That is, regression weights are learned on the second layer using ﬁxed features extracted bythe ﬁrst layer.
Revised: That is, regression weights are learned on the second layer using ﬁxed features extracted bythe ﬁrst layer.

Original: We plot the average GSNR of the second layer parameters for both the frozen andnon-frozen cases.
Revised: We plot the average GSNR of the second layer parameters for both the frozen andnon-frozen cases.

Original: Figure 3 shows that in the non-frozen case, the average GSNR over parametersof the second layer has a signiﬁcant upward process, whereas in the frozen case the average GSNRdecreases in the beginning and remains at a low level during  whole training process.
Revised: Figure 3 shows that in the non-frozen case, the average GSNR over parameters ofthe second layer shows a signiﬁcant upward process, whereas in the frozen case the average GSNRdecreases in the beginning and remains at a low level during the whole training process.

Original:  GSNR curve of individual parameters of the second layerof the non-frozen case are shown in Figure3.
Revised: In the non-frozen case, GSNR curve of individual parameters of the second layer are shown in Figure3.

Original: It can be observed that the GSNR for some parameters have a signiﬁcant upward process.
Revised: The GSNR for some parameters show a signiﬁcant upward process.

Original: Wecomputed the Pearson correlation between the features extracted from the ﬁrst layer andy corresponding to these parameters at the  beginning ofthe trainingprocess and  the maximum point of the GSNR curve.
Revised: To measure the quality ofthese features, we computed the Pearson correlation between them and the target output y , both atthe beginning of training and at the maximum point of their GSNR curves.

Original: We can see that the learning process changes these features from a random initializedone to a "good" feature with stronger correlation with y  as shown in Table 1.
Revised: We can see that thelearning process learns “good” features (high correlation value, i.e. with stronger correlation with y )from random initialized ones, as shown in Table 1.

Original: This shows thatin thetraining of DNN, the feature learning process is closely related to the GSNR increasing process. A NALYTICAL
Revised: This shows that the GSNR increasing process isrelated to feature learning.

Original: ANALYSIS OF TRAINING DYNAMICS BEHIND DNN S ’ GSNR BEHAVIOR
Revised: 3. A NALYSIS OF TRAINING DYNAMICS BEHIND DNN S ’ GSNR BEHAVIOR

Original: In this section, we willanalytically investigate the training dynamics behind the GSNR curve behavior of DNNs.
Revised: In this section, we will investigate the training dynamics behind the GSNR curve behavior.

Original: In the case of fully connected network structure, we can analytically show that thenumerator of GSNR, the squared gradient mean of model parameters, tends to increase in the earlytraining stage through feature learning.
Revised: In thecase of fully connected network structure, we can analytically show that the numerator of GSNR,i.e. the squared gradient mean of model parameters, tends to increase in the early training stagethrough feature learning.

Original: In a fully connected network, whose parameters aredenoted as θ ≡ 224 { ω 1 , b 1 , ω 2 , b 2 , ..., ω l max  , b  l max  } , ω 1 , b 1 is the weight  and bias of the ﬁrst layer, and so on.
Revised: Consider a fully connected network, whose parameters are θ = { W (1) , b (1) , ..., W ( l max ) , b ( l max ) } ,where W (1) , b (1) are the weight matrix and bias of the ﬁrst layer, and so on.

Original: We denote the activations of the l -th layer as[ al ] c 1 ( ϕ ), c 1 = { 1 , 2 , ..., C 1 } , where C 1 is the numberof activations of this layer.
Revised: We denote the activations of the l -th layer as a ( l ) = { a ( l ) s ( θ ( l − ) )

Original: ϕ is the collection of model parameters over all the layers before thislayer, i.e. ϕ = { ω 1 , b 1 , ω 2 , b 2 , ..., ω l − 1 , b  l − 1 } .
Revised: } , where s is the index for nodes/channels ofthis layer, and θ ( l − ) is the collection of model parameters in the layers before l , i.e. θ ( l − ) ={ W (1) , b (1) , ..., W ( l − 1) , b ( l − 1) } .

Original: In the forward pass of the i -th sample, a l ( ϕ ) willbe multiplied by the weightof this layer ω l , which can be expressed as matrix multiplication.
Revised: In the forward pass on data sample i , { a ls ( θ ( l − ) ) } is multipliedby the weight matrix W ( l ) : [Equation]

Original: [Equation]
Revised: [Equation]

Original: (24) where o  l  is the output of the matrix multiplication of the l -th layer and c2 = { 1 , 2 , ..., C 2 } where C 2is the number of activations of  the next layer.
Revised: where o ( l ) = { o ( l ) i,c } is the output of the matrix multiplication, for the i -th data sample, on the l -thlayer, c = 1, 2, ..., C is index of nodes/channels in the ( l + 1) -th layer.

Original: We use k l  to denote the gradientmean of weights ofthe l -th layer ω l  , i.e. k l  = 1 n ∑ ni =1 L i ∂ω l  , where L i is the loss  of the i -th sample.
Revised: We use g ( l ) D to denote theaverage gradient of weights of the l -th layer W ( l ) , i.e. g ( l ) D = 1 n ∑ n i = L i ∂ W ( l ) , where L i is the loss 230 of the i -th sample.

Original: Here weanalytically show that the feature learning ability of DNNs plays a crucial role in the GSNRincreasing process.
Revised: Here we show that the feature learning ability of DNNs plays a crucial role in the GSNR increasingprocess.

Original: To be more precise, we show that the learning of features a  l  ( ϕ ) , i.e. the learningof parameters ϕ , tends to increase the absolute value of k l  .
Revised: More precisely, we show that the learning of features a ( l ) ( θ ( l − ) ) , i.e. the learning of parameters θ ( l − ) tends to increase the absolute value of g ( l ) D .

Original: Let’s consider, the one-step change ofgradient mean ∆[ k l ] = ([ k l ]) t +1 − ([ k l ]) t with the learning rate λ → 0 .
Revised: Consider the one-step change of gradientmean ∆ g ( l ) D = g ( l ) D,t +1 − g ( l ) D,t with the learning rate λ → 0 .

Original: In one training step, θ willbe updated by ∆ θ = θ t +1 − θ t = − λ k ( θ ) .
Revised: In one training step, θ is updated by ∆ θ = θ t +1 − θ t = − λ g D ( θ ) .

Original: Use linear approximation with λ → 0 , we have
Revised: Using linear approximation with λ → 0 , we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where Q is the total number of model parameters and Q l is the number of model parametersof allthe layers before  the l -th layer.
Revised: where θ ( l − ) and θ ( l +) denote model parameters before and after the l -the layer (including the l -th),respectively.

Original: Wewill focus on the ﬁrst term of equation 25, i.e. the one-step change of k l  caused by learning ϕ . 
Revised: We focus on the ﬁrst term of eq. (27), i.e. the one-step change of g ( l ) D caused by learning θ ( l − ) . Substituting g ( l ) D = 1 n ∑ [Equation]

Original: Substituting k l =1 n ∑ ni =1 L i ∂ω l  and ∆ ϕ q = ( − λ 1 n ∑ ni =1 ∂L i ∂ϕ q ) into equation 25, we have
Revised: = L i ∂ W ( l ) and ∆ θ j = ( − λ 1 n ∑ ni =1 ∂L i ∂θ j ) into eq. (27), we have

Original: [Equation]
Revised: [Equation]

Original: The detailed derivation of equation 26 can be found in Appendix B. We can see the ﬁrst term (whichis a summation of Q l terms) in equation 26 has opposite sign with [ ω l ] c 1 c 2 .
Revised: The detailed derivation of eq. (28) can be found in Appendix B. We can see the ﬁrst term (which is asummation over parameters in θ ( l − ) ) in eq. (28) has opposite sign with W ( l ) s,c .

Original: This term will make∆[ k l ] c 1 c 2 negatively correlated with [ ω l ] c 1 c 2 .
Revised: This term will make∆ g ( l ) D,s,c negatively correlated with W ( l ) s,c .

Original: We plot the correlation between ∆[ k l ] c 1 c 2 with [ ω l ] c 1 c 2 244 for a model trained on MNIST for 200 epochsas shown in Figure 4a.
Revised: We plot the correlation between ∆ g ( l ) D,s,c with W ( l ) s,c for amodel trained on MNIST for 200 epochs in Figure 4a.

Original: In the early training stage,they are indeed negatively correlated.
Revised: In the early training stage, they are indeednegatively correlated.

Original: For top-10% weights with larger absolute values, the negativecorrelation is more signiﬁcant.
Revised: For top-10% weights with larger absolute values, the negative correlation iseven more signiﬁcant.

Original: Here we show that this negative correlation between ∆[ k l ] c 1 c 2 and [ ω l ] c 1 c 2 will tend to increasethe absolute value of [ k l ] through an interesting mechanism.
Revised: Here we show that this negative correlation between ∆ g ( l ) D,s,c and W ( l ) s,c tends to increase the absolute value of g ( l ) D through an interesting mechanism.

Original: Consider the weights [ ω l ] c 1 c 2 with249 { [ ω l ] c 1 c 2 > 0 , [ k l ] c 1 c 2 < 0 } .
Revised: Consider the weights W ( l ) s,c with { W ( l ) s,c >0 , g ( l ) D,s,c < 0 } .

Original: Learning ϕ wouldtend to decrease [ k l ] c 1 c 2 and thus increase itsabsolute value because the ﬁrst term in equation 26 is negative in this case.
Revised: Learning θ l − would decrease g ( l ) D,s,c and thus increase its absolute value becausethe ﬁrst term in eq. (28) is negative.

Original: Learning [ ω l ] c 1 c 2 wouldincrease [ ω l ] c 1 c 2 and its absolute value because ∆[ ω l ] c 1 c 2 = − λ [ k l ] c 1 c 2 is positive in this case.
Revised: On the other hand, learning W ( l ) s,c would increase W ( l ) s,c and itsabsolute value because ∆ W ( l ) s,c = − λ g ( l ) D,s,c is positive.

Original: Thiswill form a positive feedback process, in which the numerator of GSNR, ([ k l ] c 1 c 2 ) 2 , would increaseand so is the GSNR.
Revised: This will form a positive feedback process,in which the numerator of GSNR, ( g ( l ) D,s,c ) 2 , would increase and so is the GSNR.

Original: Similar things happen for the weights with { [ ω l ] c 1 c 2 < 0 , [ k l ] c 1 c 2 > 0 } .
Revised: Similar analysiscan be done for the case with { W ( l ) s,c < 0 , g ( l ) D,s,c > 0 } .

Original: Then what about the weights with { [ ω l ] c 1 c 2 [ k l ] c 1 c 2 > 0 } ? Here we show that the weights with 255 { [ ω l ] c 1 c 2 [ k l ] c 1 c 2 > 0 } tends to change into weights with { [ ω l ] c 1 c 2 [ k l ] c 1 c 2 < 0 } during training 256 process.
Revised: On the other hand, when { W ( l ) s,c g ( l ) D,s,c > 0 } , we show that the weights tend to change into the earliercase, i.e. { W ( l ) s,c g ( l ) D,s,c < 0 } during training.

Original: Consider the case where { [ ω l ] c 1 c 2 > 0 , [ k l ] c 1 c 2 > 0 } , the ﬁrst term in equation 26 will benegative, learning ϕ tends to decrease [ k l ] c 1 c 2 or even change the sign of [ k l ] c 1 c 2 .
Revised: Consider the case of { W ( l ) s,c > 0 , g ( l ) D,s,c > 0 } , the ﬁrstterm in eq. (28) is negative, learning θ l − tends to decrease g ( l ) D,s,c or even change its sign.

Original: Another posibilityis that learning [ ω l ] c 1 c2 tends to changes the sign of [ ω l ] c 1 c 2 because ∆[ ω l ] c 1 c 2 = − λ [ k l ] c 1 c 2 is 259 negative in this case.
Revised: Anotherposibility is that learning W ( l ) s, c changes the sign of W ( l ) s,c because ∆ W ( l ) s,c = − λ g ( l ) D,s,c is negative.

Original: Both case will change the weights  with { [ ω l ] c 1 c 2 [ k l ] c 1 c 2 > 0 } to weights with 260 { [ ω l ] c 1 c 2 [ k l ] c 1 c 2 < 0 } .
Revised: In both cases the weights change into the earlier case with { W ( l ) s,c g ( l ) D,s,c < 0 } .

Original: Same things happen for the weights with { [ ω l ] c 1 c 2 < 0 , [ k l ] c 1 c 2 < 0 } .
Revised: Similar analysis canbe done for the case of { W ( l ) s,c < 0 , g ( l ) D,s,c < 0 } .

Original: Therefore { [ ω l ] c 1 c 2 [ k l ] c 1 c 2 < 0 } is a more stable statethan { [ ω l ] c 1 c 2 [ k l ] c 1 c 2 > 0 } in the earlytraining process.
Revised: Therefore { W ( l ) s,c g ( l ) D,s,c < 0 } is a more stable state in the training process.

Original: For a simple model trained on Mnist, We plot the proportion of weights that satisfy 263 { [ ω l ] c 1 c 2 [ k l ] c 1 c 2 < 0 } in Figure 4b and ﬁnd that there are  more weights with { [ ω l ] c 1 c 2 [ k l ] c 1 c 2 < 0 }than the opposite.
Revised: For a simple modeltrained on MNIST, We plot the proportion of weights satisfying { W ( l ) s,c g ( l ) D,s,c < 0 } in Figure 4band ﬁnd that there are indeed more weights with { W ( l ) s,c g ( l ) D,s,c < 0 } than the opposite.

Original: Because weights with small absolute value easily change sign during training, wealso plot this proportion for the top-10% weights with larger absolute values.
Revised: Becauseweights with small absolute value easily change sign during training, we also plot this proportionfor the top-10% weights with larger absolute values.

Original: It can be seen that forthe weights with large absolute values, nearly 80% of these weights have opposite sign with theirgradient mean as training proceeds.
Revised: We can see that for the weights with largeabsolute values, nearly 80% of them have opposite signs with their gradient mean, conﬁrming ourearlier analysis.

Original: And for these weights, the numerator of GSNR, ([ k l ] c 1 c 2 ) 2 ,tends to increase through a posotive feedback process in the early training stage of DNN. Through discussions above, we further demonstrate that the learning of corresponding features tendsto increase the GSNR of a weight based on analytical analysis. This GSNR increasing process leadsto larger GSNR in the whole gradient descent training process of DNN, which in turn provides goodgeneralization performance of DNN. [Table]
Revised: For these weights, the numerator of GSNR, ( g ( l ) D,s,c ) 2 , tends to increase through thepositive feedback process as discussed above.

Original: [Table]
Revised: [Table]

Original:  S UMMARY
Revised:  S UMMARY

Original: In this paper we investigated the relation between generalization of DNNs and the GSNR  of themodel parameters in gradient descent.
Revised: In this paper we performed a series of analysis on the role of model parameters’ GSNR in deepneural networks’ generalization ability. We showed that large GSNR is a key to small generalizationgap, and gradient descent training naturally incurs and exploits large GSNR as the model discoversuseful features in learning. A A PPENDIX AA. M ODEL S TRUCTURE IN S ECTION 2.4 As shown in Table 2, all models in the experiment consist of 2 Conv-Relu-MaxPooling blocks and2 fully-connected layers, but they are different in the number of channels.

Revised: We choose the number ofchannels p from { 6 , 8 , 10 , 12 , 14 , 16 , 18 , 20 } .

Revised: [Table]

Original:  We also analyzed the GSNR behavior and the mechanismbehind it in the DNNs training process.
Revised: Different from the experiment on MNIST, we use a deeper network on CIFAR10. We also includethe Batch Normalization (BN) layer [Table]

Original: Through our analysis, we hope to shed more light on themechanisms behind DNNs impressive generalization ability.
Revised: because we ﬁnd that it’s difﬁcult for the network to convergein the absence of it. The network consists of 4 Conv-BN-Relu-Conv-BN-Relu-MaxPooling blocksand 3 fully-connected layers. More details are shown in Table [Table]

Revised: [Table]

Revised: [Equation]

Revised: The experiment is conducted under a similar setting as that of MNIST in section 2.4.

Revised: We choosen ∈ { 2000 , 4000 , 6000 , 8000 , 10000 } , p random ∈ { 0 . 0 , 0 .

Revised: 2 , 0 . 4 } , ch ∈ { 6 , 8 , 10 , 12 , 14 , 16 , 18 } .

Revised: We use the gradient descent training (Not SGD), with a small learning rate of 0 . 001 .

Revised: The left andright hand sides of 19 at different epochs are shown in Figure 5, where each point represents onespeciﬁc combination of the above settings.

Revised: Note that at the evaluation step of every epoch, we use the same mean and variance inside the BN layers as the training dataset to the network loss function are consistent between training and test.

Revised: At the beginning of training, compared to that of MNIST, the data points no longer perfectly resideson the diagonal dashed line.

Revised: We suppose that’s beacuse of the presence of BN layer, whose internalparameters, i.e. running mean and running variance, are not regular learnable parameters in theoptimization process, but change their values in a different way.

Revised: Their change affects the OSGR,yet we could not include them in the estimation of OSGR.

Original: B A PPENDIX B
Revised: However, the strong positive correlationbetween the left and right hand sides of eq. (19) can always be observed until the training begins toconverge. A. E XPERIMENT ON T OY D ATASET In this section we show a simple two-layer regression model consists of a FC-Relu structure withonly 2 inputs, 1 hidden layer with N neurons and 1 output.

Revised: A similar synthetic dataset with thetraining data used in the experiment of Section 3.2 is generated as follows.

Revised: Each data point isconstructed i.i.d.

Revised: using y = x 0 x 1 + ϵ , where x 0 and x 1 are drawn from uniform distribution of[ − 1 , 1] and ϵ is drawn from uniform distribution of [ − η noise , η noise ] .

Revised: To estimate eq.

Revised: (24), we randomly generate 100 training sets with n samples each, i.e. M =100,and a test set with 20,000 samples.

Original: The step-by-step derivation of equation∆[ k
Revised: To cover different conditions, we (1) choose n ∈ 345 { 50 , 100 , 300 , 600 , 1000 , 2000 , 6000 } ; (2) inject noise with η noise ∈ { 0 . 2 , 2 , 4 , 6 , 8 } ; (3) perturbmodel structures by choosing N ∈ { 6 , 8 , 10 , 12 , 14 , 16 , 18 , 20 } . We use gradient descent with learning rate of 0.001. Figure 6 shows a similar behavior as Fig. 1. During the early training stages, the LHS and RHS ofeq.

Original: [Equation] [Equation]
Revised: (19) are very close.

Original: ∑ q [Equation] ∂ [ k
Revised: Their highly correlated relation remains until training converges, whereasthe RHS of eq.

Original: [Equation] 1 c∂ϕ q ∆ ϕ q + other terms (27) = [Equation] ∂ϕ
Revised: (19) decreases signiﬁcantly.

Original: [Equation] ∂ϕ q ) + other terms (28)= [Equation] ∑
Revised: B A PPENDIX BDerivation of eq. (28)

Original: [Equation]
Revised: [Equation]

Original: ∑ i
Revised: i,s

Original: [Equation]
Revised: [Equation]

Original: ∑ [Equation] ∂L [Equation]
Revised: + other terms

Original: ∂ [ o [Equation] ic ′ 
Revised: Above we used ∂o ( l ) i,c ′ ∂a

Original: [Equation]

Original: [ o

Original: [Equation]

Original: ∂ [ a

Original: [Equation]
Revised: [Equation]

Original: ic

Original: [Equation] ∂ [ a l ] ic [Equation] ∂ϕ q ) + other terms (29) = − λ n
Revised: ) i,s

Original: [Equation]
Revised: [Equation]

Original: ∑ [Equation] ∂ϕ q ( n ∑ [Equation]
Revised: =

Original: + other terms

Original: [Equation]
Revised: [Equation]

Original: In the above derivation, we use relation ∂ [ o l ] ic ′ 2 ∂ [ a l ] ic ′ 1 = [ ω l ] c ′ 1 c ′2 and ∂ [ o l ] ic 2 ∂ [ ω l ] c 1 c 2 =
Revised: ( l ) s ′ ,c ′ and ∂o ( l ) i,c

Original: [ a  l ] ic 1 which can both bederived from equation 24.
Revised: ∂ W ( l ) s,c = a ( l ) i,s that can both be derived from eq.

Original: Consider the ﬁrst term of equation 31 and when c ′[-
-] =
Revised: Considerthe ﬁrst term of eq. When s ′ = s, c ′[-
-] =

Original: c1 , [-c ′ 
-]
Revised: c , [-c ′ 
-]we have

Original: =
Revised: [Equation]

Original: c 2 in thec ′ 1 , c ′ 2 summation, we have
Revised: Note that the term related to ∂ 2 L i ∂o ( l ) i,c ∂θ j a ( l ) i,s and the terms when s ′ ̸ = s or c ′ ̸ = c in eq.

Original: [Equation]
Revised: (33) are mergedinto other terms of eq. [Table] Notations [Table]

Revised: set consists of n ′ samples drawn from Z

Revised: [Table]

Revised: Model parameters, whose components are denoted as θ j

Revised: [Table]

Revised: r

Revised: [Table]

Revised: t

Revised: [Table]

Revised: single

Revised: [Table]

Revised: sample

Revised: [Table]

Revised: y

Revised: [Table]

Revised: of parameters’ gradient of a single sample

Revised: , i

Revised: [Table]

Revised: [Table]

Revised: Variance of the average

Revised: [Table]

Revised: over a training dataset of size

Revised: [Table]

Revised: Same as

Revised: [Table]

Revised: Gradient signal to noise ratio (GSNR) of model parameter θ j

Revised: L [ D ]

Revised: Empirical training

Revised: [Table]

Revised: [Table]

Revised: Empirical test loss,

Revised: [Table]

Revised: [Table]

Revised: One

Revised: [Table]

Revised: training loss decrease

Revised: [Table]

Revised: One

Revised: [Table]

Revised: step training loss decrease caused by updating one parameter θ j

Revised: [Table]

Revised: One-step generalization ratio (OSGR) for a training and test sets of size n sampled from data distribution Z ,

Revised: i

Revised: [Table]

Revised: e. , E

Revised: [Table]

Revised: One-step generalization gap increment,

Revised: i

Revised: [Table]

Revised: e. , ∆ L [ D ] - ∆ L [ D ′ ]

Revised: [Table]

Revised: Random variables

Revised: [Table]

Revised: zero mean and variance

Revised: [Table]

Revised: model parameters (weight matrix and bias) of the l -

Revised: [Table]

Revised: layer

Revised: [Table]

Revised: Collection of model parameters over all the layers before the l -th layer

Revised: [Table]

Revised: Average gradient of W ( l ) over the training dataset

Revised: [Table]

Revised: Collection of model parameters over all the layers aftering the l -th layer, including the l -th layer

Revised: [Table]

Original: Note that theterm related to ∂ 2 L i ∂ [ o l ] ic 2 ∂ϕ q [ a l ] ic 1 and the terms when c′ 1 ̸ = c 1 or c ′ 2 ̸ = c 2 in equation31 is added into other terms of equation 32.
Revised: Activations of the l -th layer, where s = { 1 , 2 , ..., S } is index of nodes/channels in the l -th layer. [Table] Outputs of matrix multiplication of the l -th layer, where c = { 1 , 2 , ..., C } is index of nodes/channels in the l + 1 -th layer. [Table]

Revised: evaluated on data sample i