Original: U NDERSTANDING W HY N EURAL N ETWORKS G ENERALIZE W ELL T HROUGH GSNR OF P ARAMETERS
Revised: U NDERSTANDING W HY N EURAL N ETWORKS G ENERALIZE W ELL T HROUGH GSNR OF P ARAMETERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: As deep neural networks (DNNs) achieve tremendous success across many application domains, researchers tried to explore in many aspects on why they generalizewell.
Revised: As deep neural networks (DNNs) achieve tremendous success across many application domains, researchers tried to explore in many aspects on why they generalizewell.

Original: In this paper, we provide a novel perspective on these issues using the gradientsignal to noise ratio (GSNR) of parameters during training process of DNNs.
Revised: In this paper, we provide a novel perspective on these issues using the gradientsignal to noise ratio (GSNR) of parameters during training process of DNNs.

Original: The
Revised: The

Original: GSNR of a parameter is deﬁned as the ratio between its gradient’s squared mean andvariance, over the data distribution.
Revised: GSNR of a parameter is deﬁned as the ratio between its gradient’s squared mean andvariance, over the data distribution.

Original: Based on several approximations, we establisha quantitative relationship between model parameters’ GSNR and the generalization gap.
Revised: Based on several approximations, we establisha quantitative relationship between model parameters’ GSNR and the generalization gap.

Original: This relationship indicates that larger GSNR during training process leadsto better generalization performance.
Revised: This relationship indicates that larger GSNR during training process leadsto better generalization performance.

Original: Moreover, we show that, different from thatof shallow models (e.g. logistic regression, support vector machines), the gradientdescent optimization dynamics of DNNs naturally produces large GSNR duringtraining, which is probably the key to DNNs’ remarkable generalization ability.
Revised: Moreover, we show that, different from thatof shallow models (e.g. logistic regression, support vector machines), the gradientdescent optimization dynamics of DNNs naturally produces large GSNR duringtraining, which is probably the key to DNNs’ remarkable generalization ability.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks typically contain far more trainable parameters than training samples, whichseems to easily cause a poor generalization performance.
Revised: Deep neural networks typically contain far more trainable parameters than training samples, whichseems to easily cause a poor generalization performance.

Original: However, in fact they usually exhibitremarkably small generalization gaps.
Revised: However, in fact they usually exhibitremarkably small generalization gaps.

Original: Traditional generalization theories such as VC dimension(Vapnik & Chervonenkis, 1991) or Rademacher complexity (Bartlett P L, 2002) cannot explain itsmechanism.
Revised: Traditional generalization theories such as VC dimension(Vapnik & Chervonenkis, 1991) or Rademacher complexity (Bartlett P L, 2002) cannot explain itsmechanism.

Original: Extensive research focuses on the generalization ability of DNNs (Neyshabur et al., 2017;
Revised: Extensive research focuses on the generalization ability of DNNs (Neyshabur et al., 2017;

Original: S. et al., 2018;
Revised: S. et al., 2018;

Original: Keskar et al., 2016; Dinh et al., 2017; Hoffer et al., 2017; R et al., 2018; Dziugaite &
Revised: Keskar et al., 2016; Dinh et al., 2017; Hoffer et al., 2017; R et al., 2018; Dziugaite &

Original: Roy, 2017; etc, 2018; Kawaguchi et al., 2017; Advani & Saxe, 2017).
Revised: Roy, 2017; etc, 2018; Kawaguchi et al., 2017; Advani & Saxe, 2017).

Original: Unlike that of shallow models such as logistic regression or support vector machines, the globalminimum of high-dimensional and non-convex DNNs cannot be found analytically, and can only beapproximated by gradient descent and its variants (Zeiler, 2012; Kingma & Ba, 2014; Graves, 2013).
Revised: Unlike that of shallow models such as logistic regression or support vector machines, the globalminimum of high-dimensional and non-convex DNNs cannot be found analytically, and can only beapproximated by gradient descent and its variants (Zeiler, 2012; Kingma & Ba, 2014; Graves, 2013).

Original: Previous work (Zhang et al., 2016; Hardt et al., 2015; Dziugaite & Roy, 2017) suggests that thegeneralization ability of DNNs is closely related to gradient descent optimization.
Revised: Previous work (Zhang et al., 2016; Hardt et al., 2015; Dziugaite & Roy, 2017) suggests that thegeneralization ability of DNNs is closely related to gradient descent optimization.

Original: For example, Hardtet al.
Revised: For example, Hardtet al.

Original: (2015) claims that any model trained with stochastic gradient descent (SGD) for reasonableepochs would exhibit small generalization error.
Revised: (2015) claims that any model trained with stochastic gradient descent (SGD) for reasonableepochs would exhibit small generalization error.

Original: Their analysis is based on the smoothness of lossfunction.
Revised: Their analysis is based on the smoothness of lossfunction.

Original: In this work, we attempt to understand the generalization behavior of DNNs through GSNRand reveal how GSNR affects the training dynamics of gradient descent.
Revised: In this work, we attempt to understand the generalization behavior of DNNs through GSNRand reveal how GSNR affects the training dynamics of gradient descent.

Original: The GSNR of a parameter is deﬁned as the ratio between its gradient’s squared mean and varianceover the data distribution.
Revised: The GSNR of a parameter is deﬁned as the ratio between its gradient’s squared mean and varianceover the data distribution.

Original: Previous work tried to use GSNR to conduct theoretical analysis ondeep learning.
Revised: Previous work tried to use GSNR to conduct theoretical analysis ondeep learning.

Original: For example, Rainforth et al.
Revised: For example, Rainforth et al.

Original: (2018) used GSNR to analyze variational bounds inunsupervised DNNs such as variational auto-encoder (VAE).
Revised: (2018) used GSNR to analyze variational bounds inunsupervised DNNs such as variational auto-encoder (VAE).

Original: Here we focus on analyzing the relationbetween GSNR and the generalization gap.
Revised: Here we focus on analyzing the relationbetween GSNR and the generalization gap.

Original: Intuitively, GSNR measures the similarity of a parameter’s gradients among different training samples.
Revised: Intuitively, GSNR measures the similarity of a parameter’s gradients among different training samples.

Original: Large GSNR implies that most training samples agree on the optimization direction of this parameter,thus the parameter is more likely to be associated with a meaningful “pattern” and we assume itsupdate could lead to a better generalization.
Revised: Large GSNR implies that most training samples agree on the optimization direction of this parameter,thus the parameter is more likely to be associated with a meaningful “pattern” and we assume itsupdate could lead to a better generalization.

Original: In this work, we prove that the GSNR is strongly relatedto the generalization performance, and larger GSNR means a better generalization.
Revised: In this work, we prove that the GSNR is strongly relatedto the generalization performance, and larger GSNR means a better generalization.

Original: To reveal the mechanism of DNNs’ good generalization ability, we show that the gradient descentoptimization dynamics of DNN naturally leads to large GSNR of model parameters and therefore good generalization.
Revised: To reveal the mechanism of DNNs’ good generalization ability, we show that the gradient descentoptimization dynamics of DNN naturally leads to large GSNR of model parameters and therefore good generalization.

Original: Furthermore, we give a complete analysis and a detailed interpretation to thisphenomenon.
Revised: Furthermore, we give a complete analysis and a detailed interpretation to thisphenomenon.

Original: We believe this is probably the key to DNNs’ remarkable generalization ability.
Revised: We believe this is probably the key to DNNs’ remarkable generalization ability.

Original: In the remainder of this paper we ﬁrst analyze the relation between GSNR and generalization (Section2).
Revised: In the remainder of this paper we ﬁrst analyze the relation between GSNR and generalization (Section2).

Original: We then show how the training dynamics lead to large GSNR of model parameters experimentallyand analytically in Section 3.
Revised: We then show how the training dynamics lead to large GSNR of model parameters experimentallyand analytically in Section 3.

Original:  L ARGER GSNR L EADS TO B ETTER G ENERALIZATION
Revised:  L ARGER GSNR L EADS TO B ETTER G ENERALIZATION

Original: In this section, we establish a quantitative relation between the GSNR of model parameters andgeneralization gap, showing that larger GSNR during training leads to better generalization.
Revised: In this section, we establish a quantitative relation between the GSNR of model parameters andgeneralization gap, showing that larger GSNR during training leads to better generalization.

Original: 2. G RADIENTS S IGNAL TO N OISE R ATIO
Revised: 2. G RADIENTS S IGNAL TO N OISE R ATIO

Original: Consider a data distribution Z = X ×Y , from which each sample ( x, y ) is drawn; a model ˆ y = f
Revised: Consider a data distribution Z = X ×Y , from which each sample ( x, y ) is drawn; a model ˆ y = f

Original: ( x, θ )parameterized by θ ; and a loss function L .
Revised: ( x, θ )parameterized by θ ; and a loss function L .

Original: The parameters’ gradient w.r.t.
Revised: The parameters’ gradient w.r.t.

Original: L and sample ( x i , y i ) is denoted by
Revised: L and sample ( x i , y i ) is denoted by

Original: [Equation]
Revised: [Equation]

Original: ) whose j -th element is g i ( θ j ) .
Revised: whose j -th element is g i ( θ j ) .

Original: Note that throughout this paper we always use i to index data examplesand j to index model parameters.
Revised: Note that throughout this paper we always use i to index data examplesand j to index model parameters.

Original: Given the data distribution Z , we have the (sample-wise) mean and variance of g i ( θ ) .
Revised: Given the data distribution Z , we have the (sample-wise) mean and variance of g i ( θ ) .

Original: We denotethem as ˜g ( θ ) =
Revised: We denotethem as ˜g ( θ ) =

Original: E ( x,y ) ∼Z ( g ( x, y, θ )) and ρ 2 ( θ ) =
Revised: E ( x,y ) ∼Z ( g ( x, y, θ )) and ρ 2 ( θ ) =

Original: Var ( x,y ) ∼Z ( g ( x, y, θ )) , respectively.
Revised: Var ( x,y ) ∼Z ( g ( x, y, θ )) , respectively.

Original: The gradient signal to noise ratio (GSNR) of one model parameter θ j is deﬁned as:
Revised: The gradient signal to noise ratio (GSNR) of one model parameter θ j is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: At a particular point of the parameter space, GSNR measures the consistency of a parameter’sgradients across different data samples.
Revised: At a particular point of the parameter space, GSNR measures the consistency of a parameter’sgradients across different data samples.

Original: [Equation] ENERALIZATION R ATIO
Revised: 2. O NE -S TEP G ENERALIZATION R ATIO

Original: In this section we introduce a new concept to help measure the generalization performance during gradient descent optimization, which we call one-step generalization ratio (OSGR).
Revised: In this section we introduce a new concept to help measure the generalization performance during gradient descent optimization, which we call one-step generalization ratio (OSGR).

Original: Considertraining set D
Revised: Considertraining set D

Original: = { ( x 0 , y 0 ) , ..., ( x n , y n ) } ∼ Z n with n samples drawn from Z , and a test set
Revised: = { ( x 0 , y 0 ) , ..., ( x n , y n ) } ∼ Z n with n samples drawn from Z , and a test set

Original: D (cid:48)
Revised: D (cid:48)

Original: = { ( x (cid:48) 0 , y (cid:48) 0 ) , ..., ( x (cid:48) n (cid:48) , y (cid:48) n (cid:48) )
Revised: = { ( x (cid:48) 0 , y (cid:48) 0 ) , ..., ( x (cid:48) n (cid:48) , y (cid:48) n (cid:48) )

Original: } ∼ Z n (cid:48) .
Revised: } ∼ Z n (cid:48) .

Original: In practice we use the loss on D (cid:48) to measure generalization.
Revised: In practice we use the loss on D (cid:48) to measure generalization.

Revised: For simplicity, we assume the sizes of training and test datasets are equal, i.e. n = n (cid:48) .

Original: We denote the empirical training and test loss as:
Revised: We denote theempirical training and test loss as:

Original: [Equation]
Revised: [Equation]

Original: respectively.
Revised: respectively.

Original: Then the empirical generalization gap is given by L [ D (cid:48) ] − L [ D ] .
Revised: Then the empirical generalization gap is given by L [ D (cid:48) ] − L [ D ] .

Original: In gradient descent optimization, both the training and test loss would decrease step by step.
Revised: In gradient descent optimization, both the training and test loss would decrease step by step.

Original: We use∆ L [ D ] and ∆ L [ D (cid:48) ] to denote the one-step training and test loss decrease during training, respectively.
Revised: We use∆ L [ D ] and ∆ L [ D (cid:48) ] to denote the one-step training and test loss decrease during training, respectively.

Original: Let’s consider the ratio between the expectations of ∆ L
Revised: Let’s consider the ratio between the expectations of ∆ L

Original: [ D (cid:48) ] and ∆ L [ D ] of one single training step,which we denote as R ( Z , n ) .
Revised: [ D (cid:48) ] and ∆ L [ D ] of one single training step,which we denote as R ( Z , n ) .

Original: [Equation]
Revised: [Equation]

Original: Note that this ratio also depends on current model parameters θ and learning rate λ .
Revised: Note that this ratio also depends on current model parameters θ and learning rate λ .

Original: We are notincluding them in the above notation as we will not explicitly model these dependencies, but rathertry to quantitatively characterize R for very small λ and for θ at the early stage of training (satisfying
Revised: We are notincluding them in the above notation as we will not explicitly model these dependencies, but rathertry to quantitatively characterize R for very small λ and for θ at the early stage of training (satisfying

Original: Assumption 2.3.1).
Revised: Assumption 2.3.1).

Original: Also note that the expectation of ∆ L [ D (cid:48) ] is over D and D (cid:48) .
Revised: Also note that the expectation of ∆ L [ D (cid:48) ] is over D and D (cid:48) .

Original: This is because the optimization step isperformed on D .
Revised: This is because the optimization step isperformed on D .

Original: We refer to R ( Z , n ) as OSGR of gradient descent optimization.
Revised: We refer to R ( Z , n ) as OSGR of gradient descent optimization.

Original: Statistically the training loss decreases faster than the test loss and OSGR would be less than 1, which usually resultsin a non-zero generalization gap at the end of training.
Revised: Statistically thetraining loss decreases faster than the test loss and OSGR would be less than 1, which usually resultsin a non-zero generalization gap at the end of training.

Original: If OSGR is large in the whole training process,generalization gap would be small when training completes, implying good generalization ability ofthe model.
Revised: If OSGR is large in the whole training process,generalization gap would be small when training completes, implying good generalization ability ofthe model.

Original: 2. R ELATION BETWEEN GSNR AND OSGR
Revised: 2. R ELATION BETWEEN GSNR AND OSGR

Original: In gradient descent optimization, we take the average gradient over training set D , which we denoteas g D ( θ ) .
Revised: In gradient descent optimization, we take the average gradient over training set D , which we denoteas g D ( θ ) .

Original: Note that we have used g i ( θ ) to denote gradient evaluated on one data sample and ˜ g
Revised: Note that we have used g i ( θ ) to denote gradient evaluated on one data sample and ˜ g

Original: ( θ ) todenote its expectation over the entire data distribution.
Revised: ( θ ) todenote its expectation over the entire data distribution.

Original: Similarly we deﬁne g D (cid:48)
Revised: Similarly we deﬁne g D (cid:48)

Original: ( θ ) to be the averagegradient over test set D (cid:48) .
Revised: ( θ ) to be the averagegradient over test set D (cid:48) .

Original: [Equation]
Revised: [Equation]

Original: Both the training and test dataset are randomly generated from the same distribution Z n , so wecan treat g D ( θ ) and g D (cid:48)
Revised: Both the training and test dataset are randomly generated from the same distribution Z n , so wecan treat g D ( θ ) and g D (cid:48)

Original: ( θ ) as random variables.
Revised: ( θ ) as random variables.

Original: At the beginning of the optimization process, θ israndomly initialized thus independent of D , so g D ( θ ) and g D (cid:48)
Revised: At the beginning of the optimization process, θ israndomly initialized thus independent of D , so g D ( θ ) and g D (cid:48)

Original: ( θ ) would obey the same distribution.
Revised: ( θ ) would obey the same distribution.

Original: After a period of training, the model parameters begin to ﬁt the training dataset and become a functionof D , i.e. θ = θ ( D ) , therefore distributions of g D ( θ ( D )) and g D (cid:48)
Revised: After a period of training, the model parameters begin to ﬁt the training dataset and become a functionof D , i.e. θ = θ ( D ) , therefore distributions of g D ( θ ( D )) and g D (cid:48)

Original: ( θ ( D )) become different.
Revised: ( θ ( D )) become different.

Original: Howeverwe choose not to model this dependency and make the following assumption for our analysis:
Revised: Howeverwe choose not to model this dependency and make the following assumption for our analysis:

Original: Assumption 2.3.1 (Non-overﬁtting limit approximation)
Revised: Assumption 2.3.1 (Non-overﬁtting limit approximation)

Original: The average gradient over the trainingdataset and test dataset g D ( θ ) and g D (cid:48)
Revised: The average gradient over the trainingdataset and test dataset g D ( θ ) and g D (cid:48)

Original: ( θ ) obey the same distribution.
Revised: ( θ ) obey the same distribution.

Original: Obviously the mean of g D ( θ ) and g D (cid:48)
Revised: Obviously the mean of g D ( θ ) and g D (cid:48)

Original: ( θ ) is just the mean gradient over the data distribution ˜ g
Revised: ( θ ) is just the mean gradient over the data distribution ˜ g

Original: ( θ ) .
Revised: ( θ ) .

Original: [Equation]
Revised: [Equation]

Original: We denote their variance as σ 2 ( θ ) , i.e.
Revised: We denote their variance as σ 2 ( θ ) , i.e.

Original: [Equation]
Revised: [Equation]

Original: It is straightforward to show that:
Revised: It is straightforward to show that:

Original: [Equation]
Revised: [Equation]

Original: where σ 2 ( θ ) is the variance of the average gradient over the dataset of size n , and ρ 2 ( θ ) is the varianceof the gradient of a single data sample.
Revised: where σ 2 ( θ ) is the variance of the average gradient over the dataset of size n , and ρ 2 ( θ ) is the varianceof the gradient of a single data sample.

Original: In one gradient descent step, the model parameter is updated by ∆ θ = θ t +1 − θ t = − λ g D ( θ )where λ is the learning rate.
Revised: In one gradient descent step, the model parameter is updated by ∆ θ = θ t +1 − θ t = − λ g D ( θ )where λ is the learning rate.

Original: If λ is small enough, the one-step training and test loss decrease can beapproximated by
Revised: If λ is small enough, the one-step training and test loss decrease can beapproximated by

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Usually there are some differences between the directions of g D ( θ ) and g D (cid:48)
Revised: Usually there are some differences between the directions of g D ( θ ) and g D (cid:48)

Original: ( θ ) ,
Revised: ( θ ) ,

Original: so statistically∆ L [ D ] tends to be larger than ∆ L
Revised: so statistically∆ L [ D ] tends to be larger than ∆ L

Original: [ D (cid:48) ] and the generalization gap would increase during training.
Revised: [ D (cid:48) ] and the generalization gap would increase during training.

Original: When λ → 0 , in one single training step the empirical generalization gap increases by ∆ L [ D ] −∆ L [ D (cid:48) ] , for simplicity we denote this quantity as (cid:53) :
Revised: When λ → 0 , in one single training step the empirical generalization gap increases by ∆ L [ D ] −∆ L [ D (cid:48) ] , for simplicity we denote this quantity as (cid:53) :

Original: [Equation]
Revised: [Equation]

Original: Here we replaced the random variables by g D ( θ ) =
Revised: Here we replaced the random variables by g D ( θ ) =

Original: ˜ g
Revised: ˜ g

Original: ( θ ) + (cid:15) and g D (cid:48)
Revised: ( θ ) + (cid:15) and g D (cid:48)

Original: ( θ ) =
Revised: ( θ ) =

Original: ˜ g
Revised: ˜ g

Original: ( θ ) + (cid:15) (cid:48) , where (cid:15) and(cid:15) (cid:48) are random variables with zero mean and variance σ 2 ( θ ) .
Revised: ( θ ) + (cid:15) (cid:48) , where (cid:15) and(cid:15) (cid:48) are random variables with zero mean and variance σ 2 ( θ ) .

Original: Since E ( (cid:15) (cid:48) ) =
Revised: Since E ( (cid:15) (cid:48) ) =

Original: E ( (cid:15) )
Revised: E ( (cid:15) )

Original: = 0 , (cid:15) and (cid:15) (cid:48) are independent, the expectation of (cid:53) is
Revised: = 0 , (cid:15) and (cid:15) (cid:48) are independent, the expectation of (cid:53) is

Original: [Equation]
Revised: [Equation]

Original: where σ 2 ( θ j ) is the variance the of average gradient of the parameter θ j .
Revised: where σ 2 ( θ j ) is the variance the of average gradient of the parameter θ j .

Original: For simplicity, when it involves a single model parameter θ j , we will use only a subscript j insteadof the full notation.
Revised: For simplicity, when it involves a single model parameter θ j , we will use only a subscript j insteadof the full notation.

Original: For example, we use σ 2 j , r j , and g D,j to denote σ 2 ( θ j ) , r ( θ j ) , and g D ( θ j )respectively.
Revised: For example, we use σ 2 j , r j , and g D,j to denote σ 2 ( θ j ) , r ( θ j ) , and g D ( θ j )respectively.

Original: Consider the expectation of ∆ L [ D ] and ∆ L
Revised: Consider the expectation of ∆ L [ D ] and ∆ L

Original: [ D (cid:48) ] when λ → 

Revised: [ D (cid:48) ] when λ →

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Substituting (18) and (15) into (4) we have:
Revised: Substituting (18) and (15) into (4) we have:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Although we derived eq.
Revised: Although we derived eq.

Original: (19) from simpliﬁed assumptions, we can empirically verify it by estimatingtwo sides of the equation on real data.
Revised: (19) from simpliﬁed assumptions, we can empirically verify it by estimatingtwo sides of the equation on real data.

Original: We will elaborate on this estimation method in section 2.4.
Revised: We will elaborate on this estimation method in section 2.4.

Original: We can rewrite eq.
Revised: We can rewrite eq.

Original: (19) as:
Revised: (19) as:

Original: [Equation]
Revised: [Equation]

Original: where E D ∼Z n ( g 2 D,j ) = V ar D ∼Z n ( g D,j ) + E 2 D ∼Z n ( g D,j ) = 1 n ρ 2 j + ˜ g 2 j .
Revised: where E D ∼Z n ( g 2 D,j ) = V ar D ∼Z n ( g D,j ) + E 2 D ∼Z n ( g D,j ) = 1 n ρ 2 j + ˜ g 2 j .

Original: We deﬁne ∆ L j [ D ] to be the training loss decrease caused by updating θ j .
Revised: We deﬁne ∆ L j [ D ] to be the training loss decrease caused by updating θ j .

Original: We can show that when λis very small ∆ L j [ D ] = λ g 2 D,j + O ( λ 2 ) .
Revised: We can show that when λis very small ∆ L j [ D ] = λ g 2 D,j + O ( λ 2 ) .

Original: Therefore when λ → 0 , we have
Revised: Therefore when λ → 0 , we have

Original: [Equation]
Revised: [Equation]

Original: r j +
Revised: r j +

Original: [Equation]
Revised: [Equation]

Original: n , where W j := E D ∼Z n (∆ L j [ D ])
Revised: n , where W j := E D ∼Z n (∆ L j [ D ])

Original: E D ∼Z n (∆ L [ D ]) with (cid:88)
Revised: E D ∼Z n (∆ L [ D ]) with (cid:88)

Original: j
Revised: j

Original: W j =(22)
Revised: W j =(22)

Original: Eq.
Revised: Eq.

Original: (22) shows that the GSNR r j plays a crucial role in the model’s generalization ability—theone-step generalization ratio in gradient descent equals one minus the weighted average of 1 r j + 1 n overall model parameters divided by n .
Revised: (22) shows that the GSNR r j plays a crucial role in the model’s generalization ability—theone-step generalization ratio in gradient descent equals one minus the weighted average of 1 r j + 1 n overall model parameters divided by n .

Original: The weight is proportional to the expectation of the training lossdecrease resulted from updating that parameter.
Revised: The weight is proportional to the expectation of the training lossdecrease resulted from updating that parameter.

Original: This implies that larger GSNR of model parametersduring training leads to smaller generalization gap growth and better generalization performance ofthe trained model.
Revised: This implies that larger GSNR of model parametersduring training leads to smaller generalization gap growth and better generalization performance ofthe trained model.

Original: Also note when n → ∞ , we have R ( Z , n ) → 1 , meaning that training on moredata helps generalization.
Revised: Also note when n → ∞ , we have R ( Z , n ) → 1 , meaning that training on moredata helps generalization.

Original: 2. E XPERIMENTAL VERIFICATION OF THE RELATION BETWEEN GSNR AND OSGR
Revised: 2. E XPERIMENTAL VERIFICATION OF THE RELATION BETWEEN GSNR AND OSGR

Original: The relation between GSNR and OSGR, i.e. eq.
Revised: The relation between GSNR and OSGR, i.e. eq.

Original: (19) or (22) can be veriﬁed empirically using anydatasets if: (1) The dataset includes enough samples to construct many training sets and a largeenough test set so that we can reliably estimate ρ 2 j , E D ∼Z n ( g 2 D,j ) and OSGR.
Revised: (19) or (22) can be veriﬁed empirically using anydatasets if: (1) The dataset includes enough samples to construct many training sets and a largeenough test set so that we can reliably estimate ρ 2 j , E D ∼Z n ( g 2 D,j ) and OSGR.

Original: The learning rateis small enough.
Revised: The learning rateis small enough.

Original: In the early training stage of gradient descent.
Revised: In the early training stage of gradient descent.

Original: To empirically verify eq.
Revised: To empirically verify eq.

Original: (19), we show how to estimate its left and right hand sides, i.e. OSGR bydeﬁnition and OSGR as a function of GSNR.
Revised: (19), we show how to estimate its left and right hand sides, i.e. OSGR bydeﬁnition and OSGR as a function of GSNR.

Original: Suppose we have M training sets each with size n , and a test set of size n (cid:48) .
Revised: Suppose we have M training sets each with size n , and a test set of size n (cid:48) .

Original: We initialize a model and train it separately on the M training sets and test itwith the same test set.
Revised: We initialize a model and train it separately on the M training sets and test itwith the same test set.

Original: For the t -th training iteration, we denote the training loss and test loss of themodel trained on the m -th training dataset as L ( m ) t and L (cid:48) ( m ) t , respectively.
Revised: For the t -th training iteration, we denote the training loss and test loss of themodel trained on the m -th training dataset as L ( m ) t and L (cid:48) ( m ) t , respectively.

Original: Then the left hand side, i.e. OSGR by deﬁnition, of the t -th iteration can be estimated by
Revised: Then the left hand side,i.e. OSGR by deﬁnition, of the t -th iteration can be estimated by

Original: [Equation]
Revised: [Equation]

Original: For the model trained on the m -th training set, we can compute the t -th step avreage gradient andsample-wise gradient variance of θ j on the corresponding training set, denoted as g m,j,t and ρ 2 m,j,t ,respectively.
Revised: For the model trained on the m -th training set, we can compute the t -th step avreage gradient andsample-wise gradient variance of θ j on the corresponding training set, denoted as g m,j,t and ρ 2 m,j,t ,respectively.

Original: Therefore the right hand side of eq.
Revised: Therefore the right hand side of eq.

Original: (19) can be estimated by
Revised: (19) can be estimated by

Original: [Equation]
Revised: [Equation]

Original: We performed the above estimations on MNIST with a simple CNN structure consists of 2 Conv-ReluMaxPooling blocks and 2 fully-connected layers.
Revised: We performed the above estimations on MNIST with a simple CNN structure consists of 2 Conv-ReluMaxPooling blocks and 2 fully-connected layers.

Original: First, to estimate eq.
Revised: First, to estimate eq.

Original: (24) with M = 10 , we randomly sample 10 training sets with size n and a test set with size 10,000.
Revised: (24) with M = 10 , we randomly sample 10 training sets with size n and a test set with size 10,000.

Original: To cover different conditions,we (1) choose n ∈ { 1000 , 2000 , 4000 , 6000 , 8000 , 10000 , 15000 } , respectively; (2) inject noise byrandomly changing the labels with probability p random ∈ { 0 . 0 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 5 } ; (3) change themodel structure by varying number of channels in the layers, ch ∈ { 6 , 8 , 10 , 12 , 14 , 16 , 18 , 20 } .
Revised: To cover different conditions,we (1) choose n ∈ { 1000 , 2000 , 4000 , 6000 , 8000 , 10000 , 15000 } , respectively; (2) inject noise byrandomly changing the labels with probability p random ∈ { 0 . 0 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 5 } ; (3) change themodel structure by varying number of channels in the layers, ch ∈ { 6 , 8 , 10 , 12 , 14 , 16 , 18 , 20 } .

Original: See
Revised: See

Original: Appendix A for more details of the setup.
Revised: Appendix A for more details of the setup.

Original: We use the gradient descent training (not SGD), with asmall learning rate of 0 .
Revised: We use the gradient descent training (not SGD), with asmall learning rate of 0 .

Original: The left and right hand sides of 19 at different epochs are shown in
Revised: The left and right hand sides of 19 at different epochs are shown in

Original: Figure 1, where each point represents one speciﬁc choice of the above settings.
Revised: Figure 1, where each point represents one speciﬁc choice of the above settings.

Original: At the beginning of training, the data points are closely distributed along the dashed line correspondingto LHS=
Revised: At the beginning of training, the data points are closely distributed along the dashed line correspondingto LHS=

Original: RHS.
Revised: RHS.

Original: This shows that eq.
Revised: This shows that eq.

Original: (19) ﬁts quite well under a variety of different settings.
Revised: (19) ﬁts quite well under a variety of different settings.

Original: As trainingproceeds, the points become more scattered as the non-overﬁtting limit approximation no longerholds, but correlation between the LHS and RHS remains high even when the training converges (atepoch 2,500).
Revised: As trainingproceeds, the points become more scattered as the non-overﬁtting limit approximation no longerholds, but correlation between the LHS and RHS remains high even when the training converges (atepoch 2,500).

Original: We also conducted the same experiment on CIFAR10 A.2 and a toy dataset A.3 andobserved the same behavior.
Revised: We also conducted the same experiment on CIFAR10 A.2 and a toy dataset A.3 andobserved the same behavior.

Original: See Appendix for these experiments.
Revised: See Appendix for these experiments.

Original: The empirical evidence together with our previous derivation of eq.
Revised: The empirical evidence together with our previous derivation of eq.

Original: (19) clearly show the relationbetween GSNR and OSGR and its implication in the model’s generalization ability.
Revised: (19) clearly show the relationbetween GSNR and OSGR and its implication in the model’s generalization ability.

Original:  T RAINING DYNAMICS OF DNN S NATURALLY LEADS TO LARGE GSNR
Revised:  T RAINING DYNAMICS OF DNN S NATURALLY LEADS TO LARGE GSNR

Original: In this section, we analyze and explain one interesting phenomenon: the parameters’ GSNR of DNNsrises in the early stages of training, whereas the GSNR of shallow models such as logistic regression or support vector machines declines during the entire training process.
Revised: In this section, we analyze and explain one interesting phenomenon: the parameters’ GSNR of DNNsrises in the early stages of training, whereas the GSNR of shallow models such as logistic regression or support vector machines declines during the entire training process.

Original: This difference gives rise to
Revised: This difference gives rise to

Original: GSNR’s large practical values during training, which in turn is associated with good generalization.
Revised: GSNR’s large practical values during training, which in turn is associated with good generalization.

Original: We analyze the dynamics behind this phenomenon both experimentally and theoretically.
Revised: We analyze the dynamics behind this phenomenon both experimentally and theoretically.

Original: 3. GSNR BEHAVIOR OF DNN S TRAINING
Revised: 3. GSNR BEHAVIOR OF DNN S TRAINING

Original: For shallow models, the GSNR of parameters decreases in the whole training process becausegradients become small as learning converges.
Revised: For shallow models, the GSNR of parameters decreases in the whole training process becausegradients become small as learning converges.

Original: But for DNNs it is not the case.
Revised: But for DNNs it is not the case.

Original: We trained DNNson
Revised: We trained DNNson

Original: the CIFAR datasets and computed the GSNR averaged over all model parameters.
Revised: the CIFAR datasets and computed the GSNR averaged over all model parameters.

Original: Because
Revised: Because

Original: E D ∼Z n ( g 2 D,j ) =
Revised: E D ∼Z n ( g 2 D,j ) =

Original: 1 n ρ 2 j + ˜ g
Revised: 1 n ρ 2 j + ˜ g

Original: 2 j
Revised: 2 j

Original: and we assume n is large, E D ∼Z n ( g 2 D,j )
Revised: and we assume n is large, E D ∼Z n ( g 2 D,j )

Original: ≈ ˜
Revised: ≈ ˜

Original: g 2 j .
Revised: g 2 j .

Original: In the case of only onelarge training datasets, we estimate GSNR of t -th iteration by
Revised: In the case of only onelarge training datasets, we estimate GSNR of t -th iteration by

Original: [Equation]
Revised: [Equation]

Original: As shown in Figure 2, the GSNR starts out low with randomly initialized parameters.
Revised: As shown in Figure 2, the GSNR starts out low with randomly initialized parameters.

Original: As learningprogresses, the GSNR increases in the early training stage and stays at a high level in the wholelearning process.
Revised: As learningprogresses, the GSNR increases in the early training stage and stays at a high level in the wholelearning process.

Original: For each model parameter, we also computed the proportion of the samples withthe same gradient sign, denoted as p same _ sign .
Revised: For each model parameter, we also computed the proportion of the samples withthe same gradient sign, denoted as p same _ sign .

Original: In Figure 2c, we plot the mean of time series of thisproportion for all the parameters .
Revised: In Figure 2c, we plot the mean of time series of thisproportion for all the parameters .

Original: This value increases from about 50% (half positive half negetivedue to random initialization) to about 56% ﬁnally, which indicates that for most parameters, thegradient signs on different samples become more consistent.
Revised: This value increases from about 50% (half positive half negetivedue to random initialization) to about 56% ﬁnally, which indicates that for most parameters, thegradient signs on different samples become more consistent.

Original: This is because meaningful featuresbegin to emerge in the learning process and the gradients of the weights on these features tend tohave the same sign among different samples.
Revised: This is because meaningful featuresbegin to emerge in the learning process and the gradients of the weights on these features tend tohave the same sign among different samples.

Original: Previous research (Zhang et al., 2016) showed that DNNs achieved zero training loss by memorizingtraining samples even if the labels were randomized.
Revised: Previous research (Zhang et al., 2016) showed that DNNs achieved zero training loss by memorizingtraining samples even if the labels were randomized.

Original: We also plot the average GSNR for modeltrained using data with randomized labels in Figure 2 and ﬁnd that the GSNR stays at a low levelthroughout the training process.
Revised: We also plot the average GSNR for modeltrained using data with randomized labels in Figure 2 and ﬁnd that the GSNR stays at a low levelthroughout the training process.

Original: Although the training loss of both the original and randomized labelsgo to zero (not shown), the GSNR curves clearly distinguish between these two cases and reveal thelack of meaningful patterns in the latter one.
Revised: Although the training loss of both the original and randomized labelsgo to zero (not shown), the GSNR curves clearly distinguish between these two cases and reveal thelack of meaningful patterns in the latter one.

Original: We believe this is the reason why DNNs trained on realand random data lead to completely different generalization behaviors.
Revised: We believe this is the reason why DNNs trained on realand random data lead to completely different generalization behaviors.

Original: T RAINING D YNAMICS BEHIND THE GSNR BEHAVIOR
Revised: T RAINING D YNAMICS BEHIND THE GSNR BEHAVIOR

Original: In this section we show that the feature learning ability of DNNs is the key reason why the GSNRcurve behavior of DNNs is different from that of shallow models during the gradient descent training.
Revised: In this section we show that the feature learning ability of DNNs is the key reason why the GSNRcurve behavior of DNNs is different from that of shallow models during the gradient descent training.

Original: To demonstrate this, a simple two-layer perceptron regression model is constructed.
Revised: To demonstrate this, a simple two-layer perceptron regression model is constructed.

Original: A syntheticdataset is generated as following.
Revised: A syntheticdataset is generated as following.

Original: Each data point is constructed i.i.d.
Revised: Each data point is constructed i.i.d.

Original: using y = x 0 x 1 + (cid:15) , wherex 0 and x 1 are drawn from uniform distribution [ − 1 , 1] and (cid:15) is drawn from uniform distribution[ − 0 .
Revised: using y = x 0 x 1 + (cid:15) , wherex 0 and x 1 are drawn from uniform distribution [ − 1 , 1] and (cid:15) is drawn from uniform distribution[ − 0 .

Original: The training set and test set sizes are 200 and 10,000, respectively.
Revised: The training set and test set sizes are 200 and 10,000, respectively.

Original: We use a verysimple two-layer MLP structure with 2 inputs, 20 hidden neurons and 1 output.
Revised: We use a verysimple two-layer MLP structure with 2 inputs, 20 hidden neurons and 1 output.

Original: We randomly initialized the model parameters and trained the model on the synthetic training dataset.
Revised: We randomly initialized the model parameters and trained the model on the synthetic training dataset.

Original: As a control setup we also tried to freeze model weights in the ﬁrst layer to prevent it from learningfeatures.
Revised: As a control setup we also tried to freeze model weights in the ﬁrst layer to prevent it from learningfeatures.

Original: Note that a two layer MLP with the ﬁrst layer frozen is equivalent to a linear regressionmodel.
Revised: Note that a two layer MLP with the ﬁrst layer frozen is equivalent to a linear regressionmodel.

Original: That is, regression weights are learned on the second layer using ﬁxed features extracted bythe ﬁrst layer.
Revised: That is, regression weights are learned on the second layer using ﬁxed features extracted bythe ﬁrst layer.

Original: We plot the average GSNR of the second layer parameters for both the frozen andnon-frozen cases.
Revised: We plot the average GSNR of the second layer parameters for both the frozen andnon-frozen cases.

Original: Figure 3 shows that in the non-frozen case, the average GSNR over parameters of the second layer shows a signiﬁcant upward process, whereas in the frozen case the average GSNRdecreases in the beginning and remains at a low level during the whole training process.
Revised: Figure 3 shows that in the non-frozen case, the average GSNR over parameters of the second layer shows a signiﬁcant upward process, whereas in the frozen case the average GSNRdecreases in the beginning and remains at a low level during the whole training process.

Original: In the non-frozen case, GSNR curve of individual parameters of the second layer are shown in Figure3.
Revised: In the non-frozen case, GSNR curve of individual parameters of the second layer are shown in Figure3.

Original: The GSNR for some parameters show a signiﬁcant upward process.
Revised: The GSNR for some parameters show a signiﬁcant upward process.

Original: To measure the quality ofthese features, we computed the Pearson correlation between them and the target output y , both at thebeginning of training and at the maximum point of their GSNR curves.
Revised: To measure the quality ofthese features, we computed the Pearson correlation between them and the target output y , both at thebeginning of training and at the maximum point of their GSNR curves.

Original: We can see that the learningprocess learns “good” features (high correlation value, i.e. with stronger correlation with y ) fromrandom initialized ones, as shown in Table 1.
Revised: We can see that the learningprocess learns “good” features (high correlation value, i.e. with stronger correlation with y ) fromrandom initialized ones, as shown in Table 1.

Original: This shows that the GSNR increasing process is relatedto feature learning.
Revised: This shows that the GSNR increasing process is relatedto feature learning.

Original: 3. A NALYSIS OF TRAINING DYNAMICS BEHIND DNN S ’ GSNR BEHAVIOR
Revised: 3. A NALYSIS OF TRAINING DYNAMICS BEHIND DNN S ’ GSNR BEHAVIOR

Original: In this section, we will investigate the training dynamics behind the GSNR curve behavior.
Revised: In this section, we will investigate the training dynamics behind the GSNR curve behavior.

Original: In the caseof fully connected network structure, we can analytically show that the numerator of GSNR, i.e. thesquared gradient mean of model parameters, tends to increase in the early training stage throughfeature learning.
Revised: In the caseof fully connected network structure, we can analytically show that the numerator of GSNR, i.e. thesquared gradient mean of model parameters, tends to increase in the early training stage throughfeature learning.

Original: Consider a fully connected network, whose parameters are θ = { W (1) , b (1) , ..., W ( l max ) , b ( l max ) } ,where W (1) , b (1) are the weight matrix and bias of the ﬁrst layer, and so on.
Revised: Consider a fully connected network, whose parameters are θ = { W (1) , b (1) , ..., W ( l max ) , b ( l max ) } ,where W (1) , b (1) are the weight matrix and bias of the ﬁrst layer, and so on.

Original: We denote theactivations of the l -th layer as a ( l ) =
Revised: We denote theactivations of the l -th layer as a ( l ) =

Original: { a ( l ) s ( θ ( l − ) )
Revised: { a ( l ) s ( θ ( l − ) )

Original: } , where s is the index for nodes/channels ofthis layer, and θ ( l − ) is the collection of model parameters in the layers before l , i.e. θ ( l − )
Revised: } , where s is the index for nodes/channels ofthis layer, and θ ( l − ) is the collection of model parameters in the layers before l , i.e. θ ( l − )

Original: ={ W (1) , b (1) , ..., W ( l − 1) , b ( l − 1) } .
Revised: ={ W (1) , b (1) , ..., W ( l − 1) , b ( l − 1) } .

Original: In the forward pass on data sample i , { a ls ( θ ( l − ) ) } is multiplied bythe weight matrix W ( l ) :
Revised: In the forward pass on data sample i , { a ls ( θ ( l − ) ) } is multiplied bythe weight matrix W ( l ) :

Original: [Equation]
Revised: [Equation]

Original: 26) where o ( l ) = { o ( l ) i,c } is the output of the matrix multiplication, for the i -th data sample, on the l -thlayer, c
Revised: 26) where o ( l ) = { o ( l ) i,c } is the output of the matrix multiplication, for the i -th data sample, on the l -thlayer, c

Original: = 1, 2, ..., C is  index of nodes/channels in the ( l + 1) -th layer.
Revised: = 1, 2, ..., C is the index of nodes/channels in the ( l + 1) -th layer.

Original: We use g ( l ) D to denote theaverage gradient of weights of the l -th layer W ( l ) , i.e. g ( l ) D = 1 n (cid:80) ni 
Revised: We use g ( l ) D to denote theaverage gradient of weights of the l -th layer W ( l ) , i.e. g ( l ) D = 1 n (cid:80) ni =

Original: [Equation]
Revised: [Equation]

Original: L i ∂ W ( l ) , where L i is the loss of 225 the i -th sample.
Revised: L i ∂ W ( l ) , where L i is the loss of 226 the i -th sample.

Original: Here we show that the feature learning ability of DNNs plays a crucial role in the GSNR increasingprocess.
Revised: Here we show that the feature learning ability of DNNs plays a crucial role in the GSNR increasingprocess.

Original: More precisely, we show that the learning of features a ( l ) ( θ ( l − ) ) , i.e. the learning ofparameters θ ( l − ) tends to increase the absolute value of g ( l ) D .
Revised: More precisely, we show that the learning of features a ( l ) ( θ ( l − ) ) , i.e. the learning ofparameters θ ( l − ) tends to increase the absolute value of g ( l ) D .

Original: Consider the one-step change ofgradient mean ∆ g ( l ) D = g ( l ) D,t +1 − g ( l ) D,t with the learning rate λ → 0 .
Revised: Consider the one-step change ofgradient mean ∆ g ( l ) D = g ( l ) D,t +1 − g ( l ) D,t with the learning rate λ → 0 .

Original: In one training step, θ is updatedby ∆ θ = θ t +1 − θ t = − λ g D ( θ ) .
Revised: In one training step, θ is updatedby ∆ θ = θ t +1 − θ t = − λ g D ( θ ) .

Original: Using linear approximation with λ → 0 , we have∆ g ( l ) D,s,
Revised: Using linear approximation with λ → 0 , we have∆ g ( l ) D [Equation] s

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where θ ( l − ) and θ ( l +) denote model parameters before and after the l -thelayer
Revised: where θ ( l − ) and θ ( l +) denote model parameters before and after the l -the

Original: We focus on the ﬁrst term of eq.
Revised: We focus on the ﬁrst term of eq.

Original: (27), i.e. the one-step change of g ( l ) D caused by learning θ ( l − ) .
Revised: (27), i.e. the one-step change of g ( l ) D caused by learning θ ( l − ) .

Original: Substituting g ( l ) D = 1 n (cid:80) ni =
Revised: Substituting g ( l ) D = 1 n (cid:80) ni =

Original: L i ∂ W ( l ) and ∆ θ j = ( − λ 1 n (cid:80) ni
Revised: L i ∂ W ( l ) and ∆ θ j = ( − λ 1 n (cid:80) ni

Original: =1 ∂L i ∂θ j ) into eq.
Revised: =1 ∂L i ∂θ j ) into eq.

Original: (27), we have
Revised: (27), we have

Original: [Equation]
Revised: [Equation]

Original: The detailed derivation of eq.
Revised: The detailed derivation of eq.

Original: (28) can be found in Appendix B. We can see the ﬁrst term (which is asummation over parameters in θ ( l − ) ) in eq.
Revised: (28) can be found in Appendix B. We can see the ﬁrst term (which is asummation over parameters in θ ( l − ) ) in eq.

Original: (28) has opposite sign with W ( l ) s,c .
Revised: (28) has opposite sign with W ( l ) s,c .

Original: This term will make∆ g ( l ) D,s,c negatively correlated with W ( l ) s,c .
Revised: This term will make∆ g ( l ) D,s,c negatively correlated with W ( l ) s,c .

Original: We plot the correlation between ∆ g ( l ) D,s,c with W ( l ) s,c for amodel trained on MNIST for 200 epochs in Figure 4a.
Revised: We plot the correlation between ∆ g ( l ) D,s,c with W ( l ) s,c for amodel trained on MNIST for 200 epochs in Figure 4a.

Original: In the early training stage, they are indeednegatively correlated.
Revised: In the early training stage, they are indeednegatively correlated.

Original: For top-10% weights with larger absolute values, the negative correlation iseven more signiﬁcant.
Revised: For top-10% weights with larger absolute values, the negative correlation iseven more signiﬁcant.

Original: Here we show that this negative correlation between ∆ g ( l ) D,s,c and W ( l ) s,c tends to increase theabsolute value of g ( l ) D through an interesting mechanism.
Revised: Here we show that this negative correlation between ∆ g ( l ) D,s,c and W ( l ) s,c tends to increase theabsolute value of g ( l ) D through an interesting mechanism.

Original: Consider the weights W ( l ) s,c with{ W ( l ) s,c > 0 , g ( l ) D,s,c < 0 } .
Revised: Consider the weights W ( l ) s,c with{ W ( l ) s,c > 0 , g ( l ) D,s,c < 0 } .

Original: Learning θ l − would decrease g ( l ) D,s,c and thus increase its absolutevalue because the ﬁrst term in eq.
Revised: Learning θ l − would decrease g ( l ) D,s,c and thus increase its absolutevalue because the ﬁrst term in eq.

Original: (28) is negative.
Revised: (28) is negative.

Original: On the other hand, learning W ( l ) s,c would increase
Revised: On the other hand, learning W ( l ) s,c would increase

Original: [Equation]
Revised: [Equation]

Original: ) s,c and its absolute value because ∆ W ( l ) s,c = − λ g ( l ) D,s,c is positive.
Revised: ) s,c and its absolute value because ∆ W ( l ) s,c = − λ g ( l ) D,s,c is positive.

Original: This will form a positivefeedback process, in which the numerator of GSNR, ( g ( l ) D,s,c ) 2 , would increase and so is the GSNR.
Revised: This will form a positivefeedback process, in which the numerator of GSNR, ( g ( l ) D,s,c ) 2 , would increase and so is the GSNR.

Original: Similar analysis can be done for the case with { W ( l ) s,c < 0 , g ( l ) D,s,c > 0 } .
Revised: Similar analysis can be done for the case with { W ( l ) s,c < 0 , g ( l ) D,s,c > 0 } .

Original: On the other hand, when { W ( l ) s,c g ( l ) D,s,c > 0 } , we show that the weights tend to change into the earliercase, i.e. { W ( l ) s,c g ( l ) D,s,c < 0 } during training.
Revised: On the other hand, when { W ( l ) s,c g ( l ) D,s,c > 0 } , we show that the weights tend to change into the earliercase, i.e. { W ( l ) s,c g ( l ) D,s,c < 0 } during training.

Original: Consider the case of { W ( l ) s,c > 0 , g ( l ) D,s,c > 0 } , the ﬁrstterm in eq.
Revised: Consider the case of { W ( l ) s,c > 0 , g ( l ) D,s,c > 0 } , the ﬁrstterm in eq.

Original: (28) is negative, learning θ l − tends to decrease g ( l ) D,s,c or even change its sign.
Revised: (28) is negative, learning θ l − tends to decrease g ( l ) D,s,c or even change its sign.

Original: Anotherposibility is that learning W ( l ) s,
Revised: Anotherposibility is that learning W ( l ) s,

Original: c changes the sign of W ( l ) s,c because ∆ W ( l ) s,c = − λ g ( l ) D,s,c is negative.
Revised: c changes the sign of W ( l ) s,c because ∆ W ( l ) s,c = − λ g ( l ) D,s,c is negative.

Original: In both cases the weights change into the earlier case with { W ( l ) s,c g ( l ) D,s,c < 0 } .
Revised: In both cases the weights change into the earlier case with { W ( l ) s,c g ( l ) D,s,c < 0 } .

Original: Similar analysis canbe done for the case of { W ( l ) s,c < 0 , g ( l ) D,s,c < 0 } .
Revised: Similar analysis canbe done for the case of { W ( l ) s,c < 0 , g ( l ) D,s,c < 0 } .

Original: Therefore { W ( l ) s,c g
Revised: Therefore { W ( l ) s,c g

Original: ( l ) D,s,c < 0 } is a more stable state in the training process.
Revised: ( l ) D,s,c < 0 } is a more stable state in the training process.

Original: For a simple model trainedon MNIST, We plot the proportion of weights satisfying { W ( l ) s,c g ( l ) D,s,c < 0 } in Figure 4b and ﬁndthat there are indeed more weights with { W ( l ) s,c g ( l ) D,s,c < 0 } than the opposite.
Revised: For a simple model trainedon MNIST, We plot the proportion of weights satisfying { W ( l ) s,c g ( l ) D,s,c < 0 } in Figure 4b and ﬁndthat there are indeed more weights with { W ( l ) s,c g ( l ) D,s,c < 0 } than the opposite.

Original: Because weights withsmall absolute value easily change sign during training, we also plot this proportion for the top-10%weights with larger absolute values.
Revised: Because weights withsmall absolute value easily change sign during training, we also plot this proportion for the top-10%weights with larger absolute values.

Original: We can see that for the weights with large absolute values, nearly80% of them have opposite signs with their gradient mean, conﬁrming our earlier analysis.
Revised: We can see that for the weights with large absolute values, nearly80% of them have opposite signs with their gradient mean, conﬁrming our earlier analysis.

Original: For theseweights, the numerator of GSNR, ( g ( l ) D,s,c ) 2 , tends to increase through the positive feedback processas discussed above.
Revised: For theseweights, the numerator of GSNR, ( g ( l ) D,s,c ) 2 , tends to increase through the positive feedback processas discussed above.

Original: [Table]
Revised: [Table]

Original:  S UMMARY
Revised:  S UMMARY

Original: In this paper we performed a series of analysis on the role of model parameters’ GSNR in deepneural networks’ generalization ability.
Revised: In this paper we performed a series of analysis on the role of model parameters’ GSNR in deepneural networks’ generalization ability.

Original: We showed that large GSNR is a key to small generalizationgap, and gradient descent training naturally incurs and exploits large GSNR as the model discoversuseful features in learning
Revised: We showed that large GSNR is a key to small generalizationgap, and gradient descent training naturally incurs and exploits large GSNR as the model discoversuseful features in learning

Original: A A PPENDIX AA. M ODEL S TRUCTURE IN S ECTION 2.4
Revised: A A PPENDIX AA. M ODEL S TRUCTURE IN S ECTION 2.4

Original: As shown in Table 2, all models in the experiment consist of 2 Conv-Relu-MaxPooling blocks and 2fully-connected layers, but they are different in the number of channels.
Revised: As shown in Table 2, all models in the experiment consist of 2 Conv-Relu-MaxPooling blocks and 2fully-connected layers, but they are different in the number of channels.

Original: We choose the number ofchannels p from { 6 , 8 , 10 , 12 , 14 , 16 , 18 , 20 } .
Revised: We choose the number ofchannels p from { 6 , 8 , 10 , 12 , 14 , 16 , 18 , 20 } .

Original: [Table]
Revised: [Table]

Original: A. E XPERIMENT ON CIFAR10
Revised: A. E XPERIMENT ON CIFAR10

Original: Different from the experiment on MNIST, we use a deeper network on CIFAR10.
Revised: Different from the experiment on MNIST, we use a deeper network on CIFAR10.

Original: We also includethe Batch Normalization (BN) layer, because we ﬁnd that it’s difﬁcult for the network to converge inthe absence of it.
Revised: We also includethe Batch Normalization (BN) layer, because we ﬁnd that it’s difﬁcult for the network to converge inthe absence of it.

Original: The network consists of 4 Conv-BN-Relu-Conv-BN-Relu-MaxPooling blocks and3 fully-connected layers.
Revised: The network consists of 4 Conv-BN-Relu-Conv-BN-Relu-MaxPooling blocks and3 fully-connected layers.

Original: More details are shown in Table 3.
Revised: More details are shown in Table 3.

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: The experiment is conducted under a similar setting as that of MNIST in section 2.4.
Revised: The experiment is conducted under a similar setting as that of MNIST in section 2.4.

Original: We choosen ∈ { 2000 , 4000 , 6000 , 8000 , 10000 } , p random ∈ { 0 . 0 , 0 .
Revised: We choosen ∈ { 2000 , 4000 , 6000 , 8000 , 10000 } , p random ∈ { 0 . 0 , 0 .

Original: 2 , 0 . 4 } , ch ∈ { 6 , 8 , 10 , 12 , 14 , 16 , 18 } .
Revised: 2 , 0 . 4 } , ch ∈ { 6 , 8 , 10 , 12 , 14 , 16 , 18 } .

Original: We use the gradient descent training (Not SGD), with a small learning rate of 0 . 001 .
Revised: We use the gradient descent training (Not SGD), with a small learning rate of 0 . 001 .

Original: The left andright hand sides of 19 at different epochs are shown in Figure 5, where each point represents onespeciﬁc combination of the above settings.
Revised: The left andright hand sides of 19 at different epochs are shown in Figure 5, where each point represents onespeciﬁc combination of the above settings.

Original: Note that at the evaluation step of every epoch, we use the same mean and variance inside the BN layers as the training datasetThat’s to ensure thenetwork and loss function are consistent between training and test.
Revised: Note that at the evaluation step of every epoch, we use the same mean and variance inside the BN layers as the training dataset to ensure the network and loss function are consistent between training and test.

Original: At the beginning of training, compared to that of MNIST, the data points no longer perfectly resideson the diagonal dashed line.
Revised: At the beginning of training, compared to that of MNIST, the data points no longer perfectly resideson the diagonal dashed line.

Original: We suppose that’s beacuse of the presence of BN layer, whose internalparameters, i.e. running mean and running variance, are not regular learnable parameters in theoptimization process, but change their values in a different way.
Revised: We suppose that’s beacuse of the presence of BN layer, whose internalparameters, i.e. running mean and running variance, are not regular learnable parameters in theoptimization process, but change their values in a different way.

Original: Their change affects the OSGR,yet we could not include them in the estimation of OSGR.
Revised: Their change affects the OSGR,yet we could not include them in the estimation of OSGR.

Original: However, the strong positive correlationbetween the left and right hand sides of eq.
Revised: However, the strong positive correlationbetween the left and right hand sides of eq.

Original: (19) can always be observed until the training begins toconverge.
Revised: (19) can always be observed until the training begins toconverge.

Original: A. E XPERIMENT ON T OY D ATASET
Revised: A. E XPERIMENT ON T OY D ATASET

Original: In this section we show a simple two-layer regression model consists of a FC-Relu structure with only2 inputs, 1 hidden layer with N neurons and 1 output.
Revised: In this section we show a simple two-layer regression model consists of a FC-Relu structure with only2 inputs, 1 hidden layer with N neurons and 1 output.

Original: A similar synthetic dataset with the trainingdata used in the experiment of Section 3.2 is generated as follows.
Revised: A similar synthetic dataset with the trainingdata used in the experiment of Section 3.2 is generated as follows.

Original: Each data point is constructedi.i.d.
Revised: Each data point is constructedi.i.d.

Original: using y = x 0 x 1 + (cid:15) , where x 0 and x 1 are drawn from uniform distribution of [ − 1 , 1] and (cid:15) isdrawn from uniform distribution of [ − η noise , η noise ] .
Revised: using y = x 0 x 1 + (cid:15) , where x 0 and x 1 are drawn from uniform distribution of [ − 1 , 1] and (cid:15) isdrawn from uniform distribution of [ − η noise , η noise ] .

Original: To estimate eq.
Revised: To estimate eq.

Original: (24), we randomly generate 100 training sets with n samples each, i.e. M =100,and a test set with 20,000 samples.
Revised: (24), we randomly generate 100 training sets with n samples each, i.e. M =100,and a test set with 20,000 samples.

Original: To cover different conditions, we (1) choose n ∈{ 50 , 100 , 300 , 600 , 1000 , 2000 , 6000 } ; (2) inject noise with η noise ∈ { 0 . 2 , 2 , 4 , 6 , 8 } ; (3) perturbmodel structures by choosing N ∈ { 6 , 8 , 10 , 12 , 14 , 16 , 18 , 20 } .
Revised: To cover different conditions, we (1) choose n ∈{ 50 , 100 , 300 , 600 , 1000 , 2000 , 6000 } ; (2) inject noise with η noise ∈ { 0 . 2 , 2 , 4 , 6 , 8 } ; (3) perturbmodel structures by choosing N ∈ { 6 , 8 , 10 , 12 , 14 , 16 , 18 , 20 } .

Original: We use gradient descent withlearning rate of 0.001.
Revised: We use gradient descent withlearning rate of 0.001.

Original: Figure 6 shows a similar behavior as Fig. 1.
Revised: Figure 6 shows a similar behavior as Fig. 1.

Original: During the early training stages, the LHS and RHS ofeq.
Revised: During the early training stages, the LHS and RHS ofeq.

Original: (19) are very close.
Revised: (19) are very close.

Original: Their highly correlated relation remains until training converges, whereas the
Revised: Their highly correlated relation remains until training converges, whereas the

Original: RHS of eq.
Revised: RHS of eq.

Original: (19) decreases signiﬁcantly.
Revised: (19) decreases signiﬁcantly.

Original: B A PPENDIX B
Revised: B A PPENDIX B

Original: Derivation of eq. (28)
Revised: Derivation of eq. (28)

Original: [Equation]
Revised: [Equation]

Revised: + other terms (33)

Original: i = [Equation] ( l ) i
Revised: Above we used ∂o ( l ) i

Original: [Equation]
Revised: [Equation]

Original: ( l )
Revised: c

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: ∂a

Original: [Equation]
Revised: [Equation]

Original: c
Revised: l

Original: [Equation]
Revised: [Equation]

Original: ) + other terms (33) Above we used ∂o ( l ) i,c (cid:48) ∂a ( l ) i,s (cid:48)
Revised: i [Equation]

Original: = 
Revised: = W [Equation]

Original: W ( l ) s (cid:48) ,c (cid:48) and ∂o ( l ) i,c
Revised: l ) s (cid:48) ,c (cid:48) and ∂o ( l ) i,c

Original: ∂ W ( l ) s,c = a ( l ) i,s that can both be derived from eq.
Revised: ∂ W ( l ) s,c = a ( l ) i,s that can both be derived from eq.

Original: Considerthe ﬁrst term of eq.
Revised: Considerthe ﬁrst term of eq.

Original: When s (cid:48)
Revised: When s (cid:48)

Original: =
Revised: =

Original: s, c (cid:48)
Revised: s, c (cid:48)

Original: =
Revised: =

Original: c , we have
Revised: c , we have

Original: [Equation]
Revised: [Equation]

Revised: other terms (34)

Original: Note that the term related to ∂ 2 L i ∂o ( l ) i,c
Revised: Note that the term related to ∂ 2 L i ∂o ( l ) i,c

Original: ∂θ j a ( l ) i,s and the terms when s (cid:48) (cid:54) =
Revised: ∂θ j a ( l ) i,s and the terms when s (cid:48) (cid:54) =

Original: s or c (cid:48) (cid:54)
Revised: s or c (cid:48) (cid:54)

Original: =
Revised: =

Original: c in eq.
Revised: c in eq.

Original: (33) are mergedinto other terms of eq.
Revised: (33) are mergedinto other terms of eq.

Original: [Table]

Original: Notations

Original: [Table]
Revised: [Table]

Original: Training set consists of n samples drawn from Z [Table] Test set consists of n (cid:48) samples drawn from Z
Revised: parameters

Original: [Table]
Revised: [Table]

Original: Model parameters, whose components are denoted as θj
Revised: components are denoted as θ

Original: [Table]
Revised: [Table]

Original: Parameters’ gradient w.r.t. a single data sample s or ( x i , y i ) [Table] values of 
Revised: Gradient signal to noise ratio (GSNR) of model parameter θ

Original: [Table]

Original: [Table]

Original: Note that,

Original: [Table]
Revised: [Table]

Original: (19), we assume n

Original: [Table]

Original: n

Original: [Table]
Revised: [Table]

Original: Variance of parameters’ gradient of a single sample, i [Table] e. ,
Revised: Random

Original: [Table]

Original: Same

Original: [Table]
Revised: [Table]

Original: Variance of the average gradient over a training dataset of size n ,

Original: [Table] Same as [Table] Gradient signal to noise ratio (GSNR) of modelparameter θ j
Revised: model

Original: [Table]
Revised: [Table]

Original: Empirical training loss,

Original: i

Original: [Table]

Original: e

Original: [Table]

Original: [Table] Empirical test loss, [Table] [Table]
Revised: (weight matrix and bias)

Original: One-

Original: [Table]
Revised: [Table]

Original: generalization ratio (OSGR) for a training and test sets of size n sampled from data distribution Z [Table] i [Table]
Revised: Collection of model parameters over all the layers before

Original: e

Original: [Table]
Revised: [Table]

Original: model [Table] (weight matrix [Table]
Revised: -th

Original: [Table]
Revised: [Table]

Original: Collection of model parameters over all the layers before
Revised: Average gradient of W

Original: [Table]
Revised: [Table]

Original: -th
Revised: l

Original: [Table]
Revised: [Table]

Original: Average gradient of W ( l ) over the training dataset
Revised: over the training dataset

Original: [Table]
Revised: [Table]

Original: Collection of model parameters over all the layers aftering the l -th layer, including the l -th layer
Revised: Collection of model parameters over all the layers aftering the l -th layer, including the l -th layer

Original: [Table]
Revised: [Table]

Original: Activations of the l -th layer, where s = { 1 , 2 , ..., S } is index of nodes/channels in the l -th layer.
Revised: Activations of the l -th layer, where s = { 1 , 2 , ..., S } is index of nodes/channels in the l -th layer.

Original: [Table]
Revised: [Table]

Original: Outputs of matrix multiplication of the l -th layer, where c = { 1 , 2 , ..., C } is index of nodes/channels in the l + 1 -th layer.[Table] a
Revised: Outputs of matrix multiplication of the l -th layer, where c = { 1 , 2 , ..., C } is index of nodes/channels in the l + 1 -th layer.

Original: [Table]
Revised: [Table]

Original: evaluated on data sample i
Revised: evaluated on data sample i