Original: T HEORETICAL A NALYSIS OF A UTO R ATE - TUNING BY B ATCH N ORMALIZATION
Revised: T HEORETICAL A NALYSIS OF A UTO R ATE - TUNING BY B ATCH N ORMALIZATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Batch Normalization (BN) has become a cornerstone of deep learning across diverse architectures, appearing to help optimization as well as generalization.
Revised: Batch Normalization (BN) has become a cornerstone of deep learning across diverse architectures, appearing to help optimization as well as generalization.

Original: While the idea makes intuitive sense, theoretical analysis of its effectiveness has been lacking.
Revised: While the idea makes intuitive sense, theoretical analysis of its effectiveness has been lacking.

Original: Here theoretical support is provided for one of its conjectured properties, namely, the ability to allow gradient descent to succeed with less tuning of learning rates.
Revised: Here theoretical support is provided for one of its conjectured properties, namely, the ability to allow gradient descent to succeed with less tuning of learning rates.

Original: It is shown that even if we ﬁx the learning rate of scale-invariant parameters (e.g., weights of each layer with BN) to a constant (say, 0 . 3 ), gradient descent still approaches a stationary point (i.e., a solution where gradient is zero) in the rate of T − 1 / 2 in T iterations, asymptotically matching the best bound for gradient descent with well-tuned learning rates.
Revised: It is shown that even if we ﬁx the learning rate of scale-invariant parameters (e.g., weights of each layer with BN) to a constant (say, 0 . 3 ), gradient descent still approaches a stationary point (i.e., a solution where gradient is zero) in the rate of T − 1 / 2 in T iterations, asymptotically matching the best bound for gradient descent with well-tuned learning rates.

Original: A similar result with convergence rate T − 1 / 4 is also shown for stochastic gradient descent.
Revised: A similar result with convergence rate T − 1 / 4 is also shown for stochastic gradient descent.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Batch Normalization (abbreviated as BatchNorm or BN) (Ioffe & Szegedy, 2015) is one of the most important innovation in deep learning, widely used in modern neural network architectures such as
Revised: Batch Normalization (abbreviated as BatchNorm or BN) (Ioffe & Szegedy, 2015) is one of the most important innovation in deep learning, widely used in modern neural network architectures such as

Original: ResNet (He et al., 2016), Inception (Szegedy et al., 2017), and DenseNet (Huang et al., 2017).
Revised: ResNet (He et al., 2016), Inception (Szegedy et al., 2017), and DenseNet (Huang et al., 2017).

Original: It also inspired a series of other normalization methods (Ulyanov et al., 2016; Ba et al., 2016; Ioffe, 2017; Wu & He, 2018).
Revised: It also inspired a series of other normalization methods (Ulyanov et al., 2016; Ba et al., 2016; Ioffe, 2017; Wu & He, 2018).

Original: BatchNorm consists of standardizing the output of each layer to have zero mean and unit variance.
Revised: BatchNorm consists of standardizing the output of each layer to have zero mean and unit variance.

Original: For a single neuron, if x 1 , . . .
Revised: For a single neuron, if x 1 , . . .

Original: , x B is the original outputs in a mini-batch, then it adds a BatchNorm layer which modiﬁes the outputs to
Revised: , x B is the original outputs in a mini-batch, then it adds a BatchNorm layer which modiﬁes the outputs to

Original: [Equation]
Revised: [Equation]

Original: where µ = 1 B (cid:80) Bi
Revised: where µ = 1 B (cid:80) Bi

Original: =1 x i and σ 2 = 1 B (cid:80) Bi
Revised: =1 x i and σ 2 = 1 B (cid:80) Bi

Original: =1 ( x i − µ ) 2 are the mean and variance within the minibatch, and γ, β are two learnable parameters.
Revised: =1 ( x i − µ ) 2 are the mean and variance within the minibatch, and γ, β are two learnable parameters.

Original: BN appears to stabilize and speed up training, and improve generalization.
Revised: BN appears to stabilize and speed up training, and improve generalization.

Original: The inventors suggested (Ioffe & Szegedy, 2015) that these beneﬁts derive from the following:
Revised: The inventors suggested (Ioffe & Szegedy, 2015) that these beneﬁts derive from the following:

Original: By stabilizing layer outputs it reduces a phenomenon called Internal Covariate Shift , whereby the training of a higher layer is continuously undermined or undone by changes in the distribution of its inputs due to parameter changes in previous layers.
Revised: By stabilizing layer outputs it reduces a phenomenon called Internal Covariate Shift , whereby the training of a higher layer is continuously undermined or undone by changes in the distribution of its inputs due to parameter changes in previous layers.

Original: Making the weights invariant to scaling, appears to reduce the dependence of training on the scale of parameters and enables us to use a higher learning rate; 3.
Revised: Making the weights invariant to scaling, appears to reduce the dependence of training on the scale of parameters and enables us to use a higher learning rate; 3.

Original: By implictly regularizing the model it improves generalization.
Revised: By implictly regularizing the model it improves generalization.

Original: But these three beneﬁts are not fully understood in theory.
Revised: But these three beneﬁts are not fully understood in theory.

Original: Understanding generalization for deep models remains an open problem (with or without BN).
Revised: Understanding generalization for deep models remains an open problem (with or without BN).

Original: Furthermore, in demonstration that intuition can sometimes mislead, recent experimental results suggest that BN does not reduce internal covariate shift either (Santurkar et al., 2018), and the authors of that study suggest that the true explanation for BN’s effectiveness may lie in a smoothening effect (i.e., lowering of the Hessian norm) on the objective.
Revised: Furthermore, in demonstration that intuition can sometimes mislead, recent experimental results suggest that BN does not reduce internal covariate shift either (Santurkar et al., 2018), and the authors of that study suggest that the true explanation for BN’s effectiveness may lie in a smoothening effect (i.e., lowering of the Hessian norm) on the objective.

Original: Another recent paper (Kohler et al., 2018) tries to quantify the beneﬁts of BN for simple machine learning problems such as regression but does not analyze deep models.
Revised: Another recent paper (Kohler et al., 2018) tries to quantify the beneﬁts of BN for simple machine learning problems such as regression but does not analyze deep models.

Original: Provable quantiﬁcation of Effect 2 (learning rates).
Revised: Provable quantiﬁcation of Effect 2 (learning rates).

Original: Our study consists of quantifying the effect of BN on learning rates.
Revised: Our study consists of quantifying the effect of BN on learning rates.

Original: Ioffe & Szegedy (2015) observed that without BatchNorm, a large learning rate leads to a rapid growth of the parameter scale.
Revised: Ioffe & Szegedy (2015) observed that without BatchNorm, a large learning rate leads to a rapid growth of the parameter scale.

Original: Introducing BatchNorm usually stabilizes the growth of weights and appears to implicitly tune the learning rate so that the effective learning rate adapts during the course of the algorithm.
Revised: Introducing BatchNorm usually stabilizes the growth of weights and appears to implicitly tune the learning rate so that the effective learning rate adapts during the course of the algorithm.

Original: They explained this intuitively as follows.
Revised: They explained this intuitively as follows.

Original: After BN the output of a neuron z = BN( w (cid:62) x ) is unaffected when the weight w is scaled, i.e., for any scalar c > 0 ,
Revised: After BN the output of a neuron z = BN( w (cid:62) x ) is unaffected when the weight w is scaled, i.e., for any scalar c > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Taking derivatives one ﬁnds that the gradient at c w equals to the gradient at w multiplied by a factor 1 /c .
Revised: Taking derivatives one ﬁnds that the gradient at c w equals to the gradient at w multiplied by a factor 1 /c .

Original: Thus, even though the scale of weight parameters of a linear layer proceeding a BatchNorm no longer means anything to the function represented by the neural network, their growth has an effect of reducing the learning rate.
Revised: Thus, even though the scale of weight parameters of a linear layer proceeding a BatchNorm no longer means anything to the function represented by the neural network, their growth has an effect of reducing the learning rate.

Original: Our paper considers the following question: Can we rigorously capture the above intuitive behavior?
Revised: Our paper considers the following question: Can we rigorously capture the above intuitive behavior?

Original: Theoretical analyses of speed of gradient descent algorithms in nonconvex settings study the number of iterations required for convergence to a stationary point (i.e., where gradient vanishes).
Revised: Theoretical analyses of speed of gradient descent algorithms in nonconvex settings study the number of iterations required for convergence to a stationary point (i.e., where gradient vanishes).

Original: But they need to assume that the learning rate has been set (magically) to a small enough number determined by the smoothness constant of the loss function — which in practice are of course unknown.
Revised: But they need to assume that the learning rate has been set (magically) to a small enough number determined by the smoothness constant of the loss function — which in practice are of course unknown.

Original: With this tuned learning rate, the norm of the gradient reduces asymptotically as T − 1 / 2 in T iterations.
Revised: With this tuned learning rate, the norm of the gradient reduces asymptotically as T − 1 / 2 in T iterations.

Original: In case of stochastic gradient descent, the reduction is like T − 1 / 4 .
Revised: In case of stochastic gradient descent, the reduction is like T − 1 / 4 .

Original: Thus a potential way to quantify the rate-tuning behavior of BN would be to show that even when the learning rate is ﬁxed to a suitable constant, say 0 .
Revised: Thus a potential way to quantify the rate-tuning behavior of BN would be to show that even when the learning rate is ﬁxed to a suitable constant, say 0 .

Original: 1 , from the start, after introducing BN the convergence to stationary point is asymptotically just as fast (essentially) as it would be with a hand-tuned learning rate required by earlier analyses.
Revised: 1 , from the start, after introducing BN the convergence to stationary point is asymptotically just as fast (essentially) as it would be with a hand-tuned learning rate required by earlier analyses.

Original: The current paper rigorously establishes such auto-tuning behavior of BN (See below for an important clariﬁcation about scale-invariance).
Revised: The current paper rigorously establishes such auto-tuning behavior of BN (See below for an important clariﬁcation about scale-invariance).

Original: We note that a recent paper (Wu et al., 2018) introduced a new algorithm WNgrad that is motivated by BN and provably has the above auto-tuning behavior as well.
Revised: We note that a recent paper (Wu et al., 2018) introduced a new algorithm WNgrad that is motivated by BN and provably has the above auto-tuning behavior as well.

Original: That paper did not establish such behavior for BN itself, but it was a clear inspiration for our analysis of BN.
Revised: That paper did not establish such behavior for BN itself, but it was a clear inspiration for our analysis of BN.

Original: Scale-invariant and scale-variant parameters.
Revised: Scale-invariant and scale-variant parameters.

Original: The intuition of Ioffe & Szegedy (2015) applies for all scale-invariant parameters, but the actual algorithm also involves other parameters such as γ and β whose scale does matter.
Revised: The intuition of Ioffe & Szegedy (2015) applies for all scale-invariant parameters, but the actual algorithm also involves other parameters such as γ and β whose scale does matter.

Original: Our analysis partitions the parameters in the neural networks into two groups W ( scale-invariant ) and g ( scale-variant ).
Revised: Our analysis partitions the parameters in the neural networks into two groups W ( scale-invariant ) and g ( scale-variant ).

Original: The ﬁrst group, W = { w (1) , . . .
Revised: The ﬁrst group, W = { w (1) , . . .

Original: , w ( m ) } , consists of all the parameters whose scales does not affect the loss, i.e., scaling w ( i ) to c w ( i ) for any c > 0 do not change the loss (see Deﬁnition 2.1 for a formal deﬁnition); the second group, g , consists of all other parameters that are not scale-invariant.
Revised: , w ( m ) } , consists of all the parameters whose scales does not affect the loss, i.e., scaling w ( i ) to c w ( i ) for any c > 0 does not change the loss (see Deﬁnition 2.1 for a formal deﬁnition); the second group, g , consists of all other parameters that are not scale-invariant.

Original: In a feedforward neural network with
Revised: In a feedforward neural network with

Original: BatchNorm added at each layer, the layer weights are  scale-invariant.
Revised: BN added at each layer, the layer weights are all scale-invariant.

Original: And this fact is also true for  Weight Normalization (Salimans & Kingma, 2016), Layer Normalization (Ba et al., 2016),Group
Revised: This is also true for BN with (cid:96) p normalization strategies (Santurkar et al., 2018; Hoffer et al., 2018) and other normalization layers, such as Weight Normalization (Salimans & Kingma, 2016), Layer Normalization (Ba et al., 2016),

Original:  Normalization (Wu & He, 2018) (see Table 1 in Ba et al.
Revised: Group Normalization (Wu & He, 2018) (see Table 1 in Ba et al.

Original: (2016) for a summary).
Revised: (2016) for a summary).

Original: 1. O UR CONTRIBUTIONS
Revised: 1. O UR CONTRIBUTIONS

Original: In this paper, we show that the scale-invariant parameters do not require rate tuning for lowering the training loss.
Revised: In this paper, we show that the scale-invariant parameters do not require rate tuning for lowering the training loss.

Original: To illustrate this, we consider the case in which we set learning rates separately for scale-invariant parameters W and scale-variant parameters g .
Revised: To illustrate this, we consider the case in which we set learning rates separately for scale-invariant parameters W and scale-variant parameters g .

Original: Under some assumptions on the smoothness of the loss and the boundedness of the noise, we show that
Revised: Under some assumptions on the smoothness of the loss and the boundedness of the noise, we show that

Original: In full-batch gradient descent, if the learning rate for g is set optimally, then no matter how the learning rates for W is set, ( W ; g ) converges to a ﬁrst-order stationary point in the rateO
Revised: In full-batch gradient descent, if the learning rate for g is set optimally, then no matter how the learning rates for W is set, ( W ; g ) converges to a ﬁrst-order stationary point in the rate

Original:  ( T − 1 / 2 ) , which asymptotically matches with the convergence rate of gradient descent with optimal choice of learning rates for all parameters (Theorem 3.1); 2.
Revised: O ( T − 1 / 2 ) , which asymptotically matches with the convergence rate of gradient descent with optimal choice of learning rates for all parameters (Theorem 3.1); 2.

Original: In stochastic gradient descent, if the learning rate for g is set optimally, then no matter how the learning rate for W is set, ( W ; g ) converges to a ﬁrst-order stationary point in the rate O
Revised: In stochastic gradient descent, if the learning rate for g is set optimally, then no matter how the learning rate for W is set, ( W ; g ) converges to a ﬁrst-order stationary point in the rate O

Original: ( T − 1 / 4 polylog( T )) , which asymptotically matches with the convergence rate of gradient descent with optimal choice of learning rates for all parameters (up to a polylog( T ) factor) (Theorem 4.2).
Revised: ( T − 1 / 4 polylog( T )) , which asymptotically matches with the convergence rate of gradient descent with optimal choice of learning rates for all parameters (up to a polylog( T ) factor) (Theorem 4.2).

Original: In the usual case where we set a uniﬁed learning rate for all parameters, our results imply that we only need to set a learning rate that is suitable for g .
Revised: In the usual case where we set a uniﬁed learning rate for all parameters, our results imply that we only need to set a learning rate that is suitable for g .

Original: This means introducing scale-invariance into neural networks potentially reduces the efforts to tune learning rates, since there are less number of parameters we need to concern in order to guarantee an asymptotically fastest convergence.
Revised: This means introducing scale-invariance into neural networks potentially reduces the efforts to tune learning rates, since there are less number of parameters we need to concern in order to guarantee an asymptotically fastest convergence.

Original: In our study, the loss function is assumed to be smooth.
Revised: In our study, the loss function is assumed to be smooth.

Original: However, BN introduces non-smoothness in extreme cases due to division by zero when the input variance is zero (see equation 1).
Revised: However, BN introduces non-smoothness in extreme cases due to division by zero when the input variance is zero (see equation 1).

Original: Note that the suggested implementation of BN by Ioffe & Szegedy (2015) uses a smoothening constant in the whitening step, but it does not preserve scale-invariance.
Revised: Note that the suggested implementation of BN by Ioffe & Szegedy (2015) uses a smoothening constant in the whitening step, but it does not preserve scale-invariance.

Original: In order to avoid this issue, we describe a simple modiﬁcation of the smoothening that maintains scale-invariance.
Revised: In order to avoid this issue, we describe a simple modiﬁcation of the smoothening that maintains scale-invariance.

Original: Also, our result cannot be applied to neural networks with ReLU, but it is applicable for its smooth approximation
Revised: Also, our result cannot be applied to neural networks with ReLU, but it is applicable for its smooth approximation

Original: softplus (Dugas et al., 2001).
Revised: softplus (Dugas et al., 2001).

Original: We include some experiments in Appendix D, showing that it is indeed the auto-tuning behavior we analysed in this paper empowers BN to have such convergence with arbitrary learning rate for scale-invariant parameters.
Revised: We include some experiments in Appendix D, showing that it is indeed the auto-tuning behavior we analysed in this paper empowers BN to have such convergence with arbitrary learning rate for scale-invariant parameters.

Original: In the generalization aspect, a tuned learning rate is still needed for the best test accuracy, and we showed in the experiments that the auto-tuning behavior of BN also leads to a wider range of suitable learning rate for good generalization.
Revised: In the generalization aspect, a tuned learning rate is still needed for the best test accuracy, and we showed in the experiments that the auto-tuning behavior of BN also leads to a wider range of suitable learning rate for good generalization.

Original: 1. R ELATED WORKS
Revised: 1. R ELATED WORKS

Original: Previous work for understanding Batch Normalization.
Revised: Previous work for understanding Batch Normalization.

Original: Only a few recent works try to theoretically understand BatchNorm.
Revised: Only a few recent works tried to theoretically understand BatchNorm.

Original: Santurkar et al.
Revised: Santurkar et al.

Original: (2018) was described earlier.
Revised: (2018) was described earlier.

Original: Kohler et al.
Revised: Kohler et al.

Original: (2018) aims to ﬁnd theoretical setting such that training neural networks with BatchNorm is faster than without BatchNorm.
Revised: (2018) aims to ﬁnd theoretical setting such that training neural networks with BatchNorm is faster than without BatchNorm.

Original: In particular, the authors analyzed three types of shallow neural networks, but rather than consider gradient descent, the authors designed task-speciﬁc training methods when discussing neural networks with BatchNorm.
Revised: In particular, the authors analyzed three types of shallow neural networks, but rather than consider gradient descent, the authors designed task-speciﬁc training methods when discussing neural networks with BatchNorm.

Original: Bjorck et al.
Revised: Bjorck et al.

Original: (2018) observes that the higher learning rates enabled by BatchNorm improves generalization.
Revised: (2018) observes that the higher learning rates enabled by BatchNorm improves generalization.

Original: Convergence of adaptive algorithms.
Revised: Convergence of adaptive algorithms.

Original: Our analysis is inspired by the proof for WNGrad (Wu et al., 2018), where the author analyzed an adaptive algorithm, WNGrad, motivated by Weight
Revised: Our analysis is inspired by the proof for WNGrad (Wu et al., 2018), where the author analyzed an adaptive algorithm, WNGrad, motivated by Weight

Original: Normalization (Salimans & Kingma, 2016).
Revised: Normalization (Salimans & Kingma, 2016).

Original: Other works analyzing the convergence of adaptive methods are (Ward et al., 2018; Li & Orabona, 2018; Zou & Shen, 2018;
Revised: Other works analyzing the convergence of adaptive methods are (Ward et al., 2018; Li & Orabona, 2018; Zou & Shen, 2018;

Original: Zhou et al., 2018).
Revised: Zhou et al., 2018).

Original: Invariance by Batch Normalization.
Revised: Invariance by Batch Normalization.

Original: Cho & Lee (2017) proposed to run riemmanian gradient descent on Grassmann manifold G (1 , n ) since the weight matrix is scaling invariant to the loss function.
Revised: Cho & Lee (2017) proposed to run riemmanian gradient descent on Grassmann manifold G (1 , n ) since the weight matrix is scaling invariant to the loss function.

Original: Hoffer et al.
Revised: Hoffer et al.

Original: (2018) observed that the effective stepsize is proportional to η w (cid:107) w t (cid:107) 2 .
Revised: (2018) observed that the effective stepsize is proportional to η w (cid:107) w t (cid:107) 2 .

Original:  G ENERAL FRAMEWORK
Revised:  G ENERAL FRAMEWORK

Original: In this section, we introduce our general framework in order to study the beneﬁts of scale-invariance.
Revised: In this section, we introduce our general framework in order to study the beneﬁts of scale-invariance.

Original: 2. M OTIVATING EXAMPLES OF NEURAL NETWORKS
Revised: 2. M OTIVATING EXAMPLES OF NEURAL NETWORKS

Original: Scale-invariance is common in neural networks with BatchNorm.
Revised: Scale-invariance is common in neural networks with BatchNorm.

Original: We formally state the deﬁnition of scale-invariance below:
Revised: We formally state the deﬁnition of scale-invariance below:

Original: Deﬁnition 2.1.
Revised: Deﬁnition 2.1.

Original: (Scale-invariance) Let F ( w , θ (cid:48) ) be a loss function.
Revised: (Scale-invariance) Let F ( w , θ (cid:48) ) be a loss function.

Original: We say that w is a scale-invariant parameter of F if for all c > 0 , F ( w , θ (cid:48) )
Revised: We say that w is a scale-invariant parameter of F if for all c > 0 , F ( w , θ (cid:48) )

Original: =
Revised: =

Original: F ( c w , θ (cid:48) ) ; if w is not scale-invariant, then we say w is a scale-variant parameter of F .
Revised: F ( c w , θ (cid:48) ) ; if w is not scale-invariant, then we say w is a scale-variant parameter of F .

Original: We consider the following L -layer “fully-batch-normalized” feedforward network Φ for illustration:
Revised: We consider the following L -layer “fully-batch-normalized” feedforward network Φ for illustration:

Original: [Equation]
Revised: [Equation]

Original: Z = { ( x 1 , y 1 ) , . .
Revised: Z = { ( x 1 , y 1 ) , . .

Original: , ( x B , y B ) } is a mini-batch of B pairs of input data and ground-truth label from a data set D .
Revised: , ( x B , y B ) } is a mini-batch of B pairs of input data and ground-truth label from a data set D .

Original: f y is an objective function depending on the label, e.g., f y could be a cross-entropy loss in classiﬁcation tasks.
Revised: f y is an objective function depending on the label, e.g., f y could be a cross-entropy loss in classiﬁcation tasks.

Original: W (1) , . . .
Revised: W (1) , . . .

Original: , W ( L ) are weight matrices of each layer.
Revised: , W ( L ) are weight matrices of each layer.

Original: σ : R → R is a nonlinear activation function which processes its input elementwise (such as ReLU, sigmoid).
Revised: σ : R → R is a nonlinear activation function which processes its input elementwise (such as ReLU, sigmoid).

Original: Given a batch of inputs z 1 , . . .
Revised: Given a batch of inputs z 1 , . . .

Original: , z B ∈ R m , BN( z b ) outputs a vector ˜
Revised: , z B ∈ R m , BN( z b ) outputs a vector ˜

Original: z b deﬁned as
Revised: z b deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where µ k = E b ∈ [ B ] [ z b,k ] and σ 2 k = E b ∈ [ B ] [( z b,k − µ k ) 2 ] are the mean and variance of z b , γ k and β k are two learnable parameters which rescale and offset the normalized outputs to retain the representation power.
Revised: where µ k = E b ∈ [ B ] [ z b,k ] and σ 2 k = E b ∈ [ B ] [( z b,k − µ k ) 2 ] are the mean and variance of z b , γ k and β k are two learnable parameters which rescale and offset the normalized outputs to retain the representation power.

Original: The neural network Φ is thus parameterized by weight matrices W ( i ) in each layer and learnable parameters γ k , β k in each BN.
Revised: The neural network Φ is thus parameterized by weight matrices W ( i ) in each layer and learnable parameters γ k , β k in each BN.

Original: BN has the property that the output is unchanged when the batch inputs z 1 ,k , . . .
Revised: BN has the property that the output is unchanged when the batch inputs z 1 ,k , . . .

Original: , z B,k are scaled or shifted simultaneously.
Revised: , z B,k are scaled or shifted simultaneously.

Original: For z b,k = w (cid:62) k
Revised: For z b,k = w (cid:62) k

Original: ˆ x b being the output of a linear layer, it is easy to see that w k is scale-invariant, and thus each row vector of weight matrices W (1) , . . .
Revised: ˆ x b being the output of a linear layer, it is easy to see that w k is scale-invariant, and thus each row vector of weight matrices W (1) , . . .

Original: , W ( L ) in Φ are scaleinvariant parameters of L ( θ ) .
Revised: , W ( L ) in Φ are scaleinvariant parameters of L ( θ ) .

Original: In convolutional neural networks with BatchNorm, a similar argument can be done.
Revised: In convolutional neural networks with BatchNorm, a similar argument can be done.

Original: In particular, each ﬁlter of convolutional layer normalized by BN is scale-invariant.
Revised: In particular, each ﬁlter of convolutional layer normalized by BN is scale-invariant.

Original: With a general nonlinear activation, other parameters in Φ , the scale and shift parameters γ k and β k in each BN, are scale-variant.
Revised: With a general nonlinear activation, other parameters in Φ , the scale and shift parameters γ k and β k in each BN, are scale-variant.

Original: When ReLU or Leaky ReLU (Maas et al., 2013) are used as the activation σ , the vector ( γ 1 , . .
Revised: When ReLU or Leaky ReLU (Maas et al., 2013) are used as the activation σ , the vector ( γ 1 , . .

Original: . , γ m , β 1 , . .
Revised: . , γ m , β 1 , . .

Original: . , β m )
Revised: . , β m )

Original: of each BN
Revised: of each BN

Original: at layer 1 ≤ i < L (except the last one) is indeed scale-invariant.
Revised: at layer 1 ≤ i < L (except the last one) is indeed scale-invariant.

Original: This can be deduced by using the the (positive) homogeneity of these two types of activations and noticing that the output of internal activations is processed by a BN in the next layer.
Revised: This can be deduced by using the the (positive) homogeneity of these two types of activations and noticing that the output of internal activations is processed by a BN in the next layer.

Original: Nevertheless, we are not able to analyse either ReLU or Leaky ReLU activations because we need the loss to be smooth in our analysis.
Revised: Nevertheless, we are not able to analyse either ReLU or Leaky ReLU activations because we need the loss to be smooth in our analysis.

Original: We can instead analyse smooth activations, such as sigmoid, tanh, softplus (Dugas et al., 2001), etc.
Revised: We can instead analyse smooth activations, such as sigmoid, tanh, softplus (Dugas et al., 2001), etc.

Original: 2. F RAMEWORK
Revised: 2. F RAMEWORK

Original: Now we introduce our general framework.
Revised: Now we introduce our general framework.

Original: Let Φ be a neural network parameterized by θ .
Revised: Let Φ be a neural network parameterized by θ .

Original: Let
Revised: Let

Original: D be a dataset, where each data point z ∼ D is associated with a loss function F z ( θ ) ( D can be the set of all possible mini-batches).
Revised: D be a dataset, where each data point z ∼ D is associated with a loss function F z ( θ ) ( D can be the set of all possible mini-batches).

Original: We partition the parameters θ into ( W ; g ) , where W = { w (1) , . . .
Revised: We partition the parameters θ into ( W ; g ) , where W = { w (1) , . . .

Original: , w ( m ) } consisting of parameters that are scale-invariant to all F z , and g contains the remaining parameters.
Revised: , w ( m ) } consisting of parameters that are scale-invariant to all F z , and g contains the remaining parameters.

Original: The goal of training the neural network is to minimize the expected loss over the dataset: L ( W ; g ) :
Revised: The goal of training the neural network is to minimize the expected loss over the dataset: L ( W ; g ) :

Original: = E z
Revised: = E z

Original: ∼D [ F z ( W ; g )] .
Revised: ∼D [ F z ( W ; g )] .

Original: In order to illustrate the optimization beneﬁts of scaleinvariance, we consider the process of training this neural network by stochastic gradient descent with separate learning rates for W and g :
Revised: In order to illustrate the optimization beneﬁts of scaleinvariance, we consider the process of training this neural network by stochastic gradient descent with separate learning rates for W and g :

Original: [Equation]
Revised: [Equation]

Original: 2. T HE INTRINSIC OPTIMIZATION PROBLEM
Revised: 2. T HE INTRINSIC OPTIMIZATION PROBLEM

Original: Thanks to the scale-invariant properties, the scale of each weight w ( i ) does not affect loss values.
Revised: Thanks to the scale-invariant properties, the scale of each weight w ( i ) does not affect loss values.

Original: However, the scale does affect the gradients.
Revised: However, the scale does affect the gradients.

Original: Let V = { v (1) , . .
Revised: Let V = { v (1) , . .

Original: , v ( m ) } be the set of normalized weights, where v ( i ) =
Revised: , v ( m ) } be the set of normalized weights, where v ( i ) =

Original: w ( i ) / (cid:107) w ( i ) (cid:107) 2 .
Revised: w ( i ) / (cid:107) w ( i ) (cid:107) 2 .

Original: The following simple lemma can be easily shown:
Revised: The following simple lemma can be easily shown:

Original: Lemma 2.2 (Implied by Ioffe & Szegedy (2015)) .
Revised: Lemma 2.2 (Implied by Ioffe & Szegedy (2015)) .

Original: For any W and g ,
Revised: For any W and g ,

Original: [Equation]
Revised: [Equation]

Original: To make (cid:107)∇ w ( i ) F z ( W ; g ) (cid:107) 2 to be small, one can just scale the weights by a large factor.
Revised: To make (cid:107)∇ w ( i ) F [Equation] [Equation] 2 to be small, one can just scale the weights by a large factor.

Original: Thus there are ways to reduce the norm of the gradient that do not reduce the loss.
Revised: Thus there are ways to reduce the norm of the gradient that do not reduce the loss.

Original: For this reason, we deﬁne the intrinsic optimization problem for training the neural network.
Revised: For this reason, we deﬁne the intrinsic optimization problem for training the neural network.

Original: Instead of optimizing W and g over all possible solutions, we focus on parameters θ in which (cid:107) w ( i ) (cid:107) 

Revised: Instead of optimizing W and g over all possible solutions, we focus on parameters θ in which (cid:107) w ( i ) (cid:107)

Original: =
Revised: =

Original: for all w ( i ) ∈ W .
Revised: for all w ( i ) ∈ W .

Original: This does not change our objective, since the scale of W does not affect the loss.
Revised: This does not change our objective, since the scale of W does not affect the loss.

Original: Deﬁnition 2.3 (Intrinsic optimization problem) .
Revised: Deﬁnition 2.3 (Intrinsic optimization problem) .

Original: Let U
Revised: Let U

Original: = { θ | (cid:107) w ( i ) (cid:107) 2 = 1 for all i } be the intrinsic domain .
Revised: = { θ | (cid:107) w ( i ) (cid:107) 2 = 1 for all i } be the intrinsic domain .

Original: The intrinsic optimization problem is deﬁned as optimizing the original problem in U :
Revised: The intrinsic optimization problem is deﬁned as optimizing the original problem in U :

Original: [Equation]
Revised: [Equation]

Original: For { θ t } being a sequence of points for optimizing the original optimization problem, we can deﬁne { ˜ θ t } , where ˜ θ t =
Revised: For { θ t } being a sequence of points for optimizing the original optimization problem, we can deﬁne { ˜ θ t } , where ˜ θ t =

Original: ( V t ; g t ) , as a sequence of points optimizing the intrinsic optimization problem.
Revised: ( V t ; g t ) , as a sequence of points optimizing the intrinsic optimization problem.

Original: In this paper, we aim to show that training neural network for the original optimization problem by gradient descent can be seen as training by adaptive methods for the intrinsic optimization problem, and it converges to a ﬁrst-order stationary point in the intrinsic optimization problem with no need for tuning learning rates for W .
Revised: In this paper, we aim to show that training neural network for the original optimization problem by gradient descent can be seen as training by adaptive methods for the intrinsic optimization problem, and it converges to a ﬁrst-order stationary point in the intrinsic optimization problem with no need for tuning learning rates for W .

Original: 2. A SSUMPTIONS ON THE LOSS
Revised: 2. A SSUMPTIONS ON THE LOSS

Original: We assume F z ( W ; g ) is deﬁned and twice continuously differentiable at any θ satisfying none of w ( i ) is 0 .
Revised: We assume F z ( W ; g ) is deﬁned and twice continuously differentiable at any θ satisfying none of w ( i ) is 0 .

Original: Also, we assume that the expected loss L ( θ ) is lower-bounded by L min .
Revised: Also, we assume that the expected loss L ( θ ) is lower-bounded by L min .

Original: Furthermore, for V = { v (1) , . . .
Revised: Furthermore, for V = { v (1) , . . .

Original: , v ( m ) } , where v ( i ) =
Revised: , v ( m ) } , where v ( i ) =

Original: w ( i ) / (cid:107) w ( i ) (cid:107) 2 , we assume that the following bounds on the smoothness:
Revised: w ( i ) / (cid:107) w ( i ) (cid:107) 2 , we assume that the following bounds on the smoothness:

Original: [Equation]
Revised: [Equation]

Original: In addition, we assume that the noise on the gradient of g in SGD is upper bounded by G g :
Revised: In addition, we assume that the noise on the gradient of g in SGD is upper bounded by G g :

Original: [Equation]
Revised: [Equation]

Original: Smoothed version of motivating neural networks.
Revised: Smoothed version of motivating neural networks.

Original: Note that the neural network Φ illustrated in
Revised: Note that the neural network Φ illustrated in

Original: Section 2.1 does not meet the conditions of the smooothness at all since the loss function could be non-smooth.
Revised: Section 2.1 does not meet the conditions of the smooothness at all since the loss function could be non-smooth.

Original: We can make some mild modiﬁcations to the motivating example to smoothen it 1 : (1).
Revised: We can make some mild modiﬁcations to the motivating example to smoothen it 1 : (1).

Original: The activation could be non-smooth.
Revised: The activation could be non-smooth.

Original: A possible solution is to use smooth nonlinearities, e.g., sigmoid, tanh, softplus (Dugas et al., 2001), etc.
Revised: A possible solution is to use smooth nonlinearities, e.g., sigmoid, tanh, softplus (Dugas et al., 2001), etc.

Original: Note that softplus can be seen as a smooth approximation of the most commonly used activation ReLU.
Revised: Note that softplus can be seen as a smooth approximation of the most commonly used activation ReLU.

Original: The formula of BN shown in equation 3 may suffer from the problem of division by zero.
Revised: The formula of BN shown in equation 3 may suffer from the problem of division by zero.

Original: To avoid this, the inventors of BN, Ioffe & Szegedy (2015), add a small smoothening parameter (cid:15) >to the denominator, i.e.,
Revised: To avoid this, the inventors of BN, Ioffe & Szegedy (2015), add a small smoothening parameter (cid:15) >to the denominator, i.e.,

Original: [Equation]
Revised: [Equation]

Original: However, when z b,k = w (cid:62) k ˆ x b , adding a constant (cid:15) directly breaks the scale-invariance of w k .
Revised: However, when z b,k = w (cid:62) k ˆ x b , adding a constant (cid:15) directly breaks the scale-invariance of w k .

Original: We can preserve the scale-invariance by making the smoothening term propositional to (cid:107) w k (cid:107) 2 , i.e., replacing (cid:15) with (cid:15) (cid:107) w k (cid:107) 2 .
Revised: We can preserve the scale-invariance by making the smoothening term propositional to (cid:107) w k (cid:107) 2 , i.e., replacing (cid:15) with (cid:15) (cid:107) w k (cid:107) 2 .

Original: By simple linear algebra and letting u :=
Revised: By simple linear algebra and letting u :=

Original: E b ∈ [ B ] [ˆ x b ] , S :=
Revised: E b ∈ [ B ] [ˆ x b ] , S :=

Original: Var b ∈ [ B ] (ˆ x b ) , this smoothed version of BN can also be written as
Revised: Var b ∈ [ B ] (ˆ x b ) , this smoothed version of BN can also be written as

Original: [Equation]
Revised: [Equation]

Original: Since the variance of inputs is usually large in practice, for small (cid:15) , the effect of the smoothening term is negligible except in extreme cases.
Revised: Since the variance of inputs is usually large in practice, for small (cid:15) , the effect of the smoothening term is negligible except in extreme cases.

Original: Using the above two modiﬁcations, the loss function is already smooth.
Revised: Using the above two modiﬁcations, the loss function is already smooth.

Original: However, the scale of scale-variant parameters may be unbounded during training, which could cause the smoothness unbounded.
Revised: However, the scale of scale-variant parameters may be unbounded during training, which could cause the smoothness unbounded.

Original: To avoid this issue, we can either project scale-variant parameters to a bounded set, or use weight decay for those parameters (see Appendix C for a proof for the latter solution).
Revised: To avoid this issue, we can either project scale-variant parameters to a bounded set, or use weight decay for those parameters (see Appendix C for a proof for the latter solution).

Original: 2. K EY OBSERVATION : THE GROWTH OF WEIGHTS
Revised: 2. K EY OBSERVATION : THE GROWTH OF WEIGHTS

Original: The following lemma is our key observation.
Revised: The following lemma is our key observation.

Original: It establishes a connection between the scale-invariant property and the growth of weight scale, which further implies an automatic decay of learning rates:
Revised: It establishes a connection between the scale-invariant property and the growth of weight scale, which further implies an automatic decay of learning rates:

Original: Lemma 2.4.
Revised: Lemma 2.4.

Original: For any scale-invariant weight w ( i ) in the network Ψ , we have:
Revised: For any scale-invariant weight w ( i ) in the network Ψ , we have:

Revised: [Equation]

Original: w ( i ) t and ∇ w ( i ) t L ( θ t ) are always perpendicular;
Revised: w [Equation] i ) t and ∇ w ( i ) t F z t ( θ t ) are always perpendicular

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let θ (cid:48)  be all the parameters  other than w ( i )  .
Revised: Let θ (cid:48) t be all the parameters in θ t other than w ( i ) t .

Original: Taking derivatives with respect to c for the both sides of F z  ( w ( i )  , θ (cid:48)  )
Revised: Taking derivatives with respect to c for the both sides of F z t ( w ( i ) t , θ (cid:48) t )

Original: =
Revised: =

Original: F z  ( c w ( i )  , θ (cid:48)  ) , we have 0 =
Revised: F z t ( c w ( i ) t , θ (cid:48) t ) , we have 0 =

Original: ∂∂c F z  ( c w ( i )  , θ (cid:48)  ) .
Revised: ∂∂c F z t ( c w ( i ) t , θ (cid:48) t ) .

Original: The right hand side equals ∇ c w ( i ) L ( c w ( i )  , θ (cid:48)  ) (cid:62) w ( i )  , then the ﬁrst proposition follows by taking c = 1 .
Revised: The right hand side equals ∇ c w ( i ) t F z t ( c w ( i ) t , θ (cid:48) t ) (cid:62) w ( i ) t , so the ﬁrst proposition follows by taking c = 1 .

Original: Applying Pythagorean theorem and Lemma 2.2, the second proposition directly follows.
Revised: Applying Pythagorean theorem and Lemma 2.2, the second proposition directly follows.

Original: Using Lemma 2.4, we can show that performing gradient descent for the original problem is equivalent to performing an adaptive gradient method for the intrinsic optimization problem:
Revised: Using Lemma 2.4, we can show that performing gradient descent for the original problem is equivalent to performing an adaptive gradient method for the intrinsic optimization problem:

Original: Theorem 2.5.
Revised: Theorem 2.5.

Original: Let Π be a projection operator which maps a vector w to w / (cid:107) w  (cid:107) 2 . Then 
Revised: Let G ( i ) t = (cid:107) w ( i ) t (cid:107) 22 . Then for all t ≥ 0 , v ( i )

Original: [Equation]
Revised: [Equation]

Revised: where Π is a projection operator which maps any vector w to w / (cid:107) w (cid:107) 2 .

Original: Remark 2.6.
Revised: Remark 2.6.

Original: Wu et al.
Revised: Wu et al.

Original: (2018) noticed that Theorem 2.5 is true for Weight Normalization by direct calculation of gradients.
Revised: (2018) noticed that Theorem 2.5 is true for Weight Normalization by direct calculation of gradients.

Original: Inspiring by this, they proposed a new adaptive method called WNGrad .
Revised: Inspiring by this, they proposed a new adaptive method called WNGrad .

Original: Our theorem is more general since it holds for any normalization methods as long as it induces scale-invariant properties to the network.
Revised: Our theorem is more general since it holds for any normalization methods as long as it induces scale-invariant properties to the network.

Original: The adaptive update rule derived in our theorem can be seen as WNGrad with projection to unit sphere after each step.
Revised: The adaptive update rule derived in our theorem can be seen as WNGrad with projection to unit sphere after each step.

Original: Proof for Theorem 2.5.
Revised: Proof for Theorem 2.5.

Original: Let G ( i ) t = (cid:107) w ( i ) t (cid:107) 22 . The second equation is by Lemma 2.4. By Lemma 2.2, w ( i ) t +1 = w ( i ) t −
Revised: Using Lemma 2.2, we have

Original: [Equation]
Revised: [Equation]

Original: which implies equation 9.
Revised: which implies the ﬁrst equation.

Original:  While popular adaptive gradient methods such as AdaGrad (Duchi et al., 2011), RMSprop (Hinton et al.), Adam (Kingma & Ba, 2014) adjust learning rates for each single coordinate, the adaptive gradient method described in Theorem 2.5 sets learning rates η w ,t /G ( i ) t for each scale-invariant parameter respectively.
Revised: The second equation is by Lemma 2.4. While popular adaptive gradient methods such as AdaGrad (Duchi et al., 2011), RMSprop (Tieleman & Hinton, 2012), Adam (Kingma & Ba, 2014) adjust learning rates for each single coordinate, the adaptive gradient method described in Theorem 2.5 sets learning rates η w ,t /G ( i ) t for each scaleinvariant parameter respectively.

Original: In this paper, we call η w ,t /G
Revised: In this paper, we call η w ,t /G

Original: ( i ) t the effective learning rate of v ( i ) t or w ( i ) t , because it’s η w ,t / (cid:107) w ( i ) t (cid:107) 22 instead of η w , t alone that really determines the trajectory of gradient descent given the normalized scale-invariant parameter v ( i ) t .
Revised: ( i ) t the effective learning rate of v ( i ) t or w ( i ) t , because it is η w ,t / (cid:107) w ( i ) t (cid:107) 22 instead of η w ,t alone that really determines the trajectory of gradient descent given the normalized scale-invariant parameter v ( i ) t .

Original: In other words, the magnitude of the initialization of parameters before BN is as important as their learning rates: multiplying the initialization of scale-invariant parameter by constant C is equivalent to dividing its learning rate by C 2 .
Revised: In other words, the magnitude of the initialization of parameters before BN is as important as their learning rates: multiplying the initialization of scale-invariant parameter by constant C is equivalent to dividing its learning rate by C 2 .

Original: Thus we suggest researchers to report initialization as well as learning rates in the future experiments.
Revised: Thus we suggest researchers to report initialization as well as learning rates in the future experiments.

Original:  T RAINING BY FULL - BATCH GRADIENT DESCENT
Revised:  T RAINING BY FULL - BATCH GRADIENT DESCENT

Original: In this section, we rigorously analyze the effect related to the scale-invariant properties in training neural network by full-batch gradient descent.
Revised: In this section, we rigorously analyze the effect related to the scale-invariant properties in training neural network by full-batch gradient descent.

Original: We use the framework introduced in Section 2.2 and assumptions from Section 2.4.
Revised: We use the framework introduced in Section 2.2 and assumptions from Section 2.4.

Original: We focus on the full-batch training, i.e., z t is always equal to the whole training set and F z t ( θ ) =
Revised: We focus on the full-batch training, i.e., z t is always equal to the whole training set and F z t ( θ ) =

Original: L ( θ ) .
Revised: L ( θ ) .

Original: 3. S ETTINGS AND MAIN THEOREM
Revised: 3. S ETTINGS AND MAIN THEOREM

Original: Assumptions on learning rates.
Revised: Assumptions on learning rates.

Original: We consider the case that we use ﬁxed learning rates for both W and g , i.e., η w , 0 = · · · = η w ,T − 1 = η w and η g , 0 = · · · = η g ,T − 1 =
Revised: We consider the case that we use ﬁxed learning rates for both W and g , i.e., η w , 0 = · · · = η w ,T − 1 = η w and η g , 0 = · · · = η g ,T − 1 =

Original: η g .
Revised: η g .

Original: We assume that η g is tuned carefully to η g =
Revised: We assume that η g is tuned carefully to η g =

Original: (1 − c g ) /L
Revised: (1 − c g ) /L

Original: gg for some constant c g ∈ (0 , 1) .
Revised: gg for some constant c g ∈ (0 , 1) .

Original: For η w , we do not make any assumption, i.e., η w can be set to any positive value.
Revised: For η w , we do not make any assumption, i.e., η w can be set to any positive value.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Consider the process of training Φ by gradient descent with η g = 2(1 − c g ) /L
Revised: Consider the process of training Φ by gradient descent with η g = 2(1 − c g ) /L

Original: gg and arbitrary η w > 0 .
Revised: gg and arbitrary η w > 0 .

Original: Then Φ converges to a stationary point in the rate of
Revised: Then Φ converges to a stationary point in the rate of

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: 10) where V t = { v (1) t , . . 
Revised: where V t = { v (1) t , . . .

Original: , v ( m ) t } with v ( i ) t = w ( i ) t / (cid:107) w
Revised: , v ( m ) t } with v ( i ) t = w ( i ) t / (cid:107) w

Original: ( i ) t (cid:107) 2 , ˜ O suppresses polynomial factors in L vv ij ,
Revised: ( i ) t (cid:107) 2 , ˜ O suppresses polynomial factors in L vv ij ,

Original: L vg i , L gg , (cid:107) w ( i ) 

Revised: L vg i , L gg , (cid:107) w ( i )

Original: (cid:107) 2 , (cid:107) w ( i ) 

Revised: (cid:107) 2 , (cid:107) w ( i )

Original: (cid:107) − 1 2 , L ( θ 0 )
Revised: (cid:107) − 1 2 , L ( θ 0 )

Original: − L min for all i, j , and we see L gg = Ω(1) .
Revised: − L min for all i, j , and we see L gg = Ω(1) .

Original: This matches the asymptotic convergence rate of GD by Carmon et al.
Revised: This matches the asymptotic convergence rate of GD by Carmon et al.

Original: 3. P ROOF SKETCH
Revised: 3. P ROOF SKETCH

Original: The high level idea is to use the decrement of loss function to upper bound the sum of the squared norm of the gradients.
Revised: The high level idea is to use the decrement of loss function to upper bound the sum of the squared norm of the gradients.

Original: Note that (cid:107)∇L ( V t ; g t ) (cid:107) 

Revised: Note that (cid:107)∇L ( V t ; g t ) (cid:107)

Original: = (cid:80) mi
Revised: = (cid:80) mi

Original: [-=1 (cid:107)∇ v ( i ) t L ( V t ; g t ) (cid:107) 
 + (cid:107)∇ g t L ( V t ; g t ) (cid:107) 22 . For the ﬁrst part (cid:80)-]
Revised: [-=1 (cid:107)∇ v ( i ) t L ( V t ; g t ) (cid:107) 
 + (cid:107)∇ g t L ( V t ; g t ) (cid:107) 22 . For the ﬁrst part (cid:80)-]=

Original: [Equation]
Revised: [Equation]

Original: v [Equation] i ) t L
Revised: For the ﬁrst part (cid:80) mi =

Original: [Equation]
Revised: [Equation]

Original: V t

Original: [Equation] g t ) [Equation] 22 , we have
Revised: v ( i ) t L ( V t ; g t ) (cid:107) 22 , we have

Original: [Equation]
Revised: [Equation]

Original: Thus the core of the proof is to show that the monotone increasing (cid:107) w ( i ) T (cid:107) 2 has an upper bound for all T .
Revised: Thus the core of the proof is to show that the monotone increasing (cid:107) w ( i ) T (cid:107) 2 has an upper bound for all T .

Original: It is shown that for every w ( i ) , the whole training process can be divided into at most two phases.
Revised: It is shown that for every w ( i ) , the whole training process can be divided into at most two phases.

Original: In the ﬁrst phase, the effective learning rate η w / (cid:107) w ( i ) t (cid:107) 22 is larger than some threshold 1 C i (deﬁned in Lemma 3.2) and in the second phase it is smaller.
Revised: In the ﬁrst phase, the effective learning rate η w / (cid:107) w ( i ) t (cid:107) 22 is larger than some threshold 1 C i (deﬁned in Lemma 3.2) and in the second phase it is smaller.

Original: Lemma 3.2 (Taylor Expansion) .
Revised: Lemma 3.2 (Taylor Expansion) .

Original: Let C i :=
Revised: Let C i :=

Original: 12 (cid:80) mj
Revised: 12 (cid:80) mj

Original: =1 L vv ij + L vg i 2 m/ ( c g L gg ) =
Revised: =1 L vv ij + L vg i 2 m/ ( c g L gg ) =

Original: ˜ O (1) .
Revised: ˜ O (1) .

Original: Then
Revised: Then

Revised: L (

Revised: [Equation]

Revised: t +

Revised: [Equation]

Revised: − L (

Original: [Equation]
Revised: [Equation]

Revised: t

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: [Equation]

Original: If (cid:107) w ( i ) t (cid:107) 2 is large enough and that the process enters the second phase, then by Lemma 3.2 in each step the loss function L will decrease by 
Revised: If (cid:107) w ( i ) t (cid:107) 2 is large enough and that the process enters the second phase, then by Lemma 3.2 in each step the loss function L will decrease by η w 2 (cid:107)∇ w [Equation] i ) t L

Original: [Equation]
Revised: [Equation]

Original: (Recall that
Revised: θ t [Equation] = [Equation]

Revised: w

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Revised: [Equation]

Revised: +

Revised: [Equation]

Revised: [Equation]

Revised: w

Original: [Equation]
Revised: [Equation]

Revised: i

Original: [Equation]
Revised: [Equation]

Revised: 22 2 η w (Recall that (cid:107) w

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: 22 −

Revised: [Equation]

Revised: i

Revised: [Equation]

Original:  by Lemma 2.4).
Revised: i ) t L [Equation] θ t ) (cid:107) 22 by Lemma 2.4).

Original: Since L is lower-bounded, we can conclude (cid:107) w ( i ) t (cid:107) 22 is also bounded .
Revised: Since L is lower-bounded, we can conclude (cid:107) w ( i ) t (cid:107) 22 is also bounded .

Original: For the second part, we can also show that by Lemma 3.
Revised: For the second part, we can also show that by Lemma 3.

Original: [Equation]
Revised: [Equation]

Original: Thus we can conclude ˜ O ( 1 √ T ) convergence rate of (cid:107)∇L ( θ t ) (cid:107) 2 as follows. 
Revised: Thus we can conclude ˜ O ( 1 √ T ) convergence rate of (cid:107)∇L ( θ t ) (cid:107) 2 as follows. min

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: The full proof is postponed to Appendix A.
Revised: The full proof is postponed to Appendix A.

Original:  T RAINING BY STOCHASTIC GRADIENT DESCENT
Revised:  T RAINING BY STOCHASTIC GRADIENT DESCENT

Original: In this section, we analyze the effect related to the scale-invariant properties when training a neural network by stochastic gradient descent.
Revised: In this section, we analyze the effect related to the scale-invariant properties when training a neural network by stochastic gradient descent.

Original: We use the framework introduced in Section 2.2 and assumptions from Section 2.4.
Revised: We use the framework introduced in Section 2.2 and assumptions from Section 2.4.

Original: 4. S ETTINGS AND MAIN THEOREM
Revised: 4. S ETTINGS AND MAIN THEOREM

Original: Assumptions on learning rates.
Revised: Assumptions on learning rates.

Original: As usual, we assume that the learning rate for g is chosen carefully and the learning rate for W is chosen rather arbitrarily.
Revised: As usual, we assume that the learning rate for g is chosen carefully and the learning rate for W is chosen rather arbitrarily.

Original: More speciﬁcally, we consider the case that the learning rates are chosen as
Revised: More speciﬁcally, we consider the case that the learning rates are chosen as

Original: [Equation]
Revised: [Equation]

Original: We assume that the initial learning rate η g of g is tuned carefully to η g =
Revised: We assume that the initial learning rate η g of g is tuned carefully to η g =

Original: (1 − c g ) /L
Revised: (1 − c g ) /L

Original: gg for some constant c g ∈ (0 , 1) .
Revised: gg for some constant c g ∈ (0 , 1) .

Original: Note that this learning rate schedule matches the best known convergence rate
Revised: Note that this learning rate schedule matches the best known convergence rate

Original: O ( T − 1 / 4 ) of SGD in the case of smooth non-convex loss functions (Ghadimi & Lan, 2013).
Revised: O ( T − 1 / 4 ) of SGD in the case of smooth non-convex loss functions (Ghadimi & Lan, 2013).

Original: For the learning rates of W , we only assume that 0 ≤ α ≤ 1 / 2 , i.e., the learning rate decays equally as or slower than the optimal SGD learning rate schedule.
Revised: For the learning rates of W , we only assume that 0 ≤ α ≤ 1 / 2 , i.e., the learning rate decays equally as or slower than the optimal SGD learning rate schedule.

Original: η w can be set to any positive value.
Revised: η w can be set to any positive value.

Original: Note that this includes the case that we set a ﬁxed learning rate η w , 0 = · · · = η w ,T − 1 = η w for W by taking α = 0 .
Revised: Note that this includes the case that we set a ﬁxed learning rate η w , 0 = · · · = η w ,T − 1 = η w for W by taking α = 0 .

Original: Remark 4.1.
Revised: Remark 4.1.

Original: Note that the auto-tuning behavior induced by scale-invariances always decreases the learning rates.
Revised: Note that the auto-tuning behavior induced by scale-invariances always decreases the learning rates.

Original: Thus, if we set α > 1 / 2 , there is no hope to adjust the learning rate to the optimal strategy Θ( t − 1 / 2 ) .
Revised: Thus, if we set α > 1 / 2 , there is no hope to adjust the learning rate to the optimal strategy Θ( t − 1 / 2 ) .

Original: Indeed, in this case, the learning rate 1 /G t in the intrinsic optimization process decays exactly in the rate of ˜Θ( t − α ) , which is the best possible learning rate can be achieved without increasing the original learning rate.
Revised: Indeed, in this case, the learning rate 1 /G t in the intrinsic optimization process decays exactly in the rate of ˜Θ( t − α ) , which is the best possible learning rate can be achieved without increasing the original learning rate.

Original: Theorem 4.2.
Revised: Theorem 4.2.

Original: Consider the process of training Φ by gradient descent with η w ,t = η w · ( t + 1) − α and η g ,t
Revised: Consider the process of training Φ by gradient descent with η w ,t = η w · ( t + 1) − α and η g ,t

Original: = η g · ( t + 1) − 1 / 2 , where η g = 2(1 − c g ) /L
Revised: = η g · ( t + 1) − 1 / 2 , where η g = 2(1 − c g ) /L

Original: gg and η w > 0 is arbitrary.
Revised: gg and η w > 0 is arbitrary.

Original: Then Φ converges to a stationary point in the rate of
Revised: Then Φ converges to a stationary point in the rate of

Original: [Equation]
Revised: [Equation]

Original: where V t = { v (1) t , . . .
Revised: where V [Equation]

Original: , v ( m ) t } with v ( i ) t= w ( i ) t / (cid:107) w
Revised: [Equation] } with v ( i [Equation] t

Original: ( i ) t (cid:107) 2 , ˜ O suppresses polynomial factors in η [-w , η − 1w , η − 1 g , L vv ij , L vg i , L gg , (cid:107) w ( i ) 
-]
Revised: [Equation] , ˜ O suppresses polynomial factors in η [-w , η − 1w , η − 1 g , L vv ij , L vg i , L gg , (cid:107) w ( i ) 
-][Equation]

Original: [-(cid:107) 2 , (cid:107) w ( i ) 
 (cid:107) − 1 2 , L ( θ 0 )-]
Revised: [-(cid:107) 2 , (cid:107) w ( i ) 
 (cid:107) − 1 2 , L ( θ 0 )-][Equation]

Original: −L min for all i, j , and we see L gg = Ω(1) .
Revised: −L min for all i, j , and we see L gg = Ω(1) .

Original: Note that this matches the asymptotic convergence rate of SGD, within a polylog( T ) factor.
Revised: Note that this matches the asymptotic convergence rate of SGD, within a polylog( T ) factor.

Original: 4. P ROOF SKETCH
Revised: 4. P ROOF SKETCH

Original: We delay the full proof into Appendix B and give a proof sketch in a simpliﬁed setting where there is no g and α ∈ [0 , 12 ) .
Revised: We delay the full proof into Appendix B and give a proof sketch in a simpliﬁed setting where there is no g and α ∈ [0 , 12 ) .

Original: We also assume there’s only one w i , that is, m = 1 and omit the index i .
Revised: We also assume there’s only one w i , that is, m = 1 and omit the index i .

Original: By Taylor expansion [Equation] we have
Revised: By Taylor expansion, we have

Original: [Equation]
Revised: [Equation]

Original: We can lower bound the effective learning rate η w ,T [Equation] w T
Revised: We can lower bound the effective learning rate η w ,T (cid:107) w T

Original: [Equation]
Revised: [Equation]

Original: 2 and upper bound the second order term respectively in the following way:
Revised: 2 and upper bound the second order term respectively in the following way:

Original: [Equation]

Original: For all 0 ≤ α 
Revised: For all 0 ≤ α <

Original: [Equation]

Original: [Equation]

Original: the effective 
Revised: the effective learning

Original: [Equation]
Revised: [Equation]

Original: Taking expectation over equation 14 and summing it up
Revised: Taking expectation over equation 14 and summing it up

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Plug the above bounds into the above inequality, we complete the proof.
Revised: Plug the above bounds into the above inequality, we complete the proof.

Original: [Equation]
Revised: [Equation]

Original:  C ONCLUSIONS AND FUTURE WORKS
Revised:  C ONCLUSIONS AND FUTURE WORKS

Original: In this paper, we studied how scale-invariances in neural networks with BN help optimization, and showed that (stochastic) gradient descent can achieve  asymptotic best convergence rate without tuning learning rates for scale-invariant parameters.
Revised: In this paper, we studied how scale-invariance in neural networks with BN helps optimization, and showed that (stochastic) gradient descent can achieve the asymptotic best convergence rate without tuning learning rates for scale-invariant parameters.

Original: Our analysis suggests that scale-invariance in nerual networks introduced by BN reduces the efforts for tuning learning rate to ﬁt the training data.
Revised: Our analysis suggests that scale-invariance in nerual networks introduced by BN reduces the efforts for tuning learning rate to ﬁt the training data.

Original: However, our analysis only applies to smooth loss functions.
Revised: However, our analysis only applies to smooth loss functions.

Original: In modern neural networks, ReLU or
Revised: In modern neural networks, ReLU or

Original: Leaky ReLU are often used, which makes the loss non-smooth.
Revised: Leaky ReLU are often used, which makes the loss non-smooth.

Original: It would have more implications by showing similar results in non-smooth settings.
Revised: It would have more implications by showing similar results in non-smooth settings.

Original: Also, we only considered gradient descent in this paper.
Revised: Also, we only considered gradient descent in this paper.

Original: It can be shown that if we perform (stochastic) gradient descent with momentum, the norm of scale-invariant parameters will also be monotone increasing.
Revised: It can be shown that if we perform (stochastic) gradient descent with momentum, the norm of scale-invariant parameters will also be monotone increasing.

Original: It would be interesting to use it to show similar convergence results for more gradient methods.
Revised: It would be interesting to use it to show similar convergence results for more gradient methods.

Revised: A CKNOWLEDGMENTS

Revised: Thanks Yuanzhi Li, Wei Hu and Noah Golowich for helpful discussions.

Revised: This research was done with support from NSF, ONR, Simons Foundation, Mozilla Research, Schmidt Foundation, DARPA, and SRC.

Original: A P ROOF FOR F ULL -B ATCH G RADIENT D ESCENT
Revised: A P ROOF FOR F ULL -B ATCH G RADIENT D ESCENT

Original: By the scale-invariant property of w ( i ) , we know that L ( W ; g ) =
Revised: By the scale-invariant property of w ( i ) , we know that L ( W ; g ) =

Original: L ( V ; g ) .
Revised: L ( V ; g ) .

Original: Also, the following identities about derivatives can be easily obtained:
Revised: Also, the following identities about derivatives can be easily obtained:

Original: [Equation]
Revised: [Equation]

Original: Thus, the assumptions on the smoothness imply
Revised: Thus, the assumptions on the smoothness imply

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Proof for Lemma 3.2.
Revised: Proof for Lemma 3.2.

Original: Using Taylor expansion, we have ∃ γ ∈ (0 , 1) , such that for w ( i (cid:48) )
Revised: Using Taylor expansion, we have ∃ γ ∈ (0 , 1) , such that for w ( i (cid:48) )

Original: t
Revised: t

Original: [Equation] Note that w ( i ) t +1 − w ( i ) t = [Equation] w
Revised: = (1 − γ ) w ( i ) t + γ w ( i ) t +1 ,

Original: [Equation]
Revised: [Equation]

Original: w ( i ) t L ( θ t ) is perpendicular to w ( i ) t , we have (cid:107) w [Equation] i
Revised: (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: ≥

Original: [Equation]
Revised: [Equation]

Original: i

Original: [Equation]
Revised: [Equation]

Original: + w
Revised: (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: i

Original: [Equation]
Revised: [Equation]

Original: (cid:107) w ( i ) t (cid:107) 2 . Thus, L (
Revised: (cid:13)(cid:13)(cid:13) 2 (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: ≤ [Equation] = [Equation]
Revised: ( j (cid:48) ) t (cid:13)(cid:13)(cid:13)+ m (cid:88) i = (cid:13)(cid:13)(cid:13) ∆ w ( i ) t (cid:13)(cid:13)(cid:13) 2 (cid:107) ∆ g t (cid:107)

Original: θ [Equation] +[Equation]
Revised: L vg i (cid:13)(cid:13)(cid:13) w ( i (cid:48) ) t (cid:13)(cid:13)(cid:13) +

Original: [Equation]
Revised: [Equation]

Original: i= [Equation] (cid:88) [Equation]
Revised: i

Original: = [Equation](cid:13)(cid:13)(cid:13) ∆ w ( i ) t (cid:13)(cid:13)(cid:13) 2 (cid:13)(cid:13)(cid:13) ∆ w ( j ) t (cid:13)(cid:13)(cid:13) L vv ij (cid:13)(cid:13)(cid:13) w ( i ) t (cid:13)(cid:13)(cid:13) 2 (cid:13)(cid:13)(cid:13) w ( j ) t (cid:13)(cid:13)(cid:13)+ m (cid:88) i = (cid:13)(cid:13)(cid:13) ∆ w ( i ) t (cid:13)(cid:13)(cid:13) 2 (cid:107) ∆ g t (cid:107)
Revised: [Equation]

Original: L vg i (cid:13)(cid:13)(cid:13) w ( i ) t (cid:13)(cid:13)(cid:13)+ m (cid:88) i = (cid:107) ∆ g t (cid:107)[-
 L gg . By the inequality of arithmetic and geometric means, we have (cid:13)(cid:13)(cid:13) ∆ w-]
Revised: (cid:107)[-
 L gg . By the inequality of arithmetic and geometric means, we have (cid:13)(cid:13)(cid:13) ∆ w-]

Original: [Equation]
Revised: [Equation]

Original: {+t (cid:107) 
+}
Revised: {+t (cid:107) 
+}

Original: i
Revised: L

Original: [Equation]
Revised: [Equation]

Original:  t 
Revised: Note that w ( i ) t +1 − w ( i ) t = η w ∇ w ( i ) t L ( θ t ) is perpendicular to w ( i ) t , we have

Original: [Equation]
Revised: [Equation]

Revised: Thus,

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: Taking ∆ w ( i ) t = − η w ∇ w ( i ) t L ( θ t ) , ∆ g t = − η g ∇ g t L ( θ t ) , we have 
Revised: By the inequality of arithmetic and geometric means [Table] we have [Table]

Original: [Equation]
Revised: [Equation]

Original: We can complete the proof by replacing 1 2 (cid:80) mj
Revised: We can complete the proof by replacing 12 (cid:80) [Equation] L [Equation]

Revised: L

Original: =1 L vv ij + L vg i 2 m/ ( c g L gg ) with C i .
Revised: [Equation] i 2 m/ ( c [Equation] L gg ) with C i .

Original: Using the assumption on the smoothness, we can show that the gradient with respect to w ( i ) is essentially bounded:
Revised: Using the assumption on the smoothness, we can show that the gradient with respect to w ( i ) is essentially bounded:

Original: Lemma A.1.
Revised: Lemma A.1.

Original: For any W and g , we have
Revised: For any W and g , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: A.1 Fix all the parameters except w ( i ) .
Revised: A.1 Fix all the parameters except w ( i ) .

Original: Then L ( W ; g ) can be written as a function f ( w ( i ) ) on the variable w ( i ) .
Revised: Then L ( W ; g ) can be written as a function f ( w ( i ) ) on the variable w ( i ) .

Original: Let S = { w ( i ) | (cid:107) w ( i ) (cid:107) 2 = 1 } .
Revised: Let S = { w ( i ) | (cid:107) w ( i ) (cid:107) 2 = 1 } .

Original: Since f is continuous and S is compact, there must exist v ( i )min ∈ S such that f ( v ( i )min ) ≤ f ( w ( i ) ) for all w ( i ) ∈ S .
Revised: Since f is continuous and S is compact, there must exist v ( i )min ∈ S such that f ( v ( i )min ) ≤ f ( w ( i ) ) for all w ( i ) ∈ S .

Original: Note that w ( i ) is scale-invariant, so w ( i )min is also a minimum in the entire domain and ∇ f ( w ( i )min ) = 0 .
Revised: Note that w ( i ) is scale-invariant, so w ( i )min is also a minimum in the entire domain and ∇ f ( w ( i )min ) = 0 .

Original: For an arbitrary w ( i ) , let v ( i ) =
Revised: For an arbitrary w ( i ) , let v ( i ) =

Original: w ( i ) / (cid:107) w ( i ) (cid:107) 2 .
Revised: w ( i ) / (cid:107) w ( i ) (cid:107) 2 .

Original: Let h : [0 , 1] → S be a curve such that h (0) = v ( i )min , h (1)
Revised: Let h : [0 , 1] → S be a curve such that h (0) = v ( i ) min , h (1)

Original: = v ( i ) , and h goes along the geodesic from v ( i )min to v ( i ) on the unit sphere S with constant speed.
Revised: = v ( i ) , and h goes along the geodesic from v ( i )min to v ( i ) on the unit sphere S with constant speed.

Original: Let H ( τ ) = ∇ f
Revised: Let H ( τ ) = ∇ f

Original: ( h ( τ )) .
Revised: ( h ( τ )) .

Original: By Taylor expansion, we have
Revised: By Taylor expansion, we have

Original: [Equation]
Revised: [Equation]

Original: Thus, 
Revised: Thus, (cid:107)∇ [Equation] and

Original: [Equation]
Revised: [Equation]

Original: The following lemma gives an upper bound to the weight scales.
Revised: The following lemma gives an upper bound to the weight scales.

Original: Lemma A.2.
Revised: Lemma A.2.

Original: For all T ≥ 0 , we have 1.
Revised: For all T ≥ 0 , we have 1.

Original: for i = 1 , . . .
Revised: for i = 1 , . . .

Original: , m , let t i be the maximum time 

Revised: , m , let t i be the maximum time

Original: ≤ τ < T satisfying(cid:107) w ( i ) τ [Equation] 22 ≤
Revised: ≤ τ < T satisfying

Original: [Equation]
Revised: [Equation]

Original: w ( t i = − 1 if no such τ exists), then (cid:107) w ( i ) t
Revised: ( t i = − 1 if no such τ exists), then (cid:107) w ( i ) t

Original: i +1 (cid:107) 2 can be bounded by
Revised: i +1 (cid:107) 2 can be bounded by

Revised: [Equation]

Revised: i

Original: [Equation]
Revised: [Equation]

Original: the following inequality on weight scales at time T holds:
Revised: the following inequality on weight scales at time T holds:

Original: [Equation]
Revised: [Equation]

Original: ) where K i =
Revised: where K i =

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: A.2 For every i = 1 , . . .
Revised: A.2 For every i = 1 , . . .

Original: , m , let S ( i ) t
Revised: , m , let S ( i ) t

Original: [Equation]
Revised: [Equation]

Original:  w ( i ) τ
Revised: 1 − C [Equation] w (cid:107) w ( i ) [Equation]

Original: (cid:107)(cid:19) .
Revised: (cid:107)(cid:19) .

Original: Also let G t := − (1 / 2)
Revised: Also let G t := − (1 / 2)

Original: c g η g (cid:80) t − 1 τ =

Revised: c g η g (cid:80) t − 1 τ =

Original: (cid:107)∇ g t L ( θ t ) (cid:107) 22 .
Revised: (cid:107)∇ g t L ( θ t ) (cid:107) 22 .

Original: By Lemma 3.2 we know that L ( θ T ) ≤ L ( θ 0 )
Revised: By Lemma 3.2 we know that L ( θ T ) ≤ L ( θ 0 )

Original: + (cid:80) mi =1 S ( i ) T + G T .
Revised: + (cid:80) mi =1 S ( i ) T + G T .

Original: Upper Bound for Weight Scales at t i + 1 .
Revised: Upper Bound for Weight Scales at t i + 1 .

Original: If t i = − 1 , then (cid:107) w ( i ) t
Revised: If t i = − 1 , then (cid:107) w ( i ) t

Original: i +1 (cid:107) 

Revised: i +1 (cid:107)

Original: = (cid:107) w ( i ) 

Revised: = (cid:107) w ( i )

Original: (cid:107) 22 .
Revised: (cid:107) 22 .

Original: For t i ≥ 0 , we can bound (cid:107) w ( i ) t
Revised: For t i ≥ 0 , we can bound (cid:107) w ( i ) t

Original: i +1 (cid:107) 22 by Lemma A.1:
Revised: i +1 (cid:107) 22 by Lemma A.1:

Original: [Equation]
Revised: [Equation]

Original: In either case, we have (cid:107) w ( i )
Revised: In either case, we have (cid:107) w ( i )

Original: t
Revised: t

Original: i +1 (cid:107) 2 2 ≤ (cid:107) w ( i ) 0 (cid:107) 2 2 + η w (cid:18) 2 C i + ( πL vv i )η
Revised: i +1 (cid:107) 22 ≤ (cid:107) w ( i ) 0 (cid:107) 22 + η w (cid:18) 2 C i + ( πL vv i )

Original: [Equation]
Revised: [Equation]

Original: (cid:19) .
Revised: (cid:19) .

Original: [Table]
Revised: Upper Bound for Weight Scales at T . Since

Original: [Table]
Revised: [Table]

Original: i )

Original: [Equation]
Revised: [Equation]

Original:  22 +{+= (cid:107) w ( i ) 
+}
Revised: where we use the fact that (cid:107) w ( i ) t (cid:107) 22 {+= (cid:107) w ( i ) 
+}

Original: [Equation]{+(cid:107) 
+}
Revised: {+(cid:107) 
+}

Original:  2w [Equation] − 1 τ =0{+=
+} (cid:107)∇ w ( i ) τ
Revised: + η 2w (cid:80) t − 1 τ {+=
+} (cid:107)∇ w ( i ) τ

Original: L ( θ τ ) [Equation] 22 at the third line.
Revised: L ( θ τ ) (cid:107) 22 at the third line.

Original: We can bound S ( i ) t i +1 by
Revised: We can bound S ( i ) t i +1 by

Original: [Equation]
Revised: [Equation]

Original: Combining them together, we have
Revised: Combining them together, we have

Original: [Equation]
Revised: [Equation]

Original: Taking sum over all i = 1 , . .
Revised: Taking sum over all i = 1 , . .

Original: , m and also subtracting G T on the both sides, we havem
Revised: , m and also subtracting G T on the both sides, we have

Original: [Equation]
Revised: [Equation]

Original: where L min ≤ L ( θ T ) ≤ L ( θ 0 ) + (cid:80) mi
Revised: where L min ≤ L ( θ T ) ≤ L ( θ 0 ) + (cid:80) mi

Original: =1 S ( i ) T + G T is used at the second line.
Revised: =1 S ( i ) T + G T is used at the second line.

Original: Combining the lemmas above together, we can obtain our results.
Revised: Combining the lemmas above together, we can obtain our results.

Original: Proof for Theorem 
Revised: Proof for Theorem 3.1.

Revised: By Lemma A.2, we have m (cid:88)

Original: [Equation]
Revised: [Equation]

Revised: m (cid:40)

Revised: [Equation]

Revised: ( i )

Revised: [Equation]

Original: By Lemma A.2, we have
Revised: (cid:107)η w (cid:41) · [Equation] i = (cid:107) w ( i ) [Equation]

Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: {+(cid:107) w ( i ) 
+}
Revised: {+(cid:107) w ( i ) 
+}

Revised: (cid:107)η w

Revised: [Equation]

Original: ≤ c g η g (cid:32) 
Revised: (cid:32) L ( θ 0 ) [Equation] L

Revised: [Equation]

Original: L ( θ 0 ) − L min + m (cid:88) i = 
Revised: (cid:88) i = K i [Equation] (cid:88) i

Revised: = (cid:107)

Revised: [Equation]

Original: K i  (cid:33) ≤
Revised: ( i ) [Equation] (cid:107)η w (cid:33) (cid:32) L ( θ 0 )

Original: [Equation]
Revised: [Equation]

Original: (cid:18)
Revised: L

Original: [Equation]
Revised: [Equation]

Original: +
Revised: (cid:88) i =

Original: [Equation]
Revised: [Equation]

Original: η 2w η g (cid:19) .
Revised: i (cid:33) [Equation] [Equation]

Original: Combining them together we have
Revised: Combining them together we have

Original: T · min 0 ≤ t<T (cid:40) m (cid:88) i =

Original: (cid:13)(cid:13)(cid:13) ∇ v ( i ) t L ( V t ; g t ) (cid:13)(cid:13)(cid:13) 2 2 + (cid:107) g t (cid:107)(cid:41) ≤

Original: m (cid:88) i =

Original: T − 1 (cid:88) t

Original: = (cid:13)(cid:13)(cid:13) ∇ v ( i ) t L ( V t ; g t ) (cid:13)(cid:13)(cid:13) 2 2 + T − 1 (cid:88) t = (cid:107)∇ g t L ( V t ; g t ) (cid:107)≤ ˜ O (cid:18)η w + η 4w + 1 + η 2w η g (cid:19) .
Revised: [Equation]

Original: Thus min 0 ≤ t<T (cid:107)∇L ( V t , g t ) (cid:107) 2 converges in the rate of min 0 ≤ t<T (cid:107)∇L[-( V t , g t ) (cid:107) 
 =-]
Revised: Thus min 0 ≤ t<T (cid:107)∇L ( V t , g t ) (cid:107) 2 converges in the rate of min 0 ≤ t<T (cid:107)∇L[-( V t , g t ) (cid:107) 
 =-]

Original: ˜ O (cid:18) 1 √ T (cid:18) 1 √ η w + η 2w + 1 + η w √ η g (cid:19)(cid:19) .
Revised: [Equation]

Original: B P ROOF FOR S TOCHASTIC G RADIENT D ESCENT
Revised: B P ROOF FOR S TOCHASTIC G RADIENT D ESCENT

Original: Let F t = σ { z 0 , . .
Revised: Let F t = σ { z 0 , . .

Original: z t − 1 } be the ﬁltration, where σ {·} denotes the sigma ﬁeld.
Revised: z t − 1 } be the ﬁltration, where σ {·} denotes the sigma ﬁeld.

Original: We use L t :=
Revised: We use L t :=

Original: L z t ( θ t ) , F t :=
Revised: L z t ( θ t ) , F t :=

Original: F z t ( θ t ) for simplicity.
Revised: F z t ( θ t ) for simplicity.

Original: As usual, we deﬁne v ( i ) t = w ( i ) t / (cid:107) w ( i ) t (cid:107) 2 .
Revised: As usual, we deﬁne v ( i ) t = w ( i ) t / (cid:107) w ( i ) t (cid:107) 2 .

Original: We use the notations ∇ v ( i )
Revised: We use the notations ∇ v ( i )

Original: t L t := ∇ v ( i ) t
Revised: t L t := ∇ v ( i ) t

Original: L z t ( V t , g t ) and ∇ v ( i ) t F t := ∇ v ( i ) t F z t ( V t , g t ) for short.
Revised: L z t ( V t , g t ) and ∇ v ( i ) t F t := ∇ v ( i ) t F z t ( V t , g t ) for short.

Original: Let ∆ w ( i ) t = − η w ,t ∇ w ( i ) t F t , ∆ g t = − η g ,t ∇ g t F t .
Revised: Let ∆ w ( i ) t = − η w ,t ∇ w ( i ) t F t , ∆ g t = − η g ,t ∇ g t F t .

Original: Lemma B.1.
Revised: Lemma B.1.

Original: For any a 0 , . . . , a T ∈ [0 , B ] with a 0 > 0 , 
Revised: For any a 0 , . . . , a T ∈ [0 , B ] with a 0 > 0 , [Equation] [Equation] t

Original: [Equation]
Revised: [Equation]

Revised: + 1 + 2 B/

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let t i be the minimum 1 ≤ t ≤ T such that (cid:80) t − 1 τ =

Revised: Let t i be the minimum 1 ≤ t ≤ T such that (cid:80) t − 1 τ =

Original: a τ ≥ a 0 · 2 i .
Revised: a τ ≥ a 0 · 2 i .

Original: Let k be the maximum i such that t i exists.
Revised: Let k be the maximum i such that t i exists.

Original: Let t k +

Revised: Let t k +

Original: =
Revised: =

Original: T + 1 .
Revised: T + 1 .

Original: Then we know that
Revised: Then we know that

Original: [Equation]
Revised: [Equation]

Revised: 1 + B a

Revised: [Equation]

Revised: 2 −

Original:  Thus, (cid:80) 
Revised: [Equation] Thus, (cid:80) T t = [Equation] (cid:80)

Original: [Equation]
Revised: [Equation]

Revised: + Ba

Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: (cid:17) =

Revised: k + 1 + 2 Ba 0 ≤ log 2 (cid:16) (cid:80) T

Original: [Equation]
Revised: [Equation]

Revised: 1 t =0 a t /a 0 (cid:17) + 1 + 2 B/a 0 .

Original: Lemma B.2.
Revised: Lemma B.2.

Original: Fix T > 0 .
Revised: Fix T > 0 .

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Revised: v ( i ) t

Revised: L t (cid:107) 2 2 + C i

Revised: T − 1 (cid:88) t

Original: {+= η 2w ,t (cid:107) w ( i ) t (cid:107) 
+}
Revised: {+= η 2w ,t (cid:107) w ( i ) t (cid:107) 
+}

Original: {+(cid:107)∇ v ( i ) t F t (cid:107) 
+}
Revised: {+(cid:107)∇ v ( i ) t F t (cid:107) 
+}

Revised: R := − 1 2 c g

Revised: T − 1 (cid:88) t

Original: {+= η g ,t (cid:107)∇ g t L t (cid:107) 
+}
Revised: {+= η g ,t (cid:107)∇ g t L t (cid:107) 
+}

Original:  Then E [ L T ] − L 0 ≤ (cid:80) mi =1 E [ S i ] + E
Revised: + U T − 1 (cid:88) t = η 2g ,t G g Then E [ L T ] − L 0 ≤ (cid:80) mi =1 E [ S i ] + E

Original: [ R ] .
Revised: [ R ] .

Original: Proof.
Revised: Proof.

Original: Conditioned on F t , by Taylor expansion, we have
Revised: Conditioned on F t , by Taylor expansion, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where Q t is
Revised: where Q t is

Original: [Equation]

Original: i = m (cid:88) j

Original: =

Original: L vv ij (cid:13)(cid:13)(cid:13) w ( i ) t (cid:13)(cid:13)(cid:13) 2 (cid:13)(cid:13)(cid:13) w ( j ) t (cid:13)(cid:13)(cid:13) 2 (cid:107) ∆ w ( i ) t (cid:107) 2 (cid:107) ∆ w ( j ) t (cid:107)+ m (cid:88) i =

Original: L vg i (cid:13)(cid:13)(cid:13) w ( i ) t (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:107) ∆ w ( i ) t (cid:107) 2 (cid:107) ∆ g t (cid:107) 2 + 1 2 L gg (cid:107) ∆ g t (cid:107)

Original: By the inequality √ ab ≤ 12 a + 12 b, we have (cid:107) ∆ w ( i ) t (cid:107) [Equation] (cid:107) [Equation]
Revised: By the inequality √ ab ≤ 12 a + 12 b

Original: w ( j ) t (cid:107)

Original: L vv ij (cid:107)

Original: [Equation]
Revised: [Equation]

Original: i

Original: [Equation]

Original: (cid:107)

Original: [Equation]

Original: j

Original: [Equation]

Original: (cid:107)

Original: [Equation]

Original: ≤ 2 (cid:107)

Original: [Equation]

Original: w ( i ) t (cid:107) L vv ij (cid:107) w ( i [Equation] (cid:107) 22 + 2 (cid:107) ∆ w ( j ) t (cid:107)
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: (cid:107) w ( j ) t (cid:107)(cid:107)

Original: [Equation]

Original: w ( i

Original: [Equation]

Original: (cid:107)

Original: [Equation]

Original: (cid:107)

Original: [Equation]

Original: (cid:107)

Original: L

Original: [Equation]

Original: i (cid:107) w ( i

Original: [Equation]

Original: (cid:107)

Original: [Equation]

Original: ≤ (cid:107)

Original: [Equation]

Original: w ( i )

Original: [Equation]

Original: (cid:107) 22 L vg

Original: i

Original: [Equation]

Original: m/ ( c

Original: [Equation]

Original: L

Original: [Equation] ) (cid:107) [Equation] Note that E [ (cid:107) ∆ g t (cid:107) 22 | F t ] ≤ η 2g ,t (cid:0) (cid:107)∇ g t F t (cid:107) 

Revised: Note that E [ (cid:107) ∆ g t (cid:107) 22 | F t ] ≤ η 2g ,t (cid:0) (cid:107)∇ g t F t (cid:107)

Original: + G 2g (cid:1) .
Revised: + G 2g (cid:1) .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: Taking this into equation 21 and summing up for all t [Equation] we have
Revised: Taking this into equation 21 and summing up for all t , we have

Original: [Equation]
Revised: [Equation]

Original: Lemma B.3.
Revised: Lemma B.3.

Original: For any T ≥ 0 , 1 ≤ i ≤ m , we have
Revised: For any T ≥ 0 , 1 ≤ i ≤ m , we have

Original: [Equation]
Revised: [Equation]

Original: w

Original: [Equation]

Original: T

Original: [Equation]

Original: Proof.
Revised: Proof.

Original: By Lemma 2.4, we have
Revised: By Lemma 2.4, we have

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  
Revised: η [Equation] [Equation] ii

Original: [Equation]
Revised: [Equation]

Original: ( πL vv ii )
Revised: vv ii [Equation]  

Original: [Equation]
Revised: [Equation]

Original: [-(cid:32) πL [Equation] ii (cid:107) w ( i ) 
 (cid:107)-]
Revised: [-(cid:32) πL [Equation] ii (cid:107) w ( i ) 
 (cid:107)-] 

Original: [Equation]
Revised: [Equation]

Original: (cid:33) 4   . For 0 ≤ α < 1 / 2 , (cid:80) T − 1 τ = 1 (
Revised: πL

Original: [Equation]
Revised: [Equation]

Original: [Equation] = O ( T [Equation]
Revised: ii )

Original: [Equation]
Revised: [Equation]

Original: ) , so η[-w ,T (cid:107) w ( i ) T (cid:107) 
 = ˜Ω( T − 1 / 2 ) ; for α = 1 / 2 , (cid:80) T − 1 τ = [Equation] [Equation]-]
Revised: η[-w ,T (cid:107) w ( i ) T (cid:107) 
 = ˜Ω( T − 1 / 2 ) ; for α = 1 / 2 , (cid:80) T − 1 τ = [Equation] [Equation]-]

Original: [Equation]
Revised: [Equation]

Original: O (log T ) , so
Revised: ii

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: T{+i ) 
+} (cid:107)
Revised: {+i ) 
+} (cid:107)

Original: [Equation]
Revised: [Equation]

Original: ( i )
Revised:  

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: For 0 ≤ α < 1 / 2 , (cid:80) T − 1 τ = 1 ( τ +1)

Original: [Equation]
Revised: [Equation]

Revised: Lemma B

Original: [Equation]
Revised: [Equation]

Original: Lemma B.4.

Original: For T > 0 ,
Revised: For T > 0 ,

Original: • S i can be bounded by S i ≤ − η w ,T (cid:107) w ( i ) T (cid:107) 22 (cid:80) T − 1 t =

Revised: • S i can be bounded by S i ≤ − η w ,T (cid:107) w ( i ) T (cid:107) 22 (cid:80) T − 1 t =

Original: (cid:107)∇ v ( i )
Revised: (cid:107)∇ v ( i )

Original: t L t (cid:107) 

Revised: t L t (cid:107)

Original: + ˜ O (log T ) ; • R can be bounded by R ≤ − ˜Ω
Revised: + ˜ O (log T ) ; • R can be bounded by R ≤ − ˜Ω

Original: ( T − 1 / 2 ) (cid:80) T − 1 t =0 (cid:107)∇ g t L t (cid:107) 

Revised: ( T − 1 / 2 ) (cid:80) T − 1 t =0 (cid:107)∇ g t L t (cid:107)

Original: + ˜ O (log T ) .
Revised: + ˜ O (log T ) .

Original: Proof.
Revised: Proof.

Original: Fix i ∈ { 1 , . . .
Revised: Fix i ∈ { 1 , . . .

Original: , m } .
Revised: , m } .

Original: First we bound S i .
Revised: First we bound S i .

Original: Recall that
Revised: Recall that

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: is non-increasing, so

Original: [Equation]
Revised: [Equation]

Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: Also note that (cid:107) w ( i ) t (cid:107) 22 ≥ (cid:107) w ( i ) 

Revised: Also note that (cid:107) w ( i ) t (cid:107) 22 ≥ (cid:107) w ( i )

Original: (cid:107) 

Revised: (cid:107)

Original: + (cid:80) t − 1 τ =

Revised: + (cid:80) t − 1 τ =

Original: η 2w ,τ (cid:107)∇ w ( i ) t F t (cid:107) 22 .
Revised: η 2 w ,τ (cid:107)∇ w ( i ) t F t (cid:107) 22 .

Original: By Lemma B.1,
Revised: By Lemma B.1,

Original: C i
Revised: C i

Original: T − 1 (cid:88)
Revised: T − 1 (cid:88)

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We can get the bound for S i by combining equation 22 and equation 23.
Revised: We can get the bound for S i by combining equation 22 and equation 23.

Original: Now we bound R .
Revised: Now we bound R .

Original: Recall that
Revised: Recall that

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term can be bounded by − ˜Ω( T − 1 / 2 ) (cid:80) T − 1 t =

Revised: The ﬁrst term can be bounded by − ˜Ω( T − 1 / 2 ) (cid:80) T − 1 t =

Original: (cid:107)∇ g t L t (cid:107) 22 by noticing[-that η g ,t ≥ η g ,T = Ω ( T − 1 / 2 ) . The second term can be bounded by ˜ O (log T ) since (cid:80) T − 1 t =
-]
Revised: (cid:107)∇ g t L t (cid:107) 22 by noticing[-that η g ,t ≥ η g ,T = Ω ( T − 1 / 2 ) . The second term can be bounded by ˜ O (log T ) since (cid:80) T − 1 t =
-]

Original: η 2g ,t

Original: [-= (cid:80) T − 1 t =
-]
Revised: [-= (cid:80) T − 1 t =
-]

Original: ˜ O

Original: (1 /t )

Original: =

Original: ˜ O (log T ) .

Original: Proof for Theorem 4.2.
Revised: Proof for Theorem 4.2.

Original: Combining Lemma B.2 and Lemma B.4, for 0 ≤ α < 1 / 2 , we have
Revised: Combining Lemma B.2 and Lemma B.4, for 0 ≤ α < 1 / 2 , we have

Original: [Equation]
Revised: [Equation]

Original: Thus, min
Revised: Thus, min

Original: [Equation]
Revised: [Equation]

Original: E

Original: [Equation]

Original: Similarly, for α = 1 / 2 , we have
Revised: Similarly, for α = 1 / 2 , we have

Original: [Equation]
Revised: [Equation]

Original: Thus, min [Equation] [Table] [Equation]
Revised: Thus, min 0 ≤ t<

Original: [Table]

Original: √

Original: T

Original: [Equation]
Revised: [Equation]

Original: C P ROOF FOR THE SMOOTHNESS OF THE MOTIVATING NEURAL NETWORK
Revised: C P ROOF FOR THE SMOOTHNESS OF THE MOTIVATING NEURAL NETWORK

Original: In this section we prove that the modiﬁed version of the motivating neural network does meet the assumptions in Section 2.4.
Revised: In this section we prove that the modiﬁed version of the motivating neural network does meet the assumptions in Section 2.4.

Original: More speciﬁcally, we assume:
Revised: More speciﬁcally, we assume:

Original: • We use the network structure Φ in Section 2.1 with the smoothed variant of BN as described in Section 2.4; • The objective f y ( · ) is twice continuously differentiable, lower bounded by f min and Lipschitz ( | f (cid:48)
Revised: • We use the network structure Φ in Section 2.1 with the smoothed variant of BN as described in Section 2.4; • The objective f y ( · ) is twice continuously differentiable, lower bounded by f min and Lipschitz ( | f (cid:48)

Original: y
Revised: y

Original: (ˆ y ) | ≤ α f ); • The activation σ ( · ) is twice continuously differentiable and Lipschitz ( | f (cid:48)
Revised: (ˆ y ) | ≤ α f ); • The activation σ ( · ) is twice continuously differentiable and Lipschitz ( | f (cid:48)

Original: y (ˆ y ) | ≤ α σ ); • We add an extra weight decay (L2 regularization) term λ 2 (cid:107) g (cid:107) 22 to the loss in equation 2 for some λ > 0 .
Revised: y (ˆ y ) | ≤ α σ ); • We add an extra weight decay (L2 regularization) term λ 2 (cid:107) g (cid:107) 22 to the loss in equation 2 for some λ > 0 .

Original: First, we show that g t (containing all scale and shift parameters in BN) is bounded during the training process.
Revised: First, we show that g t (containing all scale and shift parameters in BN) is bounded during the training process.

Original: Then the smoothness follows compactness using Extreme Value Theorem.
Revised: Then the smoothness follows compactness using Extreme Value Theorem.

Original: We use the following lemma to calculate back propagation:
Revised: We use the following lemma to calculate back propagation:

Original: Lemma C.1.
Revised: Lemma C.1.

Original: Let x 1 , . . .
Revised: Let x 1 , . . .

Original: , x B be a set of vectors.
Revised: , x B be a set of vectors.

Original: Let u := E b ∈
Revised: Let u := E b ∈

Original: [ B ] [ x b ] and S :=
Revised: [ B ] [ x b ] and S :=

Original: Var b ∈ [ B ] ( x b ) .
Revised: Var b ∈ [ B ] ( x b ) .

Original: Let z b := γ w (cid:62) ( x b − u ) (cid:107) w (cid:107) S + (cid:15)I + β.
Revised: Let [Equation]

Original: Let y :
Revised: Let y :

Original: =
Revised: =

Original: f ( z 1 , . .
Revised: f ( z 1 , . .

Original: . , z B ) =
Revised: . , z B ) =

Original: f ( z ) for some function f .
Revised: f ( z ) for some function f .

Original: If (cid:107)∇ f ( z ) (cid:107) 2 ≤ G , then
Revised: If (cid:107)∇ f ( z ) (cid:107) 2 ≤ G , then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let ˜ x ∈ R B be the vector where
Revised: Let ˜ x ∈ R B be the vector where

Original: ˜ x b :=
Revised: ˜ x b :=

Original: ( w (cid:62) ( x b − u )) / (cid:107) w (cid:107) S
Revised: ( w (cid:62) ( x b − u )) / (cid:107) w (cid:107) S

Original: + (cid:15)I .
Revised: + (cid:15)I .

Original: It is easy to see (cid:107)
Revised: It is easy to see (cid:107)

Original: ˜
Revised: ˜

Original: x
Revised: x

Original: (cid:107) 2 2 ≤ B .
Revised: (cid:107) 22 ≤ B .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: For x b , we have
Revised: For x b , we have

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Since ( w (cid:62) ( x b − u )) 2 ≤ B (cid:107) w (cid:107) 2 S + (cid:15)I , (cid:107)∇x b z b (cid:48) (cid:107) 2 ≤ γ (cid:18) (cid:107) w (cid:107)
Revised: Since ( w (cid:62) ( x b − u )) 2 ≤ B (cid:107) w (cid:107) 2 S + (cid:15)I , (cid:107)∇

Original: [Equation]
Revised: [Equation]

Original: (cid:15)I (
Revised: γ

Original: [Equation]
Revised: [Equation]

Original: − 1 /B ) (cid:107) w (cid:107) 2 +
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: w
Revised: B

Original: [Equation]
Revised: [Equation]

Original: S +(cid:15)I [Equation] [Equation] w (cid:107)
Revised: +

Original: [Equation]
Revised: [Equation]

Original: 3 γ (cid:15) . Thus,(cid:107)∇ [Equation] b
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: √
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Lemma C.2.
Revised: Lemma C.2.

Original: If (cid:107) g 0 (cid:107) 2 is bounded by a constant, there exists some constant K such that (cid:107) g t (cid:107) 2 ≤ K .
Revised: If (cid:107) g 0 (cid:107) 2 is bounded by a constant, there exists some constant K such that (cid:107) g t (cid:107) 2 ≤ K .

Original: Proof.
Revised: Proof.

Original: Fix a time t in the training process.
Revised: Fix a time t in the training process.

Original: Consider the process of back propagation.
Revised: Consider the process of back propagation.

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: where x ( i ) b,k is the output of the k -th neuron in the i -th layer in the b -th data sample in the batch.
Revised: where x ( i ) b,k is the output of the k -th neuron in the i -th layer in the b -th data sample in the batch.

Original: By the Lipschitzness of the objective, R L can be bounded by a constant.
Revised: By the Lipschitzness of the objective, R L can be bounded by a constant.

Original: If R i can be bounded by a constant, then by the Lipschitzness of σ and Lemma C.1, the gradient of γ and β in layer i can also be bounded by a constant.
Revised: If R i can be bounded by a constant, then by the Lipschitzness of σ and Lemma C.1, the gradient of γ and β in layer i can also be bounded by a constant.

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Thus γ and β in layer i can be bounded by a constant since
Revised: Thus γ and β in layer i can be bounded by a constant since

Original: [Equation]
Revised: [Equation]

Original: Also Lemma C.1 and the Lipschitzness of σ imply that R i − 1 can be bounded if R i and γ in the layer i can be bounded by a constant.
Revised: Also Lemma C.1 and the Lipschitzness of σ imply that R i − 1 can be bounded if R i and γ in the layer i can be bounded by a constant.

Original: Using a simple induction, we can prove the existence of K for bounding the norm of (cid:107) g t (cid:107) 2 for all time t .
Revised: Using a simple induction, we can prove the existence of K for bounding the norm of (cid:107) g t (cid:107) 2 for all time t .

Original: Theorem C.3.
Revised: Theorem C.3.

Original: If (cid:107) g 0 (cid:107) 2 is bounded by a constant, then Φ satisﬁes the assumptions in Section 2.4.
Revised: If (cid:107) g 0 (cid:107) 2 is bounded by a constant, then Φ satisﬁes the assumptions in Section 2.4.

Original: Proof.
Revised: Proof.

Original: Let C be the set of parameters θ satisfying (cid:107) g (cid:107) ≤ K and (cid:107) w ( i ) (cid:107) 2 = 1 for all 1 ≤ i ≤ m .
Revised: Let C be the set of parameters θ satisfying (cid:107) g (cid:107) ≤ K and (cid:107) w ( i ) (cid:107) 2 = 1 for all 1 ≤ i ≤ m .

Original: By Lemma C.2, C contains the set of ˜ θ associated with the points lying between each pair of θ t and θ t +1 (including the endpoints).
Revised: By Lemma C.2, C contains the set of ˜ θ associated with the points lying between each pair of θ t and θ t +1 (including the endpoints).

Original: It is easy to show that F z (˜ θ ) is twice continously differentiable.
Revised: It is easy to show that F z (˜ θ ) is twice continously differentiable.

Original: Since C is compact, by the Extreme Value Theorem, there must exist such constants L vv ij , L vg i , L gg , G g for upper bounding the smoothness and the difference of gradients.
Revised: Since C is compact, by the Extreme Value Theorem, there must exist such constants L vv ij , L vg i , L gg , G g for upper bounding the smoothness and the difference of gradients.

Original: D E XPERIMENTS
Revised: D E XPERIMENTS

Original: In this section, we provide experimental evidence showing that the auto rate-tuning behavior does empower BN in the optimization aspect.
Revised: In this section, we provide experimental evidence showing that the auto rate-tuning behavior does empower BN in the optimization aspect.

Original: We trained a modiﬁed version of VGGNet (Simonyan & Zisserman, 2014) on Tensorﬂow.
Revised: We trained a modiﬁed version of VGGNet (Simonyan & Zisserman, 2014) on Tensorﬂow.

Original: This network has 2 × conv64 , pooling, 3 × conv128 , pooling, 3 × conv256 , pooling, 3 × conv512 , pooling, 3 × conv512 , pooling, fc512 , fc10 layers in order.
Revised: This network has 2 × conv64 , pooling, 3 × conv128 , pooling, 3 × conv256 , pooling, 3 × conv512 , pooling, 3 × conv512 , pooling, fc512 , fc10 layers in order.

Original: Each convolutional layer has kernel size 3 × 3 and stride 1 .
Revised: Each convolutional layer has kernel size 3 × 3 and stride 1 .

Original: ReLU is used as the activation function after each convolutional or fullyconnected layer.
Revised: ReLU is used as the activation function after each convolutional or fullyconnected layer.

Original: We add a BN layer right before each ReLU .
Revised: We add a BN layer right before each ReLU .

Original: We set (cid:15) = 0 in each BN, since we observed that the network works equally well for (cid:15) being 0 or an small number (such as 10 − 3 , the default value in Tensorﬂow).
Revised: We set (cid:15) = 0 in each BN, since we observed that the network works equally well for (cid:15) being 0 or an small number (such as 10 − 3 , the default value in Tensorﬂow).

Original: We initialize the parameters according to the default conﬁguration in
Revised: We initialize the parameters according to the default conﬁguration in

Original: Tensorﬂow: all the weights are initialized by Glorot uniform initializer (Glorot & Bengio, 2010); β and γ in BN are initialized by 0 and 1 , respectively.
Revised: Tensorﬂow: all the weights are initialized by Glorot uniform initializer (Glorot & Bengio, 2010); β and γ in BN are initialized by 0 and 1 , respectively.

Original: In this network, every kernel is scale-invariant, and for every BN layer except the last one, the concatenation of all β and γ parameters in this BN is also scale-invariant.
Revised: In this network, every kernel is scale-invariant, and for every BN layer except the last one, the concatenation of all β and γ parameters in this BN is also scale-invariant.

Original: Only β and γ parameters in the last BN are scale-variant (See Section 2.1).
Revised: Only β and γ parameters in the last BN are scale-variant (See Section 2.1).

Original: We consider the training in following two settings:
Revised: We consider the training in following two settings:

Original: Train the network using the standard SGD(No momentum, learning rate decay ,weight decay and dropout); 2.
Revised: Train the network using the standard SGD (No momentum, learning rate decay, weight decay and dropout); 2.

Original: Train the network using Projected SGD (PSGD): at each iteration, one ﬁrst takes a step proportional to the negative of the gradient calculated in a random batch, and then projects each scale-invariant parameter to the sphere with radius equal to its 2 -norm before this iteration, i.e., rescales each scale-invariant parameter so that each maintains its length during training.
Revised: Train the network using Projected SGD (PSGD): at each iteration, one ﬁrst takes a step proportional to the negative of the gradient calculated in a random batch, and then projects each scale-invariant parameter to the sphere with radius equal to its 2 -norm before this iteration, i.e., rescales each scale-invariant parameter so that each maintains its length during training.

Original: Note that the projection in Setting 2 removes the adaptivity of the learning rates in the corresponding intrinsic optimization problem, i.e., G ( i ) t in equation 9 remains constant during the training.
Revised: Note that the projection in Setting 2 removes the adaptivity of the learning rates in the corresponding intrinsic optimization problem, i.e., G ( i ) t in equation 9 remains constant during the training.

Original: Thus, by comparing Setting 1 and Setting 2, we can know whether or not the auto-tuning behavior of BN shown in theory is effective in practice.
Revised: Thus, by comparing Setting 1 and Setting 2, we can know whether or not the auto-tuning behavior of BN shown in theory is effective in practice.

Original: D. S EPARATE L EARNING R ATES
Revised: D. S EPARATE L EARNING R ATES

Original: As in our theoretical analysis, we consider what will happen if we set two learning rates separately for scale-invariant and scale-variant parameters.
Revised: As in our theoretical analysis, we consider what will happen if we set two learning rates separately for scale-invariant and scale-variant parameters.

Original: We train the network in either setting with different learning rates ranging from 10 − 2 to 10 2 for 100 epochs.
Revised: We train the network in either setting with different learning rates ranging from 10 − 2 to 10 2 for 100 epochs.

Original: First, we ﬁx the learning rate for scale-variant ones to 0 .
Revised: First, we ﬁx the learning rate for scale-variant ones to 0 .

Original: 1 , and try different learning rates for scaleinvariant ones.
Revised: 1 , and try different learning rates for scaleinvariant ones.

Original: As shown in Figure 1, for small learning rates (such as 0 . 1 ), the training processes of networks in Setting 1 and 2 are very similar.
Revised: As shown in Figure 1, for small learning rates (such as 0 . 1 ), the training processes of networks in Setting 1 and 2 are very similar.

Original: But for larger learning rates, networks in Settingcan still converge to 0 for all the learning rates we tried, while networks in Setting 2 got stuck with relatively large training loss.
Revised: But for larger learning rates, networks in Settingcan still converge to 0 for all the learning rates we tried, while networks in Setting 2 got stuck with relatively large training loss.

Original: This suggests that the auto-tuning behavior of BN does takes effect when the learning rate is large, and it matches with the claimed effect of BN in Ioffe & Szegedy (2015) that BN enables us to use a higher learning rate.
Revised: This suggests that the auto-tuning behavior of BN does takes effect when the learning rate is large, and it matches with the claimed effect of BN in (Ioffe & Szegedy, 2015) that BN enables us to use a higher learning rate.

Original: Though our theoretical analysis cannot be directly applied to the network we trained due to the non-smoothness of the loss function, the experiment results match with what we expect in our analysis.
Revised: Though our theoretical analysis cannot be directly applied to the network we trained due to the non-smoothness of the loss function, the experiment results match with what we expect in our analysis.

Original: D. U NIFIED L EARNING R ATE
Revised: D. U NIFIED L EARNING R ATE

Original: Next, we consider the case in which we train the network with a uniﬁed learning rate for both scaleinvariant and scale-variant parameters.
Revised: Next, we consider the case in which we train the network with a uniﬁed learning rate for both scaleinvariant and scale-variant parameters.

Original: We also compare Setting 1 and 2 with the setting in which we train the network with all the BN layers removed using SGD (we call it Setting 3).
Revised: We also compare Setting 1 and 2 with the setting in which we train the network with all the BN layers removed using SGD (we call it Setting 3).

Original: As shown in Figure 2, the training loss of networks in Setting 1 converges to 0 .
Revised: As shown in Figure 2, the training loss of networks in Setting 1 converges to 0 .

Original: On the contrast, the training loss of networks in Setting 2 and 3 fails to converge to 0 when a large learning rate is used, and in some cases the loss diverges to inﬁnity or NaN.
Revised: On the contrast, the training loss of networks in Setting 2 and 3 fails to converge to 0 when a large learning rate is used, and in some cases the loss diverges to inﬁnity or NaN.

Original: This suggests that the auto-tuning behavior of BN has an effective role in the case that a uniﬁed learning rate is set for all parameters.
Revised: This suggests that the auto-tuning behavior of BN has an effective role in the case that a uniﬁed learning rate is set for all parameters.

Revised: For a fair comparison, we also trained neural networks in Setting 3 with initialization essentially equivalent to the ones with BN.

Revised: This is done in the same way as (Kr¨ahenb¨uhl et al., 2015; Mishkin & Matas, 2015) and Section 3 of (Salimans & Kingma, 2016): we ﬁrst randomly initialize the parameters, then feed the ﬁrst batch into the network and adjust the scaling and bias of each neuron to make its outputs have zero mean and unit variance.

Revised: In this way, the loss of the networks converges to 0 when the learning rate is smaller than 10 − 2 . 0 , but for a slightly larger learning rate such as 10 − 1 . 8 , the loss fails to converge to 0 , and sometimes even diverges to inﬁnity or NaN. Compared with experimental results in Setting 1, this suggests that the robustness of training brought by BN is independent of the fact that BN changes the effective initialization of parameters.

Original: D. G ENERALIZATION
Revised: D. G ENERALIZATION

Original: Despite in Setting 1 the convergence of training loss for different learning rates, the convergence points can be different, which lead to different performances on test data.
Revised: Despite in Setting 1 the convergence of training loss for different learning rates, the convergence points can be different, which lead to different performances on test data.

Original: In Figure 3, we plot the test accuracy of networks trained in Setting 1 and 2 using different uniﬁed learning rates, or separate learning rates with the learning rate for scale-variant parameters ﬁxed to 0 .
Revised: In Figure 3, we plot the test accuracy of networks trained in Setting 1 and 2 using different uniﬁed learning rates, or separate learning rates with the learning rate for scale-variant parameters ﬁxed to 0 .

Original: As shown in the Figure 3, the test accuracy of networks in Setting 2 decreases as the learning rate increases over 0 .
Revised: As shown in the Figure 3, the test accuracy of networks in Setting 2 decreases as the learning rate increases over 0 .

Original: 1 , while the test accuracy of networks in Setting 1 remains higher than 75% .
Revised: 1 , while the test accuracy of networks in Setting 1 remains higher than 75% .

Original: The main reason that the network in Setting 2 doesn’t perform well is underﬁtting, i.e. the network in Setting 2 fails to ﬁt the training data well when learning rate is large.
Revised: The main reason that the network in Setting 2 doesn’t perform well is underﬁtting, i.e. the network in Setting 2 fails to ﬁt the training data well when learning rate is large.

Original: This suggests that the autotuning behavior of BN also beneﬁts generalization since such behavior allows the algorithm to pick learning rates from a wider range while still converging to small test error.
Revised: This suggests that the autotuning behavior of BN also beneﬁts generalization since such behavior allows the algorithm to pick learning rates from a wider range while still converging to small test error.