Original: Channel Normalization in Convolutional Neural Networks avoids Vanishing Gradients 
Revised: Channel Normalization in Convolutional Neural Networks avoids Vanishing Gradients June 12,

Original: Abstract
Revised: Abstract

Original: Normalization layers are widely used in deep neural networks to stabilize training.
Revised: Normalization layers are widely used in deep neural networks to stabilize training.

Original: In this paper, we consider the training of convolutional neural networks with gradient descent on a single training example.
Revised: In this paper, we consider the training of convolutional neural networks with gradient descent on a single training example.

Original: This optimization problem arrises in recent approaches for solving inverse problems such as the deep image prior or the deep decoder.
Revised: This optimization problem arises in recent approaches for solving inverse problems such as the deep image prior or the deep decoder.

Original: We show that for this setup, channel normalization, which centers and normalizes each channel individually, avoids vanishing gradients, whereas without normalization, gradients vanish which prevents eﬃcient optimization.
Revised: We show that for this setup, channel normalization, which centers and normalizes each channel individually, avoids vanishing gradients, whereas without normalization, gradients vanish which prevents eﬃcient optimization.

Original: We then demonstrate that this eﬀect prevails in deep single-channel linear convolutional networks, and analytically show that without channel normalization, gradient descent takes at least exponentially many steps to come close to an optimum.
Revised: This eﬀect prevails in deep single-channel linear convolutional networks, and we show that without channel normalization, gradient descent takes at least exponentially many steps to come close to an optimum.

Original: Contrary, with channel normalization, the gradients remain bounded, thus avoiding exploding gradients.
Revised: Contrary, with channel normalization, the gradients remain bounded, thus avoiding exploding gradients.

Original:  Introduction
Revised:  Introduction

Original: Deep learning and in particular convolutional neural networks have signiﬁcantly improved the stateof-the-art in computer vision, image generation, and computational imaging, among many other ﬁelds.
Revised: Deep learning and in particular convolutional neural networks have signiﬁcantly improved the state-of-theart in computer vision, image generation, and computational imaging, among many other ﬁelds.

Original: Deep neural networks are typically trained using ﬁrst order methods such as gradient descent and the stochastic gradient methods.
Revised: Deep neural networks are typically trained using ﬁrst order methods such as gradient descent and the stochastic gradient method.

Original: However, the corresponding loss function is non-convex and therefore, depending on the initialization, convergence to an optimum is not guaranteed, and ﬁrst order methods sometimes suﬀer from  vanishing or exploding gradients.
Revised: However, the corresponding loss function is non-convex and therefore, depending on the initialization, convergence to an optimum is not guaranteed, and ﬁrst order methods sometimes suﬀer from unstable training and/or vanishing or exploding gradients.

Original: Normalization layers are widely used to avoid vanishing or exploding gradients, stabilize training,  enable learning with higher rates, andenable faster convergence.
Revised: Normalization layers are widely used to avoid vanishing or exploding gradients, stabilize training, and enable learning with higher rates and faster convergence.

Original: The perhaps most popular  technique is batch normalization [IS15]; but a number of (often closely related) variations and alternatives have been proposed such as layer normalization [LB+16], weight normalization [SK16], and instance normalization [Uly+16].
Revised: The perhaps most popular normalization technique is batch normalization [IS15]; but a number of (often closely related) variations and alternatives have been proposed such as layer normalization [LB+16], weight normalization [SK16], and instance normalization [Uly+16].

Original: A variety of recent works have proposed diﬀerent explanations for the success of normalization layers.
Revised: Avarietyof recent works have proposed diﬀerent explanations for the success of normalization layers.

Original: While the original batch normalization paper [IS15] suggested that batch normalization aids optimization by reducing a quantity called internal covariate shift, Santurkar et al.
Revised: The original batch normalization paper [IS15] suggested that batch normalization aids optimization by reducing a quantity called internal covariate shift. In contrast, Santurkar et al.

Original: [San+18] provide empirical evidence contradicting this explanation, and instead explain that batch normalization reparameterizes the underlying optimization problem and  make its landscape signiﬁcantly smoother.
Revised: [San+18] reason that batch normalization reparameterizes the underlying optimization problem and thereby make its landscape signiﬁcantly smoother.

Original: Kohler et al.
Revised: Kohler et al.

Original: [Koh+18] linked batch normalization to weight normalization [SK16], and pointed out that batch normalization accelerates the training process by splitting the optimization task into optimizing the length and direction of the parameters separately.
Revised: [Koh+19] linked batch normalization to weight normalization [SK16], and pointed out that batch normalization accelerates the training process by splitting the optimization task into optimizing the length and direction of the parameters separately, and Bjorck [Bjo+18] argues that (batch) normalization enables training with larger training rates.

Original: Wehasten to add that whether normalization layers are useful or not depends strongly on the architecture and initialization.
Revised: We add that whether normalization layers are useful or not depends strongly on the architecture and initialization.

Original: For example carefully initialized deep residual networks can be trainedsuccessfully without any normalization layers [Zha+19].
Revised: For example carefully initialized deep residual networks can be trained without any normalization layers [Zha+19].

Original: In this paper, we studynormalization techniques in the context of a convolutional generator network. Speciﬁcally, we study channel normalization, which is a special case of a number of the above mentioned normalization techniques.
Revised: In this paper, we study channel normalization, which is a special case of a number of the above mentioned normalization techniques, in the context of a convolutional generator network.

Original: Channel normalization standardizes each channel in a convolutional neural network, individually for each training example, and scales and shifts the resulting vector with a (learned) scalar.
Revised: Channel normalization standardizes each channel in a convolutional neural network, individually for each training example, and scales and shifts the resulting vector with a (trainable) scalar.

Original: Channel normalization is equivalent to  batch normalization for a single training example (then the batch size is one) and to instance normalization.
Revised: Channel normalization is equivalent to instance normalization [Uly+16] and to batch normalization for a single training example (then the batch size is one).

Original: [Figure] that channel normalization in convolutional networks avoids exploding and vanishing gradients in the context of training a convolutional network  on a single training example, as common for solving inverse problems [HH18; Uly+18].
Revised: We ﬁrst train a convolutional network with gradient descent on a single training example, a problem that occurs in solving inverse problems without training data [HH19; Uly+18], and demonstrate that channel normalization avoids exploding and vanishing gradients and enables reaching a close-to-optimal point.

Original: Contrary, without channel normalization, the optimization does not converge to an optimum in a reasonable number of iterations.
Revised: Contrary, without channel normalization, gradient descent does not converge to an optimum in a reasonable number of iterations.

Original: We then show analytically, for a special case of linear convolutional networks that  without channel normalization, gradient descent requires at least exponentially many steps to convergeunder mild initialization conditions.
Revised: We then show analytically, for a special case of linear convolutional networks, that [Figure] without channel normalization, gradient descent requires at least exponentially many steps to converge

Original: The aforementioned works [IS15; San+18; Koh+18] have studied normalization techniques by focusing on shallow networks (i e., networks with one hidden layer), since analytical gradient expressions for deep networks with non-linearities are  intractable.
Revised: The aforementioned works [IS15; San+18; Koh+19] have studied normalization techniques by focusing on shallow networks (i.e., networks with one hidden layer), since analytical gradient expressions for deep networks with non-linearities are almost intractable.

Original: Here, we sidestep this hurdle by exploring a simple model–linear convolutional neural networks with a single channel.
Revised: Here, we sidestep this hurdle by exploring a simpler model, speciﬁcally a linear convolutional network with a single channel.

Original: Studying such a simple model is justiﬁed by noting that even for this simple model, normalization is crucial for fast convergence.
Revised: Studying such a simple model is justiﬁed by observing that even for this simple model, normalization is critical for fast convergence.

Original:  Channel normalizationin convolutional networks
Revised:  Channel normalization

Original: We start by introducingthe channel normalizationoperation and then show empirically that it is critical for the optimization of convolutional generators trained on a single example.
Revised: We start by introducing channel normalization and then show empirically that it is critical for running gradient descent eﬃciently on a convolutional generators trained on a single example.

Original: The channel normalization operation normalizes each channel of a convolutional network individually.
Revised: The channel normalization operation normalizes each channel of a convolutional network individually.

Original: Let z ij be the input of  j -th channel and the i -th layer.
Revised: Let z ij be the input of the j -th channel and the i -th layer.

Original: Channel normalization performs the transformation z (cid:48) ij =
Revised: Channel normalization performs the transformation z (cid:48) ij =

Original: z ij − mean( z ij ) √ var( z ij )+ (cid:15) γ ij + β ij , where mean and var compute the empirical mean and variance, γ i,j and β ij are parameters learned independently for each channel, and (cid:15) is a ﬁxed small constant  for numerical stability.
Revised: z ij − mean( z ij ) √ var( z ij )+ (cid:15) γ ij + β ij , where mean and var compute the empirical mean and variance, γ i,j and β ij are parameters learned independently for each channel, and (cid:15) is a ﬁxed small constant added for numerical stability.

Original: We consider a variant of the deep decoder introduced in [HH18], which in its original form consists of bilinear upsampling and convolutional operations, followed by channel normalization and ReLU layers.
Revised: Weconsider a variant of the deep decoder introduced in [HH19].

Original: The network works well for image compression and for  a variety of inverse problems, when trained or ﬁtted to a single image only.
Revised: The network works well for image compression and for regularizing a variety of inverse problems, when trained or ﬁtted to a single image only.

Original: Speciﬁcally, we consider an extremely simple convolutional generator consisting of d = 5 3x3 convolutional layers, followed by channel normalization and ReLU activation functions.
Revised: Speciﬁcally, we consider an extremely simple convolutional generator consisting of d = 5 many 3x convolutional layers, followed by channel normalization and ReLU activation functions.

Original: Each layer has k = 32 channels, and the last layer is a 1x1 convolutional layer mapping the k channels to a single, 256x256 grayscale output image.
Revised: Each layer has k = 32 channels, and the last layer is a 1x1 convolutional layer mapping the k channels to a single, 256x grayscale output image.

Original: The input to the network is a 32x256x256 volume that is chosen randomly and is ﬁxed (i.e., we do not optimize over the input).
Revised: The input to the network is a 32x256x256 volume that is chosen randomly and is ﬁxed (i.e., we do not optimize over the input).

Original: Given an image x ∗ we then ﬁt the parameters of the network (i.e., the weights) by minimizing the loss L ( C ) =
Revised: Given an image x ∗ we then ﬁt the parameters of the network (i.e., the weights) by minimizing the loss L ( C ) =

Original: (cid:107) G ( C ) − x ∗ (cid:107) 22 with respect to the network parameters [Figure] loss gradients [Figure]
Revised: (cid:13)(cid:13) G ( C ) − x ∗ (cid:13)(cid:13) 2 2 with respect to the network parameters C using plain gradient descent with ﬁxed stepsize.

Original: C using plain gradient descent with a ﬁxed stepsize.

Original: Figure 1 shows the results for the phantom MRI image for a network with and without channel normalization.
Revised: Figure 1 shows the results for the phantom MRI image for a network with and without channel normalization.

Original: It can be seen that with channel normalization, the training loss converges rapidly and the gradients do not vanish.
Revised: With channel normalization, the training loss converges rapidly and the gradients do not vanish.

Original: Contrary, without normalization, the network does not converge to a small error (even though it has the capacity to represent the image exactly), and the gradients vanish.
Revised: Contrary, without normalization, the network does not converge to a small error (even though the network has the capacity to represent the image), and the gradients vanish.

Original: This eﬀect is not speciﬁc to the image (we have reproduced it using 100 randomly chosen images from imagenet), and it is also reproducible for a number of related convolutional generators, for example those including upsampling operations.
Revised: This eﬀect is not speciﬁc to the image (we have reproduced it using 100 randomly chosen images from imagenet), and it is also reproducible for a number of related convolutional generators, for example networks including upsampling operations.

Original:  Isolating the eﬀect of channel normalization
Revised:  Isolating the eﬀect of channel normalization

Original: We next show that to achieve the stabilizing eﬀect of channel normalization, the trainable coefﬁcients γ ij and β ij do not need to be learned and can be set to one and zero, respectively.
Revised: We next show that to achieve the stabilizing eﬀect of channel normalization, the trainable coeﬃcients γ ij and β ij do not need to be learned and can be set to one and zero, respectively.

Original: We also demonstrate that even for linear networks, channel normalization is critical to avoid vanishing gradients.
Revised: We also demonstrate that even for linear networks, channel normalization is critical to avoid vanishing gradients.

Original: This justiﬁes studying linear networks in the next section.
Revised: This justiﬁes our theoretical study of linear networks in the next section.

Revised: [Figure]

Original: Multiple Channels CNN: We  consider a network again only consisting of convolutional layers followed by channel normalization and ReLU activation functions.
Revised: Multiple Channels CNN: We ﬁrst consider a one-dimensional convolutional network, again only consisting of convolutional layers followed by channel normalization and ReLU activation functions.

Original: We set the dimension of input/output vector to n = 256, the number of channels to k = 4, number of hidden layers to d = 12, and convolution kernel size to 3.
Revised: We set the dimension of input/output vector to n = 256, the number of channels to k = 4, number of hidden layers to d = 12, and convolutional kernel size to 3.

Original: The entries of the input vector x are sampled from a standard uniform distribution.
Revised: The entries of the input vector x are sampled from a standard uniform distribution.

Original: As before, we minimize the least squares error with respect to the weight parameters using gradient descent with ﬁxed step size.
Revised: As before, we minimize the least squares loss with respect to the weight parameters using gradient descent with ﬁxed step size.

Original: We consider three diﬀerent normalization operations: the original channel normalization, a variant where γ ij = 1 and β ij =(called ﬁxed norm), and no normalization and ﬁt a simple step function.
Revised: We consider three diﬀerent normalization operations: the original channel normalization, a variant where γ ij = 1 and β ij = 0 (called ﬁxed norm), and no normalization. We consider the problem of ﬁtting a simple step function.

Original: The results  in Figureshow that bothversions of normalization  enable eﬃcient optimization, whereas without channel normalization the training error stops improving after a few iterations and the gradients vanish.
Revised: The results reported in Figure 2 show that both normalization versions enable eﬃcient optimization with gradient descent (the training error is near-zero), whereas without channel normalization the training error does not improve after a few iterations and the gradients vanish.

Original: Single Channel Linear CNN: Next, we consider an even simpler network with only one channel and without activation functions.
Revised: Single Channel Linear CNN: Next, we consider an even simpler network with only one channel and without activation functions.

Original: Without normalization, the network is linear.
Revised: Without normalization, the network is linear.

Original: We set the dimension of input/output vector to n = 64, number of hidden layers d = 10, and convolution kernel size k = 9.
Revised: We set the dimension of input/output vector to n = 64, number of hidden layers d = 10, and convolution kernel size k = 9.

Original: The results are very similar to the previous experiment in Figure 2 (see Figurein the appendix), and demonstrates the critical role of channel normalization.
Revised: The results are very similar to the previous experiment in Figure 2 (see Figure 4 in the appendix), and demonstrates the critical role of channel normalization.

Original:  Theoretical analysis
Revised:  Theoretical analysis

Original: From the previous section, we know that channel normalization avoids vanishing gradients even for linear, one-layer convolutional neural networks, and that the scale and shift parameters can be set to one and zero.
Revised: From the previous section, we know that channel normalization avoids vanishing gradients even for linear, one-layer convolutional neural networks, and that the scale and shift parameters can be set to one and zero.

Original: In this section, we provide theoretical justiﬁcation for the stabilizing eﬀect of channel normalization.
Revised: In this section, we provide theoretical justiﬁcation for the stabilizing eﬀect of channel normalization.

Original: Throughout this section, we consider a single channel linear convolutional neural network with d layers, with output given as f ( x , w ) =
Revised: Throughout this section, we consider a single channel linear convolutional neural network with d layers, with output given as f ( x , w ) =

Original: (cid:81) di =1 W i x , where W i ∈ R n × n are circulant matrices implementing the convolution operation, and w = ( w 1 , w 2 , . . . , w d ) is the set of weights, given by the ﬁrst columns of the respective circulant matrices W i , . .
Revised: (cid:81) di =1 W i x , where W i ∈ R n × n are circulant matrices implementing the convolution operation, and w = ( w 1 , w 2 , . . . , w d ) is the set of weights or convolutional ﬁlters, given by the ﬁrst columns of the respective circulant matrices W i , . .

Original: , W d (which deﬁne all other entries of the matrices).
Revised: , W d (which deﬁne all other entries of the matrices).

Original: We study gradient descent applied to the squared loss function L ( w , x , y ) 
Revised: We study gradient descent applied to the squared loss function L ( w , x , y ) =

Original: = 12 (cid:107) y − f ( x , w ) (cid:107) 22 .
Revised: 12 (cid:13)(cid:13) y − f ( x , w ) (cid:13)(cid:13) 2 2 .

Original: We start by showing that without channel normalizationoptimization is provably hard. Specifically, we show that without channel normalization, gradient descent needs at least exponentially many steps to converge under a standard initialization scheme.
Revised: We start by showing that without channel normalization gradient descent needs at least exponentially many steps to converge under a standard initialization scheme.

Original: Theorem 1.
Revised: Theorem 1.

Original: Suppose that the signal y doesn’t vanish, i.e., (cid:107) y (cid:107) / (cid:107) x (cid:107) ≥ dn d/ 2 τ , where τ is a constant.
Revised: Suppose that the signal y doesn’t vanish, i.e., (cid:107) y (cid:107) / (cid:107) x (cid:107) ≥ dn d/ 2 τ , where τ is a constant.

Original: Moreover, suppose that y is in the range of the generator f and that the initial weights are Gaussian random variables with zero mean and covariance matrix (1 /n 2 ) I ).
Revised: Moreover, suppose that y is in the range of the generator f and that the initial weights are Gaussian random variables with zero mean and covariance matrix (1 /n 2 ) I .

Original: Then gradient descent with constant stepsize η ≤ exp ( 12 cd ) runs at least for exp(Ω( d )) steps to be c (cid:48) close to optimal with probability larger than 1 − exp( − Ω( d )) .
Revised: Then gradient descent with constant stepsize η ≤ exp ( 1 2 cd ) runs at least for exp(Ω( d )) steps until it reaches a point that is c (cid:48) close to optimal with probability larger than 1 − exp( − Ω( d )) .

Original: Here, c and c (cid:48) are constants independent of d .
Revised: Here, c and c (cid:48) are constants independent of d .

Original: The proof, deferred to the appendix, relies on diagonalizing the circulant matrices
Revised: The proof, deferred to the appendix, relies on diagonalizing the circulant matrices

Original: W i using the Fourier transform.
Revised: W i using the Fourier transform.

Original: Then, the optimization problem reduces to n one-dimensional problems, and we can build on results by Shamir [Sha18] on the hardness of optimizing one-dimensional deep neural networks.
Revised: Then, the optimization problem reduces to n one-dimensional problems, and we can build on results by Shamir [Sha18] on the hardness of optimizing one-dimensional deep neural networks.

Original: Theorem 1 shows that the number of steps to come close to the optimum is at least exponential in the network depth d , even when the stepsize is large (exponential in d ).
Revised: Theorem 1 shows that the number of steps to come close to the optimum is at least exponential in the network depth d , even when the stepsize is large (exponential in d ).

Original: This can be interpreted as a case of gradient vanishing.
Revised: This can be interpreted as a case of gradient vanishing.

Original: Also not that if we initialize the weight away from 0, with another initialization scheme, then the norms of the gradients increase exponentially fast with the network depth, and the network becomes diﬃcult to optimize due to exploding gradients.
Revised: Also note that if we initialize the weight away from 0, with another initialization scheme, then the norms of the gradients increase exponentially fast with the network depth, and the network becomes diﬃcult to optimize due to exploding gradients.

Original: Next, we comment on the eﬀect the channel normalization on the gradients.
Revised: Next, we evaluate the eﬀect the channel normalization on the gradients.

Original: Since our experiments have shown that ﬁxing scale and shift parameters does not weaken the performance of channel normalization, we focus on the case with γ ij = 1 and β ij = 0. 
Revised: Since our experiments have shown that ﬁxing scale and shift parameters does not weaken the performance of channel normalization, we focus on the case with γ ij = 1 and β ij = 0. Suppose the the input x has zero mean.

Original: With the Fourier transformation, the circulant matrices W i can be diagonalized (see appendix). As shown in the appendix, provided the input and output vectors x and y have been centered, the loss function with channel normalization L N ( W , x , y ) can written as
Revised: Then the gradients pertaining to the loss function with normalized loss are (see Appendix C):

Original: [Equation]
Revised: [Equation]

Original: where w i is the  ﬁrst column  ofW i . To ensure the ﬁtted function can exhaust R n , we added an extra scaling parameter w d +1 ∈ R to the ﬁtted function.
Revised: where X is the circulant matrix with ﬁrst column x , and w d +1 is a scale parameter that we optimize over and necessary so that the range of the network can exhaust R n .

Original: Now, we can analyze the gradient with channel normalization (ﬁxed scale and shift). Based on the expression above, it can be shown that the gradients  of the loss function with normalization, given by ∇ w k LN ( W , x , y ) are bounded at initialization (shown in an extended version of this manuscript).
Revised: By this expression, the gradients are obtained by projecting y onto the orthogonal complement of the estimate at the k -th iteration, X k w k , followed by multiplication with X Tk / (cid:107) X k w k (cid:107) . In contrast, the norm of the un-normalized gradients is given by ∇ w k L ( W , x , y ) = X Tk ( y − X k w k ).

Original: Thus, channel normalization avoid gradient explosition.
Revised: In Figure 3 Panel (a) and (b) we plot the distribution with and without normalization at initialization for a network with n = 100 and d = 6 layers. Note that the loss typically diverges at the ﬁrst few iterations, which justiﬁes considering the gradients at initialization. The results show that the normalization leads to the gradients to be signiﬁcantly better behaved, i.e., the distribution does have a signiﬁcantly smaller tail. In Figure 3 Panel (c) and (d) we plot the distribution for a multi-channel CNN with ReLU activation functions, and likewise, the results shows that without channel normalization, the tail is signiﬁcantly larger.

Original: Contrary, without normalization, the order of the gradient norm increases in the network depth.
Revised: Thus, without normalization for a given stepsize the network is much more susceptible to vanishing or exploding gradients.

Original: Speciﬁcally, for deep linear CNNs, the norm of the gradients increase/decrease exponentially fast as the order of network, which can easily lead to gradient explosion/vanishing.
Revised: [Equation] [Figure] [Equation] [Figure]

Original: Convergence for linear one layer convolutional networks
Revised: [Equation] [Figure] [Equation]

Original: [Figure]
Revised: [Figure]

Revised: Figure 3:

Original: Proofs
Revised: Panel a) and b) show the distribution of the gradients at initialization with normalization and without normalization for a single channel linear CNN with n = 100 and d = 6; c) and d) show that for a multiple channel CNN with n = 64, d = 6, ﬁlter size 9 and number of channels 4. Code Code to reproduce the experiments is available at github.com/reinhardh/normalization dnns. Acknowledgements

Revised: RH and ZD are partially supported by NSF award IIS-1816986.

Revised: B Proof of Theorem

Original: Our proofs rely on diagonalizing the circulant matrices using the Fourier transform.
Revised: Our proof relies on diagonalizing the circulant matrices implementing the convolutions with the Fourier transform.

Original: 6. Linear single channel CNNs in the Fourier domain
Revised: B. Linear single channel CNNs in the Fourier domain

Original: Since the matrices W i are circulant, they can be diagonalized with the Fourier transformation, which makes the loss function separable.
Revised: Since the matrices W i are circulant, they can be diagonalized with the Fourier transformation.

Original: Hence, the loss function  linear CNN can be transformed to the sum ofthe loss functions of  single channel deep linear neural networks.
Revised: As a consequence, the loss function of a linear CNN becomes a sum of loss functions of one dimensional single channel deep linear neural networks.

Original: With W i = F diag( √ n F H w i ) F H , where F is the unitary n × n discrete Fourier transform matrix and w i is the ﬁrst column of W i , the network output  can be expressed as f ( x , W ) =
Revised: With W i = F diag( √ n F H w i ) F H , where F is the unitary n × n discrete Fourier transform matrix and w i is the ﬁrst column of W i , the network’s output without normalization can be expressed as f ( x , W ) =

Original: d (cid:89) i =
Revised: d (cid:89) i =

Original: W i x
Revised: W i x

Original: = d (cid:89) i =
Revised: = d (cid:89) i =

Original: F diag( √ n F H w i ) F H x =√ n d F (cid:32) d (cid:89) i = diag( F H w i ) (cid:33)
Revised: F diag( √ n F H w i ) F H x = n d/ 2 F   d (cid:89) i = diag( F H w i ) 

Original:  F H x .
Revised:  F H x .

Original: With this expression, the loss function becomes
Revised: With this expression, the loss function becomes

Original: [Equation]
Revised: [Equation]

Original: [Table]

Revised:   d

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: i = diag

Original: [Equation] 
Revised: [Equation] 

Original: [Table]
Revised:  F H x (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) F

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: n

Original: [Equation] 
Revised: [Equation] /

Original: [Table] [Equation]
Revised:  d (cid:89) i = diag( F H w i ) 

Original: [Table]
Revised:  F H x (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) = n (cid:88) j = (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) f Hj y

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: n d/  d (cid:89) i = f Hj

Original: [Equation]
Revised: [Equation]

Original:  where f j is the j -th column of the Fourier matrix F .
Revised: i   f Hj x (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) , where f j is the j -th column of the Fourier matrix F .

Original: 6. Proof of Theorem
Revised: B. Proof of Theorem

Original: In this section, we show that gradient descent takes exponentially many steps to converge under(standard) Xavier initialization.
Revised: In this section, we show that gradient descent takes exponentially many steps to converge under Xavier initialization, a standard initialization scheme.

Revised: The proof follows a similar line of arguments as a very related result by Shamir [Sha18] on the hardness of optimizing one-dimensional deep neural networks.

Original: Assume the kernel size of  i -th convolution layer is p( p ≥ 1).
Revised: Assume the kernel size of the i -th convolution layer is p ≥ 1.

Original: Thus, w i ∈ S p , where S p = { z = ( z 1 , z 2 , . . . , z n ) | z 1 , . . . , z p ∈ R , z p +1 , . . .
Revised: Thus, w i ∈ S p , where S p = { z = ( z 1 , z 2 , . . . , z n ) | z 1 , . . . , z p ∈ R , z p +1 , . . .

Original: , z n = 0 } .
Revised: , z n = 0 } .

Original: Assumption 1.
Revised: Assumption 1.

Original: Assume w 1 , w 2 , . .
Revised: Assume w 1 , w 2 , . .

Original: . , w d are drawn independently and the ﬁrst p entries of w i are drawn i.i.d from a distribution with mean µ = 0 and variance σ = 1 n 2 , which also satisﬁes
Revised: . , w d are drawn independently and the ﬁrst p entries of w i are drawn i.i.d from a distribution with mean µ = 0 and variance σ = 1 n 2 , which also satisﬁes

Original: P [ (cid:107) w i (cid:107) ≤ t ] ≤ c 1 t 2.
Revised: [Equation] and

Original: E [ (cid:107) w i (cid:107) ] ≤ 1 − c
Revised: [Equation]

Original: where the constants c 1 , c 2 > 0 are independent of d .
Revised: where the constants c 1 , c 2 > 0 are independent of d .

Original: The assumptions hold for some widely used  distributions, like the distribution N (0 , 1 n 2 ) in the statement of the theorem.
Revised: The assumption holds for some widely used initialization distributions, like the distribution N (0 , 1 n 2 ) in the statement of the theorem.

Original: Next, we show that with the initialization satisfying Assumption 1, gradient descent takes exponential time to converge with high probability.
Revised: Next, we show that with the initialization satisfying Assumption 1, gradient descent takes at least exponentially many gradient descent iterations to reach a close-to-optimal point with high probability.

Original: The key idea of the proof of Theorem 1 is to show that if we start from such a random initialization w = ( w 1 , w 2 , . . .
Revised: The key idea of the proof of Theorem 1 is to show that if we start from such a random initialization w = ( w 1 , w 2 , . . .

Original: , w d ), then gradient descent must take exponential many steps to escape a ball of radius r around w , deﬁned as
Revised: , w d ), then gradient descent must take exponentially many steps to escape a ball of radius r around w , deﬁned as

Original: [Equation]
Revised: [Equation]

Original: We then show that the  function value evaluated at any point inside the ball is sub-optimal, i.e., for v ∈ B ( w , r ), L ( v ) ≥ c (cid:48) , where c (cid:48) is a constant.
Revised: We then show that the loss function value evaluated at any point inside the ball is sub-optimal, i.e., for v ∈ B ( w , r ), L ( v ) ≥ c (cid:48) , where c (cid:48) is a constant.

Original: This concludes the proof.
Revised: This will establish the proof.

Original: To show that there exist a radius r > 0 such that gradient descent takes at least exponential steps to escape the ball B ( w , r ), we ﬁrst note that the number of iterations required to escape a ball of radius r is at least r/ ( η sup v ∈B ( w ,r ) (cid:107)∇ L ( v ) (cid:107) ), since at each step, gradient descent can at most move by η sup v ∈B ( w ,r ) (cid:107)∇ L ( v ) (cid:107) away from the initialization.
Revised: To show that there exist a radius r > 0 such that gradient descent takes at least exponentially many steps to escape the ball B ( w , r ), we ﬁrst note that the number of iterations required to escape a ball of radius r is at least r/ ( η sup v ∈B ( w ,r ) (cid:13)(cid:13) ∇ L ( v ) (cid:13)(cid:13) ), since at each step, gradient descent can at most move by η sup v ∈B ( w ,r ) (cid:13)(cid:13) ∇ L ( v ) (cid:13)(cid:13) away from the initialization.

Original: The following lemma provides an upper bound on (cid:107)∇ L ( v ) (cid:107) , which enables us to show that the number of iterations must be large.
Revised: The following lemma provides an upper bound on (cid:13)(cid:13) ∇ L ( v ) (cid:13)(cid:13) , which enables us to show that the number of iterations must be large.

Original: Lemma 1.
Revised: Lemma 1.

Original: For all v ∈ B ( w , r ) , if w satisﬁes, for some α, δ > 0 ,  max k (cid:16)(cid:81) i (cid:54) 
Revised: Suppose that the initial point w satisﬁes, for some α, δ > 0 , i) max k (cid:16)(cid:81) i (cid:54) =

Original: =
Revised: k (cid:107) w i (cid:107) (cid:17) ≤ α and ii) min i (cid:107) w i (cid:107) ≥ δ .

Original: k (cid:107) w  i (cid:107)(cid:17) ≤ α and min i (cid:107) w i (cid:107) ≥ δ , it holds that (cid:107)∇ L ( v ) (cid:107) ≤ (cid:107) D ( v )  (cid:107)(cid:16) α √ d(cid:17) exp (cid:32) √ d δ r (cid:33) , where L ( v ) =12 (cid:107) D ( v ) (cid:107) 2 and
Revised: Then, there exists a radius r such that for all v ∈ B ( w , r ) , it holds that (cid:81) di =1 (cid:107) v i (cid:107) ≤ α exp (cid:16) √ drδ (cid:17) max k (cid:107) v k (cid:107) and (cid:13)(cid:13) ∇ L ( v ) (cid:13)(cid:13) ≤ (cid:13)(cid:13) D ( v ) (cid:13)(cid:13) (cid:107) x (cid:107) α √ d exp (cid:32) √ d δ r (cid:33) , where L ( v ) =

Original:  D ( v ) =
Revised: 12 (cid:13)(cid:13) D ( v ) (cid:13)(cid:13) 2 and D ( v ) =

Original: F H y −√ n d (cid:81) di =1 diag( F H v i )
Revised: F H y − n d/ 2 (cid:81) di =1 diag( F H v i )

Original: F H x .
Revised: F H x .

Original: Lemma 1 guarantees that in a ball B ( w , r ) around the initialization, the corresponding gradient is strictly upper bounded.
Revised: Lemma 1 guarantees that in a ball B ( w , r ) around the initialization, the gradients of the loss function are strictly upper bounded.

Original: So, for a stepsize  not too large, the update of each step  is also upper bounded.
Revised: So, provided the stepsize is not too large, the progress made in each step of gradient descent is also upper bounded.

Original: Then, we show with high probability, that we can ﬁnd a radius r that is much larger than the updates in each step.
Revised: Then, we show with high probability, that there is a radius r that is much larger than the updates in each step.

Original: Evoking Lemma 1,  the number of steps required to escape the ball B ( w , r ) is at leastr
Revised: Evoking Lemma 1, given a constant stepsize η , the number of steps required to escape the ball B ( w , r ) is at least

Original: O ( ηα √ d exp( √ dδ r )) .
Revised: [Equation] is upper bounded by a numerical constant.

Original: Next, we show that we can upper bound α and lower bound δ with high probability  given Assumption 1.
Revised: Next, we show that conditions i and ii from Lemma 1 hold with high probability for α = exp( − 2 cd ) and δ = exp( − cd ) given Assumption 1.

Original: And under this condition, there exists r such that gradient descent takes exponential steps to escape the ball B ( w , r ).
Revised: We then show that there is a radius r in which (cid:13)(cid:13) D ( v ) (cid:13)(cid:13) is upper bounded and gradient descent takes at least exponentially many steps to escape the ball B ( w , r ).

Original: With Assumption 1 and Markov’s inequality, we have, for t > 0, that
Revised: We start by showing that conditions i and ii from Lemma 1 hold with high probability for α = exp( − 2 cd ) and δ = exp( − cd ). Lemma 2. Suppose w is initialized satisfying Assumption 1. With probability at least 1 − Ω( de − cd ) , the conditions

Original: Pr [Equation]
Revised: [Equation]

Original: t
Revised: and

Original: [Equation]
Revised: [Equation]

Revised: hold simultaneously, where c is a constant independent of d .

Original: [Equation]
Revised: Proof.

Original: − c
Revised: With Markov’s inequality and Assumption 1, we have, for t > 0, that

Original: [Equation]
Revised: [Equation]

Original: t . Let  exp( − 4 c ) = 1 − c 2 , t = exp( − 2 cd ), where c > 0.
Revised: Let c be a constant so that exp( − 4 c ) = 1 − c 2 and set t = exp( − 2 cd ).

Original: Then we obtain
Revised: Then we obtain

Original: [Equation]
Revised: [Equation]

Original: Again by Assumption 1, it holds for all i 
Revised: Thus, by the union bound, max k (cid:81) i (cid:54) =

Original: that Pr( (cid:107) w i (cid:107) ≤ exp( −  cd )) ≤ O (exp( −  cd )).
Revised: k (cid:107) w i (cid:107) ≤ e − 2 cd with probability at least 1 − Ω( d exp( − 2 cd )).

Revised: We next consider min i (cid:107) w i (cid:107) .

Revised: Again by Assumption 1, it holds for all i that P (cid:2) (cid:107) w i (cid:107) ≤ exp( − cd ) (cid:3) ≤ Ω(exp( − cd )).

Original: Then, by the union bound,
Revised: Then, by the union bound,

Original: [Equation]
Revised: [Equation]

Original: Finally, by Assumption 1, Markov’s inequality, and a union bound
Revised: Finally, by the union bound, Markov’s inequality, and Assumption 1, we have that

Original: [Equation]
Revised: [Equation]

Original: i
Revised: Thus, by a union bound, with probability at least

Original: [Equation]
Revised: [Equation]

Original: Thus, by a union bound, with probability at least 1 − O (exp( − 2 cd )) − O ( d exp( − cd )) − O (  d exp( − cd )) = 1 − O ( d  exp( − cd )), the conditions i) (cid:13)(cid:13)(cid:13)(cid:81)
Revised: the conditions i-iii are satisﬁed simultaneously, which concludes the proof. Next, we show that under conditions i-iii in Lemma 2, provided that d is large enough, (cid:13)(cid:13) D ( v ) (cid:13)(cid:13) is upper bounded by a constant independent of d for radius r = Ω( δ √ d ) = Ω( exp( − cd ) √ d ).

Original: i (cid:54) = k ( f Hj w i ) (cid:13)(cid:13)(cid:13) ≥ exp( − 2 cd ), ii) min i (cid:107) w i (cid:107) ≥ exp( − cd ), and ii) max i (cid:107) w i (cid:107) ≤ exp( cd ) are satisﬁed simultaneously. With i-iii) satisﬁed, the number of steps in B ( w , r ) is at least
Revised: Evoking Lemma 1, we have

Original: [Equation]
Revised: [Equation]

Original: If we let r = O ( δ √ d ) = O ( exp(− cd ) √ d ), then r
Revised: where the second inequality follows from our choice of α and exp( √ drδ )

Original: [Equation]
Revised: = Ω(1), by our choice of δ , and the third inequality follows from max k (cid:107) v k (cid:107) ≤ max k (cid:107) w k (cid:107) + r = Ω(exp( cd )).

Original: =
Revised: Then, we can upper bound (cid:13)(cid:13) D ( v ) (cid:13)(cid:13) when d is large,

Original: [Equation]
Revised: [Equation]

Original: exp
Revised: Therefore, (cid:13)(cid:13) D ( v ) (cid:13)(cid:13) is upper bounded by a constant (given y is constant, and x satisﬁes the assumption in theorem 1 independent of d . Finally, we can prove that gradient descent takes at least exponentially many steps to escape B ( w , r ) with r = Ω( exp( − cd ) √ d ). By lemma 2, with conditions i-iii satisﬁed, the number of steps in B ( w , r ) is at least r (cid:107) D ( v ) (cid:107) (cid:107) x (cid:107) Ω( ηα √ d exp( √ dδ r )) . Since (cid:13)(cid:13) D ( v ) (cid:13)(cid:13) can be upper bounded by a constant, and using that (cid:107) x (cid:107) is upper bounded by a constant (from (cid:107) x (cid:107) < (cid:107) y (cid:107) , with (cid:107) y (cid:107) upper bounded by a constant) and η ≤ exp( d 2 ) (by the assumptions in theorem 1),

Original: [Equation]
Revised: [Equation]

Original: 12 cd )), which increases exponentially in d .
Revised: , which increases exponentially in d .

Original: It remains to prove that for any v ∈ B ( w , r ) (with r as chosen above), the loss function L ( v ) is lower bounded away from the global minimum.Towards this goal, we note that, for v ∈ B ( w , r ), [Equation] Hence,
Revised: It remains to prove that for any v ∈ B ( w , r ) (with r as chosen above), the loss function L ( v ) is lower bounded away from the global minimum.

Original: [Table] Therefore, when d is large, (cid:13)(cid:13)(cid:13)(cid:81) di =1  f
Revised: We have proved that when d is large, (cid:81) di =1 (cid:12)(cid:12)(cid:12) f

Original: Hj v i (cid:13)(cid:13)(cid:13) ≤ 1.
Revised: Hj v i (cid:12)(cid:12)(cid:12) ≤ (cid:81) di

Original: From the assumption (cid:107) y (cid:107) − (cid:107) x (cid:107) d √ n d ≥ τ , there exists t ∈ [ n ] such that (cid:13)(cid:13) f Ht y (cid:13)(cid:13) 2 − √ n d (cid:13)(cid:13) f
Revised: =1 (cid:107) v i (cid:107) ≤ Ω(1).

Original: Ht x (cid:13)(cid:13) 2 ≥ τ . So, we have (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) f [Table] √ n 
Revised: From the assumption (cid:107) y (cid:107) − (cid:107) x (cid:107) dn d/ 2 ≥ τ , there exists t ∈ [ n ] such that (cid:12)(cid:12)

Original: [Table]

Original: f
Revised: f

Original: [Equation] v i [Equation] x (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≥ (cid:13)(cid:13) f
Revised: Ht y (cid:12)(cid:12) − n d/

Original: [Equation]
Revised: [Equation]

Original: y (cid:13)(cid:13) 2 − √ n d d (cid:89) i = (cid:13)(cid:13) f Ht 
Revised: f Ht x (cid:12)(cid:12) ≥ τ .

Original: [Equation] (cid:13)(cid:13) 2 (cid:13)(cid:13) f [Equation] x (cid:13)(cid:13)≥ (cid:13) (cid:13)
Revised: So, we have

Original: [Equation]
Revised: [Equation]

Original: (cid:13) (cid:13) 2 − √ n d (cid:13) (cid:13) f

Original: Ht x (cid:13) (cid:13) 2 > τ

Original: Thus, for v ∈ B ( w , r ), the loss function obeys L ( v ) =
Revised: Thus, for v ∈ B ( w , r ), the loss function obeys L ( v ) =

Original: (cid:80) nj =1 (cid:13)(cid:13)(cid:13) f
Revised: (cid:80) nj =1 (cid:12)(cid:12)(cid:12)(cid:12) f

Original: Hj y −√ n d (cid:81) di =1 (cid:16) f
Revised: Hj y − n d/ 2 (cid:81) di =1 (cid:16) f

Original: Hj v i (cid:17) f
Revised: Hj v i (cid:17) f

Original: Hj x (cid:13)(cid:13)(cid:13) 2 > τ and is thus lower bounded away from zero-training error. 
Revised: Hj x (cid:12)(cid:12)(cid:12)(cid:12) > τ and is thus lower bounded away from zero-training error. B

Original: 6.2.  Proof of Lemma
Revised:  Proof of Lemma

Original: Let v ∈ B ( w , r ) as deﬁned previously.
Revised: Let v ∈ B ( w , r ) as deﬁned previously.

Original: We have (cid:107)∇ L ( v ) (cid:107) 2 ≤ d (cid:88) k
Revised: We have (cid:13)(cid:13) ∇ L ( v ) (cid:13)(cid:13) 2 ≤ d (cid:88) k

Original: = (cid:107)∇ v k L ( v ) (cid:107)= d (cid:88) k{+(cid:13)(cid:13) 
+}
Revised: = (cid:13)(cid:13) ∇ v k L ( v ) {+(cid:13)(cid:13) 
+}

Original: = (cid:107) D ( v ) (cid:107)  (cid:89) i (cid:54) =
Revised: = d (cid:88) k = (cid:13)(cid:13) D ( v ) (cid:13)(cid:13) (cid:89) i (cid:54) =

Original: k (cid:107) v i (cid:107)  
Revised: k (cid:107) v i (cid:107)   (cid:107) x

Original:  ≤ d (cid:88) k
Revised: (cid:107)≤ d (cid:88) k

Original: = (cid:107) D ( v ) (cid:107)   max k 
Revised: = (cid:13)(cid:13) D ( v ) (cid:13)(cid:13)  max k (cid:89) i (cid:54) = k (cid:107) v i (cid:107)   (cid:107) x (cid:107) 2 . Here, we used that (cid:13)(cid:13) ∇ v k L ( v ) (cid:13)(cid:13) =

Revised: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) D ( v ) (cid:13)(cid:13) n d/ 2 diag

Original:  (cid:89) i (cid:54) =
Revised: ( F H x i ) (cid:89) i (cid:54) =

Original: k (cid:107) v i (cid:107) 
Revised: k diag( F H v i ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original:  .
Revised: [Table]

Original: The lemma now follows from max k (cid:16)(cid:81)i (cid:54)
Revised: The lemma now follows from max k (cid:16)(cid:81)

Original: =
Revised: i

Original: k (cid:107) v i (cid:107) (cid:17) ≤ (cid:16) α √ d (cid:17) exp (cid:16) √ dδ r (cid:17) .
Revised: [Equation] dδ r (cid:17) .

Original: Deﬁne r i = v
Revised: Deﬁne r i = v

Original: i − w i , for notational convenience, and note that (cid:80) ni =1 (cid:107) r i (cid:107) 2 ≤ r 2 since v ∈ B ( w , r ).
Revised: i − w i , for notational convenience, and note that (cid:80) di =1 (cid:107) r i (cid:107) 2 ≤ r 2 since v ∈ B ( w , r ).

Original: Then, we have (cid:89)i (cid:54)
Revised: Then, we have (cid:89)

Original: = k (cid:107) v i (cid:107) = (cid:89) i (cid:54) =
Revised: [Equation]

Original: k (cid:107)w i + r i (cid:107) ≤  (cid:89) i (cid:54) =
Revised: (cid:107) r i (cid:107) δ   , (5) where the second inequality follows from the assumption min

Original: k (cid:107) w i (cid:107) 
Revised: i (cid:107) w i (cid:107) ≥ δ .

Original:   (cid:89) i (cid:54) = k (cid:18) 1 + (cid:107) r i (cid:107) (cid:107) w i (cid:107) (cid:19)  ≤  (cid:89) i (cid:54) =
Revised: We also have

Original: k (cid:107) w i (cid:107)    (cid:89) i (cid:54) = k (cid:18) 1 + (cid:107) r
Revised: [Equation]

Original: i (cid:107) δ (cid:19)  =  (cid:89) i (cid:54) = k (cid:107) w i (cid:107)   exp  (cid:88) i (cid:54) =
Revised: Inserting equation (6) into equation (5), we get (cid:89) i (cid:54) =

Original: k[-log (cid:18) 
 + (cid:107) r i (cid:107) δ (cid:19)  ≤  (cid:89) i (cid:54) = k (cid:107) w i (cid:107) -]
Revised: k[-log (cid:18) 
 + (cid:107) r i (cid:107) δ (cid:19)  ≤  (cid:89) i (cid:54) = k (cid:107) w i (cid:107) -]

Original:  exp  (cid:88)

Original: [Equation]
Revised: [Equation]

Original: We also have
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: Inserting Equation (5) into Equation (4), we get (cid:89)
Revised: i

Original: [Equation]
Revised: [Equation]

Revised: δ

Revised: [Equation]

Revised: δ

Revised: [Equation]

Revised: , where the last inequality follows from the assumption max k (cid:16)(cid:81) i (cid:54) =

Revised: k (cid:107) w i (cid:107) (cid:17) ≤ α .

Revised: Therefore, (cid:81) di

Revised: =1 (cid:107) v i (cid:107) ≤ α exp (cid:16) √ drδ (cid:17) · max k (cid:107) v k (cid:107) .

Original: Application of this inequality in  ( ?? ) 
Revised: Application of this inequality in (4) yields (cid:13)(cid:13) ∇ L ( v ) (cid:13)(cid:13) ≤ √ d · max k (cid:13)(cid:13) ∇ k L ( v ) (cid:13)(cid:13) ≤ (cid:13)(cid:13) D ( v ) (cid:13)(cid:13) (cid:107) x (cid:107) (cid:16) α √ d (cid:17) exp (cid:16) √ dδ r (cid:17) , which concludes the proof. C Channel normalization on a single channel linear CNN

Original: {+We consider channel normalization with scale parameter ﬁxed to γ ij = 
+}
Revised: {+We consider channel normalization with scale parameter ﬁxed to γ ij = 
+}

Original: yields (cid:107)∇
Revised: and the shift parameter ﬁxed to β ij = 0. Hence, merely varying the scale of the parameters does not change the output of the network with channel normalization f N ( W , x ). To simplify the derivation we assume the input vector x is centered, i.e., its entries sum to zero. Let z i be the input of the i -th layer.

Revised: Then, the output of the convolutional layer can be written as W i

Revised: z i , where W i is the circulant matrix implementing the convolution operation.

Revised: Channel normalization centers the mean and adjusts the empirical variance to one.

Revised: Since W i is a convolution operation, given z i is centered,

Revised: W i

Revised: z i is centered as well.

Revised: Thus, the eﬀect of channel normalization in this setup is to normalize the scale of the vector.

Revised: Itfollows that the output of the network with channel normalization can be written as

Original: [Equation]
Revised: [Equation]

Revised: where w d +1 is a scale parameter that we introduced so that f N ( W , x ) can exhaust R n .

Revised: To see that the output of the network can be written as in equation (7), note that the input of the ﬁrst layer is z 1 = x , with x centered.

Revised: The normalization of the ﬁrst layer yields the input of the second layer by division by (cid:107) W 1 z

Revised: [Equation]

Revised: / √ n =

Revised: [Equation]

Revised: W

Revised: [Equation]

Revised: / √ n , which gives

Revised: [Equation]

Revised: The normalization operation at the second layer divides by

Original:  which concludes the proof.
Revised: [Equation] which yields [Equation] Continuing this logic yields equation (7), where we absorbed √ n in the parameter w d +1 .

Revised: C. Gradient analysis

Revised: Note that with X ∈ R n × n the circulant matrix with ﬁrst column equal to x , we have that W k x =

Revised: Xw k and the channel normalized output of the network becomes

Revised: [Equation]

Revised: where we deﬁned X k = (cid:81) i (cid:54)

Revised: =

Revised: k W i X for notational convenience.

Revised: We next compute the gradient ∇ w k L ( W , x , y ) ,

Revised: L

Revised: [Equation]

Revised: Towards this goal, ﬁrst note that

Revised: [Equation]

Revised: Thus, by the chain rule

Revised: [Equation]

Revised: From direct computation,

Revised: [Equation]

Revised: thus the gradient is orthogonal to w k .

Revised: Moreover, if w k is multiplied with a positive scalar γ , then the gradient scales with 1 /γ .