Original: On the Convex Behavior of Deep Neural Networks in Relation to the Layers’ Width
Revised: On the Convex Behavior of Deep Neural Networks in Relation to the Layers’ Width

Original: Abstract
Revised: Abstract

Original: The Hessian of neural networks can be decomposed into a sum of two matrices: (i) the positive semideﬁnite generalized Gauss-Newton matrix G , and (ii) the matrix H containing negative eigenvalues.
Revised: The Hessian of neural networks can be decomposed into a sum of two matrices: (i) the positive semideﬁnite generalized Gauss-Newton matrix G , and (ii) the matrix H containing negative eigenvalues.

Original: We observe that for wider networks, minimizing the loss with the gradient descent optimization maneuvers through surfaces of positive curvatures at the start and end of training, and close to zero curvatures in between.
Revised: We observe that for wider networks, minimizing the loss with the gradient descent optimization maneuvers through surfaces of positive curvatures at the start and end of training, and close to zero curvatures in between.

Original: In other words, it seems that during crucial parts of the training process, the Hessian in wide networks is dominated by the component G .
Revised: In other words, it seems that during crucial parts of the training process, the Hessian in wide networks is dominated by the component G .

Original: To explain this phenomenon, we show that when initialized using common methodologies, the gradients of over-parameterized networks are approximately orthogonal to H , such that the curvature of the loss surface is strictly positive in the direction of the gradient.
Revised: To explain this phenomenon, we show that when initialized using common methodologies, the gradients of over-parameterized networks are approximately orthogonal to H , such that the curvature of the loss surface is strictly positive in the direction of the gradient.

Original: 1. Introduction
Revised: 1. Introduction

Original: It has frequently been observed (Choromanska et al., 2015) that deep neural networks are relatively easy to optimize using straightforward SGD algorithms, despite having complex non-convex loss surfaces.
Revised: It has frequently been observed (Choromanska et al., 2015) that deep neural networks are relatively easy to optimize using straightforward SGD algorithms, despite having complex non-convex loss surfaces.

Original: A great body of work in recent years has focused on the properties of SGD and its ability to maneuver through saddle points and complex surface topologies, in order to explain the relative ease of the optimization process (Daneshmand et al., 2018).
Revised: A great body of work in recent years has focused on the properties of SGD and its ability to maneuver through saddle points and complex surface topologies, in order to explain the relative ease of the optimization process (Daneshmand et al., 2018).

Original: A complementary approach to analyzing the optimization process, is to consider the trajectories of the gradients during optimization, with the guiding intuition that while the loss surface deﬁned by modern neural networks might be incredibly complex and hard to analyze, the trajectories of gradient descent on these loss surfaces might prove rather simple.
Revised: A complementary approach to analyzing the optimization process, is to consider the trajectories of the gradients during optimization, with the guiding intuition that while the loss surface deﬁned by modern neural networks might be incredibly complex and hard to analyze, the trajectories of gradient descent on these loss surfaces might prove rather simple.

Original: An additional direction of research focuses on the spectrum of the Hessian of the loss during training, as the knowledge of the second order information about the loss surface can tell us quite a bit about the behavior of the optimizer, and how we could modify our algorithms to make them converge faster and to better solutions (Gupta et al., 2018; Martens & Grosse, 2015).
Revised: An additional direction of research focuses on the spectrum of the Hessian of the loss during training, as the knowledge of the second order information about the loss surface can tell us quite a bit about the behavior of the optimizer, and how we could modify our algorithms to make them converge faster and to better solutions (Gupta et al., 2018; Martens & Grosse, 2015).

Original: Recent works have empirically shown that the distribution of the eigenvalues of the Hessian is composed of a bulk, which is concentrated around zero, and the edges which are scattered away from zero (Sagun et al., 2016).
Revised: Recent works have empirically shown that the distribution of the eigenvalues of the Hessian is composed of a bulk, which is concentrated around zero, and the edges which are scattered away from zero (Sagun et al., 2016).

Original: Modern neural networks, however, contain millions of parameters, resulting in a hefty Hessian with millions of eigenvalues, the overwhelming majority of which are irrelevant.
Revised: Modern neural networks, however, contain millions of parameters, resulting in a hefty Hessian with millions of eigenvalues, the overwhelming majority of which are irrelevant.

Original: In our work, we seek to study the curvature of the Hessian in directions corresponding to the trajectory of the gradients during training, by focusing on the Hessian gradient product.
Revised: In our work, we seek to study the curvature of the Hessian in directions corresponding to the trajectory of the gradients during training, by focusing on the Hessian gradient product.

Original: Speciﬁcally, we aim to empirically and theoretically analyze the path of gradient descent algorithms by analyzing the
Revised: Speciﬁcally, we aim to empirically and theoretically analyze the path of gradient descent algorithms by analyzing the

Original: Hessian at initialization.
Revised: Hessian at initialization.

Original: We identify and empirically demonstrate novel phenomena relating to the trajectory of gradient descent neural networks under certain conditions.
Revised: We identify and empirically demonstrate novel phenomena relating to the trajectory of gradient descent neural networks under certain conditions.

Original: We observe that the gradients follow a convex path during training when performing SGD, such that the curvature of the loss is seldom negative in the direction of the gradients.
Revised: We observe that the gradients follow a convex path during training when performing SGD, such that the curvature of the loss is seldom negative in the direction of the gradients.

Original: This effect is more prominent in wider networks, i.e., in CNNs with a larger number of kernels.
Revised: This effect is more prominent in wider networks, i.e., in CNNs with a larger number of kernels.

Original: Furthermore, we theoretically prove that when using common initialization methods, the gradients of random feedforward networks are indeed pointing in a direction of positive curvature when the network is wide enough, and the input is large enough.
Revised: Furthermore, we theoretically prove that when using common initialization methods, the gradients of random feedforward networks are indeed pointing in a direction of positive curvature when the network is wide enough, and the input is large enough.

Original: 1.1. Terminology
Revised: 1.1. Terminology

Original: Given the output of some neural network y ( x i , w t ) for input x
Revised: Given the output of some neural network y ( x i , w t ) for input x

Original: i , with W t denoting the model weights at iteration t , and a convex, twice differentiable loss function
Revised: i , with W t denoting the model weights at iteration t , and a convex, twice differentiable loss function

Original: L ( y ( x i , W t )) , we denote the gradient on a batch of N samples by g t =
Revised: L ( y ( x i , W t )) , we denote the gradient on a batch of N samples by g t =

Original: [Equation]
Revised: [Equation]

Original: (cid:80)
Revised: (cid:80)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , and the Hessian by
Revised: , and the Hessian by

Original: [Equation]
Revised: [Equation]

Original: A positive curvature during training at step t is characterized by the following condition:
Revised: A positive curvature during training at step t is characterized by the following condition:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: 2. Observations
Revised: 2. Observations

Original: We approximate the Hessian gradient product g (cid:62)
Revised: We approximate the Hessian gradient product g (cid:62)

Original: t H t g t during the training of deep neural networks, using the following formula:
Revised: t H t g t during the training of deep neural networks, using the following formula:

Original: [Equation]
Revised: [Equation]

Original: Which directly leads to:
Revised: Which directly leads to:

Original: [Equation]
Revised: [Equation]

Original: where δ is the learning rate, and the loss L is computed on a minibatch { x i } Ni =1 .
Revised: where δ is the learning rate, and the loss L is computed on a minibatch { x i } Ni =1 .

Original: We track the behavior of g
Revised: We track the behavior of g

Original: [Equation]
Revised: [Equation]

Original: t H
Revised: t H

Original: [Equation]
Revised: [Equation]

Original: g t across the epochs for networks of varying widths.
Revised: g t across the epochs for networks of varying widths.

Original: In our experiments, we employ the ResNet-50 network (He et al., 2016), which has layers with 16, 32, or 64 kernels.
Revised: In our experiments, we employ the ResNet-50 network (He et al., 2016), which has layers with 16, 32, or 64 kernels.

Original: We make these layers wider by multiplying the number of kernels by a factor K = 1 , . . .
Revised: We make these layers wider by multiplying the number of kernels by a factor K = 1 , . . .

Original: We halve the learning rate at epochs 40, 80, and 120.
Revised: We halve the learning rate at epochs 40, 80, and 120.

Original: A batch size of 100 is used with a vanilla SGD optimizer.
Revised: A batch size of 100 is used with a vanilla SGD optimizer.

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: 1 for two popular datasets:
Revised: 1 for two popular datasets:

Original: CIFAR-100 (Krizhevsky, 2009) and tiny imagenet (Le &
Revised: CIFAR-100 (Krizhevsky, 2009) and tiny imagenet (Le &

Original: Yang, 2015).
Revised: Yang, 2015).

Original: In both datasets, one can observe that (i) throughout the majority of the training run, g (cid:62) t H t g t remains positive; (ii) at the start of training, and when the network converges, it obtains larger values; and (iii) typically, the wider the network, the bigger this quantity is.
Revised: In both datasets, one can observe that (i) throughout the majority of the training run, g (cid:62) t H t g t remains positive; (ii) at the start of training, and when the network converges, it obtains larger values; and (iii) typically, the wider the network, the bigger this quantity is.

Original: Fig.
Revised: Fig.

Original: 1 also presents the test error for the same benchmarks  and the same networks.
Revised: 1 also presents the test error for the same benchmarks [Figure] and the same networks.

Original: We observe that wider networks lead to smaller test errors, which means that their tendency toward a convex-mimicking behavior during training does not come at the expanse of the generalization ability.
Revised: We observe that wider networks lead to smaller test errors, which means that their tendency toward a convex-mimicking behavior during training does not come at the expanse of the generalization ability.

Original: We further conduct small scale experiments, in which one can efﬁciently decompose the empirical Hessian into its components H and G , and calculate the quantities
Revised: We further conduct small scale experiments, in which one can efﬁciently decompose the empirical Hessian into its components H and G , and calculate the quantities

Original: H ˆ g
Revised: H ˆ g

Original: = ˆ g (cid:62)
Revised: = ˆ g (cid:62)

Original: H ˆ g , G ˆ g
Revised: H ˆ g , G ˆ g

Original: = ˆ g (cid:62) G ˆ g and H ˆ g
Revised: = ˆ g (cid:62) G ˆ g and H ˆ g

Original: = ˆ g (cid:62)
Revised: = ˆ g (cid:62)

Original: H ˆ g during training.
Revised: H ˆ g during training.

Original: Speciﬁcally, we train a single output, 8 layer fully connected network with layer width of 50, 200 and 400 on a set of 1000 predetermined random inputs sampled from a Gaussian distribution with dimensionality matching the width of the network, which are then normalized to have unit norm.
Revised: Speciﬁcally, we train a single output, 8 layer fully connected network with layer width of 50, 200 and 400 on a set of 1000 predetermined random inputs sampled from a Gaussian distribution with dimensionality matching the width of the network, which are then normalized to have unit norm.

Original: The network are trained to ﬁt with a standard L2 loss random labels sampled from a binary distribution.
Revised: The network are trained to ﬁt with a standard L2 loss random labels sampled from a binary distribution.

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: As can be seen, in all cases, the network exhibits a convex-mimicking behaviour, similar to the results of our CNN experiments.
Revised: As can be seen, in all cases, the network exhibits a convex-mimicking behaviour, similar to the results of our CNN experiments.

Original: Speciﬁcally, the curvature observed in H ˆ g seems to closely follow that of
Revised: Speciﬁcally, the curvature observed in H ˆ g seems to closely follow that of

Original: G ˆ g .
Revised: G ˆ g .

Original: In addition, as predicted in our analysis, the curve of
Revised: In addition, as predicted in our analysis, the curve of

Original: H ˆ g starts closer to zero for wider networks.
Revised: H ˆ g starts closer to zero for wider networks.

Original: 3. Analysis
Revised: 3. Analysis

Original: An initial analysis is performed for linear networks (no non-linearities) at initialization, where the weights of the networks are randomly sampled.
Revised: An initial analysis is performed for linear networks (no non-linearities) at initialization, where the weights of the networks are randomly sampled.

Original: We make very few assumptions on the nature of the networks’ initialization, and the popular initialization techniques satisfy our assumptions.
Revised: We make very few assumptions on the nature of the networks’ initialization, and the popular initialization techniques satisfy our assumptions.

Original: 3.1. Overview
Revised: 3.1. Overview

Original: In our theoretical analysis, we focus on the Hessian of a single output linear network with L layers, where the output of layer l = 1 , . . .
Revised: In our theoretical analysis, we focus on the Hessian of a single output linear network with L layers, where the output of layer l = 1 , . . .

Original: , L is given by y l = W l (cid:62) y l −where W l ∈ R n l − 1 × n l , n L = 1 , y l ∈ R n l , and the input, which is also denoted by y 0 is given by a ﬁxed dataset { x s } Ns =1 , x s ∈ R n 0 .
Revised: , L is given by y l = W l (cid:62) y l −where W l ∈ R n l − 1 × n l , n L = 1 , y l ∈ R n l , and the input, which is also denoted by y 0 is given by a ﬁxed dataset { x s } Ns =1 , x s ∈ R n 0 .

Original: We denote by W the concatenation of all the parameters of the network in a column vector.
Revised: We denote by W the concatenation of all the parameters of the network in a column vector.

Original: As mentioned, we consider a twice differentiable convex loss function L .
Revised: As mentioned, we consider a twice differentiable convex loss function L .

Original: We further assume that the loss has bounded ﬁrst and second derivatives.
Revised: We further assume that the loss has bounded ﬁrst and second derivatives.

Original: The Hessian matrix is given by:
Revised: The Hessian matrix is given by:

Original: [Equation]
Revised: [Equation]

Original: where g sw is the gradient of the network output unit for input s with respect to the weights.
Revised: where g sw is the gradient of the network output unit for input s with respect to the weights.

Original: The ﬁrst term of the Hessian is comprised of a sum of positive semideﬁnite matrices, and is, therefore, positive semideﬁnite, while the second term is not, and is dependent on the architecture.
Revised: The ﬁrst term of the Hessian is comprised of a sum of positive semideﬁnite matrices, and is, therefore, positive semideﬁnite, while the second term is not, and is dependent on the architecture.

Original: In the following, we aim to show that under certain conditions pretaining to the width of the network and the dimensionality of the input, which are usually met in practice, the trajectory of GD is inﬂuenced primarily by G .
Revised: In the following, we aim to show that under certain conditions pretaining to the width of the network and the dimensionality of the input, which are usually met in practice, the trajectory of GD is inﬂuenced primarily by G .

Original: That is, denoting by index t the evaluation of a tensor at iteration t , it holds that:
Revised: That is, denoting by index t the evaluation of a tensor at iteration t , it holds that:

Original: [Equation]
Revised: [Equation]

Revised: ≈

Revised: [Equation]

Revised: >(5)

Original: 3.2. Main Results
Revised: 3.2. Main Results

Original: In our analysis, we consider the weights W k
Revised: In our analysis, we consider the weights W k

Original: =
Revised: =

Original: [ w kij ] 0 <i ≤ n k , 0 <j ≤ n k − 1 , 0 <k ≤ L sampled iid from a symmetric distribution with moments m t , such that m 2 , m 4 , m 6 ≈n k − 1 , where m t = E (cid:0) ( w kij ) t (cid:1) are the moments of the distribution.
Revised: [ w kij ] 0 <i ≤ n k , 0 <j ≤ n k − 1 , 0 <k ≤ L sampled iid from a symmetric distribution with moments m t , such that m 2 , m 4 , m 6 ≈n k − 1 , where m t = E (cid:0) ( w kij ) t (cid:1) are the moments of the distribution.

Original: Notice that this condition applies to the common initialization scheme (He et al., 2015).
Revised: Notice that this condition applies to the common initialization scheme (He et al., 2015).

Original: We ﬁrst consider the gradient and Hessian evaluated on a single sample with unit norm, i.e., (cid:107) y 0 (cid:107) = 1 , and then discuss multiple samples.
Revised: We ﬁrst consider the gradient and Hessian evaluated on a single sample with unit norm, i.e., (cid:107) y 0 (cid:107) = 1 , and then discuss multiple samples.

Original: We make the following notations: the derivative of the output unit with respect to a single weight ij in layer k is denoted by g kij =
Revised: We make the following notations: the derivative of the output unit with respect to a single weight ij in layer k is denoted by [Equation] =

Original: [Equation]
Revised: [Equation]

Original: we denote ˆ H by:
Revised: we denote ˆ H by:

Original: [Equation]
Revised: [Equation]

Original: where the triplets of indices ijk, uvl parameterize row and column indices in the matrix ˆ H (note that the ranges of ij, uv depend on the layer indices k and l ).
Revised: where the triplets of indices ijk, uvl parameterize row and column indices in the matrix ˆ H (note that the ranges of ij, uv depend on the layer indices k and l ).

Original: The next theorem gives the expected mean and variance of g (cid:62) ˆ Hg for wide networks at initialization:
Revised: The next theorem gives the expected mean and variance of g (cid:62) ˆ Hg for wide networks at initialization:

Original: Theorem 1.
Revised: Theorem 1.

Original: E (cid:16) g T
Revised: E (cid:16) g T

Original: ˆ
Revised: ˆ

Original: Hg (cid:17) = 0 and for min l<L ( n l ) >> L , it holds that E (cid:16) ( g T ˆ Hg [Equation]
Revised: Hg (cid:17) = 0 and for min l<L ( n l ) >> L , it holds that E (cid:16) ( g T ˆ Hg ) 2 (cid:17)

Original: 2 (cid:17) ≈ Θ( 
Revised: ≈ Θ( (cid:80)

Original: [Equation]
Revised: [Equation]

Original: The Hessian gradient product is not necessarily indicative of the local curvature, since we do not take into account the norm of the gradient.
Revised: The Hessian gradient product is not necessarily indicative of the local curvature, since we do not take into account the norm of the gradient.

Original: As shown in (Arpit & Bengio, 2019), a constant width network ensures that the norm of the gradient approximately equals the input.
Revised: As shown in (Arpit & Bengio, 2019), a constant width network ensures that the norm of the gradient approximately equals the input.

Original: More speciﬁcally, by having ∀ 0 ≤ l ≤ L , n l = nm l , we ensure that (cid:107) g (cid:107) 2 →(cid:80)
Revised: More speciﬁcally, by having ∀ 0 ≤ l ≤ L , n l = nm l , we ensure that (cid:107) g (cid:107) 2 →

Original: [Equation]
Revised: [Equation]

Original: (cid:107) y
Revised: Ll

Original: [Equation]
Revised: [Equation]

Original: (cid:107) {+y 
 (cid:107)+} 2 as n gets larger.
Revised: (cid:107) {+y 
 (cid:107)+} 2 as n gets larger.

Original: (The precise statement is omitted for brevity).
Revised: (The precise statement is omitted for brevity).

Original: The next theorem gives a bound on the probability of the local curvature to be above some (cid:15) , given the full Hessian.
Revised: The next theorem gives a bound on the probability of the local curvature to be above some (cid:15) , given the full Hessian.

Original: Theorem 2.
Revised: Theorem 2.

Original: Assume that ∀ 0 ≤ l ≤ L , n l = nm l , andProb
Revised: Assume that ∀ 0 ≤ l ≤ L , n l = nm l , and

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: y

Original: [Equation]
Revised: [Equation]

Original: L

Original: [Equation]
Revised: [Equation]

Original: y L ) > α > 0 , L
Revised: 0 , L

Original: [Equation]
Revised: [Equation]

Original: ( y L ) > β > 

Revised: ( y L ) > β >

Original: then it holds that:
Revised: then it holds that:

Original: [Equation]
Revised: [Equation]

Original: The above analysis provides a rather surprising result, which is that the dimensionality of the input also plays a role in determining the local curvature at initialization, in addition to the width of the network.
Revised: The above analysis provides a rather surprising result, which is that the dimensionality of the input also plays a role in determining the local curvature at initialization, in addition to the width of the network.

Original: We have discussed a single sample in our analysis so far.
Revised: We have discussed a single sample in our analysis so far.

Original: In the case of linear neural networks, the result of Thm. 1 also hold for the batch case since we did not assume anything on the input.
Revised: In the case of linear neural networks, the result of Thm. 1 also hold for the batch case since we did not assume anything on the input.

Original: Speciﬁcally, since H = (cid:80) Ns =1 L (cid:48) ( y ( x s ) L )
Revised: Speciﬁcally, since H = (cid:80) Ns [Equation] L [Equation]

Original:  ˆ H s , and considering the gradient of sample u , and ˆ H v evaluated on sample v , then it holds that g u 
Revised: ( y [Equation] ) ˆ H s , and considering the gradient of sample u , and ˆ H v evaluated on sample v , then it holds that g u (cid:62)

Original: [Equation]
Revised: [Equation]

Original: The term g (cid:62) Gg can be signiﬁcantly smaller in the batch case, even when the gradient is large.
Revised: The term g (cid:62) Gg can be signiﬁcantly smaller in the batch case, even when the gradient is large.

Original: Indeed, under mild conditions, when gradients of individual samples are orthogonal to each other, this term can be made small while keeping the overall gradient large, insuring a faster convergence to the optimum.
Revised: Indeed, under mild conditions, when gradients of individual samples are orthogonal to each other, this term can be made small while keeping the overall gradient large, insuring a faster convergence to the optimum.

Original: Precise statement and analysis are kept for future work.
Revised: Precise statement and analysis are kept for future work.

Original: 4. Discussion
Revised: 4. Discussion

Original: The literature has already linked the width of a neural network to the property of norm-preserving.
Revised: The literature has already linked the width of a neural network to the property of norm-preserving.

Original: For example, (Arpit & Bengio, 2019) show that under different sets of assumptions, the wider the network, the more likely the norm of the activations will be preserved with depth.
Revised: For example, (Arpit & Bengio, 2019) show that under different sets of assumptions, the wider the network, the more likely the norm of the activations will be preserved with depth.

Original: Our work is the ﬁrst one, as far as we know, to show a link between a convex-mimicking behavior during optimization and the network’s width.
Revised: Our work is the ﬁrst one, as far as we know, to show a link between a convex-mimicking behavior during optimization and the network’s width.

Original: The literature also teaches us about the properties of the
Revised: The literature also teaches us about the properties of the

Original: Hessian during training (Sagun et al., 2016).
Revised: Hessian during training (Sagun et al., 2016).

Original: As far as we know, we are unique in that we look at the Hessian in the direction of the gradient, where it matters most to the optimization process.
Revised: As far as we know, we are unique in that we look at the Hessian in the direction of the gradient, where it matters most to the optimization process.

Original: The line of research can lead to a better understanding of the link between different architectures and their trainability.
Revised: The line of research can lead to a better understanding of the link between different architectures and their trainability.

Original: Moreover, the signal we track can be readily used in order to drive the optimization process.
Revised: Moreover, the signal we track can be readily used in order to drive the optimization process.

Original: Speciﬁcally, it may be derivable to adjust the hyperparameters (learning rate, batch size, batch composition, etc.)
Revised: Speciﬁcally, it may be derivable to adjust the hyperparameters (learning rate, batch size, batch composition, etc.)

Original: such that the gradients are large while the product of the gradient with the Hessian implies close to zero curvature.
Revised: such that the gradients are large while the product of the gradient with the Hessian implies close to zero curvature.

Original: [Figure]
Revised: Acknowledgements

Original: [Equation] [Figure]
Revised: This project has received funding from the European Research Council (ERC) under the European Unions Horizon 2020 research and innovation programme (grant ERC CoG 725974).

Original: A. Proof of Thm.
Revised: A. Proof of Thm.

Original: Proof.
Revised: Proof.

Original: The derivative of unit j in layer k with respect to unit u in layer l is denoted by ∂y kj ∂y lu = a kjlu .
Revised: The derivative of unit j in layer k with respect to unit u in layer l is denoted by ∂y kj ∂y lu = a kjlu .

Original: Dropping the unit indexes j or u means deriving the entire layer, or with respect to the entire layer respectively.
Revised: Dropping the unit indexes j or u means deriving the entire layer, or with respect to the entire layer respectively.

Original: Dropping all superscripts means derivative of the output unit, so that ∂y L ∂y lu = a lu .
Revised: Dropping all superscripts means derivative of the output unit, so that ∂y L ∂y lu = a lu .

Original: It then holds that g kij = a ki y k − 1 j .
Revised: It then holds that g kij = a ki y k − 1 j .

Original: Finally, we denote by W l,i and W li, the i (cid:48)
Revised: Finally, we denote by W l,i and W li, the i (cid:48)

Original: th column and row vectors of W l respectively.
Revised: th column and row vectors of W l respectively.

Original: We have:
Revised: We have:

Original: [Equation]
Revised: [Equation]

Original: where the triplets of indices ijk, uvl parameterize row and column indices in the matrix ˆ H (note that the ranges of ij, uv depend on the layer indices k and l ).
Revised: where the triplets of indices ijk, uvl parameterize row and column indices in the matrix ˆ H (note that the ranges of ij, uv depend on the layer indices k and l ).

Original: The Hessian gradient product is given by: (cid:88) ijk
Revised: The Hessian gradient product is given by: (cid:88) ijk

Original: [Equation]
Revised: [Equation]

Original: The claim E (cid:16) g T ˆ Hg (cid:17) = 0 is trivially true, since each term in the Hessian gradient products will contain odd moments of the weights, which zero out for symmetric distributions.
Revised: The claim E (cid:16) g T ˆ Hg (cid:17) = 0 is trivially true, since each term in the Hessian gradient products will contain odd moments of the weights, which zero out for symmetric distributions.

Original: For deep networks, we have that:
Revised: For deep networks, we have that:

Original: [Equation]
Revised: [Equation]

Original: and so: (cid:88)
Revised: and so: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: where we assumed in the last transition that min l<L ( n l ) >> L .
Revised: where we assumed in the last transition that min l<L ( n l ) >> L .

Original: Applying expectation on the last weight layer W L , it holds that:
Revised: Applying expectation on the last weight layer W L , it holds that:

Original: E  
Revised: E  

Original: [Table]
Revised: [Table]

Original: Plugging each term in Eq.
Revised: Plugging each term in Eq.

Original: 12 yields the same asymptotic behaviour.
Revised: 12 yields the same asymptotic behaviour.

Original: For the sake of brevity, we will show this with the ﬁrst term.
Revised: For the sake of brevity, we will show this with the ﬁrst term.

Original: Without loss of generality, we assume k > k (cid:48)
Revised: Without loss of generality, we assume k > k (cid:48)

Original: + 1 > l + 2 > l (cid:48)
Revised: + 1 > l + 2 > l (cid:48)

Original: + 3 , we now recursively apply expectations over
Revised: + 3 , we now recursively apply expectations over

Original: W L − 1 ...
Revised: W L − 1 ...

Original: W k , while noticing that the following hold for ﬁxed vectors v 1 ...v 6 :
Revised: W k , while noticing that the following hold for ﬁxed vectors v 1 ...v 6 :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and so:
Revised: and so:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: ≈ n 3 k −
Revised: ≈ n 3 k −

Original: E (cid:18) (cid:88)
Revised: E (cid:18) (cid:88)

Original: [Table]
Revised: [Table]

Original: ≈ n k
Revised: ≈ n k

Original: [Table]
Revised: [Table]

Original: n
Revised: n

Original: [Table]
Revised: [Table]

Original: E
Revised: E

Original: [Table]
Revised: [Table]

Original: (cid:88)
Revised: (cid:88)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Recursively going through l = k − 2 ...
Revised: Recursively going through l = k − 2 ...

Original: 1 we have:
Revised: 1 we have:

Original: [Equation]
Revised: [Equation]

Original: ≈
Revised: ≈

Original: [Equation]
Revised: [Equation]

Original: Summing over k, k (cid:48) :
Revised: Summing over k, k (cid:48) :

Original: [Equation]
Revised: [Equation]

Original: B. Proof of Thm.
Revised: B. Proof of Thm.

Original: Proof.
Revised: Proof.

Original: We have:
Revised: We have:

Original: Prob (cid:18)
Revised: Prob (cid:18)

Original: [Equation]
Revised: [Equation]

Original: = Prob
Revised: = Prob

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:15) (cid:19)
Revised: (cid:15) (cid:19)

Original: [Equation]
Revised: [Equation]

Original: (cid:18)
Revised: (cid:18)

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: (cid:48)
Revised: (cid:48)

Original: ||
Revised: ||

Original: [Equation]
Revised: [Equation]

Original: (cid:62) ˆ
Revised: (cid:62) ˆ

Original: [Equation]
Revised: [Equation]

Original: ˆ g | < (cid:15) (cid:19) ≥ Prob (cid:18) | α
Revised: ˆ g | < (cid:15) (cid:19) ≥ Prob (cid:18) | α

Original: [Equation]
Revised: [Equation]

Original: (cid:107) g (cid:107)
Revised: (cid:107) g (cid:107)

Original: [Equation]
Revised: [Equation]

Original: 2 (cid:15) (cid:19) Prob (cid:18)
Revised: 2 (cid:15) (cid:19) Prob (cid:18)

Original: | β
Revised: | β

Original: || ˆ g (cid:62) ˆ
Revised: || ˆ g (cid:62) ˆ

Original: H ˆ g | < (cid:15) (cid:19) (18)
Revised: H ˆ g | < (cid:15) (cid:19) (18)

Original: [Table]
Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation] [Table] 
Revised: [Equation] [Table] [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation]

Original: [Table]
Revised: [Table]

Original: From The.
Revised: From The.

Original: [Table]
Revised: [Table]

Original: we have that var
Revised: we have that var

Original: [Equation]
Revised: [Equation]

Original: g
Revised: g

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: for some γ > 0 .
Revised: for some γ > 0 .

Original: Using Chebyshev’s inequality:
Revised: Using Chebyshev’s inequality:

Original: Prob
Revised: Prob

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: and so:
Revised: and so:

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Finally:
Revised: Finally:

Original: Prob
Revised: Prob

Original: [Equation]
Revised: [Equation]