Original: E LLIPSOIDAL T RUST R EGION M ETHODS FOR N EURAL N ETWORK T RAINING
Revised: E LLIPSOIDAL T RUST R EGION M ETHODS FOR N EURAL N ETWORK T RAINING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We investigate the use of ellipsoidal trust region constraints for second-order optimization of neural networks.
Revised: We investigate the use of ellipsoidal trust region constraints for second-order optimization of neural networks.

Original: This approach can be seen as a higher-order counterpart of adaptive gradient methods, which we here show to be interpretable as ﬁrst-order trust region methods with ellipsoidal constraints.
Revised: This approach can be seen as a higher-order counterpart of adaptive gradient methods, which we here show to be interpretable as ﬁrst-order trust region methods with ellipsoidal constraints.

Original: In particular, we show that the preconditioning matrix used in RMSProp and Adam satisﬁes the necessary conditions for provable convergence of second-order trust region methods with standard worst-case complexities.
Revised: In particular, we show that the preconditioning matrix used in RMSProp and Adam satisﬁes the necessary conditions for provable convergence of second-order trust region methods with standard worst-case complexities.

Original: Furthermore, we run experiments across different neural architectures and datasets to ﬁnd that the ellipsoidal constraints constantly outperform their spherical counterpart both in terms of number of backpropagations and asymptotic loss value.
Revised: Furthermore, we run experiments across different neural architectures and datasets to ﬁnd that the ellipsoidal constraints constantly outperform their spherical counterpart both in terms of number of backpropagations and asymptotic loss value.

Original: Finally, we ﬁnd comparable performance to state-ofthe-art ﬁrst-order methods in terms of backpropagations, but further advances in hardware are needed to render Newton methods competitive in terms of time
Revised: Finally, we ﬁnd comparable performance to state-ofthe-art ﬁrst-order methods in terms of backpropagations, but further advances in hardware are needed to render Newton methods competitive in terms of time

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: We consider ﬁnite-sum optimization problems of the form
Revised: We consider ﬁnite-sum optimization problems of the form

Original: [Equation]
Revised: [Equation]

Original: which typically arise in neural network training, e.g. for empirical risk minimization over a set of data points ( x i , y i )
Revised: which typically arise in neural network training, e.g. for empirical risk minimization over a set of data points ( x i , y i )

Original: ∈ R in × R out , i = 1 , . . . , n .
Revised: ∈ R in × R out , i = 1 , . . . , n .

Original: Here, (cid:96) :
Revised: Here, (cid:96) :

Original: R out × R out → R + is a convex loss function and f : R in × R d → R out represents the neural network mapping parameterized by the concatenation of layers w ∈ R d , which is non-convex due to its multiplicative nature and potentially non-linear activation functions.
Revised: R out × R out → R + is a convex loss function and f : R in × R d → R out represents the neural network mapping parameterized by the concatenation of layers w ∈ R d , which is non-convex due to its multiplicative nature and potentially non-linear activation functions.

Original: We assume that L is twice differentiable, i.e. L ∈ C 2 ( R d , R ) and consider ﬁnding a ﬁrst- and second-order stationary point ¯ w for which (cid:107)∇L ( ¯ w ) (cid:107) ≤ (cid:15) g and λ min (cid:0) ∇ 2 L ( ¯ w ) (cid:1) ≥ − (cid:15) H .
Revised: We assume that L is twice differentiable, i.e. L ∈ C 2 ( R d , R ) and consider ﬁnding a ﬁrst- and second-order stationary point ¯ w for which (cid:107)∇L ( ¯ w ) (cid:107) ≤ (cid:15) g and λ min (cid:0) ∇ 2 L ( ¯ w ) (cid:1) ≥ − (cid:15) H .

Original: Non-convex optimization problems are ubiquitous in machine learning.
Revised: Non-convex optimization problems are ubiquitous in machine learning.

Original: Among the most prominent examples are present-day deep neural networks, that have achieved outstanding results on core tasks such as collaborative ﬁltering (Wang et al., 2015), sentence classiﬁcation (Kim, 2014) and image classiﬁcation (Krizhevsky et al., 2012).
Revised: Among the most prominent examples are present-day deep neural networks, that have achieved outstanding results on core tasks such as collaborative ﬁltering (Wang et al., 2015), sentence classiﬁcation (Kim, 2014) and image classiﬁcation (Krizhevsky et al., 2012).

Original: In the era of big data and deep neural networks, stochastic gradient descent (SGD) is one of the most widely used training algorithms (Bottou, 2010).
Revised: In the era of big data and deep neural networks, stochastic gradient descent (SGD) is one of the most widely used training algorithms (Bottou, 2010).

Original: What makes SGD so attractive is its simplicity and per-iteration cost that are independent of the size of the training set ( n ) and scale linearly in the dimensionality ( d ).
Revised: What makes SGD so attractive is its simplicity and per-iteration cost that are independent of the size of the training set ( n ) and scale linearly in the dimensionality ( d ).

Original: However, gradient descent is known to be inadequate to optimize functions that are ill-conditioned (Nesterov, 2013; Shalev-Shwartz et al., 2017) and thus adaptive gradient methods that employ dynamic, coordinate-wise learning rates based on past gradients—including Adagrad (Duchi et al., 2011), RMSprop (Tieleman & Hinton, 2012) and Adam (Kingma & Ba, 2014)—have become a popular alternative, often providing signiﬁcant speed-ups over SGD.
Revised: However, gradient descent is known to be inadequate to optimize functions that are ill-conditioned (Nesterov, 2013; Shalev-Shwartz et al., 2017) and thus adaptive gradient methods that employ dynamic, coordinate-wise learning rates based on past gradients—including Adagrad (Duchi et al., 2011), RMSprop (Tieleman & Hinton, 2012) and Adam (Kingma & Ba, 2014)—have become a popular alternative, often providing signiﬁcant speed-ups over SGD.

Original: Yet, there exist no theoretical proofs that these methods are faster than gradient descent (Li & Orabona, 2018).
Revised: Yet, there exist no theoretical proofs that these methods are faster than gradient descent (Li & Orabona, 2018).

Original: From a theoretical perspective, Newton methods provide stronger convergence guarantees by appropriately transforming the gradient in ill-conditioned regions according to second-order derivatives.
Revised: From a theoretical perspective, Newton methods provide stronger convergence guarantees by appropriately transforming the gradient in ill-conditioned regions according to second-order derivatives.

Original: It is precisely this Hessian information that allows regularized Newton methods to enjoy superlinear local convergence as well as to provably escape saddle points (Conn et al., 2000).
Revised: It is precisely this Hessian information that allows regularized Newton methods to enjoy superlinear local convergence as well as to provably escape saddle points (Conn et al., 2000).

Original: While second-order algorithms have a long-standing history even in the realm of neural network training (Hagan &
Revised: While second-order algorithms have a long-standing history even in the realm of neural network training (Hagan &

Original: Menhaj, 1994; Becker et al., 1988), they were mostly considered as too computationally and memory expensive for practical applications.
Revised: Menhaj, 1994; Becker et al., 1988), they were mostly considered as too computationally and memory expensive for practical applications.

Original: Yet, the seminal work of Martens (2010) renewed interest for their use in deep learning by proposing efﬁcient Hessian-free methods that only access second-order information via matrix-vector products which can be computed at the cost of an additional backpropagation (Pearlmutter, 1994; Schraudolph, 2002).
Revised: Yet, the seminal work of Martens (2010) renewed interest for their use in deep learning by proposing efﬁcient Hessian-free methods that only access second-order information via matrix-vector products which can be computed at the cost of an additional backpropagation (Pearlmutter, 1994; Schraudolph, 2002).

Original: Among the class of regularized Newton methods, trust region (Conn et al., 2000) and cubic regularization algorithms (Cartis et al., 2011) are the most principled approaches in the sense that they yield the strongest convergence guarantees.
Revised: Among the class of regularized Newton methods, trust region (Conn et al., 2000) and cubic regularization algorithms (Cartis et al., 2011) are the most principled approaches in the sense that they yield the strongest convergence guarantees.

Original: Recently, stochastic extensions have emerged (Xu et al., 2017b; Yao et al., 2018; Kohler & Lucchi, 2017;
Revised: Recently, stochastic extensions have emerged (Xu et al., 2017b; Yao et al., 2018; Kohler & Lucchi, 2017;

Original: Gratton et al., 2017), which suggest their applicability for deep learning.
Revised: Gratton et al., 2017), which suggest their applicability for deep learning.

Original: We here propose a simple modiﬁcation to make TR methods even more suitable for neural network training.
Revised: We here propose a simple modiﬁcation to make TR methods even more suitable for neural network training.

Original: Particularly, we build upon the following alternative view on adaptive gradient methods:
Revised: Particularly, we build upon the following alternative view on adaptive gradient methods:

Original: While gradient descent can be interpreted as a spherically constrained ﬁrst-order TR method, preconditioned gradient methods—such as Adagrad—can be seen as ﬁrst-order TR methods with ellipsoidal trust region constraint.
Revised: While gradient descent can be interpreted as a spherically constrained ﬁrst-order TR method, preconditioned gradient methods—such as Adagrad—can be seen as ﬁrst-order TR methods with ellipsoidal trust region constraint.

Original: This observation is particularly interesting since spherical constraints are blind to the underlying geometry of the problem, but ellipsoids can adapt to local landscape characteristics, thereby allowing for more suitable steps in regions that are ill-conditioned.
Revised: This observation is particularly interesting since spherical constraints are blind to the underlying geometry of the problem, but ellipsoids can adapt to local landscape characteristics, thereby allowing for more suitable steps in regions that are ill-conditioned.

Original: We will leverage this analogy and investigate the use of the Adagrad and RMSProp preconditioning matrices as ellipsoidal trust region shapes within a fully stochastic second-order TR algorithm named STORM (Chen et al., 2018; Gratton et al., 2017).
Revised: We will leverage this analogy and investigate the use of the Adagrad and RMSProp preconditioning matrices as ellipsoidal trust region shapes within a fully stochastic second-order TR algorithm named STORM (Chen et al., 2018; Gratton et al., 2017).

Original: While the theory for ellipsoidal TR methods is well-studied (e.g. Conn et al.
Revised: While the theory for ellipsoidal TR methods is well-studied (e.g. Conn et al.

Original: ; Yuan (2015)), no ellipsoid ﬁts all objective functions and our main contribution thus lies in the identiﬁcation of adequate matrix-induced constraints that lead to provable convergence and signiﬁcant practical speed-ups for the speciﬁc case of deep learning.
Revised: ; Yuan (2015)), no ellipsoid ﬁts all objective functions and our main contribution thus lies in the identiﬁcation of adequate matrix-induced constraints that lead to provable convergence and signiﬁcant practical speed-ups for the speciﬁc case of deep learning.

Original: On the whole, our contribution is threefold:
Revised: On the whole, our contribution is threefold:

Original: • We provide a new perspective on adaptive gradient methods that contributes to a better understanding of their inner-workings.
Revised: • We provide a new perspective on adaptive gradient methods that contributes to a better understanding of their inner-workings.

Original: Furthermore, we empirically ﬁnd that many neural network problems exhibit diagonally dominated Hessian matrices which suggests the effectivity of diagonal preconditioning.
Revised: Furthermore, we empirically ﬁnd that many neural network problems exhibit diagonally dominated Hessian matrices which suggests the effectivity of diagonal preconditioning.

Original: (Section 3) • We investigate the ﬁrst application of ellipsoidal TR methods for deep learning.
Revised: (Section 3) • We investigate the ﬁrst application of ellipsoidal TR methods for deep learning.

Original: In Theoremwe show that the RMSProp matrix can directly be applied as constraint inducing norm in second-order TR algorithms while preserving all convergence guarantees.
Revised: In Theoremwe show that the RMSProp matrix can directly be applied as constraint inducing norm in second-order TR algorithms while preserving all convergence guarantees.

Original: (Section 4) •
Revised: (Section 4) •

Original: Finally, we provide an experimental benchmark across different real-world datasets and architectures.
Revised: Finally, we provide an experimental benchmark across different real-world datasets and architectures.

Original: We also compare against adaptive gradient methods and show results in terms of backprogations, epochs, and wall-clock time; a comparison we were not able to ﬁnd in the literature.
Revised: We also compare against adaptive gradient methods and show results in terms of backprogations, epochs, and wall-clock time; a comparison we were not able to ﬁnd in the literature.

Original: (Section 5)
Revised: (Section 5)

Original: Our main empirical results demonstrate that ellipsoidal constraints prove to be a very effective modiﬁcation of the trust region method in the sense that they constantly outperform the spherical TR method, both in terms of number of backprogations and asymptotic loss value on a variety of tasks.
Revised: Our main empirical results demonstrate that ellipsoidal constraints prove to be a very effective modiﬁcation of the trust region method in the sense that they constantly outperform the spherical TR method, both in terms of number of backprogations and asymptotic loss value on a variety of tasks.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: First-order methods The prototypical method for optimizing Eq.
Revised: First-order methods The prototypical method for optimizing Eq.

Original: (1) is SGD (Robbins & Monro, 1951).
Revised: (1) is SGD (Robbins & Monro, 1951).

Original: While the practical success of SGD in non-convex optimization is unquestioned, the theoretical foundation of this phenomenon is still rather limited.
Revised: While the practical success of SGD in non-convex optimization is unquestioned, the theoretical foundation of this phenomenon is still rather limited.

Original: Recent ﬁndings suggest the ability of this method to escape saddle points and reach local minima in polynomial time for general non-convex problems, but they either need to artiﬁcially add noise to the iterates (Ge et al., 2015; Lee et al., 2016) or make an assumption on the inherent noise of vanilla SGD (Daneshmand et al., 2018).
Revised: Recent ﬁndings suggest the ability of this method to escape saddle points and reach local minima in polynomial time for general non-convex problems, but they either need to artiﬁcially add noise to the iterates (Ge et al., 2015; Lee et al., 2016) or make an assumption on the inherent noise of vanilla SGD (Daneshmand et al., 2018).

Original: For neural network training, a recent line of research proclaims the effectiveness of SGD, but the results usually come at the cost of fairly strong assumptions such as heavy overparametrization and Gaussian inputs (Du et al., 2017; Brutzkus & Globerson, 2017; Li & Yuan, 2017; Du & Lee, 2018; Allen-Zhu et al., 2018).
Revised: For neural network training, a recent line of research proclaims the effectiveness of SGD, but the results usually come at the cost of fairly strong assumptions such as heavy overparametrization and Gaussian inputs (Du et al., 2017; Brutzkus & Globerson, 2017; Li & Yuan, 2017; Du & Lee, 2018; Allen-Zhu et al., 2018).

Original: Adaptive gradient methods (Duchi et al., 2011; Tieleman & Hinton, 2012; Kingma & Ba, 2014) build on the intuition that larger learning rates for smaller gradient components and smaller learning rates for larger gradient components balance their respective inﬂuences and thereby make the methods behave as if they were optimizing a more isotropic surface.
Revised: Adaptive gradient methods (Duchi et al., 2011; Tieleman & Hinton, 2012; Kingma & Ba, 2014) build on the intuition that larger learning rates for smaller gradient components and smaller learning rates for larger gradient components balance their respective inﬂuences and thereby make the methods behave as if they were optimizing a more isotropic surface.

Original: Such approaches have ﬁrst been suggested for neural networks by LeCun et al.
Revised: Such approaches have ﬁrst been suggested for neural networks by LeCun et al.

Original: Recently, convergence guarantees for such methods are starting to appear (Ward et al., 2018; Li & Orabona, 2018).
Revised: Recently, convergence guarantees for such methods are starting to appear (Ward et al., 2018; Li & Orabona, 2018).

Original: However, these are not superior to the O ( (cid:15) − 2 g ) worst-case complexity of standard gradient descent (Cartis et al., 2012b).
Revised: However, these are not superior to the O ( (cid:15) − 2 g ) worst-case complexity of standard gradient descent (Cartis et al., 2012b).

Original: Regularized Newton methods The most principled class of regularized Newton methods are trust region (TR) and adaptive cubic regularization algorithms (ARC) (Conn et al., 2000; Cartis et al., 2011), which repeatedly optimize a local Taylor model of the objective while making sure that the step does not travel too far such that the model stays accurate.
Revised: Regularized Newton methods The most principled class of regularized Newton methods are trust region (TR) and adaptive cubic regularization algorithms (ARC) (Conn et al., 2000; Cartis et al., 2011), which repeatedly optimize a local Taylor model of the objective while making sure that the step does not travel too far such that the model stays accurate.

Original: While the former ﬁnds ﬁrst-order stationary points within O ( (cid:15) − 2 g ) , ARC only takes at most O ( (cid:15) − 3 / 2 g ) .
Revised: While the former ﬁnds ﬁrst-order stationary points within O ( (cid:15) − 2 g ) , ARC only takes at most O ( (cid:15) − 3 / 2 g ) .

Original: However, simple modiﬁcations to the TR framework allow these methods to obtain the same accelerated rate (Curtis et al., 2017).
Revised: However, simple modiﬁcations to the TR framework allow these methods to obtain the same accelerated rate (Curtis et al., 2017).

Original: Both methods take at most O ( (cid:15) − 3 H ) iterations to ﬁnd an (cid:15) H approximate second-order stationary point (Cartis et al., 2012a).
Revised: Both methods take at most O ( (cid:15) − 3 H ) iterations to ﬁnd an (cid:15) H approximate second-order stationary point (Cartis et al., 2012a).

Original: These rates are optimal for second-order Lipschitz continuous functions (Carmon et al., 2017; Cartis et al., 2012a) and they can be retained even when only sub-sampled gradient and Hessian information is used (Kohler & Lucchi, 2017; Yao et al., 2018; Xu et al., 2017b; Blanchet et al., 2016;
Revised: These rates are optimal for second-order Lipschitz continuous functions (Carmon et al., 2017; Cartis et al., 2012a) and they can be retained even when only sub-sampled gradient and Hessian information is used (Kohler & Lucchi, 2017; Yao et al., 2018; Xu et al., 2017b; Blanchet et al., 2016;

Original: Liu et al., 2018; Cartis & Scheinberg, 2017).
Revised: Liu et al., 2018; Cartis & Scheinberg, 2017).

Original: Furthermore, the involved Hessian information can be computed solely based on Hessian-vector products, which are implementable efﬁciently for neural networks (Pearlmutter, 1994).
Revised: Furthermore, the involved Hessian information can be computed solely based on Hessian-vector products, which are implementable efﬁciently for neural networks (Pearlmutter, 1994).

Original: This makes these methods particularly attractive for deep learning, but the empirical evidence of their applicability is so far very limited.
Revised: This makes these methods particularly attractive for deep learning, but the empirical evidence of their applicability is so far very limited.

Original: We are only aware of the works of
Revised: We are only aware of the works of

Original: Liu et al.
Revised: Liu et al.

Original: (2018) and Xu et al.
Revised: (2018) and Xu et al.

Original: (2017a), which report promising ﬁrst results
Revised: (2017a), which report promising ﬁrst results

Original: but these are by no means fully encompassing.
Revised: but these are by no means fully encompassing.

Original: Gauss-Newton methods An interesting line of research proposes to replace the Hessian by (approximations of) the generalized-Gauss-Newton matrix (GGN) within a Levenberg-Marquardt framework(LeCun et al., 2012; Martens, 2010; Martens & Grosse, 2015).
Revised: Gauss-Newton methods An interesting line of research proposes to replace the Hessian by (approximations of) the generalized-Gauss-Newton matrix (GGN) within a Levenberg-Marquardt framework(LeCun et al., 2012; Martens, 2010; Martens & Grosse, 2015).

Original: These methods have been termed hessian-free since only access to GGN-vector products is required.
Revised: These methods have been termed hessian-free since only access to GGN-vector products is required.

Original: As the GGN matrix is always positive semideﬁnite, they cannot leverage negative curvature to escape saddles and hence, there exist no second-order convergence guarantees.
Revised: As the GGN matrix is always positive semideﬁnite, they cannot leverage negative curvature to escape saddles and hence, there exist no second-order convergence guarantees.

Original: Furthermore, there are cases in neural network training where the Hessian is better conditioned than the GGN matrix (Mizutani & Dreyfus, 2008).
Revised: Furthermore, there are cases in neural network training where the Hessian is better conditioned than the GGN matrix (Mizutani & Dreyfus, 2008).

Original: Nevertheless, the above works report promising preliminary results, most notably Grosse & Martens (2016) report that K-FAC can be faster than SGD on a small convnet.
Revised: Nevertheless, the above works report promising preliminary results, most notably Grosse & Martens (2016) report that K-FAC can be faster than SGD on a small convnet.

Original: On the other hand, recent ﬁndings report performance at best comparable to SGD on the much larger ResNet architecture (Ma et al., 2019).
Revised: On the other hand, recent ﬁndings report performance at best comparable to SGD on the much larger ResNet architecture (Ma et al., 2019).

Original: Moreover, Xu et al.
Revised: Moreover, Xu et al.

Original: (2017a) reports many cases where TR and GGN algorithms perform similarly.
Revised: (2017a) reports many cases where TR and GGN algorithms perform similarly.

Original: This line of work is to be seen as complementary to our approach since it is straight forward to replace the Hessian in the TR framework with the GGN matrix.
Revised: This line of work is to be seen as complementary to our approach since it is straight forward to replace the Hessian in the TR framework with the GGN matrix.

Original: Furthermore, the preconditioners used in
Revised: Furthermore, the preconditioners used in

Original: Martens (2010) and Chapelle & Erhan (2011), namely diagonal estimates of the empirical Fisher and
Revised: Martens (2010) and Chapelle & Erhan (2011), namely diagonal estimates of the empirical Fisher and

Original: Fisher matrix, respectively, can directly be used as matrix norms in our ellipsoidal TR framework.
Revised: Fisher matrix, respectively, can directly be used as matrix norms in our ellipsoidal TR framework.

Original:  A N ALTERNATIVE VIEW ON ADAPTIVE GRADIENT METHODS
Revised:  A N ALTERNATIVE VIEW ON ADAPTIVE GRADIENT METHODS

Original: Adaptively preconditioned gradient methods update iterates as w t +1 =
Revised: Adaptively preconditioned gradient methods update iterates as w t +1 =

Original: w t − η t
Revised: w t − η t

Original: A − 1 / 2 t g t , where g t is a stochastic estimate of ∇L ( w t ) and A t is a positive deﬁnite symmetric pre-conditioning matrix.
Revised: A − 1 / 2 t g t , where g t is a stochastic estimate of ∇L ( w t ) and A t is a positive deﬁnite symmetric pre-conditioning matrix.

Original: In Adagrad, A ada,t is the un-centered second moment matrix of the past gradients computed as
Revised: In Adagrad, A ada,t is the un-centered second moment matrix of the past gradients computed as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where (cid:15) > 0 , I is the d × d identity matrix and G t =
Revised: where (cid:15) > 0 , I is the d × d identity matrix and G t =

Original: [ g 1 , g 2 , . . . , g t ] .
Revised: [ g 1 , g 2 , . . . , g t ] .

Original: Building up on the intuition that past gradients might become obsolete in quickly changing non-convex landscapes, RMSprop (and Adam) introduce an exponential weight decay leading to the preconditioning matrix
Revised: Building up on the intuition that past gradients might become obsolete in quickly changing non-convex landscapes, RMSprop (and Adam) introduce an exponential weight decay leading to the preconditioning matrix

Original: [Equation]
Revised: [Equation]

Original: where β ∈ (0 , 1) .
Revised: where β ∈ (0 , 1) .

Original: In order to save computational efforts, the diagonal versions diag( A ada ) and diag( A rms ) are more commonly applied in practice, which in turn gives rise to coordinate-wise adaptive stepsizes that are enlarged (reduced) in coordinates that have seen past gradient components with a smaller (larger) magnitude.
Revised: In order to save computational efforts, the diagonal versions diag( A ada ) and diag( A rms ) are more commonly applied in practice, which in turn gives rise to coordinate-wise adaptive stepsizes that are enlarged (reduced) in coordinates that have seen past gradient components with a smaller (larger) magnitude.

Original: In that way, the optimization methods can account for gradients of potentially different scales arising from e.g. different layers of the networks.
Revised: In that way, the optimization methods can account for gradients of potentially different scales arising from e.g. different layers of the networks.

Original: 3. A DAPTIVE PRECONDITIONING AS ELLIPSOIDAL T RUST R EGION
Revised: 3. A DAPTIVE PRECONDITIONING AS ELLIPSOIDAL T RUST R EGION

Original: Starting from the fact that adaptive methods employ coordinate-wise stepsizes, one can take a principled view of these methods.
Revised: Starting from the fact that adaptive methods employ coordinate-wise stepsizes, one can take a principled view of these methods.

Original: Namely, their update steps arise from minimizing a ﬁrst-order
Revised: Namely, their update steps arise from minimizing a ﬁrst-order

Original: Taylor model of the function L within an ellipsoidal search space around the current iterate w t , where the diameter of the ellipsoid along a particular coordinate is implicitly given by η t and (cid:107) g t (cid:107)
Revised: Taylor model of the function L within an ellipsoidal search space around the current iterate w t , where the diameter of the ellipsoid along a particular coordinate is implicitly given by η t and (cid:107) g t (cid:107)

Original: A
Revised: A

Original: [Figure]
Revised: [Figure]

Original: 1 t .
Revised: 1 t .

Original: Correspondingly, vanilla (S)GD optimizes the same ﬁrst-order model within a spherical constraint.
Revised: Correspondingly, vanilla (S)GD optimizes the same ﬁrst-order model within a spherical constraint.

Original: Fig.
Revised: Fig.

Original: 1 (top) illustrates this effect by showing not only the iterates of GD and Adagrad but also the implicit trust regions within which the local models were optimized at each step.
Revised: 1 (top) illustrates this effect by showing not only the iterates of GD and Adagrad but also the implicit trust regions within which the local models were optimized at each step.

Original: Since the models are linear, the constrained minimizer is always found on the boundary.
Revised: Since the models are linear, the constrained minimizer is always found on the boundary.

Original: [Figure]
Revised: [Figure]

Original: It is well known that GD struggles to progress towards the minimizer of quadratics along lowcurvature directions (see e.g., Goh (2017)).
Revised: It is well known that GD struggles to progress towards the minimizer of quadratics along lowcurvature directions (see e.g., Goh (2017)).

Original: While this effect is negligible for well-conditioned objectives (Fig. 1, left), it leads to drastically slow-down when the problem is ill-conditioned (Fig. 1, center).
Revised: While this effect is negligible for well-conditioned objectives (Fig. 1, left), it leads to drastically slow-down when the problem is ill-conditioned (Fig. 1, center).

Original: Particularly, once the method has reached the bottom of the valley, it struggles to make progress along the horizontal axis.
Revised: Particularly, once the method has reached the bottom of the valley, it struggles to make progress along the horizontal axis.

Original: Here is precisely where the advantage of adaptive stepsize methods comes into play.
Revised: Here is precisely where the advantage of adaptive stepsize methods comes into play.

Original: As illustrated by the dashed lines, Adagrad’s search space is damped along the direction of high curvature (vertical axis) and elongated along the low curvature direction (horizontal axis).
Revised: As illustrated by the dashed lines, Adagrad’s search space is damped along the direction of high curvature (vertical axis) and elongated along the low curvature direction (horizontal axis).

Original: This allows the method to move further horizontally early on to enter the valley with a smaller distance to the optimizer w ∗ along the low curvature direction which accelerates convergence.
Revised: This allows the method to move further horizontally early on to enter the valley with a smaller distance to the optimizer w ∗ along the low curvature direction which accelerates convergence.

Original: Theorem 1 (Preconditioned gradient methods as TR) .
Revised: Theorem 1 (Preconditioned gradient methods as TR) .

Original: A preconditioned gradient step
Revised: A preconditioned gradient step

Original: [Equation]
Revised: [Equation]

Original: with stepsize η t > 0 , symmetric positive deﬁnite preconditioner A t ∈ R d × d and g t (cid:54) =minimizes a ﬁrst-order model around w t ∈ R d in an ellipsoid given by A t in the sense that
Revised: with stepsize η t > 0 , symmetric positive deﬁnite preconditioner A t ∈ R d × d and g t (cid:54) =minimizes a ﬁrst-order model around w t ∈ R d in an ellipsoid given by A t in the sense that

Original: [Equation]
Revised: [Equation]

Original: Corollary 1 (Rmsprop) .
Revised: Corollary 1 (Rmsprop) .

Original: The step s rms,t := − η t
Revised: The step s rms,t := − η t

Original: A − 1 / 2 rms,t g t minimizes a ﬁrst-order Taylor model around w t in an ellipsoid given by A 1 / 2 rms,t
Revised: A − 1 / 2 rms,t g t minimizes a ﬁrst-order Taylor model around w t in an ellipsoid given by A 1 / 2 rms,t

Original: (Eq. 3) in the sense that
Revised: (Eq. 3) in the sense that

Original: [Equation]
Revised: [Equation]

Original: Equivalent results can be established for Adam using g adam,t :=
Revised: Equivalent results can be established for Adam using g adam,t :=

Original: (1 − β ) (cid:80) tk =

Revised: (1 − β ) (cid:80) tk =

Original: β t − k g t as well as for
Revised: β t − k g t as well as for

Original: Adagrad by replacing the matrix A ada into the constraint in Eq.
Revised: Adagrad by replacing the matrix A ada into the constraint in Eq.

Original: Of course, the update procedure in Eq.
Revised: Of course, the update procedure in Eq.

Original: (5) is merely a reinterpretation of the original preconditioned update, and thus the employed trust region radii are deﬁned implicitly by the current gradient and stepsize.
Revised: (5) is merely a reinterpretation of the original preconditioned update, and thus the employed trust region radii are deﬁned implicitly by the current gradient and stepsize.

Original: 3. D IAGONAL VERSUS FULL PRECONDITIONING
Revised: 3. D IAGONAL VERSUS FULL PRECONDITIONING

Original: A closer look at Fig.
Revised: A closer look at Fig.

Original: 1 reveals that the ﬁrst two problems come with level sets that are perfectly axis-aligned , which makes these objectives particularly attractive for diagonal preconditioning.
Revised: 1 reveals that the ﬁrst two problems come with level sets that are perfectly axis-aligned , which makes these objectives particularly attractive for diagonal preconditioning.

Original: For comparison, on the right of Fig. 1, we report another quadratic problem instance, where the Hessian is no longer zero on the off-diagonals.
Revised: For comparison, on the right of Fig. 1, we report another quadratic problem instance, where the Hessian is no longer zero on the off-diagonals.

Original: As can be seen, the interaction between coordinates introduces a tilt in the level sets and reduces the superiority of diagonal Adagrad over plain GD.
Revised: As can be seen, the interaction between coordinates introduces a tilt in the level sets and reduces the superiority of diagonal Adagrad over plain GD.

Original: However, using the full preconditioner A ada re-establishes the original speed up.
Revised: However, using the full preconditioner A ada re-establishes the original speed up.

Original: Yet, non-diagonal preconditioning comes at the cost of taking the inverse square root of a large matrix, which is why this approach has been relatively unexplored (see Agarwal et al.
Revised: Yet, non-diagonal preconditioning comes at the cost of taking the inverse square root of a large matrix, which is why this approach has been relatively unexplored (see Agarwal et al.

Original: (2018) for a recent exception).
Revised: (2018) for a recent exception).

Original: Interestingly, early results by Becker et al.
Revised: Interestingly, early results by Becker et al.

Original: (1988) on the curvature structure of neural nets report a strong diagonal dominance of the Hessian matrix ∇ 2 L ( w ) .
Revised: (1988) on the curvature structure of neural nets report a strong diagonal dominance of the Hessian matrix ∇ 2 L ( w ) .

Original: This suggests that the loss surface is indeed somewhat axis-aligned.
Revised: This suggests that the loss surface is indeed somewhat axis-aligned.

Original: However, the reported numbers are only for tiny feed-forward networks of at most 256 parameters.
Revised: However, the reported numbers are only for tiny feed-forward networks of at most 256 parameters.

Original: Therefore, we generalize these ﬁndings in the following to larger networks.
Revised: Therefore, we generalize these ﬁndings in the following to larger networks.

Original: Furthermore, we contrast the diagonal dominance of real Hessian matrices to the expected behavior of random Wigner matrices.
Revised: Furthermore, we contrast the diagonal dominance of real Hessian matrices to the expected behavior of random Wigner matrices.

Original: Of course, true Hessians do not have i.i.d.
Revised: Of course, true Hessians do not have i.i.d.

Original: entries but the symmetry of Wigner matrices suggests that this baseline is not completely off.
Revised: entries but the symmetry of Wigner matrices suggests that this baseline is not completely off.

Revised: Furthermore, we also compare Hessians of Ordinary Least Squares problems (OLS) with random inputs.

Original: For this purpose, let δ A deﬁne the ratio of diagonal to overall mass of a matrix A , i.e. δ A :
Revised: For this purpose, let δ A deﬁne the ratio of diagonal to overall mass of a matrix A , i.e. δ A :=

Original: = (cid:80) i |
Revised: [Equation] i |

Original: A i,i | (cid:80) i (cid:80) j | A 
Revised: [Equation] j | A i [Equation]

Original: i,j | as in (Becker et al., 1988).
Revised: j | as in (Becker et al., 1988).

Original: Proposition 1.
Revised: Proposition 1 (Diagonal share of Wigner matrix) .

Original: For random Gaussian 3 Wigner matrix W formed as
Revised: For random Gaussian 3 Wigner matrix W formed as

Original: [Equation]
Revised: [Equation]

Original: where ∼ stands for i.i.d.
Revised: where ∼ stands for i.i.d.

Original: draws (Wigner, 1993), theexpected share of diagonal mass δ W amounts to
Revised: draws (Wigner, 1993), the diagonal mass of the expected absolute matrix amounts to

Original: [Equation]
Revised: [Equation]

Original: Thus, if we suppose the Hessian at any given point w were a random Wigner matrix we would expect the share of diagonal mass to fall with O (1 /d ) as the network grows in size.
Revised: Thus, if we suppose the Hessian at any given point w were a random Wigner matrix we would expect the share of diagonal mass to fall with O (1 /d ) as the network grows in size.

Original: Yet, as can be seen in 
Revised: A similar result can be derived for the large n limit in the case of OLS Hessians. Proposition 2 (Diagonal share of OLS Hessian) . Let X ∈ R d × n and assume each x i,j is generated i.i.d. with zero-mean ﬁnite second moment σ 2 > 0 .

Original: Fig.
Revised: Then the share of diagonal mass of the expected matrix E [ | H ols | ] amounts to [Equation] Again, we expect the diagonal mass to fall in d and interestingly, empirical simulations suggest that the result of Proposition 2 holds already in small n settings (see Fig.D.2) and it is likely that ﬁnite n results can be derived when adding assumptions such as Gaussian data.

Original: 2 the diagonal mass δ H ofreal-world neural networks staysway above this theoretical value at random initialization, during training and after convergence.
Revised: As can be seen in Fig.below, even for a practical batch size of n = 32 the diagonal mass δ H of neural networks stays above both benchmarks when its input is random as well as with real-world data at random initialization, during training and after convergence.

Original: [Figure]
Revised: [Figure]

Original: These ﬁndings are in line with Becker et al.
Revised: These ﬁndings are in line with Becker et al.

Original: and suggest that full matrix preconditioning is most probably not worth the effort for neural networks.
Revised: and suggest that full matrix preconditioning is most probably not worth the additional computational cost for neural networks.

Original: Consequently, we use diagonal preconditioning for both ﬁrst- and second-order methods in all of our experiments in Section 5.
Revised: Consequently, we use diagonal preconditioning for both ﬁrst- and second-order methods in all of our experiments in Section 5.

Revised: Further theoretical elaboration of these ﬁndings present an interesting direction of future research.

Original:  S ECOND - ORDER T RUST R EGION M ETHODS
Revised:  S ECOND - ORDER T RUST R EGION M ETHODS

Original: Cubic regularization (Nesterov & Polyak, 2006; Cartis et al., 2011) and trust region methods belong to the family of globalized Newton methods.
Revised: Cubic regularization (Nesterov & Polyak, 2006; Cartis et al., 2011) and trust region methods belong to the family of globalized Newton methods.

Original: Both frameworks compute parameter updates by optimizing regularized (former) or constrained (latter) second-order Taylor models of the objective L around the current iterate w t .
Revised: Both frameworks compute parameter updates by optimizing regularized (former) or constrained (latter) second-order Taylor models of the objective L around the current iterate w t .

Original: In particular, in iteration t the update step of the trust region algorithm is computed as
Revised: In particular, in iteration t the update step of the trust region algorithm is computed as

Original: [Equation]
Revised: [Equation]

Original: where ∆ t > 0 and g t and B t are either ∇L ( w t ) and ∇ 2 L ( w t ) or suitable approximations.
Revised: where ∆ t > 0 and g t and B t are either ∇L ( w t ) and ∇ 2 L ( w t ) or suitable approximations.

Original: The matrix A t induces the shape of the constraint set.
Revised: The matrix A t induces the shape of the constraint set.

Original: So far, the common choice for neural networks isA t :=
Revised: So far, the common choice for neural networks is

Original:  I , ∀ t which gives rise to spherical trust regions (Xu et al., 2017a; Liu et al., 2018).
Revised: A t := I , ∀ t which gives rise to spherical trust regions (Xu et al., 2017a; Liu et al., 2018).

Original: By solving the constrained problem (8), TR methods overcome the problem that pure Newton steps may be ascending, attracted by saddles or not even computable.
Revised: By solving the constrained problem (10), TR methods overcome the problem that pure Newton steps may be ascending, attracted by saddles or not even computable.

Original: See Appendix B for more details.
Revised: See Appendix B for more details.

Original: 4. C ONVERGENCE OF ELLIPSOIDAL T RUST R EGION METHODS
Revised: 4. C ONVERGENCE OF ELLIPSOIDAL T RUST R EGION METHODS

Original: Inspired by the success of adaptive gradient methods, we investigate the use of their preconditioning matrices as norm inducing matrices for second-order TR methods.
Revised: Inspired by the success of adaptive gradient methods, we investigate the use of their preconditioning matrices as norm inducing matrices for second-order TR methods.

Original: The crucial condition for convergence is that the applied norms are not degenerate during the entire minimization process in the sense that the ellipsoids do not ﬂatten out (or blow up) completely along any given direction.
Revised: The crucial condition for convergence is that the applied norms are not degenerate during the entire minimization process in the sense that the ellipsoids do not ﬂatten out (or blow up) completely along any given direction.

Original: The following deﬁnition formalizes this intuition.
Revised: The following deﬁnition formalizes this intuition.

Original: Deﬁnition 1 (Uniformly equivalent norms) .
Revised: Deﬁnition 1 (Uniformly equivalent norms) .

Original: The norms (cid:107) w (cid:107)
Revised: The norms (cid:107) w (cid:107)

Original: At :=
Revised: A

Original: ( w (cid:124)
Revised: [Equation]

Original: A t w ) 1 / 2 induced by symmetric positive deﬁnite matrices
Revised: 2 induced by symmetric positive deﬁnite matrices

Original: A t are called uniformly equivalent, if ∃ µ ≥ 1 such that
Revised: A t are called uniformly equivalent, if ∃ µ ≥ 1 such that

Original: [Equation]
Revised: [Equation]

Original: We now establish a result which shows that the RMSProp ellipsoid is indeed uniformly equivalent.
Revised: We now establish a result which shows that the RMSProp ellipsoid is indeed uniformly equivalent.

Original: Proposition 2 (Uniform equivalence) .
Revised: Proposition 3 (Uniform equivalence) .

Original: Suppose (cid:107) g t (cid:107) 2 ≤ L 2 H for all w t ∈ R d , t = 1 , 2 , . . .
Revised: Suppose (cid:107) g t (cid:107) 2 ≤ L 2 H for all w t ∈ R d , t = 1 , 2 , . . .

Original: Then there always exists (cid:15) > 

Revised: Then there always exists (cid:15) >

Original: such that the proposed preconditioning matrices
Revised: such that the proposed preconditioning matrices

Original: A rms,t (Eq. 3) are uniformly equivalent, i.e. Def. 1 holds.
Revised: A rms,t (Eq. 3) are uniformly equivalent, i.e. Def. 1 holds.

Original: The same holds for the diagonal variant.
Revised: The same holds for the diagonal variant.

Original: Consequently, the ellipsoids A rms,t can directly be applied to any convergent TR framework without losing convergence guarantees (Conn et al.
Revised: Consequently, the ellipsoids A rms,t can directly be applied to any convergent TR framework without losing convergence guarantees (Conn et al.

Original: (2000), Theorem 6.6.8).
Revised: (2000), Theorem 6.6.8).

Original: Interestingly, this result cannot be established for A ada,t , which reﬂects the widely known vanishing stepsize problem that arises since squared gradients are continuously added to the preconditioning matrix.
Revised: Interestingly, this result cannot be established for A ada,t , which reﬂects the widely known vanishing stepsize problem that arises since squared gradients are continuously added to the preconditioning matrix.

Original: At least partially, this effect inspired the development of RMSprop (Tieleman & Hinton, 2012) and Adadelta (Zeiler, 2012).
Revised: At least partially, this effect inspired the development of RMSprop (Tieleman & Hinton, 2012) and Adadelta (Zeiler, 2012).

Original: Why ellipsoids?
Revised: Why ellipsoids?

Original: There are many sources for ill-conditioning in neural networks such as un-centered and correlated inputs (LeCun et al., 2012), saturated hidden units, and different weight scales in different layers (Van Der Smagt & Hirzinger, 1998).
Revised: There are many sources for ill-conditioning in neural networks such as un-centered and correlated inputs (LeCun et al., 2012), saturated hidden units, and different weight scales in different layers (Van Der Smagt & Hirzinger, 1998).

Original: While the quadratic term of model (8) accounts for such ill-conditioning to some extent, the spherical constraint is completely blind towards the loss surface.
Revised: While the quadratic term of model (10) accounts for such ill-conditioning to some extent, the spherical constraint is completely blind towards the loss surface.

Original: Thus, it is advisable to instead measure distances in norms that reﬂect the underlying geometry (see Chapter 7.7 in Conn et al.
Revised: Thus, it is advisable to instead measure distances in norms that reﬂect the underlying geometry (see Chapter 7.7 in Conn et al.

Original: The ellipsoids we propose are such that they allow for longer steps along coordinates that have seen small gradient components in past and vice versa.
Revised: The ellipsoids we propose are such that they allow for longer steps along coordinates that have seen small gradient components in past and vice versa.

Original: Thereby the TR shape is adaptively adjusted to ﬁt the current region of the non-convex loss landscape.
Revised: Thereby the TR shape is adaptively adjusted to ﬁt the current region of the non-convex loss landscape.

Original: This procedure is not only effective when the iterates are in an ill-conditioned neighborhood of a minimizer (Figure 1), but it also helps to escape elongated plateaus (see autoencoder in Section 5).
Revised: This procedure is not only effective when the iterates are in an ill-conditioned neighborhood of a minimizer (Figure 1), but it also helps to escape elongated plateaus (see autoencoder in Section 5).

Original: 4. A STOCHASTIC TR FRAMEWORK FOR NEURAL NETWORK TRAINING
Revised: 4. A STOCHASTIC TR FRAMEWORK FOR NEURAL NETWORK TRAINING

Original: Since neural network training often constitutes a large scale learning problem in which the number of datapoints n is very high, we here opt for a fully stochastic TR framework (Chen et al., 2018) in order to circumvent memory issues and reduce computational complexity.
Revised: Since neural network training often constitutes a large scale learning problem in which the number of datapoints n is very high, we here opt for a fully stochastic TR framework (Chen et al., 2018) in order to circumvent memory issues and reduce computational complexity.

Original: Given that the involved function and derivative estimates are sufﬁciently accurate with a ﬁxed probability, such a framework retains the O (cid:0) (cid:15) − 2 g , (cid:15) − 3 H (cid:1) convergence rate of deterministic TR methods to stationary points in expectation (Blanchet et al., 2016; Gratton et al., 2017).
Revised: Given that the involved function and derivative estimates are sufﬁciently accurate with a ﬁxed probability, such a framework retains the O (cid:0) (cid:15) − 2 g , (cid:15) − 3 H (cid:1) convergence rate of deterministic TR methods to stationary points in expectation (Blanchet et al., 2016; Gratton et al., 2017).

Original: For ﬁnite-sum objectives such as Eq.
Revised: For ﬁnite-sum objectives such as Eq.

Original: (1), the required level of accuracy can be obtained by simple mini-batching.
Revised: (1), the required level of accuracy can be obtained by simple mini-batching.

Original: In that case, the same rate can be proven for Algorithm 1 with A rms ellipsoids as Proposition 2 guarantees the sufﬁcient condition for convergence with adaptive norms (see Appendix C below as well as AN.1 in Conn et al. 
Revised: In that case, the same rate can be proven for Algorithm 1 with A rms ellipsoids as Proposition 3 guarantees the sufﬁcient condition for convergence with adaptive norms (see Appendix C below as well as AN.1 in Conn et al. Algorithm

Original: 1: Input: w 0 ∈ R d , γ 1 , γ 2 > 1 , 1 > η 2 > η 1 > 0 , ∆ 0 > 0 , T ≥ 1 , |S 0 | , µ ≥ 1 , (cid:15) >2: for t = 0 , 1 , . .
Revised: 1: Input: w 0 ∈ R d , γ 1 , γ 2 > 1 , 1 > η 2 > η 1 > 0 , ∆ 0 > 0 , T ≥ 1 , |S 0 | , µ ≥ 1 , (cid:15) >2: for t = 0 , 1 , . .

Original: , until convergence do 3:
Revised: , until convergence do 3:

Original: Sample L t , g t and B t with batch sizes |S L ,t | , |S g ,t | , |S B ,t | 4:
Revised: Sample L t , g t and B t with batch sizes |S L ,t | , |S g ,t | , |S B ,t | 4:

Original: Compute preconditioner A t s.t.
Revised: Compute preconditioner A t s.t.

Original: Def. 1 holds 5:
Revised: Def. 1 holds 5:

Original: Obtain s t by solving m t ( s t ) (Eq. 
Revised: Obtain s t by solving m t ( s t ) (Eq. 10)

Original: 8) s.t.
Revised: s.t.

Original: Eq 36 holds 6:
Revised: Eq 41 holds 6:

Original: Compute actual over predicted decrease on batch
Revised: Compute actual over predicted decrease on batch

Original: [Equation]
Revised: [Equation]

Original: Set
Revised: Set

Original: [Equation]

Original: very successful

Original: [Equation]

Original: successful

Original: [Equation]
Revised: [Equation]

Original: unsuccessful

Original: [Equation] w [Equation] otherwise 8: end for
Revised: t + = (cid:26) w t + s t if ρ S ,t ≥ ηw t otherwise 8: end for

Original: The main difference between this and the existing approaches of Kohler & Lucchi (2017); Xu et al. (2017b); Yao et al.
Revised: The main difference between this and the existing approaches of Kohler & Lucchi (2017); Xu et al. (2017b); Yao et al.

Original: ; Cartis & Scheinberg (2017) lies in the computation of ρ , which is traditionally computed as full function- over stochastic model decrease.
Revised: ; Cartis & Scheinberg (2017) lies in the computation of ρ , which is traditionally computed as full function- over stochastic model decrease.

Original: Computing ρ solely based on sub-sampled quantities has the nice side-effect of disentangling two potential sources of error: an overoptimistic trust region radius or insufﬁciently small batch sizes.
Revised: Computing ρ solely based on sub-sampled quantities has the nice side-effect of disentangling two potential sources of error: an overoptimistic trust region radius or insufﬁciently small batch sizes.

Original: Indeed, the quantity ρ S from
Revised: Indeed, the quantity ρ S from

Original: Eq.
Revised: Eq.

Original: (10) is an unbiased estimate of the ρ used in fully deterministic algorithms and hence the trust region radius is adjusted purely based on the current adequacy of the local quadratic approximation.
Revised: (12) is an unbiased estimate of the ρ used in fully deterministic algorithms and hence the trust region radius is adjusted purely based on the current adequacy of the local quadratic approximation.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Trust region methods To validate our claim that ellipsoidal TR methods yield improved performance over spherical ones, we run a set of experiments on two image datasets and three types of network architectures.
Revised: Trust region methods To validate our claim that ellipsoidal TR methods yield improved performance over spherical ones, we run a set of experiments on two image datasets and three types of network architectures.

Original: As can be seen in Figure 3, the ellipsoidal TR methods consistently outperform their spherical counterpart in the sense that they reach full training accuracy substantially faster on all problems.
Revised: As can be seen in Figure 3, the ellipsoidal TR methods consistently outperform their spherical counterpart in the sense that they reach full training accuracy substantially faster on all problems.

Original: Moreover, their limit points are in all cases lower than those of the uniform TR method.
Revised: Moreover, their limit points are in all cases lower than those of the uniform TR method.

Original: Interestingly, this makes an actual difference in the image reconstruction quality of autoencoders (see Figure 11).
Revised: Interestingly, this makes an actual difference in the image reconstruction quality of autoencoders (see Figure 12).

Original: We thus draw the clear conclusion that the ellipsoidal trust region constraints we propose are to be preferred over their spherical counterpart when training neural networks.
Revised: We thus draw the clear conclusion that the ellipsoidal trust region constraints we propose are to be preferred over their spherical counterpart when training neural networks.

Original: Both the experimental and architectural details are provided in Appendix C.
Revised: Both the experimental and architectural details are provided in Appendix C.

Original:  Benchmark with SGD To put the previous results into context, we also benchmark several stateof-the-art gradient methods.
Revised: [Figure] [Table] [Figure] Benchmark with SGD To put the previous results into context, we also benchmark several stateof-the-art gradient methods.

Original: We ﬁx their sample size to 32 (as advocated e.g. in Masters & Luschi (2018)) but grid search the stepsize since it is the ratio of these two quantities that effectively determines the level of stochasticity (Jastrzebski et al., 2017).
Revised: We ﬁx their sample size to 32 (as advocated e.g. in Masters & Luschi (2018)) but grid search the stepsize since it is the ratio of these two quantities that effectively determines the level of stochasticity (Jastrzebski et al., 2017).

Original: As the TR methods have a larger batch size 7 of 128–512, we report results both in terms of number of backpropagations and epochs for a fair comparison.
Revised: As the TR methods have a larger batch size 7 of 128–512, we report results both in terms of number of backpropagations and epochs for a fair comparison.

Original: A close look at Figure 4 and 9 (Appendix) indicates that the ellipsoidal TR methods can be slightly superior in terms of backprops but at best manage to keep pace with ﬁrst-order methods [Figure] [Figure]
Revised: A close look at Figure 4 and 10 (Appendix) indicates that the ellipsoidal TR methods can be slightly superior in terms of backprops but at best manage to keep pace with ﬁrst-order methods in terms of epochs.

Original: as Figure 3.

Original: See Figure 9 for epoch results.

Original: in terms of epochs.

Original: Furthermore, the limit points of both ﬁrst- and second-order methods yield the same order of loss in most experiments.
Revised: Furthermore, the limit points of both ﬁrst- and second-order methods yield the same order of loss in most experiments.

Original: When taking gradient norms into account (plot omitted), we indeed
Revised: When taking gradient norms into account (plot omitted), we indeed

Original: ﬁnd no spurious local minima and only the autoencoders give rise to saddle points.
Revised: ﬁnd no spurious local minima and only the autoencoders give rise to saddle points.

Revised: [Figure]

Revised: See Figure 10 for epoch results.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We investigated the use of ellipsoidal trust region constraints for neural networks.
Revised: We investigated the use of ellipsoidal trust region constraints for neural networks.

Original: We have shown that the RMSProp matrix satisﬁes the necessary conditions for convergence and our experimental results demonstrate that ellipsoidal TR methods outperform their spherical counterparts signiﬁcantly.
Revised: We have shown that the RMSProp matrix satisﬁes the necessary conditions for convergence and our experimental results demonstrate that ellipsoidal TR methods outperform their spherical counterparts signiﬁcantly.

Original: We thus consider the development of further ellipsoids that can potentially adapt even better to the loss landscape such as e.g. (block-) diagonal hessian approximations (e.g. Bekas et al.
Revised: We thus consider the development of further ellipsoids that can potentially adapt even better to the loss landscape such as e.g. (block-) diagonal hessian approximations (e g. Bekas et al.

Original: or approximations of higher order derivatives as an interesting direction of future research.
Revised: or approximations of higher order derivatives as an interesting direction of future research.

Original: Yet, the gradient method benchmark indicates that the value of Hessian information for neural network training is limited for mainly three reasons: 1) second-order methods rarely yield better limit points, which suggests that saddles and spurious local minima are not a major obstacle; 2) gradient methods can run on smaller batch sizes which is beneﬁcial in terms of epoch and when memory is limited; 3) The per-iteration time complexity is noticeably lower for ﬁrst-order methods (Figure 10).
Revised: Yet, the gradient method benchmark indicates that the value of Hessian information for neural network training is limited for mainly three reasons: 1) second-order methods rarely yield better limit points, which suggests that saddles and spurious local minima are not a major obstacle; 2) gradient methods can run on smaller batch sizes which is beneﬁcial in terms of epoch and when memory is limited; 3) The per-iteration time complexity is noticeably lower for ﬁrst-order methods (Figure 11).

Original: These observations suggest that advances in hardware and distributed second-order algorithms (e g.,
Revised: These observations suggest that advances in hardware and distributed second-order algorithms (e g.,

Original: Osawa et al.
Revised: Osawa et al.

Original: ; Dünner et al.
Revised: ; Dünner et al.

Original: (2018)) will be needed before Newton-type methods can replace (stochastic) gradient methods in deep learning.
Revised: (2018)) will be needed before Newton-type methods can replace (stochastic) gradient methods in deep learning.

Original: Appendix A: Proofs A N OTATION
Revised: Appendix A: Proofs A N OTATION

Original: Throughout this work, scalars are denoted by regular lower case letters, vectors by bold lower case letters and matrices as well as tensors by bold upper case letters.
Revised: Throughout this work, scalars are denoted by regular lower case letters, vectors by bold lower case letters and matrices as well as tensors by bold upper case letters.

Original: By (cid:107) · (cid:107) we denote an arbitrary norm.
Revised: By (cid:107) · (cid:107) we denote an arbitrary norm.

Original: For a symmetric positive deﬁnite matrix A we introduce the compact notation (cid:107) w (cid:107)
Revised: For a symmetric positive deﬁnite matrix A we introduce the compact notation (cid:107) w (cid:107)

Original: A = ( w (cid:124)
Revised: A = ( w (cid:124)

Original: Aw ) 1 / 2 , where w ∈ R d .
Revised: Aw ) 1 / 2 , where w ∈ R d .

Original: B E QUIVALENCE OF P RECONDITIONED G RADIENT D ESCENT AND FIRST - ORDER T RUST R EGION M ETHODS
Revised: B E QUIVALENCE OF P RECONDITIONED G RADIENT D ESCENT AND FIRST - ORDER T RUST R EGION M ETHODS

Original: Theorem 2 (Theorem 1 restated) .
Revised: Theorem 2 (Theorem 1 restated) .

Original: A preconditioned gradient step
Revised: A preconditioned gradient step

Original: [Equation]
Revised: [Equation]

Original:  with stepsize η t > 0 , symmetric positive deﬁnite preconditioner A t ∈ R d × d and g t (cid:54) =minimizes a ﬁrst-order local model around w t ∈ R in an ellipsoid given by A t in the sense that
Revised: (13) with stepsize η t > 0 , symmetric positive deﬁnite preconditioner A t ∈ R d × d and g t (cid:54) =minimizes a ﬁrst-order local model around w t ∈ R in an ellipsoid given by A t in the sense that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We start the proof by noting that the optimization problem in Eq.
Revised: We start the proof by noting that the optimization problem in Eq.

Original: (12) is convex.
Revised: (14) is convex.

Original: For η t >the constraint satisﬁes the Slater condition since 0 is a strictly feasible point.
Revised: For η t >the constraint satisﬁes the Slater condition since 0 is a strictly feasible point.

Original: As a result, any KKT point is a feasible minimizer and vice versa.
Revised: As a result, any KKT point is a feasible minimizer and vice versa.

Original: Let L ( s , λ ) denote the Lagrange dual of Eq.
Revised: Let L ( s , λ ) denote the Lagrange dual of Eq.

Original: [Equation]
Revised: [Equation]

Original: Any point s is a KKT point if and only if the following system of equations is satisﬁed
Revised: Any point s is a KKT point if and only if the following system of equations is satisﬁed

Original: [Equation]
Revised: [Equation]

Original: For s t as given in Eq.
Revised: For s t as given in Eq.

Original: (4) we have that
Revised: (4) we have that

Original: [Equation]
Revised: [Equation]

Original: and thus 15 and 16 hold with equality such that any λ ≥ 0 is feasible.
Revised: and thus 17 and 18 hold with equality such that any λ ≥ 0 is feasible.

Original: Furthermore,
Revised: Furthermore,

Original: [Equation]
Revised: [Equation]

Original: is zero for λ
Revised: is zero for λ

Original: = (cid:107) g t (cid:107)
Revised: = (cid:107) g t (cid:107)

Original: A − 1 ≥ 0 .
Revised: A − 1 ≥ 0 .

Original: As a result, s t is a KKT point of the convex problem 5 which proves the assertion.
Revised: As a result, s t is a KKT point of the convex problem 5 which proves the assertion.

Original: To illustrate this theoretical result we run gradient descent and Adagrad as well as the two corresponding ﬁrst-order TR methods 8 on an ill-conditioned quadratic problem.
Revised: To illustrate this theoretical result we run gradient descent and Adagrad as well as the two corresponding ﬁrst-order TR methods 8 on an ill-conditioned quadratic problem.

Original: While the method 1st
Revised: While the method 1st

Original: TR
Revised: TR

Original: Eq.
Revised: Eq.

Original: (12). optimizes a linear model within a ball in each iteration, 1st TR ada optimizes the same model over the ellipsoid given by the Adagrad matrix A ada .
Revised: optimizes a linear model within a ball in each iteration, 1st TR ada optimizes the same model over the ellipsoid given by the Adagrad matrix A ada .

Original: The results in Figure 5 show that the methods behave very similar to their constant stepsize analogues.
Revised: The results in Figure 5 show that the methods behave very similar to their constant stepsize analogues.

Original: [Figure]
Revised: [Figure]

Original: C C ONVERGENCE OF ELLIPSOIDAL TR METHODS
Revised: C C ONVERGENCE OF ELLIPSOIDAL TR METHODS

Original: Spherical constrained TR methods Under standard smoothness assumptions, spherical TR algorithms achieve (cid:15) g criticality after O ( (cid:15) − 2 g ) iterations and additionally (cid:15) H almost positive curvature in O ( (cid:15) − 3 H ) iterations.
Revised: Spherical constrained TR methods Under standard smoothness assumptions, spherical TR algorithms achieve (cid:15) g criticality after O ( (cid:15) − 2 g ) iterations and additionally (cid:15) H almost positive curvature in O ( (cid:15) − 3 H ) iterations.

Original: These rates are attained without the need of actually knowing the Hessian’s
Revised: These rates are attained without the need of actually knowing the Hessian’s

Original: Lipschitz constant.
Revised: Lipschitz constant.

Original: The complete statement of the convergence results for TR methods can be found in
Revised: The complete statement of the convergence results for TR methods can be found in

Original: Theorem 4.3 of (Cartis et al., 2012a).
Revised: Theorem 4.3 of (Cartis et al., 2012a).

Original: Interestingly, these rates can be improved to match the (optimal)
Revised: Interestingly, these rates can be improved to match the (optimal)

Original: O ( (cid:15) − 3 / 2 ) ﬁrst-order worst case complexity of Cubic Regularization by applying small modiﬁcations to the TR framework.
Revised: O ( (cid:15) − 3 / 2 ) ﬁrst-order worst case complexity of Cubic Regularization by applying small modiﬁcations to the TR framework.

Original: As stated in Section B.2 the involved subproblems do not need to be solved globally in each iteration.
Revised: As stated in Section B.2 the involved subproblems do not need to be solved globally in each iteration.

Original: For both, cubic regularization and trust region methods, many stochastic extensions have emerged in literature that alleviate the need to compute exact derivative information without losing the above mentioned convergence guarantees with high probability (Kohler & Lucchi, 2017; Xu et al., 2017a;b; Blanchet et al., 2016; Gratton et al., 2017)).
Revised: For both, cubic regularization and trust region methods, many stochastic extensions have emerged in literature that alleviate the need to compute exact derivative information without losing the above mentioned convergence guarantees with high probability (Kohler & Lucchi, 2017; Xu et al., 2017a;b; Blanchet et al., 2016; Gratton et al., 2017)).

Original: For the deep learning setting, the analysis of Blanchet et al.
Revised: For the deep learning setting, the analysis of Blanchet et al.

Original: (2016) is most relevant since it also allows the algorithm to run solely based on sub-sampled function evaluations.
Revised: (2016) is most relevant since it also allows the algorithm to run solely based on sub-sampled function evaluations.

Original: Ellipsoidal constrained TR methods In order to prove such results for ellipsoidal Trust Region methods one must ensure that the applied norms are coherent during the complete minimization process in the sense that the ellipsoids do not ﬂatten out (or blow up) completely along any given direction.
Revised: Ellipsoidal constrained TR methods In order to prove such results for ellipsoidal Trust Region methods one must ensure that the applied norms are coherent during the complete minimization process in the sense that the ellipsoids do not ﬂatten out (or blow up) completely along any given direction.

Original: This intuition is formalized in Assumption 1 which we restate here for the sake of clarity.
Revised: This intuition is formalized in Assumption 1 which we restate here for the sake of clarity.

Original: Deﬁnition 2 (Deﬁnition 1 restated) .
Revised: Deﬁnition 2 (Deﬁnition 1 restated) .

Original: There exists a constant µ ≥ 1 such that
Revised: There exists a constant µ ≥ 1 such that

Original: [Equation]
Revised: [Equation]

Original: Having uniformly equivalent norms is necessary and sufﬁcient to prove that ellipsoidal TR methods enjoy the same convergence rate as classical ball constrained Trust Region algorithms.
Revised: Having uniformly equivalent norms is necessary and sufﬁcient to prove that ellipsoidal TR methods enjoy the same convergence rate as classical ball constrained Trust Region algorithms.

Original: Towards this end, Conn et al.
Revised: Towards this end, Conn et al.

Original: (2000) identify the following sufﬁcient condition on the basis of which we will prove that our proposed ellipsoid A rms is indeed uniformly equivalent under some mild assumptions.
Revised: (2000) identify the following sufﬁcient condition on the basis of which we will prove that our proposed ellipsoid A rms is indeed uniformly equivalent under some mild assumptions.

Original: Lemma 1 (Theorem 6.7.1 in Conn et al.
Revised: Lemma 1 (Theorem 6.7.1 in Conn et al.

Original: Suppose that there exists a constant ζ ≥ 1 such that
Revised: Suppose that there exists a constant ζ ≥ 1 such that

Original: [Equation]
Revised: [Equation]

Original: then Deﬁnition 1 holds.
Revised: then Deﬁnition 1 holds.

Original: Proposition 3 (Uniform equivalence) .
Revised: Proposition 4 (Uniform equivalence) .

Original: Suppose (cid:107) g t (cid:107) 2 ≤ L 2 H for all w t ∈ R d , t = 1 , 2 , . . .
Revised: Suppose (cid:107) g t (cid:107) 2 ≤ L 2 H for all w t ∈ R d , t = 1 , 2 , . . .

Original: Then there always exists (cid:15) > 

Revised: Then there always exists (cid:15) >

Original: such that the proposed preconditioning matrices
Revised: such that the proposed preconditioning matrices

Original: A rms,t (Eq. 3) are uniformly equivalent, i.e. Def. 1 holds.
Revised: A rms,t (Eq. 3) are uniformly equivalent, i.e. Def. 1 holds.

Original: The same holds for the diagonal variant.
Revised: The same holds for the diagonal variant.

Original: Proof.
Revised: Proof.

Original: The basic building block of our ellipsoid matrix consists of the current and past stochastic gradients G t :=
Revised: The basic building block of our ellipsoid matrix consists of the current and past stochastic gradients G t :=

Original: [ g 1 , g 2 , . . . , g t ] .
Revised: [ g 1 , g 2 , . . . , g t ] .

Original: We consider A rms which is built up as follows
Revised: We consider A rms which is built up as follows

Original: [Equation]
Revised: [Equation]

Original: From the construction of A rms,t it directly follows that for any unit length vector u ∈ R d \ { 0 } , (cid:107) u (cid:107) 

Revised: From the construction of A rms,t it directly follows that for any unit length vector u ∈ R d \ { 0 } , (cid:107) u (cid:107)

Original: = 1 we have
Revised: = 1 we have

Original: [Equation]
Revised: [Equation]

Original: 23) which proves the lower bound for ζ = 1 /(cid:15).
Revised: [Equation] which proves the lower bound for ζ = 1 /(cid:15)

Original: Now, let us consider the upper end of the spectrum of
Revised: Now, let us consider the upper end of the spectrum of

Original: A rms,t .
Revised: A rms,t .

Original: Towards this end, recall the geometric series expansion
Revised: Towards this end, recall the geometric series expansion

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and the fact that GG (cid:62) is a sum of exponentially weighted rank-one positive semi-deﬁnite matrices of the form
Revised: and the fact that GG (cid:62) is a sum of exponentially weighted rank-one positive semi-deﬁnite matrices of the form

Original: g i g (cid:124) i .
Revised: g i g (cid:124) i .

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: where the latter inequality holds per assumption for any sample size | S | .
Revised: where the latter inequality holds per assumption for any sample size | S | .

Original: Combining these facts we get that
Revised: Combining these facts we get that

Original: [Equation]
Revised: [Equation]

Original: As a result we have that
Revised: As a result we have that

Original: [Equation]
Revised: [Equation]

Original: Finally, to achieve uniform equivalence we need the r.h.s.
Revised: Finally, to achieve uniform equivalence we need the r.h.s.

Original: of (26) to be bounded by 1 /(cid:15) .
Revised: of (28) to be bounded by 1 /(cid:15) .

Original: This gives rise to a quadratic equation in (cid:15) , namely
Revised: This gives rise to a quadratic equation in (cid:15) , namely

Original: [Equation]
Revised: [Equation]

Original: 27) which holds for any t and any β ∈ (0 , 1) as long as
Revised: which holds for any t and any β ∈ (0 , 1) as long as

Original: [Equation]
Revised: [Equation]

Original: Such an (cid:15) always exists but one needs to choose smaller and smaller values as the upper bound on the gradient norm grows.
Revised: Such an (cid:15) always exists but one needs to choose smaller and smaller values as the upper bound on the gradient norm grows.

Original: For example, the usual value (cid:15) = 10 − 8 is valid for all L 2 H < 9 .
Revised: For example, the usual value (cid:15) = 10 − 8 is valid for all L 2 H < 9 .

Original: All of the above arguments naturally extend to the diagonal preconditioner diag( A rms ) .
Revised: All of the above arguments naturally extend to the diagonal preconditioner diag( A rms ) .

Original: D D IAGONAL D OMINANCE IN N EURAL N ETWORKSD. P ROOF OF P ROPOSITION
Revised: D D IAGONAL D OMINANCE IN N EURAL N ETWORKS

Revised: In the following, we make statements about the diagonal share of random matrices.

Revised: As E [ 1 x ] might not exist for a random variable x , we cannot compute the expectation of the diagonal share but rather of for computing the diagonal share of the expectation of the random matrix in absolute terms.

Revised: Note that this notion is still meaningful, as the average of many non-diagonally dominated matrices with positive entries cannot become diagonally dominated.

Revised: D. P ROOF OF P ROPOSITION

Original: Proposition 4 (Proposition 1 restated) .
Revised: Proposition 5 (Proposition 1 restated) .

Original: For random Gaussian Wigner matrix W formed as
Revised: For random Gaussian Wigner matrix W formed as

Original: [Equation]
Revised: [Equation]

Original: (29) where ∼ stands for i.i.d.
Revised: (31) where ∼ stands for i.i.d.

Original: draws (Wigner, 1993), theexpected share of diagonal mass δ W amounts to
Revised: draws (Wigner, 1993), the diagonal mass of the expected absolute matrix amounts to

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: which simpliﬁes to 1 d if the diagonal and off-diagonal elements come from the same Gaussian distribution ( σ 1 = σ 2 ).
Revised: which simpliﬁes to 1 d if the diagonal and off-diagonal elements come from the same Gaussian distribution ( σ 1 = σ 2 ).

Original: For the sake of simplicity we only consider Gaussian Wigner matrices but the above argument naturally extends to any distribution with positive expected absolute values, i.e. we only exclude the
Revised: For the sake of simplicity we only consider Gaussian Wigner matrices but the above argument naturally extends to any distribution with positive expected absolute values, i.e. we only exclude the

Original: Dirac delta function as probability density.
Revised: Dirac delta function as probability density.

Original: [Figure]
Revised: [Figure]

Original: Appendix B: Background on second-order optimization
Revised: D. OLS B ASELINE

Revised: When considering regression tasks, a direct competitor to neural network models is the classical Ordinary Least Squares (OLS) regression, which minimizes a quadratic loss over a linear model. In this case the Hessian simply amounts to the input-covariance matrix H ols :

Revised: = X (cid:124) X , where

Revised: X ∈ R d × n .

Revised: We here show that the diagonal share of the expected matrix itself also decays in d , when n grows to inﬁnity.

Revised: However, empirical simulations suggest the validity of this result even for much smaller values of n (see Figure D.2) and it is likely that ﬁnite n results can be derived when adding assumptions such as Gaussian data.

Revised: [Figure]

Revised: Proposition 6 (Proposition 2 restated) .

Revised: Let X ∈ R n × d and assume each x i,k is generated i.i.d.

Revised: with zero-mean ﬁnite second moment σ 2 > 0 .

Revised: Then the share of diagonal mass of the expected matrix E [ | H ols | ] amounts to

Revised: [Equation]

Revised: Proof.

Original: [Equation]
Revised: [Equation]

Revised: Where we used the fact that all x i,k are i.i.d.

Revised: variables.

Original: The canonical second-order method is Newton’s methods.
Revised: Per assumption, we have E (cid:2) x 2 i, 1 (cid:3) = σ 2 , ∀ i . Furthermore, the products x i, 1 x i, 2 are i.i.d with expectation 0 and variance σ 4 . By the central limit theorem

Revised: [Equation]

Revised: in law, with Z ∼ N (0 , σ 4 ) , since E (cid:2)

Revised: Z

Revised: [Equation]

Revised: E

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: n

Revised: [Equation]

Revised: ni

Revised: [Equation]

Revised: E [ x

Revised: [Equation]

Revised: ] E [ x 2 i, 2 ] = σdue to the independence assumption.

Revised: Then E (cid:0) | Z N

Revised: [Equation]

Revised: 1 | Z N |≥ R (cid:1) ≤ E ( | Z N | 2 /R ) ≤ σ 4 /R .

Revised: This implies that

Revised: [Equation]

Revised: As a result, we have that in the limit of large n

Revised: [Equation]

Revised: [Equation]

Revised: Appendix B: Background on second-order optimization A N EWTON

Revised: [Equation]

Revised: M ETHOD

Revised: The canonical second-order method is Newton’s methods.

Original: This algorithm uses the inverse Hessian as a scaling matrix and thus has updates of the form
Revised: This algorithm uses the inverse Hessian as a scaling matrix and thus has updates of the form

Original: [Equation]
Revised: [Equation]

Original:  which is equivalent to optimizing the local quadratic model
Revised: (37) which is equivalent to optimizing the local quadratic model

Original: [Equation]
Revised: [Equation]

Original:  to ﬁrst-order stationarity .
Revised: (38) to ﬁrst-order stationarity .

Original: Using curvature information to rescale the steepest descent direction gives
Revised: Using curvature information to rescale the steepest descent direction gives

Original: Newton’s method the useful property of being linearly scale invariant.
Revised: Newton’s method the useful property of being linearly scale invariant.

Original: This gives rise to a problem independent local convergence rate that is super-linear and even quadratic in the case of Lipschitz continuous Hessians (see Nocedal & Wright (2006) Theorem 3.5), whereas gradient descent at best achieves linear local convergence (Nesterov, 2013).
Revised: This gives rise to a problem independent local convergence rate that is super-linear and even quadratic in the case of Lipschitz continuous Hessians (see Nocedal & Wright (2006) Theorem 3.5), whereas gradient descent at best achieves linear local convergence (Nesterov, 2013).

Original: However, there are certain drawbacks associated with applying classical Newton’s method.
Revised: However, there are certain drawbacks associated with applying classical Newton’s method.

Original: First of all, the Hessian matrix may be singular and thus not invertible.
Revised: First of all, the Hessian matrix may be singular and thus not invertible.

Original: Secondly, even if it is invertible the local quadratic model (Eq. 33) that is minimized in each NM iteration may simply be an inadequate approximation of the true objective.
Revised: Secondly, even if it is invertible the local quadratic model (Eq. 38) that is minimized in each NM iteration may simply be an inadequate approximation of the true objective.

Original: As a result, the Newton step is not necessarily a descent step.
Revised: As a result, the Newton step is not necessarily a descent step.

Original: It may hence approximate arbitrary critical points (including local maxima) or even diverge.
Revised: It may hence approximate arbitrary critical points (including local maxima) or even diverge.

Original: Finally, the cost of forming and inverting the Hessian sum up to O ( nd 2 + d 3 ) and are thus prohibitively high for applications in large dimensional problems.
Revised: Finally, the cost of forming and inverting the Hessian sum up to O ( nd 2 + d 3 ) and are thus prohibitively high for applications in large dimensional problems.

Original: B T RUST R EGION M ETHODS B. O UTER ITERATIONS
Revised: B T RUST R EGION M ETHODS B. O UTER ITERATIONS

Original: Trust region methods are among the most principled approaches to overcome the above mentioned issues.
Revised: Trust region methods are among the most principled approaches to overcome the above mentioned issues.

Original: These methods also construct a quadratic model m t but constrain the subproblem in such a way that the stepsize is restricted to stay within a certain radius ∆ t within which the model is trusted to be sufﬁciently adequate
Revised: These methods also construct a quadratic model m t but constrain the subproblem in such a way that the stepsize is restricted to stay within a certain radius ∆ t within which the model is trusted to be sufﬁciently adequate

Original: [Equation]
Revised: [Equation]

Original: Hence, contrary to line-search methods this approach ﬁnds the step s t and its length (cid:107) s t (cid:107) simultaneously by optimizing (34).
Revised: Hence, contrary to line-search methods this approach ﬁnds the step s t and its length (cid:107) s t (cid:107) simultaneously by optimizing (39).

Original: Subsequently the actual decrease L ( w t ) − L ( w t + s t ) is compared to the predicted decrease m t (0) − m t ( s t ) and the step is only accepted if the ratio ρ :
Revised: Subsequently the actual decrease L ( w t ) − L ( w t + s t ) is compared to the predicted decrease m t (0) − m t ( s t ) and the step is only accepted if the ratio ρ :

Original: = L ( w t ) − L ( w t + s t ) / ( m t (0) − m t ( s t )) exceeds some predeﬁned success threshold η 1 > 0 .
Revised: = L ( w t ) − L ( w t + s t ) / ( m t (0) − m t ( s t )) exceeds some predeﬁned success threshold η 1 > 0 .

Original: Furthermore, the trust region radius is decreased whenever ρ falls below η 1 and it is increased whenever ρ exceeds the "very successful" threshold η 2 0 .
Revised: Furthermore, the trust region radius is decreased whenever ρ falls below η 1 and it is increased whenever ρ exceeds the "very successful" threshold η 2 0 .

Original: Thereby, the algorithm adaptively measures the accuracy of the second-order Taylor model – which may change drastically over the parameter space depending on the behaviour of the higher-order derivatives 10 – and adapts the effective length along which the model is trusted accordingly.
Revised: Thereby, the algorithm adaptively measures the accuracy of the second-order Taylor model – which may change drastically over the parameter space depending on the behaviour of the higher-order derivatives 10 – and adapts the effective length along which the model is trusted accordingly.

Original: See Conn et al.
Revised: See Conn et al.

Original: (2000) for more details.
Revised: (2000) for more details.

Original: As a consequence, the plain Newton step s N,t = − (cid:0) ∇ 2 L t (cid:1) − 1 ∇L t is only taken if it lies within the trust region radius and yields a certain amount of decrease in the objective value.
Revised: As a consequence, the plain Newton step s N,t = − (cid:0) ∇ 2 L t (cid:1) − 1 ∇L t is only taken if it lies within the trust region radius and yields a certain amount of decrease in the objective value.

Original: Since many functions look somehow quadratic close to a minimizer the radius can be shown to grow asymptotically under mild assumptions such that eventually full Newton steps are taken in every iteration which retains the local quadratic convergence rate (Conn et al., 2000).
Revised: Since many functions look somehow quadratic close to a minimizer the radius can be shown to grow asymptotically under mild assumptions such that eventually full Newton steps are taken in every iteration which retains the local quadratic convergence rate (Conn et al., 2000).

Original: w opt w saddle w 0 w newton w TR    2 2.
Revised: w opt w saddle w 0 w newton w TR    2 2.

Original: w opt w saddle w 0 w newton w TR
Revised: w opt w saddle w 0 w newton w TR

Original: B. S UBPROBLEM SOLVER
Revised: B. S UBPROBLEM SOLVER

Original: Interestingly, there is no need to optimize Eq.
Revised: Interestingly, there is no need to optimize Eq.

Original: (34) to global optimality to retain the remarkable global convergence properties of TR algorithms.
Revised: (39) to global optimality to retain the remarkable global convergence properties of TR algorithms.

Original: Instead, it sufﬁces to do better than the Cauchyand Eigenpoint 11 simultaneously.
Revised: Instead, it sufﬁces to do better than the Cauchyand Eigenpoint 11 simultaneously.

Original: One way to ensure this is to minimize m t ( s ) in nested Krylov subspaces.
Revised: One way to ensure this is to minimize m t ( s ) in nested Krylov subspaces.

Original: These subspaces naturally include the gradient direction as well as increasingly accurate estimates of the leading eigendirection
Revised: These subspaces naturally include the gradient direction as well as increasingly accurate estimates of the leading eigendirection

Original: [Equation]
Revised: [Equation]

Original: until (for example) the stopping criterion
Revised: until (for example) the stopping criterion

Original: [Equation]
Revised: [Equation]

Original: is met, which requires increased accuracy as the underlying trust region algorithm approaches criticality.
Revised: is met, which requires increased accuracy as the underlying trust region algorithm approaches criticality.

Original: Conjugate gradients and Lanczos method are two iterative routines that implicitly build up a conjugate and orthogonal basis for such a Krylov space respectively and they converge linearly on quadratic objectives with a square-root dependency on the condition number of the Hessian (Conn et al., 2000).
Revised: Conjugate gradients and Lanczos method are two iterative routines that implicitly build up a conjugate and orthogonal basis for such a Krylov space respectively and they converge linearly on quadratic objectives with a square-root dependency on the condition number of the Hessian (Conn et al., 2000).

Original: We here employ the preconditionied Steihaug-Toint CG method (Steihaug, 1983) in order to cope with possible boundary solutions of (34) but similar techniques exist for the Lanczos solver as well for which we also provide code.
Revised: We here employ the preconditionied Steihaug-Toint CG method (Steihaug, 1983) in order to cope with possible boundary solutions of (39) but similar techniques exist for the Lanczos solver as well for which we also provide code.

Original: As preconditioning matrix for CG we use the same matrix as for the ellipsoidal constraint.
Revised: As preconditioning matrix for CG we use the same matrix as for the ellipsoidal constraint.

Original: C D AMPED (G AUSS -)N EWTON METHODS
Revised: C D AMPED (G AUSS -)N EWTON METHODS

Original: An alternative approach to actively constraining the region within which the model is trusted is to instead penalize the step norm in each iteration in a Lagrangian manner.
Revised: An alternative approach to actively constraining the region within which the model is trusted is to instead penalize the step norm in each iteration in a Lagrangian manner.

Original: This is done by so-called damped Newton methods that add a λ > 0 multiple of the identity matrix to the second-order term in the model, which leads to the update step
Revised: This is done by so-called damped Newton methods that add a λ > 0 multiple of the identity matrix to the second-order term in the model, which leads to the update step

Original: [Equation]
Revised: [Equation]

Original: This can also be solved hessian-free by conjugate gradients (or other Krylov subspace methods).
Revised: This can also be solved hessian-free by conjugate gradients (or other Krylov subspace methods).

Original: The penalty parameter λ is acting inversely to the trust region radius ∆ and it is often updated accordingly.
Revised: The penalty parameter λ is acting inversely to the trust region radius ∆ and it is often updated accordingly.

Original: Such algorithms are commonly known as Levenberg-Marquardt algorithms and they were originally tailored towards solving non-linear least squares problems (Nocedal & Wright, 2006) but they have been proposed for neural network training already early on (Hagan & Menhaj, 1994).
Revised: Such algorithms are commonly known as Levenberg-Marquardt algorithms and they were originally tailored towards solving non-linear least squares problems (Nocedal & Wright, 2006) but they have been proposed for neural network training already early on (Hagan & Menhaj, 1994).

Original: Many algorithms in the existing literature replace the use of ∇ 2 L ( w t ) in (37) with the Generalized
Revised: Many algorithms in the existing literature replace the use of ∇ 2 L ( w t ) in (42) with the Generalized

Original: Gauss Newton matrix (Martens, 2010; Chapelle & Erhan, 2011) or an approximation of the latter (Martens & Grosse, 2015).
Revised: Gauss Newton matrix (Martens, 2010; Chapelle & Erhan, 2011) or an approximation of the latter (Martens & Grosse, 2015).

Original: This matrix constitutes the ﬁrst part of the well-known Gauss-Newton decomposition
Revised: This matrix constitutes the ﬁrst part of the well-known Gauss-Newton decomposition

Original: [Equation]
Revised: [Equation]

Original:  where l (cid:48) and l (cid:48)(cid:48) are the ﬁrst and second derivative of l : R out → R + assuming that out = 1 (binary classiﬁcation and regression task) for simplicity here.
Revised: (43) where l (cid:48) and l (cid:48)(cid:48) are the ﬁrst and second derivative of l : R out → R + assuming that out = 1 (binary classiﬁcation and regression task) for simplicity here.

Original: It is interesting to note that the GGN matrix A GGN of neural networks is equivalent to the Fisher matrix used in natural gradient descent (Amari, 1998) in many cases like linear activation function and squared error as well as sigmoid and cross-entropy or softmax and negative log-likelihood for which the extended Gauss-Newton is deﬁned (Pascanu & Bengio, 2013).
Revised: It is interesting to note that the GGN matrix A GGN of neural networks is equivalent to the Fisher matrix used in natural gradient descent (Amari, 1998) in many cases like linear activation function and squared error as well as sigmoid and cross-entropy or softmax and negative log-likelihood for which the extended Gauss-Newton is deﬁned (Pascanu & Bengio, 2013).

Original: As can be seen in (38) the matrix A GGN is positive semideﬁnite (and low rank if n < d ).
Revised: As can be seen in (43) the matrix A GGN is positive semideﬁnite (and low rank if n < d ).

Original: As a result, there exist no second-order convergence guarantees for such methods on general non-convex problems.
Revised: As a result, there exist no second-order convergence guarantees for such methods on general non-convex problems.

Original: On the other end of the spectrum, the GGN also drops possibly positive terms from the Hessian (see 38).
Revised: On the other end of the spectrum, the GGN also drops possibly positive terms from the Hessian (see 43).

Original: Hence it is not guaranteed to be an upper bound on the latter in the PSD sense.
Revised: Hence it is not guaranteed to be an upper bound on the latter in the PSD sense.

Original: Essentially, GGN approximations assume that the network is piece-wise linear and thus the GGN and Hessian matrices only coincide in the case of linear and ReLU activations or non-curved loss functions.
Revised: Essentially, GGN approximations assume that the network is piece-wise linear and thus the GGN and Hessian matrices only coincide in the case of linear and ReLU activations or non-curved loss functions.

Original: For any other activation the GGN matrix may approximate the Hessian only asymptotically and if the (cid:96) (cid:48) ( f i ( · )) terms in 38 go to zero for all i ∈ { 1 , . . . , n } .
Revised: For any other activation the GGN matrix may approximate the Hessian only asymptotically and if the (cid:96) (cid:48) ( f i ( · )) terms in 43 go to zero for all i ∈ { 1 , . . . , n } .

Original: In non-linear least squares such problems are called zero-residual problems and GN methods can be shown to have quadratic local convergence there.
Revised: In non-linear least squares such problems are called zero-residual problems and GN methods can be shown to have quadratic local convergence there.

Original: In any other case the convergence rate does not exceed the linear local convergence bound of gradient descent.
Revised: In any other case the convergence rate does not exceed the linear local convergence bound of gradient descent.

Original: In practice however there are cases where deep neural nets do show negative curvature in the neighborhood of a minimizer (Bottou et al., 2018).Finally, Dauphin et al.
Revised: In practice however there are cases where deep neural nets do show negative curvature in the neighborhood of a minimizer (Bottou et al., 2018).Finally, Dauphin et al.

Original: (2014) propose the use of the absolute Hessian instead of the GGN matrix in a framework similar to 37.
Revised: (2014) propose the use of the absolute Hessian instead of the GGN matrix in a framework similar to 42.

Original: This method has been termed saddle-free Newton even though its manifold of attraction to a given saddle is non-empty 12 .
Revised: This method has been termed saddle-free Newton even though its manifold of attraction to a given saddle is non-empty 12 .

Original: [Figure]
Revised: w opt w saddle w 0 w GN    2 2. w opt w saddle w 0 w | H |

Original: C. C OMPARISON TO TRUST REGION
Revised: C. C OMPARISON TO TRUST REGION

Original: Contrary to TR methods, the Levenberg-Marquardt methods never take plain Newton steps since the regularization is always on ( λ > 0 ).
Revised: Contrary to TR methods the Levenberg-Marquardt methods never take plain Newton steps since the regularization is always on ( λ > 0 ).

Original: Furthermore, if a positive-deﬁnite Hessian approximation like the Generalized Gauss Newton matrix is used, this algorithm is not capable of exploiting negative curvature and there are cases in neural network training where the Hessian is much better conditioned than the Gauss-Newton matrix (Mizutani & Dreyfus, 2008)
Revised: Furthermore, if a positive-deﬁnite Hessian approximation like the Generalized Gauss Newton matrix is used this algorithm is not capable of exploiting negative curvature and there are cases in neural network training where the Hessian is much better conditioned than the Gauss-Newton matrix (Mizutani & Dreyfus, 2008)

Original: (also see Figure 8).
Revised: (also see Figure 9).

Original: While some scholars believe that positive-deﬁniteness is a desirable feature (Martens, 2010; Chapelle & Erhan, 2011), we want to point out that following negative curvature directions is necessarily needed to escape saddle points and it can also be meaningful to follow directions of negative eigenvalue λ outside a saddle since they guarantee O ( | λ | 3 ) progress, whereas a gradient descent step yields at least (cid:107)∇ f ( w ) (cid:107)progress (both under certain stepsize conditions) and one cannot conclude a-priori which one is better in general (Curtis & Robinson, 2017; Alain et al., 2018).
Revised: While some scholars believe that positive-deﬁniteness is a desirable feature (Martens, 2010; Chapelle & Erhan, 2011) we want to point out that following negative curvature directions is necessarily needed to escape saddle points and it can also be meaningful to follow directions of negative eigenvalue λ outside a saddle since they guarantee O ( | λ | 3 ) progress, whereas a gradient descent step yields at least (cid:107)∇ f ( w ) (cid:107)progress (both under certain stepsize conditions) and one cannot conclude a-priori which one is better in general (Curtis & Robinson, 2017; Alain et al., 2018).

Original: Despite these theoretical considerations, many methods based on GGN matrices have been applied to neural network training (see Martens (2014) and references therein) and particularly the hessian-free implementations of (Martens, 2010;
Revised: Despite these theoretical considerations, many methods based on GGN matrices have been applied to neural network training (see Martens (2014) and references therein) and particularly the hessian-free implementations of (Martens, 2010;

Original: Chapelle & Erhan, 2011) can be implemented very cheaply (Schraudolph, 2002).
Revised: Chapelle & Erhan, 2011) can be implemented very cheaply (Schraudolph, 2002).

Original: D U SING H ESSIAN INFORMATION IN N EURAL N ETWORKS
Revised: D U SING H ESSIAN INFORMATION IN N EURAL N ETWORKS

Original: While many theoretical arguments suggest the superiority of regularized Newton methods over gradient based algorithms, several practical considerations cast doubt on this theoretical superiority when it comes to neural network training.
Revised: While many theoretical arguments suggest the superiority of regularized Newton methods over gradient based algorithms, several practical considerations cast doubt on this theoretical superiority when it comes to neural network training.

Original: Answers to the following questions are particularly unclear: Are saddles even an issue in deep learning?
Revised: Answers to the following questions are particularly unclear: Are saddles even an issue in deep learning?

Original: Is superlinear local convergence a desirable feature in machine learning applications (test error)?
Revised: Is superlinear local convergence a desirable feature in machine learning applications (test error)?

Original: Are second-order methods more "vulnerable" to sub-sampling noise?
Revised: Are second-order methods more "vulnerable" to sub-sampling noise?

Original: Do worst-case iteration complexities even matter in real-world settings?
Revised: Do worst-case iteration complexities even matter in real-world settings?

Original: As a result, the value of Hessian information in neural network training is somewhat unclear a-priori and so far a conclusive empirical study is still missing.
Revised: As a result, the value of Hessian information in neural network training is somewhat unclear a-priori and so far a conclusive empirical study is still missing.

Original: Our empirical ﬁndings indicate that the net value of Hessian information for neural network training is indeed somewhat limited for mainly three reasons: 1) second-order methods rarely yield better limit points, which suggests that saddles and spurious local minima are not a major obstacle; 2) gradient methods can indeed run on smaller batch sizes which is beneﬁcial in terms of epoch and when memory is limited; 3) The per-iteration time complexity is noticeably lower for ﬁrst-order methods.
Revised: Our empirical ﬁndings indicate that the net value of Hessian information for neural network training is indeed somewhat limited for mainly three reasons: 1) second-order methods rarely yield better limit points, which suggests that saddles and spurious local minima are not a major obstacle; 2) gradient methods can indeed run on smaller batch sizes which is beneﬁcial in terms of epoch and when memory is limited; 3) The per-iteration time complexity is noticeably lower for ﬁrst-order methods.

Original: In summary, these observations suggest that advances in hardware and distributed second-order algorithms (e.g., Osawa et al.
Revised: In summary, these observations suggest that advances in hardware and distributed second-order algorithms (e.g., Osawa et al.

Original: ; Dünner et al.
Revised: ; Dünner et al.

Original: (2018)) will be needed before Newton-type methods can replace (stochastic) gradient methods in deep learning
Revised: (2018)) will be needed before Newton-type methods can replace (stochastic) gradient methods in deep learning. Appendix C: Experiment details A E XPERIMENTAL

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: Appendix C: Experiment details A E XPERIMENTAL RESULTS A. E LLIPSOIDAL T RUST R EGION VS . F IRST - ORDER O PTIMIZER ResNet
Revised: a [Figure] h i

Original: [Figure]
Revised: [Figure]

Original: ResNet
Revised: M N I S T

Original: [Figure]
Revised: [Figure]

Original: B F URTHER E XPERIMENT D ETAILS B. D EFAULT PARAMETERS , ARCHITECTURES AND DATASETS
Revised: B F URTHER E XPERIMENT D ETAILS B. D EFAULT PARAMETERS , ARCHITECTURES AND DATASETS

Original: Parameters Table 1 reports the default parameters we consider.
Revised: Parameters Table 1 reports the default parameters we consider.

Original: Only for the larger ResNet18 on
Revised: Only for the larger ResNet18 on

Original: CIFAR-10, we adapted the batch size to 128 due to memory constraints.
Revised: CIFAR-10, we adapted the batch size to 128 due to memory constraints

Original: [Equation]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Datasets We use two real-world datasets for image classiﬁcation, namely CIFAR-10 and FashionMNIST 13 .
Revised: Datasets We use two real-world datasets for image classiﬁcation, namely CIFAR-10 and FashionMNIST 13 .

Original: While Fashion-MNIST consists of greyscale 28 × 28 images, CIFAR-10 are colored images of size 32 × 32 .
Revised: While Fashion-MNIST consists of greyscale 28 × 28 images, CIFAR-10 are colored images of size 32 × 32 .

Original: Both datasets have a ﬁxed training-test split consisting of 60,000 and 10, images, respectively.
Revised: Both datasets have a ﬁxed training-test split consisting of 60,000 and 10, images, respectively.

Original: Network architectures The MLP architectures are simple.
Revised: Network architectures The MLP architectures are simple.

Original: For MNIST and Fashion-MNIST we use a 784 − 128 − 10 network with tanh activations and a cross entropy loss.
Revised: For MNIST and Fashion-MNIST we use a 784 − 128 − 10 network with tanh activations and a cross entropy loss.

Original: The networks has 101 (cid:48) 770 parameters.
Revised: The networks has 101 (cid:48) 770 parameters.

Original: For the CIFAR-10 MLP we use a 3072 − 128 − 128 − 10 architecture also with tanh activations and cross entropy loss.
Revised: For the CIFAR-10 MLP we use a 3072 − 128 − 128 − 10 architecture also with tanh activations and cross entropy loss.

Original: This network has 410 (cid:48) 880 parameters.
Revised: This network has 410 (cid:48) 880 parameters.

Original: The Fashion-MNIST autoencoder has the same architecture as the one used in Hinton & Salakhutdinov (2006); Xu et al.
Revised: The Fashion-MNIST autoencoder has the same architecture as the one used in Hinton & Salakhutdinov (2006); Xu et al.

Original: (2017a);
Revised: (2017a);

Original: Martens (2010); Martens & Grosse (2015).
Revised: Martens (2010); Martens & Grosse (2015).

Original: The encoder structure is 784 − 1000 − 500 − 250 − 30 and the decoder is mirrored.
Revised: The encoder structure is 784 − 1000 − 500 − 250 − 30 and the decoder is mirrored.

Original: Sigmoid activations are used in all but the central layer.
Revised: Sigmoid activations are used in all but the central layer.

Original: The reconstructed images are fed pixelwise into a binary cross entropy loss.
Revised: The reconstructed images are fed pixelwise into a binary cross entropy loss.

Original: The network has a total of 2 (cid:48) 833 (cid:48)
Revised: The network has a total of 2 (cid:48) 833 (cid:48)

Original: 000 parameters.
Revised: 000 parameters.

Original: The CIFAR-10 autoencoder is taken from the implementation of https://github.com/jellycsc/PyTorch-CIFAR-10-autoencoder.
Revised: The CIFAR-10 autoencoder is taken from the implementation of https://github.com/jellycsc/PyTorch-CIFAR-10-autoencoder.

Original: For the ResNet18, we used the implementation from torchvision for CIFAR-10 as well as a modiﬁcation of it for Fashion-MNIST that adapts the ﬁrst convolution to account for the single input channel.
Revised: For the ResNet18, we used the implementation from torchvision for CIFAR-10 as well as a modiﬁcation of it for Fashion-MNIST that adapts the ﬁrst convolution to account for the single input channel.

Original: In all of our experiments each method was run on one Tesla P100 GPU using the PyTorch (Paszke et al., 2017) library.
Revised: In all of our experiments each method was run on one Tesla P100 GPU using the PyTorch (Paszke et al., 2017) library.

Original: B. R ECONSTRUCTED I MAGES FROM A UTOENCODERS
Revised: B. R ECONSTRUCTED I MAGES FROM A UTOENCODERS

Original: Original SGD Adagrad Rmsprop
Revised: [Figure]

Original: TR Uniform

Original: TR Adagrad

Original: TR RMSprop