Original: A C OMPUTATIONALLY E FFICIENT S PARSIFIED O NLINE N EWTON M ETHOD
Revised: A C OMPUTATIONALLY E FFICIENT S PARSIFIED O NLINE N EWTON M ETHOD

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Second-order methods have enormous potential in improving the convergence of deep neural network (DNN) training, but are prohibitive due to their large memory and compute requirements.
Revised: Second-order methods have enormous potential in improving the convergence of deep neural network (DNN) training, but are prohibitive due to their large memory and compute requirements.

Original: Furthermore, computing the matrix inverse or the
Revised: Furthermore, computing the matrix inverse or the

Original: Newton direction, which is needed in second-order methods, requires high precision computation for stable training as the preconditioner could have a large condition number.
Revised: Newton direction, which is needed in second-order methods, requires high precision computation for stable training as the preconditioner could have a large condition number.

Original: This paper provides a first attempt at developing computationally efficient sparse preconditioners for DNN training which can also tolerate low precision computation.
Revised: This paper provides a first attempt at developing computationally efficient sparse preconditioners for DNN training which can also tolerate low precision computation.

Original: Our new Sparsified Online Newton (SONew) algorithm emerges from the novel use of the LogDet matrix divergence measure; we combine it with sparsity constraints to minimize regret in the online convex optimization framework.
Revised: Our new Sparsified Online Newton (SONew) algorithm emerges from the novel use of the LogDet matrix divergence measure; we combine it with sparsity constraints to minimize regret in the online convex optimization framework.

Original: Our mathematical analysis allows us to reduce the condition number of our sparse preconditioning matrix, thus improving the stability of training with low precision.
Revised: Our mathematical analysis allows us to reduce the condition number of our sparse preconditioning matrix, thus improving the stability of training with low precision.

Original: We conduct experiments on a feed-forward neural-network autoencoder benchmark, where we compare training loss of optimizers when run for a fixed number of epochs.
Revised: We conduct experiments on a feed-forward neural-network autoencoder benchmark, where we compare training loss of optimizers when run for a fixed number of epochs.

Original: In the float32 experiments, our methods outperform the best-performing first-order optimizers and perform comparably to Shampoo, a state-of-the-art second-order optimizer.
Revised: In the float32 experiments, our methods outperform the best-performing first-order optimizers and perform comparably to Shampoo, a state-of-the-art second-order optimizer.

Original: However, our method is even more effective in low precision, where SONew finishes training considerably faster while performing comparably with Shampoo on training loss.
Revised: However, our method is even more effective in low precision, where SONew finishes training considerably faster while performing comparably with Shampoo on training loss.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Stochastic first order methods which use the negative gradient direction to update parameters have become the standard for training deep neural networks (DNNs).
Revised: Stochastic first order methods which use the negative gradient direction to update parameters have become the standard for training deep neural networks (DNNs).

Original: Gradient-based preconditioning involves finding an update direction, by multiplying the gradient with a preconditioner matrix carefully chosen from gradients observed in previous iterations, to improve convergence.
Revised: Gradient-based preconditioning involves finding an update direction, by multiplying the gradient with a preconditioner matrix carefully chosen from gradients observed in previous iterations, to improve convergence.

Original: (Full-matrix)
Revised: (Full-matrix)

Original: Adagrad (Duchi et al., 2011b), online Newton method (Hazan et al., 2007) and natural gradient descent (Amari, 1998) use a full-matrix preconditioner, but computing and storing the full matrix is infeasible when there are millions of parameters.
Revised: Adagrad (Duchi et al., 2011b), online Newton method (Hazan et al., 2007) and natural gradient descent (Amari, 1998) use a full-matrix preconditioner, but computing and storing the full matrix is infeasible when there are millions of parameters.

Original: Thus, diagonal versions such as diagonal Adagrad,
Revised: Thus, diagonal versions such as diagonal Adagrad,

Original: Adam (Kingma & Ba, 2014), and RMSprop (Hinton et al., 2012) are now widely used to train DNNs due to their scalability.
Revised: Adam (Kingma & Ba, 2014), and RMSprop (Hinton et al., 2012) are now widely used to train DNNs due to their scalability.

Original: Several higher-order methods have previously been applied to deep learning (Gupta et al., 2018; Anil et al., 2020;
Revised: Several higher-order methods have previously been applied to deep learning (Gupta et al., 2018; Anil et al., 2020;

Original: Goldfarb et al., 2020; Martens & Grosse, 2015).
Revised: Goldfarb et al., 2020; Martens & Grosse, 2015).

Original: All these methods use Kronecker products that reduce computational and storage costs to make them feasible for training neural networks.
Revised: All these methods use Kronecker products that reduce computational and storage costs to make them feasible for training neural networks.

Original: However, these methods rely on matrix inverses or pth-roots that require high precision arithmetic as the matrices they deal with can have large condition numbers (Anil et al., 2020; 2022).
Revised: However, these methods rely on matrix inverses or pth-roots that require high precision arithmetic as the matrices they deal with can have large condition numbers (Anil et al., 2020; 2022).

Original: Meanwhile, deep learning hardware accelerators have evolved towards using lower precision (bfloat16, float16, int8) (Henry et al., 2019; Jouppi et al., 2017) to reduce overall computational and memory costs and improve training performance.
Revised: Meanwhile, deep learning hardware accelerators have evolved towards using lower precision (bfloat16, float16, int8) (Henry et al., 2019; Jouppi et al., 2017) to reduce overall computational and memory costs and improve training performance.

Original: This calls for further research in developing efficient optimization techniques that work with low precision.
Revised: This calls for further research in developing efficient optimization techniques that work with low precision.

Original: Indeed there is recent work along these directions, from careful quantization of Adam (Dettmers et al., 2021) to 8-bits to optimizer agnostic local loss optimization (Amid et al., 2022) that leverage first-order methods to match higher-order methods.
Revised: Indeed there is recent work along these directions, from careful quantization of Adam (Dettmers et al., 2021) to 8-bits to optimizer agnostic local loss optimization (Amid et al., 2022) that leverage first-order methods to match higher-order methods.

Original: In this paper, we present a first attempt towards computationally efficient sparse preconditioners for DNN training.
Revised: In this paper, we present a first attempt towards computationally efficient sparse preconditioners for DNN training.

Original: Regret analysis when using a preconditioner reveals that the error is bounded by two summations (see (3) below); the first summation depends on the change in the preconditioning matrix, while the second depends on the generalized gradient norm.
Revised: Regret analysis when using a preconditioner reveals that the error is bounded by two summations (see (3) below); the first summation depends on the change in the preconditioning matrix, while the second depends on the generalized gradient norm.

Original: We take the approach of minimizing the second term while regularizing two successive preconditioners to be close in the
Revised: We take the approach of minimizing the second term while regularizing two successive preconditioners to be close in the

Original: LogDet matrix divergence measure (Kulis et al., 2009).
Revised: LogDet matrix divergence measure (Kulis et al., 2009).

Original: This technique gives us an Online Newton method (Hazan et al., 2007).
Revised: This technique gives us an Online Newton method (Hazan et al., 2007).

Original: To make it computationally efficient, we further sparsify the preconditioner by finding a sparse approximation that is close in LogDet divergence.
Revised: To make it computationally efficient, we further sparsify the preconditioner by finding a sparse approximation that is close in LogDet divergence.

Original: Thus we are consistent in using the same measure (LogDet divergence) in both the regularization and sparsification steps.
Revised: Thus we are consistent in using the same measure (LogDet divergence) in both the regularization and sparsification steps.

Original: This gives us our Sparsified Online Newton (SONew) method, which only requires O ( n ) time and memory complexity per iteration.
Revised: This gives us our Sparsified Online Newton (SONew) method, which only requires O ( n ) time and memory complexity per iteration.

Original: We achieve this by imposing structured sparsity, such as tridiagonal and banded sparsity patterns, in the preconditioner.
Revised: We achieve this by imposing structured sparsity, such as tridiagonal and banded sparsity patterns, in the preconditioner.

Original: This is unlike most existing online Newton methods that require at least O ( n 2 ) space and time complexity.
Revised: This is unlike most existing online Newton methods that require at least O ( n 2 ) space and time complexity.

Original: By making each step linear time, the SONew method can be applied to train modern DNNs.
Revised: By making each step linear time, the SONew method can be applied to train modern DNNs.

Original: Further, for some sparsity structures, our method is easily parallelized thus making negligible the overhead of computing the preconditioner.
Revised: Further, for some sparsity structures, our method is easily parallelized thus making negligible the overhead of computing the preconditioner.

Original: We also show that introducing sparsity allows us to reduce the condition number of the problem; as a consequence our preconditioner allows us to train DNNs even in low precision arithmetic.
Revised: We also show that introducing sparsity allows us to reduce the condition number of the problem; as a consequence our preconditioner allows us to train DNNs even in low precision arithmetic.

Original: We establish regret bound guarantees of our algorithm in the online convex optimization framework.
Revised: We establish regret bound guarantees of our algorithm in the online convex optimization framework.

Original: This involves using various properties about LogDet divergence and connections to other
Revised: This involves using various properties about LogDet divergence and connections to other

Original: Bregman matrix divergences (Bregman, 1967), such as the von Neumann matrix divergence (Kulis et al., 2009).
Revised: Bregman matrix divergences (Bregman, 1967), such as the von Neumann matrix divergence (Kulis et al., 2009).

Original: We conduct experiments on an MLP Autoencoder, where we obtain better training loss compared to first order methods.
Revised: We conduct experiments on an MLP Autoencoder, where we obtain better training loss compared to first order methods.

Original: We also conduct experiments on large-scale benchmarks in
Revised: We also conduct experiments on large-scale benchmarks in

Original: Appendix A.5, and observe comparable or improved performance than Adam (Kingma & Ba, 2014).
Revised: Appendix A.5, and observe comparable or improved performance than Adam (Kingma & Ba, 2014).

Original: Our MLP experiments in limited precision arithmetic (bfloat16) showed comparable performance with second-order methods, while being considerably faster.
Revised: Our MLP experiments in limited precision arithmetic (bfloat16) showed comparable performance with second-order methods, while being considerably faster.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: The inner product between matrices is defined as ⟨ A, B ⟩ =
Revised: The inner product between matrices is defined as ⟨ A, B ⟩ =

Original: Tr( A T B ) , where Tr( . ) denotes the matrix trace.
Revised: Tr( A T B ) , where Tr( . ) denotes the matrix trace.

Original: The Frobenius norm of a matrix A is ∥
Revised: The Frobenius norm of a matrix A is ∥

Original: A ∥ F = (cid:112)
Revised: A ∥ F = (cid:112)

Original: Tr( A T A ) , while its spectral norm is ∥ A ∥ 2 = max x ∥
Revised: Tr( A T A ) , while its spectral norm is ∥ A ∥ 2 = max x ∥

Original: Ax ∥ 2 / ∥ x ∥ 2 .
Revised: Ax ∥ 2 / ∥ x ∥ 2 .

Original: We use I n ∈ R n × n to denote an identity matrix.
Revised: We use I n ∈ R n × n to denote an identity matrix.

Original: We use S n , S ++ n to denote the set of symmetric, and positive definite matrices respectively.
Revised: We use S n , S ++ n to denote the set of symmetric, and positive definite matrices respectively.

Original: The generalized norm of a vector x ∈ R n with respect to matrix A ∈ S ++ n is defined as ∥ x ∥ A = √ x T Ax .
Revised: The generalized norm of a vector x ∈ R n with respect to matrix A ∈ S ++ n is defined as ∥ x ∥ A = √ x T Ax .

Original: We use det ( A ) to denote the determinant of matrix A , and diag( A ) to denote diagonal matrix with diag( A ) ii =
Revised: We use det ( A ) to denote the determinant of matrix A , and diag( A ) to denote diagonal matrix with diag( A ) ii =

Original: A ii .
Revised: A ii .

Original: We use G and ˜ G to denote a graph and its sub-graph with a vertex set [ n ] = { 1 , . . . , n } .
Revised: We use G and ˜ G to denote a graph and its sub-graph with a vertex set [ n ] = { 1 , . . . , n } .

Original: Let E G denote set of edges in graph G , and neig G ( i ) denote neighbours of vertex i in graph G .
Revised: Let E G denote set of edges in graph G , and neig G ( i ) denote neighbours of vertex i in graph G .

Original: The sparsity graph/pattern of a matrix A ∈ R n × n is a graph G with edges E G
Revised: The sparsity graph/pattern of a matrix A ∈ R n × n is a graph G with edges E G

Original: =
Revised: =

Original: ( i, j ) :
Revised: ( i, j ) :

Original: A ij ̸ = 0 } corresponding to the non-zero entries in A .
Revised: A ij ̸ = 0 } corresponding to the non-zero entries in A .

Original: We extensively use S n ( G ) ++ to denote the set of positive definite matrices with sparsity structure given by graph G .
Revised: We extensively use S n ( G ) ++ to denote the set of positive definite matrices with sparsity structure given by graph G .

Original: Given an index set I
Revised: Given an index set I

Original: = { i 1 , i 2 , .., i n } , we use
Revised: = { i 1 , i 2 , .., i n } , we use

Original: A II to denote the corresponding principal sub-matrix of A .
Revised: A II to denote the corresponding principal sub-matrix of A .

Original: 2. L OG D ET MATRIX DIVERGENCE
Revised: 2. L OG D ET MATRIX DIVERGENCE

Original: Let ϕ : S ++ n → R be a strictly convex, differentiable function.
Revised: Let ϕ : S ++ n → R be a strictly convex, differentiable function.

Original: Then the Bregman matrix divergence between X, Y ∈ S ++ n is defined as (Bregman, 1967; Kulis et al., 2009):
Revised: Then the Bregman matrix divergence between X, Y ∈ S ++ n is defined as (Bregman, 1967; Kulis et al., 2009):

Original: [Equation]
Revised: [Equation]

Original: Since ϕ is convex, D ϕ
Revised: Since ϕ is convex, D ϕ

Original: ( X, U ) ≥ 0 for all X, Y ≻ 0 .
Revised: ( X, U ) ≥ 0 for all X, Y ≻ 0 .

Original: A well known example is when ϕ ( X ) =
Revised: A well known example is when ϕ ( X ) =

Original: ∥
Revised: ∥

Original: X ∥ 2 F , the corresponding Bregman divergence D ϕ
Revised: X ∥ 2 F , the corresponding Bregman divergence D ϕ

Original: ( X, Y ) =
Revised: ( X, Y ) =

Original: ∥ X − Y ∥
Revised: ∥ X − Y ∥

Original: 2 F is the squared Frobenius norm.
Revised: 2 F is the squared Frobenius norm.

Original: In this paper, we extensively use the divergence when the convex function is ϕ ( X )
Revised: In this paper, we extensively use the divergence when the convex function is ϕ ( X )

Original: = − log det ( X ) ; the corresponding divergence measure D ℓ d ( X, Y ) is called the LogDet matrix divergence :
Revised: = − log det ( X ) ; the corresponding divergence measure D ℓ d ( X, Y ) is called the LogDet matrix divergence :

Original: [Equation]
Revised: [Equation]

Original: The LogDet divergence is scale invariant to invertible matrices A , i.e. D ℓ d ( A T XA, A T Y A ) =
Revised: The LogDet divergence is scale invariant to invertible matrices A , i.e. D ℓ d ( A T XA, A T Y A ) =

Original: D ℓ d ( X, Y ) .
Revised: D ℓ d ( X, Y ) .

Original: The following revealing form of LogDet divergence in terms of eigendecompositions of X = V Λ V T and Y =
Revised: The following revealing form of LogDet divergence in terms of eigendecompositions of X = V Λ V T and Y =

Original: U Θ U T (Kulis et al., 2009):
Revised: U Θ U T (Kulis et al., 2009):

Original: [Equation]
Revised: [Equation]

Original: These two properties are later used in Section 3 to highlight the significance of LogDet divergence in our algorithm.
Revised: These two properties are later used in Section 3 to highlight the significance of LogDet divergence in our algorithm.

Original:  SON EW : S PARSIFIED O NLINE N EWTON M ETHOD
Revised:  SON EW : S PARSIFIED O NLINE N EWTON M ETHOD

Original: We now present our proposed SONew algorithm.
Revised: We now present our proposed SONew algorithm.

Original: 3. R EGRET MINIMIZATION VIA L OG D ET DIVERGENCE
Revised: 3. R EGRET MINIMIZATION VIA L OG D ET DIVERGENCE

Original: We set up our problem under the online convex optimization framework (OCO) (Shalev-Shwartz et al., 2012; Hazan et al., 2016), where at each round the learner makes a prediction w t in an online fashion and receives a convex loss f t ( w t ) and gradient g t = ∇ f t ( w t ) as feedback.
Revised: We set up our problem under the online convex optimization framework (OCO) (Shalev-Shwartz et al., 2012; Hazan et al., 2016), where at each round the learner makes a prediction w t in an online fashion and receives a convex loss f t ( w t ) and gradient g t = ∇ f t ( w t ) as feedback.

Original: The goal of the learner is to reduce regret R T by predicting w t so that a low aggregate loss (cid:80) Tt =1 f t ( w t ) is achieved compared to the best possible, w ∗ =
Revised: The goal of the learner is to reduce regret R T by predicting w t so that a low aggregate loss (cid:80) Tt =1 f t ( w t ) is achieved compared to the best possible, w ∗ =

Original: arg min w (cid:80) Tt =1 f t ( w ) .
Revised: arg min w (cid:80) Tt =1 f t ( w ) .

Original: Formally, the regret is given by
Revised: Formally, the regret is given by

Original: [Equation]
Revised: [Equation]

Original: To upper bound this regret, we proceed as in Hazan et al.
Revised: To upper bound this regret, we proceed as in Hazan et al.

Original: by analyzing the error in the iterates for the update w t +1 :
Revised: by analyzing the error in the iterates for the update w t +1 :

Original: = w t − ηX t g t , where X t ∈ R n × n .
Revised: = w t − ηX t g t , where X t ∈ R n × n .

Original: Then ∥ w t +1 − w ∗ ∥ 2 X − 1 t = ∥
Revised: Then ∥ w t +1 − w ∗ ∥ 2 X − 1 t = ∥

Original: w t −
Revised: w t −

Original: [Equation]
Revised: [Equation]

Original: t g t − w
Revised: t g t − w

Original: [Equation]
Revised: [Equation]

Original: ∥ 2 X
Revised: ∥ 2 X

Original: [Equation]
Revised: [Equation]

Original: 1 t =
Revised: 1 t =

Original: [Equation]
Revised: [Equation]

Original: t +
Revised: t +

Original: [Equation]
Revised: [Equation]

Original: g
Revised: g

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: η ( w
Revised: η ( w

Original: [Equation]
Revised: [Equation]

Original: − w ∗ ) T g t .
Revised: − w ∗ ) T g t .

Original: The convexity of f t implies that f t ( w t ) − f t ( w ∗ ) ≤ ( w t − w ∗ ) T g t leading to f t ( w t ) − f t ( w ∗ ) ≤ ∥ w t − w ∗ ∥ 2 X − 1 t − ∥ w t +1 − w ∗ ∥ 2 X − 1 t + η 2 g Tt X t g t .
Revised: The convexity of f t implies that f t ( w t ) − f t ( w ∗ ) ≤ ( w t − w ∗ ) T g t leading to f t ( w t ) − f t ( w ∗ ) ≤ ∥ w t − w ∗ ∥ 2 X − 1 t − ∥ w t +1 − w ∗ ∥ 2 X − 1 t + η 2 g Tt X t g t .

Original: Summing over all t ∈ [ T ] and rearranging reveals the following upper bound on overall regret:
Revised: Summing over all t ∈ [ T ] and rearranging reveals the following upper bound on overall regret:

Original: [Equation]
Revised: [Equation]

Original: Since w ∗ is unknown, finding X t which minimizes (3) is infeasible.
Revised: Since w ∗ is unknown, finding X t which minimizes (3) is infeasible.

Original: So to minimize regret, we attempt to minimize the last term in (3) while regularizing X − 1 t to be “close” to X − 1 t − 1 .
Revised: So to minimize regret, we attempt to minimize the last term in (3) while regularizing X − 1 t to be “close” to X − 1 t − 1 .

Original: The nearness measure we choose is the LogDet matrix divergence, thus leading to the following objective
Revised: The nearness measure we choose is the LogDet matrix divergence, thus leading to the following objective

Original: [Equation]
Revised: [Equation]

Original: , such that D
Revised: , such that D

Original: [Equation]
Revised: [Equation]

Original: where D ℓ d is as in (1).
Revised: where D ℓ d is as in (1).

Original: Why do we use the LogDet divergence?
Revised: Why do we use the LogDet divergence?

Original: From (2), due to the term λ i /θ j ,
Revised: From (2), due to the term λ i /θ j ,

Original: D ℓ d ( X, X t − 1 ) prioritizes matching the smaller eigenvalues of X t − 1 with those of X , i.e., matching the larger eigenvalues of X − 1 t − 1 and X − 1 .
Revised: D ℓ d ( X, X t − 1 ) prioritizes matching the smaller eigenvalues of X t − 1 with those of X , i.e., matching the larger eigenvalues of X − 1 t − 1 and X − 1 .

Original: As a consequence, LogDet divergence regularizes X by matching up its large eigenvalues with those of X t − 1 .
Revised: As a consequence, LogDet divergence regularizes X by matching up its large eigenvalues with those of X t − 1 .

Original: For e.g., if smallest and largest eigenvalue of X t − 1 are θ n
Revised: For e.g., if smallest and largest eigenvalue of X t − 1 are θ n

Original: and θ 1 , then for an eigenvalue λ of X , when λ > θ n , θ 1 , the penalty from (2) for θ n is higher than for θ 1 , ( λ/θ n − log( λ/θ n ) − 1) >
Revised: and θ 1 , then for an eigenvalue λ of X , when λ > θ n , θ 1 , the penalty from (2) for θ n is higher than for θ 1 , ( λ/θ n − log( λ/θ n ) − 1) >

Original: ( λ/θ 1 − log( λ/θ 1 ) − 1) .
Revised: ( λ/θ 1 − log( λ/θ 1 ) − 1) .

Original: This intuition leads us to formulate (4) as our objective.
Revised: This intuition leads us to formulate (4) as our objective.

Original: We recall that there is precedence of using the LogDet divergence in the optimization literature; indeed the celebrated BFGS algorithm (Broyden, 1967;
Revised: We recall that there is precedence of using the LogDet divergence in the optimization literature; indeed the celebrated BFGS algorithm (Broyden, 1967;

Original: Fletcher, 1970; Goldfarb, 1970; Shanno, 1970) can be shown to be the unique solution obtained when the LogDet divergence between successive preconditioners, subject to a secant constraint, is minimized (as shown in the beautiful 4-page paper by Fletcher (1991)).
Revised: Fletcher, 1970; Goldfarb, 1970; Shanno, 1970) can be shown to be the unique solution obtained when the LogDet divergence between successive preconditioners, subject to a secant constraint, is minimized (as shown in the beautiful 4-page paper by Fletcher (1991)).

Original: The optimization problem in (4) is convex in X since the LogDet divergence is convex in its first argument.
Revised: The optimization problem in (4) is convex in X since the LogDet divergence is convex in its first argument.

Original: The Lagrange L ( X, λ t ) =
Revised: The Lagrange L ( X, λ t ) =

Original: g
Revised: g

Original: Tt Xg t + λ t
Revised: Tt Xg t + λ t

Original: (D ℓ d ( X, X t − 1 ) − c t ) =
Revised: (D ℓ d ( X, X t − 1 ) − c t ) =

Original: Tr( Xg t g Tt ) + λ t ( − log det (cid:0) XX − 1 t − 1 (cid:1) +
Revised: Tr( Xg t g Tt ) + λ t ( − log det (cid:0) XX − 1 t − 1 (cid:1) +

Original: Tr( XX − 1 t − 1 ) − n )) .
Revised: Tr( XX − 1 t − 1 ) − n )) .

Original: Setting ∇L ( X, λ t )
Revised: Setting ∇L ( X, λ t )

Original: = 0 , and using the fact that ∇ log det ( X ) =
Revised: = 0 , and using the fact that ∇ log det ( X ) =

Original: X − 1 we get the following update rule:
Revised: X − 1 we get the following update rule:

Original: [Equation]
Revised: [Equation]

Original: Note that setting c t = 

Revised: Note that setting c t =

Original: (equivalently λ t = ∞ ) ∀ t ∈ [ n ] in (4) results in no change to the preconditioner in any round.
Revised: (equivalently λ t = ∞ ) ∀ t ∈ [ n ] in (4) results in no change to the preconditioner in any round.

Original: In this case, with X 0 = I n , we get online gradient descent (Zinkevich, 2003).
Revised: In this case, with X 0 = I n , we get online gradient descent (Zinkevich, 2003).

Original: On the other hand, setting λ t = 1 gives the update rule of the online Newton method Hazan et al.
Revised: On the other hand, setting λ t = 1 gives the update rule of the online Newton method Hazan et al.

Original: Our update rule differs from (full-matrix) Adagrad (Duchi et al., 2011b) which has
Revised: Our update rule differs from (full-matrix) Adagrad (Duchi et al., 2011b) which has

Original: X − 2 t =
Revised: X − 2 t =

Original: X − 2 t − 1 + g t g Tt .
Revised: X − 2 t − 1 + g t g Tt .

Original: Maintaining and updating X t as in (5) is possible by using Sherman-Morrison formula but requires
Revised: Maintaining and updating X t as in (5) is possible by using Sherman-Morrison formula but requires

Original: O ( n 2 ) storage and time complexity.
Revised: O ( n 2 ) storage and time complexity.

Original: This becomes impractical when n is in the order of millions which is typically the case in DNNs.
Revised: This becomes impractical when n is in the order of millions which is typically the case in DNNs.

Original: Inputs : λ t := coefficient in the update (5), G := sparsity graph (banded/tridiagonal), ϵ :
Revised: Inputs : λ t := coefficient in the update (5), G := sparsity graph (banded/tridiagonal), ϵ :

Original: = damping parameter, T :=
Revised: = damping parameter, T :=

Original: total number of iterations/mini-batches, η t := step size/learning rate.
Revised: total number of iterations/mini-batches, η t := step size/learning rate.

Original: Output : w T + 1: H 0 = ϵI d , w 1 = 0 2: for t ∈ { 1 , . . .
Revised: Output : w T + 1: H 0 = ϵI d , w 1 = 0 2: for t ∈ { 1 , . . .

Original: , T } do 3: compute g t = ∇ f t ( w t ) 4:
Revised: , T } do 3: compute g t = ∇ f t ( w t ) 4:

Original: H t :=
Revised: H t :=

Original: H t − 1 + P G ( g t g
Revised: H t − 1 + P G ( g t g

Original: Tt /λ t ) ∈ S n ( G ) with P
Revised: Tt /λ t ) ∈ S n ( G ) with P

Original: G as in (8).
Revised: G as in (8).

Original: ▷ O ( n ) time & memory 5:
Revised: ▷ O ( n ) time & memory 5:

Original: Get L, D = S
Revised: Get L, D = S

Original: PARSIFIED
Revised: PARSIFIED

Original: I NVERSE ( H t , G ) , where X t =
Revised: I NVERSE ( H t , G ) , where X t =

Original: LDL T solves (11).
Revised: LDL T solves (11).

Original: Compute update u t =
Revised: Compute update u t =

Original: LDL T g t , 7: w t +1 =
Revised: LDL T g t , 7: w t +1 =

Original: w t − η t u t 8: end for 9: return w T +
Revised: w t − η t u t 8: end for 9: return w T +

Original: ( n
Revised: ( n

Original: Inputs : H ∈ S n ( G ) , is as (10).
Revised: Inputs : H ∈ S n ( G ) , is as (10).

Original: G :=
Revised: G :=

Original: the banded graph of band size b ≪ n
Revised: the banded graph of band size b ≪ n

Original: Outputs : lower triangular banded L ∈ R n × n and diagonal matrix D ∈ R n × n 1: function S PARSIFIED
Revised: Outputs : lower triangular banded L ∈ R n × n and diagonal matrix D ∈ R n × n 1: function S PARSIFIED

Original: I NVERSE ( H , G ) 2:
Revised: I NVERSE ( H , G ) 2:

Original: L := 0 , D :=3:
Revised: L := 0 , D :=3:

Original: L jj := 1 , ∀ j ∈ [ n ] 4: for j ∈ { 1 , . . . , n } do ▷ parallelizable 5:
Revised: L jj := 1 , ∀ j ∈ [ n ] 4: for j ∈ { 1 , . . . , n } do ▷ parallelizable 5:

Original: Let H jI j and H I j I j be defined as in Section 2, where I j = { j + 1 , . . .
Revised: Let H jI j and H I j I j be defined as in Section 2, where I j = { j + 1 , . . .

Original: , j + b } ∩ [ n ] , 6:
Revised: , j + b } ∩ [ n ] , 6:

Original: Solve for L I j j in the linear system
Revised: Solve for L I j j in the linear system

Original: H
Revised: H

Original: I j I j L I j j = − H
Revised: I j I j L I j j = − H

Original: I j j ▷ O ( b 3 ) time.
Revised: I j j ▷ O ( b 3 ) time.

Original: D jj := 1 / ( H jj + H TI j j L I j j ) 8: end for 9: return L, D 10: end function
Revised: D jj := 1 / ( H jj + H TI j j L I j j ) 8: end for 9: return L, D 10: end function

Original: 3. S PARSIFYING THE P RECONDITIONER
Revised: 3. S PARSIFYING THE P RECONDITIONER

Original: To reduce the memory required to maintain and update X t using (5), we consider the following general problem: find sparse X ≻ 0 with ∥ X ∥ 0 ≤ αn , α > 1 , such that the objective
Revised: To reduce the memory required to maintain and update X t using (5), we consider the following general problem: find sparse X ≻ 0 with ∥ X ∥ 0 ≤ αn , α > 1 , such that the objective

Original: D ℓ d ( X, ( X − 1 t − 1 + g t
Revised: D ℓ d ( X, ( X − 1 t − 1 + g t

Original: g
Revised: g

Original: Tt /λ t ) − 1 ) is minimized.
Revised: Tt /λ t ) − 1 ) is minimized.

Original: Due to the L 0 -norm constraint, this is a non-convex problem, which makes it difficult to solve exactly.
Revised: Due to the L 0 -norm constraint, this is a non-convex problem, which makes it difficult to solve exactly.

Original: Since L 1 -norm serves as a convex relaxation for the L 0 norm, we could use it instead, resulting in the following optimization problem also known as graphical lasso estimator (Friedman et al., 2008):
Revised: Since L 1 -norm serves as a convex relaxation for the L 0 norm, we could use it instead, resulting in the following optimization problem also known as graphical lasso estimator (Friedman et al., 2008):

Original: [Equation]
Revised: [Equation]

Original: The sparsity introduced by L 1 -norm penalty will reduce the memory usage.
Revised: The sparsity introduced by L 1 -norm penalty will reduce the memory usage.

Original: However, the time taken to solve the above problem, even with the current best methods (Bollh ¨ ofer et al., 2019; Hsieh et al., 2013; Fattahi & Sojoudi, 2019; Zhang et al., 2018), can still be too large (as these methods take several minutes for a matrix of size million), making it impractical to embed in DNN training since preconditioning may need to be done after processing every mini-batch.
Revised: However, the time taken to solve the above problem, even with the current best methods (Bollh ¨ ofer et al., 2019; Hsieh et al., 2013; Fattahi & Sojoudi, 2019; Zhang et al., 2018), can still be too large (as these methods take several minutes for a matrix of size million), making it impractical to embed in DNN training since preconditioning may need to be done after processing every mini-batch.

Original: In this paper, we take a different direction where we use fixed sparsity pattern constraints, specified by a fixed undirected graph G .
Revised: In this paper, we take a different direction where we use fixed sparsity pattern constraints, specified by a fixed undirected graph G .

Original: To sparsify the solution in (5), we formulate the subproblem
Revised: To sparsify the solution in (5), we formulate the subproblem

Original: [Equation]
Revised: [Equation]

Original: (6) where S n ( G ) ++ denotes the set of positive definite matrices with the fixed sparsity pattern given by graph G .
Revised: (6) where S n ( G ) ++ denotes the set of positive definite matrices with the fixed sparsity pattern given by graph G .

Original: Note that even for sparsification we use the LogDet measure; thus both the steps (4) and (6) use the same measure.
Revised: Note that even for sparsification we use the LogDet measure; thus both the steps (4) and (6) use the same measure.

Original: Algorithm 1 presents one proposed SONew method, which solves (6) using O ( n ) time and memory for banded matrices with band size b , in particular a tridiagonal matrix, corresponding to a chain graph, is a banded matrix with bandsize 1.
Revised: Algorithm 1 presents one proposed SONew method, which solves (6) using O ( n ) time and memory for banded matrices with band size b , in particular a tridiagonal matrix, corresponding to a chain graph, is a banded matrix with bandsize 1.

Original: Maintaining H t ∈ S n ( G ) in line 4 .
Revised: Maintaining H t ∈ S n ( G ) in line 4 .

Original: Solving the subproblem in (6) naively is impractical since
Revised: Solving the subproblem in (6) naively is impractical since

Original: X − 1 t − 1 is a dense matrix.
Revised: X − 1 t − 1 is a dense matrix.

Original: However, the structure of the LogDet divergence comes to the rescue; (6) can be expanded as follows:
Revised: However, the structure of the LogDet divergence comes to the rescue; (6) can be expanded as follows:

Original: [Equation]
Revised: [Equation]

Original: Let us define the projection onto S n ( G ) , P G : R n × n → R n × n as:
Revised: Let us define the projection onto S n ( G ) , P G : R n × n → R n × n as:

Original: [Equation]
Revised: [Equation]

Original: Note that the Tr( . ) term in (7) involves only the non-zero elements of X ∈ S n ( G )
Revised: Note that the Tr( . ) term in (7) involves only the non-zero elements of X ∈ S n ( G )

Original: +
Revised: +

Original: + .
Revised: + .

Original: Hence, (7) can be written as
Revised: Hence, (7) can be written as

Original: [Equation]
Revised: [Equation]

Original: Computing matrix X − 1 t − 1 can be avoided by analyzing optimality condition of (9).
Revised: Computing matrix X − 1 t − 1 can be avoided by analyzing optimality condition of (9).

Original: Let g ( X ) be the objective function in (9), then the optimality condition of (9) is P G ( ∇ g ( X ))
Revised: Let g ( X ) be the objective function in (9), then the optimality condition of (9) is P G ( ∇ g ( X ))

Original: = 0 .
Revised: = 0 .

Original: Expanding g ( X ) gives
Revised: Expanding g ( X ) gives

Original: [Equation]
Revised: [Equation]

Original: Thus we only need to maintain H t ∈ S n ( G ) , which is H t =
Revised: Thus we only need to maintain H t ∈ S n ( G ) , which is H t =

Original: P G ( X − 1 t ) .
Revised: P G ( X − 1 t ) .

Original: This matrix is updated as
Revised: This matrix is updated as

Original: H t =
Revised: H t =

Original: H t − 1 + P G ( g t
Revised: H t − 1 + P G ( g t

Original: g
Revised: g

Original: Tt /λ t ) , which can be done in O ( | E G | ) memory and time, while computing the matrix X − 1 t would have cost O ( n 2 ) .
Revised: Tt /λ t ) , which can be done in O ( | E G | ) memory and time, while computing the matrix X − 1 t would have cost O ( n 2 ) .

Original: In SONew, this key observation is used to maintain H t in line 4.
Revised: In SONew, this key observation is used to maintain H t in line 4.

Original: Computing X t in line 5 .
Revised: Computing X t in line 5 .

Original: Now that H t is known at every round t , we can replace P G ( X − 1 t − 1 + g t g T t /λ t ) in (9) with H t as:
Revised: Now that H t is known at every round t , we can replace P G ( X − 1 t − 1 + g t g T t /λ t ) in (9) with H t as:

Original: [Equation]
Revised: [Equation]

Original: For an arbitrary graph G , solving this subproblem might be difficult.
Revised: For an arbitrary graph G , solving this subproblem might be difficult.

Original: Theorems 1 and 2 show embarrassingly parallelizable explicit solutions to the subproblem 11 for tridiagonal and banded sparsity patterns.
Revised: Theorems 1 and 2 show embarrassingly parallelizable explicit solutions to the subproblem 11 for tridiagonal and banded sparsity patterns.

Original: Proofs of Theorems 1 and 2 are given in Appendix A.1.
Revised: Proofs of Theorems 1 and 2 are given in Appendix A.1.

Original: Theorem 1 (Explicit solution of (11) for tridiagonal matrix/chain graph) .
Revised: Theorem 1 (Explicit solution of (11) for tridiagonal matrix/chain graph) .

Original: Let the sparsity structure
Revised: Let the sparsity structure

Original: G be a chain with edges E G = { ( j, j + 1) : j ∈ [ n − 1] } , and let H ∈ S n ( G ) , then the solution of (11) is given by ˆ X
Revised: G be a chain with edges E G = { ( j, j + 1) : j ∈ [ n − 1] } , and let H ∈ S n ( G ) , then the solution of (11) is given by ˆ X

Original: = LDL T , where the unit lower triangular matrix L and diagonal matrix D have the following non-zero entries:
Revised: = LDL T , where the unit lower triangular matrix L and diagonal matrix D have the following non-zero entries:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The time and memory complexity required to compute (12) is O ( n ) .
Revised: The time and memory complexity required to compute (12) is O ( n ) .

Original: Note that the computation of (12) can be easily parallelized.
Revised: Note that the computation of (12) can be easily parallelized.

Original: This explicit solution can be generalized to banded sparsity structures with band size b .
Revised: This explicit solution can be generalized to banded sparsity structures with band size b .

Original: Theorem 2 (Explicit solution of (11) for banded matrices) .
Revised: Theorem 2 (Explicit solution of (11) for banded matrices) .

Original: Let the sparsity pattern G be a banded matrix of band size b, i.e. for every vertex j , let I j
Revised: Let the sparsity pattern G be a banded matrix of band size b, i.e. for every vertex j , let I j

Original: = { j + 1 , . . .
Revised: = { j + 1 , . . .

Original: , j + b } , then edges
Revised: , j + b } , then edges

Original: E G =
Revised: E G =

Original: ( (cid:83) nj =1 { j } × I j ) ∩ { ( i, j ) : i ≤ n, j ≤ n } .
Revised: ( (cid:83) nj =1 { j } × I j ) ∩ { ( i, j ) : i ≤ n, j ≤ n } .

Original: Then X t =
Revised: Then X t =

Original: LDL T is the solution of (11) with nonzero entries of L and D defined as follows :
Revised: LDL T is the solution of (11) with nonzero entries of L and D defined as follows :

Original: [Equation]
Revised: [Equation]

Original: Finding the above solution requires solving n linear systems of size b (which is small) as shown in
Revised: Finding the above solution requires solving n linear systems of size b (which is small) as shown in

Original: Algorithm 2, and takes O (( n − b + 1) b 3 ) flops.
Revised: Algorithm 2, and takes O (( n − b + 1) b 3 ) flops.

Original: Since b ≪ n , the number of flops is O ( n ) .
Revised: Since b ≪ n , the number of flops is O ( n ) .

Original: 3. A NALYSIS OF SON EW
Revised: 3. A NALYSIS OF SON EW

Original: The following theorem establishes regret guarantees of SONew in online convex optimization framework mentioned in Section 3.1.
Revised: The following theorem establishes regret guarantees of SONew in online convex optimization framework mentioned in Section 3.1.

Original: Theorem 3.
Revised: Theorem 3.

Original: When G = tridiagona/chain graph
Revised: When G = tridiagona/chain graph

Original: [Equation]
Revised: [Equation]

Original: then Algorithm 1 incurs the following regret under convex losses f t
Revised: then Algorithm 1 incurs the following regret under convex losses f t

Original: [Equation]
Revised: [Equation]

Original: where g ( i ) 1: T = [( g 1 ) i , . . .
Revised: where g ( i ) 1: T = [( g 1 ) i , . . .

Original: ( g T )
Revised: ( g T )

Original: i ] denotes gradient history of i th variable/parameter , G ( i ) ∞
Revised: i ] denotes gradient history of i th variable/parameter , G ( i ) ∞

Original: =
Revised: =

Original: ∥ g ( i ) 1: T ∥ ∞ ,
Revised: ∥ g ( i ) 1: T ∥ ∞ ,

Original: C = max t ( (cid:80) ni =1 ( w t − w ∗ ) 2 i ( G ( i ) ∞ ) 2 ) , β i = (cid:68) g
Revised: C = max t ( (cid:80) ni =1 ( w t − w ∗ ) 2 i ( G ( i ) ∞ ) 2 ) , β i = (cid:68) g

Original: ( i ) 1: T ,g ( i +1) 1: T (cid:69) (cid:114)
Revised: ( i ) 1: T ,g ( i +1) 1: T (cid:69) (cid:114)

Original: ( ϵ + (cid:13)(cid:13)(cid:13) g
Revised: ( ϵ + (cid:13)(cid:13)(cid:13) g

Original: ( i ) 1: T (cid:13)(cid:13)(cid:13) 2 2 ) · ( ϵ + (cid:13)(cid:13)(cid:13) g
Revised: ( i ) 1: T (cid:13)(cid:13)(cid:13) 2 2 ) · ( ϵ + (cid:13)(cid:13)(cid:13) g

Original: ( i +1) 1: T (cid:13)(cid:13)(cid:13) 2 2 ) denotes dot-product between gradient histories of connected parameters in the chain graph,
Revised: ( i +1) 1: T (cid:13)(cid:13)(cid:13) 2 2 ) denotes dot-product between gradient histories of connected parameters in the chain graph,

Original: β = max i ∈ [ n − 1] β i .
Revised: β = max i ∈ [ n − 1] β i .

Original: SONew is scale-invariant to diagonal transformation of gradients; if the gradients sent as feedback to the OCO learner are ¯ g t =
Revised: SONew is scale-invariant to diagonal transformation of gradients; if the gradients sent as feedback to the OCO learner are ¯ g t =

Original: Λ g t , then the iterates for the transformed problem will be ¯ x t = Λ − 1 w t = Λ − 1 x t − 1 − η
Revised: Λ g t , then the iterates for the transformed problem will be ¯ x t = Λ − 1 w t = Λ − 1 x t − 1 − η

Original: (Λ − 1 X t − 1 Λ − 1 )Λ( g t − 1 ) , this is due to scale-invariance property of LogDet divergence in Section 2.1,i.e, the transformed preconditioner is ¯ X t =
Revised: (Λ − 1 X t − 1 Λ − 1 )Λ( g t − 1 ) , this is due to scale-invariance property of LogDet divergence in Section 2.1,i.e, the transformed preconditioner is ¯ X t =

Original: Λ − 1 X t − 1 Λ − 1 .
Revised: Λ − 1 X t − 1 Λ − 1 .

Original: Furthermore, the regret bound derived in Theorem 3 for transformed and original problem is approximately same.
Revised: Furthermore, the regret bound derived in Theorem 3 for transformed and original problem is approximately same.

Original: Our regret bound is data-dependent, since if β i is nearer to 1 , then the regret is smaller due to the log(1 − β 2 i ) term.
Revised: Our regret bound is data-dependent, since if β i is nearer to 1 , then the regret is smaller due to the log(1 − β 2 i ) term.

Original: This effect is more amplified when log(

Revised: This effect is more amplified when log(

Original: + ∥ g ( i ) 1: T ∥ 22 /ϵ ) is relatively low compared to log(1 −
Revised: + ∥ g ( i ) 1: T ∥ 22 /ϵ ) is relatively low compared to log(1 −

Original: β 2 i ) .
Revised: β 2 i ) .

Original: The proof for Theorem 3 is given in Appendix A.2.
Revised: The proof for Theorem 3 is given in Appendix A.2.

Original: We also derive a regret bound with O ( √ κT 1 / 2 ) dependency on T in Appendix A.3, where the condition number κ (diag( H t )) ≤ κ .
Revised: We also derive a regret bound with O ( √ κT 1 / 2 ) dependency on T in Appendix A.3, where the condition number κ (diag( H t )) ≤ κ .

Original:  N UMERICAL S TABILITY OF SON EW
Revised:  N UMERICAL S TABILITY OF SON EW

Original: The matrix H ∈ S n ( G ) given as input to Algorithm 2 should have positive definite submatrices
Revised: The matrix H ∈ S n ( G ) given as input to Algorithm 2 should have positive definite submatrices

Original: H J j J j , J j = I j ∪ { j } , j ∈ [ n ] , where I j is as in Theorem 2, hence, it should have positive schur complements
Revised: H J j J j , J j = I j ∪ { j } , j ∈ [ n ] , where I j is as in Theorem 2, hence, it should have positive schur complements

Original: ( H jj − H TI j j H − 

Revised: ( H jj − H TI j j H −

Original: I j I j H I j j ) .
Revised: I j I j H I j j ) .

Original: But, if the matrices H J j J j and H I j I j are illconditioned, then the computed value for ( H jj + H TI j j L I j j ) =
Revised: But, if the matrices H J j J j and H I j I j are illconditioned, then the computed value for ( H jj + H TI j j L I j j ) =

Original: ( H jj − H TI j j H − 

Revised: ( H jj − H TI j j H −

Original: I j I j H I j j ) , in line 7 of Algorithm 2, can be zero or negative due to catastrophic cancellation, since finding L I j j = − H − 

Revised: I j I j H I j j ) , in line 7 of Algorithm 2, can be zero or negative due to catastrophic cancellation, since finding L I j j = − H −

Original: I j I j H I j j in floating point arithmetic can result in rounding errors.
Revised: I j I j H I j j in floating point arithmetic can result in rounding errors.

Original: To understand this issue further, we conduct perturbation analysis in Theorem 4 which establishes a componentwise condition number (Higham, 2002) of the optimization problem in (11) with a tridiagonal sparsity structure G .
Revised: To understand this issue further, we conduct perturbation analysis in Theorem 4 which establishes a componentwise condition number (Higham, 2002) of the optimization problem in (11) with a tridiagonal sparsity structure G .

Original: Theorem 4 (Condition number of tridiagonal LogDet subproblem 11) .
Revised: Theorem 4 (Condition number of tridiagonal LogDet subproblem 11) .

Original: Let H ∈ S ++ n be such that
Revised: Let H ∈ S ++ n be such that

Original: H ii = 1 for i ∈ [ n ] .
Revised: H ii = 1 for i ∈ [ n ] .

Original: Let ∆ H be a symmetric perturbation such that ∆ H ii = 0 for i ∈ [ n ] , and
Revised: Let ∆ H be a symmetric perturbation such that ∆ H ii = 0 for i ∈ [ n ] , and

Original: H + ∆ H ∈ S ++ n .
Revised: H + ∆ H ∈ S ++ n .

Original: Let P G ( H ) be the input to 11, where G is a chain graph, then
Revised: Let P G ( H ) be the input to 11, where G is a chain graph, then

Original: [Equation]
Revised: [Equation]

Original: (14) where, β i =
Revised: (14) where, β i =

Original: H ii +1 , κ ℓd ∞ := componentwise condition number of (11) for perturbation ∆ H .
Revised: H ii +1 , κ ℓd ∞ := componentwise condition number of (11) for perturbation ∆ H .

Original: So, the tridiagonal LogDet problem with inputs H as mentioned in Theorem 4, has high condition number when 1 − β 2 i
Revised: So, the tridiagonal LogDet problem with inputs H as mentioned in Theorem 4, has high condition number when 1 − β 2 i

Original: = H ii − H 2 ii
Revised: = H ii − H 2 ii

Original: +1 /H
Revised: +1 /H

Original: i +

Revised: i +

Original: i +1 are low and as a result the preconditioner X t in SONew (Algorithm 1) has high componentwise relative errors.
Revised: i +1 are low and as a result the preconditioner X t in SONew (Algorithm 1) has high componentwise relative errors.

Original: In SONew (1), the
Revised: In SONew (1), the

Original: H t =
Revised: H t =

Original: P G ( (cid:80) ts =

Revised: P G ( (cid:80) ts =

Original: g s g
Revised: g s g

Original: Ts /λ t ) generated in line 4 could be such that the matrix (cid:80) ts =
Revised: Ts /λ t ) generated in line 4 could be such that the matrix (cid:80) ts =

Original: 1 g s g
Revised: 1 g s g

Original: Ts /λ t need not be positive definite and so the schur complements H ii
Revised: Ts /λ t need not be positive definite and so the schur complements H ii

Original: − H 2 ii +1 /H
Revised: − H 2 ii +1 /H

Original: i +

Revised: i +

Original: i +1 can be zero, giving an infinite condition number κ ℓd ∞ by Theorem 4.
Revised: i +1 can be zero, giving an infinite condition number κ ℓd ∞ by Theorem 4.

Original: The following lemma describes such cases in detail for a more general banded sparsity structure case.
Revised: The following lemma describes such cases in detail for a more general banded sparsity structure case.

Original: Lemma 1 (Degenerate inputs to banded LogDet subproblem) .
Revised: Lemma 1 (Degenerate inputs to banded LogDet subproblem) .

Original: Let H
Revised: Let H

Original: = P G ( GG T ) , where G ∈
Revised: = P G ( GG T ) , where G ∈

Original: R n × T and let g ( i ) 1: T be i th row of G , which is gradients of parameter i for T rounds, then H ij = (cid:68) g
Revised: R n × T and let g ( i ) 1: T be i th row of G , which is gradients of parameter i for T rounds, then H ij = (cid:68) g

Original: ( i ) 1: T
Revised: ( i ) 1: T

Original: g ( j ) 1: T (cid:69) .
Revised: g ( j ) 1: T (cid:69) .

Original: • Case 1: For tridiagonal sparsity structure G : if g ( j ) 1: T = g ( j +1) 1: T , then H jj − H 2 jj +1 /H
Revised: • Case 1: For tridiagonal sparsity structure G : if g ( j ) 1: T = g ( j +1) 1: T , then H jj − H 2 jj +1 /H

Original: j +

Revised: j +

Original: j +1 = 0 .
Revised: j +1 = 0 .

Original: • Case 2: For b > 1 in (13): If rank( H J j J j )
Revised: • Case 2: For b > 1 in (13): If rank( H J j J j )

Original: = rank
Revised: = rank

Original: ( H I j I j )
Revised: ( H I j I j )

Original: =
Revised: =

Original: b , then ( H jj − H T I j j H − 

Revised: b , then ( H jj − H T I j j H −

Original: I j I j H I j j ) = 0 and D jj = ∞ .
Revised: I j I j H I j j ) = 0 and D jj = ∞ .

Original: If rank( H I j I j ) <
Revised: If rank( H I j I j ) <

Original: b then the inverse H − 1 I j I j doesn’t exist and D jj is not well-defined.
Revised: b then the inverse H − 1 I j I j doesn’t exist and D jj is not well-defined.

Original: If GG T = (cid:80) Ti =

Revised: If GG T = (cid:80) Ti =

Original: g
Revised: g

Original: i g
Revised: i g

Original: i is a singular matrix, then solution to the LogDet problem might not be welldefined as shown in Lemma 1.
Revised: i is a singular matrix, then solution to the LogDet problem might not be welldefined as shown in Lemma 1.

Original: For instance, Case 1 can occur when preconditioning the input layer of an image-based DNN with flattened image inputs, where j th and ( j + 1) th pixel can be highly correlated throughout the dataset.
Revised: For instance, Case 1 can occur when preconditioning the input layer of an image-based DNN with flattened image inputs, where j th and ( j + 1) th pixel can be highly correlated throughout the dataset.

Original: Case 2 can occur in the first b iterations in Algorithm 1 when the rank of submatrices rank( H I j I j )
Revised: Case 2 can occur in the first b iterations in Algorithm 1 when the rank of submatrices rank( H I j I j )

Original: < b and ϵ = 0 .
Revised: < b and ϵ = 0 .

Original: We develop Algorithm 3 which is robust to degenerate inputs H , given that H ii > 0 .
Revised: We develop Algorithm 3 which is robust to degenerate inputs H , given that H ii > 0 .

Original: It finds a subgraph ˜ G of G for which (13) is well-defined.
Revised: It finds a subgraph ˜ G of G for which (13) is well-defined.

Original: This is done by removing edges which causes inverse H − 1 I j I j to be singular or ( H jj − H TI j j H − 1 I j I j H I j j ) to be low.
Revised: This is done by removing edges which causes inverse H − 1 I j I j to be singular or ( H jj − H TI j j H − 1 I j I j H I j j ) to be low.

Original: Our ablation study in Table 4 demonstrates noticeable improvement in performance
Revised: Our ablation study in Table 4 demonstrates noticeable improvement in performance

Original: Algorithm 3 is used.
Revised: Algorithm 3 is used.

Original: Theorem 5 (Numerically stable algorithm) .
Revised: Theorem 5 (Numerically stable algorithm) .

Original: Algorithm 3 finds a subgraph ˜ G of G , such that explicit solution for ˜ G in (13) is well-defined.
Revised: Algorithm 3 finds a subgraph ˜ G of G , such that explicit solution for ˜ G in (13) is well-defined.

Original: Furthermore, when G is a tridiagonal/chain graph, the component-wise condition number upperbound in (14) is reduced upon using Algorithm 3, ˆ κ ˜ G ℓd <
Revised: Furthermore, when G is a tridiagonal/chain graph, the component-wise condition number upperbound in (14) is reduced upon using Algorithm 3, ˆ κ ˜ G ℓd <

Original: ˆ κ G ℓd , where ˆ κ ˜ G ℓd , ˆ κ G ℓd are defined as in Theorem 4 for graphs ˜ G and G respectively.
Revised: ˆ κ G ℓd , where ˆ κ ˜ G ℓd , ˆ κ G ℓd are defined as in Theorem 4 for graphs ˜ G and G respectively.

Original: The proofs for Lemma 1 and Theorem 5 are given in Appendix A.4.
Revised: The proofs for Lemma 1 and Theorem 5 are given in Appendix A.4.

Original: neighbourhood
Revised: neighbourhood

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Online Newton method is a second order method in online convex optimization framework with properties such as scale invariance (Luo et al., 2016) and logarithmic regrets in exp-concave and strongly convex functions (Hazan et al., 2007; 2016).
Revised: Online Newton method is a second order method in online convex optimization framework with properties such as scale invariance (Luo et al., 2016) and logarithmic regrets in exp-concave and strongly convex functions (Hazan et al., 2007; 2016).

Original: However, it has a time complexity of O ( n 2 ) , making it infeasible for large n .
Revised: However, it has a time complexity of O ( n 2 ) , making it infeasible for large n .

Original: A diagonal version of this method SC-Adagrad (Mukkamala &
Revised: A diagonal version of this method SC-Adagrad (Mukkamala &

Original: Hein, 2017) was proposed to make the online Newton method scalable for deep-learning.
Revised: Hein, 2017) was proposed to make the online Newton method scalable for deep-learning.

Original: SCAdagrad is equivalent to setting the sparsity pattern G in equation 11 as a null graph, furthermore, setting G as a complete graph will result in online Newton method.
Revised: SCAdagrad is equivalent to setting the sparsity pattern G in equation 11 as a null graph, furthermore, setting G as a complete graph will result in online Newton method.

Original: However, introduction of LogDet divergence measure in SONew allows us to set different sparsity graphs as G such as banded graph with band-size b , for which our preconditioning process is more computationally efficient with a time complexity of O ( b 3 ( n − b + 1)) compared to online-newton method O ( n 2 ) .
Revised: However, introduction of LogDet divergence measure in SONew allows us to set different sparsity graphs as G such as banded graph with band-size b , for which our preconditioning process is more computationally efficient with a time complexity of O ( b 3 ( n − b + 1)) compared to online-newton method O ( n 2 ) .

Original: As discussed in
Revised: As discussed in

Original: Theorem 3, SONew also utilizes correlation among gradients of parameters to improve convergence, in contrast to diagonal preconditioners such as SC-Adagrad.
Revised: Theorem 3, SONew also utilizes correlation among gradients of parameters to improve convergence, in contrast to diagonal preconditioners such as SC-Adagrad.

Original: Shampoo (Gupta et al., 2018; Anil et al., 2020) uses Kronecker factored preconditioners to reduce the memory and time complexity from O ( n 2 ) to O ( d 21 + d 22 ) and O ( d 31 + d 32 ) respectively, where n = d 1 d 2 denotes number of parameters for a linear layer of dimensions d 1 × d 2 .
Revised: Shampoo (Gupta et al., 2018; Anil et al., 2020) uses Kronecker factored preconditioners to reduce the memory and time complexity from O ( n 2 ) to O ( d 21 + d 22 ) and O ( d 31 + d 32 ) respectively, where n = d 1 d 2 denotes number of parameters for a linear layer of dimensions d 1 × d 2 .

Original: The time complexity of matrix inversion takes a heavy toll in Shampoo’s compute time even with the Kronecker product assumption on the preconditioner, whereas, our method has a time complexity of O ( b 3 d 1 d 2 ) quadratic in dimensions of the linear layer (note that b = 1 for tridiagonal structure).
Revised: The time complexity of matrix inversion takes a heavy toll in Shampoo’s compute time even with the Kronecker product assumption on the preconditioner, whereas, our method has a time complexity of O ( b 3 d 1 d 2 ) quadratic in dimensions of the linear layer (note that b = 1 for tridiagonal structure).

Original: Furthermore, precision lost in matrix inversion can be large due to high condition number matrices occurring in training (Anil et al., 2022; 2020).
Revised: Furthermore, precision lost in matrix inversion can be large due to high condition number matrices occurring in training (Anil et al., 2022; 2020).

Original: LogDet problem in equation 11 is closely related to the Maximum Determinant Matrix Completion (MDMC)
Revised: LogDet problem in equation 11 is closely related to the Maximum Determinant Matrix Completion (MDMC)

Original: (Andersen et al., 2013; Vandenberghe et al., 2015).
Revised: (Andersen et al., 2013; Vandenberghe et al., 2015).

Original: The MDMC problem is the dual of LogDet problem (11), and has explicit solutions for chordal graphs (Andersen et al., 2013).
Revised: The MDMC problem is the dual of LogDet problem (11), and has explicit solutions for chordal graphs (Andersen et al., 2013).

Original: Thus the explicit solutions in (13) are the same as the ones proved in Andersen et al.
Revised: Thus the explicit solutions in (13) are the same as the ones proved in Andersen et al.

Original: Also, we noticed that the tridiagonal explicit solution has been used previously in KFAC (Martens & Grosse, 2015) in the context of a gaussian graphical model interpretation of gradients.
Revised: Also, we noticed that the tridiagonal explicit solution has been used previously in KFAC (Martens & Grosse, 2015) in the context of a gaussian graphical model interpretation of gradients.

Original: In this paper we also analyze conditioning and degenerate cases of these explicit solutions (which can occur frequently in
Revised: In this paper we also analyze conditioning and degenerate cases of these explicit solutions (which can occur frequently in

Original: DNN training)
Revised: DNN training)

Original: and develop algorithms to avoid such cases.
Revised: and develop algorithms to avoid such cases.

Original: In addition, we have regret guarantees connecting these explicit solutions to regret in online convex optimization problem setup.
Revised: In addition, we have regret guarantees connecting these explicit solutions to regret in online convex optimization problem setup.

Original: There is prior work (Luo et al., 2016; 2019) in reducing the complexity - O ( n 2 ) flops of Online Newton Step (ONS) to O ( n ) flops using sketching.
Revised: There is prior work (Luo et al., 2016; 2019) in reducing the complexity - O ( n 2 ) flops of Online Newton Step (ONS) to O ( n ) flops using sketching.

Original: These ONS variants maintain a low rank approximation of
Revised: These ONS variants maintain a low rank approximation of

Original: H t (as in Algorithm 1) and updating it with a new gradient g t at every iteration requires conducting SVD (Luo et al.
Revised: H t (as in Algorithm 1) and updating it with a new gradient g t at every iteration requires conducting SVD (Luo et al.

Original: , 2019)/orthonormalization (Luo et al., 2016) of a tall and thin matrix in R n × r , where r denotes the rank of approximation of H t .
Revised: , 2019)/orthonormalization (Luo et al., 2016) of a tall and thin matrix in R n × r , where r denotes the rank of approximation of H t .

Original: Our proposed method (Algorithm 1) has a more parallelizable update H t :
Revised: Our proposed method (Algorithm 1) has a more parallelizable update H t :

Original: = H t − 1 + P G ( g t g Tt ) making it more suitable for DNN training.
Revised: = H t − 1 + P G ( g t g Tt ) making it more suitable for DNN training.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We conduct experiments onthe Autoencoder benchmark (Schmidhuber, 2015) using MNIST dataset (Deng, 2012).
Revised: In this section we describe our experiments on Autoencoder benchmark (Schmidhuber, 2015) using MNIST dataset (Deng, 2012).

Revised: Our results on larger benchmarks are given in Appendix A.5.

Original: We compare SONew against commonly used first order methods including SGD (Kiefer & Wolfowitz, 1952)), SGD with Momentum (Qian, 1999), Nesterov (Nesterov, 1983), Adagrad (Duchi et al., 2011a), Adam (Kingma & Ba, 2014), and Rmsprop (Tieleman & Hinton, 2012).
Revised: We compare SONew against commonly used first order methods including SGD (Kiefer & Wolfowitz, 1952)), SGD with Momentum (Qian, 1999), Nesterov (Nesterov, 1983), Adagrad (Duchi et al., 2011a), Adam (Kingma & Ba, 2014), and Rmsprop (Tieleman & Hinton, 2012).

Original: We also compare with Shampoo (Gupta et al., 2018), a state of the art second-order optimizer used in practice.
Revised: We also compare with Shampoo (Gupta et al., 2018), a state of the art second-order optimizer used in practice.

Original: Computing preconditioner at every step in shampoo could be infeasible, instead it is computed every t steps - referred to as Shampoo( t ) in the experiments.
Revised: Computing preconditioner at every step in shampoo could be infeasible, instead it is computed every t steps - referred to as Shampoo( t ) in the experiments.

Original: For SONew, we use exponentially moving average (EMA) for both first order and second order statistics.
Revised: For SONew, we use exponentially moving average (EMA) for both first order and second order statistics.

Original: EMA is commonly used in adaptive optimizers for deep-learning training Kingma & Ba (2014);
Revised: EMA is commonly used in adaptive optimizers for deep-learning training Kingma & Ba (2014);

Original: Qian (1999).
Revised: Qian (1999).

Original: Let β 1 , β 2 ∈ [0 , 1] be the coefficients for EMA and g t ∈ R n be the gradient at t th iteration/mini-batch.
Revised: Let β 1 , β 2 ∈ [0 , 1] be the coefficients for EMA and g t ∈ R n be the gradient at t th iteration/mini-batch.

Original: Let µ t ∈ R n be the first order gradient statistic, and H t ∈ S n ( G ) be the second order gradient statistic as in (10).
Revised: Let µ t ∈ R n be the first order gradient statistic, and H t ∈ S n ( G ) be the second order gradient statistic as in (10).

Original: Then, the following modification to update rules are made to in the SONew implementation.
Revised: Then, the following modification to update rules are made to in the SONew implementation.

Original: [Equation]
Revised: [Equation]

Original: Furthermore, µ t is used in w t +1 =
Revised: Furthermore, µ t is used in w t +1 =

Original: w t − η t X t µ t , replacing the gradient g t , similar to Kingma &
Revised: w t − η t X t µ t , replacing the gradient g t , similar to Kingma &

Original: Ba (2014).
Revised: Ba (2014).

Original: As discussed in Section 3.2 we need to store the values of H t only corresponding to the sparsity pattern, hence SONew uses O ( n ) space.
Revised: As discussed in Section 3.2 we need to store the values of H t only corresponding to the sparsity pattern, hence SONew uses O ( n ) space.

Original: The updates above get computed in parallel in
Revised: The updates above get computed in parallel in

Original: O (1) time.
Revised: O (1) time.

Original: Moreover, we use Algorithm 3 to make SONew numerically stable.
Revised: Moreover, we use Algorithm 3 to make SONew numerically stable.

Original: We also use grafting (Agarwal et al., 2022), a technique used to transfer step size between optimization algorithms.
Revised: We also use grafting (Agarwal et al., 2022), a technique used to transfer step size between optimization algorithms.

Original: Specifically, given an update v 1 of Optimizer-1 and v 2 of Optimizer-2, grafting allows to use the direction suggested by Optimizer-2 with step size suggested by Optimizer-1.
Revised: Specifically, given an update v 1 of Optimizer-1 and v 2 of Optimizer-2, grafting allows to use the direction suggested by Optimizer-2 with step size suggested by Optimizer-1.

Original: The final update is given by ∥ v 1 ∥ ∥ v 2 ∥ · v 2 .
Revised: The final update is given by ∥ v 1 ∥ ∥ v 2 ∥ · v 2 .

Original: Grafting has been shown to take advantage of a tuned optimizer step size and improve performance.
Revised: Grafting has been shown to take advantage of a tuned optimizer step size and improve performance.

Original: In our case, we use Adam grafting - using Adam optimizer step size ∥ v 1 ∥ with SONew direction v 2 / ∥ v 2 ∥ .
Revised: In our case, we use Adam grafting - using Adam optimizer step size ∥ v 1 ∥ with SONew direction v 2 / ∥ v 2 ∥ .

Original: We use three sparsity patterns for SONew - a) diagonal sparsity, resulting in a diagonal preconditioner similar to adaptive first order methods like Adam and Adagrad; b) tridiagonal sparsity, corresponding to a chain graph; and c) banded sparsity, represented by ”band- k ” in tables and figures for band size of k .
Revised: We use three sparsity patterns for SONew - a) diagonal sparsity, resulting in a diagonal preconditioner similar to adaptive first order methods like Adam and Adagrad; b) tridiagonal sparsity, corresponding to a chain graph; and c) banded sparsity, represented by ”band- k ” in tables and figures for band size of k .

Original: All the baselines and SONew are trained for 100 epochs and use the train set of MNIST dataset containing 60 k points.
Revised: All the baselines and SONew are trained for 100 epochs and use the train set of MNIST dataset containing 60 k points.

Original: We use a standard sized ( 2 . 72 M parameters)
Revised: We use a standard sized ( 2 . 72 M parameters)

Original: Autoencoder with layer sizes: [1000, 500, 250, 30, 250, 500, 1000] and tanh non-linearity.
Revised: Autoencoder with layer sizes: [1000, 500, 250, 30, 250, 500, 1000] and tanh non-linearity.

Original: Batch size is fixed at 1000 and we use learning rate schedule with a linear warmup of 5 epochs followed by linear decay towards 0 .
Revised: Batch size is fixed at 1000 and we use learning rate schedule with a linear warmup of 5 epochs followed by linear decay towards 0 .

Original: Over 2 k hyperparameters are searched over for each experiment using a Bayesian optimization package.
Revised: Over 2 k hyperparameters are searched over for each experiment using a Bayesian optimization package.

Original: The search space for each optimizer is mentioned in Appendix A.5.
Revised: The search space for each optimizer is mentioned in Appendix A.5.

Original: From the float32 experiments in Table 1 we observe that among first order methods, diag-SONew performs the best while taking same amount of time.
Revised: From the float32 experiments in Table 1 we observe that among first order methods, diag-SONew performs the best while taking same amount of time.

Original: Increasing the number of edges in the sparsity graph to tridiag or banded with band size 4 enhances the performance further.
Revised: Increasing the number of edges in the sparsity graph to tridiag or banded with band size 4 enhances the performance further.

Original: Tridiag-SONew performs 4 × faster than Shampoo at a marginal cost to the loss - even when Shampoo updates preconditioner once every 20 steps.
Revised: Tridiag-SONew performs 4 × faster than Shampoo at a marginal cost to the loss - even when Shampoo updates preconditioner once every 20 steps.

Original: When dealing with large scale models like Resnet50 (He et al., 2015), Shampoo requires updating preconditioner much more often (Anil et al., 2019), whereas our method is also promising in such scenarios.
Revised: When dealing with large scale models like Resnet50 (He et al., 2015a), Shampoo requires updating preconditioner much more often (Anil et al., 2019), whereas our method is also promising in such scenarios.

Original: We leave testing on such large benchmarks as a future work.
Revised: We leave comparison of SONew with Shampoo on such large benchmarks as a future work.

Original: To show efficacy of SONew at lower precision, we conduct bfloat16 experiments.
Revised: To show efficacy of SONew at lower precision, we conduct bfloat16 experiments.

Original: We notice in Table 2 that diag-SONew performs the best in first order methods, just like in float32.
Revised: We notice in Table 2 that diag-SONew performs the best in first order methods, just like in float32.

Original: Moreover SONew undergoes the least degradation in performancecompared to all other optimizers.
Revised: Moreover SONew undergoes the least degradation in performance

Original: We also provide an ablation study on effect of using Algorithm 3 on training loss in the appendix.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: compared to all other optimizers. We also provide an ablation study on effect of using Algorithmon training loss in the appendix.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: In this paper we have introduced a computationally efficient sparse preconditioner.
Revised: In this paper we have introduced a computationally efficient sparse preconditioner.

Original: Our algorithm arises from a novel regret bound analysis using LogDet Divergence, and furthermore we make it numerically stable.
Revised: Our algorithm arises from a novel regret bound analysis using LogDet Divergence, and furthermore we make it numerically stable.

Original: Experimental results on the Autoencoder benchmark confirm the effectiveness of SONew in both float32 as well as bfloat16 precision.
Revised: Experimental results on the Autoencoder benchmark confirm the effectiveness of SONew in both float32 as well as bfloat16 precision.

Original: In the future, one cantest SONew on larger benchmarks like DLRM (Naumov et al., 2019) and ResNet50 (He et al., 2015), and explore different sparsity graphs for which efficient solutions exist for the LogDet subproblem (11).
Revised: In the future, one can explore different sparsity graphs for which efficient solutions exist for the LogDet subproblem (11).

Original: A A PPENDIX A. P ROPERTIES OF L OG D ET SUBPROBLEM Proof 
Revised: A A PPENDIX A. P ROPERTIES OF L OG D ET SUBPROBLEM Proof of Theorem

Original: of Theorem

Original: The optimality condition of (11) is
Revised: The optimality condition of (11) is

Original: [Equation]
Revised: [Equation]

Original: Let Z =
Revised: Let Z =

Original: L − T D − 1 L −1 , then P G 
Revised: L − T D − 1 L − then P G ( Z ) =

Original: [Equation]
Revised: H ZL = L − T D − 1 = ⇒ ZLe

Original: =
Revised: j

Original: [Equation]
Revised: = L − T D − 1 e j

Original: Let J j
Revised: Let J j

Original: = I j ∪ j, then select J j indices of vectors on both sides of the second equality.
Revised: = I j ∪ j then select J j indices of vectors on both sides of the second equality [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that L − T is an upper triangular matrix with ones in the diagonal hence J thj block of L − T e j will be [1 , 0 , 0 , . . . ] .
Revised: Note that L − T is an upper triangular matrix with ones in the diagonal hence J thj block of L − T e j will be [1 , 0 , 0 , . . . ] .

Original: Also, since P G ( Z ) = H
Revised: Also, since P G ( Z ) = H

Original: [Equation]
Revised: [Equation]

Original: Substituting this in the linear equation
Revised: Substituting this in the linear equation

Original: [Equation]
Revised: [Equation]

Original: The lemma follows from solving the above equations.
Revised: The lemma follows from solving the above equations.

Original: Note that here we used that lower triangular halves of matrices L and H have the same sparsity patterns, which follows from the fact that banded graph is chordal and has a perfect elimination order [1 , 2 , . . . , n ] .
Revised: Note that here we used that lower triangular halves of matrices L and H have the same sparsity patterns, which follows from the fact that banded graph is chordal and has a perfect elimination order [1 , 2 , . . . , n ] .

Original: Proof of Theorem 1 The proof follows trivially from Theorem 1, when b is set to 1 .
Revised: Proof of Theorem 1 The proof follows trivially from Theorem 1, when b is set to 1 .

Original: A. R EGRET BOUND ANALYSISProof of Theorem 3.
Revised: A. R EGRET BOUND ANALYSIS

Revised: Proof of Theorem 3.

Original: To upperbound the regret we prove the following lemma about regret decompsition
Revised: To upperbound the regret we prove the following lemma about regret decompsition

Original: Lemma 2 ( Hazan et al.
Revised: Lemma 2 ( Hazan et al.

Original: In the OCO problem setup, if a prediction w t ∈ R n is made at round t and is updated as w t +1 :
Revised: In the OCO problem setup, if a prediction w t ∈ R n is made at round t and is updated as w t +1 :

Original: = w t − ηX t g t using a preconditioner matrix X t ∈ S ++ n
Revised: = w t − ηX t g t using a preconditioner matrix X t ∈ S ++ n

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Using the convexity of f t
Revised: Using the convexity of f t

Original: [Equation]
Revised: [Equation]

Original: f t ( w t ) − f t ( w ∗ ) ≤ ( w t − w ∗ ) T g t , where g t =
Revised: f t ( w t ) − f t ( w ∗ ) ≤ ( w t − w ∗ ) T g t [Equation] where g t =

Original: ∆ f t ( w t ) and summing over t ∈ [ T ]
Revised: ∆ f t ( w t ) and summing over t ∈ [ T ]

Original: [Equation]
Revised: [Equation]

Original: The first summation can be decomposed as follows
Revised: The first summation can be decomposed as follows

Original: [Equation]
Revised: [Equation]

Original: Substituting the above identity in the Equation (19) proves the lemma.
Revised: Substituting the above identity in the Equation (19) proves the lemma.

Original: Let R T ≤ T 1 + T 2 + T 3 , where
Revised: Let R T ≤ T 1 + T 2 + T 3 , where

Original: [-• T 1 = 12 η · ( ∥ w 1 − w ∗ ∥ 2 X − 1 1 − ∥ w T +
 − w ∗ ∥ X − 1 T ) • T 2 = 12 η · (cid:80) T − 1 t =1 ( w t +1 − w ∗ ) T ( X − 1 t +1 − X − 1 t )( w t +1 − w ∗ ) • T 3 = (cid:80) Tt =1 η 2 · g Tt X t g t-]
Revised: [-• T 1 = 12 η · ( ∥ w 1 − w ∗ ∥ 2 X − 1 1 − ∥ w T +
 − w ∗ ∥ X − 1 T ) • T 2 = 12 η · (cid:80) T − 1 t =1 ( w t +1 − w ∗ ) T ( X − 1 t +1 − X − 1 t )( w t +1 − w ∗ ) • T 3 = (cid:80) Tt =1 η 2 · g Tt X t g t-][Equation]

Original: The following lemmas will be used to bound T 1 , T 2 , T 3 .
Revised: The following lemmas will be used to bound T 1 , T 2 , T 3 .

Original: Lemma 3.
Revised: Lemma 3.

Original: If G = chain/tridiagonal graph and ˆ X = arg min X ∈ S n ( G ) ++ D ℓ d ( X, H − 1 ) , then the inverse ˆ X − 1 takes the following expression
Revised: If G = chain/tridiagonal graph and ˆ X = arg min X ∈ S n ( G ) ++ D ℓ d ( X, H − 1 ) , then the inverse ˆ X − 1 takes the following expression

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Where ˆ X ( j ) is the j th column of ˆ X .
Revised: Where ˆ X ( j ) is the j th column of ˆ X .

Original: Let ˆ Y denote the right hand side of Equation (20).
Revised: Let ˆ Y denote the right hand side of Equation (20).

Original: [Equation]
Revised: [Equation]

Original: The third equality is by using the following alternative form of Equation (12):
Revised: The third equality is by using the following alternative form of Equation (12):

Original: [Equation]
Revised: [Equation]

Original: where i < j .
Revised: where i < j .

Original: Similarly, the offdiagonals of ˆ Y ˆ X can be evaluated to be zero as follows.
Revised: Similarly, the offdiagonals of ˆ Y ˆ X can be evaluated to be zero as follows.

Original: [Equation]
Revised: [Equation]

Original: Lemma 4.
Revised: Lemma 4.

Original: Let y ∈ R n , β = max t max i ∈ [ n − 1]
Revised: Let y ∈ R n , β = max t max i ∈ [ n − 1]

Original: ( H t ) ii +1 / (cid:112)
Revised: ( H t ) ii +1 / (cid:112)

Original: ( H t ) ii ( H t )
Revised: ( H t ) ii ( H t )

Original: i +1 i +

Revised: i +1 i +

Original: < 1 , C = (cid:80) ni =

Revised: < 1 , C = (cid:80) n i =

Original: y 

Revised: y

Original: i ( G ( i ) ∞ ) 2 , where g ( i ) 1: T =
Revised: i ( G ( i ) ∞ ) 2 , where g ( i ) 1: T =

Original: [( g 1 ) i , . . .
Revised: [( g 1 ) i , . . .

Original: ( g T ) i ] and G ( i ) ∞ =
Revised: ( g T ) i ] and G ( i ) ∞ =

Original: ∥ g ( i ) 1: T ∥ ∞ , then y
Revised: ∥ g ( i ) 1: T ∥ ∞ , then y

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let ˜ X
Revised: Let ˜ X

Original: − 1 t 
Revised: − 1 t = diag( H t ) − 1 / 2 ˆ X t diag( H t ) − [Equation]

Original: [Equation]
Revised: [Equation]

Original:  Using the identity of spectral radius ρ ( X ) ≤ ∥ X ∥ ∞ and since ˜ X is positive definite, (cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13) 2 2 (cid:13)(cid:13)(cid:13) ˜ X − 1 t (cid:13)(cid:13)(cid:13)(22) Using the identity of spectral radius ρ ( X ) ≤ ∥ X ∥ ∞ and since ˜ X is positive definite, (cid:13)(cid:13)(cid:13)

Original: ˜ X − 1 t (cid:13)(cid:13)(cid:13) 2 ≤ ∥ ˜ X − 1 t ∥ ∞ 
Revised: ˜ X − 1 t (cid:13)(cid:13)(cid:13) 2 ≤ ∥ ˜ X − 1 t ∥ ∞ (cid:13)

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: The second inequality is using Lemma 3.
Revised: The second inequality is using Lemma 3.

Original: Using ( H t ) ii = (cid:13)(cid:13)(cid:13) g
Revised: Using ( H t ) ii = (cid:13)(cid:13)(cid:13) g

Original: ( i ) 1: T (cid:13)(cid:13)(cid:13) 2 

Revised: ( i ) 1: T (cid:13)(cid:13)(cid:13) 2

Original: + ϵ in Equation (22) will give the lemma.
Revised: + ϵ in Equation (22) will give the lemma.

Original: Lemma 5 (Upperbound of T 1 ) .
Revised: Lemma 5 (Upperbound of T 1 ) .

Original: T 1 ≤ ( C + ϵD 22 ) 2 η · 1+ β 1 − β , where D 2 = max t ∈
Revised: T 1 ≤ ( C + ϵD 22 ) 2 η · 1+ β 1 − β , where D 2 = max t ∈

Original: [ T ] ∥ w t − w ∗ ∥ 2 , where
Revised: [ T ] ∥ w t − w ∗ ∥ 2 , where

Original: C = max t ( (cid:80) ni =

Revised: C = max t ( (cid:80) ni =

Original: ( y ( t ) i ) 2 ( G ( i ) ∞ ) 2 ) .
Revised: ( y ( t ) i ) 2 ( G ( i ) ∞ ) 2 ) .

Original: Proof.
Revised: Proof.

Original: Since X T is positive definite
Revised: Since X T is positive definite

Original: [Equation]
Revised: [Equation]

Original: Using Lemma 4 proves the lemma.
Revised: Using Lemma 4 proves the lemma.

Original: Upperbounding T
Revised: Upperbounding T

Original: Let y t = w t − w ∗ , P t =
Revised: Let y t = w t − w ∗ , P t =

Original: y
Revised: y

Original: Tt X − 1 t y t , and Q t = y
Revised: Tt X − 1 t y t , and Q t = y

Original: Tt X − 1 t − 1 y t .
Revised: Tt X − 1 t − 1 y t .

Original: Let P =
Revised: Let P =

Original: [ P 2 , · · · , P T ] and Q =
Revised: [ P 2 , · · · , P T ] and Q =

Original: [ Q 2 , · · · , Q T ] be the two array representations.
Revised: [ Q 2 , · · · , Q T ] be the two array representations.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: In order to upper bound this, we derive here a generalized version of Pinsker’s inequality for our setting.
Revised: In order to upper bound this, we derive here a generalized version of Pinsker’s inequality for our setting.

Original: Lemma 6.
Revised: Lemma 6.

Original: Let A =
Revised: Let A =

Original: [ A 1 , · · · , A T ] and B =
Revised: [ A 1 , · · · , A T ] and B =

Original: [ B 1 , · · · , B T ] be two T -length arrays.
Revised: [ B 1 , · · · , B T ] be two T -length arrays.

Original: Let B i , A i > 0 ∀ i ∈ [1 , T + 1] .
Revised: Let B i , A i > 0 ∀ i ∈ [1 , T + 1] .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where the generalized KL-Divergence D KL ( B, A ) = (cid:80) Ti =1 (cid:16) B i log (cid:16) B
Revised: where the generalized KL-Divergence D KL ( B, A ) = (cid:80) Ti =1 (cid:16) B i log (cid:16) B

Original: i
Revised: i

Original: A
Revised: A

Original: i (cid:17) − B
Revised: i (cid:17) − B

Original: i
Revised: i

Original: + A
Revised: + A

Original: i (cid:17) .
Revised: i (cid:17) .

Original: Proof.
Revised: Proof.

Original: This proof is a mock up of Pinsker’s inequality for probability spaces.
Revised: This proof is a mock up of Pinsker’s inequality for probability spaces.

Original: First step involves using the following identity, for t ≥ −
Revised: First step involves using the following identity, for t ≥ −

Original: [Equation]
Revised: [Equation]

Original: Let t i
Revised: Let t i

Original: = B i
Revised: = B i

Original: A
Revised: A

Original: i − 1 .
Revised: i − 1 .

Original: Then, − 1 ≤ t i .
Revised: Then, − 1 ≤ t i .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: (Using Cauchy-Schwartz)
Revised: Using Cauchy-Schwartz)

Original: [Equation]
Revised: [Equation]

Original: Using the above to bound Equation (23) with B := P and A :

Original: = Q

Original: [Equation]

Original: we get

Original: [Equation]
Revised: [Equation]

Original: Here, ¯ y t = Λ t y t , ¯ X t = Λ t X t Λ t , ¯ X t − 1 =
Revised: Here, ¯ y t = Λ t y t , ¯ X t = Λ t X t Λ t , ¯ X t − 1 =

Original: Λ t X t − 1 Λ t D vN is von-Neuman Divergence, and last inequality is from Lindblad (1975).
Revised: Λ t X t − 1 Λ t D vN is von-Neuman Divergence [Equation] and last inequality is from Lindblad (1975).

Original: We upper bound Equation (24) using the following lemma.
Revised: We upper bound Equation (24) using the following lemma.

Original: Lemma 7.
Revised: Lemma 7.

Original: Let A, B be two PD matrices.
Revised: Let A, B be two PD matrices.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let A = V Λ V T and B = U Θ U T
Revised: Let A = V Λ V T and B = U Θ U T

Original: [Equation]
Revised: [Equation]

Original: j
Revised: Combining the above result

Original: [Equation]
Revised: [Equation]

Original: Combining the above result, we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: The second inequality is due to scale invariance of LogDet divergence.
Revised: The second inequality is due to scale invariance of LogDet divergence.

Original: If we set Λ t = diag( H t ) 1 / 2 , then using Lemma
Revised: If we set Λ t = diag( H t ) 1 / 2 , then using Lemma

Original: [Equation]
Revised: [Equation]

Original: where, D 2 = max t ∥ y t ∥ 2 , the second and third inequality is using Lemma 4.
Revised: where, D 2 = max t ∥ y t ∥ 2 , the second and third inequality is using Lemma 4.

Original: Now to bound the
Revised: Now to bound the

Original: D ℓ d ( X − 1 t − 1 , X − 1 t ) term in the above equation, we develop the following lemma.
Revised: D ℓ d ( X − 1 t − 1 , X − 1 t ) term in the above equation [Equation] we develop the following lemma.

Original: Lemma 8.
Revised: Lemma 8.

Original: Tr( X t H t ) = n , ∀ t ∈ { 2 , . . .
Revised: Tr( X t H t ) = n , ∀ t ∈ { 2 , . . .

Original: , T } in Algorithm 1 with any G
Revised: , T } in Algorithm 1 with any G

Original: Proof.
Revised: Proof.

Original: P G ( X − 1 t ) =
Revised: P G ( X − 1 t ) =

Original: H t is an optimality condition of LogDet subproblem Equation (11)
Revised: H t is an optimality condition of LogDet subproblem Equation (11)

Original: [Equation]
Revised: [Equation]

Original: The second equality is because X t follows the sparsity graph G.
Revised: The second equality is because X t follows the sparsity graph G

Revised: [Equation]

Original: Lemma9.
Revised: Lemma

Original: [Equation]
Revised: [Equation]

Original: where β 
Revised: where β i =

Original: i = (cid:68) g ( i ) 1: T ,g ( i +1) 1: T (cid:69) (cid:114) ( ϵ + (cid:13)(cid:13)(cid:13) g ( i ) 1: T (cid:13)(cid:13)(cid:13) 2 2 ) · ( ϵ + (cid:13)(cid:13)(cid:13) g
Revised: [Equation]

Original: ( i +1) 1: T (cid:13)(cid:13)(cid:13) 2 2 )

Original: Proof.
Revised: Proof.

Original: LogDet divergence is defined as follows
Revised: LogDet divergence is defined as follows

Original: [Equation]
Revised: [Equation]

Original: The second two terms on right hand side can be simplified analyzed as follows.
Revised: The second two terms on right hand side can be simplified analyzed as follows.

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Revised: T (cid:88)

Revised: [Equation]

Revised: D

Original: [Equation]
Revised: [Equation]

Original: Since LogDet divergence is positive
Revised: Since LogDet divergence is positive

Original: [Equation]
Revised: [Equation]

Original: The second inequality uses Equation (26).
Revised: The second inequality uses Equation (26).

Original: Now, using Equation (13), which is in cholesky decomposition format,
Revised: Now, using Equation (13), which is in cholesky decomposition format,

Original: [Equation]
Revised: [Equation]

Original: Using the above
Revised: Using the above

Original: [Equation]
Revised: [Equation]

Original: we can expand the log deteriminatn difference.
Revised: we can expand the log deteriminatn difference.

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Let X 0 = arg min X ∈ S n ( G ) ++ D ℓ d
Revised: Let X 0 = arg min X ∈ S n ( G ) ++ D ℓ d

Original: ( X, H − 1 0 ) , the above inequality is since log 
Revised: ( X, H − 1 0 ) , the above inequality is since log [Equation] ) det [Equation]

Revised: X

Original: [Equation]
Revised: [Equation]

Revised: 1 0 ) (cid:19)

Revised: [Equation]

Original: using Equation (27)
Revised: using Equation (27)

Original: Lemma 10 (Upperbound of T 2 ).
Revised: Lemma 10 (Upperbound of T 2 )

Original: [Equation]
Revised: [Equation]

Original: 

Original: Proof.
Revised: Proof.

Original: The proof directly follows from Equation (25) and Lemma
Revised: The proof directly follows from Equation (25) and Lemma

Original: Lemma 11 (Upperbound of T 3 ) .
Revised: Lemma 11 (Upperbound of T 3 ) .

Original: Let X 
Revised: Let X 0 = arg

Original: [Equation]
Revised: [Equation]

Original: =
Revised: X ∈

Original: [Equation]
Revised: [Equation]

Original: where β i = 
Revised: where β i = (

Original: (( g ( i ) 1: T ) T g ( i +1) 1: T )

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: g ( i

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: The second equality is using Lemma 3 and that X t ’s sparsity graph is tridiagonal.
Revised: The second equality is using Lemma 3 and that X t ’s sparsity graph is tridiagonal.

Original: The first inequality is using the property D ℓ d ( X − 1 t − 1 , X − 1 t ) ≥ 0 of LogDet divergence.
Revised: The first inequality is using the property D ℓ d ( X − 1 t − 1 , X − 1 t ) ≥ 0 of LogDet divergence.

Original: Thus summing up and using
Revised: Thus summing up and using

Original: Equation (28) will give the lemma.
Revised: Equation (28) will give the lemma.

Original: Putting together T 1 , T 2 and T 3 from Lemma 
Revised: Putting together T 1 , T 2 and T 3 from Lemma 5, Lemma 10 and Lemma T

Original: [Equation]
Revised: [Equation]

Original: Lemma 10 and Lemma
Revised: (cid:118)(cid:117)(cid:117) (cid:117)(cid:117) (cid:116)

Original: [Equation]
Revised: [Equation]

Revised: T    n (cid:88)

Revised: i = log   1 + (cid:13)(cid:13) (cid:13) g ( i ) 1: T (cid:13)(cid:13) (cid:13)

Revised: 2ϵ   

Revised: + n − 1 (cid:88)

Original:  [Equation]
Revised: i = log (1 − β 2 i )   , (29) T 3 ≤ η 2 · n (cid:88) i = log   [Equation]

Revised: + (cid:13)(cid:13)(cid:13) g ( i ) 1: T (cid:13)(cid:13)(cid:13) 2ϵ

Revised:   + n − 1 (cid:88) i

Original: {+= log (cid:0) 1 − β 
+}
Revised: {+= log (cid:0) 1 − β 
+}

Revised: i (cid:1) (30)

Original: If we set η =
Revised: If we set η =

Revised: C 1 /

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Revised: / (cid:80)

Original:  then
Revised: ni =1 log   1+ ∥ g ( i ) 1: T ∥ 2 2 ϵ   + (cid:80) n − 1 i =1 log ( 1 − β 2 i )   1 / 4 , then

Revised: R T ≤ T 1 + T 2 + T≤ O   C 1 /

Original: [Equation]
Revised: [Equation]

Original:  g ( i ) 1: T (cid:13)(cid:13)(cid:13) 2ϵ  + n − 1 (cid:88)
Revised: · T [Equation] / 4 · (cid:18) 1 + β 1 − β (cid:19) 1 /·   n (cid:88) i = log   1 + (cid:13)(cid:13)(cid:13) g ( i ) 1: T (cid:13)(cid:13)(cid:13) 2ϵ 

Revised:  + n − 1 (cid:88)

Original: i = log (cid:0) 1 − β 2 i (cid:1)
Revised: i = log (cid:0) 1 − β 2 i (cid:1)

Original:   3 / 4  
Revised:   3 / 4  

Original:  A. O ( T 1 / 2 ) R EGRET UPPER BOUND
Revised: A. O ( T 1 / 2 ) R EGRET UPPER BOUND

Original: In this section we derive a regret upper bound with a O ( T 1 / 2 ) growth, in contrast to the O ( T 3 / 4 ) obtained in A.2.
Revised: In this section we derive a regret upper bound with a O ( T 1 / 2 ) growth, in contrast to the O ( T 3 / 4 ) obtained in A.2.

Original: In (29), T 2 = (cid:80) Tt =2 ( w t − w ∗ ) T ( X − 1 t − X − 1 t − 1 )( w t − w ∗ ) is of the order
Revised: In (29), T 2 = (cid:80) Tt =2 ( w t − w ∗ ) T ( X − 1 t − X − 1 t − 1 )( w t − w ∗ ) is of the order

Original: O ( T 3 / 2 ) , which can be reduced to O ( T ) , by upper bounding each entry of X − 1 t − X − 1 t − 1 individually.
Revised: O ( T 3 / 2 ) , which can be reduced to O ( T ) , by upper bounding each entry of X − 1 t − X − 1 t − 1 individually.

Original: The following lemma helps in constructing a telescoping argument to bound (cid:12)(cid:12) ( X − 1 t − X − 1 t − 1 ) i,j (cid:12)(cid:12) .
Revised: The following lemma helps in constructing a telescoping argument to bound (cid:12)(cid:12) ( X − 1 t − X − 1 t − 1 ) i,j (cid:12)(cid:12) .

Original: Lemma 12.
Revised: Lemma 12.

Original: Let H, ˜ H ∈ S ++ n , such that ˜ H
Revised: Let H, ˜ H ∈ S ++ n , such that ˜ H

Original: =
Revised: =

Original: H + gg T , where g ∈ R n , then
Revised: H + gg T , where g ∈ R n , then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original:   
Revised:  

Original: 
Revised: 

Original: Lemma 13.
Revised: Lemma 13.

Original: Let H, ˜ H ∈ S ++ n , such that ˜ H
Revised: Let H, ˜ H ∈ S ++ n , such that ˜ H

Original: =
Revised: =

Original: H + gg T , where g ∈ R n .
Revised: H + gg T , where g ∈ R n .

Original: Also, ˜ Y = arg min X ∈ S n ( G ) ++ D ℓ d ( X, ˜ H ) and
Revised: Also, ˜ Y = arg min X ∈ S n ( G ) ++ D ℓ d ( X, ˜ H ) and

Original: Y = arg min X ∈ S n ( G ) ++ D ℓ d ( X, H ) , where G is a chain graph, then
Revised: Y = arg min X ∈ S n ( G ) ++ D ℓ d ( X, H ) , where G is a chain graph, then

Original: [Equation]
Revised: [Equation]

Original: where i, i + k ≤ n , G ∞ =
Revised: where i, i + k ≤ n , G ∞ =

Original: ∥ g ∥ ∞ and max i,j | H ij | / (cid:112) H ii H jj ≤ β < 1 .
Revised: ∥ g ∥ ∞ and max i,j | H ij | / (cid:112) H ii H jj ≤ β < 1 .

Original: Let κ (diag( H )) :
Revised: Let κ (diag( H )) :

Original: = condition number of the diagonal part of H , then κ := max( κ (diag( H )) , κ (diag( ˜ H ))) .
Revised: = condition number of the diagonal part of H , then κ := max( κ (diag( H )) , κ (diag( ˜ H ))) .

Original: Proof.
Revised: Proof.

Original: Using Lemma 3 will give the following:
Revised: Using Lemma 3 will give the following:

Original: [Equation]
Revised: [Equation]

Original: where N ij =
Revised: where N ij =

Original: H ij / (cid:112)
Revised: H ij / (cid:112)

Original: H ii H jj .
Revised: H ii H jj .

Original: Expanding ˜ N ii +1 =
Revised: Expanding ˜ N ii +1 =

Original: N ii +1 + θ ii +

Revised: N ii +1 + θ ii +

Original: (from Lemma 12), subsequently ˜ N ii +2 =
Revised: (from Lemma 12), subsequently ˜ N ii +2 =

Original: N ii +2 + θ ii
Revised: N ii +2 + θ ii

Original: +

Revised: +

Original: and so on will give(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ˜ N ii [Equation] (cid:115) [Equation]
Revised: and so on will give

Original: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: (cid:33)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) =

Original: | θ ii +1 ˜ N i +1 i +2 . . . ˜ N
Revised: [Table]

Original: i + k

Original: [Equation]
Revised: [Equation]

Original: 1 i + k

Original: [Equation] N ii +1 θ i +1 i +
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: N ii +3 . . . ˜ N
Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: k − 1 i + k + . . .

Original: + N ii

Original: +1 . . .

Original: N ii + k − 1 θ i + k − 1 i + k + N ii +1 . . . N ii + k (cid:32) 1 − (cid:115)
Revised: (cid:12)(cid:12) (cid:12)(cid:12)(cid:12) 1 − (cid:115)

Original: H ii H
Revised: H ii H

Original: i + ki + k ˜ H ii
Revised: i + ki + k ˜ H ii

Original: ˜ H
Revised: ˜ H

Original: i + ki+ k (cid:33) | ≤ ( k − 1 (cid:88) l = | θ i + li + l +1 | ) β k − 1 + β k − 1 (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) 1 − (cid:115) H ii H
Revised: i + ki

Original: i + ki + k ˜ H ii

Original: ˜ H

Original: i + ki + k (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) , = ⇒ (cid:12)(cid:12)(cid:12) ( ˜ Y −

Original: [Equation]
Revised: [Equation]

Original: Y − 1 ) ii + k  (cid:12)(cid:12)(cid:12) ≤ (cid:113) ˜ H ii ˜ H i + ki + k (cid:32) ( k
Revised: k (cid:12)(cid:12) (cid:12)(cid:12)(cid:12) (cid:33) where max i,j | N i,j | , max i,j | ˜ N i,j | ≤ β .

Original: [Equation]

Original: 1 (cid:88) l = | θ i + li + l +1 | )

Original: β k − 1 + β k − 1 (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) 1 − (cid:115)

Original: H ii H

Original: i + ki + k ˜ H ii

Original: ˜ H

Original: i + ki + k (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:33) where max i,j | N i,j | , max i,j | ˜ N i,j | ≤ β .

Original: Expanding θ i + li + l +1 from Lemma 12 in the term | θ i + li + l +1 | [Table]
Revised: Expanding θ i + li + l +1 from Lemma 12 in the term | θ i + li + l +1 | (cid:113)

Original: ˜ H ii ˜ H
Revised: ˜ H ii ˜ H

Original: i + ki + k will give:
Revised: i + ki + k will give:

Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: i
Revised: g

Original: [Table]
Revised: [Table]

Revised: Since H

Original:  li + l +1 | [Table] 1 i + l  + [Equation]
Revised: i + li + l H i + l +1 i + l +1 ≤ ˜ H

Original: (cid:33)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
Revised: i + li + l ˜ H i + l +1 i + l +1 , 1 − (cid:115)

Original: Since H
Revised: H

Original: i + li + l H i + l +1{++
+} i + l +1 ≤ ˜ H
Revised: i + li + l H i + l {++
+} i + l + ˜ H

Revised: i + li + l ˜ H i + l +1 i + l +1 ≤ max (cid:18)

Original:  i + li + l ˜ H i +  l +1 i + l +1 ,{+˜ H i + l +
+}
Revised: 1 − H i + li + l ˜ H i + li + l , 1 − H i + l +1 i + l +1 {+˜ H i + l +
+}

Original: [Equation]{+i + l + (cid:19) ≤ max (cid:18) g 2 i + l ˜ H i + li + l , g 2 i + l + ˜ H i + l +
+}
Revised: {+i + l + (cid:19) ≤ max (cid:18) g 2 i + l ˜ H i + li + l , g 2 i + l + ˜ H i + l +
+}

Original: [Equation] (cid:19)
Revised: i + l + (cid:19)

Original: Using the above, H i,i /H j,j ≤ κ , and | g i | ≤ G ∞ , ∀ i, j ∈ [ n ] , gives
Revised: Using the above, H i,i /H j,j ≤ κ , and | g i | ≤ G ∞ , ∀ i, j ∈ [ n ] , gives

Original: [Equation]
Revised: [Equation]

Original: Thus the following part of (cid:12)(cid:12)(cid:12)(cid:12)(cid:16) ˜ Y − 1 − Y − 1 (cid:17) ii + k (cid:12)(cid:12)(cid:12)(cid:12) can be upperbounded:(cid:113) ˜
Revised: Thus the following part of (cid:12)(cid:12)(cid:12)(cid:12)(cid:16) ˜ Y − 1 − Y − 1 (cid:17) ii + k (cid:12)(cid:12)(cid:12)(cid:12) can be upperbounded:

Original: H ii ˜ H i + ki + k (cid:32) ( k − 1 (cid:88) l = | θ i + li + l +1 | ) β k −(cid:33) ≤ G 2 ∞ κ (1 + β ) kβ k −
Revised: [Equation]

Original: Also, β k − 1 (cid:12)(cid:12)(cid:12) 1 − (cid:113) H ii H
Revised: Also, β k − 1 (cid:12)(cid:12)(cid:12) 1 − (cid:113) H ii H

Original: i + ki + k ˜ H ii
Revised: i + ki + k ˜ H ii

Original: ˜ H
Revised: ˜ H

Revised: i + ki + k (cid:12)(cid:12)(cid:12) ≤ β k − 1 κG 2 ∞ , so (cid:12)(cid:12)(cid:12)(cid:12)(cid:16)

Revised: [Equation]

Original: i + ki +
Revised: ii +

Original: [Equation]
Revised: [Equation]

Revised: −

Original: Lemma 14 ( O ( T ) upper bound of T 2 ) .
Revised: Lemma 14 ( O ( T ) upper bound of T 2 ) .

Original: Given that κ (diag( H t )) ≤ κ , ∥ w t − w ∗ ∥ 2 ≤ D 2 , max i,j | ( H t ) ij | / (cid:112) ( H t ) ii ( H t )
Revised: Given that κ (diag( H t )) ≤ κ , ∥ w t − w ∗ ∥ 2 ≤ D 2 , max i,j | ( H t ) ij | / (cid:112) ( H t ) ii ( H t )

Original: jj ≤ β < 1 , ∀ t ∈ [ T ] in Algorithm 1, then T 2 in (29) can be bounded as follows:
Revised: jj ≤ β < 1 , ∀ t ∈ [ T ] in Algorithm 1, then T 2 in (29) can be bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Note that T
Revised: Note that T

Original: [Equation]
Revised: [Equation]

Original: ≤

Original: [Equation]

Original: 1 t

Original: [Equation]

Original: 1 −

Original: [Equation]

Original: − 1 t )

Original: [Equation]

Original: Using ∥ A ∥ 2 = ρ ( A ) ≤ ∥
Revised: Using ∥ A ∥ 2 = ρ ( A ) ≤ ∥

Original: A ∥ ∞ for symmetric matrices
Revised: A ∥ ∞ for symmetric matrices

Original: A , we get
Revised: A , we get

Original: [Equation]
Revised: [Equation]

Original: The third inequality is using Lemma 13.
Revised: The third inequality is using Lemma 13.

Original: Expanding T 2 with this bound gives the result.
Revised: Expanding T 2 with this bound gives the result.

Original: Theorem 6 ( O ( T 1 / 2 ) regret upper bound) .
Revised: Theorem 6 ( O ( T 1 / 2 ) regret upper bound) .

Original: Setting
Revised: Setting

Original: [Equation]
Revised: [Equation]

Original: gives the following regret
Revised: gives the following regret

Original: [Equation]
Revised: [Equation]

Original: −

Original: [Equation]

Original:  where condition number κ (diag( H t )) ≤ κ , ∥ w t − w ∗ ∥ 2 ≤ D [Equation] H [Equation]
Revised: , where condition number κ (diag( H t )) ≤ κ , ∥ w t − w ∗ ∥ 2 ≤ D 2 , max i,j | ( H t ) ij | / (cid:112) ( H t ) ii ( H t )

Original: H

Original: [Equation]

Original: ii

Original: [Equation] H [Equation] jj ≤ β < 1 , ∀ t ∈ [ T ] and g t , H t are as defined in Algorithm 1.
Revised: jj ≤ β < 1 , ∀ t ∈ [ T ] and g t , H t are as defined in Algorithm 1.

Original: Proof.
Revised: Proof.

Original: The result follows from using (30) and Lemma 14.
Revised: The result follows from using (30) and Lemma 14.

Original: A. N UMERICAL STABILITY
Revised: A. N UMERICAL STABILITY

Original: Theorem 7 ( Full version of Theorem 4 ) .
Revised: Theorem 7 ( Full version of Theorem 4 ) .

Original: Let H ∈ S ++ n such that H ii
Revised: Let H ∈ S ++ n such that H ii

Original: = 1 , for i ∈ [ n ] and a symmetric perturbation ∆ H such that ∆ H ii = 0 , for i ∈ [ n ] and H + ∆ H ≻ 0 .
Revised: = 1 , for i ∈ [ n ] and a symmetric perturbation ∆ H such that ∆ H ii = 0 , for i ∈ [ n ] and H + ∆ H ≻ 0 .

Original: Let ˆ X =
Revised: Let ˆ X =

Original: arg
Revised: arg

Original: min
Revised: min

Original: X ∈ S n ( G ) ++ D ℓ d (cid:0) X, H − 1 (cid:1) and ˆ X + ∆ ˆ X = arg min X ∈ S n [Equation] G )  + 
Revised: X ∈ S n ( G ) ++ D ℓ d (cid:0) X, H − 1 (cid:1) and ˆ X + ∆ ˆ X = arg min X ∈ S n ( G ) ++ D ℓ d (cid:0) X, ( H + ∆ H ) − 1 (cid:1) , here G :=

Original: [Equation]

Original: D

Original: [Equation] d [Equation] X, ( H + ∆ H ) − 1 (cid:1) , here G := chain/tridiagonal sparsity graph and S n ( G ) ++ denotes positive definite matrices which follows the sparsity pattern G .
Revised: chain/tridiagonal sparsity graph and S n ( G ) ++ denotes positive definite matrices which follows the sparsity pattern G .

Original: [Equation]
Revised: [Equation]

Original: where, κ ℓd := condition number of the LogDet subproblem, κ 2 ( . ) :
Revised: where, κ ℓd := condition number of the LogDet subproblem, κ 2 ( . ) :

Original: = condition number of a matrix in ℓ 2 norm, β i =
Revised: = condition number of a matrix in ℓ 2 norm, β i =

Original: H ii +1 / (cid:112) H ii
Revised: H ii +1 / (cid:112) H ii

Original: H
Revised: H

Original: i +1 i
Revised: i +1 i

Original: +
Revised: +

Original: Proof.
Revised: Proof.

Original: Consider the offdiagonals for which ( ˆ X + ∆ ˆ X ) ii +1 = − H ii
Revised: Consider the offdiagonals for which ( ˆ X + ∆ ˆ X ) ii +1 = − H ii

Original: +1 / (1 − H 2 ii +1 ) =
Revised: +1 / (1 − H 2 ii +1 ) =

Original: f ( H ii +1 ) ,where f ( x ) = − x/
Revised: f ( H ii +1 ) ,where f ( x ) = − x/

Original: (1 − x 2 ) .
Revised: (1 − x 2 ) .

Original: Let y =
Revised: Let y =

Original: f ( x ) , ˆ y = f
Revised: f ( x ) , ˆ y = f

Original: ( x + ∆ x ) and | ∆ x/x | ≤ ϵ then usingTaylor series
Revised: ( x + ∆ x ) and | ∆ x/x | ≤ ϵ then using series

Original: [Equation]
Revised: [Equation]

Original: Using the above inequality, with x :
Revised: Using the above inequality, with x :

Original: = H ii +1 and y :
Revised: = H ii +1 and y :

Original: = ˆ X ii +1 ,
Revised: = ˆ X ii +1 ,

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Let g ( x ) =
Revised: Let g ( x ) =

Original: x 2 / (1 − x 2 ) , let y 

Revised: x 2 / (1 − x 2 ) , let y

Original: = g ( w 1 ) , y 2 = g ( x 2 ) , ˆ y 1 = g ( w 1 + ∆ x ) , ˆ y 2 = g ( x 2 + ∆ x ) .
Revised: = g ( w 1 ) , y 2 = g ( x 2 ) , ˆ y 1 = g ( w 1 + ∆ x ) , ˆ y 2 = g ( x 2 + ∆ x ) .

Original: Using
Revised: Using

Original: Taylor series
Revised: Taylor series

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)
Revised: (cid:12)(cid:12)(cid:12) (cid:12) (cid:12)(cid:12)(cid:12) (cid:12)

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)
Revised: (cid:12)(cid:12)(cid:12) (cid:12)

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: (cid:12)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Revised: (cid:12)(cid:12)(cid:12)(cid:12)

Original: [Table]
Revised: [Equation]

Original: Putting x 1 :
Revised: Putting x 1 :

Original: = H ii +1 , x 2 :=
Revised: = H ii +1 , x 2 :=

Original: H ii − 1 and analyzing y 1 :
Revised: H ii − 1 and analyzing y 1 :

Original: = H 2 ii +1 / (1 − H 2 ii +1 ) and y 2 :
Revised: = H 2 ii +1 / (1 − H 2 ii +1 ) and y 2 :

Original: =
Revised: =

Original: H 2 ii − 1 / (1 −
Revised: H 2 ii − 1 / (1 −

Original: H 2 ii − 1 ) will result in the following
Revised: H 2 ii − 1 ) will result in the following

Original: [Equation]
Revised: [Equation]

Original: Since ˆ X ii = 1 + H 2 ii +1 / (1 − H 2 ii +1 )
Revised: Since ˆ X ii = 1 + H 2 ii +1 / (1 − H 2 ii +1 )

Original: + H 2 ii − 1 / (1 − H 2 ii − 1 ) .
Revised: + H 2 ii − 1 / (1 − H 2 ii − 1 ) .

Original: Putting together Equation (32) and
Revised: Putting together Equation (32) and

Original: Equation (31), the theorem is proved.
Revised: Equation (31), the theorem is proved.

Original: Proof of Lemma
Revised: Proof of Lemma

Original: For b = 1 , if g ( j ) 1: T = g ( j +1) 1: T , then H jj +

Revised: For b = 1 , if g ( j ) 1: T = g ( j +1) 1: T , then H jj +

Original: = H jj
Revised: = H jj

Original: =
Revised: =

Original: H
Revised: H

Original: j +1 j +1 =
Revised: j +1 j +1 =

Original: (cid:13)(cid:13)(cid:13) g ( j ) 1: T (cid:13)(cid:13)(cid:13) 2 2 , thus H jj −
Revised: (cid:13)(cid:13)(cid:13) g ( j ) 1: T (cid:13)(cid:13)(cid:13) 2 2 , thus H jj −

Original: H 2 jj +1 /H
Revised: H 2 jj +1 /H

Original: j +

Revised: j +

Original: j +1 = 0 .
Revised: j +1 = 0 .

Original: For b > 1 , since H I j I j , using Guttman rank additivity formula, rank( H jj − H 2 jj +1 /H
Revised: For b > 1 , since H I j I j , using Guttman rank additivity formula, rank( H jj − H 2 jj +1 /H

Original: j +1 j +1 )
Revised: j +1 j +1 )

Original: =
Revised: =

Original: rank( H J j J j ) − rank ( H I j I j ) = 0 , thus H jj − H 2 jj
Revised: rank( H J j J j ) − rank ( H I j I j ) = 0 , thus H jj − H 2 jj

Original: +1 /H
Revised: +1 /H

Original: j +

Revised: j +

Original: j +1 = 0 .
Revised: j +1 = 0 .

Original: Furthermore, if rank( H ) ≤ b , then all b + 1 × b + 1 principal submatrices of H have rank b , thus ∀ j , H J j J j have a rank b , thus D jj for all j are undefined.
Revised: Furthermore, if rank( H ) ≤ b , then all b + 1 × b + 1 principal submatrices of H have rank b , thus ∀ j , H J j J j have a rank b , thus D jj for all j are undefined.

Original: Proof 
Revised: Proof of Theorem

Original: of Theorem

Original: Let I i
Revised: Let I i

Original: = { j : i < j, ( i,
Revised: = { j : i < j, ( i,

Original: j ) ∈ E G } and I ′ i = (cid:8)
Revised: j ) ∈ E G } and I ′ i = (cid:8)

Original: j : i < j, ( i,
Revised: j : i < j, ( i,

Original: j ) ∈
Revised: j ) ∈

Original: E ˜ G (cid:9) Let K = (cid:8)
Revised: E ˜ G (cid:9) Let K = (cid:8)

Original: i : H ii − H TI
Revised: i : H ii − H TI

Original: i
Revised: i

Original: i H − 1 I i
Revised: i H − 1 I i

Original: I
Revised: I

Original: i H
Revised: i H

Original: I
Revised: I

Original: i
Revised: i

Original: i is undefined or 0 , i ∈
Revised: i is undefined or 0 , i ∈

Original: [ n ] (cid:9) denote vertices which are getting removed by the algorithm, then for the new graph ˜ G , D ii = 1 /H ii , ∀ i ∈ K since H ii > 0 .
Revised: [ n ] (cid:9) denote vertices which are getting removed by the algorithm, then for the new graph ˜ G , D ii = 1 /H ii , ∀ i ∈ K since H ii > 0 .

Original: Let ¯ K
Revised: Let ¯ K

Original: = (cid:8) i : H ii − H TI
Revised: = (cid:8) i : H ii − H TI

Original: i i H − 

Revised: i i H −

Original: I i
Revised: I i

Original: I
Revised: I

Original: i H
Revised: i H

Original: I i i > 0 , i ∈ [ n ] (cid:9) .
Revised: I i i > 0 , i ∈ [ n ] (cid:9) .

Original: Let for some j ∈ ¯ K , if l = arg min { i : j < i, i ∈ K ∩ I j } , denotes the nearest connected vertex higher than j for which D ll is undefined or zero, then according to the definition E ˜ G in Algorithm 3, I ′
Revised: Let for some j ∈ ¯ K , if l = arg min { i : j < i, i ∈ K ∩ I j } , denotes the nearest connected vertex higher than j for which D ll is undefined or zero, then according to the definition E ˜ G in Algorithm 3, I ′

Original: j = { j +1 , . . .
Revised: j = { j +1 , . . .

Original: l − 1 } ⊂
Revised: l − 1 } ⊂

Original: I j , since D jj is well-defined, H I j
Revised: I j , since D jj is well-defined, H I j

Original: I j is invertible, which makes it a positive definite matrix (since H is PSD).
Revised: I j is invertible, which makes it a positive definite matrix (since H is PSD).

Original: Since H jj − H TI j j H − 

Revised: Since H jj − H TI j j H −

Original: I j I j H I j j > 0 , using Guttman rank additivity formula H J j J j ≻ 0 , where J j = I j ∪ j .
Revised: I j I j H I j j > 0 , using Guttman rank additivity formula H J j J j ≻ 0 , where J j = I j ∪ j .

Original: Since H J ′ j J ′
Revised: Since H J ′ j J ′

Original: j is a submatrix of H J j J j , it is positive definite and hence its schur complement H jj − H TI ′
Revised: j is a submatrix of H J j J j , it is positive definite and hence its schur complement H jj − H TI ′

Original: j j H
Revised: j j H

Original: −
Revised: −

Original: I ′
Revised: I ′

Original: j I ′ j H
Revised: j I ′ j H

Original: I ′ j j > 0 .
Revised: I ′ j j > 0 .

Original: Thus for all j ∈ [ n ] , the corresponding D jj ’s are well-defined in the new graph ˜ G .
Revised: Thus for all j ∈ [ n ] , the corresponding D jj ’s are well-defined in the new graph ˜ G .

Original: Note that κ ˜ G ℓd = max i ∈
Revised: Note that κ ˜ G ℓd = max i ∈

Original: [ n − 1] 1 / (1 − β 2 i ) < max
Revised: [ n − 1] 1 / (1 − β 2 i ) < max

Original: i ∈
Revised: i ∈

Original: ¯ K 1 / (1 − β 2 i ) =
Revised: ¯ K 1 / (1 − β 2 i ) =

Original: κ G ℓd , for tridiagonal graph, where β i =
Revised: κ G ℓd , for tridiagonal graph, where β i =

Original: H ii +1 , in the case where H ii = 1 .
Revised: H ii +1 , in the case where H ii = 1 .

Original: This is because the arg max i ∈ [ n − 1] 1 / (1 − β 2 i ) ∈ K . 
Revised: This is because the arg max i ∈ [ n − 1] 1 / (1 − β 2 i ) ∈ K . [Table] [Table] [Table]

Original: A. A DDITIONAL E XPERIMENTS , ABLATIONS , AND DETAILS
Revised: A. A DDITIONAL E XPERIMENTS , ABLATIONS , AND DETAILS

Original: Effect of band size in banded-SONew Increasing band size in banded-SONew captures more correlation between parameters, hence should expectedly lead to better preconditioners.
Revised: Effect of band size in banded-SONew Increasing band size in banded-SONew captures more correlation between parameters, hence should expectedly lead to better preconditioners.

Original: We confirm this through experiments on the Autoencoder benchmark where we take band size = 0 (diagSONew), 1 (tridiag-SONew),{+
 (diag-SONew),(tridiag-SONew),+} 4, and 10.
Revised: We confirm this through experiments on the Autoencoder benchmark where we take band size = {+
 (diag-SONew),(tridiag-SONew),+} 4, and 10.

Revised: Effect of mini-batch size To find the effect of mini-batch size, in Table 7, We empirically compare

Revised: SONew with state of the art first-order methods such as Adam and RMSProp, and second-order method Shampoo.

Revised: We see that SONew performance doesn’t deteriorate much when using smaller or larger batch size.

Revised: First order methods on the other hand suffer significantly.

Original:  Effect of Numerical Stability Algorithm 3 On tridiag-SONew and banded-4-SONew, we observe that using Algorithm 3 improves training loss.
Revised: We also notice that Shampoo doesn’t perform better than SONew in these regimes. Effect of Numerical Stability Algorithm 3 On tridiag-SONew and banded-4-SONew, we observe that using Algorithm 3 improves training loss.

Original: We present in Table 4 results where we observed significant performance improvements.
Revised: We present in Table 4 results where we observed significant performance improvements.

Original: [Table] [Table]{+Large Scale Benchmark Comparison To test efficacy of SONew in Deep Learning, we test our method against Adam on 3 large scale benchmarks - Resnet
+}
Revised: {+Large Scale Benchmark Comparison To test efficacy of SONew in Deep Learning, we test our method against Adam on 3 large scale benchmarks - Resnet
+}

Original: [Table]
Revised: He et al.

Original: Large Scale Benchmark Comparison To test efficacy of SONew in Deep Learning, we test our method against Adam on 3 large scale benchmarks, given in Table 5.
Revised: (2015b) on Imagenet Deng et al. (2009) training, Vision Transformer Dosovitskiy et al. (2020) on Imagenet training, and GraphNetwork Battaglia et al. ; Godwin* et al.

Original:  We see that SONew performs comparable or outperforms Adam in all the benchmark in both train loss and validation performance.
Revised: (2020) on OGBG-molpcba dataset Hu et al. The numbers are given in Table 5. We see that SONew performs comparable or outperforms Adam in all the benchmark in both train loss and validation performance.

Original: Moreover [Table] we plot the training and validation curve in Figure 2.
Revised: Moreover, we plot the training and validation curve in Figure 2.

Original: We observe that SONew has an early advantage over Adam
Revised: We observe that SONew has an early advantage over Adam.

Original: [Table]

Original: Hyperparaeter search space We provide the hyperparamter search space for experiments presented in Section 6.
Revised: Hyperparaeter search space We provide the hyperparamter search space for experiments presented in Section 6.

Original: We search over 2 k hyperparameters for each experiment using a Bayesian Optimization package.
Revised: We search over 2 k hyperparameters for each experiment using a Bayesian Optimization package.

Original: The search ranges are: first order momentum term β 1 ∈ [1 e − 1 , 0 . 999] , second order momentum term β 2 ∈ [1 e − 1 , 0 . 999] , learning rate ∈
Revised: The search ranges are: first order momentum term β 1 ∈ [1 e − 1 , 0 . 999] , second order momentum term β 2 ∈ [1 e − 1 , 0 . 999] , learning rate ∈

Original: [1 e − 7 , 1 e − 1] , ϵ ∈ [1 e − 10 , 1 e − 1] .
Revised: [1 e − 7 , 1 e − 1] , ϵ ∈ [1 e − 10 , 1 e − 1] .

Original: We give the optimal hyperparameter value for each experiment in Table 
Revised: We give the optimal hyperparameter value for each experiment in Table 6.

Original: For large scale benchmark (Table 5), we search β 1 , β 2 ∈ [0 . 1 , 0 . 999] , lr ∈ [1 e − 5 , 1 e − 1] , ϵ ∈
Revised: For large scale benchmark (Table 5), we search β 1 , β 2 ∈ [0 . 1 , 0 . 999] , lr ∈ [1 e − 5 , 1 e − 1] , ϵ ∈

Original: [1 e − 8 , 1 e − 4] , weight decay ∈ [1 e − 5 , 1 . 0] , learning rate warmup ∈
Revised: [1 e − 8 , 1 e − 4] , weight decay ∈ [1 e − 5 , 1 . 0] , learning rate warmup ∈

Original: [2% , 5% , 10%] ∗ total train steps.
Revised: [2% , 5% , 10%] ∗ total train steps.

Original: We use cosine learning rate schedule.
Revised: We use cosine learning rate schedule.

Original: For resnet50 imagenet, we also search label smoothing over [0 . 0 , 0 . 2] .
Revised: For resnet50 imagenet, we also search label smoothing over [0 . 0 , 0 . 2] .

Original: Batch size was kept = 1024, 1024, and 512 for Resnet50, Vision Transformer, and OGBG respectively.
Revised: Batch size was kept = 1024, 1024, and 512 for Resnet50, Vision Transformer, and OGBG respectively.

Original: We sweep over 100 hyperparameters in the search space for both SONew and Adam.
Revised: We sweep over 100 hyperparameters in the search space for both SONew and Adam.

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: [Table]