Original: Optimal and Adaptive Monteiro-Svaiter Acceleration
Revised: Optimal and Adaptive Monteiro-Svaiter Acceleration

Original: Abstract
Revised: Abstract

Original: We develop a variant of the Monteiro-Svaiter (MS) acceleration framework that removes the need to solve an expensive implicit equation at every iteration.
Revised: We develop a variant of the Monteiro-Svaiter (MS) acceleration framework that removes the need to solve an expensive implicit equation at every iteration.

Original: Consequently, for any p  2 we improve the complexity of convex optimization with
Revised: Consequently, for any p  2 we improve the complexity of convex optimization with

Original: Lipschitz p th derivative by a logarithmic factor, matching a lower bound.
Revised: Lipschitz p th derivative by a logarithmic factor, matching a lower bound.

Original: We also introduce an MS subproblem solver that requires no knowledge of problem parameters, and implement it as either a second- or ﬁrst-order method via exact linear system solution or MinRes, respectively.
Revised: We also introduce an MS subproblem solver that requires no knowledge of problem parameters, and implement it as either a second- or ﬁrst-order method via exact linear system solution or MinRes, respectively.

Original: On logistic regression our method outperforms previous second-order acceleration schemes, but under-performs Newton’s method; simply iterating our ﬁrst-order adaptive subproblem solver performs comparably to L-BFGS
Revised: On logistic regression our method outperforms previous second-order acceleration schemes, but under-performs Newton’s method; simply iterating our ﬁrst-order adaptive subproblem solver performs comparably to L-BFGS

Original:  Introduction
Revised:  Introduction

Original: We consider the problem of minimizing a convex function f : X !
Revised: We consider the problem of minimizing a convex function f : X !

Original: R over closed convex set
Revised: R over closed convex set

Original: X ✓ R d , given access to an oracle O : X !
Revised: X ✓ R d , given access to an oracle O : X !

Original: X that minimizes a local model of f around a given query point.
Revised: X that minimizes a local model of f around a given query point.

Original: A key motivating example of such an oracle is the cubic-regularized Newton step
Revised: A key motivating example of such an oracle is the cubic-regularized Newton step

Original: O cr ( y ) = argmin x
Revised: O cr ( y ) = argmin x

Original: [Equation]
Revised: [Equation]

Original: i.e., minimizing the second-order Taylor approximation of f around y plus a cubic regularization term.
Revised: i.e., minimizing the second-order Taylor approximation of f around y plus a cubic regularization term.

Original: However, our results apply to additional oracles including a simple gradient step, regularized higher-order Taylor expansions [5, 19, 7, 23, 8, 35, 21, 41, 36, 26], ball-constrained optimization [12], and new adaptive oracles that we develop.
Revised: However, our results apply to additional oracles including a simple gradient step, regularized higher-order Taylor expansions [5, 19, 7, 23, 8, 35, 21, 41, 36, 26], ball-constrained optimization [12], and new adaptive oracles that we develop.

Original: Seminal work by Monteiro and Svaiter [32] (MS) shows how to accelerate the basic oracle iteration x t +

Revised: Seminal work by Monteiro and Svaiter [32] (MS) shows how to accelerate the basic oracle iteration x t +

Original: = O ( x t ) .
Revised: = O ( x t ) .

Original: Their algorithm is based on the fact that many oracles, including O cr , implicitly approximate proximal points.
Revised: Their algorithm is based on the fact that many oracles, including O cr , implicitly approximate proximal points.

Original: That is, for every y and x = O ( y ) , there exists  x,y > 0 such that x ⇡ argmin x 0 2 X  f ( x 0 )
Revised: That is, for every y and x = O ( y ) , there exists  x,y > 0 such that x ⇡ argmin x 0 2 X  f ( x 0 )

Original: + 12  x,y k x 

Revised: + 12  x,y k x

Original:  y k 2 , with the approximation error controlled by a speciﬁc condition they deﬁne.
Revised:  y k 2 , with the approximation error controlled by a speciﬁc condition they deﬁne.

Original: MS prove that, under this condition, the accelerated proximal point method [22, 40] (with dynamic regularization parameter) maintains its rate of convergence.
Revised: MS prove that, under this condition, the accelerated proximal point method [22, 40] (with dynamic regularization parameter) maintains its rate of convergence.

Original: Applying their framework to O cr and assuming r 2 f is Lipschitz, they achieve error bounds that decay as O ( t  7 / 2 log t ) after t oracle calls, improving the O ( t  2 ) rate of the basic O cr iteration [37] and the O ( t  3 ) rate of an earlier accelerated method [33].
Revised: Applying their framework to O cr and assuming r 2 f is Lipschitz, they achieve error bounds that decay as O ( t  7 / 2 log t ) after t oracle calls, improving the O ( t  2 ) rate of the basic O cr iteration [37] and the O ( t  3 ) rate of an earlier accelerated method [33].

Original: Subsequent works apply variations of the MS framework to different oracles, obtaining improved theoretical guarantees for functions with continuous higher-order derivatives [19, 7, 23, 41, 2], parallel optimization [6], logistic and ` 1 regression [8, 12], minimizing functions with Hölder continuous higher derivatives [41], and distributionally-robust optimization [13, 11].
Revised: Subsequent works apply variations of the MS framework to different oracles, obtaining improved theoretical guarantees for functions with continuous higher-order derivatives [19, 7, 23, 41, 2], parallel optimization [6], logistic and ` 1 regression [8, 12], minimizing functions with Hölder continuous higher derivatives [41], and distributionally-robust optimization [13, 11].

Original: However, all of these algorithms based on the MS framework share a common drawback: the iterate y t used to produce x t +

Revised: However, all of these algorithms based on the MS framework share a common drawback: the iterate y t used to produce x t +

Original: =
Revised: =

Original: O
Revised: O

Original: ( y t ) depends on the proximal parameter  t
Revised: ( y t ) depends on the proximal parameter  t

Original: +1 =
Revised: +1 =

Original:  x t +1 ,y t , which itself depends on both x t +1 and y t .
Revised:  x t +1 ,y t , which itself depends on both x t +1 and y t .

Original: This circular dependence necessitates solving an implicit equation for
Revised: This circular dependence necessitates solving an implicit equation for

Original:  t +1 ; MS (and many subsequent results based upon it) propose bisection procedures for doing so using a number of oracle calls logarithmic in the problem parameters.
Revised:  t +1 ; MS (and many subsequent results based upon it) propose bisection procedures for doing so using a number of oracle calls logarithmic in the problem parameters.

Original: From a theoretical perspective, the additional bisection complexity introduces a logarithmic gap between the upper bounds due to
Revised: From a theoretical perspective, the additional bisection complexity introduces a logarithmic gap between the upper bounds due to

Original: MS-based algorithms and the best known lower bounds [3, 21] in a number of settings.
Revised: MS-based algorithms and the best known lower bounds [3, 21] in a number of settings.

Original: From a practical perspective, the use of bisection in the MS framework is undesirable as it potentially discards the optimization progress made by oracle calls during each bisection.
Revised: From a practical perspective, the use of bisection in the MS framework is undesirable as it potentially discards the optimization progress made by oracle calls during each bisection.

Original: In his textbook,
Revised: In his textbook,

Original: Nesterov [34, §4.3.3] argues that the logarithmic cost of bisection likely renders the MS scheme for accelerating O cr inferior in practice to algorithms whose error decays at the asymptotically worse rate
Revised: Nesterov [34, §4.3.3] argues that the logarithmic cost of bisection likely renders the MS scheme for accelerating O cr inferior in practice to algorithms whose error decays at the asymptotically worse rate

Original: O ( t  3 ) but do not require bisection; he notes that removing the bisection from the MS algorithm is an “open and challenging question in Optimization Theory.”
Revised: O ( t  3 ) but do not require bisection; he notes that removing the bisection from the MS algorithm is an “open and challenging question in Optimization Theory.”

Original: Carmon et al.
Revised: Carmon et al.

Original: [13] also point out bisection as one of the main hurdles in making their theoretical scheme practical, while Song et al.
Revised: [13] also point out bisection as one of the main hurdles in making their theoretical scheme practical, while Song et al.

Original: [41] note this limitation and propose a heuristic alternative to bisection.
Revised: [41] note this limitation and propose a heuristic alternative to bisection.

Original: (See Appendix A for extended discussion of related work, including a concurrent and independent result by Kovalev and Gasnikov [27].)
Revised: (See Appendix A for extended discussion of related work, including a concurrent and independent result by Kovalev and Gasnikov [27].)

Original: 1.1 Our contributions
Revised: 1.1 Our contributions

Original: We settle this open question, providing a variant of MS acceleration that does not require bisection (Section 2).
Revised: We settle this open question, providing a variant of MS acceleration that does not require bisection (Section 2).

Original: When combined with certain existing MS oracles (Section 3.1), our algorithm obtains complexity bounds that are optimal up to constant factors, improving over prior art by a logarithmic factor (see Table 1).
Revised: When combined with certain existing MS oracles (Section 3.1), our algorithm obtains complexity bounds that are optimal up to constant factors, improving over prior art by a logarithmic factor (see Table 1).

Original: In addition, our algorithm has no parameters sensitive to tuning.
Revised: In addition, our algorithm has no parameters sensitive to tuning.

Original: We then go a step further and (in Section 3.2) develop an adaptive alternative to O cr (Equation (1)).
Revised: We then go a step further and (in Section 3.2) develop an adaptive alternative to O cr (Equation (1)).

Original: Our oracle does not require tuning the parameter M , which in theory should be proportional to the (difﬁcult to estimate)
Revised: Our oracle does not require tuning the parameter M , which in theory should be proportional to the (difﬁcult to estimate)

Original: Lipschitz constant of r 2 f .
Revised: Lipschitz constant of r 2 f .

Original: Using our oracle, we obtain the optimal
Revised: Using our oracle, we obtain the optimal

Original: Hessian evaluation complexity O ( t  (4+3 ⌫ ) / 2 ) for functions with order- ⌫ Hölder Hessian (Lipschitz
Revised: Hessian evaluation complexity O ( t  (4+3 ⌫ ) / 2 ) for functions with order- ⌫ Hölder Hessian (Lipschitz

Original: Hessian is the ⌫ = 1 special case), without requiring any knowledge of the Hölder constant and order ⌫ .
Revised: Hessian is the ⌫ = 1 special case), without requiring any knowledge of the Hölder constant and order ⌫ .

Original: Our oracle is also efﬁcient: while existing complexity bounds for computing O cr require a logarithmic number of linear system solutions per call, our oracle requires a double-logarithmic number.
Revised: Our oracle is also efﬁcient: while existing complexity bounds for computing O cr require a logarithmic number of linear system solutions per call, our oracle requires a double-logarithmic number.

Original: Moreover, when used with our acceleration method, the number of linear system solves per iteration is essentially constant.
Revised: Moreover, when used with our acceleration method, the number of linear system solves per iteration is essentially constant.

Original: We also provide a ﬁrst-order implementation of our adaptive oracle (Section 3.3).
Revised: We also provide a ﬁrst-order implementation of our adaptive oracle (Section 3.3).

Original: It approximately solves linear systems via ﬁrst-order operations (Hessian-vector products) using MinRes/Conjugate
Revised: It approximately solves linear systems via ﬁrst-order operations (Hessian-vector products) using MinRes/Conjugate

Original: Residuals [42, 18] with a simple, adaptive, stopping criterion lifted directly from our analysis.
Revised: Residuals [42, 18] with a simple, adaptive, stopping criterion lifted directly from our analysis.

Original: Our oracle attains the optimal ﬁrst-order evaluation complexity for smooth functions up to an additive logarithmic term, without knowledge of the gradient Lipschitz constant or any parameter tuning.
Revised: Our oracle attains the optimal ﬁrst-order evaluation complexity for smooth functions up to an additive logarithmic term, without knowledge of the gradient Lipschitz constant or any parameter tuning.

Original: Moreover, it maintains an optimal outer iteration complexity for Hölder Hessian of any order.
Revised: Moreover, it maintains an optimal outer iteration complexity for Hölder Hessian of any order.

Original: Finally, we report empirical results (Section 4).
Revised: Finally, we report empirical results (Section 4).

Original: On logistic regression problems, combining our optimal acceleration scheme with our adaptive oracle outperforms previously proposed accelerated second-order methods.
Revised: On logistic regression problems, combining our optimal acceleration scheme with our adaptive oracle outperforms previously proposed accelerated second-order methods.

Original: However, we also show that (while somewhat helpful for O cr with a conservative choice of H ), adding momentum to well-tuned or adaptive second-order methods is harmful in logistic regression: simply iterating our oracle—or, better yet, applying Newton’s method—dramatically outperforms all “accelerated” algorithms.
Revised: However, we also show that (while somewhat helpful for O cr with a conservative choice of H ), adding momentum to well-tuned or adaptive second-order methods is harmful in logistic regression: simply iterating our oracle—or, better yet, applying Newton’s method—dramatically outperforms all “accelerated” algorithms.

Original: This important fact seems to have gone unobserved in the literature on accelerated second-order methods, despite logistic regression appearing in many related experiments [41, 16, 30, 24].
Revised: This important fact seems to have gone unobserved in the literature on accelerated second-order methods, despite logistic regression appearing in many related experiments [41, 16, 30, 24].

Original: Simply iterating our adaptive oracle outperforms the classical accelerated gradient descent, and performs comparably to L-BFGS.
Revised: Simply iterating our adaptive oracle outperforms the classical accelerated gradient descent, and performs comparably to L-BFGS.

Original: 1.2 Limitations and outlook
Revised: 1.2 Limitations and outlook

Original: While our algorithms resolve an enduring theoretical open problem in convex optimization, and are free of sensitive parameters that typically hinder theoretically-optimal methods, practical performance remains a limitation.
Revised: While our algorithms resolve an enduring theoretical open problem in convex optimization, and are free of sensitive parameters that typically hinder theoretically-optimal methods, practical performance remains a limitation.

Original: On logistic regression, Newton’s method is remarkably fast, and our acceleration scheme does not seem to help our adaptive oracle.
Revised: On logistic regression, Newton’s method is remarkably fast, and our acceleration scheme does not seem to help our adaptive oracle.

Original: We do not fully understand why this is so, but we suspect that it has to do with additional structure in logistic regression, which Newton’s method can automatically exploit but momentum cannot.
Revised: We do not fully understand why this is so, but we suspect that it has to do with additional structure in logistic regression, which Newton’s method can automatically exploit but momentum cannot.

Original: We believe that future research should identify the structure that makes Newton’s method so efﬁcient, and modifying momentum schemes to leverage it.
Revised: We believe that future research should identify the structure that makes Newton’s method so efﬁcient, and modifying momentum schemes to leverage it.

Original: Scalability is another important limitation.
Revised: Scalability is another important limitation.

Original: While our ﬁrst-order oracle signiﬁcantly improves scalability over the second-order oracle from which it is built, it still relies on exact gradient and
Revised: While our ﬁrst-order oracle signiﬁcantly improves scalability over the second-order oracle from which it is built, it still relies on exact gradient and

Original: Hessian-vector products.
Revised: Hessian-vector products.

Original: Therefore, it will have difﬁculty scaling up to very large datasets.
Revised: Therefore, it will have difﬁculty scaling up to very large datasets.

Original: Nevertheless, we hope that further scalability improvements may be possible by building an oracle that utilizes cheap stochastic gradient estimates instead of exact gradients, bringing with it the exciting prospect of a new and powerful adaptive stochastic gradient method.
Revised: Nevertheless, we hope that further scalability improvements may be possible by building an oracle that utilizes cheap stochastic gradient estimates instead of exact gradients, bringing with it the exciting prospect of a new and powerful adaptive stochastic gradient method.

Original: The alternative, probabilistic approximation condition we propose in Appendix C might be helpful in this regard.
Revised: The alternative, probabilistic approximation condition we propose in Appendix C might be helpful in this regard.

Original:  Removing bisection from the Monteiro-Svaiter framework
Revised:  Removing bisection from the Monteiro-Svaiter framework

Original: Input: Initial x 0 , function f , oracle O
Revised: Input: Initial x 0 , function f , oracle O

Original: Parameters: Initial  0 0 , multiplicative adjustment factor ↵ >1 Set v 0 = x 0 , A 0 =2 ˜ x 1 ,  1 =
Revised: Parameters: Initial  0 0 , multiplicative adjustment factor ↵ >1 Set v 0 = x 0 , A 0 =2 ˜ x 1 ,  1 =

Original: O ( x 0 ;  0 0 ) ,  0 1 = 3 for t = 0 , 1 , . .
Revised: O ( x 0 ;  0 0 ) ,  0 1 = 3 for t = 0 , 1 , . .

Original: , do a 0 t +1 = 1 2  0 t +  1 + p 1 + 4  0 t +1 A t 
Revised: , do a 0 t +1 = 1 2  0 t +  1 + p 1 + 4  0 t +1 A t

Original: A 0 t +1 =
Revised: A 0 t +1 =

Original: A t + a 0 t +  y t
Revised: A t + a 0 t +  y t

Original: =
Revised: =

Original: A t A 0 t +1 x t + a 0 t +1 A 0 t +1 v t if t > 0 then ˜ x t +1 ,  t
Revised: A t A 0 t +1 x t + a 0 t +1 A 0 t +1 v t if t > 0 then ˜ x t +1 ,  t

Original: +1 =
Revised: +1 =

Original: O ( y t ;  0 t +1 )
Revised: O ( y t ;  0 t +1 )

Original: if  t +1   0 t +1 then a t +1 = a 0 t +1 , A t +1 =
Revised: if  t +1   0 t +1 then a t +1 = a 0 t +1 , A t +1 =

Original: A 0 t +  x t
Revised: A 0 t +  x t

Original: +

Revised: +

Original: =
Revised: =

Original: ˜ x t +   0 t +2 =
Revised: ˜ x t +   0 t +2 =

Original: 1 ↵  0 t +  else  t +1 =
Revised: 1 ↵  0 t +  else  t +1 =

Original:  0 t +1  t +  a t
Revised:  0 t +1  t +  a t

Original: +1 =
Revised: +1 =

Original:  t +1 a 0 t +1 , A t +1 =
Revised:  t +1 a 0 t +1 , A t +1 =

Original: A t + a t +  x t
Revised: A t + a t +  x t

Original: +

Revised: +

Original: =
Revised: =

Original: (1   t +1 )
Revised: (1   t +1 )

Original: A t
Revised: A t

Original: A t + x t +  t +1 A 0 t +1 A t + ˜ x t +   0 t +2 =
Revised: A t + x t +  t +1 A 0 t +1 A t + ˜ x t +   0 t +2 =

Original: ↵ 0 t +  v t +1 =
Revised: ↵ 0 t +  v t +1 =

Original: v t  a t +1 r f (˜ x t +1 )
Revised: v t  a t +1 r f (˜ x t +1 )

Original: Algorithm
Revised: Algorithm

Original: Input: Initial x 0 , function f , oracle O
Revised: Input: Initial x 0 , function f , oracle O

Original: Parameters: Bisection limits  ` ,  h , and tolerance ⇢ >1 Set v 0 = x 0 , A 

Revised: Parameters: Bisection limits  ` ,  h , and tolerance ⇢ >1 Set v 0 = x 0 , A

Original: =2 for t = 0 , 1 , . .
Revised: =2 for t = 0 , 1 , . .

Original: , do  ` t +1 ,  h t
Revised: , do  ` t +1 ,  h t

Original: +1 =
Revised: +1 =

Original:  ` ,  h  0 t +1 =
Revised:  ` ,  h  0 t +1 =

Original:  ` t +

Revised:  ` t +

Original: +
Revised: +

Original: 
Revised:

Original: ht +1 a 0 t +1 = 1 2  0 t +  1 + p 1 + 4  0 t +

Revised: ht +1 a 0 t +1 = 1 2  0 t +  1 + p 1 + 4  0 t +

Original: A t 
Revised: A t

Original: A 0 t +1 =
Revised: A 0 t +1 =

Original: A t + a 0 t +  y t
Revised: A t + a 0 t +  y t

Original: =
Revised: =

Original: A t A 0 t +1 x t + a 0 t +1 A 0 t +1 v t ˜ x t +1 ,  t
Revised: A t A 0 t +1 x t + a 0 t +1 A 0 t +1 v t ˜ x t +1 ,  t

Original: +1 =
Revised: +1 =

Original: O ( y t ;  0 t +1 )
Revised: O ( y t ;  0 t +1 )

Original: if  t +1 2 [ 1 ⇢  0 t
Revised: if  t +1 2 [ 1 ⇢  0 t

Original: +1 ,  0 t
Revised: +1 ,  0 t

Original: +1 ] then a t +1 = a 0 t +1 , A t +1 =
Revised: +1 ] then a t +1 = a 0 t +1 , A t +1 =

Original: A 0 t +  x t
Revised: A 0 t +  x t

Original: +

Revised: +

Original: =
Revised: =

Original: ˜ x t +  else if  t +1 < 1 ⇢  0 t +1 then  ht +

Revised: ˜ x t +  else if  t +1 < 1 ⇢  0 t +1 then  ht +

Original: =  0 t +
Revised: =  0 t +

Original: Go to line else
Revised: Go to line else

Original:  lt +1 =
Revised:  lt +1 =

Original:  0 t +
Revised:  0 t +

Original: Go to line v t +1 = v t  a t +1 r f (˜ x t +1 )
Revised: Go to line v t +1 = v t  a t +1 r f (˜ x t +1 )

Original: In this section we present our acceleration algorithm
Revised: In this section we present our acceleration algorithm

Original: (Algorithm 1) which removes bisection from the
Revised: (Algorithm 1) which removes bisection from the

Original: MS method (shown in stylized form as Algorithm 0) and thereby attains optimal rates of convergence.
Revised: MS method (shown in stylized form as Algorithm 0) and thereby attains optimal rates of convergence.

Original: For simplicity, in this section and the next we focus on unconstrained optimization ( X = R d ) and assume that f is continuously differentiable, so that r f exists.
Revised: For simplicity, in this section and the next we focus on unconstrained optimization ( X = R d ) and assume that f is continuously differentiable, so that r f exists.

Original: In Appendix C we extend our framework to general closed and convex domains and non-differentiable convex objectives.
Revised: In Appendix C we extend our framework to general closed and convex domains and non-differentiable convex objectives.

Original: The key object in both the original MS algorithm and our new variant is an oracle O that approximately minimizes a local model of f at a query point y .
Revised: The key object in both the original MS algorithm and our new variant is an oracle O that approximately minimizes a local model of f at a query point y .

Original: In particular, O satisﬁes the following approximation error bound, adapted from Monteiro and Svaiter [32, eq.
Revised: In particular, O satisﬁes the following approximation error bound, adapted from Monteiro and Svaiter [32, eq.

Original: (  in [32] is 1 /  in our notation).
Revised: (  in [32] is 1 /  in our notation).

Original: Deﬁnition 1 (MS oracle) .
Revised: Deﬁnition 1 (MS oracle) .

Original: An oracle O : R d ⇥ R + !
Revised: An oracle O : R d ⇥ R + !

Original: R d ⇥ R
Revised: R d ⇥ R

Original: + is a  -MS oracle for function f :
Revised: + is a  -MS oracle for function f :

Original: R d !
Revised: R d !

Original: R if for every y 2 R d and  0 > 0 , the points ( x,  )
Revised: R if for every y 2 R d and  0 > 0 , the points ( x,  )

Original: =
Revised: =

Original: O ( y ;  0 ) satisfy
Revised: O ( y ;  0 ) satisfy

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition 1 endows the oracle with an additional output  and an additional input  0 .
Revised: Deﬁnition 1 endows the oracle with an additional output  and an additional input  0 .

Original: The value of  has the following simple interpretation: any point x satisfying (2) approximately minimizes
Revised: The value of  has the following simple interpretation: any point x satisfying (2) approximately minimizes

Original: F ( x 0 ) =
Revised: F ( x 0 ) =

Original: f ( x 0 )
Revised: f ( x 0 )

Original: +  2 k x 0  y k 2 in the sense that kr F ( x ) k   k x  y k .
Revised: +  2 k x 0  y k 2 in the sense that kr F ( x ) k   k x  y k .

Original: In particular, computing an exact proximal point x  = argmin x 0 F ( x 0 ) and outputting ( x,  ) implements a 0 -MS oracle.
Revised: In particular, computing an exact proximal point x  = argmin x 0 F ( x 0 ) and outputting ( x,  ) implements a 0 -MS oracle.

Original: The input  0 is optional: oracle implementations in prior work do not require it, but our new adaptive oracles (described in the next section) use it for improved efﬁciency.
Revised: The input  0 is optional: oracle implementations in prior work do not require it, but our new adaptive oracles (described in the next section) use it for improved efﬁciency.

Original: In Appendix C we provide a slightly more general approximation condition for MS oracles that handles non-smooth objectives and bounded domains, as well as a different, stochastic condition similar to that of [4, 11].
Revised: In Appendix C we provide a slightly more general approximation condition for MS oracles that handles non-smooth objectives and bounded domains, as well as a different, stochastic condition similar to that of [4, 11].

Original: Let us discuss the key differences between our algorithm (Algorithm 1) and the stylized MS algorithm (Algorithm 0).
Revised: Let us discuss the key differences between our algorithm (Algorithm 1) and the stylized MS algorithm (Algorithm 0).

Original: At every iteration, Algorithm 0 searches for a value  0 t +1 such that ˜ x t +1 ,  t +1 =
Revised: At every iteration, Algorithm 0 searches for a value  0 t +1 such that ˜ x t +1 ,  t +1 =

Original: O ( y t ;  0 t +1 ) satisﬁes  t +1 ⇡  0 t +1 (note that y t depends on  0 t +1 ).
Revised: O ( y t ;  0 t +1 ) satisﬁes  t +1 ⇡  0 t +1 (note that y t depends on  0 t +1 ).

Original: This is done via a bisection procedure iteratively shrinking an interval that contains a successful choice of  0 t
Revised: This is done via a bisection procedure iteratively shrinking an interval that contains a successful choice of  0 t

Original: +1 .
Revised: +1 .

Original: This bisection process is inefﬁcient in the sense that every time we reach lines 14 and 17 (highlighted in red) all of the optimization progress made by the last oracle call is discarded.
Revised: This bisection process is inefﬁcient in the sense that every time we reach lines 14 and 17 (highlighted in red) all of the optimization progress made by the last oracle call is discarded.

Original: In contrast, even though our algorithm queries O in the same way (with y t computed based on a guess  0 t +1 ), it makes use of the oracle output even if  t +1 is very far from  0 t +1 , thus never discarding progress made by the oracle.
Revised: In contrast, even though our algorithm queries O in the same way (with y t computed based on a guess  0 t +1 ), it makes use of the oracle output even if  t +1 is very far from  0 t +1 , thus never discarding progress made by the oracle.

Original: Instead of performing a bisection, we compare  0 t +1 and  t
Revised: Instead of performing a bisection, we compare  0 t +1 and  t

Original: +1 to guide our next guess  0 t +2 .
Revised: +1 to guide our next guess  0 t +2 .

Original: When  0 overshoots  , we decrease it by a factor ↵ (line 11, highlighted in green) and set x t and A t as in Algorithm 0.
Revised: When  0 overshoots  , we decrease it by a factor ↵ (line 11, highlighted in green) and set x t and A t as in Algorithm 0.

Original: When it undershoots, we multiply it by ↵ (line 16).
Revised: When it undershoots, we multiply it by ↵ (line 16).

Original: In this case, we perform an additional key algorithmic modiﬁcation which we call the momentum damping mechanism : we scale down the growth of the parameter A t +1 and replace the next iterate with a convex combination of x t and
Revised: In this case, we perform an additional key algorithmic modiﬁcation which we call the momentum damping mechanism : we scale down the growth of the parameter A t +1 and replace the next iterate with a convex combination of x t and

Original: ˜ x t +1 to ensure that our overly optimistic guess for  t
Revised: ˜ x t +1 to ensure that our overly optimistic guess for  t

Original: +1 does not destabilize the algorithm.
Revised: +1 does not destabilize the algorithm.

Original: In Appendix E.6 we demonstrate empirically that this mechanism is important for stabilizing Algorithm 1.
Revised: In Appendix E.6 we demonstrate empirically that this mechanism is important for stabilizing Algorithm 1.

Original: Different MS oracles attain different rates of convergence when accelerated via the MS framework.
Revised: Different MS oracles attain different rates of convergence when accelerated via the MS framework.

Original: In the following deﬁnition, we distill a key property that determines this rate.
Revised: In the following deﬁnition, we distill a key property that determines this rate.

Original: Deﬁnition 2 (Movement bound) .
Revised: Deﬁnition 2 (Movement bound) .

Original: For s  1 , c,  > 0 , and x, y 2 R d we say that ( x, y,  ) satisfy a ( s, c ) -movement bound if
Revised: For s  1 , c,  > 0 , and x, y 2 R d we say that ( x, y,  ) satisfy a ( s, c ) -movement bound if

Original: [Equation]
Revised: [Equation]

Original: where a (1 , c ) -movement bound simply means that   c .
Revised: where a (1 , c ) -movement bound simply means that   c .

Original: In the next section, we will show how to build MS oracles that, given query y , output ( x,  ) such that ( x, y,  ) always satisfy a ( s, c ) -movement bound, for certain s and c depending on the oracle type and function structure (e.g., level of smoothness).
Revised: In the next section, we will show how to build MS oracles that, given query y , output ( x,  ) such that ( x, y,  ) always satisfy a ( s, c ) -movement bound, for certain s and c depending on the oracle type and function structure (e.g., level of smoothness).

Original: For example, when f has H -Lipschitz Hessian, the cubic-regularized Newton step with M = 2 H is a 12 -MS oracle that guarantees a (2 , p H ) -movement bound.
Revised: For example, when f has H -Lipschitz Hessian, the cubic-regularized Newton step with M = 2 H is a 12 -MS oracle that guarantees a (2 , p H ) -movement bound.

Original: With the necessary deﬁnitions in hand, we are ready to state our main result: the iteration (and MS oracle query) complexity of Algorithm 1.
Revised: With the necessary deﬁnitions in hand, we are ready to state our main result: the iteration (and MS oracle query) complexity of Algorithm 1.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let f : R d !
Revised: Let f : R d !

Original: R be convex and differentiable, and consider Algorithm 1 with parameters ↵ > 1 ,  0 > 0 , and a  -MS oracle (Deﬁnition 1) for f with  2 [0 , 0 . 99) .
Revised: R be convex and differentiable, and consider Algorithm 1 with parameters ↵ > 1 ,  0 > 0 , and a  -MS oracle (Deﬁnition 1) for f with  2 [0 , 0 . 99) .

Original: Let s  1 and c > 0 , and suppose that for all t such that  t >  0 t or t = 1 , the iterates (˜ x t , y t  1 ,  t ) satisfy a ( s, c )
Revised: Let s  1 and c > 0 , and suppose that for all t such that  t >  0 t or t = 1 , the iterates (˜ x t , y t  1 ,  t ) satisfy a ( s, c )

Original: -movement bound (Deﬁnition 2).
Revised: -movement bound (Deﬁnition 2).

Original: There exist C ↵ ,s =
Revised: There exist C ↵ ,s =

Original: O ⇣ s min { s, ln ↵ } ↵ s +1 3 s + ⌘ and K ↵ =
Revised: O ⇣ s min { s, ln ↵ } ↵ s +1 3 s + ⌘ and K ↵ =

Original: O  1 ln ↵ ↵ 1 / such thatfor any x ? 2 R d and ✏ > 0 , we have f ( x T )  f ( x ? )  ✏ when
Revised: O  1 ln ↵ ↵ 1 / such thatfor any x ? 2 R d and ✏ > 0 , we have f ( x T )  f ( x ? )  ✏ when

Original: [Equation]
Revised: [Equation]

Original: Proof sketch.
Revised: Proof sketch.

Original: The remainder of this section is an overview of the proof of Theorem 1, which we provide in full in Appendix B. To simplify this proof sketch we treat ↵ , c and 1 / (1   ) as O (1) , and focus on s < 1 .
Revised: The remainder of this section is an overview of the proof of Theorem 1, which we provide in full in Appendix B. To simplify this proof sketch we treat ↵ , c and 1 / (1   ) as O (1) , and focus on s < 1 .

Original: To highlight the novel aspects of the proof, let us ﬁrst brieﬂy recall the analysis of Algorithm 

Revised: To highlight the novel aspects of the proof, let us ﬁrst brieﬂy recall the analysis of Algorithm

Original: For every t  T let
Revised: For every t  T let

Original: [Equation]
Revised: [Equation]

Original: The key facts about the standard MS iterations are
Revised: The key facts about the standard MS iterations are

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst fact implies that the optimality gap at iteration T is inversely proportional to A T , while the latter two facts imply that A T grows rapidly.
Revised: The ﬁrst fact implies that the optimality gap at iteration T is inversely proportional to A T , while the latter two facts imply that A T grows rapidly.

Original: More speciﬁcally, substituting the movement bound
Revised: More speciﬁcally, substituting the movement bound

Original: M t  ⌦
Revised: M t  ⌦

Original: [Equation]
Revised: [Equation]

Original:  and  0 t  ⌦ (  t ) (thanks to the bisection) yields
Revised:  and  0 t  ⌦ (  t ) (thanks to the bisection) yields

Original: [Equation]
Revised: [Equation]

Original: Combining this with the third fact in (4) and using the reverse Hölder inequality allows one to conclude that, for k = s +1 3 s +1 and k 

Revised: Combining this with the third fact in (4) and using the reverse Hölder inequality allows one to conclude that, for k = s +1 3 s +1 and k

Original: = s  1 3 s +1 , we have A kT  ⌦ ( D  k 0)
Revised: = s  1 3 s +1 , we have A kT  ⌦ ( D  k 0)

Original: P t
Revised: P t

Original: [Equation]
Revised: [Equation]

Original: T ]
Revised: T ]

Original: A k 0 i , which, upon further algebraic manipulation, yields A T  ⌦ ( T (3 s +1) / 2 D  ( s  1) / 2) .
Revised: A k 0 i , which, upon further algebraic manipulation, yields A T  ⌦ ( T (3 s +1) / 2 D  ( s  1) / 2) .

Original: Plugging this back to to the ﬁrst fact in (4) gives the claimed convergence rate.
Revised: Plugging this back to to the ﬁrst fact in (4) gives the claimed convergence rate.

Original: Having described the standard MS analysis, we move on to our algorithm.
Revised: Having described the standard MS analysis, we move on to our algorithm.

Original: Our ﬁrst challenge is re-establishing the facts (4).
Revised: Our ﬁrst challenge is re-establishing the facts (4).

Original: The difﬁcult case is  t >  0 t , where the standard cancellation that occurs in the MS analysis may fail.
Revised: The difﬁcult case is  t >  0 t , where the standard cancellation that occurs in the MS analysis may fail.

Original: This is where the momentum damping mechanism (lines 14 and 15 of our algorithm) comes into play, allowing us to show that (See Proposition 1 in the appendix)
Revised: This is where the momentum damping mechanism (lines 14 and 15 of our algorithm) comes into play, allowing us to show that (See Proposition 1 in the appendix)

Original: [Equation]
Revised: [Equation]

Original: T  D
Revised: T  D

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and
Revised: and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where S  T :
Revised: where S  T :

Original: = { t 2 [ T ] |  t   0 t } and S >T , S  T , S <T and S =
Revised: = { t 2 [ T ] |  t   0 t } and S >T , S  T , S <T and S =

Original: T are analogously deﬁned.
Revised: T are analogously deﬁned.

Original: Comparing (4) and (5), the price of removing the bisection becomes evident: at each iteration (except the ﬁrst) only one of the terms forcing the growth of A t receives a contribution.
Revised: Comparing (4) and (5), the price of removing the bisection becomes evident: at each iteration (except the ﬁrst) only one of the terms forcing the growth of A t receives a contribution.

Original: The second challenge of our proof is establishing a lower bound on p A T in terms of the 1 / p  0 t values for t 2 S >T [ { 1 } , where the movement bound holds for M t .
Revised: The second challenge of our proof is establishing a lower bound on p A T in terms of the 1 / p  0 t values for t 2 S >T [ { 1 } , where the movement bound holds for M t .

Original: This is where the multiplicative  0 update rule (lines 11 and 16 of the algorithm) comes into play: it allows us to “credit” the contribution of every “down iterate” (in S  T ) to an adjacent “up” iterate ( S >T [ { 1 } ) and furthermore argue that the contribution gets an exponential bonus based on the distance between the two.
Revised: This is where the multiplicative  0 update rule (lines 11 and 16 of the algorithm) comes into play: it allows us to “credit” the contribution of every “down iterate” (in S  T ) to an adjacent “up” iterate ( S >T [ { 1 } ) and furthermore argue that the contribution gets an exponential bonus based on the distance between the two.

Original: Consequently, we are able to identify a set Q T ✓ S >T [ { 1 } of iterates, and a sequence { r t } such that (see Lemma 1)
Revised: Consequently, we are able to identify a set Q T ✓ S >T [ { 1 } of iterates, and a sequence { r t } such that (see Lemma 1)

Original: [Equation]
Revised: [Equation]

Original: 
Revised:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Repeating the reverse Hölder argument of prior work, we obtain the recursive bound
Revised: Repeating the reverse Hölder argument of prior work, we obtain the recursive bound

Original: [Equation]
Revised: [Equation]

Original: with k = s +1 3 s +

Revised: with k = s +1 3 s +

Original: and k 

Revised: and k

Original: = s  1 3 s +1 as before.
Revised: = s  1 3 s +1 as before.

Original: The ﬁnal challenge of our proof is to show that such recursion implies sufﬁcient growth of A t .
Revised: The ﬁnal challenge of our proof is to show that such recursion implies sufﬁcient growth of A t .

Original: This is where careful algebra comes into play; we show that (6) implies that
Revised: This is where careful algebra comes into play; we show that (6) implies that

Original: [Equation]
Revised: [Equation]

Original: ⌘ (see Lemmas 3 and 4) which establishes our result since
Revised: ⌘ (see Lemmas 3 and 4) which establishes our result since

Original: [Equation]
Revised: [Equation]

Original:  MS oracle implementations
Revised:  MS oracle implementations

Original: In this section we describe several oracles that satisfy both Deﬁnition 1 (the MS condition) and
Revised: In this section we describe several oracles that satisfy both Deﬁnition 1 (the MS condition) and

Original: Deﬁnition 2 (movement bounds) and may therefore be used by Algorithm 1.
Revised: Deﬁnition 2 (movement bounds) and may therefore be used by Algorithm 1.

Original: Section 3.1 brieﬂy reviews oracles that have appeared in prior work, while Section 3.2 and Section 3.3 describe our new adaptive oracle implementations.
Revised: Section 3.1 brieﬂy reviews oracles that have appeared in prior work, while Section 3.2 and Section 3.3 describe our new adaptive oracle implementations.

Original: We summarize the key oracle properties and resulting complexity bounds in Table 1.
Revised: We summarize the key oracle properties and resulting complexity bounds in Table 1.

Original: 3.1 Oracles from prior work
Revised: 3.1 Oracles from prior work

Original: Here we consider several previously-studied oracles of the form ( x,  ) =
Revised: Here we consider several previously-studied oracles of the form ( x,  ) =

Original: O ( y ) , where we omit the second argument  0 since prior work does not leverage it to improve implementation efﬁciency.
Revised: O ( y ) , where we omit the second argument  0 since prior work does not leverage it to improve implementation efﬁciency.

Original: Assumption
Revised: Assumption

Original: Oracle
Revised: Oracle

Original: Complexity with Algorithm
Revised: Complexity with Algorithm

Revised: Lower

Original: [Table]
Revised: [Table]

Original: O ⇣
Revised: O ⇣

Original: [Table]
Revised: [Table]

Original:  ⌦
Revised: f ⌦

Original: [Table]
Revised: [Table]

Original: ⌘ [ [Table] 
Revised: [Table] ⌘ [3, 21]

Original: [Table]
Revised: [Table]

Original: ⇣
Revised: ⇣

Original: [Table]
Revised: [Table]

Revised: ⌘

Revised: Hessian

Original: [Table]
Revised: [Table]

Original:  [Table]
Revised: ⌦ [Table] [Table] [Table]

Original: is
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Lipschitz
Revised: r f is ` -Lipschitz and

Original: [Table]
Revised: [Table]

Original: order ⌫ 2 [0 , 1] .
Revised: [Table]

Original: Gradient descent step [e.g., 34].
Revised: Gradient descent step [e.g., 34].

Original: As a gentle start, consider the oracle O gd ( y )
Revised: As a gentle start, consider the oracle O gd ( y )

Original: =
Revised: =

Original: ( y  ⌘ r f ( y ) , 1 ⌘ ) , i.e., an oracle that returns x by taking standard gradient step with size ⌘ and  = 1 / ⌘ .
Revised: ( y  ⌘ r f ( y ) , 1 ⌘ ) , i.e., an oracle that returns x by taking standard gradient step with size ⌘ and  = 1 / ⌘ .

Original: Obviously, the oracle always satisﬁes a (1 , ⌘  1 ) -movement bound.
Revised: Obviously, the oracle always satisﬁes a (1 , ⌘  1 ) -movement bound.

Original: Moreover, if we assume that r f is L -Lipschitz, then k x  ( y  1  r f ( x ))
Revised: Moreover, if we assume that r f is L -Lipschitz, then k x  ( y  1  r f ( x ))

Original: k
Revised: k

Original: = ⌘ kr f ( x )  r f ( y ) k  ⌘ L k x  y k .
Revised: = ⌘ kr f ( x )  r f ( y ) k  ⌘ L k x  y k .

Original: Therefore, when ⌘  1  L/  the oracle is a  -MS oracle.
Revised: Therefore, when ⌘  1  L/  the oracle is a  -MS oracle.

Original: Taylor descent step [5, 35, 19, 7, 23, 41].
Revised: Taylor descent step [5, 35, 19, 7, 23, 41].

Original: Generalizing both O gd and the cubic-regularized Newton step oracle O cr , we deﬁne for every integer p  1 and ⌫ 2 [0 , 1] the oracle O p, ⌫ -reg , that, for parameter C and input y returns ( x,  ) =
Revised: Generalizing both O gd and the cubic-regularized Newton step oracle O cr , we deﬁne for every integer p  1 and ⌫ 2 [0 , 1] the oracle O p, ⌫ -reg , that, for parameter C and input y returns ( x,  ) =

Original: O p, ⌫ -reg ( y ) where
Revised: O p, ⌫ -reg ( y ) where

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and ˜ f p ( x ; y ) :=
Revised: and ˜ f p ( x ; y ) :=

Original: P pi =

Revised: P pi =

Original: 1 i !
Revised: 1 i !

Original: r i f ( y )
Revised: r i f ( y )

Original: [( x  y ) ⌦ i ] is the Taylor expansion of f around y evaluated at x .
Revised: [( x  y ) ⌦ i ] is the Taylor expansion of f around y evaluated at x .

Original: Oracles O gd and O cr correspond to the special cases
Revised: Oracles O gd and O cr correspond to the special cases

Original: O 1 , 1 -reg (with ⌘ = M  1 ) and O 2 , 1 -reg , respectively.
Revised: O 1 , 1 -reg (with ⌘ = M  1 ) and O 2 , 1 -reg , respectively.

Original: Clearly, by deﬁnition, the oracle always satisﬁes a ( p + ⌫  1 , ( M/p !)
Revised: Clearly, by deﬁnition, the oracle always satisﬁes a ( p + ⌫  1 , ( M/p !)

Original: 1 / ( p + ⌫  1) ) - movement bound.
Revised: 1 / ( p + ⌫  1) ) - movement bound.

Original: Moreover, it is easy to show that
Revised: Moreover, it is easy to show that

Original: [Equation]
Revised: [Equation]

Original:  For any p  1 and ⌫ 2 [0 , 1] we say that r p f is ( H, ⌫ ) -Hölder if for all x, y we have kr p f ( x )  r p f ( y ) k
Revised: p [Equation] For any p  1 and ⌫ 2 [0 , 1] we say that r p f is ( H, ⌫ ) -Hölder if for all x, y we have kr p f ( x )  r p f ( y ) k

Original: op  H k x  y k ⌫ .
Revised: op  H k x  y k ⌫ .

Original: (An ( H, 1) -Hölder derivative is H -Lipschitz.)
Revised: (An ( H, 1) -Hölder derivative is H -Lipschitz.)

Original: If r p f is ( H, ⌫ ) -Hölder, Taylor’s theorem gives kr f ( x ) r
Revised: If r p f is ( H, ⌫ ) -Hölder, Taylor’s theorem gives kr f ( x ) r

Original: ˜ f p ( x ; y ) k  Hp !
Revised: ˜ f p ( x ; y ) k  Hp !

Original: k x  y k p + ⌫  1 [41, Lemma 2.5], and so k x  ( y  1  r f ( x ))
Revised: k x  y k p + ⌫  1 [41, Lemma 2.5], and so k x  ( y  1  r f ( x ))

Original: k  HM k x  y k .
Revised: k  HM k x  y k .

Original: Therefore, when M  H/  the oracle is a  -MS oracle.
Revised: Therefore, when M  H/  the oracle is a  -MS oracle.

Original: Exploiting third-order smoothness with a second order oracle [36, 26].
Revised: Exploiting third-order smoothness with a second order oracle [36, 26].

Original: For p > 2 , computing O p, ⌫ -reg is typically intractable due to the need to compute the high-order derivative tensors r 3 f ( y ) , r 4 f ( y ) , . . .
Revised: For p > 2 , computing O p, ⌫ -reg is typically intractable due to the need to compute the high-order derivative tensors r 3 f ( y ) , r 4 f ( y ) , . . .

Original: , r p f ( y ) .
Revised: , r p f ( y ) .

Original: Nevertheless for p = 3 Nesterov [36] designs an approximate solver for (7), which we denote O 3 -reg-so , using only r 2 f ( y ) and a logarithmic number of gradient evaluations.
Revised: Nevertheless for p = 3 Nesterov [36] designs an approximate solver for (7), which we denote O 3 -reg-so , using only r 2 f ( y ) and a logarithmic number of gradient evaluations.

Original: When r 3 f is ( L 3 , 1) -Hölder, [36] shows that
Revised: When r 3 f is ( L 3 , 1) -Hölder, [36] shows that

Original: O 3 -reg-so is a valid MS-oracle satisfying a (3 , O
Revised: O 3 -reg-so is a valid MS-oracle satisfying a (3 , O

Original: ( L 3 ))
Revised: ( L 3 ))

Original: -movement bound, on par with the movement bound of O 3 , 1 -reg .
Revised: -movement bound, on par with the movement bound of O 3 , 1 -reg .

Original: Exact ball optimization oracle [12].
Revised: Exact ball optimization oracle [12].

Original: For a given query y , consider the exact minimizer of f constrained to a ball of radius r around y , i.e., consider an oracle
Revised: For a given query y , consider the exact minimizer of f constrained to a ball of radius r around y , i.e., consider an oracle

Original: O r -ball such that ( x,  ) =
Revised: O r -ball such that ( x,  ) =

Original: O r -ball ( y ) satisfy x 2 argmin x 0 : k x 0  y k r
Revised: O r -ball ( y ) satisfy x 2 argmin x 0 : k x 0  y k r

Original: f
Revised: f

Original: ( x 0 ) and  =
Revised: ( x 0 ) and  =

Original: kr f ( x ) k k x  y k .
Revised: kr f ( x ) k k x  y k .

Original: One may easily verify that (unless  = kr f ( x ) k = 0 ) we have x = y  1  r f ( x ) , and therefore the oracle is a 0 -MS oracle.
Revised: One may easily verify that (unless  = kr f ( x ) k = 0 ) we have x = y  1  r f ( x ) , and therefore the oracle is a 0 -MS oracle.

Original: Moreover, when f is convex, we have either k x  y k
Revised: Moreover, when f is convex, we have either k x  y k

Original: = r or x is a global minimizer of f , and so we may assume without loss of generality that the oracle satisﬁes an ( 1 , 1 /r ) movement bound.
Revised: = r or x is a global minimizer of f , and so we may assume without loss of generality that the oracle satisﬁes an ( 1 , 1 /r ) movement bound.

Original: Ball-Constrained Newton (BaCoN) oracle [12].
Revised: Ball-Constrained Newton (BaCoN) oracle [12].

Original: Exactly implementing O r -ball is generally in- tractable.
Revised: Exactly implementing O r -ball is generally in- tractable.

Original: Nevertheless, Carmon et al.
Revised: Nevertheless, Carmon et al.

Original: [12, Alg. 3] describe a method
Revised: [12, Alg. 3] describe a method

Original: O r -BaCoN based on solving a sequence of e
Revised: O r -BaCoN based on solving a sequence of e

Original: O (1) trust-region problems (ball-constrained Newton steps), which we call that, for functions that are O (1) -Hessian
Revised: O (1) trust-region problems (ball-constrained Newton steps), which we call that, for functions that are O (1) -Hessian

Original: stable in a ball of radius r (or 1 /r -quasi-self-concordant) and have a ﬁnite condition number, outputs ( x,  ) satisfying the 12 -MS oracle condition and an ( 1 , O (1 /r )) - movement bound.
Revised: stable in a ball of radius r (or 1 /r -quasi-self-concordant) and have a ﬁnite condition number, outputs ( x,  ) satisfying the 12 -MS oracle condition and an ( 1 , O (1 /r )) - movement bound.

Original: Implementing O r -BaCoN requires only a single Hessian evaluation and a number of linear system solutions that is polylogarithmic in problem parameters.
Revised: Implementing O r -BaCoN requires only a single Hessian evaluation and a number of linear system solutions that is polylogarithmic in problem parameters.

Original: Subsequent works implementing ball oracles [13, 4, 11] satisfy an approximation guarantee different than the MS condition, similar to the one we describe in Appendix C.
Revised: Subsequent works implementing ball oracles [13, 4, 11] satisfy an approximation guarantee different than the MS condition, similar to the one we describe in Appendix C.

Original: 3.2 An adaptive Monteiro-Svaiter-Newton oracle
Revised: 3.2 An adaptive Monteiro-Svaiter-Newton oracle

Original: The oracle implementations in Section 3.1 satisfy movement bounds by design and the MS condition (2) by assumption.
Revised: The oracle implementations in Section 3.1 satisfy movement bounds by design and the MS condition (2) by assumption.

Original: For example, the cubic-regularized Newton step oracle O cr is guaranteed to satisfy the MS condition only when the regularization parameter M is sufﬁciently larger than the Lipschitz constant of r 2 f .
Revised: For example, the cubic-regularized Newton step oracle O cr is guaranteed to satisfy the MS condition only when the regularization parameter M is sufﬁciently larger than the Lipschitz constant of r 2 f .

Original: This suggests that M must be carefully tuned to ensure good performance.
Revised: This suggests that M must be carefully tuned to ensure good performance.

Original: Prior work attempt to dynamically adjust M in order to meet certain approximation conditions [14, 20, 21, 24].
Revised: Prior work attempt to dynamically adjust M in order to meet certain approximation conditions [14, 20, 21, 24].

Original: However, even computing a single cubic-regularized Newton step entails searching for  that satisﬁes k
Revised: However, even computing a single cubic-regularized Newton step entails searching for  that satisﬁes k

Original: [ r 2 f ( y )
Revised: [ r 2 f ( y )

Original: +  I ]  1 r f ( y ) k
Revised: +  I ]  1 r f ( y ) k

Original: = M  2 .
Revised: = M  2 .

Original: Therefore, such a search over M is essentially a (potentially) redundant double search over  .
Revised: Therefore, such a search over M is essentially a (potentially) redundant double search over  .

Original: We propose a more direct and more adaptive MS oracle recipe: search for the smallest  for which the regularized Newton step x = y  [ r 2 f ( y )
Revised: We propose a more direct and more adaptive MS oracle recipe: search for the smallest  for which the regularized Newton step x = y  [ r 2 f ( y )

Original: +  I ]  1 r f ( y ) satisﬁes the MS condition (2).
Revised: +  I ]  1 r f ( y ) satisﬁes the MS condition (2).

Original: This yields valid MS oracle by construction, independently of any assumption.
Revised: This yields valid MS oracle by construction, independently of any assumption.

Original: Moreover, it is simple to argue that when r 2 f is ( H, ⌫ ) -Hölder continuous for some ⌫ 2 [0 , 1] , such oracle would guarantee the same movement bound as O 2 , ⌫ -reg with the best choice parameters M and ⌫ (see Appendix D.1)—even though our recipe requires neither of these parameters!
Revised: Moreover, it is simple to argue that when r 2 f is ( H, ⌫ ) -Hölder continuous for some ⌫ 2 [0 , 1] , such oracle would guarantee the same movement bound as O 2 , ⌫ -reg with the best choice parameters M and ⌫ (see Appendix D.1)—even though our recipe requires neither of these parameters!

Original: Exactly fulﬁlling this recipe, i.e., ﬁnding the ideal minimal  ?
Revised: Exactly fulﬁlling this recipe, i.e., ﬁnding the ideal minimal  ?

Original: that satisﬁes the MS condition, is difﬁcult.
Revised: that satisﬁes the MS condition, is difﬁcult.

Original: Fortunately, to adaptively guarantee movement bounds, it sufﬁces to ﬁnd a value  such the corresponding regularized Newton step satisﬁes the MS condition, while the step corresponding to  / 2 does not; Algorithm 2 ﬁnds precisely such a  .
Revised: Fortunately, to adaptively guarantee movement bounds, it sufﬁces to ﬁnd a value  such the corresponding regularized Newton step satisﬁes the MS condition, while the step corresponding to  / 2 does not; Algorithm 2 ﬁnds precisely such a  .

Original: Let us describe the operation of Algorithm 2.
Revised: Let us describe the operation of Algorithm 2.

Original: If the input  0 is invalid (i.e., its corresponding regularized Newton step does not satisfy the MS condition so that C HECK MS (  0 ; y,  ) evaluates to
Revised: If the input  0 is invalid (i.e., its corresponding regularized Newton step does not satisfy the MS condition so that C HECK MS (  0 ; y,  ) evaluates to

Original: False), we set  invld  0 and test a double-exponentially increasing series of  ’s, until reaching a valid  vld (line 11).
Revised: False), we set  invld  0 and test a double-exponentially increasing series of  ’s, until reaching a valid  vld (line 11).

Original: If  0 is valid and the LAZY ﬂag is set, we return it immediately.
Revised: If  0 is valid and the LAZY ﬂag is set, we return it immediately.

Original: Otherwise (if LAZY is not set) we set  vld =
Revised: Otherwise (if LAZY is not set) we set  vld =

Original:  0 and decrease it at a double-exponential rate until ﬁnding an invalid  invld (line 5).
Revised:  0 and decrease it at a double-exponential rate until ﬁnding an invalid  invld (line 5).

Original: In either case (so long as LAZY is not set) we obtain an (invalid,valid) pair (  invld ,  vld ) such that  vld /  invld = 2 2 k ?
Revised: In either case (so long as LAZY is not set) we obtain an (invalid,valid) pair (  invld ,  vld ) such that  vld /  invld = 2 2 k ?

Original: at the cost of 2 + k ? linear system solutions.
Revised: at the cost of 2 + k ? linear system solutions.

Original: We then perform precisely k ? log-scale bisection steps in order to shrink  vld /  invld down to 2 while maintaining the invariant that  vld is valid and  invld is invalid (line 15).
Revised: We then perform precisely k ? log-scale bisection steps in order to shrink  vld /  invld down to 2 while maintaining the invariant that  vld is valid and  invld is invalid (line 15).

Original: The following theorem bounds the complexity of Algorithm 2 in terms of linear-system solution number, and establishes a movement bound for its output assuming that r 2 f is locally Hölder around the query point.
Revised: The following theorem bounds the complexity of Algorithm 2 in terms of linear-system solution number, and establishes a movement bound for its output assuming that r 2 f is locally Hölder around the query point.

Original: We defer the proof of the theorem and its following corollary to Appendix D.2.
Revised: We defer the proof of the theorem and its following corollary to Appendix D.2.

Original: Theorem 2.
Revised: Theorem 2.

Original: Algorithm 2 with parameter  is a  -MS oracle O aMSN .
Revised: Algorithm 2 with parameter  is a  -MS oracle O aMSN .

Original: For any y 2 R d , computing ( x,  ) =
Revised: For any y 2 R d , computing ( x,  ) =

Original: O aMSN ( y ) requires at most 2 + 2 log
Revised: O aMSN ( y ) requires at most 2 + 2 log

Original: [Equation]
Revised: [Equation]

Original:  linear systems solutions.
Revised:  linear systems solutions.

Original: If LAZY is
Revised: If LAZY is

Original: False or  >  0 , and if r 2 f is ( H, ⌫ ) -Hölder in a ball of radius 2 k x  y k around y , then ( x, y,  ) satisfy a
Revised: False or  >  0 , and if r 2 f is ( H, ⌫ ) -Hölder in a ball of radius 2 k x  y k around y , then ( x, y,  ) satisfy a

Original: [Equation]
Revised: [Equation]

Original: Input: Query y 2 R d ,  0 > 0 .
Revised: Input: Query y 2 R d ,  0 > 0 .

Original: Flag LAZY .
Revised: Flag LAZY .

Original: Parameters: MS factor  2 (0 , 1) .
Revised: Parameters: MS factor  2 (0 , 1) .

Original: 1 if C HECK MS (  0 ; y,  ) then if LAZY then return y  [ r 2 f ( y ) +  0 I ]  1 r f ( y ) ,  else  vld  0 , k  while C HECK MS (  vld / 2 2 k ; y,  ) do  vld  vld / 2 

Revised: 1 if C HECK MS (  0 ; y,  ) then if LAZY then return y  [ r 2 f ( y ) +  0 I ]  1 r f ( y ) ,  else  vld  0 , k  while C HECK MS (  vld / 2 2 k ; y,  ) do  vld  vld / 2

Original: k k k + k ?
Revised: k k k + k ?

Original: k ,  invld  vld / 2 2 k ?
Revised: k ,  invld  vld / 2 2 k ?

Original: 9 else  invld  0 , k  while not C HECK MS (  invld 2 2 k ; y,  ) do  invld  invld 2 2 k k k + k ?
Revised: 9 else  invld  0 , k  while not C HECK MS (  invld 2 2 k ; y,  ) do  invld  invld 2 2 k k k + k ?

Original: k ,  vld  invld 2 2 k ?
Revised: k ,  vld  invld 2 2 k ?

Original: 15 while  invld <  vld / 2 do  p  invld  vld if C HECK MS (  ; y,  ) then  vld  else  invld  19 return y  [ r 2 f ( y )
Revised: 15 while  invld <  vld / 2 do  p  invld  vld if C HECK MS (  ; y,  ) then  vld  else  invld  19 return y  [ r 2 f ( y )

Original: +  vld I ]  1 r f ( y ) ,  vld 20 function C HECK MS (  ; y,  ) x = y  [ r 2 f ( y )
Revised: +  vld I ]  1 r f ( y ) ,  vld 20 function C HECK MS (  ; y,  ) x = y  [ r 2 f ( y )

Original: +  I ]  1 r f ( y ) if  x  ( y  1  r f ( x ))
Revised: +  I ]  1 r f ( y ) if  x  ( y  1  r f ( x ))

Original:    k x  y k then return True else return False
Revised:    k x  y k then return True else return False

Original: Input: y 2 R d ,  0 > 0 .
Revised: Input: y 2 R d ,  0 > 0 .

Original: Flag LAZY .
Revised: Flag LAZY .

Original: Parameters: MS factor  2 (0 , 1) .
Revised: Parameters: MS factor  2 (0 , 1) .

Original: 1   0 , F AILED C HECK
Revised: 1   0 , F AILED C HECK

Original: False 2 Repeat
Revised: False 2 Repeat

Original: A r 2 f ( y )
Revised: A r 2 f ( y )

Original: +  I , b r f ( y ) .
Revised: +  I , b r f ( y ) .

Original: Apply MinRes/Conjugate Residuals [18] until obtaining w s.t.
Revised: Apply MinRes/Conjugate Residuals [18] until obtaining w s.t.

Original: k
Revised: k

Original: Aw  b k   2 k w kx y + C ONJ R ES
Revised: Aw  b k   2 k w kx y + C ONJ R ES

Original: ( A, b,  ) if  x  ( y  1  r f ( x ))    k x  y k
Revised: ( A, b,  ) if  x  ( y  1  r f ( x ))    k x  y k

Original: then if LAZY or F AILED C HECK then return x ,  else   / else
Revised: then if LAZY or F AILED C HECK then return x ,  else   / else

Original: F AILED C HECK
Revised: F AILED C HECK

Original: True  2  12 function C ONJ R ES ( A, b,  ) w  pr
Revised: True  2  12 function C ONJ R ES ( A, b,  ) w  pr

Original: Aw 0  b .
Revised: Aw 0  b .

Original: r i
Revised: r i

Original: =
Revised: =

Original: Aw i  b sq
Revised: Aw i  b sq

Original: Ar.
Revised: Ar.

Original: q i
Revised: q i

Original: =
Revised: =

Original: Ap i i  while k r i k >  2 k w i k do w i
Revised: Ap i i  while k r i k >  2 k w i k do w i

Original: + w
Revised: + w

Original: i
Revised: i

Original: 
Revised:

Original: h r i ,s i
Revised: h r i ,s i

Original: i k q i k 2 p
Revised: i k q i k 2 p

Original: i r i
Revised: i r i

Original: + r i 
Revised: + r i

Original: h r i ,s i
Revised: h r i ,s i

Original: i k q i k 2 q
Revised: i k q i k 2 q

Original: i s i +
Revised: i s i +

Original: Ar i +  p
Revised: Ar i +  p

Original: i
Revised: i

Original: + h r
Revised: + h r

Original: i +1 ,
Revised: i +1 ,

Original: s i +

Revised: s i +

Original: i
Revised: i

Original: h r i ,s i
Revised: h r i ,s i

Original: i p i
Revised: i p i

Original: + r i +  
Revised: + r i +

Original: q
Revised: q

Original: i + h r
Revised: i + h r

Original: i +1 ,
Revised: i +1 ,

Original: s i +

Revised: s i +

Original: i
Revised: i

Original: h r i ,s i
Revised: h r i ,s i

Original: i q i
Revised: i q i

Original: + s i
Revised: + s i

Original: +  
Revised: +

Original: i
Revised: i

Original: i + return w i
Revised: i + return w i

Original: To understand the LAZY option of Algorithm 2, note that when  0 is valid we will necessarily output    0 .
Revised: To understand the LAZY option of Algorithm 2, note that when  0 is valid we will necessarily output    0 .

Original: In such case Theorem 1 does not require a movement bound (except for the ﬁrst iteration).
Revised: In such case Theorem 1 does not require a movement bound (except for the ﬁrst iteration).

Original: Therefore, we might as well save on computation and return  0 .
Revised: Therefore, we might as well save on computation and return  0 .

Original: The following Corollary 3 gives the overall complexity bound for the combination of Algorithm 1 and O aMSN , leveraging “lazy” oracle calls to show that the number of linear system solves per iteration is essentially constant.
Revised: The following Corollary 3 gives the overall complexity bound for the combination of Algorithm 1 and O aMSN , leveraging “lazy” oracle calls to show that the number of linear system solves per iteration is essentially constant.

Original: Corollary 3.
Revised: Corollary 3.

Original: Consider Algorithm 1 with initial point x 0 , parameters ↵ satisfying 1 . 1  ↵ =
Revised: Consider Algorithm 1 with initial point x 0 , parameters ↵ satisfying 1 . 1  ↵ =

Original: O (1) and  0 0 , and  -MS oracle O aMSN
Revised: O (1) and  0 0 , and  -MS oracle O aMSN

Original: (with LAZY =
Revised: (with LAZY =

Original: True in all but the ﬁrst iteration) with  2 (0 . 01 , 0 . 99) .
Revised: True in all but the ﬁrst iteration) with  2 (0 . 01 , 0 . 99) .

Original: For any H, ✏ > 0 , ⌫ 2 [0 , 1] and any x ? 2 R d , if f is convex with ( H, ⌫ ) -
Revised: For any H, ✏ > 0 , ⌫ 2 [0 , 1] and any x ? 2 R d , if f is convex with ( H, ⌫ ) -

Original: Hölder Hessian, the algorithm produces an iterate x T such that f ( x T )  f ( x ? ) +
Revised: Hölder Hessian, the algorithm produces an iterate x T such that f ( x T )  f ( x ? ) +

Original: ✏ using T =
Revised: ✏ using T =

Original: O ⇣ 
Revised: O ⇣

Original: H k x 0  x ?
Revised: H k x 0  x ?

Original: k 2+ ⌫ / ✏  2 / (4+3 ⌫ ) ⌘
Revised: k 2+ ⌫ / ✏  2 / (4+3 ⌫ ) ⌘

Original: Hessian evaluations and O ⇣
Revised: Hessian evaluations and O ⇣

Original: T + loglog max n
Revised: T + log

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Revised: max n

Original: [Equation]
Revised: [Equation]

Original: o⌘ linear system solutions, where R is the distance between x 0 and argmin x 0 f ( x 0 ) .
Revised: o⌘ linear system solutions, where R is the distance between x 0 and argmin x 0 f ( x 0 ) .

Original: Note that as long as  0 0 is in the range
Revised: Note that as long as  0 0 is in the range

Original: [Equation]
Revised: [Equation]

Original: ⇤ , the double logarithmic term in our bound on linear system solution number is O ( T ) .
Revised: ⇤ , the double logarithmic term in our bound on linear system solution number is O ( T ) .

Original: Therefore, the overall bound is O ( T ) for an extremely large range of  0 0 values.
Revised: Therefore, the overall bound is O ( T ) for an extremely large range of  0 0 values.

Original: 3.3 First-order implementation via MinRes/Conjugate Residuals
Revised: 3.3 First-order implementation via MinRes/Conjugate Residuals

Original: We now present a ﬁrst-order implementation of our adaptive oracle, O aMSN-fo (Algorithm 3), which replaces exact linear system solutions with approximations obtained via Hessian-vector products and the MinRes/Conjugate Residuals method [42, 18].
Revised: We now present a ﬁrst-order implementation of our adaptive oracle, O aMSN-fo (Algorithm 3), which replaces exact linear system solutions with approximations obtained via Hessian-vector products and the MinRes/Conjugate Residuals method [42, 18].

Original: Similar to Algorithm 2, the algorithm searches for  such that x  ⇡ y  [ r 2 f ( y )
Revised: Similar to Algorithm 2, the algorithm searches for  such that x  ⇡ y  [ r 2 f ( y )

Original: +  I ]  1 r f ( y ) satisﬁes the MS condition, but x  / 2 does not.
Revised: +  I ]  1 r f ( y ) satisﬁes the MS condition, but x  / 2 does not.

Original: Departing from the double-exponential scheme of Algorithm 2, here we adopt the following doubling scheme that allows us to control the cost of the x  approximation.
Revised: Departing from the double-exponential scheme of Algorithm 2, here we adopt the following doubling scheme that allows us to control the cost of the x  approximation.

Original: If  0 is such that x  0 does not satisfy the MS condition, we repeatedly test  = 2  0 , 4  0 , 8  0 , . . .
Revised: If  0 is such that x  0 does not satisfy the MS condition, we repeatedly test  = 2  0 , 4  0 , 8  0 , . . .

Original: and return the ﬁrst one for which x  satisﬁes the MS condition.
Revised: and return the ﬁrst one for which x  satisﬁes the MS condition.

Original: If x  0 satisﬁes the MS condition and the algorithm is LAZY , we immediately return it.
Revised: If x  0 satisﬁes the MS condition and the algorithm is LAZY , we immediately return it.

Original: Otherwise, we repeatedly test  = 12  0 , 14  0 , 18  0 , . . .
Revised: Otherwise, we repeatedly test  = 12  0 , 14  0 , 18  0 , . . .

Original: until reaching  for which x
Revised: until reaching  for which x

Original:  does not satisfy the MS condition, and return x 2  .
Revised:  does not satisfy the MS condition, and return x 2  .

Original: The subroutine C ONJ R ES of Algorithm 3 takes as input a matrix A , a vector b , and accuracy parameter  , and iteratively generates { w i } that approximate A  1 b .
Revised: The subroutine C ONJ R ES of Algorithm 3 takes as input a matrix A , a vector b , and accuracy parameter  , and iteratively generates { w i } that approximate A  1 b .

Original: The construction of the MinRes/Conjugate
Revised: The construction of the MinRes/Conjugate

Original: Residuals method guarnatees that w i minimizes the norm of the residual r
Revised: Residuals method guarnatees that w i minimizes the norm of the residual r

Original: i
Revised: i

Original: =
Revised: =

Original: Aw i
Revised: Aw i

Original:  b in the Krylov subspace span { b, Ab, . . .
Revised:  b in the Krylov subspace span { b, Ab, . . .

Original: , A i  1 b } .
Revised: , A i  1 b } .

Original: The key algorithmic decision here is when to stop the iterations: stop too early, and the approximation for the Newton step might not be accurate enough to guarantee a movement bound; stop too late, and incur a high Hessian-vector product complexity.
Revised: The key algorithmic decision here is when to stop the iterations: stop too early, and the approximation for the Newton step might not be accurate enough to guarantee a movement bound; stop too late, and incur a high Hessian-vector product complexity.

Original: We introduce a simple stopping condition (line 17) that strikes a balance.
Revised: We introduce a simple stopping condition (line 17) that strikes a balance.

Original: On the one hand, we show that whenever the condition k r i k   2 k w i k holds, the resulting point x can certify roughly the same movement bounds as exact Newton steps.
Revised: On the one hand, we show that whenever the condition k r i k   2 k w i k holds, the resulting point x can certify roughly the same movement bounds as exact Newton steps.

Original: On the other hand, by invoking the complexity bounds in [28] and using the the optimality of k r i
Revised: On the other hand, by invoking the complexity bounds in [28] and using the the optimality of k r i

Original: k
Revised: k

Original: , we guarantee that the stopping condition is met within a number of iterations proportional to 1 / p  .
Revised: , we guarantee that the stopping condition is met within a number of iterations proportional to 1 / p  .

Original: The structure of our doubling scheme for  then allows us to relate the overall ﬁrst-order complexity to the lowest value of  queried, obtaining the following guarantees.
Revised: The structure of our doubling scheme for  then allows us to relate the overall ﬁrst-order complexity to the lowest value of  queried, obtaining the following guarantees.

Original: See proofs in Appendix D.3.
Revised: See proofs in Appendix D.3.

Original: Theorem 4.
Revised: Theorem 4.

Original: Algorithm 3 with parameter  is a  -MS oracle O aMSN-fo .
Revised: Algorithm 3 with parameter  is a  -MS oracle O aMSN-fo .

Original: For any y 2 R d , computing ( x,  ) =
Revised: For any y 2 R d , computing ( x,  ) =

Original: O aMSN-fo ( y ) requires at most O ⇣q
Revised: O aMSN-fo ( y ) requires at most O ⇣q

Original: [Equation]
Revised: [Equation]

Original: Hessian-vector product and
Revised: Hessian-vector product and

Original: O (  log  ) gradient computations.
Revised: O (  log  ) gradient computations.

Original: If LAZY is False or  >  0 , and if r 2 f is ( H, ⌫ ) -Hölder, then ( x, y,  ) satisfy a  1 + ⌫ , (6 H/  ) 1 / (1+ ⌫ )  -movement bound.
Revised: If LAZY is False or  >  0 , and if r 2 f is ( H, ⌫ ) -Hölder, then ( x, y,  ) satisfy a  1 + ⌫ , (6 H/  ) 1 / (1+ ⌫ )  -movement bound.

Original: Corollary 5.
Revised: Corollary 5.

Original: Consider Algorithm 1 with initial point x 0 , parameters ↵ satisfying 1 . 1  ↵ = O (1) and  0 0 , and  -MS oracle O aMSN-fo with LAZY set to True in all but the ﬁrst iteration and  2 (0 . 01 , 0 . 99) .
Revised: Consider Algorithm 1 with initial point x 0 , parameters ↵ satisfying 1 . 1  ↵ = O (1) and  0 0 , and  -MS oracle O aMSN-fo with LAZY set to True in all but the ﬁrst iteration and  2 (0 . 01 , 0 . 99) .

Original: For any L, H, ✏ > 0 , ⌫ 2 [0 , 1] and any x ? 2 R d , if f is convex with ( H, ⌫ ) -Hölder Hessian and L -Lipschitz gradient, the algorithm produces an iterate x T such that f ( x T )  f ( x ? ) + ✏ within T =
Revised: For any L, H, ✏ > 0 , ⌫ 2 [0 , 1] and any x ? 2 R d , if f is convex with ( H, ⌫ ) -Hölder Hessian and L -Lipschitz gradient, the algorithm produces an iterate x T such that f ( x T )  f ( x ? ) + ✏ within T =

Original: O ✓⇣
Revised: O ✓⇣

Original: [Equation]
Revised: [Equation]

Original: ◆ iterations and at most
Revised: ◆ iterations and at most

Original: O ✓⇣
Revised: O ✓⇣

Original: [Equation]
Revised: [Equation]

Original: q
Revised: q

Original: [Equation]
Revised: [Equation]

Original: ◆ gradient and Hessian-vector product evaluations.
Revised: ◆ gradient and Hessian-vector product evaluations.

Original: Note that the L -Lipschitz gradient assumption implies an ( L, 0)
Revised: Note that the L -Lipschitz gradient assumption implies an ( L, 0)

Original: -Hölder
Revised: -Hölder

Original: Hessian assumption, giving the iteration complexity bound we state in Table 1.
Revised: Hessian assumption, giving the iteration complexity bound we state in Table 1.

Original: Moreover, note that our algorithm has the optimal O ( p
Revised: Moreover, note that our algorithm has the optimal O ( p

Original: L k x 0  x ?
Revised: L k x 0  x ?

Original: k 2 / ✏ ) complexity for any  0 0 in the range ⌦ ( ✏ / k x 0  x ? k 2 )
Revised: k 2 / ✏ ) complexity for any  0 0 in the range ⌦ ( ✏ / k x 0  x ? k 2 )

Original: to
Revised: to

Original: L exp n
Revised: L exp n

Original: O ( p
Revised: O ( p

Original: L k
Revised: L k

Original: [Equation]
Revised: [Equation]

Original: 0 
Revised: 0

Original: [Equation]
Revised: [Equation]

Original: k 2 / ✏ )
Revised: k 2 / ✏ )

Original: o .
Revised: o .

Original: By choosing a large  0 0 (say 10 6 ) we may guarantee that only the logarithmic term is added to the optimal ﬁrst-order evaluation complexity.
Revised: By choosing a large  0 0 (say 10 6 ) we may guarantee that only the logarithmic term is added to the optimal ﬁrst-order evaluation complexity.

Original:  Experiments
Revised:  Experiments

Original: We conduct three sets of experiments.
Revised: We conduct three sets of experiments.

Original: First, we consider O cr with a ﬁxed parameter M and compare previous acceleration schemes to Algorithm 1.
Revised: First, we consider O cr with a ﬁxed parameter M and compare previous acceleration schemes to Algorithm 1.

Original: Second, we combine Algorithm 1 with our adaptive
Revised: Second, we combine Algorithm 1 with our adaptive

Original: O aMSN and test it against previous adaptive accelerated (second-order) methods and Newton’s method.
Revised: O aMSN and test it against previous adaptive accelerated (second-order) methods and Newton’s method.

Original: Finally, we compare Algorithm 1 with our ﬁrst-order adaptive oracle O aMSN-fo to other ﬁrst-order methods.
Revised: Finally, we compare Algorithm 1 with our ﬁrst-order adaptive oracle O aMSN-fo to other ﬁrst-order methods.

Original: We provide full implementation details in Appendix E.1.
Revised: We provide full implementation details in Appendix E.1.

Original: Figure 1 summarizes our results for logistic regression on the ‘a9a’ dataset [15]; see Appendix E.2 for similar results on three additional datasets.
Revised: Figure 1 summarizes our results for logistic regression on the ‘a9a’ dataset [15]; see Appendix E.2 for similar results on three additional datasets.

Original: These experiments were conducted with no tuning of Algorithm 1: the parameters  and ↵ were simply set to 1 2 and 2 , respectively.
Revised: These experiments were conducted with no tuning of Algorithm 1: the parameters  and ↵ were simply set to 1 2 and 2 , respectively.

Original: An additional experiment, reported in Appendix E.3, shows that the algorithm is indeed insensitive to that choice.
Revised: An additional experiment, reported in Appendix E.3, shows that the algorithm is indeed insensitive to that choice.

Original: Non-adaptive methods.
Revised: Non-adaptive methods.

Original: We use the non-adaptive oracle O cr (1), and take M to be 0 .
Revised: We use the non-adaptive oracle O cr (1), and take M to be 0 .

Original: 2 ¯ H where, for feature vectors  1 , . . .
Revised: 2 ¯ H where, for feature vectors  1 , . . .

Original: ,  n , ¯ H =
Revised: ,  n , ¯ H =

Original: k 1 n
Revised: k 1 n

Original: P ni =

Revised: P ni =

Original:  i  Ti k op max
Revised:  i  Ti k op max

Original: i 2 [ n ] k  i k is an upper bound on 6 p 3 ⇡ 10 times the Lipschitz constant of the logistic regression Hessian [see, e.g., 41].
Revised: i 2 [ n ] k  i k is an upper bound on 6 p 3 ⇡ 10 times the Lipschitz constant of the logistic regression Hessian [see, e.g., 41].

Original: Fixing the MS oracle allows for a controlled comparison of different acceleration schemes: Figure 1(a) shows that standard
Revised: Fixing the MS oracle allows for a controlled comparison of different acceleration schemes: Figure 1(a) shows that standard

Original: MS acceleration with a carefully-implemented bisection outperforms standard cubic regularization (CR) and its accelerated counterpart (ACR) [33, Alg. 4.8], and removing the bisection via Algorithmyields the best results.
Revised: MS acceleration with a carefully-implemented bisection outperforms standard cubic regularization (CR) and its accelerated counterpart (ACR) [33, Alg. 4.8], and removing the bisection via Algorithmyields the best results.

Original: We also implemented the heuristic suggested by Song et al.
Revised: We also implemented the heuristic suggested by Song et al.

Original: [41], where instead of a bisection in Algorithm 

Revised: [41], where instead of a bisection in Algorithm

Original: we select a sequence  0 t such that A t =
Revised: we select a sequence  0 t such that A t =

Original: [Equation]
Revised: [Equation]

Original: In Appendix E.4 we tune the M parameter for each method separately, ﬁnding that the optimal M for CR is near 0 , so that O cr is nearly a Newton step (and not a valid MS oracle).
Revised: In Appendix E.4 we tune the M parameter for each method separately, ﬁnding that the optimal M for CR is near 0 , so that O cr is nearly a Newton step (and not a valid MS oracle).

Original: Adaptive methods and Newton’s method.
Revised: Adaptive methods and Newton’s method.

Original: We compare the following adaptive accelerated secondorder methods (which do not require an estimate of the Hessian Lipschitz constant): Adaptive
Revised: We compare the following adaptive accelerated secondorder methods (which do not require an estimate of the Hessian Lipschitz constant): Adaptive

Original: ACR [21, Algorithm 4] (which adaptively sets M in O cr ), standard MS acceleration (Algorithm 0) with O aMSN (Algorithm 2, with LAZY = False) and Algorithm 1 with O aMSN (with LAZY =
Revised: ACR [21, Algorithm 4] (which adaptively sets M in O cr ), standard MS acceleration (Algorithm 0) with O aMSN (Algorithm 2, with LAZY = False) and Algorithm 1 with O aMSN (with LAZY =

Original: True in all but the ﬁrst iteration).
Revised: True in all but the ﬁrst iteration).

Original: Figure 1(b) shows that the latter converges signiﬁcantly faster than the other adaptive acceleration schemes.
Revised: Figure 1(b) shows that the latter converges signiﬁcantly faster than the other adaptive acceleration schemes.

Original: However, the classical “unaccelerated” Newton iteration x t +1 =
Revised: However, the classical “unaccelerated” Newton iteration x t +1 =

Original:  ( r 2 f ( x t ))  1 r f ( x t ) strongly outperforms all “accelerated” methods, indicating that momentum mechanisms might actually be slowing down convergence in logistic regression problems.
Revised:  ( r 2 f ( x t ))  1 r f ( x t ) strongly outperforms all “accelerated” methods, indicating that momentum mechanisms might actually be slowing down convergence in logistic regression problems.

Original: To test this, we consider the following simple iteration of (the non-lazy variant) of our oracle: x t +1 ,  t +1 =
Revised: To test this, we consider the following simple iteration of (the non-lazy variant) of our oracle: x t +1 ,  t +1 =

Original: O aMSN ( x t ;  t / 2) ; it signiﬁcantly improves over Algorithm 1.
Revised: O aMSN ( x t ;  t / 2) ; it signiﬁcantly improves over Algorithm 1.

Original: These results beg the question: is momentum ever useful for second-order methods?
Revised: These results beg the question: is momentum ever useful for second-order methods?

Original: In Appendix E. we test different schemes on the lower bound construction [3, 21].
Revised: In Appendix E. we test different schemes on the lower bound construction [3, 21].

Original: We ﬁnd momentum is helpful for
Revised: We ﬁnd momentum is helpful for

Original: O cr , but not for the adaptive oracle O aMSN .
Revised: O cr , but not for the adaptive oracle O aMSN .

Original: What makes Newton’s method perform so well on logistic regression, and whether simply iterating O aMSN is worst-case optimal, are important questions for future work.
Revised: What makes Newton’s method perform so well on logistic regression, and whether simply iterating O aMSN is worst-case optimal, are important questions for future work.

Original: First-order methods.
Revised: First-order methods.

Original: We compare our ﬁrst-order adaptive O aMSN-fo (Algorithm 3) to the following baselines: gradient descent and accelerated gradient descent [38] with a tuned step size ⌘ , and LBFGS-B from SciPy [10, 44, 43].
Revised: We compare our ﬁrst-order adaptive O aMSN-fo (Algorithm 3) to the following baselines: gradient descent and accelerated gradient descent [38] with a tuned step size ⌘ , and LBFGS-B from SciPy [10, 44, 43].

Original: In light of the above comparison with Newton’s method, we also test the following simple iteration of (the lazy variant) of our oracle: x t +1 ,  t +1 =
Revised: In light of the above comparison with Newton’s method, we also test the following simple iteration of (the lazy variant) of our oracle: x t +1 ,  t +1 =

Original: O aMSN-fo ( x t ;  t / 2) .
Revised: O aMSN-fo ( x t ;  t / 2) .

Original: Figure 1(c) shows that forgoing (second-order) momentum is better for the ﬁrst-order oracle, too:
Revised: Figure 1(c) shows that forgoing (second-order) momentum is better for the ﬁrst-order oracle, too:

Original: Algorithm 1 performs comparably to tuned AGD (without tuning a single parameter), and the equally adaptive O aMSN-fo iteration performs comparably to with L-BFGS-B.
Revised: Algorithm 1 performs comparably to tuned AGD (without tuning a single parameter), and the equally adaptive O aMSN-fo iteration performs comparably to with L-BFGS-B.

Original: Acknowledgments
Revised: Acknowledgments

Original: We thank the anonymous reviewers for helpful questions and suggestions, leading to important writing clariﬁcations, a simpliﬁcation of the proof of Lemma 3, and an improved complexity bound for second-order methods under third-order smoothness (due to the observation that our framework is compatible with the oracle of [36]).
Revised: We thank the anonymous reviewers for helpful questions and suggestions, leading to important writing clariﬁcations, a simpliﬁcation of the proof of Lemma 3, and an improved complexity bound for second-order methods under third-order smoothness (due to the observation that our framework is compatible with the oracle of [36]).

Original: YC and DH were supported in part by the Israeli Science
Revised: YC and DH were supported in part by the Israeli Science

Original: Foundation (ISF) grant no. 2486/21, the Len Blavatnik and the Blavatnik Family foundation and
Revised: Foundation (ISF) grant no. 2486/21, the Len Blavatnik and the Blavatnik Family foundation and

Original: the
Revised: the

Original: Adelis Foundation.
Revised: Adelis Foundation.

Original: YJ was supported in part by a Stanford Graduate Fellowship and the DantzigLieberman Fellowship.
Revised: YJ was supported in part by a Stanford Graduate Fellowship and the DantzigLieberman Fellowship.

Original: AS was supported in part by a Microsoft Research Faculty Fellowship, NSF
Revised: AS was supported in part by a Microsoft Research Faculty Fellowship, NSF

Original: CAREER Award CCF-1844855, NSF Grant CCF-1955039, a PayPal research award, and a Sloan
Revised: CAREER Award CCF-1844855, NSF Grant CCF-1955039, a PayPal research award, and a Sloan

Original: Research Fellowship.
Revised: Research Fellowship.