Original: Optimal and Adaptive Monteiro-Svaiter Acceleration
Revised: Optimal and Adaptive Monteiro-Svaiter Acceleration

Original: Abstract
Revised: Abstract

Original: We develop a variant of the Monteiro-Svaiter (MS) acceleration framework that removes the need to solve an expensive implicit equation at every iteration.
Revised: We develop a variant of the Monteiro-Svaiter (MS) acceleration framework that removes the need to solve an expensive implicit equation at every iteration.

Original: Consequently, for any p ≥ 2 we improve the complexity of convex optimization with
Revised: Consequently, for any p ≥ 2 we improve the complexity of convex optimization with

Original: Lipschitz p th derivative by a logarithmic factor, matching a lower bound.
Revised: Lipschitz p th derivative by a logarithmic factor, matching a lower bound.

Original: We also introduce an MS subproblem solver that requires no knowledge of problem parameters, and implement it as either a second- or ﬁrst-order method via exact linear system solution or MinRes, respectively.
Revised: We also introduce an MS subproblem solver that requires no knowledge of problem parameters, and implement it as either a second- or ﬁrst-order method via exact linear system solution or MinRes, respectively.

Original: On logistic regression our method outperforms previous second-order acceleration schemes, but under-performs Newton’s method; simply iterating our ﬁrst-order adaptive subproblem solver performs comparably to L-BFGS.
Revised: On logistic regression our method outperforms previous second-order acceleration schemes, but under-performs Newton’s method; simply iterating our ﬁrst-order adaptive subproblem solver performs comparably to L-BFGS.

Original:  Introduction
Revised:  Introduction

Original: We consider the problem of minimizing a convex function f : X → R over closed convex set
Revised: We consider the problem of minimizing a convex function f : X → R over closed convex set

Original: X ⊆ R d , given access to an oracle O : X → X that minimizes a local model of f around a given query point.
Revised: X ⊆ R d , given access to an oracle O : X → X that minimizes a local model of f around a given query point.

Original: A key motivating example of such an oracle is the cubic-regularized Newton step
Revised: A key motivating example of such an oracle is the cubic-regularized Newton step

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original: i.e., minimizing the second-order Taylor approximation of f around y plus a cubic regularization term.
Revised: i.e., minimizing the second-order Taylor approximation of f around y plus a cubic regularization term.

Original: However, our results apply to additional oracles including a simple gradient step, regularized higher-order Taylor expansions [5, 20, 7, 24, 8,  36, 22, 42, 37, 27], ball-constrained optimization [12], and new adaptive oracles that we develop.
Revised: However, our results apply to additional oracles including a simple gradient step, regularized higher-order Taylor expansions [5, 19, 7, 23, 8, 35, 21, 41, 36, 26], ball-constrained optimization [12], and new adaptive oracles that we develop.

Original: Seminal work by Monteiro and Svaiter [33] (MS) shows how to accelerate the basic oracle iteration x t +

Revised: Seminal work by Monteiro and Svaiter [32] (MS) shows how to accelerate the basic oracle iteration x t +

Original: = O ( x t ) .
Revised: = O ( x t ) .

Original: Their algorithm is based on the fact that many oracles, including O cr , implicitly approximate proximal points.
Revised: Their algorithm is based on the fact that many oracles, including O cr , implicitly approximate proximal points.

Original: That is, for every y and x = O ( y ) , there exists λ x,y > 0 such that x ≈ argmin x (cid:48) ∈X (cid:8)
Revised: That is, for every y and x = O ( y ) , there exists λ x,y > 0 such that x ≈ argmin x (cid:48) ∈X (cid:8)

Original: f ( x (cid:48) )
Revised: f ( x (cid:48) )

Original: + 12 λ x,y (cid:107) x (cid:48)
Revised: + 12 λ x,y (cid:107) x (cid:48)

Original: − y (cid:107) 2 (cid:9) , with the approximation error controlled by a speciﬁc condition they deﬁne.
Revised: − y (cid:107) 2 (cid:9) , with the approximation error controlled by a speciﬁc condition they deﬁne.

Original: MS prove that, under this condition, the accelerated proximal point method [23, 41] (with dynamic regularization parameter) maintains its rate of convergence.
Revised: MS prove that, under this condition, the accelerated proximal point method [22, 40] (with dynamic regularization parameter) maintains its rate of convergence.

Original: Applying their framework to O cr and assuming ∇ 2 f is Lipschitz, they achieve error bounds that decay as O ( t − 7 / 2 log t ) after t oracle calls, improving the O ( t − 2 ) rate of the basic O cr iteration [38] and the O ( t − 3 ) rate of an earlier accelerated method [34].
Revised: Applying their framework to O cr and assuming ∇ 2 f is Lipschitz, they achieve error bounds that decay as O ( t − 7 / 2 log t ) after t oracle calls, improving the O ( t − 2 ) rate of the basic O cr iteration [37] and the O ( t − 3 ) rate of an earlier accelerated method [33].

Original: Subsequent works apply variations of the MS framework to different oracles, obtaining improved theoretical guarantees for functions with continuous higher-order derivatives [20, 7, 24, 42, 2], parallel optimization [6], logistic and (cid:96) ∞ regression [8, 12], minimizing functions with Hölder continuous higher derivatives [42], and distributionally-robust optimization [13, 11].
Revised: Subsequent works apply variations of the MS framework to different oracles, obtaining improved theoretical guarantees for functions with continuous higher-order derivatives [19, 7, 23, 41, 2], parallel optimization [6], logistic and (cid:96) ∞ regression [8, 12], minimizing functions with Hölder continuous higher derivatives [41], and distributionally-robust optimization [13, 11].

Original: However, all of these algorithms based on the MS framework share a common drawback: the iterate y t used to produce x t +

Revised: However, all of these algorithms based on the MS framework share a common drawback: the iterate y t used to produce x t +

Original: =
Revised: =

Original: O
Revised: O

Original: ( y t ) depends on the proximal parameter λ t
Revised: ( y t ) depends on the proximal parameter λ t

Original: +1 =
Revised: +1 =

Original: λ x t +1 ,y t , which itself depends on both x t +1 and y t .
Revised: λ x t +1 ,y t , which itself depends on both x t +1 and y t .

Original: This circular dependence necessitates solving an implicit equation for λ t +1 ; MS (and many subsequent results based upon it) propose bisection procedures for doing so using a number of oracle calls logarithmic in the problem parameters.
Revised: This circular dependence necessitates solving an implicit equation for λ t +1 ; MS (and many subsequent results based upon it) propose bisection procedures for doing so using a number of oracle calls logarithmic in the problem parameters.

Original: From a theoretical perspective, the additional bisection complexity introduces a logarithmic gap between the upper bounds due to
Revised: From a theoretical perspective, the additional bisection complexity introduces a logarithmic gap between the upper bounds due to

Original: MS-based algorithms and the best known lower bounds [3, 22] in a number of settings.
Revised: MS-based algorithms and the best known lower bounds [3, 21] in a number of settings.

Original: From a practical perspective, the use of bisection in the MS framework is undesirable as it potentially discards the optimization progress made by oracle calls during each bisection.
Revised: From a practical perspective, the use of bisection in the MS framework is undesirable as it potentially discards the optimization progress made by oracle calls during each bisection.

Original: In his textbook,
Revised: In his textbook,

Original: Nesterov [35, §4.3.3] argues that the logarithmic cost of bisection likely renders the MS scheme for accelerating O cr inferior in practice to algorithms whose error decays at the asymptotically worse rate
Revised: Nesterov [34, §4.3.3] argues that the logarithmic cost of bisection likely renders the MS scheme for accelerating O cr inferior in practice to algorithms whose error decays at the asymptotically worse rate

Original: O ( t − 3 ) but do not require bisection; he notes that removing the bisection from the MS algorithm is an “open and challenging question in Optimization Theory.”
Revised: O ( t − 3 ) but do not require bisection; he notes that removing the bisection from the MS algorithm is an “open and challenging question in Optimization Theory.”

Original: Carmon et al.
Revised: Carmon et al.

Original: [13] also point out bisection as one of the main hurdles in making their theoretical scheme practical, while Song et al.
Revised: [13] also point out bisection as one of the main hurdles in making their theoretical scheme practical, while Song et al.

Original: [42] note this limitation and propose a heuristic alternative to bisection.
Revised: [41] note this limitation and propose a heuristic alternative to bisection.

Original: (See Appendix A for extended discussion of related work, including a concurrent and independent result by Kovalev and Gasnikov [28].)
Revised: (See Appendix A for extended discussion of related work, including a concurrent and independent result by Kovalev and Gasnikov [27].)

Original: 1.1 Our contributions
Revised: 1.1 Our contributions

Original: We settle this open question, providing a variant of MS acceleration that does not require bisection (Section 2).
Revised: We settle this open question, providing a variant of MS acceleration that does not require bisection (Section 2).

Original: When combined with certain existing MS oracles (Section 3.1), our algorithm obtains complexity bounds that are optimal up to constant factors, improving over prior art by a logarithmic factor (see Table 1).
Revised: When combined with certain existing MS oracles (Section 3.1), our algorithm obtains complexity bounds that are optimal up to constant factors, improving over prior art by a logarithmic factor (see Table 1).

Original: In addition, our algorithm has no parameters sensitive to tuning.
Revised: In addition, our algorithm has no parameters sensitive to tuning.

Original: We then go a step further and (in Section 3.2) develop an adaptive alternative to O cr (Equation (1)).
Revised: We then go a step further and (in Section 3.2) develop an adaptive alternative to O cr (Equation (1)).

Original: Our oracle does not require tuning the parameter M , which in theory should be proportional to the (difﬁcult to estimate)
Revised: Our oracle does not require tuning the parameter M , which in theory should be proportional to the (difﬁcult to estimate)

Original: Lipschitz constant of ∇ 2 f .
Revised: Lipschitz constant of ∇ 2 f .

Original: Using our oracle, we obtain the optimal
Revised: Using our oracle, we obtain the optimal

Original: Hessian evaluation complexity O ( t − (4+3 ν ) / 2 ) for functions with order- ν Hölder Hessian (Lipschitz
Revised: Hessian evaluation complexity O ( t − (4+3 ν ) / 2 ) for functions with order- ν Hölder Hessian (Lipschitz

Original: Hessian is the ν = 1 special case), without requiring any knowledge of the Hölder constant and order ν .
Revised: Hessian is the ν = 1 special case), without requiring any knowledge of the Hölder constant and order ν .

Original: Our oracle is also efﬁcient: while existing complexity bounds for computing O cr require a logarithmic number of linear system solutions per call, our oracle requires a double-logarithmic number.
Revised: Our oracle is also efﬁcient: while existing complexity bounds for computing O cr require a logarithmic number of linear system solutions per call, our oracle requires a double-logarithmic number.

Original: Moreover, when used with our acceleration method, the number of linear system solves per iteration is essentially constant.
Revised: Moreover, when used with our acceleration method, the number of linear system solves per iteration is essentially constant.

Original: We also provide a ﬁrst-order implementation of our adaptive oracle (Section 3.3).
Revised: We also provide a ﬁrst-order implementation of our adaptive oracle (Section 3.3).

Original: It approximately solves linear systems via ﬁrst-order operations (Hessian-vector products) using MinRes/Conjugate
Revised: It approximately solves linear systems via ﬁrst-order operations (Hessian-vector products) using MinRes/Conjugate

Original: Residuals [43, 19] with a simple, adaptive, stopping criterion lifted directly from our analysis.
Revised: Residuals [42, 18] with a simple, adaptive, stopping criterion lifted directly from our analysis.

Original: Our oracle attains the optimal ﬁrst-order evaluation complexity for smooth functions up to an additive logarithmic term, without knowledge of the gradient Lipschitz constant or any parameter tuning.
Revised: Our oracle attains the optimal ﬁrst-order evaluation complexity for smooth functions up to an additive logarithmic term, without knowledge of the gradient Lipschitz constant or any parameter tuning.

Original: Moreover, it maintains an optimal outer iteration complexity for Hölder Hessian of any order.
Revised: Moreover, it maintains an optimal outer iteration complexity for Hölder Hessian of any order.

Original: Finally, we report empirical results (Section 4).
Revised: Finally, we report empirical results (Section 4).

Original: On logistic regression problems, combining our optimal acceleration scheme with our adaptive oracle outperforms previously proposed accelerated second-order methods.
Revised: On logistic regression problems, combining our optimal acceleration scheme with our adaptive oracle outperforms previously proposed accelerated second-order methods.

Original: However, we also show that (while somewhat helpful for O cr with a conservative choice of H ), adding momentum to well-tuned or adaptive second-order methods is harmful in logistic regression: simply iterating our oracle—or, better yet, applying Newton’s method—dramatically outperforms all “accelerated” algorithms.
Revised: However, we also show that (while somewhat helpful for O cr with a conservative choice of H ), adding momentum to well-tuned or adaptive second-order methods is harmful in logistic regression: simply iterating our oracle—or, better yet, applying Newton’s method—dramatically outperforms all “accelerated” algorithms.

Original: This important fact seems to have gone unobserved in the literature on accelerated second-order methods, despite logistic regression appearing in many related experiments [42, 16, 31, 25].
Revised: This important fact seems to have gone unobserved in the literature on accelerated second-order methods, despite logistic regression appearing in many related experiments [41, 16, 30, 24].

Original: Simply iterating our adaptive oracle outperforms the classical accelerated gradient descent, and performs comparably to L-BFGS.
Revised: Simply iterating our adaptive oracle outperforms the classical accelerated gradient descent, and performs comparably to L-BFGS.

Original: 1.2 Limitations and outlook
Revised: 1.2 Limitations and outlook

Original: While our algorithms resolve an enduring theoretical open problem in convex optimization, and are free of sensitive parameters that typically hinder theoretically-optimal methods, practical performance remains a limitation.
Revised: While our algorithms resolve an enduring theoretical open problem in convex optimization, and are free of sensitive parameters that typically hinder theoretically-optimal methods, practical performance remains a limitation.

Original: On logistic regression, Newton’s method is remarkably fast, and our acceleration scheme does not seem to help our adaptive oracle.
Revised: On logistic regression, Newton’s method is remarkably fast, and our acceleration scheme does not seem to help our adaptive oracle.

Original: We do not fully understand why this is so, but we suspect that it has to do with additional structure in logistic regression, which Newton’s method can automatically exploit but momentum cannot.
Revised: We do not fully understand why this is so, but we suspect that it has to do with additional structure in logistic regression, which Newton’s method can automatically exploit but momentum cannot.

Original: We believe that future research should identify the structure that makes Newton’s method so efﬁcient, and modifying momentum schemes to leverage it.
Revised: We believe that future research should identify the structure that makes Newton’s method so efﬁcient, and modifying momentum schemes to leverage it.

Original: Scalability is another important limitation.
Revised: Scalability is another important limitation.

Original: While our ﬁrst-order oracle signiﬁcantly improves scalability over the second-order oracle from which it is built, it still relies on exact gradient and
Revised: While our ﬁrst-order oracle signiﬁcantly improves scalability over the second-order oracle from which it is built, it still relies on exact gradient and

Original: Hessian-vector products.
Revised: Hessian-vector products.

Original: Therefore, it will have difﬁculty scaling up to very large datasets.
Revised: Therefore, it will have difﬁculty scaling up to very large datasets.

Original: Nevertheless, we hope that further scalability improvements may be possible by building an oracle that utilizes cheap stochastic gradient estimates instead of exact gradients, bringing with it the exciting prospect of a new and powerful adaptive stochastic gradient method.
Revised: Nevertheless, we hope that further scalability improvements may be possible by building an oracle that utilizes cheap stochastic gradient estimates instead of exact gradients, bringing with it the exciting prospect of a new and powerful adaptive stochastic gradient method.

Original: The alternative, probabilistic approximation condition we propose in Appendix C might be helpful in this regard.
Revised: The alternative, probabilistic approximation condition we propose in Appendix C might be helpful in this regard.

Original:  Removing bisection from the Monteiro-Svaiter framework
Revised:  Removing bisection from the Monteiro-Svaiter framework

Original: Input: Initial x 0 , function f , oracle O
Revised: Input: Initial x 0 , function f , oracle O

Original: Parameters: Initial λ (cid:48) 0 , multiplicative adjustment factor α >1 Set v 0 = x 0 , A 0 =2 ˜ x 1 , λ 1 =
Revised: Parameters: Initial λ (cid:48) 0 , multiplicative adjustment factor α >1 Set v 0 = x 0 , A 0 =2 ˜ x 1 , λ 1 =

Original: O ( x 0 ; λ (cid:48) 0 ) , λ (cid:48) 1 = λ3 for t = 0 , 1 , . .
Revised: O ( x 0 ; λ (cid:48) 0 ) , λ (cid:48) 1 = λ3 for t = 0 , 1 , . .

Original: , do a (cid:48) t +1 = 1 2 λ (cid:48)
Revised: , do a (cid:48) t +1 = 1 2 λ (cid:48)

Original: t +1 (cid:0) 

Revised: t +1 (cid:0)

Original: + (cid:112) 

Revised: + (cid:112)

Original: + 4 λ (cid:48)
Revised: + 4 λ (cid:48)

Original: t +1 A t (cid:1)
Revised: t +1 A t (cid:1)

Original: A (cid:48) t +1 =
Revised: A (cid:48) t +1 =

Original: A t + a (cid:48) t +  y t =
Revised: A t + a (cid:48) t +  y t =

Original: A t A (cid:48) t +1 x t + a (cid:48) t +1 A (cid:48)
Revised: A t A (cid:48) t +1 x t + a (cid:48) t +1 A (cid:48)

Original: t +1 v t if t > 0 then ˜ x t +1 , λ t
Revised: t +1 v t if t > 0 then ˜ x t +1 , λ t

Original: +

Revised: +

Original: =
Revised: =

Original: O ( y t ; λ (cid:48)
Revised: O ( y t ; λ (cid:48)

Original: t +1 ) if λ t +1 ≤ λ (cid:48) t
Revised: t +1 ) if λ t +1 ≤ λ (cid:48) t

Original: +1 then a t +1 = a (cid:48) t +1 , A t +1 =
Revised: +1 then a t +1 = a (cid:48) t +1 , A t +1 =

Original: A (cid:48) t
Revised: A (cid:48) t

Original: +  x t +1 =
Revised: +  x t +1 =

Original: ˜ x t +  λ (cid:48) t +

Revised: ˜ x t +  λ (cid:48) t +

Original: = 

Revised: =

Original: α λ (cid:48) t
Revised: α λ (cid:48) t

Original: +  else γ t
Revised: +  else γ t

Original: +1 = λ (cid:48)
Revised: +1 = λ (cid:48)

Original: t +1 λ t +  a t +1 = γ t +1 a (cid:48)
Revised: t +1 λ t +  a t +1 = γ t +1 a (cid:48)

Original: t +1 , A t +1 =
Revised: t +1 , A t +1 =

Original: A t + a t +  x t
Revised: A t + a t +  x t

Original: +

Revised: +

Original: =
Revised: =

Original: (1 − γ t +1 )
Revised: (1 − γ t +1 )

Original: A t
Revised: A t

Original: A t + x t + γ t
Revised: A t + x t + γ t

Original: +

Revised: +

Original: A
Revised: A

Original: (cid:48) t +1 A t + ˜ x t +  λ (cid:48)
Revised: (cid:48) t +1 A t + ˜ x t +  λ (cid:48)

Original: t +2 =
Revised: t +2 =

Original: αλ (cid:48) t
Revised: αλ (cid:48) t

Original: +  v t +1 =
Revised: +  v t +1 =

Original: v t − a t +1 ∇ f (˜ x t +1 )
Revised: v t − a t +1 ∇ f (˜ x t +1 )

Original: Input: Initial x 0 , function f , oracle O
Revised: Input: Initial x 0 , function f , oracle O

Original: Parameters: Bisection limits λ (cid:96) , λ h , and tolerance ρ >1 Set v 0 = x 0 , A 0 =2 for t = 0 , 1 , . .
Revised: Parameters: Bisection limits λ (cid:96) , λ h , and tolerance ρ >1 Set v 0 = x 0 , A 0 =2 for t = 0 , 1 , . .

Original: , do λ (cid:96) t +1 , λ h t
Revised: , do λ (cid:96) t +1 , λ h t

Original: +1 =
Revised: +1 =

Original: λ (cid:96) , λ h λ (cid:48)
Revised: λ (cid:96) , λ h λ (cid:48)

Original: t +1 = λ
Revised: t +1 = λ

Original: (cid:96)t +

Revised: (cid:96)t +

Original: + λ
Revised: + λ

Original: ht +1 a (cid:48)
Revised: ht +1 a (cid:48)

Original: t +1 = 1 2 λ (cid:48)
Revised: t +1 = 1 2 λ (cid:48)

Original: t +1 (cid:0) 

Revised: t +1 (cid:0)

Original: + (cid:112) 

Revised: + (cid:112)

Original: + 4 λ (cid:48)
Revised: + 4 λ (cid:48)

Original: t +1 A t (cid:1)
Revised: t +1 A t (cid:1)

Original: A (cid:48) t +1 =
Revised: A (cid:48) t +1 =

Original: A t + a (cid:48) t +  y t =
Revised: A t + a (cid:48) t +  y t =

Original: A t A (cid:48) t +1 x t + a (cid:48) t +1 A (cid:48)
Revised: A t A (cid:48) t +1 x t + a (cid:48) t +1 A (cid:48)

Original: t +1 v t ˜ x t +1 , λ t +

Revised: t +1 v t ˜ x t +1 , λ t +

Original: =
Revised: =

Original: O ( y t ; λ (cid:48)
Revised: O ( y t ; λ (cid:48)

Original: t +1 ) if λ t +1 ∈ [ 1 ρ λ (cid:48)
Revised: t +1 ) if λ t +1 ∈ [ 1 ρ λ (cid:48)

Original: t +1 , λ (cid:48)
Revised: t +1 , λ (cid:48)

Original: t +1 ] then a t +1 = a (cid:48) t +1 , A t +1 =
Revised: t +1 ] then a t +1 = a (cid:48) t +1 , A t +1 =

Original: A (cid:48) t
Revised: A (cid:48) t

Original: +  x t +1 =
Revised: +  x t +1 =

Original: ˜ x t +  else if λ t +1 <
Revised: ˜ x t +  else if λ t +1 <

Original: In this section we present our acceleration algorithm
Revised: In this section we present our acceleration algorithm

Original: (Algorithm 1) which removes bisection from the
Revised: (Algorithm 1) which removes bisection from the

Original: MS method (shown in stylized form as Algorithm 0) and thereby attains optimal rates of convergence.
Revised: MS method (shown in stylized form as Algorithm 0) and thereby attains optimal rates of convergence.

Original: For simplicity, in this section and the next we focus on unconstrained optimization ( X = R d ) and assume that f is continuously differentiable, so that ∇ f exists.
Revised: For simplicity, in this section and the next we focus on unconstrained optimization ( X = R d ) and assume that f is continuously differentiable, so that ∇ f exists.

Original: In Appendix C we extend our framework to general closed and convex domains and non-differentiable convex objectives.
Revised: In Appendix C we extend our framework to general closed and convex domains and non-differentiable convex objectives.

Original: The key object in both the original MS algorithm and our new variant is an oracle O that approximately minimizes a local model of f at a query point y .
Revised: The key object in both the original MS algorithm and our new variant is an oracle O that approximately minimizes a local model of f at a query point y .

Original: In particular, O satisﬁes the following approximation error bound, adapted from Monteiro and Svaiter [33, eq.
Revised: In particular, O satisﬁes the following approximation error bound, adapted from Monteiro and Svaiter [32, eq.

Original: (3.3)] ( λ in [33] is 1 /λ in our notation).
Revised: (3.3)] ( λ in [32] is 1 /λ in our notation).

Original: Deﬁnition 1 (MS oracle) .
Revised: Deﬁnition 1 (MS oracle) .

Original: An oracle O : R d × R + → R d × R + is a σ -MS oracle for function f : R d → R if for
Revised: An oracle O : R d × R + → R d × R + is a σ -MS oracle for function f : R d → R if for

Original: every y ∈ R d and λ (cid:48) > 0 , the points ( x, λ )
Revised: every y ∈ R d and λ (cid:48) > 0 , the points ( x, λ )

Original: =
Revised: =

Original: O ( y ; λ (cid:48) ) satisfy
Revised: O ( y ; λ (cid:48) ) satisfy

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition 1 endows the oracle with an additional output λ and an additional input λ (cid:48) .
Revised: Deﬁnition 1 endows the oracle with an additional output λ and an additional input λ (cid:48) .

Original: The value of λ has the following simple interpretation: any point x satisfying (2) approximately minimizes
Revised: The value of λ has the following simple interpretation: any point x satisfying (2) approximately minimizes

Original: F ( x (cid:48) )
Revised: F ( x (cid:48) )

Original: =
Revised: =

Original: f ( x (cid:48) ) + λ 2 (cid:107) x (cid:48)
Revised: f ( x (cid:48) ) + λ 2 (cid:107) x (cid:48)

Original: − y (cid:107) 2 in the sense that (cid:107)∇ F ( x ) (cid:107) ≤ λσ (cid:107) x − y (cid:107) .
Revised: − y (cid:107) 2 in the sense that (cid:107)∇ F ( x ) (cid:107) ≤ λσ (cid:107) x − y (cid:107) .

Original: In particular, computing an exact proximal point x λ = argmin x (cid:48) F ( x (cid:48) ) and outputting ( x, λ ) implements a 0 -MS oracle.
Revised: In particular, computing an exact proximal point x λ = argmin x (cid:48) F ( x (cid:48) ) and outputting ( x, λ ) implements a 0 -MS oracle.

Original: The input λ (cid:48) is optional: oracle implementations in prior work do not require it, but our new adaptive oracles (described in the next section) use it for improved efﬁciency.
Revised: The input λ (cid:48) is optional: oracle implementations in prior work do not require it, but our new adaptive oracles (described in the next section) use it for improved efﬁciency.

Original: In Appendix C we provide a slightly more general approximation condition for MS oracles that handles non-smooth objectives and bounded domains, as well as a different, stochastic condition similar to that of [4, 11].
Revised: In Appendix C we provide a slightly more general approximation condition for MS oracles that handles non-smooth objectives and bounded domains, as well as a different, stochastic condition similar to that of [4, 11].

Original: Let us discuss the key differences between our algorithm (Algorithm 1) and the stylized MS algorithm (Algorithm 0).
Revised: Let us discuss the key differences between our algorithm (Algorithm 1) and the stylized MS algorithm (Algorithm 0).

Original: At every iteration, Algorithm 0 searches for a value λ (cid:48) t +1 such that ˜ x t +1 , λ t +1 =
Revised: At every iteration, Algorithm 0 searches for a value λ (cid:48) t +1 such that ˜ x t +1 , λ t +1 =

Original: O ( y t ; λ (cid:48) t +1 ) satisﬁes λ t
Revised: O ( y t ; λ (cid:48) t +1 ) satisﬁes λ t

Original: +1 ≈ λ (cid:48)
Revised: +1 ≈ λ (cid:48)

Original: t +1 (note that y t depends on λ (cid:48)
Revised: t +1 (note that y t depends on λ (cid:48)

Original: t +1 ).
Revised: t +1 ).

Original: This is done via a bisection procedure iteratively shrinking an interval that contains a successful choice of λ (cid:48)
Revised: This is done via a bisection procedure iteratively shrinking an interval that contains a successful choice of λ (cid:48)

Original: t +1 .
Revised: t +1 .

Original: This bisection process is inefﬁcient in the sense that every time we reach lines 14 and 17 (highlighted in red) all of the optimization progress made by the last oracle call is discarded.
Revised: This bisection process is inefﬁcient in the sense that every time we reach lines 14 and 17 (highlighted in red) all of the optimization progress made by the last oracle call is discarded.

Original: In contrast, even though our algorithm queries O in the same way (with y t computed based on a guess λ (cid:48) t +1 ), it makes use of the oracle output even if λ t +1 is very far from λ (cid:48)
Revised: In contrast, even though our algorithm queries O in the same way (with y t computed based on a guess λ (cid:48) t +1 ), it makes use of the oracle output even if λ t +1 is very far from λ (cid:48)

Original: t +1 , thus never discarding progress made by the oracle.
Revised: t +1 , thus never discarding progress made by the oracle.

Original: Instead of performing a bisection, we compare λ (cid:48) t +1 and λ t +1 to guide our next guess λ (cid:48)
Revised: Instead of performing a bisection, we compare λ (cid:48) t +1 and λ t +1 to guide our next guess λ (cid:48)

Original: t +2 .
Revised: t +2 .

Original: When λ (cid:48) overshoots λ , we decrease it by a factor α (line 11, highlighted in green) and set x t and A t as in Algorithm 0.
Revised: When λ (cid:48) overshoots λ , we decrease it by a factor α (line 11, highlighted in green) and set x t and A t as in Algorithm 0.

Original: When it undershoots, we multiply it by α (line 16).
Revised: When it undershoots, we multiply it by α (line 16).

Original: In this case, we perform an additional key algorithmic modiﬁcation which we call the momentum damping mechanism : we scale down the growth of the parameter A t +1 and replace the next iterate with a convex combination of x t and
Revised: In this case, we perform an additional key algorithmic modiﬁcation which we call the momentum damping mechanism : we scale down the growth of the parameter A t +1 and replace the next iterate with a convex combination of x t and

Original: ˜ x t +1 to ensure that our overly optimistic guess for λ t +1 does not destabilize the algorithm.
Revised: ˜ x t +1 to ensure that our overly optimistic guess for λ t +1 does not destabilize the algorithm.

Original: In Appendix E.6 we demonstrate empirically that this mechanism is important for stabilizing Algorithm 1.
Revised: In Appendix E.6 we demonstrate empirically that this mechanism is important for stabilizing Algorithm 1.

Original: Different MS oracles attain different rates of convergence when accelerated via the MS framework.
Revised: Different MS oracles attain different rates of convergence when accelerated via the MS framework.

Original: In the following deﬁnition, we distill a key property that determines this rate.
Revised: In the following deﬁnition, we distill a key property that determines this rate.

Original: Deﬁnition 2 (Movement bound) .
Revised: Deﬁnition 2 (Movement bound) .

Original: For s ≥ 1 , c, λ > 0 , and x, y ∈ R
Revised: For s ≥ 1 , c, λ > 0 , and x, y ∈ R

Original: d we say that ( x, y, λ ) satisfy a ( s, c ) -movement bound if
Revised: d we say that ( x, y, λ ) satisfy a ( s, c ) -movement bound if

Original: [Equation]
Revised: [Equation]

Original: where a (1 , c ) -movement bound simply means that λ ≤ c .
Revised: where a (1 , c ) -movement bound simply means that λ ≤ c .

Original: In the next section, we will show how to build MS oracles that, given query y , output ( x, λ ) such that ( x, y, λ ) always satisfy a ( s, c ) -movement bound, for certain s and c depending on the oracle type and function structure (e.g., level of smoothness).
Revised: In the next section, we will show how to build MS oracles that, given query y , output ( x, λ ) such that ( x, y, λ ) always satisfy a ( s, c ) -movement bound, for certain s and c depending on the oracle type and function structure (e.g., level of smoothness).

Original: For example, when f has H -Lipschitz Hessian, the cubic-regularized Newton step with M = 2 H is a 12 -MS oracle that guarantees a (2 , √ H ) -movement bound.
Revised: For example, when f has H -Lipschitz Hessian, the cubic-regularized Newton step with M = 2 H is a 12 -MS oracle that guarantees a (2 , √ H ) -movement bound.

Original: With the necessary deﬁnitions in hand, we are ready to state our main result: the iteration (and MS oracle query) complexity of Algorithm 1.
Revised: With the necessary deﬁnitions in hand, we are ready to state our main result: the iteration (and MS oracle query) complexity of Algorithm 1.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let f : R d → R be convex and differentiable, and consider Algorithm 1 with parameters α > 1 , λ (cid:48) > 0 , and a σ -MS oracle (Deﬁnition 1) for f with σ ∈ [0 , 0 . 99) .
Revised: Let f : R d → R be convex and differentiable, and consider Algorithm 1 with parameters α > 1 , λ (cid:48) > 0 , and a σ -MS oracle (Deﬁnition 1) for f with σ ∈ [0 , 0 . 99) .

Original: Let s ≥ 1 and c > 0 , and suppose that for all t such that λ t > λ (cid:48) t or t = 1 , the iterates (˜ x t , y t − 1 , λ t ) satisfy a ( s, c )
Revised: Let s ≥ 1 and c > 0 , and suppose that for all t such that λ t > λ (cid:48) t or t = 1 , the iterates (˜ x t , y t − 1 , λ t ) satisfy a ( s, c )

Original: -movement
Revised: -movement

Original: bound (Deﬁnition 2).
Revised: bound (Deﬁnition 2).

Original: There exist C α,s = O (cid:16)
Revised: There exist C α,s = O (cid:16)

Original: [Equation]
Revised: [Equation]

Original: and K α
Revised: and K α

Original: = O (cid:0)
Revised: = O (cid:0)

Original: [Equation]
Revised: [Equation]

Original: 3 (cid:1) such thatfor any x (cid:63) ∈ R d and (cid:15) > 0 , we have f ( x T ) − f ( x (cid:63) ) ≤ (cid:15)
Revised: 3 (cid:1) such thatfor any x (cid:63) ∈ R d and (cid:15) > 0 , we have f ( x T ) − f ( x (cid:63) ) ≤ (cid:15)

Original: when
Revised: when

Original: [Equation]
Revised: [Equation]

Original: Proof sketch.
Revised: Proof sketch.

Original: The remainder of this section is an overview of the proof of Theorem 1, which we provide in full in Appendix B. To simplify this proof sketch, we treat α , c , and 1 / (1 − σ ) as O (1) , and focus on s < ∞ .
Revised: The remainder of this section is an overview of the proof of Theorem 1, which we provide in full in Appendix B. To simplify this proof sketch, we treat α , c , and 1 / (1 − σ ) as O (1) , and focus on s < ∞ .

Original: To highlight the novel aspects of the proof, let us ﬁrst brieﬂy recall the analysis of Algorithm 

Revised: To highlight the novel aspects of the proof, let us ﬁrst brieﬂy recall the analysis of Algorithm

Original: For every t ≤ T let
Revised: For every t ≤ T let

Original: [Equation]
Revised: [Equation]

Original: The key facts about the standard MS iterations are
Revised: The key facts about the standard MS iterations are

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst fact implies that the optimality gap at iteration T is inversely proportional to A T , while the latter two facts imply that A T grows rapidly.
Revised: The ﬁrst fact implies that the optimality gap at iteration T is inversely proportional to A T , while the latter two facts imply that A T grows rapidly.

Original: More speciﬁcally, substituting the movement bound
Revised: More speciﬁcally, substituting the movement bound

Original: M t ≥ Ω (cid:0) ( λ t ) 2 / ( s − 1) (cid:1) and λ (cid:48)
Revised: M t ≥ Ω (cid:0) ( λ t ) 2 / ( s − 1) (cid:1) and λ (cid:48)

Original: t ≥ Ω( λ t ) (thanks to the bisection) yields (cid:80) t ∈
Revised: t ≥ Ω( λ t ) (thanks to the bisection) yields (cid:80) t ∈

Original: [ T ] λ (cid:48) t s +1 s − 1 A t =
Revised: [ T ] λ (cid:48) t s +1 s − 1 A t =

Original: O ( D 0 ) .
Revised: O ( D 0 ) .

Original: Combining this with the third fact in (4) and using the reverse Hölder inequality allows one to conclude that, for k = s +1 3 s +1 and k (cid:48)
Revised: Combining this with the third fact in (4) and using the reverse Hölder inequality allows one to conclude that, for k = s +1 3 s +1 and k (cid:48)

Original: =
Revised: =

Original: s − 1 3 s +1 , we have A kT ≥ Ω( D − k
Revised: s − 1 3 s +1 , we have A kT ≥ Ω( D − k

Original: [Equation]
Revised: [Equation]

Original: ) (cid:80) t ∈ [ T ] A k (cid:48) i , which, upon further algebraic manipulation, yields A T ≥ Ω( T (3 s +1) / 2 D − ( s − 1) / 2) .
Revised: ) (cid:80) t ∈ [ T ] A k (cid:48) i , which, upon further algebraic manipulation, yields A T ≥ Ω( T (3 s +1) / 2 D − ( s − 1) / 2) .

Original: Plugging this back to to the ﬁrst fact in (4) gives the claimed convergence rate.
Revised: Plugging this back to to the ﬁrst fact in (4) gives the claimed convergence rate.

Original: Having described the standard MS analysis, we move on to our algorithm.
Revised: Having described the standard MS analysis, we move on to our algorithm.

Original: Our ﬁrst challenge is re-establishing the facts (4).
Revised: Our ﬁrst challenge is re-establishing the facts (4).

Original: The difﬁcult case is λ t > λ (cid:48) t , where the standard cancellation that occurs in the MS analysis may fail.
Revised: The difﬁcult case is λ t > λ (cid:48) t , where the standard cancellation that occurs in the MS analysis may fail.

Original: This is where the momentum damping mechanism (lines 14 and 15 of our algorithm) comes into play, allowing us to show that (See Proposition 1 in the appendix)
Revised: This is where the momentum damping mechanism (lines 14 and 15 of our algorithm) comes into play, allowing us to show that (See Proposition 1 in the appendix)

Original: [Equation]
Revised: [Equation]

Original: and (cid:112)
Revised: and (cid:112)

Original: [Equation]
Revised: [Equation]

Original: 5) where S ≥ T := { t ∈ [ T ] | λ t ≥ λ (cid:48) t } and S >T , S ≤ T , S <T and S =
Revised: 5) where S ≥ T := { t ∈ [ T ] | λ t ≥ λ (cid:48) t } and S >T , S ≤ T , S <T and S =

Original: T are analogously deﬁned.
Revised: T are analogously deﬁned.

Original: Comparing (4) and (5), the price of removing the bisection becomes evident: at each iteration (except the ﬁrst) only one of the terms forcing the growth of A t receives a contribution.
Revised: Comparing (4) and (5), the price of removing the bisection becomes evident: at each iteration (except the ﬁrst) only one of the terms forcing the growth of A t receives a contribution.

Original: The second challenge of our proof is establishing a lower bound on √ A T in terms of the 1 / (cid:112) λ (cid:48)
Revised: The second challenge of our proof is establishing a lower bound on √ A T in terms of the 1 / (cid:112) λ (cid:48)

Original: t values for t ∈ S >T ∪ { 1 } , where the movement bound holds for M t .
Revised: t values for t ∈ S >T ∪ { 1 } , where the movement bound holds for M t .

Original: This is where the multiplicative λ (cid:48) update rule (lines 11 and 16 of the algorithm) comes into play: it allows us to “credit” the contribution of every “down iterate” (in S ≤ T ) to an adjacent “up” iterate ( S >T ∪ { 1 } ) and furthermore argue that the contribution gets an exponential bonus based on the distance between the two.
Revised: This is where the multiplicative λ (cid:48) update rule (lines 11 and 16 of the algorithm) comes into play: it allows us to “credit” the contribution of every “down iterate” (in S ≤ T ) to an adjacent “up” iterate ( S >T ∪ { 1 } ) and furthermore argue that the contribution gets an exponential bonus based on the distance between the two.

Original: Consequently, we are able to identify a set Q T ⊆ S >T ∪ { 1 } of iterates, and a sequence { r t } such that (see Lemma 1) √
Revised: Consequently, we are able to identify a set Q T ⊆ S >T ∪ { 1 } of iterates, and a sequence { r t } such that (see Lemma 1) √

Original: [Equation]
Revised: [Equation]

Original: Repeating the reverse Hölder argument of prior work, we obtain the recursive bound
Revised: Repeating the reverse Hölder argument of prior work, we obtain the recursive bound

Original: [Equation]
Revised: [Equation]

Original: ≥ Ω( D − k (cid:48))
Revised: ≥ Ω( D − k (cid:48))

Original: (cid:88) t ∈Q T
Revised: (cid:88) t ∈Q T

Original: A k (cid:48)
Revised: A k (cid:48)

Original: t α kr t ≥ Ω
Revised: t α kr t ≥ Ω

Original: ( D − k (cid:48))
Revised: ( D − k (cid:48))

Original: (cid:88) t ∈Q T
Revised: (cid:88) t ∈Q T

Original: A k (cid:48) t r t (6) with k = s +1 3 s +1 and k (cid:48)
Revised: A k (cid:48) t r t (6) with k = s +1 3 s +1 and k (cid:48)

Original: =
Revised: =

Original: s − 1 3 s +1 as before.
Revised: s − 1 3 s +1 as before.

Original: The ﬁnal challenge of our proof is to show that such recursion implies sufﬁcient growth of A t .
Revised: The ﬁnal challenge of our proof is to show that such recursion implies sufﬁcient growth of A t .

Original: This is where careful algebra comes into play; we show that (6) implies that A T ≥ Ω (cid:16) (
Revised: This is where careful algebra comes into play; we show that (6) implies that A T ≥ Ω (cid:16) (

Original: (cid:80) t ∈
Revised: (cid:80) t ∈

Original: [ T ] r t ) (3 s +1) / 2 D − ( s − 1) / 2(cid:17) (see Lemmas 3 and 4) which establishes our result since (cid:80) t ∈
Revised: [ T ] r t ) (3 s +1) / 2 D − ( s − 1) / 2(cid:17) (see Lemmas 3 and 4) which establishes our result since (cid:80) t ∈

Original: [ T ] r t =
Revised: [ T ] r t =

Original: T − 1 2 .
Revised: T − 1 2 .

Original:  MS oracle implementations
Revised:  MS oracle implementations

Original: In this section we describe several oracles that satisfy both Deﬁnition 1 (the MS condition) and
Revised: In this section we describe several oracles that satisfy both Deﬁnition 1 (the MS condition) and

Original: Deﬁnition 2 (movement bounds) and may therefore be used by Algorithm 1.
Revised: Deﬁnition 2 (movement bounds) and may therefore be used by Algorithm 1.

Original: Section 3.1 brieﬂy reviews oracles that have appeared in prior work, while Section 3.2 and Section 3.3 describe our new adaptive oracle implementations.
Revised: Section 3.1 brieﬂy reviews oracles that have appeared in prior work, while Section 3.2 and Section 3.3 describe our new adaptive oracle implementations.

Original: We summarize the key oracle properties and resulting complexity bounds in Table 1.
Revised: We summarize the key oracle properties and resulting complexity bounds in Table 1.

Original: 3.1 Oracles from prior work
Revised: 3.1 Oracles from prior work

Original: Here we consider several previously-studied oracles of the form ( x, λ )
Revised: Here we consider several previously-studied oracles of the form ( x, λ )

Original: =
Revised: =

Original: O ( y ) , where we omit the second argument λ (cid:48) since prior work does not leverage it to improve implementation efﬁciency.
Revised: O ( y ) , where we omit the second argument λ (cid:48) since prior work does not leverage it to improve implementation efﬁciency.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Gradient descent step [e.g., 35].
Revised: Gradient descent step [e.g., 34].

Original: As a gentle start, consider the oracle O gd ( y )
Revised: As a gentle start, consider the oracle O gd ( y )

Original: =
Revised: =

Original: ( y − η ∇ f ( y ) , 1 η ) , i.e., an oracle that returns x by taking standard gradient step with size η and λ = 1 /η .
Revised: ( y − η ∇ f ( y ) , 1 η ) , i.e., an oracle that returns x by taking standard gradient step with size η and λ = 1 /η .

Original: Obviously, the oracle always satisﬁes a (1 , η − 1 ) -movement bound.
Revised: Obviously, the oracle always satisﬁes a (1 , η − 1 ) -movement bound.

Original: Moreover, if we assume that ∇ f is L -Lipschitz, then (cid:107) x − ( y − 1 λ ∇ f ( x ))
Revised: Moreover, if we assume that ∇ f is L -Lipschitz, then (cid:107) x − ( y − 1 λ ∇ f ( x ))

Original: (cid:107) =
Revised: (cid:107) =

Original: η (cid:107)∇ f ( x ) − ∇ f ( y ) (cid:107) ≤ ηL (cid:107) x − y (cid:107) .
Revised: η (cid:107)∇ f ( x ) − ∇ f ( y ) (cid:107) ≤ ηL (cid:107) x − y (cid:107) .

Original: Therefore, when η − 1 ≥ L/σ the oracle is a σ -MS oracle.
Revised: Therefore, when η − 1 ≥ L/σ the oracle is a σ -MS oracle.

Original: Taylor descent step [5, 36, 20, 7, 24, 42].
Revised: Taylor descent step [5, 35, 19, 7, 23, 41].

Original: Generalizing both O gd and the cubic-regularized Newton step oracle O cr , we deﬁne for every integer p ≥ 1 and ν ∈ [0 , 1] the oracle O p,ν -reg , that, for parameter C and input y returns ( x, λ ) =
Revised: Generalizing both O gd and the cubic-regularized Newton step oracle O cr , we deﬁne for every integer p ≥ 1 and ν ∈ [0 , 1] the oracle O p,ν -reg , that, for parameter C and input y returns ( x, λ ) =

Original: O p,ν -reg ( y ) where
Revised: O p,ν -reg ( y ) where

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and ˜ f p ( x ; y ) :
Revised: and ˜ f p ( x ; y ) :

Original: = (cid:80) pi =

Revised: = (cid:80) pi =

Original: i ! ∇ i f ( y )
Revised: i ! ∇ i f ( y )

Original: [( x − y ) ⊗ i ] is the Taylor expansion of f around y evaluated at x .
Revised: [( x − y ) ⊗ i ] is the Taylor expansion of f around y evaluated at x .

Original: Oracles O gd and O cr correspond to the special cases
Revised: Oracles O gd and O cr correspond to the special cases

Original: O 1 , 1 -reg (with η = M − 1 ) and O 2 , 1 -reg , respectively.
Revised: O 1 , 1 -reg (with η = M − 1 ) and O 2 , 1 -reg , respectively.

Original: Clearly, by deﬁnition, the oracle always satisﬁes a ( p + ν − 1 , ( M/p !)
Revised: Clearly, by deﬁnition, the oracle always satisﬁes a ( p + ν − 1 , ( M/p !)

Original: 1 / ( p + ν − 1) ) - movement bound.
Revised: 1 / ( p + ν − 1) ) - movement bound.

Original: Moreover, it is easy to show that
Revised: Moreover, it is easy to show that

Original: [Equation]
Revised: [Equation]

Original: For any p ≥ 1 and ν ∈ [0 , 1] we say that ∇ p f is ( H, ν ) -Hölder if for all x, y we have
Revised: For any p ≥ 1 and ν ∈ [0 , 1] we say that ∇ p f is ( H, ν ) -Hölder if for all x, y we have

Original: [Equation]
Revised: [Equation]

Original: p f
Revised: p f

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: ν .
Revised: ν .

Original: (An ( H, 1) -Hölder derivative is H -Lipschitz.)
Revised: (An ( H, 1) -Hölder derivative is H -Lipschitz.)

Original: If ∇ p f is ( H, ν ) -Hölder, Taylor’s theorem gives (cid:107)∇ f ( x ) −∇ ˜ f p ( x ; y ) (cid:107) ≤ Hp !
Revised: If ∇ p f is ( H, ν ) -Hölder, Taylor’s theorem gives (cid:107)∇ f ( x ) −∇ ˜ f p ( x ; y ) (cid:107) ≤ Hp !

Original: (cid:107) x − y (cid:107) p
Revised: (cid:107) x − y (cid:107) p

Original: + ν − 1 [42, Lemma 2.5], and so (cid:107) x − ( y − 1 λ ∇ f ( x ))
Revised: + ν − 1 [41, Lemma 2.5], and so (cid:107) x − ( y − 1 λ ∇ f ( x ))

Original: (cid:107) ≤ HM (cid:107) x
Revised: (cid:107) ≤ HM (cid:107) x

Original: − y (cid:107) .
Revised: − y (cid:107) .

Original: Therefore, when M ≥ H/σ the oracle is a σ -MS oracle.
Revised: Therefore, when M ≥ H/σ the oracle is a σ -MS oracle.

Original: Exploiting third-order smoothness with a second order oracle [37, 27].
Revised: Exploiting third-order smoothness with a second order oracle [36, 26].

Original: For p > 2 , computing O p,ν -reg is typically intractable due to the need to compute the high-order derivative tensors ∇ 3 f ( y ) , ∇ 4 f ( y ) , . .
Revised: For p > 2 , computing O p,ν -reg is typically intractable due to the need to compute the high-order derivative tensors ∇ 3 f ( y ) , ∇ 4 f ( y ) , . .

Original: , ∇ p f ( y ) .
Revised: , ∇ p f ( y ) .

Original: Nevertheless for p = 3 Nesterov [37] designs an approximate solver for (7), which we denote O 3 -reg-so , using only ∇ 2 f ( y ) and a logarithmic number of gradient evaluations.
Revised: Nevertheless for p = 3 Nesterov [36] designs an approximate solver for (7), which we denote O 3 -reg-so , using only ∇ 2 f ( y ) and a logarithmic number of gradient evaluations.

Original: When ∇ 3 f is ( L 3 , 1) -Hölder, [37] shows that
Revised: When ∇ 3 f is ( L 3 , 1) -Hölder, [36] shows that

Original: O 3 -reg-so is a valid MS-oracle satisfying a (3 , O
Revised: O 3 -reg-so is a valid MS-oracle satisfying a (3 , O

Original: ( L 3 ))
Revised: ( L 3 ))

Original: -movement bound, on par with the movement bound of O 3 , 1 -reg .
Revised: -movement bound, on par with the movement bound of O 3 , 1 -reg .

Original: Exact ball optimization oracle [12].
Revised: Exact ball optimization oracle [12].

Original: For a given query y , consider the exact minimizer of f constrained to a ball of radius r around y , i.e., consider an oracle
Revised: For a given query y , consider the exact minimizer of f constrained to a ball of radius r around y , i.e., consider an oracle

Original: O r -ball such that ( x, λ ) =
Revised: O r -ball such that ( x, λ ) =

Original: O r -ball ( y ) satisfy x ∈ argmin x (cid:48) :
Revised: O r -ball ( y ) satisfy x ∈ argmin x (cid:48) :

Original: (cid:107) x (cid:48) − y (cid:107)≤ r f ( x (cid:48) ) and λ = (cid:107)∇ f ( x ) (cid:107) (cid:107) x − y (cid:107) .
Revised: (cid:107) x (cid:48) − y (cid:107)≤ r f ( x (cid:48) ) and λ = (cid:107)∇ f ( x ) (cid:107) (cid:107) x − y (cid:107) .

Original: One may easily verify that (unless λ =
Revised: One may easily verify that (unless λ =

Original: (cid:107)∇ f ( x ) (cid:107) = 0 )
Revised: (cid:107)∇ f ( x ) (cid:107) = 0 )

Original: we have x = y − 1 λ ∇ f ( x ) , and therefore the oracle is a 0 -MS oracle.
Revised: we have x = y − 1 λ ∇ f ( x ) , and therefore the oracle is a 0 -MS oracle.

Original: Moreover, when f is convex, we have either (cid:107) x − y (cid:107) =
Revised: Moreover, when f is convex, we have either (cid:107) x − y (cid:107) =

Original: r or x is a global minimizer of f , and so we may assume without loss of generality that the oracle satisﬁes an ( ∞ , 1 /r ) movement bound.
Revised: r or x is a global minimizer of f , and so we may assume without loss of generality that the oracle satisﬁes an ( ∞ , 1 /r ) movement bound.

Original: Ball-Constrained Newton (BaCoN) oracle [12].
Revised: Ball-Constrained Newton (BaCoN) oracle [12].

Original: Exactly implementing O r -ball is generally intractable.
Revised: Exactly implementing O r -ball is generally intractable.

Original: Nevertheless, Carmon et al.
Revised: Nevertheless, Carmon et al.

Original: [12, Alg. 3] describe a method O r -BaCoN based on solving a sequence of (cid:101) O (1) trust-region problems (ball-constrained Newton steps), which we call that, for functions that are O (1)
Revised: [12, Alg. 3] describe a method O r -BaCoN based on solving a sequence of (cid:101) O (1) trust-region problems (ball-constrained Newton steps), which we call that, for functions that are O (1)

Original: -Hessian stable in a ball of radius r (or 1 /r -quasi-self-concordant) and have a ﬁnite condition number, outputs ( x, λ ) satisfying the 12 -MS oracle condition and an ( ∞ , O (1 /r )) - movement bound.
Revised: -Hessian stable in a ball of radius r (or 1 /r -quasi-self-concordant) and have a ﬁnite condition number, outputs ( x, λ ) satisfying the 12 -MS oracle condition and an ( ∞ , O (1 /r )) - movement bound.

Original: Implementing O r -BaCoN requires only a single Hessian evaluation and a number of linear system solutions that is polylogarithmic in problem parameters.
Revised: Implementing O r -BaCoN requires only a single Hessian evaluation and a number of linear system solutions that is polylogarithmic in problem parameters.

Original: Subsequent works implementing ball oracles [13, 4, 11] satisfy an approximation guarantee different than the MS condition, similar to the one we describe in Appendix C.
Revised: Subsequent works implementing ball oracles [13, 4, 11] satisfy an approximation guarantee different than the MS condition, similar to the one we describe in Appendix C.

Original: 3.2 An adaptive Monteiro-Svaiter-Newton oracle
Revised: 3.2 An adaptive Monteiro-Svaiter-Newton oracle

Original: The oracle implementations in Section 3.1 satisfy movement bounds by design and the MS condition (2) by assumption.
Revised: The oracle implementations in Section 3.1 satisfy movement bounds by design and the MS condition (2) by assumption.

Original: For example, the cubic-regularized Newton step oracle O cr is guaranteed to satisfy the MS condition only when the regularization parameter M is sufﬁciently larger than the Lipschitz constant of ∇ 2 f .
Revised: For example, the cubic-regularized Newton step oracle O cr is guaranteed to satisfy the MS condition only when the regularization parameter M is sufﬁciently larger than the Lipschitz constant of ∇ 2 f .

Original: This suggests that M must be carefully tuned to ensure good performance.
Revised: This suggests that M must be carefully tuned to ensure good performance.

Original: Prior work attempt to dynamically adjust M in order to meet certain approximation conditions [14,  21, 22, 25].
Revised: Prior work attempt to dynamically adjust M in order to meet certain approximation conditions [14, 20, 21, 24].

Original: However, even computing a single cubic-regularized Newton step entails searching for λ that satisﬁes (cid:107) [ ∇ 2 f ( y )
Revised: However, even computing a single cubic-regularized Newton step entails searching for λ that satisﬁes (cid:107) [ ∇ 2 f ( y )

Original: + λI ] − 1 ∇ f ( y ) (cid:107) =
Revised: + λI ] − 1 ∇ f ( y ) (cid:107) =

Original: Mλ 2 .
Revised: Mλ 2 .

Original: Therefore, such a search over M is essentially a (potentially) redundant double search over λ .
Revised: Therefore, such a search over M is essentially a (potentially) redundant double search over λ .

Original: We propose a more direct and more adaptive MS oracle recipe: search for the smallest λ for which the regularized Newton step x = y − [ ∇ 2 f ( y )
Revised: We propose a more direct and more adaptive MS oracle recipe: search for the smallest λ for which the regularized Newton step x = y − [ ∇ 2 f ( y )

Original: + λI ] − 1 ∇ f ( y ) satisﬁes the MS condition (2).
Revised: + λI ] − 1 ∇ f ( y ) satisﬁes the MS condition (2).

Original: This yields valid MS oracle by construction, independently of any assumption.
Revised: This yields valid MS oracle by construction, independently of any assumption.

Original: Moreover, it is simple to argue that when ∇ 2 f is ( H, ν ) -Hölder continuous for some ν ∈ [0 , 1] , such oracle would guarantee the same movement bound as O 2 ,ν -reg with the best choice parameters M and ν
Revised: Moreover, it is simple to argue that when ∇ 2 f is ( H, ν ) -Hölder continuous for some ν ∈ [0 , 1] , such oracle would guarantee the same movement bound as O 2 ,ν -reg with the best choice parameters M and ν

Original: (see Appendix D.1)—even though our recipe requires neither of these parameters!
Revised: (see Appendix D.1)—even though our recipe requires neither of these parameters!

Original: Exactly fulﬁlling this recipe, i.e., ﬁnding the ideal minimal λ (cid:63) that satisﬁes the MS condition, is difﬁcult.
Revised: Exactly fulﬁlling this recipe, i.e., ﬁnding the ideal minimal λ (cid:63) that satisﬁes the MS condition, is difﬁcult.

Original: Fortunately, to adaptively guarantee movement bounds, it sufﬁces to ﬁnd a value λ such the corresponding regularized Newton step satisﬁes the MS condition, while the step corresponding to λ/ 2 does not; Algorithm 2 ﬁnds precisely such a λ .
Revised: Fortunately, to adaptively guarantee movement bounds, it sufﬁces to ﬁnd a value λ such the corresponding regularized Newton step satisﬁes the MS condition, while the step corresponding to λ/ 2 does not; Algorithm 2 ﬁnds precisely such a λ .

Original: Let us describe the operation of Algorithm 2.
Revised: Let us describe the operation of Algorithm 2.

Original: If the input λ (cid:48) is invalid (i.e., its corresponding regularized Newton step does not satisfy the MS condition so that C HECK MS ( λ (cid:48) ; y, σ ) evaluates to
Revised: If the input λ (cid:48) is invalid (i.e., its corresponding regularized Newton step does not satisfy the MS condition so that C HECK MS ( λ (cid:48) ; y, σ ) evaluates to

Original: False), we set λ invld ← λ (cid:48) and test a double-exponentially increasing series of λ ’s, until reaching a valid λ vld (line 11).
Revised: False), we set λ invld ← λ (cid:48) and test a double-exponentially increasing series of λ ’s, until reaching a valid λ vld (line 11).

Original: If λ (cid:48) is valid and the LAZY ﬂag is set, we return it immediately.
Revised: If λ (cid:48) is valid and the LAZY ﬂag is set, we return it immediately.

Original: Otherwise (if LAZY is not set) we set λ vld = λ (cid:48) and decrease it at a double-exponential rate until ﬁnding an invalid λ invld (line 5).
Revised: Otherwise (if LAZY is not set) we set λ vld = λ (cid:48) and decrease it at a double-exponential rate until ﬁnding an invalid λ invld (line 5).

Original: In either case (so long as LAZY is not set) we obtain an (invalid,valid) pair ( λ invld , λ vld ) such that λ vld /λ invld = 2 2 k(cid:63) at the cost of 2 + k (cid:63) linear system solutions.
Revised: In either case (so long as LAZY is not set) we obtain an (invalid,valid) pair ( λ invld , λ vld ) such that λ vld /λ invld = 2 2 k(cid:63) at the cost of 2 + k (cid:63) linear system solutions.

Original: We then perform precisely k (cid:63) log-scale bisection steps in order to shrink λ vld /λ invld down to 2 while maintaining the invariant that λ vld is valid and λ invld is invalid (line 15).
Revised: We then perform precisely k (cid:63) log-scale bisection steps in order to shrink λ vld /λ invld down to 2 while maintaining the invariant that λ vld is valid and λ invld is invalid (line 15).

Original: The following theorem bounds the complexity of Algorithm 2 in terms of linear-system solution number, and establishes a movement bound for its output assuming that ∇ 2 f is locally Hölder around the query point.
Revised: The following theorem bounds the complexity of Algorithm 2 in terms of linear-system solution number, and establishes a movement bound for its output assuming that ∇ 2 f is locally Hölder around the query point.

Original: We defer the proof of the theorem and its following corollary to Appendix D.2.
Revised: We defer the proof of the theorem and its following corollary to Appendix D.2.

Original: Theorem 2.
Revised: Theorem 2.

Original: Algorithm 2 with parameter σ is a σ -MS oracle O aMSN .
Revised: Algorithm 2 with parameter σ is a σ -MS oracle O aMSN .

Original: For any y ∈ R d , computing ( x, λ ) =
Revised: For any y ∈ R d , computing ( x, λ ) =

Original: O aMSN ( y ) requires at most 

Revised: O aMSN ( y ) requires at most

Original: + 2 log 2 (cid:0) 1 + (cid:12) (cid:12) log 2 λ λ (cid:48)
Revised: + 2 log 2 (cid:0) 1 + (cid:12) (cid:12) log 2 λ λ (cid:48)

Original: (cid:12) (cid:12) (cid:1) linear systems solutions.
Revised: (cid:12) (cid:12) (cid:1) linear systems solutions.

Original: If LAZY is
Revised: If LAZY is

Original: False or λ > λ (cid:48) , and if ∇ 2 f is ( H, ν ) -Hölder in a ball of radius 2 (cid:107) x − y (cid:107) around y , then ( x, y, λ ) satisfy a (cid:0) 1 + ν, (2 H/σ ) 1 / (1+ ν ) /σ (cid:1) -movement bound.
Revised: False or λ > λ (cid:48) , and if ∇ 2 f is ( H, ν ) -Hölder in a ball of radius 2 (cid:107) x − y (cid:107) around y , then ( x, y, λ ) satisfy a (cid:0) 1 + ν, (2 H/σ ) 1 / (1+ ν ) /σ (cid:1) -movement bound.

Original: Input: Query y ∈ R d , λ (cid:48) > 0 .
Revised: Input: Query y ∈ R d , λ (cid:48) > 0 .

Original: Flag LAZY .
Revised: Flag LAZY .

Original: Parameters: MS factor σ ∈ (0 , 1) .
Revised: Parameters: MS factor σ ∈ (0 , 1) .

Original: 1 if C HECK MS ( λ (cid:48) ; y, σ ) then if LAZY then return y − [ ∇ 2 f ( y ) + λ (cid:48)
Revised: 1 if C HECK MS ( λ (cid:48) ; y, σ ) then if LAZY then return y − [ ∇ 2 f ( y ) + λ (cid:48)

Original: I ] − 1 ∇ f ( y ) , λ (cid:48) else
Revised: I ] − 1 ∇ f ( y ) , λ (cid:48) else

Original: λ vld ← λ (cid:48) , k ← while C HECK MS ( λ vld / 2 2 k ; y, σ ) do λ vld ← λ vld / 2 

Revised: λ vld ← λ (cid:48) , k ← while C HECK MS ( λ vld / 2 2 k ; y, σ ) do λ vld ← λ vld / 2

Original: k k ← k + k (cid:63) ← k , λ invld ← λ vld / 2 2 k(cid:63) 9 else λ invld ← λ (cid:48) , k ← while not C HECK MS ( λ invld 2 2 k ; y, σ ) do λ invld ← λ invld 2 2 k k ← k + k (cid:63) ← k , λ vld ← λ invld 2 2 k(cid:63) 15 while λ invld < λ vld / 2 do λ ← √ λ invld λ vld if C HECK MS ( λ ; y, σ ) then λ vld ← λ else λ invld ← λ 19 return y − [ ∇ 2 f ( y )+ λ vld I ] − 1 ∇ f ( y ) , λ vld 20 function C HECK MS ( λ ; y, σ ) x = y − [ ∇ 2 f ( y ) + λI ] − 1 ∇ f ( y ) if (cid:13)(cid:13) x − ( y − 1 λ ∇ f ( x ))
Revised: k k ← k + k (cid:63) ← k , λ invld ← λ vld / 2 2 k(cid:63) 9 else λ invld ← λ (cid:48) , k ← while not C HECK MS ( λ invld 2 2 k ; y, σ ) do λ invld ← λ invld 2 2 k k ← k + k (cid:63) ← k , λ vld ← λ invld 2 2 k(cid:63) 15 while λ invld < λ vld / 2 do λ ← √ λ invld λ vld if C HECK MS ( λ ; y, σ ) then λ vld ← λ else λ invld ← λ 19 return y − [ ∇ 2 f ( y ) + λ vld I ] − 1 ∇ f ( y ) , λ vld 20 function C HECK MS ( λ ; y, σ ) x = y − [ ∇ 2 f ( y ) + λI ] − 1 ∇ f ( y ) if (cid:13)(cid:13) x − ( y − 1 λ ∇ f ( x ))

Original: (cid:13)(cid:13) ≤ σ (cid:107) x
Revised: (cid:13)(cid:13) ≤ σ (cid:107) x

Original: − y (cid:107) then return True else return False
Revised: − y (cid:107) then return True else return False

Original: Algorithm 3:
Revised: Algorithm 3:

Original: Input: y ∈ R d , λ (cid:48) > 0 .
Revised: Input: y ∈ R d , λ (cid:48) > 0 .

Original: Flag LAZY .
Revised: Flag LAZY .

Original: Parameters: MS factor σ ∈ (0 , 1) .
Revised: Parameters: MS factor σ ∈ (0 , 1) .

Original: 1 λ ← λ (cid:48) , F AILED C HECK ← False 2 Repeat
Revised: 1 λ ← λ (cid:48) , F AILED C HECK ← False 2 Repeat

Original: A ← ∇ 2 f ( y ) + λI , b ← −∇ f ( y ) x ← y + C ONJ R ES
Revised: A ← ∇ 2 f ( y ) + λI , b ← −∇ f ( y ) x ← y + C ONJ R ES

Original: ( A, b, λσ ) if (cid:13)(cid:13) x − ( y − 1 λ ∇ f ( x ))
Revised: ( A, b, λσ ) if (cid:13)(cid:13) x − ( y − 1 λ ∇ f ( x ))

Original: (cid:13)(cid:13) ≤ σ (cid:107) x
Revised: (cid:13)(cid:13) ≤ σ (cid:107) x

Original: − y (cid:107) then if LAZY or F AILED C HECK then return x , λ else λ ← λ/ else
Revised: − y (cid:107) then if LAZY or F AILED C HECK then return x , λ else λ ← λ/ else

Original: F AILED C HECK ← True λ ← 2 λ 12 function C ONJ R ES ( A, b, λσ ) w 0 ← p 0 ← r 0 ← Aw 0 − b (cid:46) r i
Revised: F AILED C HECK ← True λ ← 2 λ 12 function C ONJ R ES ( A, b, λσ ) w 0 ← p 0 ← r 0 ← Aw 0 − b (cid:46) r i

Original: =
Revised: =

Original: Aw i − b s 0 ← q 0 ← Ar(cid:46)
Revised: Aw i − b s 0 ← q 0 ← Ar(cid:46)

Original: q i =
Revised: q i =

Original: Ap i
Revised: Ap i

Original: i ← while (cid:107) r i (cid:107) >
Revised: i ← while (cid:107) r i (cid:107) >

Original: λσ 2 (cid:107) w i (cid:107) do w i +1 ← w i − (cid:104)
Revised: λσ 2 (cid:107) w i (cid:107) do w i +1 ← w i − (cid:104)

Original: r i ,s i (cid:105) (cid:107) q
Revised: r i ,s i (cid:105) (cid:107) q

Original: i (cid:107) 2 p i r i +1 ← r i − (cid:104) r
Revised: i (cid:107) 2 p i r i +1 ← r i − (cid:104) r

Original: i ,s i (cid:105) (cid:107) q
Revised: i ,s i (cid:105) (cid:107) q

Original: i (cid:107) 2 q i s i +

Revised: i (cid:107) 2 q i s i +

Original: ←
Revised: ←

Original: Ar
Revised: Ar

Original: i +  p
Revised: i +  p

Original: i +1 ← (cid:104) r
Revised: i +1 ← (cid:104) r

Original: i +1 ,
Revised: i +1 ,

Original: s i +1 (cid:105) (cid:104) r
Revised: s i +1 (cid:105) (cid:104) r

Original: i ,s i (cid:105) p
Revised: i ,s i (cid:105) p

Original: i
Revised: i

Original: + r i +  
Revised: + r i +

Original: q
Revised: q

Original: i +1 ← (cid:104) r
Revised: i +1 ← (cid:104) r

Original: i +1 ,
Revised: i +1 ,

Original: s i +1 (cid:105) (cid:104) r
Revised: s i +1 (cid:105) (cid:104) r

Original: i ,s i (cid:105) q
Revised: i ,s i (cid:105) q

Original: i + s i +  
Revised: i + s i +

Original: i ← i + return w i
Revised: i ← i + return w i

Original: To understand the LAZY option of Algorithm 2, note that when λ (cid:48) is valid we will necessarily output λ ≤ λ (cid:48) .
Revised: To understand the LAZY option of Algorithm 2, note that when λ (cid:48) is valid we will necessarily output λ ≤ λ (cid:48) .

Original: In such case Theorem 1 does not require a movement bound (except for the ﬁrst iteration).
Revised: In such case Theorem 1 does not require a movement bound (except for the ﬁrst iteration).

Original: Therefore, we might as well save on computation and return λ (cid:48) .
Revised: Therefore, we might as well save on computation and return λ (cid:48) .

Original: The following Corollary 3 gives the overall complexity bound for the combination of Algorithm 1 and O aMSN , leveraging “lazy” oracle calls to show that the number of linear system solves per iteration is essentially constant.
Revised: The following Corollary 3 gives the overall complexity bound for the combination of Algorithm 1 and O aMSN , leveraging “lazy” oracle calls to show that the number of linear system solves per iteration is essentially constant.

Original: Corollary 3.
Revised: Corollary 3.

Original: Consider Algorithm 1 with initial point x 0 , parameters α satisfying 1 .
Revised: Consider Algorithm 1 with initial point x 0 , parameters α satisfying 1 .

Original: 1 ≤ α =
Revised: 1 ≤ α =

Original: O (1) and λ (cid:48) 0 , and σ -MS oracle O aMSN (with LAZY =
Revised: O (1) and λ (cid:48) 0 , and σ -MS oracle O aMSN (with LAZY =

Original: True in all but the ﬁrst iteration) with σ ∈ (0 . 01 , 0 . 99) .
Revised: True in all but the ﬁrst iteration) with σ ∈ (0 . 01 , 0 . 99) .

Original: For any H, (cid:15) > 0 , ν ∈ [0 , 1] and any x (cid:63) ∈ R d , if f is convex with ( H, ν ) -
Revised: For any H, (cid:15) > 0 , ν ∈ [0 , 1] and any x (cid:63) ∈ R d , if f is convex with ( H, ν ) -

Original: Hölder Hessian, the algorithm produces an iterate x T such that f ( x T ) ≤ f ( x (cid:63) )
Revised: Hölder Hessian, the algorithm produces an iterate x T such that f ( x T ) ≤ f ( x (cid:63) )

Original: + (cid:15) using T =
Revised: + (cid:15) using T =

Original: O (cid:16)(cid:0) H (cid:107) x 

Revised: O (cid:16)(cid:0) H (cid:107) x

Original: − x (cid:63) (cid:107) 2+ ν /(cid:15) (cid:1) 2 / (4+3 ν ) (cid:17) Hessian evaluations and O (cid:16) T + log log max (cid:110) HR ν λ (cid:48) 0 , λ (cid:48) 

Revised: − x (cid:63) (cid:107) 2+ ν /(cid:15) (cid:1) 2 / (4+3 ν ) (cid:17) Hessian evaluations and O (cid:16) T + log log max (cid:110) HR ν λ (cid:48) 0 , λ (cid:48)

Original: R 2 (cid:15) (cid:111)(cid:17) linear system solutions, where R is the distance between x 0 and argmin x (cid:48)
Revised: R 2 (cid:15) (cid:111)(cid:17) linear system solutions, where R is the distance between x 0 and argmin x (cid:48)

Original: f ( x (cid:48) ) .
Revised: f ( x (cid:48) ) .

Original: Note that as long as λ (cid:48) 0 is in the range (cid:2) 2 − 2 T HR ν , 2 2 T (cid:15)R − 2 (cid:3) , the double logarithmic term in our bound on linear system solution number is O ( T ) .
Revised: Note that as long as λ (cid:48) 0 is in the range (cid:2) 2 − 2 T HR ν , 2 2 T (cid:15)R − 2 (cid:3) , the double logarithmic term in our bound on linear system solution number is O ( T ) .

Original: Therefore, the overall bound is O ( T ) for an extremely large range of λ (cid:48) 0 values.
Revised: Therefore, the overall bound is O ( T ) for an extremely large range of λ (cid:48) 0 values.

Original: 3.3 First-order implementation via MinRes/Conjugate Residuals
Revised: 3.3 First-order implementation via MinRes/Conjugate Residuals

Original: We now present a ﬁrst-order implementation of our adaptive oracle, O aMSN-fo (Algorithm 3), which replaces exact linear system solutions with approximations obtained via Hessian-vector products and the MinRes/Conjugate Residuals method [43, 19].
Revised: We now present a ﬁrst-order implementation of our adaptive oracle, O aMSN-fo (Algorithm 3), which replaces exact linear system solutions with approximations obtained via Hessian-vector products and the MinRes/Conjugate Residuals method [42, 18].

Original: Similar to Algorithm 2, the algorithm searches for λ such that x λ ≈ y − [ ∇ 2 f ( y ) + λI ] − 1 ∇ f ( y ) satisﬁes the MS condition, but x λ/ 2 does not.
Revised: Similar to Algorithm 2, the algorithm searches for λ such that x λ ≈ y − [ ∇ 2 f ( y ) + λI ] − 1 ∇ f ( y ) satisﬁes the MS condition, but x λ/ 2 does not.

Original: Departing from the double-exponential scheme of Algorithm 2, here we adopt the following doubling scheme that allows us to control the cost of the x λ approximation.
Revised: Departing from the double-exponential scheme of Algorithm 2, here we adopt the following doubling scheme that allows us to control the cost of the x λ approximation.

Original: If λ (cid:48) is such that x λ (cid:48) does not satisfy the MS condition, we repeatedly test λ = 2 λ (cid:48) , 4 λ (cid:48) , 8 λ (cid:48) , . . .
Revised: If λ (cid:48) is such that x λ (cid:48) does not satisfy the MS condition, we repeatedly test λ = 2 λ (cid:48) , 4 λ (cid:48) , 8 λ (cid:48) , . . .

Original: and return the ﬁrst one for which x λ satisﬁes the MS condition.
Revised: and return the ﬁrst one for which x λ satisﬁes the MS condition.

Original: If x λ (cid:48) satisﬁes the MS condition and the algorithm is LAZY , we immediately return it.
Revised: If x λ (cid:48) satisﬁes the MS condition and the algorithm is LAZY , we immediately return it.

Original: Otherwise, we repeatedly test λ = 12 λ (cid:48) , 14 λ (cid:48) , 18 λ (cid:48) , . . .
Revised: Otherwise, we repeatedly test λ = 12 λ (cid:48) , 14 λ (cid:48) , 18 λ (cid:48) , . . .

Original: until reaching λ for which x λ does not satisfy the MS condition, and return x 2 λ .
Revised: until reaching λ for which x λ does not satisfy the MS condition, and return x 2 λ .

Original: The subroutine C ONJ R ES of Algorithm 3 takes as input a matrix A , a vector b , and accuracy parameter λσ , and iteratively generates { w i } that approximate A − 1 b .
Revised: The subroutine C ONJ R ES of Algorithm 3 takes as input a matrix A , a vector b , and accuracy parameter λσ , and iteratively generates { w i } that approximate A − 1 b .

Original: The construction of the MinRes/Conjugate
Revised: The construction of the MinRes/Conjugate

Original: Residuals method guarnatees that w i minimizes the norm of the residual r
Revised: Residuals method guarnatees that w i minimizes the norm of the residual r

Original: i
Revised: i

Original: =
Revised: =

Original: Aw i − b in the Krylov subspace span { b, Ab, . . .
Revised: Aw i − b in the Krylov subspace span { b, Ab, . . .

Original: A i − 1 b } .
Revised: A i − 1 b } .

Original: The key algorithmic decision here is when to stop the iterations: stop too early, and the approximation for the Newton step might not be accurate enough to guarantee a movement bound; stop too late, and incur a high Hessian-vector product complexity.
Revised: The key algorithmic decision here is when to stop the iterations: stop too early, and the approximation for the Newton step might not be accurate enough to guarantee a movement bound; stop too late, and incur a high Hessian-vector product complexity.

Original: We introduce a simple stopping condition (line 17) that strikes a balance.
Revised: We introduce a simple stopping condition (line 17) that strikes a balance.

Original: On the one hand, we show that whenever the condition (cid:107) r i (cid:107) ≤ λσ 2 (cid:107) w i (cid:107) holds, the resulting point x can certify roughly the same movement bounds as exact Newton steps.
Revised: On the one hand, we show that whenever the condition (cid:107) r i (cid:107) ≤ λσ 2 (cid:107) w i (cid:107) holds, the resulting point x can certify roughly the same movement bounds as exact Newton steps.

Original: On the other hand, by invoking the complexity bounds in [29] and using the the optimality of (cid:107) r i (cid:107) , we guarantee that the stopping condition is met within a number of iterations proportional to 1 / √ λ .
Revised: On the other hand, by invoking the complexity bounds in [28] and using the the optimality of (cid:107) r i (cid:107) , we guarantee that the stopping condition is met within a number of iterations proportional to 1 / √ λ .

Original: The structure of our doubling scheme for λ then allows us to relate the overall ﬁrst-order complexity to the lowest value of λ queried, obtaining the following guarantees.
Revised: The structure of our doubling scheme for λ then allows us to relate the overall ﬁrst-order complexity to the lowest value of λ queried, obtaining the following guarantees.

Original: See proofs in Appendix D.3.
Revised: See proofs in Appendix D.3.

Original: Theorem 4.
Revised: Theorem 4.

Original: Algorithm 3 with parameter σ is a σ -MS oracle O aMSN-fo .
Revised: Algorithm 3 with parameter σ is a σ -MS oracle O aMSN-fo .

Original: For any y ∈ R d , computing ( x, λ ) =
Revised: For any y ∈ R d , computing ( x, λ ) =

Original: O aMSN-fo ( y ) requires at most O (cid:16)(cid:113) 

Revised: O aMSN-fo ( y ) requires at most O (cid:16)(cid:113)

Original: + (cid:107)∇ 2 f ( y ) (cid:107) op σ min { λ,λ (cid:48) } (cid:17) Hessian-vector product and
Revised: + (cid:107)∇ 2 f ( y ) (cid:107) op σ min { λ,λ (cid:48) } (cid:17) Hessian-vector product and

Original: O ( (cid:12)(cid:12) log λλ (cid:48) (cid:12)(cid:12) )
Revised: O ( (cid:12)(cid:12) log λλ (cid:48) (cid:12)(cid:12) )

Original: gradient computations.
Revised: gradient computations.

Original: If LAZY is False or λ > λ (cid:48) , and if ∇ 2 f is ( H, ν ) -Hölder, then ( x, y, λ ) satisfy a (cid:0) 1 + ν, (6 H/σ ) 1 / (1+ ν ) (cid:1) -movement bound.
Revised: If LAZY is False or λ > λ (cid:48) , and if ∇ 2 f is ( H, ν ) -Hölder, then ( x, y, λ ) satisfy a (cid:0) 1 + ν, (6 H/σ ) 1 / (1+ ν ) (cid:1) -movement bound.

Original: Corollary 5.
Revised: Corollary 5.

Original: Consider Algorithm 1 with initial point x 0 , parameters α satisfying 1 .
Revised: Consider Algorithm 1 with initial point x 0 , parameters α satisfying 1 .

Original: 1 ≤ α = O (1) and λ (cid:48) 0 , and σ -MS oracle O aMSN-fo with LAZY set to True in all but the ﬁrst iteration and σ ∈ (0 . 01 , 0 . 99) .
Revised: 1 ≤ α = O (1) and λ (cid:48) 0 , and σ -MS oracle O aMSN-fo with LAZY set to True in all but the ﬁrst iteration and σ ∈ (0 . 01 , 0 . 99) .

Original: For any L, H, (cid:15) > 0 , ν ∈ [0 , 1] and any x (cid:63) ∈ R d , if f is convex with ( H, ν ) -Hölder Hessian and L -Lipschitz gradient, the algorithm produces an iterate x T such that f ( x T ) ≤ f ( x (cid:63) )
Revised: For any L, H, (cid:15) > 0 , ν ∈ [0 , 1] and any x (cid:63) ∈ R d , if f is convex with ( H, ν ) -Hölder Hessian and L -Lipschitz gradient, the algorithm produces an iterate x T such that f ( x T ) ≤ f ( x (cid:63) )

Original: + (cid:15) within T
Revised: + (cid:15) within T

Original: = O (cid:18)(cid:16)
Revised: = O (cid:18)(cid:16)

Original: H (cid:107) x 0 − x (cid:63) (cid:107) 

Revised: H (cid:107) x 0 − x (cid:63) (cid:107)

Original: + ν (cid:15) (cid:17) 2 / (4+3 ν ) (cid:19) iterations and at most
Revised: + ν (cid:15) [Table] iterations and at most

Original: O
Revised: O

Original: [Table]
Revised: [Table]

Original: [Equation] [Table] [Equation]
Revised: +

Original: [Table]
Revised: [Table]

Revised: gradient and Hessian-vector product evaluations.

Original: Note that the L -Lipschitz gradient assumption implies an ( L, 0)
Revised: Note that the L -Lipschitz gradient assumption implies an ( L, 0)

Original: -Hölder
Revised: -Hölder

Original: Hessian assumption, giving the iteration complexity bound we state in Table 1.
Revised: Hessian assumption, giving the iteration complexity bound we state in Table 1.

Original: Moreover, note that our algorithm has the optimal O ( (cid:112) L (cid:107) x 0 − x (cid:63) (cid:107) 2 /(cid:15) ) complexity for any λ (cid:48) 0 in the range Ω( (cid:15)/ (cid:107) x 0 − x (cid:63) (cid:107) 2 ) to
Revised: Moreover, note that our algorithm has the optimal O ( (cid:112) L (cid:107) x 0 − x (cid:63) (cid:107) 2 /(cid:15) ) complexity for any λ (cid:48) 0 in the range Ω( (cid:15)/ (cid:107) x 0 − x (cid:63) (cid:107) 2 ) to

Original: L exp (cid:110) O
Revised: L exp (cid:110) O

Original: ( (cid:112) L (cid:107) x 0 − x (cid:63) (cid:107) 2 /(cid:15) )
Revised: ( (cid:112) L (cid:107) x 0 − x (cid:63) (cid:107) 2 /(cid:15) )

Original: (cid:111) .
Revised: (cid:111) .

Original: By choosing a large λ (cid:48) 

Revised: By choosing a large λ (cid:48)

Original: (say 10 6 ) we may guarantee that only the logarithmic term is added to the optimal ﬁrst-order evaluation complexity.
Revised: (say 10 6 ) we may guarantee that only the logarithmic term is added to the optimal ﬁrst-order evaluation complexity.

Original:  Experiments
Revised:  Experiments

Original: We conduct three sets of experiments.
Revised: We conduct three sets of experiments.

Original: First, we consider O cr with a ﬁxed parameter M and compare previous acceleration schemes to Algorithm 1.
Revised: First, we consider O cr with a ﬁxed parameter M and compare previous acceleration schemes to Algorithm 1.

Original: Second, we combine Algorithm 1 with our adaptive
Revised: Second, we combine Algorithm 1 with our adaptive

Original: O aMSN and test it against previous adaptive accelerated (second-order) methods and Newton’s method.
Revised: O aMSN and test it against previous adaptive accelerated (second-order) methods and Newton’s method.

Original: Finally, we compare Algorithm 1 with our ﬁrst-order adaptive oracle O aMSN-fo to other ﬁrst-order methods.
Revised: Finally, we compare Algorithm 1 with our ﬁrst-order adaptive oracle O aMSN-fo to other ﬁrst-order methods.

Original: We provide full implementation details in Appendix E.1.
Revised: We provide full implementation details in Appendix E.1.

Original: Figure 1 summarizes our results for logistic regression on the ‘a9a’ dataset [15]; see Appendix E.2 for similar results on three additional datasets.
Revised: Figure 1 summarizes our results for logistic regression on the ‘a9a’ dataset [15]; see Appendix E.2 for similar results on three additional datasets.

Original: These experiments were conducted with no tuning of Algorithm 1: the parameters σ and α were simply set to 12 and 2 , respectively.
Revised: These experiments were conducted with no tuning of Algorithm 1: the parameters σ and α were simply set to 12 and 2 , respectively.

Original: An additional experiment, reported in Appendix E.3, shows that the algorithm is indeed insensitive to that choice.
Revised: An additional experiment, reported in Appendix E.3, shows that the algorithm is indeed insensitive to that choice.

Original: Non-adaptive methods.
Revised: Non-adaptive methods.

Original: We use the non-adaptive oracle O cr (1), and take M to be 0 .
Revised: We use the non-adaptive oracle O cr (1), and take M to be 0 .

Original: 2 ¯ H where, for feature vectors φ 1 , . .
Revised: 2 ¯ H where, for feature vectors φ 1 , . .

Original: . , φ n , ¯ H
Revised: . , φ n , ¯ H

Original: = (cid:107) 1 n (cid:80) ni
Revised: = (cid:107) 1 n (cid:80) ni

Original: =1 φ i φ Ti (cid:107) op max i ∈ [ n ] (cid:107) φ i (cid:107) is an upper bound on 6 √ 3 ≈times the Lipschitz constant of the logistic regression Hessian [see, e.g., 42].
Revised: =1 φ i φ Ti (cid:107) op max i ∈ [ n ] (cid:107) φ i (cid:107) is an upper bound on 6 √ 3 ≈times the Lipschitz constant of the logistic regression Hessian [see, e.g., 41].

Original: Fixing the MS oracleallows for a controlled comparison of different acceleration schemes: Figure 1(a) shows that standard
Revised: Fixing the MS oracle

Original: [Figure]
Revised: [Figure]

Revised: allows for a controlled comparison of different acceleration schemes: Figure 1(a) shows that standard

Original: MS acceleration with a carefully-implemented bisection outperforms standard cubic regularization (CR) and its accelerated counterpart (ACR) [34, Alg. 4.8], and removing the bisection via Algorithmyields the best results.
Revised: MS acceleration with a carefully-implemented bisection outperforms standard cubic regularization (CR) and its accelerated counterpart (ACR) [33, Alg. 4.8] [Figure] and removing the bisection via Algorithmyields the best results.

Original: We also implemented the heuristic suggested by Song et al.
Revised: We also implemented the heuristic suggested by Song et al.

Original: [42], where instead of a bisection in Algorithm 

Revised: [Figure] where instead of a bisection in Algorithm

Original: we select a sequence λ (cid:48)
Revised: we select a sequence λ (cid:48)

Original: t such that A t =
Revised: t such that A t =

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: In Appendix E.4 we tune the M parameter for each method separately, ﬁnding that the optimal M for CR is near 0 , so that O cr is nearly a Newton step (and not a valid MS oracle)
Revised: [Equation] [Figure] [Equation] [Figure]

Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Revised: In Appendix E.4 we tune the M parameter for each method separately, ﬁnding that the optimal M for CR is near 0 , so that O cr is nearly a Newton step (and not a valid MS oracle).

Original: Adaptive methods and Newton’s method.
Revised: Adaptive methods and Newton’s method.

Original: We compare the following adaptive accelerated secondorder methods (which do not require an estimate of the Hessian Lipschitz constant): Adaptive
Revised: We compare the following adaptive accelerated secondorder methods (which do not require an estimate of the Hessian Lipschitz constant): Adaptive

Original: ACR [22, Algorithm 4] (which adaptively sets M in O cr ), standard MS acceleration (Algorithm 0) with O aMSN (Algorithm 2, with LAZY = False) and Algorithm 1 with O aMSN (with LAZY =
Revised: ACR [21, Algorithm 4] (which adaptively sets M in O cr ), standard MS acceleration (Algorithm 0) with O aMSN (Algorithm 2, with LAZY = False) and Algorithm 1 with O aMSN (with LAZY =

Original: True in all but the ﬁrst iteration).
Revised: True in all but the ﬁrst iteration).

Original: Figure 1(b) shows that the latter converges signiﬁcantly faster than the other adaptive acceleration schemes.
Revised: Figure 1(b) shows that the latter converges signiﬁcantly faster than the other adaptive acceleration schemes.

Original: However, the classical “unaccelerated” Newton iteration x t +1 = − ( ∇ 2 f ( x t )) − 1 ∇ f ( x t ) strongly outperforms all “accelerated” methods, indicating that momentum mechanisms might actually be slowing down convergence in logistic regression problems.
Revised: However, the classical “unaccelerated” Newton iteration x t +1 = − ( ∇ 2 f ( x t )) − 1 ∇ f ( x t ) strongly outperforms all “accelerated” methods, indicating that momentum mechanisms might actually be slowing down convergence in logistic regression problems.

Original: To test this, we consider the following simple iteration of (the non-lazy variant) of our oracle: x t +1 , λ t
Revised: To test this, we consider the following simple iteration of (the non-lazy variant) of our oracle: x t +1 , λ t

Original: +1 =
Revised: +1 =

Original: O aMSN ( x t ; λ t / 2) ; it signiﬁcantly improves over Algorithm 1.
Revised: O aMSN ( x t ; λ t / 2) ; it signiﬁcantly improves over Algorithm 1.

Original: These results beg the question: is momentum ever useful for second-order methods?
Revised: These results beg the question: is momentum ever useful for second-order methods?

Original: In Appendix E. we test different schemes on the lower bound construction [3, 22].
Revised: In Appendix E. we test different schemes on the lower bound construction [3, 21].

Original: We ﬁnd momentum is helpful for
Revised: We ﬁnd momentum is helpful for

Original: O cr , but not for the adaptive oracle O aMSN .
Revised: O cr , but not for the adaptive oracle O aMSN .

Original: What makes Newton’s method perform so well on logistic regression, and whether simply iterating O aMSN is worst-case optimal, are important questions for future work.
Revised: What makes Newton’s method perform so well on logistic regression, and whether simply iterating O aMSN is worst-case optimal, are important questions for future work.

Original: First-order methods.
Revised: First-order methods.

Original: We compare our ﬁrst-order adaptive O aMSN-fo (Algorithm 3) to the following baselines: gradient descent and accelerated gradient descent [39] with a tuned step size η , and LBFGS-B from SciPy [10, 45, 44].
Revised: We compare our ﬁrst-order adaptive O aMSN-fo (Algorithm 3) to the following baselines: gradient descent and accelerated gradient descent [38] with a tuned step size η , and LBFGS-B from SciPy [10, 44, 43].

Original: In light of the above comparison with Newton’s method, we also test the following simple iteration of (the lazy variant) of our oracle: x t +1 , λ t
Revised: In light of the above comparison with Newton’s method, we also test the following simple iteration of (the lazy variant) of our oracle: x t +1 , λ t

Original: +

Revised: +

Original: =
Revised: =

Original: O aMSN-fo ( x t ; λ t / 2) .
Revised: O aMSN-fo ( x t ; λ t / 2) .

Original: Figure 1(c) shows that forgoing (second-order) momentum is better for the ﬁrst-order oracle, too:
Revised: Figure 1(c) shows that forgoing (second-order) momentum is better for the ﬁrst-order oracle, too:

Original: Algorithm 1 performs comparably to tuned AGD (without tuning a single parameter), and the equally adaptive O aMSN-fo iteration performs comparably to with L-BFGS-B.
Revised: Algorithm 1 performs comparably to tuned AGD (without tuning a single parameter), and the equally adaptive O aMSN-fo iteration performs comparably to with L-BFGS-B.

Original: Acknowledgments
Revised: Acknowledgments

Original: We thank the anonymous reviewers for helpful questions and suggestions, leading to important writing clariﬁcations, a simpliﬁcation of the proof of Lemma 3, and an improved complexity bound for second-order methods under third-order smoothness (due to the observation that our framework is compatible with the oracle of [37]).
Revised: We thank the anonymous reviewers for helpful questions and suggestions, leading to important writing clariﬁcations, a simpliﬁcation of the proof of Lemma 3, and an improved complexity bound for second-order methods under third-order smoothness (due to the observation that our framework is compatible with the oracle of [36]).

Original: YC and DH were supported in part by the Israeli Science
Revised: YC and DH were supported in part by the Israeli Science

Original: Foundation (ISF) grant no. 2486/21 and the Len Blavatnik and the Blavatnik Family foundation.
Revised: Foundation (ISF) grant no. 2486/21 and the Len Blavatnik and the Blavatnik Family foundation.

Original: YJ was supported in part by a Stanford Graduate Fellowship and the Dantzig-Lieberman Fellowship.
Revised: YJ was supported in part by a Stanford Graduate Fellowship and the Dantzig-Lieberman Fellowship.

Original: AS was supported in part by a Microsoft Research Faculty Fellowship, NSF CAREER Award
Revised: AS was supported in part by a Microsoft Research Faculty Fellowship, NSF CAREER Award

Original: CCF-1844855, NSF Grant CCF-1955039, a PayPal research award, and a Sloan Research Fellowship.
Revised: CCF-1844855, NSF Grant CCF-1955039, a PayPal research award, and a Sloan Research Fellowship.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] see supplementary material 3.
Revised: [Yes] see supplementary material 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] in the supplementary material (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] in the supplementary material (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] full implementation details in Appendix E. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] full implementation details in Appendix E. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] We only experimented with deterministic algorithms (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [No] We only experimented with deterministic algorithms (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [No] We only ran small scale experiments on CPU 4.
Revised: [No] We only ran small scale experiments on CPU 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] see Appendix E. (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] see Appendix E. (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] in the supplemental material (d) Did you discuss whether
Revised: [Yes] in the supplemental material (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Additional related work
Revised: A Additional related work

Original: Bisection-free methods for variational inequalities.
Revised: Bisection-free methods for variational inequalities.

Original: Monteiro and Svaiter also proposed secondorder methods for solving variational inequalities for monotone operators with continuous derivatives [32], and subsequent work provided improved rates for variational inequalities with continuous higher-order derivatives via tensor methods [9, 26].
Revised: Monteiro and Svaiter also proposed secondorder methods for solving variational inequalities for monotone operators with continuous derivatives [31], and subsequent work provided improved rates for variational inequalities with continuous higher-order derivatives via tensor methods [9, 25].

Original: These works also feature an implicit equation over a scalar regularization/step-size parameter, that necessitates a bisection and increases complexity by a logarithmic factor.
Revised: These works also feature an implicit equation over a scalar regularization/step-size parameter, that necessitates a bisection and increases complexity by a logarithmic factor.

Original: In recent papers, Lin and Jordan [30] and Adil et al.
Revised: In recent papers, Lin and Jordan [29] and Adil et al.

Original: [1] remove that logarithmic factor by developing bisection-free methods for variational inequalities.
Revised: [1] remove that logarithmic factor by developing bisection-free methods for variational inequalities.

Original: However, applying these methods directly to convex optimization with Lipschitz p th derivatives yields a rate of O ( t − ( p +1) / 2 ) rather than the optimal O ( t − (3 p +1) / 2 ) rate of our method.
Revised: However, applying these methods directly to convex optimization with Lipschitz p th derivatives yields a rate of O ( t − ( p +1) / 2 ) rather than the optimal O ( t − (3 p +1) / 2 ) rate of our method.

Original: Moreover, these works remove bisections using techniques fundamentally different from ours.
Revised: Moreover, these works remove bisections using techniques fundamentally different from ours.

Original: In particular, they do not apply a damping scheme on the A t sequence, nor do they apply a multiplicative update for the regularization parameter.
Revised: In particular, they do not apply a damping scheme on the A t sequence, nor do they apply a multiplicative update for the regularization parameter.

Original: Adaptive Newton and tensor methods.
Revised: Adaptive Newton and tensor methods.

Original: A number of works consider adaptive variants of the cubicregularized Newton method and its tensor counterparts.
Revised: A number of works consider adaptive variants of the cubicregularized Newton method and its tensor counterparts.

Original: Cartis et al.
Revised: Cartis et al.

Original: [14], Gould et al.
Revised: [14], Gould et al.

Original: [21] propose adaptive variants of cubic regularization for non-convex optimization.
Revised: [20] propose adaptive variants of cubic regularization for non-convex optimization.

Original: For convex optimization,
Revised: For convex optimization,

Original: Mishchenko [31] provides a simple adaptive scheme converging at rate O ( t − 2 ) , followed by an improvement in its guarantee to O ( t − 3 )
Revised: Mishchenko [30] provides a simple adaptive scheme converging at rate O ( t − 2 ) , followed by an improvement in its guarantee to O ( t − 3 )

Original: For tensor methods, Jiang et al.
Revised: For tensor methods, Jiang et al.

Original: [25] proposes an adaptive regularization scheme for convex functions with Lipschitz-continuous p th derivatives which achieves the rate O ( t − p − 1 ) .
Revised: [24] proposes an adaptive regularization scheme for convex functions with Lipschitz-continuous p th derivatives which achieves the rate O ( t − p − 1 ) .

Original: In addition, Grapiglia and Nesterov [22] gives analogous results under ν -Hölder continuity of the derivatives.
Revised: In addition, Grapiglia and Nesterov [21] gives analogous results under ν -Hölder continuity of the derivatives.

Original: Even for the case second-order methods with Lipschitz-continuous Hessian, an adaptive scheme with optimal rate O ( t − 3 . 5 ) remained open prior to this work.
Revised: Even for the case second-order methods with Lipschitz-continuous Hessian, an adaptive scheme with optimal rate O ( t − 3 . 5 ) remained open prior to this work.

Original: Beyond removing the bisection from the MS framework, our key algorithmic techniques include directly considering quadraticallyregularized Newton step (similar to [31, 17] and different from [22, 25]), and using the original MS approximation condition for selecting an appropriate regularization parameter, which is new in the context of adaptive methods.
Revised: Beyond removing the bisection from the MS framework, our key algorithmic techniques include directly considering quadraticallyregularized Newton step (similar to [30, 17] and different from [21, 24]), and using the original MS approximation condition for selecting an appropriate regularization parameter, which is new in the context of adaptive methods.

Original: These techniques allow us to adapt to both the constant and order of
Revised: These techniques allow us to adapt to both the constant and order of

Original: Hessian Hölder continuity simultaneously.
Revised: Hessian Hölder continuity simultaneously.

Original: Comparison to [28].
Revised: Comparison to [27].

Original: In concurrent and independent theoretical work, Kovalev and Gasnikov [28] propose an algorithm that also attains the optimal p th derivative evaluation complexity for convex optimization with Lipschitz p th derivatives.
Revised: In concurrent and independent theoretical work, Kovalev and Gasnikov [27] propose an algorithm that also attains the optimal p th derivative evaluation complexity for convex optimization with Lipschitz p th derivatives.

Original: While also inspired by MS acceleration, the algorithm of [28] is quite different from ours: they replace the implicitly deﬁned regularization parameters inherent to MS oracles by approximating proximal points with explicit, predetermined regularization parameters.
Revised: While also inspired by MS acceleration, the algorithm of [27] is quite different from ours: they replace the implicitly deﬁned regularization parameters inherent to MS oracles by approximating proximal points with explicit, predetermined regularization parameters.

Original: To obtain these proximal points they apply a tensor-extragradient method, and stop it when the MS condition is met.
Revised: To obtain these proximal points they apply a tensor-extragradient method, and stop it when the MS condition is met.

Original: By careful analysis, they show that—even though an individual outer acceleration step requires multiple derivative evaluations—the overall complexity of their method is optimal.
Revised: By careful analysis, they show that—even though an individual outer acceleration step requires multiple derivative evaluations—the overall complexity of their method is optimal.

Original: In contrast, our method makes a single oracle call (high-order derivative evaluation) per step.
Revised: In contrast, our method makes a single oracle call (high-order derivative evaluation) per step.

Original: To obtain optimal complexity, our method relies on a dynamic sequence of “guessed” regularization parameters and a momentum damping schemes that handles cases where these guesses overshoot.
Revised: To obtain optimal complexity, our method relies on a dynamic sequence of “guessed” regularization parameters and a momentum damping schemes that handles cases where these guesses overshoot.

Original: The two works offer complementary viewpoints of the algorithmic innovation required for removing bisection from the MS framework.
Revised: The two works offer complementary viewpoints of the algorithmic innovation required for removing bisection from the MS framework.

Original: In comparison to Kovalev and Gasnikov [28], we believe that our algorithm offers advantages in terms of generality and adaptivity.
Revised: In comparison to Kovalev and Gasnikov [27], we believe that our algorithm offers advantages in terms of generality and adaptivity.

Original: From a generality perspective, our algorithm applies to every setting where the original MS framework applies.
Revised: From a generality perspective, our algorithm applies to every setting where the original MS framework applies.

Original: In addition to functions with Lipschitz p th derivatives, that includes functions with Hölder derivatives [42], ball minimization oracles [12], and a second-order oracles for functions with Lipschitz third derivative [37, 27].
Revised: In addition to functions with Lipschitz p th derivatives, that includes functions with Hölder derivatives [41], ball minimization oracles [12], and a second-order oracles for functions with Lipschitz third derivative [36, 26].

Original: While extending [28] to these settings may be possible, it would require additional work in formulating and analyzing an appropriate subproblem solver.
Revised: While extending [27] to these settings may be possible, it would require additional work in formulating and analyzing an appropriate subproblem solver.

Original: Regarding adaptivity, like the original MS framework, our algorithm is agnostic to both the order of the Lipschitz derivative and its corresponding Lipschitz constant.
Revised: Regarding adaptivity, like the original MS framework, our algorithm is agnostic to both the order of the Lipschitz derivative and its corresponding Lipschitz constant.

Original: In contrast,
Revised: In contrast,

Original: Kovalev and Gasnikov [28] require the derivative order for determining the regularization parameters, and the Lipschitz constant for the subproblem solver.
Revised: Kovalev and Gasnikov [27] require the derivative order for determining the regularization parameters, and the Lipschitz constant for the subproblem solver.

Original: B Proof of Theorem
Revised: B Proof of Theorem

Original: In this section, we provide a complete proof for Theorem 1.
Revised: In this section, we provide a complete proof for Theorem 1.

Original: We begin in Appendix B.1 with establishing a potential decrease result analogous to the standard analyses of accelerated proximal methods (Proposition 1).
Revised: We begin in Appendix B.1 with establishing a potential decrease result analogous to the standard analyses of accelerated proximal methods (Proposition 1).

Original: In Appendix B.2 we then provide a lower bound of A T in terms of the values of λ (cid:48) t at a subset of “up” iterations where λ (cid:48) t > λ t (Lemma 1).
Revised: In Appendix B.2 we then provide a lower bound of A T in terms of the values of λ (cid:48) t at a subset of “up” iterations where λ (cid:48) t > λ t (Lemma 1).

Original: We apply these results along with the reverse Hölder inequality to obtain Theorem 1 in Appendix B.3.
Revised: We apply these results along with the reverse Hölder inequality to obtain Theorem 1 in Appendix B.3.

Original: The last part of the analysis relies on two technical lemmas on the growth rates of sequences satisfying certain recursive inequalities (Lemmas 3 and 4), which we prove at the end of the section in Appendix B.4.
Revised: The last part of the analysis relies on two technical lemmas on the growth rates of sequences satisfying certain recursive inequalities (Lemmas 3 and 4), which we prove at the end of the section in Appendix B.4.

Original: Beyond its utility for proving Theorem 1, Lemma 1 includes additional lower bounds on A T in terms of λ (cid:48) t which we use in the analysis of adaptive oracle implementations in Appendix D.
Revised: Beyond its utility for proving Theorem 1, Lemma 1 includes additional lower bounds on A T in terms of λ (cid:48) t which we use in the analysis of adaptive oracle implementations in Appendix D.

Original: We use the following notation for the set of “down” iterations:
Revised: We use the following notation for the set of “down” iterations:

Original: [Equation]
Revised: [Equation]

Original: and analogously deﬁne S >T (“up” iterations), S ≥ T , S <T and S = T .
Revised: and analogously deﬁne S >T (“up” iterations), S ≥ T , S <T and S = T .

Original: B.1 Potential decrease
Revised: B.1 Potential decrease

Original: Proposition 1.
Revised: Proposition 1.

Original: Under the assumptions of Theorem 1, let E t := f ( x t ) − f ( x (cid:63) ) , D t := 12 (cid:107) v t − x (cid:63) (cid:107) 2 , and N t +1 :
Revised: Under the assumptions of Theorem 1, let E t := f ( x t ) − f ( x (cid:63) ) , D t := 12 (cid:107) v t − x (cid:63) (cid:107) 2 , and N t +1 :

Original: = 12 (cid:107) ˜ x t +1 − y t (cid:107) 2 for all t ≥ 0 .
Revised: = 12 (cid:107) ˜ x t +1 − y t (cid:107) 2 for all t ≥ 0 .

Original: Then, for all t ≥
Revised: Then, for all t ≥

Original: [Equation]
Revised: [Equation]

Original: Consequently, for all T ≥ 1 , √ A T ≥ 12 (cid:80) t ∈S ≤ T 1 / (cid:112) λ (cid:48) t ,
Revised: Consequently, for all T ≥ 1 , √ A T ≥ 12 (cid:80) t ∈S ≤ T 1 / (cid:112) λ (cid:48) t ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: By deﬁnition of D t and the deﬁnition of v t +1 in line 17, we have
Revised: By deﬁnition of D t and the deﬁnition of v t +1 in line 17, we have

Original: [Equation]
Revised: [Equation]

Original: Also, by the deﬁnition y t in line 6 and A (cid:48)
Revised: Also, by the deﬁnition y t in line 6 and A (cid:48)

Original: t +1 :
Revised: t +1 :

Original: =
Revised: =

Original: A t + a (cid:48) t +1 in line 5, we have
Revised: A t + a (cid:48) t +1 in line 5, we have

Original: [Equation]
Revised: [Equation]

Original: Subtracting a (cid:48)
Revised: Subtracting a (cid:48)

Original: t +1 x (cid:63) from both sides and considering the inner product with ∇ f (˜ x t +1 ) then yields a (cid:48) t +
Revised: t +1 x (cid:63) from both sides and considering the inner product with ∇ f (˜ x t +1 ) then yields a (cid:48) t +

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: ˜
Revised: ˜

Original: [Equation]
Revised: [Equation]

Original: , x
Revised: , x

Original: [Equation]
Revised: [Equation]

Original: ( i )
Revised: ( i )

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: where we used ( i ) convexity of f and ( ii ) again that A (cid:48) t +

Revised: where we used ( i ) convexity of f and ( ii ) again that A (cid:48) t +

Original: =
Revised: =

Original: A t + a (cid:48) t +1 (line 5).
Revised: A t + a (cid:48) t +1 (line 5).

Original: Next, note that we can upper bound (cid:104)∇ f (˜ x t +1 ) , x (cid:63) − v t (cid:105) as
Revised: Next, note that we can upper bound (cid:104)∇ f (˜ x t +1 ) , x (cid:63) − v t (cid:105) as

Original: [Equation]
Revised: [Equation]

Original: where for the inequality we used that O is a σ -MS oracle for f (deﬁnition 1) and the deﬁnition of N t .
Revised: where for the inequality we used that O is a σ -MS oracle for f (deﬁnition 1) and the deﬁnition of N t .

Original: Substituting back gives
Revised: Substituting back gives

Original: [Equation]
Revised: [Equation]

Original: We separately consider the cases λ t +1 ≤ λ (cid:48) t +1 and λ t +1 > λ (cid:48)
Revised: We separately consider the cases λ t +1 ≤ λ (cid:48) t +1 and λ t +1 > λ (cid:48)

Original: t +1 .
Revised: t +1 .

Original: First, when λ t +1 ≤ λ (cid:48) t +1 , by deﬁnition in the algorithm x t +1 =
Revised: First, when λ t +1 ≤ λ (cid:48) t +1 , by deﬁnition in the algorithm x t +1 =

Original: ˜ x t +1 , a t +1 = a (cid:48)
Revised: ˜ x t +1 , a t +1 = a (cid:48)

Original: t +1 , A t +1 =
Revised: t +1 , A t +1 =

Original: A (cid:48) t +1 and by line 4 we have
Revised: A (cid:48) t +1 and by line 4 we have

Original: A t +1 = λ (cid:48) t +1 a 2 t +1 .
Revised: A t +1 = λ (cid:48) t +1 a 2 t +1 .

Original: Consequently, we can combine (10) and (11) to conclude that
Revised: Consequently, we can combine (10) and (11) to conclude that

Original: [Equation]
Revised: [Equation]

Original: On the other hand, when λ t +1 > λ (cid:48) t +1 , by the deﬁnition of γ t +1 = λ (cid:48)
Revised: On the other hand, when λ t +1 > λ (cid:48) t +1 , by the deﬁnition of γ t +1 = λ (cid:48)

Original: t +1 /λ t +1 in line 13, a t +1 , A t + in line 14, and x t +1 in line 15, we have A t +

Revised: t +1 /λ t +1 in line 13, a t +1 , A t + in line 14, and x t +1 in line 15, we have A t +

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: 1 − γ
Revised: 1 − γ

Original: [Equation]
Revised: [Equation]

Original: +1 )
Revised: +1 )

Original: A t + γ t
Revised: A t + γ t

Original: +

Revised: +

Original: A
Revised: A

Original: (cid:48) t +1 , and therefore convexity of f implies that
Revised: (cid:48) t +1 , and therefore convexity of f implies that

Original: [Equation]
Revised: [Equation]

Original: Subtracting f
Revised: Subtracting f

Original: ( x (cid:63) ) , multiplying by A t
Revised: ( x (cid:63) ) , multiplying by A t

Original: +1 , combining with (11) to bound f (˜ x t +1 ) and noting that γ t +1 a (cid:48) t +1 =
Revised: +1 , combining with (11) to bound f (˜ x t +1 ) and noting that γ t +1 a (cid:48) t +1 =

Original: a t +1 yields
Revised: a t +1 yields

Original: [Equation]
Revised: [Equation]

Original: Noting that A (cid:48) t +

Revised: Noting that A (cid:48) t +

Original: = λ (cid:48)
Revised: = λ (cid:48)

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: +1 ) 2 = λ
Revised: +1 ) 2 = λ

Original: [Equation]
Revised: [Equation]

Original: 1 γ
Revised: 1 γ

Original: [Equation]
Revised: [Equation]

Original: +1 a 2 t +1 by deﬁnition and further substituting (10) into the above display yields
Revised: +1 a 2 t +1 by deﬁnition and further substituting (10) into the above display yields

Original: [Equation]
Revised: [Equation]

Original: which, when combined with (12) yields (8).
Revised: which, when combined with (12) yields (8).

Original: The bound on A T follows from standard argument for Monteiro-Svaiter acceleration restricting to the proper set, i.e. S ≤ T , see e.g. Lemma 27 in [12], we include here for completeness.
Revised: The bound on A T follows from standard argument for Monteiro-Svaiter acceleration restricting to the proper set, i.e. S ≤ T , see e.g. Lemma 27 in [12], we include here for completeness.

Original: [Equation]
Revised: [Equation]

Original: For the second line we used that λ (cid:48) t +1 ( a (cid:48)
Revised: For the second line we used that λ (cid:48) t +1 ( a (cid:48)

Original: t +1 ) 

Revised: t +1 )

Original: = A (cid:48) t +1 and that A (cid:48) t is increasing in t .
Revised: = A (cid:48) t +1 and that A (cid:48) t is increasing in t .

Original: Finally, the conclusions in (9) follow from inductively applying (8) and using A 0 = 0 .
Revised: Finally, the conclusions in (9) follow from inductively applying (8) and using A 0 = 0 .

Original: B.2 Lower bounding A T using “up” iterates
Revised: B.2 Lower bounding A T using “up” iterates

Original: Next, we provide more ﬁne-grained bounds on the growth of A t , implied by the adaptive scheme for updating λ (cid:48) in line 11 and 16.
Revised: Next, we provide more ﬁne-grained bounds on the growth of A t , implied by the adaptive scheme for updating λ (cid:48) in line 11 and 16.

Original: Lemma 1.
Revised: Lemma 1.

Original: In the setting of Theorem 1, for any (cid:98) T ≥ 1 , there exists a non-empty set Q (cid:98) T ⊆ S > (cid:98) T ∪ { 1 } and positive numbers r t for each t ∈ Q (cid:98) T such that
Revised: In the setting of Theorem 1, for any (cid:98) T ≥ 1 , there exists a non-empty set Q (cid:98) T ⊆ S > (cid:98) T ∪ { 1 } and positive numbers r t for each t ∈ Q (cid:98) T such that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Further, the deﬁnition is consistent in the sense that for any T ≥ 1 and deﬁned Q T , for any (cid:98) T 1 , (cid:98) T 2 ∈ Q T , suppose (cid:98) T 1 < (cid:98) T 2 and r t, (cid:98) T are the numbers when applying previous argument to (cid:98) T , then Q (cid:98) T 1 ⊆ Q (cid:98) T 2 and r t, (cid:98) T 1 = r t, (cid:98) T 2 for any t ≤ (cid:98) T 1 .
Revised: Further, the deﬁnition is consistent in the sense that for any T ≥ 1 and deﬁned Q T , for any (cid:98) T 1 , (cid:98) T 2 ∈ Q T , suppose (cid:98) T 1 < (cid:98) T 2 and r t, (cid:98) T are the numbers when applying previous argument to (cid:98) T , then Q (cid:98) T 1 ⊆ Q (cid:98) T 2 and r t, (cid:98) T 1 = r t, (cid:98) T 2 for any t ≤ (cid:98) T 1 .

Original: Thus, we omit the second subscript in deﬁning r t when clear from context.
Revised: Thus, we omit the second subscript in deﬁning r t when clear from context.

Original: Furthermore, for T ≥ 1 ,
Revised: Furthermore, for T ≥ 1 ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We deﬁne Q (cid:98) T to be the set of “up-down” iterates, i.e., iterates t for which λ t > λ (cid:48) t but λ t +1 ≤ λ (cid:48)
Revised: We deﬁne Q (cid:98) T to be the set of “up-down” iterates, i.e., iterates t for which λ t > λ (cid:48) t but λ t +1 ≤ λ (cid:48)

Original: t +1 ; we also add to Q (cid:98) T the ﬁrst iterate and, if (cid:98) T ∈ S > (cid:98) T , the iterate (cid:98) T .
Revised: t +1 ; we also add to Q (cid:98) T the ﬁrst iterate and, if (cid:98) T ∈ S > (cid:98) T , the iterate (cid:98) T .

Original: Formally, we have
Revised: Formally, we have

Original: [Equation]
Revised: [Equation]

Original: We let 1 = τ 1 < τ 2 < · · · < τ S ≤ (cid:98) T denote the S
Revised: We let 1 = τ 1 < τ 2 < · · · < τ S ≤ (cid:98) T denote the S

Original: = |Q (cid:98) T | distinct elements of Q (cid:98) T in increasing order.
Revised: = |Q (cid:98) T | distinct elements of Q (cid:98) T in increasing order.

Original: For notational convenience, we also let τ S +1 :=
Revised: For notational convenience, we also let τ S +1 :=

Original: [Equation]
Revised: [Equation]

Original: T .
Revised: T .

Original: For every i ∈ [ S ] , we let n i be the index of the last “down” iterate between τ i and τ i +1 (and (cid:98) T if i = S ), that is
Revised: For every i ∈ [ S ] , we let n i be the index of the last “down” iterate between τ i and τ i +1 (and (cid:98) T if i = S ), that is

Original: [Equation]
Revised: [Equation]

Original: As an immediate consequence of the deﬁnition of τ i and n i , we have for all i < S , n i ∈
Revised: As an immediate consequence of the deﬁnition of τ i and n i , we have for all i < S , n i ∈

Original: [ τ i , τ i +1 ) .
Revised: [ τ i , τ i +1 ) .

Original: We also have that the set of n i are distinct, i.e. n i (cid:54) = n j for all i, j ∈ [ S ] with i (cid:54) =
Revised: We also have that the set of n i are distinct, i.e. n i (cid:54) = n j for all i, j ∈ [ S ] with i (cid:54) =

Original: j .
Revised: j .

Original: Note that between any two “up-down” iterates τ
Revised: Note that between any two “up-down” iterates τ

Original: i and τ i +

Revised: i and τ i +

Original: we have a sequence of “down” iterates (ending at n i ) followed by a sequence of “up iterates” (ending at τ i +1 ).
Revised: we have a sequence of “down” iterates (ending at n i ) followed by a sequence of “up iterates” (ending at τ i +1 ).

Original: In other words, for all i < S and k ∈ ( n i , τ i +1 ]
Revised: In other words, for all i < S and k ∈ ( n i , τ i +1 ]

Original: we have k ∈ S > (cid:98) T .
Revised: we have k ∈ S > (cid:98) T .

Original: Consequently, λ (cid:48)
Revised: Consequently, λ (cid:48)

Original: k +1 =
Revised: k +1 =

Original: αλ (cid:48) k for all k ∈ ( n i , τ i +1 )
Revised: αλ (cid:48) k for all k ∈ ( n i , τ i +1 )

Original: (since these are “up” iterates).
Revised: (since these are “up” iterates).

Original: Since λ (cid:48) n i +1 =
Revised: Since λ (cid:48) n i +1 =

Original: α − 1 λ (cid:48) n i (because n i is a “down” iterate), we conclude that λ (cid:48) n i = α 2 − ( τ i +1 − n i ) λ (cid:48)
Revised: α − 1 λ (cid:48) n i (because n i is a “down” iterate), we conclude that λ (cid:48) n i = α 2 − ( τ i +1 − n i ) λ (cid:48)

Original: τ i +1 .
Revised: τ i +1 .

Original: Combining this with Proposition 1 implies the following lower bound on
Revised: Combining this with Proposition 1 implies the following lower bound on

Original: [Equation]
Revised: [Equation]

Original: Further, as argued above, for k ∈ ( τ i , n i ] we have k ∈ S ≤ (cid:98) T for all and therefore λ (cid:48)
Revised: Further, as argued above, for k ∈ ( τ i , n i ] we have k ∈ S ≤ (cid:98) T for all and therefore λ (cid:48)

Original: k +1 =
Revised: k +1 =

Original: λ (cid:48)
Revised: λ (cid:48)

Original: k /α .
Revised: k /α .

Original: Consequently, when τ i < (cid:98) T we have λ (cid:48) n i = α 2 − ( n i − τ i ) λ (cid:48)
Revised: Consequently, when τ i < (cid:98) T we have λ (cid:48) n i = α 2 − ( n i − τ i ) λ (cid:48)

Original: τ i .
Revised: τ i .

Original: When τ i = (cid:98) T the inequality also holds since τ i = n i .
Revised: When τ i = (cid:98) T the inequality also holds since τ i = n i .

Original: Together with the conclusion of Proposition 1, this implies the following lower bound on (cid:112) A (cid:98) T :
Revised: Together with the conclusion of Proposition 1, this implies the following lower bound on (cid:112) A (cid:98) T :

Original: [Equation]
Revised: [Equation]

Original: We now deﬁne r i as follows
Revised: We now deﬁne r i as follows

Original: [Equation]
Revised: [Equation]

Original: Clearly we have r t ≥ 0 for all t ∈ Q (cid:98) T and (cid:80) t ∈Q (cid:98) T r t = (cid:80) i ∈
Revised: Clearly we have r t ≥ 0 for all t ∈ Q (cid:98) T and (cid:80) t ∈Q (cid:98) T r t = (cid:80) i ∈

Original: [ S ] r τ i = (cid:98) T − 1 2 , which proves (14).
Revised: [ S ] r τ i = (cid:98) T − 1 2 , which proves (14).

Original: To show (15), note that for any α ≥ 1 , 12 √ α a + 12 √ α b ≥ (cid:112) α 12 a + 12 b due to the arithmetic and geometric mean (AM-GM) inequality.
Revised: To show (15), note that for any α ≥ 1 , 12 √ α a + 12 √ α b ≥ (cid:112) α 12 a + 12 b due to the arithmetic and geometric mean (AM-GM) inequality.

Original: Averaging our two lower bounds on (cid:112) A (cid:98) T , (18) and (19), we conclude that
Revised: Averaging our two lower bounds on (cid:112) A (cid:98) T , (18) and (19), we conclude that

Original: [Equation]
Revised: [Equation]

Original: Here the ﬁrst term on the RHS bound comes purely from (cid:113) α ni − τi
Revised: Here the ﬁrst term on the RHS bound comes purely from (cid:113) α ni − τi

Original: [Equation]
Revised: [Equation]

Original: 2 λ (cid:48) τi when i
Revised: 2 λ (cid:48) τi when i

Original: = 1 since n 1 − τ 1 > 12 ( n 1 − τ 1 ) = 12 r τ 1 which leads to the coefﬁcient of 1 / 4 on RHS.
Revised: = 1 since n 1 − τ 1 > 12 ( n 1 − τ 1 ) = 12 r τ 1 which leads to the coefﬁcient of 1 / 4 on RHS.

Original: Now for the consistency arguments, note by deﬁnition of Q and r t we have Q (cid:98) T 1 ⊆ Q (cid:98) T 2 and r t, (cid:98) T 1 = r t, (cid:98) T 2 for any (cid:98) T 1 < (cid:98) T 2 ∈ Q (cid:98) T .
Revised: Now for the consistency arguments, note by deﬁnition of Q and r t we have Q (cid:98) T 1 ⊆ Q (cid:98) T 2 and r t, (cid:98) T 1 = r t, (cid:98) T 2 for any (cid:98) T 1 < (cid:98) T 2 ∈ Q (cid:98) T .

Original: To show the second inequality (16), we start again with (cid:98) T = T .
Revised: To show the second inequality (16), we start again with (cid:98) T = T .

Original: From the conclusion of Propositionand the observation that k ∈ S ≤ T for any k ∈ ( τ i , n i ] , giving
Revised: From the conclusion of Propositionand the observation that k ∈ S ≤ T for any k ∈ ( τ i , n i ] , giving

Original: [Equation]
Revised: [Equation]

Original: t =  n 1 (cid:88) t = (cid:115)λ (cid:48) t +
Revised: t =  n 1 (cid:88) t = (cid:115)λ (cid:48) t +

Original: S (cid:88) i
Revised: S (cid:88) i

Original: = n i (cid:88) t
Revised: = n i (cid:88) t

Original: =
Revised: =

Original: τ i + (cid:115)λ (cid:48) t .
Revised: τ i + (cid:115)λ (cid:48) t .

Original: Moreover, since for i < S and k ∈ ( n i , τ i +1 ] we have λ (cid:48) k
Revised: Moreover, since for i < S and k ∈ ( n i , τ i +1 ] we have λ (cid:48) k

Original: = α ( k − n i − 2)
Revised: = α ( k − n i − 2)

Original: λ (cid:48)
Revised: λ (cid:48)

Original: n i , and (cid:88) t ∈ ( n i ,τ i +1 ]
Revised: n i , and (cid:88) t ∈ ( n i ,τ i +1 ]

Original: (cid:115)λ (cid:48) t =  
Revised: (cid:115)λ (cid:48) t =  

Original: τ i +1 − n i (cid:88)
Revised: τ i +1 − n i (cid:88)

Original: j = α α j/ 
Revised: j = α α j/ 

Original: (cid:115)λ (cid:48) n i ≤ α √ α −(cid:115)λ (cid:48) n i .
Revised: (cid:115)λ (cid:48) n i ≤ α √ α −(cid:115)λ (cid:48) n i .

Original: Combining (20) and (21) with √ A T ≥ 12 (cid:80) i ∈
Revised: Combining (20) and (21) with √ A T ≥ 12 (cid:80) i ∈

Original: [ S − 1] (cid:113) 1 λ (cid:48) ni yields (16) since (cid:112)
Revised: [ S − 1] (cid:113) 1 λ (cid:48) ni yields (16) since (cid:112)

Original: A T ≥ 1(cid:32) n 1 (cid:88) t
Revised: A T ≥ 1(cid:32) n 1 (cid:88) t

Original: = (cid:115)λ (cid:48) t +
Revised: = (cid:115)λ (cid:48) t +

Original: S (cid:88) i
Revised: S (cid:88) i

Original: = n i (cid:88) t
Revised: = n i (cid:88) t

Original: =
Revised: =

Original: τ i + (cid:115)λ (cid:48) t (cid:33) +
Revised: τ i + (cid:115)λ (cid:48) t (cid:33) +

Original: S − 1 (cid:88) i = (cid:115)λ (cid:48) n i ≥ (cid:32) n 1 (cid:88) t
Revised: S − 1 (cid:88) i = (cid:115)λ (cid:48) n i ≥ (cid:32) n 1 (cid:88) t

Original: = (cid:115)λ (cid:48) t +
Revised: = (cid:115)λ (cid:48) t +

Original: S (cid:88) i
Revised: S (cid:88) i

Original: = n i (cid:88) t
Revised: = n i (cid:88) t

Original: =
Revised: =

Original: τ i + (cid:115)λ (cid:48) t (cid:33) + √
Revised: τ i + (cid:115)λ (cid:48) t (cid:33) + √

Original: α −4 α
Revised: α −4 α

Original: [Equation]
Revised: [Equation]

Original: B.3 Completing the proof of Theorem
Revised: B.3 Completing the proof of Theorem

Original: We now show how to use Proposition 1 and Lemma 1 to obtain optimal acceleration, considering the cases s ∈ (1 , ∞ ) , s = ∞ , and s = 1 in turn.
Revised: We now show how to use Proposition 1 and Lemma 1 to obtain optimal acceleration, considering the cases s ∈ (1 , ∞ ) , s = ∞ , and s = 1 in turn.

Original: The s ∈ (1 , ∞ ) case.
Revised: The s ∈ (1 , ∞ ) case.

Original: If E T ≤ 0 , the result f ( x T ) − f ( x (cid:63) ) ≤ 0 follows immediately.
Revised: If E T ≤ 0 , the result f ( x T ) − f ( x (cid:63) ) ≤ 0 follows immediately.

Original: Therefore, it sufﬁces to consider the case when E T > 0 .
Revised: Therefore, it sufﬁces to consider the case when E T > 0 .

Original: For any (cid:98) T ∈ Q T , applying Proposition 1 and Lemma(using that movement bounds hold for all iterations in Q T including the ﬁrst iterate t = 1 by assumption) yields
Revised: For any (cid:98) T ∈ Q T , applying Proposition 1 and Lemma(using that movement bounds hold for all iterations in Q T including the ﬁrst iterate t = 1 by assumption) yields

Original: [Equation]
Revised: [Equation]

Original: This implies E (cid:98) T ≤ D 0 /A (cid:98) T where (cid:112) A (cid:98)
Revised: This implies E (cid:98) T ≤ D 0 /A (cid:98) T where (cid:112) A (cid:98)

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Revised: rt

Revised: [Equation]

Original: λ
Revised: λ

Original: [Equation]
Revised: [Equation]

Original: t for (cid:80)
Revised: t for (cid:80)

Original: [Equation]
Revised: [Equation]

Original: The reverse Hölder inequality (which is a standard technique in analyzing MS acceleration [20, 7, 24, 42, 2]) states that, for all q > 1 , and any two vectors u, v with positive elements,
Revised: The reverse Hölder inequality (which is a standard technique in analyzing MS acceleration [19, 7, 23, 41, 2]) states that, for all q > 1 , and any two vectors u, v with positive elements,

Original: [Equation]
Revised: [Equation]

Original: i u
Revised: i u

Original: (cid:33)
Revised: (cid:33)

Original: [Equation]
Revised: [Equation]

Original: (cid:32)(cid:88)
Revised: (cid:32)(cid:88)

Original: i v
Revised: i v

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:33)
Revised: (cid:33)

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: We set q = 3 s +1 2( s +1) and apply the reverse Hölder inequality to obtain
Revised: We set q = 3 s +1 2( s +1) and apply the reverse Hölder inequality to obtain

Original: [Equation]
Revised: [Equation]

Original: where we used ( i ) the reverse Hölder inequality with u t =
Revised: where we used ( i ) the reverse Hölder inequality with u t =

Original: A q − 1 t √
Revised: A q − 1 t √

Original: [Equation]
Revised: [Equation]

Original: and v t =
Revised: and v t =

Original: A 1 − q t / (cid:112) λ (cid:48) t (for t ∈ Q (cid:98) T ) and − 12 · 11 − q =
Revised: A 1 − q t / (cid:112) λ (cid:48) t (for t ∈ Q (cid:98) T ) and − 12 · 11 − q =

Original: s +1 s − 1 , ( ii ) the bound (22), and ( iii ) the following lemma (proved in the next subsection) with a ← α 1 / 2 q and b ← r t ≥ 0 .
Revised: s +1 s − 1 , ( ii ) the bound (22), and ( iii ) the following lemma (proved in the next subsection) with a ← α 1 / 2 q and b ← r t ≥ 0 .

Original: Lemma 2.
Revised: Lemma 2.

Original: For all a ≥ 0 and b ≥ 1 , we have a b − 1 ≥ min { 1 , ln a } · b .
Revised: For all a ≥ 0 and b ≥ 1 , we have a b − 1 ≥ min { 1 , ln a } · b .

Original: Substituting the deﬁnitions
Revised: Substituting the deﬁnitions

Original: [Equation]
Revised: [Equation]

Original: the bound (23) can be rewritten as
Revised: the bound (23) can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: To deduce the growth rate of B t , we give the following lemma generalizing the analyses in prior work [7, 24] (see proof in the next Appendix B.4).
Revised: To deduce the growth rate of B t , we give the following lemma generalizing the analyses in prior work [7, 23] (see proof in the next Appendix B.4).

Original: Lemma 3.
Revised: Lemma 3.

Original: Let B 1 , ..., B k ∈ R > 0 , r 1 , ..., r k ∈ R ≥ 0 and β > 0 .
Revised: Let B 1 , ..., B k ∈ R > 0 , r 1 , ..., r k ∈ R ≥ 0 and β > 0 .

Original: Further, suppose that for some m > 1 and all i ∈ [ k ] it is the case that B mi ≥ β
Revised: Further, suppose that for some m > 1 and all i ∈ [ k ] it is the case that B mi ≥ β

Original: (cid:80)
Revised: (cid:80)

Original: j ∈
Revised: j ∈

Original: [ i ] B j · r j .
Revised: [ i ] B j · r j .

Original: Then for all i ∈ [ k ] we have that
Revised: Then for all i ∈ [ k ] we have that

Original: B
Revised: B

Original: [Equation]
Revised: [Equation]

Original: Applying the lemma with m = s +1 s − 1 and recalling that (cid:80) t ∈Q T r t = T − 1 2 , we obtain (for T (cid:48)
Revised: Applying the lemma with m = s +1 s − 1 and recalling that (cid:80) t ∈Q T r t = T − 1 2 , we obtain (for T (cid:48)

Original: =
Revised: =

Original: max Q T ),
Revised: max Q T ),

Original: [Equation]
Revised: [Equation]

Original: for any T ∈ Z >
Revised: for any T ∈ Z >

Original: [Equation]
Revised: [Equation]

Original: Since A T =
Revised: Since A T =

Original: B qq − 1 T and qq − 1 = 3 s +1 s − 1 this gives the desired growth rate of
Revised: B qq − 1 T and qq − 1 = 3 s +1 s − 1 this gives the desired growth rate of

Original: A T ≥ (cid:18)
Revised: A T ≥ (cid:18)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: 3 s + for any T ∈ Z > 0 , where, substituting back, we have
Revised: 3 s + for any T ∈ Z > 0 , where, substituting back, we have

Original: [Equation]
Revised: [Equation]

Original: Thus, for
Revised: Thus, for

Original: [Equation]
Revised: [Equation]

Original: = Ω
Revised: = Ω

Original: [Equation]
Revised: [Equation]

Original: we have A T ≥ 12 (cid:107) x 0 − x (cid:63) (cid:107) 2 /(cid:15) , and consequently
Revised: we have A T ≥ 12 (cid:107) x 0 − x (cid:63) (cid:107) 2 /(cid:15) , and consequently

Original: [Equation]
Revised: [Equation]

Original: The case for s ∈ (1 , ∞ ) follows immediately.
Revised: The case for s ∈ (1 , ∞ ) follows immediately.

Original: The s = ∞ case.
Revised: The s = ∞ case.

Original: Considering s = ∞ and q = 32 in (23) yields for any T ∈ Z > 0 and (cid:98) T ∈ Q T ,
Revised: Considering s = ∞ and q = 32 in (23) yields for any T ∈ Z > 0 and (cid:98) T ∈ Q T ,

Original: [Equation]
Revised: [Equation]

Original: Deﬁning
Revised: Deﬁning

Original: Bt := At 1 / 3 and β := min { 1 , 13 ln α } ·
Revised: B

Original: [Equation]
Revised: [Equation]

Original: 5 c 2 αD{+A 1 / 3 t and β := min { 1 , 
 ln+}
Revised: {+A 1 / 3 t and β := min { 1 , 
 ln+}

Original: [Equation]
Revised: [Equation]

Original: 1 − σ 2 ) (cid:1) − 1 /we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original:  for all (cid:98) T ∈ Q T .
Revised: , for all (cid:98) T ∈ Q T .

Original: We deduce an exponential rate of growth for B t using the following lemma, inspired by the analysis in [12] (and proved in the next subsection).
Revised: We deduce an exponential rate of growth for B t using the following lemma, inspired by the analysis in [12] (and proved in the next subsection).

Original: Lemma 4.
Revised: Lemma 4.

Original: Let B 1 , ..., B k ∈ R > 0 be non-decreasing and let r 1 , ..., r k ∈ R ≥ 0 and R i :=
Revised: Let B 1 , ..., B k ∈ R > 0 be non-decreasing and let r 1 , ..., r k ∈ R ≥ 0 and R i :=

Original: (cid:80)
Revised: (cid:80)

Original: j ∈
Revised: j ∈

Original: [ i ] r j for i ∈ [ k ] .
Revised: [ i ] r j for i ∈ [ k ] .

Original: Further, suppose that for some β > 0 , and all i ∈ [ k ] it is the case
Revised: Further, suppose that for some β > 0 , and all i ∈ [ k ] it is the case

Original: that B
Revised: that B

Original: i ≥ β · (cid:80) j ∈
Revised: i ≥ β · (cid:80) j ∈

Original: [ i ] B j · r j .
Revised: [ i ] B j · r j .

Original: Then B i ≥ exp( β ( R i − β − 1 )) B 1 for all i ∈ [ k ] .
Revised: Then B i ≥ exp( βR i − 1) B 1 for all i ∈ [ k ] .

Original: Applying the lemma and substituting back the deﬁnition of B t , we obtain,
Revised: Applying the lemma and substituting back the deﬁnition of B t , we obtain,

Original: [Equation]
Revised: [Equation]

Original: where we let β := min (cid:0) 1 , [-
-] ln 
Revised: where we let β := min (cid:0) 1 , [-
-]13 ln α (cid:1) ·

Original: [Equation]
Revised: [Equation]

Original: − 13 .
Revised: − 13 .

Original: Thus, for
Revised: Thus, for

Original: [Equation]
Revised: [Equation]

Original: we have A T ≥ 12 (cid:107) x 0 − x (cid:63) (cid:107) 2 /(cid:15) , and consequently
Revised: we have A T ≥ 12 (cid:107) x 0 − x (cid:63) (cid:107) 2 /(cid:15) , and consequently

Original: [Equation]
Revised: [Equation]

Original: which proves the case for s = ∞ .
Revised: which proves the case for s = ∞ .

Original: The s = 1 case.
Revised: The s = 1 case.

Original: This case corresponds to the standard analysis of Nesterov acceleration.
Revised: This case corresponds to the standard analysis of Nesterov acceleration.

Original: The (1 , c ) -movement bound guarantees that λ t ≤ c for all t .
Revised: The (1 , c ) -movement bound guarantees that λ t ≤ c for all t .

Original: Recalling that λ (cid:48) t ≤ λ t for all t ∈ Q T , the bound (15), yields
Revised: Recalling that λ (cid:48) t ≤ λ t for all t ∈ Q T , the bound (15), yields

Original: [Equation]
Revised: [Equation]

Original: where the ﬁnal bound uses Lemma 1.
Revised: where the ﬁnal bound uses Lemma 1.

Original: Consequently, the error bound (9) we have
Revised: Consequently, the error bound (9) we have

Original: [Equation]
Revised: [Equation]

Original: yielding the claimed result for s = 1 .
Revised: yielding the claimed result for s = 1 .

Original: B.4 Helper lemmas
Revised: B.4 Helper lemmas

Original: Lemma 2.
Revised: Lemma 2.

Original: For all a ≥ 0 and b ≥ 1 , we have a b − 1 ≥ min { 1 , ln a } · b .
Revised: For all a ≥ 0 and b ≥ 1 , we have a b − 1 ≥ min { 1 , ln a } · b .

Original: Proof.
Revised: Proof.

Original: Deﬁne the difference function f ( x ) :
Revised: Deﬁne the difference function f ( x ) :

Original: = a x − 1 − min (1 , ln a ) · x .
Revised: = a x − 1 − min (1 , ln a ) · x .

Original: We note that clearly f (1)
Revised: We note that clearly f (1)

Original: ≥and the ﬁrst-order derivative f (cid:48) ( x ) =
Revised: ≥and the ﬁrst-order derivative f (cid:48) ( x ) =

Original: (ln a ) · a x − 1 − min (1 , ln a ) ≥ 0 for all x ≥ 1 .
Revised: (ln a ) · a x − 1 − min (1 , ln a ) ≥ 0 for all x ≥ 1 .

Original: Consequently, by the integral formula f ( x ) = (cid:82) x 1 f (cid:48)
Revised: Consequently, by the integral formula f ( x ) = (cid:82) x 1 f (cid:48)

Original: ( z ) dz we have that f ( x ) ≥ 0 for all x ≥ 1 .
Revised: ( z ) dz we have that f ( x ) ≥ 0 for all x ≥ 1 .

Original: Lemma 3.
Revised: Lemma 3.

Original: Let B 1 , ..., B k ∈ R > 0 , r 1 , ..., r k ∈ R ≥ 0 and β > 0 .
Revised: Let B 1 , ..., B k ∈ R > 0 , r 1 , ..., r k ∈ R ≥ 0 and β > 0 .

Original: Further, suppose that for some m > 1 and all i ∈ [ k ] it is the case that B mi ≥ β
Revised: Further, suppose that for some m > 1 and all i ∈ [ k ] it is the case that B mi ≥ β

Original: (cid:80)
Revised: (cid:80)

Original: j ∈
Revised: j ∈

Original: [ i ] B j · r j .
Revised: [ i ] B j · r j .

Original: Then for all i ∈ [ k ] we have that
Revised: Then for all i ∈ [ k ] we have that

Original: B i ≥ (cid:16) m − 1 m β · (cid:80) j ∈
Revised: B i ≥ (cid:16) m − 1 m β · (cid:80) j ∈

Original: [ i ] r j (cid:17) 1 / ( m − 1) .
Revised: [ i ] r j (cid:17) 1 / ( m − 1) .

Original: Proof.
Revised: Proof.

Original: Without loss of generality we take β = 1 , since otherwise we may redeﬁne r i to be βr i .
Revised: Without loss of generality we take β = 1 , since otherwise we may redeﬁne r i to be βr i .

Original: Furthermore, we assume r 1 > 0 as otherwise we can divide into the following two cases:
Revised: Furthermore, we assume r 1 > 0 as otherwise we can divide into the following two cases:

Original: if all r i = 

Revised: if all r i =

Original: the desired inequality naively holds; 2.
Revised: the desired inequality naively holds; 2.

Original: if there exists some i 0 such that r i 0 > 0 and r i = 

Revised: if there exists some i 0 such that r i 0 > 0 and r i =

Original: for all i < i 0 , then it sufﬁces to consider the sequence starting from i 0 .
Revised: for all i < i 0 , then it sufﬁces to consider the sequence starting from i 0 .

Original: First, for i = 1 , note that B m 1 ≥ B 1 · r 1 and consequently B 1 ≥ ( r
Revised: First, for i = 1 , note that B m 1 ≥ B 1 · r 1 and consequently B 1 ≥ ( r

Original: ( m−
Revised: ( m

Original:  ≥ (
Revised: − 1) ≥ (

Original: m −
Revised: m −

Original: 1 m · r 1 ) 1 / ( m − 1) as desired.
Revised: 1 m · r 1 ) 1 / ( m − 1) as desired.

Original: Next, for any j > 1 , we have
Revised: Next, for any j > 1 , we have

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: and consequently r
Revised: and consequently r

Original: [Equation]
Revised: [Equation]

Original: Summing the above inequality for all j ∈ [ i ] , noting that for i = 1 we have r 1 ≤ mm − 1 ( B 1 r 1 ) m − 1 m , and rearranging terms yields
Revised: Summing the above inequality for all j ∈ [ i ] , noting that for i = 1 we have r 1 ≤ mm − 1 ( B 1 r 1 ) m − 1 m , and rearranging terms yields

Original: [Equation]
Revised: [Equation]

Original: Combining this with the condition B mi +1 ≥ (cid:80) j ∈ [ i +1]
Revised: Combining this with the condition B mi +1 ≥ (cid:80) j ∈ [ i +1]

Original: B j r j concludes the proof.
Revised: B j r j concludes the proof.

Original: Lemma 4.
Revised: Lemma 4.

Original: Let B 1 , ..., B k ∈ R > 0 be non-decreasing and let r 1 , ..., r k ∈ R ≥ 0 and R i :=
Revised: Let B 1 , ..., B k ∈ R > 0 be non-decreasing and let r 1 , ..., r k ∈ R ≥ 0 and R i :=

Original: (cid:80)
Revised: (cid:80)

Original: j ∈
Revised: j ∈

Original: [ i ] r j for i ∈ [ k ] .
Revised: [ i ] r j for i ∈ [ k ] .

Original: Further, suppose that for some β > 0 , and all i ∈ [ k ] it is the case
Revised: Further, suppose that for some β > 0 , and all i ∈ [ k ] it is the case

Original: that B
Revised: that B

Original: i ≥ β · (cid:80) j ∈
Revised: i ≥ β · (cid:80) j ∈

Original: [ i ] B j · r j .
Revised: [ i ] B j · r j .

Original: Then B i ≥ exp( β ( R i − β − 1 )) B 1 for all i ∈ [ k ] .
Revised: Then B i ≥ exp( βR i − 1) B 1 for all i ∈ [ k ] .

Original: Proof.
Revised: Proof.

Original: Let k 0 denote the largest element of [ k ] for which R k 0 < β − 1 and let k 

Revised: Let k 0 denote the largest element of [ k ] for which R k 0 < β − 1 and let k

Original: =
Revised: =

Original: if there is no such element.
Revised: if there is no such element.

Original: Note that exp( β ( R k 0 − β − 1 ) ≤ 1 .
Revised: Note that exp( β ( R k 0 − β − 1 ) ≤ 1 .

Original: Since B k increases monotonically in k this implies that B
Revised: Since B k increases monotonically in k this implies that B

Original: i ≥ exp (cid:0) β ( R i − β − 1 ) (cid:1) B 1 for all i ∈ [ k 0 ] .
Revised: i ≥ exp (cid:0) β ( R i − β − 1 ) (cid:1) B 1 for all i ∈ [ k 0 ] .

Original: For i = k 

Revised: For i = k

Original: + 1 , we have
Revised: + 1 , we have

Original: [Equation]
Revised: [Equation]

Original:  i , where the ﬁrst inequality is due to the deﬁnition of k 0 and that B i is non-increasing.
Revised: βr i , where the ﬁrst inequality is due to the deﬁnition of k 0 and that B i is non-increasing.

Original: Consequently,
Revised: Consequently,

Original: [Equation]
Revised: [Equation]

Original: For any i + 1 such that 2 ≤ i + 1 ≤ k and j ∈ [ i ] , we have 
Revised: For any i + 1 such that 2 ≤ i + 1 ≤ k and j ∈ [ i ] , we have (cid:88)

Original: [Equation]
Revised: [Equation]

Original: [ j +1] [Equation]
Revised: [ j +1] βB j (cid:48) r j (cid:48)

Original: 1 , and consequently
Revised:   βr j +1 , and consequently

Original: [Equation]
Revised: [Equation]

Original: j
Revised: (cid:48)

Original: [Equation]
Revised: [Equation]

Original:  [ j +1] [Equation] · B j (cid:48)r j (cid:48)
Revised: (cid:80) j (cid:48)∈ [ j +1] β · B j (cid:48)

Original: [Equation] j (cid:48)∈ [ j ] β · B j (cid:48)
Revised: r j (cid:48) (cid:80) j (cid:48)∈ [ j ] β · B j (cid:48)

Original: r j (cid:48)
Revised: r j (cid:48)

Original: t dt = log   β · (cid:80) j (cid:48) ∈
Revised: t dt = log   β · (cid:80) j (cid:48) ∈

Original: [ j +1]
Revised: [ j +1]

Original: B j (cid:48)
Revised: B j (cid:48)

Original: r j (cid:48) β · (cid:80) j (cid:48) ∈
Revised: r j (cid:48) β · (cid:80) j (cid:48) ∈

Original: [ j ]
Revised: [ j ]

Original: B j (cid:48) r j (cid:48)
Revised: B j (cid:48) r j (cid:48)

Original:   .
Revised:   .

Original: Summing up above inequalities and rearranging terms, we obtain that B
Revised: Summing up above inequalities yields that for any i ∈ [ k 0 + 1 , k ] it is the case that β

Original:  i  ≥exp (cid:0) β ( R i − β − 1 ) (cid:1) Bholds for all
Revised: [Equation] Since β (cid:80) ij = k 0 +1 r j = β ( R i − R k 0 ) ≥ β ( R i − β − 1 ) , where we deﬁne R 0 :

Original:  i > k 0 , and hence—by deﬁnition of k 0 —for all i ≥ 1 .
Revised: = 0 , we obtain that B i ≥ exp ( βR i − 1) B 1 holds for all i > k 0 , and hence for all i ≥ 1 .

Original: C Generalized oracle notions
Revised: C Generalized oracle notions

Original: In this section, we consider a setting where the convex function f may be non-differentiable (i.e., ∇ f might not exist everywhere) and the problem may be constrained (i.e., the convex closed domain
Revised: In this section, we consider a setting where the convex function f may be non-differentiable (i.e., ∇ f might not exist everywhere) and the problem may be constrained (i.e., the convex closed domain

Original: X may be different from R d ).
Revised: X may be different from R d ).

Original: We consider two types of oracles: a slight generalization of the
Revised: We consider two types of oracles: a slight generalization of the

Original: MS oracle for the non-differentiable and/or constrained setting, and a fairly different notion of a “stochastic proximal oracle” similar to the one considered in [4].
Revised: MS oracle for the non-differentiable and/or constrained setting, and a fairly different notion of a “stochastic proximal oracle” similar to the one considered in [4].

Original: We also provide a slight variation of
Revised: We also provide a slight variation of

Original: Algorithm 1 that makes use of these oracles and also allows more ﬂexibility in choosing some of the iterates, and prove convergence rate bounds for this algorithm combined with either oracle.
Revised: Algorithm 1 that makes use of these oracles and also allows more ﬂexibility in choosing some of the iterates, and prove convergence rate bounds for this algorithm combined with either oracle.

Original: To generalize Deﬁnition 1 of a MS oracle, we consider mappings that return, in addition to x ∈ X and λ > 0 , a vector g ∈ R d that replaces ∇ f ( x ) .
Revised: To generalize Deﬁnition 1 of a MS oracle, we consider mappings that return, in addition to x ∈ X and λ > 0 , a vector g ∈ R d that replaces ∇ f ( x ) .

Original: More speciﬁcally, recall the deﬁnition of the subdifferential of f at x ∈ X :
Revised: More speciﬁcally, recall the deﬁnition of the subdifferential of f at x ∈ X :

Original: [Equation]
Revised: [Equation]

Original: An element of ∂f ( x ) is called a subgradient of f at x .
Revised: An element of ∂f ( x ) is called a subgradient of f at x .

Original: Note that when f is differentiable at x we have ∇ f ( x ) ∈ ∂f ( x ) .
Revised: Note that when f is differentiable at x we have ∇ f ( x ) ∈ ∂f ( x ) .

Original: However, on the boundary of X there may additional elements in the subdifferential even when f is differentiable.
Revised: However, on the boundary of X there may additional elements in the subdifferential even when f is differentiable.

Original: Our generalized MS oracle (that originally appeared in [33]) returns g , a subgradient of f at x , such that the MS condition holds with g instead of ∇ f ( x ) .
Revised: Our generalized MS oracle (that originally appeared in [32]) returns g , a subgradient of f at x , such that the MS condition holds with g instead of ∇ f ( x ) .

Original: Deﬁnition 3 (Generalized MS oracle) .
Revised: Deﬁnition 3 (Generalized MS oracle) .

Original: An oracle O :
Revised: An oracle O :

Original: X × R + → X × R d × R + is a σ -Generalized
Revised: X × R + → X × R d × R + is a σ -Generalized

Original: MS oracle for function f : X → R if for every y ∈ X and λ (cid:48) > 0 , the points ( x, g, λ ) =
Revised: MS oracle for function f : X → R if for every y ∈ X and λ (cid:48) > 0 , the points ( x, g, λ ) =

Original: O ( y ; λ (cid:48) ) satisfy the following:
Revised: O ( y ; λ [Equation] ) satisfy the following:

Original: [Equation]
Revised: [Equation]

Original: We remark that considering subgradients instead of gradients is essential for handling constrained optimization even when f is differentiable, because even exact proximal point do not necessarily satisfy the simple MS condition (2).
Revised: We remark that considering subgradients instead of gradients is essential for handling constrained optimization even when f is differentiable, because even exact proximal point do not necessarily satisfy the simple MS condition (2).

Original: That is, letting F λ ( x ) =
Revised: That is, letting F λ ( x ) =

Original: f ( x ) + λ 2 (cid:107) x − y (cid:107) 2 , the point x λ = argmin
Revised: f ( x ) + λ 2 (cid:107) x − y (cid:107) 2 , the point x λ = argmin

Original: x
Revised: x

Original: ∈X F λ ( x ) does not necessarily satisfy x λ = y − 1 λ ∇ f ( x λ ) .
Revised: ∈X F λ ( x ) does not necessarily satisfy x λ = y − 1 λ ∇ f ( x λ ) .

Original: Nevertheless, the ﬁrst-order optimality conditions of characterizing x λ guarantee that λ ( y − x λ ) ∈ ∂f ( x λ ) .
Revised: Nevertheless, the ﬁrst-order optimality conditions of characterizing x λ guarantee that λ ( y − x λ ) ∈ ∂f ( x λ ) .

Original: Therefore, exact proximal points are 0 -Generalized MS oracles.
Revised: Therefore, exact proximal points are 0 -Generalized MS oracles.

Original: We now present a different oracle, with a probabilistic approximation condition that relates directly to the exact proximal point x λ .
Revised: We now present a different oracle, with a probabilistic approximation condition that relates directly to the exact proximal point x λ .

Original: The advantage of approximation conditions of this kind is that they can be efﬁciently satisﬁed using stochastic ﬁrst-order methods in certain non-smooth problems where certifying (24) is hard [4].
Revised: The advantage of approximation conditions of this kind is that they can be efﬁciently satisﬁed using stochastic ﬁrst-order methods in certain non-smooth problems where certifying (24) is hard [4].

Original: Deﬁnition 4 (Stochastic proximal oracle) .
Revised: Deﬁnition 4 (Stochastic proximal oracle) .

Original: A (randomized) oracle O :
Revised: A (randomized) oracle O :

Original: X × R + → X × R d × R + is a σ -stochastic proximal oracle for function f : R d → R if for every y ∈ X and λ (cid:48) > 0 , the points ( x, g, λ ) =
Revised: X × R + → X × R d × R + is a σ -stochastic proximal oracle for function f : R d → R if for every y ∈ X and λ (cid:48) > 0 , the points ( x, g, λ ) =

Original: O ( y ; λ (cid:48) ) satisfy the following:
Revised: O ( y ; λ (cid:48) ) satisfy the following:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where F λ ( x (cid:48) )
Revised: where F λ ( x (cid:48) )

Original: = f
Revised: = f

Original: ( x (cid:48) )
Revised: ( x (cid:48) )

Original: + λ 2 (cid:107) x (cid:48)
Revised: + λ 2 (cid:107) x (cid:48)

Original: − y (cid:107) 2 , x λ = argmin x (cid:48) ∈X F λ ( x (cid:48) ) and g λ = λ
Revised: − y (cid:107) 2 , x λ = argmin x (cid:48) ∈X F λ ( x (cid:48) ) and g λ = λ

Original: ( y − x λ ) , and all expectations are conditional on y, λ (cid:48) and λ .
Revised: ( y − x λ ) , and all expectations are conditional on y, λ (cid:48) and λ .

Original: Three remarks are in order.
Revised: Three remarks are in order.

Original: First, note that exact proximal points are also 0 -stochastic proximal oracles.
Revised: First, note that exact proximal points are also 0 -stochastic proximal oracles.

Original: Second, the condition E g = g λ =
Revised: Second, the condition E g = g λ =

Original: λ ( y − x λ ) implies that if the stochastic proximal oracle outputs a deterministic g then it also computes x λ exactly.
Revised: λ ( y − x λ ) implies that if the stochastic proximal oracle outputs a deterministic g then it also computes x λ exactly.

Original: Third, a σ -Generalized MS oracle output x, g, λ satisﬁes g + λ ( x − y ) ∈ ∂F λ ( x ) and therefore, by λ -strong convexity of F λ ,
Revised: Third, a σ -Generalized MS oracle output x, g, λ satisﬁes g + λ ( x − y ) ∈ ∂F λ ( x ) and therefore, by λ -strong convexity of F λ ,

Original: [Equation]
Revised: [Equation]

Original: Therefore, up to a replacing σ with σ/ √ 2 , the generalized MS-condition (24) implies the ﬁrst part of (25).
Revised: Therefore, up to a replacing σ with σ/ √ 2 , the generalized MS-condition (24) implies the ﬁrst part of (25).

Original: Moreover, when g is deterministic its variance is zero, giving the third part of the condition.
Revised: Moreover, when g is deterministic its variance is zero, giving the third part of the condition.

Original: The second part of the condition, however, is not directly implied by (24).
Revised: The second part of the condition, however, is not directly implied by (24).

Original: Nevertheless, given a procedure that for any δ ≥ 0 outputs a point x δ such that F λ ( x δ ) − min x (cid:48) ∈X
Revised: Nevertheless, given a procedure that for any δ ≥ 0 outputs a point x δ such that F λ ( x δ ) − min x (cid:48) ∈X

Original: F λ ( x (cid:48) ) ≤ λδ 2 2 (e.g., an σ -MS oracle with appropriate value of σ ), it is possible to generically obtain an estimator ˆ x λ that is unbiased for x λ via multilevel Monte Carlo [see 4, 11], thereby obtaining g
Revised: F λ ( x (cid:48) ) ≤ λδ 2 2 (e.g., an σ -MS oracle with appropriate value of σ ), it is possible to generically obtain an estimator ˆ x λ that is unbiased for x λ via multilevel Monte Carlo [see 4, 11], thereby obtaining g

Original: = λ ( y − ˆ x λ ) satisfying the second part of (25) as well as the variance bound in the third part of (25).
Revised: = λ ( y − ˆ x λ ) satisfying the second part of (25) as well as the variance bound in the third part of (25).

Original: Algorithm 4 uses either the generalized MS oracle (Deﬁnition 3) or the stochastic proximal oracle (Deﬁnition 4).
Revised: Algorithm 4 uses either the generalized MS oracle (Deﬁnition 3) or the stochastic proximal oracle (Deﬁnition 4).

Original: The differences between it and Algorithm 1 are highlighted in blue.
Revised: The differences between it and Algorithm 1 are highlighted in blue.

Original: There are two differences addition to the obvious one in the oracle interface (which now returns an additional vector g t +1 ).
Revised: There are two differences addition to the obvious one in the oracle interface (which now returns an additional vector g t +1 ).

Original: First, we use the vector g t +1 to update v t using a projected mirror descent step (here Proj X denotes the Euclidean projection unto X ).
Revised: First, we use the vector g t +1 to update v t using a projected mirror descent step (here Proj X denotes the Euclidean projection unto X ).

Original: Second, we allow the algorithm to replace the point ¯ x t output from the oracle with any other point
Revised: Second, we allow the algorithm to replace the point ¯ x t output from the oracle with any other point

Original: ˜ x t that has a lower function value.
Revised: ˜ x t that has a lower function value.

Original: We note that such option exists also in the original proposal by Monteiro and Svaiter [33] and is independent of the other generalizations studied in this section.
Revised: We note that such option exists also in the original proposal by Monteiro and Svaiter [32] and is independent of the other generalizations studied in this section.

Original: Theorem 6.
Revised: Theorem 6.

Original: Let f : X → R be convex and differentiable, let X be closed and convex, and consider
Revised: Let f : X → R be convex and differentiable, let X be closed and convex, and consider

Original: Algorithm 1 with parameters α > 1 , λ (cid:48) > 0 , and a σ -Generalized MS oracle (Deﬁnition 3) or σstochastic proximal oracle (Deﬁnition 4) for f with σ ∈ [0 , 0 . 99) .
Revised: Algorithm 1 with parameters α > 1 , λ (cid:48) > 0 , and a σ -Generalized MS oracle (Deﬁnition 3) or σstochastic proximal oracle (Deﬁnition 4) for f with σ ∈ [0 , 0 . 99) .

Original: Let p ≥ 1 and c > 0 , and suppose that for all t such that λ t > λ (cid:48) t or t = 1 , the iterates (¯ x t , y t − 1 , λ t ) satisfy a ( s, c ) -movement
Revised: Let p ≥ 1 and c > 0 , and suppose that for all t such that λ t > λ (cid:48) t or t = 1 , the iterates (¯ x t , y t − 1 , λ t ) satisfy a ( s, c ) -movement

Original: bound (Deﬁnition 2) with probability 1.
Revised: bound (Deﬁnition 2) with probability 1.

Original: There exist C α,s = O (cid:16) s min { s, ln α } α s +1 3 s +1 (cid:17) and K α
Revised: There exist C α,s = O (cid:16) s min { s, ln α } α s +1 3 s +1 (cid:17) and K α

Original: = O (cid:0) 1 ln α α 1 / 3 (cid:1) such that the following holds.
Revised: = O (cid:0) 1 ln α α 1 / 3 (cid:1) such that the following holds.

Original: Let x (cid:63) ∈ X ; if O is a stochastic proximal oracle then let x (cid:63) be a minimizer of f .
Revised: Let x (cid:63) ∈ X ; if O is a stochastic proximal oracle then let x (cid:63) be a minimizer of f .

Original: For any (cid:15) > 0 , when
Revised: For any (cid:15) > 0 , when

Original: [Equation] s
Revised: T ≥

Original: [Equation]
Revised: [Equation]

Original: we have f ( x T ) − f ( x (cid:63) ) ≤ (cid:15) with probability at least 2 / 3 .

Original: Before providing the proof, we make three more remarks.

Original: First, the success probability is relevant only for the stochastic proximal oracle; the bound for the generalized MS oracle holds with probability

Original: Input: Initial x 0 , generalized oracle O
Revised: Input: Initial x 0 , generalized oracle O

Original: Parameters: Initial λ (cid:48) 0 , multiplicative adjustment factor α >1 Set v 0 = x 0 , A 0 =2 ¯ x 1 , g 1 , λ 1 =
Revised: Parameters: Initial λ (cid:48) 0 , multiplicative adjustment factor α >1 Set v 0 = x 0 , A 0 =2 ¯ x 1 , g 1 , λ 1 =

Original: O ( x 0 ; λ (cid:48) 0 ) , λ (cid:48) 1 = λ3 for t = 0 , 1 , . .
Revised: O ( x 0 ; λ (cid:48) 0 ) , λ (cid:48) 1 = λ3 for t = 0 , 1 , . .

Original: , do a (cid:48) t +1 = 1 2 λ (cid:48)
Revised: , do a (cid:48) t +1 = 1 2 λ (cid:48)

Original: t +1 (cid:0) 

Revised: t +1 (cid:0)

Original: + (cid:112) 

Revised: + (cid:112)

Original: + 4 λ (cid:48)
Revised: + 4 λ (cid:48)

Original: t +1 A t (cid:1)
Revised: t +1 A t (cid:1)

Original: A (cid:48) t +1 =
Revised: A (cid:48) t +1 =

Original: A t + a (cid:48) t +  y t =
Revised: A t + a (cid:48) t +  y t =

Original: A t A (cid:48) t +1 x t + a (cid:48) t +1 A (cid:48)
Revised: A t A (cid:48) t +1 x t + a (cid:48) t +1 A (cid:48)

Original: t +1 v t if t > 0 then ¯ x t +1 , g t +1 , λ t
Revised: t +1 v t if t > 0 then ¯ x t +1 , g t +1 , λ t

Original: +

Revised: +

Original: =
Revised: =

Original: O ( y t ; λ (cid:48) t +1 )
Revised: O ( y t ; λ (cid:48) t +1 )

Original: Let ˜ x t +1 ∈
Revised: Let ˜ x t +1 ∈

Original: X satisfy f
Revised: X satisfy f

Original: (˜ x t +1 ) ≤ f (¯ x t +1 )
Revised: (˜ x t +1 ) ≤ f (¯ x t +1 )

Original: if λ t +1 ≤ λ (cid:48) t +1 then a t +1 = a (cid:48) t +1 , A t +1 =
Revised: if λ t +1 ≤ λ (cid:48) t +1 then a t +1 = a (cid:48) t +1 , A t +1 =

Original: A t + a t x t +1 =
Revised: A t + a t x t +1 =

Original: ˜ x t +  λ (cid:48) t +

Revised: ˜ x t +  λ (cid:48) t +

Original: = 

Revised: =

Original: α λ (cid:48) t
Revised: α λ (cid:48) t

Original: +  else γ t
Revised: +  else γ t

Original: +1 = λ (cid:48)
Revised: +1 = λ (cid:48)

Original: t +1 λ t +  a t +1 = γ t +1 a (cid:48)
Revised: t +1 λ t +  a t +1 = γ t +1 a (cid:48)

Original: t +1 , A t +1 =
Revised: t +1 , A t +1 =

Original: A t + a t x t +1 =
Revised: A t + a t x t +1 =

Original: (1 − γ t +1 )
Revised: (1 − γ t +1 )

Original: A t
Revised: A t

Original: A t + x t + γ t
Revised: A t + x t + γ t

Original: +

Revised: +

Original: A
Revised: A

Original: (cid:48) t +1 A t + ˜ x t +  λ (cid:48)
Revised: (cid:48) t +1 A t + ˜ x t +  λ (cid:48)

Original: t +2 =
Revised: t +2 =

Original: αλ (cid:48) t
Revised: αλ (cid:48) t

Original: +  v t +1 =
Revised: +  v t +1 =

Original: [Table]
Revised: [Table]

Revised: we have f ( x T ) − f ( x (cid:63) ) ≤ (cid:15) with probability at least 2 / 3 .

Revised: Before providing the proof, we make three more remarks.

Revised: First, the success probability is relevant only for the stochastic proximal oracle; the bound for the generalized MS oracle holds with probability 1.

Original: Second, the need to assume that x (cid:63) is a minimizer of f is also due to a technical issue with the analysis of the stochastic proximal oracle, pointed out in the proof below.
Revised: Second, the need to assume that x (cid:63) is a minimizer of f is also due to a technical issue with the analysis of the stochastic proximal oracle, pointed out in the proof below.

Original: Finally, we note that for stochastic proximal oracle we may require the movement bounds to hold on either (˜ x t , y t − 1 , λ t ) as stated in the theorem, or on ( x (cid:63)t , y t − 1 , λ t ) , where x (cid:63)t is the exact λ t proximal point of y t − 1 .
Revised: Finally, we note that for stochastic proximal oracle we may require the movement bounds to hold on either (˜ x t , y t − 1 , λ t ) as stated in the theorem, or on ( x (cid:63)t , y t − 1 , λ t ) , where x (cid:63)t is the exact λ t proximal point of y t − 1 .

Original: We recommend reading the proof of Theorem 1 (in Appendix B) before reading the following proof.
Revised: We recommend reading the proof of Theorem 1 (in Appendix B) before reading the following proof.

Original: Proof of Theorem 6.
Revised: Proof of Theorem 6.

Original: The proof consists of showing that a version of Proposition 1 holds under the conditions of Theorem 6; from there on the arguments on the analysis of the growth rate of A T is identical.
Revised: The proof consists of showing that a version of Proposition 1 holds under the conditions of Theorem 6; from there on the arguments on the analysis of the growth rate of A T is identical.

Original: For generalized MS oracles, the proof of Proposition 1 goes through unchanged, except for ¯ x t replacing ˜ x t , the subgradient g t ∈ ∂f (¯ x t ) replacing ∇ f (˜ x t ) , and using f (˜ x t ) ≤ f (¯ x t ) to show that A t +1 E t +1 ≤ (1 − γ t +1 )
Revised: For generalized MS oracles, the proof of Proposition 1 goes through unchanged, except for ¯ x t replacing ˜ x t , the subgradient g t ∈ ∂f (¯ x t ) replacing ∇ f (˜ x t ) , and using f (˜ x t ) ≤ f (¯ x t ) to show that A t +1 E t +1 ≤ (1 − γ t +1 )

Original: A t E t + γ t
Revised: A t E t + γ t

Original: +

Revised: +

Original: A
Revised: A

Original: (cid:48)
Revised: (cid:48)

Original: t +1 [ f (¯ x t +1 )
Revised: t +1 [ f (¯ x t +1 )

Original: − f ( x (cid:63) )] .
Revised: − f ( x (cid:63) )] .

Original: Next, we consider stochastic proximal oracles and adapt [4, Lemma 5], which considers a very similar oracle, to account for our momentum damping scheme (lines 14 to 17).
Revised: Next, we consider stochastic proximal oracles and adapt [4, Lemma 5], which considers a very similar oracle, to account for our momentum damping scheme (lines 14 to 17).

Original: Beginning with some notation, we deﬁne the ﬁltration
Revised: Beginning with some notation, we deﬁne the ﬁltration

Original: [Equation]
Revised: [Equation]

Original: so that a t +1 , a (cid:48) t +1 , A t +1 ∈ F t .
Revised: so that a t +1 , a (cid:48) t +1 , A t +1 ∈ F t .

Original: In addition, we let
Revised: In addition, we let

Original: [Equation]

Original: argmin

Original: [Equation]
Revised: [Equation]

Original: and note that ˆ x t +1 ∈ F t and moreover thatˆ
Revised: and note that ˆ x t +1 ∈ F t and moreover that

Original: [Equation]
Revised: [Equation]

Original: ] by the second part of (25).
Revised: by the second part of (25).

Original: We also deﬁne E t := f ( x t ) − f ( x (cid:63) ) , D t := 12 (cid:107) v t − x (cid:63) (cid:107) 2 , and
Revised: We also deﬁne E t := f ( x t ) − f ( x (cid:63) ) , D [Equation] := 12 (cid:107) v t − x (cid:63) (cid:107) 2 , and

Original: M t +1 :
Revised: M t +1 :

Original: = 12 (cid:107) ¯ x t +1 − y t (cid:107) 2 as in Proposition 1 (except with ¯ x t instead of ˜ x t ).
Revised: = 12 (cid:107) ¯ x t +1 − y t (cid:107) 2 as in Proposition 1 (except with ¯ x t instead of ˜ x t ).

Original: The update formula for v t gives us
Revised: The update formula for v t gives us

Original: [Equation]
Revised: [Equation]

Original: Rearranging and taking expectation, we have
Revised: Rearranging and taking expectation, we have

Original: [Equation]
Revised: [Equation]

Original: Moreover, by the second and third parts of (25),
Revised: Moreover, by the second and third parts of (25),

Original: [Equation]
Revised: [Equation]

Original:  By the deﬁnition y t and A (cid:48) t +1 =A t + a
Revised: t ] . By the deﬁnition y t and A (cid:48) t +1 =

Original: [Equation]

Original:  t +1 , we have[Equation] A t
Revised: A t + a (cid:48) t +1 , we have

Original: [Equation]
Revised: [Equation]

Original: x t

Original: [Equation]

Original: ˆ

Original: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: where we used ( i ) the fact that ˆ g t +1 ∈ ∂f (ˆ x t +1 ) and ( ii )
Revised: where we used ( i ) the fact that ˆ g t +1 ∈ ∂f (ˆ x t +1 ) and ( ii )

Original: that A (cid:48)
Revised: that A (cid:48)

Original: t +1 =
Revised: t +1 =

Original: A t + a (cid:48) t +1 and ˆ x t +1 − y t
Revised: A t + a (cid:48) t +1 and ˆ x t +1 − y t

Original: = − ˆ g t +1 /λ t +1 .
Revised: = − ˆ g t +1 /λ t +1 .

Original: To connect f (ˆ x t +1 ) to f (¯ x t +1 ) , we use the ﬁrst part of (25), which gives
Revised: To connect f (ˆ x t +1 ) to f (¯ x t +1 ) , we use the ﬁrst part of (25), which gives

Original: E [ f (¯ x [Equation] 1 ) + [Equation]
Revised: E [ f (¯ x t +1 ) + λ t +1 M

Original: 1 M

Original: [Equation]
Revised: [Equation]

Original: 1 |

Original: [Equation] ] ≤ f (ˆ [Equation] +
Revised: F

Original: [Equation]
Revised: [Equation]

Original: 1 2 (cid:107)ˆ
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: σ
Revised: E [ M t

Original: [Equation]
Revised: [Equation]

Original: λ
Revised: | F

Original: [Equation]
Revised: [Equation]

Original: +1 E

Original: [ M t +1 | F t ] .

Original: Substituting back and recalling that (cid:107)
Revised: Substituting back and recalling that (cid:107)

Original: ˆ x t +1 − y t (cid:107) =
Revised: ˆ x t +1 − y t (cid:107) =

Original: (cid:107) ˆ g t +1 (cid:107) /λ
Revised: (cid:107) ˆ g t +1 (cid:107) /λ

Original: t +1 and f (˜ x t +1 ) ≤ f (¯ x t +1 ) gives
Revised: t +1 and f (˜ x t +1 ) ≤ f (¯ x t +1 ) gives

Original: A 
Revised: A (cid:48)

Original: [Equation]
Revised: [Equation]

Revised: f

Original: [Equation]
Revised: [Equation]

Original: Let γ t +1 :
Revised: Let γ t +1 :

Original: = min (cid:110) 1 , λ (cid:48)
Revised: = min (cid:110) 1 , λ (cid:48)

Original: t +1 λ t +1 (cid:111) and note that this deﬁnition is consistent with γ t +1 as deﬁned in the algorithm and
Revised: t +1 λ t +1 (cid:111) and note that this deﬁnition is consistent with γ t +1 as deﬁned in the algorithm and

Original: that, for any value of λ (cid:48) t +1 /λ t
Revised: that, for any value of λ (cid:48) t +1 /λ t

Original: +1 we have a t +1 =
Revised: +1 we have a t +1 =

Original: γ t +1 a (cid:48)
Revised: γ t +1 a (cid:48)

Original: t +1 , A t +1 =
Revised: t +1 , A t +1 =

Original: (1 − γ t +1 )
Revised: (1 − γ t +1 )

Original: A t + γ t
Revised: A t + γ t

Original: +

Revised: +

Original: A
Revised: A

Original: (cid:48)
Revised: (cid:48)

Original: t +1 and x t +1 =
Revised: t +1 and x t +1 =

Original: (1 − γ t +1 )
Revised: (1 − γ t +1 )

Original: A t
Revised: A t

Original: A t + x t + γ t

Original: [-+
-] A 
Revised: [-+
-] A t +

Original: (cid:48) t +1 A t + ˜ x t +1 .
Revised: [Equation] ˜ x t +1 .

Original: Therefore, by convexity,
Revised: Therefore, by convexity,

Original: [Equation]
Revised: [Equation]

Original: Subtracting f
Revised: Subtracting f

Original: ( x (cid:63) ) , multiplying by A t +1 and taking expectation, we have
Revised: ( x (cid:63) ) , multiplying by A t +1 and taking expectation, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where in the second inequality we substituted (28)
Revised: where in the second inequality we substituted (28).

Original: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Substituting back into (26) and combining with (27) gives
Revised: Substituting back into (26) and combining with (27) gives

Original: [Equation]
Revised: [Equation]

Original: Plugging the above bound on a t +1 (cid:104)
Revised: Plugging the above bound on a t +1 (cid:104)

Original: ˆ g t +1 , v t − x (cid:63) (cid:105) into (29), noting that γ t +1 λ t +
Revised: ˆ g t +1 , v t − x (cid:63) (cid:105) into (29), noting that γ t +1 λ t +

Original: [Equation]
Revised: [Equation]

Original: min { λ (cid:48) t +1 , λ t +1 } , and rearranging, we obtain
Revised: min { λ (cid:48) t +1 , λ t +1 } , and rearranging, we obtain

Original: [Equation]
Revised: [Equation]

Original: Iterating this bound and noting that A t ≤ A (cid:48) t for all t , we obtain
Revised: Iterating this bound and noting that A t ≤ A (cid:48) t for all t , we obtain

Original: [Equation]
Revised: [Equation]

Original: By our assumption that x (cid:63) is a minimizer of f , we have that E T is a nonegative random variable, and consequently the above display is a bound on the expectation of a nonegative random variable.
Revised: By our assumption that x (cid:63) is a minimizer of f , we have that E T is a nonegative random variable, and consequently the above display is a bound on the expectation of a nonegative random variable.

Original: (This is the reason we require x (cid:63) to be a minimizer of f ).
Revised: (This is the reason we require x (cid:63) to be a minimizer of f ).

Original: Therefore, by Markov’s inequality, the event
Revised: Therefore, by Markov’s inequality, the event

Original: [Equation]
Revised: [Equation]

Original: holds with probability at least 2/3, implying (13), except with A 0 E 0 + D 0 multiplied by a factor of 3 .
Revised: holds with probability at least 2/3, implying (13), except with A 0 E 0 + D 0 multiplied by a factor of 3 .

Original: Moreover, the growth bounds √ A T ≥ 12 (cid:80) t ∈S ≤ T 1 / (cid:112) λ (cid:48) t holds deterministically as a consequence of the update rule for A t .
Revised: Moreover, the growth bounds √ A T ≥ 12 (cid:80) t ∈S ≤ T 1 / (cid:112) λ (cid:48) t holds deterministically as a consequence of the update rule for A t .

Original: These two facts sufﬁce to establish the growth rate of A T precisely as we do in the proof of Theorem 1, thereby obtaining the same rate of convergence (up to a constant).
Revised: These two facts sufﬁce to establish the growth rate of A T precisely as we do in the proof of Theorem 1, thereby obtaining the same rate of convergence (up to a constant).

Original: D Proofs for Section
Revised: D Proofs for Section

Original: This section contains the analysis of our adaptive oracle implementations (Algorithms 2 and 3).
Revised: This section contains the analysis of our adaptive oracle implementations (Algorithms 2 and 3).

Original: We begin by quickly showing how an idealized regularized Newton step adaptively yields optimal movement bounds without need to know the degree or order of the Hessian Hölder continuity (Appendix D.1).
Revised: We begin by quickly showing how an idealized regularized Newton step adaptively yields optimal movement bounds without need to know the degree or order of the Hessian Hölder continuity (Appendix D.1).

Original: Then, we prove movement bound and complexity guarantees for our second-order and ﬁrst-order adaptive oracle implementations in Appendices D.2 and D.3, respectively.
Revised: Then, we prove movement bound and complexity guarantees for our second-order and ﬁrst-order adaptive oracle implementations in Appendices D.2 and D.3, respectively.

Original: Finally, we provide auxiliary results used throughout the preceding proofs (Appendix D.4).
Revised: Finally, we provide auxiliary results used throughout the preceding proofs (Appendix D.4).

Original: D.1 A movement bound for the ideal Newton step
Revised: D.1 A movement bound for the ideal Newton step

Original: The following proposition shows how choosing the smallest λ for which a λ -regularized Newton step satisﬁes the MS condition yields movement bounds adaptive to Hessian Hölder continuity.
Revised: The following proposition shows how choosing the smallest λ for which a λ -regularized Newton step satisﬁes the MS condition yields movement bounds adaptive to Hessian Hölder continuity.

Original: Proposition 2.
Revised: Proposition 2.

Original: For any y ∈ R d and σ ∈ (0 , 1) , let λ (cid:63) be the smallest λ for which the λ -regularized
Revised: For any y ∈ R d and σ ∈ (0 , 1) , let λ (cid:63) be the smallest λ for which the λ -regularized

Original: Newton step x λ = y − [ ∇ 2 f ( y )+ λ (cid:63) I ] − 1 ∇ f ( y ) satisﬁes the MS condition (cid:107) x
Revised: Newton step x λ = y − [ ∇ 2 f ( y )+ λ (cid:63) I ] − 1 ∇ f ( y ) satisﬁes the MS condition (cid:107) x

Original: λ − ( y − 1 λ ∇ f ( x )) (cid:107) ≤ σ (cid:107) x
Revised: λ − ( y − 1 λ ∇ f ( x )) (cid:107) ≤ σ (cid:107) x

Original: λ − y (cid:107) .
Revised: λ − y (cid:107) .

Original: If ∇ 2 f is ( H, ν ) -Hölder continuous for any ν ∈ [0 , 1] , the triplet ( x λ (cid:63) , y, λ (cid:63) ) satisﬁes a (cid:16) 1 + ν, (cid:0) H 2 σ (cid:1) 1 / (1+ ν ) (cid:17)
Revised: If ∇ 2 f is ( H, ν ) -Hölder continuous for any ν ∈ [0 , 1] , the triplet ( x λ (cid:63) , y, λ (cid:63) ) satisﬁes a (cid:16) 1 + ν, (cid:0) H 2 σ (cid:1) 1 / (1+ ν ) (cid:17)

Original: -movement bound.
Revised: -movement bound.

Original: Proof.
Revised: Proof.

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: That is (˜ x, ˜ λ ) =
Revised: That is (˜ x, ˜ λ ) =

Original: O 2 ,ν -reg ( y ) with parameter M = H/σ .
Revised: O 2 ,ν -reg ( y ) with parameter M = H/σ .

Original: Note that (a) ˜ x, y and ˜ λ satisfy the MS condition as explained in Section 3.1, and (b) ˜ x = y − [ ∇ 2 f ( y ) + ˜ λI ] − 1 ∇ f ( y ) .
Revised: Note that (a) ˜ x, y and ˜ λ satisfy the MS condition as explained in Section 3.1, and (b) ˜ x = y − [ ∇ 2 f ( y ) + ˜ λI ] − 1 ∇ f ( y ) .

Original: Therefore the minimal λ (cid:63) satisfying the MS condition must satisfy λ (cid:63) ≤ ˜ λ
Revised: Therefore the minimal λ (cid:63) satisfying the MS condition must satisfy λ (cid:63) ≤ ˜ λ

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where ( (cid:63) ) is due to auxiliary Lemma 7.
Revised: where ( (cid:63) ) is due to auxiliary Lemma 7.

Original: This yields the movement bound.
Revised: This yields the movement bound.

Original: We remark that the same proof and movement bound hold for a slightly more relaxed notion of λ (cid:63) , namely the smallest for which the x λ satisﬁes the MS condition for all λ ≥ λ ∗ .
Revised: We remark that the same proof and movement bound hold for a slightly more relaxed notion of λ (cid:63) , namely the smallest for which the x λ satisﬁes the MS condition for all λ ≥ λ ∗ .

Original: This is the actual notion of λ (cid:63) that we approximate in the next subsections, where we ﬁnd λ such that x λ satisﬁes the
Revised: This is the actual notion of λ (cid:63) that we approximate in the next subsections, where we ﬁnd λ such that x λ satisﬁes the

Original: MS condition, but x λ/ 2 does not satisfy it (and therefore λ ≥ λ (cid:63) / 2 ).
Revised: MS condition, but x λ/ 2 does not satisfy it (and therefore λ ≥ λ (cid:63) / 2 ).

Original: D.2 Analysis of Algorithm
Revised: D.2 Analysis of Algorithm

Original: Theorem 2.
Revised: Theorem 2.

Original: Algorithm 2 with parameter σ is a σ -MS oracle O aMSN .
Revised: Algorithm 2 with parameter σ is a σ -MS oracle O aMSN .

Original: For any y ∈ R d , computing ( x, λ ) =
Revised: For any y ∈ R d , computing ( x, λ ) =

Original: O aMSN ( y ) requires at most 

Revised: O aMSN ( y ) requires at most

Original: + 2 log 2 (cid:0) 1 + (cid:12)(cid:12) log 2 λλ (cid:48)
Revised: + 2 log 2 (cid:0) 1 + (cid:12)(cid:12) log 2 λλ (cid:48)

Original: (cid:12)(cid:12)(cid:1) linear systems solutions.
Revised: (cid:12)(cid:12)(cid:1) linear systems solutions.

Original: If LAZY is
Revised: If LAZY is

Original: False or λ > λ (cid:48) , and if ∇ 2 f is ( H, ν ) -Hölder in a ball of radius 2 (cid:107) x − y (cid:107) around y , then ( x, y, λ ) satisfy a (cid:0) 1 + ν, (2 H/σ ) 1 / (1+ ν ) /σ (cid:1) -movement bound.
Revised: False or λ > λ (cid:48) , and if ∇ 2 f is ( H, ν ) -Hölder in a ball of radius 2 (cid:107) x − y (cid:107) around y , then ( x, y, λ ) satisfy a (cid:0) 1 + ν, (2 H/σ ) 1 / (1+ ν ) /σ (cid:1) -movement bound.

Original: Proof.
Revised: Proof.

Original: First, note that, by construction, the algorithm outputs a value of λ for which
Revised: First, note that, by construction, the algorithm outputs a value of λ for which

Original: C HECK MS ( λ ; y, σ ) evaluates to True, and is therefore a σ -MS oracle as per Deﬁnition 1.
Revised: C HECK MS ( λ ; y, σ ) evaluates to True, and is therefore a σ -MS oracle as per Deﬁnition 1.

Original: Next, let us bound the total number of linear system solutions in the algorithm, noting it is equal to the number of calls to C HECK MS.
Revised: Next, let us bound the total number of linear system solutions in the algorithm, noting it is equal to the number of calls to C HECK MS.

Original: The algorithm solves 1 linear system in line 1, and then solves k (cid:63) +linear systems in either the while-loop in line 5 or the while-loop in line 11.
Revised: The algorithm solves 1 linear system in line 1, and then solves k (cid:63) +linear systems in either the while-loop in line 5 or the while-loop in line 11.

Original: The algorithm then arrives at the while-loop in line 15 with two values λ vld and λ invld such that (a) C HECK MS ( λ vld ; y, σ ) is
Revised: The algorithm then arrives at the while-loop in line 15 with two values λ vld and λ invld such that (a) C HECK MS ( λ vld ; y, σ ) is

Original: True and C HECK MS ( λ invld ; y, σ ) is False and (b) λ vld /λ invld = 2 2 k(cid:63) .
Revised: True and C HECK MS ( λ invld ; y, σ ) is False and (b) λ vld /λ invld = 2 2 k(cid:63) .

Original: The while-loop maintains the invariant (a) while transforming λ vld /λ invld → (cid:112) λ vld /λ invld at each iteration.
Revised: The while-loop maintains the invariant (a) while transforming λ vld /λ invld → (cid:112) λ vld /λ invld at each iteration.

Original: After j iterations of the while-loop, we have
Revised: After j iterations of the while-loop, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, after precisely k (cid:63) iterations we obtain λ vld = 2 λ invld and the loop terminates.
Revised: Therefore, after precisely k (cid:63) iterations we obtain λ vld = 2 λ invld and the loop terminates.

Original: Hence, the overall number of linear system solutions is 1 + ( k (cid:63) + 1) + k (cid:63)
Revised: Hence, the overall number of linear system solutions is 1 + ( k (cid:63) + 1) + k (cid:63)

Original: = 2 + 2 k
Revised: = 2 + 2 k

Original: (cid:63) , or just 1 in case the ﬁrst
Revised: (cid:63) , or just 1 in case the ﬁrst

Original: C HECK MS is True and the algorithm is lazy.
Revised: C HECK MS is True and the algorithm is lazy.

Original: To bound k (cid:63) in terms of the input λ (cid:48) and output λ , consider the values of λ vld and λ invld before entering the while-loop at line 15.
Revised: To bound k (cid:63) in terms of the input λ (cid:48) and output λ , consider the values of λ vld and λ invld before entering the while-loop at line 15.

Original: First, note that λ invld ≤ λ ≤ λ vld .
Revised: First, note that λ invld ≤ λ ≤ λ vld .

Original: Second, assuming that λ > λ (cid:48) (i.e., the ﬁrst C HECK MS fails) we have
Revised: Second, assuming that λ > λ (cid:48) (i.e., the ﬁrst C HECK MS fails) we have

Original: [Equation]
Revised: [Equation]

Original: Combining these two facts, we have
Revised: Combining these two facts, we have

Original: [Equation]
Revised: [Equation]

Original: Rearranging this inequality yields k (cid:63) ≤ log 2 (cid:0) 1 + log 2 λλ (cid:48) (cid:1) as claimed.
Revised: Rearranging this inequality yields k (cid:63) ≤ log 2 (cid:0) 1 + log 2 λλ (cid:48) (cid:1) as claimed.

Original: The bound for λ <
Revised: The bound for λ <

Original: λ (cid:48) follows analogously.
Revised: λ (cid:48) follows analogously.

Original: We now turn to showing the movement bound assuming ∇ 2 f is ( H, ν ) -Hölder for ν ∈ [0 , 1] within a ball of radius 2 (cid:107) x − y (cid:107) around y , where x = y − ( ∇ 2 f ( y )
Revised: We now turn to showing the movement bound assuming ∇ 2 f is ( H, ν ) -Hölder for ν ∈ [0 , 1] within a ball of radius 2 (cid:107) x − y (cid:107) around y , where x = y − ( ∇ 2 f ( y )

Original: + λI ) − 1 ∇ f ( y ) is the output of the algorithm.
Revised: + λI ) − 1 ∇ f ( y ) is the output of the algorithm.

Original: Let λ 1 / 2 = λ/ 2 and x 1 / 2 = y − ( ∇ 2 f ( y ) + λ 1 / 2 I ) − 1 ∇ f ( y ) .
Revised: Let λ 1 / 2 = λ/ 2 and x 1 / 2 = y − ( ∇ 2 f ( y ) + λ 1 / 2 I ) − 1 ∇ f ( y ) .

Original: Let ˜ λ
Revised: Let ˜ λ

Original: and
Revised: and

Original: ˜ x = y − ( ∇ 2 f ( y ) + ˜ λI ) − 1 ∇ f ( y ) satisfy ˜ λ
Revised: ˜ x = y − ( ∇ 2 f ( y ) + ˜ λI ) − 1 ∇ f ( y ) satisfy ˜ λ

Original: = H 2 σ (cid:107) ˜ x − y (cid:107) ν , i.e.,
Revised: = H 2 σ (cid:107) ˜ x − y (cid:107) ν , i.e.,

Original: [Equation]
Revised: [Equation]

Original: argmin
Revised: argmin

Original: [Equation]
Revised: [Equation]

Original: It is immediate to see that for any λ > ˜ λ , C HECK MS ( λ ; y, σ ) must evaluate to True and thus the while loop in Line 11 is guaranteed to terminate.
Revised: It is immediate to see that for any λ > ˜ λ , C HECK MS ( λ ; y, σ ) must evaluate to True and thus the while loop in Line 11 is guaranteed to terminate.

Original: We will now argue that λ 1 / 2 < ˜ λ .
Revised: We will now argue that λ 1 / 2 < ˜ λ .

Original: First, note that λ 1 / 2 is the last value of λ invld before the while loop at line 15 terminates.
Revised: First, note that λ 1 / 2 is the last value of λ invld before the while loop at line 15 terminates.

Original: Therefore, C HECK MS ( λ 1 / 2 ; y, σ ) must evaluate to False.
Revised: Therefore, C HECK MS ( λ 1 / 2 ; y, σ ) must evaluate to False.

Original: Further, note that, by Lemma 7,
Revised: Further, note that, by Lemma 7,

Original: [Equation]
Revised: [Equation]

Original: Using the local Hessian assumption of Hölder continuity along with ( ∇ 2 f ( y ) + λ 1 / 2 I ) x 1 / 2 + ∇ f ( y ) = 0 , auxiliary Lemma 8 gives
Revised: Using the local Hessian assumption of Hölder continuity along with ( ∇ 2 f ( y ) + λ 1 / 2 I ) x 1 / 2 + ∇ f ( y ) = 0 , auxiliary Lemma 8 gives

Original: [Equation]
Revised: [Equation]

Original: with the ﬁnal equality using the deﬁnition of ˜ λ .
Revised: with the ﬁnal equality using the deﬁnition of ˜ λ .

Original: Assuming by contradiction that λ 1 / 2 ≥ ˜ λ , we have that
Revised: Assuming by contradiction that λ 1 / 2 ≥ ˜ λ , we have that

Original: [Equation]
Revised: [Equation]

Original: where the ﬁnal inequality used Lemma 7 combined with λ 1 / 2 ≥ ˜ λ to deduce that (cid:107) x 1 / 2 − y (cid:107) ≤ (cid:107) ˜ x − y (cid:107) .
Revised: where the ﬁnal inequality used Lemma 7 combined with λ 1 / 2 ≥ ˜ λ to deduce that (cid:107) x 1 / 2 − y (cid:107) ≤ (cid:107) ˜ x − y (cid:107) .

Original: This implies that C HECK MS ( λ 1 / 2 ; y, σ ) is True, giving a contradiction.
Revised: This implies that C HECK MS ( λ 1 / 2 ; y, σ ) is True, giving a contradiction.

Original: The (1 + ν, (2 H/σ ) 1 / (1+ ν ) )
Revised: The (1 + ν, (2 H/σ ) 1 / (1+ ν ) )

Original: -movement bound follows from
Revised: -movement bound follows from

Original: [Equation]
Revised: [Equation]

Original: ν σ, with the ﬁnal inequality using Lemma 7 combined with λ/ ˜ λ ≤ 2 , which implies (cid:107) ˜ x − y (cid:107) ≤ 2 (cid:107) x − y (cid:107) .
Revised: ν σ, with the ﬁnal inequality using Lemma 7 combined with λ/ ˜ λ ≤ 2 , which implies (cid:107) ˜ x − y (cid:107) ≤ 2 (cid:107) x − y (cid:107) .

Original: We remark that the termination of the while loop in line 1 (when LAZY is False) is, strictly speaking, not guaranteed.
Revised: We remark that the termination of the while loop in line 1 (when LAZY is False) is, strictly speaking, not guaranteed.

Original: For example, if the function f is quadratic, C HECK MS will always evaluate to
Revised: For example, if the function f is quadratic, C HECK MS will always evaluate to

Original: True.
Revised: True.

Original: However, it is also straightforward to verify that as long as C HECK MS is True for a given λ , the corresponding regularized Newton step x = y − ( ∇ 2 f ( y )
Revised: However, it is also straightforward to verify that as long as C HECK MS is True for a given λ , the corresponding regularized Newton step x = y − ( ∇ 2 f ( y )

Original: + λI ) − 1 ∇ f ( y ) has optimality gap bounded by λ (cid:107) x
Revised: + λI ) − 1 ∇ f ( y ) has optimality gap bounded by λ (cid:107) x

Original: − y (cid:107) 2 .
Revised: − y (cid:107) 2 .

Original: Therefore, since the loop in line 1 decreases λ at a double-exponential rate, if it fails to terminate after a small number of iterations then it means we have found an essentially optimal point.
Revised: Therefore, since the loop in line 1 decreases λ at a double-exponential rate, if it fails to terminate after a small number of iterations then it means we have found an essentially optimal point.

Original: Put differently, if we seek an (cid:15) suboptimal point, we may stop the loop in line 1 after
Revised: Put differently, if we seek an (cid:15) suboptimal point, we may stop the loop in line 1 after

Original: O (log log( λ (cid:48) 

Revised: O (log log( λ (cid:48)

Original: R 2 (cid:15) )) iterations.
Revised: R 2 (cid:15) )) iterations.

Original: We account for this possibility in the complexity bound below.
Revised: We account for this possibility in the complexity bound below.

Original: Corollary 3.
Revised: Corollary 3.

Original: Consider Algorithm 1 with initial point x 0 , parameters α satisfying 1 .
Revised: Consider Algorithm 1 with initial point x 0 , parameters α satisfying 1 .

Original: 1 ≤ α =
Revised: 1 ≤ α =

Original: O (1) and λ (cid:48) 0 , and σ -MS oracle O aMSN (with LAZY =
Revised: O (1) and λ (cid:48) 0 , and σ -MS oracle O aMSN (with LAZY =

Original: True in all but the ﬁrst iteration) with σ ∈ (0 . 01 , 0 . 99) .
Revised: True in all but the ﬁrst iteration) with σ ∈ (0 . 01 , 0 . 99) .

Original: For any H, (cid:15) > 0 , ν ∈ [0 , 1] and any x (cid:63) ∈ R d , if f is convex with ( H, ν ) -
Revised: For any H, (cid:15) > 0 , ν ∈ [0 , 1] and any x (cid:63) ∈ R d , if f is convex with ( H, ν ) -

Original: Hölder Hessian, the algorithm produces an iterate x T such that f ( x T ) ≤ f ( x (cid:63) )
Revised: Hölder Hessian, the algorithm produces an iterate x T such that f ( x T ) ≤ f ( x (cid:63) )

Original: + (cid:15) using T =
Revised: + (cid:15) using T =

Original: O (cid:16)(cid:0) H (cid:107) x 

Revised: O (cid:16)(cid:0) H (cid:107) x

Original: − x (cid:63) (cid:107) 2+ ν /(cid:15) (cid:1) 2 / (4+3 ν ) (cid:17) Hessian evaluations and O (cid:16) T + log log max (cid:110) HR ν λ (cid:48) 0 , λ (cid:48) 

Revised: − x (cid:63) (cid:107) 2+ ν /(cid:15) (cid:1) 2 / (4+3 ν ) (cid:17) Hessian evaluations and O (cid:16) T + log log max (cid:110) HR ν λ (cid:48) 0 , λ (cid:48)

Original: R 2 (cid:15) (cid:111)(cid:17) linear system solutions, where R is the distance between x 0 and argmin x (cid:48)
Revised: R 2 (cid:15) (cid:111)(cid:17) linear system solutions, where R is the distance between x 0 and argmin x (cid:48)

Original: f ( x (cid:48) ) .
Revised: f ( x (cid:48) ) .

Original: Proof.
Revised: Proof.

Original: Throughout the proof, we let T denote the index of the ﬁrst iteration of Algorithm 1 for which f ( x T ) ≤ f ( x (cid:63) )
Revised: Throughout the proof, we let T denote the index of the ﬁrst iteration of Algorithm 1 for which f ( x T ) ≤ f ( x (cid:63) )

Original: + (cid:15) .
Revised: + (cid:15) .

Original: The bound on Hessian evaluation complexity follows immediately from the validity of the MS-approximate proximal oracle and movement bounds guaranteed in Theorem 2, and the iteration bound given by Theorem 1, noting that each call to Algorithm 2 requires only one
Revised: The bound on Hessian evaluation complexity follows immediately from the validity of the MS-approximate proximal oracle and movement bounds guaranteed in Theorem 2, and the iteration bound given by Theorem 1, noting that each call to Algorithm 2 requires only one

Original: Hessian computation.
Revised: Hessian computation.

Original: To bound the total number of linear system solutions, we consider separately ( i ) the iteration where λ t = λ (cid:48) t , ( ii ) the iterations t ∈ [2 , T − 1] where λ t > λ (cid:48) t (i.e, those in S >T ), ( iii ) the ﬁrst iteration (note that λ t +1 < λ (cid:48)
Revised: To bound the total number of linear system solutions, we consider separately ( i ) the iteration where λ t = λ (cid:48) t , ( ii ) the iterations t ∈ [2 , T − 1] where λ t > λ (cid:48) t (i.e, those in S >T ), ( iii ) the ﬁrst iteration (note that λ t +1 < λ (cid:48)

Original: t +1 can only happen in the ﬁrst iteration since we are using a lazy oracle), and ( iv ) the last iteration.
Revised: t +1 can only happen in the ﬁrst iteration since we are using a lazy oracle), and ( iv ) the last iteration.

Original: Case ( i ) is easy, because at iterations where λ t +

Revised: Case ( i ) is easy, because at iterations where λ t +

Original: = λ (cid:48)
Revised: = λ (cid:48)

Original: t +1 Algorithm 2 requires at most 2 linear system solves by Theorem 2, and thereofore all such iterations combined require N ( i ) =
Revised: t +1 Algorithm 2 requires at most 2 linear system solves by Theorem 2, and thereofore all such iterations combined require N ( i ) =

Original: O ( T ) =
Revised: O ( T ) =

Original: O (cid:18)(cid:16)
Revised: O (cid:18)(cid:16)

Original: H (cid:107) x 0 − x (cid:63) (cid:107) 

Revised: H (cid:107) x 0 − x (cid:63) (cid:107)

Original: + ν (cid:15) (cid:17) 2 / (4+3 ν ) (cid:19) linear system solves.
Revised: + ν (cid:15) (cid:17) 2 / (4+3 ν ) (cid:19) linear system solves.

Original: To handle case ( ii ) , which requires the most work, we use Theorem 2 to bound the total number of linear system solves contributed by these iterates by
Revised: To handle case ( ii ) , which requires the most work, we use Theorem 2 to bound the total number of linear system solves contributed by these iterates by

Original: [Equation]
Revised: [Equation]

Original: which follows from ( a ) the complexity bound in Theorem 2; ( b ) the fact that a (1 + ν, O
Revised: which follows from ( a ) the complexity bound in Theorem 2; ( b ) the fact that a (1 + ν, O

Original: ( H 1 / (1+ ν ) )) - movement bound holds for every t ∈ S >T , meaning that λ
Revised: ( H 1 / (1+ ν ) )) - movement bound holds for every t ∈ S >T , meaning that λ

Original: = O ( H ) (cid:107) ˜
Revised: = O ( H ) (cid:107) ˜

Original: x
Revised: x

Original: t − y t (cid:107) ν ; ( c ) by the inequalities
Revised: t − y t (cid:107) ν ; ( c ) by the inequalities

Original: [Equation]
Revised: [Equation]

Original: for any z ≥ 1 and ﬁxed c ≥ 0 ; and ( d ) Lemma 9, using the assumption that f ( x T − 1 ) >
Revised: for any z ≥ 1 and ﬁxed c ≥ 0 ; and ( d ) Lemma 9, using the assumption that f ( x T − 1 ) >

Original: f ( x (cid:63) ) .
Revised: f ( x (cid:63) ) .

Original: Moreover, note that
Revised: Moreover, note that

Original: [Equation]
Revised: [Equation]

Original: by eq.
Revised: by eq.

Original: (9) in Proposition 1, which implies A T − 1 =
Revised: (9) in Proposition 1, which implies A T − 1 =

Original: O (
Revised: O (

Original: [Equation]
Revised: [Equation]

Original: x 0 − x
Revised: x 0 − x

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Substituting back into the bound on N ( ii ) , we obtain
Revised: Substituting back into the bound on N ( ii ) , we obtain

Original: [Equation]
Revised: [Equation]

Original: Therefore, perhaps surprisingly, the worst-case double-logarithmic per-iteration linear system complexity amortizes to a constant.
Revised: Therefore, perhaps surprisingly, the worst-case double-logarithmic per-iteration linear system complexity amortizes to a constant.

Original: To handle the last two edge cases, let z (cid:63) be the minimizer of f closest to x 0 , such that (cid:107) x 0 − z (cid:63) (cid:107)
Revised: To handle the last two edge cases, let z (cid:63) be the minimizer of f closest to x 0 , such that (cid:107) x 0 − z (cid:63) (cid:107)

Original: = R .
Revised: = R .

Original: In case ( iii ) , i.e., the number of linear system solves in the ﬁrst iteration.
Revised: In case ( iii ) , i.e., the number of linear system solves in the ﬁrst iteration.

Original: We consider separately the cases λ 1 ≥ λ (cid:48) 0 and λ 1 < λ (cid:48) 0 .
Revised: We consider separately the cases λ 1 ≥ λ (cid:48) 0 and λ 1 < λ (cid:48) 0 .

Original: In the former case, the movement bound guaranteed at the ﬁrst iteration yields (noting that y 

Revised: In the former case, the movement bound guaranteed at the ﬁrst iteration yields (noting that y

Original: = x 0 )
Revised: = x 0 )

Original: [Equation]
Revised: [Equation]

Original: where the last transition follows from Lemma 5 and the assumption f ( x 1 ) ≥
Revised: where the last transition follows from Lemma 5 and the assumption f ( x 1 ) ≥

Original: f
Revised: f

Original: ( z (cid:63) ) .
Revised: ( z (cid:63) ) .

Original: In the latter case ( λ 1 < λ (cid:48) 0 ), Lemma 5 and f ( x 1 ) − f
Revised: In the latter case ( λ 1 < λ (cid:48) 0 ), Lemma 5 and f ( x 1 ) − f

Original: ( z (cid:63) ) ≥ f ( x 1 ) − f ( x (cid:63) ) ≥ (cid:15) gives
Revised: ( z (cid:63) ) ≥ f ( x 1 ) − f ( x (cid:63) ) ≥ (cid:15) gives

Original: [Equation]
Revised: [Equation]

Original: Therefore, the number of linear system solutions at the ﬁrst iteration is
Revised: Therefore, the number of linear system solutions at the ﬁrst iteration is

Original: [Equation]
Revised: [Equation]

Original: Finally, we consider ( iv ) the last iteration t = T .
Revised: Finally, we consider ( iv ) the last iteration t = T .

Original: If T / ∈ S >T then there is nothing to consider, since it only contributes a single linear system solutions.
Revised: If T / ∈ S >T then there is nothing to consider, since it only contributes a single linear system solutions.

Original: If T ∈ S >T , however, we cannot treat it as in case ( ii ) , since we are not guaranteed that f ( x T ) ≥ f
Revised: If T ∈ S >T , however, we cannot treat it as in case ( ii ) , since we are not guaranteed that f ( x T ) ≥ f

Original: ( x (cid:63) ) .
Revised: ( x (cid:63) ) .

Original: However, we are guaranteed that f ( x T ) ≥ f
Revised: However, we are guaranteed that f ( x T ) ≥ f

Original: ( z (cid:63) ) .
Revised: ( z (cid:63) ) .

Original: Therefore, Lemma 9 allows us to conclude that
Revised: Therefore, Lemma 9 allows us to conclude that

Original: [Equation]
Revised: [Equation]

Original: Noting that A T
Revised: Noting that A T

Original: = O
Revised: = O

Original: ( A T − 1 ) (see Lemma 10) and A T − 1 =
Revised: ( A T − 1 ) (see Lemma 10) and A T − 1 =

Original: O ( R 2 /(cid:15) )
Revised: O ( R 2 /(cid:15) )

Original: (since (cid:15) < f ( x T − 1 ) − f ( x (cid:63) ) ≤ f ( x T − 1 ) − f
Revised: (since (cid:15) < f ( x T − 1 ) − f ( x (cid:63) ) ≤ f ( x T − 1 ) − f

Original: ( z (cid:63) ) ≤
Revised: ( z (cid:63) ) ≤

Original: R 2 2 A T − 1 ) we conclude that λ T λ (cid:48)
Revised: R 2 2 A T − 1 ) we conclude that λ T λ (cid:48)

Original: T
Revised: T

Original: = O (cid:16) HR 2+ ν (cid:15) (cid:17) .
Revised: = O (cid:16) HR 2+ ν (cid:15) (cid:17) .

Original: Since HR 2+ ν (cid:15) = HR ν λ (cid:48) 0 · λ (cid:48) 

Revised: Since HR 2+ ν (cid:15) = HR ν λ (cid:48) 0 · λ (cid:48)

Original: R 2 (cid:15) ≤ max (cid:110) HR ν λ (cid:48) 0 , λ (cid:48) 0 R 2 (cid:15) (cid:111) 2 , we conclude that the O (cid:16) log log λ T λ (cid:48) T (cid:17) contribution of case ( iv ) to the total number of linear systems is no greater than our bound for case ( iii ) .
Revised: R 2 (cid:15) ≤ max (cid:110) HR ν λ (cid:48) 0 , λ (cid:48) 0 R 2 (cid:15) (cid:111) 2 , we conclude that the O (cid:16) log log λ T λ (cid:48) T (cid:17) contribution of case ( iv ) to the total number of linear systems is no greater than our bound for case ( iii ) .

Original: We remark that departing from a normal bisection or doubling scheme we have used a “doublelogarithmic scale” in the while loops starting at Line 5 and Line 11 in Algorithm 2.
Revised: We remark that departing from a normal bisection or doubling scheme we have used a “doublelogarithmic scale” in the while loops starting at Line 5 and Line 11 in Algorithm 2.

Original: As shown in the proof above, this doesn’t affect the complexity bounds shown for case ( ii ) , but gives better complexity bounds in the analysis of case ( iii ) and ( iv ) .
Revised: As shown in the proof above, this doesn’t affect the complexity bounds shown for case ( ii ) , but gives better complexity bounds in the analysis of case ( iii ) and ( iv ) .

Original: Eventually this allows us to only have an additive double-logarithmic term in the ﬁnal complexity of linear system solves as stated in Corollary 3.
Revised: Eventually this allows us to only have an additive double-logarithmic term in the ﬁnal complexity of linear system solves as stated in Corollary 3.

Original: D.3 Analysis of Algorithm
Revised: D.3 Analysis of Algorithm

Original: Theorem 4.
Revised: Theorem 4.

Original: Algorithm 3 with parameter σ is a σ -MS oracle O aMSN-fo .
Revised: Algorithm 3 with parameter σ is a σ -MS oracle O aMSN-fo .

Original: For any y ∈ R d , computing ( x, λ ) =
Revised: For any y ∈ R d , computing ( x, λ ) =

Original: O aMSN-fo ( y ) requires at most O (cid:16)(cid:113) 

Revised: O aMSN-fo ( y ) requires at most O (cid:16)(cid:113)

Original: + (cid:107)∇ 2 f ( y ) (cid:107) op σ min { λ,λ (cid:48) } (cid:17) Hessian-vector product and
Revised: + (cid:107)∇ 2 f ( y ) (cid:107) op σ min { λ,λ (cid:48) } (cid:17) Hessian-vector product and

Original: O ( (cid:12)(cid:12) log λλ (cid:48) (cid:12)(cid:12) )
Revised: O ( (cid:12)(cid:12) log λλ (cid:48) (cid:12)(cid:12) )

Original: gradient computations.
Revised: gradient computations.

Original: If LAZY is False or λ > λ (cid:48) , and if ∇ 2 f is ( H, ν ) -Hölder, then ( x, y, λ ) satisfy a (cid:0) 1 + ν, (6 H/σ ) 1 / (1+ ν ) (cid:1) -movement bound.
Revised: If LAZY is False or λ > λ (cid:48) , and if ∇ 2 f is ( H, ν ) -Hölder, then ( x, y, λ ) satisfy a (cid:0) 1 + ν, (6 H/σ ) 1 / (1+ ν ) (cid:1) -movement bound.

Original: Proof.
Revised: Proof.

Original: Clearly, Algorithm 3 can only return a pair x, λ satisfying Deﬁnition 1.
Revised: Clearly, Algorithm 3 can only return a pair x, λ satisfying Deﬁnition 1.

Original: (At this point, we are not guaranteed that the algorithm ever returns.
Revised: (At this point, we are not guaranteed that the algorithm ever returns.

Original: However, below we prove that the check in Linesucceeds for ﬁnite
Revised: However, below we prove that the check in Linesucceeds for ﬁnite

Original: λ
Revised: λ

Original: as long as the Hessian is continuous).
Revised: as long as the Hessian is continuous).

Original: To analyze the complexity of the algorithm, we begin by noting that it approximates a Newton step with MinRes at most (cid:12)(cid:12) log 2 λλ (cid:48) (cid:12)(cid:12) + 1 times: when λ ≥ λ (cid:48) we approximate Newton steps for values for regularization parameters of the form 2 k λ (cid:48) for k = 0 , 1 , . . .
Revised: To analyze the complexity of the algorithm, we begin by noting that it approximates a Newton step with MinRes at most (cid:12)(cid:12) log 2 λλ (cid:48) (cid:12)(cid:12) + 1 times: when λ ≥ λ (cid:48) we approximate Newton steps for values for regularization parameters of the form 2 k λ (cid:48) for k = 0 , 1 , . . .

Original: , log 2 λλ (cid:48) ; when λ < λ (cid:48)
Revised: , log 2 λλ (cid:48) ; when λ < λ (cid:48)

Original: we instead consider regularization parameters of the form 2 − k λ (cid:48) for k = 0 , 1 , . .
Revised: we instead consider regularization parameters of the form 2 − k λ (cid:48) for k = 0 , 1 , . .

Original: . , log 2 λλ (cid:48)
Revised: . , log 2 λλ (cid:48)

Original: + 1 .
Revised: + 1 .

Original: These considerations immediately yield our claimed O (cid:0)(cid:12)(cid:12) log 2 λλ (cid:48)
Revised: These considerations immediately yield our claimed O (cid:0)(cid:12)(cid:12) log 2 λλ (cid:48)

Original: (cid:12)(cid:12)(cid:1) bound on the number of gradients evaluated by the algorithm.
Revised: (cid:12)(cid:12)(cid:1) bound on the number of gradients evaluated by the algorithm.

Original: To bound the total Hessian-vector product complexity, suppose that Algorithm 3 attempts to approximate a Newton step with regularization parameter λ k ; we argue that the corresponding terminates in O (cid:0)(cid:113) (cid:107)∇ 2 f ( y ) (cid:107) op + λ k λ k σ (cid:1) Hessian-vector products.
Revised: To bound the total Hessian-vector product complexity, suppose that Algorithm 3 attempts to approximate a Newton step with regularization parameter λ k ; we argue that the corresponding terminates in O (cid:0)(cid:113) (cid:107)∇ 2 f ( y ) (cid:107) op + λ k λ k σ (cid:1) Hessian-vector products.

Original: Let A = ∇ 2 f ( y )
Revised: Let A = ∇ 2 f ( y )

Original: +
Revised: +

Original: λ k
Revised: λ k

Original: I , b = −∇ f ( y ) and w (cid:63) =
Revised: I , b = −∇ f ( y ) and w (cid:63) =

Original: A − 1 b .
Revised: A − 1 b .

Original: Lemma 6 guarantees that (cid:107) r t (cid:107) =
Revised: Lemma 6 guarantees that (cid:107) r t (cid:107) =

Original: O (cid:0) (cid:107) A (cid:107) op (cid:107) w (cid:63) (cid:107) /t
Revised: O (cid:0) (cid:107) A (cid:107) op (cid:107) w (cid:63) (cid:107) /t

Original: (cid:1) .
Revised: (cid:1) .

Original: Consequently, after T λ
Revised: Consequently, after T λ

Original: = O (cid:0)(cid:113) (cid:107) A (cid:107) op
Revised: = O (cid:0)(cid:113) (cid:107) A (cid:107) op

Original: λ
Revised: λ

Original: k σ (cid:1) steps we have (cid:107) r T λ (cid:107) ≤ λ k σ 4 (cid:107) w (cid:63) (cid:107) .
Revised: k σ (cid:1) steps we have (cid:107) r T λ (cid:107) ≤ λ k σ 4 (cid:107) w (cid:63) (cid:107) .