Original: Since h ( x ) = 12 x (cid:62) Ax − b (cid:62) x is λ k -strongly-convex and r i = ∇ h ( w i ) , we have λ k (cid:107) w T λ − w (cid:63) (cid:107) ≤ (cid:107) r T
Revised: Since h ( x ) = 12 x (cid:62) Ax − b (cid:62) x is λ k -strongly-convex and r i = ∇ h ( w i ) , we have λ k (cid:107) w T λ − w (cid:63) (cid:107) ≤ (cid:107) r T

Original: λ (cid:107) , and consequently (cid:107) w T λ − w (cid:63) (cid:107) ≤ σ 4 (cid:107) w (cid:63) (cid:107) ≤ 14 (cid:107) w (cid:63) (cid:107) .
Revised: λ (cid:107) , and consequently (cid:107) w T λ − w (cid:63) (cid:107) ≤ σ 4 (cid:107) w (cid:63) (cid:107) ≤ 14 (cid:107) w (cid:63) (cid:107) .

Original: Since (cid:107) w (cid:63) (cid:107) − (cid:107) w T λ (cid:107) ≤ (cid:107) w T λ − w (cid:63) (cid:107) by the triangle inequality, we conclude that (cid:107) w (cid:63) (cid:107) ≤ 43 (cid:107) w T λ (cid:107) .
Revised: Since (cid:107) w (cid:63) (cid:107) − (cid:107) w T λ (cid:107) ≤ (cid:107) w T λ − w (cid:63) (cid:107) by the triangle inequality, we conclude that (cid:107) w (cid:63) (cid:107) ≤ 43 (cid:107) w T λ (cid:107) .

Original: Substituting back yields (cid:107) r T λ (cid:107) ≤ λ k σ 3 (cid:107) w T λ (cid:107) , and consequently the while-loop must terminate in T λ
Revised: Substituting back yields (cid:107) r T λ (cid:107) ≤ λ k σ 3 (cid:107) w T λ (cid:107) , and consequently the while-loop must terminate in T λ

Original: = O (cid:0)(cid:113)
Revised: = O (cid:0)(cid:113)

Original: (cid:107)∇ 2 f ( y ) (cid:107) op + λ k λ k σ (cid:1) steps, with each step corresponding to a single Hessian-vector product.
Revised: (cid:107)∇ 2 f ( y ) (cid:107) op + λ k λ k σ (cid:1) steps, with each step corresponding to a single Hessian-vector product.

Original: Next, we argue that for very large λ k we do not need to compute any (new) Hessian-vector product, since the while-loop terminates in one step.
Revised: Next, we argue that for very large λ k we do not need to compute any (new) Hessian-vector product, since the while-loop terminates in one step.

Original: More speciﬁcally that, λ k ≥ 4 (cid:107)∇ 2 f ( y ) (cid:107) op σ the while-loop terminates after one step, i.e., (cid:107) r 1 (cid:107) ≤ λ k σ 2 (cid:107) w 1 (cid:107) .
Revised: More speciﬁcally that, λ k ≥ 4 (cid:107)∇ 2 f ( y ) (cid:107) op σ the while-loop terminates after one step, i.e., (cid:107) r 1 (cid:107) ≤ λ k σ 2 (cid:107) w 1 (cid:107) .

Original: To see this, ﬁrst observe that (since w 1 has the smallest residual among all vectors w proportional to b ), we have (cid:107) r 1 (cid:107) ≤ (cid:107)
Revised: To see this, ﬁrst observe that (since w 1 has the smallest residual among all vectors w proportional to b ), we have (cid:107) r 1 (cid:107) ≤ (cid:107)

Original: Ab/λ k − b (cid:107) =
Revised: Ab/λ k − b (cid:107) =

Original: 1 λ k
Revised: 1 λ k

Original: (cid:107)∇ 2 f ( y ) b (cid:107) ≤ (cid:107)∇ 2 f ( y ) (cid:107) op λ k (cid:107) b (cid:107) ≤ σ 4 (cid:107) b (cid:107) .
Revised: (cid:107)∇ 2 f ( y ) b (cid:107) ≤ (cid:107)∇ 2 f ( y ) (cid:107) op λ k (cid:107) b (cid:107) ≤ σ 4 (cid:107) b (cid:107) .

Original: Moreover, since w 1 = b (cid:62) Ab b (cid:62) A 2 b b , we have (cid:107) w 1 (cid:107) ≥ 1 λ
Revised: Moreover, since w 1 = b (cid:62) Ab b (cid:62) A 2 b b , we have (cid:107) w 1 (cid:107) ≥ 1 λ

Original: [Equation]
Revised: [Equation]

Original: + (cid:107)∇ 2 f ( y ) (cid:107) op (cid:107) b (cid:107) ≥ 45 λ k (cid:107) b (cid:107) .
Revised: + (cid:107)∇ 2 f ( y ) (cid:107) op (cid:107) b (cid:107) ≥ 45 λ k (cid:107) b (cid:107) .

Original: Consequently, (cid:107) r 1 (cid:107) ≤ 5 λ k σ 16 (cid:107) w 1 (cid:107) , meaning that the while-loop terminates after the ﬁrst iterate.
Revised: Consequently, (cid:107) r 1 (cid:107) ≤ 5 λ k σ 16 (cid:107) w 1 (cid:107) , meaning that the while-loop terminates after the ﬁrst iterate.

Original: Therefore, for λ k ≥ 4 (cid:107)∇ 2 f ( y ) (cid:107) op σ no Hessian-vector product computations are necessary (since the ones before the while-loop can be computed once for all k ).
Revised: Therefore, for λ k ≥ 4 (cid:107)∇ 2 f ( y ) (cid:107) op σ no Hessian-vector product computations are necessary (since the ones before the while-loop can be computed once for all k ).

Original: Let λ min be the smallest value of λ k encountered by the algorithm, and note that λ min ≥ min { λ/ 2 , λ (cid:48) } , and that λ k = 2 k λ min for k = 0 , . . .
Revised: Let λ min be the smallest value of λ k encountered by the algorithm, and note that λ min ≥ min { λ/ 2 , λ (cid:48) } , and that λ k = 2 k λ min for k = 0 , . . .

Original: , O (cid:0)(cid:12)(cid:12) log λ λ (cid:48)
Revised: , O (cid:0)(cid:12)(cid:12) log λ λ (cid:48)

Original: (cid:12)(cid:12)(cid:1) .
Revised: (cid:12)(cid:12)(cid:1) .

Original: By the discussion above, we require Hessian-vector product computations only at the ﬁrst K
Revised: By the discussion above, we require Hessian-vector product computations only at the ﬁrst K

Original: = O (log (cid:107)∇ 2 f ( y ) (cid:107) op σλ min ) iterations.
Revised: = O (log (cid:107)∇ 2 f ( y ) (cid:107) op σλ min ) iterations.

Original: Therefore the total number of Hessian-vector products is O (cid:18)(cid:80)
Revised: Therefore the total number of Hessian-vector products is O (cid:18)(cid:80)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: 1 +
Revised: 1 +

Original: [Equation]
Revised: [Equation]

Original: λ min (cid:19) =
Revised: λ min (cid:19) =

Original: O (cid:16)(cid:113) 

Revised: O (cid:16)(cid:113)

Original: + (cid:107)∇ 2 f ( y ) (cid:107) op σ min { λ,λ (cid:48) } (cid:17) , giving the claimed bound on Hessian-vector product count.
Revised: + (cid:107)∇ 2 f ( y ) (cid:107) op σ min { λ,λ (cid:48) } (cid:17) , giving the claimed bound on Hessian-vector product count.

Original: Next, we assume that f is has an ( H, ν ) -Hölder Hessian and argue that the algorithm’s output x, λ satisﬁes a movement bound (unless LAZY is True and λ = λ (cid:48) ).
Revised: Next, we assume that f is has an ( H, ν ) -Hölder Hessian and argue that the algorithm’s output x, λ satisﬁes a movement bound (unless LAZY is True and λ = λ (cid:48) ).

Original: To do so, we ﬁrst establish an upper bound on the returned λ .
Revised: To do so, we ﬁrst establish an upper bound on the returned λ .

Original: Let ˜
Revised: Let ˜

Original: [Equation]
Revised: [Equation]

Original: = argmin
Revised: = argmin

Original: [Equation]
Revised: [Equation]

Original: and note that ˜ w = − ( ∇ 2 f ( y ) + ˜ λI ) − 1 ∇ f ( y ) for ˜ λ
Revised: and note that ˜ w = − ( ∇ 2 f ( y ) + ˜ λI ) − 1 ∇ f ( y ) for ˜ λ

Original: = Hσ (cid:107) ˜ w (cid:107) ν .
Revised: = Hσ (cid:107) ˜ w (cid:107) ν .

Original: Let us show that the MS condition check in line 5 must succeed when for λ k ≥ ˜ λ .
Revised: Let us show that the MS condition check in line 5 must succeed when for λ k ≥ ˜ λ .

Original: Let w (cid:63) = − ( ∇ 2 f ( y ) +
Revised: Let w (cid:63) = − ( ∇ 2 f ( y ) +

Original: λ k I ) − 1 ∇ f ( y ) denote the exact regularized Newton step corresponding to λ k , let w i be the output of the corresponding MinRes run, and let r
Revised: λ k I ) − 1 ∇ f ( y ) denote the exact regularized Newton step corresponding to λ k , let w i be the output of the corresponding MinRes run, and let r

Original: [Equation]
Revised: [Equation]

Original: be the corresponding residual.
Revised: be the corresponding residual.

Original: Note that, by Lemma 8 we have
Revised: Note that, by Lemma 8 we have

Original: [Equation]
Revised: [Equation]

Original: Moreover
Revised: Moreover

Original: [Equation]
Revised: [Equation]

Original: 1+ ν , and
Revised: 1+ ν , and

Original: [Equation]
Revised: [Equation]

Original: due to ( i ) Lemma 6 and ( ii ) Lemma 7 and the fact that λ k ≥ ˜ λ .
Revised: due to ( i ) Lemma 6 and ( ii ) Lemma 7 and the fact that λ k ≥ ˜ λ .

Original: Therefore, for ν ∈ [0 , 1] we have H 2 (cid:107) w i (cid:107) 1+ ν ≤ λ k σ 2 (cid:107) w i (cid:107) , and (cid:107) r
Revised: Therefore, for ν ∈ [0 , 1] we have H 2 (cid:107) w i (cid:107) 1+ ν ≤ λ k σ 2 (cid:107) w i (cid:107) , and (cid:107) r

Original: i (cid:107) ≤ λ k σ 2 (cid:107) w i (cid:107) admits an identical bound by the MinRes termination condition.
Revised: i (cid:107) ≤ λ k σ 2 (cid:107) w i (cid:107) admits an identical bound by the MinRes termination condition.

Original: Substituting back into (30), we conclude that
Revised: Substituting back into (30), we conclude that

Original: [Equation]
Revised: [Equation]

Original: and consequently reaching λ k ≥ ˜ λ ensures that the MS condition holds.
Revised: and consequently reaching λ k ≥ ˜ λ ensures that the MS condition holds.

Original: However, the algorithm returns a value of λ such that for λ k
Revised: However, the algorithm returns a value of λ such that for λ k

Original: = λ/ 2 the MS condition check fails, meaning that λ/ 2 < ˜ λ .
Revised: = λ/ 2 the MS condition check fails, meaning that λ/ 2 < ˜ λ .

Original: It remains to argue that when Algorithm 3 returns with λ ≤ 2˜ λ , an appropriate movement bound holds.
Revised: It remains to argue that when Algorithm 3 returns with λ ≤ 2˜ λ , an appropriate movement bound holds.

Original: To that end, recall that (by λ -strong convexity of the quadratic subproblem)
Revised: To that end, recall that (by λ -strong convexity of the quadratic subproblem)

Original: [Equation]
Revised: [Equation]

Original: and consequently
Revised: and consequently

Original: [Equation]
Revised: [Equation]

Original: Moreover, since λ ≤ 2˜ λ , we have
Revised: Moreover, since λ ≤ 2˜ λ , we have

Original: [Equation]
Revised: [Equation]

Original: due to ( i ) Lemma 7 and λ ≤ 2˜ λ , ( ii ) the deﬁnition of ˜ λ
Revised: due to ( i ) Lemma 7 and λ ≤ 2˜ λ , ( ii ) the deﬁnition of ˜ λ

Original: and ( iii ) λ ≤ 2˜ λ again.
Revised: and ( iii ) λ ≤ 2˜ λ again.

Original: Combining the last two displays, we have
Revised: Combining the last two displays, we have

Original: [Equation]
Revised: [Equation]

Original: for c =
Revised: for c =

Original: [Equation]
Revised: [Equation]

Original: σ)
Revised: σ

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: as required.
Revised: as required.

Original: Corollary 5.
Revised: Corollary 5.

Original: Consider Algorithm 1 with initial point x 0 , parameters α satisfying 1 .
Revised: Consider Algorithm 1 with initial point x 0 , parameters α satisfying 1 .

Original: 1 ≤ α = O (1) and λ (cid:48) 0 , and σ -MS oracle O aMSN-fo with LAZY set to True in all but the ﬁrst iteration and σ ∈ (0 . 01 , 0 . 99) .
Revised: 1 ≤ α = O (1) and λ (cid:48) 0 , and σ -MS oracle O aMSN-fo with LAZY set to True in all but the ﬁrst iteration and σ ∈ (0 . 01 , 0 . 99) .

Original: For any L, H, (cid:15) > 0 , ν ∈ [0 , 1] and any x (cid:63) ∈ R d , if f is convex with ( H, ν ) -Hölder Hessian and L -Lipschitz gradient, the algorithm produces an iterate x T such that f ( x T ) ≤ f ( x (cid:63) )
Revised: For any L, H, (cid:15) > 0 , ν ∈ [0 , 1] and any x (cid:63) ∈ R d , if f is convex with ( H, ν ) -Hölder Hessian and L -Lipschitz gradient, the algorithm produces an iterate x T such that f ( x T ) ≤ f ( x (cid:63) )

Original: + (cid:15) within T
Revised: + (cid:15) within T

Original: = O (cid:18)(cid:16)
Revised: = O (cid:18)(cid:16)

Original: H (cid:107) x 0 − x (cid:63) (cid:107) 

Revised: H (cid:107) x 0 − x (cid:63) (cid:107)

Original: + ν (cid:15)
Revised: + ν (cid:15)

Original: [Equation]
Revised: [Equation]

Original: 2 / (4+3 ν ) (cid:19) iterations and at most
Revised: 2 / (4+3 ν ) (cid:19) iterations and at most

Original: O (cid:18)(cid:16) L (cid:107) x 0 −
Revised: O (cid:18)(cid:16) L (cid:107) x 0 −

Original: [Table]
Revised: [Table]

Original: Proof.
Revised: Proof.

Original: Throughout the proof, we let T denote the index of the ﬁrst iteration of Algorithm 1 for which f ( x T ) ≤ f ( x (cid:63) )
Revised: Throughout the proof, we let T denote the index of the ﬁrst iteration of Algorithm 1 for which f ( x T ) ≤ f ( x (cid:63) )

Original: + (cid:15) ; the claimed bound on total number of iterations is an immediate corollary of both Theorem 1 and Theorem 4.
Revised: + (cid:15) ; the claimed bound on total number of iterations is an immediate corollary of both Theorem 1 and Theorem 4.

Original: We now bound the complexity of Algorithm 3 with an approach similar to the proof of Corollary 3.
Revised: We now bound the complexity of Algorithm 3 with an approach similar to the proof of Corollary 3.

Original: To do so, we categorize all iterations into the following two cases: ( i ) t > 1 and (since LAZY is true)
Revised: To do so, we categorize all iterations into the following two cases: ( i ) t > 1 and (since LAZY is true)

Original: λ t ≥ λ (cid:48) t or ( ii ) the ﬁrst iteration t = 1 .
Revised: λ t ≥ λ (cid:48) t or ( ii ) the ﬁrst iteration t = 1 .

Original: Now using Theorem 4, we know that in case ( i ) the number of Hessian-vector product and gradient evaluations is bounded by
Revised: Now using Theorem 4, we know that in case ( i ) the number of Hessian-vector product and gradient evaluations is bounded by

Original: [Equation]
Revised: [Equation]

Original: where we use ( a ) that (cid:107)∇ 2 f ( y t ) (cid:107) op ≤ L by the assumption of L -Lipschitz gradient, and that either λ t = λ (cid:48) t or λ t
Revised: where we use ( a ) that (cid:107)∇ 2 f ( y t ) (cid:107) op ≤ L by the assumption of L -Lipschitz gradient, and that either λ t = λ (cid:48) t or λ t

Original: = O ( L ) by the movement bound guaranteed from Theorem 4 (since LLipschitz gradient means ( L, 0)
Revised: = O ( L ) by the movement bound guaranteed from Theorem 4 (since LLipschitz gradient means ( L, 0)

Original: -Lipschitz Hessian), and ( b ) the bound (16) from Lemma 1.
Revised: -Lipschitz Hessian), and ( b ) the bound (16) from Lemma 1.

Original: Note that T
Revised: Note that T

Original: =
Revised: =

Original: O (cid:18)(cid:113) L (cid:107) x
Revised: O (cid:18)(cid:113) L (cid:107) x

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:15) (cid:19) due to the iteration count bound for ( L, 0) -Hölder Hessian.
Revised: (cid:15) (cid:19) due to the iteration count bound for ( L, 0) -Hölder Hessian.

Original: Moreover, noting that A T =
Revised: Moreover, noting that A T =

Original: O ( A T − 1 ) by Lemma 10 and that A T − 1 =
Revised: O ( A T − 1 ) by Lemma 10 and that A T − 1 =

Original: O ( (cid:107) x 0 − x (cid:63) (cid:107) 2 /(cid:15) ) as argued in the proof of Corollary 3, we have √ LA T
Revised: O ( (cid:107) x 0 − x (cid:63) (cid:107) 2 /(cid:15) ) as argued in the proof of Corollary 3, we have √ LA T

Original: = O (cid:18)(cid:113)
Revised: = O (cid:18)(cid:113)

Original: L
Revised: L

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: as well.
Revised: as well.

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: For case ( ii ) , Theorem 4 gives allows us to bound the number of ﬁrst-order operations by
Revised: For case ( ii ) , Theorem 4 gives allows us to bound the number of ﬁrst-order operations by

Original: N ( ii ) =
Revised: N ( ii ) =

Original: O (cid:32)(cid:115)
Revised: O (cid:32)(cid:115)

Original: [Equation]
Revised: [Equation]

Original: = O
Revised: = O

Revised: [Equation]

Revised: (cid:114)

Original: [Equation]
Revised: [Equation]

Original: where for the last equality we use log( λ (cid:48) 0 /λ 1 ) ≤ log( λ (cid:48) 

Revised: where for the last equality we use log( λ (cid:48) 0 /λ 1 ) ≤ log( λ (cid:48)

Original: /L )
Revised: /L )

Original: + log( L/λ 1 ) and the bounds (cid:113)
Revised: + log( L/λ 1 ) and the bounds (cid:113)

Original: Lλ 1 =
Revised: Lλ 1 =

Original: O ( √ LA 1 ) =
Revised: O ( √ LA 1 ) =

Original: O (cid:18)(cid:113) L (cid:107) x 0 − x (cid:63) (cid:107) 

Revised: O (cid:18)(cid:113) L (cid:107) x 0 − x (cid:63) (cid:107)

Original: (cid:15) (cid:19) and λ
Revised: (cid:15) (cid:19) and λ

Original: 1 =
Revised: 1 =

Original: O ( L ) as in previous case.
Revised: O ( L ) as in previous case.

Original: Summing up the N ( i ) and
Revised: Summing up the N ( i ) and

Original: N ( ii ) yields the the claimed bound.
Revised: N ( ii ) yields the the claimed bound.

Original: D.4 Auxiliary results
Revised: D.4 Auxiliary results

Original: Here, we list technical results invoked throughout the section.
Revised: Here, we list technical results invoked throughout the section.

Original: Lemma 5.
Revised: Lemma 5.

Original: Let f : R d → R d be convex, and suppose that for some σ ∈ (0 , 1) , x, y ∈ R d and λ >the MS condition
Revised: Let f : R d → R d be convex, and suppose that for some σ ∈ (0 , 1) , x, y ∈ R d and λ >the MS condition

Original: [Equation]
Revised: [Equation]

Original: holds.
Revised: holds.

Original: Then, for any x (cid:63) ∈ R d , we have
Revised: Then, for any x (cid:63) ∈ R d , we have

Original: [Equation]
Revised: [Equation]

Original: Therefore λ ≥ 2( f ( x ) − f ( x (cid:63) ))
Revised: Therefore λ ≥ 2( f ( x ) − f ( x (cid:63) ))

Original: (cid:107) x
Revised: (cid:107) x

Original: (cid:63) − y (cid:107)and, if f ( x ) ≥ f ( x (cid:63) ) , (cid:107) x − y (cid:107) ≤ 1 √ 1 − σ 2 (cid:107) x
Revised: (cid:63) − y (cid:107)and, if f ( x ) ≥ f ( x (cid:63) ) , (cid:107) x − y (cid:107) ≤ 1 √ 1 − σ 2 (cid:107) x

Original: (cid:63) − y (cid:107) .
Revised: (cid:63) − y (cid:107) .

Original: Proof.
Revised: Proof.

Original: Let F ( x
Revised: Let F ( x

Original: [Equation]
Revised: [Equation]

Original: ) =
Revised: ) =

Original: f ( x
Revised: f ( x

Original: [Equation]
Revised: [Equation]

Original: ) + λ
Revised: ) + λ

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: − y (cid:107) 2 .
Revised: − y (cid:107) 2 .

Original: Since F is λ -strongly convex, we have
Revised: Since F is λ -strongly convex, we have

Original: [Equation]
Revised: [Equation]

Original: for every x (cid:63) ∈ R d .
Revised: for every x (cid:63) ∈ R d .

Original: Moreover, the MS condition yields
Revised: Moreover, the MS condition yields

Original: [Equation]
Revised: [Equation]

Original: The lemma follows by substituting back and rearranging.
Revised: The lemma follows by substituting back and rearranging.

Original: Lemma 6.
Revised: Lemma 6.

Original: Let A ∈ R d × d be a positive deﬁnite symmetric matrix, let b ∈ R d , and let w (cid:63) =
Revised: Let A ∈ R d × d be a positive deﬁnite symmetric matrix, let b ∈ R d , and let w (cid:63) =

Original: A − 1 b .
Revised: A − 1 b .

Original: The iterates { w t } and residuals { r t =
Revised: The iterates { w t } and residuals { r t =

Original: Aw t − b } of the Conjugate Residuals/MinRes algorithm [43, 19] for minimizing (cid:107)
Revised: Aw t − b } of the Conjugate Residuals/MinRes algorithm [42, 18] for minimizing (cid:107)

Original: Aw − b (cid:107) satisfy
Revised: Aw − b (cid:107) satisfy

Original: (cid:107) w t (cid:107) is non-decreasing in t with (cid:107) w ∞ (cid:107)
Revised: (cid:107) w t (cid:107) is non-decreasing in t with (cid:107) w ∞ (cid:107)

Original: = (cid:107) w (cid:63) (cid:107) , 2.
Revised: = (cid:107) w (cid:63) (cid:107) , 2.

Original: (cid:107) r t (cid:107) is non-increasing in t with (cid:107) r ∞ (cid:107) = 0 , 3.
Revised: (cid:107) r t (cid:107) is non-increasing in t with (cid:107) r ∞ (cid:107) = 0 , 3.

Original: (cid:107) r t (cid:107) =
Revised: (cid:107) r t (cid:107) =

Original: O (cid:16) (cid:107) A (cid:107) op (cid:107) w (cid:63) (cid:107) t(cid:17) .
Revised: O (cid:16) (cid:107) A (cid:107) op (cid:107) w (cid:63) (cid:107) t(cid:17) .

Original: Proof.
Revised: Proof.

Original: The ﬁrst two parts of the lemma are Theorems 2.3 and 2.4 of Fong and Saunders [19], respectively.
Revised: The ﬁrst two parts of the lemma are Theorems 2.3 and 2.4 of Fong and Saunders [18], respectively.

Original: To show the third part, we cite Lee et al.
Revised: To show the third part, we cite Lee et al.

Original: [29] which give a gradient method that, for any L -smooth convex function h with minimizer x (cid:63) , produces iterates x t such that (cid:107)∇ h ( x t ) (cid:107) =
Revised: [28] which give a gradient method that, for any L -smooth convex function h with minimizer x (cid:63) , produces iterates x t such that (cid:107)∇ h ( x t ) (cid:107) =

Original: O ( L (cid:107) x
Revised: O ( L (cid:107) x

Original: (cid:63) − x 0 (cid:107) /t 2 ) [29, Corollary 1].
Revised: (cid:63) − x 0 (cid:107) /t 2 ) [28, Corollary 1].

Original: Applying this method to h ( x ) = 12 x (cid:62) Ax − b (cid:62) x with x 0 = 0 , which is convex and (cid:107) A (cid:107) op -smooth with minimizer w (cid:63) , guarantees (cid:107)
Revised: Applying this method to h ( x ) = 12 x (cid:62) Ax − b (cid:62) x with x 0 = 0 , which is convex and (cid:107) A (cid:107) op -smooth with minimizer w (cid:63) , guarantees (cid:107)

Original: Ax t − b (cid:107) =
Revised: Ax t − b (cid:107) =

Original: (cid:107)∇ h ( x t ) (cid:107) =
Revised: (cid:107)∇ h ( x t ) (cid:107) =

Original: O (cid:0) (cid:107) A (cid:107) op (cid:107) w (cid:63) (cid:107) /t
Revised: O (cid:0) (cid:107) A (cid:107) op (cid:107) w (cid:63) (cid:107) /t

Original: (cid:1) .
Revised: (cid:1) .

Original: Moreover, we note that x t is in the linear span of ∇ h (0) , ∇ h ( x 1 ) , . . .
Revised: Moreover, we note that x t is in the linear span of ∇ h (0) , ∇ h ( x 1 ) , . . .

Original: ∇ h ( x t − 1 ) and consequently in the Krylov subspace span( b, Ab, . . .
Revised: ∇ h ( x t − 1 ) and consequently in the Krylov subspace span( b, Ab, . . .

Original: , A t − 1 b ) .
Revised: , A t − 1 b ) .

Original: Therefore (cid:107) r t (cid:107) ≤ (cid:107)
Revised: Therefore (cid:107) r t (cid:107) ≤ (cid:107)

Original: Ax t − b (cid:107) by deﬁnition of the Conjugate Residuals/MinRes method.
Revised: Ax t − b (cid:107) by deﬁnition of the Conjugate Residuals/MinRes method.

Original: Lemma 7.
Revised: Lemma 7.

Original: Let A ∈ R d × d be positive semideﬁnite, let b ∈ R d and let ∆( λ )
Revised: Let A ∈ R d × d be positive semideﬁnite, let b ∈ R d and let ∆( λ )

Original: =
Revised: =

Original: (cid:107) ( A + λI ) − 1 b (cid:107) .
Revised: (cid:107) ( A + λI ) − 1 b (cid:107) .

Original: Then, for any λ 1 ≤ λ 2 we have
Revised: Then, for any λ 1 ≤ λ 2 we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: This is an immediate consequence of
Revised: This is an immediate consequence of

Original: [Equation]
Revised: [Equation]

Original: 2 I and the fact that if 0 ≺ M 1 (cid:22) M 2 and M 1 , M 2 have the same eigenvectors, then (cid:107) M − 1 1 b (cid:107) ≥ (cid:107) M − 1 2 b (cid:107) for all b , since M 1 (cid:22) M 2 implies that the eigenvalues of M 2 majorize the eigenvalues of M 1 .
Revised: 2 I and the fact that if 0 ≺ M 1 (cid:22) M 2 and M 1 , M 2 have the same eigenvectors, then (cid:107) M − 1 1 b (cid:107) ≥ (cid:107) M − 1 2 b (cid:107) for all b , since M 1 (cid:22) M 2 implies that the eigenvalues of M 2 majorize the eigenvalues of M 1 .

Original: Lemma 8.
Revised: Lemma 8.

Original: Let f : R d → R and x, y ∈ R d , and suppose that ∇ 2 f is ( H, ν ) -Hölder continuous for some ν ∈ [0 , 1] in a ball of radius (cid:107) x − y (cid:107) around y .
Revised: Let f : R d → R and x, y ∈ R d , and suppose that ∇ 2 f is ( H, ν ) -Hölder continuous for some ν ∈ [0 , 1] in a ball of radius (cid:107) x − y (cid:107) around y .

Original: Then, for any λ ,
Revised: Then, for any λ ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: The local H -Hölder continuity of ∇ 2 f around y yields
Revised: The local H -Hölder continuity of ∇ 2 f around y yields

Original: [Equation]
Revised: [Equation]

Original: Moreover, for
Revised: Moreover, for

Original: [Equation]
Revised: [Equation]

Original: algebraic manipulation yields
Revised: algebraic manipulation yields

Original: [Equation]
Revised: [Equation]

Original: and the lemma holds via the triangle inequality.
Revised: and the lemma holds via the triangle inequality.

Original: For the following lemma, recall the notation S >T = { t ≤ T | λ t > λ (cid:48) t } .
Revised: For the following lemma, recall the notation S >T = { t ≤ T | λ t > λ (cid:48) t } .

Original: Lemma 9.
Revised: Lemma 9.

Original: For every T > 0 , ν ∈ [0 , 1] and x (cid:63) ∈ R d such that f ( x T ) ≥ f ( x (cid:63) ) , the iterates of
Revised: For every T > 0 , ν ∈ [0 , 1] and x (cid:63) ∈ R d such that f ( x T ) ≥ f ( x (cid:63) ) , the iterates of

Original: Algorithm 1 with σ ∈ (0 . 01 , 0 . 99) and α ∈ (1 . 01 , O (1)) satisfy
Revised: Algorithm 1 with σ ∈ (0 . 01 , 0 . 99) and α ∈ (1 . 01 , O (1)) satisfy

Original: [Equation]
Revised: [Equation]

Original: >T
Revised: >T

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The case ν = 0 follows immediately from eq.
Revised: The case ν = 0 follows immediately from eq.

Original: (15) in Lemma 1.
Revised: (15) in Lemma 1.

Original: For ν ∈ (0 , 1] , deﬁne u t =
Revised: For ν ∈ (0 , 1] , deﬁne u t =

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: { t ∈S >T } and v t = ( λ (cid:48) t ) 

Revised: { t ∈S >T } and v t = ( λ (cid:48) t )

Original: +2 /ν
Revised: +2 /ν

Original: A t .
Revised: A t .

Original: The reverse Hölder inequality gives, for any q > 1 ,  (cid:88) t ≤ T
Revised: The reverse Hölder inequality gives, for any q > 1 ,  (cid:88) t ≤ T

Original: [Equation]
Revised: [Equation]

Original: Substituting back the deﬁnitions of u t and v t we have
Revised: Substituting back the deﬁnitions of u t and v t we have

Original: [Equation]
Revised: [Equation]

Original: with the last transition due to eq.
Revised: with the last transition due to eq.

Original: (9) in Proposition 1 and the fact that f ( x T ) ≥ f ( x (cid:63) ) .
Revised: (9) in Proposition 1 and the fact that f ( x T ) ≥ f ( x (cid:63) ) .

Original: Next, we substitute q = 4+3 ν ν and note that
Revised: Next, we substitute q = 4+3 ν ν and note that

Original: [Equation]
Revised: [Equation]

Original:   with the last transition due to
Revised:   with the last transition due to

Original: [Equation]
Revised: [Equation]

Original: for all t (cid:48) by eq.
Revised: for all t (cid:48) by eq.

Original: (16) in Lemma 1.
Revised: (16) in Lemma 1.

Original: Note that for every non-decreasing sequence 0 = B 0 ≤ B 1 ≤
Revised: Note that for every non-decreasing sequence 0 = B 0 ≤ B 1 ≤

Original: B 2 ≤ · · · ≤ B T we have (cid:88)
Revised: B 2 ≤ · · · ≤ B T we have (cid:88)

Original: [Equation]
Revised: [Equation]

Original: B
Revised: B

Original: [Equation]
Revised: [Equation]

Original: Substituting B t = (cid:80) j ≤ t ( λ (cid:48)
Revised: Substituting B t = (cid:80) j ≤ t ( λ (cid:48)

Original: j ) − 1 / 2 , we have (cid:88) t ≤ T
Revised: j ) − 1 / 2 , we have (cid:88) t ≤ T

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: = O
Revised: = O

Original: [Equation]
Revised: [Equation]

Original: t ≤ T
Revised: t ≤ T

Original: [Equation]
Revised: [Equation]

Original: with the ﬁnal bound again using (33).
Revised: with the ﬁnal bound again using (33).

Original: This implies
Revised: This implies

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Substituting q
Revised: Substituting q

Original: =
Revised: =

Original: (4 + 3 ν ) /ν
Revised: (4 + 3 ν ) /ν

Original: and the bounds (34) and (32) into (31) completes the proof.
Revised: and the bounds (34) and (32) into (31) completes the proof.

Original: Lemma 10.
Revised: Lemma 10.

Original: For every T > 0 , the sequence { A t } in Algorithm 1 with α ∈ (1 . 01 , O (1)) satisﬁes
Revised: For every T > 0 , the sequence { A t } in Algorithm 1 with α ∈ (1 . 01 , O (1)) satisﬁes

Original: A t +1 =
Revised: A t +1 =

Original: O ( A t ) .
Revised: O ( A t ) .

Original: Proof.
Revised: Proof.

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: and therefore (since A (cid:48) t +1 > A t )
Revised: and therefore (since A (cid:48) t +1 > A t )

Original: [Equation]
Revised: [Equation]

Original: due to ( i ) λ (cid:48) t +1 ≥ λ (cid:48) t /α =
Revised: due to ( i ) λ (cid:48) t +1 ≥ λ (cid:48) t /α =

Original: Ω( λ (cid:48) t ) by the algorithm’s construction and ( ii ) 1 √ λ
Revised: Ω( λ (cid:48) t ) by the algorithm’s construction and ( ii ) 1 √ λ

Original: [Equation]
Revised: [Equation]

Original: t =
Revised: t =

Original: O ( √ A t ) by the last inequality in Lemma 1.
Revised: O ( √ A t ) by the last inequality in Lemma 1.

Original: The proof is complete by noting that A t +1 ≤ A (cid:48) t +1 .
Revised: The proof is complete by noting that A t +1 ≤ A (cid:48) t +1 .

Original: E Experiments
Revised: E Experiments

Original: This section provides the full details of the experiment we report in Section 4 (in Appendix E.1), as well as results of the additional experiments: algorithm comparison across additional datasets (Appendix E.2), parameter sensitivity of our algorithm (Appendix E.3), effect of changing the parameter M in O cr (Appendix E.4), and the effect of momentum on the worst-case instance for
Revised: This section provides the full details of the experiment we report in Section 4 (in Appendix E.1), as well as results of the additional experiments: algorithm comparison across additional datasets (Appendix E.2), parameter sensitivity of our algorithm (Appendix E.3), effect of changing the parameter M in O cr (Appendix E.4), and the effect of momentum on the worst-case instance for

Original: Lipschitz-Hessian functions (Appendix E.5).
Revised: Lipschitz-Hessian functions (Appendix E.5).

Original: Finally, we also demonstrate empirically the importance of the momentum damping mechanism in Algorithm 1 (Appendix E.6).
Revised: Finally, we also demonstrate empirically the importance of the momentum damping mechanism in Algorithm 1 (Appendix E.6).

Original: E.1 Main experiment details
Revised: E.1 Main experiment details

Original: We report experiments for logistic regression objectives of the form
Revised: We report experiments for logistic regression objectives of the form

Original: [Equation]
Revised: [Equation]

Original: , where each φ i ∈ R
Revised: , where each φ i ∈ R

Original: d is a feature vector with a corresponding label c i ∈ {− 1 , 1 } .
Revised: d is a feature vector with a corresponding label c i ∈ {− 1 , 1 } .

Original: Implementation details.
Revised: Implementation details.

Original: We now provide the key implementation details for all
Revised: We now provide the key implementation details for all

Original: the algorithms considered in our experiments.
Revised: the algorithms considered in our experiments.

Original: For a complete description please refer to the Python implementations submitted with this manuscript.
Revised: For a complete description please refer to the Python implementations submitted with this manuscript.

Original: • Algorithm 1.
Revised: • Algorithm 1.

Original: Our implementation of Algorithm 1 follows its pseudocode precisely.
Revised: Our implementation of Algorithm 1 follows its pseudocode precisely.

Original: We keep λ (cid:48) 

Revised: We keep λ (cid:48)

Original: = 0 .
Revised: = 0 .

Original: 1 throughout and set α = 2 in all experiments except for those in Appendix E.3, where we test how changing it affects performance.
Revised: 1 throughout and set α = 2 in all experiments except for those in Appendix E.3, where we test how changing it affects performance.

Original: • Algorithm 0 [33].
Revised: • Algorithm 0 [32].

Original: A direct implementation of the pseudocode of Algorithm 0 would be quite inefﬁcient, since stating off the bisection with a large interval [ λ (cid:96) , λ h ] at each iterations will waste many oracle calls.
Revised: A direct implementation of the pseudocode of Algorithm 0 would be quite inefﬁcient, since stating off the bisection with a large interval [ λ (cid:96) , λ h ] at each iterations will waste many oracle calls.

Original: Instead, we implement a strong baseline for our bisection-free algorithm by starting each bisection with a guess λ 0 t +1 determined by the previous iterations.
Revised: Instead, we implement a strong baseline for our bisection-free algorithm by starting each bisection with a guess λ 0 t +1 determined by the previous iterations.

Original: We construct this guess using the scheme 8 for updating λ (cid:48) t in Algorithm 1: if the previous ﬁnal bisection output λ t and the previous initial bisection guess λ 0 t satisfy λ t > λ 0 t , we let λ 0 t +1 = 2 λ 

Revised: We construct this guess using the scheme 8 for updating λ (cid:48) t in Algorithm 1: if the previous ﬁnal bisection output λ t and the previous initial bisection guess λ 0 t satisfy λ t > λ 0 t , we let λ 0 t +1 = 2 λ

Original: and otherwise we set λ 0 t +1 = 12 λ 0 t .
Revised: and otherwise we set λ 0 t +1 = 12 λ 0 t .

Original: We take λ 01 = 0 . 1 .
Revised: We take λ 01 = 0 . 1 .

Original: To construct a bisection interval out of the initial guess λ 0 t
Revised: To construct a bisection interval out of the initial guess λ 0 t

Original: +1 , we adopt a strategy similar to the ones used in Algorithms 2 and 3.
Revised: +1 , we adopt a strategy similar to the ones used in Algorithms 2 and 3.

Original: To explain it, deﬁne the following terminology.
Revised: To explain it, deﬁne the following terminology.

Original: Consider some λ (cid:48) t + and λ t +1 computed by applying an MS oracle to y t and λ (cid:48) t +1 , with y t computed from λ (cid:48)
Revised: Consider some λ (cid:48) t + and λ t +1 computed by applying an MS oracle to y t and λ (cid:48) t +1 , with y t computed from λ (cid:48)

Original: t +1 as in lines 5 and 8 of Algorithm 0.
Revised: t +1 as in lines 5 and 8 of Algorithm 0.

Original: We say that λ (cid:48)
Revised: We say that λ (cid:48)

Original: t +1 is valid if λ t +1 ∈ [ 1 ρ λ (cid:48)
Revised: t +1 is valid if λ t +1 ∈ [ 1 ρ λ (cid:48)

Original: t +1 , λ (cid:48)
Revised: t +1 , λ (cid:48)

Original: t +1 ] , that λ t +1 is high if λ t +1 <
Revised: t +1 ] , that λ t +1 is high if λ t +1 <

Original: 1 ρ λ (cid:48)
Revised: 1 ρ λ (cid:48)

Original: t +1 , and that λ (cid:48)
Revised: t +1 , and that λ (cid:48)

Original: t +1 is low if λ t +1 > λ (cid:48) t +1 .
Revised: t +1 is low if λ t +1 > λ (cid:48) t +1 .

Original: If λ 0 t +1 is valid, we simply use it and there is no need for bisection.
Revised: If λ 0 t +1 is valid, we simply use it and there is no need for bisection.

Original: Otherwise, if it is low, we take λ (cid:96)t
Revised: Otherwise, if it is low, we take λ (cid:96)t

Original: +

Revised: +

Original: = λ 0 t +1 , and repeatedly double λ 0 t + until we ﬁnd some 2 k λ 0 t
Revised: = λ 0 t +1 , and repeatedly double λ 0 t + until we ﬁnd some 2 k λ 0 t

Original: +1 that is either valid or high.
Revised: +1 that is either valid or high.

Original: In the former case we are again done, and in the latter case we set λ h t +

Revised: In the former case we are again done, and in the latter case we set λ h t +

Original: = 2 k λ 0 t
Revised: = 2 k λ 0 t

Original: +

Revised: +

Original: and continue with the bisection as described in Algorithm 0, except that (inspired by Algorithm 2) at each iteration we take λ (cid:48) t +1 to be the geometric mean of λ (cid:96)t +

Revised: and continue with the bisection as described in Algorithm 0, except that (inspired by Algorithm 2) at each iteration we take λ (cid:48) t +1 to be the geometric mean of λ (cid:96)t +

Original: and λ ht +1 rather than the arithmetic mean.
Revised: and λ ht +1 rather than the arithmetic mean.

Original: The case that λ 0 t +1 is high is treated analogously, setting λ ht
Revised: The case that λ 0 t +1 is high is treated analogously, setting λ ht

Original: +

Revised: +

Original: = λ 0 t
Revised: = λ 0 t

Original: +

Revised: +

Original: and
Revised: and

Original: repeatedly halving it until ﬁnding 2 − k λ 0 t +1 that is either valid or low.
Revised: repeatedly halving it until ﬁnding 2 − k λ 0 t +1 that is either valid or low.

Original: Finally, we note that for σ -MS oracles with σ > 0 the bisection is only guaranteed to succeed when ρ is sufﬁciently large.
Revised: Finally, we note that for σ -MS oracles with σ > 0 the bisection is only guaranteed to succeed when ρ is sufﬁciently large.

Original: The precise value of ρ depends on σ and the order of the movement bound guaranteed by the oracle.
Revised: The precise value of ρ depends on σ and the order of the movement bound guaranteed by the oracle.

Original: Instead of attempting a precise calculation, we set ρ = 4 .
Revised: Instead of attempting a precise calculation, we set ρ = 4 .

Original: • Cubic-regularized Newton Method (CR) [38].
Revised: • Cubic-regularized Newton Method (CR) [37].

Original: The method consists of simply iterating x t +

Revised: The method consists of simply iterating x t +

Original: =
Revised: =

Original: O cr ( x t ) .
Revised: O cr ( x t ) .

Original: For when the parameter M in O cr is set to M = 0 , the method reduces to the classical Newton’s method x t +1 = − [ ∇ 2 f ( x t )] − 1 ∇ f ( x t ) .
Revised: For when the parameter M in O cr is set to M = 0 , the method reduces to the classical Newton’s method x t +1 = − [ ∇ 2 f ( x t )] − 1 ∇ f ( x t ) .

Original: • Accelerated CR (ACR) [34].
Revised: • Accelerated CR (ACR) [33].

Original: We implement [34, Alg. 4.8] without changes.
Revised: We implement [33, Alg. 4.8] without changes.

Original: • Adaptive ACR 
Revised: • Adaptive ACR [21].

Original: We implement [22, Alg. 4] without changes.
Revised: We implement [21, Alg. 4] without changes.

Original: • Song et al.
Revised: • Song et al.

Original: [42] heuristic.
Revised: [41] heuristic.

Original: Following a proposal in [42], we consider a version of Algorithmthat uses a single pre-speciﬁed sequence of λ (cid:48) t without checking whether the resulting λ t is in the interval [ 1 ρ λ (cid:48) t , λ (cid:48) t ] .
Revised: Following a proposal in [41], we consider a version of Algorithmthat uses a single pre-speciﬁed sequence of λ (cid:48) t without checking whether the resulting λ t is in the interval [ 1 ρ λ (cid:48) t , λ (cid:48) t ] .

Original: We compute the sequence by setting A t =
Revised: We compute the sequence by setting A t =

Original: A (cid:48) t =
Revised: A (cid:48) t =

Original: 1 2 HR
Revised: 1 2 HR

Original: ( t/ 3) 7 / 2 , and taking λ (cid:48) t +1 =
Revised: ( t/ 3) 7 / 2 , and taking λ (cid:48) t +1 =

Original: A t +1 a 2 t +

Revised: A t +1 a 2 t +

Original: =
Revised: =

Original: A t +1 ( A t +1 − A t ) 2 .
Revised: A t +1 ( A t +1 − A t ) 2 .

Original: Here H is an estimate of the function’s Hessian Lipschitz constant (see below), and R is an estimate of the Euclidean distance between x 0 from an optimal point.
Revised: Here H is an estimate of the function’s Hessian Lipschitz constant (see below), and R is an estimate of the Euclidean distance between x 0 from an optimal point.

Original: We obtain R by using the default scikit-learn logistic regression solver [40]; it ﬁnds a far less accurate solution than the methods we consider but provides a reasonably accurate estimate of R .
Revised: We obtain R by using the default scikit-learn logistic regression solver [39]; it ﬁnds a far less accurate solution than the methods we consider but provides a reasonably accurate estimate of R .

Original: • Iterating O
Revised: • Iterating O

Original: aMSN .
Revised: aMSN .

Original: This scheme corresponds to simply iterating x t +1 , λ t +

Revised: This scheme corresponds to simply iterating x t +1 , λ t +

Original: =
Revised: =

Original: O aMSN ( x t , λ t / 2) , with the initial λ 1 set to 0 .
Revised: O aMSN ( x t , λ t / 2) , with the initial λ 1 set to 0 .

Original: • Iterating O aMSN-fo .
Revised: • Iterating O aMSN-fo .

Original: This scheme corresponds to simply iterating x t +1 , λ t +1 =
Revised: This scheme corresponds to simply iterating x t +1 , λ t +1 =

Original: O aMSN-fo ( x t , λ t / 2) , with the initial λ 1 set to 0 .
Revised: O aMSN-fo ( x t , λ t / 2) , with the initial λ 1 set to 0 .

Original: • Gradient descent (GD).
Revised: • Gradient descent (GD).

Original: We iterate x t +1 = x
Revised: We iterate x t +1 = x

Original: t − η ∇ f ( x t ) and choose the best value of η from { 3 , 10 , 30 , 100 , 1000 , 3000 } , making sure the best value is never on the edge of the grid, i.e., 3 and 3000 are never chosen.
Revised: t − η ∇ f ( x t ) and choose the best value of η from { 3 , 10 , 30 , 100 , 1000 , 3000 } , making sure the best value is never on the edge of the grid, i.e., 3 and 3000 are never chosen.

Original: • Accelerated gradient descent (AGD) [39].
Revised: • Accelerated gradient descent (AGD) [38].

Original: We implement the algorithm precisely as described in [39], and tune the step size η as described for GD.
Revised: We implement the algorithm precisely as described in [38], and tune the step size η as described for GD.

Original: • L-BFGS-B [10, 45].
Revised: • L-BFGS-B [10, 44].

Original: We use the implementation available from SciPy [44], where we set all tolerance parameter to a very small value so that the algorithm only stops after exceeding the speciﬁed maximum number of iterations.
Revised: We use the implementation available from SciPy [43], where we set all tolerance parameter to a very small value so that the algorithm only stops after exceeding the speciﬁed maximum number of iterations.

Original: • O cr .
Revised: • O cr .

Original: To solve the problem (1) and implement O cr , we perform a bisection over λ to solve for λ that satisﬁes λ ≈ M 2 (cid:107) [ ∇ 2 f ( y )
Revised: To solve the problem (1) and implement O cr , we perform a bisection over λ to solve for λ that satisﬁes λ ≈ M 2 (cid:107) [ ∇ 2 f ( y )

Original: + λI ] − 1 ∇ f ( y ) (cid:107) , and return x = y − [ ∇ 2 f ( y )
Revised: + λI ] − 1 ∇ f ( y ) (cid:107) , and return x = y − [ ∇ 2 f ( y )

Original: + λI ] − 1 ∇ f ( y ) .
Revised: + λI ] − 1 ∇ f ( y ) .

Original: To ensure a high-quality solution to the implicit equation for λ , we stop the bisection only when λ M 2 (cid:107) [ ∇ 2 f ( y )
Revised: To ensure a high-quality solution to the implicit equation for λ , we stop the bisection only when λ M 2 (cid:107) [ ∇ 2 f ( y )

Original: + λI ] − 

Revised: + λI ] −

Original: ∇ f ( y ) (cid:107) ∈
Revised: ∇ f ( y ) (cid:107) ∈

Original: [1 − 10 − 5 , 1 + 10 − 5 ] .
Revised: [1 − 10 − 5 , 1 + 10 − 5 ] .

Original: This results in a slow implementation of O cr (requiring a lot of linear system solutions), but provides the ideal point of comparison since we measure complexity by number of Hessian evaluations.
Revised: This results in a slow implementation of O cr (requiring a lot of linear system solutions), but provides the ideal point of comparison since we measure complexity by number of Hessian evaluations.

Original: To ensure numerical stability, we also stop the bisection if the value of λ falls below λ Newton = 10 − 10 .
Revised: To ensure numerical stability, we also stop the bisection if the value of λ falls below λ Newton = 10 − 10 .

Original: • O aMSN .
Revised: • O aMSN .

Original: Our implementation follows the pseudocode of Algorithm 2 precisely, except that, to ensure numerical stability, we stop the procedure if λ falls below λ Newton = 10 − 10 .
Revised: Our implementation follows the pseudocode of Algorithm 2 precisely, except that, to ensure numerical stability, we stop the procedure if λ falls below λ Newton = 10 − 10 .

Original: When combining the oracle with Algorithm 1, we set the LAZY to be True in all iterations except the ﬁrst, as in Corollary 3.
Revised: When combining the oracle with Algorithm 1, we set the LAZY to be True in all iterations except the ﬁrst, as in Corollary 3.

Original: In all other settings we set LAZY to be False.
Revised: In all other settings we set LAZY to be False.

Original: • O aMSN-fo .
Revised: • O aMSN-fo .

Original: Our implementation follows the pseudocode of Algorithm 3, except we only implement the case that LAZY =
Revised: Our implementation follows the pseudocode of Algorithm 3, except we only implement the case that LAZY =

Original: True, since doing otherwise appears less practical; it is not hard to extend Theorem 1 and Corollary 3 to provide similar guarantees even when the ﬁrst iteration is lazy.
Revised: True, since doing otherwise appears less practical; it is not hard to extend Theorem 1 and Corollary 3 to provide similar guarantees even when the ﬁrst iteration is lazy.

Original: Initialization.
Revised: Initialization.

Original: We initialize all algorithms at the origin, i.e., with x 0 = 0 .
Revised: We initialize all algorithms at the origin, i e., with x 0 = 0 .

Original: Estimating the Hessian Lipschitz constant.
Revised: Estimating the Hessian Lipschitz constant.

Original: For all algorithms that require an estimate for the
Revised: For all algorithms that require an estimate for the

Original: Lipschitz constant H of ∇ 2 f (i e., all the algorithms that use O cr ) we set H =
Revised: Lipschitz constant H of ∇ 2 f (i.e., all the algorithms that use O cr ), we set H =

Original: 110 ¯ H  where
Revised: 110 ¯ H , where

Original: (cid:107) 1 n (cid:80) ni
Revised: (cid:107) 1 n (cid:80) ni

Original: =1 φ i φ (cid:62) i (cid:107) op max i ∈ [ n ] (cid:107) φ i (cid:107) is a conservative upper bound on the Lipschitz constant of ∇ 2 f for logistic regression [
Revised: =1 φ i φ (cid:62) i (cid:107) op max i ∈ [ n ] (cid:107) φ i (cid:107) is a conservative upper bound on the Lipschitz constant of ∇ 2 f for logistic regression [

Original: We explore the effect of varying the
Revised: We explore the effect of varying the

Original: estimate H
Revised: estimate H

Original: in Appendix
Revised: in Appendix

Original: E
Revised: E.

Original: Note that value of M given to O cr is typically 2 H : for Algorithms 0 and 1 it is H/σ and σ = 2 , while CR and ACR also use M = 2 H .
Revised: Note that value of M given to O cr is typically 2 H : for Algorithms 0 and 1 it is H/σ and σ = 2 , while CR and ACR also use M = 2 H .

Original: We also use H as the initial guess for Adaptive ACR.
Revised: We also use H as the initial guess for Adaptive ACR.

Original: Datasets and preprocessing.
Revised: Datasets and preprocessing.

Original: We compare our methods to other baselines using the following binary classiﬁcation datasets:
Revised: We compare our methods to other baselines using the following binary classiﬁcation datasets:

Original: • a9a ( n = 32 , 561 and d = 123 )
Revised: • a9a ( n = 32 , 561 and d = 123 )

Original: • w8a ( n = 49 , 749 and d = 300 ) • splice ( n = 1 , 000 and d = 60 ) • synthetic ( n = 500 and d = 200 ).
Revised: • w8a ( n = 49 , 749 and d = 300 ) • splice ( n = 1 , 000 and d = 60 ) • synthetic ( n = 500 and d = 200 ).

Original: The ﬁrst three datasets are from LIBSVM [15], which is available under a BSD 3-Clause "New" or "Revised" license.
Revised: The ﬁrst three datasets are from LIBSVM [15], which is available under a BSD 3-Clause "New" or "Revised" license.

Original: The synthetic dataset is generated by sampling half of the data points from
Revised: The synthetic dataset is generated by sampling half of the data points from

Original: N 1 ( µ 1 , I ) and the other half from N 1 ( µ 2 , I ) , where µ 1 , µ 2 ∈ R
Revised: N 1 ( µ 1 , I ) and the other half from N 1 ( µ 2 , I ) , where µ 1 , µ 2 ∈ R

Original: d are independent random vectors uniformly drawn from a sphere with radius 0 . 5 .
Revised: d are independent random vectors uniformly drawn from a sphere with radius 0 . 5 .

Original: For all datasets we normalize the feature vectors, such that for every i ∈ [ n ] , each feature vector φ i is a unit norm.
Revised: For all datasets we normalize the feature vectors, such that for every i ∈ [ n ] , each feature vector φ i is a unit norm.

Original: E.2 Replicating Figure 1 with additional datasets
Revised: E.2 Replicating Figure 1 with additional datasets

Original: In Figure 2 we compare all the algorithms described above on logistic regression with the datasets: “a9a” (panels a-c), “w8a” (panels d-f), “splice” (panels g-i) and the synthetic dataset (panels j-l).
Revised: In Figure 2 we compare all the algorithms described above on logistic regression with the datasets: “a9a” (panels a-c), “w8a” (panels d-f), “splice” (panels g-i) and the synthetic dataset (panels j-l).

Original: Among non-adaptive methods (panels a, d, g and j), Algorithm 1 outperforms the other non-adaptive methods while Algorithm 0 is consistently the second best-performing method.
Revised: Among non-adaptive methods (panels a, d, g and j), Algorithm 1 outperforms the other non-adaptive methods while Algorithm 0 is consistently the second best-performing method.

Original: Comparing adaptive methods (panels b, e, h and k), we see that our implementation of Algorithmwith the adaptive oracle O aMSN converges faster than adaptive ACR and Algorithm 0 with O aMSN for all datasets.
Revised: Comparing adaptive methods (panels b, e, h and k), we see that our implementation of Algorithmwith the adaptive oracle O aMSN converges faster than adaptive ACR and Algorithm 0 with O aMSN for all datasets.

Original: However, our scheme that only iterates O aMSN without momentum converges even faster, and Newton’s method outperforms all second-order methods.
Revised: However, our scheme that only iterates O aMSN without momentum converges even faster, and Newton’s method outperforms all second-order methods.

Original: For ﬁrst-order methods (panels c, f, i and l), iterating O aMSN-fo scheme is comparable to L-BFGS-B on 2 out of 4 datasets and is faster than tuned AGD in 3 out of 4 datasets.
Revised: For ﬁrst-order methods (panels c, f, i and l), iterating O aMSN-fo scheme is comparable to L-BFGS-B on 2 out of 4 datasets and is faster than tuned AGD in 3 out of 4 datasets.

Original: On the synthetic dataset it is about twice slower than L-BFGS-B but still faster than tuned AGD, while on w8a it is about 50% slower than L-BFGS-B and tuned AGD, which perform comparably.
Revised: On the synthetic dataset it is about twice slower than L-BFGS-B but still faster than tuned AGD, while on w8a it is about 50% slower than L-BFGS-B and tuned AGD, which perform comparably.

Original: E.3 Parameter sensitivity of Algorithm
Revised: E.3 Parameter sensitivity of Algorithm

Original: We test the sensitivity of Algorithm 1 combined with our adaptive oracle (Algorithm 2 or Algorithm 3) to the parameters α and σ .
Revised: We test the sensitivity of Algorithm 1 combined with our adaptive oracle (Algorithm 2 or Algorithm 3) to the parameters α and σ .

Original: Figure 3 shows that Algorithm 1 second-order oracle O aMSN performs essentially the same for all α in the range 1 . 2 to 8 , and that the oracle’s performance is similar for σ = 0 .
Revised: Figure 3 shows that Algorithm 1 second-order oracle O aMSN performs essentially the same for all α in the range 1 . 2 to 8 , and that the oracle’s performance is similar for σ = 0 .

Original: 1 and σ = 0 .
Revised: 1 and σ = 0 .

Original: 25 , but slightly degrades for larger and smaller σ .
Revised: 25 , but slightly degrades for larger and smaller σ .

Original: Algorithm 1 combined with the ﬁrst-order oracle O aMSN-fo is a bit more sensitive to α (performing best for α = 1 . 2 ), but is less sensitive to σ , showing similar performance for all σ values except the very smallest σ = 0 .
Revised: Algorithm 1 combined with the ﬁrst-order oracle O aMSN-fo is a bit more sensitive to α (performing best for α = 1 . 2 ), but is less sensitive to σ , showing similar performance for all σ values except the very smallest σ = 0 .

Original: E.4 Varying M for O cr
Revised: E.4 Varying M for O cr

Original: In this section we test the performance of non-adaptive methods (i.e., the methods that use O cr ) when changing the estimate of the function’s Lipschitz constant H .
Revised: In this section we test the performance of non-adaptive methods (i.e., the methods that use O cr ) when changing the estimate of the function’s Lipschitz constant H .

Original: In particular, we consider values of H of the form β ¯ H , where ¯ H = (cid:107) 1 n (cid:80) n i =1 φ
Revised: In particular, we consider values of H of the form β ¯ H , where ¯ H = (cid:107) 1 n (cid:80) n i =1 φ

Original: i φ (cid:62) i (cid:107) op max i ∈ [ n ] (cid:107) φ i (cid:107) is an upper bound on the
Revised: i φ (cid:62) i (cid:107) op max i ∈ [ n ] (cid:107) φ i (cid:107) is an upper bound on the

Original: Hessian Lipschitz constant and β varies in (cid:8) 1 , 10 − 1 , 10 − 2 , 10 − 3 , 10 − 4 , 10 − 5 , 10 − 6 , 10 − 7 , 10 − 8 (cid:9) .
Revised: Hessian Lipschitz constant and β varies in (cid:8) 1 , 10 − 1 , 10 − 2 , 10 − 3 , 10 − 4 , 10 − 5 , 10 − 6 , 10 − 7 , 10 − 8 (cid:9) .

Original: (The experiments in Appendix E.2 correspond to β = 0 . 1 ).
Revised: (The experiments in Appendix E.2 correspond to β = 0 . 1 ).

Original: Figure 4 shows that our adaptive accelerated scheme (Algorithm 1 with the O aMSN ) outperforms all non-adaptive methods with their optimal H value, except for the CR method that has optimal H ≈ 0 and therefore is almost equivalent to Newton’s method.
Revised: Figure 4 shows that our adaptive accelerated scheme (Algorithm 1 with the O aMSN ) outperforms all non-adaptive methods with their optimal H value, except for the CR method that has optimal H ≈ 0 and therefore is almost equivalent to Newton’s method.

Original: E.5 Performance on a worst-case instance
Revised: E.5 Performance on a worst-case instance

Original: Having observed that our adaptive oracle O aMSN performs better on logistic regression without the “acceleration” scheme in Algorithm 1, we now test whether Algorithm 1 demonstrably accelerates
Revised: Having observed that our adaptive oracle O aMSN performs better on logistic regression without the “acceleration” scheme in Algorithm 1, we now test whether Algorithm 1 demonstrably accelerates

Original: O aMSN on a different, harder problem.
Revised: O aMSN on a different, harder problem.

Original: In particular, we consider the worst case instance [3, 22, 16] f : R d → R given by
Revised: In particular, we consider the worst case instance [3, 21, 16] f : R d → R given by

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: We note that, for t <
Revised: We note that, for t <

Original: d , the optimal rate of convergence for any of the algorithms we consider (which can only “discover” one coordinate of f per iterations)
Revised: d , the optimal rate of convergence for any of the algorithms we consider (which can only “discover” one coordinate of f per iterations)

Original: is O
Revised: is O

Original: ( t − 2 ) , or O (cid:16)(cid:13)(cid:13) x 0 − x ( t ) (cid:63) (cid:13)(cid:13) 3 t − 3 .
Revised: ( t − 2 ) , or O (cid:16)(cid:13)(cid:13) x 0 − x ( t ) (cid:63) (cid:13)(cid:13) 3 t − 3 .

Original: 5 (cid:17) where (cid:13)(cid:13) x 0 − x ( t ) (cid:63) (cid:13)(cid:13) =
Revised: 5 (cid:17) where (cid:13)(cid:13) x 0 − x ( t ) (cid:63) (cid:13)(cid:13) =

Original: Θ( √ t ) is the distance between the initial point and the best solution with only t non-zero coordinates.
Revised: Θ( √ t ) is the distance between the initial point and the best solution with only t non-zero coordinates.

Original: In our experiments, we set d = 3 , 000 and compare the convergence rate of the following secondorder methods: standard cubic regularized method (CR), its accelerated variant (ACR), Algorithmwith O cr , Algorithm 1 with the adaptive oracle O aMSN , and iterating the oracle O aMSN .
Revised: In our experiments, we set d = 3 , 000 and compare the convergence rate of the following secondorder methods: standard cubic regularized method (CR), its accelerated variant (ACR), Algorithmwith O cr , Algorithm 1 with the adaptive oracle O aMSN , and iterating the oracle O aMSN .

Original: For methods based on O cr we estimate the Hessian Lipschitz constant to be H = 10 .
Revised: For methods based on O cr we estimate the Hessian Lipschitz constant to be H = 10 .

Original: Figure 5 shows that the slope of the accelerated methods using O cr (ACR and Algorithm 1) is sharper than the slope of the CR method, indicating a faster convergence rate due to the acceleration scheme.
Revised: Figure 5 shows that the slope of the accelerated methods using O cr (ACR and Algorithm 1) is sharper than the slope of the CR method, indicating a faster convergence rate due to the acceleration scheme.

Original: However, the convergence rate O aMSN with and without the acceleration component is optimal.
Revised: However, the convergence rate O aMSN with and without the acceleration component is optimal.

Original: Therefore, even the worst-case instance for convex optimization with Lipschitz Hessian does not provide evidence that acceleration signiﬁcantly beneﬁts O aMSN .
Revised: Therefore, even the worst-case instance for convex optimization with Lipschitz Hessian does not provide evidence that acceleration signiﬁcantly beneﬁts O aMSN .

Original: E.6 The importance of momentum damping in Algorithm
Revised: E.6 The importance of momentum damping in Algorithm

Original: We compare our method (Algorithm 1 with O aMSN ) to a variant of it that does not use the momentum damping mechanism.
Revised: We compare our method (Algorithm 1 with O aMSN ) to a variant of it that does not use the momentum damping mechanism.

Original: That is, we set x t +

Revised: That is, we set x t +

Original: =
Revised: =

Original: ˜ x t +1 and a t +1 =
Revised: ˜ x t +1 and a t +1 =

Original: a (cid:48) t +1 regardless of the value of λ t +1 .
Revised: a (cid:48) t +1 regardless of the value of λ t +1 .

Original: As Figure 6 shows, without the momentum damping mechanism Algorithm 1 fails to converge on all the datasets we test (“a9a”, “w8a”, “splice”, and the synthetic dataset).
Revised: As Figure 6 shows, without the momentum damping mechanism Algorithm 1 fails to converge on all the datasets we test (“a9a”, “w8a”, “splice”, and the synthetic dataset).

Original: [Figure]
Revised: [Figure]

Original: o p t i m a li t y g a p '
Revised: o p t i m a li t y g a p '

Original: [Figure]
Revised: [Figure]

Original: a '
Revised: a '

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]