A Damped Newton Method Achieves Global O (cid:0) 1 k 2 (cid:1) and Local Quadratic Convergence Rate
Abstract
In this paper, we present the first stepsize schedule for Newton method resulting in fast global and local convergence guarantees.
In particular, a) we prove an O (cid:0) 1 k 2 (cid:1) global rate, which matches the state-of-the-art global rate of cubically regularized
Newton method of Polyak and Nesterov (2006) and of regularized Newton method of Mishchenko (2021) and Doikov and Nesterov (2021), b) we prove a local quadratic rate, which matches the best-known local rate of second-order methods, and c) our stepsize formula is simple, explicit, and does not require solving any subproblem.
Our convergence proofs hold under affine-invariance assumptions closely related to the notion of self-concordance.
Finally, our method has competitive performance when compared to existing baselines, which share the same fast global convergence guarantees.
Introduction
Second-order optimization methods are the backbone of much of industrial and scientific computing.
With origins that can be tracked back several centuries to the pioneering works of Newton [Newton, 1687], Raphson [Raphson, 1697] and Simpson [Simpson, 1740], they were extensively studied, generalized, modified, and improved in the last century [Kantorovich, 1948, Moré, 1978, Griewank, 1981a].
For a review of the historical development of the classical Newton-Raphson method, we refer the reader to the work of Ypma [1995].
The number of extensions and applications of second-order optimization methods is enormous; for example, the survey of Conn et al.
[2000] on trust-region and quasi-Newton methods cited over a thousand papers.
1.1 Second-order methods and modern machine learning
Despite the rich history of the field, research on second-order methods has been flourishing up to this day.
Some of the most recent development in the area was motivated by the needs of modern machine learning.
Data-oriented machine learning depends on large datasets (both in number of features and number of datapoints), which are often stored in distributed/decentalized fashion.
Consequently, there is a need for scalable algorithms.
To tackle large number of features, Qu et al.
[2016], Gower et al.
[2019], Doikov and Richtárik [2018] and Hanzely et al.
[2020] proposed variants of Newton method operating in random low-dimensional subspaces.
On the other hand, Pilanci and Wainwright [2017], Xu et al.
[2020] and Kovalev et al.
[2019] developed subsampled Newton methods for solving empirical risk minimization (ERM) problems with large training datasets.
Additionally, Bordes et al.
[2009], Mokhtari and Ribeiro [2015], Gower et al.
[2016], Byrd et al.
[2016] and Kovalev et al.
[2020] proposed stochastic variants of quasi-Newton methods.
To tackle non-centralized nature of datasets, Shamir et al.
[2014], Reddi et al.
[2016], Wang et al.
[2018] and Crane and Roosta [2019] considered distributed variants of
Newton method, with improvements under various data/function similarity assumptions.
Islamov et al.
[2021], Safaryan et al.
[2022], Qian et al.
[2022], Islamov et al.
[2022] and Agafonov et al.
[2022] developed communication-efficient distributed variants of Newton method using the idea of communication compression and error compensation, without the need for any similarity assumptions.
We highlight two main research directions throughout of history of second-order methods: globally convergent methods under additional second-order smoothness (4) [Nesterov and Polyak, 2006] and local methods for self-concordant problems (12)
[Nesterov and Nemirovski, 1989].
Former approach lead to various improvements such as acceleration Nesterov [2008], Monteiro and Svaiter [2013], usage of inexact information Ghadimi et al.
[2017], Agafonov et al.
[2020], generalization to tensor methods and their acceleration Nesterov [2021a], Gasnikov et al.
[2019], Kovalev and Gasnikov [2022], superfast second-order methods under higher smoothness Nesterov [2021c,b], Kamzolov and
Gasnikov [2020].
Latter approach was a breakthrough in 1990s, it lead to interior-point methods.
Summary of the results can be found in books Nesterov and Nemirovski [1994], Nesterov [2018].
This direction is still popular up to this day Dvurechensky and Nesterov [2018], Hildebrand [2020],
Doikov and Nesterov [2022b], Nesterov [2022].
As easy-to-scale alternative to second-order methods, first-order algorithms attracted a lot of attention.
Many of their aspects have been explored, including strong results in variance reduction (Roux et al.
[2012],Gower et al.
[2020], Johnson and Zhang [2013], Nguyen et al.
[2021, 2017]), preconditioning
Jahani et al.
[2022], acceleration (Nesterov [2013], d’Aspremont et al.
[2021]) distributed/federated computation (Kone ˇ cný et al.
[2016], Chen et al.
[2022], Berahas et al.
[2016], Taká ˇ c et al.
Richtárik and Taká ˇ c [2016], Kairouz et al.
[2021]), and decentralized computation [Koloskova et al., 2020, Sadiev et al., 2021, Borodich et al., 2021].
However, the convergence of first-order methods always depends on the conditioning of the underlying problem.
Improving conditioning is fundamentally impossible without using higher-order information.
Removing this conditioning dependence is possible by incorporating information about the Hessian.
This results in second-order methods.
Their most compelling advantage is that they can converge extremely quickly, usually in just a few iterations.
1.2 Newton method: benefits and limitations
One of the most famous algorithms in optimization, Newton method, takes iterates of form
[Equation]
Its iterates satisfy the recursion ∥∇ f ( x k +1 )
∥
≤ c ∥∇ f ( x k ) ∥ 2 2 (for a constant c > 0 ), which means that Newton method converges locally quadratically.
However, convergence of Newton method is limited to only to the neighborhood of the solution.
It is well-known that when initialized far from optimum, Newton can diverge, both in theory and practice (Jarre and Toint [2016], Mascarenhas [2007]).
We can explain intuition why this happens.
Update rule of Newton (1) was chosen to minimize right hand side of Taylor approximation
[Equation]
The main problem is that Taylor approximation is not an upper bound, and therefore, global convergence of Newton method is not guaranteed.
1.3 Towards a fast globally convergent Newton method
Even though second-order algorithms with superlinear local convergence rates are very common, global convergence guarantees of any form are surprisingly rare.
Many papers proposed globalization strategies, essentially all of them require some combination of the following: line-search, trust regions, damping/truncation, regularization.
Some popular globalization strategies show non-increase [Equation] of functional value during the training.
However, this turned out to be insufficient for convergence to the optimum.
Jarre and Toint [2016], Mascarenhas [2007] designed simple functions (strictly convex with compact level sets) so that Newton method with Armijo stepsizes does not converge to the optimum.
To this day, virtually all known global convergence guarantees are for regularized
Newton methods with, which can be written as
[Equation]
where λ k ≥ 0 .
Parameter λ k is also known as Levenberg-Marquardt regularization [Moré, 1978], which was first introduced for a nonlinear least-squares objective.
For simplicity, we disregard differences in the objectives for the literature comparison.
Motivation behind update (3) is to replace Taylor approximation in (2) by an upper bound.
The first method with proven global convergence rate O (cid:0) k − 2 (cid:1) is Cubic Newton method [Nesterov and Polyak, 2006] for function f with
Lipschitz-continuous Hessian,
[Equation]
Under this condition, one can upper bound of Taylor approximation eq.
(2) as
[Equation]
Next iterate of Cubic Newton can be written as a minimizer of right hand side of (5)
[Equation]
For our newly-proposed algorithm AICN (Algorithm 1), we are using almost identical step
[Equation]
The difference between the update of Cubic Newton and AICN is that we measure the cubic regularization term in the local Hessian norms.
This seemingly negligible perturbation turned out to be of a great significance for two reasons a) model in (7) is affine-invariant, b) surprisingly, the next iterate of (7) lies in the direction of Newton method step and is obtainable without regularizer λ k ( AICN just needs to set stepsize α k ).
We elaborate on both of these points later in the paper.
Cubic Newton method (6) can be equivalently expressed in form (3) with α k = 1 and λ k =
L 2 ∥ x k − x k +1 ∥ 2 .
However, since such λ k depends on x k +1 , resulting algorithm requires additional subroutine for solving its subproblem each iteration.
Next work showing convergence rate of regularized Newton method [Polyak, 2009] avoided implicit steps by choosing λ k
∝ ∥∇ f
( x k )
∥ 2 .
However, this came with a trade-off for slower convergence rate, O (cid:0) k − 1 / 4 (cid:1) .
Finally, Mishchenko [4/2021] (see also the work of Doikov and Nesterov [12/2021]) improved upon both of these works by using explicit regularization α k = 1 ,
λ k ∝ (cid:112) L 2 ∥∇ f ( x k ) ∥ 2 , and proving global rate O (cid:0) k − 2 (cid:1) .
Contributions 2.1 AICN as a damped Newton method
In this work, we investigate global convergence for most basic globalization strategy, stepsized
Newton method without any regularizer ( λ k = 0 ).
This algorithm is also referred as damped (or truncated) Newton method; it can be written as
[Equation]
Resulting algorithm was investigated in detail as an interior-point method.
Nesterov [2018] shows quadratic local convergence for stepsizes
α 1 def = 1 1+ G 1 , α 2 def = 1+ G 1 1+ G 1 + G 21 , where
Function f is L semi -semi-strongly self-concordant (Definition 3). Instead of
[Table] [Table]
[Table] [Table]
∥ 2 L(3) ( ✗ , ✗ , ✗ ) ✗
O (cid:16) k − [Table]
Polyak [2009]
[Table]
f ( x k ) [Table] ( ✗ , ✗ , ✗ )
✓
O (cid:0) k − 2 (cid:1)
Mishchenko [4/2021] Doikov and Nesterov [12/2021]
[Table]
− 1+ √ 1+2 G G (4) ( ✓ , ✓ , ✓ ) ✓
O (cid:0) k − 2 (cid:1)
This work
[Table]
G 1 def =
L sc ∥∇ f ( x k ) ∥
∗ x k .
Our algorithm AICN is also damped Newton method with stepsize α
= − 1+ √ 1+2 G G , where 810 G def =
L semi ∥∇ f ( x k ) ∥ ∗ x k .
Mentioned stepsizes α 1 , α 2 , α share two characteristics.
Firstly, all of them depends on gradient computed in the dual norm and scaled by a smoothness constant ( G 1 or G ).
Secondly, all of these stepsizes converge to 1 from below (for ˆ α ∈ { α 1 , α 2 , α } holds 0 < ˆ α ≤ 1 and lim x
→ x ∗ ˆ α = 1 ).
Our algorithm uses stepsize bigger by orders of magnitude (see Figure 3 in Appendix A for detailed comparison).
The main difference between already established stepsizes α 1 , α 2 and our stepsize α are resulting global convergence rates.
While stepsize α 2 does not lead to a global convergence rate, and α 1 leads to rate O (cid:0) k − 1 / 2 (cid:1) , our stepsize α leads to a significantly faster, O (cid:0) k − 2 (cid:1) rate.
Our rate matches best known global rates for regularized
Newton methods.
We manage to achieve these results by carefully choosing assumptions.
While rates for α 1 and α 2 follows from standard self-concordance, our assumptions are a consequence of a slightly stronger version of self-concordance.
We will discuss this difference in detail later.
We summarize important properties of regularized Newton methods with fast global convergence guarantees and damped Newton methods in Table 1.
2.2 Summary of contributions
To summarize novelty in our work, we present a novel algorithm AICN .
Our algorithm can be interpreted in two viewpoints a) as a regularized Newton method (version of Cubic Newton method), b) as a damped Newton method.
AICN enjoys the best properties of these two worlds:
• Fast global convergence : AICN converges globally with rate O (cid:0) k − 2 (cid:1) (Theorem 2, 4), which matches state-of-the-art global rate for all regularized Newton methods.
Furthermore, it is the first such rate for Damped Newton method.
• Fast local convergence:
In addition to the fast global rate, AICN decreases gradient norms locally in quadratic rate (Theorem 3).
This result matches the best-known rates for both regularized Newton algorithms and damped Newton algorithms.
• Simplicity:
Previous works on Newton regularizations can be viewed as a popular global-convergence fix for the Newton method.
We propose an even simpler fix in the form of a stepsize schedule (Section 3).
• Implementability: Step of AICN depends on a smoothness constant L semi (Definition 3).
Given this constant, next iterate of AICN can be computed directly.
This is improvement over Cubic Newton [Nesterov and Polyak, 2006], which for a given constant L 2 needs to run line-search subroutine each iteration to solve its subproblem • Improvement: Avoiding latter subroutine yields theoretical improvements.
If we compute matrix inverses naively, iteration cost of AICN is O ( d 3 ) (where d is a dimension of the problem), which is improvement over O ( d 3 log ε − 1 ) iteration cost of Cubic Newton [Nesterov and Polyak, 2006].
• Practical performance: We show that in practice, AICN outperforms all algorithms sharing same convergence guarantees: Cubic Newton [Nesterov and Polyak, 2006] and Globally Regularized Newton [Mishchenko, 4/2021] and Doikov and Nesterov [12/2021], and fixed stepsize Damped Newton method (Section 5).
• Geometric properties: We analyze AICN under more geometrically natural assumptions.
Instead of smoothness, we use a version of self-concordance (Section 3.1), which is invariant to affine transformations and hence also to a choice of a basis.
AICN preserves affine-invariance obtained from assumptions throughout the convergence.
In contrast, Cubic Newton uses base-dependent lnorm and hence depends on a choice of a basis.
This represents an extra layer of complexity.
• Alternative analysis: We also provide alternative analysis under weaker assumptions (Appendix C).
The rest of the paper is structured as follows.
In Section 2.3 we introduce our notation.
In Section 3, we discuss algorithm AICN , affine-invariant properties and self-concordance.
In Sections 4.1 and 4.
we show global and local convergence guarantees, respectively.
In Section 5 we present an empirical comparison of AICN with other algorithms sharing fast global convergence.
2.3 Minimization problem & notation
In the paper, we consider a d -dimensional Euclidean space E .
Its dual space, E ∗ , is composed of all linear functionals on E .
For a functional g ∈ E ∗ , we denote by ⟨ g, x ⟩ its value at x ∈ E .
We consider the following convex optimization problem:
[Equation]
where f ( x ) ∈ C 2 is a convex function with continuous first and second derivatives and positive definite Hessian.
We assume that the problem has a unique minimizer x ∗ ∈ argmin x ∈ E f ( x ) .
Note, that ∇ f
( x ) ∈
E ∗ , ∇ 2 f ( x ) h ∈ E ∗ .
Now, we introduce different norms for spaces E and E ∗ .
Denote x, h ∈ E , g ∈ E ∗ .
For a self-adjoint positive-definite operator H : E → E ∗ , we can endow these spaces with conjugate Euclidean norms:
[Equation]
For identity H
=
I , we get classical Eucledian norm ∥
x ∥ I = ⟨ x, x ⟩ 1 / 2 .
For local Hessian norm
H = ∇ 2 f ( x ) , we use shortened notation
[Equation]
Operator norm is defined by
[Equation]
for H : E → E ∗ and a fixed x ∈ E .
If we consider a specific case E ← R d , then H is a symmetric positive definite matrix.
New Algorithm: Affine-Invariant Cubic Newton
Finally, we are ready to present algorithm AICN .
It is damped Newton method with updates
[Equation] [Equation]
with stepsize
[Equation]
as summarized in Algorithm 1.
Stepsize satisfy α k ≤ 1 (from AG inequality, (45)).
Also lim x
k → x ∗ α k = 1 , hence (11) converges to Newton method.
Next, we are going to discuss geometric properties of our algorithm.
1: Requires: Initial point x 0 ∈ E , constant L s t
L est ≥ L semi >2: for k = 0 , 1 , 2 . . .
do
[Equation]
[Equation]
5: end for
3.1 Geometric properties: affine invariance
One of the main geometric properties of the Newton method is affine invariance , invariance to affine transformations of variables.
Let A : E → E ∗ be a non-degenerate linear transformation.
Consider function ϕ ( y )
=
f ( A y ) .
By affine transformation, we denote f ( x ) → ϕ ( y ) =
f ( A y ) , x → A − 1 y .
Significance of norms: Note that local Hessian norm ∥ h ∥ ∇ f ( x ) is affine-invariant because
[Equation]
where h =
A z .
On the other hand, induced norm ∥ h ∥ I is not affine-invariant because
[Equation]
With respect to geometry, the most natural norm is local Hessian norm, ∥ h ∥ ∇ f ( x ) .
From affine invariance follows that for this norm, the level sets (cid:8) y ∈ E | ∥
y − x ∥ 2 x ≤ c (cid:9) are balls centered around x (all directions have the same scaling).
In comparison, scaling of the l 2 norm is dependent on eigenvalues of the Hessian.
In terms of convergence, one direction in l 2 can significantly dominate others and slow down an algorithm.
Significance for algorithms: Algorithms that are not affine-invariant can suffer from chosen coordinate system.
This is the case for Cubic Newton, as its model (4) is bound to base-dependent lnorm.
Same is true for any other method regularized with an induced norm ∥ h ∥ I .
On the other hand, (damped) Newton methods have affine-invariant models, and hence as algorithms independent of the chosen coordinate system.
We prove this claim in following lemma (note: α k = 1 and α k from (11) are affine-invariant).
Lemma 1 (Lemma 5.1.1 Nesterov [2018]) .
Let the sequence { x k } be generated by a damped
Newton method with affine-invariant stepsize α k , applied to the function f : x k +
= x
k − α k (cid:2) ∇ 2 f ( x k ) (cid:3) − 1 ∇ f ( x k ) .
For function ϕ ( y ) , damped Newton method generates { y k } :
y k +1 =
y k − α k (cid:2) ∇ 2 ϕ ( y k ) (cid:3) − 1 ∇ ϕ
( y k ) , with y
=
A − 1 x 0 .
Then y k
=
A − 1 x k .
3.2 Significance in assumptions: self-concordance
We showed that damped Newton methods preserve affine-invariance through iterations.
Hence it is more fitting to analyze them under affine-invariant assumptions.
Affine-invariant version of smoothness, self-concordance , was introduced in Nesterov and Nemirovski [1994].
Definition 1.
Convex function f ∈ C 3 is called self-concordant if
[Equation]
where for any integer p ≥ 1 , by D p f ( x )[ h ] p def = D p f ( x )[ h, . . .
, h ] we denote the p -th order directional derivative a of f at x ∈ E along direction h ∈ E .
a For example, D 1 f ( x )[ h ] =
⟨∇ f
( x ) , h ⟩ and D 2 f ( x )[ h ] 2 = (cid:10) ∇ 2 f ( x ) h, h (cid:11) .
Both sides of inequality are affine-invariant.
This assumption corresponds to a big class of optimization methods called interior-point methods.
Self-concordance implies uniqueness of the solution, as stated in following proposition.
Proposition 1 (Theorem 5.1.16, Nesterov [2018]) .
Let a self-concordant function f be bounded below.
Then it attains its minimum at a single point.
Rodomanov and Nesterov [2021] introduced stronger version of self-concordance assumption.
Definition 2.
Convex function f ∈ C 2 is called strongly self-concordant if
[Equation]
For our analysis, we introduce definition for functions between self-concordant and strongly self-concordant.
Definition 3.
Convex function f ∈ C 2 is called semi-strongly self-concordant if
[Equation]
[Equation]
Note that all of the Definitions 1 - 3 are affine-invariant, their respective classes satisfy strong self-concordance ⊆
semi-strong self-concordance ⊆ self-concordance .
Also, for a fixed strongly self-concordant function f and smallest such L sc , L semi , L str holds L sc ≤
L semi ≤ L str .
These notions are related to the convexity and smoothness; strong concordance follows from function
L 2 -Lipschitz continuous Hessian and strong convexity.
Proposition 2 (Example 4.1 from [Rodomanov and Nesterov, 2021]) .
Let H : E → E ∗ be a self-adjoint positive definite operator.
Suppose there exist µ > 0 and L 2 ≥ 0 such that the function f is µ -strongly convex and its Hessian is L 2 -Lipschitz continuous (4) with respect to the norm ∥ · ∥ H .
Then f is strongly self-concordant with constant L str =
L 2 µ 3 / 2 .
3.3 From assumptions to algorithm
From semi-strong self-concordance we can get a second-order bounds on the function and model.
Lemma 2.
If f is semi-strongly self-concordant, then
[Equation]
Consequently, we have upper bound for function value in form
[Equation]
One can show that (16) is not valid for just self-concordant functions.
For example, there is no such upper bound for − log( x ) .
Hence, the semi-strongly self-concordance is significant as an assumption.
We can define iterates of optimization algorithm to be minimizers of the right hand side of (16),
[Equation]
for an estimate constant L est ≥ L semi .
It turns out that subproblem (17) is easy to solve.
To get an explicit solution, we compute its gradient w.r.t.
h .
For solution h ∗ , it should be equal to zero,
[Equation]
We get that step (18) has the same direction as a Newton method and is scaled by α k = (cid:0)
L est 2 ∥ h ∗ ∥ x + 1 (cid:1) − 1 .
Now, we substitute h ∗ from (20) to (19)
[Equation]
We solve the quadratic equation (21) for α k , and obtain explicit formula for stepsizes of AICN , as (11).
We formalize this connection in theorem, for further explanation see proof in Appendix B.
Theorem 1.
For L est ≥ L semi , update of AICN (11) ,
[Equation] [Equation]
1 ∇ f ( x k ) , where
[Equation]
is a minimizer of upper bound (17) , x k +
=
S f,L est ( x k ) .
Convergence Results 4.1 Global convergence
Next, we focus on global convergence guarantees.
We will utilize the following assumption:
Assumption 1 (Bounded level sets) .
The objective function f has a unique minimizer x ∗ .
Also, the diameter of the level set L ( x 0 ) def = { x ∈ E : f ( x ) ≤ f ( x 0 ) } is bounded by a constant Das a , max x ∈L ( x 0 )
∥ x − x ∗ ∥
≤ D 2 < + ∞ .
Our analysis proceeds as follows.
Firstly, we show that one step of the algorithm decreases function value, and secondly, we use the technique from [Ghadimi et al., 2017] to show that multiple steps lead to O (cid:0) k − 2 (cid:1) global convergence.
We start with following lemma.
Lemma 3 (One step globally) .
Let function f be L semi -semi-strongly self-concordant, convex with positive-definite Hessian and L est ≥ L semi .
Then for any x ∈ E , we have
[Equation]
This lemma implies that step (17) decreases function value (take y ← x ).
Using notation of
Assumption 1, x k ∈ L ( x 0 ) for any k ≥ 0 .
Also, setting y ← x and x ← x ∗ in (14) yields ∥ x − x ∗ ∥ x ≤ (cid:0) ∥ x − x ∗ ∥ 2 x ∗ + L est ∥ x − x ∗ ∥ 3 x ∗ (cid:1) 12 .
We denote those distances D and R ,
[Equation]
and
[Equation]
They are both affine-invariant and R upper bounds D .
While R depends only on the level set L ( x 0 ) ,
D can be used to obtain more tight inequalities.
We avoid using common distance D 2 , as l 2 norm would ruin affine-invariant properties.
Theorem 2.
Let f ( x ) be a L semi -semi-strongly self-concordant convex function with positive-definite Hessian, constant L est satisfy L est ≥ L semi and Assumption 1 holds.
Then, after k + 1 iterations of Algorithm 1, we have the following convergence rate:
[Equation]
Consequently, AICN converges globally with a fast rate O (cid:0) k − 2 (cid:1) .
We can now present local analysis.
4.2 Local convergence
For local quadratic convergence are going to utilise following lemmas.
Lemma 4.
For convex L semi -semi-strongly self-concordant function f and for any c < 1 in the neighborhood of solution
[Equation]
Lemma 4 formalizes that a inverse hessians of a self-concordant function around the solution is non-degenerate.
With this result, we can show one-step gradient norm decrease.
Lemma 5 (One step decrease locally) .
Let function f be L semi -semi-strongly self-concordant and
L est ≥ L semi .
If x k such that (25) holds, then for next iterate x k +1 of AICN holds
[Equation]
Using Lemma 4, we shift the gradient bound to respective norms,
[Equation]
Gradient norm decreases ∥∇ f
( x k +1 ) ∥ ∗ x k +1 ≤ ∥∇ f ( x k ) ∥ ∗ x k for ∥∇ f ( x k ) ∥ ∗ x k ≤ (2 − c ) 2 − 1 2 L est .
As a result, neighbourhood of the local convergence is (cid:110) x : ∥∇ f ( x ) ∥
[Equation]
x ≤ min (cid:104) (2 − c ) [Equation] 1 2 L est ; (2 c +1) 2 − 1 2 L est (cid:105)(cid:111) .
Maximizing by c , we get c = 1 / 3 and neighrbourhood (cid:110) x : ∥∇ f ( x ) ∥
∗ x ≤ 8 9 L est (cid:111) .
One step of AICN decreases gradient norm quadratically, multiple steps leads to following decrease.
Theorem 3 (Local convergence rate) .
Let function f be L semi -semi-strongly self-concordant,
L est ≥ L semi and starting point x 0 be in the neighborhood of the solution such that ∥∇ f ( x 0 )
∥ ∗ x 0 ≤ 8 9 L est .
For k ≥ 0 , we have quadratic decrease of the gradient norms,
[Equation]
Numerical Experiments
In this section, we evaluate proposed AICN (Algorithm 1) algorithm on the logistic regression task and second-order lower bound function.
We compare it with regularized Newton methods sharing fast global convergence guarantees: Cubic Newton method [Nesterov and Polyak, 2006], and Globally Regularized Newton method ([Mishchenko, 4/2021, Doikov and Nesterov, 12/2021]) with L 2 -constant.
Because AICN has a form of a damped Newton method, we also compare it with
Damped Newton with fixed (tuned) stepsize α k
= α .
We report decrease in function value f ( x k ) , function value suboptimality f ( x k ) − f ( x ∗ ) with respect to iteration and time.
The methods are implemented as PyTorch optimizers.
The code is available at https://github.com/OPTAMI .
5.1 Logistic regression
For first part, we solve the following empirical risk minimization problem:
[Equation]
[Figure] [Figure]
where { ( a i , b i ) } mi =1 are given data samples described by features a i and class b i ∈ {− 1 , 1 } .
In Figure 1, we consider task of classification images on dataset a9a [Chang and Lin, 2011].
Number of features for every data sample is d = 123 , m = 20000 .
We take staring point x 0 def = 10[1 , 1 , . . .
, 1] ⊤ and µ = 10 − 3 .
Our choice is differ from x 0 =
(equal to all zeroes) to show globalisation properties of methods ( x 0 = 0 is very close to the solution, as Newton method converges in 4 iterations).
Parameters of all methods are fine-tuned, we choose parameters L est , L 2 , α (of AICN , Cubic Newton, Damped Newton, resp.)
to largest values having monotone decrease in reported metrics.
Fine-tuned values are L est = 0 .
97 L 2 = 0 . 000215 , α = 0 . 285 .
Figuredemonstrates that AICN converges slightly faster than Cubic Newton method by iteration, notably faster than Globally Regularized Newton and significantly faster than Damped Newton.
AICN outperforms every method by time.
5.2 Second-order lower bound function
For second part we solve the following minimization problem:
[Equation]
 
[Equation]
  , where
A =   1 − . . .
− 1 . .
  .
This function is a lower bound for a class of functions with Lipschitz continuous Hessian (4) with additional regularization [Nesterov and Polyak, 2006, Nesterov, 2021a].
In Figure 2, we take d = 20 , x 0 =
(equal to all zeroes).
Parameters L est , L 2 , α are fine-tuned to largest values having monotone decrease in reported metrics:
L est = 662 L 2 = 0 .
662 , α = 0 .
Figure 2 demonstrates that
AICN converges slightly slower than Cubic Newton method, slightly faster than Globally Regularized
Newton, and significantly faster than Damped Newton.
ain outperforms every method by time.
More experiments are presented in Appendix A. Note, that the iteration of the Cubic Newton method needs an additional line-search, so one iteration of Cubic Newton is computationally harder than one iteration of AICN .
More experiments are presented in Appendix A.
Acknowledgement
The work of D. Pasechnyuk and A. Gasnikov was supported by a grant for research centers in the field of artificial intelligence, provided by the Analytical Center for the Government of the RF in accordance with the subsidy agreement (agreement identifier 000000D730321P5Q0002) and the agreement with the Ivannikov Institute for System Programming of the RAS dated November 2,
No. 70-2021-00142.
Appendix A Extra comparisons to other methods A.1 Damped Newton method stepsize comparison
In Figure 3, we present comparison of stepizes of AICN with other damped Newton methods [Nesterov, 2018].
Our algorithm uses stepsize bigger by orders of magnitude.
For reader’s convenience, we repeat stepsize choices.
For AICN stepsize is α = − 1+ √ 1+2 G G , where G def =
L semi ∥∇ f ( x k ) ∥
∗ x k .
For damped Newton methods from Nesterov [2018], α 1 def = 1 1+ G 1 , α 2 def = 1+ G 1
+
G
+ G 21 , where G 1 def =
L sc ∥∇ f ( x k ) ∥
∗ x k .
A.2 Convergence rate comparison under various assumptions
In this subsection we present Table
A.3 Logistic regression experiments
We solve the following minimization problem:
[Equation]
To make problem and data balanced, we normalise each data point and get ∥ a i ∥ 2 = 1 for every i ∈ [1 , . . .
, m ] .
Parameters of all methods are fine-tuned to get the fastest convergence.
Note, that it is possible that for bigger L method converges faster in practice.
In Figure 4, we consider classification task on dataset w8a [Chang and Lin, 2011].
Number of features for every data sample is d = 300 , m = 49749 .
We take starting point x 0 def = 8[1 , 1 , . . .
, 1] ⊤ and µ = 10 − 3 .
Fine-tuned values are L est = 0 .
6 , L 2 = 0 . 0001 ,
α = 0 .
We can see that all methods are very close.
Damped Newton has rather big step 0 .
5 , so it is fast at the beginning but still struggle at the end because of the fixed step-size.
In Figure 5, we consider binary classification task on dataset MNIST [Deng, 2012]
(one class contains images with 0, another one — all others).
Number of features for every data sample is d = 28 2 = 784 , m = 60000 .
We take starting point x 0 def = 3 · [1 , 1 , . . .
, 1] ⊤ (such that Newton method started from this point diverges) and µ = 10 − 3 .
Fine-tuned values are L est = 10 , L 2 = 0 .
0003 for Globally Reg.
[Table]
∥ 2 x + L alt 6 ∥ h ∥ 3 x O (cid:0) k − 2 (cid:1)
Theorems 4, 3 (3)
AIC Newton f ( x + h ) − f ( x ) ≤ ⟨∇ f ( x ) ,h ⟩ + 12 ∥ h ∥ 2 x + L alt 6 ∥ h ∥ 3 x O (cid:0) k − 2 (cid:1) (4) ✗
Theorem(1)
For a Lyapunov function Φ and a constant c > 0 , we report exponent β > 1 of Φ( x k +1 ) ≤ c Φ( x k ) β .
Mark ✗ means that such β,c, Φ are not known.
(2) Self-concordance implies strong convexity locally.
(3) Under strong convexity, we can prove local convergence analogically to Theorem 3.
Convergence to a neighborhood of the solution
[Figure]
Newton and Cubic Newton, α = 0 .
We see that AICN has the same iteration convergence as Cubic
Newton but faster by time.
[Figure]
In Figure 6, we present the results for multi-class classification problem on dataset MNIST .
We train 10 different models in parallel, each one for the problem of binary classification distinguishing i -th class out of others.
Loss on current iteration for the plots is defined as average loss of 10 models.
0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 Iterations, k51525f ( x k )
Cubic Newton Damped Newton AIC Newton
Globally Reg.
Newton 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 Iterations, k 10 − 10 −5 10 − 10 −3 10 − 10 −1 1010f ( x k ) − f ( x * )
Cubic Newto Damped Newto AIC Newto Globall( Reg.
Newto      300 Time, sec 10 − 10 −5 10 − 10 −3 10 − 10 −1 1010f ( x k ) − f ( x * )
Cubic Newt n Damped Newt n AIC Newt n Gl bally Reg.
Newt n
Prediction is determined by the maximum “probability” predicted by i -th model.
The estimates for the parameters of methods are the same as in previous experiment.
We see that AICN is the same speed as Cubic Newton and Globally Reg.
Newton and much faster than Damped Newton in both of and gradient norm.
For normalized problem, we can analytically compute an upper bound for theoretical constant L
[Equation]
One can show that L 2 = √ 318 ≃ 0 .
In our experiments, we show that Cubic Newton can work with much lower constants: madelon - 0 . 015 , w8a - 0 . 00003 , a9a - 0 . 000215 .
It means that theoretical approximation of the constants can be bad and we have to tune them for all methods.
B Proofs of Results Appearing in the Paper
In this section, we present proofs of the lemmas and theorems from the main paper’s body.
B.1 Proofs regarding affine invariance (Section 3.1)
Proof of Lemma 1 (Lemma 5.1.1, Nesterov [2018]).
[Newton method is affine-invariant]
Let y k
=
A − 1 x k for some k ≥
and α k be affine-invariant.
Firstly,
[Equation]
Secondly note that ∥∇ f ( x ) ∥ ∗ x is affine invariant, as
[Equation]
Consequently, stepsizes α k from AICN (11) and Nesterov [2018] are all affine-invariant.
Hence these damped Newton algorithms are affine-invariant.
Proof of Lemma 2.
[Upper bound from from semi-strong self-concordance]
We rewrite function value approximation from the left hand side as
[Equation]
Taking its norm, we can finish the proof as
[Equation]
Proof of Theorem 1.
[Minimizer of the model (18) has form of damped Newton method (11) ]
Proof is straightforward.
To show that AICN model update minimizes S f,L est ( x ) , we compute the gradient of S f,L est ( x ) at next iterate of AICN .
Showing that it is 0 concludes x k +
=
S f,L est ( x k ) .
For simplicity, denote h def =
y − x .
We can simplify the implicit update step S f,L est ( x )
S f
[Equation]
L est (
[Equation]
= argmin
[Equation]
E
[Equation]
Taking gradient of the subproblem with respect to h ,
[Equation]
and setting h according to AICN , h = − α ∇ 2 f ( x ) − 1 ∇ f ( x ) , leads to
[Equation]
[Equation]
Finally
[Equation]
AICN stepsize α (11) is chosen as a root of quadratic function
[Equation]
[Equation]
hence the gradient of the (17) at next iterate of AICN is 0 .
This concludes the proof
[Equation]
B.2 Proof of global convergence (Section 4.1)
Proof of Lemma 3.
[One step decrease globally under semi-strong self-concordance]
This claim follows directly from Lemma 2.
[Equation]
Proof of Theorem 2.
[Global convergence under semi-strong self-concordance]
We start by taking Lemma 3 for any t ≥ 0 , we obtain
[Equation]
where for the second line we take y = x t + η t ( x ∗ − x t ) and use convexity of f for the third line.
Therefore, subtracting f ( x ∗ ) from both sides, we obtain, for any η t ∈ [0 , 1] [Equation]
[Equation]
Let us define the sequence { A t } t ≥ 0 as follows:
[Equation]
Then A t = (1 − η t )
A t − 1 .
Also
[Equation]
we define η 0 def = 1 .
Dividing both sides of (34) by A t , we get
[Equation]
Summing both sides of inequality (36) for t = 0 , . .
, k , we obtain
[Equation]
As a result,
[Equation]
To finish the proof, we need to choose η t so that (cid:80) kt =
A k η 3 t
A t =
O ( k − 2 ) .
This holds for
[Equation]
as stated in the next lemma.
Lemma 6 (Properties of η t and A t , [from (2.23), Ghadimi et al.
For choice η t as (38) and
A t as (35) we have
[Equation]
Plugging Lemma 6 inequalities to (37) concludes the proof of the Theorem 2, f
[Equation]
x
[Equation]
( k + 1)( k + 2)( k + 3)
L est D 9 k 2 ≤ 9 L est Dk≤ 9 L est Rk.
For readers convenience, we include proof of Lemma 6 in Appendix B.4.
B.3 Proofs of local convergence (Section 4.2)
Proof of Lemma 4.
[Strong convexity / Bound on inverse Hessian norm change]
Claim follows from Theorem 5.1.7 of Nesterov [2018], which states that for L sc -self-concordant function, hence also for L semi -semi-strongly self-concordant function f and x k , x k +1 such that L semi 2 ∥ x
[Equation]
− x k ∥ x k < 1 holds
[Equation]
Let c < 1 be some constant then for updates of AICN in the neighborhood
[Equation]
holds
[Equation]
In order to prove Lemma 5, we first use semi-strong self-concordance to prove a key inequality – a version of Hessian smoothness, bounding gradient approximation by difference of points.
Lemma 7.
For semi-strongly self-concordant function f holds
[Equation]
Proof of Lemma 7.
[Local smoothness assumption follows from semi-strong self-concordance]
We rewrite gradient approximation on the left hand side as
[Equation]
Now
[Equation]
we can bound it in dual norm as
[Equation]
where in second inequality we used property of operator norm (10)
[Equation]
Finally, we are ready to prove one step decrease and the convergence theorem
[Equation]
Proof of Lemma 5.
[One step decrease locally under semi-strong self-concordance]
We bound norm of ∇ f ( x k +1 ) using basic norm manipulation and triangle inequality as ∥∇
[Equation]
Using Lemma 7, we can continue
[Equation]
We use Lemma 4 to shift matrix norms.
[Equation]
We obtain neighborhood of decrease by solving L est α k 1 − c ∥∇ f ( x k ) ∥ ∗ x k ≤ 1 in ∥∇ f ( x k ) ∥
∗ x k .
Proof of Theorem 3.
[Local convergence under semi-strong self-concordance]
Let c = 13 , then for ∥∇ f
( x 0 ) ∥ ∗ x 0 < 8 9 L est , we have L est α 0 1 − c ∥∇ f
( x 0 ) ∥ ∗ x 0 ≤ 1 and c ≥
L est 2 α 0 ∥∇ f ( x 0 )
∥ ∗ x 0 .
Then from Lemma 5 we have guaranteed the decrease of gradients ∥
g k +1 ∥ ∗ x k +1 ≤ ∥
g k ∥
∗ x k < 8 9 L est .
We finish proof by chaining (43) and simplifying with α i ≤ 1 .
[Equation] [Equation]
B.4 Technical lemmas
Lemma 8 (Arithmetic mean – Geometric mean inequality) .
For c ≥
we have
[Equation]
Lemma 9 (Jensen for square root) .
Function f ( x ) = √ x is concave, hence for c ≥
we have
[Equation]
Lemma 6 [(2.23) from [Ghadimi et al., 2017]]
For η t def = t + 3 , t ≥ 0 , and
A
[Equation]
def =   1 , t =
[Equation]
(cid:81)
[Equation]
= [Equation] [Equation] ≥we have
A
[Equation]
and k (cid:88) t = η [Equation] k
[Equation]
k (cid:88) t = 9(
[Equation]
[Equation]
Proof of Lemma 6.
We have
A
[Equation]
(48) which gives, k (cid:88)
[Equation]
A
[Equation]
k
[Equation] j [Equation] (cid:88) t = (cid:81)
[Equation] [Equation] [Equation] +
[Equation]
The sum is non-decreasing.
Indeed, we have 1 ≤ 1 + t + 3 ≤ 1 + t + j , ∀ j ∈ { 1 , 2 , 3 } , and, hence, for all j ∈ { 1 , 2 , 3 } , (cid:16) 1 + 1 t +3 (cid:17)≤ 3 (cid:89)
j = (cid:18) 1 + t + j (cid:19) ⇔ (cid:16) t +4 t +3 (cid:17)
[Equation]
Thus, we have shown that the summands in the RHS of (49) are growing, whence we get the next upper bound for the sum
[Equation]
C Global Convergence with weaker assumptions on Self-Concordance
We can prove global convergence to a neighborhood of the solution without using self-concordance directly, just by utilizing the following assumptions:
Assumption 2 (Convexity) .
For function f and any x, h ∈ E holds
[Equation]
Assumption 3 (Hessian smoothness, in Hessian norms) .
Objective function f satisfy
[Equation]
Lemma 10 (One step decrease globally) .
Let Assumption 3 hold and let L est ≥ L alt .
Iterates of
AICN eq.
(11) yield function value decrease,
[Equation]
Decrease of Lemma 10 is tight up to a constant factor.
As far as ∥∇ f ( x k ) ∥ ∗ x k ≤ 4 c 1 L est , we have functional value decrease as the first line of (53), which leads to O (cid:0) k − 2 (cid:1) convergence rate.
We can obtain fast convergence to only neighborhood of solution, because close to the solution, gradient norm is sufficiently small ∥∇ f ( x k )
∥ ∗ x k ≤ 4 c 1 L est
and we get functional value decrease from second line of (53).
However, this convergence is slower then O (cid:0) k − 2 (cid:1) for ∥∇ f ( x k ) ∥ ∗ x k
≈
and it is insufficient for O ( k − 2 ) rate.
Note that third line generalizes first line; we use it to remove a constant factor gap from the neighborhood of fast local convergence.
Theorem 4 (Global convergence) .
Let Assumptions 2, 3, 1 hold, and constants c 1 , L est satisfy 0 < c 1 ≤ 1 , L est ≥ L alt .
For k until ∥∇ f ( x k ) ∥ ∗ x k ≥ 4 c 1 L est , AICN has global quadratic decrease,
[Equation]
Note that the global quadratic decrease of AICN is only to a neighborhood of the solution.
However, once AICN gets to this neighborhood, it converges using (faster) local convergence rate (Theorem 3).
Proofs of global convergence without self-concordance
Throughout the rest of proofs
[Equation]
we simplify expressions by denoting terms
[Equation]
for which holds
[Equation]
and also G k def =
L est
∥ g k ∥ ∗ x k .
Proof of Lemma 10.
We can use Assumption 3 to obtain
[Equation]
We can simplify bracket in eq.
(57) as
[Equation]
Also, we can simplify the other term in eq.
(57) as
[Equation]
and plug these two result into eq.
(57) to obtain first two lines of (53).
Third line can be obtained from the first line of (53).
For c 1 so that 0 < c 1 ≤ 1 and x k satisfying 4 c
[Equation]
est ≤ ∥∇ f
( x k
[Equation]
∥ ∗ x k <
[Equation]
L est holds
[Equation]
Proof of Theorem 4.
As a consequence of convexity (Assumption 2) and bounded level sets (Assumption 1), we have
[Equation]
Plugging it into eq.
(53) (which holds under Assumption 3) and noting that it yields
[Equation]
Denote τ def = √ c
[Equation]
and β
[Equation]
τ
[Equation]
f ( x k ) − f ∗ ) ≥ 0 .
Terms β k satisfy recurrence
[Equation]
Because β k +1 ≥ 0 , we have that β k ≤ 1 .
Proposition 1 of Mishchenko [4/2021] shows that the sequence { β k } ∞ k =0 , 0 ≤ β k ≤ 1 decreases as
O (cid:0) 1 k 2 (cid:1) .
Let c 2 be a constant satisfying β k ≤ c 2 k 2 for all k ( c 2 ≈ 3 seems to be sufficient).
Finally, fol
[Equation]
we have
[Equation]