The First Optimal Acceleration of High-Order Methods in Smooth Convex Optimization
Abstract
In this paper, we study the fundamental open question of finding the optimal highorder algorithm for solving smooth convex minimization problems.
Arjevani et al. (2019) established the lower bound Ω (cid:0)
ϵ − 2 / (3 p +1) (cid:1) on the number of the p -th order oracle calls required by an algorithm to find an ϵ -accurate solution to the problem, where the p -th order oracle stands for the computation of the objective function value and the derivatives up to the order p .
However, the existing state-of-theart high-order methods of Gasnikov et al.
(2019b)
; Bubeck et al.
; Jiang et al.
(2019) achieve the oracle complexity O (cid:0) ϵ − 2 / (3 p +1) log(1 /ϵ ) (cid:1) , which does not match the lower bound.
The reason for this is that these algorithms require performing a complex binary search procedure, which makes them neither optimal nor practical.
We fix this fundamental issue by providing the first algorithm with
O (cid:0) ϵ − 2 / (3 p +1) (cid:1) p -th order oracle complexity.
Introduction
Let R d be a finite-dimensional Euclidean space and let f ( x ): R d → R be a convex, p times continuously differentiable function with L p -Lipschitz p -th order derivatives.
Our goal is to solve the following convex minimization problem:
[Equation]
In this work we assume access to the p -th order oracle associated with function f ( x ) .
That is, given an arbitrary point x ∈ R d , we can compute the function value and the derivatives of function f ( x ) up to order p .
First-order methods.
When p = 1 , first-order methods, such as gradient descent, are typically used for solving problem (1).
The lower bound Ω( ϵ − 1 / 2 ) on the number of the gradient evaluations required by these algorithms to find an ϵ -accurate solution was established by Nemirovskij and Yudin (1983); Nesterov (2003), while the optimal algorithm matching this lower bound is called Accelerated
Gradient Descent and was developed by Nesterov (1983).
Second-order methods.
In contrast to the first-order methods, the understanding of the secondorder methods ( p = 2 ) was developed relatively recently.
Nesterov and Polyak (2006) developed the cubic regularized variant of Newton’s method.
This algorithm achieves global convergence with
[Table]
Algorithm 4 (This Paper)
[Table]
[Table]
First-Order
Methods ( p = [Table]
Second-Order
Methods ( [Table] = [Table]
High-Order
Methods ( [Table] ) the oracle complexity O ( ϵ − 1 / 2 ) , which cannot be achieved with the standard Newton’s method.
Nesterov (2008) also developed an accelerated version of the cubic regularized Newton’s method with O (cid:0) ϵ − 1 / 3 (cid:1) second-order oracle complexity.
A few years later, Monteiro and Svaiter (2013) developed the Accelerated Hybrid Proximal Extragradient (A-HPE) framework and combined it with a trust region Newton-type method.
The resulting algorithm, called Accelerated Newton Proximal
Extragradient (A-NPE), achieved the second-order oracle complexity of O (cid:0) ϵ − 2 / 7 log(1 /ϵ ) (cid:1) .
In 2018,
Arjevani et al.
(2019) established the lower bound Ω (cid:0)
ϵ − 2 / 7 (cid:1) on the number of the second-order oracle calls required by an algorithm to find an ϵ -accurate solution 5 , which is almost achieved by the
A-NPE algorithm of Monteiro and Svaiter (2013), up to the logarithmic factor log(1 /ϵ ) .
However, the optimal second-order algorithms for solving smooth convex minimization problems remain to be unknown.
High-order methods.
In the case when p > 2 , the situation is very similar to the second-order case.
Nesterov (2021a) developed the generalization of the cubic regularized Newton method to the high-order case and called them tensor methods.
Nesterov (2021a) provided both non-accelerated and accelerated p -th order tensor methods with the oracle complexity O (cid:0) ϵ − 1 /p (cid:1) and O (cid:0) ϵ − 1 / ( p +1) (cid:1) , respectively.
6 Later, three independent groups of researchers (Gasnikov et al., 2019a; Bubeck et al., 2019; Jiang et al., 2019) used the A-HPE framework to develop the near-optimal tensor methods with the oracle complexity O (cid:0) ϵ − 2 / (3 p +1) log(1 /ϵ ) (cid:1) .
Similarly to the case p = 2 , these algorithms match the lower complexity bound Ω (cid:0) ϵ − 2 / (3 p +1) (cid:1) of Arjevani et al.
(2019), up to the logarithmic factor log(1 /ϵ ) .
1.1 Main Contribution: Optimal Second-Order and High Order Methods
The review of the second-order and high-order methods that we made above identifies the following fundamental open question:
Can we design an optimal p -th order algorithm ( p ≥ 2 ) for solving smooth convex minimization problems with the oracle complexity matching the lower bounds?
The lack of an answer to this question reveals a significant gap in the understanding of the high-order optimization compared to the first-order optimization.
We give a positive answer to this question.
That is, we provide the first optimal high-order algorithm with the p -th order oracle complexity
O (cid:0) ϵ − 2 / (3 p +1) (cid:1) that matches the lower bounds of Arjevani et al.
This is the main contribution of our work. 1.2 Related Work
High-order and second-order methods have attracted a lot of interest recently.
Relevant works include but are not limited to superfast second-order methods (Nesterov, 2021c), second-order methods with gradient regularization (Mishchenko, 2021;
Doikov et al., 2022; Doikov and Nesterov, 2021), high-order methods for non-smooth optimization via smoothing technique (Bullins, 2020), and ball-constrained optimization Carmon et al.
A more detailed review of recent advances in high-order optimization can be found in the work of Kamzolov et al.
High-order methods for variational inequalities. Monteiro and Svaiter (2012) also developed second-order methods for solving monotone variational inequalities and inclusions problems for operators with Lipschitz continuous derivatives, which were generalized to the high-order setting by Bullins and Lai (2022); Jiang and Mokhtari (2022). However, similarly to the near-optimal high-order methods for minimization problems (Monteiro and Svaiter, 2013; Gasnikov et al., 2019b), these algorithms have additional logarithmic factors in the complexity that appear due to the requirement of performing the binary search procedure.
Recently, Lin et al.
; Adil et al.
(2022) removed the extra logarithmic factors and provided high-order methods for solving monotone variational inequalities and inclusions problems without any binary search procedures.
Moreover, Lin et al.
(2022) established lower complexity bounds that matched the proposed algorithms.
Hence, the problem of finding optimal high-order algorithms for solving variational inequalities and inclusions problems is solved.
In our paper, we solve a similar problem of finding optimal high-order methods for solving smooth convex minimization problems.
Note that this problem is much more challenging because optimal algorithms for solving variational inequalities and inclusion problems are typically much simpler and do not require utilising acceleration techniques.
Concurrent work of Carmon et al.
The question of finding optimal high-order methods for solving smooth convex minimization problems was solved recently in the concurrent work of
Carmon et al.
However, their approach is substantially different from the approach used in this work, and their paper appeared on arXiv 11 days later than ours.
1.3 Paper Organization
Our paper is organized as follows: (i) in Section 2, we briefly introduce the tensor approximations and provide necessary assumptions and definitions; (ii) in Section 3, we describe the existing near-optimal high-order methods and identify their main flaws that prevent them from being optimal and practical algorithms; (iii) in Section 4, we describe the development of our optimal high-order algorithm and provide its theoretical convergence analysis.
Preliminaries
By ∥ · ∥ : R d → R and ⟨· , ·⟩ : R d × R d → R , we denote the standard Euclidean norm and scalar product on R d .
Given a p times continuously differentiable function g ( x ): R d → R and index i ∈ { 1 , 2 , . . .
, p } , by ∇ i g ( x )
[ h ] i : R d → R we denote the following homogeneous polynomial:
[Equation]
where x = ( x 1 , . .
, x d ) ∈ R d , h
= ( h 1 , . . .
, h d ) ∈ R d , and ∂ i g
[Equation]
1 · · · ∂x j i ( x ) [Equation] 3) is the i -th order partial derivative of function g ( x ) at point x with respect to variables x j 1 , . . .
, x j i .
For instance, if i = 1 , then ∇ 1 g ( x )[ h ] =
⟨∇ g ( x ) , h ⟩ , where ∇ g ( x ) ∈ R
d is the gradient of function g ( x ) ; if i = 2 , then ∇ 2 g ( x )[ h ] = ⟨∇
2 f ( x ) h, h ⟩ , where ∇ 2 f ( x ) ∈ R
d × d is the Hessian of function f ( x ) .
We can write the p -th order Taylor approximation of function g ( x ) at point z ∈ R d : [Equation] = [Equation]
(cid:88)
[Equation]
It is well known that the Taylor polynomial Φ pg ( x ; z ) approximates function g ( x ) , if point x is close enough to point z :
[Equation]
where R pg ( · ; z ): R d → R is a function that satisfies lim x → z R pg ( x ; z ) = 0 .
As mentioned earlier, we assume that the objective function f ( x ) of the main problem (1) is p times continuously differentiable and has L p -Lipschitz p -th order derivatives.
It is formalized via the following definition.
Assumption 1.
Function f ( x ) is p -times continuously differentiable, convex, and has L p -Lipschitz p -th order derivatives, i.e., for all x 1 ,
x 2 ∈ R d
the following inequality holds:
[Equation]
Theorem 1 of Nesterov (2021a) implies that under Assumption 1, function f ( x ) has the following convex upper bound:
[Equation]
where M ≥ L p and z ∈ R d .
Hence, an obvious approach to solving problem (1) is to perform the minimization of this upper bound instead of minimizing the function f ( x ) .
This approach naturally leads to the following iterative process:
[Equation]
In the case p = 2 , this iterative process is known as the cubic regularized Newton’s method of
Nesterov and Polyak (2006), and in the case p > 2 , it is known as the tensor method of Nesterov (2021a).
Minimization procedures similar to (7) are widely used in high-order optimization methods.
It will also be used in the development of our optimal algorithm.
We also have the following assumption which requires problem (1) to have at least a single solution x ∗ ∈ R d .
It is a standard assumption for the majority of works on convex optimization.
Assumption 2.
There exists a constant R > 0 and at least a single solution x ∗ to problem (1) , such that ∥ x 0 − x ∗ ∥ ≤ R , where x 0 ∈ R d is the starting point that we use as an input for a given algorithm for solving the problem.
Finally, we have the following definitions that formalize the notions of ϵ -accurate solution of a problem, p -th order oracle call, and oracle complexity of an algorithm.
Definition 1.
We call vector ˆ x ∈ R d an ϵ -accurate solution of problem (1) , if for a given accuracy ϵ >
it satisfies f
(ˆ x ) − f ∗ ≤ ϵ .
Definition 2.
Given an arbitrary vector x ∈ R d by the p -th order oracle call at x , we denote the computation of the function value f ( x ) and the derivatives ∇ 1 f ( x )[ · ] , . . .
, ∇ p f ( x )[ · ] .
Definition 3.
By the p -th order oracle complexity of a p -th order algorithm for solving problem (1) , we denote the number of p -th order oracle calls required by the algorithm to find an ϵ -accurate solution of the problem for a given accuracy ϵ > 0 . [Equation] [Equation] ∈ R d λ k > 0 such that
[Equation] where x kg ∈ R d and α k ∈ (0 , 1] are defined as [Equation] and η k > 0 and β k > 0 are defined by the following system:
[Equation]
[Equation]  Near-Optimal Tensor Methods In this section, we revisit the state-of-the-art high-order optimization algorithms that include the
A-NPE method of Monteiro and Svaiter (2013) in the p = 2 case and the near-optimal tensor methods of Gasnikov et al. (2019a); Bubeck et al.
; Jiang et al.
in the general p > 2 case.
We start with describing the key ideas behind the development of these algorithms to understand how they work.
Then, we identify the main flaws of the algorithms that prevent them from being optimal and practical.
Note that the A-NPE method and near-optimal tensor methods have the following substantial similarities: (i) both the A-NPE and near-optimal tensor methods are based on the A-HPE framework of
Monteiro and Svaiter (2013); (ii) the oracle complexity of the near-optimal tensor methods recovers the oracle complexity of the A-NPE method in the case p = 2 ; (iii) these algorithms have the same issue: the requirement to perform the complex binary search procedure at each iteration which makes them neither optimal nor practical.
Hence, we will further leave out the description of the
A-NPE method of Monteiro and Svaiter (2013) and consider only the near-optimal tensor methods of
Gasnikov et al. (2019a); Bubeck et al.
; Jiang et al.
3.1 A-HPE Framework
The main component in the development of the near-optimal tensor methods of Gasnikov et al. (2019a) ; Bubeck et al.
; Jiang et al.
(2019) is the Accelerated Hybrid Proximal Extragradient (A-HPE) framework of Monteiro and Svaiter (2013).
This algorithmic framework can be seen as a generalization of the Accelerated Gradient Descent of Nesterov (1983).
It is formalized as
Algorithm 1.
Next, we recall the main theorem by Monteiro and Svaiter (2013), which describes the convergence properties of Algorithm 1.
Theorem 1 (Monteiro and Svaiter (2013)) .
The iterations of Algorithm 1 satisfy the following inequality:
[Equation]
Note that Algorithm 1 requires finding x k +1 f satisfying condition (8) on line 5.
This condition can be rewritten as follows:
[Equation]
1: input: x 0 = x [Equation] d 2: parameters: M > 0 , K ∈ { 1 , 2 , . . . } 3: β − 1 =4: for k = 0 , 1 , 2 , . . .
, K − 1 do 5: compute 
[Equation]
k >satisfying (17)
[Equation]
∈ R d satisfying (15) x
[Equation]
[Equation]
satisfying (9) η k , β k >satisfying (10) 6: x k +1 = x
k − η k ∇ f ( x k +1 f ) 7: end for 8: output: x Kf where function A λ ( · ; z ): R d → R for λ > 0 and z ∈ R d is defined as
[Equation] [Equation]
3.2 Application to High-Order Minimization
In order to perform the computation on line 5 of Algorithm 1, we need to find x k +1 f ∈ R d that satisfies condition (8).
As we mentioned earlier, condition (8) is equivalent to (12), which involves the gradient norm ∥∇ A λ k ( · ; x kg ) ∥ at point x k +1 f .
Function A λ k ( · ; x kg ) has L p -Lipschitz p -th order derivatives for p ≥ 2 due to its definition (13) and Assumption 1.[-
-]
Hence, it has the following upper bound, thanks to Theorem 1 of Nesterov (2021a):
[Equation]
It turns out that x k +1 f can be obtained by minimizing this upper bound:
[Equation] [Equation]
where M > L p .
Indeed, by Lemma 1 of Nesterov (2021a), we have
[Equation] [Equation]
Hence, to satisfy condition (12), we choose λ k in the following way:
[Equation]
Here, the upper bound on λ k ensures condition (12), while the lower bound prevents stepsize λ k from being too small, which would hurt the convergence rate.
The resulting near-optimal tensor method is formalized as Algorithm 2.
It has the following convergence rate:
[Equation]
where K is the number of iterations.
The proof of this convergence rate involves condition (17) and
Theorem 1.
It is given in the works of Gasnikov et al. (2019a);
Bubeck et al.
; Jiang et al.
[Equation]
[Equation]
We require the strict inequality to ensure the uniform convexity of upper bound (14), which implies the uniqueness and the existence of the minimizer in (15).
3.3 The Problems with
the Existing Algorithms
Algorithm 2 requires finding λ k satisfying condition (17) at each iteration.
According to line 5 of
Algorithm 2, λ k depends on x k +1 f via (17), which depends on x kg via (15), which depends on η k , β k via (9), which depend on λ k via (10).
Hence, computation of stepsize λ k depends on λ k itself and there is no explicit way to perform the computation on line 5.
The algorithms of Gasnikov et al. (2019a); Bubeck et al.
; Jiang et al.
(2019) use various binary search procedures to find λ k and perform the computation on line 5.
However, such procedures are costly and require many iterations to converge.
For instance, Bubeck et al.
(2019) show that their variant of binary search requires the following number of p -th order oracle calls to find λ k satisfying condition (17):
[Equation]
The same complexity (up to constant factors) for similar binary search procedures was established in the works of Nesterov (2021b) ; Jiang et al.
(2019), and in the work of Monteiro and Svaiter (2013) for the p = 2 case.
Hence, the total oracle complexity of Algorithm 2 is O (cid:0) ϵ − 2 / (3 p +1) log(1 /ϵ ) (cid:1) which does not match the lower bound of Arjevani et al.
The additional logarithmic factor in the oracle complexity of Algorithm 2 raises the question whether it is superior to the accelerated tensor method of Nesterov (2021a) in practice.
On the one hand,
Gasnikov et al. (2019a) provided an experimental study that showed the practical superiority of
Algorithm 2 over the algorithm of Nesterov (2021a).
However, this experimental comparison is utterly unfair because it considers only the iteration complexity of the algorithms, which does not take into account the oracle complexity of the binary search procedure.
The First Optimal Tensor Method
In the previous section, we described the main issues with the existing high-order methods that prevent them from being optimal and practical algorithms for solving problem (1).
In this section, we will show how to construct an algorithm that does not have those issues.
More precisely, we will develop the first optimal p -th order algorithm ( p ≥ 2 ) for solving main problem (1).
4.1 The Key Idea
Gasnikov et al. (2019a); Bubeck et al.
; Jiang et al.
(2019) used the following approach while creating their near-optimal algorithms: they fixed the procedure of computing x k +1 f on line 5 of Algorithm 1 using formula (15) and then developed the procedure for computing λ k , which turned out to be inefficient.
We will go the opposite way.
That is, we choose parameters
λ k in advance in such a way that they ensure the optimal convergence rate and then provide an efficient procedure for finding x k [-+
-]+1 f satisfying condition (8).
Let η k be defined as follows:
[Equation]
where η > 0 is a parameter.
Using (10), we can compute β k and λ k as follows:
[Equation]
The following lemma provides a lower bound on β k and an upper bound on λ k .
Lemma 1.
Parameters β k and λ k defined by (21) satisfy the following inequalities:
[Equation]
Lemma 1 and Theorem 1 immediately imply the convergence rate O
(1 /k (3 p +1) / 2 ) , which matches the lower bound of Arjevani et al.
Hence, the only remaining question is how to compute x k {++
+} f satisfying (8) efficiently.
To be precise, we need to develop a procedure that can perform this computation using O (1) 1: input [Equation] 2: parameters: M >3: t = −4: repeat 5: t =
[Equation]
+6: compute x k,t +1 / 2 ∈ R d as follows:
[Equation] k [Equation] [Equation]
k
[Equation]
x k
[Equation]
+
[Equation] [Equation] 8: until ∥∇ A k ( x k,t +1 / 2 ) ∥ ≤ σλ − 1 k ∥ x k,t +1 / 2 − x k, 0 ∥ 9: T k
=
t +10: output: x k +1 f
= x k,T k − 1 / 4.2 Tensor Extragradient Method for Gradient Norm Reduction In this subsection, we develop an efficient procedure for computing x k +1 f satisfying condition (8). As we mentioned earlier, condition (8) is equivalent to (12), which is an upper bound on the gradient norm ∥∇ A λ k ( · ; x kg ) ∥ at point x k +1 f .
Hence, we need an algorithm for the gradient norm reduction in the following smooth high-order convex minimization problem:
[Equation]
In this subsection, we provide such an algorithm.
We call the algorithm Tensor Extragradient Method.
It is formalized as Algorithm 3.
In the case p = 1 , this algorithm recovers the extragradient method of Korpelevich (1976).
Algorithm 3 can be seen as a generalization of the extragradient method for high-order optimization.
One can observe that due to line 8 of Algorithm 3, x k +1 f
= x k,T k − 1 / 2 satisfies condition (12), where x k,T k − 1 / 2 is the output of Algorithm 3.
This is exactly what we need.
The following theorem provides an upper bound on the number of iterations T k required by Algorithm 3 to terminate and produce the output x k +1 f .
Theorem 2.
Let M satisfy M
[Equation]
Then step (23) on line 6 of Algorithm 3 is well defined and the number of iterations T k performed by
Algorithm 3 is upper-bounded as follows: [Equation] [Equation]
where C p is defined as
[Equation]
Algorithm 3 and Theorem 2 will further be used for the construction of the optimal high-order algorithm for solving problem (1).
It is worth mentioning the potential alternatives to Algorithmthat we could use for gradient norm reduction.
For instance, we could use the tensor method of
Nesterov (2021a).
However, the upper bound on the number of iterations for this method would involve the diameter of the level set of function A λ k ( · ; x kg ) rather than the distance to the solution ∥ x kg − x k, ∗ ∥ .
This would be an obstacle towards development of the optimal algorithm.
Alternatively, we could use the accelerated tensor method of Nesterov (2021a).
It turns out that it would work as we need.
Moreover, the upper bound on the number of iterations would be even better than (26).
However, we find the accelerated tensor method of Nesterov (2021a) to be too complicated, which could make the resulting optimal high-order method hard to implement.
On the other hand, it would not give us any benefits for the construction of the optimal high-order method compared to
Algorithm 3. 1: input: x 0 = x 0 f ∈ R d 2: parameters: η > 0 , M > 0 , σ ∈ (0 , 1) , K ∈ { 1 , 2 , . . .
} 3: β − 1 =4: for k = 0 , 1 , 2 , . . .
, K − 1 do 5: η k = η
(1 + k ) (3 p − 1) /6:
β k = β k − 1 + η k , λ k = η 2 k /β k , α k
= η k /β k 7: x kg =
α k x k + (1 − α k ) x
kf 8: x k, 0 = x kg , t = −9: repeat 10: t =
t +11: x k,t +1 / 2 = arg min x ∈ R d
Φ pA λk ( · ; x kg ) ( x ; x k,t ) + pM ( p +1)!
∥ x − x k,t ∥ p + 12: x k,t +1 = x k,t − (cid:16) M ∥ x k,t +1 / 2 − x k,t ∥ p − 1 ( p − 1)!
(cid:17) − 1 ∇ A λ k
( x k,t +1 / 2 ; x kg ) 13: until ∥∇ A λ k ( x k,t +1 / 2 ; x kg ) ∥ ≤ σλ − 1 k
∥ x k,t +1 / 2 − x k, 0 ∥ 14:
T k = t +15: x k +1 f
= x k,T k − 1 /16:
x k +
=
x
k − η k ∇ f ( x k +1 f ) 17: end for 18: output: x Kf
4.3 Modification of the Analysis of A-HPE Framework
Unfortunately, we cannot use Theorem 1 for the analysis of our optimal algorithm.
This is because inequality (11) involves the distances ∥ x kg − x k +1 f ∥ on the right-hand side.
Hence, inequality (11) does not allow us to estimate the iteration complexity T k of Algorithm 3 using Theorem 2.
Further, we provide a new theorem that includes the analysis of the A-HPE framework and provides an upper bound on the distances ∥ x kg − x k, ∗ ∥ .
Theorem 3.
The iterations of Algorithm 1 satisfy the following inequality:
[Equation]
4.4 The First Optimal Tensor Method
Now, we are ready to provide the first optimal high-order algorithm for solving problem (1).
In order to construct this algorithm, we use our Tensor Extragradient Method (Algorithm 3) to perform the computations on line 5 of the A-HPE Framework (Algorithm 1).
We also use our choice of parameters η k , β k and λ k which is provided by (20) and (21).
The resulting algorithm is formalized as Algorithm 4.
Now, we are ready to prove that Algorithm 4 is an optimal algorithm.
First, we need to establish an upper bound on the number of iterations T k performed by the inner repeat-loop of Algorithm 4.
This is done by the following theorem.
Theorem 4.
Let M satisfy (25) .
Then, the following inequality holds for Algorithm 4:
[Equation]
where C p is defined by (27) .
Theorem 4 implies that with a proper choice of the parameter η , Algorithm 4 performs O (1) p -th order oracle calls per iteration on average.
Indeed, let η be chosen as follows:
[Equation]
Then, Theorem 4 immediately implies
[Equation]
Finally, the following theorem establishes the total p -th order oracle complexity of Algorithm 4.
Theorem 5.
Let M
=
L p and σ = 1 / 2 .
Let η be defined by (30) .
Then, to reach precision f ( x kf ) − f ∗ ≤ ϵ , Algorithm 4 requires no more than the following number of p -th order oracle calls:
[Equation]
where D p is defined as follows:
[Equation]
Theorem 5 shows that the total p -th order oracle complexity of Algorithm 4 is O (cid:16)(cid:0)
[Equation]
p
[Equation]
/ϵ
[Equation]
This oracle complexity matches the lower bounds of Arjevani et al.
up to a universal constant that does not depend on R , L p and ϵ .
Hence, Algorithm 4 is indeed the first optimal high-order algorithm for solving smooth convex minimization problems.
4.5 Practical Performance
In this paper we provide an experimental comparison of the proposed optimal high-order algorithm for solving smooth convex minimization problems (Algorithm 4) with the existing near-optimal high-order method of Gasnikov et al. (2019a);
Bubeck et al.
; Jiang et al.
(Algorithm 2).
In summary, the experiments show that the proposed optimal Algorithm 4 is indeed a practical algorithm which significantly outperforms the existing near-optimal Algorithm 2.
The details can be found in Appendix F.
Acknowledgments
The work of Alexander Gasnikov was supported by the strategic academic leadership program ‘Priority 2030’
(Agreement 075-02-2021-1316 30.09.2021).
Appendix A Proof of Lemma
The lower bound on β k can be obtained in the following way
[Equation]
Upper bound on λ k is obtained using the lower bound on β k and (10).
B Proof of Theorem
[Equation]
From (23) online 6 of Algorithm
[Equation]
we have
[Equation]
p
[Equation]
[Equation]
Plugging this into the previous equation and using line 7 of Algorithm 3, we get
[Equation]
where γ k,t = (cid:16) M ∥ x
[Equation]
[Equation]
/ 2 − x k,
[Equation]
∥ p
[Equation]
( p −
[Equation]
(cid:17) − 1 .
Using the convexity of function A k ( x ) , we get ∥ x k,t +
[Equation]
x k,
[Equation]
∥
[Equation]
=
[Equation]
x k
[Equation]
x k, ∗ ∥ 2 −
[Equation]
γ k,t ( A k ( x k,t +1 / 2 ) − A k ( x k, ∗ ))
[Equation]
∥≤ ∥ x
[Equation]
∥
[Equation]
γ
[Equation]
k
[Equation]
pA
[Equation]
k
[Equation]
+
[Equation]
A k
[Equation]
x k
[Equation]
t +1 /
[Equation]
∥
[Equation]
− ∥ x k
[Equation]
t − x k
[Equation]
+
[Equation]
∥
[Equation]
Using inequality (1.6) of Nesterov (2021a), we get ∥
[Equation]
p !
(cid:19)
[Equation]
∥
[Equation]
2 p − ∥ x k,t − x k,t +1 / 2 ∥= ∥ x k,t − x k, ∗ ∥ 2 − (cid:32) 1 − (cid:18) γ k,t L p p !
∥ x k,t +1 / 2 − x k,t ∥
p − 1 (cid:19) 2 (cid:33) ∥ x k,t − x k,t +1 / 2 ∥= ∥ x k,t − x k, ∗ ∥ 2 − (cid:32) 1 − (cid:18)
L p ( p − 1)!
p !
M (cid:19) 2 (cid:33) ∥ x k,t − x k,t +1 / 2 ∥= ∥ x k,t − x k, ∗ ∥ 2 − (cid:32) 1 − (cid:18)
L p pM (cid:19) 2 (cid:33) ∥ x k,t − x k,t +1 / 2 ∥ 2 .
Using Lemma 1 of Nesterov (2021a), we get
[Equation]
After telescoping and rearranging, for T ≤ T k we get
[Equation]
Taking both sides of the inequality in the power of p/ 2 gives ∥ x k,
[Equation]
− x k, ∗
[Equation]
After rearranging, we get
[Equation]
Now, let us prove upper bound (26) by a contradiction.
Suppose that (26) is not true.
Hence,
[Equation]
This implies
[Equation]
where c = (1 + σ − 1 ) − 1 and T = T k − 1 .
Using the λ − 1 k -strong convexity of A k ( x )
[Equation]
we get 0 ≥
[Equation]
t ∈{
[Equation]
Dividing by 1 − c gives
[Equation]
Plugging c
=
(1 + σ − 1 ) − 1 gives
[Equation]
This means that the inner repeat-loop of Algorithm 3 terminated after no more than T iterations, which contradicts with T k = T + 1 .
This concludes the proof.
C Proof of Theorem
Here we also provide the proof of Theorem 1 for completeness.
Using line 6 of Algorithm 1, we get
[Equation]
Using (9), we get x k = α − 1 k x kg − α − 1 k (1 − α k ) x kf , which implies
[Equation]
Using the convexity of f ( x ) and (10)
[Equation]
we get
[Equation]
Using (9), we get
[Equation]
Using the parallelogram rule, we get
[Equation]
Using (9) and (10), we get
[Equation]
Using (8)
[Equation]
we get
[Equation]
Now
[Equation]
let us bound ∥ x kg − x k, ∗ ∥ using λ − 1 k -strong convexity of A λ k ( · ; x kg ) and (8):
[Equation]
Plugging this into the previous inequality gives
[Equation]
Rearranging and telescoping concludes the proof.
D Proof of Theorem
Using Theorem 2, we get
[Equation]
Let us choose parameters τ 0 , . .
, τ K − 1 as follows:
[Equation]
Then, we have
[Equation]
Note that parameters τ k satisfy
K
[Equation]
Hence, using the convexity of function ( · ) p/ ( p − 1) , we get
[Equation]
p − 1 ∥
[Equation]
kg
[Equation]
x k, ∗ ∥
[Equation]
Using Lemma 1, we get (cid:32) K − 1 (cid:88) k
=
[Equation]
T
[Equation]
[Equation]
Using the definition of τ k , we get (cid:32) K − 1 (cid:88)
[Equation] [Equation] (cid:33) [Equation]
K
[Equation]
(cid:88)
[Equation]
[Equation]
k )
[Equation]
∥ x
[Equation]
∥
[Equation]
Using the inequality
K − 1 (cid:88) l
= (1 + l ) p − 1 ≤
K − 1 (cid:88) l
= (cid:90)0 (1 + l + t ) p − 1 dt =
K − 1 (cid:88) l
[Equation]
(cid:90) l +
[Equation]
[Equation]
p − 1 dt = (cid:90) K
+  t p − 1 dt =
(1 + K ) p −p ≤p (
[Equation]
K
[Equation]
p
[Equation]
we get
[Equation]
From (9) and Lemma 1, we get
[Equation]
Hence, (cid:32) K − 1 (cid:88) k
=
[Equation]
Using Theorem 3, we get
[Equation]
k =
[Equation]
T
[Equation]
[Equation]
pp
[Equation]
[Equation]
After taking both sides of the inequality in the power of p − 1 p we get
K − 1 (cid:88)
[Equation]
σ
[Equation]
· (cid:18)
[Equation]
σ
[Equation]
σ (cid:19) p
[Equation]
12 (cid:33) 2 p .
After rearranging, we get
K
[Equation]
k
[Equation]
(cid:19)
[Equation]
12 (cid:33)
[Equation]
p .
E Proof of Theorem
Theorem 3 implies f ( x K f ) − f ∗ ≤ R 2 / (2 β K − 1 ) .
Using Lemma 1, we get
[Equation]
Choosing K
= (cid:24)(cid:16)
[Equation]
p +1) [Equation] 2 4 ηϵ (cid:17) 2 3 p +1 (cid:25) implies f ( x Kf ) − f ∗ ≤ ϵ . Hence, we have the following upper bound on the total iteration complexity of Algorithm 4:
[Equation]
Plugging η defined by (30) gives [Equation] √ [Equation]
+
[Equation]
Using the definition of C p ( M, σ ) , we get K ≤ (cid:32) (3 p + 1) p +1 R p + 2 p [Equation] pϵ ·
[Equation]
σ
[Equation]
σ (cid:19) p
[Equation]
[Equation]
p
[Equation]
pM − L
[Equation]
pM [Equation] L
[Equation] ) p [Equation] −
[Equation] (cid:33) [Equation] p
[Equation] + 1 . Using M = L p , we get
K ≤ (cid:32) (3 p + 1) p +1 R p + 2 p +2 √ pϵ · (cid:18) 1 + σ 1 − σ (cid:19) p −· p p
L p (1 + σ − 1 ) p !
[Equation]
Using σ = 1 /
[Equation]
we get
[Equation]
Finally
[Equation]
we have that Algorithm 4 performs (1 + 2 T k ) of p -th order oracle calls at each iteration.
Hence, using (31) [Equation]
we get following upper bound on the total oracle complexity:
[Equation]
F Experiments
In this section we perform a simple numerical comparison of our Optimal Tensor Method (Algorithm 4) with the Near-Optimal Tensor Method of Gasnikov et al. (2019a); Jiang et al.
; Bubeck et al. (Algorithm 2).
We focus on the second-order case ( p = 2 ) because it is already a highly important case.
We leave investigating higher-order cases ( p ≥ 3 ) for future work.
The main goal of our experimental comparison is to illustrate the practical benefits of eliminating the binary search procedure.
We perform our experiment with logistic regression for binary classification.
That is, our objective function f ( x ) has the form
[Equation]
where a i ∈ R d and b i ∈ {− 1 , +1 } are data points and labels, and n is the number of data points.
In the experiment, we use the a9a dataset from the LIBSVM 9 dataset collection. This dataset has n = 32561 training samples with d = 123 features.
We implement both Algorithm 4 and Algorithm 2 in Python language with the help of JAX API.
To perform computation on line 5 of Algorithm 2, we use the variant of the binary search procedure described by Bubeck et al.
To perform the computation of the tensor step (7) in both algorithms, we use a procedure based on the eigenvalue decomposition of the Hessian matrix.
For a fair comparison, we use the same value of parameter M in both algorithms.
Furthermore, we choose the parameter η in Algorithm 4 in such a way that both Algorithm 2 and Algorithm 4 have almost identical convergence in the number of iterations k . It is illustrated in Figure 1. It is not a surprise that both algorithms have very similar convergence curves (with our choice of η ) since they are based on the A-HPE Framework (Algorithm 1). However, the difference between Algorithms 2 and 4 becomes clear when we compare the number of oracle calls performed at each iteration k .
It is illustrated in Figure 2 from which we can make the following conclusions:
• One can observe that the number of oracle calls performed at each iteration k by Algorithmslightly oscillates around a constant.
This is perfectly aligned with our theory which shows that the average number of oracle calls performed at each iteration k by Algorithm 4 is bounded by a constant at all times (see, for instance, inequality (31)). • At the same time, one can observe that the number of oracle calls performed by the binary search procedure at each iteration k of Algorithm 2 slowly grows with the number of iterations k . This is also perfectly aligned with the theory of Bubeck et al. ; Jiang et al.
; Nesterov (2021b) which suggests that the number of iterations performed by the binary search procedure at each iteration k grows as O (log k ) (in fact, the blue curve in Figure 2 resembles this logarithmic dependency). Overall, Figure 2 shows that Algorithm 2 has to perform substantially more oracle calls at each iteration k than Algorithm 4 due to the binary search procedure. The aforementioned major difference between the algorithms results in a significantly slower convergence of Algorithm 2 in the total number of oracle calls compared to Algorithm 4. It is illustrated in Figure 3 which shows that Algorithm 2 requires approximately 2 times more oracle calls than
Algorithm 4 to reach accuracy ∥∇ f ( x ) ∥ 2 ≤ 10 − 15 . This difference becomes even more dramatic if we compare both algorithms in the wall clock time which is shown in Table 2. One can observe that Algorithm 4 converges approximately 4 times faster than Algorithm 2.
In summary, our illustrative experiment shows that the proposed Optimal Tensor Method (Algorithm 4) is indeed a practical algorithm. Moreover, it shows that the necessity to use the binary search procedure by the Near-Optimal Tensor Method (Algorithm 2) results in a significantly slower convergence in
[Table]
both the number of oracle calls and the wall clock time, which is perfectly aligned with the existing theory on the binary search procedure (Bubeck et al., 2019; Jiang et al., 2019; Nesterov, 2021b) and our new theory for our new Algorithm 4. [Figure] [Figure]
[Figure]