O N THE L OWER B OUND OF M INIMIZING P OLYAKŁ OJASIEWICZ FUNCTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Polyak-Łojasiewicz (PL) (Polyak, 1963) condition is a weaker condition than the strong convexity but suffices to ensure a global convergence for the Gradient Descent algorithm.
In this paper, we study the lower bound of algorithms using first-order oracles to find an approximate optimal solution.
We show that any first-order algorithm requires at least Ω (cid:16) Lµ log 1 ε (cid:17) gradient costs to find an εapproximate optimal solution for a general L -smooth function that has an µ -PL constant.
This result demonstrates the optimality of the Gradient Descent algorithm to minimize smooth PL functions in the sense that there exists a “hard” PL function such that no first-order algorithm can be faster than Gradient Descent when ignoring a numerical constant.
In contrast, it is well-known that the momentum technique, e.g. (Nesterov, 2003, chap. 2) can provably accelerate Gradient Descent to O (cid:16)(cid:113)
L ˆ µ log 1 ε (cid:17) gradient costs for functions that are L -smooth and ˆ µ -strongly convex.
Therefore, our result distinguishes the hardness of minimizing a smooth PL function and a smooth strongly convex function as the complexity of the former cannot be improved by any polynomial order in general.
I NTRODUCTION
We consider the problem
[Equation]
where the function f is L -smooth and satisfies the Polyak-Łojasiewicz condition.
A function f is said to satisfy the Polyak-Łojasiewicz condition if (2) holds for some µ > 0 :
[Equation]
We refer to (2) as the µ -PL condition and simply denote inf
y ∈ R
d f ( y ) by f ∗ .
The PL condition may be originally introduced by Polyak (Polyak, 1963) and Łojasiewicz (Lojasiewicz, 1963) independently.
The PL condition is strictly weaker than strong convexity as one can show that any ˆ µ -strongly convex function which by definition satisfies:
[Equation]
is also ˆ µ -PL by minimizing both sides with respect to x (Karimi et al., 2016).
However, the PL condition does not even imply convexity.
From a geometric view, the PL condition suggests that the sum of the squares of the gradient dominates the optimal function value gap, which implies that any local stationary point is a global minimizer.
Because it is relatively easy to obtain an approximate local stationary point by first-order algorithms, the PL condition serves as an ideal and weaker alternative to strong convexity.
In machine learning, the PL condition has received wide attention recently.
Lots of models are found to satisfy this condition under different regimes.
Examples include, but are not limited to, matrix decomposition and linear neural networks under a specific initialization (Hardt & Ma, 2016;
Li et al., 2018), nonlinear neural networks in the so-called neural tangent kernel regime (Liu et al., , reinforcement learning with linear quadratic regulator (Fazel et al., 2018).
Compared with strong convexity, the PL condition is much easier to hold since the reference point in the latter only is a minimum point such that x ∗ =
argmin
y f ( y ) , instead of any y in the domain.
Turning to the theoretic side, it is known (Karimi et al., 2016) that the standard Gradient Descent algorithm admits a linear converge to minimize a L -smooth and µ -PL function.
To be specific, in order to find an ε -approximate optimal solution ˆ x such that f (ˆ x ) − f ∗ ≤ ε , Gradient Decent needs O ( Lµ log 1 ε ) gradient computations.
However, it is still not clear whether there exist algorithms that can achieve a provably faster convergence rate.
In the optimization community, it is perhaps well-known that the momentum technique, e.g. Nesterov (2003, chap. 2), can provably accelerate Gradient Descent from O
( L ˆ µ log 1 ε ) to O (cid:16)(cid:113)
L ˆ µ log 1 ε (cid:17) for functions that are L -smooth and ˆ µ -strongly convex.
Even though some works (J Reddi et al., 2016; Lei et al., 2017) have considered accelerations under different settings, probably faster convergence of first-order algorithms for PL functions is still not obtained up to now.
In this paper, we study the first-order complexities to minimize a generic smooth PL function and ask the question: “Is the Gradient Decent algorithm (nearly) optimal or can we design a much faster algorithm?”
We answer the question in the language of min-max lower bound complexity for minimizing the Lsmooth and µ -PL function class.
We analyze the worst complexity of minimizing any function that belongs to the class using first-order algorithms.
Excitingly, we construct a hard instance function showing that any first-order algorithm requires at least Ω (cid:16) Lµ log 1 ε (cid:17) gradient costs to find an εapproximate optimal solution.
This answers the aforementioned question in an explicit way: the Gradient Descent algorithm is already optimal in the sense that no first-order algorithm can achieve a provably faster convergence rate in general ignoring a numerical constant.
For the first time, we distinguish the hardness of minimizing a PL function and a strongly convex function in terms of firstorder complexities, as the momentum technique for smooth and strongly convex functions provably accelerates Gradient Descent by a certain polynomial order.
It is worth mentioning that the optimization problem under our consideration is high-dimensional and the goal is to obtain the complexity bounds that do not have an explicit dependency on the dimension.
Our technique to establish the lower bound follows from the previous lower bounds in convex (Nesterov, 2003) and non-convex optimization (Carmon et al., 2021).
The main idea is to construct a so-called “zero-chain” function ensuring that any first-order algorithm per-iteratively can only solve one coordinate of the optimization variable.
Then for a “zero-chain” function that has a sufficiently high dimension, some number of entries will never reach their optimal values after the execution of any first-order algorithm in certain iterations.
To obtain the desired Ω (cid:16) Lµ log 1 ε (cid:17) lower bound, we propose a “zero-chain” function similar to Carmon et al.
(2020), which is composed of the worst convex function designed by Nesterov (2003) and a separable function in the form as σ (cid:80)
Ti =1 v T,c ( x i ) or (cid:80)
Ti =1 v y i ( x i ) to destroy the convexity.
Different from their separable function, the one that we introduce has a large Lipshictz constant.
This property helps us to estimate the PL constant in a convenient way.
This new idea gives new insights into the constructions and analyses of instance functions, which might be potentially generalized to establish the lower bounds for other non-convex problems.
N OTATION
We use bold letters, such as x , to denote vectors in the Euclidean space R d , and bold capital letters, such as A , to denote matrices.
I d denotes the identity matrix of size d × d .
We omit the subscript and simply denote I as the identity matrix when the dimension is clear from context.
For x ∈ R d , we use x i to denote its i th coordinate.
We use supp( x ) to denote the subscripts of non-zero entries of x , i.e. supp( x ) =
{ i : x i ̸ = 0 } .
We use span (cid:8) x (1) , · · · , x ( n ) (cid:9) to denote the linear subspace spanned by x (1) , · · · , x ( n ) , i.e. (cid:8) y :
y = (cid:80)
ni
=1 a i x ( i ) , a i ∈ R (cid:9) .
We call a function f L -smooth if ∇ f is L -Lipschitz continuous, i.e. ∥∇ f ( x ) − ∇ f ( y )
∥
≤ L ∥ x − y ∥ .
We denote f ∗ = inf x f ( x ) .
We let x ∗ be any minimizer of f , i.e., x ∗ = argmin f .
We always assume the existence of x ∗ .
We say that x is an ε -approximate optimal point of f when f ( x ) − f ∗ ≤ ε .
R ELATED W ORK
Lower Bounds There has been a line of research concerning the lower bounds of algorithms on certain function classes.
To the best of our knowledge, (Nemirovskij & Yudin, 1983) defines the oracle model to measure the complexity of algorithms, and most existing research on lower bounds follow this formulation of complexity.
For convex functions and first-order oracles, the lower bound is studied in Nesterov (2003), where well-known optimal lower bound Ω( ε − 12 ) and Ω( κ log 1 ε ) are obtained.
For convex functions and n th-order oracles, lower bounds Ω (cid:16) ε − 2 3 n +1 (cid:17) have been proposed in Arjevani et al. (2019b).
When the function is non-convex, it is generally NP-hard to find its global minima, or to test whether a point is a local minimum or a saddle point (Murty & Kabadi, 1985).
Instead of finding ε -approximate optimal points, an alternative measure is finding ε -stationary points where ∥∇ f ( x ) ∥ ≤ ε .
Sometimes, additional constraints on the Hessian matrices of second-order stationary points are needed.
Results of this kind include Carmon et al.
; Fang et al.
; Zhou & Gu (2019);
Arjevani et al.
(2019a; 2020). Though a PL function may be nonconvex, it is tractable to find an ε -approximate optimal point, as local minima of a PL function must be global minima.
In this paper, we give the lower complexity bound for finding ε -approximate optimal points.
PL Condition The PL condition was introduced by Polyak (Polyak, 1963) and Łojasiewicz (Lojasiewicz, 1963) independently.
Besides the PL condition, there are other relaxations of the strong convexity, including error bounds (Luo & Tseng, 1993), essential strong convexity (Liu et al., 2014), weak strong convexity (Necoara et al., 2019), restricted secant inequality (Zhang & Yin, 2013), and quadratic growth (Anitescu, 2000).
Karimi et al.
(2016) discussed the relationships between these conditions.
All these relaxations implies the PL condition except for the quadratic growth, which implies that the PL condition is quite general.
There are many other papers that study designing practical algorithms to optimize a PL objective function under different scenarios, for example, Bassily et al.
; Nouiehed et al.
Hardt & Ma (2016); Fazel et al.
(2018); J Reddi et al.
Lei et al.
P RELIMINARIES 3. U PPER BOUND ON PL FUNCTIONS
Although the PL condition is a weaker condition than strong convexity, it guarantees linear convergence for Gradient Descent.
The result can be found in Polyak (1963) and Karimi et al. (2016).
We present it here for completeness.
Theorem 1.
If f is L -smooth and satisfies µ -PL condition, then the Gradient Descent algorithm with a constant step-size 1 L : [Equation]
[Equation]
has a linear convergence rate.
We have:
[Equation]
Theorem 1 shows that the Gradient Descent algorithm finds the ε -approximate optimal point of f in O (cid:16) Lµ log 1 ε (cid:17) gradient computations.
This gives an upper complexity bound for first-order algorithms.
However, it remains open to us whether there are faster algorithms for smooth PL functions.
We will establish a lower complexity bound on first-order algorithms, which nearly matches the upper bound.
3. D EFINITIONS OF ALGORITHM CLASSES AND FUNCTION CLASSES
An algorithm is a mapping from real-valued functions to sequences.
For algorithm A and f : R d →
R , we define A [ f ] = { x ( i ) } i ∈ N to be the sequence of algorithm
A acting on f , where x ( i ) ∈ R d .
Note here, the algorithm under our consideration works on function defined on any Euclidean space.
We call it the dimension-free property of the algorithm.
The definition of algorithms abstracts away from the the optimization process of a function.
We consider algorithms which only make use of the first-order information of the iteration sequence.
We call them first-order algorithms.
If an algorithm is a first-order algorithm, then
[Equation]
where A ( i ) is a function depending on A .
Perhaps the simplest example of first-order algorithms is
Gradient Descent.
We are interested in finding an ε -approximate point of a function f .
Given a function f : R d → R and an algorithm A , the complexity of A on f is the number of queries to the first-order oracle needed to find an ε -approximate point.
We denote T ε ( A , f ) to be the gradient complexity of A on f , then
[Equation]
In practice, we do not have the full information of the function f .
We only know that f is in a particular function class F , such as L -smooth functions.
Given an algorithm A .
We denote T ε ( A , F ) to be the complexity of A on F , and define T ε ( A , F ) as follows:
[Equation]
Thus, T ε ( A , F ) is the worst-case complexity of functions f ∈ F .
For searching an ε -approximate optimal point of a function in F , we need to find an algorithm which have a low complexity on F .
Denote an algorithm class by A .
The lower bound of an algorithm class on F describes the efficiency of algorithm class
A on function class F , which is defined to be
[Equation]
3. Z ERO - RESPECTING A LGORITHM
Among all the algorithms, a special algorithm class is called zero-respecting algorithms.
If A is a zero-respecting algorithm and A [ f ] = (cid:8) x ( t ) (cid:9) t ∈ N , then the following condition holds for all f :
[Equation]
That is to say, x ( n ) − x (0) lies in the linear subspace spanned by ∇ f ( x (0) ) , · · · , ∇ f ( x ( n − 1) ) .
We denote the collection of first-order zero-respecting algorithms with x (0)
=
by A zr .
It is shown by
Nemirovskij & Yudin (1983) that a lower complexity bound on first-order zero-respecting algorithms are also a lower complexity bound on all the first-order algorithm when the function class satisfies the orthogonal invariance property.
3. Z ERO - CHAIN
A zero-chain f is a function that safisfies the following condition:
[Equation]
In other words, the support of ∇ f ( x ) lies in a restricted linear subspace depending on the support of x . [Figure]
The “worst function in the (convex) world” in Nesterov (2003) defined as
[Equation]
(11) is a zero-chain, because if x i =
for i > n , then ( ∇ f k ( x ))
i +
=
for i > n .
A zero-chain is difficult to optimize for zero-respecting algorithms, because zero-respecting algorithms only discover one coordinate by one gradient computation.
M AIN RESULTS
According to Theorem 1, we already have an upper complexity bound O (cid:16) Lµ log 1 ε (cid:17) by applying
Gradient Descent to all the PL functions.
In this section, we establish the lower complexity bound of first-order algorithms on PL functions.
Let P (∆ , µ, L ) be the collection of all L -smooth and µ -PL functions f with f ( x (0) )
− f ∗ ≤ ∆ .
We establish a lower bound of T ε ( A zr , P (∆ , µ, L )) by constructing a function which is hard to optimize for zero-respecting algorithms, and extend the result to first-order algorithms.
We first propose a relatively simple hard instance which leads to a Ω (cid:0) κ 1 − a (cid:1) lower bound in Section 4.1, where a can be any real number that belongs to (0 , 1) . This helps in understanding our intuitions. Then we present a more complicated hard instance that can achieve the desired Ω (cid:0) κ log 1 ε (cid:1) lower bound in Section 4.2. 4. Ω (cid:0) κ 1 − a (cid:1) LOWER BOUND
We introduce a hard function f T,c,σ : R T → R for first-order algorithms:
[Equation]
where
[Equation]
is a quadratic function, and we define v T,c as follows:
[Equation]
where T is a positive integer, and c, σ are positive real numbers.
The function q can be rewritten as
[Equation]
where
[Equation]
is a positive-definite symmetric matrix.
From the definition of v T,c in (14), we obtain the derivative of v T,c , which is:
[Equation]
As we can see in (17), v ′ T,c is a piecewise linear function.
To further simplify notations, we define b T,c ( x ) in (18):
[Equation]
Then v ′ T,c ( x ) = x − b T,c ( x ) .
Figure 1 provides geometric view of v T,c and b T,c .
The quadratic part q is a translation of “the worst function in the (convex) world” in Nesterov (2003), and the definition of v T,c is inspired by the hard instance in Carmon et al.
Our hard instance differs from previous ones mainly in the large Lipschitz constant of its gradient.
We note that the controlled degree of nonsmoothness is crucial for our estimate of PL constant.
In Lemma 1 we list some important properties of f T,c,σ , which we prove in Section ?? .
Lemma 1.
If σ < 1 and T c ≥ 12 σ − 1 , then f T,c,σ satisfies the following.
g T,c,σ ( x ) =
f T,c,σ ( 1 − x ) is a zero-chain.
x ∗ = 0 , f ∗ T,c,σ = 0 , f T,c,σ ( x ) ≤ 12 x T ( A + σ I ) x . 3.
f T,c,σ is (4 + σ + 32 σT c ) -smooth.
f T,c,σ satisfies the 1 C 1 T 1+5 c -PL condition, where C 1 is a universal constant.
Now we study the lower bound of zero-respecting algorithms first.
Let ˜ f denote the following function:
[Equation]
where D is the distance between 0 and x ∗ , and T , σ are parameters to be specified later.
A change in D affects ˜ f
( 0 ) − ˜ f ( x ∗ ) and ∥ x ∗ − 0 ∥ , but does not affect the condition number of ˜ f .
In Lemma 2, we show that ˜ f is difficult to optimize for all first-order zero-respecting algorithms.
Lemma 2.
If σ < 1 and T c > 12 σ − 1 , a first-order zero-respecting algorithm with x (0) = needs at least T 2 gradient computations to find a point x satisfying ˜ f
( x ) − ˜ f ∗ ≤ 116 ( ˜ f
( x (0) ) − ˜ f ∗ ) .
We give the full proof of Lemma 2 in Appendix B.1 .
The key point is that each gradient access of a zero-respecting algorithm only reveals one coordinate of a zero-chain.
For i > T 2 , x ( k ) i remains unchanged when k ≤ T 2 , which gives a lower bound of the function value after the first T 2 gradient computations.
With Lemma 1 and Lemma 2 in hand, we are ready to give our lower complexity bound for zerorespecting first-order algorithms.
Theorem 2.
Given L ≥ µ > 0 and c < 0 .
When κ = Lµ > C 2 where C 2 is a universal constant, we let [Equation] and [Equation]
Then, ˜ f is L -smooth and µ -PL.
Moreover, any first-order zero-respecting algorithm with x (0) = {+
+}
needs at least Ω (cid:0) κ 1 / (1+6 c ) (cid:1) gradient computations to find a point x satisfying ˜ f
( x ) − ˜ f ∗ ≤ 116 ( ˜ f
( x (0) ) − ˜ f ∗ ) .
The proof of Theorem 2 is provided in Section ?? .
For a satisfying a 6(1 − a ) < 0 .
01 , let c = a 6(1 − a [Equation] , then by Theorem 2, any zero-respecting algorithm needs at least Ω( κ 1 − a ) gradient computations to find an ε -approximate optimal point of the function ˜ f .
Using the technique of Nemirovskij & Yudin (1983), for specific function classes such as PL functions, a lower complexity bound on first-order zero-respecting algorithms is also a lower complexity bound on all the first-order algorithms.
Denoting the set of all first-order algorithms by A (1) , we have the following lemma:
Lemma 3.
[Equation]
Note that given L and µ , the value ˜ f
( 0 ) can be controlled by choosing an appropriate D .
Lemmaand 3 leads to the main result of the paper:
Theorem 3.
For any 0 < a < 1 , when ε ≤ 116 ∆ ,
[Equation]
[Equation] Discussion about parameter setting In the definition of ˜ f , ˜ f has hyper-parameters T , σ , c . From our construction, to achieve a lower bound of Ω( κ 1 − a ) when a tends to zero, c also tends to zero.
Parameters T and σ are chosen according to (20) and (21). When c tends to zero, T = Θ( κ ) and σ tends to 1 . [Equation]
We show that the Ω( κ ) lower bound can be further improved to Ω (cid:0) κ log 1 ε (cid:1) with a new hard instance based on f T,c,σ and a similar technique to estimate the PL constant. Detailed proof of Lemmas and
Theorems in this subsection is provided in Appendix C . We first introduce several components of the new hard instance.
We define [Equation] where y > 0 is a constant.
By the definition of v y , we have
[Equation]
For the non-convex part, we define
[Equation]
[Equation] Then we have v ′ y ( x ) = x − b y ( x ) . For the convex part, we define q T,t ( x ) as follows (for the convenience of notation, we define x 0 = 0 ):
[Equation] where x ∈ R Tt . q T,t is a quadratic function of x , thus can be written as
[Equation] where B is a positive semi-definite symmetric matrix. Like A , B also satisfies 0 ⪯ B ⪯ 4 I , because the sum of absolute value of non-zero entries of each row of B is smaller or equal to 4 . Let y ∈
R Tt be a vector satisfying y qT + b = (cid:0) 78 (cid:1) q , where q ∈ N , b ∈ { 1 , 2 , · · · , T } .
We define the hard instance g T,t : R Tt → R as follows:
[Equation]
Now we list some properties of g T,t in Lemma 4.
Lemma 4.
f T,c,σ satisfies the following. g T,t ( y − x ) is a zero-chain. x ∗ = 0 , g ∗ T,t = 0 , g T,t ( x ) ≤ 12 x T ( B + I ) x . 3.
g T,t is 37 -smooth.
g T,t
satisfies the 1 C 3 T -PL condition, where C 3 is a universal constant.
Define ˜ g to be the following function, which is hard for first-order algorithms:
[Equation]
Similar to Lemma 2, we show that ˜ g is hard for first-order zero-respecting algorithms:
Lemma 5.
Assume that ε < {+0 . 
+} and let t = 2 (cid:106) log 87 32 ε (cid:107) . A first-order zero-respecting algorithm with x {+(0) = 
+}
needs at least 12 Tt gradient computations to find a point x satisfying ˜ g
( x ) − ˜ g ∗ ≤ ε (˜ g ( x (0) )
− ˜ g ∗ ) .
With Lemma 4 and 5, we obtain a lower bound for zero-respeting algorithms:
Theorem 4.
Given L ≥ µ > 0 .
When κ = Lµ > C 4 where C 4 is a universal constant, there exists
T and t such that ˜ g is L -smooth and µ -PL.
Moreover, any first-order zero-respecting algorithm with x {+(0) = 
 needs+} at least Ω (cid:0) κ log 1 ε (cid:1) gradient computations to find a point x satisfying ˜ g
( x ) − ˜ g
∗ ≤ ε (˜ g ( x (0) ) − ˜ g ∗ ) . Finally, we arrive at a lower bound for first-order algorithms using Lemma 3:
Theorem 5.
For any 0 < a < 1 , when ε ≤ 116 ∆ ,
[Equation]
This bound matches the convergence rate of Gradient Descent up to a constant.
[Figure]
We conduct numerical experiments on our hard instance. We consider the κ relatively large, which can reduce the factors from the numerical constants.
{+We κ = 4 . 
+}
{+× 10 8 , c = 0 . 001 in Figure 2a and κ = 4 . 
 × 10 10 , c = 0 .+}
001 in Figure 2b, then compute corresponding T and σ according to (20) and (21).
We use Gradient Descent, Nesterov’s Accelerated Gradient Descent (AGD) and Polyak’s Heavy-ball Method to optimize the hard instance.
As AGD and the Heavy-ball Method are designed for convex functions, we need to choose appropriate parameter ˆ µ in both algorithms, because our hard instance is non-convex.
The first choice is µ , which is the PL constant of our hard instance.
The second choice is σ , because when x i is far from 1 , our hard instance can be treated as a σ -strongly convex quadratic function. From Figure 2,we observe that the convergence can be roughly divided into two phases.
In phase one, the optimization methods tries to pull every coordinate of ˜ x away from 1 , and the function value does not decrease much. In phase two they converge linearly.
This may be due to the fact that when each ˜ x
i is far from 1 , our hard instance is exactly a strongly convex quadratic function.
The observation is consistent with our theoretical results.
C ONCLUSION
We construct a lower complexity bound on optimizing smooth PL functions with first-order methods.
A first-order algorithm needs at least Ω (cid:16) Lµ log 1 ε (cid:17) gradient access to find an ε -approximate optimal point of an L -smooth µ -PL function.
Our lower bound matches the convergence rate of Gradient Descent up to constants.
We only focus on deterministic algorithms in this paper.
We conjecture that our results can be extended to randomized algorithms, using the same technique in Nemirovskij & Yudin (1983) and explicit construction in Woodworth & Srebro (2016) and Woodworth & Srebro (2017).
We leave its formal derivation to the future work.