A N OVEL A NALYSIS F RAMEWORK OF L OWER C OMPLEXITY B OUNDS FOR F INITE -S UM O PTIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper studies the lower bound complexity for the optimization problem whose objective function is the average of n individual smooth convex functions.
We consider the algorithm which gets access to gradient and proximal oracle for each individual component.
For the strongly-convex case, we prove such an algorithm can not reach an ε -suboptimal point in fewer than Ω(( n + √ κn ) log(1 /ε )) iterations, where κ is the condition number of the objective function.
This lower bound is tighter than previous results and perfectly matches the upper bound of the existing proximal incremental ﬁrst-order oracle algorithm Point-SAGA.
We develop a novel construction to show the above result, which partitions the tridiagonal matrix of classical examples into n groups to make the problem difﬁcult enough to stochastic algorithms.
This construction is friendly to the analysis of proximal oracle and also could be used in general convex and average smooth cases naturally.
I NTRODUCTION
We consider the minimization of the following optimization problem
[Equation]
) where the f i ( x ) are L -smooth and µ -strongly convex.
Accordingly, the condition number is deﬁned as κ = L/µ , which is typically larger than n in real-world applications.
Many machine learning models can be formulated as the above problem such as ridge linear regression, ridge logistic regression, smoothed support vector machines, graphical models, etc.
This paper focuses on the ﬁrst order methods for solving Problem (1), which access to the Proximal Incremental First-order Oracle (PIFO) for each individual component, that is,
[Equation]
where i ∈ { 1 , . . . , n } , γ > 0 , and the proximal operation is deﬁned as
[Equation]
We also deﬁne the Incremental First-order Oracle (IFO)
[Equation]
i
[Equation]
x )] .
PIFO provides more information than IFO and it would be potentially more powerful than IFO in ﬁrst order optimization algorithms.
Our goal is to ﬁnd an ε -suboptimal solution ˆ x such that
[Equation]
by using PIFO or IFO.
There are several ﬁrst-order stochastic algorithms to solve Problem (1).
The key idea to leverage the structure of f is variance reduction which is effective for ill-conditioned problems.
For example, SVRG (Zhang et al., 2013; Johnson and Zhang, 2013; Xiao and Zhang, 2014) can ﬁnd an ε -suboptimal solution in O (( n + κ ) log(1 /ε ))
IFO calls, while the complexity of the classical Nesterov’s acceleration (Nesterov, 1983) is O ( n √ κ log(1 /ε )) .
Similar results 1 also hold for SAG (Schmidt et al., 2017) and SAGA (Defazio et al., 2014).
In fact, there exists an accelerated stochastic gradient method with √ κ dependency.
Defazio (2016) introduced a simple and practical accelerated method called Point SAGA, which reduces the iteration complexity to
O (( n + √ κn ) log(1 /ε )) .
The advantage of Point SAGA is in that it has only one parameter to be tuned, but the iteration depends on PIFO rather than IFO.
Allen-Zhu (2017) proposed the Katyusha momentum to accelerate variance reduction algorithms, which achieves the same iteration complexity as Point-SAGA but only requires IFO calls.
The lower bound complexities of IFO algorithms for convex optimization have been well studied (Agarwal and Bottou, 2015; Arjevani and Shamir, 2015; Woodworth and Srebro, 2016; Carmon et al., 2017; Lan and Zhou, 2017; Zhou and Gu, 2019).
Speciﬁcally, Lan and Zhou (2017) showed that at least Ω(( n + √ κn ) log(1 /ε ))
IFO calls 2 are needed to obtain an ε -suboptimal solution for some complicated objective functions.
This lower bound is optimal because it matches the upper bound complexity of Katyusha (Allen-Zhu, 2017).
It would be interesting whether we can establish a more efﬁcient PIFO algorithm than IFO one.
Woodworth and Srebro (2016) provided a lower bound Ω( n + √ κn log(1 /ε )) for PIFO algorithms, while the known upper bound of the PIFO algorithm Point SAGA [3] is O (( n + √ κn ) log(1 /ε )) .
The difference of dependency on n implies that the existing theory of PIFO algorithm is not perfect.
This gap can not be ignored because the number of components n is typically very large in many machine learning problems.
A natural question is can we design a PIFO algorithm whose upper bound complexity matches Woodworth and Srebro’s lower bound, or can we improve the lower bound complexity of PIFO to match the upper bound of Point SAGA.
In this paper, we prove the lower bound complexity of PIFO algorithm is Ω(( n + √ κn ) log(1 /ε )) for smooth and strongly-convex f i , which means the existing Point-SAGA (Defazio, 2016) has achieved optimal complexity and PIFO can not lead to a tighter upper bound than IFO.
We provide a novel construction, showing the above result by decomposing the classical tridiagonal matrix (Nesterov, 2013) into n groups.
This technique is quite different from the previous lower bound complexity analysis (Agarwal and Bottou, 2015; Woodworth and Srebro, 2016; Lan and Zhou, 2017; Zhou and
Gu, 2019).
Moreover, it is very friendly to the analysis of proximal operation and easy to follow.
We also use this technique to study general convex and average smooth cases (Allen-Zhu, 2018;
Zhou and Gu, 2019), and extend our result to non-convex problems (see Appendix J).
O UR A NALYSIS F RAMEWORK
In this paper, we consider the Proximal Incremental First-order Oracle (PIFO) algorithm for smooth convex ﬁnite-sum optimization.
All proofs in this section can be found in Appendices C and D for a detailed version.
We analyze the lower bounds of the algorithms when the objective functions are respectively strongly convex, general convex, smooth and average smooth (Zhou and Gu, 2019).
Deﬁnition 2.1.
For any differentiable function f : R m → R ,
• f is convex, if for any x , y ∈
R
m it satisﬁes f ( y ) ≥ f
( x ) + (cid:104)∇ f ( x ) , y − x (cid:105) .
• f is µ -strongly convex, if for any x , y ∈
R
m it satisﬁes f ( y ) ≥ f
( x ) + (cid:104)∇ f ( x ) , y − x (cid:105)
+ µ 2 (cid:107) x
− y (cid:107) 22 .
• f is L -smooth, if for any x , y ∈
R m it satisﬁes (cid:107)∇ f ( x ) − ∇ f ( y ) (cid:107) 2 ≤ L (cid:107) x − y (cid:107) 2 .
[Table]
Deﬁnition 2.2.
We say differentiable functions { f i } ni =1 , f i : R m → R , to be L -average smooth if for any x , y ∈
R m , they satisfy
[Equation]
Remark 2.3.
We point out that
if each f i is L -smooth, then { f i } n i =1 are L -average smooth.
if { f i } ni =1 are L -average smooth, then f ( x ) = 1 n (cid:80) ni =
f i ( x ) is L -smooth.
We present the formal deﬁnition for PIFO algorithm.
Deﬁnition 2.4.
Consider a stochastic optimization algorithm
A to solve Problem (1).
Let x t be the point obtained at time-step t and the algorithm starts with x 0 .
The algorithm A is said to be a PIFO algorithm
if for any t ≥ 0 , we have
[Equation]
where i t is a random variable supported on [ n ] and takes P ( i t = j ) =
p j for each t ≥ 0 and 1 ≤ j ≤ n where (cid:80) nj =1 p j = 1 .
Without loss of generality, we assume x 0 = 0 and p 1 ≤ p 2 ≤ · · · ≤ p n to simplify our analysis.
Otherwise, we can take { ˆ f i ( x ) =
f i ( x + x 0 ) } ni =1 into consideration.
On the other hand, suppose that p s 1 ≤ p s 2 ≤ · · · ≤ p s n where { s i } ni =1 is a permutation of [ n ] .
Deﬁne { ˜ f i } ni =1 such that ˜ f s i
=
f i , then A takes component ˜ f
s i in probability p
s i , i.e., A takes f i in probability p s i .
To demonstrate the construction of adversarial functions, we ﬁrst introduce the following class of matrices:
[Equation]
Then we deﬁne
[Equation]
The matrix A ( m, ω ) is widely-used in the analysis of lower bounds for convex optimization (Nesterov, 2013; Agarwal and Bottou, 2015; Lan and Zhou, 2017; Carmon et al., 2017; Zhou and Gu, 2019).
We now present a decomposition of A ( m, ω ) based on Eq.
Denote the l -th row of the matrix B ( m, ω ) by b l ( m, ω ) (cid:62) and let
[Equation]
Our construction is based on the following class of functions
[Equation]
where
[Equation]
l ∈L i (cid:13)(cid:13)
b l ( m, ω ) (cid:62) x
(cid:13)(cid:13) 2 2 + λ 2 (cid:107) x (cid:107) 22 , for i = 2 , 3 , · · · , n. (6)
We can determine the smooth and strongly-convex coefﬁcients of r i as follows.
Proposition 2.5.
For any λ 1 > 0 , λ 2 ≥ 0 , ω < √ 2 , we have that the r i are (4 λ 1 + 2 λ 2 ) -smooth
and λ 2 -strongly convex, and { r i } ni =1 is L (cid:48)
-average
smooth where
[Equation]
We deﬁne the subspaces {F k } mk =0 as
F
[Equation]
= (cid:26) span { e m , e m − 1 , · · · , e m − k +1 } , for 1 ≤ k ≤ m, { 0 } , for k = 0 .
The following technical lemma plays a crucial role in our proof.
Lemma 2.6.
For any λ 0 (cid:54) = 0 , λ 1 > 0 , λ 2 ≥ 0 and x ∈ F k , 0 ≤ k < m , we have that ∇ r i ( x ; λ 0 , λ 1 , λ 2 , m, ω ) and prox
γr i ( x ) ∈ (cid:26) F k +1 , if k ≡ i − 1(mod n ) ,
F k , otherwise .
In short, if x ∈ F k and let f i ( x ) (cid:44) r i ( x ; λ 0 , λ 1 , λ 2 , ω ) , then there exists only one i ∈ { 1 , . . . , n } such that h f ( x , i, γ ) could (and only could) provide additional information in F k +1 .
The “only one” property is important to the lower bound analysis for ﬁrst order stochastic optimization algorithms (Lan and Zhou, 2017; Zhou and Gu, 2019), but these prior constructions only work for IFO rather than PIFO.
Lemma 2.6 implies that x t = 0 will host until algorithm A draws the component f 1 .
Then, for any t < T 1 = min t { t : i t = 1 } , we have x t ∈ F 0 and x T 1 ∈ F 1 .
The value of T 1 can be regarded as the smallest integer such that x T 1 could host.
Similarly, we can deﬁne T k to be the smallest integer such that x T k ∈ F k could host.
We give the formal deﬁnition of T k recursively and connect it to geometrically distributed random variables in the following corollary.
Corollary 2.7.
Let
[Equation]
Then for any k ≥ 1 and t < T k , we have x t ∈ F k − 1 .
Moreover, T k can be written as sum of k independent random variables { Y l } 1 ≤ l ≤ k , i.e., T k = (cid:80) kl =1 Y l , where Y l follows a geometric distribution with success probability q l = p l (cid:48) where l (cid:48) ≡ l (mod n ) , 1 ≤ l (cid:48) ≤ n .
The basic idea of our analysis is that we guarantee the minimizer of r lies in F m and assure the
PIFO algorithm extend the space of span { x 0 , x 1 , . . .
, x t } slowly with t increasing.
We know that span { x 0 , x 1 , . . .
x T k } ⊆
F k by Corollary 2.7.
Hence, T k is just the quantity that reﬂects how span { x 0 , x 1 , . . .
, x t } veriﬁes.
Because T k can be written as the sum of geometrically distributed random variables, we needs to introduce some properties of such random variables which derive the lower bounds of our construction.
Lemma 2.8.
Let { Y i } 1 ≤ i ≤ N be independent random variables, and Y i follows a geometric distribution with success probability p i .
Then
[Equation]
From Lemma 2.8, the following result implies how many PIFO calls we need.
Lemma 2.9.
If M ≥ 1 satisﬁes min x ∈F M f ( x ) − min x ∈ R m f ( x ) ≥ 9 ε and N = n ( M + 1) / 4 , then we have
[-By Corollary 2.7, T M +1 can be written as T M +
-]
[Equation]
M AIN R ESULTS
We present the our lower bound results for PIFO algorithms and summarize all of results in Tableand 2 .
We ﬁrst start with smooth and strongly convex setting, then consider the general convex and average smooth cases.
Theorem 3.1.
For any PIFO algorithm A and any L, µ, n, ∆ , ε such that κ = L/µ ≥ 2 , and ε/ ∆ ≤ 0 .
5 , there exist a dimension d = O (cid:16) 1 + (cid:112) κ/n log (∆ /ε )
(cid:17) and n L -smooth and µ -strongly convex functions { f i : R d → R } ni =1 such that f ( x 0 ) − f ( x ∗ ) = ∆ .
In order to ﬁnd ˆ x ∈ R d such that E f (ˆ x ) − f ( x ∗ ) < ε , A needs at least N queries to h f , where [Equation] for n = O ( κ ) ,
[Equation]
for κ = O ( n ) .
Remark 3.2.
In fact, the lower bound in Theorem 3.1 perfectly match the upper bound of the PIFO algorithm Point SAGA (Defazio, 2016) 3 in n =
O ( κ ) case and match the the upper bound of the IFO algorithm 4 prox-SVRG (Hannah et al., 2018) in κ = O ( n ) case.
Hence, the lower bound in Theorem 3.1 is tight, while Woodworth and Srebro (2016) only provided lower bound Ω ( n + √ κn log (1 /ε )) in n
= O ( κ ) case.
The theorem also shows that the PIFO algorithm can not be more powerful than the IFO algorithm in the worst case, because Hannah et al.
(2018) proposed a same lower bound for IFO algorithms.
Next we give the lower bound when the objective function is not strongly-convex.
Theorem 3.3.
For any PIFO algorithm A and any L, n, B, ε such that ε ≤ LB 2 / 4 , there exist a dimension d = O (cid:16) 1 + B (cid:112) L/ ( nε ) (cid:17) and n L -smooth and convex functions { f i : R d → R } ni = such that (cid:107) x 0 − x ∗ (cid:107) 2 ≤ B .
In order to ﬁnd ˆ x ∈ R d such that E f (ˆ x ) − f ( x ∗ ) < ε , A needs at least Ω (cid:16) n + B (cid:112) nL/ε (cid:17) queries to h f .
Remark 3.4.
The lower bound in Theorem 3.3 is the same as the one of Woodworth and Srebro’s result.
However, our construction only requires the dimension be O (cid:16) 1 + B (cid:
) L/ ( nε ) (cid:17) , which is much smaller than O (cid:16)
L 2 B 4 ε 2 log (cid:16) nLB 2 ε (cid:17)(cid:17) in (Woodworth and Srebro, 2016).
Then we extend our results to the weaker assumption: that is, the objective function F is L -average smooth (Zhou and Gu, 2019).
We start with the case that F is strongly convex.
Theorem 3.5.
For any PIFO algorithm A and any L, µ, n, ∆ , ε such that κ = L/µ ≥ (cid:112) 3 /n (cid:0) n 2 + 1 (cid:1) , and ε/ ∆ ≤ 0 .
00327 , there exist a dimension d = O (cid:0) n − 1 / 4 √ κ log (∆ /ε ) (cid:1) and n functions { f i : R d → R } ni =1 where the { f i } ni =1 are L -average smooth and f is µ -strongly convex, such that f ( x 0 ) − f ( x ∗ ) = ∆ .
In order to ﬁnd ˆ x ∈ R d such that E f (ˆ x ) − f ( x ∗ ) < ε , A needs at least Ω (cid:0)(cid:0) n + n 3 / 4 √ κ (cid:1) log (∆ /ε )
(cid:1) queries to h f .
Remark 3.6.
Compared with Zhou and Gu’s lower bound Ω (cid:0) n + n 3 / 4 √ κ log (∆ /ε ) (cid:1) for IFO algorithms, Theorem 3.5 shows tighter dependency on n and supports PIFO algorithms additionally.
We also give the lower bound for general convex case under the L -average smooth condition.
Theorem 3.7.
For any PIFO algorithm A and any L, n, B, ε such that ε ≤ LB 2 / 4 , there exist a dimension d = O (cid:16) 1 + Bn − 1 / 4 (cid:112) L/ε (cid:17) and n functions { f i : R d → R } ni =1 which the { f i } ni =1 are
L -average smooth and f is convex, such that (cid:107) x 0 − x ∗ (cid:107) 2 ≤ B .
In order to ﬁnd ˆ x ∈ R d such that
E f
(ˆ x ) − f ( x ∗ ) < ε , A needs at least Ω (cid:16) n + Bn 3 / 4 (cid:112) L/ε (cid:17) queries to h f .
Remark 3.8.
The lower bound in Theorem 3.7 is comparable to the one of Zhou and Gu’s result, but our construction only requires the dimension be O (cid:16)
+ Bn − 1 / 4 (cid:112) L/ε (cid:17) , which is much smaller than O (cid:16) n + Bn 3 / 4 (cid:112) L/ε (cid:17) in (Zhou and Gu, 2019).
C ONSTRUCTIONS IN P ROOF OF M AIN T HEOREMS
We demonstrate the detailed constructions for PIFO lower bounds in this section.
All the omitted proof in this section can be found in Appendix for a detailed version.
4. S TRONGLY C ONVEX C ASE
The analysis of lower bound complexity for the strongly-convex case depends on the following construction.
Deﬁnition 4.1.
For ﬁxed L, µ, ∆ , n , let α = (cid:113) 2( L/µ − 1) n + 1 .
We deﬁne f SC ,i : R m → R as follows f SC
[Equation]
r i (cid:32)
[Equation]
[Equation]
(cid:33)
[Equation]
for
[Equation]
n
[Equation]
[Equation]
and
F
[Equation]
n (cid:88) i =
f SC [Equation] Note that the f SC ,i are L -smooth and µ -strongly convex, and F SC ( x 0 ) − F SC ( x ∗ ) =
∆ (see Proposition E.1 in Appendix for more details).
Next we show that the functions { f SC ,i } n i =1 are “hard enough” for any PIFO algorithm A , and deduce the conclusion of Theorem 3.1.
Theorem 4.2.
Suppose that
[Equation]
where α = (cid:113) 2( L/µ − 1) n + 1 .
In order to ﬁnd ˆ x ∈ R m such that E F SC (ˆ x ) − F SC ( x ∗ ) <
ε , PIFO algorithm
A needs at least N queries to h F SC .
where
[Equation]
for 2 ≤ Lµ <
[Equation]
For larger ε , we can apply following Lemma.
Lemma 4.3.
For any PIFO algorithm A and any L, µ, n, ∆ , ε such that ε ≤ ∆ / 2 , there exist n
L -smooth and µ -strongly convex functions { f i : R → R } ni =1 such that F ( x 0 ) − F ( x ∗ ) ≤ ∆ .
In order to ﬁnd ˆ x ∈ R such that E F (ˆ x ) − F ( x ∗ ) < ε , A needs at least Ω( n ) queries to h F .
As we explain in Remark H.1, the lower bound in Lemma 4.3 is same as the lower bound in Theorem 4.2 for ε > ∆9 (cid:16) α − 1 α +1 (cid:17) 2 .
In conclusion, we obtain Theorem 3.1.
4. C ONVEX C ASE
The analysis of lower bound complexity for non strongly-convex cases depends on the following construction.
Deﬁnition 4.4.
For ﬁxed L, B, n , we deﬁne f C ,i : R m → R as follows
[Equation]
[Equation]
and
F
[Equation]
Note that the f C ,i are L -smooth and convex, and (cid:107) x 0 − x ∗ (cid:107) 2 ≤ B (see Proposition G.1 in Appendix for more details).
Next we show the lower bound for functions f C ,i deﬁned above.
Theorem 4.5.
Suppose that
[Equation]
In order to ﬁnd ˆ x ∈ R m such that E F C (ˆ x ) − F C ( x ∗ ) <
ε , A needs at least Ω (cid:16) n + B (cid:113) nLε (cid:17) queries to h F C .
To derive Theorem 3.3, we also need the following lemma in the case ε > B 2 L 384 n .
Lemma 4.6.
For any PIFO algorithm A and any L, n, B, ε such that ε ≤ LB 2 / 4 , there exist n
L -smooth and convex functions { f i : R → R } ni =1 such that | x 0 − x ∗ | ≤ B .
In order to ﬁnd ˆ x ∈ R such that E F (ˆ x ) − F ( x ∗ ) < ε , A needs at least Ω( n ) queries to h F .
It is worth noting that if ε > B 2 L 384 n , then Ω( n ) =
Ω (cid:16) n + B (cid:113) nLε (cid:17) .
Thus combining Theorem 4.
and Lemma 4.6, we obtain Theorem 3.3.
4. A VERAGE S MOOTH C ASE
Zhou and Gu (2019) established lower bounds of IFO complexity under the average smooth assumption.
Here we demonstrate that our technique can also develop lower bounds of PIFO algorithm under this assumption.
4.3. F IS S TRONGLY C ONVEX
For ﬁxed L (cid:48) , µ, ∆ , n, ε , we set L = (cid:113) n ( L (cid:48) 2 − µ 2 )− µ 2 , and consider { f SC ,i } ni =1 and F SC deﬁned in
Deﬁnition 4.1.
Proposition 4.7.
For n ≥ 2 , we have that
F SC ( x ) is µ -strongly convex and { f SC ,i } ni
=1 is L (cid:48) -average smooth.
If L (cid:48) µ ≥ (cid:113) 3 n ( n 2 + 1) , then we have (cid:112) n 3 L (cid:48) ≤ L ≤ (cid:112) n 2 L (cid:48) and L/µ ≥ n/ 2 + 1 .
Theorem 4.8.
Suppose that
[Equation]
and m =
[Equation]
  (cid:115)(cid:114)n
L (cid:48) µ +  log (cid:18) ∆ 9 ε (cid:19) + 1 .
In order to ﬁnd ˆ x ∈ R m such that E F SC (ˆ x ) − F SC ( x ∗ ) <
ε , PIFO algorithm
A needs at least Ω (cid:16)(cid:16) n + n 3 / 4 (cid:113) L (cid:48) µ (cid:17) log (cid:0) ∆ ε (cid:1) (cid:17) queries to h F SC .
4.3. F IS C ONVEX
For ﬁxed L (cid:48) , B, n, ε , we set L = (cid:112) n 2 L (cid:48) , and consider { f C ,i } ni =1 and F C deﬁned in Deﬁnition 4.4.
It follows from Proposition 2.5 that { f C ,i } ni =1 is L (cid:48) -average smooth.
Theorem 4.9.
Suppose that
[Equation]
In order to ﬁnd ˆ x ∈ R m such that E F C (ˆ x ) − F C ( x ∗ ) <
ε , A needs at least Ω (cid:16) n + Bn 3 / 4 (cid:113) L (cid:48)
ε (cid:17) queries to h F C .
Similar to Lemma 4.6, we also need the following lemma for the case [Equation]
Lemma 4.10.
For any PIFO algorithm A and any L, n, B, ε such that ε ≤ LB 2 / 4 , there exist n functions { f i : R → R } ni =1 which is L -average smooth, such that F ( x ) is convex and (cid:107) x
0 − x ∗ (cid:107) 2 ≤
B .
In order to ﬁnd ˆ x ∈ R such that E F (ˆ x ) − F ( x ∗ ) < ε , A needs at least Ω( n ) queries to h F .
Similarly, note that if ε > √ 2 768 B 2 L (cid:48) √ n , then Ω( n ) = Ω (cid:16) n + Bn 3 / 4 (cid:113) L (cid:48)
ε (cid:17) .
In summary, we obtain
Theorem 3.7.
C ONCLUSION AND F UTURE W ORK
In this paper we have studied lower bound of PIFO algorithm for smooth convex ﬁnite-sum optimization.
We have given a tight lower bound of PIFO algorithms in the strongly convex case.
We have proposed a novel construction framework that is very useful to the analysis of proximal algorithms.
Based on this framework, we can extended our result to non-strongly convex, average smooth and non-convex problems easily (Appendix J).
It would be interesting to prove tight lower bounds in more general setting, such as F is of ( σ, L ) -smoothness while each f i is ( l, L ) -smoothness
A C OMPARISON OF REQUIRED NUMBER OF DIMENSIONS
Previous Lower Bounds
Our Lower Bounds f
i is L -smooth and µ -strongly convex
[Equation]
and
[Equation] [Equation]
[Theorem 3.1] f i is L -smooth and convex
[Equation]
and
[Equation] [Equation]
[Theorem 3.3] { f i } ni =1 is L -average smooth and f is µ -strongly convex
[Equation]
Zhou and Gu, 2019)
[Equation]
[Theorem 3.5] { f i } ni =1 is L -average smooth and f is convex
[Equation]
Theorem 3.7]
B C OMPARISON WITH EXISTING PROOFS
Recall the adversary function we used is (please see detailed deﬁntion in Section 2)
[Equation] [Equation]
where r
[Equation]
for
[Equation]
The constructions in previous work (Lan and Zhou, 2017; Zhou and Gu, 2019) for IFO algorithms employ an aggregation of r , that is,
[Equation]
i =
f i ( x ) , where f i ( x ) =
nr ( x i ) , x =   x 1 x [Equation] x n   ∈ R mn , and x i ∈ R m for i = 1 , . .
n.
The disadvantage of their construction is in that the important property from Lemma 2.6, which we can obtain information of only one extra dimension at each PIFO query, cannot be held.
Note that the framework in this paper is the ﬁrst lower bound analysis that utilize the decomposition form (11), which makes the “only one” property also hold for PIFO query.
The previous works only consider the presentation (12) and are unaware of the decomposition (11).
Moreover, the fact r : R m → R and f : R mn
→ R provide an intuitive understanding why our construction requires a smaller dimension (see Table 2).
The analysis in (Woodworth and Srebro, 2016; Fang et al., 2018) considers a very complicated approach to dealing with the proximal operator (completely different from how to deal with gradient operator).
In contrast, our construction holds “only one” property (Lemma 2.6) both for proximal and gradient operator, which leads the proof is more concise.
Our construction more clearly shows that PIFO algorithms are not more powerful than IFO algorithms in the sense of lower complexity bound.
We also use our technique to prove the tight lower bound of PIFO algorithm when κ = O ( n ) , which is a new result.
C D ETAILED P ROOF FOR S ECTION
In this section, we use (cid:107) A (cid:107) to denote the spectral radius of A .
For simplicity, let
B
[Equation]
b (cid:62) l is the l -th row of B , and f i ( x ) =
r i ( x ; λ 0 , λ 1 , λ 2 , m, ω ) .
Recall that
[Equation]
For 1 ≤ i ≤ n , let B i be a submatrix which is formed from rows L i of B , that is
[Equation]
Then f i can be wriiten as
[Equation]
where η 1 = λ 0 , η i = 0 ,
i ≥ 2 .
Proof of Proposition 2.5 .
Note that
[Equation]
where the last inequality is according to ( x + y ) 2 ≤ 2( x 2 + y 2 ) , and | l 1 − l 2 | ≥ n ≥ 2 for l 1 , l 2 ∈ L i .
Hence, (cid:13)(cid:13) B (cid:62) i B i (cid:13)(cid:13) ≤ 2 , and (cid:13)(cid:13) ∇ 2 f i ( x ) (cid:13)(cid:13) =
(cid:13)(cid:13) 2 λ 1 B (cid:62)
i B i
+ 2 λ
I (cid:13)(cid:13) ≤ 4 λ 1 + 2 λ 2 .
Next, observe that
[Equation]
Let u = x − y .
Note that
[Equation]
Thus
[Equation]
if m / ∈ L i , then
[Equation]
Similarly, if m ∈ L i , then
[Equation]
Therefore, we have
[Equation]
where we have used (2 λ 1 + λ 2 ) 2 ≤ 2 (cid:2) ( λ 1 + λ 2 ) 2 + λ 21 (cid:3) .
In summary, we get that { f i } 1 ≤ i ≤ n is L (cid:48) -average smooth, where
[Equation]
Proof of Lemma 2.6 .
For x ∈ F k ( k ≥ 1) , we have b (cid:62) l x = 0 for l > k, b l ∈ F k for l < k, b k ∈ F k +1 .
Consequently, for l (cid:54) =
k , b l b (cid:62) l x = ( b (cid:62) l x ) b l ∈ F k , and b k b (cid:62) k x ∈ F k +1 .
For k = 0 , we have x = 0 , and ∇ f 1 ( x ) = λ 0 e m ∈ F 1 , ∇ f j ( x ) =
( j ≥ 2) .
Moreover, we suppose k ≥ 1 , k ∈ L i .
Since
[Equation]
Hence, ∇ f i ( x ) ∈ F k +1 and ∇ f j ( x ) ∈ F k ( j (cid:54) = i ) .
Now, we turn to consider u = prox γf j
( x ) .
We have
[Equation]
i.e.,
[Equation]
where c 1 = 1 2 λ 2 +1 /γ , c 2 = 2 λ 1 2 λ 2 +1 /γ , and y =
η
j e m + 1 γ x .
Note that
[Equation]
If k = 0 and j > 1 , we have y = 0 and u = 0 .
If k = 0 and j = 1 , we have y = λ 0 e m .
On this case, B 1 e m = 0 , so u
= c 1 y ∈ F 1 .
For k ≥ 1 , we know that y ∈ F k .
And observe that if | l − l (cid:48)
| ≥ 2 , then b (cid:62) l b l (cid:48)
= 0 , and consequently B j B (cid:62) j is a diagonal matrix, so we can assume that 1 c 2 I + B j B (cid:62) j = diag( β j, 1 , · · · , β j, |L j | ) .
Therefore,
[Equation]
where we assume that L j = { l j, 1 , · · · , l j, |L j | } .
Thus, we have prox γf i ( x ) ∈ F k +1 for k ∈ L i and prox γf j ( x ) ∈ F k ( j (cid:54)
= i ) .
Proof of Corollary 2.7 .
Denote
[Equation]
by M t .
We know that x t ∈ M t .
Suppose that M T ⊆ F k − 1 for some T and let T (cid:48) =
arg min t : t > T, i t ≡ k (mod n ) .
By Lemma 2.6, for T < t < T (cid:48) , we can use a simple induction to obtain that
[Equation]
and M t ⊆
F k − 1 .
Moreover, since i T (cid:48) ≡ k (mod n ) , we have
[Equation]
and M T (cid:48) ⊆
F k .
Following from above statement, it is easily to check that for t <
T k , we have x t ∈ M t ⊆
F k − 1 .
Next, note that
[Equation]
mod n
[Equation]
where k (cid:48) ≡ k (mod n ) , 1 ≤ k (cid:48) ≤ n .
So T k − T k − 1 is a geometric random variable with success probability p k (cid:48) .
On the other hand, T k − T k − 1 is just dependent on i T k − 1 +1 , · · · , i T k , thus for l (cid:54) =
k , T l − T l − 1 is independent with T k − T k − 1 .
Therefore,
T k = k (cid:88) l
= ( T
[Equation]
T
[Equation]
=
[Equation]
(cid:88)
[Equation]
= Y l , where Y l follows a geometric distribution with success probability q l = p l (cid:48) where l (cid:48) ≡ l (mod n ) , 1 ≤ l (cid:48) ≤ n .
Proof of Remark 2.3 .
If each f i is L -smooth, then for any x , y ∈ R
m we have
[Equation]
and consequently,
[Equation]
If { f i } ni =1 is L -average smooth, then for any x , y ∈ R
m we have
[Equation]
(cid:107)∇
[Equation]
Proof of Lemma 2.9 .
Denote min x ∈ R
m f ( x ) by f ∗ .
For t ≤ N , we have [Equation] {+where T M +1 is deﬁned in (7), and the second inequality follows from Corollary 2.7 (if N < T M +1 , then x t ∈ F M for t ≤ N ). By Corollary 2.7, T M +1 can be written as T M +
+}
=
(cid:80) M +1 l =1 Y l , where { Y l } 1 ≤ l ≤ M +1 are independent random variables, and Y l follows a geometric distribution with success probability q l = p l (cid:48)
( l (cid:48)
≡ l (mod n ) , 1 ≤ l (cid:48) ≤ n ).
Moreover, recalling that p 1 ≤ p 2 ≤ · · · ≤ p n , we have (cid:80) M +1 l =1 q l ≤ M +1 n .
Therefore, by Lemma 2.8, we have
[Equation]
[Equation]
Hence, we can conclude that E f ( x N ) − f ∗ ≥ 9 ε P ( T M +1 > N ) ≥ ε .
Remark
In fact, a more strong conclusion hosts:
[Equation]
D R ESULTS ABOUT S UM OF G EOMETRIC D ISTRIBUTED R ANDOM V ARIABLES
Lemma D.1.
Let X 1 ∼ Geo
( p 1 ) , X 2 ∼ Geo( p 2 ) be independent random variables.
For any positive integer j , if p 1 (cid:54) =
p 2 , then
P ( X 1 + X 2 > j ) = p 2 (1 − p 1 ) j − p 1 (1 − p 2 )
j p 2 − p, (14) and if p 1 = p 2 , then
P ( X 1 + X 2 > j ) = jp 1 (1 − p 1 ) j − 1 + (1 − p 1 ) j .
Proof.
P ( X 1 + X 2 > j ) = j (cid:88) l = P ( X 1 = l ) P ( X 2 > j − l ) + P ( X 1 > j ) = j (cid:88) l = (1 − p 1 ) l − 1 p 1 (1 − p 2 )
j − l + (1 − p 1 ) j = p 1 (1 − p 2 )
j −j (cid:88) l
= (cid:18) 1 − p1 − p(cid:19) l −+ (1 − p 1 )
j
Thus if p 1 = p 2 , P ( X 1 + X 2 > j ) = jp 1 (1 − p 1 ) j − 1 + (1 − p 1 ) j .
For p 1 (cid:54) =
p 2 ,
P ( X 1 + X 2 > j ) = p 1 (1 − p 1 ) j − (1 − p 2 ) j p 2 − p+ (1 − p 1 ) j = p 2 (1 − p 1 ) j − p 1 (1 − p 2 )
j p 2 − p.
Lemma D.2.
For x ≥ 0 and j ≥ 2 ,
[Equation]
Proof.
We just need to show that
[Equation]
that is
[Equation]
Note that for l ≤ j − 2 ,
[Equation]
thus inequality (16) hosts for x ≥ 0 and j ≥ 2 .
Lemma D.3.
Let X 1 ∼ Geo
( p 1 ) , X 2 ∼ Geo( p 2 ) , Y 1 , Y 2 ∼ Geo (cid:0) p 1 + p 2(cid:1) be independent random variables with 0 < p 1 ≤ p 2 ≤ 1 .
Then for any positive integer j , we have
P ( X 1 + X 2 > j ) ≥ P ( Y 1 + Y 2 > j ) .
Proof.
If j = 1 , then P ( X 1 + X 2 > j ) =
1 = P ( Y 1 + Y 2 > j ) .
If p 1 = p 2 = 1 , then P ( X 1 + X 2 > j ) =
= P ( Y 1 + Y 2 > j ) for j ≥ 2 .
Let j ≥ 2 , and c (cid:44) p 1 + p 2 < 2 be a given constant.
We prove that f ( p 1 ) (cid:44) P ( X 1 + X 2 > j ) is a decreasing function.
Employing equation (14), for p 1 < c/ 2 , we have [Equation] c −
[Equation]
p, and f (cid:48) [Equation] − [Equation]
( c
[Equation]
=
[Equation]
−(
[Equation]
p
[Equation]
Hence f (cid:48) ( p 1 ) < 0 is equivalent to
[Equation]
Note that
[Equation]
Denote x = 1+ p 1 − c c − 2 p 1 .
If c ≤ 1 , then p 1 > 0 and x > 1 − c c ≥ 0 .
And if c > 1 , then p 1 ≥ c − 1 and x ≥ 1+ c − 1 − c 2 − c = 0 .
Rewrite inequality (17) as
[Equation]
Recall inequality (16), we have (cid:18) x
[Equation]
Consequently, f (cid:48) ( p 1 ) < 0 hosts for p 1 < c/ 2 and j ≥ 2 .
With the fact that lim p 1 → c/ 2 f ( p 1 ) =
f ( c/ 2) according to equation (15), we have
P ( X 1 + X 2 > j ) ≥ P ( Y 1 + Y 2 > j ) .
for any positive integer j and 0 < p 1 ≤ p 2 ≤ 1 .
Corollary D.4.
Let X 1 ∼ Geo( p 1 ) , X 2 ∼ [Equation] p [Equation]
, Y
[Equation]
, Y 2 ∼
[Equation]
+ [Equation] be independent random variables with 0 < p 1 ≤ p 2 ≤ 1 . Suppose Z is a random variable that takes nonnegative integer values, and Z is independent with X 1 , X 2 , Y 1 , Y 2 .
Then for any positive integer j , we have
P [Equation]
Proof.
With applying Lemma D.3, we have
[Equation]
Corollary D.5.
Let { X i } 1 ≤ i ≤ m be independent variables, and X i follow a geometric distribution with success probability p i .
For any positive integer j , we have
[Equation]
where { Y i } 1 ≤ i ≤ m are i.i.d.
random variables,
Y i ∼ Geo( (cid:80) mi =1 p i /m ) , and Y i is independent with
X i (cid:48) (1 ≤ i (cid:48) ≤ m ) .
Proof.
Let
[Equation]
Our goal is to minimize f ( p 1 , p 2 , · · · , p m ) such that (cid:80) mi =1 p i =
S < 1 .
By Corollary D.4, we know that
[Equation]
This fact implies that ( p 1 , p 2 , · · · , p m ) such that p 1 = p 2 = · · · = p m = S/m is a minimizer of the function f .
Lemma D.6.
Let { X i } 1 ≤ i ≤ m be i.i.d.
random variables, and X i follows a geometric distribution with success probability p .
We have
[Equation]
Proof.
Denote (cid:80) mi =1 X i by τ .
We know that
[Equation]
Hence, we have
[Equation]
Corollary D.7.
Let { X i } 1 ≤ i ≤ m be independent random variables, and X i follows a geometric distribution with success probability p i .
Then
[Equation]
E P ROOF OF T HEOREM 4.
Proposition E.1.
For any n ≥ 2 , m ≥ 2 , f SC ,i and F SC in Deﬁnition 4.1 satisfy: 1.
f SC ,i is L -smooth and µ -strongly convex.
The minimizer of the function F SC is
[Equation]
where q = α − 1 α
+1 .
Moreover, F SC ( x ∗ ) = − ∆ .
For 1 ≤ k ≤ m − 1 , we have
[Equation]
Proof.
Just recall Proposition 2.5.
Denote ξ = (cid:113) 2∆ n ( α +1)
( L − µ )( α − 1) .
Let ∇ F SC ( x ) = 0 , that is (cid:32)
L − µ
[Equation]
A (cid:32)(cid:114)
[Equation]
[Equation]
or  
[Equation]
[Equation]
[Equation]
[Equation]
 
[Equation]
=    ...
[Equation]
ξ α +
[Equation]
  (20)
Note that q =
α − 1 α +1 is a root of the equation
[Equation]
and
[Equation]
Hence, it is easily to check that the solution to Equation (20) is
[Equation]
and
[Equation]
If x ∈ F k , 1 ≤ k < m , then x 1 = x 2 = · · · = x m − k = 0 .
Let y = x m − k
+1: m ∈ R k and A k be last k rows and columns of the matrix in Equation (21).
Then we can rewrite F ( x ) as
[Equation]
Let ∇ F k ( y )
= 0 , that is  
[Equation]
[Equation]
By some calculation, the solution to above equation is
[Equation]
Thus
[Equation]
− µ 2 n ( α + 1) ξ 2 q 1 − q 2 k 1 + q 2 k +1 = ∆ 1 − q 2 k 1 + q 2 k +1 , and min x ∈F k F SC ( x ) − F SC ( x ∗ ) = ∆ (cid:18) 1 − 1 − q 2 k 1 + q 2 k +
(cid:19) =
∆ q 2 k 1 + q 1 + q 2 k +1 ≥ ∆ q 2 k .
Proof of Theorem 4.2 .
Let M = (cid:106) log(
ε/ ∆) 2log q (cid:107) , then we have
[Equation]
where the ﬁrst inequality is according to the third property of Proposition E.1.
Following from Lemma 2.9, for M ≥ 1 and N =
( M + 1) n/ 4 , we have
[Equation]
Therefore, in order to ﬁnd ˆ x ∈ R m such that E F SC (ˆ x ) − F SC ( x ∗ ) < ε , A needs at least N queries to h F SC .
We estimate − log( q ) and N in two cases.
If L/µ ≥ n/ 2 + 1 , then α = (cid:113) 2 L/µ − 1 n + 1 ≥ √ 2 .
Observe that function h ( β ) = 1 log ( β +1 β − 1 ) − β 2 is increasing when β > 1 .
Thus, we have
[Equation]
and
N =
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
(cid:19)
=
Ω (cid:32)(cid:32)
[Equation]
If 2 ≤ L/µ < n/ 2 + 1 , then we have
[Equation]
where the ﬁrst inequality and second inequality follow from L/µ − 1 < n/ 2 and the last inequality is according to 1 x − 1 ≤ 2 x for x
≥ 2 . Note that n ≥ 2 , thus nn − 1 ≤ 2 ≤ n L/µ − 1 , and hence n ≥ L/µ , i.e. log( nµ/L ) ≥ 0 .
Therefore,
[Equation]
Recalling that we assume that 9 ε/ ∆ ≤ q 2 , thus we have
[Equation]
Therefore, N = Ω (cid:16) n + (cid:16) n 1+log( nµ/L ) (cid:17) log (cid:0)
[Equation]
ε (cid:1)(cid:17) .
At last, we must to ensure that 1 ≤ M < m , that is
[Equation]
Note that lim
β → + ∞ h ( β ) = 0 , so − 1 / log( q ) ≤ α/ 2 .
Thus the above conditions are satisﬁed when m = log [Equation] ∆ /
[Equation]
[Equation] ≤ [Equation] and ε
[Equation]
F L OWER BOUND FOR ANOTHER FORM OF SUBOPTIMAL SOLUTION
Defazio (2016) showed that the PIFO algorithm Point SAGA has the convergence result
E (cid:107) x t − x ∗ (cid:107) 22 ≤ ( q (cid:48) )
t (cid:107) x 0 − x ∗ (cid:107) 2 , where q (cid:48) satisﬁes − 1 / log( q (cid:48) )
=
O (cid:16) n + (cid:112) nL/µ (cid:17) .
To match this form of upper bound, we point out that a similar result holds for { f SC ,i } ni =1 .
Theorem F.1.
Suppose that
[Equation]
In order to ﬁnd ˆ x ∈ R m such that E (cid:107) ˆ x − x ∗ (cid:107) 22 < ε (cid:107) x 0 − x ∗ (cid:107) 22 , PIFO algorithm
A needs at least Ω (cid:16)(cid:16) n +
[Equation]
nLµ (cid:17) log (cid:0)
[Equation]
(cid:1)(cid:17) queries to h F SC .
Proof.
Denote ξ = (cid:113) 2∆ n ( α +1)
( L − µ )( α − 1) , and M = (cid:106) log(18 ε ) 2log q (cid:107) .
For 1 ≤ M ≤ m/ 2 , N = n ( M + 1) / 4 and t ≤ N , we have
[Equation]
where T M +1 is deﬁned in (7), the second inequality follows from Corollary 2.7 (if N < T M +1 , then x t ∈ F M for t ≤ N ), and the last inequality is established because of our Corollary 2.7 (More detailed explanation refer to our proof of Lemma 2.9).
By Proposition E.1, we know that x ∗ = ξ ( q m , q m − 1 , · · · , q ) (cid:62) , and
[Equation]
Note that if x ∈ F M , then x 1 = x 2 = · · · = x m − M = 0 , thus
[Equation]
Thus, for t ≤ N and M ≤ m/ 2 , we have
[Equation]
where the second inequality is due to
[Equation]
Therefore, in order to ﬁnd ˆ x ∈ R m such that E (cid:107) ˆ x − x ∗ [Equation] 22 (cid:107) x 0 − x ∗ (cid:107) 22 < ε , A needs at least N queries to h F SC .
As we have showed in proof of Theorem 4.2, for L/µ ≥ n/ 2 + 1 , we have [Equation] µ [Equation]
n
[Equation]
, and
[Equation]
At last, we have to ensure that 1 ≤ M ≤ m/ 2 , that is
[Equation]
The above conditions are satisﬁed when
[Equation]
and ε ≤ 1 18 q 2 .
Observe that when L/µ ≤ n/ 2 + 1 , we have α ≥ √ 2 and q = α − 1 α +1 ≥ √ 2 − 1 √ 2+1 .
Hence, we just need ε ≤ 118 (cid:16) √ 2 − 1 √ 2+1 (cid:17) 2 ≈ 0 . 00164 .
G P ROOF OF T HEOREM 4.
Proposition G.1.
For any n ≥ 2 , m ≥ 2 , following properties hold:
f C ,i is L -smooth and convex.
The minimizer of the function F C is
[Equation]
where ξ = √ BL ( m +1) 3 / 2 .
Moreover, F C ( x ∗ ) = − mξ 2 nL and (cid:107) x 0 − x ∗ (cid:107) 22 ≤ B 2 .
For 1 ≤ k ≤ m , we have
[Equation]
Proof.
Just recall Proposition 2.5.
Denote ξ = √
BL ( m +1) 3 / 2 n .
Let ∇ F C ( x ) = 0 , that is
L
[Equation]
or  
[Equation]
Hence, it is easily to check that the solution to Equation (25) is
[Equation]
and
F
[Equation]
Moreover, we have
[Equation]
By similar calculation to above proof, we have
[Equation]
and
[Equation]
Thus
[Equation]
Proof of Theorem 4.5 .
Since ε ≤ B 2 L 384 n , we have m ≥ 3 .
Let ξ = √
BL ( m +1) 3 / 2 .
For M = (cid:4) m − 1 2 (cid:5) ≥ 1 , we have m − M ≥ ( m + 1) / 2 , and
[Equation]
where the ﬁrst equation is according to the 3rd property in Proposition G.1 and the last inequality follows from m + 1 ≤ B (cid:112) L/
(24 nε ) .
Similar to the proof of Theorem 4.2, by Lemma 2.9, we have
[Equation]
In other words, in order to ﬁnd ˆ x ∈ R m such that E F C ( ˆ x ) − F C ( x ∗ ) <
ε , A needs at least N queries to h F .
At last, observe that
[Equation]
where we have recalled ε ≤ B 2 L 384 n in last equation.
H P ROOF OF L EMMA 4.3, L EMMA 4.6 AND L EMMA 4.
Proof of Lemma 4.6 .
Consider the following functions { g i } 1 ≤ i ≤ n , g i : R → R , where
[Equation]
First observe that
[Equation]
and | x 0 − x ∗ | =
B .
For i > 1 , we have dg i ( x ) dx | x =
= 0 and prox
γg i (0) = 0 .
Thus x t = 0 will host till our ﬁrst-order method A draws the component f 1 .
That is, for t < T = arg min { t : i t = 1 } , we have x t = 0 .
Hence, for t ≤ 12 p 1 , we have
[Equation]
Note that T follows a geometric distribution with success probability p 1 ≤ 1 /n , and
[Equation]
where the second inequality follows from h ( z )
=
log(1 − z ) 2 z is a decreasing function.
Thus, for t ≤ 12 p 1 , we have
[Equation]
Thus, in order to ﬁnd ˆ x ∈ R such that E F (ˆ x ) − F ( x ∗ ) < ε , A needs at least 12 p 1 ≥ n/ 2 = Ω ( n ) queries to h G .
Proof of Lemma 4.10 .
Note that { g i } ni =1 deﬁned in proof of Lemma 4.6 is also L -average smooth, so Lemma 4.10 hosts for the same reason.
Proof of Lemma 4.3 .
Let B = (cid:112) 2∆ /L .
Then ε/ ∆ ≤ 1 / 2 is equivalent to ε ≤ LB 2 / 4 .
Note that { g i } ni =1 deﬁned in proof of Lemma 4.6 is also µ -strongly convex for any µ ≤ L , and satisfy | G (0) − G ( x ∗ ) | = ∆ .
Therefore Lemma 4.3 hosts for the same reason.
Remark H.1.
Suppose that
[Equation]
If κ ≥ n/ 2 + 1 , then we have α ≥ √ 2 and
[Equation]
where the second inequality follows from log(1+ x ) ≤ x and the last inequality is according to α ≥ (cid:112) 2 κ/n
That is
[Equation]
If 2 ≤ L/µ < n/ 2 + 1 , then we have
[Equation]
where the second inequality follows from (22).
That is
[Equation]
n
[Equation]
L ) (cid:19) log (cid:18)
∆ 9 ε (cid:19)
[Equation]
n (cid:19) .
I D ETAILED P ROOF FOR S ECTION 4.
Proof of Proposition 4.7 .
It is easily to check that F SC ( x ) is µ -strongly convex.
Following from Proposition 2.5, then { f SC ,i } ni =1 is ˆ L -average smooth, where
[Equation]
Clearly, L = (cid:113) n ( L (cid:48)
2 − µ 2 )
[Equation]
Furthermore, according to L (cid:48) µ ≥ (cid:113) 3 n ( n 2 + 1) , we have
[Equation]
and, L/µ ≥ (cid:112) n 3 L (cid:48) /µ ≥ n/ 2 + 1 .
Proof of Theorem 4.8 .
By 2nd property of Proposition 4.7, we know that L/µ ≥ n/ 2 + 1 .
Moreover,
[Equation]
Then, by Theorem 4.2 5 , in order to ﬁnd ˆ x ∈ R m such that E F SC (ˆ x ) − F SC ( x ∗ ) < ε , A needs at least N queries to h F SC , where
[Equation]
Proof of Theorem 4.9 .
Note that
[Equation]
By Theorem 4.5, in order to ﬁnd ˆ x ∈ R m such that E F C (ˆ x ) − F C ( x ∗ ) <
ε , A needs at least N queries to h F C , where
[Equation]
J N ON - CONVEX C ASE
In non-convex case, our goal is to ﬁnd an ε -approximate stationary point
ˆ x of our objective function f , which satisﬁes
[Equation]
J. P RELIMINARIES
We ﬁrst introduce a general concept about smoothness.
Deﬁnition J.1.
For any differentiable function f : R m +
→ R , we say f is ( l, L )
-smooth, if for any x , y ∈ R m we have
[Equation]
where L > 0 , l ∈ R .
Especially, if f is L -smooth, then it can be checked that f is ( − L, L ) -smooth.
If f is ( − σ, L ) -smooth, in order to make the operator prox γf valid, we set 1 γ > σ to ensure the function
[Equation]
is a convex function.
Next, we introduce a class of function which is original proposed in (Carmon et al., 2017).
Let
G NC : R m +1 → R be
[Equation]
where the non-convex function Γ : R → R is
[Equation]
We need following properties about G NC ( x ; α, m ) .
Proposition J.2 (Lemmas 3,4, Carmon et al.
For any 0 < α ≤ 1 , it holds that
Γ( x ) is ( − 45( √ 3 − 1) , 180) -smooth and G NC ( x ; α, m ) is ( − 45( √ 3 − 1) α, 4 + 180 α ) - smooth. 2.
G NC ( 0 ; α, m ) − min x ∈ R
m +1 G NC ( x ; α, m ) ≤ √ α/ 2 + 10 αm .
For x which satisﬁes that x m = x m +
= 0 , we have
[Equation]
J. O UR R ESULT
Theorem J.3.
For any PIFO algorithm A and any L, σ, n, ∆ , ε such that ε 2 ≤ ∆ Lα 81648 n , there exist a dimension d = (cid:106)
∆ L √ α 40824 nε 2 (cid:107) + 1 and n ( − σ, L ) -smooth nonconvex functions { f i : R d → R } ni =1 such that f ( x 0 ) − f ( x ∗ ) ≤ ∆ .
In order to ﬁnd ˆ x ∈ R d such that E (cid:107)∇ f
(ˆ x ) (cid:107) 2 < ε , A needs at least Ω (cid:16)
∆ L √ α ε 2 (cid:17) queries to h f , where we set α = min (cid:110) 1 , ( √ 3+1) nσ
L
[Equation]
n [Equation]
Remark J.4.
For n > 180 , wehave
[Equation]
Thus, our result is comparable to the one of Zhou and Gu’s result (their result only related to
IFO algorithms, so our result is more strong), but our construction only requires the dimension be O (cid:16) 1 + ∆ ε 2 min { L/n, (cid:112) σL/n } (cid:17) , which is much smaller than O (cid:16)
∆ ε 2 min { L, √ nσL } (cid:17) in (Zhou and Gu, 2019).
J. C ONSTRUCTIONS
Consider
[Equation]
Similar to our construction we introduced in Section 2, we denote the l -th row of the matrix B ( m + 1 , 4 √ α ) by b l and
[Equation]
Let G k = span { e 1 , e 2 , · · · , e k } , 1 ≤ k ≤ m , G 0 = { 0 } and compose F ( x ; α, m, λ, β ) to 
[Equation]
Clearly, F ( x ; α, m, λ, β ) = 1 n (cid:80) ni
=
f i ( x ; α, m, λ, β ) .
Moreover, by Proposition J.2, we have following properties about F ( x ; α, m, λ, β ) and { f i ( x ; α, m, λ, β ) } ni =1 .
Proposition J.5.
For any 0 < α ≤ 1 , it holds that
f i ( x ; α, m, λ, β ) is (cid:16) − 45( √ 3 − 1) αλ β, (2 n +180 α ) λ β(cid:17) -smooth.
F ( 0 ; α, m, λ, β ) − min x ∈ R m +1 F ( x ; α, m, λ, β ) ≤ λ ( √ α/ 2 + 10 αm ) .
For x which satisﬁes that x m = x m +
= 0 , we have
[Equation]
Similar to Lemma 2.6, similar conclusion hosts for { f i ( x ; α, m, λ, β ) } ni =1 .
Lemma J.6.
For x ∈ F k , 0 ≤ k < m and γ < √ 2+160 β 2 λα , we have ∇ f i ( x ; α, m, λ, β ) , prox
γf i ( x ) ∈ (cid:26)
G k +1 , if k ≡ i − 1(mod n ) ,
G k , otherwise .
Proof.
Let G ( x ) (cid:44) m
(cid:80) i =1 Γ( x i ) and Γ (cid:48)
( x ) be the derivative of Γ( x ) .
First note that Γ (cid:48) (0) = 0 , so if x ∈ G k , then ∇ G ( x ) = (cid:0) Γ (cid:48) ( x 1 ) , Γ (cid:48) ( x 2 ) , · · · , Γ (cid:48) ( x m ) (cid:1) (cid:62) ∈ G k .
Moreover, for x ∈ F G ( k ≥ 1) , we have b (cid:62) l x = 0 for l < m − k, b l ∈ G k for l > m − k, b m − k ∈
G k +1 .
Consequently, for l (cid:54) = m − k , b l b (cid:62) l x = ( b (cid:62) l x ) b l ∈ G k , and b m − k b (cid:62) m − k x ∈ G k +1 .
For k = 0 , we have x = 0 , and
[Equation]
[Equation]
For k ≥ 1 , we suppose that m − k ∈ L i .
Since
[Equation]
where η 1 = λn √ α/β , η j = 0 for j ≥ 2 .
Hence, ∇ f i ( x ) ∈ F k +1 and ∇ f j ( x ) ∈ F k ( j (cid:54) = i ) .
Now, we turn to consider v = prox γf j ( x ) .
We have
[Equation]
, that is
[Equation]
Denote
[Equation]
then we have
[Equation]
Next, if s satisﬁes (cid:26) s > max { 1 , k } for j = 1 , s > k for j > 1 , (33) then we know that the s -th element of y is 0 .
If s satisﬁes (33) and m − s ∈
L j , then the s -th and ( s + 1) -th elements of Au is (( ξ + β/γ ) u
s − ξu s +1 ) and ( − ξu s + ( ξ + β/γ ) u s +1 ) respectively where ξ = λn/β .
So by Equation (32), we have
[Equation]
Following from Lemma J.9, for 120 λαβ < (2+2 √ 2) β γ , we have u s =
u s +
= 0 .
That is
if m − s ∈ L j and s satisﬁes (33), then u s = 0 .
if m − s + 1 ∈ L j and s − 1 satisﬁes (33), then u s = 0 .
For s which satisﬁes (33), if m − s (cid:54)∈ L j and m − s + 1 (cid:54)∈ L j , then the s -th element of Au is ( β/γ u s ) .
Similarly, by Equation (32), we have
[Equation]
Following from Lemma J.8, for 120 λαβ < (2+2 √ 2) β γ , we have u s = 0 .
Therefore, we can conclude that
if s − 1 satisﬁes (33), then u s = 0 .
if s satisﬁes (33) and m − s + 1 (cid:54)∈ L j , then u s = 0 .
Moreover, we have that
if k = 0 and j = 1 , then m − 1 , m − 2 (cid:54)∈ L j , so u 2 = 0 .
if k = 0 and j > 1 , then for s = 1 , we have m − s + 1 (cid:54)∈ L j , so u 1 = 0 .
if k = 0 , then for s > 2 , we have s − 1 > 1 satisﬁes (33), so u s = 0 .
if k > 0 , then for s > k + 1 , we have s − 1 > k satisﬁes (33), so u s = 0 .
if k > 0 and m − k
(cid:54)∈ L j , then for s = k + 1 , we have m − s + 1 (cid:54)∈ L j , so
u k +1 = 0 .
In short,
if k = 0 and j > 1 , then u ∈ G 0 .
if k = 0 and j = 1 , then u ∈ G 1 .
if k > 1 and m − k
(cid:54)∈ L j , then u ∈ G k .
if k > 1 and m − k ∈
L j , then u ∈ G k +1 .
Remark J.7.
In order to make the operator prox γf i valid, γ need to satisfy
[Equation]
So for any valid PIFO call, the condition about γ in Lemma J.6 must be satisﬁed.
Lemma J.8.
Suppose that 0 < λ 2 <
(2 + 2 √ 2) λ 1 , then
z = 0 is the only real solution to the equation
[Equation]
Proof.
Since 0 < λ 2 < (2 + 2 √ 2) λ 1 , we have
[Equation]
, and consequently, for any z , ( λ 1 + λ [Equation] z 2 − λ 2 z + λ 1 > 0 .
On the other hand, we can rewrite Equation (34) as
[Equation]
Clearly, z = 0 is the only real solution to Equation (34).
Lemma J.9.
Suppose that 0 < λ 2 < (2 + 2 √ 2) λ 1 and λ 3 > 0 , then z 1 = z 2 = 0 is the only real solution to the equation
[Equation]
Proof.
If z 1 = 0 , then z 2 = 0 .
So let assume that z 1 z 2 (cid:54) = 0 .
Rewrite the ﬁrst equation of Equation (35) as
[Equation]
Note that
[Equation]
Thus, we have
[Equation]
Similarly, it also holds
[Equation]
By 0 < λ 2 < (2 + 2 √ 2) λ
1 , we know that λ
+ 1 −√ 2 2 λ 2 > 0 .
Thus
[Equation]
Since z 1 /z 2 > 1 and z 2 /z 1 > 1 can not hold at the same time, so we get a contradiction.
Following from Lemma J.6, we know following Lemma which is similar to Lemma 2.9.
Lemma J.10.
If M ≥ 1 satisﬁes min x ∈G M (cid:107)∇ F ( x ) (cid:107) 2 ≥ 9 ε and N = n ( M + 1) / 4 , then we have min t ≤ N E (cid:107)∇ F ( x t ) (cid:107) 2 ≥ ε.
Theorem J.11.
Set α =
[Equation]
Suppose that ε 2 ≤ ∆ Lα 81648 n .
In order to ﬁnd ˆ x ∈ R m +1 such that E (cid:107)∇ F (ˆ x ) (cid:107) 2 < ε , PIFO algorithm
A needs at least Ω (cid:16) ∆ L √ α ε 2 (cid:17) queries to h F .
Proof.
First, note that f i is ( − l 1 , l 2 ) -smooth, where
[Equation]
Thus each f i is ( − σ, L ) -smooth.
Next, observe that
[Equation]
For M = m − 1 , we know that
[Equation]
With recalling Lemma J.10, in order to ﬁnd ˆ x ∈ R m +1 such that E (cid:107)∇ F (ˆ x ) [Equation] 2 < ε , PIFO algorithm
A needs at least N queries to h F , where
[Equation]
At last, we need to ensure that m ≥ 2 .
By ε 2 ≤ ∆ Lα 81648 n , we have
[Equation]
and consequently m ≥ 2 .