Faster Stochastic Algorithms for Minimax Optimization under Polyak-Łojasiewicz Conditions
Abstract
This paper considers stochastic first-order algorithms for minimax optimization under Polyak-Łojasiewicz (PL) conditions.
We propose SPIDER-GDA for solving the finite-sum problem of the form min x max y f
( x, y ) ≜ 1 n (cid:80) ni =
f i ( x, y ) , where the objective function f ( x, y ) is µ x -PL in x and µ y -PL in y ; and each f i ( x, y ) is L -smooth.
We prove SPIDER-GDA could find an ϵ -approximate solution within O (cid:0) ( n + √ n κ x κ 2 y ) log(1 /ϵ ) (cid:1) stochastic first-order oracle (SFO) complexity, which is better than the state-of-the-art method whose SFO upper bound is O (cid:0) ( n + n 2 / 3 κ x κ 2 y ) log(1 /ϵ ) (cid:1) , where κ x ≜ L/µ x and κ y ≜ L/µ y .
For the ill-conditioned case, we provide an accelerated algorithm to reduce the computational cost further.
It achieves ˜ O (cid:0) ( n + √ n κ x κ y ) log 2 (1 /ϵ ) (cid:1) SFO upper bound when κ y ≳ √ n .
Our ideas also can be applied to the more general setting that the objective function only satisfies PL condition for one variable.
Numerical experiments validate the superiority of proposed methods.
Introduction
This paper focuses on smooth minimax optimization problem of the form
[Equation]
f i ( x, y ) , (1) which covers a lot of important applications in machine learning such as reinforcement learning [10, 42], AUC maximization [13, 24, 48], imitation learning [5, 32], robust optimization [11], causal inference [28], game theory [6, 29] and so on.
We are interested in the minimax problems under PL conditions [9, 32, 45].
The PL condition [35] was originally proposed to relax the strong convexity in minimization problem that is sufficient for achieving the global linear convergence rate for first-order methods.
In machine learning community, it has been successfully used to analyze the convergence behavior for overparameterized neural networks [23], robust phase retrieval [40] and a plenty of fundamental models [18].
There are many popular minimax formulations only satisfy PL condition, but lack strong convexity (or strong concavity).
The examples include PL-game [32], robust least square [45], deep AUC maximization [24] and generative adversarial imitation learning of LQR [5, 32].
Yang et al.
[45] showed that the alternating gradient descent ascent (AGDA) algorithm linearly converges to the saddle point when the objective function satisfies two-sided PL condition.
They also proposed the SVRG-AGDA method for the finite-sum problem (1), which could find ϵ -approximate
[Table]
Theorem 4.
[Table]
Theorem 5. solution within O (cid:0) ( n + n 2 / 3 κ x κ 2 y ) log(1 /ϵ ) (cid:1) stochastic first-order oracle (SFO) calls, 2 where κ x and κ y are the condition numbers with respect to PL condition for x and y respectively.
The variance reduced technique in the SVRG-AGDA leads to better a convergence rate than full batch AGDA whose SFO complexity is O (cid:0) nκ x κ 2 y log(1 /ϵ ) (cid:1) .
However, there are still some open questions left.
Firstly, Yang et al.
[45]’s theoretical analysis heavily relies on the alternating update rules.
It remains interesting whether a simultaneous version of GDA (or its stochastic variants) also has similar convergence results.
Secondly, it is unclear whether the SFO upper bound obtain by SVRG-AGDA can be improved by designing more efficient algorithms.
For one-sided PL condition, we desire to find the stationary point of g ( x ) ≜ max y ∈ R dy f
( x, y ) , since the saddle point may not exist.
Nouiehed et al.
[32] proposed the multi-step GDA method that achieves the ϵ -stationary point within O ( κ 2 y Lϵ − 2 log( κ y /ϵ ))
numbers of full gradient iterations.
The similar complexity also can be obtained by AGDA [45].
Recently, Yang et al.
[47] proposed the smoothed-AGDA that improves the upper bound into O ( κ y Lϵ − 2 ) .
Both multi-step GDA Nouiehed et al.
[32] and smoothed-AGDA [46] can be extended to online setting [14], but the formulation (1) with finite-sum structure has not been explored.
In this paper, we introduce a variance reduced first-order method, called SPIDER-GDA, which constructs the gradient estimator by stochastic recursive gradient and the iterations are based on simultaneous gradient descent ascent.
We prove that SPIDER-GDA could achieve ϵ -approximate solution of the two-sided PL problem of the form (1) within O (cid:0) ( n + √ n κ x κ 2 y ) log(1 /ϵ ) (cid:1) SFO calls, which has better dependency on n than SVRG-AGDA [45].
We also provide an acceleration framework to improve first-order methods for solving ill-conditioned minimax problems under PL conditions.
The accelerated SPIDER-GDA (AccSPIDER-GDA) could achieve ϵ -approximate solution within ˜ O (cid:0) ( n + √ n κ x κ y ) log 2 (1 /ϵ ) (cid:1) SFO calls 3 when κ y ≳ √ n , which is the best known SFO upper bound for this problem.
We summarize our main results and compare them with related work in Table 1.
Without loss of generality, we always suppose κ x ≳ κ y .
Furthermore, the proposed algorithms also work for minimax problem with one-sided PL condition.
We present the results for this case in Table 2.
Related Work
The minimax optimization problem (1) can be viewed as the following minimization problem
[Equation]
A natural way to solve such problem is the multi-step GDA algorithm
[21, 25, 32, 36] that contains double-loop iterations in which the outer loop can be regarded as running inexact gradient descent on
[Table]
Theorem B.2, [45]
[Table]
Theorem F
[Table] [Table]
[Table] [Table]
[Equation]
[Table]
[Equation]
Theorem 6.
g ( x ) and the inner loop finds the approximate solution to max y ∈ R dy
f ( x, y ) for a given x .
Another class of methods is the two-timescale (alternating) GDA algorithm
[9, 21, 44, 45] that only has singleloop iterations which update two variables with different stepsizes.
The two-timescale GDA method can be implemented more easily and typically performs better than multi-step GDA empirically.
Its convergence rate also can be established by analyzing function g ( x ) but the analysis is more challenging than the multi-step GDA.
The variance reduction is a popular technique to improve the efficiency of stochastic optimization algorithms [2–4, 7, 8, 12, 16, 17, 19, 27, 31, 33, 34, 37–39, 43, 49, 50].
It is shown that solving nonconvex minimization problems with stochastic recursive gradient estimator [12, 16, 34, 43, 51] has the optimal SFO complexity.
In the context of minimax optimization, the variance reduced algorithms also obtain the best-known SFO complexities in several settings [1, 15, 25, 26, 41, 45].
Specifically, the (near) optimal SFO algorithm for several convex-concave minimax problem has been proposed [15, 26], but the optimality for the more general case is still unclear [25, 45].
The Catalyst acceleration [20] is a useful approach to reduce the computational cost of ill-conditioned optimization problems, which is based on a sequence of inexact proximal point iterations.
Lin et al.
[22] first introduced Catalyst into minimax optimization.
Later, Luo et al.
[26], Tominin et al.
[41], Yang et al.
[46] designed the accelerated stochastic algorithms for convex-concave and nonconvex-concave problem.
Recently, Yang et al.
[47] also applied this technique to one-sided PL setting.
Notation and Preliminaries
First of all, we present the definition of saddle point.
Definition 3.1.
We say ( x ∗ , y ∗ ) ∈ R d x × R d y is a saddle point of function f : R d x × R d y → R if it holds that f ( x ∗ , y ) ≤ f ( x ∗ , y ∗ ) ≤ f ( x, y ∗ ) for any x ∈ R d x and y ∈ R d y .
Then we formally define the Polyak-Łojasiewicz (PL) condition [35] as follows.
Definition 3.2.
We say a differentiable function h : R d → R satisfies µ -PL for some µ > 0 if ∥∇ h ( z ) ∥ 2 ≥ 2 µ (cid:0) h ( z ) − min z ′ ∈ R d h ( z ′ ) (cid:1) holds for any z ∈ R d .
Note that the PL condition does not require the strongly convexity and it can be satisfied even if the function is nonconvex [18].
We are interested in the finite-sum minimax optimization problem (1) under following assumptions.
Assumption 3.1.
We suppose each component f i : R d x × R d y → R is L -smooth, i.e., there exists a constant L > 0 such that ∥∇ f i ( x, y ) − ∇ f i ( x ′ , y ′ )
∥ 2 ≤ L 2 (cid:0) ∥ x − x ′ ∥ 2 + ∥ y − y ′
∥ 2 (cid:1) holds for any x, x ′ ∈ R d x and y, y
′ ∈
R d y .
Assumption 3.2.
We suppose the differentiable function f : R d x × R d y
→ R satisfies two-sided PL condition, i.e., there exist constants µ x > 0 and µ y > 0 such that f ( · , y ) is µ x -PL for any y ∈ R d y and − f ( x, · ) is µ y -PL for any x ∈ R d x .
Under Assumption 3.1 and 3.2, we define the condition numbers of problem (1) with respect to PL conditions for x and y as κ x ≜ L/µ x and κ y ≜ L/µ y respectively.
We also introduce the following assumption for the existence of saddle point.
Assumption 3.3 (Yang et al.
We suppose the function f : R d x × R d y
→ R has at least one saddle point ( x ∗ , y ∗ ) .
We also suppose that for any fixed y ∈ R d y , the problem min x ∈ R
dx f ( x, y ) has a nonempty solution set and a finite optimal value; and for any fixed x ∈ R d x , the problem max y ∈ R dy f ( x, y ) has a nonempty solution set and a finite optimal value.
The goal of solving minimax problem under two-sided PL condition is finding an ϵ -approximate solution or ϵ -saddle point that is defined as follows.
Definition 3.3.
We say x is an ϵ -approximate solution of problem (1) if it holds that g ( x ) − g ( x ∗ ) ≤ ϵ , where g ( x ) = max y ∈ R dy f ( x, y ) .
Definition 3.4.
Under Assumption 3.3, we say ( x, y ) is an ϵ -saddle point of problem (1) if it holds that ∥ x − x ∗ ∥ 2 + ∥ y − y ∗ ∥ 2 ≤ ϵ for some saddle point ( x ∗ , y ∗ ) .
We allow the saddle point does not exist for the problem with one-sided PL condition.
In such case, it is guaranteed that g ( x ) ≜ max y ∈ R dy
f
( x, y ) is differentiable [32, Lemma A.5] and we target to find an ϵ -stationary point of g ( x ) .
Definition 3.5.
If the function g : R d x → R is differentiable, we say x is an ϵ -stationary point of g if it holds that ∥∇ g ( x ) ∥ ≤ ϵ .
A Faster Algorithm for the Two-Sided PL Condition
We first consider the two-sided PL conditioned minimax problem of the finite-sum form (1) under
Assumption 3.1, 3.2 and 3.3.
We propose a novel stochastic algorithm, which we refer to as SPIDERGDA.
The detailed procedure of our method is presented in Algorithm 1.
SPIDER-GDA constructs the stochastic recursive gradient estimators [12, 31] as follows:
[Equation]
It simultaneously updates two variables x and y by estimators G x and G y with different stepsizes τ x = Θ(1 / ( κ 2 y L )) and τ y = Θ(1 /L ) respectively.
Huang et al.
[16], Luo et al.
[25], Xian et al.
[44] have studied the SPIDER-type algorithm for nonconvex-strongly-concave problem and showed it converges to the stationary point of g ( x ) ≜ max y ∈ R dy
f ( x, y ) sublinearly.
However, solving the problem minimax problems with two-sided PL condition desires stronger linear convergence rate, which leads to our theoretical analysis be different from previous work [Equation]
We measure the convergence of SPIDER-GDA by the following Lyapunov function
[Equation]
where x ∗ ∈ arg min x ∈ R
dx g ( x ) and λ =
Θ( κ 2 y ) .
We can establish recursion for V t,k as follows
E [ V t,K ] ≤ E
[Equation]
Using the above inequality by setting M = B = √ n leads to the estimators G x (˜ x t , ˜ y t ) and
G y (˜ x t , ˜ y t ) be sufficiently close to the exact gradient and converge to zero linearly, which indicates g (˜ x t ) also converges to g ( x ∗ ) linearly.
We formally provide the convergence result for SPIDER-GDA in the following theorem and its detailed proof is shown in appendix.
Theorem 4.1.
Under Assumption 3.1, 3.2 and 3.3, we run Algorithm 1 with M = B = √ n , τ y = 1 / (5 L ) , λ =
L 2 /µ
y , τ x = τ y / (24 λ ) , K = ⌈ 4224 / ( µ x τ x ) ⌉ and T =
⌈ log(1 /ϵ )
Then the output (˜ x T , ˜ y T ) satisfies g (˜ x T )
− g ( x ∗ ) ≤ ϵ and g (˜ x T ) − f (˜ x T , ˜ y T ) ≤ 24 ϵ in expectation; and it takes no more than O (cid:0) ( n + √ nκ x κ 2 y ) log(1 /ϵ ) (cid:1) SFO calls.
1: ˜ x 0 = x 0 , ˜ y t
= y2: for t = 0 , 1 , . .
, T − 1 do 3: x t, 0 = ˜ x t , y t, 0 = ˜
y t 4: for k
, K − 1 do 5:
if mod ( k, M ) = 0 then 6:
G x ( x t,k , y t,k ) = ∇ x f ( x t,k , y t,k ) 7:
G y ( x t,k , y t,k ) = ∇ y f
( x t,k , y t,k ) 8: else 9: draw mini-batches S x and S y independently with sizes of B .
G x ( x t,k , y t,k ) =
1 B (cid:80)
i
[Equation]
S x
[ ∇ x f i ( x t,k , y t,k ) −
[Equation]
x f i ( x t,k − 1 , y t,k − 1 ) + G x ( x t,k − 1 , y t,k − 1 )] 11:
G y ( x t,k , y t,k ) =
1 B (cid:80) i ∈
S y [ ∇ y f i ( x t,k , y t,k ) − ∇ y f i ( x t,k − 1 , y t,k − 1 )
+ G y ( x t,k − 1 , y t,k − 1 )] 12: end if 13: x t,k +
= x t,k − τ x G x ( x t,k , y t,k ) 14: y t,k +
= x y,k
+ τ y G y
( x t,k , y t,k ) 15: end for 16: choose
17: end for 18: return
1: u 0 = x2: for k = 0 , 1 , . .
, K − 1 do 3: ( x k +1 , y k +1 )
=
SPIDER-GDA (cid:0) f ( x, y ) + β 2 ∥ x − u k ∥ 2 , ( x k , y k ) , T k , K, M, B, τ x , τ y (cid:1) 4: u k +1 =
x k +1 + γ ( x k +1 − x k ) 5: end for 6: option I (two-sided PL): return ( x K , y K ) 7: option II (one-sided PL): return (ˆ x, ˆ y ) chosen uniformly at random from { ( x k , y k ) } K − 1 k =
Our results provide an SFO upper bound of O (( n + √ nκ x κ 2 y ) log(1 /ϵ )) for finding an ε -approximate solution that is better than the complexity O (( n + n 2 / 3 κ x κ 2 y ) log(1 /ϵ )) derived from SVRGAGDA [45].
It is possible to use SVRG-type [17, 49] estimators to replace the stochastic recursive estimators in Algorithm 1, which results the algorithm SVRG-GDA.
We can prove that SVRGGDA also has O (( n + n 2 / 3 κ x κ 2 y ) log(1 /ϵ ))
SFO upper bound that matches the theoretical result of
SVRG-AGDA.
We provide the details in Appendix C.
Further Acceleration with Catalyst
Both the proposed SPIDER-GDA (Algorithm 1) and existing SVRG-AGDA [45] have the complexities more heavily depend on the condition number of y than the condition number of x .
It is natural to ask can we make the dependency of two condition numbers balanced like the results in the strongly-convex-strongly-concave case [22, 25, 41].
In this section, we show it is possible by introducing the Catalyst acceleration.
To make acceleration possible, we need to assume the uniqueness of the optimal set for inner problem.
Assumption 5.1.
We assume the inner problem max y ∈ R dy f
( x, y ) has an unique solution.
We proposed the accelerated SPIDER-GDA (AccSPIDER-GDA) in Algorithm 2 for reducing the computational cost further.
Each iteration of the algorithm solve the following sub-problem
[Equation]
by SPIDER-GDA (Algorithm 1).
AccSPIDER-GDA has the following convergence result if the sub-problem attain the required accuracy.
Lemma 5.1.
Under Assumption 3.1, 3.2 and 3.3, we run Algorithm 2 by β = 2 L , γ =
and the appropriate setting for the sub-problem solver such that E [ ∥ x k − ˜ x k ∥ 2 + ∥
y k − ˜ y k
∥ 2 ] ≤ δ , where (˜ x k , ˜ y k ) is a saddle point of F k − 1 ( k ≥ 1) and we set the precision
[Equation]
Then it holds that
[Equation]
The setting β = Θ( L ) in Lemma 5.1 guarantees the sub-problem (2) has condition number of the order O (1) for x .
It is more well-conditioned on x , we prefer to address the following equivalent problem
[Equation]
Since (4) is a minimax problem satisfying two sided PL condition, we can apply SPIDER-GDA to solve it.
And we can show that under Assumption 5.1, the saddle point (˜ x k , ˜ y k ) of each F k − 1 ( k ≥ 1) is unique (see Lemma E.2 in appendix) and we are able to obtain a good approximation to it.
Lemma 5.2.
Under Assumption 3.1, 3.2 and 3.3, if we use Algorithm 1 to solve each sub-problem max y ∈ R dy min x ∈ R dx F k ( x, y ) (2) with β = 2 L , M = B = √ n , τ x = 1 / (15 L ) , λ = 288 , τ y =
τ x / (24 λ ) , K = ⌈ 4224 / ( µ y τ y )
⌉ , T k = ⌈ log(1 /δ
k ) ⌉ , then it holds that
[Equation]
where (˜ x k , ˜ y k ) is the unique saddle point of F k − 1 ( k ≥ 1) .
For a short summary, Lemma 5.1 means Algorithm 2 requires O ( κ x log(1 /ϵ )) numbers of inexact proximal point iterations to find an ϵ -approximate solution of the problem.
And Lemma E.1 tells us that each sub-problem can be solved within a SFO complexity of O ( n + √ nκ y ) log(1 /δ k )) .
Thus, the total complexity for AccSPIDER-GDA becomes O (( nκ x + √ nκ x κ y ) log(1 /ϵ )
log(1 /δ k )) .
Our next step is to specify δ k which would lead to the total SFO complexity of the algorithm.
Theorem 5.1.
Under Assumption 3.1, 3.2, 3.3 and 5.1 if we let γ = 0 , β = 2 L and use Algorithmto solve
each sub-problem max y ∈ R dy min x ∈ R dx F k
( x, y ) (2) with M, B, τ x , τ y , K defined as Lemma 5.2 and T k = ⌈ log(1 /δ
k ) ⌉ , where
[Equation]
and δ is followed by the definition in (3).
Then Algorithm 2 can return x K such that g ( x K ) − g ( x ∗ ) ≤ ϵ in expectation with no more than O (( nκ x + √ nκ x κ y ) log(1 /ϵ ) log( κ x κ y /ϵ ))
SFO calls.
Lemma 5.1 does not rely on the choice of sub-problem solver, we can apply the acceleration framework in Algorithm 2 by replacing SPIDER-GDA with other algorithms.
We summarize the
SFO complexities for the acceleration of different algorithms in Table 3.
Extension to One-Sided PL Condition
In this section, we show the idea that SPIDER-GDA and its Catalyst acceleration also work for one-sided PL condition.
We relax Assumption 3.2 and 3.3 to the following one.
[Table]
Assumption 6.1.
We suppose that − f ( x, · ) is µ y -PL for any x ∈ R d x ; the problem max y ∈ R dy f ( x, y ) has a nonempty solution set and an optimal value ; g ( x ) ≜ max y ∈ R dy
f
( x, y ) is lower bounded, i.e., we have g ∗ =
inf x ∈ R dx g ( x ) > −∞ .
We first show that the SFO complexity of SPIDER-GDA outperforms SVRG-GDA 4 by a factor of
O ( n 1 / 6 ) in Theorem 6.1.
Theorem 6.1.
Under Assumption 3.1 and 6.1 , Let T = 1 and M, B, τ x , τ y , λ as defined in Theorem 4.1 and K = ⌈ 64 / ( τ x ϵ 2 )
⌉ , then Algorithm 1 can guarantee the output ˆ x to satisfy ∥∇ g (ˆ x ) ∥ ≤ ϵ in expectation with no more than O ( n + √ nκ 2 y Lϵ − 2 ) SFO calls.
The AccSPIDER-GDA also performs better than SPIDER-GDA in one-sided PL condition for ill conditioned case.
In the following lemma, we show that AccSPIDER-GDA could find an approximate stationary point if we solve the sub-problem sufficiently accurate.
Lemma 6.1.
Under Assumption 3.1 and 6.1, if it holds true that E [ ∥ x k − ˜ x k ∥ 2 + ∥
y k − ˜ y k
∥ 2 ] ≤ δ for some saddle point (˜ x k , ˜ y k ) of F k − 1 ( k ≥ 1) , where
[Equation]
Let β = 2 L , then for the output (ˆ x, ˆ y ) of Algorithm 2, it holds true that
[Equation]
Compared with two-sided PL condition, the analysis of AccSPIDER-GDA is more complicated since the precision δ k at each round are different.
By choosing the parameters of the algorithm carefully, we obtain the following results.
Theorem 6.2.
Under Assumption 3.1, 6.1 and 5.1, if we run Algorithm 2 by γ = 0 , β = 2 L and use
Algorithm 1 to solve each sub-problem max y ∈ R dy min x ∈ R dx F k ( x, y ) (2) with M, B, τ x , τ y , λ, K and T k (dependent on δ ) as in Theorem 5.1 and δ is followed by the definition in Lemma 6.1, then
Algorithm 2 can find ˆ x such that ∥∇ g (ˆ x ) ∥ ≤ ϵ in expectation within O (( n + √ nκ y )
Lϵ − 2 log( κ y /ϵ ))
SFO calls.
We can directly set β = 0 for Algorithm 2 in the case of very large n and in this case AccSPIDERGDA reduces to SPIDER-GDA.
The summary and comparison for the complexities for the one-sided
PL condition is shown in Table 2.
Besides, the algorithms of GDA and SVRG-GDA also can be accelerated with Catalyst framework and we present the corresponding results in Table 4.
[Table]
Experiments
In this section [Table] we conduct the numerical experiments to show the advantage of proposed algorithms and the source code is available 5 .
We consider the following two player Polyak-Łojasiewicz game:
[Table]
[Equation]
[Table]
[Equation]
[Table]
where
P = 1 n n (cid:88) i
= p
i
p ⊤ i ,
Q = 1 n n (cid:88) i =
q i q ⊤ i and R = 1 n n (cid:88) i =
r i r ⊤ i .
We independently sample p i , q i and r i from N (0 , Σ P ) , N (0 , Σ Q ) and N (0 , Σ R ) respectively.
We set the covariance matrix Σ P as the form of UDU ⊤ such that U ∈ R
d × r is column orthogonal matrix and D ∈ R r × r is diagonal with r < d .
The diagonal elements of D are distributed uniformly in the interval [ µ, L ] with 0 < µ < L .
The matrix Σ Q is set by the similar way to Σ P .
We also let Σ R = 0 .
1 V V ⊤ , where each element of V ∈ R d × d is sampled from N (0 , 1) independently.
Since the covariance matrices Σ P and Σ Q are rank-deficient, it is guaranteed that both P and Q are singular.
Hence, the objective function is not strongly-convex and not strongly-concave, but it satisfies the two-sided PL-condition [18].
We set n = 6000 , d = 10 , r = 5 , L = 1 for all experiments; and let µ be 10 − 5 and 10 − 9 for two different settings.
We compare the proposed SPIDER-GDA (Algorithm 1) and AccSPIDER-GDA (Algorithm 2) with the baseline algorithm SVRG-AGDA [45].
We let B = 1 and M = n for all of these algorithms and both of the stepsizes for x and y are tuned from { 10 − 1 , 10 − 2 , 10 − 3 , 10 − 4 , 10 − 5 } .
For AccSPIDER, we set β = L/ (20 n ) and γ = 0 .
We present the results of the number of SFO calls against the norm of gradient and the distance to the saddle point in Figure 1 and Figure 2.
It is clear that our algorithms outperform than baselines.
Conclusion and Future Work
In this paper, we have investigated stochastic optimization for PL conditioned minimax problem with the finite-sum objective.
We have proposed the SPIDER-GDA algorithm, which reduces the dependency of the sample numbers in SFO complexity.
Moreover, we have introduced a Catalyst scheme to accelerate our algorithm for solving the ill-conditioned problems.
We improve the SFO upper bound of the state-of-the-art algorithms for both two-sided and one-sided PL conditions.
However, the optimality of SFO algorithms for the PL conditioned minimax problem is still unclear.
It is interesting to construct the lower bound for verifying the tightness of our results.
It is also possible to extend our algorithm to online setting.
[Figure]
SVRG
SPIDER
AccSPIDER
[Figure]
SVRG
SPIDER
AccSPIDER
[Figure] [Figure]
Acknowledgements
This work is supported by National Natural Science Foundation of China (No. 62206058) and
Shanghai Sailing Program (22YF1402900)