SAPD+ : An Accelerated Stochastic Method for Nonconvex-Concave Minimax Problems
Abstract
We propose a new stochastic method SAPD+ for solving nonconvex-concave minimax problems of the form min max L ( x, y ) =
f ( x ) + Φ( x, y ) − g ( y ) , where f, g are closed convex and Φ( x, y ) is a smooth function that is weakly convex in x , (strongly) concave in y .
For both strongly concave and merely concave settings,
SAPD+ achieves the best known oracle complexities of O
( Lκ y (cid:15) − 4 ) and O ( L 3 (cid:15) − 6 ) , respectively, without assuming compactness of the problem domain, where κ y is the condition number and L is the Lipschitz constant.
We also propose SAPD+ with variance reduction, which enjoys the best known oracle complexity of O
( Lκ 2 y (cid:15) − 3 ) for weakly convex-strongly concave setting.
We demonstrate the efﬁciency of
SAPD+ on a distributionally robust learning problem with a nonconvex regularizer and also on a multi-class classiﬁcation problem in deep learning.
Introduction
We consider the following saddle-point (SP) problem:
[Equation]
where X and Y are, n and m dimensional Euclidean spaces the function Φ :
X × Y → R is smooth and possibly nonconvex in x ∈ X and µ y -strongly concave in y ∈ Y for some µ y ≥ 0 –with the convention that for µ y = 0 , Φ is merely concave (MC) in y , and the functions f and g are closed, convex and possibly nonsmooth.
In this paper, we consider a particular case of nonconvexity, i.e., we assume that Φ( · , y ) is weakly convex (WC) for any ﬁxed y ∈ dom g ⊂ Y .
Weakly convex functions constitute a rich class of non-convex functions and arise naturally in many practical settings for machine learning (ML) applications [9, 37], precise deﬁnitions will be given later in Section 2.
In practice, WC assumption is widely satisﬁed, e.g., under smoothness –see remark 1, in fact it can be argued that most of the work in related literature considering nonconvex-(strongly) concave SP problems provide their analyses under the premise of weak convexity.
The problem (1) with µ y >is called a weakly convex-strongly concave (WCSC) saddle-point problem, whereas for µ y = 0 , it is called a weakly convex-merely concave (WCMC) saddle-point problem.
Both problems arise frequently in many ML problems including constrained optimization of WC objectives based on
Lagrangian duality [22], Generative Adversarial Networks (GAN) (where x denotes the parameters of the generator network whereas y represents the parameters of the discriminator network [13]), distributional robust learning with weakly convex loss functions such as those arising in deep learning [14, 37] and learning problems with non-decomposable losses [37].
There are two important settings for (1): (i) the deterministic setting , where the partial gradients of Φ are exactly available, (ii) the stochastic setting , where only stochastic estimates of the gradients are available.
Although, recent years have witnessed signiﬁcant advances in the deterministic setting [7, 17, 19, 23, 24, 26, 35, 38, 40]; our focus in this paper will be mainly the stochastic setting which is more relevant and more applicable to ML problems.
Indeed, due to large-dimensions and the sheer size of the modern datasets, computing gradients exactly is either infeasible or impractical in ML practice and gradients are often estimated stochastically based on mini-batches (randomly sampled subset of data points) as in the case of stochastic gradient-type algorithms.
There is a growing literature on the WCSC and WCMC problems in the stochastic setting.
Several metrics for quantifying the quality of an approximate solution to (1) have been proposed in the literature.
A common way to assess the performance is to deﬁne the primal function φ ( · ) (cid:44) max y ∈Y L ( · , y ) and measure the violation of ﬁrst-order necessary conditions for the non-convex problem min x ∈X φ ( x ) .
Given the primal iterate sequence { x k } k ≥ 0 of a stochastic SP algorithm and a threshold (cid:15) > 0 , a commonly used metric is the gradient norm of the Moreau envelope (GNME); indeed, the objective is to provide a bound K (cid:15) such that E [ (cid:107)∇ φ λ ( x k ) (cid:107) ] ≤ (cid:15) for all k ≥ K (cid:15) , where φ λ denotes the Moreau envelope of the primal function φ –see Deﬁnitions 3, 4 and 5.
Another commonly used natural metric is the gradient norm of the primal function φ ( · ) [4, 17, 16, 27, 39], abbreviated as GNP, where the aim is to derive K (cid:15) such that E [ (cid:107)∇ φ ( x k ) (cid:107) ] ≤ (cid:15) for all k ≥ K (cid:15) .
Other metrics such as the notion of (cid:15) -ﬁrst-order Nash equilibrium (FNE) and its generalized versions also exist in the literature [34, 35].
When using any of the aforementioned metrics, the ultimate goal is to establish a bound on the oracle (sampling) complexity, i.e., (cid:80) K (cid:15) k
=
b k , where b k denotes the batch-size for iteration k ≥ 0 .
For the
WCSC setting, it crucial to note that GNME, GNP and FNE metrics are all equivalent in the sense that convergence in either of them implies convergence in the other two metric for WCSC problems [23].
In this paper, for the WCSC setting, we adopt both GNME and GNP as the main performance metrics to analyze our algorithms; indeed, in Theorem 2 we show that, when the non-smooth part f ( · ) = 0 , we can convert a GNME guarantee to a GNP guarantee by incurring only little additional cost compared to the computational cost required for the GNME guarantee, and the overall worst-case complexity (in terms of worst-case dependency to the target accuracy (cid:15) ) remains the same for both metrics.
When the non-smooth part f ( · ) (cid:54) = 0 , we also obtain similar guarantees and show equivalence between the metrics based on GNME and the generalized gradient mapping.
On the other hand, for the WCMC setting, we provide our guarantees in GNME metric as φ is not necessarily differentiable for this scenario.
Moreover, our work accounts for the individual effects of L xx , L xy , L yx and L yy , i.e., the Lipschitz constants of ∇ x Φ( · , y ) , ∇ x Φ( x, · ) , ∇ y Φ( x, · ) and ∇ y Φ( · , y )
(see Assumption 2), respectively, instead of using the worst-case parameters L (cid:44) max { L xx , L xy , L yx , L yy } , while the majority of related work ignore the inﬂuence of these block Lipschitz constants in their analyses.
We emphasize that using the worst-case parameters will lead to a theoretically conservative step sizes, and this phenomenon has been validated in the work [44].
Contributions.
Table 1 summarizes the relevant existing work for WCSC and WCMC problems closest to our setting.
More speciﬁcally, in Table 1, for the stochastic setting, we report the (oracle) complexity with respect to the GNP and GNME as the performance metrics for WCSC and WCMC problems, respectively, and the batch-size (number of data points in the mini-batches) required at every iteration.
We also report whether the method is based on a variance-reduction (VR) technique.
VR-based methods mentioned in Table 1 use a small batch-size b (cid:48) in general and need a large batchsize b ≥ b (cid:48) once in every q iterations.
The period q is equal to the number of times small batches are sampled consecutively plus one, and it is also an algorithm parameter.
Therefore, for VR-methods, we report the batch size as a triplet ( b (cid:48) , b, q ) .
In the column “Compactness", we list whether achieving the speciﬁc complexity requires assuming compactness of the primal and/or dual domains.
[Table]
Table notes: ∗ For WCSC setting, [37] assumes Φ( · , y ) (cid:44) c
(cid:62) ( · ) y is weakly convex and g ( · ) is strongly convex.
In [41], L = Φ and Φ need not be smooth, rather second moment of stochastic subgradients is assumed to be uniformly bounded.
When Φ is L -smooth, Φ( · , y ) and Φ( x, · ) are L Φ -Lipschitz, the results in [41] imply
O ( L 2Φ κ 2 y (cid:15) − log
2 ( √ κ y
L
Φ
/(cid:15)
) complexity.
‡ , § The complexity results reported here are different than those in [17, 16].
The issues in their proofs leading to the wrong complexity results are explained in Appendix I.
The notation ˜ O ignores logarithmic factors
To make the comparison of our results with the existing work easier, we provide the results in the table for the worst-case setting, where κ y (cid:44)
Lµ y , and we report the (cid:15) -, κ y - and L -dependency of the complexity results for the existing algorithms.
That being said, our results have ﬁner granularity in terms of their dependence to the individual effects of L xx , L xy , L yx and L yy as we mentioned earlier.
Our contributions (also summarized in section 1) are as follows:
• We propose a new stochastic method, SAPD+ , based on the inexact proximal point method (iPPM).
In this framework, one inexactly solves strongly convex-strongly concave (SCSC) saddle point sub-problems using an accelerated primal-dual method, SAPD [44].
In Theorem 3, we establish an oracle complexity of O
( Lκ y (cid:15) − 4 ) for WCSC problems, and unlike the majority of existing work we do not require compactness for neither the primal nor the dual domain.
To our knowledge, our bound has the best κ y dependence in the literature; indeed, prior to this work, without using variance reduction, the best known complexity was O
( Lκ 3 y (cid:15) − 4 ) ; hence, we establish a O ( κ 2 y ) improvement.
• We propose a variance-reduced version of SAPD+ in Theorem 4.
For WCSC setting, SAPD+ using variance reduction achieves an oracle complexity of O
( Lκ 2 y (cid:15) − 3 ) –this bound has the best (cid:15) -dependency in the literature to our knowledge, and among all the methods with the O ( (cid:15) − 3 ) complexity, our approach has the best condition number, κ y , dependency; indeed, prior to this work, the best known complexity was O
( Lκ 3 y (cid:15) − 3 ) ; hence, we establish O ( κ y ) factor improvement.
• For the WCMC case, our proposed algorithm SAPD+ results in O ( L 3 (cid:15) − 6 ) complexity, which is the best to our knowledge, improving the best known complexity by log 3 ( L/(cid:15) 2 ) factor.
Finally, we demonstrate the efﬁciency of SAPD+ on a distributionally robust learning problem and also on a (worst-case) multi-class classiﬁcation problem in deep learning.
Notation.
Throughout the paper, (cid:107) · (cid:107) denotes the Euclidean norm.
Given f : R n → R ∪ {∞} a closed convex function, prox λf ( x ) (cid:44) argmin w f ( w ) + 1 2 λ (cid:107) w − x (cid:107) 2 denotes the proximal map of f .
Given random ω , let ˜ ∇ x Φ( x, y ; ω ) and ˜ ∇ y Φ( x, y ; ω ) denote unbiased estimators of ∇ Φ x ( x, y ) and ∇ Φ y ( x, y ) .
Moreover, given a random mini-batch B = { ω i } bi =1 , we let ˜ ∇ x Φ B ( x, y )
= 1 b (cid:80) bi =1 ˜ ∇ x Φ( x, y ; ω i ) to denote the stochastic gradient estimate based on the batch
B , and we deﬁne ˜ ∇ y Φ B ( · , · ) similarly.
Preliminaries
We start with describing the notion of weak convexity.
Deﬁnition 1.
h : R d → R ∪ { + ∞} is γ -weakly convex if x (cid:55)→ h ( x ) + γ 2 (cid:107) x (cid:107) 2 is convex.
Deﬁnition 2.
A differentiable function h : R d → R ∪ { + ∞} is L -smooth if ∃ L > 0 such that for ∀ x, x (cid:48) ∈ dom h , (cid:107)∇ h ( x ) − ∇ h
( x (cid:48) )
(cid:107) ≤ L (cid:107) x − x (cid:48)
(cid:107) .
Remark 1.
If a function is L -smooth, then it is also L -weakly convex.
Remark 1 shows that weak convexity is a rich class containing the class of smooth functions.
In the rest of the paper, we consider the SP problem in (1).
Next, we introduce our assumptions.
Assumption 1.
f : X → R ∪ {∞} and g :
Y → R ∪ {∞} are proper, closed, convex functions.
Let Φ : X × Y → R be such that (i) for any y ∈ dom g ⊂ Y , Φ( · , y ) is γ -weakly convex and bounded from below; (ii) for any x ∈ dom f ⊂ X , Φ( x, · ) is µ y -strongly concave for some µ y ≥ 0 ; (iii) Φ is differentiable on an open set containing dom f × dom g .
Assumption 2.
There exist L xx , L yy ≥ 0 , L xy , L yx > 0 such that (cid:107)∇ x Φ( x, y ) − ∇ x Φ(¯ x, ¯ y ) (cid:107) ≤
L xx (cid:107) x − ¯ x (cid:107) + L xy (cid:107)
y − ¯ y (cid:107) , and (cid:107)∇ y Φ( x, y ) − ∇ y Φ(¯ x, ¯ y ) (cid:107) ≤
L yx (cid:107) x − ¯ x (cid:107) + L yy (cid:107)
y − ¯ y (cid:107) for all x, ¯ x ∈ dom f ⊂ X , and y, ¯ y ∈ dom g ⊂ Y .
Assumption 1 allows non-convexity in x while requiring (strong) concavity in the y variable.
Assumption 2 is standard in the analysis of ﬁrst-order methods for solving SP problems.
It should be noticed that when L yx =
L xy = 0 , the problem in (1) can be solved separately for the primal and dual variables; hence, it is natural to assume L yx , L xy > 0 .
Suppose that we implement SAPD , stated in Algorithm 1, on the SCSC problem
[Equation]
(2) for some given µ x > 0 and x 0 ∈ X –strong convexity follows from L ( · , y ) being γ -weakly convex.
[Table]
We make the following assumption on the statistical nature of the gradient noise as in, e.g., [5, 11, 44].
Assumption 3.
Given arbitrary x 0 ∈ X and µ x > 0 , let { x k , y k } sequence be generated by SAPD , stated in Algorithm 1, running on (2) .
There exist δ x , δ y ≥ 0 such that for all k ≥ 0 , the stochastic gradients ˜ ∇ x Φ( x k , y k +1 ; ω xk ) , ˜ ∇ y Φ( x k , y k ; ω yk ) and random sequences { ω xk } k , { ω yk } k satisfy the conditions:
(i) E
[ ˜ ∇ x Φ( x k , y k +1 ; ω x k ) | x k , y k +1 ] = ∇ x Φ( x k , y k +1 ) ; (ii) E [ ˜ ∇ y Φ( x k , y k ; ω yk ) | x k , y k ] = ∇ y Φ( x k , y k ) ; (iii) E [ (cid:107) ˜ ∇ x Φ( x k , y k +1 ; ω xk ) − ∇ x Φ( x k , y k +1 )
(cid:107) 2 | x k , y k +1 ] ≤ δ 2 x ; (iv) E
[ (cid:107) ˜ ∇ y Φ( x k , y k ; ω yk ) − ∇ y Φ( x k , y k ) (cid:107) 2 | x k , y k ] ≤ δ 2 y .
Assumption 3 says that the gradient noise conditioned on the iterates is unbiased with a ﬁnite variance 1 .
Such assumptions are common in the literature, e.g., [5, 11, 44], and can be satisﬁed when gradients are estimated from randomly sampled data points with replacement.
For WCSC minimax problems, a commonly adopted deﬁnition for (cid:15) -stationary is based on Moreau envelope, e.g., see [23, 41].
It is inspired by Davis and Drusvyatskiy’s work [9] for solving weakly convex minimization problems.
For the sake of completeness, we brieﬂy review this idea below.
Deﬁnition 3.
Let φ : R d → R ∪ { + ∞} be γ -weakly convex.
Then, for any λ ∈ (0 , γ − 1 ) , Moreau envelope of φ is deﬁned as φ λ : R d → R such that φ λ ( x ) (cid:44) min w ∈X φ ( w ) + 12 λ (cid:107) w − x (cid:107) 2 .
Lemma 1.
Let φ : R d → R ∪ { + ∞} be a γ -weakly convex function.
For any given λ ∈ (0 , γ − 1 ) , φ λ ( · ) is well-deﬁned on X .
Moreover, ∇ φ λ ( x ) =
λ − 1 ( x − prox λφ ( x )) for x ∈ X ; hence, φ λ is λ − 1 -smooth, where prox λφ ( x ) = argmin w ∈X { φ ( w ) + 12 λ (cid:107) w − x (cid:107) 2 } .
Deﬁnition 4.
Under Assumption 1, let φ, φ s : R d → R ∪ { + ∞} such that φ ( x ) (cid:44) max y ∈Y L ( x, y ) and φ s ( x ) =
φ ( x ) − f ( x ) for x ∈ X , i.e., φ s ( x ) (cid:44) max y ∈Y Φ( x, y ) − g ( y ) for x ∈ dom f .
Remark 2.
Under Assumption 1, since Φ( · , y ) is γ -weakly convex for any y ∈ dom g , φ s is γ -weakly convex 2 ; hence, φ is also γ -weakly convex.
Note that
[Equation]
Furthermore, when µ y > 0 , φ s is differentiable on dom f .
In the following deﬁnition, we introduce the notion of (cid:15) -stationary with respect to the GNME metric.
Deﬁnition 5.
A point x (cid:15) is an (cid:15) -stationary point of a γ -weakly convex function φ if (cid:107)∇ φ λ ( x (cid:15) ) (cid:107) ≤ (cid:15) for some λ ∈ (0 , γ − 1 ) .
If (cid:15) = 0 , then x (cid:15) is a stationary point of φ .
Thus, from Lemma 1, computing an (cid:15) -stationary point x (cid:15) for φ is equivalent to searching for x (cid:15) such that (cid:107) x (cid:15) − prox λφ ( x (cid:15) )
(cid:107) is small.
Recall that for any λ ∈ (0 , γ − 1 ) , prox λφ ( x ) is well-deﬁned and unique.
We also observe from (3) that prox λφ ( · ) computation is indeed an SCSC SP problem.
To compute x (cid:15) such that (cid:107) x (cid:15) − prox λφ ( x (cid:15) ) (cid:107) is small, it is natural to consider the iPPM algorithm –see e.g. [18].
A generic iPPM generates { x t 0 } t ≥ 0 such that x t + ≈ prox λφ ( x t 0 ) , i.e. proximal steps are “inexactly” computed for t ≥ 0 , starting from an arbitrary given point x 00 ∈ X .
In the next section, we describe the proposed SAPD+ method, an iPPM algorithm employing SAPD to inexactly solve the SCSC subproblems arising in the iPPM iterations.
The proposed algorithm SAPD+ and its analysis
The convergence and robustness properties of SAPD for SCSC SP problems are analyzed in [44].
For the WCSC SP problems, as we explained in the previous section, our main idea is to apply the iPPM framework as stated in SAPD+ (see algorithm 2) which requires successively solving SCSC SP problems.
In the rest, the counter for iPPM outer iterations is denoted with t ∈ Z + .
At each outer iteration t ≥ 1 , we inexactly compute the prox map, i.e., x t + ≈ prox λφ ( x t 0 ) , which is well-deﬁned for λ ∈ (0 , γ − 1 ) ; hence, to derive our preliminary results, we ﬁx λ
= ( µ x + γ ) − 1 for some given µ x > 0 – thus, L ( x, y ) + µ x + γ 2 (cid:107) x − x t 0 (cid:107) 2 is SCSC in ( x, y ) with moduli ( µ x , µ y ) and has a unique saddle point.
Consider the following SCSC SP problem: min x ∈X max y ∈Y L t ( x, y ) (cid:44) f
( x )+Φ t ( x, y ) − g
( y ) , where Φ t ( x, y ) =
Φ( x, y )+ µ x + γ(cid:107) x − x t 0 (cid:107) 2 .
1: Input: { τ, σ, θ, µ x } , ( x 00 , y 00 )
∈
X × Y , { N t } t ≥ 0 ∈ Z + 2: for t = 0 , 1 , 2 , ..., T do 3: if VR-flag =
=
false then 4: ( x t +10 , y t +10 ) ← SAPD ( τ, σ, θ, µ x , x t 0 , y t 0 , N t ) 5: else 6: ( x t +10 , y t +10 ) ← VR-SAPD ( τ, σ, θ, µ x , x t 0 , y t 0 , N t ) 7: end if 8: end for
We will construct { x t 0 } Tt =1 ⊂ dom f by inexactly solving (4) at each outer iteration t ∈ Z + through running SAPD for N t ∈ Z + iterations –we will specify N t ∈
Z + later.
Next, we brieﬂy explain the main step of SAPD+ with
VR-flag = false .
The statement in line 4 of algorithm 2 means that ( x t +10 , y t +10 ) is generated using SAPD , as stated in algorithm 1 –indeed, SAPD is run on (4) for N t iterations with SAPD parameters ( τ, σ, θ ) and starting from the initial point ( x t 0 , y t 0 ) .
To analyze the convergence of SAPD+ , we ﬁrst deﬁne the gap function G t for t -th SAPD+ iteration:
[Equation]
Recall that L t is an SCSC function; therefore, i) it has a unique saddle point denoted by ( x t ∗ , y t ∗ ) , and it is important to note that x t ∗ =
prox λφ ( x t 0 ) for φ ( x ) = max y ∈Y L ( x, y ) and λ = ( γ + µ x ) − 1 ; ii) for any ( x, y ) ∈
dom f × dom g , the following quantities are well-deﬁned:
[Equation]
Thus, it follows that G t ( x, y ) =
L t ( x, y ∗ ( x )) − L t ( x t ∗ ( y ) , y ) .
Moreover, we also deﬁne G ( x, y ) (cid:44) sup y (cid:48) ∈Y L ( x, y (cid:48) ) − inf x (cid:48)
∈X L ( x (cid:48) , y ) .
Assumption 1 ensures that G is well deﬁned.
Next, we ﬁrst provide our oracle complexity in the GNME metric under the compactness assumption of the primal-dual domains; later, in section 3.1, we show that under a particular subdifferentiability assumption compactness requirement can be avoided.
Assumption 4.
dom f and dom g are compact sets.
Theorem 1.
Suppose Assumptions 1, 2, 3, and 4 hold.
Let µ x = γ , θ = 1 , τ, σ and N be chosen as
[Equation]
(7) Then, for any (cid:15) > 0 , when VR-flag = false , SAPD+ guarantees (cid:15) -stationary, min t =0 ,...,T E
[ (cid:107)∇ φ λ ( x t 0 ) (cid:107) ] ≤ (cid:15) , for T ≥ 96 G ( x 00 , y 00 ) · γ(cid:15) 2 + 1 , which requires C (cid:15) stochastic ﬁrst-order oracle calls in total where
[Equation]
Proof.
See appendix A for the proof.
Remark 3.
Since E [min t =0 ,...,T (cid:107)∇ φ λ ( x t 0 ) (cid:107) ] ≤ min t =0 ,...,T E [ (cid:107)∇ φ λ ( x t 0 ) (cid:107) ] , the guarantees given in Theorem 1 also hold for achieving E (cid:2) min t =0 ,...,T (cid:107)∇ φ λ ( x t 0 ) (cid:107) (cid:3) ≤ (cid:15) .
Remark 4.
For any y ∈ dom g , since Φ( · , y )
L xx -smooth, it is necessarily L xx -weakly convex; hence, γ ≤ L xx .
To get a worst-case complexity, let
[Equation]
[Equation]
[Equation]
Our oracle complexity C (cid:15) in Theorem 1 can be simpliﬁed as C (cid:15)
[Equation]
O
[Equation]
[Equation]
In fact, Li et al.
[21] (see also [43]) provide a lower complexity bound for a class of ﬁrst-order stochastic algorithms that do not use variance reduction.
The lower bound for ﬁnding (cid:15) -stationary points of smooth WCSC problems in GNP metric is Ω( L ∆ φ ( √ κ y (cid:15) − 2 + κ 13 y (cid:15) − 4 )) ,
where ∆ φ (cid:44) φ ( x 0 ) − min x ∈X φ ( x ) and x 0 is an arbitrary initial point.
Consider φ = f
+ φ s as given in deﬁnition 4.
For λ > 0 , the map G λ : R d → R d deﬁned as
[Equation]
9) is called the generalized gradient mapping and its norm is frequently used in optimization for assessing stationarity (see e.g. [10]).
Theorem 1 provides guarantees in the GNME metric.
Theorem 2 shows that given x (cid:15) , an (cid:15) -stationary point in GNME metric (see deﬁnition 5) in expectation, we can generate ˜ x such that E [ (cid:107) G λ
(˜ x ) (cid:107) ] ≤ (cid:15) for some λ > 0 , i.e., an (cid:15) -stationary point in generalized gradient mapping metric, within ˜ O
(1 /(cid:15) 2 ) SAPD iterations.
Indeed, when f ( · ) = 0 , this metric and the GNP metric are the same.
Theorem 2.
Suppose Assumptions 1, 2, 3 hold.
Given x (cid:15) , an (cid:15) -stationary point for the γ -weakly convex function φ ( · ) =
max y ∈Y L ( · , y ) in expectation, i.e., E [ (cid:107)∇ φ λ ( x (cid:15) )
(cid:107) ] ≤ (cid:15) 2 for some ﬁxed λ ∈ (0 , γ − 1 ) .
Consider φ s ( · ) =
max y ∈Y Φ( · , y ) − g
( y )
so that φ = f
+ φ s as in Deﬁnition 4.
Then, initialized from x (cid:15) , SAPD , stated in algorithm 1, can generate ˜ x such that E (cid:2) (cid:107) G λ
(˜ x ) (cid:107) ≤ (cid:15) within ˜ O
( 1 (cid:15) 2 ) stochastic ﬁrst-order oracle calls.
Proof.
See appendix B for the proof.
Remark 5.
Based on Remark 3, the random vector x ε in Theorem 2 can be chosen as x t ∗ 0 where t ∗ (cid:44) argmin 0 ≤ t ≤ T (cid:107)∇ φ λ ( x t 0 ) (cid:107) .
Remark 6.
Suppose SAPD+ requires T (cid:15) iterations to output x (cid:15) such that E
[ (cid:107)∇ φ λ ( x (cid:15) ) (cid:107) ] ≤ (cid:15) 2 .
Then generating ˜ x from x (cid:15) such that E (cid:2) (cid:107) G λ
(˜ x ) (cid:107) (cid:3) ≤ (cid:15) requires an additional SAPD+ iteration, i.e., T (cid:15) +iterations in total, where in the last iteration we run SAPD for ˜ O
( (cid:15) − 2 ) iterations rather than N with a different choice of parameters τ, σ, θ –see eq.
(35) in appendix B for our speciﬁc choice.
3.1 Relaxing the compactness assumption
In Theorem 1, we assume that dom f and dom g are compact sets, e.g., f ( · ) = 1 X ( · ) and g ( · ) =
Y ( · ) , where X ⊂ X and Y ⊂ Y are compact convex sets.
In this section, we show that SAPD+ can also handle unbounded domains under the following assumption.
Assumption 5.
For f and g closed convex, suppose ∃ B f , B g >
such that inf {(cid:107) s f (cid:107) : s f ∈ ∂f ( x ) }
≤ B f for all x ∈ dom f and inf {(cid:107) s g (cid:107) :
s g ∈ ∂g
( y ) } ≤ B g for all y ∈ dom g .
Remark 7.
Assumption 5 holds when f is an indicator function of a closed convex set (not necessarily bounded) or for f : R d → R ∪ {∞} such that dom f is open and f is Lipschitz.
Two important examples for this scenario: (i) f ( · ) = 0 , (ii) f is a norm, e.g., (cid:96) 1 -, (cid:96) 2 -, or the Nuclear norms.
The existing work based on iPPM framework either require compactness, e.g., [41], or some special structure on L , e.g., [37].
This is also true for VR-based methods, e.g.,[16, 27, 39].
To our knowledge, ours is the ﬁrst one to overcome this difﬁculty and strictly improve the best known complexity bound for the WCSC setting without compactness assumption; moreover, the same idea also works simultaneously with a variance reduction technique that will be discussed later (see section 4).
Finally, the same trick for removing compactness assumption for the WCSC setting also helps removing the compactness assumption for the primal domain in WCMC setting and we still improve the best known complexity for this setting as well (see section 5).
Remark 8.
In [23], when f = g = 0 , boundedness of dual space is required while Assumption 5 is a weaker requirement.
Furthermore, based on the discussion with the authors of [41], compactness of the domain is needed for their proof to hold.
In [17], the sub-level set { x : φ ( x ) + f ( x ) ≤ α } is required to be compact for all α > 0 .
There are simple convex functions that do not satisfy this condition such as f ( x ) = max { 0 , x } .
Bot and ¨Bohm [4] use milder assumptions than [23] without requiring compactness; however, their complexity is the same as the complexity of [23].
Theorem 3.
The result of Theorem 1 continues to hold, if one replaces the compact domain assumption, i.e., Assumption 4, with Assumption 5.
Proof.
See appendix D for the proof.
Variance reduction
[Table] [Table]
|
[Table]
|
[Table]
, y k +1 ) 17: else 18:
Let
I
yk
+1 be mini-batch sample with |I yk +
| =
b y 19: ˜ q k +1 ← ˜ ∇ y Φ I yk +
( x k +1 , y k +1 )
− ˜ ∇ y Φ I yk +
( x k , y k ) 20: w k +1 ← w k + ˜ q k
+1 21: end if 22: ˜ s k +1 ← (1 + θ ) w k +1 − θw k 23: end for 24: Output : (¯ x N , ¯ y N ) = 1 N (cid:80) N − 1 k =
( x k +1 , y k +1 )
Variance reduction techniques have been found useful for solving SCSC problems in ﬁnite sum form, e.g., [36] –see also [6] for improving error bounds using Richardson-Romberg extrapolation in solving SCSC problems with noisy gradients.
In this section, we equip SAPD+ with SPIDER variance reduction technique [12], a variant of SARAH [32, 33].
More precisely, for inexactly solving SCSC subproblems given in (4), we propose using VR-SAPD as stated in algorithm 3.
Note VR-SAPD employs a large batchsize of b in every q iterations and use small batchsizes of b x and b y for the rest.
We prove that
SAPD+ using variance reduction, i.e., with VR-flag = true , achieves an oracle complexity of O
( Lκ 2 y (cid:15) − 3 ) ; hence, we show an O ( κ y ) factor improvement over the best known complexity in the literature to our knowledge.
Here, we use ˜ ∇ y Φ t B k ( x k , y k ) to represent 1 |B k |
(cid:80) ω
ik ∈B k ˜ ∇ y Φ( x k , y y ; ω ik ) , where B k = { ω ik } bi =1 is the mini-batch with |B k | =
b and we deﬁne ˜ ∇ x Φ t B k ( x k , y k +1 ) similarly.
In addition, I xk = { ω x,ik } and I yk = { ω y,ik } with |I xk | =
b x and |I yk | =
b y denote the small mini-batches for generating ˜ ∇ y
Φ t I yk ( x k , y k ) and ˜ ∇ x Φ t I xk ( x k , y k +1 ) .
When we run VR-SAPD on a generic subproblem as in (2), we use the convention that ˜ ∇ x ¯Φ B k ( x k , y k +1 ) (cid:44) ˜
∇ x Φ B k ( x k , y k +1 )
+ ( µ x + γ )( x k − x 0 ) .
Throughout this section we make a continuity assumption on the stochastic ﬁrst-order oracles similar to [17, 16, 27, 39].
Assumption 6.
∃ L xx , L xy , L yx , L yy ≥ 0 such that ∀ x, ¯ x ∈ dom f ⊂ X and ∀ y, ¯ y ∈ dom
g ⊂ Y ,
[Equation]
Assumption 7.
Consider SAPD+ with VR-flag = true .
For any k ≥ 0 , I xk and I yk consist of independent elements; moreover, I xk is independent of ( x i , y i +1 ) and I yk is independent of ( x i , y i ) for i ∈ { k − 1 , k } .
Remark 9.
For ﬁnite-sum type problems of the form min x max y 1 n (cid:80) ni
=
Φ i ( x, y ) , we can set the stochastic gradient according to ˜ ∇ x Φ( x, y ; ω ) =
∇ x Φ ω ( x, y ) and ˜ ∇ y Φ( x, y ; ω ) =
∇ y Φ ω
( x, y ) where ω is uniformly drawn at random from { 1 , . . . , n } .
Therefore, if batch samples I xk , I yk are drawn from { 1 , . . . , n } uniformly at random with replacement; batches will be independent of the past iterates satisfying Assumption 7.
Theorem 4.
Suppose Assumptions 1,3,6 and 7 hold.
Moreover, either Assumption 4 or Assumption 5 holds.
Let µ x = γ , θ = 1 , and τ , σ , b and N be chosen as follows:
[Equation]
For any (cid:15) > 0 and parameters b x , b y , q ∈ N + , when VR-flag = true , SAPD+ guarantees (cid:15) -stationary, min t =0 ,...,T E [ (cid:107)∇ φ λ ( x t 0 ) (cid:107) ] ≤ (cid:15) , for T ≥ 288 G ( x 00 , y 00 ) · γ(cid:15) 2 , which requires T ( Nb/q + N ( b x + b y )) stochastic ﬁrst-order oracle calls in total, where
[Equation]
Proof.
See appendix E for the proof.
Remark 10.
For any y ∈ dom g , since Φ( · , y )
L xx -smooth, it is necessarily L xx -weakly convex; hence, γ ≤ L xx .
To get a worst-case complexity, consider the setting in (8) , and let b x = b y = b (cid:48) .
Then, Theorem 4 implies that setting b =
O (cid:16) κ y δ 2 (cid:15) 2 (cid:17) , N = O (cid:0) κ y + κ 2 y qb (cid:48) (cid:1) , and T =
O (cid:16) L G ( x 00 ,y 00 )
(cid:15)(cid:17) leads to Nb/q + Nb (cid:48)
=
O (cid:16) κ y bq + κ 2 y bb (cid:48)
+ κ y b (cid:48)
+ κ 2 y q (cid:17) .
Thus, setting q = (cid:113) bκ y and b (cid:48)
= (cid:112) bκ y leads to the oracle complexity of T ( Nb/q + Nb (cid:48) )
=
O (cid:16) κ 2 y δ(cid:15) · L G ( x 00 ,y 00 ) (cid:15)(cid:17) .
Weakly convex-merely concave (WCMC) problems
In this section, we state the convergence guarantees of SAPD+ for solving WCMC problems.
In particular, we will consider (1) such that f ( · ) = 0 and µ y = 0 , i.e., Φ( x, · ) is merely concave for all x ∈ X .
Instead of directly solving (1) in WCMC setting, we will solve an approximate model obtained by smoothing the primal problem in a similar spirit to the technique in [31].
More precisely, we approximate (1) with the following WCSC problem: given an arbitrary ˆ y ∈ dom g , consider
[Equation]
where ˆΦ( x, y [Equation] [Equation]
Theorem 5.
Under Assumptions 1, 2, 3, consider (1) such that f ( · ) ≡ 0 , µ y = 0 , and D Y (cid:44) sup y 1 ,y 2 ∈ dom g (cid:107) y 1 − y 2 (cid:107) < ∞ .
When either Assumption 4 or Assumption 5 holds, for any given (cid:15) > 0 , SAPD+ with VR-flag = false , applied to (13) with ˆ µ y =
Θ( (cid:15) 2 / ( L D 2 y )) , is guaranteed to generate x (cid:15) ∈
X such that E
[ (cid:107)∇ φ λ ( x (cid:15) ) (cid:107) ] ≤ (cid:15) for λ
= 1 / (2 γ ) within O ( L 3 (cid:15) − 6 ) stochastic ﬁrst-order oracle calls.
Proof.
See appendix F for the proof.
Numerical experiments
The experiments are conducted on a PC with 3.6 GHz Intel Core i7 CPU and NVIDIA RTX
GPU.
We consider distributionally robust optimization and fair classiﬁcation.
In the rest, n and d represent the number of samples in the dataset and the dimension of each data point, respectively.
In this section, SAPD+ means calling SAPD+ with VR-flag = false , and SAPD+VR means calling SAPD+ with VR-flag = true .
Distributionally Robust Optimization (DRO).
First, we consider nonconvex-regularized variant of DRO problem [1, 29, 20, 27, 44, 42] which arises in distributionally robust learning.
Let { a i , b i } ni = be the dataset where a i ∈ R d are the features and b i ∈ {− 1 , 1 } are labels.
The DRO problem is
[Equation]
where (cid:96) i ( x ) = log(
+ exp( − b i a (cid:62) i x )) is the logistic loss, f ( x ) =
η 1 (cid:80) di
=1 αx 2 i 1+ αx 2 i is a nonconvex regularizer [2], g ( y ) =
12 η 2 (cid:107) ny − 1 (cid:107) 2 , and Y (cid:44) { y ∈ R d
+ : 1 (cid:62) y
= 1 } – here, 1 denotes the vector with all entries equal to one.
This problem can be viewed as a robust formulation of empirical risk minimization where the weights y
i are allowed to deviate from 1 /n ; and the aim is to minimize the worst-case empirical risk.
We perform experiments on three data sets: i ) a9a with n = 32561 , d = 123 ; ii ) gisette with n = 6000 , d = 5000 ; iii ) sido0 with n = 12678 , d = 4932 .
The dataset sido0 is obtained from Causality Workbench 3 while the others can be downloaded from
LIBSVM repository 4 .
Parameter tuning.
We set the parameters according to [42, 27, 20], i.e., , α = 10 , η 1 = 10 − 3 , η 2 = 1 /n 2 .
We compare SAPD+ and SAPD+VR against PASGDA [4], SREDA [27], SMDA , SMDA-VR [17] algorithms.
As suggested in [27], we tune the primal stepsizes of all the algorithms based on a grid-search over the set { 10 − 3 , 10 − 2 , 10 − 1 } and the ratio of the primal stepsize to dual stepsize, i.e., τ/σ , is varied to take values in the set { 10 , 10 2 , 10 3 , 10 4 } .
For all variance reduction-based algorithms, i.e., for SAPD+VR , SREDA , SMDA-VR , we tune the large batch size b (cid:44) |B| from the set { 3000 , 6000 } , and the small batch size b (cid:48) (cid:44) | I | from grid search over the set { 10 , 100 , 200 } .
For the frequency parameter q , we let q = b (cid:48)
=
| I | for SAPD+VR and SMDA-VR (as suggested in [17]); for
SREDA , when we set q and m ( SREDA ’s inner loop iteration number) to O ( n/ | I | ) as suggested in [27], we noticed that SREDA does not perform well against SAPD+VR and SMDA-VR .
Therefore, to optimize the performance of SREDA further, we tune q, m from a grid search over { 10 , 100 , 200 } .
For methods without variance reduction, i.e., for SAPD+ , SMDA and PASGDA , we also use mini-batch to estimate the gradients and tune the batch size from { 10 , 100 , 200 } as well.
For SAPD+ and SAPD+VR , we tune the momentum θ from { 0 . 8 , 0 . 85 , 0 . 9 } and the inner iteration number from N = { 10 , 50 , 100 } .
Results.
To fairly compare the performances of algorithms using different batch sizes, we plot loss against epochs in x-axis 5 .
In ﬁg. 1, we plot the average loss against the epoch number based on 30 simulations (runs).
The standard deviations of the runs are also illustrated around the average in lighter color as shaded regions.
We observe that SAPD+ and SAPD+VR consistently outperforms over other algorithms.
For a9a , gisette , sido0 datasets, the average training accuracy of SAPD+ are 84 .
43% , and of SAPD+VR are 84 .
46% , respectively.
The best performance for a9a , gisette , sido0 among all the other algorithms are 75 .
43% , respectively.
More importantly, we observe that as an accelerated method, SAPD+VR enjoys fast convergence properties while still being robust to gradient noise.
Fair Classiﬁcation.
In the context of multi-class classiﬁcation, Mohri et al.
[28] propose training a fair classiﬁer thorough minimizing the worst-case loss over the classiﬁcation categories.
In the spirit of [34, 17], we adopt a nonconvex convolutional neural network (CNN) model as a classiﬁer and set the number of categories to 3 , resulting in a minimax problem of the form:
[Equation]
where x ∈ R p represents the parameters of the CNN, and (cid:96) 1 , (cid:96) 2 , (cid:96) 3 correspond to the loss of three categories, g ( y )
=
η 2 (cid:107) y (cid:107) 2 2 is a regularizer with η > 0 .
We train (15) on the datasets to classify: i ) gray- scale hand-written digits { 0 , 2 , 3 } from MNIST ; ii ) fashion images with target classes {T-shirt/top,
Sandal, Ankle boot} from F-MNIST ; iii ) RBG colored images with target classes {Plane, Truck,
Deer} from CIFAR10 .
For both MNIST and F-MNIST p = 43831 , n = 18000 and d = 28 × 28 × 1 , and for CIFAR10 p = 61411 , n = 15000 , and d = 32 × 32 × 3 .
We let the regularization parameter η = 0 .
1 as suggested in [17].
We compare SAPD+VR against the other VR-based algorithms SREDA and SMDA-VR over 30 runs.
We tune the primal stepsizes of
SAPD+VR and SREDA by a grid search over the set { 10 − 2 , 5 × 10 − 3 , 10 − 3 } and the ratio of primal to dual stepsizes, i.e., τ/σ , is chosen from { 10 , 10 2 , 5 × 10 2 , 10 3 } .
For SMDA-VR , the primal and dual stepsizes are 10 − 3 and 10 − 5 as suggested in [17] –we also tried stepsizes bigger than the suggested; but, it caused convergence issues in the experiments.
We set the large batchsize |B| = 3000 and the small batchsize |I| = 200 for all algorithms and data sets; the frequency q = 200 is used for SAPD+VR and SMDA-VR , and we tune q for SREDA taking values from { 10 , 50 , 100 , 200 } .
The momentum θ for SAPD+VR is tuned taking values from { 0 . 8 , 0 . 85 , 0 . 9 } and inner iteration number is tuned from
N = { 10 , 50 , 100 } .
For SREDA , we tune the inner loop iteration from { 10 , 50 , 100 } .
Fig.
2 shows that SAPD+VR outperforms the other VR-based algorithms clearly in terms of both the average loss and the standard deviation of the loss.
Conclusion
In this paper, we considered both WCSC and WCMC saddle-point problems assuming we only have an access to an unbiased stochastic ﬁrst-oracle with a ﬁnite variance.
This setting arises in many applications ranging from distributionally robust learning to GANs.
We proposed a new method
SAPD+ , which achieves an improved complexity in terms of target accuracy (cid:15) for both WCSC and
WCMC problems; moreover, our bound for SAPD+ has a better dependency to the condition number κ y for the WCSC scenario.
We also showed that our algorithm SAPD+ can support the SPIDER variance-reduction technique.
Finally, we provided numerical experiments demonstrating that SAPD+ can achieve a state-of-the-art performance on distributionally robust learning and on multi-class classiﬁcation problems arising in ML.