Nest Your Adaptive Algorithm for Parameter-Agnostic Nonconvex Minimax Optimization
Abstract
Adaptive algorithms like AdaGrad and AMSGrad are successful in nonconvex optimization owing to their parameter-agnostic ability – requiring no a priori knowledge about problem-speciﬁc parameters nor tuning of learning rates.
However, when it comes to nonconvex minimax optimization, direct extensions of such adaptive optimizers without proper time-scale separation may fail to work in practice.
We provide such an example proving that the simple combination of Gradient Descent Ascent (GDA) with adaptive stepsizes can diverge if the primal-dual stepsize ratio is not carefully chosen; hence, a fortiori, such adaptive extensions are not parameter-agnostic.
To address the issue, we formally introduce a Nested Adaptive framework, NeAda for short, that carries an inner loop for adaptively maximizing the dual variable with controllable stopping criteria and an outer loop for adaptively minimizing the primal variable.
Such mechanism can be equipped with off-the-shelf adaptive optimizers and automatically balance the progress in the primal and dual variables.
Theoretically, for nonconvex-stronglyconcave minimax problems, we show that NeAda with AdaGrad stepsizes can achieve the near-optimal e
O ( ✏  2 ) and e
O ( ✏  4 ) gradient complexities respectively in the deterministic and stochastic settings, without prior information on the problem’s smoothness and strong concavity parameters.
To the best of our knowledge, this is the ﬁrst algorithm that simultaneously achieves near-optimal convergence rates and parameter-agnostic adaptation in the nonconvex minimax setting.
Numerically, we further illustrate the robustness of the NeAda family with experiments on simple test functions and a real-world application.
Introduction
Adaptive gradient methods, whose stepsizes and search directions are adjusted based on past gradients, have received phenomenal popularity and are proven successful in a variety of large-scale machine learning applications.
Prominent examples include AdaGrad [17], RMSProp [31], AdaDelta [84],
Adam [41], and AMSGrad [69], just to name a few.
Their empirical success is especially pronounced for nonconvex optimization such as training deep neural networks.
Besides improved performance, being parameter-agnostic is another important trait of adaptive methods.
Unlike (stochastic) gradient descent, adaptive methods often do not require a priori knowledge about problem-speciﬁc parameters
(such as Lipschitz constants, smoothness, etc.).
On the theoretical front, some adaptive methods can achieve nearly the same convergence guarantees as (stochastic) gradient descent [17, 79, 69].
Recently, adaptive methods have sprung up for minimax optimization:
[Equation]
where f is l -Lipschitz smooth jointly in x and y , Y is closed and convex, and ⇠ is a random vector.
Such problems have found numerous applications in generative adversarial networks (GANs) [25, 4],
Wasserstein GANs [4], generative adversarial imitation learning [32], reinforcement learning [14, 60], adversarial training [74], domain-adversarial training of neural networks [21] , etc.
A common practice is to simply combine adaptive stepsizes with popular minimax optimization algorithms such as Gradient Descent Ascent (GDA), extragradient method (EG) and the like; see e.g., [23, 27, 24].
It is worth noting that these methods are reported successful in some applications yet at other times can suffer from training instability.
In recent years, theoretical behaviors of such adaptive methods are extensively studied for convex-concave minimax optimization; see e.g., [6, 3, 1, 18, 72, 22, 58, 15].
However, for minimax optimization in the important nonconvex regime, little theory related to adaptive methods is known.
Unlike the convex-concave setting, a key challenge for nonconvex minimax optimization lies in the necessity of a problem-speciﬁc time-scale separation of the learning rates between the min-player and max-player when GDA or EG methods are applied, as proven in [82, 50, 70, 8].
This makes the design of adaptive methods fundamentally different from and more challenging than nonconvex minimization.
Several recent attempts [28, 33, 34] studied adaptive methods for nonconvex-stronglyconcave minimax problems; yet, they all require explicit knowledge of the problems’ smoothness and strong concavity parameters to maintain a stepsize ratio proportional to the condition number.
Such a requirement evidently undermines the parameter-agnostic trait of adaptive methods.
This then raises a couple of interesting questions: (1) Without a problem-dependent stepsize ratio, does simple combination of GDA and adaptive stepsizes still converge?
(2) Can we design an adaptive algorithm for nonconvex minimax optimization that is truly parameter-agnostic and provably convergent?
In this paper, we address these questions and make the following key contributions:
• We investigate two generic frameworks for adaptive minimax optimization: one is a simple (non-nested) adaptive framework, which performs one step of update of x and y simultaneously with adaptive gradients; the other is Nested Adaptive (NeAda) framework, which performs multiple updates of y after one update of x , each with adaptive gradients.
Both frameworks allow ﬂexible choices of adaptive mechanisms such as Adam, AMSGrad and AdaGrad.
We provide an example proving that the simple adaptive framework can fail to converge without setting an appropriate stepsize ratio; this applies to any of the adaptive mechanisms mentioned above, even in the noiseless setting.
In contrast, the NeAda framework is less sensitive to the stepsize ratio, as numerically illustrated in Figure 1.
• We provide the convergence analysis for a representative of NeAda that uses AdaGrad stepsizes for x and a convergent adaptive optimizer for y , in terms of nonconvex-stronglyconcave minimax problems.
Notably, the convergence of this general scheme does not require to know any problem parameters and does not assume the bounded gradients.
We demonstrate that NeAda is able to achieve e O ( ✏  2 ) oracle complexity for the deterministic setting and e O ( ✏  4 ) for the stochastic setting to converge to ✏ -stationary point, matching best known bounds.
To the best of our knowledge, this seems to be the ﬁrst adaptive framework for nonconvex minimax optimization that is provably convergent and parameter-agnostic.
• We further make two complementary contributions, which can be of independent interest.
First, we propose a general AdaGrad-type stepsize for strongly-convex problems without knowing the strong convexity parameters, and derive a convergence rate comparable to SGD.
It can
serve as a subroutine for NeAda.
Second, we provide a high probability convergence result for the primal variable of NeAda under a subGaussian assumption.
[Figure]
• Finally, we numerically validate the robustness of the NeAda framework on several test functions compared to the non-nested adaptive framework, and demonstrate the effectiveness of the NeAda framework on distributionally robust optimization task with a real dataset.
1.1 Related work Adaptive algorithms.
Duchi et al.
[17] and Streeter and McMahan [73] introduce AdaGrad for convex online learning and achieve O ( p T ) regrets.
Li and Orabona [48] and Ward et al.
[79] show an e
O ( ✏  4 ) complexity for AdaGrad in the nonconvex stochastic optimization.
There are an extensive number of works on AdaGrad-type methods; to list a few, [59, 45, 2, 39, 65].
Another family of algorithms uses more aggressive stepsizes of exponential moving average of the past gradients, such as Adam [41] and RMSProp [31].
Reddi et al.
[69] point out the non-convergence of Adam and provide a remedy with non-increasing stepsizes.
There is a surge in the study of Adamtype algorithms due to their popularity in the deep neural network training [83, 11, 51].
Some work provides the convergence results for adaptive methods in the strongly-convex optimization [78, 44, 62].
Line search and stochastic line search are another effective strategy that can detect the objective’s curvature and have received much attention [75, 77, 76].
Notably, many adaptive algorithms are parameter-agnostic [17, 69, 79].
Nonconvex minimax optimization.
Stationary convergence of GDA in NC-SC setting was ﬁrst provided by Lin et al.
[50], showing O ( ✏  2 ) oracle complexity and O ( ✏  4 ) sample complexity with minibatch.
Recently, Chen et al.
[10] and Yang et al.
[82] achieve this sample complexity in the stochastic setting without minibatch.
GDmax is a double loop algorithm that maximizes the dual variable to a certain accuracy.
It achieves nearly the same complexity as GDA [64].
Sebbouh et al.
[70] recently discuss the relation between the two-time-scale and number of inner steps for GDmax.
Very recently, Li et al.
[47] show that time-scale separation is necessary for GDA to converge to
Stackelberg equilibrium.
Besides NC-SC setting, some work provides convergent algorithms when the objective is (non-strongly) concave about the dual variable[85, 55, 81].
Nonconvex-nonconcave regime is only explored under some special structure [53, 15], such as Polyak- Ł ojasiewicz (PL) condition [19].
All algorithms mentioned above require prior knowledge about problem parameters, such as smoothness modulus, strong concavity modulus, and noise variance.
Adaptive algorithms in minimax optimization.
There exist many adaptive and parameteragnostic methods designed for convex-concave minimax optimization as a special case of monotone variational inequality [6, 3, 1, 18, 72, 22, 58, 15].
Most of them combine extragradient method, mirror prox [63] or the like, with AdaGrad mechanism.
Liu et al.
[52] and Dou and Li [16] relax convexityconcavity assumption to the regime where Minty variational inequality (MVI) has a solution.
In these settings, time-scale separation of learning rates is not required even for non-adaptive algorithms.
For nonconvex-strongly-concave problems, Huang and Huang [33], Huang et al.
[34], Guo et al.
[28] propose adaptive methods, which set the learning rates based on knowledge about smoothness and strong-concavity modulus and the bounds for adaptive stepsizes.
In this section, we investigate two generic frameworks that can incorporate most existing adaptive methods into minimax optimization.
We remark that many variants encapsulated in these two families are already widely used in practice, such as training of GAN [24], distributionally robust optimization [71], etc.
These two frameworks, coined as non-nested and nested adaptive methods, can be viewed as adaptive counterparts of GDA and GDmax.
We aim to illustrate the difference between these two adaptive families, even though GDA and GDmax are often considered “twins”.
Non-nested adaptive methods.
In Algorithm 1, non-nested methods update the primal and dual variables in a symmetric way.
Weighted gradients m xt and m yt are the moving average of the past stochastic gradients with the momentum parameters  x and  y .
The effective stepsizes of x and y are ⌘ x / p v xt and ⌘ y / p v yt , where the division is taken coordinate-wise.
We refer to ⌘ x and ⌘ y as learning rates, and v xt , v yt are some average of squared-past gradients through function .
Many popular choices of adaptive stepsizes are captured in this framework, see also [69]: (GDA)  =
[Equation]
sample ⇠ t and let g xt = r x F ( x t , y t ; ⇠ t ) and g yt = r y F ( x t , y t ; ⇠ t ) 4: // update the first moment m xt
+
=
x m xt + (1   x ) g xt and m yt [-+
-]+1 =
y m yt
+ (1   y )
g yt 5: // update the second moment v xt +
=  v x 0 , { ( g xi ) 2 } ti =  and v yt [-+
-]+1 =
v y 0 , { ( g yi ) 2 } ti =  6: // update variables x t [-+
-]+1 = x
t  ⌘ x p v xt +1 m xt {++
+}
and y t +1 =
y t + ⌘ y p v y t
+1 m yt + 7: end for
1: Input: x 0 and y2: for t = 0 , 1 , 2 , ... do 3: for k = 0 , 1 , 2 , ... until a stopping criterion is [-satisfied do 4: sample ˆ ξ kt and g yt,k = ∇ y F ( x t , y kt ; ˆ ξ kt ) 5: m yt,k +
-]satisﬁed
do 4: sample ˆ ⇠ kt and g yt,k = r y F ( x t , y kt ; ˆ ⇠ kt ) 5: m yt, k +1 =
y m yt,k
+ (1   y ) g yt,k 6: v yt,k +1 =
y  v yt, 0 , { ( g yt,i ) 2 } ki =  7: y k +1 t = y kt + ⌘ y p v yt,
k +1 m yt,k + 8: end for 9: v yt +1 ,
= v yt,k +
and m yt +1 ,
= m yt,k + 10: sample ⇠ t and g xt = r x F ( x t , y k +1 t ; ⇠ t ) 11: m xt +1 =  x m xt + (1   x
[Equation]
[Equation] [Equation]
14: end for
Nested adaptive (NeAda) methods.
NeAda, presented in Algorithm 2, has a nesting inner loop to maximize y until some stopping criterion is reached (see details in Section 3).
Instead of using a ﬁxed number of inner iterations or a ﬁxed target accuracy as in GDmax [50, 64], NeAda gradually increases the accuracy of the inner loop as the outer loop proceeds to make it fully adaptive.
We refer to the ratio between two learning rates, i.e. ⌘ y / ⌘ x , as the two-time-scale.
The current analysis of GDA in nonconvex-strongly-concave setting requires two-time-scale to be proportional with the condition number  = l/µ , where l and µ are Lipschitz smoothness and strongly-concavity modulus [50, 82].
We provide an example showing that the problem-dependent two-time-scale is necessary for GDA and most non-nested methods even in the deterministic setting.
Lemma 2.1. Consider the function f
( x, y ) =  12 y {+
+} + Lxy  L 2 2 x 2 in the deterministic setting.
Let r ⌘ x = ⌘ y .
(1) GDA will not converge to the stationary point when r  L 2 :
[Equation]
(2) Assume the averaging function x and y are the same, and satisfy that for any ⌧ , if v xt = ⌧ v yt and ( g xt ) [-
-]2 = ⌧ ( g yt ) 2 then v xt +
= ⌧ v yt
+1 .
With  x =  y , v x 0 = v y 0 =
and m x 0 = m y
= 0 (which are commonly used in practice), non-nested adaptive method will not converge when r  L :
[Equation]
When r = L , r x f ( x t , y t ) = r x f ( x 0 , y 0 ) for all t .
Remark 1.
Most popular adaptive stepsizes we mentioned before, such as Adam, AMSGrad and
AdaGrad, have averaging functions satisfying the assumption in the lemma.
Any point on the line y
=
Lx is a stationary point for the above function, and the distance from a point to this line is proportional to its gradient norm, so the divergence in gradient norm will also implies that of iterates.
In the proof, we will also show that the averaged or best iterate will still diverge under the same condition.
The lemma implies that for any given time-scale r , there exists a problem for which the non-nested algorithm does not converge to the stationary point, so they are not parameter-agnostic.
We compare non-nested and nested methods combined with different stepsizes schemes: Adam,
AMSGrad, AdaGrad and ﬁxed stepsize, on the function:  12 y 2 + 2 xy  2 x 2 .
In the experiments of this section, we halt the inner loop when the (stochastic) gradient about y is smaller than 1 /t or the number iteration is greater than t .
We observe from Figure 1 that the thresholds for the non-convergence of non-nested methods ( r = 2 for adaptive methods and r = 4 for GDA) are exactly as predicted by the lemma.
Although the adaptive methods admit a smaller two-time-scale threshold than GDA in this example, it is not a universal phenomenon from our experiments in Section 4.
Interestingly, nested adaptive methods are robust to different two-time-scales and always have the trend to converge to the stationary point.
Convergence Analysis of NeAda-AdaGrad
In this section, we reveal the secret behind the robust performance of NeAda by providing the convergence guarantee for a representative member in the family.
For sake of simplicity and clarity, we mainly focus on NeAda with AdaGrad.
Adam-type mechanism can suffer from non-convergence already for nonconvex minimization despite its good performance in practice.
Our result also sheds light on the analysis of other more sophisticated members such as AMSGrad in the family.
NeAda-AdaGrad: Presented in Algorithm 3, NeAda-AdaGrad adopts the scalar AdaGrad scheme [73] for the x -update in the outer loop and uses mini-batch in the stochastic setting.
For the inner loop for maximizing y , we run some adaptive algorithm for maximizing y until some easily checkable stopping criterion is satisﬁed.
We suggest two criteria here: at t -th outer loop: (I) the squard gradient mapping norm about y is smaller than 1 / ( t + 1) in the deterministic setting, (II) the number of inner loop iterations reaches t + 1 in the stochastic setting.
[Table]
v do y t  an adaptive algorithm A for maximizing f ( x t , · ) to obtain y t
a Proj t + r y t ,
t )) + b) criterion II (stochastic):
t + 1 inner loop iterations. [Equation]
[Equation] [Equation]
where are samples
[Equation]
⌘ 6: end for
For the purpose of theoretical analysis, we mainly focus on the minimax problem of the form (1) under the nonconvex-strongly-concave (NC-SC) setting 3 , formally stated in the following assumptions.
Assumption 3.1 (Lipschitz smoothness) .
There exists a positive constant l > 0 such that max
[Equation]
holds for all x 1 , x 2 2 R d , y 1 , y 2 2 Y .
Assumption 3.2 (Strong-concavity in y ) .
There exists µ > 0 such that: f ( x, y 1 )  f ( x, y 2 ) + hr y f ( x, y 1 ) , y 1  y 2 i + µ 2 k y 1  y 2 k [Equation]
[Equation] 2 R d , y 1 , y 2 2 Y .
For simplicity of notation, deﬁne  = l/µ as the condition number,  ( x ) = max y 2 Y f ( x, y )
as the primal function, and y ⇤ ( x ) =
arg max Y f
( x, y ) as the optimal y w.r.t x .
Since the objective is nonconvex about x , we aim at ﬁnding an ✏ -stationary point ( x t , y t )
such that E
kr x f ( x t , y t ) k  ✏ and E k y t  y ⇤ ( x t ) k  ✏ , where the expectation is taken over the randomness in the algorithm.
3.1 Convergence in deterministic and stochastic settings
Assumption 3.3 (Stochastic gradients) .
r x F ( x, y ; ⇠ ) and r y F ( x, y ; ⇠ ) are unbiased stochastic estimators of r x f ( x, y ) and r y f ( x, y ) and have variances bounded by  2  0 .
We assume the unbiased stochastic gradients have the variance  2 , and the problem reduces to the deterministic setting when  = 0 .
Now we provide a general analysis of the convergence for any adaptive optimizer used in the inner loop.
Theorem 3.1.
Deﬁne the expected cumulative suboptimality of inner loops as E =
E hP
[Equation]
i .
Under Assumptions 3.1, 3.2 and 3.3, the output from Algorithm 3 satisﬁes
[Equation]
where A = 2  ⌘ +
[Equation]
Remark 2.
The general analysis is built upon milder assumptions than existing work on AdaGrad in nonconvex optimization, not requiring either bounded gradient in [79] or prior knowledge about the smoothness modulus in [48].
This theorem implies the algorithm attains convergence for the nonconvex variable x with any constant ⌘ > 0 and v 0 > 0 that does not depend on any problem parameter, so it is parameter-agnostic.
Remark 3.
Another beneﬁt of this analysis is that the variance  appears in the leading term T  14 , which means the convergence rate can interpolate between the deterministic and stochastic settings.
It implies a complexity of e
O ( ✏  2 ) in the deterministic setting and e
O ( ✏  4 ) in the stochastic setting for the primal variable as long as the accumulated suboptimality for the inner-loops E is e O (1) , regardless of the batch size M .
However, M can control the number of outer loops and there affect the sample complexity for the dual variable.
In the next two theorems, we derive the total complexities, in the deterministic and stochastic settings, of ﬁnding ✏ -stationary point by controlling the cumulative suboptimality E in Theorem 3.1 for subroutine A with speciﬁc convergence rate.
In fact, we can also use any off-the-shelf adaptive optimizer for solving the inner maximization problem up to the desired accuracy.
Note that (stochastic)
GDmax ﬁxes each inner-loop’s accuracy or steps to be related with µ , ` and ✏ so that E can be easily bounded [50, 64].
In contrast, since we do not have access to the problem parameters and ✏ , Algorithm 3 gradually increases the inner-loop accuracy.
In the proof of the following theorems, we will show that with our proposed stopping criteria and desired subroutines, E is bounded by O (log T ) .
Theorem 3.2 (deterministic) .
Suppose we have a linearly-convergent subroutine A for maximizing any strongly concave function h ( · ) :
[Equation]
where y k is k -th iterate, y ⇤ is the optimal solution, and a 1 > 0 and 0 < a 2 < 1 are constants that can depend on the parameters of h .
Under the same setting as Theorem 3.1 with  = 0 , for
Algorithm 3 with M = 1 and a subroutine A under stopping criterion I, there exists t ⇤  e O  ✏  such that ( x t ⇤ , y t ⇤ ) is an ✏ -stationary point.
Therefore, the total gradient complexity is e O  ✏  .
Remark 4.
This complexity is optimal in ✏ up to logarithmic term [86], similar to GDA [50].
Note that many adaptive and parameter-agnostic algorithms can achieve the linear rate when solving smooth and strongly concave maximization problems; to list a few, gradient ascent with backtracking line-search [75], SC-AdaNGD [44] and polyak stepsize [30, 54, 66] 4 .
Here we can also pick more general subproblem accuracy in criterion I that only needs to scale with 1 /t .
Theorem 3.3 (stochastic) .
Suppose we have a sub-linearly-convergent subroutine A for maximizing any strongly concave function h ( · ) : after K = k + 1 iterations
[Equation]
where y k is k -th iterate, y ⇤ is the optimal solution, and b 1 , b 2 > 0 are constants that can depend on the parameters of h .
Under the same setting as Theorem 3.1, for Algorithm 3 with M = ✏  2 and subroutine A under the stopping criterion II, there exists t ⇤  e O  [Equation]  such that ( x t ⇤ , y t ⇤ ) is an ✏ -stationary point.
Therefore, the total stochastic gradient complexity is e O  [Equation]  .
Remark 5.
This e O [Equation]  complexity is nearly optimal in the dependence of ✏ for stochastic NC-SC problems [46].
Here we set M = ✏  2 for the simplicity of exposition, and a similar result also holds for gradually increasing M .
The sublinear rate speciﬁed above for solving the stochastic strongly convex subproblem can be achieved by several existing parameter-agnostic algorithms under some additional assumptions, such as F REE R EX M OMENTUM [12] and Coin-Betting [13] 5 .
Parameter-free
SGD [9] is partially parameter-agnostic that only requires the stochastic gradient bound rather than the strongly-convexity parameter.
Mukkamala and Hein [62] and Wang et al.
[78] introduce the variants of AdaGrad, RMSProp and Adam for strongly-convex online learning, but they need to know both gradient bounds and strongly-convexity parameter for setting stepsizes.
We will show in the next subsection that AdaGrad with a slower decaying rate is parameter-agnostic.
We note that the analysis of this theorem is not the simple gluing of the outer loop and inner loop complexity, but requires more sophisticated control of the cumulative suboptimality E .
With the popularity of computational resource demanding deep neural networks, in both minimization and minimax applications, people ﬁnd high probability guarantees for a single run of an algorithm useful [40, 49].
Given the lack of such guarantee in the minimax optimization, we provide a high probability convergence result for NeAda-AdaGrad in Appendix C, which shows a similar sample complexity as Theorem 3.3 under the subGaussian noise.
3.2 Generalized AdaGrad for strongly-convex subproblem
We now introduce the generalized AdaGrad for minimizing strongly convex objectives, which can serve as an adaptive subroutine for Algorithm 3, without requiring knowledge on the strongly convex parameter.
We analyze it for the more general online convex optimization setting: at each round t , the learner updates its decision x t , then it suffers a loss f t ( x t ) and receives the sub-gradient of f t .
The generalized AdaGrad, described in Algorithm 4, keeps the cumulative gradient norm v t and takes the stepsize ⌘ /v ↵ t with a decaying rate ↵ 2 (0 , 1] .
When ↵ = 1 / 2 , it reduces to the scalar version of AdaGrad [73]; when ↵ = 1 , it reduced to the scalar version of SC-AdaGrad [62].
Theorem 3.4.
Consider Algorithm 4 for online convex optimization and assume that (i) f t is continuous and µ -strongly convex, (ii) X is convex and compact with diameter D ; (iii) k g t k  G for every t .
Then for 0 < ↵ < 1 with any ⌘ > 0 , the regret of Algorithm 4 satisﬁes:
[Equation]
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure] max x 2 X
T  [Equation] where c ↵ and d ↵ are constants depending on the problem parameters, ↵ and ⌘ .
The theorem implies a logarithmic regret for the case ↵ = 1 , but the stepsize needs knowledge about problem’s parameters µ and G ; similar results are shown for SC-AdaGrad [62] and SAdam [78].
When ↵ < 1 , the algorithm becomes parameter-agnostic and attains an O ( T 1  ↵ ) regret.
Such parameter-agnostic phenomenon for smaller decaying rates is also observed for SGD in stochastic optimization [20].
Proving the regret bound for the generalized AdaGrad with ↵ < 1 in the online setting is challenging, since the adversarial g t can lead to a “sudden” change in the stepsize.
In the proof, we bound the possible number of times such “sudden” change could happen.
To the best of our knowledge, this is the ﬁrst regret bound for adaptive methods with general decaying rates in the strongly convex setting.
By online-to-batch conversion [38], it can be converted to
O ( T  ↵ ) rate in the strongly convex stochastic optimization.
Xie et al.
[80] prove the O (1 /T ) convergence rate, or a linear convergence rate when the smoothness parameter is known, for AdaGrad with ↵ = 1 / 2 in this setting, but under a strong assumption — Restricted Uniform Inequality of
Gradients (RUIG) — that requires the loss function with respect to each sample ⇠ to satisfy the error bound condition with some probability.
Experiments
To evaluate the performance of NeAda, we conducted experiments on simple test functions and a real-world application of distributional robustness optimization (DRO).
In all cases, we compare
NeAda with the non-nested adaptive methods using the same adaptive schemes.
For notational simplicity, in all ﬁgure legends, we label the non-nested methods with the names of the adaptive mechanisms used.
We observe from all our experiments that: 1) while non-nested adaptive methods can diverge without the proper two-time-scale, NeAda with adaptive subroutine always converges; 2) when the non-nested method converges, NeAda can achieve comparable or even better performance.
[Figure]
4.1 Test functions
In Section
[Figure]
we have compared NeAda with non-nested methods on a quadratic function in Figureand the observations match Lemma 2.1.
Now we consider a more complicated function that is composed of McCormick function in x , a bilinear term, and a quadratic term in y ,
[Equation]
For this function, we compare the adaptive frameworks in the stochastic setting with Gaussian noise.
As demonstrated in Figure
[Figure]
non-nested methods are sensitive to the selection of the two-time-scale.
When the learning rate ratio is too small, e.g., ⌘ y / ⌘ x = 0 .
01 , non-nested Adam, AMSGrad and GDA all fail to converge.
We observe that GDA converges when the ratio reaches 0.03, while non-nested
Adam and AMSGrad still diverge until 0.05.
Although non-nested adaptive methods require a smaller ratio than GDA in Lemma 2.1, this example illustrates that adaptive algorithms sometimes can be more sensitive to the time separation.
In comparison, NeAda with adaptive subroutine always converges regardless of the learning rate ratio.
4.2 Distributional robustness optimization
To justify the effectiveness of NeAda on real-world applications, we carried out experiments on distributionally robust optimization [71], where the primal variable is the model weights to be learned by minimizing the empirical loss while the dual variable is the adversarial perturbed inputs.
The dual variable problem targets ﬁnding perturbations that maximize the empirical loss but not far away from the original inputs.
Formally, for model weights x and adversarial samples y , we have:
[Equation]
where n is the total number of training samples, v i is the i -th original input and f i is the loss function for the i -th sample.
is a trade-off parameter between the empirical loss and the magnitude of perturbations.
When  is large enough, this problem is nonconvex-strongly-concave, and following the same setting as [71, 70], we set  = 1 . 3 .
For NeAda, we use both stopping criterion I with stochastic gradient and criterion II in our experiments.
For the results, we report the training loss and the test accuracy on adversarial samples generated from fast gradient sign method (FGSM) [26].
FGSM can be formulated as x adv = x + ✏ · sign ( r x f ( x )) , where ✏ is the noise level.
To get reasonable test accuracy, NeAda with Adam as subroutine is compared with Adam with ﬁxedinner loop iterations, which is consistent with the choice of inner loop steps in [71], and such choice obtains much better test accuracy than the completely non-nested Adam.
Our experiments include a synthetic dataset and MNIST [43] with code modiﬁed from [56].
Results on Synthetic Dataset.
We use the same data generation process as in [71].
The inputs are 2-dimensional i.i.d.
random Guassian vectors, i.e., x i ⇠ N (0 , I 2 ) , where I 2 is the 2 ⇥ 2 identity matrix.
The corresponding y
i is deﬁned as y i = sign ( k x i k 2  p 2) .
Data points with norm in range
[Figure]
As shown in Figure 3(a), when the learning rates are set to different scales, i.e., ⌘ x = 0 . 01 , ⌘ y =
.(red curves in the ﬁgure), both methods achieve reasonable test errors.
In this case, NeAda has higher test accuracy and reaches such accuracy faster than Adam.
If we change the learning rates to the same scale, i.e., ⌘ x = 0 . 01 , ⌘ y = 0 .
01 (blue curves in the ﬁgure), NeAda retains good accuracy while
Adam drops to an unsatisfactory performance.
This demonstrates the adaptivity and less-sensitivity to learning rates of NeAda.
In addition, Figure 3(b) illustrates the convergence speeds on the loss function, and NeAda (solid lines) always decreases the loss faster than Adam.
Note that Adam with the same learning rates converges to a lower loss but suffers from overﬁtting, as shown in Figure 3(a) that its test accuracy is only about 68%.
Results on MNIST Dataset.
For MNIST, we use a convolutional neural network with three convolutional layers and one ﬁnal fully-connected layer.
Following each convolutional layer, ELU activation and batch normalization are used.
We compare NeAda with Adam under three different noise levels and the accuracy is shown in
Figures 4(a) to 4(c).
Under all noise levels, NeAda outperforms Adam with the same learning rates.
When we have proper time-scale separation (the red curves), both methods achieve good test accuracy, and NeAda achieves higher accuracy and converges faster.
After we change to the same learning rates for the primal and dual variables (the blue curves), the accuracy drop of NeAda is slighter compared to Adam, especially when ✏ = 0 .
As for the training loss shown in Figure 4(d), NeAda (the solid curves) is always faster at the beginning.
We also observed that with proper time-scale separation,
NeAda reaches a lower loss.
Conclusion
Both non-nested and nested adaptive methods are popular in nonconvex minimax problems, e.g., the training of GANs.
In this paper, we demonstrate that non-nested algorithms may fail to converge when the time-scale separation is ignorant of the problem parameter even when the objective is strongly-concave in the dual variable with noiseless gradients information.
[-tk =1 x k and S 
-]
[- 
-]
[-Note that ( g xt ) 
-]
[-∥ (cid:1) − σ 2 /M √ v t +1 (cid:112) v t + ∥∇ x f ( x t , y t ) ∥ 
-]
[-(cid:112) v t + ∥∇ x f ( x t , y t ) ∥ 
-]
[-∥ 
-]
[-Proof for Theorem 3.
-]
[-( A + E ) 
-]
[-In this part we wrap up everything for two cases: i) 
-]
We propose ﬁxes to this problem with a family of nested algorithms–NeAda, that nests the max oracle of the [-bounded gradient. Assumption C.
 (Bounded gradient in x ) .-]dual variable under an inner loop stopping criterion.
The proper stopping criterion will help to balance the outer loop progress and inner loop accuracy.
NeAda-AdaGrad attains the near-optimal complexity without a priori knowledge of problem parameters in the nonconvex-strongly-concave setting.
It can be a future direction to design parameter-agnostic algorithms for nonconvex-concave minimax problems or more general regimes by leveraging recent progress in nonconvex minimax optimization and the adaptive analysis in this paper.
Another interesting direction is to investigate the convergence behavior of Adam-type algorithms with general decaying rates in the strongly convex online optimization.
Acknowledgement
This work was supported by an ETH Research Grant funded through the ETH Zurich Foundation.
[-Regarding this inequality as a quadratic of (cid:113)(cid:80) T − 1 t =
-]