Nest Your Adaptive Algorithm for Parameter-Agnostic Nonconvex Minimax Optimization
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Adaptive algorithms like AdaGrad and AMSGrad are successful in nonconvexoptimization owing to their parameter-agnostic ability – requiring no a prioriknowledge about problem-speciﬁc parameters nor tuning of learning rates.
However, when it comes to nonconvex minimax optimization, direct extensions of suchadaptive optimizers without proper time-scale separation may fail to work in practice.
We provide such an example proving that the simple combination of Gradient Descent Ascent (GDA) with adaptive stepsizes can diverge if the primal-dual stepsize ratio is not carefully chosen; hence, a fortiori, such adaptive extensions are notparameter-agnostic.
To address the issue, we formally introduce a Nested Adaptiveframework, NeAda for short, that carries an inner loop for adaptively maximizingthe dual variable with controllable stopping criteria and an outer loop for adaptivelyminimizing the primal variable.
Such mechanism can be equipped with off-theshelf adaptive optimizers and automatically balance the progress in the primal anddual variables.
Theoretically, for nonconvex-strongly-concave minimax problems,we show that NeAda can achieve the near-optimal e
O ( ✏  2 ) and e
O ( ✏  4 ) gradientcomplexities respectively in the deterministic and stochastic settings, without priorinformation on the problem’s smoothness and strong concavity parameters.
Tothe best of our knowledge, this is the ﬁrst algorithm that simultaneously achievesnear-optimal convergence rates and parameter-agnostic adaptation in the nonconvexminimax setting.
Numerically, we further illustrate the robustness of the NeAdafamily with experiments on simple test functions and a real-world application.
Introduction
Adaptive gradient methods, whose stepsizes and search directions are adjusted based on past gradients,have received phenomenal popularity and are proven successful in a variety of large-scale machinelearning applications.
Prominent examples include AdaGrad [15], RMSProp [27], AdaDelta [69],
Adam [34], and AMSGrad [55], just to name a few.
Their empirical success is especially pronouncedfor nonconvex optimization such as training deep neural networks.
Besides improved performance,being parameter-agnostic is another important trait of adaptive methods.
Unlike (stochastic) gradientdescent, adaptive methods often do not require a priori knowledge about problem-speciﬁc parameters(such as Lipschitz constants, smoothness, etc.).
1 On the theoretical front, some adaptive methods canachieve nearly the same convergence guarantees as (stochastic) gradient descent [15, 64, 55].
Recently, adaptive methods have sprung up for minimax optimization:
[Equation]
where f is l -Lipschitz smooth jointly in x and y , Y is closed and convex, and ⇠ is a random vector.
Such problems have found numerous applications in generative adversarial networks (GANs) [23, 4],
Wasserstein GANs [4], generative adversarial imitation learning [28], reinforcement learning [12, 49],adversarial training [59], domain-adversarial training of neural networks [19] , etc.
A common practice is to simply combine adaptive stepsizes with popular minimax optimizationalgorithms such as Gradient Descent Ascent (GDA), extragradient method (EG) and the like; seee.g.
It is worth noting that these methods are reported successful in some applicationsyet at other times can suffer from training instability.
In recent years, theoretical behaviors ofsuch adaptive methods are extensively studied for convex-concave minimax optimization; see e.g.,[5, 3, 1, 16, 58, 20, 48, 13].
However, for minimax optimization in the important nonconvex regime,little theory relative to adaptive methods is known.
Unlike the convex-concave setting, a key challenge for nonconvex minimax optimization lies in thenecessity of a problem-speciﬁc time-scale separation of the learning rates between the min-playerand max-player when GDA or EG methods are applied, as proven in [67, 41, 56, 6].
This makesthe design of adaptive methods fundamentally different from and more challenging than nonconvexminimization.
Several recent attempts [25, 29, 30] studied adaptive methods for nonconvex-stronglyconcave minimax problems; yet, they all require explicit knowledge of the problems’ smoothnessand strong concavity parameters to maintain a stepsize ration proportional to the condition number.
Such a requirement evidently undermines the parameter-agnostic trait of adaptive methods.
This thenraises a couple of interesting questions: (1) Without a problem-dependent stepsize ratio, does simplecombination of GDA and adaptive stepsizes still converge?
(2) Can we design an adaptive algorithmfor nonconvex minimax optimization that is truly parameter-agnostic and provably convergent?
In this paper, we address these questions and make the following key contributions:
• We investigate two generic frameworks for adaptive minimax optimization: one is a simple
(non-nested) adaptive framework, which performs one step of update of x and y simulta-
neously with adaptive gradients; the other is Nested Adaptive (NeAda) framework, whichperforms multiple updates of y after one update of x , each with adaptive gradients.
Both
frameworks allow ﬂexible choices of adaptive mechanism such as Adam, AMSGrad and
AdaGrad.
We provide an example proving that the simple adaptive framework can fail
to converge without setting an appropriate stepsize ratio; this applies to any of the adap-
tive mechanisms mentioned above, even in the noiseless setting.
In contrast, the NeAda
framework is less sensitive to the stepsize ratio, as numerically illustrated in Figure 1.
• We provide the convergence analysis for a representative of NeAda that uses AdaGrad
stepsizes for x and a convergent adaptive optimizer for y , in terms of nonconvex-strongly-
concave minimax problems.
Notably, the convergence of this general scheme does not
require to know any problem parameters and does not assume the bounded gradients.
We
demonstrate that NeAda is able to achieve e O ( ✏  2 ) oracle complexity for the deterministic
setting and e O ( ✏  4 ) for the stochastic setting to converge to ✏ -stationary point, matching best
known bounds.
To the best of our knowledge, this seems to be the ﬁrst adaptive framework
for nonconvex minimax optimization that is provably convergent and parameter-agnostic.
• We further make two complementary contributions, which can be of independent interest.
First, we propose a general AdaGrad-type stepsize for strongly-convex problems without
knowing the strong convexity parameters, and derive a convergence rate comparable to SGD.It can serve as a subroutine for NeAda.
Second, we provide a high probability convergence
result for the primal variable of NeAda under a subGaussian assumption
Finally, we numerically validate the robustness of the NeAda framework on several test
functions compared to the non-nested adaptive framework and demonstrate the effectiveness of the NeAda framework on distributionally robust optimization task on a real dataset [Figure]
1.1 Related workAdaptive algorithms.
Duchi et al.
[15] introduce AdaGrad for convex online learning and achieve
O ( p T ) regrets.
Li and Orabona [39] and Ward et al.
[64] show an e
O ( ✏  4 ) complexity for AdaGradin the nonconvex stochastic optimization.
There are an extensive number of works on AdaGrad-typemethods; to list a few, [37, 2, 32, 53].
Another family of algorithms uses more aggressive stepsizes ofexponential moving average of the past gradients, such as Adam [34] and RMSProp [27].
Reddi et al.[55] point out the non-convergence of Adam and provide a remedy with non-increasing stepsizes.
There is a surge in the study of Adam-type algorithms due to their popularity in the deep neuralnetwork training [68, 9, 42].
Some work provides the convergence results for adaptive methods in thestrongly-convex optimization [63, 36, 50].
Line search and stochastic line search are another effectivestrategy that can detect the objective’s curvature and have received much attention [60, 62, 61].
Notably, many adaptive algorithms are parameter-agnostic [15, 55, 64].
Nonconvex minimax optimization.
In NC-SC setting, stationary convergence of GDA in NC-SCsetting was ﬁrst provided by Lin et al.
[41], showing O ( ✏  2 ) oracle complexity and O ( ✏  4 ) samplecomplexity with minibatch.
Recently, Chen et al.
[8] and Yang et al.
[67] achieve this samplecomplexity in the stochastic setting without minibatch.
GDmax is a double loop algorithm thatmaximizes the dual variable to a certain accuracy.
It achieves nearly the same complexity as GDA [52].
Sebbouh et al.
[56] recently discuss the relation between the two-time-scale and number of inner stepsfor GDmax.
Besides NC-SC setting, some work provides convergent algorithms when the objectiveis (non-strongly) concave about the dual variable[70, 46, 66].
In addition, nonconvex-nonconcaveregime is only explored under some special structure [44, 13], such as Polyak- Ł ojasiewicz (PL)condition [17].
All algorithms mentioned above require prior knowledge about problem parameters,such as smoothness modulus, strong concavity modulus, and noise variance.
Adaptive algorithms in minimax optimization
There exist many adaptive and parameteragnostic methods designed for convex-concave minimax optimization as a special case of monotonevariational inequality [5, 3, 1, 16, 58, 20, 48, 13].
Most of them combine extragradient method, mirrorprox [51] or the like, with AdaGrad mechanism.
Liu et al.
[43] and Dou and Li [14] relax convexityconcavity assumption to the regime where Minty variational inequality (MVI) has a solution.
In thesesettings, time-scale separation of learning rates is not required even for non-adaptive algorithms.
Fornonconvex-strongly-concave problems, Huang and Huang [29], Huang et al.
[30], Guo et al.
[25]propose adaptive methods, which set the learning rates based on knowledge about smoothness andstrong-concavity modulus and the bounds for adaptive stepsizes.
Non-nested and nested adaptive methods
In this section, we investigate two generic frameworks that can incorporate most existing adaptivemethods into minimax optimization.
We remark that many variants encapsulated in these twofamilies are already widely used in practice, such as training of GAN [22], distributionally robustoptimization [57], etc.
These two frameworks, coined as non-nested and nested adaptive methods, can be viewed as adaptive counterparts of GDA and GDmax.
We aim to illustrate the differencebetween these two adaptive families, even though GDA and GDmax are often considered “twins”.
Non-nested adaptive methods.
In Algorithm 1, non-nested methods update the primal and dualvariables in a symmetric way.
Weighted gradients m xt and m yt are the moving average of the paststochastic gradients with the momentum parameters  x and  y .
The effective stepsizes of x and yare ⌘ x / p v xt and ⌘ y / p v yt , where the division is taking coordinate-wise.
We refer to ⌘ x and ⌘ y aslearning rates, and v xt , v yt are some averaging of squared-past gradients through function .
Manypopular choices of adaptive stepsizes are captured in this framework, see also [55]:
[Equation]
1: Input: x 0 and y2: for t = 0 , 1 , 2 , ... do 3: sample ⇠ t and let g
[Equation]
⇠
[Equation]
and g
[Equation] =
[Equation] [Equation] ; ⇠ t ) 4: // update the first moment
[-+
 =-][Equation]
and [Equation]
yt + (1   y )
g yt 5: // update the second moment v xt +
=  v x 0 , { ( g xi ) 2 } ti =
and v yt +1 =
v y 0 , { ( g yi ) 2 } ti =
6: // update variables x t +1 = x
t  ⌘ x p v xt +1 m xt +
and y t +1 =
y t + ⌘ y p v yt
+1 m yt
+ 7: end for
1: Input: x 0 and y2: for t = 0 , 1 , 2 , ... do 3: for k = 0 , 1 , 2 , ... until a stopping criterion is satisﬁed
do 4: sample ˆ ⇠ kt and g yt,k = r y F ( x t , y kt ; ˆ ⇠ kt ) 5: m yt, k +1 =
y m yt,k + (1   y ) g yt,k 6: v yt,k
+1 =
y  v yt, 0 , { ( g yt,i ) 2 } ki =  7: y k +1 t = y kt + ⌘ y p v yt,
k +1 m yt,k + 8: end for 9: v yt +1 ,
= v yt,k +
and m yt +1 ,
= m yt,k + 10: sample ⇠ t and g xt =
r x F ( x t , y k +1 t ; ⇠ t ) 11: m
[Equation] =
[Equation] [Equation]   x ) g xt 12: v xt +
= x  v x 0 , { ( g xi ) 2 } ti =
13: x t +1 = x
t  ⌘ x p v xt +1 m xt + 14: end for
Nested adaptive (NeAda) methods.
NeAda, presented in Algorithm 2, has a nesting inner loop tomaximize y until some stopping criterion is reached (see details in Section 3).
Instead of using a ﬁxednumber of inner iterations or a ﬁxed target accuracy as in GDmax [52], NeAda gradually increasesthe accuracy of the inner loop as the outer loop proceeds to make it fully adaptive.
We refer to the ratio between two learning rates, i.e. ⌘ y / ⌘ x , as the two-time-scale.
The currentanalysis of GDA in nonconvex-strongly-concave setting requires two-time-scale to be proportionalwith the condition number  = l/µ , where l and µ are Lipschitz smoothness and strongly-concavitymodulus [41, 67].
We provide an example showing that the problem-dependent two-time-scale isnecessary for GDA and most non-nested methods even in the deterministic setting.
Lemma 2.1 Consider the function {+ 12 y 
+}
[-( x, y ) =  12 y 
-]
+ Lxy  L 2 2 x 2 in the deterministic setting.
Let r ⌘ x = ⌘ y .(1)
GDA will not converge to the stationary point when r  L 2 :
[Equation]
(2) Assume the averaging function x and y are the same, and satisfy that for any ⌧ , if v xt = ⌧ v ytand ( g xt ) 2 = ⌧ ( g yt ) 2 then v xt +
= ⌧ v yt
+1 .
With  x =  y , v x 0 = v y 0 =
and m x 0 = m y 
[-= 
-]
= 0 (which are commonly used in practice), non-nested adaptive method will not converge when r  L :
[Equation]
When r = L , r x f ( x t , y t ) =
r x f ( x 0 , y 0 ) for all t .
Remark 1 Most popular adaptive stepsizes we mentioned before, such as Adam, AMSGrad and
AdaGrad, have averaging functions satisfying the assumption in the lemma.
Any point on the liney =
Lx is a stationary point for the above function.
The lemma implies that for any given time-scaler , there exists a problem for which the non-nested algorithm does not converge to the stationary point,so they are not parameter-agnostic.
We compare non-nested and nested methods combined with different stepsizes schemes: Adam,
AMSGrad, AdaGrad and ﬁxed stepsize, on the function:  12 y 2 + 2 xy  2 x 2 .
In the experimentsof this section, we halt the inner loop when the (stochastic) gradient about y is smaller than 1 /tor
the number iteration is greater than t .
We observe from Figure 1 that the thresholds for thenon-convergence of non-nested methods ( r = 2 for adaptive methods and r = 4 for GDA) are exactlyas predicted by the lemma.
Although the adaptive methods admit a smaller two-time-scale thresholdthan GDA in this example, it is not a universal phenomenon from our experiments in Section 4.
Interestingly, nested adaptive methods are robust to different two-time-scales and always have thetrend to converge to the stationary point.
Convergence Analysis of NeAda-AdaGrad
In this section, we reveal the secret behind the robust performance of NeAda by providing theconvergence guarantee for a representative member in the family.
For sake of simplicity and clarity,we mainly focus on NeAda with AdaGrad.
Adam-type mechanism can suffer from non-convergencealready for nonconvex minimization despite its good performance in practice.
Our result also shedslight on the analysis of other more sophisticated members such as AMSGrad in the family.
NeAda-AdaGrad: Presented in Algorithm 3, NeAda-AdaGrad adopts the scalar AdaGrad schemefor the x -update in the outer loop and uses mini-batch in the stochastic setting.
For the inner loopfor maximizing y , we run some adaptive algorithm for maximizing y until some easily checkablestopping criterion is satisﬁed.
We suggest two criteria here: at t -th outer loop: (I) the gradient normabout y is smaller than 1 / ( t + 1) in the deterministic setting, (II) the number of inner loop iterationsreaches t + 1 in the stochastic setting. [Equation] [Equation] [Equation]
[Equation]
For the purpose of theoretical analysis, we mainly focus on the minimax problem of the form (1) underthe nonconvex-strongly-concave (NC-SC) setting 2 , formally stated in the following assumptions.
Assumption 3.1 (Lipschitz smoothness)
There exists a positive constant l > 0 such thatmax
[Equation]
[Table]
holds for all x 1 , x 2 2 R d , y 1 , y 2 2 Y .
Assumption 3.2 (Strong-concavity in y )
There exists µ > 0 such that: f ( x, y 1 )  f ( x, y 2 ) +hr [Equation] f
[Equation]
[Equation]
[Equation]
Since the objective is nonconvex about x , we aim at ﬁnding an ✏ -stationary point ( x t , y t ) such that E kr x f ( x t , y t ) k  ✏ and E kr y f ( x t , y t ) k  ✏ , where the expectation is taken over the randomness 174 in the algorithm. For simplicity of notation, deﬁne  =
l/µ as the condition number,  ( x ) =max y
2 Y f ( x, y ) as the primal function, and y ⇤ ( x ) =
arg max Y f
( x, y ) as the optimal y w.r.t x .
3.1 Convergence in deterministic and stochastic settings
Assumption 3.3 (Stochastic gradients) r x F ( x, y ; ⇠ ) and r y F ( x, y ; ⇠ ) are unbiased stochasticestimators of r x f ( x, y ) and r y f ( x, y ) and have variances bounded by  2  0 .
We assume the unbiased stochastic gradients have the variance  2 , and the problem reduces to thedeterministic setting when  = 0 .
Now we provide a general analysis of the convergence for anyadaptive optimizer used in the inner loop.
Theorem 3.1 Deﬁne the expected cumulative suboptimality of inner loops as E =
[Equation]
i .
Under Assumptions 3.1, 3.2 and 3.3, the output from Algorithm 3 satisﬁes
[Equation]
where A =
[Equation]
Remark {+
+}
The general analysis is built upon milder assumptions than existing work on AdaGrad innonconvex optimization, not requiring either bounded gradient in [64] or prior knowledge about thesmoothness modulus in [39].
This lemma implies the algorithm attains convergence for the nonconvexvariable x with any choice ⌘ > 0 and v 0 > 0 , so it is parameter-agnostic.
Remark {+
+}
Another beneﬁt of this analysis is that the variance  appears in the leading term T  14 ,which means the convergence rate can interpolate between the deterministic and stochastic settings.
It implies a complexity of e
O ( ✏  2 ) in the deterministic setting and e
O ( ✏  4 ) in the stochastic setting forthe primal variable, regardless of the batch size M and the choice of subroutine.
However, M cancontrol the balance between the number of outer loops and the number of samples in each loop.
In the next two theorems, we derive the total complexities, in the deterministic and stochastic settings,of ﬁnding ✏ -stationary point by controlling the cumulative suboptimality E in Theorem 3.1 for speciﬁcsubroutine A .
In fact, we can also use any off-the-shelf adaptive optimizer for solving the innermaximization problem up to the desired accuracy.
Theorem 3.2 (deterministic) Under the same setting as Lemma 3.1 with  = 0 , for Algorithm 3with any linearly-convergent subroutine A under criterion I and M = 1 , there exist t ⇤  e
O  ✏  such that ( x t ⇤ , y t ⇤ ) is an ✏ -stationary point.
Therefore, the total gradient complexity is e
O  ✏  .
Remark {+
+}
This complexity is optimal in ✏ up to logarithmic term [71], similar to GDA [41].
Notethat many adaptive and parameter-agnostic algorithms can achieve the linear rate when solvingsmooth and strongly concave maximization problems; to list a few, gradient ascent with backtrackingline-search [60], SC-AdaNGD [36] and polyak stepsize [26, 45, 54] 3 .
Here we can also pick moregeneral subproblem accuracy in criterion I that only needs to scale with 1 /t .
Theorem 3.3 (stochastic) Under the same setting as Lemma 3.1, for Algorithm 3 with M = ✏  2and any sublinearly-convergent subroutine A such that E k y t  y ⇤ ( x t )
k 2  b 1 k y t  1 {+ y ⇤ ( x t ) k 
+}
+ b 2 tunder the stopping criterion II, where b 1 , b 2 > 0 are constants, there exist t ⇤  e
O  ✏  such that( x t ⇤ , y t ⇤ ) is an ✏ -stationary point.
Therefore, the total stochastic gradient complexity is e
O  ✏  .
Remark {+
+}
This e
O  ✏  complexity is nearly optimal in the dependence of ✏ for stochastic NC-SCproblems [38].
Here we set M = ✏  2 for the simplicity of exposition, and a similar result also holdsfor gradually increasing M .
The sublinear rate speciﬁed above for solving the stochastic stronglyconvex subproblem can be achieved by several existing parameter-agnostic algorithms under someadditional assumptions, such as F REE R EX M OMENTUM [10] and Coin-Betting [11] 4 .
Parameter-free
SGD [7] is partially parameter-agnostic that only requires the stochastic gradient bound rather thanthe strongly-convexity parameter.
Mukkamala and Hein [50] and Wang et al.
[63] introduce thevariants of AdaGrad, RMSProp and Adam for strongly-convex online learning, but they need to knowboth gradient bounds and strongly-convexity parameter for setting stepsizes.
We will show in thenext subsection that AdaGrad with a slower decaying rate is parameter-agnostic.
We note that theanalysis of this theorem is not the simple gluing of the outer loop and inner loop complexity, butrequires more sophisticated control of the cumulative suboptimality E .
With the popularity of computational resource demanding deep neural networks, in both minimizationand minimax applications, people ﬁnd high probability guarantees for a single run of an algorithmuseful [33, 40].
Given the lack of such guarantee in the minimax optimization, we provide a highprobability convergence result for NeAda in Appendix C, which shows a similar sample complexityas Theorem 3.2 under the subGaussian noise.
3.2 Generalized AdaGrad for strongly-convex subproblem
We now introduce the generalized AdaGrad for minimizing strongly convex objectives, which canserve as an adaptive subroutine for Algorithm 3, without requiring knowledge on the strongly convexparameter.
We analyze it for the more general online convex optimization setting: at each round t , thelearner updates its decision x t , then it suffers a loss f t ( x t ) and receives the sub-gradient of f t .
Thegeneralized AdaGrad, described in Algorithm 4, keeps the cumulative gradient norm v t and takes thestepsize ⌘ /v ↵ t with a decaying rate ↵ 2 (0 , 1] .
When ↵ = 1 / 2 , it reduces to the scalar version of theoriginal AdaGrad [15]; when ↵ = 1 , it reduced to the scalar version of SC-AdaGrad [50].
Theorem 3.4 Consider Algorithm 4 for online convex optimization and assume that (i) f t is continuous and µ -strongly convex, (ii) X is convex and compact with diameter D ; (iii) k g t k  G for every t .
Then for ↵ = 1 with ⌘  G 2 2 µ or 0 < ↵ < 1 with any ⌘ > 0 , the regret of Algorithm 4 satisﬁes:
[Equation]
where c 1 and c 2 are constants depending on the problem parameters and ↵
[Figure]
The theorem implies a logarithmic regret for the case ↵ = 1 , but the stepsize needs knowledge aboutproblem’s parameters µ and G ; similar results are shown for SC-AdaGrad [50] and SAdam [63].
When ↵ < 1 , the algorithm becomes parameter-agnostic and attains an e O ( T 1  ↵ ) regret.
Suchparameter-agnostic phenomenon for smaller decaying rates is also observed for SGD in stochasticoptimization [18].
Proving the regret bound for the generalized AdaGrad with ↵ < 1 in the onlinesetting is much more challenging, since the adversarial g t can lead to a “sudden” change in thestepsize.
In the proof, we bound the possible number of times such “sudden” change could happen.
To the best of our knowledge, this is the ﬁrst regret bound for adaptive methods with general decayingrates in the strongly convex setting.
By online-to-batch conversion [31], it can be converted toe
O ( T  ↵ ) rate in the strongly convex stochastic optimization.
Xie et al.
[65] prove the O (1 /T )convergence rate for AdaGrad with ↵ = 1 / 2 in this setting, but under a strong assumption, Restricted
Uniform Inequality of Gradients (RUIG), that requires the loss function with respect to each sample⇠ to satisfy the error bound condition with some probability.
Experiments
To evaluate the performance of NeAda, we conducted experiments on simple test functions and areal-world application of distributional robustness optimization (DRO).
In all cases, we compare
NeAda with the non-nested adaptive methods using the same adaptive schemes.
For notationalsimplicity, in all ﬁgure legends, we label the non-nested methods with the names of the adaptivemechanisms used.
We observe from all our experiments that: 1) while non-nested adaptive methodscan diverge without the proper two-time-scale, NeAda with adaptive subroutine always converges; 2)when the non-nested one converges, NeAda can achieve comparable or even better performance.
Test functions.
In Section 2, we have compared NeAda with non-nested methods on a quadraticfunction in Figure 1 and the observations match Lemma 2.1.
Now we consider a more complicatedfunction that is composed of McCormick function in x , a bilinear term, and a quadratic term in y ,
[Equation]
For this function, we compare the adaptive frameworks in the stochastic setting with Gaussian noise.
As demonstrated in Figure 2, non-nested methods are sensitive to the selection of the two-time-scale.
When the learning rate ratio is too small, e.g., ⌘ y / ⌘ x = 0 .
01 , non-nested Adam, AMSGrad and GDAall fail to converge.
We observe that GDA converges when the ratio reaches 0.03, while non-nested
Adam and AMSGrad still diverge until 0.04.
Although non-nested adaptive methods require a smallerratio than GDA in Lemma 2.1, this example illustrates that adaptive algorithms sometimes canbe more sensitive to the time separation.
In comparison, NeAda with adaptive subroutine alwaysconverges regardless of the learning rate ratio.
Distributional robustness optimization.
To justify the effectiveness of NeAda on real-worldapplications, we carried out experiments on distributionally robust optimization [57], where the
[Figure]
primal variable is the model weights to be learned by minimizing the empirical loss while the dualvariable is the adversarial perturbed inputs.
The dual variable problem targets ﬁnding perturbationsthat maximize the empirical loss but not far away from the original inputs.
Formally, for modelweights x and adversarial samples y , we have:
[Equation]
where f ( x, y ) : = 1 n [Equation] where n is the total number of training samples, v i is the i -th original input and f i is the loss functionfor the i -th sample.
is a trade-off parameter between the empirical loss and the magnitude ofperturbations.
When  is large enough, this problem is nonconvex-strongly-concave and followingthe same setting as [57, 56], we set  = 1 . 3 .
With the code modiﬁed from [47], we train a three-layer
MLP with ELU activations on MNIST dataset [35] and report the test accuracy on adversarial samplesgenerated from fast gradient sign method with ✏ = 0 .
To make the test accuracy meaningful, NeAdawith Adam as subroutine is compared with Adam with ﬁxed 15 inner loop iterations, which isconsistent with the choice of inner loop steps in [57], and such choice obtains much better testaccuracy than the completely non-nested Adam (which results in only about 67% test accuracy).
As shown in Figure 3(a), when the learning rates are set to different scales, i.e., ⌘ x = 0 . 01 , ⌘ y = 0 .
08(red curves in the ﬁgure), both methods achieve reasonable test errors.
In this case, NeAda has highertest accuracy and reaches such accuracy faster than Adam.
If we change the learning rates to thesame scale, i.e., ⌘ x = 0 . 01 , ⌘ y = 0 .
01 (blue curves in the ﬁgure), NeAda retains good accuracy while Adam drops to an unsatisfactory performance.
This demonstrates the adaptivity and less-sensitivityto learning rates of NeAda.
In addition, Figure 3(b) illustrates the convergence speeds on the lossfunction, and NeAda (solid lines) always decreases the loss faster than Adam.
Note that Adam withthe same learning rates converges to a lower loss but suffers from overﬁtting, as shown in Figure 3(a)that its test accuracy is only about 68%.
Conclusion
Both non-nested and nested adaptive methods are popular in nonconvex minimax problems, e.g.,training of GANs.
In this paper, we demonstrate that non-nested algorithms may fail to convergewhen the time-scale separation is ignorant of the problem parameter even when the objective isstrongly-concave in the dual variable with noiseless gradients information.
We propose ﬁxes to thisproblem with a family of nested algorithms–NeAda, that nests the max oracle of the dual variableunder an inner loop stopping criterion.
The proper stopping criterion will help to balance the outerloop progress and inner loop accuracy.
NeAda-AdaGrad attains the near-optimal complexity withouta priori knowledge of problem parameters in the nonconvex-strongly-concave setting.
It can be afuture direction to design parameter-agnostic algorithms for nonconvex-concave minimax problemsor more general regimes by leveraging recent progress in nonconvex minimax optimization and theadaptive analysis in this paper.
Another interesting direction is to investigate the convergence behaviorof Adam-type algorithms with general decaying rates in the strongly convex online optimization.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[N/A] We consider our work fundamental research on optimization and there is no foreseeablesocietal
impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
[Table]
If you ran experiments...
[Table]
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
[Table]
mental results (either in the supplemental material or as a URL)?
[Yes] We include the
[Table]
code in supplemental materials.
[Table]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
[Table]
were chosen)?
[Yes]
[Table]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
[Table]
ments multiple times)?
[N/A] We obtain the same observation for different runs,
[Table]
therefore, report only one run for more clear illustration.
[Table]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A] Our experiments do not require
[Table]
large resource of computation.
[Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes]
In the experiment
[Table]
section.(b)
Did you mention the license of the assets?
[N/A]
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table]
We include our code in supplemental materials.
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board
[Table]
) approvals, if applicable?
[N
[Table]
A]
[Table]