TiAda: A Time-scale Adaptive Algorithm for Nonconvex Minimax Optimization
Abstract
Adaptive gradient methods have shown their ability to adjust the stepsizes on the fly in a parameteragnostic manner, and empirically achieve faster convergence for solving minimization problems.
When it comes to nonconvex minimax optimization, however, current convergence analyses of gradient descent ascent (GDA) combined with adaptive stepsizes require careful tuning of hyperparameters and the knowledge of problem-dependent parameters.
Such a discrepancy arises from the primal-dual nature of minimax problems and the necessity of delicate time-scale separation between the primal and dual updates in attaining convergence.
In this work, we propose a singleloop adaptive GDA algorithm called TiAda for nonconvex minimax optimization that automatically adapts to the time-scale separation.
Our algorithm is fully parameter-agnostic and can achieve near-optimal complexities simultaneously in deterministic and stochastic settings of nonconvexstrongly-concave minimax problems.
The effectiveness of the proposed method is further justified numerically for a number of machine learning applications.
1. Introduction
Adaptive gradient methods, such as AdaGrad [10], Adam [24] and AMSGrad [43], have become the default choice of optimization algorithms in many machine learning applications owing to their robustness to hyper-parameter selection and fast empirical convergence.
Classic analyses of gradient descent for smooth functions require the stepsize to be less than 2 /l , where l is the smoothness parameter and often unknown, whereas many adaptive schemes can automatically adapt to them [48, 49].
Such tuning-free algorithms are called parameter-agnostic , as they do not require any prior knowledge of problem-specific parameters, e.g., the smoothness or strong-convexity parameter.
In this work we aim to bring the benefits of adaptive stepsizes to solving the following problem:
[Equation]
where P is an unknown distribution from which we can drawn i.i.d.
samples, Y ⊂ R d 2 is closed and convex, and f : R d
× R d 2 → R is nonconvex in x .
We call x the primal variable and y the dual variable.
This minimax formulation has found vast applications in modern machine learning [2, 7, 11–14, 37, 39].
Albeit theoretically underexplored, adaptive methods are widely deployed in combination with popular minimax optimization algorithms such as (stochastic) gradient descent ascent (GDA), extragradient (EG) [25], and optimistic GDA [41, 42]; see, e.g., [8, 16, 38, 44].
While it seems natural to directly extend adaptive stepsizes to minimax optimization algorithms, a recent work by Yang et al.
[50] pointed out that such schemes may not always converge without knowing problem-dependent parameters.
Unlike the case of minimization, convergent analyses of
[Figure]
GDA and EG for nonconvex minimax optimization are subject to time-scale separation [4, 35, 46, 51] — the stepsize ratio of primal and dual variables needs to be smaller than a problem-dependent threshold — which is recently shown to be necessary even when the objective is strongly concave in y with true gradients [32].
Moreover, Yang et al.
[50] showed that GDA with standard adaptive stepsizes, fails to adapt to the time-scale separation requirement.
Take the following nonconvexstrongly-concave function as a concrete example:
[Equation]
where L > 0 is a constant.
Yang et al.
[50] proved that directly using adaptive stepsizes like
AdaGrad, Adam and AMSGrad will fail to converge if the ratio of initial stepsizes of x and y (denoted as η x and η y ) is large.
We illustrate this phenomenon in Figures 1( a ) and 1( c ), where
AdaGrad diverges.
To sum up, adaptive stepsizes designed for minimization, are not time-scale adaptive for minimax optimization and thus not parameter-agnostic .
To circumvent this time-scale separation bottleneck, Yang et al.
[50] introduced NeAda for problem (1) with nonconvex-strongly-concave objectives.
Although the algorithm is agnostic to the smoothness and strong-concavity parameters, there are several limitations: (a)
In the stochastic setting, it gradually increases the number of inner loop steps ( k steps for the k -th outer loop) to improve the inner maximization problem accuracy, resulting in a possible waste of inner loop updates if the maximization problem is already well solved; (b) NeAda needs a large batchsize of order Ω (cid:0) ϵ − 2 (cid:1) to achieve the near-optimal convergence rate in theory; (c) It is not fully adaptive to the gradient noise, since it deploys different strategies for deterministic and stochastic settings.
In this work, we address all of the issues above by proposing TiAda ( Ti me-scale Ada ptive Algorithm), a single-loop algorithm with time-scale adaptivity for minimax optimization.
Specifically, one of our major modifications is setting the effective stepsize, i.e., the scale of (stochastic) gradient used in the updates, of the primal variable to the reciprocal of the maximum between the primal and dual variables’ second moments, i.e., the sums of their past gradient norms.
This ensures the effective stepsize ratio of x and y being upper bounded by a decreasing sequence, which eventually reaches the desired time-scale separation.
Taking the test function (2) as an example, Figure 1 illus- trates the time-scale adaptivity of TiAda:
In Stage I, the stepsize ratio quickly decreases below the threshold; in Stage II, the ratio is stabilized and the gradient norm starts to converge fast.
In summary, our contributions are as follows:
• We introduce the first single-loop and fully parameter-agnostic adaptive algorithm, TiAda, for nonconvex-strongly-concave (NC-SC) minimax optimization.
It adapts to the necessary timescale separation without large batchsize or any knowledge of problem-dependant parameters or target accuracy.
TiAda finds an ϵ -stationary point with an optimal complexity of O (cid:0) ϵ − 2 (cid:1) in the deterministic case, and a near-optimal sample complexity of O (cid:0) ϵ − (4+ δ ) (cid:1) for any small δ > 0 in the stochastic case.
It shaves off the extra logarithmic terms in the complexity of NeAda with AdaGrad stepsize for both primal and dual variables [50].
TiAda is proven to be noise-adaptive, which is the first of its kind among nonconvex minimax optimization algorithms.
• While TiAda is based on AdaGrad stepsize, in Appendix A, we generalize TiAda with other existing adaptive schemes, and conduct experiments on several tasks.
We show that TiAda converges faster and is more robust compared with NeAda or GDA with other existing adaptive stepsizes.
1.1. Related Work
Adaptive gradient methods.
The original AdaGrad was introduced for online convex optimization and maintains coordinate-wise stepsizes.
In nonconvex stochastic optimization, AdaGradNorm with one learning rate for all directions is shown to achieve the same complexity as SGD [33, 48], even with the high probability bound [23, 34].
In comparison, RMSProp [19] and Adam [24] use the decaying moving average of past gradients, but may suffer from divergence [43].
Many variants of Adam are proposed, and a wide family of them, including AMSGrad, are provided with convergence guarantees [6, 9, 53, 54].
One of the distinguishing traits of adaptive algorithms is that they can achieve order-optimal rates without knowledge about the problem parameters [22, 29, 48].
Adaptive minimax optimization algorithms.
In the convex-concave regime, several adaptive algorithms are designed based on EG and AdaGrad stepsize, and they inherit the parameter-agnostic characteristic [1, 3].
In sharp contrast, when the objective function is nonconvex about one variable, most existing adaptive algorithms require knowledge of the problem parameters [17, 20, 21].
Very recently, it was proved that a parameter-dependent ratio between two stepsizes is necessary for GDA in NC-SC minimax problems with non-adaptive stepsize [32] and most existing adaptive stepsizes [50].
Heusel et al.
[18] shows the two-time-scaled GDA with non-adaptive stepsize or Adam will converge, but assuming the existence of an asymptotically stable attractor.
1.2. Notations
We denote l as the smoothness parameter, µ as the strong-concavity parameter, and κ :=
l/µ as the condition number.
For the minimax problem (1), we denote y ∗ ( x ) := arg max y ∈Y f ( x, y ) as the solution of the inner maximization problem, Φ( x ) :
= f ( x, y ∗ ( x )) as the primal function, and
P Y ( · ) as projection operator onto set Y .
We assume access to stochastic gradient oracle returning ∇ x F ( x, y ; ξ ) for x and ∇ y F ( x, y ; ξ ) for y .
2. Method
1: Input do 3: sample i.i.d. ξ xt {+and ξ yt , and let g xt = ∇ x F ( x t , y t ; ξ xt ) and g yt = ∇ y F ( x t , y t ; ξ yt ) 4: v xt +
+}
=
v xt + {+∥ g xt ∥ 2 and v yt +
 =+} v yt + ∥ g yt ∥5: x t +1 = x
t − η x max { v xt +1 ,v yt +1 } α g xt and y t +1 = P Y (cid:18) y t + η y ( v yt +1 ) β g yt (cid:19) 6: end for
We formally introduce the TiAda method in Algorithm 1, and the major difference with AdaGrad lies in line 5.
Like AdaGrad, TiAda stores the accumulated squared (stochastic) gradient norm of the primal and dual variables in v xt and v yt , respectively.
We refer to hyper-parameters η x
and η y as the initial stepsizes , and the actual stepsizes for updating in line 5 as effective stepsizes which are denoted by η xt and η yt .
TiAda adopts effective stepsizes
η xt = η x / max (cid:8) v xt +1 , v yt +1 (cid:9) α and η yt = η y / (cid:0) v yt +
(cid:1) β , while AdaGrad uses η x / (cid:0) v xt +1 (cid:1) 1 / 2 and η y / (cid:0) v yt +
(cid:1) 1 / 2 .
In Section 3, our theoretical analysis suggests to choose α > 1 / 2 > β .
2.1. The Time-Scale Adaptivity of TiAda
Current analyses of GDA with non-adaptive stepsizes require the time-scale, η xt /η yt , to be smaller than a threshold depending on problem constants such as the smoothness and the strong-concavity parameter [35, 51].
It is tempting to expect adaptive stepsizes to automatically find a suitable time-scale separation.
However, the quadratic example (2) given by Yang et al.
[50] shattered the illusion.
In this example, the effective stepsize ratio stays the same along the run of existing adaptive algorithms, including AdaGrad (see Figure 1( b )), Adam and AMSGrad, and they fail to converge if the initial stepsizes are not carefully chosen (see Yang et al.
[50] for details).
As v xt and v yt only separately contain the gradients of x and y , the effective stepsizes of two variables in these methods depend on their own history, which prevents them from cooperating to adjust the ratio.
Now we explain how TiAda adapts to both the required time-scale separation and small enough stepsizes.
First, the ratio of our modified effective stepsizes is upper bounded by a decreasing sequence when α > β :
[Equation] [Equation]
) as v yt is the sum of previous gradient norms and is increasing.
Regardless of the initial stepsize ratio η x /η y , we expect the effective stepsize ratio to eventually drop below the desirable threshold for convergence.
On the other hand, the effective stepsizes for the primal and dual variables are also upper bounded by decreasing sequences, η x / (cid:0) v xt +1 (cid:1) α and η y / (cid:0) v yt +1 (cid:1) β , respectively.
Similar to
AdaGrad, such adaptive stepsizes will reduce to small enough, e.g., O (1 /l ) , to ensure convergence.
To demonstrate the time-scale adaptivity of TiAda, we conducted experiments on the quadratic minimax example (2) with L = 2 .
As shown in Figure 1( b ), while the effective stepsize ratio of AdaGrad stays unchanged for this particular function, TiAda progressively decreases the ratio.
According to Lemma 2.1 of Yang et al.
[50], 1 /κ is the threshold where GDA starts to converge.
We label the time period before reaching this threshold as Stage I, during which as shown in Figure 1( c ), the gradient norm for TiAda increases.
However, as soon as it enters Stage II, i.e., when the ratio drops below 1 /κ , TiAda converges fast to the stationary point.
In contrast, since the stepsize ratio of AdaGrad never reaches this threshold, the gradient norm keeps growing.
3. Theoretical Analysis of TiAda
In this section, we study the convergence of TiAda under NC-SC setting with both deterministic and stochastic gradient oracles.
We make the following assumptions to develop our convergence results.
Assumption 3.1 (smoothness) Function f ( x, y ) is l -smooth ( l > 0 ) in both x and y , that is, for any x 1 , x
2 ∈ R
d 1 and y 1 , y 2 ∈ Y , we have
[Equation]
Assumption 3.2 (strong-concavity in y )
Function f ( x, y ) is µ -strongly-concave ( µ > 0 ) in y , that is, for any x ∈ R d 1 and y 1 , y 2 ∈ Y , we have
[Equation]
Assumption 3.3 (interior optimal point)
For any x ∈ R d 1 , y ∗ ( x ) is in the interior of Y .
Remark
The last assumption ensures ∇ y f
( x, y ∗ ( x ))
= 0 , which is important for AdaGrad-like stepsizes.
If the gradient about y is not 0 at y ∗ ( x ) , the stepsize will keep decreasing even near the optimal point, leading to slow convergence.
This assumption could be potentially alleviated by using generalized AdaGrad stepsizes [3].
We aim to find a near stationary point for the minimax problem (1).
Here, ( x, y ) is defined to be an ϵ stationary point
if ∥∇ x f ( x, y ) ∥ ≤ ϵ and ∥∇ y f ( x, y ) ∥
≤ ϵ in the deterministic setting, or E ∥∇ x f ( x, y ) ∥ 2 ≤ ϵ 2 and E ∥∇ y f
( x, y ) ∥ 2 ≤ ϵ 2 in the stochastic setting, where the expectation is taken over all the randomness in the algorithm.
This stationarity notion can be easily translated to the near-stationarity of the primal function Φ( x ) = max y ∈Y ( x, y )
Under our analyses, TiAda is able to achieve the optimal O (cid:0) ϵ − 2 (cid:1) complexity in the deterministic setting and a near-optimal
O (cid:0) ϵ − (4+ δ ) (cid:1) sample complexity for any small δ > 0 in the stochastic setting.
Firstly we assume to have access to the exact gradients of f ( · , · ) , and therefore we can replace ∇ x F ( x t , y t ; ξ xt ) and ∇ y F ( x t , y t ; ξ yt ) by ∇ x f ( x t , y t ) and ∇ y f ( x t , y t ) in Algorithm 1.
Theorem 2 (deterministic setting) Under Assumptions 3.1 to 3.3, Algorithm 1 with deterministic gradient oracles satisfies that for any 0 < β < α < 1 , after T iterations,
[Equation]
This theorem implies that for any initial stepsizes, TiAda finds an ϵ -stationary point within
O ( ϵ − 2 ) iterations.
Such complexity is comparable to that of nonadaptive methods, such as vanilla
GDA [35], and is optimal in the dependency of ϵ [52].
Like NeAda [50], TiAda does not need any prior knowledge about µ and l , but it improves over NeAda by removing the logarithmic term in the complexity.
Notably, we provide a unified analysis for a wide range of α and β .
while most existing literature on only validates a specific choice, e.g., α = 1 / 2 , for AdaGrad-like stepsizes [22, 48].
Now, we assume the access to a stochastic gradient oracle, that returns unbiased noisy gradients, ∇ x F ( x, y ; ξ ) and ∇ y F ( x, y ; ξ ) .
Also, we make the following additional assumptions.
Assumption 3.4 (stochastic gradients)
For z ∈ { x, y } , we have E ξ [ ∇ z F ( x, y, ξ )]
= ∇ z f ( x, y ) .
In addition, there exists a constant G such that ∥∇ z F ( x, y, ξ ) ∥ ≤ G for any x ∈ R d 1 and y ∈ Y .
Assumption 3.5 (bounded primal function value)
There exists a constant Φ max ∈ R such that for any x ∈ R d 1 , Φ( x ) is upper bounded by Φ max .
Remark 3 Many works on adaptive algorithms have the two assumptions above [23, 29].
This implies the domain of y is bounded, which is also assumed in the analyses of AdaGrad [28, 30].
Assumption 3.6 (second order Lipschitz continuity for y )
For any x 1 , x 2 ∈ R
d
and y 1 , y 2 ∈
Y , there exists constant L such that (cid:13)(cid:13) ∇ 2 xy f ( x 1 , y 1 ) − ∇ 2 xy f ( x 2 , y 2 ) (cid:13)(cid:13) ≤ L ( ∥ x 1 − x 2 ∥ + ∥ y 1 − y 2 ∥ ) and (cid:13)(cid:13) ∇ 2 yy f ( x 1 , y 1 ) − ∇ 2 yy f ( x 2 , y 2 ) (cid:13)(cid:13) ≤ L ( ∥ x 1 − x 2 ∥ + ∥ y 1 − y 2 ∥ ) .
Remark 4 Chen et al.
[5] also impose this assumption to achieve the optimal O (cid:0) ϵ − 4 (cid:1) complexity for GDA with non-adaptive stepsizes for solving NC-SC minimax problems.
Together with Assumption 3.3, we can show that y ∗ ( · ) is smooth.
Nevertheless, without this assumption, Lin et al.
[35] only show a worse complexity of O (cid:0) ϵ − 5 (cid:1) for GDA without large batchsize.
Theorem 5 (stochastic setting) Under Assumptions 3.1 to 3.6, Algorithm 1 with stochastic gradient oracles satisfies that for any 0 < β < α < 1 , after T iterations,
[Equation]
TiAda can achieve the complexity arbitrarily close to the optimal sample complexity, O (cid:0) ϵ − 4 (cid:1) [31], by choosing α and β arbitrarily close to 0 .
Specifically, TiAda achieves a complexity of
O (cid:0) ϵ − (4+ δ ) (cid:1) for any small δ > 0 if we set α = 0 .
5+ δ/ (8+2 δ ) and β = 0 .
5 − δ/ (8+2 δ ) .
Notably, this matches the complexity of NeAda with AdaGrad stepsizes for both variables [50].
Theorem 5 implies that TiAda is fully agnostic to problem parameters, e.g., µ , l and σ .
Compared with the only parameter-agnostic algorithm, NeAda, our algorithm has several advantages.
First, TiAda is a single-loop algorithm, while NeAda [50] needs increasing inner-loop steps and a huge batchsize of order Ω (cid:0) ϵ − 2 (cid:1) to achieve its best complexity.
Second, our stationary guarantee is for E ∥∇ x f ( x, y ) ∥ 2 ≤ ϵ 2 , which is stronger than E ∥∇ x
f ( x, y ) ∥ ≤ ϵ guarantee in NeAda.
Last but not least, NeAda needs to know whether σ = 0 as it uses different stopping criteria for the inner loop in deterministic and stochastic settings.
In comparison, TiAda achieves the (near) optimal complexity in both settings with the same strategy.
4. Conclusion
In this work, we bring in adaptive stepsizes to nonconvex minimax problems in a parameter-agnostic manner.
We designed the first time-scale adaptive algorithm, TiAda, which progressively adjusts the effective stepsize ratio and reaches the desired time-scale separation.
TiAda is also noise adaptive and does not require large batchsizes compared with the existing parameter-agnostic algorithm for nonconvex minimax optimization.
Furthermore, TiAda is able to achieve optimal and near-optimal complexities respectively wtih deterministic and stochastic gradient oracles.
We also empirically showcased the advantages of TiAda over NeAda and GDA with adaptive stepsizes.
Appendix A. Experiments
In this section, we first present extensions of TiAda that accommodate other adaptive schemes besides AdaGrad and are more practical in deep models.
Then we present empirical results of TiAda and compare it with (i) simple combinations of GDA and adaptive stepsizes, which are commonly used in practice, and (ii) NeAda with different adaptive mechanism [50].
For fair comparisons, we use the same hyper-parameters when comparing our TiAda with other algorithms.
Our experiments include test functions proposed by Yang et al.
[50], the NC-SC distributional robustness optimization [47], and training the nonconvex-nonconcave Wasserstein GAN with gradient penalty [16].
We believe that this not only validates our theoretical results but also shows the potential of our algorithm in real-world scenarios.
To show the strength of being parameter-agnostic of TiAda, in all the experiments, we merely select α = 0 .
6 and β = 0 .
4 without further tuning those two hyperparameters.
For notational simplicity, we will use the name of an existing adaptive algorithm to refer to the simple combination of GDA and it, i.e., setting the stepsize of GDA to that adaptive scheme separately for both x and y .
For instance “AdaGrad” for minimax problems stands for the algorithm that uses AdaGrad stepsizes separately for x and y in GDA.
A.1. Extensions to Other Adaptive Stepsizes and High-dimensional Models
Although we design TiAda upon AdaGrad-Norm, it is easy and intuitive to apply other adaptive schemes like Adam and AMSGrad.
To do so, for z ∈ { x, y } , we replace the definition of g zt and v zt +1 in line 3 and 4 of Algorithm 1 to
[Equation]
Algorithms
[Table]
where { β zt } is the momentum parameters and ψ is the second moment function.
Some common stepsizes that fit in this generalized framework can be seen in Table 1.
Since Adam is widely used in many deep learning tasks, we also implement generalized TiAda with Adam stepsizes in our experiments for real-world applications, and we label it “TiAda-Adam”.
Besides generalizing TiAda to accommodate different stepsize schemes, for high-dimensional models, we also provide a coordinate-wise version of TiAda.
Note that we cannot simply change everything in Algorithm 1 to be coordinate-wise, because we use the gradients of y in the stepsize of x and there are no corresponding relationships between the coordinates of x and y .
Therefore we use the global accumulated gradient norms to dynamically adjust the stepsize of x .
Denote the second moment (analogous to v xt +1 in Algorithm 1) for the i -th coordinate of x at the t -th step as v xt +1 ,i and globally v xt
+1 :
= (cid:80) d 1 i =1 v xt +1 ,i .
We also use similar notations for y .
Then, the update for the i -th parameter, i.e., x i and y i , can be written as
[Equation]
If we look at the effective stepsize of x in the above update scheme, when the overall gradients of y are small (i.e., v yt
+1 < v xt +1 ), meaning the inner maximization problem is well solved, then the first factor becomes 1 and the effective stepsize of x is just the second factor, similar to the AdaGrad updates.
If the term v yt +1 dominates over v xt +1 , the first factor would be smaller than 1 , allowing to slow down the update of x and waiting for a better approximation of y ∗ ( x ) .
Our results in the following subsections provide strong empirical evidence for the effectiveness of these TiAda variants, and developing convergence guarantees for them would be an interesting future work.
We believe our proof techniques for TiAda, together with existing convergence results for coordinate-wise AdaGrad and AMSGrad [6, 9, 54], can shed light on the theoretical analyses of these variants.
A.2. Test Functions
Firstly, we examine TiAda on the quadratic function (2) that shows the non-convergence of simple combinations of GDA and adaptive stepsizes [50].
Since our TiAda is based on AdaGrad, we compare it with GDA with AdaGrad stepsize and NeAda-AdaGrad [50].
For Figure 1 and the first row of Figure 2, we conduct experiments on problem (2) with L = 2 .
We use initial stepsize η y = 0 .
2 and initial point (1 , 0 . 01) for all runs.
As shown in Figure 2, when the initial ratio is poor, [Figure]
TiAda and NeAda-AdaGrad always converge while AdaGrad diverges.
NeAda also suffers from slow convergence when the initial ratio is poor, e.g., 1 and 1 / 2 after 2000 iterations.
In contrast,
TiAda automatically balances the stepsizes and converges fast under all ratios.
Ablation Study on Convergence Behavior with Different α and β
We conduct more experiments on the quadratic function to study the effect of hyper-parameters α and β on the convergence behavior of TiAda.
As discussed in Sections 1 and 2, we refer to the period before the stepsize ratio reduce to the convergence threshold as Stage I, and the period after that as Stage II.
In order to accentuate the difference between these two stages, we pick a large initial stepsize ratio η x /η y = 20 .
We compare 4 different pairs of α and β : α ∈ { 0 . 59 , 0 . 6 , 0 . 61 , 0 . 62 } and β = 1 − α .
From Figure 3, we observed that as soon as TiAda enters Stage II, the norm of gradients start to drop.
Moreover, the closer α and β are to 0.5, the more time TiAda remains in Stage I, which confirms the intuitions behind our analysis in Appendix C.3.
For the stochastic case, we follow Yang et al.
[50] and conduct experiments on the McCormick function which is more complicated and 2-dimensional: f ( x, y ) =
sin( x 1 + x 2 )+( x 1 − x 2 ) 2 − 32 x 1 +2 x 2 +1+ x 1 y 1 + x 2 y 2 − 12 ( y 21 + y 22 ) .
We chose η y = 0 .
01 , and the noises added to the gradients are from zero-mean Gaussian distribution with variance 0 .
TiAda consistently outperforms AdaGrad and NeAda-AdaGrad as demonstrated in the second row of Figure 2 regardless of the initial ratio.
In this function, we also run an ablation study on the effect of our design that uses max-operator in the update of x .
We compare TiAda with and its variant without the max-operator, TiAda without
MAX (Algorithm 2 in Appendix C.4) whose effective stepsizes of x are η x / (cid:0) v xt +1 (cid:1) α .
According to
Figure 2( h ), TiAda converges to smaller gradient norms under all configurations of α and β .
A.3. Distributional robustness optimization
In this subsection, we consider the distributional robustness optimization [47].
We target training the model weights, the primal variable x , to be robust to the perturbations in the image inputs, the [Figure] dual variable y .
The problem can be formulated as:
[Equation]
where f i is the loss function of the i -th sample, v i is the i -th input image, and y i is the corresponding perturbation.
There are a total of n samples and γ is a trade-off hyper-parameter between the original loss and the penalty of the perturbations.
If γ is large enough, the problem is NC-SC.
We adapt code from Lv [36], and used the same hyper-parameter setting as Sebbouh et al.
[46], Sinha et al.
[47], i.e., γ = 1 . 3 .
The model we used is a three layer convolutional neural network (CNN) with a final fully-connected layer.
For each layer, batch normalization and ELU activation are used.
The width of each layer is (32 , 64 , 128 , 512) .
The setting is the same as Sinha et al.
[47], Yang et al.
We set the batchsize as 128 , and for the Adam-like optimizers, including Adam, NeAda-Adam and
TiAda-Adam, we use β 1 = 0 .
9 , β 2 = 0 .
999 for the first moment and second moment parameters.
We conduct the experiments on the MNIST dataset [27].
Since it is common in practice to update y 15 times after each x update [47] for better generalization error, we implement AdaGrad using both single and 15 iterations of inner loop (update of y ).
We use a grid of stepsize combinations to evaluate TiAda and compare it with NeAda and GDA with corresponding adaptive stepsizes.
For
AdaGrad-like algorithms, we use { 0 . 1 , 0 . 05 , 0 . 01 , 0 . 0005 } for both η x and η y , and the results are reported in Figure 4.
In all cases, TiAda outperforms NeAda and AdaGrad, especially when η y = 0 .or 0 .
05 , the performance gap is large.
For Adam-like algorithms, we use { 0 . 001 , 0 . 0005 , 0 . 0001 } for η x and { 0 . 1 , 0 . 05 , 0 . 005 , 0 . 001 } for η y , and the results are shown in Figure 5.
In this case, we find that TiAda is not only faster, but also more stable comparing to Adam with single inner loop iteration.
We note that since Adam uses the reciprocal of the moving average of gradient norms, it is extremely unstable when the gradients are small.
Therefore, Adam-like algorithms often experience instability when they are near stationary points.
[Figure]
A.4. Generative Adversarial Networks
Another successful and popular application of minimax optimization is generative adversarial networks.
In this task, a discriminator (or critic) is trained to distinguish whether an image is from the dataset.
At the same time, a generator is mutually trained to synthesize samples with the same distribution as the training dataset so as to fool the discriminator.
We use WGAN-GP loss [16], which imposes the discriminator to be a 1 -Lipschitz function, with CIFAR-10 dataset [26] in our experiments.
We use the code adapted from Green9 [15].
A four layer CNN and a four layer CNN with transpose convolution layers are used respectively for the discriminator and generator.
Following a similar setting as Daskalakis et al.
[8], we set batchsize as 512 , the dimension of latent variable as 50 and the weight of gradient penalty term as 10 − 4 .
For the Adam-like optimizers, we set β 1 = 0 .
5 , β 2 = 0 .
To get the inception score [45], we feed the pre-trained inception network [Figure] [Figure]
TiAda-Adam ( η x ,η y = 0 .
Adam ( η x ,η y = 0 . 01)
TiAda-Adam ( η x ,η y = 0 .
Adam ( η x ,η y = 0 . 005) TiAda-Adam ( η x ,η y = 0 . Adam ( η x ,η y = 0 . 001) with 8000 synthesized samples. Since TiAda is a single-loop algorithm, for fair comparisons, we also update the discriminator only once for each generator update in Adam.
In Figure 6, we plot the inception scores of TiAda-Adam and Adam under different initial stepsizes.
We use the same color for the same initial stepsizes, and different line styles to distinguish the two methods, i.e., solid lines for TiAda-Adam and dashed lines for Adam.
For all the three initial stepsizes we consider, TiAda-Adam achieves higher inception scores.
Also, TiAda-Adam is more robust to initial stepsize selection, as the gap between different solid lines at the end of training is smaller than the dashed lines.
Appendix B. Helper Lemmas
Lemma 6 Let x 1 , ..., x T be a sequence of non-negative real numbers, x 1 > 0 and 0 < α < 1 .
Then we have
[Equation]
When α = 1 , we have
[Equation]
Proof For the first inequality in the case 0 < α < 1 , we have
[Equation]
The second inequality in the case 0 < α < 1 is proved in Lemma 3 of [29].
For α = 1 , it is proved in Lemma 3.2 of Ward et al.
Lemma 7 (smoothness of Φ( · ) and Lipschitzness of y ∗ ( · ) .
Lemma 4.3 in Lin et al.
Under
Assumptions 3.1 and 3.2, we have Φ( · ) is ( l + κl ) -smooth with ∇ Φ( x ) = ∇ x f ( x, y ∗ ( x )) , and y ∗ ( · ) is κ -Lipschitz.
Lemma 8 (smoothness of y ∗ ( · ) .
Lemma 2 in Chen et al.
Under Assumptions 3.1, 3.2 and 3.6, we have for some constant (cid:98) L ,
[Equation]
Appendix C. Proofs
In this section, we first show the proofs for theorems in Section 3 and then present an analysis for a
TiAda variant without the max-operator.
For notational convenience in the proofs, we denote the stochastic gradient as ∇ x (cid:101) f
( x t , y t ) and ∇ y (cid:101) f
( x t , y t ) .
Also denote y ∗ t = y ∗ ( x t ) , η [Equation]
= [Equation] and ∆Φ = Φ max − Φ ∗ .
We use 1 as the indicator function.
C.1. Proof of Theorem
We present a formal version of Theorem
[Equation]
Theorem 9 (deterministic setting) Under Assumptions 3.1 to 3.3, Algorithm 1 with deterministic gradient oracles satisfies that for any 0 < β < α < 1 , after T iterations,
[Equation] where [Equation] [Equation]
<
[Equation]
with ∆Φ = Φ( x 0 ) − Φ ∗ ,
[Equation]
{+In addition, denoting the above upper bound for (cid:80) T − 1 t =
+}
∥∇ x f ( x t , y t ) ∥ 2 as C 3 , we have
T − 1 (cid:88)
[Equation]
Proof Let us start from the smoothness of the primal function Φ( · ) .
By Theorem 7, Φ( x t +1 )
[Equation]
where in the second to last inequality
[Equation]
we used the strong-concavity of f ( x, · ) :
[Equation]
Telescoping and rearranging the terms, we have
[Equation]
≤ 2∆Φ +
[Equation]
≤ 2∆Φ +
[Equation]
[Equation]
We proceed to bound (cid:80) T − 1 t =
γ t ∥∇ y f ( x t , y t ) ∥ 2 .
Let t 0 be the first iteration such that (cid:0) v yt 0 +1 (cid:1) β > c 2 := max (cid:110)
η y µl µ + l , η y ( µ + l ) (cid:111) .
We have v yt 0 ≤ c 1 /β 2 , and for t ≥ t 0 ,
[Equation] [Equation]
where λ t > 0 will be determined later.
For l -smooth and µ -strongly convex function g ( x )
[Equation]
according to Theorem 2.1.12 in Nesterov [40]
[Equation]
we have
[Equation]
Therefore,
Term (A)
[Equation]
Let λ t =
η y µl
[Equation]
yt +
[Equation]
−
[Equation]
y µl .
Note that λ t > 0 after t 0 .
Then
Term (A)
[Equation]
As 1 + λ t ≥ 1 and (cid:0) v yt +1 (cid:1) β ≥ η y ( µ + l )
[Equation]
we have term (B) ≤ − η y ( µ + l ) ( v yt +1 ) β .
Putting them back, we can get
[Equation]
Then, by telescoping, we have
[Equation]
For the first term in the RHS, using Young’s inequality with τ to be determined later
[Equation]
we have
[Equation]
Combined with Equation (6) [Equation] we have
[Equation]
By adding terms from 0 to t 0 − 1 and η y v y 0 ( v y 0 ) β from both sides
[Equation]
we have
[Equation] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
The LHS can be bounded by (cid:0) v yT (cid:1) 1 − β by Theorem 6.
Then we get two useful inequalities from above:
[Equation]
Now bring it back to Equation (5), we get
[Equation]
For the LHS, we have
[Equation]
[Equation]
From here, by combining two inequalites above and noting that (cid:80) T − 1 t =0 ∥∇ x f ( x t , y t ) ∥ 2 ≤ v xT , we {+can already conclude that (cid:80) T − 1 t =
+}
∥∇ x f ( x t , y t ) ∥ 2 = O (1) . Now we will provide an explicit bound. We consider two cases:
[Equation]
If v yT ≤ v xT , then
[Equation]
where we used x − m ( c + log x ) ≤ e cm em for x > 0 , m > 0 and c ∈ R in the last inequality. Also, if 0 < α i < 1 and b i are positive constants, and x ≤ (cid:80) ni =1 b i x α i , then we get x ≤ n (cid:80) ni =1 b 1 / (1 − α i )
i [Equation]
Now consider v x T as the x in the previous statement, and note that the LHS of Equation (8) equals to v xT − v x 0 . Then we {+can get v xT ≤ 5 v x 
 +(cid:18) 2∆Φ η x (cid:19)+}
[Equation]
[Table]
[Equation]
If v yT ≤ v xT , then we can use the upper bound for v yT from Equation (7). We now discuss two cases: 1. 2 α < 1 + β .
Then we have [Equation] Note that since α > β
[Equation]
we have [Equation] Therefore, with the same reasoning as Equation (9),
[Equation]
which gives us constant RHS.
α ≥ 1 + β .
Then we have
[Equation]
[Equation]
which implies
[Equation]
Now we also get only a constant on the RHS.
Summarizing all the cases, we finish the proof.
C.2. Intermediate Lemmas for Theorem
Lemma
Under the same setting as Theorem 5, if for t = t 0 to t 1 − 1 and any λ t > 0 , S t ,
[Equation]
then we have
[Equation]
Proof Letting λ t :=
[Equation]
we have
[Equation]
[Equation]
By multiplying 1 γ t (1+ λ t ) and rearranging the terms, we can get
[Equation]
By telescoping from t = t 0 to t 1 − 1 , we have
[Equation]
Now we take the expectation and get
E [ LHS ] ≥
[Equation]
where we used strong-concavity in the last inequality
[Equation]
Lemma
Under the same setting as Theorem 5, if v yt +1 ≤ C for t = 0 , ..., t 0 − 1 , then we have
E
[Equation]
Proof By Young’s inequality
[Equation]
we have
[Equation]
Then letting λ t = µγ t 2 and by Theorem
[Equation]
we have
[Equation]
We now remain to bound the last term:
[Equation]
By Theorem 7 we have[-t 0 − 1 (cid:88) t = (cid:13)(cid:13) y ∗ t +1 − y ∗ t (cid:13)(cid:13) 2 ≤ κt 0 − 1 (cid:88) t =
 ∥ x-]
[Equation]
[Equation]
where we applied Theorem 6 in the last inequality.
Bringing back this result, we finish the proof
[Equation]
Lemma
Under the same setting as Theorem 5, if t 0 is the first iteration such that v yt 0 +1 > C , then we have
[Equation]
Proof By the Lipschitzness of y ∗ ( · ) as in Theorem
[Equation]
we have
[Equation]
For Term (C)
[Equation]
by the Cauchy-Schwarz and Lipschitzness of y ∗ ( · ) , [Equation]
[Equation]
where we used Young’s inequality in the last step and λ t > 0 will be determined later
[Equation]
For Term (D), according to Cauchy-Schwarz and the smoothness of y ∗ ( · ) as shown in Theorem 8,
[Equation]
where in the last step we used Young’s inequality and τ >
[Equation]
Therefore
[Equation]
in total [Equation] we have
[Equation]
Note that we can upper bound η t by
[Equation]
and
[Equation]
which, plugged into the previous result
[Equation]
implies
[Equation]
Now we choose λ t =[-µη y 4 ( v yt +1 ) β and τ = µη y ( v y 0 ) 
-]
[Equation]
and τ [Equation]
[-2 + (cid:98) L 2 G 2 ( η x )µη y ( v y 0 ) 
 α − β (cid:33) η 2 t (cid:13)(cid:13) (cid:13) ∇ x (cid:101) f-], and get
[Equation]
Then Theorem 10 gives us
[Equation]
Now we proceed to bound each term
[Equation]
Term (E)
Term (E) ≤
[Equation]
[Equation]
where we used Theorem 6 in the last step
[Equation]
[Table]
[Equation]
Term (G)
For simplicity [Equation] denote m t :=
[Equation]
Since y ∗ ( · ) is κ -Lipschitz as in Theorem
[Equation]
| m t | can be upper bounded as | m t |
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table]
[Equation]
[-= t 0 + η t − 1 m t + T − 1 (cid:88) t = t 
 + ( η t − η t − 1 ) m t (cid:35)-][Table]
[Equation]
Also note that γ t and y t +1 does not depend on ξ xt , so E ξ xt [ m t ] = 0 . Next, we look at Term (G).
[Equation]
Summarizing all the results, we finish the proof.
Lemma
Under the same setting as Theorem 5, we have
[Equation]
Proof
E
[Equation]
≤
[Equation] For Term (H) [Equation] we will bound it using the same strategy as in [50].
The general idea is to show that
[Equation]
v y t +1 (cid:1) β − µη y 2 − ( v yt ) β is positive for only a constant number of times.
If the term is positive at iteration t , then we have
[Equation]
y
[Equation]
where in the last inequality we used Bernoulli’s inequality.
By rearranging the terms
[Equation]
we have the two following conditions
[Equation]
This indicates that at each time the term is positive, the gradient norm must be large enough and the accumulated gradient norm, i.e., v yt +1 , must be small enough.
Therefore, we can have at most
[Equation]
constant number of iterations when the term is positive.
When the term is positive, it is also upper bounded by using the result from Equation (10):
[Equation]
which is a constant.
In total, Term (H) is bounded by
[Equation]
Bringing it back, we get the desired result
[Equation]
Lemma
Under the same setting as Theorem [Equation] for any constant C
[Equation]
we have
[Equation]
Proof By Theorem 11 and Theorem
[Equation]
we have for any constant C ,
[Equation]
The first term can be bounded by Theorem 13.
For the second term [Equation] we have E
[Equation]
≤ η y E [Equation] T −
[Equation] [Equation] (cid:13)(cid:13)(cid:13) [Equation]
≤
[Equation]
where the last inequality follows from Theorem 6.
Then the proof is completed.
C.3. Proof of Theorem
We present a formal version of Theorem 5. Theorem 15 (stochastic setting) Under Assumptions 3.1 to 3.6, Algorithm 1 with stochastic gradient oracles satisfies that for any 0 < β < α < 1 , [-a desirable time-scale separation. This means that v yt +1 has to grow to nearly (1 /c ) 
-]after T iterations,
[Equation] and
E
[Equation]
Proof By smoothness of the primal function [Equation] we have
[Equation]
By multiplying 1 η t on both sides and taking the expectation w.r.t.
the noise of current iteration
[Equation]
we have
[Equation]
Summing over t = 0 to T − 1 , rearranging and taking total expectation, we get
E
[Equation]
Term (I) 2 E
[Equation]
− η
[Equation]
(cid:19)(cid:35)
[Equation]
η [Equation] − Φ [Equation]
η T
[Equation]
+ [Equation] t = Φ max (cid:18)η t − η t −(cid:19)(cid:35)
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
Term (K)
According to the smoothness of f ( x t , · )
[Equation]
we have
[Equation]
where the last inequality follows the strong-concavity of y .
Now we let
[Equation]
and apply Theorem
[Equation]
in total
[Equation]
we have
[Equation]
It remains to bound ( v zT ) m for z ∈ { x, y } and m ≥ 0 :
[Equation]
Bringing it back, we conclude our proof.
C.4. TiAda without Accessing Opponent’s Gradients
The effective stepsize of x requires the knowledge of gradients of y , i.e., v yt +1 .
At the end of
Section 3, we discussed the situation when such information is not available.
Now we formally introduce the algorithm and present the convergence result.
Theorem 16 (stochastic) Under Assumptions 3.1, 3.2, 3.4 and 3.5, Algorithm 2 with stochastic gradient oracles satisfies that for any 0 < β < α < 1 , after T iterations,
[-) 4: v xt +
-]
[-= v xt + ∥ g xt ∥ v yt +
-]
[Equation]
η
) β
≤ 2∆Φ G 2 α
[Equation]
and
[Equation]
Remark 17 The best rate achievable is (cid:101) O (cid:0) ϵ − 6 (cid:1) by choosing α = 1 / 2 and β = 1 / 3 .
Proof Theorems 10 and 13 can be directly used here because they do not have or expand the effective stepsize of x , i.e., η t .
This is also the case for the beginning part of Appendix C.3, the proof of Theorem 5, up to Equation (11).
However, we need to bound Terms (I), (J) and (K) in Equation (11) differently.
According to our assumption on bounded stochastic gradients, we know that v xT and v [-yt +
-]yT are both upper bounded by T G 2 , which we will use throughout the proof.
Term (I)
[Equation]
η
[Equation]
Term (J)
[Equation]
Term
(K)
According to the smoothness and strong-concavity of f ( x t
[Equation]
[Equation]
we have
E
[Equation]
To bound the RHS
[Equation]
we use Young’s inequality and have
[Equation]
Then applying Theorem 10 with λ t = µγ t 2 gives us
[Equation]
where the first term is O (1) according to Theorem 13.
The other two terms can be bounded as follow
[Equation]
Term (L)
[Equation]
Term (M)
[Equation]
E
[Equation]
[Equation]
where we used the the Lipschitzness of y ∗ ( · ) in the third inequality.
Summarizing all the terms, we finish the proof.