Online Min-max Optimization: Nonconvexity, Nonstationarity, and Dynamic Regret
Abstract
Online min-max optimization has recently gained considerable interest due to its rich applications to game theory, multi-agent reinforcement learning, online robust learning, etc.
Theoretical understanding in this field has been mainly focused on convex-concave settings.
Online min-max optimization with nonconvex geometries, which captures various online deep learning problems, has yet been studied so far.
In this paper, we make the first effort and investigate online nonconvexstrongly-concave min-max optimization in the nonstationary environment.
We first introduce a natural notion of dynamic Nash equilibrium (NE) regret, and then propose a novel algorithm coined
SODA to achieve the optimal regret.
We further generalize our study to the setting with stochastic first-order feedback, and show that a variation of SODA can also achieve the same optimal regret in expectation.
Our theoretical results and the superior performance of the proposed method are further validated by empirical experiments.
To our best knowledge, this is the first exploration of efficient online nonconvex min-max optimization.
1. Introduction
Online optimization [7] is a powerful paradigm for modeling many applications that require decision making based on information available sequentially.
Specially, at each time instance, an online player needs to make a decision based on the history information, and then receives a feedback (which can be a possibly adversarial and nonstationary reward or loss value) that may be used in the future.
There have been extensive studies in this field for various scenarios, such as online convex optimization [17, 36], online bilevel optimization [39], online federated learning [8] etc.
Recently, the online min-max (i e., saddle point) problem has gained considerable interest due to its broad applications in game theory [34, 43], multi-agent reinforcement learning [5, 42], online robust learning [3, 13], to name a few.
On the theoretical side, a line of works have explored provably efficient algorithms for online min-max optimization.
Specifically, [6, 12, 19, 44] considered the zero-sum matrix games where the online objective function takes a bilinear form.
[33, 34] studied a more general online minmax problem, where the objective is strongly-convex and strongly-concave.
[30] focused on multiobjective online min-max games, where the reward is convex-concave in each coordinate.
Despite many efforts so far, existing literature on online min-max optimization has mainly focused on online convex-concave problems and did not take nonconvexity into consideration.
However, in practice, nonconvexity occurs very often in online min-max problems, particularly those that apply deep neural networks (DNNs) for decision making.
For instance, in the time-varying two-player zero-sum stochastic games [28, 34, 44], where the payoffs change with time, the policies are modeled by DNNs with strong regularization, and hence the online objective function is nonconvex-strongly-concave.
Motivated by the aforementioned practical problems, the goal of this paper is to take the first step towards exploring the online nonconvex-strongly-concave min-max problem with dynamic (and hence nonstationary) loss functions.
Due to the nonconvexity and nonstationarity nature of the problem, two new challenges arise as we explain below.
First, how to define an appropriate notion of regret for the nonstationary environment under the online nonconvex setting?
The standard notion of Nash Equilibrium (NE)-regret, e.g., [33] for online convex-concave problems, which quantifies the difference between the cumulative loss of players and the min-max value of the cumulative payoff loss, is highly unreasonable for nonconvexconcave setting, since the min-max comparator is intractable for a nonconvex-concave function.
Hence, new surrogate for regret is in demand.
Second, with a desirable notion of regret, how to design efficient algorithms?
A natural strategy to handle the nonstationarity is that at each round, the decision maker first learns a good enough decision based on history losses and then applies it to the adversarial loss of current round.
Two key difficulties will arise during this process.
First, how to identify a good decision?
In nonconvex min-max problems, a good decision usually refers to a stationary point.
The standard definition of a stationary point involves an optimization oracle, which is unknown to the decision maker.
Thus the decision maker needs to find a surrogate to identify a near stationary point at each round.
Second, when applying the decision based on history information to the adversarial loss, mismatch errors arise due to variability of the environment, which motivates the need for nonstationarity measures.
1.1. Our contributions
In this paper, we handle the aforementioned challenges by introducing a new regret measure and developing efficient algorithms for online nonconvex min-max problem with optimal regret guarantees.
The main contributions are highlighted below.
• We first introduce a novel notion of dynamic regret for online nonconvex-strongly-concave min-max problem, called local Nash equilibrium (NE)-regret , which jointly captures the nonconvexity, nonstationarity, and min-max nature of our problem.
• Based on the regret notion, we propose an efficient online min-max optimization algorithm, named time-Smoothed Online gradient Descent Ascent (SODA).
The main idea underlying SODA is to output a near-stationary point at each round by performing two-timescale gradient descent ascent and utilizing a specially designed stop criterion component.
• We show that the local NE-regret of SODA scales as O ( Tw 2 ) with a iteration complexity of O ( T w ) , which matches the Ω( Tw 2 ) regret lower bound and the order of iteration complexity of O ( T w ) provided in [15] for online minimization (where we set the maximization to be over a singleton).
Thus, SODA achieves the optimal performance for online nonconvex-stronglyconcave min-max optimization.
• We further generalize our study to the setting with stochastic first-order feedback and show that a variation of SODA can also achieve a regret of O
( Tw 2 ) and defer the details to Appx.
C • We verify our theoretical results and demonstrate the effectiveness of our algorithm through several empirical experiments on real-world datasets and defer the details to Appx.
D. To our best knowledge, this is the first study on online nonconvex min-max optimization with theoretical characterization of the regret performance.
2. Problem Setup
We consider solving the following online min-max (i.e., saddle-point) problem:
[Equation]
where f t : R m × R n → R is generally nonconvex in x but concave in y and where Y is a convex set.
At each round t ∈ [ T ] , the environment first incurs a loss function f t .
Without knowing the knowledge of f t , the x -learner and y -learner are tasked with predicting x t and y t respectively to solve eq.
(1) based on loss functions up to round t − 1 , i.e., { f i } t − 1 i =1 .
The learners then observe the function f t ( · ) and suffer a loss of f t ( x t , y t ) .
The following regularity assumptions for f t are made throughout the entire paper:
Assumption 1 (Smoothness) f t is ℓ -smooth ∀ t ∈
[ T ] , i.e., ∀ ( x , y ) , ( x ′ , y ′ ) , it holds that ∥∇ f t ( x , y ) − ∇ f t ( x ′ , y ′ )
∥ ≤ ℓ ∥ ( x , y ) − ( x ′ , y ′ )
∥ .
Assumption 2 (Strong Concavity)
The function f t ( x , · ) is µ -strongly concave ∀ t ∈
[ T ] , i.e., given x ∈ R m , ∀ y , y ′ , it holds that f t ( x , y ) ≤ f t ( x , y ′ )
+ ⟨∇ y f t ( x , y ′ ) , y − y ′ ⟩ − µ 2 ∥
y − y ′
∥ 2 .
Assumption 3 (Boundness)
The set Y is a convex and bounded set with diameter D ≥ 0 .
There exists M > 0 , s.t.
| f t ( x , y ) | ≤ M , ∀ t ∈
[ T ] , x ∈ R m , y ∈ Y .
These assumptions are standard in literature of online [16] and min-max optimization [25, 26].
3. How to Measure the Performance? 3.1. Local Nash Equilibrium (NE)-Regret
We introduce a new definition of a local regret that suits online nonconvex-strongly-concave minmax problems.
Our new metric is motivated by the online nonconvex optimization literature; see for example [14, 15].
Specifically, for each t , we first define the smoothed functions of f t over a sliding-window of size w as:
[Equation]
For notation convenience, we treat f t ( x , y ) as 0 for all t < 0 .
Moreover, since the averaging preserves strongly-convexity, which implies F t,w is strongly-concave in y , the maximization problem max y ∈Y F t,w ( x , y ) can be solved efficiently.
Then, we can naturally define the following function:
[Equation]
The overall goal of online min-max optimization can be viewed as online minimization over the above defined Φ t,w ( · ) function.
Thus, we define the following regret metric with respect to Φ t,w ( · ) .
Definition 1 (Local Nash Equilibrium (NE)-Regret) Let { f t } t ∈
[ T ] satisfy Assumption 1-3, with Φ t,w ( · ) defined in eq.
The w -local Nash Equilibrium (NE)-Regret is defined as:
[Equation]
(4) ∇ Φ t,w is well-defined since Φ t,w is differentiable for nonconvex-strongly-concave min-max problem [25].
We next justify the above notion of the local NE-regret from three aspects 1 .
Why norm of gradient as metric?
In online convex-concave min-max optimization, it is standard to consider the Nash Equilibrium (NE)-Regret [33] metric, defined as: | (cid:80)
Tt =1 f t ( x t , y t )
− min x ∈ R m max y ∈Y (cid:80)
Tt =1 f t ( x , y ) | .
However, the above metric of NE-regret is inappropriate and face a major issue in the online nonconvex-concave formulation.
The core challenge is that, even in the offline case ( T = 1) , it is hard to efficiently find global optimum of min x ∈ R
m max y ∈Y f 1 ( x , y ) in hindsight.
Clearly, the problem is equivalent to min x ∈ R m Φ( x ) , where Φ( · ) = max y ∈Y f 1 ( · , y ) is generally nonconvex, and hence finding global minimum for Φ( x ) is NP hard.
A common surrogate for the global minimum of Φ in the offline nonconvex-strongly-concave min-max literature is the notion of ϵ -stationary point [25, 26] for differentiable Φ , i.e., there exists some iterate x t for which ∥∇ Φ( x t ) ∥ 2 < ϵ .
If ϵ = 0 , then x t is a stationary point.
Therefore, it is reasonable to leverage such a norm of gradient as the optimality criterion from the offline nonconvex min-max analysis.
Why sliding-window averaging?
The motivation behind window averaging is two-fold: ( i ) F t,w and Φ t,w represent average performance during the window, which is widely adopted to handle noises and fluctuations when the environment and the loss function
f t encounter mild perturbations and variations.
For instance, when each loss function f t is an unbiased noisy realization of some f , the expected gradient norm of a randomly selected update inside the window is a standard measure in nonconvex stochastic optimization literature [4] and can reduce the variation caused by noises.
Such smoothed notion is also a common practice in the field of online nonconvex optimization 2 [2, 14, 15, 45].
( ii )
The average performance itself is also a typical notion people are interested in real applications.
Suppose a decision maker in time-varying environment (with loss functions f t ) has only finite term memory w .
Then she naturally wishes to find the best decision based on the entire finite term memory and will choose the average loss function F t,w and Φ t,w as performance metrics.
3.2. Variability of Environment
Intuitively, if the environment (and hence the loss function f t ) changes drastically over time, it will be hard to obtain meaningful guarantees efficiently.
To handle this problem, dynamic [34, 44] or local [14] regret serves as better performance metrics to take the changing environment into consideration.
Such notions typically rely on certain nonstationarity measures of the environment in order to reflect how the system dynamics affects the performance.
Therefore, in this subsection, we introduce such measures of variation for loss functions, which will be crucial in our analysis and capture the nonstationarity of our online min-max settings.
Definition 2 (Variation of Sliding-window) Let us denote y ∗ t,w ( x ) = arg max y ∈Y F t,w ( x , y ) .
The sliding-window variation in x is defined as:
[Equation]
Moreover, the sliding-window variation in y is defined as:
[Equation]
[Equation]
Remark
Clearly, V x ,w [ T ] and V y ,w [ T ] are O ( T ) if the gradients of f t are bounded and can be zero in the offline setting, i.e., T = 1 .
A key observation is that if the loss function encounters a periodic shift with certain period length of w ∗ , i.e., f t + w ∗ =
f t , then for w = w ∗ , f t = f t − w and y ∗ t,w = y ∗ t − 1 ,w , which is implied by the fact that F t +1 ,w = F t,w .
As a consequence, for the well-tuned w , the sliding-window variations could be considerably small compared to T , especially
V x ,w [ T ] =
V y ,w [ T ] = 0 in the above case.
Input: window size w ≥ 1 , stepsizes ( η x , η y ) , tolerance δ >
Initialization: ( x 1 , y 1 ) 1: for t = 1 to T do 2:
Predict ( x t , y t ) .
Observe the cost function f t : R m × R n → R 3:
Set ( x t +1 , y t +1 )
← ( x t , y t ) 4: repeat 5: x t +1 ← x t +1 − η x ∇ x F t,w ( x t +1 , y t +1 ) 6: y t +1 ← P Y
( y t +1 + η y ∇ y F t,w ( x t +1 , y t +1 ))
7: until κ 2 η 2 y ∥
y t +1 − P Y ( y t +1 + η y ∇ y F t,w ( x t +1 , y t +1 ))
∥
+ ∥∇ x F t,w ( x t +1 , y t +1 )
∥ 2 ≤ δ 2 2 w8: end for
4. SODA: Time-Smoothed Online Gradient Descent Ascent
In this section, we present our proposed method, named time-Smoothed Online gradient Descent
Ascent (SODA), for online nonconvex-strongly-concave problem, and we show that our approach is capable of efficiently achieving a favorable local NE-regret bound.
4.1. Proposed Algorithm
At the high-level, our algorithm plays following the-leader iterates, aiming to find a suitable approximating stationary point at each round using two-timescale gradient descent ascent (GDA).
At each round t , SODA performs gradient descent over the variable x with the stepsize η x and gradient ascent over the variable y with the stepsize η y on function F t,w ( x , y ) until the stop condition is satisfied.
Then, SODA observes the loss function f t +1 to be used in the next round.
The pseudocode of SODA is summarized in Algorithm 1.
Discussions about stopping criterion.
Due to the online nature, the design of the stopping condition is to guarantee that the learner outputs a good x t +1 with small local regret at round t , i.e., ∥∇ Φ t,w ( x t +1 )
∥ 2 is small enough.
However, we do not have direct access to the first order oracle of Φ t,w .
To circumvent this issue, we adopt the global error bound condition from the seminal paper Drusvyatskiy and Lewis [11] to translate conditions on ∇ Φ t,w ( x t +1 ) into restrictions on tractable ∇ F t,w .
Specifically, we prove that ∥∇ Φ t,w ( x t +1 )
∥ 2 is upper bounded by the left-hand side of inequality in Algorithm 1 line 7 (see Theorem 8).
Therefore, alternatively we can utilize the accessible information of ∇ F t,w to terminate the inner loop iterations at time t .
Last-iterate guarantee.
At each round t , the stop condition will be triggered only when the local regret of last iteration is small enough.
Such a last-iterate type guarantee is different by nature from existing offline nonconvex-strongly-concave min-max results [25, 26], which are only guaranteed to visit an ϵ -stationary point within a certain number of iterations, i.e., where the return ¯ x is uniformly drawn from previous iterations.
Crucially, we will establish the total iteration bound (see Theorem 5) in the next subsection, which indicates that such last-iterate type outputs can be obtained efficiently.
Furthermore, since the stopping criterion leads to stronger guarantee, our result is incomparable with former offline iteration complexity in the special case that T = 1 .
4.2. Theoretical Guarantees
In this subsection, we provide the regret and computational complexity guarantees of our algorithm under local NE-regret and highlight several connections with the existing results from offline minmax optimization and online nonconvex problem.
Theorem 4 (Local NE regret minimization) Let κ = ℓ/µ denote the condition number.
Under
Assumptions 1-3, and letting the stepsizes be chosen as η x = Θ (cid:0) 1 /κ 3 ℓ (cid:1) and η y = Θ(1 /ℓ ) , then
Algorithm 1 enjoys the following local NE-regret bound:
[Equation]
Theorem 5 (Iteration bound) Let τ denote the total number of iterations incurred by Algorithm 1.
Then τ can be upper bounded as:
[Equation]
[Equation]
Theorems 4 and 5 together reveal trade-offs between the impact of size w on the regret and computational complexity, where larger w leads to smaller regret bound but incurs more gradient calls.
Robustness of SODA.
Our results in Theorems 4 and 5 are expressed in terms of variation measures
V x ,w [ T ] and V y ,w [ T ] of the environment introduced in Section 3.2.
If we make the more restrictive assumption similar to that in [15] that the gradient of f t is bounded, the above theorems provide a robust guarantee for SODA; namely, no matter how the environment changes at each round, SODA always ensures O ( Tw 2 ) local NE-regret with O ( T w ) iterations since V x ,w [ T ] and V y ,w [ T ] are O ( T ) by definitions.
Therefore, the regret can be made sublinear in T if w is selected accordingly.
Interestingly, following SODA, the local NE-regret can achieve the same order without the bounded gradient assumption depending on the nonstationarity.
Particularly, as we discussed in Theorem 3, for the scenario that f t is periodic with period w , V x ,w [ T ] =
V y ,w [ T ] = 0 .
Optimality of regret bound.
Note that the basic online nonconvex minimization problem can be viewed as a special case of our online nonconvex min-max problem, if f t ( x, y ) takes values independent of y .
In such a degenerate case, our local NE-regret is equivalent to the local regret analyzed in [14, 15].
Consequently, the adversarial example that incurs the local regret of Ω( Tw 2 ) constructed in [14] can also serve as a worst case example for our online noncovex min-max setting.
Moreover, under the same assumption made in [15] (which is more restrictive than our assumption here), we achieve a robust regret upper bound of O
( Tw 2 ) (as discussed in the previous paragraph), which matches the worst-case lower bound, indicating that our bound Theorem 4 for online nonconvex min-max problem is optimal.
Comparison to offline min-max optimization.
When the environment is fixed, i.e. f t ≡ f or T = 1 with w = 1 , our problem specializes to offline min-max optimization and V x ,w [ T ] =
V y ,w [ T ] =will disappear from our results.
Therefore, an immediate implication from our theorems is that
GDA is guaranteed to find ϵ -stationary point with O ( κ 3 ϵ − 2 ) iteration complexity.
The best known complexity bound for GDA in offline min-max optimization is O ( κ 2 ϵ − 2 ) [25].
However, as we discussed in Section 4.1, SODA aims to output x with last-iterate type guarantee, which is a stronger notion than that considered in [25], where GDA are only guaranteed to visit an ϵ -stationary point within a certain number of iterations.
Thus, these results are not directly comparable.
5. Conclusions
This paper provides the first analysis for the online nonconvex-concave min-max optimization problem.
We introduced a novel notion of local Nash Equilibrium regret to capture the nonconvexity and nonstationary of the environment.
We developed and analyzed algorithms SODA and its stochastic version with respect to the proposed notions of regret, establishing favorable regret and complexity guarantees.
Furthermore, we conduct experiments with real-world data to validate the theoretical statements and show its superiority in practice.
Appendix A Missing Discussions and Related Work
A.1 Why capturing the dynamic nature? . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B Missing Proof of Section
B.1 Technical Lemma . .
B.2 Local Regret: Proof of Theorem 4 . . . . . . . . . . . . . . . . . . . . . . . . . .
B.3 Oracle Queries:
Proof of Theorem 5 . . . . . . . . . . . . . . . . . . . . . . . . .
B.4 Proof of Theorem 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C SODA with Stochastic First-order Oracle
C.
Proposed Algorithm . .
C.2 Theoretical Guarantees . .
C.3 Missing Proof of Appx.
C.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D Experiments
D.1 Implementation Details . .
D.2 Results . .
Notation [ T ] ≜ { 1 , . .
, T } .
We use bold lower-case letters to denote vectors as in x , y , and denote its ℓ 2 -norm as ∥ · ∥ .
We use calligraphic upper case letters to denote sets as in Y , and use the notation P Y to denote projections onto the set.
For a differentiable function Φ( · ) : R m → R , we let ∇ Φ( x ) denote the gradient of Φ at x .
For a function f ( · , · ) : R m × Y → R of two variables, ∇ x f ( x , y ) (or ∇ y f ( x , y )) denotes the partial gradient of f with respect to the first variable (or the second variable) at the point ( x , y ) .
We also use ∇ f ( x , y ) to denote the full gradient at ( x , y ) where ∇ f ( x , y )
=
( ∇ x f ( x , y ) , ∇ y f ( x , y )) .
Finally, we use the notation O ( · ) and Ω( · ) to hide constant factors which are independent of problem parameters.
Appendix A. Missing Discussions and Related Work A.1. Why capturing the dynamic nature?
It is desirable that the regret can capture how well the players adapt their actions to the best decision at each round if the environment is nonstationary and changes over time.
In the well-studied online convex-concave setting, the notion of dynamic regret [44] is defined for this purpose, since its definition of | (cid:80) Tt =1 f t ( x t , y t ) − (cid:80)
Tt =1 min x ∈ R m max y ∈Y f t ( x , y ) | evaluates the gap to the minmax comparator at each round instead of the min-max solution of the sum of functions over all rounds.
For the nonconvex min-max setting, the best min-max comparator at each round can be set as the stationary point of the window function Φ t,w ( · ) , which has zero gradient.
Hence, our local regret in eq.
(4) can be interpreted as evaluating the gap between ∥∇ Φ t,w ( x t ) ∥ 2 and its comparator (which equals zero gradient) at each round , and thus implicitly captures the player’s adaption to the dynamic setting.
A.2. Related Work
Online min-max optimization.
Recently, online min-max optimization, also known as online saddle-point game, has emerged as an interesting optimization framework, and has been studied under various settings.
More specifically, the zero-sum matrix game considers the special case that the function is bilinear with a payoff matrix A t , where the objective function is given by f t ( x , y ) =
x ⊤ A t y t .
Several works, for example, [6, 12, 19, 44] proposed and analyzed algorithms with respect to different notions of regret.
For more general objective functions, [33, 34] studied the case where the loss function f t is strongly-convex-strongly-concave.
Very recently, [30] formulated a general multi-objective framework, where the goal is to minimize the maximum coordinate of the cumulative vector-valued loss with convex-concave function in every coordinate.
We emphasize that all of the above studies did not consider nonconvexity in their objective functions, which is the focus of our study here.
Online nonconvex optimization.
As online nonconvex optimization is an active research area, various works have taken different approaches to handle the nonconvexity.
Assuming access to an offline nonconvex optimization oracle to approximate minimizers of perturbed nonconvex functions, [1, 38] studied the performance of “follow the perturbed leader” (FTPL) algorithm [21], and their regrets are all static regret.
Further, [2, 14, 15] considered online nonconvex problems under nonstationary environments, and utilized sliding windows method with window size w .
They proposed different notions of dynamic regrets and algorithms, and achieved an order of O ( Tw 2 ) according regret notions.
Additionally, [18] studied online nonconvex optimization with imperfect feedback.
Except first-order optimization, [18, 35] considered zeroth-order online nonconvex optimization and [23] studied second-order online nonconvex optimization.
Offline min-max optimization.
There is a rich literature that studies a diverse set of algorithms for min-max optimization with nonconvexity in the offline setting.
We next describe only those studies highly relevant to our study here.
One celebrated approach is the nested-loop type algorithm [22, 31, 32, 40], where the outer loop can be treated as an inexact gradient descent on a nonconvex function while the inner loop aims to find an approximate solution to the maximization problem (see [25] and references therein for a good collection of such studies).
Another approach, manifesting in the recent works of [27] and [25] considers less complicated single-loop structures.
Specifically, the two-timescale GDA analyzed in [25] is closest to the implementation at each round of our proposed
SODA method.
But it is not straightforward to generalize the design to the online setting, and our analysis of the new local NE-regret for online optimization is also very different from such a offline min-max problem.
Appendix B. Missing Proof of SectionB.1. Technical Lemma
Recall that Φ t,w ( x ) = max y ∈Y F t,w ( x , y ) and y ∗ t,w ( x ) =
arg max y ∈Y F t,w ( x , y ) .
In this section, we first present some technical lemmas to characterize the structure of the function Φ t,w and y ∗ t,w in the nonconvex-strongly-concave setting, which will be essential throughout the analysis.
Lemma 6 Φ t,w ( · ) is ( ℓ + κℓ ) -smooth with ∇ Φ t,w ( · ) = ∇ x F t,w (cid:0) · , y ⋆t,w ( · ) (cid:1) .
Also, y ⋆t,w ( · ) is κLipschitz.
Proof Since the averaging maintains the strongly-nonconcavity and smoothness,
i
[Equation]
e. F t,w ( x , y ) is still µ -strongly-convave in y and ℓ -smooth.
Thus, the proof directly follow Lemma 4.3 in [25] and we omit the details
[Equation]
Furthermore, we derive the following lemma to provide the smoothness property of y ∗ t,w ( x ) with respect to t .
In other words, given any fixed x , the movement of y ∗ t,w ( x ) when t changes can be controlled by the variability of environment of the sliding window.
Lemma
For any x ∈ R m , t ∈ [ T ] , it holds that
[Equation]
Proof By the optimality of y ∗ t
[Equation]
w ( x ) and y ∗ t − 1 ,w ( x ) , for ∀ x , we have
[Equation]
Summing up Equation (9) with y =
y ∗ t − 1 ,w ( x ) and Equation (10) with y =
y ∗ t,w ( x ) yields that
[Equation]
[Equation]
By the definition of F t,w ( x , y ) , we have
[Equation]
Since for any t and fixed x
[Equation]
the f t ( x , · ) is µ -strongly-concave
[Equation]
we have
[Equation]
Plug Equations (12) and (13) into Equation (11), then we have
[Equation]
As a result
[Equation]
where the last inequality follows from Cauchy-Schwartz inequality.
Finally, by some algebra manipulation
[Equation]
we finish the proof as following
[Equation]
The next lemma provides an upper bound for the gradient norm of ∇ Φ t,w in term of notions about ∇ F t,w
[Equation]
which justifies our design of stop conditions
[Equation]
Lemma
Given a pair ( x , y ) ∈
R m × Y , for t ∈ [ T ] and w > 0 , it holds that
[Equation]
Proof By Cauchy-Schwartz inequality, we have
[Equation]
where the last inequality holds by combining Theorem 6 and the fact that F t,w is ℓ -smooth.
Since
F t,w ( x , · ) is µ -strongly-concave over Y , from the global error bound condition in [11], we obtain
[Equation]
Thus, we complete the proof.
B.2. Local Regret: Proof of Theorem
Proof [Proof of Theorem 4] Recall the definition of Φ t,w and notice that
[Equation]
Then
[Equation]
14) where the second term in last inequality follows from that
[Equation]
where ( a ) is implied by Theorem 7.
Moreover, for the first term in Equation (14), by Theorem 8, and the stop condition, we obtain
[Equation]
Summing over t = 1 , · · · , T , and combining the definition of variation measures V x ,w and V y ,w , then we have
[Equation]
B.3. Oracle Queries: Proof of Theorem
Denote the sequence generated in the inner loop at time t ∈
[ T ] by
[Equation]
Let τ t be the number of times the gradient update is executed at the t -th iteration.
Note that x
τt = x
t +1 and y
τt =
y t +1 .
B.3.1. S UPPORTING L EMMAS
We present three key lemmas which are important step descent lemmas.
In this section, we focus on a crucial quantity, δ kt,w =
(cid:13)(cid:13)
y ⋆t,w (cid:0) x kt (cid:1) − y kt
(cid:13)(cid:13) 2 , which are useful for the subsequent analysis.
Throughout our analysis, we choose η x = 1 8 κ 3 ℓ and η y = 1 ℓ .
Lemma 9 Denote τ t the total iteration of inner loop at step t , for 0 ≤ k ≤ τ t −
[Equation]
Proof Since Φ t,w is ( ℓ + κℓ ) -smooth and ℓ + κℓ ≤ 2 κℓ , for any x, x + ∈ R m , we have
[Equation]
Φ
[Equation]
−
[Equation]
Plugging x + − x = − η x ∇ x F t,w ( x , y ) yields that
[Equation]
,w ( x , y )
By Young’s inequality, we have
[Equation]
Since ∇ Φ t,w ( x ) = ∇ x F t,w (cid:0) x , y ∗ t,w ( x ) (cid:1) , we have
[Equation]
Putting these pieces together, we obtain
[Equation]
Lemma
For any t, k ≥ 0 , the following statement holds true,
[Equation]
Proof By Young’s inequality, we have
[Equation]
Lemma 11 Let δ kt,w =
(cid:13)(cid:13)
y ⋆t,w (cid:0) x kt (cid:1) − y kt
(cid:13)(cid:13) 2 , the following statement holds true,
[Equation]
Proof Since f t ( x , · ) is µ -strongly concave and η y = 1 /ℓ
we have
[Equation]
By Young’s inequality, we have
[Equation]
Since y ⋆t,w ( · ) is κ -Lipschitz, we have
[Equation]
Thus, plug into eq.
[Equation]
B.4. Proof of Theorem
Proof [Proof of Theorem 5]
Denote γ = 1 − 1 2 κ , from Theorem 11 and using telescoping we have
[Equation]
Specially, for t > 1 ,
[Equation]
Then plug Equation (18) into Equations (15) and (16) from Theorems 9 and 10, and sum over outer loop number.
[Equation]
Letting η x = 1 8 κ
[Equation]
we have
[Equation]
Therefore add Equation (19) × η x 8 and Equation (20) ×
η x 10 we have η x
[Equation]
Denote Φ 0 ,w ( x ) = 0 , we notice that Φ T,
[Equation]
=
[Equation]
w
[Equation] [Equation]
≤w
[Equation]
t
[Equation]
where ( i ) follows from that y ∗ t − 1 ,w ( x t ) is the maximizer of F t − 1 ,w ( x t , · ) .
By some algebra, we have
[Equation]
Sum Equation (21) over t , we have
[Equation]
Hence
[Equation]
Appendix C. SODA with Stochastic First-order Oracle
In this section
[Equation]
we extend our online min-max framework to an online stochastic version.
This setting is motivated by the fact that, in real world application, such as training a neural network, an oracle with access to the gradient of loss function is hard to reach.
Instead, a stochastic first-order oracle (SFO) is used to approximate the ground truth gradient.
Similar settings have been studied in [14, 15, 29].
Specifically, the formal SFO definition is as follows.
Definition 12 (Stochastic first-order oracle)
A stochastic first-order oracle (SFO) is a function
S σ such that, given a point ( x
, y ) ∈
R m ×Y , a random seed ζ , and a smooth function h : R m ×Y →
R satisfies:
• S σ ( x , y ; ζ, h ) is an unbiased estimate of ∇ h ( x , y ) :
E ( S ( x , y ; ζ, h ) − ∇ h ( x , y ))
= 0 ;
• S σ ( x , y ; ζ, h ) has variance bounded by σ 2 > 0 : E (cid:16) ∥S ( x , y ; ζ, h ) − ∇ h ( x , y ) ∥ 2 (cid:17) ≤ σ 2 .
C.1. Proposed Algorithm
With the above definition of SFO, we introduce the stochastic version of Algorithm 1, named
SODA-SFO (see Algorithm 2).
Similarly, SODA-SFO also follows the-leader iterates using twotime scale GDA.
Taking the noise brought by SFO into consideration, nested loops and special stopping criterion (in line 6 in Algorithm 2) are modified accordingly.
Specially, ( i ) SFO results
Algorithm
Input: window size w ≥ 1 , stepsizes ( η x , η y ) , tolerance δ >
Initialization: ( x 1 , y 1 ) 1: for t = 1 to T do 2:
Cost function f t : R m × R n → R is updated; 3:
Sample ˜ ∇ f t ( x t , y t ) ← S σ/w ( x t , y t ; ζ, f t ) 4:
Set ˜ ∇ F t,w ( x t , y t ) =
˜ ∇ F t − 1 ,w ( x t , y t ) + 1 w ( ˜ ∇ f t − w ( x t , y t ) − ˜ ∇ f t ( x t , y t )) 5:
Set x 0 t = x t , y 0 t =
y t , G 0 x ,t = ˜ ∇ x F t,w ( x t , y t ) , G 0 y ,t =
˜ ∇ y F t,w ( x t , y t ) , k =6: while 2 κ 2 η 2 y ∥ y kt − P Y (cid:0) y kt + η y G k y ,t (cid:1) ∥ 2 + ∥ G k x ,t ∥ 2 > δ 2 / 3 w 2 do 7: x k +1 t ← x kt − η x G k x ,t 8: y k +1 t ← P Y (cid:0) y kt + η y G k y ,t (cid:1) 9:
Sample ˜ ∇ f i ( x k +1 t , y k +1 t ) ← S σw ( x k +1 t , y k +1 t ; ζ, f i ) for i = t − w + 1 , · · · , t ; 10:
Set G k +1 t :=
( G k +1 t, x , G k +1 t, y )
= 1 w (cid:80) t i =
t − w +1 ˜ ∇ f i ( x k +1 t , y k +1 t ) 11: k ← k +12: end while 13: x t +1 = x kt , y t +1 = y kt , and ˜ ∇ F t,w ( x t +1 , y t +1 )
=
G kt 14: end for in different coefficients in stop criterion compared to SODA.
( ii )
The stopping criterion in SODASFO only ensures that ∥∇ Φ t,w ( x t +1 )
∥ 2 is bounded by the threshold plus the variation of SFO.
But the variation here does not play an important role, since sliding windows serve variance reduction purpose to reduce the variation in the final expectation regret.
C.2. Theoretical Guarantees
Denote τ t as the number of iterations of inner loop at round t and thus τ = (cid:80) Tt =1 τ t .
We first establish that for each round t , the inner loop terminates with finite iterations τ t provided that δ is not too small (recall that δ is the tolerance for stopping criterion), which justifies that SODA-SFO is computation tractable.
Theorem 13 (Finite iteration with SFO) Let κ = ℓ/µ denote the condition number, and let the stepsizes be chosen as η x = Θ (cid:0) 1 /κ 3 ℓ (cid:1) and η y = Θ(1 /ℓ ) .
Under Assumptions 1-3, for any t ∈ [ T ] , if δ, w and σ satisfy that δ 2 =
O ( κ 4 ℓ 2 σ 2 w ) , then τ t and τ is finite with high probability.
Specially, when K ∈ R is large enough, P ( τ t > K ) =
O (1 /K ) .
With the finite step stopping guarantee on hand, we next characterize the performance of SODASFO with expectation local NE-regret formally in terms of w, T, V x ,w [ T ] , V y ,w [ T ] .
Theorem 14 (Expectation local NE-regret with SFO)
Under the setting of Theorem 13, SODASFO enjoys the following expectation local NE regret bound:
[Equation]
,w [ T ] .
Beyond finite stopping and the regret bound, people may wonder whether the inner loop is meaningful if the per-stage calls of SFOs increase greatly, and are also interested in the total complexity of SFO calls.
To address such an issue, we further provide an upper bound on the complexity of SFO calls similar to Theorem 5.
In the stochastic online nonconvex min-max setting, we further need the following widely adopted assumption [14, 24] to control the noise caused by SFO calls.
Assumption 4 Given any point ( x
, y ) ∈
R m ×Y , random seed ζ , and smooth function h : R m ×Y , the SFO defined in Theorem 12 satisfys
that ∥S ( x , y ; ζ, h ) − ∇ h
x , y ) ∥ 2 ≤ σ 2 .
Remark 15 We remark here that Theorems 13 and 14 do not require Assumption 4, and Theorem 13 provide the finite iteration guarantee with high probability and Theorem 14 provides an upper bound for expectation regret.
With Assumption 4, which is slightly stronger than the assumptions in Theorem 12, we are able to provide the following deterministic bound on iterations and the number of SFO calls as in Theorem 16.
Furthermore, Theorem 16 can provide deterministic guarantees rather than high probability guarantees because Assumption 4 controls the variation of noise in an absolute and deterministic manner.
Theorem 16 (Iterations and SFO calls bounds)
Under the setting of Theorem 13 and Assumption 4, and suppose that δ 2 > 540 κ 4 σ 2 .
Then the total number of iterations satisfies
[Equation]
Furthermore, the number of SFO calls is bounded by O ( wτ ) .
The above results also provide a robust guarantee for SODA-SFO, where SODA-SFO achieves a expectation regret of O
( Tw 2 ) with at most O ( T w ) iterations and hence O ( T w 2 ) calls of SFO, as long as V x ,w [ T ] and V y ,w [ T ] scale with O ( T ) .
Following the similar discussions from Theoremand Section 4.2, such condition can hold with relaxed assumptions depending on nonstationarity.
Specially, if the variance of SFO defined in Theorem 12 is zero, then SFO reduces to perfect first order feedback.
Hence, as discussed in Section 4.2, the adversarial example provided by [15] is also applicable to the stochastic setting, and thus indicates that the expectation regret O ( Tw 2 ) reaches optimality.
If the set Y is a singleton, online nonconvex min-max problem with SFO reduces to the online nonconvex problem with SFO.
In this case, the term consisting of V y ,w [ T ] will disappear in our analysis, and our theorems recover the results in [14].
C.3. Missing Proof of Appx. C.
In this section, we first make some notation clearly here, G k
[Equation]
1 t
[Equation]
˜ ∇ F t,
[Equation]
x k +1 t
[Equation]
[Equation]
[Equation]
And for simplification, we denote y kt
=
y τ t t for any k ≥ τ t .
Before our theoretical analysis of Algorithm 2 and proof of Appx.
C.2, we define the filtration in
Algorithm 2 formally to describe clearly what is known and what is unknown at certain stage.
Definition 17 (Filtration) For any t ≥ 1 , we denote filtration F t to be the σ -fields that corresponds to the randomness of all gradient feedback up to stage t − 1 and the decision of f t at stage t .
In particular, F t includes f t , x t and ˜ ∇ F t − 1 ,w ( x t , y t ) , but doesn’t include ˜ ∇ f t ( x t , y t ) , ˜ ∇ F t,w ( x t , y t ) .
For any t ≥ 1 , k ≥ 1 , we denote filtration F kt to be the σ -fields that corresponds to the randomness of all gradient feedback up to the k -th iteration in line 6 at stage t in Algorithm 2.
In particular,
F kt includes f t , x kt , y kt , ˜ ∇ F t,w ( x t , y t ) , { ˜ ∇ f i ( x k − 1 t , y k − 1 t ) } ti = t − w +
and G k − 1 t , but doesn’t include
G kt , { ˜ ∇ f i ( x kt , y kt ) } ti =
t − w +1 .
C.3.1. S UPPORTING L EMMAS
Generally speaking, the lemmas in this section extends lemmas in Appx.
B.3.1 to noisy setting.
We first provide a descend lemma for Φ t,w ( x ) in each iteration of inner loop.
Lemma
Denote τ t the total iteration of inner loop at stage t and δ kt,w =
(cid:13)(cid:13)
y ⋆t,w (cid:0) x kt (cid:1) − y kt
(cid:13)(cid:13) 2 , for 0 ≤ k ≤ τ t −
[Equation]
k +
[Equation]
Proof Since Φ t,w is ( ℓ + κℓ ) -smooth, for any x, x + ∈ R m , we have Φ t,w
[Equation]
Set x + = x k +1 t , x = x kt , we have x +
[Equation]
x = x k +
[Equation]
, which yeilds that
[Equation]
By Young’s inequality
[Equation]
we have
[Equation]
∇
[Equation] [Equation]
Since ∇ Φ t,w (cid:0) x kt (cid:1) = ∇ x F t,w (cid:0) x kt , y ∗ t,w ( x kt ) (cid:1) , we have ∥∇
[Equation]
Putting Equations (22) to (24) together, we obtain Φ t,
[Equation]
The next lemma characterizes the descent property of distance to the maximizer y ∗ t,w .
Lemma 19 Let δ kt,w =
(cid:13)(cid:13)
y ⋆t,w (cid:0) x kt
[Equation]
− y kt (cid:13)(cid:13) 2 , the following statement holds true,
[Equation]
Proof Since f ( x , · ) is µ -strongly concave and η y = 1 /ℓ
[Equation]
we have
[Equation]
By Young’s inequality, we have
[Equation]
Since y ⋆t,w ( · ) is κ -Lipschitz
[Equation]
we have
[Equation]
Thus, plug into
[Equation]
The next lemma shows that updates over y can be controlled by δ kt,w plus a noisy term
[Equation]
Lemma
For any t, k ≥ 0 , the following statement holds true,
[Equation]
Proof By Young’s inequality
[Equation]
we have
[Equation]
where ( i ) follows from Equation (25)
[Equation]
C.3.2. F INITE I TERATION : P ROOF OF T HEOREM
Proof [Proof of Theorem 13] From Theorem
[Equation]
Denote γ = 1 − 1 4 κ , Given F k − 1 t we have
[Equation]

[Equation] [Equation]
where the first term of ( i ) follows from that Y is bounded with D , and the second term of ( i ) follows from the stopping criterion of Algorithm 2 and (cid:80)
k − 1 j =0 γ k − 1 − j ≤ 4 κ .
Notice that for any fixed t, k and j ∈ [ k − 1] ,
[Equation]
where ( i ) follows from the property of conditional expectation, ( ii ) follows from that the SFO calls in line 9 of Algorithm 2 is independent and ( iii ) follows from definition of SFO and filtration F jt .
Thus take expectation over two sides of Equation (26)
[Equation]
we have
[Equation]
Then by Theorem
[Equation]
By Theorem
[Equation]
Sum Equation (29) and Equation (30)
[Equation]
we have
[Equation]
Rearranging the term, we have
[Equation]
Take expectation over both sides of Equation (31), plug into Equation (28) and follow from the similar step of Equation (27)
[Equation]
we have
[Equation]
Because γ = 1 − 14 κ ≤
[Equation]
there exist a constant ˜ K such that γ ˜ K D 2 ≤ max (cid:110) 32 κ 4 η 2 x 3 w 2 , 8 κ 2 σ 2 ℓ 2 w 3 (cid:111) .
Thus for k ≥ ˜ K
[Equation]
we have
[Equation]
x k +1 t (cid:17)(cid:105) ≥
η x δ32 w 2 − 5 κ 2 ℓ
[Equation]
η x (cid:18) 35 κ 4 η 2 x δ3 w+ 9 κ 2 σℓ 2 w(cid:19) − (cid:18)4 ℓ +(cid:19) σw≥ 25 η x δ256 w 2 − 45 κ 4 η x σw− (cid:18)4 ℓ +(cid:19) σw 3 .
when δ 2 > 2304 κ 4 σ 2 5 w + 256(4 ℓ +1) σ 2 25 η x w , we set α = 25 η x δ 2 256 w 2 − 45 κ 4 η x σ 2 w− (cid:0) 154 ℓ + 1 (cid:1) σ 2 w 3 > 0 .
Then for
K ≥ ˜ K , we have
[Equation]
=
[Equation]
≥ α
K
[Equation]
k =
[Equation]
where the third equation follows from the Optional Stopping Theorem.
Consequently, we have τ t is finite in probability, which implies that τ = (cid:80) Tt =1 τ t is finite in probability since it is the finite sum of finite variables in probability.
C.3.3. L OCAL R EGRET : P ROOF OF T HEOREM
Proof [Proof of Theorem 14] Following from Equation (14), we have
[Equation]
For the first term
[Equation]
Consider (cid:13)(cid:13)
y ⋆t − 1 ( x τ t − 1 t − 1 ) − y τ t − 1 t − 1 (cid:13)(cid:13)(cid:13)(cid:13)
y ⋆t − 1 ( x τ t
[Equation]
1 t − 1 ) − y τ t − 1 t
[Equation]
1 (cid:13)(cid:13)( i ) ≤ κ 2 · η 2 y ℓ 2 (cid:13)(cid:13)
y τ t − 1 t − 1 − P Y (cid:0) y t + η y ∇ y F t
[Equation]
1 ,w ( x
[Equation]
1 , y τ
[Equation]
− 1 ) (cid:1)(cid:13)(cid:13)≤ 2 κ 2 · η
[Equation]
y ℓ 2 (cid:13)(cid:13)(cid:13)
y τ t
[Equation]
t
[Equation]
1 − P Y (cid:16) y
[Equation]
+ η y ˜ ∇ y F t −
[Equation]
w (
[Equation]
t
[Equation]
1 t −
[Equation]
y τ t − 1 t − 1 ) (cid:17)(cid:13)(cid:13)(cid:13)
[Equation]
where ( i ) follows from the global error bound condition in [9] and ( ii ) follows from the project operator is a contraction.
Then
[Equation]
where ( i ) follows from the stopping condition of inner loop and η y = 1 /ℓ .
Plug Equation (33) into Equation (32) and sum over t , we have
[Equation]
Notice that for any t ∈ [ T ] ,
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where ( i ) follows from the property of conditional expectation
[Equation]
( ii ) follows from that the SFO calls in line 9 of Algorithm 2 is independent and ( iii ) follows from definition of SFO.
Similarly, for any t
[Equation]
we have
[Equation]
Plug Equations (35) and (36) into Equation (34)
[Equation]
we have
E [
[Equation]
w ( T )] =
[Equation]
C.3.4. I TERATION AND SFO C ALLS B OUND : P ROOF OF T HEOREM
Proof [Proof of Theorem 16] From Theorem
[Equation]
Denote γ = 1 − 14 κ , Given F t we have
[Equation]
Then by Theorem
[Equation]
Then plugging Equation (37) into Equation (38) and summing up them over k = 0 , . .
, τ t − 1 , we have
[Equation]
where the last inequality follows from that (cid:80)
τ t − 1 k =
[Equation]
γ k = 1 − γ τt 1 −
[Equation]
≤ 4 κ and changing the order of summation over j and k .
Rearranging the terms, we have
[Equation]
x t ) − Φ t,w ( x t +1 )
+ 4 κη x ℓ 2 δ 0 t
[Equation]
τ t −
[Equation]
k = (cid:13)(cid:13)(cid:13)
[Equation]
t
[Equation]
(cid:33)
[Equation]
By Theorem
[Equation]
Then
[Equation] [Equation]
∇
[Equation]
f
[Equation]
Letting η x = 1 32 κ 3 ℓ , we have
[Equation]
Therefore add Equation (39) × η x 8 and Equation (40) × η x 9 , we have η x
[Equation]
Denote Φ 0 ,w ( x ) = 0 , we notice that
[Equation]
where ( i ) follows from that y ∗ t − 1 ,w ( x t ) is the maximizer of F t − 1 ,w ( x t , · )
[Equation]
By some algebra
[Equation]
we have
T (cid:88)
[Equation]
Sum Equation (41) over t and take expectation, we have
[Equation] [Equation]
T
[Equation] [Equation]
4 ℓ
[Equation]
+ µ 2 ( w −
[Equation]
where the first inequality follows from Assumption
[Equation]
Thus
[Equation]
Appendix D. Experiments
In this section, we evaluate the efficiency of the proposed SODA algorithm and verify the theoretical results through numerical simulations.
We consider the min-max problem of training an empirical
Wasserstein robustness model (WRM) [37], which has the following form 3 :
[Equation]
where ℓ is the cross-entropy loss function, N is the number of training samples, x is the network parameter, ( ξ i , z i )
∈ D corresponds to the i -th data sample and label, respectively, and y i is the adversarial sample corresponding to ξ i .
Denote { y i } Ni =1 as y .
We simulate the online WRM model as follows.
We randomly split the given dataset into T pieces {D t } Tt =1 , and the learner sequentially receives D t .
At each round t , f t ( x , y ) =
L ( x , y ; D t ) .
D.1. Implementation Details
The real-world datasets we consider are MNIST [10] and Fashion-MNIST [41], each containing 60k samples.
We choose T = 100 for the online setting.
The network architecture mainly follows [37], which consists of three convolution blocks with filters of size 8 × 8 , 6 × 6 and 5 × 5 respectively activated by ELU function, then followed by a fully connected layer and softmax output.
Furthermore, we set the adversarial perturbation γ ∈ { 0 . 4 , 1 . 3 } , which is consistent with [37].
For the results presented in Figure 1 to investigate the effect of window size, the two-timescale learning rates are set to η x = 5 × 10 − 3 , η y = 0 .
01 for MNIST and η x = 10 − 4 , η y =
× 10 −for Fashion-MNIST.
Moreover, considering experiments in Figure 2, which compared SODA and onlineGDmax, for onlineGDmax, the learning rates are set to η x = η y = 5 × 10 − 3 for MNIST and η x = η y = 10 − 3 for Fashion-MNIST; And for SODA, the learning rates are set to η x = 10 − 3 , η y = 5 × 10 − 3 for MNIST and η x = 10 − 4 , η y = 5 × 10 − 3 for Fashion-MNIST.
The window size is 10 .
Metrics.
Since we do not have access to the first-order oracle of ∇ Φ t,w in practice, two alternative performance metrics are considered, which capture the essence of the online setting and are consistent with the definition of our local NE-regret.
The first metric is the stronger notion we utilize in the stop criterion, which provides an upper bound for ∥∇ Φ t,w ( x t ) ∥ 2 .
Observing that the projected gradient of y does not change significantly in experiments, we only compute ∥∇ x F t,w ( x t , y t )
∥and report the average R avg ≜ 1 t (cid:80) tj =1 ∥∇ x F j,w ( x j , y j ) ∥ 2 of these at each round t , which serves as an approximation of 1 t ℜ w − NE ( t ) .
The second metric is the average accuracy, where we evaluate the test accuracy of output ( x t , y t ) from the last round on the newly coming D t and report the average from round 1 to t .
[Figure] [Figure]
D
[Figure]
Algorithm 3 OnlineGDmax
Input: window size w ≥ 1 , stepsizes ( η x , η y ) , tolerance δ > 0 , ascent step K
Initialization: ( x 1 , y 1 ) 1: for t = 1 to T do 2:
Predict ( x t , y t ) .
Observe the cost function f t : R m × R n → R 3:
Set ( x t +1 , y t +1 )
← ( x t , y t ) 4: repeat 5: x t +1 ← x t +1 − η x ∇ x F t,w ( x t +1 , y t +1 ) 6: k =7: while k < K do 8: y t +1 ← P Y
( y t +1 + η y ∇ y F t,w ( x t +1 , y t +1 ))
9: k = k +10: end
while 11: until κ 2 η 2 y ∥ y t +1 − P Y ( y t +1 + η y ∇ y F t,w ( x t +1 , y t +1 ))
∥
+ ∥∇ x F t,w ( x t +1 , y t +1 )
∥ 2 ≤ δ 2 2 w12: end for
The effect of window size w .
In Figure 1, we plot R avg of SODA on MNIST and Fashion-MNIST with different w .
It can be observed that as w increases from 2 to 10 , the local regret becomes smaller, which verifies the bound in Theorem 4 and justifies the usage of large window size.
Comparison of SODA and baseline algorithm.
To further investigate the performance of SODA, we conduct experiments to compare it with a baseline algorithm.
Note that to our best knowledge, there has been no existing formal studies on the performance of any developed algorithm for online nonconvex min-max problems.
Here, we consider a baseline algorithm, which is a natural extension of the well-known offline min-max method GDmax [20] to the online framework, named onlineGDmax .
Specifically, onlineGDmax replaces the inner-loop procedure of SODA by the nested-loop GDmax, i.e., at each iteration in the inner loop of round t , onlineGDmax will firstly maximize the function by multi-step gradient ascent for y , which is 10 steps in our setting, then perform one-step GD for x .
Typically, the stepsizes for GDmax are chosen to be equal, i.e. η x , η y [37].
OnlineGDmax is summarized in Algorithm 3.
As shown in Figure 2, to achieve similar accuracy, onlineGDmax requires significantly larger number of gradient calls than SODA, which demonstrates the efficiency of our algorithm.