Original: M INIMAX O PTIMIZATION WITH S MOOTH A LGORITHMIC A DVERSARIES
Revised: M INIMAX O PTIMIZATION WITH S MOOTH A LGORITHMIC A DVERSARIES

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper considers minimax optimization min x max y f
Revised: This paper considers minimax optimization min x max y f

Original: ( x, y ) in the challenging setting where f can be both nonconvex in x and nonconcave in y .
Revised: ( x, y ) in the challenging setting where f can be both nonconvex in x and nonconcave in y .

Original: Though such optimization problems arise in many machine learning paradigms including training generative adversarial networks (GANs) and adversarially robust models, from a theoretical point of view, two fundamental issues remain: (i) the absence of simple and efﬁciently computable optimality notions, and (ii) cyclic or diverging behavior of existing algorithms.
Revised: Though such optimization problems arise in many machine learning paradigms including training generative adversarial networks (GANs) and adversarially robust models, from a theoretical point of view, two fundamental issues remain: (i) the absence of simple and efﬁciently computable optimality notions, and (ii) cyclic or diverging behavior of existing algorithms.

Original: This paper proposes a new theoretical framework for nonconvex-nonconcave minimax optimization that addresses both of the above issues.
Revised: This paper proposes a new theoretical framework for nonconvex-nonconcave minimax optimization that addresses both of the above issues.

Original: The starting point of this paper is the observation that, under a computational budget, the max-player can not fully maximize f ( x, · ) since nonconcave maximization is NP-hard in general.
Revised: The starting point of this paper is the observation that, under a computational budget, the max-player can not fully maximize f ( x, · ) since nonconcave maximization is NP-hard in general.

Original: So, we propose a new framework, and a corresponding algorithm, for the min-player to play against smooth algorithms deployed by the adversary (i.e., the max-player) instead of against full maximization.
Revised: So, we propose a new framework, and a corresponding algorithm, for the min-player to play against smooth algorithms deployed by the adversary (i.e., the max-player) instead of against full maximization.

Original: Our algorithm is guaranteed to make monotonic progress (thus having no limit cycles or diverging behavior), and to ﬁnd an appropriate “stationary point” in a polynomial number of iterations.
Revised: Our algorithm is guaranteed to make monotonic progress (thus having no limit cycles or diverging behavior), and to ﬁnd an appropriate “stationary point” in a polynomial number of iterations.

Original: Our framework covers practically relevant settings where the smooth algorithms deployed by the adversary are multi-step stochastic gradient ascent, and its accelerated version.
Revised: Our framework covers practically relevant settings where the smooth algorithms deployed by the adversary are multi-step stochastic gradient ascent, and its accelerated version.

Original: We further present experimental results that conﬁrm our theoretical ﬁndings and demonstrate the effectiveness of the proposed approach in practice on simple, conceptual settings.
Revised: We further present experimental results that conﬁrm our theoretical ﬁndings and demonstrate the effectiveness of the proposed approach in practice on simple, conceptual settings.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: This paper considers minimax optimization min x max y f
Revised: This paper considers minimax optimization min x max y f

Original: ( x, y ) in the context of two-player zero-sum games, where the min-player (controlling x ) tries to minimize objective f assuming a worst-case opponent (controlling y ) that acts so as to maximize it.
Revised: ( x, y ) in the context of two-player zero-sum games, where the min-player (controlling x ) tries to minimize objective f assuming a worst-case opponent (controlling y ) that acts so as to maximize it.

Original: Minimax optimization naturally arises in a variety of important machine learning paradigms, with the most prominent examples being the training of generative adversarial networks (GANs) (Goodfellow et al., 2014) and adversarially robust models (Madry et al., 2018).
Revised: Minimax optimization naturally arises in a variety of important machine learning paradigms, with the most prominent examples being the training of generative adversarial networks (GANs) (Goodfellow et al., 2014) and adversarially robust models (Madry et al., 2018).

Original: These applications commonly engage deep neural networks with various techniques such as convolution, recurrent layers, and batch normalization.
Revised: These applications commonly engage deep neural networks with various techniques such as convolution, recurrent layers, and batch normalization.

Original: As a result, the objective function f is highly nonconvex in x and nonconcave in y .
Revised: As a result, the objective function f is highly nonconvex in x and nonconcave in y .

Original: Theoretically, minimax optimization has been extensively studied starting from the seminal work of von Neumann (Neumann, 1928), with many efﬁcient algorithms proposed for solving it (Robinson, 1951; Korpelevich, 1976; Nemirovski, 2004).
Revised: Theoretically, minimax optimization has been extensively studied starting from the seminal work of von Neumann (Neumann, 1928), with many efﬁcient algorithms proposed for solving it (Robinson, 1951; Korpelevich, 1976; Nemirovski, 2004).

Original: A majority of these classical results have been focused on convex-concave functions, and heavily rely on the minimax theorem, i.e., min x max y f
Revised: A majority of these classical results have been focused on convex-concave functions, and heavily rely on the minimax theorem, i.e., min x max y f

Original: ( x, y ) =
Revised: ( x, y ) =

Original: max y min
Revised: max y min

Original: x f ( x, y ) , which no longer holds beyond the convex-concave setting.
Revised: x f ( x, y ) , which no longer holds beyond the convex-concave setting.

Original: Recent line of works (Lin et al., 2020a; Nouiehed et al., 2019; Thekumparampil et al., 2019; Lin et al., 2020b;
Revised: Recent line of works (Lin et al., 2020a; Nouiehed et al., 2019; Thekumparampil et al., 2019; Lin et al., 2020b;

Original: Ostrovskii et al., 2020) address the nonconvex-concave setting where f is nonconvex in x but concave in y by proposing meaningful optimality notions and designing algorithms to ﬁnd such points.
Revised: Ostrovskii et al., 2020) address the nonconvex-concave setting where f is nonconvex in x but concave in y by proposing meaningful optimality notions and designing algorithms to ﬁnd such points.

Original: A key property heavily exploited in this setting is that the inner maximization over y given a ﬁxed x can be computed efﬁciently, which unfortunately does not extend to the nonconvex-nonconcave setting.
Revised: A key property heavily exploited in this setting is that the inner maximization over y given a ﬁxed x can be computed efﬁciently, which unfortunately does not extend to the nonconvex-nonconcave setting.

Original: Consequently, nonconvex-nonconcave optimization remains challenging, and two of the most fundamental theoretical issues are still unresolved: (i) what is an appropriate notion of optimality that can be computed efﬁciently?
Revised: Consequently, nonconvex-nonconcave optimization remains challenging, and two of the most fundamental theoretical issues are still unresolved: (i) what is an appropriate notion of optimality that can be computed efﬁciently?

Original: and (ii) can we design algorithms that do not suffer from cycling or diverging behavior?
Revised: and (ii) can we design algorithms that do not suffer from cycling or diverging behavior?

Original: Practitioners often use simple and popular algorithms such as gradient descent ascent (GDA) and other variants for solving these challenging optimization problems.
Revised: Practitioners often use simple and popular algorithms such as gradient descent ascent (GDA) and other variants for solving these challenging optimization problems.

Original: While these algorithms seem to perform well in terms of producing high quality images in GANs and robust models in adversarial training, they are also highly unstable, particularly in training GANs.
Revised: While these algorithms seem to perform well in terms of producing high quality images in GANs and robust models in adversarial training, they are also highly unstable, particularly in training GANs.

Original: Indeed the instability of GDA and other empirically popular methods is not surprising since they are known to not converge even in very simple settings (Daskalakis & Panageas, 2018a; Bailey et al., 2020).
Revised: Indeed the instability of GDA and other empirically popular methods is not surprising since they are known to not converge even in very simple settings (Daskalakis & Panageas, 2018a; Bailey et al., 2020).

Original: This current state of affairs strongly motivates the need to develop a strong theoretical foundation for nonconvex-nonconcave minimax optimization and to design better algorithms for solving them.
Revised: This current state of affairs strongly motivates the need to develop a strong theoretical foundation for nonconvex-nonconcave minimax optimization and to design better algorithms for solving them.

Original: This work considers the challenging nonconvex-nonconcave setting.
Revised: This work considers the challenging nonconvex-nonconcave setting.

Original: Our framework sprouts from the practical consideration that under a computational budget, the max-player cannot fully maximize f ( x, · ) since nonconcave maximization is NP-hard in general.
Revised: Our framework sprouts from the practical consideration that under a computational budget, the max-player cannot fully maximize f ( x, · ) since nonconcave maximization is NP-hard in general.

Original: In fact, in both the settings of GAN and adversarial training, in practice, the max-player employs simple gradient based algorithms such as gradient ascent, run for a few steps – on the order of 5 for GAN training and 40 for adversarial training (see, e.g., Arjovsky et al. 2017; Madry et al. 2018) – to estimate argmax y f ( x, y ) .
Revised: In fact, in both the settings of GAN and adversarial training, in practice, the max-player employs simple gradient based algorithms such as gradient ascent, run for a few steps – on the order of 5 for GAN training and 40 for adversarial training (see, e.g., Arjovsky et al. 2017; Madry et al. 2018) – to estimate argmax y f ( x, y ) .

Original: To capture this aspect, we assume that the max-player has a toolkit of multiple (potentially randomized) algorithms
Revised: To capture this aspect, we assume that the max-player has a toolkit of multiple (potentially randomized) algorithms

Original: A 1 , A 2 , · · · , A k in an attempt to solve the maximization problem given ﬁxed x , and picks the best solution among these algorithms.
Revised: A 1 , A 2 , · · · , A k in an attempt to solve the maximization problem given ﬁxed x , and picks the best solution among these algorithms.

Original: This motivates us to study the surrogate of the minimax optimization problem as
Revised: This motivates us to study the surrogate of the minimax optimization problem as

Original: [Equation]
Revised: [Equation]

Original: where ∆ k denotes the k -dimensional simplex, and A i ( x ) denotes the output of
Revised: where ∆ k denotes the k -dimensional simplex, and A i ( x ) denotes the output of

Original: algorithm A
Revised: algorithm A

Original: i for a given x .
Revised: i for a given x .

Original: When both the objective function f and the algorithms {A i } ki =1 are smooth (deﬁned formally in Section 3), we can show that (1) becomes a smooth nonconvex-concave minimax optimization problem, where recent advances can be leveraged in solving such problems.
Revised: When both the objective function f and the algorithms {A i } ki =1 are smooth (deﬁned formally in Section 3), we can show that (1) becomes a smooth nonconvex-concave minimax optimization problem, where recent advances can be leveraged in solving such problems.

Original: In particular, given the smooth algorithms deployed by the adversary (i.e. the max-player), this paper proposes two algorithms for solving problems in (1).
Revised: In particular, given the smooth algorithms deployed by the adversary (i.e. the max-player), this paper proposes two algorithms for solving problems in (1).

Original: The ﬁrst algorithm is based on stochastic gradient descent (SGD), which is guaranteed to ﬁnd an appropriate notion of “ (cid:15) -approximate stationary point” in O ( (cid:15) − 4 ) gradient computations.
Revised: The ﬁrst algorithm is based on stochastic gradient descent (SGD), which is guaranteed to ﬁnd an appropriate notion of “ (cid:15) -approximate stationary point” in O ( (cid:15) − 4 ) gradient computations.

Original: The second algorithm is based on proximal algorithm, in the case of deterministic adversarial algorithms {A i } ki =1 , this algorithm has an improved gradient complexity O ( (cid:15) − 3 ) or ˜ O (poly( k ) /(cid:15) 2 ) depending on the choice of subroutine within the algorithm.
Revised: The second algorithm is based on proximal algorithm, in the case of deterministic adversarial algorithms {A i } ki =1 , this algorithm has an improved gradient complexity O ( (cid:15) − 3 ) or ˜ O (poly( k ) /(cid:15) 2 ) depending on the choice of subroutine within the algorithm.

Original: All our algorithms are guaranteed to make monotonic progress, thus having no limit cycles.
Revised: All our algorithms are guaranteed to make monotonic progress, thus having no limit cycles.

Original: Our second set of results show that, many popular algorithms deployed by the adversary such as multi-step stochastic gradient ascent, and multi-step stochastic Nesterov’s accelerated gradient ascent are in fact smooth.
Revised: Our second set of results show that, many popular algorithms deployed by the adversary such as multi-step stochastic gradient ascent, and multi-step stochastic Nesterov’s accelerated gradient ascent are in fact smooth.

Original: Therefore, our framework readily applies to those settings in practice.
Revised: Therefore, our framework readily applies to those settings in practice.

Original: We also present complementing experimental results showing that our theoretical framework and algorithm succeed on simple, conceptual examples of GAN and adversarial training.
Revised: We also present complementing experimental results showing that our theoretical framework and algorithm succeed on simple, conceptual examples of GAN and adversarial training.

Original: While more work is needed to scale our approach to large scale benchmarks in GAN and adversarial training, our experimental results serve as a proof of concept demonstration that our algorithm converges to desirable points in practice, at least on simple conceptual examples.
Revised: While more work is needed to scale our approach to large scale benchmarks in GAN and adversarial training, our experimental results serve as a proof of concept demonstration that our algorithm converges to desirable points in practice, at least on simple conceptual examples.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Due to lack of space, we focus our discussion here on directly related works and present a more detailed overview of related work in Appendix A.
Revised: Due to lack of space, we focus our discussion here on directly related works and present a more detailed overview of related work in Appendix A.

Original: Many existing works that propose local optimality notions in nonconvex-nonconcave minimax optimization suffer from nonexistence of such points in general (Ratliff et al., 2013; Ratliff et al., 2016; Jin et al., 2020; Fiez et al., 2020; Zhang et al., 2020a; Farnia & Ozdaglar, 2020).
Revised: Many existing works that propose local optimality notions in nonconvex-nonconcave minimax optimization suffer from nonexistence of such points in general (Ratliff et al., 2013; Ratliff et al., 2016; Jin et al., 2020; Fiez et al., 2020; Zhang et al., 2020a; Farnia & Ozdaglar, 2020).

Original: To the best of our knowledge, the only works that propose a relaxed local optimality notion that is shown to exist and be computable in polynomial time is due to Keswani et al.
Revised: To the best of our knowledge, the only works that propose a relaxed local optimality notion that is shown to exist and be computable in polynomial time is due to Keswani et al.

Original: Mangoubi & Vishnoi (2021).
Revised: Mangoubi & Vishnoi (2021).

Original: The aforementioned works are similar to this paper in the sense that the min-player faces the max-player with computational restrictions, but are different from ours in terms of the model of the max-player and the algorithms to solve the problem.
Revised: The aforementioned works are similar to this paper in the sense that the min-player faces the max-player with computational restrictions, but are different from ours in terms of the model of the max-player and the algorithms to solve the problem.

Original: Concretely, Keswani et al.
Revised: Concretely, Keswani et al.

Original: Mangoubi & Vishnoi (2021) consider the stationary points of a “smoothed" greedy-max function, which is computed by maximizing along a locally ascending path when ﬁxing the min-player.
Revised: Mangoubi & Vishnoi (2021) consider the stationary points of a “smoothed" greedy-max function, which is computed by maximizing along a locally ascending path when ﬁxing the min-player.

Original: In contrast, our paper considers the stationary points of a “max" function computed by particular smooth algorithms given their initializations, which is a more faithful surrogate to the objective that the min-player wishes to minimize.
Revised: In contrast, our paper considers the stationary points of a “max" function computed by particular smooth algorithms given their initializations, which is a more faithful surrogate to the objective that the min-player wishes to minimize.

Original: Depending on the choice of smooth algorithms and their initializations, the optimal points of Keswani et al.
Revised: Depending on the choice of smooth algorithms and their initializations, the optimal points of Keswani et al.

Original: Mangoubi & Vishnoi (2021) and the optimal points in our paper can be very different, making them incomparable.
Revised: Mangoubi & Vishnoi (2021) and the optimal points in our paper can be very different, making them incomparable.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: In this section, we present problem formulation and preliminaries.
Revised: In this section, we present problem formulation and preliminaries.

Original: We consider function f satisfying
Revised: We consider function f satisfying

Original: Assumption 1.
Revised: Assumption 1.

Original: We denote w =
Revised: We denote w =

Original: ( x, y ) , and assume f : R d 

Revised: ( x, y ) , and assume f : R d

Original: × R d 2 → R is:
Revised: × R d 2 → R is:

Original: (a) B -bounded 
Revised: (a) B -bounded i.e., | f ( w ) | ≤ B , (b) G -Lipschitz

Original: i
Revised: i

Revised: [Equation]

Revised: e.,

Revised: [Equation]

Revised: f

Revised: [Equation]

Original: e., | f ( w ) | ≤ B , (b) G -Lipschitz i.e., | f ( w 1 ) − f ( w 2 ) 
Revised: w 1 ) − f [Equation] w 2 ) [Equation]

Revised: ≤

Revised: [Equation]

Revised: (cid:107)

Revised: [Equation]

Revised: − w

Original: | ≤ G (cid:107) w 1 − w 2 (cid:107) , (c) L -gradient Lipschitz i.e., (cid:107)∇ f ( w 1 ) − ∇ f ( w 2 ) (cid:107) ≤ L (cid:107) w 1 − w 2 (cid:107) , (d) ρ -Hessian Lipschitz i.e., (cid:107)∇ 2 f ( w 1 ) − ∇ 2 f ( w 2 ) (cid:107) ≤ ρ (cid:107) w 1 − w 2 (cid:107) .
Revised: [Equation] (cid:107) , (c) L -gradient Lipschitz i e., (cid:107)∇ f ( w 1 ) − ∇ f ( w 2 ) (cid:107) ≤ L (cid:107) w 1 − w 2 (cid:107) , (d) ρ -Hessian Lipschitz i.e., (cid:107)∇ 2 f ( w 1 ) − ∇ 2 f ( w 2 ) (cid:107) ≤ ρ (cid:107) w 1 − w 2 (cid:107) .

Original: where (cid:107)·(cid:107) denotes Euclidean norm for vectors and operator norm for matrices.
Revised: where (cid:107)·(cid:107) denotes Euclidean norm for vectors and operator norm for matrices.

Original: We aim to solve min x ∈ R d 1 max y ∈ R
Revised: We aim to solve min x ∈ R d 1 max y ∈ R

Original: d 2 f ( x, y ) .
Revised: d 2 f ( x, y ) .

Original: Since max y ∈ R d 2 f ( x, y ) involves non-concave maximization and hence is NP-hard in the worst case, we intend to play against algorithm(s) that y -player uses to compute her strategy.
Revised: Since max y ∈ R d 2 f ( x, y ) involves non-concave maximization and hence is NP-hard in the worst case, we intend to play against algorithm(s) that y -player uses to compute her strategy.

Original: Concretely, given x ∈ R d 1 , we assume that the y -player chooses her (potentially random) strategy (cid:98)
Revised: Concretely, given x ∈ R d 1 , we assume that the y -player chooses her (potentially random) strategy (cid:98)

Original: y z ( x ) =
Revised: y z ( x ) =

Original: A i ∗ ( x ) ( x, z i ∗ ( x ) ) , where we use shorthand z :=
Revised: A i ∗ ( x ) ( x, z i ∗ ( x ) ) , where we use shorthand z :=

Original: ( z 1 , · · · , z k ) , as i ∗ ( x ) =
Revised: ( z 1 , · · · , z k ) , as i ∗ ( x ) =

Original: argmax i ∈
Revised: argmax i ∈

Original: [ k ] f ( x, A i ( x, z i )) , where A 1 , · · · , A k are k deterministic algorithms that take as input x and a random seed z i ∈ R (cid:96) , where z i are all independent.
Revised: [ k ] f ( x, A i ( x, z i )) , where A 1 , · · · , A k are k deterministic algorithms that take as input x and a random seed z i ∈ R (cid:96) , where z i are all independent.

Original: Note that the framework captures randomized algorithms e.g., A could be stochastic gradient ascent on f ( x, · ) , with initialization, minibatching etc. determined by the random seed z .
Revised: Note that the framework captures randomized algorithms e.g., A could be stochastic gradient ascent on f ( x, · ) , with initialization, minibatching etc. determined by the random seed z .

Original: This also incorporates running the same algorithm multiple times, with different seeds and then choosing the best strategy.
Revised: This also incorporates running the same algorithm multiple times, with different seeds and then choosing the best strategy.

Original: We now reformulate the minimax objective function to:
Revised: We now reformulate the minimax objective function to:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: For general algorithms
Revised: For general algorithms

Original: A i , the functions f ( x, A i ( x, z i )) need not be continuous even when f satisﬁes Assumption 1.
Revised: A i , the functions f ( x, A i ( x, z i )) need not be continuous even when f satisﬁes Assumption 1.

Original: However, if the algorithms A i are smooth as deﬁned below, the functions f ( x, A i ( x, z i )) behave much more nicely.
Revised: However, if the algorithms A i are smooth as deﬁned below, the functions f ( x, A i ( x, z i )) behave much more nicely.

Original: Deﬁnition 1 (Algorithm Smoothness) .
Revised: Deﬁnition 1 (Algorithm Smoothness) .

Original: A randomized algorithm A : R d 1 × R (cid:96) → R d 2 is:
Revised: A randomized algorithm A : R d 1 × R (cid:96) → R d 2 is:

Original: (a) G -Lipschitz, if (cid:107)A ( x 1 , z ) − A ( x 2 , z ) (cid:107) ≤ G (cid:107) x 1 − x 2 (cid:107) for any z .
Revised: (a) G -Lipschitz, if (cid:107)A ( x 1 , z ) − A ( x 2 , z ) (cid:107) ≤ G (cid:107) x 1 − x 2 (cid:107) for any z .

Original: (b) L -gradient Lipschitz,
Revised: (b) L -gradient Lipschitz,

Original: if (cid:107) D A ( x 1 , z ) − D A ( x2 , z ) (cid:107) ≤ L (cid:107) x 1 − x 2 (cid:107) for any z .
Revised: if (cid:107) D A ( x 1 , z ) − D A ( x , z ) (cid:107) ≤ L (cid:107) x 1 − x 2 (cid:107) for any z .

Original: Here D A ( x, z ) ∈ R
Revised: Here D A ( x, z ) ∈ R

Original: d 1 × R d 2 is the Jacobian of the function A ( · , z ) for a ﬁxed z .
Revised: d 1 × R d 2 is the Jacobian of the function A ( · , z ) for a ﬁxed z .

Original: The following lemma tells us that f ( x, A ( x, z )) behaves nicely whenever A is a Lipschitz and gradient Lipschitz algorithm.
Revised: The following lemma tells us that f ( x, A ( x, z )) behaves nicely whenever A is a Lipschitz and gradient Lipschitz algorithm.

Original: For deterministic algorithms, we also use the shortened notation
Revised: For deterministic algorithms, we also use the shortened notation

Original: A ( x ) and D A ( x ) .
Revised: A ( x ) and D A ( x ) .

Original: Lemma 1.
Revised: Lemma 1.

Original: Suppose A is G (cid:48) -Lipschitz and L (cid:48)
Revised: Suppose A is G (cid:48) -Lipschitz and L (cid:48)

Original: -gradient
Revised: -gradient

Original: Lipschitz and f satisﬁes Assumption 1.
Revised: Lipschitz and f satisﬁes Assumption 1.

Original: Then, for a ﬁxed z , function f ( · , A ( · , z )) is G (1 + G (cid:48) )
Revised: Then, for a ﬁxed z , function f ( · , A ( · , z )) is G (1 + G (cid:48) )

Original: -Lipschitz and L (1 + G (cid:48) )
Revised: -Lipschitz and L (1 + G (cid:48) )

Original: 2 + GL (cid:48) -gradient Lipschitz.
Revised: 2 + GL (cid:48) -gradient Lipschitz.

Original: While g ( x ) deﬁned in (2) is not necessarily gradient Lipschitz, it can be shown to be weakly-convex as deﬁned below.
Revised: While g ( x ) deﬁned in (2) is not necessarily gradient Lipschitz, it can be shown to be weakly-convex as deﬁned below.

Original: Note that an L -gradient Lipschitz function is L -weakly convex.
Revised: Note that an L -gradient Lipschitz function is L -weakly convex.

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: A function g : R d 1 → R is L -weakly convex if ∀ x , there exists a vector u x satisfying:
Revised: A function g : R d 1 → R is L -weakly convex if ∀ x , there exists a vector u x satisfying:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Any vector u x satisfying this property is called the subgradient of g at x and is denoted by ∇ g ( x ) .
Revised: Any vector u x satisfying this property is called the subgradient of g at x and is denoted by ∇ g ( x ) .

Original: An important property of weakly convex function is that the maximum over a ﬁnite number of weakly convex function is still a weakly convex function.
Revised: An important property of weakly convex function is that the maximum over a ﬁnite number of weakly convex function is still a weakly convex function.

Original: Lemma 2.
Revised: Lemma 2.

Original: Given L -weakly convex functions g 1 , · · · , g k : R d → R , the maximum function g ( · ) := max i ∈
Revised: Given L -weakly convex functions g 1 , · · · , g k : R d → R , the maximum function g ( · ) := max i ∈

Original: [ k ] g i ( · ) is also L -weakly convex and the set of subgradients of g ( · ) at x is given by: 
Revised: [ k ] g i ( · ) is also L -weakly convex and the set of subgradients of g ( · ) at x is given by: ∂g

Original: [Equation]
Revised: [Equation]

Revised: x )

Original: [Equation]
Revised: [Equation]

Original:  j = 1 } , where S ( x ) := argmax i ∈ [ k ] g i ( x ) .
Revised: x ) [Equation] g j ( x ) : λ j ≥ 0 , (cid:80) j ∈ S ( x ) λ j = 1 } , where S ( x ) : = argmax i ∈ [ k ] g i ( x ) .

Original: Thus, under Assumption 1 and the assumptions A i are all G (cid:48)
Revised: Thus, under Assumption 1 and the assumptions A i are all G (cid:48)

Original: -Lipschitz and L (cid:48)
Revised: -Lipschitz and L (cid:48)

Original: -gradient Lipschitz, then g ( · ) deﬁned in (2) is L (1 + G (cid:48) )
Revised: -gradient Lipschitz, then g ( · ) deﬁned in (2) is L (1 + G (cid:48) )

Original: 2 + GL (cid:48)
Revised: 2 + GL (cid:48)

Original: -weakly convex.
Revised: -weakly convex.

Original: The usual optimality notion for weakly-convex functions is approximate ﬁrst order stationary points (Davis & Drusvyatskiy, 2018).
Revised: The usual optimality notion for weakly-convex functions is approximate ﬁrst order stationary points (Davis & Drusvyatskiy, 2018).

Original: Approximate ﬁrst-order stationary point for weakly convex functions :
Revised: Approximate ﬁrst-order stationary point for weakly convex functions :

Original: In order to deﬁne approximate stationary points, we also need the notion of Moreau envelope.
Revised: In order to deﬁne approximate stationary points, we also need the notion of Moreau envelope.

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: The Moreau envelope of a function g : R d 1 → R and parameter λ is:
Revised: The Moreau envelope of a function g : R d 1 → R and parameter λ is:

Original: [Equation]
Revised: [Equation]

Original: The following lemma provides useful properties of the Moreau envelope.
Revised: The following lemma provides useful properties of the Moreau envelope.

Original: Lemma 3.
Revised: Lemma 3.

Original: For an L -weakly convex function g : R d 1 → R and λ < 1 /L , we have:
Revised: For an L -weakly convex function g : R d 1 → R and λ < 1 /L , we have:

Original: (a) The minimizer ˆ x λ ( x ) =
Revised: (a) The minimizer ˆ x λ ( x ) =

Original: arg min x (cid:48) ∈ R d 1 g ( x (cid:48) )
Revised: arg min x (cid:48) ∈ R d 1 g ( x (cid:48) )

Original: +(2 λ ) − 1 (cid:107) x − x (cid:48) (cid:107) 2 is unique and g (ˆ x λ ( x ))
Revised: +(2 λ ) − 1 (cid:107) x − x (cid:48) (cid:107) 2 is unique and g (ˆ x λ ( x ))

Original: ≤
Revised: ≤

Original: g λ ( x ) ≤ g ( x ) .
Revised: g λ ( x ) ≤ g ( x ) .

Original: Furthermore, arg min x g ( x ) =
Revised: Furthermore, arg min x g ( x ) =

Original: arg min x g λ ( x ) .
Revised: arg min x g λ ( x ) .

Original: (b) g λ is λ − 1 (1 + (1 − λL ) − 1 ) -smooth and thus differentiable, and (c) min u ∈ ∂g
Revised: (b) g λ is λ − 1 (1 + (1 − λL ) − 1 ) -smooth and thus differentiable, and (c) min u ∈ ∂g

Original: (ˆ x λ ( x )) (cid:107) u (cid:107) ≤ λ − 1 (cid:107) ˆ x λ ( x ) − x (cid:107) =
Revised: (ˆ x λ ( x )) (cid:107) u (cid:107) ≤ λ − 1 (cid:107) ˆ x λ ( x ) − x (cid:107) =

Original: (cid:107)∇ g λ ( x ) (cid:107) .
Revised: (cid:107)∇ g λ ( x ) (cid:107) .

Original: First order stationary points (FOSP) of a non-smooth nonconvex function are well-deﬁned, i.e., x ∗ is a FOSP of a function g ( x ) if, 0 ∈ ∂f ( x ∗ ) .
Revised: First order stationary points (FOSP) of a non-smooth nonconvex function are well-deﬁned, i.e., x ∗ is a FOSP of a function g ( x ) if, 0 ∈ ∂f ( x ∗ ) .

Original: However, unlike smooth functions, it is nontrivial to deﬁne an approximate FOSP.
Revised: However, unlike smooth functions, it is nontrivial to deﬁne an approximate FOSP.

Original: For example, if we deﬁne an ε -FOSP as the point x with min u ∈ ∂g
Revised: For example, if we deﬁne an ε -FOSP as the point x with min u ∈ ∂g

Original: ( x ) (cid:107) u (cid:107) ≤ ε , where ∂g ( x ) denotes the subgradients of g at x , there may never exist such a point for sufﬁciently small ε , unless x is exactly a FOSP.
Revised: ( x ) (cid:107) u (cid:107) ≤ ε , where ∂g ( x ) denotes the subgradients of g at x , there may never exist such a point for sufﬁciently small ε , unless x is exactly a FOSP.

Original: In contrast, by using above properties of the Moreau envelope of a weakly convex function, it’s approximate FOSP can be deﬁned as (Davis & Drusvyatskiy, 2018):
Revised: In contrast, by using above properties of the Moreau envelope of a weakly convex function, it’s approximate FOSP can be deﬁned as (Davis & Drusvyatskiy, 2018):

Original: Deﬁnition 4.
Revised: Deﬁnition 4.

Original: Given an L -weakly convex function g , we say that x ∗ is an ε -ﬁrst order stationary point ( ε -FOSP) if, (cid:107)∇ g 1 / 2 L ( x ∗ ) (cid:107) ≤ ε , where g 1 / 2 L is the Moreau envelope with parameter 1 / 2 L .
Revised: Given an L -weakly convex function g , we say that x ∗ is an ε -ﬁrst order stationary point ( ε -FOSP) if, (cid:107)∇ g 1 / 2 L ( x ∗ ) (cid:107) ≤ ε , where g 1 / 2 L is the Moreau envelope with parameter 1 / 2 L .

Original: Using Lemma 3, we can show that for any ε -FOSP x ∗ , there exists ˆ x such that (cid:107) ˆ x − x ∗ (cid:107) ≤ ε/ 2 L and min u ∈ ∂g (ˆ x ) (cid:107) u (cid:107) ≤ ε .
Revised: Using Lemma 3, we can show that for any ε -FOSP x ∗ , there exists ˆ x such that (cid:107) ˆ x − x ∗ (cid:107) ≤ ε/ 2 L and min u ∈ ∂g (ˆ x ) (cid:107) u (cid:107) ≤ ε .

Original: In other words, an ε -FOSP is O ( ε ) close to a point ˆ x which has a subgradient smaller than ε .
Revised: In other words, an ε -FOSP is O ( ε ) close to a point ˆ x which has a subgradient smaller than ε .

Original: Other notions of FOSP proposed recently such as in Nouiehed et al.
Revised: Other notions of FOSP proposed recently such as in Nouiehed et al.

Original: (2019) can be shown to be a strict generalization of the above deﬁnition.
Revised: (2019) can be shown to be a strict generalization of the above deﬁnition.

Original:  M AIN R ESULTS
Revised:  M AIN R ESULTS

Original: In this section, we present our main results.
Revised: In this section, we present our main results.

Original: Assuming that the adversary employs Lipschitz and gradient-Lipschitz algorithms (Assumption 2), Section 4.1 shows how to compute (stochastic) subgradients of g ( · ) (deﬁned in (2)) efﬁciently.
Revised: Assuming that the adversary employs Lipschitz and gradient-Lipschitz algorithms (Assumption 2), Section 4.1 shows how to compute (stochastic) subgradients of g ( · ) (deﬁned in (2)) efﬁciently.

Original: Section 4.2 further shows that stochastic subgradient descent (SGD) on g ( · ) can ﬁnd an (cid:15) -FOSP in O (cid:0) (cid:15) − 4 (cid:1) iterations while for the deterministic setting, where the adversary uses only deterministic algorithms, Section 4.3 provides a proximal algorithm that can ﬁnd an (cid:15) -FOSP faster than SGD.
Revised: Section 4.2 further shows that stochastic subgradient descent (SGD) on g ( · ) can ﬁnd an (cid:15) -FOSP in O (cid:0) (cid:15) − 4 (cid:1) iterations while for the deterministic setting, where the adversary uses only deterministic algorithms, Section 4.3 provides a proximal algorithm that can ﬁnd an (cid:15) -FOSP faster than SGD.

Original: For convenience, we denote g z,i ( x ) :
Revised: For convenience, we denote g z,i ( x ) :

Original: = f ( x, A i ( x, z i )) and recall g ( x ) :=
Revised: = f ( x, A i ( x, z i )) and recall g ( x ) :=

Original: E z (cid:2) max i ∈
Revised: E z (cid:2) max i ∈

Original: [ k ] g z,i ( x ) (cid:3) .
Revised: [ k ] g z,i ( x ) (cid:3) .

Original: For deterministic A i , we drop z and just use g i ( x ) .
Revised: For deterministic A i , we drop z and just use g i ( x ) .

Original: 4. C OMPUTING STOCHASTIC SUBGRADIENTS OF g ( x )
Revised: 4. C OMPUTING STOCHASTIC SUBGRADIENTS OF g ( x )

Original: In this section, we give a characterization of subgradients of g ( x ) and show how to compute stochastic subgradients efﬁciently under the following assumption.
Revised: In this section, we give a characterization of subgradients of g ( x ) and show how to compute stochastic subgradients efﬁciently under the following assumption.

Original: Assumption 2.
Revised: Assumption 2.

Original: Algorithms A i in (2) are G (cid:48)
Revised: Algorithms A i in (2) are G (cid:48)

Original: -Lipschitz and L (cid:48)
Revised: -Lipschitz and L (cid:48)

Original: -gradient
Revised: -gradient

Original: Lipschitz as per Deﬁnition 1.
Revised: Lipschitz as per Deﬁnition 1.

Original: Under Assumptions 1 and 2, Lemma 1 tells us that g z,i ( x ) is a G (1+ G (cid:48) )
Revised: Under Assumptions 1 and 2, Lemma 1 tells us that g z,i ( x ) is a G (1+ G (cid:48) )

Original: -Lipschitz and L (1+ G (cid:48) )
Revised: -Lipschitz and L (1+ G (cid:48) )

Original: 2 +
Revised: 2 +

Original: GL (cid:48) -gradient
Revised: GL (cid:48) -gradient

Original: Lipschitz function for every i ∈ [ k ] with
Revised: Lipschitz function for every i ∈ [ k ] with

Original: [Equation]
Revised: [Equation]

Original: where we recall that
Revised: where we recall that

Original: D
Revised: D

Original: A i ( x, z i ) ∈
Revised: A i ( x, z i ) ∈

Original: R d 1 × d 2 is the Jacobian matrix of A i ( · , z i ) :
Revised: R d 1 × d 2 is the Jacobian matrix of A i ( · , z i ) :

Original: R d 1 → R d 2 at x and ∇ x f ( x, A i ( x, z i )) denotes the partial derivative of f with respect to the ﬁrst variable at ( x, A i ( x, z i )) .
Revised: R d 1 → R d 2 at x and ∇ x f ( x, A i ( x, z i )) denotes the partial derivative of f with respect to the ﬁrst variable at ( x, A i ( x, z i )) .

Original: While there is no known general recipe for computing D
Revised: While there is no known general recipe for computing D

Original: A i ( x, z i ) for an arbitrary algorithm
Revised: A i ( x, z i ) for an arbitrary algorithm

Original: A
Revised: A

Original: i , most algorithms used in practice such as stochastic gradient ascent (SGA), stochastic Nesterov accelerated gradient (SNAG), ADAM, admit efﬁcient ways of computing these derivatives e.g., higher package in PyTorch (Grefenstette et al., 2019).
Revised: i , most algorithms used in practice such as stochastic gradient ascent (SGA), stochastic Nesterov accelerated gradient (SNAG), ADAM, admit efﬁcient ways of computing these derivatives e.g., higher package in PyTorch (Grefenstette et al., 2019).

Original: For concreteness, we obtain expression for gradients of SGA and SNAG in Section 5 but the principle behind the derivation holds much more broadly and can be extended to most algorithms used in practice (Grefenstette et al., 2019).
Revised: For concreteness, we obtain expression for gradients of SGA and SNAG in Section 5 but the principle behind the derivation holds much more broadly and can be extended to most algorithms used in practice (Grefenstette et al., 2019).

Original: In practice, the cost of computing ∇ g z,i ( x ) in (5) is at most twice the cost of evaluating g z,i ( x ) —it consists a forward pass for evaluating g z,i ( x ) and a backward pass for evaluating its gradient (Grefenstette et al., 2019).Input: initial point x 0 , step size η , S
Revised: In practice, the cost of computing ∇ g z,i ( x ) in (5) is at most twice the cost of evaluating g z,i ( x ) —it consists a forward pass for evaluating g z,i ( x ) and a backward pass for evaluating its gradient (Grefenstette et al., 2019).

Original: Lemma 2 shows that g ( x ) :=
Revised: Lemma 2 shows that g ( x ) :=

Original: E z (cid:2) max i ∈
Revised: E z (cid:2) max i ∈

Original: [ k ] g z,i ( x ) (cid:3) is a weakly convex function and a stochastic subgradient of g ( · ) can be computed by generating a random sample of z 1 , · · · , z k as: 
Revised: [ k ] g z,i ( x ) (cid:3) is a weakly convex function and a stochastic subgradient of g ( · ) can be computed by generating a random sample of z 1 , · · · , z k as: (cid:98) ∇ [Equation] j

Original: [Equation]
Revised: [Equation]

Original:  for any λ ∈ ∆ k , where S ( x ) :=
Revised: S [Equation] λ j ∇ g z,i ( x ) for any λ ∈ ∆ k , where S ( x ) :=

Original: argmax i ∈ [ k ] g z,i ( x ) (6)
Revised: argmax i ∈ [ k ] g z,i ( x ) (6)

Original: Here ∆ k is the k -dimensional probability simplex.
Revised: Here ∆ k is the k -dimensional probability simplex.

Original: It can be seen that E z [ ˆ ∇ g ( x )] ∈ ∂g ( x ) .
Revised: It can be seen that E z [ ˆ ∇ g ( x )] ∈ ∂g ( x ) .

Original: Furthermore, if all A i are deterministic algorithms, then the above is indeed a subgradient of g .
Revised: Furthermore, if all A i are deterministic algorithms, then the above is indeed a subgradient of g .

Original: 4. C ONVERGENCE RATE OF SGD
Revised: 4. C ONVERGENCE RATE OF SGD

Original: The SGD algorithm to solve (2) is given in Algorithm 1.
Revised: The SGD algorithm to solve (2) is given in Algorithm 1.

Original: The following theorem shows that Algorithm 1 ﬁnds an (cid:15) -FOSP of g ( · ) in S
Revised: The following theorem shows that Algorithm 1 ﬁnds an (cid:15) -FOSP of g ( · ) in S

Original: = O (cid:0) (cid:15) − 4 (cid:1) iterations.
Revised: = O (cid:0) (cid:15) − 4 (cid:1) iterations.

Original: Since each iteration of Algorithmrequires computing the gradient of g z,i ( x ) for each i ∈ [ k ] at a single point x s , this leads to a total of
Revised: Since each iteration of Algorithmrequires computing the gradient of g z,i ( x ) for each i ∈ [ k ] at a single point x s , this leads to a total of

Original: S =
Revised: S =

Original: O (cid:0) (cid:15) − 4 (cid:1) gradient computations for each g z,i ( x ) .
Revised: O (cid:0) (cid:15) − 4 (cid:1) gradient computations for each g z,i ( x ) .

Original: Theorem 1.
Revised: Theorem 1.

Original: Under Assumptions 1 and 2, if S ≥ 16 B (cid:98) L (cid:98) G 2 (cid:15) − 4 and learning rate η = (2 B/ [ (cid:98) L (cid:98) G 2 ( S + 1)]) 1 / 2 then output of Algorithm 1 satisﬁes E [ (cid:107)∇ g 1 / 2 (cid:98) L (¯ x ) (cid:107) 2 ] ≤ (cid:15) 2 , where (cid:98) L :=
Revised: Under Assumptions 1 and 2, if S ≥ 16 B (cid:98) L (cid:98) G 2 (cid:15) − 4 and learning rate η = (2 B/ [ (cid:98) L (cid:98) G 2 ( S + 1)]) 1 / 2 then output of Algorithm 1 satisﬁes E [ (cid:107)∇ g 1 / 2 (cid:98) L (¯ x ) (cid:107) 2 ] ≤ (cid:15) 2 , where (cid:98) L :=

Original: L (1 + G (cid:48) )
Revised: L (1 + G (cid:48) )

Original: 2 + GL (cid:48) and (cid:98) G :=
Revised: 2 + GL (cid:48) and (cid:98) G :=

Original: G (1 + G (cid:48) ) .
Revised: G (1 + G (cid:48) ) .

Original: Theorem 1 claims that the expected norm squared of the Moreau envelope gradient of the output point satisﬁes the condition for being an (cid:15) -FOSP.
Revised: Theorem 1 claims that the expected norm squared of the Moreau envelope gradient of the output point satisﬁes the condition for being an (cid:15) -FOSP.

Original: This, together with Markov’s inequality, implies that at least half of x 0 , · · · , x S are 2 (cid:15) -FOSPs, so that the probability of outputting a 2 (cid:15) -FOSP is at least 0 .
Revised: This, together with Markov’s inequality, implies that at least half of x 0 , · · · , x S are 2 (cid:15) -FOSPs, so that the probability of outputting a 2 (cid:15) -FOSP is at least 0 .

Original: Proposition 1 in Appendix C shows how to use an efﬁcient postprocessing mechanism to output a 2 (cid:15) -FOSP with high probability.
Revised: Proposition 1 in Appendix C shows how to use an efﬁcient postprocessing mechanism to output a 2 (cid:15) -FOSP with high probability.

Original: Theorem 1 essentially follows from the results of (Davis &
Revised: Theorem 1 essentially follows from the results of (Davis &

Original: Drusvyatskiy, 2018), where the key insight is that, in expectation, the SGD procedure in Algorithmalmost monotonically decreases the Moreau envelope evaluated at x s i.e., E (cid:104) g
Revised: Drusvyatskiy, 2018), where the key insight is that, in expectation, the SGD procedure in Algorithmalmost monotonically decreases the Moreau envelope evaluated at x s i.e., E (cid:104) g

Original: 1 / 2 (cid:98) L ( x s ) (cid:105) is almost monotonically decreasing.
Revised: 1 / 2 (cid:98) L ( x s ) (cid:105) is almost monotonically decreasing.

Original: This shows that Algorithm 1 makes (almost) monotonic progress in a precise sense and hence does not have limit cycles.
Revised: This shows that Algorithm 1 makes (almost) monotonic progress in a precise sense and hence does not have limit cycles.

Original: In contrast, none of the other existing algorithms for nonconvex-nonconcave minimax optimization enjoy such a guarantee.
Revised: In contrast, none of the other existing algorithms for nonconvex-nonconcave minimax optimization enjoy such a guarantee.

Original: Note that this claim includes extra-gradient and optimistic gradient methods that have been touted as empirically mitigating cycling behavior; see Appendix F for examples demonstrating nonconvergence of these methods.
Revised: Note that this claim includes extra-gradient and optimistic gradient methods that have been touted as empirically mitigating cycling behavior; see Appendix F for examples demonstrating nonconvergence of these methods.

Original: 4. P ROXIMAL ALGORITHM WITH FASTER CONVERGENCE FOR DETERMINISTIC ALGORITHMS
Revised: 4. P ROXIMAL ALGORITHM WITH FASTER CONVERGENCE FOR DETERMINISTIC ALGORITHMS

Original: While the rate achieved by SGD is the best known for weakly-convex optimization with a stochastic subgradient oracle, faster algorithms exist for functions which can be written as maximum over a ﬁnite number of smooth functions with access to exact subgradients of these component functions .
Revised: While the rate achieved by SGD is the best known for weakly-convex optimization with a stochastic subgradient oracle, faster algorithms exist for functions which can be written as maximum over a ﬁnite number of smooth functions with access to exact subgradients of these component functions .

Original: These conditions are satisﬁed when A i are all deterministic and satisfy Assumption 2.
Revised: These conditions are satisﬁed when A i are all deterministic and satisfy Assumption 2.

Original: A pseudocode of such a fast proximal algorithm, inspired by Thekumparampil et al.
Revised: A pseudocode of such a fast proximal algorithm, inspired by Thekumparampil et al.

Original: (2019, Algorithm 3), is presented in Algorithm 2.
Revised: (2019, Algorithm 3), is presented in Algorithm 2.

Original: However, in contrast to the results of Thekumparampil et al.
Revised: However, in contrast to the results of Thekumparampil et al.

Original: (2019), the following theorem provides two alternate ways of implementing Step 3 of Algorithm 2, resulting in two different (and incomparable) convergence rates.
Revised: (2019), the following theorem provides two alternate ways of implementing Step 3 of Algorithm 2, resulting in two different (and incomparable) convergence rates.

Original: Theorem 2. Under Assumptions 1 and 2, if (cid:98) L :=
Revised: Input: initial point x 0 , target accuracy (cid:15) , smoothness parameter (cid:98) L 1 (cid:98) (cid:15) ← (cid:15) 2 / (64 (cid:98) L ) 2 for t = 0 , 1 , . . S do

Original: L (1 + G (cid:48) )
Revised: Find x s +1 such that

Original: 2 + GL (cid:48)
Revised: [Equation]

Original: + kG (1 + G (cid:48) ) and S ≥ 200 (cid:98) LB(cid:15) −2 then Algorithm 2 returns x s satisfying (cid:107)∇ g 1 / 2 (cid:98) L ( x s ) (cid:107) ≤ (cid:15) .
Revised: if max i ∈ [ k ] g i ( x s +1 ) + (cid:98) L (cid:107) x s − x s +1 (cid:107) 2 ≥ max i ∈ [ k ] g i ( x s ) − 3 (cid:98) (cid:15)/ 4 then return x s

Original: Depending on whether we use (Thekumparampil et al., 2019, Algorithm 1) or cutting plane method (Lee et al., 2015) for
Revised: Theorem 2.

Original: Input: initial point x 0 , target accuracy (cid:15) , smoothness parameter (cid:98) L 1 (cid:98) (cid:15) ← (cid:15) 2 / (64 (cid:98) L ) 2 for t = 0 , 1 , . . S do Find x s +1 such that
Revised: Under Assumptions 1 and 2, if (cid:98) L :=

Original: [Equation] if max i ∈ [ k ] g [Equation] ( x s + 
Revised: L (1 + G (cid:48) )

Original: [Equation] + (cid:98) L (cid:107) x [Equation] − x
Revised: 2 + GL (cid:48)

Original: [Equation] (cid:107) [Equation] ≥ max
Revised: + kG (1 + G (cid:48) ) and

Original: [Equation] ∈ [ k ] g [Equation] ( x 
Revised: S ≥ 200 (cid:98) LB(cid:15) − 2 then Algorithm 2 returns x s satisfying (cid:107)∇ g 1 / 2 (cid:98) L ( x s ) (cid:107) ≤ (cid:15) .

Original: [Equation] ) − [Equation] (cid:98) (cid:15)/ 4 then return x s solving Step 3 of Algorithm 2, the total number of gradient computations of each g i is (cid:101) O (cid:16) (cid:98)
Revised: Depending on whether we use (Thekumparampil et al., 2019, Algorithm 1) or cutting plane method (Lee et al., 2015) for solving Step 3 of Algorithm 2, the total number of gradient computations of each g i is (cid:101) O (cid:16) (cid:98)

Original: L 2 B (cid:15) 3 (cid:17) or
Revised: L 2 B (cid:15) 3 (cid:17) or

Original: O (cid:16) (cid:98) LB(cid:15) 2 · poly( k ) log (cid:98) L(cid:15) (cid:17) respectively.
Revised: O (cid:16) (cid:98) LB(cid:15) 2 · poly( k ) log (cid:98) L(cid:15) (cid:17) respectively.

Original: Ignoring the parameters L, G, L (cid:48) and G (cid:48) , the above theorem tells us that Algorithm 2 outputs an (cid:15) -FOSP using O (cid:0) k(cid:15) − 3 (cid:1) or O (cid:0) poly( k ) (cid:15) − 2 log 1 (cid:15) (cid:1) gradient queries to each g
Revised: Ignoring the parameters L, G, L (cid:48) and G (cid:48) , the above theorem tells us that Algorithm 2 outputs an (cid:15) -FOSP using O (cid:0) k(cid:15) − 3 (cid:1) or O (cid:0) poly( k ) (cid:15) − 2 log 1 (cid:15) (cid:1) gradient queries to each g

Original: i depending on whether
Revised: i depending on whether

Original: Thekumparampil et al.
Revised: Thekumparampil et al.

Original: (2019, Algorithm 3) or cutting plane method (Lee et al., 2015) was used for implementing Step 3 .
Revised: (2019, Algorithm 3) or cutting plane method (Lee et al., 2015) was used for implementing Step 3 .

Original: While the proximal algorithm itself works even when A i are randomized algorithms, there are no known algorithms that can implement Step (3) with fewer than O (cid:0) (cid:15) − 2 (cid:1) stochastic gradient queries.
Revised: While the proximal algorithm itself works even when A i are randomized algorithms, there are no known algorithms that can implement Step (3) with fewer than O (cid:0) (cid:15) − 2 (cid:1) stochastic gradient queries.

Original: Hence, this does not improve upon the O (cid:0) (cid:15) − 4 (cid:1) guarantee for Algorithmwhen A i are randomized algorithms.
Revised: Hence, this does not improve upon the O (cid:0) (cid:15) − 4 (cid:1) guarantee for Algorithmwhen A i are randomized algorithms.

Original: The proof of Theorem 2 shows that the iterates x s monotonically decrease the value g
Revised: The proof of Theorem 2 shows that the iterates x s monotonically decrease the value g

Original: ( x s ) , guaranteeing that there are no limit cycles for Algorithm 2 as well.
Revised: ( x s ) , guaranteeing that there are no limit cycles for Algorithm 2 as well.

Original:  S MOOTHNESS OF P OPULAR A LGORITHMS
Revised:  S MOOTHNESS OF P OPULAR A LGORITHMS

Original: In this section, we show that two popular algorithms—namely, T -step stochastic gradient ascent (SGA) and T -step stochastic Nesterov’s accelerated gradient ascent (SNAG)—are both Lipschitz and gradient-Lipschitz satisfying Assumption 2 and hence are captured by our results in Section 4.
Revised: In this section, we show that two popular algorithms—namely, T -step stochastic gradient ascent (SGA) and T -step stochastic Nesterov’s accelerated gradient ascent (SNAG)—are both Lipschitz and gradient-Lipschitz satisfying Assumption 2 and hence are captured by our results in Section 4.

Original: Consider the setting f ( x, y )
Revised: Consider the setting f ( x, y )

Original: =
Revised: =

Original: 1 n (cid:80) j ∈ [ n ] f j ( x, y ) .
Revised: 1 n (cid:80) j ∈ [ n ] f j ( x, y ) .

Original: Let z be a random seed that captures the randomness in the initial point as well as minibatch order in SGA and SNAG.
Revised: Let z be a random seed that captures the randomness in the initial point as well as minibatch order in SGA and SNAG.

Original: We ﬁrst provide the smoothness results on T -step SGA for different assumptions on the shape of the function f and for T -step SNAG.
Revised: We ﬁrst provide the smoothness results on T -step SGA for different assumptions on the shape of the function f and for T -step SNAG.

Original: After giving these results, we make remarks interpreting their signiﬁcance and the implications.
Revised: After giving these results, we make remarks interpreting their signiﬁcance and the implications.

Original: T -step SGA: For a given x and random seed z , the T -step SGA update is givenby:
Revised: T -step SGA: For a given x and random seed z , the T -step SGA update is given

Original: [Equation]
Revised: by: y t +1 =

Original:  where σ : [ T ] → [ N ] is a sample selection function and η is the stepsize.
Revised: y t + η ∇ y f σ ( t ) ( x, y t ) , where σ : [ T ] → [ N ] is a sample selection function and η is the stepsize.

Original: Observe that with the same randomness z , the initial point does not depend on x i.e., y 0 ( x ) =
Revised: Observe that with the same randomness z , the initial point does not depend on x i.e., y 0 ( x ) =

Original: y 

Revised: y

Original: ( x (cid:48) ) , so Dy 

Revised: ( x (cid:48) ) , so Dy

Original: = 0 .
Revised: = 0 .

Original: The following theorems provide the Lipschitz and gradient Lipschitz constants of y T ( x ) (as generated by T -step SGA) for the general nonconvex-nonconcave setting as well as the settings in which the function f is nonconvex-concave and nonconvex-strongly concave.
Revised: The following theorems provide the Lipschitz and gradient Lipschitz constants of y T ( x ) (as generated by T -step SGA) for the general nonconvex-nonconcave setting as well as the settings in which the function f is nonconvex-concave and nonconvex-strongly concave.

Original: Theorem 3 (General Case) .
Revised: Theorem 3 (General Case) .

Original: Suppose for all j ∈ [ n ] , f j satisﬁes Assumption 1.
Revised: Suppose for all j ∈ [ n ] , f j satisﬁes Assumption 1.

Original: Then, for any ﬁxed randomness z , T -step SGA is (1 + ηL ) T -Lipschitz and 4( ρ/L ) · (1 + ηL ) 2 T -gradient Lipschitz.
Revised: Then, for any ﬁxed randomness z , T -step SGA is (1 + ηL ) T -Lipschitz and 4( ρ/L ) · (1 + ηL ) 2 T -gradient Lipschitz.

Original: Theorem 4 (Concave Case) .
Revised: Theorem 4 (Concave Case) .

Original: Suppose for all j ∈ [ n ] , f j satisﬁes Assumption 1 and f j ( x, · ) is concave for any x .
Revised: Suppose for all j ∈ [ n ] , f j satisﬁes Assumption 1 and f j ( x, · ) is concave for any x .

Original: Then, for any ﬁxed randomness z , T -step SGA is ηLT -Lipschitz and ( ρ/L ) · (1 + ηLT ) 3gradient Lipschitz.
Revised: Then, for any ﬁxed randomness z , T -step SGA is ηLT -Lipschitz and ( ρ/L ) · (1 + ηLT ) 3gradient Lipschitz.

Original: Theorem 5 (Strongly-concave Case) .
Revised: Theorem 5 (Strongly-concave Case) .

Original: Suppose for all j ∈ [ n ] , f j satisﬁes Assumption 1 and f j ( x, · ) is α -strongly concave for any x .
Revised: Suppose for all j ∈ [ n ] , f j satisﬁes Assumption 1 and f j ( x, · ) is α -strongly concave for any x .

Original: Then, for any ﬁxed randomness z , T -step SGA is κ -Lipschitz and 4( ρ/L ) · κ 3 -gradient Lipschitz, where κ =
Revised: Then, for any ﬁxed randomness z , T -step SGA is κ -Lipschitz and 4( ρ/L ) · κ 3 -gradient Lipschitz, where κ =

Original: L/α is the condition number.
Revised: L/α is the condition number.

Original: T -step SNAG:
Revised: T -step SNAG:

Original: For a given random seed z , the T -step SNAG update is given by:
Revised: For a given random seed z , the T -step SNAG update is given by:

Original: [Equation]
Revised: [Equation]

Original:  where η is the stepsize, θ ∈ [0 , 1] is the momentum parameter.
Revised: and [Equation] where η is the stepsize, θ ∈ [0 , 1] is the momentum parameter.

Original: The output of the algorithm is given by A ( x, z ) =
Revised: The output of the algorithm is given by A ( x, z ) =

Original: y T ( x ) .
Revised: y T ( x ) .

Original: Furthermore, we have the following guarantee.
Revised: Furthermore, we have the following guarantee.

Original: Theorem 6 (General Case) .
Revised: Theorem 6 (General Case) .

Original: Suppose for all j ∈ [ n ] , f j satisﬁes Assumption 1.
Revised: Suppose for all j ∈ [ n ] , f j satisﬁes Assumption 1.

Original: Then, for any ﬁxed seed z , T -step SNAG is T (1+ ηL/θ ) T -Lipschitz and 50( ρ/L ) · T 3 (1+ ηL/θ ) 2 T -gradient Lipschitz.
Revised: Then, for any ﬁxed seed z , T -step SNAG is T (1+ ηL/θ ) T -Lipschitz and 50( ρ/L ) · T 3 (1+ ηL/θ ) 2 T -gradient Lipschitz.

Original: Remarks on the Impact of the Smoothness Results: The Lipschitz and gradient Lipschitz parameters for T -step SGA and T -step SNAG in the setting where f is nonconvex- nonconcave are all exponential in T , the duration of the algorithm.
Revised: Remarks on the Impact of the Smoothness Results: The Lipschitz and gradient Lipschitz parameters for T -step SGA and T -step SNAG in the setting where f is nonconvex- nonconcave are all exponential in T , the duration of the algorithm.

Original: In general, this seems unavoidable in the worst case for the above algorithms and also seems to be the case for most of the other popular algorithms such as ADAM, RMSProp, etc.
Revised: In general, this seems unavoidable in the worst case for the above algorithms and also seems to be the case for most of the other popular algorithms such as ADAM, RMSProp, etc.

Original: In contrast, in the nonconvex- concave and nonconvex- strongly concave settings, our results show that the smoothness parameters of T -step SGA are no longer exponential in
Revised: In contrast, in the nonconvex- concave and nonconvex- strongly concave settings, our results show that the smoothness parameters of T -step SGA are no longer exponential in

Original: T .
Revised: T .

Original: In particular, in the nonconvex- concave the Lipschitz parameter is linear in T and the gradient
Revised: In particular, in the nonconvex- concave the Lipschitz parameter is linear in T and the gradient

Original: Lipschitz parameter is polynomial in T while in the nonconvex- strongly concave , the analogous smoothness parameters are no longer dependent on T .
Revised: Lipschitz parameter is polynomial in T while in the nonconvex- strongly concave , the analogous smoothness parameters are no longer dependent on T .

Original: We conjecture this is also the case for T -step
Revised: We conjecture this is also the case for T -step

Original: SNAG, though the proof appears quite tedious and hence, we opted to leave that for future work.
Revised: SNAG, though the proof appears quite tedious and hence, we opted to leave that for future work.

Original: For problems of practical importance, however, we believe that the smoothness parameters are rarely exponential in T .
Revised: For problems of practical importance, however, we believe that the smoothness parameters are rarely exponential in T .

Original: Our experimental results conﬁrm this intuition (see Figure 2).
Revised: Our experimental results conﬁrm this intuition (see Figure 2).

Original: Furthermore, note that the function g ( x ) (2), and its stationary points, vary as one varies T .
Revised: Furthermore, note that the function g ( x ) (2), and its stationary points, vary as one varies T .

Original: While in adversarial training,
Revised: While in adversarial training,

Original: T is essentially determined by the power of adversary against which we wish the model to be robust, in GAN training, T is actually an algorithmic knob that can be tuned to improve the ﬁnal generator performance.
Revised: T is essentially determined by the power of adversary against which we wish the model to be robust, in GAN training, T is actually an algorithmic knob that can be tuned to improve the ﬁnal generator performance.

Original: Indeed, larger T does not necessarily lead to better generators and small values of T are of vital interest; as is pointed out in the original paper (Goodfellow et al., 2014), fully optimizing the discriminator can result in overﬁtting and too strong of a discriminator can limit the ability of the generator to learn.
Revised: Indeed, larger T does not necessarily lead to better generators and small values of T are of vital interest; as is pointed out in the original paper (Goodfellow et al., 2014), fully optimizing the discriminator can result in overﬁtting and too strong of a discriminator can limit the ability of the generator to learn.

Original: Finally, while we prove the Lipschitz and gradient Lipschitz properties only for SGA and SNAG, we believe that the same techniques could be used to prove similar results for popular algorithms such as ADAM, RMSProp, etc.
Revised: Finally, while we prove the Lipschitz and gradient Lipschitz properties only for SGA and SNAG, we believe that the same techniques could be used to prove similar results for popular algorithms such as ADAM, RMSProp, etc.

Original: However, there are other algorithms, particularly those involving projection that are not gradient Lipschitz (see Proposition 3 in Appendix E.5).
Revised: However, there are other algorithms, particularly those involving projection that are not gradient Lipschitz (see Proposition 3 in Appendix E.5).

Original:  E MPIRICAL R ESULTS
Revised:  E MPIRICAL R ESULTS

Original: This section presents empirical results evaluating our SGD algorithm (Algorithm 1) for generative adversarial networks (Goodfellow et al., 2014) and adversarial training (Madry et al., 2018).
Revised: This section presents empirical results evaluating our SGD algorithm (Algorithm 1) for generative adversarial networks (Goodfellow et al., 2014) and adversarial training (Madry et al., 2018).

Original: Our results demonstrate that our framework results in stable monotonic improvement during training and converges to desirable solutions in both GAN and adversarial training problems.
Revised: Our results demonstrate that our framework results in stable monotonic improvement during training and converges to desirable solutions in both GAN and adversarial training problems.

Original: While optimization through the algorithm of the adversary increases the complexity of each update by a factor of two, our results demonstrate that it also leads to faster convergence.
Revised: While optimization through the algorithm of the adversary increases the complexity of each update by a factor of two, our results demonstrate that it also leads to faster convergence.

Original: Finally, we show the gradient norms do not grow exponentially in the number of gradient ascent steps T taken by the adversary in practice.
Revised: Finally, we show the gradient norms do not grow exponentially in the number of gradient ascent steps T taken by the adversary in practice.

Original: Generative Adversarial Networks.
Revised: Generative Adversarial Networks.

Original: Generative adversarial network formulations can be characterized by the following minimax problem (Nagarajan & Kolter, 2017; Mescheder et al., 2018):[Equation]
Revised: Generative adversarial network formulations can be characterized by the following minimax problem (Nagarajan & Kolter, 2017; Mescheder et al., 2018):

Original: [Equation]
Revised: [Equation]

Original: In this formulation G θ : Z → X is the generator network parameterized by θ that maps from the latent space Z to the input space X , D ω : X → R is discriminator network parameterized by ω that maps from the input space X to real-valued logits, and p X and p Z are the distributions over the input and latent spaces.
Revised: In this formulation G θ : Z → X is the generator network parameterized by θ that maps from the latent space Z to the input space X , D ω : X → R is discriminator network parameterized by ω that maps from the input space X to real-valued logits, and p X and p Z are the distributions over the input and latent spaces.

Original: The loss function deﬁnes the objective where (cid:96) ( w ) =
Revised: The loss function deﬁnes the objective where (cid:96) ( w ) =

Original: − log(1 + exp( − w )) gives the original “saturating” generative adversarial networks formulation (Goodfellow et al., 2014).
Revised: − log(1 + exp( − w )) gives the original “saturating” generative adversarial networks formulation (Goodfellow et al., 2014).

Original: Dirac–GAN.
Revised: Dirac–GAN.

Original: The Dirac-GAN (Mescheder et al., 2018) is a simple and common baseline for evaluating the efﬁcacy of generative adversarial network training methods.
Revised: The Dirac-GAN (Mescheder et al., 2018) is a simple and common baseline for evaluating the efﬁcacy of generative adversarial network training methods.

Original: In this problem, the generator distribution G θ ( z ) =
Revised: In this problem, the generator distribution G θ ( z ) =

Original: δ θ is a Dirac distribution concentrated at θ , the discriminator network
Revised: δ θ is a Dirac distribution concentrated at θ , the discriminator network

Original: D ω ( x ) = − ωx is linear, and the real data distribution p X = δ 0 is a Dirac distribution concentrated at zero.
Revised: D ω ( x ) = − ωx is linear, and the real data distribution p X = δ 0 is a Dirac distribution concentrated at zero.

Original: The resulting objective after evaluating (7) with the loss (cid:96) ( w ) =
Revised: The resulting objective after evaluating (7) with the loss (cid:96) ( w ) =

Original: − log(1 + exp( − w )) is min θ max ω f ( θ, ω ) =
Revised: − log(1 + exp( − w )) is min θ max ω f ( θ, ω ) =

Original: (cid:96) ( θω ) + (cid:96) (0) =
Revised: (cid:96) ( θω ) + (cid:96) (0) =

Original: − log(1 + e − θω ) − log(2) .
Revised: − log(1 + e − θω ) − log(2) .

Original: To mimic the real data distribution, the generator parameter θ should converge to θ ∗ = 0 .
Revised: To mimic the real data distribution, the generator parameter θ should converge to θ ∗ = 0 .

Original: Notably, simultaneous and alternating gradient descent-ascent are known to cycle and fail to converge on this problem (see Figure 1).
Revised: Notably, simultaneous and alternating gradient descent-ascent are known to cycle and fail to converge on this problem (see Figure 1).

Original: We consider an instantiation of our framework where the discriminator
Revised: We consider an instantiation of our framework where the discriminator

Original: [Figure]
Revised: [Figure]

Original:  samples an initialization uniformly between [ − 0 . 1 , 0 . 1] and performs T = 10 steps of gradient ascent between each generator update.
Revised: We see that training is stable and results in monotonic progress towards the samples an initialization uniformly between [ − 0 . 1 , 0 . 1] and performs T = 10 steps of gradient ascent between each generator update.

Original: The learning rates for both the generator and the discriminator are η = 0 .
Revised: The learning rates for both the generator and the discriminator are η = 0 .

Original: We present the results in Figure 1.
Revised: We present the results in Figure 1.

Original: Notably, the generator parameter monotonically converges to the optimal θ ∗ = 0 and matches the real data distribution using our training method.
Revised: Notably, the generator parameter monotonically converges to the optimal θ ∗ = 0 and matches the real data distribution using our training method.

Original: We also show the performance when the generator descends using partial gradient ∇ θ f ( x, A ( θ )) instead of the total gradient ∇ f
Revised: We also show the performance when the generator descends using partial gradient ∇ θ f ( x, A ( θ )) instead of the total gradient ∇ f

Original: ( θ, A ( θ )) in Algorithm 1.
Revised: ( θ, A ( θ )) in Algorithm 1.

Original: This method is able to converge to the optimal generator distribution but at a slower rate.
Revised: This method is able to converge to the optimal generator distribution but at a slower rate.

Original: Together, this example highlights that our method ﬁxes the usual cycling problem by reinitializing the discriminator and also that optimizing through the discriminator algorithm is key to fast convergence.
Revised: Together, this example highlights that our method ﬁxes the usual cycling problem by reinitializing the discriminator and also that optimizing through the discriminator algorithm is key to fast convergence.

Original: Additional results are given in Appendix G.
Revised: Additional results are given in Appendix G.

Original: Mixture of Gaussians.
Revised: Mixture of Gaussians.

Original: We now demonstrate that the insights we developed from the Dirac-GAN (stability and monotonic improvement) carry over to the more complex problem of learning a 2dimensional mixture of Gaussians.
Revised: We now demonstrate that the insights we developed from the Dirac-GAN (stability and monotonic improvement) carry over to the more complex problem of learning a 2dimensional mixture of Gaussians.

Original: This is a common example and a number of papers (see, e.g., Metz et al. 2017; Mescheder et al. 2017; Balduzzi et al. 2018) show that standard training methods using simultaneous or alternating gradient descent-ascent can fail.
Revised: This is a common example and a number of papers (see, e.g., Metz et al. 2017; Mescheder et al. 2017; Balduzzi et al. 2018) show that standard training methods using simultaneous or alternating gradient descent-ascent can fail.

Original: The setup for the problem is as follows.
Revised: The setup for the problem is as follows.

Original: The real data distribution consists of 2-dimensional Gaussian distributions with means given by µ = [sin( φ ) , cos( φ )] for φ ∈ { kπ/ 4 } 7 k =0 and each with covariance σ 2 I where σ 2 = 0 .
Revised: The real data distribution consists of 2-dimensional Gaussian distributions with means given by µ = [sin( φ ) , cos( φ )] for φ ∈ { kπ/ 4 } 7 k =0 and each with covariance σ 2 I where σ 2 = 0 .

Original: For training, the real data x ∈ R 2 is drawn at random from the set of Gaussian distributions and the latent data z ∈ R 16 is drawn from a standard normal distribution with batch sizes of 512.
Revised: For training, the real data x ∈ R 2 is drawn at random from the set of Gaussian distributions and the latent data z ∈ R 16 is drawn from a standard normal distribution with batch sizes of 512.

Original: The network for the generator and discriminator contain two and one hidden layers respectively, each of which contain 32 neurons and ReLU activation functions.
Revised: The network for the generator and discriminator contain two and one hidden layers respectively, each of which contain 32 neurons and ReLU activation functions.

Original: We consider the objective from (7) with (cid:96) ( w ) =
Revised: We consider the objective from (7) with (cid:96) ( w ) =

Original: − log(1 + exp( − w )) which corresponds to the “saturating” generative adversarial networks formulation (Goodfellow et al., 2014).
Revised: − log(1 + exp( − w )) which corresponds to the “saturating” generative adversarial networks formulation (Goodfellow et al., 2014).

Original: This objective is known to be difﬁcult to train since with typical training methods the generator gradients saturate early in training.
Revised: This objective is known to be difﬁcult to train since with typical training methods the generator gradients saturate early in training.

Original: We show results using our framework in Figure 3 where the discriminator performs T = 15 steps of gradient ascent and the initialization between each generator step is obtained by the default network initialization in Pytorch.
Revised: We show results using our framework in Figure 3 where the discriminator performs T = 15 steps of gradient ascent and the initialization between each generator step is obtained by the default network initialization in Pytorch.

Original: The generator and discriminator learning rates are both ﬁxed to be η = 0 .
Revised: The generator and discriminator learning rates are both ﬁxed to be η = 0 .

Original: We see that our method has stable improvement during the course of training and recovers close to the real data distribution.
Revised: We see that our method has stable improvement during the course of training and recovers close to the real data distribution.

Original: We demonstrate in Appendix G that this result is robust by presenting the ﬁnal output of 10 runs of the procedure.
Revised: We demonstrate in Appendix G that this result is robust by presenting the ﬁnal output of 10 runs of the procedure.

Original: Notably, the training algorithm recovers all the modes of the distribution in each run.
Revised: Notably, the training algorithm recovers all the modes of the distribution in each run.

Original: We also show results using Adam for the discriminator in Appendix G.
Revised: We also show results using Adam for the discriminator in Appendix G.

Original: Adversarial Training.
Revised: Adversarial Training.

Original: Given a data distribution D over pairs of examples x ∈ R d and labels y ∈
Revised: Given a data distribution D over pairs of examples x ∈ R d and labels y ∈

Original: [ k ] , parameters θ of a neural network, a set S ⊂ R d of allowable adversarial perturbations, and a loss function (cid:96) ( · , · , · ) dependent on the network parameters and the data, adversarial training amounts to considering a minmax optimization problem of the form min θ E ( x,y )
Revised: [ k ] , parameters θ of a neural network, a set S ⊂ R d of allowable adversarial perturbations, and a loss function (cid:96) ( · , · , · ) dependent on the network parameters and the data, adversarial training amounts to considering a minmax optimization problem of the form min θ E ( x,y )

Original: ∼D [max δ ∈S (cid:96) ( θ, x + δ, y )] .
Revised: ∼D [max δ ∈S (cid:96) ( θ, x + δ, y )] .

Original: In practice (Madry et al., 2018), the inner maximization problem max δ ∈S (cid:96) ( θ, x + δ, y ) is solved using projected gradient ascent.
Revised: In practice (Madry et al., 2018), the inner maximization problem max δ ∈S (cid:96) ( θ, x + δ, y ) is solved using projected gradient ascent.

Original: However, as described in Section 5, this is not a smooth algorithm and does not ﬁt our framework.
Revised: However, as described in Section 5, this is not a smooth algorithm and does not ﬁt our framework.

Original: So, we use gradient ascent, without projection , for the inner maximization.
Revised: So, we use gradient ascent, without projection , for the inner maximization.

Original: We run an adversarial training experiment with the MNIST dataset, a convolutional neural network, and the cross entropy loss function.
Revised: We run an adversarial training experiment with the MNIST dataset, a convolutional neural network, and the cross entropy loss function.

Original: We compare Algorithm 1 with usual adversarial training (Madry et al., 2018) which descends ∇ θ f
Revised: We compare Algorithm 1 with usual adversarial training (Madry et al., 2018) which descends ∇ θ f

Original: ( θ, A ( θ )) instead of ∇ f
Revised: ( θ, A ( θ )) instead of ∇ f

Original: ( θ, A ( θ )) , and a baseline of standard training without adversarial training.
Revised: ( θ, A ( θ )) , and a baseline of standard training without adversarial training.

Original: For each algorithm, we train for 100 passes over the training set using a batch size of 50.
Revised: For each algorithm, we train for 100 passes over the training set using a batch size of 50.

Original: The minimization procedure has a ﬁxed learning rate of η 1 = 0 . 0001 and the maximization procedure runs for T = 10 steps with a ﬁxed learning rate of η 2 = 4 .
Revised: The minimization procedure has a ﬁxed learning rate of η 1 = 0 . 0001 and the maximization procedure runs for T = 10 steps with a ﬁxed learning rate of η 2 = 4 .

Original: We evaluate the test classiﬁcation accuracy during the course of training against gradient ascent or Adam optimization adversarial attacks.
Revised: We evaluate the test classiﬁcation accuracy during the course of training against gradient ascent or Adam optimization adversarial attacks.

Original: The results are presented in Figure 4 where the mean accuracies are reported over 5 runs and the shaded regions show one standard deviation around the means.
Revised: The results are presented in Figure 4 where the mean accuracies are reported over 5 runs and the shaded regions show one standard deviation around the means.

Original: We observe that the adversarial training procedure gives a signiﬁcant boost in robustness compared to standard training.
Revised: We observe that the adversarial training procedure gives a signiﬁcant boost in robustness compared to standard training.

Original: Moreover, consistent with the previous experiments, our algorithm which uses total gradient outperforms standard adversarial training which uses only partial gradient.
Revised: Moreover, consistent with the previous experiments, our algorithm which uses total gradient outperforms standard adversarial training which uses only partial gradient.

Original: We present results against more attacks in Appendix G. As suggested in Section 5, we also ﬁnd that in practice, the gradient norms (cid:107)∇ f ( θ, A ( θ )) (cid:107) do not grow exponentially in the number of gradient ascent steps T in the adversary
Revised: We present results against more attacks in Appendix G. As suggested in Section 5, we also ﬁnd that in practice, the gradient norms (cid:107)∇ f ( θ, A ( θ )) (cid:107) do not grow exponentially in the number of gradient ascent steps T in the adversary

Original: algorithm A (see Figure 2).
Revised: algorithm A (see Figure 2).

Original: For further details and additional results see Appendix G.
Revised: For further details and additional results see Appendix G.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Motivated by the facts that (i) nonconcave maximization is NP-hard in the worst case and (ii) in practice, simple gradient based algorithms are used for maximization, this paper presents a new framework for solving nonconvex-nonconcave minimax optimization problems.
Revised: Motivated by the facts that (i) nonconcave maximization is NP-hard in the worst case and (ii) in practice, simple gradient based algorithms are used for maximization, this paper presents a new framework for solving nonconvex-nonconcave minimax optimization problems.

Original: Assuming that the min player has knowledge of the smooth algorithms being used by max player, we propose new efﬁcient algorithms and verify that these algorithms ﬁnd desirable solutions in practice on smallscale generative adversarial network and adversarial training problems.
Revised: Assuming that the min player has knowledge of the smooth algorithms being used by max player, we propose new efﬁcient algorithms and verify that these algorithms ﬁnd desirable solutions in practice on smallscale generative adversarial network and adversarial training problems.

Original: Furthermore, the framework proposed in this paper extends naturally to two player non-zero sum games as well.
Revised: Furthermore, the framework proposed in this paper extends naturally to two player non-zero sum games as well.

Original: There are several interesting directions for future  work (i) scaling our algorithm to large scale, state of the art GAN benchmarks and understanding the quality ofsolutions found by our algorithm in those settings, (ii) developing new techniques to deal with nonsmooth algorithms such as projected gradient ascent.
Revised: An important future direction of work is to scale our algorithm to large scale, state of the art GAN benchmarks. The key challenge here is that reinitializing the discriminator parameters necessitates running several steps of gradient ascent for discriminator for every generator update. One approach to overcome this could be to use pretrained networks for initializing the discriminator.

Original: A D ETAILED R ELATED W ORK
Revised: A D ETAILED R ELATED W ORK

Original: Nonconvex-Nonconcave Zero-Sum Games.
Revised: Nonconvex-Nonconcave Zero-Sum Games.

Original: The existing work on nonconvex-nonconcave zerosum games has generally focused on (1) deﬁning and characterizing local equilibrium solution concepts and (2) analyzing the local stability and convergence behavior of gradient-based learning algorithms around ﬁxed points of the dynamics.
Revised: The existing work on nonconvex-nonconcave zerosum games has generally focused on (1) deﬁning and characterizing local equilibrium solution concepts and (2) analyzing the local stability and convergence behavior of gradient-based learning algorithms around ﬁxed points of the dynamics.

Original: The concentration on local analysis stems from the inherent challenges that arise in nonconvex-nonconcave zero-sum games from both a dynamical systems perspective and a computational perspective.
Revised: The concentration on local analysis stems from the inherent challenges that arise in nonconvex-nonconcave zero-sum games from both a dynamical systems perspective and a computational perspective.

Original: In particular, it is know that broad classes of gradient-based learning dynamics can admit limit cycles and other non-trivial periodic orbits that are antithetical to any type of global convergence guarantee in this class of games (Hsieh et al., 2020; Letcher, 2021).
Revised: In particular, it is know that broad classes of gradient-based learning dynamics can admit limit cycles and other non-trivial periodic orbits that are antithetical to any type of global convergence guarantee in this class of games (Hsieh et al., 2020; Letcher, 2021).

Original: Moreover, on constrained domains, it has been shown that ﬁnding even a local equilibrium is computationally intractable (Daskalakis et al., 2021).
Revised: Moreover, on constrained domains, it has been shown that ﬁnding even a local equilibrium is computationally intractable (Daskalakis et al., 2021).

Original: A number of local equilibrium notions for nonconvex-nonconcave zero-sum games now exist with characterizations in terms of gradient-based conditions relevant to gradient-based learning.
Revised: A number of local equilibrium notions for nonconvex-nonconcave zero-sum games now exist with characterizations in terms of gradient-based conditions relevant to gradient-based learning.

Original: This includes the local Nash (Ratliff et al., 2013; Ratliff et al., 2016) and local minmax (Stackelberg)
Revised: This includes the local Nash (Ratliff et al., 2013; Ratliff et al., 2016) and local minmax (Stackelberg)

Original: (Jin et al., 2020; Fiez et al., 2020) equilibrium concepts, which both amount to local reﬁnements and characterizations of historically standard game-theoretic equilibrium notions.
Revised: (Jin et al., 2020; Fiez et al., 2020) equilibrium concepts, which both amount to local reﬁnements and characterizations of historically standard game-theoretic equilibrium notions.

Original: In terms of provable guarantees, algorithms incorporating higher-order gradient information have been proposed and analyzed that guarantee local convergence to only local Nash equilibria (Adolphs et al., 2019;
Revised: In terms of provable guarantees, algorithms incorporating higher-order gradient information have been proposed and analyzed that guarantee local convergence to only local Nash equilibria (Adolphs et al., 2019;

Original: Mazumdar et al., 2019) or local convergence to only local minmax equilibria (Wang et al., 2020;
Revised: Mazumdar et al., 2019) or local convergence to only local minmax equilibria (Wang et al., 2020;

Original: Zhang et al., 2020b; Fiez et al., 2020) in nonconvex-nonconcave zero-sum games.
Revised: Zhang et al., 2020b; Fiez et al., 2020) in nonconvex-nonconcave zero-sum games.

Original: Beyond the local Nash and minmax equilibrium, notions including the proximal equilibrium concept (Farnia &
Revised: Beyond the local Nash and minmax equilibrium, notions including the proximal equilibrium concept (Farnia &

Original: Ozdaglar, 2020), which is a class between the set of local Nash and local minmax equilibria, and the local robust equilibrium concept (Zhang et al., 2020a), which includes both local minmax and local maxmin equilibria, have been proposed and studied.
Revised: Ozdaglar, 2020), which is a class between the set of local Nash and local minmax equilibria, and the local robust equilibrium concept (Zhang et al., 2020a), which includes both local minmax and local maxmin equilibria, have been proposed and studied.

Original: It is worth noting that a shortcoming of each of the local equilibrium notions is that may fail to exist on unconstrained domains.
Revised: It is worth noting that a shortcoming of each of the local equilibrium notions is that may fail to exist on unconstrained domains.

Original: Signiﬁcant attention has been given to the local stability and convergence of simultaneous gradient descent-ascent in nonconvex-nonconcave zero-sum games.
Revised: Signiﬁcant attention has been given to the local stability and convergence of simultaneous gradient descent-ascent in nonconvex-nonconcave zero-sum games.

Original: This stems from the fact that it is the natural analogue of learning dynamics for zero-sum game optimization to gradient descent for function optimization.
Revised: This stems from the fact that it is the natural analogue of learning dynamics for zero-sum game optimization to gradient descent for function optimization.

Original: Moreover, simultaneous gradient descent-ascent is know to often perform reasonably well empirically and is ubiquitous in a number of applications such as in training generative adversarial networks and adversarial learning.
Revised: Moreover, simultaneous gradient descent-ascent is know to often perform reasonably well empirically and is ubiquitous in a number of applications such as in training generative adversarial networks and adversarial learning.

Original: However, it has been shown that while local Nash are guaranteed to be stable equilibria of simultaneous gradient descent-ascent (Mazumdar et al., 2020;
Revised: However, it has been shown that while local Nash are guaranteed to be stable equilibria of simultaneous gradient descent-ascent (Mazumdar et al., 2020;

Original: Daskalakis & Panageas, 2018b; Jin et al., 2020), local minmax may not be unless there is sufﬁcient timescale separation between the minimizing and maximizing players (Jin et al., 2020; Fiez & Ratliff, 2021).
Revised: Daskalakis & Panageas, 2018b; Jin et al., 2020), local minmax may not be unless there is sufﬁcient timescale separation between the minimizing and maximizing players (Jin et al., 2020; Fiez & Ratliff, 2021).

Original: Speciﬁc to generative adversarial networks, it has been shown that simultaneous gradient descent-ascent locally converges to local equilibria under certain assumptions on the generator network and the data distribution (Nagarajan & Kolter, 2017; Mescheder et al., 2018).
Revised: Speciﬁc to generative adversarial networks, it has been shown that simultaneous gradient descent-ascent locally converges to local equilibria under certain assumptions on the generator network and the data distribution (Nagarajan & Kolter, 2017; Mescheder et al., 2018).

Original: Later in this section we discuss in further detail learning dynamics studied previously in games which bear resemblance to that which we consider in this paper depending on the model of the maximizing player.
Revised: Later in this section we discuss in further detail learning dynamics studied previously in games which bear resemblance to that which we consider in this paper depending on the model of the maximizing player.

Original: The challenges of nonconvex-nonconcave zero-sum games we have highlighted limit the types of provable guarantees that can be obtained and consequently motivate tractable relaxations including to nonconvex-concave zero-sum games and the general framework we formulate in this work.
Revised: The challenges of nonconvex-nonconcave zero-sum games we have highlighted limit the types of provable guarantees that can be obtained and consequently motivate tractable relaxations including to nonconvex-concave zero-sum games and the general framework we formulate in this work.

Original: Before moving on, we mention that from a related perspective, a line of recent work (Keswani et al., 2020;
Revised: Before moving on, we mention that from a related perspective, a line of recent work (Keswani et al., 2020;

Original: Mangoubi & Vishnoi, 2021) in nonconvex-nonconcave zero-sum games proposes relaxed equilibrium notions that are shown to be computable in polynomial time and are guaranteed to exist.
Revised: Mangoubi & Vishnoi, 2021) in nonconvex-nonconcave zero-sum games proposes relaxed equilibrium notions that are shown to be computable in polynomial time and are guaranteed to exist.

Original: At a high level, the equilibria correspond to a joint strategy at which the maximizing player is at an approximate local maximum of the cost function and the minimizing player is at an approximate local minimum of a smoothed and relaxed best-response function of the maximizing player.
Revised: At a high level, the equilibria correspond to a joint strategy at which the maximizing player is at an approximate local maximum of the cost function and the minimizing player is at an approximate local minimum of a smoothed and relaxed best-response function of the maximizing player.

Original: The aforementioned works are similar to this paper in the sense that the minimizing player faces a maximizing player with computational restrictions, but diverge in terms of the model of the maximizing player and the algorithms for solving the problem.
Revised: The aforementioned works are similar to this paper in the sense that the minimizing player faces a maximizing player with computational restrictions, but diverge in terms of the model of the maximizing player and the algorithms for solving the problem.

Original: Nonconvex-Concave Zero-Sum Games.
Revised: Nonconvex-Concave Zero-Sum Games.

Original: The past few years has witnessed a signiﬁcant amount of work on gradient-based dynamics in nonconvex-concave zero-sum games.
Revised: The past few years has witnessed a signiﬁcant amount of work on gradient-based dynamics in nonconvex-concave zero-sum games.

Original: The focus of existing work on nonconvex-concave zero-sum games has key distinctions from that in nonconvex-nonconcave zero-sum games.
Revised: The focus of existing work on nonconvex-concave zero-sum games has key distinctions from that in nonconvex-nonconcave zero-sum games.

Original: Generally, the work on nonconvex-concave zero-sum games has analyzed dynamics on constrained domains, where typically the strategy space of the maximizing player is constrained to a closed convex set and occasionally the minimizing player also faces a constraint.
Revised: Generally, the work on nonconvex-concave zero-sum games has analyzed dynamics on constrained domains, where typically the strategy space of the maximizing player is constrained to a closed convex set and occasionally the minimizing player also faces a constraint.

Original: In contrast, nonconvex-nonconcave zero-sum games have generally been analyzed on unconstrained domains.
Revised: In contrast, nonconvex-nonconcave zero-sum games have generally been analyzed on unconstrained domains.

Original: Moreover, instead of focusing on computing notions of game-theoretic equilibrium as is typical in nonconvex-nonconcave zero-sum games, the body of work on nonconvex-concave zero-sum games has focused on achieving stationarity of the game cost function f ( · , · ) or the best-response function Φ( · ) = max y f
Revised: Moreover, instead of focusing on computing notions of game-theoretic equilibrium as is typical in nonconvex-nonconcave zero-sum games, the body of work on nonconvex-concave zero-sum games has focused on achieving stationarity of the game cost function f ( · , · ) or the best-response function Φ( · ) = max y f

Original: ( · , y ) .
Revised: ( · , y ) .

Original: The structure present in nonconvex-concave zero-sum games has been shown to simplify the problem compared to nonconvex-nonconcave zero-sum games so that global ﬁnite-time convergence guarantees are achievable.
Revised: The structure present in nonconvex-concave zero-sum games has been shown to simplify the problem compared to nonconvex-nonconcave zero-sum games so that global ﬁnite-time convergence guarantees are achievable.

Original: Thus, work in this direction has focused on improving the rates of convergence in terms of the gradient complexity to ﬁnd (cid:15) –approximate stationary points of f ( · , · ) or Φ( · ) , both with deterministic and stochastic gradients.
Revised: Thus, work in this direction has focused on improving the rates of convergence in terms of the gradient complexity to ﬁnd (cid:15) –approximate stationary points of f ( · , · ) or Φ( · ) , both with deterministic and stochastic gradients.

Original: Guarantees on the former notion of stationarity can be translated to guarantees on the latter notion of stationarity with extra computational cost (Lin et al., 2020a).
Revised: Guarantees on the former notion of stationarity can be translated to guarantees on the latter notion of stationarity with extra computational cost (Lin et al., 2020a).

Original: For the the class of nonconvex-strongly-concave zero-sum games, a series of works design algorithms that are shown to obtain (cid:15) –approximate stationary points of the functions f ( · , · ) or Φ( · ) with a gradient complexity of (cid:101) O
Revised: For the the class of nonconvex-strongly-concave zero-sum games, a series of works design algorithms that are shown to obtain (cid:15) –approximate stationary points of the functions f ( · , · ) or Φ( · ) with a gradient complexity of (cid:101) O

Original: ( (cid:15) − 2 ) in terms of (cid:15) in the deterministic setting (Jin et al., 2020; Raﬁque et al., 2021;
Revised: ( (cid:15) − 2 ) in terms of (cid:15) in the deterministic setting (Jin et al., 2020; Raﬁque et al., 2021;

Original: Lu et al., 2020; Lin et al., 2020a;b).
Revised: Lu et al., 2020; Lin et al., 2020a;b).

Original: In the deterministic nonconvex-strongly concave problem, the notions of stationarity are equivalent in terms of the dependence on (cid:15) up to a logarithmic dependence (Lin et al., 2020a).
Revised: In the deterministic nonconvex-strongly concave problem, the notions of stationarity are equivalent in terms of the dependence on (cid:15) up to a logarithmic dependence (Lin et al., 2020a).

Original: Lower bounds for this problem have also been established (Zhang et al., 2021; Li et al., 2021).
Revised: Lower bounds for this problem have also been established (Zhang et al., 2021; Li et al., 2021).

Original: In the stochastic nonconvex-strongly-concave problem, existing work has developed algorithms that are shown to obtain (cid:15) –approximate stationary points of the function Φ( · ) in gradient complexities of (cid:101) O
Revised: In the stochastic nonconvex-strongly-concave problem, existing work has developed algorithms that are shown to obtain (cid:15) –approximate stationary points of the function Φ( · ) in gradient complexities of (cid:101) O

Original: ( (cid:15) − 4 )
Revised: ( (cid:15) − 4 )

Original: (Raﬁque et al., 2021; Jin et al., 2020; Lin et al., 2020a) and (cid:101) O ( (cid:15) − 3 )
Revised: (Raﬁque et al., 2021; Jin et al., 2020; Lin et al., 2020a) and (cid:101) O ( (cid:15) − 3 )

Original: (Luo et al., 2020) in terms of (cid:15) dependence.
Revised: (Luo et al., 2020) in terms of (cid:15) dependence.

Original: In the deterministic nonconvex-concave problem, a number of algorithms with provable guarantees to (cid:15) –approximate stationary points of the function f ( · , · ) have been shown with gradient complexities of (cid:101) O
Revised: In the deterministic nonconvex-concave problem, a number of algorithms with provable guarantees to (cid:15) –approximate stationary points of the function f ( · , · ) have been shown with gradient complexities of (cid:101) O

Original: ( (cid:15) − 4 )
Revised: ( (cid:15) − 4 )

Original: (Lu et al., 2020), (cid:101) O
Revised: (Lu et al., 2020), (cid:101) O

Original: ( (cid:15) − 3 . 5 )
Revised: ( (cid:15) − 3 . 5 )

Original: (Nouiehed et al., 2019), and (cid:101) O ( (cid:15) − 2 . 5 ) (Ostrovskii et al., 2020;
Revised: (Nouiehed et al., 2019), and (cid:101) O ( (cid:15) − 2 . 5 ) (Ostrovskii et al., 2020;

Original: Lin et al., 2020b).
Revised: Lin et al., 2020b).

Original: Similarly, in this class of problems, there exist results on algorithms that guarantee convergence to an (cid:15) –approximate stationary points of the function Φ( · ) with gradient complexities (cid:101) O
Revised: Similarly, in this class of problems, there exist results on algorithms that guarantee convergence to an (cid:15) –approximate stationary points of the function Φ( · ) with gradient complexities (cid:101) O

Original: ( (cid:15) − 6 )
Revised: ( (cid:15) − 6 )

Original: (Raﬁque et al., 2021; Jin et al., 2020; Lin et al., 2020a) and (cid:101) O ( (cid:15) − 3 ) (Thekumparampil et al., 2019; Zhao, 2020; Kong & Monteiro, 2019; Lin et al., 2020b).
Revised: (Raﬁque et al., 2021; Jin et al., 2020; Lin et al., 2020a) and (cid:101) O ( (cid:15) − 3 ) (Thekumparampil et al., 2019; Zhao, 2020; Kong & Monteiro, 2019; Lin et al., 2020b).

Original: Finally, existing results in the stochastic setting for achieving an (cid:15) –approximate stationary point of Φ( · ) show gradient complexities of (cid:101)
Revised: Finally, existing results in the stochastic setting for achieving an (cid:15) –approximate stationary point of Φ( · ) show gradient complexities of (cid:101)

Original: O
Revised: O

Original: ( (cid:15) − 6 )
Revised: ( (cid:15) − 6 )

Original: (Raﬁque et al., 2021) and (cid:101) O ( (cid:15) [Table] , 2020a [Table]
Revised: (Raﬁque et al., 2021) and (cid:101) O ( (cid:15) − 8 ) (Lin et al., 2020a).

Original: In this work, we build on the developments for nonconvex-concave problems to obtain our results.
Revised: In this work, we build on the developments for nonconvex-concave problems to obtain our results.

Original: Gradient-Based Learning with Opponent Modeling.
Revised: Gradient-Based Learning with Opponent Modeling.

Original: A number of gradient-based learning schemes have been derived is various classes of games based on the following idea: if a player knows how the opponents in a game are optimizing their cost functions, then it is natural to account for this behavior in the players own optimization procedure.
Revised: A number of gradient-based learning schemes have been derived is various classes of games based on the following idea: if a player knows how the opponents in a game are optimizing their cost functions, then it is natural to account for this behavior in the players own optimization procedure.

Original: The simultaneous gradient descent learning dynamics can be viewed as the simplest instantiation of this perspective, where each player is optimizing their own cost function assuming that all other players in the game will remain ﬁxed.
Revised: The simultaneous gradient descent learning dynamics can be viewed as the simplest instantiation of this perspective, where each player is optimizing their own cost function assuming that all other players in the game will remain ﬁxed.

Original: In general, the more sophisticated existing learning dynamics based on opponent modeling assume the opponents are doing gradient descent on their cost function and this prediction is incorporated into the objective being optimized in place of the current strategies of opponents.
Revised: In general, the more sophisticated existing learning dynamics based on opponent modeling assume the opponents are doing gradient descent on their cost function and this prediction is incorporated into the objective being optimized in place of the current strategies of opponents.

Original: A key conceptual distinction between this approach and our work is that in existing opponent modeling methods the dynamics of the players are always updated simultaneously whereas the procedure we consider is sequential in nature with the opponent initializing again at each interaction.
Revised: A key conceptual distinction between this approach and our work is that in existing opponent modeling methods the dynamics of the players are always updated simultaneously whereas the procedure we consider is sequential in nature with the opponent initializing again at each interaction.

Original: Moreover, the types of guarantees we prove are distinct compared to existing work in this realm.
Revised: Moreover, the types of guarantees we prove are distinct compared to existing work in this realm.

Original: In this modern literature, gradient-based learning with opponent modeling dates back to the work of Zhang & Lesser (2010).
Revised: In this modern literature, gradient-based learning with opponent modeling dates back to the work of Zhang & Lesser (2010).

Original: They study simple two-player, two-action, general-sum matrix games, and analyze a set of learning dynamics called iterated descent descent with policy prediction (IGAPP) and show asymptotic convergence to a Nash equilibrium.
Revised: They study simple two-player, two-action, general-sum matrix games, and analyze a set of learning dynamics called iterated descent descent with policy prediction (IGAPP) and show asymptotic convergence to a Nash equilibrium.

Original: In this set of learning dynamics, each player assumes the other player is doing gradient descent and this prediction is used in the objective.
Revised: In this set of learning dynamics, each player assumes the other player is doing gradient descent and this prediction is used in the objective.

Original: In particular, each player i has a choice variable x
Revised: In particular, each player i has a choice variable x

Original: i and a cost function f i ( x i , x − i ) that after incorporating the prediction becomes f i ( x it , x − i [Equation] − γ ∇ − i f − i ( x it , x − i t )) .
Revised: i and a cost function f i ( x i , x − i ) that after incorporating the prediction becomes f i ( x it , x − i t − γ ∇ − i f − i ( x it , x − i t )) .

Original: To optimize the objective, each player takes a ﬁrst-order Taylor expansion of their cost function to give the augmented objective
Revised: To optimize the objective, each player takes a ﬁrst-order Taylor expansion of their cost function to give the augmented objective

Original: [Equation]
Revised: [Equation]

Original: Each player in the game simultaneously follows the gradient of their augmented objective which is given by
Revised: Each player in the game simultaneously follows the gradient of their augmented objective which is given by

Original: [Equation]
Revised: [Equation]

Original: This gradient computation is derived based on the fact that the assumed update of the other player ∇ − i f − i ( x it , x − i t ) does not depend on the optimization variable.
Revised: This gradient computation is derived based on the fact that the assumed update of the other player ∇ − i f − i ( x it , x − i t ) does not depend on the optimization variable.

Original: Similar ideas have recently been revisited in more general nonconvex multiplayer games (Foerster et al., 2018;
Revised: Similar ideas have recently been revisited in more general nonconvex multiplayer games (Foerster et al., 2018;

Original: Letcher et al., 2019).
Revised: Letcher et al., 2019).

Original: In learning with opponent learning awareness (LOLA)
Revised: In learning with opponent learning awareness (LOLA)

Original: (Foerster et al., 2018), players again assume the other players are doing gradient descent and take their objective to be f i ( x it , x − i t − γ ∇ − i f − i ( x it , x − i t )) .
Revised: (Foerster et al., 2018), players again assume the other players are doing gradient descent and take their objective to be f i ( x it , x − i t − γ ∇ − i f − i ( x it , x − i t )) .

Original: To derive the learning rule, an augmented objective is again formed by computing a ﬁrst-order Taylor expansion, but now the term ∇ − i f − i ( x it , x − i t ) in the augmented objective is treated as dependent on the optimization variable so that the gradient of the augmented objective is given by
Revised: To derive the learning rule, an augmented objective is again formed by computing a ﬁrst-order Taylor expansion, but now the term ∇ − i f − i ( x it , x − i t ) in the augmented objective is treated as dependent on the optimization variable so that the gradient of the augmented objective is given by

Original: [Equation]
Revised: [Equation]

Original: Finally, to arrive at the ﬁnal gradient update for each player, the middle term in the equation above is removed and each player takes steps along the gradient update
Revised: Finally, to arrive at the ﬁnal gradient update for each player, the middle term in the equation above is removed and each player takes steps along the gradient update

Original: [Equation]
Revised: [Equation]

Original: While no convergence results are given for LOLA, a follow-up work shows local convergence guarantees to stable ﬁxed points for IGA-PP and learning dynamics called stable opponent shaping (SOS) that interpolate between IGA-PP and LOLA (Letcher et al., 2019).
Revised: While no convergence results are given for LOLA, a follow-up work shows local convergence guarantees to stable ﬁxed points for IGA-PP and learning dynamics called stable opponent shaping (SOS) that interpolate between IGA-PP and LOLA (Letcher et al., 2019).

Original: A related work derives learning dynamic based on the idea that the opponent selects a best-response to the chosen strategy (Fiez et al., 2020).
Revised: A related work derives learning dynamic based on the idea that the opponent selects a best-response to the chosen strategy (Fiez et al., 2020).

Original: The resulting learning dynamics can be viewed as LOLA with the opponent selecting a
Revised: The resulting learning dynamics can be viewed as LOLA with the opponent selecting a

Original: Newton learning rate.
Revised: Newton learning rate.

Original: For nonconvex-nonconcave zero-sum games, local convergence guarantees to only local Stackelberg equilibrium are given in for this set of learning dynamics (Fiez et al., 2020).
Revised: For nonconvex-nonconcave zero-sum games, local convergence guarantees to only local Stackelberg equilibrium are given in for this set of learning dynamics (Fiez et al., 2020).

Original: It is worth remarking that gradient-based learning with opponent modeling is historically rooted in the general framework of consistent conjectural variations (see, e.g., Ba¸sar & Olsder 1998, Chapter 4.6), a concept that is now being explored again and is closely related to the previously mentioned learning dynamics (Chasnov et al., 2020).
Revised: It is worth remarking that gradient-based learning with opponent modeling is historically rooted in the general framework of consistent conjectural variations (see, e.g., Ba¸sar & Olsder 1998, Chapter 4.6), a concept that is now being explored again and is closely related to the previously mentioned learning dynamics (Chasnov et al., 2020).

Original: Perhaps the closest work on gradient-based learning with opponent modeling to this paper is that of unrolled generative adversarial networks (Metz et al., 2017).
Revised: Perhaps the closest work on gradient-based learning with opponent modeling to this paper is that of unrolled generative adversarial networks (Metz et al., 2017).

Original: In unrolled generative adversarial networks, the generator simulates the discriminator doing a ﬁxed number of gradient steps from the current parameter conﬁgurations of the generator and discriminator.
Revised: In unrolled generative adversarial networks, the generator simulates the discriminator doing a ﬁxed number of gradient steps from the current parameter conﬁgurations of the generator and discriminator.

Original: The resulting discriminator parameters are then used in place of the current discriminator parameters in the generator objective.
Revised: The resulting discriminator parameters are then used in place of the current discriminator parameters in the generator objective.

Original: The generator then updates following the gradient of this objective, optimizing through the rolled out discriminator update by computing the total derivative.
Revised: The generator then updates following the gradient of this objective, optimizing through the rolled out discriminator update by computing the total derivative.

Original: Simultaneously with the generator update, the discriminator updates its parameters by performing a gradient step on its objective.
Revised: Simultaneously with the generator update, the discriminator updates its parameters by performing a gradient step on its objective.

Original: In our framework, for generative adversarial networks when the discriminator is modeled as performing T -steps of gradient ascent, the procedure we propose is similar but an important difference is that when the generator simulates the discriminator unrolling procedure the discriminator parameters are initialized from scratch and there is no explicit discriminator being trained simultaneously with the generator.
Revised: In our framework, for generative adversarial networks when the discriminator is modeled as performing T -steps of gradient ascent, the procedure we propose is similar but an important difference is that when the generator simulates the discriminator unrolling procedure the discriminator parameters are initialized from scratch and there is no explicit discriminator being trained simultaneously with the generator.

Original: Extragradient methods : Starting from the seminal work of Korpelevich (1976), extragradient methods have been studied extensively for solving minimax optimization problems.
Revised: Extragradient methods : Starting from the seminal work of Korpelevich (1976), extragradient methods have been studied extensively for solving minimax optimization problems.

Original: While such methods have been shown to obtain faster convergence rates (Nemirovski, 2004) and ﬁnal iterate convergence (Daskalakis et al., 2018; Lei et al., 2021) for convex-concave minimax optimization, there is no result showing that it converges in general nonconvex-nonconcave settings.
Revised: While such methods have been shown to obtain faster convergence rates (Nemirovski, 2004) and ﬁnal iterate convergence (Daskalakis et al., 2018; Lei et al., 2021) for convex-concave minimax optimization, there is no result showing that it converges in general nonconvex-nonconcave settings.

Original: In fact, in
Revised: In fact, in

Original: Appendix F we present simple examples demonstrating that the extragradient method as well as another variant called optimistic gradient descent ascent, do not converge for general nonconvexnonconcave minimax optimization.
Revised: Appendix F we present simple examples demonstrating that the extragradient method as well as another variant called optimistic gradient descent ascent, do not converge for general nonconvexnonconcave minimax optimization.

Original: Games with computationally bounded adversaries : There are also a few works in the game theory literature which consider resource/computationally bounded agents.
Revised: Games with computationally bounded adversaries : There are also a few works in the game theory literature which consider resource/computationally bounded agents.

Original: For example Freund et al.
Revised: For example Freund et al.

Original: (1995) considers repeated games between resource bounded agents, (Sandhlom & Lesser, 1997) considers coalition formation between resource bounded agents in cooperative games and Halpern et al.
Revised: (1995) considers repeated games between resource bounded agents, (Sandhlom & Lesser, 1997) considers coalition formation between resource bounded agents in cooperative games and Halpern et al.

Original: (2014) shows that resource constraints in otherwise rational players might lead to some commonly observed human behaviors while making decisions.
Revised: (2014) shows that resource constraints in otherwise rational players might lead to some commonly observed human behaviors while making decisions.

Original: However, the settings, models of limited computation and the focus of results considered in all of these prior works are distinct from those of this paper.
Revised: However, the settings, models of limited computation and the focus of results considered in all of these prior works are distinct from those of this paper.

Original: Stability of algorithms in numerical analysis : To our knowledge such results on the smoothness of the classes of algorithms we study—i.e., gradient-based updates such as SGA and SNAG—with respect to problem parameters (e.g., in this case, x ) have not been shown in the machine learning and optimization literature.
Revised: Stability of algorithms in numerical analysis : To our knowledge such results on the smoothness of the classes of algorithms we study—i.e., gradient-based updates such as SGA and SNAG—with respect to problem parameters (e.g., in this case, x ) have not been shown in the machine learning and optimization literature.

Original: This being said, in the study of dynamical systems—more speciﬁcally differential equations—the concept of continuity (and Lipschitzness) with respect to parameters and initial data has been studied using a variational approach wherein the continuity of the solution of the differential equation is shown to be continuous with respect to variations in the parameters or initial data by appealing to nonlinear variation of parameters results such as the Bellman-Grownwall inequality or Alekseev’s theorem (see classical references on differential equations such as Coddington & Levinson (1955, Chapter 2) or Hartman (2002, Chapter IV.2)).
Revised: This being said, in the study of dynamical systems—more speciﬁcally differential equations—the concept of continuity (and Lipschitzness) with respect to parameters and initial data has been studied using a variational approach wherein the continuity of the solution of the differential equation is shown to be continuous with respect to variations in the parameters or initial data by appealing to nonlinear variation of parameters results such as the Bellman-Grownwall inequality or Alekseev’s theorem (see classical references on differential equations such as Coddington & Levinson (1955, Chapter 2) or Hartman (2002, Chapter IV.2)).

Original: In numerical methods, such results on the “smoothness” or continuity of the differential equation with respect to initial data or problem parameters are used to understand stability of particular numerical methods (see, e.g., Atkinson et al. 2011, Chapter 1.2).
Revised: In numerical methods, such results on the “smoothness” or continuity of the differential equation with respect to initial data or problem parameters are used to understand stability of particular numerical methods (see, e.g., Atkinson et al. 2011, Chapter 1.2).

Original: In particular, a initial value problem is only considered well-posed if there is continuous dependence on initial data.
Revised: In particular, a initial value problem is only considered well-posed if there is continuous dependence on initial data.

Original: For instance, the simple scalar differential equation
Revised: For instance, the simple scalar differential equation

Original: [Equation]
Revised: [Equation]

Original: has solution y ( t ) ≡ 1 , yet the perturbed problem,
Revised: has solution y ( t ) ≡ 1 , yet the perturbed problem,

Original: [Equation]
Revised: [Equation]

Original: has solution y (cid:15) (
Revised: has solution y [Equation]

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: + (cid:15)e − t so that
Revised: (cid:15)e − t so that

Original: [Equation]
Revised: [Equation]

Original: If the maximum error (cid:107) y (cid:15) − y (cid:107) ∞ is (much) larger than (cid:15) then the initial value problem is ill-conditioned and any typical attempt to numerically solve such a problem will lead to large errors in the computed solution.
Revised: If the maximum error (cid:107) y (cid:15) − y (cid:107) ∞ is (much) larger than (cid:15) then the initial value problem is ill-conditioned and any typical attempt to numerically solve such a problem will lead to large errors in the computed solution.

Original: In short, the stability properties of a numerical method (i.e., discretization of the differential equation) are fundamentally connected to the continuity (smoothness) with respect to initial data.
Revised: In short, the stability properties of a numerical method (i.e., discretization of the differential equation) are fundamentally connected to the continuity (smoothness) with respect to initial data.

Original: Observe that methods such as gradient ascent can be viewed as a discretization of an differential equation:
Revised: Observe that methods such as gradient ascent can be viewed as a discretization of an differential equation:

Original: [Equation]
Revised: [Equation]

Original: As such, the techniques for showing continuity of the solution of a differential equation with respect to initial data or other problem parameters (e.g., in this case x ) can be adopted to show smoothness of the T -step solution of the discretized update.
Revised: As such, the techniques for showing continuity of the solution of a differential equation with respect to initial data or other problem parameters (e.g., in this case x ) can be adopted to show smoothness of the T -step solution of the discretized update.

Original: Our approach to showing smoothness, on the other hand, leverages the recursive nature of the discrete time updates deﬁning the classes of algorithms we study.
Revised: Our approach to showing smoothness, on the other hand, leverages the recursive nature of the discrete time updates deﬁning the classes of algorithms we study.

Original: This approach simpliﬁes the analysis by directly going after the smoothness parameters using the update versus solving the difference (or differential) equation for y T ( x ) and then ﬁnding the smoothness parameters which is the method typically used in numerical analysis of differential equations.
Revised: This approach simpliﬁes the analysis by directly going after the smoothness parameters using the update versus solving the difference (or differential) equation for y T ( x ) and then ﬁnding the smoothness parameters which is the method typically used in numerical analysis of differential equations.

Original: An interesting direction of future research is to more formally connect the stability analysis from numerical analysis of differential equations to robustness of adversarial learning to initial data and even variations in problem parameters.
Revised: An interesting direction of future research is to more formally connect the stability analysis from numerical analysis of differential equations to robustness of adversarial learning to initial data and even variations in problem parameters.

Original: B P ROOF OF RESULTS IN S ECTION
Revised: B P ROOF OF RESULTS IN S ECTION

Original: Proof of Lemma 1.
Revised: Proof of Lemma 1.

Original: For any ﬁxed z , we note that A ( · , z ) is a deterministic algorithm.
Revised: For any ﬁxed z , we note that A ( · , z ) is a deterministic algorithm.

Original: Consequently, it sufﬁces to prove the lemma for a deterministic algorithm A ( · ) .
Revised: Consequently, it sufﬁces to prove the lemma for a deterministic algorithm A ( · ) .

Original: By chain rule, the derivative of f ( x, A ( x )) is given by:
Revised: By chain rule, the derivative of f ( x, A ( x )) is given by:

Original: [Equation]
Revised: [Equation]

Original: where D A ( x ) ∈ R d 1 ×
Revised: where D A ( x ) ∈ R d 1 ×

Original: d 2 is the derivative of A ( · ) : R d 1 → R d 2 at x and ∇ x f ( x, A ( x )) and ∇ y f ( x, A ( x )) denote the partial derivatives of f with respect to the ﬁrst and second variables respectively at ( x, A ( x )) .
Revised: d 2 is the derivative of A ( · ) : R d 1 → R d 2 at x and ∇ x f ( x, A ( x )) and ∇ y f ( x, A ( x )) denote the partial derivatives of f with respect to the ﬁrst and second variables respectively at ( x, A ( x )) .

Original: An easy computation shows that
Revised: An easy computation shows that

Original: [Equation]
Revised: [Equation]

Original: This shows that f ( x, A ( x )) is (1 + G (cid:48) )
Revised: This shows that f ( x, A ( x )) is (1 + G (cid:48) )

Original: G -Lipschitz.
Revised: G -Lipschitz.

Original: Similarly, we have:
Revised: Similarly, we have:

Original: [Equation]
Revised: [Equation]

Original: For the ﬁrst term
Revised: For the ﬁrst term

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Similarly, for the second term we have:
Revised: Similarly, for the second term we have:

Original: [Equation]
Revised: [Equation]

Original: This proves the lemma.
Revised: This proves the lemma.

Original: Proof of Lemma 2.
Revised: Proof of Lemma 2.

Original: Given any x and y
Revised: Given any x and y

Original: [Equation]
Revised: [Equation]

Original: and any λ such that λ j ≥ 0 and (cid:80) j ∈ S ( x ) λ j =
Revised: and any λ such that λ j ≥ 0 and (cid:80) j ∈ S ( x ) λ j =

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: proving the lemma.
Revised: proving the lemma.

Original: Proof of Lemma 3.
Revised: Proof of Lemma 3.

Original: We re-write f λ ( x ) as minimum value of a ( 1 λ − L ) -strong convex function φ λ,x , as g is L -weakly convex (Deﬁnition 2) and 12 λ (cid:107) x − x (cid:48) (cid:107) 2 is differentiable and 1 λ -strongly convex, g λ ( x ) =
Revised: We re-write f λ ( x ) as minimum value of a ( 1 λ − L ) -strong convex function φ λ,x , as g is L -weakly convex (Deﬁnition 2) and 12 λ (cid:107) x − x (cid:48) (cid:107) 2 is differentiable and 1 λ -strongly convex, g λ ( x ) =

Original: min x (cid:48)
Revised: min x (cid:48)

Original: [Equation]
Revised: [Equation]

Original: Then ﬁrst part of (a) follows trivially by the strong convexity.
Revised: Then ﬁrst part of (a) follows trivially by the strong convexity.

Original: For the second part notice the following,
Revised: For the second part notice the following,

Original: [Equation]
Revised: [Equation]

Original: Furthermore, we will show that arg min x g λ ( x ) =
Revised: Furthermore, we will show that arg min x g λ ( x ) =

Original: arg min x g ( x ) .
Revised: arg min x g ( x ) .

Original: Consider x ∗ ∈ argmin x g ( x ) .
Revised: Consider x ∗ ∈ argmin x g ( x ) .

Original: We have that for any x ,
Revised: We have that for any x ,

Original: [Equation]
Revised: [Equation]

Original: This shows that x ∗ ∈ argmin x g λ ( x ) and hence argmin x g ( x ) ⊆
Revised: This shows that x ∗ ∈ argmin x g λ ( x ) and hence argmin x g ( x ) ⊆

Original: argmin x g λ ( x ) .
Revised: argmin x g λ ( x ) .

Original: Similarly, for any x ∗ ∈ argmin x g λ ( x ) , we have:
Revised: Similarly, for any x ∗ ∈ argmin x g λ ( x ) , we have:

Original: [Equation]
Revised: [Equation]

Original: Input: point x , stochastic subgradient oracle for function g , error (cid:15) , failure probability δ 1 Find (cid:101) x such that g ( (cid:101) x ) + L (cid:107) x − (cid:101) x (cid:107) 2 ≤ (cid:0) min x (cid:48) g
Revised: Input: point x , stochastic subgradient oracle for function g , error (cid:15) , failure probability δ 1 Find (cid:101) x such that g ( (cid:101) x ) + L (cid:107) x − (cid:101) x (cid:107) 2 ≤ (cid:0) min x (cid:48) g

Original: ( x (cid:48) )
Revised: ( x (cid:48) )

Original: + L (cid:107) x − x (cid:48) (cid:107) 2 (cid:1)
Revised: + L (cid:107) x − x (cid:48) (cid:107) 2 (cid:1)

Original: + (cid:15) 2 4 L using (Harvey et al., 2019, Algorithm 1).
Revised: + (cid:15) 2 4 L using (Harvey et al., 2019, Algorithm 1).

Original: 2 return 2 L ( x − (cid:101) x ) .
Revised: 2 return 2 L ( x − (cid:101) x ) .

Original: Since x ∗ ∈ argmin x g λ ( x ) , the above inequality is infact an equality and hence ˆ x λ ( x ∗ ) = x ∗ .
Revised: Since x ∗ ∈ argmin x g λ ( x ) , the above inequality is infact an equality and hence ˆ x λ ( x ∗ ) = x ∗ .

Original: So, g λ ( x ∗ ) =
Revised: So, g λ ( x ∗ ) =

Original: g ( x ∗ ) and hence x ∗ ∈ argmin x g ( x ) implying that argmin x g λ ( x ) ⊆
Revised: g ( x ∗ ) and hence x ∗ ∈ argmin x g ( x ) implying that argmin x g λ ( x ) ⊆

Original: argmin x g ( x ) .
Revised: argmin x g ( x ) .

Original: Thus arg min x g λ ( x ) =
Revised: Thus arg min x g λ ( x ) =

Original: arg min x g ( x ) .
Revised: arg min x g ( x ) .

Original: For ( b ) we can re-write the Moreau envelope g λ as,
Revised: For ( b ) we can re-write the Moreau envelope g λ as,

Original: [Equation]
Revised: [Equation]

Original: where ( · ) ∗ is the Fenchel conjugation operator.
Revised: where ( · ) ∗ is the Fenchel conjugation operator.

Original: Since L < 1 /λ , using L -weak convexity of g , it is easy to see that λg ( x (cid:48) )
Revised: Since L < 1 /λ , using L -weak convexity of g , it is easy to see that λg ( x (cid:48) )

Original: + (cid:107) x (cid:48) (cid:107) 2is
Revised: + (cid:107) x (cid:48) (cid:107) 2is

Original: (1 − λL ) -strongly convex, therefore its Fenchel conjugate would be(1 − λL ) -smooth
Revised: (1 − λL ) -strongly convex, therefore its Fenchel conjugate would be(1 − λL ) -smooth

Original: (Kakade et al., 2009, Theorem 6).
Revised: (Kakade et al., 2009, Theorem 6).

Original: This, along with 1 λ -smoothness of ﬁrst quadratic term implies that g λ ( x ) is (cid:0) 1 λ + 1 λ (1 − λL ) (cid:1) -smooth, and thus differentiable.
Revised: This, along with 1 λ -smoothness of ﬁrst quadratic term implies that g λ ( x ) is (cid:0) 1 λ + 1 λ (1 − λL ) (cid:1) -smooth, and thus differentiable.

Original: For ( c ) we proceed as follows.
Revised: For ( c ) we proceed as follows.

Original: Since ˆ x λ ( x ) = argmin x (cid:48) φ λ,x ( x (cid:48) ) , by ﬁrst order KKT optimality conditions, we have that 
Revised: Since ˆ x λ ( x ) = argmin x (cid:48) φ λ,x ( x (cid:48) ) , by ﬁrst order KKT optimality conditions, we have that [Equation]

Original: [Equation]
Revised: [Equation]

Original: Further, from proof of part (a) we have that φ λ,x
Revised: Further, from proof of part (a) we have that φ λ,x

Original: ( x (cid:48) ) (1 − λL ) -strongly-convex in x (cid:48) and it is quadratic (and thus convex) in x .
Revised: ( x (cid:48) ) (1 − λL ) -strongly-convex in x (cid:48) and it is quadratic (and thus convex) in x .

Original: Danskin’s theorem (Bertsekas, 2009, Section 6.11) thus implies that for g λ ( x ) =
Revised: Danskin’s theorem (Bertsekas, 2009, Section 6.11) thus implies that for g λ ( x ) =

Original: min x (cid:48) ∈
Revised: min x (cid:48) ∈

Original: R
Revised: R

Original: d 1 φ λ,x ( x (cid:48) ) , we have that ∇ g λ ( x ) =
Revised: d 1 φ λ,x ( x (cid:48) ) , we have that ∇ g λ ( x ) =

Original: ( x − ˆ x λ ( x ))
Revised: ( x − ˆ x λ ( x ))

Original: /λ .
Revised: /λ .

Original: Letting ˆ u λ ( x ) :
Revised: Letting ˆ u λ ( x ) :

Original: = λ − 1 ( x − ˆ x λ ( x )) ∈ ∂g
Revised: = λ − 1 ( x − ˆ x λ ( x )) ∈ ∂g

Original: ( x ) , we have that:
Revised: ( x ) , we have that:

Original: [Equation]
Revised: [Equation]

Original: C P ROOFS OF R ESULTS IN S ECTION 4. 
Revised: C P ROOFS OF R ESULTS IN S ECTION 4.

Original: In order to prove convergence of this algorithm, we ﬁrst recall the following result from (Davis &
Revised: In order to prove convergence of this algorithm, we ﬁrst recall the following result from (Davis &

Original: Drusvyatskiy, 2018).
Revised: Drusvyatskiy, 2018).

Original: Theorem 7 (Corollary 2.2 from (Davis & Drusvyatskiy, 2018)) .
Revised: Theorem 7 (Corollary 2.2 from (Davis & Drusvyatskiy, 2018)) .

Original: Suppose g ( · ) is L -weakly convex, and E z 1 , ··· ,z k (cid:104) (cid:107) (cid:98) ∇ g ( x ) (cid:107) 2 (cid:105) ≤ G 2 .
Revised: Suppose g ( · ) is L -weakly convex, and E z 1 , ··· ,z k (cid:104) (cid:107) (cid:98) ∇ g ( x ) (cid:107) 2 (cid:105) ≤ G 2 .

Original: Then, the output ¯ x of Algorithm 1 with stepsize η = γ √ S + satisﬁes:
Revised: Then, the output ¯ x of Algorithm 1 with stepsize η = γ √ S + satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: Lemmas 1 and 2 tell us that g ( x ) is (cid:98) L -weakly convex and for any choice of z , the stochastic subgradient (cid:98) ∇ g ( x ) is bounded in norm by (cid:98) G .
Revised: Lemmas 1 and 2 tell us that g ( x ) is (cid:98) L -weakly convex and for any choice of z , the stochastic subgradient (cid:98) ∇ g ( x ) is bounded in norm by (cid:98) G .

Original: Consequently, Theorem 7 with the stated choice of S proves Theorem 1.
Revised: Consequently, Theorem 7 with the stated choice of S proves Theorem 1.

Original: Proposition 1.
Revised: Proposition 1.

Original: Given a point x , an L -weakly convex function g and a G -norm bounded and a stochastic subgradient oracle to g (i.e., given any point x (cid:48) , which returns a stochastic vector u such that E [ u ] ∈ ∂g
Revised: Given a point x , an L -weakly convex function g and a G -norm bounded and a stochastic subgradient oracle to g (i.e., given any point x (cid:48) , which returns a stochastic vector u such that E [ u ] ∈ ∂g

Original: ( x (cid:48) ) and (cid:107) u (cid:107) ≤ G ), with probability at least 1 − δ , Algorithm 3 returns a vector u satisfying (cid:107) u − ∇ g λ ( x ) (cid:107) ≤ (cid:15) with at most O (cid:16) G 2 log 

Revised: ( x (cid:48) ) and (cid:107) u (cid:107) ≤ G ), with probability at least 1 − δ , Algorithm 3 returns a vector u satisfying (cid:107) u − ∇ g λ ( x ) (cid:107) ≤ (cid:15) with at most O (cid:16) G 2 log

Original: δ (cid:15)(cid:17) queries to the stochastic subgradient oracle of g .
Revised: δ (cid:15)(cid:17) queries to the stochastic subgradient oracle of g .

Original: Proof of Proposition 1.
Revised: Proof of Proposition 1.

Original: Let Φ 12 L ( x (cid:48) , x ) :
Revised: Let Φ 12 L ( x (cid:48) , x ) :

Original: = g ( x (cid:48) )
Revised: = g ( x (cid:48) )

Original: + L (cid:107) x − x (cid:48) (cid:107) 2 .
Revised: + L (cid:107) x − x (cid:48) (cid:107) 2 .

Original: Recall the notation of Lemma(cid:98)
Revised: Recall the notation of Lemma(cid:98)

Original: x
Revised: x

Original: 12 L ( x ) :=
Revised: 12 L ( x ) :=

Original: argmin x (cid:48)
Revised: argmin x (cid:48)

Original: Φ 12 L ( x (cid:48) , x ) and g λ ( x ) =
Revised: Φ 12 L ( x (cid:48) , x ) and g λ ( x ) =

Original: min x (cid:48)
Revised: min x (cid:48)

Original: Φ 12 L ( x (cid:48) , x ) .
Revised: Φ 12 L ( x (cid:48) , x ) .

Original: The proof of Lemma 3 tells us that∇ g 12 L (
Revised: The proof of Lemma 3 tells us that

Original: [Equation]
Revised: [Equation]

Original: = [Equation] −
Revised: L (

Original: [Equation]
Revised: [Equation]

Original: x 12 L ( x ) λ and also that (cid:107) (cid:98) x 12 L ( x ) − x (cid:107) ≤ G 2 L .
Revised: and also that (cid:107) (cid:98) x 12 L ( x ) − x (cid:107) ≤ G 2 L .

Original: Since Φ 12 L ( · , x ) is a L -strongly convex and G Lipschitz function in the domain (cid:110) x (cid:48) : (cid:107) (cid:98) x 12 L ( x ) − x (cid:107) ≤ G 2 L (cid:111) , (Harvey et al., 2019, Theorem 3.1) tells us that we can use SGD with O (cid:16) G 2 log 1 δ L (cid:101) (cid:15) (cid:17) stochastic gradient oracle queries to implement
Revised: Since Φ 12 L ( · , x ) is a L -strongly convex and G Lipschitz function in the domain (cid:110) x (cid:48) : (cid:107) (cid:98) x 12 L ( x ) − x (cid:107) ≤ G 2 L (cid:111) , (Harvey et al., 2019, Theorem 3.1) tells us that we can use SGD with O (cid:16) G 2 log 1 δ L (cid:101) (cid:15) (cid:17) stochastic gradient oracle queries to implement

Original: Step 1 of Algorithm 3 with success probability at least 1 − δ , where (cid:101) (cid:15) :
Revised: Step 1 of Algorithm 3 with success probability at least 1 − δ , where (cid:101) (cid:15) :

Original: = (cid:15) 2 4 L .
Revised: = (cid:15) 2 4 L .

Original: Simplifying this expression gives us a stochastic gradient oracle complexity of O (cid:16) G 2 log 1 δ (cid:15)(cid:17) .
Revised: Simplifying this expression gives us a stochastic gradient oracle complexity of O (cid:16) G 2 log 1 δ (cid:15)(cid:17) .

Original: D P ROOFS OF R ESULTS IN S ECTION 4. 
Revised: D P ROOFS OF R ESULTS IN S ECTION 4.

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: Letting h ( x, λ ) :
Revised: Letting h ( x, λ ) :

Original: = (cid:80) i ∈
Revised: = (cid:80) i ∈

Original: [ k ] λ i g i ( x ) , we note that (cid:107)∇ xx h ( x, λ ) (cid:107) =
Revised: [ k ] λ i g i ( x ) , we note that (cid:107)∇ xx h ( x, λ ) (cid:107) =

Original: (cid:107) (cid:80) i ∈
Revised: (cid:107) (cid:80) i ∈

Original: [ k ] λ i ∇ 2 g
Revised: [ k ] λ i ∇ 2 g

Original: i
Revised: i

Original: ( x ) (cid:107) ≤ L (1 + G (cid:48) )
Revised: ( x ) (cid:107) ≤ L (1 + G (cid:48) )

Original: 2 + GL (cid:48) , where we used Lemma 1 and the fact that (cid:80) i | λ i | ≤ 1 .
Revised: 2 + GL (cid:48) , where we used Lemma 1 and the fact that (cid:80) i | λ i | ≤ 1 .

Original: On the other hand, (cid:107)∇ xλ h ( x, λ ) = (cid:107) (cid:80) i ∈
Revised: On the other hand, (cid:107)∇ xλ h ( x, λ ) = (cid:107) (cid:80) i ∈

Original: [ k ] ∇ g i ( x ) (cid:107)(cid:107) ≤ kG (1 + G (cid:48) ) , where we again used
Revised: [ k ] ∇ g i ( x ) (cid:107)(cid:107) ≤ kG (1 + G (cid:48) ) , where we again used

Original: Lemma 1.
Revised: Lemma 1.

Original: Since ∇ λλ h ( x, λ ) = 0 , we can conclude that h is an (cid:98) L -gradient Lipschitz function with (cid:98) L :=
Revised: Since ∇ λλ h ( x, λ ) = 0 , we can conclude that h is an (cid:98) L -gradient Lipschitz function with (cid:98) L :=

Original: L (1 + G (cid:48) )
Revised: L (1 + G (cid:48) )

Original: 2 + GL (cid:48)
Revised: 2 + GL (cid:48)

Original: + kG (1 + G (cid:48) ) .
Revised: + kG (1 + G (cid:48) ) .

Original: Consequently, g ( x ) = max λ ∈ S h
Revised: Consequently, g ( x ) = max λ ∈ S h

Original: ( x, λ ) , where
Revised: ( x, λ ) , where

Original: S :=
Revised: S :=

Original: (cid:110)
Revised: (cid:110)

Original: λ ∈ R
Revised: λ ∈ R

Original: k : λ i ≥ 0 , (cid:80) i ∈
Revised: k : λ i ≥ 0 , (cid:80) i ∈

Original: [ k ] λ i = 1 (cid:111) , is (cid:98) L -weakly convex and the Moreau envelope
Revised: [ k ] λ i = 1 (cid:111) , is (cid:98) L -weakly convex and the Moreau envelope

Original: g 12 (cid:98) L is well deﬁned.
Revised: g 12 (cid:98) L is well deﬁned.

Original: Denote (cid:98) g ( x, x s ) :
Revised: Denote (cid:98) g ( x, x s ) :

Original: = max i ∈
Revised: = max i ∈

Original: [ k ] g i ( x ) + (cid:98) L (cid:107) x − x s (cid:107) 2 .
Revised: [ k ] g i ( x ) + (cid:98) L (cid:107) x − x s (cid:107) 2 .

Original: We now divide the analysis of each of iteration of Algorithm 2 into two cases.
Revised: We now divide the analysis of each of iteration of Algorithm 2 into two cases.

Original: Case I, (cid:98) g ( x s +1 , x s ) ≤ max i ∈ [ k ] g i ( x s ) − 3 (cid:98) (cid:15)/ 4 : Since max i ∈ [ k ] g i ( x s +1 ) ≤ (cid:98) g ( x s +1 , x s ) ≤ max i ∈ [ k ] g i ( x s ) − 3 (cid:98) (cid:15)/ 4 , we see that in this case g ( x s ) decreases monotonically by at least 3 (cid:98) (cid:15)/in each iteration.
Revised: Case I, (cid:98) g ( x s +1 , x s ) ≤ max i ∈ [ k ] g i ( x s ) − 3 (cid:98) (cid:15)/ 4 : Since max i ∈ [ k ] g i ( x s +1 ) ≤ (cid:98) g ( x s +1 , x s ) ≤ max i ∈ [ k ] g i ( x s ) − 3 (cid:98) (cid:15)/ 4 , we see that in this case g ( x s ) decreases monotonically by at least 3 (cid:98) (cid:15)/in each iteration.

Original: Since by Assumption 1, g is bounded by B in magnitude, and the termination condition in Step 4 guarantees monotonic decrease in every iteration, there can only be at most 2 B/ (3 (cid:98) (cid:15)/ 4) = 8 B/ (cid:98) (cid:15) such iterations in Case I.
Revised: Since by Assumption 1, g is bounded by B in magnitude, and the termination condition in Step 4 guarantees monotonic decrease in every iteration, there can only be at most 2 B/ (3 (cid:98) (cid:15)/ 4) = 8 B/ (cid:98) (cid:15) such iterations in Case I.

Original: Case II, (cid:98) g
Revised: Case II, (cid:98) g

Original: ( x s +1 , x s ) ≥ max i ∈ [ k ] g i ( x s ) − 3 (cid:98) (cid:15)/ 4 :
Revised: ( x s +1 , x s ) ≥ max i ∈ [ k ] g i ( x s ) − 3 (cid:98) (cid:15)/ 4 :

Original: In this case, we claim that x s is an (cid:15) -FOSP of g = max i ∈ [ k ] g i ( x ) .
Revised: In this case, we claim that x s is an (cid:15) -FOSP of g = max i ∈ [ k ] g i ( x ) .

Original: To see this, we ﬁrst note that
Revised: To see this, we ﬁrst note that

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Let x ∗ s :=
Revised: Let x ∗ s :=

Original: arg min x (cid:98) g ( x ; x k ) .
Revised: arg min x (cid:98) g ( x ; x k ) .

Original: Since g is (cid:98) L -gradient Lipschitz, we note that (cid:98) g ( · ; x s ) is (cid:98) L -strongly convex.
Revised: Since g is (cid:98) L -gradient Lipschitz, we note that (cid:98) g ( · ; x s ) is (cid:98) L -strongly convex.

Original: We now use this to prove that x s is close to x ∗ s :
Revised: We now use this to prove that x s is close to x ∗ s :

Original: [Equation]
Revised: [Equation]

Original: where ( a ) uses (12).
Revised: where ( a ) uses (12).

Original: Now consider any (cid:98) x , such that 4 (cid:112)(cid:98) (cid:15)/L ≤ (cid:107) (cid:98) x − x s (cid:107) .
Revised: Now consider any (cid:98) x , such that 4 (cid:112)(cid:98) (cid:15)/L ≤ (cid:107) (cid:98) x − x s (cid:107) .

Original: Then,
Revised: Then,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  where ( a ) uses uses (cid:98) L -strong convexity of (cid:98) g ( · ; x s ) at its minimizer x ∗ s , ( b ) uses (12), and ( b ) and ( c ) use triangle inequality, (13) and 4 (cid:113) (cid:98) (
Revised: ) where ( a ) uses uses (cid:98) L -strong convexity of (cid:98) g ( · ; x s ) at its minimizer x ∗ s , ( b ) uses (12), and ( b ) and ( c ) use triangle inequality, (13) and 4 (cid:113) (cid:98) (

Original: cid:15)/ (cid:98) L ≤ (cid:107) (cid:98) x − x s (cid:107) .
Revised: cid:15)/ (cid:98) L ≤ (cid:107) (cid:98) x − x s (cid:107) .

Original: Now consider the Moreau envelope, g 1 2 (cid:98) L ( x ) =
Revised: Now consider the Moreau envelope, g 1 2 (cid:98) L ( x ) =

Original: min x (cid:48) φ 1 2 (cid:98) L ,x ( x (cid:48) )
Revised: min x (cid:48) φ 1 2 (cid:98) L ,x ( x (cid:48) )

Original: where φ 1 2 (cid:98) L ,x ( x (cid:48) )
Revised: where φ 1 2 (cid:98) L ,x ( x (cid:48) )

Original: =
Revised: =

Original: g ( x (cid:48) )
Revised: g ( x (cid:48) )

Original: + L (cid:107) x − x (cid:48) (cid:107) 2 .
Revised: + L (cid:107) x − x (cid:48) (cid:107) 2 .

Original: Then, we can see that φ 12 (cid:98) L ,x s ( x (cid:48) ) achieves its minimum in the ball { x (cid:48) | (cid:107) x (cid:48) − x s (cid:107)
Revised: Then, we can see that φ 12 (cid:98) L ,x s ( x (cid:48) ) achieves its minimum in the ball { x (cid:48) | (cid:107) x (cid:48) − x s (cid:107)

Original: [Equation]
Revised: [Equation]

Original: 4 (cid:113) (cid:98) (cid:15)/ (cid:98) L } by (14) and Lemma 3(a).
Revised: 4 (cid:113) (cid:98) (cid:15)/ (cid:98) L } by (14) and Lemma 3(a).

Original: Then, with Lemma 3(b,c) and (cid:98) (cid:15) =
Revised: Then, with Lemma 3(b,c) and (cid:98) (cid:15) =

Original: ε 2 64 (cid:98) L , we get that,
Revised: ε 2 64 (cid:98) L , we get that,

Original: [Equation]
Revised: [Equation]

Original: i.e., x s is an ε -FOSP of g .
Revised: i.e., x s is an ε -FOSP of g .

Original: By combining the above two cases, we establish that 8 B 3 (cid:98) (cid:15) “outer” iterations ensure convergence to a ε -FOSP.
Revised: By combining the above two cases, we establish that 8 B 3 (cid:98) (cid:15) “outer” iterations ensure convergence to a ε -FOSP.

Original: We now compute the gradient call complexity of each of these “outer" iterations, where we have two options for implementing Step 3 of Algorithm 2.
Revised: We now compute the gradient call complexity of each of these “outer" iterations, where we have two options for implementing Step 3 of Algorithm 2.

Original: Note that this step corresponds to solving min x max λ ∈ S
Revised: Note that this step corresponds to solving min x max λ ∈ S

Original: h ( x, λ ) up to an accuracy of (cid:98) (cid:15)/ 4 .
Revised: h ( x, λ ) up to an accuracy of (cid:98) (cid:15)/ 4 .

Original: Option I , (Thekumparampil et al., 2019, Algorithm 2): Since the minimax optimization problem here is (cid:98) L -strongly-convex–concave and 2 (cid:98) L -gradient Lipschitz, (Thekumparampil et al., 2019, Theorem 1) tells us that this requires at most m gradient calls for each g i where,
Revised: Option I , (Thekumparampil et al., 2019, Algorithm 2): Since the minimax optimization problem here is (cid:98) L -strongly-convex–concave and 2 (cid:98) L -gradient Lipschitz, (Thekumparampil et al., 2019, Theorem 1) tells us that this requires at most m gradient calls for each g i where,

Original: [Equation]
Revised: [Equation]

Original: Therefore the number of gradient computations required for each iteration of inner problem is
Revised: Therefore the number of gradient computations required for each iteration of inner problem is

Original: [Equation]
Revised: [Equation]

Original: Option II, Cutting plane method (Lee et al., 2015): Let us consider u ( λ ) :
Revised: Option II, Cutting plane method (Lee et al., 2015): Let us consider u ( λ ) :

Original: = min x h ( x, λ ) + (cid:98) L (cid:107) x − x s (cid:107) 2 , which is a (cid:98) L -Lipschitz, concave function of λ .
Revised: = min x h ( x, λ ) + (cid:98) L (cid:107) x − x s (cid:107) 2 , which is a (cid:98) L -Lipschitz, concave function of λ .

Original: (Lee et al., 2015) tells us that we can use cutting plane algorithms to obtain (cid:98) λ satisfying u ( (cid:98) λ ) ≥ max λ ∈ S u ( λ ) − (cid:101) (cid:15) using O (cid:16) k log k[Equation] L
Revised: (Lee et al., 2015) tells us that we can use cutting plane algorithms to obtain (cid:98) λ satisfying u ( (cid:98) λ ) ≥ max λ ∈ S u ( λ ) − (cid:101) (cid:15) using O (cid:16) k log k

Original: [Equation]
Revised: [Equation]

Original: (cid:17) gradient queries to u and poly( k log (cid:98) L (cid:101) (cid:15) ) computation.
Revised: (cid:17) gradient queries to u and poly( k log (cid:98) L (cid:101) (cid:15) ) computation.

Original: The gradient of u is given by ∇ u ( λ ) = ∇ λ h ( x ∗ ( λ ) , λ ) , where x ∗ ( λ ) :
Revised: The gradient of u is given by ∇ u ( λ ) = ∇ λ h ( x ∗ ( λ ) , λ ) , where x ∗ ( λ ) :

Original: = argmin x h ( x, λ ) + (cid:98) L (cid:107) x − x s (cid:107) 2 .
Revised: = argmin x h ( x, λ ) + (cid:98) L (cid:107) x − x s (cid:107) 2 .

Original: Since h ( x, λ ) + (cid:98) L (cid:107) x − x s (cid:107) 2 is a 3 (cid:98) L -smooth and (cid:98) L -strongly convex function in x , x ∗ ( λ ) can be computed up to (cid:101) (cid:15) error using gradient descent in
Revised: Since h ( x, λ ) + (cid:98) L (cid:107) x − x s (cid:107) 2 is a 3 (cid:98) L -smooth and (cid:98) L -strongly convex function in x , x ∗ ( λ ) can be computed up to (cid:101) (cid:15) error using gradient descent in

Original: O (cid:16) log (cid:98) L (cid:101) (cid:15) (cid:17) iterations.
Revised: O (cid:16) log (cid:98) L (cid:101) (cid:15) (cid:17) iterations.

Original: If we choose (cid:101) (cid:15) = (cid:98) (cid:15) 2 / poly( k, (cid:98) L/µ ) , then Proposition 2 tells us that x ∗ ( (cid:98) λ ) satisﬁes the requirements of Step (3) of Algorithm 2 and the total number of gradient calls to each g
Revised: If we choose (cid:101) (cid:15) = (cid:98) (cid:15) 2 / poly( k, (cid:98) L/µ ) , then Proposition 2 tells us that x ∗ ( (cid:98) λ ) satisﬁes the requirements of Step (3) of Algorithm 2 and the total number of gradient calls to each g

Original: i is at most O (cid:16) poly( k ) log (cid:98) L(cid:15) (cid:17) in each outer iteration of Algorithm 2.
Revised: i is at most O (cid:16) poly( k ) log (cid:98) L(cid:15) (cid:17) in each outer iteration of Algorithm 2.

Original: Proposition 2.
Revised: Proposition 2.

Original: Suppose h : R d 1 × U → R be such that h ( · , λ ) is µ -strongly convex for every λ ∈ U , h ( x, · ) is concave for every x ∈ R d 

Revised: Suppose h : R d 1 × U → R be such that h ( · , λ ) is µ -strongly convex for every λ ∈ U , h ( x, · ) is concave for every x ∈ R d

Original: and h is L -gradient Lipschitz.
Revised: and h is L -gradient Lipschitz.

Original: Let (cid:98) λ be such that min x h ( x, (cid:98) λ ) ≥ max λ min x h ( x, λ ) − (cid:15) and let x ∗ ( (cid:98) λ ) :=
Revised: Let (cid:98) λ be such that min x h ( x, (cid:98) λ ) ≥ max λ min x h ( x, λ ) − (cid:15) and let x ∗ ( (cid:98) λ ) :=

Original: argmin x h ( x, (cid:98) λ ) .
Revised: argmin x h ( x, (cid:98) λ ) .

Original: Then, we have max λ h ( x ∗ ( (cid:98) λ ) , λ ) ≤ min x max λ h ( x, λ ) + c (cid:16) Lµ · (cid:15) + LD U √ µ · √ (cid:15) (cid:17) , where D U = max λ 1 ,λ 2 ∈U (cid:107) λ 1 − λ 2 (cid:107) is the diameter of U .
Revised: Then, we have max λ h ( x ∗ ( (cid:98) λ ) , λ ) ≤ min x max λ h ( x, λ ) + c (cid:16) Lµ · (cid:15) + LD U √ µ · √ (cid:15) (cid:17) , where D U = max λ 1 ,λ 2 ∈U (cid:107) λ 1 − λ 2 (cid:107) is the diameter of U .

Original: Proof of Proposition 2.
Revised: Proof of Proposition 2.

Original: From the hypothesis, we have:
Revised: From the hypothesis, we have:

Original: [Equation]
Revised: [Equation]

Original: where ( x ∗ , λ ∗ ) is the Nash equilibrium and the second step follows from the fact that λ ∗ =
Revised: where ( x ∗ , λ ∗ ) is the Nash equilibrium and the second step follows from the fact that λ ∗ =

Original: argmax λ h ( x ∗ , λ ) and the third step follows from the fact that x ∗ ( (cid:98) λ ) =
Revised: argmax λ h ( x ∗ , λ ) and the third step follows from the fact that x ∗ ( (cid:98) λ ) =

Original: argmin x h ( x, (cid:98) λ ) .
Revised: argmin x h ( x, (cid:98) λ ) .

Original: Consequently, we have that (cid:107) x ∗ − x ∗ ( (cid:98) λ ) (cid:107) ≤ (cid:112) 2 (cid:15)/µ .
Revised: Consequently, we have that (cid:107) x ∗ − x ∗ ( (cid:98) λ ) (cid:107) ≤ (cid:112) 2 (cid:15)/µ .

Original: Let ¯ λ : = argmax λ h ( x ∗ ( (cid:98) λ ) , λ ) .
Revised: Let ¯ λ : = argmax λ h ( x ∗ ( (cid:98) λ ) , λ ) .

Original: We now have that:
Revised: We now have that:

Original: [Equation]
Revised: [Equation]

Original: where ( ζ 1 ) follows from the fact that h ( x ∗ ( (cid:98) λ ) , · ) is concave and x ∗ =
Revised: where ( ζ 1 ) follows from the fact that h ( x ∗ ( (cid:98) λ ) , · ) is concave and x ∗ =

Original: argmin x h ( x, λ ∗ ) , ( ζ 2 ) follows from the bound (cid:107)
Revised: argmin x h ( x, λ ∗ ) , ( ζ 2 ) follows from the bound (cid:107)

Original: x ∗ − x ∗ ( (cid:98) λ )
Revised: x ∗ − x ∗ ( (cid:98) λ )

Original: (cid:107) ≤ (cid:112) 2 (cid:15)/µ , ( ζ 3 ) follows from the L -gradient Lipschitz property of h , and the last step follows from the fact that ∇ λ h ( x ∗ , λ ∗ ) = 0 .
Revised: (cid:107) ≤ (cid:112) 2 (cid:15)/µ , ( ζ 3 ) follows from the L -gradient Lipschitz property of h , and the last step follows from the fact that ∇ λ h ( x ∗ , λ ∗ ) = 0 .

Original: This proves the proposition.
Revised: This proves the proposition.

Original: E P ROOFS OF RESULTS IN S ECTION
Revised: E P ROOFS OF RESULTS IN S ECTION

Original: In this appendix, we present the proofs for the lemmas in Section 5.
Revised: In this appendix, we present the proofs for the lemmas in Section 5.

Original: Recall the SGA update:
Revised: Recall the SGA update:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore, the Jacobian of the T -step SGA update is given by
Revised: Therefore, the Jacobian of the T -step SGA update is given by

Original: [Equation]
Revised: [Equation]

Original: A ( x
Revised: A ( x

Original: [Equation]
Revised: [Equation]

Original: z ) =
Revised: z ) =

Original: y T ( [Equation]
Revised: y T ( x )

Original: [Equation]
Revised: [Equation]

Original: E. P ROOF OF T HEOREM
Revised: E. P ROOF OF T HEOREM

Original: Theorem 3 (General Case) .
Revised: Theorem 3 (General Case) .

Original: Suppose for all j ∈ [ n ] , f j satisﬁes Assumption 1.
Revised: Suppose for all j ∈ [ n ] , f j satisﬁes Assumption 1.

Original: Then, for any ﬁxed randomness z , T -step SGA is (1 + ηL ) T -Lipschitz and 4( ρ/L ) · (1 + ηL ) 2 T -gradient Lipschitz.
Revised: Then, for any ﬁxed randomness z , T -step SGA is (1 + ηL ) T -Lipschitz and 4( ρ/L ) · (1 + ηL ) 2 T -gradient Lipschitz.

Original: Proof.
Revised: Proof.

Original: Lipschitz of y t ( x ) .
Revised: Lipschitz of y t ( x ) .

Original: We ﬁrst show the Lipschitz claim.
Revised: We ﬁrst show the Lipschitz claim.

Original: We have the following bound on the
Revised: We have the following bound on the

Original: Jacobian of the update equation given in (18):
Revised: Jacobian of the update equation given in (18):

Original: [Equation]
Revised: [Equation]

Original: Since Dy 0 ( x ) = 0 , the above recursion implies that
Revised: Since Dy 0 ( x ) = 0 , the above recursion implies that

Original: [Equation]
Revised: [Equation]

Original: Gradient-Lipschitz of y t ( x ) .
Revised: Gradient-Lipschitz of y t ( x ) .

Original: Next, we show the claimed gradient Lipschitz constant.
Revised: Next, we show the claimed gradient Lipschitz constant.

Original: As above, using the update equation in (18), we have the following bound on the Jacobian:
Revised: As above, using the update equation in (18), we have the following bound on the Jacobian:

Original: [Equation]
Revised: [Equation]

Original: 1 ( x 2 ) (cid:107) ≤(cid:107)
Revised: ( x 2 ) (cid:107) ≤(cid:107)

Original: ( I + η ∇ 2 yy f ( x 1 , y t ( x 1 )))
Revised: ( I + η ∇ 2 yy f ( x 1 , y t ( x 1 )))

Original: ( Dy t ( x 1 ) − Dy t ( x 2 )) (cid:107) + η (cid:107)∇ 2 yx f ( x 1 , y t ( x 1 ))
Revised: ( Dy t ( x 1 ) − Dy t ( x 2 )) (cid:107) + η (cid:107)∇ 2 yx f ( x 1 , y t ( x 1 ))

Original: − ∇ 2 yx f ( x 2 , y t ( x 2 ))
Revised: − ∇ 2 yx f ( x 2 , y t ( x 2 ))

Original: (cid:107) + η (cid:107) [ ∇ 2 yy f
Revised: (cid:107) + η (cid:107) [ ∇ 2 yy f

Original: ( x 1 , y t ( x 1 ))
Revised: ( x 1 , y t ( x 1 ))

Original: − ∇ 2 yy f
Revised: − ∇ 2 yy f

Original: ( x 2 , y t ( x 2 ))]
Revised: ( x 2 , y t ( x 2 ))]

Original: Dy t ( x 2 ) (cid:107) ≤ (1 + ηL ) (cid:107) Dy t
Revised: Dy t ( x 2 ) (cid:107) ≤ (1 + ηL ) (cid:107) Dy t

Original: ( x 1 ) − Dy t ( x 2 ) (cid:107) + ηρ (1 + (cid:107)
Revised: ( x 1 ) − Dy t ( x 2 ) (cid:107) + ηρ (1 + (cid:107)

Original: Dy t ( x 2 ) (cid:107) )( (cid:107) x 1 − x 2 (cid:107) + (cid:107) y t
Revised: Dy t ( x 2 ) (cid:107) )( (cid:107) x 1 − x 2 (cid:107) + (cid:107) y t

Original: ( x 1 ) − y t ( x 2 ) (cid:107) ) ≤ (1 + ηL ) (cid:107) Dy t
Revised: ( x 1 ) − y t ( x 2 ) (cid:107) ) ≤ (1 + ηL ) (cid:107) Dy t

Original: ( x 1 ) − Dy t ( x 2 ) (cid:107) + 4 ηρ (1 + ηL ) 2 t (cid:107) x 1 − x 2 (cid:107) .
Revised: ( x 1 ) − Dy t ( x 2 ) (cid:107) + 4 ηρ (1 + ηL ) 2 t (cid:107) x 1 − x 2 (cid:107) .

Original: The above recursion implies the claimed Lipschitz constant.
Revised: The above recursion implies the claimed Lipschitz constant.

Original: Indeed,
Revised: Indeed,

Revised: [Equation]

Revised: x

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: E. P ROOF OF T HEOREM
Revised: E. P ROOF OF T HEOREM

Original: Theorem 4 (Concave Case) .
Revised: Theorem 4 (Concave Case) .

Original: Suppose for all j ∈ [ n ] , f j satisﬁes Assumption 1 and f j ( x, · ) is concave for any x .
Revised: Suppose for all j ∈ [ n ] , f j satisﬁes Assumption 1 and f j ( x, · ) is concave for any x .

Original: Then, for any ﬁxed randomness z , T -step SGA is ηLT -Lipschitz and ( ρ/L ) · (1 + ηLT ) 3gradient Lipschitz.
Revised: Then, for any ﬁxed randomness z , T -step SGA is ηLT -Lipschitz and ( ρ/L ) · (1 + ηLT ) 3gradient Lipschitz.

Original: Proof.
Revised: Proof.

Original: Lipschitz of y t ( x ) .
Revised: Lipschitz of y t ( x ) .

Original: We ﬁrst show the Lipschitz claim.
Revised: We ﬁrst show the Lipschitz claim.

Original: Using the update equation in (18), we have the following bound on the Jacobian:
Revised: Using the update equation in (18), we have the following bound on the Jacobian:

Original: [Equation]
Revised: [Equation]

Original: Since Dy 0 ( x ) = 0 , the above recursive implies that (cid:107) Dy t ( x ) (cid:107) ≤ ηLt .
Revised: Since Dy 0 ( x ) = 0 , the above recursive implies that (cid:107) Dy t ( x ) (cid:107) ≤ ηLt .

Original: Gradient-Lipschitz of y t ( x ) .
Revised: Gradient-Lipschitz of y t ( x ) .

Original: Next, we show the claimed gradient Lipschitz constant.
Revised: Next, we show the claimed gradient Lipschitz constant.

Original: Using the update equation in (18):, we have the following bound on the Jacobian:
Revised: Using the update equation in (18):, we have the following bound on the Jacobian:

Original: [Equation]
Revised: [Equation]

Revised: x 2 ) (cid:107) ≤(cid:107)

Revised: [Equation]

Revised: [Equation]

Revised: ( (cid:107)

Revised: [Equation]

Revised: − x

Revised: [Equation]

Revised: (cid:107) + (cid:107) y t

Revised: ( x 1 ) − y t ( x 2 ) (cid:107)

Revised: [Equation]

Revised: ≤(cid:107)

Revised: [Equation]

Revised: t

Revised: [Equation]

Revised: (cid:107)

Revised: [Equation]

Revised: (cid:107)

Revised: [Equation]

Original: This recursion implies the following gradient Lipschitz constant:
Revised: This recursion implies the following gradient Lipschitz constant:

Original: [Equation]
Revised: [Equation]

Original: E. P ROOF OF T HEOREM
Revised: E. P ROOF OF T HEOREM

Original: Theorem 5 (Strongly-concave Case) .
Revised: Theorem 5 (Strongly-concave Case) .

Original: Suppose for all j ∈ [ n ] , f j satisﬁes Assumption 1 and f j ( x, · ) is α -strongly concave for any x .
Revised: Suppose for all j ∈ [ n ] , f j satisﬁes Assumption 1 and f j ( x, · ) is α -strongly concave for any x .

Original: Then, for any ﬁxed randomness z , T -step SGA is κ -Lipschitz and 4( ρ/L ) · κ 3 -gradient Lipschitz, where κ =
Revised: Then, for any ﬁxed randomness z , T -step SGA is κ -Lipschitz and 4( ρ/L ) · κ 3 -gradient Lipschitz, where κ =

Original: L/α is the condition number.
Revised: L/α is the condition number.

Original: Proof.
Revised: Proof.

Original: Denote the condition number κ =
Revised: Denote the condition number κ =

Original: L/α .
Revised: L/α .

Original: Lipschitz of y t ( x ) .
Revised: Lipschitz of y t ( x ) .

Original: We ﬁrst show the claimed Lipschitz constant.
Revised: We ﬁrst show the claimed Lipschitz constant.

Original: Using the update equation in (18), we have that
Revised: Using the update equation in (18), we have that

Original: [Equation]
Revised: [Equation]

Original: Since Dy 0 ( x ) = 0 , the above recursion gives the following bound:
Revised: Since Dy 0 ( x ) = 0 , the above recursion gives the following bound:

Original: [Equation]
Revised: [Equation]

Original: Gradient-Lipschitz of y t ( x ) .
Revised: Gradient-Lipschitz of y t ( x ) .

Original: Next we show the claimed gradient Lipschitz constant.
Revised: Next we show the claimed gradient Lipschitz constant.

Original: Again, using the update equation, we have that
Revised: Again, using the update equation, we have that

Original: [Equation]
Revised: [Equation]

Original: yy

Original: [Equation]

Original: y

Original: [Equation]

Original: y

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: This recursion implies that
Revised: This recursion implies that

Original: [Equation]
Revised: [Equation]

Original: E. P ROOF OF T HEOREM
Revised: E. P ROOF OF T HEOREM

Original: Theorem 6 (General Case) .
Revised: Theorem 6 (General Case) .

Original: Suppose for all j ∈ [ n ] , f j satisﬁes Assumption 1.
Revised: Suppose for all j ∈ [ n ] , f j satisﬁes Assumption 1.

Original: Then, for any ﬁxed seed z , T -step SNAG is T (1 + ηL/θ ) T -Lipschitz and 50( ρ/L ) · T 3 (1 + ηL/θ ) 2 T -gradient Lipschitz.
Revised: Then, for any ﬁxed seed z , T -step SNAG is T (1 + ηL/θ ) T -Lipschitz and 50( ρ/L ) · T 3 (1 + ηL/θ ) 2 T -gradient Lipschitz.

Original: Proof.
Revised: Proof.

Original: Recall the SNAG update
Revised: Recall the SNAG update

Original: [Equation]
Revised: [Equation]

Original: Observe that the update equation for T -step SNAG implies that
Revised: Observe that the update equation for T -step SNAG implies that

Original: [Equation]
Revised: [Equation]

Original: Lipschitz of y t ( x ) , v t ( x ) .
Revised: Lipschitz of y t ( x ) , v t ( x ) .

Original: We ﬁrst show the claimed Lipschitz constant.
Revised: We ﬁrst show the claimed Lipschitz constant.

Original: By the update equations in (21), we have that
Revised: By the update equations in (21), we have that

Original: [Equation]
Revised: [Equation]

Original: Denote δ t = (cid:107)
Revised: Denote δ t = (cid:107)

Original: Dy t − Dy t − 1 (cid:107)
Revised: Dy t − Dy t − 1 (cid:107)

Original: [Equation]
Revised: [Equation]

Original: and note that Dy 

Revised: and note that Dy

Original: = Dy
Revised: = Dy

Original: [Equation]
Revised: [Equation]

Original: [-= 
-] so that δ 0 = 0 .
Revised: [-= 
-] so that δ 0 = 0 .

Original: By the equation above, we have that
Revised: By the equation above, we have that

Original: [Equation]
Revised: [Equation]

Original: In the following
Revised: In the following

Original: [Equation]
Revised: [Equation]

Original: we use induction to prove that
Revised: we use induction to prove that

Original: [Equation]
Revised: [Equation]

Original: It is easy to verify that this is true for the base case δ 0 = 0 ≤ 1 .
Revised: It is easy to verify that this is true for the base case δ 0 = 0 ≤ 1 .

Original: Suppose the claim is true for all τ ≤ t , then we have that
Revised: Suppose the claim is true for all τ ≤ t , then we have that

Original: [Equation]
Revised: [Equation]

Original: This proves the induction claim.
Revised: This proves the induction claim.

Original: Therefore, by (22), we have the following two bounds:
Revised: Therefore, by (22), we have the following two bounds:

Original: [Equation]
Revised: [Equation]

Original: Gradient-Lipschitz of y t ( x ) .
Revised: Gradient-Lipschitz of y t ( x ) .

Original: Next, we show the claimed gradient Lipschitz constant.
Revised: Next, we show the claimed gradient Lipschitz constant.

Original: For any ﬁxed x 1 , x 2 , denote w t =
Revised: For any ﬁxed x 1 , x 2 , denote w t =

Original: Dy t ( x 1 ) − Dy t ( x 2 ) , we have w t +1 =(
Revised: Dy t ( x 1 ) − Dy t ( x 2 ) , we have w t +1 =(

Original: I + η ∇ yy f σ ( t ) ( x 1 , ˜
Revised: I + η ∇ yy f σ ( t ) ( x 1 , ˜

Original: y t ( x 1 )))
Revised: y t ( x 1 )))

Original: ( w t + (1 − θ )( w t − w t − 1 ))
Revised: ( w t + (1 − θ )( w t − w t − 1 ))

Original: + η ( ∇ yx f σ ( t ) ( x 1 , ˜ y t ( x 1 )) − ∇ yx f σ ( t ) ( x 2 , ˜ y t ( x 2 )))
Revised: + η ( ∇ yx f σ ( t ) ( x 1 , ˜ y t ( x 1 )) − ∇ yx f σ ( t ) ( x 2 , ˜ y t ( x 2 )))

Original: (cid:124) (cid:123)(cid:122) (cid:125)
Revised: (cid:124) (cid:123)(cid:122) (cid:125)

Original: T+ η ( ∇ yy f σ ( t ) ( x 1 , ˜
Revised: T+ η ( ∇ yy f σ ( t ) ( x 1 , ˜

Original: y t ( x 1 ))
Revised: y t ( x 1 ))

Original: − ∇ yy f σ ( t ) ( x 2 , ˜
Revised: − ∇ yy f σ ( t ) ( x 2 , ˜

Original: y t ( x 2 )))
Revised: y t ( x 2 )))

Original: ( Dy t ( x 2 ) + (1 − θ )
Revised: ( Dy t ( x 2 ) + (1 − θ )

Original: ( Dy t ( x 2 ) − Dy t − 1 ( x 2 )))
Revised: ( Dy t ( x 2 ) − Dy t − 1 ( x 2 )))

Original: (cid:124) (cid:123)(cid:122) (cid:125)
Revised: (cid:124) (cid:123)(cid:122) (cid:125)

Original: T.
Revised: T.

Original: We note that we can upper bound the last two terms above as follows:
Revised: We note that we can upper bound the last two terms above as follows:

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: let ζ t = (cid:107) w t − w t − 1 (cid:107)
Revised: let ζ t = (cid:107) w t − w t − 1 (cid:107)

Original: [Equation]
Revised: [Equation]

Original: and ∆ = (cid:107) x 1 − x 2 (cid:107)
Revised: and ∆ = (cid:107) x 1 − x 2 (cid:107)

Original: [Equation]
Revised: [Equation]

Original: we have the following:
Revised: we have the following:

Original: [Equation]
Revised: [Equation]

Original: In the following
Revised: In the following

Original: [Equation]
Revised: [Equation]

Original: we use induction to prove that
Revised: we use induction to prove that

Original: [Equation]
Revised: [Equation]

Original: It is easy to verify that this is true for the base case ζ 0 = 0 .
Revised: It is easy to verify that this is true for the base case ζ 0 = 0 .

Original: Suppose the claim is true for all τ ≤ t , then we have that
Revised: Suppose the claim is true for all τ ≤ t , then we have that

Original: [Equation]
Revised: [Equation]

Original: (1 + ηL/θ ) 2( t +1)
Revised: 1 + ηL/θ ) 2( t +1)

Original: −(1 + ηL/θ ) 2 −+ 24( ηL/θ ) t 2 (1 + ηL/θ ) 2 t ]∆ + (1 − θ ) ψ
Revised: −(1 + ηL/θ ) 2 −+ 24( ηL/θ ) t 2 (1 + ηL/θ ) 2 t ]∆ + (1 − θ ) ψ

Original: ( t + 1) ≤ θ ( ρ/L ) · [25 t 2 (1 + ηL/θ ) 

Revised: ( t + 1) ≤ θ ( ρ/L ) · [25 t 2 (1 + ηL/θ )

Original: ( t +1)
Revised: ( t +1)

Original: + 24 t 2 (1 + ηL/θ ) 2( t +1) ]
Revised: + 24 t 2 (1 + ηL/θ ) 2( t +1) ]

Original: ∆ + (1 − θ ) ψ ( t + 1) ≤ θ ( ρ/L ) · [50 t 2 (1 + ηL/θ ) 

Revised: ∆ + (1 − θ ) ψ ( t + 1) ≤ θ ( ρ/L ) · [50 t 2 (1 + ηL/θ )

Original: ( t +1) ]
Revised: ( t +1) ]

Original: ∆ + (1 − θ ) ψ ( t + 1) ≤ ψ ( t + 1) .
Revised: ∆ + (1 − θ ) ψ ( t + 1) ≤ ψ ( t + 1) .

Original: This proves the induction claim.
Revised: This proves the induction claim.

Original: Therefore, by (23), we have that
Revised: Therefore, by (23), we have that

Original: [Equation]
Revised: [Equation]

Original: E. P ROJECTED GRADIENT ASCENT IS NOT GRADIENT -L IPSCHITZ
Revised: E. P ROJECTED GRADIENT ASCENT IS NOT GRADIENT -L IPSCHITZ

Original: Proposition 3.
Revised: Proposition 3.

Original: Consider f ( x, y )
Revised: Consider f ( x, y )

Original: =
Revised: =

Original: xy for ( x, y ) ∈
Revised: xy for ( x, y ) ∈

Original: X × Y , where X = [0 , 10] and Y = [0 , 1] .
Revised: X × Y , where X = [0 , 10] and Y = [0 , 1] .

Original: 1 -step projected gradient ascent given by:
Revised: 1 -step projected gradient ascent given by:

Original: [Equation]
Revised: [Equation]

Original: where η > 1 / 10 is not a gradient-Lipschitz algorithm.
Revised: where η > 1 / 10 is not a gradient-Lipschitz algorithm.

Original: Proof.
Revised: Proof.

Original: We see that y 1 ( x ) = min(1 , ηx ) and f ( x, y 1 ( x ))
Revised: We see that y 1 ( x ) = min(1 , ηx ) and f ( x, y 1 ( x ))

Original: = x min(1 , ηx ) .
Revised: = x min(1 , ηx ) .

Original: For η < 1 / 10 , we see that f ( x, y 1 ( x )) is not gradient-Lipschitz at x = 1 /η ∈ (0 , 10) .
Revised: For η < 1 / 10 , we see that f ( x, y 1 ( x )) is not gradient-Lipschitz at x = 1 /η ∈ (0 , 10) .

Original: F O N N ONCONVERGENCE O F E XTRAGRADIENT M ETHOD AND O PTIMISTIC G RADIENT D ESCENT A SCENT 
Revised: F O N N ONCONVERGENCE O F E XTRAGRADIENT M ETHOD AND O PTIMISTIC G RADIENT D ESCENT A SCENT (

Original: [Equation]
Revised: OGDA)

Original: In this section, we show that there exist nonconvex-nonconcave problems where the extragradient method and OGDA get trapped in limit cycles.
Revised: In this section, we show that there exist nonconvex-nonconcave problems where the extragradient method and OGDA get trapped in limit cycles.

Original: While the example we construct is a quadratic function that is not bounded over all of the space, it will be clear from the proof that the same results hold for any function that matches the constructed quadratic locally around origin.
Revised: While the example we construct is a quadratic function that is not bounded over all of the space, it will be clear from the proof that the same results hold for any function that matches the constructed quadratic locally around origin.

Original: Consider the following two dimensional quadratic function parametrized by a, b ∈ R :
Revised: Consider the following two dimensional quadratic function parametrized by a, b ∈ R :

Original: [Equation]
Revised: [Equation]

Original: Denote z := ( x, y ) , Φ := (cid:18) ab −
Revised: Denote z := ( x, y ) , Φ := (cid:18) a

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: and deﬁne:
Revised: and deﬁne:

Original: [Equation]
Revised: [Equation]

Original: Let the eigenvalues of matrix Φ be λ 1 , λ 2 .
Revised: Let the eigenvalues of matrix Φ be λ 1 , λ 2 .

Original: We know they satify the characteristic equation ( λ − a ) 

Revised: We know they satify the characteristic equation ( λ − a )

Original: + b 2 = 0 .
Revised: + b 2 = 0 .

Original: By Vieta’s formulas, we know that λ
Revised: By Vieta’s formulas, we know that λ

Original: 1 + λ 2 = 2 a , λ 1 λ 2 =
Revised: 1 + λ 2 = 2 a , λ 1 λ 2 =

Original: a 2 + b 2 .
Revised: a 2 + b 2 .

Original: This justiﬁes the following proposition.
Revised: This justiﬁes the following proposition.

Original: Proposition 4.
Revised: Proposition 4.

Original: For any complex number c ∈ C , there exist a, b ∈ R s.t.
Revised: For any complex number c ∈ C , there exist a, b ∈ R s.t.

Original: the eigenvalues of Φ are c and ¯ c .
Revised: the eigenvalues of Φ are c and ¯ c .

Original: Here ¯ c is the complex conjugate of c .
Revised: Here ¯ c is the complex conjugate of c .

Original: Proof.
Revised: Proof.

Original: Let c
Revised: Let c

Original: = p + qi for some p, q ∈ R , we can choose a = ( c + ¯ c ) / 2 = p , b = ±√ c ¯ c − a 2 = ± q .
Revised: = p + qi for some p, q ∈ R , we can choose a = ( c + ¯ c ) / 2 = p , b = ±√ c ¯ c − a 2 = ± q .

Original: It is easy to check such matrix Φ has eigenvalues c and ¯ c .
Revised: It is easy to check such matrix Φ has eigenvalues c and ¯ c .

Original: F.
Revised: F.

Original: E XTRAGRADIENT M ETHODS
Revised: E XTRAGRADIENT M ETHODS

Original: The extragradient algorithm (Korpelevich, 1976) uses the following update:
Revised: The extragradient algorithm (Korpelevich, 1976) uses the following update:

Original: [Equation]
Revised: [Equation]

Original: For the case of quadratic function speciﬁed by (24), this update is equivalent to
Revised: For the case of quadratic function speciﬁed by (24), this update is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Clearly, the eigenvalues of matrix ( I − η Φ+ η 2 Φ 2 ) are simply (1 − ηλ 1 + η 2 λ 21 ) and (1 − ηλ 2 + η 2 λ 22 ) .
Revised: Clearly, the eigenvalues of matrix ( I − η Φ+ η 2 Φ 2 ) are simply (1 − ηλ 1 + η 2 λ 21 ) and (1 − ηλ 2 + η 2 λ 22 ) .

Original: Proposition 5.
Revised: Proposition 5.

Original: For the extragradient algorithm with any ﬁxed learning rate η , there exists a quadratic function, and appropriate initial points such that the extragradient algorithm is trapped in a limit cycle.
Revised: For the extragradient algorithm with any ﬁxed learning rate η , there exists a quadratic function, and appropriate initial points such that the extragradient algorithm is trapped in a limit cycle.

Original: Proof.
Revised: Proof.

Original: For any given θ ∈ (0 , 2 π ) , we can simply pick λ 1 ∈ C to be the solution of equation 1 − ηλ 1 + η 2 λ 21 =
Revised: For any given θ ∈ (0 , 2 π ) , we can simply pick λ 1 ∈ C to be the solution of equation 1 − ηλ 1 + η 2 λ 21 =

Original: e iθ .
Revised: e iθ .

Original: This is a quadratic equation of λ 1 , which always has nonzero complex roots.
Revised: This is a quadratic equation of λ 1 , which always has nonzero complex roots.

Original: For λ 2 = ¯ λ 1 , we verify that 1 − ηλ 2 + η 2 λ 

Revised: For λ 2 = ¯ λ 1 , we verify that 1 − ηλ 2 + η 2 λ

Original: =
Revised: =

Original: e − iθ .
Revised: e − iθ .

Original: Finally, by Proposition 4, we know that there exists a choice of a, b such that, the two eigenvalues of ( I − η Φ + η 2 Φ 2 ) are precisely e iθ and e − iθ .
Revised: Finally, by Proposition 4, we know that there exists a choice of a, b such that, the two eigenvalues of ( I − η Φ + η 2 Φ 2 ) are precisely e iθ and e − iθ .

Original: If the initial point
Revised: If the initial point

Original: [Equation]
Revised: [Equation]

Original: can be decomposed as
Revised: can be decomposed as

Original: [Equation]
Revised: [Equation]

Original: z 2 , where z 1 and z 2 are the two eigenvectors of Φ corresponding to eigenvalues λ 1 and λ 2 respectively, we see that the extra gradient algorithm follows the trajectory
Revised: z 2 , where z 1 and z 2 are the two eigenvectors of Φ corresponding to eigenvalues λ 1 and λ 2 respectively, we see that the extra gradient algorithm follows the trajectory

Original: [Equation]
Revised: [Equation]

Original: , thereby getting trapped in a limit cycle.
Revised: , thereby getting trapped in a limit cycle.

Original: F. O PTIMISTIC G RADIENT D ESCENT A SCENT (OGDA)
Revised: F. O PTIMISTIC G RADIENT D ESCENT A SCENT (OGDA)

Original: The OGDA algorithm (Daskalakis & Panageas, 2018a) has the following update:
Revised: The OGDA algorithm (Daskalakis & Panageas, 2018a) has the following update:

Original: [Equation]
Revised: [Equation]

Original: (Daskalakis & Panageas, 2018a)[Section 4.1] presents an example where OGDA is empirically not shown to converge for some initialization points.
Revised: (Daskalakis & Panageas, 2018a)[Section 4.1] presents an example where OGDA is empirically not shown to converge for some initialization points.

Original: Here, we present an example for which nonconvergence of OGDA is shown rigorously.
Revised: Here, we present an example for which nonconvergence of OGDA is shown rigorously.

Original: We can rewrite this update as
Revised: We can rewrite this update as

Original: [Equation]
Revised: [Equation]

Original: I −
Revised: I −

Original: [Equation]
Revised: [Equation]

Original: η Φ η Φ
Revised: Φ

Original: I (cid:19) (cid:18)
Revised: I (cid:19) (cid:18)

Original: z t z t −(cid:19)
Revised: z t z t −(cid:19)

Original: There are four eigenvalues for the matrix (cid:18)
Revised: There are four eigenvalues for the matrix (cid:18)

Original: I − 2 η Φ η Φ
Revised: I − [Equation] Φ

Revised: [Equation]

Revised: Φ

Original: I (cid:19) .
Revised: I (cid:19) .

Original: Two of them µ 1 , µ 2 are eigenvalues of the matrix
Revised: Two of them µ 1 , µ 2 are eigenvalues of the matrix

Original: [Equation]
Revised: [Equation]

Original: (cid:19) while the other two µ 3 , µ 4 are eigenvalues of the matrix (cid:18)
Revised: (cid:19) while the other two µ 3 , µ 4 are eigenvalues of the matrix (cid:18)

Original: [Equation]
Revised: [Equation]

Original: (cid:19) , where λ 1 and λ 2 are the eigenvalues of Φ .
Revised: (cid:19) , where λ 1 and λ 2 are the eigenvalues of Φ .

Original: Proposition 6.
Revised: Proposition 6.

Original: For the OGDA algorithm with any ﬁxed learning rate η , there exists a quadratic function, and a choice of initial points such that the iterates are trapped in a limit cycle.
Revised: For the OGDA algorithm with any ﬁxed learning rate η , there exists a quadratic function, and a choice of initial points such that the iterates are trapped in a limit cycle.

Original: Proof.
Revised: Proof.

Original: We will ﬁst show that there is a choice for a and b so that the resulting eigenvalues µ 1 , µ 2 of(cid:18)
Revised: We will ﬁst show that there is a choice for a and b so that the resulting eigenvalues µ 1 , µ 2 of

Original: [Equation]
Revised: [Equation]

Original: and µ 3 , µ 4 of
Revised: and µ 3 , µ 4 of

Original: [Equation]
Revised: [Equation]

Original: satisfy | µ 1 | = | µ 3 | = 1 and | µ 2 | =
Revised: satisfy | µ 1 | = | µ 3 | = 1 and | µ 2 | =

Original: | µ 4 | < 1 .
Revised: | µ 4 | < 1 .

Original: Let us ﬁrst ﬁx µ 1 = e iθ and choose ηλ 1 = µ 1 ( µ 1 − 1) 2 µ 1 − 1 so that we have µ 1 is a root of µ 2 − 
Revised: Let us ﬁrst ﬁx µ 1 = e iθ and choose ηλ 1 = µ 1 ( µ 1 − 1) 2 µ 1 − 1 so that we have µ 1 is a root of µ 2 − (1 −

Original: [Equation]
Revised: [Equation]

Original:  µ + ηλ 1 = 0 .
Revised: 1 ) µ + ηλ 1 = 0 .

Original: We note that µ 2 is the other root of the above equation.
Revised: We note that µ 2 is the other root of the above equation.

Original: We have that:
Revised: We have that:

Original: [Equation]
Revised: [Equation]

Original: Therefore, for any θ ∈ (0 , π/ 4] , we have | ηλ 1 | < 1 .
Revised: Therefore, for any θ ∈ (0 , π/ 4] , we have | ηλ 1 | < 1 .

Original: That is | µ 1 µ 2 | < 1 , since | µ 1 | = 1 , we have | µ 2 | < 1 .
Revised: That is | µ 1 µ 2 | < 1 , since | µ 1 | = 1 , we have | µ 2 | < 1 .

Original: By symmetry, we know by choosing λ 

Revised: By symmetry, we know by choosing λ

Original: = ¯ λ
Revised: = ¯ λ

Original: 1 , the two eigenvalues of
Revised: 1 , the two eigenvalues of

Original: [Equation]
Revised: [Equation]

Original: are µ 3 = ¯ µ 1 and µ 4 = ¯ µ 2 .
Revised: are µ 3 = ¯ µ 1 and µ 4 = ¯ µ 2 .

Original: In sum, we have proved that for any θ ∈ (0 , π/ 4] , and any learning rate η , there exists a choice of a, b such that the four eigenvalues of (cid:18)
Revised: In sum, we have proved that for any θ ∈ (0 , π/ 4] , and any learning rate η , there exists a choice of a, b such that the four eigenvalues of (cid:18)

Original: I − 2 η Φ η Φ
Revised: I − 2 η Φ η Φ

Original: I (cid:19) are e ± iθ , c (cid:48) , ¯ c (cid:48) where | c (cid:48)
Revised: I (cid:19) are e ± iθ , c (cid:48) , ¯ c (cid:48) where | c (cid:48)

Original: | = | ¯ c (cid:48) | < 1 .
Revised: | = | ¯ c (cid:48) | < 1 .

Original: If the initial points (cid:18) z 1 z 0 (cid:19) have the decomposition (cid:80) 4 i =

Revised: If the initial points (cid:18) z 1 z 0 (cid:19) have the decomposition (cid:80) 4 i =

Original: α i ¯
Revised: α i ¯

Original: z
Revised: z

Original: i , where ¯ z i is the eigenvector corresponding to eigenvalue µ i , then the iterates z t and z t − 1 are given by (cid:18)
Revised: i , where ¯ z i is the eigenvector corresponding to eigenvalue µ i , then the iterates z t and z t − 1 are given by (cid:18)

Original: z t z t −(cid:19) =
Revised: z t z t −(cid:19) =

Original: (cid:80) 4 i
Revised: (cid:80) 4 i

Original: =

Revised: =

Original: α i µ ti ¯ z i .
Revised: α i µ ti ¯ z i .

Original: As t → ∞ , (cid:18) z t z t −(cid:19) converges to α 1 e
Revised: As t → ∞ , (cid:18) z t z t −(cid:19) converges to α 1 e

Original: iθt ¯
Revised: iθt ¯

Original: z 

Revised: z

Original: + α 3 e − iθt
Revised: + α 3 e − iθt

Original: ¯ z 3 , proving that
Revised: ¯ z 3 , proving that

Original: OGDA iterates get trapped in a limit cycle and neither converge to a single point nor escape to inﬁnity.
Revised: OGDA iterates get trapped in a limit cycle and neither converge to a single point nor escape to inﬁnity.

Original: G A DDITIONAL E XPERIMENTS AND D ETAILS
Revised: G A DDITIONAL E XPERIMENTS AND D ETAILS

Original: In this appendix section, we provide additional experimental results and details.
Revised: In this appendix section, we provide additional experimental results and details.

Original: Dirac-GAN.
Revised: Dirac-GAN.

Original: In the results presented in Section 6 for this problem, the discriminator sampled its initialization uniformly from the interval [ − 0 .
Revised: In the results presented in Section 6 for this problem, the discriminator sampled its initialization uniformly from the interval [ − 0 .

Original: 1 , 0 . 1] and performed T = 10 steps of gradient ascent.
Revised: 1 , 0 . 1] and performed T = 10 steps of gradient ascent.

Original: For the results given in Figure 5, we allow the discriminator to sample uniformly from the interval [ − 0 .
Revised: For the results given in Figure 5, we allow the discriminator to sample uniformly from the interval [ − 0 .

Original: 5 , 1] and consider the discriminator performing T = 

Revised: 5 , 1] and consider the discriminator performing T =

Original: (Figure 5b) and T = 1000 (Figure 5c) gradient ascent steps.
Revised: (Figure 5b) and T = 1000 (Figure 5c) gradient ascent steps.

Original: The rest of the experimental setup is equivalent to that described in Section 6.
Revised: The rest of the experimental setup is equivalent to that described in Section 6.

Original: For the result presented in Figure 5b, we see that with this distribution of initializations for the discriminator and T = 100 gradient ascent steps, the generator is not able to converge to the optimal parameter of θ ∗ = 0 to recreate the underlying data distribution using our algorithm which descends ∇ f
Revised: For the result presented in Figure 5b, we see that with this distribution of initializations for the discriminator and T = 100 gradient ascent steps, the generator is not able to converge to the optimal parameter of θ ∗ = 0 to recreate the underlying data distribution using our algorithm which descends ∇ f

Original: ( θ, A ( θ )) or the algorithm that descends ∇ θ f
Revised: ( θ, A ( θ )) or the algorithm that descends ∇ θ f

Original: ( θ, A ( θ )) .
Revised: ( θ, A ( θ )) .

Original: However, we see that our algorithm converges signiﬁcantly closer to the optimal parameter conﬁguration.
Revised: However, we see that our algorithm converges signiﬁcantly closer to the optimal parameter conﬁguration.

Original: Furthermore, we still observe stability and convergence from our training method, whereas standard training methods using simultaneous or alternating gradient descent-ascent always cycle.
Revised: Furthermore, we still observe stability and convergence from our training method, whereas standard training methods using simultaneous or alternating gradient descent-ascent always cycle.

Original: This example highlights that the optimization through the algorithm of the adversary is important not only for the rate of convergence, but it also inﬂuences what the training method converges to and gives improved results in this regard.
Revised: This example highlights that the optimization through the algorithm of the adversary is important not only for the rate of convergence, but it also inﬂuences what the training method converges to and gives improved results in this regard.

Original: Finally, in the result presented in Figure 5b, we see that with this distribution of initializations for the discriminator and T = 1000 gradient ascent steps, the generator is able to converge to the optimal parameter of θ ∗ = 0 to recreate the underlying data distribution using our algorithm which descends ∇ f
Revised: Finally, in the result presented in Figure 5b, we see that with this distribution of initializations for the discriminator and T = 1000 gradient ascent steps, the generator is able to converge to the optimal parameter of θ ∗ = 0 to recreate the underlying data distribution using our algorithm which descends ∇ f

Original: ( θ, A ( θ )) or the algorithm that descends ∇ θ f
Revised: ( θ, A ( θ )) or the algorithm that descends ∇ θ f

Original: ( θ, A ( θ )) .
Revised: ( θ, A ( θ )) .

Original: Thus, while with T = 100 we did not observe convergence to the optimal generator parameter, with a stronger adversary we do see convergence to the optimal generator parameter.
Revised: Thus, while with T = 100 we did not observe convergence to the optimal generator parameter, with a stronger adversary we do see convergence to the optimal generator parameter.

Original: This behavior can be explained by the fact that when the discriminator is able to perform enough gradient ascent steps to nearly converge, the gradients ∇ f ( θ, A ( θ )) and ∇ θ f
Revised: This behavior can be explained by the fact that when the discriminator is able to perform enough gradient ascent steps to nearly converge, the gradients ∇ f ( θ, A ( θ )) and ∇ θ f

Original: ( θ, A ( θ )) are nearly equivalent.
Revised: ( θ, A ( θ )) are nearly equivalent.

Original: We remark that we repeated the experiments 5 times with different random seeds and show the mean generator parameters during the training with a window around the mean of a standard deviation.
Revised: We remark that we repeated the experiments 5 times with different random seeds and show the mean generator parameters during the training with a window around the mean of a standard deviation.

Original: The results were very similar between runs so the window around the mean is not visible.
Revised: The results were very similar between runs so the window around the mean is not visible.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Mixture of Gaussians.
Revised: Mixture of Gaussians.

Original: We noted in Section 6 that we repeated our experiment training a generative adversarial network to learn a mixture of Gaussians 10 times and observed that for each run of the experiment our training algorithm recovered all modes of the distribution.
Revised: We noted in Section 6 that we repeated our experiment training a generative adversarial network to learn a mixture of Gaussians 10 times and observed that for each run of the experiment our training algorithm recovered all modes of the distribution.

Original: We now show those results in Figure 6.
Revised: We now show those results in Figure 6.

Original: In particular, in Figure 6a we show the real data distribution and in Figures
Revised: In particular, in Figure 6a we show the real data distribution and in Figures

Original: 6b–6k we show the ﬁnal generated distribution from 10 separate runs of the training procedure after 150k generator updates.
Revised: 6b–6k we show the ﬁnal generated distribution from 10 separate runs of the training procedure after 150k generator updates.

Original: Notably, we observe that each run of the training algorithm is able to generate a distribution that closely resembles the underlying data distribution, showing the stability and robustness of our training method.
Revised: Notably, we observe that each run of the training algorithm is able to generate a distribution that closely resembles the underlying data distribution, showing the stability and robustness of our training method.

Original: We also performed an experiment on the mixture of Gaussian problem in which the discriminator algorithm was the Adam optimization procedure with parameters ( β 1 , β 2 ) =
Revised: We also performed an experiment on the mixture of Gaussian problem in which the discriminator algorithm was the Adam optimization procedure with parameters ( β 1 , β 2 ) =

Original: (0 . 99 , 0 . 999) and learning rate η 2 = 0 .
Revised: (0 . 99 , 0 . 999) and learning rate η 2 = 0 .

Original: and the generator learning rate was η 1 = 0 .
Revised: and the generator learning rate was η 1 = 0 .

Original: The rest of the experimental setup remained the same.
Revised: The rest of the experimental setup remained the same.

Original: We ran this experiment 10 times and observed that for 7 out of the 10 runs of the ﬁnal generated distribution was reasonably close to the real data distribution, while for 3 out of the 10 runs the generator did not learn the proper distribution.
Revised: We ran this experiment 10 times and observed that for 7 out of the 10 runs of the ﬁnal generated distribution was reasonably close to the real data distribution, while for 3 out of the 10 runs the generator did not learn the proper distribution.

Original: This is to say that we found the training algorithm was not as stable when the discriminator used Adam versus normal gradient ascent.
Revised: This is to say that we found the training algorithm was not as stable when the discriminator used Adam versus normal gradient ascent.

Original: The ﬁnal generated distribution from the 7 of 10 runs with reasonable distributions are shown in Figure 7.
Revised: The ﬁnal generated distribution from the 7 of 10 runs with reasonable distributions are shown in Figure 7.

Original: [Figure]
Revised: [Figure]

Original: Adversarial Training.
Revised: Adversarial Training.

Original: We now provide some further background on the adversarial training experiment and additional results.
Revised: We now provide some further background on the adversarial training experiment and additional results.

Original: It is now well-documented that the effectiveness of deep learning classiﬁcation models can be vulnerable to adversarial attacks that perturb the input data (see, e.g.,
Revised: It is now well-documented that the effectiveness of deep learning classiﬁcation models can be vulnerable to adversarial attacks that perturb the input data (see, e.g.,

Original: Biggio et al. 2013; Szegedy et al. 2014; Kurakin et al. 2017; Madry et al. 2018).
Revised: Biggio et al. 2013; Szegedy et al. 2014; Kurakin et al. 2017; Madry et al. 2018).

Original: A common approach toward remedying this vulnerability is by training the classiﬁcation model against adversarial perturbations.
Revised: A common approach toward remedying this vulnerability is by training the classiﬁcation model against adversarial perturbations.

Original: Recall from Section 6 that given a data distribution D over pairs of examples x ∈ R d and labels y ∈
Revised: Recall from Section 6 that given a data distribution D over pairs of examples x ∈ R d and labels y ∈

Original: [ k ] , parameters θ of a neural network, a set S ⊂ R d of allowable adversarial perturbations, and a loss function (cid:96) ( · , · , · ) dependent on the network parameters and the data, adversarial training amounts to considering a minmax optimization problem of the form min θ E ( x,y )
Revised: [ k ] , parameters θ of a neural network, a set S ⊂ R d of allowable adversarial perturbations, and a loss function (cid:96) ( · , · , · ) dependent on the network parameters and the data, adversarial training amounts to considering a minmax optimization problem of the form min θ E ( x,y )

Original: ∼D [max δ ∈S (cid:96) ( θ, x + δ, y )] .
Revised: ∼D [max δ ∈S (cid:96) ( θ, x + δ, y )] .

Original: A typical approach to solving this problem is an alternating optimization approach (Madry et al., 2018).
Revised: A typical approach to solving this problem is an alternating optimization approach (Madry et al., 2018).

Original: In particular, each time a batch of data is drawn from the distribution, T -steps of projected gradient ascent are performed by ascending along the sign of the gradient of the loss function with respect to the data and projecting back onto the set of allowable perturbations, then the parameters of the neural network are updated by descending along the gradient of the loss function with the perturbed examples.
Revised: In particular, each time a batch of data is drawn from the distribution, T -steps of projected gradient ascent are performed by ascending along the sign of the gradient of the loss function with respect to the data and projecting back onto the set of allowable perturbations, then the parameters of the neural network are updated by descending along the gradient of the loss function with the perturbed examples.

Original: The experimental setup we consider is analogous but the inner maximization loop performs T -steps of regular gradient ascent (not using the sign of the gradient and without projections).
Revised: The experimental setup we consider is analogous but the inner maximization loop performs T -steps of regular gradient ascent (not using the sign of the gradient and without projections).

Original: For the adversarial training experiment considered in Section 6, we also evaluated the trained models against various other attacks.
Revised: For the adversarial training experiment considered in Section 6, we also evaluated the trained models against various other attacks.

Original: Recall that the models were training using T = 10 steps of gradient ascent in the inner optimization loop with a learning rate of η 2 = 4 .
Revised: Recall that the models were training using T = 10 steps of gradient ascent in the inner optimization loop with a learning rate of η 2 = 4 .

Original: To begin, we evaluated the trained models against gradient ascent attacks with a ﬁxed learning rate of η 2 = 4 and a number of steps T ∈ { 5 , 10 , 20 , 40 } .
Revised: To begin, we evaluated the trained models against gradient ascent attacks with a ﬁxed learning rate of η 2 = 4 and a number of steps T ∈ { 5 , 10 , 20 , 40 } .

Original: We also evaluated the trained models against gradient ascent attacks with a ﬁxed budget of Tη 2 = 40 and various choices of T and η 2 .
Revised: We also evaluated the trained models against gradient ascent attacks with a ﬁxed budget of Tη 2 = 40 and various choices of T and η 2 .

Original: These results are presented in Figure 9.
Revised: These results are presented in Figure 9.

Original: Finally, we evaluated the trained models against attacks using the Adam optimization method with a ﬁxed budget of Tη 2 = 0 .
Revised: Finally, we evaluated the trained models against attacks using the Adam optimization method with a ﬁxed budget of Tη 2 = 0 .

Original: 04 and various choices of T and η 2 .
Revised: 04 and various choices of T and η 2 .

Original: These results are presented in Figure 10.
Revised: These results are presented in Figure 10.

Original: Notably, we see that our algorithm outperforms the baselines and similar conclusions can be drawn as from the experiments for adversarial training presented in Section 6.
Revised: Notably, we see that our algorithm outperforms the baselines and similar conclusions can be drawn as from the experiments for adversarial training presented in Section 6.

Original: The additional experiments highlight that our method of adversarial training is robust against attacks that the algorithm did not use in training when of comparable computational power and also that it improves robustness against attacks of greater computational power than used during training.
Revised: The additional experiments highlight that our method of adversarial training is robust against attacks that the algorithm did not use in training when of comparable computational power and also that it improves robustness against attacks of greater computational power than used during training.

Original: In Section 6, we showed the results of evaluating the gradient norms (cid:107)∇ f ( θ, A ( θ ))
Revised: In Section 6, we showed the results of evaluating the gradient norms (cid:107)∇ f ( θ, A ( θ ))

Original: (cid:107) as a function of the number of gradient ascent steps T in the adversary algorithm A and observed that it grows much slower than exponentially.
Revised: (cid:107) as a function of the number of gradient ascent steps T in the adversary algorithm A and observed that it grows much slower than exponentially.

Original: Here we provide more details on the setup.
Revised: Here we provide more details on the setup.

Original: We took a run of our algorithm trained with the setup described in Section 6 and retrieved the models that were saved after 25, 50, 75, and 100 training epochs.
Revised: We took a run of our algorithm trained with the setup described in Section 6 and retrieved the models that were saved after 25, 50, 75, and 100 training epochs.

Original: For each model, we then sampled 100 minibatches of data and for each minibatch performed T ∈ { 20 , 30 , 40 , 50 , 60 , 70 , 80 , 90 , 100 } steps of gradient ascent with learning rate η 2 = 4 (the learning rate from training) and then computed the norm of the gradient ∇ f
Revised: For each model, we then sampled 100 minibatches of data and for each minibatch performed T ∈ { 20 , 30 , 40 , 50 , 60 , 70 , 80 , 90 , 100 } steps of gradient ascent with learning rate η 2 = 4 (the learning rate from training) and then computed the norm of the gradient ∇ f

Original: ( θ, A ( θ )) where A corresponds to the gradient ascent procedure with the given number of steps and learning rate.
Revised: ( θ, A ( θ )) where A corresponds to the gradient ascent procedure with the given number of steps and learning rate.

Original: In Figure 2, which is reproduced here in Figure 11a, the mean of the norm of the gradients over the sampled minibatches are shown with the shaded window indicating a standard deviation around the mean.
Revised: In Figure 2, which is reproduced here in Figure 11a, the mean of the norm of the gradients over the sampled minibatches are shown with the shaded window indicating a standard deviation around the mean.

Original: We also repeated this procedure using η 2 = 1 and show the results in
Revised: We also repeated this procedure using η 2 = 1 and show the results in

Original: Figure 11b from which similar conclusions can be drawn.
Revised: Figure 11b from which similar conclusions can be drawn.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Finally, we provide details on the convolutional neural network model for the adversarial training experiments.
Revised: Finally, we provide details on the convolutional neural network model for the adversarial training experiments.

Original: In particular, this model is exactly the same as considered in (Nouiehed et al., 2019) and we reproduce it in Table 1.
Revised: In particular, this model is exactly the same as considered in (Nouiehed et al., 2019) and we reproduce it in Table 1.

Original: Experimental Details.
Revised: Experimental Details.

Original: For the experiments with neural network models we used two Nvidia GeForce
Revised: For the experiments with neural network models we used two Nvidia GeForce

Original: GTX 1080 Ti GPU and the PyTorch higher library(Deleu et al., 2019) to compute ∇ f
Revised: GTX 1080 Ti GPU and the PyTorch higher library(Deleu et al., 2019) to compute ∇ f

Original: ( θ, A ( θ )) .
Revised: ( θ, A ( θ )) .

Original: In total, running all the experiments in the paper takes about half of a day with this computational setup.
Revised: In total, running all the experiments in the paper takes about half of a day with this computational setup.

Original: The code for the experiments is included in the supplementary material with instructions on how to run.
Revised: The code for the experiments is included in the supplementary material with instructions on how to run.

Original: We built our code for the adversarial training based off code openly provided by the authors of (Nouiehed et al., 2019).
Revised: We built our code for the adversarial training based off code openly provided by the authors of (Nouiehed et al., 2019).