T OWARDS CONVERGENCE TO N ASH EQUILIBRIA IN TWO - TEAM ZERO - SUM GAMES
A BSTRACT
Contemporary applications of machine learning in two-team e-sports and the superior expressivity of multi-agent generative adversarial networks raise important and overlooked theoretical questions regarding optimization in two-team games.
Formally, two-team zero-sum games are defined as multi-player games where players are split into two competing sets of agents, each experiencing a utility identical to that of their teammates and opposite to that of the opposing team.
We focus on the solution concept of Nash equilibria (NE).
We first show that computing NE for this class of games is hard for the complexity class CLS.
To further examine the capabilities of online learning algorithms in games with fullinformation feedback, we propose a benchmark of a simple —yet nontrivial— family of such games.
These games do not enjoy the properties used to prove convergence for relevant algorithms.
In particular, we use a dynamical systems perspective to demonstrate that gradient descent-ascent, its optimistic variant, optimistic multiplicative weights update, and extra gradient fail to converge (even locally) to a Nash equilibrium.
On a brighter note, we propose a first-order method that leverages control theory techniques and under some conditions enjoys lastiterate local convergence to a Nash equilibrium.
We also believe our proposed method is of independent interest for general min-max optimization.
I NTRODUCTION
Online learning shares an enduring relationship with game theory that has a very early onset dating back to the analysis of fictitious play by (Robinson, 1951) and Blackwell’s approachability theorem (Blackwell, 1956).
A key question within this context is whether self-interested agents can arrive at a game-theoretic equilibrium in an independent and decentralized manner with only limited feedback from their environment.
Learning dynamics that converge to different notions of equilibria are known to exist for two-player zero-sum games (Robinson, 1951; Arora et al., 2012; Daskalakis et al., 2011), potential games (Monderer & Shapley, 1996), near-potential games (Anagnostides et al., 2022b), socially concave games (Golowich et al., 2020), and extensive form games (Anagnostides et al., 2022a).
We try to push the boundary further and explore whether equilibria —in particular, Nash equilibria— can be reached by agents that follow decentralized learning algorithms in two-team zero-sum games.
Team competition has played a central role in the development of game theory (Marschak, 1955; von Stengel & Koller, 1997; Bacharach, 1999; Gold, 2005), economics (Marschak, 1955; Gottinger, 1974), and evolutionary biology (Nagylaki, 1993; Nowak et al., 2004).
Recently, competition among teams has attracted the interest of the machine learning community due to the advances that multiagent systems have accomplished: e.g. , multi-GAN’s (Hoang et al., 2017; Hardy et al., 2019) for generative tasks, adversarial regression with multiple learners (Tong et al., 2018), or AI agents competing in e-sports ( e.g., CTF (Jaderberg et al., 2019) or Starcraft (Vinyals et al., 2019)) as well as card games (Moravˇc´ık et al., 2017; Brown & Sandholm, 2018; Bowling et al., 2015).
Our class of games.
We turn our attention to two-team zero-sum games a quite general class of min-max optimization problems that include bilinear games and a wide range of nonconvexnonconcave games as well.
In this class of games, players fall into two teams of size n, m and submit their own randomized strategy vectors independently.
We note that the games that we focus on are not restricted to team games in the narrow sense of the term “team” as we use it in sports, games, and so on; the players play independently and do not follow a central coordinating authority.
Rather, for the purpose of this paper, teams are constituted by agents that merely enjoy the same utility function.
This might already hint that the solution concept that we engage with is the Nash equilibrium (NE).
Another class of games that is captured by this framework is the class of adversarial potential games.
In these games, the condition that all players of the same team experience the same utility is weakened as long as there exists a potential function that can track differences in the utility of each player when they unilaterally deviate from a given strategy profile (see Appendix A.2 for a formal definition).
A similar setting has been studied in the context of nonatomic games (Babaioff et al., 2009).
Positive duality gap.
In two-player zero-sum games, i.e. , n = m = 1 , min-max (respectively max-min) strategies are guaranteed to form a Nash equilibrium due to Von Neumann’s minmax theorem (Von Neumann, 1928); ultimately endowing the game with a unique value.
The challenges arise for the case of n, m > 1 ; Schulman & Vazirani (2019b) prove that, in general, two-team games do not have a unique value.
They do so by presenting a family of team games with a positive duality gap, together with bounds concerning this gap.
These bounds quantify the effect of exchanging the order of commitment to their strategy either between the teams as a whole or the individual players.
Solution concept.
In this work, we examine the solution concept of Nash equilibrium (NE).
Under a Nash equilibrium, no player can improve their utility by unilaterally deviating.
The main downside of a NE for team games is the fact that such an equilibrium can be arbitrarily suboptimal for the team (Basilico et al., 2017a).
This is one of the reasons that the solution concept of team-maxmin equilibrium with a coordination device (TMECor) has dominated contemporary literature of team games, especially in regard to applications (Farina et al., 2018; Zhang et al., 2020; Cacciamani et al., 2021).
Under a TMECor, players are allowed to communicate before the game and decide upon combinations of strategies to be played during the game using an external source of randomness.
The undeniable advantage of a TMECor is that the expected utility of the team under it is greater than the expected utility under a NE (Basilico et al., 2017a).
Nevertheless, this favorable property of TMECor can by no means render the study of NE irrelevant.
In fact, the study of NE is always of independent interest within the literature of algorithmic game theory —especially questions corresponding to computational complexity.
Moreover, there exist settings in which ex ante coordination cannot be expected to be possible or even sensible; for example in (i) environments where the external sources of randomness are unreliable or nonexistent or visible to the adversarial team, (ii) games in which players cannot know in advance who they share a common utility with, (iii) adversarial potential games .
These games can model naturally occurring settings such as (a) security games with multiple uncoordinated defenders versus multiple similarly uncoordinated attackers, (b) the load balancing “game” between telecommunication service providers trying to minimize the maximum delay of service experienced by their customers versus the service users that try to individually utilize the maximum amount of broadband possible, and (c) the weak selection model of evolutionary biology where a species as a whole is a team, the genes of its population are the players and the alleles of each gene are in turn the actions of a player; the allele frequencies are independent across genes (Nagylaki, 1993; Nowak et al., 2004; Mehta et al., 2015).
Concluding, we could not possibly argue for a single correct solution concept for two-team games; there is no silver bullet.
In contrast, one has to assess which is the most fitting based on the constraints of a given setting.
A Nash equilibrium is a cornerstone concept of game theory and examining its properties in different games is always important.
The optimization point of view.
We focus on the solution concept of NE and we first note that computing local-NE in general nonconvex-nonconcave games is PPAD -complete
(Daskalakis et al., 2009; 2021).
Thus, all well-celebrated online learning, first-order methods like gradient descentascent (Lin et al., 2020; Daskalakis & Panageas, 2019), its optimistic (Popov, 1980; Chiang et al., 2012; Sridharan & Tewari, 2010), optimistic multiplicative weights update (Sridharan, 2012), and the extra gradient method (Korpelevich, 1976) would require an exponential number of steps in the parameters of the problem in order to compute an approximate NE under the oracle optimization model of (Nemirovskij & Yudin, 1983).
Additionally, in the continuous time regime, similar classes of games exhibit behaviors antithetical to convergence like cycling, recurrence, or chaos (VlatakisGkaragkounis et al., 2019).
Second, even if a regret notion within the context of team-competition could be defined, no-regret dynamics are guaranteed to converge only to the set of coarse correlated equilibria (CCE) (Fudenberg, 1991; Hannan, 2016).
CCE is a weaker equilibrium notion whose solutions could potentially be exclusively supported on strictly dominated strategies, even for simple symmetric two-player games (See also (Viossat & Zapechelnyuk, 2013)).
Surely, the aforementioned intractability remarks for the general case of nonconvex-nonconcave min-max problems provide a significant insight.
But, they cannot per se address the issue of computing Nash equilibria when the game is equipped with a particular structure, i.e. , that of two-team zero-sum games.
In fact, our paper addresses the following questions:
Can we get provable convergence guarantees to NE of decentralized first-order methods in two-team zero-sum games?
Our results.
First, with regards to computational complexity, we prove that computing an approximate (and possibly mixed) NE in two-team zero-sum games is
CLS -hard
Theorem 3.1); i.e. , it is computationally harder than finding pure NE in a congestion game or computing an approximate fixed point of gradient descent.
Second, regarding online learning for equilibrium computation, we prove that a number of established, decentralized, first-order methods are not fit for the purpose and fail to converge even asymptotically.
Specifically, we present a simple —yet nontrivial— family of two-team zero-sum games (with each team consisting of two players) where projected gradient descent-ascent (GDA), optimistic gradient descent-ascent (OGDA), optimistic multiplicative weights update (OMWU), and the extra gradient method (EG) fail to locally converge to a mixed NE (Theorem 3.3).
More broadly, in the case of GDA in nondegenerate team games with unique mixed NE, one could acquire an even stronger result for any high-dimensional configuration of actions and players (Theorem 3.2).
To the best of our knowledge, the described family of games is the first-of-its-kind in which all these methods provably fail to converge at the same time.
Third, we propose a novel first-order method inspired by adaptive control (Bazanella et al., 1997;
Hassouneh et al., 2004).
In particular, we use a technique that manages to stabilize unstable fixed points of a dynamical system without prior knowledge of their position and without introducing new ones.
It is important to note that this method is a modification of GDA that uses a stabilizing feedback which maintains the decentralized nature of GDA.
Finally, in Section 4 we provide a series of experiments in simple two-team zero-sum GAN’s.
We also show that multi-GAN architectures achieve better performance than the single-agent ones, relative to the network capacity, when they are trained on synthetic or real-world datasets like CIFAR10.
Existing algorithms for NE in multiplayer games.
The focus of the present paper is examining algorithms for the setting of repeated games (Cesa-Bianchi & Lugosi, 1999, Chapter 7).
If we do not restrict ourselves to this setting, there are numerous centralized algorithms (Lipton et al., 2003;
Berg & Sandholm, 2017) and heuristics (Gemp et al., 2021) that solve the problem of computing
Nash equilibria in general multi-player games .
P RELIMINARIES
Our setting.
A two-team game in normal form is defined as a tuple Γ( N , M , A , B , { U A , U B } ) .
The tuple is defined by
(i) a finite set of n = |N| players belonging to team A , as well as a finite set of m = |M| players belonging to team B ; (ii) a finite set of actions (or pure strategies )
A i = { α 1 , . . .
, α n i } per player i ∈ N ; where A := (cid:81)
i A
i denotes the ensemble of all possible action profiles of team A , and respectively, a finite set of actions (or pure strategies )
B i
= { β 1 , . .
, β n i } per player i ∈ M , where B := (cid:81) i B i .
(iii) a utility function for team A , U A : A × B → R (resp.
U B : A × B → R for team B )
We also use α = ( α 1 , . . .
α n ) to denote the strategy profile of team
A players and β = ( β 1 , . .
. , β m ) the strategy profile of team B players.
Finally, each team’s payoff function is denoted by U A , U B : A × B → R , where the individual utility of a player is identical to her teammates, i.e., U i =
U A & U j
= U B ∀ i ∈ N and j ∈ M .
In this general context, players could also submit mixed strategies , i.e, probability distributions over actions.
Correspondingly, we define the product distributions x = ( x 1 , . . .
, x n ) ,
y =
( y 1 , . . .
, y m ) as team A and B ’s strategies respectively, in which x i ∈ ∆( A i ) and y j ∈ ∆( B j ) .
Conclusively, we will write X :=
(cid:81) i ∈N X i = (cid:81) i ∈N ∆( A i ) , Y := (cid:81) i ∈M Y i = (cid:81) i ∈M ∆( B i ) the space of mixed strategy profiles of teams A, B .
A two-team game is called two-team zero-sum if U B = − U
A =
U which is the main focus of this paper.
Moreover, we assume that the game is succinctly representable and satisfies the polynomial expectation property (Daskalakis et al., 2006).
This means that given a mixed strategy profile, the utility of each player can be computed in polynomial time in the number of agents, the sum of the number of strategies of each player, and the bit number required to represent the mixed strategy profile.
A Nash equilibrium (NE) is a strategy profile ( x ∗ , y ∗ ) ∈
X × Y such that
[Equation]
A first approach to computing NE in Two-Team Zero-Sum games.
Due to the multilinearity of the utility and the existence of a duality gap, the linear programming method used in two-player zero-sum games cannot be used to compute a Nash equilibrium.
For the goal of computing Nash equilibrium in two-team zero-sum games, we have experimented with a selection of online learning, first-order methods that have been utilized with varying success in the setting of the two-person zero-sum case.
Namely, we analyze the following methods: (i) gradient descent-ascent (GDA) (ii) optimistic gradient descent-ascent (OGDA) (iii) extra gradient method (EG) (iv) optimistic multiplicative weights update method (OMWU).
For their precise definitions, we refer to Appendix B.
The below folklore fact will play a key role hereafter.
Fact 2.1.
Any fixed point of the aforementioned discrete-time dynamics (apart from OMWU) on the utility function necessarily corresponds to a Nash Equilibrium of the game.
Hence, an important test for the asymptotic behavior of GDA, OGDA, EG, and OMWU methods is to examine whether these methods stabilize around their fixed points which effectively constitute the
Nash equilibria of the game.
In Section 3.2, we show that in the absence of pure Nash equilibria, all the above methods fail to stabilize on their fixed points even for a simple class of two-team game with ( n = 2 , m = 2) .
Consequently, they fail to converge to the mixed Nash equilibrium of the game.
The presence of these results demonstrates the need for a different approach that lies outside the scope of traditional optimization techniques.
Inspired by the applications of washout filters to stabilize unknown fixed points and the adaptive control generalizations of the former, we design a new variant of GDA “vaned” with a feedback loop dictated by a pair of two matrices.
In contrast to the aforementioned conventional methods, our proposed technique surprisingly accomplishes asymptotic last-iterate convergence to its fixed point, i.e. , the mixed Nash equilibria of the team game.
( K , P ) -vaned GDA Method.
After concatenating the vectors of the minimizing and the maximizing agents — z ( k ) =
( x ( k ) , y ( k ) ) — our method for appropriate matrices K , P reads:
[Equation]
Intuitively, the additional variable θ ( k ) holds an estimate of the fixed point, and through the feedback law η K ( z ( k ) − θ ( k ) )
the vector z stabilizes around that estimate which slowly moves towards the real fixed point of the GDA dynamic.
2. T WO I LLUSTRATIVE E XAMPLES
Our first example plays a dual role: first, it demonstrates how two-team min-max competition can capture the formulation of multi-agent GAN architectures; second, it hints at the discrepancy be-
tween the results of optimization methods, since —as we will see— GDA will not converge to the Nash equilibrium/ground-truth distribution.
Generally, the solution that is sought after is the min max solution of the objective function (Goodfellow et al., 2014) which are NP -hard to compute in the general case (Borgs et al., 2008); nevertheless, applications of GANs have proven that first-order stationary points of the objective function suffice to produce samples of very good quality.
2.1. L EARNING A MIXTURE OF GAUSSIANS WITH MULTI - AGENT GAN’ S
Consider the case of O , a mixture of gaussian distribution with two components, C 1 ∼ N ( µ , I ) and
C 2 ∼ N ( − µ , I ) and mixture weights π 1 , π 2 to be positive such that π 1 + π 2 = 1 and π 1 , π 2 ̸ = 12 .
To learn the distribution above, we utilize an instance of a Team -WGAN in which there exists a generating team of agents G p : R → R , G θ : R n → R n , and a discriminating team of agents
D v : R n → R , D w : R n → R , all described by the following equations:
[Equation]
The generating agent G θ maps random noise ξ ∼ N (0 , I ) to samples while generating agent G p ( ζ ) , utilizing an independent source of randomness ζ ∼ N (0 , 1) , probabilistically controls the sign of the output of the generator G θ .
The probability of ultimately generating a sample y =
ξ + θ is in expectation equal to p , while the probability of the sample being y =
−
z − θ is equal to 1 − p .
On the other end, there stands the discriminating team of D v , D w .
Discriminators, D v ( y ) , D w ( y ) map any given sample y to a scalar value accounting for the “realness” or “fakeness” of it – negative meaning fake, positive meaning real.
The discriminators are disparate in the way they measure the realness of samples as seen in their definitions.
We follow the formalism of the Wasserstein GAN to form the optimization objective: max v , w min θ ,p
[Equation]
Equation (2) yields the simpler form:
[Equation]
It is easy to check that Nash equilibria of (2) must satisfy:
[Equation]
Figure 1 demonstrates both GDA’s failure and OGDA, EG, and our proposed method, KPV-GDA succeeding to converge to the above Nash equilibria and simultaneously discovering the mixture of the ground-truth.
2.1. M ULTIPLAYER M ATCHING P ENNIES
Interestingly enough, there are non-trivial instances of two-team competition settings in which even
OGDA and EG fail to converge.
Such is the case for a team version of the well-known game of matching pennies.
The game can be shortly described as such: “ coordinate with your teammates to play a game of matching pennies against the opposing team, coordinate not and pay a penalty ”.
(We note that this game is a special case of the game presented in Section 3.3.)
As we can see in Figures 2a and 2b, this multiplayer generalized matching pennies game constitutes an excellent benchmark on which all traditional gradient flow discretizations fail under the perfect competition setting.
Interestingly, we are not aware of a similar example in min-max literature and it has been our starting point for seeking new optimization techniques inspired by Control theory.
Indeed, the
KPV-GDA variation with ( K , P ) =
( − 1 .
· I , 0 .
3 · I ) achieves to converge to the unique mixed Nash
Equilibrium of the game.
In the following sections, we provide theorems that explain formally the behavior of the examined dynamics.
(a) Generalized matching pennies under different algorithms.
For the precise definition of the game, we refer to appendix C. (b) Projected Trajectory of Team A under differ- ent algorithms.
The sketched surface is not part of the feasible team strategy profiles (product of distributions).
M AIN RESULTS
In this section, we will prove that computing a Nash equilibrium in two-team zero-sum games is computationally hard and thus getting a polynomial-time algorithm that computes a Nash equilibrium is unlikely.
Next, we will demonstrate the shortcomings of an array of commonly used online learning, first-order methods, and then we will provide a novel, decentralized, first-order method that locally converges to NE under some conditions.
3. O N THE HARDNESS OF COMPUTING NE IN TWO - TEAM ZERO - SUM GAMES
As promised, our first statement characterizes the hardness of NE computation in two-team zero-sum games:
Theorem 3.1 ( CLS -hard) .
Computing a Nash equilibrium in a succinctly represented two-team zero-sum game is CLS -hard.
The main idea of the proof of Theorem 3.1 relies on a reduction of approximating Nash equilibria in congestion games, which has been shown to be complete for the CLS complexity class.
The class
CLS contains the problem of continuous optimization.
We defer the proof of the above theorem to the paper’s supplement.
3. F AILURE OF COMMON ONLINE , FIRST - ORDER METHODS
The negative computational complexity result we proved for two-team zero-sum games (Theorem 3.1) does not preclude the prospect of attaining algorithms (learning first-order methods) that converge to Nash equilibria.
Unfortunately, we prove that these methods cannot guarantee convergence to Nash equilibria in two-team zero-sum games in general.
In this subsection, we are going to construct a family of two-team zero-sum games with the property that the dynamics of GDA, OGDA, OMWU, and EG are unstable on Nash equilibria.
This result is indicative of the challenges that lie in the min-max optimization of two-team zero-sum games and the reason that provable, nonasymptotic convergence guarantees of online learning have not yet been established.
Before defining our benchmark game, we prove an important theorem which states that GDA does not converge to mixed Nash equilibria.
This fact is a stepping stone in constructing the family of team
-zero sum games later.
We present the proof of all of the below statements in detail in the paper’s appendix (Appendix B).
Weakly-stable Nash equilibrium.
(Kleinberg et al., 2009; Mehta et al., 2015) Consider the set of
Nash equilibria with the property that if any single randomizing agent of one team is forced to play any strategy in their current support with probability one, all other agents of the same team must remain indifferent between the strategies in their support.
This type of Nash equilibria is called weakly-stable.
We note that pure Nash equilibria are trivially weakly-stable.
It has been shown that mixed Nash equilibria are not weakly-stable in generic games
We can show that Nash equilibria that are not weakly-stable Nash are actually unstable for GDA.
Moreover, through standard dynamical systems machinery, that the set of initial conditions that converges to Nash equilibria that are not weakly-stable should be of Lebesgue measure zero.
Formally, we prove that:
Theorem 3.2 (Non weakly-stable Nash are unstable) .
Consider a two-team zero-sum game with the utility function of Team B ( y vector) being U ( x , y ) and Team A ( x vector) being − U ( x , y ) .
Moreover, assume that ( x ∗ , y ∗ ) is a Nash equilibrium of full support that is not weakly-stable.
It follows that the set of initial conditions so that GDA converges to ( x ∗ , y ∗ ) is of measure zero for step size
η < 1 L where L is the Lipschitz constant of ∇ U .
3. G ENERALIZED M ATCHING P ENNIES (GMP)
Inspired by Theorem 3.2, in this section we construct a family of team zero-sum games so that
GDA, OGDA, OMWU, and EG methods fail to converge (if the initialization is a random point in the simplex, the probability of convergence of the aforementioned methods is zero).
The intuition is to construct a family of games, each of which has only mixed Nash equilibria (that are not weakly-stable), i.e. , the constructed games should lack pure Nash equilibria; using Theorem 3.2, it would immediately imply our claim for GDA.
It turns out that OGDA, OMWU, and EG also fail to converge for the same family.
HH
HT/TH
TT
HH 1 , −ω, − ω − 1 ,
HT/TH − ω,ω 0 ,− ω,ω
TT − 1 ,ω, − ω 1 , −
Definition of GMP.
Consider a setting with two teams (Team A , Team B ), each of which has n = 2 players.
Inspired by the standard matching pennies game and the game defined in (Schulman & Vazirani, 2019a), we allow each agent i to have two strategies/actions that is S
=
{ H, T } for both teams with 2 4 possible strategy profiles.
In case all the members of a Team choose the same strategy say H or T then the Team “agrees” to play H or
T (otherwise the Team “does not agree”).
Thus, in the case that both teams “agree”, the payoff of each team is actually the payoff for the two-player matching pennies.
If one team “agrees” and the other does not, the team that “agrees” enjoys the payoff ω ∈ (0 , 1) and the other team suffers a penalty ω .
If both teams fail to “agree”, both teams get payoff zero.
Let x i with i ∈ { 1 , 2 } be the probability that agent i of Team A chooses H and 1 − x i the probability that she chooses T .
We also denote x as the vector of probabilities for
Team A .
Similarly, we denote y i for i ∈ { 1 , 2 } be the probability that agent i of Team B chooses H and 1 − y i the probability that she chooses T and y the probability vector.
Properties of GMP.
An important remark on the properties of our presented game is due.
Existing literature tackles settings with (i) (weak-)monotonocity (Mertikopoulos et al., 2019; Diakonikolas et al., 2021), (ii) cocoercivity (Zhu & Marcotte, 1996), (iii) zero-duality gap (Von Neumann, 1928), (iv) unconstrained solution space (Golowich et al., 2020) .
Our game is carefully crafted and – although it has a distinct structure and is nonconvex-nonconcave only due to multilinearity – satisfies none of the latter properties.
This makes the (local) convergence of our proposed method even more surprising.
(See also Appendix B.6.)
The first fact about the game that we defined is that for ω ∈ (0 , 1) , there is only one Nash equilibrium ( x ∗ , y ∗ ) , which is the uniform strategy, i.e. , x ∗ 1 = x ∗ 2 = y ∗ 1 = y ∗ 2 = 12 for all agents i.
Lemma 3.1 (GMP has a unique Nash) .
The Generalized Matching Pennies game exhibits a unique
Nash equilibrium which is ( x ∗ , y ∗ ) =
Remark 1.
The fact that the game we defined has a unique Nash equilibrium that is in the interior of [0 , 1] 4 is really crucial for our negative convergence results later in the section as we will show that it is not a weakly-stable Nash equilibrium and the negative result about GDA will be a corollary due to Theorem 3.2.
We also note that if ω = 1 then there are more Nash equilibria, in particular the ( 0 , 0 ) , ( 1 , 0 ) , ( 0 , 1 ) , ( 1 , 1 ) which are pure.
The following Theorem is the main (negative) result of this section.
Theorem 3.3 (GDA, OGDA, EG, and OMWU fail) .
Consider GMP game with ω ∈ (0 , 1) .
Assume that η GDA < 14 , η OGDA < min( ω, 18 ) , η EG < ω 2 , and η OMWU < min (cid:0) 14 , ω 2 (cid:1) (bound on the stepsize for GDA, OGDA, OMWU, and EG methods respectively).
It holds that the set of initial conditions so that GDA, OGDA, OMWU, and EG converge (stabilize to any point) is of measure zero.
Remark 2.
Theorem 3.3 formally demonstrates that the behavior of algorithms mentioned in Section 2.1.2 are not a result of “bad parametrization”, and in fact, the probability that any of them converges to the NE is equal to the probability that the initialization of the variables coincides with the NE (Lebesgue measure zero).
Remark 3 (Average iterate also fails) .
One might ask what happens when we consider average iterates instead of the last iterate.
It is a well-known fact (Syrgkanis et al., 2015) that the average iterate of no-regret algorithms converges to coarse correlated equilibria (CCE) so we expect that the average iterate stabilizes.
Nevertheless, CCE might not be Nash equilibria.
Indeed we can construct examples in which the average iterate of GDA, OGDA, OMWU, and EG experimentally fail to stabilize to Nash equilibria.
In particular, we consider a slight modification of GMP; players and strategies are the same but the payoff matrix has changed and can be found reads
[Figure]
Figure 3 shows that the average iterates of GDA, OGDA, OMWU, and EG stabilize to points that are not Nash equilibria.
We note that since our method (see next subsection) converges locally, the average iterate should converge locally to a Nash equilibrium.
3. O UR PROPOSED METHOD
The aforementioned results prove that the challenging goal of computing two-team zero-sum games calls for an expansion of existing optimization techniques.
The mainstay of this effort and our positive result is the KPV-GDA method defined in (KPV-GDA) which is inspired by techniques of adaptive control literature.
The first statement we make is that KPV-GDA stabilizes around any
Nash equilibrium for appropriate choices of matrices K , P :
Theorem 3.4 (KPV-GDA stabilizes) .
Consider a team zero-sum game so that the utility of Team
B is U ( x , y ) and hence the utility of Team A is − U ( x , y ) and a Nash equilibrium ( x ∗ , y ∗ ) of the game.
Moreover, we assume
[Equation]
For any fixed step size η > 0 , we can always find matrices K, P so that KPV-GDA method defined in (KPV-GDA) converges locally to ( x ∗ , y ∗ ) .
This is an existential theorem and cannot be generally useful in practice.
Further, this dynamic would not be necessarily uncoupled and the design of matrices K and P could necessitate knowledge of the NE we are trying to compute.
Instead, our next statement provides sufficient conditions under which a simple parametrization of matrices K , P results in an uncoupled, converging dynamic:
Theorem 3.5.
Consider a two-team zero-sum game so that the utility of Team B is U ( x , y ) , the utility of Team A is − U ( x , y ) , and a Nash equilibrium ( x ∗ , y ∗ ) .
Moreover, let
[Equation]
and E be the set of eigenvalues ρ of H with real part positive, that is E =
Eigenvalues of matrix H , ρ : Re ( ρ ) > 0 } .
We assume that H is invertible and moreover
[Equation]
We set K = k · I , P = p · I .
There exist small enough step size η > 0 and scalar p > 0 and for any k ∈ ( − β, − α ) so that (KPV-GDA) with chosen K , P converges locally to ( x ∗ , y ∗ ) .
E XPERIMENTS
In this section, we perform a series of experiments to further motivate the study of two-team zerosum games, especially in the context of multi-agent generative adversarial networks (multi-GANs).
A multi-agent generative adversarial network (multi-GAN) (Arora et al., 2017; Hoang et al., 2017; 2018; Zhang et al., 2018; Tang, 2020; Hardy et al., 2019; Albuquerque et al., 2019) is a generative adversarial network (GAN) that leverages multiple “agents” (generators and/or discriminators) in order achieve statistical and computational benefits.
In particular, Arora et al.
formally proved the expressive superiority of multi-generator adversarial network architectures something that we empirically verify in Section 4.
In this direction, researchers strive to harness the efficacy of distributed processing by utilizing shallower networks that can collectively learn more diverse datasets 4 .
At first, the superiority of multi-GANs might appear to contrast our theoretical findings; but in reality, the superiority comes from the quality of solutions that are attainable from multi-agent architectures ( expressivity ) and the fact that hardness ( complexity ) translates to rates of convergence but not non-convergence.
Single agent GANs quickly converge to critical points that are not guaranteed to capture the distribution very well.
In figure 4 we see the fast convergence of a single-agent GAN to solutions of bad quality versus the convergence of a multi-GAN to an obviously better solution.
Due to space constraints, we defer further discussion of the experiments at Section D.1.
C ONCLUSIONS AND O PEN P ROBLEMS
In this work we study the wide class of nonconvex-nonconcave games that express the two-team competition, inspired broadly by the structure of the complex competition between multi-agent generators and discriminators in GAN’s.
Furthermore, in this setting of two-team zero-sum games , we have presented a number of negative results about the problem of computing a Nash equilibrium.
Moreover, through a simple family of games that we construct, we prove the inability of commonly used methods for min-max optimization such as GDA, OGDA, OMWU, and EG to converge both in average and in the last iterate to Nash Equilibria which comes to a stark contrast to recent literature that is concerned with simpler games.
We have also presented an optimization method (called
KPV-GDA) that manages to stabilize around Nash equilibria.
A CKNOWLEDGEMENTS
Ioannis Panageas would like to acknowledge a start-up grant. Emmanouil V. VG is grateful for the financial support by FODSI Postdoctoral Fellowship. This work was partially completed while IP and EVVG were visiting research fellows at the Simons Institute for Theory of Computing during Learning and Games Semester
A F URTHER RELATED WORK
The properties of team games have given rise to a large amount of literature which we cannot hope to review here; instead we refer the reader to (Kim et al., 2019; Gold, 2005) and references therein.
Additionally, convergence properties of online learning, first-order methods ( e.g. GDA, OGDA,
OMWU, EG) in team games are not well understood.
Team games.
For team games, the solution concept of team-maxmin equilibria (TME), a notion introduced by von Stengel & Koller (1997), is a rather well-understood solution concept.
In this category of two-team zero-sum games, a team of n players plays defensively against a single adversarial player.
TME’s provide crucial information about robustness since they give the team players the highest payoff possible when they employ uncoordinated strategies.
Unfortunately, TMEs correspond to the global optimum of a nonconvex nonlinear program which is NP -hard to compute and as such precludes the use of online learning, first-order algorithms for their computation.
Basilico et al.
(2017b) provided heuristics with iterative linear programming and support enumeration methods with proved approximation guarantees.
For the case of extensive or large-sized normal form games, Basilico et al. (2017a); Celli & Gatti (2018) introduced a relaxed notion of team-maxmin equilibrium with coordination device (TMECor).
In a TMECor, the team of players decides their joint strategies ex-ante against the adversary; that is, the team members are allowed to discuss and agree on tactics before the game starts, but they cannot communicate during the game.
While TMECor’s share some properties with NE’s in zero-sum two-player games ( e.g ., exchangeability), the proposed algorithms by Celli & Gatti (2018) leverage tools like Mixed-Integer Linear
Program (MILP) that involve a large number of integer variables.
In order to tackle the large size of the program heuristically, Zhang et al.
; Zhang & An (2020) have proposed modified versions of the incremental strategy generation (ISG) algorithm
(See McMahan et al.
for large-sized games.
Multi-GAN’s.
We cannot aspire to survey the literature of multi-GAN’s in its entirety.
Scratching only the surface, we mention that: (a) In MGAN Hoang et al.
and MAD-GAN Li et al.
(2019), we have a mixture of generative models with asymptotically infinite capacity networks.
(b) In the seminal case of MIX+GAN Arora et al.
(2017), an extra regularization term is typically added to discourage the weights of a mixture of generators being too far away from uniform.
(c) In Stackelberg GAN, Zhang et al.
(2018) exploit the leader-followers relation of discriminator vs generators to achieve smaller minimax gap and better Frech ´ et metrics.
For the case of multiple-discriminators : (d) In GMAN Durugkar et al.
(2016), an increased number of agents are used to achieve higher quality images, while (e) MD-GAN Hardy et al.
(2019) is designed to generalize over different datasets and multiple tasks that are spread on multiple workers.
A. D ISCUSSION ABOUT DIFFERENT SOLUTION CONCEPTS
In this subsection, we would like to stress some differences between the different equilibrium notions (TMECor, TME, and NE).
Initially, it is easy to see that TMECor similar to coarse correlated equilibria can achieve better social welfare but, as in the case of congestion games, their price of anarchy can become significantly worse Roughgarden (2009).
In order to improve this, TMECor request prior knowledge of the game.
For example, as Zhang et al.
(2020) mention: For example, in multiplayer poker games, a team may play against an adversary player, but they cannot communicate and discuss their strategy during the game due to the rule.
However in many contemporary challenging AI models, the agents/players of each team do not necessarily have knowledge of the actual game.
Thus, the only thing the team players can decide in advance is the decision rules they will follow.
Thus, the main conceptual reason that we focus on NE’s in comparison to TMECor is is because it is a way of reasoning about games in which players do not necessarily need to know in which team they fall into in advance and/or they cannot be expected to be able to coordinate their actions with players of common interest.
Problems of this kind are prevalent in modern ML applications like strategic conflict resolution (Leibo et al., 2017), coordination between autonomous vehicles (Cao et al., 2012) , or collaboration of agents in defensive escort teams (Sheikh et al., 2019).
In this kind of games each agent is simulta- neously working towards maximizing its own payoff (local reward) as well as the collective success of the team (global reward) .
Conclusively , the notion of Nash equilibrium seems to be a solution concept that needs to be investigated thoroughly in the setting of two-team zero-sum games.
A. A DVERSARIAL POTENTIAL GAMES .
Consider a normal form game with two sets of players N and M .
Following the conventions of
Section 2, the minimizers N will be associated with cost functions
c i , i ∈ [ n ] , while the maximizers
M will be associated with utilities u j , j ∈ [ m ] .
Further, there exists an underlying potential function Φ : X × Y → R , so that
[Equation]
While this setting is not a two-team zero-sum game, it can still be captured by one, namely Γ( n, m, Φ) .
More precisely, (5) implies that c
i ( x , y ) =
Φ( x , y )
+ Ψ i ( x − i , y ) , for any i ∈ N and
( x , y ) ∈
X × Y , where Ψ i does not depend on x i , and (6) implies that u j ( x , y ) =
Φ( x , y ) + Ψ ′ j
( x , y − j ) , where Ψ j does not depend on y j .
As a result, by the same reasoning we used in the proof of Theorem 3.1, we arrive at the following conclusion:
Proposition A.1.
A strategy profile that is an ϵ -approximate Nash equilibrium with respect to the two-team zero-sum game Γ( n, m, Φ) will be an ϵ -approximate Nash equilibrium of the original adversarial potential game.
B G AME D YNAMICS
The following algorithms are ubiquitous in the literature of min-max optimization and have known variable success under different settings.
Gradient descent-ascent (GDA) is the prototypical method upon which optimistic gradient descent (OGDA), extra gradient (EG), and our proposed method ( K , P ) -vaned gradient descent-ascent (KPV-GDA) are built.
Optimistic multiplicative weights update is the optimistic variant of the MWU.
B. G RADIENT D ESCENT -A SCENT
[Equation]
B. O PTIMISTIC
[Equation] RADIENT D ESCENT - [Equation]
SCENT
[Equation]
B. E XTRA G RADIENT M ETHOD
 
[Equation]
B. O PTIMISTIC M ULTIPLICATIVE W EIGHTS U PDATE M ETHOD
 
[Table]
[Equation]
[Table]
[Equation]
Notations x ( t ) i ( or y ( t ) j ) stand for the strategy vector of the i -th minimizing (or J -th maximizing) agent at time-step t ; while x i,k and y j,k denote the k -th pure strategy of player i of team A and player
J of team B respectively.
The step size is denoted as η .
Operators Π X
i , Π Y j , Π Z are the projection operators to the corresponding products of simplices.
The projection operator is important since remaining inside of the set of feasible solutions is not guaranteed at every step except for the case of the OMWU method.
B. P ROPERTIES OF GMP
We will formally demonstrate how a series of conditions needed for the convergence of the extra gradient method are not satisfied for GMP.
Variational Inequalities Let a saddle-point problem (which can be seen as a zero-sum game between agents x , y ):
[Equation]
Also, let an operator F : R d → R d that is L -Lipschitz continuous for a norm ∥ · ∥ p .
The saddle-point problem can be formulated in the form of the equivalent Stampacchia variational inequality (SVI), i.e. find a z ∗ ∈ R d , z ∗ =
( x ∗ , y ∗ ) , such that for every z :
[Equation]
Operator F is said to be monotone if for every z , w ∈ R d :
[Equation]
One more noteworthy variational inequality is the Minty Variational Inequality (MVI) with respect to a point z ∗ is defined as:
[Equation]
Mertikopoulos et al.
(2019) prove the convergence of the extra gradient method by utilizing the monotonicity of F .
In this case, the solutions to the MVI, z ∗ , coincide with the solutions to the SVI.
In particular F is taken to be F ( x , y )
= (cid:16) ∇ x U ( x , y ) , −∇ y U ( x , y ) (cid:17) .
Diakonikolas et al.
(2021) prove convergence under a weaker form of the MVI for operator F :
[Equation] [Equation]
With ρ a constant in the interval [0 , 14 L ] .
We note that with ρ = 0 we retrieve the MVI which we proved is not satisfied.
Setting z
=
we see that the MVI is violated: ⟨ F ( z ) , z − z ∗ ⟩ = − ω 9 .
As for the weak-MVI, we numerically verify that for ω = 1 / 2 and every ρ in { k 10 3 | k = 0 , 1 , ..., 10 3 · 14 L } that a z can be found such that is violates the inequality.
Hence, already known techniques (Mertikopoulos et al., 2018; Diakonikolas et al., 2021; Golowich et al., 2020) cannot tackle the problem of computing a Nash equilibrium in GMP.
Additionally,
Golowich et al.
(2020) only analyze the unconstrained setting.
Cocoercivity Cocoerciviy is a condition that is stronger than the MVI.
Hence, it follows that cocoercivity does not hold in GMP.
Hidden convex-concave games.
Flokas et al.
(2021) proved global convergence for gradient flow (continuous time) when a game is strictly hidden convex-concave.
Our setting of GMP does not meet neither criteria.
B. GMP SATISFIES THE SUFFICIENT CONDITIONS OF T HEOREM 3.
We can verify that the generalized matching pennies game satisfies the sufficient conditions of Theorem 3.5.
Namely, the eigenvalues of H are the following
[Equation]
Hence, the set of the eigenvalues with a positive real part E is E = { 2 ω − 2 i, 2 ω
+ 2 i } .
As such, we see that indeed,
[Equation]
For any choice of k ∈ ( − 2 ω − 2 ω , − 2 ω ) the algorithm converges.
For the case that ω = 1 / 2 , it suffices that k ∈ ( − 5 , − 1) .
C P ROOFS FOR S ECTION 2. C. D ERIVATION OF THE MIN - MAX OBJECTIVE IN EQUATION
By the definition of variance, we get that : Var (cid:2)
x
i (cid:3) =
[Equation]
[ x
[Equation]
[Equation]
[Equation]
Var
[Equation]
i
[Equation]
(cid:0) E [ x i ] (cid:1) 2 .
More precisely for any x ∼ N ( µ, I ) , we get E [ x 2 i ] = 1 + µ 2 i .
Additionally, after calculations, we can get that:
[Equation]
Moreover, it is easy to check that for a mixture D ( µ 1 , σ 1 I , µ 2 , σ 2 I , π 1 , π 2 ) of two multi-dimensional normal distributions N ( µ 1 , σ 1 I ) , N ( µ 1 , σ 1 I ) with corresponding weights π 1 , π 2 :
[Equation]
Specifically for the case of µ 1 = µ = − µ 2 , this equivalent with:
[Equation]
Hence:
[Equation]
So we can apply the following manipulations in the objective (The notation [ x 2 i ] stands for the vector that has as entries the values of the vector x squared):
[Table]
[Equation]
C. P ROOFS FOR S ECTIONC. P ROOF OF T HEOREM 3.
We will reduce the problem of finding a Nash equilibrium in congestion games to the problem of finding a Nash equilibrium in two-team zero-sum games.
The result then will follow since computing Nash equilibria in congestion games is CLS -hard (Babichenko & Rubinstein, 2021).
As a recent result (Fearnley et al., 2021) shows CLS is equal to the intersection of PLS and PPAD , two important classes of total problems.
PPAD captures diverse problems in combinatorics and (non-)cooperative game theory, like the ε -approximation of a mixed Nash Equilibrium in a graphical game or the computation of market equilibria.
PLS , for “Polynomial Local Search”, captures problems of finding a local minimum of an objective function f, in contexts where any candidate solution x has a local neighborhood within which we can readily check for the existence of some other point having a lower value of f. Many diverse local optimization problems have been shown complete for PLS , attesting to its importance.
Examples include searching for a local optimum of the TSP according to the Lin-Kernighan heuristic (Papadimitriou, 1992) and finding pure Nash equilibria in many-player congestion games (Fabrikant et al., 2004).
The complexity class CLS (“Continuous Local Search”) was introduced by Daskalakis and Papadimitriou (Daskalakis & Papadimitriou, 2011) to classify various important problems that lie in both PPAD and PLS .
CLS is seen as a strong candidate for capturing the complexity of some of those important problems, like
In our reduction, a congestion game is defined by the tuple ( N ; E ; ( S i ) i ∈ N ; ( c e ) e ∈ E ) where N is the set of agents , E is a set of resources (also known as edges or facilities ), and each player i has a set S i of subsets of E .
Each strategy s i ∈ S
i is a set of edges (a path ), and c e is a cost (negative utility) function associated with facility e .
For a strategy profile s =
( s 1 , s 2 , . .
, s N ) , the cost of player i is given by c
i ( s ) =
(cid:80) e ∈
s i
c e ( ℓ e ( s )) , where ℓ e ( s ) is the number of players using e in s (the load of edge e ).
It is a well-known result (Rosenthal, 1973) that congestion games exhibit a potential function Φ( s ) , that
[Equation]
with the property that if any agent i changes her strategy to s ′ i it holds that
[Equation]
Succinct representation.
We assume that the congestion game and the team game are succinctly representable (Daskalakis et al., 2006) and admits the polynomial expectation property (Daskalakis et al., 2006).
That is, the expected cost of each player i , c i ( · ) , for any given mixed strategy profile ( s 1 , . .
, s n ) is computed in time polynomial in: (i) the number agents n , (ii) (cid:80) i | S
i | where S i is the finite set of agent i ’s strategies, (iii) the number of bits required to represent the mixed strategy profile ( s 1 , . . .
, s n ) .
Reduction.
Consider a congestion game ( N ; E ; ( S i ) i ∈ N ; ( c e ) e ∈ E ) with n = | N | players and potential function Φ .
We define a team zero-sum game as follows:
Team A has n players, in which each agent i chooses strategies from S i .
Team B has n players, with each agent j having only one possible choice (singleton set of actions) call it d , i.e., these are dummy players.
If players from
Team A choose strategy profile s (Team B has only one choice) then they get utility u A ( s , d )
= − Φ( s ) .
The utility members of Team B get is u B ( s , d ) =
Φ( s ) .
Let x ∗ :
= ( x ∗ 1 , ..., x ∗ n ) and ( d, ..., d ) be a (possibly mixed)
Nash equilibrium in the team zero-sum game we defined.
We shall show that ( x ∗ 1 , ..., x ∗ n ) is a Nash equilibrium of the original congestion game and the reduction will be complete.
Aiming for contradiction, suppose ( x ∗ 1 , ..., x ∗ n ) is not a
Nash equilibrium of the original congestion game.
Then there exists an agent i that can deviate from strategy x ∗ i to ˜ x
i and decrease her expected cost.
Hence we have that
[Equation]
We conclude that E s ∼ x ∗ [ u A ( s , d )]
[Equation]
=
E s ∼ (˜ x ∗ i , x ∗− i )
[ u A ( s , d )] which is a contradiction since ( x ∗ 1 , ..., x ∗ n ) is a Nash equilibrium for the team zero-sum game hence if player i deviates, her payoff (i.e., the payoff of her Team) should not increase.
C. M ULTIPLAYER M ATCHING P ENNIES
Below, we present the exact definition of the (2x2) two-team of two-players matching pennies that we discuss in Section 3.
Team B
HH
HT/TH
TT
Team A
HH − 1 ,− 1 / 2 , 1 /1 , −
HT/TH 1 / 2 , − 1 /0 ,1 / 2 , − 1 /
TT 1 , −− 1 / 2 , 1 /− 1 ,
C. P ROOF OF T HEOREM 3.
Since ( x ∗ , y ∗ ) is not weakly-stable, there exist players i, j from the same team (say B without loss of generality) and strategies k, l, l ′ so that if i is forced to play k , then J ’s best response is l and that gives larger payoff than another strategy l ′ in her support.
Formally it holds that (by multi-linearity of U )
[Equation]
and also ∂U ( x ∗ , y
[Equation]
x
[Equation]
We shall show that ∇ 2 yy U ( x ∗ , y ∗ ) has a strictly positive eigenvalue tangent in the product of simplices (note that if we were working with A, we would show that ∇ 2 xx U ( x ∗ , y ∗ ) has a strictly negative eigenvalue).
Consider a vector of size (cid:80) i | S i | , (where | S i | is the cardinality of the strategy space of agent i in Team B ) which has 1 at coordinates ( i, k ) , ( j, l ) , − 1 at coordinate ( j, l ′ ) and from which we subtract y ∗ i ; we denote by v the resulting vector.
We shall show that v ⊤ ∇ 2 xx U ( x ∗ , y ∗ ) v < 0 .
By multilinearity of U it follows that
[Equation]
) for all agents J and strategies s, s ′ and the same is true for x variables (team A).
We conclude that
[Equation]
Therefore ∇ 2 yy U ( x ∗ , y ∗ ) has a positive eigenvalue and as a result
[Equation]
must have a positive and a negative eigenvalue (since the trace is zero).
We consider the Jacobian of the GDA dynamics at ( x ∗ , y ∗ ) .
The corresponding matrix is the following:
[Equation]
We will show that J GDA has an eigenvalue (possible complex) with an absolute value greater than one.
It suffices to show that J GDA − I has an eigenvalue with a positive real part (because then J GDA would have an eigenvalue with a real part greater than 1 and hence magnitude greater than one).
Due to (**), we get that J GDA −
I has trace zero.
To reach contradiction suppose that no eigenvalue of
J GDA − I has a positive real part, then all eigenvalues of J GDA − I should be imaginary or zero.
But an imaginary eigenvalue (that is not zero) also results in an eigenvalue with a magnitude greater than one for J GDA , therefore all eigenvalues of J GDA − I should be zero.
We use Ky Fan inequalities which states that the sequence (in decreasing order) of the eigenvalues of 12 ( H + H ⊤ ) majorizes the real part of the sequence of the eigenvalues of H (see (Moslehian, 2011, p. 4)) for any matrix H .
We choose H = 1 η · ( J GDA − I ) .
Since R has both a negative and a positive eigenvalue, we get that
H has an eigenvalue with a negative real part.
The claim follows since H has a trace equal to zero, thus it must have an eigenvalue with a positive real part as well.
We conclude that J GDA has an eigenvalue with an absolute value greater than one.
Using (Daskalakis & Panageas, 2018, Theorem 2.2), it occurs that the set of initial conditions so that
GDA converge to ( x ∗ , y ∗ ) is of measure zero (for the particular choice of the step size).
C. P ROOF OF L EMMA 3.
First, we start with the min-max optimization objective of the GMP game with ω = 12 : min
[Equation]
or after simplification, it is equivalent with
[Equation]
Remark 4.
We highlight that the min-max optimization objective is multilinear and the degree of each variable in every summand is at most one (total degree is 2 ).
Moreover, we note that due to the nonconvexity-nonconcavity of the function above, the max-min is not equal to the min-max.
Let ( x ∗ 1 , x ∗ 2 , y ∗ 1 , y ∗ 2 ) be a Nash equilibrium.
Assuming x ∗ 1 , x ∗ 2 , y ∗ 1 , y ∗ 2 ∈ (0 , 1) from (15) and firstorder conditions we get the system of equations
1. ω + 1 − y ∗ 1 − y ∗ 2 − 2 ωx ∗ 2 = 0 , 2. ω
+ 1 − y ∗ 1 − y ∗ 2 − 2 ωx ∗ 1 = 0 , 3.
1 − ω − x ∗ 1 − x ∗ 2 + 2 ωy
[Equation]
Combining the first two equations, we have x ∗ 1 = x ∗ 2 and combining the last two it follows y ∗
= y ∗ 2 .
Dividing equation one by ω and subtracting three, it follows that 1 ω − 2 y ∗ 1 ω + ω − 2 ωy ∗ 1 =
Hence we conclude that y
∗ 1 = 12 .
As a result y ∗ 2 = 12 and substituting in first equation x ∗ 1 = x ∗ 2 = 12 .
• Assume now that x ∗ 1 = 0 and x ∗ 2 , y ∗ 1 , y ∗ 2 ∈ (0 , 1) .
Following the same idea, now only equations 2, 3, 4 hold and instead of the first we have the constraint ω +1 − y ∗ 1 − y ∗ 2 − 2 ωx ∗ 2 ≥ 0 .
From 3, 4 we conclude that y ∗ 1 =
y ∗ 2 and using 2 it holds that y ∗ 1 =
y ∗ 2 = 1+ ω 2 .
Using 3 follows that 1 − ω − x ∗ 2 + ω ( ω + 1) = 0 .
Thus x ∗ 2 = 1 + ω 2 > 1 (this is not possible because x ∗ 2 ∈ [0 , 1] ).
• Consider the case
that x ∗ 1 = 1 and x ∗ 2 , y ∗ 1 , y ∗ 2 ∈ (0 , 1) .
Only equations 2, 3, 4 hold and instead of the first we have the constraint ω
+ 1 − y ∗ 1 − y ∗ 2 − 2 ωx ∗ 2 ≤ 0 .
From 3, 4 we conclude that y ∗ 1 =
y ∗ 2 and using 2 it holds that y ∗ 1 =
y ∗ 2 = 1 − ω 2 .
Using 3 follows that − ω − x ∗ 2 + ω (1 − ω ) = 0 .
Thus x ∗ 2 = − ω 2 <
(this is not possible because x ∗ 2 ∈ [0 , 1] ).
By symmetry the same happens when x ∗ 2 = 0 or x ∗ 2 = 1 and x ∗ 1 , y ∗ 1 , y ∗ 2 ∈ (0 , 1) .
• Case x ∗ 1 = x ∗ 2 = 0 and y ∗ 1 , y ∗ 2 ∈ (0 , 1) .
Using 3, 4 we get y ∗
= y ∗ 2 = ω − 1 2 ω <
(this is not possible).
• Case x ∗ 1 = x ∗ 2 = 1 and y ∗ 1 , y ∗ 2 ∈ (0 , 1) .
Using 3, 4 we get y ∗
= y ∗ 2 = ω
+12 ω > 1 (this is not possible).
• Case x ∗ 1 = 0 and x ∗ 2 =
and y ∗ 1 , y ∗ 2 ∈ (0 , 1) .
Using 3, 4 we get y ∗
= y ∗ 2 = 12 .
Moreover one becomes ω − 2 ωx ∗ 2 ≥ 0 and two ω − 2 ωx ∗ 1 ≤ 0 , that is x ∗ 1 ≥ 12 and x ∗ 2 ≤(contradiction).
The case x ∗ 1 = 1 and x ∗ 2 = 0 and y ∗ 1 , y ∗ 2 ∈ (0 , 1) is symmetric.
Similarly one can consider the case where the y team plays pure and x ∗ 1 , x ∗ 2 ∈ (0 , 1) .
One can also check that all possible pure strategy profiles are not Nash equilibria.
C. P ROOF OF T HEOREM 3.
We split the proof into 3 parts.
Before we start the proof, note that the Hessian of U (19) has infinity norm less than 4 (since ω ∈ (0 , 1) ), so U has gradient Lipschitz with L ≤ 4 .
Thus for the rest of the proof for GDA, we choose η OGDA < 14 .
GDA.
For GDA the proof will be straightforward.
We will show that ( x ∗ 1 , x ∗ 2 , y ∗ 1 , y ∗ 2 ) is a weakly Nash equilibrium.
Then the claim about GDA will follow because of Lemma 3.1,
Theorem 3.2 and Remark 2.1.
Assume that player x 1 fixes his strategy to x 1 = 0 .
and y 1 , y 2 keep their strategy ( 12 , 12 ) .
We shall show that x 2 is not indifferent in his support and would like to change his mixed strategy x 2 = 12 to pure.
When x 1 = 0 and y 1 =
y 2 = 12 the payoff of Team A ( x variables) becomes − ωx 2 − 1 + ω 2 .
Since ω ∈ (0 , 1) , x 2 prefers to play x 2 = 0 (instead of 12 she had).
We conclude that ( 12 , 12 , 12 , 12 ) is not a weakly-stable Nash equilibrium.
OGDA.
The Jacobian of the update rule of OGDA dynamics (use the same machinery of
Section 3 in (Daskalakis & Panageas, 2018)) is the following:
[Equation]
where U is the payoff of Team B (max) and − U is the payoff of team A .
Substituting for MPG payoff at Nash equilibrium, we have
[Equation]
The corresponding Jacobian matrix becomes:
[Equation] [Equation]
[Equation]
[Equation]
[Equation] [Equation]
[Equation]
[Equation]
  ,
[Equation]
It turns out that matrix (17) has a characteristic polynomial (computed with the help of Mathematica)
[Equation]
[Equation]
A root of π ( λ ) is 12 (cid:16)
[Equation]
OGDA
[Equation]
, which is in absolute value greater than one for [-
-]0 < η OGDA ≤ ω .
It is also easy to see the Hessian of U , that is:
[Equation]
(19) is invertible for ω ∈ (0 , 1) (this is true as the eigenvalues are − 2 ω, 2 ω, − 2 √ 1 + ω 2 , 2 √ 1 + ω 2 , non of which is zero).
From Theorem 3.2 in (Daskalakis & Panageas, 2018) follows that the initial conditions so that OGDA converges to the Nash equilibrium is of measure zero for step size η OGDA < 12 L ≤ 18 (where L is the Lipschitz constant of ∇ U ).
We choose η OGDA ≤ min( 18 , ω ) and the claim follows.
EG.
The Jacobian of EG dynamics computed at the Nash equilibrium (fixed point) is given below (made use of chain rule):
[Equation]
We substitute with the values and we get
[Equation]
The eigenvalues of J EG are 1 + η EG ( − 2 ω
+ 4 η EG ω 2 ) , 1 + η EG ( − 2 ω + 4 η EG ω 2 ) , 1 + η EG (cid:16) − 4 η EG + 2 ω + 4 η EG ω 2 − 2 (cid:112) − 1 − 8 η EG ω − 16 η 2 EG ω 2 (cid:17) , and 1 + η EG (cid:16) − 4 η EG + 2 ω + 4 η EG ω 2 + 2 (cid:112) − 1 − 8 η EG ω − 16 η 2 EG ω 2 (cid:17) .
If − 4 η EG + 2 ω + 4 η EG ω 2 >then J EG will have an eigenvalue with absolute value greater than one.
A sufficient condition for that is η EG ≤ ω 2 .
Finally, for the same choice of stepsizes we have that J EG is invertible for all ( x , y ) , hence the EG dynamics is a local diffeomorphism.
From Theorem 2 in arxiv version of (Lee et al., 2019)
the claim for EG method follows.
OMWU.
The Jacobian of the update rule of OMWU dynamics (use the same machinery of Section 3 in (Daskalakis & Panageas, 2019)) is the following:
[Equation]
  .
[Equation]
It turns out that matrix (22) has a characteristic polynomial (retrieved with the help of Mathematica)
[Equation]
One root of the polynomial π ( λ ) above is 12 (1 + 2 ηj + ηω + (cid:112) 1 − 4 η 2 + 4 η 2 ωj + η 2 ω 2 ) .
As in the analysis of OGDA, it turns out that for 0 < η < ω the aforementioned root has an absolute value greater than one.
Therefore we conclude that the Nash equilibrium where each agent plays ( 12 , 12 ) is repelling (and thus the fact that the set of initial conditions so that OMWU converges to that particular fixed point is of measure zero is derived from standard arguments from (Lee et al., 2019).
To conclude the proof we need to exclude that OMWU will stabilize in other fixed points.
Note that OMWU can stabilize to points that are not Nash equilibria (e.g., all pure strategy profiles are fixed points).
To exclude such stabilization, the trick is that if OMWU stabilizes to a point, it should be (approximate coarse correlated equilibrium where the approximation depends on the size of the stepsize).
However, none of the pure strategy profiles is a ω/ 2 - approximate coarse correlated equilibrium, so OMWU does not stabilize.
Hence if we choose 0 < η < ω/ 2 , OMWU does not stabilize.
C. P ROOF OF T HEOREM 3.
As previously, let a zero-sum game with payoff function U ( x , y ) for the maximizing team (it can either be a single player or more players in each team).
The maximizing team controls y while the minimizing team controls x .
It is easy to retrieve a black-box argument for the proof of Theorem 3.4 using the proof (Hassouneh et al., 2004).
For the sake of completeness, we reprove the theorem from the bottom up in our own vocabulary.
Hassouneh et al.
(2004) extend the proof (Bazanella et al., 2000) to the discrete-time case without using the singular-perturbation analysis that the latter use.
Let A =
[Equation]
=
[Equation]
+
[Equation]
be the Jacobian of GDA around a Nash equilibrium.
and z be the concatenation of x and y .
We can locally represent the dynamic of GDA as a linear time-invariant dynamical system:
[Equation]
Here, the feedback law, u , is trivially zero, but we note it for the sake of clarity.
By assumption, A = J GDA is invertible, hence full-rank.
Also, B has to be the identity matrix,
B =
I , as we want every player to be able to make decisions with access to their own estimation vector.
From a known theorem (the PBH controllability test) the pair of matrices ( A , B ) , with A ∈ R n × n is controllable if and only if, for every eigenvalue λ of A s , the rank of the block matrix [ A − λ I , B ] is equal to n .
As we know, stabilizability implies controllability.
Hence, the pair of matrices ( A , B ) =
( J GDA , I ) is stabilizable.
We can now write down the dynamic of (KPV-GDA) localized around a given (possibly Lyapunov unstable)
Nash equilibrium as the following linear time-invariant system:
[Equation]
This dynamic can be represented in a block matrix fashion as:
[Equation]
Knowing that eigenvalues are preserved under similarity transformations, we define two similarity transformations T 1 , T 2 which we will apply consecutively on M .
Let Q be any matrix.
We define:
[Equation]
and:
[Equation]
By consecutively applying the latter transformations we get:
[Equation]
We now want to eliminate the top-left entry of M ′ in order to have an easy application of the Schur complement equations.
We are free to do so if we can find an appropriate matrix Q .
Remember that
Q can be any matrix.
First, we write matrices P and Q as (non-singular) perturbations of some appropriate matrices for ϵ > 0 sufficiently small:
[Equation]
[Equation]
Setting the top right
[Equation]
Gathering terms w.r.t.
to the power of ϵ we get, for O (1) terms:
[Equation]
And for O ( ϵ ) terms we get:
[Equation]
From equation 35 we get that
[Equation]
From equation 36, by observing that we can always select a K such that 1 is not an eigenvalue of
J GDA + K we get that:
[Equation]
We observe that Q 1 is uniquely defined through a matrix inversion.
Hence, from the Implicit Function Theorem applied to equation 33 we get that there exists a locally unique matrix Q : [Equation]
[Equation]
We return to matrix M ′ and substitute Q with what we calculated in order to get:
[Equation]
I Λ (cid:19) (40)
With Λ =
J GDA + K + ϵ (cid:16) ( J GDA − I ) P 1 ( J GDA + K − I ) − 1 − P 1 (cid:17) + O ( ϵ 2 ) .
We have assumed η
[Equation]
=
[Equation]
− I to be invertible, hence in order for M ′ to be stable it suffices that:
• I − ϵ ( J GDA − I ) P 1 ( J GDA + K − I ) − 1 + O ( ϵ 2 ) is stable • J GDA + K is stable with eigenvalues away from 1 , so that term ϵ (cid:16) ( J GDA − I )
P 1 ( J GDA + K − I ) − 1 − P 1 (cid:17) does not cause the eigenvalues of J GDA + K to get a a magnitude greater than 1 (not to become unstable).
For the first item it easy to see that an appropriate P 1 always exists, that is, P 1 = ( J GDA −
I ) − 1 ( J GDA + K − I ) .
As for the second item, it follows from the fact that ( J GDA , I ) is stabilizable as we have demonstrated.
C. P ROOF OF T HEOREM 3.
We first compute the Jacobian of KPV-GDA dynamics (KPV-GDA) where we have eliminated one variable per agent (to avoid using the projection operator).
The Jacobian has the following form:
[Equation]
It suffices to show that the spectral radius of J KPV-GDA is less than one for step size η small enough.
We first show the claim below
Claim C.1.
Let J
= I + η M where I is the identity matrix.
Suppose that M has all of its eigenvalues with a real part that is negative.
Then, there exists an interval (0 , η 0 ) so that if η ∈ (0 , η 0 ) then J has all of its eigenvalues with an absolute value less than one.
Proof.
Let a i
+ b i · j be an eigenvalue of M such that a i < 0 .
Assume that η < − 2 a i a 2 i + b 2 i , then the corresponding eigenvalue of J is 1 + η ( a i + b i · j ) , the magnitude of which is (1 + ηa i ) 2 + η 2 b 2 i = 1 +
ηa
i + η 2 ( a 2 i + b 2 i ) < 1 since − 2 a i > η ( a 2 i + b 2 i ) (by assumption).
Hence we can choose η 0 to be min i − 2 a i a 2 i + b 2 i > 0 .
Using Claim C.1, we conclude that as long as the matrix M below has eigenvalues with real part negative then ( x ∗ , y ∗ ) is attracting:
[Equation]
−
[Equation]
By setting M 11 , M 12 , M 21 , M 22 as the corresponding block matrices we compute the characteristic polynomial of M .
It holds
[Equation]
det( M − λ I ) has roots at λ = − p and when
[Equation]
Therefore, if M has all eigenvalues with real part negative, we must have p > 0 .
Let ρ be an eigenvalue of
[Equation]
It must hold that
[Equation]
We need to investigate under what assumptions λ will have a real part negative.
We expand (46) and we get
[Equation]
We solve Equation (47) and we get
[Equation]
We provide sufficient conditions so that Re ( λ 1 , 2 ) < 0 .
We consider the following cases:
• ρ is real and negative.
In this case observe that for p =
we have that λ 1 ,
=
k
+ ρ and 0 .
Note that for k < 0 , the first eigenvalue has real part negative and the second eigenvalue is zero.
We compute the first derivative at p =
and this gives 12 (cid:16) − 1 + − k
+ ρ | k + ρ |
(cid:17)
=
[Equation]
− k
−
ρ < 0 .
Therefore the λ 1 , 2 as a function of p is strictly decreasing at zero.
We conclude that for p sufficiently small and positive, both eigenvalues of M will be real and negative.
Hence if an eigenvalue of H is real and negative then for p sufficiently small positive and k <both eigenvalues of M will be negative.
• ρ is complex with | Im ( ρ ) | ̸ = 0 .
In this case observe that for p =
we have that λ 1 ,
=
k
+ ρ and 0 .
If we choose min(0 , − Re ( ρ ))
> k , then the first eigenvalue will have real part negative.
Now using the same idea as for the first case, we compute the first derivative of the real part of λ 1 , 2 as a function of p and we get
[Equation]
The expression above is negative as long as − Re ( ρ ) 2 − Im ( ρ ) 2 − k
Re ( ρ ) < 0 .
The equation is trivially satisfied when Re ( ρ ) < 0 since k is chosen to be negative.
Assume Re ( ρ ) >(if it is zero then the above is trivially true since the imaginary part is non-zero).
It occurs that the inequality above is true when k > − | ρ | 2 Re ( ρ ) .
Since k is chosen to be smaller than min(0 , − Re ( ρ )) , it suffices to show that
[Equation]
by our assumptions.
The above is true as long as Im ( ρ ) ̸ = 0 .
Let E be the set of eigenvalues ρ of H with real part positive (and non-zero imaginary part by assumption).
There exists a choice for η, k, p , in which η, p > 0 and sufficiently small and k is negative and chosen to be
[Equation]
D E XPERIMENTS
We elaborate on the experiment mentioned in the main body.
Our experiment includes a dataset of a mixture of 2-D Gaussians with 8 modes.
Our architecture includes 8 “shallow” generators and discriminators with 2 layers of 2-16-2 ReLU activations, compared with a “large” single-agent GAN with 4 layers of 2-128-256-1024-2 activations.
Interestingly, the giant one fails in a double sense;
It demonstrates both mode-collapsing and mode-drop phenomena without stabilizing.
On the other hand, our architecture with a small number of neurons achieves to fit the data well.
The iterations illustrate the step at which the model roughly stops learning/converges.
We observed no change after the demonstrated number of iterations.
[Figure]
D. MGAN VS . WGAN
CIFAR-10 is a well-established testbed for various GAN architectures.
It contains 10 balanced classes of images of different objects.
Just to further illustrate the merits of multi-agent GAN’s, we offer a selection of images generated by WGAN-GP (Arjovsky et al., 2017) and the MGAN (Hoang et al., 2017).
In the pictures provided by the WGAN-GP we see that across iterations, the generated samples tend to cover only certain classes of the dataset while it takes longer for the samples to become realistic.
On the other hand, the MGAN architecture from an early stage provides with diverse samples that tend to be more realistic from early on.
E E MPIRICAL PERFORMANCE OF THE PROPOSED METHOD
In this section, we perform a number of empirical experiments to assess the performance of our proposed method.
As we highlighted, the conditions of Theorem 3.5 are only sufficient and not necessary.
In the following figure, we plot the curves of the Nash equilibrium gap for 500 random two-team zero-sum games.
In every such game, every player has 2 available strategies.
We note that for all of these games we use a random initialization for the players’ strategies and the same parameters η = 0 .
05 , k
= − 1 . 2 , p = 0 .
02 for our proposed method.
It is notable that the Nash equilibrium gap vanishes for virtually all of these instances.
This indicates that the conditions of
Theorem 3.5 are only sufficient and a more complex analysis could show less restrictive conditions for convergence with the same simple parametrization.
For completeness, we include the definition of the Nash equilibrium gap:
Definition E.1 (NE-gap) .
We define the Nash equilibrium gap as the sum of the differences between the best-response and the strategy every player plays:
[Equation]
[Equation]
= = − Additionally, we apply the same method with fixed parameters η = 0 .
05 , k
= − 1 . 05 , p = 0 .to
two-team zero-sum games with 3 strategies per player and 2 players in each team.
Again, most random games seem to converge to a 0 NE-gap.
Lastly, we use the same method with fixed parameters η = 0 .
1 , k = − 1 .
15 , p = 0 .
01 forrandom two-player zero-sum games where each player has 5 .
The method converges for virtually all games despite of not tuning the parameters for every game.