C HAOS OF L EARNING B EYOND Z ERO - SUM AND C OORDINATION VIA G AME D ECOMPOSITIONS
A BSTRACT
It is of primary interest for Machine Learning to understand how agents learn and interact dynamically in competitive environments and games (e.g. GANs).
But this has been a difﬁcult task, as irregular behaviors are commonly observed in such systems.
This can be explained theoretically, for instance, by the works of Cheung & Piliouras (2019; 2020), which showed that in two-person zero-sum games, if agents employ one of the most well-known learning algorithms, Multiplicative Weights Update (MWU), then Lyapunov chaos occurs everywhere in the cumulative payoff space.
In this paper, we study how persistent chaos can occur in the general normal-form game settings, where the agents might have the motivation to coordinate (which is not true for zero-sum games) and the number of agents can be arbitrary.
We characterize bimatrix games where MWU, its optimistic variant (OMWU) or
Follow-the-Regularized-Leader (FTRL) algorithms are Lyapunov chaotic almost everywhere in the cumulative payoff space.
Our characterizations are derived by extending the volume-expansion argument of Cheung & Piliouras via the canonical game decomposition into zero-sum and coordination components.
Interestingly, the two components induce opposite volume-changing behaviors, so the overall behavior can be analyzed by comparing the strengths of the components against each other.
The comparison is done via our new notion of “matrix domination” or via a linear program.
For multi-player games, we present a local equivalence of volume change between general games and graphical games, which is used for volume and chaos analyses of MWU and OMWU in potential games.
I NTRODUCTION
In Machine Learning (ML), it is of primary interest to understand how agents learn in competitive environments.
This is more strongly propelled recently due to the success of Generative Adversarial Networks (GANs), which can be viewed as two neural-networks playing a zero-sum game.
As such,
Evolutionary Game Theory (EGT) (Hofbauer & Sigmund (1998); Sandholm (2010)), a decades-old area devoted to the study of adaptive (learning) behaviors of agents in competitive environments arising from Economics, Biology, and Physics, has drawn attention from the ML community.
In contrast with the typical optimization (or no-regret) approach in ML, EGT provides a dynamicalsystemic perspective to understand ML processes, which has already provided new insights into a number of ML-related problems.
This perspective is particularly helpful in studying “learning in games”, where irregular behaviors are commonly observed, but the ML community currently lacks of a rigorous method to analyze such systems.
In this paper, we study Lyapunov chaos , a central notion that captures instability and unpredictability in dynamical systems.
We characterize general normal games where popular learning algorithms exhibit chaotic behaviors.
Lyapunov chaos captures the butterﬂy effect : when the starting point of a dynamical system is slightly perturbed, the resulting trajectories and ﬁnal outcomes diverge quickly; see Deﬁnition 1 for a formal deﬁnition.
The perturbations correspond to round-off errors of numerical algorithms in ML (and Computer Science in general).
1 While signiﬁcant efforts have been spent in analyzing and minimizing round-off effects of ﬂoating-point computations (Demmel (1997)), they are unavoidable in general 2 .
As round-offs are inevitable, and the round-off schemes can vary from machine to machine due to various hardware and software factors, we surely want to avoid chaotic learning that does not fulﬁll our primary goals in building predictable and reproducible learning systems.
Such issues are exempliﬁed by a quote from Ali Rahimi’s NIPS’2017 test-of-time award speech: “Someone on another team changed the default rounding mode of some Tensorﬂow internals from ‘truncate toward zero’ to ‘round to even’.
Our training broke, our error rate went from less than 25% error to ∼ 99 . 97% error.”
To avoid chaotic learning, we ﬁrst need to understand how it can arise.
This is the main motivation of our work.
Recently, in the context of “learning in games”, Cheung & Piliouras (2019; 2020) presented interesting theoretical analyses to show that in two-person zero-sum and graphical constant-sum games, if the agents employ Multiplicative Weights Update (MWU) or Follow-theRegularized-Leader (FTRL) algorithms, then Lyapunov chaos occurs everywhere in the cumulative payoff space; the same result holds for the optimistic variant of MWU (OMWU) in coordination games.
While zero-sum and coordination games are interesting in their own rights, they are rather small subspaces within the family of general normal games.
In this paper, we present techniques and tools for characterizing general games where MWU, FTRL, or OMWU are Lyapunov chaotic almost everywhere.
Next, we give an overview of our contributions and a discussion of related work.
Our Contributions.
To show the results about chaos mentioned above, Cheung & Piliouras (2019; 2020) used a classical technique in the study of dynamical systems called volume analysis .
Volume analysis considers a set of starting points of positive Lebesgue measure, e.g. a ball centred at a point; volume is an alternative name for Lebesgue measure in this context.
When this set of starting points evolves according to the rule of dynamical system, it becomes a new set with a different volume.
Intuitively, volume is a measure of the range of possible outcomes, so the larger it is, the more unpredictable the system is.
If the set’s volume increases exponentially with time, then its diameter increases exponentially too, which implies Lyapunov chaos.
This indicates that when players repeatedly play the games by employing the respective learning algorithms, a slight perturbation on the initiating condition can lead to a wide range of possible cumulative payoffs in the long run.
This can be shown to imply instability in the mixed strategy space.
The technical starting point of Cheung & Piliouras is to show that when all agents in a bimatrix game G use MWU with step-size (cid:15) , and when a set S in the cumulative payoff space evolves for one time step, its volume change can be expressed as (cid:15) 2 (cid:82) S C G ( s ) ds + O ( (cid:15) 4 ) , where C G is a function that depends on the game G , which we will deﬁne in Section 2.
Clearly, the sign of C G dictates the volume change behaviors for all small enough (cid:15) .
Cheung & Piliouras showed that C G is a positive function when G is a two-person zero-sum game.
For a large region in the cumulative payoff space, this implies the volume change per time step is Ω( (cid:15) 2 ) · volume ( S ) , i.e. volume expands exponentially.
They also showed that C G is a negative function if G is a two-person coordination game.
To extend their volume expansion results to general bimatrix games, we ﬁrst discover that C G admits a clean decoupling w.r.t.
the canonical decomposition of such games into zero-sum and coordination components (Basar & Ho (1974); Kalai & Kalai).
Precisely, given any two-person general game ( A , B ) , it can be written uniquely as a direct sum of a zero-sum game ( Z , − Z ) and a coordination game ( C , C ) , where Z =
( A − B ) / 2 and C =
( A + B ) / 2 .
Interestingly, we ﬁnd that
C G ( · ) =
C ( Z , − Z ) ( · ) + C ( C , C ) ( · ) in Lemma 6.
Recall from the last paragraph that C
( Z , − Z )
( · ) is always positive, while C ( C , C ) ( · ) is always negative.
Thus, to see whether volume expansion occurs, it boils down to comparing the strengths of C ( Z , − Z ) ( · ) and − C ( C , C ) ( · ) .
We also discover that the function C G is invariant upon additions of trivial matrices to the game
G ; see Deﬁnition 7 and Lemma 8.
An immediate application of trivial matrices is for bimatrix potential games (Monderer & Shapley (1996)), for which we show it can be transformed to a coordination game via additions of trivial matrices.
With the result in Cheung & Piliouras (2020), this immediately implies that OMWU in any bimatrix potential game is Lyapunov chaotic everywhere in the cumulative payoff space (Observation 10).
Based on the above discoveries, we identify two characterizations of bimatrix games where
MWU and FTRL are Lyapunov chaotic almost everywhere (Theorems 15 and 17).
As said before, the key is to compare the strengths of the C -functions of the zero-sum and the coordination components.
The comparison is done via our new notion of matrix domination (see Deﬁnition 11), and also a linear program (Eqn. (7)) which is designed to prune out the trivial-matrix projection and keep the remaining part minimal.
Such family of games has positive Lebesgue measure in the bimatrix game space, so it is not conﬁned to any proper game subspace 4 .
This justiﬁes the claim that the occurrences of chaos are not only circumstantial, but a rather substantial issue in learning in games.
Analogous result holds for OMWU.
For games with any number of players (multi-player games), we use an observation in Cheung &
Piliouras (2019), coupled with our new ﬁndings about bimatrix games discussed above, to present a new family of graphical games where MWU is Lyapunov chaotic almost everywhere (Theorem 18); the new family strictly includes all graphical constant-sum games.
To facilitate volume analyses of learning in multi-player games, we establish their local equivalence of volume change with graphical games.
Brieﬂy, we show that C G ( p ) for a general game G is the same as C H ( p ) for some graphical game H ; H will depend on the point p , that’s why we say the equivalence is local (Theorem 19).
This provides an intuitive procedure for understanding volume changes, as the volume change of learning in a graphical game is easier to compute.
This is used to show that the volume-changing behaviors of MWU and OMWU are opposite to each other.
We use these to analyze MWU and
OMWU in multi-player potential games; in particular, we show that C G ( p ) of a multi-player potential game G is identical to C C ( p ) of a corresponding multi-player graphical coordination game, while C C ( p ) ≤ 0 for any p (Proposition 21).
Related Work.
MWU and its variant, such as FTRL and Optimistic MWU, play important roles in online learning.
We recommend the texts of Cesa-Bianchi & Lugosi (2006) and Hart & Mas-Colell (2013) for a modern overview of online learning from Machine Learning or Economics perspectives.
Recently, there is a stream of works that examine how learning algorithms behave in games or min-max optimization from a dynamical-systemic perspective.
This provides new insights into learning systems which could hardly be obtained with classical tools in ML.
For instance, some learning systems are shown to be nearly-periodic under the notion of Poincar´e recurrence (Piliouras & Shamma (2014); Mertikopoulos et al.
Using potential functions ﬁrst proposed in EGT and other tools from mathematics, surprising behaviors of ﬁrst-order methods in zero-sum games and min-max optimization were discovered (Daskalakis & Panageas (2018; 2019); Bailey & Piliouras (2018); Cheung (2018)).
In Appendix A, we give an account on some further related work.
Empirical evidences of Lyapunov chaos of learning in games were reported by Sato et al.
and Galla & Farmer (2013).
Li-Yorke chaos , another classical chaos notion, was proved to occur in several learning-in-game systems (Palaiopanos et al.
(2017); Chotibut et al.
Volume analysis has long been a technique of interest in the study of population and game dynamics.
It was discussed in a number of famous texts; see (Hofbauer & Sigmund, 1998, Section 11), (Fudenberg & Levine, 1998, Section 3) and (Sandholm, 2010, Chapter 9).
We use game decomposition in this paper, which is a natural and generic approach to extend compelling results for a speciﬁc family of games to more general games.
Let H denote the speciﬁc family of games with some compelling properties.
Given a general game, we seek to decompose it into a sum of its projection on H , plus one or more residue components.
If the residues are small, then it is plausible that those compelling properties extend (approximately).
In seeking of games where learning is stable, game decompositions were used (Candogan et al.
(2011; 2013a;b);
Letcher et al.
with H being potential games.
P RELIMINARY
In this paper, every bold lower-case alphabet denotes a vector, every bold upper-case alphabet denotes a matrix or a game.
When we say a “game”, we always mean a normal-form game.
Given n , let ∆ n denote the mixed strategy space of dimension n − 1 6 , i.e. { ( z 1 , z 2 , · · · , z n ) | (cid:80) {+nj =
+} z j = 1 } .
Normal-Form Games.
Let N denote the number of players in a game.
Let S i denote the strategy set of Player i , and S :
= S 1 × · · · × S N .
Let n i = | S i | .
s =
( s 1 , · · · , s N ) ∈ S denotes a strategy proﬁle of all players, and u i ( s ) denotes the payoff to Player i when each player picks s i .
A mixed strategy proﬁle is denoted by x = ( x 1 , · · · , x N ) ∈ ∆ n 1 × · · · × ∆ n N , and u i is extended to take mixed strategies as inputs via u i ( x ) =
E s ∼ x
[ u i ( s )] .
We let − ( i 1 , · · · , i g ) denote the player set other than players i 1 , · · · , i g .
We also let
[Equation]
which is the expected payoff to Player i 1 when: for 1 ≤ f ≤ g , Player i f picks strategy
j f , while for each player i / ∈ { i 1 , · · · i g } , she picks a strategy randomly following x i .
We also use U i 1 i 2 ··· i g j 1 j 2 ··· j g if x is clear from the context.
We say a game is a zero-sum game if (cid:80) i u i ( s )
=
for all s ∈ S , and we say a game is a coordination game if u i ( s ) =
u k ( s ) for all Players i and k and for all s ∈ S .
When N = 2 , such games are called bimatrix games , for which we adopt the notations below.
Let ( A , B ) denote a bimatrix game, where for any j ∈ S 1 , k ∈ S 2 , A jk :=
u 1 ( j, k ) , B jk := u 2 ( j, k ) .
x and y denote mixed strategies of Players 1 and 2 respectively.
A bimatrix game is a zero-sum game if A = − B ; it is a coordination game if A = B .
Note that U 1 j =
[ Ay ] j ,
U 2 k =
[ B T x ] k , which we denote by A j , B k respectively when x , y are clear from context; B j , A k are deﬁned analogously.
MWU, FTRL and OMWU in Games.
All three algorithms have a step-size (cid:15) , and can be implemented as updating in the cumulative payoff (dual) space.
In each round, the players’ actions (mixed strategies) in the strategy space are functions of the cumulative payoff vectors to be deﬁned below, and these actions are then used to determine the payoffs in the next round.
For a player with d strategies, let p t ∈ R
d denote her cumulative payoff vector at time t , and let p 0 ∈ R
d denote the starting point chosen by the player.
For MWU in a game, the update rule for Player i is
[Equation]
where U ij is the function deﬁned in Eqn.
(1), and x t is the mixed strategy as below:
[Equation]
For OMWU in a game, the update rule for Player i starts with p 1 = p 0 , and for t ≥ 2 , p t +1 j = p tj + (cid:15) · (cid:2) 2 U ij ( x t ) − U ij ( x t − 1 ) (cid:3) , where x t is determined by Eqn.
For FTRL in a game, the update rule for Player i is same as Eqn.
(2), but x t is determined as below using a convex regularizer function h
i : ∆ d → R : x t = arg max x ∈ ∆
d {(cid:104) p t , x (cid:105) − h
i ( x ) } .
As all the results for MWU can be directly generalized to FTRL as discussed in (Cheung & Piliouras, 2019, Appendix D), to keep our exposition simple, in the rest of this paper, we focus on MWU and
OMWU, and their comparisons.
For bimatrix game, we use p , q to denote the cumulative payoff vectors of Players 1 and 2 respectively.
Dynamical Systems, Lyapunov Chaos and Volume Analysis.
A learning-in-game system can be viewed as a discrete-time dynamical system, for which we present a simpliﬁed deﬁnition which suits our need.
A discrete-time dynamical system in R d is determined by a starting point r
(0) ∈ R d and an update rule r ( t +1) =
f ( r ( t )) , where f : R d → R d is a function.
7 The sequence r (0) , r (1) , r (2) , · · · is called a trajectory of the dynamical system.
When f is clear from the context, we let Φ : ( N ∪ { 0 } ) × R d
→ R d denote the function such that Φ( t, r ) is the value of r ( t ) generated by the dynamical system with starting point set to r .
Given a set U ⊂ R d , we let Φ( t, U ) =
{ Φ( t, r ) | r ∈ U} .
Let
B ( r , z ) denote the open ball with center r and radius z .
There are several similar but not identical deﬁnitions of Lyapunov chaos, all capturing the butterﬂy effect : when the starting point is slightly perturbed, the resulting trajectories diverge quickly.
We use the following deﬁnition, which was also used by Cheung & Piliouras (2019; 2020) implicitly.
Intuitively, a system is Lyapunov chaotic in an open set O ⊂ R d if for any r ∈ O and any open ball
B around r , as long as Φ( t, B ) remains inside O , there exists r (cid:48)
∈ B such that (cid:107) Φ( t, r (cid:48) ) − Φ( t, r ) (cid:107) grows exponentially with t .
Lyapunov exponent in the deﬁnition below is a measure of how fast the exponential growth is; the larger it is, the more unpredictable the dynamical system is.
Deﬁnition 1.
A dynamical system is Lyapunov chaotic 8 in an open set
O ⊂ R d if there exists a constant λ > 0 and a Lyapunov exponent γ ≡ γ ( O ) > 0 , such that for any r ∈ O , for any sufﬁciently small δ > 0 and for all t satisfying 0 ≤ t < min { τ | τ ≥ 0 , Φ( τ, B ( r , δ ))
(cid:54)⊂ O} ,
[Equation]
r
[Equation]
Deﬁnition 2.
A dynamical system is Lyapunov chaotic everywhere if it is Lyapunov chaotic in any bounded open subset of R d .
In the above deﬁnitions, all norms and radii are Euclidean norms.
For capturing round-off errors in computer algorithms and ML systems, it is more natural to use (cid:96) ∞ -norm with δ be the maximum round-off error per step, say ∼ 10 − 16 when IEEE 754 binary64 (standard double) is used.
When O is a small set, it is easy to determine whether a dynamical system is Lyapunov chaotic in O , since the dynamic can be locally approximated by a linear dynamical system, where the eigenvalues of the local Jacobian characterizes chaotic behaviors.
But when O is large, determining whether Lyapunov chaos occurs is difﬁcult in general.
Cheung & Piliouras (2019) found that volume analysis can be useful in this regard, based on the following simple observation.
Proposition 3.
In R d , if a set U has volume at least v , then its radius w.r.t.
any point r ∈ U is at least v 1 /d / 2 .
Thus, if the volume of Φ( t, U ) of some dynamical system is Ω(exp( γt )) for some γ > 0 , then the radius of Φ( t, U ) w.r.t.
any point r ∈ Φ( t, U ) is Ω(exp( γd [Equation]
Cheung and Piliouras showed Lemma 4 below, which, for bimatrix games, reduces volume analysis to analyzing the sign of the function C ( A , B ) ( p , q ) deﬁned in Eqn.
(4) below; the sign also determines the local volume-changing behavior around the point ( p , q ) when MWU is used.
Based on Proposition 3 that converts volume expansion to radius expansion, the sign can be used to determine if the dynamical system is Lyapunov chaotic.
Let A j = (cid:80) [Equation]
A
[Equation] y
[Equation] = ∇ x j [ x T Ay ] and A k = (cid:80) j (cid:48)
x j (cid:48)
A j (cid:48)
k = ∇ y k
[ x T Ay ] ; and, similarly,
B j = (cid:80)
[Equation]
(cid:48) B jk (cid:48)
y k (cid:48)
= ∇ x j [ x T By ] and B k = (cid:80) [Equation]
(cid:48) x j (cid:48)
B j (cid:48)
k = ∇ y k
[ x T By ] .
Then,
[Equation]
Note that here x and y are the shorthand for x ( p ) and y ( q ) , which are the mixed strategies (i.e. probability distributions over strategies) of Players 1 and 2 respectively, as computed via
Eqn.
Also, E x , y [ f ( j, k )]
=
E ( j,k ) ∼ ( x ( p ) , y ( q ))
[ f ( j, k )] is the expected value of f ( j, k ) when the strategies j and k are randomly chosen according to the distrubutions x ( p ) and y ( q ) respectively.
For multi-player game G , the analogous function C G ( · ) is given below; the U quantities were deﬁned in Eqn.
Lemma 4 is adapted from Cheung & Piliouras (2019) for games with any number of players.
Derivation of Eqn.
(5) uses the Jacobian of the corresponding dynamical system and integration by substitution; see Appendix D.
[Equation]
Lemma 4.
Suppose O is a set in the cumulative payoff space R d where d = n 1 + · · · + n N , and
[Equation]
Then for the dynamical system in which MWU with any sufﬁciently small step-size (cid:15) is employed to play the game G , it is Lyapunov chaotic in O with Lyapunov exponent ¯ c G ( O ) · (cid:15) 2 / 2 d .
If MWU is replaced by OMWU, then the same result holds by replacing the condition (6) with ¯ c G ( O ) := inf ( p 1 , ··· , p N ) ∈O [ − C G ( p 1 , · · · , p N )] > 0 .
Note that if we start from a Nash equilibrium in the strategy space, MWU and OMWU will stay at the equilibrium.
However, if this equilibrium ( x ∗ , y ∗ ) satisﬁes the conditions in Corollary 5 below, there are points arbitrarily close to the equilibrium that keep moving away from the equilibrium (if the region { ( x (cid:48) , y (cid:48) )
=
( x ( p ) , y ( q ))
| ( p , q ) ∈ O} is large in the strategy simplex ∆ n 1 × ∆ n 2 ).
Corollary 5 (Adapted from (Cheung & Piliouras, 2020, Theorem 5)) .
Let ( x ∗ , y ∗ ) be a point in the interior of the strategy space.
Suppose that there exists ( p , q ) in the cumulative payoff space, such that x ∗ = x ( p ∗ ) and y ∗ =
y ( q ∗ ) .
Furthermore, suppose C ( A , B ) ( p ∗ , q ∗ ) > 0 , and ( p , q ) ∈
O where O is the set described in Lemma 4.
Then there are strategy points arbitrarily close to ( x ∗ , y ∗ ) such that MWU in the game ( A , B ) eventually leaves the corresponding strategy set of O , i.e. { ( x (cid:48) , y (cid:48) )
=
( x ( p ) , y ( q ))
| ( p , q ) ∈ O} .
We give the intuitions behind the proof of Corollary 5.
Suppose the contrary, i.e. for any open neighbourhood of ( p ∗ , q ∗ ) , its ﬂow never escapes from O .
Then there are two contradicting facts.
First, the volume of the ﬂow expands at least exponentially with time.
Second, by Eqn.
(2), each p tj grows at most linearly with t (since | U ij | ≤ max j,k {| A jk | , | B jk |} ), and thus the volume of the ﬂow can only expand at most polynomially with time.
When the game is zero-sum, i.e., B = − A , hence C
( A , B ) ( p , q ) =
E x , y (cid:2) ( A jk − A j − A k ) 2 (cid:3) −
E x , y
[( A jk )] 2 .
Since E x , y [ A jk ] = E x , y
[ A j ] = E x , y
[ A k ] and hence E x , y
[ A jk − A j − A k ] = − E x , y
[ A jk ] , C ( A , B ) ( p , q ) is indeed the variance of the random variable A jk − A j − A k , and thus is non-negative.
By Eqn.
(4), we have C ( A , B ) ( p , q ) =
− C
( A , − B ) ( p , q ) .
Thus, for any coordination game ( A , A ) , we have C ( A , A ) ( p , q ) =
− C
( A , − A )
( p , q ) ≤ 0 .
B IMATRIX G AMES
In this section, we focus on general bimatrix games ( A , B ) .
In Section 3.1, we present two tools for analyzing C ( A , B ) ( · ) , and then we provide an example to show how to use these tools.
In Section 3.2, we present two characterizations such that the dynamics are Lyapunov chaotic almost everywhere.
3. T OOLS FOR A NALYZING B IMATRIX G AME
First Tool: Canonical Decomposition for Bimatrix Games.
For every bimatrix game ( A , B ) , it admits a canonical decomposition (Basar & Ho (1974); Kalai & Kalai) into the sum of a zero-sum game ( Z , − Z ) and a coordination game ( C , C ) , where Z = 12 ( A − B ) and C = 12 ( A + B ) , i.e.
( A , B ) =
( Z , − Z )
+ ( C , C ) .
We call ( Z , − Z ) the zero-sum part of the game ( A , B ) , and ( C , C ) the coordination part of the game.
Our ﬁrst result shows that the function C ( · ) can be decomposed neatly into the two parts too.
Lemma 6.
For any
[Equation]
Proof.
We use Eqn.
(4) to expand the following:
[Equation]
By the end of Section 2, we discussed that C ( Z , − Z )
( p , q ) is always non-negative and
C ( C , C ) ( p , q ) is always non-positive.
By the above lemma, we can analyze the volume-changing behavior of a bimatrix game ( A , B ) by looking at its zero-sum and coordination parts independently.
One simple intuition is that if the coordination (resp.
zero-sum) part is small, then the volumechanging behavior of ( A , B ) is closer to the behavior of the zero-sum (resp. coordination) part.
We realize this intuition quantitatively in the next subsection.
Second Tool: Trivial matrix.
Trivial matrices are matrices which do not affect the volume-changing behavior, as depicted in Lemma 8 below.
Deﬁnition 7 (Trivial Matrix) .
T ∈ R n × m is a trivial matrix if there exists real numbers u 1 , u 2 , · · · , u n and v 1 , v 2 , · · · , v m such that T jk =
u j + v k for all j ∈ [ n ] , k ∈ [ m ] .
Lemma 8.
For any two trivial matrices T 1 , T 2 , for any two matrices A , B ∈ R n × m ,
C ( A , B ) ( p , q ) ≡
[Equation]
( A + T 1 , B
[Equation]
T
[Equation]
[Equation]
p ,
[Equation]
One immediate application of this lemma is for two player potential games.
Deﬁnition 9.
A game G is a potential game if there exists a potential function P : S → R such that for any Player i and any strategy proﬁle
[Equation]
For the potential game, we have the following observation:
Observation 10.
For any bimatrix potential game ( A , B ) , there is a coordination game ( P , P ) such that A − P , B − P are trivial matrices.
P is the matrix representation of the potential function P .
This observation immediately implies that the volume-changing behavior of a potential game is equivalent to that of a corresponding coordination game.
We give a concrete example to show how these tools help us to analyze the C ( A , B ) ( · ) .
A Simple Example.
We will show how to use our tools to demonstrate C ( · ) ≥ 0 everywhere for the following game.
In the example, each player has three strategies.
The payoff bimatrix ( A , B ) is given below.
The ﬁrst number gives the payoff of the row player, who chooses a strategy from { a, b, c } ; the second number gives the payoff of the column player, who chooses a strategy from { 1 , 2 , 3 } .
We ﬁrst use our ﬁrst tool to decompose this game into zero-sum part ( Z , − Z ) and
Strategy Strategy
Strategy
Strategy a (4 , 4) (12 , − 4) ( − 6 , 10)
Strategy b ( − 8 , 8) (0 , 0) (12 , − 4)
Strategy c (14 , − 2) ( − 8 , 8) (4 , 4) coordination part ( C , C ) , where Z = (cid:104) 0 8 − 8 − 8 0 8 8 0 0 (cid:105) and C =
(cid:104)
4 2 0 0 4 6 0 4 (cid:105) .
At this point, we still cannot easily ﬁgure out which one is larger between C ( Z , − Z ) ( · ) and C ( C , − C ) ( · ) .
However, we can further decompose the coordination part by the second tool : C = (cid:104) 4 2 4 2 0 2 4 2 4 (cid:105) + (cid:104) 0 2 − 2 − 2 0 2 2 − 2 0 (cid:105) , where the ﬁrst matrix on the RHS is a trivial matrix (using notations in Deﬁnition 7, u = v T =
It’s easy to see the second matrix on the RHS is 14 Z .
Then by Lemmas 6 and 8, and the deﬁnition of the function C , for any point ( p , q ) in the cumulative payoff space,
[Equation]
3. R ESULTS FOR B IMATRIX G AMES
In this subsection, we identify several characterizations for general bimatrix games in which we have chaotic behavior with MWU dynamic in a following set S δ in the cumulative payoff space R n 1 + n 2 .
Note that when δ is tiny, its strategy correspondence covers almost the entirety of the strategy simplex ∆ n 1 × ∆ n 2 , thus we informally say that if the dynamical system is Lyapunov chaotic in S δ for a tiny δ , then it is Lyapunov chaotic almost everywhere .
[Equation]
In order to show chaotic behavior of MWU in a speciﬁc bimatrix game ( A , B ) , it is sufﬁcient to show C ( A , B ) ( p , q ) is strictly positive in the region S δ , due to Lemma 4.
In the previous subsection, we show that for each game ( A , B ) , it can be decomposed into a zero-sum part ( Z , − Z ) and a coordination part ( C , C ) .
Furthermore, C ( A , B ) ( p , q ) =
C ( Z , − Z ) ( p , q ) + C ( C , C ) ( p , q ) .
We also raise an intuition that if the zero-sum part is small, then the volume behavior in the game ( A , B ) will be similar that in the coordination part; conversely, if the coordination part is small, then the volume behavior will be similar to the zero-sum part.
However, we have not yet presented a way to compare the largeness of the two parts.
This is what we do here.
3.2. F IRST C HARACTERIZATION : M ATRIX D OMINATION
The ﬁrst characterization we identify is matrix domination.
In this part, we show that under certain conditions, the zero-sum part is always no less than the coordination part,
i.e. C
( Z , − Z ) ( p , q ) ≥ − C
( C , C ) ( p , q ) for all ( p , q ) .
This directly implies C ( A , B ) ( p , q ) will be non-negative in the whole cumulative payoff space.
Interestingly, the condition we identify is both necessary and sufﬁcient.
Similar result can also be achieved in the case that coordination part is always no less than the zero-sum part.
We ﬁrst introduce the deﬁnition of the matrix domination.
Deﬁnition 11.
We say matrix K dominates matrix L if they are of the same dimension, and for any row indices j, j (cid:48) and column indices k, k (cid:48) ,
[Equation]
Note that the domination induces a partial order on all matrices: if K dominates L and L dominates M , then K dominates M .
The theorem below gives the necessary and sufﬁcient condition.
Theorem 12.
C ( A , B ) ( p , q ) is non-negative for all ( p , q ) if and only if matrix of the zero-sum part
Z dominates the matrix of the coordination part C .
The above theorem is based on the following crucial observation.
Observation 13.
For any matrix Z ,
[Equation]
Matrix domination only implies C ( A , B ) ( p , q ) is non-negative.
In order to have C ( A , B ) ( p , q ) to be strictly positive in the set S , we need θ -domination.
Deﬁnition 14.
We say matrix K θ -dominates ( θ > 0 )
matrix L if K dominates L , and there exist j , j (cid:48) , k , k (cid:48) such that | K jk + K j (cid:48)
[Equation]
The following theorem holds due to Lemma 4.
Theorem 15.
For any general bimatrix game ( A , B ) which is decomposed into zero-sum part ( Z , − Z ) and coordination part ( C , C ) , if Z θ -dominates C , then MWU with any sufﬁciently small step-size (cid:15) in the game ( A , B ) is Lyapunov chaotic in S δ with Lyapunov exponent
[Equation]
Note that in Deﬁnition 14, K θ -dominates L if a ﬁnite number of inequalities are satisﬁed.
In the context of Theorem 15, it is easy to see that there are quite many games ( A , B ) , such that Z θdominates C with all those inequalities strictly satisﬁed.
Thus, there exists an open neighbourhood around these games such that every game in the neighbourhood has its zero-sum part θ -dominates its coordination part.
This shows that such family of games has positive Lebesgue measure.
3.2. S ECOND C HARACTERIZATION : L INEAR P ROGRAM
Note that matrix domination is not always true.
In some scenarios, the zero-sum matrix might not dominate the coordination matrix.
Yet, it is still possible that C ( A , B ) ( p , q ) is strictly positive in the region S δ , when every entry in the coordination matrix is small .
Precisely, for a general bimatrix game ( A , B ) , if its coordination part ( C , C ) is small in the sense that the absolute values of all entries in C are smaller than some constant r , then we can bound C ( C , − C ) ( · ) by O ( r 2 ) .
This is not the only case where we can bound C ( C , − C ) ( · ) by a small term.
Even if the entries in matrix C are large, we can use trivial matrices to reduce them without affecting C ( C , − C ) ( · ) .
This is done via a linear programming approach described below.
Given a matrix K , let r ( K ) be the optimal value of following linear program:
[Equation]
Note that { g j + h k } j,k is a trivial matrix.
Let K (cid:48)
=
K − { g j + h k } j,k .
By Lemma 8, C ( K , − K ) ( · ) =
C ( K (cid:48) , − K (cid:48) ) ( · ) .
The following lemma shows that the value of C ( K , − K ) ( · ) is closely related to r ( K ) .
Lemma 16.
For any ( p , q ) in S δ = { ( p , q ) |∀ j, k, x j ( p ) ≥ δ and y k ( q ) ≥ δ } , ( r ( K ) · δ ) 2 ≤ C ( K , − K ) ( p , q ) ≤ r ( K ) 2 .
Then the theorem below follows by applying Lemma 16 with Lemma 4.
Theorem 17.
For any general bimatrix game ( A , B ) which is decomposed into zero-sum part ( Z , − Z ) and coordination part ( C , C ) , if there exists θ > 0 such that ( r ( Z ) · δ ) 2 ≥ ( r ( C ))
+ ( θδ 2 ) 2 , then MWU with any sufﬁciently small step-size (cid:15) in the game ( A , B ) is Lyapunov chaotic in
S δ with Lyapunov exponent
[Equation]
Intuitively, r ( Z ) is a distance measure from the zero-sum game ( Z , − Z ) to the trivial game space; analogously, r ( C ) is a distance measure from the coordination game ( C , C ) to the trivial game space.
Theorem 17 shows that if the coordination part is much closer to the trivial game space than the zero-sum [-part (depicted by the condition ( r ( Z ) · δ ) 2 ≥ ( r ( C )) 
 + ( θδ 2 ) 2 ),-]part, then MWU in this game is Lyapunov chaotic in S δ .
E XPERIMENT
[Figure]
To illuminate that volume expansion occurs when MWU is employed in game with small coordination part, we simulate
MWU in the reduced payoff space 10 in a game which is the sum of zero-sum game (cid:0) [
1 ] , (cid:2) [Figure]
[Figure]
(cid:3)(cid:1) and coordination game (cid:0)(cid:2)
[Figure]
, (cid:2)
[Figure]
(cid:3)(cid:1) .
In the strategy space, x ∗ =
y ∗ =
(0 . 5 , 0 . 5) is the unique Nash equilibrium of the game.
In the reduced dual space, the origin corresponds to the equilibrium.
We pick a square of side length
.around the origin as the set of starting points (the small red square in the middle of Figure 1).
As these starting points are evolved via
MWU with step-size 0 .
02 , we take snapshots after every 1900 time steps, which are shown with colors blue, pink, lime, purple, orange and green (then the colors repeat) respectively.
As shown in the ﬁgure, the volume (which is area in two-dimensional space) increases, and its shape changes from a square to a parallelogram.
C ONCLUSION AND F UTURE W ORKS
In this paper, we analyze the volume-changing behavior of several well-known learning algorithms (MWU, OMWU, FTRL) on general bimatrix games and multi-player games, which leads to a Lyapunov chaos analysis.
For bimatrix games, we do this by decomposing a game into the zero-sum part and the coordination part.
This decomposition turns the volume analysis into comparing the strengths of volume-expansion (zero-sum part) and volume-contraction (coordination part) of the
MWU dynamics.
The comparison of strengths is made via the notion of matrix domination and the use of a linear program.
For multi-player games, by the local equivalence, we show that the volume-changing behavior of MWU and OMWU are opposite to each other even in multi-player games.
We also show, for a general multi-player potential game, the key function C G is equal to a corresponding multi-player coordination game, which implies it is not positive.
Studying learning in matrix (normal-form) games, which are among one of the most classical game models, is a good theoretical starting point.
Matrix games admit mathematically amenable analyses, as demonstrated in our work and many previous works.
For future works, we are immensely interested in chaos analyses on settings that are more relevant to applications in ML, e.g. general GANs and differential games.
We believe the techniques we use (volume analysis, game decomposition, etc.) can be applicable.
A CKNOWLEDGMENTS
We thank several anonymous reviewers for their suggestions, which help to improve the readability of this paper from its earlier version.
Yixin Tao acknowledges NSF grant CCF-1527568, CCF1909538 and ERC Starting Grant ScaleOpt-757481.
Yun Kuen Cheung acknowledges Singapore
NRF 2018 Fellowship NRF-NRFF2018-07.
A F URTHER R ELATED W ORK
In the study of no-regret learning (e.g. Littlestone & Warmuth (1994); Freund & Schapire (1995)), a vast literature concerns general or even adversarial settings, in which the online arrivals of payoff values come with no pattern or even from an adversary.
More recently, settings where the online payoffs are more well-behaved, under the term of “predictable sequence” coined by Rakhlin & Sridharan (2013b), have been studied.
These settings include game dynamics, as the online payoffs are determined by the mixed strategy choices of the players, while these choices are updated gradually and somewhat predictably.
For these settings, online learning algorithms that perform particularly well, e.g. achieving regret bound below the canonical O ( √ T ) limit, are designed and studied (Hazan & Kale (2010); Chiang et al.
; Syrgkanis et al.
For instance, Nesterov’s excessive gap technique and optimistic mirror descent are found to achieve near-optimal regret O (log T ) in zerosum games (Daskalakis et al. (2015); Rakhlin & Sridharan (2013a)), and thus the empirical average of the learning sequence converges to Nash equilibrium of the game (see Freund & Schapire (1996) for an explanation).
OMWU (with time-varying step-sizes), and more generally optimistic variant of FTRL (Rakhlin & Sridharan (2013b)), are some canonical examples of such online learning algorithms.
Recently, there is a stream of work that examines how learning algorithms behave in games or min-max optimization from a dynamical-systemic perspective.
Replicator dynamics (RD; the continuous-time analogue of MWU) and continuous-time FTRL are found to achieve optimal regret in general settings (Mertikopoulos et al.
Furthermore, RD in zero-sum games or graphical constant-sum games admits a constant of motion and preserves volume; these two properties are used to show that such dynamical systems are near-periodic (Piliouras & Shamma (2014)
; Mertikopoulos et al.
; Boone & Piliouras (2019); Vlatakis-Gkaragkounis et al. ; Perolat et al. (2020)), captured rigorously under the notion of Poincar´e recurrence (Poincar ´ e (1890); Barreira (2006)).
However, when MWU, the forward Euler discretization of RD, is used in discrete-time setting in zero-sum games, the near-periodicity is destroyed totally; indeed, the system will never visit the same point (or its tiny neighbourhood) twice, converge to the boundary of the strategy simplex, and ﬂuctuate there irregularly (Bailey & Piliouras (2018); Cheung (2018)).
In contrast, (discrete-time) OMWU in zero-sum game is shown to converge to Nash equilibrium (Daskalakis & Panageas (2019)); yet, in the more general setting of min-max optimization, it was found that Optimistic Gradient Descent Ascent (OGDA) can have limit points other than (local) min-max solutions (Daskalakis & Panageas (2018)).
B P ROOFS IN S ECTION
Proof of Lemma 8.
First, observe that it sufﬁces to prove that the lemma holds when T 1 is a trivial matrix and T 2 is the zero matrix.
Then the lemma holds for any trivial matrices T 1 , T 2 due to symmetry:
C ( A , B ) ( p , q ) =
C ( A + T 1 , B ) ( p , q ) =
C ( A + T 1 , B + T 2 ) ( p , q ) .
Due to the deﬁnition of trivial matrix, we can write T 1 jk = u
j + v k .
Then
C [Equation] A [Equation]
A [Equation] B [Equation]
− E (cid:34)(cid:32)
[Equation]
(cid:96) ∈ Su (cid:96) x (cid:96) (cid:33) ( B jk − B j − B k ) (cid:35) + E
[ A jk + u j + v k ] · E [ B jk ] + E
[( A jk − A j − A k )
( B jk − B j − B k )] − E
[ A jk ] · E [ B jk ] = − E (cid:34)(cid:32) −
[Equation]
Sv (cid:96) y
[Equation]
(cid:96) ∈ S
[Equation]
(cid:96) x (cid:96) (cid:33) (
[Equation]
) (cid:35)
+ E [ u j + v k ] · E [ B jk ] = E [ v k + u j ] · E [ B jk − B j − B k ] + E [ u j + v k ] · E [ B jk ] . By recalling that E [ B jk − B j − B k ] = − E [ B jk ] , we have C ( A + T 1 , B ) ( p , q )
− C
( A , B ) ( p , q ) = 0 .
Proof of Observation 10.
Let P jk be the potential value of a potential game when Player 1 plays strategy j and Player 2 plays strategy k .
Then according to the deﬁnition the potential function, for any j 1 , j 2 and k ,
[Equation]
In particular, for any j, k , A jk = P jk + A 1 k − P 1 k .
This implies that there exists v such that
A jk =
P jk + v k for any j and k .
Similarly, there exists u such that B jk =
P jk + u j for any j and k .
This implies that any two-player potential games are coordination games plus trivial matrices.
Proof of Theorem 12.
We ﬁrst prove that if Z dominates C , then C ( A , B ) ( p , q ) is always nonnegative.
By Observation 13,
[Equation]
In contrast, if Z does not dominate C , then there exist ˆ j , ˆ j (cid:48) , ˆ k , ˆ k (cid:48) and δ > 0 such that
[Equation]
For each η > 0 , we construct p and q such that x ˆ j ( p ) =
x ˆ j (cid:48)
[Equation]
Furthermore, we let Υ denote the maximum absolute value of all entries in matrices A and B .
Then, for all j and k , | Z jk | ≤ Υ and | C jk | ≤ Υ .
Therefore,
[Equation]
The last inequality holds as ( C jk + C j (cid:48) k [Equation] − C jk
[Equation]
− C j
[Equation]
k ) 2 − ( Z jk + Z j
[Equation]
k [Equation] − Z jk (cid:48) − Z j (cid:48) k ) 2 ≤ 16Υ 2 .
The value of − δ (cid:0) 1 − η 2 (cid:1) 4 + | S 1 | 2 · | S 2 | 2 · η · 16Υ 2 will be negative if we pick a small enough η .
Proof of Observation 13.
Consider a random process, where j, j (cid:48) ∈ S 1 are randomly picked according to distribution x ( p ) , and k, k (cid:48) ∈ S 2 are randomly picked according to distribution y ( q ) .
Then the RHS of Observation 13 can be expressed as 14 · E (cid:2) ( Z jk + Z j (cid:48) k
(cid:48)
− Z jk (cid:48) − Z j (cid:48) k ) 2 (cid:3) .
Then we expand the squared term in the expectation.
Observing the symmetries within the expansion, we immediately have
[Equation]
Let Z j :=
[ Zy ] j and Z k =
[ Z T x ] k .
Then we have
[Equation]
Similarly, E [ Z jk Z j (cid:48) k ]
= E (cid:2) ( Z k ) 2 (cid:3) .
Lastly, E [ Z jk Z j (cid:48) k (cid:48) ]
=
E [ Z jk ] 2 .
Thus, the RHS of Observation 13 is simpliﬁed to
[Equation]
We complete the proof by noting that from the deﬁnition of C ( Z , − Z ) ( · ) in Eqn.
(4), C ( Z , − Z ) ( · ) can be rewritten as
[Equation]
while E [ Z j Z jk ] = (cid:80) j x j Z j (cid:80) k
y k Z jk =
(cid:80) j x j Z j Z j = E (cid:2) ( Z j ) 2 (cid:3) , and similarly E [ Z j Z jk ] =
E (cid:2) ( Z k ) 2 (cid:3) .
Proof of Theorem 15.
By Lemma 4, it sufﬁces to prove ¯ c ( A
[Equation]
≥ θ
[Equation]
δ 4 .
This holds because the matrix
Z θ -dominates C , which implies there exist j , j (cid:48) , k , and k (cid:48) such that
[Equation]
By applying Observation 13, C ( Z , − Z ) ( p , q ) ≥ C ( C , − C ) ( p , q ) + θ 2 δ 4 , because for ( p , q ) ∈ S δ , every x j ( p ) , y k ( q ) , x j (cid:48) ( p ) , y k (cid:48)
( q ) is at least δ .
By noting that C ( C , C ) ( p , q ) =
− C ( C , − C ) ( p , q ) and C ( A , B ) ( p , q ) =
C ( Z , − Z ) ( p , q ) + C ( C , C ) ( p , q ) , the result follows.
Proof of Lemma 16.
A key observation is the following equality:
[Equation]
F
[Equation]
g , h [Equation] =
[Equation]
Recall the notations K j = (cid:80)
k y k K jk and K k
= (cid:80) j x j K jk , and we let e := (cid:80) j,k x j y k K jk ≡
E [ K jk ] .
The equality (8) holds due to the following observations: (i) F ( g , h ) is a smooth convex function of its variables, thus all minimum points have the same function value; (ii) if ∂F∂g j and ∂F∂h k are all zero at some point ( g , h ) , then the point is a minimal point of F ;
(iii) C ( K , − K ) ( p , q ) is the variance of the random variable K jk − K j − K k (see the end of Section 2), and thus by a deﬁnition of variance,
[Equation]
since E [Equation] jk
[Equation]
, where g # j = K j − e and h # k
= K k ; and (iv) at ( g # , h # ) , the partial derivatives stated in (ii) are all zero.
With this observation and comparing this with the deﬁnition of r ( Z ) , it’s easy to ﬁgure out that
C ( K , − K ) ( p , q ) ≤ r ( K ) 2 .
To see C ( K , − K ) ( p , q ) ≥ ( r ( K )) · δ )
, we ﬁrst let g ∗ and h ∗ to be the optimal choice of g and h in C ( K , − K ) ( p , q ) =
min g , h (cid:80) j,k x j ( p ) · y k ( q ) · ( K jk − g j − h k ) 2 .
Due to the speciﬁcation of the linear program (7), we have
[Equation]
Therefore,
[Equation]
This immediately implies that K ) (
, q ) which make r ( ) smaller.
C M ULTI -P LAYER G AMES
Computing volume change of learning algorithm in multi-player game is slightly more involved than the two-player case.
We present a local equivalence formula of volume change between normal-form and graphical games.
This provides an intuitive procedure for understanding volume changes.
Proposition 20 shows that in multi-player game, the volume-changing behaviors of MWU and OMWU are again opposite to each other (which was shown for bimatrix game in Cheung &
Piliouras (2020)).
Graphical Games.
A graphical game Kearns et al.
(2001) is a special type of N -player game where the payoffs can be compactly represented.
In a graphical game H , for each pair of players i, k , there is an edge-game which is a bimatrix game between the two players, denoted by ( H i,k , ( H k,i ) T ) , where H i,k ∈ R n i × n k is the payoff matrix that denotes the payoffs to Player i .
Then the payoff to
Player i at strategy proﬁle s =
( s 1 , s 2 , · · · , s N ) is the sum of payoffs to Player i in all her edgegames, i.e. u i ( s )
=
(cid:80) k (cid:54) =
i H i,ks i ,s k .
As is standard, this payoff function is extended via expectation when the inputs are mixed strategies.
Here, we ﬁrst use an observation from Cheung & Piliouras (2019) to construct a family of multiplayer graphical games where MWU is Lyapunov chaotic in S N,δ :=
{ ( p 1 , · · · , p N ) |∀ i ∈
[ N ] , j ∈
S i , x ij ( p i ) ≥ δ } .
It was observed that the function C H ( p ) deﬁned in Eqn.
(5) is the sum of
C ( H i,k , ( H k,i ) T ) ( p i , p k ) of all pairs of Players i < k .
This observation yields Theorem 18.
Theorem 18.
Let G ↑ denote the family of bimatrix games which satisfy the condition either in
Theorem 15 or in Theorem 17.
In an N -player graphical game where each edge-game is drawn from G ↑ , if all players are employing MWU with a sufﬁciently small step-size (cid:15) , then the dynamical system is Lyapunov chaotic in S N,δ with Lyapunov exponent N ( N − 1) θ 2 δ 4 (cid:15) 2 / 4 (cid:80) Ni =1 n i .
Local Equivalence of General Games and Graphical Games.
Next, we present a theorem which connects the value of C G ( p ) of a general game to C H ( p ) , where H is a graphical game.
Theorem 19.
Given an N -player normal-form game G and any point p in the cumulative payoff space, the value of C G ( p ) is the same as C H ( p ) , where H is a graphical game speciﬁed as follows: for each pair of Players i, k and j ∈ S i , (cid:96)
∈ S k , the payoff to Player i in her edge-game with Player k when Player i picks j and Player k picks (cid:96) is H ikj(cid:96) :
= U ikj(cid:96) , where U ikj(cid:96) is deﬁned in Eqn.
This theorem shows that for any game G , the value of C G ( p ) is the same as in a particular graphical game, where each pair of players, ( i, k ) play a bimatrix game whose utility is exactly the utility of the original game G , but taking the expectation on the randomness of the other players’ strategies.
If the original game G is a graphical game, then in the graphical game H ikj(cid:96)
=
U ikj(cid:96)
+ c − i, − k , where c − i, − k is a parameter which does not depend on Players i and k .
Theorem 19 will be used in Appendix D to show the following proposition, which shows that the volume-changing behaviors of MWU and OMWU are opposite to each other in multi-player game, generalizing a prior result in Cheung & Piliouras (2020).
Proposition 20.
The volume integrands of MWU and OMWU in a multi-player game G are respectively
1 + C G ( p ) · (cid:15) 2 + O ( (cid:15) 3 ) and 1 − C G ( p ) · (cid:15)
+ O ( (cid:15) 3 ) .
Thus, volume expands locally around a cumulative payoff point p for MWU (resp.
OMWU)
if C G ( p ) is positive (resp. negative).
Multiplayer Potential Game.
By Observation 10, we know that the volume behavior of a potential game is equivalent to a corresponding coordination game in bimatrix game.
In this section, we want to show, this holds even in the multi player setting.
Proposition 21.
Suppose P is the potential function of a potential game U .
Let U P be a game that all players will receive P ( s ) when players play strategies s .
Then C U ( p ) =
C U P ( p ) ≤ 0 .
In Appendix E, we will discuss some situations where C U ( p ) is strictly less than 0 , thus OMWU is Lyapunov chaotic therein.
D L OCAL E QUIVALENCE OF V OLUME C HANGE BETWEEN N ORMAL - FORM AND G RAPHICAL G AMES
Here, we concern the volume change of a learning algorithm in multi-player game.
We ﬁrst recap from Cheung & Piliouras (2020) on how the volume change is computed for dynamical systems which are gradual (i.e. those governed by a small step-size), followed by a continuous-time analogue of OMWU in games, which are crucial for analyzing the volume change of discrete-time OMWU.
Then we compute the volume changes of MWU and OMWU in multi-player graphical games and normal-form games respectively.
Once these are done, the proofs of Proposition 20 and Theorembecome apparent.
D. D ISCRETE -T IME D YNAMICAL S YSTEMS AND V OLUME OF F LOW
We consider discrete-time dynamical systems in R d .
Such a dynamical system is determined recursively by a starting point s (0)
∈ R d and an update rule of the form s ( t + 1) =
G ( s ( t )) , for some function G : R d → R d .
Here, we focus on the special case when the update rule is gradual , i.e. it is in the form of
[Equation]
where F : R d → R d is a smooth function and step-size (cid:15) > 0 .
When F and (cid:15) are given, the ﬂow of the starting point s (0) at time t , denoted by Φ( t, s (0)) , is simply the point s ( t ) generated by the above recursive update rule.
Then the ﬂow of a set S ⊂ R d at time t , denoted by Φ( t, S ) , is the set { Φ( t, s ) | s ∈ S } .
Since F does not depend on time t , we have the following equality: Φ( t 1 + t 2 , S ) =
Φ( t 2 , Φ( t 1 , S )) .
By equipping R d with the standard Lebesgue measure, the volume of a measurable set S , denoted by vol ( S ) , is simply its measure.
Given a bounded and measurable set S ⊂ R d , if the discrete ﬂow in one time step maps S to S (cid:48)
= Φ(1 , S ) injectively, then by integration by substitution for multivariables,
[Equation]
where I is the identity matrix, and J ( s ) is the Jacobian matrix deﬁned below:
[Equation]
Clearly, analyzing the determinant in the integrand in Eqn.
(9) is crucial in volume analysis; we call it the volume integrand .
When the determinant is expanded using the Leibniz formula, it becomes a polynomial of (cid:15) , in the form of 1 + C ( s ) · (cid:15) h + O ( (cid:15) h +1 ) for some integer h ≥ 1 .
Thus, when the step-size (cid:15) is sufﬁciently small, the sign of C ( s ) dictates on whether the volume expands or contracts.
D. C ONTINUOUS -T IME A NALOGUE OF OMWU
OMWU does not fall into the category of dynamical systems deﬁned above, since its update rule is in the form of s ( t + 1)
=
G ( s ( t ) , s ( t − 1)) .
Fortunately, Cheung and Piliouras Cheung &
Piliouras (2020) showed that OMWU can be well-approximated by the online Euler discretization of a system of ordinary differential equations (ODE), and thus it can be well-approximated by a dynamical system.
The ODE system is given below.
p is a dual (cumulative payoff) vector variable, u : R + → R d is the function such that u ( t ) gives the instantaneous payoff vector at time t .
We assume that u is twice differentiable with bounded second-derivatives, and ˙ u denotes the time-derivative of u .
[Equation]
Online Euler discretization (OED) of Eqn.
(11) refers to the following time-discretization of the
ODE system.
In applications, ˙ u might not be explicitly given, and the sequence u (0) , u (1) , u (2) , · · · are available online (i.e., at time t we only have access of u ( τ ) for τ = 0 , 1 , · · · , t ).
As the discretization step is (cid:15) , we approximate ˙ u ( t ) by ( u ( t ) − u ( t − 1)) /(cid:15) .
By using this approximation,
OED of Eqn.
(11) yields
[Equation]
which is exactly the OMWU update rule in general context.
When compared the OED with the standard Euler discretization
[Equation]
OED incurs a local error that appears due to the approximation of ˙ u ( t ) .
The local error can be bounded by O ( (cid:15) 3 ) .
Cheung and Piliouras Cheung & Piliouras (2020) showed that eventually the determinant of the volume integrand is a of the form 1 + C ( s ) · (cid:15) 2 + O ( (cid:15) 3 ) , the local error does not affect the ﬁrst and second highest-order terms, and hence can be ignored henceforth.
D. MWU IN G RAPHICAL G AMES
Let H be a graphical game of N players, where between every pair of Players i and k , the payoff bimatrices are ( H ik , ( H ki ) T ) .
In the cumulative payoff space, let p =
( p 1 , · · · , p N ) denote the cumulative payoff proﬁle, and let x = ( x 1 , · · · , x N ) denote the corresponding mixed strategy proﬁle, where x i is a function of p i .
We will write x i and x i ( p i ) interchangeably.
The expected payoff to strategy j of Player i is
[Equation]
which will be used to compute the Jacobian matrices of MWU and OMWU.
For MWU, the Jacobian matrix J is a squared matrix with each row and each column indexed by ( i, j ) , where i is a Player and j ∈ S i .
The precise values of its entries are given below:
[Equation]
and
[Equation]
Then by expansion using Leibniz formula
[Equation]
the determinant of ( I + (cid:15) · J ) is
[Equation]
By noting the similarity of the double summation to C ( A , B ) ( · ) in Eqn.
[Equation]
we can immediately rewrite the above expression as
[Equation]
D. OMWU IN G RAPHICAL G AMES
For OMWU, as we pointed out already, we will ﬁrst consider its continuous analogue ﬁrst.
Thus, we need to compute ˙
u in the continuous-time setting.
By chain rule, we have ˙ [Equation] ij
[Equation]
p
[Equation]
and hence
[Equation]
Note that this is a recurrence formulae for d p d t .
By iterating it 12 , we have
[Equation]
Hence, its standard Euler discretization, which approximates the OED with local error O ( (cid:15) 3 ) , can be written as below (where we ignore the O ( (cid:15) 3 ) error terms):
[Equation]
With this, we are ready to compute the Jacobian matrix J for OMWU.
For all j 1 , j 2 ∈ S i , (cid:15)J ( i,j
[Equation]
, ( i,j
[Equation] = (cid:15) 2 (cid:88)
[Equation]
and for all i (cid:54)
=
k , j ∈ S i , (cid:96) ∈
S k ,
[Equation]
Then by expansion using Leibniz formula, the determinant of ( I + (cid:15) · J ) is
[Equation]
By a direct expansions on T 1 and T 2 , it is easy to see that T 1 = 2 T 2 (after ignoring O ( (cid:15) 3 ) terms).
On the other hand, the coefﬁcient of (cid:15) 2 in T 2 is exactly the same as the double summation in Eqn.
(14), thus it equals to − (cid:80) i,k :1 ≤ i<k ≤ N C
( H ik , ( H ki ) T ) ( p i , p k ) .
Overall, we show that the determinant equals to
[Equation]
Observation 22.
The coefﬁcient of (cid:15) 2 in Eqn.
(18) is the exact negation of the coefﬁcient of (cid:15) 2 in
Eqn.
D. C OMPLETING THE L OCAL E QUIVALENCE P ROOF
In a multiplayer normal-form game G , recall that notation Eqn.
We point out the following formulae:
[Equation]
MWU.
Here, MWU update rule is p ij ( t + 1) =
p ij ( t ) + (cid:15) · U ij .
When computing the Jacobian matrix for this update rule using the formulae above, and comparing it with the Jacobian matrix computed in Eqn.
(12) and Eqn.
(13), it is immediate that they are the same by setting H ikj(cid:96)
=
U ikj(cid:96) .
This derives Eqn.
(5), and completes the proof of Theorem 19.
OMWU.
As before, we use the continuous analogue and compute ˙ u .
By the chain rule and the above formulae, we have
[Equation]
and hence
[Equation]
Iterating the above recurrence yields
[Equation]
Its standard Euler discretization is
[Equation]
Now we compute the Jacobian matrix for this standard Euler discretization.
For j 1 , j 2 ∈ S i , (cid:15)J ( i
[Equation]
j 1 ) ,
[Equation]
i,j 2 ) =
[Equation]
and for all i (cid:54)
=
k , j ∈ S i , (cid:96) ∈
S k ,
[Equation]
=
[Equation]
By comparing this computed Jacobian matrix with the Jacobian matrix computed in Eqn.
(16) and
Eqn.
(17), it is immediate to see that their determinants are the same (after ignoring all O ( (cid:15) 3 ) terms) by setting H ikj(cid:96) =
U ikj(cid:96) .
With the result we just derived, together with Observation 22 and Theorem 19, Proposition 20 follows.
E M ULTI - PLAYER P OTENTIAL G AME
Proof of Proposition 21.
We know that the potential game satisﬁes the following condition:
[Equation]
Therefore, u i ( s i , s − i ) =
P ( s i , s − i )
+ v i ( s − i ) .
Note that v i ( s − i ) does not depend on s i , the strategy of player i .
By Theorem 19, let H ( U ) be the induced graphical game of U and H ( U P ) be the induced graphical game of U P .
Then,
[Equation]
=
C H ( U ) ( p ) (Theorem 19) = (cid:88) i,k
C ( H ( U ) ik , ( H ( U ) ki ) T ) ( p i , p k ) (By equation 15) =
(cid:88) i,k
C ( H ( U P ) ik , ( H ( U P ) ki ) T ) ( p i , p k ) (see explanation below) =
C H ( U P ) ( p ) (By equation 15) =
C U P ( p ) .
(Theorem 19)
The third equality holds as the difference between H ( U ) ik and H ( U P ) ik is a trivial matrix:
[Equation]
where 13 E − ( i,k ) (cid:2) v i ( s − i ) (cid:3) doesn’t depend on j , the strategy of player i , and only depends on l , the strategy of player k .
The same argument applies for ( H ( U ) ki ) T and ( H ( U P ) ki ) T .
To see C U ( p ) ≤ 0 , observe that the induced graphical game of U P between player i and k , ( H ( U P )
ik , ( H ( U P ) ki ) T ) , is also a bimatrix coordination game, which implies
C ( H ( U P ) ik , ( H ( U P ) ki ) T ) ( · ) ≤ 0 .
As C U ( p ) =
(cid:80) i,k C
[Equation]
H
[Equation]
U P )
ik , ( H ( U P ) ki ) T ) ( p i , p k ) , the result follows.
Next, we identify several cases such that C U ( p ) is strictly negative in the region
[Equation]
The conditions we pose are on the corresponding potential function P .
Note that H ( U P ) ik , the induced edge-game between player i and k , is also a coordination game, i.e. H ( U P )
ik =
( H ( U P ) ki ) T .
[Equation]
≥ θ, where P ikj(cid:96) =
E s − ( i,k ) (cid:2) P ( s i =
j, s k = (cid:96), s − ( i,k ) ) (cid:3) .
With this condition, we can prove that
C U p ) ≤ − θδ 2 for any p in S δ .
One key observation for this is true is that
[Equation]
as H ( U P
ik = U P ik = P ik
• Case 2 : If U is a graphical game
then if there exists a pair of player i 1 and i
such that the game between i
and i 2 is a non-trivial game
then C U will be strictly negative in S δ .
• Case 3 : Consider the payoff matrix of U P , the coordination game, between players i
and i 2 given a strategy proﬁle of the other players.
There are total (cid:81) i (cid:54)
=
i 1 ,i 2 n i such matrices, one for each strategy proﬁle of the other players, and each matrix is of dimension n i 1 × n i 2 .
We call these matrices the projected matrices for players i 1 , i 2 .
Let M denote the matrix space of n i 1 × n i 2 .
On the other hand, trivial matrices form a subspace of dimension n i 1 + n i 2 − 1 .
14 Let’s call this the trivial space , denoted by T .
We consider the direct decomposition M = T ⊕ V .
Let a set of bases of M be
B 1 , B 2 , B 3 , · · · , B n i 1 n i 2 , where the ﬁrst n i 1 + n i 2 − 1 bases form a basis of T , and the remaining bases form a basis of V .
Without loss of generality, we assume that all bases are of L 2 norm 1 .
Recall that a matrix T as { u j + v k } j,k .
Consider the natural linear map L such that L ( u 1 , u 2 , · · · , u n i 1 , v 1 , v 2 , · · · , v n i 2 ) maps to the trivial matrix T . that the kernel of L is of dimension 1 ,
( u 1 , u 2 , · · · , u n
, v v n
(cid:123)(cid:122) (cid:125) the u part , − 1 , − 1 , · · · , − 1 (cid:124) (cid:123)(cid:122) (cid:125)
v part
Thus n i n Given the above-mentioned bases of M , each of the projected matrices can be written into a unique linear combination of these bases.
Now, suppose there is a base B (cid:96) for l ≥ n i 1 + n i 2 (i.e. this base is in the set of bases for V ), such that all projected matrices have non-positive (or non-negative) coefﬁcients of this base, and at least one of these projected matrices (which we call a special projected matrix ) has strictly negative (or strictly positive) coefﬁcient of the base.
Then we claim that C ( H ( U P )
i 1 i 2 , ( H ( U P ) i 2 i 1 ) T ) ( p i 1 , p i 2 ) will be strictly negative in S δ .
This is because H ( U P ) i 1 i 2 is a convex combination of all those projected matrices, and by our assumption above, when H ( U P ) i 1 i 2 is expressed as the linear combinations of the bases of M , the coefﬁcient of B (cid:96) is strictly negative (or strictly positive), thus H ( U P )
i 1 i 2 cannot be a trivial matrix.
Suppose further that there exists θ > 0 such that a special projected matrix has negative (or positive) coefﬁcient for B (cid:96) which is smaller (or bigger) than − θ (or θ ), then we are guaranteed that H ( U P ) i 1 i 2 is bounded away from T for a distance of θδ N − 2 , 16 and hence as the calculations below show, C ( H ( U P )
i 1 i 2 , ( H ( U P ) i 2 i 1 ) T ) ( p i 1 , p i 2 ) ≤ − θ
δ 2 N − 2 .
If there exists a pair of player i 1 and i 2 such that this condition holds, then C U ≤ − θ
δ 2 N − 2 .
[Equation]
where { g ∗ j + h ∗ k } jk is projection of H ( U P )
i 1 i 2 on the trivial space.
The ﬁrst inequality follows as p ∈ S δ ; the second equality holds as the projection minimizing the distance to the trivial space, and the ﬁnal inequality comes from the distance from H ( U P )
i 1 i 2 to the trivial space.
For all these cases, we can have OMWU is C U to be strictly negative in domain S δ , which implies OMWU is Lyapunov chaotic in S δ .