C HAOS OF L EARNING B EYOND Z ERO - SUM AND C OORDINATION VIA G AME D ECOMPOSITIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
It is of primary interest for ML to understand how agents learn and interact dynamically in competitive environments and games (e.g. GANs).
But this has been a difﬁcult task, as irregular behaviors are commonly observed in such systems.
This can be explained theoretically, for instance, by the works of Cheung & Piliouras (2019; 2020), which showed that in two-person zero-sum games, if agents employ one of the most well-known learning algorithms, Multiplicative Weights
Update (MWU), then Lyapunov chaos occurs everywhere in the payoff space.
In this paper, we study how persistent chaos can occur in the general normal-form game settings, where the agents might have the motivation to coordinate (which is not true for zero-sum games) and the number of agents can be arbitrary.
We characterize bimatrix games where MWU, its optimistic variant (OMWU) or
Follow-the-Regularized-Leader (FTRL) algorithms are Lyapunov chaotic almost everywhere in the payoff space.
Technically, our characterization is derived by extending the volume-expansion argument of Cheung & Piliouras via the canonical game decomposition into zero-sum and coordination components.
Interestingly, the two components induce opposite volume-changing behaviors, so the overall behavior can be analyzed by comparing the strengths of the components against each other.
The comparison is done via our new notion of “matrix domination” or via a linear program.
For multi-player games, we present a local equivalence of volume change between general games and graphical games, which is used to perform volume and chaos analyses of MWU and OMWU in potential games.
I NTRODUCTION
In ML, it is of primary interest to understand how agents learn in competitive environments.
This is more strongly propelled recently due to the success of Generative Adversarial Networks (GANs), which can be viewed as two neural-networks (agents) playing a zero-sum game.
As such, Evolutionary Game Theory (EGT) (Hofbauer & Sigmund (1998); Sandholm (2010)), a decades-old area devoted to the study of adaptive (learning) behaviors of agents in competitive environments arising from Economics, Biology and Physics, has drawn attention from the ML community.
In contrast with the typical optimization (or no-regret) approach in ML, EGT provides a dynamical-systemic perspective to understand ML processes; see Appendix A which summarizes the recent successes of how this perspective helps tackle ML-related problems.
The dynamical-systemic perspective is particularly helpful in studying “learning in games”, where irregular behaviors are commonly observed, but the ML community currently lacks of a rigorous mean to do the relevant analyses.
Our theme is chaos , a central notion in the study of dynamical systems that captures instability and unpredictability.
In Computer Science and ML, chaos is an important issue but it has not received as much attention as it should, as chaos captures sensitivity to round-off errors, which affect numerical robustness of algorithms, and also the predictability and reproducibility of learning systems.
We characterize games in which popular learning algorithms exhibit chaotic behaviors, by employing game decomposition techniques.
Next, we discuss chaos and game decompositions.
All missing proofs are in the appendix.
Chaos.
Chaos generally means a system becomes unpredictable in the long run.
Lyapunov chaos is one of the most popular chaos notions which captures the butterﬂy effect : when the starting point of a dynamical system is slightly perturbed, the resulting trajectories and ﬁnal outcomes diverge quickly; see Deﬁnition 1 for a formal deﬁnition.
A Lyapunov chaotic system is sensitive to roundoff errors in computer simulations, and to errors in measuring payoffs.
Indeed, Edward Lorenz, a pioneer of the modern chaos theory, started working on the topic because he found round-off errors led to hugely different outcomes in his weather simulation program (Lorenz (1963)).
In the context of learning-in-game, chaos indicates that Nash equilibrium is not achievable in general.
To see the importance for the ML community to understand chaos, recall that one of our primary targets is to build predictable and reproducible ML systems.
The soundness of a newly proposed
ML system is usually supported by a theory, which is in turn supported by experimental evidences.
However, there is a gap between theory and experiments: theory is built upon the assumption of inﬁnite precision, but experiments are done using ﬁnite-precision computers where round-off occurs in every step.
While the round-off error per step is small, it is unclear how it can affect the ﬁnal outcome of the system, let alone the cumulating effect of all the errors across multiple steps.
Moreover, how the round-off is done depends on the OS, the CPU/GPU architecture, the compiler and more, which vary from computer to computer, so reproducing the same result across multiple computers is not as trivial as some of us had presumed.
These unpredictability and irreproducibility issues are exempliﬁed by two quotes from Ali Rahimi’s NIPS’2017 test-of-time award speech: “Someone on another team changed the default rounding mode of some Tensorﬂow internals from ‘truncate toward zero’ to ‘round to even’.
Our training broke, our error rate went from less than 25% error to ∼ 99 . 97% error.”
“If a machine learning algorithm does crazy things on a linear model, it’s going to do crazy things on complex non-linear models too.”
Game Decomposition.
To see how game decomposition works, we ﬁrst compare the dynamicalsystemic and optimization approaches.
Historically, there is a trade-off of them between generality of settings and scope of results.
1 No-regret learning works even in general and adversarial settings, but its effectiveness is benchmarked w.r.t.
the average of history , while it sheds little insight on the daily behaviors of the learning-in-game systems.
The dynamical-systemic approach primarily aims at understanding the daily behaviors, but compelling results are often limited to speciﬁc families of games because non-linear dynamical systems are inherently difﬁcult to analyze in general.
A natural and generic approach to extend those compelling results to more general families of games is via game decomposition.
Suppose there is a speciﬁc family of games, denoted by
H , for which some compelling results are shown.
Given a general game, we seek to decompose it into a sum of its projection on H and a residue component.
If the residue is small, then it is plausible that those compelling results extend (approximately).
For instance, if the residue is small, then any Nash equilibrium of the projection is an approximate Nash equilibrium of the original game.
More generally, a game might be decomposed into three or more components to be studied separately.
In seeking of games where learning is stable, the game decomposition approaches were used (Candogan et al.
(2011; 2013a;b);
Letcher et al.
with H being potential games.
While we also employ game decomposition technique, our target is in the opposite end of those of the cited works above, which is to seek broad families of games where popular learning algorithms are
Lyapunov chaotic.
Our Contributions.
Our starting point is the recent works of Cheung & Piliouras (2019; 2020).
They considered a classical technique in the study of dynamical systems called volume analysis .
Volume analysis considers a set of starting points of positive volume (e.g. a ball centred at a point).
When this set of starting points evolves according to the rule of dynamical system, it becomes a new set with a different volume.
Intuitively, volume is a measure of the range of possible outcomes, so the larger it is, the more unpredictable the system is.
If the set’s volume increases exponentially with time, then its diameter increases exponentially too, which implies Lyapunov chaos.
Cheung & Piliouras employed the volume-expansion argument to show that Multiplicative
Weights Update (MWU) and Follow-the-Regularized-Leader (FTRL) algorithms in two-player zerosum and graphical constant-sum games are Lyapunov chaotic everywhere in the cumulative payoff (dual) space 2 ; analogous result holds for the optimistic variant of MWU (OMWU) in two-player coordination games.
This indicates that when players repeatedly play the games by employing the respective learning algorithms, a slight perturbation on the initiating condition can lead to a wide range of cumulative payoffs in the long run.
This can be shown to imply instability in the mixed strategy (primal) space.
The volume-expansion argument crucially relies on analyzing a function C G ( · ) in the cumulative payoff space, where G denotes the underlying game.
Volume expands with MWU/FTRL (resp.
OMWU)
if and only if C G ( · ) is positive (resp. negative).
Cheung & Piliouras proved that
C G ( · ) is always positive if G is two-person zero-sum, and it is always negative if G is a two-person coordination game.
It is natural to ask: are these chaos results isolated in the sense that they hold due to the very speciﬁc structures of those games, or do they generalize broadly?
In the other way around, we ask:
Does the Lyapunov chaos phenomena of learning extend beyond two-player zero-sum games, two-player coordination games and multi-player graphical constant-sum games?
We answer the above question afﬁrmatively, both for two-player and multi-player settings.
To keep the exposition simple, we focus on the technical aspects in the two-player setting in the main text, and defer the formal statements of results and the details for multi-player setting to Appendix C.
Two-player normal form games.
For the family of two-player normal-form games (bimatrix games)
G , we present two new techniques to go beyond zero-sum game and coordination games.
The ﬁrst technique is the well-known direct-sum decomposition G = Z ⊕ C , where every bimatrix game ( A , B ) is decomposed into the sum of a zero-sum game ( Z , − Z ) and a coordination game ( C , C ) (Basar & Ho (1974); Kalai & Kalai).
We show that C G ( · ) is a simple sum of the zero-sum game part and the coordination part: C G ( · ) =
C ( Z , − Z ) ( · ) + C ( C , C ) ( · ) (Lemma 4).
Recall that
C ( Z , − Z ) ( · ) is always positive (as ( Z , − Z ) is a zero-sum game) and C ( C , C ) ( · ) is always negative (as ( C , C ) is a coordination game).
Thus, if C ( Z , − Z ) ( · ) is always relatively larger than − C ( C , C ) ( · ) , then we always have volume expansion in G .
The second technique is the trivial matrices (see Deﬁnition 2).
Intuitively, trivial matrices are a set of matrices which do not affect the volume changing behavior of the game: for any bimatrix game ( A , B ) and any trivial matrices T 1 and T 2 , C ( A , B ) ( · ) ≡ C ( A + T 1 , B + T 2 ) ( · ) (Lemma 5).
An immediate application of trivial matrices is for bimatrix potential games (Monderer & Shapley (1996)).
For any bimatrix potential game P , C P ( · ) is identical to C ( C , C ) ( · ) for some coordination game ( C , C ) , which implies OMWU is Lyapunov chaotic everywhere in P (Observation 6).
Based on these techniques, we identify two characterizations of bimatrix games where MWU and FTRL are Lyapunov chaotic almost everywhere (Theorems 9 and 11).
The characterizations are based on our new notion of matrix domination (see Deﬁnition 4), and a linear program (Eqn. 7) which is designed to prune out the trivial-matrix projection and keep the residue part minimal.
Such family of games has positive Lebesgue measure in the bimatrix game space, so it is not conﬁned to any proper game subspace 3 .
This justiﬁes the claim that the occurrences of chaos are not only circumstantial, but a rather substantial issue in learning in games.
Analogous result holds for OMWU.
Multi-player normal-form games.
For the family of multi-player games, we ﬁrst use an observation in Cheung & Piliouras (2019), coupled with our new ﬁndings about bimatrix games discussed before, to present a new family of graphical games in which MWU is Lyapunov chaotic almost everywhere (Theorem 12); the new family strictly includes all graphical constant-sum games.
To facilitate volume analyses in general normal-form games, we establish their local equivalence of volume change with graphical games.
Precisely, we show that C G ( p ) for a general game G is the same as C H ( p ) for some graphical game H ; H will depend on the point p , that’s why we say the equivalence is local (Theorem 13).
This provides an intuitive procedure for understanding volume changes.
We show that the volume-changing behaviors of MWU and OMWU are opposite to each other in multi-player game (Proposition 14).
We use these to analyze MWU and OMWU in multiplayer potential games; in particular, we show that C G ( p ) of a multi-player potential game is equal to C C ( p ) of a corresponding multi-palyer coordination game, while C C ( p ) ≤ 0 (Proposition 15).
(Theorem 12, Theorem 13, Proposition 14 and Proposition 15 are formally stated in Appendix C.)
P RELIMINARY
In this paper, every bold lower-case alphabet denotes a vector, every bold upper-case alphabet denotes a matrix or a game.
When we say a “game”, we always mean a normal-form game.
Given n , let ∆ n denote the mixed strategy space of dimension n − 1 4 , i.e. { ( z 1 , z 2 , · · · , z n ) | (cid:80) nj =
z j = 1 } .
Normal-Form Games.
Let N denote the number of players in a game.
Let S i denote the strategy set of Player i , and S :
= S 1 × · · · × S N .
Let n i = | S i | .
s =
( s 1 , · · · , s N ) ∈ S denotes a strategy proﬁle of all players, and u i ( s ) denotes the payoff to Player i when each player picks s i .
A mixed strategy proﬁle is denoted by x = ( x 1 , · · · , x N ) ∈ ∆ n 1 × · · · × ∆ n N , and u i is extended to take mixed strategies as inputs via u i ( x ) =
E s ∼ x
[ u i ( s )] .
We let − ( i 1 , · · · , i g ) denotes the player set other than players i 1 , · · · , i g .
We also let
[Equation]
which is the expected payoff to Player i 1 when: for 1 ≤ f ≤ g , Player i f picks strategy
j f , while for each player i / ∈ { i 1 , · · · i g } , she picks a strategy randomly following x i .
We also use U i 1 i 2 ···
[Equation]
j 1 j 2 ··· j g if x is clear from the context.
We say a game is a zero-sum game if (cid:80) i u i ( s )
=
for all s ∈ S , and we say a game is a coordination game if u i ( s ) =
u k ( s ) for all Players i and k and for all s ∈ S .
When N = 2 , such games are called bimatrix games , for which we adopt the notations below.
Let ( A , B ) denote a bimatrix game, where for any j ∈ S 1 , k ∈ S 2 , A jk :=
u 1 ( j, k ) , B jk := u 2 ( j, k ) .
x and y denote mixed strategies of Players 1 and 2 respectively.
A bimatrix game is a zero-sum game if A = − B ; it is a coordination game if A = B .
Note that U 1 j =
[ Ay ] j ,
U 2 k =
[ B T x ] k , which we denote by A j , B k respectively when x , y are clear from context; B j , A k are deﬁned analogously.
MWU, FTRL and OMWU in Games.
All three algorithms have a step-size (cid:15) , and can be implemented as updating in the cumulative payoff (dual) space.
In each round, the players’ actions (mixed strategies) in the strategy space are functions of the cumulative payoff vectors to be deﬁned below, and these actions are then used to determine the payoffs in the next round.
For a player with d strategies, let p t ∈ R
d denote her cumulative payoff vector at time t , and let p 0 ∈ R
d denote the starting point chosen by the player.
For MWU in a game, the update rule for Player i is
[Equation]
where U ij is the function deﬁned in Eqn.
(1), and x t is the mixed strategy as below:
[Equation]
For OMWU in a game, the update rule for Player i starts with p 1 = p 0 , and for t ≥ 2 , p t +1 j = p tj + (cid:15) · (cid:2) 2 U ij ( x t ) − U ij ( x t − 1 ) (cid:3) , where x t is determined by Eqn.
For FTRL in a game, the update rule for Player i is same as Eqn.
(2), but x t is determined as below using a convex regularizer function h
i : ∆ d → R : x t = arg max x ∈ ∆
d {(cid:104) p t , x (cid:105) − h
i ( x ) } .
As all the results for MWU can be directly generalized to FTRL as discussed in (Cheung & Piliouras, 2019, Appendix D), to keep our exposition simple, in the rest of this paper, we focus on MWU and
OMWU, and their comparisons.
For bimatrix game, we use p , q to denote the cumulative payoff vectors of Players 1 and 2 respectively.
Dynamical Systems, Lyapunov Chaos and Volume Analysis.
A learning-in-game system can be viewed as a discrete-time dynamical system, for which we present a simpliﬁed deﬁnition which suits our need.
A discrete-time dynamical system (d.s.) in R d is determined by a starting point r
(0) ∈ R d and an update rule r ( t + 1) =
f ( r ( t )) , where f : R d → R d is a function.
5 The sequence r (0) , r (1) , r (2) , · · · is called a trajectory of the d.s..
When f is clear from the context, we let Φ : ( N ∪ { 0 } ) × R d
→ R d denote the function such that Φ( t, r ) is the value of r ( t ) generated by the d.s. with starting point set to r .
Given a set U ⊂ R d , we let Φ( t, U ) =
{ Φ( t, r ) | r ∈ U} .
Let
B ( r , z ) denote the open ball with center r and radius z .
There are several similar but not identical deﬁnitions of Lyapunov chaos, all capturing the butterﬂy effect : when the starting point is slightly perturbed, the resulting trajectories diverge quickly.
We use the following deﬁnition, which was also used by Cheung & Piliouras (2019; 2020) implicitly.
Intuitively, a system is Lyapunov chaotic in an open set O ⊂ R d if for any r ∈ O and any open ball
B around r , as long as Φ( t, B ) remains inside O , there exists r (cid:48)
∈ B such that (cid:107) Φ( t, r (cid:48) ) − Φ( t, r ) (cid:107) grows exponentially with t .
Lyapunov exponent in the deﬁnition below is a measure of how fast the exponential growth is; the larger it is, the more unpredictable the d.s. is.
Deﬁnition 1.
A d.s. is Lyapunov chaotic in an open set
O ⊂ R d if there exists a constant λ > 0 and a Lyapunov exponent γ ≡ γ ( O ) > 0 , such that for any r ∈ O , for any sufﬁciently small δ > 0 and for all t satisfying 0 ≤ t < min { τ | τ ≥ 0 , Φ( τ, B ( r , δ ))
(cid:54)⊂ O} ,
[Equation]
A d.s. is Lyapunov chaotic everywhere if it is Lyapunov chaotic in any bounded open set.
In the above deﬁnition, all norms and radii are Euclidean norms.
For capturing round-off errors in computer algorithms and ML systems, it is more natural to use (cid:96) ∞ -norm with δ be the maximum round-off error per step, say ∼ 10 − 16 when IEEE 754 binary64 (standard double) is used.
When O is a small set, it is easy to determine whether a d.s. is Lyapunov chaotic in O , since the dynamic can be locally approximated by a linear d.s., where the eigenvalues of the local Jacobian characterizes chaotic behaviors.
But when O is large, determining whether Lyapunov chaos occurs is difﬁcult in general.
Cheung & Piliouras (2019) found that volume analysis can be useful in this regard, based on the following simple observation.
Proposition 1.
In R d , if a set U has volume at least v , then its radius w.r.t.
any point r ∈ U is at least v 1 /d / 2 .
Thus, if the volume of Φ( t, U ) of some d.s. is Ω(exp( γt )) for some γ > 0 , then the radius of Φ( t, U ) w.r.t.
any point r ∈ Φ( t, U ) is Ω(exp( γd · t )) .
Cheung and Piliouras showed Lemma 2 below, which, for bimatrix games, reduces volume analysis to analyzing the sign the function C ( A , B ) ( p , q ) deﬁned in Eqn.
(4) below; the sign also determines the local volume-changing behavior around the point ( p , q ) when MWU is used.
6 Based on
Proposition 1 that converts volume expansion to radius expansion, the sign can be used to determine if the dynamic system is Lyapunov chaotic.
Let A j = (cid:80) k (cid:48)
A jk (cid:48)
y k (cid:48) and
A k = (cid:80) j (cid:48)
x j (cid:48)
A j (cid:48) k ; and, similarly, B j = (cid:80) k (cid:48)
B jk (cid:48)
y k (cid:48) and B k
= (cid:80) j (cid:48)
x j (cid:48)
B j (cid:48) k .
Then,
[Equation]
Note that here x and y are the shorthand for x ( p ) and y ( q ) , which are the mixed strategies (i.e. probability distributions over strategies) of Players 1 and 2 respectively, as computed via
Eqn.
Also, E x , y [ f ( j, k )]
=
E ( j,k ) ∼ ( x ( p ) , y ( q ))
[ f ( j, k )] is the expected value of f ( j, k ) when the strategies j and k are randomly chosen according to the distrubutions x ( p ) and y ( q ) respectively.
For multi-player game G , the analogous function C G ( · ) is given below; the U quantities were deﬁned in Eqn.
Lemma 2 is adapted from Cheung & Piliouras (2019) for games with any number of players.
Derivation of Eqn.
(5) uses the Jacobian of the corresponding d.s. and integration by substitution; see Appendix D.
[Equation]
Lemma 2.
Suppose O is a set in the cumulative payoff space R d where d = n 1 + · · · + n N , and ¯
[Equation]
G
[Equation]
O
[Equation] [Equation]
Then for the d.s. in which MWU with any sufﬁciently small step-size (cid:15) is employed to play the game
G , it is Lyapunov chaotic in O with Lyapunov exponent ¯
[Equation]
/ 2 d .
If MWU is replaced by OMWU, then the same result holds by replacing the condition (6) with ¯ c ( O ) := inf ( p 1 , ··· , p N ) ∈O [ − C G ( p 1 , · · · , p N )] > 0 .
Note that if we start from a Nash equilibrium in the strategy space, MWU and OMWU will stay at the equilibrium.
However, if this equilibrium ( x ∗ , y ∗ ) satisﬁes the conditions in Corollary 3 below, there are points arbitrarily close to the equilibrium that keep moving away from the equilibrium (if the region { ( x (cid:48) , y (cid:48) )
=
( x ( p ) , y ( q ))
| ( p , q ) ∈ O} is large in the strategy simplex ∆ n 1 × ∆ n 2 ).
Corollary 3 (Adapted from (Cheung & Piliouras, 2020, Theorem 5)) .
Let ( x ∗ , y ∗ ) be a point in the interior of the strategy space.
Suppose that there exists ( p , q ) in the cumulative payoff space, such that x ∗ = x ( p ) and y ∗ =
y ( q ) .
Furthermore, suppose C ( A , B ) ( p , q ) > 0 , and ( p , q )
∈
O where
O is the set described in Lemma 2.
Then there are strategy points arbitrarily close to ( x ∗ , y ∗ ) such that MWU in the game ( A , B ) eventually leaves the corresponding strategy set of O , i.e. { ( x (cid:48) , y (cid:48) )
=
( x ( p ) , y ( q ))
| ( p , q ) ∈ O} .
When the game is zero-sum, i.e., B = − A , hence C
( A , B ) ( p , q ) =
E x , y (cid:2) ( A jk − A j − A k ) 2 (cid:3) −
E x , y
[( A jk )] 2 .
Since E x , y [ A jk ] = E x , y
[ A j ] = E x , y
[ A k ] and hence E x , y
[ A jk − A j − A k ] = − E x , y
[ A jk ] , C ( A , B ) ( p , q ) is indeed the variance of the random variable A jk − A j − A k , and thus is non-negative.
By Eqn
(4), we have C (
A
, B
( p , q ) = −
C
A
, −
B p
, q )
Thus, for any coordination game ( A , A
, we have C (
A
, A ) ( p , q ) = −
C ( A , − A
( p
, q
) ≤ 0 .
B IMATRIX G AMES
In this section, we focus on general bimatrix games ( A , B ) .
In Section 3.1, we present two tools for analyzing C ( A , B ) ( · ) , and then we provide an example to show how to use these tools.
In Section 3.2, we present two characterizations such that the dynamics are Lyapunov chaotic almost everywhere.
3. T OOLS FOR A NALYZING B IMATRIX G AME
First Tool: Canonical Decomposition for Bimatrix Games.
For every bimatrix game ( A , B ) , it admits a canonical decomposition (Basar & Ho (1974); Kalai & Kalai) into the sum of a zero-sum game ( Z , − Z ) and a coordination game ( C , C ) , where Z = 12 ( A − B ) and C = 12 ( A + B ) , i.e.
( A , B ) =
( Z , − Z )
+ ( C , C ) .
We call ( Z , − Z ) the zero-sum part of the game ( A , B ) , and ( C , C ) the coordination part of the game.
Our ﬁrst result shows that the function C ( · ) can be decomposed neatly into the two parts too.
Lemma 4.
For any bimatrix game ( A , B ) ,
[Equation]
Proof.
We use Eqn.
(4) to expand the following:
[Equation]
− E (cid:2) ( A jk
[Equation]
B jk − A j − B j − A k −
[Equation]
k
[Equation]
2 (cid:3)
[Equation]
=
E (cid:2)
( A
[Equation]
A j
[Equation]
j − A k
[Equation]
k ) 2 − ( A jk
[Equation]
B jk − A j
[Equation]
j
[Equation]
A k −
[Equation] [Equation]
=
[Equation]
By the end of Section 2, we discussed that C ( Z , − Z )
( p , q ) is always non-negative and
C ( C , C ) ( p , q ) is always non-positive.
By the above lemma, we can analyze the volume-changing behavior of a bimatrix game ( A , B ) by looking at its zero-sum and coordination parts independently.
One simple intuition is that if the coordination (resp.
zero-sum) part is small, then the volumechanging behavior of ( A , B ) is closer to the behavior of the zero-sum (resp. coordination) part.
We realize this intuition quantitatively in the next subsection.
Second Tool: Trivial matrix.
Trivial matrices are matrices which do not affect the volume-changing behavior, as depicted in Lemma 5 below.
Deﬁnition 2 (Trivial Matrix) .
T ∈ R n × m is a trivial matrix if there exists real numbers u 1 , u 2 , · · · , u n and v 1 , v 2 , · · · , v m such that T jk =
u j + v k for all j ∈ [ n ] , k ∈ [ m ] .
Lemma 5.
For any two trivial matrices T 1 , T 2 , for any two matrices A , B ∈ R n × m ,
C ( A , B ) ( p , q ) ≡ C ( A + T 1 , B + T 2 ) ( p , q ) .
One immediate application of this lemma is for two player potential games.
Deﬁnition 3.
A game G is a potential game if there exists a potential function P : S → R such that for any Player i and any strategy proﬁle s ∈ S , P ( s i , s − i )
−P ( s (cid:48)
i , s − i )
=
u i ( s i , s − i )
− u i ( s i (cid:48) , s − i ) .
For the potential game, we have the following observation:
Observation 6.
For any bimatrix potential game ( A , B ) , there is a coordination game ( P , P ) such that A − P , B − P are trivial matrices.
P is the matrix representation of the potential function P .
This observation immediately implies that the volume-changing behavior of potential game is equivalent to that of a corresponding coordination game.
We give a concrete example to show how these tools help us to analyze the C ( A , B ) ( · ) .
A Simple Example.
We will show how to use our tools to demonstrate C ( · ) ≥ 0 everywhere for the following game.
In the example, each player has three strategies.
The payoff bimatrix ( A , B ) is given below.
The ﬁrst number gives the payoff of the row player, who chooses strategy from { a, b, c } ; the second number gives the payoff of the column player, who chooses strategy from { 1 , 2 , 3 } .
We ﬁrst use our ﬁrst tool to decompose this game into zero-sum part ( Z , − Z ) and
Strategy
Strategy Strategy
Strategy a
[Equation]
Strategy b
[Equation]
c
[Equation]
coordination part ( C , C ) , where Z = (cid:104) 0 8 − 8 − 8 0 8 8 0 0 (cid:105) and C =
(cid:104)
4 2 0 0 4 6 0 4 (cid:105) .
At this point, we still cannot easily ﬁgure out which one is larger between C ( Z , − Z ) ( · ) and C ( C , − C ) ( · ) .
However, we can further decompose the coordination part by the second tool : C = (cid:104) 4 2 4 2 0 2 4 2 4 (cid:105) + (cid:104) 0 2 − 2 − 2 0 2 2 − 2 0 (cid:105) , where the ﬁrst matrix on the RHS is a trivial matrix (using notations in Deﬁnition 2, u = v T =
It’s easy to see the second matrix on the RHS is 14 Z .
Then by Lemmas 4 and 5, and the deﬁnition of the function C , for any point ( p , q ) in the cumulative payoff space,
[Equation]
3. R ESULTS FOR B IMATRIX G AMES
In this subsection, we identify several characterizations for general bimatrix games in which we have chaotic behavior with MWU dynamic in a following set S δ in the cumulative payoff space R n 1 + n 2 .
Note that when δ is tiny, its strategy correspondence covers almost the entirety of the strategy simplex ∆ n 1 × ∆ n 2 , thus we informally say that if the d.s. is Lyapunov chaotic in S δ for a tiny δ , then it is Lyapunov chaotic almost everywhere .
[Equation]
In order to show chaotic behavior of MWU in a speciﬁc bimatrix game ( A , B ) , it is sufﬁcient to show C ( A , B ) ( p , q ) is strictly positive in the region S δ , due to Lemma 2.
In the previous subsection, we show that for each game ( A , B ) , it can be decomposed into a zero-sum part ( Z , − Z ) and a coordination part ( C , C ) .
Furthermore, C ( A , B ) ( p , q ) =
C ( Z , − Z ) ( p , q ) + C ( C , C ) ( p , q ) .
We also raise an intuition that if the zero-sum part is small, then the volume behavior in the game ( A , B ) will be similar that in the coordination part; conversely, if the coordination part is small, then the volume behavior will be similar to the zero-sum part.
However, we have not yet presented a way to compare the largeness of the two parts.
This is what we do here.
3.2. F IRST C HARACTERIZATION : M ATRIX D OMINATION
The ﬁrst characterization we identify is matrix domination.
In this part, we show that under certain conditions, the zero-sum part is always no less than the coordination part,
i.e. C
( Z , − Z ) ( p , q ) ≥ − C
( C , C ) ( p , q ) for all ( p , q ) .
This directly implies C ( A , B ) ( p , q ) will be non-negative in the whole cumulative payoff space.
Interestingly, the condition we identify is both necessary and sufﬁcient.
Similar result can also be achieved in the case that coordination part is always no less than the zero-sum part.
We ﬁrst introduce the deﬁnition of the matrix domination.
Deﬁnition 4.
We say matrix K dominates matrix L if they are of the same dimension, and for any row indices j, j (cid:48) and column indices k, k (cid:48) ,
[Equation]
Note that the domination induces a partial order on all matrices: if K dominates L and L dominates M , then K dominates M .
The theorem below gives the necessary and sufﬁcient condition.
Theorem 7.
C ( A , B ) ( p , q ) is non-negative for all p and q if and only if matrix of the zero-sum part
Z dominates the coordination part C .
The above theorem is based on the following crucial observation.
Observation 8.
For any matrix Z ,
[Equation]
Z j (cid:48) k (cid:48) − Z jk (cid:48) − Z j (cid:48) k ) 2 .
Matrix domination only implies C ( A , B ) ( p , q ) is non-negative.
In order to have C ( A , B ) ( p , q ) to be strictly positive in the set S , we need θ -domination.
Deﬁnition 5.
We say matrix K θ -dominates ( θ > 0 )
matrix L if K dominates L , and there exist j , j (cid:48) , k , k (cid:48) such that | K jk + K j (cid:48) k (cid:48)
− K jk (cid:48) − K j (cid:48) k
| ≥ | L jk + L j (cid:48) k (cid:48) − L jk (cid:48) − L j (cid:48)
k | + θ.
The following theorem holds due to Lemma 2.
Theorem 9.
For any general bimatrix game ( A , B ) which is decomposed into zero-sum part ( Z , − Z ) and coordination part ( C , C ) , if Z θ -dominates C , then MWU with any sufﬁciently small step-size (cid:15) in the game ( A , B ) is Lyapunov chaotic in S δ with Lyapunov exponent θ 2 δ 4 2( n 1 + n 2 ) (cid:15) 2 .
Note that in Deﬁnition 5, K θ -dominates L if a ﬁnite number of inequalities are satisﬁed.
In the context of Theorem 9, it is easy to see that there are quite many games ( A , B ) , such that Z θdominates C with all those inequalities strictly satisﬁed.
Thus, there exists an open neighbourhood around these games such that every game in the neighbourhood has its zero-sum part θ -dominates its coordination part.
This shows that such family of games has positive Lebesgue measure.
3.2. S ECOND C HARACTERIZATION : L INEAR P ROGRAM
Note that matrix domination is not always true.
In some scenarios, the zero-sum matrix might not dominate the coordination matrix.
Yet, it is still possible that C ( A , B ) ( p , q ) is strictly positive in the region S δ , when every entry in the coordination matrix is small .
Precisely, for a general bimatrix game ( A , B ) , if its coordination part ( C , C ) is small in the sense that the absolute values of all entries in C are smaller than some constant r , then we can bound C ( C , − C ) ( · ) by O ( r 2 ) .
This is not the only case we can bound C ( C , − C ) ( · ) by a small term.
Even the entries in matrix C are large, we can use trivial matrices to reduce them without affecting
C ( C , − C ) ( · ) .
This is done via a linear programming approach described below.
Given a matrix K , let r ( K ) be the optimal value of following linear program:
[Equation]
Note that { g j + h k } j,k constructs a trivial matrix.
Let K (cid:48)
=
K − { g j + h k } j,k .
By Lemma 5,
C ( K , − K ) ( · ) =
C ( K (cid:48) , − K (cid:48) ) ( · ) .
The following lemma shows that the value of C ( K , − K ) ( · ) is closely related to r ( K ) .
Lemma 10.
For any ( p , q ) in S δ = { ( p , q ) |∀ j, k, x j ( p ) ≥ δ and y k ( q ) ≥ δ } ,
[Equation]
Then, the theorem below follows by applying Lemma 10 with Lemma 2.
Theorem 11.
For any general bimatrix game ( A , B ) which is decomposed into zero-sum part ( Z , − Z ) and coordination part ( C , C ) , if there exists θ > 0 such that ( r ( Z ) · δ ) 2 ≥ ( r ( C ))
+ ( θδ 2 ) 2 , then MWU with any sufﬁciently small step-size (cid:15) in the game ( A , B ) is Lyapunov chaotic in
S δ with Lyapunov exponent
[Equation]
Intuitively, r ( Z ) is a distance measure from the zero-sum game ( Z , − Z ) to the trivial game space; analogously, r ( C ) is a distance measure from the coordination game ( C , C ) to the trivial game space.
Theorem 11 shows that if the coordination part is much closer to the trivial game space than the zero-sum part (depicted by the condition ( r ( Z ) · δ ) 2 ≥ ( r ( C ))
+ ( θδ 2 ) 2 ), then MWU in this game is Lyapunov chaotic in S δ .
E XPERIMENT
[Figure]
To illuminate that volume expansion occurs when MWU is employed in game with small coordination part, we simulate
MWU in the reduced payoff space 7 in a game which is the sum of zero-sum game (cid:0) [
[Figure]
1 ] , (cid:2)
[Figure]
[Table]
and
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
In the strategy space, x ∗ =
y ∗ =
(0 . 5 , 0 . 5) is the unique Nash equilibrium of the game.
In the reduced dual space, the origin corresponds to the Nash equilibrium.
We pick a square of side length 0 .
around the origin as the set of starting points (the small red square in the middle of Figure 1)
As these starting points are evolved via MWU with step-size 0 .
02 , we take snapshots after every 1900 time steps, which are shown with colors blue, pink, lime, purple, orange and green (and then the colors repeat) respectively.
As shown in the ﬁgure, its volume (which, in two-dimensional space, is area) increases, and its shape changes from a square to a parallelogram.
C ONCLUSION AND F UTURE W ORKS
In this paper, we analyze the volume-changing behavior of several well-known learning algorithms (MWU, OMWU, FTRL) on general bimatrix games and multi-player games, which leads to a Lyapunov chaos analysis.
For bimatrix games, we do this by decomposing a game into the zero-sum part and the coordination part.
This decomposition turns the volume analysis into comparing the strengths of volume-expansion (zero-sum part) and volume-contraction (coordination part) of the
MWU dynamics.
The comparison of strengths is made via the notion of matrix domination and the use of a linear program.
For multi-player games, by the local equivalence, we show that the volume-changing behavior of MWU and OMWU are opposite to each other even in multi-player games.
We also show, for a general multi-player potential game, the key function C G is equal to a corresponding multi-player coordination game, which implies it is not positive.
We do a simulation on a simple game; the simulation matches with the theory.
Studying learning in matrix (normal-form) games, which are among one of the most classical game models, is a good theoretical starting point.
Matrix games admit mathematically amenable analyses, as demonstrated in our work and many previous works.
For future works, we are immensely interested in chaos analyses on settings that are more relevant to applications in ML, e.g. general GANs and differential games.
We believe the techniques we use (volume analysis, game decomposition, etc.) can be applicable.
A F URTHER R ELATED W ORK
Volume analysis has long been a technique of interest in the study of population and game dynamics.
It was discussed in a number of famous texts; see (Hofbauer & Sigmund, 1998, Section 11), (Fudenberg & Levine, 1998, Section 3) and (Sandholm, 2010, Chapter 9).
For a modern overview of online learning algorithms from Machine Learning or Economics perspectives, which includes the discussion about MWU and its variants, no-regret learning and potential games, we recommend the texts of Cesa-Bianchi & Lugosi (2006) and Hart & Mas-Colell (2013).
In the study of no-regret learning (e.g. Littlestone & Warmuth (1994); Freund & Schapire (1995)), a vast literature concerns general or even adversarial settings, in which the online arrivals of payoff values come with no pattern or even from an adversary.
More recently, settings where the online payoffs are more well-behaved, under the term of “predictable sequence” coined by Rakhlin & Sridharan (2013b), have been studied.
These settings include game dynamics, as the online payoffs are determined by the mixed strategy choices of the players, while these choices are updated gradually and somewhat predictably.
For these settings, online learning algorithms that perform particularly well, e.g. achieving regret bound below the canonical O ( √ T ) limit, are designed and studied (Hazan & Kale (2010); Chiang et al.
; Syrgkanis et al.
For instance, Nesterov’s excessive gap technique and optimistic mirror descent are found to achieve near-optimal regret O (log T ) in zero-sum games (Daskalakis et al.
; Rakhlin & Sridharan (2013a)), and thus the empirical average of the learning sequence converges to Nash equilibrium of the game (see Freund & Schapire (1996) for an explanation).
OMWU (with time-varying step-sizes), and more generally optimistic variant of FTRL (Rakhlin & Sridharan (2013b)), are some canonical examples of such online learning algorithms.
Recently, there is a stream of work that examines how learning algorithms behave in games or min-max optimization from a dynamical-systemic perspective.
Replicator dynamics (RD; the continuous-time analogue of MWU) and continuous-time FTRL are found to achieve optimal regret in general settings (Mertikopoulos et al.
Furthermore, RD in zero-sum games or graphical constant-sum games admits a constant of motion and preserves volume; these two properties are used to show that such dynamical systems are near-periodic (Hofbauer & Sigmund (1998);
Piliouras & Shamma (2014); Mertikopoulos et al.
; Boone & Piliouras (2019)), captured rigorously under the notion of Poincar´e recurrence (Poincar ´ e (1890); Barreira (2006)).
However, when
MWU, the forward Euler discretization of RD, is used in discrete-time setting in zero-sum games, the near-periodicity is destroyed totally; indeed, the system will never visit the same point (or its tiny neighbourhood) twice, converge to the boundary of the strategy simplex, and ﬂuctuate there irregularly (Bailey & Piliouras (2018); Cheung (2018)).
In contrast, (discrete-time) OMWU in zero-sum game is shown to converge to Nash equilibrium (Daskalakis & Panageas (2019)); yet, in the more general setting of min-max optimization, it was found that Optimistic Gradient Descent Ascent (OGDA) can have limit points other than (local) min-max solutions (Daskalakis & Panageas (2018)).
Another notion of chaos called Li-Yorke chaos was shown to exist when a variant of MWU is used in congestion games (Palaiopanos et al.
B P ROOFS IN S ECTION
Proof of Lemma 5.
First, observe that it sufﬁces to prove that the lemma holds when T 1 is a trivial matrix and T 2 is the zero matrix.
Then the lemma holds for any trivial matrices T 1 , T 2 due to symmetry:
C ( A , B ) ( p , q ) =
C ( A + T
[Equation]
B )
[Equation]
p
[Equation]
q ) =
[Equation]
( A + T 1 , B + T 2 )
[Equation]
p , q ) .
Due to the deﬁnition of trivial matrix, we can write T 1 jk = u
j + v k .
Then
[Equation]
A
[Equation] [Equation]
[Equation]
By recalling that E [ B jk − B j − B k ] = − E [ B jk ] , we have C ( A + T 1 , B ) ( p , q )
− C
( A , B ) ( p , q ) = 0 .
Proof of Observation 6.
Let P jk be the potential value of a potential game when Player 1 plays strategy j and Player 2 plays strategy k .
Then according to the deﬁnition the potential function, for any j 1 , j 2 and k ,
[Equation]
In particular, for any j, k , A jk = P jk + A 1 k − P 1 k .
This implies that there exists v such that
A jk =
P jk + v k for any j and k .
Similarly, there exists u such that B jk =
P jk + u j for any j and k .
This implies that any two-player potential games are coordination games plus trivial matrices.
Proof of Theorem 7.
We ﬁrst prove that if Z dominates C , then C ( A , B ) ( p , q ) is always nonnegative.
By Observation 8,
[Equation]
In contrast
[Equation]
if Z does not dominate C
[Equation]
then there exist ˆ j , ˆ j (cid:48) , ˆ k , ˆ k (cid:48) and δ > 0 such that
[Equation]
k
[Equation]
2 + δ.
For each η > 0 , we construct p and q such that x ˆ j ( p ) =
x ˆ j (cid:48) ( p ) =
y ˆ k ( q ) =
y ˆ k (cid:48) ( q ) =
1 − η 2 .
Furthermore, we let Υ denote the maximum absolute value of all entries in matrices A and B .
Then, for all j and k , | Z jk | ≤ Υ and | C jk | ≤ Υ .
Therefore,
[Equation]
The last inequality holds as ( C jk + C j (cid:48) k (cid:48) − C jk (cid:48) − C j (cid:48) k ) 2 − ( Z jk + Z j (cid:48) k (cid:48) − Z jk (cid:48)
− Z j (cid:48) k ) 2 ≤ 16Υ 2 .
The value of − δ (cid:0) 1 − η 2 (cid:1) 4 + | S 1 | 2 · | S 2 | 2 · η · 16Υ 2 will be negative if we pick a small enough η .
Proof of Observation 8.
Consider a random process, where j, j (cid:48) ∈ S 1 are randomly picked according to distribution x ( p ) , and k, k (cid:48) ∈ S 2 are randomly picked according to distribution y ( q ) .
Then the RHS of Observation 8 can be expressed as
[Equation]
Then we expand the squared term in the expectation.
Observing the symmetries within the expansion, we immediately have
[Equation]
Let Z j :=
[ Zy ] j and Z k =
[ Z T x ] k .
Then we have
[Equation]
Similarly, E [ Z jk Z j (cid:48) k ]
= E (cid:2) ( Z k ) 2 (cid:3) .
Lastly, E [ Z jk Z j (cid:48) k (cid:48) ]
=
E [ Z jk ] 2 .
Thus, the RHS of Observation 8 is simpliﬁed to
[Equation]
We complete the proof by noting that from the deﬁnition of C ( Z , − Z ) ( · ) in Eqn.
(4), C ( Z , − Z ) ( · ) can be rewritten as
[Equation]
while E [ Z j Z jk ] = (cid:80)
[Equation]
x j Z j (cid:80) k y k Z jk =
(cid:80) j x j Z j Z j = E (cid:2) ( Z j ) 2 (cid:3) , and similarly E [ Z j Z jk ] =
E (cid:2) ( Z k ) 2 (cid:3) .
Proof of Theorem 9.
By Lemma 2, it sufﬁces to prove ¯ c ( A , B ) ( S δ ) =
inf ( p , q )
∈
S δ C ( A , B ) ( p , q ) ≥ θ 2 δ 4 .
This holds because the matrix
Z θ -dominates C , which implies there exist j , j (cid:48) , k , and k (cid:48) such that
[Equation]
By applying Observation 8, C ( Z , − Z ) ( p , q ) ≥ C ( C , − C ) ( p , q ) + θ 2 δ 4 , because for ( p , q ) ∈ S δ , every x j ( p ) , y k ( q ) , x j (cid:48) ( p ) , y k (cid:48)
( q ) is at least δ .
By noting that C ( C , C ) ( p , q ) =
− C ( C , − C ) ( p , q ) and C ( A , B ) ( p , q ) =
C ( Z , − Z ) ( p , q ) + C ( C , C ) ( p , q ) , the result follows.
Proof of Lemma 10.
A key observation is the following equality:
[Equation] [Equation]
F
[Equation]
g , h ) , where F
[Equation]
g , h
[Equation]
Recall the notations K j = (cid:80)
k y k K jk and K k
= (cid:80) j x j K jk , and we let e := (cid:80) j,k x j y k K jk ≡
E [ K jk ] .
The equality (8) holds due to the following observations: (i) F ( g , h ) is a smooth convex function of its variables, thus all minimum points have the same function value; (ii) if ∂F∂g j and ∂F∂h k are all zero at some point ( g , h ) , then the point is a minimal point of F ;
(iii) C ( K , − K ) ( p , q ) is the variance of the random variable K jk − K j − K k (see the end of Section 2), and thus by a deﬁnition of variance,
[Equation]
(since E [ K jk
[Equation]
K
[Equation]
E [ K
[Equation]
e
[Equation]
where g # j = K j − e and h # k
= K k ; and (iv) at ( g # , h # ) , the partial derivatives stated in (ii) are all zero.
With this observation and comparing this with the deﬁnition of r ( Z ) , it’s easy to ﬁgure out that
C ( K , − K ) ( p , q ) ≤ r ( K ) 2 .
To see C ( K , − K ) ( p , q ) ≥ ( r ( K )) · δ )
, we ﬁrst let g ∗ and h ∗ to be the optimal choice of g and h in C ( K , − K ) ( p , q ) =
min g , h (cid:80) j,k x j ( p ) · y k ( q ) · ( K jk − g j − h k ) 2 .
Due to the speciﬁcation of the linear program (7), we have
[Equation]
Therefore,
[Equation]
This immediately implies that C ( K , − K ) ( p , q ) ≥ ( r ( K ) · δ ) 2 .
C M ULTI -P LAYER G AMES
Computing volume change of learning algorithm in multi-player game is slightly more involved than the two-player case.
We present a local equivalence formula of volume change between normal-form and graphical games.
This provides an intuitive procedure for understanding volume changes.
Proposition 14 shows that in multi-player game, the volume-changing behaviors of MWU and OMWU are again opposite to each other (which was shown for bimatrix game in Cheung &
Piliouras (2020)).
Graphical Games.
A graphical game Kearns et al.
(2001) is a special type of N -player game where the payoffs can be compactly represented.
In a graphical game H , for each pair of players i, k , there is an edge-game which is a bimatrix game between the two players, denoted by ( H i,k , ( H k,i ) T ) , where H i,k ∈ R n i × n k is the payoff matrix that denotes the payoffs to Player i .
Then the payoff to
Player i at strategy proﬁle s =
( s 1 , s 2 , · · · , s N ) is the sum of payoffs to Player i in all her edgegames, i.e. u i ( s )
=
(cid:80) k (cid:54) =
i H i,ks i ,s k .
As is standard, this payoff function is extended via expectation when the inputs are mixed strategies.
Here, we ﬁrst use an observation from Cheung & Piliouras (2019) to construct a family of multiplayer graphical games where MWU is Lyapunov chaotic in S N,δ :=
{ ( p 1 , · · · , p N ) |∀ i ∈
[ N ] , j ∈
S i , x ij ( p i ) ≥ δ } .
It was observed that the function C H ( p ) deﬁned in Eqn.
(5) is the sum of
C ( H i,k , ( H k,i ) T ) ( p i , p k ) of all pairs of Players i < k .
This observation yields Theorem 12.
Theorem 12.
Let G ↑ denote the family of bimatrix games which satisfy the condition either in
Theorem 9 or in Theorem 11.
In an N -player graphical game where each edge-game is drawn from
G ↑ , if all players are employing MWU with a sufﬁciently small step-size (cid:15) , then the dynamical system is Lyapunov chaotic in S N,δ with Lyapunov exponent N ( N − 1) θ 2 δ 4 (cid:15) 2 / 4 (cid:80) Ni =1 n i .
Local Equivalence of General Games and Graphical Games.
Next, we present a theorem which connects the value of C G ( p ) of a general game to C H ( p ) , where H is a graphical game.
Theorem 13.
Given an N -player normal-form game G and any point p in the cumulative payoff space, the value of C G ( p ) is the same as C H ( p ) , where H is a graphical game speciﬁed as follows:
for each pair of Players i, k and j ∈ S i , (cid:96)
∈ S k , the payoff to Player i in her edge-game
with Player k when Player i picks j and Player k picks (cid:96) is H ikj(cid:96) :
= U ikj(cid:96) , where U ikj(cid:96) is deﬁned in Eqn.
This theorem shows that for any game G , the value of C G ( p ) is the same as in a particular graphical game, where each pair of players, ( i, k ) play a bimatrix game whose utility is exactly the utility of the original game G , but taking the expectation on the randomness of the other players’ strategies.
If the original game G is a graphical game, then in the graphical game H ikj(cid:96)
=
U ikj(cid:96)
+ c − i, − k , where c − i, − k is a parameter which does not depend on Players i and k .
Theorem 13 will be used in Appendix D to show the following proposition, which shows that the volume-changing behaviors of MWU and OMWU are opposite to each other in multi-player game, generalizing a prior result in Cheung & Piliouras (2020).
Proposition 14.
The volume integrands of MWU and OMWU in a multi-player game G are respectively
1 + C G ( p ) · (cid:15) 2 + O ( (cid:15) 3 ) and 1 − C G ( p ) · (cid:15)
+ O ( (cid:15) 3 ) .
Thus, volume expands locally around a cumulative payoff point p for MWU (resp.
OMWU)
if C G ( p ) is positive (resp. negative).
Multiplayer Potential Game.
By Observation 6, we know that the volume behavior of a potential game is equivalent to a corresponding coordination game in bimatrix game.
In this section, we want to show, this holds even in the multi player setting.
Proposition 15.
Suppose P is the potential function of a potential game U .
Let U P be a game that all players will receive P ( s ) when players play strategies s .
Then C U ( p ) =
C U P ( p ) ≤ 0 .
In Appendix E, we will discuss some situations where C U ( p ) is strictly less than 0 , thus OMWU is Lyapunov chaotic therein.
D L OCAL E QUIVALENCE OF V OLUME C HANGE BETWEEN N ORMAL - FORM AND G RAPHICAL G AMES
Here, we concern the volume change of a learning algorithm in multi-player game.
We ﬁrst recap from Cheung & Piliouras (2020) on how the volume change is computed for dynamical systems which are gradual (i.e. those governed by a small step-size), followed by a continuous-time analogue of OMWU in games, which are crucial for analyzing the volume change of discrete-time OMWU.
Then we compute the volume changes of MWU and OMWU in multi-player graphical games and normal-form games respectively.
Once these are done, the proofs of Proposition 14 and Theorembecome apparent.
D. D ISCRETE -T IME D YNAMICAL S YSTEMS AND V OLUME OF F LOW
We consider discrete-time dynamical systems in R d .
Such a dynamical system is determined recursively by a starting point s (0)
∈ R d and an update rule of the form s ( t + 1) =
G ( s ( t )) , for some function G : R d → R d .
Here, we focus on the special case when the update rule is gradual , i.e. it is in the form of
[Equation]
where F : R d → R d is a smooth function and step-size (cid:15) > 0 .
When F and (cid:15) are given, the ﬂow of the starting point s (0) at time t , denoted by Φ( t, s (0)) , is simply the point s ( t ) generated by the above recursive update rule.
Then the ﬂow of a set S ⊂ R d at time t , denoted by Φ( t, S ) , is the set { Φ( t, s ) | s ∈ S } .
Since F does not depend on time t , we have the following equality: Φ( t 1 + t 2 , S ) =
Φ( t 2 , Φ( t 1 , S )) .
By equipping R d with the standard Lebesgue measure, the volume of a measurable set S , denoted by vol ( S ) , is simply its measure.
Given a bounded and measurable set S ⊂ R d , if the discrete ﬂow in one time step maps S to S (cid:48)
= Φ(1 , S ) injectively, then by integration by substitution for multivariables,
[Equation]
where I is the identity matrix, and J ( s ) is the Jacobian matrix deﬁned below:
[Equation]
Clearly, analyzing the determinant in the integrand in Eqn.
(9) is crucial in volume analysis; we call it the volume integrand .
When the determinant is expanded using the Leibniz formula, it becomes a polynomial of (cid:15) , in the form of 1 + C ( s ) · (cid:15) h + O ( (cid:15) h +1 ) for some integer h ≥ 1 .
Thus, when the step-size (cid:15) is sufﬁciently small, the sign of C ( s ) dictates on whether the volume expands or contracts.
D. C ONTINUOUS -T IME A NALOGUE OF OMWU
OMWU does not fall into the category of dynamical systems deﬁned above, since its update rule is in the form of s ( t + 1)
=
G ( s ( t ) , s ( t − 1)) .
Fortunately, Cheung and Piliouras Cheung &
Piliouras (2020) showed that OMWU can be well-approximated by the online Euler discretization of a system of ordinary differential equations (ODE), and thus it can be well-approximated by a dynamical system.
The ODE system is given below.
p is a dual (cumulative payoff) vector variable, u : R + → R d is the function such that u ( t ) gives the instantaneous payoff vector at time t .
We assume that u is twice differentiable with bounded second-derivatives, and ˙ u denotes the time-derivative of u .
[Equation]
Online Euler discretization (OED) of Eqn.
(11) refers to the following time-discretization of the
ODE system.
In applications, ˙ u might not be explicitly given, and the sequence u (0) , u (1) , u (2) , · · · are available online (i.e., at time t we only have access of u ( τ ) for τ = 0 , 1 , · · · , t ).
As the discretization step is (cid:15) , we approximate ˙ u ( t ) by ( u ( t ) − u ( t − 1)) /(cid:15) .
By using this approximation,
OED of Eqn.
(11) yields
[Equation]
which is exactly the OMWU update rule in general context.
When compared the OED with the standard Euler discretization
[Equation]
OED incurs a local error that appears due to the approximation of ˙ u ( t ) .
The local error can be bounded by O ( (cid:15) 3 ) .
Cheung and Piliouras Cheung & Piliouras (2020) showed that eventually the determinant of the volume integrand is a of the form 1 + C ( s ) · (cid:15) 2 + O ( (cid:15) 3 ) , the local error does not affect the ﬁrst and second highest-order terms, and hence can be ignored henceforth.
D. MWU IN G RAPHICAL G AMES
Let H be a graphical game of N players, where between every pair of Players i and k , the payoff bimatrices are ( H ik , ( H ki ) T ) .
In the cumulative payoff space, let p =
( p 1 , · · · , p N ) denote the cumulative payoff proﬁle, and let x = ( x 1 , · · · , x N ) denote the corresponding mixed strategy proﬁle, where x i is a function of p i .
We will write x i and x i ( p i ) interchangeably.
The expected payoff to strategy j of Player i is
[Equation]
which will be used to compute the Jacobian matrices of MWU and OMWU.
For MWU, the Jacobian matrix J is a squared matrix with each row and each column indexed by ( i, j ) , where i is a Player and j ∈ S i .
The precise values of its entries are given below:
[Equation]
(12) and ∀ i (cid:54) =
k
[Equation]
Then by expansion using Leibniz formula
[Equation]
the determinant of ( I + (cid:15) · J ) is
[Equation]
By noting the similarity of the double summation to C ( A , B ) ( · ) in Eqn.
[Equation]
we can immediately rewrite the above expression as
[Equation]
D. OMWU IN G RAPHICAL G AMES
For OMWU, as we pointed out already, we will ﬁrst consider its continuous analogue ﬁrst.
Thus, we need to compute ˙
u in the continuous-time setting.
By chain rule, we have
[Equation]
and hence
[Equation]
Note that this is a recurrence formulae for d p d t .
By iterating it 9 , we have d p ij
[Equation]
Hence, its standard Euler discretization, which approximates the OED with local error O ( (cid:15) 3 ) , can be written as below (where we ignore the O ( (cid:15) 3 ) error terms):
[Equation]
With this, we are ready to compute the Jacobian matrix J for OMWU.
For all j 1 , j 2 ∈ S i ,
[Equation]
16) and for all i (cid:54)
=
k , j ∈ S i , (cid:96) ∈
S k ,
[Equation]
Then by expansion using Leibniz formula, the determinant of ( I + (cid:15) · J ) is
[Equation]
By a direct expansions on T 1 and T 2 , it is easy to see that T 1 = 2 T 2 (after ignoring O ( (cid:15) 3 ) terms).
On the other hand, the coefﬁcient of (cid:15) 2 in T 2 is exactly the same as the double summation in Eqn.
(14), thus it equals to − (cid:80) i,k :1 ≤ i<k ≤ N C
( H ik , ( H ki ) T ) ( p i , p k ) .
Overall, we show that the determinant equals to
[Equation]
Observation 16.
The coefﬁcient of (cid:15) 2 in Eqn.
(18) is the exact negation of the coefﬁcient of (cid:15) 2 in
Eqn.
D. C OMPLETING THE L OCAL E QUIVALENCE P ROOF
In a multiplayer normal-form game G , recall that notation Eqn.
We point out the following formulae:
[Equation]
MWU.
Here, MWU update rule is p ij ( t + 1) =
p ij ( t ) + (cid:15) · U ij .
When computing the Jacobian matrix for this update rule using the formulae above, and comparing it with the Jacobian matrix computed in Eqn.
(12) and Eqn.
(13), it is immediate that they are the same by setting H ikj(cid:96)
=
U ikj(cid:96) .
This derives Eqn.
(5), and completes the proof of Theorem 13.
OMWU.
As before, we use the continuous analogue and compute ˙ u .
By the chain rule and the above formulae, we have
[Equation]
and hence
[Equation]
Iterating the above recurrence yields
[Equation]
Its standard Euler discretization is
[Equation]
Now we compute the Jacobian matrix for this standard Euler discretization.
For j 1 , j 2 ∈ S i ,
[Equation]
and for all i (cid:54)
=
k , j ∈ S i , (cid:96) ∈ S k , (cid:15)J ( i,j ) , ( k,(cid:96) ) =
(cid:15)x
[Equation]
By comparing this computed Jacobian matrix with the Jacobian matrix computed in Eqn.
(16) and
Eqn.
(17), it is immediate to see that their determinants are the same (after ignoring all O ( (cid:15) 3 ) terms) by setting H ikj(cid:96) =
U ikj(cid:96) .
With the result we just derived, together with Observation 16 and Theorem 13, Proposition 14 follows.
E M ULTI - PLAYER P OTENTIAL G AME
Proof of Proposition 15.
We know that the potential game satisﬁes the following condition:
[Equation]
Therefore, u i ( s i , s − i ) =
P ( s i , s − i )
+ v i ( s − i ) .
Note that v i ( s − i ) does not depend on s i , the strategy of player i .
By Theorem 13, let H ( U ) be the induced graphical game of U and H ( U P ) be the induced graphical game of U P .
Then,
[Equation]
The third equality holds as the difference between H ( U ) ik and H ( U P ) ik is a trivial matrix:
[Equation]
where 10 E − ( i,k ) (cid:2) v i ( s − i ) (cid:3) doesn’t depend on j , the strategy of player i , and only depends on l , the strategy of player k .
The same argument applies for ( H ( U ) ki ) T and ( H ( U P ) ki ) T .
To see C U ( p ) ≤ 0 , observe that the induced graphical game of U P between player i and k , ( H ( U P )
ik , ( H ( U P ) ki ) T ) , is also a bimatrix coordination game, which implies
C ( H ( U P ) ik , ( H ( U P ) ki ) T ) ( · ) ≤ 0 .
As C U ( p ) =
(cid:80) i,k C
( H ( U P )
ik , ( H ( U P ) ki ) T ) ( p i , p k ) , the result follows.
Next, we identify several cases such that C U ( p ) is strictly negative in the region
[Equation]
The conditions we pose are on the corresponding potential function P .
Note that H ( U P ) ik , the induced edge-game between player i and k , is also a coordination game, i.e. H ( U P )
ik =
( H ( U P ) ki ) T .
• Case 1 :
[Equation]
where P ikj(cid:96)
=
E s − ( i,k ) (cid:2) P ( s i =
j, s k = (cid:96), s − ( i,k ) ) (cid:3) .
With this condition, we can prove that
C U ( p ) ≤ − θδ 2 for any p in S δ .
One key observation for this is true is that
[Equation]
as
[Equation]
• Case 2 :
If U is a graphical game, then if there exists a pair of player i 1 and i 2 , such that the game between i
and i 2 is a non-trivial game, then
C U will be strictly negative in S δ .
• Case 3 : Consider the payoff matrix of U P , the coordination game, between players i
and i 2 given a strategy proﬁle of the other players.
There are total (cid:81) i (cid:54)
=
i 1 ,i 2 n i such matrices, one for each strategy proﬁle of the other players, and each matrix is of dimension n i 1 × n i 2 .
We call these matrices the projected matrices for players i 1 , i 2 .
Let M denote the matrix space of n i 1 × n i 2 .
On the other hand, trivial matrices form a subspace of dimension n i 1 + n i 2 − 1 .
11 Let’s call this the trivial space , denoted by T .
We consider the direct decomposition M = T ⊕ V .
Let a set of bases of M be
B 1 , B 2 , B 3 , · · · , B n i 1 n i 2 , where the ﬁrst n i 1 + n i 2 − 1 bases form a basis of T , and the remaining bases form a basis of V .
Without loss of generality, we assume that all bases are of L 2 norm 1 .
Given the above-mentioned bases of M , each of the projected matrices can be written into a unique linear combination of these bases.
Now, suppose there is a base B (cid:96) for l ≥ n i 1 + n i 2 (i.e. this base is in the set of bases for V ), such that all projected matrices have non-positive (or non-negative) coefﬁcients of this base, and at least one of these projected matrices (which we call a special projected matrix ) has strictly negative (or strictly positive) coefﬁcient of the base.
Then we claim that C ( H ( U P )
i 1 i 2 , ( H ( U P ) i 2 i 1 ) T ) ( p i 1 , p i 2 ) will be strictly negative in S δ .
This is because H ( U P ) i 1 i 2 is a convex combination of all those projected matrices, and by our assumption above, when H ( U P ) i 1 i 2 is expressed as the linear combinations of the bases of M , the coefﬁcient of B (cid:96) is strictly negative (or strictly positive), thus H ( U P )
i 1 i 2 cannot be a trivial matrix.
Suppose further that there exists θ > 0 such that a special projected matrix has negative (or positive) coefﬁcient for B (cid:96) which is smaller (or bigger) than − θ (or θ ), then we are guaranteed that H ( U P ) i 1 i 2 is bounded away from T for a distance of θδ N − 2 , 13 and hence as the calculations below show, C ( H ( U P )
i 1 i 2 , ( H ( U P ) i 2 i 1 ) T ) ( p i 1 , p i 2 ) ≤ − θ
δ 2 N − 2 .
If there exists a pair of player i 1 and i 2 such that this condition holds, then C U ≤ − θ
δ 2 N − 2 .
[Equation]
where { g ∗ j + h ∗ k } jk is projection of H ( U P )
i 1 i 2 on the trivial space.
The ﬁrst inequality follows as p ∈ S δ ; the second equality holds as the projection minimizing the distance to the trivial space, and the ﬁnal inequality comes from the distance from H ( U P )
i 1 i 2 to the trivial space.
For all these cases, we can have OMWU is C U to be strictly negative in domain S δ , which implies OMWU is Lyapunov chaotic in S δ .
j for all j, k , and hence the kernel of L must be the span of the vector (1 , 1 , · · · , 1 (cid:124) (cid:123)(cid:122) (cid:125) the u part , − 1 , − 1 , · · · , − 1 (cid:124) (cid:123)(cid:122) (cid:125) the v part ) .
Thus, the dimension of all trivial matrices is the dimension of the domain of L , which is n i 1 + n i 2 , minus the dimension of the kernel of L .
12 Here, the norm is deﬁned w.r.t.
the standard Frobenius matrix inner product.
To see why, when the coefﬁcient for B (cid:96) is bounded away from zero, we are guaranteed that the special projected matrix has a strictly positive distance from T , and this distance is at least θ .
Then H ( U P ) i 1 i 2 , which is a convex combination of all projected matrices where each projected matrix (in particular, the special projected matrix) has a weight at least δ N − 2 , has a strictly positive distance from T too, which is at least θδ N − 2 .