T HE EVOLUTION OF UNCERTAINTY OF LEARNING IN GAMES
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning-in-games has become an object of intense interest for ML due to its connections to numerous AI architectures.
We study standard online learning in games but from a non-standard perspective.
Instead of studying the behavior of a single initial condition and whether it converges to equilibrium or not, we study the behavior of a probability distribution/measure over a set of initial conditions .
This initial uncertainty is well motivated both from a standard game-theoretic perspective (e.g. a modeler’s uncertainty about the agents’ initial beliefs, random external signals) as well as from a ML one (e.g. noisy measurements, system initialization from a dataset distribution).
Despite this, little is formally known about whether and under what conditions uncertainty is ampliﬁed or reduced in these systems.
We use the popular measure of differential entropy to quantify the evolution of uncertainty.
We ﬁnd that such analysis shares an intimate relationship with volume analysis, a technique which was recently used to demonstrate the occurrence of Lyapunov chaos when using Multiplicative Weights Update (MWU) or Followthe-Regularized-Leader (FTRL) algorithms in zero-sum games.
This allows us to show that the differential entropy of these learning-in-game systems increases linearly with time, formalizing their increased unpredictability over time.
We showcase the power of the framework by applying it in the study of multiple related systems, including different standard online optimization algorithms in numerous games and dynamics of evolutionary game theory.
I NTRODUCTION
A primary goal of ML research is to understand the behaviors of learning algorithms in various settings.
One standard approach is from each initial condition, we determine whether a learning algorithm converges to a local optimum or stable state.
Yet, in the context of online learning in games, and more generally in distributed multi-agent learning, it is natural to consider the evolution of a probability distribution over initial conditions instead.
In these settings, each agent forms an initial belief on her own, and she typically does not reveal her belief to other agents or external modelers/analysts.
For a modeler to understand the behaviors of the system while being uncertain of the agents’ beliefs, one natural approach is to infer how the initialization distributes by using data of observations from the past.
The modeler then use this inference to predict the likelihoods of different outcomes in the future, either by simulation or by analysis.
Also, random initialization can happen due to random external signals, e.g. weather, as well as noisy measurements.
For readers who want to know more intuitions and mathematical aspects of such models, see Appendix A.
In such cases, it is critical to understand whether the initial probability distribution evolves toward stability, or if its uncertainty gets ampliﬁed.
Such analysis provides insight into the learning system’s robustness against random initialization and environment perturbations.
If a coordinator of the system desires stability but the analysis shows the uncertainty gets ampliﬁed, she ought to coordinate with the agents to make changes, e.g. encourage them to use other online learning algorithms.
To analyze how uncertainty evolves, we need a measure of it.
A natural choice is entropy .
In his seminal work in 1948, Claude Shannon formulated an axiomatic foundation of information theory and introduced the seminal deﬁnition of Shannon entropy (SE): given a discrete random variable with possible outcomes x 1 , . . .
, x n which occur with probability p 1 , . . . , p n , its SE is − (cid:80) ni
=1 p i log p i
=
E [log(1 /p i )] .
Entropy is the canonical measure of uncertainty: when p i =
for some i , the distribution is considered certain and its entropy is zero; uniform distribution is viewed as the most uncertain, and it attains the maximum possible entropy of log n .
For continuous random variables with probability density function g , Shannon proposed an extension of SE called the differential entropy (DE), deﬁned as E [log(1 /g ( x ))]
= − (cid:82) X g ( x ) log g ( x ) d x , where X is the support set of the random variable.
We will analyze how DE evolves in multi-agent learning.
Our Contributions.
In our model, a learning-in-game system starts with a distribution over a set of initial conditions in the cumulative payoff space , a coordinate system which is inherent in the implementations of many online learning algorithms.
The initial distribution evolves over time depending on the combination of agents’ learning algorithms and the underlying game.
We focus on two popular learning algorithms, Multiplicative Weights Update (MWU) and its optimistic variant (OMWU).
The game settings include the standard two-player matrix games, and one-population games which are fundamental in biological evolution models.
We show that the DE of a broad range of learning-in-game systems increases linearly with time (up to a certain limit), formalizing their increased unpredictability.
Such systems include MWU in zero-sum games, OMWU in coordination games, and MWU in certain one-population games which reward more to intra-species play than to inter-species play.
Our results apply to any smooth initial distribution with bounded support.
At this point one may naturally wonder: What level of uncertainty does a linear increase of DE indicate?
What are its implications?
To answer these questions, it is best to compare ourselves against other simple benchmarks.
Consider the following simple learning system: in each round, the payoffs for each action are generated from a uniform distribution over [ − 1 , 1] .
In the cumulative payoff space, the distribution at time T converges to a multivariate Gaussian distribution with variance Θ( T ) , and hence the entropy grows at a rate of O (log T ) , much slower than linear growth.
In Section 4.1, we present an implication of linear DE growth in learning-in-game systems.
Brieﬂy speaking, the DE cannot increase substantially for an indeﬁnite amount of time in such systems.
Thus, the distribution after a sufﬁciently long time must concentrate in a region that yields slower
DE growth or decline; such region does not cover any point in the cumulative payoff space that corresponds to an interior Nash equilibrium.
We will refer to this phenomenon by “the grand escape”.
For the implications in information theory, we refer readers to Chapter 8 of Cover & Thomas (2006).
The central tool in analyzing the changes of DE is the Jacobian matrix of our multi-agent dynamical system.
This is also the key notion in volume analysis , which was used in a recent series of papers to demonstrate Lyapunov chaos in learning-in-game systems.
(Cheung & Piliouras (2019; 2020); Cheung & Tao (2020))
By showing that the determinant of the Jacobian matrix is strictly above 1 in a large domain of the cumulative payoff space, they showed that the volume (Lebesgue measure) of a small set of initial conditions will blow up exponentially to become a set of large diameter, thus demonstrating a general Lyapunov chaos phenomenon in learning-in-game.
Indeed, the same property of Jacobian matrix guarantees linear increase of DE.
Additionally, we present the ﬁrst Jacobian analysis of online learning in one-population games.
Finally, our DE analysis is robust against small perturbations.
Consequently, our results extend to the setting where the game in each round can be different, as long as the payoffs are perturbations of each other.
Such settings capture many games in the real world, where we know the “reference values” of the payoffs, but the accurate payoffs in each round differ slightly due to unknown external (random or adversarial) factors.
Our model and analysis can be viewed as an a strengthening of volume analysis.
Volume analysis focuses on whether it is possible to reach a state or not, whereas in our model we are also concerned with how probable/likely it is to reach a state.
More explicitly, a state can be reached but only with a tiny chance.
When studying chaos and volume, such states matter, but less so when studying uncertainty as their contributions to entropy will be low.
To illuminate this, we simulate MWU (with step-size 0 . 01 ) in Matching Pennies game, and present a few plots in Figure 2.
The top-left plot shows the initial set, which is a small square around the unique Nash equilibrium.
The top-right plot shows the range of possibility after 40000 steps (a similar ﬁgure was ﬁrst presented by Cheung & Piliouras (2019)).
In our model, we assume the initial distribution is uniform over the small square, and plot the heat-map of probability densities after 40000 steps (bottom-left).
We observe that the states in the boundary of the vortex are more probable to occur, while the densities around the Nash equilibrium decline.
The bottom-right plot shows the densities that generate the heat-map.
Further Related Work.
Learning models that explicitly study the effects random initialization have received relatively little attention in game theory where static equilibrium concepts are typically
[Figure]
the preferred solution.
Lahkar & Seymour (2013) studies online learning under random initialization in population dynamics, where one population game is a common payoff model.
In population dynamics, there exists a large population of agents who have different initial choices of mixed strategies, modeled as a distribution.
The dynamics proceed by randomly pairing up agents in each round to play a game.
Lahkar & Seymour (2014) extends this framework to other learning dynamics to describe the evolution of a distribution of mixed strategies in a population playing variants of RockPaper-Scissors games establishing local convergence to the interior equilibrium.
Recently, further learning models inspired by Q-learning dynamics have been deﬁned (Hu et al., 2019).
Very little is formally known about the behavior of such models particularly under instability.
For zero-sum games with an interior Nash equilibrium, “the grand escape” phenomenon follows from Bailey & Piliouras (2018); Cheung (2018), who showed that MWU diverges away from Nash.
Our results are more general, since their analysis only works with those zero-sum game dynamics, while our technique applies to many other game dynamics as well.
Such instability results in zerosum games are undesirable for ML applications such as GANs thus a lot of effort has been invested in developing algorithms that achieve convergence Daskalakis et al.
; Mertikopoulos et al.
(2019); Daskalakis & Panageas (2018); Gidel et al.
; Mokhtari et al.
; Perolat et al.
Our research direction here is in some sense orthogonal.
We aim to characterize in a more detailed fashion the behavior of learning dynamics despite their instability.
The instability of MWU, FTRL and other optimization-driven dynamics in games has attracted a lot of attention in recent years.
One formal technique to establish the unpredictability of MWU and other dynamics is based on proving Li-Yorke chaos, which has been established robustly in different families of routing/potential/Cournot games (Palaiopanos et al.
(2017); Chotibut et al.
Bielawski et al.
(2021); Cheung et al.
The techniques in those papers are independent from ours as they require the identiﬁcation of one dimensional invariant manifolds, and Li-Yorke chaos implies the existence of periodic orbits, which is not possible in our systems.
In a very recent series of works, Flokas et al.
Giannou et al.
(2021) established that all (even partially) mixed equilibria in all games are not asymptotically stable for any choice for FTRL dynamics both in discrete as well as in the idealized continuous-time limit.
In fact, it is possible to construct simple matrix games such that the orbits of the resulting dynamics can be arbitrarily complex Andrade et al.
If, inspired by ML applications, we allow for more complex differentiable games it becomes even simpler to establish strong instability results for effectively any type of training algorithms (Letcher (2020);
Balduzzi et al.
The sweeping nature of these strong negative results showcases the practical interest in uncovering more information about the nature of unstable dynamics of games.
Our proposed framework introduces a novel, quantitative methodology for their study.
The notion of differential entropy (DE) has been used as uncertainty measure in many works across multiple disciplines.
We note that while the extended deﬁnition of DE seems natural, it is not considered as the perfect generalization of SE since it misses some favorable properties, and therefore, as quoted from the popular information theory text of Cover & Thomas (2006), “there is need for some care in using the concept”.
Nevertheless, DE remains as an effective measure of uncertainty 3 , which is commonly used in physics, economics, control theory, engineering, biology, medical research and beyond.
For a general discussion, we refer readers to the text of Cover &
Thomas (2006).
For background information of evolutionary game theory, we recommend Hofbauer & Sigmund (1998);
Sandholm (2010); Vega-Redondo (1996).
M ODEL
We use bold small letters to denote vectors, and bold capital letters to denote matrices.
Let ∆ d denote the probability simplex of dimension d : ∆ d := { x ∈ R d : for each j, x j ≥ 0 , and (cid:80) dj
=1 x j = 1 } .
MWU and OMWU.
When an agent uses MWU, she wants to choose among d ≥ 2 pure actions based on the past cumulative payoffs to these actions.
The process starts at discrete time t = 0 , where the agent chooses an initial cumulative payoff vector p 0 ∈ R d .
At any time t ≥ 1 , upon receiving the payoffs to the each of the d actions at time t − 1 , which are denoted by a vector r t − 1 ∈ R d , the agent updates the cumulative payoff vector by the rule ∀ j = 1 , 2 , . . .
, d, p tj ← p t − 1 j + (cid:15) · r t −
[Equation]
where (cid:15) > 0 is the step-size of the update.
At any time t ≥ 0 , the agent chooses randomly among the d pure actions depending on the value of p t .
Precisely, the agent chooses a mixed strategy x t ∈ ∆ d which is a function of p t .For
MWU, x t is determined by the rule
[Equation]
For OMWU, we start with p
= p 1 , and for t ≥ 2 ,
[Equation]
and x t is determined by the same rule (2).
In general, the payoff vectors r t − 1 can be arbitrary, and they may or may not depend on the mixed strategy x
t − 1 .
However, in the context of learning-in-game, the underlying game generates the payoff vectors that depend on the mixed strategies of the agents involved.
We discuss two popular game models relevant to our work, namely two-player matrix games and one-population games .
Two-Player Matrix Games.
In a two-player matrix game, Player 1 has n actions and Playerhas m actions.
The game is speciﬁed by two matrices A , B ∈ [ − 1 , 1] n × m .
When Player 1 chooses action j and Player 2 chooses action k , A jk , B jk are the payoffs to Players 1 and 2 respectively.
When the players choose mixed strategies, the payoffs are extended via expectation.
Denote the mixed strategies of Players 1 and 2 by x ∈ ∆ n and y ∈ ∆ m .
The payoff to action j of Player 1 is [ Ay ] j , while the payoff to action k of Player 2 is [ B T x ] k .
When both players use MWU or OMWU to play this game repeatedly, we have the following discrete-time dynamical system, where p t , q t denote the cumulative payoff vectors at time t of Players 1 and 2 respectively.
The two players choose initial cumulative payoff vectors p 0 , q
[Equation]
[Equation]
Note that x t − 1 is a function of p t − 1 as speciﬁed in (2); similarly, y t − 1 is a function of q t − 1 .
Thus, we can also view (4) as an iterated function that maps from ( p t − 1 , q t − 1 ) to ( p t , q t ) .
In the above setting, the game ( A , B ) is the same at all times t .
It is also interesting to consider scenarios where the game varies in each round.
Our results extend to settings where there is a “reference game” ( A , B ) , but at each round the actual game being played is a perturbation of the reference game.
Precisely, the game at time t is ( A t , B t ) =
( A , B ) + ( ∆ tA , ∆ tB ) , where ∆ tA , ∆ tB are matrices with maximum absolute entry at most β , for some small β > 0 .
One-Population Games.
One-population games is a fundamental model in mathematical biology, which is widely used to explain the evolution of a population of different of species over time.
An one-population game is similar to a two-player game, but now Player 1 is playing against herself; there is only one player (or in biology term, one population) in this game, while a mixed strategy x t represents the fraction of different species among the population.
An one-population game is speciﬁed by a matrix A ∈ R n × n .
When the mixed strategy of the player is x ∈ ∆ n , the payoff to action j is [ Ax ] j .
The resultant discrete-time dynamical system with MWU is
[Equation]
(5) where x t − 1 is a function of p t − 1 as in (2).
Random Initial Cumulative Payoff Vectors.
In Section 1, we explained why we consider uncertainty of the initial cumulative payoff vectors.
This can be captured by a model where these vectors follow a distribution.
We use two-player game for discussion below; for one-population game it is similar.
Assume that the initial vectors ( p 0 , q 0 ) follow a joint probability distribution in the domain R n × R m .
As the initial vectors are random and they are updated according to the discrete-time dynamical system (4), for any t ≥ 0 , ( p t , q t ) also follows a distribution, which is the main object we analyze in this work.
We are interested in knowing certain statistics of uncertainty of this distribution, say differential entropy, which is the subject of the next section.
D IFFERENTIAL E NTROPY
In this section, we discuss how changes of differential entropy (DE) can be analyzed in general, and provide some intuition of the relationship between DE and volume.
To proceed, we need a crucial notion called the Jacobian matrix .
Let f : R d → R d be a smooth function.
Its Jacobian matrix is
[Equation]
Let X be a continuous random variable in R d .
Let g be its probability density function, whose support set is X .
The DE of X is h ( X ) :
= − (cid:82) X g ( x ) log g ( x ) d x .
For any function f : R d → R d , f ( X ) is another random variable; we denote its probability density function by ˆ g , and its support set by Y .
If f is a diffeomorphism, it is known that ˆ g ( y ) =
g ( f
[Equation]
Using multivariate integration by substitution, the DE of f ( X ) can be computed as follows:
[Equation]
By the Inverse Function Theorem, (cid:12)(cid:12) det J f
[Equation]
1 ( f ( x )) (cid:12)(cid:12) · | det J f ( x ) | = 1 .
We have (Duvenaud et al.
[Equation] [Equation]
Thus, the determinant of the Jacobian matrix directly affects the change of DE.
In particular, if there exists α > 0 such that | det J f ( x ) | ≥ 1+ α for all x ∈ X , then h ( f ( X )) − h ( X ) ≥ log(1+ α ) > 0 .
In order to use (6) to analyze the changes of DE of learning in two-player matrix games, we view the dynamical system (4) as a function that maps from ( p t − 1 , q t − 1 ) to ( p t , q t ) in each round t ≥ 1 .
If the change of DE per round is at least log(1 + α ) > 0 , then the DE increases linearly with time.
The Relationship Between DE and Volume.
Before discussing the relationship, we ﬁrst recap what is volume.
Let X ⊂ R d be a measurable set with positive Lebesgue measure (which is sometimes called the volume of the set), and let f : R d → R d be a diffeomorphism.
The volume of f ( X ) is (cid:82) X | det J f ( x ) | d x .
When | det J f ( x ) | ≥ 1 + α for all x ∈ X , the volume increases by a factor of at least (1+ α ) .
If this occurs in a dynamical system every round, the volume of the initial set grows exponentially with time, and hence also its diameter, establishing that Lyapunov chaos occurs.
The occurrence of the Jacobian in both the formulae for computing volume and the change of
DE in (6) is not a coincidence, but with a simple intuition behind.
The Jacobian yields a linear approximation of f locally: for any x ∈ R d and any small perturbation vector ∆ x ∈ R d , f ( x + ∆ x ) ≈ f
( x ) + J f ( x ) · ∆ x .
We consider a tiny hypercube with volume v around x , and its image under the function f .
By the linear approximation, the image is approximately a parallelotope (the high-dimensional analogue of parallelogram), whose volume is known to be | det J f ( x ) | · v .
By partitioning X into inﬁnitesimal tiny hypercubes and summing up, this is intuitively how the integral formula for computing volume is obtained.
Next, suppose the probability density at x is g ( x ) .
The probability of the tiny hypercube is approximately g ( x ) · v .
After the transformation f , this amount of probability is spread approximately uniformly in the parallelotope, thus the new density at f ( x ) is g ( x ) / | det J f ( x ) | .
If the volume increases locally at x , i.e. | det J f ( x ) | > 1 , it intuitively means the uncertainty level increases locally.
This intuition is captured by DE: the contribution to DE by the tiny hypercube is − g ( x ) · v log( g ( x ) · v ) , while the contribution to DE by the parallelotope is − g ( x ) ·
v log g ( x ) · v | det J f ( x ) | .
Thus, the local change of DE is g ( x ) · v log | det J f ( x ) | , which has the same sign as ( | det J f ( x ) | − 1) .
To summarize, local volume increase is equivalent local DE increase .
This equivalence allows us to spot that for those learning systems in two-player matrix games studied in the series of volume analysis papers of Cheung & Piliouras (2019; 2020);
Cheung &
Tao (2020), their DE with our model increases linearly in a large domain of the cumulative payoff space.
We present this result formally Section 4.1, and present an interesting consequence of the DE growth.
Partly motivated by Lahkar & Seymour (2013), we also transfer the techniques to analyzing one-population games in Section 4.2, and we present two sufﬁcient conditions on these games that leads to linear DE growth with MWU.
As ﬁrst spotted by Cheung & Piliouras (2019), these results not only cover MWU and OMWU, but also the broad family of FTRL algorithms, since the properties of the Jacobian for FTRL is very similar to that of MWU.
(This is not surprising since FTRL is a generalization of MWU.)
A PPLICATIONS
In this section, we consider two applications: MWU in two-player zero-sum games and onepopulation games.
In Appendix B, we also consider OMWU dynamics in two-player coordination games and one-population games, and the settings where the game in each round is perturbed.
A two-player game ( A , B ) is zero-sum if A = − B .
We focus on non-trivial games, deﬁned below.
Deﬁnition 4.1 (Cheung & Piliouras (2019))
A zero-sum game ( A , − A ) is trivial if there exists real numbers a 1 , a 2 , · · · , a n and b 1 , b 2 , · · · , b m such that A jk = a j + b k .
The same deﬁnition applies for one-population game A .
Cheung & Piliouras pointed out that a trivial game is not interesting as the players have dominant strategies.
They provided a measure of distance of a zero-sum game ( A , − A ) from triviality by
[Equation]
4. MWU IN TWO - PLAYER ZERO - SUM GAME
Recall from (4) that in zero-sum game ( A , − A ) , MWU dynamic in the cumulative payoff space is p t +
=
p t + (cid:15) ·
Ay ( q t ) and q t +1 = q t + (cid:15) · ( − A ) (cid:62) x ( p t ) where x ( p t ) and y ( q t ) are the mixed strategies of two players at time t : x j ( p t ) =
exp( p tj ) / (cid:80) j (cid:48) exp( p tj (cid:48) ) and y k ( q t ) = exp( q tk ) / (cid:80) k (cid:48) exp( q tk (cid:48) ) .
Two different regions are considered: the interior region S 1 :=
{ ( p , q ) | x ( p ) ≥ δ and y ( q ) ≥ δ } , and the boundary region S 2 , which is the complement to S 1 .
Note that if the game possesses an interior Nash equilibrium, then the equilibrium lies in S 1 for any sufﬁciently small δ .
Let g t ( p , q ) denote the probability density of ( p , q ) at round t .
Recall from 6 that the change of entropy is (cid:82) g t ( p , q ) log | det J f ( p , q ) | d ( p , q ) .
If g t ( p , q ) is fully supported by S 1 , then by using the Jacobian analysis of Cheung & Piliouras (2019), who showed that the determinant of the
Jacobian matrix is strictly larger than 1 in S 1 , we show the differential entropy increases linearly.
Theorem 4.2 For any non-trivial two-player zero-sum game ( A , − A ) , with MWU dynamics of step-size (cid:15) ≤ min { 1 / (32 n 2 m 2 ) , δ 2 c ( A ) 2 / 8 } , if g t ( p , q ) > 0 only for ( p , q ) ∈ S 1 , then the differential entropy will increase by at least δ 2 c ( A ) 2 · (cid:15) 2 / 8 1+ δ 2 c ( A ) 2 · (cid:15) 2 / 8 in round t .
Proof: For MWU dynamics, f ( p , q ) =
( p + (cid:15)
Ay ( q ) , q − (cid:15)
A (cid:62) x ( p )) by (2).
Cheung & Piliouras (2019) showed that if (cid:15) ≤ min { 1 / (32 n 2 m 2 ) , δ 2 c ( A ) 2 / 8 } , then det J f ( p , q ) ≥ 1 + δ 2 c ( A ) 2·
[Equation]
for any ( p , q ) ∈ S 1 , where c ( A ) is the measure of the distance of a zero-sum game ( A , − A ) from triviality (7).
As log(1 + r ) ≥ r 1+ r for any r > − 1 ,
[Equation]
Actually, the assumption that g t ( p , q ) is fully supported in S 1 can be weakened.
We show that if a sufﬁciently large probability lies in S 1 , then the differential entropy still increases linearly.
With this result, we can show Theorem 4.5 below, which states that there exists inﬁnitely many rounds, such that in each of these rounds t , the probability of ( p t , q t ) ∈ S 2 will be at least some constant.
Theorem 4.
If (cid:82) S 1 g ( p , q ) d ( p , q ) ≥ λ , then the entropy will increase by at least δ 2 c ( A ) 2 · (cid:15) 2 / 8 1+ δ 2 c ( A ) 2 · (cid:15) 2 / 8 · λ − (cid:15) 3 1 − (cid:15) 3 · (1 − λ ) in round t , which is strictly positive whenever λ >
[Equation]
[Equation] [Equation]
Proof: Cheung & Piliouras (2019) showed that if (cid:15) ≤ min { 1 / (32 n 2 m 2 ) , δ 2 c ( A ) 2 / 8 } , then det J f ( p , q ) ≥ 1+ δ 2 c ( A ) 2· (cid:15) 2 for any ( p , q ) ∈ S 1 , and det J f ( p , q ) ≥ 1 − (cid:15) 3 for any ( p , q ) ∈ S 2 .
Recall that the change of entropy is (cid:82) g t ( p , q ) log | det J f ( p , q ) | d ( p , q ) .
As log(1 + r ) ≥ r 1+ r for any r > − 1 , the change of entropy is at least
[Equation]
In Theorem 4.3, for any ﬁxed δ and c ( A ) , the lower bound of λ decreases with (cid:15) .
In particular, if (cid:15) =
δ 2 c ( A ) 2 / 64 , the lower bound is 1 / 4 , i.e. it only requires a small probability in S 1 for the DE to strictly increase.
Theorem 4.3 indicates that the DE growth is at least linear ( Ω( T ) ) as long as the distribution has a sufﬁcient amount of probability lying in S 1 .
Next, in Lemma 4.4 we show that the DE growth is at most logarithmic ( O (log T ) ) whenever the initial distribution has bounded support.
These two seemingly contradicting bounds can only be compatible for one reason: the condition needed by
Theorem 4.3 will eventually be violated, i.e. there is insufﬁcient amount of probability lying in Safter a sufﬁciently long time.
We formalize this in Theorem 4.5 below.
Lemma 4.4 Suppose the initial distribution has bounded support in [ − b, b ] n + m , for some b > 0 .
Then the differential entropy at time T is at most ( n + m ) log( (cid:15)T + b ) .
Proof: Since every entry in A is bounded in the interval [ − 1 , 1] , we have | p t +1 j − p tj | ≤ (cid:15) and | q t +1 k − q tk | ≤ (cid:15) .
Thus, the distribution at time T has bounded support in [ − (cid:15)T − b, (cid:15)T + b ] n + m .
Within this box, the maximal differential entropy is attained by the uniform distribution over it, which is ( n + m ) log( (cid:15)T + b ) .
(cid:3)
Theorem 4.5 For any non-trivial two-player zero-sum games ( A , − A ) , with MWU dynamics of step-size (cid:15) ≤ min { 1 / (32 n 2 m 2 ) , δ 2 c ( A ) 2 / 8 } , for any initial distribution with bounded support, and for any λ > 16 (cid:15) δ 2 c ( A ) 2 , there exists an inﬁnite sequence of times ( T 1 , T 2 , · · · ) such that the probability in S 2 at each round T i is at least 1 − λ .
The above theorem gives a lower bound of 1 − λ on the probability in the boundary area for inﬁnitely many steps, which is the ”grand escape” phenomenon we mentioned in the introduction.
4. MWU IN ONE POPULATION GAME
Recall from (5) that in one-population game A , MWU dynamics in the cumulative payoff space is p t +1 =
p t + (cid:15) ·
Ax ( p t ) , where x k ( p t ) =
[Equation]
Similar to the two-player zero-sum games, we also consider two different regions: the interior region S
= { p | x ( p ) ≥ δ } , and the boundary region S 2 , which is the complement to S 1 .
Let g t ( p ) denote the probability density of p at time t .
We will show that for some sub-classes of onepopulation games A , MWU dynamics increase differential entropy linearly under some conditions.
Also we show the analogue of Theorem 4.5: there exists inﬁnitely many rounds, such that at each of these rounds t , the probability of p t ∈ S 2 will be at least a constant.
First of all, we present the
Jacobian analysis of such systems.
Jacobian Analysis.
The Jacobian of the dynamics is J
= I + (cid:15) · R , where I is the n × n identity matrix, and R is an n × n matrix, in which
[Equation]
For simplicity, with a little abuse of notation, we use x k as a shorthand of x k ( p ) .
Next, we calculate the determinant of the Jacobian.
When we compute it using the Leibniz formula, it can be expressed in the following form, which is a polynomial of (cid:15) :
[Equation]
where the ﬁrst order term is
[Equation]
Thus, the following lemma gives the condition with which the ﬁrst order term is non-negative.
Note that in order to have a increasing differential entropy, non-negative ﬁrst order term is a necessary condition for a small enough step size.
Lemma 4.
The ﬁrst order term is non-negative for all possible x if and only if A kk + A k (cid:48)
k (cid:48) ≥
A kk (cid:48)
+ A k (cid:48)
k for all k, k (cid:48) .
Moreover, if A kk + A k (cid:48) k (cid:48) ≥
A kk (cid:48)
+ A k (cid:48)
k for all k, k (cid:48)
and the inequality is strict for some k, k (cid:48) , then the ﬁrst order term is strictly positive for all x ≥ δ .
In the terminology of mathematical biology, A kk is the payoff to species k for the intra-species play within the species, while A kk (cid:48) is the payoff to species k for the inter-species play between species k and k (cid:48) .
Informally speaking, the condition in the above lemma is: the one-population game rewards intra-species play more than to inter-species play.
Two Sufﬁcient Conditions for Linear DE Growth.
Recall that to demonstrate linear DE growth, we want the determinant of the Jacobian to be strictly larger than 1 .
By Lemma 4.6, a clear sufﬁcient condition is: A kk + A k (cid:48) k (cid:48) ≥
A kk (cid:48)
+ A k (cid:48)
k for all k and k (cid:48) , and there exist k, k (cid:48)
such that the inequality is strict.
But when there is no k, k (cid:48) with strict inequality, it is still possible to have linear
DE growth if the second order term is strictly positive.
We present precise statements regarding these two cases below.
To proceed, we deﬁne s ( A ) , which is positive for the ﬁrst case: s ( A ) =
[Equation]
For the ﬁrst case, we now bound the higher order terms.
Note that since | A kk (cid:48) | ≤ 1 and hence | R kk (cid:48) | ≤ 2 .
When expanding the determinant using the Leibniz formula, we get a polynomial of (cid:15) .
For any i ≥ 2 , each (cid:15) i term in the expansion must be of the format (cid:15)
i times a product of i factors of the form R k 1 ,k 2 , so each such term has absolute value bounded by 2 i (cid:15) i .
To count the number of such terms, note that within each term, the collection of k 1 ’s is same as the collection of k 2 ’s, but the order of k 2 can be any permutation of the k 1 ’s.
A simple counting shows there are at most (cid:0) ni (cid:1) · i ! ≤ n i such terms, and hence the i -th order coefﬁcient is bounded by n i · 2 i .
Therefore, when (cid:15) ≤ δ 2 s ( A ) 16 n 2 , det J ( p ) ≥
+ δ 2 s ( A )· (cid:15) for all p ∈ S 1 , and det J ( p ) ≥ 1 − 8 n 2 (cid:15) 2 for all p ∈ S 2 .
With a similar argument as in two-player zero-sum game, we achieve the following theorems.
Theorem 4.7 If (cid:82) S 1 g t ( p ) d p ≥ λ , then the entropy will increase by at least δ 2 s ( A ) · (cid:15)/ 2 1+ δ 2 s ( A ) · (cid:15)/ 2 · λ − 8 n 2 (cid:15) 2 1 − 8 n 2 (cid:15) 2 · (1 − λ ) in round t , which is strictly positive whenever λ > 32 n 2 (cid:15) δ 2 s ( A ) .
Theorem 4.8 In a population game
A such that A kk + A k (cid:48) k (cid:48) ≥
A kk (cid:48)
+ A k (cid:48)
k for all k and k (cid:48)
and there exist k and k (cid:48)
such that the inequality is strict, with MWU dynamics of step-size (cid:15) ≤
δ 2 s ( A ) 16 n 2 , for any initial distribution with bounded support, and for any λ > 32 n 2 (cid:15) δ 2 s ( A ) , there exists an inﬁnite series of time ( T 1 , T 2 , · · · ) such that, for any i , the probability in S 2 at round T i is no less than 1 − λ .
In Appendix C, in the same spirit we did in Figure 2, we present a few plots of MWU in an one-population game that satisﬁes the condition in the above theorem.
Next, we consider the case that A kk + A k (cid:48) k (cid:48)
=
A kk (cid:48)
+ A k (cid:48)
k for all k and k (cid:48) .
In this case, the ﬁrst order term is 0 .
Therefore, we will focus on the second order term, which is (cid:88)
[Equation]
A k 2 ,k (cid:48) x k (cid:48)
(cid:33)  .
Let C ( A , B ) ( x , y )
= − (cid:80) k 1 ,k 2 x k 1 y k 2 ( A k 1 k 2 − (cid:80)
k (cid:48)
A k 1 k (cid:48)
y k (cid:48) )
( B k 1 k 2 − (cid:80) k (cid:48)
B k (cid:48) ,k 2 x k (cid:48) ) .
Then the second order term is 12 C ( A , A (cid:62) )
( x , x ) .
We will show this term is non-negative.
Lemma 4.9 If A kk + A k
cid:48)
k (cid:48)
=
A kk (cid:48)
+ A k (cid:48)
k for all k and k (cid:48) , then 12 C ( A , A (cid:62) )
( x, x ) = 12 C
( A , − A )
( x, x ) ≥ δ 2 c ( A ).
Proof: By Cheung & Tao (2020), C ( A , B ) ( x, y )
=
C ( A , B + T ) ( x, y ) if T kk (cid:48) =
u k + v k (cid:48) for all k and k (cid:48) .
As ( A (cid:62) − ( − A )) ij =
A ji + A ij =
A ii + A jj , the ﬁrst equality holds.
The second inequality follows from Cheung & Piliouras (2019).
(cid:3)
Thus, for the determinant of the Jacobian, the ﬁrst order term is 0 and the second order term is 12 C ( A , A (cid:62) ) , which is at least δ 2 c
( A )in S 1 .
Moreover, the i -th order term is bounded by n
i 2 i (cid:15) i .
Therefore, when (cid:15) ≤ δ 2 c ( A ) 256 n 3 , det J ( p ) ≥ 1+ δ 2 c ( A )· (cid:15) 2 for any p ∈ S 1 , and det J ( p ) ≥ 1 − 16 n 3 (cid:15)for x ∈ S 2 .
With this, we obtain the following theorems.
Theorem 4.10 If (cid:82) S 1 g t ( p ) d p ≥ λ then the entropy will increase by at least δ 2 c ( A ) · (cid:15) 2 / 16
+ δ 2 c
( A ) · (cid:15) 2 / 16 · λ − 16 n 3 (cid:15) 3 1 − 16 n 3 (cid:15) 3 · (1 − λ ) in round t , which is strictly positive whenever λ > 512 n 3 (cid:15) δ 2 c
( A ) .
Theorem 4.11 In a population game
A such that A kk + A k (cid:48) k (cid:48)
=
A kk (cid:48)
+ A k (cid:48)
k for all k and k (cid:48) and A is non-trivial, with MWU dynamics of step-size (cid:15) ≤
δ 2 c ( A ) 256 n 3 , for any initial distribution with bounded support, and for any λ > 512 n 3 (cid:15) δ 2 c ( A ) , there exists an inﬁnite series of time ( T 1 , T 2 , · · · ) such that, for any i , the probability in S 2 at round T i is at least 1 − λ .
R EPRODUCIBILITY S TATEMENT
We present complete proofs for all theoretical results in this work.
E THICS S TATEMENT
We do not see any ethical or future societal consequences of this work.
A M OTIVATING E XAMPLE
To explain our mathematical model intuitively, we present the following simple one-dimensional example. Let f ( x ) = 2 x , and suppose that X 0 is a random variable uniform on the interval [0 , 1] . Then X 1 = f
( X 0 ) becomes another random variable, which is uniform on [0 , 2] .
Analogously,
X 2 = f
( X 1 ) is a new random variable being uniform on [0 , 4] .
Generally speaking, the problem is to understand how the initial random variable X 0 evolves under a given deterministic iterated update rule f , and to study how the uncertainty evolves using the measure of differential entropy.
In our paper, we focus on such systems which arise in the context of learning-in-games.
In a learning-in-game system, X 0 = ( p 0 , q 0 ) represents a distribution over initial beliefs (initial cumulative payoffs) of the two players, and X t
=
( p t , q t ) represents the distribution of cumulative payoffs at time t .
Such a probabilistic formulation is natural from the perspective of an external analyst (who is not a player of the game), who cannot be certain about what the initial beliefs the players have.
For her to proceed with an analysis of the system, a natural approach is to infer the initial distributions from either the past data, or her knowledge about how random external signals inﬂuence the initializations.
The iterated update rule f in this case is determined by the learning algorithms used by the players, and by the underlying game.
For instance, if both players are using
MWU in a bimatrix game ( A , B ) , the update rule f is given by (4).
Unlike the motivating example, for learning-in-game systems the dimensions can be arbitrarily high, and the update rule is non-linear.
Thus, deriving the exact evolution of the distribution of X t is very hard in general.
But we can still use the popular statistical measure, differential entropy (DE), to understand the evolution of uncertainty.
We show that the DE increases linearly with time for various combinations of learning algorithms and games, including MWU in zero-sum games,
OMWU in coordination games, and MWU in certain one-population games.
Section 2 has precise descriptions of the learning algorithms and the games.
These results can be extended to a broad family of non-zero-sum and non-coordination games, via the Jacobian analysis by Cheung & Tao (2020).
The results also extend to gradient ascent, as it is known to be a special case of FTRL.
B M ORE APPLICATIONS B. OMWU IN TWO - PLAYER COORDINATION GAMES
A two-player game ( A , B ) is a coordination game if A = B .
Similar to the MWU dynamics in twoplayer zero-sum games, here we also look at two different areas: S 1 = { ( p, q ) | x ( p ) ≥ δ and y ( q ) ≥ δ } and S 2 , which is the complement to S 1 , S 2 = S 1 .
For a coordination game ( A , A ) , Cheung &
Piliouras (2020) showed that when ( p, q ) ∈ S 1 , then det J f ( x ) ≥ 1 + δ 2 c ( A ) 2· (cid:15) 2 when (cid:15) is small enough and, when ( p, q ) ∈ S 2 , then det J f ( x ) ≥ 1 − O ( (cid:15) 3 ) .
Therefore, by a calculation which is similar to MWU in two-player zero-sum game, we obtain the following theorems.
Theorem [-A.
-]{+B.
+}
If (cid:82) S 1 g t ( p , q ) d ( p , q ) ≥ λ then the entropy increases by at least δ 2 c ( A ) 2 · (cid:15) 2 / 8
+ δ 2 c
( A ) 2 · (cid:15) 2 / 8 · λ −
O ( (cid:15) 3 ) · (1 − λ ) .
Theorem [-A.
-]B.2 For any non-trivial two-player coordination games ( A , A ) , with OMWU dynamics of a small enough step-size (cid:15) , for any initial distribution with bounded support, and for any λ such that (cid:15)
=
o ( λ ) , there exists an inﬁnite series of time ( T 1 , T 2 , · · · ) , such that the probability in S 2 at round T i is at least 1 − λ .
B. OMWU IN ONE POPULATION GAMES
As pointed out in Cheung & Piliouras (2020), the continuous analogue of OMWU, in the context of online learning, is as follows.
There are n actions.
Let p ( t ) ∈
R n denote the cumulative payoff vector up to time t , and u ( t ) ∈ R n denote the instantaneous payoff vector to the n actions at time t .
The continuous analogue is
[Equation]
In one-population game, u ( t ) =
A · x ( t ) , where for the logit conversion used in MWU and OMWU, x k = exp( p k ) / (cid:80) (cid:96) exp
( p (cid:96) ) .
It is easy to compute that
[Equation]
and ∂x
[Equation]
when (cid:96) (cid:54) =
k.
Then for any (cid:96) = 1 , 2 , . . . , n , we have
[Equation]
Following the notation in MWU case, ∂
[ Ax ] j ∂p (cid:96) = R j(cid:96) .
By the chain rule, we have
[Equation]
Putting the above equality into (8) yields
[Equation]
which is a recurrence of the variables ˙ p 1 , ˙ p 2 , . .
. , ˙ p n .
Unwinding this recurrence yields
[Equation]
The above system, upon Euler discretization to a discrete-time dynamical system with time interval (cid:15) , becomes
[Equation]
Observe that on the RHS of (9), the ﬁrst two terms coincide with that in the MWU case.
The remaining terms can be viewed as a second-order perturbation .
Consequently, the Jacobian of the system equation 9 is a second-order perturbation of the Jacobian for the MWU case, and hence the constant and ﬁrst-order terms of the determinants of these two Jacobians are the same, which is (cid:80) nj =1 R jj .
Therefore, for the case that A kk + A k (cid:48) k (cid:48) ≥ A kk (cid:48)
+ A k (cid:48)
k for all k and k (cid:48)
and there exist k and k (cid:48)
such that the inequality is strict, we obtain the following theorems.
Recall that S 1 = { p | x ( p ) ≥ δ } and S 2 , which is the complement to S 1 , S 2 = S 1 ; and s ( A ) =
max k,k (cid:48)
{ A kk + A k (cid:48) k (cid:48) − A kk (cid:48) − A k (cid:48) k } .
Theorem [-A.
-]{+B.
+}
If (cid:82) S 1 g t ( p ) d p ≥ λ , then the entropy increases by at least δ 2 s ( A ) (cid:15)/ 2 1+ δ
s
( A ) (cid:15)/ 2 · λ − O ( (cid:15) 2 ) · (1 − λ ) for a small enough step size (cid:15) .
Theorem [-A.
-]{+B.
+}
In an one-population game A such that A kk + A k (cid:48) k ( cid:48) ≥ A kk (cid:48)
+ A k (cid:48)
k for all k and k (cid:48)
and there exist k and k (cid:48)
such that the inequality is strict, with OMWU dynamics with a small enough step-size (cid:15) , for any initial distribution with bounded support, and for any λ such that (cid:15)
=
o ( λ ) , there exists an inﬁnite series of time ( T 1 , T 2 , · · · ) such that, for any i , the probability in S 2 at round T i is no less than 1 − λ .
For the case that A kk + A k (cid:48) k (cid:48)
=
A kk (cid:48)
+ A k (cid:48)
k for all k and k (cid:48)
, we consider the second-order term.
The second-order terms of the two determinants (MWU and OMWU) are not the same though.
The second-order term of the determinant of Jacobian for the OMWU case is (cid:88)
[Equation]
Observe that the ﬁrst two summations are just the same as those in the MWU case.
Next, a straightforward calculation gives
[Equation]
where 1 j = (cid:96) = 1 if j = (cid:96) , and 0 otherwise.
Hence n (cid:88)
j = n
[Equation]
n [Equation] = n (cid:88) j
[Equation] [Equation] n (cid:88) j = x j
[ Ax [Equation] j n (cid:88) (cid:96) = R j(cid:96) − n (cid:88)
j =
R jj [ Ax ] j n (cid:88) (cid:96) = x (cid:96) . Since (cid:80) n(cid:96) =1 x (cid:96) = 1 and (cid:80) n(cid:96)
=1 R j(cid:96)
= 0 , we have (cid:80) nj =1 (cid:80) n(cid:96)
=1 ∂R j(cid:96)
∂p j · [ Ax ] (cid:96) = 0 .
To conclude, the second-order term is (cid:88) 1 ≤ j<k ≤ n
R jj R kk − (cid:88) 1 ≤ j<k ≤ n
R jk R kj + n (cid:88) j = n (cid:88) (cid:96) =
R j(cid:96) R (cid:96)j = n (cid:88)
j = ( R jj )
+ (cid:88) 1 ≤ j<k ≤ n
R jj R kk + (cid:88) 1 ≤ j<k ≤ n
R jk R kj =   n (cid:88) j =
R jj   − (cid:88)
1 ≤ j<k ≤ n
R jj R kk + (cid:88) 1 ≤ j<k ≤ n
R jk R kj
Note that when A kk + A k (cid:48) k (cid:48)
=
A kk (cid:48)
+ A k (cid:48)
k for all k and k (cid:48) , (cid:80) j R jj = 0 .
Therefore, the second order term of OMWU is the negative of the second order term of MWU, which will be smaller than − δ 2 c
( A ).
Theorem [-A.
-]B.5 In an one-population game A such that A kk + A k (cid:48) k (cid:48)
≥
A kk (cid:48)
+ A k (cid:48)
k for all k and k (cid:48) , with OMWU dynamics, if (cid:82) p ∈ S 1 g ( p )
d p ≥ λ , then the entropy decreases by at least δ 2 c
( A ) 2 (cid:15) 2 1+ δ 2 c
( A ) 2 (cid:15) 2 · λ − O ( (cid:15) 3 ) · (1 − λ ) for a small enough step size (cid:15) .
B. P ERTURBATIONS TO P AYOFFS
In the main body, we focus on two-player zero-sum games.
In this appendix, we discuss the Jacobian analysis for two-player general games based on Cheung & Tao (2020), and explain why it is robust against perturbations.
First, we deﬁne the following function on the cumulative payoff space of two-player games, which depends on a matrix M ∈ R n × m .
[Equation]
where x j , x j (cid:48) , y k , y k (cid:48) is as deﬁned in (2).
For any general game ( A , B ) , we can decompose it into the sum of a zero-sum game and a coordination game as ( A , B ) =
( Z , − Z )
+ ( C , C ) , where Z
=
( A − B ) / 2 and C =
( A + B ) / 2 .
Cheung & Tao (2020) showed that the determinant of the Jacobian matrix of MWU with step-size (cid:15) in the game ( A , B ) is
[Equation]
We rewrite L Z ( p , q ) − L C ( p , q ) as
[Equation]
Thus, if max j,
[Equation]
[Equation]
10) and
[Equation]
(11) then in the domain S 1 = { ( p , q ) | ∀
j, k, x j ( p ) ≥ δ and y k ( q ) ≥ δ } , the determinant is at least 1 + δ 4 L(cid:15) 2 / 8 for all sufﬁciently small (cid:15) .
Since the product ( A jk + A j (cid:48) k (cid:48) − A jk (cid:48) − A j (cid:48) k )( B jk + B j (cid:48) k (cid:48) − B jk (cid:48) − B j (cid:48) k ) depends on A , B continuously, it is natural to expect that when A , B are slightly perturbed, the value of the product changes also slightly.
To make this precise, we suppose that in each round t , the actual payoff matrices are ( A t , B t ) , and their entries are perturbations from ( A , B ) by at most an absolute value of β ≤ 2 .
5 Recall that the entries in A , B are bounded in the interval [ − 1 , 1] , so the absolute values of A jk + A j (cid:48) k (cid:48) − A jk (cid:48)
− A j (cid:48) k and B jk (cid:48) + B j (cid:48)
k − B jk − B j (cid:48) k (cid:48) are bounded by 4 , while the perturbations to their values are bounded by 4 β .
By the lemma below, for the game ( A t , B t ) , the bound in (10) still hold by replacing its lower bound with L − 34 β , and the bound in (11) still hold by replacing its lower bound with − 34 β .
Thus the determinant of the Jacobian matrix in S 1 is at least 1 + ( δ 4 L/ 4 − 34 β ) (cid:15) 2 / 2 ≥ 1 + δ 4 L(cid:15) 2 / 16 if β ≤ δ 4 L/ 272 .
Lemma [-A.
-]{+B.
+}
If β ≤ 2 , | a | , | b | ≤ 4 and | a (cid:48) − a | , | b (cid:48) − b | ≤ 4 β , then a (cid:48) b (cid:48) ≥ ab − 34 β .
Proof: Let a (cid:48) − a = β a , b (cid:48) − b = β b .
Then a (cid:48) b (cid:48) − ab = bβ a + aβ b + β a β b ≥ − 16 β − 16 β − β 2 ≥ − 34 β .
(cid:3)
C P LOTS OF U NCERTAINTY E VOLUTION OF MWU IN O NE -P OPULATION G AME
[Figure]
[Figure]
0 .1 .−
1 .− 1 .0 .1 .1 .−
1 .0 .(cid:35)