G ENERALIZED N ATURAL G RADIENT F LOWS IN H IDDEN C ONVEX -C ONCAVE G AMES AND GAN S
Anonymous authors
Paper under double-blind review
A BSTRACT
Game-theoretic formulations in machine learning have recently risen in prominence, whereby entire modeling paradigms are best captured as zero-sum games.
Despite their popularity, however, their dynamics are still poorly understood.
This lack of theory is often substantiated with painful empirical observations of volatile training dynamics and even divergence.
Such results highlight the need to develop an appropriate theory with convergence guarantees that are powerful enough to inform practice.
This paper studies the generalized Gradient Descent-Ascent (GDA) ﬂow in a large class of non-convex non-concave Zero-Sum games dubbed Hidden Convex-Concave games, a class of games that includes GANs.
We focus on two speciﬁc geometries: a novel geometry induced by the hidden convex-concave structure that we call the hidden mapping geometry and the Fisher information geometry.
For the hidden mapping geometry, we prove global convergence under mild assumptions.
In the case of Fisher information geometry, we provide a complete picture of the dynamics in an interesting special setting of team competition via invariant function analysis.
I NTRODUCTION
Min-max optimization has found extensive applications in modern Machine Learning (ML) and
Deep Learning.
Popular application settings include Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), adversarial training (Madry et al., 2018), and multi-agent reinforcement learning (Silver et al., 2017).
In all of these cases, a pair of networks is typically trained towards ﬁnding an approximate equilibrium of a highly non-convex non-concave problem.
However, such settings go beyond classical and well-known results in game theory for which equilibria only exist in more restrictive settings, i.e., convex-concave games (Sion et al., 1958).
Unfortunately, there is no parallel guarantee if the payoff is not convex-concave, and all these applications are indeed based upon non-convex non-concave games.
Even worse, many negative results occur when dealing with such payoffs: global or local minimax may not exist (Jin et al., 2020), and even if they exist, there is, in general, no “reasonable algorithm” 1 that can globally converge to any meaningful notion of local optimum (Letcher, 2021; Hsieh et al., 2020).
In order to overcome these negative results, we analyze a speciﬁc restricted class of non-convex non-concave games called Hidden Convex-Concave (HCC) games (Flokas et al., 2020; Gidel et al., 2021; Flokas et al., 2021) that include many Machine Learning applications such as GANs (Goodfellow et al., 2014), Adversarial Example Games (AEG) (Bose et al., 2020), or Minimax Estimation of Conditional Moment
(Dikkala et al., 2020).
Our contributions.
In the setting of HCC games, our analysis tackles unique challenges not captured l standard in the convex-concave games setting.
As a way to overcome those challenges, we propose a new type of dynamics, dubbed Natural Hidden Gradient dynamics (NHG) , and we prove its convergence to stationary points of the HCC game.
Critically, our convergence results are global , i.e., we do not make any assumptions about initial conditions, e.g., that the initial points belong to some local neighborhood.
This novel algorithm is, in fact, inspired by the generalization of gradient ﬂows in different Banach spaces than the standard Euclidean one.
Arguably, the most well-known non-Euclidean gradient is the natural gradient ﬂow induced by the Fisher information matrix, which enjoys deep connections to the Replicator Dynamics (RD).
Thus, a natural question emerges: Can we have similar convergence guarantees for RD?
In that regard, we provide a complete picture of the resulting dynamics in a setting of team competition via invariant function analysis.
Speciﬁcally, the ﬂow of the dynamics is shown to be reduced to a competition between two generalized gradients, one for each team.
We show that the behavior of the dynamics in this setting is constrained by a maximal number of invariant functions, which, combined with the actual geometry of the available strategies, shows that the system can either converge or cycle.
These results showcase, both, the importance of adapting the “algorithmic geometry” to the application and the data at hand, as well as the unexplored effects of feasibility constraints on the complexity of well-known game dynamics.
R ELATED W ORK
Despite the popularity of deep learning applications involving non-convex non-concave games, our understanding of their optimization dynamics and the nature of their solution concepts is still preliminary.
However, this space has already witnessed a few important early works that focus on identifying new solution concepts.
These solution concepts—which are also broadly applicable in general min-max games—include (local/differential)
Nash equilibria (Adolphs et al., 2019; Mazumdar & Ratliff, 2019), (local/differential)
Stackelberg equilibria (Fiez et al., 2020; Wang et al., 2020), local minmax (Daskalakis & Panageas, 2018), local robust points (Zhang et al., 2020), and approximate minimax theorems (Jin et al., 2019; Gidel et al., 2021).
Numerous solutions concepts such as cycles (Vlatakis-Gkaragkounis et al., 2019), chaotic behavior (Cheung & Piliouras, 2019; Cheung &
Tao, 2021), and computational issues (Daskalakis et al., 2021) indicate that solving min-max games, in general, might involve challenging and complex behavior.
Many algorithms have been proposed to solve restricted classes of non-convex non-concave games such as Polyak-Łojasiewicz games (Nouiehed et al., 2019; Yang et al., 2020), nonconvex-concave games (Lin et al., 2020; Ostrovskii et al., 2021; Yang et al., 2020; Kong & Monteiro, 2021), as well as classes of games inspired by variational inequalities (Mertikopoulos et al., 2019; Diakonikolas et al., 2021; Lee & Kim, 2021).
However, even though they are signiﬁcant advances in the understanding of non-convex non-concave game dynamics, such classes of games may not encompass games where the players are parameterized neural networks such as GANs.
While convergence in GANs has been a topic of exploration (Kodali et al., 2017; Heusel et al., 2017; Mescheder et al., 2018; Gemp & Mahadevan, 2018;
Li et al., 2018; Hsieh et al., 2019; Cao & Guo, 2020), its hidden convex-concave structure has not been exploited before, and there were no theoretical global convergence guarantees in a general setting up to this date. In particular, Hsieh et al.
(2019) use a lifting trick and proceed in solving a relaxation of the problem in the distribution space, while in our work we work entirely in the parameter space.
With this respect, we consider
Flokas et al. (2021), and Gemp & Mahadevan (2018) as the closest related works. On the one hand, in Gemp & Mahadevan (2018), the authors propose crossing-the-curl , a second-order technique, and provide global convergence guarantees for the Wasserstein Linear Quadratic GAN (W-LQGAN).
While the W-LQGANs is a class of non-convex non-concave GANs it remains far from the GAN formulation used in practice where the discriminator and the generator are neural networks.
On the other hand, the idea of Hidden Convex-Concave (HCC) games was ﬁrst proposed by Flokas et al.
(2021) and Gidel et al.
While Flokas et al.
(2021) study the Gradient Descent-Ascent (GDA) dynamics in the HCC setting, their work relies on hard-to-verify safety conditions.
Our global convergence results without hard-to-test assumptions on initialization are ﬁrst of their kind, to the best of knowledge.
P RELIMINARIES 3. H IDDEN C ONVEX -C ONCAVE GAMES
Many game-theoretic applications in machine learning often involve a speciﬁc structure where the models’ payoff is a convex-concave function (e.g., minimizing Jensen-Shannon-Divergence when training GANs).
In order to model this speciﬁc structure, we propose the following deﬁnition of
Hidden Convex-Concave (HCC) games where intuitively, the game’s payoff is convex-concave while the actions are parametrized by non-convex mappings.
Deﬁnition 1 (Hidden Convex-Concave game) .
A Hidden Convex-Concave game comprises a collection of payoff ( L x , x (cid:48) )
x , x (cid:48) ∈ R d , a distribution p , and two parametrized mappings, F : R M × R d → R , and G : R N × R d (cid:48)
→ R , such that the minimax game of interest is
[Equation]
Ψ ( θ , φ ) where Ψ ( θ , φ ) =
L ( F θ , G φ ) : = E ( x , x (cid:48) ) ∼ p [ L x , x (cid:48) ( F θ ( x ) , G φ ( x (cid:48) ))] .
In this setting, while we do not expect Ψ ( θ , φ ) to be convex-concave, we assume the function L : F × G → R to be convex-concave where F and G are, respectively, convex subsets of { F : R d → R } and { G : R p
→ R } .
2 We extended Flokas et al.
(2021)’s deﬁnition to now be able to include most minimax machine learning applications such as GANs or AEG.
Example 1 (Hidden Matching Pennies (HMP) games) .
Let us consider p, q ∈
[0 , 1] the probabilities of picking HEADS for the ﬁrst and the second player, respectively, in a Matching Pennies game with payoff matrix A ∈ R 2 × 2 , where A i,j = 1 if i = j ; − 1 , otherwise.
Then the payoff of this game is deﬁned via
[Equation]
[Equation]
p
[Equation]
(Player 1 matches the pennies)
[Equation] otherwise [Equation] Now, let us consider any mappings F : R M × R d → [0 , 1] and G : R N × R d
→
[0 , 1] , such that their output does not depend on the d -dimensional input, i.e., F θ ( x ) =
f ( θ ) , and G φ ( x ) =
g ( φ ) , ∀ x ∈
R d .
The payoff Ψ ( θ , φ ) =
L ( F θ , G φ ) :
=
E [ L x , x (cid:48) ( F θ (
[Equation]
deﬁnes a HCC game.
In this example, the two agents play the typical bilinear game of Matching Pennies.
However, they do not act on it directly (i.e., choose randomized actions to apply, e.g., the probability of playing
HEADS), instead they choose the input parameters θ , and φ , which are fed into functions f , and g , respectively, whose outputs deﬁne the probability of playing HEADS for each agent.
Example 2 (GANs) .
A Generative Adversarial Network (GAN) is a minimax game where the ﬁrst player, i.e., generator, aims at learning a distribution p θ similar to a reference data distribution p data .
In practice, the reference data distribution is taken to be the empirical data distribution.
Conversely, the second player, usually called the discriminator or critic, D φ , tries to distinguish the distributions of p θ and p data .
The payoff, Ψ , of this game is deﬁned as:
[Equation]
Assuming that p data and p θ have a density with respect to Lebesgue measure, 3 and that p data has full support, we can consider the distribution p such that p ( x , x (cid:48) )
=
p data ( x ) if x = x (cid:48) ; 0 , otherwise, and set L x , x ( p (cid:48) , D ) :
= log D + p (cid:48)
p data ( x ) log(1 − D ) , and L x , x (cid:48) ( p (cid:48) , D ) = 0 , ∀ x (cid:54) = x (cid:48) . Thus L x , x (cid:48) is convex-concave for any x ∈ R d .
We have that Ψ ( θ , φ ) =
E ( x , x (cid:48) ) ∼ p
[ L x , x (cid:48) ( p θ ( x ) , D φ ( x (cid:48) ))] , which is a HCC game.
A GAN formulates the generative modeling task as ﬁnding a Nash equilibrium of a minimax game.
The generator of a GAN is deﬁned as a function that aims to produce realistic data samples by transforming samples drawn from a ﬁxed noise distribution, e.g., N (0 , I d ) .
Here, we notice that the
GAN payoff is convex (actually linear) with respect to the density of the generated distribution.
An alternative formulation of a GAN is a Wasserstein GAN (WGAN)
(Arjovsky et al., 2017). It turns out that this GAN is also a HCC game.
Example 3 (WGANs) .
A Wasserstein GAN is a constrained minimax game, where the second player
D φ is a 1 -Lipchitz function and where the payoff is
[Equation]
Similarly, as in Example 2, the WGAN payoff can be shown to be a HCC game.
As the last class of examples of HCC games we present the Adversarial Example Games (AEG) (Bose et al., 2020).
Example 4 (AEG) .
An Adversarial Example Game is a minimax game between a generator G θ and a classiﬁer f .
Given samples ( x , y ) ∼ p data , the generator G θ aims at ﬁnding adversarial examples x (cid:48) such that (cid:107) x − x (cid:48) (cid:107) ∞ ≤ (cid:15) and that x (cid:48) is not classiﬁed as y by f , thus generating a distribution p θ .
Overall, the payoff of this game is
[Equation]
[Equation]
)] for
[Equation]
x
[Equation]
[Equation]
[Equation]
p [Equation] [Equation] 5) and where (cid:96) is the cross-entropy loss and G θ is such that (cid:107) x − G θ ( x ) (cid:107) ≤ (cid:15), ∀ x .
By using a similar construction as in Example 2, we can show that (5) is a HCC game with respect to p θ and f φ .
Throughout this work, we make the assumption that the minimax problem induced by L admits a solution:
Assumption 1.
The HCC game deﬁned by (1) admits a Nash equilibrium, ( F ∗ , G ∗ ) , i.e.,
[Equation]
Such an assumption is relatively mild since, by Sion et al.
(1958), it holds when the set F is compact (since, by deﬁnition, it is convex).
Here we do not assume that this solution is achievable, i.e., we do not assume that there exists θ ∗ and φ ∗ such that ( F θ ∗ , G φ ∗ ) =
( F ∗ , G ∗ ) .
Such sufﬁcient conditions for the existence of a Nash equilibrium of L : F × G → R are discussed in detail in Gidel et al.
(2021, Prop. 1), e.g., Assumption 1 holds if the parameters θ and φ are bounded.
From a high-level perspective, this assumption is analogous to the existential assumption of a optimizer in non-convex optimization.
In that sense, we use the equilibrium ( F ∗ , G ∗ ) as a target to build a Lyapunov function for a natural gradient ﬂow.
3. N ATURAL GRADIENT FLOW
In this section, we introduce the notion of a natural gradient ﬂow (Amari, 1985; 1998).
Let us consider a function f : S → R and a class of symmetric positive deﬁnite matrices ( P θ ) θ ∈ S (cid:31)
that we will refer to as metric tensors .
The natural gradient ﬂow is the ﬂow given by the steepest descent direction (Ollivier et al., 2017) with respect to the geometry induced by the matrices P θ ,
[Equation]
When P θ
=
I , we consider the standard Euclidean geometry and recover the standard gradient ﬂow.
One celebrated example of a natural gradient in machine learning is the natural gradient induced by the Fisher information matrix (Amari, 1998; Martens, 2020).
Example 5 (The natural gradient ﬂow of the Fisher information matrix) .
Let us consider P ( X ) the space of probability distributions on a set X ⊆ R with the metric induced by the Kullback–Leibler (KL) divergence.
Then the natural gradient ﬂow of the Fisher information matrix is given by
[Equation]
Moreover, if n := | X | = dim S is ﬁnite, and if p θ = θ , the ﬂow (8) is the natural gradient ﬂow of the Shahshahani metric (Shahshahani, 1979) induced by the metric tensors [Equation]
[Equation]
3. C ONNECTIONS BETWEEN R EPLICATOR D YNAMICS AND GRADIENT FLOWS
The Replicator Dynamics (RD) are standard dynamics used in Evolutionary Game Theory and learning in games.
It is, arguably, the most widely used model of evolutionary selection with multiple applications in economics, biology, and other ﬁelds.
Interestingly, RD enjoys a close connection to gradient ﬂows (see Sigmund (1984); Hofbauer et al.
(1998); Harper (2009); Mertikopoulos &
Sandholm (2018)).
Speciﬁcally, in the case of a symmetric and linear ﬁtness landscape, the gradient induced by the Shahshahani metric of the mean ﬁtness is a special case of RD.
Such a landscape can be formally represented by a Potential game .
A Potential game is a n -player game
G = ( n, S :=
[ m 1 ] × . . .
×
[ m n ] , u : S → R n ) , with payoff function u , characterized by the existence of a potential function Φ : S
→ R that satisﬁes
[Equation]
The connection between the Potential games and the gradient ﬂows induced by the Shahshahani metric can be formally shown by a generalization of Hofbauer et al.
(1998)’s lemma in the space of a Cartesian product of simplices.
Proposition 1.
The Replicator Dynamics of a potential game G with potential function Φ is an (extended) Shahshahani gradient in int(∆ m 1 × . . .
×
∆ m n ) having potential Ψ( θ ) :
=
E s i ∼ θ
i i ∈
[ n ] [Φ( s )] .
It follows that, in the context of potential games, RD are a gradient ﬂow with respect to a very speciﬁc geometry.
In the next section, we consider a different geometry. One that is induced by the HCC structure, and which we leverage to obtain a new natural gradient ﬂow that we call Natural
Hidden Gradient ﬂow (NGH).
3. T HE N ATURAL H IDDEN G RADIENT FLOW FOR HCC G AMES
In HCC games, we assume Ψ ( θ , φ ) =
L ( F θ , G φ ) (see Deﬁnition 1), and, therefore, since the entities that characterize a solution to the minimax problem are the mappings F θ and G φ instead of their parametrizations , θ and φ , respectively, a natural geometry for consideration is the one deﬁned by the L 2 distance in the space of F × G .
Formally, let θ and θ (cid:48) be two parameterizations for F (similarly, for G ).
The L 2 distance between F θ and F θ (cid:48) is
[Equation]
(11) where p x is the ﬁrst marginal (second for G ) of p .
We can then derive the metric tensors that deﬁne this geometry.
Proposition 2 (Metric tensors of the model space) .
For any θ , we have that
[Equation]
θ δ θ (cid:105)
+ o ( (cid:107) δ θ (cid:107) 2 )
where
A θ := E [ ∇ θ F θ ( x ) ∇ θ F θ ( x ) (cid:124) ] .
[Equation]
Consequently, A θ deﬁnes a metric on the parameter space in which the “distance” between two values, θ , and θ (cid:48) , corresponds to the L 2 distance between F θ and F θ (cid:48) .
We can, thus, construct the natural gradient ﬂow given by this metric.
Proposition 3 (Natural Hidden Gradient dynamics) .
The ﬂow induced by the geometry (11) is
[Equation]
where A θ := E x ∼ p x [ ∇ θ F θ ( x ) ∇ θ F θ ( x ) (cid:124) ] and B φ :=
E x (cid:48) ∼ p x (cid:48)
[ ∇ φ G φ ( x ) ∇ φ G φ ( x ) (cid:124) ] and C † denotes the pseudo-inverse of a matrix C .
C ONVERGENCE OF THE N ATURAL H IDDEN G RADIENT FLOW
In this section, we propose a new type of dynamics, dubbed Natural Hidden Gradient dynamics (NHG) , and in the following two theorems, we prove their convergence in HCC games and GANs.
At the heart of our analysis lies the construction of a proper Lyapunov function that measures the distance from the game’s equilibrium point.
By proving that the Lyapunov function is proper, i.e., monotonic, we will prove our proposed dynamics are approaching a Nash Equilibrium, i.e., converging to a stationary point.
4. W ARM - UP : A SINGLE DATAPOINT
In this section, as a warm-up, we will consider the single datapoint case.
In this case, the payoff that deﬁnes the HCC game, can be written as
[Equation]
[Equation] where L : R × R → R is convex-concave (see Example 1).
In this case, since the mapping F θ and G φ are evaluated at a single point, one can simplify the notation and consider F θ ( x ) =
f ( θ ) and G φ ( x (cid:48) )
=
g ( φ ) where f : R M → R and g : R N → R are real-valued mappings that do not depend on an input x or x (cid:48) .
This situation is already non-trivial since, as illustrated in Example 1, it can correspond to a non-convex non-concave parametrization of the Matching Pennies game.
Let us recall that we want to use a Nash equilibrium ( f ∗ , g ∗ ) of this game as a target.
In order to solve this game we consider the Natural gradient of the metric deﬁned in Proposition 2.
Proposition 4.
In the uni-dimensional case, the Natural Hidden Gradient ﬂow D1 takes the form of
[Equation]
and
[Equation]
Using Proposition 4, it is relatively straightforward to show that the distance to the optimum is a
Lyapunov function for the Natural Hidden Gradient ﬂow.
Theorem 1.
Let Ψ be the payoff of an HCC game deﬁned as in (13) and consider the game dynamics in (D2) .
Then, the quantity
[Equation]
is a Lyapunov function, i.e., it is positive, non-increasing, and null if and only if it evaluated at a game solution.
Moreover, if L is strictly convex-concave, we have that V is decreasing and that any limit point ( θ , φ ) satisﬁes ∇ θ L ( f ( θ ) , g ( φ )) = ∇ φ L ( f ( θ ) , g ( φ ))
= 0 .
In order to see this, one can compute the time derivative of V .
After some elementary computations, it follows ˙
V ( θ
[Equation]
φ ) =
[Equation]
g [Equation]
g ( φ ) , and, thus, by the convex-concavity of L , we have that ˙ V ≤ 0 .
However, generalizing this theorem to HCC games with non-constant mappings (with respect to x ) is non-trivial since we drastically used the simplicity of the mappings (in Proposition 4) to simplify the expression of the ﬂow.
In the next section, we propose to extend our convergence analysis to ﬁnite sum HCC games.
4. T HE GENERAL FINITE SUM CASE
In this section, we consider a ﬁnite-sum version of the HCC games as they appear in Deﬁnition 1.
In this case, the payoff is deﬁned as
[Equation]
where the function L : R n × R m → R is assumed to be convex-concave.
We note L i,j :=
L x i , x (cid:48)
j for compactness.
Let us recall that we assumed the existence of a Nash equilibrium ( F ∗ , G ∗ ) for the minimax problem (Assumption 1).
In this situation, the Natural gradient deﬁned in Propositionhas the following form:
[Equation]
where A θ := 1 n (cid:80) ni
=1 ∇ θ F θ ( x i ) ∇ θ F θ ( x
[Equation]
(cid:124) and B φ := 1 m
[Equation]
G
[Equation]
We will generalize the Lyapunov function considered in the uni-dimensional case (14).
The idea is to consider the L 2 distance between ( F θ ( x i ) , G φ ( x (cid:48) j ))
i,j and ( F ∗ ( x i ) , G ∗ ( x (cid:48) j ))
i,j as our Lyapunov function V .
However, in order to prove our result, we will need the following technical assumption.
Assumption 2.
For any θ ∈ R M and φ ∈ R N , we have that the families of vectors ( ∇ θ F θ ( x i ))
i ∈
[ n ] and ( ∇ φ G φ ( x (cid:48) j ))
j ∈ [ m ] are linearly independent.
This assumption is relatively mild since it can be insured by an arbitrarily small perturbation of the considered vectors.
In practice, it suggests regularizing the matrices
A θ and B φ by adding (cid:15) · I d which is a standard way to stabilize methods requiring matrix inversions.
Theorem 2.
Let Ψ be the payoff of a ﬁnite-sum HCC game given by (15) and consider the game dynamics in (D3) .
Under Assumption 1, Assumption 2, and that d > n we have that the quantity
[Equation]
[Equation]
is a Lyapunov function, i.e., is positive, non-increasing and null if and only if evaluated at a game solution.
Moreover, if L is strictly convex-concave, V is decreasing as long as ( θ , φ ) (cid:54) =
( θ ∗ , φ ∗ )
and if L is a µ -strongly convex-concave function we have that V is decreasing exponentially as V ( θ , φ ) = V ( θ 0 , φ 0 ) exp( − µt ) .
The proofs of Theorem 1 and Theorem 2 can be found in §A.3.
C HARACTERIZATIONS OF R EPLICATOR D YNAMICS IN HCC GAMES
In this section, we consider a speciﬁc instance of HMP games (cf.
Example 1), dubbed 2-Team HMP games or the XOR-XOR games .
Although the possibility of cycling orbits for RD in such games was established before (Piliouras & Schulman, 2018), in this section, we show stronger results.
Speciﬁcally, as we prove in subsection 5.2, by enforcing restrictions to the game’s parameters, it is possible to affect the game’s outcome, e.g., we may deviate from the well-known cyclic behavior in the unrestricted setting, and moreover, enforce divergence away from the game’s original equilibrium and convergence to novel ﬁxed points.
We completely characterize the geometry of possible limit cycles in such a restricted setting by exploiting intuitions developed via the connection between RD and the Shahshahani information geometry.
Speciﬁcally, we show the dynamics are controlled by invariant functions, which correspond to weighted sums of the cross-entropy of the current mixed strategies of opposing members relative to the uniform, equilibrium strategies (20).
For the complete proofs of this section, we refer the interested reader to §A.4.
5. 2-T EAM H IDDEN M ATCHING P ENNIES GAMES
We introduce the 2-Team HMP game , G = ( n := n 1 + n 2 , S :
= { 0 , 1 } n , u : S → R n ) , between n 1 + n 2 members divided into two teams.
The ﬁrst team, team 1 , consists of n 1 members, while the second team, team 2 , consists of n 2 members.
The payoff function u for a strategy proﬁle s :=
[Equation]
s 1 , s 2 ) ∈ S is given by [Equation]
[Equation]
where XOR( s k ) = 1 if |{ s k,i | s k,i = 1 }| is odd, and 0 , otherwise.
Given a mixed-strategy proﬁle, ( θ , 1 − θ ) , θ :=
( θ k ) k ∈ [2] ∈ [0 , 1] n , the expected payoff of the i -th member of team k is given by
[Equation]
) where f ( θ 1 ) : =
E s 1 ∼ Ber( θ 1 )
[XOR( s 1 )] and g ( θ 2 ) :=
E s 2 ∼ Ber( θ 2 )
[XOR( s 2 )] .
Notice that, since each member of a given team aims at maximizing the same payoff, G is a Hidden Matching Pennies game (Example 1) with hidden mappings f ( θ 1 ) , and g ( θ 2 ) ,
[Equation]
It is not difﬁcult to prove that the RD exhibit cyclic behavior in this setting.
The following theorem provides a ﬁne-grained characterization of the dynamics of the HCC game in (19) since we know that the trajectories will lie on the intersection of level sets of the invariant functions identiﬁed by the following theorem:
Theorem 3.
Consider the Replicator Dynamics of G .
Given any interior initial condition, the resulting orbit is a cycle that satisﬁes the following n 1 + n 2 − 1 independent invariant functions:
[Equation] [Equation]
5. R EPLICATOR D YNAMICS
H IN A RESTRICTED
Next, we introduce restrictions in the range of each member’s strategies in G such that θ k,i ∈ S k,i :=
[ α k,i , β k,i ] ⊆
[0 , 1] , ∀ k ∈ [2] , i ∈ [ n k ] .
These restrictions reﬂect in RD as halts, i.e., ˙ θ k,i = 0 , every time the strategy of the i -th member of the k -th team exceeds those bounds.
To ease our notation, we let S k :=
{ i ∈ [ n k ] | {+
 2+} ∈ S k,i } , k ∈ [2] denote all the members of the k -th team for which ( 1 2 , 1 2 ) is an allowed mixed strategy, and, to simplify this part of the analysis, we also make the following mild assumption on the initialization of the dynamics:
Assumption 3.
For all k ∈ [2] and i ∈ [ n k ] , θ k,i (0) ∈ S k,i ⊂ (0 , 1) .
A signiﬁcant observation is that for any mixed-strategy proﬁle
( θ , 1 − θ ) , θ :=
( θ 1 , θ 2 ) ∈ [0 , 1] n , if there exists some i k ∈ [ n k ] , ∀ k ∈ [2] such that θ
i
k
= 1 2 , then θ is an equilibrium point; in fact, it is not difﬁcult to see that these are the only interior equilibrium points of G , which implies that an interior equilibrium point is reachable if and only if S k (cid:54) = ∅ , ∀ k ∈ [2] .
In our ﬁrst result, we prove that if the Nash Equilibrium of the unrestricted case is not reachable by both teams due to the enforced constraints, then the dynamics are bound to converge to a point which only depends on those restrictions.
Theorem 4.
Under Assumption 3, if S k = ∅ , ∀ k ∈ [2] , the restricted RD of G converge to [Equation] [Equation] i
[Equation]
even or
[Equation]
if
[Equation]
[Equation]
otherwise
[Equation]
[Equation]
where S :
= { ( k, i ) | k ∈ [2] , i ∈ [ n k ] , α k,i > 12 } .
The key idea behind this result is that at any given time t ≥ 0 , the direction of a strategy θ k,i ( t ) only depends on whether | S | , the total number of members who can access the uniform strategy, is odd or even.
Thus, we can decouple the evolution of the dynamics of each of the members and analyze its behavior separately.
On the other hand, if an equilibrium point of the unrestricted setting is reachable by both teams, i.e., S k (cid:54) = ∅ , ∀ k ∈ [2] , then the dynamics converge to an invariant set whose degrees of freedom depend on the number of members who have access to the uniform strategy, ( 12 , 12 ) .
Theorem 5.
Under Assumption 3, if S k (cid:54) = ∅ , ∀ k ∈ [2] , then the restricted RD of G converge to an invariant set deﬁned by the | S 1 | + | S 2 |− 1 independent invariant functions, V i 1 ,i 2 ( θ ) , i k ∈ S k , ∀ k ∈ [2] , where V i 1 ,i 2 ( θ ) is given as in (20) .
One can prove this result by partitioning the time based on the set of members that have halted.
The analysis of V i 1 ,i 2 ( θ ) in each time-partition is almost trivial, and the result follows by a continuity argument on V i 1 ,i 2 ( θ ) .
For completeness, we remark that the intermediate case where ∃ k, k (cid:48) ∈ [2] with S k = ∅ , and S k (cid:48) (cid:54) = ∅ , still leads to the convergence of RD, though the actual point of convergence depends on the initial conditions θ (0) .
However, its analysis merely combines elements of the other two cases without providing any interesting insight, and, therefore, we omit it.
These results depict how the behavior of RD depends on the parameter space of the game.
Notably, we show that if no equilibrium point is feasible, the RD converge to a point described entirely by the strategy space restrictions.
On the other hand, if an equilibrium point is feasible, we prove the existence of a maximal number of invariant functions, with close connections to the KL divergence.
The latter does not merely show that the RD cycle in this setting, but that they actually converge to an invariant set with speciﬁc degrees of freedom , and which we characterize.
For exemplary parameterizations that visualize the behavior of RD in such restricted settings, please see §B.1.
E XPERIMENTAL RESULTS
NHG dynamics:
The multi-dimensional case.
As a ﬁrst experiment, we consider a hidden convexconcave objective
[Equation]
where M is the payoff matrix of the Rock-Paper-Scissors game (see e.g. Gidel et al.
We note F θ ( x i ) =
[ F θ ] i , i ∈ [3] , and 1 3 , i.e., the uniform distribution, is the game’s equilibrium.
The mappings F and G are 2-layer MLP with 130 parameters and GELU non-linearities (Hendrycks &
Gimpel, 2016).
In Figure 1, we depict a comparison between the performance of GDA and NHG dynamics on the task of solving min θ max φ Ψ ( θ , φ ) .
We remark, our NHG dynamics converge smoothly (Figure 1 (Right) ), compared to GDA (Figure 1 (Center) ), which fail to converge.
The value of the Lyapunov function of the game described is depicted in (Figure 1 (Left) ).
NHG dynamics: GANs.
For our second experiment, we implement the NHG dynamics to train a
GAN.
We consider a synthetic experiment to learn a sine wave sampled uniformly from 0 to π with 1024 observations.
Regarding the GAN, we consider a Flow-GAN architecture where our generator,
[Figure] G , is a Real NVP consisting of 8 coupling layers (Dinh et al., 2017; 2014), and the discriminator is a 4-layer MLP with 256-128-64-1 output features.
We adapt K-FAC (Martens & Grosse) as an approximator to compute NHG, with a learning rate of 10 − 4 , selected using the standard parameter optimizer package Optuna (Akiba et al., 2019).
We used gradient clipping with gradient clip to a maximum norm of 1 on, both the discriminator and the generator, to stabilize the optimization.
As observed in (Figure 2 (Center) ), by iteration 200 , the GAN optimized using K-FAC learns the true sinusoid almost perfectly and converges to a better Wasserstein-1 score than the conventional GDA (Figure 2 (Right) ).
The experiments reveal that our approach provides good performance and convergence guarantees.
However, we observe instabilities during training, and lack of convergence in certain instances, which stay in line with the empirical observations regarding the difﬁculty of GAN training, and the instability of matrix inversions close to singularities.
We remark that this experiment goes slightly beyond the theoretical results: while in theory, we assume a natural gradient ﬂow on a “full-batch”, our experiments are based on discrete stochastic updates, and an approximation of the pseudo-inverses of the matrices (see (D1)) using K-FAC.
Thus, this experiment acts as a proof-of-concept rather than a large-scale comparison between NHG and GDA.
The details of both experiments can be found within the source-code ﬁles included with this work.
[Figure]
D ISCUSSION
In this work, we proposed a novel version of Gradient Descent Ascent dynamics in Hidden ConvexConcave games, a subset of non-convex non-concave games.
In this class of games, which includes
GANs, the utility is convex-concave in the function space.
Still, training happens in the parameter space, where the mappings between the input and output are non-convex non-concave functions.
Our analysis of the convergence of the new type of dynamics, Natural Hidden Gradient (NHG) dynamics, uses ideas from Game Theory and Dynamical Systems.
We proved global convergence guarantees for NHG in HCC games to local stationary points via Lyapunov function analysis in the ﬁnite-sum case.
To the best of our knowledge, such a non-local convergence result in HCC games and GAN-like settings, is one of the ﬁrst of its kind.
We explored the dynamics of gradient ﬂows induced by different Banach spaces (e.g., the Fischer information geometry) that led to the discovery of Lyapunov functions suited to these geometries.
This work shows a deep connection between the natural gradients and the Lyapunov function analysis of complex non-convex non-concave game dynamics that pave the way to a better understanding of learning in games.
We show that we can obtain promising experimental results on practical GANs using NHG and standard gradient approximation techniques such as KFAC.
However, we are aware that the current formulation of NHG may be challenging to scale up to large neural networks because of the tensor pseudo-inversion step that is part of the dynamics.
Investigating experimentally novel versions of Gradient Descent Ascent dynamics with a richer set of experiments and developing techniques to scale our results for large neural networks is a natural direction for our future work.
S TATEMENT OF R EPRODUCIBILITY
We made sure to provide sufﬁcient details to ensure the reproducibility of our results.
The complete proofs of the theoretical results can be found in §A, and all the assumptions have been stated and are referenced in each statement.
We provide details regarding the experimental results, such as code language, required libraries and parametrization to execute and reproduce the experiments in section 6 and §B. We also include the source code ﬁles and the necessary input ﬁles in the supplementary material that accompanies this work
A O MITTED PROOFS A. O MITTED PROOFS OF S ECTION 3.
Proposition A.1.
Let us consider P ( X ) the space of probability distributions on a set X ⊆ R with the metric induced by the Kullback–Leibler (KL) divergence.
If n := | X | = dim S , and p θ := θ ∈ P ( X ) , then the natural gradient ﬂow of the Fisher information matrix in (8) is the natural gradient ﬂow of the Shahshahani metric induced by the metric tensors in (9) .
Proof.
All we need to show is that F θ = S θ , ∀ θ ∈ S .
Simply, note that ∀ i, j ∈ [ n ] :
[Equation]
A. O MITTED PROOFS OF SECTION 3.
Let G =
( n, S :=
[ m 1 ] × . . .
×
[ m n ] , u : S → R n ) be a n -player (exact) potential game with payoff function u and potential Φ : S → R , and let M := ∆ m 1 × . . .
× ∆ m n be the mixed-strategy space of G . We are going to consider the RD of G in int M = int ∆ m 1 × . . . × int ∆ m n as given in the following proposition.
Proposition A.2.
The Replicator Dynamics of G in int M are given by the following dynamical system of equations:
[Equation]
where Ψ( θ ) :
=
E s i ∼ θ
i i ∈
[ n ] [Φ( s )] is the the expected potential function of the game.
Proof.
First note that
[Equation]
From which follows that
[Equation]
Next
[Equation] let us consider a point θ ∈ int M
[Equation]
and note that the following proposition holds:
Proposition A.3.
For any θ ∈ int M = int ∆ m 1 × . . .
× int ∆ m n , the tangent plane of int M at θ is
[Equation]
Proof.
We ﬁrst note that int M is an open-subset of the surface D := { µ ∈ R m 1 × . . .
×
R m n |
U i ( µ ) = 1 } ,
where U i ( µ ) = (cid:80) j µ i,j = 1 ; hence, T θ (int M )
=
T θ D , up to isomorphism.
Let r ( t ) be a smooth curve in D with r (0)
= θ .
Then, by deﬁnition, U i ( r ( t ))
= 1 for all i , and, by differentiating with respect to t , we have
[Equation]
However, ddt r ( t ) is tangent to D for all t ; hence, ddt r ( t ) (cid:12) (cid:12)(cid:12) t =0 ∈ T θ D .
It follows that ∇ U
i ( r (0))
= ∇ U
i ( θ ) is normal to T θ D for all i , and ,thus, we have
[Equation]
Finally, notice that dim { µ ∈ R m 1 × . . .
×
R m n | (cid:80) j µ i,j = 0 } =
dim D , which implies the proposition.
Proposition 1.
The Replicator Dynamics of a potential game G with potential function Φ is an (extended) Shahshahani gradient in int(∆ m 1 × . . .
×
∆ m n ) having potential Ψ [Equation] θ
[Equation]
s i ∼ θ i i ∈
[ n ] [ [Equation]
Proof.
Let θ ∈ int M :=
int ∆ m 1 × . . .
× int ∆ m n .
Then for every ξ ∈ T θ (int M ) we have:
[Equation]
Hence, by deﬁnition, we have that ∇ Ψ( θ ) = ˙
θ , where the Del operator is deﬁned with respect to the Shahshahani metric, which implies the lemma.
Now, let us consider a different class of games, which we deﬁne as 2-Team Zero-Sum games .
Deﬁnition 2.
A 2-Team Zero-Sum game, G = ( m := m 1 + m 2 , S :=
[ n 1 ] × . . .
× [ n m ] , u : S
→ R m ) , is a game between m players with strategy-space S whose utility function, u , satisﬁes the following:
[Equation]
for some function Φ : S → R .
We are going to, collectively, refer to the ﬁrst m 1 players of a 2-Team Zero-Sum game, G = ( m := m 1 + m 2 , S :=
[ n 1 ] × . . .
× [ n m ] , u : S
→ R m ) , as team 1 and to the rest of them as team 2.
Notice that if we consider each team as a single player, then the two teams are playing a 2-Player
Zero-Sum game, G (cid:48)
= (2 , S (cid:48) :
= S 1 × S 2 , u (cid:48) : S (cid:48)
→ R 2 ) where S 1 = [ n 1 ] × . . .
× [ n m 1 ] , S 2 =
[ n m 1 +1 × . . .
× n m ] , and the utility function u (cid:48) k is the total utility of the corresponding team k , i.e.
∀ s ∈ S we have
[Equation]
Note that, by deﬁnition, the 2-Team Hidden Matching Pennies game (see section 5) is a a 2-Team
Zero-Sum game.
Let us consider the Replicator Dynamics of G given by the following dynamical system of equations:
[Equation]
where, as before, we can rewrite (D5) in terms of Ψ( x ) :
=
E S k ∼ x k
[Φ( S )] as
[Equation]
Notice that the Replicator equations of each team are similar to the Replicator equations of a Potential game.
In fact, it’s not difﬁcult to prove the following lemma.
Lemma 1.
Let G =
( m := m 1 + m 2 , S :=
[ n 1 ] × . . .
× [ n m ] , u : S
→ R m ) be a 2-Team
Zero-Sum game.
If we assume the strategies of team 2 to be time-invariant, then the Replicator
Dynamics of G , given by (D5) (or (D6) , equivalently), is a ( m 1 -scaled)
Shahshahani gradient in int M := int ∆ n 1 × . . .
× int ∆ n m with potential Ψ (cid:48) 1 ( θ ) =
Ψ ◦ Π 1 ( θ ) where Ψ( θ ) :
=
E s i ∼ θ
i
[Φ( s )] , and Π 1 : int M → int M is the natural projection
[Equation]
i
[Equation]
if i in team k
[Equation]
Likewise, if we assume the strategies of team 1 to be time-invariant, then the Replicator Dynamics of G is a ( m 2 -scaled)
Shahshahani gradient in int M with potential Ψ (cid:48)
= − Ψ ◦
Π 2 ( θ ) .
Proof.
This proof is similar to the proof of Proposition 1, but we need to perform the correct projection before applying the deﬁnition of a gradient ﬂow.
Let θ ∈ int M .
Without any loss of the generality, let us assume that the strategies of team 2 are time-invariant, i.e., ˙ θ i,j = 0 for all i ≥ m 1 .
Then, for every ξ ∈ T θ (int M ) we have
[Equation]
Hence, by deﬁnition, we have that d Ψ (cid:48) 1 ( θ ) = ˙
θ (cid:91) , which implies the theorem.
Proposition 2 (Metric tensors of the model space) .
For any θ , we have that
[Equation]
A θ [Equation] = E [
[Equation]
( x
[Equation]
Proof.
To prove that we use the fact that for all x ,
[Equation]
Then we have that
[Equation]
which concludes the proof
[Equation]
Proposition 3 (Natural Hidden Gradient dynamics) .
The ﬂow induced by the geometry (11) is
[Equation]
where A θ := E x ∼ p x [ ∇ θ F θ ( x ) ∇ θ F θ ( x ) (cid:124) ] and B φ :=
E x (cid:48) ∼ p x (cid:48)
[ ∇ φ G φ ( x ) ∇ φ G φ ( x ) (cid:124) ] and C † denotes the pseudo-inverse of a matrix C .
Proof.
The proposition directly follows from Proposition 2 applied to the deﬁnition of (7).
However, in order to convey more intuition we could use the deﬁnition that the natural gradient ﬂow of the objective function f induced by the distance d if
[Equation]
By using the L 2 distance and noting that when λ → 0 we have f ( θ + λδ θ ) =
f ( θ )+ λ ∇ f ( θ ) (cid:124)
δ
θ + o ( λ ) and 12 λ 2 (cid:107) F θ + λδ θ
− F θ (cid:107) 2 = 12 (cid:104) δ θ , A θ δ θ (cid:105) + o (1) we have that the RHS of the equation above is
[Equation]
which is minimized for δ θ = − A † θ ∇ f ( θ ) .
A. O MITTED P ROOFS OF SECTION
Proposition 4.
In the uni-dimensional case, the Natural Hidden Gradient ﬂow D1 takes the form of
[Equation]
and
[Equation]
Proof.
For the uni-dimensional case, A θ = ∇ f ( θ ) ∇ f ( θ ) (cid:124) is a rank- 1 matrix that projects any vector in the direction of ∇ f ( θ ) and scales it by (cid:107)∇ f ( θ ) (cid:107) 2 .
Thus, by deﬁnition of the pseudoinverse, we have that for any vector u ∈ R d :
[Equation]
Finally, notice that
[Equation]
which leads to the stated proposition.
Theorem 1.
Let Ψ be the payoff of an HCC game deﬁned as in (13) and consider the game dynamics in (D2) .
Then, the quantity
[Equation]
(14) is a Lyapunov function, i.e., it is positive, non-increasing, and null if and only if it evaluated at a game solution.
Moreover, if L is strictly convex-concave, we have that V is decreasing and that any limit point ( θ , φ ) satisﬁes ∇
[Equation]
Proof.
By taking the time derivative of V ( θ , φ ) , we get
[Equation]
where the last inequality holds because L is convex-concave.
Speciﬁcally, let L : R M × R N → R be any differentiable convex-concave function with saddle point ( f ∗ , g ∗ ) , then we have, respectively [Equation]
g [Equation]
g [Equation] g [Equation]
By adding the two inequalities, we get
[Equation] [Equation]
where the last inequality holds because ( f ∗ , g ∗ ) is a saddle point.
Hence, by deﬁnition, L is a
Lyapunov function.
Finally, when L is strictly convex-concave, if there exists a limit point of ( θ , φ ) that is not a point where ∇ θ L ( f ( θ ) , g ( φ ))
=
and ∇ φ L ( f ( θ ) , g ( φ ))
= 0 then by strict convex-concavity we get that V should decrease by a “signiﬁcant enough amount” to create a contradiction with the fact that
V does converge.
Next, before proving the general case of Theorem 1, we’ll have to prove the following lemma:
Lemma 2.
Let ( u i ) i ∈
[ n ] be a linearly independent family of vectors on R d .
Then, the matrix
[Equation]
is the Gram matrix of the family ( u i ) i ∈
[ n ] and and we have that (cid:104) u
i , A † u j (cid:105) =
δ i,j , ∀ i, j ∈ [ n ] .
Proof.
Let us introduce the matrix P :=
[ u 1 , . . . , u n ] (cid:124) ; we can easily verify that A = P (cid:124) A .
Let
P = UDV (cid:124) be the SVD decomposition of P and, thus, A † =
V ( D [Equation] D ) † V (cid:124) where D (cid:124) D is a diagonal matrix.
Then, we have
[Equation]
To conclude this lemma, we just need to notice that the matrix D is a matrix with non-zero entries on the diagonal and, since we assumed that the vectors in ( u i ) i ∈ [ n ] are linearly independent, we have that
D i,i > 0 , ∀ i ∈ [ n ] .
Thus, by a direct computation, we get that
[Equation]
n which leads to (cid:104) u i , A † u i (cid:105) = δ i,j .
Theorem 2.
Let Ψ be the payoff of a ﬁnite-sum HCC game given by (15) and consider the game dynamics in (D3) .
Under Assumption 1, Assumption 2, and that d > n we have that the quantity
[Equation]
is a Lyapunov function, i.e., is positive, non-increasing and null if and only if evaluated at a game solution.
Moreover, if L is strictly convex-concave, V is decreasing as long as ( θ , φ ) (cid:54) =
( θ ∗ , φ ∗ )
and if L is a µ -strongly convex-concave function we have that V is decreasing exponentially as
V ( θ , φ ) =
V ( θ 0 , φ 0 ) exp( − µt ) .
Proof.
Similarly as the proof of Theorem 1, we consider the time derivative of V ( θ , φ ) .
For compactness, we are going to simplify the notation slightly by setting F i :=
F θ ( x i ) , G j :=
G φ ( x (cid:48) j ) ,
F ∗ i :=
F ∗ ( x i ) , G ∗ j :=
G ∗ ( x (cid:48) j ) , and L i,j :=
L x i , x (cid:48)
j .
[Equation]
From Lemma 2 we have that (cid:104)∇ θ F i , A † θ ∇ θ F k (cid:105) =
n · δ i,k , ∀ i ∈ [ n ] , k ∈ [ m ] , and that (cid:104)∇ φ G j , B † φ ∇ φ F k (cid:105) = m · δ j,k .
Hence, it follows
[Equation]
[Equation] G
[Equation]
G
[Equation]
where [ ∂ F L ( F, G )] ( i,j ) :
= ∂L
i,j ( F,G j ) ∂F
[Equation]
j , and F , G , F ∗ , and G ∗ are indexed by ( i, j ) , as well (while repeating vector elements as necessarily).
Thus, using the same reasoning as in Theorem 1, it follows that if L is convex-concave we have that V is nonincreasing, and, if L is strictly convex-concave, ˙ V ( θ , φ ) < 0 whenever ( θ , φ ) (cid:54) =
( θ ∗ , φ ∗ ) .
Finally, if L is µ -strongly convex-concave, we have by deﬁnition that
[Equation]
Thus we conclude that V ( θ , φ ) ≤ V ( θ 0 , φ 0 ) exp( − µt ) .
A. O MITTED PROOFS OF SECTION
In order to prove (18) [Equation] and (D9) we are [Equation]
ﬁrst
[Equation]
going to prove the following useful lemma:
Lemma 3.
Let h θ
( i ) :
=
E x j ∼ Ber( θ j )
[XOR( x 1 , . . .
, x i )] where θ ∈ [0 , 1] n , and i ≤ n .
Then the following equality holds:
[Equation]
Proof.
The easiest way to prove this relationship is by induction on n ∈ N .
For n = 1 , we only need to verify that (28) holds for i = 1 .
Indeed, we have,
[Equation]
Next [Equation] let us assume that Lemma 3 holds for some n = n (cid:48) ∈ N .
We are going to prove that Lemmaalso holds for n = n (cid:48) + [Equation] and this comes down in proving that (28) holds for i = n (cid:48) + 1 :
[Equation]
[Equation]
And we that, the proof by induction is complete
[Equation]
Proposition A.4.
Let G =
( n := n 1 + n 2 , S :
= { 0 , 1 } n , u : S → R n ) be a 2-Team Hidden Matching
Pennies game with payoff function given by (17) .
Then, the expected payoff of the i -th member of team k is given by (18) .
Proof.
The ﬁrst equality follows, trivially, from the deﬁnition of Φ( s ) .
All is left to prove is that −
[Equation]
i [Equation]
θ k,i ) [
[Equation]
[Equation] ) for all θ ∈ [0 , 1] n .
We deﬁne, x k
= XOR( s k ) , k ∈ [2] .
Note that, by deﬁnition, x k ∼ Ber( p k ) , where p k :=
E s k,i ∼ Ber( θ k,i )
[XOR( s k )] ; hence,
[Equation]
Proposition A.5.
Let G =
( n := n 1 + n 2 , S :
= { 0 , 1 } n , u : S → R n ) be a 2-Team Hidden Matching
Pennies game with payoff function given by (17) .
Then, the RD of G is given by the dynamical system of equations:
[Equation]
Proof.
From (18) we have that
[Equation]
Then, by the deﬁnition of RD of G
[Equation]
we get
[Equation]
Theorem 3.
Consider the Replicator Dynamics of G .
Given any interior initial condition, the resulting orbit is a cycle that satisﬁes the following n 1 + n 2 − 1 independent invariant functions:
[Equation]
[Equation]
Proof.
For all k ∈ [2] and i ∈ [ n k ] , we have
[Equation]
Subsequently, we have
[Equation]
Observe that the above imply the existence of n 1 + n 2 − 1 independent invariant functions.
Hence, the dynamics converge to a limit set of a single degree of freedom, i.e., a cycle.
Proposition A.6.
Let G =
( n := n 1 + n 2 , S :
= { 0 , 1 } n , u : S → R n ) be a 2-Team Hidden
Matching Pennies game with payoff function given by (17) .
Given a mixed-strategy proﬁle
( θ , 1 − θ ) , θ :=
( θ 1 , θ 2 ) ∈ (0 , 1) n , ( θ , 1 − θ ) is an equilibrium of G if and only if ∃ i k , ∀ k ∈ [2] such that θ k,i k
= 12 , ∀ k ∈ [2] .
Proof.
We know that G is equivalent to a Hidden Matching Pennies game (Example 1) with payoff Ψ ( θ ) =
(1 − 2 f ( θ 1 ))
(1 − 2 g ( θ 2 )) , where f ( θ 1 ) :
=
E s 1 ,i ∼ Ber( θ 1 ,i )
[XOR( s 1 )] , and g ( θ 2 ) :=
E s 2 ,i ∼ Ber( θ 2 ,i )
[XOR( s 2 )] are its hidden mappings.
The only fully mixed-Nash equilibrium of this Hidden Matching Pennies game is ( f ( θ 1 ) , g ( θ 2 ))
= ( 12 , 12 ) ; hence, a mixed-strategy proﬁle ( θ , 1 − θ ) is a fully mixed-Nash equilibrium of G , if and only if f ( θ 1 ) =
g ( θ 2 ) = 12 .
From f ( θ 1 ) =
12 we get
[Equation]
We remark that, for any i k ∈ [ n k ] , k ∈ [2] ,
V i
[Equation]
where H ( p, q ) is the cross-entropy of a distribution q relative to a distribution p .
In other words, every one of the invariant functions in (20) is the weighted sum of the cross entropy of two mixedstrategies (of the i 1 -th member of team 1, and the i 2 -th member of team 2) relative to the uniform strategy.
That is, each invariant function measures (up to a constant) the Kullback–Leibler divergence of two opposing members to an actual equilibrium of the game.
Notice that, for any equilibrium point of the Hidden Matching Pennies game, there exists at least on such pair of opposing members i 1 ∈ [ n 1 ] , and i 2 ∈ [ n 2 ] such that
[Equation]
For the rest of this section we are going to consider the RD of a 2-Team Hidden Matching Pennies game G = ( n := n 1 + n 2 , S :
= { 0 , 1 } n , u : S → R n ) in a restricted setting, given by the following dynamical system of equations:
[Equation]
where D k,i ( θ ) :
= (cid:81) k (cid:48) ∈ [2] (cid:81) j ∈ [ n k (cid:48) ]
( k (cid:48) ,j ) (cid:54) =(
k,i ) (1 − 2 θ k (cid:48) ,j ) , k ∈ [2] i ∈ [ n k ] , and where we restrict each mixed-strategy proﬁle ( θ , 1 − θ ) such that θ k,i ∈
S k,i :=
[ α k,i , β k,i ] , ∀ k ∈ [2] , i ∈ [ n k ] .
We let Ω be an orbit deﬁned by this RD whose initial conditions satisfy Assumption 3.
This assumption serves a dual purpose.
To begin with, it ensures that any orbit is initialized inside the restricted parameter space that is deﬁned by S k,i .
Furthermore, it makes sure the initial strategy proﬁle has full support, i.e., ( θ k,i (0) , 1 − θ k,i (0)) is an interior point of the simplex for all k ∈ [2] , and i ∈ [ n k ] .
It is easy to see that any dimension of the strategy space initially without support is impossible to be updated by the RD in (D10); hence, it would be irrelevant for the analysis.
Before we proceed, we are going to introduce a couple of useful lemmas.
Lemma 4.
∀ k ∈ [2] , and i ∈ [ n k ] if θ k,i (0)
∈
S k,i , then θ k,i ( t )
∈
S k,i , ∀ t ≥ 0 .
Proof.
Let k ∈
[2] and i ∈ [ n k ] such that θ k,i (0)
∈
S k,i = ⇒
α k,i ≤ θ k,i (0) ≤ β k,i .
We are going to prove our case by abduction.
Suppose ∃ t 0 > 0 such that θ k,i ( t 0 ) / ∈ S k,i and, without any loss of the generality, let us assume that θ k,i ( t 0 ) > β k,i .
We let T = { t ∈ (0 , t 0 ) | θ k,i ( t )
=
β k,i and we note that, since θ k,i (0) ≤ β k,i , it is implied by the continuity of θ k,i ( t ) (Equation D10) and by the Intermediate Value Theorem that T (cid:54)
= ∅ .
Finally we deﬁne t max = max( T ) .
Let us now consider the value of θ k,i ( t ) for some t ∈ ( t max , t 0 ) and observe that if θ k,i ( t )
=
β k,i we have t ∈
[Equation]
= ⇒
[Equation]
max
[Equation]
=
[Equation]
max = ⇒
t max < t ≤ t max .
This contradiction implies that θ k,i ( t ) (cid:54) =
β k,i .
However, if θ k,i ( t ) <
β k,i , the Intermediate Value
Theorem, once again, implies ∃ t (cid:48) ∈ ( t, t 0 ) such that θ k,i ( t (cid:48) )
=
β k,i , which as before implies t max < t (cid:48) ≤ t max .
Hence, it must be the case that, for all t ∈ ( t max , t 0 ) :
[Equation]
However, by applying the Mean Value Theorem , if follows ∃ t (cid:48) ∈ ( t max , t 0 ) such that
[Equation]
which is once again a contradiction.
Lemma 5.
Under Assumption 3, if S k = ∅ , ∀ k ∈ [2] then the following hold:
(a) Ψ 1 ,i 1 ( θ ( t ))
Ψ 2 ,i 2 ( θ ( t )) < [-
-]0 for all i k ∈ [ n k ] , k ∈ [2] , and t ≥ 0 .
(b) Ψ k,i k
( θ ( t )) preserves sign for all i k ∈ [ n k ] , k ∈ [2] .
Proof.
Since S k = ∅ for all k ∈ [2] , it follows, by deﬁnition, that
[Equation]
Furthermore, by Assumption 3, we have that θ k,i (0)
∈
S k,i for all k ∈ [2] , i ∈
[ n k ] .
Hence, it follows, by Lemma 4, that ∀ k ∈
[2] , i ∈ [ n k ] , and t ≥ 0 :
[Equation]
Then, by Equation 18, we have
[Equation]
That implies (a).
We can now condition on Ψ k,i ( θ (0)) .
Since Ψ k,i ( θ ( t )) (cid:54) = 0 for all t ≥ 0 , it must be the case that either Ψ k,i ( θ (0)) > 0 or Ψ k,i ( θ (0)) < 0 ; let us, ﬁrst, assume the former case.
We are going to prove, by abduction, that Ψ k,i ( θ ( t )) > 0 , ∀ t ≥ 0 .
Suppose ∃ t (cid:48) > 0 such that Ψ k,i ( θ ( t (cid:48) )) ≤ 0 .
Since Ψ k,i ( θ ( t )) (cid:54) = 0 for all t ≥
it follows that Ψ k,i ( θ ( t (cid:48) ))
< 0 must be the case.
However, by the continuity of Ψ k,i ( θ ( t )) and the Intermediate
Value Theorem, it follows that ∃ t (cid:48)(cid:48) ∈ (0 , t (cid:48) ) such that Ψ k,i ( θ ( t (cid:48)(cid:48) ))
= 0 , and that is, indeed, a contradiction.
It follows that it must be the case Ψ k,i ( θ ( t )) > 0 , ∀ t ≥ 0 .
In a similar manner we may prove that Ψ k,i ( θ (0)) < 0 =
⇒ Ψ k,i ( θ ( t )) < 0 , ∀ t ≥ 0 ; hence, (b) holds.
Theorem 4.
Under Assumption 3, if S k = ∅ , ∀ k ∈ [2] , the restricted RD of G converge to θ ∗ 1 ,i = (cid:26) α 1 ,i , if β 1 ,i <β 1 ,i , otherwise i ∈ [ n 1 ] θ ∗ 2 ,i = (cid:26) β 2 ,i , if β 2 ,i <α 2 ,i , otherwise i ∈
[ n 2 ] (cid:124) (cid:123)(cid:122) (cid:125) if | S | is even or θ ∗ 1 ,i = (cid:26) β 1 ,i , if β 1 ,i < α 1 ,i , otherwise i ∈ [ n 1 ] θ ∗ 2 ,i = (cid:26) α 2 ,i , if β 2 ,i <β 2 ,i , otherwise i ∈ [ n 2 ] (cid:124) (cid:123)(cid:122) (cid:125) if | S
[Equation]
(21) where S :
= { ( k, i ) | k ∈ [2] , i ∈ [ n k ] , α k,i > 12 } .
Proof.
We begin by conditioning on the value of | S | and, since the proof is similar in both cases, and without any loss of the generality, we are going to assume that | S | is even.
Since S k = ∅ for all k ∈ [2] , i.e., 12 / ∈ S k,i for all k ∈ [2] , i ∈ [ n k ] , and, by Assumption 3, θ k,i (0)
∈
S k,i :=
[ α k,i , β k,i ] for k ∈ [2] , i ∈ [ n k ] it follows that θ k,i (0) > 12 , if α k,i > 12 ; θ k,i (0) < 12 ; otherwise.
Then, for all i ∈ [ n 1 ] , we have
[Equation]
since | S | is even.
Subsequently, by Lemma 5, Ψ 1 ,i ( θ ( t )) > 0 , ∀ t ≥ 0 .
That implies that Ψ 1 ,i ( θ ( t )) > 0 , ∀ i ∈ [ n 1 ] , t ≥ 0 , and, hence, by Lemma 5, we also have that Ψ 2 ,i ( θ ( t )) < 0 , ∀ i ∈ [ n 2 ] , t ≥ 0 .
Take any k ∈ [2] , and i ∈ [ n k ] .
In order to complete the proof, we’ll have to condition on the value of k , and on whether ( k, i ) ∈ S .
There are four cases in total that we”ll have to consider, but, since in all of them we follow a similar reasoning, we’ll just go ahead and demonstrate the single case of k = 1 , and ( k, i ) ∈ S .
We are going to prove that there exists t ∗ 1 ,i ≥ 0 such that θ 1 ,i ( t ) =
β 1 , i, ∀ t ≥ t ∗ 1 ,i .
By Assumption 3, and Lemma 4, we have α 1 ,i ≥ θ 1 ,i ( t ) ≥ β 1 ,i for all t ≥ 0 .
Furthermore, since (1 , i ) ∈ S , and, since S k = ∅ , we have that α 1 ,i ≥ β 1 ,i > 12 .
That is, θ 1 ,i ( t ) > 12 for all t ≥ 0 , and, hence,
[Equation]
Then, by Equation D10 it follows that ∀ t ≥ 0 :
[Equation]
Hence, θ 1 ,i = β 1 ,i is the single attracting point of the dynamical system described by D11 and, hence, by deﬁnition, ∃ t ∗ 1 ,i ≥ 0 such that θ 1 ,i ( t )
= θ ∗ 1 ,i = β 1 ,i , ∀ t ≥ t ∗ 1 ,i .
Similarly, we can prove that ∃ t ∗ k,i ≥ 0 such that θ k,i ( t )
= θ ∗ k,i , ∀ t ≥ t ∗ k,i , and by letting t ∗ = max k ∈ [2] i ∈ [ n k ] ( t ∗ k,i ) , Theoremfollows.
Theorem 5.
Under Assumption 3, if S k (cid:54) = ∅ , ∀ k ∈ [2] , then the restricted RD of G converge to an invariant set deﬁned by the | S 1 | + | S 2 |− 1 independent invariant functions, V i 1 ,i 2 ( θ ) , i k ∈ S k , ∀ k ∈ [2] , where V i 1 ,i 2 ( θ ) is given as in (20) .
Proof.
We are going to begin by deﬁning the following time-dependent set:
[Equation]
We perform a time partitioning based on the values of C ( t ) , t ≥ 0 .
Speciﬁcally, we let P = { ( t 1 , t 2 )
| t 1 , t 2 ∈ T : t / ∈ T , ∀ t ∈ ( t 1 , t 2 ) } , where
[Equation]
It is not difﬁcult to see that the continuity of θ ( t ) implies that T consists entirely of isolated points and, due to this fact, P is well-deﬁned.
Let I ∈ P be one of these partitions.
We are going to prove that ˙
V i 1 ,i 2 ( θ ( t )) ≥ 0 , ∀
i k ∈ S k , ∀ k ∈ [2] , t ∈ I .
First of all, observe that, by deﬁnition, the following two properties have to hold:
(a) C ( t 1 ) =
C ( t 2 ) , ∀ t 1 , t 2 ∈ I
(b) By Equation D10, we have that ∀ t ∈
I ,
[Equation]
For any i k ∈ S k , k ∈ [2] , we proceed by conditioning on the value of C ( t ) , t ∈ I and, since C t 1 ) =
C ( t 2 ) , ∀ t 1 , t 2 ∈
I , we distinct only three cases:
(a) ( k, i k ) ∈ C ( t ) , ∀ k ∈ [2] , t ∈ I .
(b) ( k, i k ) / ∈ C ( t ) , ∀ k ∈ [2] , t ∈ I .
(c) ∃ k, k (cid:48) ∈ [2] such that ( k, i k ) ∈ C ( t ) and ( k (cid:48) , i k (cid:48) )
/ ∈ C ( t ) for all t ∈ I .
The analysis of the ﬁrst two cases is relatively straightforward.
Let t ∈
I ; then, if ( k, i k ) ∈
C ( t ) , ∀ k ∈ [2] is the case, then by Equation D12, we have that ˙ θ k,i k ( t )
= 0 , ∀ k ∈ [2] and, hence,
[Equation]
On the other hand, if ( k, i k ) / ∈ C ( t ) , ∀ k ∈ [2] is the case, then, once again, by Equation D12, we have that ˙ θ k,i k ( t ) =
( − 1) k [Equation] 1 2 n k θ k,i (1 − θ k,i )
D k,i ( θ ) , ∀ k ∈ [2] .
That implies, ˙
[Equation]
In order to proceed with the ﬁnal case we’ll ﬁrst need to prove a small technical lemma:
Lemma 6.
If ( k, i ) ∈ C ( t ) for some t ∈
I , I ∈ P = { ( t 1 , t 2 ) | t 1 , t 2 ∈ T : t / ∈ T , ∀ t ∈ ( t 1 , t 2 ) } then one of the following holds:
a) θ k,i ( t ) =
α k,i , ∀ t ∈ I .
b) θ k,i ( t )
=
β k,i , ∀ t ∈ I .
To see that, let I ∈ P and ( k, i ) ∈ C ( t ) for some t ∈ I .
Then it, holds, by deﬁnition, that
C ( t 1 ) = C ( t 2 ) , ∀ t 1 , t 2 ∈ I = ⇒ C ( t (cid:48) )
=
C ( t ) , ∀ t (cid:48) ∈
I
And, since ( k, i ) ∈ C ( t ) , we also have that ( k [Equation] i [Equation]
∈
[Equation]
( t (cid:48) ) , ∀ t (cid:48) ∈
I =
⇒
θ k,i ( t (cid:48) )
∈ { α k,i , β k,i }
Let us assume
that θ k,i ( t ) =
α k,i and suppose ∃ t (cid:48) ∈
I such that θ k,i ( t (cid:48) )
(cid:54) =
α k,i = ⇒ θ k,i ( t (cid:48) )
=
β k,i and α k,i (cid:54) =
β k,i .
By the Intermediate Value Theorem, it follows ∃ t (cid:48)(cid:48) ∈ (min( t, t (cid:48) ) , max( t, t (cid:48) ))
such that θ k,i ( t (cid:48)(cid:48) ) ∈ ( α k,i , β k,i ) =
⇒ ( k, i ) / ∈ C ( t (cid:48)(cid:48) )
That is a contradiction, and, hence, a) holds.
a) follows by a similar argument, assuming θ k,i ( t ) =
βk, i is the case, instead.
Having established Lemma 6, we continue with the proof of Theorem 5.
Let us assume that ∃ k, k (cid:48) ∈
[2] such that ( k, i k ) ∈ C ( t ) and ( k (cid:48) , i k (cid:48) ) / ∈ C ( t ) .
Then Lemma 6 implies that either θ k,i k ( t (cid:48) )
=
α k,i k , ∀ t (cid:48) ∈
I or θ k,i k
( t (cid:48) )
=
β k,i k , ∀ t (cid:48) ∈ I .
Let us assume, without any loss of the generality, the former case and let us consider some t ∈ I .
Since S k,i k ∈ S k , i.e., α k,i k ≤ 12 , that implies 1 − 2 θ k,i k ( t ) ≥ 0 .
Next, Let us consider the value of ( − 1) k − 1 D k,i ( θ ) .
We are going to prove by abduction that ( − 1) k − 1 D k,i ( θ ) ≤ 0 .
If ( − 1) k − 1 D k,i ( θ ) > 0 then by Equation D10 we have that ˙ θ k,i k ( t ) =
( − 1) k − 1 2 n k θ k,i (1 − θ k,i )
D k,i ( θ ) (cid:54) = 0 , where the last inequality follows by Assumption 3.
By the continuity θ ( t ) , it follows that ∃ (cid:15) 0 > 0 such that ∀ (cid:15) ∈ (0 , (cid:15) 0 ) , θ k,i k
( t + (cid:15) )
(cid:54)
=
θ k,i k ( t )
[Equation]
that is, indeed, a contradiction.
It must then be the case that ( − 1) k − 1 D k,i ( θ ) ≤ 0 and, thus, we have that
[Equation]
Thus, we showed that in every case ˙
V i 1 ,i 2 ( θ ( t )) ≥ 0 , ∀ t ∈ I .
However, since θ ( t ) is continuous (since it is differentiable), we can extend this property for any t ≥ 0 , i.e., ˙ V i 1 ,i 2 ( θ ( t )) ≥ 0 , ∀ t ≥ 0 .
Finally, notice that V i 1 ,i 2 ( θ ( t )) is bounded, and, hence, it follows that − V
i 1 ,i 2 ( θ ( t )) is a Lyapunov function (up to a constant) of the dynamical system described by Equation D10, and, Theoremfollows by the deﬁnition of a Lyapunov function.
B S UPPLEMENTARY EXPERIMENTAL RESULTS B. E XPERIMENTAL RESULTS FOR 2-T EAM H IDDEN M ATCHING P ENNIES GAMES
In this section we present exemplary settings for the dynamics presented in section 5.
For an overview of the conditions that characterize the behaviors presented in these examples, we reference the interested reader to section 5; for formal deﬁnitions and the proofs of these concepts, see §A.4.
Let G =
( n := n 1 + n 2 , S :
= { 0 , 1 } n , u : S → R n ) be a 2-Team Hidden Matching Pennies game with n 1 = n 2 = 3 and payoff function given by (17).
Figure 3 depicts the behavior of the RD in an unrestricted instance of G , i.e., S k,i =
[0 , 1] for all i ∈ [3] , k ∈ [2] .
The restrictions applied to team 1 and team 2 are visualized by the feasible range of each member’s strategies (Figure 3 (Left) , and (Right) , respectively).
The black dot in each range indicates the initial strategy of each member, i.e., θ k,i (0) .
Note that, by Assumption 3, this strategy proﬁle, ( θ (0) , 1 − θ (0)) , is assumed to lie inside the feasible region enforced by the constraints S k,i .
We solve the initial value problem of the ODE that corresponds to this RD using the RADAU integration method implemented by the scipy package in Python 3, and we perform 500 evaluations over the time interval t ∈ [0 , 100] .
The orbit of the RD with initial parametrization θ (0) is depicted as a curve on the F × G space deﬁned by the hidden mappings in (19) (Figure 3 (Center) ).
The red dot indicates the the point ( f ( θ 1 ( t )) , g ( θ 2 ( t )) at the end of the simulation, i.e., at time t = 100 .
The corresponding strategies of each member are indicated by blue (team 1) and orange (team 2) dots inside the corresponding feasible regions (Figure 3 (Left) , and (Right) ).
The individual trajectory of each θ k,i , i ∈ [3] , k ∈ [2] is depicted in Figure 3 (Bottom) , where the curve labeled T k P i corresponds to the trajectory of the i -th member of team k
if k = 1 , or the i − n 1 -th member of team k
if k = 2 .
In the case that the RD cycle, e.g., in the unrestricted setting, we depict one of the Lyapunov functions of the RD (Figure 3 (Bottom) ).
Speciﬁcally, the Lyapunov function of our choice is
[Equation]
the average value of all the linearly independent Lyapunov functions deﬁned in (20).
We selected this speciﬁc Lyapunov function as the average is, in general, more numerically stable and for compactness.
In Figure 3, we verify that the RD of G , indeed, cycle in an unrestricted setting (Theorem 3).
The choice of initial points does not matter in this case; for completeness, we note that θ 1 = (0 . 1 , 0 .
9 , 0 . 2) , and θ 2 =
8) in all of the examples in this section.
As the ﬁrst example in a restricted setting (Figure 4), we are going to enforce S 1 , 1 = S 1 , 3 = S 2 , 1 = [0 . 05 , 0 . 4] , and S 1 , 2 = S 2 , 2 = S 2 , 3 = [0 . 6 , 0 .
First, observe that S k,i ⊂ (0 , 1) for all i ∈ [3] , k ∈ [2] ; hence, Assumption 3 is satisﬁed.
Next, notice that 12 / ∈ S k,i for all i ∈ [3] , k ∈ [2] .
It follows, by Theorem 4, that the RD should converge in this case (Figure 4 (Center) , and (Bottom) ), and the point of convergence is, indeed, θ ∗ 1 = (0 . 4 , 0 . 6 , 0 . 4) , and θ ∗ 2 = (0 . 05 , 0 .
95 , 0 . 95) , as given in Theorem 4.
There exists a similar setting (Figure 5), where the RD of G converge, as well.
Note, in this case, while 12 / ∈ S 2 ,i for all i ∈ [3] , we let 12 ∈ S 1 ,i for some i ∈ [3] (speciﬁcally, 12 ∈ S 1 ,i for all i ∈ [3] in this particular example).
Although we did not provide a convergence analysis for this case, it is not difﬁcult to see that the convergence guarantees from Theorem 4 can be generalized to include this case.
As is visualized in Figure 5 (Center) , the exhibited behavior is a hybrid of the two behaviors described by Theorem 4, and Theorem 5.
In particular, the behavior of the dynamics depends on the quadrant of the F × G space that the θ ( t ) lies.
Ultimately though, the dynamics can be shown to converge to a point θ ∗ , which depends not only on the initialization θ (0) but also on the restrictions in the space parameters, S k,i , i ∈ [3] , k ∈ [2] .
The particular point of convergence for this example is θ ∗ 1 = (0 . 05 , 0 .
95) , and θ ∗ 2 = (0 . 4 , 0 . 6 , 0 . 6) , where the space restrictions are given as S 2 , 1 =
4] , S 2 , 2 = S 2 , 3 = [0 . 6 , 0 .
95] , and S 1 ,i =
95] for all i ∈ [3] .
Finally, we present an example of cycling behavior in the restricted setting (Figure 6) where we applied the restrictions S 1 , 1 = S 1 , 3 = [0 . 05 , 0 . 8] , S 1 , 2 = S 2 , 2 = [0 . 6 , 0 .
95] , S 2 , 1 = [0 .
05 , 0 . 4] , and S 2 , 3 =
Note that 1 2 ∈ S 1 , 1 , S 1 , 3 , S 2 , 3 ; hence, | S 1 | + | S 2 | = 3 , and there exists i k ∈ [ n k ] for all k ∈ [2] such that 12 ∈ S k,i k .
By Theorem 5, it follows that the restricted RD cycle in this case; speciﬁcally, they converge to an invariant set deﬁned by V 1 , 3 ( θ ) = 0 , and V 3 , 3 ( θ ) = 0 as given by (20).
This behavior is depicted in Figure 6 (Center) .
Notice how the Lyapunov function, V ( θ ) (Figure 6 (Bottom) ) is monotonically increasing, and stabilizes at approximately t = 25 .