F INDING M IXED S TRATEGY N ASH E QUILIBRIUM FOR C ONTINUOUS G AMES THROUGH D EEP L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Nash equilibrium has long been a desired solution concept in multi-player games, especially for those on continuous strategy spaces, which have attracted a rapidly growing amount of interests due to advances in research applications such as the generative adversarial networks.
Despite the fact that several deep learning based approaches are designed to obtain pure strategy Nash equilibrium, it is rather luxurious to assume the existence of such an equilibrium.
In this paper, we present a new method to approximate mixed strategy Nash equilibria in multi-player continuous games, which always exist and include the pure ones as a special case.
We remedy the pure strategy weakness by adopting the pushforward measure technique to represent a mixed strategy in continuous spaces.
That allows us to generalize the Gradient-based Nikaido-Isoda (GNI) function to measure the distance between the players’ joint strategy proﬁle and a Nash equilibrium.
Applying the gradient descent algorithm, our approach is shown to converge to a stationary
Nash equilibrium under the convexity assumption on payoff functions, the same popular setting as in previous studies.
In numerical experiments, our method consistently and signiﬁcantly outperforms recent works on approximating Nash equilibrium for quadratic games, general blotto games, and GAMUT games.
I NTRODUCTION
Nash equilibrium (Nash, 1950) is one of the most important solution concepts in game scenario with multiple rational participants.
It plays an important role in theoretical analysis of games to guide rational decision-making processes in multi-agent systems.
With the recent success of machine learning applications in games, it attracts even more research interests on applying machine learning technique for unsolved game theory problems, for example, computation of Nash equilibrium for multi-player games.
In this paper, we focus on games with continuous action spaces, which include the famous application for Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), as well as many important game types such as the colonel blotto game (Gross & Wagner, 1950),
Cournot competition (R, 1996).
We develop a solution which signiﬁcantly improves the status-quo.
There have been several successful approaches to compute Nash equilibrium for multi-player (mostly 2-player) continuous game (Raghunathan et al., 2019; Balduzzi et al., 2018; Letcher et al., 2018).
These works seek Nash equilibria corresponding to pure strategies, in which each player takes a speciﬁc action to achieve its best payoff given other players’ actions.
A major concern for such a solution concept is its possible non-existence.
As a result, the convergences to a Nash equilibrium for these approaches were proven under the assumption for the existence of a pure strategy
Nash equilibrium, which can hardly be checked in practice, and their applicability is limited to speciﬁc types of games.
On the contrary, it is known that mixed strategy Nash equilibria always exist under mild conditions.
And note that any pure strategy Nash equilibrium is also a mixed strategy
Nash equilibrium, which means the latter one is a much more desired solution concept.
However, a key challenge that obstructs the study of computing a mixed strategy Nash equilibrium, especially for a continuous game, lies on how to design an efﬁcient method to represent the mixed strategy.
To be precise, a pure strategy can be represented by a single variable choosing from some region.
But as a distribution on each player’s action space, a mixed strategy with respect to the player is deﬁned in a (subspace of) real space R .
More generally, exact representation for a mixed strategy of a player usually requires many variables in a continuous space.
In addition, the corresponding probability distribution may not have a density function in closed-form.
To address this challenge, we introduce a pushforward measure technique.
It is a common tool in measure theory to transfer a measure to some speciﬁc measure space (Bogachev, 2007).
Speciﬁc to a continuous game, the probability distribution corresponding to a mixed strategy is obtained via a mapping parameterized by neural nets from a multi-dimensional uniform distribution.
With this pushforward representation, we generalize the Gradient-based Nikaido-Isoda (GNI) function introduced by Raghunathan et al. (2019), to handle mixed strategy Nash equilibria.
The original
GNI function can be viewed as a measure for the distance between any joint strategy proﬁle and a
Nash equilibrium after applying the payoff functions of players.
With proper generalization and modiﬁcation, we develop its mixed strategy version as a proper measure for a Nash equilibrium.
We prove that the distance becomes zero if and only if a stationary mixed Nash equilibrium is obtained.
Then we apply the gradient descent algorithm to the general GNI function, which converges to a stationary mixed Nash equilibrium under the convexity assumptions on the payoff functions.
Finally, we compare our method with baseline algorithms in numerical experiments.
Our approach shows effective convergence property in all the randomly generated quadratic games, general blotto games, GAMUT games and Delta-Dirac GAN (in appendix), which outperforms other baselines.
B ACKGROUND AND P ROBLEM D ESCRIPTION
The discrete action space Nash equilibrium computation has been most widely studied in the literatures.
Most well-known being the Lemke–Howson algorithm (Lemke & Howson, 1964) for solving the bimatrix game.
The state-of-art theoretical work of Tsaknakis & Spirakis (2007) provided a solution of 1 / 3 approximation in polynomial time .
And empirical work (Fearnley et al., 2015) shows it also performs well against practical game solving methods for the bimatrix game.
However, the continuous action space game computation is widely used in practice, while few methods are known for the general Nash equilibrium computation and all restricted to pure strategies.
The
GNI function is considered by Raghunathan et al. as one possible technique for multi-player continuous game, but they only consider pure strategies.
With further assumption that the utility functions of players are twice continuous differentiable, gradient-based algorithms are developed (Balduzzi et al., 2018;
Letcher et al., 2018).
But they do not provide a way to parameterize mixed strategy of continuous action space, and even we are able to do so through neural networks,
the
Hessian of utility functions, necessary in these approaches, is extremely hard to compute or store.
Game-theoretical approach has had useful applications to machine learning such as the optimization of GAN network training (Daskalakis et al., 2017; Gidel et al., 2018) and adjustment on the gradient descent method (Balduzzi et al., 2018).
However they are limited to 2-player zero-sum games.
We are the ﬁrst work to study the mixed strategy continuous game Nash equilibrium computation.
Our work is motivated by the utilization of the Nikaido-Isoda (NI) function for loss function minimization (Uryas’ ev & Rubinstein, 1994; Raghunathan et al., 2019).
We start to establish a theoretical formulation of the extend mixed strategy
continuous action space
Nash equilibrium as a result of the minimization on a functional variation-based Nikaido-Isoda function.
2. C ONTINUOUS G AME N ASH E QUILIBRIUM
[Equation]
Here N denotes the number of players, and x i ∈ R n i the strategy of the i -th player where n i is the dimension of its action space.
Let n = (cid:80) Ni =1 n i , and x = ( x 1 , x 2 , · · · , x N ) ∈ R n denotes the joint pure strategy among all players while x − i =
( x 1 , · · · , x i − 1 , x i +1 , · · · , x N ) ∈ R n − n i the joint pure strategy among players except i .
f i : R n → R denotes the utility function (cost) of i -th player.
A solution x ∗ to (1) is called a pure strategy Nash equilibrium.
2. N IKAIDO -I SODA (NI) F UNCTION
In the paper by Nikaidˆo et al.
(1955), Nikaido-Isoda (NI) function is introduced as:
[Equation]
From the Equation (2), we know φ ( x ) (cid:62) 0 for ∀ x ∈ R n , and φ ( x ) = 0 is the global minimum of NI function which can only be achieved at a Nash equilibrium (NE).
Therefore, a common algorithm of computing NE points is minimizing the NI function above.
However, it is a huge difﬁculty to handle the global inﬁmum.
On one hand, the global inﬁmum can not be obtained in ﬁnite time.
On the other hand, the inﬁmum can be unbounded below in some games, for example the twoplayer bi-linear games, where f 1 ( x ) =
x T 1 Mx 2 = − f 2 ( x ) .
All of the facts above show us the shortcomings of NI function, and in order to rectify them, Raghunathan et al.
(2019) introduce the following Gradient-based Nikaido-Isoda (GNI) function.
2. G RADIENT - BASED N IKAIDO -I SODA (GNI) F UNCTION
If we calculate local inﬁmum in the NI function φ ( x ) instead of global inﬁmum, the time complexity and unbounded inﬁmum are no longer shortcomings.
To be precise, given the local radius λ , local inﬁmum can be approximated by steepest descent direction, and we get the following GNI function:
[Equation]
By minimizing V ( x , λ ) , a stationary Nash point x ∗ , where ∇ x i f i ( x ∗ ) = 0 for ∀ i , can be approximated efﬁciently.
Furthermore, if all the utility functions f i are convex, then the stationary Nash points (SNP) obtained are actually Nash Equilibrium (NE).
(M IXED -GNI) G RADIENT - BASED N IKAIDO -I SODA F UNCTION OF M IXED S TRATEGY ON C ONTINUOUS G AMES
In this section, we are going to introduce our novel Gradient-based Nikaido-Isoda function of mixed strategy on continuous games (Mixed-GNI), which is used to get an approximated solution of the following optimization problem.
[Equation]
Before we solve this optimization problem, there is another fundamental question, which is how we should represent (or parametrize) a distribution π i .
The simplest way to do so is to parametrize its density function.
However, not every distribution has its density function, such as Dirac distribution, and it will be inconvenient for us to do sampling from only a density function.
Therefore, we introduce another way, adopting the pushforward measure to represent a distribution.
Given a distribution µ 0 and a mapping g ( · ) , data x drawn from µ 0 can be transported into a new distribution µ 1 (constituted by g ( x ) ).
Technically speaking, µ 1 is called the pushforward measure of µ 0 by mapping g , denoted by µ 1 = g # ( µ 0 ) .
Here, for ∀ j ∈ [ N ] , we prepare each distribution π j a corresponding pushforward function g j :
R d → R n j , and we have:
[Equation]
where U stands for the uniform distribution on [0 , 1] d .
Each time we want to sample from distribution π i , we only need to sample several ω i ∈ [0 , 1] d from distribution U and calculate g i ( ω i ) .
Then, these g i ( ω i ) form a sample set from distribution π i .
And optimization problem (3) becomes:
[Equation]
To solve the optimization problem above
[Equation]
we consider the following Gradient-based Nikaido-Isoda function of Mixed strategy on Continuous games (Mixed-GNI), generalized from the GNI function introduced above
[Equation]
and we call this function V the local regret :
[Equation]
Here, δ g i F i stands for the 1-st order variation of functional F i on element function g i and
[Equation]
By minimizing the functional V ( g 1 , g 2 , · · · , g N ; λ ) , we can approximately get stationary Nash points (SNP), and even get Nash equilibrium if all the utility functions
f i are convex.
We will prove them in the next section.
In practice, we further parametrize these pushforward functions as: g i ( · ) =
g i ( · , θ i ) , to efﬁciently calculate derivatives instead of variations.
For simplicity, we denote g i as g θ i .
In order to obtain a better expressibility, we use neural networks as the architecture to parametrize these pushforward functions.
Then, Mixed-GNI function V can be transformed to:
[Equation]
Finally, the Mixed-GNI function can be estimated by sampling the points from distribution U as an estimator of these F i and minimized by implying gradient descent on these function parameters θ i , i ∈
[ N ] , the convergence of which is proved in the next section.
T HEORETICAL A NALYSIS OF M IXED -GNI 4. T HE S UFFICIENT AND N ECESSARY C ONDITION OF S TATIONARY N ASH P OINT
As a mixed strategy of an N -player continuous game, ( π 1 , π 2 , · · · , π N ) = ( g #1 U, g #2 U, · · · , g # N U ) is a stationary Nash point (SNP)
if and only if for ∀ i ∈ [ N ] , the 1-st order variation
[Equation]
holds at each direction σ ( x ) .
Here:
[Equation]
is the expectation of the i -th player’s utility with the form of N -variable functional.
Now, we compute the variation above and deduce the sufﬁcient and necessary condition of SNP.
[Equation]
where:
[Equation]
For SNP, Equation (6) holds at each direction σ ( x ) , i.e. G ( ω i ) ≡ 0 .
Therefore, we have
Theorem 1.
π =
( π 1 , π 2 , · · · , π N ) =
( g #1 U, g #2 U, · · · , g # N U ) is a stationary Nash point (SNP) for an N -player continuous game if and only if:
[Equation]
holds for all i ∈ [ N ] .
From Equation (7), we also know that:
[Equation]
In other words, the steepest direction is:
[Equation]
Then we show the relationship between stationary Nash point and Nash equilibrium.
Theorem 2.
Denote S SNP , S NE as the set of SNPs and NEs of a particular N -player continuous game.
Obviously, S NE ⊆ S SNP .
If all utility functions f i are convex, we have: S NE = S SNP
Proof.
Suppose π =
π 1 , π 2 , · · · , π N ) =
( g #1 U, g #2 U, · · · , g # N U ) is an SNP, we will prove it an NE when all functions
f
i are convex.
According to the convexity and the condition of SNPs, we know that for ∀ i ∈
[ N ] and any other pushforward function ˜
g
i :
[Equation]
which leads to our conclusion
[Equation]
that π = ( g #1 U, g #2 U, · · · , g # N U ) is a global Nash equilibrium.
Next, we show the relationship between the zeros of Mixed-GNI function V ( g 1 , g 2 , · · · , g N ) and
SNPs of the N -player continuous game
[Equation]
Lemma 1.
Assume f : R d
→ R is a twice differentiable function, and its 1-st order gradient ∇ f is
L f -Lipschitz continuous.
Then for ∀ x, y ∈ R d , we have:
[Equation]
Proof.
According to the condition of f , there holds the following equations.
[Equation]
With this lemma, we can show that each global minimum of V ( g 1 , g 2 , · · · , g N ) is also an SNP.
Theorem 3.
If each utility function f i is twice differentiable and its 1-st order gradient ∇ f
i is
L f -Lipschitz continuous.
Then:
[Equation]
holds when 0 < λ (cid:54) 1 L f .
Here, (cid:107) · (cid:107) 2 is a functional norm which means:
[Equation]
Proof.
[Equation] [Equation]
Then, according to Lemma 1:
[Equation]
F
[Equation]
And the other side of this inequality is similar.
The theorem above tells us that, V ( g 1 , g 2 , · · · , g N ; λ ) is always non-negative as long as λ (cid:54) 1 L f .
And its global minima, or in the other words, its zeros, are surely SNPs, because for ∀ i ∈
[ N ] :
[Equation]
Finally, we analyze the stability of SNPs.
In the following theorem, we show that the 2-nd order variation of functional V is a positive semideﬁnite operator, which conﬁrms the stability of SNPs.
Theorem 4.
The 2-nd order variation δ 2 V ( g ∗ ; λ ) is a positive semideﬁnite operator for ∀ g ∗ ∈
S SNP and 0 (cid:54) λ (cid:54) 1 L f .
Proof.
The 1-st and 2-nd order variation of V i ( g ; λ ) satisfy:
[Equation]
where g = ( g 1 , g 2 , · · · , g N ) , ˜ g =
( g 1 , · · · , g i − 1 , g i − λδ
g
i F i , · · · , g N ) and
[Equation]
) is a n × n matrix.
Given g ∗ ∈ S SNP , then δF i ( g ∗ ) = 0 .
[Equation]
which is positive semideﬁnite.
Therefore:
[Equation]
is also positive semideﬁnite.
4. C ONVERGENCE A NALYSIS
In this section, we analyze the convergence analysis of gradient descent:
[Equation]
According to the deﬁnition of functional V ( g ; λ ) , it can be rewritten as the following form:
[Equation]
] where G V = (cid:80) Ni =
1 f i ( y 1 , y 2 , · · · , y N ) − f i ( y 1 , · · · , y i − 1 , y i − λ ∇ i f i
( y 1 , y 2 , · · · , y N ) , · · · , y N ) .
Theorem 5.
Suppose ∇ G V ( x ) is L G -Lipschitz continuous.
Through gradient descent, the function sequence g ( k ) converges sublinearly to a stationary Nash point (SNP) g ∗ if ρ < 1 L G , λ (cid:54) 1 L f .
Proof.
According to Lemma 1, we have:
[Equation]
Let k = 0 , 1 , · · · , K
[Equation]
and add them up
[Equation]
we have:
[Equation]
Since λ (cid:54) 1 L f , we know that V ( g ( K +1) ; λ ) (cid:62) 0 by Theorem 3, we have
[Equation]
which completes our proof.
E XPERIMENTS
To evaluate the practical performance of our approach, we apply it to three types of games, twoplayer quadratic games, general blotto games, and GAMUT games, the most popular games for evaluation of Nash equilibrium algorithms.
In all the experiments, we set the local radius λ = 1 eand
we use gradient descent as our optimization method with step size ρ = 1 e -2 and momentum κ = 0 .
The network architecture we use for the pushforward functions g θ is a 6-layer fully connected neural network with the size of each layer as: 20, 40, 160, 160, 40, 20.
The size of its output layer is the dimension of each player’s action space.
From forward to backward, the activation function we use is: tanh , tanh , tanh , ReLU, tanh , tanh .
We mainly compare our approach with three recent studies, gradient descent for GNI function (Raghunathan et al., 2019) (gradGNI in short), Symplectic Gradient Adjustment algorithm
(Balduzzi et al., 2018) (SGA in short), and Stable Opponent Shaping (Letcher et al., 2018)
(SOS in short) as they outperformed other existing algorithms applicable to continuous game settings.
For all these methods, we either follow the standard hyper-parameters mentioned in the original papers, or the ones resulting in the best convergence.
5. T WO - PLAYER Q UADRATIC G AME
The two-player quadratic game is deﬁned by the the players’ payoff functions
f i ( i = 1 , 2 ):
[Equation]
16) where Q i ∈ R ( n 1 + n 2 ) × ( n 1 + n 2 ) , r i ∈ R n 1 + n 2 , x = ( x 1 , x 2 ) and x i ∈ R n i .
In our experiments, we choose n 1 = n 2 ∈ { 3 , 5 , 10 } .
For each pair of n i , we randomly generate 100 instances for the matrix Q i and r i for i = 1 , 2 .
Each item in each matrix Q i and each vector r i follows the uniform distribution on [0 , 1] independently.
We show the converging process of all algorithms for one game instance ( n 1 = n 2 = 3 ) in Fig. 1(a) as an example.
As we can see, our approach effectively converges to a stationary Nash equilibrium point.
While gradGNI also converge in this instance, its result has larger local regret.
In other words, it obtain worse approximation to Nash equilibrium, which coincides with the essential difference between pure strategy and mixed strategy.
The Mixed-GNI approach searches for the equilibrium in the mixed strategy space, which includes the pure strategy space that gradGNI searches in.
On the other hand, SGA and SOS diverge in this game instance.
We further take the average of the ﬁnal local regrets after 2000 iterations for all the 100 instances, summarized in Tab. 1.
All the algorithms show consistency as the dimension of action space increases, and Mixed-GNI outperforms others regardless of the randomness of game structures.
5. G ENERAL B LOTTO G AME
We next consider the general blotto game, which differs from previous games in the action space of each player for which further constraints apply.
In a blotto game, player 1 and 2 (sometimes known as two colonels) have a budget of resource X 1 ,
X 2 respectively.
W.l.o.g we set X 1 ≤ X 2 .
There are m battleﬁelds in total.
In each battleﬁeld j , when two players allocate x 1 j , x 2 j resource on it, the payoff of player i is:
[Equation]
f
[Equation]
χ
[Equation]
=
[Equation]
χ ) ,
[Equation] [Equation]
where − i denotes the player other than player i .
Each player’s payoff across all m battleﬁelds is the sum of the payoffs across the individual battleﬁelds.
For each player i , a feasible pure strategy x i =
( x i 1 , . . .
, x im ) ∈ R m + must also satisﬁes (cid:80) mj
=1 x ij ≤ X i .
Here we adopt the generalized
[Figure] [Figure]
[Table]
GAMUT
[Table]
GAMUT
[Table]
blotto game proposed by Golman & Page (2009) with continuous payoff functions.
The payoff functions in vanilla blotto game is discontinuous, for which our method as well as baselines fails.
In our experiments, we set m ∈ { 3 , 5 , 10 } .
For each m , we randomly generate 100 instance for the budget
X i , following the uniform distribution on [0 , 1] independently.
We show the converging process of all algorithms for one game instance ( m = 3 ) in Fig. 1(b) as an example.
All the algorithms converges for this game, while gradGNI, SGA and SOS converge faster and more smoothly comparing with our Mixed-GNI.
However, similar to the quadratic game, their ﬁnal results have larger local regrets.
This coincides with the fact that the mixed strategy is a better solution concept than the pure strategy, especially in blotto games.
We further take the average of the ﬁnal local regrets after 2000 iterations for all the 100 instances, summarized in Tab.
All the algorithms show consistency as the dimension of action space increases, and Mixed-GNI outperforms others regardless of the randomness of game structures.
5. GAMUT G AMES
Finally, we apply our method on the game instance generated by the comprehensive GAMUT suite of game generators designated for testing game-theoretic algorithms (Nudelman et al., 2004).
GAMUT includes a group of random distributions, based on each of which the payoff of each player for each pure strategy proﬁle can be drawn independently.
To be precise, we extend the quadratic game to a multi-player version, where r i = 0 , and 100 game instances with 4 players are generated.
For each instance, one of the distributions from the GAMUT set is selected, and each item in each matrix Q
i is sampled according to it independently.
We show the converging process of all algorithms for one game instance in Fig.
1(c).
Both MixedGNI and SGA converge, but SGA has a much worse ﬁnal result than our Mixed-GNI.
And this time, gradGNI diverges while SOS converges much slower.
Furthermore, we take the average of the ﬁnal local regrets after 2000 iterations for all the 100 instances, shown in Table 1.
From these different games, we know that our Mixed-GNI converges and performs better than two baselines in all of the three games, which shows the effectiveness and efﬁciency of our Mixed-GNI model.
As the ﬁrst algorithm to compute the mixed strategy Nash equilibrium of games with continuous action space, we believe that the technique we introduced here will enable new optimization researches of many exciting interaction domains of algorithmic game theory and deep learning.
A E XPERIMENTS ON D ELTA -D IRAC GAN S
Gidel et al.
(2018) introduce a one-dimensional GAN where the real data follows a Dirac-Delta distribution.
This experiment is also conducted in our baseline paper by Raghunathan et al.
As a two-player game, the payoff functions for the two players are listed as follows:
[Equation]
where the θ is the location of the Dirac spike.
In Figure 2, we show the converging process of all algorithms for one game instance (i.e. for a single θ ).
As we can see, our Mixed-GNI and original gradGNI algorithms converge while the other two diverge.
Compared with gradGNI, our approach is converging much faster.
We further take the average of the ﬁnal local regrets afteriterations for all the 100 instances (i.e. 100 different θ s) and show the results in Table 2.
In this experiment, our Mixed-GNI algorithm again outperforms the others, regardless of the randomness of game structures.
[Table]
[Table]