A U NIFIED A PPROACH TO R EINFORCEMENT L EARNING , Q UANTAL R ESPONSE E QUILIBRIA , AND T WO -P LAYER Z ERO -S UM G AMES
Anonymous authors
Paper under double-blind review
A BSTRACT
Algorithms designed for single-agent reinforcement learning (RL) generally fail to converge to equilibria in two- player zero sum (2p0s) games
On the other hand, game
-theoretic algorithms for approximating Nash and regularized equilibria in 2p0s games are not typically competitive for RL and can be difficult to scale.
As a result, algorithms for these two cases are generally developed and evaluated separately.
In this work, we show that a single algorithm can produce strong results in both settings , despite their fundamental differences. This algorithm, which we call magnet mirror descent (MMD), is a simple extention to mirror descent and a special case of a non-Euclidean proximal gradient algorithm.
From a theoretical standpoint, we prove a novel linear convergence for this non-Euclidean proximal gradient algorithm for a class of variational inequality problems.
It follows from this result that MMD converges linearly to quantal response equilibria (i e., entropy regularized Nash equilibria) in extensive-form games—this is the first time linear convergence has been proven for a first order solver.
Moreover, applied as a tabular Nash equilibrium solver via self-play, we show empirically that MMD produces results competitive with CFR in both normal-form and extensive-form games—this is the first time that a standard RL algorithm has done so.
Furthermore, for singleagent deep RL, on a small collection of Atari and Mujoco tasks, we show that MMD can produce results competitive with those of PPO.
Lastly , for multi-agent deep RL, we show MMD can outperform NFSP in 3x
Abrupt Dark Hex
I NTRODUCTION
This work concerns approximating (regularized) optimal policies in single-agent settings and approximating (regularized)
Nash equilibria in two-player zero-sum (2p0s) games.
While these problem settings are similar in certain intuitive senses—both involve control policies converging to welldefined notions of optimality—they also possess fundamental differences.
For example, because of cyclical dynamics in 2p0s games (e.g., rock-paper-scissors), standard reinforcement learning (RL) algorithms generally fail to converge to equilibria when applied in self-play.
On the other hand, convergent algorithms for 2p0s games are not generally competitive as single-agent algorithms and often involve nuances that make them difficult to scale.
Our contribution is a single algorithm that yields strong performance in both problem settings, despite their substantial differences.
This algorithm, which we call magnetic mirror descent (MMD), is an extension of mirror descent (Beck & Teboulle, 2003; Nemirovsky & Yudin, 1983) with proximal regularization and is a special case of a non-Euclidean proximal gradient method.
We prove a new linear convergence result for this non-Euclidean proximal gradient method for a class of variational inequality problems (Facchinei & Pang, 2003) with composite structure. As a consequence of our general analysis, we attain formal guarantees for MMD by showing that solving for quantal response equilibria (McKelvey & Palfrey, 1995) (i.e., entropy regularized Nash equilibria) in extensive-form games (EFGs) can be modeled as variational inequality problems via the sequence form (Romanovskii, 1962; Von Stengel, 1996; Koller et al., 1996).
These guarantees provide the first linear convergence results to quantal response equilibria (QREs) in EFGs for a first order method—previously known results only exist for normal-form games (NFGs) (Cen et al., 2021).
While such algorithms can be applied to EFGs, they require first converting EFGs to NFGs, which causes an exponential blow up in the size of the game.
On the other hand, while there exist scalable algorithms that operate directly over the extensive-form (Farina et al., 2019; Ling et al., 2019), they possess neither last-iterate nor linear convergence guarantees.
Our empirical contribution investigates MMD as a single-agent RL algorithm and as a 2p0s regularized equilibrium and last-iterate equilibrium approximation algorithm across a variety of benchmarks.
We begin by confirming our theory—showing that MMD converges exponentially fast to QREs in both NFGs and EFGs.
We also find that, empirically, MMD converges to agent QREs (AQREs) (McKelvey & Palfrey, 1998)—an alternative formulation of QREs for extensive-form games—when applied as a self-play RL algorithm.
These results lead us to examine MMD as an
RL algorithm for approximating Nash equilibria.
On this front, we show competitive performance with counterfactual regret minimization (CFR) (Zinkevich et al., 2007).
This is the first instance of a standard RL algorithm 1 yielding empirically competitive performance with CFR in tabular benchmarks when applied in self play.
Next, for single-agent deep RL, we show that MMD yields results that are competitive with those of PPO (Schulman et al., 2017) for a small collection of
Atari (Bellemare et al., 2013) and Mujoco (Todorov et al., 2012) tasks.
Lastly, we present multi-agent deep RL results for 3x
Abrupt Dark Hex, where we show that MMD can outperform neural fictitious self play (NFSP) (Heinrich & Silver, 2016).
In addition to the above, we provide extensive additional experiments in the appendix.
In aggregate, these results suggest that MMD is effective both as a single-agent RL algorithm and as multi-agent RL algorithm for 2p0s games.
B ACKGROUND
Sections 2.1 and 3.3 provide a casual treatment of our problem settings and solution concepts and a summary of our algorithm and some of our theoretical results.
Sections 2.2 through 3.2 give a more formal and detailed treatment of the same material—these sections are self-contained and safe-to-skip for readers less interested in our theoretical results.
2. P ROBLEM S ETTINGS AND S OLUTION C ONCEPTS
This work is concerned with both single-agent settings, such as Markov decision processes (Puterman, 2014) and partially observable Markov decision processes (Kaelbling et al., 1998), and 2p0s games— settings with two players in which the reward for one player is the negation of the reward for the other player.
This latter setting is often formalized as an NFG, a partially observable stochastic game (Hansen et al., 2004) or a perfect-recall EFG (von Neumann & Morgenstern, 1947).
An important idea is that it is possible to convert any EFG into an equivalent NFG.
The actions of the equivalent NFG correspond to the deterministic policies of the EFG.
The payoffs for a joint action are dictated by the expected returns of the corresponding joint policy in the EFG.
We use different names for the solution concepts studied in this work, depending on whether the setting is single agent or multi agent.
In single-agent settings, we call them optimal policies and soft-optimal policies.
We say a policy is optimal if there does not exist another policy achieving a greater expected return (Sutton & Barto, 2018).
We say a policy is α -soft optimal in the normal sense if it maximizes a weighted combination of its expected action value and its entropy:
[Equation]
where π is a policy, ∆ p
A q is the action simplex, q is the action-value function, α is the regularization temperature, and H is Shannon entropy.
More generally, we say a policy is α -soft optimal in the behavioral sense if it satisfies equation (1) at every decision point.
In 2p0s settings, we refer to the primary solution concepts used in this work as Nash equilibria and
QREs.
We say a joint policy is a Nash equilibrium if each player’s policy is optimal, conditioned on the other player not changing its policy.
We say a joint policy is a QRE 2 (McKelvey & Palfrey, 1995) if each player’s policy is soft optimal in the normal sense, conditioned on the other player not changing its policy.
We say a joint policy is an agent QRE (AQRE) (McKelvey & Palfrey, 1998) if each player’s policy is soft optimal in the behavioral sense, subject to the opponent’s policy being fixed.
Note that
AQREs of EFGs do not generally correspond with the QREs of their normal-form equivalents.
Outside of (A)QREs, our results also apply to other regularized solution concepts, such as those having KL regularization toward a non-uniform policy.
2. N OTATION
We use superscript to denote a particular coordinate of x “ p x 1 , ¨ ¨ ¨ , x n q P R n and subscript to denote time x t .
We use the standard inner product denoted as x x, y y
“ ř ni “ 1 x i y i .
For a given norm } ¨ } on R n we define its dual norm } y } ˚ “ sup } x }“ 1 x y, x y .
For example, the dual norm to } x } 1 “ ř ni “ 1 | x i | is } x } 8 “ max i | x i | .
We assume all functions f : R n Ñ p´8 , `8s to be closed, with domain of f as dom f “ t x : f p x q ă `8u , and corresponding interior int dom f .
If f is convex and differentiable, then its minimum x ˚ P arg min x P C f p x q over a closed convex set C satisfies x ∇ f p x ˚ q , x ´ x ˚ y ě 0 for any x P C .
We use the Bregman divergence of ψ to generalize the notion of distance.
Let ψ be a convex function differentiable over int dom ψ .
Then the Bregman divergence with respect to ψ is B ψ : dom ψ ˆ int dom ψ Ñ R , defined as B ψ p x ; y q “ ψ p x q ´ ψ p y q ´ x ∇ ψ p y q , x ´ y y .
We say that f is µ -strongly convex over C with respect to } ¨ } if B f p x ; y q ě µ 2 } x ´ y } 2 for any x P C, y P C X int dom ψ .
Similarly we define relative strong convexity (Lu et al., 2018).
We say g is µ -strongly convex relative to ψ over C if x ∇ g p x [Equation]
Note both ψ and B ψ p¨ ; y q are 1 -strongly convex relative to ψ and that relative strong convexity can be equivalently stated as B g p x ; y q ě µB f p x ; y q (Lu et al., 2018).
2. Z ERO -S UM G AMES AND QRE S
In 2p0s games, the solution of a QRE can be written as the solution to a negative entropy regularized saddle point problem.
To model QREs (and more), we consider the regularized min max problem
[Equation]
where X Ă R n , Y Ă R m are closed and convex (and possibly unbounded) and g 1 : R n Ñ R , g 2 : R m Ñ R , f : R n ˆ R m Ñ R .
Moreover, g 1 and f p¨ , y q are differentiable and convex for every y .
Similarly ´ g 2 , f p x, ¨q are differentiable and concave for every x .
A solution p x ˚ , y ˚ q to equationis a Nash equilibrium in the regularized game with the following best response conditions along with their equivalent first order optimality conditions
[Equation]
In the context of QREs we have that X “ ∆ n , Y “ ∆ m with f p x, y q “ x J Ay for some payoff matrix A , and g 1 , g 2 are negative entropy.
The corresponding best response conditions (3-4) can be written in closed form as x ˚ 9 exp p ´
Ay ˚ { α q , y ˚ 9 exp p
A J x ˚ { α q .
Similarly, for EFGs, normal-form
QREs take the form of equation 2 (Ling et al., 2018) with g 1 , g 2 being dilated entropy (Hoda et al., 2010), f p x, y q “ x J Ay ( A being the sequence-from payoff matrix), and X , Y the sequence-form strategy spaces of both players.
2. C ONNECTION BETWEEN ZERO - SUM GAMES AND VARIATIONAL INEQUALITIES
More generally, solutions to equation 2 (including QREs) can be written as solutions to variational inequalities (VIs) with specific structure.
The equivalent VI formulation stacks both first-order best response conditions (3-4) into one inequality.
Definition 2.1 (Variational Inequality Problem (VI)) .
Given Z Ď R n and mapping G : Z Ñ R n , the variational inequality problem VI p Z , G q is to find z ˚ P Z such that
[Equation]
In particular, the optimality conditions (3-4) are equivalent to VI p Z , G q where G “ F ` α ∇ g ,
Z “ X ˆ Y and g : Z Ñ R , p x, y q ÞÑ g 1 p x q ` g 2 p
y q , with corresponding operators F p
z q “ r ∇ x f p x, y q , ´ ∇ y f p x, y qs J , and ∇ g “ r ∇ x g 1 p x q , ∇ y g 2 p
y qs J .
For more details see Facchinei &
Pang (2003)(Section 1.4.2).
Note that VIs are more general than min-max problems; they also include fixed-point problems and Nash equilibria in n -player general-sum games (Facchinei & Pang, 2003).
However, in the case of convex-concave zero-sum games and convex optimization, the problem admits efficient algorithms since the corresponding operator G is monotone (Rockafellar, 1970).
Definition 2.2.
G is said to be strongly monotone if, for µ ą 0 , and any z, z 1 where G is defined, x G p
z q ´ G p
z
1 q , z ´ z 1 y ě µ } z ´ z 1 } 2 .
G is monotone if this is true for µ “ 0 .
Definition 2.3.
G is said to be L -smooth with respect to } ¨ } if, for any z, z 1 where G is defined, } G p z q ´ G p z 1 q} ˚ ď L } z ´ z 1 } .
For EFGs, Ling et al.
(2018) showed that the QRE is the solution of a min-max problem of the form equation 2 where f is bilinear and each g i could be non smooth.
Therefore, we can write the problem as a VI with strongly monotone operator G having composite structure, a smooth part coming from f and non-smooth part from the regularization g 1 , g 2 .
Proposition 2.4.
Solving a normal-form reduced QRE in a two-player zero-sum EFG is equivalent to solving VI p Z , F ` α ∇ ψ q where Z is the cross-product of the sequence form strategy spaces and ψ is the sum of the dilated entropy functions for each player.
The function ψ is strongly convex with respect to } ¨ } .
Furthermore, F is monotone and max ij |
A ij | -smooth ( A being the sequence-form payoff matrix) with respect to } ¨ } and F ` α ∇ ψ is strongly monotone.
A LGORITHMIC F RAMEWORK
In Proposition 2.4, we provided a new perspective to QRE problems that draws connections to VIs with special composite structure.
Motivated by this connection, in Section 3.1, we consider an approach to solve such problems via a non-Euclidean proximal gradient method Tseng (2010); Beck (2017) and prove a novel linear convergence result. Thereafter, in Section 3.2, we demonstrate how this general algorithm specializes to MMD and splits into two decentralized simultaneous updates in 2p0s games (one for each player). Finally, in Section 3.3, we discuss specific instances of MMD, giving new algorithms for RL and QRE solving, as well as, summarize our linear convergence result for QREs. 3. V ARIATIONAL I NEQUALITY T HEORY
We now present our main algorithm, a non-Euclidean proximal gradient method to solve VI p Z , F ` α ∇ g q . Since ∇ g is possibly not smooth, we incorporate g as a proximal regularization.
Algorithm 3.1.
Starting with z 1 P int dom ψ X Z at each iteration t do
[Equation]
To ensure that z t ` 1 is well defined, we make the following assumption.
Assumption 3.2 (Well-defined) .
Assume ψ is 1 -strongly convex with respect to } ¨ } over Z and, for any ℓ , stepsize η ą 0 , [Equation] , z [Equation]
p
[Equation]
z t q P int dom ψ .
We also make some assumptions on F and g .
Assumption 3.3.
Let F be monotone and L -smooth with respect to } ¨ } and g be 1 -strongly convex relative to ψ over Z with g differentiable over int dom ψ .
These assumptions imply F ` α ∇ g is strongly monotone 3 with unique solution z ˚ (Bauschke et al., 2011).
Our result shows that, if z ˚ P int dom ψ 4 , then Algorithm 3.1 converges linearly to z ˚ .
Theorem 3.4.
Let Assumptions 3.2 and 3.3 hold and assume the unique solution z ˚ to VI p Z , F ` α ∇ g q satisfies z ˚ P int dom ψ .
Then Algorithm 3.1 converges if η ď αL 2 and guarantees
[Equation]
Note α ą 0 is necessary to converge to the solution.
If α “ 0 in the context of solving equation 2, Algorithm 3.1 with ψ p z q “ 12 } z } 2 becomes projected gradient descent ascent, which is known to diverge or cycle for any positive stepsize.
However, choosing the strong convexity constants of g and ψ to beis for convenience—the theorem still holds with arbitrary constants, in which case the stepsize condition becomes proportional to the relative strong convexity constant of g (see Corollary D.6 for details).
Due to the generality of VIs, we have the following convex optimization result.
Corollary 3.5.
Consider the composite optimization problem min z P Z f p z q ` αg p z q .
Then under the same assumptions as Theorem 3.4 with F “ ∇ f , Algorithm 3.1 converges linearly to the solution.
In contrast to existing results (Tseng, 2010; Bauschke et al., 2017; Hanzely et al., 2021), Corollary 3.5 guarantees linear convergence due to the assumption that g is relatively-strongly convex.
3. A PPLICATION OF M AGNETIC M IRROR D ESCENT TO T WO -P LAYER Z ERO -S UM G AMES
We define MMD to be Algorithm 3 .
1 with g taken to be either ψ or B ψ p¨ ; z 1 q for some z 1 ; in both cases the 1 -relative strongly convex assumption is satisfied, and z t ` 1 is attracted to either min z P Z ψ p
z q or z 1 , which we call the magnet.
Algorithm 3.6 (Magnetic Mirror Descent (MMD)) .
[Equation]
Remark 3.7.
MMD has the same computational cost as mirror descent since the updates can be equivalently written as z t ` 1 “ arg min z P Z x ℓ,
z y ` ψ p z q (e.g. ℓ “ p ηF p z t q´ ∇ ψ p x t qq { p 1 ` ηα q for equation 6).
In fact, Proposition D.7 shows that MMD is equivalent to mirror descent on the regularized loss with a different stepsize.
MMD and, more generally, Algorithm 3.1 can be used to derive a descent-ascent method to solve the zero-sum game equation 2.
If g 1 “ ψ 1 and g 2 “ ψ 2 are strongly convex over X and Y , then we can let ψ p z q “ ψ 1 p x q ` ψ 2 p y q , which makes ψ strongly convex over Z .
Then the MMD update rule equation 6 converges to the solution of equation 2 and splits into simultaneous descent-ascent updates:
[Equation]
3. M AGNET M IRROR D ESCENT S UMMARY
MMD’s update is parameterized by four objects: a stepsize η , a regularization temperature α , a mirror map ψ , and a magnet, which we notate as either ρ or ζ depending on the ψ .
The stepsize η dictates the extent to which moving away from the current iterate is penalized; the regularization temperature α dictates the extent to which being far away from the magnet (i.e., ρ or ζ ) is penalized; the mirror map ψ determines how distance is measured.
If we take ψ to be negative entropy, then, in reinforcement learning language, MMD takes the form
[Equation]
10) where π t is the current policy, q t is the Q-value vector for time t , and ρ is a magnet policy.
For parameterized problems, if ψ “ 12 } ¨ } 22 , MMD takes the form
[Equation]
where θ t is the current parameter vector, L is the loss, and ζ is the magnet.
In settings with discrete actions, these instances of MMD possess close forms, as shown below
[Equation]
[Table]
E x t e n s i v eFo r m
Q R E s
M ea n
K L
D i v e r g e n ce
E x t e n s i v eFo r m
A Q R
E s
[Figure]
[Table]
10
[Table]
/sHxcOjpo4SxbDBIhGptk81Ci6xYbgR2I4V0tAX2PLHtzO/9YRK80g+mEmMvZAOJQ84o8ZKLc99TC+q036x5JbdOcgq8TJSggz1fvGrO4hYEqI0TFCtO54bm15KleFM4LTQTTTGlI3pEDuWShqi7qXzc6fkzCoDEkTKljRkrv6eSGmo9ST0bWdIzUgvezPxP6+TmOC6l3IZJwYlWywKEkFMRGa/kwFXyIyYWEKZ4vZWwkZUUWZsQgUbgrf88ippXpa9SvnqvlKq3WRx5OEETuEcPKhCDe6gDg1gMIZneIU3J3ZenHfnY9Gac7KZY/gD5/MHRx+O4w==</latexit> 10 <latexit sha1_base64="SXDK0HWqcwZU3hP7HyDuDV+fRRM=">AAAB7nicbVDLSgNBEOyNrxhfUY9eBoPgxbArEfUW9OIxgnlAsobZSScZMju7zMwKYclHePGgiFe/x5t/4yTZgyYWNBRV3XR3BbHg2rjut5NbWV1b38hvFra2d3b3ivsHDR0limGdRSJSrYBqFFxi3XAjsBUrpGEgsBmMbqd+8wmV5pF8MOMY/ZAOJO9zRo2Vmp77mJ5dT7rFklt2ZyDLxMtICTLUusWvTi9iSYjSMEG1bntubPyUKsOZwEmhk2iMKRvRAbYtlTRE7aezcyfkxCo90o+ULWnITP09kdJQ63EY2M6QmqFe9Kbif147Mf0rP+UyTgxKNl/UTwQxEZn+TnpcITNibAllittbCRtSRZmxCRVsCN7iy8ukcV72KuWL+0qpepPFkYcjOIZT8OASqnAHNagDgxE8wyu8ObHz4rw7H/PWnJPNHMIfOJ8/SimO5Q==</latexit> 10 <latexit sha1_base64="28OONLrmby1Xtw05r12r5o7BRX4=">AAAB73icbVBNSwMxEJ2tX7V
[Table] [Table]
↵
[Table] [Table]
/vG6qwys/YyJJNRVksShMua1je/a+PWCSEs0nhiCRzNxqkxFKJNqEVDIheMsvr5JW1fFqzsV9rVK/yeMowgmcwjl4cAl1uIMGNIGAgGd4hTdLWS/Wu/WxaC1Y+cwx/IH1+QNjfJAV</latexit>
[Figure]
<latexit sha1_base64="TPj7ZRHzUE6LBV7+NzSLnU8sLrk=">AAAB8XicbVBNS8NAEJ3Ur1q/qh69BIvgKSTSoheh6MVjBfuBbSiT7aZdutmE3Y1QQv+FFw+KePXfePPfuG1z0NYHA4/3ZpiZFyScKe2631ZhbX1jc6u4XdrZ3ds/KB8etVScSkKbJOax7ASoKGeCNjXTnHYSSTEKOG0H49uZ336iUrFYPOhJQv0Ih4KFjKA20mMPeTLCa9ep9csV13HnsFeJl5MK5Gj0y1+9QUzSiApNOCrV9dxE+xlKzQin01IvVTRBMsYh7RoqMKLKz+YXT+0zowzsMJamhLbn6u+JDCOlJlFgOiPUI7XszcT/vG6qwys/YyJJNRVksShMua1je/a+PWCSEs0nhiCRzNxqkxFKJNqEVDIheMsvr5LWheNVndp9tVK/yeMowgmcwjl4cAl1uIMGNIGAgGd4hTdLWS/Wu/WxaC1Y+cwx/IH1+QNoCJAY</latexit>
[Figure]
<latexit sha1_base64="0VXXdbYgYISBa9T50LdFhRw/L0o=">AAAB63icbVBNS8NAEJ3Ur1q/qh69LBbBU0mkoseiF48V7Ae0sWy2m3bp7ibsboQS+he8eFDEq
[Table]
/qh69LBbBU0mkoseiF48V7Ae0sWy2m3bp7ibsboQS+he8eFDEq3/Im//GTZqDtj4YeLw3w8y8IOZMG9f9dkpr6xubW+Xtys7u3v5B9fCoo6NEEdomEY9UL8CaciZp2zDDaS9WFIuA024wvc387hNVmkXywcxi6gs8lixkBJtM8txHd1ituXU3B1olXkFqUKA1rH4NRhFJBJWGcKx133Nj46dYGUY4nVcGiaYxJlM8pn1LJRZU+2l+6xydWWWEwkjZkgbl6u+JFAutZyKwnQKbiV72MvE/r5+Y8NpPmYwTQyVZLAoTjkyEssfRiClKDJ9Zgoli9lZEJlhhYmw8FRuCt/zyKulc1L1G/fK+UWveFHGU4QRO4Rw8uIIm3EEL2kBgAs/
[Table]
+OcF6cd+dj0Vpyiplj+
[Table]
</
[Table] [Table]
<
[Table]
sha1_
[Table]
="SJUNDXPj9V3b23Lx4Q4/42OIzrE=
[Table]
>
[Table]
/ggvHhTx6u/x5r9x2+agrQ8GHu/NMDMvSATXxnW/nZXVtfWNzcJWcXtnd2+/dHDY1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hoduq3nlBpHssHM07Qj+hA8pAzaqzU8tzH7Nyb9Eplt+LOQJaJl5My5Kj3Sl/dfszSCKVhgmrd8dzE+BlVhjOBk2I31ZhQNqID7FgqaYTaz2bnTsipVfokjJUtachM/T2R0UjrcRTYzoiaoV70puJ/Xic14bWfcZmkBiWbLwpTQUxMpr+TPlfIjBhbQpni9lbChlRRZmxCRRuCt/jyMmleVLxq5fK
[Table]
/cZLsQRMLGoqqbrq7/FhwbVz328mtrW9sbuW3Czu7e/sHxcOjpo4SxbDBIhGptk81Ci6xYbgR2I4V0tAX2PLHtzO/9YRK80g+mEmMvZAOJQ84o8ZKLc99TC+q036x5JbdOcgq8TJSggz1fvGrO4hYEqI0TFCtO54bm15KleFM4LTQTTTGlI3pEDuWShqi7qXzc6fkzCoDEkTKljRkrv6eSGmo9ST0bWdIzUgvezPxP6+TmOC6l3IZJwYlWywKEkFMRGa/kwFXyIyYWEKZ4vZWwkZUUWZsQgUbgrf88ippXpa9Srl6XynVbrI48nACp3AOHlxBDe6gDg1gMIZneIU3J3ZenHfnY9Gac7KZY
[Table]
=
=</
[Table]
<latexit sha1_base64="7U3u/
cpF49kdEcsvjtgLvpfrZV4=">AAAB7nicbVDLSgNBEOyNrxhfUY9eBoPgxbAr8XEMevEYwTwgWcPspJMMmZ1dZmaFsOQjvHhQxKvf482/cZLsQRMLGoqqbrq7glhwbVz328mtrK6tb+Q3C1vbO7t7xf2Dho4SxbDOIhGpVkA1Ci6xbrgR2IoV0jAQ2AxGt1O/+YRK80g+mHGMfkgHkvc5o8ZKTc99TM8uJ91iyS27M5Bl4mWkBBlq3eJXpxexJERpmKBatz03Nn5KleFM4KTQSTTGlI3oANuWShqi9tPZuRNyYpUe6UfKljRkpv6eSGmo9TgMbGdIzVAvelPxP6+dmP61n3IZJwYlmy/qJ4KYiEx/Jz2ukBkxtoQyxe2thA2poszYhAo2BG/x5WXSOC97lfLFfaVUvcniyMMRHMMpeHAFVbiDGtSBwQie4RXenNh5cd6dj3lrzslmDuEPnM8fRZqO4g==</
[Table]
<latexit sha1_base64="0VXXdbYgYISBa9T50LdFhRw
[Table]
[Figure]
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
latexit sha1_base64=
[Table]
+
[Table]
[Table] [Table]
[Table] [Table]
</
[Table]
<
[Table] [Table]
[Table] [Table]
[Table] [Table]
+/
[Table] [Table]
/RJGCtb0pCZ+nsio5HW4yiwnRE1Q73oTcX/vE5qwms/4zJJDUo2XxSmgpiYTH8nfa6QGTG2hDLF7a2EDamizNiEijYEb/HlZdK8qHjVyuV9tVy7yeMowDGcwBl4cAU1uIM6NIDBCJ7hFd6cxHlx3p2PeeuKk88cwR84nz9CkI7g</latexit>
[Table]
<latexit sha1_base64="xvcH9C5TuLdxC9cDiey45U+toyc=">AAAB7nicbVDLSgNBEOyNrxhfUY9eBoPgxbArCXoMevEYwTwgiWF20psMmZ1dZmaFsOQjvHhQxKvf482/cZLsQRMLGoqqbrq7/FhwbVz328mtrW9sbuW3Czu7e/sHxcOjpo4SxbDBIhGptk81Ci6xYbgR2I4V0tAX2PLHtzO/9YRK80g+mEmMvZAOJQ84o8ZKLc99TC+q036x5JbdOcgq8TJSggz1fvGrO4hYEqI0TFCtO54bm15KleFM4LTQTTTGlI3pEDuWShqi7qXzc6fkzCoDEkTKljRkrv6eSGmo9ST0bWdIzUgvezPxP6+TmOC6l3IZJwYlWywKEkFMRGa/kwFXyIyYWEKZ4vZWwkZUUWZsQgUbgrf88ippXpa9Srl6XynVbrI48nACp3AOHlxBDe6gDg1gMIZneIU3J3ZenHfnY9Gac7KZY/gD5/MHRBWO4Q==</latexit>
[Table]
<latexit sha1_base64="7U3u/
cpF49kdEcsvjtgLvpfrZV4=">AAAB7nicbVDLSgNBEOyNrxhfUY9eBoPgxbAr8XEMevEYwTwgWcPspJMMmZ1dZmaFsOQjvHhQxKvf482/cZLsQRMLGoqqbrq7glhwbVz328mtrK6tb+Q3C1vbO7t7xf2Dho4SxbDOIhGpVkA1Ci6xbrgR2IoV0jAQ2AxGt1O/+YRK80g+mHGMfkgHkvc5o8ZKTc99TM8uJ91iyS27M5Bl4mWkBBlq3eJXpxexJERpmKBatz03Nn5KleFM4KTQSTTGlI3oANuWShqi9tPZuRNyYpUe6UfKljRkpv6eSGmo9TgMbGdIzVAvelPxP6+dmP61n3IZJwYlmy/qJ4KYiEx/Jz2ukBkxtoQyxe2thA2poszYhAo2BG/x5WXSOC97lfLFfaVUvcniyMMRHMMpeHAFVbiDGtSBwQie4RXenNh5cd6dj3lrzslmDuEPnM8fRZqO4g
[Table]
=</latexit>
[Figure]
<latexit sha1_base64="jSFUO5ClO4n3yRf/LxuFc2YNhrA=">AAAB7XicbVBNSwMxEJ2tX7V+VT16CRbBU9mIoseiF48V7Ae0a8mm2TY2myxJVihL/4MXD4p49f9489+YtnvQ1gcDj/dmmJkXJoIb6/vfXmFldW19o7hZ2tre2d0r7x80jUo1ZQ2qhNLtkBgmuGQNy61g7UQzEoeCtcLRzdRvPTFtuJL3dpywICYDySNOiXVSE/sPGZ70yhW/6s+AlgnOSQVy1Hvlr25f0TRm0lJBjOlgP7FBRrTlVLBJqZsalhA6IgPWcVSSmJkgm107QSdO6aNIaVfSopn6eyIjsTHjOHSdMbFDs+hNxf+8TmqjqyDjMkktk3S+KEoFsgpNX0d9rhm1YuwIoZq7WxEdEk2odQGVXAh48eVl0jyr4vPqxd15pXadx1GEIziGU8BwCTW4hTo0gMIjPMMrvHnKe/HevY95a8HLZw7hD7zPH9M8jqY=</latexit>
[Figure]
[Table]
/Im/
[Table]
=
=</latexit>
[Figure]
<latexit sha1_base64="FhtoraXXOURooW75ux8CSEPf/Nk=">AAAB7nicbVDLSgNBEOyNrxhfUY9eBoPgxbArEXMMevEYwTwgiWF20psMmZ1dZmaFsOQjvHhQxKvf482/cZLsQRMLGoqqbrq7/FhwbVz328mtrW9sbuW3Czu7e/sHxcOjpo4SxbDBIhGptk81Ci6xYbgR2I4V0tAX2PLHtzO/9YRK80g+mEmMvZAOJQ84o8ZKLc99TC+q036x5JbdOcgq8TJSggz1fvGrO4hYEqI0TFCtO54bm15KleFM4LTQTTTGlI3pEDuWShqi7qXzc6fkzCoDEkTKljRkrv6eSGmo9ST0bWdIzUgvezPxP6+TmKDaS7mME4OSLRYFiSAmIrPfyYArZEZMLKFMcXsrYSOqKDM2oYINwVt+eZU0L8tepXx1XynVbrI48nACp3AOHlxDDe6gDg1gMIZneIU3J3ZenHfnY9Gac7KZY/gD5/MHSKSO5A==</latexit>
[Figure]
<latexit sha1_base64="hKBmBJCwE7xCXxZwmESnt+gX6OM=">AAAB73icbVDLSgNBEOyNrxhfUY9eBoPgxbArET0GvXiMYB6QrGF2MkmGzM6uM71CWPITXjwo4tXf8ebfOEn2oIkFDUVVN91dQSyFQdf9dnIrq2vrG/nNwtb2zu5ecf+gYaJEM15nkYx0K6CGS6F4HQVK3oo1p2EgeTMY3Uz95hPXRkTqHscx90M6UKIvGEUrtTz3IT3z3Em3WHLL7gxkmXgZKUGGWrf41elFLAm5QiapMW3PjdFPqUbBJJ8UOonhMWUjOuBtSxUNufHT2b0TcmKVHulH2pZCMlN/T6Q0NGYcBrYzpDg0i95U/M9rJ9i/8lOh4gS5YvNF/UQSjMj0edITmjOUY0so08LeStiQasrQRlSwIXiLLy+TxnnZq5Qv7iql6nUWRx6O4BhOwYNLqMIt1KAODCQ8wyu8OY/Oi/PufMxbc042cwh/4Hz+AK15jxc=</latexit>
Our main result, Theorem 3.4, and Proposition 2.4 imply that if both players simultaneously update their policies using equation (10) with a uniform magnet in 2p0s NFGs, then their joint policy converges to the α -QRE exponentially fast.
Similarly, in EFGs, if both players use a type of policy called sequence form with ψ taken to be dilated entropy, then their joint policy converges to the α -QRE exponentially fast.
Both of these results also hold more generally for equilibria induced by non-uniform magnet policies.
MMD can also be a considered as a behavioral-form algorithm in which update rule (10) or (11) is applied at each information state.
If ρ is uniform, a fixed point of this instantiation is an α -AQRE; more generally, fixed points are regularized equilibria (i.e., fixed points of a regularized best response operator).
E XPERIMENTS
Our main body focuses on highlighting the high level takeaways of our main experiments.
Additional discussion of each experiment, as well as additional experiments, are included in the appendix.
Experimental Domains For tabular normal-form settings, we used a 2p0s variant of Diplomacy—an
AI benchmark (Paquette et al., 2019) in which each turn can be viewed as a normal-form game because all players act simultaneously and the board state is fully observed.
These games have payoff matrices of shape p 50 , 50 q , p 35 , 43 q , p 50 , 50 q , and p 4 , 4 q , respectively, and were constructed using an open-source value function (Bakhtin et al., 2021).
For tabular extensive-form settings, we used games implemented in OpenSpiel (Lanctot et al., 2019):
Kuhn Poker, 2x
Abrupt Dark Hex, 4-Sided Liar’s Dice, and Leduc Poker.
These games have 54, 471, 8176, and 9300 non-terminal histories, respectively.
For function approximation single-agent settings, we used the selection of
Atari (Bellemare et al., 2013) and Mujoco (Todorov et al., 2012) games chosen by (Huang et al., 2022).
For function approximation multi-agent settings, we used 3x3 Abrupt Dark Hex, which is also implemented in OpenSpiel.
We found that 2x2 and 3x2 have 471 and 245 , 243 non-terminal histories, respectively, but were unable to compute the number for 3x3 due to its size.
Convergence to Quantal Response Equilibria
First, we examine MMD’s performance as a QRE solver.
We used Ling et al.
’s solver to compute ground truth solutions for NFGs and Gambit (McKelvey, Richard D., McLennan, Andrew M., and Turocy, Theodore L.) to compute ground truth solutions for EFGs.
We show the results in Figure 1.
We compared against algorithms introduced by Cen et al.
We show results in the top row of the figure, with each algorithm using the largest stepsize allowed by theory.
All three algorithms converge exponentially fast, as is guaranteed by theory.
The middle row shows results for QREs on EFG benchmarks.
For Kuhn Poker and 2x
Abrupt Dark Hex, we observe that MMD’s divergence converges exponentially fast, as is also guaranteed by theory.
For 4-Sided Liar’s Dice and Leduc Poker, we found that Gambit had difficulty approximating [Table] [Table] +Xtys7u3v5B9fCoo6NEEdomEY9UL8CaciZp2zDDaS9WFIuA024wvc387hNVmkXywcxi6gs8lixkBJtM8txHd1ituXU3B1olXkFqUKA1rH4NRhFJBJWGcKx133Nj46dYGUY4nVcGiaYxJlM8pn1LJRZU+2l+6xydWWWEwkjZkgbl6u+JFAutZyKwnQKbiV72MvE/r5+Y8NpPmYwTQyVZLAoTjkyEssfRiClKDJ9Zgoli9lZEJlhhYmw8FRuCt/zyKulc1L1G/fK+UWveFHGU4QRO4Rw8uIIm3EEL2kBgAs/wCm+OcF6cd+dj0Vpyiplj+APn8wcNTo2Z</latexit>
[Figure]
i [Figure]
ili [Figure] [Table]
the QREs, due to the size of the games.
Thus, we instead report the saddle point gap (the sum of best response values in the regularized game), for which we observe linear convergence, as is guaranteed by Proposition D.6.
The bottom row shows results for AQREs using behavioral form MMD (with η “ α { 10 ) on the same benchmarks, where we also observe convergence (despite a lack of guarantees).
For further details, see Sections G.3 for the QRE experiments and Section G.4 for the AQRE experiments.
Exploitability Experiments From our AQRE experiments, it immediately follows that it is possible to use behavioral-form MMD with constant stepsize, temperature, and magnet to compute strategies with low exploitabilities.
Indeed, we show such results (again with η “ α { 10 and a uniform magnet) in the top row of Figure 2.
A natural follow up question to these experiments is whether MMD can be made into a Nash equilibrium solver by either annealing the amount of regularization over time or by having the magnet trail behind the current iterate.
We investigate this question in the bottom row of Figure 2 by comparing i) MMD with an annealed temperature, annealed stepsize, and constant magnet; ii)
MMD with a constant temperature, constant stepsize, and moving magnet; iii) CFR (Zinkevich et al., 2007); and iv) CFR+ (Tammelin, 2014).
While CFR+ yields the strongest performance, suggesting that it remains the best choice for tabularly solving games, we view the results as very positive.
Indeed, not only do both variants of MMD exhibit convergent behavior, they also perform competitively with (or better than) CFR.
This is the first instance of a standard RL algorithm yielding results competitive with tabular CFR in classical 2p0s benchmark games.
For further details, see Section H.3 for the annealing temperature experiments and Section H.5 for the moving magnet experiments.
Deep Single-Agent Reinforcement Learning Next, we examine whether MMD is competitive as a single-agent RL algorithm compared to PPO (Schulman et al., 2017).
We implement MMD as an adaptation of Huang et al.’s
PPO implementation.
We show results, compared to those reported for
PPO by Huang et al.
(2022) in Table 1.
The Atari results are for 10M time steps; the Mujoco results are for 2M time steps.
While the exact numbers should be taken lightly, as they are over only three seeds, the results suggest that MMD can perform roughly comparably to PPO in these settings.
This should not be seen as a surprising result, but rather a confirmation of the reality that MMD can be implemented in a fashion that shares many similarities with PPO.
For further details, see Section I.
Deep Multi-Agent Reinforcement Learning Our last experiment examines MMD as a deep multi-agent RL algorithm using self play.
We benchmarked against NFSP (Heinrich & Silver, 2016).
We implemented MMD by modifying RLlib’s
(Liang et al., 2018) independent PPO implementation, [Table]
[Table] and used OpenSpiel’s (Lanctot et al., 2019) NFSP implementation.
This setup gives an advantage to NFSP in that RLlib’s OpenSpiel interface does not inform the agent about which actions are legal (so MMD is playing a harder version of the game), but an advantage to MMD in that it is likely that RLlib’s PPO implementation is more optimized than OpenSpiel’s NFSP implementation.
As the game is too large to compute exact exploitability, we approximate exploitability using a DQN best response, trained for 10 million time steps.
The results are shown in Table 2, with standard error over five training runs.
The results include checkpoints after both 1 million and 10 million time steps, as well as bots that select the first legal action and that select actions uniformly at random.
As expected, both NFSP and MMD both yield lower approximate exploitability after 10M steps than they do after 1M steps; that said, among the two, MMD produces substantially better results at both checkpoints.
We also show results of head-to-head match-ups in Table 3 for the 10M time step checkpoints.
We find the MMD tends to outperform NFSP, both in direct head-to-head match-ups and in terms of exploiting the first action taking and uniformly random bots.
For further details, see Section J.
Subject Matter of Additional Experiments In the appendix, we include 14 additional experiments:
Trajectory visualizations for MMD applied as a QRE solver for NFGs (Section D.4); 2. Trajectory visualizations for MMD applied as a saddle point solver (Section D.5) 3. Solving for QREs in NFGs with black box (b.b.) sampling (Section G.2); [-5) results for QRE solving for Kuhn Poker and 2x
-]4.
{+Solving for QREs in NFGs with b b. sampling under different gradient estimators (Section G.2); 5. Solving for QREs in Kuhn Poker & 2x
+} Abrupt Dark Hex, shown in saddle point gap (Section G.3); 6.
Solving for Nash in NFGs with full feedback (Section H.1); 7. Solving for Nash in NFGs with black box sampling (Section H.2); 8. Solving for Nash in EFGs with a reach-probability-weighted stepsize (Section H.3); 9. Solving for Nash in EFGs with black box sampling (Section H.4); 10.
Solving for Nash in EFGs using MaxEnt and MiniMaxEnt objectives (Section H.6); 11. Solving for Nash in EFGs with a Euclidean mirror map (Section H.7); 12. Solving for MiniMaxEnt equilibria (Section H.8); 13. Learning in EFGs with constant hyperparameters and a MiniMaxEnt objective (Section H.9); 14.
Deep MARL in 3x
Abrupt Dark Hex stratified by training run (Section J).
R ELATED W ORK
We discuss the main related work below.
Additional related work concerning average policy deep reinforcement learning for 2p0s games can be found in Section K.
Convex Optimization and Variational Inequalities Like MMD and Algorithm 3.1, the extragradient method (Korpelevich, 1976; Gorbunov et al., 2022) and the optimistic method (Popov, 1980) have also been studied in the context of zero-sum games and variational inequalities more generally.
However, in contrast to MMD, neither of these methods are guaranteed to converge without smoothness.
Outside the context of variational inequalities, analogues of MMD and Algorithm 3. have been studied in convex optimization under the non-Euclidean proximal gradient method (Beck, 2017) originally proposed by Tseng (2010).
But, in contrast to Theorem 3.4, existing convex optimization results (Beck, 2017; Tseng, 2010; Hanzely et al., 2021; Bauschke et al., 2017) are without linear rates because they do not assume the proximal regularization to be relatively-strongly convex.
In addition to convex optimization, the non-Euclidean proximal gradient algorithm has also been studied in online optimization under the name composite mirror descent (Duchi et al.,
Duchi et al.
(2010) show a O p?
t q regret bound without strong convexity assumptions on the proximal term.
In the case where the proximal term is relatively strongly convex, Duchi et al.
(2010) give an improved rate of O p log t q —implying that MMD has average iterate convergence with a rate of O p log t { t q for bounded problems, like QRE solving.
Quantal Response Equilibria Among QRE solvers for NFGs, the PU and OMWPU algorithms from Cen et al.
(2021), which also possess linear convergence rates for NFGs, are most similar to
MMD.
However, both PU and OMWPU require two steps per iteration (because of their similarities to mirror-prox
(Nemirovski, 2004) and optimistic mirror descent (Rakhlin & Sridharan, 2013)), and
PU requires an extra gradient evaluation.
In contrast, our algorithm needs only one simple step per iteration (with the same computation cost as mirror descent) and our analysis applies to various choices of mirror map, meaning our algorithm can be used to compute a larger class of regularized equilibria, rather than only QREs.
Among QRE solvers for EFGs, existing algorithms differ from
MMD in that they either require second order information (Ling et al., 2018) or are first order methods with average iterate convergence (Farina et al., 2019; Ling et al., 2019).
In contrast to these methods,
MMD attains linear last-iterate convergence.
Single-Agent Deep Reinforcement Learning Considered as a reinforcement learning algorithm,
MMD with a negative entropy generating function bears close resemblance to both KL-PPO (a variant of PPO that served as motivation for the widely adopted gradient clipping variant) (Schulman et al., 2017) and MDPO (Tomar et al., 2020).
In short, the negative entropy instantiation of MMD corresponds with KL-PPO with a flipped KL term and with MDPO when there is entropy regularization.
We describe these relationships using symbolic expressions in Section L.
Regularized Follow-the-Regularized-Leader Another line of work has combined follow-theregularized-leader with additional regularization, under the names follow-the-regularized-leader (F-FoReL) (Pérolat et al., 2021) and piKL (Jacob et al., 2022), in an analogous fashion to how we combine mirror descent with additional regularization.
These two methods have deep similarities but were created for different purposes.
Similarly to our work, F-FoReL was designed for the purpose of achieving last iterate convergence in two-player zero-sum games.
In terms of convergence guarantees, we prove discrete-time linear convergence for NFGs, while Pérolat et al.
(2021) give continuous-time linear convergence (which does not imply discrete time) for EFGs under trajectory importance re-weighting.
In terms of easeof-use, MMD offers the advantage that it is decentralizable, whereas the version of F-FoReL that
Pérolat et al.
(2021) present is not.
In terms of scalability, MMD offers the advantage that it only requires approximating bounded quantities; in contrast, F-FoReL requires estimating an arbitrarily accumulating sum (in practice, Pérolat et al.
(2021) clip the sum using the approach of Hennes et al.
Lastly, in terms of empirical performance, the tabular results presented in this work for
MMD are substantially better than those presented for F-FoReL. For example, F-FoReL’s best result in Leduc is an exploitability of about 0.08 after 200 , 000 iterations—it takes MMD fewer than 1 ,iterations to achieve the same value.
On the other hand, piKL was motivated by improving the prediction accuracy of imitation learning via decision-time planning.
We believe the success of piKL in this context suggests that MMD may also perform well in such a setting.
While Jacob et al.
(2022) also attains convergence to KL-regularized equilibria in NFGs, our results differ in two ways: First, our results only handle the full feedback case, whereas Jacob et al.
(2022)’s
results allow for stochasticity.
Second, our results give linear last-iterate convergence, whereas Jacob et al.
(2022) only show O p log t { t q average-iterate convergence.
C ONCLUSION
In this work, we introduced MMD—an algorithm for reinforcement learning in single-agent settings and 2p0s games, and regularized equilibrium solving.
We presented a proof that MMD converges exponentially fast to QREs in EFGs—the first algorithm of its kind to do so.
We showed empirically that MMD exhibits desirable properties as a tabular equilibrium solver, as a single-agent
deep
RL algorithm, and as a multi-agent deep RL algorithm.
This is the first instance of an algorithm exhibiting such strong performance across all of these settings simultaneously.
We hope that, due to its simplicity, MMD will help open the door to 2p0s games research for RL researchers without game theoretic backgrounds.
We provide directions for future work in Section M. Appendices Table of Contents C
C.2 Proof for Proposition 2.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.3 MMD Normal-Form QREs over the Sequence-Form . . D Proofs D.1 Supporting Lemmas and Propositions . . . . . . . . . . . . . . . . . . . . . . .
D.2 Proof of Theorem 3.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.3 Equivalence between MMD and MD . . . . . . . . . . . . . . . . . . . . . . .
D.4 Negative Entropy MMD Example . . . . . . . . . . . . . . . . . . . . . . . . .
{+D.
+}
Euclidean MMD Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.6 Bounding the Gap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E MMD for Logit-AQREs and MiniMaxEnt Equilibria
F Experimental Domains
G QRE Experiments
G.
1 Full Feedback QRE Convergence Diplomacy
G
2 Black Box QRE Convergence Diplomacy
G.3 Full Feedback QRE Convergence EFGs
G.4 Full Feedback AQRE Convergence EFGs
H Exploitability Experiments
H
1 Full Feedback Nash Convergence Diplomacy
H
2 Black Box Nash Convergence Diplomacy
H
3 Full Feedback Nash Convergence EFGs
H
4 Black Box Nash Convergence EFGs
H.5 Moving Magnet
H.6 MaxEnt and MiniMaxEnt Objectives
H.7 Euclidean Mirror Map Over Logits
H.8 MiniMaxEnt Exploitability with Fixed Parameters
H.9 Exploitability with Fixed Parameters
I Atari and
Mujoco
Experiments
{+J 3x
+}
Abrupt Dark Hex Experiments
K Additional Related Work
L Relationship to KL-PPO and MDPO
M Future Work
A P ROBLEM S ETTING
In our notation, we use
• s P S to notate Markov states, • a i P A i to notate actions, • o i P O i to notate observations, • h
i P H
i “ Ť t p
O i ˆ
A
i q t
ˆ O i to denote information states (i
e., decision points).
We use
• T : S ˆ A Ñ ∆ p S Y tKuq to notate the transition function, where K notates termination, • R
i : S ˆ A Ñ R to notate a reward function, • O i : S ˆ
A Ñ
O i to notate an observation function.
A
i : H i Ñ A i to notate a legal action function.
We are interested in 2p0s games, in which i P t 1 , 2 u and @ s, a, R 1 p s, a q “ ´ R 2 p s, a q .
For convenience, we use ´ i to notate the player “not i ”.
Single-agent settings are captured as a special case in which the second player has a trivial action set | A 2 | “ 1 .
Normal-form games are captured as a special case in which there is only one state s and the transition function only supports termination: @ a, supp p T p s, a qq “ tKu .
(Here, we use supp p X q to denote the support of a distribution X —i.e., the subset of the domain of X that is mapped to a value greater than zero: t x : X p x q ą 0 u .)
Each agent’s goal is to maximize its expected return
[Equation]
using its policy π i , which dictates a distribution over actions for each information state
[Equation]
In game theory literature, these policies are called behavioral form and assume perfect recall.
We notate the expected value for an agent’s return a i at an information state h
i at time t under joint policy π as
[Equation]
ff
[Equation]
Here, the first expectation samples the current Markov state S and the current opponent action
A
´ i from the posterior induced by player i reaching information state h
i , when each player uses its part of joint policy π to determine its actions.
The second expectation is over trajectories under the same conditions, with the additional condition that a i is the agent’s action at the current time step.
A. R EDUCTION TO N ORMAL F ORM
Given any game of the above form, we can reduce the game to normal form as follows.
Let ¯Π i denote the set of deterministic policies—i.e., the set of policies that support exactly one action at a time:
[Equation]
The action space of the normal-form game is the space of deterministic policies:
˜ A i “ ¯Π i .[-
-]
The reward function of the normal-form game is dictated by the expected return of the deterministic joint policy:
[Equation]
Remark A.1.
Any policy π i can be expressed as a finite mixture over policies in ¯Π i in a fashion that induces the same distribution over trajectories (against arbitrary, but fixed, opponents).
Conversely, any finite mixture over policies in ¯Π i can be expressed as a policy π i that induces the same distribution over trajectories (against arbitrary, but fixed, opponents).
By the remark above, joint policies in the original game possess counterparts in the normal-form game (and vice versa) achieving identical expected returns.
It is in the sense that the normal-form game is equivalent to the original game.
A more detailed exposition on this equivalence can be found in Shoham & Leyton-Brown (2008).
B S OLUTION C ONCEPTS
Nash equilibria are perhaps the most commonly sought-after solution concept in 2p0s games.
A joint policy π 1 , π 2 is a Nash equilibrium if neither player can improve its expected return by changing its policy (assuming the other player does not change its policy):
[Equation]
Note that, in single-agent settings, this corresponds with the notion of an optimal policy in reinforcement learning.
Another solution concept is a logit quantal response equilibrium (McKelvey & Palfrey, 1995; 1998).
As we only deal with logit quantal response equilibria, we generally drop logit and refer to them simply as quantal reponse equilibria.
In normal-form games, there are multiple equivalent ways to define a quantal response equilibrium.
One way is using entropy regularization.
We say a joint policy is an α -QRE in a normal-form game if each player maximizes a weighted combination of expected return and policy entropy
[Equation]
In a temporally-extended game, we say a joint policy is an α -QRE if the equivalent mixture over deterministic joint policies is an α -QRE of the equivalent normal-form game.
An alternative way to extend QREs to temporally extended settings is to ask that they satisfy the normal-form QRE condition at each information state:
[Equation]
When a joint policy satisfies this condition, it is called an agent QRE (as it is as if there is a separate agent playing a part of a normal-form QRE at each information state).
In single-agent settings, α -AQREs correspond with the fixed point of the instantiation of expected SARSA (Sutton & Barto, 2018) in which the policy is a softmax distribution over Q-values with temperature α .
The last solution concept that we investigate is called the MiniMaxEnt equilibirum.
A joint policy is an α -MiniMaxEnt equilibrium if it satisfies condition (13) for MiniMaxEnt Q -values
[Equation]
Alternatively, α -MiniMaxEnt equilibria can be defined as the saddlepoint of the α -MiniMaxEnt objective [Equation] While the name MiniMaxEnt is novel to this work, the concept has been studied in recent existing work (Pérolat et al., 2021; Cen et al., 2021). C R EDUCED N ORMAL -F ORM L OGIT -QRE S AND MMD C. S EQUENCE -F ORM B ACKGROUND
A Nash-equilibrium in a 2p0s extensive-form game can be formulated as a bilinear saddle point problem over the sequence form (Nisan et al., 2007)
[Equation]
where X and Y are the sequence form polytopes, which equivalently can be viewed as treeplexes (Hoda et al., 2010; Kroer et al., 2020).
We provide some background on the sequence form in the context of the min player (player 1); the max player follows similarly.
Recall all the decision points for player 1 are denoted as H 1 (also known as information states) and the actions available at decision point h P H 1 are A 1 p h q .
Recall that a policy (a.k.a behavioral-form strategy) is denoted as π 1 with π 1 p
h q P ∆ p
A 1 p h qq being the policy at decision point h .
For convenience, let π 1 p h, a q denote the probability of taking action a P A 1 p
h
q at decision point h .
Next we denote p p
h q as the parent sequence to reach decision point h —that is, the unique previous decision point and action taken by the player before reaching h .
Note that this parent is unique due to perfect recall and that it is possible for many decision points to share the same parent.
Then we can construct the sequence form from the top down, where the p h, a q sequence of x P X is given by
[Equation]
For convenience, the root sequence ∅ is defined to be the parent of all initial decision points of the game and is set to the constant x ∅ “ 1 .
We denote x h as the slice of x “ p x p h,a q q h P H 1 ,a P A p h q corresponding to decision point h .
Note we have the following relationship
[Equation]
Because x p h,a q corresponds to the probability of player 1 choosing all actions along the sequence until reaching p h, a q , we get that x
J Ay is the expected payoff for player 2 given a pair of sequence-form strategies x, y .
Thus the sequence form allows us to get a bilinear objective.
Given the bilinear structure of the sequence-form problem, we convert the problem into a VI using first-order optimality conditions.
In order to apply MMD (or other first-order methods), we need a good choice of a mirror map for X and Y .
A such choice is the class of dilated distance generating functions (Hoda et al., 2010):
[Equation]
where p β h q h P H 1 ą 0 are per-decision-point weights and ψ h is a distance-generating function for the simplex associated to h .
If ψ h is taken to be the negative entropy then we say ψ is the dilated entropy function.
In the normal-form setting the dilated entropy is simply the standard negative entropy.
Recently it was shown that an α -QRE (for the reduced normal form) is the solution to the following saddle point problem over the sequence form (Ling et al., 2018), [Equation]
[Equation]
where ψ 1 , ψ 2 are dilated entropy functions with weights β h “ 1 .
Note that we have the normal form α -QRE as a special case of equation 16.
C. P ROOF FOR P ROPOSITION 2.
Proposition 2.4.
Solving a normal-form reduced QRE in a two-player zero-sum EFG is equivalent to solving VI p Z , F ` α ∇ ψ q where Z is the cross-product of the sequence form strategy spaces and ψ is the sum of the dilated entropy functions for each player.
The function ψ is strongly convex with respect to } ¨ } .
Furthermore, F is monotone and max ij |
A ij | -smooth ( A being
Proof.
The problem of finding a reduced normal-form logit QRE is equivalent to solving the saddlepoint problem stated in equation 16 (Ling et al., 2018).
Therefore, due to the convexity of ψ 1 and ψ 2 and the discussion from Section 2.3, we have that the solution to equation 16 is equivalent to the solution of VI p Z , F ` ∇ ψ q where,
[Equation]
From Hoda et al.
we know there exists constants µ 1 , µ 2 such that ψ 1 is µ 1 -strongly convex over X with respect to } ¨ } 1 and ψ 2 is µ 2 -strongly convex over Y with respect to } ¨ } 1 (Hoda et al.
(2010) do not show bounds on these constants, but we only need them to exist).
Therefore, ψ is also strongly convex over Z with constant min t µ 1 , µ 2 u with respect to } z } “ a } x } 21 ` } y } 21 since, for z “ p x, y q , and z 1 “ p x 1 , y 1 q we have
[Equation]
Following Theorem 3.4, it is useful to characterize the smoothness of F under the same norm for which ψ is strongly-convex.
First, notice that for any matrix A we have that } Ax ´ Ay } 8 ď max ij | A ij |} x ´ y } 1 (see for example Bubeck et al.
(2015)[Section 5.2.4]).
Therefore altogether we have
[Equation]
showing that F is L “ max ij | A ij | -smooth with respect to } ¨ } .
The strong-monotonicity of F ` ∇ ψ follows since F is monotone and ∇ ψ is strongly monotone since ψ is strongly convex.
Note that in general the Hessian can have unbounded entries (Kroer et al., 2020) meaning that ∇ ψ cannot be L -smooth (Beck, 2017).
Our MMD algorithm which handles ψ in closed form allows us to sidestep this issue.
We also have that the dual norm of }¨} is simply } z } ˚ “
[Equation]
8 ` } y } 2 8 (Bubeck et al., 2015; Nemirovski, 2004).
C. MMD FOR F INDING R EDUCED N ORMAL -F ORM QRE S OVER THE S EQUENCE -F ORM
From Proposition 2.4 and Corollary D.6 we have that the MMD descent-ascent updates (8-9) with ψ 1 , ψ 2 , taken to be dilated entropy with η ď α { max ij | A ij | 2 converges linearly to the solution of equation 16.
The updates, as mentioned by Remark 3.7, can be computed in closed-form as a one-line change to mirror descent with dilated-entropy (Kroer et al., 2020).
Indeed, setting g t “ Ay t (the gradient for the min player), we have that the update for the min player can be written as follows
[Equation]
Updates can be computed in closed-form starting from decision points h without any children and progressing upwards in the game tree.
D P ROOFS D. S UPPORTING L EMMAS AND P ROPOSITIONS
Proposition D.1 ((Bauschke et al., 2003)Proposition 2.3) .
Let t x, y u Ă dom ψ and t u, v u Ă int dom ψ .
Then
[Equation]
The following result is also known as the non-Euclidean prox theorem (Beck, 2017)[Theorem 9.12] or the three-point property(Tseng, 2008).
Proposition D.2.
Assume Z closed convex and both f and ψ are differentiable at ¯ z (defined below)
[Equation]
Then the following statements are equivalent
[Equation]
Proof.
[Equation]
The first equivalence follows by the first-order optimality condition and the last one by Proposition
D.1.
Lemma D.3.
One step of Algorithm 3.1, under the assumptions of Theorem 3.4, guarantees that, for all z P Z ,
[Equation]
Proof.
Immediate from Proposition D.2.
Lemma D.4.
Under the same assumptions as Theorem 3.
[Equation]
let z ˚ be the solution to VI p Z , F ` α ∇ g q .
Then, for any z P Z X int dom ψ [Equation] the following inequality holds
[Equation]
Proof.
[Equation]
Note that ∇ g p z q , ∇ g p z ˚ q , ∇ ψ p z q , ∇ ψ p z ˚ q are all well-defined because z ˚ P int dom ψ and Assumptions (3.2-3.3).
The first inequality follows since F is monotone and z ˚ P
sol VI p Z , F ` α ∇ g q .
The second inequality follows since g is 1-strongly convex relative to ψ and the last equality by
Proposition D.1.
D. P ROOF OF T HEOREM 3.
Theorem 3.4.
Let Assumptions 3.2 and 3.3 hold and assume the unique solution z ˚ to VI p Z , F ` α ∇ g q satisfies z ˚ P int dom ψ .
Then Algorithm 3.1 converges if η ď αL 2 and guarantees
[Equation]
Proof. [Equation]
` B ψ p z
[Equation] z t ` 1 qq ď
[Equation] z
[Equation]
The first inequality follows from Lemma D.3 and the second inequality from Lemma D.4; the third inequality by the generalized Cauchy-Schwarz inequality and the smoothness of F ; the fourth inequality by elementary inequality ab ď ρa 2 2 ` b 2 2 ρ @ ρ ą 0 ; and the fifth inequality by the strong convexity of ψ since 12 } x ´ y } 2 ď B ψ p x ; y q .
Therefore altogether we have
[Equation]
Iterating the inequality yields the result.
Corollary D.6.
Under the same assumptions as Theorem 3.4, if g is µ -strongly convex relative to ψ and ψ is µ ψ strongly convex, then if η ď αµL 2 , Algorithm 3.1 guarantees
[Equation]
Proof.
Observe that ¯ ψ “ ψµ ψ is 1 -strongly convex and ¯ g “ gµµ ψ is 1 -strongly convex relative to ¯ ψ ,
[Equation]
Rewriting the update of Algorithm 3.1 in terms of ¯ g and ¯ ψ gives
[Equation]
The result follows from Theorem 3.4 with stepsize ¯ η “ ηµ ψ and ¯ α “ µµ ψ α .
D. E QUIVALENCE BETWEEN MMD AND MD
In this section we show that MMD is equivalent to mirror descent (MD) with a different stepsize when an extra regularized loss is added.
In the game context this implies that MMD can be implemented as mirror descent ascent on the regularized game with a particular stepsize.
Proposition D.7.
Magnetic mirror descent updates (6, 7) are equivalent to the following updates respectively (where the second update uses a magnet z 1 ):
[Equation]
with stepsize ¯ η “
[Equation]
Proof.
We begin by proving the first equivalence, between equation (6) and (23):
[Equation]
The second equivalence follows from similar steps:
[Equation]
q ô z t ` 1 “ arg min z P
Z x ¯ η `
F p z t q ` α ∇ ψ p z t q ´ α ∇ ψ p z 1 q ˘ ´ ∇ ψ p z t q , z y ` ψ p z q ô z t ` 1 “ arg min z P Z x ¯ η ` F p z t q ` α ∇ z t B ψ p z t ; z 1 q ˘ ´ ∇ ψ p z t q , z y ` ψ p z q ô z t ` 1 “ arg min z P Z ¯ η x F p z t q ` α ∇ z t B ψ p z t ; z 1 q , z y ` B ψ p z ; z t q ..
[Table]
+lynAmcFLoJBpjykZ0gG1LJQ1R++ns3Ak5sUqP9CNlSxoyU39PpDTUehwGtjOkZqgXvan4n9dOTP/aT7mME4OSzRf1E0FMRKa/kx5XyIwYW0KZ4vZWwoZUUWZsQgUbgrf48jJpnJe9SvnyvlKq3mRx5OEIjuEUPLiCKtxBDerAYATP8ApvTuy8OO/Ox7w152Qzh/AHzucPQQuO3w==</latexit>
[Table]
<latexit sha1_base64="xvcH9C5TuLdxC9cDiey45U+toyc=">AAAB7nicbVDLSgNBEOyNrxhfUY9eBoPgxbArCXoMevEYwTwgiWF20psMmZ1dZmaFsOQjvHhQxKvf482/cZLsQRMLGoqqbrq7/FhwbVz328mtrW9sbuW3Czu7e/sHxcOjpo4SxbDBIhGptk81Ci6xYbgR2I4V0tAX2PLHtzO/9YRK80g+mEmMvZAOJQ84o8ZKLc99TC+q036x5JbdOcgq8TJSggz1fvGrO4hYEqI0TFCtO54bm15KleFM4LTQTTTGlI3pEDuWShqi7qXzc6fkzCoDEkTKljRkrv6eSGmo9ST0bWdIzUgvezPxP6+TmOC6l3IZJwYlWywKEkFMRGa/kwFXyIyYWEKZ4vZWwkZUUWZsQgUbgrf88ippXpa9Srl6XynVbrI48nACp3AOHlxBDe6gDg1gMIZneIU3J3ZenHfnY9Gac7KZY/gD5/MHRBWO4Q==</latexit>
[Table]
<latexit sha1_base64="5FGM+A/RphYM3UE+MaGVVGtOUWU=">AAAB7nicbVDLSgNBEOyNrxhfUY9eBoPgxbArkXgMevEYwTwgiWF20psMmZ1dZmaFsOQjvHhQxKvf482/cZLsQRMLGoqqbrq7/FhwbVz328mtrW9sbuW3Czu7e/sHxcOjpo4SxbDBIhGptk81Ci6xYbgR2I4V0tAX2PLHtzO/
[Table]
+
[Table]
+q036x5JbdOcgq8TJSggz1fvGrO4hYEqI0TFCtO54bm15KleFM4LTQTTTGlI3pEDuWShqi7qXzc6fkzCoDEkTKljRkrv6eSGmo9ST0bWdIzUgvezPxP
[Table]
TmOC6l3IZJwYlWywKEkFMRGa/kwFXyIyYWEKZ4vZWwkZUUWZsQgUbgrf88ippXpa9SvnqvlKq3WRx5OEETuEcPKhCDe6gDg1gMIZneIU3J3ZenHfnY9Gac7KZY
[Table]
</
[Table]
>
[Table]
+
[Table]
=">
[Table]
[Table]
[Table]
+8wmV5pF8MOMY/ZAOJO9zRo2Vmp77mJ5dT7rFklt2ZyDLxMtICTLUusWvTi9iSYjSMEG1bntubPyUKsOZwEmhk2iMKRvRAbYtlTRE7aezcyfkxCo90o+ULWnITP09kdJQ63EY2M6QmqFe9Kbif147Mf0rP+UyTgxKNl/UTwQxEZn+TnpcITNibAllittbCRtSRZmxCRVsCN7iy8ukcV72KuWL+0qpepPFkYcjOIZT8OASqnAHNagDgxE8wyu8ObHz4rw7H/PWnJPNHMIfOJ8/SimO5Q==</latexit>
[Table]
<latexit sha1_base
[Table]
< [Table] < [Table]
[Table]
> [Table] +KePV3vPk3TpI9aGJBQ1HVTXdXEAuujet+Oyura+sbm7mt/PbO7t5+4eCwoaNEMayzSESqFVCNgkusG24EtmKFNAwENoPRzdRvPqHSPJL3ZhyjH9KB5H3OqLFSy3Mf0nOvMukWim7JnYEsEy8jRchQ6xa+Or2IJSFKwwTVuu25sfFTqgxnAif5TqIxpmxEB9i2VNIQtZ/O7p2QU6v0SD9StqQhM/X3REpDrcdhYDtDaoZ60ZuK/3ntxPSv/JTLODEo2XxRPxHERGT6POlxhcyIsSWUKW5vJWxIFWXGRpS3IXiLLy+TxkXJK5cqd+Vi9TqLIwfHcAJn4MElVOEWalAHBgKe4RXenEfnxXl3PuatK042cwR/4Hz+ALUSjxw=</latexit>
KL Divergence to QRE
[Figure]
Simplex Trajectories
Payo ﬀ Matrix
An example showing the simplex trajectories of MMD applied to a small NFG is shown in Figure 3.
D. E UCLIDEAN MMD E XAMPLE
We discuss the Euclidean case for update equation 6 (update 7 is similar).
In the Eucldiean case were ψ “ 12 } ¨ } 22 we have that update equation 6 reduces to
[Equation]
where Π Z denotes the Euclidean projection onto Z .
In the context of solving min max problems where ψ “ ψ 1 ` ψ 2 , the sum of 12 } ¨ } 22 then the descent-ascent updates of (8,9) become
[Equation]
Note that our results don’t require bounded constraints—in the unconstrained setting there would be no projection step.
By Theorem 3.4 the above iterations converge linearly to the solution of
[Equation]
provided f is smooth in the sense that F p x, y q “ r ∇ x f p x, y q , ´ ∇ y f p x, y qs J is smooth.
For example, in the 1-D case we have that the following unconstrained saddle point problem can be solved with Euclidean MMD:
[Equation]
Where a, b are constants.
In this case F p x, y q “ r y ´ b, ´p x ´ a qs J , which is 1 -smooth.
Therefore, with stepsize η “ α the following update rule converges linearly to the solution:
[Equation]
See Figure 4 for a visualization with a “ b “ 1 .
600 Iterations
[Figure]
Divergence to equilibrium for Euclidean MMD =0.2 =0.
=1 = 0.
2.0 x 0.
y
Trajectories of Euclidean MMD =0.2 =0.
=1 =
[Figure]
B OUNDING THE G AP
Below we show how Theorem 3.4 can be used to guarantee linear convergence of the gap.
Proposition D.8.
Suppose the assumptions of Theorem 3.4 hold.
Moreover, assume that g is twice continuously differentiable over int dom ψ and Z is bounded.
In that case, there exists a constant C and a time step t 1 such that, for any t ě t 1 ,
[Equation]
Proof.
By Theorem 3.4 we know z t Ñ z ˚ where t z t u t ě 1 Y t
z ˚ u Ď int dom ψ .
Therefore, we have that t z t u t ě 1 Y t
z ˚ u is eventually within a closed ball centered at z ˚ .
That is, there exists tand a closed ball B such that t z t u t ě t 1 Y t
z ˚ u Ď B Ď int dom ψ .
Since B is compact and ∇ 2 g is continuous over B , we have that ∇ 2 g p
z q is bounded on B .
Therefore, there exists L B such that } ∇ g p z 1 q ´ ∇ g p z q} ˚ ď L B } z ´ z 1 } for any z, z 1 P B .
Setting G “ F ` ∇ g , we have that, for any z, z 1 P B , } G p
z
q ´ G p z 1 q} ˚ ď ˜ L } z ´ z 1 } for ˜ L “ L ` L B .
Then for any z P Z , t ě t 1 , denoting x ˚ as the solution to VI p Z , G q , we have
[Equation]
where D is such that max z,z 1 P Z } z ´ z 1 } ď D and C “ } G p z ˚ q} ˚ ` ˜ LD .
The first inequality is by the generalized Cauchy-Schwarz inequality and the Lipschitz property of G .
The second inequality is by boundedness of Z .
The third inequality is by the fact that B ψ p z ˚ ; z t q ě 1 2 } z ˚ ´ z t } 2 .
The fourth inequality is by applying Theorem 3.4 inductively.
Note that we have the following well-known inequality between the saddle-point gap
[Equation]
and
[Equation]
as shown below:
[Equation]
for some pair p
[Equation]
[Equation]
y
[Equation]
P
[Equation]
Therefore Proposition D.6 gives a guarantee on the saddle-point gap ξ p x, y q .
E MMD FOR L OGIT -AQRE S AND M INI M AX E NT E QUILIBRIA
By Proposition D
[Equation]
[Equation]
the MMD update equation 6, restated below, has fixed points corresponding to the solutions of VI p Z , F ` ∇ ψ q :
[Equation]
If Z is the cross-product of policy spaces for both players (cross product of sets of behavioral-form policies) and ψ is the sum of negative entropy over all decision points (information states)
[Equation]
and F includes the the negative q-values for both players, then the iteration above reduces to
[Equation]
with fixed points corresponding to
[Equation]
or, equivalently, the solution to VI p Z , F ` ∇ ψ q , which corresponds to a logit-AQRE.
If F includes the negative MiniMaxEnt Q -values for both players, the fixed point instead corresponds to a MiniMaxEnt equilibrium.
F E XPERIMENTAL D OMAINS
For our experiments with normal-form games, we used No-Press Diplomacy stage games.
No-Press
Diplomacy is a seven-player Markov game in which players compete to conquer Europe.
Because the game is a Markov game (which means that the game is fully observable but that the players move simultaneously), each turn of the game resembles a normal-form game.
We constructed the normalform games that we used for our experiments by querying an open source value function (Bakhtin et al., 2021) in different circumstances for a two-player variant of the game, similarly to Zhang et al.
These games have payoff matrices of shape p 50 , 50 q (game A), p 35 , 43 q (game B), p 50 , 50 q (game C), and p 4 , 4 q (game D).
We normalized the payoffs of each game to r 0 , 1 s .
For our extensive-form games, we used the implementations of Kuhn Poker, 2x2 (and also 3x3)
Abrupt Dark Hex, 4-Sided Liar’s Dice, and Leduc Poker provided by OpenSpiel (Lanctot et al., 2019).
Kuhn poker (Kuhn, 1951) is a simplified poker game with three cards (J, Q, K).
It hasnon-terminal histories (not counting chance nodes).
Abrupt Dark Hex is a variant of the classical board game Hex (Bakst & Gardner, 1962).
In Hex, two players take turns placing stones onto a board.
One player’s goal is to create a path of its stones connecting the east end of the board with the west end, while the other player’s goal is to do the same with the north end and south end.
Dark
Hex is a variant in which players cannot see where their opponents are placing stones.
Abrupt Dark
Hex is a variant of Dark Hex in which placing a stone in an occupied position results in a loss of turn.
The prefix n x n describes the size of the board.
2x
Abrupt Dark Hex has 471 non-terminal histories.
3x
Abrupt Dark Hex has too many non-terminal histories to enumerate on our hardware.
Liar’s Dice (Ferguson & Ferguson, 1991) is a dice game in which players privately roll dice and place bids based on the observed outcomes, similarly to poker games.
The prefix n -sided means that the players play with 4-sided dice.
4-Sided Liar’s Dice has 8176 non-terminal histories (not counting chance nodes).
Leduc Poker (Southey et al., 2005) is a small poker game with three card values (J, Q,
K), each of which have two instances in the deck.
It has 9300 non-terminal histories non-terminal histories (not counting chance nodes).
For our single-agent deep RL experiments, we use three Atari games (Bellemare et al., 2013) and three Mujoco games (Todorov et al., 2012).
We selected these games because Huang et al.
(2022) used them to benchmark an open source implementation of PPO.
G QRE E XPERIMENTS
[Table]
[Figure]
Iterations
K L
D i v e r g e n c e
MMDPUOMWU Game A Game B Game C Game D ↵ = 0 .<latexit sha1_base64="h5GWBU8xdbIouEj0X/2rS5y50hA=">AAAB9HicbVBNSwMxEJ2tX7V+VT16CRbBU9kVi16EohePFewHtEuZTbNtaDa7JtlCKf0dXjwo4tUf481/Y9ruQVsfDPN4b4ZMXpAIro3rfju5tfWNza38dmFnd2//oHh41NBxqiir01jEqhWgZoJLVjfcCNZKFMMoEKwZDO9mfnPElOaxfDTjhPkR9iUPOUVjJb+DIhkguSFu2a10iyXb5iCrxMtICTLUusWvTi+macSkoQK1bntuYvwJKsOpYNNCJ9UsQTrEPmtbKjFi2p/Mj56SM6v0SBgrW9KQufp7Y4KR1uMosJMRmoFe9mbif147NeG1P+EySQ2TdPFQmApiYjJLgPS4YtSIsSVIFbe3EjpAhdTYnAo2BG/5y6ukcVH2LsuVh8tS9TaLIw8ncArn4MEVVOEealAHCk/wDK/w5oycF+fd+ViM5pxs5xj+wPn8AYotkKY=</latexit> ↵ = 0 .<latexit
sha1_base64="FkJ7uu6hz6zvjIpJemKY/7tGCq0=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBU0ikoheh6MVjBfsBTSiT7aZdutksuxuhhP4NLx4U8eqf8ea/cdvmoK0PBh7vzTAzL5KcaeN5305pbX1jc6u8XdnZ3ds/qB4etXWaKUJbJOWp6kagKWeCtgwznHalopBEnHai8d3M7zxRpVkqHs1E0jCBoWAxI2CsFATA5QjwDfZcv1+tea43B14lfkFqqECzX/0KBinJEioM4aB1z/ekCXNQhhFOp5Ug01QCGcOQ9iwVkFAd5vObp/jMKgMcp8qWMHiu/p7IIdF6kkS2MwEz0sveTPzP62Umvg5zJmRmqCCLRXHGsUnxLAA8YIoSwyeWAFHM3orJCBQQY2Oq2BD85ZdXSfvC9evu5UO91rgt4iijE3SKzpGPrlAD3aMmaiGCJHpGr+jNyZwX5935WLSWnGLmGP2B8/kDEvaQaA==</latexit>
[Table]
.<latexit sha1_base64="XGsaTepsGGlauStOKtuLYcKwWv8=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBU0hKRS9C0YvHCrYWmlA22027dLNZdjdCCf0bXjwo4tU/481/47bNQVsfDDzem2FmXiQ508bzvp3S2vrG5lZ5u7Kzu7d/UD086ug0U4S2ScpT1Y2wppwJ2jbMcNqViuIk4vQxGt/O/McnqjRLxYOZSBomeChYzAg2VgoCzOUIo2vkufV+tea53hxolfgFqUGBVr/6FQxSkiVUGMKx1j3fkybMsTKMcDqtBJmmEpMxHtKepQInVIf5/OYpOrPKAMWpsiUMmqu/J3KcaD1JItuZYDPSy95M/M/rZSa+CnMmZGaoIItFccaRSdEsADRgihLDJ5Zgopi9FZERVpgYG1PFhuAvv7xKOnXXb7gX941a86aIowwncArn4MMlNOEOWtAGAhKe4RXenMx5cd6dj0VrySlmjuEPnM8fFHqQaQ==</latexit>
[Table]
.<latexit sha1_base64="27puadG0Hs7oU6pa3zWHWvrRUXs=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBU0jEoheh6MVjBfsBTSiT7aZdutksuxuhhP4NLx4U8eqf8ea/cdvmoK0PBh7vzTAzL5KcaeN5305pbX1jc6u8XdnZ3ds/qB4etXWaKUJbJOWp6kagKWeCtgwznHalopBEnHai8d3M7zxRpVkqHs1E0jCBoWAxI2CsFATA5QjwDfbcer9a81xvDrxK/ILUUIFmv/oVDFKSJVQYwkHrnu9JE+agDCOcTitBpqkEMoYh7VkqIKE6zOc3T/GZVQY4TpUtYfBc/T2RQ6L1JIlsZwJmpJe9mfif18tMfB3mTMjMUEEWi+KMY5PiWQB4wBQlhk8sAaKYvRWTESggxsZUsSH4yy+vkvaF61+69YfLWuO2iKOMTtApOkc+ukINdI+aqIUIkugZvaI3J3NenHfnY9FacoqZY/QHzucPGQaQbA==</latexit>
We perform various QRE experiments under full feedback for Diplomacy stage games.
Full feedback means that each player outputs a fully specified policy and receives its exact Q-values (given both players’ policies) as feedback.
Both players then perform the update
[Equation]
For our experiments, we set η “ α (for each α ) for MMD, which is the maximal value that retains a linear convergence guarantee for normal-form games with a max payoff magnitude of one.
For PU and OMWU (Cen et al., 2021), we also used the maximal values that guarantee linear convergence.
We solved for the QRE for each game using Ling et al.’s
Newton’s method approach.
We show iterations on the x -axis and KL p solution , iterate q on the y -axis.
We count each query to the oracle as an iterate, meaning that OMWU uses two iterates for every update (contrasting MMD and PU, which only use one).
The results of the experiment, found in Figure 5, show that all three algorithms converge linearly with faster rates for larger values of alpha, as is guaranteed by theory.
We find that, for our Diplomacy games, MMD converges faster than PU and OMWU.
However, we found that all three algorithms also exhibited faster convergence with larger than theoretically allowed stepsizes.
G. B LACK B OX QRE C ONVERGENCE D IPLOMACY
Our second set of experiments examine convergence to QREs for our Diplomacy stage games with black box feedback.
In this context, black box feedback means that each player i outputs an action A i sampled from its current policy and that player i receives R p¨ ,
A i , A
´ i q (but not A ´ i ) as feedback.
One way to approach such a setting is to construct an unbiased estimate of the exact Q-values.
Letting r be the observed reward
[Equation]
otherwise is such an estimate.
To see that this is true, observe
[Equation]
π t p a i q ¨ R p¨
[Equation]
a i , A ´ i q π t p a i q ` ÿ a 1 i ‰ a i π t p a 1 i q ¨fi fl “ E A
´ i „
π t „ π t p a i q ¨ R p¨ , a i , A
´ i q π t p a i q ȷ
[Equation]
E
[Equation]
i
[Equation]
p¨
[Equation]
a i
[Equation]
A
[Equation]
i
[Equation]
In Figure 6, we show results for each of MMD, PU and OMWU, with the exact Q-values q t replaced by the unbiased estimates ˆ q t .
For each algorithm, the stepsize at iteration t was set to be equal to the maximal step size for which there exists an exponential convergence guarantee divided by 10 ?
t .
In other words,
[Equation]
Each line is an average over 30 runs.
The bands depict estimates of 95% confidence intervals computed using bootstrapping.
Although none of the algorithms possess existing black box convergence guarantees, we observe that they all exhibit convergent behavior empirically.
In terms of convergence speed, we observe that MMD compares favorably to PU and OMWU for α P t 0 .
2 u ; however, for α “ 0 . 5 , OMWU performed the best, with the exception of game
D.
It is likely that all algorithms could achieve better performance, as we did not perform much hyperparameter tuning.
[Equation]
<latexit sha1_base64="h5GWBU8xdbIouEj0X/2rS5y50hA=">AAAB9HicbVBNSwMxEJ2tX7V+VT16CRbBU9kVi16EohePFewHtEuZTbNtaDa7JtlCKf0dXjwo4tUf481/Y9ruQVsfDPN4b4ZMXpAIro3rfju5tfWNza38dmFnd
[Table]
/kDEvaQaA==</latexit>
[Table]
=
[Table]
.<latexit
sha1_base64="XGsaTepsGGlauStOKtuLYcKwWv8=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBU0hKRS9C0YvHCrYWmlA22027dLNZdjdCCf0bXjwo4tU/481/47bNQVsfDDzem2FmXiQ508bzvp3S2vrG5lZ5u7Kzu7d/UD086ug0U4S2ScpT1Y2wppwJ2jbMcNqViuIk4vQxGt/O/McnqjRLxYOZSBomeChYzAg2VgoCzOUIo2vkufV+tea53hxolfgFqUGBVr/6FQxSkiVUGMKx1j3fkybMsTKMcDqtBJmmEpMxHtKepQInVIf5/OYpOrPKAMWpsiUMmqu/J3KcaD1JItuZYDPSy95M/M/rZSa+CnMmZGaoIItFccaRSdEsADRgihLDJ5Zgopi9FZERVpgYG1PFhuAvv7xKOnXXb7gX941a86aIowwncArn4MMlNOEOWtAGAhKe4RXenMx5cd6dj0VrySlmjuEPnM8fFHqQaQ==</latexit>
[Table]
<latexit sha1_
[Table]
="27puadG0Hs7oU6pa3zWHWvrRUXs=">AAAB83icbVBNS8NAEN3Ur1q/
[Table]
/cdvmoK0PBh7vzTAzL5KcaeN5305pbX1jc6u8XdnZ3ds/qB4etXWaKUJbJOWp6kagKWeCtgwznHalopBEnHai8d3M7zxRpVkqHs1E0jCBoWAxI2CsFATA5QjwDfbcer9a81xvDrxK/ILUUIFmv/oVDFKSJVQYwkHrnu9JE+agDCOcTitBpqkEMoYh7VkqIKE6zOc3T
[Table]
K L
D i v e r g e n ce
MMDPUOMWU
[Table]
/gD
[Figure]
MHP4aO3g
[Figure]
</
[Figure]
>
[Figure]
<
[Figure]
sha1_base
[Figure]
="
[Figure]
AAAB7nicbVDLSgNBEOyNrxhfUY9eBoPgxbCrET0GvXiMYB6QrGF20kmGzM4uM7NCWPIRXjwo4tXv8ebfOEn2oIkFDUVVN91dQSy4Nq777eRWVtfWN/Kbha3tnd294v5BQ0eJYlhnkYhUK6AaBZdYN9wIbMUKaRgIbAaj26nffEKleSQfzDhGP6QDyfucUWOlpuc
[Figure]
+
[Figure]
/kx5XyIwYW0KZ4vZWwoZUUWZsQgUbgrf48jJpnJe9SvnyvlKq3mRx5OEIjuEUPLiCKtxBDerAYATP8ApvTuy8OO/
[Figure]
/AHzucPQQuO3w==</latexit
[Figure]
We also investigate the performance of other methods for estimating Q-values for the black box setting.
One such method uses an unbiased baseline to reduce variance (Schmid et al., 2019; Davis et al., 2020).
The premise of this approach is the idea that any quantity that is zero in expectation can be subtracted from an unbiased Q-value estimate without introducing bias.
As a result, if the quantity is correlated with the estimator, subtracting it from the estimate can reduce variance “for free”.
We call this quantity a baseline.
For our baseline, we used
[Equation]
By a similar argument as above, this quantity is zero in expectation
[Equation]
Also, if ˜ q is close to q , our baseline will be correlated with ˆ q .
Thus, it satisfies our desired criteria.
For ˜ q , we used a running estimate of the reward observed after selecting action a i .
Specifically, every time action a
i was selected, we updated
[Equation]
We used ˜ η “ 1 { 2 , inspired by Schmid et al.
We also investigated the use of biased Q-value estimates, as this is the setting that corresponds with function approximation.
For this approach, we plugged in ˜ q , as computed above, instead of the exact
Q-values q
[Table]
[Figure]
Iterations
K L
D i v e r g e n ce
[Figure]
[Table]
[Figure]
[Table]
/Ox7w152Qzh/AHzucPQQuO3w==</latexit>
[Table]
<latexit sha1_base64="0VXXdbYgYISBa9T50LdFhRw/L0o=">AAAB63icbVBNS8NAEJ3Ur1q/qh69LBbBU0mkoseiF48V7Ae0sWy2m3bp7ibsboQS+he8eFDEq3/Im//GTZqDtj4YeLw3w8y8IOZMG9f9dkpr6xubW+Xtys7u3v5B9fCoo6NEEdomEY9UL8CaciZp2zDDaS9WFIuA024wvc387hNVmkXywcxi6gs8lixkBJtM8txHd1ituXU3B1olXkFqUKA1rH4NRhFJBJWGcKx133Nj46dYGUY4nVcGiaYxJlM8pn1LJRZU+2l
[Table]
6xydWWWEwkjZkgbl6u
[Table]
[Table] [Table]
/fK
[Table] [Table]
OcF6cd
[Table]
dj0Vpyiplj
[Table]
</
[Table]
<
[Table]
sha1_
[Table]
="
SJUNDXPj9V3b23Lx4Q4/
[Table]
=
[Table] [Table]
[Table] [Table]
[Table]
/QHzucPGQaQbA==</latexit>
[Figure]
<latexit sha1_base64="YUNPuFgwU0RHQIQIaz9LXkLvpr0=">AAAB7nicbVBNS8NAEJ34WetX1aOXxSJ4sSRS0WPRi8cK9gPaWDbbSbt0swm7G6GE/ggvHhTx6u/x5r9x2+agrQ8GHu/NMDMvSATXxnW/nZXVtfWNzcJWcXtnd2+/dHDY1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hoduq3nlBpHssHM07Qj+hA8pAzaqzU8tzH7Lw66ZXKbsWdgSwTLydlyFHvlb66/ZilEUrDBNW647mJ8TOqDGcCJ8VuqjGhbEQH2LFU0gi1n83OnZBTq/RJGCtb0pCZ+nsio5HW4yiwnRE1Q73oTcX/vE5qwms/4zJJDUo2XxSmgpiYTH8nfa6QGTG2hDLF7a2EDamizNiEijYEb/HlZdK8qHjVyuV9tVy7yeMowDGcwBl4cAU1uIM6NIDBCJ7hFd6cxHlx3p2PeeuKk88cwR84nz9CkI7g</latexit>
We show the results of the experiment if Figure 7.
The column shows the temperature for the QRE.
The y-axis shows the KL divergence to the corresponding logit-QRE.
The x-axis shows the number of iterations.
For each algorithm, the step size at iteration t was set to be equal to the maximal step size for which there exists an exponential convergence guarantee divided by
Overall, we find that both using unbiased baselines and biased Q-value estimates appears to improve convergence speed
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
D i v e r g e n ce
o
f
D il a t e d
E n t r o p y
[Table]
/qB4etXWaKUJbJOWp6kagKWeCtgwznHalopBEnHai8d3M7zxRpVkqHs1E0jCBoWAxI2CsFATA5QjwDfbcer9a81xvDrxK/ILUUIFmv/oVDFKSJVQYwkHrnu9JE+agDCOcTitBpqkEMoYh7VkqIKE6zOc3T/GZVQY4TpUtYfBc/T2RQ6L1JIlsZwJmpJe9mfif18tMfB3mTMjMUEEWi+KMY5PiWQB4wBQlhk8sAaKYvRWTESggxsZUsSH4yy+vkvaF61+69YfLWuO2iKOMTtApOkc+ukINdI+aqIUIkugZvaI3J3NenHfnY9FacoqZY/QHzucPGQaQbA==</latexit>
[Figure]
2x
Abrupt Dark Hex
[Figure]
<latexit sha1_base64="h5GWBU8xdbIouEj0X/2rS5y50hA=">AAAB9HicbVBNSwMxEJ2tX7V+VT16CRbBU9kVi16EohePFewHtEuZTbNtaDa7JtlCKf0dXjwo4tUf481/Y9ruQVsfDPN4b4ZMXpAIro3rfju5tfWNza38dmFnd2//oHh41NBxqiir01jEqhWgZoJLVjfcCNZKFMMoEKwZDO9mfnPElOaxfDTjhPkR9iUPOUVjJb+DIhkguSFu2a10iyXb5iCrxMtICTLUusWvTi+macSkoQK1bntuYvwJKsOpYNNCJ9UsQTrEPmtbKjFi2p/Mj56SM6v0SBgrW9KQufp7Y4KR1uMosJMRmoFe9mbif147NeG1P+EySQ2TdPFQmApiYjJLgPS4YtSIsSVIFbe3EjpAhdTYnAo2BG/5y6ukcVH2LsuVh8tS9TaLIw8ncArn4MEVVOEealAHCk/wDK/w5oycF+fd+ViM5pxs5xj+wPn8AYotkKY=</latexit> ↵ =0 .<latexit
sha1_base64="FkJ7uu6hz6zvjIpJemKY/7tGCq0=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBU0ikoheh6MVjBfsBTSiT7aZdutksuxuhhP4NLx4U8eqf8ea/cdvmoK0PBh7vzTAzL5KcaeN5305pbX1jc6u8XdnZ3ds/qB4etXWaKUJbJOWp6kagKWeCtgwznHalopBEnHai8d3M7zxRpVkqHs1E0jCBoWAxI2CsFATA5QjwDfZcv1+tea43B14lfkFqqECzX/0KBinJEioM4aB1z/ekCXNQhhFOp5Ug01QCGcOQ9iwVkFAd5vObp/jMKgMcp8qWMHiu/p7IIdF6kkS2MwEz0sveTPzP62Umvg5zJmRmqCCLRXHGsUnxLAA8YIoSwyeWAFHM3orJCBQQY2Oq2BD85ZdXSfvC9evu5UO91rgt4iijE3SKzpGPrlAD3aMmaiGCJHpGr+jNyZwX5935WLSWnGLmGP2B8/kDEvaQaA==</latexit> ↵ =0 .<latexit
sha1_base64="XGsaTepsGGlauStOKtuLYcKwWv8=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBU0hKRS9C0YvHCrYWmlA22027dLNZdjdCCf0bXjwo4tU
[Table]
[Figure]
[Table]
+gX6OM=">AAAB73icbVDLSgNBEOyNrxhfUY9eBoPgxbArET0GvXiMYB6QrGF2MkmGzM6uM71CWPITXjwo4tXf8ebfOEn2oIkFDUVVN91dQSyFQdf9dnIrq2vrG/nNwtb2zu5ecf+gYaJEM15nkYx0K6CGS6F4HQVK3oo1p2EgeTMY3Uz95hPXRkTqHscx90M6UKIvGEUrtTz3IT3z3Em3WHLL7gxkmXgZKUGGWrf41elFLAm5QiapMW3PjdFPqUbBJJ8UOonhMWUjOuBtSxUNufHT2b0TcmKVHulH2pZCMlN/T6Q0NGYcBrYzpDg0i95U/M9rJ9i/8lOh4gS5YvNF/UQSjMj0edITmjOUY0so08LeStiQasrQRlSwIXiLLy+TxnnZq5Qv7iql6nUWRx6O4BhOwYNLqMIt1KAODCQ8wyu8OY/Oi/PufMxbc042cwh/4Hz+AK15jxc=</latexit>
Kuhn
[Figure]
latexit sha1_base64=
[Figure]
/2rS5y50hA
[Figure]
>AAAB9HicbVBNSwMxEJ2tX7V+VT16CRbBU9kVi16EohePFewHtEuZTbNtaDa7JtlCKf0dXjwo4tUf
[Figure]
[Table]
/wDK/w5oycF+fd+ViM5pxs5xj+wPn8AYotkKY=</latexit> ↵ =0 .<latexit
sha1_base64="FkJ7uu6hz6zvjIpJemKY/7tGCq0=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBU0ikoheh6MVjBfsBTSiT7aZdutksuxuhhP4NLx4U8eqf8ea/cdvmoK0PBh7vzTAzL5KcaeN5305pbX1jc6u8XdnZ3ds/qB4etXWaKUJbJOWp6kagKWeCtgwznHalopBEnHai8d3M7zxRpVkqHs1E0jCBoWAxI2CsFATA5QjwDfZcv1+tea43B14lfkFqqECzX/0KBinJEioM4aB1z/ekCXNQhhFOp5Ug01QCGcOQ9iwVkFAd5vObp/jMKgMcp8qWMHiu/p7IIdF6kkS2MwEz0sveTPzP62Umvg5zJmRmqCCLRXHGsUnxLAA8YIoSwyeWAFHM3orJCBQQY2Oq2BD85ZdXSfvC9evu5UO91rgt4iijE3SKzpGPrlAD3aMmaiGCJHpGr+jNyZwX5935WLSWnGLmGP2B8/kDEvaQaA==</latexit> ↵ =0 .<latexit
sha1_base64="XGsaTepsGGlauStOKtuLYcKwWv8=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBU0hKRS9C0YvHCrYWmlA22027dLNZdjdCCf0bXjwo4tU/481/47bNQVsfDDzem2FmXiQ508bzvp3S2vrG5lZ5u7Kzu7d/UD086ug0U4S2ScpT1Y2wppwJ2jbMcNqViuIk4vQxGt/O/McnqjRLxYOZSBomeChYzAg2VgoCzOUIo2vkufV+tea53hxolfgFqUGBVr/6FQxSkiVUGMKx1j3fkybMsTKMcDqtBJmmEpMxHtKepQInVIf5/OYpOrPKAMWpsiUMmqu/J3KcaD1JItuZYDPSy95M/M/rZSa+CnMmZGaoIItFccaRSdEsADRgihLDJ5Zgopi9FZERVpgYG1PFhuAvv7xKOnXXb7gX941a86aIowwncArn4MMlNOEOWtAGAhKe4RXenMx5cd6dj0VrySlmjuEPnM8fFHqQaQ==</latexit> ↵ =0 .<latexit sha1_base64="27puadG0Hs7oU6pa3zWHWvrRUXs=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBU0jEoheh6MVjBfsBTSiT7aZdutksuxuhhP4NLx4U8eqf8ea/cdvmoK0PBh7vzTAzL5KcaeN5305pbX1jc6u8XdnZ3ds/qB4etXWaKUJbJOWp6kagKWeCtgwznHalopBEnHai8d3M7zxRpVkqHs1E0jCBoWAxI2CsFATA5QjwDfbcer9a81xvDrxK/ILUUIFmv/oVDFKSJVQYwkHrnu9JE+agDCOcTitBpqkEMoYh7VkqIKE6zOc3T/GZVQY4TpUtYfBc/T2RQ6L1JIlsZwJmpJe9mfif18tMfB3mTMjMUEEWi+KMY5PiWQB4wBQlhk8sAaKYvRWTESggxsZUsSH4yy+vkvaF61+69YfLWuO2iKOMTtApOkc+ukINdI+aqIUIkugZvaI3J3NenHfnY9FacoqZY/QHzucPGQaQbA==</latexit>
[Figure]
We perform several experiments for solving reduced normal-form logit QREs by using MMD over the sequence form with dilated entropy.
We use the descent-ascent updates
[Equation]
The method is full feedback since ∇ x t f p x t , y t q “ Ay t and ∇ y t f p x t , y t q “ A J x t , where A is the sequence form payoff matrix.
Note in the normal form setting ´
Ay t and A J x t are the Q-values for both players and the algorithm is the same as described in Section G.1.
We set the stepsize to be η “ α { p max ij | A ij |q 2 , the largest possible allowed from Theorem 3.4.
For more details on the sequence form algorithm, see Section C.3.
For Kuhn Poker and 2x
Abrubt Dark Hex, we used Gambit (McKelvey, Richard D., McLennan,
Andrew M., and Turocy, Theodore L.; Turocy, 2005) to compute the reduced normal-form QRE.
We check the convergence of MMD by plotting the sum of Bregman divergences with respect to dilated entropy B ψ p z ˚ ; z t q “ B ψ 1 p x ˚ ; x t q ` B ψ 1 p y ˚ ; y t q , with respect to the solution z ˚ “ p x ˚ , y ˚ q .
As predicted by Theorem 3.4 we observer linear convergence with faster convergence for larger values of α .
For 4-Sided Liar’s Dice and Leduc Poker, the games were too large for Gambit (McKelvey, Richard
D., McLennan, Andrew M., and Turocy, Theodore L.; Turocy, 2005) to compute the reduced normalform QRE on our hardware.
Therefore, we check the convergence of MMD by plotting the saddle point gap ξ p x t , y t q of the min max problem given by Ling et al.
[Equation]
Theorem 3.4 guarantees that the gap will converge to zero.
Note the gap is zero if and only if at the solution and, by Proposition D.6, the gap is also guaranteed to converge linearly.
In both 4-Sided
Liar’s Dice and Leduc Poker we observe linear convergence of the gap, with faster convergence for larger values of α .
Additionally, due to the O p log p t qq regret bound from Duchi et al.
(2010), we have that the gap is guaranteed to converge at a rate of O
[Equation]
for the average iterates of both players.
G. F ULL F EEDBACK AQRE C ONVERGENCE EFG S
Next, we investigate whether MMD can be made to converge to AQREs in extensive-form games.
For these experiments we applied MMD in behavioral form, as described in Section E. Specifically, we computed q t p h
i q for each player
i and each information state h
i .
Then, we applied the update rule
[Equation]
for each player i and information state h i .
For each setting, we used
[Equation]
M ea n K L
D i v e r g e n ce 10 <latexit
sha1_base64="SJUNDXPj9V3b23Lx4Q4/42OIzrE=">AAAB7nicbVBNS8NAEJ34WetX1aOXxSJ4sSRS0WPRi8cK9gPaWDbbSbt0swm7G6GE/ggvHhTx6u/x5r9x
[Table]
/cZLsQRMLGoqqbrq7glhwbVz328mtrK6tb+Q3C1vbO7t7xf2Dho4SxbDOIhGpVkA1Ci6xbrgR2IoV0jAQ2AxGt1O/+YRK80g+mHGMfkgHkvc5o8ZKTc99TM8uJ91iyS27M5Bl4mWkBBlq3eJXpxexJERpmKBatz03Nn5KleFM4KTQSTTGlI3oANuWShqi9tPZuRNyYpUe6UfKljRkpv6eSGmo9TgMbGdIzVAvelPxP6+dmP61n3IZJwYlmy/qJ4KYiEx/Jz2ukBkxtoQyxe2thA2poszYhAo2BG/x5WXSOC97lfLFfaVUvcniyMMRHMMpeHAFVbiDGtSBwQie4RXenNh5cd6dj3lrzslmDuEPnM8fRZqO4g==</latexit>
[Figure]
<latexit sha1_base64="0VXXdbYgYISBa9T50LdFhRw/L0o=">AAAB63icbVBNS8NAEJ3Ur1q/qh69LBbBU0mkoseiF48V7Ae0sWy2m3bp7ibsboQS+he8eFDEq3/Im//GTZqDtj4YeLw3w8y8IOZMG9f9dkpr6xubW+Xtys7u3v5B9fCoo6NEEdomEY9UL8CaciZp2zDDaS9WFIuA024wvc387hNVmkXywcxi6gs8lixkBJtM8txHd1ituXU3B1olXkFqUKA1rH4NRhFJBJWGcKx133Nj46dYGUY4nVcGiaYxJlM8pn1LJRZU+2l+6xydWWWEwkjZkgbl6u+JFAutZyKwnQKbiV72MvE/r5+Y8NpPmYwTQyVZLAoTjkyEssfRiClKDJ9Zgoli9lZEJlhhYmw8FRuCt/zyKulc1L1G/fK+UWveFHGU4QRO4Rw8uIIm3EEL2kBgAs/wCm+OcF6cd+dj0Vpyiplj+APn8wcNTo2Z</latexit>
[Figure]
Iterations ↵ = 0 .
[Figure]
<latexit sha1_base64="h5GWBU8xdbIouEj0X
[Table]
/ILUUIFmv/oVDFKSJVQYwkHrnu9JE+agDCOcTitBpqkEMoYh7VkqIKE6zOc3T/GZVQY4TpUtYfBc/T2RQ6L1JIlsZwJmpJe9mfif18tMfB3mTMjMUEEWi+KMY5PiWQB4wBQlhk8sAaKYvRWTESggxsZUsSH4yy+vkvaF61+69YfLWuO2iKOMTtApOkc+ukINdI+aqIUIkugZvaI3J3NenHfnY9FacoqZY/QHzucPGQaQbA==</latexit>
[Figure]
We show the results in Figure 10.
We measure convergence against solutions computed using Gambit (McKelvey, Richard D., McLennan, Andrew M., and Turocy, Theodore L.; Turocy, 2010).
Despite a lack of proven convergence guarantees, we observe that MMD converges to the AQRE in each game, for each temperature.
While the convergence is not monotonic, it is roughly linear over large time scales.
H E XPLOITABILITY E XPERIMENTS
Next, we investigate the convergence of MMD as a Nash equilibrium solver.
To induce convergence, in most of our experiments, we anneal the temperature of the regularization over time.
H.
F ULL F EEDBACK N ASH C ONVERGENCE D IPLOMACY
In our full feedback Nash convergence Diplomacy experiments, we used
[Equation]
We show the results of the experiment in Figure 11.
Over short iteration horizons, we observe that
CFR tends to outperform MMD.
However, for longer horizons, we find that MMD tends to catch up with CFR.
In game D, the qualitatively different behavior is likely to due the fact that the Nash equilibrium is a pure strategy, unlike the Nash equilibria of the first three games, which are mixed.
Iterations b ili
<latexit sha1_base64="1RZGl+duRB0AcksLpWnWYdkSmBY=">AAAB63icbVBNS8NAEJ3Ur1q/qh69LBbBU0mkoseiF48V7Ae0sWy2m3bp7ibsboQS+he8eFDEq3/Im//GTZqDtj4YeLw3w8y8IOZMG9f9dkpr6xubW+Xtys7u3v5B9fCoo6NEEdomEY9UL8CaciZp2zDDaS9WFIuA024wvc387hNVmkXywcxi6gs8lixkBJtM8txHb1ituXU3B1olXkFqUKA1rH4NRhFJBJWGcKx133Nj46dYGUY4nVcGiaYxJlM8pn1LJRZU+2l+6xydWWWEwkjZkgbl6u+JFAutZyKwnQKbiV72MvE/r5+Y8NpPmYwTQyVZLAoTjkyEssfRiClKDJ9Zgoli9lZEJlhhYmw8FRuCt/zyKulc1L1G/fK+UWveFHGU4QRO4Rw8uIIm3EEL2kBgAs/wCm+OcF6cd+dj0Vpyiplj
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
<latexit sha1_base64="SJUNDXPj9V3b23Lx4Q4/42OIzrE=">AAAB7nicbVBNS8NAEJ34WetX1aOXxSJ4sSRS0WPRi8cK9gPaWDbbSbt0swm7G6GE
[Table]
/latexit>
[Figure]
H. B LACK B OX N ASH C ONVERGENCE D IPLOMACY
For our black box Nash convergence experiments, we compare against the “opponent on-policy” variant of Monte Carlo CFR (Lanctot et al., 2009).
In this variant, the two players alternate between an updating player and an on-policy player.
The updating player plays off-policy according to a policy that provides sufficiently large support to each action (in our Diplomacy experiments we used a uniform policy).
The advantage to this setup is that it guarantees that the updating player will receive bounded gradients, which is necessary for Monte Carlo CFR’s convergence proof.
In contrast, we show results for an on-policy Monte Carlo variant of MMD, despite the fact that this causes unbounded gradients.
This is not a fair comparison in the sense that the same “opponent on-policy” setup is equally applicable to MMD and would keep the gradients bounded, whereas the “on-policy” version of Monte Carlo CFR does not converge.
We made this decision because the on-policy Monte
Carlo variant of MMD is simpler and more elegant.
Nevertheless, we believe that the “opponent on-policy” version of MMD remains an interesting direction for future, and would very possibly yield faster convergence.
We again investigated three ways of estimating Q-values.
For our unbiased estimator with no baseline we used
[Equation]
for games A, B, and C, and
[Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
for
[Equation] [Equation]
for
D
We show results in Figure with averages and estimates of 95% computed from
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
</
[Table]
[Figure]
[Table]
+hA8pAzaqzU8tzH7Nyb9Eplt+LOQJaJl5My5Kj3Sl/dfszSCKVhgmrd8dzE+BlVhjOBk2I31ZhQNqID7FgqaYTaz2bnTsipVfokjJUtachM/T2R0UjrcRTYzoiaoV70puJ/Xic14bWfcZmkBiWbLwpTQUxMpr+TPlfIjBhbQpni9lbChlRRZmxCRRuCt/jyMmleVLxq5fK+Wq7d5HEU4BhO4Aw8uIIa3EEdGsBgBM/wCm9O4rw4787HvHXFyWeO4A+czx8+AY7d</latexit>
[Figure]
For MMD, we observe that biased Q-value estimates generally perform best, followed by an unbiased estimate with baseline, followed by an unbiased estimate without baseline, except on game D, where the unbiased baseline performs similarly to biased Q-value estimates.
We also find that CFR tends to follow this trend, though the difference between biased Q-value estimates and an unbiased baseline is less pronounced, except on game D, where the unbiased baseline performs poorly.
Between MMD and CFR, CFR tends to perform better on an estimator-to-estimator basis in games A, B and C, though MMD is relatively competitive with CFR under biased Q-value estimates.
For game D, we observe that this comparison is more favorable for MMD than the other games.
H. F ULL F EEDBACK N ASH C ONVERGENCE EFG S
For our full feedback Nash convergence EFG experiments, we examined two variants of MMD.
The first, which we call unweighted MMD, corresponds with the version tested in the AQRE experiments
[Equation]
The second, which we call weighted MMD, uses
[Equation]
In other words, it weights the stepsize of the update by the probability of reaching that information state under the current policy.
We test this variant because it corresponds with a “determinized” version of black box sampling for temporally extended settings.
For unweighted MMD, we used
[Equation]
for Kuhn Poker,
[Equation] [Equation]
[Table] [Table]
[Equation]
[Table]
[Equation] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
/AHzucPQQuO3w==</latexit>
[Figure]
We show the results of our experiments in Figure 13.
We find that both weighted MMD and unweighted MMD exhibit convergent behavior.
Furthermore, they converge at rates comparable with
CFR on average across the games.
H. B LACK B OX N ASH C ONVERGENCE EFG S
For our black box Nash convergence EFG experiments, we used the Monte Carlo CFR implementation in OpenSpiel (Lanctot et al., 2019), which uses an update policy with a 0 .
4 weight on the current policy and a 0 .
6 weight on the uniform policy.
For MMD, we used the sampling version of weighted
MMD, meaning that the information states touched during the trajectory are updated with the full stepsize, while information not touched during the trajectory are not updated.
For Kuhn Poker, we used
[Equation]
For 2x2 Abrupt Dark Hex, we used
[Equation]
For 4-Sided Liar’s Dice, we used
[Equation]
we used [Figure]
[Equation]
[Figure]
Noting again that the caveats about comparing on-policy MMD to opponent on-policy Monte Carlo
CFR also apply here, we present the results in Figure 13.
Results are averaged across 30 runs and shown with 95% confidence intervals estimated from bootstrapping.
As in the normal-form experiments, we find that Monte Carlo CFR generally outperforms MMD for unbiased gradient estimates with no baseline.
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
+ySZIWy9C948aCIV/+QN/+N2XYP2vpg4PHeDDPzgpgzbVz32ylsbG5t7xR3S3v7B4dH5eOTjo4SRWibRDxSvQBrypmkbcMMp71YUSwCTrvB9Dbzu09UaRbJBzOLqS/wWLKQEWwyyXMf68Nyxa26C6B14uWkAjlaw/LXYBSRRFBpCMda9z03Nn6KlWGE03lpkGgaYzLFY9q3VGJBtZ8ubp2jC6uMUBgpW9Kghfp7IsVC65kIbKfAZqJXvUz8z+snJrz2UybjxFBJlovChCMToexxNGKKEsNnlmCimL0VkQlWmBgbT8mG4K2+vE46V1WvVq3f1yrNmzyOIpzBOVyCBw1owh20oA0EJvAMr/DmCOfFeXc+lq0FJ585hT9wPn8AFOKNng==</latexit>
[Figure]
<latexit sha1_base64="7vHrL539BumllXTvcFV4ghbJm1E=">AAAB63icbVBNSwMxEJ2tX7V+VT16CRbBU9mVVj0WvXisYD+gXUs2zbahSXZJskJZ+he8eFDEq3/Im//GbLsHbX0w8Hhvhpl5QcyZNq777RTW1jc2t4rbpZ3dvf2D8uFRW0eJIrRFIh6pboA15UzSlmGG026sKBYBp51gcpv5nSeqNIvkg5nG1Bd4JFnICDaZ5LmPl4Nyxa26c6BV4uWkAjmag/JXfxiRRFBpCMda9zw3Nn6KlWGE01mpn2gaYzLBI9qzVGJBtZ/Ob52hM6sMURgpW9Kgufp7IsVC66kIbKfAZqyXvUz8z+slJrz2UybjxFBJFovChCMToexxNGSKEsOnlmCimL0VkTFWmBgbT8mG4C2/vEraF1WvVq3f1yqNmzyOIpzAKZyDB1fQgDtoQgsIjOEZXuHNEc6L8+58LFoLTj5zDH/gfP4AFmaNnw==</latexit>
[Figure]
latexit sha1_base64="0VXXdbYgYISBa9T50LdFhRw/L0o=">AAAB63icbVBNS8NAEJ3Ur1q/qh69LBbBU0mkoseiF48V7Ae0sWy2m3bp7ibsboQS+he8eFDEq3/Im//GTZqDtj4YeLw3w8y8IOZMG9f9dkpr6xubW+Xtys7u3v5B9fCoo6NEEdomEY9UL8CaciZp2zDDaS9WFIuA024wvc387hNVmkXywcxi6gs8lixkBJtM8txHd1ituXU3B1olXkFqUKA1rH4NRhFJBJWGcKx133Nj46dYGUY4nVcGiaYxJlM8pn1LJRZU+2l+6xydWWWEwkjZkgbl6u+JFAutZyKwnQKbiV72MvE/r5+Y8NpPmYwTQyVZLAoTjkyEssfRiClKDJ9Zgoli9lZEJlhhYmw8FRuCt/zyKulc1L1G/fK+UWveFHGU4QRO4Rw8uIIm3EEL2kBgAs/wCm+OcF6cd+dj0Vpyiplj+APn8wcNTo2Z</latexit> H. [Figure]
M OVING M AGNET
Next, we investigate using a moving magnet, rather than an annealing temperature, to induce convergence to a Nash equilibrium.
In the moving magnet setup, updates take the form
[Equation]
where ρ t slowly trails behind π t .
In our experiment, we used
[Equation]
For each game, we used α “ 1 , η “ 0 . 1 . For Kuhn poker, we used ˜ η “ 1 e ´ {+3 , for 2x
+}
Abrupt Dark Hex, we used ˜ η “ 1 e ´ 3 , for 4-Sided Liar’s Dice, we used ˜ η “ 5 e ´ 5 , for Leduc Poker, we used ˜ η “ 5 e ´ 5 . Iterations
E x p l
o i t
[Figure]
b ili t y
[Figure]
We show the results in Figure 15, compared against CFR and MMD with an annealing temperature (with the same hyperparameters as before).
Encouragingly, we find that that moving the magnet behind the current iterate also appears to induce convergence.
Furthermore, convergence may occur at a much faster rate than that which is induced by annealing the temperature.
H. M AX E NT AND M INI M AX E NT O BJECTIVES
Next, we examine the convergence properties of other related objectives.
We consider two different types.
One involves an information state entropy bonus, wherein α H p π t p h
i qq is added to the reward for player i for reaching information state h
i .
This corresponds with a maximum entropy objective in reinforcement learning (Ziebart et al., 2008); we call this objective MaxEnt.
The second involves simultaneously giving an information state entropy bonus (like MaxEnt), while also penalizing the player with the opponent’s information state entropy.
This second approach can be viewed as a modification of the first approach that makes the game zero-sum.
It is the objective that was examined in Pérolat et al.
We call this objective MiniMaxEnt.
For each algorithm, we used
[Equation]
for Kuhn Poker,
{+[Equation] for 2x
 Dark Hex, [Equation]+}
for 4-Sided Liar’s Dice, and
[Equation]
[Figure] [Equation] [Figure] [Equation]
[Figure]
[Equation]
for Leduc Poker.
[Figure]
We show the results in Figure 16.
We find that MMD exhibits convergent behavior with each of the objectives.
[Equation] [Figure]
[Equation]
Next, we examine an instantiation of MMD that optimizes the logits using a Euclidean mirror map ( ψ “ 12 } ¨ } 22 ), as discussed in Section D.5, rather than reverse KL regularization.
The update rule for this approach is given by
[Equation]
where π t p h
i q “ softmax p z t p h
i qq and ζ is the magnet.
The closed form is
[Equation]
We test the convergence of Euclidean MMD for Leduc poker, using
[Equation]
We show the results in Figure 17.
We find that Euclidean MMD also exhibits convergence behavior.
However, convergence may be slower than the negative entropy variant.
[Table]
/gD5/MHP4aO3g==</latexit>
[Figure]
Leduc
[Figure] [Figure]
Next, we examine using MMD for the purposes of computing MiniMaxEnt equilibria. For each temperature α , we used η “ α { 10 . We show the results in Figure 18.
In the figure, convergence is measured in terms of exploitability in the entropy regularized game. Similarly to our AQRE results, we find that, althought convergence is non-monotonic, the empirical rate appears roughly linear over long time scales. This is the first empirical demonstration of convergence in MiniMaxEnt exploitability in EFGs. [Figure]
a x E n t
[Figure]
l
[Figure] i
[Figure]
H. E XPLOITABILITY WITH F IXED P ARAMETERS
In our AQRE and MiniMaxEnt experiments, we observed that MMD exhibits convergent behavior to AQREs and MiniMaxEnt equilibria with fixed parameter values.
It follows that MMD can achieve relatively good exploitability values without using any scheduling.
We show these results in Figure 19 and Figure 20.
These results suggest that it may be reasonable to instantiate MMD as a deep RL algorithm for 2p0s games without using any annealing.
↵ = 0 .<latexit
sha1_base64="h5GWBU8xdbIouEj0X/2rS5y50hA=">AAAB9HicbVBNSwMxEJ2tX7V [Table] [Table] >
[Table]
< [Table] >
[Table]
/zyKulc1L1G/fK+UWveFHGU4QRO4Rw8uIIm3EEL2kBgAs/wCm+OcF6cd+dj0Vpyiplj+APn8wcNTo2Z</latexit>
E x p l
o i t a b ili t y
[Figure]
[Figure]
I A TARI AND M UJOCO E XPERIMENTS
For our single-agent deep RL experiments, we implemented MMD as a modification to Huang et al.
’s
implementation of PPO.
For Atari, we added a reverse KL penalty with a coefficient 1 { η “ 0 . 001 ; we kept the temperature as the default value set by Huang et al.
( α “ 0 . 01 ).
For Mujoco, we added a reverse KL penalty with a coefficient 1 { η “ 0 . 1 ; we added an entropy bonus (Huang et al.
(2022) do not use an entropy bonus) with a value of α “ 0 . 0001 .
Otherwise, for both Atari and
Mujoco, the hyperparameters were set to those selected by Huang et al.
We show the results again in Table 4 for convenience.
The baseline results for PPO are copied directly from Huang et al.
The exact numbers should be interpreted cautiously as they are averaged over only three runs, leaving high levels of uncertainty.
That said, the results in Table 4 provide evidence that MMD can perform comparably to PPO.
But, even without looking at empirical results, the idea that a deep form of MMD can perform comparably to PPO should not be surprising, as MMD can be implemented in a way that resembles PPO in many aspects.
[Table]
414 ˘21 .
0 ˘ 0 .2549 ˘2898 ˘2215 ˘3638 ˘
J 3 X 3 A BRUPT D ARK H EX E XPERIMENTS
For our 3x3 Abrupt Dark Hex experiments, we implemented MMD as a modification to PPO, as implemented by RLlib (Liang et al., 2018).
This involved modifying the loss to add entropy regularization, as well a changing the adaptive forward KL regularization to a constant reverse KL regularization.
We used a schedule
[Equation]
where t is the number of time steps—not the number of episodes.
Otherwise, we used the default hyperparameters.
We ran this implementation in self-play using RLlib’s OpenSpiel environment wrapper, modified to work with information states, rather than observations.
For NFSP (Heinrich & Silver, 2016), we used the same hyperparameters as those found in the NFSP Leduc example in the OpenSpiel codebase.
For the best response, we used the OpenSpiel’s DQN best response code, without modifying any hyperparameters.
We ran the best response for 10 million time steps and evaluated all match-ups over 2000 games (with each agent being the first-moving player in 1000).
There are two caveats to consider in interpreting these experiments.
First, it is likely that RLlib’s default PPO hyperparameters are generally stronger than the default hyperparameters for NFSP in the
OpenSpiel.
In this respect, the results we present may be unfair to NFSP.
Second, RLlib’s OpenSpiel wrapper does endow agents with knowledge about which actions are legal—instead, if an illegal action is selected, the agent is given a small penalty and a random legal action is executed.
In contrast,
OpenSpiel’s implementation of NFSP uses information about the legal actions to perform masking.
In other words, MMD faces a harder version of the game than NFSP faces.
In this respect, the results we present are unfair to MMD.
[Table]
We show the results of our approximate exploitability experiments again in Table 5 for convenience.
The agents we evaluated are: a bot that always takes the first legal action, a bot that takes actions uniformly at random, NFSP after 1 million time steps, NFSP after 10 million time steps, MMD after 1 million time steps, and MMD and 10 million time steps.
Note that there are three kinds of uncertainty involved here: 1) uncertainty over training the agent (captured in table by the seed), 2) uncertainty over training the best response (also captured in the table by the seed), and 3) uncertainty in evaluating the match-up (captured in the table by the standard error).
First, we observe that there is a substantial difference in approximate exploitability between playing totally randomly and playing deterministically—the uniform random bot’s approximate exploitability ranged from 0 .
74 to 0 .
76 , while the first action taking bot had full exploitability for every seed.
Second, we observe that NFSP appears to be significantly slower to converge than MMD.
Aftermillion time steps, none of the five NFSP seeds have approximate exploitability that outperformed playing uniformly randomly in terms of approximate exploitability.
In contrast, with the same budget, all five of the MMD seeds outperform uniform random play.
After 10 million time steps, NFSP convincingly outperforms uniform random play, but generally does not even match the performance of MMD(1M steps).
MMD(10M steps) yields the best performance, with approximate exploitability ranging from 0 .
20 to 0 .
In terms of winning percentage, this means the exploiter defeated
MMD(10M steps) between 60% and 62% of the time.
[Table]
We show results of head-to-head match-ups in Table 6 for convenience.
The agents we evaluated are the ones that were trained for 10 million time steps.
Again, note that there are three uncertainties involved: training each of the two agents and evaluating the match-up.
First, we observe that, despite the large difference in approximate exploitability between always taking the first legal action and playing uniformly at random, the two bots are relatively evenly matched in head-to-head games.
Furthermore, they generally achieve similar expected returns against the MMD and NFSP agents.
This observation reflects the significant difference between worst case performance and average case performance.
In general, we find that MMD exploits the first legal action taking bot and the uniform random bot more than NFSP: MMD’s performance ranges from 0 .
62 to 0 .
70 , while, with the exception of one seed, NFSP’s performance ranges from 0 .
36 to 0 .
However, NFSP also has a seed with outlyingly strong performance against the first action taking bot that achieves an expected return of 0 .
In terms of head-to-head performance against each other, we found that MMD yielded stronger performance than NFSP—MMD’s expected return ranged from 8 to 38 .
In terms of winning percentage, this means that MMD won between 54% of the games and 69% of the games against
NFSP.
K A DDITIONAL R ELATED W ORK
Average Policy Deep Reinforcement Learning for Two-Player Zero-Sum Games One class of deep reinforcement learning methods for two-player zero-sum games, which includes NFSP (Heinrich & Silver, 2016) and PSRO (Lanctot et al., 2017), scales oracle-based approaches (Brown, 1951; McMahan et al., 2003) by using single-agent reinforcement learning as a subroutine to compute approximate best responses. While this class of methods is very scalable, it can require computing many best responses (McAleer et al., 2021), making it very slow in some cases.
MMD differs from this class in that it does not use a best response subroutine and in that it does not use averages over historical policies.
Another class of methods, which includes deep CFR (Brown et al., 2019) and double neural CFR (Li et al., 2020), is motivated by scaling CFR (Zinkevich et al., 2007)—the dominant paradigm in tabular settings—to function approximation.
Unfortunately, the sampling variant of CFR (Lanctot et al., 2009) requires importance sampling across trajectories, making straightforward extensions to stochasticity (Steinberger et al., 2020) difficult to apply to games with long trajectories, though more recent extensions may make progress toward resolving this issue (Gruslys et al., 2021).
MMD differs from this class both in that it neither requires policy averaging nor importance sampling over trajectories.
L R ELATIONSHIP TO KL-PPO AND MDPO
On the single-agent deep reinforcement learning side, MMD most closely resembles KL-PPO (Schulman et al., 2017) and MDPO (Tomar et al., 2020).
KL-PPO uses the policy loss function
E t
[Equation]
where ˆ A t is an advantage function (a learned estimate of q π t p s t , a t q ´ v π t p s t q ).
In expectation, the first term acts as x π t p s t q , q π t p s t qy , which is the first term of MMD’s loss function.
The second term is the same entropy bonus as exists in MMD’s loss function, using a uniform magnet with a negative entropy mirror map.
However, unlike MMD, KL-PPO’s KL regularization goes forward
KL p π old p s t q , π p s t qq .
In contrast, MMD’s KL regularization goes backward KL p
π p s t q , π old p s t qq .
MDPO uses the policy loss function
[Equation]
where ˆ A t is the approximate advantage function (a learned estimate of q π t p s t , a t q ´ v π t p s t q ).
In the context of a negative entropy mirror map and a uniform magnet, MDPO differs from MMD in that it does not necessarily include an entropy regularization term α H p
π p s t qq ; in the case that it does include such an entropy term, MDPO and MMD coincide. MMD with a negative entropy mirror map a uniform magnet takes the form
[Equation]
where β acts as an inverse stepsize.
M F UTURE W ORK
Our work opens up a multitude of important directions for future work.
On the theoretical side, these directions include pursuing results for black box sampling, behavioral-form convergence, convergence and relaxing the smoothness assumption to relative smoothness (Lu et al., 2018;
Bauschke et al., 2017) while removing strong convexity assumptions.
On the empirical side, these directions include constructing an adaptive mechanism for adapting the stepsize, temperature, and magnet (Badia et al., 2020; Fan & Xiao, 2022), pushing the limits of MMD as a deep RL algorithm for large scale 2p0s games, and investigating MMD as an optimizer for differentiable games (Goodfellow et al., 2014;
Letcher et al., 2019).