A CTOR -C RITIC P OLICY O PTIMIZATION IN A L ARGES CALE I MPERFECT -I NFORMATION G AME
Anonymous authors
Paper under double-blind review
A BSTRACT
The deep policy gradient method has demonstrated promising results in many large-scale games, where the agent learns purely from its own experience.
Yet, policy gradient methods suffer convergence problems to a Nash Equilibrium (NE) in multi-agent situations.
Counterfactual regret minimization has a convergence guarantee to a NE in 2-player zero-sum games, but it usually needs domain-speciﬁc abstractions and model-based traversing to deal with large-scale games.
Inheriting merits from both methods, we extend the actor-critic algorithm framework in deep reinforcement learning to tackle a large-scale 2-player zero-sum imperfectinformation game, 1-on-1 Mahjong, whose information set size and game length are much larger than poker.
In particular, we modify the policy optimization objective from originally maximizing the discounted returns to minimizing a type of weighted cumulative counterfactual regrets.
This modiﬁcation is achieved by approximating the regrets via a deep neural network and minimizing the regrets via generating self-play strategies using Hedge.
We name the proposed algorithm Actor-Critic Hedge (ACH) and derive its theoretical connection to CFR.
We prove the convergence of ACH to a NE under certain conditions.
Experimental results on the proposed 1-on-1 Mahjong benchmark and benchmarks from the literature demonstrate that ACH outperforms related state-of-the-art methods.
Also, the agent obtained by ACH defeats a human champion in 1-on-1 Mahjong.
I NTRODUCTION
Policy gradient methods using deep neural networks as policy and value approximators have been successfully applied to many large-scale games (Berner et al., 2019; Vinyals et al., 2019;
Ye et al., 2020).
Usually, a score function representing the discounted returns is maximized by the policy, i.e., the actor.
In the meantime, a value function, known as the critic, is learned to guide the directions and magnitudes of the policy gradients.
This type of actor-critic methods are efﬁciently scalable with regard to the game size and the amount of computational resources.
However, as pointed out in Srinivasan et al.
and Hennes et al.
(2020), policy gradient methods have no convergence guarantee to optimal solutions in competitive Imperfect-Information Games (IIGs).
The main reason is that the policy gradient theorem (Sutton et al., 1999) is established within the single agent situation, where the environment is Markovian.
However, learning becomes non-stationary and non-Markovian when multiple agents learn simultaneously in a competitive environment.
An optimal solution to a 2-player zero-sum IIG usually refers to a Nash Equilibrium (NE), where no player could improve by unilaterally deviating to a different policy.
Tremendous progress in computing such optimal solutions has been made by a family of tabular methods: Counterfactual Regret Minimization (CFR) (Zinkevich et al., 2008).
CFR is a type of iterative self-play algorithm based on regret minimization, and it guarantees to converge to a NE with regard to the average policy in 2-player zero-sum IIGs.
A game model is required in CFR to sample many if not all actions from a state during self-play.
To handle large-scale IIGs with CFR, abstractions 1 , applied to either the action space or the state space, are usually employed to reduce the game to a manageable size (Moravˇcík et al., 2017; Brown & Sandholm, 2018; 2019).
However, abstractions are domain speciﬁc (Waugh et al., 2009; Johanson et al., 2013; Ganzfried & Sandholm, 2014).
Besides, some large-scale IIGs may be inherently difﬁcult to be abstracted.
In this paper, we investigate a large-scale IIG, Mahjong.
Li et al. (2020b) has recently developed a strong Mahjong AI based on a novel integration of supervised learning and traditional Reinforcement Learning (RL).
In comparison, we study Mahjong from a game-theoretic perspective, i.e., aiming for a NE.
We focus on a 2-player (1-on-1) zero-sum Mahjong game, whose information set size and game length are much larger than poker 2 .
We are interested in methods using only trajectory samples to learn, as it is infeasible to consistently sample multiple actions for each state in game trees of large-scale IIGs with long episodes.
Also, we employ deep neural networks to generalize across states, since the state or action abstraction in 1-on-1 Mahjong is not as straightforward as in poker, which is explained in the Appendix A.2.
To summarize, we make the following contributions.
• Inheriting the scalability of deep RL methods and the convergence property of CFR, we develop a new actor-critic algorithm, named Actor-Critic Hedge (ACH), for approaching a NE in largescale 2-player zero-sum IIGs, such as 1-on-1 Mahjong.
ACH employs a deep neural network to approximate a type of weighted cumulative counterfactual regrets.
In the meantime, ACH minimizes the regrets via generating self-play strategies using Hedge (Freund & Schapire, 1997).
• We prove that the exploitability of the average strategy of ACH under certain conditions decreases at the rate of O ( T − 1 / 2 )
+ O (1) , where T denotes the number of iterations in ACH.
The term O (1) disappears if the behavioral policy (i
e., a separate policy for sampling actions in self-play) is constant across iterations and samples any action with non-zero probability.
• To facilitate research on large-scale 2-player zero-sum IIGs and Mahjong, we propose a 1-onMahjong benchmark.
The corresponding game enjoys a large population in online games.
• We build a 1-on-1 Mahjong agent,
named JueJong, based on ACH.
In an initial evaluation against human players including a Mahjong champion 3 , JueJong demonstrates superior performance.
N OTATIONS AND B ACKGROUND
In this section, we introduce notations in the context of IIGs and NE.
Related background on RL (focusing on policy gradient methods) and CFR are brieﬂy described.
2. I MPERFECT -I NFORMATION G AMES AND N ASH E QUILIBRIUM
An IIG is usually represented in an extensive-form, i.e., a game tree.
A node (history) h ∈ H in the tree represents all information of the current situation.
Given a history h , P : H → P ∪ { c } maps h to the unique player p ∈ P who acts at h or the chance player c , where the chance player is responsible for taking actions for random events.
When P ( h ) ∈ P , the player P ( h ) has to take an action a ∈ A .
We use A ( h ) to represent the set of legal actions of P ( h ) .
The set of terminal nodes is denoted as Z .
At each z ∈ Z , there is a payoff u p : Z → R for each player p ∈ P .
In this paper, we focus on 2-player zero-sum games, where P = { 0 , 1 } and u 0 ( z ) + u 1 ( z ) = 0 for each z ∈ Z .
For either player p , the set of histories H is partitioned into information sets (infosets).
We denote the set of infosets for player p as I p and an infoset in I p as I p .
Two histories h, h (cid:48) ∈ H are in the same infoset if and only if h and h (cid:48) are indistinguishable from the perspective of player p .
As a result, a player p ’s strategy π p maps an infoset to a probability distribution over legal actions.
For the same reason, we deﬁne A ( I p ) =
A ( h ) , ∀
h ∈
I p and P ( I p ) =
P ( h ) , ∀ h ∈ I p .
A strategy proﬁle π is simply a tuple of strategies ( π p , π − p ) , where π − p represents the player p ’s opponent strategy.
The expected payoff for player p under π is then denoted as u p ( π p , π − p ) .
We use ∆( I ) to denote the range of payoffs reachable from a history h in infoset I .
Let ∆ =
max I ∈I p ,p ∈P ∆( I ) .
f π ( h ) denotes the joint reaching probability of h under π . f πp ( h ) is the contribution of player p to f π ( h ) , and f π − p ( h ) is the contribution of the opponent and chance: f π ( h )
=
f πp ( h ) f π − p
( h ) .
We focus on the perfect-recall setting, where each player recalls the sequence of their own infosets reached.
We deﬁne f
πp ( I p ) =
f
πp ( h ) , ∀ h ∈
I p , f π − p
( I p ) =
(cid:80) h ∈
I p
f π − p
( h ) , and f π ( I p ) =
(cid:80) h ∈
I p
f π ( h ) =
f πp ( I p ) f π − p ( I p ) .
A best response BR ( π − p ) to π − p is a player p ’s strategy, which satisﬁes u p ( BR ( π − p ) , π − p ) =
max π (cid:48) p u p ( π (cid:48) p , π − p ) .
A NE is a strategy proﬁle
π ∗ , where each player plays a best response to the other: u p ( π ∗ p , π ∗− p ) =
max π (cid:48) p u p ( π (cid:48) p , π ∗− p ) , ∀ p ∈ P .
The exploitability of a player’s strategy, denoted as e ( π p ) , measures the performance gap between π p and a NE strategy π ∗ p : e ( π p ) =
u p ( π ∗ p , π ∗− p ) − u p ( π p , BR ( π p )) .
The average exploitability is (cid:15) =
|P| (cid:80) p ∈P e ( π p ) .
2. R EINFORCEMENT L EARNING AND P OLICY G RADIENT M ETHODS
RL usually assumes a Markov decision process, where the agent selects an action a i from the action set A ( s i ) in state s i ∈ S 4 at each time step i .
The agent then receives a reward r i from the environment and transitions to a new state s i +1 .
The objective in RL is to learn a policy that maximizes the expected discounted returns, i.e., the state value, starting from any state s :
V π ( s ) =
E π [ (cid:80) ∞
j = i γ j − i r
j | s i
= s ]
=
E π
[ G ] , with the discount factor γ ∈ [0 , 1) .
A large family of methods in RL belong to policy gradient methods, in which the parameters θ of policy π ( a | s ; θ ) are updated by performing gradient ascent directly on E
π θ [ G ] .
One early example is the standard REINFORCE algorithm (Williams, 1992) that updates θ in the direction ∇ θ log π ( a | s ; θ ) G , which is an unbiased estimate of ∇ θ E π θ [ G ] .
To further reduce the variance of the gradient, an action-independent baseline is often subtracted from G : ∇ θ log π ( a | s ; θ )[ G − B ( s )] .
A recent policy gradient method, named Advantage Actor-Critic (A2C) (Mnih et al., 2016), learns a parameterized value function as the baseline: B ( s ) =
V π θ ( s ; w ) , where w often shares some parameters with θ .
Moreover, the value G − B ( s ) is replaced in A2C with the estimated advantage of action a in state s :
A ( s, a ) =
Q ( s, a ) − V ( s ) .
Q ( s, a ) is usually estimated using sampled rewards and predicted values of future states.
A2C updates parameters θ and w in a synchronous manner, while its asynchronous version is Asynchronous Advantage Actor-Critic (A3C) (Mnih et al., 2016).
In an asynchronous training environment, the behavioral policy is usually different from the learning policy.
Proximal Policy Optimization (PPO) (Schulman et al., 2017) takes this discrepancy into account by multiplying A2C gradient with an importance ratio r ( a | s ; θ )
=
π ( a | s ; θ ) /π
( a | s ; θ old ) , which results in r ( a | s ; θ ) ∇ θ log π ( a | s ; θ ) A ( s, a ) = ∇ θ r ( a | s ; θ )
A ( s, a ) .
Furthermore, PPO constrains the KL divergence between the learning policy π θ and the old behavioral policy π θ old by clipping the ratio to a small interval around 1 . 0 .
2. C OUNTERFACTUAL R EGRET M INIMIZATION
CFR is an iterative algorithm that minimizes the total regret of strategies by minimizing the cumulative counterfactual regrets in every state (infoset).
The cumulative counterfactual regret of an action a in state s is deﬁned as R ct ( s, a ) = (cid:80) tk
=1 r ck ( s, a ) , where r ck ( s, a ) denotes the instantaneous counterfactual regret of action a in state s at iteration k .
r ct ( s, a ) is equal to f π t − p ( s )
A π t ( s, a ) (as proven in Srinivasan et al.
(2018)), where A π t ( s, a ) is the advantage function of player
p =
P ( s ) at state s as deﬁned in traditional RL.
Intuitively, r ct ( s, a ) represents how regretful player p is that he does not select the action a in state s at iteration t .
The term f π t − p ( s ) =
(cid:80) h ∈ s f π t − p ( h ) is needed here to reﬂect the fact that the reaching of state s is also controlled by the opponent − p .
To minimize the counterfactual regrets, a regret minimizer is utilized in each state at each iteration, generating a series of local policies π 1 ( s ) , . .
, π t ( s ) .
Once π t is obtained, r ct ( s, a ) is generated by a model-based game tree traversing using π t , and R ct ( s, a ) is updated accordingly: R ct ( s, a ) =
R ct − 1 ( s, a ) + r ct ( s, a ) .
Deﬁne the total regret of player p as R cT = max π (cid:48) (cid:80) Tt =1 ( u p ( π (cid:48) , π − p,t ) − u p ( π p,t , π − p,t )) .
It is proven in Zinkevich et al.
(2008) that R cT ≤ (cid:80) s ∈S
[ R cT ( s )] + , where [ · ] + = max {· , 0 } and
R cT ( s ) = max a ∈A ( s ) R cT ( s, a ) .
As a result, the total regret can be minimized by minimizing the cumulative counterfactual regrets at each state.
Moreover, the average strategy ¯ π T ( a | s ) = (cid:80)
t ∈ T f π t p ( s ) π t ( a | s ) / (cid:80) t ∈ T f π t p ( s ) converges to a NE if R cT grows sub-linearly with T (Zinkevich et al., 2008).
There are two commonly used regret minimizers: Regret Matching (RM) and Hedge (Cesa-Bianchi & Lugosi, 2006).
In RM, a player selects an action with probability in proportion to its positive cumulative counterfactual regret.
In Hedge, the strategy π t +1 ( a | s ) is decided according to:
[Equation]
If a player plays according to Hedge in state s and η ( s ) =
(cid:112) 8 log( |A ( s ) | ) / (∆ 2 ( s ) T ) , then R cT ( s ) ≤ ∆( s ) (cid:112) log( |A ( s ) | )
T/ 2 (Cesa-Bianchi & Lugosi, 2006).
In other words, the total regret grows sub-linearly with T , and therefore the average strategy ¯ π T ( a | s ) converges to a NE.
A CTOR -C RITIC H EDGE
In this section, we develop the new actor-critic algorithm, i.e., ACH.
We make modiﬁcations within the actor-critic algorithm framework in deep RL such that ACH remains highly scalable.
Key to ACH is the idea that we change the policy net from predicting action probabilities in deep RL to approximating the expectation of the sum of sampled advantages R at ( s, a ) :
=
E [ (cid:80) tk =1 ˜
A π k ( s, a )] , where ˜
A π k ( s, a ) =
A π k ( s, a ) if s is visited at iteration k (given that only one trajectory is sampled at each iteration).
Otherwise, ˜
A π k ( s, a ) = 0 .
In other words, the expectation E [ ˜ A π k ( s, a )] depends on both the advantage
A π k ( s, a ) and the sampling strategy µ k at iteration k .
The strategy π t at each iteration in ACH is obtained via Hedge on the output of the policy net.
In the next section, we will deduce that the policy target R at ( s, a ) in ACH can be viewed as a type of weighted cumulative counterfactual regret .
As a result, ACH is theoretically connected to CFR with Hedge, more details of which will be given in the next section.
Suppose at the beginning of iteration t , R at − 1 ( s, a ) is well approximated by the output y ( a | s ; θ t − 1 ) of the policy net in ACH, where θ t − 1 represents the parameters of the policy net in ACH.
The strategy π t for iteration t is ﬁrst obtained via Hedge, i.e., softmaxing on η ( s ) y ( a | s ; θ t − 1 ) .
The reason we use
Hedge instead of RM is that softmaxing is shift-invariant, which may be more robust to the function approximation error compared to the threshold operation in RM.
M trajectories are then sampled into the buffer B v via model-free self-play using the strategy proﬁle
π t =
( π p,t , π − p,t ) .
Those samples in
B v are used to train the action value net by minimizing the squared loss: 12 [ Q ( s, a ; ω ) − G ] 2 , where ω represents the parameters of the action value net and G is the sampled return.
Afterwards, another M trajectories are sampled into the buffer B π via self-play using another sampling strategy proﬁle µ t = ( µ p,t , π − p,t ) , where µ p,t is the behavioral policy of player p and π − p,t is the current policy of player − p .
We use an additional behavioral policy µ p,t for player p other than π p,t , because µ p,t can provide more ﬂexibility for the convergence (more details are provided in the next section).
The policy θ is then optimized according to the following squared loss function:
[Equation]
otherwise ,
[Equation]
where τ i is the i th sampled trajectory in B π .
The next iteration of ACH begins after the policy training ﬁnishes.
The pseudocode of ACH is given in Algorithm 1.
Note that we present ACH from the perspective of player p , and the same procedure applies to player − p .
ACH runs concurrently for both players and synchronizes at the beginning of each iteration.
P ROPERTIES OF A CTOR -C RITIC H EDGE
In this section, we prove the convergence of ACH to an approximate NE.
To this end, we ﬁrst deduce that the policy target in ACH is R at ( s, a ) , which is a type of weighted cumulative counterfactual regret.
We then deﬁne a new family of CFR algorithms: weighted CFR.
Finally, we show that ACH is equivalent to a type of weighted CFR with Hedge under certain conditions. The convergence properties of weighted CFR with Hedge and hence ACH are proven accordingly.
We can rewrite the ACH policy loss (in Equation 2) at iteration t as L π = (cid:80) s ∈S (cid:80) a { y ( a | s ; θ ) − [ y ( a | s ; θ t − 1 ) + 1 M (cid:80) Mi
=1 1 s ∈ τ
i A π t ( s, a )] } 2 .
In other words, the target of y ( a | s ; θ ) at iteration t is y ( a | s ; θ t − 1 ) + 1 M (cid:80) Mi =1 1 s ∈ τ i A π t ( s, a ) , whose expectation is E
[Equation]
Also, E τ
[Equation]
∼
[Equation]
s, a ) = f
[Equation]
p
[Equation]
s )
f
[Equation]
k − p ( s )
A
[Equation]
k ( s, a ) , where f µ k p ( s ) f π k − p ( s ) is the reaching probability of state s at iteration k .
Therefore, R at ( s, a ) =
(cid:80) tk =1 f µ k p ( s )
f π k − p ( s )
A π k ( s, a ) .
Since
Algorithm
Function ACH( p, N, M, θ , ω ) : ω 0 ← ω, θ 0 ← θ .
for t ← 1 to N do # Obtain the policy : π t ← Softmax( η ( s ) y ( a | s ; θ t − 1 )) .
# Train the action value net :
Reset B v ← ∅ .
for i ← 1 to M do τ i ∼ SelfPlay( π p,t , π − p,t ) , B v ← B v ∪ τ
i
Train ω on the loss E ( s,a,G ) ∼B v { 12 [ Q ( s, a ; ω ) − G ] 2 } ω t ← ω
# Train the policy net :
Reset B π ← ∅ .
for i ← 1 to M do τ i ∼ SelfPlay( µ p,t , π − p,t ) , B π ← B π ∪
τ
i
Estimate A π t ( s, a ) , ∀ a ∈ A ( s ) , ∀ s ∈ B π as Q ( s, a ; ω t ) − (cid:80) b π t ( b | s ) Q ( s, b ; ω t ) .
Sum aggregate advantage A π t ( s, a ) in B π by state s .
Train θ on the loss E s ∼S {L π } , where L π is deﬁned in Equation 2.
θ t ← θ .
r ck ( s,
[Equation]
) = [Equation]
p
[Equation]
s
[Equation]
A
[Equation]
k ( s, a ) (Srinivasan et al., 2018), we have
[Equation]
Hence, R at ( s, a ) is a type of weighted cumulative counterfactual regret that weighs each instantaneous regret with f µ k p ( s ) .
Without loss of generality, we deﬁne a new family of CFR algorithms, weighted
CFR, below.
Afterwards, we present Theorem 1.
Deﬁnition 1.
Weighted CFR follows the same procedure as the original CFR, except that the instantaneous regret r ct ( s, a ) is weighted by some weight w t ( s ) , w t ( s ) > 0 and (cid:80) ∞ t =
w t ( s ) = ∞ .
The original CFR is a type of weighted CFR with w t ( s )
= 1 . 0 .
Theorem 1.
ACH is equivalent to a type of weighted CFR with Hedge when w t ( s ) =
f µ t p ( s ) > 0 , given that enough trajectories are sampled and y ( a | s ; θ t ) is sufﬁciently close to R at ( s, a ) .
Further, if η ( s ) = (cid:112) 8 ln |A ( s ) | / { [ w h ( s )] 2 ∆ 2 ( s ) T } and w t ( s ) =
f µ t p ( s ) ∈ [ w l ( s ) , w h ( s )]
⊂ (0 , 1] , t = 1 , . .
, T , the average strategy π of the corresponding weighted CFR with Hedge and equivalently ACH with π p ( a | s
[Equation] = (cid:80)
[Equation] / (cid:80) [Equation]
[Equation]
π t p ( s ) , ∀ p ∈ P , converges to a (cid:15) -Nash equilibrium after T iterations, where
[Equation]
In Theorem 1, we proved that the exploitability of ACH is bounded by (cid:15) when y ( a | s ; θ t ) is sufﬁciently close to R at ( s, a ) .
There are two terms in the bound (cid:15) .
The ﬁrst term converges to zero at the rate of
O ( T − 1 / 2 ) , and the second term is O (1) , which is weighted by the sum of the normalized range of the weights (cid:80) s ∈S ( w h ( s )
− w l ( s ))
/w
h ( s ) .
However, it is possible to reduce the second term to an arbitrarily small value via tightening the range of f µ t p ( s ) .
Corollary 1.
If the behavioral policy µ p,t for each player p ∈ P is constant across iterations, and f µ t p ( s ) > 0 , ∀ s ∈ S , t > 0 , ACH is equivalent to CFR with Hedge when y ( a | s ; θ t ) is sufﬁciently close to R at ( s, a ) .
As shown in Corollary 1, when the behavioral policy µ p,t for each player p ∈ P is time-invariant, i.e., w h ( s ) =
f µ t p ( s )
=
w l ( s ) , ∀ s ∈ S , t > 0 , the second term in (cid:15) disappears, and CFR with Hedge is recovered.
All the proofs are given in the Appendix C.
S OME C ONSIDERATIONS ON I MPLEMENTING ACH FOR 1- ON -1 M AHJONG
To strike a balance between theoretical soundness and practical efﬁciency, there are the following three considerations when implementing ACH for the large-scale game 1-on-1 Mahjong. The current policy and the average policy. Theorem 1 and Corollary 1 state the convergence property of the average policy of ACH.
Yet, as pointed out in Srinivasan et al.
, Hennes et al.
(2020), and Perolat et al.
(2021), obtaining the average policy with deep neural nets in large-scale games is inherently difﬁcult, due to either the computation or the accuracy issue.
Alternatively, we could turn to some additional technique to hopefully induce the current policy convergence towards a
NE.
Srinivasan et al.
and Hennes et al.
(2020) handled this by adding an entropy regularization to the current policy training, which is, to some extent, theoretically justiﬁed later in Perolat et al.
We decide to add an entropy regularization when applying ACH to 1-on-1 Mahjong as well.
The behavior policy.
The behavior policy µ p,t in ACH could be set to either the current policy π p,t or simply a uniform sampling policy.
Corollary 1 gives a tighter bound on the exploitability of the average policy of ACH if µ p,t is stationary over iterations.
In a preliminary experiment (Appendix E), we ﬁnd that the performance of the current strategy in ACH (trained with an entropy regularization) is not sensitive to the choice of µ p,t .
We use µ p,t = π p,t on 1-on-1 Mahjong.
Training on states not sampled.
Theoretically, in order to optimize Equation 2, we need to collect both sampled and non-sampled states.
Optimizing with only sampled states makes y ( a | s ; θ t ) a biased estimation of R at ( s, a ) .
Yet, collecting non-sampled states may be intractable in large-scale games (Li et al., 2020a).
In practice, we ﬁnd that optimizing with only sampled states gives us satisfactory results on 1-on-1 Mahjong.
Based on the above practical considerations, we provide a parallel implementation of ACH by employing a framework of decoupled acting and learning, similar to IMPALA (Espeholt et al., 2018).
The parallel implementation trains the network with mini batches and handles asynchronous training with the importance ratio clipping of PPO.
All experimental results with regard to ACH are based on the parallel implementation, more details of which are presented in the Appendix D.
R ELATED W ORK
To obviate the need of abstractions, various neural forms of CFR methods have been developed.
An early work of this direction is regression CFR (Waugh et al., 2015), which calculates weights for a number of hand-crafted features to approximate the regrets.
Deep CFR (Brown et al., 2019) is similar to regression CFR but employs a neural network to approximate the regrets.
Also, deep CFR traverses a part of the game tree using external sampling, in comparison to the full traversal of the game tree in regression CFR.
Double neural CFR (Li et al., 2020a) is another method that approximates the regrets and the average policy using deep neural networks.
The learned cumulative target in Li et al.
(2020a) is scaled by the inverse of sampling probability, which may vary signiﬁcantly in games with long episodes.
In comparison, the advantage, which tends to 0 , is accumulated and learned in ACH.
Both deep CFR and double neural CFR build on the tabular Monte Carlo CFR (MCCFR) (Lanctot et al., 2009), where either outcome sampling (sampling one action in a state) or external sampling (sampling all actions in a state) could be employed.
When dealing with games with long episodes, a necessity may be that only trajectory samples are allowed.
To improve the learning performance with only trajectory samples, DREAM (Steinberger et al., 2020) adapts deep CFR by using a learned Q-baseline, which is inspired by the variance reduction techniques in tabular MCCFR (Schmid et al., 2019; Davis et al., 2020).
Another recent work using only trajectory samples is ARMAC (Gruslys et al., 2020).
By replaying through past policies and using a history-based critic, ARMAC predicts conditional advantages, based on which the policy for each iteration is generated.
Other popular neural network based methods, which learn from trajectory samples and are inspired by game-theoretic approaches other than CFR, include neural ﬁctitious self-play (Heinrich & Silver, 2016), policy space response oracles (Lanctot et al., 2017), and exploitability descent (Lockhart et al., 2019), all of which require to compute an approximate best response at each iteration.
Such computation may be prohibitive in large-scale games.
The most related methods to ACH are Regret Policy Gradient (RPG)
(Srinivasan et al., 2018) and Neural Replicator Dynamics (NeuRD) (Hennes et al., 2020), both of which employ the actor-critic framework and thus have similar computation and memory complexities as ACH.
RPG minimizes a loss that is an upper bound on the regret after threshold, and the corresponding policy gradient is ∇ RPGθ ( s )
= − (cid:80) a ∇ θ [ Q ( s, a ; w ) − (cid:80)
b
π ( b | s ; θ ) Q ( s, b ; w )] + .
However, RPG requires a l 2 projection after every gradient step for the convergence to a NE, while such projection is not required in ACH. NeuRD is inspired by the replicator dynamics, a well studied model in evolutionary game theory (Gatti et al., 2013). The policy gradient in NeuRD is ∇ NeuRDθ ( s )
= (cid:80) a [ ∇ θ y ( a | s ; θ )] [ Q ( s, a ; w ) − (cid:80) b π ( b | s ) Q ( s, b ; w )] , where y ( a | s ; θ ) is the output of the policy net.
There are important differences between ACH and NeuRD in how the algorithm is motivated and how the policy net at each iteration is optimized. Also, the convergence analysis was given only for the single-state, all-actions, tabular NeuRD (Hennes et al., 2020), while we link ACH to a type of weighted CFR with Hedge and prove its convergence in full extensive-form games.  E XPERIMENTAL S TUDIES
We ﬁrstly introduce a 1-on-1 Mahjong benchmark, on which we compare ACH with related stateof-the-art methods of similar computation complexity: PPO, RPG, and NeuRD.
Since our goal is to approximate a NE, the standard and default performance metric, exploitability, is employed.
We approximate a lower bound on the exploitability of an agent (i.e., solution) by training a best response against it as suggested in Timbers et al.
(2020) and Steinberger et al.
(2020), because traversing the full game tree to compute the exact exploitability is intractable in such a large-scale game as 1-onMahjong.
As a complement, head-to-head performance of different methods on 1-on-1 Mahjong is also presented.
Moreover, the agent obtained by ACH is evaluated against practised Mahjong human players.
To further validate the performance of ACH in IIGs other than 1-on-1 Mahjong, experimental results on a non-trivial poker game, i.e., heads-up Flop Hold’em Poker (FHP) (Brown et al., 2019) are presented.
Also, deep CFR with outcome sampling (OS-DCFR) and DREAM are added to enable a more thorough comparison.
Additional results on smaller benchmarks from OpenSpiel (Lanctot et al., 2019) are reported in the Appendix F. Note that results are reported for the current policy in[-Srinivasan et al. (2018) and Hennes et al. We use TensorFlow 2 . 2 . 
-]
ACH, PPO, NeuRD, and RPG and the average policy in OS-DCFR and DREAM.
7. A 2-P LAYER Z ERO -S UM M AHJONG B ENCHMARK
Mahjong is a tile-based game that is played world wide with many regional variations, such as
Japanese Riichi Mahjong and Competition Mahjong.
Like poker, Mahjong is an IIG and is full of strategy, chance, and calculation.
To facilitate Mahjong research from a game-theoretic perspective, we propose a 2-player zero-sum Mahjong benchmark 5 , whose game rules are similar to Competition
Mahjong.
The corresponding game, “2-player Mahjong Master”, is played by humans in Tencent mobile games.
A full description of the game rules is in the Appendix A.1.
Apart from being the ﬁrst benchmark for the 1-on-1 Mahjong game, our benchmark has a larger infoset size (resulting in a larger history size) and a longer game length, compared with existing poker benchmarks (Lanctot et al., 2019).
The infoset size (i.e., the number of distinct histories in an infoset) in [-1v
-]1-on-1 Mahjong is around 10 11 , compared to 10 3 in poker.
This is due to the fact that only two private cards are invisible in poker, while there are 13 invisible tiles in 1-on-1 Mahjong.
In addition, players can decide up to about 40 sequential actions in 1-on-1 Mahjong, whereas most 1-on-1 poker games end withinsteps.
More details about the 1-on-1 Mahjong benchmark are given in the Appendix A.
7. R ESULTS ON O UR 1- ON -1 M AHJONG B ENCHMARK
All methods run in an asynchronous training platform with overall 800 CPUs, 3200 GB memory, and 8 M40 GPUs in the Ubuntu 16 .
04 operating system.
Each method shares the same neural network architecture, a full description of which is given in
the Appendix B. We performed a mild hyper-parameter search on PPO and shared the best setting for all methods. The advantage value is estimated by the Generalized Advantage Estimator (GAE( λ )) (Schulman et al., 2016) for all methods. An overview of the hyper-parameters is listed in the Appendix G.1.
Approximate a Lower Bound on the Exploitability .
To approximate a lower bound on the exploitability of the agents obtained by each method, we train a best response against each agent.
The agent of each method is selected at the 1 e 6 th training step.
Note that the agent is ﬁxed as a part of
1.0 Training Steps 1e
A v e r a g e S
c
o r e s
Lo s t
PPORPGNeuRDACH (Ours) (
[Figure]
Approximate Exploitability 0.
1.0 Training Steps 1e
A v e r a g e S
c
o r e s W o n
PPORPGNeuRDACH (Ours)
[Figure]
the 1-on-1 Mahjong environment when training the best response.
We train the best response using
PPO with the same hyper-parameters that were used to train the PPO agent 6 .
During the training of the best response, we evaluate the best response every 500 training steps using 10 , 000 head-to-head plays against each agent.
According to the average scores each agent loses to its best response in Figure 1(a), we may conclude that ACH is signiﬁcantly more difﬁcult to exploit than other methods in the large-scale 1-on-1 Mahjong environment.
Head-to-Head Evaluation .
We then compare the head-to-head performance of PPO, RPG, NeuRD, and ACH in the 1-on-1 Mahjong environment.
First, we compare the training process of each method by evaluating each agent every 500 training steps against a common rule-based agent 7 using 10 ,head-to-head plays, the results of which are shown in Figure 1(b).
As we can see, all methods beat the common rule-based agent signiﬁcantly, while ACH has a clear advantage over other methods in terms of stability and ﬁnal performance.
The relatively slow convergence of RPG may due to the threshold operation on the advantage, which could reduce sample efﬁciency in large-scale IIGs.
Second, the superior performance of ACH is further validated in head-to-head evaluations with other agents in Table 1, where all the agents are selected at the 1 e 6 th training step.
The agent of ACH wins all other agents by a signiﬁcant margin.
[Table]
Human Evaluation .
We evaluate the agent, selected at the 1 e 6 th training step, of ACH against human players.
First, the agent, named JueJong, is roughly evaluated by playing over 7 , 700 games against 157 practiced Mahjong players, where JueJong won an average of 4 .
56 scores per game.
Second, we select the top 4 out of 157 players according to their performances against JueJong and play JueJong against the four players for 200 games each.
As shown in Figure 2(a), the average winning scores of JueJong oscillate in the ﬁrst 120 games but all plateau above 0 afterwards.
More importantly, we evaluate JueJong against the Mahjong champion Haihua Cheng for 1 , 000 games, as shown in Figure 2(b).
After playing 1 , 000 games, JueJong won the champion by a score of 0 .
82 ± 0 .
96 (mean ± standard deviation), with a p-value of 0 .
19 under one-tailed t-test.
Hence, we may conclude that Haihua Cheng failed to exploit JueJong effectively within 1 , 000 games. [Figure]
[Figure]
7. R ESULTS ON THE FHP B ENCHMARK
We further evaluate ACH and compare it with OS-DCFR, DREAM, A2C 8 , RPG
[Figure]
and NeuRD on FHP.
FHP is a simpliﬁed Heads-up Limit Texas Hold’em (HULH), which includes only the ﬁrst two of the four bettings in HULH.
It is a medium-sized game with over 10 12 nodes and 10 9 infosets.
All the methods share the same neural network architecture proposed in Brown et al.
We perform a mild hyper-parameter search for ACH, A2C, RPG, and NeuRD.
For OS-DCFR and DREAM, we follow the hyper-parameters presented in Steinberger et al.
All the hyper-parameters and the running environment are described in the Appendix G.2.
As shown in Figure 3, ACH performs competitively with OS-DCFR and slightly worse than DREAM on FHP in terms of exploitability per episodes generated.
However, ACH is much more sample efﬁcient: ACH achieves an exploitability of 10 almost 100 times faster than DREAM and 1,000 times faster than OS-DCFR.
Also, ACH converges signiﬁcantly faster and achieves a lower exploitability than methods (A2C, RPG, and NeuRD) with similar sample efﬁciency.
C ONCLUSIONS
In this paper, we investigated the problem of adapting policy gradient methods in deep RL to tackle a large-scale IIG, i.e., 1-on-1 Mahjong.
To this end, we developed a new model-free actor-critic algorithm, i.e, ACH, for approximating a NE in large-scale IIGs.
ACH is memory and computation efﬁcient, as it uses only trajectory samples at the current iteration and requires no computation of best response.
As a result, ACH is efﬁciently scalable.
We proved the convergence of ACH to an approximate NE in 2-player zero-sum IIGs under certain conditions, by linking ACH to a type of weighted CFR with Hedge.
The superior performance of ACH was validated on both our 1-onMahjong benchmark and other common benchmarks.
Secondly, to facilitate research on large-scale IIGs, we proposed the ﬁrst 1-on-1 zero-sum Mahjong benchmark, whose infoset size and game length are much larger than poker.
Finally, using ACH we obtained the 1-on-1 Mahjong agent JueJong, which has demonstrated stronger performance against the Mahjong champion Haihua Cheng.
R EPRODUCIBILITY S TATEMENT
The experiments on the 1-on-1 Mahjong benchmark were run in a large cluster of thousands of machines, on which we have developed an efﬁcient actor-learner training platform, similar to IMPALA (Espeholt et al., 2018).
The codes of both the platform and the 1-on-1 Mahjong benchmark are not released currently but are planned to be open sourced in the near future.
Yet, the codes for both the actor part and the learner part of each algorithm (PPO, RPG, NeuRD, and ACH) are included in the supplementary materials.
All the hyper-parameters are listed in the Appendix G.
We implement the FHP benchmark as well as the FHP speciﬁc best response algorithm based on the OpenSpiel project.
All the algorithms tested on FHP are implemented using TensorFlow (1.15)
C++ API.
The codes will be open sourced in the future.
All the hyper-parameters are listed in the Appendix G.
The experiments on the benchmarks in OpenSpiel were run using the OpenSpiel environment.
We include the necessary codes, rand seeds, and instructions to reproduce all the results on OpenSpiel in the supplementary materials.
All the hyper-parameters are listed in the Appendix G.
All the theoretical results with regard to ACH are presented in the main text, with all the proofs included in the Appendix C.
A I NTRODUCTION OF THE 1- ON -1 M AHJONG G AME
Mahjong is a tile-based game that is played world wide with many regional variations, such as
Japanese Riichi Mahjong and Competition Mahjong.
Like poker, Mahjong is an IIG and is full of strategy, chance and calculation.
In this paper, we investigate a 1-on-1 Mahjong game, whose information set size is much larger than poker, as shown in Figure 4.
The game rules of 1-onMahjong are similar to Competition Mahjong.
The corresponding game, “2-player Mahjong Master", is played by humans in Tencent [Figure]
[Figure]
A. T HE G AME R ULES
There are 24 unique tiles and 72 tiles in total in the 1-on-1 Mahjong game, as shown in Figure 5.
At the beginning of the 1-on-1 Mahjong game, each player is dealt with 13 tiles, the content of which is invisible to the other.
Afterwards, each player takes actions in turn.
Typically, the ﬁrst player draws a tile from the deck wall and then discards a tile, and the next player takes the same types of actions in sequence.
There are exceptional cases where the player does not draw a tile from the deck wall but Chow, Pong, or Kong the tile the opponent just discarded.
Afterwards, the player discards a tile, and the game proceeds.
Also, after drawing a tile from the deck wall, there are cases where the player could Kong, after which the player draws and discards a tile in sequence.
There are 10 types of actions with 105 different actions in total, a full description of which is listed in Table 2.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
The goal of each player is to complete a legal hand prior to the opponent, by drawing a tile or using the tile the opponent just discarded.
A legal hand is generally in the form of four melds and a pair, with an exception of 7 pairs.
Different categories of legal hands, 64 in total, come with different points, which is fully described in Table 3.
Besides, a legal hand can belong to multiple categories, and the score of a legal hand is the sum of points of corresponding categories.
Legal hands with higher points are generally more difﬁcult to form in terms of either luck or strategy, and it is critical for a player to trade off the winning probability and the corresponding winning points.
If no player completes a legal hand before the tiles are exhausted, the game is tied.
A ﬂow chart of the game is illustrated in Figure 6.
[Table]
3 character tiles.
Short Straight A hand with two successive sequences.
Two Terminal Chows
A hand with two sequences 123 and 789.
Melded Kong
A hand with exposed quads.
Edge Wait A hand completion with the situation there is only one tile name to complete because you have the incomplete sequence in edge (12 or 89).
Closed
Wait A hand completion with the situation there is only one tile name to complete because you have the incomplete sequence lacking center (like 24 and 79).
Single Wait A hand completion solely waiting on a tile to form a pair.
Self Draw A hand completion by draw.
Flower Tile Whenever a player draws a ﬂower or season tile, this is counted as 1 point.
Ready Hand A hand that is one tile away from winning.
A hand with a triplet or quad of dragon tile.
12 Concealed Hand
A hand without exposed melds and completed by seizing the discarded tile.
All
A hand with four sequences and a pair in characters.
A hand with four same tiles, other than a melded quad.
Two Concealed
hand with two concealed triplets or quads.
Concealed
with a concealed quad.
All Simples
consisting of no character 1, 9 and honor tile
A
hand that includes terminals and honors in each meld, including the pair
19 Fully Concealed
Hand
A hand without exposed melds and completed by drawing a tile.
Self Draw
Kongs
A hand exposed quads.
Melded Kong
Last Tile
Winning on a tile that is the last of its kind
Three of
the other
tiles on the table already revealed to all players
Little Three
Winds
A hand with two wind triplets or quads and a wind pair.
All Pungs
A hand with four triplets or quads.
Half Flush A hand consisting of character and honor tiles.
Two Dragon Pungs A hand containing two dragon triplets or quads. Dragon Pung
Two
Concealed Kongs A hand two concealed quads.
{+Concealed Kong 
+}
Melded Hand Every set in the hand must be completed with tiles discarded by other players. Single Wait
Out
Replacement Tile Hand completion with supplemental tile when you melding quad.
Self Draw
Rob Kong
Winning off the tile that opponent adds to a melded triplet(to create a Kong)
Last Tile Last Tile
Claim
Winning off another player on the last tile (of the game).
Pure Straight
A hand with three sequences 123,
[Table]
Seven Pairs
A hand with seven pairs.
Concealed Hand,
Single Wait,
Fully Concealed
Hand.
Pure Triple
Chow
A hand with three identical sequences.
Pure Double
Chow
Pure Shifted
Pungs
A hand with three number triplets or quads with successive numbers.
Pure Triple
Chow
Four Pure
Shifted Chows
Four shifted number sequences, each shifted either one or two numbers up from the last, but not a combination of both.
Pure Shifted
Chows,
Short Straight,
Two Terminal
Chows,
Pure Double
Chow
Three Kongs
A hand with three quad melds.
Two Melded
Kongs, Melded
Kong, Two
Concealed Kongs,
Concealed Kong 43 All Terminals and Honours
A hand consisting of only terminal and honor tiles.
All Pungs,
Outside Hand 44 Heavenly Ready
Hand
A hand that is one tile away from winning at the beginning of the game.
Ready Hand 45 Quadruple Chow
A hand with four identical sequences.
Pure Triple
Chow,
Tile Hog,
Pure Double
Chow,
Two Terminal
Chow,
Pure Shifted
Pungs
Four Pure
Shifted Pungs
A hand with four character triplets or quads with successive numbers.
Pure Shifted
Pungs,
Pure Triple
Chow,
All Pungs
Four Winds
Seven Pairs
Seven pairs hand with four different wind pairs.
Seven Pairs,
Concealed Hand,
Single Wait,
Fully Concealed
Hand 48 Three Dragons
Seven Pairs
Seven pairs hand with three different dragon pairs.
Seven Pairs,
Concealed Hand,
Single Wait,
Fully Concealed
Hand
Little Four
Winds
A hand with three winds be triplets or quads, and the last wind be pair.
Big Three
Winds,
Little Three
Winds
Little Three
Dragons
A hand with two dragons be triplets or quads, and the last dragon be pair.
Two Dragon
Pungs,
Dragon Pung
All Honours
A hand consisting of only honor tiles.
All Terminals
And Honours,
All Pungs,
Four Honour
Pungs,
Outside Hand 52 Four Concealed
Pungs
A hand with four concealed triplets or quads.
Concealed Hand,
All Pungs,
Three Concealed
Pungs,
Two Concealed
Pungs,
Fully Concealed
Hand 53 Pure Terminal
Chows
A hand with two Two Terminal Chows and a pair of number 5 in character.
All Chows,
Seven Pairs,
Full Flush,
Pure Double
Chow,
Two Terminal
Chows
Big Four
Winds
A hand with triplets or quads of all four winds and an pair.
All Pungs,
Little Three
Winds,
Big Three
Winds,
Four Honor
Pungs
Big Three
Dragons
A hand with triplets or quads of all three dragons.
Dragon Pung,
Two Dragon
Pungs
Nine Gates
Collecting number tileswithout melding, and completing with any tile of characters.
Full Flush,
Concealed Hand,
Fully Concealed
Hand
Four Kongs
A hand with four quad melds.
Three Kongs,
Two Melded
Kongs,
Melded Kong,
Single Wait,
Concealed Kong,
Two Concealed
Kongs,
All Pungs
Seven Shifted
Pairs
Seven pairs hand with successive seven numbers in characters.
Seven Pairs,
Single Wait,
Concealed Hand,
Full Flush,
Fully Concealed
Hand
[Table]
Seven
[Table]
Four
[Table]
Seven Pairs,
Three Dragons
Seven Pairs,
Outside Hand
[Table]
Concealed Hand
[Table]
[-All Honours 
-]Fully
[Table]
Hand
[Table]
A. T HE S TATE AND A CTION S PACE
The state space size of 1-on-1 Mahjong, shown as the information set count in Figure 4, is approximately 10 74 .
Yet, the state space of 1-on-1 Mahjong is not as easily abstracted as in poker.
The primary reason is that a single tile difference in the state could signiﬁcantly impact the strategy, e [Table] g., making a legal hand illegal and vice versa. In contrast, states that have similar strength in poker could share a common strategy.
For instance, the optimal preﬂop strategy could be very similar for “Ace-Four" and “Ace-Three" in poker.
Another reason is that a state in 1-on-1 Mahjong is divided into different information groups, as demonstrated in Figure 7(a).
Different information groups have signiﬁcantly different meanings.
For instance, one group denotes the player’s hand, which is invisible to the opponent, while another group denotes the player’s discarded tiles, which are visible to both players.
There are 105 different actions in total, as demonstrated in Table 2.
The number of legal actions in a state is relatively small compared to poker.
Yet, the game length in 1-on-1 Mahjong is larger than that in poker.
Players can decide up to about 40 sequential actions in 1-on-1 Mahjong, whereas most 1-on-1 poker games end within 10 steps.
As a result, the reaching probability of states in 1-onMahjong may vary more signiﬁcantly than that in poker.
B T HE M ODEL D ESIGN OF O UR 1- ON -1 M AHJONG AGENT J UE J ONG
The model of JueJong is an end-to-end neural network that takes all relevant information as input and outputs both the probabilities of all actions and the state value.
This is different from ﬁve separated neural networks in Suphx (Li et al., 2020b), representing ﬁve different types of actions.
Also, we
P i draws a tile
P i discards a tile
P i selects a legal action
Game starts
Is Concealed-Kong?
Deal cards for P 0 , Pi=
P i selects a legal action i = 1-i
Is Kong?
P i discards a tile
P i selects a legal action
Is Concealed-Kong or Add-Kong?
yes
yes
no no
no yes
no no
yes yes yes yes
no no yes
Game ends yes
Is Pong?
Is Chow?
Is Hu? Is Hu?
no
Is Hu?
No tiles?
no
P i draws a tile train JueJong from zero by pure self-play using ACH, while the ﬁve neural networks in Suphx were trained by supervised learning on human data, with only the “Discard” network further enhanced by
RL.
Figure 7 gives an overview of the model design of JueJong.
The marked regions in Figure 7(a) summarize the information an agent can observe.
Regions 1 to 7 are encoded in image-like features, which represent the player’s hand, the player’s Chow, Pong, and Kong, the player’s concealed-Kong, the player’s Discard, the opponent’s Chow, Pong, and Kong, the opponent’s concealed-Kong, and the opponent’s Discard respectively.
A feature map, as shown in Figure 7(b), of height 4 and width 17 is employed, where a 0 or 1 in the i th row and j th column means whether there are i tiles of the j th type for the input set of tiles.
Therefore, a single or two successive convolutional layers with 3 × 3 kernels efﬁciently derive row and column combinations.
Note that, in Suphx (Li et al., 2020b), the feature maps are one-dimensional vectors, where 1 ×kernels are used.
The black tile in Figure 7(b) is designed exclusively for Region 6 to indicate how many concealed-Kongs the opponent has.
For discarded tiles in Region 4 or 7 , 24 such feature maps are used to indicate the latest 24 discarded tiles in order, with one feature map encoding one discard tile.
All the feature maps are concatenated in the channel dimension, and therefore the ﬁnal image-like features are in the shape of 4 × 17 × 53 ( h × w × c ).
Regions 8 to 11 are encoded in one-hot features, which represent the player’s position ( 0 or 1 ), the is _ ready state of both players, and the bonus tiles of both players.
Additionally, the last action (not shown as a region in Figure 7(a)) for each player is encoded in a one-hot feature vector as well. [Figure]
We apply residual blocks (He et al., 2016) to transform the image-like features.
There are totally 3 stages with 3 residual blocks and 1 transition layer in each stage, as shown in Figure 7(c).
Each block contains two convolutional layers with kernel size 3 × 3 and number of output feature maps 32 , 64 , and 128 for each stage.
The transition layer is point-wise convolution 1 × 1 , and the one in the ﬁnal stage scales the input to 32 channels.
Subsequently, the transformed image-like features are reshaped to a vector, which is concatenated with the one-hot feature vectors.
A fully-connected layer (dimension 1024 ) is then used to transform the concatenated feature vector, and two branches with two fully-connected layers (dimension 512 × 512 ) each output the action probabilities and the state value respectively.
Besides, we apply batch normalization (Ioffe & Szegedy, 2015) and ReLU
C T HEORETICAL P ROPERTIES OF ACH C. P ROOF FOR T HEOREM
In order to prove Theorem 1, we ﬁrst prove the following Lemma.
Lemma 1.
For a weighted CFR, if w t ( s ) ∈ [ w l ( s ) , w h ( s )] , 0 < w l ( s ) ≤ w h ( s ) ≤ 1 , then max a ∈A (
[Equation]
Proof.
First, for any state s ∈ S and action a ∈ A ( s ) , we have
[Equation]
So,
[Equation]
In other words
[Equation]
[Equation]
Then we can prove the Theorem:
Proof.
For ACH, the policy at iteration t is generated according to Hedge:
[Equation]
) where
[Equation]
On the other hand
[Equation]
weighted CFR with Hedge generates policy at iteration t according to
[Equation]
where
[Equation]
So, when w k ( s ) =
f µ k p ( s ) ≥ w l ( s ) > 0 for any s ∈ S and k > 0 , ACH and weighted CFR with
Hedge generates the same policy at each iteration, and therefore ACH is equivalent to weighted CFR with Hedge.
Then [Equation] we can prove the convergence property of ACH.
Let
[Equation]
We have
[Equation]
On the other hand [Equation] for each k = 1 , . . . , t ,
[Equation]
Since
[Equation]
Note that r ck
[Equation]
s, a
[Equation]
a
[Equation]
Therefore,
[Equation]
So,
[Equation]
According to Lemma
[Equation]
When η ( s ) =
(cid:112) 8 ln |A
[Equation] s ) [Equation] / { [ w h ( s )] 2 ∆ 2 ( s ) T } and w t ( s ) ∈ [ w l ( s ) , w h ( s )]
⊂ (0 , 1] , t = 1 , . .
, T , we have
[Equation]
According to Theorem 2 in Zinkevich et al.
, the total regret
[Equation]
As a result [Equation] according to the folk theorem in Zinkevich et al.
(2008), the average strategy converges to an (cid:15) -equilibrium, where
[Equation]
C. P ROOF FOR C OROLLARY
Proof.
When the behavioral policy µ p,k of each player p ∈ P across iterations k > 0 is constant, the reaching probability f µ k p ( s ) of any state s ∈ S is also constant.
Assume f µ k p ( s ) =
w ( s ) , then,
[Equation]
In other words, R at − 1 ( s, a ) is equal to the cumulative counterfactual regret scaled by a time-invariant weight w ( s ) .
Hence, the policy at iteration t is
[Equation]
where the new η (cid:48) ( s ) is set to (cid:112) 8 ln |A ( s ) | / [∆ 2 ( s ) T ] .
So, when µ p,k is constant, ACH is equivalent to CFR with Hedge when y ( a | s ; θ t ) is sufﬁciently close to R at ( s, a ) .
Furthermore, since w l ( s ) =
f µ k p ( s )
=
w h ( s ) , the second term in (25) disappears, i.e.,
[Equation]
As a result, the exploitability bound of CFR with Hedge is recovered.
C
C ONTINUOUS -T IME D YNAMICS OF ACH
We illustrate the convergence dynamics of ACH in the game of Rock-Paper-Scissors, in comparison with other actor-critic methods (PPO/A2C, RPG, and NeuRD), in Figure 8.
S
R
P 0 .
ACH
S
R
P 0 .
PPO
S
R
P 0 .
RPG
S
R
P 0 .
NeuRD
D A P ARALLEL I MPLEMENTATION OF ACH
In order to utilize distributed clusters, we provide a parallel implementation of ACH, which employs a framework of decoupled acting and learning, similar to IMPALA (Espeholt et al., 2018).
The parallel implementation of ACH maintains a policy net y ( a | s ; θ ) and a value net V ( s ; ω ) , where θ and ω share a large portion of parameters (see Figure 7).
Both players use the same θ and ω .
We do not use an additional time-invariant behavioral policy for sampling actions.
Instead, we use the current policy π t . As a result, we can use the same samples to train both the value net and the policy net. η ( s ) is incorporated into the learned target value in the policy net, so the policy π ( a | s ) is obtained by directly softmaxing on y ( a | s ; θ ) . The advantage A ( s, a ) is estimated by the GAE( λ ) (Schulman et al., 2016), using sampled rewards and V ( s ; ω ) , for only sampled states and actions.
The value and policy nets are updated as soon as a mini batch of samples is available.
In other words, we update θ and ω once using a single mini batch at each iteration.
As a result, the policy loss reduces to L π = η ( s ) y ( a | s ; θ ) π old ( a | s )
A ( a, s ) , where 1 π old ( a | s ) accounts for the fact that a was sampled using π old ( a | s ) .
The parallel implementation handles asynchronous training with the importance ratio clipping of PPO (Schulman et al., 2017).
To
Initialize the policy and critic parameters: θ and ω .
Start multiple actor and learner threads in parallel.
Actors : while true do
Fetch the latest model from the learners.
Generate samples via self-play in the form: [ a, s, A ( s, a ) , G, π old ( a | s )] .
Send the samples to the replay buffer.
Learners : for t ∈ 1 , 2 , 3 , ... do
Fetch a mini batch of samples from the replay buffer.
L sum = 0 .
for each sample [ a, s, A ( s, a ) , G, π old ( a | s )] ∈ the mini batch do
Let c be the clipping mask:
[Equation]
Update θ and ω once using gradient on L sum .
avoid numerical issues, the mean is subtracted from the policy output, which is then clipped within a range.
The pseudocode of the parallel implementation is given in Algorithm 2.
We employ an entropy loss to encourage exploration during training and hopefully the convergence of current strategy to a NE (Srinivasan et al., 2018).
The parallel implementation updates θ and ω simultaneously, and the overall loss is:
[Equation]
Theoretically, y ( a | s (cid:48) ; θ ) for unsampled states s (cid:48) should also be trained with the target y
( a | s (cid:48) ; θ old ) .
Yet, in the parallel implementation, we only update θ once using a single mini-batch at each iteration.
Therefore, θ is equal to θ old before the update.
In other words, the policy loss for non-sampled states is 0 in the parallel implementation of ACH.
E T HE E FFECT OF THE B EHAVIOR P OLICY ON ACH IN FHP
As we noted in the paper, the behavior policy µ p,t in ACH could be set to either the current policy π p,t or simply a uniform sampling policy. As Corollary 1 states, a tighter bound on the exploitability of the average policy of ACH can be obtained, if µ p,t is stationary over iterations. However, since the average strategy is difﬁcult to obtain in ACH and we have decided to use the current policy (trained with an entropy regularization) for evaluation, the effect of the behavior policy is unclear from a theoretical perspective.
Hence, we conduct experiments to investigate this.
In Figure 9, we compare the exploitability of the current policy of ACH on FHP, by setting the behavior policy in ACH to either the current policy or a uniform sampling policy.
From the comparison, it seems that the performance of the current policy of ACH is not sensitive to the choices of behavior policies.
The reason might be that the additional entropy loss forces the current strategy to be stable and prone to a uniform random strategy.
In other words, when using the current strategy as the behavior policy, the behavior policy tends to be stable.
F A DDITIONAL R ESULTS ON S MALL IIG B ENCHMARKS IN O PEN S PIEL
We further evaluate ACH and compare it with A2C, RPG, and NeuRD on three benchmarks from
OpenSpiel: Kuhn poker, Leduc poker, and Liar’s Dice.
All the experiments were run single-threaded
[Figure]
on a 2.24GHz CPU.
We use the network architecture in OpenSpiel, which has a 128-neurons fullyconnected layer followed by ReLU and two separate linear layers for the policy and the state/action value.
For A2C, RPG, and NeuRD, we use the default hyper-parameters in OpenSpiel.
ACH shares most of the hyper-parameters with A2C.
All the hyper-parameters are listed in the Appendix G.3.
The exploitability of an agent is exactly calculated using tools in OpenSpiel.
For each method, we compute the exploitability of each agent every 1 e 5 training steps, the results of which are plotted in
Figure 10.
Clearly, ACH converges signiﬁcantly faster and achieves a lower exploitability than other methods across the three benchmarks.
There is still some gap between 0 and the exploitability ACH converges to.
This may due to the neural network approximation error and the fact that we use the current strategy instead of the average strategy for the evaluation.
A2C has the worst performance, since it is designed for single-agent environments.
Moreover, the superiority of ACH is most signiﬁcant on the Liar’s Dice benchmark, which is the most complex one of the three benchmarks.
[Figure]
E x p l
o i t a b ili t y
A2CRPGNeuRDACH(Ours) (b) Leduc poker 0.
1.0 Training Steps 1e7 0.
E x p l
o i t a b ili t y
A2CRPGNeuRDACH(Ours)
[Figure]
As a complement, we present the head-to-head performance of A2C, RPG, NeuRD, and ACH on the three benchmarks in OpenSpiel: Kuhn poker, Leduc poker, and Liars Dice.
As demonstrated in Table 4, the agent of ACH won all other agents across all the three benchmarks.
The superior performance of ACH is most signiﬁcant on the Liars Dice benchmark, which is the most complex one of the three benchmarks.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
G H YPER - PARAMETERS G. H YPER - PARAMETERS FOR THE 1- ON -1 M AHJONG E XPERIMENT
We used the Adam optimizer (Kingma & Ba, 2014) for all experiments on the 1-on-1 Mahjong benchmark.
We performed a mild hyper-parameter search on PPO and used the best setting for the shared hyper-parameters of all methods (PPO, RPG, NeuRD, and ACH).
For each setting in the hyper-parameter search, we conducted 3 independent runs with different random seeds.
Tablegives an overview of hyper-parameters for each method.
Also, we report the performance of current strategy, instead of the average strategy, for each method.
[Table]
G. H YPER - PARAMETERS FOR THE FHP E XPERIMENT
We used the Adam optimizer Kingma & Ba (2014) for all experiments on FHP.
All the experiments were run multi-threaded and synchronously using 10 2.24GHz CPUs.
We update the neural networks of ACH and other methods (A2C, RPG, and NeuRD)
every 1000 episodes, with a batch consisting all the samples collected in the 1000 episodes.
Since the average length of FHP is around 2, the batch size is roughly 2000.
We performed a mild hyper-parameter search for ACH, A2C, RPG, and NeuRD, as shown in Table 6.
Note that we do not need to clip the advantages in a synchronous method, so the ratio clip hyper-parameter is not needed.
For FHP, we introduced a reward normalizer to normalize the rewards as the returns in FHP are in [ − 700 , 700] .
Besides, we found that A2C, RPG, and NeuRD are very sensitive to the entropy coefﬁcient, and we had to use a larger entropy coefﬁcient in these algorithms than ACH.
For OS-DCFR and DREAM, we use the same hyper-parameters presented in
Steinberger et al.
All the hyper-parameters are given in Table 6.
G. H YPER - PARAMETERS FOR THE E XPERIMENT ON O PEN S PIEL
We used stochastic gradient descent with constant learning rate for all experiments on benchmarks from OpenSpiel.
For A2C and RPG, we used the default implementations and hyper-parameters in OpenSpiel.
NeuRD was originally implemented using counterfactual regrets in OpenSpiel.
We re-implemented NeuRD using predicted advantages and set the shared hyper-parameters of NeuRD identical to those in RPG.
All methods employ an entropy loss to encourage exploration during training and hopefully the convergence of current strategy to a NE.
Also, we report the performance of current strategy, instead of the average strategy, for each method.
In OpenSpiel, the value parameters are updated separately and more frequently compared to the policy parameters for A2C, RPG, and NeuRD.
Yet, the value loss and the policy loss are combined in
ACH, and all parameters are updated simultaneously, as illustrated in Algorithm 2.
Also note that, in [Table] [Table] the single-threaded OpenSpiel training environment, the actor and the learner run in sequence.
As a result, π ( a | s ; θ ) is always equal to π old ( a | s ) in Algorithm 2.
We performed a mild hyper-parameter search for ACH, which is illustrated in Table 7.
For each setting in the hyper-parameter search, we conducted 4 independent runs with different random seeds.
The ﬁnal hyper-parameters used for each method are listed in Table 8, with 3 additional hyper-parameters of ACH listed in the “Best” column in Table
[Table]
H T HE R ELATIONSHIP B ETWEEN ACH AND S UPHX
Recently, Suphx (Li et al., 2020b) has achieved stunning performance on Japanese Riichi Mahjong.
The development of Suphx is enabled by a novel integration of existing supervised learning and RL methods in addition to some newly developed techniques.
Three new techniques were introduced in Suphx: global reward prediction, oracle guiding, and run-time policy adaptation.
The global reward prediction technique is to handle the multi-round game situation, which is irrelevant to our 1-on-1 Mahjong setting (only one round per game in our test setting).
The oracle guiding technique decays the invisible feature during training, which is of independent interest in dealing with imperfectinformation.
The run-time policy adaptation technique adapts the trained policy at test time, and this can be combined with ACH, which is a training algorithm.
In summary, Suphx is more of a novel system than a new algorithm.
For this reason, we did not implement and compare Suphx with ACH in our 1-on-1 Mahjong experiment.
Nonetheless, the oracle guiding technique may be complementary to ACH in handling imperfect-information, and we will investigate this in our future work.
[Table]