D OUBLE N EURAL C OUNTERFACTUAL R EGRET M INIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Counterfactual Regret Minimization (CRF) is a fundamental and effective technique for solving Imperfect Information Games (IIG).
However, the original
CRF algorithm only works for discrete state and action spaces, and the resulting strategy is maintained as a tabular representation.
Such tabular representation limits the method from being directly applied to large games and continuing to improve from a poor strategy proﬁle.
In this paper, we propose a double neural representation for the imperfect information games, where one neural network represents the cumulative regret, and the other represents the average strategy.
Furthermore, we adopt the counterfactual regret minimization algorithm to optimize this double neural representation.
To make neural learning efﬁcient, we also developed several novel techniques including a robust sampling method, mini-batch Monte Carlo Counterfactual Regret Minimization (MCCFR) and
Monte Carlo Counterfactual Regret Minimization Plus (MCCFR+) which may be of independent interests.
Experimentally, we demonstrate that the proposed double neural algorithm converges signiﬁcantly better than the reinforcement learning counterpart.
I NTRODUCTION
In Imperfect Information Games ( IIG ), a player only has partial access to the knowledge of her opponents before making a decision.
This is similar to real-world scenarios, such as trading, trafﬁc routing, and public auction.
Thus designing methods for solving IIG is of great economic and societal beneﬁts.
Due to the hidden information, a player has to reason under the uncertainty about her opponents’ information, and she also needs to act so as to take advantage of her opponents’ uncertainty about her own information.
Nash equilibrium is a typical solution concept for a two-player extensive-form game.
Many algorithms have been designed over years to approximately ﬁnd Nash equilibrium for large games.
One of the most effective approaches is CFR (Zinkevich et al., 2007).
In this algorithm, the authors proposed to minimize overall counterfactual regret and prove that the average of the strategies in all iterations would converge to a Nash equilibrium.
However, the original CFR only works for discrete state and action spaces, and the resulting strategy is maintained as a tabular representation.
Such tabular representation limits the method from being directly applied to large games and continuing to improve if starting from a poor strategy proﬁle.
To alleviate CFR’s large memory requirement in large games such as heads-up no-limit Texas
Hold’em, Moravcik et al.
(2017) proposed a seminal approach called DeepStack which uses fully connected neural networks to represent players counterfactual values and obtain a strategy online as requested.
However, the strategy is still represented as a tabular form and the quality of this solution depends a lot on the initial quality of the counterfactual network.
Furthermore, the counterfactual network is estimated separately, and it is not easy to continue improving both counterfactual network and the tabular strategy proﬁle in an end-to-end optimization framework.
Heinrich et al.
; Heinrich & Silver (2016) proposed end-to-end ﬁctitious self-play approaches (XFP and NFSP respectively) to learn the approximate Nash equilibrium with deep reinforcement learning.
In a ﬁctitious play model, strategies are represented as neural networks and the strategies are updated by selecting the best responses to their opponents’ average strategies.
This approach is advantageous in the sense that the approach does not rely on abstracting the game, and in theory, the strategy should continually improve as the algorithm iterates more steps.
However, these methods do not explicitly take into account the hidden information in a game, because they are optimized based on the transition memory and the reward of the intermediate node is the utility of the game rather than the counterfactual value which consider the distribution of hidden variables (opponent’s private information).
In experiments for games such as Leduc Hold’em, these methods converge slower than tabular based counterfactual regret minimization algorithms.
Waugh et al.
(2015) used handcraft features of the information sets to estimates the counterfactual regret.
However, it need traverse the full game tree which is infeasible in large games.
Thus it remains an open question whether the purely neural-based end-to-end approach can achieve comparable performance to tabular based CFR approach.
In the paper, we partially resolve this open question by designing a double neural counterfactual regret minimization algorithm which can match the performance of tabular based counterfactual regret minimization algorithm.
We employed two neural networks, one for the cumulative regret, and the other for the average strategy.
We show that careful algorithm design allows these two networks to track the cumulative regret and average strategy respectively, resulting in a converging neural strategy.
Furthermore, in order to improve the convergence of the neural algorithm, we also developed a new sampling technique which has lower variance than the outcome sampling, while being more memory efﬁcient than the external sampling.
In experiments with One-card poker and a large Leduc Hold’em containing more than 10 7 nodes, we showed that the proposed double neural algorithm has a strong generalization and compression ability even though only a small proportion of nodes are visited in each iteration.
In addition, this method can converge to comparable results produced by its tabular counterpart while performing much better than deep reinforcement learning method.
The current results open up the possibility for a purely neural approach to directly solve large IIG.
B ACKGROUND
In this section, we will introduce some background on IIG and existing approaches to solve them.
2. R EPRESENTATION OF E XTENSIVE -F ORM G AME
We deﬁne the components of an extensive-form game following Osborne & Ariel (1994) (page 200 ∼ 201 ).
A ﬁnite set N = { 0 , 1 , ..., n − 1 } of players .
Deﬁne h vi as the hidden variable of player i in IIG, e.g. , in poker game h vi refers to the private cards of player i .
H refers to a ﬁnite set of histories.
Each member h
= ( h vi ) i =0 , 1 ,...,n − 1 ( a l ) l =0 ,...,L − 1 = h
v 0 h v 1 ...h vn − 1 a 0 a 1 ...a L − 1 of H denotes a possible history (or state), which consists of each player’s hidden variable and L actions taken by players including chance.
For player i , h also can be denoted as h vi h v − i a 0 a 1 ...a L − 1 , where h v − i refers to the opponent’s hidden variables.
The empty sequence ∅ is a member of H .
h j (cid:118) h denotes h j is a preﬁx of h , where h j = ( h vi ) i =0 , 1 ,...,n − 1 ( a l ) l =1 ,...,L (cid:48) − 1 and 0 < L (cid:48) <
L .
Z ⊆
H denotes the terminal histories and any member
z ∈ Z is not a preﬁx of any other sequences.
A ( h ) =
{ a : ha ∈ H } is the set of available actions after non-terminal history
h ∈ H \ Z .
A player function P assigns a member of N ∪ { c } to each non-terminal history, where c denotes the chance player id, which usually is -1.
P ( h ) is the player who takes an action after history h .
I i of a history { h ∈ H :
P ( h ) =
i } is an information partition of player i .
A set I i ∈
I i is an information set of player
i
and I i ( h ) refers to information set
I i at state h .
Generally, I i could only remember the information observed by player i including player i (cid:48) s hidden variable and public actions.
Therefore
I i indicates a sequence in IIG, i.e. , h vi a 0 a 2 ...a L − 1 .
For I i ∈
I i we denote by A ( I i ) the set
A ( h ) and by P ( I i ) the player P ( h ) for
any h ∈
I i .
For each player i ∈ N a utility function u
i ( z ) deﬁne the payoff of the terminal state z .
A more detailed explanation of these notations and deﬁnitions is presented in section B.
S TRATEGY AND N ASH EQUILIBRIUM
A strategy proﬁle σ
= { σ i | σ i ∈ Σ
i , i ∈ N } is a collection of strategies for all players, where Σ i is the set of all possible strategies for player i .
σ − i refers to strategy of all players other than player i .
For play i ∈ N the strategy σ i ( I i ) is a function, which assigns an action distribution over
A ( I i ) to information set I i .
σ i ( a | h ) denotes the probability of action a taken by player i ∈ N ∪ { c } at state h .
In IIG, ∀ h 1 , h 2 ∈ I i , we have I
i
= I
i ( h 1 )
=
I i ( h 2 ) , σ i ( I i )
=
σ i ( h 1 ) = σ i ( h 2 ) , σ i ( a | I i )
=
σ i ( a |
h 1 ) =
σ i ( a | h 2 ) .
For iterative method such as CFR, σ t refers to the strategy proﬁle at t -th iteration.
The state reach probability of history h is denoted by π σ ( h ) if players take actions according to σ .
For an empty sequence π σ ( ∅ ) = 1 .
The reach probability can be decomposed into π σ ( h ) =
(cid:81) i ∈ N ∪{ c } π σi ( h ) =
π σi ( h ) π σ − i ( h ) according to each player’s contribution, where π σi ( h ) =
(cid:81) h (cid:48) a (cid:118) h,P ( h (cid:48) )
=
P ( h ) σ i ( a | h (cid:48) ) and π σ − i ( h ) =
(cid:81) h (cid:48)
[Table]
h,P ( h
[Table]
) (cid:54) =
P ( h ) σ − i ( a | h (cid:48) ) .
The information set reach probability of I i is deﬁned as π σ ( I i ) =
(cid:80) h ∈
I i π σ ( h ) .
If h (cid:48) (cid:118) h , the interval state reach probability from state h (cid:48) to h is deﬁned as π σ ( h (cid:48) , h ) , then we have π σ ( h (cid:48) , h ) =
π σ ( h ) /π
σ ( h (cid:48) ) .
π σi ( I i ) , π σ − i ( I i ) , π σi ( h (cid:48) , h ) , and π σ − i ( h (cid:48) , h ) are deﬁned similarly.
2. C OUNTERFACTUAL R EGRET M INIMIZATION
In large and zero-sum IIG, CFR is proved to be an efﬁcient method to compute Nash equilibrium (Zinkevich et al., 2007; Brown & Sandholm, 2017).
We present some key ideas of this method as follows.
Lemma 1: The state reach probability of one player is proportional to posterior probability of the opponent’s hidden variable, i.e., p ( h v − i
| I i )
∝ π σ − i
( h ) , where h vi and I
i indicate a particular h .
(see the proof in section G.1)
For player i and strategy proﬁle σ , the counterfactual value (CFV) v σi ( h ) at state h is deﬁne as
[Equation]
where u (cid:48)
i ( z ) =
π σ − i ( z ) u
i ( z ) is the expected reward of player i with respective to the approximated posterior distribution of the opponent’s hidden variable.
The action counterfactual value of taking action a is v σ i ( a | h )
=
v σ i ( ha ) and the regret of taking this action is r σ i ( a | h )
=
v σ i ( a | h ) − v σ i ( h ) .
Similarly, the CFV of information set I
i is v σi ( I i ) =
(cid:80) h ∈
I i v σi ( h ) and the regret is r σi ( a |
I i ) =
(cid:80) z
∈ Z,ha
[Equation] [Equation]
i π
[Equation]
ha, z
[Equation]
i
[Equation]
z ) −
[Equation]
z ∈ Z,h (cid:118) z,h ∈
I
i π σi ( h, z ) u (cid:48)
i ( z ) .
Then the cumulative regret of action a after T iterations is
[Equation]
where R
i ( a | I i ) = 0 .
Deﬁne R T, + i ( a | I i ) =
max( R Ti ( a | I i ) , 0) , the current strategy (or behavior strategy) at T + 1 iteration will be updated by
[Equation]
otherwise
[Equation]
The average strategy ¯ σ iT from iteration 1 to T is deﬁned as:
[Equation]
where π σ t i ( I i ) denotes the information set reach probability of I
i at t -th iteration and is used to weight the corresponding current strategy σ ti ( a | I i ) .
Deﬁne
[Equation]
I i ) σ ti
[Equation]
a | I i ) as the additional numerator in iteration t , then the cumulative numerator can be deﬁned as
[Equation]
where S 0 ( a | I i ) = 0 .
2. M ONTE C ARLO CFR
When solving a game, CFR needs to traverse the entire game tree in each iteration, which will prevent it from handling large games with limited memory.
To address this challenge, Lanctot et al.
(2009) proposed a Monte Carlo CFR to minimize counterfactual regret.
Their method can compute an unbiased estimation of counterfactual value and avoid traversing the entire game tree.
Since only subsets of all information sets are visited in each iteration, this approach requires less memory than standard CFR.
Deﬁne Q = { Q 1 , Q 2 , ..., Q m } , where Q j ∈ Z is a block of sampling terminal histories in each iteration, such that Q j spans the set Z .
Generally, different Q j may have an overlap according to the specify sampling schema.
Speciﬁcally, in the external sampling and outcome sampling, each block Q j ∈ Q is a partition of Z .
Deﬁne q Q j as the probability of considering block Q j , where (cid:80) mj
=1 q Q j = 1 .
Deﬁne q ( z ) =
(cid:80)
j : z ∈ Q
j q Q j as the probability of considering a particular terminal history z .
Speciﬁcally, vanilla CFR is a special case of MCCFR, where Q = { Z } only contain one block and q Q 1 = 1 .
In outcome sampling, only one trajectory will be sampled, such that ∀ Q j ∈ Q ,
| Q j | = 1 and |Q
j | = | Z | .
For information set I i , a sample estimate of counterfactual value is ˜ v
σi ( I i | Q j ) =
(cid:80) h ∈
I i ,z ∈ Q j ,h (cid:118)
z
1 q ( z ) π σ − i ( z )
π σi ( h, z ) u i ( z ) .
Lemma 2:
The sampling counterfactual value in MCCFR is the unbiased estimation of actual counterfactual value in CFR.
E j ∼ q Qj [˜ v σi ( I i | Q j )]
= v σi ( I i ) (Lemma 1, Lanctot et al.
Deﬁne σ rs as sampling strategy proﬁle , where σ rsi is the sampling strategy for player i and σ rs − i are the sampling strategies for players expect i .
Particularly, for both external sampling and outcome sampling proposed by (Lanctot et al., 2009), σ rs − i =
σ − i .
The regret of the sampled action a ∈ A ( I i ) is deﬁned as
[Equation]
where u rsi ( z ) =
u i ( z ) π σrsi ( z ) is a new utility weighted by 1 π σrsi ( z ) .
The sample estimate for cumulative regret of action a after T iterations is ˜ R Ti (( a |
I i ) | Q j ) =
˜ R T − 1 i (( a | I i ) | Q j ) + ˜ r σ T i (( a | I i ) | Q j ) with ˜ R
i (( a | I i ) | Q j ) = 0 .
D OUBLE N EURAL C OUNTERFACTUAL
R EGRET M INIMIZATION 𝐼𝐼𝐼𝐼 4 𝑧𝑧𝑧𝑧𝑧𝑎𝑎𝑎𝑎𝑎𝑎𝑎 7 𝑎𝑅 𝑖 𝑡−1 (𝑎|𝐼 𝑖 )
+ 𝑅 𝑖𝑡 (𝑎|𝐼 𝑖 )
𝑠 𝑖𝑡 (𝑎|𝐼 𝑖 ) 𝑆 𝑖𝑡−1 (𝑎|𝐼 𝑖 )
+ 𝑆 𝑖𝑡 (𝑎|𝐼 𝑖 ) 𝑞𝑢𝑒𝑟𝑦 𝑞𝑢𝑒𝑟𝑦
Regret Matching 𝐼𝐼𝐼𝐼 4 𝑧𝑧𝑧𝑧𝑧𝑎𝑎𝑎𝑎𝑎𝑎𝑎 7 𝑎𝑠 𝑖𝑡 (𝑎|𝐼 𝑖 )
[Figure]
Matching all 𝑖𝑛𝑓𝑜𝑟𝑚𝑎𝑡𝑖𝑜𝑛𝑠𝑒𝑡𝑠 all 𝑖𝑛𝑓𝑜𝑟𝑚𝑎𝑡𝑖𝑜𝑛𝑠𝑒𝑡𝑠
RegretSumNetwork
[Figure]
𝑠𝑎𝑚𝑝𝑙𝑒𝑑𝑖𝑛𝑓𝑜𝑟𝑚𝑎𝑡𝑖𝑜𝑛𝑠𝑒𝑡𝑠
[Figure]
𝑟 𝑖 𝜎 𝑡
[Figure]
𝑖𝜎 𝑡 (
[Figure]
𝐼 𝑖
[Figure]
In this section, we will explain our double neural CFR algorithm, where we employ two neural networks, one for the cumulative regret, and the other for the average strategy.
As shown in Figure 1 (A), standard CFR-family methods such as CFR (Zinkevich et al., 2007), outcome-sampling MCCFR, external sampling MCCFR
(Lanctot et al., 2009), and CFR+ (Tammelin, 2014) need to use two large tabular-based memories M R and M S to record the cumulative regret and average strategy for all information sets.
Such tabular representation makes these methods difﬁcult to apply to large extensive-form games with limited time and space (Burch, 2017).
In contrast, we will use two deep neural networks to compute approximate Nash equilibrium of IIG as shown in Figure 1 (B).
Different from NFSP, our method is based on the theory of CFR, where the ﬁrst network is used to learn the cumulative regret and the other is to learn the cumulative numerator of the average strategy proﬁle.
With the help of these two networks, we do not need to use a large memory to save the key information of the entire game tree.
In practice, the proposed double neural method can achieve a lower exploitability with fewer iterations than NFSP.
In addition, we present experimentally that our double neural CFR can also continually improve after initialization from a poor tabular strategy.
3. O VERALL F RAMEWORK
An algorithm in the CFR framework needs to be able to answer two queries:
what is the current strategy σ t +1 ( a | I i ) for iteration t + 1 ; 2.
and what is the average strategy ¯ σ ti ( a | I i ) after t iterations; ∀ i ∈ N, ∀ I i ∈
I i , ∀
a ∈ A ( I i ) , ∀ t ∈ [1 , T ] .
Thus, our neural networks are designed to address the
needs
for these two queries
respectively.
For the ﬁrst query .
According to Eq.
(3), current strategy σ t +1 ( a | I i ) is computed by the cumulative regret R t ( a | I i ) .
Given information set I i and action a , we design a neural network
RegretSumNetwork(RSN)
R ( a, I i | θ t R ) to learn R t ( a | I i ) , where θ t R is the parameter in the network at t -th iteration.
As shown Figure 1 (b), deﬁne memory M R =
{ ( I i , ˜
r
σ t i (( a | I i ) | Q j )) |∀ i ∈ N, ∀ a ∈
A ( I i ) , h ∈
I
i , h (cid:118) z, z ∈ Q j } .
Each member of M R is the visited information set I
i and the corresponding regret ˜ r σ t i (( a | I i ) | Q j ) , where Q j is the sampled block in t -th iteration.
According to Eq.
(2), we can estimate R ( a, I i | θ t +1 R ) using the following optimization:
[Equation]
For the second query .
According to Eq.
(4), the approximate Nash equilibrium is the weighted average of all previous strategies over T iterations.
We only need to track the numerator in Eq.
(4) since the denominator is used to normalize the summation.
Similar to the cumulative regret, we employ another deep neural network AvgStrategyNetwork(ASN) to learn the cumulative numerator of the average strategy.
Deﬁne M S =
{ ( I i , π σ
[Equation]
i ( I i ) σ ti (
[Equation]
|
[Equation]
)) |
[Equation]
N
[Equation]
A (
[Equation] [Equation]
h ∈
[Equation]
z, z ∈ Q j } .
Each member of M S is the visited information set I i and the value of π σ t
i ( I i ) σ ti ( a | I i ) , where Q j is the sampled block in t -th iteration.
Then the parameter θ t +1 S can estimated by the following optimization:
[Equation]
Relation between CFR, MCCFR and our double neural method .
As shown in Figure 1, these three methods are based on the CFR framework.
The CFR computes counterfactual value and regret by traversing the entire tree in each iteration, which makes it computationally intensive to be applied to large games directly.
MCCFR samples a subset of information sets and will need less computation than CFR in each iteration.
However, both CFR and MCCFR need two huge memories to save the cumulative regrets and the numerators of average strategy for all information sets after multiple iterations, which prevents these two methods to be used in large games directly.
The proposed neural method keeps the beneﬁt of MCCFR yet without the need for a large memory.
Remark 1: Deﬁne ∆ = max
[Equation]
i , [Equation] − [Equation]
− [Equation] | [Equation]
∀
[Equation] ∈ I , [Equation] ) , t ∈ [1 , T ] , we have R Ti ( a | I i ) ≤ ∆ (cid:112) | A | T according to the Theorem 6 in (Burch, 2017), where | A | = max I i ∈I
| A ( I i ) | .
As the increasing of the number of iterations T , the value of R Ti ( a | I i ) will become increasingly large, which makes neural network difﬁcult to learn.
In practice, we can use a normalization factor of √ T and update the cumulative regret by
[Equation]
where R (cid:48)
i ( a | I i ) = 0 .
Remark 2:
In each iteration, only a small subset of information sets are sampled, which may lead to the neural networks forgetting values for those unobserved information sets.
To avoid such catastrophic forgetting, we used the neural network parameters from previous iterations as initialization, which gives an online learning/adaptation to the update.
Furthermore, due to the generalization ability of the neural networks, even samples from a small number of information sets are used to update the new neural networks, we ﬁnd that the newly updated neural networks can produce very good value for the cumulative regret and the strategy mixture.
Remark 3:
The optimization problem for the double neural networks is different from that in
DQN (Mnih et al., 2015), where the target network is ﬁxed for several steps and only one step of gradient descent is performed.
In our setting, both RSN and ASN perform several steps of gradient descent with stochastic mini-batch samples.
Furthermore, in DQN, the Q-value for the greedy action is used in the update, while in our setting, we do not use greedy actions.
Algorithm E gives further details on how to optimize the objectives in Eq.
(7) and Eq.
3. R ECURRENT N EURAL N ETWORK R EPRESENTATION FOR I NFORMATION S ET
In order to deﬁne our R and S network, we need to represent the information set I i ∈
I in extensive-form games.
In such games, players take action in alternating fashion and each player makes a decision according to the observed history.
Because of the dynamic sequence, in this paper, we model the behavior history as a recurrent neural network and each action in the sequence
[Figure]
c
[Figure] [Figure]
[Figure] [Figure]
+ e
[Figure] [Figure]
[Figure]
a a a a
[Figure]
corresponds to a cell in RNN.
This architecture is different from the one in DeepStack, which used a fully connected deep neural network to estimate counterfactual value.
Figure 2 (A) provides an illustration of the proposed deep sequential neural network representation for information sets.
Hochreiter & Schmidhuber (1997) proposed a long short-term memory method (LSTM) which is suitable for sequential decision-making problem.
Furthermore, different position in the sequence may contribute differently to the decision making, we will add an attention mechanism (Desimone & Duncan, 1995; Cho et al., 2015) to the LSTM architecture to enhance the representation.
For example, the player may need to take a more aggressive strategy after beneﬁcial public cards are revealed.
Thus the information, after the public cards are revealed may be more important.
In practice, we ﬁnd this kind of architecture can help us obtain more a high convergence rate than the original LSTM.
Section D provides further details about the proposed architecture.
In the experiment, we also explore other neural architectures, such as LSTM and GRU variants (Cho et al., 2014).
3. C ONTINUAL I MPROVEMENT
With the proposed framework of double neural CFR, it is easy to initialize the neural networks from an existing strategy proﬁle based on the tabular representation or neural representation.
For information set I i and action a , in an existing strategy proﬁle, deﬁne R (cid:48)
i ( a | I i ) as the cumulative regret and S (cid:48) ( a | I i ) as the cumulative numerator of average strategy.
We can clone the cumulative regret for all information sets and actions by optimizing
[Equation]
Similarly, the parameters θ ∗S for cloning the cumulative numerator of average strategy can be optimized in the same way.
Based on the learned θ ∗R and θ ∗S , we can warm start the double neural networks and continually improve beyond the tabular strategy proﬁle.
Remark:
Generally, Eq.
(10) is an idea of behavior cloning algorithm.
Clone a good initialization, and we then continuously update the two neural networks.
In the large extensive game, the initial strategy is obtained from an abstracted game which has a manageable number of information sets.
The abstracted game is generated by domain knowledge, such as clustering similar hand strength cards into the same buckets.
3. O VERALL A LGORITHM
Algorithm 1 provides a summary of the proposed double neural counterfactual regret minimization algorithm.
In the ﬁrst iteration, if the system warm starts from tabular based CFR or MCCFR methods, the techniques in section 3.3 will be used to clone the cumulative regrets and strategy.
If there is no warm start initialization, we can start our algorithm by randomly initializing the parameters in RSN and ASN at iteration t = 1 .
Then sampling methods will return the counterfactual regret and the numerator of average strategy for the sampled information sets in this iteration, and they will be saved in memories M R and M S respectively.
Then these samples will be used by the NeuralAgent algorithm from Algorithm 2 to optimize RSN and ASN.
Further details for the sampling methods and the NeuralAgent ﬁtting algorithm will be discussed in the next section.
(cid:46)
(cid:46) θ t
(cid:46) update θ t
E FFICIENT T RAINING
In this section, we will propose three techniques to improve the efﬁciency of the double neural method.
These algorithms can also be used separately in other CFR-family methods.
4. R OBUST S AMPLING T ECHNIQUES
In this paper, we proposed a new robust sampling technique which has lower variance than outcome sampling, while being more memory efﬁcient than the external sampling.
In this robust sampling method, the sampling proﬁle is deﬁned as σ rs ( k ) =
( σ rs ( k ) i , σ − i ) , where player i will randomly select k actions according to sampling strategy σ rs ( k ) i
( I i ) for each information set I
i and other players will randomly select one action according to strategy σ − i .
Speciﬁcally, if player i randomly selects min ( k, | A ( I i ) | ) actions according to discrete uniform distribution unif (0 , | A ( I i ) | ) at information set I i , i.e. , σ rs ( k )
i
[Equation]
, then
[Equation]
and the weighted utility u rs ( k ) i ( z ) will be a constant number in each iteration, which has a low variance.
In addition, because the weighted utility no longer requires explicit knowledge of the opponent’s strategy, we can use this sampling method for online regret minimization.
For simplicity, k = max refers to k = max I i ∈I
| A ( I i ) | in the following sections.
Lemma 3: If k = max and ∀ i ∈ N, ∀ I i ∈
I i , ∀
a ∈ A ( I i ) , σ rs ( k ) i ( a | I i ) ∼ unif (0 , |
A ( I i ) | ) , then robust sampling is the same as external sampling.
Lemma 4: If k = 1 and σ rs ( k ) i
=
σ i , then robust sampling is the same as outcome sampling.
Lemma 3 and Lemma 4 provide the relationship between outcome sampling, external sampling, and the proposed robust sampling algorithm.
The detailed theoretical analysis are presented in
Appendix G.2.
4. M INI - BATCH T ECHNIQUES
Mini-batch MCCFR: Traditional outcome sampling and external sampling only sample one block in an iteration and provide an unbiased estimation of origin CFV according to Lemma 2 .
In this paper, we present a mini-batch Monte Carlo technique and randomly sample b blocks in one iteration.
Let Q j denote a block of terminals sampled according to the scheme in section 4.1 at j − th time, then mini-batch CFV with b mini-batches for information set I i can be deﬁned as
[Equation]
Furthermore, we can show that ˜ v σi ( I i | b ) is an unbiased estimator of the counterfactual value of I i :
Lemma 5:
[Equation] [Equation]
(see the proof in section G.3)
Similarly, the cumulative mini-batch regret of action a is
[Equation]
where ˜ R
i (( a | I i ) | b ) = 0 .
In practice, mini-batch technique can sample b blocks in parallel and help
MCCFR to converge faster.
Mini-Batch MCCFR+:
When optimizing counterfactual regret, CFR+ (Tammelin, 2014) substitutes the regret-matching algorithm (Hart & Mas-Colell, 2000) with regret-matching+ and can converge faster than CFR.
However, Burch (2017) showed that MCCFR+ actually converge slower than MCCFR when mini-batch is not used.
In our paper, we derive mini-batch version of MCCFR+ which updates cumulative mini-batch regret ˜ R T, + (( a | I i ) | b ) up to iteration T by
[Equation]
if T =
[Equation]
where ( x ) + = max ( x, 0) .
In practice, we ﬁnd that mini-batch MCCFR+ converges faster than mini-batch MCCFR when specifying a suitable mini-batch size.
E XPERIMENT
The proposed double neural CFR algorithm will be evaluated in the One-Card-Poker game withcards and a large No-Limit Leduc Hold’em (NLLH) with stack size 5, 10, and 15.
The largest NLLH in our experiment has over 2 × 10 7 states and 3 .
7 × 10 6 information sets.
We will compare it with tabular CFR and deep reinforcement learning based method such as NFSP.
The experiments show that the proposed double neural algorithm can converge to comparable results produced by its tabular counterpart while performing much better than deep reinforcement learning method.
With the help of the neural network, our method has a strong generalization and compression ability to converge to an approximate Nash equilibrium by using fewer parameters than the number of information sets.
The current results open up the possibility for a purely neural approach to directly solve large
IIG.
Due to space limit, we present experimental results for One-Card-Poker and the analysis in section C.
The hyperparameters and setting about the neural networks can be found in section E.
Settings.
To simplify the expression, the abbreviations of different methods are deﬁned as follows.
XFP refers to the full-width extensive-form ﬁctitious play method.
NFSP refers to the reinforcement learning based ﬁctitious self-play method.
RS-MCCFR refers to the proposed robust sampling
MCCFR.
This method with regret matching+ acceleration technique is denoted by RS-MCCFR+ .
These methods only containing one neural network are denoted by RS-MCCFR+-RSN and
RS-MCCFR+-ASN respectively.
RS-MCCFR+-RSN-ASN refers to the proposed double neural
MCCFR.
According to Lemma 3 , if k = max , ES-MCCFR is the same with RS-MCCFR.
More speciﬁcally, we investigated the following questions.
Is mini-batch sampling helpful?
Figure 3(A) presents the convergence curves of the proposed robust sampling method with k = max under different mini-batch sizes (b=1, 1000, 5000,respectively).
The experimental results show that larger batch sizes generally lead to better strategy proﬁles.
Furthermore, the convergence for b = 5000 is as good as b = 10000 .
Thus in the later experiments, we set the mini-batch size equal to 5000.
Is robust sampling helpful?
Figure 3 (B) and (C) presents convergence curves for outcome sampling, external sampling( k = max ) and the proposed robust sampling method under the different number of sampled actions.
The outcome sampling cannot converge to a low exploitability smaller than 0.1 after 1000 iterations
The proposed robust sampling algorithm with k = 1 , which only samples one trajectory like the outcome sampling, can achieve a better strategy proﬁle after the same number of iterations.
With an increasing k , the robust sampling method achieves an even better convergence rate.
Experiment results show k = 3 and 5 have a similar trend with k = max , which demonstrates that the proposed robust sampling achieves similar strategy proﬁle but requires less memory than the external sampling.
We choose k = 3 for the later experiments in Leduc
Hold’em Poker.
Figure 3 (C) presents the results in a different way and displays the relation between exploitability and the cumulative number of touched nodes.
The robust sampling with small k is just as good as the external sampling while being more memory efﬁcient on the condition that each algorithm touches the same number of nodes.
[Figure]
[Figure]
How does double neural CRF compare to the tabular counterpart, XFP, and NFSP?
To obtain an approximation of Nash equilibrium, Figure 4(A) demonstrates that NFSP needs 10 6 iterations to reach a 0.06-Nash equilibrium, and requires 2 × 10 5 state-action pair samples and 2 × 10samples for supervised learning respectively.
The XFP needs 10 3 iterations to obtain the same exploitability, however, this method is the precursor of NFSP and updated by a tabular based full-width ﬁctitious play.
Our proposed neural method only needs 200 iterations to achieve the same performance which shows that the proposed double neural algorithm converges signiﬁcantly better than the reinforcement learning counterpart.
In practice, our double neural method can achieve an exploitability of 0.02 after 1000 iterations, which is similar to the tabular method.
What is the individual effect of RSN and ASN?
Figure 4(B) presents ablation study of the effects of RSN and ASN network respectively.
Both MCCFR+-RSN and MCCFR+-ASN, which only employ one neural network, perform only slightly better than the double neural method.
All the proposed neural methods can match the performance of the tabular based method.
How well does continual improvement work?
In practice, we usually want to continually improve our strategy proﬁle from an existing checkpoint (Brown & Sandholm, 2016).
In the framework of the proposed neural counterfactual regret minimization algorithm, warm starting is easy and friendly.
Firstly, we employ two neural networks to clone the existing tabular based cumulative regret and the numerator of average strategy by optimizing Eq.
Then the double neural methods can continually improve the tabular based methods.
As shown in Figure 4(C), warm start from either full-width based or sampling based CFR the existing can lead to continual improvements.
Speciﬁcally, the ﬁrst 10 iterations are learned by tabular based CFR and RS-MCCFR+.
The remaining iterations are continually improved by the double neural method, where b = 5000 , k = max .
A
B
C
D 0.
Does neural network have generalization?
In order to inspect the generalization ability, we perform the neural CFR with small mini-batch sizes (b=50, 100, 500), where only 3 .
59% , and 13 .
06% information sets are observed in each iteration.
According to the results in Figure 5(A), all of these settings can arrive at exploitability less than 0.1 within only 1000 iterations.
Do fewer parameters still work well?
In the proposed neural architecture, the embedding sizeand 16 will leads to 1048 and 2608 parameters respectively in NLLH(5), both of which are much less than the tabular memory, which saves about (cid:80) i ∈ N,I i ∈I |
A ( I i ) | values (more than 10 4 ).
Both these two embedding sizes can converge to a good strategy proﬁle as shown in Figure 5(B).
Does neural method converge in the larger game?
Figure 5(C) presents the log-log convergence curve of different stack size 5, 10, 15.
The largest game size contains over 2 × 10 7 states and 3 .
7 × 10information sets.
Let mini-batch size be 500, there are 13 .
06% , 2 . 39% and 0 .
53% information sets that are observed respectively in each iteration.
Even though only a small subset of nodes are sampled, the double neural method can still achieve O (1 / √ T ) convergence rate.
Is attention in the neural architecture helpful?
Figure 5(D) presents the convergence curves of several different deep neural architectures (LSTM, LSTM plus attention, original RNN plus attention, and GRU plus attention).
The recurrent neural network with LSTM cell plus attention helps us obtain a better convergence rate than the counterpart after hundreds of iterations.
C ONCLUSION AND F UTURE W ORK
In this paper, we present a novel double neural counterfactual regret minimization method to solve large imperfect information game, which has a strong generalization and compression ability and can match the performance of tabular based CFR approach.
We also developed a new sampling technique which has lower variance than the outcome sampling, while being more memory efﬁcient than the external sampling.
In the future, we plan to explore much more ﬂexible methods and apply the double neural method to larger games, such as No-Limit Texas Hold’em.
A PPENDIX A G AME R ULES
Leduc Hold’em a two-players IIG of poker, which was ﬁrst introduced in (Southey et al., 2012).
In
Leduc Hold’em, there is a deck of 6 cards comprising two suits of three ranks.
The cards are often denoted by king, queen, and jack.
In No-Limit Leduc Hold’em( NLLH ), the player may wager any amount of chips up to a maximum of that player’s remaining stack.
There is also no limit on the number of raises or bets in each betting round.
There are two rounds.
In the ﬁrst betting round, each player is dealt one card from a deck of 6 cards.
In the second betting round, a community (or public) card is revealed from a deck of the remaining 4 cards.
In this paper, we use NLLH(x) refer to the
No-Limit Leduc Hold’em, whose stack size is x .
One-Card Poker is a two-players IIG of poker described by (Gordon, 2005).
The game rules are deﬁned as follows.
Each player is dealt one card from a deck of X cards.
The ﬁrst player can pass or bet, If the ﬁrst player bet, the second player can call or fold.
If the ﬁrst player pass, the second player can pass or bet.
If second player bet, the ﬁrst player can fold or call.
The game ends with two pass, call, fold.
The fold player will lose 1 chips.
If the game ended with two passes, the player with higher card win 1 chips, If the game end with call, the player with higher card win 2 chips.
A PPENDIX B D EFINITION OF E XTENSIVE -F ORM G AMES
[Table]
F
C
P
B
P
B
F
C 𝒉
[Table]
𝒉𝒉
F
C
P
B
P
B
F
C 𝒛𝒛𝒛𝒛𝒛𝒛𝒛𝒛
[Table] 𝒉′𝒉′𝒉′ 𝟕 𝒛′𝒛′ F C
P
B
P
B
F
C 𝒉′𝒉′𝒉′𝒉′
F
C
P
B
P
B
F
C 𝒛′𝒛′𝒛′𝒛′𝒛′𝒛′𝒛′𝒛′
B. A DDITIONAL D EFINITIONS
For player i , the expected game utility u σi =
(cid:80)
z ∈
Z π σ ( z ) u i ( z ) of σ is the expected payoff of all possible terminal nodes.
Given a ﬁxed strategy proﬁle σ − i , any strategy σ ∗ i = max σ (cid:48) i ∈
Σ
i u ( σ (cid:48)
i ,σ − i ) i of player i that achieves maximize payoff against π σ − i is a best response .
For two players’ extensive-form games, a Nash equilibrium is a strategy proﬁle
σ ∗ = ( σ ∗ 0 , σ ∗ 1 ) such that each player’s strategy is a best response to the opponent.
An (cid:15) -Nash
equilibrium is an approximation of a Nash equilibrium, whose strategy proﬁle σ ∗ satisﬁes: ∀ i ∈ N , u σ ∗
i i + (cid:15) ≥ max σ (cid:48)
i ∈ Σ
i u ( σ (cid:48) i ,σ − i ) i .
Exploitability of a strategy σ i is deﬁned as (cid:15) i ( σ i ) =
u σ ∗ i − u ( σ i ,σ ∗− i ) i .
A strategy is unexploitable if (cid:15) i ( σ i ) = 0 .
In large two player zero-sum games such poker, u σ ∗ i is intractable to compute.
However, if the players alternate their positions, the value of a pair of games is zeros, i.e. , u σ ∗
+ u σ ∗ 1 = 0 .
We deﬁne the exploitability of strategy proﬁle σ as (cid:15) ( σ ) =
( u ( σ 0 ,σ ∗ 1 )+ u ( σ ∗ 0 ,σ 1 )) / 2 .
B. E XPLANATION BY E XAMPLE
To provide a more detailed explanation, Figure 6 presents an illustration of a partial game tree in
One-Card Poker.
In the ﬁrst tree, two players are dealt (queen, jack) as shown in the left subtree and (queen, king) as shown in the right subtree.
z i denotes terminal node and h i denotes non-terminal node.
There are 19 distinct nodes, corresponding 9 non-terminal nodes including chance h 0 andterminal nodes in the left tree.
The trajectory from the root to each node is a history of actions.
In an extensive-form game, h i refers to this history.
For example, h 3 consists of actions 0:Q, 1:J and P. h 7 consists
Q
J
In IIG, the private card of player 1 is invisible to player 0, therefore h 7 and h 8 are actually the same for player 0.
We use information set to denote the set of these undistinguished states.
Similarly, hand h 2 are in the same information set.
For the right tree of Figure 6, h (cid:48) 3 and h (cid:48) 5 are in the same information set.
h (cid:48) 4 and h (cid:48) 6 are in the same information set.
Generally, any I i ∈ I could only remember the information observed by player i including player i (cid:48) s hidden variable and public actions.
For example, the information set of h 7 and h 8 indicates a sequence of 0:Q, P, and B. Because h 7 and h 8 are undistinguished by player
in IIG, all the states have a same strategy.
For example, I 0 is the information set of h 7 and h 8 , we have I
= I 0 ( h 7 ) =
I 0 ( h 8 ) , σ 0 ( I 0 ) =
σ 0 ( h 7 ) = σ 0 ( h 8 ) , σ 0 ( a | I 0 )
=
σ 0 ( a | h 7 ) =
σ 0 ( a | h 8 ) .
A PPENDIX C A DDITIONAL E XPERIMENT D ETAILS C. F EATURE E NCODING OF P OKER G AMES
The feature is encoded as following.
As shown in the ﬁgure 2 (A), for a history h and player
P ( h ) , we use one-hot encoding (Harris & Harris) to represent the observed actions including chance player.
For example, the input feature x l for l -th cell is the concatenation of three one-hot features including the given private cards, the revealed public cards and current action a .
Both the private cards and public cards are encoded by one-hot technique, where the value in the existing position is 1 and the others are 0.
If there are no public cards, the respective position will be ﬁlled with 0.
Because the action taking by chance is also a cell in the proposed sequential model.
Thus in a
No-Limit poker, such as Leduc Hold’em, action a could be any element in { fold , cumulative spent }∪ { public cards } , where cumulative spent denotes the total chips after making a call or raise.
The length of the encoding vector of action a is the quantities of public cards plus 2, where cumulative spent is normalized by the stack size.
C. A DDITIONAL E XPERIMENT R ESULTS
[Figure]
Figure 7 (A) presents the convergence rate for the proposed robust sampling method of different mini-batch size b
=
The experimental results are similar to Leduc Hold’em poker, larger mini-batch size indicates a better exploitability.
Figure 7 (B) demonstrates that the convergence rate for different sampling methods including outcome sampling and robust sampling under k = 1 , 2 .
The conclusion is that RS-MCCFR+ converges signiﬁcantly faster than OS-MCCFR+ after touching the same number of nodes.
Experiment results show that k = 1 has a similar trend with k = 2 (external sampling).
Because only one trajectory is sampled, the proposed RS-MCCFR+ will require less memory than the external sampling.
Figure 7 (C) compares the performance between the tabular method and the double neural method.
Experimental results demonstrate that RS-MCCFR+-RSN-ASN can achieve an exploitability of less than 0.0004 in One-Card Poker, which matches the performance of the tabular method.
For RSN and ASN, we set neural batch size 4, hidden size 32 and learning rate 0.001.
A PPENDIX D D ETAILS OF R ECURRENT N EURAL N ETWORK
In order to deﬁne our R and S network, we need to represent the information set I i ∈
I in extensive-form games.
In such games, players take action in alternating fashion and each player makes a decision according to the observed history.
In this paper, we model the behavior sequence as a recurrent neural network and each action in the sequence corresponds to a cell in RNN.
Figure 2 (A) provides an illustration of the proposed deep sequential neural network representation for information sets.
In standard RNN, the recurrent cell will have a very simple structure, such as a single tanh or sigmoid layer.
Hochreiter & Schmidhuber (1997) proposed a long short-term memory method (LSTM) with the gating mechanism, which outperforms the standard version and is capable of learning long-term dependencies.
Thus we will use LSTM for the representation.
Furthermore, different position in the sequence may contribute differently to the decision making, we will add an attention mechanism (Desimone & Duncan, 1995; Cho et al., 2015) to the LSTM architecture to enhance the representation.
For example, the player may need to take a more aggressive strategy after beneﬁcial public cards are revealed.
Thus the information, after the public cards are revealed may be more important.
More speciﬁcally, for l -th cell, deﬁne x l as the input vector (which can be either player or chance actions), e l as the hidden layer embedding, φ ∗ as a general nonlinear function.
Each action is represented by a LSTM cell, which has the ability to remove or add information to the cell state with three different gates.
Deﬁne the notation · as element-wise product.
The ﬁrst forgetting gate layer is deﬁned as g fl = φ f ( w f [ x l , e l − 1 ]) , where [ x l , e l − 1 ] denotes the concatenation of x l and e l − 1 .
The second input gate layer decides which values to update and is deﬁned as g il =
φ i ( w i [ x l , e l − 1 ]) .
A nonlinear layer output a vector of new candidate values ˜ C l = φ c ( w l [ x l , e l − 1 ]) to decide what can be added to the state.
After the forgetting gate and the input gate, the new cell state is updated by
C l = g fl · C l − 1 + g il · ˜ C l .
The third output gate is deﬁned as g ol =
φ o ( w o [ x l , e l − 1 ]) .
Finally, the updated hidden embedding is e l = g ol · φ e ( C l ) .
As shown in Figure 2 (A), for each LSTM cell j , the vector of attention weight is learned by an attention network .
Each member in this vector is a scalar α j = φ a ( w a e j ) .
The attention embedding of l -th cell is then deﬁned as e al = (cid:80) lj
=1 α j · e j , which is the summation of the hidden embedding e j and the learned attention weight α j .
The ﬁnal output of the network is predicted by a value network , which is deﬁned as
[Equation] [Equation]
where θ is the parameters in the deﬁned sequential neural networks.
Speciﬁcally, φ f , φ i , φ o are sigmoid functions.
φ c and φ e are hyperbolic tangent functions.
φ a and φ v are rectiﬁed linear functions.
The proposed R SN and A SN share the same neural architecture, but use different parameters.
That is R ( a, I i | θ t R ) =
f ( a, I i | θ t R ) and S ( a,
I i | θ t S )
=
f ( a, I i | θ t S ) .
R ( · , I i | θ t R ) and
S ( · , I i | θ t S ) denote two vectors of inference value for all a ∈ A ( I i ) .
A PPENDIX E N EURAL A GENT FOR O PTIMIZING N EURAL R EPRESENTATION
Deﬁne β epoch as training epoch, β lr as learning rate, β loss as the criteria for early stopping, β re as the upper bound for the number of iterations from getting the minimal loss last time, θ t − 1 as the parameter to optimize, f ( ·| θ t − 1 ) as the neural network, M as the training sample consisting information set and the corresponding target.
To simplify notations, we use β ∗ to denote the set of hyperparameters in the proposed deep neural networks.
β ∗ R and β ∗ S refer to the
1 Function NeuralAgent f
·| θ T − 1 ) , M , θ T − 1 , β ∗ ) : initialize
(cid:46) gradient descent optimizer and learning rate scheduler θ T ← θ T − 1 , l best ← ∞ , t best ←(cid:46)
warm starting from the checkpoint of the last iteration do
] (cid:46) initialize loss as an empty list do
=
M (cid:46) sampling a mini-batch from M
=
i
|
T −
+ i ) −
i
(cid:46) d is dimension of θ T
(cid:46) sets of hyperparameters in RSN and ASN respectively.
According to our experiments, we ﬁnd a carefully designed optimization method can help us obtain a relatively higher convergence rate of exploitability.
Algorithm 2 presents the details of how to optimize the proposed neural networks.
Both R ( a, I i | θ t +1 R ) and S ( a, I i | θ t S ) are optimized by mini-batch stochastic gradient descent method.
In this paper, we use Adam optimizer (Kingma & Ba, 2014) with both momentum and adaptive learning rate.
Some other optimizers such as Nadam, RMSprop, Nadam from (Ruder, 2017) are also tried in our experiments, however, they do not achieve better experimental results.
In practice, existing optimizers may not return a relatively low enough loss because of potential saddle point or local minima.
To obtain a relatively higher accuracy and lower optimization loss, we use a carefully designed scheduler to reduce the learning rate when the loss has stopped decrease.
Speciﬁcally, the scheduler reads a metrics quantity, e.g , mean squared error, and if no improvement is seen for a number of epochs, the learning rate is reduced by a factor.
In addition, we will reset the learning rate in both optimizer and scheduler once loss stops decrease in β re epochs.
Gradient clipping mechanism is used to limit the magnitude of the parameter gradient and make optimizer behave better in the vicinity of steep cliffs.
After each epoch, the best parameter will be updated.
Early stopping mechanism is used once the lowest loss is less than the speciﬁed criteria β loss .
In experiments, we set the network hyperparameters as follow.
For RSN, we set the hyperparameters as follows: neural batch size is 256 and learning rate β lr = 0 .
A scheduler, who will reduce the learning rate based on the number of epochs and the convergence rate of loss, help the neural agent to obtain a high accuracy.
The learning rate will be reduced by 0.5 when loss has stopped improving after 10 epochs.
The lower bound on the learning rate of all parameters in this scheduler is 10 − 6 .
To avoid the algorithm converging to potential local minima or saddle point, we will reset the learning rate to 0.001 and help the optimizer to learn a better performance.
θ Tbest is the best parameters to achieve the lowest loss after T epochs.
If average loss for epoch t is less than the speciﬁed criteria β loss = 10 − 4 , we will early stop the optimizer.
We set β epoch = 2000 and update the optimizermaximum epochs.
For ASN, we set the loss of early stopping criteria as 10 − 5 .
The learning rate will be reduced by 0.7 when loss has stopped improving after 15 epochs.
Other hyperparameters in
ASN are similar to RSN.
A PPENDIX F O PTIMIZE C OUNTERFACTUAL R EGRET M INIMIZATION WITH T WO D EEP N EURAL N ETWORKS
1 Function Mini-Batch-MCCFR-NN( t ) :
M R ← ∅ , M S ← ∅
For all i = 1 to b do in parallel then
MCCFR-NN( t, ∅ , 0 , 1 , 1 )
MCCFR-NN( t, ∅ , 1 , 1 , 1 ) return M R , M S 8 Function MCCFR-NN( t, h , i , π i , π rs ( k ) i ) :
I i ← I i ( h ) (cid:46) information set at state h
if h ∈ Z then return u i ( h ) π rs ( k ) i (cid:46) return game payoff else if P ( h )
= − 1 then a ∼ σ − i ( I i )
(cid:46) Sample an action from σ − i ( h ) return
MCCFR-NN( t, ha, i, π i , π rs ( k ) i ) else if P ( h )
=
i then
ˆ R i ( ·| I i ) ← R ( · , I i | θ t R ) if t > 1 else −→(cid:46) inference the vector of cumulative regret ∀
a ∈ A ( I i ) σ
i ( I i )
← CalculateStrategy ( ˆ R i ( ·| I i ) , I i )
(cid:46) calculate current strategy v i ( h ) ← 0 ,
r i ( ·| I i ) ← (cid:126) 0 , s i ( ·| I i ) ← (cid:126)(cid:46) r i ( ·| I i ) and i ( ·| I i ) are two vectors over A ( I i )
A rs ( k ) ( I i ) ← sampling k different actions according to σ rs ( k )
i
For a ∈ A rs ( k ) ( I i ) do v i ( a | h ) ← MCCFR-NN ( t, ha, i, π i σ i ( a | I i ) , π rsi σ rs ( k ) i ( a | I i ))
v i ( h ) ← v i ( h )
+ v i ( a | h ) σ i ( a | I i ) (cid:46) update counterfactual value
For a ∈ A rs ( k ) ( I i ) do r i ( a | I i )
← v i ( a | h ) − v i ( h ) (cid:46) update cumulative regret s i ( a | I i ) ← π σi ( I i ) σ i ( a | I i )
(cid:46) update average strategy numerator
Store updated cumulative regret tuple ( I i , r i ( ·| I i )) in M R
Store updated current strategy dictionary ( I i , s i ( ·| I i )) in M S else
ˆ R − i ( ·| I i ) ← R ( · , I i | θ t R ) if t > 1 else −→(cid:46) inference cumulative regret σ − i ( I i )
← CalculateStrategy ( ˆ R − i ( ·| I i ) , I i )
(cid:46) calculate current strategy a ∼ σ − i ( I i )
(cid:46) Sample an action from σ − i
( I i ) return MCCFR-NN( t, ha, i, π i , π rs ( k ) i )
Function CalculateStrategy
( R i ( ·| I i ) , I i ) : sum ← (cid:80) a ∈ A ( I i ) max( R i ( a | I i ) , 0)
For a ∈ A ( I i ) do σ i ( a | I i )
=
max( R i ( a | I i ) , 0) sum if sum > 0 else 1 | A ( I i ) | return σ i ( I i )
Algorithm 3 presents one application scenario of the proposed double neural method, which is based on the proposed mini-batch robust sampling method.
The function MCCFR-NN will traverse the game tree like tabular MCCFR, which starts from the root history h = ∅ .
Deﬁne I
i as the information set of h .
Suppose that player i will sample k actions according to the robust sampling.
Then the function can be deﬁned as follows.
(1) If the history is terminal, the function returns the weighted utility.
(2) If the history is the chance player, one action a ∈ A ( I i ) will be sampled according to the strategy σ −
i
( I i ) .
Then this action will be added to the history, i.e. , h ← ha .
(3) If P ( I i ) =
i , the current strategy can be updated by the cumulative regret predicted by RSN.
Then we sample k actions according the speciﬁed sampling strategy proﬁle σ rs ( k ) i .
After a recursive updating, we can obtain the counterfactual value and regret of each action at I i .
For the visited node, their counterfactual regrets and numerators of the corresponding average strategy will be stored in M R and M S respectively.
(4) If P ( I i ) is the opponent, only one action will be sampled according the strategy σ −
i ( I i ) .
The function Mini-Batch-MCCFR-NN presents a mini-batch sampling method, where b blocks will be sampled in parallel.
This mini-batch method can help the MCCFR to achieve a more accurate estimation of CFV.
The parallel sampling makes this method efﬁcient in practice.
A PPENDIX G T HEORETICAL A NALYSIS G. R EACH P ROBABILITY AND P OSTERIOR P ROBABILITY
Lemma 1: The state reach probability of one player is proportional to posterior probability of the opponent’s hidden variable, i.e., p ( h v − i
| I i )
∝ π σ − i
( h ) .
Proof: For player i at information set I i and ﬁxed i (cid:48)
s strategy proﬁle
σ i , i.e. , ∀ h ∈
I
i , π σi ( h ) is constant.
Based on the deﬁnation of extensive-form game in Section 2.1, the cominbation of I i and opponent’s hidden state h v − i can indicate a particular history
h
= h vi h v − i a 0 a 1 ...a L − 1 .
With Bayes’
Theorem, we can inference the posterior probability of opponent’s private cards with Equation
[Equation]
L
[Equation]
G. R OBUST S AMPLING , OUTCOME SAMPLING AND EXTERNAL SAMPLING
Lemma 3: If k = max I i ∈I |
A ( I i ) | and ∀ i ∈ N, ∀ I i ∈
I i , ∀
a ∈ A ( I i ) , σ rs ( k ) i ( a | I i ) ∼ unif (0 , |
A ( I i ) | ) , then robust sampling is same with external sampling.
(see the proof in section
G.2)
Lemma 4: If k = 1 and σ rs ( k ) i
=
σ i , then robust sampling is same with outcome sampling.
(see the proof in section G.2)
For robust sampling, given strategy proﬁle σ and the sampled block Q j according to sampling proﬁle σ rs ( k ) = ( σ rs ( k ) i , σ − i ) , then q ( z ) =
π σ rs ( k ) i ( z ) π σ − i ( z ) , and the regret of action
a ∈
A rs ( k ) ( I i ) is ˜ r σi (( a |
I i ) | Q j ) =
˜ v σi (( a | I i ) | Q j ) − ˜ v σi ( I i | Q j ) =
(cid:88)
[Equation] [Equation]
where u rsi ( z ) =
u i ( z ) π σrs ( k ) i ( z ) is the weighted utility according to reach probability π σ rs ( k ) i ( z ) .
Because the weighted utility no long requires explicit knowledge of the opponent’s strategy, we can use this sampling method for online regret minimization.
Generally, if player i randomly selects min ( k, | A ( I i ) | ) actions according to discrete uniform distribution unif (0 , | A ( I i ) | ) at information set I i , i.e. , σ rs (
[Equation]
, then
[Equation]
and u rsi ( z ) is a constant number when given the sampling proﬁle σ rs ( k ) .
Speciﬁcally,
• if k = max I i ∈ I
| A ( I i ) | , then σ rs ( k
[Equation]
i
[Equation]
Therefore, robust sampling is same with external sampling when k = max I
i ∈ I
| A ( I i ) | .
• if k = 1 and σ rs ( k ) i
= σ i , only one history z is sampled in this case,then u rs ( k ) i ( z )
=
u i ( z ) π σii ( z ) , ∃ h ∈
I i , for a ∈
A rs ( k ) ( I i )
[Equation]
For a (cid:54)∈ A rs ( k ) ( I i ) , the regret will be ˜ r σi (( a | h ) | j ) =
− ˜ v σi ( h | j ) .
Therefore, robust sampling is same with outcome sampling when k
= 1 and σ rs ( k ) i
=
σ i .
• if k = 1 , and player i randomly selects one action according to discrete uniform distribution
unif
(0 , | A ( I i ) | ) at information set I i , then u rs (1)
i
[Equation]
is a
constant
∃
h ∈ I
i
, for
a ∈
A rs
( k ) ( I i
) ˜
[Equation]
if action a is not sampled at state h , the regret is ˜ r σi (( a | h ) | j ) =
− ˜ v σi ( h | j ) .
Compared to outcome sampling, the robust sampling in that case have a lower variance because
of the constant u
rs (1)
i
( z ) .
G.
M
-B ATCH MCCFR GIVES AN UNBIASED ESTIMATION OF COUNTERFACTUAL VALUE
Lemma 5: E Q j ∼ Robust Sampling
[˜ v σ i ( I i | b )]
= v σ i
( I i ) .
Proof:
[Equation]