PerfectDou: Dominating DouDizhu with Perfect Information Distillation
Anonymous Author(s)
AffiliationAddress email
Abstract
As a challenging multi-player card game, DouDizhu has recently drawn muchattention for analyzing competition and collaboration in imperfect-informationgames.
In this paper, we propose PerfectDou, a state-of-the-art DouDizhu AIsystem that dominates the game, in an actor-critic framework with a proposedtechnique named perfect information distillation.
In detail, we adopt a perfecttraining-imperfect-execution framework that allows the agents to utilize the globalinformation to guide the training of the policies as if it is a perfect informationgame and the trained policies can be used to play the imperfect information gameduring the actual gameplay.
To this end, we characterize card and game featuresfor DouDizhu to represent the perfect and imperfect information.
To train oursystem, we adopt proximal policy optimization with generalized advantage estimation in a parallel training paradigm.
In experiments we show how and why
PerfectDou beats all existing AI programs, and achieves state-of-the-art performance.
Introduction
With the fast development of Reinforcement Learning (RL), game AI has achieved great success inmany types of games, including board games (e g., Go [21], chess [20]), card games (e.g., Texas
Hold’em [3], Mahjong [14]) and video games (e.g., Starcraft [25], Dota [1]).
As one of the mostpopular card games in China, DouDizhu has not been studied in depth until very recently.
In perfectinformation games such as Go, agent can observe all the events occurred previously including initialhand of each agent and all agents’ actions.
In contrast, DouDizhu is an imperfect-information gamewith special structure, and an agent does not know other agents’ initial hands but can observe allagents’ actions.
One challenge in DouDizhu is that it is a three-player game with both competitionand collaboration: the two Peasant players need to cooperate as a team to compete with the third
Landlord player.
In addition, DouDizhu has a large action space that is hard to be abstracted forsearch-based methods [30].
Although various methods have been proposed for tackling these challenges [28, 9], they are eithercomputationally expensive or far from optimal, and highly rely on abstractions with human knowledge [30].
Recently, [30] proposed DouZero, which applies simple Deep Monte-Carlo (DMC)method to learn the value function with pre-designed features and reward function.
DouZero isregarded as the state-of-the-art (SoTA) AI system of DouDizhu for its superior performance andtraining efficiency compared with previous works.
Unfortunately, we find DouZero still has severe limitations in many battle scenarios, which willbe characterized in Section D.3.
To establish a stronger and more robust DouDizhu bot, in thispaper, we present a new AI system named PerfectDou, and show that it leads to significantlybetter performance than existing AI systems including DouZero.
The name of our program fol- lows the key technique we use – perfect information distillation.
The proposed technique utilizesa Perfect-Training-Imperfect-Execution (PTIE) framework, a variant of the popular CentralizedTraining-Decentralized-Execution (CTDE) paradigm in multi-agent RL literature [5, 10].
Namely,we feed perfect-information to the agent in the training phase to guide the training of the policy, andonly imperfect-information can be used when deploying the learned policy for actual game play.
Correspondingly, we further design the card and game features to represent the perfect and imperfect information.
To train PerfectDou, we utilize Proximal Policy Optimization (PPO) [18] with
Generalized Advantage Estimation (GAE) [17] by self-play in a distributed training system.
In experiments, we show PerfectDou beats all the existing DouDizhu AI systems and achieves the
SoTA performance in a 10k-decks tournament; moreover, PerfectDou is the most training efficient,such that the number of samples required is an order of magnitude lower than the previous SoTAmethod; for application usage, PerfectDou can be deployed in online game environment due to itslow inference time.
Preliminaries
Imperfect-Information Extensive-Form Games.
An imperfect-information extensive-form (ortree-form) game can be described as a tuple G =
( P , H , Z , A , T , χ, ρ, r, I ) , where P denotes afinite set of players , A is a finite set of actions, and H is a finite set of nodes at which players cantake actions and are similar to states in an RL problem.
At a node h ∈ H , χ : H → 2 A is the actionfunction that assigns to each node h ∈ H a set of possible actions, and ρ : H → P represents theunique acting player.
An action a ∈ A ( h ) that leads from h to h ′ is denoted by the successor function
T : H ×
A → H as h ′ =
T ( h, a ) .
Z ⊆
H are the sets of terminal nodes for which no actions areavailable.
For each player p ∈ P , there is a reward function r p ∈ r = r 1 , r 2 , . . .
, r |P| : Z → R .
Furthermore, I =
{I p | p ∈ P} describes the information sets (infosets) in the game where I p is apartition of all the nodes with acting player p .
If two nodes h, h ′ belong to the same infoset I ofplayer p , i.e., h, h ′ ∈ I ∈
I p , these two nodes are indistinguishable to p and will share the sameaction set.
We use I ( h ) to denote the infoset of node h .
Upon a certain infoset, a policy (or abehavior strategy)
π p for player p describes which action the player would take at each infoset.
Apolicy can be stochastic, and we use π p ( I ) to denote the probability vector over player p ’s availableactions at infoset I .
With a slight abuse of notation, we use π p ( h ) to denote the stochastic actionplayer p will take at node h .
For two nodes h, h ′ that belong to the same infoset I , it is clear thatπ p ( h )
=
π p ( h ′ ) =
π p ( I ) .
Therefore, the objective for each player p is to maximize its own totalexpected return at the end of the game: R p ≜
E Z ∼ π
[ r p ( Z )] , Z ∈ Z .
The Three-Players DouDizhu Game.
DouDizhu (a.k.a.
Fight the Landload) is a three-playercard game that is popular in China and is played by hundreds of millions of people.
Among thethree players, two of them are called the Peasants , and they need to cooperate as a team to competeagainst the other player called the Landlord .
The standard game consists of two phases, bidding andcardplay.
The bidding phase designates the roles to the players and deals leftover cards to the Landlord .
In the cardplay phase, the three players play cards in turn in clock-wise order.
Within a gameepisode, there are several rounds , and each begins with one player showing a legal combination ofcards (solo, pair, etc.).
The subsequent players must either choose to pass or beat the previous handby playing a more superior combination of cards, usually in the same category.
The round continuesuntil two consecutive players choose to pass and the player who played the last hand initiates to thenext round.
DouDizhu is in the genre of shedding where the player wins by emptying his’s hand, orloses vice versa.
Therefore, in this game, the suit does not matter but the rank does.
The score of agame is calculated as the base score multiplied by a multiplier determined by specialized categoriesof cards (Appendix B.2 shows the details).
In this paper, we only consider the cardplay phase, whichcan be formulated as an imperfect-information game.
More detailed information about the game canbe referred to [30].
The key challenges of DouDizhu include how the Peasants work as a team to beat the Landlordwith card number advantage using only imperfect information.
For example, one Peasant can tryto help his teammate to win by always trying the best to beat the Landlord ’s cards and play cardsin a category where the teammate has an advantage.
In addition, the action space of DouDizhu isparticularly large, and there are 27,472 possible combinations of cards that can be played in totalwith hundreds of legal actions in a hand.
Furthermore, the action space cannot be easily abstracted since improperly playing a card may break other potential card combinations in the following roundsand lead to losing the game.
Methodology
In this section, we first introduce how perfect-training-imperfect-execution works for a generalimperfect-information game.
Then, we formulate the DouDizhu game as an imperfect-informationgame to solve.
3.1 Perfect Information Distillation
[Figure]
In card games as DouDizhu, the imperfect-information property comes from the fact that players do not show their handcards to the others.
And therefore the critical challenge foreach player is to deal with the indistinguishable nodes fromthe same infoset.
For such games, consider we can constructa strategically identical perfect-information game and allowone player to observe distinguishable nodes, then, the decisions at each node can rely on the global information andhe may have more chances to win the game, like owning acheating plug-in.
This motivates us to utilize the distinguishable nodes for training the agents of imperfect-informationgames, and therefore we propose the technique of perfectinformation distillation.
In general, the perfect information distillation is anactor-critic framework trained in perfect-training-imperfectexecution (PTIE) paradigm, a variant of centralized-trainingdecentralized-execution (CTDE) [5, 10], as illustrated in
Fig.
Particularly, CTDE constructs the value function withall agents’ observations and actions for general multi-agenttasks.
By comparison, our proposed PTIE is designed forimperfect-information games where additional perfect information is introduced to the value function.
Actor-critic [24] is a template of policy gradient (PG) methods, proposed in the RL literaturetowards maximizing the expected reward of the policy function through PG with a value function:
[Equation]
where s denotes a state in an RL problem, Q is the state-action value function learned by a functionapproximator, usually called the critic.
Notice that the critic is playing the role of evaluating howgood an action is taken at a specific situation, but only at the training time.
When the agent isdeployed into inference, only the policy π can be used to inferring feasible actions.
Therefore, forimperfect-information games, we can provide additional information about the exact node the playeris in to train the critic with self-play, as long as the actor does not take such information for decisionmaking.
Intuitively, we are distilling the perfect information into the imperfect policy.
Formally, for each node h , we construct a distinguishable node D ( h ) for the strategically identical perfect-information game.
Then, we define the value function at D ( h ) , V π p ( D ( h ))
=
129 E a
[Equation]
p ,
[Equation]
=
[Equation] [Equation]
] =
[Equation]
[ r p ( Z )] as the expected value of distinguishable nodes.
In 130 the sequel, we propose a simple extension of actor-critic policy gradient considering parameterizedpolicy π θ p for each player p :
[Equation]
In practice, we use a policy network (actor) to represent the policy π p for each player, which takesas input a vector describing the representation at an indistinguishable node h that the player canobserve during the game.
For estimating the critic, a value network is utilized, which takes representation of the global information at the distinguishable node D ( h ) .
In other words, the valuenetwork takes additional information (such as other players’ cards in Poker games) as input, whilethe policy network does not.
During the training, the value function updates the values for all distinguishable nodes; then, it trains the policy on every node on the same infoset from sampled data, which implicitly gives an expected value estimation on each infoset.
In practice, the generalizationability of neural networks enables the policy to find a better solution, which is also the advantage forusing the proposed PTIE framework.
PTIE is a general way for training imperfect-information game agents.
In addition, the optimalitypoint of independent RL in multi-agent learning is exactly one of an Nash equilibria (NE) and it hasbeen proved that training RL algorithms with self-play can converge to an NE in two-player cases(although no convergence guarantees for three-player games) [8, 11].
We expect that with PTIE,players can leverage the perfect information during inference to derive coordination and strategicpolicies.
In experiments, we show that this allows PerfectDou to cooperate with each other (as
Peasants ) or compete against the team (as the Landlord ).
3.2 DouDizhu as An Imperfect-Information Game
As is mentioned in Section 2, the cardplay phase of DouDizhu can be regarded as an imperfectinformation game with three players.
At each node h , its infoset I p contains all combinations of theother’s invisible handcards.
Then at each level of the game tree, the three players take clockwiseturns to choose an available action with policies π p depending on the infoset I p ( h ) of the currentnode h with the reward function r p .
The path from the root of the game tree to a node h containsthe initial hand of all players and all the historical moves of all players.
The reward functions at leafnodes are set to be the score the players win or lose at the end of the game.
PerfectDou System Design
[Figure]
In this section, we explain how we construct our PerfectDou system in detail, with theproposed perfect information distillation technique, and several novel components designedfor DouDizhu that help it summit the game.
Particularly, PTIE requires different representations as input layer for the policy and the valuenetwork by feeding the value function withperfect information (distinguishable nodes) andthe policy with imperfect information (indistinguishable nodes).
4.1 Card Representation
In our system, we encode each feasible cardcombination with a 12 × 15 matrix, as shownin Fig.
Specifically, we first encode different ranks and numbers with a 4 × 15 matrix,where the columns correspond to the 15 ranks(including jokers) and similar to [30], the number of ones in the four rows of a single column represents the number of cards of that rank in theplayer’s hand.
Different from [30], we further propose to encode the legal combination of cards withthe player’s current hand, to help the agent realize the different property of various kinds of cards(see Section 4.2).
The feature sizes of each part are shown in Appendix C.1.
4.2 Node Representation
In the game of DouDizhu, the distinguishable node D ( h ) should cover all players’ hand cards at h ,along with the game and player status.
Therefore, we propose to represent h with imperfect featuresand D ( h ) with perfect feature designs, shown in Tab. 1.
In detail, the imperfect features include aflatten matrix 1 of 23 × 12 × 15 and a game state array of 6 × 1 .
On the contrary, the perfect featuresconsist of a flatten card matrix of 25 × 12 × 15 and a game state array of 8 × 1 .
Therefore, they aretotally asymmetric, and the imperfect features are a subset of the perfect feature.
[Table]
4.3 Network Structure and Action Representation
The PerfectDou system follows the general actor-critic design, and we take PPO [18] with GAE [17]as the learning algorithm.
Slightly different from Eq.
(2), PPO estimates the advantage A p
=
R p −
V π p as the critic instead of Q π p .
For value network, we use an MLP to handle encoded features (thedetailed structure is shown in Appendix C.3).
As for the policy network, we first utilize an LSTMto encode all designed features; to encourage the agent to pay attention to specific card types, theproposed network structure will encode all the available actions into feature vectors, as depictedin Tab. 6.
The output of the legal action probability is then computed with the action and gamefeatures, as illustrated in Fig.
Formally, we concatenate the node representation e s with eachaction representation e a i separately, and get the legal action distribution:
[Equation]
where a i is the i -th action, [ · ] denotes the concatenation operation for N available actions, and f arelayers of MLPs.
This resembles the target attention mechanism in [27].
4.4 Node Reward Design
If we only care about the result at the end of the game, the reward at leaf nodes is rather sparse; inaddition, players can only estimate their advantage of winning the game using imperfect informationduring the game, which could be inaccurate and fluctuated.
To that end, we propose an augmentedreward function for DouDizhu at each node.
Instead of letting the players estimate the advantageseparately, we utilize an oracle 2 for evaluating each player, particularly, the minimum steps neededto play out all cards, which can be treated as a simple estimation of the distance to win.
The rewardfunction is then defined as the advantage difference computed by the relative distance to win of thetwo camps in two consecutive timesteps.
Formally, at timestep t , the reward function is:
[Equation] [Equation]
where l is a scaling factor, and N t is the minimum steps to play out all cards at timestep t .
For instance, in a round, at timestamp t , the distance of the Landlord to win is 5 and the distances oftwo Peasants are 3 and 7, which means Peasants have a larger advantage since the relative distanceis 2 for Peasants and -2 for the Landlord .
However, if the Landlord plays a good hand such that2 Implemented as a dynamic programming algorithm, see Appendix E for details.
all Peasants can not suppress, the Landlord will in result get a positive reward due to the decreasedrelative distance of the Landlord , i.e., from 2 to 1.
Correspondingly, the Peasants would get anegative reward as their relative distances are getting larger.
Such a reward function can encouragethe cooperation between Peasants , since the winning distance is defined by the minimum steps ofboth players.
In our implementation, the computation of the rewards is carried out after a round ofthe game, hence to promote training efficiency.
4.5 Distributed Training Details
[Figure]
To further expedite the training procedure, we design a distributed training system represented in Fig. 4.
Specifically, the system contains a setof rollout workers for collecting theself-play experience data and sending it to a pool of GPUs; these GPUsasynchronously receive the data andstore it into their local buffers.
Then,each learner randomly samples minibatches from its own buffer and compute the gradient separately, which isthen synchronously averaged acrossall GPUs and back propagated to update the neural networks.
After eachround of updating, new parametersare sent to every rollout worker.
Andeach worker will load the latest modelafter 24 (8 for each player) steps sampling.
Such a decoupled training-sampling structure will allow
PerfectDou to be extended to large scale experiments.
Our design of the distributed system borrowsa lot from IMPALA [4], which also keeps a set of rollout workers to receive the updated model,interact with the environment and send back rollout trajectories to a learner.
The main differenceis derived from the learning algorithm where we use PPO with GAE instead of actor-critic with
V-trace [4].
Moreover, we keep three different models for Landlord and two Peasants separatelywhich is only updated by their own data against the latest opponent model.
[Figure]
Related Work
Imperfect-Information Games.
Many popular card games are imperfect-information games andhave attracted much attention.
For instance, [14] worked on the four-player game Mahjong andproposed a distributed RL algorithm combined with techniques like global reward prediction, oracle guiding, and run-time policy adaptation to win against most top human players; in addition,[12] adopted search-based and imitation methods to learn the playing policy for Hanabi.
Iterativealgorithms such as Counterfactual Regret Minimization (CFR) and its variants [32, 16, 3] are alsowell-used for a fully competitive game, Hold’em Poker, whose action space is generally designed atthe scale of tens (fold, call, check and kinds of bet) and the legal actions at each decision point areeven less [16, 31].
In addition, CFR does not proven to converged to an Nash equilibrium in games
[Table]
with more than two players [13].
Consequently, they can be hardly designed for DouDizhu due tothe large action space with difficult abstraction and the mixed game property (both cooperative andcompetitive, although there are success few cases for such a setting [15]).
DouDizhu AI systems.
Besides the recent SoTA work DouZero [30], many researchers have madeefforts on utilizing the power of RL into solving DouDizhu.
However, simply applying RL algorithms such as DQN and A3C into the game can hardly make benefits [28].
Therefore, [28]proposed Combinational Q-Network (CQN) that reduces the action space by heuristics action decoupling; moreover, DeltaDou [9] utilized Monte-Carlo Tree Search (MCTS) for DouDizhu, alongwith Bayesian inference for the hidden information and a pre-trained kicker network for action abstraction.
DeltaDou was also reported as reaching human-level performance.
Experiments
We conduct comprehensive experiments to investigate the following research questions.
RQ1 : Howgood is PerfectDou against SoTA DouDizhu AI?
RQ2 : What are the key ingredients of PerfectDou?
RQ3 : How is the inference efficiency of PerfectDou?
To answer RQ1 , we empirically evaluatethe performance against existing DouDizhu programs.
Regarding RQ2 , we conduct ablation studies on key components in our design.
And for RQ3 , we calculate the average inference time forall algorithms involved.
Finally, we conduct in-depth analysis, provide interesting case studies of
PerfectDou.
In the appendix, we report more results including a battle against skilled human players.
6.1 Experimental Setups
Baselines.
We evaluate PerfectDou against following algorithms under the open-source RLCard
Environment [29]: 1) DouZero [30]: A recent SoTA baseline method that had beaten every existing DouDizhu AI system using Deep Monte-Carlo algorithm.
2) DeltaDou [ 9]: An MCTS- based algorithm with Beyesian inference.
It achieved
comparable performance as human experts
Combinational Q-Network (CQN) [28]: Based on card decomposition and Deep Q-Learning.
4) Rule-Based Algorithms : Including the open-source heuristic-based program RHCP-v2 [9, 30],
the rule model in RLCard and a Random program with uniform legal moves.
For evaluation, we
directly take their public (or provided) codes and pre-trained models
Metrics.
The performance of DouDizhu are mainly quantified following the same metrics in previous researches [9, 30].
Specifically, given two algorithms A against B , we calculate: 1)
WP
(Winning
Percentage): The proportion of winning by A in a number of games. 2) ADP (Average Differencein Points):
The per-game averaged difference of scores between A and B .
In other words, positive
ADP means gaining scores while negative represents losing it.
This is a more reasonable metric forevaluating DouDizhu AI systems, because in real games players are evaluated by the scores obtainedinstead of their winning rate, as further discussed in Appendix B.2.
In our experiments, we choose ADP as the basic reward (except the tournament results of the columnof WP shown in Tab. 2), which is augmented with the proposed reward signal in Section 4.4 duringthe early training stage.
6.2 Comparative Evaluations
We conduct a tournament to demonstrate the advantage of our PerfectDou, where each pair of thealgorithms plays 10,000 decks, shown in Tab.
( RQ1 ).
Since the bidding performance in each algorithm varies and poor bidding would affect game results significantly, for fair comparison, we omitthe bidding phase and focus on the phase of cardplay.
In detail, all games are randomly generatedand each game would be played two times, i.e., each competing algorithm is assigned as Landlordor Peasant once.
We use WP and ADP as the basic reward respectively for comparing over thesetwo metrics for all evaluating methods.
Overall, PerfectDou dominates the leaderboard by beating all the existing AI programs, no matterrule-based or learning based algorithms, with significant advantage on both WP and ADP.
Specifically, as noted that DouDizhu has a large variance where the initial hand cards can seriously determine the advantage of the game; even though, PerfectDou still consistently outperforms the current
SoTA baseline – DouZero.
However, we find that PerfectDou is worse than the result publishedin DouZero paper [30] when competing against RHCP.
To verify this problem, we test the publicmodel of DouZero (denoted as DouZero (Public) with grey color).
To our surprise, its performancecan match most of the reported results in their paper except the one against RHCP, where it onlytakes a WP of 0.452 lower than 0.5, indicating that the public model of DouZero can not beat RHCPas suggested in the original paper, and in fact PerfectDou is the better one.
It is also observed that some competition outcome has a high WP and a negative ADP.
A potentialreason can be explained as such agents are reckless to play out the bigger cards without consideringthe left hand, leading to winning many games of low score, but losing high score in the other gameswhen their hand cards are not good enough.
From our statistics of online human matches, the WPof winner is usually in a range of 0 .
52 ∼ 0 .
55 when the player tries to maximize its ADP.
[Table]
We further reveal the sample efficiency of
[Table]
PerfectDou by comparing the competing
[Table]
performance w.r.t.
different training steps.
As shown in Tab. 3, we compare two ver-
[Table]
sions of PerfectDou (1e9 and 2.5e9 steps)against two versions of DouZero (roughly1e9 and 1e10 steps).
From the tournamentin Tab. 2, we know the final version of PerfectDou (2.5e9 steps) outperforms the final version of
DouZero ( ∼ 1e10 steps).
However, to our surprise, an early stage of PerfectDou is able to beat
DouZero.
With the same 1e9 training samples, PerfectDou wins DouZero with a large gap (WPof 0.732 and ADP of 1.270), which is even better than the 1e10 sample-trained DouZero.
This indicates that PerfectDou is not only the best performance but also the most training efficient.
Therelated training curves are shown in Appendix D.5.
6.3 Ablation Studies
[Table]
We want to further investigate thekey to the success of our AI system( RQ2 ).
Specifically, we would like toanalyse how our design of the featureand the training framework help PerfectDou dominate the tournament of DouDizhu.
To this end, weevaluate different variants of PerfectDou and the previous SoTA AI system – DouZero, including: a)
ImperfectDouZero
: DouZero with our proposed imperfect-information features.
b) ImperfectDou :
PerfectDou with only imperfect-features as inputs for the value function.
c) RewardlessDou : PerfectDou without node reward.
d) Vanilla PPO : Naive actor-critic training with imperfect-featuresonly and without additional reward.
The ablation experiments are designed as competitions among
ImperfectDou, RewardlessDou and PerfectDou against DouZero for comparing the effectiveness ofperfect information distillation and perfect intermediate reward separately; while the battle of ImperfectDouZero and DouZero against PerfectDou are designed for excluding the benefit from featureengineering.
Results for all comparisons are shown in Tab. 4.
Even with the imperfect featuresonly, ImperfectDou can still easily beat DouZero with the same training steps; however, DouZeroturns the tide with much more training data.
Furthermore, our proposed node features seems not appropriate for DouZero to achieve a better results compared with its original design.
Additionally,without the node reward, PerfectDou still beats DouZero with higher WP (in spite of sacrificing alot of ADP), indicating the effectiveness of perfect reward in training, without which it would risklosing points to win one game.
Finally, without both node reward and perfect feature design forthe value function, vanilla PPO simply can not perform well.
Therefore, we can conclude that ouractor-critic based algorithm along with the PTIE training provides a high sample efficiency underour feature design, and the node reward benefits the rationality of our AI.
6.4 Runtime Analysis
[Figure]
We further conduct runtime analysis to show the efficiency of PerfectDou w.r.t.
the inference time ( RQ3 ),which is reported in Fig.
All evaluations are conducted on a single core of Intel(R)
Xeon(R)
Gold
CPU @ 2.10GHz.
The inference time of each AI couldbe attributed to its pipeline and implementation in theplaying time.
CQN uses a large Q network (nearly10 × parameters larger than ours) with a complex carddecomposer to derive reasonable hands.
As a result,the inference time of CQN is the longest.
Besides, both DeltaDou and RHCP-V2 contain lots oftimes of Monte Carlo simulations, thus slowing down the inference time.
As comparisons, DouZeroand PerfectDou only require one network forward inference time with a similar number of parameters.
For RLCard, only handcraft rules are computed.
Therefore, we can notice that PerfectDouis significantly faster than previous programs like DeltaDou, CQN and RHCP, yet is slightly slowerthan DouZero.
To be more accurate, the average inference time of DouZero is 2 milliseconds compared with 6 milliseconds of PerfectDou.
And the reason why PerfectDou is a bit slower than
DouZero may due to the more complex feature processing procedure.
Note that this can be furtheroptimized in practice such as changing the implementation from Python to C++, which is commonfor AI deployment.
In addition, we test the model inference time of DouZero is 1.9 ms while PerfectDou is 4 ms, the difference is slight.
The above analysis suggests that PerfectDou is applicableand affordable to real-world applications such as advanced game AI.
6.5 In-depth Analysis and Case Studies
In our experiments, we find that DouZero is leaky and unreasonable in many battle scenarios, while
PerfectDou performs better therein.
To quantitatively evaluate whether PerfectDou is stronger andmore reasonable, we conduct an in-depth analysis by collecting the statistics among the games, andadditionally illustrate some of the observations for qualitatively comparing the behavior of DouZeroand PerfectDou.
Detailed results can be further referred to Appendix D.2 and Appendix D.3, andhere we list the key conclusions below:
The statistic analysis claims the rationality of PerfectDou: (i) when playing as the Landlord , PerfectDou plays fewer bombs to avoid losing scores and tends to control the game even when the Peasantsplay more bombs; (ii) when playing as the Peasant , two PerfectDou agents cooperate better withmore bombs to reduce the control time of the Landlord and its chance to play bombs; (iii) whenplaying as the Peasant , the right-side Peasant agent (play after the Landlord ) of PerfectDou throwsmore bombs to suppress the Landlord than DouZero, which is more like human strategy.
From behavior observations, we also find: 1) DouZero is more aggressive but less thinking.
PerfectDouis better at guessing and suppressing.
3) PerfectDou is better at card combination.
4) PerfectDou ismore calm.
Beyond these, we also include battle results against skilled human players, additionaltraining results and complete tournament results in the appendix.
Conclusion
In this paper, we propose PerfectDou, a SoTA DouDizhu AI system that dominates the game.
PerfectDou takes the advantage of the perfect-training-imperfection-execution training paradigm, andis trained within a distributed training framework.
In experiments we extensively investigate howand why PefectDou can achieve the SoTA performance by beating all existing AI programs withreasonable strategic actions.
AppendixA Additional Related Work
Train with global information.
Utilizing global information to reduce the complexity ofimperfect-information games has also been investigated in some works.
For example, AlphaStar [25], a grand-master level AI system for StarCraft II.
In their implementation, the value networkof the agent can observe the full information about the game state, including those that are hiddenfrom the policy.
They argue that such a training style improves training performance.
In our work,we formulate the idea as Perfect-Training-Imperfect-Execution (PTIE) or perfect information distillation technique for imperfect-information games, and show the effectiveness on complicated cardgames like DouDizhu.
Moreover, in Suphx [14], a strong Mahjong AI system, they used a similarmethod namely oracle guiding.
Particularly, in the beginning of the training stage, all global information is utilized; then, as the training goes, the additional information would be dropped out slowlyto none, and only the information that the agent is allowed to observe is reserved in the subsequenttraining stage.
However, there are obvious difference between Suphx and PerfectDou.
In Suphx, theperfect information is used by the actor and thus has to be dropped before the inference stage; on thecontrary, PerfectDou feeds the critic with additional observations and distill the global informationto the actor.
Relation to sample-based CFR.
CFR aims to minimizes the total regret of policy by minimizingthe cumulative counterfactual regret in each infoset.
The definition of regret highly relates to thedefinition of advantage used in RL community, which has been shown in lots of previous works[22, 6].
Vanilla CFR [32] and many variants [26, 2] applies model-based approach to calculatedall the weights of the game tree to update and obtain a good strategy (policy).
However, when thegame has long episodes and is hard for searching across the game tree, it is necessary to computethrough trajectory samples, called sampled-based CFR methods [23, 7].
This resembles the learningprocedure of RL algorithms.
Recently, [6] proposed a new form of sample-based CFR algorithm,and shown that PPO is exactly a practical implementation of it (but not PTIE).
Thereafter, we cansee close connections between CFR and RL.
B More About DouDizhuB.1 Term of Categories
In the work of [30], they had shown a comprehensive introduction of DouDizhu game, so we thinkit may be wordy to repeat the stereotyped rules.
However, for better understanding the cases shownin this paper, we introduce the typical term of categories in DouDizhu that are commonly used asfollows.
Note that all cards can suppress the cards in the same category with a higher rank, yet bombcan suppress any categories except the bomb with a higher rank.
Rocket is the highest-rank bomb.
Kicker refer to the unrelated or useless cards that players can deal out when playing some kind ofcategories of main cards (see below), which can be either a solo or a pair.
Solo : Any single card.
Pair : Two matching cards of equal rank.
Trio : Three individual cards of equal rank.
Trio with Solo : Three individual cards of equal rank with a Solo as the kicker.
Trio with Pair : Three individual cards of equal rank with a Pair as the kicker.
Chain of Solo : Five or more consecutive individual cards.
Chain of Pair : Three or more consecutive Pairs.
Chain of Trio (Plane) :
Two or more consecutive Trios.
Plane with Solo : Two or more consecutive Trios with each has a distinct individual kicker
card and Plane as the main cards.
Quad with Solo : Four-of-a-kind with two Solos as the kicker and Four-of-a-kind as the
main cards.
Quad with Pair : Four-of-a-kind with a set of Pair as the kicker and Quad with Pair as the
[Table]
main cards.
[Table]
Quad with Pairs : Four-of-a-kind with two sets of Pair as the kicker and Quad with Pairs
[Table]
as the main cards.
[Table]
Bomb : Four-of-a-kind.
[Table]
Rocket : Red and black jokers.
[Table]
B.2 Scoring Rules
[Table]
In [30], they pay more attention to the win/lose result of the game but care less about the score.
[Table]
However, in real competitions, players must play for numbers of games and are ranked by the scorethey win.
And that is why we think ADP is a better metric for evaluating DouDizhu AI systems [Table] because a bad AI player can win a game with few scores but lose with much more scores. [Table]
Specifically, in each game, the Landlord and the Peasants have base scores of 2 and 1 respectively.
[Table]
When there is a bomb shown in a game, the score of each player doubles.
For example, a Peasantplayer first shows a bomb of 4 and then the Landlord player suppresses it with a rocket, then thebase score of each Peasant becomes 4 and the Landlord becomes 8.
A player will win all his scoresafter winning the game, or loses all of them vice versa.
C Additional System Design Details
[Table]
C.1 Card Representation Details
[Table]
In the system of PerfectDou, we augment the basic card in hand matrix with explicitly encoded cardtypes as additional features, in order to allow the agent realizing the different properties of differentkind of cards.
The size details are shown in Tab. 5.
[Table]
C.2 Action Feature Details
[Table]
[Table]
The action features are a flatten matrix from 12 × 15 action card matrix plus 1 × 6 extra dimensionsdescribing the property of the cards as shown in Tab. 6.
Since the number of actions in each gamestate varies, which can lead to different lengths of action features, a fixed length matrix is flattenedto store all action features where the non-available ones are marked as zero.
C.3 Value Network Structure
[Figure]
The value network of PerfectDou is designed to evaluate the current situation of players, and weexpect that the value function can utilize the global information, in other words, know the exactnode the player is in.
Therefore, we should feed additional information that the policy is not allowedto see in our design.
Specifically, as shown in Fig.
6, the imperfect feature for indistinguishablenodes is encoded using the shared network as in the policy network; besides, we also encode theperfect feature of distinguishable nodes that the policy cannot observe during its game playing.
Theencoded vector are then concatenated to a simple MLP to get the scalar value output.
[Figure]
D Experiments
[Figure]
D.1 Hyperparameters and Setups
In our implementation, a small distributed training cluster is built using 880 CPUs cores and
GPUs.
Horovod [19] is used to synchronize gradients between GPUs, the total batch size is 1024,128 for each GPU.
During self-play, we find a better practical solution for DouDizhu is to keep threedifferent models for Landlord and two Peasants separately which is only updated by their own dataagainst the latest opponent model.
In the early training stage, the total reward function will be a basicreward (either WP or ADP) augmented with the designed oracle reward as shown in Section 4.4 tohelp convergence.
In the later stage, the augmented reward will be removed.
The most importanthyperparameters in our experiment are shown in Tab. 7.
Specifically, in our design, we simplify thediscrete action space from 27472 (include all possible combinations) into an abstract action spaceof 621 for learning the actor, followed by an decoding strategy to get the final action.
D.2 In-Depth Statistical Analysis
In our experiments, we find that DouZero is leaky and unreasonable in many battle scenarios, while
PerfectDou performs better therein.
To quantitatively evaluate whether PerfectDou is stronger andmore reasonable, we conduct an in-depth analysis and collect the statistics among the games between
DouZero and PerfectDou.
Particularly, we organize games between PerfectDou and Douzero toplay in different roles for 100,000 decks in each setting.
Since the roles are assigned randomlyinstead of opting by agents themselves in our experiments, and the Landlord has a higher base scorewith three extra cards, we observe that playing as a Landlord is always harder to win and leads tonegative ADPs.
From the statistics shown in Tab. 8, we learn many lessons about the rationalityof PerfectDou: (i) when playing as the Landlord , PerfectDou plays fewer bombs to avoid losingscores and tends to control the game even the Peasants play more bombs; (ii) when playing as the
Peasant , two PerfectDou agents cooperate better with more bombs to reduce the control time of the
Landlord and its chance to play bombs; (iii) when playing as the Peasant , the right-side Peasant
[Table]
[Table]
agent (play after the Landlord ) of PerfectDou throws more bombs to suppress the Landlord than
DouZero, which is more like human strategy
[Table]
D.3 Case Study: Behavior of DouZero vs PerfectDou
In this section, we list some of the observations during the games for comparing the behavior of
DouZero and PerfectDou to qualitatively support our analysis.
DouZero is more aggressive but less thinking.
The first observation is that DouZero is extremelyaggressive without considering the left hands.
For instance, as shown in Fig. 7(a), in the beginning
DouZero chooses a chain of solo but leaves the pair of 3, which can be dangerous since the pair of3 is the one of the minimum cards and cannot suppress any card; Fig. 7(b) illustrates another strongcase, where DouZero also chooses a chain of solo to suppress the opponent without consideringthe consequence of leaving a hand of solos.
On the contrary, PerfectDou is more conservative andsteady.
We believe the proposed perfect information distillation mechanism helps PerfectDou toinfer global information in a more reasonable way.
PerfectDou is better at guessing and suppressing.
We observe another fact that the usage ofperfect information distillation within the PTIE framework benefits PerfectDou a lot by suppressingthe opponents in advance.
In Fig. 7(c) shows a case when the teammate puts a pair of T 4 , DouZerochooses to pass; on the contrary, PerfectDou chooses suppressing by a pair of Q – the minimal pairof the Landlord .
PerfectDou is better at card combination.
In the battle shown in Fig.
7(d), PerfectDou showsthe better ability on the strategy of card combination.
Specifically, PerfectDou chooses to split theplane (999, TTT since it considers there is a chain of solo ( 9 TJQK ) left.
However, DouZero onlytakes the trio, which will be easily suppressed by the opponent.
This benefits from the proper designof the card representation and the action feature of PerfectDou.
P e a s a n t P e a s a n t
Landlord
Last hand: NULL
PerfectDou
DouZero
P e a s a n t
L a nd l o r d
Peasant
Last hand: Landlord
PerfectDou
DouZero (
[Figure]
P e a s a n t
[Figure]
P e a s a n t P e a s a n t
Landlord
[Figure]
: NULL
[Figure] [Figure]
PerfectDou is more calm.
Fig.
7(e) depicts a typical and interesting scenario where PerfectDoushows its calm and careful consideration over the whole.
In the game, the last hand is of the Landlordwith a solo 2, and it only has 8 cards left in the hand.
DouZero seems afraid and splits the rocketbomb; however, PerfectDou benefits from the advantage reward design and is calm considering thereis a greater chance on winning the game with a higher score by keeping the bomb.
D.4 Battle Results Against Skilled Human Players
We further invite some skilled human players to play against PerfectDou.
Particularly, each humanplayer plays with two AI players.
In other words, each game is involved with either two AI Peasantsagainst one human Landlord , or one AI Peasant cooperating with one human Peasant against onehuman Landlord .
The results are shown in Tab. 9.
One can easily observe that PerfectDou takesevident advantage during the game.
[Figure]
[Figure]
[Table]
D.5 Additional Training Results
[Figure]
Fig.
8 shows the learning curves of WP and ADP against DouZero for PerfectDou with a singlerun, and every evaluation contains 10000 decks.
As shown in the figure, PerfectDou can easily beat
DouZero (on WP) without considering the scores (ADP) at the beginning of the training; but after1.5e9 steps of training, PerfectDou is able to fully beat DouZero (both WP and ADP).
D.6 Extended Competition against DouZero
Cycling Different Hands for the Landlord .
In our main competition conducted in Section 6.2, allgames are randomly generated and played twice with the same assigned hand cards for the Landlord ,once each algorithm controlling the Landlord and once two Peasants .
In this section, we test cyclingthe 3 hand cards of every randomly generated game for the Landlord , and test PerfectDou against
DouZero by controlling the Landlord seperately, which leads to 6 times of battle.
In this situation,we obtain the results as follows showing that PerfectDou has consistent advantage (Tab. 10).
[Table]
Peasants Paired with Different AIs.
We also include an interesting battle by pairing the peasants with different algorithms.
Specifically, DouZero or PerfectDou plays the Landlord while one
Peasant is assigned with DouZero and the other Peasant is played by PerfectDou.
The results areconcluded in Tab. 11, which reveals that when playing as the peasant, PerfectDou can better cooperate with its teammate than DouZero.
And when playing as the landlord, PerfectDou outperforms
DouZero against all types of opponents.
[Table]
[Table] [Table]
D.7 Complete Tournament Results of ADP for Landlord and Peasants
We report the complete tournament results of ADP and WP for Landlord and Peasants in Tab.
12and
Tab. 13.
PerfectDou tends to have more advantage of Peasants than that of Landlord , especiallywhen compete against stronger baselines.
We believe that the proposed perfect informtation distillation technique allows for better cooperation between two Peasants .
In addition, since the roles areassigned instead of opting according to hand in our competition
[Table]
and the Landlord has extra threecards and can lose a higher base score, the Peasants seems having more chance to win the game.
Therefore, almost all methods can play better results as a Peasant than that as a Landlord .
[Table]
E More Implementation DetailsE.1 The Oracle for Minimum Steps to Play Out All cards
In our paper, as mentioned in Section 4.4, we utilize an oracle for evaluating the minimum steps toplay out all cards.
Particularly, the oracle is implemented by a dynamic programming algorithmcombined with depth-first-search, which can be referred to https://www.cnblogs.com/
SYCstudio/p/7628971.html (which is also a competition problem of National Olympiadin Informatics in Provinces (NOIP) 2015).
We also summerize the pseudocode for implementingsuch an algorithm in Algo.
E.2 Detailed Action Space
In our paper, we utilize a simplified discrete action space of 621 for learning the actor, since weobserve that the original action space of 27472 (include all possible combinations) contains a largenumber of actions that can be abstract.
For instance, actions like Bomb with kickers and Trio withkickers occupy the action space most due to the large number of combinations of kickers.
To thisend, we abstract actions with the same main cards into one action, and significantly reduce theaction space.
When the policy chooses an abstract action, we further deploy a simple decodingfunction to obtain the most preferred action in the original action space, as illustrated in Algo. 2.
Wenote that this part is similar to an old implementation of RLCard [29]: https://github.com/datamllab/rlcard/blob/d100952f144e4b0fd7186cc06e79ef277cda9722/rlcard/envs/doudizhu.py#L67 .
Below we list all the 621 discrete actions of PerfectDou in 15 categories, where the notation * incategory (9) (10) (11) and (12) denotes the kicker.
(1) Solo (15 actions): 3, 4, 5, 6, 7, 8, 9, T, J, Q, K, A, 2, B, R
(2) Pair (13 actions): 33, 44, 55, 66, 77, 88, 99, TT, JJ, QQ, KK, AA, 22(3) Trio (13 actions):
[Table]
[Table]
[Table]
[Table] [Table] [Table] , TTT
[Table]
JJJ
[Table]
QQQ, KKK,
[Table]
, 222(4) Trio with Solo
[Table]
actions):
[Table]
333A
[Table]
,4442, 444B, 444R, [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
, 555J,
[Table]
, 555K, 555A, 5552,555B, 555R, 3666, 4666, 5666,
[Table]
[Table] [Table] [Table] [Table]
, 666Q, 666K,
[Table] [Table] [Table] [Table]
666R, 3777, 4777, 5777,
[Table] [Table] [Table] [Table]
[Table]
, 777Q, 777K, 777A,
[Table]
, 777B,
[Table]
,3888, 4888, 5888, 6888, 7888, 8889, 888T, 888J, 888Q, 888K, 888A, 8882, 888B, 888R, 3999,4999, 5999, 6999, 7999, 8999, 999T, 999J, 999Q, 999K, 999A, 9992, 999B, 999R, 3TTT, 4TTT,5TTT, 6TTT, 7TTT, 8TTT, 9TTT, TTTJ, TTTQ, TTTK, TTTA, TTT2, TTTB, TTTR, 3JJJ, 4JJJ,5JJJ, 6JJJ, 7JJJ, 8JJJ, 9JJJ, TJJJ, JJJQ, JJJK, JJJA, JJJ2, JJJB, JJJR, 3QQQ, 4QQQ, 5QQQ, 6QQQ,7QQQ, 8QQQ, 9QQQ, TQQQ, JQQQ, QQQK, QQQA, QQQ2, QQQB, QQQR, 3KKK, 4KKK,5KKK, 6KKK, 7KKK, 8KKK, 9KKK, TKKK, JKKK, QKKK, KKKA, KKK2, KKKB, KKKR,3AAA, 4AAA, 5AAA, 6AAA, 7AAA, 8AAA, 9AAA, TAAA, JAAA, QAAA, KAAA, AAA2,
AAAB, AAAR, 3222, 4222, 5222, 6222, 7222, 8222, 9222, T222, J222, Q222, K222, A222, 222B,222R(5) Trio with Pair (156 actions): 33344, 33355, 33366, 33377, 33388, 33399, 333TT, 333JJ, 333QQ,333KK, 333AA, 33322, 33444, 44455, 44466, 44477, 44488, 44499, 444TT, 444JJ, 444QQ,444KK, 444AA, 44422, 33555, 44555, 55566, 55577, 55588, 55599, 555TT, 555JJ, 555QQ,555KK, 555AA, 55522, 33666, 44666, 55666, 66677, 66688, 66699, 666TT, 666JJ, 666QQ,666KK, 666AA, 66622, 33777, 44777, 55777, 66777, 77788, 77799, 777TT, 777JJ, 777QQ,777KK, 777AA, 77722, 33888, 44888, 55888, 66888, 77888, 88899, 888TT, 888JJ, 888QQ,888KK, 888AA, 88822, 33999, 44999, 55999, 66999, 77999, 88999, 999TT, 999JJ, 999QQ,999KK, 999AA, 99922, 33TTT, 44TTT, 55TTT, 66TTT, 77TTT, 88TTT, 99TTT, TTTJJ, TTTQQ,
TTTKK, TTTAA, TTT22, 33JJJ, 44JJJ, 55JJJ, 66JJJ, 77JJJ, 88JJJ, 99JJJ, TTJJJ, JJJQQ, JJJKK,
JJJAA, JJJ22, 33QQQ, 44QQQ, 55QQQ, 66QQQ, 77QQQ, 88QQQ, 99QQQ, TTQQQ, JJQQQ,
QQQKK, QQQAA, QQQ22, 33KKK, 44KKK, 55KKK, 66KKK, 77KKK, 88KKK, 99KKK,
TTKKK, JJKKK, QQKKK, KKKAA, KKK22, 33AAA, 44AAA, 55AAA, 66AAA, 77AAA,88AAA, 99AAA, TTAAA, JJAAA, QQAAA, KKAAA, AAA22, 33222, 44222, 55222, 66222,77222, 88222, 99222, TT222, JJ222, QQ222, KK222, AA
1: function M AIN 2:
N 1 ← all possible number of single card 3:
N 2 ← all possible number of pair card 4:
N 3 ← all possible number of trio card 5:
N 4 ← all possible number of bomb card 6: function I NIT M ATRIX ( F ) 7: for action ∈ { Solo , Pair , Trio , Bomb , Chain-of-Trio , Trio-with-Pair ,
Quad-withSolos , Quad-with-Pair , Quad-with-Pairs } do 8: d 1 ← the number of Solo card in action 9: d 2 ← the number of Pair card in action 10: d 3 ← the number of Trio card in action 11: d 4 ← the number of Bomb card in action 12:
F [ N 1 , N 2 , N 3 , N 4 ] ← min( F [ N 1 , N 2 , N 3 , N 4 ] , F [ N 1 − d 1 , N 2 − d 2 , N 3 − d 3 , N 4 − d 4 ] + 1) 13: if action is Trio then 14:
F [ N 1 , N 2 , N 3 , N 4 ] ← min( F [ N 1 , N 2 , N 3 , N 4 ] , F [ N 1 +1 , N 2 +2 , N 3 − 1 , N 4 ]) 15: end if 16: if action is Bomb then 17:
F [ N 1 , N 2 , N 3 , N 4 ] ← min( F [ N 1 , N 2 , N 3 , N 4 ] , F [ N 1 +1 , N 2 +2 , N 3 − 1 , N 4 ]) 18: end if 19: end for 20: end function 21: function N OW S TEP ( Cards ) 22: if Rocket ∈ Cards then 23: left cards ← left cards after playing out Rocket 24: return min ( NowStep ( left cards ) + 1, NowStep ( left cards ) + 2 ) 25: end if 26: if only one Joker ∈ Cards then 27: left cards ← left cards after playing out Joker 28: return NowStep ( left cards )
+29: end if 30:
if no Joker ∈ Cards then 31: calculate number of Solo N 1 , Pair N 2 , Trio N 3 and Bomb N 4 of Cards 32: return F [ N 1 , N 2 , N 3 , N 4 ] 33: end if 34: end function 35: function DFS ( step , ans , Cards ) 36: if step > ans then 37: return ans 38: end if 39: ans ← min ( step , NowStep ( Cards ) ) 40: for Chain-of-Solo ∈ Cards do 41: left cards ← left cards after playing out Chain-of-Solo 42:
DFS ( step + 1, ans , left cards of Cards ) 43: end for 44: for Chain-of-Pair in Cards do 45: left cards ← left cards after playing out Chain-of-Pair 46:
DFS ( step + 1, ans , left cards of Cards ) 47: end for 48: for Plane-with-Solo ∈ Cards do 49: left cards ← left cards after playing out Plane-with-Solo 50:
DFS ( step + 1, ans , left cards of Cards ) 51: end for 52: end function 53:
Create a matrix F of size [ N 1 , N 2 , N 3 , N 4 ] 54:
InitMatrix ( F ) 55: step ← 0 , ans ← + ∞ , Cards ← all Cards to be calculated 56:
DFS ( 0, ans , Cards ) 57: end function
1: function D ECODE ( M ) 2:
A ← get all available actions from current hand 3:
K ← get all kickers using the main card M from A 4: for k in K do 5: calculate score s of each k 6:
N ← number of actions contains k in A , rank k ← card rank of k 7: s ← 1 . 0 ∗ N + 0 .
1 ∗ rank k 8: end for 9: return k with minimum s 10: end function (6) Chain of Solo (36 actions): 34567, 45678, 56789, 6789T, 789TJ, 89TJQ, 9TJQK, TJQKA,345678, 456789, 56789T, 6789TJ, 789TJQ, 89TJQK, 9TJQKA, 3456789, 456789T, 56789TJ,6789TJQ, 789TJQK, 89TJQKA, 3456789T, 456789TJ, 56789TJQ, 6789TJQK, 789TJQKA,3456789TJ, 456789TJQ, 56789TJQK, 6789TJQKA, 3456789TJQ, 456789TJQK, 56789TJQKA,3456789TJQK, 456789TJQKA, 3456789TJQKA(7) Chain of Pair (52 actions): 334455, 445566, 556677, 667788, 778899, 8899TT, 99TTJJ,
TTJJQQ, JJQQKK, QQKKAA, 33445566, 44556677, 55667788, 66778899, 778899TT, 8899TTJJ,99TTJJQQ, TTJJQQKK, JJQQKKAA, 3344556677, 4455667788, 5566778899, 66778899TT,778899TTJJ, 8899TTJJQQ, 99TTJJQQKK, TTJJQQKKAA, 334455667788, 445566778899,5566778899TT, 66778899TTJJ, 778899TTJJQQ, 8899TTJJQQKK, 99TTJJQQKKAA,33445566778899, 445566778899TT, 5566778899TTJJ, 66778899TTJJQQ, 778899TTJJQQKK,8899TTJJQQKKAA, 33445566778899TT, 445566778899TTJJ, 5566778899TTJJQQ,66778899TTJJQQKK, 778899TTJJQQKKAA, 33445566778899TTJJ, 445566778899TTJJQQ,5566778899TTJJQQKK, 66778899TTJJQQKKAA, 33445566778899TTJJQQ,445566778899TTJJQQKK, 5566778899TTJJQQKKAA(8) Chain of Trio (45 actions): 333444, 444555, 555666, 666777, 777888, 888999, 999TTT, TTTJJJ,
JJJQQQ, QQQKKK, KKKAAA, 333444555, 444555666, 555666777, 666777888, 777888999,888999TTT, 999TTTJJJ,
TTTJJJQQQ,
JJJQQQKKK,
QQQKKKAAA, 333444555666,444555666777, 555666777888, 666777888999, 777888999TTT, 888999TTTJJJ, 999TTTJJJQQQ,
TTTJJJQQQKKK,
JJJQQQKKKAAA, 333444555666777, 444555666777888,555666777888999, 666777888999TTT, 777888999TTTJJJ, 888999TTTJJJQQQ, 999TTTJJJQQQKKK,
TTTJJJQQQKKKAAA, 333444555666777888, 444555666777888999,555666777888999TTT, 666777888999TTTJJJ, 777888999TTTJJJQQQ, 888999TTTJJJQQQKKK, 999TTTJJJQQQKKKAAA(9) Plane with Solo (38 actions): 333444**, 444555**, 555666**, 666777**, 777888**,888999**, 999TTT**, TTTJJJ**, JJJQQQ**, QQQKKK**, KKKAAA**, 333444555***,444555666***, 555666777***, 666777888***, 777888999***, 888999TTT***, 999TTTJJJ***,
TTTJJJQQQ***, JJJQQQKKK***, QQQKKKAAA***, 333444555666****, 444555666777****,555666777888****, 666777888999****, 777888999TTT****, 888999TTTJJJ****, 999TTTJJJQQQ****,
TTTJJJQQQKKK****,
JJJQQQKKKAAA****, 333444555666777*****,444555666777888*****, 555666777888999*****, 666777888999TTT*****, 777888999TTTJJJ*****, 888999TTTJJJQQQ*****, 999TTTJJJQQQKKK*****, TTTJJJQQQKKKAAA*****(10) Plane with Pair (30 actions): 333444****, 444555****, 555666****, 666777****,777888****, 888999****, 999TTT****,
TTTJJJ****,
JJJQQQ****,
QQQKKK****,
KKKAAA****, 333444555******, 444555666******, 555666777******, 666777888******,777888999******, 888999TTT******, 999TTTJJJ******,
TTTJJJQQQ******,
JJJQQQKKK******, QQQKKKAAA******, 333444555666********, 444555666777********,555666777888********, 666777888999********, 777888999TTT********,888999TTTJJJ********, 999TTTJJJQQQ********,
TTTJJJQQQKKK********,
JJJQQQKKKAAA********(11)
Quad with Solo (13 actions): 3333**, 4444**, 5555**, 6666**, 7777**, 8888**, 9999**,
TTTT**, JJJJ**, QQQQ**, KKKK**, AAAA**, 2222**
(12) Quad with Pair (13 actions):
, QQQQ
, AAAA
13) Bomb (
, QQQQ, KKKK,
AAAA
(14) Rocket (1 action): BR
Pass
: PASS
F Additional Discussions
The main technique proposed by the paper is the perfect-training-imperfect-execution (PTIE) framework, where we expect the value function can distill the perfect information to the policy whichcan only receive imperfect information.
Intuitively, PTIE is a general way for training imperfectinformation game agents.
Although in this paper we only discuss its success on one of the hard pokergames, DouDizhu, we believe it has the power to further improve the ability for other imperfectinformation games, remaining a space of imagination to be explored by more future works.