M ODELING S TRONG AND H UMAN - LIKE G AMEPLAY WITH KL-R EGULARIZED S EARCH
A BSTRACT
We consider the task of building strong but human-like policies in multi-agent decision-making problems, given examples of human behavior.
Imitation learning is effective at predicting human actions but may not match the strength of expert humans, while self-play learning and search techniques (e g. AlphaZero) lead to strong performance but may produce policies that are difficult for humans to understand and coordinate with.
We show in chess and Go that regularizing search based on the KL divergence from an imitation-learned policy results in higher human prediction accuracy and stronger performance than imitation learning alone.
We then introduce a novel regret minimization algorithm that is regularized based on the KL divergence from an imitation-learned policy and show that using this algorithm for search in no-press Diplomacy yields a policy that matches the human prediction accuracy of imitation learning while being substantially stronger
I NTRODUCTION
Self-play AI algorithms have matched or exceeded expert human performance in many games, such as chess (Campbell et al., 2002; Silver et al., 2018), Go (Silver et al., 2016; 2017) and poker (Moravˇcík et al., 2017; Brown & Sandholm, 2017; 2019).
However, the resulting policies often differ markedly from how humans play (McIlroy-Young et al., 2020a).
This is a serious problem for human-computer interactions that involve cooperation rather than purely competition.
In such settings, modeling the other participants accurately is important for success.
For example, it is important for a self-driving car at a four-way stop sign to conform to existing human conventions rather than its own self-play solution to the problem (Lerer & Peysakhovich, 2019).
Even in purely adversarial games, the alien nature of AI policies makes it difficult for humans to understand and learn from superhuman bots.
The classic approach toward modeling human behavior is imitation learning (IL) on human data.
Yet, evidence in multiple games indicates that IL on expert human data produces policies that are much weaker than actual expert human play in domains with complex strategic planning.
In this paper, we study the problem of producing policies that are both strong and human-like in games with complex strategic planning like chess, Go, Hanabi, and Diplomacy.
In all four, we find that conducting search with KL-regularization towards an IL policy matches or exceeds the prior state of the art for prediction accuracy of expert humans while also better matching expert human performance.
In Section 3, we show that Monte Carlo tree search (MCTS) with a human imitation-learned policy prior and value function surpasses prior state-of-the-art results for human prediction accuracy in chess and Go.
As explained by Grill et al.
(2020), standard MCTS with a policy prior can be viewed as a form of KL-regularized search, optimizing a value function subject to a KL-divergence term with that prior.
Although MCTS has been extensively studied for developing strong agents, it has been explored much less in the context of developing human-like agents.
Section 4 builds on these findings and shows how to generalize them to a class of imperfectinformation games (in which ordinary MCTS is unsound and cannot be applied) via a new algorithm for KL-regularized regret minimization.
We show that existing regret minimization algorithms achieve low accuracy in predicting expert human actions in no-press Diplomacy.
We then introduce the first regret minimization algorithm to incorporate a cost term proportional to the KL divergence between the search policy and a human-imitation learned anchor policy .
We call this algorithm policy-regularized hedge , or piKL-hedge .
We prove that piKL-hedge converges to an equilibrium in which all players’ policies are optimal given the joint policies of the players and the cost of deviating from the anchor policy.
We then present results in no-press Diplomacy showing that piKL-hedge produces policies that predict human actions as accurately as imitation learning while also improving head-to-head performance in a population of prior agents.
Appendix M additionally shows that applying KL-regularization toward a human IL policy in the search algorithm SPARTA Lerer et al.
(2020) produces similar or better human prediction accuracy while greatly improving performance in the benchmark domain of Hanabi Bard et al.
Our experiments demonstrate the benefits of KL-regularized search in all four of chess, Go, no-press
Diplomacy, and Hanabi to producing agents that are simultaneously more human-like and closer in strength to actual human experts than purely imitation-learned models.
P RELIMINARIES
We study the problem of learning policies for multiplayer games.
Here we briefly introduce the key ingredients of both classes of games we study; Section 3 and Section 4 give a more formal presentation tailored to individual game types and learning algorithms.
An ( N -player) game is defined by a state space S , an action space A , a (deterministic) transition function T : S × A N → S , and a collection of reward functions
u i .
We model the behavior of each player in a game as a policy
π
i : S → ∆( A ) (a distribution over actions given states).
In every round of a game, each player observes a (possibly incomplete) view s ti of the current state.
One or more players select actions a ti ∼ π i ( · | s ti ) , then each player receives a reward u ti ( s t , a t = a t 1 , . . .
, a tn ) , and the game transitions into a new state s t
+1 =
T ( s t , a t ) .
Each player i aims to maximizes its expected reward, and the optimal policies for doing so may depend on the policies π − i
= { π 1 , . .
π i − 1 , π i +1 , . .
, π N } of the other players.
The sequential decision-making problem described above is extremely general, and in this paper we focus on two special cases.
In perfect-information games, players make moves sequentially (e.g., u 1 and s 2 depend only on a 1 , u 2 and s 3 depend only on a 2 , etc.).
Many important games, including chess and Go, fall into this category.
Next, we study a more general class of imperfect-information, simultaneous-action games that make no assumptions about the dependence of different u i and
T on a ; here we focus on games with only a single round, also called matrix games.
Owing to the large differences between these two settings, the tools for computing strong policies are quite different.
The remainder of this paper accordingly offers a deeper exploration of each class of games: perfect-information games in Section 3 and imperfect-information games in Section 4.
P ERFECT -I NFORMATION G AMES : P OLICY R EGULARIZATION IN M ONTE C ARLO T REE S EARCH
In this section, we focus on developing strong human-like agents for perfect-information games.
Monte Carlo tree search (MCTS) has been highly successful for developing strong, but not necessarily human-like agents in this setting, and is a key component of general learning algorithms such as
AlphaZero and MuZero capable of achieving superhuman performance in chess, Go, and similar games (Silver et al., 2018; Schrittwieser et al., 2020).
By contrast, for developing human-like agents, the best prior human move prediction accuracies for chess and Go were all achieved with pure imitation learning on human data McIlroy-Young et al.
(2020a)
; Cazenave (2017); Silver et al. (2017).
The state of the art for predicting human moves in chess is the Maia engine created by McIlroy-Young et al.
(2020a) via pure imitation learning without any search.
However, this approach appears to be of limited effectiveness for modeling sufficiently strong humans.
Although the weakest Maia models at low temperatures appear to outperform the players they imitate (due to “averaging away” of the imitated players’ individual idiosyncratic mistakes (Anderson et al., 2021)) each successive model on data from stronger players improves by much less than the players improve.
The strongest model, trained to predict human 1900-1999 rated players, even with low temperature appears to be clearly below a 1950-average level of performance in all but the minority of bullet-speed games (in which very little time is available for planning and players are forced to rely more heavily on intuition).
Similarly, in Go, pure imitation-learning agents have not exceeded mid-expert level on various online servers despite being trained on top-expert and master-level games (Cazenave, 2017).
In contrast, search-based reinforcement learning (RL) agents such as AlphaZero that do not use a human policy prior play at a superhuman level, but often in non-human ways that humans find difficult to understand even when given access to interactively query and inspect the agent’s analysis (Silver et al., 2017; Egri-Nagy & Törmänen, 2020).
However, we show in both chess and Go that if the human-learned model is used in MCTS with appropriate parameters, MCTS outperforms those models’ human prediction accuracy while simultaneously reducing the shortcomings in those models’ strength.
3. B ACKGROUND
We consider sequential games where each player i alternatively chooses action a from a policy
π i where, a ∼ π i ( · | s ) .
Each action deterministically transitions the game into a new state s ′ =
T ( s, a ) and gives rewards u i ( s, a ) .
Notationally, we may elide the player i in some places when it is clear that i is the next player to move in the state being considered.
For this work, following Silver et al.
, we implement one of the most common modern forms of MCTS, which uses a value function V predicting the expected total future reward
V i ( s ) =
E [ (cid:80) t u i ( s t , a t ) | π 1 , π 2 , s
= s ] and a policy prior τ (typically both the outputs of a trained deep neural net) and attempts to produce an improved policy π .
Each turn, MCTS builds a game tree over multiple iterations rooted at the current state.
Each iteration t , MCTS explores a single path down the tree by simulating at each successive state s with player i to move the action:
[Equation]
where Q ( s, a ) is the estimated expected future reward for i from playing action a in state s , the visit count N ( s, a ) is the number of times a has been explored from s , τ ( s, a ) is the prior policy probability, and c puct is a tunable parameter trading off exploration versus exploitation.
Upon reaching a state s t not yet seen, MCTS queries the value function V i ( s t ) for each player i , and based on V i ( s t ) and any intermediate rewards received, updates all Q ( s, a ) estimates on the path traversed.
The final agent policy is π ( s, a ) =
N ( s, a ) / (cid:80) b N ( s, b ) where s is the root state, or optionally we may also have π ( s, a ) ∼ N ( s, a ) 1 /T where T is a temperature parameter.
See also
Appendix G for a fuller description of MCTS.
Grill et al.
(2020) show that the agent policy π computed by this form of MCTS is an approximate solution to the optimization problem:
[Equation]
where λ ∼ c puct √ N and N is the total number of iterations.
[Figure]
MCTS on top of current state-of-the-art models improves human prediction accuracy significantly
[Figure]
In other words, at every node of the tree recursively, MCTS implicitly optimizes its expected future reward subject to KL regularization of its policy towards the prior policy τ with strength controlled by λ .
For any fixed computational budget N , we can therefore tune c puct to vary the strength of that prior, with c puct = ∞ approximating the prior policy before search, and c puct → 0 approaching a greedy argmax of the Q value estimates.
If our goal is a strong human-like agent rather than solely a strong agent, and the KL-regularizing policy is learned from human data, then that policy serves not just as a prior, but also as an anchor policy that regularizing towards is desirable in and of itself.
With good choice of c puct , MCTS can improve that policy while remaining close to human.
Our experiments confirm that MCTS improves on the strength and human prediction accuracy of the best existing models in both chess and Go.
3. E XPERIMENTS IN C HESS AND G O
In chess and Go, we ran two main experiments each.
First, in chess using the prior state-of-the-art
Maia models from McIlroy-Young et al.
(2020a)
and in Go using a model trained on professional games from the GoGoD dataset, we demonstrate that MCTS with that model outperforms the raw model in human prediction accuracy.
Second, we sanity-check that MCTS with the same parameters greatly improves strength relative to the raw imitation policy of those models in chess and Go.
3.2.1 Data and Model Architecture
In chess, for the human-learned anchor policy we use the pre-trained Maia1100, Maia1500, and Maia1900 models from McIlroy-Young et al.
(2020a), achieving state-of-the-art performance on rating-conditional human move prediction.
These models follow a standard AlphaZero-like residual block architecture, including both a policy and a value head, and were trained to imitate players in Elo ratings “buckets” 1100, 1500, and 1900 respectively, based on roughly 10 million games each from the popular Lichess server (each bucket contains games between players from rating N to N+99) from January 2016 to November 2019.
We then test on the same holdout set of December 2019 games as McIlroy-Young et al.
(2020a).
For Go, we trained a deep neural net on the GoGoD professional game dataset 2 .
We match Cazenave (2017) in using games from 1900 through 2014 for training and 2015-2016 as the test set, with roughly 73000 and 6500 games, respectively.
Our architecture matches the 20-block residual net structure used by some versions of AlphaZero (Silver et al., 2017), except adds squeeze-and-excitation layers which have been successful in image processing tasks (Hu et al., 2018) and self-play learning in chess and Go (LC0, 2020; Troisi, 2019).
See Appendix F for additional details.
3.2.2 Improved Human Prediction and Strength
In Table 1 we show the top-1 accuracy of MCTS (i.e. frequency with which MCTS’s top move matches the actual human move) at predicting players in chess and Go.
MCTS on top of each model tested outperforms that model at predicting human moves. [Figure]
Standard error is very small, the slight thin shade around each line.
MCTS most improves prediction of a model on players of its target rating and higher.
In chess, the benefit provided increases greatly as the rating of players predicted increases, while the optimal choice for c puct appears to decrease (allowing increasingly small value differences to affect the search).
This is consistent with the intuitive hypothesis that stronger players plan more deeply, increasing the value of explicitly modeling planning, and that they are more sensitive to small future value differences.
In Go, despite our baseline model being equal or better than all prior imitation-learning models on the GoGoD human pro games dataset, MCTS improves it yet further.
In Figure 2, for chess we see that while KL-regularized search improves each model’s accuracy on players of its target rating, surprisingly, the improvement grows yet larger when each model predicts players of higher rating than it was trained on.
This suggests that as human players improve, the incremental average change in their behavior resembles or is correlated with the way that highly-regularized search improves the strength of a baseline policy.
Additionally, in Appendix D, we show that if we apply post-processing to the MCTS policy based on Grill et al.
(2020), MCTS improves cross entropy with human moves in both chess and Go, not just top-1 accuracy.
In other words, not only does policy-regularized search improve the prediction of the top move, but it also better models the overall distribution of moves that humans may likely play.
We measure the strength impact of regularized search with 1000 games 3 per c puct setting between the raw model policy and the MCTS policy, sampling each at temperature 1.
Figure 1 shows the change in human prediction accuracy of MCTS in both chess and Go plotted jointly versus winrate of MCTS against the raw model.
Rather than solely a tradeoff between strength and accuracy, most c puct values in the range we tested increase both , some achieving more than 90% winrate while still improving human prediction.
See Appendix E for results at lower temperature and evidence that accuracy improves further at longer time controls.
Although we did not test against humans directly to calibrate, this gives clear evidence that a welltuned human-regularized MCTS agent would be better able to match the 1900-1999-rated chess players that Maia1900 currently falls hundreds of Elo short of imitating, while simultaneously being more accurate to their move-by-move behavior, and similarly for human-imitation agents in Go.
I MPERFECT -I NFORMATION G AMES : P OLICY - REGULARIZED R EGRET M INIMIZATION
While MCTS is a popular search algorithm for perfect-information deterministic games, it is not able to compute optimal policies in imperfect-information games.
Instead, iterative algorithms based on regret minimization are the leading approach to search in imperfect-information games.
Hedge (Littlestone & Warmuth, 1994; Freund & Schapire, 1997) is an iterative regret minimization algorithm that in general converges to a coarse correlated equilibrium (CCE) (Hannan, 1957).
In the special case of two-player zero-sum games, it also converges to a Nash equilibrium (NE) (Nash, 1951).
Regret Matching (RM) (Blackwell et al., 1956; Hart & Mas-Colell, 2000) is another equilibriumfinding algorithm similar to Hedge that has historically been more popular and that we compare our algorithm to in this paper.
The effectiveness of the implicit KL-regularization in MCTS that we study in Section 3 motivates us to develop an equilibrium-finding algorithm called piKL-Hedge that similarly biases the search towards an anchor policy.
In Section 4.3, we show that piKL-Hedge achieves better empirical performance against baseline human-imitation models than Hedge and RM in a large imperfect-information game, as well as much higher human prediction accuracy.
4. B ACKGROUND
We consider a game with N players where each player i chooses an action a from a set of actions A i .
We denote the actions of all players other than i as a − i .
After all players simultaneously choose an action, player i receives a reward of u i ( a, a − i ) .
Players may also choose a probability distribution over actions, where the probability of action a is denoted π i ( a ) and the vector of probabilities is denoted π i .
We also define the fixed policy that we are interested in biasing player i towards as the anchor policy τ i ∈ ∆( A i ) .
Each player i maintains a regret for each action.
The regret on iteration t is denoted R t i ( a ) .
Initially, all regrets are zero.
On each iteration t of Hedge, π ti ( a ) is set according to
[Equation]
Next, each player samples an action a ∗ from A i according to π ti and all regrets are updated such that
[Equation]
It is proven that the average policy of Hedge over all iterations converges to a NE in two-player zero-sum games and, more broadly, the players’ joint policy distribution converges to a CCE as t → ∞ (Freund & Schapire, 1997; Cesa-Bianchi & Lugosi, 2006).
We wish to model agents that seek to maximize their expected reward, while at the same time playing “close” to the anchor policy.
The two goals can be reconciled by defining a composite utility function that adds a penalty based on the “distance” between the player policy and their anchor policy, with coefficient λ i ∈ [0 , ∞ ) scaling the penalty.
For each player i , we define i ’s utility as a function of the the agent policy π i ∈ ∆( A i ) given policies π − i of all other agents:
[Equation]
When λ is large, the utility function is dominated by the KL-divergence term λ i D KL ( π i ∥ τ i ) , and so the agent will naturally tend to play a policy π i close to the anchor policy τ i 4 .
When λ i is small, the dominating term is the rewards
u
i ( π i , a t − i ) and so the agent will tend to maximize reward without as closely matching the anchor policy
τ i .
These statements are made precise in Theorem 1 and
Theorem 2.
4. N O -R EGRET L EARNING FOR P OLICY -R EGULARIZED U TILITIES
In this section, we present Algorithm 1, a no-regret algorithm based on Hedge for any player i to learn strong policies relative to the regularized utilities defined in equation 5.
As we show in Propositionin Appendix B, it guarantees that each player i accumulates sublinear regret (of order log T ) with respect to the regularized utility functions:
[Equation]
no matter the opponents’ actions a t − i at each time t .
, D KL ( τ ∥ π ) . We choose this direction for greater ease of theoretical analysis and implementation in the context of regret minimization; for our use cases we have not found the exact form of the loss to be critical so much as simply doing any reasonable regularized search. Data: • A
i [Table] ; • u i reward
[Table] i ; • η >
[Table]
learning
[Table]
hyperparameter.
1 function I NITIALIZE () t ← 0for each action a ∈ A i do
CV 0 i ( a ) ←5 function P LAY () t ← t + let π ti be the policy such that
[Equation]
( a
[Equation]
∝ exp (cid:26) η CV t − 1 i ( a ) + tλ i η log
τ
i ( a ) 1 + tλ i η (cid:27) .
[Equation]
6) sample an action a t ∼ π ti play a t and observe actions a t − i played by the opponents for each a ∈ A i do
CV ti ( a ) ← CV t − 1 i ( a ) + u i ( a, a t − i )
As with many other regret-minimization methods, we consider the average policy of each player i over
T iterations:
[Equation]
where π ti is defined in equation 6 in Algorithm 1.
We take ¯ π Ti to be the final agent policy to play in the real environment.
As shown in Appendix B, the KL-divergence of ¯ π Ti from the anchor policy τ i converges to be inversely proportional to λ i :
Theorem 1. (piKL stays close to the anchor policy)
Upon running Algorithm 1 for T iterations in a multiplayer general-sum game, the policy ¯ π Ti is at a distance D KL (¯ π Ti ∥ τ i ) ≤
1 λ i (cid:16) R Ti T + D i (cid:17) , where
D i is any upper bound on possible rewards for
Player i .
In particular, if η > 0 is set so that R T i = o ( T ) , then D KL (¯ π Ti ∥ τ i )
→ D i /λ i as T → + ∞ .
We can also show (Appendix B) that in the case of a two-player zero-sum game, ¯ π Ti approximates a
Nash equilibrium of the original utility functions, with the approximation guarantee controlled by λ :
Theorem 2.
Let (¯ π 1 , ¯ π 2 ) be any limit point of the average policies (¯ π T 1 , ¯ π T 2 ) of the players.
Almost surely, (¯ π 1 , ¯ π 2 ) is a (max i =1 , 2 { λ i β i } )
-approximate
Nash equilibrium policy with respect to the original utility functions u i , where β i is a constant depending on τ i (see equation 21 in Appendix B).
Lastly, we remark that in the special case that τ i is the uniform policy for all players i , the above results imply that Algorithm 1 converges towards a quantal response equilibrium (McKelvey & Palfrey, 1995a), in which an imperfect agent is modeled as choosing actions with probability exponentially decaying in the amount that each action is worse than the best action(s), given that all other agents behave the same way.
Our method can be seen as a generalization that takes into account a human-learned prior for what actions may be more likely.
4. D IPLOMACY E XPERIMENTS
Diplomacy is a benchmark 7-player simultaneous-action game featuring both cooperation and competition.
In Appendix H, we summarize the rules of the game.
Using piKL-Hedge, we develop an agent piKL-HedgeBot and show that it improves upon prior approaches.
In Appendix C, we also illustrate the key features of piKL-Hedge in Blotto, a famous 2-player simultaneous action game.
4.3.1 Algorithms and Models
In no-press Diplomacy, we compare the different equilibrium search algorithms (RM, Hedge and piKL-Hedge) using the procedure introduced by Gray et al.
We perform 1-ply lookahead where on each turn, we sample up to 30 of the most likely actions for each player from a policy network trained via imitation learning on human data (IL policy).
We then consider the 1-ply subgame consisting of those possible actions where the rewards for a given joint action are given by querying a value network trained on human game data as in Gray et al.
We play according to the approximate equilibrium computed for that subgame by that algorithm.
For piKL-Hedge, the anchor policy is simply the same human-trained policy network.
Our baseline policy and value models also contain a few improvements over prior models for no-press Diplomacy, described in Appendices J and K.
In our experiments, we label our RM, Hedge, and piKL-Hedge agents as RMBot, HedgeBot, and piKL-HedgeBot respectively.
We compare also against SearchBot
Gray et al.
(2020) (similar to RMBot but using the models from Gray et al.
(2020) rather than our models).
See Appendix I for more details about the hyperparameters used.
4.3.2 Strong, human-like play with piKL-Hedge
Similar to Chess and Go, we compare the human prediction accuracy of RMBot, HedgeBot, piKL-HedgeBot (with different λ s) to the IL anchor policy, as well as testing their head-to-head strength.
In particular, we test their ability to predict human moves in 226 no-press Diplomacy games from a validation set, and measure their score against the IL policy across 700 games each. [Figure] Pseudo-ELO 0 .0 .0 .0 . U n it
O r d e r T opA cc u r ac y
[Figure] [Figure] [Table] In Figure 3 (Left), we present the average top-1 accuracy of unit orders in each action predicted by these methods as well as their average scores against 6 IL anchor policies.
The raw IL model ( λ = ∞ ) predicts human moves with high accuracy but is weak and achieves low average score.
Unregularized Hedge and RM ( λ = 0 ) achieve high score but low human prediction accuracy.
By contrast, piKL-HedgeBot with different λ achieves a variety of highly favorable combinations of the two.
λ
= 10 − 1 gives about the same top-1 accuracy as the IL policy but improves score by a factor of 1.4x over the IL anchor policy.
λ
= 10 − 3 outperforms unregularized search methods in both score (by ~5%) and human prediction accuracy (by ~6%).
Mild regularization improves average score, rather than harming it.
We also tested pure RL agents and found they perform poorly in predicting human moves.
In particular, the recently proposed DORA and HumanDNVI-NPU algorithms Bakhtin et al.
(2021) achieve top-1 accuracy of only 29.1% and 37.8% respectively.
Next, in Figure 3 (Right), we compare the top-1 accuracy of these methods across players of different pseudo-Elo ratings.
The pseudo-Elo ( e i ) for player i is constructed based on the logit rating s
i introduced in Gray et al.
(2020), where, e i
=
s i · 400 log(10)
+ 1000 .
The top-1 accuracy for all the searchbased policies increases with pseudo-Elo, indicating that they are better at modeling stronger players than weaker players.
piKL-Hedge ( λ = 10 − 1 ) performs just as well as the anchor policy across pseudo-Elos while being significantly stronger than the anchor, while λ = 10 − 3 is as strong or stronger than Hedge and RM but matches human play far better.
4.3.3 piKL-HedgeBot performs well against a varied pool of agents
We also develop a new head-to-head evaluation setting, where rather than testing one agent vs 6x of another agent, allagents per game are sampled uniformly from a pool.
The 1v6 head-to-head scores used in prior work
Gray et al.
(2020); Bakhtin et al.
Anthony et al.
; Paquette et al.
(2019) indicate whether a population of 6x agents can be invaded by a 1x agent, and hence whether the 6 agents constitute an
Evolutionarily Stable Strategy (ESS) (Taylor & Jonker, 1978; Smith, 1982).
By contrast, assigning the 7 agents per game randomly from a pool studies the robustness of an agent to a variety of other agents.
We experiment with a pool of 8 agents.
Five are previously published agents: DipNet, DipNet
RL Paquette et al.
(2019), Blueprint, BRBot,
SearchBot Gray et al.
(2020) and three are our agents:
IL policy, RMBot and piKL-HedgeBot.
Doing well in this population requires playing well with both human-like policies (DipNet, Blueprint) and equilibrium policies (SearchBot, RMBot).
Each experiment only compares one lambda value of piKL-HedgeBot for fairness.
The results of these experiment with piKL-HedgeBot ( λ = 10 − 3 ) is presented in Table 2.
piKL-HedgeBot ( λ = 10 − 3 ) outperforms all other agents.
Overall, unlike Chess and Go, we do not find that piKL-Hedge clearly improves human prediction accuracy over the IL model.
However, unlike Chess and Go, we observe that piKL-Hedge does improve playing strength over prior search methods against various past agents.
In general-sum games like Diplomacy, it appears that piKL-hedge with a human anchor policy allows for slightly better play in a population containing human-like agents while still doing well against equilibrium-searchers, or alternatively can improve strength over IL to a lesser degree with no cost at all to prediction accuracy.
C ONCLUSION
In this paper, we showed across several domains that regularizing search policies according to a KL-divergence loss with an imitation learned (IL) policy produces policies that maintain high human prediction accuracy while being far stronger than the original learned policy.
In chess and
Go, applying standard MCTS regularized toward a human-learned policy achieves state-of-the-art prediction accuracy, surpassing imitation learning, while also winning more than 85% of games against an IL model.
We then introduced a novel regret minimization algorithm that is regularized based on the KL divergence from an IL policy.
In no-press Diplomacy, this algorithm yields both a policy that predicts human play with the same accuracy as imitation learning alone while increasing win rate against state-of-the-art baselines by a factor of 1.4, or alternately a policy that outperforms unregularized search while achieving much higher human prediction accuracy.
We presented in
Appendix M similar successful results for KL-regularized search in Hanabi.
There are several directions for future work, such as extending piKL-Hedge to handle extensive-form games.
Additionally, there may be better ways to regularize search than KL-divergence.
Finally, it remains to be seen how KL-regularized search performs when combined with RL.
A UTHOR C ONTRIBUTIONS
A. P. Jacob was the primary researcher for piKL-hedge and contributed to the direction, experiments, and writing of the entire paper.
D. J. Wu was the primary researcher for MCTS in chess and Go, and contributed to the direction, experiments, and writing of the entire paper.
G. Farina was the primary formulator of the piKL-hedge algorithm and handled all the theory in the paper.
A. Lerer contributed to the direction of the project, the formulation of piKL-hedge, its experimental evaluation, and paper writing.
H. Hu was the primary researcher for the extension of piKL to Hanabi covered in Appendix M. A. Bakhtin contributed to the experimental evaluation of piKL-hedge. J. Andreas contributed to the direction of the project and to paper writing. N. Brown initiated the project and contributed to the direction of the project, the formulation of piKL-hedge, its experimental evaluation, and paper writing. A. R EGULARIZED L EARNING AND P LANNING
Several prior works have explored augmenting reinforcement learning with supervised data from expert demonstrations (Vecerik et al., 2017; Nair et al., 2018).
For example, Hester et al.
(2018) augment Deep Q Learning with a margin loss on demonstration data that aims to make Q ( a ) for each demonstration action higher than that of other actions.
KL-regularization has also been used successfully to incorporate expert demonstrations into RL training (Rudner et al., 2021;
Ng et al., 2000; Boularias et al., 2011; Wu et al., 2019; Peng et al., 2019; Siegel et al., 2019).
In these settings, the standard RL objective is augmented by a KL divergence penalty that expresses the dissimilarity between the online policy and a reference policy derived from demonstrations.
This helps guide exploration in RL or ameliorate inaccurate modeling of the environment in domains such as robotics.
AlphaStar Vinyals et al.
(2019), which achieved expert human performance in StarCraft 2, uses self-play RL initialized from supervised policies with a KL penalty term for deviating from the supervised policy during training to "aid in exploration and to preserve strategic diversity".
In our work, we use the KL term to better approximate human play during inference-time search.
Prior work has explored entropy-regularized utilities in games.
Ling et al.
(2018) show that a particular type of entropic regularization in extensive-form games leads to quantal response equilibria.
Cen et al.
(2021) give fast online optimization algorithms for entropy-regularized utilities in the context of quantal response.
Farina et al.
(2019) regularize utilities with a
KL divergence term from a precomputed Nash equilibrium strategy to design agents that trade off game-theoretic safety and exploitation.
In our work, we leverage the KL divergence term towards a human-imitation learned policy instead, to regularize the utilities.
And unlike previous works, we empirically study our approach in a much larger imperfect information game.
A. S TRONG H UMAN -C OMPATIBLE P OLICIES Prior work
in multi-agent reinforcement learning has emphasized the importance of human-compatible policies in cooperative multi-agent environments.
Lerer & Peysakhovich (2019) demonstrate that self-play policies may perform poorly with other agents if they do not conform to the population equilibrium (social conventions), and propose a combination of policy gradient and imitation loss directly on samples of population data.
Human-compatible policies have also been studied on the benchmark game of Hanabi, where ad hoc play with humans is regarded as an open challenge problem Bard et al.
Most work on this challenge has focused on zero-shot coordination with humans, in which an agent must adapt to human play with no prior experience with human partners Hu et al.
(2020; 2021b)
; Cui et al.
Learning human-compatible policies from a combination of human data and planning are less well-studied in this setting.
A.2.1 MCTS, Chess and Go
Prior work in tree search methods, especially in chess and Go, has typically focused on developing strong agents without concern for accurately modeling human behavior.
For example in Go, a significant body of older work investigates imitation learning (IL) to obtain a baseline policy prior to use with MCTS, but tunes and evaluates the final agent via playing strength alone Tian & Zhu (2016)
; Cazenave (2017).
Regarding the use of search for human modeling, recent work in chess by McIlroy-Young et al.
(2020b) found that pure IL outperformed all other approaches and that adding MCTS with the parameters of a standard engine significantly harmed human prediction accuracy.
However in our work, using a different range of parameters, we show clear results to the contrary.
In Go,
Wang et al.
(2017) report promising results on human prediction and playing strength using search, albeit with a specialized architecture and rollout method.
Baier et al.
(2018) report in the card game Spades both excellent human accuracy and playing strength by adding a human policy bias to a variant of MCTS.
To our knowledge, our work is the first to demonstrate a clear gain in human prediction accuracy over deep learning models in the highly-studied domains of chess and Go via simple well-established methods of policy-regularized planning.
A.2.2 Diplomacy
Diplomacy is a benchmark 7-player game that involves communication, negotiation, cooperation, and competition in a strategic multi-agent setting.
While chess and Go are two-player zero-sum games for which optimal play is well-defined and can be computed through self-play Nash (1951), Diplomacy has no such guarantees and strong play likely requires modeling other agents, even in the no-press variant where natural-language communication is not allowed Bakhtin et al.
Paquette et al.
(2019) showed that neural network IL on human data in no-press Diplomacy can reasonably approximate human play, but that bootstrapping RL from this agent leads to a breakdown in cooperation.
Anthony et al.
(2020) developed new RL methods based on fictitious play that improve the performance of agents in no-press Diplomacy, and Gray et al.
(2020) showed that an equilibrium-finding regret minimization search procedure on top of human IL models achieves human-level performance in no-press Diplomacy.
However, although both methods rely on the human IL model to generate a restricted action set for RL or search, neither contains any explicit regularization when choosing among those actions, and we show that the equilibrium search in Gray et al.
(2020) greatly decreases the accuracy of modeling human players.
Similarly, Bakhtin et al.
(2021) achieved strong results in no-press Diplomacy via self-play RL both from scratch and initialized with a human-learned policy, but we show that the resulting final agents do not ultimately model humans well.
In this Appendix, we present detailed proofs of Proposition 1 and Theorem 2.
B. K NOWN RESULTS
We start by recalling a few standard results.
First, we recall the follow-the-regularized-leader (FTRL) algorithm, one of the most well-studied algorithms in online optimization.
At every time t , the FTRL algorithm instantiated with domain X , 1 -strongly-convex regularizer ϕ : X → R , and learning rate η > 0 , produces iterates according to
[Equation]
where ℓ 1 , . .
. , ℓ t are the convex utility functions gave as feedback by the environment.
The FTRL algorithm guarantees the following regret bound.
Lemma 1 (Rakhlin (2009), Corollary 7) .
The iterates x t ∈ X produced by the FTRL algorithm set up with constant step size η > 0 and 1 -strongly convex regularizer
ϕ
satisfy the regret bound
[Equation]
In the analysis of Algorithm 1 we will also make use of the following technical lemma, a proof of which can be obtained starting using the same techniques as Abernethy & Rakhlin (2009, Lemma A.4)
Lemma 2.
Let p ∈ ∆( A ) be a distribution over a discrete set A , q ∈ R |
A | be a vector, and D > 0 be any constant such that max a,
a ′ ∈ A { q ( a ) − q ( a ′ ) } ≤ M .
Then,
[Equation]
Proof.
Let q min := min a ∈
A q ( a ) and ˜ q ( a ) :
= q ( a ) − q min be a shifted version of q ( a ) so
that 0 ≤ ˜ q
( a ) ≤ M for all a ∈ A .
Let now X denote a random variable with value ˜
q
( a ) with probability p ( a ) for all a ∈ A .
Then,
[Equation]
Applying Lemma A.4 from Abernethy & Rakhlin (2009) we obtain
[Equation]
which is exactly the statement.
B. B OUNDING THE DISTANCE BETWEEN THE ITERATES OF A LGORITHM
Lemma 3.
At all times t and for all players i , the policies π ti produced by the FTRL algorithm set up with constant step size η and negative entropy regularizer φ ( x ) := (cid:80) a ∈
A i x ( a ) log x ( a ) , when observing the utilities U ti , match the policies π ti produced by Algorithm 1.
Proof.
Plugging the particular choices of utilities and regularizer into equation FTRL, we obtain π
[Equation]
A well-known closed form solution to the above entropy-regularized problem is given by the softmax function.
In particular, let
[Equation]
Then,
[Equation]
which coincides with the iterate produced by Algorithm 1.
The next observation shows that the iterates π i do not change if the utility function u i is first shifted to be in the range [0 , D i ] .
Remark 1.
Consider the shifted utilities
[Equation]
and let v be defined as equation 10 using
˜ u
i in place of u i , that is,
[Equation]
Then, the iterates π i can be equivalently expressed as
[Equation]
Proof.
Let γ := min a ∈ A 1 ×···× A n u i ( a ) denote the minimum utility that Player i can get against the actions of the opponents.
Since the argmax of a function does not change if a constant is added to the objective, from equation 9 we can write
[Equation]
where the second equality follows from the fact that π ∈ ∆( A i ) .
A solution is again given by softmax function
[Equation]
In the rest of the proof we will use equation 13 to analyze the iterates π i produced by the algorithm.
Lemma 4.
Let η ≤ 1 / ( λ i β i + 2 D i ) .
Then, at all times t ,
[Equation]
Proof.
At all times t , introduce the vector ξ ti ∈ R | A i | defined as
[Equation]
At all times t and for all a , it holds that
[Equation]
Substituting equation 15 we can write
[Equation]
[Equation]
Expanding the definition of the local norm induced by ∇ 2 φ ( π ti ) we find
[Equation]
where equation 17 follows from substituting equation 16.
We now apply Lemma 2, applied with q = ξ ti , p = π ti
[Equation]
and A = A i .
First, we study the range D i = max a,a
[Equation]
∈
[Equation] [Equation]
) − ξ ti ( a ′ ) } used in the statement of the Lemma.
In particular, using equationwe have
[Equation]
[Equation]
where the first inequality follows from upper bounding the max of a sum with the sum of max of each term, and the second inequality follows from noting that
[Equation]
and
[Equation]
Applying Lemma 2 to the right-hand side of equation 18 using the bound on the range of ξ ti shown in equation 19 yields
[Equation]
Using the fact that any convex combination of values is upper bounded by the maximum value
[Equation]
we can further bound the right-hand side of equation 20 as
[Equation]
where the equality follows from equation 15.
Hence, expanding the definition of v ti and v t +1 i , ∥
π t +1 i
[Equation]
i η ) log τ i ( a ) + η 1 + tλ i η ˜ u i ( a, a t − i ) − λ
i η(1 + tλ i η )(1 + ( t − 1)
λ i η ) t − 1 (cid:88) t ′ =
˜ u
i ( a, a t ′ − i ) (cid:41)≤ 3exp { 2 η ( λ i β i + 2 D i ) }
η 2 ( λ i β i + 2 D i )max a ∈
A i (cid:40)(cid:18)
λ
i η (1 + tλ i η )(1 + ( t − 1) λ i η ) log
τ i ( a ) (cid:19)+ (cid:18) η 1 + tλ i η ˜ u i ( a, a t − i ) (cid:19)+ (cid:32) λ
i η(1 + tλ i η )(1 + ( t − 1)
λ i η ) t − 1 (cid:88) t ′ =
˜ u
i ( a, a t ′ − i ) (cid:33) 2  
=
(1 + tλ i η )exp { 2 η
( λ i β i
+ 2 D i ) }
η 2 ( λ i β i + 2 D i )max a ∈
A i (cid:40)(cid:18)
λ
i η 1 + ( t − 1) λ i η log
τ
i ( a ) (cid:19)+ (cid:0) η ˜ u
i ( a, a t − i ) (cid:1)+ (cid:32) λ
i η(1 + ( t − 1) λ i η ) t − 1 (cid:88) t ′ =
˜ u
i ( a, a t ′ − i ) (cid:33) 2  
≤
[Equation]
η
[Equation]
λ
[Equation]
η
[Equation]
β
[Equation]
Using the hypothesis that η ≤ 1 / ( λ i β i + 2 D i ) and taking square roots yields the statement.
B. C OMPLETING THE ANALYSIS
Proposition 1.
Fix a player i ∈ { 1 , . . . , n } .
The regret
[Equation]
incurred up to any time T by policies π ti defined in equation 6 where the learning rate is set to any value 0 < η ≤ 1 / ( λ i β i
[Equation]
D i ) , satisfies
[Equation]
where D i is any upper bound on the range of the possible rewards of Player i , and
[Equation]
Proof.
Let
[Equation]
and note that, by definition of the regularized utilities
U i ,
[Equation]
where the first inequality follows by convexity and the second inequality by the generalized Cauchy-Schwarz inequality with the primal-dual norm pair ∥ · ∥ ∇ 2 φ ( π ti ) and ∥ · ∥ ∇ − 2 φ ( π ti ) .
A bound for the second term in the product if given by Lemma 4.
We now bound the first norm.
First,
[Equation]
where the second inequality follows from the fact that x log 2 x ≤ 1 for all x ∈ [0 , 1] .
Taking square roots, we find
[Equation]
So, using Lemma
[Equation]
we can write
[Equation]
Plugging in the above expression into Lemma 1 yields
[Equation]
which is the statement.
B. P ROOF OF T HEOREM
Theorem 1. (piKL stays close to the anchor policy) Upon running Algorithm 1 for T iterations in a multiplayer general-sum game, the policy ¯ π Ti is at a distance D KL (¯ π Ti ∥ τ i ) ≤
1 λ i (cid:16) R Ti T
+ D i (cid:17) , where D i is any upper bound on possible rewards for
Player i .
In particular, if η > 0 is set so that R Ti = o ( T ) , then D KL (¯ π Ti ∥ τ i )
→ D i /λ i as T → + ∞ .
Proof.
By definition of regret,
[Equation]
[Table]
[Equation]
where the first inequality holds since the KL divergence is nonnegative, the second inequality by convexity of the KL divergence, and the third inequality by definition of D i .
Rearranging yields the inequality in the statement.
B. R ELATIONSHIP WITH N ASH E QUILIBRIUM
In this subsection, we show that when all players play according to Algorithm 1 in a two-player zero-sum game, then the average policies ¯ π Ti converge to a Nash equilibrium of the regularized game whose utilities are U i .
Proposition 2.
For any T ∈ N , η > 0 , and δ ∈ (0 , 1) , define the quantity
[Equation]
Upon running Algorithm 1 for any T iterations with learning rate η > 0 , the average policies ¯ π Ti of each player form a ξ T ( δ )
-approximate
Nash equilibrium with respect to the regularized utility functions U i with probability at least 1 − δ , for any δ ∈ (0 , 1) .
Proof.
Fix a player i ∈ { 1 , 2 } , and any policy π ∗ ∈ ∆( A i ) , and introduce the discrete-time stochastic process
[Equation]
Since the opponent player − i plays according to Algorithm 1, its action a t − i at all times t is selected by sampling (unbiasedly) an action from the policy
π t − i .
Therefore, w t is a martingale difference sequence.
Furthermore, by expanding the definition of U i , the absolute value of w t satisfies
[Equation]
Hence, using Azuma-Hoeffding’s inequality, for any δ ∈ (0 , 1) ,
[Equation]
where R T i is as defined in Proposition 1.
Since the above expression holds for any π ∗ ∈ ∆( A i ) , in particular, using the union bound on each a ∈
A i ,
[Equation]
for any player i ∈ { 1 , 2 } and any δ ∈ (0 , 1) .
Summing Inequality equation 22 for i ∈ { 1 , 2 } and using the union bound, we can further write
[Equation]
Dividing by T and noting that for any player i ∈ { 1 , 2 }
[Equation]
further yields
[Equation]
We now analyze the term in parenthesis
[Equation]
that is,
[Equation]
Plugging in the definition of U 1 and U 2 , that is,
[Equation]
into ( ♣ ) yields
[Equation]
where equation 24 follows from convexity of the KL divergence function
[Equation]
and equation 25 follows again from the definition of Uand U 2 .
Substituting equation 25 back into equation
[Equation]
we find
[Equation]
Since
[Equation]
and
[Equation]
the inequality above in particular implies that
[Equation]
which is equivalent to the statement after making the variable substitution δ :
= δ ′ / 2 .
In particular, when η ≤ 1 / ( λ i β i + 2 D i ) for both players i ∈ { 1 , 2 } , Proposition 2 implies that the average strategy profile is
a
O (1 / √ T ) -Nash equilibrium with respect to the regularized utility functions
U i .
A standard application of the Borel-Cantelli lemma enables to convert from the high-proability guarantees of Proposition 2 at finite time to almost-sure convergence in the limit.
Specifically,
Corollary 1.
Let (¯ π 1 , ¯ π 2 ) be any limit point of the average policies (¯ π T 1 , ¯ π T 2 ) of the players.
Almost surely, (¯ π 1 , ¯ π 2 ) is a Nash equilibrium with respect to the regularized utility functions U 1 , U 2 , respectively.
From there, it is immediate to give guarantees with respect to the original ( i.e. , unregularized) game, and Theorem 2 follows.
Theorem 2.
Let (¯ π 1 , ¯ π 2 ) be any limit point of the average policies (¯ π T 1 , ¯ π T 2 ) of the players.
Almost surely, (¯ π 1 , ¯ π 2 ) is a (max i =1 , 2 { λ i β i } )
-approximate
Nash equilibrium policy with respect to the original utility functions u i , where β i is a constant depending on τ i (see equation 21 in Appendix B).
Proof.
From Corollary 1, almost surely (¯ π 1 , ¯ π 2 ) is a Nash equilibrium of the regularized game whose players’ utilities are U 1 and
U 2 , respectively.
Expanding the definition of Nash equilibrium relative to Player
[Equation]
we have that
[Equation]
where the first inequality follows since the KL divergence is alwasy nonnegative, the second inequality since the negative entropy function is nonpositive on the simplex, and the third inequality follows from the definition of β 1 .
Symmetrically, for Player 2 we find that
[Equation]
Hence, the exploitability of ¯ π 1 is at most λ 1 β 1 , while the exploitability of ¯ π 2 is at most λ 2 β 2 , which immediately implies the statement.
C I LLUSTRATIONS OF PI KL-H EDGE IN B LOTTO
Colonel Blotto is a famous 2-player simultaneous action game that has a large action space but has rules that are short and simple.
In Blotto, each player has c coins to be distributed across f fields.
The aim is to win the most fields by allocating the player’s coins across the fields.
A field is won by contributing the most coins to that field (and drawn if there is a tie).
The winner receives a reward of +1 and the loser receives -1.
Both receive 0 in the case of a tie.
In Figure 4 we illustrate the key features of piKL-Hedge in Blotto, using a uniform anchor policy for convenience.
Incidentally, piKL-Hedge with a uniform anchor policy converges to a quantal response equilibrium McKelvey & Palfrey (1995b).
piKL-Hedge finds policies that play close to the anchor policy while having low regret, with λ controlling the relative optimality of these two desiderata.
[Equation]
[Equation]
EGULARIZED S EARCH ALSO IMPROVES C ROSS -E NTROPY
Model
[Table]
Here we show that not only does policy-regularized search improve top-1 accuracy, it also improves cross entropy for a reasonable range of parameter values in both chess and Go, as well as performing well in Diplomacy.
For Chess and Go, to obtain the policy distribution with which to compute its cross entropy with the human data, unlike for topaccuracy or for play we cannot directly use the MCTS visit distribution because occasionally simply due to discretization MCTS may give zero visits to the actual move that a human played, resulting in an undefined (i.e. infinite) cross-entropy.
Instead, we leverage the result of Grill et al.
(2020) that PUCT-style MCTS with a policy prior can be seen as a discrete approximation to solving a smooth optimization:
[Equation]
[Figure]
U n it
O r d e r
C r o ss - E n t r opy
[Figure]
where
[Equation]
where Q is the current value estimate from search for each action, τ is the anchor or prior policy, λ controls the strength of the regularization towards that prior as a function of the number of visits, c puct is the MCTS exploration coefficient, n a is the number of times a was explored and k is an arbitrary constant not affecting the asymptotic results.
We perform MCTS exactly the same as normal, the only difference is that at the very end, rather than using visit counts, we compute π optimizing the above objective using the human imitation-learned anchor policy for τ and the MCTS-estimated
Q-values for Q (and using the same unweighted average child value for moves that lack a Q-value estimate due to having zero visits as described in Appendix G).
We use this resulting smooth π as the final policy prediction and compute its cross entropy with the actual human moves.
Note that the authors choose k = |
A | , the number of legal actions, for mathematical convenience, corresponding to adding one extra visit per every possible action (Grill et al., 2020), but since in our experiments we use only use 50 visits including the root visit (i.e. (cid:80) a n a = 49 ), and the branching factor can be as large as 362 in Go, adding one extra visit per legal action in our case greatly overestimates the total number of visits, which in practice gives a less accurate correspondence between MCTS and this smoother regularized solution, so we instead choose k = 0 .
In Table 3 we show the results.
Across roughly the same parameter ranges, the regularized search policy using this smoothed
MCTS postprocessing achieves lower cross-entropy with human moves than the raw imitation-learned policy without search.
This suggests that not only does search improve on the raw imitation-learned policy at pinpointing the top action, it also gives a more accurate model of the overall distribution of likely human actions.
Similarly, in Figure 5, we show that piKL-Hedge achieves better average cross entropy of unit orders in no-press Diplomacy compared to unregularized search methods, and matches that of imitation learning at λ = 0 .
This corroborates the results of
Section 4.3 and shows that piKL-Hedge provides the same benefits in modeling the overall distribution of human actions as it does on predicting the top move - outpredicting unregularized search (while playing as well or slightly better against human-like opponents), or equaling the prediction quality of imitation learning (while playing much more strongly than IL).
[Table] [Table]
This section summarizes the results of a small number of additional experiments in chess and Go.
Whereas Figure 1 used a temperature of 1.0 when sampling from the agent policy, we show in Table 4 that MCTS also achieves similar winrates versus the raw model when both are sampled using a much lower temperature of 0.3.
Additionally
[Table]
we confirm in Go the same result that McIlroy-Young et al.
(2020a) reported in chess, that current top RL agents are far worse at matching human moves than even just imitation learning.
We tested the current top open-source Go program
KataGo (Wu, 2020) using its final 6, 10, and 15 block models 5 which range from upper-amateur to superhuman level, and found that they achieve accuracies of about 35%, 43%, and 43%, all more than 10% lower than the results in Table 1.
Lastly, in Table 5 we show that in chess the amount of improvement given by MCTS over the raw model in predicting human players on the Lichess test set tends to be larger for games with longer time controls than for shorter time controls, going from about 0.2% to about 0.7% between the shortest and longer time controls for 1500-1599-rated players, and going from about 0.8% to around 2.0% for 1900-1999-rated players.
This is consistent with the intuition that humans rely more heavily on planning when they have more time to think, increasing the gain from modeling that planning.
F B
M
A RCHITECTURE AND T RAINING FOR G O
As summarized in Section 3.2, for training baseline imitation-learning models to play on the 19x19 board in Go, our architecture follows the same 20-block 256-channel residual net described in Silver et al.
(2017), except with the addition of squeeze-andexcitation layers at the end of each residual block (Hu et al., 2018).
In particular, the following additional operations are inserted just prior to each skip connection that adds the output R of a residual block to the trunk X :
• Channelwise global average pooling of R from 19 × 19 × 256 channels to 256 channels.
• A fully connected layer including bias from 256 channels to 64 channels.
• A ReLU nonlinearity.
• A fully connected layer including bias from 64 channels to 512 channels, which are split two vectors S and B ofchannels each.
• Output R × sigmoid( S ) + B to be added back to the trunk X , instead of R as in a normal residual net.
In other words, the final result of the residual block as a whole is ReLU( X + R × sigmoid
S )
+
B ) instead of ReLU( X + R ).
Additionally, some games are played with a komi (compensation given to White for playing second) that is not equal to the value of 7.5 used by Silver et al.
Therefore, for the final feature of the input encoding, rather than a binary-valued feature equalling 1 if the player to move is White and 0 if the player to move is Black, we instead use the real-valued feature of komi/10 if the player to move is White or -komi/10 if the player to move is Black.
We additionally exclude a very tiny number of games with extreme komi values, outside of the range [-60,60].
We train using a mini-batch size of 2048 distributed as 8 batches of 256 across 8 GPUs, and train for a total of 64 epochs - roughly 475000 minibatches for the GoGoD dataset.
We use SGD with momentum 0.9, weight decay coefficient of 1e-4, and a learning rate schedule of of 1e-1, 1e-2, 1e-3, 1e-4 for the first 16, next 16, next 16, and last 16 epochs respectively.
We train both the policy and values heads jointly, minimizing the cross entropy of the policy head with respect to the one-hot move made in the actual game, and the MSE of the value head with respect to the game result of -1 or 1, except similarly to Silver et al.
(2017) we weight the MSE value loss by 0.01 to avoid overfitting of the value head.
G MCTS A LGORITHMIC D ETAILS
In this appendix, we summarize the details of the version of MCTS used in our experiments, including one often-overlooked detail.
We follow a standard MCTS implementation very similar to that of Silver et al.
Each turn, the algorithm builds and expands a game tree over multiple iterations rooted at the current state for that turn.
On each iteration t MCTS starts at the root and descends the tree by exploring at each state s an action a according to some exploration method.
Upon reaching a state s t not yet explored, it adds s t to the tree, queries the value function V i ( s t ) for each i to estimate the total expected future reward, and updates the statistics of all nodes traversed based on V i ( s t ) and any intermediate rewards received.
Subsequent iterations begin again from the root.
For our work in chess and Go, we follow the convention where win, loss, and draw have reward 1,-1, and 0.
The statistics tracked at the node for each state s where player i is to move include the visit counts N ( s, a ) which are the number of iterations that reached state s and tried action a , and Q ( s, a ) the average value of those iterations from the perspective of player i , i.e. Q ( s, a ) =
(1 /N ( s, a ))
(cid:80) t V i ( s t )
+ U
i
( s, s t ) where the sum ranges only over those iterations t that reached state s and tried action a
and U
i ( s, s t ) is the total intermediate reward on the path from s to s t .
When descending the tree, the exploration method is to always select the action:
[Equation]
where τ ( s, a ) is the prior policy probability for action a in state s , and c puct is a tunable parameter controlling the tradeoff between exploration and exploitation.
The final agent policy π is simply proportional to the visit counts for the root, i.e. π ( s, a ) =
N ( s, a ) / (cid:80) b N ( s, b ) where s is the root state, or optionally we may also have π ( s, a ) ∼ N ( s, a ) 1 /T where T is a temperature parameter.
One final often-overlooked detail concerns how to evaluate states for which there is no Q ( s, a ) estimate.
Since the tree policy depends on the Q ( s, a ) estimates of the possible actions, there is a nontrivial choice of what Q value to use for an action a that has been tried zero times and therefore never estimated.
Since the tree branches exponentially, deeper in the MCTS tree there will always be many actions with zero visits, and so this choice can affect the behavior of MCTS even in the limit of large amounts of search.
Unfortunately, the details of this choice have sometimes been left undiscussed and undocumented in major past work, and as a result major MCTS implementations have not standardized on it, variously choosing game-loss (Lai, 2018), the current running average parent Q or a Q -value minus a heuristic offset parameter (Tian, 2019), or many other options.
In our work, we use the equal-weighted average value of all actions at the parent node that have been visited at least once, i.e. (cid:80) a Q ( s, a ) I ( N ( s, a ) > 0) / (cid:80) a I ( N ( s, a ) > 0) .
This can be viewed as corresponding to a naive prior that the values of actions are i.i.d draws from an unknown distribution.
While not perfect, our choice is simple, parameter-free, behaves in a way that is invariant to any global translation or scaling of the game’s rewards, and works reasonably in practice for our purposes.
H B RIEF D ESCRIPTION OF D IPLOMACY
We briefly summarizing the rules of Diplomacy.
See Paquette et al.
for a more detailed description.
The board is a map of Europe partitioned into 75 regions, 34 of which are supply centers (SCs) that players compete to control.
Players command multiple units and each turn privately issue orders for each unit they own (to hold, move, support another unit, or convoy).
These orders are revealed at the same time, thereby making Diplomacy a simultaneous-action game.
A player wins the game by controlling a majority (18) of the SCs.
A game may also end in a draw if all remaining players agree.
In this case, we use the
Sum-of-Squares (SoS) scoring system as used in prior works Paquette et al.
; Gray et al.
(2020); Bakhtin et al.
If no player wins, SoS defines the score of player i as C 2 i / (cid:80) i ′
C
i ′ , where C i is the SC count for player i .
Diplomacy is specifically designed so that a player is unlikely to achieve victory without help from other players.
The full game allows unrestricted private natural-language communication between players each turn prior to choosing orders, but we focus on the simpler no-press variant, in which no such communication is allowed, but modeling how opponents will behave continues to be important.
I D IPLOMACY HYPER - PARAMETERS
We describe the describe the search parameters used in this work and compare it to those in previous works.
As compared to Gray et al.
(2020), we use a much less expensive set of search parameters for all results in the main section of this paper (See, Table 7).
In Table 8, we show that these tuned-down set of parameters, slightly reduces piKL-HedgeBot’s performance but allows us to make similar conclusions when comparing against SearchBot Gray et al.
and supervised learning-based bots from prior works.
Additionally, in Table 8, we also show that when our agent (piKL-HedgeBot ( λ = 10 − 3 )) uses the same search parameters as Gray et al.
(2020), it outperforms SearchBot by a big margin.
In our experiments, to compare against DipNet (Paquette et al., 2019)
we use the original model checkpoint
and we sample from the policy with temperature 0 .
1 (as used in prior works).
Similarly, to compare against SearchBot (Gray et al., 2020) agent we use the released checkpoint 7 and agent configuration 8 .
The only other search parameter unique and new to our piKL-HedgeBot algorithm is η in Algorithm 1, which we heuristically set on each Hedge iteration t to c/ ( σ √ t ) where σ is the standard deviation across iterations of the average utility experienced by the agent i being updated.
We find c = 10 / 3 works well for no-press Diplomacy.
[Table]
J D IPLOMACY M ODEL A RCHITECTURE AND I NPUT F EATURES
Our imitation learning policy model for Diplomacy uses the same transformer-encoder LSTM-decoder architecture as Bakhtin et al.
(2021) for reinforcement learning in Diplomacy, but applied to imitation learning over human games.
This architecture also
[Table] [Table]
resembles the architecture used by a significant amount of past work Gray et al.
Anthony et al.
; Paquette et al.
but replaces the graph-convolution encoder with a transformer, which we find to produce good results.
Additionally, we slightly modify the input feature encoding relative to Gray et al.
(2020), removing a small number of redundant channels and adding channels to indicate the “home centers” of each of the 7 powers (Austria, England, France,... etc), which are the locations where that power is allowed to build new armies or fleets.
See Table 9 for the new list of input features.
By adding the home centers to the input encoding instead of leaving them implicit, the game becomes entirely equivariant to permutations of those powers - e.g. if one swaps all the units of England and France, and all their centers, and which centers are their home centers, the resulting game is isomorphic to the original except with the two powers renamed.
This allows us to then augment the training data via equivariant permutations of the seven possible powers in the encoding.
Every time we sample a position from the dataset for training, we also choose among all 7-factorial permutations of the powers uniformly at random, and correspondingly permute both the input and output, to reduce overfit and improve the model’s generalization given the limited human data available
[Table]
K
For no-press Diplomacy, we note that Gray et al.
(2020) observed that their search agent benefits from short rollouts using the trained human policy before applying the human-learned value model to evaluate the position.
Doing so appears to result in more accurate evaluations reflecting the likely outcomes from a given game state, which the raw value model may failed to learn sufficiently accurately on the limited human dataset.
Since expectation of the learned value model after a short rollout appears to be better than the learned value model itself, this motivates training a model to directly approximate the former.
In a fashion broadly similar to Silver et al.
(2016) generating rollout games to train a more accurate value head for Go, we therefore generated a large stream of data by uniformly sampling positions from the human game dataset for Diplomacy, rolling them forward between 4-8 phases of game play via the same rollout settings as Gray et al.
i.e. policy sampling temperature 0.75, top-p 0.95, and training a new value model to predict the resulting post-rollout value estimate of the old value model.
Samples were continuously and asynchronously added to a replay buffer of 10000 batches, and the buffer was continuously sampled to train the same transformer-based architecture as the human-trained model from Appendix J initialized with the weights of that model.
Training was constrained to never exceed the rate of data generation by more than a factor of 2 (i.e. using each sample twice in expectation) and proceeded for 128000 mini-batches of 1024 samples each using the ADAM optimizer with a fixed learning rate of 1e-5.
L M ORE E XPERIMENTS IN D IPLOMACY
This section compiles additional performance results from evaluation games
. L. H EAD - TO -H EAD P ERFORMANCE
In Table 10, we compare the performance of piKL-HedgeBotin 1v6 head-to-head games against the underlying imitation anchor policy, following prior work Gray et al.
(2020); Bakhtin et al.
; Paquette et al.
Anthony et al.
We find that the λ = 10 − 1 policy is substantially stronger than the imitation policy while matching the accuracy in predicting human moves, while the λ = 10 − 3 policy outperforms unregularized search methods while playing much closer to the human policy.
[Table]
L. PI KL-H EDGE B OT ’ S PERFORMANCE IN POPULATION - BASED EXPERIMENTS
In this section, we provide all the results from the population experiments across various piKL-HedgeBot’s lambda values.
Figureand Table 11 show the results.
piKL-HedgeBot with λ = 10 − 3 performs best across individual population experiments with an
SoS score
[Table]
[Figure]
M D EC -POMDP G AMES : P OLICY - REGULARIZED SPARTA ON H ANABI
In this section, we extend KL-regularized search to decentralized partially observable Markov decision processes (Dec-POMDP) and test it on the Hanabi benchmark (Bard et al., 2020).
We first train an imitation learning policy (IL policy) from human data.
We then use the IL policy as the blueprint policy in SPARTA (Lerer et al., 2020), a search technique for Dec-POMDPs, and apply
KL-regularization toward the IL policy in SPARTA.
We call this new algorithm piKL-SPARTA.
We show that piKL-SPARTA matches or even slightly improves the original IL policy in human move prediction accuracy while greatly improving self-play performance.
M. B ACKGROUND
A Dec-POMDP is an N -player fully cooperative game with state space S that is partially observed by each player i through their individual observation function o i
= Ω i ( s ) for s ∈ S , with joint action space
A =
A 1 × A 2 × · · · × A N and transition function
T : S ×
A → P ( S ) that returns the distribution of next state given current state and joint action.
The reward function R : S ×
A → R assigns a scalar reward for the entire team at each time step.
A trajectory is denoted as τ t = ( s 0 , a 0 , r 0 , . . .
, s t ) while the action-observation history (AOH) of each player is defined as τ ti =
( o 0 , a 0 , r 0 , . . .
, o t ) .
A full trajectory or full AOH that reaches the terminal state may be denoted more simply as τ or τ i respectively.
The policy for each individual player π i ( a ti | τ ti ) takes as input the AOH and returns a distribution over valid actions.
The joint policy π = ( π 1 , . . .
, π N ) is a tuple containing all players’ policies.
The goal is to find a policy to maximize the expected total return π ∗ = arg max π J ( π ) =
E τ ∼ P ( τ | π ) R 0 ( τ ) where R t ( τ ) =
(cid:80) t
′ ≥ t γ ( t ′ − t ) r ′ t is the forward looking return with optional discount factor γ ≤ 1 .
Hanabi is a well-established large-scale Dec-POMDP benchmark (Bard et al., 2020).
It is a 2 to 5 player card game with a deck of 50 cards equally divided into 5 color suits.
Each color consists of five ranks with three 1s, two 2s, two 3s, two 4s, and one 5.
Each player draws five cards from a randomly shuffled deck to start the game.
The goal of the team is to play cards in order of increasing rank from 1 to 5 for every color suit.
Players take turns to either play a card, discard a card, or give a hint to another player about their cards.
When giving a hint, the acting player picks a receiver of the hint and a rank or color of any card in the receiver’s hand.
The recipient will then learn exactly which cards in their hand match the given rank or color.
Hinting costs one information token.
The team starts with eight information tokens and they can recoup one information token after discarding a card or successfully playing a 5 of any color.
If a player makes an invalid play, e.g. playing a red 3 when a red 2 is not played yet, the team loses one life token.
After each play or a discard, a player draws a new card if possible.
The game ends when three life tokens are lost, in which case the team receives 0 points, or one round after the entire deck is exhausted, in which case the score equals the number of cards successfully played.
The majority of existing works in the Hanabi domain focus on learning human compatible policies without using human data (Bard et al., 2020;
Siu et al., 2021;
Hu et al., 2021b).
Fewer works have explored better modeling human policies using human game data, partly due to the lack of publicly available datasets.
To the best of our knowledge, the strongest supervised learning agent trained from human data was done by Hu et al.
(2021b) where the authors use it as an unseen test-time partner agent in evaluation to estimate how well their agents might collaborate with humans.
No prior work has been done to better predict human moves in
Hanabi.
A few search techniques such as SPARTA (Lerer et al., 2020) and RL-search (Fickinger et al., 2021) have been proposed for large scale Dec-POMDPs and have specifically been applied in Hanabi.
In this work we choose SPARTA as our backbone for simplicity, while noting that our KL-regularization methods may also be generalized to other search techniques.
SPARTA is a test-time policy improvement algorithm that can be applied on top of any policy.
SPARTA assumes that a blueprint policy (BP) π is common knowledge in the Dec-POMDP and all players play their part of the blueprint unless they should deviate according to the SPARTA rule.
Here we briefly discuss the single-agent variant where the search agent assumes that other agents will always play the blueprint.
Given blueprint π , SPARTA first defines a belief function that tracks the distribution of the real trajectory given the search agent’s own AOH B i ( τ t ) =
P ( τ t | τ ti , π ) .
Then the search agent i computes the expected value for each action a using
Monte Carlo rollouts:
[Equation]
) where:
[Equation]
is the expected forward looking return on τ t assuming that the search agent will perform the action a for the current step and follow the blueprint afterwards while other agents always follow the blueprint.
SPARTA computes the belief B i ( τ t ) analytically.
It maintains a distribution of all possible trajectories and adjusts it at every step by removing the trajectories that contradict public knowledge or would have led to different joint actions according to the known joint policy π .
We start with an imitation learning policy (IL policy) trained from human gameplay data collected from an online Hanabi game platform.
The IL policy is used as both the baseline for predicting human moves as well as the blueprint policy for our piKL-SPARTA.
The analytical belief update procedure in SPARTA requires full knowledge of the partners’ policy.
This is challenging when predicting human moves and playing with humans because our IL policy models the average behavior of the entire population of players in the training dataset and a player at test time may perform actions that the IL policy will never do, leading to a null belief space that will terminate the search.
Therefore, we follow the practice in (Hu et al., 2021a) and train an approximate neural network belief model on self-play data generated by the IL policy.
Given the IL policy π as blueprint and the approximate belief model ˆ B i that replaces the B i in Eq. 29, piKL-SPARTA selects actions for the search player i following
[Equation]
M. E XPERIMENTAL S ETUP
We use a similar dataset acquired from en .
boardgamearena . com as in Hu et al.
(2021b).
The dataset consists of 240, 2-player Hanabi games.
We randomly sample 1,000 games to create a validation set and another 4,000 games for the test set.
The training set contains the remaining 235,954 games with an average score of 15.88.
Each game records the AOH
τ i , i ∈ { 1 , 2 } for both players.
The IL policy π θ is parameterized by a neural network θ and is trained to minimize the cross-entropy loss
[Equation]
with stochastic gradient descent.
The training set D is dynamically augmented with color shuffling (Hu et al., 2020) where a random color permutation that changes both observation and action space is applied to each trajectory τ i sampled from the dataset before feeding it to the network.
Hu et al.
(2021b) shows that this data augmentation method greatly reduces overfitting and leads to better policies.
The network θ uses the Public-LSTM structure in Hu et al.
(2021a), which eliminates the need to re-unroll
LSTM on sampled trajectories from the beginning of the game as they share the same public observations as the real trajectory.
To construct the approximate belief model ˆ B , we train a neural network ϕ to predict each player’s own hand, which is the only hidden information in Hanabi.
The network predicts each card in hand from oldest to newest auto-regressively.
We denote the hidden cards as { h ji } with h 1 i being the oldest and h mi being the newest, m ≤ 5 .
Then the cross-entropy loss for ϕ in an N -player setting becomes
[Equation]
In Hanabi, it is sufficient to reconstruct τ t given τ ti and { h ij } .
The model is trained on infinite stream of data generated by π θ through self-play to avoid overfitting.
We train two variants of the belief model, one with sampled action a ∼ π θ while the other with greedy action a = arg max π θ .
They are used by piKL-SPARTA and piKL-SPARTA-G(reedy) respectively.
We first run piKL-SPARTA on the test set to compare its ability to predict human moves against the IL policy.
For each τ ti in the test set, we sample K |
A i | hands from the belief model P ϕ where K is the total number of searches for this step and
| A i | is the number of legal actions of the search player.
In all our experiments we set K = 10 , 000 .
In practice, we sample 2 K | A i | hands and take the top K |
A i | samples not contradicting the public knowledge.
If the belief model fails to produce any samples that comply with the public knowledge, we revert back to the blueprint.
We then compute the expected value for each search action by unrolling the
IL policy until the end of the game for each sampled trajectory and compare a pred = arg max a π θ ( a | τ ti ) · exp[ Q π ( τ ti ,a ) λ ] against the human move a ti from the data.
We experiment with different λ to study its effect on prediction accuracy.
We also evaluate piKL-SPARTA in self-play to compare its performance under different λ against the IL policy.
We run piKL-SPARTA and the IL policy on 4,000 games with different seeds for the deck.
At each step, the IL policy acts following a t ∼ π θ ( a | τ ti ) while piKL-SPARTA acts following Eq.
(30) again using K = 10 , 000 rollouts.
The λ in these experiments are significantly higher than those in Diplomacy or implicitly in MCTS in Chess and Go 9 because λ represents the scale of utility difference that offsets a particular KL penalty, and the range of the utilities Q π ( τ ti , a ) in Hanabi is [ − 25 , 25] whereas the range in other games are either [0 , 1] or [ − 1 , 1] .
[Table]
We experiment with both 1p piKL-SPARTA, where one player uses piKL-SPARTA and the other follows the blueprint, and 2p piKL-SPARTA, where both players run the same single-agent version of piKL-SPARTA independently.
The latter is not theoretically sound and 2p SPARTA has in past papers produced worse performance than 1p SPARTA (Lerer et al., 2020) because 1p SPARTA assumes that partners are playing according to the blueprint policy while in actuality the partners are playing according to a SPARTA policy.
Despite the lack of theoretical soundness, we are interested in whether 2p piKL-SPARTA can obtain empirical improvement over the 1p version, since piKL-SPARTA regularizes towards the blueprint and therefore the mismatch between assuming the partner follows the blueprint and the policy they actually play would likely be less severe.
Lastly, in Dec-POMDPs, it is common to select actions greedily in self-play instead of sampling from a mixed policy, since in fully cooperative settings there is no need to avoid being deterministic or predictable to an adversary.
Therefore, we also experimented with piKL-SPARTA-G(reedy) where every agent, including the baseline IL policy, play according to the arg max of their action distributions at every step, and the belief model for piKL-SPARTA-G is trained on trajectories produced by a greedy IL policy.
M. R ESULTS
presented in
Table 12 summarizes the results for human move prediction.
On the full test set, piKL-SPARTA with λ = 10 and λ =outperform the IL policy slightly, although not statistically confidently, and vastly outperform unregularized SPARTA.
We also investigate the prediction accuracy on games with score ≥ 10 or ≥ 20 that more predominantly come from more experienced human players.
The improvement of piKL-SPARTA over the IL policy may be larger on these games, although this result is noisy and at best should be considered only mildly suggestive since filtering on final score also introduces other major confounding factors as well.
[Table]
In Table
[Table]
we show the self-play performance piKL-SPARTA and IL policy.
The top two rows show the results of the sampling version while the bottom two rows show those of the greedy version.
The conclusions are consistent across both cases.
Both 1p and 2p variants of piKL-SPARTA outscore IL for all λ tested.
Together with Table 12 we find with λ = 10 , piKL-SPARTA maintains IL prediction accuracy and outscores IL greatly in self-play, an overall improvement without any tradeoff.
For smaller λ = 2 or λ = 5 , piKL-SPARTA further outscores IL while losing some prediction accuracy on human moves, but remains vastly more accurate than unregularized SPARTA ( λ = 0 ).
Through qualitative analysis of the games played by both methods, we find that IL makes many mistakes due to both sampling low probability actions and the fact that the training set contains bad moves.
piKL-SPARTA, on the other hand, avoids many of these mistakes while still otherwise following the same strategies and humanlike signaling conventions.
It is particularly good at preventing catastrophic failures where the agents lose all life tokens and points since the Q values for good and bad actions differ much more widely in those cases.
The 2p piKL-SPARTA variants, despite being theoretically unsound, result in a further score improvement.
Meanwhile, we notice that 2p versions of the plain SPARTA ( λ = 0 ) underperform their 1p variants, which is consistent with the observations from Lerer et al.
despite us using an approximate learned belief model instead of exact belief.
This suggests that regularization towards the blueprint IL policy is successful in keeping the trajectories similar enough to those from the blueprint that the unsound assumption that the partner plays the blueprint does not cause major problems, while still allowing both players to deviate enough to correct major blunders that they would otherwise make.
Lastly, we notice that 1p piKL-SPARTA with λ = 0 .
5 outperforms the unregularized version in self-play.
The reason could be that the quality of the samples from the learned belief model worsen as the trajectories it sees at test time becomes more off-distribution for smaller λ .
For reference, we also run the original SPARTA which does not have this problem as it computes beliefs analytically.
The original SPARTA gets 23.49 ± 0.02, which is better than 1p piKL-SPARTA with λ = 0 .
5 but worse than 2p piKL-SPARTA with the same λ .
The computational cost of 1p SPARTA with exact beliefs and 2p piKL-SPARTA with approximate learned beliefs is roughly the same because the exact beliefs computation accounts for 50% of the entire computation.
Therefore, 2p piKL-SPARTA could be a preferable option to improve performance in Dec-POMDPs via self play without incurring the huge cost of joint SPARTA or RL-search (Fickinger et al., 2021).