H UMAN -L EVEL P ERFORMANCE IN N O -P RESS D IPLOMACY VIA E QUILIBRIUM S EARCH
Anonymous authors
Paper under double-blind review
A BSTRACT
Prior AI breakthroughs in complex games have focused on either the purely adversarial or purely cooperative settings.
In contrast, Diplomacy is a game of shifting alliances that involves both cooperation and competition.
For this reason, Diplomacy has proven to be a formidable research challenge.
In this paper we describe an agent for the no-press variant of Diplomacy that combines supervised learning on human data with one-step lookahead search via external regret minimization.
External regret minimization techniques have been behind previous AI successes in adversarial games, most notably poker, but have not previously been shown to be successful in large-scale games involving cooperation.
We show that our agent greatly exceeds the performance of past no-press Diplomacy bots, is unexploitable by expert humans, and achieves a rank of 23 out of 1,128 human players when playing anonymous games on a popular Diplomacy website.
I NTRODUCTION
A primary goal for AI research is to develop agents that can act optimally in real-world multi-agent interactions (i.e., games ).
In recent years, AI agents have achieved expert-level or even superhuman performance in benchmark games such as backgammon (Tesauro, 1994), chess (Campbell et al., 2002), Go (Silver et al., 2016; 2017; 2018),
poker (Morav ˇ cík et al., 2017; Brown & Sandholm, 2017; 2019b), and real-time strategy games (Berner et al., 2019; Vinyals et al., 2019).
However, previous large-scale game AI results have focused on either purely competitive or purely cooperative settings.
In contrast, real-world games, such as business negotiations, politics, and trafﬁc navigation, involve a far more complex mixture of cooperation and competition.
In such settings, the theoretical grounding for the techniques used in previous AI breakthroughs falls apart.
In this paper we augment neural policies trained through imitation learning with regret minimization search techniques, and evaluate on the benchmark game of no-press Diplomacy.
Diplomacy is a longstanding benchmark for research that features a rich mixture of cooperation and competition.
Like previous researchers, we evaluate on the widely played no-press variant of Diplomacy, in which communication can only occur through the actions in the game (i.e., no cheap talk is allowed).
Speciﬁcally, we begin with a blueprint policy that approximates human play in a dataset of Diplomacy games.
We then improve upon the blueprint during play by approximating an equilibrium for the current phase of the game, assuming all players (including our agent) play the blueprint for the remainder of the game.
Our agent then plays its part of the computed equilibrium.
The equilibrium is computed via external regret matching (ERM) (Blackwell et al., 1956; Hart & Mas-Colell, 2000).
Search via ERM has led to remarkable success in poker.
However, ERM only converges to a Nash equilibrium in two-player zero-sum games and other special cases, and ERM was never previously shown to produce strong policies in a mixed cooperative/competitive game as complex as no-press
Diplomacy.
Nevertheless, we show that our agent exceeds the performance of prior agents and for the ﬁrst time convincingly achieves human-level performance in no-press Diplomacy.
Speciﬁcally, we show that our agent soundly defeats previous agents, that our agent is far less exploitable than previous agents, that an expert human cannot exploit our agent even in repeated play, and, most importantly, that our agent achieves a score of 25.6% when playing anonymously with humans on a popular Diplomacy website, compared to an average human score of 14.3%.
B ACKGROUND AND R ELATED W ORK
Search has previously been used in almost every major game AI breakthrough, including backgammon (Tesauro, 1994), chess (Campbell et al., 2002), Go (Silver et al., 2016; 2017; 2018), poker (Moravˇcík et al., 2017; Brown & Sandholm, 2017; 2019b), and Hanabi (Lerer et al., 2020).
A major exception is real-time strategy games (Vinyals et al., 2019; Berner et al., 2019).
Similar to
SPARTA as used in Hanabi (Lerer et al., 2020), our agent conducts one-ply lookahead search (i.e., changes the policy just for the current game turn) and thereafter assumes all players play according to the blueprint.
Similar to the Pluribus poker agent (Brown & Sandholm, 2019b), our search technique uses external regret matching to compute an approximate equilibrium.
In a manner similar to the sampled best response algorithm of Anthony et al.
(2020), we sample a limited number of actions from the blueprint policy rather than search over all possible actions, which would be intractable.
Learning effective policies in games involving cooperation and competition has been studied extensively in the ﬁeld of multi-agent reinforcement learning (MARL) (Shoham et al., 2003).
Nash-Q and CE-Q applied Q learning for general sum games by using Q values derived by computing Nash (or correlated) equilibrium values at the target states (Hu & Wellman, 2003; Greenwald et al., 2003).
Friend-or-foe Q learning treats other agents as either cooperative or adversarial, where the Nash Q values are well deﬁned Littman (2001).
The recent focus on "Deep" MARL has led to learning rules from game theory such as ﬁctitious play and regret minimization being adapted to Deep reinforcement learning (Heinrich & Silver, 2016; Brown et al., 2019), as well as work on game-theoretic challenges of mixed cooperative/competitive settings such as social dilemmas and multiple equilibria in the MARL setting (Leibo et al., 2017; Lerer & Peysakhovich, 2017; 2019).
Diplomacy in particular has served for decades as a benchmark for multi-agent AI research (Kraus & Lehmann, 1988; Kraus et al., 1994; Kraus & Lehmann, 1995; Johansson & Håård, 2005; Ferreira et al., 2015).
Recently, Paquette et al.
(2019) applied imitation learning (IL) via deep neural networks on a dataset of more than 150,000 Diplomacy games.
This work greatly improved the state of the art for no-press Diplomacy, which was previously a handcrafted agent (van Hal, 2013).
Paquette et al.
(2019) also tested reinforcement learning (RL) in no-press Diplomacy via Advantage Actor-Critic (A2C)
(Mnih et al., 2016).
Anthony et al.
introduced sampled best response policy iteration, a self-play technique, which further improved upon the performance of Paquette et al.
2. D ESCRIPTION OF D IPLOMACY
The rules of no-press Diplomacy are complex; a full description is provided by Paquette et al.
No-press Diplomacy is a seven-player zero-sum board game in which a map of Europe is divided into 75 provinces.
34 of these provinces contain supply centers (SCs), and the goal of the game is for a player to control a majority (18) of the SCs.
Each players begins the game controlling three or four SCs and an equal number of units.
The game consists of three types of phases: movement phases in which each player assigns an order to each unit they control, retreat phases in which defeated units retreat to a neighboring province, and adjustment phases in which new units are built or existing units are destroyed.
During a movement phase, a player assigns an order to each unit they control.
A unit’s order may be to hold (defend its province), move to a neighboring province, convoy a unit over water, or support a neighboring unit’s hold or move order.
Support may be provided to units of any player.
We refer to a tuple of orders, one order for each of a player’s units, as an action .
That is, each player chooses one action each turn.
There are an average of 26 valid orders for each unit (Paquette et al., 2019), so the game’s branching factor is massive and on some turns enumerating all actions is intractable.
Importantly, all actions occur simultaneously.
In live games, players write down their orders and then reveal them at the same time.
This makes the game an imperfect-information game in which an optimal policy may need to be stochastic in order to prevent predictability.
Diplomacy is designed in such a way that cooperation with other players is almost essential in order to achieve victory, even though only one player can ultimately win.
A game may end in a draw on any turn if all remaining players agree.
Draws are a common outcome among experienced players because players will often coordinate to prevent any individual from reaching 18 centers.
The two most common scoring systems for draws are draw-size scoring (DSS) , in which all surviving players equally split a win, and sum-of-squares scoring (SoS) , in which player i receives a score of
C 2 i (cid:80) j ∈N C 2 j , where C i is the number of SCs that player i controls (Fogel, 2020).
Throughout this paper we use SoS scoring except in anonymous games against humans where the human host chooses a scoring system.
2. E XTERNAL R EGRET M ATCHING
External Regret Matching (ERM) (Blackwell et al., 1956; Hart & Mas-Colell, 2000) is an iterative algorithm that converges to a Nash equilibrium (NE) (Nash, 1951) in two-player zero-sum games and other special cases, and converges to a coarse correlated equilibrium (CCE) (Hannan, 1957) in general.
We consider a game with N players where each player i chooses an action a i from a set of actions
A i .
We denote the joint action as a = ( a 1 , a 2 , . . .
, a N ) , the actions of all players other than i as a − i , and the set of joint actions as A .
After all players simultaneously choose an action, player i receives a reward of v i ( a ) (which can also be represented as v i ( a i , a − i ) ).
Players may also choose a probability distribution over actions, where the probability of action a i is denoted π i ( a i ) and the vector of probabilities is denoted π i .
Normally, each iteration of ERM has a computational complexity of Π i ∈N |A
i | .
In a seven-player game, this is typically intractable.
We therefore use a sampled form of ERM in which each iteration has a computational complexity of (cid:80) i ∈N |A
i | .
We now describe this sampled form of ERM.
Each agent i maintains an external regret value for each action a
i ∈
A i , which we refer to simply as regret .
The regret on iteration t is denoted R ti ( a i ) .
Initially, all regrets are zero.
On each iteration t of ERM, π ti ( a i ) is set according to
[Equation]
Next, each player samples an action a ∗ i from A i according to π ti and all regrets are updated such that
[Equation]
This sampled form of ERM guarantees that R ti ( a i ) ∈ O ( √ t ) with high probability (Lanctot et al., 2009).
If R ti ( a i ) grows sublinearly for all players’ actions, as is the case in ERM, then the average policy over all iterations converges to a NE in two-player zero-sum games and in general the empirical distribution of players’ joint policies converges to a CCE as t → ∞ .
In order to improve empirical performance, we use linear ERM (Brown & Sandholm, 2019a), which weighs updates on iteration t by t .
1 We also use optimism (Syrgkanis et al., 2015), in which the most recent iteration is counted twice when computing regret.
Additionally, the action our agent ultimately plays is sampled from the ﬁnal iteration’s policy, rather than the average policy over all iterations.
This reduces the risk of sampling a non-equilibrium action due to insufﬁcient convergence.
In theory, sampling from the ﬁnal iteration may increase exploitability, but this technique has been used successfully in past poker agents (Brown & Sandholm, 2019b).
A GENT D ESCRIPTION
Our agent is composed of two major components.
The ﬁrst is a blueprint policy and state-value function trained via imitation learning on human data.
The second is a search algorithm that utilizes the blueprint.
This algorithm is executed on every turn, and approximates an equilibrium policy (for all players, not just the agent) for the current turn via ERM, assuming that the blueprint is played by all players for the remaining game beyond the current turn.
3. S UPERVISED L EARNING
We construct a blueprint policy via imitation learning on a corpus of 46,148 Diplomacy games collected from online play, building on the methodology and model architecture described by Paquette et al.
(2019) and Anthony et al.
A blueprint policy and value function estimated from human play is ideal for performing search in a general-sum game, because it is likely to realistically approximate state values and other players’ actions when playing with humans.
Our blueprint supervised model is based on the DipNet agent from Paquette et al.
, but we make a number of modiﬁcations to the architecture and training.
A detailed description of the architecture is provided in Appendix A; in this section we highlight the key differences from prior work.
We trained the blueprint policy using only a subset of the data used by Paquette et al.
(2019), specifically those games obtained from webdiplomacy.net .
For this subset of the data, we obtained metadata about the press variant (full-press vs. no-press) which we add as a feature to the model, and anonymized player IDs for the participants in each game.
Using the IDs, we computed ratings s i for each player i and only trained the policy on actions from players with above-average ratings.
Appendix B describes our method for computing these ratings.
Our model closely follows the architecture of Paquette et al.
(2019), with additional dropout of 0.4 between GNN encoder layers.
We model sets of build orders as single tokens because there are a small number of build order combinations and it is tricky to predict sets auto-regressively with teacher forcing.
We adopt the encoder changes of Anthony et al.
(2020), but do not adopt their relational order decoder because it is more expensive to compute and leads to only marginal accuracy improvements after tuning dropout.
We make a small modiﬁcation to the encoder GNN architecture that improves modeling.
In addition to the standard residual that skips the entire GNN layer, we replace the graph convolution 2 with the sum of a graph convolution and a linear layer.
This the model a hierarchy
[Equation]
where A is a learned linear transformation.
Finally, we achieve a substantial improvement in order prediction accuracy using a featurized order decoder .
Diplomacy has over 13,000 possible orders, many of which will be observed infrequently in the training data.
Therefore, by featurizing the orders by the order type, and encodings of the source, destination, and support locations, we observe improved prediction accuracy.
Speciﬁcally, in a standard decoder each order o has a learned representation e o , and for some board encoding x and learned order embedding e o , P ( o ) =
softmax ( x · e o ) .
With order featurization, we use ˜ e o =
e o +
Af o , where f o are static order features and A is a learned linear transformation.
The order featurization we use is the concatenation of the one-hot order type with the board encodings for the source, destination, and support locations.
We found that representing order location features by their location encodings works better than one-hot locations, presumably because the model can learn more state-contextual features.
We add an additional value head to the model immediately after the dipnet encoder, that is trained to estimate the ﬁnal SoS scores given a board situation.
We use this value estimate during equilibrium search (Sec. 3.2) to estimate the value of a Monte Carlo rollout after a ﬁxed number of steps.
The value head is an MLP with one hidden layer that takes as input the concatenated vector of all board position encodings.
A softmax over powers’ SoS scores is applied at the end to enforce that all players’ SoS scores sum to 1. [Table]
3. E QUILIBRIUM S EARCH
The policy that is actually played results from a search algorithm which utilizes the blueprint policy.
Let s be the current state of the game.
On each turn, the search algorithm computes an equilibrium for a subgame and our agent plays according to its part of the equilibrium solution for its next action.
Conceptually, the subgame is a well-deﬁned game that begins at state s .
The set of actions available to each player is a subset of the possible actions in state s in the full game, and are referred to as the subgame actions .
Each player i chooses a subgame action a i , resulting in joint subgame action a .
After a is taken, the players make no further decisions in the subgame.
Instead, the players receive a reward corresponding to the players sampling actions according to the blueprint policy π b for the remaining game.
The subgame actions for player i are the Mk i highest-probability actions according to the blueprint model, where k i is the number of units controlled by player i and M is a hyperparameter (usually set to 5).
The effect of different numbers of subgame actions is plotted in Figure 1 (left).
Rolling out π b to the end of the game is very expensive, so in practice we instead roll out π b for a small number of turns (usually 2 or
movement phases in our experiments) until state s (cid:48) is reached, and then use the value for s (cid:48) from the blueprint’s value network as the reward vector.
Figure(right) shows the performance of our search agent using different rollout lengths.
We do not observe improved performance for rolling out farther than 3 or 4 movement phases.
We compute a policy for each agent by running the sampled regret matching algorithm described in Section 2.2.
The search algorithm used 256 – 4,096 iterations of RM and typically required between 2 minutes and 20 minutes per turn using a single Volta GPU and 8 CPU cores, depending on the hyperparameters used for the game.
Details on the hyperparameters we used are provided in Appendix F.
R ESULTS
Using the techniques described in Section 3, we developed an agent we call SearchBot.
Our experiments focus on two formats.
The ﬁrst evaluates the head-to-head performance of SearchBot playing against the population of human players on a popular Diplomacy website, as well as against prior
AI agents.
The second measures the exploitability of SearchBot.
4. P ERFORMANCE AGAINST A POPULATION OF HUMAN PLAYERS
The ultimate test of an AI system is how well it performs in the real world with humans.
To measure this, we had SearchBot anonymously play no-press Diplomacy games on the popular Diplomacy website webdiplomacy.net .
Since there are 7 players in each game, average human performance is a score of 14.3%.
In contrast, SearchBot scored 25.6% ± 4.8%.
The agent’s performance is shown in Table 2 and a detailed breakdown is presented in Table 5 in Appendix F. [Table]
In addition to raw score, we measured SearchBot’s performance using the Ghost-Rating system (Anthony, 2020), which is a Diplomacy rating system inspired by the Elo system that accounts for the relative strength of opponents and that is used to semi-ofﬁcially rank players on webdiplomacy.net .
Among no-press Diplomacy players on the site, our agent ranked 23 out of 1,128 players with a Ghost-Rating of 176.0 as of September 30th, 2020.
7 Details on the setup for experiments are provided in Appendix F. 4. P ERFORMANCE A GAINST O THER AI A GENTS In addition to playing against humans, we also evaluated SearchBot against prior AI agents from Paquette et al. (one based on supervised learning, and another based on reinforcement learning) and our blueprint agent.
Table 3 shows these results.
Following prior work, we compute average scores in ‘1v6’ games containing a single agent of type A and six agents of type
B. The average score of an identical agent should therefore be 1 / 7 ≈ 14 . SearchBot achieves its highest 1v6 score when matched against its own blueprint, since it is most accurately able to approximate the behavior of that agent. It outperforms all three agents by a large margin, and none of the three baselines is able to achieve a score of more than 1% against our search agent.
4. E XPLOITABILITY
While performance of an agent within a population of human players is the most important metric, that metric alone does not capture how the population of players might adapt to the agent’s presence.
For example, if our agent is extremely strong then over time other players might adopt the bot’s playstyle.
As the percentage of players playing like the bot increases, other players might adopt a policy that seeks to exploit this playstyle.
Thus, if the bot’s policy is highly exploitable then it might eventually do poorly even if it initially performs well against the population of human players.
Motivated by this, we measure the exploitability of our agent.
Exploitability of a policy proﬁle π (denoted e ( π ) ) measures worst-case performance when all but one agents follows π .
Formally, the [Table] exploitability of π is deﬁned as e ( π ) =
(cid:80) i ∈N (max π ∗ i v i
( π ∗ i , π − i ) − v i ( π )) /N , where π − i denotes the policies of all players other than i .
Agent i ’s best response to π − i is deﬁned as BR ( π − i ) =
arg max π i v i
( π i , π − i ) .
We estimate our agent’s full-game exploitability in two ways: by training an RL agent to best respond to the bot, and by having expert humans repeatedly play against six copies of the bot.
We also measure the ‘local’ exploitability in the search subgame and show that it converges to an approximate Nash equilibrium.
4.3. P ERFORMANCE AGAINST A BEST - RESPONDING AGENT
When the policies of all players but one are ﬁxed, the game becomes a Markov Decision Process (MDP)
(Howard, 1960) for the non-ﬁxed player because the actions of the ﬁxed players can be viewed as stochastic transitions in the “environment”.
Thus, we can estimate the exploitability of π by ﬁrst training a best response policy BR ( π − i ) for each agent i using any single-agent RL algorithm, and then computing (cid:80) i ∈N v i ( BR ( π − i ) , π − i ) /N .
Since the best response RL policy will not be an exact best response (which is intractable to compute in a game as complex as no-press
Diplomacy) this only gives us a lower-bound estimate of the exploitability.
Following other work on environments with huge action spaces (Vinyals et al., 2019; Berner et al., 2019), we use a distributed asynchronous actor-critic RL approach to optimize the exploiter policy (Espeholt et al., 2018).
We use the same architecture for the exploiter agent as for the ﬁxed model.
Moreover, to simplify the training we initialize the exploiter agent from the ﬁxed model.
We found that training becomes unstable when the policy entropy gets too low.
The standard remedy is to use an entropy regularization term.
However, due to the immense action space, an exact computation of the entropy term, E a log p θ ( a ) , is infeasible.
Instead, we optimize a surrogate loss that gives an unbiased estimate of the gradient of the entropy loss (see Appendix D).
We found this to be critical for the stability of the training.
Training an RL agent to exploit SearchBot is prohibitively expensive.
Even when choosing hyperparameters that would result in the agent playing as fast as possible, SearchBot typically requires at least a full minute in order to act each turn.
Instead, we collect a dataset of self-play games of
SearchBot and train a supervised agent on this dataset.
The resulting agent, which we refer to as
SearchBot-clone, is weaker than SearchBot but requires only a single pass through the neural network in order to act on a turn.
By training an agent to exploit SearchBot-clone, we can obtain a (likely) upper bound on what the performance would be if a similar RL agent were trained against
SearchBot.
We report the reward of the exploiter agents against the blueprint and SearchBot-clone agents in Figure 2.
The results show that SearchBot-clone is highly exploitable, with a best responder able to score at least 42% against SearchBot-clone.
Any score above 14.3% means the best responder is winning in expectation.
However, SearchBot-clone appears to be much less exploitable than the blueprint, which a best responder can beat with a score of at least 54%.
Also, we emphasize again that SearchBot is almost certainly far less exploitable than SearchBot-clone.
[Figure] SearchBot [Figure] 4.3. P ERFORMANCE A GAINST E XPERT H UMAN E XPLOITERS
In addition to training a best-responding agent, we also invited the 1st and 2nd place ﬁnishers in the 2017 World Diplomacy Convention (widely considered the world championship for full-press
Diplomacy) to play games against six copies of our agent.
The purpose was to determine whether the human experts could discover exploitable weaknesses in the bot.
The humans played games against three types of bots:
DipNet (Paquette et al., 2019) (with temperature set to 0.5), our blueprint agent (with temperature set to 0.5), and SearchBot.
In total, the participants played 35 games against each bot; each of the seven powers was controlled by a human player ﬁve times, while the other six powers were controlled by identical copies of the bot.
The performance of the humans is shown in Table 4.
While the sample size is relatively small, the results suggest that our agent is less exploitable than prior bots.
Our improvements to the supervised learning policy reduced the humans’ score from 39.1% to 22.5%, which is itself a large improvement.
However, the largest gains came from search.
Adding search reduced the humans’ score from 22.5% to 5.7%.
A score below 14.3% means the humans are losing on average.
Power 1 Human vs
[Equation] 6 DipNet [Equation] [Equation]
[Table]
4.3. E XPLOITABILITY IN L OCAL S UBGAME
We ﬁrst investigate the exploitability of our agent in the local subgame deﬁned by a given board state, sampled actions, and assumed blueprint policy for the rest of the game.
We simulate 7 games between a search agent and 6 DipNet agents [Equation] and plot the total exploitability of the average strategy of the search procedure as a function of the number of ERM iterations [Equation]
as well as the exploitability of the blueprint policies.
Utilities u i are computed using Monte Carlo rollouts with the same (blueprint) rollout policy used during ERM
[Equation]
and total exploitability for a joint policy π is computed as e ( π ) = [Equation]
[Equation]
u [Equation]
( π ) .
The exploitability curves aggregated over all phases are shown in Figure 3 (left) and broken down by phase in the Appendix.
In Figure 3 (right), we verify that the average of policies from multiple independent executions of ERM also converges to an approximate Nash.
For example, it is possible that if each agent independently running ERM converged to a different incompatible equilibrium and played their part of it, then the joint policy of all the agents would not be an equilibrium.
However, we observe that the exploitibility of the average of policies closely matches the exploitability of the individual policies.
C ONCLUSIONS
No-press Diplomacy is a complex game involving both cooperation and competition that poses major theoretical and practical challenges for past AI techniques.
Nevertheless, our AI agent achieves human-level performance in this game with a combination of supervised learning on human data and one-ply search using external regret minimization.
The massive improvement in performance from conducting search just one action deep matches a larger trend seen in other games, such as chess, Go, poker, and Hanabi, in which search dramatically improves performance.
While external regret minimization has been behind previous AI breakthroughs in purely competitive games such as poker, it was never previously shown to be successful in a complex game involving cooperation.
The success of ERM in no-press Diplomacy suggests that its use is not limited to purely adversarial games.
Our work points to several avenues for future research.
SearchBot conducts search only for the current turn.
In principle, this search could extend deeper into the game tree using counterfactual regret minimization (CFR) (Zinkevich et al., 2008).
However, the size of the subgame grows exponentially with the depth of the subgame.
Developing search techniques that scale more effectively with the depth of the game tree may lead to substantial improvements in performance.
Another direction is combining our search technique with reinforcement learning.
Combining search with reinforcement learning has led to tremendous success in perfect-information games (Silver et al., 2018) and more recently in two-player zero-sum imperfect-information games as well (Brown et al., 2020).
Finally, it remains to be seen whether similar search techniques can be developed for variants of Diplomacy that allow for coordination between agents.
A F ULL I MITATION L EARNING M ODEL A RCHITECTURE D ESCRIPTION
In this section, we provide a full description of the model architecture used for the supervised Diplomacy agent.
The model architecture is shown in Figure 4 .
It consists of a shared encoder and separate policy and value decoder heads.
The input to the encoder is a set of features for each of the 81 board locations (75 regions +coasts).
The features provided for each location are shown in Figure 5.
The ﬁrst 7 feature types ( features/location) match those of Paquette et al.
(2019), and the last three feature types (in italics) are for global board features that are common across all locations: a 20-d embedding of the current season; the net number of builds for each power during a build phase; and a single feature identifying whether it is a no-press or full-press game.
The encoder has two parallel GNN trunks.
The Current State Encoder takes the current board state as input, while the Previous State Encoder takes the board state features from the last movement phase, concatenated with a 20-dimensional feature at each location that is a sum of learned embeddings for each order that ocurred at this location during all phases starting at the last movement phase.
The current board state and previous board state are each fed to an 8-layer GNN, whose layer architecture is shown in Figure 6.
The GraphConv operation consists of a separate learned linear transformation applied at each board location, followed by application of the normalized board adjacency matrix.
In parallel with the GraphConv is a linear layer that allows the learning of a feature hierarchy without graph smoothing.
This is followed by batchnorm, a ReLU non-linearity, and dropout probability 0.4.
The 81 × 120 output features from each of these GNNs are then concatenated and fed to another 8-layer GNN with dimensional 240 .
The output of this encoder arefeatures for each board location.
The value head takes as input all 81 × 240 encoder features concatenated into a single 19440dimensional vector.
An MLP with a single hidden layer followed by a softmax produces predicted sum-of-squares values for each power summing to 1.
They are trained to predict the ﬁnal SoS values with an MSE loss function.
The policy head auto-regressively predicts the policy for a particular power one unit at a time with a 2-layer LSTM of width 200.
Unit orders are predicted in the global order used by Paquette et al.
The input to the LSTM at each step is the concatenation of (a) the 240-d encoder features of the location whose orders are being predicted; (b) a 60-d embedding of the power whose policy is being computed; (c) an 80-d embedding of the predicted orders for the last unit.
A standard decoder matrix can be thought of as a learned vector for each valid order, whose dot product with the LSTM output predicts the logit of the order probability given the state.
We featurize these order vectors as shown in Figure 7.
The decoder vector for “PAR S RUH - BUR” , for example, would be the sum of a learned vector for this order and learned linear transformations of (a) one-hot features for the order source (RUH), destination (BUR), type (S); and (b) the encoder features of the source (RUH) and destination (BUR) locations.
Build orders are different than other order types because the set of source locations for build orders is not known in advance.
So the decoder needs to predict an unordered set of build orders during a build phase, which is tricky to compute correctly with an autoregressive decoder and teacher forcing.
Luckily, since a power can only build in its 3-4 home supply centers, there are only 170 sets of build orders across all powers, so we just consider each possible set of builds as a single order, rather than predicting them one by one.
We do not do featurized order decoding for build orders.
B C OMPUTING P LAYER R ATINGS IN H UMAN D ATA
To compute player ratings, we used a regularized logistic outcome model.
Speciﬁcally, we optimized the loss
[Equation]
across all pairs of players ( i, j ) ∈ D where player i achieved a "better" outcome than player j in a game.
We found this approach led to more plausible scores than Elo (Elo, 1978) or TrueSkill (Herbrich et al., 2007) ratings.
Paquette et al.
(2019) took an orthogonal approach to ﬁlter poor players from the training data: they only trained on "winning" powers, i.e. those who ended the game with at least 7 SCs.
This ﬁltering is sensible for training a policy for play, but is problematic for training policies for search.
In a general-sum game, it is crucial for the agent to be able to predict the empirical distribution of actions even for other agents who are destined to lose.
C S EARCH E XAMPLE
The listing below shows the policy generated by one run of our search algorithm for the opening move of Diplomacy when running for 512 iterations and considering 8 possible actions per power.
For each possible action, the listing below shows
probs : The probability of the action in the ﬁnal strategy.
bp_p : The probability of the action in the blueprint strategy.
avg_u :
The average predicted sum-of-squares utility of this action.
orders : The orders for this action
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Table]
D RL DETAILS
Each action a in the MDP is a sequence of orders ( o 1 , . . .
, o t ) .
The probability of the order a under policy π θ is deﬁned by an LSTM in auto regressive fashion, i.e., π θ ( a ) = (cid:81) ti =1 ( π θ ( o i | o 1 . . .
o i − 1 )) .
To make training more stable, we would like to prevent entropy H ( π θ ) :
= − E a ∼ π θ log( π θ ( a ) from collapsing to zero.
The naive way to optimize the entropy of the joint distribution is to use a sum of entropies for each individual order, i.e., ddθ H ( π θ ( • ))
≈ (cid:80) ti =1 ddθ H
( π θ ( •| o 1 . . .
o i − 1 )) .
However, we found that this does not work well for our case probably because there are strong correlations between orders.
Instead we use an unbiased estimate of the joint entropy that is agnostic to the size of the action space and requires only to be able to sample from a model and to adjust probabilities of the samples.
Statement 1.
Let π θ ( • ) be a probability distribution over a discrete set A ,
such that ∀ a ∈
A π θ ( a ) is a smooth function of a vector of parameters θ .
Then
[Equation]
Proof.
Proof is similar to one for REINFORCE:
[Equation]
E S UBGAME E XPLOITABILITY R ESULTS
Figure E plots the total exploitability of joint policies computed by ERM in the subgame used for equilibrium search at each phase in 7 simulated Diplomacy games.
F D ETAILS ON E XPERIMENTAL S ETUP
Most games on webdiplomacy.net were played with 24-hour turns, though the agent also played in some “live” games with 5-minute turns.
Different hyperparameters were used for live games versus non-live games.
In non-live games, we typically ran ERM for 2,048 iterations with a rollout length of 3 movement phases, and set M (the constant which is multiplied by the number of units to determine the number of subgame actions) equal to 5.
This typically required aboutminutes to compute.
In live games (including games in which one human played against six bots) and in games against non-human opponents, we ran ERM for 256 iterations with a rollout length of 2 movement phases, and set M equal to 3.5.
This typically required about 2 minutes to compute.
In all cases, the temperature for the blueprint in rollouts was set to 0.75.
The experiments on webdiplomacy.net occurred over a three-month timespan, with games commonly taking one to two months to complete (players are typically given 24 hours to act).
Freezing research and development over such a period would have been impractical, so our agent was not ﬁxed for the entire time period.
Instead, serious bugs were ﬁxed, improvements to the algorithm were made, and the model was updated.
In games on webdiplomacy.net , a draw was submitted if SearchBot did not gain a center in two years, or if the agent’s projected sum-of-squares score was less than the score it would achieve by an immediate draw.
Since there was no way to submit draws through the webdiplomacy.net
API, draws were submitted manually once the above criteria was satisﬁed.
Games against other bots, shown in Table 3, automatically ended in a draw if no player won by 1935.
G Q UALITATIVE A SSESSMENT OF S EARCH B OT
Qualitatively, we observe that SearchBot performs particularly well in the early and mid game.
However, we observe that it sometimes struggles with endgame situations.
In particular, when it is clear that one power will win unless the others work together to stop it, SearchBot will sometimes continue to attack its would-be allies.
There may be multiple contributing factors to this.
One important limitation, which we have veriﬁed in some situations, is that the sampled subgame actions may not contain any action that could prevent a loss.
This is exacerbated by the fact that players typically control far more units in the endgame and the number of possible actions grows expo- [Table] [Table] nentially with the number of units, so the sampled subgame actions contain a smaller fraction of all possible actions.
Another possible contributing factor is that the state space near the end of the game is far larger, so there is relatively less data for supervised learning in this part of the game.
Finally, another possibility is that because the network only has a dependence on the state of the board and the most recent player actions, it is unable to sufﬁciently model the future behavior of other players in response to being attacked.
Although the sample size is small, the results suggest that SearchBot performed particularly well with the central powers of Austria, Germany, and Italy.
These powers are considered to be the most difﬁcult to play by humans because they are believed to require an awareness of the interactions between all players in the game.