A DVERSARIAL D IVERSITY IN H ANABI
A BSTRACT
Many Dec-POMDPs admit a qualitatively diverse set of “reasonable” joint policies, where reasonableness is indicated by symmetry equivariance, nonsabotaging behaviour and the graceful degradation of performance when paired with ad-hoc partners.
Some of the work in diversity literature is concerned with generating these policies.
Unfortunately, existing methods fail to produce teams of agents that are simultaneously diverse, high performing, and reasonable.
In this work, we propose a novel approach, adversarial diversity (ADVERSITY), which is designed for turn-based Dec-POMDPs with public actions.
ADVERSITY relies on off-belief learning to encourage reasonableness and skill, and on “repulsive” fictitious transitions to encourage diversity.
We use this approach to generate new agents with distinct but reasonable play styles for the card game Hanabi and opensource our agents to be used for future research on (ad-hoc) coordination.
I NTRODUCTION
A key objective of cooperative multi-agent reinforcement learning (MARL) is to produce agents capable of coordinating with novel partners, including other artificial agents and ultimately humans.
In order to make progress on this objective, a number of works have focused on the general challenge of ad-hoc team play , which is to create autonomous agents able to “ efficiently and robustly collaborate with previously unknown teammates on tasks to which they are all individually capable of contributing as team members ” (Stone et al., 2010).
To evaluate such agents, many works on ad-hoc coordination rely on evaluation setups similar to the one proposed by Stone et al.
(2010), pairing the agents at test time with partners sampled from a pre-determined pool.
The value of such evaluations depends on the size and quality of the pool of partners.
A pool that is too small or too homogeneous may not be representative of all possible play-styles, and provide an inaccurate evaluation of the coordination capabilities of an agent.
For this reason, previous works in coordination have relied on various approaches to generate a diverse pool of partners.
A first approach is to handcraft policies, either directly or by shaping the reward at train-time (Albrecht, 2015; Barrett et al., 2017; Zand et al., 2022), but it requires domain knowledge and scales poorly.
Another is to train a population with varying hyperparameters or by deploying multiple RL algorithms on the same task (Nekoei et al., 2021;
Zand et al., 2022; Albrecht, 2015).
The diversity achieved this way is unclear, since it is a byproduct of the variability of the algorithms used rather than being actively optimized for.
Yet other works augment training with a diversity loss (Lupu et al., 2021) or save multiple checkpoints (Strouse et al., 2021) but often do not report the level of diversity achieved.
Measures of diversity based on policy similarity struggle in settings where not all different actions result in “meaningfully” different outcomes 1 .
Furthermore, the number of possible trajectories is often so large that it becomes easy to maximize diversity objectives without learning qualitatively different policies – imagine a humanoid robot that wiggles a finger at any single time step in the episode rather than learning different walking styles.
This is particularly true in multi-agent settings, where the number of trajectories is exponential in the number of agents.
[Figure]
To avoid such pitfalls, another approach, followed by Charakorn et al., is to require distinct policies to be incompatible by training them to obtain a low score when paired in mixed teams.
In Section 7, we show that this results in policies that simply identify whether they are playing in self-play (SP, with themselves) or in cross play (XP), with another agent.
In the latter case, they purposely “ sabotage ” the game by selecting actions that minimize return, such as playing unplayable cards in the card game Hanabi.
Adapting to such policies in an ad-hoc pool is a non-goal, since they do not represent meaningfully different policies but rather actively poor and adversarial game play .
This is in line with previous findings that partners trained with SP rely on arbitrary conventions and symmetry breaking, making collaboration with them difficult (Hu et al., 2020; 2021b; Lupu et al., 2021).
As such, producing strong and meaningfully diverse policies in Dec-POMDPs remains an important unsolved problem.
We address this problem in turn-based settings with public actions by introducing adversarial diversity (ADVERSITY), a policy training method which, given a repulser agent , produces an “adversary” whose conventions are fundamentally incompatible with those of the repulser.
The key insight of ADVERSITY is that it prevents the adversary from identifying whether it is currently in SP or playing with the repulser agent by randomizing between the two at every time step.
In other words, even if the adversary is in an action-observation history (AOH) that is incompatible with the repulser agent, the adversary is paired with the repulser agent with a fixed probability λ , in which case the next reward is inverted.
Likewise, with probability 1 − λ the adversary is instead paired with itself.
Crucially, the choice of the current partner determines not only the sign of the reward (positive or negative) and the partners’ action, but also how the entire AOH thus far is interpreted .
Here, we build on top of the fictitious transition mechanism from off-belief learning (Hu et al., 2021b, OBL) and use the belief model of the repulser policy on the corresponding repulsive transition .
In a nutshell, if the adversary is currently paired with the repulser policy, the transition is sampled from a belief distribution that assumes the repulser policy took all actions thus far.
When the adversary is paired with itself rather than the repulser, we must avoid the feedback loop between induced beliefs and future actions, which would allow the adversary to form arbitrary conventions .
Thus, we train in a hierarchy: we start with the grounded belief , like in OBL, and at each level ℓ we compute the vanilla transitions using the belief model of the level below, ℓ − 1 .
The adversary is trained to maximize a “ difference value function ”, which estimates the forward looking discounted difference between adversarial and vanilla transitions under their corresponding beliefs.
For the first time, ADVERSITY enables us to produce a number of high performing, diverse, and symmetry-invariant policies for the challenging collaborative card game Hanabi.
R ELATED WORK
Stone et al.
and Bowling & McCracken (2005) were among the first to formulate the ad-hoc teamwork (“impromptu team play”) setting, requiring autonomous agents to collaborate with novel teammates.
Works in the literature have often taken a type-based approach, where potential partners are grouped in a number of possible types, which must be identified at test time.
Different types (or classes) of polices have notably been generated through genetic algorithms (Albrecht et al., 2015) to induce diversity.
However, these methods usually require hand-coded heuristics and are difficult to scale to complex, high dimensional environments that we consider in this paper.
Other works promote diversity in multi-agent RL (MARL) to robustify an agent by having it train against a pool of partners (Lupu et al., 2021; Strouse et al., 2021).
Lupu et al.
uses an auxiliary loss to induce diversity, while Strouse et al.
selects older checkpoints of the model; both suffer from the arbitrariness and sabotage issues which we address in our work.
Canaan et al.
(2019) generates a diverse pool of agents through MAP-Elites (Mouret & Clune, 2015), but the algorithm relies on lowdimensional rule based agents and fails to produce strong Hanabi agents (scores > 19 points).
In contrast, our method scales to high-dimensional settings and does not require manual hard-coding.
Hu et al.
(2020) introduce a setting where the goal is to maximize the cross-play (mixed team) performance between independently trained agents from the same training algorithm.
For clarity, we refer to this metric as the intra-algorithm cross-play (intra-AXP) in this paper.
Hu et al.
(2020) highlighted the importance of “reasonable” policies, in particular focusing on symmetry breaking.
Ma et al.
(2022) also investigates producing reasonable policies through the inductive biases of various model architectures.
They find that certain architectures achieve diversity in simple settings.
The work most similar to ours is that of Charakorn et al.
They learn incompatible policies by maximizing SP scores, while minimizing XP scores in a pool, and optimizing the lower bound on variations between policies.
As we show in Section 7, this approach leads to agents that obtain low XP scores by sabotaging the game rather than discovering fundamentally incompatible conventions.
B ACKGROUND 3. T URN - BASED D EC -POMDP S WITH P UBLIC A CTIONS
In this work, we assume a turn-based Dec-POMDP (Oliehoek, 2012) which can be described by a tuple ( n, S , A , P, r, O , γ ) , with number of agents n , state space S , action space A , transition function P : S ×
A × S → [0 , 1] determining the probability over next states, reward function r : S ×
A → R , observations function O ( s )
=
o , and discount factor γ ∈ [0 , 1] .
We also define the trajectory up to time t as τ t =
( o 0 , a 0 , . .
. , a t − 1 , o t ) and the action-observation history (AOH) of agent i as τ it =
( o i 0 , a 0 , . .
. , a t − 1 , o it ) .
For a given player i , a policy π i ( a | τ it ) is a function that maps an AOH to a distribution over actions.
Importantly, this setting is turn based, meaning only one agent acts at any given time step.
We also assume public actions, such that all agents observe the action selected by the acting agent—a limitation we inherit from OBL.
Multiple trajectories can produce the same AOH.
The relationship between them is therefore probabilistic and depends on the policy π that generated the trajectory up to time t .
We refer to this distribution over trajectories as the belief , B π ( τ t | τ it )
=
P ( τ t | τ it , π ) .
3. S ELF -P LAY T RAINING
Self-play (SP) is a general class of training methods for multi-agent RL.
When training the (joint) policy π in SP, we unroll the policy up to time step t , producing the partial trajectory τ t .
Each agent i then observes τ it and samples an action from π ( ·| τ it ) .
The team receives a reward r t = r ( τ t , a t ) and transitions to trajectory τ t +1 .
In two-player turn-based games, the process is repeated for agent − i to obtain r t +
and τ it +2 .
Finally, SP computes the TD target
[Equation]
which assumes that actions will be selected according to policy π at all future time-steps.
During SP training, past, present and future actions are assumed to be sampled from the policy being trained.
In fully cooperative settings, this enables agents to communicate additional information about their private observation (and therefore the trajectory τ t ) by selecting actions to sharpen their partner’s belief B π − i ( τ t | τ it ) .
Arbitrary correlations arising during training (e.g. due to random initialization) are therefore reinforced, as they provide useful information to other agents at future time-steps.
While such correlations can improve the team’s return, they are unlikely to reoccur on independent training runs, resulting in a policy that is brittle and difficult to cooperate with.
3. O FF -B ELIEF L EARNING
Off-Belief Learning
(Hu et al., 2021b, OBL) is a training algorithm designed to address the shortcomings of SP training in turn-based Dec-POMDPs with public actions.
It prevents agents from learning arbitrary and brittle conventions peculiar to the random correlations of a particular run.
The general issue in applying SP training to cooperative MARL is that it enables information feedback loops that reinforce spurious correlations between actions and meanings.
For example, consider in the cooperative card game Hanabi a policy π A , where by chance π A often takes a particular action X when its partner has a playable card, and otherwise takes action Y. Upon observing X, the partner’s belief over the card will be that it is playable.
The partner therefore plays it, resulting in a positive reward which reinforces the convention that “X means playable”.
Crucially, this occurs even if neither X nor Y reveal any extra information about the playable card.
The key insight of OBL is to break this loop by fixing a belief model B 0 , which is independent of π A .
At each step, OBL then reinterprets the AOH based on this B 0 by sampling a new fictitious trajectory τ ′ t ∼ B 0 .
The correlation between the trajectory seen by π A and its action are then restricted to what “survives” given this trajectory resampling.
Thus, if π A takes action X when a card is playable, but that same card isn’t playable under τ ′ t , then no reward is obtained if the partner plays the card.
In particular, OBL uses B 0 =
B π 0 , where π 0 is fully random.
Thus, in expectation, “X means playable” will only be reinforced if X carries verifiable information about the playability of the card.
More formally: like SP, when training policy π , OBL unrolls π to obtain the trajectory τ t , AOH τ it and action a it ∼ π ( ·| τ it ) for agent i .
However, OBL breaks the information feedback loop by entirely ignoring the reward r ( τ t , a it ) .
Instead, it assumes access to the environment simulator, as well as to a belief model B π 0 of another policy π 0 , and samples a fictitious trajectory τ ′ t ∼ B π 0 ( ·| τ it ) .
In this fictitious trajectory, OBL first applies the “real” action a it and then fictitious action a − i t +1 ′ ∼ π ( ·| τ ′ t +1 ) .
It thus receives fictitious rewards r ′ t =
r ( τ ′ t , a it ) and r ′ t +1 =
r
( τ ′ t +1 , a − i t +1 ′ ) , and fictitious future trajectory τ ′ t +2 and AOH τ it +2 ′ .
The TD target used for training the policy is therefore
[Equation]
In essence, rather than training on the real rewards seen during the transition, OBL samples a fictitious transition at every time step.
This fictitious transition reinterprets the AOH of agent i as having been produced by policy π 0 rather than π .
As a result, actions that sharpen the posterior distribution
B π ( τ t | τ it ) over trajectories given π are no longer reinforced in general.
Instead, agents must rely on actions which convey information about the trajectory in spite of this resampling.
As stated earlier, OBL assumes π 0 to be a fully random policy.
As a result, spurious correlations between the real trajectory and the observations do not propagate and agents can only rely on actions that convey verifiable information about the state.
We refer to such agents as “grounded.”
OBL can be iterated to produce a hierarchy of policies, with each level π ℓ being trained on fictitious transitions sampled from a belief model B ℓ − 1 =
B π ℓ − 1 ( τ t | τ it ) .
This process was shown to reliably produce policies with similar conventions that are devoid of symmetry breaking and are altogether considered as more “reasonable” partners for coordination.
In practice, we use neural networks ˆ B l trained with supervised learning to approximate B l to enable fast inference and sampling.
P ROBLEM S ETTING AND M OTIVATION
Many works seek to train a large pool of diverse policies from scratch.
Instead, given access to a fixed repulser policy µ , our goal is to train an adversary π —a new agent with a different play style.
We predict that many methods addressing the latter task could theoretically be deployed at larger scale to produce a population of agents.
However, this is not the primary focus of our paper.
We aim for the adversary to exhibit meaningful diversity from the repulser: to adopt conventions and strategies that differ drastically from those of the repulser, to the extent permitted by the environment. This is in contrast to policies that merely exhibit small differences in action probabilities or state occupancy but otherwise converge to the same high level strategy (Lupu et al., 2021). Furthermore, we seek adversaries that achieve high return and that are reasonable teammates. We further discuss these desiderata in section 4.2.
4. M OTIVATION
The main motivation to our work is to generate quality partners to serve as a test suite for adhoc coordination.
While it is often easy to produce a population by varying the training method or hyperparameters, there is no guarantee that such approaches will produce meaningfully diverse policies.
Additionally, it is concerning if small training variations do produce diversity, as it suggests that the training algorithms used do not reliably output policies with consistent conventions.
Indeed, such policies will fail in the context of intra-AXP, making them poor partners for ad-hoc evaluation.
Instead, with a method that is able to take a small number of reasonable partners and output highly skilled and very distinct adversaries, it becomes possible to generate a partner pool that both covers a more substantial portion of the policy set and is less likely to be populated by poor collaborators.
With access to such a method, it will become easier to test the ability of an agent to generalize to unseen and meaningfully diverse partners.
A method for producing adversaries to skilled agents has other possible applications.
Previous studies (Hu et al., 2021b; Lupu et al., 2021) have shown that biases in the training method or the environment structure can result in RL agents repeatedly converging to the same equilibrium.
Discovering new ways of solving a task is therefore an open problem.
Were it solved, it would have applications to software testing, for instance by discovering new ways of breaking a feature or simulating different user behaviours when using a program.
4. D ESIDERATA FOR A DVERSARIES
Before proposing an approach to train adversaries, we first establish how to measure success.
Whether two policies are meaningfully diverse is environment and task-specific.
For instance, two humanoid robots moving their arms differently may be considered irrelevant if the goal is to discover new gait patterns.
Similarly, the feasibility of training strong and distinct policies is environmentdependent (e.g. a task may have a unique equilibrium).
Therefore, we do not formally define the notion of “meaningful diversity”, and instead determine the desiderata of such agents, each understood to be potentially limited by the environment itself.
Skill level: Past works indicate that diversity often comes at the cost of performance.
While the environment structure may in itself be at cause, this effect can also be attributed to the difficulty of simultaneously optimizing and diversity
A good method would minimize this drop, and produce adversaries that are as strong as allowed by the setting.
Note that this is not claiming that ad-hoc partners of beginner or intermediate levels are not useful.
However, it is usually easy to obtain such partners by selecting earlier checkpoints of a trained policy.
Thus, our goal is to produce adversaries near expert level (i.e. as close to SOTA as possible).
“True” Diversity: Another core criteria to adversaries is that they adopt distinct strategies from those of their respective repulser policy.
The first way to evaluate whether the adversary π and its repulser µ adopted different strategies is by evaluating them in XP, as a mixed team.
Low XP can be indicative of distinct and incompatible conventions, but it is not sufficient.
Indeed, low XP can also be explained by having brittle policies that fail at the slightest deviation from SP.
Even worse, since π is a function of µ , it is possible that π learns to identify when it is paired with µ , and deliberately performs poorly, i.e. sabotages the game (see Section 7).
While secondary, it is also desirable that the adversary strategies differ in an interpretable way from those of the repulser policy.
For instance, an adversary to a robot sports team that is particularly aggressive may instead play much more defensively.
Reasonableness:
We require adversaries to be ”reasonable” or ”well-behaved” in an informal sense.
First, this means policies that do not sabotage, as explained above.
Secondly, we wish to avoid arbitrary conventions or symmetry breaking, since those are unlikely to be recovered even by subsequent runs of the same algorithm, making for very inflexible and uncooperative partners. While we do not have a problem agnostic means of identifying the failure modes listed above, we do have Hanabi specific metrics as explored in Section 6. Furthermore, as explained in Section 5, our method avoids these failure modes by design , building on the desirable properties of OBL.  A DVERSARIAL D IVERSITY
Given a repulser policy µ , how do we train an adversary policy π = Adv ( µ ) satisfying the criteria above?
A simple starting point is to train π to maximize SP return while minimizing return when paired with µ .
However, as we show in section 7, this approach leads to agents that identify whether they are playing in SP and sabotage if not.
Thus, it fails to produce reasonable policies.
We therefore introduce ADVERSITY, which overcomes these issues with two key insights.
[Table] [Table]
procedure ADVERSITY( Q θ , π ℓ , µ, ˆ B µ , D ) Initialize training episode E =
EmptyList and sample initial environment τ 0 ∼ P ( τ 0 ) while NotTerminal( τ ) do
Get observation τ it
= O i ( τ t ) and action a it ∼ π ℓ ( τ it ) for the active player i
if x ∼ U (0 , 1); x < λ then
B = ˆ B µ , π partner = µ , w = − 1 else
B =
ˆ B ℓ − 1 , π partner = π ℓ , w = 1 end if
Sample fictitious trajectory τ ′ t ∼ B ( ·| τ it )
Apply active player’s action on the fictitious trajectory τ ′ t
+1 =
P ( ·| τ ′ t , a it ) and collect reward r ′ t
Partner observes and picks fictitious action a − i t +1 ′
=
π partner ( O − i ( τ ′ t +1 ))
Apply partner’s action on the fictitious trajectory τ ′ t +
= P ( ·| τ ′ t +1 , a ′− i t +1 ) and collect reward r ′ t
+1 .
Compute target Q t = wr ′
t + wγr ′
t
Firstly, we prevent sabotages by re-sampling the partner for every AOH, such that the adversary cannot condition on the partner’s identity to choose its action.
At every time step, the adversary’s partner is the repulser with probability λ , and itself otherwise.
However, doing so directly perturbs the trajectory distribution and impacts learning. For that reason, we leave the trajectory being unrolled intact and perform this partner switching within the fictitious transitions of OBL. Thus, in ”repulsive” transitions (when π is paired with µ ), we not only sample the next action from the repulser, but also use the repulser’s belief model to resample the current trajectory. This reduces the likelihood of µ being off-distribution and reinforces actions that are incompatible with µ .
Conceptually, on those transitions we pretend that all prior actions in the episode were taken by the repulser.
Secondly, we prevent arbitrary conventions by training a hierarchy of adversary policies, π ℓ .
On vanilla transitions (when the adversary is paired with itself), each π ℓ uses a fixed belief model from the adversary policy at the level below, π ℓ − 1 .
Like OBL, the lowest level belief is the grounded belief, i.e. the unique belief corresponding to a uniformly random policy.
All belief models are trained using the supervised learning procedure described in Hu et al.
(2021a).
Technically, we proceed as follows: at a given level ℓ , ADVERSITY follows a similar training pattern to OBL by unrolling policy π ℓ on the real trajectory and computing a fictitious transition given τ ′ t ∼ ˆ B ℓ − 1 ( τ − i t +1 ′ ) and a − i t +1 ′ ∼ π ( ·| τ − i t +1 ′ ) .
However, with probability λ , the algorithm utilizes a “repulsive” fictitious transition rather than the vanilla OBL one.
In that case, we instead sample τ ′ t ∼ ˆ B µ ( τ − i t +1 ′ ) and the partner’s action from µ .
We then flip the fictitious rewards to obtain the repulsive target
[Equation]
This procedure is summarized in Algorithm 1 and each transition is illustrated in Figure 1. [Table]
Accounting for both transitions, at every time step the policy learns difference Q-values that estimate the expected future discounted reward difference between vanilla and repulsive transitions:
[Equation]
Breaking it down, the first line corresponds to vanilla OBL.
It implies reinterpreting the past as having been produced by a given policy π ℓ − 1 and acting according to π ℓ ever after, reinforcing only actions that lead to high expected return when interpreting the partner’s actions according to π ℓ − 1 .
This prevents feedback loops in SP where the agent can learn spurious beliefs from noise about which past actions correspond to what unobserved trajectories, and then use those beliefs to signal information and reinforce them into arbitrary and brittle conventions.
The second line represents repulsive transitions.
Resampling the current trajectory assuming the
AOH was produced by µ results in a it being reinforced if it is incompatible with µ ’s conventions and leads to low immediate rewards.
At the next step, we assume our partner is µ , thus simulating XP between the repulser and the adversary.
Here again, negating r ′ t +1 means we reinforce actions that are misinterpreted by µ .
Finally, in both vanilla and repulsive transitions, we maximize expected future discounted difference reward , denoted by V diff ℓ ( τ it +2 ′ ) .
Conceptually, these difference value functions assume that at every point in the future the repulser will intervene for one time step with probability λ , at which point rewards are inverted and the entire past is re-interpreted according to their belief model ˆ B π µ .
Overall, this procedure pushes π ℓ to select actions that achieve high return under ˆ B π ℓ − 1 , and that are simultaneously incompatible with µ ’s conventions.
This section describes training a single policy π ℓ on top of ˆ B ℓ − 1 and µ .
Because the initial belief ˆ
B 0 is restricted to only rely on grounded information, the skill level of π 1 is limited.
Therefore, to improve skill we follow the procedure from Hu et al.
(2021b)
and iteratively learn higher levels π ℓ .
At each level, we decrease λ , up to λ = 0 , at which point our training reverts to vanilla OBL.
Nonetheless, since each level uses the belief level of the previous level, in our settings the final play style is incompatible to µ , as we show in Section 7.
This shows that we truly obtain novel equilibria.
E XPERIMENTAL S ETUP
Here we describe the evaluation setting and baseline.
Training details are included in the Appendix.
6. H ANABI
We implement and test our method in Hanabi, a large scale cooperative card game proposed as a challenging benchmark for Dec-POMDP research (Bard et al., 2020).
Hanabi is played with 8 hint tokens, 3 life tokens and a deck of 50 cards, each having a rank between 1 and 5 and one of five colors.
It is a game for 2-5 players, but we restrict ourselves to the 2-player version.
[Table]
The goal is to form stacks of cards in rank order for each of the five colors.
The final score, between 0 and 25, is given by the number of cards successfully stacked by the end of the game.
At any given time, player’s have five cards in their hands, and can only see their partner’s cards.
The players must therefore communicate effectively with their partners so that they can act in an informed manner.
One their turn, a player has up to 20 different actions.
They can either a) discard one of their cards, b) attempt to play one of their cards, c) hint at all the cards of a chosen color in the partner’s hand, or d) hint at all the cards of a given rank in the partner’s hand.
Discarding replenishes one hint token.
Playing a card results in it being placed on top of the pile of its color if it’s the next logical card in that pile.
Otherwise, the card is lost and the team loses a life.
Hinting provides limited information to the partner and consumes a hint token.
Hinting is not allowed if there are no hint tokens left, and discarding is not allowed if all 8 tokens are available.
If the team loses all 3 lives, the game ends prematurely, with a score of 0.
Finally, the game also ends when there are no cards left in the deck.
6. S ELF -P LAY W ORST R ESPONSE
As a baseline, we implement a “Self-Play Worst Response” (SPWR) – a PPO agent using the same neural network architecture and hyperparameters as above but trained on SP data with probability 1 − λ and in XP with the repulser with probability λ .
Our SWPR experiments set λ = 0 .
An entire game is either SP or XP, with no switching within a single game.
When in XP, the rewards received are inverted.
This is a simpler version of LIPO from Charakorn et al.
R ESULTS
We evaluate the skill level, diversity, and reasonableness of our method against the SPWR baseline.
For both our method and our baseline, we train 3 adversary seeds for each of 4 repulser policies.
In addition to vanilla OBL level 5, the repulsers are 3 of the baseline policies inspired by Hu et al.
(2021b); namely Rank Bot, which is an Other-Play (Hu et al., 2020) (and therefore color equivariant) policy favouring rank hints, Color Bot, which is a reward-shaped policy favouring color, and Clone Bot, which is a supervised learning bot trained on human data.
We first see in Table 1 that both the SPWR and ADVERSITY agents achieve high SP, corresponding to high skill in Hanabi, and very low XP scores when paired with their respective repulser, showing that both methods produce policies that are incompatible with their repulser.
However, ADVERSITY shows a clear advantage over the SPWR in terms of intra-AXP scores, computed between independent adversary seeds.
SPWR produces different policies every run, resulting in a very low Intra-AXP score.
In contrast, ADVERSITY agents tend to be quite similar, as shown by the lower
SP-XP gap.
Adversaries to Color Bot and OBL are exceptions and have low Intra-AXP scores.
To evaluate whether the adversaries exhibit meaningful diversity from their repulser, we first look at conditional action matrices , presented in appendix A.4.
These display the conditional probability of a player’s action a it +1 given the previous action, a − i t .
We find that both ADVERSITY and the SPWR exhibit meaningful diversity from their repulsers and between adversaries to different repulsers.
For example, when Rank Bot is the repulser policy, ADVERSITY consistently produces adversaries that use color to indicate playable cards.
Similarly, while OBL tends to respond to discard actions by also discarding a card, adversaries to OBL learn instead to discard their last card to hint play.
r a n k _ b o t c
o l o r _ b
o t c l o n e _ b o t o b l r b _ W R c
b _ W R
c
l o n e _ W R
o
b
l _ W R
[Table]
bot color_bot clone_bot obl rb_WR cb_WR clone_WR
obl_WR
[Table]
r a n k _ b o t c
o l o r _ b
o t c l o n e _ b o t o b l r b _ a d v c b _ a d v c l o n e _ a d v o b l _ a d v
[Table]
cb_ [Table] obl
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Figure]
In terms of method consistency, the action matrices for different ADVERSITY seeds tend to be similar, reinforcing the idea that it is reproducible.
SPWR seeds, on the other hand, differ wildly, explaining the low inter-AXP score mentioned previously.
Finally, Figure 2b shows the XP matrices between bots including the repulsers and one of each adversary seed.
Notice that SPWR adversaries have near-zero XP scores with virtually every partner; a red flag supporting the hypothesis that they learned to identify when not in SP and purposely throw the game.
Meanwhile, ADVERSITY agents exhibit a graceful degradation of ad-hoc performance depending on the similarity to the partner’s policy, indicating much more reasonable policies.
Sabotaging: We verify that unlike ADVERSITY, SPWR adversaries exhibit sabotaging behavior in Hanabi.
We do this by measuring sabotages, the number of knowingly unplayable cards (based on revealed information) played by the agent.
We measure the average number of sabotages per game when SPWR and ADVERSITY are paired with their respective repulser agents and report results in Table 2.
ADVERSITY consistently has a low number of sabotages ( < 0 .
1 ) per game, whereas SPWR has at least 1 sabotage per game and in many cases > 2 .
The sabotages are lower for SPWR(Color Bot) simply because a color hint does not immediately reveal whether a card is definitely unplayable.
The SPWR(Color Bot) agent simply plays unhinted cards blindly, which is not necessarily a sabotage by our strict definition, but a poor move nonetheless.
Moreover, SPWR agents sabotage all non-SP games, not just the ones with the repulser they were trained with.
This indicates that the poor XP performance of SPWR comes not from playing a reasonable rewardmaximizing strategy that happens to be meaningfully different and incompatible with other agents, but from “deliberately” playing bad actions upon identifying that its current partner is not itself.
We also verified that the < 0 .
1 mean incidence of sabotaging for ADVERSITY is in line with vanilla
OBL evaluated in SP, i.e. corresponds to a standard rate of mistakes or risky bets.
On a different metric, SPWR is responsible for a dominant amount of 2.67 ± 0.05 out of 3 life losses per game while, ADVERSITY is only responsible for 1.42 ± 0.05—roughly half of the total mistakes.
This also indicates that SPWR tries hard to terminate games deliberately, while ADVERSITY policies fail due to meaningful incompatibility, without any party trying to explicitly sabotage.
C ONCLUSION AND F UTURE W ORK
In this paper, we introduce ADVERSITY, a method for producing highly skilled and reasonable policies for a fully cooperative task that play according to meaningfully diverse conventions.
While our results show that both ADVERSITY and our baseline produce agents that exhibit high skill and meaningful diversity from their repulser, only ADVERSITY agents are also reproducible on independent runs and reasonable, as indicated by the graceful degradation of their performance with different ad-hoc partners.
The main limitation of our method is the high computational cost, making it difficult to scale the method to a large number of adversaries.
Were this issue solved, ADVERSITY could theoretically be used to produce a large pool of diverse agents by iteratively computing adversaries to past models.
A A PPENDIX A. T RAINING D ETAILS
Our implementation is based on the open sourced OBL code with two main modifications.
We first replace the recurrent Q-learning backbone with PPO as it runs faster and requires significantly less memory.
Then, we implement a synchronous method that trains all OBL levels simultaneously.
Otherwise, we use a distributed training set up similar to (Hu et al., 2021b), which we detail in the appendix.
All bots are trained for 3000 epochs, each epoch consists of 1000 gradient steps.
We select the checkpoint with the highest SP score.
The original OBL trains multiple levels of policies sequentially, using the output policy of the previous level as the input policy of the new level.
In ADVERSITY, we train all levels simultaneously for faster wall-clock time.
These policies are denoted as π 0 , π 1 . . .
π L and their corresponding belief models are denoted as ˆ B 0 , . . .
, ˆ B L .
To warm up the belief model and avoid having too many invalid samples, we first train a belief model ˆ B 0 on the uniform random base policy π 0 and initialize all ˆ
B l = ˆ B 0 .
Then, L policy training tasks and L belief training tasks start at the same time.
The belief task of ˆ B l gets a latest copy of π l every 50 gradient steps and the policy task of π l gets a latest copy of ˆ
B l − 1 every 50 gradient steps.
The details of each individual belief follows the exact configurations of the original OBL paper and each policy task uses the PPO-OBL method described above.
For each adversary, we train a hierarchy of 7 levels, setting λ = 0 .
25 for l = 1 and decreasing by 0 .
every level (min. 0).
Levels l ≤ 4 are trained simultaneously, followed by levels l ≥ 5 , also trained simultaneously and with beliefs initialized at ˆ B l
= ˆ B 4 .
This split was forced by limitations on the concurrent compute available to the authors, but we anticipate no change in performance if all levels were trained simultaneously.
The ADVERSITY numbers reported in Section 7 all refer to the highest level of the hierarchy.
A. P OLICY T RAINING D ETAILS
We use a large scale distributed training framework for policy training.
To train a single policy, we run 6400 games in parallel, each adding to a centralized replay buffer.
We achieve this by runningthreads in parallel, with 80 games running per thread.
All models are on GPUs and we dynamically batch all model calls in order to increase inference speed.
This schema also allows games on the same thread to forward environment calls while certain games wait for GPU calls.
As done in (Wang et al., 2016), when an environment terminates, each game grabs all necessary objects: observations, actions, and targets, pads everything to a length of 80 and adds it to a centralized replay buffer.
For every training step we apply the PPO update rule, but instead of using the real reward and advantage, we use the fictitious values.
Every m = 10 training steps, we update the environment actors with the weights for the updated policy.
As done in Cui et al.
(2021) synchronously train our hierarchy of beliefs and policies, querying for and updating all dependencies every p = 50 training steps.
We utilize the same policy architecture as Hu et al.
(2021b).
We utilize their public-private LSTM architecture.
The public observation is encoded by a one-layer feedforwards neural network followed by a LSTM.
The private observation is encoded by a three-layer neural network.
We combine these encodings via element wise multiplication.
For all OBL experiments we compute the target with r = 1 fictitious steps.
We also sample the belief model s = 10 times and use the first sampled trajectory that doesn’t violate card constraints to compute the fictitious targets.
We then use a simulator to produce transitions from the valid trajectory.
Like Hu et al.
, we discard the fictitious transition whenever the belief fails to produce a valid sample, which in practice happens on less than 1% of transitions.
Implementation
The policy is represented by a public-LSTM network π θ with a value head and a policy head.
A large number of parallel workers generate data by sampling from a slightly outdated policy π θ ′ and write that data into a replay buffer D .
One datapoint in D is an entire trajectory τ j .
Although PPO normally does not need a replay buffer, we still use one here to fully decouple inference and training for maximum speed.
Its size is set to a small value of 1024 to minimize the instability caused by stale data.
π θ is trained with the Adam optimizer (Kingma & Ba, 2014) on minibatches of data uniformly sampled from the replay buffer.
The value loss is E τ i ∼D (cid:80) t [ r t + γV θ ( τ it +1 ) − V
θ
( τ it )] 2 .
The policy loss is E τ i ∼D (cid:80) t min[ r t ( θ ) ˙
A t , clip ( r t ( θ ) , 1 ± ϵ ) ˙
A t ] where r t ( θ ) =
π θ ( a it | τ it )
π θ ′ ( a it | τ it ) , ˙ A t =
StopGradient [ r t + γv θ ( τ it +1 ) − V θ ( τ it )] .
We perform one gradient step per minibatch.
We use 1-step bootstrapped value target instead of (cid:80) t r t because it converges significantly faster and it fits well in the OBL fictitious target computation.
π θ ′ is synced with π θ every 10 gradient updates.
A. B ELIEF T RAINING D ETAILS
We utilize the same distributed training schema from policy training for belief training.
This has also been done by (Hu et al., 2021b).
As done in policy training, we query and update dependencies every p = 50 training steps.
For belief training we store the true hand of the player along with the observation to train the belief.
For training, we train an autoregressive belief model that predicts cards oldest to newest via supervised learning.
More precisely, the belief model is trained to minimize the loss
[Equation]
[Equation]
) where h k is the k th card in the player’s hand and n is the hand size (usually 5).
A. A DDITIONAL R ESULTS
[Equation]
[Figure] [Table] D1D2D3D4D5P1P2P3P4P5C1C2C3C4C5R1R2R3R4R D1D2D3D4D5P1P2P3P4P5C1C2C3C4C5R1R2R3R4R
[Table]
D1D2D3D4D5P1P2P3P4P5C1C2C3C4C5R1R2R3R4R
D1D2D3D4D5P1P2P3P4P5C1C2C3C4C5R1R2R3R4R
[Table]
D1D2D3D4D5P1P2P3P4P5C1C2C3C4C5R1R2R3R4R
[Table]
D1D2D3D4D5P1P2P3P4P5C1C2C3C4C5R1R2R3R4R
D1D2D3D4D5P1P2P3P4P5C1C2C3C4C5R1R2R3R4R [Table] [Figure] [Table]
D1D2D3D4D5P1P2P3P4P5C1C2C3C4C5R1R2R3R4R
D1D2D3D4D5P1P2P3P4P5C1C2C3C4C5R1R2R3R4R
[Table]
D1D2D3D4D5P1P2P3P4P5C1C2C3C4C5R1R2R3R4R
D1D2D3D4D5P1P2P3P4P5C1C2C3C4C5R1R2R3R4R
[Table]
D1D2D3D4D5P1P2P3P4P5C1C2C3C4C5R1R2R3R4R
[Table]
D1D2D3D4D5P1P2P3P4P5C1C2C3C4C5R1R2R3R4R
D1D2D3D4D5P1P2P3P4P5C1C2C3C4C5R1R2R3R4R
[Table]
[Figure]
r a n k _ b o t c
o l o r _ b
o t c l o n e _ b o t o b l
S P W R ( r b ) _ S EE
D a
S P W R ( r b ) _ S EE
D b
S P W R ( r b ) _ S EE
D c
S P W R ( c b ) _ S EE
D a
S P W R ( c b ) _ S EE
D b
S P W R ( c b ) _ S EE
D c
S P W R ( c l o n e ) _ S
EE D a
S P W R ( c l o n e ) _ S
EE D b
S P W R ( c l o n e ) _ S
EE D c
S P W R ( o b l ) _ S EE
D a
S P W R ( o b l ) _ S EE
D b
S P W R ( o b l ) _ S EE
D c
[Figure] [Figure] [Figure] [Figure]
[Figure]
[Figure]
[Figure]
A D V ( r b ) _ S E E D
a
A D V ( r b ) _ S E E D b A
[Figure] V ( r b ) _ S E E D c
A D V ( c b ) _ S E E D
a
A D V ( c b ) _ S E E D b A D V ( c b ) _ S E E D c
A D V ( c l o n e ) _ S E
E D a
A D V ( c l o n e ) _ S E
E D b
A D V ( c l o n e ) _ S E
E D c
A D V ( o b l ) _ S E E D
a
A D V ( o b l ) _ S E E D b A D V ( o b l ) _ S E E D
[Figure]
[Table]
[Figure] [Figure]