A DVERSARIAL D IVERSITY IN H ANABI
Anonymous authors
Paper under double-blind review
A BSTRACT
Many Dec-POMDPs admit a qualitatively diverse set of “reasonable” joint policies, where reasonableness is indicated by symmetry equivariance, nonsabotaging behaviour and the graceful degradation of performance when paired with ad-hoc partners.
Some of the work in diversity literature is concerned with generating these policies.
Unfortunately, existing methods fail to produce teams of agents that are simultaneously diverse, high performing, and reasonable.
In this work, we propose a novel approach, adversarial diversity (ADVERSITY), which is designed for turn-based Dec-POMDPs with public actions.
ADVERSITY relies on off-belief learning to encourage reasonableness and skill, and on “repulsive” fictitious transitions to encourage diversity.
We use this approach to generate new agents with distinct but reasonable play styles for the card game Hanabi and opensource our agents so that they may be used as starting points for future research on (ad-hoc) coordination.
I NTRODUCTION
A key objective of cooperative multi-agent reinforcement learning (MARL) is to produce agents capable of coordinating with novel partners, including other artificial agents and ultimately humans.
In order to make progress on this objective, a number of works have focused on the general challenge of ad-hoc team play , which is to create autonomous agents able to “ efficiently and robustly collaborate with previously unknown teammates on tasks to which they are all individually capable of contributing as team members ” (Stone et al., 2010).
To evaluate such agents, many works on ad-hoc coordination rely on evaluation setups similar to the one proposed by Stone et al.
(2010), pairing the agents at test time with partners sampled from a pre-determined pool.
The value of such evaluations depends on the size and quality of the pool of partners.
A pool that is too small or too homogeneous may not be representative of all possible play-styles, and provide an inaccurate evaluation of the coordination capabilities of an agent. For this reason, previous works in coordination have relied on various approaches to generate a diverse pool of partners.
A first approach is to handcraft policies, either directly or by shaping the reward at train-time (Albrecht, 2015; Barrett et al., 2017; Zand et al., 2022), but it requires domain knowledge and scales poorly.
Another is to train a population with varying hyperparameters or by deploying multiple RL algorithms on the same task (Nekoei et al., 2021; Zand et al., 2022; Albrecht, 2015). The diversity achieved this way is unclear, since it is a byproduct of the variability of the algorithms used rather than being actively optimized for. Yet other works augment training with a diversity loss (Lupu et al., 2021) or save multiple checkpoints (Strouse et al., 2021) but often do not report the level of diversity achieved.
Measures of diversity based on policy similarity struggle in settings where not all different actions result in “meaningfully” different outcomes 1 .
Furthermore, the number of possible trajectories is often so large that it becomes easy to maximize diversity objectives without learning qualitatively different policies – imagine a humanoid robot that wiggles a finger at any single time step in the episode rather than learning different walking styles. This is particularly true in multi-agent settings, where the number of trajectories is exponential in the number of agents.
To avoid such pitfalls, another approach is to require distinct policies to be incompatible by training them to obtain a low score when paired in mixed teams.
However, as we show in Section 7, this results in policies that simply identify whether they are playing in self play (SP, with themselves) or in cross play (XP), with another agent.
In the latter case, they purposely “ sabotage ” the game by [Figure] selecting actions that minimize return, such as playing illegal cards in the card game Hanabi.
Adapting to such policies in an ad-hoc pool is clearly a non-goal, since they do not represent meaningfully different policies but rather actively poor and adversarial game play .
This is in line with previous findings that partners trained with SP rely on arbitrary conventions and symmetry breaking, making collaboration with them difficult (Hu et al., 2020; Lupu et al., 2021;
Hu et al., 2021b).
As such, producing strong and meaningfully diverse policies in Dec-POMDPs remains an important unsolved problem.
We address this problem in turn-based settings with public actions by introducing adversarial diversity (ADVERSITY), a policy training method which, given a repulser agent , produces an “adversary” whose conventions are fundamentally incompatible with those of the repulser.
The key insight of ADVERSITY is that it prevents the adversary from identifying whether it is currently in SP or playing with the repulser agent by randomizing between the two at every time step.
In other words, even if the adversary is in an action-observation history (AOH) that is incompatible with the repulser agent, the adversary is paired with the repulser agent with a fixed probability λ , in which case the next reward is inverted.
Likewise, with probability 1 − λ the adversary is instead paired with itself.
Crucially, the choice of the current partner determines not only the sign of the reward (positive or negative) and the partners’ action, but also how the entire AOH thus far is interpreted .
Here, we build on top of the fictitious transition mechanism from off-belief learning (Hu et al., 2021b, OBL) and use the belief model of the repulser policy on the corresponding repulsive transition .
In a nutshell, if the adversary is currently paired with the repulser policy, the next trajectory is sampled from a belief distribution that assumes the repulser policy took all actions thus far.
As for when the adversary is paired with itself rather than the repulser policy, in principle, we could continuously co-train a belief model for the adversary and use this up-to-date belief model to sample the fictitious transition up to the current time step.
However, this would allow the adversary to form arbitrary conventions and break symmetries—just like SP agents—due to the feedback loop between induced beliefs and future actions.
Instead, to regularize learning, we train in a hierarchy: we start out with the grounded belief , like in OBL and at each level ℓ , use the belief model of the level below, ℓ − 1 , for the corresponding transitions.
The adversary is trained to maximize a “ difference value function ”, which estimates the forward looking discounted difference between adversarial and SP transitions under their corresponding belief functions.
For the first time, ADVERSITY enables us to produce a number of high performing, diverse, and symmetry-invariant policies for the challenging collaborative card game Hanabi.
R ELATED WORK
Stone et al.
and Bowling & McCracken (2005) were among the first to formulate the ad-hoc teamwork (“impromptu team play”) setting, requiring autonomous agents to collaborate with novel teammates.
Works in the literature have often taken a type-based approach, where potential partners are grouped in a number of possible types, which must be identified at test time.
Different types (or classes) of polices have notably been generated through genetic algorithms (Albrecht et al., 2015) to induce diversity.
However, these methods usually both require hand-coded heuristics and are difficult to scale to complex, high dimensional environments that we consider in this paper.
Other works have promoted diversity in multi-agent RL (MARL) to robustify an agent by having it train against a pool of partners (Lupu et al., 2021; Strouse et al., 2021).
Lupu et al.
(2021) relied on an auxiliary loss to induce diversity, while Strouse et al.
(2021) simply selects older checkpoints of the model.
These methods suffer from the arbitrariness and sabotage issues which we address in our work.
Canaan et al.
(2019) generates a diverse pool of agents through MAP-Elites (Mouret & Clune, 2015); however, the algorithm relies on low-dimensional rule based agents and fails to produce strong Hanabi agents (scores > 19 points).
In contrast, our method is based on learning approaches that scale to high-dimensional settings and does not require manual hard-coding.
Hu et al.
(2020) introduce a setting where the goal is to maximize the cross-play (mixed team) performance between independently trained agents from the same training algorithm.
For clarity, we refer to this metric as the intra-algorithm cross-play (intra-AXP) in this paper.
Hu et al.
(2020) highlighted the importance of “reasonable” policies, in particular focusing on symmetry breaking.
Ma et al.
(2022) also investigates producing reasonable policies through the inductive biases of various model architectures.
They find that certain architectures achieve diversity in simple settings.
The work most similar to ours is that of Charakorn et al.
They learn incompatible policies by maximizing self-play scores, while minimizing XP scores in a pool, and optimizing the lower bound on variations between policies.
As we show in Section 7, this type of simple adversarial diversity leads to agents that obtain low XP scores by sabotaging the game rather than discovering fundamentally incompatible, reasonable conventions.
B ACKGROUND 3. T URN - BASED D EC -POMDP S WITH P UBLIC A CTIONS
In this work, we assume a turn-based Dec-POMDP (Oliehoek, 2012) which can be described by a tuple ( n, S , A , P, r, O , γ ) . We restrict ourselves to Dec-POMDPs with public actions—a limitation we inherit from OBL.
Here, n is the number of agents, S is the state space,
A is the action space, P : S × A × S → [0 , 1] is the transition function determining the probability over next states, r : S ×
A → R is the reward function, O is the observations function O ( s )
=
o , and γ ∈ [0 , 1] is a discount factor.
We also define the trajectory up to time t as τ t =
( o 0 , a 0 , . .
. , a t − 1 , o t ) and the action-observation history (AOH) of agent i as τ it =
( o i 0 , a 0 , . .
. , a t − 1 , o it ) .
For a given player i , a policy π i ( a | τ it ) is a function that maps an AOH to a distribution over actions.
Note that, importantly, this setting is turn based, meaning only one agent acts at any given time step.
Furthermore, actions are public, meaning all agents are made aware of the action selected by the acting agent.
Multiple trajectories can produce the same AOH.
The relationship between them is therefore probabilistic and depends on the policy π that generated the trajectory up to time t .
We refer to this distribution over trajectories as the belief , B π ( τ t | τ it )
=
P ( τ t | τ it , π ) .
3. S ELF -P LAY T RAINING
Self play (SP) is a general class of training methods for multi-agent RL.
When training the (joint) policy π in SP, we unroll the policy up to time step t , producing the partial trajectory τ t .
Each agent i then observes τ it and samples an action from π ( ·| τ it ) .
The team receives a reward r t = r ( τ t , a t ) and transitions to trajectory τ t +1 .
In two-player turn-based games, the process is repeated for agent − i to obtain r t +
and τ it +2 .
Finally, SP computes the TD target
[Equation]
1) assuming that actions will be selected according to policy π at all future time-steps.
In summary, during SP training, past, present and future actions are assumed to be sampled from the policy being trained.
In fully cooperative settings, this enables agents to communicate additional information about their private observation (and therefore the trajectory τ t ) by selecting actions such as to sharpen the belief B π ( τ t | τ it ) for their partners.
This means that arbitrary correlations arising during training (e.g. due to random initialization) are potentially reinforced, as they provide useful information to the other agents at future time-steps.
While such correlations can improve the team’s return, they are unlikely to reoccur on independent training runs.
Thus, the resulting policy may be difficult to reproduce.
3. O FF -B ELIEF L EARNING
Off-Belief Learning
(Hu et al., 2021b, OBL) is a training algorithm designed to address the shortcomings of SP training in turn-based Dec-POMDPs with public actions.
It prevents agents from learning arbitrary and brittle conventions peculiar to the random correlations of a particular run.
The general issue in applying SP training to cooperative MARL is that it enables information feedback loops that reinforce spurious correlations between actions and meanings.
For example, consider in the cooperative card game Hanabi a policy π A , where by chance π A often takes a particular action
X when its partner has a playable card, and otherwise takes action Y. Upon observing X, the partner’s belief over the card will be that it is playable.
It therefore plays it, resulting in a positive reward which reinforces the convention that “X means playable”.
Crucially, this occurs even if neither X nor Y reveal any extra information about the playable card.
The key insight of OBL is to break this loop by fixing a belief model B 0 , which is independent of π A .
At each step, OBL then reinterprets the AOH based on this B 0 by sampling a new fictitious trajectory τ ′ t ∼ B 0 .
The correlation between the trajectory seen by π A and its action are then restricted to what “survives” given this trajectory resampling.
Thus, if π A takes action X when a card is playable, but that same card isn’t playable under τ ′ t , then no reward is obtained if the partner plays the card.
In particular, OBL uses B 0 =
B π 0 , where π 0 is fully random.
Thus, in expectation, “X means playable” will only be reinforced if X carries verifiable information about the playability of the card.
More formally: like SP, when training policy π , OBL unrolls π to obtain the trajectory τ t , trajectory τ it and action a it ∼ π ( ·| τ it ) for agent i .
However, OBL breaks the information feedback loop by entirely ignoring the reward r ( τ t , a it ) .
Instead, it assumes access to the environment simulator, as well as to a belief model B π 0 of another policy π 0 , and samples a fictitious trajectory τ ′ t ∼ B π 0 ( ·| τ it ) .
In this fictitious trajectory, OBL first applies the “real” action a it and then fictitious action a − i t +1 ′ ∼ π ( ·| τ ′ t +1 ) .
It thus receives fictitious rewards r ′ t =
r ( τ ′ t , a it ) and r ′ t +1 =
r
( τ ′ t +1 , a − i t +1 ′ ) , and fictitious future trajectory τ ′ t +2 and τ it +2 ′ .
The TD target used for training the policy is therefore
[Equation]
In essence, rather than training on the real rewards seen during the transition, OBL applies a fictitious transition at every time step.
This fictitious transition reinterprets the AOH of agent i as having been produced by policy π 0 rather than π .
As a result, actions that sharpen the posterior distribution
B π ( τ t | τ it ) over trajectories given π are no longer reinforced in general.
Instead, agents must rely on actions which convey information about the trajectory in spite of this resampling.
As stated earlier OBL assumes π 0 to be a fully random policy.
As a result, spurious correlations between the real trajectory and the observations do not propagate and agents can only rely on actions that convey verifiable information about the state, independently of which policy is assumed to have produced them.
We refer to such policies as “grounded”.
We can then iterate OBL to produce a hierarchy of policies, with each level π ℓ being trained on fictitious transitions sampled from a belief model B ℓ − 1 =
B π ℓ − 1 ( τ t | τ it ) .
This process was shown to reliably produce policies with similar conventions that are devoid of symmetry breaking and are altogether considered as more “reasonable” partners for coordination.
In practice, approximate neural network belief models ˆ B l are trained with supervised learning for fast inference and sampling.
P ROBLEM S ETTING AND M OTIVATION
Many works seek to create a large pool of diverse policies from scratch, often by training them simultaneously in a population.
Instead, given access to a repulser policy µ , our goal is to train an adversary π – a new agent with a different play style.
That is, we wish the conventions and strategies adopted by the adversary to differ drastically from those of the repulser.
Crucially, µ is assumed to be fixed, and it may be produced e.g. by reinforcement learning, behaviour cloning, or hand crafting.
It simply reflects one known way of solving the task from which π must deviate.
We predict that many methods addressing the latter task could theoretically be deployed at larger scale to produce a population of agents.
However, this is not the primary focus of our paper.
Instead, we aim for to train a small number of meaningfully diverse agents that maintain a high skill level and “reasonable” conventions, in a way which we detail here.
4. M OTIVATION
The main motivation to our work is in generating quality partners to serve as a test suite for adhoc coordination.
While it is often easy to produce a population by varying the training method or hyperparameters, there is no guarantee that such approaches will produce meaningfully diverse policies.
Additionally, if small training variations do produce diversity, then there is reason for concern, as it suggests that the training algorithms used do not reliably output policies with consistent conventions.
Indeed, by definitions such policies fail in the context of intra-AXP, making them poor partners with which to evaluate an ad-hoc agent.
Instead, with a method that is able to take a small number of reasonable partners and output highly skilled and very distinct adversaries, it becomes possible to generate a partner pool that at the same time covers a more substantial portion of the policy set and is less likely to be populated by poor collaborators.
With access to such a method, it will become easier to test the ability of an agent to adapt to meaningfully diverse agents.
For instance, one could study whether an agent trained with a set of partners will be able to generalize to the adversaries of those partners.
A method for producing adversaries to skilled agents has other possible applications.
Previous studies (Hu et al., 2021b; Lupu et al., 2021) have shown that biases in the training method or the environment structure can result in RL agents repeatedly converging to the same equilibrium.
Discovering new ways of solving a task is therefore an open problem.
Were it solved, it would have applications to software testing, for instance by discovering new ways of breaking a feature or simulating different user behaviours when using a program.
4. D ESIDERATA FOR A DVERSARIES
Before proposing an approach to train adversaries, we first establish how to measure success.
Whether two policies are meaningfully diverse or not is environment and task-specific.
For instance, two humanoid robots moving their arms differently may be considered irrelevant if the goal is to discover new gait patterns.
Similarly, the feasibility of training strong and distinct policies is environment-dependent (e.g. a task may have a unique equilibrium).
Therefore, we do not formally define the notion of “meaningful diversity”, and instead determine the desiderata of such agents, each understood to be potentially limited by the environment itself.
Skill level: Past works indicate that diversity often comes at the cost of performance, or skill level.
While the environment structure may in itself be at cause, in many cases the performance drop occurs due to the difficulty of simultaneously optimizing for return and diversity.
A good method would minimize this drop, and thus produce adversaries that are as strong as allowed by the setting.
Note that this is not claiming that ad-hoc partners of beginner or intermediate levels are not useful.
However, it is usually easy to obtain such partners by selecting earlier checkpoints of a trained policy.
Thus, our goal is to produce adversaries near expert level (i.e. as close to SOTA as possible).
“True” Diversity: Another core criteria to adversaries is that they adopt distinct strategies from those of their respective repulser policy.
The first way to evaluate whether the adversary π and its repulser µ adopted different strategies is by evaluating them in XP, as a mixed team.
Low XP can be indicative of distinct and incompatible conventions, but it is not sufficient.
Indeed, low XP can also be explained by having brittle policies that fail at the slightest deviation from SP.
Even worse, since π is a function of µ , it is possible that π learns to identify when it is paired with µ , and deliberately performs poorly, i.e. sabotages the game (see Section 7).
While we do not have a problem agnostic means of identifying the failure modes listed above, we do have Hanabi specific metrics as explored in Section 6.
Furthermore, as explained in Section 5, our method avoid these failure modes by design .
While secondary, it is also desirable that the adversary strategies differ in an interpretable way from those of the repulser policy.
For instance, an adversary to a robot sports team that is particularly aggressive may instead play much more defensively.
A DVERSARIAL D IVERSITY
Given a repulser policy µ , how do we train an adversary policy π = Adv ( µ ) satisfying the criteria above?
A simple starting point is to train π to maximize SP return while minimizing return when paired with µ .
However, as we show in section 7, this approach leads agents that identify whether they are in SP mode and sabotage if not.
Thus, it fails to produce reasonable policies.
We introduce
ADVERSITY, our principled approach which overcomes these issues with two key insights.
[Table] [Table]
procedure ADVERSITY(
[Table]
) Initialize training episode E = EmptyList and sample initial environment [Table] ∼ P ( [Table]
0 ) while NotTerminal( τ ) do
Get observation τ i t
= O i ( τ t ) and action a i t ∼ π ℓ ( τ i t ) for the active player i
if x ∼ U (0 , 1); x < λ then
B = ˆ B µ , π partner = µ , w = − 1 else
B =
ˆ B ℓ − 1 , π partner = π ℓ , w = 1 end if
Sample fictitious trajectory τ ′ t ∼ B ( ·| τ it )
Apply active player’s action on the fictitious trajectory τ ′ t
+1 =
P ( ·| τ ′ t , a it ) and collect reward r ′ t
Partner observes and picks fictitious action a − i t +1 ′
=
π partner ( O − i ( τ ′ t +1 ))
Apply partner’s action on the fictitious trajectory τ ′ t +
= P ( ·| τ ′ t +1 , a ′− i t +1 ) and collect reward r ′ t
+1 .
Compute target Q t = wr ′
t
First of all, it prevents sabotage (i.e. actively poor play in XP) by i.i.d.
re-sampling of the transition type (“repulsive” or vanilla) for each action-observation history.
In other words, the probability of currently being paired with the repulser is λ , even in AOH that are incompatible with the repulser.
We accomplish this by building on top of the OBL fictitious transition mechanism: in transitions paired with repulser we not only sample the next action from repulser, but also use the corresponding belief model to determine the next trajectory.
Conceptually, on those transitions we pretend that all prior actions in the episode were taken by the current partner.
Secondly, we prevent arbitrary conventions by training a hierarchy of adversary policies, π ℓ .
On the vanilla transitions (i.e. when paired with the adversary) each π ℓ uses a fixed belief model from the adversary policy at the level below, π ℓ − 1 .
Like OBL, the lowest level belief is the grounded belief, i.e. the unique belief corresponding to a uniformly random policy.
All belief models are trained using the supervised learning procedure described in Hu et al.
(2021a).
Technically, we proceed as follows:
At a given level ℓ , ADVERSITY follows a similar training pattern as OBL by unrolling policy π ℓ on the real trajectory and computing a fictitious transition given τ ′ t ∼ ˆ B ℓ − 1 and a − i t +1 ′ ∼ π ( ·| τ − i t +1 ′ ) .
However, with probability λ , the algorithm utilizes a “repulsive” fictitious transition rather than the vanilla OBL one.
In that case, we instead sample τ ′ t ∼ ˆ B µ and partner’s action from µ .
We then flip the fictitious rewards to obtain the repulsive target
[Equation]
This procedure is summarized in Algorithm 1 and each transition is illustrated in Figure 1.
Putting both transitions together, at every time step the policy learns difference Q-values that estimate the expected future discounted reward difference between vanilla and repulsive transition:
[Equation]
Breaking it down, the first line corresponds to vanilla OBL.
It implies reinterpreting the past as having been produced by a given policy π ℓ − 1 and acting according to π ℓ ever after, reinforcing only actions that lead to high expected return when interpreting the partner’s actions according to π ℓ − 1 .
This prevents feedback loops in SP where the agent can learn spurious beliefs from noise about which past actions correspond to what unobserved trajectories, and then use those beliefs to signal information and reinforce them into arbitrary and brittle conventions.
The second line represents repulsive transitions.
Resampling the current trajectory assuming the
AOH was produced by µ results in a it being reinforced if it is incompatible with µ ’s conventions and leads to low immediate rewards.
At the next step, we assume our partner is µ , thus simulating XP between the repulser and the adversary.
Here again, negating r ′ t +1 means we reinforce actions that are misinterpreted by µ .
Finally, in both vanilla and repulsive transitions, we maximize expected future discounted difference reward , denoted by V diff ℓ ( τ it +2 ′ ) .
Conceptually, these difference value functions assume that at every point in the future the repulser will intervene for one time step with probability λ , at which point rewards are inverted and the entire past is re-interpreted according to their belief model ˆ B π µ .
Overall, this procedure pushes π ℓ to select actions that achieve high return under ˆ B π ℓ − 1 , and that are simultaneously incompatible with µ ’s conventions.
This section describes training a single policy π ℓ on top of ˆ B ℓ − 1 and µ .
Because the initial belief ˆ
B 0 is restricted to only rely on grounded information, the skill level of π 1 is restricted.
Thus, to improve skill we follow the procedure from Hu et al.
(2021b)
and iteratively learn higher levels π ℓ .
At each level, we decrease λ , up to λ = 0 , at which point our training reverts to vanilla OBL.
Nonetheless, since each level uses the belief level of the previous level, in our settings the final play style is incompatible to µ , as we show in Section 7.
This shows that we truly obtain novel equilibria.
E XPERIMENTAL S ETUP
Here we describe the evaluation setting and baseline.
Training details are included in the Appendix.
6. H ANABI
We implement and test our method in Hanabi, a large scale cooperative card game proposed as a challenging benchmark for Dec-POMDP research (Bard et al., 2020).
Hanabi is played with 8 hint tokens, 3 life tokens and a deck of 50 cards, each having a rank between 1 and 5 and one of five colours.
It is a game for 2-5 players, but we restrict ourselves to the 2-player version.
The goal is to form stacks of cards in rank order for each of the five colours.
The final score, between 0 and 25, is given by the number of cards successfully stacked by the end of the game.
At any given time, player’s have five cards in their hands, and can only see their partner’s cards.
The players must therefore communicate effectively with their partners so that they can act in an informed manner.
One their turn, a player has up to 20 different actions.
They can either a) discard one of their cards, b) attempt to play one of their cards, c) hint at all the cards of a chosen colour in the partner’s hand, or d) hint at all the cards of a given rank in the partner’s hand.
Discarding replenishes one hint token.
Playing a card results in it being placed on top of the pile of its colour if it’s the next logical card in that pile.
Otherwise, the card is lost and the team loses a life.
Hinting provides limited information to the partner and consumes a hint token.
Hinting is not allowed if there are no hint tokens left, and
[Table]
[Table]
discarding is not allowed if all 8 tokens are available.
If the team loses all 3 lives, the game ends prematurely, with a score of 0.
Finally, the game also ends when there are no cards left in the deck.
6. S ELF -P LAY W ORST R ESPONSE
As a baseline, we implement a “Self-Play Worst Response” (SPWR) – a PPO agent using the same neural network architecture and hyperparameters as above but trained on SP data with probability 1 − λ and in XP with the repulser with probability λ .
Our SWPR experiments set λ = 0 .
An entire game is either SP or XP, with no switching within a single game.
When in XP, the rewards received are inverted.
This is a simpler version of LIPO from Charakorn et al.
R ESULTS
We evaluate the skill level, diversity, and reasonableness of our method against the SPWR baseline.
For both our method and our baseline, we train 3 adversary seeds for each of 4 repulser policies.
In addition to vanilla OBL level 5, the repulsers are 3 of the baseline policies inspired by Hu et al.
(2021b); namely Rank Bot, which an Other-Play (Hu et al., 2020) policy favouring rank hints,
Color Bot, which is a reward-shaped policy favouring color, and a Clone Bot, which is a supervised learning bot that we trained on human data.
We first see in Table 1 that both the SPWR and ADVERSITY agents achieve high SP, corresponding to high skill in Hanabi, and very low XP scores when paired with their respective repulser, showing that both methods produce policies that are incompatible with their repulser.
However, ADVERSITY shows a clear advantage over the SPWR in terms of intra-AXP scores, computed between independent adversary seeds.
SPWR produces different policies with every run, resulting in a very low Intra-AXP score.
In contrast, ADVERSITY agents tend to be quite similar, as shown by the low SP-XP gap.
The exception to this are adversaries to Color Bot, with an Intra-AXP score of 10.29.
To evaluate whether the adversaries exhibit meaningful diversity from their repulser, we first look at “conditional action matrices”, presented in the appendix, which display the conditional probability of a player’s action a
it +1 given the previous action, a − i t .
We find that both ADVERSITY and the SPWR exhibit meaningful diversity from their repulsers and between adversaries to different repulsers.
However, the action matrices for different ADVERSITY seeds tend to be similar, reinforcing the idea that it is reproducible.
SPWR seeds, on the other side, differ wildly, explaining the low inter-AXP score mentioned previously.
r a n k _ b o t c
o
l
o r _ b
o t c l o n e _ b o t o b l r b _ W R c
b _ W R
c
l o n e _ W R
o
b l _ W R
[Table]
bot color_bot clone_bot obl rb_WR cb_WR clone_WR
obl_WR
[Table]
[Table]
r a n k _ b o t c
o l o r _ b
o t c l o n e _ b o t o b l r b _ a d v c b _ a d v c l o n e _ a d v o b l _ a d v
[Table]
_bot obl rb_adv cb_adv clone_adv obl_adv 24.14 4.
[Table]
[Table]
[Figure]
Finally, Figure 2b shows the XP matrices between a selection of bots including the repulsers and one of each adversary seed.
What we observe is that SPWR adversaries have near-zero XP scores with virtually every partner, which is a red flag supporting the hypothesis that they learned to identify when not in SP and purposely throw the game.
Meanwhile, ADVERSITY agents exhibit a graceful degradation of ad-hoc performance depending on the similarity to the partner’s policy, which is a sign that the policies are much more reasonable.
Sabotaging: We verify that unlike ADVERSITY, SPWR adversaries exhibit sabotaging behavior in Hanabi.
We do this by measuring sabotages, the number of knowingly unplayable cards (based on revealed information) played by the agent.
We measure the average number of sabotages per game when SPWR and ADVERSITY are paired with their respective repulser agents and report results in Table 2.
ADVERSITY consistently has a low number of sabotages ( < 0 .
1 ) per game, whereas SPWR has at least 1 sabotage per game and in many cases > 2 .
The sabotages are lower for SPWR(Color Bot) simply because a color hint does not immediately reveal whether a card is definitely unplayable.
The SPWR(Color Bot) agent simply plays unhinted cards blindly, which is not necessarily a sabotage by our strict definition, but a poor move nonetheless.
Moreover, SPWR agents sabotages all non-self-play games, not just the ones with the repulser that they are trained with.
This indicates that the poor XP performance of SPWR comes not from playing a reasonable rewardmaximizing strategy that happens to be meaningfully different and incompatible with other agents, but from “deliberately” playing bad actions upon identifying that its current partner is not itself.
We also verified that the < 0 .
1 mean incidence of sabotaging for ADVERSITY is in line with vanilla OBL evaluated in SP, i.e. corresponds to a standard rate of mistakes or risky bets.
On a different metric, SPWR is responsible for a dominant amount of 2.67 ± 0.05 out of 3 life losses per game while, ADVERSITY is only responsible for 1.42 ± 0.05—roughly half of the total mistakes.
This also indicates that SPWR tries hard to terminate games deliberately, while ADVERSITY policies fail due to meaningful incompatibility, without any party trying to sabotage explicitly.
C ONCLUSION AND F UTURE W ORK
In this paper, we introduce ADVERSITY, a method for producing highly skilled and reasonable policies for a fully cooperative task that play according to meaningfully diverse conventions.
While our results show that both ADVERSITY and our baseline produce agents that exhibit high skill and meaningful diversity from their repulser, only ADVERSITY agents are also reproducible on independent runs and reasonable, as indicated by the graceful degradation of their performance with different ad-hoc partners.
The main limitation of our method is the high computational cost, making it difficult to scale the method to a large number of adversaries.
Were this issue solved, ADVERSITY could theoretically be used to produce a large pool of diverse agents by iteratively computing adversaries to past models. A A PPENDIX A. T RAINING D ETAILS
Our implementation is based on the open sourced OBL code with two main modifications.
We first replace the recurrent Q-learning backbone with PPO as it runs faster and requires significantly less memory.
Then, we implement a synchronous method that trains all OBL levels simultaneously.
Otherwise, we use a distributed training set up similar to (Hu et al., 2021b), which we detail in the appendix.
All bots are trained for 3000 epochs, each epoch consists of 1000 gradient steps.
We select the checkpoint with the highest SP score.
The original OBL trains multiple levels of policies sequentially, using the output policy of the previous level as the input policy of the new level.
In ADVERSITY, we train all levels simultaneously for faster wall-clock time.
These policies are denoted as π 0 , π 1 . . .
π L and their corresponding belief models are denoted as ˆ B 0 , . . .
, ˆ B L .
To warm up the belief model and avoid having too many invalid samples, we first train a belief model ˆ B 0 on the uniform random base policy π 0 and initialize all ˆ
B l = ˆ B 0 .
Then, L policy training tasks and L belief training tasks start at the same time.
The belief task of ˆ B l gets a latest copy of π l every 50 gradient steps and the policy task of π l gets a latest copy of ˆ
B l − 1 every 50 gradient steps.
The details of each individual belief follows the exact configurations of the original OBL paper and each policy task uses the PPO-OBL method described above.
For each adversary, we train a hierarchy of 7 levels, setting λ = 0 .
25 for l = 1 and decreasing by 0 .
every level (min. 0).
Levels l ≤ 4 are trained simultaneously, followed by levels l ≥ 5 , also trained simultaneously and with beliefs initialized at ˆ B l
= ˆ B 4 .
This split was forced by limitations on the concurrent compute available to the authors, but we anticipate no change in performance if all levels were trained simultaneously.
The ADVERSITY numbers reported in Section 7 all refer to the highest level of the hierarchy.
A. P OLICY T RAINING D ETAILS
We use a large scale distributed training framework for policy training.
To train a single policy, we run 6400 games in parallel, each adding to a centralized replay buffer.
We achieve this by runningthreads in parallel, with 80 games running per thread.
All models are on GPUs and we dynamically batch all model calls in order to increase inference speed.
This schema also allows games on the same thread to forward environment calls while certain games wait for GPU calls.
As done in (Wang et al., 2016), when an environment terminates, each game grabs all necessary objects: observations, actions, and targets, pads everything to a length of 80 and adds it to a centralized replay buffer.
For every training step we apply the PPO update rule, but instead of using the real reward and advantage, we use the fictitious values.
Every m = 10 training steps, we update the environment actors with the weights for the updated policy.
As done in Cui et al.
(2021) synchronously train our hierarchy of beliefs and policies, querying for and updating all dependencies every p = 50 training steps.
We utilize the same policy architecture as Hu et al.
(2021b).
We utilize their public-private LSTM architecture.
The public observation is encoded by a one-layer feedforwards neural network followed by a LSTM.
The private observation is encoded by a three-layer neural network.
We combine these encodings via element wise multiplication.
For all OBL experiments we compute the target with r = 1 fictitious steps.
We also sample the belief model s = 10 times and use the first sampled trajectory that doesn’t violate card constraints to compute the fictitious targets.
We then use a simulator to produce transitions from the valid trajectory.
Like Hu et al.
, we discard the fictitious transition whenever the belief fails to produce a valid sample, which in practice happens on less than 1% of transitions.
Implementation
The policy is represented by a public-LSTM network π θ with a value head and a policy head.
A large number of parallel workers generate data by sampling from a slightly outdated policy π θ ′ and write that data into a replay buffer D .
One datapoint in D is an entire trajectory τ j .
Although PPO normally does not need a replay buffer, we still use one here to fully decouple inference and training for maximum speed.
Its size is set to a small value of 1024 to minimize the instability caused by stale data.
π θ is trained with the Adam optimizer (Kingma & Ba, 2014) on minibatches of data uniformly sampled from the replay buffer.
The value loss is E τ i ∼D (cid:80) t [ r t + γV θ ( τ it +1 ) − V
θ
( τ it )] 2 .
The policy loss is E τ i ∼D (cid:80) t min[ r t ( θ ) ˙
A t , clip ( r t ( θ ) , 1 ± ϵ ) ˙
A t ] where r t ( θ ) =
π θ ( a it | τ it )
π θ ′ ( a it | τ it ) , ˙ A t =
StopGradient [ r t + γv θ ( τ it +1 ) − V θ ( τ it )] .
We perform one gradient step per minibatch.
We use 1-step bootstrapped value target instead of (cid:80) t r t because it converges significantly faster and it fits well in the OBL fictitious target computation.
π θ ′ is synced with π θ every 10 gradient updates.
A. B ELIEF T RAINING D ETAILS
We utilize the same distributed training schema from policy training for belief training.
This has also been done by (Hu et al., 2021b).
As done in policy training, we query and update dependencies every p = 50 training steps.
For belief training we store the true hand of the player along with the observation to train the belief.
For training, we train an autoregressive belief model that predicts cards oldest to newest via supervised learning.
More precisely, the belief model is trained to minimize the loss
[Equation]
(4) where h k is the k th card in the player’s hand and n is the hand size (usually 5).
A. A DDITIONAL R ESULTS
[Equation]
[Figure]
[Table]
D1D2D3D4D5P1P2P3P4P5C1C2C3C4C5R1R2R3R4R
[Table]
D1D2D3D4D5P1P2P3P4P5C1C2C3C4C5R1R2R3R4R
[Table]
[Figure]
[Table]
D1D2D3D4D5P1P2P3P4P5C1C2C3C4C5R1R2R3R4R
[Table]
D1D2D3D4D5P1P2P3P4P5C1C2C3C4C5R1R2R3R4R
[Table]
[Figure]
r a n k _ b o t c
o l o r _ b
o t c l o n e _ b o t o b l
S P W R ( r b ) _ S EE
D a
S P W R ( r b ) _ S EE
D b
S P W R ( r b ) _ S EE
D c
S P W R ( c b ) _ S EE
D a
S P W R ( c b ) _ S EE
D b
S P W R ( c b ) _ S EE
D c
S P W R ( c l o n e ) _ S
EE D a
S P W R ( c l o n e ) _ S
EE D b
S P W R ( c l o n e ) _ S
EE D c
S P W R ( o b l ) _ S EE
D a
S P W R ( o b l ) _ S EE
D b
S P W R ( o b l ) _ S EE
D c
[Figure]
[Figure]
A D V ( r b ) _ S E E D
a
A D V ( r b ) _ S E E D b
A D V ( r b ) _ S E E D c
A D V ( c b ) _ S E E D
a
A D V ( c b ) _ S E E D b
A D V ( c b ) _ S E E D c
A D V ( c l o n e ) _ S E
E D a
A D V ( c l o n e ) _ S E
E D b
A D V ( c l o n e ) _ S E
E D c
A D V ( o b l ) _ S E E D
a
A D V ( o b l ) _ S E E D b
A D V ( o b l ) _ S E E D
[Figure]
[Table]
[Figure]
[Figure]
[Figure]