Off-Team Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Zero-shot coordination (ZSC) is a framework for coordination that evaluates agentsunder cross-play (XP), i.e. paired with partners independently trained with thesame algorithm.
Off-belief learning (OBL) is a recent method for ZSC whichprevents agents from learning arbitrary conventions.
It achieves state-of-the-artresults in ZSC, ad-hoc teamplay, and proxy human-AI coordination in the complexcard game Hanabi.
However, OBL suffers from two issues: First of all, it reliesheavily on a belief model which is trained on a fixed, given policy π 0 , but evaluatedon the trained policy π 1 , potentially leading to large covariate shifts.
Secondly, OBL agents are trained only on the narrow distribution induced by their trainingpartner, which means they can easily be taken off distribution at test time.
Wepresent two methods addressing these issues.
The first, off-team belief learning(OT-BL), is a general algorithm that allows training belief models off-team—i.e.,to learn the belief for a given policy, π on the distribution induced by a differentteam, π b .
The second, off-team reinforcement learning (OT-RL), allows traininga policy off-team—i.e., learning values and policies corresponding to one policy,π , on rollouts from a different team, π b .
We empirically show that OBL+OT-BLoutperforms OBL in challenging variants of Hanabi.
Furthermore, we show that OBL+OT-RL achieves near-optimal ZSC and can be applied to mitigate largecovariate shifts in ad-hoc teamplay and proxy human-AI coordination.
Introduction
A core objective of multi-agent reinforcement learning (MARL) is to develop methods for coordinating with previously unseen partners such as other artificial agents or humans.
MARL has beenstudied through the lens of self-play (SP) where agents are only evaluated with their training partners[21].
In partially observable, fully cooperative settings SP policies develop efficient but opaquearbitrary conventions that make collaboration with novel partners, or even independent training runsof the same algorithm, extremely difficult [6, 4].
To study this issue, the zero-shot coordination (ZSC)framework was introduced as a new proxy setting towards human-AI coordination [6].
Here agentsare evaluated in cross-play (XP), i.e. with other, independently trained agents from the same method.
Under ZSC agents cannot rely on arbitrary conventions and must instead adopt strategies that can beindependently discovered and interpreted by other agents.
Off-belief learning (OBL) [7] is a recent MARL training algorithm designed to address the problemof arbitrary conventions in ZSC.
OBL starts by training a policy π 1 that reinterprets past actions ashaving been taken by a fully random policy π 0 .
This forces π 1 to only take actions based on groundedinformation rather than inferring the private information of other agents from observing their pastactions.
Similarly, this grounded policy does not actively communicate private information since itassumes this information will be ignored in future time steps.
OBL then iterates upon this groundedpolicy by training a hierarchy where each level builds upon the conventions induced by optimal playat the level below.
For each π ℓ , OBL trains a belief model ˆ B π ℓ through supervised learning, which is then used to interpret past actions when training π ℓ +1 .
OBL not only achieved state of the artresults in ZSC and human-AI coordination on the popular benchmark Hanabi [1], but formally alsoconverges to a unique policy given any fixed π 0 .
In contrast to the empirical success, there are two major problems with the current formulation ofthe algorithm from [7].
First of all, the belief model ˆ B π ℓ is trained via supervised learning on thedistribution of trajectories induced by π ℓ .
However, it is evaluated on the trajectories induced by the
OBL policy at the next level, π ℓ +1 .
When the distributions of π ℓ and π ℓ +1 differ substantially, thiswill bring ˆ B π 0 off-distribution leading to covariate shift and cause it to be unreliable.
For instance, afully random π 0 may lead to the early termination of episodes, while an optimal π 1 will play longerepisodes.
Crucially, the goal is then to learn a belief model which predicts the state of the world hadthe agent arrived at the current action-observation history by playing with π b , e.g. a random partner,even if getting there with that partner is very unlikely in the first place.
Secondly, while OBL prevents agents from learning arbitrary conventions, OBL policies are stillfundamentally brittle, since they are only trained on the distribution of trajectories induced by theirtraining partner.
As a consequence they can easily be taken off-distribution and experience covariateshift when paired with a novel test time partner, e.g. a human or another AI agent.
To address the covariate shift experienced by ˆ B π ℓ we propose off-team belief learning (OT-BL).
Rather than training ˆ B π ℓ only on the distribution induced by π ℓ , we introduce a belief-bootstrappingoperator that allows us to train our belief model off-team on the distribution induced by π ℓ +1 .
Weevaluate OT-BL empirically and show that it resolves the covariate shift experienced by belief modelswhen training OBL and significantly outperforms vanilla OBL in the challenging 2-life variant of thecard game Hanabi.
Next, to mitigate the test time covariate shift experienced by π ℓ when paired with an unseen agent,we propose off-team reinforcement learning (OT-RL).
OT-RL builds on the insight that OBL can beapplied off-team , allowing OBL to be trained on broader distributions to reduce the test time covariateshift.
We evaluate OT-RL empirically showing it achieves near optimal, state-of-the-art XP scores,higher than standard OBL.
Additionally, we apply OT-RL a small number of target trajectories fromad-hoc teamplay and proxy human-AI coordination.
With this process we are able to improve uponvanilla OBL and reduce large test time covariate shift.
Related Work2.1 Prior Work on Zero-Shot Coordination (ZSC)
Our framework of interest, zero-shot coordination (ZSC) formalizes the task of coordinating withpartners unseen during train time.
It was first introduced by [6] and further analyzed in [22], and led toa series of works meant to address the challenges of the setting.
In their original paper, [6] introduced
Other-Play, a method for training policies that are equivariant with respect to the symmetries of thegame, thus avoiding conventions that rely on arbitrary symmetry breaking.
Other works have trainedhierarchies of policies for ZSC [5] or yet leveraged diverse populations to train a robust common bestresponse [12, 19].
One of the most significant breakthroughs in ZSC is off-belief learning (OBL)
[7],which we detail in section 3.3 and upon which our work builds directly.
2.2 Belief learning
One crucial requirement for scaling OBL to large settings is learning beliefs about the private information of other players.
Belief states can be explicitly tracked.
Modern approaches to this probleminvolve deep sequence models, such as recurrent networks [20] or transformers [25].
However, thestandard training setup for these models leverages Monte Carlo sampling.
In contrast, OT-BL trainsbeliefs models via approximate dynamic programming.
In this respect, OT-BL is most similar tothe work of Sokota et al.
[16], who propose a framework for belief fine-tuning.
In this framework,a belief model for the current time-step is used to bootstrap a belief model for the next time-stepusing approximate dynamic programming.
Sokota et al.
show that belief fine-tuning can improve thequality of belief models and adapt them to new policies at inference time.
As mentioned above, belieffine-tuning and OT-BL are similar in that they use bootstrapping to learn beliefs, however whereas
Sokota et al.
[16] examined belief fine-tuning in an inference-time context, we present OT-BL as amechanism for performing off-team train-time learning.
Background3.1 Dec-POMDPs
This work considers decentralized partially observable Markov decision processes (Dec-POMDP)
Dec-POMDPs have state space s ∈ S , observation functions o i
=
Ω i ( s ) for each agent i throughwhich they obtain a partial observation of the underlying state, joint action space
A =
A 1 × A 2 ×· · · × A N , reward function R : S ×
A → R and transition function T : S ×
A → ∆ S that definesthe distribution of next state given current state and joint action.
We further denote a trajectory asτ t = ( s 1 , a 1 , r 1 , · · · , s t ) and each players action observation history (AOH) τ it =
( o i 1 , a 1 , r 1 , · · · , o it ) .
When t equals to terminal T , the subscript is dropped and the two notations are simplified as τ andτ i respectively.
Each agent acts according to their policy π i ( a | τ it ) that returns a distribution overactions given action observation history and the joint policy is simply defined as π
= ( π 1 , · · · , π N ) .
An important quantity in Dec-POMDPs is the belief function B π ( τ t | τ it ) =
P ( τ t | τ it , π ) that returnsan agent’s belief over the distribution of the state given their AOH and the group’s joint policy.
3.2 Problem Settings
Ad-Hoc Teamplay
The ad-hoc teamplay setting’s goal is to cooperate effectively with unknownteammates, where each teammate can contribute meaningfully to the task [17].
This setting hasbeen used in numerous environments including Overcooked [4], RoboCup 2D [2], and Hanabi[5, 7].
Some works leverage small amounts of data to guide agents towards equilibria or “socialconventions" [11, 23].
The ad-hoc teamplay setting includes cooperating with any well performingpolicy.
However, in Hanabi it is well known that strong SP policies aren’t necessarily well suited forcoordination [6].
As a result, for all experiments involving ad-hoc teamplay, we choose policies thatare amenable to meaningful coordination.
Zero-Shot Coordination (ZSC) Training in SP has been shown to give rise to policies that employ“arbitrary" conventions.
One example is converging on one out of combinatorially many, mutuallyincompatible possible protocols for conveying information through their actions.
Informally, thishappens because both past observations and future actions are produced by the policy being optimized.
This forms a feedback loop in which coincidences that occur when agents explore are reinforced untilthey form arbitrary conventions.
Clearly, this kind of failure points to a fundamental incompatibilityof current MARL methods with the coordination setting.
Inspired by this, zero-shot coordination (ZSC) [6] was proposed as a proxy setting towards human-AIcoordination.
Under ZSC the goal is to maximise the cross-play (XP) score, the average scorebetween independently trained agents under the same high level algorithm.
Clearly, this rules out anymethods that rely on arbitrary conventions for maximizing rewards, since those lead to incompatibleresults across independent training runs.
ZSC closely mirrors the independent reasoning process thathumans are able to undertaken when coordination with others without prior agreement.
Human-AI Coordination
The long standing goal of AI has been to coordinate with humans.
Multiplestudies have evaluated AI agents in coordinating with humans including in Hanabi [6, 15], Overcooked[4, 18], and Dota 2 [13].
A commonly used proxy setting for human-AI coordination is to evaluatewith an agent trained on human data [4, 5, 7].
We use this proxy setting in this work.
3.3 Off-Belief Learning
Intuition.
Off-belief learning (OBL) [7] is an algorithm designed to break the standard SP trainingfeedback loop and avoid arbitrariness.
At every time step OBL reinterprets the AOH as if it generatedby playing with fixed, given policy π 0 , preventing the development of conventions.
When π 0 is afully random policy past actions themselves are treated as conveying no information at all, since theprobability of an action is independent of the state of the environment.
Thus, the only informationretained from an action is what it explicitly reveals about the environment.
In turn, any accidentalattempt to encode private information through the actions will be ignored in subsequent time stepsand will not be reinforced.
Instead, agents have to adopt grounded play , acting only on the basis ofthe grounded information and prioritizing actions that reveal grounded information in turn.
Implementation.
In practice, OBL trains a sequence of policies starting with π 0 a uniform randompolicy and iteratively using the trained policy π 1 as the new π 0 until the improvement ∆( J ) =
J ( π 1 ) − J ( π 0 ) diminishes.
Keeping track of the exact belief is computationally expensive, thus OBLis implemented by first training a neural network belief model on π 0 using supervised learning.
Thismodel is used to sample fictitious states τ ′ t by evaluating it on π 1 ’s data, i.e. τ ′ t ∼ ˆ B π 0 ( τ t | τ it ) , where τ t ∼ P π 1 ( τ it ) .
At every timestep t , the policy π 1 is queried to produce an action on the fictitious state .
This action is used to perform a fictitious transition , which is used to calculate the target value fortraining OBL (See figure 2 for a visual representation of the fictitious transition).
The Off-Team Learning Problem
In single agent RL, off-policy RL is the task of learning information about a target policy π , fromrollouts generated by a distinct behavior policy π b .
Similarly, in MARL we define off-team (OT)learning to be the task of learning policies or values for target joint policy π , from rollouts generatedby playing with a distinct behavior team π b .
Despite the similarities in definitions, note that themethods for the former do not transfer to the latter.
Indeed, unlike in single-agent settings the AOH isonly Markov given fixed teammates.
We therefore need to carefully approach off-team learning toprevent π b ’s conventions from bleeding into π , the joint policy being learned.
Formally, we definethe OT-Belief Learning (OT-BL) problem as follows
[Equation]
In words, the goal is to learn an accurate belief model, B π , for the policy π when evaluated on thedistribution of AOH induced by the policy π b .
The loss function, L , is defined in eq. 4.
Similarly, formally, we define the OT-RL problem as follows
[Equation]
where ⊥ ( π ) denotes π but caveats that it cannot be optimized.
Notably, unlike in off-policy reinforcement learning, where it is often desirable to re-weight the data according to the target policy reachprobabilities, in off-team learning, the fact that the objective is weighted according to the behaviorpolicy reach probabilities is a core part of the problem setting.
Off-Team Learning
Off-team learning is a difficult problem because it requires weighting the loss values by the reachprobabilities induced by π b .
However, we can’t simply naively train our learned belief model, ˆ B π ,on data generated by π b , since this would result in a belief model for π b rather than π .
Similarly,we cannot simply naively train our policy on data generated by π b because such a policy wouldleverage the correlations between π b ’s actions and its private information.
To address this problemwe introduce off-team belief learning (OT-BL) and off-team reinforcement learning (OT-RL) , whichuse bootstrapping to perform forward induction on the beliefs and backward induction the values(respectively), and trajectory resampling to ensure that each loss is weighted by the correct value.
Say we have learned a belief model
ˆ B π up to time t − 1 , and a value function Q π for time t andthereafter.
We describe off-team learning’s procedure for generating data for time t in three steps.
Off-team learning samples trajectories according to the behavior policy π b , thereby guaranteeing
that the samples τ it − 1 are weighted by the behavior distribution
Off-team learning corrects for the fact that τ t − 1 was played to t − 1 by π b , rather than π .
It doesso by resampling τ ′ t − 1 using the learned belief model, ˆ B π ( τ t − 1 | τ it − 1 ) , taking advantage of thebelief model being accurate up to time t − 1 .3.
We propagate this trajectory forward in time by sampling an action a ∼ π ( τ ′ t − 1 ) from the targetpolicy, π , and sampling a trajectory for the next time step τ ′ t ∼ T ( τ ′ t − 1 , a ) using this sampledaction.
This τ ′ t is distributed approximately according to B π ( τ i ′ t ) but only generated by OT-BLwith the reach probability induced by π b .
Thus, by training ˆ B π ( τ i ′ t ) to maximize the likelihood ofτ ′ t , we are minimizing the loss defined in expression 1.
Analogously, by performing a TD-updatebetween Q ( τ i ′ t − 1 , a ) and R ( τ ′ t − 1 , a ) + max a ′ Q ( τ i ′
t , a ′ ) , we are maximizing the objective definedin expression 2.
The procedure described above is described in pseudocode in algorithms 1 and 2 and visualizationsare provided in Figure 1 and Figure 2.
Experimental Setup
6.1 Hanabi
Hanabi [1] is a challenging benchmark for Dec-POMDP research.
It is a cooperative game for 2-5players played with 8 hint tokens, 3 life tokens, and a deck of 50 cards, each having a rank between 1and 5 and one of five colors.
In the 2-player variant, on which this paper focuses, each player holds5 cards, backwards, such that they can see their partners’ hand but not their own.
On their turn, aplayer can either play , discard or hint .
The goal is to play cards in rank order for each color, witheach correctly played card providing one point for a maximum score of 25.
The team loses a lifetoken each time a player attempts to play an unplayable card.
Hinting consumes a hint token and canbe used to point all cards of a chosen color or rank in a partner’s hand 1 .
The team regains a hint tokenwhen a player discards a card or completes a stack by successfully playing the 5 of the correspondingcolor.
A player draws a new card after playing or discarding.
The game terminates when all 5 stacksare completed, the deck is empty, or all three lives are lost.
In the latter case the team loses all thepoints they have earned.
6.2 Training Details
We implement our algorithms based on the open-sourced code for OBL 2 and extend it with ideasfrom synchronous training [10, 5], training all models simultaneously, thus enabling effective imple- [Figure] mentations of OT-BL and OT-RL.
When synchronously training OBL models, every n = 50 steps wesave a copy of the model then query for and update all dependencies.
We follow practices in the original OBL paper [7] to train each policy and belief model.
Thebackbone is R2D2 [8], a distributed recurrent Q-learning method with parallel environment workersand centralized replay buffer and trainer.
We briefly describe the loss functions for the Q-networkand belief models here.
For more details, please refer to Appendix A.1 or the OBL paper [7].
The Q-network is trained to minimize the TD-error between its Q-value and targets from fictitioustransition:
[Equation]
[Equation]
where ˆ θ is a slightly outdated target network.
We pre-compute the fictitious target G ′ t and store theminto the replay buffer to avoid branching LSTM rollouts into ficitious transitions at training time.
The belief models are trained to predict the player’s hand given τ i .
At a given timestep t , the beliefmodel will take in τ it as input and predict the hand of player i auto-regressively from oldest card tonewest.
We train all OBL models as recurrent models via supervised learning to optimize:
[Equation]
where h k is the k th card in hand and n is the hand size.
6.3 Evaluation
We repeat each training method five times with different seeds.
For SP and XP, every team is evaluatedon 5000 games and then averaged over five seeds.
The XP score is the average score of all teamsformed by all independently trained agents.
The goal of ZSC is to evaluate the consistency of analgorithm across different seeds.
To better reflect improvement in ZSC, we introduce the self-playcross-play gap (SXG) metric, which is computed as SXG = 100 · ( SP − XP ) SP .
This normalizesincreases in XP score relative to the theoretical maximum self-play score.
Additionally, SXG acts asa proxy for the covariate shift as it compares the evaluated training distribution (SP) with the testdistribution (XP).
The higher the SXG the larger the covariate shift.
6.4 Experimental Design for OT-BL
We design experiments for OT-BL to emphasize its advantage over the normal belief learningprocedure when evaluated off-team.
In normal OBL, the belief model is trained on trajectories fromπ ℓ but utilized on trajectories from π ℓ +1 to produce ficitious transitions, making it vulnerable to covariate shift.
With OT-BL, half of environments used to train ˆ B π ℓ follow typical belief modeltraining.
The other half uses OT-BL to train ˆ B π ℓ off-team on P π ℓ +1 ( τ it ) .
Ideally, we want to evaluate our algorithm on environments with a larger covariate shift .
Therefore,we make a few modifications to the Hanabi environment to exacerbate the covariate shift.
Firstwe reduce the total number of life tokens from 3 to 2, making the game harder so that the averagescore and average trajectory length of different OBL levels differ more significantly.
It is particularlychallenging for the belief model only trained on shorter trajectories to generalize to longer unseentrajectories.
Second , we consider sampling valid hands .
Prior works including OBL utilize a checkerprovided in the Hanabi challenge [1] to check if a hand satisfies the known public knowledge.
If ahand is invalid, the environment can fail to produce key input features.
We further modify the 2-life
Hanabi environment to remove the checker and allow invalid samples to corrupt the environment asis, which only affects the fictitious transition.
Specifically, if a sampled card contradicts the publicknowledge of that card, the observation encoder will just encode the card knowledge for that card tobe all zero instead of rejecting the sampled hand.
We call this invalid card penalty .
Finally, we measure the belief improvements through thepercentage of valid samples for fictitious transitions.
Forthis we focus on the level 1 policy training process in the
OBL hierarchy, where there is the largest covariate shift.
To train the level 1 policy, we need to unroll a belief modelˆ B π 0 on trajectories generated by the learning policy π 1 .
At convergence, π 1 can achieve 20 points.
However, thebelief model is trained on π 0 , a uniform random policywith average score close to zero.
We compute the percentage of timesteps where ˆ B π 0 is able to produce a validsample when training π 1 as a proxy for the amount thebelief model has deteriorated due to covariate shift.
6.5 Experimental Design for OT-RL
OT-RL for ZSC We design experiments to evaluate OTRL’s performance in ZSC.
For normal OBL, the policy,π ℓ , is trained on trajectories from P π ℓ ( τ it ) but evaluatedon P π ℓ ,π b ( τ it ) .
This means π ℓ experiences test time covariate shift.
To evaluate OT-RL in ZSC, when training a given policy π ℓ , we train on real trajectories fromfrom P π ℓ − 1 ,π ℓ ( τ it ) , P π ℓ ( τ it ) , P π ℓ ,π ℓ +1 ( τ it ) and use OT-RL 266 to train off-team.
By training on broader distributions, weincrease the support of our policies.
Since π ℓ − 1 and π ℓ +1are OBL policies, they are also reasonable and do wellin Hanabi.
We also test a variant where we use OT-RLand robustify the belief by also training the belief modelon P π ℓ − 1 ,π ℓ ( τ it ) , P π ℓ ( τ it ) , P π ℓ ,π ℓ +1 ( τ it ) .
We refer to this 272 variant as OT-RL + robust belief.
OT-RL for ad-hoc teamplay and human-AI coordination This set of experiments aims to demonstrate theability for OT-RL to improve test time performance withonly a small amount of data from test time agents.
Theseexperiments address the ad-hoc teamplay and human-AIcoordination settings where there are large covariate shifts .
We utilize rank bot for the ad-hocteamplay setting and clone bot , a bot trained via supervised learning on human data, as a proxyfor human-AI coordination.
See appendix A.3 and A.4 for more details on training rank and clonebot respectively.
Each setting we assume access to only a small number, 3200 , of full trajectoriesgenerated by the target agents.
We retrain a new policy, π , with a belief model trained on level 3 ofsequentially trained OBL with the typical OBL training procedure.
In this case half of the games inour distributed RL environment are the real target trajectories and we train π in an off-policy mannerwith OT-RL on these trajectories.
By training on these target trajectories, we can partially mitigate thecovariate shift experienced when π is evaluated.
We compare our results with a vanilla sequentiallytrained OBL level 4.
[Table] [Table]
OT-RL prevents overfitting and continues to improve its XP score.
OT-RL almost always has a higher
XP scores than vanilla OBL.
OT-RL has a very small SXG < 1% and this gap is up to 10 times lower than that of vanilla OBL.
By training our belief models on broader distributions, we further increase our XP score while maintaing a low SXG.
The ± the team’s standard error.
Results and Discussion7.1 OT-BL in 2-life Hanabi
The the XP and SXG results of OT-BL on 2-life Hanabi is shown in Table 1.
Without the invalid cardpenalty , both OBL with vanilla belief learning and OT-BL are able to achieve reasonably high XPscores in 2-life Hanabi.
However, OT-BL consistently achieves higher XP scores in 2-life Hanabiacross all levels.
At level 3, OT-BL is able to achieve a final XP score of 23 .
76 ± 0 .
Lastly, the
SXG for OT-BL is always lower than that of vanilla OBL (8-16x lower).
In Table 1, we also show the performance of OT-BL on 2-life Hanabi with invalid card penalty onthe
right.
Similarly, OT-BL consistently achieves higher XP scores across all levels.
OT-BL is ableto achieve a small SXG (always < 1% ) and this gap is up to 18 × lower than vanilla OBL.
OT-BLachieves similar results between the penalized and unpenalized environments since the belief modelis more on distribution, thus it almost always samples valid samples ( > 99 .
5% of the time).
However,
OBL with vanilla belief learning suffers significantly from the penalty, especially for level 1 wherethe covariate shift from a uniform random policy to a learned OBL policy is the largest.
This indicatesthat the test time covariate shift is lower for OT-BL when compared to Vanilla OBL.
The reducedcovariate shift improves training and leads to better test time results.
We plot the percentage of timesteps ˆ B π 0 is able to produce a valid sample when training π 1 in Figure3, to illustrate the deteriorating belief model in normal OBL and show the benefit of OT-BL.
Withoutpenalty , vanilla OBL steadily decreases in the percentage of timesteps producing valid samples.
Withpenalty , vanilla OBL is unable to produce a high percentage of valid samples and can never recover.
This high percentage of invalid samples contributes to the steep score drop for OBL level 1 in theinvalid state experiments.
On the other hand, OT-BL mitigates the belief model’s training covariateshift and consistently samples valid states (over 99 .
5% of the time).
This contributes to the highscores for OT-BL and explains why OT-BL maintains high scores in penalized Hanabi.
7.2 OT-RL
ZSC Results We present the XP and SXG results of OT-RL in table 2.
All variants, normal OBL,
OT-RL , and OT-RL + robust belief all achieve high XP scores.
However, at level 4 vanilla OBL startsto overfit, whereas OT-RL and OT-RL + robust belief continue to improve.
This results in OT-RLachieving a XP score of 24 .
24 ± 0 .
01 and OT-RL + robust belief achieves a XP score of 24 . 30 ± 0 . 01 .
As Vanilla OBL overfits, the SXG increases, indicating an increase in the test time covariate shift.
On the other hand, both OT-RL and OT-RL + robust belief always have a SXG < 1% and at level
5the SXG is almost 0 .
By level 5, the SXG of OT-RL and OT-RL + robust belief are at least 8 times
[Table]
, we indicating we only broadening lower than that of vanilla OBL.
Overall, by training on broader distributions, we reduce the test timecovariate shift, achieve near optimal ZSC performance, and improve OBL in off-team evaluation.
Ad-Hoc Teamplay and Proxy Human-AI Coordination Results We present the results appling
OT-RL Ad-hoc teamplay and Human-AI coordination in table 3.
With OT-RL and only 3200 targetpolicy trajectories, we can increase rank bot to 14 .
47 ± 0 .
67 to 15 .
66 ± 0 .
and increase clonebot scores from 16 .
19 ± 0 . 21 to 16 . 64 ± 20 without heavily impacting XP scores or the SXG.
Additionally, vanilla OBL in XP achieves near identical scores to OT-RL trained on additionaltrajectories is paired with vanilla OBL.
Therefore, the additional trajectories merely broaden thetraining policies support and don’t negatively impact policy training.
Overall, by training on targetdistributions off-policy and off-team with OT-RL we are able to reduce the test time covariate shiftand increase ad-hoc teamplay and proxy human-AI coordination scores.
Limitations
Although OT-BL enables B π to train off-team on any distribution P π b ( τ it ) , it assumes π is also ondistribution when we produce fictitious moves with OT-BL.
As a result, if π is too off distribution
OT-BL might not fully mitigate the covariate shift.
Additionally, although our work provides apositive step to mitigate the large covariate shift that can occur in ad-hoc teamplay and human-AIcoordination with little target policy data, there is still exists covariate shift.
Conclusion
We present off-team belief learning , a method that allows us to train any belief off-team .
Beliefmodels are very ubiquitous in POMDPs and Dec-POMDPs.
However, they can experience covariateshift when deployed on unseen distributions.
We demonstrate that standard belief model training cansuffer from covariate shift, but OT-BL reduces it.
Overall, by training OBL with OT-BL, we are ableto achieve strong results in ZSC in 2-life Hanabi and minimize the gap between SP and XP.
We also present off-team reinforcement learning , a method that allows us to train policies off-team .
The goal of off-team training is to train with various teams with the goal of improving performancewith unseen teams.
With OT-RL, we can train policies on broader distributions, increase their support,and improve test time performance.
By training OBL with OT-RL we were able to achieve nearoptimal-ZSC performance and improve ad-hoc teamplay and proxy human-AI coordination.
For follow-up work, a promising direction involves combining multiple ways of reducing test timecovariate shift.
For instance, Strouse et al.
[19] trained on a population of various checkpoints, Cuiet al.
[5] synchronously trained cognitive hierarchies, and Lupu et al.
[12] learned a common bestresponse to diverse agents; all works which could complement our methods by further broadeningthe training distribution.
Additional research is also needed to produce agents that can accommodatehuman behaviour such as “finessing" or “bluffing" in Hanabi, which relies on theory of mind and canalso induce large covariate shifts.
Broader Impact
We have demonstrated both OT-RL and OT-BL can reduce covariate shift.
OT-BL reduces covariateshift in the difficult 2-life Hanabi case in ZSC compared to vanilla OBL.
OT-RL reduces covariateshift in ad-hoc teamplay, ZSC, and proxy human-AI coordination.
Overall, OT-RL and OT-BL aregeneral methods for belief and policy robustification.
As a result, we have provided positive stepstowards numerous difficult coordination settings.
Although, no technology is immune to misuse, weare working in purely cooperative settings.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets? • Did you include the license to the code and datasets?
[No]• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]