Equivariant Networks for Zero-Shot Coordination
Abstract
Successful coordination in Dec-POMDPs requires agents to adopt robust strategies and interpretable styles of play for their partner.
A common failure mode is symmetry breaking , when agents arbitrarily converge on one out of many equivalent but mutually incompatible policies.
Commonly these examples include partial observability, e.g. waving your right hand vs. left hand to convey a covert message.
In this paper, we present a novel equivariant network architecture for use in Dec-POMDPs that prevents the agent from learning policies which break symmetries, doing so more effectively than prior methods.
Our method also acts as a “coordinationimprovement operator” for generic, pre-trained policies, and thus may be applied at test-time in conjunction with any self-play algorithm.
We provide theoretical guarantees of our work and test on the AI benchmark task of Hanabi, where we demonstrate our methods outperforming other symmetry-aware baselines in zeroshot coordination, as well as able to improve the coordination ability of a variety of pre-trained policies.
In particular, we show our method can be used to improve on the state of the art for zero-shot coordination on the Hanabi benchmark.
Introduction
A popular method for learning strategies in partially-observable cooperative Markov games is via self-play (SP), where a joint policy controls all players during training and at test time [21, 37, 45, 53].
While SP can yield highly effective strategies, these strategies often fail in zero-shot coordination (ZSC), where independently trained strategies are paired together for one step at test time.
A common cause for this failure is mutually incompatible symmetry breaking (e g. signalling 0 to refer to a “cat” vs. a “dog”).
Speciﬁcally, this symmetry breaking results in policies that act differently in situations which are equivalent with respect to the symmetries of the environment [22, 38].
To address this, we are interested in devising equivariant strategies, which we illustrate in Figure 1.
Equivariant policies are such that symmetric changes to their observation cause a corresponding change to their output.
In doing so, we fundamentally prevent the agent from breaking symmetries over the course of training.
In earlier work, the Other-Play (OP) learning algorithm [22] addressed symmetry in this setting by training agents to maximize expected return when randomly matched with symmetry-equivalent policies of their training time partners.
This learning rule is implemented by independently applying a random symmetry permutation to the action-observation history of each agent during each episode.
OP, however, has some pitfalls, namely: 1) it does not guarantee equivariance; 2) it poses the burden of learning environmental symmetry onto the agent; and 3) it is only applicable during training time and cannot be used as a policy-improvement operator.
36th Conference on Neural Information Processing Systems (NeurIPS 2022).
To address all of this, our approach is to encode symmetry not in the data, but in the network itself .
There are many approaches to building equivariant networks in the literature.
Most work focuses on analytical derivations of equivariant layers [11, 49, 51, 52], which can prove time-consuming to engineer for each problem instance and inhibits rapid prototyping.
Algorithmic approaches to constructing equivariant networks are restricted to layer-wise constraints [10, 17, 18, 35], which require computational overhead to enforce.
Furthermore, for complex problems such as ZSC in challenging benchmarks such as Hanabi [5], these approaches are insufﬁcient because 1) they are not scalable; and 2) they usually do not allow for recurrent layers, which are the standard method for observation histories in partially observable settings.
In this paper, we propose the equivariant coordinator (EQC), which uses our novel, scalable approach towards equivariant modelling in Dec-POMDPs.
Speciﬁcally, our main contributions are:
Our method, EQC, that mathematically guarantees symmetry-equivariance of multi-agent policies, and can be applied solely at test time as a coordination-improvement operator; and 2.
Showing that EQC outperforms prior symmetry-robust baselines on the AI benchmark Hanabi.
Background
This section formalises the problem setting (2.1), provides a brief primer to some group-theoretic notions (2.2), introduces equivariance (2.3), provides background to prior multi-agent work on symmetry (2.4), and offers a small, concrete example to help tie these notions together and facilitate intuition (2.5).
1 Dec-POMDPs
We formalise the cooperative multi-agent setting as a decentralized partially-observable Markov decision process (Dec-POMDP) [32] which is a 9-tuple ( S , N , {A i } ni =1 , {O i } ni =1 , T , R , {U i } ni =1 , T,  ), for ﬁnite sets
S , N , {A i } ni =1 , {O i } ni =1 , denoting the set of states, agents, actions and observa- tions, respectively, where a superscript i denotes the set pertaining to agent i 2 N = { 1 , . . . , n } (i.e.,
A i and O i are the action and observation sets for agent i , and a i 2 A
i and o i {+
+}
O i are a speciﬁc action and observation of agent i ).
We also write A = ⇥ i
A i and O = ⇥
i O i , the sets of joint actions and observations, respectively.
s t 2 S is the state at time t and s t = { s kt } k , where s kt is state feature k of s t .
a t 2 A is the joint action of all agents taken at time t , which changes the state according to the transition distribution s t +1 ⇠ T ( s t +1 | s t , a t ) .
The subsequent joint observation of the agents is o t +1 2 O , distributed according to o t {++
+}
⇠ U ( o t +1 | s t +1 , a t ) , where U = ⇥
i U i ; observation features of o it +
2 O i are notated analogously to state features; that is, o it +1 =
{ o i,kt +1 } k .
The reward r t +1 2 R is distributed according to r t +1 ⇠ R ( r t +1 | s t +1 , a t ) .
T is the horizon and  2 [0 , 1] is the discount factor.
Notating ⌧ it =
( a i 0 , o i 1 , . .
. , a it  1 , o it ) for the action-observation history of agent i , agent i acts according to a policy a it ⇠ ⇡ i ( a it | ⌧ it ) .
The agents seek to maximize the return, i.e., the expected discounted sum of rewards:
[Equation]
where ⌧ t = ( s 0 , a 0 , o 1 , r 1 , . .
. , a t  1 , o t , r t , s t ) is the trajectory until time t .
2.2 Groups
A group G is a set with an associative binary operation on G , such that, relative to the binary operation, the inverse of each element in the set is also in the set, and the set contains an identity element.
For our purposes, each element g 2 G can be thought of as an automorphism over some space X (i.e. an isomorphism from X to X ), and the binary operation is function composition.
The cardinality of the group is referred to as its order.
If a subset K of G is also a group under the binary operation of G (a.k.a.
a subgroup), we denote it K < G .
A group action 1 is a function G ⇥ X !
X satisfying ex = x (where e is the identity element) and ( g · h ) x
= g · ( hx ) , for all g, h 2 G .
In particular, for a ﬁxed g , we get a function ↵ g : X !
X given by x 7!
gx .
g can be thought to be “relabeling” the elements of X , which aligns with our deﬁnition for Dec-POMDP symmetry introduced Section 2.4.
Groups have speciﬁc mathematical properties, such as associativity, containing the inverse of each element, and closure (closure is implicit of the binary operation being over G itself) which are necessary for proving Propositions 1 and 2 and ensuring equivariance.
2.3 Equivariance
Let G be a group.
Consider the set of all neural networks of a given architecture whose ﬁrst and ultimate layers are linear; that is,
[Equation] } , where f, h are linear functions, and ˆ is a non-linear function.
We say a network is equivariant (with respect to a group G , or
G -equivariant)
if
[Equation]
where L g is the group action of g 2 G on inputs, K g is similarly the group action on outputs, and d is the dimension of our data.
If satisﬁes Equation 2, then we say equiv .
In the context of reinforcement learning, one may consider as the neural parameterization of a policy ⇡ 2 , so a policy is G -equivariant if
[Equation]
If for any choice of g 2 G we have that K g
=
I , the identity function/matrix, then we say ⇡ (or ) is invariant to g .
2.4 Other-Play and Dec-POMDP Symmetry
The OP learning rule [22] of a joint policy, ⇡ = ( ⇡ 1 , ⇡ 2 ) , is formally the maximization of the following objective (which we state for the two-player case for ease of notation):
[Equation]
where  is the class of equivalence mappings (symmetries) for the given Dec-POMDP, so that each  2  is an automorphism of S , A , O whose application leaves the Dec-POMDP unchanged up to relabeling (  is shorthand for
= {  S ,  A ,  O } ).
The expectation in Equation 4 is taken with respect to the uniform distribution on  .
Each  2  acts on the action-observation history as
[Equation]
and acts on policies as
[Equation]
Policies ⇡ , ˆ ⇡ in Equation 6 are said to be symmetry-equivalent to one another with respect to  .
To correspond with Equation 3, we have L  = {  A ,  O } and K  =  A .
2.5 Idealized Example
Consider a game where you coordinate with an unknown stranger.
Each of you must pick a lever from a set of 10 levers.
Nine of the levers have 1 written on them, and one has 0 .
9 written.
If you and the stranger pick the same lever then you are paid out the amount that is written on the lever, otherwise you are paid out nothing.
We have G :=
S 9   , where S 9 is the symmetric group over 9 elements (all possible ways to permute 9 elements), since the nine levers with 1 written on them are symmetric.
If you (using policy ⇡ 1 ) pick the 1 -point lever l 1 and the stranger (using policy ⇡ 2 ) picks the 1 -point lever l 2 , then ⇡ 1 and ⇡ 2 would be symmetry-equivalent with respect to any  2 G that permutes the lever choices (e.g.
( l 1 ) =
l 2 , or equivalently L  ( l 1 ) =
K  ( l 1 ) =
l 2 , as the levers are both observations and actions).
If, as is likely, l 1 6 = l 2 , you and the stranger leave empty-handed, illustrating symmetry breaking.
If during training you and the stranger constrained yourselves to the space of equivariant policies whilst trying to maximize payoff, you would both converge to the policy that picks the 0 .
9 -point lever (which is not only equivariant, but invariant to all  2 G ), and so the choice of using equivariant policies allows symmetry to be maintained and payout to be guaranteed.
Method
We now present our methodology and architectural choices, as well as several theoretical results.
3.1 Architecture
We ﬁrst consider the symmetrizer introduced in [35], which is a functional that transforms linear maps to equivariant linear maps.
Inspired by this we present a generalisation of the symmetrizer that maps from to the equivariant subspace, equiv , namely S : [Equation] equiv , which we deﬁne as
[Equation]
so that the ﬁrst layer of , denoted f , being linear, is composed with each L g , and the ultimate layer, denoted h , is similarly composed with K  1 g (i.e. K  1 g [Equation] L [Equation]
K  [Equation]  h [Equation]
ˆ [Equation] f  L g )) , where
= h ( ˆ ( f ) ).
Below we establish properties of S that were analogously shown for the linear symmetrizer case [35]:
Proposition 1.
(Symmetric Property) S ( )equiv , for all ; that is, S maps neural networks to equivariant neural networks.
Proposition 2.
(Fixing Property) equiv =
Ran ( S ); that is, the range of S covers the entire equivariant subspace.
The proofs of these propositions can be found in the Appendix.
The proof of Proposition 1 relies on properties of the group structure (such as closure of the group); it is thus critical for the symmetrizer to sum over permutations that together form a group structure, and not any random collection of permutations.
Propositions 1 and 2 show in combination that the architecture of S ( ) is indeed equivariant, and that optimizing over the class Ran ( S ) (that is, keeping the permutation matrices 3 L g and K  1 g for all g frozen while updating the other weights) is equivalent to optimizing over equiv .
The sum over each g 2 G of S ( ) is highly parallelizable [19], which allows for maintaining tractability.
In this way we algorithmically enforce equivariance, mitigating the need to design equivariant layers by hand, as is typical [11, 12, 48, 49, 51, 52].
We note that for a network containing LSTM [20] layers, S will still map the network to an equivariant one no matter the values used for the hidden and cell states; indeed, Propositions 1, 2 and 4 all hold regardless of how the hidden and cell states are treated.
However, how the hidden and cell states are treated matter for the overall performance of the network (equivariance is not a sufﬁcient condition itself for success in ZSC, consider the uniformly random policy).
We ergo propose the following: for each hidden state h t,g and cell state c t,g produced by the LSTM layer over input ( L g ( ⌧ ) , h t  1 , c t  1 ) , we compute h t = 1 | G |
P g 2 G h t,g and c t = 1 | G | P g 2 G c t,g , and proceed inductively.
This is a nonobvious design choice, since each h t,g and c t,g is the output of a non-linear function; a more obvious choice would be to choose h t = h t,e and c t =
c t,e where e 2 G is the identity element, but to ensure symmetry-aware play we experimented with this averaging scheme to “symmetrically adjust” the short and long-term memories.
Since all the permutations over the input are computed in parallel with matrix multiplication, this design choice retains tractability.
We test this choice empirically in
Section 4.
See the Appendix for further experimentation surrounding this design choice.
3.2 A Theory for Symmetry And Coordination
The following propositions provide guarantees for EQC’s ability to prevent all symmetry breaking:
Proposition 3.
If ⇡ 1 and ⇡ 2 are symmetry-equivalent policies with respect to  2 G and ⇡ 1 is
G -equivariant, then ⇡ 1 = ⇡ 2 .
Proof.
We have ⇡ 1 =  ( ⇡ 2 ) for some  2 G , meaning {+⇡ 
+}
(   1 ( a ) |  ( ⌧ )) =
⇡ 2 ( a | ⌧ ) .
Identifying
L  = {  A ,  O } and K  =  A , by equivariance (Equation 3)
we observe that ⇡ 1 ( a | ⌧ ) =
⇡ 1 ( K
K  1  ( a ) | ⌧ ) =
⇡ 1 ( K  1  ( a ) | L  ( ⌧ )) =
⇡ 1 (   1 ( a ) |  ( ⌧ )) .
Thus ⇡ 1 ( a | ⌧ ) =
⇡ 1 (   1 ( a ) |  ( ⌧ )) =
⇡ 2 ( a | ⌧ ) .
Proposition 4.
If ⇡ 1 and ⇡ 2 are symmetry-equivalent policies with respect to  2 G , then S ( ⇡ 1 ) =
S ( ⇡ 2 ) .
Proof.
This is a corollary of the proof of Proposition 1, such that the symmetrizer architecture is ﬁxed under any  2 G .
G -equivariance thus reduces cross-play to self-play between policies that are symmetry-equivalent with respect to any  2 G .
This is critical, as this means equivariance merges policies of large classes together, such that symmetry-equivalent policies now indeed become equivalent in coordination.
Put another way, the set of policies that are symmetry-equivalent with respect to any  2 G form an equivalence class, and G -equivariance reduces the cardinality of each equivalence class to 1 , thus allowing ﬂuent coordination of policies within an equivalence class (as it merely becomes self-play).
3.3 Algorithmic Approach
Based on Propositions 1 and 2, a “naive approach” to obtaining a G -equivariant agent, for a given group G , might then be as follows: 1) initialize a random seed ; 2) map to S ( ) ; 3) train
S ( ) using any self-play algorithm, updating the weights while keeping the permutation matrices L g and K  1 g for all g frozen; 4) S ( ) is ready for inference.
This naive approach is sound, but time and memory complexity inhibit scaling to large choices of G .
For this reason we present the following two options (that together summarize EQC):
As a ﬁrst option, we introduce “ G -OP”, where during training we sample random subsets of G for each mini-batch (i.e. we select a random g 2 G for each x in the mini-batch, so that g group-theoretically acts on x , and update the weights of ) and at test time, we use all the permutations of G by deploying S ( ) .
As a second option, we train under any self-play algorithm and at test time deploy S ( ) . With
Option 1 we train over the permutations of G at training time.
With Option 2 we train under standard SP and use S ( · )
to obtain S ( ) at
test time , which can be viewed as a coordinationimprovement operator. We
empirically validate both
options
in Section
From here on, we
refer
to S ( ) as “symmetrizing
We expect compatibility between the G -OP trained agents and the symmetrizer at test time, since at training time, these agents were trained over these permutations.
For Option 2, where we have a network  not trained with permutations, it is worthwhile to inspect S (  ) ; while we know S (  ) is equivariant, we know little else regarding the conventions and style of play it would use.
While symmetry is a critical component of ZSC, other factors (such as choosing robust actions that are likely to succeed under minimal assumptions [2, 13]) are also signiﬁcant.
The following points justify why
S maps any network to one that is not only equivariant, but overall stronger at ZSC: 1) the network averages over how would act under each g 2 G , thus ensuring play that accounts for multiple symmetries; 2) the group closure property ﬁxes the architecture under any permutation in the group (see the proof of Proposition 1), forcing consistent play across different symmetries.
We verify the empirical efﬁcacy of Options 1 and 2 in Section 4.
What remains is the choice of the group G for our G -equivariant agents.
One might consider the choice of G =  , but this in general scales factorially with the size of the Dec-POMDP (e.g. the automorphism group over an environment with m symmetric features grows asymptotically with the number of permutations over m elements), and so quickly becomes infeasible.
Hence, we treat
G as a hyperparameter, and in Section 4 we analyze how different choices of G affect ZSC.
In any case, picking G as a subgroup of  may be motivated in and of itself, when enumerating all possible symmetries of an environment is intractable, or if an environment is obfuscated such that some symmetries cannot be known (as is the case in many large, realistic domains).
In this way, our method can relax the assumption of requiring access to all environment symmetries by ﬂexibly choosing equivariance with respect to a subgroup.
Experiments
The primary test bed for our methodology is the AI benchmark task Hanabi [5].
Hanabi is a unique and challenging card game that requires agents to formulate informative implicit conventions in order to be successful.
Hanabi has served as the primary test bed for many algorithms designed for ZSC [14, 22, 23, 28] due to its representation as a formidable Dec-POMDP task (see the Appendix for an introduction to Hanabi).
To make sense of the ensuing results, recall that a perfect score in Hanabi is 25 , and that “bombing out” refers to expending all three fuse tokens and ending the game with a score of 0 .
Agents thus must be cautious not to expend all fuse tokens.
In Hanabi, the class of environment symmetries are the permutations of the card colors, because assuming there is no supplemental information, a permutation of the colors of the cards leaves the game unchanged.
We therefore have  = S 5 , the symmetric group over 5 elements (all possible permutations over 5 elements).
5 here represents the number of different colors.
| S 5 | = 5!
= 120 .
In this section we analyze differences in play and the relationship to environmental symmetry between our equivariant agents and OP agents (4.1) and evaluate various pre-trained agents before and after symmetrization (4.2).
These subsections evaluate Options 1 and 2, respectively (see Section 3).
For further details of the training setup and hyperparameters used in the following experiments, please refer to the Appendix.
[Table]
4.1 Symmetry-Robust Agents
Here we compare the 12 OP agents from [22] with our equivariant agents.
Speciﬁcally, we conduct the following ablation study: we ﬁx two subgroups of S 5 : C 5 and D 10 , which are the cyclic group of order 5 and the dihedral group of order 10 , respectively; the geometric and/or algebraic interpretations of these groups is not important, for us groups purely serve as structured collections of permutations [9].
Next, we train 10 agents with the C 5 -OP learning rule (see Section 3), then map these 10 agents to equivariant ones using the C 5 -symmetrizer.
Similarly, we train 10 agents with the D 10 -OP learning rule, then map them with the D 10 -symmetrizer.
Finally, we compare the ZSC performance between these three pools of 10 agents.
This is an ablation, because we control for the order of the group by considering both C 5 and D 10 (in fact, C 5 is a subgroup of D 10 ) so to observe the effect of group order.
Each agent uses Recurrent Replay Distributed Deep Q-Networks (R2D2)
[25], and is trained using value decomposition networks (VDN) [43] and the simpliﬁed action decoding (SAD) algorithm [21].
The average self-play scores achieved by the OP agents is 23 .
93 ± 0 .
02 , of the C 5 -equivariant agents is 23 .
96 ± 0 .
03 , and the D 10 -equivariant agents is 23 .
95 ± 0 . 03 .
Table 1 shows the ﬁndings of our ablation analysis.
We see that both equivariant pools outperform the OP agents, in spite of the OP agents being trained over far more permutations than any of our equivariant agents: | C 5 | = 5 , | D 10 | = 10 , | S 5 | = 120 (the OP agents are trained on permutations from S 5 ).
This is consistent with earlier work where equivariant networks outperform data augmentation approaches [36, 47, 49, 50, 52, 55].
We conduct a Monte Carlo permutation test [15, 16] to compare the D 10 -equivariant agent with the OP agents, for which we bound the derived p -value in a 99% binomial conﬁdence interval.
We also observe that the D 10 -equivariant agents outperform the C 5 -equivariant agents, which we largely attribute to C 5 < D 10 and Propositions 3 and 4: if we have a group G <  , and choose  2  uniformly at random, then P (  2 G ) =
| G | |  | , and so for two groups G, H with | G |  | H | , we have P (  2 G )  P (  2 H ) .
4.2 Equivariance as a Policy-Improvement Operator
Here we consider the extent to which test-time symmetrizing improves ZSC performance across diverse policy types: we train 10 IQL agents [44], and use the 12 OP agents and 12 SAD agents from [22] and the 5 agents from [23] (level 5 ), and examine the effect of transforming each of these self-play trained agents with our equivariant architecture at test time.
This selection of agent types is diverse in terms of both overall performance level as well as the style of play.
Furthermore, we consider agents that are both trained for cross-play (OP and OBL) and trained for the self-play setting (SAD and IQL), which is an important dimension to account for in consideration of EQC’s applicability.
All agents use R2D2 [25], and the SAD and OP agents also use VDN [43].
The average self-play scores achieved by the SAD agents is 23 .
97 ± 0 .
04 , the IQL agents is 23 .
15 ± 0 .
02 , the OP agents is 23 .
93 ± 0 .
02 , and the OBL agents is 24 .
20 ± 0 .
Table 2 summarizes our ﬁndings over the policy types, where we consider symmetrization with both the cyclic group of order 5 , C 5 , and the dihedral group of order 10 , D 10 .
We ﬁnd that mapping any of these networks through EQC with either choice of group improves its cross-play ability with novel agents at test time.
We conduct Monte Carlo permutation tests [15, 16] to compare the
D 10 -symmetrized agents with their unsymmetrized counterparts for each policy type, for which we bound each derived p -value in a 99% binomial conﬁdence interval.
The p -values for the respective one-tailed tests for each of the SAD, IQL, OP and OBL agents are found to be within (0 , 0 . 00028] , (0 , 0 . 00028] , (0 , 0 . 00028] , and [0 . 0825 , 0 . 0973] respectively, making the differences signiﬁcant at
[Table]
the level ↵ = 0 . 01 for the SAD, IQL [Table] and OP agents
[Table]
and at the level ↵ = 0 .
1 for the OBL agents.
Therefore, the improvement in coordination found by symmetrizing agents is highly statistically signiﬁcant.
In particular, we see that OP agents improve in ZSC when symmetrized.
In the proof for Proposition 2, we showed that equivariant networks are ﬁxed under the symmetrizer.
Since these OP agents were trained under the same permutations they are being symmetrized with, yet still improve upon symmetrization, this suggests that the data augmentation of OP does not lead to full equivariance, and so encoding environmental symmetry at the model level is necessary to realise crucial properties such as Propositions 3 and 4, as well as to improve overall performance.
It is especially signiﬁcant that EQC maps networks not trained with permutations (SAD, IQL, OBL) to ones that are overall better at ZSC; we justify the mechanism by which it accomplishes this in
Section 3.
In particular, we demonstrate that EQC can improve on OBL, the state of the art for zero-shot coordination on the Hanabi benchmark at the time of writing.
Figure 2 shows the conditional action matrices (i.e. P ( a it | a jt  1 ) ) of a symmetrized and unsymmetrized IQL agent (see the Appendix for the plots of other agents).
This plot informs us on how differently the agent responds to possible actions of its partner, where large differences indicate specialized play, and therefore likely discrepancy between the conventions used by this agent and potential partners, which would be detrimental to cross-play.
We see that the IQL agent has learned specialized conventions (for example, hinting the fourth color means for the partner to discard their ﬁrst card), but that upon symmetrization its actions and responses become more consistent.
Another observation is that the D 10 -symmetrizer does not always outperform the C 5 -symmetrizer: namely for the case of the SAD agents in terms of bombout rate.
The combination of C 5 < D 10 and
Propositions 3 and 4 implies that the D 10 -symmetrizer corrects for symmetry breaking better than the
C 5 does, so the worse bombout rate may only be attributed to the kind of play the symmetrized policy encourages, with a likely explanation as follows: if a policy is such that it acts very differently in each symmetry, then symmetrizing this policy over a large number of symmetry permutations will result in a policy with higher entropy (i.e. averaging over a large number of very different distributions results in one that is approximately uniform (see the Appendix)).
But behaving “randomly” is not in general conducive to effective coordination, where one should try to use unambiguous conventions to facilitate play.
Thus, when symmetrizing an arbitrary network, we tacitly assume some degree of regularity of the network across symmetries (we control this regularity when we opt for the proposed
G -OP learning rule).
Nonetheless, both choices of C 5 and D 10 result in better ZSC than without any symmetrization for all the tested policy types (even with SAD, which learns notoriously specialized play [22, 23]), suggesting that policies in practice do generally satisfy these regularity requirements.
This phenomenon, however, should be kept in mind and motivates future research into deriving the optimal choice of G for G -equivariant agents.
Related Work
Zero-Shot Coordination.
Zero-shot coordination is the problem of constructing independent agents that can effectively engage with one another in cross-play at test time.
Many directions to this problem have been considered: [14, 23] explore controlling the cognitive-reasoning depth so to avoid formation of arbitrary conventions that can complicate cross-play.
However, [14] is computationally exacting, requiring days of GPU running time, and [23] requires simulator (environment) access.
[38] offers a modular approach to separately learn rule-dependent and convention-dependent behaviors to facilitate coordination, but their method comes at the expense of scalability, with experiments limited to bandit problems and 1-color Hanabi.
[34] uses repeated eigenvalues of the Hessian during training to ﬁnd policies suited for ZSC, but is again inhibited by scalability, with experiments focused on lever games.
[31] investigate using life-long learning for ZSC, but their approach requires access to a pool of pre-trained policies.
[22], which we have referred to at length throughout this paper, takes to data augmentation to train symmetry-robust agents.
EQC can be used along with any of these approaches (even with [22, 23], as we demonstrated in Section 4), so to correct for symmetry breaking and to encourage robust play.
Ad-Hoc Teamwork.
Similar to zero-shot coordination is the ad-hoc teamwork setting [6, 30, 41], where an agent is pitted with an existing group of agents, and is tasked with learning to coordinate with this group during interaction.
While ZSC and ad-hoc teamwork are closely related problems, they differ in that ad-hoc team play seeks policies that do well with arbitrary agents (often an undeﬁned problem setting), while ZSC agents only need to perform well with agents that are optimized for ZSC using a common high-level approach.
In principle EQC can also be used to improve performance in ad-hoc teamplay, which we leave for future work.
Equivariance.
Equivariant networks are neural networks with symmetry constraints built in, such that for a transformation of the input, the output is appropriately transformed as well [3, 11, 46, 49, 52].
Equivariant policies have shown improved data efﬁciency in single agent reinforcement learning [1, 29, 33, 35, 40, 47, 54].
In the multi-agent case, prior works use permutation symmetries to ensure agent anonymity [8, 24, 27, 42, 43] or rotational equivariance over multiple agents [36].
While these approaches consider equivariance in multi-agent settings, they do not tackle the problem of ZSC.
In terms of architecture, the symmetry ensemble of [39] is related to ours, but we propose adaptations that tailor recurrent layers to multi-agent coordination, thus suiting our design not just for
Markov Decision Processes (MDPs) [7] but for more general POMDPs [4] and Dec-POMDPs.
Conclusion
In this work we proposed equivariant modelling to solve symmetry breaking.
To this aim we presented EQC, which we validated via mathematical proof and experimentation over the complex
Dec-POMDP task of Hanabi.
We showed that EQC greatly improves on [22] by guaranteeing symmetry-equivariance of policies, and can be used as a policy-improvement operator to prevent symmetry breaking and promote robust play for a diversity of agents.
In this way, we also showed the extent to which symmetrizing improves overall performance in coordination.
Furthermore, we showed that EQC is “symmetry efﬁcient”, such that it does not require access to all environment symmetries in order to perform well.
In addition, we empirically validated that agents perform well under our proposed architectural choices.
There are many directions for future work.
One is to consider how to derive optimal choices of the group G for G -equivariant agents effective at ZSC, relative to a given policy type and task.
Another direction is exploring how to efﬁciently uncover environment symmetries from a domain when they are not given nor assumed to be known.
While our work focused on symmetry breaking, future work could explore other fundamental aspects of coordination.
Acknowledgements
The experiments were made possible by a generous equipment grant from NVIDIA.
Christian
Schroeder de Witt is generously funded by the Cooperative AI Foundation.
We thank Jacob Beck for his constructive criticism of the manuscript.