Equivariant Networks for Zero-Shot Coordination
Anonymous Author s)
AffiliationAddress email
Abstract
Successful coordination in Dec-POMDPs requires agents to adopt robust strategiesand interpretable styles of play for their partner.
A common failure mode is symmetry breaking , when agents arbitrarily converge on one out of many equivalentbut mutually incompatible policies.
Commonly these examples include partialobservability, e.g. waving your right hand vs. left hand to convey a covert message.
In this paper, we present a novel equivariant network architecture for usein Dec-POMDPs that prevents the agent from learning policies which break symmetries, doing so more effectively than prior methods.
Our method also acts asa “coordination-improvement operator” for generic, pre-trained policies, and thusmay be applied at test-time in conjunction with any self-play algorithm.
We providetheoretical guarantees of our work and test on the AI benchmark task of Hanabi,where we demonstrate our methods outperforming other symmetry-aware baselinesin zero-shot coordination, as well as able to improve the coordination ability of avariety of pre-trained policies.
Introduction
A popular method for learning strategies in partially-observable cooperative Markov games is viaself-play (SP), where a joint policy controls all players during training and at test time [19, 34, 42, 50].
While SP can yield highly effective strategies, these strategies often fail in zero-shot coordination(ZSC), where independently trained strategies are paired together one time.
A common cause forthis failure is mutually incompatible symmetry breaking (e g. signalling 0 to refer to a “cat” vs. a“dog”).
Specifically, this symmetry breaking results in policies that act differently in situations whichare equivalent with respect to the symmetries of the environment [20, 35].
To address this we are interested in devising equivariant strategies, which we illustrate in Figure1.
Equivariant policies are such that symmetric changes to their observation cause a correspondingchange to their output.
In doing so, we fundamentally prevent the agent from breaking symmetriesover the course of training.
In earlier work, the Other-Play (OP) learning algorithm [20] addressed symmetry in this setting bytraining agents to maximize expected return when randomly matched with symmetry-equivalentpolicies of their training time partners.
This learning rule is implemented by independently applyinga random symmetry permutation to the action-observation history of each agent during each episode.
OP, however, has some pitfalls, namely: 1) it does not guarantee equivariance; 2) it poses the burdenof learning environmental symmetry onto the agent; and 3) it is only applicable during training timeand cannot be used as a policy-improvement operator
To address all of this, our approach is to encode symmetry not in the data, but in the network itself .
There are many approaches to building equivariant networks in the literature.
Most work focuseson analytical derivations of equivariant layers [11, 46, 48, 49], which can prove time-consumingto engineer for each problem instance and inhibits rapid prototyping.
Algorithmic approaches to constructing equivariant networks are restricted to layer-wise constraints [10, 15, 16, 32], whichrequire computational overhead to enforce.
Furthermore, for complex problems such as ZSC inchallenging benchmarks such as Hanabi [5], these approaches are insufficient because 1) they are notscalable; and 2) they usually do not allow for recurrent layers, which are the standard method forencoding action-observation histories.
In this paper, we propose the equivariant coordinator (EQC), which uses our novel, scalable approachtowards equivariant modelling in Dec-POMDPs.
Specifically, our main contributions are:
Our method, EQC, that mathematically guarantees symmetry-equivariance of multi-agentpolicies, and can also be applied solely at test time as a coordination-improvement operator;and2.
Showing that EQC outperforms prior symmetry-robust baselines on the AI benchmarkHanabi.
Background
This section formalises the problem setting (2.1), provides a brief primer to some group-theoreticnotions (2.2), introduces equivariance (2.3), provides background to prior multi-agent work onsymmetry (2.4), and offers a small, concrete example to help tie these notions together and facilitateintuition (2.5).
2.1 Dec-POMDPs
We formalise the cooperative multi-agent setting as a decentralized partially-observable Markovdecision process (Dec-POMDP) [29] which is a 9-tuple ( S , N , {A i } ni =1 , {O i } ni =1 , T , R , {U i } ni =1 ,
T, γ ), for finite sets S , N , {A i } ni =1 , {O i } ni =1 , denoting the set of states, agents, actions and observations, respectively, where a superscript i denotes the set pertaining to agent i ∈ N = { 1 , . . . , n } (i.e.,
A i and O i are the action and observation sets for agent i , and
a i ∈ A i and o i ∈
O i are a specificaction and observation of agent i ).
We also write A = ×
i A
i and O =
× i
O i , the sets of joint actionsand observations, respectively.
s t ∈ S is the state at time t and s t = { s kt } k , where s kt is state featurek of s t .
a t ∈ A is the joint action of all agents taken at time t , which changes the state according tothe transition distribution s t
+1 ∼ T ( s t +1 | s t , a t ) .
The subsequent joint observation of the agentsis
o t +1 ∈ O , distributed according to o t +1 ∼ U
( o t +1 | s t +1 , a t ) , where U
=
× i
U i ; observationfeatures of o it +
∈
O i are notated analogously to state features; that is, o it +1 =
{ o i,kt +1 } k .
The reward r t +1 ∈ R is distributed according to r t +1 ∼ R ( r t +1 | s t +1 , a t ) .
T is the horizon and γ ∈ [0 , 1] isthe discount factor.
Notating τ it =
( a i 0 , o i 1 , . .
. , a it − 1 , o it ) for the action-observation history of agent i , agent i actsaccording to a policy a it ∼ π
i ( a it | τ it ) .
The agents seek to maximize the return, i.e., the expecteddiscounted sum of rewards:
[Equation]
[ (cid:88)
[Equation]
(1) where τ t = ( s 0 , a 0 , o 1 , r 1 , . . .
, a t − 1 , o t , r t , s t ) is the trajectory until time t .
2.2 Groups
A group G is a set with an associative binary operation on G , such that, relative to the binary operation,the inverse of each element in the set is also in the set, and the set contains an identity element.
Forour purposes, each element g ∈ G can be thought of as an automorphism over some space X (i.e. anisomorphism from X to X ), and the binary operation is function composition.
The cardinality of thegroup is referred to as its order.
If a subset K of G is also a group under the binary operation of G(a.k.a.
a subgroup), we denote it K < G .
A group action 1 is a function G × X
→ X satisfying ex = x (where e is the identity element) and( g · h ) x
= g · ( hx ) , for all g, h ∈ G .
In particular, for a fixed g , we get a function α g : X →
X givenby x (cid:55)→ gx .
g can be thought to be “relabeling” the elements of X , which aligns with our definitionfor Dec-POMDP symmetry introduced Section 2.4.
Groups have specific mathematical properties, such as associativity, containing the inverse of eachelement, and closure (closure is implicit of the binary operation being over G itself) which arenecessary for proving Propositions 1 and 2 and ensuring equivariance.
2.3 Equivariance
Let G be a group.
Consider the set of all neural networks Ψ of a given architecture whose first andultimate layers are linear; that is, Ψ :=
{ ψ | ψ = h ( ˆ ψ ( f )) } , where f, h are linear functions, andˆ ψ is a non-linear function.
We say a network ψ ∈ Ψ is equivariant (with respect to a group G , or
G -equivariant)
if
[Equation]
where L g is the group action of g ∈ G on inputs, K g is similarly the group action on outputs, andd is the dimension of our data.
If ψ satisfies Equation 2, then we say ψ ∈ Ψ equiv .
In the context ofreinforcement learning, one may consider ψ as the neural parameterization of a policy π 2 , so a policyis G -equivariant if
[Equation]
If for any choice of g ∈ G we have that K g
=
I , the identity function/matrix, then we say π (or ψ ) isinvariant to g .
2.4 Other-Play and Dec-POMDP Symmetry
The OP learning rule [20] of a joint policy, π = ( π 1 , π 2 ) , is formally the maximization of thefollowing objective (which we state for the two-player case for ease of notation):
[Equation]
(4) where Φ is the class of equivalence mappings (symmetries) for the given Dec-POMDP, so
that eachϕ ∈
Φ is an automorphism of S , A , O whose application leaves the Dec-POMDP unchanged upto relabeling ( ϕ is shorthand for ϕ
= { ϕ S , ϕ A , ϕ O } ).
The expectation in Equation 4 is taken withrespect to the uniform distribution on Φ .
Each ϕ ∈ Φ acts on the action-observation history as
[Equation]
and acts on policies as
[Equation]
Policies π, ˆ π in Equation 6 are said to be symmetry-equivalent to one another with respect to ϕ .
Tocorrespond with Equation 3, we have L ϕ =
{ ϕ A , ϕ O } and K ϕ =
ϕ A .
2.5 Idealized Example
Consider a game where you coordinate with an unknown stranger.
Each of you must pick a leverfrom a set of 10 levers.
Nine of the levers have 1 written on them, and one has 0 .
9 written.
If youand the stranger pick the same lever then you are paid out the amount that is written on the lever,otherwise you are paid out nothing.
We have G :=
S 9 ≤ Φ , where S 9 is the symmetric group over 9 elements (the number of ways wecan permute 9 elements), since the nine levers with 1 written on them are symmetric.
If you (usingpolicy π 1 ) pick the 1 -point lever l 1 and the stranger (using policy π 2 ) picks the 1 -point lever l 2 , thenπ 1 and π 2 would be symmetry-equivalent with respect to any ϕ ∈ G that permutes the lever choices(e.g.
ϕ ( l 1 ) = l 2 , or equivalently L ϕ ( l 1 ) =
K ϕ ( l 1 ) =
l 2 , as the levers are both observations andactions).
If l 1 ̸ = l 2 , which is likely since no communication is possible, you and the stranger leaveempty-handed, illustrating symmetry breaking.
The policy π equiv that picks the 0 .
9 -point lever is G -equivariant (and in particular, it is invariant to all ϕ ∈ G ), and so the choice of using (deterministic)equivariant policies maintains symmetry and guarantees payout.
Method
We now present our methodology and architectural choices, as well as several theoretical results.
We first consider the symmetrizer introduced in [32], which is a functional that transforms linearmaps to equivariant linear maps.
Inspired by this we present a generalisation of the symmetrizer thatmaps from Ψ to the equivariant subspace, Ψ equiv , namely S : Ψ
→ Ψ equiv , which we define as
[Equation]
so that the first layer of ψ , denoted f , being linear, is composed with each L g , and the ultimatelayer, denoted h , is similarly composed with K − 1 g (i.e. K − 1 g ψ ( L g )
=
K − 1 g ◦ h ( ˆ ψ ( f ◦ L g )) , whereψ =
h ( ˆ ψ ( f ) ).
Below we establish properties of S that were analogously shown for the linear symmetrizer case [32]:
Proposition 1.
(Symmetric Property) S ( ψ )
∈
Ψ equiv , for all ψ ∈ Ψ ; that is, S maps neural networksto equivariant neural networks.
Proposition 2.
(Fixing Property) Ψ equiv =
Ran ( S ); that is, the range of S covers the entire equivariant subspace.
The proofs of these propositions are in the Appendix.
The proof of Proposition 1 relies on propertiesof the group structure (such as closure of the group); it is thus critical for the symmetrizer to sum overpermutations that together form a group structure, and not any random collection of permutations.
Propositions 1 and 2 show in combination that the architecture of S ( ψ ) is indeed equivariant, andthat optimizing over the class Ran ( S ) (that is, keeping the permutation matrices L g and K − 1 g for allg frozen while updating the other weights) is equivalent to optimizing over Ψ equiv .
The sum overeach g ∈ G of S ( ψ ) is highly parallelizable [17], which allows for maintaining tractability.
In thisway we algorithmically enforce equivariance, mitigating the need to design equivariant layers byhand, as is typical [11, 12, 45, 46, 48, 49].
To make our symmetrizer S compatible with networks containing LSTM [18] layers and to encourage
S to map to policies suitable for coordination, we propose the following: for each hidden state h t,g and cell state c t,g produced by the LSTM layer over input ( L g ( τ ) , h t − 1 , c t − 1 ) , we computeh t = 1 | G | (cid:80)
g ∈ G h t,g and c t = 1 | G | (cid:80) g ∈ G c t,g , and proceed inductively.
This is a non-obvious designchoice, since each h t,g and c t,g is the output of a non-linear function; a more obvious choice wouldbe to choose h t :
= h t,e and c t :=
c t,e where e ∈ G is the identity element, but to ensure symmetryaware play we experimented with this averaging scheme to “symmetrically adjust” the short andlong-term memories.
Since all the permutations over the input are computed in parallel with matrixmultiplication, this design choice retains tractability.
We test this choice empirically in Section 4.
The following propositions provide guarantees for EQC’s ability to prevent all symmetry breaking:
Proposition 3.
If π 1 and π 2 are symmetry-equivalent policies with respect to ϕ ∈ G and π 1 is
G -equivariant, then π 1 = π 2 .
Proof.
We have π 1 = ϕ
( π 2 ) for some ϕ ∈ G , meaning π 1 ( ϕ − 1 ( a ) | ϕ
( τ ))
=
π 2 ( a | τ ) .
Identifying
L ϕ = { ϕ A , ϕ O } and K ϕ = ϕ
A , by equivariance (Equation 3) we have π 1 ( a | τ ) =
π 2 ( a | τ ) .
Proposition 4.
If π 1 and π 2 are symmetry-equivalent policies with respect to ϕ ∈ G , then S ( π 1 ) =
S ( π 2 ) .
Proof.
This is a corollary of the proof of Proposition 1, such that the symmetrizer architecture isfixed under any ϕ ∈ G .
G -equivariance thus reduces cross-play to self-play between policies that are symmetry-equivalentwith respect to any ϕ ∈ G .
This is critical, as this means equivariance merges policies of large classestogether, such that symmetry-equivalent policies now indeed become equivalent in coordination.
Putanother way, the set of policies that are symmetry-equivalent with respect to any ϕ ∈ G form anequivalence class, and G -equivariance reduces the cardinality of each equivalence class to 1 , thusallowing fluent coordination of policies within an equivalence class (as it merely becomes self-play).
Based on Propositions 1 and 2, a first approach to obtaining a G -equivariant agent, for a given group G , might then be as follows: 1) initialize a random seed ψ ∈ Ψ ; 2) map ψ to S ( ψ ) ; 3) train S ( ψ )using any self-play algorithm, updating the weights while keeping the permutation matrices L g and K − 1 g for all g frozen; 4) S ( ψ ) is ready for inference.
This first approach is sound, but time andmemory complexity inhibit scaling to large choices of G .
For this reason we present the followingtwo options (that together summarize EQC):
As a first option, we introduce “ G -OP”, where during training we sample random subsets of
G for each mini-batch (i
e. we select a random g ∈ G and update the weights of K − 1 g ψ ( L g ) , keeping the permutation matrices frozen) and at inference time, we use all the permutations
of G by deploying S ( ψ ) .
As a second option, we train under any self-play algorithm and at inference time deploy
S ( ψ ) .
With Option 1 we train over the permutations of G at training time.
With Option 2 we train understandard SP and use S ( · ) to obtain S ( ψ ) at inference time, which can be viewed as a coordinationimprovement operator.
We empirically validate both options in Section 4.
From here on, we refer to
S ( ψ ) as “symmetrizing ψ ”.
We expect compatibility between the G -OP trained agents and the symmetrizer at inference time,since at training time, these agents were trained over these permutations.
For Option 2, where wehave a network ϕ not trained with permutations, it is worthwhile to inspect S ( ϕ ) ; while we know
S ( ϕ ) is equivariant, we know little else regarding the conventions and style of play it would use.
While symmetry is a critical component of ZSC, other factors (such as choosing robust actions thatare likely to succeed under minimal assumptions [2, 13]) are also significant.
The following pointsjustify why S maps any network to one that is not only equivariant, but overall stronger at ZSC: 1)the network averages over how ψ would act under each g ∈ G , thus ensuring play that accounts formultiple symmetries; 2) the group closure property fixes the architecture under any permutation inthe group (see the proof of Proposition 1), forcing consistent play across different symmetries.
Weverify the empirical efficacy of Options 1 and 2 in Section 4.
[20] proved that if both agents use OP as their learning rule then that is the best possible metaequilibrium, where a meta-equilibrium is a learning rule for each agent such that neither agent canimprove their expected payoff by unilaterally deviating to a different learning rule.
We have theanalogous result for our equivariant architecture: Proposition 5. Assume G = Φ . If an agent chooses either the first approach or the G -OP learningrule, then choosing either of these is payoff maximizing for the agent’s partner.
In addition, bothplayers choosing either the first approach or the G -OP learning rule is the best possible metaequilibrium.
The proof of this proposition is in the Appendix.
Proposition 5 demonstrates that the equivarianceconstraint is in a sense stronger than the OP constraint.
Our work improves on [32] in many ways.
We improve on scalability by avoiding computing anequivariant basis; this is done by keeping the permutation matrices as frozen layers and training overthe rest of the network (Option 1), or simply by training as normal then using S ( ψ ) at inference(Option 2).
Computing an equivariant basis of Ψ equiv is infeasible for a couple of reasons: 1) Ψ equiv isincredibly high-dimensional for deep, neural architectures; 2) our basis elements would be non-linear,and so cannot be computed in parallel as in [32] (i.e., by adding the basis element weights andcomputing one pass over the input).
What remains is the choice of the group G for our G -equivariant agents.
One might consider thechoice of G =
Φ , but this in general scales factorially with the size of the Dec-POMDP (e.g. theautomorphism group over an environment with m symmetric features grows asymptotically withthe number of permutations over m elements), and so quickly becomes infeasible.
Hence, we treat
G as a hyperparameter, and in Section 4 we analyze how different choices of G affect ZSC.
In anycase, picking G as a subgroup of Φ may be motivated in and of itself, when enumerating all possiblesymmetries of an environment is intractable, or if an environment is obfuscated such that somesymmetries cannot be known (as is the case in many large, realistic domains).
In this way, our methodrelaxes the assumption of requiring access to all environment symmetries, improving on [20].
Experiments
The primary test bed for our methodology is the AI benchmark task Hanabi [5].
Hanabi is a uniqueand challenging card game that requires agents to formulate informative implicit conventions inorder to be successful.
Hanabi has served as the primary test bed for many algorithms designed forzero-shot coordination [14, 20, 21, 26] due to its representation as a formidable Dec-POMDP task(see the Appendix for an introduction to Hanabi).
To make sense of the ensuing results, recall that a perfect score in Hanabi is 25 , and that “bombingout” refers to expending all three fuse tokens and ending the game with a score of 0 .
Agents thusmust be cautious not to expend all fuse tokens.
In Hanabi, the class of environment symmetries are the permutations of the card colors, becauseassuming there is no supplemental information, a permutation of the colors of the cards leaves thegame unchanged.
We therefore have Φ = S 5 , the symmetric group over 5 elements (the number ofways we can permute 5 elements).
5 here represents the number of different colors.
| S 5 | = 5!
= 120 .
In this section we analyze differences in play and the relationship to environmental symmetry betweenour equivariant agents and OP agents (4.1) and evaluate various pre-trained agents before and aftersymmetrization (4.2).
These subsections evaluate Options 1 and 2, respectively (see Section 3).
For further details of the training setup and hyperparameters used in the following experiments, pleaserefer to the Appendix.
4.1 Symmetry-Robust Agents
Here we compare OP agents with our equivariant agents.
Specifically, we conduct the followingablation study: first, we train 10 agents with the OP learning rule.
We then fix two subgroups of S 5 : C 5 and D 10 , which are the cyclic group of order 5 and the dihedral group of order 10 , respectively; thegeometric and/or algebraic interpretations of these groups is not important, for us groups purely serveas structured collections of {+permutations [9]. Next, we train 10 agents with the C 
 -OP learning rule+}
[Table] [Table] [Table] (see Section 3), then map these 10 agents to equivariant ones using the C 5 -symmetrizer.
Similarly, wetrain 10 agents with the D 10 -OP learning rule, then map them with the D 10 -symmetrizer.
Finally, wecompare the ZSC performance between these three pools of 10 agents.
This is an ablation, becausewe control for the order of the group by considering both C 5 and D 10 (in fact, C 5 is a subgroup of D 10 ) so to observe the effect of group order.
Each agent uses Recurrent Replay Distributed Deep
Q-Networks (R2D2) [23], and is trained using value decomposition networks (VDN) [40] and thesimplified action decoding (SAD) algorithm
The average self-play scores achieved by the OPagents is 22 .
71 ± 0 .
03 , of the C 5 -equivariant agents is 22 .
91 ± 0 .
02 , and the D 10 -equivariant agentsis 22 .
82 ± 0 .
Table 1 shows the findings of our ablation analysis.
We see that both equivariant pools outperformthe OP agents, in spite of the OP agents being trained over far more permutations than any of ourequivariant agents: | C 5 | = 5 , | D 10 | = 10 , | S 5 | = 120 .
This is consistent with earlier work whereequivariant networks outperform data augmentation approaches [33, 44, 46, 47, 49, 52].
We also observe that the D 10 -equivariant agents outperform the C 5 -equivariant agents, which welargely attribute to C 5 < D 10 and Propositions 3 and 4: if we have a group G < Φ , and chooseϕ ∈ Φ uniformly at random, then P ( ϕ ∈ G ) =
| G | | Φ | , and so for two groups G, H with | G | ≤ | H | , wehave P ( ϕ ∈ G ) ≤ P ( ϕ ∈ H ) .
Future work should explore whether, for a given domain, the marginalbenefit eventually decreases as group order increases (our work already suggests this is likely), andfinding the choice of G that maximizes cooperative performance while maintaining tractability.
4.2 Equivariance as a Policy-Improvement Operator
Here we consider the extent to which test-time symmetrizing improves ZSC performance acrossdiverse policy types: we train 10 IQL agents [41], 10 SAD agents [19], and 10 OP agents [20] (same OP agents as in Section 4.1), and examine the effect of transforming each of these self-play trainedagents with our equivariant architecture at inference time.
This selection of agent types is diversein terms of both overall performance level as well as the style of play.
Furthermore, we consideragents that are both trained for cooperation (OP) and trained for the self-play setting (SAD and IQL),which is an important dimension to account for in consideration of EQC’s applicability.
All agentsuse R2D2 [23], the SAD and OP agents also use VDN [40].
The average self-play scores achievedby the SAD agents is 23 .
07 ± 0 .
03 , that of the IQL agents is 22 .
96 ± 0 .
02 , and the OP agents is22 .
71 ± 0 . 03 .
Table 2 summarizes our findings over the policy types, where we consider symmetrization with boththe cyclic group of order 5 , C 5 , and the dihedral group of order 10 , D 10 .
We find that mapping anyof these networks through EQC with either choice of group improves its cross-play ability with novelagents at test time.
In particular, we see that OP agents improve in ZSC when symmetrized.
In the proof for Proposition2, we showed that equivariant networks are fixed under the symmetrizer.
Since these OP agentswere trained under the same permutations they are being symmetrized with, yet still improve uponsymmetrization, this suggests that the data augmentation of OP does not lead to full equivariance,and so encoding environmental symmetry at the model level is necessary to realise crucial propertiessuch as Propositions 3 and 4, as well as to improve overall performance.
Nonetheless, we note thatthe improvement gained by symmetrizing OP is much less than by the other policies, which alignswith our expectations since OP policies are already relatively symmetry-robust.
It is especially significant that EQC maps networks not trained with permutations (SAD, IQL) to onesthat are overall better at ZSC; we justify the mechanism by which it accomplishes this in Section 3.
Figure 2 shows the conditional action matrices (i.e. P ( a it | a jt − 1 ) ) of a symmetrized and unsymmetrized IQL agent (see the Appendix for the plots of other agents).
This plot informs us on howdifferently the agent responds to possible actions of its partner, where large differences indicatespecialized play, and therefore likely discrepancy between the conventions used by this agent andpotential partners, which would be detrimental to cross-play.
We see that the IQL agent has learnedspecialized conventions (for example, hinting the fourth color means for the partner to discard theirfirst card), but that upon symmetrization its actions and responses become more consistent.
Another observation is that the D 10 -symmetrizer does not always outperform the C 5 -symmetrizer:namely for the case of the SAD agents in terms of bombout rate.
The combination of C 5 < D 10 and
Propositions 3 and 4 implies that the D 10 -symmetrizer corrects for symmetry breaking better than the
C 5 does, so the worse bombout rate may only be attributed to the kind of play the symmetrized policyencourages, with a likely explanation as follows: if a policy is such that it acts very differently in eachsymmetry, then symmetrizing this policy over a large number of symmetry permutations will resultin a policy with higher entropy (i.e. averaging over a large number of very different distributionsresults in one that is approximately uniform (see the Appendix)).
But behaving “randomly” is not ingeneral conducive to effective coordination, where one should try to use unambiguous conventionsto facilitate play.
Thus, when symmetrizing an arbitrary network, we tacitly assume some degree ofregularity of the network across symmetries (we control this regularity when we opt for the proposed
G -OP learning rule).
Nonetheless, both choices of C 5 and D 10 result in better ZSC than without anysymmetrization for all the tested policy types (even with SAD, which learns notoriously specializedplay [20, 21]), suggesting that policies in practice do generally satisfy these regularity requirements.
This phenomenon, however, should be kept in mind and further motivates future research into derivingthe optimal choice of G for G -equivariant agents.
Related Work
Zero-Shot Coordination.
Zero-shot coordination is the problem of constructing independent agentsthat can effectively engage with one another in cross-play at test time.
Many directions to thisproblem have been considered: [14, 21] explore controlling the cognitive-reasoning depth so to avoidformation of arbitrary conventions that can complicate cross-play.
However, [14] is computationallyexacting, requiring days of GPU running time, and [21] requires simulator access.
[35] offers amodular approach to separately learn rule-dependent and convention-dependent behaviors to facilitatecoordination, but their method comes at the expense of scalability, with experiments limited tobandit problems and 1-color Hanabi.
[31] uses repeated eigenvalues of the Hessian during trainingto find policies suited for ZSC, but is again inhibited by scalability, with experiments focused onlever games.
[28] investigate using life-long learning for ZSC, but their approach requires accessto a pool of pre-trained policies.
[20], which we have referred to at length throughout this paper,takes to data augmentation to train symmetry-robust agents.
EQC can be used along with any ofthese approaches (even with [20], as we demonstrated in Section 4), so to correct for symmetrybreaking and to encourage robust play.
Specifically, future work that considers the intersection of ourequivariant methods with directions such as [14, 21] should be promising.
Ad-Hoc Teamwork.
Similar to zero-shot coordination is the ad-hoc teamwork setting [6, 38], wherean agent is pitted with an existing group of agents, and is tasked with learning to coordinate withthis group during interaction.
While ZSC and ad-hoc teamwork are closely related problems, theydiffer in that ad-hoc team play seeks policies that do well with arbitrary agents (often an undefinedproblem setting), while ZSC agents only need to perform well with agents that are optimized for ZSCusing a common high-level approach.
In principle EQC can also be used to improve performance inad-hoc teamplay, which we leave for future work.
Equivariance.
Equivariant networks are neural networks with symmetry constraints built in, suchthat for a transformation of the input, the output is appropriately transformed as well [3, 11, 43,46, 49].
Equivariant policies have shown improved data efficiency in single agent reinforcementlearning [1, 27, 30, 32, 37, 44, 51].
In the multi-agent case, prior works use permutation symmetriesto ensure agent anonymity [8, 22, 25, 39, 40] or rotational equivariance over multiple agents [33].
While these approaches consider equivariance in multi-agent settings, they do not tackle the problemof zero-shot coordination.
In terms of architecture, the symmetry ensemble of [36] is related to ours,but we propose adaptations that tailor recurrent layers to our architecture, thus suiting our design notjust for Markov Decision Processes (MDPs) [7] but for general POMDPs [4] and Dec-POMDPs.
Conclusion
In this work we proposed equivariant modelling to solve symmetry breaking.
To this aim wepresented EQC, which we validated via mathematical proof and experimentation over the complex
Dec-POMDP task of Hanabi.
We showed that EQC greatly improves on [20] by guaranteeingsymmetry-equivariance of policies, and can be used to prevent symmetry breaking and promoterobust play for a diversity of agents.
In this way, we also showed the extent to which correcting forsymmetry breaking improves overall performance in coordination.
Furthermore, we showed that EQC is “symmetry efficient”, such that it does not require access to all environment symmetriesin order to perform well.
In addition, we empirically validated that agents perform well under ourproposed architectural choices.
There are many directions for future work.
One is to consider how to derive optimal choices of thegroup G for G -equivariant agents effective at coordination, relative to a given policy type and task;in this work we treated G as a hyperparameter to be tuned.
Another direction is exploring how toefficiently uncover environment symmetries from a domain when they are not given nor assumed tobe known, as our methods assume a priori access to such symmetries.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Sections 3 and
(b) Did you describe the limitations of your work?
[Yes] See Sections 3, 4 and
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See the Appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
In Section 3 and theAppendix3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Section 4, theAppendix, and the supplementary material
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 4 and the Appendix
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See the Appendix
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
In the supplement material
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A] We use rollouts of Hanabi games played by
[Table]
AI agents.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if [Table] applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
[Table]
A Equivariance Property Proofs
Here we write the proofs for the Symmetric and Fixing Properties of our methodology introduced in
Section 3, which are analogous to those of [32]:
Proposition.
(Symmetric Property) S ( ψ ) ∈
Ψ equiv , for all ψ ∈ Ψ ; that is, S maps neural networksto equivariant neural networks.
Proof.
Without loss of generality take g ′ ∈ G and ψ ∈ Ψ .
Then,
[Equation]
Thus K g ′ S ( ψ )( x ) = S ( ψ )( L g ′ x ) .
Proposition.
(Fixing Property) Ψ equiv =
Ran ( S ); that is, the range of S covers the entire equivariantsubspace.
Proof.
By Proposition 1, we have Ran ( S ) ⊂
Ψ equiv .
It thus suffices to show Ψ equiv ⊂
Ran ( S ) .
Takeψ ∈ Ψ equiv .
Then,
[Equation]
where we have used Equation 2 since ψ ∈ Ψ equiv .
Thus, ψ is a fixed point of S , and so ψ ∈
Ran ( S ) .
B Optimal Meta-Equilibrium Proof
Proposition.
Assume G = Φ .
If an agent chooses either the first approach or the G -OP learning rule,then choosing either of these is payoff maximizing for the agent’s partner.
In addition, both playerschoosing either the first approach or the G -OP learning rule is the best possible meta-equilibrium.
Proof.
If an agent π 1 has an architecture π 1 ∈ Ran ( S ) or uses the G -OP learning rule, then theynecessarily have the objective to maximize
[Equation]
because either of these choices forces consideration of all ϕ ∈ G equally.
Since G = Φ , we have
[Equation]
where the expectation is taken with respect to the uniform distribution of Φ .
We have thus derived the
OP objective, and so what we sought so show is a corollary of [20] (Proposition 2 in their paper).
C Averaging Over Very Different Distributions
This passage concerns why if given a policy that acts very differently in different symmetries, whichwe then symmetrize with respect to a large group G , the resulting policy would be one that selectsactions approximately uniformly at random.
Consider l categorical distributions µ 1 , . .
. , µ l sampled uniformly from the collection of all categoricaldistributions over k categories.
Denoting µ i,j as the probability of the j th category of distribution i ,by symmetry in the categories and by assumption we have that µ i,r and µ i,s are equally distributedfor all 1 ≤ r, s ≤ k , 1 ≤ i ≤ l .
Therefore, by the law of large numbers, 1 l (cid:80) li =1 µ i,r ≈ 1 l (cid:80) li =1 µ i,swhen l is large, for all 1 ≤ r, s ≤ k .
But since 1 l (cid:80) li =1 µ i is a probability distribution, this implies1 l (cid:80) li =1 µ i,j → 1 k as l becomes large, for all 1 ≤ j ≤ k ; i.e., 1 l (cid:80) li =1 µ i approaches the uniformdistribution over k values.
The reader may identify l = | G | , k as the number of legal actions, and each µ i as the policy’s actiondistribution under symmetry i , where we took each µ i uniformly at random to mimic a policy thatacts entirely differently per Dec-POMDP symmetry.
While the uniform distribution over actions isindeed symmetry-equivariant (and invariant), it is not in general useful.
D Hanabi
Hanabi is a cooperative card game that can be played with 2 to 5 people.
Hanabi is a popular game,having been crowned the 2013 “Spiel des Jahres” award, a German industry award given to thebest board game of the year.
Hanabi has been proposed as an AI benchmark task to test models ofcooperative play that act under partial information [5].
To date, Hanabi has one of the largest statespaces of all Dec-POMDP benchmarks.
The deck of cards in Hanabi is comprised of five colors (white, yellow, green, blue and red), and fiveranks (1 through 5), where for each color there are three 1’s, two each of 2’s, 3’s and 4’s, and one 5,for a total deck size of fifty cards.
Each player is dealt five cards (or four cards if there are 4 or 5players).
At the start, the players collectively have eight information tokens and three fuse tokens, theuses of which shall be explained presently.
In Hanabi, players can see all other players’ hands but their own.
The goal of the game is to playcards to collectively form five consecutively ordered stacks, one for each color, beginning with a cardof rank 1 and ending with a card of rank 5.
These stacks are referred to as fireworks, as playing thecards in order is meant to draw analogy to setting up a firework display 3 .
We call the player whose turn it is the active agent.
The active agent must conduct one of threeactions:
• Hint - The active agent chooses another player to grant a hint to.
A hint involves the active
agent choosing a color or rank, and revealing to their chosen partner all cards in the partner’s
hand that satisfy the chosen color or rank.
Performing a hint exhausts an information token.
If the players have no information tokens, a hint may not be conducted and the active agent
must either conduct a discard or a play.
• Discard - The active agent chooses one of the cards in their hand to discard.
The identity of
the discarded card is revealed to the active agent and becomes public information.
Discarding
a card replenishes an information token should the players have less than eight.
• Play - The active agent attempts to play one of the cards in their hand.
The identity of
the played card is revealed to the active agent and becomes public information.
The activeagent has played successfully if their played card is the next in the firework of its color to beplayed, and the played card is then added to the sequence.
If a firework is completed, theplayers receive a new information token should they have less than eight.
If the player isunsuccessful, the card is discarded, without replenishment of an information token, and the
players lose a fuse token.
The game ends when all three fuse tokens are spent, when the players successfully complete all fivefireworks,
or when the last
card in
the
deck is drawn and all players take one last
turn.
If the gamefinishes by depletion of all fuse tokens (i.
e.
by “bombing out”), the players receive a score of 0.
Otherwise , the score of the finished game is the sum of the highest card ranks in each firework, for ahighest possible score of 25.
E Experiment Details
Many of the hyperparameter choices are taken from [19].
The main body of the network for eachagent consists of 1 fully connected layer with a hidden dimension of 512, 2 LSTM layers of 512units, and two output heads for value and advantage, respectively.
The networks are updated usingthe Adam optimizer [24] with learning rate 6 .
25 × 10 − 5 and ϵ = 1 . 5 × 10 − 5 .
We use a batchsize of64 for training.
The replay buffer size is 10 5 episodes, and is warmed up with 10 4 episodes beforetraining commences.
We use one GPU for asynchronous actors to collect rollouts and record toan experience replay, and one GPU for computing gradients and model updates.
The trainer sendsits network weights to all actors every 10 updates and the target network is synchronized with theonline network every 2 .
5 × 10 3 updates.
Training was concluded after 10 3 epochs, where each modelreached (or near reached) convergence; each epoch consisted of 10 3 updates.
The compute we usedwere 2 NVIDIA GeForce RTX 2080 Ti GPUs and 40 CPU cores.
Our complete code for training and symmetrizing agents can be found in the supplemental material.
We use the off-belief learning codebase [21] and the Hanabi Learning Environment (HLE).
P1-5 and D1-5 mean playing and discarding the 1st-5th cards in the hand.
We selected a random agent C 5 -equivariant agent and a ranomd D 10 -equivariant agent, and each plot is thereby computed by running 1000 episodes of self-play with each agent to compute the statistics.
G Broader Impact
We have found that equivariant networks can effectively solve symmetry breaking and encourage playsuited for cooperative settings.
No technology is safe from being used for malicious purposes, whichequally applies to our research.
However, fully-cooperative settings target benevolent applications.