Equivariant Networks for Zero-Shot Coordination
Anonymous Author s)
AffiliationAddress email
Abstract
Successful coordination in Dec-POMDPs requires agents to adopt robust strategiesand interpretable styles of play for their partner.
A common failure mode is symmetry breaking , when agents arbitrarily converge on one out of many equivalentbut mutually incompatible policies.
Commonly these examples include partialobservability, e.g. waving your right hand vs. left hand to convey a covert message.
In this paper, we present a novel equivariant network architecture for usein Dec-POMDPs that prevents the agent from learning policies which break symmetries, doing so more effectively than prior methods.
Our method also acts asa “coordination-improvement operator” for generic, pre-trained policies, and thusmay be applied at test-time in conjunction with any self-play algorithm.
We providetheoretical guarantees of our work and test on the AI benchmark task of Hanabi,where we demonstrate our methods outperforming other symmetry-aware baselinesin zero-shot coordination, as well as able to improve the coordination ability of avariety of pre-trained policies.
In particular, we use our method to improve on thestate of the art for zero-shot coordination on the Hanabi benchmark.
Introduction
A popular method for learning strategies in partially-observable cooperative Markov games is viaself-play (SP), where a joint policy controls all players during training and at test time [21, 37, 45, 53].
While SP can yield highly effective strategies, these strategies often fail in zero-shot coordination(ZSC) where independently trained strategies are paired together one time.
A common cause forthis failure is mutually incompatible symmetry breaking (e g. signalling 0 to refer to a “cat” vs. a“dog”).
Specifically, this symmetry breaking results in policies that act differently in situations whichare equivalent with respect to the symmetries of the environment [22, 38].
To address this, we are interested in devising equivariant strategies, which we illustrate in Figure1.
Equivariant policies are such that symmetric changes to their observation cause a correspondingchange to their output.
In doing so, we fundamentally prevent the agent from breaking symmetriesover the course of training.
In earlier work, the Other-Play (OP) learning algorithm [22] addressed symmetry in this setting bytraining agents to maximize expected return when randomly matched with symmetry-equivalentpolicies of their training time partners.
This learning rule is implemented by independently applyinga random symmetry permutation to the action-observation history of each agent during each episode.
OP, however, has some pitfalls, namely: 1) it does not guarantee equivariance; 2) it poses the burdenof learning environmental symmetry onto the agent; and 3) it is only applicable during training timeand cannot be used as a policy-improvement operator
To address all of this, our approach is to encode symmetry not in the data, but in the network itself .
There are many approaches to building equivariant networks in the literature.
Most work focuseson analytical derivations of equivariant layers [11, 49, 51, 52], which can prove time-consuming to engineer for each problem instance and inhibits rapid prototyping.
Algorithmic approaches toconstructing equivariant networks are restricted to layer-wise constraints [10, 17, 18, 35], whichrequire computational overhead to enforce.
Furthermore, for complex problems such as ZSC inchallenging benchmarks such as Hanabi [5], these approaches are insufficient because 1) they are notscalable; and 2) they usually do not allow for recurrent layers, which are the standard method forencoding action-observation histories.
In this paper, we propose the equivariant coordinator (EQC), which uses our novel, scalable approachtowards equivariant modelling in Dec-POMDPs.
Specifically, our main contributions are:
Our method, EQC, that mathematically guarantees symmetry-equivariance of multi-agentpolicies, and can also be applied solely at test time as a coordination-improvement operator;and2.
Showing that EQC outperforms prior symmetry-robust baselines on the AI benchmarkHanabi.
Background
This section formalises the problem setting (2.1), provides a brief primer to some group-theoreticnotions (2.2), introduces equivariance (2.3), provides background to prior multi-agent work onsymmetry (2.4), and offers a small, concrete example to help tie these notions together and facilitateintuition (2.5).
2.1 Dec-POMDPs
We formalise the cooperative multi-agent setting as a decentralized partially-observable Markovdecision process (Dec-POMDP) [32] which is a 9-tuple ( S , N , {A i } ni =1 , {O i } ni =1 , T , R , {U i } ni =1 ,
T, γ ), for finite sets S , N , {A i } ni =1 , {O i } ni =1 , denoting the set of states, agents, actions and observations, respectively, where a superscript i denotes the set pertaining to agent i ∈ N = { 1 , . . . , n } (i.e.,
A i and O i are the action and observation sets for agent i , and
a i ∈ A i and o i ∈
O i are a specificaction and observation of agent i ).
We also write A = ×
i A
i and O =
× i
O i , the sets of joint actionsand observations, respectively.
s t ∈ S is the state at time t and s t = { s kt } k , where s kt is state featurek of s t .
a t ∈ A is the joint action of all agents taken at time t , which changes the state according tothe transition distribution s t
+1 ∼ T ( s t +1 | s t , a t ) .
The subsequent joint observation of the agentsis
o t +1 ∈ O , distributed according to o t +1 ∼ U
( o t +1 | s t +1 , a t ) , where U
=
× i
U i ; observationfeatures of o it +
∈
O i are notated analogously to state features; that is, o it +1 =
{ o i,kt +1 } k .
The reward r t +1 ∈ R is distributed according to r t +1 ∼ R ( r t +1 | s t +1 , a t ) .
T is the horizon and γ ∈ [0 , 1] isthe discount factor.
Notating τ it =
( a i 0 , o i 1 , . .
. , a it − 1 , o it ) for the action-observation history of agent i , agent i actsaccording to a policy a it ∼ π
i ( a it | τ it ) .
The agents seek to maximize the return, i.e., the expecteddiscounted sum of rewards:
[Equation]
[ (cid:88)
[Equation]
(1) where τ t = ( s 0 , a 0 , o 1 , r 1 , . . .
, a t − 1 , o t , r t , s t ) is the trajectory until time t .
2.2 Groups
A group G is a set with an associative binary operation on G , such that, relative to the binary operation,the inverse of each element in the set is also in the set, and the set contains an identity element.
Forour purposes, each element g ∈ G can be thought of as an automorphism over some space X (i.e. anisomorphism from X to X ), and the binary operation is function composition.
The cardinality of thegroup is referred to as its order.
If a subset K of G is also a group under the binary operation of G(a.k.a.
a subgroup), we denote it K < G .
A group action 1 is a function G × X
→ X satisfying ex = x (where e is the identity element) and( g · h ) x
= g · ( hx ) , for all g, h ∈ G .
In particular, for a fixed g , we get a function α g : X →
X givenby x (cid:55)→ gx .
g can be thought to be “relabeling” the elements of X , which aligns with our definitionfor Dec-POMDP symmetry introduced Section 2.4.
Groups have specific mathematical properties, such as associativity, containing the inverse of eachelement, and closure (closure is implicit of the binary operation being over G itself) which arenecessary for proving Propositions 1 and 2 and ensuring equivariance.
2.3 Equivariance
Let G be a group.
Consider the set of all neural networks Ψ of a given architecture whose first andultimate layers are linear; that is, Ψ :=
{ ψ | ψ = h ( ˆ ψ ( f )) } , where f, h are linear functions, andˆ ψ is a non-linear function.
We say a network ψ ∈ Ψ is equivariant (with respect to a group G , or
G -equivariant)
if
[Equation]
where L g is the group action of g ∈ G on inputs, K g is similarly the group action on outputs, andd is the dimension of our data.
If ψ satisfies Equation 2, then we say ψ ∈ Ψ equiv .
In the context ofreinforcement learning, one may consider ψ as the neural parameterization of a policy π 2 , so a policyis G -equivariant if
[Equation]
If for any choice of g ∈ G we have that K g
=
I , the identity function/matrix, then we say π (or ψ ) isinvariant to g .
2.4 Other-Play and Dec-POMDP Symmetry
The OP learning rule [22] of a joint policy, π = ( π 1 , π 2 ) , is formally the maximization of thefollowing objective (which we state for the two-player case for ease of notation):
[Equation]
(4) where Φ is the class of equivalence mappings (symmetries) for the given Dec-POMDP, so
that eachϕ ∈
Φ is an automorphism of S , A , O whose application leaves the Dec-POMDP unchanged upto relabeling ( ϕ is shorthand for ϕ
= { ϕ S , ϕ A , ϕ O } ).
The expectation in Equation 4 is taken withrespect to the uniform distribution on Φ .
Each ϕ ∈ Φ acts on the action-observation history as
[Equation]
and acts on policies as
[Equation]
Policies π, ˆ π in Equation 6 are said to be symmetry-equivalent to one another with respect to ϕ .
Tocorrespond with Equation 3, we have L ϕ =
{ ϕ A , ϕ O } and K ϕ =
ϕ A .
2.5 Idealized Example
Consider a game where you coordinate with an unknown stranger.
Each of you must pick a leverfrom a set of 10 levers.
Nine of the levers have 1 written on them, and one has 0 .
9 written.
If youand the stranger pick the same lever then you are paid out the amount that is written on the lever,otherwise you are paid out nothing.
We have G :=
S 9 ≤ Φ , where S 9 is the symmetric group over 9 elements (all possible ways topermute 9 elements), since the nine levers with 1 written on them are symmetric.
If you (using policyπ 1 ) pick the 1 -point lever l 1 and the stranger (using policy π 2 ) picks the 1 -point lever l 2 , then π 1 andπ 2 would be symmetry-equivalent with respect to any ϕ ∈ G that permutes the lever choices (e.g.ϕ ( l 1 )
= l 2 , or equivalently L ϕ ( l 1 ) =
K ϕ ( l 1 ) =
l 2 , as the levers are both observations and actions).
If, as is likely, l 1 ̸ = l 2 , you and the stranger leave empty-handed, illustrating symmetry breaking.
If during training you and the stranger constrained yourselves to the space of equivariant policieswhilst trying to maximize payoff, you would both converge to the policy that picks the 0 .
9 -point lever(which is not only equivariant, but invariant to all ϕ ∈ G ), and so the choice of using equivariantpolicies allows symmetry to be maintained and payout to be guaranteed.
Method
We now present our methodology and architectural choices, as well as several theoretical results.
3.1 Architecture
We first consider the symmetrizer introduced in [35], which is a functional that transforms linearmaps to equivariant linear maps.
Inspired by this we present a generalisation of the symmetrizer thatmaps from Ψ to the equivariant subspace, Ψ equiv , namely S : Ψ
→ Ψ equiv , which we define as
[Equation]
so that the first layer of ψ , denoted f , being linear, is composed with each L g , and the ultimatelayer, denoted h , is similarly composed with K − 1 g (i.e. K − 1 g ψ ( L g )
=
K − 1 g ◦ h ( ˆ ψ ( f ◦ L g )) , whereψ =
h ( ˆ ψ ( f ) ).
Below we establish properties of S that were analogously shown for the linear symmetrizer case [35]:
Proposition 1.
(Symmetric Property) S ( ψ )
∈
Ψ equiv , for all ψ ∈ Ψ ; that is, S maps neural networksto equivariant neural networks.
Proposition 2.
(Fixing Property) Ψ equiv =
Ran ( S ); that is, the range of S covers the entire equivariant subspace.
The proofs of these propositions are in the Appendix.
The proof of Proposition 1 relies on propertiesof the group structure (such as closure of the group); it is thus critical for the symmetrizer to sum overpermutations that together form a group structure, and not any random collection of permutations.
Propositions 1 and 2 show in combination that the architecture of S ( ψ ) is indeed equivariant, andthat optimizing over the class Ran ( S ) (that is, keeping the permutation matrices 3 L g and K − 1 g for all g frozen while updating the other weights) is equivalent to optimizing over Ψ equiv .
The sum overeach g ∈ G of S ( ψ ) is highly parallelizable [19], which allows for maintaining tractability.
In thisway we algorithmically enforce equivariance, mitigating the need to design equivariant layers byhand, as is typical [11, 12, 48, 49, 51, 52].
We note that for a network containing LSTM [20] layers, S will still map the network to an equivariantone no matter the values used for the hidden and cell states; indeed, Propositions 1, 2 and 4 all holdregardless of how the hidden and cell states are treated.
However, how the hidden and cell states aretreated matter for the overall performance of the network (equivariance is not a sufficient conditionitself for success in ZSC, consider the uniformly random policy).
We ergo propose the following: foreach hidden state h t,g and cell state c t,g produced by the LSTM layer over input ( L g ( τ ) , h t − 1 , c t − 1 ) ,we compute h t = 1 | G | (cid:80)
g ∈ G h t,g and c t = 1 | G | (cid:80) g ∈ G c t,g , and proceed inductively.
This is a nonobvious design choice, since each h t,g and c t,g is the output of a non-linear function; a more obviouschoice would be to choose h t :
= h t,e and c t :=
c t,e where e ∈ G is the identity element, but toensure symmetry-aware play we experimented with this averaging scheme to “symmetrically adjust”the short and long-term memories.
Since all the permutations over the input are computed in parallelwith matrix multiplication, this design choice retains tractability.
We test this choice empirically in
Section 4.
3.2 A Theory for Symmetry And Coordination
The following propositions provide guarantees for EQC’s ability to prevent all symmetry breaking:
Proposition 3.
If π 1 and π 2 are symmetry-equivalent policies with respect to ϕ ∈ G and π 1 is
G -equivariant, then π 1 = π 2 .
Proof.
We have π 1 = ϕ
( π 2 ) for some ϕ ∈ G , meaning π 1 ( ϕ − 1 ( a ) | ϕ
( τ ))
=
π 2 ( a | τ ) .
Identifying
L ϕ = { ϕ A , ϕ O } and K ϕ = ϕ
A , by equivariance (Equation 3)
we observe that π 1 ( a | τ ) =
π 1 ( K ϕ ◦
K − 1 ϕ ( a ) | τ ) =
π 1 ( K − 1 ϕ ( a ) | L ϕ ( τ ))
=
π 1 ( ϕ − 1 ( a ) | ϕ
( τ )) .
Thus π 1 ( a | τ ) =
π 1 ( ϕ − 1 ( a ) | ϕ
( τ ))
=
π 2 ( a | τ ) .
Proposition 4.
If π 1 and π 2 are symmetry-equivalent policies with respect to ϕ ∈ G , then S ( π 1 ) =
S ( π 2 ) .
Proof.
This is a corollary of the proof of Proposition 1, such that the symmetrizer architecture isfixed under any ϕ ∈ G .
G -equivariance thus reduces cross-play to self-play between policies that are symmetry-equivalentwith respect to any ϕ ∈ G .
This is critical, as this means equivariance merges policies of large classestogether, such that symmetry-equivalent policies now indeed become equivalent in coordination.
Putanother way, the set of policies that are symmetry-equivalent with respect to any ϕ ∈ G form anequivalence class, and G -equivariance reduces the cardinality of each equivalence class to 1 , thusallowing fluent coordination of policies within an equivalence class (as it merely becomes self-play).
3.3 Algorithmic Approach
Based on Propositions 1 and 2, a “naive approach” to obtaining a G -equivariant agent, for a givengroup G , might then be as follows: 1) initialize a random seed ψ ∈ Ψ ; 2) map ψ to S ( ψ ) ; 3) train
S ( ψ ) using any self-play algorithm, updating the weights while keeping the permutation matrices L gand K − 1 g for all g frozen; 4) S ( ψ ) is ready for inference.
This naive approach is sound, but time andmemory complexity inhibit scaling to large choices of G .
For this reason we present the followingtwo options (that together summarize EQC):
As a first option, we introduce “ G -OP”, where during training we sample random subsets of
G for each mini-batch (i
e. we select a random g ∈ G for each x in the mini-batch, so that ggroup-theoretically acts on x , and update the weights of ψ ) and at inference time, we use all
the permutations of G by deploying S ( ψ ) .
As a second option, we train under any self-play algorithm and at inference time deploy
S ( ψ ) .
With Option 1 we train over the permutations of G at training time.
With Option 2 we train understandard SP and use S ( · ) to obtain S ( ψ ) at inference time, which can be viewed as a coordinationimprovement operator.
We empirically validate both options in Section 4.
From here on, we refer to
S ( ψ ) as “symmetrizing ψ ”.
We expect compatibility between the G -OP trained agents and the symmetrizer at inference time,since at training time, these agents were trained over these permutations.
For Option 2, where wehave a network ϕ not trained with permutations, it is worthwhile to inspect S ( ϕ ) ; while we know
S ( ϕ ) is equivariant, we know little else regarding the conventions and style of play it would use.
While symmetry is a critical component of ZSC, other factors (such as choosing robust actions thatare likely to succeed under minimal assumptions [2, 13]) are also significant.
The following pointsjustify why S maps any network to one that is not only equivariant, but overall stronger at ZSC: 1)the network averages over how ψ would act under each g ∈ G , thus ensuring play that accounts formultiple symmetries; 2) the group closure property fixes the architecture under any permutation inthe group (see the proof of Proposition 1), forcing consistent play across different symmetries.
Weverify the empirical efficacy of Options 1 and 2 in Section 4.
What remains is the choice of the group G for our G -equivariant agents.
One might consider thechoice of G =
Φ , but this in general scales factorially with the size of the Dec-POMDP (e.g. theautomorphism group over an environment with m symmetric features grows asymptotically withthe number of permutations over m elements), and so quickly becomes infeasible.
Hence, we treat
G as a hyperparameter, and in Section 4 we analyze how different choices of G affect ZSC.
In anycase, picking G as a subgroup of Φ may be motivated in and of itself, when enumerating all possiblesymmetries of an environment is intractable, or if an environment is obfuscated such that somesymmetries cannot be known (as is the case in many large, realistic domains).
In this way, our methodcan relax the assumption of requiring access to all environment symmetries by flexibly choosingequivariance with respect to a subgroup.
Experiments
The primary test bed for our methodology is the AI benchmark task Hanabi [5].
Hanabi is a uniqueand challenging card game that requires agents to formulate informative implicit conventions in orderto be successful.
Hanabi has served as the primary test bed for many algorithms designed for ZSC[14, 22, 23, 28] due to its representation as a formidable Dec-POMDP task (see the Appendix for anintroduction to Hanabi).
To make sense of the ensuing results, recall that a perfect score in Hanabi is 25 , and that “bombingout” refers to expending all three fuse tokens and ending the game with a score of 0 .
Agents thusmust be cautious not to expend all fuse tokens.
In Hanabi, the class of environment symmetries are the permutations of the card colors, becauseassuming there is no supplemental information, a permutation of the colors of the cards leaves thegame unchanged.
We therefore have Φ = S 5 , the symmetric group over 5 elements (all possiblepermutations over 5 elements).
5 here represents the number of different colors.
| S 5 | = 5!
= 120 .
In this section we analyze differences in play and the relationship to environmental symmetry betweenour equivariant agents and OP agents (4.1) and evaluate various pre-trained agents before and aftersymmetrization (4.2).
These subsections evaluate Options 1 and 2, respectively (see Section 3).
For further details of the training setup and hyperparameters used in the following experiments, pleaserefer to the Appendix.
4.1 Symmetry-Robust Agents
Here we compare the 12 OP agents from [22] with our equivariant agents.
Specifically, we conductthe following ablation study: we fix two subgroups of S 5 : C 5 and D 10 , which are the cyclic group oforder 5 and the dihedral group of order 10 , respectively; the geometric and/or algebraic interpretationsof these groups is not important, for us groups purely serve as structured collections of permutations[9].
Next, we train 10 agents with the C 5 -OP learning rule (see Section 3), then map these 10 agentsto equivariant ones using the C 5 -symmetrizer.
Similarly, we train 10 agents with the D 10 -OP learningrule, then map them with the D 10 -symmetrizer.
Finally, we compare the ZSC performance between
[Table] [Table]
these three pools of 10 agents.
This is an ablation, because we control for the order of the group byconsidering both C 5 and D 10 (in fact, C 5 is a subgroup of D 10 ) so to observe the effect of grouporder.
Each agent uses Recurrent Replay Distributed Deep Q-Networks (R2D2)
[25], and is trainedusing value decomposition networks (VDN) [43] and the simplified action decoding (SAD) algorithm[21].
The average self-play scores achieved by the OP agents is 23 .
93 ± 0 .
02 , of the C 5 -equivariantagents is 23 .
96 ± 0 .
03 , and the D 10 -equivariant agents is 23 .
95 ± 0 . 03 .
Table 1 shows the findings of our ablation analysis.
We see that both equivariant pools outperformthe OP agents, in spite of the OP agents being trained over far more permutations than any of ourequivariant agents: | C 5 | = 5 , | D 10 | = 10 , | S 5 | = 120 (the OP agents are trained on permutationsfrom S 5 ).
This is consistent with earlier work where equivariant networks outperform data augmentation approaches [36, 47, 49, 50, 52, 55].
We conduct a Monte Carlo permutation test [15, 16] tocompare the D 10 -equivariant agent with the OP agents, for which we bound the derived p -value in a99% binomial confidence interval.
We also observe that the D 10 -equivariant agents outperform the C 5 -equivariant agents, which welargely attribute to C 5 < D 10 and Propositions 3 and 4: if we have a group G < Φ , and chooseϕ ∈ Φ uniformly at random, then P ( ϕ ∈ G ) =
| G | | Φ | , and so for two groups G, H with | G | ≤ | H | , wehave P ( ϕ ∈ G ) ≤ P ( ϕ ∈ H ) .
4.2 Equivariance as a Policy-Improvement Operator
Here we consider the extent to which test-time symmetrizing improves ZSC performance acrossdiverse policy types: we train 10 IQL agents [44], and use the 12 OP agents and 12 SAD agentsfrom [22] and the 5 agents from [23] (level 5 ), and examine the effect of transforming each of theseself-play trained agents with our equivariant architecture at inference time.
This selection of agenttypes is diverse in terms of both overall performance level as well as the style of play.
Furthermore,we consider agents that are both trained for cross-play (OP and OBL) and trained for the self-playsetting (SAD and IQL), which is an important dimension to account for in consideration of EQC’sapplicability.
All agents use R2D2 [25], and the SAD and OP agents also use VDN [43].
The averageself-play scores achieved by the SAD agents is 23 .
97 ± 0 .
04 , the IQL agents is 23 .
15 ± 0 .
02 , the OPagents is 23 .
93 ± 0 .
02 , and the OBL agents is 24 .
20 ± 0 .
Table 2 summarizes our findings over the policy types, where we consider symmetrization with boththe cyclic group of order 5 , C 5 , and the dihedral group of order 10 , D 10 .
We find that mappingany of these networks through EQC with either choice of group improves its cross-play ability with novel agents at test time.
We conduct Monte Carlo permutation tests [15, 16] to compare the
D 10 -symmetrized agents with their unsymmetrized counterparts for each policy type, for which webound each derived p -value in a 99% binomial confidence interval.
The p -values for the respectiveone-tailed tests for each of the SAD, IQL, OP and OBL agents are found to be within (0 , 0 . 00028] ,(0 , 0 . 00028] , (0 , 0 . 00028] , and [0 . 0825 , 0 . 0973] respectively, making the differences significant atthe level α = 0 .
01 for the SAD, IQL, and OP agents, and at the level α = 0 .
for the OBL agents.
Therefore, the improvement in coordination found by symmetrizing agents is highly statisticallysignificant.
In particular, we see that OP agents improve in ZSC when symmetrized.
In the proof for Proposition2, we showed that equivariant networks are fixed under the symmetrizer.
Since these OP agentswere trained under the same permutations they are being symmetrized with, yet still improve uponsymmetrization, this suggests that the data augmentation of OP does not lead to full equivariance,and so encoding environmental symmetry at the model level is necessary to realise crucial propertiessuch as Propositions 3 and 4, as well as to improve overall performance.
It is especially significant that EQC maps networks not trained with permutations (SAD, IQL, OBL)to ones that are overall better at ZSC; we justify the mechanism by which it accomplishes this in
Section 3.
As well, since OBL was up to now the state of the art for ZSC, we consequently have, atthe time of writing, OBL + EQC as the latest state of the art for ZSC on the Hanabi benchmark.
Figure 2 shows the conditional action matrices (i.e. P ( a it | a jt − 1 ) ) of a symmetrized and unsymmetrized IQL agent (see the Appendix for the plots of other agents).
This plot informs us on howdifferently the agent responds to possible actions of its partner, where large differences indicatespecialized play, and therefore likely discrepancy between the conventions used by this agent andpotential partners, which would be detrimental to cross-play.
We see that the IQL agent has learnedspecialized conventions (for example, hinting the fourth color means for the partner to discard theirfirst card), but that upon symmetrization its actions and responses become more consistent.
Another observation is that the D 10 -symmetrizer does not always outperform the C 5 -symmetrizer:namely for the case of the SAD agents in terms of bombout rate.
The combination of C 5 < D 10 and
Propositions 3 and 4 implies that the D 10 -symmetrizer corrects for symmetry breaking better than the
C 5 does, so the worse bombout rate may only be attributed to the kind of play the symmetrized policyencourages, with a likely explanation as follows: if a policy is such that it acts very differently in eachsymmetry, then symmetrizing this policy over a large number of symmetry permutations will resultin a policy with higher entropy (i.e. averaging over a large number of very different distributionsresults in one that is approximately uniform (see the Appendix)).
But behaving “randomly” is not ingeneral conducive to effective coordination, where one should try to use unambiguous conventionsto facilitate play.
Thus, when symmetrizing an arbitrary network, we tacitly assume some degree ofregularity of the network across symmetries (we control this regularity when we opt for the proposed
G -OP learning rule).
Nonetheless, both choices of C 5 and D 10 result in better ZSC than without anysymmetrization for all the tested policy types (even with SAD, which learns notoriously specializedplay [22, 23]), suggesting that policies in practice do generally satisfy these regularity requirements.
This phenomenon, however, should be kept in mind and motivates future research into deriving theoptimal choice of G for G -equivariant agents.
Related Work
Zero-Shot Coordination.
Zero-shot coordination is the problem of constructing independent agentsthat can effectively engage with one another in cross-play at test time.
Many directions to thisproblem have been considered: [14, 23] explore controlling the cognitive-reasoning depth so to avoidformation of arbitrary conventions that can complicate cross-play.
However, [14] is computationallyexacting, requiring days of GPU running time, and [23] requires simulator (environment) access.
[38]offers a modular approach to separately learn rule-dependent and convention-dependent behaviors tofacilitate coordination, but their method comes at the expense of scalability, with experiments limitedto bandit problems and 1-color Hanabi.
[34] uses repeated eigenvalues of the Hessian during trainingto find policies suited for ZSC, but is again inhibited by scalability, with experiments focused on levergames.
[31] investigate using life-long learning for ZSC, but their approach requires access to a poolof pre-trained policies.
[22], which we have referred to at length throughout this paper, takes to dataaugmentation to train symmetry-robust agents.
EQC can be used along with any of these approaches
Ad-Hoc Teamwork.
Similar to zero-shot coordination is the ad-hoc teamwork setting [6, 30, 41],where an agent is pitted with an existing group of agents, and is tasked with learning to coordinatewith this group during interaction.
While ZSC and ad-hoc teamwork are closely related problems, theydiffer in that ad-hoc team play seeks policies that do well with arbitrary agents (often an undefinedproblem setting), while ZSC agents only need to perform well with agents that are optimized for ZSCusing a common high-level approach.
In principle EQC can also be used to improve performance inad-hoc teamplay, which we leave for future work.
Equivariance.
Equivariant networks are neural networks with symmetry constraints built in, suchthat for a transformation of the input, the output is appropriately transformed as well [3, 11, 46,49, 52].
Equivariant policies have shown improved data efficiency in single agent reinforcementlearning [1, 29, 33, 35, 40, 47, 54].
In the multi-agent case, prior works use permutation symmetriesto ensure agent anonymity [8, 24, 27, 42, 43] or rotational equivariance over multiple agents [36].
While these approaches consider equivariance in multi-agent settings, they do not tackle the problemof ZSC.
In terms of architecture, the symmetry ensemble of [39] is related to ours, but we proposeadaptations that tailor recurrent layers to multi-agent coordination, thus suiting our design not just for
Markov Decision Processes (MDPs) [7] but for more general POMDPs [4] and Dec-POMDPs.
Conclusion
In this work we proposed equivariant modelling to solve symmetry breaking.
To this aim wepresented EQC, which we validated via mathematical proof and experimentation over the complex
Dec-POMDP task of Hanabi.
We showed that EQC greatly improves on [22] by guaranteeingsymmetry-equivariance of policies, and can be used as a policy-improvement operator to preventsymmetry breaking and promote robust play for a diversity of agents.
In this way, we also showedthe extent to which symmetrizing improves overall performance in coordination.
Furthermore, weshowed that EQC is “symmetry efficient”, such that it does not require access to all environmentsymmetries in order to perform well.
In addition, we empirically validated that agents perform wellunder our proposed architectural choices.
There are many directions for future work.
One is to consider how to derive optimal choices of thegroup G for G -equivariant agents effective at ZSC, relative to a given policy type and task.
Anotherdirection is exploring how to efficiently uncover environment symmetries from a domain when theyare not given nor assumed to be known.
While our work focused on symmetry breaking, future workcould explore other fundamental aspects of coordination.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Sections 3 and
(b) Did you describe the limitations of your work?
[Yes] See Sections 3, 4 and
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See theAppendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
In Section 3 and theAppendix
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Section 4, the
Appendix, and the supplementary material
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 4 and the Appendix
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Section
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See the Appendix
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
In the supplement material
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A] We use rollouts of Hanabi games played by
[Table]
AI agents.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
[Table]
A Equivariance Property Proofs
Here we write the proofs for the Symmetric and Fixing Properties of our methodology introduced in
Section 3, which are analogous to those of [35]:
Proposition.
(Symmetric Property) S ( ψ ) ∈
Ψ equiv , for all ψ ∈ Ψ ; that is, S maps neural networksto equivariant neural networks.
Proof.
Without loss of generality take g ′ ∈ G and ψ ∈ Ψ .
Then,
[Equation]
Thus K g ′ S ( ψ )( x ) = S ( ψ )( L g ′ x ) .
Proposition.
(Fixing Property) Ψ equiv =
Ran ( S ); that is, the range of S covers the entire equivariantsubspace.
Proof.
By Proposition 1, we have Ran ( S ) ⊂
Ψ equiv .
It thus suffices to show Ψ equiv ⊂
Ran ( S ) .
Takeψ ∈ Ψ equiv .
Then,
[Equation]
where we have used Equation 2 since ψ ∈ Ψ equiv .
Thus, ψ is a fixed point of S , and so ψ ∈
Ran ( S ) .
B Optimal Meta-Equilibrium Proof
Proposition.
Assume G = Φ .
If an agent chooses either the naive approach or the G -OP learningrule, then choosing either of these is payoff maximizing for the agent’s partner.
In addition, bothplayers choosing either the first approach or the G -OP learning rule is the best possible metaequilibrium.
Proof.
If an agent π 1 has an architecture π 1 ∈ Ran ( S ) or uses the G -OP learning rule, then theynecessarily have the objective to maximize
[Equation]
because either of these choices forces consideration of all ϕ ∈ G equally.
Since G = Φ , we have
[Equation]
where the expectation is taken with respect to the uniform distribution of Φ .
We have thus derived the
OP objective, and so what we sought so show is a corollary of [22] (Proposition 2 in their paper).
C Averaging Over Very Different Distributions
This passage concerns why if given a policy that acts very differently in different symmetries, whichwe then symmetrize with respect to a large group G , the resulting policy would be one that selectsactions approximately uniformly at random.
Consider l categorical distributions µ 1 , . .
. , µ l sampled uniformly from the collection of all categoricaldistributions over k categories.
Denoting µ i,j as the probability of the j th category of distribution i , bysymmetry in the categories and by assumption we have that µ a,r and µ b,s are equally distributed forall 1 ≤ r, s ≤ k , 1 ≤ a, b ≤ l .
Therefore, by the law of large numbers, 1 l (cid:80) li =1 µ i,r ≈ 1 l (cid:80) li =1 µ i,swhen l is large, for all 1 ≤ r, s ≤ k .
But since 1 l (cid:80) li =1 µ i is a probability distribution, this implies1 l (cid:80) li =1 µ i,j → 1 k as l becomes large, for all 1 ≤ j ≤ k ; i.e., 1 l (cid:80) li =1 µ i approaches the uniformdistribution over k values.
The reader may identify l = | G | , k as the number of legal actions, and each µ i as the policy’s actiondistribution under symmetry i , where we took each µ i uniformly at random to mimic a policy thatacts entirely differently per Dec-POMDP symmetry.
While the uniform distribution over actions isindeed symmetry-equivariant (and invariant), it is not in general useful.
D Hanabi
Hanabi is a cooperative card game that can be played with 2 to 5 people.
Hanabi is a popular game,having been crowned the 2013 “Spiel des Jahres” award, a German industry award given to thebest board game of the year.
Hanabi has been proposed as an AI benchmark task to test models ofcooperative play that act under partial information [5].
To date, Hanabi has one of the largest statespaces of all Dec-POMDP benchmarks.
The deck of cards in Hanabi is comprised of five colors (white, yellow, green, blue and red), and fiveranks (1 through 5), where for each color there are three 1’s, two each of 2’s, 3’s and 4’s, and one 5,for a total deck size of fifty cards.
Each player is dealt five cards (or four cards if there are 4 or 5players).
At the start, the players collectively have eight information tokens and three fuse tokens, theuses of which shall be explained presently.
In Hanabi, players can see all other players’ hands but their own.
The goal of the game is to playcards to collectively form five consecutively ordered stacks, one for each color, beginning with a cardof rank 1 and ending with a card of rank 5.
These stacks are referred to as fireworks, as playing thecards in order is meant to draw analogy to setting up a firework display 4 .
We call the player whose turn it is the active agent.
The active agent must conduct one of threeactions:
• Hint - The active agent chooses another player to grant a hint to.
A hint involves the active
agent choosing a color or rank, and revealing to their chosen partner all cards in the partner’s
hand that satisfy the chosen color or rank.
Performing a hint exhausts an information token.
If the players have no information tokens, a hint may not be conducted and the active agent
must either conduct a discard or a play.
• Discard - The active agent chooses one of the cards in their hand to discard.
The identity of
the discarded card is revealed to the active agent and becomes public information.
Discarding
a card replenishes an information token should the players have less than eight.
• Play - The active agent attempts to play one of the cards in their hand.
The identity of
the played card is revealed to the active agent and becomes public information.
The active
agent has played successfully if their played card is the next in the firework of its color to beplayed, and the played card is then added to the sequence.
If a firework is completed, the
players receive a new information token should they have less than eight.
If the player is
unsuccessful, the card is discarded, without replenishment of an information token, and the
players lose a fuse token.
The game ends when all three fuse tokens are spent, when the players successfully complete all fivefireworks, or
when the last
card in
the
deck is drawn and all players take one last turn.
If the gamefinishes by depletion of all fuse tokens (i.
e
by “bombing out”), the players receive a score of 0.
Otherwise, the score of the finished game is the sum of the highest card ranks in each firework, for ahighest possible score of 25.
E Experiment Details
Many of the hyperparameter choices are taken from [21, 22].
The main body of the network for eachtrained agent consists of 1 fully connected layer with a hidden dimension of 512, 2 LSTM layers of512 units, and two output heads for value and advantage, respectively.
The networks are updatedusing the Adam optimizer [26] with learning rate 6 .
25 × 10 − 5 and ϵ = 1 . 5 × 10 − 5 .
We use a batchsizeof 128 for training.
The replay buffer size is 10 5 episodes, and is warmed up with 10 4 episodes beforetraining commences.
We use two GPUs for asynchronous actors to collect rollouts and record toan experience replay, and one GPU for computing gradients and model updates.
The trainer sendsits network weights to all actors every 10 updates and the target network is synchronized with theonline network every 2 .
5 × 10 3 updates.
Training was concluded after 10 3 epochs, where each modelreached (or near reached) convergence; each epoch consisted of 10 3 updates.
The compute we usedwere 3 NVIDIA GeForce RTX 2080 Ti GPUs and 40 CPU cores.
Our complete code for training and symmetrizing agents can be found in the supplemental material.
We use the off-belief learning codebase [23] and the Hanabi Learning Environment (HLE).
[Figure]
P1-5 and D1-5 mean playing and discarding the 1st-5th cards in the hand.
We selected a random agent C 5 -equivariant agent and a ranomd D 10 -equivariant agent, and each plot is thereby computed by running 1000 episodes of self-play with each agent to compute the statistics.
G Broader Impact
We have found that equivariant networks can effectively solve symmetry breaking and encourage playsuited for cooperative settings.
No technology is safe from being used for malicious purposes, whichequally applies to our research.
However, fully-cooperative settings target benevolent applications.