Universally Expressive Communication in Multi-Agent Reinforcement Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Allowing agents to share information through communication is crucial for solvingcomplex tasks in multi-agent reinforcement learning.
In this work, we considerthe question of whether a given communication protocol can express an arbitrarypolicy.
By observing that many existing protocols can be viewed as instances ofgraph neural networks (GNNs), we demonstrate the equivalence of joint actionselection to node labelling.
With standard GNN approaches provably limited intheir expressive capacity, we draw from existing GNN literature and consideraugmenting agent observations with: (1) unique agent IDs and (2) random noise.
We provide a theoretical analysis as to how these approaches yield universallyexpressive communication, and also prove them capable of targeting arbitrary setsof actions for identical agents.
Empirically, these augmentations are found toimprove performance on tasks where expressive communication is required, whilst,in general, the optimal communication protocol is found to be task-dependent.
Introduction
Communication lies at the heart of many multi-agent reinforcement learning (MARL) systems.
In
MARL, multiple agents must account for each other’s actions during both training and executionand, indeed, solving complex tasks in high-dimensional spaces often requires a cooperative jointpolicy that is difﬁcult, or even impossible, to learn independently.
Therefore, allowing agents to shareinformation is crucial and how best to achieve this has remained a keen area of research since theseminal proposals of learned communication by Foerster et al.
[14] and Sukhbaatar et al.
Whilstno single universally-adopted approach has emerged, considerations for MARL communicationinclude inductive biases that aid learning.
For example, an agent’s policy should often not depend onthe order in which messages are received at a given time step.
i.e. be permutation invariant .
In this context, graph neural networks (GNNs) provide a rich framework for MARL communication.
It is natural to consider agents as nodes in a graph, with communication channels correspondingto edges between them.
GNNs are speciﬁcally designed to respect this (typically non-Euclidian)structure [5] and, indeed, many of the most successful MARL communication models fall withinthis paradigm, including CommNet [51], IC3Net [49], GA-Comm [29], MAGIC [37], Agent-Entity
Graph [2], IP [44], TARMAC [9], IMMAC [52], DGN [24], VBC [64], MAGNet [33], and TMC [65].
However, although traditional GNNs – such as those used in MARL to date – can readily providepermutation invariant communication, they are not universally expressive.
The expressivity of GNNs is often considered in the context of the 1-WL graph coloring algorithm [60].
In brief, 1-WL tests if two graphs are non-isomorphic by iteratively re-coloring the nodes has been proven to not be universally expressive (i.e. there exist non-isomorphic graphs that 1-WL can’tdistinguish).
Moreover, Morris et al.
[34] and Xu et al.
[61] proved that for any two non-isomorphicgraphs indistinguishable by 1-WL, there is no GNN that can produce different outputs for those twographs.
An example of such graphs is given in Figure 1.
This direct correspondence between GNNsand 1-WL equivalently limits the expressivity of any MARL communication built on top of GNNs.
Whist higher-order GNN architectures which go beyond 1-WL expressivity have been proposed (see[35] for an overview), many of these models do not scale well and are computationally infeasiblein practice.
However, recent works have shown that augmenting the node features can provide analternative path to increased expressivity without computationally expensive architectural changes[1, 10].
It is then natural to ask if, and how, these advancements can be brought into the MARLsetting.
In this paper, we investigate the effectiveness of GNNs for universally expressive communicationin MARL.
We deﬁne Graph Decision Networks (GDNs), a framework for MARL communicationwhich captures many of the most successful methods.
We highlight the correspondence between
GDNs and the node labeling problem in GNNs, thus making concrete the limits of GDN expressivity.
For moving beyond these limits, we consider two augmentations from the GNN literature – randomnode initialization (RNI) [1] and colored local iterative procedure (CLIP) [10], where random noiseand unique labels are added to graph nodes, respectively.
We provide a theoretical analysis as tohow these algorithms yield universally expressive communication in MARL, and also prove theirability to solve coordination problems where the optimal policy requires arbitrary sets of actionsfrom identical agents.
We then perform an empirical study where we augment several state-of-the-art
MARL communication algorithms with RNI and unique labels.
By evaluating performance acrossboth standard benchmarks and speciﬁcally designed tasks, we show that when complex non-localcoordination or symmetry breaking is required, universally expressive communication can providesigniﬁcant performance improvements.
However, in more moderate cases, augmented communicationcan reduce convergence speeds and result in suboptimal policies.
Therefore, whilst more expressive
GNN architectures are required to improve performance on certain problems, a more complete picturerelating expressivity to downstream performance remains an open question for future work.
Background
Multi-Agent Reinforcement Learning We consider the setting of Decentralized Partially Observable Markov Decision Processes [39] augmented with communication between agents.
At eachtimestep t every agent i 2 { 1 , ..., N } gets a local observation o ti , takes an action a ti , and receives areward r ti .
We consider two agent paradigms: value-based [53] and actor-critic [15, 32].
For brevity,we collectively refer to the policy network in actor-critic methods and the Q-network in value-basedmethods as the actor network .
In this paper, we consider by default parameter sharing between agent’snetworks, which is often used to yield faster and more stable training in MARL [14, 18, 45, 62].
Graph Neural Networks GNNs can refer to a large variety of models; in this paper, we deﬁne theterm to correspond to the deﬁnition of Message Passing Neural Networks (MPNNs) by Gilmer et al.[17], which is the most common GNN architecture.
Notable instances of this architecture include
Graph Convolutional Networks (GCNs) [12], GraphSAGE [19], and Graph Attention Networks(GATs) [56].
A GNN consists of multiple message-passing layers, each of which updates the node attributes / labels (terms used interchangeably).
For layer m and node i with current attribute v mi , thenew attribute v m
+
i is computed as
[Equation]
where N ( i ) is all nodes with edges connecting to i and ✓ m , ✓ 0 m , ✓ 00 m are the (possibly trainable)parameters of the update, aggregation, and readout functions for layer m .
Parameters may be sharedbetween layers, e.g. ✓ 0 = ✓ 1 .
The functions f ✓ 0 m aggr , f ✓ 00 m read are permutation invariant.
Importantly, GNNsare invariant / equivariant graph functions.
Expressivity of Multi-Agent Communication3.1 Graph Decision Networks
Many of the most successful MARL communication methods can be captured within the followingframework.
At each time step, deﬁne an attributed graph G =
( V, E, a ) with nodes V ( G ) :
={ all agents } , edges E ( G ) :
= { ( i, j ) | agent i is communicating with j } , and for all agents i , the node 85 i is labeled with the observation of i .
This graph is passed through a GNN f which outputs valuesfor each node and passes each resulting node value through the actor network of the correspondingagent.
Assuming that the actor networks use shared weights, we can substitute them for a ﬁnal
GNN layer M , where f ✓ M update ( v, ⇠ ) := P ( v ) and P represents the shared actor network.
We refer tocommunication methods that fall within this paradigm as graph decision networks (GDNs).
Given the above assumption of shared weights, any GDN simply reduces to a GNN node labellingproblem, where the correct label for a given node is the corresponding actor network output thatcollectively maximizes the joint reward (or the individual reward, depending on how the agent istrained).
Going forward, we only deal with such GDNs (ones with a shared actor network).
In thecase of stochastic policies [15, 63], the target labels are parameters of output distributions, instead ofatomic actions.
This applies to both discrete and continuous distributions.
Note that this is not how
RL agents are actually trained (i.e. they use reward signals, not supervised learning).
However, given that this paper aims to analyze expressivity, we argue that it does not matter howthe GDN is trained.
All that expressivity is concerned with is the ability of a model to produce acertain output, not how the training paradigm causes the model to converge to the solution.
All wehave to know is that there are “optimal” actor network outputs for each agent, under some metric ofoptimality, and then we can reason about the ability of the model to provide these outputs.
Scenarioswith heterogeneous agents can still be considered within this paradigm, by allocating a portion ofthe observations to indicate the agent type (e.g. through a one-hot encoding)
Models withrecurrent networks also fall within the paradigm, where the hidden or cell states for the networks canbe considered as part of the agent observations.
Due to the reduction of GDNs to a GNN node labelling problem, GDNs suffer from the sameexpressivity limits as GNNs, about which there is a plethora of work [4, 8, 16, 30, 31, 34, 38, 40, 61].
These are expanded upon in Appendix A.4.
For our analysis, we focus particularly on ways to achieveuniversal Weisfeiler-Lehman expressivity, but note that the above reduction unlocks many tools forreasoning about the expressivity of GDNs.
3.2 Desired Properties of MARL Communication
Whilst conventional GDNs cannot capture functions with expressive power beyond 1-WL [34, 61],recent GNN architectures have been proposed to achieve expressivity beyond 1-WL, even ones whichare able to express any equivariant graph function.
We can use these insights to construct moreexpressive GDNs.
However, we note that classes of models which always yield equivariant functionsare not necessarily desirable, since they cannot break symmetries between agents when required.
However, if agents have identical observations and communication graph structure in a pure GDNframework, there is no way for them to disambiguate between each other and distribute the requiredactions amongst themselves.
For a simple example, consider a setting where two agents have identicalobservations but must take opposite actions – then the only way for them to solve the environment isto communicate in such a way that they can break this symmetry and take different actions from oneanother.
More formally, since GNNs are equivariant graph functions, GDNs are equivariant functions on theagent observations and communication graph structure.
This means that agents within the samegraph orbit will always produce the same output.
For a graph G , consider two nodes u, v 2 G .
Ifthere exists an automorphism ↵ of G such that ↵ ( u ) = v , then u and v are said to be similar nodes .
The relation is similar to forms an equivalence relation on the nodes of G. Each equivalence class iscalled an orbit .
Intuitively, every node in an orbit “has the same structure”.
We denote the set of allorbits of G by R ( G ) : this forms a partition of V ( G ) .
Theorem 1.
Given a GDN f , observations O = { o 1 , ..., o n } , and communication graph G such thatnodes i and j are similar in G and o i
=
o j , then it holds that f ( O ) i
= f
( O ) j .
Full proofs for all theorems in this paper can be found in Appendix B. We formally state the desiredbehaviour of GDNs – which we refer to as symmetry breaking – that would enable them to solve suchcoordination problems.
Given a graph G with orbits R ( G ) , a GDN g ought to be able to produce, ortarget , a set of labels A k for each orbit r k :
[Equation]
where g ( G ) i is the output of g ( G ) for agent i .
Thus, ideally, MARL communication methods shouldpossess all the following properties: (1) universal expressivity for equivariant graph functions, (2)symmetry breaking for coordination problems, and (3) computational efﬁciency.
We apply twoexisting GNN augmentations to GDNs to achieve this, both of which come with minimal extracomputational cost.
3.3 Expressive Graph Decision Networks
Random Node Initialization Sato et al.
[47] propose augmenting GNNs with random nodeinitialization (RNI), where for each node in the input graph, a number of randomly sampled valuesare concatenated to the original node attribute.
For all graphs / nodes, the random values are sampledfrom the same distribution.
Abboud et al.
[1] prove that such GNNs are universal and can approximateany permutation invariant / equivariant graph function.
Technically, random initialization breaks thenode invariance in GNNs, since the result of the message passing will depend on the structure of thegraph as well as the values of the random initializations.
However, when one views the model ascomputing a random variable, the random variable is still invariant when using RNI.
In expectation,the mean of random features will be used for GNN predictions, and is the same across each node.
However, the variability of the random samples allows the GNN to discriminate between nodes thathave different random initializations, breaking the 1-WL upper bound.
Abboud et al.
[1] formally state and prove a universal approximation result for invariant graphfunctions.
They note that it can be extended to equivariant functions, which is what GDNs are.
Assuch, we adapt and state the theorem for equivariant functions.
Let G n be the class of all n -nodegraphs.
Let f : G n !
R n , a graph function which outputs a real value for each node in V ( G ) .
Wesay that a randomized function X that associates with every graph G 2 G n a sequence of randomvariables X 1 ( G ) , X 2 ( G ) , ..., X n ( G ) , one for each node, is an ( ✏ ,  ) - approximation of f if for all
G 2 G n it holds that
i 2 { 1 , 2 , ..., n } , Pr ( | f ( G ) i  X
i ( G ) |  ✏ )  1   , where f ( G ) i is the output 161 of f ( G ) for node i .
Note that a GNN h with RNI computes such functions X .
If X is computed byh , we say that h ( ✏ ,  ) - approximates f .
We can now state the following theorem:
Theorem 2.
Let n  1 and let f : G n !
R n be equivariant.
Then for all ✏ ,  > 0 , there is a GNNwith RNI that ( ✏ ,  ) - approximates f .
Such GNNs are also able to solve symmetry-breaking coordination problems by using RNI todisambiguate between otherwise identical agents.
To formally state this property, we need to deﬁne( ✏ ,  ) -approximation for sets.
We say that two sets A, B containing random variables are ( ✏ ,  ) -equal,denoted A ⇠ = ✏ ,  B , if there exists a bijection ⌧ : A !
B such that 8 a 2 A, Pr ( | a  ⌧ ( a ) |  ✏ )  1   .
Theorem 3.
Let n  1 and consider a set T , where each ( G, A ) 2 T is a graph-labels pair, suchthat G 2 G n
and there is a set of target labels A k 2 A for each orbit r k 2 R ( G ) , with | A k | = | r k | .
Then for all ✏ ,  > 0 there is a GNN with RNI g which satisﬁes:
[Equation]
In the GDN case, adding RNI means concatenating noise to the agent observations, thus achievinguniversal approximation and enabling the solving of symmetry-breaking coordination problems.
Unique Node Identiﬁers Dasoulas et al.
[10] augment GNNs with a coloring scheme to deﬁnecolored local iterative procedure (CLIP).
They use colors to differentiate otherwise identical nodeattributes, with k -CLIP corresponding to k different colorings being sampled and maximized over.
They prove theoretically that when maximizing over all such possible colorings, 1 -CLIP canrepresent any invariant graph function.
Assigning nodes unique IDs is equivalent to 1-CLIP, since this guarantees that every node withidentical attributes will have a unique “color”: its particular unique ID.
Therefore, we can leveragethe universality result for 1-CLIP (Theorem 4 in [10]), which states that with any given degree ofprecision, 1-CLIP can approximate any invariant graph function.
However, Dasoulas et al.
[10] notethat such solutions may be difﬁcult to converge to and require a large number of training steps inpractice.
Intuitively, this is because the GNN has to learn to deal with n !
permutations of unique IDs.
Similarly to the RNI case, we extend their theorem to equivariant functions.
Theorem 4.
Let n  1 and let f : G n !
R n be equivariant.
Then for all ✏ > 0 , there is a GNN withunique node IDs that ✏ - approximates f .
Such GNNs can also solve symmetry-breaking coordination problems, in a similar way to ones with
RNI.
We say that two sets A, B , which do not contain random variables, are ✏ -equal, denoted A ⇠ = ✏ B ,if there exists a bijection ⌧ : A !
B such that 8 a 2 A, | a  ⌧ ( a ) |  ✏ .
Theorem 5.
Let n  1 and consider a set T , where each ( G, A ) 2 T is a graph-labels pair, suchthat G 2 G n
and there is a set of target labels A k 2 A for each orbit r k 2 R ( G ) , with | A k | = | r k | .
Then for all ✏ > 0 there is a GNN with unique node IDs g which satisﬁes:
[Equation]
In the GDN case, this means that by giving each agent a unique ID in its observations, we can achieveuniversal approximation and enabling the solving of symmetry-breaking coordination problems.
Experiments
4.1 Methods
Baselines For evaluation, we adopt a diverse selection of MARL communication methods whichfall under the GDN paradigm.
These are shown in Table 1, along with the communication graphstructure, agent model, and GNN architecture.
We use the code provided by Jiang et al.
[24], Niuet al.
[37] as starting points.
All of the implementations are extended to support multiple roundsof message-passing and the baselines are augmented with the ability for their communication to bemasked by the environment (e.g. based on distance or obstacles in the environment).
We ﬁx thenumber of message-passing rounds to be 4 and otherwise use the original models and hyperparametersfrom [24, 37] experiment and hyperparameter details
Appendix C, fullresults
Appendix D.
[Table]
Environments Predator-Prey [9, 27, 29, 37, 49] and Trafﬁc Junction [9, 27, 29, 37, 49, 51] arecommon MARL communication benchmarks.
In Predator-Prey, predator agents are tasked withcapturing prey and in Trafﬁc Junction, agents need to successfully navigate a trafﬁc intersection(full descriptions of each environment are given in Appendix C.2).
We perform evaluations on thesebenchmarks to test how well our universally expressive GDN models perform when there is notnecessarily a beneﬁt to having communication expressivity beyond 1-WL.
We also introduce twonew environments, Drone Scatter and Box Pushing, to respectively test symmetry-breaking andcommunication expressivity beyond 1-WL.
Drone Scatter consists of 4 drones in a homogeneous ﬁeld surrounded by a fence.
Their goal is tomove around and ﬁnd a target hidden in the ﬁeld, which they can only notice when they get closeto.
The drones do not have GPS and can only see directly beneath them using their cameras, as wellas observing their last action.
The best way for them to locate the target is to split up and search indifferent portions of the ﬁeld, despite them all having the same observations; thus, they are givenrewards for splitting up.
Box Pushing consists of 10 robots in a 12x12 construction site, which has boxes within that need tobe moved to the edge of the site: the clearing area.
Robots attach themselves to boxes before theycan move them; when attached, robots can no longer see around themselves.
Free-roaming robotscan communicate with any other free-roaming robots, but attached robots can only communicatewith the robots directly adjacent to them.
The environment either spawns with one large box or twosmall boxes and agents spawn already attached.
4 attached robots all moving in the same directionare needed to move a small box, and 8 all power moving in the same direction to move a large box.
To solve the environment, the robots need to be able to communicate with each other to ﬁgure outwhich type of box they are on and all push correctly, at the same time, and in the same direction.
Since the communication graphs corresponding to the scenarios with small and large boxes are 1-WLindistinguishable, communication beyond 1-WL is needed to optimally solve the environment.
Evaluation Procedure We augment baseline communication methods with RNI and unique IDs toperform our evaluations.
Agent IDs are represented by one-hot encodings and “0.25 RNI” refers to25% of the observation space being randomly initialized.
We sample each RNI value uniformly from[  1 , 1] .
For each scenario and for every baseline communication method, we compare 4 models:the baseline without modiﬁcations, the baseline augmented with unique IDs for each agent, thebaseline augmented with 0.75 RNI, and ﬁnally 0.25 RNI.
The only exceptions are the Drone Scatterevaluations, where 0.25 RNI is not used since the observation space is not large enough, and the
Drone Scatter experiments using stochastic evaluation, where DGN is not used since it does notsupport stochastic evaluation.
For each run, corresponding to a random initialization (one seed) of the model in question, weperform periodic evaluations during training.
Each epoch consists of 5000 training episodes, afterwhich 100 evaluation episodes are used to report aggregate metric scores, yielding an evaluationscore for the model after every epoch.
Following an established practice in MARL evaluation[15, 22, 41, 46, 59, 63, 66], we take the value of a metric for a run to be the best value achieved duringtraining, so that our metrics are robust against runs which converge at some point and then degrade inperformance as they continue to train.
In such cases, one would use the parameters from the best
[Table]
performing model found during training for real-world evaluation; thus, that performance makesmore sense to report than the model performance once training has ﬁnished.
We utilize 10 seeds for
Box Pushing experiments and 5 for all others.
For each scenario, metric, baseline communicationmethod, and variant thereof, we report the mean metric value across all seeds and a 95% conﬁdenceinterval.
To calculate the conﬁdence interval, we assume a normal distribution and compute theinterval as 1 .
96 ⇥ SEM (standard error of the mean).
Finally, for all Box Pushing experiments, wemake use of a form of hybrid imitation learning to help deal with exceptionally sparse rewards (fulldetails are given in Appendix C.3).
4.2 Results
Benchmark Environments Experimental results on the benchmark environments are shown in
Table 2 for Easy Trafﬁc Junction, Table 3 for Predator-Prey, and Table 4 for Medium Trafﬁc Junction.
In general, unique IDs tends to perform comparably to the baseline.
The only exception to this is for
IC3Net on Medium Trafﬁc Junction, where unique IDs struggle.0.75 RNI is categorically the worst method, consistently getting outperformed by all other methodsand only coming out on top for MAGIC on Medium Trafﬁc Junction, which is not signiﬁcant due tothe instability of that set of results.
75% of observations being randomly initialized appears far toomuch for the system to be able to learn effective policies.
However, universality results still hold forlower ratios of RNI.0.
RNI exhibits strong performance on Easy Trafﬁc Junction, always solving the environment andalmost always outperforming the baseline.
However, on sparse-reward problems (such as PredatorPrey and, to a lesser extent, Medium Trafﬁc Junction)
RNI methods typically take longer to convergethan the baseline and unique IDs, and 0.25 RNI can struggle to reach the performance of baselinemethods.
This aligns with Abboud et al.
[1]’s observation that GNNs with RNI take signiﬁcantlylonger to converge than normal GNNs.
This is only is exacerbated in a MARL setting with sparsereward signals, where slow convergence is expected regardless of the RNI augmentation.
Indeed, onall examples, RNI methods typically take longer to converge than the baselines and unique IDs.
Overall, we conclude that both unique IDs and 0.25 RNI achieve sufﬁcient performance on thebenchmarks to qualify them for use, especially given that the extra expressivity they provide is notstrictly necessary.
With respect to the different baselines, we note that simple baselines such as
CommNet work the best when the optimal policy is also simple, such as for Easy Trafﬁc Junction,but that more sophisticated baselines outperform them on the complex environments.
We also notethe very unstable performance of MAGIC for the Trafﬁc Junction environments.
Weisfeiler-Lehman Expressivity Results for the Box Pushing environment are shown in Table5.
0.25 RNI is the clear winner, achieving the top performance across almost all baselines and onlyslightly worse performance in the others.
It is never outperformed by the baseline.
This indicatesthat when communication expressivity beyond 1-WL is helpful for solving the environment, using
RNI is the clear choice.
Across all but one method, unique IDs also outperformed the baseline,demonstrating the beneﬁt of having higher expressivity.
However, unique IDs tend to yield less stablesolutions and less effective policies than 0.25 RNI.
[Table] [Table]
For completeness, we note that expressivity beyond 1-WL is not strictly needed to solve the Box
Pushing environment, as demonstrated by several baselines achieving a “ratio cleared” score greaterthan 0.5, meaning they learned to sometimes clear both types of boxes.
This is because the environment can be solved, albeit inefﬁciently, by recurrent policies which learn to alternate actions between“normal” and “power” moves each time step.
Such policies are guaranteed to move the box every 2time steps.
Symmetry Breaking Results for the Drone Scatter experiments are shown in Table 6 and Table 7,for agents with stochastic and greedy evaluation respectively.
Across all of them, unique IDs exhibitsconsistently superior performance than 0.75 RNI and the baseline, since it deterministically breaksthe symmetry between agents and allows them to split up easily to solve the environment.
In thestochastic case, 0.75 RNI performs similarly to the baseline, but is markedly superior to the baselinein the greedy case.
The comparison to a purely random agent indicates that the models are learningmuch more effective policies than just moving around at random.
Baseline methods with stochastic evaluation achieve consistently higher pairwise distances than theirgreedy counterparts, meaning they are learning to split up to ﬁnd the target.
They are capable of thisdue to a combination of 3 things: a stochastic policy, recurrent networks, and agents observing theirlast actions.
Initially, agents cannot differentiate between each other, and all produce the same actiondistribution.
However, if the distribution is diverse, then they are expected to produce different actionssince they sample randomly from this distribution, which are observed in the next time step.
Thedifferent observations lead to different hidden states in the recurrent networks, effectively changingthe observations between agents and allowing them to differentiate between each other.
This is notthe case for greedy action selection, which is common when doing policy evaluation in MARL, wherethe actions chosen will always be the same. [Table] [Table] [Table]
Conclusion
We introduce GDNs, a framework for MARL communication, and formally show how it correspondsto the node labelling problem in GNNs.
Our theoretical contributions use this observation todemonstrate that existing MARL communication methods relying on conventional GNN architectureshave provably limited expressivity.
Driven by this, we prove how augmenting agent observationswith unique IDs or random noise yields universally expressive invariant communication in MARL,whilst also providing desirable properties such as being able to perform symmetry-breaking: targetingarbitrary sets of joint actions for identical agents.
Experimentally, we compare these augmentations across 6 different MARL communication baselinesthat fall within the GDN paradigm, using 3 benchmark communication environments and 2 tasksdesigned to separately test expressivity and symmetry-breaking.
Ultimately, we ﬁnd that, whilstunique IDs or smaller RNI augmentations can typically be applied without detriment on standardenvironments, they do not readily provide improved performance either.
However, on environmentswhere more complex coordination is required, these augmentations are essential for strong performance.
With RNI and unique IDs being best suited to environments requiring increased expressivityand symmetry-breaking, respectively, it is interesting to note that no single method emerges which wecan recommend as the de facto choice for MARL practitioners.
This suggests that a more completepicture of the relationship between communication expressivity and downstream performance onrelevant tasks remains an open question for future research.
Furthermore, insights into GNN architectures can be leveraged in GDNs, which opens many promising avenues for future work in MARLcommunication.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] The theoretical assumptions arestated concretely in Section 3.
The limitation of our empirical analysis not providinga complete picture relating expressivity to performance on MARL environments isclearly stated in the abstract and introduction.(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Weidentiﬁed no negative societal impacts.(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Assumptionsare stated in Section 3 and Appendix B.(b) Did you include complete proofs of all theoretical results?
[Yes] Full proofs for all ofour theoretical results are provided in Appendix B.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Code,environments, and instructions for reproducibility are included in the supplementalmaterial.(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] All training details are stated in Section 4.1 and Appendix C.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] 95% CI error bars are reported for all results in Section4.2 and Appendix C.(d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] The above details are provided inAppendix D.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes] [24, 37](b) Did you mention the license of the assets?
[Yes] See Appendix C.1.(c)
Did you include any new assets either in the supplemental material or as a URL?
[Yes]New environments and code provided in the supplemental material.(d)
Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A] [Table]