S TATEFUL ACTIVE FACILITATOR : C OORDINATION AND E NVIRONMENTAL H ETEROGENEITY IN C OOPERATIVE M ULTI -A GENT R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
In cooperative multi-agent reinforcement learning, a team of agents works together to achieve a common goal.
Different environments or tasks may require varying degrees of coordination among agents in order to achieve the goal in an optimal way.
The nature of coordination will depend on properties of the environment—its spatial layout, distribution of obstacles, dynamics, etc.
We term this variation of properties within an environment as heterogeneity .
Existing literature has not sufficiently addressed the fact that different environments may have different levels of heterogeneity.
We formalize the notions of coordination level and heterogeneity level of an environment and present HECOGrid , a suite of multi-agent RL environments that facilitates empirical evaluation of different MARL approaches across different levels of coordination and environmental heterogeneity by providing a quantitative control over coordination and heterogeneity levels of the environment.
Further, we propose a Centralized Training Decentralized Execution learning approach called Stateful Active Facilitator (SAF) that enables agents to work efficiently in high-coordination and high-heterogeneity environments through a differentiable and shared knowledge source used during training and dynamic selection from a shared pool of policies.
We evaluate SAF and compare its performance against baselines IPPO and MAPPO on
HECOGrid
Our results show that SAF consistently outperforms the baselines across different tasks and different heterogeneity and coordination levels.
I NTRODUCTION
Multi-Agent Reinforcement Learning (MARL) studies the problem of sequential decision-making in an environment with multiple actors.
A straightforward approach to MARL is to extend single agent RL algorithms such that each agent learns an independent policy (Tan, 1997). de Witt et al.
(2020) recently showed that PPO, when used for independent learning in multi-agent settings (called Independent PPO or IPPO) is in fact capable of beating several state-of-the-art approaches in MARL on competitive benchmarks such as StarCraft (Samvelyan et al., 2019).
However, unlike most singleagent RL settings, learning in a multi-agent RL setting is faced with the unique problem of changing environment dynamics as other agents update their policy parameters, which makes it difficult to learn optimal behavior policies.
To address this problem of environment non-stationarity, a class of approaches called Centralized Training Decentralized Execution (CTDE) such as MADDPG (Lowe et al., 2017), MAPPO (Yu et al., 2021), HAPPO and HTRPO (Kuba et al., 2021) was developed.
This usually consists of a centralized critic during training which has access to the observations of every agent and guides the policies of each agent. In many settings, MARL manifests itself in the form of cooperative tasks in which all the agents work together in order to achieve a common goal. This requires efficient coordination among the individual actors in order to learn optimal team behavior.
Efficient coordination among the agents further aggravates the problem of learning in multi-agent settings.
Another challenge in practical multi-agent learning problems is heterogeneity in the environment.
Environment heterogeneity in reinforcement learning has previously been studied in the context of federated learning in Jin et al. (2022) which considers the problem of jointly optimizing the behavior of n agents located in n identical environments (same state space, same action space,
[Table] and same reward function) with differing state-transition functions. However, in some real-world multi-agent learning problems, environment properties such as structure, dynamics, etc, may also vary within an environment, as compared to varying across different environments. Unmanned guided vehicles (UGVs) used for search and exploration may encounter different conditions, such as different distribution of obstacles or different terrains leading to differing dynamics in different regions.
Warehouse robots coordinating to pick up a bunch of items might have to work in conditions varying from one section of the warehouse to another such as different organization of aisles.
Similarly, as argued in Jin et al.
(2022), an autonomous drone should be able to adapt to different weather conditions that it encounters during its flight. We build upon the formulation of Jin et al. (2022) to address the broader problem of heterogeneity within the environment.
We formally define two properties of an environment: heterogeneity , which is a quantitative measure of the variation in environment dynamics within the environment, and coordination which is a quantitative measure of the amount of coordination required amongst agents to solve the task at hand (we formally define heterogeneity and coordination in Section 3).
The difficulty of an environment can vary based on the amount of heterogeneity and the level of coordination required to solve it.
In order to investigate the effects of coordination and environmental heterogeneity in MARL, we need to systematically analyze the performance of different approaches on varying levels of these two factors.
Recently, several benchmarks have been proposed to investigate the coordination abilities of MARL approaches, however, there exists no suite which allows systematically varying the heterogeneity of the environment.
A quantitative control over the required coordination and heterogeneity levels of the environment can also facilitate testing the generalization and transfer properties of MARL algorithms across different levels of coordination and heterogeneity.
A detailed analysis of the existing benchmarks can be found in Appendix A.
To address these limitations, we propose HECOGrid , a procedurally generated Multi-Agent Benchmark built upon MARLGrid (Ndousse et al., 2021).
HECOGrid consists of three different environments which allow the testing of algorithms across different coordination and environmental heterogeneity levels.
Each environment consists of N agents and M treasures, where the goal is to maximize the total number of treasures picked up by the agents in one episode.
c agents are required to pick up a single treasure, where c is the coordination level of the environment.
The environment is spatially divided into h zones, and the environment dynamics vary from zone to zone.
h is the level of heterogeneity of the environment.
c , h , N , and M are controllable parameters.
Table 1 presents a qualitative comparison between HECOGrid and other commonly used Multi-Agent RL environment suites.
Further, to enable efficient training of MARL agents in high coordination and heterogeneity, we introduce a novel approach called Stateful Active Facilitator ( SAF ).
SAF uses a shared knowledge source during training which learns to sift through and interpret signals provided by all the agents before passing them to the centralized critic.
In this sense, the knowledge source acts as an information bottleneck and helps implement a more efficient form of centralization by refining the information being passed to the critic.
Further, recent work in modular deep learning (Goyal et al., 2021a;d;b;
Rahaman et al., 2021) has shown that different neural modules trained on a common objective lead to the emergence of specialist neural modules which help in improving performance via decomposition of the task.
We hypothesize that a similar form of modularity can also be helpful in tackling the problem of heterogeneity.
Instead of each agent using an individual monolithic policy, we propose the use of a pool of policies that are shared across different agents (Goyal et al., 2021a).
At each time step, each agent picks one policy from the pool which is used to determine its next action where the selection being conditioned on the current state of the agent. During execution the parameters of the shared pool of policies can be distributed to each agent which can then operate in a completely decentralized manner.
Hence our method falls under the umbrella of Centralized Training Decentralized Execution (CTDE) methods.
Contributions.
We introduce a set of cooperative MARL environments with adjustable coordination and heterogeneity levels.
We also propose SAF - which consists of a shared knowledge source which is empirically shown to improve performance in high-level coordination settings, and a pool of policies that agents can dynamically choose from, which helps in tackling environmental heterogeneity.
We show that the proposed approach consistently outperforms established baselines MAPPO (Yu et al., 2021) and IPPO (de Witt et al., 2020) on environments across different coordination and heterogeneity levels.
The knowledge source is the key to improved performance across different coordination levels whereas further ablation studies show that the pool of policies is the key to good performance across different levels of environmental heterogeneity.
R ELATED W ORK
Centralized Training Decentralized Execution (CTDE).
These approaches are among the most commonly adopted variations for MARL in cooperative tasks and address the problem of environment non-stationarity in multi-agent RL.
They usually involve a centralized critic which takes in global information, i.e. information from multiple agents, and decentralized policies whose learning are guided by the critic.
Lowe et al.
(2017) first proposed an extension of DDPG (Lillicrap et al., 2016) to a multi-agent framework by using a shared critic and agent specific policies during training, and decentralized execution.
Yu et al.
(2021) proposes the extension PPO (Schulman et al., 2017) to a multi-agent framework in a similar manner.
Kuba et al.
(2021) extends trust region learning to a cooperative MARL setting in a way that the agents do not share parameters. Foerster et al. (2018) uses the standard centralized critic decentralized actors framework with a counterfactual baseline .
Li et al.
(2021) uses an information theory-based objective to promote novel behaviors in CTDE-based approaches.
Value Decomposition (Sunehag et al., 2018; Rashid et al., 2018; Wang et al., 2020;
Mahajan et al., 2019), (Rashid et al., 2018) approaches learn a factorized state-action value function.
Sunehag et al.
(2018) proposes Value Decomposition Networks (VDN) which simply add the stateaction value function of each agent to get the final state-action value function.
Rashid et al.
(2018) uses a mixing network to combine the action-value functions of each agent in a non-linear fashion.
Coordination in Multi Agent Reinforcement Learning. There have been several definitions of coordination in MARL, most of which come from the agents’ perspective. Guestrin et al.
(2002) and Busoniu et al. (2008) define coordination as the ability of agents to find optimal joint actions. Kapetanakis & Kudenko (2004) defines coordination as consensus among agents.
Choi & Ahn (2010) defines coordination as agents’ ability to achieve a common goal. In contrast, we analyze coordination levels from the angle of the RL environment. Developing MARL algorithms that train coordinated policies requires sufficient exploration due to the presence of multiple equilbria. A large section of recent approaches revolves around explicitly taking into account the states and actions of other agents by learning differentiable communication channels between agents in order to train coordinated policies.
Foerster et al.
(2016) proposes DRQN-based communication protocols DIAL and RIAL.
Sukhbaatar et al.
(2016) proposes CommNet which uses a shared recurrent module which calculates the state of each agent conditioned on the previous state and the mean of messages received from all the other agents.
Jiang & Lu (2018) and Das et al.
(2019) use attention based communication protocols where attention is used to transmit and integrate messages sent by other agents.
Similar to our work, Kim et al.
(2020a) attempts to learn coordinated behavior in a CTDE setting without introducing explicit communication, by using the mutual information of the agents’ actions. Environmental Heterogeneity in MARL.
Environmental heterogeneity is a relatively uncharted land in MARL and has only been explored to a very limited extent in RL as a whole.
Jin et al.
(2022) analyzed environmental heterogeneity in a federated learning setting. The authors define heterogeneity as different state transition functions among siloed clients in the federated system, while for each client the environment is homogeneous.
In another more recent study by Xie & Song
(2022), heterogeneity of initial state distribution and heterogeneity of environment dynamics are both taken into consideration.
Our problem is also closely related to Hidden Parameter Markov Decision
Processes (HiP-MDPs) (Doshi-Velez & Konidaris, 2016) which consider a set of closely related
MDPs which can be fully specified with a bounded number of latent parameters.
Our approach can also be seen as being related to the multi-task reinforcement learning setting, where the goal is to learn an optimal policy that can be generalized to a set of closely related tasks.
In all of the above works, heterogeneity is considered to be arising from variations across different environments or tasks.
In contrast, we focus on heterogeneity within an environment.
Extended related works can be found in appendix A.1.
P RELIMINARIES
Notation.
In this work, we consider a multi-agent version of Partially Observable Markov Decision Processes (POMDP) (Zhang & Lesser, 2013).
The environment is defined as ( N , S , O , A , Π , R, γ ) .
N = { 1 , ..., N } denotes a set of N > 1 agents.
Like Zhang & Lesser (2013) we define the joint state S as follows : S = S e × S 1 × . . . S N where S e is the set of possible states that the external environment can be in and S i is the set of possible local states that an agent i can be in, where i ∈ N .
A =
A 1 ×· · ·× A N denotes the joint action space and a i,t ∈
A i refers to the action of agent i at time step t .
O
=
O 1 × · · · × O N denotes the set of partial observations where o i,t ∈
O i stands for partial observation of agent i at time step t .
Π is the set of policies available to the agents.
To choose actions, agent i uses a stochastic policy π θ i : O i × A i
(cid:55)→ [0 , 1] .
Actions from all agents together produce the transition to the next state according to transition function T : ( s t , a 1 ,t , . .
. , a N,t ) (cid:55)→ s t +1 where s t = ( s e,t , s 1 ,t , . . . , s N,t ) ∈ S is the joint state at timestep t , where s e,t is the external environment’s state and { s i,t } i ∈N are the agents’ states at timestep t .
R : S × A (cid:55)→ R is the global reward function conditioned on the joint state and actions.
At timestep t , the agent team receives a reward r t =
R ( s t , a 1 ,t , . .
. , a N,t )
based on the current joint state s t and the joint action a 1 ,t , . . .
, a N,t .
γ is the discount factor for future rewards.
In this study, we focus on cooperative MARL with partial observations.
Coordination level in cooperative MARL task.
Here, we quantitatively define the coordination level required in a cooperative MARL task used in this study.
Recall that R : S × A (cid:55)→ R is the global reward function conditioned on the joint state and actions.
At time step t , the agent team receives a reward r t =
R ( s t , a t ) based on the current total state of all agents s t and joint action a t .
Just as in the previous section, the joint state is factorized as follows: s t = ( s e,t , s 1 ,t , . . .
, s N,t ) where s e,t is the state of the external environment and s i,t is the local state of agent i at timestep t and i ∈ N . Let G ⊂ N denote a subset of |G| =
k agents and different subsets can overlap. We define
R G ( s t , a t ) = R G ( s e,t , s G t , a G t ) as the joint reward that can only be obtained when a subset of k agents cooperate, where s G t = { s i,t } i ∈G and a G t = { a i,t } i ∈G .
We can then write the joint reward as the sum of rewards contributed by all subset of agents: [Equation]
Hence, the level of coordination c t can be defined as the positive reward that can be obtained at time t if no less than c t agents are involved in it:
[Equation] Where |G| is the number of elements in G .
The global coordination level of the environment c can then simply be defined as: c = max t ≥ 0 { c t } . This means that if there’s at least one task in the environment that must be solved using the largest number of agents, then that number of agents ( c t ) is defined as the coordination level of that environment. It is worth mentioning the difference between the problem we explore and the formulation in previous studies such as (Zhang & Lesser, 2013). We define coordination in a way that some rewards require at least k agents to coordinate with each other to obtain but different subsets of agents G j do not have to be disjoint, i.e., one agent can be involved in obtaining more than one reward in a single time step.
Heterogeneity level of cooperative MARL task.
Another aspect we like to explore is the heterogeneity of the RL environment.
It is worth pointing out that the heterogeneity of the RL environment is different from the heterogeneity of agents or heterogeneity of policies as explored in previous studies (Mondal et al.
; Kapetanakis & Kudenko (2004).
For simplification, we define heterogeneity in a single-agent RL environment, which can be easily unwrapped into a multi-agent setting.
We assume that the environment has a collection of K different state-transition functions {T k : ( s t , a t ) (cid:55)→ s t +1 } 1 ≤ k ≤ K .
At each timestep t , whenever the agent takes an action, its next state is governed by one of the K state-transition functions, and that choice is decided according to some (possibly latent) variable ν t .
K is then defined as the level of heterogeneity of the environment, if K = 1 then the environment is said to be homogeneous. In this study, we implement ν t as the position of the agent in the environment, which means the state-transition function depends on where the agent is in the environment.
HECOG RID : MARL ENVIRONMENTS FOR VARYING COORDINATION AND ENVIRONMENTAL HETEROGENEITY LEVELS
We introduce a set of three cooperative MARL environments, which we collectively call HECOGrid, that allows manipulation of coordination and environmental heterogeneity levels in addition to several other properties.
HECOGrid consists of TeamSupport , TeamTogether and
KeyForTreasure environments as shown in Fig.
In this section, we describe each HECOGrid environment in detail.
TeamTogether Environment.
The cooperative team task in this environment is to collect as many treasures as possible in a limited number of time steps.
Each treasure is presented as a bright box in the environment and becomes grey once collected.
In order for a treasure to be collected, a certain number of agents need to step onto it simultaneously.
The number of agents required for collection is the level of coordination of the task.
TeamSupport Environment.
This environment is similar to TeamTogether except that in order for an agent to collect a treasure, instead of being on the treasure together simultaneously with other agents, it needs to step onto the box and with a certain number of agents within a fixed distance (set to 2 by default) to support the collection. This number of agents required for collection support (including the agent that actually collects) is defined as the level of coordination of the task. Rewards are distributed equally across all agents in the whole team.
KeyForTreasure Environment.
This environment is similar to TeamSupport except that an agent can only collect a treasure if it is carrying a key, and to collect the treasure, a certain number of agents need to be on the box simultaneously.
This additional key-searching step increases the difficulty of the task.
If an agent picks up a key, its color changes.
In all the environments, all the rewards are distributed equally across all agents on the team. Environmental heterogeneity in HECOGrid. We implement environmental heterogeneity by dividing the grid into K zones.
For each zone, the transition function T is different.
Concretely, each action leads to a different state depending on which zone the agent is in {+(e.g. action number 
 may [Figure] or+}
SAF : T HE S TATEFUL A CTIVE F ACILIATOR
In this section we describe the proposed method that consists of two main components: a Knowledge
Source (KS) that enhances coordination among the agents and a policy pool that allows agents to dynamically select a policy, enabling agents to exhibit diverse behaviors and have distinct goals, as well as handling heterogeneous environments.
In the following subsection, we explain each step.
For further details, see Algorithm 1 in Appendix A.3. Step 1: Generating the messages.
Each agent i receives a partial observation
o i,t at each time step t .
These observations are encoded into messages which are written into the KSby a common encoder g θ : m ′ i,t
= g θ
( o i,t ) ; m ′ i,t ∈ R d m .
We denote the set of messages generated by the agents at time step t by M ′ t :
[Equation]
Step 2: Writing into the Knowledge Source.
The messages M ′ t generated in step one are distilled into a latent state which we term as a Knowledge Source or KS.
We represent the KS state at time step t by F t .
F t consists of L slots { l 0 , l 1 , ..
l L − 1 } , each of dimension d l so that F t ∈ R L × d l .
The messages in M ′ t compete with each other to write into each KS’s state slot via a cross-attention mechanism.
The query, in this case, is a linear projection of the F t , i.e., (cid:101) Q = F t (cid:102) W q , whereas the keys and values are linear projections of the messages M ′ t .
KS state is updated as:
[Equation]
After this, self-attention is applied to the KS using a transformer encoder tower constituting a perceiver encoder structure (Jaegle et al., 2022).
Step 3: Reading from the Knowledge Source.
The KS makes the updated state available to the agents should they deem to use it.
We again utilize cross attention to perform the reading operation.
All the agents create queries Q st = { q si,t | 1 ≤ i ≤ N } ∈ R N × d e where q si,t = W q read s i,t and s i,t are encoded partial observations.
Generated queries are matched with the keys κ = F t W e ∈ R l × d e from the updated state of SAF .
As a result, the attention mechanism can be written as:
[Equation]
where M t = { m i,t | 1 ≤ i ≤ N } .
Consequently, the read messages are used to define a more informative state s SAF i,t = g ϕ ([ s i,t , m i,t ]) , where g ϕ is parameterized as a neural network.
Finally, the new state s SAF i,t is used by the critic to compute values.
Interestingly, since s SAF i,t is exclusively used by the critic, which is only used during training, SAF do not uses communication during execution.
Step 4: Policy Selection.
In order to perform policy selection for each policy we define an associated signature key which is initialized randomly at the start of the training: k Π = { k π u | 1 ≤ u ≤ U } .
These keys are matched against queries computed as deterministic function of the encoded partial observation q policy i,t = g psel ( s i,t ) , where g psel is parametrized as a neural network.
[Equation]
As a result of this attention procedure, agent i selects a policy π index i .
This operation is performed independently for each agent, i.e. each agent selects a policy from the policy pool.
Therefore, it does not involve communication among different agents.
E XPERIMENTS
In this section, we design empirical experiments to understand the performance of SAF and its potential limitations by exploring the following questions: (a) how much difficulty do high levels of coordination and environmental heterogeneity cause to cooperative MARL tasks?
(b) does SAF perform well when coordination or/and heterogeneity levels are high?
(c) Is SAF robust to changes of coordination and heterogeneity levels?
and (d) SAF has three components.
How does each component contribute to the performance at high coordination and heterogeneity levels?
Baseline Methods.
We compare SAF with two widely used algorithms with related architectural designs and similar number of parameters to SAF , namely Independent PPO (IPPO) (de Witt et al., 2020) and multi-agent PPO (MAPPO)
(Yu et al., 2021)
(Table A.3). In IPPO, each agent has its own actor and critic and does not share information with other agents.
In MAPPO, instead of being trained in a decentralized manner, the critic takes information from all agents in each step as inputs during training, and agents operate in a decentralized manner without sharing information during execution. SAF has similar training and execution strategy as MAPPO but uses an added componentthe KS before passing information to the critic during training, and a shared pool of policies instead of a single shared policy for each agent.
[Figure]
High levels of coordination and environmental heterogeneity. To understand how much difficulty high levels of coordination in the environment cause, we conducted experiments in all three HECOGrid environments for coordination levels 1 to 3 with heterogeneity level set to 1.
We train all methods for 10M steps for all experiments. Our results, as shown in Figure 3 show that the performance of all three methods decreases dramatically as coordination levels increase.
Performance of all three methods show a more than 50% decrease in performance at coordination level 2, as compared with coordination level 1, in all environments. At a coordination level of 3, all methods fail to show meaningful behavior. These observations indicate that tasks requiring more agents to work together for reward collection are extremely challenging in a cooperative MARL setting.
To understand how much difficulty high levels of environmental heterogeneity cause, we conducted experiments in all three HECOGrid environments for heterogeneity levels 1 to 5 with coordination level set to 1. All methods show a decrease in performance as the environments become more heterogeneous, though to a smaller extent as compared with coordination levels (see Figure 4).
We provide further results for experiments performed in cases where coordination and heterogeneity levels are high simultaneously in Appendix A.2. [Figure] [Figure]
i
[Figure]
c
[Figure]
t
[Figure]
r n s
[Figure]
i s
[Figure]
c
[Figure]
t
[Figure] r [Figure] s
[Figure] Robustness to changes in coordination and heterogeneity levels.
In most real-world applications, such as robots in warehouses, the coordination levels as well as environmental heterogeneity levels can change over time and may even be unknown to the agents. Therefore, the agents’ robustness to such changes is important.
To understand if agents trained with SAF can still function well in these out-of-distribution (OOD) settings, we conduct experiments to test the agents’ performance on TeamSupport and TeamTogether environments with heterogeneity or coordination levels that are different as compared to the ones used during training.
First, we train the agents in environments with a coordination level of 2 and a heterogeneity level of 1 and test their performance at coordination levels between 1 and 3, and a heterogeneity level of 1.
As shown in Figure 5, SAF shows better transfer in the more difficult TeamTogether environment than other methods but fails to perform as well as MAPPO in the TeamSupport environment. Next, we train the agents in environments with a coordination level of 1 and a heterogeneity level of 2 and test their performance at a coordination level of 1 and heterogeneity levels between 1 and 5.
As shown in figure 5, SAF shows superior performance in TeamTogether environment and matches the performance of MAPPO in TeamSupport environments. This suggests that SAF has similar robustness to changes in coordination and heterogeneity levels as some of the widely used baselines in the MARL community. [Figure] increasing levels of heterogeneity.
(b) Performance of the
KS and the pool of policies against baselines in increasing levels of coordination. KS contributes to performance in all settings and PP especially improves performance in heterogeneous environments. Contribution of each component of SAF . In the method section, we design SAF by hypothesizing that in order to tackle coordination and environment heterogeneity, two key elements are necessary: the use of a shared knowledge source (KS) and a shared pool of policies (PP) from which the agents can dynamically choose.
We wish to understand how much each component contributes to the performance of SAF in different scenarios. To investigate this question, we conduct experiments using different ablated versions of SAF in cooperative tasks with different levels of heterogeneity and coordination. As seen in Figure 6, our experimental results indicate that the knowledge source contributes to the performance of SAF in all cooperative tasks while the shared pool of policies significantly improves the performance of the agents in heterogeneous environments and has minimal contribution to tasks requiring high coordination.  C ONCLUSION
In this work, we explore coordination and heterogeneity levels of cooperative MARL environments by developing a set of environments, HECOGrid, which allows full quantitative control over coordination and heterogeneity levels. Moreover, we propose a novel algorithm that enables agents to perform well in difficult environments with high levels of coordination and heterogeneity. Our experimental results suggest that high coordination and heterogeneity do make cooperative tasks challenging and our SAF method allow agents to gain better performance in these environments. However, when the coordination and heterogeneity levels are further increased, all presented methods failed to gain significant rewards if any, and that suggests that further work needs to be allocated to address these limitations which we hope can be better studied through our HECOGrid environment suite.
E THIC STATEMENT
To the best of the authors’ knowledge this study does not involve any ethical issues.  R EPRODUCIBILITY The authors aim to maximize the reproducibility of the study.
The codes of this project including the new environment constructed will be released in the camera-ready version.
In the methods section, notions align with existing literature.
A detailed description of each step in the SAF algorithm is given in the method section and a full algorithm is provided in the appendix
A A PPENDIX A. A DDITIONAL R ELATED W ORK
Information Bottleneck.
With the emergence of modular deep learning architectures (Vaswani et al., 2017; Goyal et al., 2021b; Scarselli et al., 2008; Bronstein et al., 2017; Kipf et al., 2018; Battaglia et al., 2018) which require communication among different model components, there has been a development of methods that introduce a bottleneck in this communication to a fixed bandwidth which helps to communicate only the relevant information. (Liu et al., 2021) use a VQVAE (Oord et al., 2017) to discretize the information being communicated. Inspired by the theories in cognitive neuroscience (Baars, 1988; Shanahan, 2006; Dehaene et al., 2017), (Goyal et al., 2021c) proposes the use of a generic shared workspace which acts as a bottleneck for communication among different components of multi-component architectures and promotes the emergence of specialist components.
We use SAF , which is similar to the shared workspace that different agents compete to write information to and read information from.
Communication in MARL.
Communication involves deciding which message to be shared and determining how the message-sending process is implemented.
Foerster et al.
and Sukhbaatar et al.
(2016) implemented learnable inter-agent communication protocols.
Jiang & Lu (2018) first proposed using attention for communication where attention is used for integrating the received information as well as determining when communication is needed.
Das et al.
(2019) uses multiple rounds of direct pairwise inter-agent communication in addition to the centralized critic where the messages sent by each agent are formed by encoding its partial observation, and the messages received by each agent are integrated into its current state by using a soft-attention mechanism.
Kim et al.
(2020b) uses intentions represented as encoded imagined trajectories as messages where the encoding is done via a soft-attention mechanism with the messages received by the agent.
Wang et al.
(2021) trains a model for each agent to infer the intentions of other agents in a supervised manner, where the communicated message denotes the intentions of each agent.
The above-mentioned approaches require a computational complexity that is quadratic in the number of agents whereas our approach has a computational complexity that is linear in the number of agents.
Moreover, we show that our approach is able to outperform several standard baselines using messages which can be computed as simply encoding each agent’s partial observation.
Weis et al.
(2020) developed a transformer-based multi-agent reinforcement learning method that models MARL decision-making as a sequential model.
Coordination in MARL and the Pareto-optimal Nash equilibrium .
In the field of MARL, coordination is usually defined as the ability for agents to make optimal decisions to achieve a common goal by finding an optimal joint action in a dynamic environmentChoi & Ahn (2010);
Kapetanakis & Kudenko (2004).
One way to find the optimal joint actions by a group of agents is by studying the Pareto-optimal Nash equilibrium Zhukovskiy & Kudryavtsev (2016), which describes the optimal solution as one in which no agent’s expected gain can be further increased without compromising other agents’ gain.
However, there exist several challenges in cooperative MARL systems to achieve Pareto-optimal solutions.
In the following sections, we are going to explain three of these challenges, which are the ones we seek to tackle in this study as well as their links to coordination and environmental heterogeneity levels.
Environmental heterogeneity and the non-stationarity problem In MARL, the transition probabilities associated with the action of a single agent change over time as the action dynamics of the other agents change (Bowling & Veloso, 2000).
To solve this problem of non-stationarity, most recent MARL methods follow the Centralized Training Decentralized Execution paradigm.
The most extreme case of centralized training is when all agents share the same set of parameters.
However, parameter sharing also assumes that all agents have the same behaviors, which is not true when there is heterogeneity either among the agents themselves or in the environment.
Previous studies use indicator-based methods to personalize a shared policy in a group of heterogeneous agents(Terry et al., 2020), however, environmental heterogeneity has been less explored in the literature (Jin et al., 2022).
Environmental heterogeneity and the alter-exploration problem Another problem environmental heterogeneity may cause in cooperative MARL is the alter-exploration problem.
The balance between exploration and exploitation is crucial for all reinforcement learning tasks.
In cooperative MARL this problem arises when exploration of one agent may penalize other agents and their corresponding policies during training as the cooperative agents share rewardsBoutoustous et al.
; Matignon et al. (2012).
Environmental heterogeneity could potentially lead to worse alter-exploration problems as there tend to be more unseen states for an exploring agent which may result in higher and more frequent penalties.
In this study, we seek to solve the above-mentioned problem using a combination of inter-agent communication via an active facilitator and a shared pool of policies.
Existing Environments in MARL.
Some of the existing benchmarks based on online multiplayer games, attempt to move away from the toy-like grid world setting for MARL environments in favor of more realistic environments, by making use of high-dimensional observation and action spaces, continuous action spaces, challenging dynamics, and partial observability (Vinyals et al., 2017; Berner et al., 2019; Leibo et al., 2021; Zheng et al., 2017).
These benchmarks focus on decentralized control in cooperative tasks and agents are heterogeneous (i.e., different types of agents having different abilities).
In principle, it is possible to vary the levels of coordination and heterogeneity since the difficulties of different environments vary.
However, there is no well-defined notion of the two concepts and these can’t be varied in a controlled fashion.
MeltingPot, which was recently proposed in (Leibo et al., 2021) focuses on test-time generalization abilities of a group of agents includes a wide range of scenarios: competitive games, games of pure common interest, team-based competitive games, and mixed motion games which stress test the coordination abilities of the agents.
However, similar to other benchmarks, there is no systematic decomposition nor a quantitative notion of the concepts of coordination and environmental heterogeneity .
A. A DDITIONAL R ESULTS
In this section, we show the training curves for SAF , MAPPO and IPPO on KeyForTreasure ,
TeamSupport and TeamTogether environments.
We additionally present more ablation results in the Out-of-Distribution setting.
Performance when increasing coordination or/and heterogeneity levels.
SAF shows significant performance improvement upon MAPPO and IPPO at coordination levels 1 and 2 (Figure 3).
In addition, SAF shows faster performance increase at an early stage of the training process (figure A.2).
This suggests potential advantages for training agents using SAF in cooperative tasks requiring high coordination.
At most heterogeneity levels, SAF shows performance improvement upon MAPPO and IPPO in all HECOGrid environments and a faster increase in performance at early stages of the training (Figure 4 and A.2).
This suggests potential advantages for training agents using SAF in a cooperative environment which are heterogeneous.
In addition to manipulating coordination and heterogeneity levels separately, experiments are conducted to understand if SAF can perform well in environments in which both parameters are high.
In the relatively easy TeamSupport environment with both coordination and heterogeneity set at 2, 3 and 4, SAF again shows improved performance over IPPO and MAPPO (Figure 7(c)).
Figure 7(a) shows the training curves for SAF ,
MAPPO, and IPPO on KeyForTreasure , TeamSupport and TeamTogether environments for a coordination level of 2.
It shows a gap in performance between SAF and the baselines and this gap is further enlarged when it comes to a heterogeneity level of 2 (see Figure 7(b)) which shows that
SAF is effectively able to handle changes in the environment’s dynamics.
[Figure]
[Table]
Ablation Study for OOD generalization Figure 8(a) shows test-time generalization results on the TeamTogether and TeamSupport environments where the training coordination level was set to 2 and the heterogeneity was set to 1.
The pool of policies in SAF is important in getting good performance especially when it’s tested on levels of coordination not seen during training.
Moreover,
Figure 8(b) further validates that the pool of policies is important in handling varying environment dynamics as SAF was trained on a heterogeneity level of 2 and a coordination level of 1 and the results on unseen levels of heterogeneity are better than SAF trained without a pool of policies.
These ablations show that the introduced pool of policies in SAF is key to its performance.
[Figure]
coordination level 2 and heterogeneity level 1 are tested across different coordination levels (Figure 8(a)).
In a similar manner, The models trained on coordination level 1 and heterogeneity level 2 are tested across different heterogeneity levels (Figure 8(b)).
A. A LGORITHM
In this section, we summarize the components of SAF and its computations in the form of a pseudocode outlined in algorithm 1.
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]