Self-Organized Group for Cooperative Multi-agent Reinforcement Learning
Abstract
Centralized training with decentralized execution (CTDE) has achieved great success in cooperative multi-agent reinforcement learning (MARL) in practical applications.
However, CTDE-based methods typically suffer from poor zero-shot generalization ability with dynamic team composition and varying partial observability.
To tackle these issues, we propose a spontaneously grouping mechanism, termed Self-Organized Group (SOG), which is featured with a conductor election (CE) and a message summary (MS) mechanism.
In CE, a certain number of conductors are elected every T time-steps to temporally construct groups, each with conductor-follower consensus where the followers are constrained to only communicate with their conductor.
In MS, each conductor summarize and distribute the received messages to all affiliate group members to hold a unified scheduling.
SOG provides zero-shot generalization ability to the dynamic number of agents and the varying partial observability.
Sufficient experiments on mainstream multi-agent benchmarks exhibit superiority of SOG.
Introduction
Cooperative multi-agent deep reinforcement learning algorithms (MARL) have been ubiquitously applied to real-world scenarios, such as autonomous vehicle teams [45], sensor networks [48], and social science [13].
Recently MARL has gained extraordinary performance in various multiplayer games like Dota [3], StartCraft [30] and soccer [16].
The framework of centralized training with decentralized execution (CTDE) [8, 28] is one of the popular frameworks for solving cooperative multi-agent tasks.
Centralized training renders the CTDE paradigm better agent cooperation while independent execution endows the multi-agent system with more efficiency and scalability.
Classical CTDE algorithms like QMIX [28] and MADDPG [21] are confined to the fixed size of agents.
However, the number of involved agents tends to vary in real-world multi-agent scenarios.
Recently, some methods introduce the attention mechanism [40] to train on the varying number of agents simultaneously [12, 11].
They only seek for a solution covering a range of varying team size, unable to provide the generalization ability to the cases not in the range.
Agarwal et al.
[1] and Liu et al.
[18] further introduce the communication mechanism to provide the adaptability on the dynamic team composition (i.e., the team size varies).
In the paper, we propose a method called “Self-Organized Group (SOG)”to possess zero-shot generalization for multi-agent reinforcement learning, which provides strong adaptability to the scenarios with varying number of agents and even varying sight of agents.
In spirit of the individuals’ cooperation in social science [10] and the leader-following formation in multi-agent system [9], part of the agents are elected as conductors at intervals, and for each conductor a temporary group is formed with conductor-follower consensus.
In each group, the followers can only communicate with the conductor, while the conductor summarizes all the received messages and distributes the refined message to all group members aiming for a unified target.
Our insight is that, an organized group under the unified command of a conductor can better adapt to an unseen scenario than individuals.
To be specific, each agent’s behavior is prone to diverging from the training pattern when presented with an unseen scenario.
They may behave in contradictory ways and pose a threat to the system’s stability.
On the contrary, a conductor could send unified commands, making each agent behave in order, thus avoiding the overall collapse of the training pattern.
In the meantime, our method largely reduces the bandwidth and the cost required for communication.
For efficient and economical message passing, we also design a variational message summarizer (MS), which compresses the local observation information and helps predict the future trajectories.
Since the conductor navigates the information propagation in the whole group, Conductor Election (CE) plays a critical role in SOG.
Except for the simple random CE which satisfies the fully decentralized execution paradigm, we further propose a heuristic variant: determinantal point process (DPP) [22]-based CE, and a learning-based variant: policy gradient (PG) based CE.
DPP-based
CE chooses conductors in the principle of maximum entropy and avoids the occurrence of homogeneous conductors.
PG-based CE formulates the CE as a learning process and takes the long-term benefits of the selection into consideration.
We conduct experiments on three commonly used multi-agent benchmarks, including a resource collection task, a predator-prey task, and a set of customized StarCraft micromanagement tasks.
To validate the zero-shot generalization ability of our method, we evaluate trained models in more complicated scenarios than training.
We not only increase the number of agents for evaluation, but also adjust the agents’ utility, e.g., reducing the sight range of each agent.
The results show that SOG has better zero-shot generalization ability, not only for the dynamic team composition, but also for the unseen environment condition, than current state-of-the-art methods on all three benchmarks.
The contribution of our paper is summarized as follows: (1) We propose Self-Organized Group, a mechanism for enhancing agents’ zero-shot generalization ability.
(2) We design two conductor election ways for reasonable group forming and a variational message summarizer for efficient and economical message passing.
(3) We achieve better generalization ability on three commonly used multi-agent benchmarks than current state-of-the-art methods.
Related Work
Centralized Training with Decentralized Execution .
We mainly focus on the typical paradigm of centralized training with decentralized execution (CTDE) for cooperative multi-agent tasks in this part.
It combines the advantages of independent Q-learning [38] and joint action learning [4].
A series of work concentrate on factorizing Q functions [35, 28] and deriving theoretical guarantee for the policy optimality [33, 41, 29].
Some other works use actor-critic methods with a centralized critic [21, 7, 50].
Although the CTDE paradigm shows great empirical performance in many multi-agent tasks, it may fail on some simple situations without communication on execution due to the partial observability of the local agent [18].
In addition, traditional CTDE methods usually concentrate on invariable environment conditions and fixed team size.
Communication for MARL .
Many efforts have been made for communicating agents in multi-agent systems [32].
Recently some works assume the information can propagate among homogeneous agents during decentralized execution [6, 36, 49, 34, 19], some of which use the graph neural network to handle the message passing [23, 24, 26, 24].
Some works focus on communication between heterogenous agents [25, 31], but the role of each agent is usually fixed, therefore limiting the model’s generalization ability.
These works usually pre-define the number of agents, and can hardly handle the dynamic team composition.
Liu et al.
[18] uses a predefined coach with global information to instruct dynamic number of agents.
However, a global coach that can communicate with all agents is not often available in real-world domains.
In contrast to these methods, our method enables all agents to become a conductor, while reserves the efficiency and scalability of CTDE.
Transfer Learning and Curriculum Learning .
An important topic for multi-agent systems is the adaptation to the mutable environment.
A bunch of works attempt to deal with the increasingly challenging environment by designing a set of curricula [2, 20, 44].
Another line of works utilize the attention mechanism or graph neural networks to handle the diversity of the local observation [14,
Wang et al.
[43] allocates roles with limited action space to agents, and the study shows the model’s ability of task transfer by increasing the action space for role clusters.
Iqbal et al.
[12] trains simultaneously on multiple tasks by randomly masking out some entities the agent observes.
Unlike these methods, our method bypasses the overlarge training burden of curriculum learning, and is more robust in the dynamic team composition and varying environment conditions.
Background
In our work, we consider a fully collaborative multi-agent task with n agents, which can be modeled as a decentralised partially observable Markov decision process (Dec-POMDP) [27]
G = ⟨ S, A, I, P, r, Z, O, n, γ ⟩ , where s ∈ S is the true state of the environment.
At each time step t , each agent i ∈ I ≡ { 1 , ..., n } chooses an action a i ∈ A , which is an element of a joint action
a ∈
A ≡ A n .
P ( s t +1 | s t , a t ) : S ×
A × S → [0 , 1] is the state transition function of the environment.
All agents share the same reward function r ( s, a ) : S ×
A → R .
γ ∈ [0 , 1) is the discount factor.
Due to the partial observability , each agent i has its local observations o i ∈
O drawn from the observation function Z ( s, i ) :
S × I → O .
In our setting, we assume agents can communicate within its sight range.
Each agent i may receive a message µ i ∈ U from its neighbors.
Each agent chooses an action by its stochastic policy π i ( a i |
ρ i , µ i ) :
Γ × U → ∆([0 , 1] | A | ) , where ρ i ∈ Γ ≡ ( O × A ) ∗ denotes the action-observation history of agent i , and ρ is the action-observation histories of all agents.
The agents’ joint policy π induces a joint action-value function : Q π ( s t , a t ) =
E s t +1: ∞ , a t +1: ∞
[ R t | s t , a t ] , where R t = (cid:80) ∞ k =0 γ k r t + k is the discounted accumulated reward.
The goal of our method is to find the optimal joint policy π ∗ such that Q π ∗ ( s, a ) ≥ Q π ( s, a ) , for all π and ( s, a ) ∈ S × A .
In practice, to handle the input with dynamic team size, we use the entity-wise input rather than the vector input, detailed in Appendix A.
We use the popular centralized training with decentralized execution (CTDE) paradigm [8, 28] as the baseline, also allowing the communication between adjacent agents.
During training, the method has the access to the full state and each agent’s action-observation history.
During testing (execution), each agent only has the access to its own action-observation history and communication messages from neighbors.
Method
In this section, we first highlight the importance of the communication during decentralized execution, and propose self-organized group for decentralized communication.
Next we introduce the learning objective of the communication message.
Finally we present the whole training procedure together with the model architecture.
4.1 Downsides of CTDE
Consider a simple scenario in Fig. 1(a).
The task is fulfilled only when the three buttons are pressed synchronously.
Therefore, agents
A, B, and C need to press the three nonrecurring buttons respectively.
The button 1 andare ten meters apart, while button 3 is at the midpoint of them.
The sight range of each agent is a circle with radius six meters.
As a result, the agent who is in charge of the button 2 can’t perceive if some agent is pressing the button 1, due to sight range limitation.
If we use the common centralized training with decentralized execution (CTDE) method, like QMIX [28], to deal with this task, these agents can not efficiently coordinate beyond the individual sight range, since each agent’s execution is conditioned on its own action-observation history.
The communication between agents is one of the easy and flexible solutions to handle the partial observability.
To extend the CTDE paradigm based on the communication mechanism, we now present the SelfOrganized Group (SOG), a mechanism to group all agents and select a conductor for each group.
4.2 Self-Organized Group
Aiming for a cooperative multi-agent task with dynamic team composition, every T time-steps, we elect a certain number of conductors and the groups are constructed accordingly.
In this section we first show the formulation of SOG.
Then we present the message summarizer in Sec.
4.2.1, and we discuss in Sec.
4.2.2 the strategies for conductor election.
An example group is illustrated as in
Fig. 1(b), where agent C is appointed as the conductor, with the collected information from A and
B to broaden the sight range.
Since our main task lies in MARL, we simplify the communication procedure, assuming that all the messages can be delivered accurately without delay and each agent can handle all the received messages at the same time.
Our goal is to build a communication mechanism that satisfies the following properties:
• Lightweight .
The communication message should be brief and informative, e
g., a threedimensional summary vector of the agent’s current state.
• Robust .
The group should be able to handle the varying number of group members and the unfamiliar environmental conditions (e.g. the varying sight range).
The process of group organization is as follows (Fig. 2): every T time-steps, some agents are elected to be conductors.
We allow multiple conductors in the same time.
Then conductors send group invitations to all agents within their sight ranges.
The non-conductor agents who receive one invitation then send their personal messages ζ back to the conductor.
Here the personal message ζ is delicately designed by a message summarizer, described in Sec.
For non-conductor agents who receive multiple group invitations, they randomly choose one as the conductor and reply.
The conductor and its interactive agents temporarily form a group.
For those who receive no group invitation or leave the sight of the constructor during in-group communication, they form a 1-agent group of themselves.
After the group is formed, the conductor processes the messages from all the followers (including itself).
Then it sends the processed message ξ j back to each member that belongs to group j .
Each agent only needs to communicate with its group’s conductor, therefore it greatly reduces the bandwidth required for communication.
We use a non-parametric message mixer for each conductor:
[Equation]
where M j is the size of group j .
In this way, the conductor takes no extra computational cost for message mixing and distributing procedure, and each agent can play the role of the conductor without changing its network structure.
The communication takes place every T time-steps.
Each agent i uses its local observation
o i and the message µ i to predict its action.
µ i is defined as:
[Equation]
where I ( t, T ) is the indicator whether t is divisible by T .
Through such mechanism, agents can share the information of the whole group with the minimal communication cost and coordinate more efficiently.
We present the quantitative analysis on the communication cost in Appendix H, which is 1 /M of the cost of the fully-connected communication like Agarwal et al.
[1], where M is the average number of neighbors for each agent.
In unfamiliar scenarios, the conductor integrates the followers’ information and all group members execute a unified command, driven by a delicate message summarizer.
Such a in-group communication mechanism empowers the system better stability compared to the non-conductor-following formation.
With time-varying conductor election, SOG shows strong zero-shot generalization ability to the dynamic team composition and unseen environmental conditions.
4.2.1 Message Summary (MS)
After the communication protocol is built, we now present the content of the communication message.
Inspired by recent works on role learning and state summary by variational inference [42, 18, 17], we propose a message summarizer for agents to summarize their local observations as brief latent variables.
A summarizer can extract valuable information from the whole trajectory, and reduce the computational complexity.
We first aim to distill the information about the agent i ’ s transition of future T -steps.
Let τ it =
( o it +1 , a it +1 , ...,
o it + T − 1 , a it + T − 1 ) .
The personal message is a random Gaussian variable of
C dimensions sampled from an encoder, i.e., ζ it ∼ f ψ ( o it ) .
We maximize the mutual information between ζ it and τ it conditioned on o it .
I ( ζ it ; τ it | o it ) has the following lower bound:
[Equation]
where q ϕ ( · ) is the variational estimator, which is only used for centralized training.
We define the opposite of the lower bound as future predictor loss L FP .
The derivation is defered to Appendix D.
In addition, we expect the personal message to help discard the irrelevant information while retain the future trajectory’s information, which can endow the agent with a small state representation and accelerate training.
We take advantage of the conditional entropy bottleneck (CEB) objective [5].
We maximize the mutual information between the personal message ζ it and the future trajectory τ it , and minimize the mutual information between the current observation o it and the personal message ζ it conditioned on the future trajectory τ it simultaneously.
The objective has the following lower bound:
[Equation]
where f ψ ( · ) is the encoder for personal message, and K is the size of the mini-batch sampled for training.
We define the opposite of the lower bound as L CEB .
See Appendix D for detailed derivation.
4.2.2 Conductor Election (CE)
“MHA” means multi-head attention.
We use this module to deal with the entity-wise input for dynamic team composition, detailed in
Appendix A. h i t is the hidden state for GRU cell.
In this section, we introduce three conductor election mechanisms: Random CE, Determinantal Point
Process (DPP) [22] based CE, and Policy Gradient [37] based CE.
Note that CE differs from the concept of leader election (LE) [39] in distributed system.
We elect conductors to obtain better agent cooperation for maximizing the team reward, and we assume a perfect communication channel.
LE is usually deployed in real multi-agent systems with the probability of agent disconnection, with the target to maintain the cognitive consistency among agents.
Random CE .
Each agent is elected as a conductor with an independent and identical probability p l , which is fully decentralized.
The expectation of the number of groups can be controlled by p l .
Since all group members need to communicate with their neighbors only for message passing, SOG with random CE doesn’t need centralized commander during execution.
Therefore, it can be parallelized easily during execution.
DPP-based CE .
In this part we expect to increase the generalization ability of SOG by maximizing the intra-group diversity.
We achieve this by maximizing the diversity between conductors considering the conductor is the structural center of a group.
Specifically, we formalize the conductor election as a determinantal point process P .
Definition 1.
A determinantal point process (DPP) P is a probability distribution defined on the power set 2 Y of a discrete finite basic point set Y = { 1 , 2 , ..., n } .
L ∈ R n × n measures the similarity between any point pairs in Y .
Let B be a random subset drawn by P , then ∀ C ⊆
Y , we have
[Equation]
where L C :=
[ L i,j ] i,j ∈ C is the submatrix of L , whose entries are indexed by the elements in C , and
I is an n × n identity matrix.
det( · ) means the determinant of a matrix.
In practice, we use all agents to construct the point set Y .
Correlations are always non-positive in
DPP [15].
The more similar two points are, the less possible they will appear in a subset sampled by DPP.
Therefore, once we need to elect conductors, we just sample a subset B by DPP and use all elements in B as the conductors, which greatly reduces the probability to elect conductors with similar observations (e.g., two agents close to each other).
For the similarity matrix L , we use the cosine similarity of agents’ personal feature V i :
[Equation]
Another concern for DPP is the high computational cost for calculating the determinant of all submatrices.
Instead of the traditional sampler by Schmidt orthogonalization [47], we find a method to calculate all determinants in parallel with a GPU, which greatly accelerates the computation.
Details can be found in Appendix B.
PG-based CE .
The conductor election might affect the group’s accumulative rewards in the following time-steps.
Therefore, we regard the conductor election task as a reinforcement learning problem.
We use a policy π ( a | s ) =
(cid:81) ni
=1 π i ( a | s ) with the input of global state to decide the probability of each agent to become a conductor, and do policy gradient on it.
The gradient ∇L CERL can be written as:
[Equation]
where ˆ t is the time that CE happens, i.e., every T time-steps.
We omit the π ’s dependence on the parameter ω of the neural network for brevity.
Electing conductors by PG takes the long-term benefits of the selection into consideration, and in experiments it shows great generalization ability.
4.3 Algorithm Outline
As shown in Fig.
3, our model is constructed using centralized training with decentralized execution paradigm.
We utilize the entity-based input like in Iqbal et al.
[12], so that the model can deal with the dynamic number of agents.
The partial observability of local agents is assured by masking out the unseen entities.
All agents share the same parameters.
It’s worth noting that the variational estimator q ϕ is designed to include information of future T -steps.
Since at time-step t it takes the output of
GRU cell h it as input, q ϕ should be used to calculate the loss with message ζ before T steps, i.e. ζ it − T .
The local Q function uses the message ξ j sent by the conductor as part of input if it is in group j when communication takes place, and uses personal message ζ i in place at other time.
Let µ t denote the set of the messages used for local Q prediction at time-step t .
The mean square Bellman error objective for Q-learning is as follows:
[Equation]
where ˆ θ is the parameters of the target network, and D is the replay buffer.
The overall loss can be written as:
[Equation]
where λ 1 and λ 2 are hyper-parameters.
We summarize the training procedure in Algorithm 1.
Experiments
In this section we design experiments to answer the following questions: (1) Whether the SelfOrganized Group helps the agents coordinate better than fully-connected communication methods?
Algorithm
Initialize θ, ϕ, ψ .
Set learning rate ← η , communication interval ← T , D ← {} , λ 1 , λ 2 , t max for each episode iteration do for t = 1 , 2 , . . .
, t max do if t mod T ̸ = 0 then
For each agent i , µ it ← Sample ( f ψ ( o it )) else
Elect conductors and form groups { G j } Kj =
For each agent i , ζ it ← Sample ( f ψ ( o it ))
For each conductor j , ξ jt ← Mix ( ζ it ) , i ∈ G ( j )
For each agent i , µ it ← ξ jt , i ∈ G ( j ) end if
Generate tuple { s t , a t , r t , s t +1 , ζ t , ζ t +1 } by executing f θ ( τ t , µ t ) end for
D ← D ∪ { s t , a t , r t , s t +1 , ζ t , ζ t +1 } t max t =1 , θ ← θ + η ˆ ∇ θ L RL ( D ) ψ ← ψ + η ( ˆ ∇ ψ L FP ( D ) + ˆ ∇ ψ L CEB ( D )) , ϕ ← ϕ
+ η ( ˆ ∇ ϕ L FP ( D ) + ˆ ∇ ϕ L CEB ( D )) end for (2) Can Self-Organized Group promote zero-shot generalization to the dynamic team composition and the varying partial observability, even on complicated tasks?
(3) What’s the effect of the conductor election way?
(4) What factor contributes to Self-Organized Group most?
We test our idea on three commonly used multi-agent benchmarks: Resource Collection, Predator-Prey, and StarCraft II micromanagement tasks.
The previous two are built on the multi-agent particle environment [21], and the last is modified from Iqbal et al.
The number of the agents is not fixed, sampling from a pre-defined set at the beginning of each episode in all scenarios.
For all the three tasks the testing environments are more complicated than the training ones.
Agents need to handle the situations that is not encountered during training.
Each experiment is repeated 3 or 5 times with different seeds.
In experiments, the name “SOG” means our method with random CE, whose execution is decentralized.
“SOG_dpp” and “SOG_rl” means the SOG with DPP-based CE and PG-based CE, respectively.
Except for our method, we evaluate four state-of-the-art CTDE methods that are suitable for dynamic team composition: A-QMIX [46], REFIL [12], EMP [1] and COPA [18].
For a fair comparison, we extend the original EMP’s decentralized-training structure to CTDE.
For A-QMIX and REFIL, agents have no communication during execution, while for EMP and COPA, agents communicate with their neighbors or a global coach.
Since many MARL methods with communication are not suitable for dynamic team composition (e.g., CommsNet [34] and G2ANet [19]), we extract one scenario of Resource Collection and show results in Appendix G.1.
We also compare SOG with graph-based methods including NCC [23], MAGIC [26] and Gated-ACML [24] in Appendix I.
5.1 Resource Collection
This scenario is modified from the environment described in Liu et al.
Agents need to collect resources from 6 resource points and transport the goods home.
The number of the agents for training is uniformly sampled from {2,3,4,5}, while for testing it is sampled from {6,7,8}.
Each agent has a sight range SR .
Entities including other agents and resource points that exceed agent i ’s SR are invisible to agent i .
For more details of the environment, please refer to Appendix F.1.
We train on two kinds of SR : 0.5 and 1.0.
We show in Fig.
4(a-b) the results of all methods.
SOG significantly outperforms the other methods when SR = 0 .
5 and SR = 1 . 0 .
Given that the testing scenario includes 6/7/8 agents while the training scenario only has 2/3/4/5, the communication mechanism of SOG greatly strengthens SOG’s zero-shot generalization ability for dynamic team composition.
The CE methods have no obvious difference when the team size varies.
Then we test the zero-shot generalization ability in different environmental conditions.
We save the previously trained models with a fixed sight range 0 .
5 or 1 .
and evaluate them on different agent sight range settings.
Each setting is repeated for 160 testing episodes and the results are averaged over 3 models with different seeds.
We show the performance of each method in Fig.
4(c-d).
Except for the difficult setting SR = 0 .
2 that all methods have poor performance, SOG and SOG_rl can achieve similar or even better performance in different scenarios ranging from SR = 0 .
5 to SR = 1 . 0 than in training scenarios.
SOG_rl even performs better transferring from SR = 1 . 0 to SR = 0 .
In the meantime, other methods show notable performance drop when transferred from training settings to others.
The results show SOG’s strong adaptation to unseen scenarios.
5.2 StarCraft Micromanagement Tasks
[Figure]
We apply our method to the StarCraft multi-agent challenge (SMAC) [30].
We use the map designed by Iqbal et al.
[12] and Liu et al.
The maps randomly initialize 3-8 agents with enemies of the same number at the start of each episode.
Agents are initialized together with the sight range 9 [12] or divided into 2-4 groups with the sight range 3 [18].
We call the previous one “G(ather)” and the latter “D(isperse)”.
In the scenario “Disperse”, the enemies are divided into 1-2 groups, which means each group of enemies is stronger than that of the agents.
We limit the number of agents to 3-5 in training procedure and test the model with agent number 6-8.
Similar to Iqbal et al.
[12], Liu et al.
[18], we also incorporate the imaginary objective.
We first explore the adaptation to the dynamic team composition.
We show in Fig.
5(a)
the results on 4 maps.
Our method has the highest test winning rate against the preset AI when testing the model with a larger size of agents and enemies than training.
And the PG-based CE improves the performance on 3-8sz.
Combining the results in Resource Collection, we conclude that PG-based CE can improve agents’ zero-shot generalization ability on dynamic team composition in many cases, due to the more reasonable conductor election mechanism than random CE.
Then, we study the adaptation to the varying environment conditions of our method.
We save the models trained on map 3-8sz_D with the sight range 3, which is a hard case that agents are split and have restricted horizons.
Then we evaluate the models on other initial conditions and sight ranges.
The results are shown on Fig. 5(b).
The initial conditions (Gather or Disperse) have no obvious effect on the performance for all four methods.
However, when agents can see more entities than training, other three methods all show large performance drop while our method has comparable performance to that in training.
This is probably due to the unified messaging passing mechanism within the group.
An interesting phenomenon is that the random CE has comparable or better performance to rl-based CE on 2 scenarios.
We speculate this is due to the similar pattern of random CE in different environment settings, and analyse this result in detail in Appendix G.2.
The results show that our method can better adapt to the varying partial observability.
To further validate the effect of our method, we visualize the situation for four methods (SOG, SOG-rl,
SOG without MS, and EMP3) on the same SMAC map in Fig. 5(c).
We can see that agents trained by
SOG learn to gather and concentrate fire on certain enemies surrounded by agents.
SOG-rl sacrifices a group of 2 agents to exchange the the elimination of 3 enemies.
SOG without message summary learns to gather, but the target of each agent’s attack is not focused.
EMP3 does not even learn to attack simultaneously.
The results show that CE and MS both promote better agent cooperation.
5.3 Predator Prey
In this scenario, agents play the role of predators, aiming to catch some preys with random walk.
The training number of predators is sampled from {3,4}, along with only 1 prey.
During execution we initialize 5 or 6 predators, as well as 1 or 2 preys.
For environment details, please refer to
Appendix F.2.
It is a relatively easy-to-learn scenario, and we make ablations on it.
[Table]
SR" is the average return of 160 testing episodes on simple training scenarios, while "CR" is the counterpart on complicated evaluation scenarios.
First, we explore the effect of the message summarizer.
Except for the L FP and L CEB , we also try an entropy regularizer similar to the implementation of Wang et al.
It aims to maximize the entropy of the message to encourage exploration.
As shown in Table 1, the combination of L FP and L CEB obtains the best average testing return, exceeding the performance of CTDE method A-QMIX, and communication method EMP.
Although the entropy regularizer helps stabilize the training (the variance is reduced when combined with the regularizer), it makes no contribution to the average performance.
Therefore we use the L FP + L CEB as our default loss for message summarizer.
Notice that EMP has similar performance to SOG on training scenarios.
Nevertheless, SOG performs better when transferred to complicated evaluation scenarios.
[Table]
Then we analyze the choice related to communication in
Table 2, including the number of groups in expectation, the message dimension, and the communication interval.
The number of group 2 has better performance than number 1 or 4.
Since the rule that a prey needs to be caught by 3 predators simultaneously, it is suitable to divide 5 or 6 agents into 2 groups.
When the message dimension is reduced from 10 to 3, the results have no obvious drop.
However, it decrease greatly when the message dimension is set to 1.
The results show that a 3-dimensional message is sufficient for communication for Predator-Prey task.
As for communication interval, T = 2 is a little better than T = 4 , and T = 10 is not enough for sufficient communication.
Discussion on Limitations
As stated in the section 4.2.2, we only test SOG on the perfect communication channel.
Its performance may fall down when faced with broken communication channel.
Another limitation for random
CE is that the expectation of the size of each group is decided by a pre-defined hyperparameter, i.e., the probability of agent elected as a conductor.
When transferred to unseen scenarios, if the size of each group differs a lot from the training condition, it may cause the performance drop.
We try to git rid of the hyperparameter by introducing DPP-based CE and RL-based CE, but they both require a centralized conductor elector, which is not full CTDE.
Negative Societal Impact
There is a possibility that our method could be employed in real-world multi-agent systems such as UAV formation or intelligent warehouse management.
Utilizing the policy our method derives directly is risky, since there is a domain gap between the training virtual environment and real-world scenarios.
The practitioners are supposed to restrict their policy under human supervision to avoid harmful options.
Conclusion
In this paper we propose Self-Organized Group (SOG) for cooperative multi-agent reinforcement learning.
In SOG, a certain number of agents are randomly elected to be conductors and the corresponding groups are constructed with conductor-follower consensus, allowing the groups to be re-organized every T time-steps.
We find the organized group under the unified command of a conductor embeds the multi-agent system with stronger zero-shot generalization ability compared to the traditional CTDE methods with fully-connected communication mechanism.
Furthermore, we derive a variational message summarizer for efficient and economical message passing, and we propose DPP-based and PG-based conductor election strategies for better group organization.
We take experiments on three commonly used multi-agent benchmarks.
SOG shows better zero-shot generalization ability not only for the dynamic team composition, but also for the varying partial observability on all three benchmarks.
Acknowledgement
This work was supported by the National Key R&D Program of China under Grant 2018AAA0102801,
National Natural Science Foundation of China under Grant 61620106005.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] .
(b) Did you describe the limitations of your work?
[Yes] See Sec.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Sec.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] . 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Sec. 3 and Sec. 4.2.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix D. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We submit the whole code.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[Yes] See Sec.
5 and Appendix E, F. (c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Sec.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix E. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] .
(b) Did you mention the license of the assets?
[N/A] .
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] .
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] .
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] .
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] .
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] .
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A] .
A Entity-wise Input
It’s critical to deal with the dynamic team composition in real-world multi-agent scenarios.
However, traditional methods usually fix the team size when the network is built.
A common solution is to introduce a multi-head attention ( MHA ).
To feed MHA, the raw state s of the environment is represented as a series of entities e i , i.e., s e := { e i } , i ∈ [1 , n e ] with the same vector length, where n e is the maximum number of entities.
The entities include agents we can or can not control, and other substrates in the multi-agent scenario (e.g. obstacles).
After some linear layers, s e is updated as a matrix X ∈ R n e × d where d is the dimensionality of the entity representation.
Each agent’s partial observability o i can be obtained by a mask m with size n e multiplied on s e .
Elements in m equal one if the corresponding entity is in the agent’s sight range else 0.
Combining all agents’ masks together, the matrix M ∈ R | I |× n e indicating all agents’ observability, where I is the agent set.
An
MHA module, denoted as MHA ( I, X , M ) , can integrate information not blocked by masks across all entities.
Taking only global entities as inputs during the decentralized execution, CTDE with
MHA can easily achieve agents’ partial observability with entity-wise expression.
Compared to the traditional agent-wise customized observation which may bring information redundancy, entity-wise input is concise and efficient.
For detailed structure of the MHA module, please refer to Iqbal et al.
B Implementation of the Fast DPP Sampler
If we want to sample from a DPP, we need to calculate the determinant of all submatrices of the similarity matrix L ∈ R n × n .
We show in Alg.
2 the procedure.
# n: number of agents.
# cos_matrix
: the similarity matrix
with
shape [
batch
_size, n,
n] # Generate all submatrices with
shape
n
×
n
Fill
the not-in-submatrix
place with 0.
mask =
2 ** torch.arange(n - 1, -1, -1) x
= torch.arange(2 ** n) y = x.unsqueeze(-1).bitwise_and(mask).ne(0)
sub_matrix_mask
= torch.logical_and(y.unsqueeze(1), y.unsqueeze(-1))
cofactor_matrix = cos_matrix.unsqueeze(1)
* sub_matrix_mask.unsqueeze(0) # shape
[batch_size, 2 n , n, n] # Fill the positions in the diagonal that are not in the submatrix with 1.
missed_ones =
torch.diag(torch.ones(n)).unsqueeze(0)*torch.logical_not(sub_matrix_mask)
cofactor_matrix += missed_ones.unsqueeze(0)
# Calculate the determinants in parallel.
p = torch.linalg.det(cofactor_matrix)
# Sample the conductors.
cat = Categorical(probs=p) conductors =
cat.sample() # Get a mask of shape [batch_size, n].
1 means a conductor, and 0 means a follower.
conductor_mask = conductors.unsqueeze(-1).bitwise_and(mask).ne(0)
C Code and Reproducibility
We commit our code in https://github.com/thu-rllab/SOG .
D Derivation of Message Summarizer
First we give the derivation of L FP .
To simplify the notation, we omit the superscript i and the subscript t , since the message is summarized by any agent i at any time-step t .
We fully derive the
Equation.
(3) as follows:
[Equation]
The inequality holds because any KL ( · , · ) ≥ 0 .
Then we derive L CEB .
For simplicity we omit the subscript t .
Recall that we want to maxamize − I
( o it ; ζ i | τ i )
+ I
( τ i ; ζ i ) .
The first term can be written as:
[Equation]
Since the message is summarized before the future trajectories are generated, the message ζ i is independent of the future trajectory τ i .
Therefore we can use a neural network parameterized by ψ , which only takes o i as the input, to represent the message summarizer, i.e., we can use f ψ ( ζ i
| o i ) to represent p ( ζ i | o
i , τ i ) .
To get a variational lower bound of the second term
I ( τ i ; ζ i ) , we use the
CatGen formulation, i.e., Equation.(5) in Lee et al.
[Equation]
where we estimate the mutual information
I ( τ i ; ζ i ) for a single example from a minibatch with size
K , and the K messages are sampled independently.
The future trajectories τ i depends on the current observation
o i , so we can reuse the variational encoder q ϕ ( ζ i |
o i , τ i ) to represent p ( ζ i | τ i ) .
Therefore, we can write the whole CEB objective as follows:
[Equation]
E Hardware and Hyperparameters
We use an AMD Ryzen 3975WX CPU with 32-Cores and three RTX-3090-11G GPUs to run all the experiments with three seeds.
One experiment of Resource collection and Predator-prey takes around 10 hours.
For StarCraft Micromanagement Tasks, it takes about 20 hours to run one experiment.
We summarize some hyper-parameters in Table.
F Detailed Environment Description F.1 Resource Collection
In Resource Collection, agents need to collect resources from 6 resource points and transport the goods home in a map of boundary [-1,1].
The location of the resource point and the home are random
[Table]
Clipping value
[Table]
places uniformly sampled from the whole map.
The radius of the home and the resource location & agent is 0.1 and 0.05, respectively.
There are 3 kinds of resources, and each agent i has its own ability b ei uniformly sampled from {0.1, 0.5, 0.9} to collect each kind of resource e .
Each agent can accelerate towards 4 directions or apply no forces at each time-step.
Each agent has its maximal speed uniformly sampled from {0.3, 0.5, 0.7} and the acceleration is fixed to 3.0.
Every time the agent i collects resource e , the team will get a reward 10 ∗ b ei .
When an agent brings the resource home, the team will get a reward 1.
An agent can only carry one resource at a time, which means an agent needs to bring the collected resource home before it starts to collect the next one.
The episode limit is 145.
The number of the agents for training is uniformly sampled from {2,3,4,5}, while for testing it is sampled from {6,7,8}.
Each agent has a sight range SR .
Entities including other agents and resource points that exceed agent i ’s SR are invisible to agent i .
F.
2 Predator Prey
In this scenario, agents play the role of predators, aiming to catch some preys with random walk in a map of boundary [-1,1].
The acceleration for predator/prey is 3.0/4.5.
The maximal speed for predator is uniformly sampled from {0.2, 0.3, 0.5}, while it is {1.0, 1.2} for prey.
At each episode there areobstacles with radius 0.05 generated, together with 2 tunnels.
A prey stepping into one tunnel will be immediately transported to the other.
Agents need to learn to cooperate and guard at the exit of the two tunnels to catch the preys.
Only when at least 3 predators touch the prey simultaneously, the prey is caught.
The team will get the reward 25 for catching one prey, and is penalized for − 0 .
1 ∗ (cid:80) j min i D ij at each time step.
D ij is the distance of agent i and prey j .
The episode ends if all preys are caught, or it reaches the episode time limit.
The training number of predators is sampled from {3,4}, along with only 1 prey.
During execution we initialize 5 or 6 predators, as well as 1 orpreys.
G More Results on Methods with Fixed Agent Input and Heuristic Conductor Election. G.1 Fixed Agent Input
Many communication baselines can only handle a fixed number of agents.
Therefore, it is difficult to apply them to our environments with dynamic team composition.
To make a fair comparison with them, we fix the agent number in Resource Collection to 6 and run 3 seeds for CommsNet [34],
G2ANet[19] and COPA [18] separately.
For COPA we also run it on the dynamic team composition in Fig. 6(a).
We show in Fig. 6(c)
the results.
On the fixed environment, COPA performs a little better than SOG, while CommsNet and G2ANet almost fail to learn effective policy.
We also test the trained model on different sight ranges in Fig.
6(d).
SOG shows better adaptation to unseen sight ranges than
COPA.
We emphasize that SOG is easier to distribute since it enjoys decentralized execution while
COPA, CommsNet and G2ANet are not fully decentralized.
G.2 Heuristic Conductor Election
The main body shows that random conductor election has a little better zero-shot generalization ability than DPP and policy gradient in some scenarios.
In Fig.
6(a,b), we also add two kinds of heuristic conductor election mechanisms in comparison.
The first (SOG-LD-M) elcets conductors with the maximal neighbor numbers, while in the second manner (SOG-LD-P), the probabilities of agents elected as conductors are proportional to their neighbor numbers.
We can see that SOG-LD-M performs a little better for the unseen agent numbers, while both heuristic conductors (and COPA, which has a central conductor) show obvious performance drop in unseen sight ranges.
Possible reasons are as follow: Although in fixed environments, a delicate conductor election can obviously improve the agents’ performance, we propose that random conductor election can benefit the agents’ zero-shot generalization ability.
Agents may face more combinations of different conductors and team members with random conductor election, which can improve their ability and robustness to deal with unseen scenarios.
It can be regarded as a structural exploration on agents’ relationship, which may find better coordination style than heuristic ways.
Similar ideas are proposed in REFIL [12].
They randomly masked out some teammates to deal with dynamic team composition.
H Quantitative Analysis on the Communication Cost
Assume the number of agents is N , and the average number of each agent’s neighbors is M .
The number of messages delivered at each time-step for fully-connected communication like Agarwal et al.
[1] is O ( MN ) , since each agent is required to send messages to all its neighbors.
By contrast, in our proposed method, the overall cost is just O ( N ) .
The proof is as follows: We divide the messages into two types: sent by conductors and sent by followers.
For followers, each agent only needs to send one message to its conductor, therefore the communication cost is O ( N ) .
For conductors, each follower is only subject to one conductor, so each agent can only receive at most one message sent by conductors, costing O ( N ) .
Therefore the overall messages delivered at one time-step is O ( N ) in our proposed method.
I Comparison to Attention-based Methods
Figure 7 shows a comparison of SOG-rl, NCC [23], Gated_Qatten [24] and MAGIC [26].
All methods are re-implemented to adapt to our environment with dynamic team composition.
By keeping the core part unchanged, we apply all these methods to the value-based framework to make a fair comparison.
J Learning Curves for Training Performance
To test whether SOG’s advantage is already visible during training or specific to its generalisation ability, we plot the training return curves in Resource Collection (agent number sampled in [2,3,4,5]) in Fig 8.
To make the curve more clear, we do not show algorithms with relatively low returns, including QMIX_atten, EMP and REFIL.
The results show that SOG has no obvious superior performance than NCC or MAGIC in training, which indicates that the advantage of SOG is specific to its generalisation ability.
Next we give a detailed discussion about the source of generalization ability.
The generalization ability mainly comes from the time-varying group communication mechanism.
Many existing proximity-based methods regard the communication between agents as a graph.
When transferred to unseen scenarios, the graph’s degree and the number of edges may change a lot, i.e., the communication pattern changes a lot, thus causing the performance drop.
In contrast, through our
SOG mechanism, the agent can maintain a similar communication pattern to the training condition.
For example, when training a model with 2-agent team to fulfill a task, it may perform worse in a 4-agent scenario, since the agent is only trained to cooperate with another one agent, and it may be confused by the message sent by the other 2 agents.
However, our proposed SOG mechanism has a high probability to divide the 4-agent into 2 groups, and then preventing the message delivery between the two groups.
By doing so, in unseen scenarios, the agent may find the communication pattern is similar to that in training, and perform a similar 2-agent coordination pattern.
Therefore, an organized group under the unified command of a conductor can better adapt to an unseen scenario than individuals.