Self-Organized Group for Cooperative Multi-agent Reinforcement Learning
Anonymous Author(s)
Abstract
Centralized training with decentralized execution (CTDE) has achieved greatsuccess in cooperative multi-agent reinforcement learning (MARL) in practicalapplications.
However, CTDE-based methods typically suffer from poor zero-shotgeneralization ability with dynamic team composition and varying partial observability.
To tackle these issues, we propose a spontaneously grouping mechanism,termed Self-Organized Group (SOG), which is featured with a conductor election(CE) and a message summary (MS) mechanism.
In CE, a certain number of conductors are elected every T time-steps to temporally construct groups, each withconductor-follower consensus where the followers are constrained to only communicate with their conductor.
In MS, each conductor summarize and distribute thereceived messages to all affiliate group members to hold a unified scheduling.
SOGprovides zero-shot generalization ability to the dynamic number of agents and thevarying partial observability.
Sufficient experiments on mainstream multi-agentbenchmarks exhibit superiority of SOG.
Introduction
Cooperative multi-agent deep reinforcement learning algorithms (MARL) have been ubiquitouslyapplied to real-world scenarios, such as autonomous vehicle teams [], sensor networks [47] andsocial science [13].
Recently MARL has gained extraordinary performance in various multiplayergames like Dota [3], StartCraft [30] and soccer [16].
The framework of centralized training withdecentralized execution (CTDE) [8, 28] is one of the popular frameworks for solving cooperativemulti-agent tasks.
Centralized training renders the CTDE paradigm better agent cooperation whileindependent execution endows the multi-agent system with more efficiency and scalability.
Classical CTDE algorithms like QMIX [28] and MADDPG [21] are confined to the fixed size ofagents.
However, the number of involved agents tends to vary in real-world multi-agent scenarios.
Recently, some methods introduce the attention mechanism [40] to train on the varying number ofagents simultaneously [12, 11].
They only seek for a solution covering a range of varying team size,unable to provide the generalization ability to the cases not in the range.
Agarwal et al.
[1] and Liuet al.
[18] further introduce the communication mechanism to provide the adaptability on the dynamicteam composition (i.e., the team size varies).
In the paper, we propose a method called “Self-Organized Group (SOG)”to possess zero-shotgeneralization for multi-agent reinforcement learning, which provides strong adaptability to thescenarios with varying number of agents and even varying sight of agents.
In spirit of the individuals’cooperation in social science [10] and the leader-following formation in multi-agent system [9], partof the agents are elected as conductors at intervals, and for each conductor a temporary group isformed with conductor-follower consensus.
In each group, the followers can only communicate withthe conductor, while the conductor summarizes all the received messages and distributes the refined message to all group members aiming for a unified target.
Our insight is that, an organized groupunder the unified command of a conductor can better adapt to an unseen scenario than individuals.
Tobe specific, each agent’s behavior is prone to diverging from the training pattern when presented withan unseen scenario.
They may behave in contradictory ways and pose a threat to the system’s stability.
On the contrary, a conductor could send unified commands, making each agent behave in order, thusavoiding the overall collapse of the training pattern.
In the meantime, our method largely reduces thebandwidth and the cost required for communication.
For efficient and economical message passing,we also design a variational message summarizer (MS), which compresses the local observationinformation and helps predict the future trajectories.
Since the conductor navigates the information propagation in the whole group, Conductor Election (CE) plays a critical role in SOG.
Except for the simple random CE which satisfies the fullydecentralized execution paradigm, we further propose a heuristic variant: determinantal point process (DPP) [22]-based CE, and a learning-based variant: policy gradient (PG) based CE.
DPP-based
CE chooses conductors in the principle of maximum entropy and avoids the occurrence of homogeneous conductors.
PG-based CE formulates the CE as a learning process and takes the long-termbenefits of the selection into consideration.
We conduct experiments on three commonly used multi-agent benchmarks, including a resourcecollection task, a predator-prey task, and a set of customized StarCraft micromanagement tasks.
To validate the zero-shot generalization ability of our method, we evaluate trained models in morecomplicated scenarios than training.
We not only increase the number of agents for evaluation, butalso adjust the agents’ utility, e.g., reducing the sight range of each agent.
The results show that SOGhas better zero-shot generalization ability, not only for the dynamic team composition, but also forthe unseen environment condition, than current state-of-the-art methods on all three benchmarks.
The contribution of our paper is summarized as follows: (1) We propose Self-Organized Group, amechanism for enhancing agents’ zero-shot generalization ability.
(2) We design two conductorelection ways for reasonable group forming and a variational message summarizer for efficient andeconomical message passing.
(3) We achieve better generalization ability on three commonly usedmulti-agent benchmarks than current state-of-the-art methods.
Related Work
Centralized Training with Decentralized Execution .
We mainly focus on the typical paradigm ofcentralized training with decentralized execution (CTDE) for cooperative multi-agent tasks in this part.
It combines the advantages of independent Q-learning [38] and joint action learning [4].
A series ofwork concentrate on factorizing Q functions [35, 28] and deriving theoretical guarantee for the policyoptimality [33, 41, 29].
Some other works use actor-critic methods with a centralized critic [21, 7, 49].
Although the CTDE paradigm shows great empirical performance in many multi-agent tasks, it mayfail on some simple situations without communication on execution due to the partial observabilityof the local agent [18].
In addition, traditional CTDE methods usually concentrate on invariableenvironment conditions and fixed team size.
Communication for MARL .
Many efforts have been made for communicating agents in multi-agentsystems [32].
Recently some works assume the information can propagate among homogeneousagents during decentralized execution [6, 36, 48, 34, 19], some of which use the graph neural networkto handle the message passing [23, 24, 26, 24].
Some works focus on communication betweenheterogenous agents [25, 31], but the role of each agent is usually fixed, therefore limiting the model’sgeneralization ability.
These works usually pre-define the number of agents, and can hardly handlethe dynamic team composition.
Liu et al.
[18] uses a predefined coach with global information toinstruct dynamic number of agents.
However, a global coach that can communicate with all agentsis not often available in real-world domains.
In contrast to these methods, our method enables allagents to become a conductor, while reserves the efficiency and scalability of CTDE.
Transfer Learning and Curriculum Learning .
An important topic for multi-agent systems is theadaptation to the mutable environment.
A bunch of works attempt to deal with the increasinglychallenging environment by designing a set of curricula [2, 20, 44].
Another line of works utilize theattention mechanism or graph neural networks to handle the diversity of the local observation [14,1, 11, 12].
Wang et al.
[43] allocates roles with limited action space to agents, and the study showsthe model’s ability of task transfer by increasing the action space for role clusters.
Iqbal et al.
[12] trains simultaneously on multiple tasks by randomly masking out some entities the agent observes.
Unlike these methods, our method bypasses the overlarge training burden of curriculum learning, andis more robust in the dynamic team composition and varying environment conditions.
Background
In our work, we consider a fully collaborative multi-agent task with n agents, which can bemodeled as a decentralised partially observable Markov decision process (Dec-POMDP) [27]
G = ⟨ S, A, I, P, r, Z, O, n, γ ⟩ , where s ∈ S is the true state of the environment.
At each time stept , each agent i ∈
I ≡ { 1 , ..., n } chooses an action a i ∈ A , which is an element of a joint actiona ∈
A ≡ A n .
P ( s t +1 | s t , a t ) : S ×
A × S → [0 , 1] is the state transition function of the environment.
All agents share the same reward function r ( s, a ) : S ×
A → R .
γ ∈ [0 , 1) is the discount factor.
Dueto the partial observability , each agent i has its local observations o
i ∈
O drawn from the observationfunction Z ( s, i ) :
S × I → O .
In our setting, we assume agents can communicate within its sightrange.
Each agent i may receive a message µ i ∈ U from its neighbors.
Each agent chooses an actionby its stochastic policy π i ( a i |
ρ i , µ i ) :
Γ × U → ∆([0 , 1] | A | ) , where ρ i ∈ Γ ≡ ( O × A ) ∗ denotesthe action-observation history of agent i , and ρ is the action-observation histories of all agents.
Theagents’ joint policy π induces a joint action-value function : Q π ( s t , a t ) =
E s t +1: ∞ , a t +1: ∞
[ R t | s t , a t ] ,where R t = (cid:80) ∞ k =0 γ k r t + k is the discounted accumulated reward.
The goal of our method is to findthe optimal joint policy π ∗ such that Q π ∗ ( s, a ) ≥ Q π ( s, a ) , for all π and ( s, a ) ∈ S × A .
In practice,to handle the input with dynamic team size, we use the entity-wise input rather than the vector input,detailed in Appendix A.
We use the popular centralized training with decentralized execution (CTDE) paradigm [8, 28] as thebaseline, also allowing the communication between adjacent agents.
During training, the method hasthe access to the full state and each agent’s action-observation history.
During testing (execution),each agent only has the access to its own action-observation history and communication messagesfrom neighbors.
Method
In this section, we first highlight the importance of the communication during decentralized execution,and propose self-organized group for decentralized communication.
Next we introduce the learningobjective of the communication message.
Finally we present the whole training procedure togetherwith the model architecture.
4.1 Downsides of CTDE
[Figure]
Consider a simple scenario in Fig. 1(a).
Thetask is fulfilled only when the three buttonsare pressed synchronously.
Therefore, agents
A, B, and C need to press the three nonrecurring buttons respectively.
The button 1 and 2are ten meters apart, while button 3 is at themidpoint of them.
The sight range of eachagent is a circle with radius six meters.
As aresult, the agent who is in charge of the button2 can’t perceive if some agent is pressing thebutton 1, due to sight range limitation.
If weuse the common centralized training with decentralized execution (CTDE) method, like QMIX [28], to deal with this task, these agents can notefficiently coordinate beyond the individual sight range, since each agent’s execution is conditionedon its own action-observation history.
The communication between agents is one of the easy andflexible solutions to handle the partial observability.
To extend the CTDE paradigm based on the communication mechanism, we now present the SelfOrganized Group (SOG), a mechanism to group all agents
4.2 Self-Organized Group
Aiming for a cooperative multi-agent task with dynamic team composition, every T time-steps, weelect a certain number of conductors and the groups are constructed accordingly.
In this sectionwe first show the formulation of SOG.
Then we present the message summarizer in Sec.
4.2.1, andwe discuss in Sec.
4.2.2 the strategies for conductor election.
An example group is illustrated as in
Fig. 1(b), where agent C is appointed as the conductor, with the collected information from A and
B to broaden the sight range.
Since our main task lies in MARL, we simplify the communicationprocedure, assuming that all the messages can be delivered accurately without delay and each agentcan handle all the received messages at the same time.
Our goal is to build a communicationmechanism that satisfies the following properties:
• Lightweight .
The communication message should be brief and informative, e.g., a three-
dimensional summary vector of the agent’s current state.
• Robust .
The group should be able to handle the varying number of group members and the
unfamiliar environmental conditions (e.g. the varying sight range).
The process of group organization is as follows (Fig. 2): every T time-steps, some agents areelected to be conductors.
We allow multiple conductors in the same time.
Then conductors send groupinvitations to all agents within their sight ranges.
Thenon-conductor agents who receive one invitation thensend their personal messages ζ back to the conductor.
Here the personal message ζ is delicately designed bya message summarizer, described in Sec.
Fornon-conductor agents who receive multiple group invitations
, they randomly choose one as the conductorand reply.
The conductor and its interactive agentstemporarily form a group.
For those who receive nogroup invitation or leave the sight of the constructorduring in-group communication, they form a 1-agentgroup of themselves.
After the group is formed, the conductor processes the messages from all the followers (includingitself).
Then it sends the processed message ξ j back to each member that belongs to group j .
Each agent only needs to communicate with its group’s conductor, therefore it greatly reduces thebandwidth required for communication.
We use a non-parametric message mixer for each conductor:
[Equation]
where M j is the size of group j .
In this way, the conductor takes no extra computational cost formessage mixing and distributing procedure, and each agent can play the role of the conductor withoutchanging its network structure.
The communication takes place every T time-steps.
Each agent i uses its local observation
o i and themessage µ i to predict its action.
µ i is defined as:
[Equation]
where I ( t, T ) is the indicator whether t is divisible by T .
Through such mechanism, agents can share the information of the whole group with the minimalcommunication cost and coordinate more efficiently.
We present the quantitative analysis on thecommunication cost in Appendix H, which is 1 /M of the cost of the fully-connected communicationlike Agarwal et al.
[1], where M is the average number of neighbors for each agent.
In unfamiliar scenarios, the conductor integrates the followers’ information and all group membersexecute a unified command, driven by a delicate message summarizer.
Such a in-group communicationmechanism empowers the system better stability compared to the non-conductor-following formation.
With time-varying conductor election, SOG shows strong zero-shot generalization ability to thedynamic team composition and unseen environmental conditions.
4.2.1 Message Summary (MS)
After the communication protocol is built, we now present the content of the communication message.
Inspired by recent works on role learning and state summary by variational inference [42, 18, 17],we propose a message summarizer for agents to summarize their local observations as brief latentvariables.
A summarizer can extract valuable information from the whole trajectory, and reduce thecomputational complexity.
We first aim to distill the information about the agent i ’ s transition of future T -steps.
Letτ it =
( o it +1 , a it +1 , ...,
o it + T − 1 , a it + T − 1 ) .
The personal message is a random Gaussian variable of
C dimensions sampled from an encoder, i.e., ζ it ∼ f ψ ( o it ) .
We maximize the mutual informationbetween ζ it and τ it conditioned on o it .
I ( ζ it ; τ it | o it ) has the following lower bound:
[Equation]
where q ϕ ( · ) is the variational estimator, which is only used for centralized training.
We define theopposite of the lower bound as future predictor loss L FP .
The derivation is defered to Appendix D.
In addition, we expect the personal message to help discard the irrelevant information while retainthe future trajectory’s information, which can endow the agent with a small state representation andaccelerate training.
We take advantage of the conditional entropy bottleneck (CEB) objective [5].
Wemaximize the mutual information between the personal message ζ it and the future trajectory τ it , andminimize the mutual information between the current observation o it and the personal message ζ itconditioned on the future trajectory τ it simultaneously.
The objective has the following lower bound:
[Equation]
where f ψ ( · ) is the encoder for personal message, and K is the size of the mini-batch sampled fortraining.
We define the opposite of the lower bound as L CEB .
See Appendix D for detailed derivation.
4.2.2 Conductor Election (CE)
“MHA” means multi-head attention.
We use this module to deal with the entity-wise input for dynamic team composition, detailed in
Appendix A. h
it is the hidden state for GRU cell.
In this section, we introduce three conductor election mechanisms: Random CE, Determinantal Point
Process (DPP) [22] based CE, and Policy Gradient [37] based CE.
Note that CE differs from theconcept of leader election (LE) [39] in distributedsystem.
We elect conductors to obtain better agentcooperation for maximizing the team reward, and weassume a perfect communication channel.
LE is usually deployed in real multi-agent systems with theprobability of agent disconnection, with the targetto maintain the cognitive consistency among agents.
Random CE .
Each agent is elected as a conductor with an independent and identical probability p l ,which is fully decentralized.
The expectation of thenumber of groups can be controlled by p l .
Sinceall group members need to communicate with theirneighbors only for message passing, SOG with random CE doesn’t need centralized commander duringexecution.
Therefore, it can be parallelized easilyduring execution.
DPP-based CE .
In this part we expect to increase the generalization ability of SOG by maximizingthe intra-group diversity.
We achieve this by maximizing the diversity between conductors consideringthe conductor is the structural center of a group.
Specifically, we formalize the conductor election asa determinantal point process P .
Definition 1.
A determinantal point process (DPP) P is a probability distribution defined on thepower set 2 Y of a discrete finite basic point set Y = { 1 , 2 , ..., n } .
A symmetric positive semi-definite matrix L ∈ R n × n measures the similarity between any point pairs in Y .
Let B be a random subsetdrawn by P , then ∀ C ⊆
Y , we have
[Equation]
where L C :=
[ L i,j ] i,j ∈ C is the submatrix of L , whose entries are indexed by the elements in C , and
I is an n × n identity matrix.
det( · ) means the determinant of a matrix.
In practice, we use all agents to construct the point set Y .
Correlations are always non-positive in
DPP [15].
The more similar two points are, the less possible they will appear in a subset sampledby DPP.
Therefore, once we need to elect conductors, we just sample a subset B by DPP and useall elements in B as the conductors, which greatly reduces the probability to elect conductors withsimilar observations (e.g., two agents close to each other).
For the similarity matrix L , we use thecosine similarity of agents’ personal feature V i :
[Equation]
Another concern for DPP is the high computational cost for calculating the determinant of allsubmatrices.
Instead of the traditional sampler by Schmidt orthogonalization [46], we find a methodto calculate all determinants in parallel with a GPU, which greatly accelerates the computation.
Details can be found in Appendix B.
PG-based CE .
The conductor election might affect the group’s accumulative rewards in the followingtime-steps.
Therefore, we regard the conductor election task as a reinforcement learning problem.
Weuse a policy π ( a | s ) =
(cid:81) ni
=1 π i ( a | s ) with the input of global state to decide the probability of eachagent to become a conductor, and do policy gradient on it.
The gradient ∇L CERL can be written as:
[Equation]
where ˆ t is the time that CE happens, i.e., every T time-steps.
We omit the π ’s dependence on theparameter ω of the neural network for brevity.
Electing conductors by PG takes the long-term benefitsof the selection into consideration, and in experiments it shows great generalization ability.
4.3 Algorithm Outline
As shown in Fig.
3, our model is constructed using centralized training with decentralized executionparadigm.
We utilize the entity-based input like in Iqbal et al.
[12], so that the model can deal withthe dynamic number of agents.
The partial observability of local agents is assured by masking out theunseen entities.
All agents share the same parameters.
It’s worth noting that the variational estimatorq ϕ is designed to include information of future T -steps.
Since at time-step t it takes the output of
GRU cell h it as input, q ϕ should be used to calculate the loss with message ζ before T steps, i.e. ζ it − T .
The local Q function uses the message ξ j sent by the conductor as part of input if it is in group jwhen communication takes place, and uses personal message ζ i in place at other time.
Let µ t denotethe set of the messages used for local Q prediction at time-step t .
The mean square Bellman errorobjective for Q-learning is as follows:
[Equation]
where ˆ θ is the parameters of the target network, and D is the replay buffer.
where λ 1 and λ 2 arehyper-parameters.
We summarize the training procedure in Algorithm 1.
Experiments
In this section we design experiments to answer the following questions: (1) Whether the SelfOrganized Group helps the agents coordinate better than fully-connected communication methods?(2)
Can Self-Organized Group promote zero-shot generalization to the dynamic team composition andthe varying partial observability, even on complicated tasks?
(3) What’s the effect of the conductor
Algorithm
Initialize θ, ϕ, ψ .
Set learning rate ← η , communication interval ← T , D ← {} , λ 1 , λ 2 , t max for each episode iteration do for t = 1 , 2 , . . .
, t max do if t mod T ̸ = 0 then
For each agent i , µ it ← Sample ( f ψ ( o it )) else
Elect conductors and form groups { G j } Kj =
For each agent i , ζ it ← Sample ( f ψ ( o it ))
For each conductor j , ξ jt ← Mix ( ζ it ) , i ∈ G ( j )
For each agent i , µ it ← ξ jt , i ∈ G ( j ) end if
Generate tuple { s t , a t , r t , s t +1 , ζ t , ζ t +1 } by executing f θ ( τ t , µ t ) end for
D ← D ∪ { s t , a t , r t , s t +1 , ζ t , ζ t +1 } t max t =1 , θ ← θ + η ˆ ∇ θ L RL ( D ) ψ ← ψ + η ( ˆ ∇ ψ L FP ( D ) + ˆ ∇ ψ L CEB ( D )) , ϕ ← ϕ
+ η ( ˆ ∇ ϕ L FP ( D ) + ˆ ∇ ϕ L CEB ( D )) end for election way?
(4) What factor contributes to Self-Organized Group most?
We test our idea onthree commonly used multi-agent benchmarks: Resource Collection, Predator-Prey, and StarCraft IImicromanagement tasks.
The previous two are built on the multi-agent particle environment [21],and the last is modified from Iqbal et al.
The number of the agents is not fixed, sampling from apre-defined set at the beginning of each episode in all scenarios.
For all the three tasks the testingenvironments are more complicated than the training ones.
Agents need to handle the situations thatis not encountered during training.
Each experiment is repeated 3 or 5 times with different seeds.
In experiments, the name “SOG” means our method with random CE, whose execution is decentralized.
“SOG_dpp” and “SOG_rl” means the SOG with DPP-based CE and PG-based CE, respectively.
Except for our method, we evaluate four state-of-the-art CTDE methods that are suitable for dynamicteam composition: A-QMIX [45], REFIL [12], EMP [1] and COPA [18].
For a fair comparison,we extend the original EMP’s decentralized-training structure to CTDE.
For A-QMIX and REFIL,agents have no communication during execution, while for EMP and COPA, agents communicatewith their neighbors or a global coach.
Since many MARL methods with communication are notsuitable for dynamic team composition (e.g., CommsNet [34] and G2ANet [19]), we extract onescenario of Resource Collection and show results in Appendix G.1.
We also compare SOG withgraph-based methods including NCC [23], MAGIC [26] and Gated-ACML [24] in Appendix J.
5.1 Resource Collection
This scenario is modified from the environment described in Liu et al.
Agents need to collectresources from 6 resource points and transport the goods home.
The number of the agents for trainingis uniformly sampled from {2,3,4,5}, while for testing it is sampled from {6,7,8}.
Each agent hasa sight range SR .
Entities including other agents and resource points that exceed agent i ’s SR areinvisible to agent i .
For more details of the environment, please refer to Appendix F.1.
We train on two kinds of SR : 0.5 and 1.0.
We show in Fig.
4(a-b) the results of all methods.
SOGsignificantly outperforms the other methods when SR = 0 .
5 and SR = 1 . 0 .
Given that the testingscenario includes 6/7/8 agents while the training scenario only has 2/3/4/5, the communicationmechanism of SOG greatly strengthens SOG’s zero-shot generalization ability for dynamic teamcomposition.
The CE methods have no obvious difference when the team size varies.
Then we test the zero-shot generalization ability in different environmental conditions.
We save thepreviously trained models with a fixed sight range 0 .
5 or 1 .
and evaluate them on different agentsight range settings.
Each setting is repeated for 160 testing episodes and the results are averaged over3 models with different seeds.
We show the performance of each method in Fig.
4(c-d).
Except forthe difficult setting SR = 0 .
2 that all methods have poor performance, SOG and SOG_rl can achievesimilar or even better performance in different scenarios ranging from SR = 0 .
5 to SR = 1 . {+
 thanin+} training scenarios.
SOG_rl even performs better transferring from SR = 1 . 0 to SR = 0 .
In the meantime, other methods show notable performance drop when transferred from training settings
SOG’s
5.2 StarCraft Micromanagement Tasks
[Figure]
We apply our method to the StarCraft multi-agent challenge (SMAC) [30].
We use the map designedby Iqbal et al.
[12] and Liu et al.
The maps randomly initialize 3-8 agents with enemies of thesame number at the start of each episode.
Agents are initialized together with the sight range 9 [12]or divided into 2-4 groups with the sight range 3 [18].
We call the previous one “G(ather)” and thelatter “D(isperse)”.
In the scenario “Disperse”, the enemies are divided into 1-2 groups, which meanseach group of enemies is stronger than that of the agents.
We limit the number of agents to 3-5 intraining procedure and test the model with agent number 6-8.
Similar to Iqbal et al.
[12], Liu et al.[18], we also incorporate the imaginary objective.
We first explore the adaptation to the dynamic team composition.
We show in Fig.
5(a)
the resultson 4 maps.
Our method has the highest test winning rate against the preset AI when testing themodel with a larger size of agents and enemies than training.
And the PG-based CE improves theperformance on 3-8sz.
Combining the results in Resource Collection, we conclude that PG-based CEcan improve agents’ zero-shot generalization ability on dynamic team composition in many cases,due to the more reasonable conductor election mechanism than random CE.
Then, we study the adaptation to the varying environment conditions of our method.
We save themodels trained on map 3-8sz_D with the sight range 3, which is a hard case that agents are split andhave restricted horizons.
Then we evaluate the models on other initial conditions and sight ranges.
The results are shown on Fig. 5(b).
The initial conditions (Gather or Disperse) have no obvious effecton the performance for all four methods.
However, when agents can see more entities than training,other three methods all show large performance drop while our method has comparable performanceto that in training.
This is probably due to the unified messaging passing mechanism within thegroup.
An interesting phenomenon is that the random CE has comparable or better performance torl-based CE on 2 scenarios.
We speculate this is due to the similar pattern of random CE in differentenvironment settings, and analyse this result in detail in Appendix G.2.
The results show that ourmethod can better adapt to the varying partial observability.
To further validate the effect of our method, we visualize the situation for four methods (SOG, SOG-rl,
SOG without MS, and EMP3) on the same SMAC map in Fig. 5(c).
We can see that agents trained by
SOG learn to gather and concentrate fire on certain enemies surrounded by agents.
SOG-rl sacrificesa group of 2 agents to exchange the the elimination of 3 enemies.
SOG without message summarylearns to gather, but the target of each agent’s attack is not focused.
EMP3 does not even learn toattack simultaneously.
The results show that CE and MS both promote better agent cooperation.
5.3 Predator Prey
In this scenario, agents play the role of predators, aiming to catch some preys with random walk.
The training number of predators is sampled from {3,4}, along with only 1 prey.
During executionwe initialize 5 or 6 predators, as well as 1 or 2 preys.
For environment details, please refer to
Appendix F.2.
It is a relatively easy-to-learn scenario, and we make ablations on it.
[Table]
First, we explore the effect of the message summarizer.
Except for the L FP and L CEB , we also try an entropyregularizer similar to the implementation of Wang et al.[42].
It aims to maximize the entropy of the messageto encourage exploration.
As shown in Table 1, thecombination of L FP and L CEB obtains the best average testing return, exceeding the performance of CTDEmethod A-QMIX, and communication method EMP.
Although the entropy regularizer helps stabilize thetraining (the variance is reduced when combined withthe regularizer), it makes no contribution to the average performance.
Therefore we use the L FP + L CEBas our default loss for message summarizer.
Noticethat EMP has similar performance to SOG on trainingscenarios.
Nevertheless, SOG performs better when transferred to complicated evaluation scenarios.
[Table]
Then we analyze the choice related to communication in
Table 2, including the number of groups in expectation, themessage dimension, and the communication interval.
Thenumber of group 2 has better performance than number1 or 4.
Since the rule that a prey needs to be caught by3 predators simultaneously, it is suitable to divide 5 or6 agents into 2 groups.
When the message dimension isreduced from 10 to 3, the results have no obvious drop.
However, it decrease greatly when the message dimensionis set to 1.
The results show that a 3-dimensional messageis sufficient for communication for Predator-Prey task.
As for communication interval, T = 2 is alittle better than T = 4 , and T = 10 is not enough for sufficient communication.
Conclusion
In this paper we propose Self-Organized Group (SOG) for cooperative multi-agent reinforcementlearning.
In SOG, a certain number of agents are randomly elected to be conductors and thecorresponding groups are constructed with conductor-follower consensus, allowing the groups tobe re-organized every T time-steps.
We find the organized group under the unified command of aconductor embeds the multi-agent system with stronger zero-shot generalization ability comparedto the traditional CTDE methods with fully-connected communication mechanism.
Furthermore,we derive a variational message summarizer for efficient and economical message passing, and wepropose DPP-based and PG-based conductor election strategies for better group organization.
Wetake experiments on three commonly used multi-agent benchmarks.
SOG shows better zero-shotgeneralization ability not only for the dynamic team composition, but also for the varying partialobservability on all three benchmarks.
Limitations Due to the space limitation, we defer the discussion of SOG’s limitations in Appendix L.