G ROUP - ORIENTED C OOPERATION IN M ULTI -A GENT R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Grouping is ubiquitous in natural systems and is essential for promoting efﬁciency in team coordination.
This paper introduces the concept of grouping into multi- agent reinforcement learning (MARL) and provides a novel formulation of Group- oriented MARL (GoMARL).
In contrast to existing approaches that attempt to directly learn the complex relationship between the joint action-values and indi- vidual values, we empower groups as a bridge to model the connection between a small set of agents and encourage cooperation among them, thereby improving the efﬁciency of the whole team.
In particular, we factorize the joint action-values as a combination of group-wise values, which guide agents to improve their policies in a ﬁne-grained fashion.
We propose a ﬂexible grouping mechanism inspired by variable selection and sparse regularization to generate dynamic groups and group action-values.
We further propose a hierarchical control for policy learning that drives the agents in the same group to specialize in similar policies and possess diverse strategies for various groups.
Extensive experiments on a challenging set of StarCraft II micromanagement tasks and Google Research Football scenarios verify our method’s effectiveness and learning efﬁciency.
Detailed component studies show how grouping works and enhances performance
I NTRODUCTION
Cooperative multi-agent reinforcement learning (MARL) aims to coordinate multiple agents’ actions through shared team rewards and has become a helpful tool for solving multi-agent decisionmaking problems, such as network routing (Ye et al., 2015), robot swarm control (H ¨ uttenrauch et al.
,2017), crewless aerial vehicles (Xu et al., 2018), etc.
Learning centralized policies is a natural wayto address the cooperative MARL problem.
It treats the team as a single actor with a joint actionspace.
Although single-agent RL algorithms can be trivially transplanted to this setting, the globalinformation to train the model is usually unavailable during execution due to partial observabilityor communication constraints, and the joint action space grows exponentially with the agent number (Gupta et al., 2017; OroojlooyJadid & Hajinezhad, 2019; Gronauer & Diepold, 2021).
An alternative paradigm is to learn decentralized policies (Tan, 1993;
Witt et al., 2020; Yu et al., 2021) byindependently training agents based on their local observations.
However, simultaneous explorationbrings non-stationarity that causes unstable learning and convergence difﬁculties (Hernandez-Lealet al., 2019; Zhang et al., 2021).
The centralized training with decentralized execution (CTDE)paradigm inherits the advantages of the above two paradigms and allows learning decentralizedpolicies in a centralized fashion (Oliehoek et al., 2008; Kraemer & Banerjee, 2016).
Although the CTDE paradigm solves many multi-agent problems and opens up the possibility foragents to share information during training, it still faces two challenges.
On the one hand, learningefﬁcient team cooperation by directly estimating the joint action-values from individual utilities isexceptionally difﬁcult, especially when the agent number is enormous.
Most value function factorization approaches have been only evaluated in domains with a handful of agents.
The ﬂat factorization scheme is proven to lead to performance bottleneck (Phan et al., 2021), where it gets difﬁcult toprovide sufﬁciently informative training signals for each agent.
Although the state information provides the complete knowledge needed for learning, it is burdensome for agents to extract effectiveguidance that facilitates cooperation.
On the other hand, the guideline of sharing parameters is stillan open question.
The full parameter-sharing mechanism limits the diversity of agents’ behaviorstrategies (Li et al., 2021), leading agents’ policies to be similar or the same.
Both nature (Jeanson et al., 2005; Wittemyer & Getz, 2007) and MARL (Phan et al., 2021) havevalidated grouping as a means to promote efﬁcient cooperation and break performance bottlenecks.
MARL method VAST (Phan et al., 2021) approximates a factorization for agent sub-teams to overcome the performance bottleneck caused by the ﬂat value factorization which directly assigns a centralized value function to each agent.
Although grouping provides new possibilities for MARL, howto formulate a general grouping criterion for complex and diverse environments without any domainknowledge is still an open question and a matter of great interest.
Most previous grouping works areproposed for well-structured tasks, e.g. , software engineering (Pav´on & G´omez-Sanz, 2003; Bresciani et al., 2004), and typically predeﬁne speciﬁc responsibilities or forms of task decomposition.
VAST, on the other hand, artiﬁcially sets the team amount to half or a quarter of the number ofagents.
These methods all require apriori knowledge or settings that are potentially unavailable orunreasonable in practice and may discourage methods’ transferring to diverse environments.
To address the above challenges faced by the CTDE paradigm and group learning, we propose anovel formulation of Group-oriented MARL (GoMARL), a method for learning dynamic groupingwithout any domain knowledge or a priori setting.
Instead of formulating a concrete grouping criterion, GoMARL implicitly learns dynamic groups with a novel sparsity-driven scheme.
Concretely,it learns a dual hierarchy of value function factorization, where the learning weights of the decomposition from the group value to local utilities determine whether an agent is suitable for the currentgroup.
A sparsity regularization drives this end-to-end automatic group learning with the Least Absolute Shrinkage and Selection Operator (LASSO) (Tibshirani, 1996).
Furthermore, GoMARLrelies on well-designed architecture to transform the individual, group-related, and global information into the weights of corresponding networks in a manner reminiscent of hypernetworks (Ha et al.,2017), which ﬂexibly adapts to the changes in group number and group size.
To alleviate the ﬁrst problem faced by the CTDE paradigm, GoMARL adopts the dual-hierarchyvalue factorization to learn on a more focused and compact input representation.
Different from
VAST and other methods that learn value factorization with no information (learn blindly) or onlywith the state information (complete but hard to extract efﬁcient guidance), we propose a ﬁnegrained learning scheme to integrate information from the group perspective into the policy gradient by hypernetworks.
It promotes intra-group coordination with group information and facilitatesinter-group cooperation with the global state to further break the performance bottleneck.
As forthe second problem, parameter sharing prevents efﬁcient cooperation due to policy similarity.
However, our dynamic grouping naturally serves as an intermediary to encourage diversity while sharingparameters.
We introduce a latent space to describe group information for hierarchical control toestablish the connection between group and policy.
Agents condition their behaviors on their groupinformation embedded by a shared encoder, which is learned following specialization guidance, i.e. ,imposing similarity within a group and diversity between groups.
In this way, GoMARL synergizesgroups with specialized policies, providing a fully-sharing mechanism for learning diverse policies.
We summarize our main contributions in this paper as follows:
We introduce an end-to-end adaptive group learning MARL solution with no domain knowledgeor a priori setting, partitioning agents into dynamic groups to promote efﬁcient cooperation.
Structurally, we present an architecture to estimate global and group-wise joint action-values with
varying group numbers and agent numbers per group, realizing a dynamic end-to-end training.3.
Functionally, we propose agents that condition their behavior on latent group information toachieve group specialization and diversity while sharing all their parameters.
We test our method on a challenging set of StarCraft II micromanagement tasks (Samvelyan et al.
and Google Research Football (Kurach et al., 2020) scenarios.
GoMARL achieves superiorperformance with higher efﬁciency compared with notable baseline methods.
We also provide detailed ablation studies to give insights into how grouping works and enhances learning performance.
B ACKGROUND AND P RELIMINARIES
Dec-POMDP .
This paper focuses on cooperative tasks with n agents A
= { a 1 , ..., a n } as a DecPOMDP (Oliehoek & Amato, 2016) deﬁned by a tuple G = (cid:104) S, U, P, r, Z, O, n, γ (cid:105) .
The environment has a true state s ∈ S .
Each agent a chooses an action u at from its action space U a at timestept and forms a joint action u t ∈ ( U 1 × ...
× U n ) ≡ U n that induces a transition in the environment according to the state transition function P ( s t +1 | s t , u t ) :
S × U n × S → [0 , 1] .
r ( s, u ) : S × U n → R is the reward function yielding a global reward, and γ ∈ [0 , 1) is the discount factor.
We considerpartially observable scenarios in which agent a acquires its local observation z a ∈ Z drawn from
O ( s t , a ) : S ×
A → Z .
Each agent has an action-observation history τ a ∈ T ≡ ( U × Z ) ∗ , onwhich it conditions a policy π a ( u a | τ a ) : T × U → [0 , 1] .
We denote joint quantities over agents inbold and joint quantities over agents other than a given agent a with the superscript − a .
Value Function Factorization .
We consider cooperative MARL with centralized training and decentralized execution paradigm, which has been a major focus in recent efforts (Foerster et al., 2018;
Sunehag et al., 2018; Rashid et al., 2018; Iqbal & Sha, 2019; Mahajan et al., 2019).
Some methodsachieve CTDE through value function factorization, i.e. , factoring action-value functions into combinations of per-agent utilities.
The individual utility only depends on the local history of actionsand observations, allowing agents to maximize their local utility functions independently.
Amongthese attempts, the representative deep MARL approach QMIX (Rashid et al., 2018) improves thesimple summation of individual utilities (Sunehag et al., 2018) by introducing a more expressive factorization: Q tot = f ( Q 1 ( τ 1 , u 1 ; θ Q ) , · · · , Q n ( τ n , u n ; θ Q ); θ f ) , where θ f denotes the parameters ofthe monotonic mixing function generated by a hypernetwork (Ha et al., 2017).
Variable Selection in Regression .
Finding critical explanatory variables (factors) in predicting theresponse variable is vital when solving regression problems.
Each explanatory factor may be represented by a group of derived input variables.
Consider the general regression problem with Mfactors: Y = (cid:80)
Mm =
X m β m + (cid:15) , where Y is an n × 1 response vector, X m is an n × p m matrixcorresponding to the m -th factor, β m is a coefﬁcient vector of size p m , and (cid:15) is a perturbation thatfollows a Gaussian distribution.
The most considered model selection problem is a particular casewhen p 1 = · · · = p m = 1 .
Model selection methods have been widely introduced (George &
McCulloch, 1993; Shen & Ye, 2002; Efron et al., 2004).
Among them, the Least Absolute Shrinkage and Selection Operator (LASSO) (Tibshirani, 1996) is the renowned and classic one, which isdeﬁned as ˆ β LASSO ( λ ) =
arg min β (cid:0) (cid:107) Y − Xβ (cid:107)
+ λ (cid:107) β (cid:107) l 1 (cid:1) .
(cid:107) · (cid:107) l 1 stands for the vector l 1 -normpenalty inducing sparsity in the solution , and λ is a tuning parameter.
G ROUP - ORIENTED L EARNING F RAMEWORK
[Figure]
This section introduces the Group-oriented Multi-agent Reinforcement Learning (GoMARL) framework that integrates thegroup notion with MARL in a principled manner.
Followingthe value function factorization approaches that represent theglobal action-value as an aggregation of individual utilities,
GoMARL decomposes the joint action-value into group-wisevalues and trains agents by groups in a ﬁne-grained scheme.
Figure 1 illustrates the overview of the group-oriented learning framework.
It consists of an automatic grouping module,specialized agent networks generating local utilities Q
i (cid:0) τ i , · (cid:1) ,and a mixing network among groups.
In Section 3.1, we introduce the automatic grouping module.
It progressively dividesthe team into dynamic groups as training proceeds.
Based onthe grouping, we propose specialized agent networks that achieve similarity within each group anddiversity among groups to generate the local Q i in Section 3.2.
Section 3.3 further presents themixing of Q i to estimate the group-wise and global action-values and the overall training procedure.
3. A UTOMATIC G ROUPING
Deﬁnition 1 ( Individual and Group ) .
Given a cooperative task with n agents A
= { a 1 , ..., a n } ,we have a set of groups
G = { g 1 , ..., g m } , 1 ≤ m ≤ n .
Each group g i contains n i (1 ≤ n i ≤ n )different agents g i
= { a i 1 , ..., a in i } ⊆
A , where g i ∩ g j = ∅ , i (cid:54) =
j , ∪ i g i
=
A , and i, j ∈ [1 , m ] .
Inthis paper, we denote all variables with a superscript to describe the variable owner, e.g. , e i and s g jare variables of agent a i and group g j , respectively.
The automatic grouping module aims to learn a mapping relationship f
g : A (cid:55)→ G .
Predeﬁnedexplicit formulations and apriori knowledge are not necessities, as they are unavailable in complexenvironments.
Our key idea is to divide the system into dynamic groups in an end-to-end fashion by maximizing the expected global return
[Equation]
tot
[Equation]
151 Value function factorization approaches represent the joint as an the
Rashid contribute
The
, and grouping G adjusted as training proceeds.
It is worth noting that the proposed dynamic grouping guideline does not require a predeﬁned group number like VAST or any other domain knowledge.
GoMARL ﬂexibly utilizes hypernetworks to accommodate the changeable group size (agent numberper group).
We construct individual w 1 generators
f i w (cid:0) · ; θ i w 1 (cid:1) :
τ i → w i 1 that map each agent a i ’shidden state h i with history information τ i to a k -dimensional weight vector.
All agents’ w 1 are sentto a group adjustment operator
O g and decide on a new grouping based on the proposed groupingguideline.
According to the learned grouping G , O g concatenates the w i 1 of agents in the same groupto form a set of group-wise w g 1 , i.e. , O g : { w 11 , · · · , w n 1 } G −→ { w g 1 1 , · · · , w g m 1 } .
This ﬁxed networkarchitecture can adapt to the grouping dynamics since each w i 1 is tied to agent a i with hypernetworkf i w .
No matter which group g j agent a i belongs to, Q i engages in the mixing of Q g j group through w i 1 .
We achieve automatic group learning by proposing a sparsity-driven scheme.
Speciﬁcally, GoMARLselects agents whose utilities contribute a little to their current group-values and adjusts their grouping; thus, a regularization for sparsity on the w 1 generators f iw (cid:0) · ; θ i w 1 (cid:1) is necessary.
Applicableregularizers are various, and we provide a straightforward but effective attempt.
We choose LASSOfrom among many feasible sparse regularizers due to its simplicity and not introducing additionallearnable parameters.
The w 1 generators are then trained by minimizing the following loss function:
[Equation]
where B is a replay buffer, and (cid:107) · (cid:107) l 1 stands for LASSO’s l 1 -norm penalty.
In practice, the weights’shrinkage to zero is infeasible.
It is also challenging to determine a ﬁxed threshold for groupsof various sizes in diverse environments.
We empirically utilize seventy percent of each group’saverage contribution (weights) to assess whether an agent is suitable for its current group.
Theexperiments in Section 4 show that this setting is generic to different scenarios and testbeds.
Thegrouping shifts every c timesteps, and each selected agent is assigned to the next group until itproperly contributes to where it belongs.
Appendix A shows insight into this grouping shift.
3. S PECIALIZED AGENT NETWORKS SHARING ALL THE PARAMETERS
Policy decentralization with shared parameters is widely utilized to improve scalability and learningefﬁciency.
However, agents tend to behave similarly when sharing parameters, preventing effectiveexploration and complex cooperative policies.
In addition, it is also undesirable to entirely forgoshared parameters in pursuit of diverse strategies since proper sharing accelerates learning.
To makefull use of the grouping, we introduce a hierarchical control for policy learning that drives the agentsin the same group to specialize in similar policies and possess diverse strategies across groups.
As shown in Figure 3, we construct an agentinfo encoder f e ( · ; θ e ) to embed agents’ hidden states.
The acquired agent info summarizesagent’s history from the group perspective .
Toachieve this group-related view, we train the encoder network to be an extractor by a group specialization regularizer , where the agent info ofagents from the same group is similar.
To avoidall agents’ info collapsing in a similar manner,the regularizer also encourages info diversity between agents from different groups.
Formally, wehave the following Similarity-Diversity objective to train the info encoder:
[Equation]
where I ( i, j ) =
[Equation]
[Equation]
The SD-loss trains the encoder to extract agent info e i that is recognizable to agents’ group.
Thisidentiﬁable agent info is then fed into a decoder network f d ( · ; θ d ) to generate the parameters ofthe agent network’s upper MLP.
This decoder hypernetwork is trained by the TD-loss introduced in
Section 3.3.
In this way, the value-based agents condition their behavior on their agent info withgroup-related information embedded, achieving specialized policies through hierarchical control.
The proposed agent network has two merits.
First, different from promising approaches like CDS (Liet al., 2021) that promote diversity by sharing only a fraction of the network, GoMARL enablesdiversiﬁed policies while sharing all the parameters , hybridizing the efﬁcient learning of parametersharing and diversity for complex cooperation.
Second, we utilize hypernetworks f d to integratethe informative extracted agent info e a i into the gradients to provide group-related information.
Unlike all previous methods that either learn with no information (learn blindly) or only with states (complete but hard to extract effective guidance), GoMARL achieves ﬁne-grained learning withagent info from the group perspective in the gradient for efﬁcient policy learning.
Concretely, thepartial derivative ∂Q tot ∂θ π for updating the policy parameters θ π of the GRU and the bottom MLP is:
[Equation]
where v is the representation after the GRU. Eqn.(3) shows that e i is deeply involved in agent
a i ’spolicy updating, facilitating group-related guidance on policy learning for better cooperation.
3. O VERALL L EARNING F RAMEWORK
We next introduce the estimation of the group-wise Q g group and global joint action-value function
Q tot .
Section 3.1 presents the generation of each agent’s w i 1 that determines the grouping G .
Although w g i 1 enables a weighted mixing of local utilities of agents in group g i to generate the groupaction-value Q g i group , the naive mixture of (cid:80) a j ∈ g i w j 1 Q j + b lacks the guidance of group-related information to reﬂect the action-value in a speciﬁc group-wise state .
Therefore, we build a two-layermixing structure that embeds the group state into the weights w g 2 of the second layer to generate
Q g group .
In particular, group g i ’s state s g
i is a fusion of the agent info e j presented in Section 3.2 forall
a j ∈ g i .
To cohesively summarize the group state based on the agent info of all agents in thisgroup, we apply the max pooling operation (Ranzato et al., 2007) over each dimension of the agentinfo
e g i to generate the group state s g i describing the current group status.
The pooling operationalso ensures adaptability to the changeable agent number per group.
We further build a group-wisew 2 generator to map the group state into the weights of the second mixing layer.
Equipped with this w 2 generator hypernetwork, s g is integrated into the gradients to provide group-related informationfor mixing Q g group (similar to Eqn.(3)) and facilitates efﬁcient intra-group coordination .
The overall learning framework of GoMARL isshown in Figure 4, and the bottom part illustrates the generation of s g and w 2 .
The twolayer mixing estimates the group-wise value byboth w 1 that decides the grouping and w 2 thatcarries group state information.
Just as ourﬁxed network can dynamically adapt to various group sizes, the hypernetwork w 2 generator ties each group g j to its w g j 2 and enablesour architecture to ﬂexibly adapt to the groupnumber changes as well.
We estimate the jointvalue Q tot by mixing all the Q g group in a similar fashion.
Concretely, the two layers of the total mixing network are generated by two hypernetworks, respectively taking group states s g i and the globalstate s as inputs.
Similar to Eqn.(3), the group and global state are deeply involved in the value gradient and guide the inter-group cooperation .
Each layer’s biases are produced in the same manner asthe corresponding weights.
The architecture of the total mixing network is akin to the group mixingnetwork and is omitted in Figure 4.
The formulation gives the TD-loss of the estimated Q tot :
[Equation]
(4) where ¯ Q tot is a target network with periodic updates.
The overall learning objective is:
[Equation] [Equation]
where θ =
( θ π , θ e , θ d , θ w ) , θ w denotes the parameters of hypernetworks producing all mixingweights and biases, λ g and λ SD are two scaling factors.
Although containing two mixing networks to respectively estimate Q g group and Q tot , GoMARL’s totalmixing-net size is quite close to or even smaller than the size of QMIX’s single mixing network, asveriﬁed in experiments.
This is mainly attributed to the input dimension reduction of the weightsgenerator hypernetworks.
The commonly used QMIX’s mixing network takes the global state s asthe input of all the hypernetworks.
In contrast, we take speciﬁc information of the grouping ( i.e. ,agents’ hidden states h , group state s g , and the global state s is only used in the top hypernetwork)as inputs.
This parameter reduction offsets the increase of an extra mixing network.
Comparedwith QMIX’s mixing only with the state information, our dual-mixing structure allows ﬁne-grainedmixing of Q i or Q g group to be guided by more detailed information ( i.e. , local history, group state,and global state) embedded in the gradients, facilitating intra- and inter-group coordination.
Wefurther prove that GoMARL’s dual-hierarchy factorization maintains decentralizability by satisfyingthe IGM (Individual-Global-Max) for Q tot and Q i for arbitrary grouping G in Appendix B.
E XPERIMENTS
Baselines .
We compare GoMARL with prominent baselines to verify its effectiveness and efﬁciency.
Hu et al.
(2021) fairly compared existing MARL methods without code-level optimizationsand reported that QMIX (Rashid et al., 2018) and QPLEX (Wang et al., 2021a) are the top twovalue function factorization methods.
The authors also ﬁnetuned QMIX (denoted as Ft-QMIX inour paper), which attains higher win rates than the vanilla QMIX.
Therefore, we compare GoMARLwith Ft-QMIX, and QPLEX to show its performance as a value factorization method (QMIX isalso shown ).
Besides, the baselines also include role-based methods (ROMA (Wang et al., 2020),
RODE (Wang et al., 2021b)) and the representative credit assignment method RIIT (Hu et al., 2021).
The latter combines effective modules of noticeable methods and has recently gotten much attention.
Experimental setup .
All methods are trained with 8 parallel runners for 10M steps.
We evaluatethem every 10K steps with 32 episodes and report the 1st, median, and 3rd quartile win rates across5 random seeds.
The detailed settings, including fair comparison, are introduced in Appendix C.1.
4. P ERFORMANCE ON S TAR C RAFT II MICROMANAGEMENT TASKS
All methods are evaluated on a set of challenging SMAC maps that vary in difﬁculty by Hard( 3s vs 5z , 5m vs 6m , 8m vs 9m ) and Super Hard ( corridor , MMM2 , 3s5z vs 3s6z ).
Easy scenarios are not chosen as they can be easily solved, and the performances of all methods are close.
The chosen tasks involve homogeneous and heterogeneous teams with asymmetric battles, allowinga holistic study of all methods.
Appendix C.3 introduces the traits of these scenarios.
Parameter size for value mixing .
GoMARL maintains two mixing networks to respectively estimate Q g group and Q tot .
If GoMARL has more parameters for value mixing is a natural question.
Appendix C.2 gives the parameter sizes for value mixing of all compared methods.
When the agentamount exceeds 5, our dual-hierarchy architecture has the least parameters among all the baselines.
Overall Performance .
The comparison of GoMARL against baseline algorithms on the SMACtasks is shown in Figure 5.
As the results show, each baseline method only achieves satisfactoryperformance on some of the challenging benchmarks with speciﬁc properties they specialize in,e.g. , RIIT performs well on MMM2 but converges much slower in other tasks, QPLEX’s leaning is notefﬁcient in 8m vs 9m and corridor .
RODE attains better overall performance than ROMA, butits hyperparameters are sensitive to scenarios; thus, its learning is not stable as QPLEX.
Finetuned
QMIX signiﬁcantly over-performs the vanilla QMIX and has more efﬁcient learning than otherbaselines.
Our method has a similar win rate as Ft-QMIX in 8m vs 9m and corridor .
However,its superiority in both efﬁciency and effectiveness can be clearly validated in all the other challengingtasks.
Next, we conduct detailed ablation studies and component analyses on GoMARL’s modulesto illustrate how our method improves learning efﬁciency and enhances performance.
Ablation study and component analysis .
GoMARL contains two key components: (1) an automatic grouping mechanism that progressively divides the team into proper groups as training proceeds; and (2) specialized agent networks that generate diversiﬁed policies according to the grouping while sharing all the parameters.
We conduct ablation studies on the three Super Hard tasks( corridor and MMM2 , and 3s5z vs 3s6z ) to show how each module inﬂuences performance.(1)
Ablation of the proposed grouping mechanism .
We validate our grouping mechanism by comparing GoMARL’s dynamic grouping with other intuitive groupings in the top row of Figure 6.
Allcompared methods utilized the same architecture as GoMARL to reﬂect how grouping itself inﬂuences performance.
Setting all agents as a group in corridor converges faster but has signiﬁcantvariances, as the shadow illustrates.
This may be due to the hard exploration of efﬁcient cooperation without grouping guidance.
Another two intuitive groupings, each agent a group and an equaldivision into two groups {{ a 1 , a 2 , a 3 } , { a 4 , a 5 , a 6 }} , have minor variance.
However, their learningefﬁciency is affected since inappropriate groupings fail to promote cooperative behaviors.
MMM2contains heterogeneous agents; thus, a natural grouping is to keep the agents of the same type in agroup.
As the results show, this natural grouping over-performs setting each agent in a group andall agents in a group.
On the other hand, our mechanism dynamically adjusts the grouping and converges faster.
Setting all agents as a group in 3s5z vs 3s6z also has a great variance.
It is difﬁcultfor agents to learn complex cooperation if each is set to be a group, as the bad performance in bothwin rate and efﬁciency shows.
3s5z vs 3s6z is another scenario with heterogeneous agents, andthe natural grouping of homogeneous agents as a group is studied.
It performs nearly the same asour approach because our dynamic grouping mechanism learns the grouping exactly according tothe agents’ type in this scenario.
We also analyze this map with a grouping containing three groups{{ a 1 , a 4 , a 5 } , { a 2 , a 6 , a 7 } , { a 3 , a 8 }} (1s2z, 1s2z, 1s1z); however, this balanced grouping fails to form effective cooperation.
We can see from these results that multi-agent systems with the samelearning framework perform diversely with various grouping.
Appropriate grouping facilitates efﬁcient cooperation and accelerates learning.
The proposed grouping mechanism can automaticallylearn adaptive grouping in different tasks and assist GoMARL with superior and stable performance.(2)
Learned grouping analysis .
To better demonstrate whether the learned grouping makes sense,we further visualize the ﬁnal trained strategy in one corridor battle, as illustrated in Figure 7.
Six allied Zealots ﬁght twenty-four Zerglings on this super-hard map.
The massive disparity inunit numbers between the two sides implies that the whole team cannot launch an attack together.
The only winning strategy is to sacriﬁce a small number of agents who leave the team and attract theattention of most enemies.
Taking this opportunity, our large force eliminates the rest of the enemies.
The surviving agents then use the same tactic to attract several enemies to the team every time andkill them together.
In our visualization, Agent 1 and Agent 2 sacriﬁce themselves to attract mostenemies and bring enough time for the team to eliminate the remaining enemies.
Other agents ﬁghtas a small group and successfully kill all the surviving enemies.
Our dynamic grouping mechanismlearns a two-group setting in this battle, where Agent 1 and Agent 2 are in the same group while theothers are set in another group.
This grouping is explicable in light of the combat situation, and thisreasonable grouping guidance contributes to the superior performance of our approach.(3) Ablation of the specialized agent networks .
As shown in the bottom row of Figure 6, our specialized agent networks greatly improve the learning efﬁciency.
Although GoMARL’s agents alsoshare all the parameters like vanilla parameter-sharing agents, our proposed info encoder embedsgroup-related information into the agent info with similarity and diversity regularizer. The agentinfo decoder further produces each agent’s upper MLP to generate diversiﬁed policies.
Compared tothe vanilla paramete-sharing mechanism that limits diversity, our agents have various styles of behaviors related to their group, which encourages extensive exploration and accelerates learning.
Inaddition, our agents share all the parameters (GRU, bottom MLP, agent info encoder, and decoder)to obtain their policies, preserving the advantages in scalability of the vanilla parameter-sharingmechanism.
Detailed studies on the inner component of λ SD and the performance improvement ofbaseline methods with our agents are shown in Appendix D to further demonstrate the effectiveness.
4. P ERFORMANCE ON G OOGLE R ESEARCH F OOTBALL
We also test GoMARL on two challenging Google Research Football Academy offensive scenarios,3 vs 1 with keeper and counterattack easy .
Agents in this environment need to coordinate timing and positions for organizing offense to seize ﬂeeting opportunities, and only scoringleads to rewards.
The tasks in GRF are far more difﬁcult than those in SMAC, making many methods that work well in SMAC invalid.
Therefore, a secondary test on GRF is better proof of ourapproach’s effectiveness.
Appendix C.4 details the basic information about the GRF environment.
Performance .
The left side of Figure 8 shows the performance comparison of all methods.
Only
Ft-QMIX and GoMARL achieve more than 30% of the score reward in both scenarios.
QPLEXand ROMA only score with a small probability, while other methods fail to give a single goal.
GoMARL maintains its superior overall performance with outstanding learning efﬁciency.
Theexcellent performance in the second testbed further demonstrates the transferability of our method.
Visualizations .
The trained strategies are visualized to check if the learned grouping makes sense.
Figure 8 (right) visualizes the 3 vs 1 with keeper , and the visualization of counterattackis illustrated in
Appendix E.
The cooperation of two players is enough to score a goal in3 vs 1 with keeper .
As illustrated, Agent 1 passes the ball to Agent 2 (shown in yellow
arrow)when
the opponent player rushing (blue arrow) to tackle.
GoMARL agents master the cooperativetiming and position, and Agent 2 smoothly receives the ball and shoots at the best timing (red arrow).
Our method learns the grouping of {{ a 1 , a 2 } , { a 3 }} that is explicable to the game.
R ELATED W ORK
This paper focuses on cooperative MARL problems with the CTDE paradigm (Oliehoek & Amato,2016).
GoMARL utilizes value function factorization, an approach aiming to address the multiagent credit assignment problem.
Early attempts at value function factorization (Schneider et al.
,1999; Russell & Zimdars, 2003) require apriori knowledge to predeﬁne speciﬁc responsibilitiesor design suitable team reward decompositions.
Deep MARL methods learn value factorizationwith no domain knowledge by treating agents as independent factors.
VDN (Sunehag et al., 2018)learns a linear decomposition into a sum of local utility functions used for greedy action selection.
QMIX (Rashid et al., 2018) enlarges the functions that can be represented by the mixing networkbut still faces the monotonicity limitation.
QTRAN (Son et al., 2019) further improves the expressivity by using constraints between individual utilities and the global action-value; however,the constraints are computationally intractable, and the relaxations often lead to unsatisﬁed performances.
MAVEN (Mahajan et al., 2019) learns latent embeddings to integrate a diverse ensemble ofmonotonic approximations.
LICA (Zhou et al., 2020) learns end-to-end differentiable policy optimization to remove the expressivity constraint.
VMIX (Su et al., 2021) combines A2C with QMIXto extend the monotonicity constraint to value networks.
QPLEX (Wang et al., 2021a) decomposes
Q values with a dueling structure, transferring the monotonicity condition to advantage values.
Focus on group development enables agents to maintain diversiﬁed policies and promote efﬁcientcollaborations.
Early works are proposed only for tasks with a clear structure and train agentswith similar traits to specialize in speciﬁc sub-tasks (DeLoach & Garcia-Ojeda, 2010; Bonjeanet al., 2014).
Recent works attempt to learn individual connections implicitly.
VAST (Phan et al.,2021) learns value factorization for sub-teams based on a priori setting on group number.
Wanget al.
(2019) considers pairwise mutual inﬂuence to encourage interdependence between agents.
ROMA (Wang et al., 2020) learns dynamic roles that depend on the context each agent observes.
Iqbal et al.
(2021) randomly groups agents into related and unrelated groups, allowing agents toexplore only speciﬁc entities.
RODE (Wang et al., 2021b) decomposes the joint action spaces andintegrates the action effects into the role policies to boost learning.
CDS (Li et al., 2021) incorporatesagent-speciﬁc modules to promote sharing among agents while keeping necessary diversity.
C ONCLUSION
Grouping like natural systems is essential to promote efﬁcient cooperation of multi-agent systems.
Instead of predeﬁning grouping utilizing apriori knowledge, this paper proposes an automatic grouping mechanism that gradually learns reasonable grouping as training proceeds.
Based on the dynamic grouping, we further encourage specialization in policies to promote individual similarityand group diversity, achieving efﬁcient intra- and inter-group cooperation.
With these novelties, ourmethod, GoMARL, achieves impressive performance on both SMAC and GRF benchmarks.
A T HE INSIGHT OF GROUP SHIFTING
The proposed automatic grouping mechanism dynamically adjusts the team division as the trainingproceeds.
In the beginning, all agents belong to the same group.
As introduced in Section 3.1, weexamine each w i 1 ( i ∈ [1 , n ]) every c timesteps to check if the grouping needs adjustment.
If thereare agents who take actions but contribute little to their group, it indicates that these agents do notbelong to their current group.
All these selected agents are assigned to the next group (a new onefor agents in the last group) until they appropriately contribute to where they belong.
We utilize the example in Figure 2 (also Figure A1 for your reading convenience) to illustrate theinsight of this grouping shift, i.e. , this shifting scheme guarantees that agents belonging to the samegroup will not be misclassiﬁed into different groups during the dynamic adjustments.
In this example, agent a
i is ﬁrst selected after α 1 · c timesteps’ training, and the initial grouping
G = { a 1 , a 2 , · · · , a n } shifts to G (cid:48)
= {{ a 1 , ·
· · , a i − 1 , a i +1 , · · · , a n } , { a i }} .
Subsequently, afteranother training period, at timestep α 2 · c , two agents a j and a k in the ﬁrst group were selectedsimultaneously to be moved out of the ﬁrst group because of their small contribution.
At this point,they were automatically placed in the second group, i.e. , the group where agent a
i is located.
G (cid:48)(cid:48)
=
{{A − i,j,k } , { a i , a j , a k }} .
Instead of placing a j and a k in a brand new group, our shifting ensuresthat a j and a k have the opportunity to train with a i together as a group, determining if a i , a j , anda k (or two of them) are supposed to be in a group.
Later on, agent a l in the ﬁrst group and agenta i , a k in the second group are chosen at timestep α 3 · c .
The selection of a i and a k indicates thatagent a j cannot cooperate well with them.
Therefore, a i and a k are set in a new group (they shouldnot go back to the ﬁrst group since they have been proved inappropriate for the ﬁrst group), whilea j stays alone in the second group.
Agent a l from the ﬁrst group is assigned automatically to thesecond group ( i.e. , { a j } ) in this example.
A natural question is why this agent is not sent to thethird group { a i , a k } .
There is no need to worry about this under our shifting.
If a l is supposedto be with a i and a k , it will be selected afterward since a j fails to form efﬁcient cooperation with{ a i , a k } . Therefore, a j also cannot cooperate well with a l , and a l will be selected later on and setin group { a i , a k } .
Equipped with the shifting that assigns selected agents to its following group,our proposed automatic grouping mechanism ensures that each grouping is tested by a ﬁxed periodof cooperative attempts.
Therefore, each agent can be grouped with appropriate agents that canefﬁciently cooperate.
B D UAL - HIERARCHY FACTORIZATION SATISFYING IGM
The IGM (Individual-Global-Max) principle proposed by QTRAN (Son et al., 2019) is deﬁned onthe correspondence between individual greedy actions and joint greedy actions.
Formally, if thereexist individual action-value functions (cid:2) Q i (cid:3) n i =1 that satisfy:
[Equation]
where Q tot ( τ , u ) is the joint action-value function with joint action observation histories τ and jointaction u , we say that (cid:2) Q i (cid:3) n i =1 satisﬁes IGM for Q tot .
VDN (Sunehag et al., 2018) and QMIX (Rashid et al., 2018) are renowned methods that attempt tofactorize Q tot assuming additivity and monotonicity respectively that satisﬁes the IGM.
VAST (Phanet al., 2021) combines the value factorization operators of VDN and QTRAN to heritage the IGMproperty.
Here, we formulate the following Proposition 1 to show that GoMARL also maintainsdecentralizability by satisfying the IGM principle for Q tot and Q i .
Proposition 1.
Given a multi-agent system M = (cid:104)A , S, U, P, r, Z, O ; G(cid:105) , where each agent a i ∈ Awith local utility Q i belongs to only one group g j ∈ G for GoMARL’s dual-hierarchy factorization.
IGM is satisﬁed for Q tot and (cid:2) Q i (cid:3) for each agent a i ∈
g j , g j ∈ G .
Proof.
The dynamic grouping mechanism in Section 3.1 regards the mixing weights of the individual utilities as the contribution of Q i to the group value. Therefore, it restricts the learned weights tobe non-negative, as in the monotonic mixing that satisﬁes IGM.
Thus, the maximization of (cid:2) Q i (cid:3) n i =
1maximizes Q g j group
such that u g j = (cid:2) u i (cid:3) i ∈
g j , where u g j = arg max u gj ∈ [ U ] ai ∈ gj Q g j group ( τ g j , u g j )and u i = arg max u i Q i (cid:0) τ 1 , u 1 (cid:1) .
As introduced in Section 3.3, GoMARL’s factorization of Q totinto Q g j group adopts a similar fashion with the factorization of Q g j group into Q tot .
Therefore, thefactorization of Q tot into Q g j group also satisﬁes the IGM such that:
[Equation]
(A2)
Therefore, GoMARL ensures that the greedy local action set of all agents
A = { a 1 , ..., a n } maximizes Q tot for time-varying grouping G according to the IGM principle in Eqn.(A1).
C E XPERIMENT DETAILS
C. D ETAILED EXPERIMENTAL SETUP FOR FAIR COMPARISON
We compare all methods in six StarCraft II micromanagement tasks (SMAC) (Samvelyan et al.
and two challenging Google Research Football (GRF) (Kurach et al., 2020) scenarios.
Methods are trained with 8 parallel runners for 10M steps in both testbeds.
We evaluate each method every10K steps with 32 episodes and report the 1st, median, and 3rd quartile win rates across 5 randomseeds.
The detailed setting of GoMARL’s hyperparameters is introduced in our source code.
Many algorithms introduce implementation tricks when they are implemented.
These code-leveloptimizations were studied in depth in Witt et al.
Yu et al.
; Hu et al.
and wereshown to have a signiﬁcant impact on algorithm performance.
Considering that different baselinealgorithms may use some of these code-level optimizations and thus cause unfair experimental comparisons, this paper conducts experiments under strict control on tricky code-level implementations( e.g. , reward clipping/scaling/normalization, gradient clipping, observation normalization, learningrate annealing, death agent masking, etc. ), ensuring the comparisons are as fair as possible.
Besides,speciﬁc parameter tuning in diverse scenarios is unfair to compare methods, so our experiments useﬁxed parameters for all methods in all environments.
To further ensure fair comparisons, our experiments are based on the PyMARL2 framework proposed for the purpose of fairly comparing algorithms, which is the fairest open-sourced frameworkwe could ﬁnd.
Although RODE and ROMA are not included in this framework, we utilized thesame PyMARL2 settings for them to ensure fairness.
The performance of RODE and ROMA inthis paper is one of the best among public papers, better than their performances in papers from thesame authors or research team (Wu et al., 2021; Zheng et al., 2021) and other research (Jiang & Lu,2021; Jeon et al., 2022), and even better than their original papers in some environments, with absolutely no unfair comparisons.
Please refer to PyMARL2’s open-source implementation 1 for furthertraining details and fair comparison settings.
C. P ARAMETER SIZE FOR VALUE MIXING
GoMARL maintains two mixing networks to estimate Q g group and Q tot respectively.
Whether GoMARL has more parameters for value mixing is a natural question.
We analyze that our inputdimension reduction of the hypernetworks offsets the increase of an extra mixing network in Section 3.3.
Here, we give the detailed mixing network size (GoMARL’s two mixing networks arecounted) of all the methods in Table A1.
The results of GoMARL is the average size of the 5 runssince each run may learn a slightly different grouping with diverse group amounts.
Our two-mixingarchitecture has fewer parameters than the baseline QMIX when there are a large number of agents.
[Table]
C
[Table]
D ETAILED INFORMATION ABOUT SMAC AND ITS SCENARIOS
A group of units controlled by decentralized agents cooperates to defeat the enemy agent system controlled by handcrafted heuristics in each SMAC micromanagement problem.
Each agent’s partialobservation comprises the attributes (such as health , location , unit type ) of all units shownup in its view range.
The global state information includes all agents’ positions and health , and allied units’ last actions and cooldown , which is only available to agents during centralized training.
The agents’ discrete action space consists of attack[enemy id] , move[direction] , stop ,and no-op for the dead agents only.
Particular unit Medivac has no action attack[enemy id]but has heal[enemy id] .
Agents can only attack enemies within their shooting range.
Propermicromanagement requires agents to maximize the damage to the enemies and take as little damageas possible in combat, so they need to cooperate with each other or even sacriﬁce themselves.
Wefollow the default setup of SMAC in our experiments, and more settings, including rewards and detailed observation/state information, can be acquired from the original paper Samvelyan et al.
(2019)or implementation 2 .
Based on baseline algorithms’ performances, the scenarios in SMAC are broadly grouped into threecategories: Easy , Hard , and Super Hard .
The key to winning some Hard or Super Hard battles ismastering speciﬁc micro techniques, such as focus ﬁre , kiting , avoid overkill , et cetera.
The battlescan be symmetric or asymmetric, and the group of agents can be homogeneous or heterogeneous.
Here we provide some characteristics of each Super Hard scenario to help gain insights into thegood or poor performance of the methods:
• corridor is a Super Hard map that need extensive exploration.
Six allied Zealots ﬁght twenty-four Zerglings on this super hard map.
The massive disparity in unit numbers be- 1 https://github.com/hijkzzz/pymarl2https://github.com/oxwhirl/smac
tween the two sides implies that the whole team cannot launch an attack together.
The only
winning strategy is to sacriﬁce a small number of agents who leave the team and attract the
attention of most enemies.
Taking this opportunity, our large force eliminates the rest of
the enemies.
The surviving agents then use the same tactic to attract several enemies to the
team every time and kill them together.
• MMM2 is a representative Super Hard asymmetric battle between two heterogeneous teams with three kinds of units.
One Medivac, two Marauders, and seven Marines have to battleagainst a team with one more Marine.
Marauder has greater attack damage and health thanMarine but with a longer attack cooldown .
Medivac has no damage but can heal anyother agent in the team.
This map with three kinds of units and many agents requires more cooperation between agents, so we picked this map for our ablation studies.
• 3s5z vs 3s6z is another Super Hard map that requires breaking the bottleneck of explo- ration, where three Stalkers and ﬁve Zealots battle against three Stalkers and six Zealots.
C. D ETAILED INFORMATION ABOUT GRF AND ITS SCENARIOS
Google Research Football (GRF) includes several scenarios which can be commonly found in football games.
Agents need to coordinate timing and positions for organizing offense to seize ﬂeetingopportunities, and only scoring leads to rewards.
Each agent’s partial observation contains the absolute positions and moving direction of the ego-agent, relative positions and moving directions ofother agents, and the ball.
The global state information includes all agents’ and ball’s absolute positions/directions.
Agents have a discrete action space of 19, including moving in eight directions,sliding, shooting, and passing.
Proper cooperation requires agents to pass and shoot effectively.
Other details can be acquired from the original paper (Kurach et al., 2020) or its implementation 3 .
Here we provide an introduction of the two scenarios we utilized to compare methods to help gaininsights into the good or poor performance of the methods:
• 3 vs 1 with keeper contains three of our agents and two opponents players (a de-
fender and a keeper). Our agents try to score from the edge of the penalty area. One of
them stands in the middle, while the others are located on both sides of the area.
Initially,
the agent at the center keeps the ball and directly faces the defender.
• academy counterattack easy contains four of our agents and two opponent players(a defender and a keeper).
Agents are initialized far from the penalty area and stand evenlyin an arc centered on the goal. The second agent from the top initially keeps the ball andhas to pass it to a teammate at the appropriate time to avoid interception.
D E XTRA COMPONENT STUDIES ON THE SPECIALIZED AGENT NETWORK
In Section 4.1, we conducted ablation experiments on the proposed specialized agent networks.
Asshown in the bottom row of Figure 6, our specialized agent networks signiﬁcantly improve learningefﬁciency.
Compared to agent networks utilizing the vanilla parameter-sharing mechanism thatlimits policy diversity, our agents have various styles of behaviors related to their group, whichencourages extensive exploration and accelerates learning.
Besides, we study the inner component of the specialized agent networks to provide deeper insight.
Policy specialization in GoMARL is driven by a specialization regularizer, i.e. , the similaritydiversity objective to train the info encoder in Eqn.(2).
The inﬂuence of the scaling factor λ SD onthe performance is shown in Figure A2.
According to these component studies, we set λ
SD
= 0 .
03in GoMARL for all the other experiments.
Furthermore, the proposed specialized agent network is highly transferable.
To further validateits effectiveness, we perform the specialized agents on other baseline methods (Ft-QMIX, QPLEX,
RODE, RIIT) to see if they can perform better.
ROMA is not included since ROMA’s agent networksare produced by its learned roles, and the replacement will destroy the main idea of the method.
Asshown in Figure A3, The performance of all methods is further improved when equipped with ourspecialized agent network.
Speciﬁcally, the learning efﬁciency of Ft-QMIX is boosted.
The variance h of RIIT is markedly reduced, and the win rate is increased by about 10%.
The improvement of
RODE is the most obvious, both the learning efﬁciency and win rate are enhanced, and the varianceis very clearly reduced.
QPLEX’s improvement is not very obvious; however, it obtains a slightlyhigher learning speed and achieves similar performance to GoMARL at the end of training.
Most importantly, even equipped with our specialized network, all baseline algorithms fail to surpass
GoMARL in terms of learning efﬁciency and ﬁnal performance.
GoMARL with vanilla parametersharing looks much inferior to GoMARL with specialized agents, so it is worth questioning whetherthe dynamic grouping module is effective.
This experiment fully illustrates that although dynamicgroup learning may reduce the learning efﬁciency to a certain extent in the early stage of training ,however, in the middle and late training stages , the learned grouping will have a signiﬁcant effectwhen equipped with the proposed specialized agent network.
Therefore, the two main modules of
GoMARL, the dynamic grouping module and specialized agents, are both crucial.
E E XTRA VISUALIZATION OF G OOGLE R ESEARCH F OOTBALL MATCH
In Section 4.2, we visualize a match of 3 vs 1 with keeper to prove the rationality of the grouping GoMARL learned.
Here we give another visualization of the counterattack scenario toshow a more complex strategy and its corresponding learned grouping.
As shown in Figure A4(a), Agent 2 holds the ball at the beginning and faces an opponent rushingtoward him.
Agent 2 passes the ball to Agent 1 to prevent the ball from being stolen.
Subsequently, in(b), Agent 1 carries the ball and tries to break through.
However, the opponent goalkeeper blocks hisattacking route, and Agent 1 chooses to continue passing after a short carry.
Agent 3 makes a goodrun and catches the ball smoothly but shoots quickly in Figure A4(c) since the opponent is close.
The goalkeeper easily saves this hasty attack.
Agent 4 in (d), who learned excellent coordinationwith Agent 3, stops the ball and immediately adds another shot to create the goal.
During this complex goal, GoMARL’s automatic grouping module learned a reasonable grouping {{ Agent 1 , Agent 2 } , { Agent 3 , Agent 4 }} , in which the ﬁrst group successfully brought the ballinto the penalty area through smooth coordination, while the second group ﬁnally created the goalthrough skillful cooperation of shooting and a second shooting.