Heterogeneous Skill Learning for Multi-agent Tasks
Anonymous Author(s)
AffiliationAddress email
Abstract
Heterogeneous behaviours are widespread in many multi-agent tasks, which havenot been paid much attention in the community of multi-agent reinforcement learning.
It would be a key factor for improving the learning performance to efficientlycharacterize and automatically find heterogeneous behaviours.
In this paper, weintroduce the concept of the skill to explore the ability of heterogeneous behaviours.
We propose a novel skill-based multi-agent reinforcement learning framework toenable agents to master diverse skills.
Specifically, our framework consists of theskill representation mechanism, the skill selector and the skill-based policy learningmechanism.
We design an auto-encoder model to generate the latent variable as theskill representation by incorporating the environment information, which ensuresthe distinguishable of agents for skill selection and the discriminability for theskill learning.
With the representation, a skill selection mechanism is invented torealize the assignment from agents to skills.
Meanwhile, diverse skill-based policies are generated through a novel skill-based policy learning method.
To promoteefficient skill discovery, a mutual information based intrinsic reward function isconstructed.
Empirical results show that our framework obtains the best performance on three challenging benchmarks, i.e., StarCraft II micromanagement tasks,
Google Research Football and GoBigger, over state-of-the-art MARL methods.
Introduction
The cooperative multi-agent reinforcement learning (MARL) has obtained great improvements inmany multi-agent systems, such as real-time strategy games [2], artificial swarms of robots [13] andautonomous vehicles [14].
The original solution is the independent learning method, in which eachagent treats other agents as part of the environment.
The main drawback is that simultaneous learningof all agents will result in a non-stationary environment from the perspective of each agent [3].
Toavoid this problem, fully centralized learning approaches [18] have been studied, which aim to learnan optimal joint action of agents with respect to the joint observation.
However, the solution spacegrows exponentially with the number of agents, making it hard to find a good policy in a limitedtime [4].
Later, following the centralized training with decentralized execution (CTDE) paradigm,value decomposition based MARL methods have been proposed which attracts widely attention[21, 26, 33, 31].
The main idea is that each agent learns its policy based on the local observationwhile a centralized mechanism is designed to guide the training of all agents.
In most such methods,agents share their neural networks to compute actions, which is known as the parameter sharingtechnique.
In this way, the training efficiency is significantly improved due to the reduction of thenumber of parameters, and enough samples shared by all agents in the centralized training procedure.
However, heterogeneous behaviours are hard to be generated for such kind of methods when facingcomplex multi-agent tasks.
To address this challenge, some recent methods learn different roles for agents based on theirobservations and actions.
Agents are grouped into different roles which carry out different tasks.
ROMA [27] assumes a Gaussian distribution for learning roles and RODE [28] uses a pre-trainedmodel with random samples to restrict the action space for each role.
However, roles are sometimesnot easy to be distinguished based on only observation-action pairs.
Moreover, the diversity ofheterogeneous behaviours is hard to be fully explored.
Recently, CDS [4] introduces diversity intothe MARL approach by adding an extra local network for each agent to the shared network.
It tries tomake a balance between the diversity and the high learning efficiency brought by parameter sharing.
However, extra introduced networks for all agents hinder the application of CDS on large-scale tasks,which will be illustrated in the experiment.
Different from previous works, we consider studying heterogeneous behaviours in multi-agent tasksby introducing the skill learning [20].
The main idea is to enable agents to learn diverse skills, andthen select proper skills for agents to complete certain tasks.
For instance, in Google Research
Football game, if multiple useful skills such as off-the-ball moving, dribbling with the ball, passingand shooting can be learned, agents can later use these skills to easily cooperate to win the match ina short policy search time.
Introducing skill learning into MARL approaches has two advantages.
On one hand, distinct skill representation will help to generate diverse policies and enable agents toselect heterogeneous behaviours.
On the other hand, the skill learning process is compatible with theparameter sharing technique, which does not bring much overhead for the training efficiency.
Thenthree questions arise: (1) how skills are represented;(2) how skills are learned;(3) how agents selectproper skills.
There are some works studying the problem of skill generation in the single-agent area.
Early worksuse the human-designed reward function to discover skills [11].
However, expert knowledge is oftendifficult to acquire.
Autonomous acquisition of useful skills without human-supervised signals canbe divided into two types.
One is adopting mutual information maximization for skill discovery[6].
It uses model-free unsupervised RL methods to learn distinguishable skills which can performdistinct tasks.
Other works [5, 19] dispose of skill discovery with intrinsic motivation which drivesthe exploration of agents.
However, for those works, skills are learned one by one from differentsingle-agent scenarios.
This is quite different from multi-agent tasks, in which different kinds ofskills must be learned in one scenario.
Moreover, multiple agents should make decisions on theselection of skills and the decision will change with the game goes on.
For multi-agent tasks, HSD[32] studies the problem of skill discovery with a hierarchical RL method.
However, it does not makefull considerations on distinguishable and diverse skill discovery, leading to poor performance forheterogeneous problems.
To learn effective skills in multi-agent tasks, we propose a novel skill-based learning framework.
We design an auto-encoder model to generate the latent variable as the skill representation.
Toreflect the potential characters of agents and ensure diversity of skills, a set of skill-ids as well asobservations and actions of agents are used as the input while the reward of the environment andthe next observations are used as the output.
With the representation, we devise a skill selectionmechanism that assigns an agent with a skill.
Due to characters encoded in the skill representation,this mechanism could select proper skills for agents based on their observations.
Finally, we introducethe information-theoretic paradigm of mutual information to discover distinguishable and diverseskills.
The optimization objective of mutual information is further represented as an intrinsic reward,which together with the environment reward is used to perform skill-based policy learning.
Forskill-based policy learning, we still adopt the parameter sharing technique, but we further computethe latent variable conditioned policy for each skill.
In this way, our framework learns distinguishableskills and encourages each skill to explore the environment and perform its task more effectively.
We make extensive experiments to evaluate our framework on three challenging benchmarks, i.e.,
StarCraft II micromanagement environment, Google Research Football and GoBigger.
To our bestknowledge, the proposed framework achieves state-of-the-art performance over all tasks.
Background
Multi-agent Markov Games : We consider a partially observable multi-agent game, which canbe modelled as multi-agent extension of Markov Decision Processes [34].
It is denoted by⟨ N, S, O, { A i } ni =1 , { R i } ni =1 , P, γ ⟩ , where N is a finite set of n agents.
S is the state space, Ois the partial observation space of agents,
A i is the action of agent
i , R i : S ×
A 1 × · · · × A N → Ris the reward function of agent i and P = S
× A 1 × · · · × A N × S → [0 , 1] is the state transition
{𝒊𝒅_𝒐𝒇_𝒔𝒌𝒊𝒍𝒍|𝑖𝑑=1,2,…,𝑀} 𝑂𝑛𝑒𝐻𝑜𝑡𝐸𝑛𝑐𝑜𝑑𝑒 … … …… …… …… … …
𝑧 1 𝑧 2 𝑧 𝑗 𝑧 𝑀
Skill_IDPool 𝑠 : 𝑜 _ 𝑙 𝑎 𝑡 𝑒 𝑛 𝑡 𝑞(𝑎|𝑠,𝑧 1 ) …
𝑠 : 𝑜 _ 𝑙 𝑎 𝑡 𝑒 𝑛 𝑡 𝑠 : 𝑜 _ 𝑙 𝑎 𝑡 𝑒 𝑛 𝑡 𝑞(𝑎|𝑠,𝑧 𝑗 ) 𝑞(𝑎|𝑠,𝑧 M )
𝑃𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑆ℎ𝑎𝑟𝑖𝑛𝑔 𝑺𝒌𝒊𝒍𝒍𝑺𝒌𝒊𝒍𝒍 𝒋 𝑺𝒌𝒊𝒍𝒍 𝑴 … …
GRU 𝑜 𝑖 𝑜_𝑙𝑎𝑡𝑒𝑛𝑡 𝑖 ℎ 𝑖 ℎ′ 𝑖
Agent 𝑖
A ss i g n m e n t
Skill Representation
Skill Selector
Skill-based Policy Learning …
IntrinsicReward 𝐼(𝑧;𝑠) 𝐼(𝑎;𝑧|𝑠)
𝑄(𝑎 𝑖 ;𝑧 j |𝑜 𝑖 ) 𝑜 𝑖 𝑎 𝑖 𝒛_𝒍𝒂𝒕𝒆𝒏𝒕 … 𝑒 𝑖 𝑜′ 𝑟 𝑑 𝑐𝑜𝑠 (𝑧_𝑙𝑎𝑡𝑒𝑛𝑡 𝑖 ,𝑧_𝑙𝑎𝑡𝑒𝑛𝑡 𝑗 ) 𝑑 𝑐𝑜𝑠
Learning
Representation 𝑧_𝑙𝑎𝑡𝑒𝑛𝑡𝑧_𝑙𝑎𝑡𝑒𝑛𝑡 𝑀 𝑧_𝑙𝑎𝑡𝑒𝑛𝑡 1 𝑧_𝑙𝑎𝑡𝑒𝑛𝑡 𝑗 𝑧_𝑙𝑎𝑡𝑒𝑛𝑡 𝑀 … 𝑄 𝜈 𝐻(𝑎|𝑧,𝑠) 𝜃 𝑒 𝜃 𝑑𝑜 𝜃 𝑑𝑟 𝑧_𝑙𝑎𝑡𝑒𝑛𝑡 𝑗 𝑧_𝑙𝑎𝑡𝑒𝑛𝑡 𝑀 𝑧𝑧 𝑗 𝑧 𝑀 … function.
A popular solution approach for multi-agent games is the value decomposition based
MARL.
Each agent i is associated with a neural network, which is used to compute the Q-value Q ibased on its local observation o i .
During the training process, a global Q-value Q tot is computedas a function of Q i , for example, the summation used in VDN [24] and the mixing network used in
QMIX [21].
Mutual information : Mutual information is a core quantity in the information-theory that measuresthe dependence between random variables using a Shannon entropy-based approach, defined as
Equation (1).
[Equation]
According to the definition, maximizing the mutual information I ( · ; · ) means maximizing the entropy
H ( X ) and minimizing the conditional entropy H ( X | Y ) at the same time.
In the context of reinforcement learning, X refers to the function of the state and Y is the functionof actions.
Maximizing the mutual information encourages the entropy of the state to be high,encouraging exploration in policy learning.
Skill-based RL methods such as [6, 5] adopt the mutualinformation to perform diverse skill discovery.
Method
In this section, we present a novel heterogeneous skill learning MARL framework (HSL) method formulti-agent tasks.
The framework is shown in Figure 1.
We introduce the concept skill which is alatent-conditioned ( Z conditioned) policy π
( A | S, Z ) to enable agents to perform diverse behaviours.
Suppose there are M skills, of which ids are encoded in a one-hot manner, and they are denotedby z j , j = 1 , 2 , ...
M .
We first encode the environment information, i.e, observations and actions ofagents, with skill-ids as latent skill variables z _ latent j , j = 1 , 2 , ...
M .
This process is called skillrepresentation.
Based on latent variables as key features, we design a skill assignment mechanism,which associates each agent with a skill such that agents could learn heterogeneous policies.
Withskill representations and skill assignments, diverse skill policies are learned through the skill learningmechanism.
3.1 Skill Representation Learning
Skill representation learning generates latent skill variables, which are used for skill assignment andskill learning.
On one hand, generated latent skill variables should reflect the characters of differentagents, which helps agents make proper choices over different skills.
On the other hand, they providekey information that makes the learned skills as different as possible.
To achieve the aforementioned goals, we utilize key information from the environment such as theenvironmental state transition function and the reward function for skill representation.
We designan auto-encoder model to generate skill variables.
The encoder is denoted by f e ( · ; θ e ) where θ erepresents parameters of the model.
It encodes the one-hot vector of skill-id z j to the latent skillvariable z _ latent j = f e ( z j ; θ e ) .
There are two decoders with the same input which is a concatenationof the observation o i , the action a i of agent i and all latent skill variables z _ latent =
{ z _ latent j | j =1 , 2 , ..., M } .
The output of one decoder, denoted by f dr ( · ; θ dr ) , is the estimation of environmentreward while that of the other decoder, denoted by f do ( · ; θ do ) , is the estimation of next observationof agent i .
The learning objective of the auto-encoder model is shown as Equation (2).
With the firsttwo terms, the auto-encoder model could generate latent skill variables which reveal the effect ofdifferent skills on observations, actions and the environment.
Consequently, it is beneficial for theskill selector to distinguish agents when their states and actions are similar.
[Equation]
where λ 1 , λ 2 are the scaling factors.
The third term in Equation (2) aims to maximize the cosine distance between any two skills, whichencourages diverse policy learning for the skill learning module.
The skill representation learningmechanism is first trained in the early exploration of our framework.
After the early exploration isfinished, we fix the parameters of the encoder in this module to generate representation features forall skill ids.
3.2 Skill Selector
Here, we introduce a skill selection mechanism that assigns each agent a skill.
This procedure isessentially a many-to-many assignment problem.
It is different from the single-agent skill-based RLproblem, which is only a one-to-many assignment problem.
Relying only on observation features may result in the skill selector assigning the same skill toagents with similar observation features.
Therefore, as shown in the middle part of Figure 1, the skillselector chooses skills based on latent observation features of agents and the latent skill variablesgenerated in Section 3.1.
The latent observation feature o _ latent i is obtained through an MLP and a
GRU unit based on the observation o i , which is similar to other MARL methods.
Then o _ latent iis disposed by another MLP module, of which the output is multiplied by the latent skill variablesz _ latent j , j = 1 , 2 , ..., M .
We use Ω to represent the state space of the skill selector and Λ torepresent the action space.
The Q-values of all agents’ actions are denoted by Q ν 1 , Q ν 2 , ..., Q νN .
In the execution mode, the index corresponds to the maximum value is the skill that should beassigned to the agent i .
For the training of the skill selector, the learning objective is shown as
Equation (3).
Similar to the work in [21], a mixing network is introduced to compute Q νtot .
Notethat the assignment is computed every k steps, thus the reward for each assignment is computed as k t {+′ =
+}
r t + t ′ .
Moreover, in the early training stage, we encourage exploration and incentivize theskill selections to be as diverse as possible.
The policy with high entropy forces the skill selectorto explore distinguished states for agents and chooses different skills for agents, which benefits thetraining procedure.
Therefore, maximizing the policy entropy is also a goal G
=
H
[ Q νi | Ω] in ourobjective.
We replace it with an intrinsic reward r s = − E i [log Q νi ( o _ latent
i , z _ latent i )] and addinto Equation (3) for maximizing the expectation G .
[Equation]
3.3 Skill-based Policy Learning
In this section, we introduce a skill learning mechanism, which consists of a skill discovery processand a skill-based policy learning process.
The main idea of skill discovery includes two aspects.
Thefirst is that skills should do effective exploration in the environment, which means that different skillsshould explore differently and access different states of the environment.
Ideally, several skills shouldcover as much of the exploration space of the environment as possible.
The second aspect is thatskills should be learned as diverse as possible.
In order to avoid duplication policies, each skill shouldcompute the action that differs significantly from the others.
For example, in cooperative football games, it is clear that the two skills of passing and shooting the ball should compute different actionsso that the respective tasks can be completed.
We introduce the mutual information from information theory to realize the aforementioned objectives.
To achieve the first goal, we encourage each skill z to associate with the observation o , i.e. maximizingthe mutual information I ( o ; z ) between skills and observations, where I represents the mutualinformation.
As for the diversity of skills, we hope that different skills generate actions as differentas possible.
Therefore, we can strengthen the correlation between the action a and the skill zthrough maximizing the mutual information I ( a ; z | o ) conditioned on the given state o .
Finally, wecan improve the diversity of skills by maximizing the policy entropy H [ a | o, z ]) , where H [ · ] is the
Shannon entropy.
In summary, the final maximization objective is shown as Equation (4):
[Equation]
For term 2 ⃝ in Equation (4), using the toolkit of variational inference and applying the bound from[1], we can construct the variational lower bound which is shown in Equation (5) based on Theorem1 in Appendix.
(the proof is shown Appendix A).
[Equation]
Further, we can transform E p ( a,z, [Equation] [Equation] into Equation (6).
[Equation]
Inserting Equation (5) and (6) into Equation (4) yields Equation (7).
Note that
H
[ z | o ] in term 1 ⃝ and3 ⃝ are eliminated (see Appendix A for details).
[Equation]
p ( z ) is the skill prior distribution.
We fix this prior distribution to be uniform to ensure its maximization entropy inspired by [6].
To make full use of the information contained in all actions, inspiredby [10], we use the Boltzmann Softmax distribution to generate a policy distribution, which isconsistent with the original Q function with the greedy policy, i.e., p ( a | z, o ) =
Softmax( q θ a ( a | z, o )) .
In this way, it takes into account all actions and makes maximum use of the information.
Since it isdifficult to exactly compute p ( z | a, o ) covering all states, actions and skills, we make a variationalapproximation to p ( z | a, o ) and optimize the lower bound of the variational posterior distributionq θ z ( z | a, o ) .
The final optimization objective is shown as Equation (8).
[Equation]
The discovered skills can be chosen by agents to perform skill-based policy learning.
Each agentcomputes its skill-based policy Q i ( a i , o i | z j ) where z j is the representation of the selected skill.
Toperform policy learning based on the global reward r from the environment, we use a QMIX-stylemixing network to mix all individual Q-values into a global Q-value Q total .
The TD-loss of theskill-based policy learning can be described as Equation (9).
[Equation]
3.4 Overall Optimization Objective
Finally, we describe how skill selector, skill discovery and skill-based policy learning are combinedin our framework.
It is organized as a bi-level learning structure.
Inspired by [6], the optimization goal r m of skill discovery is embedded as an intrinsic reward in L pfor skill-based policy learning, which is described as follows Equation (10).
[Equation]
[Figure]
We train our framework end-to-end by optimizing the following objective function:
[Equation]
(11) where L s is the objective function for the skill selector and η s is the scaling factor in Section 3.
[Figure]
Experiments
In this section, we test the performance of our framework on three challenging multi-agent tasks, i.e.,
StarCraft II micromanagement multi-agent challenge (SMAC)
[22], Google Research Football (GRF)[15] and GoBigger [8].
We compare our approach (HSL) with classical multi-agent value decomposition methods, i.e., QMIX [21] and QPLEX [26], role-based methods , i.e., ROMA[27],RODE[28],diversity-based method CDS [4] and skill-based method HSD [32].
All experiments are conductedover five random seeds.
The detailed setting of the experimental setup is described in Appendix B.2.
4.1 Performance
SMAC : SMAC is a cooperative multi-agent task, in which each agent cooperates with teammates tokill enemies.
Maps in SMAC can be classified as Easy, Hard and Super Hard.
In this section, weconcentrate on one Hard scenario 5m_vs_6m and all Super Hard scenarios, i.e., MMM2, 3s5z_vs_3s6z,27m_vs_30m, corridor, 6h_vs_8z .
Figure 2 shows comparing results of the proposed method, HSL, with other baseline methods.
Aswe can see, HSL achieves the best performance over all scenarios.
In the first three scenarios, evenif there exists a baseline which could achieve a satisfactory win rate, HSL still has a much fasterconvergent speed.
In the last three scenarios, HSL obtains the best win rate.
Especially for 6 h _ vs _ 8 zand 5 m _ vs _ 6 m , HSL improves the win rate by around 10% and 20% respectively, compared to
CDS.
It is interesting to see that CDS does not perform well in the scenario 27 m _ vs _ 30 m .
It is consistentwith the explanation in the introduction that CDS is not suited for large-scale scenarios.
CDSintroduces extra neural networks for all agents without parameter sharing to ensure the diversity.
Although HSL also introduces neural networks for learning skills, different skill representationsensure the diversity and thus parameter sharing still could be applied.
Another notification is that the role-based MARL, i.e., RODE, performs better than CDS in the firsttwo scenarios.
The reason is that agents in the two scenarios are heterogeneous, which is easy for
RODE to distinguish roles based on observations and actions of agents.
HSL is still better than RODEby introducing skills.
For the last three scenarios in which agents are homogeneous, it is difficultfor RODE to distinguish roles.
HSL could obtain the best performance by discovering diverse skills.
In addition, we can observe that HSD cannot win in Super Hard scenarios.
The skill discoverymechanism based only on each agent’s local observation makes it difficult to learn winning strategieseffectively.
[Figure]
GRF :
In GRF, agents work with each other to decide when and where to move and keep the ball aslong as possible to maximize the chances of scoring.
We consider three scenarios, i.e., 3 vs 1 withkeeper, Hard Counter-attack and Corner .
Results are shown in Figure 3, and again HSL performs the best.
It is clear to see that classical valuedecomposition MARL methods, i.e., QMIX and QPLEX, perform poorly on these scenarios due tothe lack of disposing of heterogeneous tasks.
Role-based MARL methods cannot achieve satisfactoryperformance because agents in these scenarios usually get similar states and actions, making it hardto distinguish roles for agents.
Due to the effective skill representation and skill selector mechanism,
HSL performs better than role-based MARL methods.
Further, we could see that CDS is not ableto find effective strategies to win on Corner .
Still, the reason is that the number of agents in thisscenario is much higher than that in the other two scenarios, which leads to an adverse effect ontraining extra networks in CDS.
[Figure]
GoBigger :
In GoBigger tasks, agents control one or more circular balls and cooperate to becomelarge as much as possible by eating food balls and other smaller balls in the environment.
Meanwhile,each agent needs to prevent itself from being eaten by other larger enemy balls.
Agents can move,eject, split and stop, which could be discretized as 13 actions, including moving, ejecting and splittingin four orthogonal directions and stoping.
For each episode, the team with the largest overall size willwin at the end of the game.
We show the performance comparison in Figure 4.
We can observe that these two maps in GoBiggerare quite difficult.
Only HSL and CDS can learn winning policies.
Role-based MARL methodsdo not work at all due to the reason explained before.
In GoBigger, it becomes quite hard forrole-based MARL methods to find roles based on observations and actions.
Due to the effective skillrepresentation and skill assignment, agents in HSL can select useful skills to cooperate with eachother to eat more food and eliminate enemies.
4.2 Ablation Study
In this section, we conduct ablation experiments to evaluate the influence of the three components: (A)
Skill representation learning; (B) Skill selector; (C) Skill-based policy learning.
To test component
A, we replace representations of skills with raw states, which is denoted by HSL No Skill Repr.
Forevaluating component B, we introduce HSL Random Skill , where each agent randomly selects a skill.
Also, we remove the intrinsic reward in the learning objective of the skill selector, which is associatedwith HSL
No Skill Intri.
Reward .
To test component C, we consider HSL No State which removesterm 1 ⃝ , and HSL No Action which removes term 2 ⃝ and 3 ⃝ in Equation (4).
[Figure] [Figure]
Results are shown in Figure 5.
HSL Random Skill obtains the worst performance, which indicatesthat the skill selector plays a key role in the performance of HSL.
Compared with HSL Random Skill ,
HSL No Skill Repr.
achieves higher performance in SMAC:6h_vs_8z and GRF:Hard Counter-attack ,since the raw state of agents could also work for skill assignment.
However, it still performs worsethan our skill representation technique.
It becomes obvious in the task Gobigger:3_vs_3 .
SM No
Skill Repr.
does not work at all.
HSL No Action achieves higher performance than HSL No Statedoes in SMAC:6h_vs_8z and GRF:Hard Counter-attack , which shows that the term 1 ⃝ in (4) isimportant in these scenarios.
In contrast, other terms in (4) are important in HSL No State because theperformance is higher in this scenario.
As for the intrinsic reward in the skill selector, we can observethat the performance gap between HSL and HSL
No Skill Intri.
Reward in GRF:Hard Counter-attackand Gobigger:3_vs_3 is larger than that in SMAC:6h_vs_8z .
The reason is that dense rewards in
SMAC:6h_vs_8z help the skill selector learn effective skill selection policies.
Conversely, the sparcereward in GRF:Hard Counter-attack and the inefficient reward in Gobigger:3_vs_3 increase thepolicy learning difficulty of the skill selector in HSL
No Skill Intri.
Reward .
4.3 Skill Demonstration
In this section, we conduct a case study on scenario SMAC:
6h_vs_8z to demonstrate what ourskill-based framework learn.
We carry out an experiment with the obtained strategy and give a skilldemonstration in Figure 6.
It is clear to see that our framework has learned 3 skills including escaping(red), attacking (green) and kiting (yellow).
The middle part shows the skill selection of agents
duringthe game.
For example, Agent 1 selects the red skill at the beginning and the yellow skill until dies.
The top and the bottom part give visualizations of discovered skills.
In the first two graphs, two agents select the escaping skill to absorb damage and other agents selectthe attack skill to produce damage to enemies.
In the third graph, one agent close to enemies selects escaping skill.
Four other agents select the kiting skill to run in the opposite direction to make damageto enemies.
In the fourth graph and fifth graph, two of the alive agents still choose the kiting skill.
Agent 3 firstly chooses the attacking skill since it is far away from enemies, and soon it chooses toescape when enemies are approaching.
In the last graph, an escaped agent is died, while the remainingagents select the kiting skill to eliminate enemies.
Results on GRF and GoBigger benchmarks aredetailed in Appendix B.3.
Related Work
Deep MARL algorithms have drawn broad attentions in recent years.
Most works take advantage ofthe CTDE paradigm [9] for cooperative policy learning for agents.
Policy-based MARL algorithmsdesign a centralized critic to compute gradients for decentralized actors such as MADDPG[16],
COMA [7] and DOP [30].
Value-based MARL algorithms achieve centralized optimization anddecentralized execution via value decomposition.
These methods introduce a mixing network to mixthe local value functions of each agent to estimate the joint value function.
VDN [25], QMIX [21] and
QTRAN [23] have gradually improved the decomposition capabilities of the mixing networks.
Qatten[34] refines the representation of the joint value function by implementing a multi-head attentionformation to approximate the value decomposition procedure.
Qplex [26] designs a duplex duelingnetwork to encode the IGM principle for higher representation capabilities of the IGM class.
Although many MARL works under the CTDE paradigm have achieved promising results, mostof them apply the parameter sharing technique to share knowledge and experience among agentsto compute cooperative actions.
However, when faced complex multi-agent tasks, centralizedtraining and parameter sharing lead to difficulties in generating heterogeneous behavior.
Therefore,a series of works have been developed to learn heterogeneous policy learning in MARL.
ROMA[27] introduces the concept of role to compute independent actions for different agents.
RODE[28] decomposes the action space into sub-groups in order to reveal proper roles for agents forheterogeneous policy learning.
Recent works introduce diversity to enable agents to learn diversepolicies to solve complex tasks.
MAVEN [17] designs a latent space to learn heterogeneous policiesand encourage exploration.
EITI and EDIT [29] use mutual information to encourage each agentto perform independent exploration to optimize team performance.
EMC [35] proposes a curiositydriven exploration method to make full use of the explored valuable experiences for each agent’sefficient policy learning.
EOI [12] learns each agent’s policy via the combination of the gradientfrom the intrinsic value function and the joint value function.
CDS [4] adds an extra local policynetwork for each agent and tries to make a balance between the high learning efficiency broughtby diversity and parameter sharing.
The most relevant work is HSD [32], which encourages skilldiscovery in MARL settings.
It proposes a bi-level hierarchies to discover cooperative skills for eachagent.
However, HSD selects skills for agents only based on historical observations, which leads topoorly distinguishable skill discovery.
Skill-based policy learning mechanism in HSD cannot learndiverse skills because it is also conditioned on skill features.
By comparison, our approach doesnot totally rely on trajectories, but instead designs a skill representation mechanism and introducesintrinsic rewards for skill selection.
Furthermore, our approach computes intrinsic targets for skilldiscovery and skill-based policy learning via mutual information maximization.
Conclusion
This paper proposes a novel skill-based MARL framework to deal with the heterogeneous problem incomplex cooperative multi-agent tasks.
The concept of skill is introduced, which is represented aslatent variables generated by an auto-encoder model.
The representation reflects the characters ofagents and ensures the diversity of skills, which play an important role in generating and selectingskills for agents.
The skill selection mechanism outputs a skill for an agent based on its observationand skill representation.
The selection is dynamically changed with the real-time game state, whichhelps agents to use proper skills to adapt to complex situation.
To ensure efficient policy learning, wepropose a skill-based policy learning mechanism based on mutual information optimization under theinformation theory paradigm.
Empirical results on three challenging MARL benchmarks show thatour framework significantly pushes forward the performance of state-of-the-art MARL methods.
Weexpect that our approach could provide valuable guidance for future MARL in using skills to solvecomplex cooperative problems in the real world.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Appendix D.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeAppendix E.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section3.3 and Appendix A.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] Include insupplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix B.2.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See all results in Section 4.(d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix B.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Appendix B.3.
(b) Did you mention the license of the assets?
[Yes] See [22], [15] and [8].
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]