LDSA: Learning Dynamic Subtask Assignment in Cooperative Multi-Agent Reinforcement Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Cooperative multi-agent reinforcement learning (MARL) has made prominentprogress in recent years.
For training efficiency and scalability, most of the MARLalgorithms make all agents share the same policy or value network.
However, inmany complex multi-agent tasks, different agents are expected to possess specificabilities to handle different subtasks.
In those scenarios, sharing parameters indiscriminately may lead to similar behavior across all agents, which will limit theexploration efficiency and degrade the final performance.
To balance the trainingcomplexity and the diversity of agent behavior, we propose a novel framework tolearn dynamic subtask assignment (LDSA) in cooperative MARL.
Specifically,we first introduce a subtask encoder to construct a vector representation for eachsubtask according to its identity.
To reasonably assign agents to different subtasks,we propose an ability-based subtask selection strategy, which can dynamicallygroup agents with similar abilities into the same subtask.
In this way, agents dealingwith the same subtask share their learning of specific abilities and different subtaskscorrespond to different specific abilities.
We further introduce two regularizers toincrease the representation difference between subtasks and stabilize the trainingby discouraging agents from frequently changing subtasks, respectively.
Empirical results show that LDSA learns reasonable and effective subtask assignmentfor better collaboration and significantly improves the learning performance onthe challenging StarCraft II micromanagement benchmark and Google Research Football.
Introduction
Cooperative multi-agent reinforcement learning (MARL) has recently received much attention dueto its broad prospects on many real-world challenging problems, such as traffic light control [1],autonomous cars [2] and robot swarm control [3].
Compared to single-agent scenarios, multi-agenttasks pose more challenges.
On the one hand, the observation transition function of each agent isrelated to the policies of other agents, which are constantly updated during training.
Hence, fromthe perspective of any individual agent, the environment is extremely non-stationary, which will beexacerbated with more agents.
On the other hand, the joint action-observation space of the multi-agenttask grows exponentially with the number of agents.
These two issues prevent MARL algorithmsfrom scaling to more agents.
To cope with a large number of agents, most of the recent MARL works, including value-based [4–9]and policy gradient [10–15], utilize a technique of policy decentralization with shared parameters [16],whereby all agents share a decentralized policy or value network.
There are several merits that makeparameter sharing so popular in MARL.
First and foremost, it considerably reduces the total numberof trainable parameters and makes the learning complexity tractable.
Besides, agents with the same parameters can share training experience with each other, which can effectively mitigate the slowdownof convergence rate due to non-stationarity [17].
However, many complex multi-agent tasks consist ofa set of subtasks, where the transition or reward functions are distinct [18, 19].
Solving every subtaskrequires some specific abilities.
In contrast, fully-shared parameters may cause all agents to behavesimilarly and hinder the diversity of agents’ policies.
For example, on the Google Research Footballtask [20], all agents will compete for the ball if sharing parameters [16].
The similar behaviors acrossall agents will limit the exploration efficiency and degrade the final performance [21].
Therefore, it’sdifficult for a single neural network to learn the specific abilities required by all subtasks.
Alternatively, another solution is to learn a separate policy for each agent without any parametersharing, which allows diverse policies but leads to high training complexity.
To balance the trainingcomplexity and the diversity of agents’ behaviors, a better method is to learn a distinct neural networkfor each subtask and group the agents by subtasks.
Agents dealing with the same subtask typicallylearn similar policies and thus can share their training experiences to accelerate training.
However, thismethod poses two new problems: (1) how to decompose a multi-agent task into subtasks and (2) howto assign agents to subtasks.
Most of the previous works [22–25] predefine the task decomposition byusing a rich prior knowledge of application domains, which may not be available in practice.
To thebest of our knowledge, only one recent work, RODE [26], learns explicit task decomposition withoutprior domain knowledge.
RODE defines the subtasks based on joint action space decompositionduring pretraining.
Specifically, RODE learns effect-based action representations to cluster actionsand then treats each cluster of actions as a subtask, where each subtask only needs to fulfill thefunctionality of a subset of actions.
But it may fail to solve subtasks when some basic actionsare necessary for all subtasks, such as the actions of moving in the StarCraft II micromanagementtasks [27].
What’s more, the effect of each action changes dynamically with the environment andthus it may be difficult to determine the actions’ effect only through pretraining.
In this work, we propose a novel framework to learn dynamic subtask assignment (LDSA) in cooperative MARL.
Our method first proposes a subtask encoder that constructs a vector representation foreach subtask according to its identity.
The action-observation history typically reflects the behavioralhabits and potential abilities of each agent, which is an important clue for selecting subtasks.
Therefore, we employ a trajectory encoding network to obtain the action-observation history of each agent.
Then, for every timestep, each agent acquires a categorical distribution of subtask selection based onthe cosine similarity of its action-observation history and representations of all subtasks, and samplesa subtask using Gumbel-Softmax [28] for training, which is a reparameterization trick allowingbackpropagate through samples.
After that, we learn a separate policy for each subtask and enableagents dealing with the same subtask to share their learning.
To associate the policy of each subtaskwith its representation, we introduce a subtask decoder to generate the policy parameters of eachsubtask based on its representation, which can also avoid similar policies between different subtasks.
Furthermore, we introduce two regularizers to increase the representation difference between subtasksand avoid agents changing subtasks frequently to stabilize training, respectively.
We evaluate our method on the challenging StarCraft II micromanagement tasks (SMAC) [27].
The results show that our LDSA significantly improves the learning performance on the SMACbenchmark compared to the baselines, especially on Hard and Super Hard scenarios.
Additionalexperiments on Google Research Football (GRF) [20] further demonstrate the effectiveness of LDSAon various multi-agent tasks.
The ablation studies confirm the benefits of the two regularizers and thehigh efficiency of LDSA.
Moreover, visualizations on SMAC further reveal that LDSA effectivelydecomposes the task with dynamic subtask assignment for better collaboration.
Preliminaries
In this section, we introduce the necessary background knowledge to understand this paper.
We firstdescribe the problem formulation and the definition of subtasks for a fully cooperative multi-agenttask.
Then, we present the value function factorization methods with the centralized training withdecentralized execution (CTDE) paradigm [29, 30], which will be adopted in this paper.
2.1 Problem formulation
This work is focused on a fully cooperative multi-agent task with only partial observation for eachagent.
This task is typically modeled as a decentralized partially observable markov decision process (Dec-POMDP) [31] defined by a tuple G = ⟨ A, S, U, P, r, Z, O, γ ⟩ , where A ≡ { g 1 , g 2 , · · · , g n }denotes the finite set of agents, γ ∈ [0 , 1) is the discount factor and S is the set of global state of theenvironment.
At each timestep, each agent g a ∈
A selects an action u a ∈ U , forming a joint actionu ∈
U n , where a ∈ { 1 , 2 , · · · , n } is the agent identity.
This leads to a transition on the environmentaccording to the state transition function P ( s ′ | s, u ) : S × U n × S → [0 , 1] and all agents receive ashared team reward r ( s, u ) :
S × U n → R , where s ∈ S describes the global state.
Due to the partialobservability, each agent g a only obtains a partial observation o a ∈ Z according to the observationfunction O ( s, g a ) : S ×
A → Z and holds an action-observation history τ a ∈ T ≡ ( Z × U ) ∗ .
Thejoint action-observation history of all agents is denoted as τ .
To achieve scalability, a popular approach adopted by many recent MARL works [4–8] is that eachagent g a constructs a decentralized policy π a ( u a | τ a ; θ ) with shared parameters θ .
However, suchsimple parameter sharing may fail to deal with many complex multi-agent tasks that require diverseabilities among agents.
To this end, we propose to decompose a fully cooperative multi-agent taskinto subtasks.
We present the definition of subtasks in the following.
Definition 1 (Subtasks) .
For a fully cooperative multi-agent task G = ⟨ A, S, U, P, r, Z, O, γ ⟩ , weassume there exists a set of k subtasks, denoted as Φ ≡ { ϕ 1 , ϕ 2 , · · · , ϕ k } , where k ∈ N + is unknownand we consider k as a tunable hyperparameter.
Each subtask ϕ
i is defined by a tuple ⟨ x ϕ i , G ϕ i , π ϕ i ⟩ ,where i ∈ { 1 , 2 , · · · , k } is the identity of subtask, x ϕ i ∈ R m is a vector representation (or latentembeddings) of subtask ϕ
i , G ϕ i = ⟨ A ϕ
i , S, U, P, r, Z, O, γ ⟩ is the Dec-POMDP model of subtask ϕ
i ,and π ϕ i : T × U → [0 , 1] is the policy of subtask ϕ
i .
A ϕ
i is the set of agents assigned to subtaskϕ i and each agent can only select one subtask to solve at each timestep, i.e., (cid:83) ki
=
A ϕ i
= A and
A ϕ i (cid:84)
A ϕ j = ∅ if i ̸
= j .
Each agent g a ∈
A ϕ i shares the policy parameters of π ϕ i .
Our objective is to learn a optimal set of subtasks Φ ∗ so as to maximize the expected global return
Q Φ tot ( τ , u ) =
E s 1: ∞ , u 1: ∞
[ (cid:80) ∞ t =0 γ t r t | s 0 = s, u 0 = u , Φ] , where r t is the team reward at timestep t .
Therefore, we are required to learn the vector representation x ϕ i , the agent assignment
A ϕ i and thepolicy
π ϕ i for each subtask ϕ
i , which will be introduced in detail in Sec. 3.
2.2 Value function factorization with CTDE paradigm
To make policy learning stable and scalable in MARL, the paradigm of CTDE [29, 30] has beenproposed and gained substantial attention.
In this paradigm, agents learn their policies togetherwith access to global information during training in a centralized way and only rely on their localobservations during execution in a decentralized way.
The CTDE paradigm has been exploited bymany recent MARL algorithms [4–8, 10, 11], among which value function factorization methods [4–8] have shown superior performance on challenging cooperative multi-agent tasks [27].
Thesemethods factorize the global Q-value function Q tot into individual Q-value functions Q a for eachagent g a , where Q a is only based on the local action-observation history τ a for decentralized execution.
To guarantee the consistency between greedy action selection in global and individual Q-values, suchfactorization has to satisfy Individual-Global-Maximum (IGM) principle [6] as follows:
[Equation]
In this work, we consider a representative value function factorization method, QMIX [5], whichestimates global Q-value as a non-linear combination of individual Q-values as follows:
[Equation]
where f w is a monotonic function that conditions on the global state s , i.e. , ∂f w ∂Q a ≥ 0 , ∀ a ∈{ 1 , 2 , · · · , n } .
Although Eq. 2 is only sufficient and unnecessary to IGM, QMIX is a lightweight andefficient method showing state-of-the-art performance on SMAC [27].
Method
In this section, we present the LDSA learning framework as shown in Fig.
We first introduce howto construct a set of distinct subtasks for decomposing a multi-agent task.
Next, we discuss how
Trajectory Encoder 𝑥 𝜏 𝑎 𝑥 𝜙... 𝑥 𝜙𝑥 𝜙 𝑘 𝑜 𝑎 (a) (b) (c)
Subtask Encoder ℎ 𝑎 ℎ 𝑎1′ 𝑜 𝑎 𝑜 𝑎 ℎ 𝑎 ℎ 𝑎2′
Gumbel Softmax
Sampling ...
𝑥 𝜙 𝑘 ℎ 𝜏 𝑎 𝑄 𝑎,𝜙 𝑘
Subtask Decoder 𝜃 𝜙 𝑘
One-hot 𝑄 𝑎
MixingNetwork 𝑸 −𝑎 𝑄 𝑡𝑜𝑡
FC FC
GRU GRU 𝑥 𝜙 𝑖 𝑑 𝑖
FC
FC FC
FC 𝑥 𝜏 𝑎 . ..
Subtask 𝝓 𝟏 policy
Subtask 𝝓 𝟐 policy
Subtask 𝝓 𝒌 policy ℎ 𝜏 𝑎 each agent selects a subtask according to its abilities. After grouping agents by subtasks, we presentthe policy of each subtask that is representation-dependent.
Finally, we show the overall trainingobjective and the inference strategy.
3.1 Distinct subtask representation
Many complex multi-agent tasks involve a set of subtasks that have different responsibilities.
Forexample, running a company requires multiple departments to work together.
Previous work typicallyleverages the prior domain knowledge to decompose a complex task, which is not practical for manyuncertain environments.
To remove the dependence on prior knowledge and apply it to broadermulti-agent tasks, we propose to learn a vector representation x ϕ i for each subtask ϕ
i according to itsidentity i .
Specifically, we use a two-layer fully-connected (FC) network with a tanh activation function in
Fig.
1(a) to learn a subtask encoder f e ( · ; θ e )
: R k
→ R m , parameterized by θ e .
The subtask encodermaps the one-hot identity d
i ∈ R k of subtask ϕ
i to an m -dimensional representation space.
The tanhactivation function is to constraint the value range of the representation space.
Then, we employ thesubtask encoder to construct a vector representation x ϕ i ∈
R m for each subtask ϕ
i , i.e. ,
[Equation]
If subtasks are similar, the task decomposition makes little sense.
Therefore, to keep differencesbetween subtasks, we propose a regularizer that maximizes the L 2 distance between representationsof different subtasks as follows:
[Equation]
where D is the replay buffer.
The subtask representation learning continues throughout the trainingprocess, which can automatically adapt to the dynamic changes in the environment.
The overalloptimization objective of the subtask encoder will be introduced in Sec. 3.4.
3.2 Ability-based subtask selection
With the representation of subtasks, we need to design a subtask selection strategy for each agentbased on its ability.
In real life, we can easily infer a person’s role from the trajectory of his or herbehavior.
Similarly, the action-observation history of an agent can reflect its behavioral habits andpotential abilities.
Therefore, we utilize a shared trajectory encoder (shown in Fig. 1(b))
consisting ofa GRU [32] and two fully-connected networks to obtain the action-observation history of each agent.
The trajectory encoder f h ( · ; θ h ) , parameterized by θ h , encodes the action-observation history of eachagent g a into a vector x τ a ∈ R m , which has the same length with the subtask representation.
Then,agent g a treats x τ a as its ability representation and selects a subtask to solve based on x τ a .
In our implementation, for each agent g a , we first calculate the cosine similarity of its actionobservation history representation x τ a and representations of all subtasks x Φ :=
[ x ϕ i ] ki =1 , i.e. ,similarity ( x τ a , x ϕ i ) =
( x T τ a x ϕ i ) / ( ∥ x τ a ∥∥ x ϕ i ∥ ) .
Since representations of all subtasks have thesame value range through the tanh activation function, we use x T τ a x ϕ i to approximate the cosinesimilarity for simplicity.
Then, we employ softmax function on the cosine similarity and obtain acategorical distribution of subtask selection p (Φ | x τ a , x Φ ) :
= [ p ( ϕ i | x τ a , x Φ )] ki =1 . p ( ϕ i | x τ a , x Φ ) isthe probability that agent g a selects subtask ϕ i , given by:
[Equation]
where exp ( · ) is the exponential function.
Sampling a subtask directly from the categorical distributionp (Φ | x τ a , x Φ ) is not differentiable.
To make the subtask selection process trainable, we use the
Straight-Through Gumbel-Softmax Estimator [28] to sample a subtask ϕ
j that will be discretized asa k -dimensional one-hot vector, i.e. , the one-hot subtask identity d j .
For every timestep, each agent will select a subtask to dynamically cooperate with each other.
Thisdynamic subtask assignment scheme may make training unstable when an agent frequently changesthe subtask in an episode.
To smooth the agents’ subtask selection and stabilize training, we introducea second regularizer to minimize the KL divergence between the subtask selection distributions forany two adjacent timesteps as follows:
[Equation]
(6) where p ′ (Φ ′ | x ′ τ a , x ′ Φ ) is the subtask selection distribution at the next timestep, D KL ( ·∥· ) is the KLdivergence operator and the sum is performed across all agents.
3.3 Representation-dependent subtask policy
After grouping agents to different subtasks based on their abilities, we learn the policy for eachsubtask which is illustrated in Fig.
1(c).
Generally, agents dealing with the same subtask sharethe policy parameters and different subtasks have distinct policy parameters.
To this end, we firstuse a new shared trajectory encoder f τ ( ·| θ τ ) with parameters θ τ to generate the action-observationhistory of each agent g a , denoted as h τ a .
The new trajectory encoder f τ ( ·| θ τ ) only consists of afully-connected network and a GRU.
The policy of each subtask ϕ
i is a fully-connected networkf ϕ
i ( · ; θ ϕ i ) with parameters θ ϕ i .
Then, for each agent g a ∈
A ϕ i that solves the subtask ϕ
i , we feed itsaction-observation history
h τ a into the subtask policy f ϕ i ( · ; θ ϕ i ) and generate the individual Q-value
Q a .
To associate the policy of each subtask with its representation, we utilize a subtask decoderf d ( ·| θ d ) conditioning on the subtask representation to generate the policy parameters of each subtask.
The subtask decoder f d ( ·| θ d ) with parameters θ d is just a single-layer fully-connected network, whichmaps x ϕ i to θ ϕ
i for each subtask ϕ
i .
By virtue of the differences between subtask representations,such subtask decoder can further increase the diversity of subtask policies.
In our implementation, for each agent g a , we feed h τ a into all subtask policies to get the individual
Q-value Q a,ϕ i for every subtask ϕ
i .
The individual Q-value Q a is the sum of [ Q a,ϕ i ] ki =1 weighted bythe one-hot identity of the selected subtask.
If agent g a selects subtask ϕ j , Q a is actually equal to
Q a,ϕ j .
Hence, each agent only trains the policy parameters of its selected subtask.
In this way, agentswith similar abilities tend to select the same subtask and thus can share their experiences to acceleratetraining and improve performance.
Moreover, on the tasks that require to assign different subtasksto agents with similar trajectories, since each agent’s observation input contains its one-hot identity,
LDSA could learn to put more weights on the agent’s identity input, and make the agent’s identityinput become the main factor of the output subtask selection distribution.
Then two agents can havedistinct subtask selection distributions even if they have similar observation from the environment.
3.4 Overall training and inference
Similar to value function factorization methods [4–8], we use a mixing network with parametersθ w to map all agents’ individual Q-values ( Q a , Q − a ) into the global Q-value Q tot , where Q a is theindividual Q-value of agent g a and Q − a are the individual Q-values of other agents.
In this work,we adopt the mixing network introduced by QMIX [5] thanks to its simple structure and superiorperformance.
It can be easily extended to other more complex mixing networks [7, 8].
Then, all theparameters θ :=
( θ e , θ h , θ τ , θ d , θ w ) of our framework can be optimized by minimizing the TD lossof Q tot as follows:
[Equation]
where θ − are the parameters of the target network that are periodically copied from θ .
Consideringthe two regularizes in Eq.
4 and Eq.
6, the overall optimization objective of LDSA is:
[Equation]
where λ ϕ and λ h are positive coefficients of the two regularizers, respectively.
During the inferencephase ( i.e. , test the decentralized policy), each agent g a selects the subtask with the maximumprobability on the subtask selection distribution, i.e. , arg max ϕ i p ( ϕ i | x τ a , x Φ ) and then chooses agreedy action according to the individual Q-value Q a for decentralized execution.
Experiments
In this section, we conduct several experiments to investigate the following questions: (1)
Can
LDSA improve the performance compared to the baselines?
(Sec. 4.2) (2) How do the two proposedregularizers influence the performance?
(Sec. 4.3) (3) Whether the superior performance of ourmethod comes from the increase in the number of parameters?
If not, which component contributesthe most to our method?
(Sec. 4.3) (4) Can LDSA learn dynamic subtask assignment and groupagents reasonably?
If so, agents with similar abilities solve the same subtask and each subtask hasspecific responsibility.
(Sec. 4.4)
4.1 Experimental setup
Environment We evaluate LDSA on the SMAC benchmark [27], a challenging benchmark forcooperative MARL.
There are two armies of units in the SMAC environment.
Each ally unit iscontrolled by a decentralized agent that can only act based on its local observation and the enemyunits are controlled by built-in handcrafted heuristic rules.
The goal of the MARL algorithm is tomaximize the test win rate for each battle scenario.
In this paper, we adopt the default environmentsettings for SMAC.
The version used in this work is SC2.4.10.
We consider all 14 scenarios on the
SMAC benchmark that can be classified into three different levels of difficulties: Easy , Hard and
Super Hard scenarios.
Baselines We select QMIX [5], ROMA [33] and RODE [26] as our baselines.
All of the baselinesand LDSA belong to value function factorization methods and use the QMIX-style mixing network [5]for a fair comparison.
Therefore, LDSA differs from the baselines only by the individual Q-valuenetworks.
QMIX is a natural baseline in which all agents share the individual Q-value network.
ROMA and RODE are two recent works that learn implicit and explicit task decomposition in MARL,respectively.
We implement all the baselines using their open-source codes based on PyMARL [27].
Note that, our experiments are conducted under the assumption that there is no prior knowledgeabout the environment.
As mentioned in Sec. 1, RODE may not work when some basic actionsare necessary for all subtasks.
In the SMAC environment, there are four actions representing theagents’ movement in four cardinal directions, which are important for all subtasks.
To be effective,
RODE uses several rules based on prior knowledge of the environment in its implementation, whichmanually enforce the four actions of moving be available to some subtasks forever under certainconditions.
Therefore, to not involve prior knowledge, we remove these rules when we implement
RODE.
We use "RODE_nr" to represent RODE without manual rules in our experiments.
Hyperparameters For all experiments, the number of subtasks is set to 4 and the length of subtaskrepresentations is set to 64, i.e. , k = 4, m = 64.
We carry out a grid search for regularizers’coefficients λ ϕ and λ h on the SMAC scenario corridor and then set them to 10 − 3 and 10 − 3 ,respectively, for all scenarios.
All the common hyperparameters of our method and baselines areset to be the same as that in the default implementation of PyMARL.
We use lightweight networkstructures for the subtask encoder, subtask decoder, trajectory encoder and subtask policy.
Thedetailed hyperparameters and network structures will be provided in Appendix A.1.
4.2 Performance on SMAC
We compare LDSA with the baselines across all 14 scenarios on the SMAC benchmark [27].
Forevery scenario, we carry out 5 independent runs with 5 different random seeds for all methods andshow the median performance and 25-75% percentiles.
Fig.
3 presents the averaged median testwin rate across all 14 scenarios.
We can observe that LDSA significantly improves the learningperformance and surpasses about 7% median test win rate averaged across all 14 scenarios.
[Figure]
Fig.
2 shows the learning curves of LDSA and the baselines on six Hard or Super Hard scenarios.
The resultson the other eight scenarios are shown in Appendix A.2.
LDSA outperforms all the baselines on all Super Hardand Hard scenarios with faster convergence.
Specifically,on the scenarios that require diverse micromanagementtechniques: corridor , 3s5z_vs_3s6z and 6h_vs_8z ,the test win rate of LDSA exceeds that of the baselines atleast 15%.
On the Easy scenarios, LDSA shows similarperformance as QMIX, which indicates that our methodmay not improve the learning performance on simple tasksnot requiring task decomposition.
Moreover, ROMA thatlearns implicit task decomposition performs even worsethan QMIX on most of the scenarios.
Without using rulesbased on prior knowledge, RODE fails to learn efficientpolicies for subtasks, which demonstrates that discovering subtasks based on joint action spacedecomposition and restricting the action space of subtasks may severely limit the learning of subtaskpolicies.
These results further confirm that LDSA can learn effective task decomposition with dynamic subtask assignment to solve complex tasks.
In Appendix A.4, we compare LDSA with anotherbaseline CDS [16] that celebrates diversity among agents, and show the benefits of LDSA to balance the training complexity and the diversity of agent behavior.
We also evaluate LDSA on Google
Research Football (GRF) [20] to demonstrate the effectiveness of LDSA on various multi-agent tasksin Appendix A.5.
4.3 Ablation studies
In this subsection, we conduct ablation studies on three scenarios: corridor , 5m_vs_6m and2c_vs_64zg , to show the effect of the two regularizers and test which component contributes most to
LDSA.
Fig.
4 reports the performance of LDSA when we remove each or both of the two regularizers
L h and L ϕ .
It can be observed that the performance of LDSA without both of the two regularizers is even worse than that of QMIX.
Either of the two regularizers can improve performance and L ϕimproves more than L h , which indicates that keeping differences between subtasks is more importantwhile L h can stabilize training.
To figure out why LDSA performs better than QMIX, we investigate the contributions of threecomponents of LDSA: the increase in the number of parameters, the ability-based subtask selectionand the subtask decoder.
As shown in Fig.
5, QMIX with similar parameters as LDSA can’t improveperformance and even may make training slower and degrade performance, which proves thatthe superior performance of LDSA is not due to the increase in the number of parameters.
Theperformance of LDSA without the subtask decoder is lower than that of LDSA, which demonstratesthat associating subtask policy with subtask representation benefits the performance.
What’s more,the large margin between the performance of LDSA with random subtask selection and that of LDSAreveals that the ability-based subtask selection contributes the most to our method.
In summary, thesuperior performance of LDSA is largely due to the efficient subtask assignment.
We also studythe effect of the number of subtasks in Appendix A.3 and conduct two more ablations for LDSA in
Appendix A.6.
4.4 Visualization of dynamic subtask assignment
In this subsection, we visualize the dynamic subtask assignment in one episode on the SMAC scenariocorridor as shown in Fig.
The scenario corridor consists of 6 ally Zealots and 24 enemy
Zerglings.
Due to the huge disparity in the number of allies and enemies, it’s hard to win if all ally
Zealots rush to attack the enemy.
Therefore, at the beginning ( t = 3), our method learns to sacrifice
[Figure]
two allies (agents g 1 and g 4 ) to attract most enemies and then the other 4 allies (agents g 2 , g 3 , g 5 andg 6 ) can focus fire to kill a small part of enemies.
After that, the remaining enemy Zerglings controlledby heuristic rules will go through a narrow corridor, and stay at the bottom left corner of the scenarioif there are no Zealots within the sight range.
In the middle of the episode ( t = 64), the remainingenemy forces are still strong, our method assigns the healthiest ally (agent g 2 ) to draw out a smallnumber of enemies and kill them with the other allies (agents g 3 , g 5 and g 6 ).
This process will repeatuntil all enemies are killed.
Moreover, we find that our method will assign the dead allies (agents g 1and g 4 ) to a fixed subtask (subtask ϕ 3 ), which can prevent them from interfering with the learning ofother subtasks.
Besides, it is worth noting that subtask ϕ 1 and subtask ϕ 4 have similar responsibilitiesthat agents focus fire to kill enemies, but under different situations related to the agent’s health point.
When an agent needs to focus fire to kill enemies, it tends to select subtask ϕ 1 if it has a near fullhealth point otherwise select subtask ϕ 4 .
These visualizations demonstrate the effectiveness andrationality of the dynamic subtask assignment in our method.
Related work
Parameter sharing in MARL To learn efficiency and scalability, most of the MARL worksemploy the technique of parameter sharing among agents [4–15].
Terry et al.
[17] demonstratethat parameter sharing can effectively alleviate the non-stationarity problem in MARL.
Despite itsefficiency, fully-shared parameters among agents may destroy the diversity of agent behavior inmany complex multi-agent tasks [20, 27].
To trade off experience sharing and behavioral diversityamong agents, SePS [21] groups agents based on their identities during pretraining and each groupshares one policy for training.
CDS [16] proposes to decompose the policy of each agent as the sumof a shared part and a non-shared part.
In this work, we don’t fix the group each agent can sharewith.
Each agent in our method can adaptively choose a group to share based on its trajectory duringtraining, which enables dynamic parameter sharing.
We discuss more related works in Appendix A.7.
Conclusion
Task decomposition is an important approach to simplify complex multi-agent tasks and has notbeen well solved without using prior knowledge.
To this end, we propose to decompose the taskinto several subtasks represented by latent embeddings.
Agents select subtasks according to theirabilities, which are indicated by their action-observation histories.
In this way, agents dealing withthe same subtask can share their learning to solve the subtask, which can learn the specific abilitiesrequired by all subtasks under a tractable training complexity.
Although the embedding representationfor each subtask may be abstract, it essentially clusters agents with similar abilities into the samegroup and thus agents can dynamically share their experiences to accelerate training and improveperformance.
After training, each agent is aware of all subtask policies and can adaptively choosethe most appropriate subtask policy to execute based on its ability.
The empirical results furtherdemonstrate that LDSA learns effective task decomposition with dynamic subtask assignment andsignificantly improves the learning performance compared to the baselines.
We hope our methodcould provide a new perspective for task decomposition and subtask learning in MARL.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] In Sec. 4.2, we mention that
our
method may not improve the learning performance on simple tasks not requiring task
decomposition.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Although
we haven’t evaluated our method on real applications, we believe our work doesn’t
have potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We include the
code, data, and instructions needed to reproduce the main experimental results in the
supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Sec.
4.1 and Appendix A.1.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We report the median performance as well as the 25-75%
percentiles across 5 random seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix A.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We implement our
method based on the open-sourced PyMARL framework, which has been cited inSec.
(b) Did you mention the license of the assets?
[Yes] PyMARL is licensed under the Apache
License 2.0.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]
A AppendixA.1 Architecture and hyperparameters
In this paper, we use lightweight network structures for the subtask encoder, subtask decoder,trajectory encoder and subtask policy.
Specifically, the subtask encoder is a two-layer 64-dimensionalfully-connected network and the output layer is followed by the tanh activation.
Agents sharea trajectory encoder to construct their ability representations.
The trajectory encoder consists ofthree layers, a 64-dimensional fully-connected layer, followed by a 64-bit GRU, and followed byanother 64-dimensional fully-connected layer.
The new shared trajectory encoder only consists ofa 64-dimensional fully-connected layer followed by a 64-bit GRU and the outputs will be fed intothe subtask policy.
The policy of each subtask is a fully-connected layer of n _ actions dimensions,where n _ actions is the number of actions.
The parameters of the subtask policy are generated by thesubtask encoder, which is also a fully-connected layer.
We use the QMIX-style mixing network withthe same setting as the original paper.
For all methods, we run 5 million timesteps on three Super Hard scenarios: corridor ,3s5z_vs_3s6z , 6h_vs_8z and 2 million timesteps on the other 11 scenarios.
The replay buffer isa first-in-first-out queue with a max size of 5000 episodes.
For every one episode data collected,we sample a batch of 32 episodes from the replay buffer for training.
The decentralized policy isevaluated every 10 k timesteps with 32 episodes.
We use ϵ -greedy exploration.
The ϵ anneals linearlyfrom 1.0 to 0.05 over 50 k timesteps and keeps constant for the rest of the learning.
The trainingobjective is optimized by RMSprop with a learning rate of 5 × 10 − 4 .
All experiments are conductedon GeForce RTX 2080Ti GPU.
A.2 Remaining results on the SMAC benchmark
[Figure]
on the remaining SMAC scenarios, including five Easy scenarios: 2s_vs_1sc , 2s3z , 3s5z , 1c3s5z , 10m_vs_11m and one Hard scenario: bane_vs_bane and two Super Hard scenarios: 27m_vs_30m , MMM2 .
A.3 Effect of the number of subtasks
The number of subtasks k is a core hyperparameter of LDSA.
To study the effect of k , we test LDSAwith different values of k on the SMAC scenario corridor as shown in Fig.
It can be observedthat LDSA can obtain a better performance than QMIX when k = 2 and increasing the number ofsubtasks benefits the performance.
But LDSA with more than 4 subtasks shows little improvementand learning more subtasks will lead to higher computational cost.
Therefore, we set k = 4 throughoutthe work to balance the effectiveness and efficiency.
[Figure]
[Equation]
[Figure]
FRUULGRU N  N
 N  N

A.4 Comparison of LDSA with CDS on SMAC
In this section, we compare LDSA with CDS [16] on three SMAC scenarios: 5m_vs_6m , 10m_vs_11mand 27m_vs_30m as shown in Fig.
We can see that LDSA performs better than CDS on all threescenarios.
Moreover, CDS can’t learn anything on the scenario 27m_vs_30m that needs to control alarge number of agents.
This may be because CDS requires to learn one shared policy network and nnon-shared policy networks ( i.e. , a total of n + 1 policy networks) if we have n agents, which maylead to high training complexity when the multi-agent task contains a large number of agents.
Ourmethod only needs to learn k subtask policies, where k ≪ n when controlling a large number ofagents.
Therefore, compared with CDS, LDSA can achieve a better balance between the trainingcomplexity and the diversity of agent behavior.
with CDS on three SMAC scenarios: 5m_vs_6m , 10m_vs_11m and 27m_vs_30m .
A.5 Performance on GRF
To validate the benefits of LDSA on more multi-agent tasks, we evaluate our method on two challenging Google Research Football (GRF) [20] academy scenarios academy_3_vs_1_with_keeperand academy_counterattack_easy .
On both scenarios, we can control more players against twoenemy players.
We can solve these two tasks similarly by learning two main different subtasks.
Oneis to dribble the ball to the top side of the field and attract the attention of the enemy players to create an enemy vacancy in the down side of the field.
The other is to run to the down side of the fieldand wait for a long pass from teammates on the top side, and then shoot.
We show the median and25-75% percentiles of the test score reward across five random seeds in Fig.
LDSA could alsoachieve better performance than baselines on GRF, which demonstrates the effectiveness of LDSAon various multi-agent tasks.
[Figure]
A.6 Additional ablations
In this section, We conduct two more ablations for LDSA, named LDSA_DireProb and LDSA_Mix,respectively.
LDSA_DireProb means to directly estimate the subtask selection distribution fromagents’ observed trajectories.
LDSA_Mix represents that each agent uses a mixture of subtaskpolicies as opposed to sampling one subtask policy.
We compare these two ablations with LDSA onthe SMAC Super Hard scenario corridor as shown in Fig.
We can observe that the performanceof LDSA_Mix is lower than that of LDSA, which indicates that it’s better for each agent to learnone subtask rather than a mixture of all subtasks at each timestep.
Besides, the comparison between
LDSA and LDSA_DireProb highlights the importance of computing the subtask selection distributionbased on similarity of agent trajectories and subtask representations.
In other words, if we haveexplicit subtask representation and the subtask policy depend on its representation, it’s hard to learnto select the subtask if we have no information about the subtask.
These two more ablations furtherconfirm the effectiveness of LDSA.   
  7LPHVWHSV0       0 HG L DQ  7 H V W : L Q   D W H    FRUULGRU /'6$ /'6$B0L[ /'6$B'LUH3URE
A
Related
work
Value-based MARL Value-based MARL algorithms have achieved great progress in recent years.
IQL [34] simply trains an independent Q-value network for each agent, which treats the otheragents as part of the environment.
This method may not converge due to the non-stationarity of theenvironment caused by the changing policies of other agents.
The other extreme method is to treatall agents as a single agent and learn a global Q-value based on the joint action-observation space, which alleviates the non-stationarity but suffers from the scalability challenge, as the joint actionobservation space grows exponentially with the number of agents.
To trade off these two methods,most of the value-based MARL algorithms factorize the global Q-value into individual Q-valuesfor centralized training and decentralized execution.
VDN [4] and QMIX [5] factorize the global
Q-value by additivity and monotonicity, respectively.
QTRAN [6] transforms the original global
Q-value function into an easily factorizable one to expand the representation capacity.
QPLEX [8]proposes a duplex dueling network architecture to implement the complete IGM [6] function class.
These methods focus on designing the mixing network.
There are other value-based works studying
MARL from the perspective of communication [35, 36], exploration [37, 38] and robustness [39, 40].
Task decomposition in MARL Task decomposition plays an important role in many complexreal-world multi-agent systems, such as software engineering [41], healthcare [42] and traffic management [43].
Once the multi-agent task is decomposed, agents can be assigned to the restrictedsubtasks that are easier to solve, and thus the learning complexity is greatly reduced.
However, it ischallenging to come up with a set of subtasks that can effectively decompose the whole multi-agenttask.
The most straightforward way is to predefine the subtasks by leveraging the prior domainknowledge [22–25].
But the prior knowledge may not be available in many uncertain environments.
To solve this problem, ROMA [33] introduces the concept of roles for each agent based on its localobservation and conditions agents’ policies on their roles.
Nguyen et al.
[44] redesign the first layer ofthe mixing network in QMIX [5] and view the output nodes of the first layer as different roles.
Thesetwo works focus on implicit task decomposition.
RODE [26] explicitly defines the roles/subtasksbased on joint action space decomposition during pretraining, where each subtask corresponds to asubset of actions.
In this work, we learn vector representations for subtasks that can automaticallyadapt to the environment during training.