LDSA: Learning Dynamic Subtask Assignment in Cooperative Multi-Agent Reinforcement Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Cooperative multi-agent reinforcement learning (MARL) has made prominentprogress in recent years.
For training efficiency and scalability, most of the MARLalgorithms make all agents share the same policy or value network.
However, inmany complex multi-agent tasks, different agents are expected to possess specificabilities to handle different subtasks.
In those scenarios, sharing parameters indiscriminately may lead to similar behavior across all agents, which will limit theexploration efficiency and degrade the final performance.
To balance the trainingcomplexity and the diversity of agent behavior, we propose a novel framework tolearn dynamic subtask assignment (LDSA) in cooperative MARL.
Specifically,we first introduce a subtask encoder to construct a vector representation for eachsubtask according to its identity.
To reasonably assign agents to different subtasks,we propose an ability-based subtask selection strategy, which can dynamicallygroup agents with similar abilities into the same subtask.
In this way, agents dealingwith the same subtask share their learning of specific abilities and different subtaskscorrespond to different specific abilities.
We further introduce two regularizers toincrease the representation difference between subtasks and stabilize the trainingby discouraging agents from frequently changing subtasks, respectively.
Empiricalresults show that LDSA learns reasonable and effective subtask assignment forbetter collaboration and significantly improves the learning performance on thechallenging StarCraft II micromanagement benchmark.
Introduction
Cooperative multi-agent reinforcement learning (MARL) has recently received much attention dueto its broad prospects on many real-world challenging problems, such as traffic light control [1],autonomous cars [2] and robot swarm control [3].
Compared to single-agent scenarios, multi-agenttasks pose more challenges.
On the one hand, the observation transition function of each agent isrelated to the policies of other agents, which are constantly updated during training.
Hence, fromthe perspective of any individual agent, the environment is extremely non-stationary, which will beexacerbated with more agents.
On the other hand, the joint action-observation space of the multi-agenttask grows exponentially with the number of agents.
These two issues prevent MARL algorithmsfrom scaling to more agents.
To cope with a large number of agents, most of the recent MARL works, including value-based [4–9]and policy gradient [10–15], utilize a technique of policy decentralization with shared parameters [16],whereby all agents share a decentralized policy or value network.
There are several merits that makeparameter sharing so popular in MARL.
First and foremost, it considerably reduces the total numberof trainable parameters and makes the learning complexity tractable.
Besides, agents with the sameparameters can share training experience with each other, which can effectively mitigate the slowdown of convergence rate due to non-stationarity [17].
However, many complex multi-agent tasks consist ofa set of subtasks, where the transition or reward functions are distinct [18, 19].
Solving every subtaskrequires some specific abilities.
In contrast, fully-shared parameters may cause all agents to behavesimilarly and hinder the diversity of agents’ policies.
For example, on the Google Research Footballtask [20], all agents will compete for the ball if sharing parameters [16].
The similar behaviors acrossall agents will limit the exploration efficiency and degrade the final performance [21].
Therefore, it’sdifficult for a single neural network to learn the specific abilities required by all subtasks.
Alternatively, another solution is to learn a separate policy for each agent without any parametersharing, which allows diverse policies but leads to high training complexity.
To balance the trainingcomplexity and the diversity of agents’ behaviors, a better method is to learn a distinct neural networkfor each subtask and group the agents by subtasks.
Agents dealing with the same subtask typicallylearn similar policies and thus can share their training experiences to accelerate training.
However, thismethod poses two new problems: (1) how to decompose a multi-agent task into subtasks and (2) howto assign agents to subtasks.
Most of the previous works [22–25] predefine the task decomposition byusing a rich prior knowledge of application domains, which may not be available in practice.
To thebest of our knowledge, only one recent work, RODE [26], learns explicit task decomposition withoutprior domain knowledge.
RODE defines the subtasks based on joint action space decompositionduring pretraining.
Specifically, RODE learns effect-based action representations to cluster actionsand then treats each cluster of actions as a subtask, where each subtask only needs to fulfill thefunctionality of a subset of actions.
But it may fail to solve subtasks when some basic actionsare necessary for all subtasks, such as the actions of moving in the StarCraft II micromanagementtasks [27].
What’s more, the effect of each action changes dynamically with the environment andthus it may be difficult to determine the actions’ effect only through pretraining.
In this work, we propose a novel framework to learn dynamic subtask assignment (LDSA) in cooperative MARL.
Our method first proposes a subtask encoder that constructs a vector representation foreach subtask according to its identity.
The action-observation history typically reflects the behavioralhabits and potential abilities of each agent, which is an important clue for selecting subtasks.
Therefore, we employ a trajectory encoding network to obtain the action-observation history of each agent.
Then, for every timestep, each agent acquires a categorical distribution of subtask selection based onthe cosine similarity of its action-observation history and representations of all subtasks, and samplesa subtask using Gumbel-Softmax [28] for training, which is a reparameterization trick allowingbackpropagate through samples.
After that, we learn a separate policy for each subtask and enableagents dealing with the same subtask to share their learning.
To associate the policy of each subtaskwith its representation, we introduce a subtask decoder to generate the policy parameters of eachsubtask based on its representation, which can also avoid similar policies between different subtasks.
Furthermore, we introduce two regularizers to increase the representation difference between subtasksand avoid agents changing subtasks frequently to stabilize training, respectively.
We evaluate our method on the challenging StarCraft II micromanagement tasks (SMAC) [27].
The results show that our LDSA significantly improves the learning performance on the SMACbenchmark compared to the baselines, especially on Hard and Super Hard scenarios.
The ablationstudies confirm the benefits of the two regularizers and the high efficiency of LDSA.
Visualizations on SMAC further reveal that LDSA effectively decomposes the task with dynamic subtask assignmentfor better collaboration.
Preliminaries
In this section, we introduce the necessary background knowledge to understand this paper.
We firstdescribe the problem formulation and the definition of subtasks for a fully cooperative multi-agenttask.
Then, we present the value function factorization methods with the centralized training withdecentralized execution (CTDE) paradigm [29, 30], which will be adopted in this paper.
2.1 Problem formulation
This work is focused on a fully cooperative multi-agent task with only partial observation for eachagent.
This task is typically modeled as a decentralized partially observable markov decision process(Dec-POMDP) [31] defined by a tuple G = ⟨ A, S, U, P, r, Z, O, γ ⟩ , where A ≡ { g 1 , g 2 , · · · , g n }denotes the finite set of agents, γ ∈ [0 , 1) is the discount factor and S is the set of global state of the environment.
At each timestep, each agent g a ∈
A selects an action u a ∈ U , forming a joint actionu ∈
U n , where a ∈ { 1 , 2 , · · · , n } is the agent identity.
This leads to a transition on the environmentaccording to the state transition function P ( s ′ | s, u ) : S × U n × S → [0 , 1] and all agents receive ashared team reward r ( s, u ) :
S × U n → R , where s ∈ S describes the global state.
Due to the partialobservability, each agent g a only obtains a partial observation o a ∈ Z according to the observationfunction O ( s, g a ) : S ×
A → Z and holds an action-observation history τ a ∈ T ≡ ( Z × U ) ∗ .
Thejoint action-observation history of all agents is denoted as τ .
To achieve scalability, a popular approach adopted by many recent MARL works [4–8] is that eachagent g a constructs a decentralized policy π a ( u a | τ a ; θ ) with shared parameters θ .
However, suchsimple parameter sharing may fail to deal with many complex multi-agent tasks that require diverseabilities among agents.
To this end, we propose to decompose a fully cooperative multi-agent taskinto subtasks.
We present the definition of subtasks in the following.
Definition 1 (Subtasks) .
For a fully cooperative multi-agent task G = ⟨ A, S, U, P, r, Z, O, γ ⟩ ,there exists a set of k subtasks, denoted as Φ ≡ { ϕ 1 , ϕ 2 , · · · , ϕ k } .
Each subtask ϕ
i is defined by a tuple ⟨ G ϕ i , x ϕ i , π ϕ i ⟩ , where i ∈ { 1 , 2 , · · · , k } is the identity of subtask, G ϕ i =
⟨ A ϕ i , S, U, P ϕ
i , r ϕ i , Z, O, γ ⟩ is the Dec-POMDP model of subtask ϕ
i , x ϕ i ∈ R m is a vector representation of subtask ϕ
i and π ϕ i : T × U → [0 , 1] is the policy of subtask ϕ
i .
A ϕ
i is the set of agentsassigned to subtask ϕ i and each agent can only select one subtask to solve at each timestep, i.e., 107 (cid:83) k i = 1 A ϕ i
[-=
-]=
A and A ϕ i (cid:84)
A ϕ j = ∅ if i ̸
= j .
The state transition function P ϕ i and reward function 108 r ϕ
i may be distinct between subtasks.
Each agent g a ∈
A ϕ i shares the policy parameters of π ϕ i .
Our objective is to learn a optimal set of subtasks Φ ∗ so as to maximize the expected global return
Q Φ tot ( τ , u ) =
E s 1: ∞ , u 1: ∞
[ (cid:80) ∞ t =0 γ t r t | s 0 = s, u 0 = u , Φ] , where r t is the team reward at timestep t .
Therefore, we are required to learn the vector representation x ϕ i , the agent assignment
A ϕ i and thepolicy
π ϕ i for each subtask ϕ
i , which will be introduced in detail in Sec. 3.
2.2 Value function factorization with CTDE paradigm
To make policy learning stable and scalable in MARL, the paradigm of CTDE [29, 30] has beenproposed and gained substantial attention.
In this paradigm, agents learn their policies togetherwith access to global information during training in a centralized way and only rely on their localobservations during execution in a decentralized way.
The CTDE paradigm has been exploited bymany recent MARL algorithms [4–8, 10, 11], among which value function factorization methods [4–8 ] have shown superior performance on challenging cooperative multi-agent tasks [27].
Thesemethods factorize the global Q-value function Q tot into individual Q-value functions Q a for eachagent g a , where Q a is only based on the local action-observation history τ a for decentralized execution.
To guarantee the consistency between greedy action selection in global and individual Q-values, suchfactorization has to satisfy Individual-Global-Maximum (IGM) principle [6] as follows:
[Equation]
In this work, we consider a representative value function factorization method, QMIX [5], whichestimates global Q-value as a non-linear combination of individual Q-values as follows:
[Equation]
where f w is a monotonic function that conditions on the global state s , i.e. , ∂f w ∂Q a ≥ 0 , ∀ a ∈{ 1 , 2 , · · · , n } .
Although Eq. 2 is only sufficient and unnecessary to IGM, QMIX is a lightweight andefficient method showing state-of-the-art performance on SMAC [27].
Method
In this section, we present the LDSA learning framework as shown in Fig.
We first introduce howto construct a set of distinct subtasks for decomposing a multi-agent task.
Next, we discuss howeach agent selects a subtask according to its abilities. After grouping agents by subtasks, we present
Trajectory Encoder 𝑥 𝜏 𝑎 𝑥 𝜙... 𝑥 𝜙𝑥 𝜙 𝑘 𝑜 𝑎 (a) (b) (c)
Subtask Encoder ℎ 𝑎 ℎ 𝑎′ 𝑜 𝑎 𝑜 𝑎 ℎ 𝑎 ℎ 𝑎′
Gumbel Softmax
Sampling ...
𝑥 𝜙 𝑘 ℎ 𝜏 𝑎 𝑄 𝑎,𝜙 𝑘
Subtask Decoder 𝜃 𝜙 𝑘
One-hot 𝑄 𝑎
MixingNetwork 𝑸 −𝑎 𝑄 𝑡𝑜𝑡
FC FC
GRU GRU 𝑥 𝜙 𝑖 𝑑 𝑖
FC
FC FC
FC 𝑥 𝜏 𝑎 . ..
Subtask 𝝓 𝟏 policy
Subtask 𝝓 𝟐 policy
Subtask 𝝓 𝒌 policy ℎ 𝜏 𝑎 the policy of each subtask that is representation-dependent.
Finally, we show the overall trainingobjective and the inference strategy.
3.1 Distinct subtask representation
Many complex multi-agent tasks involve a set of subtasks that have different responsibilities.
Forexample, running a company requires multiple departments to work together.
Previous work typicallyleverages the prior domain knowledge to decompose a complex task, which is not practical for manyuncertain environments.
To remove the dependence on prior knowledge and apply it to broadermulti-agent tasks, we propose to learn a vector representation x ϕ i for each subtask ϕ
i according to itsidentity i .
Specifically, we use a two-layer fully-connected (FC) network with a tanh activation function in
Fig.
1(a) to learn a subtask encoder f e ( · ; θ e )
: R k
→ R m , parameterized by θ e .
The subtask encodermaps the one-hot identity d
i ∈ R k of subtask ϕ
i to an m -dimensional representation space.
The tanhactivation function is to constraint the value range of the representation space.
Then, we employ thesubtask encoder to construct a vector representation x ϕ i ∈
R m for each subtask ϕ
i , i.e. ,
[Equation]
If subtasks are similar, the task decomposition makes little sense.
Therefore, to keep differencesbetween subtasks, we propose a regularizer that maximizes the L 2 distance between representationsof different subtasks as follows:
[Equation]
where D is the replay buffer.
The subtask representation learning continues throughout the trainingprocess, which can automatically adapt to the dynamic changes in the environment.
The overalloptimization objective of the subtask encoder will be introduced in Sec. 3.4.
3.2 Ability-based subtask selection
With the representation of subtasks, we need to design a subtask selection strategy for each agentbased on its ability.
In real life, we can easily infer a person’s role from the trajectory of his or herbehavior.
Similarly, the action-observation history of an agent can reflect its behavioral habits andpotential abilities.
Therefore, we utilize a shared trajectory encoder (shown in Fig. 1(b))
consisting ofa GRU [32] and two fully-connected networks to obtain the action-observation history of each agent.
The trajectory encoder f h ( · ; θ h ) , parameterized by θ h , encodes the action-observation history of each agent g a into a vector x τ a ∈ R m , which has the same length with the subtask representation.
Then,agent g a treats x τ a as its ability representation and selects a subtask to solve based on x τ a .
In our implementation, for each agent g a , we first calculate the cosine similarity of its actionobservation history representation x τ a and representations of all subtasks x Φ :=
[ x ϕ i ] ki =1 , i.e. ,similarity ( x τ a , x ϕ i ) =
( x T τ a x ϕ i ) / ( ∥ x τ a ∥∥ x ϕ i ∥ ) .
Since representations of all subtasks have thesame value range through the tanh activation function, we use x T τ a x ϕ i to approximate the cosinesimilarity for simplicity.
Then, we employ softmax function on the cosine similarity and obtain acategorical distribution of subtask selection p (Φ | x τ a , x Φ ) :
= [ p ( ϕ i | x τ a , x Φ )] ki =1 . p ( ϕ i | x τ a , x Φ ) isthe probability that agent g a selects subtask ϕ i , given by:
[Equation]
where exp ( · ) is the exponential function.
Sampling a subtask directly from the categorical distributionp (Φ | x τ a , x Φ ) is not differentiable.
To make the subtask selection process trainable, we use the
Straight-Through Gumbel-Softmax Estimator [28] to sample a subtask ϕ
j that will be discretized asa k -dimensional one-hot vector, i.e. , the one-hot subtask identity d j .
For every timestep, each agent will select a subtask to dynamically cooperate with each other.
Thisdynamic subtask assignment scheme may make training unstable when an agent frequently changesthe subtask in an episode.
To smooth the agents’ subtask selection and stabilize training, we introducea second regularizer to minimize the KL divergence between the subtask selection distributions forany two adjacent timesteps as follows:
[Equation]
where p ′ (Φ ′ | x ′ τ a , x ′ Φ ) is the subtask selection distribution at the next timestep, D KL ( ·∥· ) is the KLdivergence operator and the sum is performed across all agents.
3.3 Representation-dependent subtask policy
After grouping agents to different subtasks based on their abilities, we learn the policy for eachsubtask which is illustrated in Fig.
1(c).
Generally, agents dealing with the same subtask sharethe policy parameters and different subtasks have distinct policy parameters.
To this end, we firstuse a new shared trajectory encoder f τ ( ·| θ τ ) with parameters θ τ to generate the action-observationhistory of each agent g a , denoted as h τ a .
The new trajectory encoder f τ ( ·| θ τ ) only consists of afully-connected network and a GRU.
The policy of each subtask ϕ
i is a fully-connected networkf ϕ
i ( · ; θ ϕ i ) with parameters θ ϕ i .
Then, for each agent g a ∈
A ϕ i that solves the subtask ϕ
i , we feed itsaction-observation history
h τ a into the subtask policy f ϕ i ( · ; θ ϕ i ) and generate the individual Q-value
Q a .
To associate the policy of each subtask with its representation, we utilize a subtask decoderf d ( ·| θ d ) conditioning on the subtask representation to generate the policy parameters of each subtask.
The subtask decoder f d ( ·| θ d ) with parameters θ d is just a single-layer fully-connected network, whichmaps x ϕ i to θ ϕ
i for each subtask ϕ
i .
By virtue of the differences between subtask representations,such subtask decoder can further increase the diversity of subtask policies.
In our implementation, for each agent g a , we feed h τ a into all subtask policies to get the individual
Q-value Q a,ϕ i for every subtask ϕ
i .
The individual Q-value Q a is the sum of [ Q a,ϕ i ] ki =1 weightedby the one-hot identity of the selected subtask.
If agent g a selects subtask ϕ j , Q a is actually equalto Q a,ϕ j .
Hence, each agent only trains the policy parameters of its selected subtask.
In this way,agents with similar abilities tend to select the same subtask and thus can share their experiences toaccelerate training and improve performance.
3.4 Overall training and inference
Similar to value function factorization methods [4–8], we use a mixing network with parametersθ w to map all agents’ individual Q-values ( Q a , Q − a ) into the global Q-value Q tot , where Q a is theindividual Q-value of agent g a and Q − a are the individual Q-values of other agents.
In this work, we adopt the mixing network introduced by QMIX [5] thanks to its simple structure and superiorperformance.
It can be easily extended to other more complex mixing networks [7, 8].
Then, all theparameters θ :=
( θ e , θ h , θ τ , θ d , θ w ) of our framework can be optimized by minimizing the TD lossof Q tot as follows:
[Equation]
where θ − are the parameters of the target network that are periodically copied from θ .
Consideringthe two regularizes in Eq.
4 and Eq.
6, the overall optimization objective of LDSA is:
[Equation]
where λ ϕ and λ h are positive coefficients of the two regularizers, respectively.
During the inferencephase ( i.e. , test the decentralized policy), each agent g a selects the subtask with the maximumprobability on the subtask selection distribution, i.e. , arg max ϕ i p ( ϕ i | x τ a , x Φ ) and then chooses agreedy action according to the individual Q-value Q a for decentralized execution.
Experiments
In this section, we conduct several experiments to investigate the following questions: (1)
Can
LDSA improve the performance compared to the baselines?
(Sec. 4.2) (2) How do the two proposedregularizers influence the performance?
(Sec. 4.3) (3) Whether the superior performance of ourmethod comes from the increase in the number of parameters?
If not, which component contributesthe most to our method?
(Sec. 4.3) (4) Can LDSA learn dynamic subtask assignment and groupagents reasonably?
If so, agents with similar abilities solve the same subtask and each subtask hasspecific responsibility.
(Sec. 4.4)
4.1 Experimental setup
Environment We evaluate LDSA on the SMAC benchmark [27], a challenging benchmark forcooperative MARL.
There are two armies of units in the SMAC environment.
Each ally unit iscontrolled by a decentralized agent that can only act based on its local observation and the enemyunits are controlled by built-in handcrafted heuristic rules.
The goal of the MARL algorithm is tomaximize the test win rate for each battle scenario.
In this paper, we adopt the default environmentsettings for SMAC.
The version used in this work is SC2.4.10.
We consider all 14 scenarios on the
SMAC benchmark that can be classified into three different levels of difficulties: Easy , Hard and
Super Hard scenarios.
Baselines We select QMIX [5], ROMA [33] and RODE [26] as our baselines.
All of the baselinesand LDSA belong to value function factorization methods and use the QMIX-style mixing network [5]for a fair comparison.
Therefore, LDSA differs from the baselines only by the individual Q-valuenetworks.
QMIX is a natural baseline in which all agents share the individual Q-value network.
ROMA and RODE are two recent works that learn implicit and explicit task decomposition in MARL,respectively.
We implement all the baselines using their open-source codes based on PyMARL [27].
Note that, as mentioned in Sec. 1, RODE may not work when some basic actions are necessary for allsubtasks.
In the SMAC environment, there are four actions representing the agents’ movement infour cardinal directions, which are important for all subtasks.
To be effective, RODE uses severalrules based on prior knowledge of the environment in its implementation, which manually enforcethe four actions of moving be available to some subtasks forever under certain conditions.
Therefore,to not involve prior knowledge, we remove these rules when we implement RODE.
Hyperparameters For all experiments, the number of subtasks is set to 4 and the length of subtaskrepresentations is set to 64, i.e. , k = 4, m = 64.
We carry out a grid search for regularizers’coefficients λ ϕ and λ h on the SMAC scenario corridor and then set them to 10 − 3 and 10 − 3 ,respectively, for all scenarios.
All the common hyperparameters of our method and baselines areset to be the same as that in the default implementation of PyMARL.
We use lightweight network structures for the subtask encoder, subtask decoder, trajectory encoder and subtask policy.
Thedetailed hyperparameters and network structures will be provided in Appendix A.1.
4.2 Performance on SMAC
We compare LDSA with the baselines across all 14 scenarios on the SMAC benchmark [27].
Forevery scenario, we carry out 5 independent runs with 5 different random seeds for all methods andshow the median performance and 25-75% percentiles.
Fig.
3 presents the averaged median testwin rate across all 14 scenarios.
We can observe that LDSA significantly improves the learningperformance and surpasses about 7% median test win rate averaged across all 14 scenarios.
[Figure]
Fig.
2 shows the learning curves of LDSA and the baselines on six Hard or Super Hard scenarios.
The resultson the other eight scenarios are shown in Appendix A.2.
LDSA outperforms all the baselines on all Super Hardand Hard scenarios with faster convergence.
Specifically,on the scenarios that require diverse micromanagementtechniques: corridor , 3s5z_vs_3s6z and 6h_vs_8z ,the test win rate of LDSA exceeds that of the baselines atleast 15%.
On the Easy scenarios, LDSA shows similarperformance as QMIX, which indicates that our methodmay not improve the learning performance on simple tasksnot requiring task decomposition.
Moreover, ROMA thatlearns implicit task decomposition performs even worsethan QMIX on most of the scenarios.
Without using rulesbased on prior knowledge, RODE fails to learn efficientpolicies for subtasks, which demonstrates that discovering subtasks based on joint action spacedecomposition and restricting the action space of subtasks may severely limit the learning of subtask policies.
These results further confirm that LDSA can learn effective task decomposition withdynamic subtask assignment to solve complex tasks.
4.3 Ablation studies
In this subsection, we conduct ablation studies on three scenarios: corridor , 5m_vs_6m and2c_vs_64zg , to show the effect of the two regularizers and test which component contributes most to
LDSA.
Fig.
4 reports the performance of LDSA when we remove each or both of the two regularizers
L h and L ϕ .
It can be observed that the performance of LDSA without both of the two regularizersis even worse than that of QMIX.
Either of the two regularizers can improve performance and L ϕimproves more than L h , which indicates that keeping differences between subtasks is more importantwhile L h can stabilize training.
To figure out why LDSA performs better than QMIX, we investigate the contributions of threecomponents of LDSA: the increase in the number of parameters, the ability-based subtask selectionand the subtask decoder.
As shown in Fig.
5, QMIX with similar parameters as LDSA can’t improveperformance and even may make training slower and degrade performance, which proves thatthe superior performance of LDSA is not due to the increase in the number of parameters.
Theperformance of LDSA without the subtask decoder is lower than that of LDSA, which demonstratesthat associating subtask policy with subtask representation benefits the performance.
What’s more,the large margin between the performance of LDSA with random subtask selection and that of LDSAreveals that the ability-based subtask selection contributes the most to our method.
In summary, thesuperior performance of LDSA is largely due to the efficient subtask assignment.
We also study theeffect of the number of subtasks in Appendix A.3.
4.4 Visualization of dynamic subtask assignment
In this subsection, we visualize the dynamic subtask assignment in one episode on the SMAC scenariocorridor as shown in Fig.
The scenario corridor consists of 6 ally Zealots and 24 enemy
Zerglings.
Due to the huge disparity in the number of allies and enemies, it’s hard to win if all ally
Zealots rush to attack the enemy.
Therefore, at the beginning ( t = 3), our method learns to sacrificetwo allies (agents g 1 and g 4 ) to attract most enemies and then the other 4 allies (agents g 2 , g 3 , g 5 andg 6 ) can focus fire to kill a small part of enemies.
After that, the remaining enemy Zerglings controlledby heuristic rules will go through a narrow corridor, and stay at the bottom left corner of the scenarioif there are no Zealots within the sight range.
In the middle of the episode ( t = 64), the remainingenemy forces are still strong, our method assigns the healthiest ally (agent g 2 ) to draw out a smallnumber of enemies and kill them with the other allies (agents g 3 , g 5 and g 6 ).
This process will repeatuntil all enemies are killed.
Moreover, we find that our method will assign the dead allies (agents g 1and g 4 ) to a fixed subtask (subtask ϕ 3 ), which can prevent them from interfering with the learning ofother subtasks.
These visualizations demonstrate the effectiveness and rationality of the dynamicsubtask assignment in our method. [Figure]
Related work
Value-based MARL Value-based MARL algorithms have achieved great progress in recent years.
IQL [34] simply trains an independent Q-value network for each agent, which treats the otheragents as part of the environment. This method may not converge due to the non-stationarity of theenvironment caused by the changing policies of other agents. The other extreme method is to treatall agents as a single agent and learn a global Q-value based on the joint action-observation space,which alleviates the non-stationarity but suffers from the scalability challenge, as the joint actionobservation space grows exponentially with the number of agents. To trade off these two methods,most of the value-based MARL algorithms factorize the global Q-value into individual Q-valuesfor centralized training and decentralized execution.
VDN [4] and QMIX [5] factorize the global Q-value by additivity and monotonicity, respectively.
QTRAN [6] transforms the original global Q-value function into an easily factorizable one to expand the representation capacity. QPLEX [8]proposes a duplex dueling network architecture to implement the complete IGM [6] function class. These methods focus on designing the mixing network.
There are other value-based works studying MARL from the perspective of communication [35, 36], exploration [37, 38] and robustness [39, 40]. Task decomposition in MARL Task decomposition plays an important role in many complexreal-world multi-agent systems, such as software engineering [41], healthcare [42] and traffic management [43]. Once the multi-agent task is decomposed, agents can be assigned to the restrictedsubtasks that are easier to solve, and thus the learning complexity is greatly reduced.
However, it ischallenging to come up with a set of subtasks that can effectively decompose the whole multi-agenttask. The most straightforward way is to predefine the subtasks by leveraging the prior domainknowledge [22–25]. But the prior knowledge may not be available in many uncertain environments. To solve this problem, ROMA [33] introduces the concept of roles for each agent based on its localobservation and conditions agents’ policies on their roles, which achieves implicit task decomposition.
RODE [26] explicitly defines the subtasks based on joint action space decomposition duringpretraining, where each subtask corresponds to a subset of actions. In this work, we learn vectorrepresentations for subtasks that can automatically adapt to the environment during training.
Conclusion
Task decomposition is an important approach to simplify complex multi-agent tasks and has notbeen well solved without using prior knowledge.
To this end, we propose to decompose the task intoseveral subtasks represented by latent embeddings.
Agents select subtasks according to their abilities,which are indicated by their action-observation histories.
In this way, agents dealing with the samesubtask can share their learning to solve the subtask, which can learn the specific abilities required byall subtasks under a tractable training complexity.
Although the embedding representation for eachsubtask may be abstract, it essentially clusters agents with similar abilities into the same group andthus agents can dynamically share their experiences to accelerate training and improve performance.
The empirical results on the SMAC benchmark further demonstrate that LDSA learns effective taskdecomposition with dynamic subtask assignment and significantly improves the learning performancecompared to the baselines.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] In Sec. 4.2, we mention that ourmethod may not improve the learning performance on simple tasks not requiring
taskdecomposition
c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our method is implemented on the StarCraft II
micromanagement tasks and we haven’t evaluated our
method
on real
applications So we are not clear about the potential negative societal impacts of our work.
d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
a)
Did you include the code data and instructions needed to reproduce the main experimental results (
either in the supplemental material or as a URL)
[Yes We include
thecode , data, and instructions needed to reproduce the main experimental results in the
supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Sec.
4.1 and Appendix A.1.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We report the median performance as well as the 25-75%percentiles across 5 random seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix A.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We implement our
method based on the open-sourced PyMARL framework, which has been cited in Sec. 4.1.
(b) Did you mention the license of the assets?
[Yes] PyMARL is licensed under the ApacheLicense 2.0.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
[Table]
[Table]
A AppendixA.1 Architecture and hyperparameters
In this paper, we use lightweight network structures for the subtask encoder, subtask decoder,trajectory encoder and subtask policy.
Specifically, the subtask encoder is a two-layer 64-dimensionalfully-connected network and the output layer is followed by the tanh activation.
Agents sharea trajectory encoder to construct their ability representations.
The trajectory encoder consists ofthree layers, a 64-dimensional fully-connected layer, followed by a 64-bit GRU, and followed byanother 64-dimensional fully-connected layer.
The new shared trajectory encoder only consists ofa 64-dimensional fully-connected layer followed by a 64-bit GRU and the outputs will be fed intothe subtask policy.
The policy of each subtask is a fully-connected layer of n _ actions dimensions,where n _ actions is the number of actions.
The parameters of the subtask policy are generated by thesubtask encoder, which is also a fully-connected layer.
We use the QMIX-style mixing network withthe same setting as the original paper.
For all methods, we run 5 million timesteps on three Super Hard scenarios: corridor ,3s5z_vs_3s6z , 6h_vs_8z and 2 million timesteps on the other 11 scenarios.
The replay buffer isa first-in-first-out queue with a max size of 5000 episodes.
For every one episode data collected,we sample a batch of 32 episodes from the replay buffer for training.
The decentralized policy isevaluated every 10 k timesteps with 32 episodes.
We use ϵ -greedy exploration.
The ϵ anneals linearlyfrom 1.0 to 0.05 over 50 k timesteps and keeps constant for the rest of the learning.
The trainingobjective is optimized by RMSprop with a learning rate of 5 × 10 − 4 .
All experiments are conductedon GeForce RTX 2080Ti GPU.
A.2 Remaining results on the SMAC benchmark
[Figure]
baselines on the remaining SMAC scenarios, including five Easy scenarios: 2s_vs_1sc , 2s3z , 3s5z , 1c3s5z , 10m_vs_11m and one Hard scenario: bane_vs_bane and two Super Hard scenarios: 27m_vs_30m , MMM2 .
A.3 Effect of the number of subtasks
The number of subtasks k is a core hyperparameter of LDSA.
To study the effect of k , we test LDSAwith different values of k on the SMAC scenario corridor as shown in Fig.
It can be observedthat LDSA can obtain a better performance than QMIX when k = 2 and increasing the number ofsubtasks benefits the performance.
But LDSA with more than 4 subtasks shows little improvementand learning more subtasks will lead to higher computational cost.
Therefore, we set k = 4 throughoutthe work to balance the effectiveness and efficiency.
[Figure]
FRUULGRU N  N
 N  N
