ALMA: Hierarchical Learning for Composite Multi-Agent Tasks
Anonymous Author(s)
AffiliationAddress email
Abstract
Despite significant progress on multi-agent reinforcement learning (MARL) inrecent years, coordination in complex domains remains a challenge.
Work in
MARL often focuses on solving tasks where agents interact with all other agentsand entities in the environment; however, we observe that real-world tasks areoften composed of several isolated instances of local agent interactions (subtasks),and each agent can meaningfully focus on one subtask to the exclusion of all elsein the environment.
In these composite tasks , successful policies can often bedecomposed into two levels of decision-making: agents are allocated to specificsubtasks and each agent acts productively towards their assigned subtask alone.
This decomposed decision making provides a strong structural inductive bias,significantly reduces agent observation spaces, and encourages subtask-specificpolicies to be reused and composed during training, as opposed to treating eachnew composition of subtasks as unique.
We introduce ALMA, a general learningmethod for taking advantage of these structured tasks.
ALMA simultaneouslylearns a high-level subtask allocation policy and low-level agent policies.
Wedemonstrate that ALMA learns sophisticated coordination behavior in a number ofchallenging environments, outperforming strong baselines.
ALMA’s modularityalso enables it to better generalize to new environment configurations.
Finally,we find that while ALMA can integrate separately trained allocation and actionpolicies, the best performance is obtained only by training all components jointly.
Introduction
A certain set of real-world multi-agent tasks increase in complexity in proportion to the inter-agentcoordination required rather than any inherent difficulty in the low-level skills.
Take the exampleof firefighting (Figure 1): while coordinating firefighters across a large city is more complex thanin a small one, the individual sub-task of putting out a fire remains similar across cities of all sizes.
Despite the recent success of cooperative multi-agent reinforcement learning (MARL) methods on awide range of domains [2, 16, 35, 5], they still struggle to learn sophisticated coordination behaviorin a sample-efficient manner on tasks of real-world complexity.
And this is no surprise—not only dothese tasks require that agents learn complex low-level skills to execute sub-tasks, they must alsoincorporate coherent high-level strategies into their policies.
Integrating both strategies and skills intothe same action-level policy makes learning difficult.
Fortunately, for many relevant multi-agent tasks, while there may exist many different objectivesthat the agent population as a whole must attend to, individual agents need only focus on isolatedaspects of the environment at any given time.
Consider again the setting of firefighting in a city givena distributed set of resources.
While there may be many fires occurring in the city at once, eachfirefighter can realistically only fight one at a time.
Thus, the optimal behavior in this setting can be expressed in terms of allocating firefighters to the right fires, and each firefighter effectively fightingthe fire to which they are assigned.
Suitable policies for this and many other realworld problems can be formulated as bi-leveldecision-making processes [20]; agents are allocated to the most relevant subtasks in the environment, then each agent selects actions with thepurpose of completing its assigned subtask.
Wecall settings that can be formulated in this way“composite tasks”.
Decomposing these tasksallows for more efficient learning for a number of reasons.
First, the learning process maybenefit from the added structural inductive bias,alleviating the problem of incorporating bothskills and strategy into the same low-level policy.
Second, by focusing solely on its assignedsubtask, and ignoring the rest of the environment, agent action-level policies can learn overa significantly reduced state-space.
Finally, as we assume subtasks are drawn from the same distribution, learning policies for individual subtasks and allowing the high-level controller to compose themmay be simpler than learning a single policy over all compositions of these subtasks.
Our contributions are as follows.
First, we introduce a general learning method, AL locator-Actor
M ulti-Agent
A rchitecture ( ALMA ), for taking advantage of composite multi-agent tasks (§4).
ALMA learns a subtask allocation policy and low-level agent policies simultaneously and is designedin a modular manner to promote the reuse of components across different compositions of subtasks.
Learning the high-level controller poses challenges due to its large action space—at any time step,the number of unique allocations of n agents to m subtasks is m n .
Our solution leverages recentmethods [31] in learning value functions over massive action spaces and is designed modularly tohandle variable quantities of agents and subtasks drawn from a shared distribution.
While action-valuefunction typically depend on the global state due to shared transition dynamics and reward functionsthat depend on all agents, we observe that for composite tasks we can decompose the environment intoindependent sub-environments and redefine subtask-specific action-value functions to only depend onlocal information, improving their reusability and sample-efficiency.
We next evaluate our proposedmethod on two challenging multi-agent domains (§5).
Our complete method outperforms both state-ofthe-art hierarchical and standard MARL methods as well as strong allocation heuristics.
Furthermore,we evaluate the practicality of our subtask action-value function decomposition, demonstrating thatlearning fails without it while highlighting cases in which the required assumptions do not hold.
Lastly, we show the importance of learning action policies in concert with allocation policies.
Related Work
Multi-Agent Task Allocation
Task allocation is a long-studied problem in multi-agent systemsand robotics [26, 9, 12, 8].
Given a set of tasks and a set of agents, the goal in this setting is to assignagents to tasks in a manner that maximizes total utility.
A utility function which maps agent team-taskpairs to their utility is assumed to be given.
Within the taxonomy of sub-classes to this problemformalized by Gerkey and Matari´c [9], our problem setting can be classified as ST-MR-IA (single-taskagents, multi-agent tasks, instantaneous allocation).
In other words, agents cannot multi-task, tasksrequire the cooperation of several agents, and they are assigned instantaneously without incorporatingany information regarding future task availability.
We note that work in multi-agent task allocationoften assumes the execution of tasks to be trivial; the focus is placed entirely on how to assign agentsto a set of tasks in an efficient manner, and some knowledge of agents’ effectiveness in each taskis assumed.
In our reinforcement learning setting, we only assume the ability to execute actionsin an environment and observe global and subtask-specific rewards.
Our focus is on enabling theapplication of MARL to complex composite tasks, rather than learning task allocation policies thatcompete with traditional approaches.
MDP Formulations for Multi-Agent Task Allocation
Several works have proposed formulationsof task allocation as an MDP, enabling the use of learning and planning-based approaches.
Proper and
Tadepalli [20] introduce the decomposition of the simultaneous task setting into task allocation and task execution levels.
However, their allocation selection procedure does not consider how allocationsmay change in the future.
Campbell et al.
[6] address this shortcoming by taking future allocationsinto account when allocating agents to subtasks; however, they assume the pre-existence of low-levelaction policies.
Notably, neither approach learns a tractable policy over the allocation space, andinstead relies on exhaustive search or hand-crafted heuristics in order to select allocations.
Multi-Agent Reinforcement Learning Several recent works in MARL have addressed relatedproblems to ours.
Most relevantly, Carion et al.
[7] introduce a learning-based approach for taskallocation that scales to complex tasks.
Unlike our proposed setting, their work assumes pre-existingsubtask execution policies, as well as domain knowledge regarding how much an agent “contributes”to subtasks, which are used as constraints in a linear/quadratic program to prevent assigning moreagents than necessary to subtasks.
Our method, on the other hand, learns low-level policies, assumesless prior knowledge, and solves the problem of finding the best allocation by learning an allocationcontroller from experience rather than using an off-the-shelf solver with inputs generated by a learnedfunction and constraints from domain knowledge.
Shu and Tian [25] devise a method whereby a central “manager” agent provides incentives to selfinterested “worker” agents to perform specific subtasks such that the manager’s reward is maximizedand incentives provided to workers are minimized.
In this case, agents work on subtasks independentlyand are not cooperating towards a shared goal.
Yang et al.
[37] also consider a setting where multiplesimultaneous tasks (described as goals ) exist in a shared environment; however, they assume thatgoals are pre-assigned to agents and therefore do not tackle the task allocation problem.
Liu et al.[15] introduce COPA, a hierarchical MARL method which learns a centralized controller in order tocoordinate decentralized agents.
Unlike our method, which utilizes task allocation for the purposeof decomposing complex settings to simplify learning, COPA’s motivation is primarily to alleviatethe problem of partial observability during decentralized execution.
We evaluate our method against
COPA in order to assess the effectiveness of our hierarchical decomposition in comparison to a moregeneric hierarchical MARL method with similar assumptions (i.e. no pre-trained policies, cooperativetasks, and no pre-existing task allocation).
Preliminaries
Task Framework Our setting builds on the general framework of Decentralized POMDPs [19] withentities [23, 10] by incorporating structure for multiple coexisting subtasks.
Note that we differentiatebetween subtasks and tasks.
We refer to independent jobs within an environment as “subtasks” andthe global collection of them (specified by the Dec-POMDP) as a task.
This framework describes fullycooperative problems which can be specified by the tuple: ( S , U , O , P , r , ( r i , E i | i ∈ I ) , A , µ, I ) .
A is the set of agent entities.
Each entity e has a state s e .
We denote, for example, the statespace of all agents as S A where { s a | a ∈ A} ∈ S A .
The state space of other sets of entities is 127 denoted similarly.
Each agent a can execute actions u a , and the joint action of all agents is denotedu = { u a | a ∈ A} ∈ U .
Within the environment exist a set of subtasks I where each is defined 129 by a set of subtask-specific entities
E i and the subtask reward function r i : S E i × S
A × U → R .
The relevant state (i.e. set of entity states) for subtask i is then denoted as s i
= { s e | e ∈ E
i ∪ A} .
We define
E as the set of all entities (including agents) in the environment: E := (cid:0)(cid:83) i ∈I
E i (cid:1) ∪ A .
The global state is the set s :=
{ s e | e ∈ E} ∈ S .
Not all entities may be visible to each agent, so 133 we define a binary observability mask: µ ( s a , s e ) ∈ { 1 , 0 } .
Thus, an agent’s observation is defined 134 as o a = { s e | µ ( s a , s e ) = 1 , e ∈ E} ∈ O .
P is the state transition distribution which defines the 135 probability P ( s ′
| s , u ) .
Finally, we define the global reward function r : S × U → R .
In the simplestcase, the global reward function can be a sum of the individual subtask rewards; however, it canoptionally encode global objectives (e.g. a scalar reward upon the completion of all subtasks).
Cooperative MARL via Value Function Factorization Cooperative MARL methods are concerned with learning a set of policies which maximize the expected discounted sum of global rewards.
Value function based methods do this by learning an approximation of the optimal Q -function:
[Equation]
An optimal policy can be derived from this Q -function by taking the maximum valued action ufrom the current state; however, in the multi-agent setting it is often desirable for each agent toexecute their policy independently (i.e. in a decentralized fashion).
In order to derive decentralized policies, the Q -function is represented in a manner that realizes the Individual-Global-Max (IGM)principle [27] which states that each agent can greedily maximize their local utility Q a and maximize
Q tot as a result.
This has been accomplished through representing Q as a summation of utilities [28],a monotonically increasing combination [21] of them, as well as several works which attempt toextend the representational capacity of the global Q -function to all functions satisfying the IGMprinciple [27, 36].
The value function approximation is learned by training a neural network tominimize the following loss derived from the Bellman equation:
[Equation]
(1) where ¯ θ are the parameters of a target network that is copied from θ periodically to improve stability [17] and D is a replay buffer [14] that stores transitions collected by an exploratory policy(typically ϵ -greedy).
In cases of partial observability, the history of local actions and observationsτ at = ( o a 1 , u a 1 , . . .
, o at ) is used as a proxy for the state s t as input to the agents’ utility functions Q a .
Hierarchical MARL Value-based single-agent Hierarchial RL methods [e.g. 13] typically learn atemporally abstracted high-level Q -function which selects goals upon which a low-level controller isconditioned.
In adapting these methods for the multi-agent setting, one can choose to select goalsin a fully decentralized fashion [29] or through a centralized controller [15, 38, 1].
Our work willconsider the latter setting, where a single controller learns a value function over a shared goal spaceb ∈ B :
[Equation]
This controller operates over a dilated time scale; one step from its perspective amounts to N t stepsin the environment executed by low-level controllers conditioned on the goal, π b .
The low-levelcontrollers are trained to maximize goal-specific rewards which can be manually defined or learned.
Allocator-Actor Multi-Agent Architecture
In the last section we introduced a generic framework for value-based hierarchical MARL.
Nowwe will present the details specific to our hierarchical MARL framework for composite tasks usingsubtask allocation, ALMA .
In this case we define the action space of the high-level controller (whosevalue function is defined in Eqn. 2) as the set of all possible allocations of agents to subtasks.
Asubtask allocation refers to a specific set of agent-subtask assignments, where each agent can onlybe assigned to one subtask at a time, and it is denoted by b = { b a | a ∈ A} ∈ B , where b a ∈ I 171 represents the subtask that agent a is assigned to.
We then slightly abuse notation and denote the setof agents assigned to subtask i as b i ⊆ A and the set of all agents not assigned to task i as b \ i whereb i ∪ b \
i =
A .
We then specify the joint action for agents assigned to subtask i as u b i = { u a | a ∈ b i } .
174 Recall that the subtask-specific state s
i includes the state of all agents.
We denote the subtask-specificstate including only the assigned agents as s b i = { s e | e ∈ E i ∪ b i } .
4.1 Subtask Allocation Controllers
One of the main challenges of learning a Q -function for the high-level allocation controller is themassive action space.
Our formulation of subtask allocation can be seen as a set partitioning problem,which is known to be NP-Hard [24, 9].
Q -Learning with the allocation action space requires finding the allocation with the highest Q -value at each step (Eqn. 2), as does deriving an action policy fromthis Q -function.
At each step, we have a choice of |I| |A| possible unique allocations of agents tosubtasks, and it is prohibitively expensive to evaluate each one and select the best.
Van de Wieleet al.
[31] introduce “Amortized Q -Learning” which addresses the problem of massive action spacesin Deep Q -Learning by defining a “proposal distribution” which they train to maximize the densityof actions with high values given a state.
We adapt this idea for our allocation controller, wheref ( b | s ; ϕ ) is our proposal distribution over allocations.
We can then sample from this distribution andselect the allocation with the highest value, effectively approximating the maximization procedurerequired for Q -Learning.
Our proposal distribution is learned with the following loss:
[Equation]
where b ∗ ( s ) is the highest-valued allocation from a set of N p samples from the proposal distribution.
Formally, b ∗ ( s ) := arg max b ∈ B samp ( s ) Q ( s , b ) where B samp ( s ) :=
{ b 1 , . . .
, b N p ∼ f ( ·| s ; ϕ ) } .
We then learn an approximation of Eq.
2 with the following loss, adapted from Eqn. 1:
[Equation]
where transitions ( s t , b t , (cid:80) N t n r t + n , s t + N t ) are sampled from a replay buffer.
Learning a proposaldistribution and value function over a combinatorial space closely parallels recent work in learnedcombinatorial optimization [3, 11, 4].
We construct our proposal distribution modularly so that each module can leverage the fact thatsubtasks are drawn from a shared distribution.
These modules consist of an agent embedding functionf h : S a → R d , a subtask embedding function f g : S E i → R d , and a subtask embedding updatefunction f u : R 2 d → R d which is used to update the embedding of the subtask that an agent wasassigned to.
We first construct f in a factorized auto-regressive form: f ( b | s ) =
(cid:81) a ∈A f ( b a | s , b <a ) ,where b <a are the allocations made prior to assigning agent a .
Thus, we are iteratively assigningagents to subtasks while accounting for allocations already determined.
Since the number of subtasks present at any time is variable (e.g. some subtasks may be finishedbefore others), we construct each auto-regressive factor as a composition of outputs generated bythe modules described above, in a fashion similar to pointer-networks [34].
Pointer networks arecommonly used in RL applications to handle variable-sized action spaces where the individual actionscorrespond to entities/groups [35].
We first generate a set of subtask embeddings
g i
=
f g ( s E i )encoding the state of the non-agent entities belonging to each subtask, as well as embeddingsdescribing each agent’s state h
a = f h ( s a ) .
Logits are computed as the dot-product between agentembeddings and subtask embeddings: f
( b a | s , b <a ) =
softmax i ∈I (cid:0)
g ⊤ i h a (cid:1) .
We then sample b a ∼ f ,and update the selected subtask’s embedding as g ′ b a = g b a + f u ( g b a , h a ) such that other agents’allocations can take into account existing ones.
4.2 Subtask Execution Controllers
Given a subtask allocation b , we must learn low-level action-value functions as described in Section 3.
Specifically, for each team b i we learn a value function based on the subtask rewards r i .
Recall thatthe subtask reward function depends on all agents’ actions and states since any agent can potentiallycontribute to any subtask.
From the perspective of the team assigned to subtask i , we treat the othersubtasks’ agents as part of the environment by taking the expectation over actions sampled from theiroptimal policies:
[Equation]
While this function depends on the full global state since all entities can feasibly influence all otherentities’ state transitions and any agent can potentially contribute to the rewards of any subtask,in composite tasks the set of truly relevant information may be much smaller for optimal policies.
Reconsider the example from Sec.
1 where firefighters are agents and buildings are subtasks.
While itis possible for any firefighter to put out any fire, given optimal policies we expect firefighters to onlyput out the fire they are assigned to.
Moreover, in the optimal setting, firefighters will not be directlyinteracting with firefighters at other buildings.
As such, for composite tasks, we can decompose theenvironment into a set of independent sub-environments and rewrite Eqn.
5 as follows:
[Equation]
where P b i is the transition distribution of sub-environment i and the transitions of the full environmentcan be written in the following factored form: P ( s ′
[Equation]
s ,
[Equation]
Given subtask allocation b , S ∗ b is the set of global states visited by the optimal subtask policies, π ∗ b i .
Finally, r b i is the sub-environment reward function that depends only on local information defined as:
[Equation]
Note that the Q -function now only depends on a “local” state, and, as such, can be approximatedmore easily due to the smaller function input space.
Moreover, in the case where subtasks are drawnfrom the same distribution, Q -functions defined over the local state will more easily generalize toother instantiations of the same subtask since they no longer depend on the context outside of thatsubtask’s state.
We validate the usefulness of this decomposition empirically in Section 5.
[Figure]
Learning a function that approximates Eqn. 6 requires predicting Q -values for teams of varyingsizes, as the quantity of agents assigned to a subtask is not fixed.
In fact, each unique combinationof agents assigned to subtask i can be seen as a unique Dec-POMDP.
We rely on recent work [10]which learns factorized multi-task Q -functions for teams of varying sizes by sharing parameters viaattention mechanisms.
This work falls under the category of factorized value function methods forcooperative MARL described in §3, and as such, we represent each subtask-specific Q -function as a monotonic mixture of agent utility functions computed in a decentralized fashion, such that agents can act independently without communication.
4.3 Training and Execution Details
We provide an overview of the complete subtask allocation and execution procedure in Figure 2.
Subtask allocations b are selected in a centralized fashion every N t steps which then determinesthe set of low level policies to execute.
We define the high-level policy Π( b | s ) :
= 1 ( b =
b ∗ ( s ))where b ∗ ( s ) is the highest valued allocation sampled from the proposal distribution, as definedin Section 4.1.
The per-agent low-level policy for agent a assigned to subtask i is defined asπ a ( u a | s b i ; b ) :
= 1 ( u a = arg max u a ′ Q
ai ( s b i , u a ′ ; b )) where Q ai is the agent’s utility functionwhich is monotonically mixed with all a ∈ b i to form Q tot i .
When collecting data during training, we select random low-level actions with probability ϵ in orderto promote exploration.
We use two types of exploration in the allocation controller.
With probabilityϵ p we select a full allocation sampled from the proposal distribution, rather than taking the highestvalued one.
Then, with probability ϵ r , we randomly select subtask allocations independently ona per-agent basis.
All exploration probabilities are annealed over the course of training, and theannealing schedules (along with all hyperparameters) are provided in Appendix D.
To handle state spaces of variable size (i.e. variable quantity of entities), we use attention models [33]as in [10] for all components.
Network architectures are provided in Appendix A.
In order to achievethe partial views over the global state required by our redefined low-level Q -function in Eqn. 6, weuse masking in attention models to prevent agents from seeing certain entities.
Experiments
We evaluate ALMA on two challenging environments, described below (more in Appendix B).
5.1 Environments
S AVE T HE C ITY
Inspired by the classical Dec-POMDP task “Factored Firefighting” [18], and basedon the scenario described in Sec.
1, we introduce S AVE T HE C ITY , an environment where agents mustcoordinate to extinguish fires and rebuild a burning city (see Fig. 3).
In this version, agents are capableof contributing to any subtask, must physically navigate between buildings through low-level actions,and there are different agent types each with distinct abilities that are crucial for success.
The subtaskallocation function must discover these abilities from experience and use this knowledge to distributeagents effectively.
The task also shares similarities with the Search-and-Rescue task from [15], butwe do not consider partial observability and we introduce agents with diverse capabilities.
S TAR C RAFT
Our next environment is the StarCraft multi-agent challenge (SMAC)
[22] whichinvolves a set of agents engaging in a battle with enemy units in the StarCraft II video game.
Wespecifically consider the multi-task setting presented by Iqbal et al.
[10] where each task presentsa unique combination and quantity of agent and enemy types.
As such, agents cannot learn fixed
Step × 10 7 0 .0 .0 .0 .0 .
S u cce ss
R a t e
Method ALMA
NomaskHeur.(dyn.)
Heur.
(match)
COPAFlat 0 .0 .0 .
Step × 10 7 0 .0 .0 .0 .0 .0 .0 .0 .0 .1 . 
 Step × 10 7 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .1 .
Step × 10 7 0 .0 .0 .0 .0 .
strategies, and must instead adapt strategies to their set of teammates and enemies.
While the originalsetting consists of a single enemy army attacking the agents, inspired by the full game of StarCraftwe increase the complexity by introducing multiple enemy armies (see Fig. 5) which periodicallyattack and retreat independently and the additional objective of defending a base.
By doing do, we areessentially composing several tasks of similar complexity to those considered by the current literatureand requiring agents to learn not only how to solve those tasks but also how to distribute agents acrosstasks in a manner that is globally optimal.
Within this setting, we consider four different variationsof army compositions based on those presented in [10].
In one set of settings we give agents andenemies a symmetric set of units, while in the other, agents have one fewer unit than the enemies.
Within each set we consider two different pools of unit types: “Stalkers and Zealots” (S&Z) and“Marines, Marauders, and Medivacs” (MMM).
The former includes a mixture of melee and rangedunits, while the latter includes units that are capable of healing.
5.2 Results
[Figure]
Numbers are
[Figure]
now, but when you export they become centered (weird interactions between images and vectors)
[Figure]
In our experimental validation we aim to answer the following questions: 1) Learning Efficacy: Is ALMA effective in improving learning efficiency and asymptotic performance incomparison to state-of-the-art hierarchical andnon-hierarchical MARL methods? 2) Allocation
Strategic Complexity : Are the learned allocation policies non-trivial or are any benefits of
ALMA purely gained from the subtask decomposition?
3) Decomposition Validity : What, ifanything, do we gain from the sub-environmentand how does ALMA fare when the requiredassumptions are broken?
4) Joint Training : Dowe receive any benefits from training allocationand execution controllers jointly?
Learning Efficacy First, we aim to evaluate the effectiveness of our hierarchical abstraction incomparison to state-of-the-art cooperative MARL methods in variable entity multi-task settings.
Wecompare to two non-hierarchical multi-agent baselines: QMIX [21] augmented with self-attention[33], to extract information from variable quantities of entities (A-QMIX), and REFIL [10], a stateof-the-art approach built on top of A-QMIX for generalizing across tasks with different compositionsof agent and entity types.
Both methods are referred to as “Flat” in our figures and differ dependingon the environment.
REFIL is used for S TAR C RAFT tasks, but we find that it does not significantlyimprove on A-QMIX in S AVE T HE C ITY , so we use A-QMIX in that setting.
Evaluating these methodson our tasks allows us to isolate the effectiveness of our hierarchical decomposition, as we use thesame algorithms to train low-level controllers in ALMA.
Next, we compare to a state-of-the-art approach in hierarchical MARL for varying entity settings:
COPA [15].
COPA makes the assumption, similar to ALMA , of a centralized agent that is able tocommunicate with decentralized agents periodically.
We ensure that the communication frequency of
COPA matches that of our method and that it is provided subtask labels for entities, so it receives asimilar amount of information, even though it does not explicitly leverage the subtask decomposition.
In both S AVE T HE C ITY and S TAR C RAFT , we find that ALMA is able to outperform all baselines inmost settings.
Both A-QMIX (Flat) and COPA are unable to converge to a reasonable policy withinthe allotted timesteps in S AVE T
HE C ITY (Fig. 3), and REFIL (Flat) and COPA are only competitivewith ALMA in terms of both sample efficiency and asymptotic performance in one setting: MMM t =t
=t = 45 t =t = 75  (a) (b) (c) (d) a b c d e f g h (e) (f) (g) (h)
[Figure]
Symmetric (Fig. 4d).
These results highlight the importance of ALMA ’s hierarchical abstraction aswell as the sub-environment decomposition used in order to accelerate learning of low-level policies.
Interestingly ALMA appears to supply the greatest performance gains on the environments that aremost difficult, as judged by the absolute success rates.
ALMA stands out in S&Z disadvantage (Fig.4), where the next best methods achieve 20% success, while it only roughly matches the performanceof the top performing methods in MMM symmetric where the highest success rates are around 75%.
We hypothesize that when agents are faced with such a disadvantage, coordinated strategy becomesmore crucial.
Allocation Strategic Complexity Next, we hope to learn whether our allocation controller islearning complex non-trivial strategies.
As such, we implement allocation heuristics derived fromdomain knowledge in each setting and only learn the low-level controllers.
These methods serve as astrong baseline for learned allocation, as they simplify the learning problem for low-level controllersby leveraging the sub-environment decomposition (i.e. masking subtask-irrelevant entities) andhaving a fixed high-level allocation strategy.
For S AVE T HE C ITY , the heuristic allocates each agent tothe nearest building at which they are most useful according to their capabilities.
While relativelysimple, devising a more sophisticated allocation strategy is nontrivial as there are many factors toweigh.
For S TAR C RAFT , we devise two heuristics: the first ( matching ) allocates agents to enemyarmies by matching unit types such that each individual battle is fair.
The other ( dynamic ) onlyconsiders enemy armies that are currently attacking and also attempts to match the unit composition.
In S AVE T HE C ITY (Fig. 3) we find the heuristic is able to learn quickly at the beginning as theaction-level policies improve, but it converges lower than ALMA .
In S TAR C RAFT (Fig. 4) we findthat at least one of the heuristics can be competitive in some cases (e.g. S&Z Symmetric), althoughwhich heuristic performs best is not consistent across settings, and ALMA always outperforms it.
With these results we can conclude decomposing the task into subtasks to be solved by simpler taskexecution controllers is not sufficient for superior performance in our environments and sophisticatedhigh-level strategy must be learned.
In Fig.
5 we qualitatively demonstrate some of the strategies that ALMA is able to learn.
We findthat ALMA is able to allocate agents to subtasks in a manner that considers long term consequencesand balances priorities gracefully (e.g. defending the base vs. defeating enemies).
Decomposition Validity
In order to validate the sub-environment decomposition introduced in
Eqn.
6 we ablate our approach to exclude the task-specific masking performed on each agent’sobservation.
This method is referred to as “No mask” in Figures 3 and 4, where we find thatour method experiences a significant deterioration in performance when not masking irrelevantinformation.
[Figure]
S u cce ss
R a t e
ALMAFlat
ALMA ALMA(Heur.
π ) Heuristic 0 .0 .0 .0 .0 .
S u cce ss
R a t e
To test the boundaries of the required assumptions (independent transitions), we evaluate onsettings in S TAR C RAFT which violate them.
While we train the agents only in cases where enemy armies are maximally spread out from eachother around the agents’ base, in our evaluationsetting we allow for armies to spawn arbitrarilyclose to one another.
In this case, for example,agents assigned to one army may be attackedby other armies or bump into agents attackingother armies, violating subtask transition independence.
We plot the performance of ALMAalongside REFIL (Flat) in Fig.
6a, where thedistance between armies varies along the x-axis.
The further right, the more similar the tasks are towhat is seen during training.
While REFIL’s performance deteriorates smoothly as the composition of subtasks become moredissimilar to those seen during training, ALMA maintains steady performance and even sees aslight uptick in performance towards the middle.
While surprising, this difference in generalizationcapability can be attributed to the modularity of our approach.
While REFIL treats each uniquecomposition of subtasks as novel, the modules comprising our approach (specifically the subtaskexecution controllers and allocation proposal network) only see information relevant to their subtask,which are individually drawn from the same distribution as seen during training.
ALMA ’s increasein performance in the middle can be attributed to the fact that armies being closer together makesit easier for agents to switch between tasks.
Ultimately, as armies become closer together, ourassumptions begin to fail, subtasks blur together, and we finally see ALMA ’s performance drop off.
Joint Training Although ALMA is able to learn all components (subtask allocation and execution/action controllers) jointly, we evaluate its ability to utilize agent policies trained by a differentmethod in a zero-shot manner.
In particular, we see what happens when we replace ALMA ’s actionlevel policies with those trained with a heuristic.
We note that the procedure for training the low-levelcontrollers is identical between these methods and the only difference is the allocation strategy usedthroughout training.
We see in Figure 6b that without further training, ALMA ’s superior allocationstrategy is able to quadruple the effectiveness of the policies learned with the heuristic.
Ultimatelythough, the best performance is only attainable through training allocation and execution controllersjointly, as the execution controllers can learn to succeed in the settings the allocation controllers putthem in, and vice versa, creating a virtuous feedback loop.
Conclusion and Future Work
In this work we have introduced ALMA , a general learning method for tackling a variety of multiagent composite tasks in which each subtasks’ rewards and transitions can be assumed as independent.
By simultaneously learning a high-level allocation policy and action-level agent policies, ALMAis able to succeed in settings which are difficult for flat methods, and in which well-performingheuristics are difficult to come by.
One promising direction for future work is to equip ALMA ’sallocation policy with more sophisticated exploration methods.
Although the stochasticity resultantfrom the ϵ -greedy procedure and the proposal distribution sampling yields surprisingly good results, amore intelligent exploration procedure could even better alleviate the challenges inherent to learningover such a large combinatorial action space.
We expect such an improvement would, for example,prevent some of the variance we observe in S AVE T
HE C ITY (Figure 3).
Another direction for futurework is to discover subtasks and/or observation masks from the environment automatically.
Webelieve this work has immediate practical implications in real-world settings that can be described ascomposite tasks, such as warehouse robotics
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Throughout the text, but particu-
larly in Section 5 in the paragraph entitled “Decomposition Validity”
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This work
is focused on fundamental MARL algorithms and tests on simulated environments.
While MARL algorithms may be implemented for potentially harmful applications, we
do not believe this work uniquely enables such applications.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes]
In the
supplemental material
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
In the appendix
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] All plots include shaded region over a standard deviation
w/ several seeds (the number is specified in the captions)
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] In the Appendix
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Our code builds on
the public code of REFIL [10]
(b) Did you mention the license of the assets?
[No] It’s not mentioned in the paper, but
will be in our public code release
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
AppendixA Network Architectures
Our allocation proposal network and Q network are illustrated in Figures 7 and 8.
Low-level actionutility functions and mixing networks are similar to those described in Iqbal et al.
[10] with the only difference being a replacement of the RNN layers with standard fully connected layers.
We omit
RNNs, as we do not consider settings with significant partial observability.
Furthermore, the run timeis significantly faster since all time steps can be batched during training.
We find the omission of
RNNs does not affect performance in the domains initially used by Iqbal et al.
[Figure] [Figure]
B Environment detailsS AVE T HE C ITY
This environment is inspired by the classical Dec-POMDP task “Factored Firefighting” [18] where several firefighters must cooperate to put out independent fires; however, weintroduce several additional degrees of complexity.
First, agents are capable of contributing to anysubtask, such that the task is amenable to subtask allocation, and we can not use a fixed value functionfactorization.
Second, agents are embodied and must physically move themselves to buildingsthrough low-level actions, rather than being fixed and only having a high-level action space selectingbuildings to fight fires at.
Finally, we introduce several types of agents with differing capabilities(all of which are crucial for success), such that the subtask allocation function must learn whichsubtasks require which capabilities and how to balance these.
The task also shares similarities withthe Search-and-Rescue task from [15]; however, we do not consider partial observability and weintroduce agents with diverse capabilities.
In each episode, there are N =
[2 , 5] agents (circles)
The full map is a 16x16 grid and buildings are randomly spawned across the map.
Agents alwaysstart episodes in a cluster in the center.
Buildings begin episodes on fire at a 40% rate.
Agents arerewarded for increasing a building’s health, completing a building, putting out a fire, and completingall buildings (global reward only).
Agents are penalized for a building burning down or its healthdecreasing due to fire.
S TAR C RAFT
The StarCraft multi-agent challenge (SMAC)
[22] involves a set of agents engaging ina battle with enemy units in the StarCraft II video game.
We consider the multi-task setting presentedby Iqbal et al.
[10] where each task presents a unique combination and quantity of agent and enemytypes.
We increase the complexity by introducing multiple enemy armies and the additional objectiveof defending a centralized base.
These enemy armies attack the base from multiple angles, and theagents must defeat all enemy units while preventing any single enemy unit from entering their basein order to succeed.
As we do for S AVE T HE C ITY , we train simultaneously on tasks with variabletypes and quantities of agents.
We consider four settings based on those presented in [10] consisting of unique combinations of unittypes which require varying strategies in order to succeed.
In one set of settings we give agents andenemies a symmetric (i.e. matching types) set of units, while in the other agents have one fewerunit than the enemies.
Within each set we consider two different pools of unit types: “Stalkers and
Zealots” (S&Z) and “Marines, Marauders, and Medivacs” (MMM).
The former includes a mixture ofmelee and ranged units, while the latter includes units that are capable of healing.
In each setting,there can be between 3 and 8 agents and 2 or 3 enemy armies.
Each enemy army consists of up to 4units.
Stalkers are units that are capable of shooting enemies from afar and are useful for causingdamage without taking as much damage, as they can run immediately after shooting.
Zealots aremelee units (i.e. they must walk up to their enemies to damage them), and they are especially strongagainst Stalkers.
Marines are ranged units that are relatively weak with respect to damage output andhealth.
Marauders are also ranged units and have more health and damage output.
Medivacs are shipsthat float above the battlefield and are able to heal their friendly units.
Notably, multiple Medivacscannot heal the same unit simultaneously, so agents can overpower a Medivac’s healing by targetingthe same unit.
Agents are rewarded for damaging enemy agents’ health, defeating enemy agents, and defeatingenemy armies.
The global reward also includes an additional reward for defeating all armies.
C Experimental Details
Our experiments were performed on a desktop machine with a 6-core Intel Core i7-6800K CPU and3 NVIDIA Titan Xp GPUs, and a server with 2 16-core Intel Xeon Gold 6154 CPUs and 10 NVIDIA
Titan Xp GPUs.
Each experiment is run with 8 parallel environments for data collection and a single
GPU.
D Hyperparameters and Implementation Details
We use Pop-Art [32] to normalize returns across varying scales such that we can use similar hyperparameters across environments.
Hyperparameters for REFIL [10] and COPA [15] are taken directlyfrom those works.
New hyperparameters (e.g. allocation length, epsilon schedules, etc.) were chosenby separately tuning each one.
[Table]