E-MAPP: Efﬁcient Multi-Agent Reinforcement Learning with Parallel Program Guidance
s
Abstract
A critical challenge in multi-agent reinforcement learning (MARL) is for multipleagents to efﬁciently accomplish complex, long-horizon tasks.
The agents oftenhave difﬁculties in cooperating on common goals, dividing complex tasks, andplanning through several stages to make progress.
We propose to address thesechallenges by guiding agents with programs designed for parallelization, sinceprograms as a representation contain rich structural and semantic information, andare widely used as abstractions for long-horizon tasks.
Speciﬁcally, we introduce
E fﬁcient M ulti-
A gent Reinforcement Learning with P arallel P rogram Guidance (EMAPP), a novel framework that leverages parallel programs to guide multipleagents to efﬁciently accomplish goals that require planning over 10+ stages.
EMAPP integrates the structural information from a parallel program, promotesthe cooperative behaviors grounded in program semantics, and improves the timeefﬁciency via a task allocator.
We conduct extensive experiments on a seriesof challenging, long-horizon cooperative tasks in the Overcooked environment.
Results show that E-MAPP outperforms strong baselines in terms of the completionrate, time efﬁciency, and zero-shot generalization ability by a large margin.
Introduction
Multi-agent reinforcement learning (MARL) has achieved signiﬁcant progress by advancing the cooperation of multiple agents to accomplish complex tasks, e g., multi-robot control [16], autonomousdriving [41, 47] and video games [43, 4].
Most recent advances in MARL focus on the tasksthat feature behavior coordination [33] or joint motion planning [35].
However, for long-horizontasks such as preparing a dish, existing methods often suffer from the inability to understand thetask compositionality and subtasks’ dependencies, resulting in inefﬁcient cooperation and frequentconﬂicts.
Therefore, a natural question to ask here is how we can solve long-horizon tasks in MARL,in the face of large state/action spaces and sparse feedback.
Long-horizon tasks are usually blessed with rich structure, and thus can be divided into a sequenceof subtasks that can be resolved separately.
Previous work [38, 45] has introduced programs asinstructions to help a single agent understand the task hierarchy and accomplish the task.
Inspiredby them, we develop a general multi-agent framework, where agents can leverage programs foraccomplishing long-horizon tasks together.
This is a challenging problem, and three substantial issueswill emerge if multiple agents are naively enforced to follow sequential programs: ﬁrst, sequentialprograms do not explicitly express the dependencies among subtasks, thus hindering the division ofjobs among agents; second, different agents might have different feasibility to accomplish certainsubtasks or lines of programs; third, when assigned to a subtask together, multiple agents might needto collaborate without blocking resources with each other.
As modern CPUs dispatch instructions to parallel processors, we propose a new multi-agent framework, E fﬁcient M ulti-
A gent Reinforcement Learning with P arallel P rogram Guidance (E-MAPP),guiding cooperation and execution by automatically inferring the structure of parallelism from programs.
Speciﬁcally, we ﬁrst design a domain-speciﬁc language (DSL) for multi-agent cooperation,and use multi-stage learning to ground subroutines of a given program into the agents’ policy.
Then,we learn feasibility functions, which entail the ability of agents to complete speciﬁc subroutines inthe program in the status quo.
Finally, we leverage the learned task structure to automatically enforcecooperation and division of labor among agents.
We conduct experiments on gradually more difﬁcult challenges in the Overcooked [43] environment.
The Overcooked environment requires the agents to cooperate on very long-horizon tasks, such aspreparing dishes, while avoiding conﬂicting behaviors.
Our method signiﬁcantly outperforms otherstrong baselines in completion rates and efﬁciency.
The program structure also enables E-MAPP todeliver superior compositional generalization to novel scenes.
Our main contributions can be summarized as follows:
• We formulate a novel task of learning multi-agent cooperation via the guidance of parallel programs.
• We present a novel framework for program grounding and long-horizon planning and instantiate the framework into practical multi-agent reinforcement learning algorithms.
• We demonstrate the effectiveness of E-MAPP in completion rates and generalization ability
over existing strong baselines and provide empirical analysis in long-horizon tasks.
Related Work
Cooperative Multi-Agent Reinforcement Learning.
In a multi-agent cooperative game, agentscollaborate with each other on a common goal [27].
Researchers have investigated many waysto facilitate agent coordination [17, 10, 46, 28, 15].
Value-based MARL algorithms engage indiscovering the relationship between global value function and local value functions [39, 32], whilepolicy-based MARL algorithms use a centralized critic [44, 22] to coordinate agent behaviors.
Morespeciﬁcally, MAPPO [44] and MADDPG [22] leverage a fully-observable central critic to solve theissue of non-stationarity [27].
Value factorization approaches [32, 39, 31] decompose the globalvalue function into a combination of local agent-wise utilities to cope with scalability.
Reinforcement Learning for Long-horizon Tasks.
Reinforcement learning agents usually lackthe ability to plan and reason over a long time horizon due to sparse rewards [42, 12, 34, 13, 14, 26].
Goal-conditioned reinforcement learning [30] is one of the popular paradigms to address the sparsesupervision problem.
Another line of work has presented automatic goal generation and selectionalgorithms [3, 29, 21, 8]; however, this introduces new challenges to design a suitable goal spacethat enjoys rich semantic meanings [11].
By contrast, our work uses the subtasks corresponding topossible subroutines in a program as goals, which are associated with domain knowledge.
Hierarchicalreinforcement learning [HRL; 40, 36, 24, 37, 25] is another path to solve long-horizon tasks, usinga high-level policy for long-term planning and low-level policies for motion planning or speciﬁcbehaviors.
While our work is related to multi-agent hierarchical reinforcement learning [23], whichexplicitly provides the directed acyclic task graph and the necessity of cooperation of each subtask,we focus on learning task structure such as subtask dependencies, loops, and branchings from theprogram and judging the necessity of cooperation without additional information.
Instruction-Guided Agents.
Many recent advances have testiﬁed the advantages of leveragingstructured prior knowledge such as task graphs [1, 19], natural languages [2, 6, 20] and programs [38,45] to promote efﬁcient policy learning.
In contrast to other structured priors, programs stand outbecause of their strictly formatted and composable subroutines [5, 7].
Previous works leverageprograms to enable a single agent to learn complex tasks by following programs [38, 45].
However, aplethora of new challenges have been introduced including task dependencies, collaboration schemes,etc; hence, it is difﬁcult to trivially extend the existing works in the MARL settings.
To enablemulti-thread orderless policy execution, we leverage “plug-and-play” auxiliary functions to infer therelationships among subtasks
Problem Statement
[Equation]
3.1 Program Guided Cooperative Markov Game
An inﬁnite-horizon Markov Game is deﬁned by a tuple ( N , S , A , T , R ,  ) , where N = { 1 , . . .
, N } 89 denotes the set of N engaging agents, S denotes the state space, A
=
A 1 ⇥ · · · ⇥ A N denotes the 90 Cartesian product of all the N agents’ action space, T : S ⇥ A ! S denotes the state transition function 91 from current state s to the next state s 0 for the joint action
a = ( a 1 , . . . , a N ) , R = R 1 ⇥ · · · ⇥ R Ndenotes the Cartesian product of all the N agents’ reward functions, where each R i determines theimmediate reward for the i -th agent from the current state s and joint action a , and  is the discountfactor.
At each time t , each agent i observes the current state s t , makes the decision a it , and receives the reward r it .
In a cooperative game, the collective goal is to optimize the joint policy ⇡ : S !
A to maximize the sum of each agent’s expected cumulative discounted rewards
E a it ⇠ ⇡ i ( ·| s t ) , s t ⇠ P hP 1 t =
P
Ni =1  t r it  s t , a it i .
A program-guided Markov game is a Markov game with a factorized state space.
Speciﬁcally, state
S = S e ⇥ S p , where S e is the common state space of the environment and S p is the multi-pointer 101 program space (see Section 3.2).
Accordingly, the state transition function takes as input the currentcompounded states ( s e , s p ) and joint actions a and then returns the next joint state ( s 0 e , s 0 p ) .
In thisstudy, the transition function of the program space is based on predeﬁned rules.
3.2 Parallel Programs
The program space consists of three components: a domain-speciﬁc language (DSL) that containsall the possible subroutines, a set of pointers that point to relevant subroutines, and a control ﬂowthat manages the pointers.
A complete speciﬁcation of the DSL used by our framework is inthe Appendix A.1.
Inside this DSL, a subroutine is a minimal executable unit in the programthat corresponds to a subtask in the domain (e.g., Chop(Tomato) ).
As in previous works [38,45], we consider two types of subroutines: perception primitives (e.g., IsOnFire() ), which queryabout the status of the environment; and behavior primitives (e.g., Chop(Tomato) , which issue aninstruction.
Control ﬂow involves branching statements ( if/else ), loops ( for/while ), and parallelismindicators ( repeat/parallel ).
The parallelism indicators are designed for multi-thread execution.
The subroutines in a parallel block are possible but not guaranteed to run concurrently, i.e., theagents should reason about what subroutines in a parallel block can be executed concurrently.
Thesubroutine in a repeat block can be executed many times simultaneously.
We summarize all thecurrent subroutines that are possible and executable as the Possible Subroutine Set .
The multi-pointerpoints to all the subroutines in this set.
3.3 Multi-Agent RL with Parallel Programs
[Figure] In our study, we aim at developing a framework for optimizing the joint policy in a parallel programguided cooperative Markov game.
To this end, the agents must keep track of the pointers in theprogram, learn to reason the primitives to choose the right branches, and perform the action eithercollaboratively or individually to pursue high efﬁciency.
Method
[Figure]
Our goal is to enable multiple agents to cooperate to solve long-horizon tasks guided by parallelprograms.
There are three important factors we should consider: ﬁrst, the agents should learn subtaskconditioned policies that can be composed together to accomplish long-horizon tasks and furthergeneralize compositionally to unseen tasks; second, the agents should reason about task dependenciesso that they can automatically parallelize tasks among them; and third, the agents should distinguishbetween cooperative tasks and non-cooperative tasks that can be achieved by a single agent to avoidcompetition for common resources.
We propose E fﬁcient M ulti-
A gent Reinforcement Learning with P arallel P rograms (E-MAPP), amulti-agent reinforcement learning framework where agents can cooperate to solve long-horizontasks by following program guidance.
As shown in Figure 1, E-MAPP includes four components:a perception module that is able to judge whether the queried status exists in the state; a parallelprogram executor that keeps track of subroutines and updates them based on the perception moduleoutput; a task allocator, which extracts queries of interest and obtains the corresponding set of possiblesubtasks from the program executor, and assigns tasks to agents; and a multi-agent policy module,which agents use to make their decisions based on the input subtask.
The following of this section willintroduce these four key components of E-MAPP.
The complete algorithm is shown in Appendix A.3.
4.1 Parallel Program Executor
The program executor keeps a set of pointers pointing to possible subroutines in a domain-speciﬁc
Possible Subroutine Set .
There are four types of control ﬂows: if -routine, while -routine, parallelroutine, and repeat -routine.
Meanwhile, there are two types of subroutines in our program: behaviorprimitives and perception primitives.
Control ﬂows.
An if -routine contains a condition statement (usually a perception primitive) andsubroutines in the corresponding blocks.
A while -routine contains a condition statement and a loopingblock of subroutines.
A parallel -routine contains parallel blocks of subroutines that are possible to beexecuted simultaneously.
A repeat -routine contains an unconditioned block of subroutines that canbe executed multiple times by different agents.
[Figure]
Subroutines.
A behavior primitive (e.g., Chop(Tomato) ) corresponds to a subtask that requires tobe completed by the agents.
As for a perception primitive (e.g., IsOnFire() ), it is a query that requiresa boolean response.
After an action is performed or a perceptive query is responded, the program executor updates itspointers and the Possible Subroutine Set.
Detailed updating rules are shown in Appendix A.2.
Wenote that if an achieved subtask does not correspond to any subroutine in the Possible Subroutine Set,the program executor will terminate the program immediately.
This guarantees that no exceptionswill occur due to violating the instructed order of subtasks.
4.2 Perception Module
The perception module learns to map a query q and the current state s t to a boolean answer h
= ( q, s t ) .
For example, when a perception primitive IsOnFire() is passed to the perception module,it learns to check the existence of ﬁre in the environment and returns true/false.
The architecture isshown in Figure 2.
Speciﬁcally, we randomly sample states, queries, and the ground-truth perceptionh gt as the training dataset and train the network  in a supervised manner.
We use binary crossentropy (BCE) loss L perception =
BCELoss ( h pred , h gt ) as the objective, where h pred denotes theperception output.
In terms of architecture, we encode the perception primitive s p and the commonstate s e with a neural encoder.
The encoded primitives are represented as  and  to modulate theencoded common state.
At the last layer, we use a sigmoid function to obtain binary output.
Trainingdetails and detailed architecture descriptions can be found in Appendix A.5.
In this way, this modulecan determine whether the queried primitive exists in the state to aid the agents’ decision-making.
4.3 Policy Module
The policy module grounds agents’ actions with the programs and encourages cooperation forcompleting the tasks.
The overall policy learning procedure advocates a subtask-conditioned reinforcement learning framework as shown in Figure 3.
Our algorithm backbone is MAPPO [44]and consists of two stages.
More concretely, we ﬁrstly learn a policy for each agent where otheragents’ policies are ﬁxed.
The input is the state and the encoded subtask.
The reward signal is basedon whether a subroutine is executed correctly.
Then we enable multiple agents to coordinate bylearning a joint policy for accomplishing collective goals cooperatively and efﬁciently, with the helpof auxiliary functions and a task allocator.
We use the same architecture to fuse state and goal featuresas that in the perception module in Figure 2.
We also leverage self-imitation learning to tackle thechallenge of sparse rewards, which is shown in Appendix A.
[Figure]
Learning to cooperate on a subtask.
After obtaining single agent policies, we then encourageagents to accomplish harder tasks that require cooperation.
When assigned a cooperative subtask,one of the agents is in charge of ﬁnalizing the task.
The rest of the agents are assistive.
For example,the assistive agents might pass an onion to the leading agents to chop.
In practice, we randomlyappoint one agent as the leading agent and the other ones as the assistive agents.
The leading agent isrewarded if this subtask is completed.
The reward function for the assistive agents is calculated basedon the reachability improvement of the leading agent.
We defer the formal deﬁnition of this rewardfunction to Section 4.4.
Such reward function encourages the assistive agents to help the leadingagents to obtain rewards.
Then, we use a MAPPO-style algorithm to obtain cooperation policy.
Wealso encode the leading agent’s goal into the observation space of the assistive agents to enhanceinformation sharing.
4.4 Task Allocator
The task allocator assigns subtasks to each agent to accomplish a long-horizon task together.
Speciﬁcally, we design the task allocator for efﬁcient coordination based on the following principles: 1) Itonly assigns subtasks that are feasible for the agent(s) without additional prerequisite subtasks.
If asubtask is cooperative, the task allocator would assign the subtask to a number of agents.
3) It assignsa subtask to the agent that has the lowest cost in terms of execution time.
To facilitate the task allocation, we propose to learn three auxiliary functions: a reachability function ,a feasibility function , and a cost-to-go function as illustrated in Figure 4.
Training details of theauxiliary functions are in Appendix A.
[Figure]
The reachability function f reach = f reach ( s t , i, ⌧ ) is deﬁned as a boolean value toindicate if agent i can complete task ⌧ alone at the state s t .
For a cooperative subtask ⌧ and a selectedleading agent i , an extrinsic reward f reach ( s 0 t , i, ⌧ )  f reach ( s t , i, ⌧ ) is provided to the assistive agentsfor learning altruistic behaviors.
To train this reachability function, we randomly sample triplets( s t , i, ⌧ ) and obtain the ground-truth value f gtreach through running the pre-trained non-cooperativepolicy.
Then we optimize the network by reducing the binary cross entropy BCE ( f predreach , f gtreach ) .
Feasibility.
The feasibility function f feas =
f feas ( s t , i, ⌧ ) is deﬁned as a boolean variable indicating if an agent i can complete a subtask ⌧ with others’ assistance at the state s t .
The trainingprocedure of the feasibility function is similar to that of the reachability function, except that weleverage the cooperative policy instead of the single agent policy to collect training data.
Cost-to-go.
The cost-to-go function f cost = f cost - to - go ( s t , i, ⌧ ) denotes the remaining timestepsfor agent i to accomplish the subtask from the state s t .
We leverage trajectories generated by apre-trained intermediate cooperative policy from E-MAPP to train this cost function.
Speciﬁcally,we randomly sample triplets ( s t , i, ⌧ ) and execute the pre-trained cooperative policy in Section 4.3to obtain the ground-truth timesteps f gtcost - to - go to complete the subtask ⌧ .
Then we optimize thenetwork by reducing the Mean Squared Error (MSE) L cost - to - go =
MSE ( f predcost - to - go , f gtcost - to - go ) .
Criteria for subtask allocation.
For a speciﬁc Possible Subroutine Set { ⌧ 1 , . . .
, ⌧ m } , we denote alegal subtask allocation as { T 1 , T 2 , . . .
, T m } such that
• T i is a list of n i agents where n i = |
T i | .
The agents are those in the agent set { 1 , . . .
, N }who aim at completing the subtask ⌧ i .
• For all i 2 { 1 , . . .
, m } , if n i is larger than one, then the agent T 1 i is selected as the leadingagent and the agent(s) T 2 i , . . .
, T n i i are selected as the assistive agent(s).
• For 1  i < j  m , we have that T i \ T j = ; , which means no agent is assigned two
subtasks simultaneously.
We compute the cost of each possible subtask allocation { T 1 , T 2 , . . .
, T m } as
[Equation]
where w feas , w reach , and w cost are tunable hyperparameters and I [ · ] is the indicator function.
The feasibility term encourages the agents to choose the feasible subtasks, the cost-to-go termencourages the agents to choose the less costly subtasks, and the reachability term guarantees thatthe subtask assigned to only one agent is non-cooperative.
We search from the possible allocations{ T 1 , T 2 , . . .
, T m } and apply the one with minimal cost as ﬁnal allocation.
In practice, we also use two hyperparameters: c r to be subtracted from the cost-to-go function toencourage the agents to ﬁnish their ongoing subtasks, and c i to be added to the cost-to-go function toavoid allocating subtasks to an agent who can never accomplish a task within a timeout threshold t o .
Experiments
In this section, we aim to investigate the following key questions.
First,is the program guidance helpful for agents to understand long-horizontasks in comparison with other structured information guidance?
Second, are the parallel structures in the programs bring forth cooperativebehaviors among the agents?
Third, does the task allocator improve thecompletion rates and the time efﬁciency of the long-horizon tasks byvirtue of the auxiliary functions?
5.1 Environment Description
To evaluate the proposed framework, we extend the previous environment [43] to a more challenging discrete grid-world environmentmimicking the video game “Overcooked”.
Concretely, we extend temporally for the horizon length of a task by adding additional behaviorssuch as “wash dishes” and “put out ﬁre”.
As shown in Figure 5, agentscan navigate through the grid world, interact with objects (e.g., tomatoes, knives), and deliver the dishes to the customers (the yellow star).
The goal of the agents is to serve dishes according to the given recipes that can be divided intosubtasks.
More details about the environment can be found in Appendix A.7.
5.2 Setup
Tasks.
We test the agents on a variety of tasks with different difﬁculty levels.
The easy tasks containonly one subroutine for veriﬁcation.
The medium tasks contain two or three subroutines.
The hardtask requires the agents to complete multiple dishes that require at least two subroutines each whileputting out the randomly appearing ﬁre.
We conduct evaluation in two different patterns, based onseen or unseen tasks.
Note that both evaluation scheme are conducted on novel maps, which makesthe task more challenging.
The unseen tasks share the subroutines with the seen tasks but is neverused as a goal during training.
The unseen tasks are used to test whether the methods can generalizecompositionally.
We also note that for the tasks with a repeat command (e.g., repeatedly pick anonion from the supply) [Table]
Metrics.
We use the completion rates and the average scores as the metrics.
The completion rateis deﬁned as the percentage of tasks that can be completed in an episode.
The average score is thediscounted accumulative rewards in an episode across all testing tasks.
Agents will receive a rewardof 1 when the ﬁnal goal is achieved and receive a reward of 0 .
2 when a correct subtask is completed.
Each algorithm is tested in 1000 environments and we report their average in the tables.
Baselines.
We compare our model with two baselines: MAPPO [44] and a natural language-guidedagent.
Detailed descriptions are in Appendix A.9.
5.3 Results
[Table]
Results with seen tasks in novel maps.
Table 1 shows the completion rate and average scoreson seen tasks in novel maps, demonstrating that E-MAPP excels at understanding the structureof complex tasks.
As expected, the end-to-end MAPPO model performs well on short-horizontasks, but suffers from a signiﬁcant performance drop when the task becomes complex.
The naturallanguage–guided model also has a large performance drop when the horizon of the tasks becomeslonger.
We attribute this to that the agents fail to understand the complex task structure described innatural language without explicit structure.
By contrast, E-MAPP performs well even when the taskshave a long horizon with various accidental events happening.
Generalization with unseen tasks in novel maps.
Table 2 shows the results on the zero-shotgeneralization scheme.
The scenarios and tasks are very different from the training domain, posingextra challenge to all the algorithms including the auxiliary functions of E-MAPP.
We ﬁnd that EMAPP is signiﬁcantly better than both the previous multi-agent RL algorithms and a language-guidedagent.
We attribute the success of E-MAPP to the fact that programs have better compositionality andless ambiguity.
In the training time, E-MAPP learns certain subroutines from other tasks; it appliesthe behavior compositionally to achieve goals that is never seen during training.
More visualizedresults are in Appendix A.8. [Table]
Natural
[Table]
Ablation of auxiliary functions.
We analyze the signiﬁcance of the proposed key components inour model by comparing our model with the variants that removes 1) the feasibility predictor 2) thereachability predictor 3) the cost-to-go predictor.
The completion rates and the average scores areshown in Table 3.
We ﬁnd that all the components are important to E-MAPP.
Speciﬁcally, weﬁnd removing the reachability predictor from E-MAPP leads to a signiﬁcantly increased averagetimesteps, and this is the component that provides the largest performance gain.
We attribute this tothat the reachability function pointed out the subtasks that can be completed individually, providingan impetus for agents to parallelize.
[Table]
Ablation of parallelism in the programs.
We compare the proposed parallel programs withsequential programs in the same environments.
In Table 3, we ﬁnd that the agents with parallelprograms use 15% fewer time steps to accomplish the same goal as those with sequential programs.
This results show that the parallel programs are central to E-MAPP in terms of time efﬁciency.
Extension to continuous control.
We also investigate whether E-MAPP can be effective on taskswith continuous action space based on the Stacking environment.
In this environment, two 7 DoFfranka arms are required to stack blocks following speciﬁc orders and location constraints, where thelow-level policy to pick is provided.
Demos in Stacking can be found in Appendix A.10.
5.4 Visualization of Learned Behaviors
[Figure]
Figure 6a that two are two concurrently
Figure 6b shows that both of the agents are trying to perform a repeated task
Figure 6c shows that one agent is passing an onion to the agent who can chop it. In Figure 6, we show the representative learned behaviors of E-MAPP. In Figure 6(a), the twosubroutines are allocated to two agents separately. This demonstrates the effectiveness of thefeasibility function, with which the task allocator successfully identiﬁed the parallelizable subtasksthat can be executed concurrently.
We note that the parallel indicator in the program only suggeststhe agent trying to identify what subtasks are parallelizable rather than providing a ground-truthtask structure.
In Figure 6(b), we ﬁnd that the two agents are performing the same task in a repeatsubroutine independently.
We show the allocator can assign tasks to multiple agents that are reachableto the goal.
In Figure 6(c), we show that an assistive agent is passing an onion to the leading agent tochop.
This shows how the agents learn to cooperate with each other.
Conclusion
In this paper, we ﬁrst formulate the problem of program guided multi-agent tasks.
We propose
Efﬁcient Multi-Agent Reinforcement Learning with Parallel Programs (E-MAPP), an effectiveframework that uses a type of parallelism-aware program for multi-agents’ collaboration and a taskallocation strategy via a set of learnable auxiliary functions.
The results show that our algorithm caninfer the task structure and signiﬁcantly boost completion rates, efﬁciency, and generalization abilityon long-horizon tasks.
Limitations.
Our current framework does not consider more challenging scenarios such as dynamicsscenes or generalization to novel objects.
We believe that this work opens a welcoming avenue to thisresearch direction and more future works will address additional challenges.