M 3 RL: M IND - AWARE M ULTI - AGENT M ANAGEMENT
R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Most of the prior work on multi-agent reinforcement learning (MARL) achieves optimal collaboration by directly controlling the agents to maximize a common reward.
In this paper, we aim to address this from a different angle.
In particular, we consider scenarios where there are self-interested agents (i.e., worker agents) which have their own minds (preferences, intentions, skills, etc.) and can not be dictated to perform tasks they do not wish to do.
For achieving optimal coordination among these agents, we train a super agent (i.e., the manager) to manage them by ﬁrst inferring their minds based on both current and past observations and then initiating contracts to assign suitable tasks to workers and promise to reward them with corresponding bonuses so that they will agree to work together.
The objective of the manager is maximizing the overall productivity as well as minimizing payments made to the workers for ad-hoc worker teaming.
To train the manager, we propose Mind-aware Multi-agent Management Reinforcement
Learning (M 3 RL), which consists of agent modeling and policy learning.
We have evaluated our approach in two environments, Resource Collection and Crafting, to simulate multi-agent management problems with various task settings and multiple designs for the worker agents.
The experimental results have validated the effectiveness of our approach in modeling worker agents’ minds online, and in achieving optimal ad-hoc teaming with good generalization and fast adaptation.
I NTRODUCTION
As the main assumption and building block in economics, self-interested agents play a central roles in our daily life.
Selﬁsh agents, with their private beliefs, preferences, intentions, and skills, could collaborate effectively to make great achievement with proper incentives and contracts, an amazing phenomenon that happens every day in every corner of the world.
However, most existing multi-agent reinforcement learning (MARL) methods focus on collaboration when agents selﬂessly share a common goal, expose its complete states and are willing to be trained towards the goal.
While this is plausible in certain games, few papers address the more practical situations, in which agents are self-interested and inclined to show off, and only get motivated to work with proper incentives.
In this paper, we try to model such behaviors.
We have multiple workers and a manager, together to work on a set of tasks.
The manager gets an external reward upon the completion of some tasks, or one speciﬁc task.
Each worker has a skill set and preference over the tasks.
Note that their skills and preferences may not align with each other (Fig. 1(a)), and are not known to the manager (Fig. 1(b)).
Furthermore, manager may not get any external reward until a speciﬁc task is complete, which depends on other tasks.
By default, the self-interested workers simply choose the most preferred tasks, which is often unproductive from the perspective of the entire project.
Therefore, the manager gives additional incentives in the form of contracts .
Each contract assigns a goal and a bonus for achieving the goal to a worker.
With the external incentives, workers may choose different goals than their preferences.
Upon completion of assigned goals, the manager receives the rewards associated with those goals and makes the promised payments to the workers.
To generate optimal contracts, the manager must infer the workers’ minds and learn a good policy of goal and reward assignment.
Conventional approaches of mechanism design tackle similar problems by imposing strong assumptions (e.g., skill/preference distributions, task dependencies, etc) to ﬁnd an analytic solution.
In con-
A C B A C B
A C B
Skill
Desire
A C B A C B
A C B ? ? ?
A
B
C
A, $$
B, $
C, $$
Contract Task dependency
Nature of the workers
Incomplete information
Contract generation
Worker
Worker Worker
A C B
A C B A C B
Desire + Bonus Skill
A C B A C B
A C B
Skill
Desire
A C B A C B
A C B ? ? ?
A
B
C
A, $$
B, $
C, $$
Contract Task dependency
Nature of the workers
Incomplete information
Contract generation
Worker
Worker Worker
A C B
A C B A C B
Desire + Bonus Skill
A C B A C B
A C B
Skill
Desire
A C B A C B
A C B ? ? ?
A
B
C
A, $$
B, $
C, $$
Contract Task dependency
Nature of the workers
Incomplete information
Contract generation
Worker
Worker Worker
A C B A C B
A C B
Desire + Bonus Skill Preference Skill
Preference + Bonus Skill
[Figure]
trast, we aim to train a manager using reinforcement learning to i) assess minds of workers (skills, preferences, intentions, etc.) on the ﬂy, ii) to optimally assign contracts to maximize a collaborative reward, and iii) is adapted to diverse and even evolving workers and environments.
For this, we propose a novel framework – Mind-aware Multi-agent Management Reinforcement
Learning (M 3 RL), which entails both agent modeling for estimating workers’ minds and policy learning for contract generation.
For agent modeling, we infer workers’ identities by their performance history, and track their internal states with a mind tracker trained by imitation learning (IL).
For contract generation, we apply deep reinforcement learning (RL) to learn goal and bonus assignment policies.
To improve the learning efﬁciency and adaptation, we also propose high-level successor representation (SR) learning (Kulkarni et al., 2016) and agent-wise (cid:15) -greedy exploration.
As a proof of concept, we evaluate our approach in two environments: Resource Collection and
Crafting in 2D Minecraft, to simulate multi-agent management problems.
The setup and underlying assumptions are designed to mimic real world problems, where workers are not compelled to reveal their true preferences and skills, and there may be dependency between tasks resulting in delayed and sparse reward signals.
Workers may also be deceitful (e.g., accepting a contract even when the assigned goal is unreachable).
Our experiments demonstrate that the manager trained by our approach can i) estimate the mind of each worker from the recent behaviors, ii) motivate the workers to ﬁnish less preferable or intermediate tasks by assigning the right bonuses, iii) is adaptive to changing teams, e.g., change of members and/or change of workers’ skills and preferences, iv) and has good generalization in different team sizes and novel environments.
We have conducted substantial ablation studies by removing the key components, including IL,
SR, agent-wise (cid:15) -greedy exploration, and performance history.
Our approach shows a consistent performance in standard settings as well as in more challenging ones where workers’ policies are stochastic and sub-optimal, instantiated by randomized rule-based policies and RL policies, or there are multiple levels of bonuses required to motivate workers.
R ELATED W ORK
Multi-agent reinforcement learning .
For collaboration problems, common multi-agent reinforcement learning (Littman, 1994; Busoniu et al., 2008) usually trains agents (Oliehoek et al., 2008;
Foerster et al., 2016; Peng et al., 2017; Omidshaﬁei et al., 2017; Lowe et al., 2017)
so that they will jointly maximize a shared reward.
There also have been work on contributing different credits to agents by factorized value functions (Koller & Parr, 1999; Guestrin et al., 2001;
Sunehag et al., 2018;
Rashid et al., 2018), but the spontaneous collaboration assumption is still required.
In contrast, we instead train a manager to manage multiple self-interested workers for an optimal collaboration.
Principal-agent problems .
Our problem setup is closely related to principal-agent problems (or moral hazard problems) in economics.
Our manager and workers can be considered as the principal and agents respectively, where agents and principal have different objectives, and the principal needs to provide the right incentives to ensure that the agents make the best choices for what the principal delegates. These problems face similar technical challenges as our problem setup, e.g., information asymmetry between principals and agents, how to setup incentive cost, how to infer agents types, how to monitor their behaviors, etc. Traditional approaches in economics (Laffont & Martimort, 2002) build mathematical models to address these issues separately in stateless games, often with the assumption that the utility functions and the behavior patterns of the agents are known, leading to complicated models with many tunable parameters. In comparison, our paper provides a practical end-to-end computational framework to address this problem in more complex games in a datadriven way without any assumption about the agents’ utilities and their decision making processes.
Moreover, this framework is adaptive to changes of agents preferences and capabilities, which very few papers in economics have addressed.
Mechanism design . Similar to our problem setting, mechanism design also tackles problems where agents have different and private preferences (Myerson, 1981; Conitzer & Sandholm, 2002).
Its core idea is to set up rules so that the agents will truthfully reveal their preferences for their own interests, and ultimately an optimal collective outcome can be achieved.
Our work differs from mechanism design in several ways.
First, in addition to preferences, we also acknowledge the fact that agents may have different skills.
Second, mechanism design does not consider sequential decision problems, whereas we have to dynamically change the contracts over time.
Optimal reward design .
The contract generation in our work can be seen as reward design.
Some prior work has proposed optimal reward design approaches (Zhang et al., 2009; Zhang & Parkes, 2008; Sorg et al., 2010; Ratner et al., 2018), where a teacher designs the best reward so that the student will learn faster or alter its policy towards the target policy.
In contrast, we try to use deep
RL to train optimal reward design policies to manage multi-agents in more complex tasks.
Meta-learning .
Our work also resembles meta-learning (Wang et al., 2016;
Finn et al., 2017), which typically aims at learning a meta strategy for multiple tasks (Maclaurin et al., 2015; Duan et al., 2017;
Hariharan & Girshick, 2017; Wichrowska et al., 2017; Yu et al., 2018; Baker et al., 2017) with good sample efﬁciency, or for a fast adaptation (Al-Shedivat et al., 2018).
The meta-learning in this paper is for addressing the problem of ad-hoc teaming by training from a limited set of worker population.
Theory of Mind .
Our agent modeling is inspired by the prior work on computational theory of mind, where both Bayesian inference (Baker et al., 2009) and end-to-end training (Rabinowitz et al., 2018) have been applied to understand a single agent’s decision making by inferring their minds.
In this work, we extend this to optimal multi-agent management by understanding agents’ minds.
P ROBLEM S ETUP
In an environment, there is a set of goals G corresponding to several tasks, N self-interested workers with different minds, and a manager which can observe workers’ behaviors but is agnostic of their true minds.
Different from the common Markov game setting for MARL in prior work (Littman, 1994), we use an independent Markov Decision Process (MDP), i.e., (cid:104)S i , A
i , R i , T i (cid:105) , ∀ i ∈ N , to model each worker, where S i and A i are the state space and action space, R i : S
i × G i
→ R is the reward function, and T i : S
i ×
A i → S i is the state transition probabilities.
For achieving goals, a worker has its own policy
π i : S i × G i
→ A i .
We deﬁne the key concepts in this work as follows.
Contract .
A contract is a combination of goal and bonus assignment initiated by the manager to a speciﬁc worker.
For simplicity, we consider discrete bonuses sampled from a ﬁnite set B .
Thus, for worker i at time t , it will receive a contract deﬁned as ( g ti , b ti ) , where g ti ∈ G is the goal and b ti ∈ B is the corresponding bonus for achieving the goal.
Note that the contract will change over time.
Worker’s mind .
We model a worker’s mind by its preferences, intentions, and skills.
We do not study worker agents’ beliefs in this paper, which we leave as future work.
Preference .
A worker’s preference is formally deﬁned as its bounded internal utilities of achieving different goals, u i =
( u ig : g ∈ G ) , where 0 ≤ u ig ≤ u max .
Combined with received contract, the worker agent’s reward function can be deﬁned as
[Equation]
where s g is the goal state.
Intention .
The intention of a worker is the goal it is pursuing at any time, i.e., I ti ∈ G , which is not revealed to the manager.
Based on the reward deﬁned in Eqn.
(1), there are multiple ways to choose the goal.
For a rational worker who is clear about its skills, it will choose the goal by maximizing expected return.
I.e., I t i = arg max g E [ (cid:80) ∞ t =
γ t i r t ig ] , where 0 < γ i ≤ 1 is its discount
P i
F l a tt e n E n c o d e r h
i
L S T M
Worker Policy
F u s i o n m ti
Mind Tracker ( s ti ,a ti ,g ti ,b ti ) ˆ ⇡ ( s t +1 i ,g t +1 i ,b t +1 i )
F u s i o n
Goal Policy
Bonus Policy c t + ⇡ g
High-level SR  g ( s t
+1 i ,m ti ,h i ) ⇡
b
F u s i o n
P oo li n g
F u s i o n
F u s i o n ( s t +11 ,m t 1 ,h 1 ) ( s t +1 N ,m tN ,h N ) … …
b
[Figure]
Module factor.
However, this requires a worker to have a good estimate of its skills and to be honest, which is not always true.
E.g., a worker may want to pursue some valuable goal that it can not reach.
So an alternative way is to maximize the utility instead: I ti = arg max g u ig
+ 1 ( g = g ti ) b ti . This will make a worker’s behavior more deceptive as it may agree to pursue a goal but will rarely produce a fruitful result. In this work, we focus on the second way to achieve a more realistic simulation. Skill .
The skill of a worker is jointly determined by its state transition probabilities T i and its policy conditioned on its intention, i.e., π i ( ·| s ti , I ti ) .
Manager’s objective .
The manager in our setting has its own utility v =
( v g : g ∈ G ) , where v g ≥ 0 is the utility of achieving goal g .
To maximize its gain, the manager needs to assign contracts to workers optimally.
For the sake of realism, we do not assume that the manager knows for sure if a worker agent is really committed to the assignment. The only way to conﬁrm this is to check whether the goal achieved by the worker is consistent with its last assigned goal. If so, then the manager will gain certain reward based on its utility of that goal and pay the promised bonus to the worker. Thus, we may deﬁne the manager’s reward function as:
[Equation]
where S t +1 = { s t
+1 i : i = 1 , · · · , N } is the collective states of all present worker agents at time t
+ 1 .
The objective of the manager is to ﬁnd optimal contract generation to maximize its expected return E [ (cid:80) ∞ t =
γ t r t ] , where 0 < γ ≤ 1 is the discount factor for the manager.
Note that the manager may get the reward of a goal for multiple times if workers reach the goal respectively.
Population of worker agents .
The trained manager should be able to manage an arbitrary composition of workers rather than only speciﬁc teams of workers.
For this, we maintain a population of worker agents during training, and sample several ones from that population in each episode as the present workers in each episode.
The identities of these workers are tracked across episodes.
In testing, we will sample workers from a new population that has not been seen in training.
A PPROACH
Our approach has three main components as shown in Figure 2: i) performance history module for identiﬁcation, ii) mind tracker module for agent modeling, and iii) manager module for learning goal and bonus assignment policies.
We introduce the details of these three components as follows.
4. P ERFORMANCE H ISTORY M ODULE AND M IND T RACKER M ODULE
To model a worker’s mind, we ﬁrst need to infer its identity so that the manager can distinguish it from other agents.
Previous work (Rabinowitz et al., 2018) typically identiﬁes agents via their trajectories in recent episodes.
This only works when diverse past trajectories of agents are available beforehand.
However, this is impractical in our problem as the past trajectories of a worker depends on the manager’s policy, and thus are highly correlated and can hardly cover all aspects of that agent.
In this work, we propose performance history for agent identiﬁcation, which is inspired by the upper conﬁdence bound (UCB) algorithm (Auer et al., 2002) for multi-bandit arm (MAB) problems.
Formally, the performance history of worker i is a set of matrices P i
= { P t i = ( ρ t igb ) :
t = 1 , · · · , T } , where 0 ≤ ρ tigb ≤ 1 is an empirical estimation of the probability of worker i ﬁnishing goal g within t steps if promised with a bonus of b .
We discuss how to update this estimate in Algorithm 1.
These matrices are then ﬂatten into a vector and we encode it to a history representation,
h i , for worker i .
With identiﬁcation, the manager uses an independent mind tracker module with shared weights to update its belief of a worker’s current mental state online by encoding both current and past information: M (Γ ti , h i ) , where Γ ti = { ( s τi , a τi , g τi , b τi ) : τ = 1 , · · · , t } is a trajectory of the worker’s behavior and the contracts it has received upon current time t in the current episode.
4. M ANAGER M ODULE
For contract generation, the manager has to consider all present workers as a context.
Thus, we encode each worker’s information and pool them over to obtain a context representation, i.e., c t +1 =
C ( { ( s t +1 i , m ti , h i ) : i = 1 , . . . , N } ) .
With both individual information and the context, we deﬁne goal policy, π g ( ·| s t
+1 i , m ti , h i , c t +1 ) , and bonus policy, π b ( ·| s t
+1 i , m ti , h i , c t +1 ) , for each worker.
In addition to learning policies for individual workers, we also want the manager to estimate the overall productivity of a team.
A common choice in previous literature (e.g., Lowe et al.
(2017)) is to directly learn a centralized value function based on the context.
However, this is not informative in our case, as the ﬁnal return depends on achieving multiple goals and paying different bonuses.
It is necessary to disentangle goal achievements, bonus payments, and the ﬁnal net gain.
To this end, we adopt the idea of successor representation (SR) (Kulkarni et al., 2016; Zhu et al., 2017; Barreto et al., 2017; Ma et al., 2018), but use it to estimate the expectation of accumulated goal achievement and bonus payment in the future instead of expected state visitation.
By deﬁning two vectors φ g ( c t ) and φ b ( c t ) indicating goal achievement and bonus payment at time t respectively, we may deﬁne our high-level SR, Φ g and Φ b , as Φ g ( c t ) =
E [ (cid:80) ∞ τ =
γ τ φ g ( c t + τ )] and Φ b ( c t ) =
E [ (cid:80) ∞ τ =
γ τ φ b ( c t + τ )] .
We discuss the details in Appendix A.1.
4. L EARNING
For a joint training of these three modules, we use advantage actor-critic (A2C)
(Mnih et al., 2016) to conduct on-policy updates, and learn SR similar to Kulkarni et al.
In addition, we also use imitation learning (IL) to improve the mind tracker.
In particular, we predict a worker’s policy based on its mental state representation, i.e., ˆ π ( ·| s ti , g ti , b ti , m t − 1 i ) , which is learned by an additional cross-entropy loss for action prediction.
Section A.2 summarizes the details.
As our experimental results in Section 5 and Appendix C show, in difﬁcult settings such as random preferences and multiple bonus levels, the policies based on the mental state representation trained with IL have a much better performance than the ones without it.
As the manager is agnostic of workers’ minds, it is important to equip the manager with a good exploration strategy to fully understand each worker’s skills and preferences.
A common exploration strategy in RL is (cid:15) -greedy, where an agent has a chance of (cid:15) to take random actions.
However, this may cause premature ending of contracts where a worker does not have sufﬁcient amount of time to accomplish anything.
Therefore, we adopt an agent-wise (cid:15) -greedy exploration, where a worker has as a chance of (cid:15) to be assigned with a random goal at the beginning of an episode and the manager will never change that goal assignment throughout the whole episode.
In this way, it is easier for a manager to understand why or why not a worker is able to reach an assigned goal.
The details can be seen from the rollout procedure (Algorithm 1) in Appendix B.
E XPERIMENTS 5. G ENERAL T ASK S ETTINGS
We introduce the general task settings as follows.
Note that without additional speciﬁcation, workers are implemented as rule-based agents (detailed in Appendix D.2).
5.1. R ESOURCE C OLLECTION
In Resource Collection, the goals are deﬁned as collecting certain type of resources.
There aretypes of resources on a map (Figure 3a) and the total quantity is 10. A worker can ﬁnd any resources [Figure]
Recipes: [Figure] but only has the skills to dig out certain types of resources.
Note that it may not be skilled at collecting its preferred resources.
We consider three different settings:
• S1: Each agent can collect up to three types of resources including its preferred type.
• S2: Each agent can only collect one type of resource which may or may not be its preferred one.
• S3:
Similar to S2, except that an agent has a different random preference in each episode and thus its preference can not be inferred from history.
A worker can take ﬁve actions: “move forward”, “turn left”, “turn right”, “collect”, and “stop”, and its skill is reﬂected by the effect of taking the “collect’ action.
For workers, the internal utility of a resource is 1 if it is preferred; otherwise it is 0.
The manager receives a reward of 3 for every resource collected under the contracts, and can choose to pay a worker with a bonus of 1 or 2.
5.1. C RAFTING
Different from previous work (Andreas et al., 2017) where all items can be directly crafted from raw materials, we consider three-level recipes (Figure 3b): crafting a top-level item requires crafting certain intermediate item ﬁrst.
There are four work stations (colored blocks) for crafting the four types of items respectively.
For the manager, each top-level item is worth a reward of 10, but collecting raw materials and crafting intermediate items do not have any reward.
Note that certain materials are needed for crafting both top-level items, so the manager must strategically choose which one to craft.
In each episode, there are raw materials sufﬁcient for crafting one to two toplevel items.
All collected materials and crafted items are shared in a common inventory.
We deﬁne 8 goals including collecting raw materials and crafting items.
Each worker prefers one of the collecting goals (the internal utility is 1), and is only capable of crafting one type of items.
We expands the action space in Section 5.1.1 to include “craft”, which will only take effect if it has the ability of crafting the intended item and there are sufﬁcient materials and/or intermediate items.
The manager can choose a bonus from 0 to 2 for the contracts, where 0 means no employment.
B ASELINES
For comparison, we have evaluated the following baselines:
• Ours w/o SR: Learning a value function directly w/o successor representations.
• Ours w/o IL:
Removing action prediction loss.
• Temporal (cid:15) -greedy:
Replacing the agent-wise exploration with conventional (cid:15) -greedy exploration.
• Agent identiﬁcation using recent trajectories:
Encoding an agent’s trajectories in the most recent 20 episodes instead of its performance history, which is adopted from Rabinowitz et al.
• UCB: Applying UCB (Auer et al., 2002) by deﬁning the management problem as N multi-armed bandit sub-problems, each of which is for a worker agent.
In each MAB sub-problem, pulling an arm is equivalent to assigning a speciﬁc goal and payment combination to a worker agent (i
e., there are |G| · |B| arms for each worker agent).
• GT types known: Revealing the ground-truth skill and preference of each worker and removing the performance history module, which serves as an estimation of the upper bound performance.
5. L EARNING E FFICIENCY
[Figure]
During training, we maintain a population of 40 worker agents.
In each episode, we sample a few of them (4 workers in Resource Collection and 8 workers in Crafting).
All approaches we have evaluated follow the same training protocol.
The learning curves shown in Figure 4 demonstrate that ours consistently performs the best in all settings, and its converged rewards are comparable to the one trained using ground-truth agent types as part of the observations.
Moreover, in more difﬁcult settings, e.g., S3 of Resource Collection and Crafting, the beneﬁts of IL, SR, agent-wise (cid:15) -greedy exploration, and the history representations based on the performance history are more signiﬁcant.
In particular, when there are tasks that do not have any reward themselves such as in
Crafting, SR and IL appear to offer the most critical contributions.
Without them, the network hardly gets any training signals.
In all cases, the agent identiﬁcation by encoding recent trajectories learns extremely slowly in Resource Collection and fails to learn anything at all in Crafting.
[Figure]
[Figure]
5. A DAPTATION AND G ENERALIZATION
In real world scenarios, the population of worker agents and their skills may evolve over time, which requires the manager to continuously and quickly adapt its policy to the unforeseeable changes [Figure] [Figure] through a good exploration.
Thus we compare our agent-wise (cid:15) -greedy exploration with the temporal (cid:15) -greedy exploration in two cases: i) training with a population where workers’ skills change drastically after 100,000 episodes (the manager does not know when and which workers’ skill sets have been updated), and ii) testing with a team where 75% of the workers will be replaced with new ones after every 2,000 episodes.
Both strategies keep the same constant exploration coefﬁcient, i.e., (cid:15) = 0 .
To have a better sense of the upper bound in the testing case, we also show the performance of the baseline that knows ground-truth agent information where no exploration is need.
The results of the two cases are demonstrated in Figure 5 and in Figure 6 respectively.
In the ﬁrst case, there are moments when the signiﬁcant change in a population’s skill distribution (i.e., how many workers can reach a speciﬁc goal) will need the manager to greatly change its policy.
E.g., the ﬁrst two changes in Figure 5a result in new types of resources being collected; the changes in Figure 5b force the team to craft a different type of top-level item.
In such cases, our agentwise (cid:15) -greedy exploration signiﬁcantly improves the learning efﬁciency and increases the converged rewards.
When the change is moderate, the policy learned by ours is fairly stable.
In the second case, the managers trained by the three methods achieve similar converged rewards in training.
While the converged reward of our approach is slightly lower than the upper bound due to exploration, it allows the manager to quickly adapt itself to a new team where it has never seen the most team members.
The temporal (cid:15) -greedy on the other hand never achieves a comparable reward even though its performance is comparable to ours when managing a ﬁxed population.
We also want the manager’s policy to have good generalization in novel scenarios unseen in training, which, in our problems, has two aspects: i) generalization in different numbers of present worker agents, and ii) generalization in new environments.
It can be seen from Figure 7 that as the number of workers increases, the manager achieves higher reward until it hits a plateau.
Our approach consistently performs better in all settings.
It even gains higher rewards than the one with groundtruth does when there are fewer workers.
We also add a few walls to create novel environments unseen in training.
With the additional obstacles, workers’ paths become more complex, which increases the difﬁculty of inferring their true minds.
As suggested by Figure 8, the performance indeed decreases the most in S3 of Resource Collection where online intention inference is critical as the workers do not have ﬁxed preferences.
So far, we have only considered rule-based worker agents with deterministic plans.
To see if our approach can handle stochastic and sub-optimal worker policies, we may randomize certain amount of actions taken by the workers (Figure 9) and train a manager with these random policies.
When the randomness is moderate (e.g., ≤ 20% ), the performance is still comparable to the one without random actions.
As randomness increases, we start to see larger decrease in reward.
In Crafting speciﬁcally, random policies make the workers unlikely to achieve assigned goals within the time limit, thus the manager may never get top-level items if the policies are too random.
More results .
In addition to the main experimental results discussed above, we further test our approach from different perspectives: i) showing the effect of the minimum valid period of a contract (i.e., constraints for the manager’s commitment), ii) multiple bonus levels, and iii) training RL agents as workers.
We summarize these results in Appendix C.
C ONCLUSIONS
In this paper we propose Mind-aware Multi-agent Management Reinforcement Learning (M 3 RL) for solving the collaboration problems among self-interested workers with different skills and preferences.
We train a manager to simultaneously infer workers’ minds and optimally assign contracts to workers for maximizing the overall productivity, for which we combine imitation learning and reinforcement learning for a joint training of agent modeling and management policy optimization.
We also improve the model performance by a few techniques including learning high-level successor representation, agent-wise (cid:15) -greedy exploration and agent identiﬁcation based on performance history.
Results from extensive experiments demonstrate that our approach learns effectively, generalizes well and has a fast and continuous adaptation
A D ETAILS OF A PPROACH A. H IGH - LEVEL S UCCESSOR R EPRESENTATION
We deﬁne two vectors indicating the goal achievement and bonus payment at time t : φ g ( c t ) =
( (cid:80) i 1 ( s t +1 i =
s g )
( g ti = g ) : g ∈ G ) and φ b ( c t ) =
( (cid:80) i 1 ( s t +1 i =
s g ti )
( b ti = b ) : b ∈ B ) .
Let w
=
( b : b ∈ B ) be the weights for different bonus payments, then the reward for the manager at the current moment can be written as r t = v (cid:62) φ g ( c t ) − w (cid:62) φ b ( c t ) .
Following the typical SR deﬁnition, we deﬁne our high-level SR as
[Equation]
and
[Equation]
Thus, the value function can be written as
[Equation]
A. D ETAILS OF L EARNING
The policy gradient for the goal assignment is:
[Equation]
where A ( c t ) is the advantage estimation deﬁned as A ( c t
[Equation]
= (cid:80)
[Equation]
Φ g ( c t ) − w (cid:62) Φ b ( c t )) and H ( · ) is the entropy regularization weighted by the constant λ = 0 .
01 for encouraging exploration.
Similarly, the policy gradient for the bonus assignment is
[Equation]
The successor representations may be updated by the following gradient:
[Equation]
For imitation learning, we use the following cross-entropy loss:
[Equation]
Note that the gradient from IL will be combined with Eqn. 6, Eqn. 7, and Eqn. 8 to update corresponding parameters (see Algorithm 2 in Appendix B for details).
B P SEUDO C ODE OF O UR A LGORITHMS
We summarize the rollout algorithm and the learning algorithm in Algorithm 1 and Algorithmrespectively.
: Maximum max constraint
=
Output:
Γ
=
1: Initialize the environment 2: Set performance history update rate η = 0 . 1 . 3: t ←4:
Γ 1 i ← ∅ , ∀ i = 1 , · · · , N 5: repeat 6:
Observe current states ( s ti , a ti ) , ∀ i = 1 , · · · , N , from the environment 7: if t = 1 then 8: for i = 1 , · · · , N do 9: Γ 0 i ← ∅ 10:
Sample a random goal g 0 i ∼ G , and set a minimum bonus b
i as the initial contract.
11: e i ∼ U (0 , 1) 12: τ i ← 1 13:
Assign the contract ( g 0 i , b 0 i ) to worker i 14: Γ ti ← Γ t − 1 i ∪ { ( s ti , a ti , g ti , b ti ) } 15: end for 16: r t ←17: else 18: r t = (cid:80) g ∈G (cid:80) Ni
=1 1 ( s ti = s g )
( g = g t − 1 i )( v g − b t − 1 i ) 19:
ρ τ i ig t − 1 b t − 1 ← (1 − η ) ρ τ i ig t − 1 b t − 1 + η 1 (cid:16) s ti
=
s g t − 1 i (cid:17) #
[Table]
performance history 20:
Update h i based on the new P i 21: m t −
i ← M (Γ t − 1 i , h i ) , ∀ i
= 1 , · · · , N 22: c t ← C ( { ( s ti , m t − 1 i , h i ) : i = 1 , . . . , N } ) 23: for i = 1 , · · · , N do 24: if τ i < T c or e i < (cid:15) then # Commitment constraint and agent-wise (cid:15) -greedy 25: g ti ← g t − 1 i 26: b ti ← b t − 1 i 27:
τ i ←
τ i + 1 28: else 29:
Sample a new goal g ti ∼ π g ( ·| s ti , m t − 1 i , h i , c t ) 30:
Sample a new bonus b ti ∼ π b ( ·| s ti , m t − 1 i , h i , c t ) 31: τ i ← 1 32: end if 33:
Assign the contract ( g ti , b ti ) to worker i 34: Γ ti ← Γ t − 1 i ∪ { ( s ti , a ti , g ti , b ti ) } 35: end for 36: end if 37:
R ← R ∪ { r t } 38: t ← t + 1 39: until t = T max or the task is ﬁnished 40: T ← t
[Figure]
[Figure]
C M ORE E XPERIMENTAL R ESULTS C. C ONSTRAINING THE M ANAGER ’ S C OMMITMENT
The manager’s commitment is deﬁned as the shortest time a contract must remain unchanged, which essentially constrains how frequent the manager can change its goal and bonus assignment.
While short-term commitment allows the manager to quickly update contracts once it has a better mind estimation or once a goal has been reached, long-term commitment often leads to a more accurate skill assessment when the tasks are difﬁcult (e.g., crafting high-level items depends on the results of other tasks and thus needs a longer time).
This is supported by the results in Figure 10: shorter commitment works better in Resource Collection while Crafting needs a longer commitment.
C. M ULTIPLE B ONUS L EVELS
In previous experiments, the internal utility of goals for a worker agent is either 0 or 1.
Here, we sample the internal utility from 0 to 5 in Resource Collection and 0 to 4 in Crafting.
Consequently, the manager needs to select the right bonus from multiple choices to pay each worker.
In Resource
Collection, the manager will get a reward of 5 for every collected resource; in crafting, the reward for a top-level item is still 10.
As shown in Figure 11, the advantage of our approach is even more signiﬁcant compared to the ones in single payment level.
C. RL A GENTS AS W ORKERS
Finally, we train a population of 40 RL worker agents for Resource Collection, where each one is trained with only one goal, and for each goal we train 10 agents using different random seeds.
This creates a population with similar skill distributions as in S2, but with very different policies.
Figure 12 suggests that training to manage RL agents is slower as their policies are less predictable and less rational, but our approach can still gradually learn a good policy whose performance is comparable to the one using rule-based worker agents.
D I MPLEMENTATION D ETAILS D. N ETWORK A RCHITECTURE
Performance History Module .
We ﬂatten the matrices in worker’s performance history P i and concatenate them together to get a single vector.
We then encode this vector into a 128-dim history representation h i .
Mind Tracker Module .
We represent the state of a worker by multiple channels corresponding to different types of items.
We also use four channels to indicate its orientation.
We augment the state with additional |A||G||B| channels, where a channel is either all ones or all zeros for indicating the action it takes, and the goal and bonus it receives.
We then encode the state into a 128-dim hidden state by a convolutional layer with 64 channels and kernels of 1 × 1 , a fully connected (FC) layer (128-dim), and an LSTM with 128 hidden units.
We fuse this vector with the history representation h i .
Speciﬁcally, we adopt an attention-based mechanism for the fusion, where we ﬁrst get an attention vector (128-dim) from the history representation by an FC layer with sigmoid activation, and then do element-wise product between the attention vector and the hidden state from the LSTM.
The fused vector becomes m ti .
This can be formally written as m ti = f
( (cid:96) ti , h i ) =
(cid:96) ti (cid:12) σ ( h i ) , where σ ( · ) is an FC layer with sigmoid activation, (cid:96) ti is the hidden state from the LSTM, and (cid:12) is element-wise product. We fuse it with the state using the same mechanism: f ( φ ( s t +1 i , g t +1 i , b t +1 i ) , m ti ) = φ ( s t +1 i ,
g t
+1 i , b t +1 i ) (cid:12) σ ( m ti ) , where φ ( s t +1 i , g t +1 i , b t +1 i )
is the state encoding.
By feeding the fused vector to an FC layer with softmax activation, we may get the predicted worker policy.
Manager Module .
For each worker, we concatenate its mind representation and history representation together and fuse it with the worker’s state using the attention-based mechanism where the attention vector comes from the concatenated vector.
By pooling over these fused vectors of individual workers, we can get the context vector, from which we construct the two successor representations by two separate FC layers.
Here, we use average pooling, but one may also use other pooling mechanisms.
Finally, for each worker, we concatenate the context vector with its fused vector we obtained before pooling, and consequently get the goal policy and bonus policy by two FC layers with softmax activation.
All modules are trained with RMSProp (Tieleman & Hinto, 2012) using a learning rate of 0.0004.
D. R ULE - BASED W ORKER A GENTS
Each rule-based worker ﬁnds a shortest path to the nearest location related to a goal, and its skill is deﬁned as the post effect of its “collect” and “craft” actions.
In particular, for collecting certain resource/material, it will go to the closest one that has the same type as the goal indicates and is currently not being collected by other agents, whereas for crafting an item, it will go to the corresponding work station if it is currently unoccupied.
If a worker can perform collecting tasks, then after it takes “collect” action, the item will be collected from the map and appears in the inventory; otherwise no real effect will appear.
This applies to crafting tasks as well, except in crafting, task dependencies must also be satisﬁed before “craft” action can take real effect.
When considering random actions, for each step, we sample a random action with the speciﬁed chance to replace the action from the rule-based plan.
D. RL W ORKER A GENTS
We implement all RL worker agents using the same network architecture, where an agent’s state is augmented by additional channels to include the reward for each goal (i.e., |G||B| channels).
We use a convolution layer with 64 channels and kernels of 1 × 1 to encode the state, and feed it to an 128-dim FC layer and then an LSTM with a 128-dim hidden state.
We then predict the policy using a FC layer with softmax activation based on the hidden state from the LSTM.
For each goal, we train 10 RL worker agents using 10 random seeds.
For each episode, we randomly assign a reward from b ∈ B to an agent as the hypothetical reward it may receive from a manager.
We then set the corresponding channel to be all ones and set the remaining |G||B| − 1 channels to be all zeros.
Note that we assume all RL workers have the ability to perform “collect” and “craft” actions.