L EARNING E XPENSIVE C OORDINATION : A N E VENT -B ASED D EEP RL A PPROACH
Anonymous authors
Paper under double-blind review
A BSTRACT
Existing works in deep Multi-Agent Reinforcement Learning (MARL) mainly focus on coordinating cooperative agents to complete certain tasks jointly.
However, in many cases of the real world, agents are self-interested such as employees in a company and clubs in a league.
Therefore, the leader, i.e., the manager of the company or the league, needs to provide bonuses to followers for efﬁcient coordination, which we call expensive coordination .
The main difﬁculties of expensive coordination are that i) the leader has to consider the long-term effect and predict the followers’ behaviors when assigning bonuses and ii) the complex interactions between followers make the training process hard to converge, especially when the leader’s policy changes with time.
In this work, we address this problem through an event-based deep RL approach.
Our main contributions are threefold.
(1) We model the leader’s decision-making process as a semi-Markov Decision
Process and propose a novel multi-agent event-based policy gradient to learn the leader’s long-term policy.
(2) We exploit the leader-follower consistency scheme to design a follower-aware module and a follower-speciﬁc attention module to predict the followers’ behaviors and make accurate response to their behaviors.
(3) We propose an action abstraction-based policy gradient algorithm to reduce the followers’ decision space and thus accelerate the training process of followers.
Experiments in resource collections, navigation, and the predator-prey game reveal that our approach outperforms the state-of-the-art methods dramatically.
I NTRODUCTION
Deep Multi-Agent Reinforcement Learning (MARL) has been widely used in coordinating cooperative agents to jointly complete certain tasks where the agent is assumed to be selﬂess (fully cooperative), i.e., the agent is willing to sacriﬁce itself to maximize the team reward.
However, in many cases of the real world, the agents are self-interested, such as taxi drivers in a taxi company (ﬂeets) and clubs in a league.
For instance, in the example of taxi ﬂeets (Miao et al., 2016), drivers may prefer to stay in the area with high customer demand to gain more reward.
It is unfair and not efﬁcient to compel the taxi driver to selﬂessly contribute to the company, e.g., to stay in the low customer demand area.
Forcing the drivers to selﬂessly contribute may increase the income for the company in a short-term but it will ﬁnally causes the low efﬁcient and unsustainable of that company in the long run because the unsatisﬁed drivers may be demotivated and even leave the company.
Another important example is that the government wants some companies to invest on the poverty area to achieve the fairness of the society, which may inevitably reduce the proﬁts of companies.
Similar to previous example, the companies may leave when the government forces them to invest.
A better way to achieve coordination among followers and achieve the leader’s goals is that the manager of the company or the government needs to provide bonuses to followers, like the taxi company pays extra bonuses for serving the customers in rural areas and the government provides subsidies for investing in the poverty areas, which we term as expensive coordination .
In this paper, we solve the large-scale sequential expensive coordination problem with a novel RL training scheme.
There are several lines of works related to the expensive coordination problem, including mechanism design (Nisan & Ronen, 2001) and the principal-agent model (Laffont & Martimort, 2009).
However, these works focus more on static decisions (each agent only makes a single decision).
To consider sequential decisions, the leader-follower MDP game (Sabbadin & Viet, 2013; 2016) and
the
RL-based mechanism design (Tang, 2017; Shen et al., 2017) are introduced but most of their works only focus on matrix games or small-scale Markov games, which cannot be applied to the case with the large-scale action or state space.
The most related work is M 3 RL (Shu & Tian, 2019) where the leader assigns goals and bonuses by using a simple attention mechanism (summing/averaging the features together) and mind (behaviors) tracking to predict the followers’ behaviors and makes response to the followers’ behaviors.
But they only consider the rule-based followers, i.e., followers with ﬁxed preference, and ignore the followers’ behaviors responding to the leader’s policy, which signiﬁcantly simpliﬁes the problem and leads the unreasonability of the model.
In the expensive coordination problem, there are two critical issues which should be considered: 1) the leader’s long-term decision process where the leader has to consider both the long-term effect of itself and long-term behaviors of the followers when determining his action to incentivise the coordination among followers, which is not considered in (Sabbadin & Viet, 2013; Mguni et al., 2019); and 2) the complex interactions between the leader and followers where the followers will adapt their policies to maximize their own utility given the leader’s policy, which makes the training process unstable and hard, if not unable, to converge in large-scale environment, especially when the leader changes his actions frequently, which is ignored by (Tharakunnel & Bhattacharyya, 2007;
Shu & Tian, 2019).
In this work, we address these two issues in the expensive coordination problem through an abstraction-based deep RL approach.
Our main contributions are threefold.
(1) We model the leader’s decision-making process as a semiMarkov Decision Process (semi-MDP) and propose a novel event-based policy gradient to learn the leader’s policy considering the long-term effect (leader takes actions at important points rather than at each step to avoid myopic decisions.)
(Section 4.1).
(2) A well-performing leader’s policy is also highly dependent on how well the leader knows the followers.
To predict the followers’ behaviors precisely, we show the leader-follower consistency scheme.
Based on the scheme, the follower-aware module, the follower-speciﬁc attention module, and the sequential decision module are proposed to capture these followers’ behaviors and make accurate response to their behaviors (Section 4.2).
(3) To accelerate the training process, we propose an action abstraction-based policy gradient algorithm for the followers.
This approach is able to reduce followers’ decision space and thus simpliﬁes the interaction between the leader and followers as well as accelerates the training process of followers (Section 4.3).
Experiments in resource collections, navigation and predatorprey show that our method outperforms the state-of-the-art methods dramatically.
R ELATED W ORKS
Our works are closely related to leader-follower RL, temporal abstraction RL, and event-based RL.
Leader-follower RL.
The leader-follower RL targets at addressing the issue of expensive coordination where the leader wants to maximize the social beneﬁt (or the leader’s self-beneﬁt) by coordinating non-cooperative followers through providing them bonuses.
Previous works have investigated different approaches to solve the expensive coordination, including the vanilla leader-follower
MARL (Sabbadin & Viet, 2013; Laum ˆ onier & Chaib-draa, 2005), leader semi-MDP (Tharakunnel & Bhattacharyya, 2007), multiple followers and sub-followers MARL (Cheng et al., 2017), followers abstraction (Sabbadin & Viet, 2016), and Bayesian optimization (Mguni et al., 2019).
But most of them focus on simple tabular games or small-scale Markov games.
The most related work (Shu & Tian, 2019) leverages the deep RL approach to compute the leader’s policy of assigning goals and bonuses to rule-based followers.
But their method performs poorly when the followers are RLbased.
In this work, we aim to compute the leader’s policy against the RL-based followers in the complex and sequential scenarios.
Temporal abstraction RL.
Our methods are also related to temporal abstraction method (Sutton et al., 1998; Daniel et al., 2016; Bacon et al., 2017; Smith et al., 2018; Zhang & Whiteson, 2019;
Vezhnevets et al., 2016).
The basic idea of temporal abstraction is to divide the original one-level decision process into a two-level decision process where the high-level part is to decide the meta goal while the low-level policy is to select the primitive actions.
Our leader’s decision process is different from those methods mentioned above because the leader’s policy can naturally form as an intermittent (temporal abstraction) decision process (semi-MDP) (Tharakunnel & Bhattacharyya, 2007) and it is unnecessary to design the two-level decision process for the leader (since the low-level decision process is the follower).
Based on the nature of the leader, a novel training method is introduced.
Event-based RL & Planning .
Previous studies also focus on using events to capture important elements (e.g., whether agent reaches a goal) during the whole episode.
Upadhyay et al.
(2018) regard the leader’s action and the environment feedback as events in the continuous time environment.
Becker et al.
; Gupta et al.
(2018) leverage events to capture the fact that an agent has accomplished some goals.
We adopt this idea by depicting the event as the actions taken by the leader at some time steps and design a novel event-based policy gradient to learn the long-term leader’s policy.
S TACKELBERG M ARKOV G AMES
Our research focuses on single-leader multi-follower Stackelberg Markov Games (SMG) (Mguni et al., 2019; Sabbadin & Viet, 2013), which can be formulated as a tuple G =
(cid:104)N , S , A , Ω , P, R , γ (cid:105) .
N is the set of N followers, i.e., |N| =
N .
S is the set of states.
s 0 ∈ S
⊂ S is an initial state and
S 0 is the set of initial states.
A = × k ∈N A k is the set of joint actions for followers where a k ∈
A k is an action for the k -th follower.
ω ∈ Ω =
× k ∈N Ω k is an action for the leader and ω k
= { g k , b k } ∈ Ω k is a goal and a bonus that the leader assigns to the k -th follower.
P : S ×
A → ∆( S ) is the transition function 1 and R =
× k ∈N r k × r l is the reward function set where r k : S ×
A × Ω → R is the reward function for the k -th follower and r l : S ×
A × Ω → R is the reward function for the leader.
γ is the discount factor and a is a joint action of followers.
The leader’s policy is deﬁned as µ = (cid:104) µ k (cid:105) k ∈N where µ k : Ω × S → ∆(Ω k ) is the leader’s action to the k -th follower given the leader’s action in the previous timestep ω
t − 1 and the current state s t .
∆( · ) is a probability distribution.
The followers’ joint policy is deﬁned as π = (cid:104) π k (cid:105) where π k : Ω k × S → ∆( A k ) is the k -th follower policy given the leader’s action ω kt and the current state s t .
Given the policy proﬁle of the leader and followers (cid:104) µ , π (cid:105) , the follower’s utility is deﬁned as
J k ( µ , π ) =
E (cid:104)(cid:80)
Tt =
γ t r kt ( s t , a t , ω t ) (cid:105) and the leader’s utility is J ( µ , π ) =
E (cid:104)(cid:80)
Tt =
γ t r lt ( s t , a t , ω t ) (cid:105) .
We assume that the leader and followers aim to maximize their own utilities.
We deﬁne the trajectory τ as a sequence of state, leader’s action, and followers’ actions (cid:104) ω − 1 , ( s t , a t , ω t )
Tt =0 (cid:105) where ω − 1 is the ﬁrst step leader’s action and is set to zero.
M ETHODOLOGY
In this section, we propose a novel training scheme to train a well-performing leader policy against both rule-based and RL-based followers in the expensive coordination problem.
We address the two issues, the leader’s long-term decision process and the complex interactions between the leader and followers, with three key steps: (a) we model the leader’s decision-making process as a semiMarkov Decision Process (semi-MDP) and propose a novel event-based policy gradient to take actions only at important time steps to avoid myopic policy; (b) to accurately predict followers’ behaviors, we construct a follower-aware module based on the leader-follower consistency, including a novel follower-speciﬁc attention mechanism , and a sequential decision module to predict followers’ behaviors precisely and make accurate response to these behaviors; and (c) an action abstractionbased policy gradient method for followers is proposed to simplify the decision process for the followers and thus simplify the interaction between leader and followers, and accelerate the convergence of the training process.
4. E VENT -B ASED T RAJECTORY O PTIMIZATION FOR L EADER
We ﬁrst describe the event-based trajectory optimization for the leader.
As we mentioned above, the leader’s decision process can be naturally formulated as a semi-MDP (Tharakunnel & Bhattacharyya, 2007).
Therefore, we ﬁrstly describe the basic ideas of semiMDP using the modiﬁed option structure.
We deﬁne the modiﬁed option as a tuple: (cid:104) µ , (cid:0) β k (cid:1) k ∈N (cid:105) where µ is the leader’s policy as we deﬁned above and β k ( s t , ω t − 1 ) : S ×
[Figure]
Ω → [0 , 1] is the termination function for the k -th follower, to indicate the probability whether the leader’s action to the k -th follower changes ( ω kt −(cid:54)
= ω kt ).
Based on these deﬁnitions, we formulate the one-step option-state transition function with decay as:
P
[Equation]
s
[Equation]
, ω
[Equation] [Equation]
[Equation]
β
[Equation]
s
[Equation]
, ω
[Equation]
) µ k (cid:0) ω kt | s t , ω t − 1 (cid:1) } , where 1 is the indicator function and π ( a t | s t , ω t ) =
(cid:81) k ∈N π k (cid:0) a k t | s t , ω k t (cid:1) is the joint policy for followers.
Notice that this is an extension of the augmented process mentioned in (Bacon et al., 2017).
Differently, we do not have the low-level policy here (the low-level policy is the follower) and since we only focus on the ﬁnite time horizon, γ is set to be 1 .
Our modiﬁed option is used to depict the long-term decision process for the leader as shown in Fig.
Now we start to discuss our leader’s policy gradient.
In fact, it is not easy to directly optimize the leader’s utility based on this multi-agent option-state transition function since this form includes leader’s different action stages to different followers.
Notice that for a sampled trajec- tory, the occurrence of the leader actions is deterministic.
Therefore, we can regard the time step and the action the leader takes at that step as an event and deﬁne the (universal) event set U
T = {(cid:104) t i , ω kt i (cid:105)| t i ≤ T, k ∈ N} .
We use the notation e ki = (cid:104) t i , ω kt i (cid:105) to represent the leader’s action to the k -th follower at step t i , i is the index of the event.
Since we focus on the change of the actions from the leader, we further deﬁne a set that represents a collection of new actions ( ω kt (cid:54) =
ω kt − 1 ) taken by the leader within that trajectory: A T
= { e ki | ω kt i (cid:54) =
ω kt
i − 1 , t i ≤ T, k ∈ N} ⊆
U T , where t i − 1 is the previous time step.
A T represents when and how the leader commits to a new action (an example can be found in Fig. 2a).
For brevity, e kj (cid:54)∈ A T means e kj ∈ U T \
A T .
The probability of A
T can be represented as:
[Equation]
where t j − 1 is the previous time step for t j .
This equation illustrates that the probability of the occurrence of a certain leader’s event set within a trajectory.
Concretely, the leader changes action to the k -th follower at t i ∈
e ki while maintaining the same action within the interval from t i − 1 ∈ e ki −to t i ( s.t. , t i ∈
e ki ) .
Similarly, we can further deﬁne the probability of the whole trajectory τ as:
[Equation]
Comparing with P ( A T ) , P ( τ ) includes the probability of the followers as well as the state transition.
Do note that our goal is to maximize max A T E P ( τ )
[ R τ ( T )] , indicating that the leader is required to select an action that can maximize the accumulated reward,
where R τ ( T ) =
(cid:80) Tt =
γ t r lt is the accumulated reward and τ is to stress that its accumulated reward is from the trajectory τ .
Following the REINFORCE trick (Sutton & Barto, 1998), the policy gradient for the termination function and the leader’s policy function can be formulated under the following proposition:
Proposition 1.
The policy gradients for the termination function β k ( s t i , ω t i ) and leader’s policy function µ k (cid:0) ω kt i | s t i , ω
t i − 1 (cid:1) can be written as:
∇
[Table]
where θ and ϑ are the parameters for the termination function β kθ and leader’s policy µ kϑ .
I ( · ) and
I (cid:48) ( · ) are the piece-wise functions:
[Equation]
All the proofs can be found in Appendix A. Proposition 1 implies that under the event-based method, whether the leader’s commitment to a new action will induce different policy gradients for both termination function and the policy function.
However, from the empirical results, we ﬁnd that the leader’s policy function updates rarely during the whole episode because the policy only updates when the leader commits to a new action, which causes the sample inefﬁciency.
Notice that in fact the leader commits to the same action when e ki / ∈ A T .
Therefore, the policy indication function I (cid:48) ( · ) can be formulated in an alternative way:
[Equation]
This form considers both committing to a new action and maintaining the same actions (Details can be found in Remark 2), which we call the Event-Based Policy Gradient (EBPG) and the previous one as the sparse EBPG respectively.
Intuitively, the dense EBPG is better than the sparse EBPG because it updates the leader’s policy function more frequently than the sparse one.
For example, in time step t , supposing that the leader chooses a wrong action for follower k and receives a negative reward.
Then, the leader should learn to diminish the action chosen that state by EBPG.
The sparse EBPG only do one PG during before terminating the action (at the committing action step) while the dense one does PG in each step before terminating the action.
The latter can provide more signal to correct the wrong action.
Experiments also reveal that the dense one is better (Sec. D.3.3).
4. N EURAL N ETWORK BASED L EADER
The EBPG approach is able to improve leader’s performance.
However, it is still very hard for the leader to choose actions considering long-term effect only based on the current state information.
This is because the followers change their behaviors over time according to the leader’s policy.
Therefore, we introduce new modules and training schemes so as to capture the change of the followers’ behaviors as well as the global state.
To abstract the complicated state information, we use neural networks to learn the state representation.
To capture the followers’ behaviors and make accurate response to their behaviors, we design three modules: (1) we exploit the leader-follower consistency under game regularization and policy bound conditions, (2) based on the consistency, a follower-aware module is introduced and (3) based on the follower-aware module, a novel attention mechanism, and sequential decision making module is designed to make accurate response to these followers’ behaviors as shown in Fig.
Leader-Follower Consistency.
In previous works, a surge of researches focus on predicting other agents’ behaviors through historical information, where the other agents are assumed to be opponents of that agent, which is only suitable for zero-sum games (Zheng et al., 2018;
Foerster et al., 2018;
He et al., 2016).
However, these methods cannot be directly applied to our case because SMG is not zero-sum.
We note that Shu & Tian (2019) attempt to directly use the followers’ behavior prediction module (use the history of the followers to predict their future actions) but do not analyze when and how it works.
To ensure that the leader can predict the followers’ behaviors, we introduce the following assumptions.
Assumption 1.
(Game regularization)
The leader-follower state-action space ( A × Ω × S ) is compact and r k is a continuous function w.r.t.
µ bounded by R max .
This assumption is inspired by (Antos et al., 2008).
We only extend it into the multi-agent forms.
This assumption indicates that the action and states space should be limited and the reward function for the leader action should be smooth.
Assumption 2.
(Policy Bound)
For any agent k , reward function r k and policy is consistency, i.e.,
[Equation]
Where C 2 is a constant that satisﬁes C 2 > 0 .
π (cid:48)
k is the k -th follower’s new policy.
µ (cid:48) is the leader’s new policy.
This assumption is inspired by (Mguni et al., 2019).
π − k indicates the joint policy without the k -th agent’s.
This assumption indicates that the change of the leader causes only slightly changes on each followers policy.
Based on these two assumptions, we propose a proposition here:
Proposition 2.
(Leader-Follower Consistency.)
If both the assumptions of game regularization and policy bound are satisﬁed, for ∀ (cid:15) > 0 , k ∈ N , there exists δ > 0 , such that | µ − µ (cid:48) | ≤ (cid:15) implies (cid:12)(cid:12)
π k − π (cid:48) k
(cid:12)(cid:12) ≤ δ , where µ (cid:48) and π (cid:48)
k are the new policies for the leader and the k -th follower respectively.
This proposition reveals that the change of the leader causes only slightly changes on each follower’s policy under the game regularization assumption and the policy bound assumption, which is fundamental to follower-aware learning.
Roughly speaking, the game regularization requires that the states, actions, and rewards are bounded while the policy bound states that a little change of a follower’s policy does not change its utility so much.
The former is from the game itself and we only focus on the latter.
To satisfy the latter, one possible method is to make the µ and µ (cid:48) close because the followers always ﬁnd the best response to the leader’s policy and if the leader changes a little, the followers do not change too much since the new best response to µ (cid:48) is not far away from best response to µ .
One direct method is to slow down the learning rate of the leader to make µ (cid:48) and µ close.
Moreover, for the leader part, taking the right actions is also an important way to guarantee the second assumption because the taken action will more precisely decrease the probability of huge change of the whole process and stabilize the training process.
There is an interesting phenomenon that on one hand, knowing more about the followers can diminish the wrong decision and thus aids the establishment of the consistency.
On the other hand, the consistency will further guarantee the accuracy of the follower-aware module.
Therefore, they form a positive feedback.
Follower-Aware Module.
Based on the leader-follower consistency, we can safely implement the follower-aware module to our network.
Before we discuss this module in details, we ﬁrst deﬁne the history for both the leader and followers.
For the k -th follower, its history at time step t is a sequence of states, its own actions, and the leader’s actions to it, i.e., h kt = (cid:104) ( s t (cid:48) , a kt (cid:48) , ω kt (cid:48) ) t (cid:48) ≤ t (
cid:105) ∈
H kt while the leader’s history is the stack of all followers histories h t = (cid:104)
h kt (cid:105)
k ∈N ∈
H lt .
Then, we deﬁne the history-based leader’s policy as: µ k h t ( s t ) =
Z − 1 p ( ω kt | s t , ˆ a t , h t ) p (ˆ a t
| s t , h t ) ∝ p ( ω kt | s t , ˆ a t , h t ) (cid:81) k p k (cid:0) ˆ a kt | s t , h kt (cid:1) , where Z is the normalization term, p k is the predicted action probability of the k -th follower and ˆ a deﬁnes the predicted action (predicted by the leader).
p is to stress the output is a probability.
Since we cannot directly obtain an accurate estimation of a k , we adopt an alternative way to leverage history information and imitation learning to make a prediction of other agents’ action probability function p k (cid:0) ˆ a kt | s t , h kt (cid:1) (Implementation details can be found in Appendix B) and p ( ω kt | s t , ˆ a t , h t ) is designed using the attention mechanism as well as the sequential decision module presented below.
Follower-Speciﬁed Attention Mechanism.
Inspired by (Chen et al., 2018), we introduce a follower-speciﬁc attention mechanism to identify the important followers where the important followers are followers who has just ﬁnished a task and the leader has to commit new actions to these followers.
The attention mechanism is as follows:
[Equation]
Where w k is the weight of the k -th follower, A ( · ) : R d s × d akt × d hkt → R d c is a function to blend various information of an agent together ( d c means the dimension of the output of the A ( · ) ), c t is the total attention value (the followers’ interaction information), and c kt is the k -th agent attention value (the compression of history, states and actions for follower k as well as other followers).
ˆ a kt is the output of p k (cid:0) ˆ a kt | s t , h kt (cid:1) , the predicted the k -th follower’s action.
This attention mechanism is better because it quantiﬁes the importance of each follower in each state through learning while the original methods only adds/averages all the features ( s t , ˆ a kt , h kt ) together (Shu & Tian, 2019).
Another advantage is that its weights can be visualized to see which follower is important to the leader at current step.
(Details can be found in Appendices D.2.1 & D.3.4).
c kt then is used by β k and µ k .
Sequentially Determining Goals and Bonuses.
Also notice that the goal and the bonus are sequentially correlated.
Therefore, it is better for the leader to choose the bonus and the goal sequentially rather than select them independently.
Therefore, to consider the goal and bonus jointly when making a decision, we build a probabilistic graph-based model as: p ( ω kt | s t , ˆ a t , h t )
≈ p ( g kt ; b kt | c kt )
∝
[Figure]
p ( b kt | g kt , c kt ) × p ( g kt | c kt ) , the ﬁrst approximate equation is established because c kt is the compression of ( s t , ˆ a t , h t ) .
p ( b kt | g kt , c kt ) and p ( g kt | c kt ) means the policy for bonuses and goals (Implementation details can be found in Appendix D.2.1).
4. F OLLOWER A CTION A BSTRACTION P OLICY G RADIENT
These methods mentioned above are fully implemented can enhance the performance dramatically.
But when facing the RL-based followers, the SMG is still hard to converge.
This is because in SMG, the policies of the leader and followers are always changing depending on other agents’ performance.
To guarantee convergence, the leader can only update its policy when the followers reach (or are near to) the best response policy (Fiez et al., 2019).
However, when the followers are RL-based agents, there is no way to ensure the followers’ policies are (near) the best response policies in large-scale
SMG and the commonly-seen idea is to provide enough training time but it is unbearable in practice due to the limitation of computing power (Mguni et al., 2019).
To accelerate the training process, inspired by the action abstraction approach which is commonlyseen in Poker (Brown & Sandholm, 2019; Tuyls et al., 2018) and action abstraction RL (Chandak et al., 2019), we collect the followers’ primitive actions sharing the same properties together as a meta policy.
Then, the followers only need to select the meta action to make a decision.
Therefore, the original game is converted into a meta game, which is easy to solve.
Speciﬁcally, we deﬁne the policy for the k -th follower as:
[Equation]
s, z ) , where ˆ s = (cid:104) s, ω k (cid:105) is the augmented state for the follower (the combination of current state and the leader’s action).
π kmeta ( z | ˆ s ) is the meta policy for the k -th follower and z is the high-level (meta) action.
We hypothesize that the lower-level policy (the policy to choose the primitive actions) is already known (rule-based) and deterministic, i.e., π klower ( a k | ˆ s, z ) = 1 .
For instance, given the example of the navigation task, the π kmeta can be the selection to which landmark to explore while π klower is a speciﬁc route planning algorithm (such as Dijkstra Algorithm).
Based on this assumption, we can design a novel policy gradient to train the meta policy: ∇ λ k J k ∝ E (cid:2) ∇
[Equation]
log π kmeta ( z | ˆ s )
R k (cid:3) , where λ k is the parameter for meta-policy π kmeta (Details can be found in Lemma 3).
4. L OSS F UNCTIONS
In this section, we discuss how to design the leader’s and followers’ loss functions.
Loss
Functions for the Leaders.
The basic structure for the leader is the actor-critic structure (Sutton & Barto, 1998).
We ﬁnd that adding regularizers can enhance the leader’s performance and we implement the maximum entropy for the leader’s policy function as well as the L2 regularization for the termination function, i.e., L enp = − (cid:80) k (cid:80) ω k µ k ( ω k | s, h ) log µ k ( ω k | s, h ) and L reg =
β 2 .
We also use imitation learning to learn the predicted action function p k .
Following the same logic of (Shu & Tian, 2019), two baseline functions φ g ( c t ) and φ b ( c t ) are also introduced to further reduce the variance.
Details can be found in Appendix B.
Loss Functions for the RL-Based Followers.
The basic structure for each follower is also based on the actor-critic structure.
We leverage the action abstraction policy gradient as we mentioned above.
The learning rate between the leader and follower should satisfy the two time-scale principle (Roughly speaking, the leader learns slower than the follower(s)), similar to (Borkar, 1997).
Details can be found in Appendix B and the pseudo-code can be found in Appendix C.
E XPERIMENTAL R ESULTS 5. S ETUP
[Figure]
Tasks.
We evaluate the following tasks to testify the performance of our proposed method.
All of these tasks are based on SMG mentioned above.
(1) resource collections: each follower collects three types of resources including its preferred one and the leader can choose two bonuses levels (Shu & Tian, 2019); (2) multi-bonus resources collections: based on (1), the leader can choose four bonuses levels; (3) modiﬁed navigation: followers are required to navigate some landmarks and after one of the landmarks is reached, the reached landmark disappears and new landmark will appear randomly.
(4) modiﬁed predator-prey: followers are required to capture some randomly moving preys, prizes will be given after touching them.
Both (3) and (4) are based on (Lowe et al., 2017) and we modify them into our SMG setting.
Moreover, to increase the difﬁculty, in each episode, the combinations of the followers will change, i.e., in each task, there are 40 different followers and at each episode, we randomly choose some followers to play the game.
More details can be found in Appendix D.
Baselines & Ablations.
To evaluate our method, we compare a recently proposed method as our baseline: M 3 RL (Shu & Tian, 2019).
We do not include other baselines because other methods cannot be used in our problems, as justiﬁed in (Shu & Tian, 2019).
For the ablations of the leader part, we choose: (1) ours: the full implementation of our method.
(2) ours w/o EBPG: removing the event-based policy gradient part; (3) ours w/o Attention: replacing follower-speciﬁed attention model by the original attention model mentioned in (Shu & Tian, 2019).
For the follower part, we choose (a) with rule-based follower (b) with vanilla RL-based follower, and (c) with action abstraction RL-based follower to testify the ability of our methods when facing different followers.
Hyper-Parameters.
Our code is implemented in Pytorch (Paszke et al., 2017).
If no special mention, the batch size is 1 (online learning).
Similar to (Shu & Tian, 2019), we set the learning rate as 0 .
001 for the leader’s critic and followers while 0 .
for the leader’s policy.
The optimization algorithm is Adam (Kingma & Ba, 2014).
Our method takes less than two days to train on a NVIDIA
Geforce GTX 1080Ti GPU in each experiment.
For the loss function, we set the λ 1 = 0 .
01 and λ 2 = 0 .
The total training episode is 250 ,for all the tasks (including both the rule-based followers and the RL-based followers).
To encourage exploration, we use the ι -greedy 2 .
For the leader, the exploration rate is set to 0 .
1 and slightly decreases to zero ( 5000 episode).
For the followers, the exploration rate for each agent is always 0 .(except
for the noise experiments).
5. L EARNING E FFICIENCY
[Table]
The quantitative results with different tasks are shown in Figs.& 4.
For the rule-based followers , from Fig. 3, we ﬁnd that our method outperforms the state-of-the-art method in all the tasks, showing that our method is sample efﬁcient and fast to coverage.
There is an interesting phenomenon that in the task of multi-bonus resource collections and navigation, only our method obtains a positive reward, indicating that our method can work well in complicated environments.
For ablations, we can see that ours w/o attention and ours w/o EBPG are worse than ours, representing these components do enhance the performance.
For the RL-based followers , from Fig. 4, we observe that when facing the
RL-based method with action abstraction
, our approach outperforms the baseline method in all the tasks (in predator-prey game, the reward for ours is twice as that of the state-of-the-art).
We also ﬁnd that without action abstraction, the reward is less than zero , revealing that the abstraction does play a crucial role in stabilizing training.
5. R OBUSTNESS
This experiment is to evaluate whether our method is robust to the noise, i.e., the follower randomly takes actions.
We make this experiment by introducing noise into the follower decision.
From
Table 1, we can ﬁnd that our method reaches a higher total reward (more than 5 ) among all the environment with noise than the state-of-the-art, indicating that our method is robust to the noise.
We also observe that the total reward for the baseline method becomes lower with the increase of the noise while our method is more robust to the change.
Moreover, for the incentive (the total gain), we ﬁnd that our method gains much more incentive than the state-of-the-art method, showing that our method coordinates have a better coordination the followers than the state-of-the-art method.
5. M ORE E XPERIMENTS
We also do a substantial number of experiments.
However, due to the space limitation, we can only provide some results here: (1) The total incentives: incentive can reveal the performance of successful rate interacting with the followers.
Our method outperforms the state-of-the-art method, indicating that our method has a better ability to interact with the followers.
(2) Sparse EBPG: we compare the performance gap between sparse EBPG and (dense) EBPG.
This results show that the sparse one is worse than the dense one, supporting the assumption that the dense signal can improve the sample efﬁciency.
(3) Visualizing attention: We visualize the attention module to ﬁnd what it actually learns and the result indicates that our attention mechanism does capture the followers whom the leader needs to assign bonuses to.
(4) Two time-scale training: We testify whether our two timescale training scheme works and the ablation shows that this scheme does play an important role in improving the performance of both the leader and the followers.
The committing interval: We observe that the dynamic committing interval (our method) performs better than the one with ﬁxed committing intervals.
(6) Reward for RL-based followers: we show the reward for the followers, which can provide the situation of the followers.
The result represents that our method aids the followers to gain more than the state-of-the-art method.
(7) Number of RL-based followers: ﬁnally, we testify our method in cases with different number of RL-based followers.
The result shows that our method always performs well.
The full results can be found in Appendix D.
C ONCLUSION R EMARKS
This paper proposes a novel RL training scheme for
Stackelberg Markov Games with single leader and multiple self-interested followers, which considers the leader’s long term decision process and complicated interaction between followers with three contributions
1) To consider the long-term effect of the leader’s behavior, we develop an event-based policy gradient for the leader’s policy. 2)
To predict the followers’ behaviors and make accurate response to their behaviors, we exploit the leader-follower consistency to design a novel follower-aware module and follower-speciﬁc attention mechanism.
We propose an action abstraction-based policy gradient algorithm to accelerate the training process of followers.
Experiments in resource collections, navigation, and predator-prey game reveal that our method outperforms the state-of-the-art methods dramatically.
We are willing to highlight that SMGs contribute to the RL (especially MARL) community with three key aspects: 1).
As we mentioned in the Introduction, most of the existing MARL methods assume that all the agents are willing to sacriﬁce themselves to maximize the total rewards, which is not true in many real-world non-cooperative scenarios.
On the contrary, our proposed method realistically assumes that agents are self-interested.
Thus, SMGs provide a new scheme focusing more on the self-interested agents.
We think this aspect is the most signiﬁcant contribution to the
RL community. 2).
The SMGs can be regarded as the multi-agent system with different roles (the leader and the followers)
(Wilson et al., 2008) and our method provides a solution to that problem.
Our methods also contribute to the hierarchical RL, i.e., it provides a non-cooperative training scheme between the high-level policy (the leaders) and the low-level policy (the followers), which plays an important role when the followers are self-interested.
Moreover, our EBPG also propose an novel policy gradient method for the temporal abstraction structure.
There are several directions we would like to investigate to further extend our SMG model: i) we will consider multiple cooperative/competitive leaders and multiple self-interested followers, which is the case in the labor market, ii) we will consider multi-level leaders, which is the case in the hierarchical organizations and companies and iii) we will consider the adversarial attacks to our
SMG model, which may induce extra cost to the leader for efﬁcient coordination.
We believe that our work is a preliminary step towards a deeper understanding of the leader-follower scheme in both research and the application to society.
B L OSS F UNCTIONS B. L EADER L OSS F UNCTIONS
We add a baseline function to reduce the variance of the event-based policy gradient for the leader.
We adopt the idea of successor representation (Rabinowitz et al., 2018; Shu & Tian, 2019) as two expected baseline functions: ˆ φ g ( c t ) and ˆ φ b ( c t ) .
For the gain baseline function:
[Equation]
For the bonus-based baseline function:
[Equation]
Two baseline neural network functions with parameters ˆ θ g
and ˆ θ b are trained through minimizing the mean square error:
[Equation]
Where c t is the attention-based latent variable
[Equation]
To this end
[Equation]
the gradient for the leader can be formulated as:
[Equation]
Where
[Equation]
are the baseline policy gradients.
We also leverage the imitation learning to learn the action probability function p k (cid:16) a kt | s t , h kt , ˆ θ I (cid:17) similar to (Shu & Tian, 2019), where θ I is the parameters for follower-aware module:
[Equation]
The history encoder, the state encoder and the attention module are updated with the leader’s policy gradient end-to-end.
B. F OLLOWER L OSS F UNCTIONS
The follower policy gradient is from Lemma 3 and we ﬁnd that adding the history h k can improve the performance:
[Equation]
and the learning rate of the follower α and the leader β satisfy (cid:88) t
[Equation]
t
[Equation]
=
[Equation]
, (cid:88)
[Equation]
Which indicates the follower’s learning rate is much higher than the leader.
ˆ β is the learning rate for the leader’s critic function, which is the same as α .
Followers do not have critic function.
C A LGORITHM Algorithm
Input: The initialized leader’s parameters
Output: Well-trained leader and followers; 1 while not converge do if Rollout Stage then for do do
; end
// transition function
end do
// the follower’s parameters end θ b ← ˆ θ b − ˆ β ∇ ˆ θ b L baseline ; // the critic’s parameters ˆ θ I ←
I βL
∇ θ I L IL ; // the follower-aware module’s parameters
L policyl ; // the termination function’s parameters
− β ∇ ϑ L policyl ; // the parameters of leader’s policy; 21 end
Algorithm
Input: Leader’s policy µ
Output: goals g and bonuses b ; 1 for k ∈ N do sample the termination function β k t ; if β kt terminates previous leader’s action ω kt − 1 then choose new action ω kt ;
else ω kt = ω kt − 1 ; // maintain previous action.
[Figure]
[Figure]
D E XPERIMENT D ETAILS D. T ASKS D ETAILS
The illustration of experimental scenarios can be found in Figure 5.
Here we give some details about these environments:
Resource Collections.
This task is similar to (Shu & Tian, 2019), which is based on the scene that the leader and the followers collect some resources.
Each follower has its own preference which might be the same (or against) to the leader’s preference.
In order to make the followers obey the leader’s instruction, the leader should pay the followers bonuses.
There are total 4 types of resources and for different resources each agent has different preferences.
The leader owns two type of bonus (1 or 2) and 4 types of goals (each resource is a goal).
The number of leader is 1 while the number of followers is 4 .
Multi-Bonus Resource Collections.
This task is similar to Resource Collections .
Except that the leader can take 4 level bonuses (a bonus from 1 to 4) while each agent owns one skill.
The number of leader is 1 while the number of followers is 4 .
Modiﬁed Navigation.
This task is original from (Lowe et al., 2017).
We make some modiﬁcations here to make it suitable our SMG: the leader and the followers are going to navigate some landmarks.
Each follower has its own preference which might be the same (or against) to the leader’s preference.
When a landmark has been navigated, it disappears immediately and a new landmark will appear.
There are total 6 types of landmarks and for different landmarks, each agent has different preferences.
The leader owns two type of bonuses and 6 types of goals (each landmark is a goal).
The number of leader is 1 while the number of followers is 8 and the number of the landmarks is 6 .
Modiﬁed Predator-Prey.
This task is also original from (Lowe et al., 2017).
We make some modiﬁcation here to make it suitable our SMG: the leader and the followers are going to catch some preys.
Each follower has its own preference which might be the same (or against) to the leader’s preference.
In each step, whether a prey has been caught, it randomly chooses a direction to go.
Catching a prey will not make it disappear, which means that the preys can exist until the game ends.
There are total 8 types of preys and for different preys, each agent has different preferences.
The leader owns two type of bonuses and 8 types of goals (each prey is a goal).
The followers are faster than the preys.
The number of leader is 1 while the number of followers is 10 and the number of the landmarks is 8 .
Reward Design.
The rewards mentioned in Section 3 are the general forms.
Here, we deﬁne two speciﬁed forms of the leader and followers reward function in our experiments:
Leader Reward.
We deﬁne v g as the prize (utility) for ﬁnishing task g .
We set the reward function for the leader at step t as: r lt = (cid:80) g ∈G (cid:80) k ∈N I (cid:0) g = g kt (cid:1) (cid:0) I (cid:0) s kt =
s g (cid:1) v g − b kt (cid:1) , formulated by the total gain (cid:80) g ∈G (cid:80) k ∈N I (cid:0)
g = g kt (cid:1) I (cid:0) s kt =
s g (cid:1) v g (the total prizes got by the leader at time t )
minus the total payment − (cid:80) g ∈G (cid:80) k ∈N
I (cid:0) g = g kt (cid:1)
b kt (the total bonuses paid to the followers).
We should emphasize that our leader reward is total different from the (Shu & Tian, 2019): in their approaches, the leader changes its mind after signing a contract will not be punished.
To make it suitable to the real world, we modify the reward as the leader should pay the followers bonuses immediately after signing the contract and cannot get back if it gives up the contract.
Follower Reward.
For the followers, we set the reward for the k -th follower as: r kt = (cid:80) g r kg,t = (cid:0) u k,g I (cid:0) s kt =
s g (cid:1)
+ I (cid:0) g lt = g kt (cid:1) ×
b kt (cid:1) , where u k,g reveals the payoff of the k -th follower when ﬁnishing task g (the preference).
Speciﬁcally, r kt indicates that the follower can either follow the leader’s instruction or just do what it prefers to.
The followers will receive reward immediately after signing the contract (the leader and the followers achieve an agreement).
I (cid:0) s kt =
s g (cid:1) means that the follower ﬁnishes the task g at step t .
A penalty is added to the followers if the followers betray the leader (the followers and the leader sign the contract but the followers do not agree to work).
D. T RAINING D ETAILS D.2. N ETWORK D ESIGN
Our network is based on (Shu & Tian, 2019).
Some do not suit our method.
We do some modiﬁcation here: (1) We change the vanilla attention mechanism (sum/average all the history and action of each follower together) to a follower-speciﬁed one: each follower has a weight which indicates how important the follower is at the current step.
The output for g and b are changed into the sequential form, i.e., we ﬁrst calculate p ( g kt | c kt ) to get g kt , then based on g kt , we choose p ( b kt | g kt , c kt ) .
The history information h kt is compressed by the LSTM (Gers et al., 2000).
Leader’s Network.
For the history information, we are willing to stress that the history consists of two parts, one is the statistical information similar to (Shu & Tian, 2019) and the other is the past information given by the neural networks.
We leverage the LSTM with 128 hidden units and two fully-connected layers to compress the past information and the statistical information and obtain h kt .
Each layer contains 128 neurons.
For the state information, we encode it into a 128-dimension hidden state by a state encoder.
For resource collections task, the state encoder consist of a convolutional layer with 64 channels and kernels of 1 × 1 and two fully connected layers with 128 neurons.
For navigation and predator-prey task, we encode the state using two fully connected layers with 128 neurons.
For the follower-speciﬁc attention module, we ﬁrstly predict the ˆ a kt from the follower-aware module which contains an LSTM with 128 hidden units and two fully-connected layers with 128 neurons.
Secondly, we transform ˆ a kt , h kt and the output of the state encoder into A ( s t , ˆ a kt , h kt ) using a fully connected layer with 128 neurons.
The hidden vector is generated by concatenating all the
A ( s t , ˆ a kt , h kt ) .
Then the attention weight w is calculated using two fully connected layers with softmax activation.
The c t is calculated as a weighted summation of w and A ( · ) .
Finally, we obtain the k -th attention value c kt by concatenating c t and A ( s t , ˆ a kt , h kt ) .
For sequential graph-based model, we ﬁrstly calculate p ( g kt | c kt ) using two fully connected layers.
Each layer consists of 128 neuron units.
The input of layers is c kt and the output is g kt .
Then we obtain b kt by two fully connected layers.
The input of layers is the concatenation of c kt and g kt .
Follower’s Network .
We construct the RL-based followers by a state encoder and two fully connected layers.
Each fully connected layer contains 128 neurons.
For resource collections task, the state encoder consists of a convolutional layer with 64 channels and kernels of 1 × 1, two fully connected layers with 128 neurons and an LSTM with 128 hidden units.
For navigation and predatorprey task, we encode the state using two fully connected layers with 128 neurons and an LSTM with 128 hidden units.
Notice that the network structure of action-abstraction RL-based follower and non-action-abstraction RL-based follower is similar.
The only difference is that the output of the former is high-level action z and the output of the latter is low-level action a kt .
[Figure] [Figure]
n
[Figure]
e
[Figure]
D. M ORE E XPERIMENTAL R ESULTS D.3. T OTAL I NCENTIVE F UNCTION
Total incentive (income) functions R in = (cid:80) t
r lt + (cid:80) t (cid:80) k b kt can reveal how well the leader interacts with the followers; the higher the R in is, the more successful the coordination between the leader and the followers.
From Figure 6 and
[Figure]
comparing with the state-of-the-art method, we can see that our method far outperforms the state-of-the-art method, which reveals that our method does have a better ability to coordinate with the followers.
In all of the scenarios, without the EBPG, the performance of our method is worse than ours with EBPG.
Speciﬁcally, in some scenarios (e.g., multi-bonus resource collections, navigation), without the EBPG, the performance of our method is (or nearly) similar to the performance of M 3 RL, showing the effectiveness of our novel policy gradient.
Moreover, we can notice that in navigation environment, without follower-speciﬁed attention, the performance of our method diminishes rapidly, which implies that in some scenarios, attention does play an important role.
[Figure] [Figure]
D.3. R OBUST OF O UR M ETHOD
In this section
[Figure]
we are going to testify the robust of our method.
Speciﬁcally
[Figure]
we evaluate whether our method is robust to the noise.
We make this experiment by introducing noise into the follower decision.
For example, if we set the noise function as 30% , indicating that there is 30% probability that the followers will choose action randomly.
D.3. S PARSE E VENT - BASED P OLICY G RADIENT
This experiment testiﬁes whether the dense event-based policy gradient increases the leader’s performance comparing with the sparse event-based policy gradient.
We make ablations here: (1) Ours: the full structure of our method; (2) Sparse Event-Based Policy Gradient (sparse EBPG): the fully structure of ours except that the EBPG is replaced by sparse event-based policy gradient; (3) sparse
EBPG w/o attention: replacing the follower-speciﬁed attention mechanism by averaging the input features.
From Figure 8 and 9 we can ﬁnd that if the policy gradient is sparse, its performance is worse than the dense one, implying that the dense method does improve the leader’s performance.
There is also an interesting phenomenon that sparse EBPG with follower-speciﬁed attention mechanism performs better than that without, revealing that the attention can stabilize training when the training signal is sparse.
[Figure]
[Figure] [Figure]
- ers).
D.3. V ISUALIZING A TTENTION
Following the same logic of (Iqbal & Sha, 2019), we visualize the weight of the attention when the leader takes actions.
From Figure 10, we ﬁnd that the attention mechanism does learn to strongly attend to the followers that the leader needs to take actions.
The followers with leader’s commitment obtain much higher attention weight than others, showing that the attention module actually learn to identify the important followers while leader committing new action.
Thus, the attention mechanism does play an important role in improving the performance.
D.3. T WO T IME -S CALE U PDATING
In order to evaluate the performance of our two time-scale update scheme (TTSU), we do an ablation study as shown in Fig 11.
We can ﬁnd that the performance where the followers’ learning rate α ( 1 × 10 − 3 ) is much larger than the leader’s β ( 3 × 10 − 4 ) is better than the performance where the leader’s learning rate is similar to the followers ( 1 × 10 − 3 ).
Moreover, without TTSU, the reward curves of training methods become unstable, revealing that TTSU can stabilize the training process.
In fact, TTSU improves the rate of convergence and play an important role in improving performance
. D.3. C OMMITTING I NTERVAL
We evaluate the leader’s performance between static committing interval and our dynamic committing interval.
As shown in Figure 12, we observe that all the different ﬁxed committing intervals only change the rate of convergence and do not enhance the leader’s performance.
All the ﬁxed
[Figure] [Figure]
committing intervals are much worse than our dynamic committing approach, revealing the fact that our dynamic committing approach aids a lot in improving the leader’s performance used.
ModulesEBPG (cid:88) (cid:88) (cid:88) (cid:88)
[Table]
(cid:88) (cid:88) (cid:88) (cid:88)
[Table]
(cid:88) (cid:88) (cid:88) (cid:88)
[Table]
D.3. R EWARD FOR RL- BASED F OLLOWERS
We are interesting in the reward for the RL-based follower(s).
Intuitively, a well-performing leader can make the follower gain more.
As shown in Figure 13, the reward for RL follower is higher than
M 3 RL follower in all the tasks.
This represents the leader can coordinate the followers better and make them gain more reward than other methods, which forms a win-win strategy.
[Figure] [Figure]
D.3. N UMBER OF THE FOLLOWER AGENTS
Finally, we evaluate the leader’s performance with different number of RL-based followers.
As shown in Figure 15, we ﬁnd that our method outperforms the state-of-the-art method when facing different number of RL-based workers.
D.3. D IFFERENT COMBINATIONS STUDY
To further illustrate the performance of different combinations of our methods
[Figure]
we make an extra ablation here.
We choose the resource collections as the environments.
Our analysis is as follows:
For the RL based followers scenario, As shown in Table 2, we ﬁnd that the action abstraction policy gradient is very important to converge.
Additionally, adding different modules can improve the performance and the method with all the modules reach the highest reward than other combinations.
The contribution ranking for each module is: Action abstraction policy gradient > EBPG > Leaderfollower consistency.
[Figure]