T RUST R EGION P OLICY O PTIMISATION IN M ULTI -A GENT R EINFORCEMENT L EARNING Anonymous authors
Paper under double-blind review
A BSTRACT
Trust region methods rigorously enabled reinforcement learning (RL) agents to learn monotonically improving policies, leading to superior performance on a variety of tasks.
Unfortunately, when it comes to multi-agent reinforcement learning (MARL), the property of monotonic improvement may not simply apply; this is because agents, even in cooperative games, could have conﬂicting directions of policy updates.
As a result, achieving a guaranteed improvement on the joint policy where each agent acts individually remains an open challenge.
In this paper, we extend the theory of trust region learning to cooperative MARL.
Central to our ﬁndings are the multi-agent advantage decomposition lemma and the sequential policy update scheme .
Based on these, we develop Heterogeneous-Agent
Trust Region Policy Optimisation (HATPRO) and Heterogeneous-Agent Proximal Policy Optimisation (HAPPO) algorithms.
Unlike many existing MARL algorithms, HATRPO/HAPPO do not need agents to share parameters, nor do they need any restrictive assumptions on decomposibility of the joint value function.
Most importantly, we justify in theory the monotonic improvement property of
HATRPO/HAPPO.
We evaluate the proposed methods on a series of
Multi-Agent
MuJoCo and StarCraftII tasks
Results show that HATRPO and HAPPO signiﬁcantly outperform strong baselines such as IPPO
MAPPO and MADDPG on all tested tasks, thereby establishing a new state of the art 1 .
I NTRODUCTION
Policy gradient (PG) methods have played a major role in recent developments of reinforcement learning (RL) algorithms (Silver et al., 2014; Schulman et al., 2015a;
Haarnoja et al., 2018).
Among the many PG variants, trust region learning (Kakade & Langford, 2002), with two typical embodiments of Trust Region Policy Optimisation (TRPO) (Schulman et al., 2015a) and Proximal Policy
Optimisation (PPO) (Schulman et al., 2017) algorithms, offer supreme empirical performance in both discrete and continuous RL problems (Duan et al., 2016; Mahmood et al., 2018).
The effectiveness of trust region methods largely stems from their theoretically-justiﬁed policy iteration procedure.
By optimising the policy within a trustable neighbourhood of the current policy, thus avoiding making aggressive updates towards risky directions, trust region learning enjoys the guarantee of monotonic performance improvement at every iteration.
In multi-agent reinforcement learning (MARL) settings (Yang & Wang, 2020), naively applying policy gradient methods by considering other agents as a part of the environment can lose its effectiveness.
This is intuitively clear: once a learning agent updates its policy, so do its opponents; this however changes the loss landscape of the learning agent, thus harming the improvement effect from the PG update.
As a result, applying independent PG updates in MARL offers poor convergence property (Claus & Boutilier, 1998).
To address this, a learning paradigm named centralised training with decentralised execution (CTDE) (Lowe et al., 2017b; Foerster et al., 2018; Zhou et al., 2021) was developed.
In CTDE, each agent is equipped with a joint value function which, during training, has access to the global state and opponents’ actions.
With the help of the centralised value function that accounts for the non-stationarity caused by others, each agent adapts its policy parameters accordingly.
As such, the CTDE paradigm allows a straightforward extension of single-agent
PG theorems (Sutton et al., 2000; Silver et al., 2014) to multi-agent scenarios (Lowe et al., 2017b;
Kuba et al., 2021; Mguni et al., 2021).
Consequently, fruitful multi-agent policy gradient algorithms have been developed (Foerster et al., 2018; Peng et al., 2017; Zhang et al., 2020; Wen et al., 2018; 2020; Yang et al., 2018).
Unfortunately, existing CTDE methods offer no solution of how to perform trust region learning in
MARL.
Lack of such an extension impedes agents from learning monotonically improving policies in a stable manner.
Recent attempts such as IPPO (de Witt et al., 2020a) and MAPPO (Yu et al., 2021) have been proposed to ﬁll such a gap; however, these methods are designed for agents that are homogeneous (i.e., sharing the same action space and policy parameters), which largely limits their applicability and potentially harm the performance.
As we show in Proposition 1 later, parameter sharing could suffer from an exponentially-worse suboptimal outcome.
On the other hand, although
IPPO/MAPPO can be practically applied in a non-parameter sharing way, it still lacks the essential theoretical property of trust region learning, which is the monotonic improvement guarantee.
In this paper, we propose the ﬁrst theoretically-justiﬁed trust region learning framework in MARL.
The key to our ﬁndings are the multi-agent advantage decomposition lemma and the sequential policy update scheme .
With the advantage decomposition lemma, we introduce a multi-agent policy iteration procedure that enjoys the monotonic improvement guarantee.
To implement such a procedure, we propose two practical algorithms: Heterogeneous-Agent Trust Region Policy Optimisation (HATRPO) and Heterogeneous-Agent Proximal Policy Optimisation (HAPPO).
HATRPO/HAPPO adopts the sequential update scheme, which saves the cost of maintaining a centralised critic for each agent in CTDE.
Importantly, HATRPO/HAPPO does not require homogeneity of agents, nor any other restrictive assumptions on the decomposibility of the joint Q-function (Rashid et al., 2018).
We evaluate HATRPO and HAPPO on benchmarks of StarCraftII and Multi-Agent MuJoCo against strong baselines such as MADDPG (Lowe et al., 2017a), IPPO (de Witt et al., 2020b) and MAPPO (Yu et al., 2021); results clearly demonstrate its state-of-the-art performance across all tested tasks.
P RELIMINARIES
In this section, we ﬁrst introduce problem formulation and notations for MARL, and then brieﬂy review trust region learning in RL and discuss the difﬁculty of extending it to MARL.
We end by surveying existing MARL work that relates to trust region methods and show their limitations.
2. C OOPERATIVE MARL P ROBLEM F ORMULATION AND N OTATIONS
We consider a Markov game (Littman, 1994), which is deﬁned by a tuple (cid:104)N , S , A , 𝑃, 𝑟, 𝛾 (cid:105) .
Here,
N = { 1 , . . .
, 𝑛 } denotes the set of agents, S is the ﬁnite state space, A = (cid:206)
𝑛𝑖
=
1 A 𝑖 is the product of ﬁnite action spaces of all agents, known as the joint action space, 𝑃 : S ×
A × S → [ 0 , 1 ] is the transition probability function, 𝑟 : S ×
A → ℝ is the reward function, and 𝛾 ∈ [ 0 , 1 ) is the discount factor.
The agents interact with the environment according to the following protocol: at time step 𝑡 ∈ ℕ , the agents are at state s 𝑡 ∈ S ; every agent 𝑖 takes an action a 𝑖𝑡 ∈
A 𝑖 , drawn from its policy 𝜋 𝑖 (·| s 𝑡 ) , which together with other agents’ actions gives a joint action a 𝑡 = ( a 1 𝑡 , . . .
, a 𝑛𝑡 ) ∈ A , drawn from the joint policy π (·| s 𝑡 ) =
(cid:206)
𝑛𝑖
= 1 𝜋 𝑖 (· 𝑖 | s 𝑡 ) ; the agents receive a joint reward r 𝑡 = 𝑟 ( s 𝑡 , a 𝑡 ) ∈ ℝ , and move to a state s 𝑡
+ 1 with probability 𝑃 ( s 𝑡 + 1 | s 𝑡 , a 𝑡 ) .
The joint policy π , the transition probabililty function 𝑃 , and the initial state distribution 𝜌 0 , induce a marginal state distribution at time 𝑡 , denoted by 𝜌 𝑡 π .
We deﬁne an (improper) marginal state distribution 𝜌 π (cid:44) (cid:205) ∞
𝑡 =
𝛾 𝑡 𝜌 𝑡 π .
The state value function and the state-action value function are deﬁned: 𝑉 π ( 𝑠 ) (cid:44) 𝔼 a 0: ∞ ∼ π , s 1: ∞ ∼ 𝑃 (cid:2) (cid:205) ∞
𝑡 =
𝛾 𝑡 r 𝑡 (cid:12)(cid:12) s
=
𝑠 (cid:3) and 𝑄 π ( 𝑠, a ) (cid:44) 𝔼 s 1: ∞ ∼ 𝑃, a 1: ∞ ∼ π (cid:2) (cid:205) ∞
𝑡 =
𝛾 𝑡 r 𝑡 (cid:12)(cid:12) s
= 𝑠, a 0 = a (cid:3) .
The advantage function is written as 𝐴 π
( 𝑠, a ) (cid:44) 𝑄
π
( 𝑠, a )− 𝑉 π ( 𝑠 ) .
In this paper, we consider a fully-cooperative setting where all agents share the same reward function, aiming to maximise the expected total reward:
[Equation]
Throughout this paper, we pay close attention to the contribution to performance from different subsets of agents.
Before proceeding to our methods, we introduce following novel deﬁnitions.
Deﬁnition 1.
Let 𝑖 1: 𝑚 denote an ordered subset { 𝑖 1 , . . .
, 𝑖 𝑚 } of N , and let − 𝑖 1: 𝑚 refer to its complement.
We write 𝑖 𝑘 when we refer to the 𝑘 th agent in the ordered subset.
Correspondingly, the multi-agent state-action value function is deﬁned as
[Equation]
and for disjoint sets 𝑗 1: 𝑘 and 𝑖 1: 𝑚 , the multi-agent advantage function is
[Equation]
Hereafter, the joint policies π
= ( 𝜋 1 , . .
𝜋 𝑛 ) and ¯ π = ( ¯ 𝜋 1 , . .
, ¯ 𝜋 𝑛 ) shall be thought of as the “current”, and the “new” joint policy that agents update towards, respectively.
2. T RUST R EGION A LGORITHMS IN R EINFORCEMENT L EARNING
Trust region methods such as TRPO (Schulman et al., 2015a) were proposed in single-agent RL with an aim of achieving a monotonic improvement of J ( 𝜋 ) at each iteration.
Formally, it can be described by the following theorem.
Theorem 1.
(Schulman et al., 2015a, Theorem 1) Let 𝜋 be the current policy and ¯ 𝜋 be the next candidate policy.
We deﬁne
[Equation]
·| 𝑠 ) , ¯ 𝜋 (·| 𝑠 )) .
Then the inequality of
[Equation]
holds, where 𝐶 =
[Equation]
𝛾
[Equation]
The above theorem states that as the distance between the current policy 𝜋 and a candidate policy ¯ 𝜋 decreases, the surrogate 𝐿 𝜋 ( ¯ 𝜋 ) , which involves only the current policy’s state distribution, becomes an increasingly accurate estimate of the actual performance metric J ( ¯ 𝜋 ) .
Based on this theorem, an iterative trust region algorithm is derived; at iteration 𝑘 + 1 , the agent updates its policy by
[Equation]
Such an update guarantees a monotonic improvement of the policy, i.e., J ( 𝜋 𝑘 + 1 ) ≥ J ( 𝜋 𝑘 ) .
To implement this procedure in practical settings with parameterised policies 𝜋 𝜃 , Schulman et al.
(2015a) approximated the KL-penalty with a KL-constraint, which gave rise to the TRPO update of
[Equation]
subject to
[Equation]
At each iteration 𝑘 + 1 , TRPO constructs a KL-ball 𝔅 𝛿 ( 𝜋 𝜃 𝑘 ) around the policy 𝜋 𝜃 𝑘 , and optimises 𝜋 𝜃 𝑘 + 1 ∈ 𝔅 𝛿 ( 𝜋 𝜃 𝑘 ) to maximise 𝐿 𝜋 𝜃𝑘 ( 𝜋 𝜃 ) .
By Theorem 1, we know that the surrogate objective 𝐿 𝜋 𝜃𝑘 ( 𝜋 𝜃 ) is close to the true reward
J ( 𝜋 𝜃 ) within 𝔅 𝛿 ( 𝜋 𝜃 𝑘 ) ; therefore, 𝜋 𝜃 𝑘 leads to improvement.
Furthermore, to save the cost on 𝔼 s ∼ 𝜌 𝜋𝜃𝑘 (cid:2) 𝐷 KL ( 𝜋 𝜃 𝑘 , 𝜋 𝜃 ) (cid:3) when computing Equation (3), Schulman et al.
(2017) proposed an approximation solution to TRPO that uses only ﬁrst order derivatives, known as PPO.
PPO optimises the policy parameter 𝜃 𝑘 + 1 by maximising the PPO-clip objective of 𝐿 PPO 𝜋 𝜃𝑘 ( 𝜋 𝜃
[Equation]
= 𝔼
[Equation]
∼ 𝜌 𝜋𝜃𝑘
[Equation]
a ∼ 𝜋 𝜃𝑘 (cid:20)
[Equation]
(cid:18) 𝜋 𝜃 ( a | s ) 𝜋 𝜃 𝑘 ( a | s
[Equation]
𝐴 𝜋 𝜃𝑘 ( s
[Equation]
a
[Equation]
, clip
[Equation]
The clip operator replaces the ratio 𝜋 𝜃 ( a | s ) 𝜋 𝜃𝑘 ( a | s ) with 1 + 𝜖 or 1 − 𝜖 , depending on whether or not the ratio is beyond the threshold interval.
This effectively enables PPO to control the size of policy updates.
2. L IMITATIONS OF E XISTING T RUST R EGION M ETHODS IN MARL
Extending trust region methods to MARL is highly non-trivial.
One naive approach is to equip all agents with one shared set of parameters and use agents’ aggregated trajectories to conduct policy optimisation at every iteration.
This approach was adopted by MAPPO (Yu et al., 2021) in which the policy parameter 𝜃 is optimised by maximising the objective of
[Equation]
Unfortunately, this simple approach has signiﬁcant drawbacks.
An obvious demerit is that parameter sharing requires that all agents have identical action spaces, i.e. , A 𝑖 =
A 𝑗 , ∀ 𝑖, 𝑗 ∈ N , which limits the class of MARL problems to solve.
Importantly, enforcing parameter sharing is equivalent to putting a constraint 𝜃 𝑖 =
𝜃 𝑗 , ∀ 𝑖, 𝑗 ∈ N on the joint policy space.
In principle, this can lead to a suboptimal solution.
To elaborate, we demonstrate through an example in the following proposition.
Proposition 1.
Let’s consider a fully-cooperative game with an even number of agents 𝑛 , one state, and the joint action space { 0 , 1 } 𝑛 , where the reward is given by 𝑟 ( 0 𝑛 / 2 , 1 𝑛 / 2 )
=
𝑟 ( 1 𝑛 / 2 , 0 𝑛 / 2 ) = 1 , and 𝑟 ( a 1: 𝑛 ) = 0 for all other joint actions.
Let J ∗ be the optimal joint reward, and J ∗ share be the optimal joint reward under the shared policy constraint.
Then
[Figure]
For proof see Appendix B.
In the above example, we show that parameter sharing can lead to a suboptimal outcome that is exponentially worse with the increasing number of agents.
We also provide an empirical veriﬁcation of this proposition in Appendix F.
Apart from parameter sharing, a more general approach to extend trust region methods in
MARL is to endow all agents with their own parameters, and at each iteration 𝑘 + 1 , agents construct trust regions of { 𝔅 𝛿 ( 𝜋 𝑖𝜃 𝑖𝑘 )} 𝑖 ∈N , and optimise their objectives { 𝐿 π
[Figure]
𝑘 ( 𝜋 𝑖𝜃
[Figure]
𝑖
[Figure]
𝑖𝑘
[Figure]
𝑖
[Figure]
We initialise agents’ 1-D Gaussian policies with 𝜇 1 = − 0 .
25 , 𝜇 2 = 0 .
Orange intervals represent the KL-ball of 𝛿 = 0 .
Individual trust region update (red) jointly decreases the performance, whereas our sequential updates (blue) lead to improvement.
Admittedly, this approach can still be supported by the current
MAPPO implementation (Yu et al., 2021) if one turns off parameter sharing, thus distributing the summation in Equation (5) to all agents.
However, such an approach cannot offer a rigorous guarantee of monotonic improvement during training.
In fact, agents’ local improvements in performance can jointly lead to a worse outcome.
For example, in Figure 1, we design a single-state differential game where two agents draw their actions from Gaussian distributions with learnable means 𝜇 1 , 𝜇and unit variance, and the reward function is 𝑟 ( 𝑎 1 , 𝑎 2 ) =
𝑎 1 𝑎 2 .
The failure of MAPPO-style approach comes from the fact that, although the reward function increases in each of the agents’ (one-dimensional) update directions, it decreases in the joint (two-dimensional) update direction.
Having seen the limitations of existing trust region methods in
MARL, in the following sections, we ﬁrst introduce a multiagent policy iteration procedure that enjoys theoreticallyjustiﬁed monotonic improvement guarantee.
To implement this procedure, we propose HATRPO and HAPPO algorithms, which offer practical solutions to apply trust region learning in MARL without the necessity of assuming homogeneous agents while still maintaining the monotonic improvement property.
M ULTI -A GENT T RUST R EGION L EARNING
The purpose of this section is to develop a theoretically-justiﬁed trust region learning procedure in the context of multi-agent learning.
In Subsection 3.1, we present the policy iteration procedure with monotonic improvement guarantee, and in Subsection 3.2, we analyse its properties during training and at convergence.
Throughout the work, we make the following regularity assumptions.
Assumption 1.
There exists 𝜂 ∈ ℝ , such that 0 < 𝜂 (cid:28) 1 , and for every agent 𝑖 ∈ N , the policy space Π 𝑖 is 𝜂 -soft; that means that for every 𝜋 𝑖 ∈ Π 𝑖 , 𝑠 ∈ S , and 𝑎 𝑖 ∈
A 𝑖 , we have 𝜋 𝑖 ( 𝑎 𝑖 | 𝑠 ) ≥ 𝜂 .
3. T RUST R EGION L EARNING IN MARL WITH M ONOTONIC I MPROVEMENT G UARANTEE
We start by introducing a pivotal lemma which shows that the joint advantage function can be decomposed into a summation of each agent’s local advantages.
Importantly, this lemma offers a critical intuition behind the sequential policy-update scheme that our algorithms later apply.
Lemma 1 (Multi-Agent Advantage Decomposition) .
In any cooperative Markov games, given a joint policy π , for any state 𝑠 , and any agent subset 𝑖 1: 𝑚 , the below equations holds.
[Equation]
For proof see Appendix C.2.
Notably, Lemma 1 holds in general for cooperative Markov games, with no need for any assumptions on the decomposibility of the joint value function such as those in
VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2018) or Q-DPP (Yang et al., 2020).
Lemma 1 indicates an effective approach to search for the direction of performance improvement (i.e., joint actions with positive advantage values) in multi-agent learning.
Speciﬁcally, let agents take actions sequentially by following an arbitrary order 𝑖 1: 𝑛 , assuming agent 𝑖 1 takes an action ¯ 𝑎 𝑖such that 𝐴 𝑖 1 ( 𝑠, ¯ 𝑎 𝑖 1 ) > 0 , and then, for the rest 𝑚 = 2 , . . .
, 𝑛 , the agent 𝑖 𝑚 takes an action ¯ 𝑎 𝑖 𝑚 such that 𝐴 𝑖 𝑚
( 𝑠, ¯ a 𝑖 1: 𝑚 − 1 , ¯ 𝑎 𝑖 𝑚 ) > 0 .
For the induced joint action ¯ a , Lemma 1 assures that 𝐴 π θ ( 𝑠, ¯ a ) is positive, thus the performance is guaranteed to improve.
To formally extend the above process into a policy iteration procedure with monotonic improvement guarantee, we need the following deﬁnitions.
Deﬁnition 2.
Let π be a joint policy, ¯ π 𝑖 1: 𝑚 − 1 = (cid:206)
𝑚 − 1 𝑗 = 1 ¯ 𝜋 𝑖
𝑗 be some other joint policy of agents 𝑖 1: 𝑚 − 1 , and ˆ 𝜋 𝑖 𝑚 be some other policy of agent 𝑖 𝑚 .
Then
[Equation]
Note that, for any ¯ π 𝑖 1: 𝑚 − 1 , we have
[Equation]
Building on Lemma 1 and Deﬁnition 2, we can ﬁnally generalise Theorem 1 of TRPO to MARL.
Lemma 2.
Let π be a joint policy.
Then, for any joint policy ¯ π , we have
[Equation]
For proof see Appendix C.2.
This lemma provides an idea about how a joint policy can be improved.
Namely, by Equation (6), we know that if any agents were to set the values of the above summands 𝐿 𝑖 1:
[Equation]
( ¯ π
[Equation]
: 𝑚
[Equation]
1 , ¯ 𝜋 𝑖 𝑚 ) − 𝐶𝐷 maxKL ( 𝜋 𝑖 𝑚 , ¯ 𝜋 𝑖 𝑚 ) by sequentially updating their policies, each of them can always make its summand be zero by making no policy update (i.e., ¯ 𝜋 𝑖 𝑚 =
𝜋 𝑖 𝑚 ) .
This implies that any positive update will lead to an increment in summation.
Moreover, as there are 𝑛 agents making policy updates, the compound increment can be large, leading to a substantial improvement.
Lastly, note that this property holds with no requirement on the speciﬁc order by which agents make their updates; this allows for ﬂexible scheduling on the update order at each iteration.
To summarise, we propose the following Algorithm 1.
We want to highlight that the algorithm is
-(joint)action pairs ( 𝑠, a ) .
Compute 𝜖 = max 𝑠, a | 𝐴 π 𝑘 ( 𝑠, a )| and 𝐶 = 4 𝛾𝜖 ( 1 − 𝛾 ) 2 .
Draw a permutaion 𝑖 1: 𝑛 of agents at random.
6: for 𝑚 = 1 : 𝑛 do 7:
Make an update 𝜋 𝑖 𝑚 𝑘 + 1 = arg max 𝜋 𝑖𝑚 (cid:104) 𝐿 𝑖 1: 𝑚 π 𝑘 (cid:16) π 𝑖 1: 𝑚 − 1 𝑘 + 1 , 𝜋 𝑚 (cid:17) − 𝐶𝐷 max KL ( 𝜋 𝑖 𝑚 𝑘 , 𝜋 𝑖 𝑚 ) (cid:105) .
8: end for 9: end for markedly different from naively applying the TRPO update, i.e. , Equation (3), on the joint policy of all agents.
Firstly, our Algorithm 1 does not update the entire joint policy at once, but rather update each agent’s individual policy sequentially.
Secondly, during the sequential update, each agent has a unique optimisation objective that takes into account all previous agents’ updates, which is also the key for the monotonic improvement property to hold.
3. T HEORETICAL A NALYSIS
Now we justify by the following theorm that Algorithm 1 enjoys monotonic improvement property.
Theorem 2.
A sequence ( π 𝑘 ) ∞ 𝑘 = 0 of joint policies updated by Algorithm 1 has the monotonic improvement property, i.e., J ( π 𝑘 + 1 ) ≥ J ( π 𝑘 ) for all 𝑘 ∈ ℕ .
For proof see Appendix C.2.
With the above theorem, we ﬁnally claim a successful introduction of trust region learning to MARL, as this generalises the monotonic improvement property of TRPO.
Moreover, we take a step further to study the convergence property of Algorithm 1.
Before stating the result, we introduce the following solution concept.
Deﬁnition 3.
In a fully-cooperative game, a joint policy π ∗ =
( 𝜋 1 ∗ , . . .
, 𝜋 𝑛 ∗ ) is a Nash equilibrium (NE) if for every 𝑖 ∈ N , 𝜋 𝑖 ∈ Π 𝑖 implies J ( π ∗ ) ≥ J (cid:0) 𝜋 𝑖 , π − 𝑖
∗
cid:1) .
NE (Nash, 1951) is a well-established game-theoretic solution concept.
Deﬁnition 3 characterises the equilibrium point at convergence for cooperative MARL tasks.
Based on this, we have the following result that describes Algorithm 1’s asymptotic convergence behaviour towards NE.
Proposition 2.
Supposing in Algorithm 1 any permutation of agents has a ﬁxed non-zero probability, a sequence ( π 𝑘 ) ∞ 𝑘 = 0 of joint policies generated by Algorithm 1, in a cooperative Markov game, has a non-empty set of limit points, each of which is a Nash equilibrium.
For proof see Appendix C.3.
In deriving this result, the novel details introduced by Algorithmplayed an important role.
The monotonic improvement property (Theorem 2), achieved through the multi-agent advantage and the sequential update scheme, provided us with a guarantee on the convergence of the return.
Furthermore, randomisation of the update order assured that, at convergence, none of the agents is incentiﬁed to make an update.
The proof is ﬁnalised by excluding a possibility that the algorithm converges at non-equilibrium points.
P RACTICAL A LGORITHMS
When implementing Algorithm 1 in practice, large state and action spaces could prevent agents from designating policies 𝜋 𝑖 (·| 𝑠 ) for each state 𝑠 separately.
To handle this, we parameterise each agent’s policy 𝜋 𝑖𝜃 𝑖 by 𝜃 𝑖 , which, together with other agents’ policies, forms a joint policy π θ parametrised by θ =
( 𝜃 1 , . .
, 𝜃 𝑛 ) .
In this section, we develop two deep MARL algorithms to optimise the θ .
4. HATRPO
Computing 𝐷 maxKL (cid:16) 𝜋 𝑖 𝑚 𝜃 𝑖𝑚𝑘 ,
𝜋 𝑖 𝑚 𝜃 𝑖𝑚 (cid:17) in Algorithm 1 is challenging; it requires evaluating the KLdivergence for all states at each iteration.
Similar to TRPO, one can ease this maximal KLdivergence penalty 𝐷 maxKL (cid:16) 𝜋 𝑖 𝑚 𝜃 𝑖𝑚𝑘 , 𝜋 𝑖 𝑚 𝜃 𝑖𝑚 (cid:17) by replacing it with the expected KL-divergence constraint
[Equation]
𝑖𝑚𝑘
[Equation]
·| s )
[Equation]
𝜋
[Equation]
s )
[Equation]
≤ 𝛿 where 𝛿 is a threshold hyperparameter, and the expectation can be easily approximated by stochastic sampling.
With the above amendment, we propose practical HATRPO algorithm in which, at every iteration 𝑘 + 1 , given a permutation of agents 𝑖 1: 𝑛 , agent 𝑖 𝑚 ∈{ 1 ,...,𝑛 } sequentially optimises its policy parameter 𝜃 𝑖 𝑚 𝑘 + 1 by maximising a constrained objective:
[Equation]
subject to
[Equation]
To compute the above equation, similar to TRPO, one can apply a linear approximation to the objective function and a quadratic approximation to the KL constraint; the optimisation problem in
Equation (7) can be solved by a closed-form update rule as
[Equation]
where H 𝑖 𝑚 𝑘 = ∇ 2 𝜃 𝑖𝑚 𝔼 s ∼ 𝜌 πθ 𝑘 (cid:2) 𝐷 KL (cid:0)
𝜋 𝑖 𝑚 𝜃 𝑖𝑚𝑘 (·| s ) , 𝜋 𝑖 𝑚 𝜃 𝑖𝑚 (·| s ) (cid:1)(cid:3)(cid:12)(cid:12) 𝜃 𝑖𝑚
= 𝜃 𝑖𝑚𝑘 is the Hessian of the expected KLdivergence,
g 𝑖 𝑚 𝑘 is the gradient of the objective in Equation (7), 𝛼 𝑗 < 1 is a positive coefﬁcient that is found via backtracking line search, and the product of ( H 𝑖 𝑚 𝑘 ) − 1 g 𝑖 𝑚 𝑘 can be efﬁciently computed with conjugate gradient algorithm.
The last missing piece for HATRPO is to estimate 𝔼
a
[Equation] [Equation]
which poses new challenges because each agent’s objective has to take into account all previous agents’ updates, and the size of input vaires
Fortunately, with the following proposition, we can efﬁciently estimate this objective by employing a joint advantage estimator.
Proposition 3.
Let π = (cid:206) 𝑛𝑗 =
𝜋 𝑖 𝑗 be a joint policy, and 𝐴 π ( s , a ) be its joint advantage function.
Let ¯ π 𝑖 1: 𝑚 − 1 = (cid:206)
𝑚 − 1 𝑗 = 1 ¯ 𝜋 𝑖
𝑗 be some other joint policy of agents 𝑖 1: 𝑚 − 1 , and ˆ 𝜋 𝑖 𝑚 be some other policy of agent 𝑖 𝑚 .
Then, for every state 𝑠 ,
[Equation]
For proof see Appendix D.1.
One beneﬁt of applying Equation (9) is that agents only need to maintain a joint advantage estimator 𝐴 π ( s , a ) rather than one centralised critic for each individual agent (e.g., unlike CTDE methods such as MADDPG).
Another practical beneﬁt one can draw is that, given an estimator ˆ 𝐴 ( s , a ) of the advantage function 𝐴 π θ 𝑘 ( s , a ) , for example GAE (Schulman et al., 2015b), we can estimate 𝔼 a 𝑖 1: 𝑚
[Equation]
1 ∼ π 𝑖 1: 𝑚
[Equation]
a
[Equation]
𝑚
[Equation]
with an estimator of
[Equation]
𝑠
[Equation]
, where 𝑀 𝑖 1: 𝑚 = ¯ π 𝑖 1: 𝑚
[Equation]
1 ( a 𝑖 1: 𝑚
[Equation]
1 | 𝑠 )
π 𝑖 1: 𝑚
[Equation]
1 ( a 𝑖
[Equation]
𝑚
[Equation]
1 | 𝑠 ) ˆ 𝐴 (cid:0) 𝑠, a (cid:1) .
Notably, Equation (10) aligns nicely with the sequential update scheme in HATRPO.
For agent 𝑖 𝑚 , since previous agents 𝑖 1: 𝑚 − 1 have already made their updates, the compound policy ratio for 𝑀 𝑖 1: 𝑚 in Equation (10) is easy to compute.
Given a batch B of trajectories with length 𝑇 , we can estimate the gradient with respect to policy parameters (derived in Appendix D.2) as follows,
[Equation]
The term − 1 · 𝑀 𝑖 1: 𝑚 ( s , a ) of Equation (10) is not reﬂected in ˆ g 𝑖 𝑚 𝑘 , as it only introduces a constant with zero gradient.
Along with the Hessian of the expected KL-divergence, i.e. , H 𝑖 𝑚 𝑘 , we can update 𝜃 𝑖 𝑚 𝑘 + 1 by following Equation (8).
The detailed pseudocode of HATRPO is listed in Appendix D.3.
4. HAPPO
To further alleviate the computation burden from H 𝑖 𝑚 𝑘 in HATRPO, one can follow the idea of PPO in Equation (4) by considering only using ﬁrst order derivatives.
This is achieved by making agent 𝑖 𝑚 choose a policy parameter 𝜃 𝑖 𝑚 𝑘 + 1 which maximises the clipping objective of
[Equation]
The optimisation process can be performed by stochastic gradient methods such as Adam (Kingma & Ba, 2014).
We refer to the above procedure as HAPPO and Appendix D.4 for its full pseudocode.
4. R ELATED W ORK
We are fully aware of previous attempts that tried to extend TRPO/PPO into MARL.
Despite empirical successes, none of them managed to propose a theoretically-justiﬁed trust region protocol in multi-agent learning, or maintain the monotonic improvement property.
Instead, they tend to impose certain assumptions to enable direct implementations of TRPO/PPO in MARL problems.
For example, IPPO (de Witt et al., 2020a) assume homogeneity of action spaces for all agents and enforce parameter sharing.
Yu et al.
(2021) proposed MAPPO which enhances IPPO by considering a joint critic function and ﬁner implementation techniques for on-policy methods.
Yet, it still suffers similar drawbacks of IPPO due to the lack of monotonic improvement guarantee especially when the parameter-sharing condition is switched off.
Wen et al.
(2021) adjusted PPO for MARL by considering a game-theoretical approach at the meta-game level among agents.
Unfortunately, it can only deal with two-agent cases due to the intractability of Nash equilibrium.
Recently, Li & He (2020) tried to implement TRPO for MARL through distributed consensus optimisation; however, they enforced the same ratio ¯ 𝜋 𝑖
( 𝑎 𝑖 | 𝑠 )/ 𝜋
𝑖 ( 𝑎 𝑖 | 𝑠 ) for all agents (see their Equation (7)), which, similar
[Figure]
1e
[Figure] [Figure]
[Figure]
E v a l u a t e
W i nn i n g R a t e
[Figure]
a l u
[Figure]
t e
[Figure]
Since all methods achieve 100% win rate, we believe SMAC is not sufﬁciently difﬁcult to discriminate the capabilities of these algorithms, especially when non-parameter sharing is not required.
to parameter sharing, largely limits the policy space for optimisation.
Moreover, their method comes with a 𝛿 / 𝑛 KL-constraint threshold that fails to consider scenarios with large agent number.
One of the key ideas behind our HATRPO/HAPPO is the sequential update scheme.
A similar idea of multi-agent sequential update was also discussed in the context of dynamic programming (Bertsekas, 2019) where artiﬁcial “in-between” states have to be considered.
On the contrary, our sequential update sceheme is developed based on Lemma 1, which does not require any artiﬁcial assumptions and hold for any cooperative games.
Furthermore, Bertsekas (2019) requires to maintain a ﬁxed order of updates that is pre-deﬁned for the task, whereas the order in HATRPO/MAPPO can be randomised at each iteration, which also offers desirable convergence property, as stated in Proposition 2 and also veriﬁed through ablation studies in
Appendix F.
The idea of sequential update also appeared in principal component analysis; in EigenGame (Gemp et al., 2020) eigenvectors, represented as players, maximise their own utility functions one-by-one.
Although EigenGame provably solves the PCA problem, it is of little use in MARL, where a single iteration of sequential updates is insufﬁcient to learn complex policies.
Furthermore, its design and analysis rely on closed-form matrix calculus, which has no extension to MARL.
Lastly, we would like to highlight the importance of the decomposition result in Lemma 1.
This result could serve as an effective solution to value-based methods in MARL where tremendous efforts have been made to decompose the joint Q-function into individual Q-functions when the joint Q-function are decomposable (Rashid et al., 2018).
Lemma 1, in contrast, is a general result that holds for any cooperative MARL problems regardless of decomposibility.
As such, we think of it as an appealing contribution to future developments on value-based MARL methods.
E XPERIMENTS AND R ESULTS
We consider two most common benchmarks—StarCraftII Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019) and Multi-Agent MuJoCo (de Witt et al., 2020b)—for evaluating MARL algorithms.
All hyperparameter settings and implementations details can be found in Appendix E.
StarCraftII Multi-Agent Challenge (SMAC) .
SMAC contains a set of StarCraft maps in which a team of ally units aims to defeat the opponent team.
IPPO (de Witt et al., 2020a) and MAPPO (Yu et al., 2021) are known to achieve supreme results on this benchmark.
By adopting parameter sharing, these methods achieve a winning rate of 100 % on most maps, even including the maps that have heterogeneous agents.
Therefore, we hypothesise that non-parameter sharing is not necessarily required and the trick of sharing policies is sufﬁcient to solve SMAC tasks.
We test our methods on two hard maps and one super-hard; results on Figure 2 conﬁrm that SMAC is not sufﬁciently difﬁcult to show off the capabilities of HATRPO/HAPPO when compared against existing methods.
Multi-Agent MuJoCo .
In comparison to SMAC, we believe Mujoco enviornment provides a more suitable testing case for our methods.
MuJoCo tasks challenge a robot to learn an optimal way of motion; Multi-Agent MuJoCo models each part of a robot as an independent agent, for example, a leg for a spider or an arm for a swimmer.
With the increasing variety of the body parts, modelling heterogeneous policies becomes necessary .
Figure 3 demonstrate that, in all scenarios,
HATRPO and HAPPO enjoy superior performance over those of parameter-sharing methods: IPPO and MAPPO, and also outperform non-parameter sharing MADDPG (Lowe et al., 2017b)
both in terms of reward values and variance.
It is also worth noting that the performance gap between HATRPO and its rivals enlarges with the increasing number of agents.
Meanwhile, we can observe that
[Figure]
E p
i s o d
[Figure]
R e w a r
[Figure]
A v e r a g e E p
i s o d e R e w a r d
[Figure]
i
[Figure]
(j) 17x1-Agent
[Figure]
HATRPO outperforms HAPPO in almost all tasks; we believe it is because the hard KL constraint in HATRPO, compared to the clipping version in HAPPO, relates more closely to Algorithm 1 that attains monotonic improvement guarantee.
C ONCLUSION
In this paper, we successfully apply trust region learning to multi-agent settings by proposing the ﬁrst
MARL algorithm that attains theoretically-justiﬁed monotonical improvement property.
The key to our development is the multi-agent advantage decomposition lemma that holds in general with no need for any assumptions on agents sharing parameters or the joint value function being decomposable.
Based on this, we introduced two practical deep MARL algorithms: HATRPO and HAPPO.
Experimental results on both discrete and continuous control tasks ( i.e. , SMAC and Multi-Agent
Mujoco) conﬁrm their state-of-the-art performance.
For future work, we will consider incorporating the safety constraint into HATRPO/HAPPO and propose rigorous safety-aware MARL solutions.
Appendices A Preliminaries
A.1 Deﬁnitions and Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Proofs of Preliminary Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B Proof of Proposition
C Derivation and Analysis of Algorithm
C.1 Recap of Existing Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.2 Analysis of Training of Algorithm 1 . . . . . . . . . . . . . . . . . . . . . . . . .
C.3 Analysis of Convergence of Algorithm 1 . . . . . . . . . . . . . . . . . . . . . . .
D HATRPO and HAPPO
D.1 Proof of Proposition 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.2 Derivation of the gradient estimator for HATRPO . . . . . . . . . . . . . . . . . .
D.3 Pseudocode of HATRPO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.4 Pseudocode of HAPPO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E Hyper-parameter Settings for Experiments
F Ablation Experiments
G Ablation Study on Non-Parameter Sharing MAPPO/IPPO
H Multi-Agent Particle Environment Experiments
A P RELIMINARIES A. D EFINITIONS AND A SSUMPTIONS
Assumption 1.
There exists 𝜂 ∈ ℝ , such that 0 < 𝜂 (cid:28) 1 , and for every agent 𝑖 ∈ N , the policy space Π 𝑖 is 𝜂 -soft; that means that for every 𝜋 𝑖 ∈ Π 𝑖 , 𝑠 ∈ S , and 𝑎 𝑖 ∈
A 𝑖 , we have 𝜋 𝑖 ( 𝑎 𝑖 | 𝑠 ) ≥ 𝜂 .
Deﬁnition 3.
In a fully-cooperative game, a joint policy π ∗ =
( 𝜋 1 ∗ , . . .
, 𝜋 𝑛 ∗ ) is a Nash equilibrium (NE) if for every 𝑖 ∈ N , 𝜋 𝑖 ∈ Π 𝑖 implies J ( π ∗ ) ≥ J (cid:0) 𝜋 𝑖 , π − 𝑖
∗
cid:1) .
Deﬁnition 4.
Let 𝑋 be a ﬁnite set and 𝑝 : 𝑋 → ℝ , 𝑞 : 𝑋 → ℝ be two maps.
Then, the notion of distance between 𝑝 and 𝑞 that we adopt is given by || 𝑝 − 𝑞
|| (cid:44) max 𝑥 ∈ 𝑋 | 𝑝 ( 𝑥 ) − 𝑞 ( 𝑥 )| .
A. P ROOFS OF P RELIMINARY R ESULTS
Lemma 3.
Every agent 𝑖 ’s policy space Π 𝑖 is convex and compact under the maximum norm.
Proof.
We start from proving convexity od the policy space: we prove that, for any two policies 𝜋 𝑖 , ¯ 𝜋 𝑖 ∈ Π 𝑖 , for every 𝛼 ∈ [ 0 , 1 ] , 𝛼𝜋 𝑖 + ( 1 − 𝛼 ) ¯ 𝜋 𝑖 ∈ Π 𝑖 .
Clearly, for all 𝑠 ∈ S and 𝑎 𝑖 ∈ A 𝑖 , we have 𝛼𝜋 𝑖 ( 𝑎 𝑖 | 𝑠 )
+ ( 1 − 𝛼 ) ¯ 𝜋 𝑖
( 𝑎 𝑖 | 𝑠 ) ≥ 𝛼𝜂 + ( 1 − 𝛼 )
𝜂 = 𝜂.
Also, for every state 𝑠 , (cid:213) 𝑎 𝑖
[Equation]
𝛼𝜋
[Equation]
𝑎 𝑖
[Equation]
𝑠 ) + ( 1 − 𝛼 ) ¯ 𝜋 𝑖 ( 𝑎 𝑖 | 𝑠 )
[Equation]
=
𝛼
[Equation]
𝑎 𝑖 𝜋 𝑖 ( 𝑎 𝑖 | 𝑠 ) + ( 1 − 𝛼 ) (cid:213) 𝑎 𝑖 ¯ 𝜋 𝑖 ( 𝑎 𝑖 | 𝑠 ) =
𝛼 + 1 − 𝛼 = 1 , which establishes convexity.
For compactness, we ﬁrst prove that Π 𝑖 is closed.
Let (cid:0) 𝜋 𝑖𝑘 (cid:1) ∞
𝑘 = 0 be a convergent sequence of policies of agent 𝑖 .
Let 𝜋 𝑖 be the limit.
We will prove that 𝜋 𝑖 is a policy.
First, by Assumption 1, for any 𝑘 ∈ ℕ , 𝑠 ∈ S , and 𝑎 𝑖 ∈ A 𝑖 , we have 𝜋 𝑖𝑘 (cid:0) 𝑎 𝑖 | 𝑠 (cid:1) ≥ 𝜂 .
Hence,
[Equation]
Furtheremore, for any 𝑘 and 𝑠 , we have (cid:205)
𝑎 𝑖 𝜋 𝑖𝑘 (cid:0) 𝑎 𝑖 | 𝑠 (cid:1)
= 1 .
Hence
[Equation]
With these two conditions satisﬁed, 𝜋 𝑖 is a policy, which proves the closure.
Further, for all policies 𝜋 𝑖 , states 𝑠 and actions 𝑎 , | 𝜋 𝑖 ( 𝑎 𝑖 | 𝑠 )| ≤ 1 .
This means that || 𝜋 𝑖 || max ≤ 1 , which proves boundedness.
Hence, Π 𝑖 is compact.
[Equation]
Lemma 4 (Continuity of 𝜌 𝜋 ) .
The improper state distribution 𝜌 𝜋 is continuous in 𝜋 .
Proof.
First, let us show that for any 𝑡 ∈ ℕ , the distribution 𝜌 𝑡𝜋 is continous in 𝜋 .
We will do it by induction.
This obviously holds when 𝑡 = 0 , as 𝜌 0 does not depend on policy.
Hence, we can assume that for some 𝑡 ∈ ℕ , the distribution 𝜌 𝑡𝜋 is continuous in 𝜋 .
Let us now consider two policies 𝜋 and ˆ 𝜋 .
Let 𝑠 (cid:48) ∈ S .
We have
[Equation] [Equation]
Hence
[Equation]
we obtain
[Equation]
Using the base case
[Equation]
taking the limit as ¯ 𝜋 → 𝜋
[Equation]
we get that the right-hand-side of Equation (12) converges to 0 , which proves that every 𝜌 𝑡 + 1 𝜋 is continuous in 𝜋 , and ﬁnishes the inductive step.
We can now prove that the total marginal state distribution 𝜌 𝜋 is continous in 𝜋 .
To do that, let’s take an arbitrary 𝜖 > 0 , and a natural 𝑇 such that 𝑇 > log (cid:104) 𝜖
[Equation]
1 − 𝛾 ) 4 (cid:105) log 𝛾 .
Equivalently, we choose 𝑇 such that 2 𝛾 𝑇 1 − 𝛾 < 𝜖 2 .
Let 𝜋 and ˆ 𝜋 be two policies.
We have
[Equation]
( 𝑠 )
[Equation]
𝜌
[Equation]
ˆ 𝜋 ( 𝑠 )
[Equation]
Now, by continuity of each of 𝜌 𝑡𝜋 , for 𝑡 = 0 , 1 , . . .
,𝑇 − 1 , we have that there exists a 𝛿 > 0 such that ||
𝜋 − ˆ 𝜋 || < 𝛿 implies || 𝜌 𝑡𝜋 − 𝜌 𝑡 ˆ 𝜋 || < 𝜖 2 𝑇 .
Taking such 𝛿 , by Equation (13), we get || 𝜌 𝜋 − 𝜌 ˆ 𝜋 || ≤ 𝜖 , which ﬁnishes the proof.
[Equation]
Lemma 5 (Continuity of 𝑄 𝜋 ) .
Let 𝜋 be a policy.
Then 𝑄 𝜋 ( 𝑠, 𝑎 ) is Lipschitz-continuous in 𝜋 .
Proof.
Let 𝜋 and ˆ 𝜋 be two policies.
Then we have
[Equation] [Equation]
Hence, we get
[Equation]
which implies
[Equation]
Equation (14) establishes Lipschitz-continuity with a constant 𝛾 𝑄 max ·|A| 1 − 𝛾
[Equation]
Corollary 1.
From Lemma 5 we obtain that the following functions are Lipschitz-continuous in 𝜋 :
the state value function 𝑉 𝜋 = (cid:205)
𝑎 𝜋 ( 𝑎 | 𝑠 ) 𝑄 𝜋 ( 𝑠, 𝑎 ) , the advantage function 𝐴 𝜋 ( 𝑠, 𝑎 ) = 𝑄 𝜋 ( 𝑠, 𝑎 ) − 𝑉 𝜋 ( 𝑠 ) , and the expected total reward J ( 𝜋 ) =
𝔼 s ∼ 𝜌
[ 𝑉 𝜋 ( s )] .
Lemma 6.
Let 𝜋 and ˆ 𝜋 be policies.
The quantity 𝔼 s ∼ 𝜌 𝜋 , a ∼ ˆ 𝜋 [ 𝐴 𝜋 ( s , a )] is continuous in 𝜋 .
Proof.
We have
[Equation]
Continuity follows from continuity of each 𝜌 𝜋 ( 𝑠 ) (Lemma 4) and
𝐴 𝜋 ( 𝑠, 𝑎 ) (Corollary 1).
[Equation]
Corollary 2 (Continuity in MARL) .
All the results about continuity in 𝜋 extend to MARL.
Policy 𝜋 can be replaced with joint policy π ; as π is Lipschitz-continuous in agent 𝑖 ’s policy 𝜋 𝑖 , the above continuity results extend to conitnuity in 𝜋 𝑖 .
Thus, we will quote them in our proofs for MARL.
B P ROOF OF P ROPOSITION
Proposition 1.
Let’s consider a fully-cooperative game with an even number of agents 𝑛 , one state, and the joint action space { 0 , 1 } 𝑛 , where the reward is given by 𝑟 ( 0 𝑛 / 2 , 1 𝑛 / 2 )
=
𝑟 ( 1 𝑛 / 2 , 0 𝑛 / 2 ) = 1 , and 𝑟 ( a 1: 𝑛 ) = 0 for all other joint actions.
Let J ∗ be the optimal joint reward, and J ∗ share be the optimal joint reward under the shared policy constraint.
Then
[Equation]
Proof.
Clearly J ∗ = 1 .
An optimal joint policy in this case is, for example, the deterministic policy with joint action ( 0 𝑛 / 2 , 1 𝑛 / 2 ) .
Now, let the shared policy be ( 𝜃, 1 − 𝜃 ) , where 𝜃 determines the probability that an agent takes action 0 .
Then, the expected reward is
[Equation]
In order to maximise J ( 𝜃 ) , we must maximise 𝜃 ( 1 − 𝜃 ) , or equivalently, (cid:112) 𝜃 ( 1 − 𝜃 ) .
By the artithmetic-geometric means inequality, we have
[Equation]
where the equality holds if and only if 𝜃 = 1 − 𝜃 , that is 𝜃 = 12 .
In such case we have
[Equation]
which ﬁnishes the proof.
[Equation]
C D ERIVATION AND A NALYSIS OF A LGORITHMC. R ECAP OF E XISTING R ESULTS
Lemma 7 (Performance Difference) .
Let ¯ 𝜋
and 𝜋 be two policies.
Then, the following identity holds,
[Equation]
Proof.
See Kakade & Langford (2002) (Lemma 6.1) or Schulman et al.
(2015a)
(Appendix A).
(cid:3)
Theorem 1.
(Schulman et al., 2015a, Theorem 1)
Let 𝜋 be the current policy and ¯ 𝜋 be the next candidate policy.
We deﬁne
[Equation]
𝜋 ) + 𝔼 s ∼ 𝜌 𝜋 , a ∼ ¯ 𝜋
[ 𝐴 𝜋 ( 𝑠, 𝑎 )] , 𝐷 maxKL ( 𝜋, ¯ 𝜋 ) = max 𝑠 𝐷 KL ( 𝜋 (·| 𝑠 ) , ¯ 𝜋 (·| 𝑠 )) .
Then the inequality of
[Equation]
holds, where 𝐶 = 4 𝛾 max 𝑠,𝑎 | 𝐴 𝜋 ( 𝑠,𝑎 )| ( 1 − 𝛾 ).
Proof.
See Schulman et al.
(2015a)
(Appendix A and Equation (9) of the paper).
(cid:3)
C. A NALYSIS OF T RAINING OF A LGORITHM
Lemma 1 (Multi-Agent Advantage Decomposition) .
In any cooperative Markov games, given a joint policy π , for any state 𝑠 , and any agent subset 𝑖 1: 𝑚 , the below equations holds.
[Equation]
Proof.
By the deﬁnition of multi-agent advantage function,
[Equation]
which ﬁnishes the proof.
Note that a similar ﬁnding has been shown in Kuba et al.
[Equation]
Lemma 8.
Let π = (cid:206)
𝑛𝑖
= 1 𝜋 𝑖 and ¯ π = (cid:206) 𝑛𝑖 =
1 ¯ 𝜋 𝑖 be joint policies.
Then
[Equation]
Proof.
For any state 𝑠
[Equation]
we have
[Equation]
Now, taking maximum over 𝑠 on both sides yields
[Equation]
as required.
[Equation]
Lemma 2.
Let π be a joint policy.
Then, for any joint policy ¯ π , we have
[Equation]
Proof.
By Theorem
[Equation]
which by Lemma 1 equals
[Equation]
and by Lemma 8 this is at least
[Equation]
Theorem 2.
A sequence ( π 𝑘 ) ∞ 𝑘 = 0 of joint policies updated by Algorithm 1 has the monotonic improvement property, i.e., J ( π 𝑘 + 1 ) ≥ J ( π 𝑘 ) for all 𝑘 ∈ ℕ .
Proof.
Let π 0 be any joint policy.
For every 𝑘 ≥ 0 , the joint policy π 𝑘 + 1 is obtained from π 𝑘 by
Algorithm 1 update; for 𝑚 = 1 , . . .
, 𝑛 ,
[Equation]
By Theorem 1, we have
[Equation]
which by Lemma 8 is lower-bounded by
[Equation]
and as for every 𝑚 , 𝜋 𝑖 𝑚 𝑘 + 1 is the argmax, this is lower-bounded by
[Equation]
which, as mentioned in Deﬁnition 2, equals
[Equation]
where the last inequality follows from Equation (6).
This proves that Algorithm 1 achieves monotonic improvement.
[Equation]
C. A NALYSIS OF C ONVERGENCE OF A LGORITHM
Proposition 2.
Supposing in Algorithm 1 any permutation of agents has a ﬁxed non-zero probability, a sequence ( π 𝑘 ) ∞ 𝑘 = 0 of joint policies generated by Algorithm 1, in a cooperative Markov game, has a non-empty set of limit points, each of which is a Nash equilibrium.
Proof.
Step 1 (convergence).
Firstly, it is clear that the sequence (J ( π 𝑘 )) ∞
𝑘 = 0 converges as, by
Theorem 2, it is non-decreasing and bounded above by 𝑅 max 1 − 𝛾 .
Let us denote the limit by ¯ J .
For every 𝑘 , we denote the tuple of agents, according to whose order the agents perform the sequential updates, by 𝑖 𝑘 1: 𝑛 , and we note that (cid:0) 𝑖 𝑘 1: 𝑛 (cid:1) 𝑘 ∈ ℕ is a random process.
Furthermore, we know that the sequence of policies ( π 𝑘 ) is bounded, so by Bolzano-Weierstrass Theorem, it has at least one convergent subsequence.
Let ¯ π
be any limit point of the sequence (note that the set of limit points is a random set), and (cid:0) π 𝑘 𝑗 (cid:1) ∞
𝑗 = 0 be a subsequence converging to ¯ π (which is a random subsequence as well).
By continuity of J in π (Corollary 1), we have
[Equation]
For now, we introduce an auxiliary deﬁnition.
Deﬁnition 5 (TR-Stationarity) .
A joint policy ¯ π is trust-region-stationary (TR-stationary) if, for every agent 𝑖 ,
[Equation]
where 𝐶 ¯ π = 4 𝛾𝜖 ( 1 − 𝛾 ) 2 , and 𝜖 = max 𝑠, a | 𝐴 ¯ π
( 𝑠, a )| .
We will now establish the TR-stationarity of any limit point joint policy ¯ π (which, as stated above, is a random variable).
Let 𝔼 𝑖 0: ∞ 1: 𝑛 [·] denote the expected value operator under the random process ( 𝑖 0: ∞ 1: 𝑛 ) .
Let also 𝜖 𝑘 = max 𝑠, a | 𝐴 π 𝑘
( 𝑠, a )| , and 𝐶 𝑘 = 4 𝛾𝜖 𝑘 ( 1 − 𝛾 ) 2 .
We have
[Equation]
π 𝑘 + 1 ) − J ( π 𝑘 )] ≥ lim 𝑘 →∞ 𝔼 𝑖 0: ∞ 1: 𝑛 (cid:2)
𝐿 π 𝑘 ( π 𝑘 + 1 ) − 𝐶 𝑘 𝐷 maxKL ( π 𝑘 , π 𝑘 + 1 ) (cid:3) by Theorem≥ lim
[Equation]
𝔼 𝑖 0:
[Equation]
1: 𝑛
[Equation]
𝜋
[Equation]
by Equation (15) and the fact that each of its summands is non-negative
Now, we consider an arbitrary limit point ¯ π from the (random) limit set, and a (random) subsequence (cid:0) π 𝑘 𝑗 (cid:1) ∞
𝑗 = 0 that converges to ¯ π .
We get
[Equation]
As the expectation is taken of non-negative random variables, and for every 𝑖 ∈ N and 𝑘 ∈ ℕ , with some positive probability 𝑝 𝑖 , we have 𝑖 𝑘 𝑗 1 =
𝑖
(because every permutation has non-zero probability), the above is bounded from below by
[Equation]
which, as π 𝑘 𝑗 converges to ¯ π , equals to
[Equation]
by Equation (6).
For convergence of 𝐿 𝑖 π 𝑘𝑗 ( 𝜋 𝑖 ) we used Deﬁnition 2 combined with Lemma 6, for convergence of 𝐶 𝑘 𝑗
we used Corollary 1, and the convergence of 𝐷 maxKL follows from continuity of 𝐷 KL and max .
This proves that, for any limit point ¯ π of the random process ( π 𝑘 ) induced by Algorithm 1, max
[Equation]
(cid:1)(cid:3) = 0 , which is equivalent with Deﬁnition 5.
Step 2 (dropping the penalty term).
Now, we have to prove that TR-stationary points are NEs of cooperative Markov games.
The main step is to prove the following statement: a TR-stationary joint policy ¯ π , for every state 𝑠 ∈ S , satisﬁes
[Equation]
We will use the technique of the proof by contradiction.
Suppose that there is a state 𝑠 0 such that there exists a policy ˆ 𝜋 𝑖 with
[Equation]
Let us parametrise the policies 𝜋 𝑖 according to the template
[Equation]
where the values of 𝜃 𝑖𝑗 ( 𝑗 = 1 , . .
, 𝑑 𝑖 − 1 ) are such that 𝜋 𝑖 (·| 𝑠 0 ) is a valid policy.
Then we can rewrite our quantity of interest (the objective of Equation (17) as
[Equation]
which is an afﬁne function of the policy parameterisation.
It follows that its gradient (with respect to 𝜃 𝑖 ) and directional derivatives are constant in the space of policies at state 𝑠 0 .
The existance of policy ˆ 𝜋 𝑖 (·| 𝑠 0 ) , for which Inequality (18) holds, implies that the directional derivative in the direction from
¯ 𝜋 𝑖 (·| 𝑠 0 ) to ˆ 𝜋 𝑖 (·| 𝑠 0 ) is strictly positive.
We also have
[Equation]
, when evaluated at
[Equation]
19) which means that the KL-penalty has zero gradient at ¯ 𝜋 𝑖 (·| 𝑠 0 ) .
Hence, when evaluated at 𝜋 𝑖 (·| 𝑠 0 )
=
¯ 𝜋 𝑖 (·| 𝑠 0 ) , the objective
[Equation]
has a strictly positive directional derivative in the direction of ˆ 𝜋 𝑖 (·| 𝑠 0 ) .
Thus, there exists a policy (cid:101) 𝜋 𝑖 (·| 𝑠 0 ) , sufﬁciently close to ¯ 𝜋 𝑖 (·| 𝑠 0 ) on the path joining it with ˆ 𝜋 𝑖 (·| 𝑠 0 ) , for which
[Equation]
Let 𝜋 𝑖 ∗ be a policy such that 𝜋 𝑖 ∗ (·| 𝑠 0 ) =
(cid:101) 𝜋 𝑖 (·| 𝑠 0 ) , and 𝜋 𝑖 ∗ (·| 𝑠 ) =
¯ 𝜋 𝑖 (·| 𝑠 ) for states 𝑠 ≠ 𝑠 0 .
As for these states we have
[Equation]
a
[Equation]
and 𝐷 KL
[Equation]
¯ 𝜋
[Equation]
·| 𝑠
[Equation]
𝜋 𝑖 ∗
[Equation]
·| 𝑠
[Equation]
=
[Equation]
it follows that
[Equation]
which is a contradiction with TR-stationarity of ¯ π .
Hence, the claim of Equation (17) is proved.
Step 3 (optimality).
Now, for a ﬁxed joint policy ¯ π − 𝑖 of other agents, ¯ 𝜋 𝑖 satisﬁes
[Equation]
which is the Bellman optimality equation (Sutton & Barto, 2018).
Hence, for a ﬁxed joint policy ¯ π − 𝑖 , the policy ¯ 𝜋 𝑖 is optimal:
[Equation]
As agent 𝑖 was chosen arbitrarily, ¯ π is a Nash equilibrium.
[Equation]
D HATRPO AND HAPPO D. P ROOF OF P ROPOSITION
Proposition 3.
Let π = (cid:206) 𝑛𝑗 =
𝜋 𝑖 𝑗 be a joint policy
[Equation]
and 𝐴 π ( s , a ) be its joint advantage function.
Let ¯ π 𝑖 1: 𝑚 − 1 = (cid:206)
𝑚 − 1 𝑗 = 1 ¯ 𝜋 𝑖
𝑗 be some other joint policy of agents 𝑖 1: 𝑚 −
[Equation]
and ˆ 𝜋 𝑖 𝑚 be some other policy of agent 𝑖 𝑚 .
Then, for every state 𝑠 ,
[Equation]
Proof.
We have
[Equation]
which, by Lemma
[Equation]
equals
[Equation]
D. D ERIVATION OF THE GRADIENT ESTIMATOR FOR HATRPO
[Equation]
Evaluated at 𝜃 𝑖 𝑚 = 𝜃 𝑖 𝑚 𝑘 , the above expression equals
[Equation]
which ﬁnishes the derivation.
D. P SEUDOCODE OF HATRPO
do 4:
Collect
Compute advantage a
Set 𝑀 s a
= ˆ 𝐴 a
10: for agent
𝑖 𝑚 =
𝑖 1 , . .
. , do 11:
Estimate the gradient of the agent’s maximisation objective
[Equation]
Use the conjugate gradient algorithm to compute the update direction
[Equation]
where ˆ H 𝑖 𝑚 𝑘 is the Hessian of the average KL-divergence
[Equation]
Estimate the maximal step size allowing for meeting the KL-constraint
[Equation]
Update agent 𝑖 𝑚 ’s policy by
[Equation]
where 𝑗 ∈ { 0 , 1 , . . .
, 𝐿 } is the smallest such 𝑗 which improves the sample loss by at least 𝜅𝛼 𝑗 ˆ 𝛽 𝑖 𝑚
𝑘 ˆ x 𝑖 𝑚 𝑘 · ˆ g 𝑖 𝑚 𝑘 , found by the backtracking line search.
Compute
[Equation]
15: end for 16:
Update V-value network by following formula: 17:
[Equation]
18: end for
D. P SEUDOCODE OF HAPPO
for do 4:
Collect a set of trajectories by running the joint policy π θ 𝑘 = ( 𝜋 1 𝜃 1 𝑘 , . .
𝜋 𝑛𝜃 𝑛𝑘 ) .
Push transitions {( 𝑜 𝑖𝑡 , 𝑎 𝑖𝑡 , 𝑜 𝑖𝑡 + 1 , 𝑟 𝑡 ) , ∀ 𝑖 ∈ N , 𝑡 ∈ 𝑇 } into B .
Sample a random minibatch of 𝐵 transitions from B .
Compute advantage function ˆ 𝐴 ( s a based global V-value network with GAE.
Draw a random permutation of agents 𝑖 1: 𝑛 .
Set 𝑀 𝑖 1 ( s , a )
=
ˆ 𝐴 ( s , a ) .
10: for agent
𝑖 𝑚 =
𝑖 1 , . .
𝑖 𝑛 do 11:
Update actor 𝑖 𝑚 with 𝜃 𝑖 𝑚 𝑘
+ 1 , the argmax of the PPO-Clip objective
[Table]
[Equation]
[Table]
Compute
[Table]
𝑖𝑚 | o 𝑖𝑚 ) 𝜋 𝑖𝑚 𝜃𝑖𝑚𝑘 ( a 𝑖𝑚 | o 𝑖𝑚 )
𝑀 𝑖 1: 𝑚 ( s , a ) .
//Unless 𝑚
= 𝑛 .
13: end for 14:
Update V-value network by following formula: 15:
[Equation]
=(cid:0) 𝑉
[Equation]
( 𝑠 𝑡 ) − ˆ 𝑅 𝑡 (cid:1)16: end for
E H YPER - PARAMETER S ETTINGS FOR E XPERIMENTS
hyperparameters value hyperparameters value hyperparameters value
[Table]
gamma
[Table]
batch size gain
[Table]
hidden layer training threads actor network mlp
[Table]
mini
[Table]
batch
[Table]
threads hypernet
[Table]
[Table]
episode length activation
ReLU hidden layer dim use huber loss
True
[Table]
The implementation of MADDPG is adopted from the Tianshou framework (Weng et al., 2021), all hyperparameters left unchanged at the origin best-performing status.
[Table]
[Table] [Table]
[Table] [Table]
activation
ReLU episode length
F A BLATION E XPERIMENTS
In this section
[Figure]
we conduct ablation study to investigate the importance of two key novelties
that
our
HATRPO introduced; they are heterogeneity of agents’ parameters and the randomisation of order of agents in the sequential update scheme.
We compare the performance of original HATRPO with a version that shares parameters
[Figure]
and with a version where the order in sequential update scheme is ﬁxed throughout training.
We run the experiments on two MAMuJoCo tasks (2-agent & 6-agent).
[Figure]
The experiments reveal that, although the modiﬁed versions of HATRPO still outperform baselines (represented by MAPPO), their deviation from the theory harms performance.
In particular, parameter sharing introduces extra variance to training, harms the monotonic improvement property (Theorem 2 assumes heterogeneity), and causes HATRPO to converge to suboptimal policies.
The suboptimality is more severe in the task with more agents, as suggested by Proposition 1.
Similarly, ﬁxed order in the sequential update scheme negatively affected the performance at convergence (especially in the task with 6 agents), as suggested by Proposition 2.
We conclude that the ﬁne performance of HATRPO relies strongly on the close connection between theory an implementation.
The connection becomes increasingly important with the growing number of agents.
G A BLATION S TUDY ON N ON -P ARAMETER S HARING MAPPO/IPPO
We verify that heterogeneous-agent trust region algorithms (represented by HATRPO) achieve superior performance to, originally homogeneous, IPPO/MAPPO algorithms with the parameter-sharing function switched off.
[Figure]
H M ULTI -A GENT P ARTICLE E NVIRONMENT E XPERIMENTS
We verify that heterogeneous-agent trust region algorithms (represented here by HAPPO) quickly solve cooperative MPE tasks.
[Figure]