M ULTI -A GENT C ONSTRAINED P OLICY O PTIMISATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Developing reinforcement learning algorithms that satisfy safety constraints is becoming increasingly important in real-world applications.
In multi-agent reinforcement learning (MARL) settings, policy optimisation with safety awareness is particularly challenging because each individual agent has to not only meet its own safety constraints, but also consider those of others so that their joint behaviour can be guaranteed safe.
Despite its importance, the problem of safe multiagent learning has not been rigorously studied; very few solutions have been proposed, nor a sharable testing environment or benchmarks.
To ﬁll these gaps, in this work, we formulate the safe MARL problem as a constrained Markov game and solve it with policy optimisation methods.
Our solutions— Multi-Agent Constrained Policy Optimisation (MACPO) and MAPPO-Lagrangian —leverage the theories from both constrained policy optimisation and multi-agent trust region learning.
Crucially, our methods enjoy theoretical guarantees of both monotonic improvement in reward and satisfaction of safety constraints at every iteration.
To examine the effectiveness of our methods, we develop the benchmark suite of Safe
Multi-Agent MuJoCo that involves a variety of MARL baselines.
Experimental results justify that MACPO/MAPPO-Lagrangian can consistently satisfy safety constraints, meanwhile achieving comparable performance to strong baselines.
I NTRODUCTION
In recent years, reinforcement learning (RL) techniques have achieved remarkable successes on a variety of complex tasks (Silver et al., 2016; 2017; Vinyals et al., 2019).
Powered by deep neural networks, deep RL enables learning sophisticated behaviours.
On the other hand, deploying neural networks turns the optimisation procedure from policy space to parameter space; this enables gradient-based methods to be applied (Sutton et al., 1999; Lillicrap et al., 2015; Schulman et al., 2017).
For policy gradient methods, at every iteration, the parameters of a policy network are updated in the direction of the gradient that maximises return.
However, policies that are purely optimised for reward maximisation are rarely applicable to realworld problems.
In many applications, an agent is often required not to visit certain states or take certain actions, which are thought of as “unsafe” either for itself or for other elements in the background (Moldovan & Abbeel, 2012; Achiam et al., 2017).
For instance, a robot carrying materials in a warehouse should not damage its parts while delivering an item to a shelf, nor should a selfdriving car cross on the red light while rushing towards its destination (Shalev-Shwartz et al., 2016).
To tackle these issues, Safe RL (Moldovan & Abbeel, 2012; Garcıa & Fern´andez, 2015) is proposed, aiming to develop algorithms that learn policies that satisfy safety constraints.
Despite the additional requirement of safety on solutions, algorithms with convergence guarantees have been proposed (Xu et al., 2021; Wei et al., 2021).
Developing safe policies for multi-agent systems is a challenging task.
Part of the difﬁculty comes from solving multi-agent reinforcement learning (MARL) problems itself (Deng et al., 2021); more importantly, tackling safety in MARL is hard because each individual agent has to not only consider its own safety constraints, which already may conﬂict its reward maximisation, but also consider the safety constraints of others so that their joint behaviour is guaranteed to be safe.
As a result, there are very few solutions that offer effective learning algorithms for safe MARL problems.
In fact, many of the existing methods focus on learning to cooperate (Foerster et al., 2018;
Rashid et al., 2018; Yang & Wang, 2020).
However, they often require certain structures on the solution; for example, Rashid et al.
(2018) and Yang et al.
(2020) adopt greedy maximisation on the local component of a monotonic joint value function, and Foerster et al.
(2018) estimates the policy gradient based on the counterfactual value from a joint critic.
Therefore, it is unclear how to directly incorporate safety constraints into these solution frameworks.
Consequently, developing agents’ collaborations towards reward maximisation under safety constraints remains an unsolved problem.
The goal of this paper is to increase practicality of MARL algorithms through endowing them with safety awareness.
For this purpose, we introduce a general framework to formulate safe MARL problems, and solve them through multi-agent policy optimisation methods.
Our solutions leverage techniques from both constrained policy optimisation (Achiam et al., 2017) and multi-agent trust region learning (Kuba et al., 2021a).
The resulting algorithm attains properties of both monotonic improvement guarantee and constraints satisfaction guarantee at every iteration during training.
To execute the optimisation objectives, we introduce two practical deep MARL algorithms: MACPO and MAPPO-Lagrangian.
As a side contribution, we also develop the ﬁrst safe MARL benchmark within the MuJoCo environment, which include a variety of MARL baseline algorithms.
We evaluate MACPO and MAPPO-Lagrangian on a series of tasks, and results clearly conﬁrm the effectiveness of our solutions both in terms of constraints satisfaction and reward maximisation.
To our best knowledge, MACPO and MAPPO-Lagrangian are the ﬁrst safety-aware model-free MARL algorithms that work effectively in the challenging MuJoCo tasks with safety constraints.
R ELATED W ORK
Considering safety in the development of AI is a long-standing topic (Amodei et al., 2016).
When it comes to safe reinforcement learning (Garcıa & Fern ´ andez, 2015), a commonly used framework is Constrained Markov Decision Processes (CMDPs) (Altman, 1999).
In a CMDP, at every step, in addition to the reward, the environment emits costs associated with certain constraints.
As a result, the learning agent must try to satisfy those constraints while maximising the total reward.
In general, the cost from the environment can be thought of as a measure of safety.
Under the framework of CMDP, a safe policy is the one that explores the environment safely by keeping the total costs under certain thresholds.
To tackle the learning problem in CMDPs, Achiam et al.
(2017) introduced Constrained Policy Optimisation (CPO), which updates agent’s policy under the trust region constraint (Schulman et al., 2015) to maximise surrogate return while obeying surrogate cost constraints.
However, solving a constrained optimisation at every iteration of CPO can be cumbersome for implementation.
An alternative solution is to apply primal-dual methods, giving rise to methods like TRPO-Lagrangian and PPO-Lagrangian (Ray et al., 2019).
Although these methods achieve impressive performance in terms of safety, the performance in terms of reward is poor (Ray et al., 2019).
Another class of algorithms that solves CMDPs is by Chow et al.
(2018; 2019); these algorithms leverage the theoretical property of the Lyapunov functions and propose safe value iteration and policy gradient procedures.
In contrast to CPO, Chow et al.
(2018; 2019) can work with off-policy methods; they also can be trained end-to-end with no need for line search.
Safe multi-agent learning is an emerging research domain.
Despite its importance (Shalev-Shwartz et al., 2016), there are few solutions that work with MARL in a model-free setting.
The majority of methods are designed for robotics learning.
For example, the technique of barrier certiﬁcates (Borrmann et al., 2015; Ames et al., 2016;
Qin et al., 2020) or model predictive shielding (Zhang et al., 2019) from control theory is used to model safety.
These methods, however, are speciﬁcally derived for robotics applications; they either are supervised learning based approaches, or require speciﬁc assumptions on the state space and environment dynamics.
Moreover, due to the lack of a benchmark suite for safe MARL algorithms, the generalisation ability of those methods is unclear.
The most related work to ours is Safe Dec-PG (Lu et al., 2021) where they used the primal-dual framework to ﬁnd the saddle point between maximising reward and minimising cost.
In particular, they proposed a decentralised policy descent-ascent method through a consensus network.
However, reaching a consensus equivalently imposes an extra constraint of parameter sharing among neighbouring agents, which could yield suboptimal solutions (Kuba et al., 2021a).
Furthermore, multi-agent policy gradient methods can suffer from high variance (Kuba et al., 2021b).
In contrast, our methods employ trust region optimisation and do not assume any parameter sharing.
HATRPO (Kuba et al., 2021a) introduced the ﬁrst multi-agent trust region method that enjoys theoretically-justiﬁed monotonic improvement guarantee.
Its key idea is to make agents follow a sequential policy update scheme so that the expected joint advantage will always be positive, thus increasing reward.
In this work, we show how to further develop this theory and derive a protocol which, in addition to the monotonic improvement, also guarantees to satisfy the safety constraint at every iteration during learning.
The resulting algorithm (Algorithm 1) successfully attains theoretical guarantees of both monotonic improvement in reward and satisfaction of safety constraints.
P ROBLEM F ORMULATION : C ONSTRAINED M ARKOV G AME
We formulate the safe MARL problem as a constrained Markov game (cid:104)N , S , A , p , 𝜌 0 , 𝛾, 𝑅, C , c (cid:105) .
Here, N = { 1 , . .
. , 𝑛 } is the set of agents, S is the state space, A = (cid:206) 𝑛𝑖
=
1 A 𝑖 is the product of the agents’ action spaces, known as the joint action space, p : S ×
A × S → ℝ is the probabilistic transition function, 𝜌 0 is the initial state distribution, 𝛾 ∈ [ 0 , 1 ) is the discount factor, 𝑅 : S×
A → ℝ is the joint reward function, C
=
{ 𝐶 𝑖𝑗 } 𝑖 ∈N 1 ≤ 𝑗 ≤ 𝑚 𝑖 is the set of sets of cost functions (every agent 𝑖 has 𝑚 𝑖 cost functions) of the form 𝐶 𝑖𝑗 : S
×
A 𝑖 → ℝ , and ﬁnally the set of corresponding cost-constraining values is given by c
= { 𝑐 𝑖𝑗 } 𝑖 ∈N 1 ≤ 𝑗 ≤ 𝑚 𝑖 .
At time step 𝑡 , the agents are in a state s 𝑡 , and every agent 𝑖 takes an action a 𝑖 𝑡 according to its policy 𝜋 𝑖 ( a 𝑖 | s 𝑡 ) .
Together with other agents’ actions, it gives a joint action a 𝑡 = ( a 1 𝑡 , . . .
, a 𝑛𝑡 ) and the joint policy π ( a | s ) =
(cid:206)
𝑛𝑖
= 1 𝜋 𝑖 ( a 𝑖 | s ) .
The agents receive the reward 𝑅 ( s 𝑡 , a 𝑡 ) , meanwhile each agent 𝑖 pays the costs 𝐶 𝑖𝑗 ( s 𝑡 , a 𝑖𝑡 ) , ∀ 𝑗 = 1 , . . .
, 𝑚 𝑖 .
The environment then transits to a new state s 𝑡 + 1 ∼ p (·
| s 𝑡 , a 𝑡 ) .
In this paper, we consider a fully-cooperative setting where all agents share the same reward function, aiming to maximise the expected total reward of
[Equation]
meanwhile trying to satisfy every agent 𝑖 ’s safety constraints, written as
[Equation]
We deﬁne the state-action value and the state-value functions in terms of reward as 𝑄
[Equation]
and 𝑉
[Equation]
[Equation]
(cid:44)
[Equation]
The joint policies π that satisfy the Inequality (1) are referred to as feasible .
Notably, in the above formulation, although the action a 𝑖𝑡 of agent 𝑖 does not directly inﬂuence the costs { 𝐶 𝑘𝑗 ( s 𝑡 , a 𝑘𝑡 )} 𝑚 𝑘 𝑗 =of other agents 𝑘 ≠ 𝑖 , the action a 𝑖𝑡 will implicitly inﬂuence their total costs due to the dependence on the next state s 𝑡 + 1 2 .
For the 𝑗 th cost function of agent 𝑖 , we deﬁne the 𝑗 th state-action cost value function and the state cost value function as
[Equation]
{+∞ (cid:213) 𝑡 =𝛾 𝑡 𝐶 𝑖𝑗 ( s 𝑡 , a 𝑖𝑡 ) (cid:12)(cid:12) s 
+}
= 𝑠, a 𝑖 [-
-]0 =
𝑎 𝑖 (cid:105) , and, 𝑉 𝑖𝑗, π ( 𝑠 ) (cid:44) 𝔼 a ∼ π , s 1: ∞ ∼ p , a 1: ∞ ∼ π (cid:104) ∞ (cid:213) 𝑡 =𝛾 𝑡 𝐶 𝑖𝑗 ( s 𝑡 , a 𝑖𝑡 ) (cid:12)(cid:12) s
= 𝑠 (cid:105) , respectively.
Notably, the cost value functions 𝑄 𝑖𝑗, π and 𝑉 𝑖𝑗, π , although similar to traditional 𝑄 π and 𝑉 π , involve extra indices 𝑖 and 𝑗 ; the superscript 𝑖 denotes an agent, and the subscript 𝑗 denotes its 𝑗 th cost.
Throughout this work, we pay a close attention to the contribution to performance from different subsets of agents, therefore, we introduce the following notations.
We denote an arbitrary subset
{ 𝑖 1 , . . . , 𝑖 ℎ } of agents as 𝑖 1: ℎ ; we write − 𝑖 1: ℎ to refer to its complement.
Given the agent subset 𝑖 1: ℎ , we deﬁne the multi-agent state-action value function :
[Equation]
On top of it, the multi-agent advantage function 3 is deﬁned as follows,
[Equation]
An interesting fact about the above multi-agent advantage function is that any advantage 𝐴 𝑖 1: ℎ π can be written as a sum of sequentially-unfolding multi-agent advantages of individual agents, that is,
Lemma 1 (Multi-Agent Advantage Decomposition, Kuba et al.
(2021b)) .
For any state 𝑠 ∈ S , subset of agents 𝑖 1: ℎ ⊆ N , and joint action a 𝑖 1: ℎ , the following identity holds
[Equation]
M ULTI -A GENT C ONSTRAINED P OLICY O PTIMISATION
In this section, we ﬁrst present a theoretically-justiﬁed safe multi-agent policy iteration procedure, which leverages multi-agent trust region learning and constrained policy optimisation to solve constrained Markov games.
Based on this, we propose two practical deep MARL algorithms, enabling optimising neural-network based policies that satisfy safety constraints.
Throughout this work, we refer the symbols π and ¯ π to be the “current” and the “new” joint policies, respectively.
4. M ULTI - AGENT T RUST R EGION L EARNING W ITH C ONSTRAINTS
Kuba et al.
(2021a) introduced the ﬁrst multi-agent trust region method—HATRPO—that enjoys theoretically-justiﬁed monotonic improvement guarantee.
Speciﬁcally, it relies on the multi-agent advantage decomposition in Lemma 1, and the “surrogate” return that is given as follows.
Deﬁnition 1.
Let π be a joint policy, ¯ π 𝑖 1: ℎ − 1 be some other joint policy of agents 𝑖 1: ℎ − 1 , and ˆ 𝜋 𝑖
ℎ be a policy of agent 𝑖 ℎ .
Then we deﬁne
[Equation]
With the above deﬁnition, we can see that Lemma 1 allows for decomposing the joint surrogate return 𝐿 π ( ¯ π ) (cid:44) 𝔼 s ∼ 𝜌 π , a ∼ ¯ π
[ 𝐴 π ( s , a )] into a sum over surrogates of 𝐿 𝑖 1: ℎ π ( ¯ π 𝑖 1: ℎ − 1 , ¯ 𝜋 𝑖 ℎ ) , for ℎ = 1 , . . .
, 𝑛 .
This can be used to justify that if agents, with a joint policy π , update their policies by following a sequential update scheme , that is, if each agent in the subset 𝑖 1: ℎ sequentially solves the following optimisation problem:
[Equation]
where 𝜈
= 4 𝛾 max 𝑠, a | 𝐴 π ( 𝑠, a )| ( 1 − 𝛾 ), and 𝐷 maxKL ( 𝜋 𝑖 ℎ , ˆ 𝜋 𝑖 ℎ ) (cid:44) max 𝑠 𝐷 KL ( 𝜋 𝑖 ℎ (·| 𝑠 ) , ˆ 𝜋 𝑖 ℎ (·| 𝑠 )) , then the resulting joint policy ¯ π will surely improve the expected return, i.e., 𝐽 ( ¯ π ) ≥ 𝐽 ( π ) (see the proof in Kuba et al.
(2021a, Lemma 2)).
We know that due to the penalty term 𝐷 maxKL ( 𝜋 𝑖 ℎ , ˆ 𝜋 𝑖 ℎ ) , the new policy ¯ 𝜋 𝑖 ℎ will stay close ( w.r.t max-KL distance) to 𝜋 𝑖 ℎ .
For the safety constraints, we can extend Deﬁnition 1 to incorporate the “surrogate” cost, thus allowing us to study the cost functions in addition to the return.
Deﬁnition 2.
Let π be a joint policy, and ¯ 𝜋 𝑖 be some other policy of agent 𝑖 .
Then, for any of its costs of index 𝑗 ∈ { 1 , . . .
, 𝑚 𝑖 } , we deﬁne
[Equation]
By generalising the result about the surrogate return in Equation (1), we can derive how the expected costs change when the agents update their policies.
Speciﬁcally, we provide the following lemma.
Lemma 2.
Let π and ¯ π be joint policies.
Let 𝑖 ∈ N be an agent, and 𝑗 ∈ { 1 , . .
, 𝑚 𝑖 } be an index of one of its costs.
The following inequality holds
[Equation]
See proof in Appendix A.
The above lemma suggests that, as long as the distances between the policies 𝜋 ℎ and ¯ 𝜋 ℎ , ∀ ℎ ∈ N , are sufﬁciently small, then the change in the 𝑗 th cost of agent 𝑖 , i.e., 𝐽 𝑖𝑗 ( ¯ π ) − 𝐽 𝑖𝑗 ( π ) , is controlled by the surrogate 𝐿 𝑖𝑗, π ( ¯ 𝜋 𝑖 ) .
Importantly, this surrogate is independent of other agents’ new policies.
Hence, when the changes in policies of all agents are sufﬁciently small, each agent 𝑖 can learn a better policy ¯ 𝜋 𝑖 by only considering its own surrogate return and surrogate costs.
To summarise, we provide the following algorithm that guarantees both safety constraints satisfaction and monotonic performance improvement. π π 𝑘 ( 𝑠, a
[Equation]
[Equation]
𝛾
[Equation] , and 𝜈 𝑖𝑗 = 4 𝛾 max 𝑠,𝑎𝑖 [Equation]
𝐴 𝑖𝑗,
[Equation]
𝑘 ( 𝑠
[Equation]
, ∀ 𝑖
[Equation]
N , 𝑗
[Equation]
[Equation]
[Equation]
𝑖
[Equation]
Draw a permutaion 𝑖 1: 𝑛 of agents at random.
6: for ℎ = 1 : 𝑛 do 7:
Compute the radius of the KL-constraint 𝛿 𝑖 ℎ // see Appendix B for the setup of 𝛿 𝑖 ℎ .
Make an update 𝜋
[Equation]
argmax
[Equation]
where Π 𝑖 ℎ is a subset of safe policies of agent 𝑖 ℎ , given by
[Equation]
9: end for 10: end for
In the above algorithm, in addition to sequentially maximising agents’ surrogate returns, the agents must assure that their surrogate costs stay below the corresponding safety thresholds.
Meanwhile, they have to constrain their policy search to small local neighbourhoods ( w.r.t max-KL distance).
As such, Algorithm 1 demonstrates two desirable properties: reward performance improvement and satisfaction of safety constraints, which we justify in the following theorem.
Theorem 1.
If a sequence of joint policies ( π 𝑘 ) ∞ 𝑘 = 0 is obtained from Algorithm 1, then it has the monotonic improvement property, 𝐽 ( π 𝑘 + 1 ) ≥ 𝐽 ( π 𝑘 ) , as well as it satisﬁes the safety constraints, 𝐽 𝑖𝑗 ( π 𝑘 ) ≤ 𝑐 𝑖𝑗 , for all 𝑘 ∈ ℕ , 𝑖 ∈ N , and 𝑗 ∈ { 1 , . . . , 𝑚 𝑖 } .
See proof in Appendix B.
The above theorem assures that agents that follow Algorithm 1 will only explore safe policies; meanwhile, every new policy will be guaranteed to result in performance improvement.
These two properties hold under the conditions that only restrictive policy updates are made; this is due to the KL-penalty term in every agent’s objective (i.e., 𝜈𝐷 maxKL ( 𝜋 𝑖 ℎ 𝑘 , 𝜋 𝑖 ℎ ) ), as well as the constraints on cost surrogates (i.e., the conditions in Π 𝑖 ℎ ).
In practice, it can be intractable to evaluate 𝐷 KL (cid:0) 𝜋 𝑖 ℎ 𝑘 (·| 𝑠 ) , 𝜋 𝑖 ℎ (·| 𝑠 ) (cid:1) at every state in order to compute 𝐷 maxKL ( 𝜋 𝑖 ℎ 𝑘 , 𝜋 𝑖 ℎ ) .
In the following subsections, we describe how we can approximate Algorithm 1 in the case of parameterised policies, similar to TRPO/PPO implementations (Schulman et al., 2015; 2017).
4. MACPO: M ULTI -A GENT C ONSTRAINED P OLICY O PTIMISATION
Here we focus on the practical settings where large state and action spaces prevent agents from designating policies 𝜋 𝑖 (·| 𝑠 ) for each state separately.
To handle this, we parameterise each agent’s 𝜋 𝑖𝜃 𝑖 by a neural network 𝜃 𝑖 .
Correspondingly, the joint policies π θ are parametrised by θ =
( 𝜃 1 , . .
, 𝜃 𝑛 ) .
Let’s recall that at every iteration of Algorithm 1, every agent 𝑖 ℎ maximises its surrogate return with a KL-penalty, subject to surrogate cost constraint.
Yet, direct computation of the max-KL constraint is intractable in practical settings, as it would require computation of KL-divergence at every single state.
Instead, one can relax it by adopting a form of expected KL-constraint 𝐷 KL ( 𝜋 𝑖 ℎ 𝑘 , 𝜋 𝑖 ℎ ) ≤ 𝛿 where 𝐷 KL ( 𝜋 𝑖 ℎ 𝑘 , 𝜋 𝑖 ℎ ) (cid:44) 𝔼 s ∼ 𝜌 π 𝑘 (cid:2) 𝐷 KL ( 𝜋 𝑖 ℎ 𝑘 (·| s ) , 𝜋 𝑖 ℎ (·| s )) (cid:3) .
Such an expectation can be approximated by stochastic sampling.
As a result, the optimisation problem solved by agent 𝑖 ℎ is written as
[Equation]
.t.
[Equation]
and 𝐷 KL
[Equation]
We can further approximate Equation (3) by Taylor expansion of the optimisation objective and cost constraints up to the ﬁrst order, and the KL-divergence up to the second order.
Consequently, the optimisation problem can be written as
[Equation]
and
[Equation]
where g 𝑖 ℎ is the gradient of the objective of agent 𝑖 ℎ in Equation (3), 𝑑 𝑖𝑗 =
𝐽 𝑖𝑗 ( π θ 𝑘 ) − 𝑐 𝑖𝑗 , and
H 𝑖 ℎ = ∇ 2 𝜃 𝑖ℎ 𝐷 KL ( 𝜋 𝑖 ℎ 𝜃 𝑖ℎ𝑘 , 𝜋 𝑖 ℎ ) (cid:12)(cid:12) 𝜃 𝑖ℎ =
𝜃 𝑖ℎ𝑘 is the Hessian of the average KL divergence of agent 𝑖 ℎ , and b 𝑖 ℎ 𝑗 is the gradient of agent of the 𝑗 th constraint of agent 𝑖 ℎ .
Similar to Chow et al.
(2017) and Achiam et al.
(2017), one can take a primal-dual optimisation approach to solve the linear quadratic optimisation in Equation (4).
Speciﬁcally, the dual form can be written as:
[Equation]
where r 𝑖 ℎ (cid:44) ( g 𝑖 ℎ )
[Equation]
Given the solution to the dual form in Equation (5), i.e., 𝜆 𝑖 ℎ ∗ and v 𝑖 ℎ ∗ , the solution to the primal problem in Equation (4) can thus be written by
[Equation]
In practice, we use backtracking line search starting at 1 / 𝜆 𝑖 ℎ ∗ to choose the step size of the above update.
Furthermore, we note that the optimisation step in Equation (4) is an approximation to the original problem from Equation (3); therefore, it is possible that an infeasible policy 𝜋 𝜃 𝑖ℎ𝑘 + 1 will be generated.
Fortunately, as the policy optimisation takes place in the trust region of 𝜋 𝑖 ℎ 𝜃 𝑖ℎ𝑘 , the size of update is small, and a feasible policy can be easily recovered.
In particular, for problems with one safety constraint, i.e., 𝑚 𝑖 ℎ = 1 , one can recover a feasible policy by applying a TRPO step on the cost surrogate, written as
[Equation]
where 𝛼 𝑗 is adjusted through backtracking line search.
To put it together, we refer to this algorithm as MACPO , and provide its pseudocode in Appendix D.
4. MAPPO-L AGRANGIAN
In addition to MACPO, one can use Lagrangian multipliers in place of optimisation with linear and quadratic constraints to solve Equation (3).
The Lagrangian method is simple to implement, and it does not require computations of the Hessian H 𝑖 ℎ whose size grows quadratically with the dimension of the parameter vector 𝜃 𝑖 ℎ .
Before we proceed, let us brieﬂy recall the optimisation procedure with a Lagrangian multiplier.
Suppose that our goal is to maximise a bounded real-valued function 𝑓 ( 𝑥 ) under a constraint 𝑔 ( 𝑥 ) ; max 𝑥 𝑓 ( 𝑥 ) , s.t. 𝑔 ( 𝑥 ) ≤ 0 .
We can introduce a scalar variable 𝜆 and reformulate the optimisation by
[Equation]
Suppose that 𝑥 + satisﬁes 𝑔 ( 𝑥 + ) > 0 .
This immediately implies that − 𝜆𝑔 ( 𝑥 + ) →
−∞ , as 𝜆 → +∞ , and so Equation (7) equals −∞ for 𝑥
= 𝑥
+ .
On the other hand, if 𝑥 − satisﬁes 𝑔 ( 𝑥 − ) ≤ 0 , we have that − 𝜆𝑔 ( 𝑥 − ) ≥ 0 , with equality only for 𝜆 = 0 .
In that case, the optimisation objective’s value equals 𝑓 ( 𝑥 − ) >
−∞ .
Hence, the only candidate solutions to the problem are those 𝑥 that satisfy the constraint 𝑔 ( 𝑥 ) ≤ 0 , and the objective matches with 𝑓 ( 𝑥 ) .
We can employ the above trick to the constrained optimisation problem from Equation (3) by subsuming the cost constraints into the optimisation objective with Lagrangian multipliers.
As such, agent 𝑖 ℎ computes ¯ 𝜆 𝑖 ℎ 1: 𝑚 𝑖ℎ and 𝜃 𝑖 ℎ 𝑘
+ 1 to solve the following min-max optimisation problem
[Equation]
[Equation]
Although the objective from Equation (8) is afﬁne in the Lagrangian multipliers 𝜆 𝑖 ℎ 𝑢 ( 𝑢 = 1 , . .
𝑚 𝑖 ℎ )
[Equation]
which enables gradient-based optimisation solutions
[Equation]
computing the KL-divergence constraint still complicates the overall process.
To handle this
[Equation]
one can further simplify it by adopting the PPO-clip objective (Schulman et al., 2017), which enables replacing the KL-divergence constraint with the clip operator
[Equation]
and update the policy parameter with ﬁrst-order methods.
We do so by deﬁning
[Equation]
and rewriting the Equation (8) as
[Equation]
The objective in Equation (9) takes a form of an expectation with quadratic constraint on the policy.
Up to the error of approximation of KL-constraint with the clip operator
[Equation]
it can be equivalently transformed into an optimisation of a clipping objective.
Finally
[Equation]
the objective takes the form of
[Equation]
[Figure]
The clip operator replaces the policy ratio with 1 − 𝜖 , or 1 + 𝜖 , depending on whether its value is below or above the threshold interval.
As such, agent 𝑖 ℎ can learn within its trust region by updating 𝜃 to maximise Equation (10), while the Lagrangian multipliers are updated towards the direction opposite to their gradients of Equation (8), which can be computed analytically.
We refer to this algorithm as MAPPO-Lagrangian , and give a detailed pseudocode of it in Appendix E.
E XPERIMENTS
Although MARL researchers have long had a variety of environments to test different algorithms, such as StarCraftII (Samvelyan et al., 2019) and Multi-Agent MuJoCo (Peng et al., 2020), no public safe MARL benchmark has been proposed; this impedes researchers from evaluating and benchmarking safety-aware multi-agent learning methods.
As a key contribution of this paper, we introduce Safe Multi-Agent MuJoCo Benchmark , a safety-aware extension of the MuJoCo environment that is designed for safe MARL research.
We show example tasks in Figure 1, in our environment, safety-aware agents have to learn not only skilful manipulations of a robot, but also to avoid crashing into unsafe obstacles and positions.
For more details of the setup, please refer to Appendix F.
We use Safe MAMuJoCo to examine if the MACPO/MAPPO-Lagrangian agents can satisfy their safety constraints and cooperatively learn to achieve high rewards, compared to existing
MARL algorithms.
Notably, our proposed methods adopt two different approaches for achieving safety.
MACPO reaches safety via hard constraints and backtracking line search, while MAPPOLagrangian maintains a rather soft safety awareness by performing gradient descents on the PPOclip objective.
Figure 2 shows cost and reward performance comparisons between MACPO,
MAPPO-Lagrangian, MAPPO (Yu et al., 2021), IPPO (de Witt et al., 2020), and HAPPO (Kuba et al., 2021a) algorithms on three challenging tasks.
Figure 2 should be interpreted at three-folds; each subﬁgure represents a different robot, within each subﬁgure, three task setups in terms of multiagent control are considered, for each task, we plot the cost curves (the lower the better) in the upper row, and plot the reward curves (the higher the better) in the bottom row.
Detailed hyperparameter settings are described in Appendix I.
The experiments reveal that both MACPO and MAPPO-Lagrangian quickly learn to satisfy safety constraints, and keep their explorations within the feasible policy space.
This stands in contrast to
IPPO, MAPPO, and HAPPO which largely violate the constraints thus being unsafe.
Furthermore, our algorithms achieve comparable reward scores; both methods are often better than IPPO.
In general, the performance (in terms of reward) of MAPPO-Lagrangian is better than of MACPO; moreover, MAPPO-Lagrangian outperforms the unconstrained MAPPO on challenging Ant tasks.
We note that on none of the tasks the reward of HAPPO was exceeded though it is unsafe.
C ONCLUSION
In this paper, we tackled multi-agent policy optimisation problems with safety constraints.
Central to our ﬁndings is the safe multi-agent policy iteration procedure that attains theoretically-justiﬁed monotonic improvement guarantee and constraints satisfaction guarantee at every iteration during learning.
Based on this, we proposed two practical algorithms: MACPO and MAPPO-Lagrangian.
To demonstrate their effectiveness, we introduced a new benchmark suite of Safe Multi-Agent MuJoCo and compared our methods against strong MARL baselines.
Results show that both of our methods can signiﬁcantly outperform existing state-of-the-art methods such as IPPO, MAPPO and
HAPPO in terms of safety, meanwhile maintaining comparable performance in terms of reward.
[Figure]
[Figure]
A v e r a g e E p
i s o d
e C o s t
[Figure]
g [Figure] i
[Figure]
w [Figure] [Figure] [Figure]
[Figure]
[Figure] [Figure]
tasks of Safe ManyAgent Ant, Safe Ant, and Safe HalfCheetah in terms of cost (the ﬁrst row) and reward (the second row).
The safety constraint values are:for ManyAgent Ant, 0 . 2 for Ant, and 5 for HalfCheetah.
Our methods consistently achieve almost zero costs, thus satisfying safe constraints, on all tasks.
In terms of reward, our methods outperform
IPPO and MAPPO on some tasks but underperform HAPPO, which is also an unsafe algorithm. Appendices
A Proofs of preliminary results
B Auxiliary Results for Algorithm
C Auxiliary Results for Implementation of MACPO
D MACPO
E MAPPO-Lagrangian
F Safe Multi-Agent MuJoCo
G Ablation Experiments
H Experiments in Safe Many-Agent Ant Environments
I Details of Settings for Experiments
A P ROOFS OF PRELIMINARY RESULTS
Lemma 1 (Multi-Agent Advantage Decomposition, Kuba et al.
(2021b)) .
For any state 𝑠 ∈ S , subset of agents 𝑖 1: ℎ ⊆ N
[Equation]
and joint action a 𝑖 1: ℎ , the following identity holds
[Equation]
Proof.
We write the multi-agent advantage as in its deﬁnition
[Equation]
and expand it in a telescoping sum.
[Equation]
Lemma 2.
Let π and ¯ π be joint policies.
Let 𝑖 ∈ N be an agent
[Equation]
and 𝑗 ∈ { 1 , . .
, 𝑚 𝑖 } be an index of one of its costs.
The following inequality holds
[Equation]
Proof.
From the proof of Theorem 1 from Schulman et al.
(2015) (in particular, equations (41)-(45)), applied to joint policies π
and ¯ π , we conclude that
[Equation]
where
[Equation]
Using the inequality 𝐷 TV ( 𝑝, 𝑞 ) 2 ≤ 𝐷 KL ( 𝑝, 𝑞 ) (Pollard, 2000; Schulman et al., 2015), we obtain
[Equation]
where
[Equation]
Notice now that we have 𝔼 s ∼ 𝜌 π , a ∼ ¯ π
[Equation]
π
[Equation]
∼ 𝜌
[Equation]
a
[Equation]
∼
[Equation]
𝜋 𝑖
[Equation]
𝐴 𝑖𝑗, π ( s , a 𝑖 ) (cid:105) , and 𝐷
[Equation]
𝐷
[Equation]
𝑠
[Equation]
𝑙
[Equation]
≤ 𝑛
[Equation]
𝑙 =max
[Equation]
𝑛
[Equation]
[Equation] Setting [Equation] , we ﬁnally obtain
[Equation]
B A UXILIARY R ESULTS FOR A LGORITHM
Remark 1.
In Algorithm 1, we compute the size of KL constraint as
[Equation]
Note that 𝛿 𝑖 1 (i.e., ℎ = 1 ) is guaranteed to be non-negative if π 𝑘 satisﬁes safety constraints; that is because then 𝑐 𝑖 𝑙 𝑗 ≥ 𝐽 𝑖 𝑙 𝑗 ( π 𝑘 ) for all 𝑙 and 𝑗 , and the set { 𝑙 | 𝑙<ℎ } is empty.
This formula for 𝛿 𝑖 ℎ , combined with Lemma 2, assures that the policies 𝜋 𝑖 ℎ within 𝛿 𝑖 ℎ max-KL distance from 𝜋 𝑖 ℎ 𝑘 will not violate other agents’ safety constraints, as long as the base joint policy
π 𝑘 did not violate them (which assures 𝛿 𝑖 1 ≥ 0 ).
To see this, notice that for every 𝑙 = 1 , . . .
, ℎ − 1 , and 𝑗 = 1 , . .
. , 𝑚 𝑙 ,
[Equation]
implies 𝐽 𝑖
[Equation]
By Lemma 2, the left-hand side of the above inequality is an upper bound of 𝐽 𝑖 𝑙 𝑗 ( π 𝑖 1: ℎ − 1 𝑘 + 1 , 𝜋 𝑖 ℎ , π − 𝑖 1: ℎ 𝑘 ) , which implies that the update of agent 𝑖 ℎ doesn’t violate the constraint of 𝐽 𝑖 𝑙 𝑗 .
The fact that the constraints of 𝐽 𝑖 𝑙 𝑗 for 𝑙 ≥ ℎ + 1 are not violated, i.e.,
[Equation]
is analogous.
Theorem 1.
If a sequence of joint policies ( π 𝑘 ) ∞ 𝑘 = 0 is obtained from Algorithm 1, then it has the monotonic improvement property, 𝐽 ( π 𝑘 + 1 ) ≥ 𝐽 ( π 𝑘 ) , as well as it satisﬁes the safety constraints, 𝐽 𝑖𝑗 ( π 𝑘 ) ≤ 𝑐 𝑖𝑗 , for all 𝑘 ∈ ℕ , 𝑖 ∈ N , and 𝑗 ∈ { 1 , . . . , 𝑚 𝑖 } .
Proof.
Safety constraints are assured to be met by Remark 1.
It sufﬁces to show the monotonic improvement property.
Notice that at every iteration 𝑘 of Algorithm 1, 𝜋 𝑖 ℎ 𝑘 ∈ Π 𝑖 ℎ .
Clearly 𝐷 maxKL ( 𝜋 𝑖 ℎ 𝑘 , 𝜋 𝑖 ℎ 𝑘 ) = 0 ≤ 𝛿 𝑖 ℎ .
Moreover,
[Equation]
where the inequality is guaranteed by updates of previous agents [Equation] as described in Remark 1 (Inequality 12).
By Theorem 1 from Schulman et al.
[Equation] we have
[Equation]
which by Equation 11 is lower-bounded by
[Equation]
which by Lemma 1 equals
[Equation]
and as for every ℎ [Equation] 𝜋 𝑖 ℎ 𝑘 + 1 is the argmax
[Equation]
this is lower-bounded by
[Equation]
which, as follows from Deﬁnition
[Equation]
equals
[Equation]
0 =
𝐽 ( π 𝑘 ) , which ﬁnishes the proof.
(cid:3)
C A UXILIARY R ESULTS FOR I MPLEMENTATION OF MACPO
Theorem 2.
The solution to the following problem
[Equation]
where g , b , x ∈ ℝ 𝑛 , 𝑐, 𝛿 ∈ ℝ , 𝛿 > 0 , H ∈ 𝕊 𝑛 , and H (cid:31) 0 .
When there is at least one strictly feasible point, the optimal point x ∗ satisﬁes:
[Equation]
where 𝜆 ∗ and 𝑣 ∗ are deﬁned by
[Equation]
otherwise where q = g 𝑇 H − 1 g , r = g 𝑇 H − 1 b , and s = b 𝑇 H − 1 b .
Furthermore, let Λ 𝑎 (cid:44) { 𝜆 | 𝜆𝑐 − 𝑟 > 0 , 𝜆 ≥ 0 } , and Λ 𝑏 (cid:44) { 𝜆 | 𝜆𝑐 − 𝑟 ≤ 0 , 𝜆 ≥ 0 } .
The value of 𝜆 ∗ satisﬁes
[Equation]
[Equation] where 𝜆 ∗ = 𝜆 𝑎 ∗ if 𝑓 𝑎 (cid:0) 𝜆 𝑎 ∗ (cid:1)
>
𝑓 𝑏 (cid:0) 𝜆 𝑏 ∗ (cid:1) and 𝜆 ∗ = 𝜆 ∗ otherwise, and Proj ( 𝑎, S ) is the projection of a point 𝑥 on to a set S .
Note the projection of a point 𝑥 ∈ ℝ onto a convex segment of ℝ , [ 𝑎, 𝑏 ] , has value Proj ( 𝑥, [ 𝑎, 𝑏 ]) =
max ( 𝑎, min ( 𝑏, 𝑥 )) .
Proof.
See Achiam et al.
(2017) (Appendix 10.2).
(cid:3)
D
MACPO
− do
Collect a set θ 𝑘 𝜋 1 𝜃 1 𝑘 , . .
. , 𝜋 𝑛𝜃 𝑛𝑘 into a minibatch
Compute advantage function ˆ 𝐴 ( s , a ) based on global V-value network with GAE.
Compute cost-advantage functions ˆ 𝐴 𝑖𝑗 ( s , a 𝑖 ) based on individual 𝑉
𝑖 -cost critics with GAE.
Draw a random permutation of agents 𝑖 1: 𝑛 .
Set 𝑀 𝑖 1 ( s , a )
=
ˆ 𝐴 ( s , a ) .
11: for agent 𝑖 ℎ = 𝑖 1 , . . .
, 𝑖
𝑛 do 12:
Estimate the gradient of the agent’s maximisation objective
[Equation]
13: for 𝑗 = 1 , . . .
, 𝑚 𝑖 ℎ do 14:
Estimate the gradient of the agent’s 𝑗 th cost
[Equation]
15: end for 16:
Set
[Equation]
Compute ˆ H 𝑖 ℎ 𝑘 , the Hessian of the average KL-divergence
[Equation]
Solve the dual (5) for 𝜆 𝑖 ℎ ∗ , v 𝑖 ℎ ∗ .
Use the conjugate gradient algorithm to compute the update direction
[Equation]
Update agent 𝑖 ℎ ’s policy by
[Equation]
where 𝑗 ∈ { 0 , 1 , . . .
, 𝐿 } is the smallest such 𝑗 which improves the sample loss, and satisﬁes the sample constraints, found by the backtracking line search.
20: if the approximate is not feasible then 21:
Use equation (6) to recover policy 𝜃 𝑖 ℎ 𝑘 + 1 from unfeasible points.
22: end if 23:
Compute 𝑀
[Equation]
24: end for 25:
Update V-value network by following formula: 26:
[Equation]
27: end for
E MAPPO-L AGRANGIAN
θ 𝑘 =
1 𝜃 1 𝑘 , . .
. , 𝑛𝑘 )
Set 𝑀 𝑖 1 ( s , a )
=
ˆ 𝐴 ( s , a
11: for agent 𝑖 ℎ = do 12:
Initialise a policy parameter 𝜃 𝑖 ℎ =
𝜃 𝑖 ℎ 𝑘 , and Lagrangian multipliers 𝜆 𝑖 ℎ 𝑗 = 0 , ∀ 𝑗 = 1 , . . .
, 𝑚 𝑖 ℎ .
Make the Lagrangian modiﬁcation step of objective construction 𝑀
[Equation]
[Equation] [Equation]
[Equation]
𝑗 =𝜆 𝑖 ℎ 𝑗 ˆ 𝐴 𝑖 ℎ 𝑗 ( s 𝑡 , a 𝑖 ℎ 𝑡 ) .
14: for 𝑒 = 1 , . . .
, 𝑒 PPO do 15:
Differentiate the Lagrangian PPO-Clip objective Δ 𝜃 𝑖ℎ = ∇ 𝜃 𝑖ℎ 1 𝐵 𝐵 (cid:205) 𝑏 =𝑇 (cid:205)
𝑡 = 0 min (cid:169)(cid:173) (cid:171) 𝜋 𝑖ℎ 𝜃𝑖ℎ (cid:16) 𝑎 𝑖ℎ𝑡 | 𝑜 𝑖ℎ𝑡 (cid:17) 𝜋 𝑖ℎ 𝜃𝑖ℎ𝑘 (cid:16) 𝑎 𝑖ℎ𝑡 | 𝑜 𝑖ℎ𝑡 (cid:17)
𝑀 𝑖 ℎ , ( 𝜆 ) ( 𝑠 𝑡 , a 𝑡 ) , clip (cid:169)(cid:173) (cid:171) 𝜋 𝑖ℎ 𝜃𝑖ℎ (cid:16) 𝑎 𝑖ℎ𝑡 | 𝑜 𝑖ℎ𝑡 (cid:17) 𝜋 𝑖ℎ 𝜃𝑖ℎ𝑘 (cid:16) 𝑎 𝑖ℎ𝑡 | 𝑜 𝑖ℎ𝑡 (cid:17) , 1 ± 𝜖 (cid:170)(cid:174) (cid:172)
𝑀 𝑖 ℎ , ( 𝜆 ) ( 𝑠 𝑡 , a 𝑡 ) (cid:170)(cid:174) (cid:172) .
Update temprorarily the actor paramaters 𝜃 𝑖 ℎ ← 𝜃 𝑖 ℎ + 𝛼 𝜃 Δ 𝜃 𝑖ℎ .
17: for 𝑗 = 1 , . . .
, 𝑚 𝑖 ℎ do 18:
Approximate the constraint violation 𝑑 𝑖 ℎ 𝑗 = 1 𝐵𝑇 𝐵 (cid:205)
𝑏 =𝑇 (cid:205) 𝑡
=ˆ 𝑉 𝑖 ℎ 𝑗 ( 𝑠 𝑡 ) − 𝑐 𝑖 ℎ 𝑗 .
Differentiate the constraint Δ 𝜆 𝑖 ℎ 𝑗 = − 1 𝐵 𝐵 (cid:205) 𝑏 = 1 (cid:169)(cid:173) (cid:171) 𝑑 𝑖 ℎ 𝑗
1 − 𝛾 ) + 𝑇 (cid:205)
𝑡 =𝜋 𝑖ℎ 𝜃𝑖ℎ ( 𝑎 𝑖ℎ𝑡 | 𝑜 𝑖ℎ𝑡 )
𝜋 𝑖ℎ 𝜃𝑖ℎ𝑘 ( 𝑎 𝑖ℎ𝑡 | 𝑜 𝑖ℎ 𝑡 )
ˆ 𝐴 𝑖 ℎ 𝑗 ( 𝑠 𝑡 , 𝑎 𝑖 ℎ 𝑡 ) (cid:170) (cid:174) (cid:172) .
20: end for 21: for 𝑗 = 1 , . . .
, 𝑚 𝑖 ℎ do 22:
Update temporarily the Lagrangian multiplier 𝜆 𝑖 ℎ 𝑗 ← ReLU (cid:16) 𝜆 𝑖 ℎ 𝑗 − 𝛼 𝜆 Δ 𝜆 𝑖 ℎ 𝑗 (cid:17) .
23: end for 24: end for 25:
Update the actor parameter 𝜃 𝑖 ℎ
[Equation]
Compute
[Equation]
27: end for 28:
Update V-value network (and V-cost networks analogously) by following formula: 29:
[Equation]
30: end for
[Equation]
Safe MAMuJoCo is an extension of MAMuJoCo (Peng et al., 2020).
In particular, the background environment, agents, physics simulator, and the reward function are preserved.
However, as oppose to its predecessor, Safe MAMuJoCo environments come with obstacles, like walls or bombs.
Furthermore, with the increasing risk of an agent stumbling upon an obstacle, the environment emits cost (Brockman et al., 2016).
According to the scheme from Zanger et al.
(2021), we characterise the cost functions for each task below.
[Equation] The width of the corridor set by two walls is 9 𝑚 (ManyAgent Ant), The width of the corridor set by three folding line walls with an angle of 30 degrees is 10 𝑚 (Ant).
The environment emits the cost of 1 for an agent, if the distance between the robot and the wall is less than 1 . 8 𝑚 , or when the robot topples over.
This can be described as
[Equation]
where z torso ,𝑡 + 1 is the robot’s torso’s 𝑧 -coordinate [Equation] and x torso ,𝑡 + 1 is the robot’s torso’s 𝑥
-coordinate, at time 𝑡 + 1 ; x wall is the 𝑥 -coordinate of the wall.
H UMANOID & M ANY A GENT S WIMMER
In both tasks the width of the corridor built from two walls is 4 .
6 𝑚 .
The environment emits cost of 1 if the robot is closer than 1 . 8 𝑚 from the wall. c
[Equation] for [Equation] .1 , otherwise .
[Figure]
In these tasks, the agents move inside a corridor (which constraints their movement, but does not induce costs).
Together with them, there are bombs moving inside the corridor.
If an agent ﬁnds itself too close to a bomb, the distance between an agent and a bomb is less than 9 𝑚 , a cost of 1 will be emitted.
[Equation]
where y torso ,𝑡 + 1 is the 𝑦 -coordinate of the robot’s torso, and y obstacle is the 𝑦 -coordinate of the moving obstacle.
G A BLATION E XPERIMENTS
Here we conduct the ablation experiments, studying the effect of different values of the Lagrangian stepsize ( 𝛼 𝜆 in Algorithm 3).
[Figure]
We provide additional results on the Safe Many-Agent ant tasks. The width of the corridor is 12 𝑚 ; its walls fold at the angle of 30 degrees. The environment emits the cost of 1 for an agent, if the distance between the robot and the wall is less than 1 . 8 𝑚 , or when the robot topples over.
This can be described as c 𝑡 = (cid:26) 0 , for 0 . 2 ≤ z torso ,𝑡 + 1 ≤ 1 . 0 and (cid:13)(cid:13) x torso ,𝑡 + 1 − x wall (cid:13)(cid:13) 2 ≥ 1 .1 , otherwise .
where z torso ,𝑡 +
[Figure]
I D ETAILS OF S ETTINGS FOR E XPERIMENTS
In this section, we introduce the details of settings for our experiments.
The code is available at https://github.com/Anonymous
[Table]
ICLR2022/
[Table]
[Table]
[Table]