H ETEROGENEOUS -A GENT M IRROR L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
The necessity for cooperation among independent intelligent machines has popularised cooperative multi-agent reinforcement learning (MARL) in the artificial intelligence (AI) research community.
However, many research endeavours have been focused on developing practical MARL algorithms whose effectiveness has been studied only empirically, thereby lacking theoretical guarantees.
As recent studies have revealed, MARL methods often achieve performance that is unstable in terms of reward monotonicity or suboptimal at convergence.
To resolve these issues, in this paper,we introduce a novel framework named Heterogeneous-Agent
Mirror Learning (HAML) that provides a general template for MARL actor-critic algorithms.
We prove that algorithms derived from the HAML template satisfy the desired properties of the monotonic improvement of the joint reward and the convergence to Nash equilibrium .
We verify the practicality of HAML by proving that the current state-of-the-art cooperative MARL algorithms, HATRPO and HAPPO, are in fact HAML instances.
Next, as a natural outcome of our theory, we propose HAML extensions of two well-known RL algorithms, HAA2C (for A2C) and HADDPG (for DDPG), and demonstrate their effectiveness against strong baselines on StarCraftII and Multi-Agent MuJoCo tasks.
I NTRODUCTION
While the policy gradient (PG) formula has been long known in the reinforcement learning (RL) community (Sutton et al., 2000), it has not been until trust region learning (Schulman et al., 2015a) that deep RL algorithms started to solve complex tasks such as real-world robotic control successfully.
Nowadays, methods that followed the trust-region framework, including TRPO (Schulman et al., 2015a)
, PPO (Schulman et al., 2017) and their extensions (Schulman et al., 2015b; Hsu et al., 2020), became effective tools for solving challenging AI problems (Berner et al., 2019).
It was believed that the key to their success are the rigorously described stability and the monotonic improvement property of trust-region learning that they approximate.
This reasoning, however, would have been of limited scope since it failed to explain why some algorithms following it ( e.g. PPO-KL) largely underperform in contrast to success of other ones ( e.g. PPO-clip) (Schulman et al., 2017).
Furthermore, the trustregion interpretation of PPO has been formally rejected by recent studies both empirically (Engstrom et al., 2020) and theoretically (Wang et al., 2020); this revealed that the algorithm violates the trust-region constraints—it neither constraints the KL-divergence between two consecutive policies, nor does it bound their likelihood ratios.
These findings have suggested that, while the number of available RL algorithms grows, our understanding of them does not, and the algorithms often come without theoretical guarantees either.
Only recently, Kuba et al.
(2022b) showed that the well-known algorithms, such as PPO, are in fact instances of the so-called mirror learning framework, within which any induced algorithm is theoretically sound.
On a high level, methods that fall into this class optimise the mirror objective , which shapes an advantage surrogate by means of a drift functional —a quasi-distance between policies.
Such an update provably leads them to monotonic improvements of the return, as well as the convergence to the optimal policy.
The result of mirror learning offers RL researchers strong confidence that there exists a connection between an algorithm’s practicality and its theoretical properties and assures soundness of the common RL practice.
While the problem of the lack of theoretical guarantees has been severe in RL, in multi-agent reinforcement learning (MARL) it has only been exacerbated.
Although the PG theorem has been successfully extended to the multi-agent PG (MAPG) version (Zhang et al., 2018), it has only recently been shown that the variance of MAPG estimators grows linearly with the number of agents (Kuba et al., 2021).
Prior to this, however, a novel paradigm of centralised training for decentralised execution (CTDE) (Foerster et al., 2018; Lowe et al., 2017b) greatly alleviated the difficulty of the multi-agent learning by assuming that the global state and opponents’ actions and policies are accessible during the training phase; this enabled developments of practical MARL methods by merely extending single-agent algorithms’ implementations to the multi-agent setting.
As a result, direct extensions of TRPO (Li & He, 2020) and PPO (de Witt et al., 2020a; Yu et al., 2021) have been proposed whose performance, although is impressive in some settings, varies according to the version used and environment tested against.
However, these extensions do not assure the monotonic improvement property or convergence result of any kind (Kuba et al., 2022a).
Importantly, these methods can be proved to be suboptimal at convergence in the common setting of parameter sharing (Kuba et al., 2022a) which is considered as default by popular multi-agent algorithms (Yu et al., 2021) and popular multi-agent benchmarks such as SMAC (Samvelyan et al., 2019) due to the computational convenience it provides.
In this paper, we resolve these issues by proposing Heterogeneous-Agent Mirror Learning (HAML)— a template that can induce a continuum of cooperative MARL algorithms with theoretical guarantees for monotonic improvement as well as Nash equilibrium (NE) convergence.
The purpose of HAML is to endow MARL researchers with a template for rigorous algorithmic design so that having been granted a method’s correctness upfront, they can focus on other aspects, such as effective implementation through deep neural networks.
We demonstrate the expressive power of the HAML framework by showing that two of existing state-of-the-art (SOTA) MARL algorithms, HATRPO and
HAPPO (Kuba et al., 2022a), are rigorous instances of HAML.
This stands in contrast to viewing them as merely approximations to provably correct multi-agent trust-region algorithms as which they were originally considered.
Furthermore, although HAML is mainly a theoretical contribution, we can naturally demonstrate its usefulness by using it to derive two heterogeneous-agent extensions of successful RL algorithms: HAA2C (for A2C (Mnih et al., 2016)) and HADDPG (for DDPG (Lillicrap et al., 2015)), whose strength are demonstrated on benchmarks of StarCraftII (SMAC) (de Witt et al., 2020a) and Multi-Agent MuJoCo (de Witt et al., 2020b) against strong baselines such as MADDPG (Lowe et al., 2017b) and MAA2C (Papoudakis et al., 2021).
P ROBLEM F ORMULATION
We formulate the cooperative MARL problem as cooperative Markov game (Littman, 1994) defined by a tuple ⟨N , S , A , r, P, γ, d ⟩ .
Here, N = { 1 , . .
. , n } is a set of n agents, S is the state space,
A = × ni =
A i is the products of all agents’ action spaces, known as the joint action space.
Although our results hold for general compact state and action spaces, in this paper we assume that they are finite, for simplicity.
Further, r : S ×
A → R is the joint reward function, P : S ×
A × S → [0 , 1] is the transition probability kernel, γ ∈ [0 , 1) is the discount factor, and d ∈ P ( S ) (where P ( X ) denotes the set of probability distributions over a set X ) is the positive initial state distribution.
In this work, we will also use the notation P ( X ) to denote the power set of a set X .
At time step t ∈ N , the agents are at state s t (which may not be fully observable); they take independent actions a i t , ∀ i ∈ N drawn from their policies π i ( · i | s t ) ∈ P
( A i ) , and equivalently, they take a joint action a t = ( a 1 t , . . .
, a nt ) drawn from their joint policy π ( ·| s t ) = (cid:81) ni
{+=
+} π i ( · i | s t ) ∈ P ( A ) .
We write Π i ≜ {× s ∈S π i ( · i | s ) |∀ s ∈ S , π i ( · i | s ) ∈ P ( A i ) } to denote the policy space of agent i , and Π ≜ (Π 1 , . . .
Π n ) to denote the joint policy space.
It is important to note that when π i ( · i | s ) is a
Dirac delta ditribution, the policy is referred to as deterministic (Silver et al., 2014) and we write µ i ( s ) to refer to its centre.
Then, the environment emits the joint reward r ( s t , a t ) and moves to the next state s t +1 ∼ P ( ·| s t , a t ) ∈ P ( S ) .
The initial state distribution d , the joint policy π , and the transition kernel P induce the (improper) marginal state distribution ρ π ( s ) ≜ (cid:80) ∞ t =
γ t Pr ( s t = s | d, π ) .
The agents aim to maximise the expected joint return, defined as
[Equation]
We adopt the most common solution concept for multi-agent problems which is that of Nash equilibria (Nash, 1951; Yang & Wang, 2020).
We say that a joint policy π NE ∈ Π is a NE if none of the agents can increase the joint return by unilaterally altering its policy.
More formally, π NE is a NE if ∀ i ∈ N , ∀ π
i ∈
Π i , J ( π i , π − i NE ) ≤ J ( π NE ) .
To study the problem of finding a NE, we introduce the following notions.
Let i 1: m =
( i 1 , . .
, i m ) ⊆
N be an ordered subset of agents.
We write − i 1: m to refer to its complement, and i and − i , respectively, when m = 1 .
We define the multi-agent state-action value function as
[Equation]
When m = n (the joint action of all agents is considered), then i 1: n ∈ Sym ( n ) , where Sym ( n ) denotes the set of permutations of integers 1 , . . . , n , known as the symmetric group .
In that case we write Q i 1: n π
( s, a i 1: n ) =
Q π ( s, a ) which is known as the (joint) state-action value function.
On the other hand, when m = 0 , i.e. , i 1: m = ∅ , the function takes the form V π ( s ) , known as the state-value function.
Consider two disjoint subsets of agents, j 1: k and i 1: m .
Then, the multi-agent advantage function of i 1: m with respect to j 1: k is defined as
[Equation]
As it has been shown in Kuba et al.
(2021), the multi-agent advantage function allows for additive decomposition of the joint advantge function by the means of the following lemma.
Lemma 1 (Multi-Agent Advantage Decomposition) .
Let π be a joint policy, and i 1 , . .
, i m be an arbitrary ordered subset of agents.
Then, for any state s and joint action a
i 1: m ,
[Equation]
Although the multi-agent advantage function has been discovered only recently and has not been studied thoroughly, it is this function and the above lemma that builds the foundation for the development of the HAML theory.
T HE S TATE OF A FFAIRS IN MARL
Before we review existing SOTA algorithms for cooperative MARL, we introduce two settings in which the algorithms can be implemented.
Both of them can be considered appealing depending on the application, but these pros also come with “traps” which, if not taken care of, may provably deteriorate an algorithm’s performance.
3. H OMOGENEITY vs. H ETEROGENEITY
The first setting is that of homogeneous policies, i.e. , those where agents all agents share one policy: π i = π, ∀N , so that
π = ( π, . . .
, π ) (de Witt et al., 2020a; Yu et al., 2021).
This approach enables a straightforward adoption of an RL algorithm to MARL, and it does not introduce much computational and sample complexity burden with the increasing number of agents.
Nevertheless, sharing one policy across all agents requires that their action spaces are also
the same, i.e. , A i =
A j , ∀ i, j ∈ N .
Furthermore, policy sharing prevents agents from learning different skills.
This scenario may be particularly dangerous in games with a large number of agents, as shown in Proposition 1, proved by
Kuba et al.
(2022a).
Proposition 1 (Trap of Homogeneity) .
Let’s consider a fully-cooperative game with an even number of agents n , one state, and the joint action space { 0 , 1 } n , where the reward is given by r ( 0 n/ 2 , 1 n/ 2 ) =
r ( 1 n/ 2 , 0 n/ 2 ) = 1 , and r ( a 1: n ) =
for all other joint actions.
Let J ∗ be the optimal joint reward, and J ∗ share be the optimal joint reward under the shared policy constraint.
Then
[Equation]
A more ambitious approach to MARL is to allow for heterogeneity of policies among agents, i.e. , to let π i and π j be different functions when i ̸ =
j ∈ N .
This setting has greater applicability as heterogeneous agents can operate in different action spaces.
Furthermore, thanks to this model’s flexibility they may learn more sophisticated joint behaviours.
Lastly, they can recover homogeneous policies as a result of training, if that is indeed optimal.
Nevertheless, training heterogeneous agents is highly non-trivial.
Given a joint reward, an individual agent may not be able to distill its own contibution to it—a problem known as credit assignment (Foerster et al., 2018).
Furthermore, even if an agent identifies its improvement direction, it may be conflicting with those of other agents, which then results in performance damage, as we exemplify in Proposition 2, proved in Appendix A.2.
Proposition 2 (Trap of Heterogeneity) .
Let’s consider a fully-cooperative game with 2 agents, one state, and the joint action space { 0 , 1 } 2 , where the reward is given by r (0 , 0)
= 0 , r (0 , 1)
=
r (1 , 0) = 2 , and r (1 , 1) = − 1 .
Suppose that π i old (0) > 0 .
6 for i = 1 , 2 .
Then, if agents i update their policies by
[Equation]
then the resulting policy will yield a lower return,
[Equation]
Consequently, these facts imply that homogeneous algorithms should not be applied to complex problems, but they also highlight that heterogeneous algorithms should be developed with extra cares.
In the next subsection, we describe existing SOTA actor-critic algorithms which, while often performing greatly, are still not impeccable, as they fall into one of the above two traps.
3. A S ECOND L OOK AT SOTA MARL A LGORITHMS
Multi-Agent Advantage Actor-Critic MAA2C (Papoudakis et al., 2021) extends the A2C (Mnih et al., 2016) algorithm to MARL by replacing the RL optimisation (single-agent policy) objective with the MARL one (joint policy),
[Equation]
which computes the gradient with respect to every agent i ’s policy parameters, and performs a gradient-ascent update for each agent.
This algorithm is straightforward to implement and is capable of solving simple multi-agent problems (Papoudakis et al., 2021).
We point out, however, that by simply following their own MAPG, the agents perform uncoordinated updates, thus getting caught by Proposition 2.
Furthermore, MAPG estimates suffer from large variance which grows linearly with the number of agents (Kuba et al., 2021), thus making the algorithm unstable.
To assure greater stability, the following MARL extensions of stable RL approaches have been developed.
Multi-Agent Deep Deterministic Policy Gradient MADDPG (Lowe et al., 2017a) is a MARL extension of the popular DDPG algorithm (Lillicrap et al., 2015).
At every iteration, every agent i updates its deterministic policy by maximising the following objective
[Equation]
) where β µ old is a state distribution that is not necessarily equivalent to ρ µ old , thus allowing for offpolicy training.
In practice, MADDPG maximises Equation (3) by a few steps of gradient ascent.
The main advantages of MADDPG include small variance of its MAPG estimates—a property granted by deterministic policies (Silver et al., 2014), as well as low sample complexity due to learning from off-policy data.
Such a combination gives the algorithm a strong performance in continuous-action tasks (Lowe et al., 2017a; Kuba et al., 2022a).
However, this method’s strengths also constitute its limitations—it is applicable to continuous problems only (discrete problems require categorical-distribution policies), and relies on large memory capacity to store the off-policy data.
Multi-Agent PPO MAPPO (Yu et al., 2021) is a relatively straightforward extension of PPO (Schulman et al., 2017) to MARL.
In its default formulation, the agents employ the trick of policy sharing described in the previous subsection.
As such, the policy is updated to maximise
[Equation]
where the clip ( · , 1 ± ϵ ) operator clips the input to 1 − ϵ / 1 + ϵ if it is below/above this value.
Such an operation removes the incentive for agents to make large policy updates, thus stabilising the training effectively.
Indeed, the algorithm’s performance on the StarCraftII benchmark is remarkable, and it is accomplished by using only on-policy data.
Nevertheless, the policy-sharing strategy limits the algorithm’s applicability and leads to its suboptimality, as we discussed in Proposition 1.
Trying to avoid this issue, one can implement the algorithm without policy sharing, thus making the agents simply take simultaneous PPO updates meanwhile employing a joint advantage estimator.
In this case, the updates are not coordinated, making MAPPO fall into the trap of Proposition 2.
In summary, all these algorithms do not possess performance guarantees.
Altering their implementation settings to escape one of the traps from Subsection 3.1 makes them, at best, fall into another.
This shows that the MARL problem introduces additional complexity into the single-agent RL setting, and needs additional care to be rigorously solved.
With this motivation, in the next section, we develop a theoretical framework for development of MARL algorithms with correctness guarantees.
H ETEROGENEOUS -A GENT M IRROR L EARNING
In this section, we introduce heterogeneous-agent mirror learning (HAML)—a template that includes a continuum of MARL algorithms which we prove to solve cooperative problems with correctness guarantees.
HAML is designed for the general and expressive setting of heterogeneous agents, thus avoiding Proposition 1, and it is capable of coordinating their updates, leaving Proposition 2 behind.
4. S ETTING UP HAML
We begin by introducing the necessary definitions of HAML attributes: the drift functional.
Definition 1.
Let i ∈ N , a heterogeneous-agent drift functional (HADF) D i,ν of i consists of a map, which is defined as
[Equation]
such that for all arguments, under notation D i π (cid:0) ˆ π i | s, ¯ π j 1: m (cid:1) ≜
D i
[Equation]
(cid:0) ˆ π i ( · i | s )
| ¯ π j
[Equation]
m , s (cid:1) ,
[Equation] [Equation]
(cid:0) ˆ π i | s, ¯ π j 1: m (cid:1) has all Gâteaux derivatives zero at ˆ π i =
π i (zero gradient), and a probability distribution ν
i π , ˆ π i ∈ P ( S ) over states that can (but does not have to) depend on π and ˆ π i , and such that the drift D i,ν of ˆ π i from π i with respect to ¯ π j 1: m , defined as
[Equation]
i,
[Equation]
, is continuous with respect to π , ¯ π j 1: m , and ˆ π i .
We say that the HADF is positive if D i,ν π (ˆ π i | ¯ π j 1: m ) =implies ˆ π
i
=
π i , and trivial if D i,ν π
(ˆ π i | ¯ π j 1: m )
=
for all π , ¯ π j 1: m , and ˆ π i , .
Intuitively, the drift D i,ν π (ˆ π i | ¯ π j 1: m ) is a notion of distance between π i and ˆ π i , given that agents j 1: m just updated to ¯ π j 1: m .
We highlight that, under this conditionality, the same update (from π i to ˆ π i ) can have different sizes—this will later enable HAML agents to softly constraint their learning steps in a coordinated way.
Before that, we introduce a notion that renders hard constraints, which may be a part of an algorithm design, or an inherrent limitation.
Definition 2.
Let i ∈ N .
We say that, U i : Π × Π i
→ P (Π i ) is a neighbourhood operator if ∀ π
i ∈
Π i ,
U i π ( π i ) contains a closed ball, i.e. , there exists a state-wise monotonically non-decreasing metric χ : Π i × Π i
→ R such that ∀ π i ∈
Π i there exists δ i > 0 such that χ
( π i , ¯ π i ) ≤ δ i = ⇒
¯ π i ∈
U
i π ( π i ) .
Throughout this work, for every joint policy π , we will associate it with its sampling distribution —a positive state distribution β π ∈ P ( S ) that is continuous in π (Kuba et al., 2022b).
With these notions defined, we introduce the main definition of the paper.
Definition 3.
Let i ∈ N , j 1: m ∈ P ( − i ) , and D i,ν be a HADF of agent i .
The heterogeneous-agent mirror operator (HAMO) integrates the advantage function as
[Equation]
We note two important facts.
First, when ν i π , ˆ π i = β π , the fraction from the front of the HADF in
HAMO disappears, making it only a difference between the advantage and the HADF’s evaluation.
Second, when ˆ π i =
π i , HAMO evaluates to zero.
Therefore, as the HADF is non-negative, a policy ˆ π i that improves HAMO must make it positive, and thus leads to the improvement of the multi-agent advantage of agent i .
In the next subsection, we study the properties of HAMO in more details, as well as use it to construct HAML—a general framework for MARL algorithm design.
4. T HEORETICAL P ROPERTIES OF HAML
It turns out that, under certain configuration, agents’ local improvements result in the joint improvement of all agents, as described by the lemma below.
Lemma 2 (HAMO Is All You Need) .
Let π old and π new be joint policies and let i 1: n ∈ Sym ( n ) be an agent permutation.
Suppose that, for every state s ∈ S and every m = 1 , . . . , n ,
[Equation]
Then, π new is jointly better than π old , so that for every state s ,
[Equation]
Subsequently, the monotonic improvement property of the joint return follows naturally, as
[Equation]
However, the conditions of the lemma require every agent to solve |S| instances of Inequality (5), which may be an intractable problem.
We shall design a single optimisation objective whose solution satisfies those inequalities instead.
Furthermore, to have a practical application to large-scale problems, such an objective should be estimatable via sampling.
To handle these challenges, we introduce the following Algorithm Template 1 which generates a continuum of HAML algorithms.
Algorithm Template 1: Heterogeneous-Agent Mirror Learning
Initialise a joint policy π 0 = π 10 , .
, π n 0 ) ; for k = 0 , 1 , . . .
do
Compute the advantage function A π k ( s, a ) for all state-(joint)action pairs ( s, a ) ; Draw a permutaion i 1: n of agents at random \\from a positive distribution p ∈
P ( Sym ( n )) ; for m = 1 : n do
Make an update π i m k [Equation] =
[Equation]
Output : A limit-point joint policy π ∞
Based on Lemma 2 and the fact that π i ∈ U
i π ( π i ) , ∀ i ∈ N , π i ∈
Π i , we can know any HAML algorithm (weakly) improves the joint return at every iteration.
In practical settings, such as deep
MARL, the maximisation step of a HAML method can be performed by a few steps of gradient ascent on a sample average of HAMO (see Definition 1).
We also highlight that if the neighbourhood operators U i can be chosen so that they produce small policy-space subsets, then the resulting updates will be not only improving but also small.
This, again, is a desirable property while optimising neural-network policies, as it helps stabilise the algorithm.
One may wonder why the order of agents in HAML updates is randomised at every iteration; this condition has been necessary to establish convergence to NE, which is intuitively comprehendable: fixed-point joint policies of this randomised procedure assure that none of the agents is incentivised to make an update, namely reaching a NE.
We provide the full list of the most fundamental HAML properties in Theorem 1 which shows that any method derived from Algorithm Template 1 solves the cooperative MARL problem.
Theorem 1 (The Fundamental Theorem of Heterogeneous-Agent Mirror Learning) .
Let, for every agent i ∈ N , D i,ν be a HADF,
U i be a neighbourhood operator, and let the sampling distributions
β π depend continuously on π .
Let π 0 ∈ Π , and the sequence of joint policies ( π k ) ∞ k =0 be obtained by a HAML algorithm induced by D i,ν , U
i , ∀ i ∈ N , and β π .
Then, the joint policies induced by the algorithm enjoy the following list of properties
Attain the monotonic improvement property, J
[Equation]
[Equation]
Their expected returns converge to a Nash return,
[Equation]
Their ω -limit set consists of Nash equilibria.
See the proof in Appendix C. With the above theorem, we can conclude that HAML provides a template for generating theoretically sound, stable, monotonically improving algorithms that enable agents to learn solving multi-agent cooperation tasks.
4. E XISTING HAML I NSTANCES : HATRPO AND HAPPO
As a sanity check for its practicality, we show that two SOTA MARL methods—HATRPO and
HAPPO (Kuba et al., 2022a)—are valid instances of HAML, which also provides an explanation for their excellent empirical performance.
We begin with an intuitive example of HATRPO, where agent i m (the permutation i 1: n is drawn from the uniform distribution) updates its policy so as to maximise (in ¯ π i m )
[Equation]
This optimisation objective can be casted as a HAMO with the trivial HADF D i m ,ν ≡ 0 , and the
KL-divergence neighbourhood operator
[Equation]
The sampling distribution used in HATRPO is β π = ρ π .
Lastly, as the agents update their policies in a random loop, the algorithm is an instance of HAML.
Hence, it is monotonically improving and converges to a Nash equilibrium set.
In HAPPO, the update rule of agent i m is changes with respect to HATRPO as
[Equation]
where r (¯ π i ) =
¯ π i ( a i | s )
π i old ( a i | s ) .
We show in Appendix D that this optimisation objective is equivalent to
[Equation]
The purple term is clearly non-negative due to the presence of the ReLU funciton.
Furthermore, for policies ¯ π i m sufficiently close to π i m old , the clip operator does not activate, thus rendering r (¯ π i m ) unchanged.
Therefore, the purple term is zero at and in a region around ¯ π i m =
π i m old , which also implies that its Gâteaux derivatives are zero.
Hence, it evaluates a HADF for agent i m , thus making
HAPPO a valid HAML instance.
Finally, we would like to highlight that these results about HATRPO and HAPPO significantly strengthen the work originally by Kuba et al.
(2022a) who only derived these learning protocols as approximations to a theoretically sound algorithm (see Algorithm 1 in Kuba et al.
(2022a)), yet without such level of insights.
4. N EW HAML I NSTANCES : HAA2C AND HADDPG
In this subsection, we exemplify how HAML can be used for derivation of principled MARL algorithms, and introduce heterogeneous-agent extensions of A2C and DDPG, different from those in
Subsection 3.2.
Our goal is not to refresh new SOTA performance on challenging tasks, but rather to verify the correctness of our theory, as well as deliver more robust versions of multi-agent extensions of popular RL algorithms such as A2C and DDPG.
[Figure]
HAA2C intends to optimise the policy for the joint advantage function at every iteration, and similar to A2C, does not impose any penalties or constraints on that procedure.
This learning procedure is accomplished by, first, drawing a random permutation of agents i 1: n , and then performing a few steps of gradient ascent on the objective of
[Equation]
with respect to π i m parameters, for each agent i m in the permutation, sequentially.
In practice, we replace the multi-agent advantage
A i m π old ( s , a i 1: m − 1 , a i m ) with the joint advantage estimate which, thanks to the joint importance sampling in Equation (6), poses the same objective on the agent (see
Appendix E for full pseudocode).
HADDPG aims to maximise the state-action value function off-policy.
As it is a deterministicaction method, and thus importance sampling in its case translates to replacement of the old action inputs to the critic with the new ones.
Namely, agent i m in a random permutation i 1: n maximises
[Equation]
with respect to µ i m , also with a few steps of gradient ascent.
Similar to HAA2C, optimising the state-action value function (with the old action replacement) is equivalent to the original multi-agent value (see Appendix E for full pseudocode).
We are fully aware that none of these two methods exploits the entire abundance of the HAML framework—they do not possess HADFs or neighbourhood operators, as oppose to HATRPO or
HAPPO.
Thus, we speculate that the range of opportunities for HAML algorithm design is yet to be discovered with more future work.
Nevertheless, we begin this search from these two straightforward methods, and analyse their performance in the next section.
E XPERIMENTS AND R ESULTS
In this section (1) , we challenge the capabilities of HAML in practice by testing HAA2C and HADDPG on the most challenging MARL benchmarks—we use StarCraft Multi-Agent Challenge (Samvelyan et al., 2019) and Multi-Agent MuJoCo
(Peng et al., 2020), for discrete (only HAA2C) and continuous action settings, respectively.
We begin by demonstrating the performance of HAA2C. As a baseline, we use its “naive” predecessor,
MAA2C, in both policy-sharing (MAA2C-S) and heterogeneous (MAA2C-NS) versions.
Results on
Figures 1 & 2 show that HAA2C generally achieves higher rewards in SMAC than both versions of
MAA2C, while maintaining lower variance.
The performance gap increases in MAMuJoCo, where the is at https
//github com/anonymouswater/HAML
[Figure] [Figure]
agents must learn diverse policies to master complex movements (Kuba et al., 2022a).
Here, echoing
Proposition
[Figure]
the homogeneous MAA2C-S fails completely, and conventional, heterogeneous
MAA2C-NS underperforms by a significant margin (recall Proposition 2)
[Figure]
As HADDPG is a continuous-action method, we test it only on MAMuJoCo (precisely 6 tasks) and compare it to MADDPG.
Figure 3 revelas that HADDPG achieves higher reward than MADDPG, while sometimes displaying significantly lower variance ( e.g in Reacher-2x1 and Swimmer-2x1).
Hence, we conclude that HADDPG performs better.
C ONCLUSION
In this paper
[Figure]
we described and addressed the problem of lacking principled treatments for cooperative
MARL tasks.
Our main contribution is the development of heterogeneous-agent mirror learning (HAML), a class of provably correct MARL algorithms, whose properties are rigorously profiled.
We verified the correctness and the practicality of HAML by interpreting current SOTA methods—
HATRPO and HAPPO—as HAML instances and also by deriving and testing heterogeneous-agent extensions of successful RL algorithms, named as HAA2C and HADDPG.
We expect HAML to help create a template for designing both principled and practical MARL algorithms hereafter.
A P ROOFS OF P RELIMINARY R ESULTS A. P ROOF OF L EMMA
Lemma 1 (Multi-Agent Advantage Decomposition) .
Let π be a joint policy, and i 1 , . .
, i m be an arbitrary ordered subset of agents.
Then, for any state s and joint action a
i 1: m ,
[Equation]
Proof.
(We quote the proof from (Kuba et al., 2022a).)
We start as expressing the multi-agent advantage as a telescoping sum, and then rewrite it using the definition of multi-agent advantage,
[Equation]
A. P ROOF OF P ROPOSITION
Proposition 2 (Trap of Heterogeneity) .
Let’s consider a fully-cooperative game with 2 agents, one state, and the joint action space { 0 , 1 } 2 , where the reward is given by r (0 , 0)
= 0 , r (0 , 1)
=
r (1 , 0) = 2 , and r (1 , 1) = − 1 .
Suppose that π i old (0) > 0 .
6 for i = 1 , 2 .
Then, if agents i update their policies by
[Equation]
then the resulting policy will yield a lower return,
[Equation]
As there is only one state, we can ignore the infinite horizon and the discount factor γ , thus making the state-action value and the reward functions equivalent, Q ≡ r .
Let us, for brevity, define π i =
π
i old (0) > 0 . 6 , for i = 1 , 2 .
We have
[Equation]
The update rule stated in the proposition can be equivalently written as
[Equation]
We have
[Equation]
and similarly
E
[Equation]
Hence, if π − i > 0 . 6 , then
[Equation]
Therefore, for every i , the solution to Equation (8) is the greedy policy π i new (1) = 1 .
Therefore,
[Equation]
which finishes the proof.
B P ROOF OF HAMO Is All You Need L EMMA
Lemma 2 (HAMO Is All You Need) .
Let π old and π new be joint policies and let i 1: n ∈ Sym ( n ) be an agent permutation.
Suppose that, for every state s ∈ S and every m = 1 , . . . , n ,
[Equation]
Then, π new is jointly better than π old
[Equation]
so that for every state s ,
[Equation]
Proof.
Let (cid:101) D
π old ( π new
[Equation]
) ≜ (cid:80) nm
[Equation]
[Equation]
Combining this with Lemma 1 gives
[Equation]
by Inequality (5)
[Equation]
The resulting inequality can be equivalently rewritten as
[Equation]
We use it to prove the claim as follows,
[Equation]
by Inequality (9)
[Equation]
Hence V π new ( s ) −
[Equation]
Taking infimum over s and simplifying
[Equation]
Therefore, inf s (cid:2) V π new ( s ) − V π old ( s ) (cid:3) ≥ 0 , which proves the lemma.
C P ROOF OF T HEOREM
Lemma 3.
Suppose an agent i m maximises the expected HAMO
[Equation]
Then, for every state s ∈ S
[Equation]
Proof.
We will prove this statement by contradiction.
Suppose that there exists s 0 ∈ S such that
[Equation]
Let us define the following policy ˆ π i m .
[Equation]
Note that ˆ π i m is (weakly) closer to π i m old than π i m new at s 0 , and at the same distance at other states.
Together with π i m new ∈
U
i m π old ( π i m old )
[Equation]
this implies that ˆ π i m ∈
U i m π old ( π i m old ) .
Further,
[Equation]
[Table]
The above contradicts π i m new as being the argmax of Inequality (11), as ˆ π i m is strictly better.
The contradiciton finishes the proof.
Theorem 1 (The Fundamental Theorem of Heterogeneous-Agent Mirror Learning) .
Let, for every agent i ∈ N , D i,ν be a HADF,
U i be a neighbourhood operator, and let the sampling distributions
β π depend continuously on π .
Let π 0 ∈ Π , and the sequence of joint policies ( π k ) ∞ k =0 be obtained by a HAML algorithm induced by D i,ν , U
i , ∀ i ∈ N , and β π .
Then, the joint policies induced by the algorithm enjoy the following list of properties
Attain the monotonic improvement property,
[Equation]
Their value functions converge to a Nash value function V NE
[Equation]
Their expected returns converge to a Nash return,
[Equation]
Their ω -limit set consists of Nash equilibria.
Proof.
Proof of Property 1.
It follows from combining Lemmas 2 & 3.
Proof of Properties 2, 3 & 4.
Step 1: convergence of the value function.
By Lemma 2, we have that V π k ( s ) ≤ V π k +1 ( s ) , ∀ s ∈ S , and that the value function is upper-bounded by V max .
Hence, the sequence of value functions ( V π k ) k ∈ N converges.
We denote its limit by V .
Step 2: characterisation of limit points.
As the joint policy space Π is bounded, by BolzanoWeierstrass theorem, we know that the sequence ( π k ) k ∈ N has a convergent subsequence.
Therefore, it has at least one limit point policy.
Let ¯ π be such a limit point.
We introduce an auxiliary notation: for a joint policy π and a permutation i 1: n , let HU ( π , i 1: n ) be a joint policy obtained by a HAML update from π along the permutation i 1: n .
Claim: For any permutation z 1: n ∈ Sym ( n ) ,
[Equation]
Proof of Claim .
Let ˆ π = HU ( ¯ π , z 1: n ) ̸
= ¯ π
and ( π k r ) r ∈
N be a subsequence converging to ¯ π .
Let us recall that the limit value function is unique and denoted as V .
Writing E i 0: ∞ 1: n [ · ] for the expectation operator under the stochastic process ( i k 1: n ) k ∈ N of update orders, for a state s ∈ S , we have
[Equation]
as every choice of permutation improves the value function
[Equation]
By the continuity of the expected HAMO (following from the continuity of the value function (Kuba et al., 2022a, Appendix A), HADFs, neighbourhood operators, and the sampling distribution)
we obtain that the first component of HU ( π k r , z 1: n ) , which is π z 1 k r +1 , is continuous in π k r by Berge’s
Maximum Theorem (Ausubel & Deneckere, 1993).
Applying this argument recursively for z 2 , . .
, z n , we have that HU ( π k r , z 1: n ) is continuous in π k r .
Hence, as π k r converges to ¯ π , its HU converges to the HU of ¯ π , which is ˆ π .
Hence, we continue wiriting the above derivation as
[Equation]
As s was arbitrary, the state-value function of ˆ π is the same as that of π :
V
ˆ π = V π , by the Bellman equation (Sutton & Barto, 2018):
Q ( s, a ) =
r ( s, a )+ γ E V ( s ′ ) , this also implies that their state-value and advantage functions are the same: Q ˆ π = Q ¯ π and
A ˆ π =
A ¯ π .
Let m be the smallest integer such that ˆ π z m ̸
= ¯ π z m .
This means that ˆ π z m achieves a greater expected HAMO than ¯ π z m , for which it is zero.
Hence,
[Equation]
and as the expected value of the multi-agent advantage function is zero
[Equation]
This is a contradiction, and so the claim in Equation (12) is proved, and the Step 2 is finished.
Step 3: dropping the HADF.
Consider an arbitrary limit point joint policy ¯ π .
By Step 2, for any permutation i 1: n , considering the first component of the HU, and writing ν i
=
ν i ¯ π ,π i ,
[Equation]
As the HADF is non-negative, and at π i 1 =
¯ π i 1 its value and of its all Gâteaux derivatives are zero, it follows by Step 3 of Theorem 1 of (Kuba et al., 2022b)
that for every s ∈ S ,
[Equation]
Step 4: Nash equilibrium.
We have proved that ¯ π satisfies
[Equation]
Hence, by considering ¯ π − i fixed, we see that
¯ π i satisfies the condition for the optimal policy (Sutton & Barto, 2018), and hence
[Equation]
Thus, ¯ π is a Nash equilibrium.
Lastly, this implies that the value function corresponds to a Nash value function V NE , the return corresponds to a Nash return J NE .
D C ASTING HAPPO AS HAML
The maximisation objective of agent i m in HAPPO is
[Equation]
Fixing s and a i 1: m − 1 , we can rewrite it as
[Equation]
By the multi-agent advantage decomposition
[Equation]
Hence
[Equation]
the presence of the joint advantage of agents i 1: m is equivalent to the multi-agent advantage of i m given a i 1: m − 1 that appears in HAMO.
Hence, we only need to show that that the subtracted term is an HADF.
Firstly, we change min into max with the identity − min f ( x ) = max[ − f ( x )] .
[Equation]
which we then simplify
[Equation]
As discussed in the main body of the paper, this is an HADF.
E A LGORITHMS
Algorithm
Initialize the for k = 0 , 1 , . .
, K − 1 do
Collect a set of trajectories by letting the agents act according their policies, a i ∼ π iθ i ( · i | o i ) ;
Push transitions { ( o it , a it , o it +1 , r t ) , ∀ i ∈ N , t ∈ T } into B ;
Sample a random minibatch of B transitions from B ;
Estimate the returns R and the advantage function, ˆ A ( s , a ) , using ˆ V ϕ and GAE;
Draw a permutation of agents i 1: n at random;
Set M i 1 ( s , a )
=
ˆ A ( s , a ) ; for agent i m =
i 1 , . .
, i n do
Set π
i m 0 ( a i
[Equation]
| o i m ) =
π
[Equation]
im ( a i m | o i m ) ; for mini-epoch
= 1 , . .
, e do
Compute agent i m ’s policy gradient
[Equation]
Update agent i m ’s policy by θ i m
= θ
i m
+ α
g
i m .
Compute M i m +
( s , a ) =
π imθim ( a im |
o im )
π im
( a im | o im )
M i m ( s , a ) //unless m = n ;
Update the critic by gradient descent on 1 B (cid:80) s (cid:0) ˆ V ϕ
( s b ) − R b (cid:1) 2 .
Discard ϕ .
Deploy { θ i } i ∈N in execution;
Algorithm 3:
; for k = 0 , 1 , . . . , K − 1 do
Collect a set of transitions by letting the agents act according to their deterministic policies with the exploratory noise
[Equation]
Push transitions { ( o it , a it , o it +1 , r t ) , ∀ i ∈ N , t ∈ T } into B ;
Sample a random minibatch of B transitions from B ;
Compute the critic targets
[Equation]
Update the critic by minimising the loss
[Equation]
Draw a permutation of agents i 1: n at random; for agent i m = i 1 , . .
, i n do Update agent i m by solving
[Equation]
with e mini-epochs of deterministic policy gradient ascent;
Update the target critic network smoothly
[Equation]
Discard ϕ .
Deploy { θ i } i ∈N in execution;
F E XPERIMENTS F. C OMPUTE RESOURCES
For compute resources, We used one internal compute servers which consists consisting of 6x RTX 3090 cards and 112 CPUs, however each model is trained on at most 1 card
. F. H YPERPARAMETERS
We implement the MAA2C and HAA2C based on HAPPO/HATRPO (Kuba et al., 2022a).
We offer the hyperparameter we use for SMAC in table 1 and for Mujoco in table 2.
[Table]
[Table]
In addition to those common hyperparameters
[Table]
we set the mini-epoch for HAA2C as 5.
For actor learning rate, we set it as 2e-4 for HalfCheetah and Ant while 1e-4 for Walker2d.
We implement the MADDPG and HADDPG based on the Tianshou framework (Weng et al., 2021).
We offer the hyperparameter we use in table 3 and
[Table]
[Table] [Table]