M
ULTI -A GENT T RUST R EGION L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Trust-region methods are widely used in single-agent reinforcement learning.
One advantage is that they guarantee a lower bound of monotonic payoff improvement for policy optimization at each iteration.
Nonetheless, when applied in multi-agent settings, such guarantee is lost because an agent’s payoff is also determined by other agents’ adaptive behaviors.
In fact, measuring agents’ payoff improvements in multi-agent reinforcement learning (MARL) scenarios is still challenging.
Although game-theoretical solution concepts such as Nash equilibrium can be applied, the algorithm (e.g., Nash-Q learning) suffers from poor scalability beyond twoplayer discrete games.
To mitigate the above measurability and tractability issues, in this paper, we propose Multi-Agent Trust Region Learning (MATRL) method.
MATRL augments the single-agent trust-region optimization process with the multiagent solution concept of stable ﬁxed point that is computed at the policy-space meta-game level.
When multiple agents learn simultaneously, stable ﬁxed points at the meta-game level can effectively measure agents’ payoff improvements, and, importantly, a meta-game representation enjoys better scalability for multi-player games.
We derive the lower bound of agents’ payoff improvements for MATRL methods, and also prove the convergence of our method on the meta-game ﬁxed points.
We evaluate the MATRL method on both discrete and continuous multiplayer general-sum games; results suggest that MATRL signiﬁcantly outperforms strong MARL baselines on grid worlds, multi-agent MuJoCo, and Atari games.
I NTRODUCTION
Multi-agent systems (MAS) (Shoham & Leyton-Brown, 2008) have received much attention from the reinforcement learning community.
In real-world, automated driving (Cao et al., 2012), StarCraft
II (Vinyals et al., 2019) and Dota 2 (Berner et al., 2019) are a few examples of the myriad of applications that can be modeled by MAS.
Due to the complexity of multi-agent problems (Chatterjee et al., 2004), investigating if agents can learn to behave effectively during interactions with environments and other agents is essential (Fudenberg et al., 1998).
This can be achieved naively through the independent learner (IL)
(Tan, 1993), which ignores the other agents and optimizes the policy assuming a stable environment (Bus¸oniu et al., 2010; Hernandez-Leal et al., 2017).
Due to their theoretical guarantee and good empirical performance in real-world applications, trust region methods (e.g., PPO (Schulman et al., 2015; 2017)) based ILs are popular (Vinyals et al., 2019; Berner et al., 2019).
In single-agent learning, trust region methods can produce a monotonic payoff improvement guarantee (Kakade & Langford, 2002) via line search (Schulman et al., 2015).
In multi-agent scenarios, however, an agent’s improvement is affected by other agent’s adaptive behaviors (i.e., the multi-agent environment is non-stationary (Hernandez-Leal et al., 2017)).
As a result, trust region learners can measure the policy improvements of the agents’ current policies, but the improvements of the updated opponents’ policies are unknown (shown in Fig. 1).
Therefore, trust region based ILs act less well in MAS as in single-agent tasks.
Moreover, the convergence to a ﬁxed point , such as a Nash equilibrium (Nash et al., 1950; Bowling & Veloso, 2004; Mazumdar et al., 2020), is a common and widely accepted solution concept for multi-agent learning.
Thus, although independent learners can best respond to other agents’ current policies, they lose their convergence guarantee (Laurent et al., 2011).
One solution to address the convergence problem for independent learners is Empirical GameTheoretic Analysis (EGTA) (Wellman, 2006), which approximates the best response to the policies generated by the independent learners (Lanctot et al., 2017; Muller et al., 2019).
Although EGTA based methods (Lanctot et al., 2017; Omidshaﬁei et al., 2019; Balduzzi et al., 2019) establish

[Table]
[Table]
[Table] [Table]
/sn3ebhwUy0FfgKW9CCAHbhEH5DD/rA4AZu4R4evDvvyXuuedPSfxG+wBur1V8AfZi0PA==</latexit>
Improvementisunknown
Improvementisunknown convergence guarantees in several games classes, the computational cost is also large when empirically approximating and solving the increasing meta-game (Yang et al., 2019).
Other multi-agent learning approaches collect or approximate additional information such as communication (Foerster et al., 2016) and centralized joint critics (Lowe et al., 2017;
Foerster et al., 2017; Sunehag et al., 2018;
Rashid et al., 2018).
Nevertheless, these methods usually require centralized parameters or centralized communication assumptions.
Thus, there is considerable interest in multi-agent learning to ﬁnd an algorithm that, while having minimal requirements and computational cost as independent learners, also improves convergence performance at the same time.
This paper presents the Multi-Agent Trust Region Learning (MATRL) algorithm that augments the trust-region ILs with a meta-game analysis to improve the stability and efﬁciency of learning.
In
MATRL, a trust region trial step for an agents’ payoff improvement is implemented by independent learners, which gives a predicted policy based on the current policy.
Then, an empirical policy-space meta-game is constructed to compare the expected advantage of predicted policies with the current policies.
By solving the meta-game, MATRL ﬁnds a restricted step by aggregating the current and predicted policies using meta-game Nash Equilibrium.
Finally, MATRL takes the best responses based on the aggregated policies from last step for each agent to explore because the found TSR is not always strict stable.
MATRL is, therefore, able to provide a weak stable solution compared with the naive independent learners.
Based on trust region independent learners, MATRL does not need extra parameters, simulations, or modiﬁcations to the independent learner itself.
We provide insights into the empirical meta-game in Section 3.2, showing that an approximated Nash equilibrium of the meta-game is a weak stable ﬁxed point of the underlying game.
Our experiments demonstrate that
MATRL signiﬁcantly outperforms deep independent learners (Schulman et al., 2017) with the same hyper-parameters, centralized VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2018) methods in discrete action grid-worlds, centralized MADDPG (Lowe et al., 2017) in a continuous action multi-agent MuJoCo task (de Witt et al., 2020) and zero-sum multi-agent Atari (Terry & Black, 2020).
P RELIMINARY
A Stochastic Game (Shapley, 1953; Littman, 1994) can be deﬁned as:
G =
(cid:104)N , S , {A i } , {R i } , P , p 0 , γ (cid:105) , where N is a set of agents, n = |N| is the number of agents and
S denotes the state space.
A i is the action space for agent i .
A =
A 1 × · · · × A n = A i × A −
i is the joint action space, and for the simplicity we use − i denotes the other agents except agent i .
R i
= R i
( s, a i , a − i ) is the reward function for agent i ∈ N .
P : S ×
A × S → [0 , 1] is the transition function.
p 0 is the initial state distribution, γ ∈ [0 , 1] is a discount factor.
Each agent i ∈ N has a stochastic policy π i ( a i | s ) :
S × A i → [0 , 1] , and aims to maximize its long term discounted return: η i ( π i , π − i ) =
E s 0 ,a
i ,a 0 − i ··· (cid:34) ∞ (cid:88) t = γ t R i ( s t , a ti , a t − i ) (cid:35) , (1) where s∼ p 0 , s t + ∼
P ( s t +1 | s t , a ti , a t − i ) , a ti ∼ π i ( a ti | τ ti ) .
We then can have the standard deﬁnitions of the state-action value function Q π i ,π − i i ( s t , a ti , a t − i ) =
E s t +1 ,a t +1 i ,a t +1 − i ··· [ (cid:80) ∞
l =
γ l R i ( s t + l , a t + l i , a t + l − i )] , the value function
V π i ,π − i i ( s t ) =
E a ti ,a t − i ,s t +1 ··· [ (cid:80) ∞ l =
γ l R i ( s t + l , a t + l i , a t + l − i )] , and the advantage function A π i ,π − i i ( s t , a ti , a t − i ) =
Q π i ,π − i i ( s t , a ti , a t − i ) −
[Equation]
π i ,π − i i ( s t ) given the state and joint action.
M ULTI -A GENT T RUST R EGION P OLICY O PTIMIZATION
A trust region algorithm aims to answer two questions: how to compute a trust region trial step and whether a trial step should be accepted.
In multi-agent learning, a trust region trial step towards agents’ payoff improvement can be easily implemented with independent learners, and we call the independent payoff improvement area as Trust Payoff Region ( TPR ).
The remaining issue is
[Table]
⇡ ⇤ i
[Table]
[Table]
⇡
i
[Table]
⇡  i
[Table]
⇡ ⇤ i
[Table] [Table]
⇡ 0
i
[Table]
¯ ⇡  i
[Table]
=</latexit>
[Figure]
π i , π − i are the current policies for two agents.
ˆ π i , ˆ π − i predicted policies within TPR, ( π ∗ i , π ∗− i ) forms Nash equilibrium, π (cid:48)
i and π (cid:48)− i are the best responses to the weak stable ﬁxed point (¯ π i , ¯ π − i ) .
(a) : independent trust region learning, an agent i only considers itself’s policy improvement against the ﬁxed opponent policy π − i .
(b) : multi-agent trust region learning, agents’ policy improvement should be explored in the joint policy space ( π i , π − i ) toward a stable region.
revolved by ﬁnding a restricted step leading to a stable point in the joint policy space, denoted as
Trust Stable Region ( TSR ).
In other words, multi-agent trust region learning (MATRL) decomposes the trust region learning into two parts: ﬁrstly, ﬁnd a trust payoff region between current policy π i and predicted policy ˆ π i ; then, with the help of the predicted policy, a precise method can, to some extent, approximate a weak stable ﬁxed point .
Instead of line searching in a single-agent payoff improvement, MATRL searches for the joint policy space to achieve a weak stable ﬁxed point (see Fig. 2).
Essentially, MATRL is a simple extension of the single-agent TRPO to MAS where independent learners learn to ﬁnd a stable point between current policy and predicted policy.
To solve the TSR, we assume the knowledge about other agents’ policies during the training to ﬁnd weak stable points via empirical meta-game analysis, while the execution can still be fully decentralized.
We explain every step of MATRL in detail in the following sections.
3. I NDEPENDENT T RUST P AYOFF I MPROVEMENT
Single-agent reinforcement learning algorithms can be straightforwardly applied to multi-agent learning, where we assume that all agents behave independently (Tan, 1993).
In this section, we have chosen the policy-based reinforcement learning method as independent learners.
In multi-agent games, the environment becomes a Markov decision process for agent i when each of the other agents plays according to a ﬁxed policy.
We set agent i ’s to make a monotonic improvement against the ﬁxed opponent policies.
Thus, at each iteration, the policy is updated by maximizing the utility function
η i over a local neighborhood of the current joint policy π i , π − i :
[Equation]
) based on the trajectories sampled by π i , π − i .
We can adopt trust region policy optimization (e.g.,
PPO (Schulman et al., 2017)), which constrains step size in the policy update:
[Equation]
where D is a distance measurement and δ i is a constraint.
Independent trust region learners produce the monotonically improved policy ˆ π i which guarantees η i (ˆ π i , π − i ) ≥
η i ( π i , π − i ) and give a trust payoff bound by ˆ π i .
Due to the simultaneous policy improvement without awareness of other agents , however, the lower bound of payoff improvement from single-agent Schulman et al.
(2015) no longer holds for multi-agent payoff improvement.
Following the similar proof procedures, we can obtain a precise lower bound for a multi-agent simultaneous trust payoff region in Theorem 1:
Theorem 1 (Independent Trust Payoff Region) .
Denote the expected advantage gain when π i , π − i → ˆ π i , ˆ π − i as:
[Equation]
Let α i
= D max TV ( π i , ˆ π i ) =
max s D TV ( π i ( ·| s ) (cid:107) ˆ π i ( ·| s )) for agent i , where D TV is total variation divergence (Schulman et al., 2015).
Then, the following lower bound can be derived for multi-agent independent trust region optimization:
[Equation]
where (cid:15) i =
[Equation]
Proof.
See Appendix B.
Based on the independent trust payoff improvement, although the predicted policy ˆ π i will guide us in determining the step size of the TPR, but the stability of (ˆ π i , ˆ π − i ) is still unknown.
As shown in Theorem 1, an agent’s lower bound is roughly O
(4 α 2 ) , which is four times larger than the single-agent lower bound trust region of O ( α 2 ) (Kakade & Langford, 2002).
Furthermore, (cid:15) i = max s,a − i ,a − i (cid:12)(cid:12)
A π i ,π − i i ( s, a i , a − i ) (cid:12)(cid:12) depends on the other agents’s action a − i that would be very large when agents have conﬂicting interests.
Therefore, the most critical issue underlying the multi-agent trust region learning is to ﬁnd a TSR after the TPR.
In next section, we will illustrate how to search for a weak stable ﬁxed point within the TPR, based on the policy-space meta-game analysis.
3. A PPROXIMATING W EAK S
In multi-agent trust region learning, TSR is one of the essential parts.
Since each iteration of MATRL requires solving the TPR and TSR sub-problems, ﬁnding the efﬁcient solver for stable trust region sub-problems is very important.
Instead of using the stable ﬁxed points (Balduzzi et al., 2018) as TSR, we choose weak stable ﬁxed point in Deﬁnition 1 which is much easier to be found.
To maximize the objective deﬁned in Eq.
1 we could ask that reasonable algorithms avoid all strict minimums (a.k.a unstable ﬁxed points), which imposes only that agents are well-behaved regarding strict minimum even if their individual behaviors is not self-interested (Letcher, 2020), and we say a point is in TSR if it is weak stable ﬁxed point.
Deﬁnition 1 (Weak Stable Fixed Point in Restricted Underlaying Game) .
Consider a restricted underlying game , where each agent’s policy space is restricted to open sets ¯Π i =
[ π i , ˆ π i ] ⊆
Π i .
Denote the simultaneous gradient of the restricted underlying game as ξ =
( ∇ π
i g i , ∇ π − i g − i ) and Hessian H = ∇ ξ .
We call (¯ π i , ¯ π − i ) a ﬁxed point if ξ (¯ π i , ¯ π − i )
= 0 .
We then say (¯ π i , ¯ π − i ) is a weak stable ﬁxed point if H (¯ π i , ¯ π − i ) (cid:7) 0 1 , which avoids unstable ﬁxed points (strict minimum).
A trust stable region within weak stable ﬁxed points is reasonable if it converges only to ﬁxed points and avoids unstable ﬁxed points almost surely.
Given that we already have the TPR, which produces a predicted policy, with the knowledge about all the agents policies, it is natural to conduct an empirical game-theoretic analysis (Tuyls et al., 2018) to search for a weak stable ﬁxed point in the area bounded by current policy pair predicted policy pair.
We then deﬁne a meta-game that each agent i has only two strategies π i , ˆ π
i for each agent i :
[Equation]
where g ˆ i, − ˆ i i =
g π i ,π − i i (ˆ π i , ˆ π − i ) (as deﬁned in Eq. 3) is an empirical payoff entry of the metagame, and note g i, − i i = 0 as it has an expected advantage over itself.
Compared with using the η i (ˆ π i , ˆ π − i )
=
η i ( π i , π − i )
+ g
ˆ
i, − ˆ i i as the meta-game payoff, g ˆ i, − ˆ
i
i has lower variance and is easier to approximate because η i ( π i , π − i ) is a constant baseline.
However, the most of entries in M are unknown, it often requires lots of extra simulations to estimate the payoff entries (e.g., g ˆ i, − ˆ i i ) in
EGTA.
Instead, we reuse the trajectories in the TPR step to approximate the g ˆ i, − ˆ i i by ignoring small changes in state visitation density caused by the π i →
ˆ π i (Schulman et al., 2015).
Take two-agent case as an example, as we can see in Eq.
5, the meta-game M becomes a 2 ×matrix-form game, which is much smaller in size than the whole underlaying game.
To this end, we can use the existing Nash solvers (e.g. CMA-ES (Hansen et al., 2003)) for matrix-form games to compute a mixed Nash equilibrium
ρ i , ρ − i =
NashSolver( M ) for the meta-game M , where ρ i and ρ − i ∈ [0 , 1] , and the mixed Nash equilibrium of the meta-game is also an approximated equilibrium of the restricted underlying game (Tuyls et al., 2020).
Then, the trust stable region policies ¯ π i , ¯ π − i can be aggregated based on current policy π i and predicted policy ˆ π i in TPR for each agent i .
In the
TPR step, we require ILs enjoy the monotonic improvement against the ﬁxed opponent policies, in which the change from π i to ˆ
π i is usually constrained by a small step size.
Therefore, it is reasonable to assume there is a continuous and monotonic change in the restricted policy space between π i and ˆ π i .
In this case, with ρ i being agent i ’s Nash Equilibrium policy in the meta-game, ¯ π i can be derived via the linear mixture:
¯ π i =
ρ
i π i + (1 − ρ i )
ˆ π i , which delimit agent i ’s trust stable region.
Now we can prove that (¯ π i , ¯ π − i ) is a weak stable ﬁxed point for the underlying game in Theorem 2.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
[Table]
/Oh/
[Table]
gXr8p/+</latexit
[Table]
<
[Table]
</
[Table] [Table]
[Table]
i, i i ,
[Table]
i,
[Table]
<
[Table]
[Equation]
[Table]
[Equation]
<
[Table]
+pURjUbMBbO4hdNe7TO</latexit>
[Figure]
<latexit sha1_base64="c8WZPQ42qH3CmIbJ3KseYDuF5+Q=">AAACU3icdVFNSwMxEE3Xr1q/qh69BItQQcuuFNSboAePFWwV3LJk02kbzH6YzApl2T/kr/Hiof4SwYvZdg+21YHAy5s3k5kXP5ZCo22PS9bS8srqWnm9srG5tb1T3d3r6ChRHNo8kpF69JkGKUJoo0AJj7ECFvgSHvzn6zz/8ApKiyi8x1EM3YANQtEXnKGhvOqNC8i89FRkdTcWXiqyE+oOGabmlk34Y+oO4IX+pctBrvCqNbthT4IuAqcANVJEy6t+ub2IJwGEyCXT+smxY+ymTKHgErKKm2iIGX9mA3gyMGQB6G462TajR4bp0X6kzAmRTtjfFSkLtB4FvlEGDId6PpeT/+Xyjnrm/dQP5ubB/kU3FWGcIIR8Ok4/kRQjmhtMe0IBRzkygHElzEaUD5liHM03VIxVzrwxi6Bz1nCajcu7Zu3qojCtTA7IIakTh5yTK3JLWqRNOHkj72RMPksfpW/LspanUqtU1OyTmbC2fgBM0bU8</latexit> stable ﬁxed
Theorem 2 (Existence of Weak Stable Fixed Point) .
Consider the restricted underlying game where policy space is bounded in a linear continuous policy-space [ π i , ˆ π i ] , where ˆ π i is monotonically improved based on π i within TPR.
If ( ρ i , ρ − i ) is a Nash equilibrium of the meta-game M , then, the linear mixture joint policy (¯ π i , ¯ π − i ) is a weak stable ﬁxed point for the restricted underlying game.
Proof.
see Appendix C.
According to Theorem 2, (¯ π i , ¯ π − i ) is a weak stable ﬁxed point of the restricted underlying game.
Although the weak stable ﬁxed point is relatively weak compared to the stable ﬁxed points (Balduzzi et al., 2018), as we have stated, a weak stable ﬁxed point is a reasonable (not strong as rational) requirement for an algorithm to avoid the minimum.
Furthermore, the weak stable ﬁxed points can suit for general game settings.
As shown in Appendix C, in cooperative, competitive, and generalsum games, the ﬁxed-point found by the meta-game analysis can be either stable or saddle points.
Similarly, a local Nash equilibrium can be a stable or saddle in different differential games (Mazumdar et al., 2020).
Therefore, the goodness of stable concepts would depend on speciﬁc settings.
If we make some additional game class assumptions, we can easily obtain a stronger ﬁxed-point types.
Nevertheless, it comes with a cost, requiring additional computation or assumptions which may break the most general settings.
Besides, when the meta-game has multiple Nash equilibria, an equilibrium is randomly selected in our work.
Some equilibria can produce a more stable ﬁxed point, however, we leave the equilibrium selection problem for future work.
Extra Cost for Approximating and Solving meta-game .
There are two major-cost sources in common meta-game analysis: approximating and solving the meta-game (Muller et al., 2019).
In our case, the meta-game is restricted to a local two-action game, where two actions π i and ˆ π i are close to each other.
This proximity property reduces the meta-game approximation cost (without extra sampling) by reusing the collected trajectories in the TPR step (Tuyls et al., 2020).
The next crucial problem is how to solve the n -agent two-action meta-game, which consists of the 2 n entries of each of the n payoff matrices.
This is much simpler than solving the whole underlying game, which increases exponentially with state size, action size, agent number, and time horizons.
As the general-sum matrix-form game has no fully polynomial-time approximation for computing Nash equilibria (Chen et al., 2006), it usually costs a lot to solve the game (Daskalakis et al., 2009).
If we only require an approximated Nash equilibrium, when n is small, for example, n ≤ 10 , it is affordable to ﬁnd a meta-game Nash equilibrium in a sub-exponential complexity (Lipton et al., 2003).
However, this problem still exists when n is large.
In this case, we could try mean-ﬁeld approximation (Yang et al., 2018) or utilizing special payoff structure assumptions (e.g., graphical game (Littman et al., 2002; Daskalakis et al., 2009), which is polynomial-time computable.)
in the meta-game to reduce the computation complexity.
3. I MPROVEMENT A GAINST W EAK S
Although the weak stable ﬁxed point, (¯ π i , ¯ π − i ) , binds the policy update to another ﬁxed point, there are still undesired saddle points according to Theorem 1.
It is difﬁcult to generalize for the other parts of the policy-space not reached by these saddle points, especially in the anti-coordination games (Lanctot et al., 2017).
Similar to the extra-gradient method (Mertikopoulos et al., 2018), to escape the saddle points we apply the best response against the weak stable ﬁxed point:
[Equation]
To perform the best response, we need another round to collect the experiences and do a gradient step in Eq. 6.
However, in practice, since we already have the trajectories in the TPR step and so the best response to the weak stable ﬁxed point can be easily estimated through importance sampling.
Alternatively, through deﬁning
[Equation]
as truncated importance sampling weights, we can re-write the best response update to Eq.
6 into an equivalent form to the following one in terms of expectations: π (cid:48)
i [Equation]
Connections to Existing Methods .
MATRL generalizes many existing methods with the best response.
In extreme cases where the meta-game Nash is ( ρ i , ρ − i ) =
(1 , 1) , which means the Nash aggregated policies always keep the current policies, MATRL degenerates to independent learners.
Here, we always best response to the other agents’ current policy π i and π (cid:48)
i = arg max π i η i ( π i , π − i ) following the Eq.
The policy prediction (Zhang & Lesser, 2010; Foerster et al., 2018; Letcher et al., 2018), extra-gradient (Antipin, 2003) and exploitability descent (Tang et al., 2018;
Lockhart et al., 2019) methods are also special instances of MATRL when meta-game Nash is ( ρ i , ρ − i ) =
This means the best response to the most aggressive predicted policy ˆ π − i and π (cid:48)
i = arg max π i η i ( π i , ˆ π − i ) .
Global Convergence .
MATRL is a gradient-based algorithm with the best response to policies within TSR, which is essentially a variant of LookAhead methods(e.g.
, LOLA (Foerster et al., 2018),
SOS (Letcher et al., 2018) and IGA-PP (Zhang & Lesser, 2010).
More speciﬁcally, MATRL enhances the classic LookAhead method with variable step size scaling (Bowling & Veloso, 2002) or two time-scale update rule (Heusel et al., 2017) at each TSR step, which is controlled by the restricted meta-game analysis.
It has been proven that LookAhead method can locally converge and avoid strict saddles in all differentiable games (Letcher et al., 2018), and enjoys the better convergence with variable step size scaling (Song et al., 2019).
The convergence analysis of gradient-based algorithms is usually based on ﬁxed-point iterations and dynamical systems.
And please note, here, to investigate the convergence, the ﬁxed-point iterations are conducted on the whole learning process.
While the meta-game analysis step in MATRL borrows the concepts of different ﬁxed-points to show the meta-game analysis is reasonable to avoid unstable ﬁxed-points.
Unlike LOLA, which uses ﬁrst-order Taylor expansion to estimate the best response to a predicted policy, we elaborately design the look-ahead step within the TSR and do the best response gradient steps to TSR.
We also show that MATRL empirically outperforms the typical LookAhead method independent learner the policy prediction (IL-PP) in the experiments.
In summary, independent trust region learners’ learning in MTARL will be constrained by a weak stable ﬁxed point.
By analyzing the relatively simpler meta-game, we can easily approximate this weak stable ﬁxed point without extra rollouts or simulation.
Although MATRL’s training is centralized, its execution is fully decentralized.
It also does not require any extra centralized parameters or higherorder gradient computation.
Fig.
3 shows the overview of MATRL.
We also give the pseudocode of
MATRL in Algo.
1, which is compatible with any policy-based independent learner.
R ELATED W ORK
The study of gradient-based methods in multi-agent learning is quite extensive (Mazumdar et al., 2020; Bus¸oniu et al., 2010).
Some works on learning in games have mostly focused on adjusting the step size, which attempts to use a multiple-timescales learning scheme (Leslie & Collins, 2005; Leslie et al., 2003; Bowling & Veloso, 2002) to achieve convergence.
Balduzzi et al.
Mazumdar et al.
; Letcher et al.
(2018) tried to utilize the second-order methods to shape the step size.
However, the computational cost for second-order methods is very limiting in many cases.
Alternatively, MATRL approximates the second-order ﬁxed-point information via a small meta-game with less cost comparing to real Hessian computation.
An alternative augments the gradient-based algorithms with the best response to predicted polices (Antipin, 2003; Zhang &
Lesser, 2010; Lin et al., 2020; Foerster et al., 2018; Tang et al., 2018; Lockhart et al., 2019), [Figure] [Table] which target the challenge of instability caused by agents’ change policies.
Instead of taking the best response to the approximated opponent’s policy, MATRL exploits the ideas from both streams and and introduces the improvement over the weak stable ﬁxed point.
The research also focuses on the EGTA (Tuyls et al., 2018; Jordan & Wellman, 2009; Tuyls et al., 2020), which creates a policy-space meta-game for modeling the multi-agent interactions.
Using various evaluation metrics, it then updates and extends the policies based on the analysis of the meta policies (Lanctot et al., 2017; Muller et al., 2019; Omidshaﬁei et al., 2019; Balduzzi et al., 2019;
Yang et al., 2019).
Although these methods are broad with respect to multi-agent tasks, they require extensive computing resources to estimate the empirical meta-game and solve it with its increasing size (Omidshaﬁei et al., 2019; Yang et al., 2019).
In our method, we adopt the idea of a policy-space meta-game to approximate the ﬁxed point.
Unlike previous works, we only maintain the current policies and predicted policies to construct the meta-game, which is computationally achievable in most cases.
The payoff entry in MATRL’s meta-game is the expected advantage, which has a lower estimation variance compared to the commonly used empirically-estimated return in EGTAs.
Regardless, we can reuse the trajectories in the TPR step to estimate the payoffs without incurring additional sampling costs.
Recently, due to the use of neural networks as a function approximation for policies and values, there have emerged many works on deep reinforcement learning (DRL) (Mnih et al., 2013; Lillicrap et al., 2015).
Trust region policy optimization (Kakade & Langford, 2002; Schulman et al., 2015; 2017) is one of the most successful DRL methods in the single-agent setting, which puts constraints on the step size of policy updates, preserving any improvements monotonically.
Based the monotonic improvement in single-agent trust region policy optimization (TRPO) (Schulman et al., 2015),
MATRL extends the improvement guarantee to the multi-agent level, towards a weak stable ﬁxed point.
Some works directly apply fully decentralized single-agent DRL methods (Tan, 1993), which can be unstable during when learning due to the non-stationary issue.
Whereas (Foerster et al., 2016; Sukhbaatar et al., 2016; Peng et al., 2017) added an extra communication channel during the training and execution in a centralized way to avoid this non-stationarity issue.
Sunehag et al.
(2018); Rashid et al.
; Foerster et al.
; Lowe et al.
(2017) further exploit the setting of centralized learning decentralized execution (CTDE).
These methods provide solutions for training agents in complex multi-agent environments, and the experimental results show the effectiveness compared with independent learners.
Similar to the CTDE setting, the MATRL also enjoy fully decentralized execution.
Although MATRL still needs knowledge about the other agents’ policies during the training, it only requires a centralized mechanism to adjust the step size rather than the additional centralized critic or communication.
E XPERIMENTS
We design the experiments to answer the following questions: 1).
Can the MATRL method empirically contribute to the convergence in the general game settings, including cooperative/competitive and continuous/discrete games? 2).
How is the performance of MATRL compared to the ILs with the same hyper-parameters and other strong MARL baselines in discrete and continuous games with various agent number? 3).
Do the meta-game and best response to the weak stable ﬁxed point bring beneﬁts?
We ﬁrst evaluate the convergence performance of MATRL in matrix form games to answer the ﬁrst question and validate the effectiveness of convergence.
For Question 2, we show that MATRL largely outperforms ILs (PPO (Schulman et al., 2017)) and other centralized baselines (QMIX (Rashid et al., 2018), QTRAN (Son et al., 2019) and VDN (Sunehag et al., 2018)) for discrete grid world games [Figure] IL [Figure]
IL
VDNQMIX [Figure] that have coordination problems. It also outperforms MADDPG (Lowe et al., 2017) for continuous multi-agent MuJoCo games.
Besides, we test the algorithms with 2-agent pong Atari game to investigate if MATRL can mitigate unstable cyclic behaviors (Balduzzi et al., 2019) in zero-sum games.
In these tasks, MATRL uses the same PPO conﬁgurations as ILs to examine the effectiveness of the trust region gradient-update mechanism, and we use ofﬁcial implementations for the other baselines.
The step-by-step PPO based MATRL algorithm is given in Algo.
Finally, ablation studies are conducted by: 1. removing the best response, called the MATRL w/o BR; 2. skipping the trust-stable region estimation, named IL-PP, which has similar procedures as LOLA Foerster et al.
; Zhang & Lesser (2010), which approximated the best response to the predicted policies via
Taylor expansion, but IL-PP takes the best response gradient steps to the predicted policies.
These conﬁgurations provide insights about how much does the trust stable region and the best response contribute to the MATRL’s performance if any.
We also provide more environment details and extra experiment results, including 4-agent Ant (multi-agent mujuco), in the Appendix D and E with detailed experiment settings and hyper-parameters used for the algorithms.
The code and experiment scripts are also anonymously available at https://github.com/matrl-project/matrl .
Matrix Game and Random 2 × 2 Matrix Games .
To illustrate the effectiveness of MATRL, we conducted an experiment on well known zero-sum matching pennies (MP) (Bruns, 2015) game and devise the 2 × 2 random matrix games.
Using IGA (Singh et al., 2000) as ILs of MATRL, the learning dynamics of MATRL on MP are shown in Fig.
4, where the blue arrow is trust payoff direction and the pale blue area is TSR..
The MATRL reaches the Nash Equilibrium (central red star point) by updating the policies with the constraints from the trust stable region (the pale blue area).
It would be trapped to a cyclic loop if following the original trust pay off direction (the dark blue arrow).
To adequately examine the MATRL on border matrix games, we randomly generate three thousand 2 × 2 games for three types: coordination, anti-coordination, and cyclic (Pangallo et al., 2017).
More details about the game generation are provided in Section D.
We choose the IGA and IGA-PP (Zhang & Lesser, 2010) as baselines, and the results in Table 1 show that MATRL has a higher convergence rate and needs fewer steps for convergence in all types of games.
Grid World Checker and Switch .
We evaluated MATRL in two grid world games from MAGym (Koul, 2019), two-agent checker, and four-agent switch, which are similar to games in Sunehag et al.
(2018), but with more agents to examine if the MATRL can handle the games that have more than two agents.
In the checker game, two agents cooperate in collecting fruits on the map; the sensitive agent gets 5 for apple and − 5 for lemon, while the other one gets 1 and − 1 respectively.
So the optimal solution is to let the sensitive agent get the apple and the less sensitive one get the lemon.
In the four-agent switch game, two rooms are connected with a corridor, each room has two agents, and the four agents try to go through one corridor to the target in the opposite room.
Only one agent can pass the corridor at one time, and agents get − 0 .
1 for each step and 5 for reaching targets, so they need to cooperate to get optimal scores.
In both games, The agents can move in four directions and only partially observe their position.
Although our formulation uses a fully observable setting, in this game, the methods adopt to the partially observable by pretending the observation is a state.
We compare the MATRL with the PPO based IL and two off-policy centralized training and decentralized execution baselines:
VDN (Sunehag et al., 2018), QTRAN (Son et al., 2019) and QMIX (Rashid et al., 2018).
Results are given in Fig.
5a and 5b, where MATRL has a stable improvement and outperforms other baselines.
In two-player checker, using the best response, our method can achieve a total reward of 18 , while the independent learners’ rewards stay at − 2 .
Besides, although PPO-based MATRL uses on-policy learning, it achieved better ﬁnal results in fewer time steps compared to the off-policy baselines.
As for the four-player switch, as shown in Fig.
5b, MATRL can continuously improve the total rewards to 6 . 5 , which is the closest to the optimal score for this game when compared with
Under review as a conference paper at ICLR
[Figure] [Figure]
lemon.
In the four-agent switch game, each side of the corridor has a room, each room has two agents, and the four agents try to go through one corridor to the target in the opposite room.
Only one agent can pass the corridor at one time, and agents get  0 .
1 for each step and 5 for reaching targets, so they need to cooperate to get optimal scores.
In both games, The agents can move in four directions and only partially observe their position.
We compare the MATRL with the PPO based
IL and two off-policy centralized training and decentralized execution baselines:
VDN (Sunehag et al., 2018) and QMIX (Rashid et al., 2018).
Results are given in Fig.
5a and 5b, where MATRL has a stable improvement and outperforms other baselines.
In two-player checker, using the best response, our method can achieve a total reward of 18 , while the independent learners’ rewards stay at  2 .
Besides, although PPO-based MATRL uses on-policy learning, it achieved better ﬁnal results in fewer time steps compared to the off-policy baselines.
As for the four-player switch, as shown in Fig.
5b, MATRL can continuously improve the total rewards to 6 . 5 , which is the closest to the optimal score for this game when compared with other baselines.
The result in the four-agent switch also demonstrates the effectiveness of MATRL in guaranteeing the stable policy improvement for the games that have more than two agents.
Multi-Agent MuJoCo Hopper .
We also examined MATRL in a multi-agent continuous control task with a three-agent hopper from (de Witt et al., 2020).
Here, three agents cooperatively control each part of a hopper to move forward.
The agents are rewarded with distance and the number of steps they make before falling.
Fig.
5c shows that MATRL signiﬁcantly outperforms IL, MADDPG, and also the benchmarks in de Witt et al.
(2020) within the same amount of time.
Effect and Cost of Trust Stable Region and Best Response to Fixed Point .
This section analyzes the effect of the TSR from meta-game Nash and the best response against the weak stable ﬁxed point.
The ablation settings are obtained by removing the trust stable region (IL-PP) and the best response (MATRL w/o BR).
In Fig.
5, we can observe that in all the tasks, without the best response to the ﬁxed point, the learning curves of MATRL o/w BR have higher variance and the lowest ﬁnal scores.
This establishes the importance of the best response to stabilize and improve agents’ performance.
Also, without the TSR to select a ﬁxed point, the MATRL recovers to independent learners with the policy prediction (IL-PP) (Zhang & Lesser, 2010; Foerster et al., 2018).
Similarly, the curves of IL-PP have lower ﬁnal scores, and the convergence speed is not as good as the MATRL, which suggests that the TSR provides beneﬁts.
The MATRL w/o BR has lower variance compared to the
IL-PP, which reveals the trust stable region can stabilize the learning via weak stable ﬁxed point constraints.
Finally, when comparing to IL and IL-PP, the time for each training step in MATRL is other baselines.
The result in the four-agent switch also demonstrates the effectiveness of MATRL in guaranteeing the stable policy improvement for the games that have more agents.
Multi-Agent MuJoCo Hopper .
We also examined MATRL in a multi-agent continuous control task with a three-agent hopper from (de Witt et al., 2020).
Here, three agents cooperatively control each part of a hopper to move forward.
The agents are rewarded with distance and the number of steps they make before falling.
Fig.
5c shows that MATRL signiﬁcantly outperforms IL, MADDPG, and also the benchmarks in de Witt et al.
(2020) within the same amount of time.
Multi-Agent Pong Atari Game .
In the 2-agent pong game experiments, we used raw pixel as observation and train the MATRL and IL agents independently.
Following training, we compare these models’ pair-wise performance by pitting their ten checkpoints against one another and recording the average scores.
We report the results in Fig.
6, which shows MATRL outperforms IL in MATRL vs.
IL setting in most of the policy pairs.
Besides, from the MATRL vs. MATRL and IL vs. IL settings’ results, we can see MATRL has a more transitive learning process than IL, which means MATRL can mitigate the common cyclic behaviors in zero-sum games.
[Figure]
Effect and Cost of Trust Stable Region and Best Response to Fixed Point .
This section analyzes the effect of the TSR from meta-game Nash and the best response against the weak stable ﬁxed point.
The ablation settings are obtained by removing the trust stable region (IL-PP) and the best response (MATRL w/o BR).
In Fig.
5, we can observe that in all the tasks, without the best response to the ﬁxed point, the learning curves of MATRL o/w BR have higher variance and the lowest ﬁnal scores.
This establishes the importance of the best response to stabilize and improve agents’ performance, and empirically shows that the MATRL has better convergence ability than other baselines.
Also, without the TSR to select a ﬁxed point, the MATRL recovers to independent learners with the policy prediction (IL-PP) (Zhang & Lesser, 2010; Foerster et al., 2018).
Similarly, the curves of IL-PP have lower ﬁnal scores, and the convergence speed is not as good as the MATRL, which suggests that the TSR provides beneﬁts.
The MATRL w/o BR has lower variance compared to the IL-PP, which reveals the trust stable region can stabilize the learning via weak stable ﬁxed point constraints.
Finally, when comparing to IL and IL-PP, as shown in Fig. 7, in 2-4 agents games with 20,000 environment steps and 50 gradient steps, the training time of MATRL is empirically about 1.1-1.2 times slower.
We think this extra computational cost from the TSR and the best response is acceptable given the performance improvement brought by these operations.
C ONCLUSION
We proposed and analyzed the trust region method for multi-agent learning problems, which considers the trust payoff region and the trust stable region to meet the multi-agent learning objectives.
In practice, based on independent trust payoff learners, we provide a convenient way to approximate a further restricted step size within TSR via the meta-game.
This ensures that the MATRL is generalized, ﬂexible, and easily implemented to deal with multi-agent learning problems in general.
Our experimental results justify the fact that MATRL method signiﬁcantly outperforms independent learners using the same conﬁgurations, and other strong MARL baselines on both continuous and discrete games with various agent numbers.
A MATRL A LGORITHM B ASED ON PPO
for do 2:
Using π 1 ( θ 1 ) , π 2 ( θ 2 ) to collect trajectories τ 1 , τ 2 .
Compute GAE reward ˆ R i for each i .
Compute estimated advantages ˆ A 1 , ˆ
A 2 based on the current value functions V φ 1 , V φ 2 .
5: for i ∈ { 1 , 2 } do 6:
Compute a trust payoff region policy ˆ
π i using Eq
Update the policy by maximizing the
PPO-Clip objective: ˆ θ i = argmax θ i 1 | τ i | T (cid:80) τ ∈ τ i (cid:80)
Tt =
min (cid:16) π i ( a t | s t ; θ ) π i ( a 1 ,t | s t ; θ i )
A π 1 ,π 2 i ( s t ,a 1 ,t ,a 2 ,t ) , g ( (cid:15),A π 1 ,π 2 i ( s t ,a 1 ,t ,a 2 ,t )) (cid:17) , where g is a clipping function.
Fit value function by regression on mean-squared error: φ (cid:48)
i =
arg min φ i | τ i | T (cid:88) τ ∈ τ
i
T (cid:88) t = (cid:16) V φ ( s t ) − ˆ R
i,t (cid:17)9: end for 10:
Construct the meta-game M ( π 1 ( θ 1 ) , ˆ π 1 (ˆ θ 1 ) , π 2 ( θ 2 ) , ˆ π 2 (ˆ θ 2 )) .
Solve M and obtain meta Nash
ρ 1 , ρ 2 .
Compute aggregated weak stable ﬁxed point (¯ π 1 , ¯ π 2 ) .
13: for i ∈ { 1 , 2 } do 14:
Compute π ( (cid:48) )
i which best responses to ¯ π − i using Eq.
Estimate the best response by importance sampling:
[Equation]
16: end for 17: θ 1 ← θ (cid:48) 1 , θ 2 ← θ (cid:48) 2 .
18: end for
Output: π 1 ( θ 1 ) , π 2 ( θ 2 ) .
B I NDEPENDENT T RUST P AYOFF R EGION
We use the total variation divergence, which is deﬁned by D TV ( p (cid:107) q ) = 12 (cid:80) j | p j − q j | for discrete probability distributions p, q (Schulman et al., 2015).
D maxTV ( π, ˜ π ) is deﬁned as:
[Equation]
Based on this, we can deﬁne α -coupled policy as:
Deﬁnition 2 ( α -Coupled Policy (Schulman et al., 2015)) .
( π, π (cid:48) ) is an α -coupled policy pair if it deﬁnes a joint distribution ( a, a (cid:48) )
| s , such that P ( a (cid:54) = a (cid:48) | s ) ≤ α for all s .
π and π (cid:48) will denote the marginal distributions of a and a (cid:48) , respectively.
When the joint policy pair π i , π − i changes to π (cid:48)
i , π (cid:48)− i and coupled with α i and α − i correspondingly:
[Equation]
where
[Equation]
The proofs are as following:
Lemma 1.
Given that ( π i , π (cid:48) i ) and ( π − i , π (cid:48)− i ) are both α -coupled policies bounded by α i and α − i respectively, for all s ,
[Equation]
Proof.
[Equation]
where P ( a i (cid:54) = a (cid:48)
i ∨ a − i (cid:54) =
a (cid:48)− i | s ) = 1 − (1 − α i )(1 − α − i )
=
α i + α − i − α i
α − i .
Lemma 2.
Let ( π i , π (cid:48) i ) and ( π − i , π (cid:48)− i ) are α -coupled policy pairs.
Then,
[Equation]
Proof.
The preceding Lemma bounds the difference in expected advantage at each time step t .
When t (cid:48)
= 0 indicates that π i , π − i and π (cid:48)
i , π (cid:48)− i both agreed on all time steps less than t .
By the deﬁnition of α i , α − i , P ( π i , π − i :=
π (cid:48)
i , π (cid:48)− i | t = i ) ≥ (1 −
[Equation]
i )
[Equation]
1 −
[Equation]
− i ) , so P
[Equation]
t
[Equation]
= 0)
[Equation]
(1 − α i ) t (1 − α − i ) t and P ( t (cid:48) > 0) ≤ 1 − (1 − α i ) t (1 − α − i ) t .
We can sum over time to bind the difference between η i ( π (cid:48)
i , π (cid:48)− i ) and η i ( π i , π − i ) .
[Equation]
(20) where (cid:15)
=
max s,a i ,a − i (cid:12)(cid:12)
A π i ,π − i i ( s, a i , a − i ) (cid:12)(cid:12) .
Note that
L π i ,π − i
i
[Equation] [Equation]
Then, we can have
[Equation]
C P ROOF OF T HEOREM
At each iteration, denote ∇ i g i = ∇ π
i g π i ,π − i i and ∇ i, − i g i = ∇ π i ∇ π −
i g π i ,π − i i for each i .
Consider the simultaneous gradient ξ of the expected advantage gains and the corresponding Hessian H :
[Equation]
[Equation] [Equation]
For a restricted underlying game, where policy space is bounded: π i ∈ [ π i , ˆ π i ] .
Assume π
i is the linear mixture of π i , ˆ π i , and ¯ π i
=
ρ
i π i + (1 − ρ i )
ˆ π i , where ρ i ∈ [0 , 1] .
Therefore, we can re-write the g π i ,π − i i
( π i , π − i ) in the form of: g π i ,π − i
[Equation]
Then we have:
[Equation]
and ξ ( π i , π − i ) =
ξ ( ρ i , ρ − i ) .
Given a meta Nash policy pair (¯ π i , ¯ π − i ) , where ¯ π i = ¯ ρ i π i +(1 − ¯ ρ i )
ˆ π i , according to the Nash deﬁnition, we have:
[Equation]
which implies:
[Equation]
When ¯ ρ i , ¯ ρ − i ∈ (0 , 1) in accordance with the Nash condition in Eq.
28, ∇ i
g i (¯ ρ − i ) =
∇ − i g − i (¯ ρ i )
= 0 .
It shows that (¯ π i , ¯ π − i ) is a ﬁxed point due to ξ (¯ π i , ¯ π − i ) =
ξ (¯ ρ i , ¯ ρ − i )
= 0 .
For the boundary case, where ¯ ρ i or ¯ ρ − i ∈ { 0 , 1 } , because they are constrained to the unit square [0 , 1] × [0 , 1] , the gradients on the boundaries of the unit square are projected onto the unit square, which means additional points of zero gradient exist.
In other words, ∇ i g i and ∇ −
i g − i are still equal to zero in boundary case, and the (¯ π i , ¯ π − i ) is a ﬁxed point in both cases.
Next, we determine what types of the ﬁxed point that (¯ π i , ¯ π − i ) belongs to.
According to the Eq.
24, we have the exact Hessian Matrix for the restricted game:
[Equation]
The eigenvalue λ of H can be computed:
[Equation]
Denotes ¯ g i :=
g ˆ i, − ˆ i i − g i, − ˆ i
i − g
ˆ i, − i i , we have λ = ±√ ¯
g i ¯ g − i .
Therefore, we can have following cases for the ﬁxed point (¯ ρ i , ¯ ρ − i ) :
Fully cooperative games: ¯ g i ≤ 0 , ¯ g − i ≤ 0 , then H (¯ ρ i , ¯ ρ − i ) (cid:22) 0 , which means (¯ ρ i , ¯ ρ − i ) is a stable ﬁxed point as we are maximizing the objective.
Fully competitive games: ¯ g i > 0 , ¯ g − i < 0 or ¯ g i < 0 , ¯ g − i > 0 , all λ have two pure imaginary eigenvalues with zero real part, where (¯ ρ i , ¯ ρ − i ) is a saddle point.
General sum games: they are in-between the cooperative and competitive games, which means (¯
ρ i , ¯ ρ − i ) can be either stable ﬁxed point or saddle point.
Because we assume ˆ π i monotonically improved compared to π i , then even in zero-sum case, there is at least one negative value in ¯ g i and ¯ g − i .
Therefore, in all the situations, (¯ ρ i , ¯ ρ − i ) is not unstable, and could be a stable point or saddle point.
We deﬁne them as a weak stable ﬁxed point.
It also has a tighter lower bound than the independent trust region improvement seen in Remark 1:
[Figure]
[Figure]
o li c y o f P l a y e r
[Figure]
Remark 1.
Let ( ρ i , ρ − i ) be a Nash equilibrium of the policy-space meta-game M ( π i , ˆ π i , π − i , ˆ π − i ) , which is used for computing the linear mixture policies ¯ π i , ¯ π − i .
For simplicity, deﬁne ¯ ρ i = 1 − ρ i , then we have the payoff improvement lower bound for ¯ π i , ¯ π − i :
η i (¯ π i , ¯ π − i ) − η i ( π i , π − i ) ≥ g π i ,π − i i (¯ π i , ¯ π − i ) − 4 γ(cid:15) i (1 − γ ) 2 (
[Equation]
that is a tighter lower bound compared with Theorem 1.
Finally, we obtain MATRL as follows:
First, an agent i collects a set of trajectories using its current policy π i by independent play with other agents.
Then a predicted policy ˆ π i can be estimated using the single-agent trust region methods, which has a trust payoff improvement against the other agents’ current policy π − i .
However, this trust payoff improvements would not beneﬁt convergence requirements for the multi-agent system due to other agents adaptive learning.
To solve this problem, we approximate a n -agent two-action meta-game in policy-space by reusing the trajectories from the last TPR step.
In this game, each agent i has two pure strategies: choosing the current policy π i or predicted policy ˆ π i and the corresponding payoffs are the expected advantages (deﬁned in
Eq. 3) of the joint policy pairs.
By constructing such a meta-game, we transform a complex multiagent interactions problem into game-theoretic analysis concerning the underlying game restricted in [ π i , ˆ π i ] .
Then we can obtain a weak stable ﬁxed point as TSR within the TPR by solving the meta-game,.
When the ﬁxed point is a saddle point we then take the best response to the weak stable ﬁxed point to get the next iteration’s policies.
This encourages exploration and avoid stagnation at an unexpected saddle point.
D E NVIRONMENT D ETAILS
Random 2 × 2 Matrix Games .
We created a generator of 2 × 2 matrix games based on the category provided by (Pangallo et al., 2017).
Coordination games have characteristics enabling one agent to improve the payoff without decreasing the payoff of the other agent.
Anti-coordination games are ones where one agent improves the payoff while the other agent’s payoff decreases.
Both coordination and anti-coordination games can have two pure NEs and one mixed strategy NE.
In cyclic games, the action selections of agents that is based on their actions will form a cycle, ensuring that there is no pure NE in the game.
Instead only mixed strategy NE will be found.
Grid World Games .
In two-player checker, as shown in Fig.
8a, there is one sensitive player who gets reward 5 when they collect an apple and 5 when they collect a lemon; a less sensitive player gets 1 for apple and 1 for lemon.
The learning goal is to let the sensitive player get apples and the other one get lemons to have a higher total reward.
In four-player switch, as shown in Fig.
8b, to reach the targets, agents need to ﬁgure out a way to go through a narrow corridor.
The agent gets − 1 for taking each step and 5 when arriving at a target.
Four-player switch uses the same map as two-player switch, where two agents start from the left side and the others from the right side to go through the corridor to reach the targets.
With more agents in four-player switch, learning becomes more challenging.
MATRL agents achieved higher total rewards compared to baseline algorithms within the same number of steps.
Multi-Agent MuJoCo Tasks .
We used the three-agent Hopper environment described in (de Witt et al., 2020), and Fig. 8c, where three agents control three joints of the robot and learn to cooperate to move forward as far as possible.
The agent is rewarded by the number of time steps that they move without falling.
Each agent has 3 continuous output values as the action, and all the agents have a full observation of the states of size 17 .
We use the same hyper-parameters for MATRL, MATRL w/o
BR, and IL-PP.
For MADDPG agent, we use the hyper-parameters described in the paper (de Witt et al., 2020).
Multi-Agent Atari Game .
The pong game is a multi-agent Atari version 2 of table tennis, as shown in Fig.
Two players must prevent a ball from whizzing past their paddles and allowing their opponent to score.
The game ends when one side earns 21 points.
E E XPERIMENTAL P ARAMETER S ETTINGS
For all the tasks, the most important hyper-parameters are learning rate/step size, the number of update steps, batch size and value, policy loss coefﬁcient.
Appropriate learning rate and update steps plus larger batch size give a more stable learning curve.
And for different environments, policy and value network loss coefﬁcients that keep two losses at the same scale are essential in improving the learning result and speed.
Also, for meta-game construction and best response update where we use the importance ratio to do estimation, a clipping factor of the ration is vital to achieving a stable and monotonic improving result.
The followings are the detailed parameter settings for each task.
Matrix Game and Random 2 × 2 Matrix Games .
The hyper-parameters settings for MATRL,
IGA-PP, and WoLF are listed in Table 2.
As shown in Fig.
9, we also listed the additional convergence analysis in classical Chicken and Prisoners’ Dilemma Games, which demonstrate good convergence performance of MATRL on both games.
For MATRL, we have the KL-divergence coefﬁcient as an extra hyper-parameter to add the KL-divergence as part of the loss in policy updating.
And for the baseline algorithm WoLF, we give the real NE of the game as part of the parameters.
In all the games, all the algorithms shared the same initial policy values [0 . 9 , 0 . 1] for player 1 and [0 . 2 , 0 . 8] for player 2.
Grid World Games and Multi-agent Continuous Control Task .
The hyper-parameters settings for MATRL are given in Table 3.
We used the same hyper-parameters for MATRL, MATRL w/o BR,
IL-PP, and IL.
The only difference is whether to use Best Response and the meta-game or not.
We
[Table] [Table]
used Leaky ReLU as the activation function for both policies and value networks.
For the training, we used paralleled workers to collect experience data and update the network weights separated then synchronize all the works to have the ﬁnal updated weights.
We used different value loss and policy loss coefﬁcients to balance the weights of two losses.
For the Switch games, we used small value loss coefﬁcients because the value loss is between [0 − 10] while the absolute value policy loss is smaller than 1 e − 2 .
For the Checker game, the value loss and policy loss are at the same scale between [1 e − 4 , 1 e − 2] .
Also, we added entropy loss and KL loss to encourage exploration and limit the policy update for each step.
We used (ˇSebek, 2013) as the Nash equilibrium solver for ﬁnding the meta-game Nash.
The Nash solver is CMAES for all the experiments.
If not particularly indicated, all the baselines use common settings as listed in Table 3.
VDN, QMIX use common individual action-value networks as those used by MATRL; each consists of two 128-width hidden layers.
We includes more experiment result on 4-agent ant task multi-agent MuJoCo task in Fig.
9c, which also demonstrate the superior performance of MATRL compared to other settings.
The specialized parameter settings for each algorithm are provided in Table 4 and 5:
Multi-agent Atari Pong .
The hyper-parameters setting for MATRL are listed in Table 6.
We used the same hyper-parameters for MATRL and IL.
We take the raw pixel input from the Atari environment, and we processed it with a convolution network, which has ﬁlter sizes [8,4,3], kernel sizes (3,3,3), and stride sizes [4,2,1] and ”VALID” as padding.
Then we pass the processed embedding to a 2 layer fully connected network to get the policy.
[Table]
[Table] [Table]