DOP: O FF -P OLICY M ULTI -A GENT D ECOMPOSED P OLICY G RADIENTS
A BSTRACT
Multi-agent policy gradient (MAPG) methods recently witness vigorous progress.
However, there is a signiﬁcant performance discrepancy between MAPG methods and state-of-the-art multi-agent value-based approaches.
In this paper, we investigate causes that hinder the performance of MAPG algorithms and present a multi-agent decomposed policy gradient method (DOP).
This method introduces the idea of value function decomposition into the multi-agent actor-critic framework.
Based on this idea, DOP supports efﬁcient off-policy learning and addresses the issue of centralized-decentralized mismatch and credit assignment in both discrete and continuous action spaces.
We formally show that DOP critics have sufﬁcient representational capability to guarantee convergence.
In addition, empirical evaluations on the StarCraft II micromanagement benchmark and multi-agent particle environments demonstrate that DOP outperforms both state-of-the-art value-based and policy-based multi-agent reinforcement learning algorithms.
Demonstrative videos are available at https://sites.google.com/view/dop-mapg/ .
I NTRODUCTION
Cooperative multi-agent reinforcement learning (MARL) has achieved great progress in recent years (Hughes et al., 2018;
Jaques et al., 2019; Vinyals et al., 2019; Zhang et al., 2019; Baker et al., 2020; Wang et al., 2020c).
Advances in valued-based MARL (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019; Wang et al., 2020e) contribute signiﬁcantly to the progress, achieving state-ofthe-art performance on challenging tasks, such as StarCraft II micromanagement (Samvelyan et al., 2019).
However, these value-based methods present a major challenge for stability and convergence in multi-agent settings (Wang et al., 2020a), which is further exacerbated in continuous action spaces.
Policy gradient methods hold great promise to resolve these challenges.
MADDPG (Lowe et al., 2017) and COMA (Foerster et al., 2018) are two representative methods that adopt the paradigm of centralized critic with decentralized actors (CCDA), which not only deals with the issue of nonstationarity (Foerster et al., 2017; Hernandez-Leal et al., 2017) by conditioning the centralized critic on global history and actions but also maintains scalable decentralized execution via conditioning policies on local history.
Several subsequent works make improvements to the CCDA framework by introducing the mechanism of recursive reasoning (Wen et al., 2019) or attention (Iqbal & Sha, 2019).
Despite the progress, most of the multi-agent policy gradient (MAPG) methods do not provide satisfying performance, e.g., signiﬁcantly underperforming value-based methods on benchmark tasks (Samvelyan et al., 2019).
In this paper, we analyze this discrepancy and pinpoint three major issues that hinder the performance of MAPG methods.
(1) Current stochastic MAPG methods do not support off-policy learning, partly because using common off-policy learning techniques is computationally expensive in multi-agent settings.
In the CCDA paradigm, the suboptimality of one agent’s policy can propagate through the centralized joint critic and negatively affect policy learning of other agents, causing catastrophic miscoordination, which we call centralized-decentralized mismatch (CDM).
(3) For deterministic MAPG methods, realizing efﬁcient credit assignment (Tumer et al., 2002; Agogino & Tumer, 2004) with a single global reward signal largely remains challenging.
In this paper, we ﬁnd that these problems can be addressed by introducing the idea of value decomposition into the multi-agent actor-critic framework and learning a centralized but factorized critic.
This framework decomposes the centralized critic as a weighted linear summation of individual critics that condition on local actions.
This decomposition structure not only enables scalable learning on the critic, but also brings several beneﬁts.
It enables tractable off-policy evaluations of stochastic policies, attenuates the CDM issues, and also implicitly learns an efﬁcient multi-agent credit assignment.
Based on this decomposition, we develop efﬁcient off-policy multi-agent decomposed policy gradient methods for both discrete and continuous action spaces.
A drawback of an linearly decomposed critic is its limited representational capacity (Wang et al., 2020b), which may induce bias in value estimations.
However, we show that this bias does not violate the policy improvement guarantee of policy gradient methods and that using decomposed critics can largely reduce the variance in policy updates.
In this way, a decomposed critic achieves a great bias-variance trade-off.
We evaluate our methods on both the StarCraft II micromanagement benchmark (Samvelyan et al., 2019)
(discrete action spaces) and multi-agent particle environments (Lowe et al., 2017; Mordatch & Abbeel, 2018)
(continuous action spaces).
Empirical results show that DOP is very stable across different runs and outperforms other MAPG algorithms by a wide margin.
Moreover, to our best knowledge, stochastic DOP provides the ﬁrst MAPG method that outperforms state-of-the-art valuedbased methods in discrete-action benchmark tasks.
Related works on value decomposition methods.
In value-based MARL, value decomposition (Guestrin et al., 2002b; Castellini et al., 2019) is widely used.
These methods learn local
Q-value functions for each agent, which are combined with a learnable mixing function to produce global action values.
In VDN (Sunehag et al., 2018), the mixing function is an arithmetic summation.
QMIX (Rashid et al., 2018; 2020) proposes a non-linear monotonic factorization structure.
QTRAN (Son et al., 2019) and QPLEX (Wang et al., 2020b) further extend the class of value functions that can be represented.
NDQ (Wang et al., 2020e) addresses the miscoordination problem by learning nearly decomposable architectures.
A concurrent work (de Witt et al., 2020) ﬁnds that a decomposed centralized critic in QMIX style can improve the performance of MADDPG for learning in continuous action spaces.
In this paper, we study how and why linear value decomposition can enable efﬁcient policy-based learning in both discrete and continuous action spaces.
In Appendix F, we discuss how DOP is related to recent progress in multi-agent reinforcement learning and provide detailed comparisons with existing multi-agent policy gradient methods.
B ACKGROUND
We consider fully cooperative multi-agent tasks that can be modelled as a Dec-POMDP (Oliehoek et al., 2016)
G = (cid:104)
I, S, A, P, R, Ω , O, n, γ (cid:105) , where I is the ﬁnite set of agents, γ ∈ [0 , 1) is the discount factor, and s ∈ S is the true state of the environment.
At each timestep, each agent i receives an observation o
i ∈ Ω drawn according to the observation function
O ( s, i ) and selects an action a i ∈ A , forming a joint action a ∈
A n , leading to a next state s (cid:48) according to the transition function P ( s (cid:48) | s, a ) and a reward r = R ( s, a ) shared by all agents.
Each agent learns a policy π i ( a i |
τ i ; θ i ) , which is parameterized by θ i and conditioned on the local history τ i ∈ T ≡ (Ω × A ) ∗ .
The joint policy π , with parameters θ = (cid:104) θ 1 , · · · , θ n (cid:105) , induces a joint action-value function: Q π tot ( τ , a ) =
E s 0: ∞ , a
0: ∞ [ (cid:80) ∞ t =0 γ t R ( s t , a t ) | s
= s, a 0 = a , π ] .
We consider both discrete and continuous action spaces, for which stochastic and deterministic policies are learned, respectively.
To distinguish deterministic policies, we denote them by µ = (cid:104) µ 1 , · · · , µ n (cid:105) .
Multi-Agent Policy Gradients
The centralized training with decentralized execution (CTDE) paradigm (Foerster et al., 2016; Wang et al., 2020d) has recently attracted attention for its ability to address non-stationarity while maintaining decentralized execution.
Learning a centralized critic with decentralized actors (CCDA) is an efﬁcient approach that exploits the CTDE paradigm.
MADDPG and COMA are two representative examples.
MADDPG (Lowe et al., 2017) learns deterministic policies in continuous action spaces and uses the following gradients to update policies:
[Equation] [Equation]
where A π i ( τ , a ) =
Q π tot ( τ , a ) −
[Equation]
a
[Equation]
i Q π tot ( τ , ( a - i , a (cid:48) i )) is a counterfactual advantage ( a - i is the joint action other than agent i ) that deals with the issue of credit assignment and reduces variance.
A NALYSIS
In this section, we investigate challenges that limit the performance of state-of-the-art multi-agent policy gradient methods.
3. O FF -P OLICY L EARNING FOR M ULTI -A GENT S TOCHASTIC P OLICY G RADIENTS
Efﬁcient stochastic policy learning in single-agent settings relies heavily on using off-policy data (Lillicrap et al., 2015;
Wang et al., 2016; Fujimoto et al., 2018; Haarnoja et al., 2018), which is not supported by existing stochastic MAPG methods (Foerster et al., 2018).
In the CCDA framework, off-policy policy evaluation—estimating Q π tot from data drawn from behavior policies β = (cid:104) β 1 , . .
, β n (cid:105) —encounters major challenges.
Importance sampling (Meuleau et al., 2000;
Jie & Abbeel, 2010; Levine & Koltun, 2013) is a simple way to correct for the discrepancy between π and β , but, it requires computing (cid:81)
i π i ( a i | τ i ) β
i ( a i | τ i ) , whose variance grows exponentially with the number of agents in multi-agent settings.
An alternative is to extend the tree backup technique (Precup et al., 2000; Munos et al., 2016) to multi-agent settings and use the k -step tree backup update target for training the critic:
[Equation]
where τ 0 = τ , a 0 = a .
However, the complexity of computing E π
[ Q π tot ( τ t +1 , · )] is O
( | A | n ) , which becomes intractable when the number of agents is large.
Therefore, it is challenging to develop off-policy stochastic MAPG methods.
3. T HE C ENTRALIZED -D ECENTRALIZED M ISMATCH I SSUE
In the centralized critic with decentralized actors (CCDA) framework, agents learn individual policies, π i ( a i | τ i ; θ i ) , conditioned on the local observation-action history.
However, the gradients for updating these policies are dependent on the centralized joint critic, Q π tot ( τ , a ) (see Eq.
1 and 2), which introduces the inﬂuence of actions of other agents.
Intuitively, gradient updates will move an agent in the direction that can increase the global Q value, but the presence of other agents’ actions incurs large variance in the estimates of such directions.
Formally, the variance of policy gradients for agent i at ( τ i , a i ) is dependent on other agents’ actions:
[Equation]
where Var a - i [ Q π tot ( τ , ( a i , a - i ))] can be very large due to the exploration or suboptimality of other agents’ policies, which may cause suboptimality in individual policies.
For example, suppose that the optimal joint action under τ is a ∗ =
(cid:104) a ∗ 1 , . . .
, a ∗ n (cid:105) .
When E a - i ∼ π - i
[ Q π tot ( τ , ( a ∗ i , a - i ))] < 0 , π i ( a ∗ i | τ i ) will decrease, possibly resulting in a suboptimal π i .
This becomes problematic because a negative feedback loop is created, in which the joint critic is affected by the suboptimality of agent i , which disturbs policy updates of other agents.
We call this issue centralized-decentralized mismatch (CDM).
Does CDM occur in practice for state-of-the-art algorithms?
To answer this question, we carry out a case study in Sec. 5.1.
We can see that the variance of DOP gradients is signiﬁcantly smaller than COMA and MADDPG (Fig.
2 left).
This smaller variance enables DOP to outperform other algorithms (Fig.
2 middle).
We will explain this didactic example in detail in Sec. 5.1.
In Sec. 5.
and 5.3, we further show that CDM is exacerbated in sequential decision-making settings, causing divergence even after a near-optimal strategy has been learned.
3. C REDIT A SSIGNMENT FOR M ULTI -A GENT D ETERMINISTIC P OLICY G RADIENTS
MADDPG (Lowe et al., 2017) and MAAC (Iqbal & Sha, 2019) extend deterministic policy gradient algorithms (Silver et al., 2014; Lillicrap et al., 2015) to multi-agent settings, enabling efﬁcient offpolicy learning in continuous action spaces.
However, they leave the issue of credit assignment (Tumer et al., 2002; Agogino & Tumer, 2004) largely untouched in fully cooperative settings, where agents learn policies from a single global reward signal.
In stochastic cases, COMA assigns credits by designing a counterfactual baseline (Eq. 2).
However, it is not straightforward to extend COMA to deterministic policies, since the output of polices is no longer a probability distribution.
As a result, it remains challenging to realize efﬁcient credit assignment in deterministic cases.
D ECOMPOSED O FF -P OLICY P OLICY G RADIENTS
To address the limitations of existing MAPG methods discussed in Sec. 3, we introduce the idea of value decomposition into the multi-agent actor-critic framework and propose a Decomposed
Off-Policy policy gradient (DOP) method.
We factor the centralized critic as a weighted summation of individual critics across agents:
[Equation]
where φ and φ i are parameters of the global and local Q functions, respectively, and k i ≥ 0 and b are generated by learnable networks whose inputs are global observation-action histories.
In the following sections, we show that this linear decomposition helps address existing limitations of previous methods.
A concern is the limited expressivity of linear decomposition (Wang et al., 2020b), which may introduce bias in value estimations.
We will show that this limitation does not violate the policy improvement guarantee of DOP.
Fig.
1 shows the architecture for learning decomposed critics.
We learn individual critics Q
[Equation]
i by backpropagating gradients from global TD updates dependent on the joint global reward, i.e., Q φ i i is learned implicitly rather than from any reward speciﬁc to agent i .
We enforce k i ≥ 0 by applying an absolute activation function at the last layer of the network.
The network structure is described in detail in Appendix H.
Based on the critic decomposition learning, the following sections will introduce decomposed off-policy policy gradients for learning stochastic policies and deterministic policies, respectively.
Similar to other actor-critic methods, DOP alternates between policy evaluation —estimating the value function for a policy—and policy improvement —using the value function to update the policy (Barto et al., 1983).
4. S TOCHASTIC D ECOMPOSED O FF -P OLICY P OLICY G RADIENTS
For learning stochastic policies, the linearly decomposed critic plays an essential role in enabling tractable multi-agent tree backup for off-policy policy evaluation and attenuating the CDM issue while maintaining provable policy improvement.
4.1. O FF -P OLICY L EARNING
Policy Evaluation: Train the Critic As discussed in Sec. 3.1, using tree backup (Eq. 3) to carry out multi-agent off-policy policy evaluation requires calculating E π [ Q φtot ( τ t +1 , · )] , which needs O ( | A | n ) steps of summation when a joint critic is used.
Fortunately, using the linearly decomposed critic,
DOP reduces the complexity of computing this expectation to O ( n | A | ) :
[Equation]
(6) making the tree backup technique tractable (detailed proof can be found in Appendix A.1).
Another challenge of using multi-agent tree backup (Eq. 3) is that the coefﬁcient c t = (cid:81) tl =1 λ π ( a l | τ l ) decays as t gets larger, which may lead to relatively lower training efﬁciency.
To solve this issue, we propose to mix off-policy tree backup updates with on-policy TD ( λ ) updates to trade off sample efﬁciency and training efﬁciency.
Formally, DOP minimizes the following loss for training the critic:
[Equation]
where κ is a scaling factor, β is the joint behavior policy
[Equation]
and φ is the parameters of the critic.
The ﬁrst loss item is L DOP - TB β
[Equation]
=
[Equation] [Equation]
y
[Equation]
Q φtot
[Equation]
τ , a )) 2 ] , where y DOP - TB is the update target of the proposed k -step decomposed multi-agent tree backup algorithm:
[Equation]
Here, φ (cid:48) is the parameters of a target critic, and a t ∼ β ( ·| τ t ) .
The second loss item is L On π ( φ ) =
E π
[( y On − Q φtot ( τ , a )) 2 ] , where y On is the on-policy update target as in TD ( λ ) :
[Equation]
In practice, we use two buffers, an on-policy buffer for computing L On π ( φ ) and an off-policy buffer for estimating L DOP - TB β ( φ ) .
Policy Improvement: Train Actors Using the linearly decomposed critic architecture, we can derive the following on-policy policy gradients for learning stochastic policies:
[Equation]
In Appendix A.2, we provide the detailed derivation and an off-policy version of stochastic policy gradients.
This update rule reveals two important insights.
(1) With a linearly decomposed critic, each agent’s policy update only depends on the individual critic Q
φ i i .
(2) Learning the decomposed critic implicitly realizes multi-agent credit assignment, because the individual critic provides credit information for each agent to improve its policy in the direction of increasing the global expected return.
Moreover, Eq. 10 is also the policy gradients when assigning credits via the aristocrat utility (Wolpert & Tumer, 2002)
(Appendix A.2).
Eq.
7 and 10 form the core of our DOP algorithm for learning stochastic policies, which we call stochastic DOP and is described in detail in Appendix E.
The CDM Issue occurs when decentralized policies’ suboptimality exacerbates each other through the joint critic.
As an agent’s stochastic DOP gradients do not rely on the actions of other agents, they attenuate the effect of CDM.
We empirically show that DOP can reduce variance in policy gradients in Sec.
5.1 and can attenuate the CDM issue in complex tasks in Sec.
4.1. S TOCHASTIC DOP P OLICY I MPROVEMENT T HEOREM
In this section, we theoretically demonstrate that stochastic DOP can converge to local optimal despite the fact that a linearly decomposed critic has limited representational capability.
Since an accurate analysis for a complex function approximator (e.g., neural network) is difﬁcult, we adopt several mild assumptions used in previous work (Feinberg et al., 2018; Degris et al., 2012).
We ﬁrst show that the linearly decomposed structure ensures that the learned local value functions
Q φ i i ( τ , a i ) preserve the order of Q π i ( τ , a i ) =
(cid:80) a - i π - i ( a - i | τ - i )
Q
π tot ( τ , a ) for a wide range of function class.
Fact 1.
Under mild assumptions, when value evaluation converges, ∀ π , Q φ i i satisﬁes that
[Equation]
Detailed proof of Fact 1 can be found in Appendix C.1 as well as more detailed discussion of its implications.
Furthermore, we prove the following proposition to show that policy improvement can be guaranteed as long as the function class expressed by Q φ i i is sufﬁciently large and the loss of critic training is minimized.
Proposition 1.
Suppose the function class expressed by Q φ i i ( τ , a i ) is sufﬁciently large (e.g. neural networks) and the following loss L ( φ ) is minimized
[Equation]
where Q φtot ( τ , a ) ≡ (cid:80) i k i ( τ ) Q φ i i ( τ , a i ) + b ( τ ) .
Then, we have
[Equation]
which means stochastic DOP policy gradients are the same as those calculated using centralized critics (Eq. 2).
Therefore, policy improvement is guaranteed.
The proof can be found in Appendix C.2, which is inspired by Wang et al.
(2020a).
4. D ETERMINISTIC D ECOMPOSED O FF -P OLICY P OLICY G RADIENTS 4.2. O FF -P OLICY L EARNING
To enable efﬁcient learning with continuous actions, we propose deterministic DOP .
As in singleagent settings, because deterministic policy gradient methods avoid the integral over actions, it greatly eases the cost of off-policy learning (Silver et al., 2014).
For policy evaluation , we train the critic by minimizing the following TD loss:
[Equation]
(11) where D is a replay buffer, and φ (cid:48) , θ (cid:48) are the parameters of the target critic and actors, respectively.
For policy improvement , we derive the following deterministic DOP policy gradients:
[Equation]
Detailed proof can be found in Appendix B.1.
Similar to the stochastic case, This result reveals that updates of individual deterministic policies depend on local critics when a linearly decomposed critic is used.
Based on Eq.
11 and Eq.
12, we develop the DOP algorithm for learning deterministic policies in continuous action spaces, which is described in Appendix E and called deterministic DOP .
4.2. R EPRESENTATION C APACITY OF D ETERMINISTIC DOP C RITICS
In continuous and smooth environments, we ﬁrst show that a DOP critic has sufﬁcient expressive capability to represent Q values in the proximity of µ ( τ ) , ∀ τ with a bounded error.
For simplicity, we denote O δ
( τ ) =
{ a | (cid:107) a − µ ( τ ) (cid:107) 2 ≤ δ } .
Fact 2.
Assume that ∀ τ , a , a (cid:48) ∈ O δ ( τ ) , (cid:107) ∇ a Q µ tot ( τ , a ) − ∇ a (cid:48) Q µ tot ( τ , a (cid:48) )
(cid:107) 2 ≤ L (cid:107) a − a (cid:48) (cid:107) 2 .
The estimation error of a DOP critic can be bounded by O ( Lδ 2 ) for a ∈ O δ ( τ ) , ∀ τ .
Detailed proof can be found in Appendix D.
Here we assume that the gradients of Q-values with respect to actions are Lipschitz smooth under the deterministic policy µ .
This assumption is reasonable given that Q-values of most continuous environments with continuous policies are rather smooth.
We further show that when Q-values in the proximity of µ ( τ ) , ∀ τ are well estimated with a bounded error, deterministic DOP policy gradients are good approximation to the true gradients (Eq. 1).
Approximately, |∇ a i Q µ tot ( τ , a ) − ∇ a
i
k
i ( τ ) ∇ a i Q
φ i i ( τ , a i )
| ∼ O
( Lδ ) , ∀ i when δ (cid:28) 1 .
For detailed proof, we refer readers to Appendix D.
E XPERIMENTS
We design experiments to answer the following questions: (1) Does the CDM issue commonly exist and can decomposed critics attenuate it?
(Sec. 5.1, 5.2.1, and 5.3) (2) Can our decomposed multi-agent tree backup algorithm improve the efﬁciency of off-policy learning?
(Sec. 5.2.1) (3)
Can deterministic DOP learn reasonable credit assignment?
(Sec. 5.3) (4) Can DOP outperform state-of-the-art MARL algorithms?
For evaluation, all the results are averaged over 12 different random seeds and are shown with 95% conﬁdence intervals.
[Figure]
5. D IDACTIC E XAMPLE : T HE CDM I SSUE AND B IAS -V ARIANCE T RADE -O FF
We use a didactic example to demonstrate how DOP attenuates CDM and achieves bias-variance trade-off.
In a state-less game with 3 agents and 14 actions, if agents take action1 , 5 , 9 , respectively, they get a team reward of 10 ; otherwise - 10 .
We train stochastic DOP, COMA
[Figure]
and MADDPG for 10 K timesteps and show the gradient variance, value estimation bias
[Figure]
and learning curves in Fig.
Gumbel-Softmax trick (Jang et al., 2017; Maddison et al., 2017) is used to enable MADDPG to learn in discrete action spaces.
Fig.
2-right shows the average bias in the estimations of all Q values.
We see that linear decomposition introduces extra estimation errors.
However, the variance of DOP policy gradients is much smaller than other algorithms (Fig.
2-left).
As discussed in Sec. 3.2, large variance of other algorithms is due to the CDM issue that undecomposed joint critics are affected by actions of all agents.
Free from the inﬂuence of other agents, DOP preserves the order of local Q-values (bottom of Fig.
2-right) and effectively reduces the variance of policy gradients.
In this way, DOP sacriﬁces value estimation accuracy for accurate and low-variance policy gradients, which explains why it can outperform other algorithms (Fig.
2-middle).
5. D ISCRETE A CTION S PACES : T HE S TAR C RAFT II M ICROMANAGEMENT B ENCHMARK
We evaluate stochastic DOP on the challenging SMAC benchmark (Samvelyan et al., 2019) for its high control complexity.
We compare our method with the state-of-the-art multi-agent stochastic policy gradient method (COMA), value-based methods (VDN, QMIX, QTRAN (Son et al., 2019),
NDQ (Wang et al., 2020e), and QPLEX (Wang et al., 2020b)), exploration method (MAVEN, Mahajan et al.
and role-based method (ROMA, Wang et al. (2020c)).
For stochastic DOP, we ﬁx the hyperparameter setting and network structure in all experiments which are described in Appendix H.
For baselines, we use their default hyperparameter settings that have been ﬁne-tuned on the SMAC benchmark.
Results are shown in Fig.
Stochastic DOP signiﬁcantly outperforms all the baselines by a wide margin.
To our best knowledge, this is the ﬁrst time that a MAPG method has signiﬁcantly better performance than state-of-the-art value-based methods.
[Figure]
T (
[Figure]
T e s t W i n %
[Figure] [Figure]
[Figure]
T e s t W
i n % 2s3z     5 T (mil)
T e s t W i n %
MMM     5 T (mil)
[Figure]
T e s t W i n %
[Figure] [Figure]
5.2. A BLATIONS
Stochastic DOP has three main components: (a) off-policy policy evaluations, (b) the decomposed critic, and (c) decomposed multi-agent tree backup.
By design, component (a) improves sample efﬁciency, component (b) can attenuate the CDM issue, and component (c) makes off-policy policy evaluations tractable.
We test the contribution of each component by carrying out the following ablation studies.
Off-Policy Learning
In our method, κ controls the "off-policyness" of training.
For DOP, we set κ to 0 .
To demonstrate the effect of off-policy learning, we change κ to 0 and 1 and compare the performance.
In Fig.
4, we can see that both DOP and off-policy DOP perform much better than the on-policy version ( κ =0), highlighting the importance of using off-policy data.
Moreover, purely off-policy learning generally needs more samples to achieve similar performance to DOP.
Mixing with on-policy data can largely improve training efﬁciency.
The CDM Issue On-Policy DOP uses the same decomposed critic structure as DOP, but is trained only with on-policy data and does not use tree backup.
The only difference between On-Policy
DOP and COMA is that the former one uses a decomposed joint critic.
Therefore, given that a
COMA critic has a more powerful expression capacity than a DOP critic, the outperformance of
On-Policy DOP against COMA shows the effect of CDM.
COMA is not stable and may diverge after a near-optimal policy has been learned.
For example, on map so _ many _ baneling , COMA policies degenerate after 2M steps.
In contrast, On-Policy DOP can converge with efﬁciency and stability.
Decomposed Multi-Agent Tree Backup DOP with Common Tree Backup (DOP without component (c)) is the same as DOP except that E π [ Q φtot ( τ , · )] is estimated by sampling 200 joint actions from π .
Here, we estimate this expectation by sampling because direct computation is intractable (for example, 20 10 summations are needed on the map MMM ).
Fig.
4 shows that when the number of agents increases, sampling becomes less efﬁcient, and common tree backup performs even worse than On-Policy DOP .
In contrast, DOP with decomposed tree backup can quickly and stably converge using a similar number of summations.
5. C ONTINUOUS A CTION S PACES : M ULTI -A GENT P ARTICLE E NVIRONMENTS
We evaluate deterministic DOP on multi-agent particle environments (MPE, (Mordatch & Abbeel, 2018)), where agents take continuous actions in continuous spaces.
We compare our method with
MADDPG (Lowe et al., 2017) and MAAC (Iqbal & Sha, 2019).
Hyperparameters and the network structure are ﬁxed for deterministic DOP across experiments, which are described in Appendix H.
The CDM Issue We use task Aggregation as an example to show that deterministic DOP attenuates the CDM issue.
In this task, 5 agents navigate to one landmark.
Only when all agents reach the landmark will they get a team reward of 10 and successfully end the episode; otherwise, an episode ends after 25 timesteps and agents get a reward of − 10 .
Aggregation is a typical example where other agents’ actions can inﬂuence an agent’s local policy through an undecomposed joint critic.
Intuitively, as long as one agent does not reach the landmark, the centralized Q value is negative, confusing other agents who get to the landmark.
This intuition is supported by the empirical results
[Figure]
shown in Fig.
5-left – methods with undecomposed critics can ﬁnd rewarding conﬁgurations but then quickly diverge, while DOP converges with stability.
Credit Assignment We use task Mill to show that DOP can learn effective credit assignment mechanisms.
In this task, 10 agents need to rotate a millstone clockwise.
They can push the millstone clockwise or counterclockwise with force between 0 and 1 .
If the millstone’s angular velocity, ω , gets greater than 30 , agents are rewarded 3 per step.
If ω exceeds 100 in 10 steps, the agents win the episode and get a reward of 10 ; otherwise, they lose and get a punishment of - 10 .
Fig.
5-right shows that deterministic DOP can gradually learn a reasonable credit assignment during training, where rotating the millstone clockwise has much larger Q-values.
This explains why deterministic
DOP outperforms previous state-of-the-art deterministic MAPG methods, as shown in Fig.
5-middle.
C LOSING R EMARKS
This paper pinpointed drawbacks that hinder the performance of state-of-the-art MAPG algorithms: on-policy learning of stochastic policy gradient methods, the centralized-decentralized mismatch problem, and the credit assignment issue in deterministic policy learning.
We proposed decomposed actor-critic methods (DOP) to address these problems.
Theoretical analyses and empirical evaluations demonstrate that DOP can achieve stable and efﬁcient multi-agent off-policy learning.
A CKNOWLEDGMENTS
We would like to thank the anonymous reviewers for their insightful comments and helpful suggestions.
This work is supported in part by Science and Technology Innovation 2030 – “New Generation
Artiﬁcial Intelligence” Major Project (No. 2018AAA0100904) and a grant from the Institute of Guo
Qiang, Tsinghua University
A M ATHEMATICAL DETAILS FOR STOCHASTIC DOP A. D ECOMPOSED CRITICS ENABLE TRACTABLE MULTI - AGENT TREE BACKUP
In Sec. 4.1.
[Equation]
we propose to use tree backup (Precup et al., 2000; Munos et al., 2016) to carry out multi-agent off-policy policy evaluation.
When a joint critic is used
[Equation]
calculating E π (cid:104) Q φtot ( τ , · ) (cid:105) requires O ( | A | n ) steps of summation.
To solve this problem
[Equation]
DOP uses a linearly decomposed critic
[Equation]
and it follows that:
[Equation]
which means the complexity of calculating this expectation is reduced to O ( n | A | ) .
A. S TOCHASTIC DOP POLICY GRADIENTS A.2. O N - POLICY VERSION
In Sec. 4.1.
[Equation]
we give the on-policy stochastic DOP policy gradients:
[Equation]
We now derive it in detail
[Equation]
Proof.
We use the aristocrat utility (Wolpert & Tumer, 2002) to perform credit assignment:
[Equation]
It is worth noting that U i is independent of other agents’ actions.
Then, for the policy gradients, we have:
[Equation]
(cid:35) .
A.2. O FF - POLICY VERSION
In Appendix A.2, we derive the on-policy policy gradients for updating stochastic multi-agent policies.
Similar to policy evaluation, using off-policy data can improve the sample efﬁciency with regard to policy improvement.
Using the linearly decomposed critic architecture, the off-policy policy gradients for learning stochastic policies are:
[Equation]
Proof.
The objective function is:
[Equation]
Similar to Degris et al.
[Equation]
we have:
[Equation]
B M ATHEMATICAL DETAILS FOR DETERMINISTIC DOP B. D ETERMINISTIC DOP POLICY GRADIENT THEOREM
In Sec.
4.2.1, we give the following deterministic DOP policy gradients:
[Equation]
Now we present the derivation of this update rule.
Proof.
Drawing inspirations from single-agent cases (Silver et al., 2014), we have:
[Equation]
C T HEORETICAL JUSTIFICATION FOR STOCHASTIC DOP POLICY IMPROVEMENT
In order to understand how DOP works despite the biased Q π ( τ , a ) estimation, we provide some theoretical justiﬁcation for the policy update.
Unfortunately, a thorough analysis on deep neural network and TD-learning is too complex to be carried out.
Thus, we make some assumptions for the mathematical proof.
The following two subsections provide two different view points of theoretical understanding.
In the ﬁrst view (Sec. C.1), we assume some mild assumptions on value evaluation which holds on a wide range of function class.
In this way, we can prove a policy improvement theorem similar to Degris et al.
In the second view, we remove the M ONOTONE condition from a practical point of view.
We then prove that when the loss of value evaluation is minimized (individual critics output Q φ i i ( τ , a i ) to be a good estimate of Q π i ( τ , a i ) ), the DOP gradients in Eq.
12 equal to Eq.which is the standard gradient form.
C. P ROOF OF STOCHASTIC DOP POLICY IMPROVEMENT THEOREM
Inspired by previous work (Degris et al., 2012), we relax the requirement that Q φtot is a good estimate of Q π tot and show that stochastic DOP still guarantees policy improvement.
First, we deﬁne
[Equation]
Directly analyzing the minimization of TD-error is challenging.
To make it tractable, some works (Feinberg et al., 2018) simplify this analysis to an MSE problem.
For the analysis of stochastic
DOP, we adopt the same technique and formalize the critic’s learning as the following problem:
[Equation]
(17) where Q π tot ( τ , a ) are the true values, which are ﬁxed during optimization.
In the following analysis, we assume distinct parameters for different τ .
We ﬁrst show that Fact 1 holds for a wide range of function class of Q φ i i .
To this end, we ﬁrst prove the following lemma.
Lemma 1.
Without loss of generality, we consider the following optimization problem:
[Equation]
Here, f ( Q φ ( τ , a )) : R n → R , and Q φ ( τ , a ) is a vector whose i th entry is Q φ
i i ( τ , a i ) .
In DOP, f satisﬁes that ∂f
[Equation]
for any i, a i .
If ∇ φ i Q φ i i ( τ , a i ) (cid:54) = 0 , ∀ φ i , a i it holds that:
[Equation]
Proof.
When the optimization converges, φ i reaches a stationary point where ∇ φ i L τ ( φ ) = 0 , ∀ i .
π i ( a i | τ i ) (cid:88)
a
[Equation]
Since ∇ φ i Q φ i i ( τ , a i ) (cid:54) = 0 , this implies that ∀
i, a i , we have (cid:88)
[Equation]
⇒ (cid:88) a -
[Equation]
π
[Equation]
We consider the function q ( τ , a i ) =
(cid:80) a - i π - i ( a - i | τ − i )
f
( Q φ ( τ , a ))) , which is a function of Q φ .
Its partial derivative w.r.t Q φ i i ( τ , a i ) is:
[Equation]
i
[Equation]
Therefore, if Q π i ( τ , a i ) ≥ Q π i ( τ , a (cid:48) i ) , then any local minimal of L τ ( φ ) satisﬁes Q φ i i ( τ , a i ) ≥
Q φ i i ( τ , a (cid:48) i ) .
We argue that ∇ φ i Q φ
i i ( τ , a i ) (cid:54)
= 0 is a rather mild assumption and holds for a large range of function class of Q φ
i i .
Fact 3.
[Formal Statement of Fact 1]
For the following choices of Q φ i i :
Tabular expression of Q φ i i which requires Q ( n |
A || τ | ) space; 2.
Linear function class where a i are one-hot coded:
[Equation]
3. 2-layer neural networks ( φ i (cid:54) = 0 ) with strictly monotonic increasing activation functions (e.g. tanh, leaky-relu).
Arbitrary k -layer neural networks whose activation function at the ( k − 1) th layer is sigmoid.
when value evaluation converges, ∀ π , Q φ i i satisﬁes that
[Equation]
Proof.
We need to prove that ∇ φ i Q
φ i i ( τ , a i ) (cid:54) = 0 .
For brevity, we use a ki to denote the k th element of the one-hot coding, and use φ ta ki i to denote the weight connecting the t th element of the upper layer and the a ki element.
(1 & 2) For tabular expression and linear functions, ∀ a i
=
k
we have
[Equation]
The 2-layer neural network can be written as Q φ
i
i
( τ , a i ) =
W 2 σ ( W 1 ( τ , a i )) .
Besides, we denote the hidden layer as h .
Since φ i (cid:54) = 0 , we consider some nonzero element φ W 2 1 t,i .
For the k th action, the gradient of the parameter φ W 1 tk,i is
[Equation]
(4) Without loss of generality, we consider the last layer φ W k 1 t,i :
[Equation]
These are the cases where ∇ φ i Q
φ i i (cid:54) = 0 .
Even when ∃ φ i , ∇ φ i Q φ i i = 0 , such φ i usually occupy only a small parameter space and happen with a small probability.
As a result, we conclude that ∇ φ i Q
φ i i ( τ , a i ) (cid:54) = 0 is a rather mild assumption.
Based on Fact 1, we are able to prove the policy improvement theorem for stochastic DOP.
We will show that even without an accurate estimate of Q π tot , the stochastic DOP policy updates can still improve the objective function J ( π ) =
E π [ (cid:80) t γ t r t ] .
We ﬁrst prove the following lemma.
Lemma 2.
For two sequences { a i } , { b i } , i ∈ [ n ] listed in an increasing order.
If (cid:80) i b i = 0 , then (cid:80) i a i b i ≥ 0 .
Proof.
We denote ¯ a = 1 n (cid:80) i a i , then (cid:80) i a i b i = ¯ a ( (cid:80) i b i ) + (cid:80) i ˜ a i b i where (cid:80) i ˜ a i = 0 .
Without loss of generality, we assume that ¯ a i = 0 , ∀ i .
j and k which a j ≤ 0 , a j +1 ≥ 0 and b k ≤ 0 , b k +1 ≥ 0 .
Since a, b are symmetric, we assume j ≤ k .
Then we have (cid:88) i ∈
[ n ] a i b i = (cid:88) i ∈ [1 ,j ] a
i b i + (cid:88) i ∈
[ j +1 ,k ] a i b i + (cid:88) i ∈
[ k +1 ,n ] a i b i ≥ (cid:88) i ∈
[ j +1 ,k ] a i b i + (cid:88) i ∈
[ k +1 ,n ] a i b i ≥ a k (cid:88) i ∈
[ i +1 ,k ] b i + a k + (cid:88) i ∈
[ k +1 ,n ] b i
As (cid:80) i ∈ [ j +1 ,n ] b i ≥ 0 , we have − (cid:80) i ∈
[ j +1 ,k ] b i ≤ (cid:80) i ∈
[ k +1 ,n ] b i .
Thus, (cid:80) i ∈ [ n ] a i b i ≥ ( a k +1 − a k ) (cid:80) i ∈
[ k +1 ,n ] b i ≥ 0 .
Based on Fact 1 and Lemma 2, we prove the following proposition.
Proposition 2.
[Stochastic DOP policy improvement theorem] Under mild assumptions, for any pre-update policy π o which is updated by Eq.
10 to π , denote π i ( a i | τ i )
=
π oi ( a i | τ i ) + β a i , τ δ , where δ > 0 is a sufﬁciently small number.
If it holds that ∀ τ, a (cid:48)
i , a i , Q φ i
[Equation]
τ
[Equation]
Q
[Equation]
τ ,
[Equation]
⇐⇒ β
a i , τ ≥ β a (cid:48)
i , τ ( M ONOTONE condition, and φ i is the parameters before update.), then we have
J ( π
[Equation]
≥ J
[Equation]
π
[Equation]
(19) i.e., the joint policy is improved by the update.
Proof.
Under Fact 1 , it follows that
[Equation]
Since J ( π ) = (cid:80) τ 0 p ( τ 0 )
V π tot ( τ 0 ) , it sufﬁces to prove that ∀ τ t , V π tot ( τ t ) ≥
V π o tot ( τ t ) .
We have:
[Equation]
Since δ is sufﬁciently small, in the following analysis we omit o ( δ ) .
Observing that (cid:80) a i π i ( a i | τ i ) = 1 , ∀ i , we get (cid:80) a i β a i , τ = 0 .
Thus, by Lemma 2 and Eq. 21, we have
[Equation]
Similar to the policy improvement theorem for tabular MDPs (Sutton & Barto, 2018) , we have
V
[Equation] [Equation]

[Equation]
γ
[Equation]
τ t + p ( τ t +1 | τ t , a t )  (cid:88) a t + π ( a t +1 | τ
t +1 )
Q π o tot ( τ t +1 , a t +1 ) 
  
[Equation]
≤ V
[Equation] [Equation]
This implies J ( π ) ≥ J ( π o ) for each update.
Moreover, we verify that ∀ τ, a (cid:48)
i , a i , Q φ i i ( τ , a i ) >
Q φ i i ( τ , a (cid:48) i )
⇐⇒ β
a i , τ ≥ β a (cid:48)
i , τ (the M ONOTONE condition) holds for any π with a tabular expression.
For these π , let π i ( a i | τ i )
=
θ a i , τ , then it holds that (cid:80) a i
θ a
i , τ = 1 .
Since the gradient of policy update can be written as:
[Equation]
where d π ( τ ) is the occupancy measure w.r.t our algorithm.
With a tabular expression, the update of each θ a i , τ is proportion to β a i , τ β a i , τ
[Equation]
Clearly, β a (cid:48) i , τ ≥ β a i , τ ⇐⇒ Q
φ i i ( τ , a (cid:48) i ) ≥ Q φ i i ( τ , a i ) .
C. A NALYSIS WITHOUT M ONOTONE CONDITION
For practical implementation of policy π i ( a i | τ i ) , the M ONOTONE condition is too strong to be satisﬁed for all π i .
Analyzing the policy update when the condition is violated is difﬁcult with only
Fact 1 at hand.
Therefore, it is beneﬁcial to understand policy improvement without the M ONOTONE condition.
To bypass the M ONOTONE condition, we require a stronger property of the learnt Q φ
i i ( τ , a i ) in addition to order preserving (Fact 1).
Theorem 1 in Wang et al.
(2020a) offers a closed form solution of additive decomposition and we restate it as the following lemma
Lemma 3 (Restatement of Theorem 1 in Wang et al.
(2020a)) .
If we consider the solution of
[Equation]
∀ i ∈ [ n ] , ∀ τ , a the individual action-value function Q i ( τ , a i ) =
[Equation]
The residual term w is an arbitrary vector satisfying ∀ s , (cid:80) ni
=
w i ( s ) = 0 .
Based on this lemma, we can derive another proposition to theoretically justify the DOP architecture.
Proposition 1.
Suppose the function class expressed by Q φ i i ( τ , a i ) is sufﬁciently large (e.g. neural networks) and the following loss L ( φ ) is minimized
[Equation]
where Q φtot ( τ , a ) ≡ (cid:80)
i
[Equation]
τ
[Equation]
τ , a i ) + b ( τ ) .
Then, we have
[Equation]
which means stochastic DOP policy gradients are the same as those calculated using centralized critics (Eq. 2).
Therefore, policy improvement is guaranteed.
Proof.
For brevity
[Equation]
we denote Qk
[Equation]
i
[Equation]
τ
[Equation]
Q
[Equation]
( τ , a i ) .
Then L ( φ ) can be written as
[Equation]
According to Lemma
[Equation]
when L ( φ ) is minimized
[Equation]
we have
[Equation]
Then
[Equation]
Therefore, in expectation, stochastic DOP gradients are the same as those calculated using centralized critics (Eq. 2).
We no longer require the M ONOTONE condition to guarantee improvement of the policy update.
Proposition 1 is another point of view to explain the performance guarantee of
DOP despite its constrained critics.
D R EPRESENTATIONAL CAPABILITY OF DETERMINISTIC DOP CRITICS
In Sec.
4.2.2, we present the following facts about deterministic DOP:
Fact 2.
Assume that ∀ τ , a , a (cid:48) ∈ O δ
[Equation]
τ ) , (cid:107) ∇ a Q µ tot ( τ , a ) − ∇ a (cid:48) Q
[Equation]
tot
[Equation]
τ ,
[Equation]
2 ≤ L (cid:107) a −
[Equation]
(cid:107) 2 .
The estimation error of a DOP critic can be bounded by O ( Lδ 2 ) for a ∈ O δ ( τ ) , ∀ τ .
We consider the Taylor expansion with Lagrange remainder of Q µ tot ( τ , a ) .
Namely,
Q µ
[Equation]
Since ∀ a ∈ O δ ( π ( τ )) , we have
[Equation]
Noticing that the ﬁrst order Taylor expansion of Q µ tot has the form (cid:80) [ n ] k i ( τ ) Q φi ( τ , a i ) + b ( τ ) .
Therefore, the optimal solution of the MSE problem in Eq.
17 under DOP critics has an error term less than O ( Lδ 2 ) for arbitrary sampling distribution p ( τ , a ) of a ∈ O δ ( µ ( τ )) .
When Q values in the proximity of µ ( τ ) , ∀ τ is well estimated within a bounded error and δ (cid:28) 1 , approximately, we have
[Equation]
tot
[Equation]
E A LGORITHMS
In this section, we describe the details of our algorithms, as shown in Algorithm 1 and 2.
Initialize a critic network Q φ , actor networks π θ i , and a mixer network M ψ with random parameters φ , θ i , ψ .
Initialize target networks: φ (cid:48) =
φ , θ (cid:48)
= θ , ψ (cid:48)
=
ψ
Initialize an off-policy replay buffer D off and an on-policy replay buffer D on .
for t = 1 to T do
Generate a trajectory and store it in D off and D on
Sample a batch consisting of N 1 trajectories from D on
Update decentralized policies using the gradients described in Eq.
Calculate L On ( φ )
Sample a batch consisting of N 2 trajectories from D off
Calculate L DOP-TB ( φ )
Update critics using L On ( φ ) and L DOP-TB ( φ ) if t mod d = 0 then
Update target networks: φ (cid:48)
=
φ , θ (cid:48)
= θ , ψ (cid:48)
=
ψ end if end for
Initialize a critic network Q φ , actor networks µ θ i and a mixer network M ψ with random parameters θ, φ, ψ
Initialize target networks: φ (cid:48) =
φ , θ (cid:48)
= θ , ψ (cid:48)
=
ψ
Initialize replay buffer D for t = 1 to T do
Select action with exploration noise a ∼ µ ( τ )+ (cid:15) , generate a transition and store the transition tuple in D
Sample N transitions from D
Update the critic using the loss function described in Eq.
Update decentralized policies using the gradients described in Eq.if t mod d = 0 then
Update target networks: φ (cid:48)
=
αφ +(1 − α ) φ (cid:48) , θ (cid:48)
=
αθ +(1 − α ) θ (cid:48) , ψ (cid:48)
=
αψ +(1 − α ) ψ (cid:48) end if end for
F DOP WITH C OMMUNICATION
Although DOP can solve many coordination problems, as shown by the comparison against IQL in Fig.6, its complete decomposition critic raises the concern that DOP can not deal with the miscoordination problem induced by highly uncertain and partial observable environments.
We use an example to illustrate the causes of miscoordination problems and argue that introducing communication into DOP can help address these problems.
In hallway (Fig. 7(a)), two agents randomly start at states a 1 to a m and b 1 to b n , respectively.
Agents can observe their position and choose to move left , move right , or keep still at each timestep.
Agents win and are rewarded 10 if they arrive at state g simultaneously.
Otherwise, if any agent arrives at g earlier than the other, the team gets no reward, and the next episode begins.
The horizon is set to max( m, n ) + 10 to avoid an inﬁnite loop.
Without communication, one agent cannot know the position of its teammates, so it is difﬁcult to coordinate actions.
This explains why on hallway with m = n = 4 , the team can win only 25% of the games (Fig. 7(b)).
Equipping DOP with communication can largely solve the problem – agents learn to communicate their positions and move left at a 1 or b 1 simultaneously.
For communication, we
[Figure] [Figure]
use the technique introduced by Wang et al.
(2020e).
Agents share a communication module, and messages are passed both between actors and individual Q-functions.
Such miscoordination problems are common in complex multi-agent tasks (Wang et al., 2020e).
We believe introducing communication into DOP can help it solve a larger range of problems.
G B ASELINE BY S AMPLING
One problem of existing MAPG methods is the CDM issue, which describes the large variance in policy gradients caused by the inﬂuence of other agents’ actions introduced through the joint critic.
Another technique that is frequently used to reduce the variance in policy gradients in the single-agent
RL literature is by using baselines (Sutton & Barto, 2018).
In this section, we investigate whether using baselines can effectively reduce variance in multi-agent settings.
We start from centralized critics.
COMA uses a baseline where local actions are marginalized.
Since the variance and performance of COMA have been discussed in Sec. 5, we omit it here and study the baseline where actions of all agents are marginalized.
In multi-agent settings, the calculation of this baseline requires computing an expectation over the joint action space, which is generally intractable.
To solve this problem, we estimate the expectation by sampling.
We compare stochastic DOP, COMA, and On-Policy DOP against this method, which we call Regular
Critics with Baseline .
Results are shown in Fig.
We can see that Regular Critics with Baseline performs better than COMA.
However, Regular Critics with Baseline performs worse than On-Policy
DOP .
These results indicate that a linearly decomposed critic can reduce variance in policy gradients more efﬁciently.
H I NFRASTRUCTURE , ARCHITECTURE , AND HYPERPARAMETERS
Experiments are carried out on NVIDIA P100 GPUs and with ﬁxed hyper-parameter settings, which are described in the following sections.
[Figure]
T e
[Figure]
t
[Figure]
i n
[Figure] [Figure]
[Figure]
T e s t W
i n % 2s3z
[Figure] [Figure]
[Figure]
T e s t W i n %
MMM
[Figure]
T (
[Figure] [Figure]
t
[Figure]
i n %
[Figure]
H. S TOCHASTIC DOP
In stochastic DOP, each agent has a neural network to approximate its local utility.
The local utility network consists of two 256-dimensional fully-connected layers with ReLU activation.
Since the critic is not used when execution, we condition local Q networks on the global state s .
The output of the local utility networks is Q φ
i i ( τ , · ) for each possible local action, which are then linearly combined to get an estimate of the global Q value.
The weights and bias of the linear combination, k i and b , are generated by linear networks conditioned on the global state s .
k i is enforced to be non-negative by applying absolute activation at the last layer.
We then divide k i by (cid:80) i
k i to scale k i to [0 , 1] .
The local policy network consists of three layers, a fully-connected layer, followed by a 64 bit GRU, and followed by another fully-connected layer that outputs a probability distribution over local actions.
We use ReLU activation after the ﬁrst fully-connected layer.
For all experiments, we set κ = 0 .
5 and use an off-policy replay buffer storing the latestepisodes and an on-policy buffer with a size of 32 .
We run 4 parallel environments to collect data.
The optimization of both the critic and actors is conducted using RMSprop with a learning rate of 5 × 10 − 4 , α of 0.99, and with no momentum or weight decay.
For exploration, we use (cid:15) -greedy with (cid:15) annealed linearly from 1 . 0 to 0 .
05 over 500 k time steps and kept constant for the rest of the training.
Mixed batches consisting of 32 episodes sampled from the off-policy replay buffer andepisodes sampled from the on-policy buffer are used to train the critic.
For training actors, we sample 16 episodes from the on-policy buffer each time.
The framework is trained on fully unrolled episodes.
The learning rates for the critic and actors are set to 1 × 10 − 4 and 5 × 10 − 4 , respectively.
And we use 5 -step decomposed multi-agent tree backup.
All experiments on StarCraft II use the default reward and observation settings of the SMAC benchmark.
H. D ETERMINISTIC DOP
The critic network structure of deterministic DOP is similar to that of stochastic DOP, except that local actions are part of the input in deterministic DOP.
For actors, we use a fully-connected forward network with two 64-dimensional hidden layers with ReLU activation, and the output of actors is a local action.
We use an off-policy replay buffer storing the latest 10000 transitions, from which 1250 transitions are sampled each time to train the critic and actors.
The learning rates of both the critic and actors are set to 5 × 10 − 3 .
To reduce variance in the updates of actors, we update the actors and target networks only after 2 updates to the critic, as proposed by Fujimoto et al.
We also use this technique of delaying policy update in all the baselines.
For all the algorithms, we run a single environment to collect data, because we empirically ﬁnd it more sample efﬁcient than parallel environments in the MPE benchmark.
RMSprop with a learning rate of 5 × 10 − 4 , α of 0.99, and with no momentum or weight decay is used to optimize the critic and actors, which is the same as in stochastic DOP.
I R ELATED WORKS
Cooperative multi-agent reinforcement learning provides a scalable approach to learning collaborative strategies for many challenging tasks (Vinyals et al., 2019; Berner et al., 2019; Samvelyan et al., 2019; Jaderberg et al., 2019) and a computational framework to study many problems, including the emergence of tool usage (Baker et al., 2020), communication (Foerster et al., 2016; Sukhbaatar et al., 2016; Lazaridou et al., 2017; Das et al., 2019), social inﬂuence (Jaques et al., 2019), and inequity aversion (Hughes et al., 2018).
Recent work on role-based learning (Wang et al., 2020c; 2021) introduces the concept of division of labor into multi-agent learning and grounds MARL into more realistic applications.
Centralized learning of joint actions can handle coordination problems and avoid non-stationarity.
However, the major concern of centralized training is scalability, as the joint action space grows exponentially with the number of agents.
The coordination graph (Guestrin et al., 2002b;a) is a promising approach to achieve scalable centralized learning, which exploits coordination independencies between agents and decomposes a global reward function into a sum of local terms.
Zhang & Lesser (2011) employ the distributed constraint optimization technique to coordinate distributed learning of joint action-value functions.
Sparse cooperative Q-learning (Kok & Vlassis, 2006) learns to coordinate the actions of a group of cooperative agents only in the states where such coordination is necessary.
These methods require the dependencies between agents to be pre-supplied.
To avoid this assumption, value function decomposition methods directly learn centralized but factorized global Q-functions.
They implicitly represent the coordination dependencies among agents by the decomposable structure (Sunehag et al., 2018;
Rashid et al., 2018; Son et al., 2019; Wang et al., 2020e).
The stability of multi-agent off-policy learning is a long-standing problem.
Foerster et al.
(2017); Wang et al.
(2020a) study this problem in value-based methods.
In this paper, we focus on how to achieve efﬁcient off-policy policy-based learning.
Our work is complementary to previous work based on multi-agent policy gradients, such as those regarding multi-agent multi-task learning (Teh et al., 2017; Omidshaﬁei et al., 2017) and multi-agent exploration (Wang et al., 2020d).
Multi-agent policy gradient algorithms enjoy stable convergence properties compared to value-based methods (Gupta et al., 2017; Wang et al., 2020a) and can extend MARL to continuous control problems.
COMA (Foerster et al., 2018) and MADDPG (Lowe et al., 2017) propose the paradigm of centralized critic with decentralized actors to deal with the non-stationarity issue while maintaining decentralized execution.
PR2 (Wen et al., 2019) and MAAC (Iqbal & Sha, 2019) extend the CCDA paradigm by introducing the mechanism of recursive reasoning and attention, respectively.
Another line of research focuses on fully decentralized actor-critic learning (Macua et al., 2017; Zhang et al., 2018;
Yang et al., 2018; Cassano et al., 2018; Suttle et al., 2019; Zhang & Zavlanos, 2019).
Different from the setting of this paper, agents have local reward functions and full observation of the true state in these works.