D IVERGENCE -R EGULARIZED M ULTI -A GENT A CTORC RITIC
Anonymous authors
Paper under double-blind review
A BSTRACT
Entropy regularization is a popular method in reinforcement learning (RL).
Although it has many advantages, it alters the RL objective and makes the converged policy deviate from the optimal policy of the original Markov Decision Process.
Though divergence regularization has been proposed to settle this problem, it cannot be trivially applied to cooperative multi-agent reinforcement learning (MARL).
In this paper, we investigate divergence regularization in cooperative MARL and propose a novel off-policy cooperative MARL framework, divergence-regularized multi-agent actor-critic (DMAC).
Mathematically, we derive the update rule of
DMAC which is naturally off-policy, guarantees a monotonic policy improvement and is not biased by the regularization.
DMAC is a ﬂexible framework and can be combined with many existing MARL algorithms.
We evaluate DMAC in a didactic stochastic game and StarCraft Multi-Agent Challenge and empirically show that
DMAC substantially improves the performance of existing MARL algorithms.
I NTRODUCTION
Regularization is a common method for single-agent reinforcement learning (RL).
The optimal policy learned by traditional RL algorithm is always deterministic (Sutton and Barto, 2018).
This property may result in the inﬂexibility of the policy facing with unknown environments (Yang et al., 2019).
Entropy regularization is proposed to settle this problem by learning a policy according to the maximum-entropy principle (Haarnoja et al., 2017).
Moreover, entropy regularization is beneﬁcial to exploration and robustness for RL algorithms (Haarnoja et al., 2018).
However, entropy regularization is imperfect.
Eysenbach and Levine (2019) pointed out maximum-entropy RL is a modiﬁcation of the original RL objective because of the entropy regularizer.
Maximum-entropy RL is actually learning an optimal policy for the entropy-regularized Markov Decision Process (MDP) rather than the original
MDP, i.e. , the converged policy may be biased.
Nachum et al.
(2017) analysed a more general case for regularization in RL and proposed what we call divergence regularization .
Divergence regularization can avoid the bias of the converged policy as well as be beneﬁcial to exploration.
Wang et al.
(2019) employed divergence regularizer and proposed a single-agent RL algorithm, DAPO, which prevents the altering-objective drawback of entropy regularization.
Regularization can also be applied to cooperative multi-agent reinforcement learning (MARL) (Agarwal et al., 2020; Zhang et al., 2021).
However, most of cooperative MARL algorithms do not use regularizer (Lowe et al., 2017;
Foerster et al., 2018; Rashid et al., 2018; Son et al., 2019; Jiang et al., 2020; Wang et al., 2021a).
Only few cooperative MARL algorithms such as FOP (Zhang et al., 2021) use entropy regularization, which may suffer from the drawback aforementioned.
Divergence regularization, on the other hand, could potentially beneﬁt cooperative MARL.
In addition to its advantages mentioned above, divergence regularization can also help to control the step size of policy update which is similar to conservative policy iteration (Kakade and Langford, 2002) in single-agent
RL.
Conservative policy iteration and its successive methods such as TRPO (Schulman et al., 2015) and PPO (Schulman et al., 2017) can stabilize policy improvement (Touati et al., 2020).
These methods use a surrogate objective for policy update, but decentralized policies in centralized training with decentralized execution (CTDE) paradigm may not preserve the properties of the surrogate objective.
Moreover, DAPO (Wang et al., 2019) cannot be trivially extended to cooperative MARL settings.
Even with some tricks like V-trace (Espeholt et al., 2018) for off-policy correction, DAPO is essentially an on-policy algorithm and thus may not be sample-efﬁcient in cooperative MARL settings.
In the paper, we propose and analyze divergence policy iteration in general cooperative MARL settings and a special case combined with value decomposition.
Based on divergence policy iteration, we derive the off-policy update rule for the critic, policy, and target policy and propose divergenceregularized multi-agent actor-critic (DMAC), a novel off-policy cooperative MARL framework.
We theoretically show that DMAC guarantees a monotonic policy improvement and is not biased by the regularization.
Besides, DMAC is beneﬁcial to exploration and stable policy improvement by applying our update rule of target policy.
Moreover, DMAC is a ﬂexible framework and can be combined with many existing cooperative MARL algorithms to substantially improve their performance.
We empirically investigate DMAC in a didactic stochastic game and StarCraft Multi-Agent Challenge (Samvelyan et al., 2019).
We combine DMAC with ﬁve representative MARL methods, i.e. , COMA (Foerster et al., 2018) for on-policy multi-agent policy gradient, MAAC (Iqbal and Sha, 2019) for off-policy multi-agent actor-critic, QMIX (Rashid et al., 2018) for value decomposition, DOP (Wang et al., 2021b) for the combination of value decomposition and policy gradient, and FOP (Zhang et al., 2021) for the combination of value decomposition and entropy regularization.
Experimental results show that DMAC indeed induces better performance, faster convergence, and better stability in most tasks, which veriﬁes the beneﬁts of DMAC and demonstrates the advantages of divergence regularization over entropy regularization in cooperative MARL.
R ELATED W ORK
MARL.
MARL has been a hot topic in the ﬁeld of RL.
In this paper, we focus on cooperative MARL.
Cooperative MARL is usually modeled as Dec-POMDP (Oliehoek et al., 2016), where all agents share a reward and aim to maximize the long-term return.
Centralized training with decentralized execution (CTDE) (Lowe et al., 2017) paradigm is widely used in cooperative MARL.
CTDE usually utilizes a centralized value function to address the non-stationarity for multi-agent settings and decentralized policies for scalability.
Many MARL algorithms adopt CTDE paradigm such as COMA, MAAC,
QMIX, DOP and FOP.
COMA (Foerster et al., 2018) employs the counterfactual baseline which can reduce the variance as well as settle the credit assignment problem.
MAAC (Iqbal and Sha, 2019) uses self-attention mechanism to integrate local observation and action of each agent and provides the structured information for the centralized critic.
Value decomposition (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019; Yang et al., 2020; Wang et al., 2021a;b; Zhang et al., 2021) is a popular class of cooperative MARL algorithms.
These methods express the global Q-function as a function of individual Q-functions to satisfy Individual-Global-Max (IGM), which means the optimal actions of individual Q-functions are corresponding to the optimal joint action of global Q-function.
QMIX (Rashid et al., 2018) is a representative of value decomposition methods.
It uses a hypernet to ensure the monotonicity of the global Q-function in terms of individual Q-functions, which is a sufﬁcient condition of IGM.
DOP (Wang et al., 2021b) is a method that combines value decomposition with policy gradient.
DOP uses a linear value decomposition which is another sufﬁcient condition of IGM and the linear value decomposition helps the compute of policy gradient.
FOP (Zhang et al., 2021) is a method that combines value decomposition with entropy regularization and uses a more general condition, Individual-Global-Optimal, to replace IGM.
In this paper, we will combine DMAC with these algorithms and show its improvement.
Regularization.
Entropy regularization was ﬁrst proposed in single-agent RL.
Nachum et al.
(2017) analyzed the entropy-regularized MDP and revealed the properties about the optimal policy and the corresponding Q-function and V-function.
They also showed the equivalence of value-based methods and policy-based methods in entropy-regularized MDP.
Haarnoja et al.
(2018) pointed out maximum-entropy RL can achieve better exploration and stability facing with the model and estimation error.
Although entropy regularization has many advantages, Eysenbach and Levine (2019) showed entropy regularization modiﬁes the MDP and results in the bias of the convergent policy.
Yang et al.
(2019) revealed the drawbacks of the convergent policy of general RL and maximum-entropy
RL.
The former is usually a deterministic policy (Sutton and Barto, 2018) which is not ﬂexible enough for unknown situations, while the latter is a policy with non-zero probability for all actions which may be dangerous in some scenarios.
Neu et al.
(2017) analyzed the entropy regularization method from several views.
They revealed a more general form of regularization which is actually divergence regularization and showed entropy regularization is just a special case of divergence regularization.
Wang et al.
(2019) absorbed previous result and proposed an on-policy algorithm, i.e. ,
DAPO.
However, DAPO cannot be trivially applied to MARL.
Moreover, its on-policy learning is not sample-efﬁcient for MARL settings and its off-policy correction trick V-trace (Espeholt et al., 2018) is also intractable in MARL.
There are some previous studies in single-agent RL which use similar target policy to ours, but their purposes are quite different.
Trust-PCL (Nachum et al., 2018) introduces a target policy as a trust region constraint for maximum-entropy RL, but the policy is still biased by entropy regularizer.
MIRL (Grau-Moya et al., 2019) uses a distribution which is only related to actions as the target policy to compute a mutual-information regularizer, but it still changes the objective of the original RL.
P RELIMINARIES
Dec-POMDP is a general model for cooperative MARL.
A Dec-POMDP is a tuple M = { S, A, P, Y, O, I, n, r, γ } .
S is the state space, n is the number of agents, γ is the discount factor, and I = { 1 , 2 · · · n } is the set of all agents.
A =
A 1 × A 2 × · · · × A n represents the joint action space where A i is the individual action space for agent i .
P ( s (cid:48) | s, a ) : S ×
A × S → [0 , 1] is the transition function, and r ( s, a ) :
S ×
A → R is the reward function of state s and joint action a .
Y is the observation space, and O ( s, i ) :
S × I
→ Y is a mapping from state to observation for each agent.
The objective of Dec-POMDP is to maximize J ( π ) =
E π [ (cid:80) t =
γ t r ( s t , a t )] , and thus we need to ﬁnd the optimal joint policy π ∗ = arg max π J ( π ) .
To settle the partial observable problem, history τ i ∈ T i =
( Y × A i ) ∗ is often used to replace observation o i ∈ Y .
As for policies in CTDE, each agent i has an individual policy π i ( a i | τ i ) and the joint policy π is the product of each π i .
Though we calculate individual policy as π i ( a i | τ i ) in practice, we will use π i ( a i | s ) in analysis and proofs for simplicity.
Entropy regularization adds the logarithm of current policy to the reward function.
It modiﬁes the optimization objective as J ent ( π ) =
E π [ (cid:80) t =
γ t ( r ( s t , a t ) − λ log π ( a t | s t ))] .
We also have the corresponding Q-function Q π ent ( s, a ) =
r ( s, a ) + γ E
[ V π ent ( s (cid:48) )] and V-function
V π ent ( s ) =
E [ Q π ent ( s, a ) − λ log π ( a | s )] .
Given these deﬁnitions, we can deduce an interesting property V π ent ( s ) =
E [ Q π ent ( s, a )] + λ H ( π ( ·| s )) , where H ( π ( ·| s )) represents the entropy of policy π ( ·| s ) .
V π ent ( s ) includes an entropy term which is the reason it is called entropy regularization .
M ETHOD
In this section, we ﬁrst give the deﬁnition of divergence regularization.
Then we propose and analyze divergence policy iteration .
Finally, based on divergence policy iteration, we derive the update rules of the critic, policy, and target policy for divergence-regularized MARL.
4. D IVERGENCE R EGULARIZATION
We maintain a target policy
ρ i for each agent i , which is different from the policy π i .
Then we have a joint target policy ρ = (cid:81) [-ni =
-]n i =
ρ i .
This joint target policy ρ modiﬁes the objective function as J ρ ( π ) =
E π (cid:104)(cid:80) {+t =
+}
γ t (cid:16) r ( s t , a t ) − λ log π ( a t | s t ) ρ ( a t | s t ) (cid:17)(cid:105) .
That is, a regularizer log π ( a t | s t ) ρ ( a t | s t ) , which describes the discrepancy between policy π and target policy ρ , is added to the reward function just like entropy regularization.
Given ρ , we can deﬁne corresponding V-function and Q-function for divergence regularization as follows,
[Equation]
Further, by simple deduction, we have
[Equation]
V πρ ( s ) includes an extra term which is the KL divergence between π and ρ , and thus this regularizer is referred to as divergence regularization .
4. D IVERGENCE P OLICY I TERATION
From the perspective of policy evaluation, we can deﬁne an operator Γ πρ as
[Equation]
and have the following lemma.
Note that all the proofs are given in Appendix A.
Lemma 1 (Divergence Policy Evaluation)
For any initial Q-function Q 0 ( s, a ) : S ×
A → R , we deﬁne a sequence (cid:8) Q k (cid:9) given operator Γ πρ as Q k +
=
Γ πρ Q k .
Then, the sequence will converge to
Q πρ as k → ∞ .
After the evaluation of the policy, we need a method to improve the policy.
We have the following lemma about policy improvement.
Lemma 2 (Divergence Policy Improvement)
If we deﬁne π new satisfying
[Equation]
where u ( ·| s )
=
ρ ( ·| s ) exp ( Q π old ρ
[Equation]
and Z π old ( s ) is a normalization term, then for all actions a and all states s we have Q π new ρ
[Equation]
Lemma 1 and 2 could be seen as corollaries of the conclusion of Haarnoja et al.
Lemmaindicates that given a policy π old , if we ﬁnd a policy π new according to (4), then the policy π new is better than π old .
Lemma 2 does not make any assumption and is for general settings.
Further, the policy improvement can be established and simpliﬁed based on value decomposition.
In the following, we give an example for linear value decomposition like DOP ( i.e. , Q ( s, a ) =
(cid:80)
i k
i ( s ) Q i ( s, a i )
+ b ( s ) )
(Wang et al., 2021b).
Lemma 3 (Divergence Policy Improvement with Linear Value Decomposition)
If Q-functions satisfy Q πρ ( s, a ) =
(cid:80)
i k
i ( s ) Q π i ρ ( s, a i ) + b ( s ) and we deﬁne π i new satisfying
[Equation]
∀
[Equation]
, where u i ( ·| s ) =
ρ i ( ·| s ) exp (cid:18)
[Equation]
s
[Equation]
old ρ
[Equation]
s
[Equation]
Z πi old ( s ) and Z π i old ( s ) is a normalization term, then for all actions a and all states s we have Q π new ρ
[Equation]
Lemma 3 further tells us that if the MARL setting satisﬁes the linear value decomposition condition, then each agent can optimize its individual policy with an objective of its own individual Q-function, which immediately improves the joint policy.
By combining divergence policy evaluation and divergence policy improvement, we have the following theorem of divergence policy iteration .
Theorem 1 (Divergence Policy Iteration)
By iteratively using Divergence Policy Evaluation and
Divergence Policy Improvement, we will get a sequence (cid:8) Q k (cid:9) and this sequence will converge to the optimal Q-function Q ∗ ρ and the corresponding policy sequence will converge to the optimal policy π ∗ ρ .
Theorem 1 shows that with repeated application of divergence policy improvement and divergence policy evaluation, the policy can be monotonically improved and converge to the optimal policy.
Let π ∗ ρ , V ∗ ρ ( s ) , and Q ∗ ρ ( s, a ) denote the optimal policy, Q-function, and V-function respectively, given a target policy ρ .
We have the following proposition.
Proposition 1 If π ∗ ρ = arg max π J ρ ( π ) , and V ∗ ρ ( s ) =
V π ∗ ρ ρ ( s ) and Q ∗ ρ ( s, a ) =
Q π ∗ ρ ρ ( s, a ) are respectively the corresponding Q-function and V-function of π ∗ ρ , then they satisfy the following properties:
[Equation]
With all these results above
[Equation]
we have enough tools to obtain the practical update rule of the critic, policy and target policy of DMAC
[Equation]
4. D IVERGENCE -R EGULARIZED C RITIC
From Proposition
[Equation]
we can obtain
[Equation]
By rearranging the equation
[Equation]
we have
[Equation]
which is tenable for all actions a ∈ A .
Therefore
[Equation]
we have the following corollary
[Equation]
Corollary
[Equation]
is tenable for all actions a (cid:
) ∈
A
[Equation]
Corollary 1 gives an iterative formula for Q ∗ ρ ( s, a ) , with which we can design a loss function and update rule for learning the critic,
[Equation]
[Equation]
where φ and ˜ φ are respectively the weights of Q-function and target Q-function.
The update of
Q-function is similar to that in general MDP, except that the action for next state could be chosen arbitrarily while it must be the action that maximizes Q-function for next state in general MDP.
This property greatly enhances the ﬂexibility of learning Q-function, e.g. , we can easily extend it to TD( λ ).
4. D IVERGENCE -R EGULARIZED A CTORS
DAPO (Wang et al., 2019) analyzes the divergence-regularized MDP from the perspective of policy gradient theorem (Sutton et al., 2000) and gives an on-policy update rule for single-agent RL.
Unlike existing work, we focus on a different perspective and derive an off-policy update rule by taking into consideration the characteristics of MARL.
From Lemma 2, we can obtain an optimization target for policy improvement,
[Equation]
Then, we can deﬁne the objective of the actors,
[Equation]
where D is the replay buffer.
Suppose each individual policy π i has a corresponding parameterization θ i .
We can obtain the following policy gradient for each agent with some derivation and the detail is given in Appendix A.6,
[Equation]
We need to point out that the key to off-policy update is that Lemma 2 does not limit the state distribution.
It only requires the condition is satisﬁed for each state.
Therefore, we can maintain a replay buffer to cover different states as much as possible, which is a common practice in off-policy learning.
DAPO uses a similar formula to ours, but it obtains the formula from policy gradient theorem, which requires the state distribution of the current policy.
Further, we can add a counterfactual baseline to the gradient.
First, we have the following equation about the counterfactual baseline (Foerster et al., 2018),
[Equation]
where a − i denotes the joint action of all agents except agent i .
Next, we take the baseline as
[Equation]
Then, the gradient for each agent i can be modiﬁed as follows,
[Equation]
In addition to variance reduction and credit assignment, this counterfactual baseline eliminates the policies of other agents from the gradient.
This property makes it convenient to calculate the gradient and easy to select the target policy for each agent.
Moreover, if the linear value decomposition condition is satisﬁed, we have the following gradient formula,
[Equation]
where A π i ρ ( s, a i ) =
Q π i ρ ( s, a i ) − E ˜ a i ∼ π i (cid:2)
Q π i ρ ( s, ˜ a i ) (cid:3) .
4. T ARGET P OLICY
We have discussed the update rule of the critic and actors, and now we focus on the selection and update rule of the target policy.
With the update rules above, we can obtain divergence policy iteration given a ﬁxed target policy ρ .
Then we need to devise the update rule of ρ to prevent the bias of regularization and beneﬁt the learning procedure.
Intuitively, this regularizer log π ( a t | s t ) ρ ( a t | s t ) could help to balance exploration and exploitation.
For example, for some action a , if ρ ( a | s ) >
π ( a | s ) , then the regularizer is equivalent to adding a positive value to the reward and vice versa .
Therefore, if we choose previous policy as target policy, the regularizer will encourage agents to take actions whose probability has decreased and discourage agents to take actions whose probability has increased.
Additionally, the regularizer actually controls the discrepancy between current policy and previous policy, which could stabilize the policy improvement (Kakade and Langford, 2002; Schulman et al.,
[Equation]
To derive the update rule of target policy, we need to further analyze the regularizer theoretically.
Let µ π denote the state-action distribution given a policy π .
That is, µ π ( s, a ) =
d π ( s ) π ( a | s ) , where d π ( s ) =
(cid:80) t =
γ t Pr( s t = s | π ) is the stationary distribution of states given π .
With µ π , we can rewrite the optimization objective J ρ ( π ) as follows,
[Equation] [Equation]
where D C ( µ π (cid:107) µ ρ ) = (cid:80) s, a µ π ( s, a ) log π ( a | s )
ρ ( a | s ) is a Bregman divergence (Neu et al., 2017).
Therefore, the optimization
[Equation]
is a mirror descent process which guarantees the convergence.
With this property, similar to Neu et al.
(2017) and Wang et al.
(2019), we can use the following iterative process,
[Equation] [Equation]
This process guarantees that when the policy converges, D C ( µ π
[Equation]
µ
[Equation]
) → 0 ; i.e. , the regularizer will vanish which means the policy will converge to the optimal policy of the original MDP.
However, it is intractable to perform this update rule in practice because every iteration in (18) needs a convergent policy.
Thus, we propose an alternative approximate method.
For each agent, we update the policy
π i and the target policy ρ i as follows,
[Equation]
where β is the learning rate,
˜ θ
i is the weights of ρ i , and τ is the hyperparameter for soft update.
Here we use one gradient step (19) to replace the max operator in (18).
From Theorem 1 and previous discussion, we know that optimizing L π can maximize J ρ ( π ) , so we use ∇ θ
i
L π in the gradient step for off-policy training instead of the gradient step directly optimizing J ρ ( π ) in (16).
Moreover, as the convergence of (17) is guaranteed only if the target policy ρ is ﬁxed, we softly update the target policy as the moving average of the policy to prevent the instability caused by the large change of the target policy and hence obtain stable policy improvement.
Now we have all the update rules of DMAC.
The training of DMAC is a typical off-policy learning process, which is given in Appendix B for completeness.
E XPERIMENTS
In this section, we ﬁrst empirically study the beneﬁts of DMAC and investigate how DMAC improves the performance of existing MARL algorithms in a didactic stochastic game and ﬁve SMAC tasks.
Then, we demonstrate the advantages of divergence regularizer over entropy regularizer in cooperative
MARL.
5. I MPROVEMENTS OF E XISTING M ETHODS
DMAC is a ﬂexible framework and can be combined with many existing MARL algorithms.
In the experiments, we choose four representative algorithms for different types of methods: COMA (Foerster et al., 2018) for on-policy multi-agent policy gradients, MAAC (Iqbal and Sha, 2019) for off-policy multi-agent actor-critic, QMIX (Rashid et al., 2018) for value decomposition, DOP (Wang et al., 2021b) for the combination of value decomposition and policy gradient.
These algorithms need minor modiﬁcations to ﬁt the framework of DMAC.
We denote these modiﬁed algorithms as
COMA+DMAC, MAAC+DMAC, QMIX+DMAC, and DOP+DMAC.
Generally, our modiﬁcation is limited and tries to keep the original architecture so as to fairly demonstrate the improvement of DMAC.
The details of the modiﬁcations are included in Appendix C.1.
More details about hyperparameters are available in Appendix C. All the curves in our plots correspond to the mean value of ﬁve training runs with different random seeds, and shaded regions indicate 95% conﬁdence interval.
5.1. A D IDACTIC E XAMPLE
We ﬁrst test the four groups of methods in a stochastic game where agents share the reward.
The stochastic game is generated randomly for the reward function and transition probabilities with 30 states, 3 agents and 5 actions for each agent.
Each episode contains 30 steps in this game.
[Figure]
The performance of these methods is illustrated in Figure 1.
We could ﬁnd that DMAC performs better than the baseline at the end of the training in all the four groups.
Moreover, COMA+DMAC, QMIX+DMAC and MAAC+DMAC learn faster than their baselines.
Though DOP learns faster than DOP+DMAC at the start, it falls into a sub-optimal policy and DOP+DMAC ﬁnds a better policy in the end
[Figure]
We also show the beneﬁt of exploration in this stochastic game for the convenience of statistics.
We evaluate the exploration in terms of the cover rate of all state-action pairs,
i
[Figure]
e. , the ratio of the explored state-action pairs to all state-action pairs.
The cover rates of COMA and COMA+DMAC are illustrated in Figure 2.
We use COMA here as a representation of the traditional policy gradient method in cooperative MARL.
We could ﬁnd that the cover rate of COMA+DMAC is higher than COMA, which can be an evidence for the beneﬁt of exploration of DMAC.
The cover rates of other three groups of algorithms are available in Appendix D.
5.1. SMAC
We test all the methods in ﬁve tasks of StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019).
The introduction of the SMAC environment and the training details are included in Appendix
C. The learning curves in terms of win rate of all the methods in the ﬁve SMAC tasks are illustrated in Figure 3 (four columns for four groups of algorithms and ﬁve rows for ﬁve different maps in
SMAC).
In addition, the learning curves in terms of rewards are available in Appendix D.
We show the empirical result of DAPO in the map of 3m in the ﬁrst ﬁgure of the second column in Figure 3.
It can be seen that DAPO cannot obtain a good performance in the simple task of SMAC, so we skip it in other SMAC tasks.
The reason for the low performance of DAPO may be that DAPO omits the correction for d π ( s ) /d
π t ( s ) in policy update which introduces bias in the gradient of policy, and
[Figure] [Figure]
r
[Figure]
t e s [Figure] uses V-trace as off-policy correction which however is biased. These drawbacks may be magniﬁed in MARL settings.
The superiority of our naturally off-policy method over the biased off-policy correction method can be partly seen from the large performance gap between COMA+DMAC and
DAPO.
In all the ﬁve tasks, MAAC+DMAC outperforms MAAC signiﬁcantly, but MAAC+DMAC does not change the network architecture of MAAC, which shows the beneﬁts of divergence regularizer.
As for the result of COMA and COMA+DMAC.
We ﬁnd that COMA+DMAC has higher win rates than COMA in most cases at the end of the training, which can be attributed to the beneﬁts of off-policy training and exploration of divergence regularizer.
Though in some cases COMA learns faster than COMA+DMAC, it falls into sub-optimal in the end.
This phenomenon can be observed more clearly in the plots of episode rewards in Appendix D, especially in the hard tasks like 3s5z.
This can be an evidence for the advantage of divergence regularizer which helps the agents ﬁnd a better policy.
The stable policy improvement of divergence regularizer can be showed in the variance of the learning curves especially in the comparison between QMIX and QMIX+DMAC.
In most tasks, we ﬁnd that QMIX+DMAC learns substantially faster than QMIX and gets higher win rates in harder tasks.
The results of DOP groups are illustrated in the fourth column of Figure 3.
DOP+DMAC learns faster than DOP in most cases and ﬁnally obtains a better performance.
The difference of DOP and
DOP+DMAC can also partly show the advantage of naturally off-policy method to the off-policy correction method, as DOP+DMAC replaces the tree backup loss with off-policy TD( λ ).
DMAC improves the performance and/or convergence speed of the evaluated algorithms in most tasks.
This empirically demonstrates the beneﬁts of divergence regularizer.
Moreover, the superiority of our naturally off-policy learning over the biased off-policy correction method can be partly witnessed from the empirical results.
5. C OMPARISON WITH E NTROPY R EGULARIZATION
FOP (Zhang et al., 2021) combines the value decomposition with entropy regularization, which obtained the state-of-the-art performance in SMAC tasks.
FOP has a well tuned scheme for the temperature parameter of the entropy, so we take FOP as a strong baseline for entropy-regularized methods in cooperative MARL.
We compare FOP and FOP+DMAC in three SMAC tasks, 3s_vs_3z, 2c_vs_64zg, and MMM2, which respectively correspond to the three levels of difﬁculties ( i.e. , easy, hard, and super hard) for SMAC tasks.
These tasks are taken from the original paper of FOP.
The modiﬁcations of FOP+DMAC are also included in Appendix C.1.
The win rates of FOP and
FOP+DMAC are illustrated in Figure 4.
We could ﬁnd that FOP+DMAC learns much faster than
FOP in 3s_vs_3z, while it performs better than FOP in other two harder tasks.
These results could be an evidence for the advantages of DMAC and the bias of entropy regularization.
C ONCLUSION
We propose a multi-agent actor-critic framework, DMAC, for cooperative MARL.
We investigate divergence regularization, derive divergence policy iteration, and deduce the update rules for the critic, policy, and target policy in multi-agent settings.
DMAC is a naturally off-policy framework and the divergence regularizer is beneﬁcial to exploration and stable policy improvement.
Unlike entropy regularizer, the divergence regularizer will not bias the converged policy.
DMAC is also a ﬂexible framework and can be combined with many existing MARL algorithms with limited modiﬁcation.
It
A P ROOFS A. L EMMA
Proof .
We deﬁne a new reward function
[Equation]
then we can rewrite the deﬁnition of operator Γ πρ as
[Equation]
With this formula, we can apply the traditional convergence result of policy evaluation in Sutton and
Barto (2018).
[Equation]
A. L EMMA
For the proof of Lemma
[Equation]
we need the following lemma (Haarnoja et al., 2018) about improving policy in entropy-regularized MDP.
Lemma
If we have a new policy π new and
[Equation]
where Z π old ( s ) represents the normalization term
[Equation]
then we have
[Equation]
With Lemma
[Equation]
we have the following proof of Lemma 2.
Proof .
Let ˆ Q be the same as the deﬁnition in Proof A.5.
Then we have ˆ Q π ( s, a ) =
Q πρ ( s, a ) + λ log ρ ( a | s ) , ∀ π .
According to Lemma 4,
[Equation]
With the deﬁnition
[Equation]
we have
[Equation]
A. L EMMA
Proof .
From the equation
[Equation]
we can obtain
[Equation]
By taking expectation on the both side of (21), we can obtain the followings.
[Equation]
Moreover
[Equation]
we can easily have the following derivation,
[Equation] [Equation]
Then we have
[Equation]
The fourth equation is from (23) and the ﬁfth inequality is from (22)
[Equation]
By repeatedly applying (24) and the relation Q π old ρ
[Equation]
, we can complete the proof as followings
[Equation]
A. T HEOREM
Proof .
First, we will show that Divergence Policy Iteration will monotonically improve the policy.
From Lemma 2, we know that
[Equation]
The ﬁrst inequality is from the conclusion of Lemma 2 that
[Equation]
and the second inequality is from the deﬁnition of π new that
[Equation]
Here we have V π new ( s ) ≥ V π old ( s ) , ∀ s ∈ S , and thus J ρ ( π new ) ≥
J ρ ( π old ) .
So, Divergence
Policy Iteration will monotonically improve the policy.
Since the Q πρ is bounded above (the reward function is bounded), the sequence of Q-function (cid:8) Q k (cid:9) of Divergence Policy Iteration will converge and the corresponding policy sequence will also converge to some policy π conv .
We need to show π conv = π ∗ ρ .
[Equation]
ρ
[Equation]
(cid:21) · · · ≥ V πρ ( s ) .
The ﬁrst inequality is from the deﬁnition of π conv that
[Equation]
and all the other inequalities are just iteratively using the ﬁrst inequality and the relation of Q πρ and
V πρ .
With iterations, we replace all the π conv with π in the expression of V π conv ρ ( s ) and ﬁnally we get V πρ ( s ) .
Therefore, we have
[Equation]
A. P ROPOSITION
Before the proof of Proposition 1, we need some results about the optimal Q-function Q ∗ ent , the optimal V-function V ∗ ent , and the optimal policy π ∗ ent in entropy-regularized MDP.
We have the following lemma
Lemma
[Equation]
With Lemma
[Equation]
we can complete the proof of Proposition
[Equation]
Proof .
Let ˆ r
( s, a ) =
r ( s, a ) + λ log ρ ( a | s ) , we consider the objective function
[Equation]
Let ˆ π ∗ ( a | s ) , ˆ V ∗ ( s ) and ˆ Q ∗ ( s, a ) be the corresponding optimal policy, V-function and Q-function of ˆ J ( π ) .
By deﬁnition we can obtain
[Equation]
According to Lemma
[Equation]
we have
[Equation]
r ( s, a ) + γ E s (cid:48) ∼ P ( ·| s,a ) (cid:2) V ∗ ρ ( s (cid:48) )
(cid:3)(cid:1) /λ (cid:1)
Q ∗ ρ ( s, a ) =
ˆ Q ∗ ( s, a ) − λ log ρ ( a | s ) =
ˆ r
( s, a ) + γλ E s (cid:48) ∼ P ( ·| s,a ) (cid:34) log (cid:88) a (cid:48) exp (cid:16)
ˆ Q ∗ ( s (cid:48) , a (cid:48) )
/λ (cid:17)(cid:35) − λ log ρ ( a | s ) =
r ( s, a ) + γλ E s (cid:48) ∼ P ( ·| s,a ) (cid:34) log (cid:88) a (cid:48) ρ ( a | s ) exp (cid:0) Q ∗ ρ ( s (cid:48) , a (cid:48) ) /λ
(cid:1)(cid:35) .
(cid:3)
A. D ERIVATION OF G RADIENT
[Equation]
B A LGORITHM
Algorithm 1 gives the training procedure of DMAC.
Algorithm 1 DMAC
1: for episode = 1 to m do 2:
Initialize the environment and receive initial state s 3: for t = 1 to max-episode-length do 4:
For each agent i , select action a i ∼ π i ( ·| s ) 5:
Execute joint-action a = ( a 1 , a 2 , · · · , a n ) and observe reward r and next state s (cid:48) 6:
Store ( s, a , r, s (cid:48) ) in replay buffer D 7: end for 8:
Sample a random minibatch of K samples from D , { ( s k , a k , r k , s (cid:48) k ) } K 9: for agent i = 1 to n do 10:
Update policy π i : θ i
= θ
i + β ∇ θ i
L π 11:
Update target policy ρ i : ˜
θ i = (1 − τ )
˜ θ
i
+ τθ
i 12: end for 13:
Update critic: φ = φ − α ∇ φ L Q 14:
Update target critic: ˜ φ
= (1 − τ )˜
φ + τφ 15: end for
C I MPLEMENTATION D ETAILS
SMAC is a MARL environment based on the game StarCraft II (SC2).
Agents control different units in SC2 and can attack, move or take other actions.
The general mode of SMAC tasks is that agents control a team of units to counter another team controlled by built-in AI.
The target of agents is to wipe out all the enemy units and agents will gain rewards when hurting or killing enemy units.
Agents have an observation range and can only observe information of units in this range, but the information of all the units can be accessed in training.
We test all the methods in totally 8 tasks/maps: 3m, 2s3z, 3s5z, 8m,1c3s5z,3s_vs_3z,2c_vs_64zg, and MMM2.
In SMAC tasks, we evaluate 20 episodes in every 10000 training steps in the one million steps training procedure for COMA, MAAC, QMIX and in every 20000 time steps for DOP and FOP.
In evaluation, we select greedy actions for QMIX and FOP (following the setting in the FOP paper) and sample actions according to action distribution for stochastic policy (COMA, MAAC, DOP and divergence-regularized methods).
C. M ODIFICATIONS OF THE B ASELINE M ETHODS
The modiﬁcations of the baseline methods, COMA,MAAC,QMIX,DOP and FOP, are as follows.
• COMA.
We keep the original critic and actor networks and add a target policy network with the same architecture as the actor.
As COMA is on-policy but COMA+DMAC is off-policy, we add a replay buffer for experience replay.
• MAAC already has a target policy for stability, so we do not need to modify the network architecture.
We only change the update rule for the critic and actors.
[Figure]
• QMIX is a value-based method, so we need to add a policy network and a target policy network for each agent.
We keep the original individual Q-functions to learn the critic in QMIX+DMAC.
In divergence-regularized MDP, the max operator is not needed in the critic update, so we abandon the hypernet and use an MLP, which takes individual Q-values and state as input and produces the joint Q-value.
This architecture is simple and its expressive capability is not limited by QMIX’s IGM condition.
• DOP.
We keep the original critic and actor networks and add a target policy network with the same architecture as the actor.
We keep the value decomposition structure and use off-policy TD( λ ) for all samples in training to replace the tree backup loss and on-policy TD( λ ) loss.
• FOP.
We replace the entropy regularizers with divergence regularizers in FOP and use the update rules of DMAC.
We keep the original architecture of FOP.
C. H YPERPARAMETERS
As all tasks in our experiments are cooperative with shared reward, so we use parameter-sharing policy network and critic network for MAAC and MAAC+DMAC to accelerate training.
Besides, we add a RNN layer to the policy network and critic network in MAAC and MAAC+DMAC to settle the partial observability.
All the policy networks are the same as two linear layers and one GRUCell layer with ReLU activation and the number of hidden units is 64.
The individual Q-networks for QMIX group is the same as the policy network mentioned before.
The critic network for COMA group is a MLP with three 128-unit hidden layers and ReLU activation.
The attention dimension in the critic networks of MAAC group is 32.
The number of hidden units of mixer network in QMIX group is 32.
The learning rate for critic is 10 − 3 and the learning rate for actor is 10 − 4 .
We train all networks with RMSprop optimizer.
The discouted factor is γ = 0 . 99 .
The coefﬁcient of regularizer is λ = 0 .
The td_lambda factor used in COMA group is 0 . 8 .
The parameter used for soft updating target policy is τ = 0 .
Our
[Figure]
1.0 step 1e 5151c3s5z 0.
1.0 step 1e    1c3s5z
[Figure]
[Figure] code is based on the implementation of PyMARL (Samvelyan et al., 2019), MAAC (Iqbal and Sha, 2019), DOP (Wang et al., 2021b), FOP (Zhang et al., 2021) and an open source code for algorithms in SMAC ( https://github.com/starry-sky6688/StarCraft ).
C. E XPERIMENT S ETTINGS
We trained each algorithms for ﬁve runs with different random seeds.
In SMAC tasks, we train each algorithm for one million time steps in each run for COMA, QMIX, MAAC, and DOP groups and two million timesteps for FOP groups.
We do all the experiments by a server with 2 NVIDIA A
GPUs.
D A DDITIONAL R ESULTS
[Figure]
[Figure]
[Figure]
w i n r a t
e s
MMM QMIX [Figure] DMACQMIX 0.
[Figure]
[Figure]
[Figure]
DOP [Figure] Figure 5 shows the learning curves in terms of cover rates of COMA, QMIX, MAAC and DOP groups in the randomly generated stochastic game. Figure 6 shows the learning curves of COMA, MAAC, QMIX and DOP groups in terms of mean episode rewards in each SMAC map.
Figure 7 shows the learning curves of FOP+DMAC and FOP in terms of mean episode rewards in 3s_vs_3z, 2c_vs_64zg and MMM2. Figure 8 shows the learning curves in terms of mean episode rewards of COMA and DOP groups in the CDM environment used by the DOP paper. Figure 9 shows the learning curves of QMIX and DOP groups in terms of win rate, and COMA and MAAC groups in terms of episode reward on the map MMM2 in SMAC.