G REEDY - BASED V ALUE R EPRESENTATION FOR E FFICIENT C OORDINATION IN M ULTI - AGENT R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Due to the representation limitation of the joint Q value function, multi-agent reinforcement learning (MARL) methods with linear or monotonic value decomposition can not ensure the optimal consistency (i.e. the correspondence between the individual greedy actions and the maximal true Q value), leading to instability and poor coordination.
Existing methods focus on addressing the representation limitation through learning the complete expressiveness, which is impractical and may deteriorate the performance in complex tasks.
In this paper, we introduce the True-Global-Max (TGM) condition for linear and monotonic value decomposition to achieve the optimal consistency directly, where the TGM condition can be ensured under the unique stability of the optimal greedy action.
Therefore, we propose the greedy-based value representation (GVR) to stabilize the optimal greedy action via inferior target shaping and destabilize non-optimal greedy actions via superior experience replay.
We conduct experiments on various benchmarks, where GVR signiﬁcantly outperforms state-of-the-art baselines.
Experiment results demonstrate that our method can ensure the optimal consistency under sufﬁcient exploration and is more efﬁcient than methods with complete expressiveness capacity.
I NTRODUCTION
By taking advantage of the deep learning technique, cooperative multi-agent reinforcement learning (MARL) shows great scalability and excellent performance on challenging tasks (Vorotnikov et al., 2018; Wu et al., 2020) such as StarCraft unit micromanagement (Foerster et al., 2018).
As an efﬁcient paradigm of cooperative
MARL, centralized training with decentralized execution (CTDE) (Oliehoek et al., 2008; Foerster et al., 2016; Lowe et al., 2017) gains growing attention.
A simple and effective approach to adopt CTDE in value-based cooperative MARL is linear value decomposition (LVD) or monotonic value decomposition (MVD).
However, both LVD and MVD suffer from relative overgeneralization (Panait et al., 2006; Wei et al., 2018) due to the representation limitation of the joint Q value function.
Recent works address the problem from two different perspectives.
The ﬁrst kind of method aims to solve the representation limitation directly through value functions with complete expressiveness capacity (e.g.,
QTRAN (Son et al., 2019) and QPLEX (Wang et al., 2020)).
However, learning the complete expressiveness is impractical in complicated MARL tasks because the joint action space increases exponentially with the number of agents.
The other kind of method tries to overcome the adverse effects of relative overgeneralization by learning a bias (e.g., WQMIX (Rashid et al., 2020) and MAVEN (Mahajan et al., 2019)), which lacks of theoretical and quantitative analysis of those adverse effects and is only applicable in speciﬁc tasks.
As a result, these methods can not ensure optimal coordination.
More discussions about related works are provided in Appendix A.
Value decomposition is a popular approach to assign credit for individual agents in fully cooperative MARL tasks, where the main concern is the optimality of coordination.
In a successful case of credit assignment , individual agents act according to their local policies and achieve the best team’s performance.
To evaluate the value decomposition, we propose the optimal consistency , a criterion concerning the optimality of coordination. The optimal consistency can be decomposed into two conditions: Individual-Global-Max (IGM) and True-Global-Max (TGM) . In this paper, to achieve the optimal consistency efﬁciently, we investigate the requirements of the TGM condition and go deep into the mechanism of the value representation for LVD and MVD, where the IGM condition always holds.
We ﬁrst derive the expression of the joint Q value function of LVD and MVD, by which we draw some interesting conclusions. Firstly , LVD and MVD share the same expression of the joint Q value function. Secondly , the correspondence between the joint greedy action and the maximal Q true value (i.e., the team’s best performance) depends heavily on the task-speciﬁc reward function for LVD and MVD. Thirdly , there may be multiple stable points of the joint greedy action.
As a result, the joint policy may converge to different results. More importantly, in some of the stable points, the true Q value of the joint greedy action is not maximal, which is the root cause of non-optimal coordination and relative overgeneralization. To ensure the TGM condition for LVD and MVD, the stable point satisfying the TGM condition (we call it the optimal stable point) is required to be the unique stable point , which is the target problem to be solved in this paper. To solve the target problem, we propose the greedy-based value representation (GVR).
According to previous conclusions, the stable points are task-speciﬁc due to their dependency on the reward function, for which we propose the inferior target shaping (ITS). ITS dynamically modiﬁes the true Q value of inferior samples (i.e., the samples worse than the current greedy) according to current greedy Q value, which is theoretically proved to stabilize the optimal point under any reward function. Besides, under ITS, the stability of a non-optimal point depends only on the proportion ratio of superior samples (i.e., the samples better than the current greedy) to the non-optimal sample, where the non-optimal stable points can be eliminated under a large enough ratio. We prove two simple ways to raise the ratio applied by previous works (i.e. applying weight on the superior samples (Rashid et al., 2020) and improving exploration (Mahajan et al., 2019)) are inapplicable because the probability of superior samples decreases rapidly with the joint action space.
Therefore, we further propose the superior experience replay (SER), which achieves almost constant probability of superior samples by saving them in a superior buffer. SER is theoretically proved to eliminate the non-optimal stable points under ITS.
We have three contributions in this work.
(1) This is the ﬁrst work to derive the exact expression of the joint Q value function for LVD and MVD. (2) We point out the root cause of non-optimal coordination and further propose the target problem to be solved for LVD and MVD.
(3) We propose the GVR method, which is proved theoretically to ensure the optimal consistency under sufﬁcient exploration, and our method outperforms state-of-the-art baselines in various benchmarks.
P RELIMINARIES 2. D EC -POMDP
We model a fully cooperative multi-agent task as a decentralized partially observable Markov decision process (Dec-POMDP) described by a tuple G = < S, U, P, r, Z, O, n, γ > (Guestrin et al., 2001; Oliehoek & Amato, 2016).
s ∈ S denotes the true state of the environment.
At each time step, each agent a ∈
A ≡ { 1 , 2 , · · · , n } receives a local observation z a ∈ Z produced by the observation function O : S
×
A → Z , and then chooses an individual action u a ∈ U according to a local policy π a ( u a | τ a ) : T × U → [0 , 1] , where τ a ∈ T ≡ ( Z × U ) ∗ denotes the local action-observation history.
The joint action of n agents u results in a shared reward r ( s, u ) and a transition to the next state s (cid:48) ∼ P ( ·| s, u ) .
γ ∈ [0 , 1) is a discount factor.
We denote the joint variable of group agents with bold symbols, e.g., the joint action u ∈ U ≡ U n , the joint action-observation history τ ∈ T ≡ T n , and the joint interactive policy (i.e. the policy interacts with environment to generate trajectories) π ( u | τ ) .
The true Q value of π ( u t | τ t ) is denoted by Q π ( s t , u t ) =
E s t +1: ∞ , u t +1: ∞
[ R t | s t , u t ] , where R t = (cid:80) ∞
i =
γ i r t +1 is the discounted return.
The action-state value function of agent a and the group of agents are deﬁned as utility function U a ( u a , τ a ) and joint Q value function Q ( u , τ ) respectively.
The true Q value is the target of the joint Q value in training, serving as the unique external criterion of the team’s performance. The greedy action is deﬁned as the joint action with the maximal joint Q value u ∗ : = argmax u Q ( u , τ ) . The optimal action is deﬁned as the joint action with the best team’s performance u opt := argmax u Q ( s, u ) .
For brevity, we sometimes omit the preﬁx ”joint” for the joint variables. 2. O PTIMAL C ONSISTENCY AND TGM C ONDITION In CTDE paradigm, agents are expected to act individually according to their local policies (i.e., the individual greedy actions) while achieve the optimal coordination (i.e., the maximal true Q value).
Here we deﬁne the correspondence between the individual greedy actions and the maximal true Q value as the optimal consistency.
Deﬁnition 1 (Optimal consistency) .
Given a set of utility functions {U 1 ( u 1 , τ 1 )) , · · · , U n ( u n , τ n ) } , and the true Q value Q ( s, u ) , if the following holds
[Equation]
then we say the set of utility functions {U 1 ( u 1 , τ 1 )) , · · · , U n ( u n , τ n ) } satisﬁes the optimal consistency.
For simplicity, we ignore situations with non-unique optimal actions.
The optimal consistency can be decomposed into two conditions: Individual-Global-Max (IGM) and True-Global-Max (TGM).
The IGM condition proposed by QTRAN (Son et al., 2019) is deﬁned on the correspondence between individual greedy actions and the joint greedy actions (formally, { argmax u 1 U 1 ( u 1 , τ 1 ) , · · · , argmax u n U n ( o n , τ n ) } = argmax u Q ( u , τ ) ).
To achieve the optimal consistency, the correspondence between the joint greedy action and the maximal true Q value is required, for which we deﬁne the TGM condition:
Deﬁnition 2 (TGM) .
Given a joint value function Q ( u , τ ) , and the true Q value Q ( s, u ) , if the following holds
[Equation]
then we say the joint value function Q ( u , τ ) satisﬁes the TGM condition.
For simplicity, we ignore situations with non-unique optimal actions.
I NVESTIGATION OF THE TGM C ONDITION FOR LVD & MVD
Linear value decomposition (LVD) and monotonic value decomposition (MVD) are simple and naturally meet the IGM condition.
To achieve the optimal consistency, we investigate the requirements of the TGM condition for LVD and MVD.
According to Def.2, the TGM condition is related to the joint Q value function Q ( u , τ ) . In this section, we ﬁrst derive the expression of Q ( u , τ ) for LVD and MVD under (cid:15) − greedy visitation.
The expression indicates there may be non-optimal stable points that violate the TGM condition, which is the root cause of non-optimal coordination.
3. E XPRESSION OF THE J OINT Q V ALUE F UNCTION FOR LVD & MVD
Firstly, take two-agent linear value decomposition as an example, where the joint Q value function
Q ( u 1 i , u 2 j , τ ) is linearly factorized into two utility functions Q ( u 1 i , u 2 j , τ ) = {+U 1 ( u 1 i , τ 1 ) + U 2 ( u 2 j , τ 2 ) . u 
+}
i , u 2 j ∈ { u 1 , · · · , u m } denote the individual actions of agent 1,2 respectively, where { u 1 , · · · , u m } is the discrete individual action space. Specially, we denote the individual greedy action of agent 1,2 with u 1 i ∗ , u 2 j ∗ respectively.
For brevity, Q ( u 1 i , u 2 j , τ ) and U a ( u ai , τ a ) are represented by Q ij and U ai ( a ∈ { 1 , 2 } ) respectively.
Through the derivation provided in Appendix B.1, Q ij can be represented by the true Q values as
[Equation]
Veriﬁcation of the expression is provided in Appendix B.2. For monotonic value decomposition, the expression is identical to Eq.3 (the proof is provided in Appendix C), which indicates the coefﬁcients and bias on utility functions do not affect the joint Q value function.
For situations with more than two agents, by referring to the derivation in Appendix B.1 and C, the expression of joint Q values can also be obtained.
3. R EQUIREMENTS TO E NSURE THE TGM C ONDITION FOR LVD & MVD
According to Eq.3, the joint Q values change with the greedy action { u 1 i ∗ , u 2 j ∗ } . An example of the joint Q values under 3 different { u 1 i ∗ , u 2 j ∗ } is shown in Tab.1(b,c,d), where (cid:15)
= 0 . [Table] Notice that the example in Tab.1(c) is an unstable point , because { u 1 i ∗ , u 2 j ∗ } = { 1 , 1 } (cid:54) = argmax Q ij = { 2 , 2 } .
To explain this, assume { u 1 i ∗ , u 2 j ∗ } and Q ij of iteration t are { u 1 i ∗ , u 2 j ∗ } t and Q ij,t respectively, where { u 1 i ∗ , u 2 j ∗ } t ( { u 1 i ∗ , u 2 j ∗ } in Tab.1(c)) = argmax Q ij,t = { 1 , 1 } . After training, { u 1 i ∗ , u 2 j ∗ } t +1 = argmax Q ij,t +1 ( Q ij in Tab.1(c)) =
{ 2 , 2 } , which means { u 1 i ∗ , u 2 j ∗ } t changes with iteration . Since Q ij,t +1 is related to { u 1 i ∗ , u 2 j ∗ } t , both { u 1 i ∗ , u 2 j ∗ } t and Q ij,t +1 are unstable.
As a result, Tab.1(c) ﬁnally converges to the stable point in Tab.1(d).
Here we deﬁne the stable point of LVD and MVD. Deﬁnition 3 (Stable point of LVD and MVD) .
Given the joint Q value function Q ( u , τ ) which has converged under LVD (or MVD) with the target Q ( s, u ) , where Q ( s, u ) is the true Q value, and given an interactive policy π ( u , τ ) with the greedy action u ∗ ( u ∗ = argmax π ( s, u ) ), if the following holds
[Equation]
then we say Q ( u , τ ) and u ∗ are stable, and this is a stable point of LVD (or MVD). According to Def.3, the examples in Tab.1(b) ( argmax Q ij = { u 1 i ∗ , u 2 j ∗ } = { 0 , 0 } ) and Tab.1(d)
( argmax Q ij = { u 1 i ∗ , u 2 j ∗ } = { 2 , 2 } ) are both stable points (i.e., the joint Q values and the greedy action may converge to Tab.1(b) or Tab.1(d)).
However, the stable points in Tab.1(d) violates the TGM condition (i.e., argmax Q ij (cid:54) = argmax Q ij ), which is the root cause of non-optimal coordination.
Tab.1(b) and Tab.1(d) are deﬁned as non-optimal stable point and optimal stable point respectively.
To ensure the TGM condition, the optimal stable point is required to be the unique stable point.
More discussion about the stable points is provided in appendix H.
M ETHOD
In this section, we introduce the greedy-based value representation (GVR).
We ﬁrst propose inferior target shaping (ITS) to ensure the optimal stable point under any reward function.
Besides, under ITS, the stability of a non-optimal point depends only on the ratio of superior samples to the non-optimal sample.
To eliminate the non-optimal stable points, we then investigate two simple methods (i.e., improving exploration and applying a weight on the superior samples) which raise the ratio. However, both methods are proved inapplicable due to the extremely small probability of superior samples under large joint action space. Finally, we describe superior experience replay, which achieves an almost constant proportion of superior samples through a superior buffer.
4. I NFERIOR T ARGET S HAPING
According to Eq.3, the joint Q value of any action is related to the true Q values of the whole joint action space for LVD and MVD, which indicates the stable points depend heavily on the reward function (proofs are provided in Appendix H). Firstly, we consider to remove the dependency.
Because the exact true Q values of non-optimal samples are uninformative, we only represent the ”non-optimality” of these samples.
Here we propose the ITS target Q its ( s, u )
[Equation]
where u ∗ is the joint greedy action and α ∈ (0 , 1] .
A large enough α prevents the confusion between greedy and inferior samples.
The samples in the ﬁrst case are called inferior samples .
Besides, the action samples u which satisfy Q ( s, u ) > Q ( s, u ∗ ) are called superior samples .
ITS target only remains the information of non-optimality for inferior samples, which eases representation.
Under ITS, given the greedy action u ∗ and any action u s ( u s
[Equation]
= u ∗ ) , assuming Q ( s, u ∗ ) > 0 , we have
[Equation]
where e Q = Q ( s, u s ) −Q ( s, u ∗ ) Q ( s, u ∗ ) , η 1 = ( (cid:15)m ) n − 1 and η 2 = (1 − (cid:15) + (cid:15)m ) n − 1 .
We provide two different versions of proof for Eq. ??
in Appendix D, and the calculation result is veriﬁed in the experimental part (Fig.1).
It is proved in Appendix D.1 that there is always an optimal stable point under ITS .
When ∆ Q ( u s , τ ) > 0 , argmax u Q ( u , τ ) (cid:54) =
u ∗ , which suggests current greedy action is unstable.
If u ∗ is a non-optimal action, to destabilize it, let ∆ Q ( u s , τ ) > 0 and assume Q ( u ∗ , τ )
≈ Q ( s, u ∗ ) (this assumption is quite accurate, as veriﬁed in Appendix E.2), we have
[Equation]
which indicates the non-optimal stable points can be eliminated by raising η 1 η 2 under ITS. A simple way to raise η 1 η 2 is improving exploration . Substituting the expression of η 1 and η 2 into Eq.7, we have
[Equation] When u s = argmax u Q ( s, u ) , we obtain the lower bound of (cid:15) (denoted by (cid:15) 0 ) from the right side of Eq.8.
However, as the number of agents n and the size of individual action spaces m increases, (cid:15) 0 grows close to 1, which is inapplicable in tasks with long episodes.
Another way to raise η 1 η 2 is increasing the relative weights of the superior samples u s .
It is proved in
Appendix E that the non-optimal stable points can be eliminated by applying a weight w ( w > α ( η 2 − η 1 ) e Q η 1 ) to the superior samples under ITS.
When u s = argmax u Q ( s, u ) , we obtain the lower bound of w (denoted by w 0 ).
However, the w 0 grows rapidly as the number of agents n and the size of individual action spaces m increases, which introduces instability in training.
Since η 1 η 2 decreases rapidly as the joint action space grows, the essence to raise η 1 η 2 is increasing the proportion of superior samples and freeing the raised proportion from the dependency on the joint action space size , for which we introduce the superior experience replay.
4. S UPERIOR E XPERIENCE R EPLAY
To distinguish superior samples, we ﬁrst introduce a critic V ( s ) , which approximates the true Q value of the joint greedy sample.
The target of the critic is deﬁned as
[Equation]
others [Equation] Inspired by prioritized experience replay (Schaul et al., 2015; Zhang & Sutton, 2017), we introduce a superior buffer where the episodic trajectories are stored.
The number of superior samples which satisfy
Q ( s, u ) >
V ( s ) and u (cid:54) =
u ∗ within a trajectory is deﬁned as the priority of the trajectory.
In the superior buffer, trajectories are ranked according to their priorities.
The training batch consists of two parts: trajectories randomly sampled from the replay buffer and the top k trajectories from the superior buffer.
At the end of each episode, the trajectories sampled from the superior buffer will be put back after the update of their priorities.
The working principle of GVR and the algorithm is given in Appendix G.
With SER, the superior samples are consist of two part: sampled from the superior buffer with a small probability and sampled from the superior buffer with a constant probability.
As a result, the proportion of superior sample is mainly determined by the relative weight on samples from superior buffer, which is a constant and irrelevant to the joint action space size.
It is proved in Appendix F that SER can eliminate the non-optimal stable points under ITS.
E XPERIMENTS
In this section, ﬁrstly we verify the attributes of stable points for linear value decomposition (LVD) in matrix games, where we also evaluate the improvements of GVR.
Secondly, to evaluate the stability and scalability of our method, we test the performance of GVR in predator-prey tasks with extreme reward shaping.
Thirdly, we conduct experiments on challenging tasks of StarCraft multi-agent challenge (SMAC) (Samvelyan et al., 2019).
Finally, we design ablation studies to investigate the improvement of GVR.
Our method is compared with state-of-the-art baselines including QMIX (Rashid et al., 2018), QPLEX (Wang et al., 2020), and
WQMIX (Rashid et al., 2020).
All results are evaluated over 5 seeds.
Details about the experimental settings and ablation studies are provided in Appendix I.
5. O NE - STEP M ATRIX G AME
Matrix game is a simple fully cooperative multi-agent task, where the shared reward is deﬁned by a payoff matrix.
In one-step matrix games, the true Q values are directly accessible from the payoff matrix, which is convenient for the veriﬁcation of the optimal consistency.
The veriﬁcation of stable points for LVD.
We conduct experiments on two-agent one-step matrix game to verify the expression of joint Q values (i.e., Eq.3).
We also verify the effect of reward function and exploration rate (cid:15) on the stable points.
The experimental results and conclusions are provided in Appendix G.
Evaluation of GVR.
Since the stability under ITS is determined by Eq. ?? , we ﬁrst verify Eq. ??
under a random inferior true Q values ( Q ). Two payoff matrices of size 3 4 (i.e., m=3 and n=4) are generated for VDN and ITS respectively: [Equation] others
[Equation]
others
[Equation]
[Equation]
where e Q = 0 .
3 , α = 0 .
We measure ∆ Q opt = Q (0 , 0 , 0 , 0) − Q (2 , 2 , 2 , 2) for VDN and ITS trained with corresponding matrices, where the greedy action is ﬁxed to u ∗ = { 2 , 2 , 2 , 2 } .
As shown in Fig.1(a), the tested ∆ Q ( u opt , τ ) consists with our calculation result ( ∆ Q ( u opt , τ ) = − 1 . 951 ) from Eq.6.
Besides, ∆ Q ( u opt , τ ) of ITS with random inferior Q equals to that of VDN with ﬁxed inferior Q , which suggests that under ITS target, the stable points are irrelevant to the return of inferior samples. As a result, ITS largely frees the stable points from the interference of the reward function.
We also evaluate the ratio of optimal stable points for VDN, ITS and GVR as (cid:15) grows.
The experiments are conducted under random inferior Q , where the payoff matrices are generated according to Q its overseeds.
At each value of (cid:15) , 100 times of independent training and test are executed.
According to Fig.1(b), for VDN, the optimal point become unstable (i.e., y = [-
-]0 ) when (cid:15) > 0 . 8 .
For ITS, the optimal points is always stable (i.e., y > 0 ( ∀ (cid:15) ∈
Besides, ITS eliminates most of the non-optimal stable points under large exploration ( (cid:15) > 0 . 837 ), which consists with our calculation result (the red dash line) from Eq.8.
For GVR, the optimal stable point is the unique stable point under a wild range of (cid:15)
GVR vs methods with complete expressiveness capacity.
To compare the efﬁciency between GVR and methods with complete expressiveness capacity, we evaluate GVR, QPLEX and CWQMIX in matrix games with different scales of action space.
Similar to Q ( its ) in Eq.10, we generate random matrices of size 3 2 , 6 3 , and 12 4 over 5 seeds, where the ﬁrst and the last element are set to be 8 and 6 respectively.
From Fig.2, in the last two tasks ( U n = 6 3 and U n = 12 4 ), the representation errors of CWQMIX and QPLEX do not decrease during training, which suggests that they are unable to learn the complete expressiveness within given steps.
We do not measure the representation error of GVR because the representation target is modiﬁed by ITS.
GVR is the only method ensuring the optimal coordination (i.e. median test return = 8 ).
5. P REDATOR -P REY
In this subsection, we compare GVR with state-of-the-art baselines in the predator-prey tasks (B¨ohmer et al., 2020).
In our experiments, 8 predators are trained to capture 8 preys, where the preys are controlled by random policies.
The reward is shared by all predators, and a punishment is applied to the reward when only a single agent capture a prey.
Our experiments are carried out under 3 punishment values.
From Fig.3, VDN and QMIX fail in all experiments, where all agents tend to avoid the preys.
QPLEX also fails in spite of its complete expressiveness capacity.
WQMIX can only solve the task with a small punishment of -2, while GVR is able to solve the tasks under all punishments.
5. S TAR C RAFT M ULTI - AGENT C HALLENGE
StarCraft multi-agent challenge (SMAC) is a popular benchmark for the evaluation of MARL algorithms.
We compare GVR with baselines in various difﬁcult tasks of SMAC.
The game version is 69232.
Each algorithm is trained for 2e6 steps in MMM2, 2c vs 64 zg and 6h vs 8z, with (cid:15) damping from 1 to 0.05 during the ﬁrst 5e4 steps.
Especially, in 6h vs 8z, each algorithm is trained for 5e6 steps, with (cid:15) damping from 1 to 0. during the ﬁrst 1e6 steps.
From Fig.4, GVR shows the best performance.
Different from predator-prey with abnormal punishments, the reward function in SMAC is more reasonable, where the linear and monotonic value decomposition can meet the TGM condition approximately.
As a result, the algorithms with full representation expressiveness capacity (QPLEX, OWQMIX, CWQMIX) do not perform better than QMIX due to the difﬁculty of complete expressiveness.
C ONCLUSION AND F UTURE W ORK
This paper discusses the optimal coordination in fully cooperative MARL tasks and proposes a new criterion (i.e., optimal consistency) to evaluate the optimality of coordination in value decomposition.
To achieve the optimal consistency, we introduce the TGM condition for linear and monotonic value decomposition, where it is proved the TGM can be ensured if the optimal stable point is the unique stable point.
Therefore, we propose the GVR algorithm. Which ensures the optimal stable points via ITS and eliminates the non-optimal stable points via SER.
In experiments on matrix games, we verify our calculation results, which directly demonstrates the effect of GVR. Besides, in experiments on predator-prey and SMAC, GVR outperforms state-of-the-art baselines.
However, GVR is unable to ensure the TGM condition in hard exploration tasks, where the superior samples are difﬁcult to obtain.
We are interested in the combination of GVR with efﬁcient exploration approaches in future work.