ResQ: A Residual Q Function-based Approach for Multi-Agent Reinforcement Learning Value Factorization
Abstract
The factorization of state-action value functions for Multi-Agent Reinforcement
Learning (MARL) is important.
Existing studies are limited by their representation capability, sample efﬁciency, and approximation error.
To address these challenges, we propose, ResQ, a MARL value function factorization method, which can ﬁnd the optimal joint policy for any state-action value function through residual functions.
ResQ masks some state-action value pairs from a joint state-action value function, which is transformed as the sum of a main function and a residual function.
ResQ can be used with mean-value and stochastic-value RL.
We theoretically show that
ResQ can satisfy both the individual global max (IGM) and the distributional IGM principle without representation limitations.
Through experiments on matrix games, the predator-prey, and StarCraft benchmarks, we show that ResQ can obtain better results than multiple expected/stochastic value factorization methods.
Introduction
Many real-world tasks involve multiple agents acting together, such as robot control [1].
Such tasks can be modelled as Multi-Agent Reinforcement Learning (MARL) problems where a group of agents must cooperate to achieve a common goal.
MARL has attracted great research interest because of its social and economic impact.
However, MARL is highly stochastic and hard to learn due to the partial-observability [2] and changing policies of agents.
To address these issues, many approaches adopt the Centralized Training with Decentralized Execution paradigm (CTDE)
In the CTDE paradigm, value factorization approaches [4] are widely adopted due to their remarkable performance and sample efﬁciency.
A joint state-action value function is factorized into per-agent utilities Q i , and each agent i acts greedily according to Q i .
To enable effective CTDE, it is critical to ensure the Individual-Global-Max (IGM) principle [5] that the optimal joint action should be equivalent to the collection of each agent’s greedy actions.
VDN [6] and QMIX [4] are two popular value factorization methods, which satisfy the IGM principle.
Most value factorization approaches focus on the deterministic state-action value functions.
Albeit
MARL is highly stochastic, the full distribution of per-agent utilities and the multi-agent system are overlooked and represented only as expected value of the full distribution.
Such distributional information could be beneﬁcial for policy learning.
DMIX and DDN [7] extend QMIX and VDN with distributional RL.
They can factorize stochastic joint state-action value pairs into stochastic per-agent utilities, and satisfy the distributional individual-global-max(DIGM) principle [7].
However, because of the representation limitations of QMIX and VDN, DMIX and DDN cannot factorize distributional value functions whose mean value is non-monotonic.
To address the representation limitations, QTRAN [5], WQMIX [8], and QPLEX [9] have been proposed.
However, QTRAN and QPLEX are sample-inefﬁcient, and WQMIX has high approximation errors to non-optimal actions.
These methods may lead to incorrect estimations of optimal values.
Moreover, they satisfy only the IGM principle.
Achieving the IGM and the DIGM principles without representation limitations remains an open challenge.
We address these challenges through, ResQ , a residual Q function-based approach.
Given a hard-tofactorize joint state-action value function Q jt (e.g., shown in Fig. 1), the key insight of our work is described as follows.
We could obtain an easy-to-factorize state-action value function, the main function Q tot , if we can mask out some state-action value pairs from Q jt .
If the main function
Q tot shares the same optimal policy with Q jt , then we can obtain the optimal policy of Q jt through the factorization of Q tot by just using some simple mixers (e.g., [6]).
After the masking, Q jt is divided into two value functions: a main function Q tot and a residual function Q r .
Q r are the masked state-action value pairs from Q jt .
Besides mean-value RL (e.g., Q jt ), the idea of ResQ can be used with distributional RL [10] as well.
ResQ can factorize stochastic joint state-action value functions Z jt into per-agent stochastic utilities
Z i with a mask and a residual function Z r .
We have shown that ResQ satisﬁes both the IGM and the
DIGM principles without representation limitations.
For evaluation, we conduct extensive experiments on one-step matrix games, the StarCraft II MARL tasks [11], and predator-prey tasks.
The experimental results show that ResQ can obtain better results than multiple competitive value factorization methods, and the ablation study shows that through the use of residual functions for multiple factorization methods, ResQ can improve the performance of these methods. The source code of ResQ can be visited at https://github.com/xmu-rl-3dv/ResQ.
Background 2.1 Dec-POMDPs
We consider cooperative Multi-Agent Reinforcement Learning (MARL) scenarios which can be modelled as Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs) [12] deﬁned as G for n agents.
G =
(cid:104)S , {U i } ni
=
1 , P, r, {O i } ni
=1 , { σ i } ni =1 , n, γ (cid:105) .
S denotes the set of states and U i the set of actions available to agent i and we consider discrete actions only.
A joint action of all agents is deﬁned as u t ∈ U N :=
U 1 × . . .
× U n .
At a discrete time step t and state s t , after the joint action is issued, the next state s t +1 ∈ S of the environment is drawn from the transition function s t +1 ∼ P ( ·| s t , u t ) .
All the agents receive a reward r t after the state transition happens.
It is a partially observable environment that agent
i observes a part of the environment o ti ∈
O i which is drawn from o ti ∼ σ i ( ·| s t ) .
γ is the discount factor.
Each agent i maintains an action-observation history τ
i =
( O i × U i ) ∗ 1 , and acts according to policy π i ( u i | τ i ) .
We denote τ ∈ T N := τ 1 × . . .
×
τ n as the joint-observation history.
The learning objective is to ﬁnd the optimal policy π = <
π 1 , ..., π n > .
2.2 Value Function Factorization
The Individual-Global-Max ( IGM ) principle proposed in [5] is important to realize the factorization of MARL state-action value function.
It is deﬁned as follows.
Deﬁnition 1 (IGM) .
For a joint state-action value function Q jt : T N ×U N (cid:55)→ R , where τ ∈ T N is a joint action-observation history, if there exist individual state-action functions [ Q i : T i ×U i (cid:55)→ R ] Ni =1 , such that the following conditions are satisﬁed
[Equation]
then, [ Q i ] satisfy IGM for Q jt under τ .
We can state that Q jt ( τ , u ) is factorized by [ Q i ( τ i , u i )]
Ni =1 .
In this work, we assume the argmax operator is unique, the action with smallest index is selected to break ties if a tie exists
QMIX is a widely used monotonically increasing mixer function which proposes the following sufﬁcient conditions for IGM:
[Equation]
There are tasks whose joint state-action value function cannot be expressed well by monotonic increasing conditions, as shown in Fig. 1.
2.3 Distributional RL
Distributional RL models the stochastic return of state-action pair through Z ( τ , u ) explicitly. They models full return distribution Z ( τ , u ) instead of Q ( τ , u ) .
The distribution of return can be approximated through a categorical distribution [13] or a quantile function [10].
Implicit Quantile Function (IQN) [10] models the stochastic value function Z ( τ, u ) as a quantile function F − 1 ( τ, u | w ) , where F − 1 is the generalized inverse cumulative distribution function (CDF), w ∈ [0 , 1] is a quantile sample.
IQN deﬁned a distributional Bellman operator, and use it to updates its Z ( τ , u ) .
After applying the distributional Bellman operator on Z ( τ , u ) , its resulting Z ( τ (cid:48) , u (cid:48) ) remains in the same distribution as Z ( τ , u ) .
The loss ρ w i ,w j t for temporal-difference error δ w i ,w j t is deﬁned as Huber quantile regression loss, where w i and w j are two quantiles.
During execution, the action with the largest expected return arg max u E [ Z ( τ , u )] is chosen.
MARL is highly stochastic, and distributional RL could be used to deal with the stochasticity of
MARL.
[7] proposes the Distributional Individual-Global-Max ( DIGM ) principle, which is deﬁned as follows.
Deﬁnition 2 (DIGM) .
Given a set of stochastic individual state-action value function [ Z i ( τ i , u i )]
Ni = and a stochastic joint state-action value function Z jt ( τ , u ) , if the following conditions are satisﬁed
[Equation]
then, [ Z i ( τ i , u i )] Ni =1 satisfy DIGM for Z jt under τ .
We can state that Z jt ( τ , u ) is distributionally factorized by [ Z i ( τ i , u i )]
Ni =1 .
Related Work
Value factorization approaches are widely adopted in MARL [14].
VDN [6] factorizes the value function Q tot as the sum of per-agents’ utility Q i .
QMIX [4] supports monotonic relationships among
Q i and Q tot .
During execution, each agent acts greedily according to Q i .
RMIX [15] learns return distribution of each agent, and integrates risk-sensitive RL with QMIX.
[7] proposes DMIX and
DDN, which extend QMIX and VDN with distributional RL through mean-shape decomposition.
They satisfy the DIGM principle but suffer from the representation limitation of QMIX and VDN.
They cannot model stochastic value functions whose mean-value are non-monotonic.
QAtten [16] and REFIL [17] adopt the attention mechanisms to focus on certain agents/scenarios when factorizing value function.
QRelation [24] considers the relationships among agents for value factorization.
However, they cannot represent non-monotonic value functions well.
To address the representational limitations, WQMIX [8] prioritizes the estimation of the optimal stateaction value.
It assigns higher weights to the optimal joint actions when minimizing approximation errors.
WQMIX can be viewed as a special version of ResQ. It pays attention to the optimal actions via masking out all the sub-optimal actions.
The Q r of WQMIX consists of all the sub-optimal stateaction pairs, and the Q tot models only the optimal ones.
However, it assigns high learning priorities to Q tot , which puts the learning of sub-optimal state-action pairs and Q jt in trouble.
Moreover, it is difﬁcult to assign proper weights to obtain satisfactory performance.
QTRAN [5] and QTRAN++ [18] transform the joint state-action value function Q jt into an easyto-factorize one through a series of linear constraints.
QTRAN learns an approximated function
Q tran ( τ , u ) =
(cid:80) Ni =
1 Q i ( τ i , u i ) + V jt ( τ ) to approximate the optimal policy of Q jt .
They can be viewed as using Q tot =
(cid:80) Ni =
1 Q i ( τ i , u i ) to model the optimal actions, and constraining Q tran ≥ Q jt for sub-optimal actions.
QTRAN use soft constraints (MSE losses) to implement IGM which could lead to the violation of IGM.
RQN [19] extends QTRAN by adding an individual correction factor for each utility function to compute an adjusted utility function.
RQN can be viewed as a special case of ResQ. It uses the sum of per-agent utility as the main function Q tot , and the sum of individual correction factors as the residual function Q r .
QPlex decomposes Q jt as the sum of a value function V tot and a non-positive advantage function
A tot .
The role of A tot is similar to the main function Q tot used in ResQ. QPlex obtains agents’ policies from the factorization of A tot , which must be ≤ 0 , but ResQ does not have such restrictions.
Although QPlex does not suffer from representation limitation issues, it may struggle to learn a good policy in complex tasks [8].
MAVEN [20] deals with the inefﬁcient exploration problem in MARL.
Inefﬁcient exploration problems could interact with the representation limitation problems.
It adopts QMIX as its value factorization method.
CDS [27] improves the effective cooperation of agents with efﬁcient exploration. Different from them, ResQ focus on value factorization, and can be used in MAVEN and CDS.
COMNet [21] and DIAL [22] study communication among agents.
DCG [23] uses graphs for coordination.
COPA [25] and NDQ [26] factorize the value function with communication. The communication method adopted by them could be used with ResQ to improve MARL performance.
There exist various MARL actor-critic methods, such as MADDPG [28], MAAC [29], and COMA [30].
ResQ is a value-based method.
Other methods exist.
For example, MAPPO [31] combines MARL with PPO [32].
UPDeT [33] proposes a MARL policy decoupling method.
ResQ is orthogonal to them.
Method
Motivating Example Fig.
1 shows an example of a non-monotonic payoff matrix, which is hard-tofactorized.
Agent 1 has three different action-value orderings, which depend on the action of Agent 2.
Moreover, the second-best state-action value 7 . 9 is only slightly lower than the optimal value 8 , which causes difﬁculties for methods with representation limitations or overestimations.
Given a hard-to-factorize state-action value function Q jt , the key insight to ResQ is that if we can mask out some state-action value pairs from Q jt , then we could obtain an easy-to-factorize value function Q tot .
As shown in Fig.
1, if we mask out those action value pairs colored in red, then the main function Q tot can be factorized easily by mixers such as QAtten or QMIX.
ResQ seeks to ﬁnd a mask w , a main functions Q tot , and a residual function Q r which satisfy the following equation.
[Equation]
where masks w tot ( τ , u ) , w r ( τ , u ) ∈ { 0 , 1 } .
The main Q tot shares the same greedy optimal policy as Q jt .
Q tot ( τ , u ) is factorized into per-agent utilities Q i , and each agent selects its action greedily with respect to Q i .
There are many possibilities to construct the masks w tot and w r , in this work, we set w tot to be constant 1 , and focus on w r .
We describe how ResQ factorizes an expected-value joint state-action value function Q jt into peragent utilities in Sec. 4.1, and how ResQ decomposes a stochastic joint state-action value function
Z jt into stochastic individual utilities in Sec.
4.1 Residual Q
For a given joint action-observation history τ , consider a factorizable joint state-action value function
Q jt ( τ , u ) that can be expressed in the form as (5).
Let ¯ u i = arg max u i Q
i ( τ i , u i ) , ¯ u =
[ ¯ u i ]
Ni =1 ,
Theorem 1 states the sufﬁcient conditions for state-action value functions [ Q i ( τ i , u i )]
Ni =1 that satisfy the IGM principle for Q jt .
Theorem 1.
A joint state-action function
[Equation]
is factorized by [ Q i ( τ i , u i )]
Ni =1 , if Q r ( τ , u ) ≤ 0 , Q tot ( τ , u ) and [ Q i ( τ i , u i )]
Ni =1 satisfy the monotonicity conditions (2), and
[Equation]
The proof of Theorem 1 is shown in the appendix.
In Theorem 1, we have shown that for any factorizable value function Q jt that can be expressed as (5), we can ﬁnd a main function Q tot that shares the same optimal policy as Q jt .
In Theorem 2, we show that for any joint state-action value function Q , we can ﬁnd a joint state-action value function Q jt that can be expressed as (5), and it has the same optimal policy as Q .
Theorem 2.
For any joint state-action function Q ( τ , u ) , we can ﬁnd Q jt ( τ , u ) =
Q tot ( τ , u ) + w r ( τ , u ) Q r ( τ , u ) that
[Equation]
Q tot ( τ , u ) monotonically increases with [ Q i ( τ i , u i )]
Ni =1 , w r ( τ , u ) satisﬁes (6), and Q r ( τ , u ) ≤ 0 .
The proof of Theorem 2 and other theorems are provided in the appendix.
From Theorem 2, for any joint state-action function Q ( τ , u ) , we can ﬁnd Q jt ( τ , u ) that shares the same optimal policy as Q .
Moreover, Q jt can be expressed as (5).
Combining Theorem 1 and 2, for any Q ( τ , u ) , we can ﬁnd
[ Q i ( τ i , u i )]
Ni =1 satisfy the IGM principle for Q ( τ , u ) through a residual function Q r ( τ , u ) ≤ 0 , a main function Q tot ( τ , u ) , and a mask function w r ( τ , u ) ∈ { 0 , 1 } .
The policies [ Q i ( τ i , u i )]
Ni =1 can be obtained through factorizing Q tot ( τ , u ) using mixers such as QMIX [4].
Furthermore, Formula (8) in Theorem 2 shows that Q jt ( τ , u ) can track Q ( τ , u ) closely ∀ u (cid:54) =
¯ u .
Thus, ResQ can model Q values precisely.
4.2 Distributional Residual Q
One promising way to deal with the MARL stochasticity is to model the return of state-action pairs as distributional values [10].
Researchers [7] have proposed a distributional MARL approach that satisﬁes the DIGM principle.
However, it cannot model stochastic joint value functions whose value-expectation are non-monotonic.
ResQ can address this issue through using Z dmix ( τ , u ) as the main function and Z r as the residual function, where Z dmix ( τ , u ) is the factorization function of
DMIX or DDN [7], and Z dmix , Z r are both stochastic value functions.
Formula (9) in Theoremdescribes how Z jt is decomposed.
Theorem 3.
A stochastic joint state-action function
[Equation] Z
[Equation]
is factorized by [ Z i ( τ i , u i )]
Ni =1 , if Z r ( τ , u ) ≤ 0 and w r ( τ , u ) = 0 when u
= ¯ u , otherwise 1 .
¯ u
i = arg max u i E
[ Z i ( τ i , u i )] , ¯ u =
[ ¯ u i ]
Ni =1 , Z dmix ( τ , u )
=
Z mean ( τ , u ) + Z shape ( τ , u ) , E [ Z shape ( τ , u )] = 0 , Q i =
E
[ Z i ( τ i , u i )] .
Z mean ( τ , u ) is a monotonically increasing function with respect to Q i . where Z mean ( τ , u ) models the mean of Z dmix . In DMIX [7], it is implemented using QMIX, and in DDN, it is implemented as VDN. Z shape models the variation of Z dmix .
Theorem 3 shows that
ResQ can remove the representation limitations of DMIX. Besides freeing [7] from representation limitations, ResQ can factorize a stochastic value function Z jt into individual stochastic utilities Z i with a linear main function Z tot , a residual function Z r , and a mask function w r .
Theorem 4 states the sufﬁcient condition for [ Z i ( τ i , u i )]
Ni =1 satify the DIGM principle.
Theorem 4.
A stochastic joint state-action function
[Equation]
is factorized by [ Z i ( τ i , u i )]
Ni =1 , if Z r ( τ , u ) ≤ 0 , Z tot ( τ , u ) =
(cid:80) Ni =
k
i Z i ( τ i , u i ) k
i ≥ 0 and w r ( τ , u )
=
when u = ¯ u , otherwise 1 , where ¯ u =
[ ¯ u i ]
Ni =1 ¯ u i = arg max u i E
[ Z i ( τ i , u i )] .
The proofs of Theorem 3 and 4 are provided in the appendix.
Theorem 4 states that ResQ can ﬁnd individual stochastic utilities satisfying the DIGM principle for Z jt .
And it does not limit the representation capacity of stochastic joint state-action function Z jt .
In (10), the main function Z tot is a positive weighted sum of Z i instead of a monotonic mixing individual utilities as in (5).
This is because not all monotonic increasing functions can be used as the main function.
Some monotonic increasing function (e.g., Z tot = (cid:80) Ni
=
Z 2 i ) could lead to incorrect estimation of the optimal actions (please refer to the Appendix for an example), they cannot satisfy the DIGM principle.
4.3 Neural Networks
For any state-action value function Q jt , if we can construct the main function Q tot , the mask w r , and the residual function Q r as (5), then we can recover the optimal policy for Q jt .
Similarly, for stochastic function Z jt , we can recover the optimal actions for Z jt through the main function Z tot , the residual function Z r and the mask as (10).
The neural network of ResQ is depicted in Fig.
Main Function For Q jt , Q tot ( τ , u ) can be any monotonically increasing function.
The most well known monotonic increasing function of per-agent utility [ Q i ]
Ni =1 are QMIX, VDN, and QAtten.
For the stochastic case, the stochastic main function Z tot ( τ , u ) can be a positive weighted sum of individual utilities
Z i ( τ i , u i ) .
Z tot ( τ , u ) = (cid:80) {+Ni =
+}
k
i Z i ( τ i , u i ) k
i ≥ 0 .
Speciﬁcally, we model k i by the attention mechanism [16].
The residual function Q r ( τ , u ) is a feed-forward network that takes actionobservation history τ , agents’ utilities Q i , and action [ u i ]
Ni =1 as input.
Unlike Q tot , Q r is not constrained to be monotonic or positive weighted-sum of utilities, thus allowing it to model a larger class of functions.
For [ Q i ] Ni =1 , Q r is a simple feed-forward network which takes [ Q i ]
Ni =1 as inputs, and the negative absolute function is the activation function of the last neural network layer to ensure
Q r ≤ 0 .
For the stochastic case, the residual function Z r is modeled as −| (cid:80) Ni =
w i
Z i | .
Mask Function
The mask w r ( τ , u ) requires knowing the maximal joint actions for the joint stateaction value function.
However, it is intractable to obtain the optimal actions without searching over the entire action space.
For practical implementation, we use approximation, and w r is deﬁned as [Table] [Table] [Table]
[Table]
follows.
[Equation]
where (cid:101) u = arg max u Q tot ( τ, u ) for the expectation case.
For the stochastic case, (cid:101) u = arg max u E
[ Z tot ( τ , u )] .
Loss For the expected-value case, the loss L of ResQ consists of two temporal-difference losses: L ∗ and L jt .
L =
L ∗ + L jt .
L ∗ =
(cid:80) bk
=
( y k − Q jt ( τ k , u k , θ )) 2 , where y k
= r + γQ jt ( τ k +1 , (cid:101) u, θ − ) , (cid:101) u
= [ (cid:101) u
i ]
Ni =1 , (cid:101) u i = arg max Q
i ( τ k +1 i , u i ) .
θ − are the parameters of a target network.
L jt measure the distance between Q jt and Q tot + w r Q r .
It is deﬁned as L jt =
(cid:80) ( Q jt − Q tot − w r Q r ) 2 .
For the distributional case, L = L z ∗ + L zjt .
L z ∗ and L zjt are similar to L ∗ and L jt , respectively.
L z ∗ and L zjt use the pair-wise sampled temporal difference error instead of the standard temporal error, and the Huber quantile regression loss [10] rather than the mean square error loss.
Evaluation
We evaluate the performance of ResQ on one-step matrix games, the StarCraft II Multi-Agent
Challenge benchmark (SMAC) [11], and predator-prey with multiple algorithms.
The experimental results show that ResQ can obtain better results than state-of-the-art methods, and it satisﬁes both the IGM and DIGM principle together without representation limitation.
The ablation study shows that ResQ can improve the performance of multiple value factorization methods through using the residual function.
Please refer to the appendix for detailed setup and more results.
5.1 Experimental Setup
The studied algorithms are listed in Table 1.
ResZ is the distributional version of ResQ factorized according to (10).
DAtten, designed by us, integrates distributional RL [10] and the attention mechanism.
Other algorithms are conﬁgured with their default parameters, and the number of rollouts is one.
Each experiment is repeated at least 5 times with different seeds.
The conﬁguration of ResQ follows WQMIX [8].
5.2 Matrix Game
We study the representation power of multiple methods for a matrix game (depicted in Fig.
1 and
Table 2).
All the algorithms are ran through a full exploration (cid:15)
= 1 for (cid:15) -greedy conducted over 50,000 steps.
This setting guarantees the exploration of all possible actions.
Table 2 depicts the Q jt of multiple methods. As it is observed in the table, only ResQ/ResZ, QTran, and CW QMIX can obtain the optimal policy, DMIX, DDN, and QPlex learn the second-best policy, and OW QMIX learns a wrong policy.
Albeit CW QMIX can ﬁnd the optimal action, its learned Q jt has a high approximation error, as it focuses on the optimal actions only.
The detailed factorization results are depictd in Table 1 in the appendix.
Further, we modify the matrix into a distributional matrix by adding a nomral distribution (mean = 0, std=1) value into all action-value.
The payoff matrix and the reconstructed Q values are shown in Table 2 in the appendix.
We ﬁnd that ResQ/ResZ and QTran can ﬁnd the optimal policy while most of the algorithms cannot.
These results demonstrate the ability of ResQ to factorize difﬁcult state-action value functions with low approximation errors without representation limitations.
5.3 StarCraft II Multi-Agent Challenge (SMAC)
In SMAC [11], two teams of agents ﬁght against each other.
For the SMAC tasks, we train each algorithm for 1 to 2 million steps, and for every 10,000 steps, the learned policies are evaluated. The
[Table] [Figure] test win rate, which measures the average win ratio of the agents controlled by MARL algorithms for each 32 test episodes, is reported.
As shown in Figure 3 (a), ResQ and ResZ achieve the best performance, and QMIX has the third-best results in the MMM2 scenario.
The performance improvement over QMIX comes from the fact that
ResQ/ResZ can model better the non-monotonic state-action function of the MMM2 scenario.
For the MMM scenario, ResQ/ResZ has the best test win rate.
The results for the 3s_vs_5z tasks are depicted in Fig.
3 (c).
ResQ can obtain close-to-optimal performance after 1 million steps, while others cannot.
For the bane_vs_bane scenario, in Fig.
3 (d),
ResQ and ResZ can learn the optimal policy in less than 0.2 million steps, while the other methods cannot.
The results indicate that through masking state-action value pair from Q jt / Z jt , ResQ/ResZ can learn the optimal policy in short time, and the performance is stable .
Albeit QPlex and DMIX can obtain close-to-optimal performance at a much later time, but their performance is unstable.
The results for CW/OW QMIX for bane_vs_bane and 27m_vs_30m are not shown due to out-of-memory error on an NVIDIA Geforce RTX 3090 GPU.
ResQ/ResZ can obtain the close-to-optimal performance after 0.25 million steps in the 2s_vs_1sc scenario, shown in Fig.
3 (e).
Although the other methods can learn close-to-optimal policy with more training steps, their performance is not stable.
For the 1c3s5z scenario, depicted in Figure 2 (f),
ResQ/ResZ are among the best-performing algorithms.
In the 2c_vs_64zg scenario, 2 Colossi agents ﬁghts against 64 enemy agents.
It has a large action space, and the reward of each action varies rather than is constant as in other scenarios.
As it is depicted in Fig.
3 (g), ResZ can obtain the best performance.
This indicates ResZ’s ability to model distributional reward.
The results for 8m_vs_9m and 27m_vs_30m are depicted in Fig.
3 (h) and (f), respectively.
ResQ and ResZ are the second-best performing algorithms.
DMIX is good at scenarios which consist of homogeneous agents (e.g., 8m_vs_9m), but it under-performs for scenarios with heterogeneous agents (e.g., MMM2, 3s_vs_5z), where agent interactions are complex.
5.4 Predator Prey
In the predator prey environment [23], 8 agents hunt 8 preys in a 10 × 10 grid world.
Each agent has 6 actions: moves in either 4 directions, stands still, and catches prey.
If two adjacent agents execute simultaneously the catch action to a nearby prey, a reward r = 10 is given to the agents, but a failed catch by a single agent is punished by a reward p ≤ 0 .
The more negative p is, the higher level of coordination is needed for the agents.
Following [8], the (cid:15) of the (cid:15) -greedy strategy for QPlex, ResQ, and CW/OW QMIX is gradually annealed from 1 to 0.05 within 1 million steps.
The test return for three punishments: p = 0 , p = − 2 , and p = − 4 are shown in Fig.
For the most easy scenario p = 0 , ResQ, QMIX, and QPlex can learn the optimal policies quickly.
The performance of CW/OW QMIX drops in the middle of training.
For the difﬁcult p = − 2 case, ResQ can obtain the best returns.
CW QMIX and OW QMIX have the second and third best scores.
QTRAN and QMIX can not obtain any positive returns.
For p = − 4 , ResQ can obtain the best returns in the end.
These results indicate ResQ’s ability for challenging cooperation scenarios.
5.5 Ablation Study
We study the impact of different implementations for the main function and the residual function, and Starcraft versions on the SMAC benchmark.
We ﬁnd that ResQ can improve the performance of multiple value factorization approaches through the use of the residual function Q r .
Main Functions In SMAC, QMIX is used as the implementation of Q tot . For ResQ, the results of using QMIX, QAtten, and VDN as the main function Q tot are depicted as ResQ, Qtot-QAtten, Qtot-VDN in Fig.
5 (a) and (b).
As it is shown, for the MMM2 and 8m_vs_9m scenarios, ResQ, QtotQAtten, and Qtot-VDN perform better than QMIX, QAtten, and VDN, respectively.
This indicates that ResQ can improve the performance of value factorization approaches (the main function) through the use of residual functions and mask.
Moreover, we ﬁnd that QMIX is the preferred main function for the SMAC benchmark.
For the distributional case, shown in Fig.
5 (c).
The results of using DAtten, DDN, and DMIX as the main function Z tot are depicted as ResZ, Ztot-DDN, and Ztot-DMIX.
DAtten is designed by us; it integrates IQN [10] with QAtten.
ResZ, Ztot-DDN, and Ztot-DMIX perform better than DAtten,
DDN, and DMIX, respectively.
This indicates that ResZ can improve the performance of stochastic value factorization methods through residual functions.
Residual Functions
The results of using different mixers (factorization methods) for the residual functions are shown in the Fig.
5 (d) and (e).
The performance of using a feed-forward network (FF) as Q r (denoted as ResQ) is similar to that of using advanced mixers such as QMIX (Qr-QMIX) and VDN (Qr-VDN), and higher than that of QAtten (Qr-QAtten).
In ResQ, we do not restrict the function classes of Q r .
This suggests that using a simple feed-forward network may be sufﬁcient for Q r .
For the distributional case, the performance of using DAtten, FF, DDN, DMIX is depicted as ResZ, Zr-FF, Zr-DDN, Zr-DMIX. As it is depicted in Fig.
5 (e), we ﬁnd that using DAtten as the residual function Z r is preferred.
The Starcraft II version used in this work is SC2.4.6.2.69232 (2.4.6 for short), which is the same version used as QMIX [4] and WQMIX [8].
Some methods (e.g.,[7]) uses the version SC2.4.10.
We compare ResQ with Pymarl2_QMIX [14], which is a ﬁne-tuned QMIX, across two versions.
Figure 5 (f) depicts the results for different methods.
The version of StarCraft is written inside the parentheses after the name of each method.
For example, ResQ (2.4.6) denotes runing ResQ against
SC2.4.6. As it is shows that ResQ/ResZ perform better than QMIX and Pymarl2_QMIX for both two versions in the MMM2 scenario.
As Pymarl2_QMIX performs better than most algorithms in most case for SC2.4.10 [14], this suggests that ResQ/ResZ may perform better than other algorithms in SC2.4.10.
Conclusion
In this work, we propose, ResQ, a residual function-based approach for Multi-Agent Reinforcement
Learning (MARL) value function factorization.
ResQ recovers the optimal policy for any joint state-action value function by masking out state-action value pairs from the value function.
We show that ResQ can satisfy the individual-global-max (IGM) principle and the distributional IGM principle without representation limitations.
Through extensive experiments on multiple MARL tasks, we show that ResQ can obtain promising results.
Acknowledgement This work was partially supported by the National Natural Science Foundation of China (61872376, 61972409), by open fund of PDL, by the China Postdoctoral Science Foundation (No.2021M690094); the FuXiaQuan National Independent Innovation Demonstration Zone
Collaborative Innovation Platform (No.3502ZCQXT2021003).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] The complete proofs are included in the appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] It is provided in the supplemental material, and the code can be viewed at https:
//github.com/xmu-rl-3dv/ResQ (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] The standard errors are shown.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Each experiment is ran with a NVIDIA 3090 GPU. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We use some of the code from Pymarl, Pymarl2, and DFAC (b) Did you mention the license of the assets?
[Yes] It is source code released by the authors of papers.
Their code licensed under the Apache License v2.0.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] The source code of this work is included in the supplemental material, and it will be open-sourced.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Appendix
In Section A.1, we provide the ResQ theorems and proofs.
Section A.2 describes more experimental results. We describe the exeprimental setup in Section A.2.1 in detail, the factorization results in Section A.2.2 and shows another payoff matrix that cannot represent well by state-of-the-art algorithms in Section A.2.3. We show more results about the SMAC benchmark in Section A.2.4.
We compare ResQ with QTRAN in Section A.3.2, with QPlex in Section A.3.3, with weighted QMIX in Section A.3.4, and study the impact of the implementation of the inequality condition in Section A.3.5, and discuss the limitation of ResQ in Section A.3.6.
A.1 Theorems
In this section, we show the proof of all the Theorems of this work.
Theorem 1.
A joint state-action function
[Equation]
(1) is factorized by [ Q i ( τ i , u i )]
Ni =1 , if Q r ( τ, u ) ≤ 0 , Q tot ( τ, u ) and [ Q i ( τ i , u i )]
Ni =1 satisfy the monotonicity conditions (2), and
[Equation]
(2b)
Proof.
Theorem 1 shows that if condition (2) holds and Q r ( τ, u ) ≤ 0 , then [ Q i ( τ i , u i )]
Ni =1 satisfy the
IGM principle for Q jt .
We will show that arg max u Q jt ( τ, u ) =
¯ u , where ¯ u
i
=
arg max u i Q
i ( τ i , u i ) and ¯ u =
[¯ u i ]
Ni =1 .
[Equation]
[Equation]
(5) comes from w r ( τ, u )
= 1 ∀ u (cid:54) =
¯ u and Q r ( τ, u ) ≤ 0 .
(3) to (6) mean that ¯ u
=
[¯ u i ]
Ni =1 maximizes
Q jt ( τ, u ) .
Thus [ Q i ( τ i , u i )]
Ni =1 satisﬁes IGM for Q jt ( τ, u ) .
Lemma 1.
For any joint state-action function Q ( τ, u ) , we can ﬁnd a Q tot that satisfy the monotonicity conditions, and they share the same optimal policy.
Proof.
Theorem 1 of [8] shows that Weighted QMIX can always ﬁnd a monotonically increasing function Q tot that shares the same optimal policy as the true value function of Q ( s, u ) .
This indicates that for any Q ( s, u ) , there exists a monotonically increasing function Q tot ( s, u ) that shares the same optimal policy as Q ( s, u ) .
Theorem 2.
For any joint state-action function Q ( τ, u ) , we can ﬁnd Q jt ( τ, u ) =
Q tot ( τ, u ) + w r ( τ, u ) Q r
[Equation]
τ, u ) that
[Equation]
, u ) is a monotonic increasing function with respect to [ Q i ( τ, u )]
Ni =1 , w r ( τ, u ) satisﬁes the condition (2), and Q r ( τ, u ) ≤ 0 .
Proof.
Lemma (1) shows that for any joint state-action function Q ( τ, u ) , we can ﬁnd a Q tot that satisfy the monotonicity conditions and ¯ u = arg max Q tot ( τ, u ) [Equation]
arg max Q ( τ, u ) .
Let
[Equation] [Equation]
Obviously, the new function Q (cid:48) tot ( τ, u ) satisfy the monotonic conditions, and arg max u Q ( τ, u ) = arg max u Q (cid:48) tot ( τ, u ) .
Deﬁne the residual Q function as
[Equation]
Then
[Equation] [Equation] [Equation]
[Equation]
We can write Q ( τ, u ) as follows.
[Equation] [Equation]
16) where
[Equation]
17b)
Let
[Equation]
18a)
[Equation] 18b [Equation] Through deﬁnition, Q (cid:48) r ( τ, u ) ≤ 0 .
Let
[Equation]
Obviously, Q (cid:48)(cid:48) tot ( τ, u ) satisfy the monotonic conditions with [ Q i ] Ni =1 .
[Equation]
Combining (18) and (19), we can ﬁnd a function
[Equation]
Thus, we show that for any joint state action function Q ( τ, u ) , it can ﬁnd a function Q jt that can be expressed in the form of ResQ (1), and share the same optimal actions with Q .
Further [Equation] we analysis the property of Q jt ( τ, u ) ∀ u (cid:54) =
¯ u .
[Equation]
Then [Equation] we analysis the property of Q jt ( τ, ¯ u ) .
[Equation]
Combining (23) and (27), we reach the conclusion that the two functions Q ( τ, u ) and Q jt ( τ, u ) differ only in the maximal values.
And Q jt ( τ, u ) can be expressed as the form of the addition of one monotonicity function Q tot ( τ, u ) and a residual state-action function Q r ( τ, u )
Theorem 3.
A stochastic joint state-action function
[Equation]
) is factorized by [ Z i ( τ i , u i )]
Ni =1 , if Z r ( τ, u ) ≤ 0 and w r ( τ, u ) = 0 when u
= ¯ u , otherwise 1 .
Z dmix ( τ, u ) is the factorization method proposed in [7].
Z dmix ( τ, u ) =
Z mean ( τ, u )+ Z shape ( τ, u ) , E [ Z shape ( τ, u )] = 0 , Q i =
E
[ Z i ( τ i , u i )] .
Z mean ( τ, u ) is a monotonically increasing function with respect to Q i or the sum of [ Q i ] N 1 .
Proof.
We will show that arg max u E [ Z jt ( τ, u )]
= ¯ u , where ¯ u
i
=
arg max u i E
[ Z i ( τ i , u i )] and ¯ u =
[¯ u i ]
Ni =1 .
E [ [Equation]
] ( From (28) w ( τ, ¯ u ) [Equation]
[Equation]
(29) =
[Equation]
[Equation]
τ, ¯ u )] (30) =
E [ Z mean ( τ, ¯ u )]
( From E [ Z shape ( τ, u )] = 0) (31) =
E [ Q mix ( τ, ¯ u )] ( Deﬁnition ) (32) ≥ E [ Q mix ( τ, u )] ∀ u (cid:54) = ¯ u ( Mononicity relation )
[Equation]
(37) comes from w r ( τ, u ) = 1 ∀ u (cid:54) =
¯ u and Z r ( τ, u ) ≤ 0 .
(29) to (39) mean that ¯ u =
[¯ u i ]
Ni = maximizes E [ Z jt ( τ, u )] .
Thus [ Z i ( τ i , u i )]
Ni =1 satisﬁes DIGM for Z jt ( τ, u ) .
Theorem 4.
A stochastic joint state-action function
[Equation]
[Equation] is factorized by [ Z i ( τ i , u i )]
Ni =1 , if Z r ( τ, u ) ≤ 0 , Z tot ( τ, u ) =
(cid:80) Ni =
k
i Z i ( τ i , u i ) k
i ≥ 0 and w r ( τ, u )
=
when u = ¯ u , otherwise 1 .
Proof.
Theorem 4 shows that if Z r ( τ, u ) ≤ 0 , Z tot ( τ, u ) = (cid:80) Ni =
k
i Z i ( τ i , u i ) k
i ≥ 0 and w r ( τ, u )
=
when u = ¯ u , otherwise 1 , then [ Z i ( τ i , u i )]
Ni =1 satisfy the DIGM principle for Z jt ( τ, u ) .
We will show that arg max u E [ Z jt ( τ, u )]
= ¯ u , where ¯ u
i
=
arg max u i E
[ Z i ( τ i , u i )] and ¯ u =
[¯ u i ]
Ni =1 .
[Equation]
Z jt
[Equation]
Because Z tot ( τ, u ) = (cid:80) Ni =1 k
i Z i ( τ i , u i ) k
i ≥ 0 , the inequality formula (42) is established.
(43) comes from w r ( τ, u ) = 1 ∀ u (cid:54) =
¯ u and Q r ( τ, u ) ≤ 0 .
(44) comes from w r ( τ, u )
Z r ( τ, u ) =
E [ w r ( τ, u )
Z r ( τ, u )] .
(41) to (45) mean that ¯ u
= [¯ u i ]
Ni =1 maximizes E [ Z jt ( τ, u )] .
Thus [ Z i ( τ i , u i )]
Ni =1 satisﬁes DIGM for Z jt ( τ, u ) .
In Theorem 4, the main function Z tot is a positive weighted sum of Z i instead of a monotonic mixing of Z i , because not all monotonic mixing functions satisfy the DIGM principle. We will show an example in the proof of Theorem 5. Theorem 5.
The factorization method CW/OW QMIX may fail to satisfy the DIGM theorem. Proof. We prove this theorem through an example. In expectation case, CW/OW QMIX learn Q tot ( τ, u ) to approximate the optimal policy of the stateaction value function Q ( τ, u ) , where Q tot is a monotonic increasing function with respect to Q i .
For the stochastic case, CW/OW QMIX learn Z tot ( τ, u ) to approximate the true value function, where Z tot is a monotonic increasing function with respect to Z i . That is ∂Z tot ( τ, u ) /∂Z i
( τ i , u i ) ≥ Let’s consider a simple one-step matrix game where there are only two agents each with actions a and b. Let’s assume that the state is full observational.
The learned function is Z tot ( τ, u ) =
Z tot ( s, u ) =
Z 21 ( s, u 1 ) + Z 22 ( s, u 2 ) , where Z 1 = Z 2 .
Z tot is the summation of the square of the stochastic utilities. Z tot could lead to incorrect estimation of the optimal actions. Let’s assume that, for action a ,
Z 1 ( s, a ) = 2 for 100% of the time; for action b Z 1 ( s, b ) = 3 for 50% of the time and Z 1 ( s, b ) =[-
-]
for 50% of the time.
Clearly E Z 1 ( s, a ) = {+
+}
and E Z 1 ( s, b ) = 1 . 5 .
If Z tot and Z i satisfy the DIGM theorem, then the optimal action for Z tot should be ( a, a ) =
(arg max E Z 1 ( τ, u ) , arg max E Z 2 ( τ, u )) .
However, E Z 21 ( τ, a ) = 4 and E Z 21 ( τ, b ) = 4 . 5 .
arg max E Z tot =
( b, b ) rather than ( a, a )
A.2 Experimental Results A.2.1 Experimental Setup
We adopt the PyMARL [11] implementation of QMIX,VDN, QAtten, QTran, OW QMIX, CW
QMIX, QPlex, and DMIX from their open-source repositories 234 .
Their code are released under the
Apache License V2.0.
Their hyper-paramters are the same as that in PyMARL.
ResQ is designed based on PyMARL as well.
All the algorithms is trained with 1 rollout process for 2 million steps (the default setup for Weighted QMIX).
At each 10,000 training step, the algorithm is tested on test episodes to evaluate their performance.
For ResQ, the RMSProp optimizer ( l r = 1 e − 3 ) is used.
The batch size and buffer size are 32 and 5,000, respectively.
In the matrix game and SMAC, QMIX is chosen as the main function Q tot .
Q tot has 32 hidden dimensions, equiped with 2 hypernets, each with 64 dimensions.
In predator-predy, VDN is chosen as Q tot .
The (cid:15) used in (cid:15) − greedy annealed from 1 to 0.05 within 100K steps.
TD lambda is used, and λ = 0 .
All the experiments are ran within a local cluster.
Each experiment is ran within a NVIDIA 3090 GPU.
A.2.2 A one-step Matrix Game
Table 1 show the detailed factorization results of the one-step pay-off matrix (depicted in Figure 1).
[Table]
[Table]
Table 2 shows another difﬁcult pay-off matrix.
In this matrix, only ResQ and QTran can recover the optimal action, the other algorithms cannot.
A.2.3 A distributional one-step Matrix Game
We study the representation power of multipe methods for a distributional matrix game.
The matrix game is modiﬁed from the matrix depicted in Figure 1.
We add a nomral distribution (mean = 0, std=1) value into all action-value of the matrix.
The payoff matrix of the game is depicted in Table(a).
All the cells show the mean and the variance of a normal distributed reward.
For example, (8,1) indicates that the mean and the variance of the reward distribution are 8 and 1, respectively.
In this matrix game, all the algorithm are ran through a full exploration (cid:15) =
1 for (cid:15) -greedy conducted over 20,000 steps.
This setting guarantees the exploration of all possible actions.
As it is observed in Table 3, ResQ/ResZ, and QTRAN can obtain the optimal policy, DMIX, DDN,
CW QMIX, and QPlex learn the second-best policy, and OW QMIX learns a wrong policy.
These result demonstrates the ability of ResQ to factorize difﬁcult state-action value functions with low approximation error and high sample efﬁciency.
Moreover, we ﬁnd that ResZ has lower approximation error than ResQ for this stochastic pay-off matrix.
A.2.4 StarCraft II Multi-Agent Challenge (SMAC)
We study the performance of ResQ and other algorithms on a well known MARL benchmark, the
StarCraft II Multi-Agent Challenge (SMAC).
In each environment of SMAC, two teams of agents [Table] [Table] ﬁght against each other.
One team is controlled by the built-in game artiﬁcial intelligence, which is handcrafted carefully.
We choose the highest difﬁcult built-in game script.
Another team of agents are controlled by the decentralized policies learned by the MARL algorithms.
Each agent can receive circular observation around the agent, and attack enemies close-by.
In each episode, if the agents eliminate all the enemies within a limited time, the game episode is counted as won.
Otherwise, the game is counted as a failure.
For the SMAC tasks, the agents must learn the policies which maximize their expected total rewards.
The reward of SMAC is based on the damage dealt, enemy agents killed, and the game won.
We adopt the default SMAC reward schema.
The Starcraft II version used in ResQ is 2.4.6.2.69232, which is the same version used in QMIX and
Weighted QMIX.
For pymarl2 [14], it used another version SC 2.4.10.
As it is indicated in Weighted
QMIX and conﬁrmed by us, " performance is not comparable across versions ".
We have shown in
Figure 5 (f), ResQ/ResZ perform better than the ﬁne-tuned QMIX [14] for SC2.4.10 in the MMM scenario.
We have shown the results for ResQ, ResZ, QPLEX, QTran, QMIX, VDN, DMIX, OW QMIX, and CW QMIX.
We have also evaluated the performance of DDN [7], and REFIL [17].
Their results are ploted in Figure 1.
Due to the poor performance of REFIL, we only evaluate its performance on the MMM2, MMM, 3s_vs_5z, 1c3s5z, 8m_vs_9m scenarios.
As it is shown in the picutre, ResQ performs better than REFIL and DDN.
Figure 2 depicts the results for three super-hard scenarios (Corridor, 3s5z_vs_3s6z, 6h_vs_8z) of the
SMAC benchmark.
AS it is shown in the ﬁgure, all algorithms perform poorly for these super-hard scenarios.
This indicates that better exploration techniques should be used together with the value factorization methods.
A.3 Comparison to DMIX, QTRAN, QPlex, and weighted QMIX
In this section, we compare ResQ with DMIX, QTRAN, QPlex, and weighted QMIX. A.3.1 Larger DMIX
ResQ and ResZ use more parameters than DMIX with the residual functions.
By modifying the dimension of the last layer of the hypernet inside DMIX, we have increased the number of parameters of DMIX from 85K to 350K, which is bigger than that of ResQ (320K) and that of ResZ(316K).
Let’s denote DMIX with more parameters as DMIX-larger, and test its performance in the MMM2, the
MMM, and the 3s_vs_5z scenarios.
The experimental results are depicted in Figure 3.
DMIX-larger performs slightly better than DMIX in the MMM2 scenario but worse than DMIX in the MMM and the 3s_vs_5z scenarios.
ResQ and ResZ performs better than DMIX and DMIX-larger in these scenarios.
This means that the performance improvement of ResQ over DMIX does not come from the use of more parameters.
We can reformulate QTRAN in the form of ResQ. QTRAN approximates the true value function Q ( τ , u ) via Q tran ( τ , u ) . Q tran ( τ , u ) = Q tot ( τ , u ) + w r ( τ , u ) V ( τ ) , where Q tot ( τ , u ) = (cid:80) i Q i ( τ i , u i ) and w r ( τ , u )
= 1 .
And it must satisfy the inequity condition from 4b of Theoremof QTRAN, which requires that Q tran ( τ , u ) ≥ Q ( τ , u ) for all non-optimal actions.
Thus, QTRAN could over-estimate state-action value pairs.
Further, QTRAN uses MSE loss to implement the inequality conditions, which could lead to violation of the IGM principle.
ResQ learns Q jt ( τ , u ) to approximate the state-action value function Q ( τ , u ) .
Theorem 1 in
ResQ shows that the state-action value function Q jt ( τ , u ) satisfy Q jt ( τ , ¯ u ) ≥ Q jt ( τ , u ) , where ¯ u =
[ ¯ u i ] ni = ¯
u
i = arg max u i Q i ( τ i , u i ) .
Further, as it is shown in Formula 8 of Theorem 2 in ResQ, Q jt ( τ , u ) = Q ( τ , u ) ∀ u (cid:54) = ¯ u , ResQ can ﬁnd a Q jt ( τ , u ) that matches the sub-optimal state-actions of Q ( τ , u ) closely.
QTRAN cannot guarantee that the learned approximated function Q tran satisﬁes this property.
Further, ResQ uses Q r ( τ, u ) to model the residual part; it has more input (i.e., u ); it can consider the impact of actions. Thus, ResQ is more ﬂexible than the residual part V ( τ ) of QTRAN.
To study the reason why ResQ performs better than QTRAN, We use two variants of ResQ: Qtot-VDN and Qtot-VDN-MSE. Qtot-VDN uses VDN as the main function same as QTRAN. The difference between Qtot-VDN between QTran is the residual function, the mask, and the inequality conditions.
Qtot-VDN-MSE uses MSE loss to implement Q r ≤ 0 , and it uses VDN as its main function.
Their results are depicted in Figure 4.
For MMM2, Qtot-VDN can obtain a win rate of 0.6.
The win rate for QTRAN is 0.
For the 8m_vs_9m scenario, Qtot-VDN can obtain a win rate of 0.8.
And QTRAN can obtain a win rate of 0.4.
The performance gap does not come from the implementation of the main function; it comes from the residual function, the mask function, and the inequality conditions.
Further, Qtot-VDN-MSE performs similarly to Qtot-VDN, and it performs better than QTRAN.
This suggests that the performance gap between ResQ and QTRAN does not come from the implementation of inequality conditions for ResQ. A.3.3 Comparison to QPlex
ResQ can be viewed as a generalization of QPlex. According to QPlex, QPlex learns a function Q plex ( τ, u ) to approximate the state-action value function Q ( τ, u ) . Q plex ( τ, u ) =
V ( τ )+ Adv ( τ, u ) , where V ( τ ) = max u Q ( τ, u ) .
According to Formula (11) of the QPlex paper [9], we can rewrite Q plex in the form of ResQ as
[Equation]
(46) where w plexr ( τ, u ) = 1 , Q tot ( τ, u ) = (cid:80) i Q i ( τ, u i ) and Q plexr ( τ, u )
= (cid:80) i
( λ i − 1)
A i ( τ, u i ) .
Q tot is implemented as a summation of the utilities, which has limited representation abilities.
To ensure the IGM principle, QPlex places restrictions
A i ( τ, u i ) =
Q i ( τ, u i ) − max u i Q
i ( τ, u i ) among the main and the residual function.
Different from QPlex, ResQ does not require any relationships among the main and the residual functions, it only requires that Q r ≤ 0 .
In summary, ResQ places fewer restrictions on the relationships between the main and the residual function than QPlex.
And ResQ can use a more expressive neural network to model the main function than QPlex.
Thus, ResQ can model Q value function in a better and more ﬂexible way than QPlex.
A.3.4 Comparison to weighted QMIX
CW/OW QMIX can be viewed as variants of ResQ (see Formula 4 in Section 4).
CW/OW QMIX learns Q wqmix =
w tot ( τ, u ) Q tot ( τ, u ) + (1 − w tot ( τ, u ))
{+Q r ( τ, u ) to approximate the true value function, where w tot ( τ, ¯ u ) = 1 and w tot ( τ, u ) = 
+}
∀ u (cid:54) =
¯ u .
They assign high learning priorities to
Q tot , which puts the learning of sub-optimal state-action pairs and Q wqmix in trouble.
A.3.5 Inequality condition implementations
ResQ does not signiﬁcantly affect by the implementation of the inequality conditions (6b of Theorem 1).
We have evaluated the performance of a variant of ResQ, ResQ-MSE, which uses the MSE loss to implement Q r ≤ 0 .
We have studied the performance of ResQ-MSE in the MMM2, MMM, and 8m_vs_9m scenarios.
The results are depicted in Figure 5.
The performance of ResQ-MSE and ResQ are similar in these three scenarios.
This indicates that the violation of inequality conditions does not signiﬁcantly affect the performance for these two scenarios for ResQ. A.3.6 Limitations
Knowing the optimal action over Q is computationally intractable, we make approximations to derive a practical algorithm.
It is difﬁcult for ResQ to ﬁnd the optimal actions for a scenario requiring highly-coordinated agent exploration.
The approximated mask function will fail if the approximated optimal action differs signiﬁcantly from the optimal action.
In ResQ, we assume that the argmax operator of Q tot can lead to correct optimal actions.
However, this assumption does not always hold.
For a discrete-action environment with a tabular Q value, we think that all state-action values can be factorized using ResQ if the masking function can ﬁnd the correct optimal action.
As we use neural networks to represent states and actions, the approximation error of neural networks can interact with the error of the approximated argmax operator.
This may make ResQ fail.
We think that combining efﬁcient-exploration approaches (e.g., MAVEN) with ResQ can make the mask function more robust than using ResQ alone.