ResQ: A Residual Q Function-based Approach for Multi-Agent Reinforcement Learning Value Factorization
Abstract
The factorization of state-action value functions for Multi-Agent Reinforcement
Learning (MARL) is important.
Existing studies are limited by their representation capability, sample efﬁciency, and approximation error.
To address these challenges, we propose, ResQ, a MARL value function factorization method, which can ﬁnd the optimal joint policy for any state-action value function through residual functions.
ResQ masks some state-action value pairs from a joint state-action value function, which is transformed as the sum of a main function and a residual function.
ResQ can be used with mean-value and stochastic-value RL.
We theoretically show that
ResQ can satisfy both the individual global max (IGM) and the distributional IGM principle without representation limitations.
Through experiments on matrix games, the predator-prey, and StarCraft benchmarks, we show that ResQ can obtain better results than multiple expected/stochastic value factorization methods.
Introduction
Many real-world tasks involve multiple agents acting together, such as robot control [1].
Such tasks can be modelled as Multi-Agent Reinforcement Learning (MARL) problems where a group of agents must cooperate to achieve a common goal.
MARL has attracted great research interest because of its social and economic impact.
However, MARL is highly stochastic and hard to learn due to the partial-observability [2] and changing policies of agents.
To address these issues, many approaches adopt the Centralized Training with Decentralized Execution paradigm (CTDE)
In the CTDE paradigm, value factorization approaches [4] are widely adopted due to their remarkable performance and sample efﬁciency.
A joint state-action value function is factorized into per-agent utilities Q i , and each agent i acts greedily according to Q i .
To enable effective CTDE, it is critical to ensure the Individual-Global-Max (IGM) principle [5] that the optimal joint action should be equivalent to the collection of each agent’s greedy actions.
VDN [6] and QMIX [4] are two popular value factorization methods, which satisfy the IGM principle.
Most value factorization approaches focus on the deterministic state-action value functions.
Albeit
MARL is highly stochastic, the full distribution of per-agent utilities and the multi-agent system are overlooked and represented only as expected value of the full distribution.
Such distributional information could be beneﬁcial for policy learning.
DMIX and DDN [7] extend QMIX and VDN with distributional RL.
They can factorize stochastic joint state-action value pairs into stochastic per-agent utilities, and satisfy the distributional individual-global-max(DIGM) principle [7].
However, because of the representation limitations of QMIX and VDN, DMIX and DDN cannot factorize distributional value functions whose mean value is non-monotonic.
To address the representation limitations, QTRAN [5], WQMIX [8], and QPLEX [9] have been proposed.
However, QTRAN and QPLEX are sample-inefﬁcient, and WQMIX has high approximation errors to non-optimal actions.
These methods may lead to incorrect estimations of optimal values.
Moreover, they satisfy only the IGM principle.
Achieving the IGM and the DIGM principles without representation limitations remains an open challenge.
We address these challenges through, ResQ , a residual Q function-based approach.
Given a hard-tofactorize joint state-action value function Q jt (e.g., shown in Fig. 1), the key insight of our work is described as follows.
We could obtain an easy-to-factorize state-action value function, the main function Q tot , if we can mask out some state-action value pairs from Q jt .
If the main function
Q tot shares the same optimal policy with Q jt , then we can obtain the optimal policy of Q jt through the factorization of Q tot by just using some simple mixers (e.g., [6]).
After the masking, Q jt is divided into two value functions: a main function Q tot and a residual function Q r .
Q r are the masked state-action value pairs from Q jt .
Besides mean-value RL (e.g., Q jt ), the idea of ResQ can be used with distributional RL [10] as well.
ResQ can factorize stochastic joint state-action value functions Z jt into per-agent stochastic utilities
Z i with a mask and a residual function Z r .
We have shown that ResQ satisﬁes both the IGM and the
DIGM principles without representation limitations.
For evaluation, we conduct extensive experiments on one-step matrix games, the StarCraft II MARL tasks [11], and predator-prey tasks.
The experimental results show that ResQ can obtain better results than multiple competitive value factorization methods, and the ablation study shows that through the use of residual functions for multiple factorization methods, ResQ can improve the performance of these methods.
The source code of ResQ can be visited at https://github.com/xmu-rl-3dv/ResQ.
Background 2.1 Dec-POMDPs
We consider cooperative Multi-Agent Reinforcement Learning (MARL) scenarios which can be modelled as Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs) [12] deﬁned as G for n agents.
G =
(cid:104)S , {U i } ni
=
1 , P, r, {O i } ni
=1 , { σ i } ni =1 , n, γ (cid:105) .
S denotes the set of states and U i the set of actions available to agent i and we consider discrete actions only.
A joint action of all agents is deﬁned as u t ∈ U N :=
U 1 × . . .
× U n .
At a discrete time step t and state s t , after the joint action is issued, the next state s t +1 ∈ S of the environment is drawn from the transition function s t +1 ∼ P ( ·| s t , u t ) .
All the agents receive a reward r t after the state transition happens.
It is a partially observable environment that agent
i observes a part of the environment o ti ∈
O i which is drawn from o ti ∼ σ i ( ·| s t ) .
γ is the discount factor.
Each agent i maintains an action-observation history τ
i =
( O i × U i ) ∗ 1 , and acts according to policy π i ( u i | τ i ) .
We denote τ ∈ T N := τ 1 × . . .
×
τ n as the joint-observation history.
The learning objective is to ﬁnd the optimal policy π = <
π 1 , ..., π n > .
2.2 Value Function Factorization
The Individual-Global-Max ( IGM ) principle proposed in [5] is important to realize the factorization of MARL state-action value function.
It is deﬁned as follows.
Deﬁnition 1 (IGM) .
For a joint state-action value function Q jt : T N ×U N (cid:55)→ R , where τ ∈ T N is a joint action-observation history, if there exist individual state-action functions [ Q i : T i ×U i (cid:55)→ R ] Ni =1 , such that the following conditions are satisﬁed
[Equation]
then, [ Q i ] satisfy IGM for Q jt under τ .
We can state that Q jt ( τ , u ) is factorized by [ Q i ( τ i , u i )]
Ni =1 .
In this work, we assume the argmax operator is unique, the action with smallest index is selected to break ties if a tie exists
QMIX is a widely used monotonically increasing mixer function which proposes the following sufﬁcient conditions for IGM:
[Equation]
There are tasks whose joint state-action value function cannot be expressed well by monotonic increasing conditions, as shown in Fig. 1.
2.3 Distributional RL
Distributional RL models the stochastic return of state-action pair through Z ( τ , u ) explicitly.
They models full return distribution
Z ( τ , u ) instead of Q ( τ , u ) .
The distribution of return can be approximated through a categorical distribution [13] or a quantile function [10].
Implicit Quantile
Function (IQN) [10] models the stochastic value function Z ( τ, u ) as a quantile function F − 1 ( τ, u | w ) , where F − 1 is the generalized inverse cumulative distribution function (CDF), w ∈ [0 , 1] is a quantile sample.
IQN deﬁned a distributional Bellman operator, and use it to updates its Z ( τ , u ) .
After applying the distributional Bellman operator on Z ( τ , u ) , its resulting Z ( τ (cid:48) , u (cid:48) ) remains in the same distribution as Z ( τ , u ) .
The loss ρ w i ,w j t for temporal-difference error δ w i ,w j t is deﬁned as Huber quantile regression loss, where w i and w j are two quantiles.
During execution, the action with the largest expected return arg max u E [ Z ( τ , u )] is chosen.
MARL is highly stochastic, and distributional RL could be used to deal with the stochasticity of
MARL.
[7] proposes the Distributional Individual-Global-Max ( DIGM ) principle, which is deﬁned as follows.
Deﬁnition 2 (DIGM) .
Given a set of stochastic individual state-action value function [ Z i ( τ i , u i )]
N i = and a stochastic joint state-action value function Z jt ( τ , u ) , if the following conditions are satisﬁed
[Equation]
then, [ Z i ( τ i , u i )]
Ni =1 satisfy DIGM for Z jt under τ .
We can state that Z jt ( τ , u ) is distributionally factorized by [ Z i ( τ i , u i )]
Ni =1 .
Related Work
Value factorization approaches are widely adopted in MARL [14].
VDN [6] factorizes the value function Q tot as the sum of per-agents’ utility Q i .
QMIX [4] supports monotonic relationships among
Q i and Q tot .
During execution, each agent acts greedily according to Q i .
RMIX [15] learns return distribution of each agent, and integrates risk-sensitive RL with QMIX.
[7] proposes DMIX and
DDN, which extend QMIX and VDN with distributional RL through mean-shape decomposition.
They satisfy the DIGM principle but suffer from the representation limitation of QMIX and VDN.
They cannot model stochastic value functions whose mean-value are non-monotonic.
QAtten [16] and REFIL [17] adopt the attention mechanisms to focus on certain agents/scenarios when factorizing value function.
QRelation [18] considers the relationships among agents for value factorization.
However, they cannot represent non-monotonic value functions well.
To address the representational limitations, WQMIX [8] prioritizes the estimation of the optimal stateaction value.
It assigns higher weights to the optimal joint actions when minimizing approximation errors.
WQMIX can be viewed as a special version of ResQ. It pays attention to the optimal actions via masking out all the sub-optimal actions.
The Q r of WQMIX consists of all the sub-optimal stateaction pairs, and the Q tot models only the optimal ones.
However, it assigns high learning priorities to Q tot , which puts the learning of sub-optimal state-action pairs and Q jt in trouble.
Moreover, it is difﬁcult to assign proper weights to obtain satisfactory performance.
QTRAN [5] and QTRAN++ [19] transform the joint state-action value function Q jt into an easyto-factorize one through a series of linear constraints.
QTRAN learns an approximated function
Q tran ( τ , u ) =
(cid:80) Ni =
1 Q i ( τ i , u i ) + V jt ( τ ) to approximate the optimal policy of Q jt .
They can be viewed as using Q tot =
(cid:80) Ni =
1 Q i ( τ i , u i ) to model the optimal actions, and constraining Q tran ≥ Q jt for sub-optimal actions.
QTRAN use soft constraints (MSE losses) to implement IGM which could lead to the violation of IGM.
RQN [20] extends QTRAN by adding an individual correction factor for each utility function to compute an adjusted utility function.
RQN can be viewed as a special case of ResQ. It uses the sum of per-agent utility as the main function Q tot , and the sum of individual correction factors as the residual function Q r .
QPlex decomposes Q jt as the sum of a value function V tot and a non-positive advantage function
A tot .
The role of A tot is similar to the main function Q tot used in ResQ. QPlex obtains agents’ policies from the factorization of A tot , which must be ≤ 0 , but ResQ does not have such restrictions.
Although QPlex does not suffer from representation limitation issues, it may struggle to learn a good policy in complex tasks [8].
MAVEN [21] deals with the inefﬁcient exploration problem in MARL.
Inefﬁcient exploration problems could interact with the representation limitation problems.
It adopts QMIX as its value factorization method.
CDS [22] improves the effective cooperation of agents with efﬁcient exploration.
Different from them, ResQ focus on value factorization, and can be used in MAVEN and CDS.
COMNet [23], DIAL [24], and GraphComm [25] study communication among agents.
DCG [26] uses graphs for coordination.
COPA [27] and NDQ [28] factorize the value function with communication.
The communication method adopted by them could be used with ResQ to improve MARL performance.
There exist various MARL actor-critic methods, such as MADDPG [29], MAAC [30], and COMA [31].
ResQ is a value-based method.
Other methods exist.
For example, MAPPO [32] combines MARL with PPO [33].
UPDeT [34] proposes a MARL policy decoupling method.
ResQ is orthogonal to them.
Method
Motivating Example Fig.
1 shows an example of a non-monotonic payoff matrix, which is hardto-factorized.
Agent 1 has different action-value orderings, which depend on the action of Agent 2.
Moreover, the second-best state-action value 7 . 9 is only slightly lower than the optimal value 8 , which causes difﬁculties for methods with representation limitations or overestimations.
Given a hard-to-factorize state-action value function Q jt , the key insight to ResQ is that if we can mask out some state-action value pairs from Q jt , then we could obtain an easy-to-factorize value function Q tot .
As shown in Fig.
1, if we mask out those action value pairs colored in red, then the main function Q tot can be factorized easily by mixers such as QAtten or QMIX.
ResQ seeks to ﬁnd a mask w , a main functions Q tot , and a residual function Q r which satisfy the following equation.
[Equation]
where masks w tot ( τ , u ) , w r ( τ , u ) ∈ { 0 , 1 } .
The main Q tot shares the same greedy optimal policy as Q jt .
Q tot ( τ , u ) is factorized into per-agent utilities Q i , and each agent selects its action greedily with respect to Q i .
There are many possibilities to construct the masks w tot and w r , in this work, we set w tot to be constant 1 , and focus on w r .
We describe how ResQ factorizes an expected-value joint state-action value function Q jt into peragent utilities in Sec. 4.1, and how ResQ decomposes a stochastic joint state-action value function
Z jt into stochastic individual utilities in Sec.
4.1 Residual Q
For a given joint action-observation history τ , consider a factorizable joint state-action value function
Q jt ( τ , u ) that can be expressed in the form as (5).
Let ¯ u i = arg max u i Q
i ( τ i , u i ) , ¯ u =
[ ¯ u i ]
Ni =1 ,
Theorem 1 states the sufﬁcient conditions for state-action value functions [ Q i ( τ i , u i )]
Ni =1 that satisfy the IGM principle for Q jt .
Theorem 1.
A joint state-action function
[Equation]
is factorized by [ Q i ( τ i , u i )]
Ni =1 , if Q r ( τ , u ) ≤ 0 , Q tot ( τ , u ) and [ Q i ( τ i , u i )]
Ni =1 satisfy the monotonicity conditions (2), and
[Equation]
The proof of Theorem 1 is shown in the appendix.
In Theorem 1, we have shown that for any factorizable value function Q jt that can be expressed as (5), we can ﬁnd a main function Q tot that shares the same optimal policy as Q jt .
In Theorem 2, we show that for any joint state-action value function Q , we can ﬁnd a joint state-action value function Q jt that can be expressed as (5), and it has the same optimal policy as Q .
Theorem 2.
For any joint state-action function Q ( τ , u ) , we can ﬁnd Q jt ( τ , u ) =
Q tot ( τ , u ) + w r ( τ , u ) Q r ( τ , u ) that
[Equation]
Q tot ( τ , u ) monotonically increases with [ Q i ( τ i , u i )]
Ni =1 , w r ( τ , u ) satisﬁes (6), and Q r ( τ , u ) ≤ 0 .
The proof of Theorem 2 and other theorems are provided in the appendix.
From Theorem 2, for any joint state-action function Q ( τ , u ) , we can ﬁnd Q jt ( τ , u ) that shares the same optimal policy as Q .
Moreover, Q jt can be expressed as (5).
Combining Theorem 1 and 2, for any Q ( τ , u ) , we can ﬁnd
[ Q i ( τ i , u i )]
Ni =1 satisfy the IGM principle for Q ( τ , u ) through a residual function Q r ( τ , u ) ≤ 0 , a main function Q tot ( τ , u ) , and a mask function w r ( τ , u ) ∈ { 0 , 1 } .
The policies [ Q i ( τ i , u i )]
Ni =1 can be obtained through factorizing Q tot ( τ , u ) using mixers such as QMIX [4].
Furthermore, Formula (8) in Theorem 2 shows that Q jt ( τ , u ) can track Q ( τ , u ) closely ∀ u (cid:54) =
¯ u .
Thus, ResQ can model Q values precisely.
4.2 Distributional Residual Q
One promising way to deal with the MARL stochasticity is to model the return of state-action pairs as distributional values [10].
Researchers [7] have proposed a distributional MARL approach that satisﬁes the DIGM principle.
However, it cannot model stochastic joint value functions whose value-expectation are non-monotonic.
ResQ can address this issue through using Z dmix ( τ , u ) as the main function and Z r as the residual function, where Z dmix ( τ , u ) is the factorization function of
DMIX or DDN [7], and Z dmix , Z r are both stochastic value functions.
Formula (9) in Theorem ?? describes how Z jt is decomposed.
Theorem 3.
A stochastic joint state-action function
[Equation]
is factorized by [ Z i ( τ i , u i )]
Ni =1 , if Z r ( τ , u ) ≤ 0 and w r ( τ , u ) = 0 when u
= ¯ u , otherwise
[Equation] [Equation] [Equation]
[Equation]
Z mean ( τ , u ) is a monotonic increasing function with respect to Q i .
where Z mean ( τ , u ) models the mean of Z dmix .
In DMIX [7], it is implemented using QMIX, and in DDN, it is implemented as VDN.
Z shape models the variation of Z dmix .
Theorem 3 shows that
ResQ can remove the representation limitations of DMIX.
Besides freeing [7] from representation limitations, ResQ can factorize a stochastic value function
Z jt into individual stochastic utilities Z i with a linear main function Z tot , a residual function Z r , and a mask function w r .
Theorem 4 states the sufﬁcient condition for [ Z i ( τ i , u i )]
Ni =1 satify the DIGM principle.
Theorem 4.
A stochastic joint state-action function
[Equation]
is factorized by [ Z i ( τ i , u i )]
Ni =1 , if Z r ( τ , u ) ≤ 0 , Z tot ( τ , u ) =
(cid:80) Ni =
k
i Z i ( τ i , u i ) k
i ≥ 0 and w r ( τ , u )
=
when u = ¯ u , otherwise 1 , where ¯ u =
[ ¯ u i ]
Ni =1 ¯ u i = arg max u i E
[ Z i ( τ i , u i )] .
The proofs of Theorem 3 and 4 are provided in the appendix.
Theorem 4 states that ResQ can ﬁnd individual stochastic utilities satisfying the DIGM principle for Z jt .
And it does not limit the representation capacity of stochastic joint state-action function Z jt .
In (10), the main function Z tot is a positive weighted sum of Z i instead of a monotonic mixing individual utilities as in (5).
This is because not all monotonic increasing functions can be used as the main function.
Some monotonic increasing function (e.g., Z tot = (cid:80) Ni
=
Z 2 i ) could lead to incorrect estimation of the optimal actions (please refer to the Appendix for an example), they cannot satisfy the DIGM principle.
4.3 Neural Networks
For any state-action value function Q jt , if we can construct the main function Q tot , the mask w r , and the residual function Q r as (5), then we can recover the optimal policy for Q jt .
Similarly, for stochastic function Z jt , we can recover the optimal actions for Z jt through the main function Z tot , the residual function Z r and the mask as (10).
The neural network of ResQ is depicted in Fig.
Main Function For Q jt , Q tot ( τ , u ) can be any monotonically increasing function.
The most well known monotonic increasing function of per-agent utility [ Q i ]
Ni =1 are QMIX, VDN, and QAtten.
In default, ResQ uses QMIX as Q tot .
For the stochastic case, the stochastic main function Z tot ( τ , u ) can be a positive weighted sum of individual utilities
Z i ( τ i , u i ) .
Z tot ( τ , u ) =
(cid:80) Ni =
k
i Z i ( τ i , u i ) k
i ≥ 0 .
Speciﬁcally, we model k i by the attention mechanism [16].
Residual Function The residual function Q r ( τ , u ) is a feed-forward network that takes actionobservation history τ , agents’ utilities Q i , and action [ u i ]
Ni =1 as input.
Unlike Q tot , Q r is not constrained to be monotonic or positive weighted-sum of utilities, thus allowing it to model a larger class of functions.
For [ Q i ] Ni =1 , Q r is a simple feed-forward network which takes [ Q i ]
Ni =1 as inputs, and the negative absolute function is the activation function of the last neural network layer to ensure
Q r ≤ 0 .
For the stochastic case, the residual function Z r is modeled as −| (cid:80) Ni =
w i
Z i | .
[Table]
Mask Function
The mask w r ( τ , u ) requires knowing the maximal joint actions for the joint stateaction value function.
However, it is intractable to obtain the optimal actions without searching over the entire action space.
For practical implementation, we use approximation, and w r is deﬁned as follows.
[Equation]
where (cid:101) u = arg max u Q tot ( τ, u ) for the expectation case.
For the stochastic case, (cid:101) u = arg max u E
[ Z tot ( τ , u )] .
Loss For the expected-value case, the loss L of ResQ consists of two temporal-difference losses: L ∗ and L jt .
L =
L ∗ + L jt .
L ∗ =
(cid:80) bk
=
( y k − Q jt ( τ k , u k , θ )) 2 , where y k
= r + γQ jt ( τ k +1 , (cid:101) u, θ − ) , (cid:101) u
= [ (cid:101) u
i ] N i =1 , (cid:101) u i = arg max Q i ( τ k +1 i , u i ) .
θ − are the parameters of a target network.
L jt measure the distance between Q jt and Q tot + w r Q r .
It is deﬁned as L jt =
(cid:80) ( Q jt − Q tot − w r Q r ) 2 .
For the distributional case, L = L z ∗ + L zjt .
L z ∗ and L zjt are similar to L ∗ and L jt , respectively.
L z ∗ and L zjt use the pair-wise sampled temporal difference error instead of the standard temporal error, and the Huber quantile regression loss [10] rather than the mean square error loss.
Evaluation
We evaluate the performance of ResQ on one-step matrix games, the StarCraft II Multi-Agent
Challenge benchmark (SMAC) [11], and predator-prey with multiple algorithms.
The experimental results show that ResQ can obtain better results than state-of-the-art methods, and it satisﬁes both the IGM and DIGM principle together without representation limitation.
The ablation study shows that ResQ can improve the performance of multiple value factorization methods through using the residual function.
Please refer to the appendix for detailed setup and more results.
5.1 Experimental Setup
The studied algorithms are listed in Table 1.
ResZ is the distributional version of ResQ factorized according to (10).
DAtten, designed by us, integrates distributional RL [10] and the attention mechanism.
Other algorithms are conﬁgured with their default parameters, and the number of rollouts is one.
Each experiment is repeated at least 5 times with different seeds.
The conﬁguration of ResQ follows WQMIX [8].
5.2 Matrix Game
We study the representation power of multiple methods for a matrix game (depicted in Fig.
1 and
Table 2).
All the algorithms are ran through a full exploration (cid:15)
= 1 for (cid:15) -greedy conducted over 50,000 steps.
This setting guarantees the exploration of all possible actions.
Table 2 depicts the Q jt of multiple methods.
As it is observed in the table, only ResQ/ResZ, QTran, and CW QMIX can obtain the optimal policy, DMIX, DDN, and QPlex learn the second-best policy, and OW QMIX learns a wrong policy.
Albeit CW QMIX can ﬁnd the optimal action, its learned Q jt has a high approximation error, as it focuses on the optimal actions only.
The detailed factorization results are depictd in Table ?? in the appendix.
Further, we modify the matrix into a distributional matrix by adding a nomral distribution (mean = 0, std=1) value into all action-value.
The payoff matrix and the reconstructed Q values are shown in Table 3 in the appendix.
We ﬁnd that ResQ/ResZ and QTran can ﬁnd the optimal policy while most of the algorithms cannot.
These results demonstrate the ability of ResQ to factorize difﬁcult state-action value functions with low approximation errors without representation limitations.
[Table]
[Figure]
5.3 StarCraft II Multi-Agent Challenge (SMAC)
In SMAC [11], two teams of agents ﬁght against each other.
For the SMAC tasks, we train each algorithm for 1 to 2 million steps, and for every 10,000 steps, the learned policies are evaluated.
The test win rate, which measures the average win ratio of the agents controlled by MARL algorithms for each 32 test episodes, is reported.
As shown in Figure 3 (a), ResQ and ResZ achieve the best performance, and QMIX has the third-best results in the MMM2 scenario.
The performance improvement over QMIX comes from the fact that
ResQ/ResZ can model better the non-monotonic state-action function of the MMM2 scenario.
For the MMM scenario, ResQ/ResZ has the best test win rate.
The results for the 3s_vs_5z tasks are depicted in Fig.
3 (c).
ResQ can obtain close-to-optimal performance after 1 million steps, while others cannot.
For the bane_vs_bane scenario, in Fig.
3 (d),
ResQ and ResZ can learn the optimal policy in less than 0.2 million steps, while the other methods cannot.
The results indicate that through masking state-action value pair from Q jt / Z jt , ResQ/ResZ can learn the optimal policy in short time, and the performance is stable .
Albeit QPlex and DMIX can obtain close-to-optimal performance at a much later time, but their performance is unstable.
The results for CW/OW QMIX for bane_vs_bane and 27m_vs_30m are not shown due to out-of-memory error on an NVIDIA Geforce RTX 3090 GPU.
ResQ/ResZ can obtain the close-to-optimal performance after 0.25 million steps in the 2s_vs_1sc scenario, shown in Fig.
3 (e).
Although the other methods can learn close-to-optimal policy with more training steps, their performance is not stable.
For the 1c3s5z scenario, depicted in Figure 2 (f),
ResQ/ResZ are among the best-performing algorithms.
In the 2c_vs_64zg scenario, 2 Colossi agents ﬁghts against 64 enemy agents.
It has a large action space, and the reward of each action varies rather than is constant as in other scenarios.
As it is depicted in Fig.
3 (g), ResZ can obtain the best performance.
This indicates ResZ’s ability to model distributional reward.
The results for 8m_vs_9m and 27m_vs_30m are depicted in Fig.
3 (h) and (f), respectively.
ResQ and ResZ are the second-best performing algorithms.
DMIX is good at scenarios which consist of homogeneous agents (e.g., 8m_vs_9m), but it under-performs for scenarios with heterogeneous agents (e.g., MMM2, 3s_vs_5z), where agent interactions are complex.
5.4 Predator Prey
In the predator prey environment [26], 8 agents hunt 8 preys in a 10 × 10 grid world.
Each agent has 6 actions: moves in either 4 directions, stands still, and catches prey.
If two adjacent agents execute simultaneously the catch action to a nearby prey, a reward r = 10 is given to the agents, but a failed catch by a single agent is punished by a reward p ≤ 0 .
The more negative p is, the higher level of coordination is needed for the agents.
Following [8], the (cid:15) of the (cid:15) -greedy strategy for QPlex, ResQ, and CW/OW QMIX is gradually annealed from 1 to 0.05 within 1 million steps.
The test return for three punishments: p = 0 , p = − 2 , and p = − 4 are shown in Fig.
For the most easy scenario p = 0 , ResQ, QMIX, and QPlex can learn the optimal policies quickly.
The performance of CW/OW QMIX drops in the middle of training.
For the difﬁcult p = − 2 case,
ResQ can obtain the best returns.
CW QMIX and OW QMIX have the second and third best scores.
QTRAN and QMIX can not obtain any positive returns.
For p = − 4 , ResQ can obtain the best returns in the end.
These results indicate ResQ’s ability for challenging cooperation scenarios.
5.5 Ablation Study
We study the impact of different implementations for the main function and the residual function, and Starcraft versions on the SMAC benchmark.
We ﬁnd that ResQ can improve the performance of multiple value factorization approaches through the use of the residual function Q r .
Main Functions In SMAC, QMIX is used as the implementation of Q tot .
For ResQ, the results of using QMIX, QAtten, and VDN as the main function Q tot are depicted as ResQ, Qtot-QAtten,
Qtot-VDN in Fig.
5 (a) and (b).
As it is shown, for the MMM2 and 8m_vs_9m scenarios, ResQ, QtotQAtten, and Qtot-VDN perform better than QMIX, QAtten, and VDN, respectively.
This indicates that ResQ can improve the performance of value factorization approaches (the main function) through the use of residual functions and mask.
Moreover, we ﬁnd that QMIX is the preferred main function for the SMAC benchmark.
For the distributional case, shown in Fig.
5 (c).
The results of using DAtten, DDN, and DMIX as the main function Z tot are depicted as ResZ, Ztot-DDN, and Ztot-DMIX.
DAtten is designed by us; it integrates IQN [10] with QAtten.
ResZ, Ztot-DDN, and Ztot-DMIX perform better than DAtten,
DDN, and DMIX, respectively.
This indicates that ResZ can improve the performance of stochastic value factorization methods through residual functions.
Residual Functions
The results of using different mixers (factorization methods) for the residual functions are shown in the Fig.
5 (d) and (e).
The performance of using a feed-forward network (FF) as Q r (denoted as ResQ) is similar to that of using advanced mixers such as QMIX (Qr-QMIX) and VDN (Qr-VDN), and higher than that of QAtten (Qr-QAtten).
In ResQ, we do not restrict the function classes of Q r .
This suggests that using a simple feed-forward network may be sufﬁcient for Q r .
For the distributional case, the performance of using DAtten, FF, DDN, DMIX is depicted as ResZ, Zr-FF, Zr-DDN, Zr-DMIX.
As it is depicted in Fig.
5 (e), we ﬁnd that using DAtten as the residual function Z r is preferred.
The Starcraft II version used in this work is SC2.4.6.2.69232 (2.4.6 for short), which is the same version used as QMIX [4] and WQMIX [8].
Some methods (e.g.,[7]) uses the version SC2.4.10.
We compare ResQ with Pymarl2_QMIX [14], which is a ﬁne-tuned QMIX, across two versions.
Figure 5 (f) depicts the results for different methods.
The version of StarCraft is written inside the parentheses after the name of each method.
For example, ResQ (2.4.6) denotes runing ResQ against
SC2.4.6.
As it is shows that ResQ/ResZ perform better than QMIX and Pymarl2_QMIX for both two versions in the MMM2 scenario.
As Pymarl2_QMIX performs better than most algorithms in most case for SC2.4.10 [14], this suggests that ResQ/ResZ may perform better than other algorithms in
SC2.4.10.
Conclusion
In this work, we propose, ResQ, a residual function-based approach for Multi-Agent Reinforcement
Learning (MARL) value function factorization.
ResQ recovers the optimal policy for any joint state-action value function by masking out state-action value pairs from the value function.
We show that ResQ can satisfy the individual-global-max (IGM) principle and the distributional IGM principle without representation limitations.
Through extensive experiments on multiple MARL tasks, we show that ResQ can obtain promising results.
Acknowledgement This work was partially supported by the National Natural Science Foundation of China (61872376, 61972409), by open fund of PDL (WDZC20215250113), by the China Postdoctoral Science Foundation (No.2021M690094); the FuXiaQuan National Independent Innovation
Demonstration Zone Collaborative Innovation Platform (No.3502ZCQXT2021003).
We would like thank the anonymous reviewers for their valuable comments.
[-Theorem 4 shows that if Z r ( τ, u ) ≤ 0 , Z tot ( τ, u ) = (cid:80) Ni =
-]
[-= 
-]
[-Clearly E Z 1 ( s, a ) = 
-]
[-Q r ( τ, u ) to approximate the true value function, where w tot ( τ, ¯ u ) = 1 and w tot ( τ, u ) = 
-]