ResQ: A Residual Q Function-based Approach for Multi-Agent Reinforcement Learning Value Factorization
Abstract
The factorization of state-action value functions for Multi-Agent Reinforcement
Learning (MARL) is important.
Existing studies are limited by their representationcapability, sample efﬁciency, and approximation error.
To address these challenges,we propose, ResQ, a MARL value function factorization method, which can ﬁnd theoptimal joint policy for any state-action value function through residual functions.
ResQ masks some state-action value pairs from a joint state-action value function,which is transformed as the sum of a main function and a residual function.
ResQcan be used with mean-value and stochastic-value RL.
We theoretically show that
ResQ can satisfy both the individual global max (IGM) and the distributional IGMprinciple without representation limitations.
Through experiments on matrix games,the predator-prey, and StarCraft benchmarks, we show that ResQ can obtain betterresults than multiple expected/stochastic value factorization methods.
Introduction
Many real-world tasks involve multiple agents acting together, such as robot control [1].
Such taskscan be modelled as Multi-Agent Reinforcement Learning (MARL) problems where a group of agentsmust cooperate to achieve a common goal.
MARL has attracted great research interest because ofits social and economic impact.
However, MARL is highly stochastic and hard to learn due to thepartial-observability [2] and changing policies of agents.
To address these issues, many approachesadopt the Centralized Training with Decentralized Execution paradigm (CTDE)
In the CTDE paradigm, value factorization approaches [4] are widely adopted due to their remarkableperformance and sample efﬁciency.
A joint state-action value function is factorized into per-agentutilities Q i and each agent i acts greedily according to Q i .
To enable effective CTDE, it is criticalto ensure the Individual-Global-Max (IGM) principle [5] that the optimal joint action should beequivalent to the collection of each agent’s greedy actions.
VDN [6] and QMIX [4] are two popularvalue factorization methods, which satisfy the IGM principle.
Most value factorization approaches focus on the deterministic state-action value functions.
Albeit
MARL is highly stochastic, the full distribution of per-agent utilities and the multi-agent systemare overlooked and represented only as expected value of the full distribution.
Such distributionalinformation could be beneﬁcial for policy learning.
DMIX and DDN [7] extend QMIX and VDN withdistributional RL.
They can factorize stochastic joint state-action value pairs into stochastic per-agentutilities, and satisfy the distributional individual-global-max(DIGM) principle [7].
However, becauseof the representation limitations of QMIX and VDN, DMIX and DDN cannot factorize distributionalvalue functions whose mean value is non-monotonic.
To address the representation limitations, QTRAN [5], WQMIX [8], and QPLEX [9] have been proposed.
However, QTRAN and QPLEX are sample-inefﬁcient, and WQMIX has high approximationerrors to non-optimal actions.
These methods may lead to incorrect estimations of optimal values.
Moreover, they satisfy only the IGM principle.
Achieving the IGM and the DIGM principles withoutrepresentation limitations remains an open challenge.
We address these challenges through, ResQ , a residual Q function-based approach.
Given a hard-tofactorize joint state-action value function Q jt (e.g., shown in Fig. 1), the key insight of our workis described as follows.
We could obtain an easy-to-factorize state-action value function, the mainfunction Q tot , if we can mask out some state-action value pairs from Q jt .
If the main function
Q tot shares the same optimal policy with Q jt , then we can obtain the optimal policy of Q jt throughthe factorization of Q tot by just using some simple mixers (e.g., [6]).
After the masking, Q jt isdivided into two value functions: a main function Q tot and a residual function Q r .
Q r are the maskedstate-action value pairs from Q jt .
Besides mean-value RL (e.g., Q jt ), the idea of ResQ can be used with distributional RL [10] as well.
ResQ can factorize stochastic joint state-action value functions Z jt into per-agent stochastic utilities
Z i with a mask and a residual function Z r .
We have shown that ResQ satisﬁes both the IGM and the DIGM principles without representation limitations.
For evaluation, we conduct extensive experiments on one-step matrix games, the StarCraft II MARLtasks [11], and predator-prey tasks.
The experimental results show that ResQ can obtain better resultsthan multiple competitive value factorization methods, and the ablation study shows that through theuse of residual functions for multiple factorization methods, ResQ can improve the performance ofthese methods.
Background2.1 Dec-POMDPs
We consider cooperative Multi-Agent Reinforcement Learning (MARL) scenarios which can bemodelled as Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs) [12]deﬁned as G for n agents.
G =
(cid:104)S , {U i } ni
=
1 , P, r, {O i } ni
=1 , { σ i } ni =1 , n, γ (cid:105) .
S denotes the set ofstates and U i the set of actions available to agent i and we consider discrete actions only.
A jointaction of all agents is deﬁned as u t ∈ U N :=
U 1 × . . .
× U n .
At a discrete time step t and states t , after the joint action is issued, the next state s t +1 ∈ S of the environment is drawn from thetransition function s t +1 ∼ P ( ·| s t , u t ) .
All the agents receive a reward r t after the state transitionhappens.
It is a partially observable environment that agent
i observes a part of the environmento ti ∈
O i which is drawn from o ti ∼ σ i ( ·| s t ) .
γ is the discount factor.
Each agent i maintains anaction-observation history
τ i
=
( O i × U i ) ∗ 1 , and acts according to policy π i ( u i | τ i ) .
We denoteτ ∈ T N := τ 1 × . . .
×
τ n as the joint-observation history.
The learning objective is to ﬁnd theoptimal policy π = <
π 1 , ..., π n > .
2.2 Value Function Factorization
The Individual-Global-Max ( IGM ) principle proposed in [5] is important to realize the factorizationof MARL state-action value function.
It is deﬁned as follows.
Deﬁnition 1 (IGM) .
For a joint state-action value function Q jt : T N ×U N (cid:55)→ R , where τ ∈ T N is ajoint action-observation history, if there exist individual state-action functions [ Q i : T i ×U i (cid:55)→ R ] Ni =1 , such that the following conditions are satisﬁed
[Equation]
then, [ Q i ] satisfy IGM for Q jt under τ .
We can state that Q jt ( τ , u ) is factorized by [ Q i ( τ i , u i )]
Ni =1 .
In this work, we assume the argmax operator is unique, the action with smallest index is selected tobreak ties if a tie exists
QMIX is a widely used monotonically increasing mixer function which proposes the followingsufﬁcient conditions for IGM:
[Equation]
There are tasks whose joint state-action value function cannot be expressed well by monotonicincreasing conditions, as shown in Fig. 1.
2.3 Distributional RL
Distributional RL models the stochastic return of state-action pair through Z ( τ , u ) explicitly, ratherthan estimating a single scalar Q-value.
The distribution of return can be approximated througha categorical distribution [13] or a quantile function [10].
Implicit Quantile Function (IQN) [10]models the stochastic value function Z ( τ, u ) as a quantile function F − 1 ( τ, u | w ) , where F − 1 is thegeneralized inverse cumulative distribution function (CDF), w ∈ [0 , 1] is a quantile sample.
Distributional RL models full return distribution
Z ( τ , u ) instead of Q ( τ , u ) .
IQN deﬁned a distributional
Bellman operator, and use it to updates its Z ( s, u ) .
After applying the distributional Bellman operatoron Z ( τ , u ) , its resulting Z ( s (cid:48) , u (cid:48) ) remains in the same distribution as Z ( s, u ) .
The loss ρ w i ,w j t fortemporal-difference error δ w i ,w j t is deﬁned as Huber quantile regression loss, where w i and w j aretwo quantiles.
During execution, the action with the largest expected return arg max u E [ Z ( τ, u )] ischosen.
MARL is highly stochastic, and distributional RL could be used to deal with the stochasticity of
MARL.
[7] proposes the Distributional Individual-Global-Max ( DIGM ) principle, which is deﬁnedas follows.
Deﬁnition 2 (DIGM) .
Given a set of stochastic individual state-action value function [ Z i ( τ i , u i )]
Ni =1and a stochastic joint state-action value function Z jt ( τ , u ) , if the following conditions are satisﬁed
[Equation]
then, [ Z i ] satisfy DIGM for Z jt under τ .
We can state that Z jt ( τ , u ) is distributionally factorized by[ Z i ( τ i , u i )]
Ni =1 .
Related Work
Value factorization approaches are widely adopted in MARL [14].
VDN [6] factorizes the valuefunction Q tot as the sum of per-agents’ utility Q i .
QMIX [4] supports monotonic relationships among
Q i and Q tot .
During execution, each agent acts greedily according to Q i .
RMIX [15] learns returndistribution of each agent, and integrates risk-sensitive RL with QMIX.
[7] proposes DMIX and
DDN, which extend QMIX and VDN with distributional RL through mean-shape decomposition.
They satisfy the DIGM principle but suffer from the representation limitation of QMIX and VDN.
They cannot model stochastic value functions whose mean-value are non-monotonic.
QAtten [16]and REFIL [17] adopt the attention mechanisms to focus on certain agents/scenarios when factorizingvalue function.
However, they cannot represent non-monotonic value functions well.
To address the representational limitations, WQMIX [8] prioritizes the estimation of the optimal stateaction value.
It assigns higher weights to the optimal joint actions when minimizing approximationerrors.
WQMIX can be viewed as a special version of ResQ. It pays attention to the optimal actionsvia masking out all the sub-optimal actions.
The Q r of WQMIX consists of all the sub-optimal stateaction pairs, and the Q tot models only the optimal ones.
However, it assigns high learning prioritiesto Q tot , which puts the learning of sub-optimal state-action pairs and Q jt in trouble.
Moreover, it isdifﬁcult to assign proper weights to obtain satisfactory performance.
QTRAN [5] and QTRAN++ [18] transform the joint state-action value function Q jt into an easyto-factorize one through a series of linear constraints. QTran learns an approximated function Q tran ( τ , u ) =
(cid:80) Ni = 1 Q i ( τ i , u i ) + V jt ( τ ) to approximate the optimal policy of Q jt . They can beviewed as using Q tot =
(cid:80) Ni = 1 Q i ( τ i , u i ) to model the optimal actions, and constraining Q tran ≥ Q jtfor sub-optimal actions.
QPlex decomposes Q jt as the sum of a value function V tot and a non-positive advantage function
A tot .
The role of A tot is similar to the main function Q tot used in ResQ. QPLEX obtains agents’policies from the factorization of A tot , which must be ≤ 0 , but ResQ does not have such restrictions.
Although QPLEX does not suffer from representation limitation issues, it may struggle to learn agood policy in complex tasks [8].
The concurrent work, Residual-Q-Network (RQN) [19], deals with MARL value factorization too.
It extends QTran by adding an individual correction factor for each utility function to compute anadjusted utility function (individual Q value).
And the sum of the adjust utility function is used as thetransformed Q value Q tran which is deﬁned in QTran.
RQN can be viewed as a special case of ResQ.
It uses the sum of per-agent utility as the main function Q tot , and the sum of individual correctionfactors as the residual function Q r .
MAVEN [20] deals with the inefﬁcient exploration problem in MARL.
Inefﬁcient explorationproblems could interact with the representation limitation problems.
As it is stated in the MAVENpaper, this may push the algorithm towards sub-optimal policies.
In MAVEN, value-based agentscondition their behaviour based on a shared latent variable.
And the latent variable is controlled by ahierarchical policy.
MAVEN relies on the underlying value factorization method to decompose Qvalues.
And it adopts QMIX as its value factorization method.
ResQ can be used in MAVEN as itsfactorization method.
We will compare ResQ with MAVEN in the related work in the new version ofthis work.
COMNet [21] and DIAL [22] study communication among agents.
DCG [23] uses graphs forcoordination.
The communication method adopted by them could be used with ResQ to improve
MARL performance.
COPA [24] and NDQ [25] factorize the value function with communication.
CDS [26] improves the effective cooperation of agents by decomposing value functions.
Differentfrom them, ResQ focus on value factorization only.
There exist various MARL actor-critic methods,such as MADDPG [27], MAAC [28], and COMA [29].
ResQ is a value-based method.
Othermethods exist.
For example, MAPPO [30] combines MARL with PPO [31].
UPDeT [32] proposes a
MARL policy decoupling method.
ResQ is orthogonal to them.
Method
Motivating Example Fig.
1 shows an example of a non-monotonic payoff matrix, which is hard-tofactorized.
Agent 1 has three different action-value orderings, which depend on the action of Agent2.
Moreover, the second-best state-action value 7 . 9 is only slightly lower than the optimal value 8 ,which causes difﬁculties for methods with representation limitations.
Given a hard-to-factorize state-action value function Q jt , the key insight to ResQ is that if we canmask out some state-action value pairs from Q jt , then we could obtain an easy-to-factorize valuefunction Q tot .
As shown in Fig.
1, if we mask out those action value pairs colored in red, then themain function Q tot can be factorized easily by mixers such as QAtten or QMIX.
ResQ seeks to ﬁnd amask w , a main functions Q tot , and a residual function Q r which satisfy the following equation.
[Equation]
where masks w tot ( τ , u ) , w r ( τ , u ) ∈ { 0 , 1 } .
The main Q tot shares the same greedy optimal policyas Q jt .
Q tot ( τ , u ) is factorized into per-agent utilities Q i , and each agent selects its action greedilywith respect to Q i .
There are many possibilities to construct the masks w tot and w r , in this work, weset w tot to be constant 1 , and focus on w r .
We describe how ResQ factorizes an expected-value joint state-action value function Q jt into peragent utilities in Sec. 4.1, and how ResQ decomposes a stochastic joint state-action value function
Z jt into stochastic individual utilities in Sec.
4.1 Residual Q
For a given join action-observation history τ , consider a factorizable joint state-action value function
Q jt ( τ , u ) that can be expressed in the form as (5).
Let ¯ u i = arg max u i Q
i ( τ i , u i ) , ¯ u =
[ ¯ u i ]
Ni =1 ,
Theorem 1 states the sufﬁcient conditions for state-action value functions [ Q i ( τ i , u i )]
Ni =1 that satisfythe IGM principle for Q jt .
Theorem 1.
A joint state-action function
[Equation]
is factorized by [ Q i ( τ i , u i )]
Ni =1 , if Q r ( τ , u ) ≤ 0 , Q tot ( τ , u ) and [ Q i ( τ i , u i )]
Ni =1 satisfy the monotonicity conditions (2), and
[Equation]
The proof of Theorem 1 is shown in the appendix.
In Theorem 1, we have shown that for anyfactorizable value function Q jt that can be expressed as (5), we can ﬁnd a main function Q tot thatshares the same optimal policy as Q jt .
In Theorem 2, we show that for any joint state-action valuefunction Q , we can ﬁnd a joint state-action value function Q jt that can be expressed as (5), and it hasthe same optimal policy as Q .
Theorem 2.
For any joint state-action function Q ( τ , u ) , we can ﬁnd Q jt ( τ , u ) =
Q tot ( τ , u ) +w r ( τ , u ) Q r ( τ , u ) that
[Equation]
[Equation]
vebaru
[Equation]
[Equation]
Q tot ( τ , u ) monotonically increases with [ Q i ( τ , u )]
Ni =1 , w r ( τ , u ) satisﬁes (6), and Q r ( τ , u ) ≤ 0 .
The proof of Theorem 2 is provided in the appendix.
From Theorem 2, for any joint state-actionfunction Q ( τ , u ) , we can ﬁnd Q jt ( τ , u ) that shares the same optimal policy as Q .
Moreover, Q jtcan be expressed as (5).
Combining Theorem 1 and 2, for any Q ( τ , u ) , we can ﬁnd
[ Q i ( τ i , u i )]
Ni =1satisfy
the IGM principle for Q ( τ , u ) through a residual function Q r ( τ , u ) ≤ 0 , a main function
Q tot ( τ , u ) , and a mask function w r ( τ , u ) ∈ { 0 , 1 } .
The policies [ Q i ( τ i , u i )]
Ni =1 can be obtainedthrough factorizing Q tot ( τ , u ) .
Furthermore, Theorem 2 shows that Q jt ( τ , u ) can track Q ( τ , u )closely ∀ u (cid:54) =
¯ u .
Thus, ResQ could have low approximation errors.
4.2 Distributional Residual Q
One promising way to deal with the MARL stochasticity is to model the return of state-action pairsas distributional values [10].
Researchers [7] have proposed a distributional MARL approach thatsatisﬁes the DIGM principle.
However, it cannot model stochastic joint value functions whosevalue-expectation are non-monotonic.
ResQ can address this issue through using Z dmix ( τ , u ) as themain function and Z r as the residual function, where Z dmix ( τ , u ) is the factorization function of
DMIX or DDN [7].
Z dimx and Z r are both stochastic value functions.
Formula (11) in Theorem 3describes how Z jt is decomposed.
Theorem 3.
A stochastic joint state-action function
Z jt ( τ , u ) =
Z dmix ( τ , u ) + w r ( τ , u ) Z r
[Equation]
, u ) (11) is factorized by [ Z i ( τ i , u i )]
Ni =1 , if Z r ( τ , u ) ≤ 0 and w r ( τ , u ) = 0 when u
= ¯ u , otherwise1 .
¯ u
i = arg max u i E
[ Z i ( τ i , u i )] , ¯ u =
[ ¯ u i ]
Ni =1 , Z dmix ( τ , u )
=
Z mean ( τ , u ) + Z shape ( τ , u ) , 201 E [ Z shape ( τ , u )]
= 0 , Q i =
E
[ Z i ( τ i , u i )] .
Z mean ( τ , u ) is a monotonically increasing function withrespect to Q i or the sum of [ Q i ] N 1 .
Besides freeing [7] from representation limitations, ResQ can factorize Z jt into individual stochasticutilities Z i with a linear main function Z tot , a residual function Z r , and a mask function w r .
Theorem 4 states the sufﬁcient condition for [ Z i ( τ i , u i )]
Ni =1 satify the DIGM principle.
Theorem 4.
A stochastic joint state-action function
[Equation]
is factorized by [ Z i ( τ i , u i )]
Ni =1 , if Z r ( τ , u ) ≤ 0 , Z tot ( τ , u ) =
(cid:80) Ni =
k
i Z i ( τ i , u i ) k
i ≥ 0 andw r ( τ , u )
=
when u = ¯ u , otherwise 1 .
¯ u
i = arg max u i E
[ Z i ( τ i , u i )] , ¯ u =
[ ¯ u i ]
Ni =1 .
The proofs of Theorem 3 and 4 are provided in the appendix.
Theorem 4 states that ResQ canﬁnd individual stochastic utilities satisfying the DIGM principle for Z jt .
And it does not limit therepresentation capacity of stochastic joint state-action function Z jt .
In (12), the main function Z tot is a positive weighted sum of Z i instead of a monotonic mixingindividual utilities as in (5).
This is because not all monotonic increasing functions can be used as themain function.
Some monotonic increasing function (e.g., Z tot = (cid:80) Ni
=
Z 2 i ) could lead to incorrectestimation of the optimal actions (please refer to the Appendix for an example).
4.3 Neural Networks
For any state-action value function Q jt , if we can construct the main function Q tot , the mask w r ,and the residual function Q r as (5), then we can recover the optimal policy for Q jt .
Similarly, forstochastic function Z jt , we can recover the optimal actions for Z jt through the main function Z tot ,the residual function Z r and the mask as (12).
The neural network of ResQ is depicted in Fig.
Main Function For Q jt , Q tot ( τ , u ) can be any monotonically increasing function.
The most wellknown monotonically increasing mixers of per-agent utility [ Q i ]
Ni =1 are QMIX, VDN, and QAtten.
In default, we use QMIX as the main function Q tot .
For the stochastic case, the stochastic mainfunction Z tot ( τ , u ) can be a positive weighted sum of individual utilities
Z i ( τ i , u i ) .
Z tot ( τ , u ) = 225 (cid:80) N i =1 k
i
Z
i
τ i , u i )
k i ≥ 0 .
Speciﬁcally, we model k i by the attention mechanism [16].
Residual Function The residual function Q r ( τ , u ) is a feed-forward network that takes actionobservation history τ , agents’ utilities Q i , and action [ u i ]
Ni =1 as input.
Unlike Q tot , Q r is notconstrained to be monotonic or positive weighted-sum of utilities, thus allowing it to model a largerclass of functions.
For [ Q i ] Ni =1 , Q r is a simple feed-forward network which takes [ Q i ]
Ni =1 as inputs,and the negative absolute function is the activation function of the last neural network layer to ensure
Q r ≤ 0 .
For the stochastic case, the residual function Z r is modeled as −| (cid:80) Ni =
w i
Z i | .
Mask Function
The mask w r ( τ , u ) requires knowing the maximal joint actions for the joint stateaction value function.
However, it is intractable to obtain the optimal actions without searching over the entire action space.
For practical implementation, we use approximation, and w r is deﬁned as follows.
[Equation]
where (cid:101) u = arg max u Q tot ( τ, u ) for the expectation case.
For the stochastic case, (cid:101) u =arg max
[Equation]
[Equation] [Equation] Loss For the expected-value case, the loss L of ResQ consists of two temporal-difference losses: L ∗and L jt .
L =
L ∗ + L jt .
L ∗ =
(cid:80) bk
=
( y k − Q jt ( τ k , u k , θ )) 2 , where y k
= r + γQ jt ( τ k +1 , (cid:101) u, θ − ) ,(cid:101) u
= [ (cid:101) u
i ]
Ni =1 , (cid:101) u i = arg max Q
i ( τ k +1 i , u i ) .
θ − are the parameters of a target network.
L jt measurethe distance between Q jt and Q tot + w r Q r .
It is deﬁned as L jt =
(cid:80) ( Q jt − Q tot − w r Q r ) 2 .
Forthe distributional case,
L =
L z ∗ + L zjt .
L z ∗ and L zjt are similar to L ∗ and L jt , respectively.
L z ∗and L zjt use the pair-wise sampled temporal difference error instead of the standard temporal error,and the Huber quantile regression loss [10] rather than the mean square error loss.
Evaluation
We evaluate the performance of ResQ on one-step matrix games, the StarCraft II Multi-Agent
Challenge benchmark (SMAC) [11], and predator-prey with multiple algorithms.
The experimentalresults show that ResQ can obtain better results than state-of-the-art methods, and it satisﬁes boththe IGM and DIGM principle together without representation limitation.
The ablation study showsthat ResQ can improve the performance of multiple value factorization methods through using theresidual function.
Please refer to the supplement materials for detailed setup and more results.
5.1 Experimental Setup
The studied algorithms are listed in Table 1.
ResZ is the distributional version of ResQ factorizedaccording to (12).
DAtten, designed by us, integrates distributional RL [10] and the attentionmechanism.
Other algorithms are conﬁgured with their default parameters, and the number of rolloutsis one.
Each experiment is repeated at least 5 times with different seeds.
The conﬁguration of ResQfollows WQMIX [8].
[Table]
5.2 Matrix Game
We study the representation power of multiple methods for a matrix game (depicted in Fig.
1 and
Table 2).
All the algorithms are ran through a full exploration (cid:15)
= 1 for (cid:15) -greedy conducted over50,000 steps.
This setting guarantees the exploration of all possible actions.
As it is observed in Table 2, only ResQ/ResZ, QTran, and CW QMIX can obtain the optimal policy,
DMIX, DDN, and QPlex learn the second-best policy, and OW QMIX learns a wrong policy.
Albeit
CW QMIX can ﬁnd the optimal action, its learned Q jt has a high approximation error, as it focuseson the optimal actions only.
The detail factorization results are depictd in Table 3.
Further, we modifythe matrix into a distributional matrix by adding a nomral distribution (mean = 0, std=1) value into allaction-value.
ResQ/ResZ and QTran can ﬁnd the optimal policy while most of the algorithms cannot.
Detailed results are shown in the appendix.
These results demonstrate the ability of ResQ to factorizedifﬁcult state-action value functions with low approximation errors and high sample efﬁciency.
5.3 StarCraft II Multi-Agent Challenge (SMAC)
In SMAC [11], two teams of agents ﬁght against each other.
For the SMAC tasks, we train eachalgorithm for 1 to 2 million steps, and for every 10,000 steps, the learned policies are evaluated.
Thetest win rate, which measures the average win ratio of the agents controlled by MARL algorithms foreach 32 test episodes, is reported.
As shown in Figure 3 (a), ResQ and ResZ achieve the best performance, and QMIX has the third-bestresults in the MMM2 scenario.
The performance improvement over QMIX comes from the fact that
ResQ/ResZ can model better the non-monotonic state-action function of the MMM2 scenario.
Forthe MMM scenario, ResQ/ResZ has the best test win rate.
The results for the 3s_vs_5z tasks are depicted in Fig.
3 (c).
ResQ can obtain close-to-optimalperformance after 1 million steps, while others cannot.
For the bane_vs_bane scenario, in Fig.
3 (d),
[Table] [Table] ResQ and ResZ can learn the optimal policy in less than 0.2 million steps, while the other methodscannot.
The results indicate that through masking state-action value pair from Q jt / Z jt , ResQ/ResZcan learn the optimal policy in short time, and the performance is stable .
Albeit QPlex and DIMXcan obtain close-to-optimal performance at a much later time, but their performance is unstable.
Theresults for CW/OW QMIX for bane_vs_bane and 27m_vs_30m are not shown due to out-of-memoryerror on an NVIDIA Geforce RTX 3090 GPU.
ResQ/ResZ can obtain the close-to-optimal performance after 0.25 million steps in the 2s_vs_1scscenario, shown in Fig.
3 (e).
Although the other methods can learn close-to-optimal policy withmore training steps, their performance is not stable.
For the 1c3s5z scenario, depicted in Figure 2 (f),
ResQ/ResZ are among the best-performing algorithms.
In the 2c_vs_64zg scenario, 2 Colossi agents ﬁghts against 64 enemy agents.
It has a large actionspace, and the reward of each action varies rather than is constant as in other scenarios.
As it isdepicted in Fig.
3 (g), ResZ can obtain the best performance.
This indicates ResZ’s ability to modeldistributional reward.
The results for 8m_vs_9m and 27m_vs_30m are depicted in Fig.
3 (h) and (f), respectively.
ResQ and ResZ are the second-best performing algorithms.
DMIX is good at scenarioswhich consist of homogeneous agents (e.g., 8m_vs_9m), but it under-performs for scenarios withheterogeneous agents (e.g., MMM2, 3s_vs_5z), where agent interactions are complex.
5.4 Predator Prey
In the predator prey environment [23], 8 agents hunt 8 preys in a 10 × 10 grid world.
Each agent has6 actions: moves in either 4 directions, stands still, and catches prey.
If two adjacent agents executesimultaneously the catch action to a nearby prey, a reward r = 10 is given to the agents, but a failedcatch by a single agent is punished by a reward p ≤ 0 .
The more negative p is, the higher level ofcoordination is needed for the agents.
Following [8], the (cid:15) of the (cid:15) -greedy strategy for QPlex, ResQ,and CW/OW QMIX is gradually annealed from 1 to 0.05 within 1 million steps.
The test return for three punishments: p = 0 , p = − 2 , and p = − 4 are shown in Fig.
For themost easy scenario p = 0 , ResQ, QMIX, and QPlex can learn the optimal policies quickly.
Theperformance of CW/OW QMIX drops in the middle of training.
For the difﬁcult p = − 2 case, ResQcan obtain the best returns.
CW QMIX and OW QMIX have the second and third best scores.
QTranand QMIX can not obtain any positive returns.
For p = − 4 , ResQ can obtain the best returns in theend.
These results indicate ResQ’s ability for challenging cooperation scenarios.
5.5 Ablation Study
We study the impact of different implementations for the main function and the residual function,and Starcraft versions on the SMAC benchmark.
We ﬁnd that ResQ can improve the performance ofmultiple value factorization approaches through the use of the residual function Q r .
Main Functions For ResQ, the results of using QMIX, QAtten, and VDN as the main function Q totare depicted as ResQ, Qtot-QAtten, Qtot-VDN in Fig.
5 (a) and (b).
As it is shown, for the MMM2and 8m_vs_9m scenarios, ResQ, Qtot-QAtten, and Qtot-VDN perform better than QMIX, QAtten,and VDN, respectively.
This indicates that ResQ can improve the performance of value factorizationapproaches (the main function) through the use of residual functions and mask.
Moreover, we ﬁndthat QMIX is the preferred main function.
For the distributional case, shown in Fig.
5 (c).
The results of using DAtten, DDN, and DMIX asthe main function Z tot are depicted as ResZ, Ztot-DDN, and Ztot-DMIX.
DAtten is designed by us;it integrates IQN [10] with QAtten.
ResZ, Ztot-DDN, and Ztot-DMIX perform better than DAtten,
DDN, and DMIX, respectively.
This indicates that ResZ can improve the performance of stochasticvalue factorization methods through residual functions.
Residual Functions
The results of using different mixers (factorization methods) for the residualfunctions are shown in the Fig.
5 (d) and (e).
The performance of using a feed-forward network (FF)as Q r (denoted as ResQ) is similar to that of using advanced mixers such as QMIX (Qr-QMIX) and
VDN (Qr-VDN), and higher than that of QAtten (Qr-QAtten).
In ResQ, we do not restrict the functionclasses of Q r .
This suggests that using FF may be sufﬁcient for Q r .
For the distributional case, theperformance of using DAtten, FF, DDN, DMIX is depicted as ResZ, Zr-FF, Zr-DDN, Zr-DMIX.
Weﬁnd that using DAtten as the residual function Z r is preferred.
The Starcraft II version used in this work is SC2.4.6, which is the same version used as QMIXand WQMIX.
Some methods (e.g.,[7]) uses the version SC2.4.10.
We compare ResQ with Pymarl2_QMIX [14], which is a ﬁne-tuned QMIX, across two versions.
Figure 5 (f) shows that
ResQ/ResZ perform better than QMIX and Pymarl2_QMIX for both two versions in the MMM2scenario.
As Pymarl2_QMIX performs better than most algorithms in most case for SC2.4.10 [14],this suggests that ResQ/ResZ may perform better than other algorithms in SC2.4.10.
Conclusion
In this work, we propose, ResQ, a residual function-based approach for Multi-Agent Reinforcement
Learning (MARL) value function factorization.
ResQ recovers the optimal policy for any jointstate-action value function by masking out state-action value pairs from the value function.
We showthat ResQ can satisfy the individual-global-max (IGM) principle and the distributional IGM principlewithout representation limitations.
Through extensive experiments on multiple MARL tasks, weshow that ResQ can obtain promising results.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] The complete proofs are included in the appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] in the
supplemental material
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] The standard errors are shown.
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Each experiment is ran with aNVIDIA 3090 GPU.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We use some of the
code from Pymarl, Pymarl2, and DFAC
(b) Did you mention the license of the assets?
[Yes] It is source code released by the authors of papers.
Their code licensed under the Apache License v2.0.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]The source code of this work is included in the supplemental material, and it will be
open-sourced.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Appendix
Thanks for your time and interest to read this supplementary document.
In Section A.1, we providethe ResQ theorems and proofs.
Section A.2 describes the experimental setup in detail, shows anotherpayoff matrix that cannot represent well by state-of-the-art algorithms, and depicts more results aboutthe SMAC benchmark.
The source code of ResQ is included in the supplementary materials.
A.1 Theorems
In this section, we show the proof of all the Theorems of this work.
Theorem 1.
A joint state-action function
[Equation]
is factorized by [ Q i ( τ i , u i )]
Ni =1 , if Q r ( τ, u ) ≤ 0 , Q tot ( τ, u ) and [ Q i ( τ i , u i )]
Ni =1 satisfy the monotonicity conditions (2), and
[Equation]
Proof.
Theorem 1 shows that if condition (2) holds and Q r ( τ, u ) ≤ 0 , then [ Q i ( τ i , u i )]
Ni =1 satisfy the
IGM principle for Q jt .
We will show that arg max u Q jt ( τ, u ) =
¯ u , where ¯ u
i
=
arg max u i Q
i ( τ i , u i ) and ¯ u =
[¯ u i ]
Ni =1 .
[Equation]
2a
[Equation] (5) comes from w r ( τ, u )
= 1 ∀ u (cid:54) =
¯ u and Q r ( τ, u ) ≤ 0 .
(3) to (6) mean that ¯ u
=
[¯ u i ] Ni =1 maximizes
Q jt ( τ, u ) .
Thus [ Q i ( τ i , u i )]
Ni =1 satisﬁes IGM for Q jt ( τ, u ) .
Lemma 1.
For any joint state-action function Q ( τ, u ) , we can ﬁnd a Q tot that satisfy the monotonicityconditions, and they share the same optimal policy.
Proof.
Theorem 1 of [8] shows that Weighted QMIX can always ﬁnd a monotonically increasingfunction Q tot that shares the same optimal policy as the true value function of Q ( s, u ) .
This indicatesthat for any Q ( s, u ) , there exists a monotonically increasing function Q tot ( s, u ) that shares the sameoptimal policy as Q ( s, u ) .
Theorem 2.
For any joint state-action function Q ( τ, u ) , we can ﬁnd Q jt ( τ, u ) =
Q tot ( τ, u ) +w r ( τ, u ) Q r ( τ, u ) that
[Equation]
[Equation] Q tot ( τ, u ) is a monotonic increasing function with respect to [ Q i ( τ, u )]
Ni =1 , w r ( τ, u ) satisﬁes thecondition (2), and Q r ( τ, u ) ≤ 0 .
Proof.
Lemma (1) shows that for any joint state-action function Q ( τ, u ) , we can ﬁnd a Q tot thatsatisfy the monotonicity conditions and ¯ u = arg max Q tot ( τ, u ) =
arg max Q ( τ, u ) .
Let
[Equation] [Equation]
Obviously, the new function Q (cid:48) tot ( τ, u ) satisfy the monotonic conditions, and arg max u Q ( τ, u ) =arg max u Q (cid:48) tot ( τ, u ) .
Deﬁne the residual Q function as
[Equation]
Then
[Equation]
[Equation]
[Equation]
We can write Q ( τ, u ) as follows.
[Equation] [Equation]
16) where
[Equation]
Let
[Equation]
Through deﬁnition, Q (cid:48) r ( τ, u ) ≤ 0 .
Let
[Equation]
Obviously, Q (cid:48)(cid:48) tot ( τ, u ) satisfy the monotonic conditions with [ Q i ] Ni =1 .
[Equation]
Combining (18) and (19), we can ﬁnd a function
[Equation]
[Equation]
Thus, we show that for any joint state action function Q ( τ, u ) , it can ﬁnd a function Q jt that can beexpressed in the form of ResQ (1), and share the same optimal actions with Q .
Further, we analysis the property of Q jt ( τ, u ) ∀ u (cid:54) =
¯ u .
[Equation]
Then, we analysis the property of Q jt ( τ, ¯ u ) .
[Equation]
Combining (23) and (27), we reach the conclusion that the two functions Q ( τ, u ) and Q jt ( τ, u ) differonly in the maximal values.
And Q jt ( τ, u ) can be expressed as the form of the addition of onemonotonicity function Q tot ( τ, u ) and a residual state-action function Q r ( τ, u )
Theorem 3.
A stochastic joint state-action function
[Equation]
(28) is factorized by [ Z i ( τ i , u i )]
Ni =1 , if Z r ( τ, u ) ≤ 0 and w r ( τ, u ) = 0 when u
= ¯ u , otherwise 1 .
Z dmix ( τ, u ) is the factorization method proposed in [7].
Z dmix ( τ, u ) =
Z mean ( τ, u )+ Z shape ( τ, u ) , 508 E [ Z shape ( τ, u )]
= 0 , Q i =
E
[ Z i ( τ i , u i )] .
Z mean ( τ, u ) is a monotonically increasing function withrespect to Q i or the sum of [ Q i ] N 1 .
Proof.
We will show that arg max u E [ Z jt ( τ, u )]
= ¯ u , where ¯ u
i
=
arg max u i E
[ Z i ( τ i , u i )]
and¯ u =
[¯ u i ]
Ni =1 .
E [ Z jt ( τ, ¯ u )] =
E [ Z dmix ( τ, ¯ u )] ( From (28) w ( τ, ¯ u ) = 0) (29) = E [ Z mean ( τ, ¯ u ) + Z shape ( τ, ¯ u )] (30) =
E [ Z mean ( τ, ¯ u )]
( From E [ Z shape ( τ, u )] = 0) (31) =
E [ Q mix ( τ, ¯ u )] ( Deﬁnition ) (32)
[Equation]
relation )
[Equation]
33) =
[Equation]
34) =
[Equation]
u
[Equation] [Equation]
[Equation]
[Equation]
[Equation]
[Equation]
39) (37) comes from w r ( τ, u ) = 1 ∀ u (cid:54) =
¯ u and Z r ( τ, u ) ≤ 0 .
(29) to (39) mean that ¯ u =
[¯ u i ]
Ni =1maximizes
E [ Z jt ( τ, u )] .
Thus [ Z i ( τ i , u i )]
Ni =1 satisﬁes DIGM for Z jt ( τ, u ) .
Theorem 4.
A stochastic joint state-action function
[Equation]
(40) is factorized by [ Z i ( τ i , u i )]
Ni =1 , if Z r ( τ, u ) ≤ 0 , Z tot ( τ, u ) =
(cid:80) Ni =
k
i Z i ( τ i , u i ) k
i ≥ 0 andw r ( τ, u )
=
when
Proof.
Theorem 4 shows that if Z r ( τ, u ) ≤ 0 , Z tot ( τ, u ) = (cid:80) Ni =
k
i Z i ( τ i , u i ) k
i ≥ 0 andw r ( τ, u )
=
when u = ¯ u , otherwise 1 , then [ Z i ( τ i , u i )]
Ni =1 satisfy the DIGM principle for Z jt ( τ, u ) .
We will show that arg max u E [ Z jt ( τ, u )]
= ¯ u , where ¯ u
i
=
arg max u i E
[ Z i ( τ i , u i )] and ¯ u =
[¯ u i ]
Ni =1 .
E [Equation] Z jt
[Equation]
¯ u
[Equation]
[Equation]
E [
[Equation]
tot
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
Because Z tot ( τ, u ) = (cid:80) Ni =1 k
i Z i ( τ i , u i ) k
i ≥ 0 , the inequality formula (42) is established.
(43)comes from w r ( τ, u ) = 1 ∀ u (cid:54) =
¯ u and Q r ( τ, u ) ≤ 0 .
(44) comes from w r ( τ, u )
Z r ( τ, u ) = 522 E [ w r ( τ, u )
Z r ( τ, u )] .
(41) to (45) mean that ¯ u
= [¯ u i ]
Ni =1 maximizes E [ Z jt ( τ, u )] .
Thus[ Z i ( τ i , u i )]
Ni =1 satisﬁes DIGM for Z jt ( τ, u ) .
Example: Monotonic increasing function fails .
In Theorem 4, the main function Z tot is a positiveweighted sum of Z i instead of a monotonic mixing of Z i , because not all monotonic mixing functionssatisfy the DIGM principle (e.g., Z jt
=
Z tot = (cid:80) N i =
Z 2 i ), it could lead to incorrect estimation of theoptimal actions.
We show an example that Z (cid:48) tot
=
Z 21 + Z 22 fails.
Let Z 1 ( τ, a )
= 2 for 100% of thetime, and Z 1 ( τ, b ) = 3 for 50% of the time, Z 1 ( τ, b ) =
for 50% of the time.
Clearly E Z 1 ( τ, a ) =
2and E Z 1 ( τ, b ) = 1 . 5 .
Let Z 2 =
Z 1 .
If Z jt and Z i satisfy the DIGM theorem, then the optimal actionfor Z jt should be ( a, a ) =
(arg max E Z 1 ( τ, u ) , arg max E Z 2 ( τ, u )) .
However, E Z 21 ( τ, a ) = 4 and 531 E
Z 21 ( τ, b )
= 4 .
arg max Z jt =
( b, b ) rather than ( a, a )
A.2 Experimental Setup
We adopt the PyMARL [11] implementation of QMIX,VDN, QAtten, QTran, OW QMIX, CW
QMIX, QPlex, and DMIX from their open-source repositories 234 .
Their code are released under the
Apache License V2.0.
Their hyper-paramters are the same as that in PyMARL.
ResQ is designedbased on PyMARL as well.
All the algorithms in this submission is trained with 1 rollout process for2 million steps (the default setup for Weighted QMIX).
At each 10,000 training step, the algorithm istested on test episodes to evaluate their performance.
For ResQ, the RMSProp optimizer ( l r = 1 e − 3 )is used.
The batch size and buffer size are 32 and 5,000, respectively.
Q tot has 32 hidden dimensions,equiped with 2 hypernets, each with 64 dimensions.
The (cid:15) used in (cid:15) − greedy annealed from 1 to0.05 within 100K steps.
TD lambda is used, and λ = 0 .
All the experiments are ran within a localcluster.
Each experiment is ran within a NVIDIA 3090 GPU.
A.2.1 A distributional one-step Matrix Game
We study the representation power of multipe methods for a distributional matrix game.
The matrixgame is modiﬁed from the matrix depicted in Figure 1.
We add a nomral distribution (mean = 0,std=1) value into all action-value of the matrix.
The payoff matrix of the game is depicted in Table 1(a).
All the cells show the mean and the variance of a normal distributed reward.
For example, (8,1)indicates that the mean and the variance of the reward distribution are 8 and 1, respectively.
In this matrix game, all the algorithm are ran through a full exploration (cid:15) =
1 for (cid:15) -greedy conductedover 20,000 steps.
This setting guarantees the exploration of all possible actions.
As it is observed in Table 1, only ResQ/ResZ, and CW QMIX can obtain the optimal policy, DMIX,
DDN, QTran, and QPlex learn the second-best policy, and OW QMIX learns a wrong policy.
Albeit
CW QMIX can ﬁnd the optimal action, its learned Q jt has a high approximation, as it focus on theoptimal actions only.
These result demonstrates the ability of ResQ to factorize difﬁcult state-action value functionswith low approximation error and high sample efﬁciency.
Moreover, we ﬁnd that ResZ has lowerapproximation error than ResQ. [Table] [Table]
A.2.2 StarCraft II Multi-Agent Challenge (SMAC)
We study the performance of ResQ and other algorithms on a well known MARL benchmark, the
StarCraft II Multi-Agent Challenge (SMAC).
In each environment of SMAC, two teams of agentsﬁght against each other.
One team is controlled by the built-in game artiﬁcial intelligence, which ishandcrafted carefully.
We choose the highest difﬁcult built-in game script.
Another team of agentsare controlled by the decentralized policies learned by the MARL algorithms.
Each agent can receivecircular observation around the agent, and attack enemies close-by.
In each episode, if the agentseliminate all the enemies within a limited time, the game episode is counted as won.
Otherwise, thegame is counted as a failure.
For the SMAC tasks, the agents must learn the policies which maximizetheir expected total rewards.
The reward of SMAC is based on the damage dealt, enemy agents killed,and the game won.
We adopt the default SMAC reward schema.
The Starcraft II version used in ResQ is 2.4.6.2.69232, which is the same version used in QMIX and
Weighted QMIX.
For pymarl2, it used another version SC 2.4.10.
As it is indicated in Weighted
QMIX and conﬁrmed by us, " performance is not comparable across versions ".
We have shown in
Figure 5 (f), ResQ/ResZ perform better than the ﬁne-tuned QMIX [14] for SC2.4.10 in the MMM2scenario.
We have shown the results for ResQ, ResZ, QPLEX, QTran, QMIX, VDN, DMIX, OW QMIX,and CW QMIX.
We have also evaluated the performance of DDN [7], REFIL [17], and QTran++.
Their results are ploted in Figure 1.
Due to the poor performance of REFIL, we only evaluate itsperformance on the MMM2, MMM, 3s_vs_5z, 1c3s5z, 8m_vs_9m scenarios.
For QTran++, weonly evaluate its performance on the MMM2 and MMM scenarios due to its poor performance andcomputation resource limitation.
Figure 2 depicts the results for three super-hard scenarios (Corridor, 3s5z_vs_3s6z, 6h_vs_8z) of the
SMAC benchmark.
AS it is shown in the ﬁgure, all algorithms perform poorly for these super-hardscenarios.
This indicates that better exploration techniques should be used together with the valuefactorization methods.
B Response to Reviewers
ResQ and ResZ use more parameters than DMIX with the residual functions.
By modifying thedimension of the last layer of the hypernet inside DMIX, we have increased the number of parametersof DMIX from 85K to 350K, which is bigger than that of ResQ (320K) and that of ResZ(316K).
Let’sdenote DMIX with more parameters as DMIX-larger, and test its performance in the MMM2, the
MMM, and the 3s_vs_5z scenarios.
The experimental results are depicted in Figure 3.
DMIX-larger performs slightly better than DMIX in the MMM2 scenario but worse than DMIX in the MMMand the 3s_vs_5z scenarios.
ResQ and ResZ performs better than DMIX and DMIX-larger in thesescenarios.
This means that the performance improvement of ResQ over DMIX does not come fromthe use of more parameters.
To study the reason why ResQ performs better than QTran, We use two variants of ResQ: Qtot-VDNand Qtot-VDN-MSE.
Qtot-VDN uses VDN as the main function same as QTran.
The differencebetween Qtot-VDN between Qtran is the residual function, the mask, and the inequality conditions.
ResQ requires Q r ≤ 0 and QTran requires that Q tran ( τ, u ) ≥ Q ( τ, u ) ∀ u (cid:54) =
¯ u .
Qtot-VDN-MSEuses MSE loss to implement Q r ≤ 0 , and it uses VDN as its main function.
Their results are depictedin Figure 4.
For MMM2, Qtot-VDN can obtain a win rate of 0.6.
The win rate for QTran is 0.
For the
8m_vs_9m scenario, Qtot-VDN can obtain a win rate of 0.8.
And QTran can obtain a win rate of0.4.
The performance gap does not come from the implementation of the main function; it comesfrom the residual function, the mask function, and the inequality conditions.
Further, Qtot-VDNMSE performs similarly to Qtot-VDN, and it performs better than QTran.
This suggests that theperformance gap between ResQ and QTran does not come from the implementation of inequalityconditions for ResQ.
ResQ does not signiﬁcantly affect by inequality violations.
We have evaluated the performance of avariant of ResQ, ResQ-MSE, which uses MSE loss to implement Q r ≤ 0 .
Further, we have studiedthe performance of ResQ-MSE in the MMM2, MMM, and 8m_vs_9m scenarios.
The results aredepicted in Figure 5.
The performance of ResQ-mse and ResQ are similar in these three scenarios.
This indicates that the violation of inequality conditions does not signiﬁcantlyaffect the performancefor these two scenarios.