SHAQ: Incorporating Shapley Value Theory into Multi-Agent Q-Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Value factorisation is a useful technique for multi-agent reinforcement learning(MARL) in global reward game, however its underlying mechanism is not yet fullyunderstood.
This paper studies a theoretical framework for value factorisation withinterpretability via Shapley value theory.
We generalise Shapley value to Markovconvex game called Markov Shapley value (MSV) and apply it as a value factorisation method in global reward game, which is obtained by the equivalence betweenthe two games.
Based on the properties of MSV, we derive Shapley-Bellmanoptimality equation (SBOE) to evaluate the optimal MSV, which corresponds toan optimal joint deterministic policy.
Furthermore, we propose Shapley-Bellmanoperator (SBO) that is proved to solve SBOE.
With a stochastic approximationand some transformations, a new MARL algorithm called Shapley Q-learning(SHAQ) is established, the implementation of which is guided by the theoreticalresults of SBO and MSV.
We also discuss the relationship between SHAQ andrelevant value factorisation methods.
In the experiments SHAQ exhibits not onlysuperior performances on all tasks but also the interpretability that agrees with thetheoretical analysis.
Introduction
Cooperative games are a critical research area in multi-agent reinforcement learning (MARL).
Manyreal-life tasks can be modeled as cooperative games, e g. the coordination of autonomous vehicles[1], autonomous distributed logistics [2] and distributed voltage control in power networks [3].
Inthis paper, we consider global reward game (a.k.a. team reward game), an important subclass ofcooperative games wherein agents aim to jointly maximize cumulative global rewards over time.
There are two categories of methods to solve this problem: (i) each agent identically maximizescumulative global rewards,
i e. learning with a shared value function [4–6]; and (ii) each agentindividually maximizes distributed values, i e. learning with (implicit) credit assignments (e.g.,marginal contribution and value factorisation) [7–11].
By the view of non-cooperative game theory, global reward game are equivalent to Markov game [12]with global reward (a.k.a. team reward).
Its aim is to learn a stationary joint policy to reach a Markovequilibrium so that no agent tends to unilaterally change its policy to maximize cumulative globalrewards.
Standing by this view, learning with value factorisation cannot be directly explained [13].
Inthis paper, to clearly interpret the value factorisation, we take the perspective of cooperative gametheory [14], wherein agents are partitioned into coalitions and a payoff distribution scheme is foundto distribute optimal values to coalitions.
The corresponding solution is called Markov core, wherebyno agent has an incentive to deviate.
When all agents are partitioned into one coalition (called grandcoalition), the payoff distribution scheme naturally plays the role of value factorisation for an optimalglobal value.
to
Wang et al.
[13] extended convex game (a game model in cooperative game theory)
[14] to dynamicscenarios, which we name as Markov convex game in this paper.
We construct the analytic form of
Shapley value for Markov convex game, and prove that it reaches the Markov core under the grandcoalition, named as Markov Shapley value.
The optimal Markov Shapley value implies not onlythe optimal global value but also that no agent has incentives to deviate from the grand coalition.
Additionally, Markov Shapley value enjoys the following properties: (i) identiﬁability of dummyagents; (ii) efﬁciency; (iii) proportional to the contribution; and (iv) symmetry.
These properties aidthe interpretation and validity of value factorisation in the global reward game, and such transparencyand reliability is critical to industrial applications [3].
Based on the efﬁciency property, we derive Shapley-Bellman optimality equation that is an extensionof Bellman optimality equation [15, 16].
Moreover, we propose Shapley-Bellman operator and proveits convergence to the Shapley-Bellman optimality equation and its optimal joint deterministic policy.
With a stochastic approximation of Shapley-Bellman operator and some transformations, we derivean algorithm called Shapley Q-learning (SHAQ).
SHAQ learns to approximate the optimal Markov
Shapley Q-value (an equivalent form of the optimal Markov Shapley value).
Moreover, we enable
SHAQ decentralised in order to ﬁt the decentralised execution framework and this decentralisationstill remain the convergence condition of Shapley-Bellman operator.
The proposed method, SHAQ, is evaluated on two global reward games such as Predator-Prey [17]and multi-agent StarCraft benchmark tasks [18].
In the experiments, SHAQ shows not only generallygood performances on solving all tasks but also the interpretability that is deﬁcient in the state-of-theart baselines.
Speciﬁcally, the learned Markov Shapley Q-value reﬂects the contribution of each agentand identiﬁes dummy agents.
Besides, the optimal Shapley Q-value of each agent is demonstratedequal (i.e., a sort of fairness).
All of these support our theoretical analyses.
Markov Convex Game
We now formally deﬁne Markov convex game (MCG) that can be described as a tuple￿ N ,
S ,
A ,T,
CS , ⇡ ,R t ,  , ( max ⇡ i x i ( s ))
i ∈ N ￿ .
N is the set of all agents.
S is the set of statesand
A = × i ∈ N A
i is the joint action set of all agents wherein
A i is each agent’s action set.
T ( s ′ ￿ s , a ) is deﬁned as the transition probability between states.
CS = { C 1 ,...,
C n } is called the coalitionstructure , where
C i ⊆
N called a coalition is a subset of all agents.
￿ and
N are two special cases ofcoalitions i.e. the empty coalition and the grand coalition respectively.
Conventionally, it is assumedthat
C
[Equation]
×
[Equation]
⇡ i is the joint policy of all agents.
For any coalition
C ,it is equipped with a coalition policy ⇡
[Equation] [Equation]
[Equation]
￿ s ) deﬁned over the coalition action set
A C = × i ∈
C A i .
Therefore, ⇡ can be seen as the grand coalition policy .
R t
∶ S × A C → [ 0 , ∞ ) (i.e.,a characteristic function) is the coalition reward at time step t .
Accordingly, R t ( s , a ) is the grandcoalition reward (i.e., equivalent to the global reward) at time step t that is written as R ( s , a ) or R forconciseness in the rest of paper.
∈ ( 0 ,) is the discounted factor.
The inﬁnite long-term discountedcumulative coalition rewards is deﬁned as V ⇡ C
[Equation]
[ 0 , ∞ ) ,called a coalition value .
Moreover, the empty coalition value V ⇡ ￿ ( s ) = 0 and V ⇡ ( s ) denotes thegrand coalition value (i.e., also called the global value since the equivalence proof from [13]).
Thesolution of MCG is to ﬁnd a tuple
[Equation]
( s ))
i ∈ N ￿ , where ( max ⇡ i x i ( s ))
i ∈ N indicates thepayoff distributions (i.e., credit assignments) under the optimal joint policy given a coalition structure.
Under the assumption
[Equation]
, the condition for MCG is as follows:
[Equation]
In MCG with the grand coalition i.e.,
CS = { N } , Markov core , a solution concept describing stability,is deﬁned as a set of payoff distribution schemes by which no agent has incentives to deviate fromthe grand coalition to gain more proﬁts.
Mathematically, Markov core can be expressed as:
[Equation]
C
[Equation]
where max ⇡
C x ( s ￿ C )
=
∑ i ∈ C max ⇡ i x i ( s ) .
It aims to ﬁnd a payoff distribution scheme ( x i ( s ))
i ∈ N that can ﬁnally converge to Markov core under the optimal joint policy.
To assist the application on Q-learning, we similarly deﬁne coalition Q-value as Q ⇡ C ( s , a
C ) ∈ [ 0 , + ∞ )
C ⊂ N .
Following the above convention, the grand coalition Q-value (or the global
Q-value) can be written as Q ⇡ ( s , a ) .
Moreover, the optimal coalition Q-value of
C w.r.t.
the optimaljoint policy of
D ⊆ C (i.e., ⇡ ∗ D ) and the suboptimal joint policy of
C ￿ D (i.e., ⇡
C ￿ D ) is deﬁned as
Q ⇡ ∗ D ( s , a
C ) .
Therefore, the optimal coalition Q-value of
C w.r.t.
the optimal joint policy of
C isdeﬁned as Q ⇡ ∗ C ( s , a
C ) .
Accordingly, the optimal global coalition Q-value w.r.t.
the optimal jointpolicy of the grand coalition is denoted as Q ⇡ ∗ ( s , a ) .
Markov Shapley Value
By the view of cooperative game theory, the grand coalition is progressively formed by a permutationof agents.
Accordingly, marginal contribution measures the contribution of agent i to an arbitraryintermediate coalition, as shown in Deﬁnition 1.
Deﬁnition 1.
In Markov convex game, with a permutation of agents ￿ j 1 ,j 2 ,...,j ￿ N ￿ ￿ , ∀ j n ∈ N formingthe grand coalition
N , where n ∈ { 1 ,..., ￿ N ￿ } ,j a ≠ j b if a ≠ b , the marginal contribution of an agenti is deﬁned as the following equation such that [Equation]
[Equation]
where
C i
[Equation]
[Equation]
,...,j n
[Equation]
} for j n = i is an arbitrary intermediate coalition where agent i would joinduring the process of grand coalition formation.
Proposition 1.
Agent i ’s action marginal contribution can be derived as follows:
[Equation]
As Proposition 1 shows, an agent’s action marginal contribution (analogous to Q-value) can bederived according to Eq.4.
It is usually more useful in solving MARL problems.
It is apparent that marginal contribution only considers one permutation to form the grand coalition.
By the viewpoint from Shapley [19], the fairness is achieved through considering how much the agenti increases the optimal values (i.e., an implementation of the credits proportional to the contributionscalled marginal contributions) of the coalitions in all possible permutations when it joins in, i.e.,max ⇡
[Equation]
i } . Therefore, we construct Shapley value under Markov dynamics based on the marginal contributions shown in Deﬁnition 2, named as Markov
Shapley value (MSV).
Deﬁnition 2.
Markov Shapley value is represented as
[Equation]
With the deterministic policy, Markov Shapley value can be equivalently represented as
[Equation]
i ) . (6) where  i ( s ￿ C i ) is deﬁned in Eq.3 and  i ( s ,a i ￿ C i ) is deﬁned in Eq.4.
For convenience, we name Eq.6 as Markov Shapley Q-value (MSQ).
Brieﬂy, MSV calculates theweighted average of marginal contributions.
Since a coalition may repeatedly appear among allpermutations (i.e., ￿ N ￿ !
permutations), the ratio between the occurrence frequency ￿ C i ￿ !
( ￿ N ￿ − ￿ C
i ￿ − ) !
and the total frequency ￿ N ￿ ! is used as a weight to describe the importance of the correspondingmarginal contribution.
Besides, the sum of all weights is equal to 1, so each weight can be interpretedas a probability distribution.
Consequently, MSV can be seen as the expectation of marginalcontributions, denoted as E
C i ∼
Pr ( C i ￿ N ￿ { i })
[  i ( s ￿ C i )] .
Note that Pr ( C i ￿ N ￿ { i }) is a bell-shapedprobability distribution.
By the above relationship, Remark 1 is directly obtained.
Remark 1.
Uniformly sampling different permutations is equivalent to directly sampling from
Pr ( C i ￿ N ￿ { i }) , since the coalition generation is from the permutation to form the grand coalition.
Proposition 2.
Markov Shapley value possesses properties as follows: (i) identiﬁability of dummyagents: V  i ( s ) = 0 ; (ii) efﬁciency: max ⇡ V ⇡ ( s ) =
∑ i ∈ N max ⇡ i V
i ( s ) ; (iii) proportional to thecontribution; and (iv) symmetry.
Proposition 2 shows three properties of MSV.
The most important property is Property (ii) thataids the formulation of Shapley-Bellman optimality equation.
Property (iii) provides a fundamentalmechanism to quantitatively describe “fairness” among agents.
Property (i) and (iii) play importantroles in interpretation for value factorisation (or credit assignment).
Property (iv) indicates that if twoagents are symmetric, then their optimal MSVs should be equal, but the reverse does not necessarilyhold.
All these properties that deﬁne the fairness are inherited from the original Shapley value.
Shapley Q-Learning
4.1 Deﬁnition and Formulation
Shapley-Bellman Optimality Equation.
Based on Bellman optimality equation [15] and thefollowing conditions (the explanation of which is left to Section 4.2):
C.1.
Efﬁciency of MSV obtained from Proposition 2;
C.
[Equation] [Equation]
b i
[Equation]
s
[Equation]
, where w i ( s ,a i ) > 0 and b i ( s ) ≥ 0 are boundedand
[Equation]
we derive Shapley-Bellman optimality equation (SBOE) for evaluating the optimal MSQ (an equivalent form to optimal MSV) such that
[Equation]
where w ( s , a ) =
[Equation]
( s ) =
[ b i ( s )] ￿ ∈
R ￿ N ￿ ≥ 0 ; Q  ∗ ( s , a ) =
Q  ∗ i ( s ,a i )]
￿ ∈
R ￿ N ￿ ≥ 0and Q  ∗ i ( s ,a i ) denotes the optimal MSQ.
If Eq.7 holds, the optimal MSQ is achieved.
Moreover,it reveals an implication that for any s ∈ S and a ∗ i = arg max a i Q  ∗ i ( s ,a i ) , we have the solutionw i ( s ,a ∗ i ) =
￿￿ N ￿ (see Appendix E.4.1).
Literally, the assigned credits would be equal and eachagent would receive Q ⇡ ∗ ( s , a ) ￿￿ N ￿ if performing the optimal actions.
It is apparent that the efﬁciencystill holds under this situation, which can be interpreted as an extremely fair credit assignment suchthat the credit to each agent should not be discriminated if all of them perform optimally, regardless oftheir roles.
The equal credit assignment was also revealed by Wang et al.
[20] recently from anotherperspective of analysis.
Nevertheless, w i ( s ,a i ) for a i ≠ arg max a i Q  ∗ i ( s ,a i ) needs to be learned.
Shapley-Bellman Operator.
To ﬁnd an optimal solution described by Eq.7, we now propose anoperator called Shapley-Bellman operator (SBO), i.e., ⌥
∶ × i ∈ N
Q  i ( s ,a i ) ￿ × i ∈ N
Q  i ( s ,a i ) , whichis deﬁned as follows:
[Equation]
[Equation]
(8) where w i ( s ,a i ) = ￿￿ N ￿ when a i = arg max a i Q  i ( s ,a i ) .
We prove that the optimal joint deterministic policy can be achieved by recursively running SBO in Theorem 1.
Theorem 1.
Shapley-Bellman operator is able to converge to the optimal Markov Shapley Q-valueand the corresponding optimal joint deterministic policy when max s [Equation] ∑
[Equation]
N max a i w i (
[Equation]
￿ < 1  .
Shapley Q-Learning.
For easy implementation, we conduct transformation for the stochasticapproximation of SBO and derive Shapley Q-learning (SHAQ) whose TD error is shown as follows:
[Equation]
where
i
[Equation]
Actually, the closed-form expression of  i ( s ,a i ) is written as ￿ N ￿
[Equation]
1 w i ( s ,a i ) − 1 .
If inserting thecondition that w i ( s ,a i ) = ￿￿ N ￿ when a i = arg max a i Q  i ( s ,a i ) as well as deﬁning  i ( s ,a i ) as↵ i ( s ,a i ) when a i ≠ arg max a i Q  i ( s ,a i ) , Eq.10 is obtained.
The term b ( s ) is cancelled in Eq.9thanks to the condition such that ∑
[Equation]
[Equation]
( s ) = 0 .
Note that the condition to w i ( s ,a i ) in
Theorem 1 should hold for the convergence of SHAQ in implementation (see Appendix E.4.4).
4.2 Validity and Interpretability
In this section, we show the validity of SBOE and the interpretability of SHAQ, i.e., providing thereasons why SBOE is valid to be formulated and SHAQ is an interpretable value factorisation methodfor global reward game.
Theorem 2.
The optimal Markov Shapley value is a solution in the Markov core under Markovconvex game with the grand coalition.
Remark 2.
For an arbitrary state s ∈ S , by C.2 it is not difﬁcult to check that even if an arbitraryagent i is dummy (i.e., Q  ∗ i ( s ,a i ) = 0 for some i ∈ N ), Q ⇡ ∗ ( s , a ) and Q  ∗ j ( s ,a j ) , ∀ j ≠ i wouldnot be zero if b i ( s ) ≠ 0 .
If the extreme case happens that for an arbitrary state s ∈ S all agentsare dummies, since ∑ i ∈ N w i ( s ,a i ) − 1 b i ( s ) =
we are allowed to set b i ( s ) = 0 , ∀ i ∈ N so that
Q ⇡ ∗ ( s , a ) = 0 and efﬁciency such that max a Q ⇡ ∗ ( s , a ) =
∑ i ∈ N max a i Q  ∗ i ( s ,a i ) is still valid.
First, we give a proof for showing that the optimal MSV is a solution in Markov core under the grandcoalition, as Theorem 2 shows.
Since a solution in Markov core implies the optimal global value(see Remark 5 in Appendix D.2.2), we can conclude that the optimal MSV can lead to the optimalglobal value (a.k.a.
social welfare), which links C.1 to Markov core.
As a result, solving SBOEis equivalent to solving Markov core under the grand coalition and SHAQ is actually a learningalgorithm that reliably converges to Markov core.
As per the deﬁnition in Section 2, we can say that
SHAQ leads to the result that no agents have incentives to deviate from the grand coalition, whichprovides an interpretation of value factorisation for global reward game.
C.2 maintains the validityof the relationship between the optimal MSQ and the optimal global Q-value if there exist dummyagents (see Remark 2), so that the deﬁnition of SBOE is valid for MCG and MSQ in almost everycase, which preserves the completeness of the theory.
4.3 Implementations
We now describe a practical implementation of SHAQ for Dec-POMDP [21] (i.e., the global rewardgame but with partial observations).
First, the global state is replaced by the history of each agentto guarantee the optimal deterministic joint policy [21].
Accordingly, Markov Shapley Q-value isdenoted as Q  i ( ⌧ i ,a i ) , wherein ⌧ i is a history of partial observations of agent i .
Since the paradigmof centralised training decentralised execution (CTDE) [22] is applied, the global state (i.e., s ) forˆ ↵ i ( s ,a i ) can be obtained during training.
Compatible with the decentralised execution, we use only one parametric function ˆ Q i ( ⌧ i ,a i ) todirectly approximate Q  i ( ⌧ i ,a i ) .
With speciﬁc conditions (see Proposition 8 in Appendix E.6) theinformation of coalition formation can be equivalently transferred to ˆ i ( s ,a i ;
a
C i ) .
As a result, i ( s ,a i ) is equivalent to the form as follows:
[Equation]
wherein ˆ ↵ i ( s ,a i ) =
E
C i ∼
Pr ( C i ￿ N ￿ { i })
[ ˆ i ( s ,a i ; a
C i )] .
To solve partial observability, ˆ Q i ( ⌧ i ,a i ) isempirically represented as recurrent neural network (RNN) with GRUs [23].
ˆ i ( s ,a i ; a
C i ) is approximated by a parametric function F s + 1 and thus ˆ ↵ i ( s ,a i ) can be expressed as the following equation:
[Equation]
where ˆ Q
C ki ( ⌧
C ki , a
C ki )
=
￿ C ki ￿ ∑ j ∈ C
ki ˆ Q j ( ⌧ j ,a j ) and
C ki is sampled M times from Pr ( C i ￿ N ￿ { i })
(i.e.,implemented as Remark 1 suggests) to approximate E
C i ∼
Pr ( C i ￿ N ￿ { i })
[ ˆ i ( s ,a i ; a
C i )] using Monte
Carlo approximation; and F s is a monotonic function, followed by an absolute activation function,whose weights are generated from hyper-networks w.r.t.
the global state.
We show that Eq.12 satisﬁesthe condition to w i ( s ,a i ) in Theorem 1 and therefore the implementation of SHAQ is reliable (see
Appendix E.6.1).
By using the framework of ﬁtted Q-learning [24] to solve large number of states (i.e., could be usuallyinﬁnite) and plugging in the above designed modules, the practical least-square-error loss function derived from Eq.9 is therefore stated as follows:
[Equation]
+  [Equation] i ∈ N max
a i ˆ Q i ( ⌧ ′ i ,a i ; ✓ − ) − ￿ i ∈ N ˆ  i ( s ,a i ;  ) ˆ Q i ( ⌧ i ,a i ; ✓ ) ￿ ￿ , (13) where all agents share the parameters of ˆ Q i ( s ,a i ; ✓ ) and ˆ ↵ i ( s ,a i ;  ) respectively; and ˆ Q i ( s ′ ,a i ; ✓ − ) works as the target where ✓ − is periodically updated.
The general training procedure follows theparadigm of DQN [25], with a replay buffer to store the online collection of agents’ episodes.
Todepict an overview of the algorithm, the pseudo code is shown in Appendix A.
Related Work
Value Factorisation in MARL.
To deal with the instability during training in global reward gameby independent learners [26], the centralised training and decentralised execution (CTDE) [22] wasproposed and it became a general paradigm for MARL.
Based on CTDE, MADDPG [27] learns aglobal Q-value that can be regarded as assigning the same credits to all agents during training [13],which may cause the unfair credit assignment [28].
To avoid this problem, VDN [8] was proposed tolearn the factorised Q-value, assuming that any global Q-value equals to the sum of decentralised
Q-values.
Nevertheless, this factorisation may limit the representation of the global Q-value.
Tomitigate this issue, QMIX [9] and QTRAN [10] were proposed to represent the global Q-value with aricher class w.r.t.
decentralised Q-values, based on the assumption (called Individual-Global-Max)of convergence to the optimal joint deterministic policy.
Markov Shapley value proposed in thispaper belongs to the family of value factorisation, based on the game-theoretical framework called
MCG that enjoys the interpretability.
From the conventional cooperative games (e.g., network ﬂowgame [29], induced subgraph game [30] that can be used for modelling social networks, and facilitylocation game [31]), it is insightful that the coalition introduced in this paper exists.
In many scenarios,however, the information of coalition might be unknown.
Therefore, the latent coalition is assumed,and we only need to concentrate on the observable information, e.g., the global reward.
Relationship to VDN.
By setting  i ( s ,a i ) = 1 for all state-action pairs, SHAQ degrades to VDN[8].
Although VDN tried to tackle the problem of dummy agents, Sunehag et al.
[8] did not givea theoretical guarantee on identifying it.
The Markov Shapley value theory proposed in this paperwell addresses this issue from both theoretical and empirical aspects.
These aspects show that VDNis a subclass of SHAQ.
The theoretical framework proposed in this paper answers to why VDNworks well in most scenarios but performs poorly in some scenarios (i.e.,  i ( s ,a i )
= 1 in Eq.9 wasincorrectly deﬁned over the suboptimal actions).
Relationship to COMA.
Compared with COMA [7], each agent i ’s
credit assignment ¯ Q i ( s ,a i ) ismathematically expressed as follows:
[Equation]
where subscript − i indicates the agents excluding i .
¯ Q i ( s ,a i ) can be seen as the action marginalcontribution between the grand coalition Q-value and the coalition Q-value excluding the agent i ,under some permutation to form the grand coalition wherein agent i is located at the last position .
The efﬁciency is obviously violated (i.e., the sum of optimal action marginal contributions deﬁnedhere is unlikely to be equal to the optimal grand coalition Q-value).
In contrast to COMA, SHAQconsiders all permutations to form the grand coalition to preserve the efﬁciency.
Relationship to Independent Learning.
Independent learning (e.g., IQL [26]) can be also seenas a special credit assignment, however, the credit assigned to each agent is still with no intuitiveinterpretation.
Mathematically, suppose that ¯ Q i ( s ,a i ) is the independent Q-value of agent i , we canrewrite it in the form consisting of action marginal contributions such that
[Equation]
It is intuitive to see that the independent Q-value is a direct approximation of MSQ, ignoring coalitionformation, while SHAQ considers coalition formation in approximation.
This gives an explanationfor why independent learning works well in some cooperative tasks [32].
Nevertheless, it encountersthe same issue as in COMA, the loss of properties led by the coalition formation.
Relationship to SQDDPG.
We now discuss the relationship between SQDDPG [13] and SHAQ.
Interms of algorithms, SQDDPG belongs to policy gradient methods (i.e., an approximation of policyiteration) while SHAQ belongs to value based methods (i.e., an approximation of value iteration).
Since policy iteration (with one-step policy evaluation) can be regarded equivalent to value iteration[33] (at least under a discrete action space), the theory behind SHAQ directly ﬁlls the gap in SQDDPGon theoretical guarantees of convergence to optimal joint policy .
Speciﬁcally, the learning procedureof SQDDPG iteratively performs the following stages:
[Equation]
It can be observed that both SQDDPG and SHAQ ideally converge to the same optimal MSQ w.r.t.the optimal actions such that
[Equation]
However, about suboptimal actions, SQDDPG does not provide any theoretical guarantee , whereas
SHAQ does with speciﬁc implementations as shown in Eq.12 to match the theoretical results shownin this paper.
Note that this is critical to reliable interpretations of the optimal MSQ w.r.t.
suboptimalactions (e.g., for detecting adversarial attacks on controllers if deployed in industry [34]).
Experiments
In this section, we show the experimental results of SHAQ on Predator-Prey [17] and various tasks in
StarCraft Multi-Agent Challenge (SMAC) 1 .
The baselines that we select for comparison are COMA[7], VDN [8], QMIX [9], MASAC [36], QTRAN [10], QPLEX [37] and W-QMIX (including CWQMIX and OW-QMIX) [35].
The implementation details of our algorithm are shown in Appendix
B.1, whereas the implementation of baselines are from [35] 2 .
We also compare SHAQ with SQDDPG[13] 3 , which is shown in Appendix C.3.
For all experiments, we use the ✏ -greedy exploration strategy,where ✏ is annealed from 1 to 0.05.
The annealing time steps vary among different experiments.
For Predator-Prey, we apply 1 million time steps for annealing, following the setup from [37].
Forthe easy and hard maps in SMAC, we apply 50k time steps for annealing, the same as that in [18];while for the super-hard maps in SMAC, we apply 1 million time steps for annealing to obtain moreexplorations so that more state-action pairs can be visited.
About the replay buffer size, we set as5000 for all algorithms that is the same as that in [35].
To fairly evaluate all algorithms, we runeach experiment with 5 random seeds.
All graphs showing experimental results are plotted with themedian and 25%-75% quartile shading.
The ablation study of SHAQ is shown in Appendix C.4.
6.1 Predator-Prey
[Figure]
In this section, we run the experiments on a partially-observable task called Predator-Prey [17],wherein 8 predators that we can control aim to capture 8 preys with random policies in a 10x10 grid world.
Each agent’s observation is a 5x5 sub-grid centering around it.
If a prey is captured bycoordination of 2 agents, predators will be rewarded by 10.
On the other hand, each unsuccessfulattempt by only 1 agent will be punished by a negative reward p.
In this experiment, we study thebehaviors of each algorithm under different values of p (that describes different levels of coordination).
As [35] reported, only QTRAN and W-QMIX can solve this task, while [37] found that the failurewas primarily due to the lack of explorations.
As a result, we apply the identical epsilon annealingschedule (i.e. 1 million time steps) employed in [37].
As Figure 1 shows, SHAQ can always solvethe tasks with different values of p. With the epsilon annealing strategy from [37], W-QMIX doesnot perform as well as reported in [35].
The reason could be its poor robustness to the increasedexplorations [35] for this environment (see the evidential experimental results in Appendix C.6).
The good performance of VDN validates our analysis in Section 5, whereas the performance of
QTRAN is surprisingly almost invariant to the value of p.
The performances of QPLEX and QMIXbecome obviously worse when p=-2.
The failure of MASAC and COMA could be due to that relativeovergeneralisation 4 prevents policy gradient methods from better coordination [39].
6.2 StarCraft Multi-Agent Challenge
We now evaluate SHAQ on the more challenging SMAC tasks, the environmental settings of whichare the same as that in [18].
To broadly compare the performance of SHAQ with baselines, weselect 4 easy maps: 8m, 3s5z, 1c3s5z and 10m_vs_11m; 3 hard maps: 5m_vs_6m, 3s_vs_5z and2c_vs_64zg; and 4 super-hard maps: 3s5z_vs_3s6z, Corridor, MMM2 and 6h_vs_8z.
All training isthrough online data collection.
Due to the limited space, we only show partial results in the main partof paper and leave the rest in Appendix C.1.
Performance Analysis.
First, we compare performances between SHAQ and baselines.
It shows in
Figure 2 that SHAQ outperforms all baselines on all maps, except for 6h_vs_8z.
On 6h_vs_8z, SHAQcan beat all baselines except for CW-QMIX.
VDN performs well on 4 maps but bad on the other 2maps, which still veriﬁes our analysis in Section 5.
QMIX and QPLEX perform well on the most ofmaps, except for 3s_vs_5z, 2c_vs_64zg and 6h_vs_8z.
As for COMA, MADDPG and MASAC, theirpoor performances could be due to the weak adaptability to challenging tasks.
Although QTRANcan theoretically represent the complete class of the global Q-value [10], its complicated learningparadigm could impede the convergence to the value function for challenging tasks and thereforeresult in the poor performance.
Although W-QMIX performs well on some maps, owing to lacking alaw on hyperparameter tuning [35] it is difﬁcult to be adapted for all scenarios (see Appendix C.2).
[Figure]
Interpretability of SHAQ.
To show the interpretability of SHAQ, we conduct a test on 3m (i.e., asimple task in SMAC), demonstrating the learned MSQs of both ✏ -greedy policy (for obtaining mixedoptimal and suboptimal actions) and greedy policy (for obtaining pure optimal actions).
As seenfrom Figure 3a, Agent 3 faces the direction opposite to enemies, meanwhile, the enemies are out ofits attacking range.
It can be understood as that Agent 3 does not contribute to the team and thus is almost a dummy agent.
Its MSQ is 0.84 (around 0) that correctly catch the manner of a dummyagent (verifying (i) in Proposition 2).
In contrast, Agent 1 and Agent 2 are attacking enemies, while
Agent 1 suffers from more attacks (with lower health) than Agent 2.
As a result, Agent 1 contributesmore than Agent 2 and therefore its MSQ is greater, which reﬂects the credits proportional to thecontributions (verifying (iii) in Proposition 2).
On the other hand, we can see from Figure 3e thatwith the optimal policies all agents receive almost identical MSQs (verifying the theoretical results in Section 4.1).
The above results well verify the theoretical analysis that we deliver before.
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
To verify that MSQs learned by SHAQ are non-trivial, we also show the results of VDN, QMIX and
QPLEX.
It is surprising that the Q-values of these baselines are also almost identical among agentsfor the optimal actions (however, the property disappears in more complicated scenarios as shownin Appendix C.5 while the property of SHAQ is still valid).
Since VDN is a subclass of SHAQ andpossesses the same form of loss function for optimal actions, it is reasonable that it obtains the similarresults to SHAQ.
The explanation for the results of QMIX and QPLEX deserves to be studied inthe future work.
As for the suboptimal actions, VDN does not possess an explicit interpretation as
SHAQ due to the incorrect deﬁnition of  i ( s ,a i )
= 1 over suboptimal actions (verifying the statementin Section 5).
Similarly, QMIX and QPLEX cannot show explicit interpretations over suboptimalactions either.
To enable the conclusion to be more convincing, we also visualise the results of SHAQon a more complicated scenario, 3s5z_vs_3s6z (see Appendix C.5).
Conclusion
Summary.
This paper generalises Shapley value to Markov convex game, called Markov Shapleyvalue.
Markov Shapley value inherits a number of properties: (i) identiﬁability of dummy agents;(ii) efﬁciency; (iii) proportional to the contribution and (iv) symmetry.
Based on Property (ii), wederive Shapley-Bellman optimality equation, Shapley-Bellman operator and SHAQ.
We prove thatsolving Shapley-Bellman optimality equation is equivalent to solving the Markov core (i.e., no agenthas incentives to deviate from the grand coalition).
Markov convex game with the grand coalitionis equivalent to global reward game [13], wherein Markov Shapley value plays the role of valuefactorisation.
Since SHAQ is a stochastic approximation of Shapley-Bellman operator that is provedto solve Shapley-Bellman optimality equation, global reward game with value factorisation becomesvalid standing by the cooperative game theoretical framework (i.e., solving Markov core).
Property (i)and (iii) in Proposition 2 are demonstrated in the experiments showing the interpretability of SHAQ.
Limitation and Future Work.
The value of Markov convex game is not limited to solving problemswith the grand coalition, though in this paper we design SHAQ that only focuses on the scenarioswith the grand coalition.
By removing the condition of supermodularity (see Eq.1), this frameworkcan be used to study more general coalition games where different coalitions of agents as units maycompete/cooperate with each other.
Since the grand coalition and Markov Shapley value is not asolution in Markov core yet, the learning process becomes more complicated to converge to Markovcore.
One research direction in future is to investigate dynamically forming the coalition structureand conducting credit assignments simultaneously.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately
reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 7.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Appendix F.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See AppendixE.1.(b)
Did you include complete proofs of all theoretical results?
[Yes] See Appendix E.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See supplementary material.(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix B.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 6.
[Table] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix B.1.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes]
[Table]
(b) Did you mention the license of the assets?
[N/A](c)
Did you include any new assets either in the supplemental material or as a URL?
[Yes](d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
[N/A]5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]