T OWARDS U NDERSTANDING L INEAR V ALUE D ECOMPOSITION IN C OOPERATIVE M ULTI -A GENT Q-L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Value decomposition is a popular and promising approach to scaling up multiagent reinforcement learning in cooperative settings
However
, the theoretical understanding of such methods is limited
In this paper, we introduce a variant of the ﬁtted Q-iteration framework for analyzing multi-agent Q-learning with value decomposition.
Based on this framework, we derive a closed-form solution to the empirical Bellman error minimization with linear value decomposition.
With this novel solution, we further reveal two interesting insights: i) linear value decomposition implicitly implements a classical multi-agent credit assignment called counterfactual difference rewards ; and ii) On-policy data distribution or richer Q function classes can improve the training stability of multi-agent Qlearning.
In the empirical study, our experiments demonstrate the realizability of our theoretical closed-form formulation and implications in the didactic examples and a broad set of StarCraft II unit micromanagement tasks, respectively
I NTRODUCTION
Cooperative multi-agent reinforcement learning (MARL) has great promise for addressing coordination problems in a variety of applications, such as robotic systems (Hüttenrauch et al., 2017), autonomous cars (Cao et al., 2012), and sensor networks (Zhang & Lesser, 2011).
Such complex tasks often require MARL to learn decentralized policies for agents to jointly optimize a global cumulative reward signal, and post a number of challenges, including multi-agent credit assignment (Wolpert &
Tumer, 2002; Nguyen et al., 2018), non-stationarity
Zhang & Lesser, 2010; Song et al., 2019), and scalability (Zhang & Lesser, 2011; Panait & Luke, 2005).
Recently, by leveraging the strength of deep learning techniques, cooperative MARL has made a series of great progress (Sunehag et al., 2018;
Baker et al., 2020; Wang et al., 2020b;a), particularly in value-based methods that demonstrate stateof-the-art performance on challenging tasks such as StarCraft unit micromanagement (Samvelyan et al., 2019).
Sunehag et al.
(2018) proposed a popular approach called value-decomposition network (VDN) based on the paradigm of centralized training with decentralized execution (CTDE; Foerster et al., 2016).
VDN learns a centralized but factorizable joint value function Q tot , represented as the summation of individual value functions Q i .
During the execution, decentralized policies can be easily derived for each agent i by greedily selecting actions with respect to its local value function
Q i .
By utilizing this decomposition structure, an implicit multi-agent credit assignment is realized because Q i is learned by neural network backpropagation from the total temporal-difference error on the single global reward signal, rather than on a local reward signal speciﬁc to agent i .
This decomposition technique signiﬁcantly improves the scalability of multi-agent Q-learning algorithms and fosters a series of subsequent works, including QMIX (Rashid et al., 2018), QTRAN (Son et al., 2019), and QPLEX (Wang et al., 2020a).
In spite of the empirical success in a broad class of tasks, multi-agent Q-learning with linear value decomposition has not been theoretically well-understood.
Because of its limited representation complexity, the standard Bellman update is not a closed operator in the joint action-value function class with linear value decomposition.
The approximation error induced by this incompleteness is known as inherent Bellman error (Munos & Szepesvári, 2008), which usually deviates Q-learning to an unexpected behavior.
To develop a deeper understanding of learning with value decomposition, this paper introduces a multi-agent variant of the popular Fitted Q-Iteration (FQI; Ernst et al., 2005;
Levine et al., 2020) framework and derives a closed-form solution to its empirical Bellman error minimization.
To the best of our knowledge, it is the ﬁrst theoretical analysis that characterizes the underlying mechanism of linear value decomposition in cooperative multi-agent Q-learning, which can serve as a powerful toolkit to establish follow-up profound theories and explore potential insights from different perspectives in this popular value decomposition structure.
By utilizing this novel closed-form solution, this paper formally reveals two interesting insights: 1)
Learning linear value decomposition implicitly implements a classical multi-agent credit assignment method called counterfactual difference rewards (Wolpert & Tumer, 2002), which draws a connection with COMA (Foerster et al., 2018), a multi-agent policy-gradient method.
2) Multi-agent Q-learning with linear value decomposition potentially suffers from the risk of unbounded divergence from arbitrary initialization.
On-policy data distribution or richer Q function classes can provide local or global convergence guarantees for multi-agent Q-learning, respectively.
Finally, we set up an extensive set of experiments to demonstrate the realizability of our theoretical implications.
Besides the FQI framework, we also consider deep-learning-based implementations of different multi-agent value decomposition structures.
Through didactic examples and the StarCraft II benchmark, we design several experiments to illustrate the consistency of our closed-form formulation with the empirical results, and that online data distribution and richer Q function classes can signiﬁcantly alleviate the limitations of VDN on the ofﬂine training process (Levine et al., 2020).
R ELATED W ORK
Deep Q-learning algorithms that use neural networks as function approximators have shown great promise in solving complicated decision-making problems (Mnih et al., 2015).
One of the core component of such methods is iterative Bellman error minimization, which can be modelled by a classical framework called Fitted Q-Iteration (FQI; Ernst et al., 2005).
FQI utilizes a speciﬁc Q function class to iteratively optimize empirical Bellman error on a dataset D .
Great efforts have been made towards theoretically characterizing the behavior of FQI with ﬁnite samples and imperfect function classes (Munos & Szepesvári, 2008; Farahmand et al., 2010; Chen & Jiang, 2019).
From an empirical perspective, there is also a growing trend to adopt FQI for empirical analysis of deep ofﬂine Q-learning algorithms (Fu et al., 2019; Levine et al., 2020).
In MARL, the joint Q function class grows exponentially with the number of agents, leading many algorithms (Sunehag et al., 2018;
Rashid et al., 2018) to utilize different value decomposition structures with limited expressiveness to improve scalability.
In this paper, we extend FQI to a multi-agent variant as our grounding theoretical framework for analyzing cooperative multi-agent Q-learning with linear value decomposition.
To achieve superior effectiveness and scalability in multi-agent settings, centralized training with decentralized executing (CTDE) has become a popular MARL paradigm (Oliehoek et al., 2008;
Kraemer & Banerjee, 2016).
Individual-Global-Max (IGM) principle (Son et al., 2019) is a critical concept for value-based CTDE (Mahajan et al., 2019), that ensures the consistency between joint and local greedy action selections and enables effective performance in both training and execution phases.
VDN (Sunehag et al., 2018) utilizes linear value decomposition to satisfy a sufﬁcient condition of
IGM.
The simple additivity structure of VDN has achieved excellent scalability and inspired many follow-up methods.
QMIX (Rashid et al., 2018) proposes a monotonic Q network structure to improve the expressiveness of the factorized function class.
QTRAN (Son et al., 2019) tries to realize the entire IGM function class, but its method is computationally intractable and requires two extra soft regularizations to approximate IGM (which actually loses the IGM guarantee).
QPLEX (Wang et al., 2020a) encodes the IGM principle into the Q network architecture and realizes a complete IGM function class, but it may also have potential limitations in scalability.
Based on the advantages of
VDN’s simplicity and scalability, linear value decomposition becomes very popular in MARL (Son et al., 2019;
Wang et al., 2020a;d).
This paper focuses on the theoretical and empirical understanding of multi-agent Q-learning with linear value decomposition to explore its underlying implications.
N OTATIONS AND P RELIMINARIES 3. M ULTI - AGENT M ARKOV D ECISION P ROCESS (MMDP)
To support theoretical analysis on multi-agent Q-learning, we adopt the framework of MMDP (Boutilier, 1996), a special case of Dec-POMDP (Oliehoek et al., 2016), to model fully cooperative multi-agent decision-making tasks.
MMDP is deﬁned as a tuple M =
(cid:104)N , S , A , P, r, γ (cid:105) .
N ≡ { 1 , . . . , n } is a ﬁnite set of agents.
S is a ﬁnite set of global states.
A denotes the action space for an individual agent.
The joint action a ∈
A ≡ A n is a collection of individual actions [ a i ] ni =1 .
At each timestep t , a selected joint action a t results in a transition s t +1 ∼ P ( ·| s t , a t ) and a global reward signal r ( s t , a t ) .
γ ∈ [0 , 1) is a discount factor.
The goal for MARL is to construct a joint policy
π = (cid:104) π 1 , . . .
, π n (cid:105) maximizing expected discounted rewards
V π ( s ) =
E [ (cid:80) ∞ t =0 γ t r ( s t , π ( s t )) | s 0 =
s ] , where π i : S (cid:55)→
A denotes an individual policy of agent i .
The corresponding action-value function is denoted as Q π ( s, a ) =
r ( s, a )+ γ E s (cid:48) ∼ P ( ·| s, a )
[ V π ( s (cid:48) )] .
We use Q ∗ and V ∗ to denote the action-value function and the state-value function corresponding to the optimal policy π ∗ , respectively.
Dec-POMDP (Oliehoek et al., 2016) is a generalized model of MMDP with the consideration of partial observability.
In Dec-POMDPs, each agent can only access to its local observations rather than the full information of global states.
As inﬁnite-horizon Dec-POMDPs is undecidable in general (Madani et al., 1999), this paper focuses theoretical analyses on settings with full observability.
In practice, partial observability is not a hard constraint.
A Dec-POMDP can be transformed to an
MMDP when communication is available. Many prior efforts have been made to construct efﬁcient communication protocols for exchanging information among agents (Foerster et al., 2016; Das et al., 2019; Wang et al., 2020c).
By constructing belief states through extended observation scopes, these methods can approximately transform learning problems in Dec-POMDPs to that in MMDPs.
From this perspective, we consider MMDP as a simpliﬁcation of notations to make the underlying insights more accessible.
3. C ENTRALIZED T RAINING WITH D ECENTRALIZED E XECUTION (CTDE)
Most deep multi-agent Q-learning algorithms with value decomposition adopt the paradigm of centralized training with decentralized execution (Foerster et al., 2016).
In the training phase, the centralized trainer can access all global information, including global states, shared global rewards, agents’ polices, and value functions.
In the decentralized execution phase, every agent makes individual decisions based on its local observations.
Note that this paper considers MMDP as a simpliﬁed setting which rules out the concerns of partial observability.
Thus our notations do not distinguish the concepts of states and observations.
Individual-Global-Max (IGM) (Son et al., 2019) is a common principle to realize effective decentralized policy execution.
It enforces the action selection consistency between the global joint action-value Q tot and individual action-values [ Q i ] ni =1 , which are speciﬁed as follows:
[Equation]
As stated in Eq.
(2), the additivity constraint adopted by VDN (Sunehag et al., 2018) is a sufﬁcient condition for the IGM constraint stated in Eq.
However, this linear decomposition structure is not a necessary condition and induces a limited joint action-value function class because the linear number of individual functions cannot represent a joint action-value function class, which is exponential with the number of agents.
[Equation]
3. F ITTED Q-I TERATION (FQI) FOR M ULTI -A GENT Q-L EARNING
For multi-agent Q-learning with value decomposition, we use Q tot to denote the global but factorized value function, which can be factorized as a function of individual value functions [ Q i ] ni =1 .
In other words, we can use [ Q i ] ni =1 to represent Q tot .
For brevity, we overload Q to denote both of them.
In the MMDP settings, the shared reward signal can only supervise the training of the joint value function Q tot , which requires us to modify the notation of Bellman optimality operator T as follows:
[Equation]
Fitted Q-iteration (FQI) (Ernst et al., 2005) provides a uniﬁed framework which extends the above operator to solve high-dimensional tasks using function approximation.
It follows an iterative optimization framework based on a given dataset D
= { ( s, a , r, s (cid:48) ) } ,
[Equation]
(cid:34)(cid:18)
[Equation]
max
[Equation]
[Equation]
a
[Equation]
) −
[Equation] tot ( [Equation] a ) (cid:19) 2 (cid:35) , (4) where an initial solution Q (0) is selected arbitrarily from a function class Q .
By constructing a speciﬁc function class Q that only contains instances satisfying the IGM condition stated in Eq.
(1) (Sunehag et al., 2018; Rashid et al., 2018), the centralized training procedure in Eq.
(4) will naturally produce suitable individual values [ Q i ] ni =1 , from which individual policies can be derived for decentralized execution.
M ULTI -A GENT Q-L EARNING WITH L INEAR V ALUE D ECOMPOSITION
In the literature of deep MARL, constructing a speciﬁc value function class Q satisfying the IGM condition is a critical step to realize the paradigm centralized training with decentralized execution.
Linear value decomposition proposed by VDN (Sunehag et al., 2018) is a simple yet effective method to implement this paradigm.
In this section, we provide theoretical analysis towards a deeper understanding of this popular decomposition structure.
Our result is based on a multi-agent variant of ﬁtted Q-iteration (FQI) with linear value decomposition, named FQI-LVD.
We derive the closed-form update rule of FQI-LVD, and then reveal the underlying credit assignment mechanism realized by linear value decomposition learning.
4. M ULTI -A GENT F ITTED Q-I TERATION WITH L INEAR V ALUE D ECOMPOSITION (FQI-LVD)
To provide a clear perspective on the effects of linear value decomposition, we make an additional assumption to simplify the notations and facilitate the analysis.
Assumption 1 (Adequate and Factorizable Dataset) .
The dataset D contains all applicable stateaction pairs ( s, a ) whose empirical probability is factorizable with respect to individual behaviors of multiple agents.
Formally, let p D ( a | s ) denote the empirical probability of joint action a executed on state s , which can be factorized to the production of individual components,
[Equation]
where p D ( a i | s ) denotes the empirical probability of the event that agent i executes a i on state s .
Assumption 1 is based on the fact that an adequate dataset is necessary for FQI algorithms to ﬁnd a feasible solution (Farahmand et al., 2010; Chen & Jiang, 2019).
In practice, the property of factorizable data distribution can be directly induced by a decentralized data collection procedure.
When agents perform fully decentralized execution, the empirical probability of an event ( s, a ) in the collected dataset D is naturally factorized.
Now we deﬁne FQI with linear value decomposition as follows.
Deﬁnition 1 (FQI-LVD) .
Given a dataset D , FQI-LVD speciﬁes the action-value function class
[Equation]
and induces the empirical Bellman operator T LVD D :
[Equation]
( s, a i ) (cid:33)= arg min Q ∈Q LVD (cid:88) ( s, a ) ∈S× A p D ( a | s ) (cid:32) y ( t ) ( s, a ) − n (cid:88) i = Q i ( s, a i ) (cid:33), (7) where ˆ y ( t ) ( s, a , s (cid:48) )
=
r ( s, a ) + γ max a (cid:48) Q ( t ) tot ( s (cid:48) , a (cid:48) ) denotes the sample-based regression target.
y ( t ) ( s, a ) =
( T Q ( t ) ) tot ( s, a ) =
r ( s, a ) + γ E s (cid:48) ∼ P ( ·| s, a ) (cid:104) max a (cid:48) Q ( t ) tot ( s (cid:48) , a (cid:48) )
(cid:105) denotes the groundtruth target value derived by Bellman optimality operator.
p D ( a , s (cid:48) | s ) =
p D ( a | s ) P ( s (cid:48) | s, a ) denotes the empirical probability of the event that agents execute joint action a on state s and transit to s (cid:48) .
Q tot and [ Q i ] n i =1 refer to the discussion of CTDE deﬁned in Section 3.3.
The proof of Eq.
(7) is deferred to Lemma 1 in Appendix A. Value-decomposition network (VDN) (Sunehag et al., 2018) provides an implementation of FQI-LVD, in which individual value functions [ Q i ] ni =1 are parameterized by deep neural networks, and the joint value function Q tot can be simply formed by their summation.
4. I MPLICIT C REDIT A SSIGNMENT IN L INEAR V ALUE D ECOMPOSITION
In the formulation of FQI-LVD, the empirical Bellman error minimization in Eq.
(7) can be regarded as a weighted linear least-squares problem, which contains n |S||A| variables to form individual value functions [ Q i ] ni =1 and |S||A| n data points corresponding to all entries of the regression target y ( t ) ( s, a ) .
To solve this least-squares problem, we derive a closed-form solution stated in Theorem 1, which can be veriﬁed through Moore-Penrose inverse (Moore, 1920) for weighted linear regression analysis.
Proofs for all theorems, lemmas, and propositions in this paper are deferred to Appendix.
Theorem 1.
Let Q ( t +1) =
T LVD D Q ( t ) denote a single iteration of the empirical Bellman operator.
Then ∀ i ∈ N , ∀ ( s, a ) ∈ S × A , the individual action-value function Q ( t +1) i ( s, a i ) =
[Table]
[Equation]
[Table]
[Equation]
where we denote a i ⊕ a (cid:48)− i = (cid:104) a (cid:48) 1 , . . .
, a (cid:48) i − 1 , a i , a (cid:48)
i +1 , . .
, a (cid:48) n (cid:105) .
a (cid:48)− i denotes the action of all agents except for agent i .
The residue term w ≡ [ w i ] ni =1 is an arbitrary vector satisfying ∀ s , (cid:80) ni
=
w i ( s ) = 0 .
As shown in Theorem 1, the local action-value function Q ( t +1) i consists of three terms.
The ﬁrst term is the expectation of one-step TD target value over the actions of other agents, which evaluates the expected return of executing an individual action a i .
The second term is the expectation of one-step target TD values over all joint actions, which can be regarded as a baseline function evaluating the average performance.
The arbitrary vector w indicates the entire valid individual action-value function space.
We can ignore this term because w does not affect the local action selection of each agent and will be eliminated in the summation operator of linear value decomposition (see Eq.
(2)), which indicates that joint action-value Q ( t +1) tot
[Equation]
has a unique closed-form solution.
We compare the theoretical analysis of FQI-LVD with the empirical results of VDN to demonstrate and verify the accuracy of our closed-form updating rule (see Eq.
(8)) in Section 6.1.
Note that, if we regard the empirical probability p D ( a | s ) within the dataset D as a default policy , the ﬁrst term of Eq.
(8) is the expected value of an individual action a i , and the second term is the expected value of the default policy, which is considered as the counterfactual baseline .
Their difference corresponds to a credit assignment mechanism called counterfactual difference rewards , which has been used by counterfactual multi-agent policy gradient (COMA)
(Foerster et al., 2018).
Implication 1.
As shown in Eq.
(8), linear value decomposition implicitly implements a counterfactual credit assignment mechanism, which is similar to what is used by COMA.
Compared to COMA, this implicit credit assignment is naturally served by empirical Bellman error minimization through linear value decomposition, which is much more scalable.
The extra importance weight ( n − 1) /n brings our derived credit assignment to be more consistent and meaningful in the sense that all global rewards should be assigned to agents.
Consider a simple case where all joint actions generate the same reward signals, Eq.
(8) will assign 1 /n unit of rewards to each agent, but
COMA will assign 0.
This gap will gradually close when n becomes sufﬁciently large.
I MPROVING THE L EARNING S TABILITY OF V ALUE D ECOMPOSITION
In the previous section, we have derived the closed-form update rule of FQI-LVD, which reveals the underlying credit assignment mechanism of linear value decomposition structure.
This derivation also enables us to investigate more algorithmic functionalities of linear value decomposition in multi-agent
Q-learning.
Although linear value decomposition holds superior scalability in multi-agent settings, we ﬁnd that FQI-LVD has the potential risk of unbounded divergence from arbitrary initialization.
To improve the stability of linear value decomposition training, we theoretically demonstrate that on-policy data distribution or richer Q function classes can provide some convergence guarantees.
Moreover, we also utilize a concrete MMDP example to visualize our implications.
[Figure]
(cid:7) (cid:7)
[Figure]
5. U NBOUNDED D IVERGENCE IN O FFLINE T RAINING
We will provide an analysis of the convergence of FQI-LVD with ofﬂine training on a dataset D .
Proposition 1.
The empirical Bellman operator T LVD D is not a γ -contraction, i.e., the following important property of the standard Bellman optimality operator T does not hold for T LVD D anymore.
[Equation]
For the standard Bellman optimality operator T (Sutton & Barto, 2018), γ -contraction is critical to derive the theoretical guarantee.
In the context of FQI-LVD, the additivity constraint limits the joint action-value function class that it can express, which deviates the empirical Bellman operator
T LVD D from the original Bellman optimality operator T (see Theorem 1).
This deviation is induced by the negative importance weight ( n − 1) /n stated in Eq.
(8) and is also known as inherent Bellman error (Munos & Szepesvári, 2008), which corrupts a broad set of stability properties, including γ -contraction.
To serve a concrete example, we construct a simple MMDP with two agents, two global states, and two actions (see Figure 1a).
The optimal policy of this MMDP is simply executing the action A (1) at state s 2 , which is the only way for two agents to obtain a positive reward.
The learning curve of (cid:15) = 1 .
(green one) in Figure 1b refers to an ofﬂine setting with uniform data distribution, in which an unbounded divergence can be observed as depicted by the following proposition.
Proposition 2.
There exist MMDPs such that, when using uniform data distribution, the value function of FQI-LVD diverges to inﬁnity from an arbitrary initialization Q (0) .
Note that the unbounded divergence discussed in Proposition 2 would happen to an arbitrary initialization Q (0) .
To provide an implication for practical scenarios, we also investigate the performance of several deep multi-agent Q-learning algorithms in this MMDP.
As shown in Figure 1c, VDN (Sunehag et al., 2018), a deep-learning-based implementation of FQI-LVD, results in unbounded divergence.
We postpone the discussion of other deep-learning-based algorithms to the next subsection.
5. L OCAL AND G LOBAL C ONVERGENCE I MPROVEMENTS
To improve the training stability of FQI-LVD, we investigate methods to enable local and global convergence of value decomposition learning, respectively.
Local Convergence Improvement.
As shown in Theorem 1, the choice of training data distribution affects the output of the empirical Bellman operator T LVD D .
We ﬁnd that FQI-LVD has a local convergence property in an on-policy mode, i.e., the dataset D is accumulated by running an (cid:15) -greedy policy (Mnih et al., 2015).
Here we include an informal statement of local stability of FQI-LVD and defer the precise version, its proof, and the algorithm box of on-policy FQI-LVD to Appendix C.1.
Theorem 2 (Informal) .
On-policy FQI-LVD will locally converge to the optimal policy and have a ﬁxed point value function when the hyper-parameter (cid:15) is sufﬁciently small.
Theorem 2 indicates that multi-agent Q-learning with linear value decomposition has a convergent region, where the value function induces optimal actions.
By combining this local stability with
Brouwer’s ﬁxed-point theorem (Brouwer, 1911), we can further verify the existence of a ﬁxed-point solution for the on-policy Bellman operator T LVD D t .
Figure 1b visualizes the performance of on-policy
FQI-LVD with different values of the hyper-parameter (cid:15) .
With a smaller (cid:15) (such as 0.1 or 0.01), on-policy FQI-LVD demonstrates numerical stability, and their corresponding collected datasets are closer to on-policy data distribution.
Global Convergence Improvement.
Linear value decomposition structure limits the joint actionvalue function class Q LVD , which is the origin of the deviation of the empirical Bellman operator T LVD D , discussed in Proposition 1.
Another way to improve training stability is to enrich the expressiveness of value decomposition.
We consider a multi-agent ﬁtted Q-iteration (FQI) with a full action-value function class derived from IGM, named FQI-IGM, whose action-value function class is as follows:
[Equation]
i =1 with Eq.
(1) is satisﬁed (cid:111) .
Note that Q LVD ⊂ Q IGM indicates that linear decomposition structure stated in Eq.
(2) is a sufﬁcient condition for the IGM constraint.
The formal deﬁnition of FQI-IGM is deferred to Appendix C.2 and its global convergence property is established by the following theorem.
Theorem 3.
FQI-IGM will globally converge to the optimal value function.
Theorem 3 relies on a fact that Q IGM is complete in MMDP settings, i.e., inherent Bellman errors discussed in Proposition 1 can reach zero and its empirical Bellman operator T IGM D is a γ -contraction.
Using universal function approximation of neural networks, QPLEX (Wang et al., 2020a), a deeplearning-based implementation of FQI-IGM, theoretically realizes the complete IGM function class.
QTRAN (Son et al., 2019) is an approximate implementation of FQI-IGM which uses soft penalties to realize IGM constraints.
As the basis of comparison, VDN (Sunehag et al., 2018) is the deeplearning-based implementation of FQI-LVD.
An intermediate version, QMIX (Rashid et al., 2018), establishes a non-linear monotonic mapping between local and global value functions.
The value function class of QMIX can be summarized as follows:
[Equation]
which is known to underrepresent the IGM function class since the monotonic correspondence is not necessary for the IGM constraint stated in Eq.
(1) (Mahajan et al., 2019).
Formally, Q LVD ⊂
Q QMIX ⊂ Q IGM is a sequence of strict inclusion relations.
As shown in Figure 1c, QPLEX and QTRAN, two algorithms with representational capacity Q IGM , perform outstanding numerical stability in the proposed MDP example.
By contrast, the phenomenon of unbounded divergence happens to both VDN and QMIX, whose function classes are incomplete in terms of the IGM constraint.
This experiment is a didactic study connection between our theoretical implications and practical algorithms.
Combining the theoretical and empirical results above, we summarize this section by the following insights.
Implication 2.
Multi-agent Q-learning with linear value decomposition potentially suffers from the risk of unbounded divergence from arbitrary initialization.
On-policy data distribution or richer Q function classes can improve its local or global convergence, respectively.
E MPIRICAL A NALYSIS
In this section, we conduct an empirical study to connect our theoretical implications to practical scenarios of deep multi-agent Q-learning algorithms.
An empirical analysis of a didactic example, a two-state MMDP, has been carried out in Section 5, which shows that the linear value decomposition structure needs to improve training stability in ofﬂine mode.
In order to verify other implications, here we evaluate four state-of-the-art deep-learning-based methods, VDN (Sunehag et al., 2018),
QMIX (Rashid et al., 2018), QTRAN (Son et al., 2019), and QPLEX (Wang et al., 2020a) on the matrix game proposed by QTRAN and StarCraft Multi-Agent Challenge (SMAC) benchmark tasks (Samvelyan et al., 2019).
The implementation details of four baselines and experimental settings are deferred to Appendix F.
We test all experiments with 6 random seeds and demonstrate them with median performance and 25-75% percentiles. [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Figure]
M e d i a n
T e s t W i n R a t e % (l) 3h_vs_4z
6. I S O UR C LOSED -F ORM U PDATE R ULE OF L INEAR V ALUE D ECOMPOSITION C ONSISTENT WITH THE D EEP -L EARNING -B ASED E MPIRICAL R ESULTS ?
As shown in Theorem 1, we derive the closed-form update rule of FQI-LVD.
From an optimization perspective, FQI-LVD and VDN share the same objective function (see Deﬁnition 1) but have different optimization methods, i.e., arg min vs. gradient descent.
Starting from a common matrix game used by QTRAN (Son et al., 2019) and QPLEX (Wang et al., 2020a) stated in Table 1a, we will illustrate the correctness of our closed-form formulation.
This matrix game describes a simple cooperative multi-agent, which includes two agents and three actions.
Miscoordination penalties are also considered and the optimal strategy for two agents is to perform action A (1) simultaneously.
We adopt a uniform data distribution to conduct this didactic example.
Table 1b and 1c show the joint action-value functions of FQI-LVD and VDN, respectively.
Comparing with these two joint action-value functions, we ﬁnd that the estimation error of VDN is only (cid:107) Q FQI-LVDtot − Q VDNtot (cid:107) ∞ = 0 .
01 , which corresponds to a 0.2% relative error.
This simulation result strongly illustrates the accuracy of Theorem 1.
A learning curve of relative error for Table 1c is provided in Appendix G.1.
In addition, as discussed by QTRAN and QPLEX, VDN with limited function class cannot learn the optimal policy in this didactic matrix game.
The joint action-value functions of QPLEX, QTRAN, and QMIX are deferred to Appendix G.2, where QPLEX and QTRAN can solve this task, but QMIX cannot.
6. I S L INEAR V ALUE D ECOMPOSITION L IMITED IN O FFLINE T RAINING ?
Section 5 shows that in ofﬂine training mode, linear value decomposition is limited in a didactic
MMDP task.
In order to generalize our implications to complex domains, we investigate the collection.
Recently, ofﬂine reinforcement learning has attracted great attention because it can equip with multi-source datasets and is regarded as a key step towards real-world applications (DulacArnold et al., 2019; Levine et al., 2020).
Differing from other related work studying distributional shift (Fujimoto et al., 2019; Levine et al., 2020; Yu et al., 2020), we aim to adopt a diverse dataset to investigate the effect of the expressiveness of a value decomposition structure on ofﬂine training, i.e., which value decomposition structure is suitable for multi-agent ofﬂine reinforcement learning.
These datasets are constructed by training a behavior policy of VDN (Sunehag et al., 2018) and collecting a ﬁxed number of experienced episodes during the whole training procedure.
We evaluate the learning curve of StarCraft II on nine common maps.
The results are shown in Figure 2.
To approximate the MMDP setting, we concatenate the global state with the local observations for each agent to handle partial observability.
Figure 2(b-d,f-h,j-l) illustrate that VDN (Sunehag et al., 2018) and QMIX (Rashid et al., 2018) performs poorly and cannot utilize well the ofﬂine dataset collected by an unfamiliar behavior policy.
In contrast, QPLEX (Wang et al., 2020a) and
QTRAN (Son et al., 2019) with richer Q function class perform pretty well, which indicates that the expressiveness of value decomposition structures dramatically affects the performance of multi-agent ofﬂine Q-learning.
The learning curves of Behavior line are shown in Figure 2(a,e,i), which is implemented by VDN with (cid:15) -greedy online data collection.
Figure 2(a,e,i) show that VDN with online data collection can solve these nine tasks, but cannot with ofﬂine data collection, that is, there is a considerable gap between online and ofﬂine training with linear value decomposition.
Although the distribution shift (Levine et al., 2020) can be a potential cause of this gap, the remarkable performance of QPLEX and QTRAN suggests that our datasets should be sufﬁcient for ofﬂine training.
In contrast to the theoretical convergence analysis stated in Section 5, in this subsection, empirical experiments aim to conduct the performance of the deep-learning-based implementation of linear value decomposition (i.e., VDN) in the online and ofﬂine data collection settings.
We have designed several comparative experiments to demonstrate two empirical implications shared with the above theoretical understanding (see Implication 2): 1) VDN with online data collection has superior performance than ofﬂine data collection.
VDN, a deep-learning-based algorithm with a linear value decomposition structure, has considerable limitations in ofﬂine training; while QPLEX and
QTRAN, two deep-learning-based algorithms with or approximately with complete IGM function class, are the state-of-the-art value decomposition algorithms for multi-agent ofﬂine training.
C ONCLUSION
This paper makes an initial effort to provide theoretical analysis on multi-agent Q-learning with value decomposition.
We derive a closed-form solution to the empirical Bellman error minimization with linear value decomposition.
Based on this novel result, we reveal the implicit credit assignment mechanism of linear value decomposition learning and provide a formal analysis of its learning stability and convergence.
We also formally show that on-policy training or a richer value function class can improve the stability of factorized multi-agent Q-learning.
Empirical results are conducted with state-of-the-art deep multi-agent Q-learning with value decomposition and verify theoretical insights in both didactic examples and complex StarCraft II benchmark tasks.
To close this paper, we connect our results with an additional related literature named relative overgeneralization pathology (Wiegand, 2003).
In the empirical studies of cooperative learning, the behaviors of individual agents are usually negatively affected by their uncooperative partners.
Focusing on a similar issue, our theoretical analysis of the implicit counterfactual credit assignment provides a detailed characterization to understand this pathological phenomenon in linear value decomposition, which also provides insights for the corresponding feasible solutions.
Regarding relative overgeneralization pathology, coordination graphs (Böhmer et al., 2020) explore a different methodology for cooperative multi-agent Q-learning.
This method allows collaborated action-selection through communications, which does not follow the principle of IGM consistency.
In comparison with linear value decomposition, coordination graphs use a higher-order value decomposition structure (Castellini et al., 2019) in the view of function approximation.
Supplementary to the results of this paper, the value factorization of coordination graphs provides a different and promising perspective for future studies.
A O MITTED P ROOFS IN S ECTION
Deﬁnition 1 (FQI-LVD) .
Given a dataset D , FQI-LVD speciﬁes the action-value function class
[Equation]
and induces the empirical Bellman operator T LVD D :
[Equation]
where ˆ y ( t ) ( s, a , s
[Equation]
) =
[Equation]
+ γ
[Equation]
t )
[Equation]
) denotes the sample-based regression target.
[Equation]
, a (cid:48) ) (cid:105) denotes the groundtruth target value derived by Bellman optimality operator.
p D ( a , s (cid:48) | s ) =
p D ( a | s ) P ( s (cid:48) | s, a ) denotes the empirical probability of the event that agents execute joint action a on state s and transit to s (cid:48) .
Q tot and [ Q i ] ni
=1 refer to the discussion of CTDE deﬁned in Section 3.3.
Lemma 1.
The empirical Bellman operator T LVD D deﬁned in Deﬁnition 1 is equivalent to
[Equation]
Proof.
Recall Deﬁnition 1,
[Equation]
The ﬁrst term is a constant since y ( t ) and ˆ y ( t ) are ﬁxed targets.
The second term is equal to zero since
[Equation]
The third term exactly corresponds to Eq.
Lemma 2.
Considering following weighted linear regression problem
[Equation]
where A ∈ R m n × mn , x ∈ R mn , b , p ∈ R m n , m, n ∈ Z + .
Besides, A is m-ary encoding matrix namely ∀ i ∈
[ m n ] , j ∈ [ mn ]
[Equation]
i
[Equation] [Equation]
otherwise .
[Equation] [Equation]
For simplicity, j th row of A corresponds to a m-ary number (cid:126)a j = ( j ) m where (cid:126)a = a 0 a 1 . . .
a n − 1 , with a u ∈ [ m ] , ∀ u ∈ [ n ] .
Assume p is a positive vector which follows that
[Equation]
p u ( a u,j ) , where p u : [ m ] → (0 , 1) and (cid:88) a u ∈ [ m ] p
[Equation] [Equation]
The optimal solution of this problem is the following.
Denote i =
u
× m + v, v ∈ [ m ] , u ∈ [ n ] and an arbitrary vector w ∈ R mn
[Equation]
Proof.
For brevity, denote
[Equation]
Then the weighted linear regression becomes a standard Linear regression problem w.r.t A p , b p .
To compute the optimal solutions, we need to calculate the Moore-Penrose inverse of A p .
The sufﬁcient and necessary condition of this inverse matrix A p, † ∈ R mn × m n is the following three statements (Moore, 1920):
[Equation]
We consider the following matrix as A p, † and we prove that it satisﬁes all three statements.
For ∀ i ∈ [ mn ] , i =
u
× m + v, u ∈ [ n ] , v ∈ [ m ] , j ∈ [ m n ]
[Equation] [Equation]
(23) where p ( (cid:126)a − u ) =
(cid:81) u (cid:48)
(cid:54) =
u p u (cid:48) ( a u (cid:48) ) .
First, we verify that A p A p, † is a m n × m n self-adjoint matrix in statement (1).
For simplicity,
O ( (cid:126)a i ,(cid:126)a j ) = { u | a u,i = a u,j , u ∈ [ n ] } .
( A p A p, † ) i,j = (cid:88)
u ∈ [ n ]
[Equation]
p
[Equation] [Equation]
[Equation]
Observe that p u ( a u,j ) =
p u ( a u,i ) if a u,i = a u,j , thus ( A p
A p, † ) i,j = ( A p
A p, † ) j,i for any i, j ∈ [ m n ] .
This proves that A p A p, † is self-adjoint.
Second, we prove that A p, † A p is a mn × mn self-adjoint matrix and has surprisingly succinct form.
Let i =
u
× m + v, u ∈ [ n ] , v ∈ [ m ] .
i = i (cid:48) .
Besides, O ( i ) =
{ (cid:126)a ∈ [ m n ] | a u = v } ( A p, † A p ) i,i = (cid:88) (cid:126)a ∈ O
( i ) (cid:112) p ( (cid:126)a )
[ (cid:115) p ( (cid:126)a − u ) p u ( a u ) · 1 ( a u = v ) − n − 1 n (cid:112) p ( (cid:126)a ) − 1 m (cid:115) p ( (cid:126)a − u ) p u ( a u ) +mn n − 1 (cid:88) u (cid:48) =
(cid:115) p ( (cid:126)a − u (cid:48) )
p u (cid:48)
( a u (cid:48) )]
= (cid:88) (cid:126)a ∈ O ( i ) p ( (cid:126)a ) p u ( a u ) −
n −n p ( (cid:126)a ) − 1 m p ( (cid:126)a ) p u ( a u ) + mn n − 1 (cid:88) u (cid:48)
=
p ( (cid:126)a ) p u (cid:48) ( a u (cid:48) )
= (cid:88) (
cid:126)a ∈ O
( i )
(cid:32) p ( (cid:126)a − u ) − 1 m p ( (cid:126)a − u ) + 1 mn n − 1 (cid:88) u (cid:48)
=
p ( (cid:126)a − u (cid:48) )
(cid:33) − n −n p u ( a u = v ) = 1 − 1 m − n − 1 n p u ( a u = v ) + 1 mn (cid:88) u (cid:48)∈ [ n ] u (cid:48) (cid:54) =
u (cid:88) (cid:126)a ∈ O ( i ) p ( (cid:126)a − u (cid:48) )
+mn (cid:88) (cid:126)a ∈ O ( i ) p
[Equation]
i =
u
× m + v, i (cid:48)
=
u
× m + v (cid:48) , v (cid:54) = v (cid:48) .
This
implies
that Q ( i ) ∩ O ( i (cid:48) )
= ∅
[Equation]
A p, † A p ) i,i (cid:48)
= (cid:88)
[Table]
i
[Table]
p
[Equation]
[Table]
[Equation]
i =
u 1 ×
[Equation]
+
v
[Equation] [Equation]
Observe that A p, † A p is self-adjoint by equation (2,3,4) and the expression is succinct.
Third, we verify statement (2).
Since we have computed A p, † A p , the veriﬁcation is straightforward.
For brevity, denote A p, † A p as A p
[Equation]
Thus, A p A p, † A p =
A p .
Similarly, we can verify statement (3).
Suppose i
= u 0 × m + v 0 , we have
[Table]
− 1 (cid:88) u (cid:48) =
(cid:115)
[Equation] [Equation]
u (cid:48) )
[Equation]
(cid:48)
[Equation]
u (cid:48) )]
+
[Equation]
v
[Equation] [Equation]
[Equation]
mn
[Equation] [Equation]
=
[Equation]
mn
[Equation]
+
[Equation]
Clearly
[Equation]
we have the following relations
[Equation]
Thus
[Equation]
This proves A p, † =
A p, † A p
A p, † in statement (3) and A p, † is the Moore-Penrose inverse of A p .
Since the optimal solution x ∗ =
A p, † b p + ( I mn × mn − A p, † A p ) w where w ∈ R mn is any vector (Moore, 1920).
Denote x p =
A p, † b p .
We have ∀ i =
u
× m + v
[Equation]
From equation (2, 3, 4), we have i =
u
× m + v, i (cid:48)
[Equation]
If we consider w as the following i 0 =
[Equation]
Then for i = u
× m + v
[Equation]
Notice that this is exactly the last two terms in equation (5).
Therefore
[Equation]
the optimal solutions of this weighted linear regression problem can be written as: i = u
× m + v, v ∈ [ m ] , u ∈ [ n ] and an arbitrary vector w ∈ R mn .
[Equation]
This completes the proof.
Theorem 1.
Let Q ( t +1) =
T LVD D Q ( t ) denote a single iteration of the empirical Bellman operator.
Then ∀ i ∈ N , ∀ ( s, a ) ∈ S × A , the individual action-value function Q ( t +1)
i
[Equation]
[Table]
[Equation]
[Table]
[Equation]
+ w
[Equation]
s
[Equation]
(8) where we denote a i ⊕ a (cid:48)− i = (cid:104) a (cid:48) 1 , . . .
, a (cid:48)
i − 1 , a i , a (cid:48)
i +1 , . .
, a (cid:48) n (cid:105) .
a (cid:48)− i denotes the action of all agents except for agent i .
The residue term w ≡ [ w i ] ni =1 is an arbitrary vector satisfying ∀ s , (cid:80) ni
=
w i ( s ) = 0 .
Proof.
In the formulation of FQI-LVD stated in Deﬁnition 1, the empirical Bellman error minimization in Eq.
(7) can be regarded as a weighted linear least squares problem as follows: ∀ s ∈ S ,
[Equation]
where let m, n ∈ Z + denote the size of action space |A| and the number of agents, respectively; A ∈ R m n × mn denotes the multi-agent credit assignment coefﬁcient matrix of actionvalue functions with linear value decomposition; x ∈ R mn denotes individual action-value functions (cid:104) Q ( t ) i ( s, ·
) ∈ R m (cid:105) n i
=1 under the empirical Bellman error minimization; according to Lemma 1 , b ∈ R m n denotes the regression target y ( t ) ( s, · ) derived by Bellman optimality operator ; p ∈ R m n denotes the empirical probability of joint action a executed on state s , p D ( a | s ) , which can be factorized to the production of individual components illustrated in Assumption 1.
Besides, A is m-ary encoding matrix namely ∀ i ∈
[ m n ] , j ∈ [ mn ]
[Equation]
[Equation] [Equation]
otherwise .
For simplicity, j th row of A corresponds to a m-ary number (cid:126)a j = ( j ) m where (cid:126)a = a 0 a 1 . . .
a n − 1 , with a u ∈ [ m ] , ∀ u ∈ [ n ] .
According to the factorizable empirical probability p D shown in Assumption 1, p is a corresponding positive vector which follows that
[Equation]
p
[Equation]
According to Lemma 2, we derive the optimal solution of this problem is the following.
Denote i =
u
× m + v, v ∈ [ m ] , u ∈ [ n ] and an arbitrary vector w ∈ R mn
[Equation]
which means ∀ i ∈ N , ∀ ( s, a ) ∈ S × A , the individual action-value function Q ( t +1)
i
[Equation]
44) where we denote a i ⊕ a (cid:48)− i = (cid:104) a (cid:48) 1 , . .
, a (cid:48)
i − 1 , a i , a (cid:48)
i +1 , . .
, a (cid:48) n (cid:105) .
a (cid:48)− i denotes the action of all agents except agent i .
The residue term w ≡ [ w i ] ni =1 is an arbitrary vector satisfying ∀ s , (cid:80) ni
=
w i ( s ) = 0 .
B O MITTED P ROOFS IN S ECTION 5.
Proposition 1.
The empirical Bellman operator T LVD D is not a γ -contraction, i.e., the following important property of the standard Bellman optimality operator T does not hold for T LVD D anymore.
[Equation]
Proof.
Assume the empirical Bellman operator T LVD D is a γ -contraction.
For any MMDPs, when using a uniform data distribution, the value function of FQI-LVD will converge (Ernst et al., 2005) because of the contraction of the distance (inﬁnity norm) between any pair of Q .
However, one counterexample is indicated in Proposition 2, which shows that there exists MMDPs such that, when using a uniform data distribution, the value function of FQI-LVD diverges to inﬁnity from an arbitrary initialization Q (0) .
The assumption of γ -contraction is not hold and the empirical Bellman operator
T LVD D is not a γ -contraction.
Proposition 2.
There exist MMDPs such that, when using uniform data distribution, the value function of FQI-LVD diverges to inﬁnity from an arbitrary initialization Q (0) .
Proof.
We consider the following MMDP with 2 agents, 2 states ( s 1 , s 2 ) and each agent ( i = 1 , 2) has 2 actions A ≡ (cid:8) A (1) , A (2) (cid:9) .
The reward function is listed below which r ( s j , a ) denotes the reward of ( s j , a ) , where a = (cid:104) a 1 , a 2 (cid:105) .
[Equation]
Besides, the transition is deterministic.
[Equation]
Furthermore, γ ∈ ( 45 , 1) .
(In practice, γ is usually chosen as 0 . 99 or 0 .
The following proves that this MMDP will diverge for any initialization.
Denote Q ti ( s j , a i ) as the decomposed Q-value of agent i after t th value-iteration at state s j with action a i .
Then, the total Q-value can be described as Q t tot ( s j , a ) =
Q t 1 ( s j , a 1 ) + Q t 2 ( s j , a 2 ) .
For brevity, 0 th Q-value is its initialization.
First, we clarify the process of each iteration.
Since the value-iteration for linear decomposed function class is solving the MSE problem in Lemma 2.
b is target one-step TD-value w.r.t the Q-value of the last iteration.
Through described in Lemma 2, the optimal solution of this MSE problem is not unique.
We can ignore the term of an arbitrary vector w when considering the joint action-value functions because w does not affect the local action selection of each agent and will be eliminated in the summation operator of linear value decomposition.
In addition, under uniformed sampling, we observe that p u ( a u ) = 12 for any (cid:126)a, u .
Then, in equation
[Equation]
Second, we denote V t tot ( s j ) = max a Q t tot ( s j , a ) and observe that ∀ t ≥ 1 , s j
[Equation]
The second equation holds because the transition T and the reward R are symmetric for both agents.
Thus, we omit the subscript of local Q-values as Q t ( s j , a ) when t ≥ 1 .
Third, we analyze the Q-values on state s 1 .
Clearly, its iteration is irrelevant to s 2 .
According to equation 48, ∀ a ∈ A , t ≥
[Equation]
Clearly, when t ≥ 1 , Q t (cid:0) s 1 , A (1) (cid:1)
=
Q t (cid:0) s 1 , A (2) (cid:1) .
Therefore, we observe that Q t ( s 1 , · ) =
γ t q 1 , ∀ t ≥ 1 where q
Last, we consider state s 2 .
It is straightforward to observe the following recursion for t ≥ 2 from equation
[Equation]
We consider some δ > 0 and t δ = (cid:108) log γ δ 6 | q 1 | (cid:109) .
Then, t > t δ
[Equation]
[Equation]
Denote (cid:98) Q t (cid:0) s 2 , A (2) (cid:1)
= γ 4 (cid:98) Q t − 1 (cid:0) s 2 , A (2) (cid:1) − 1+ δ 8 , ∀ t > t δ and (cid:98) Q t δ (cid:0) s 2 , A (2) (cid:1)
=
Q t δ
[Equation]
s 2 , A
[Equation]
2) (cid:1) .
Consequently, Q t ( s 2 , a 2 ) ≥ (cid:98) Q t δ (cid:0) s 2 , A (2) (cid:1) , ∀ t ≥ t δ by equation 54.
Since t ≥ t δ
[Equation] [Equation]
Furthermore, γ ∈ ( 45 , 1) .
There exists some T δ ≥ t δ which
[Equation]
According to equation 52 and let δ < 111 .
[Equation] [Equation]
[Equation]
Similar to equation 54, we observer from equation 52 that ∀ t > T δ = 111 +
[Equation]
60) and
[Equation] [Equation]
Since 5 γ 4 > 1 and the initial point at T δ = 111 + 1 is larger than 18 , this suggests that V t tot ( s 2 ) will eventually diverge.
Noticing that our proof holds with respect to any (cid:8) Q 0 tot ( s j , a ) |∀ j ∈ S , a ∈ A (cid:9) .
Thus, value-iteration on linear decomposed function class w.r.t this MDP will diverge evnetually under any circumstances.
C O MITTED A LGORITHM B OX , T HEOREM , AND D EFINITION IN S ECTION 5. C. L OCAL C ONVERGENCE I MPROVEMENT
[Equation] [Equation]
[Equation] [Equation]
Algorithm 1 is a variant of ﬁtted Q-iteration which adopts an on-policy sample distribution.
At line 3, an exploratory noise is integrated into the greedy policy, since the function approximator generally requires an extensive set of samples to regularize extrapolation values.
Particularly, we investigate a standard exploration module called (cid:15) -greedy, in which every agent takes a small probability to explore actions with non-maximum values.
To make the underlying insights more accessible, we assume the data collection procedure at line 4 can obtain inﬁnite samples, which makes the dataset
D t become a sufﬁcient coverage over the state-action space (see Assumption 1).
This algorithmic framework serves as a foundation for discussions on local stability.
We consider an additional assumption stated as follows.
Assumption 2 (Unique Optimal Policy) .
The optimal policy π ∗ is unique.
The intuitive motivation of this assumption is to have the optimal policy π ∗ be a potential stable solution.
In situations where the optimal policy is not unique, most Q-learning algorithms will oscillate around multiple optimal policies (Simchowitz & Jamieson, 2019), and Assumption 2 helps us to rule out these non-interesting cases.
Based on this setting, the local stability of FQI-LVD can be characterized by the following lemma.
Lemma 3.
There exists a threshold δ > 0 such that the on-policy Bellman operator T LVD (cid:15) is closed in the following subspace B ⊂ Q LVD , when the hyper-parameter (cid:15) is sufﬁciently small.
[Equation]
Formally, ∃ δ > 0 , ∃ (cid:15) > 0 , ∀ Q ∈ B , there must be T LVD (cid:15)
Q
[Equation]
Lemma 3 indicates that once the value function Q steps into the subspace B , the induced policy π Q will converge to the optimal policy π ∗ .
By combining this local stability with Brouwer’s ﬁxedpoint theorem (Brouwer, 1911), we can further verify the existence of a ﬁxed-point solution for the on-policy Bellman operator T LVD (cid:15) (see Theorem 4).
Theorem 4 (Formal version of Theorem 2) .
Besides Lemma 3, Algorithm 1 will have a ﬁxed point value function expressing the optimal policy if the hyper-parameter (cid:15) is sufﬁciently small.
Theorem 4 indicates that, multi-agent Q-learning with linear value decomposition has a convergent region, where the value function induces optimal actions.
Note that Q LVD is a limited function class, which even cannot guarantee to contain the one-step TD target T LVD D
Q .
From this perspective, on-policy data distribution becomes necessary to make the one-step TD target projected to a small set of critical state-action pairs, which help construct the stable subspace B stated in Lemma 3.
G LOBAL C ONVERGENCE I MPROVEMENT
Deﬁnition 2 (FQI-IGM) .
Given a dataset D , FQI-IGM speciﬁes the action-value function class
Q IGM = (cid:110) Q (cid:12)(cid:12)(cid:12) Q tot ∈ R |S||A| n and (cid:104) ∀
Q i ∈ R |S||A| (cid:105) n i =1 with that Eq.
(1) is satisﬁed (cid:111) .
and induces the empirical Bellman operator
[Equation]
where y ( t ) ( s, a ) =
r ( s, a ) + γ max a (cid:48) Q ( t ) tot ( s (cid:48) , a (cid:48) ) denotes the regression target derived by Bellman optimality operator.
Q tot and [ Q i ]
ni =1 refer to the interfaces of CTDE deﬁned in Section 3.3.
Compared with FQI-LVD stated in Deﬁnition 1, the differences are the Q function class, i.e, Q IGM vs.
Q LVD .
D O MITTED P ROOFS OF T HEOREM
Lemma 4.
The empirical Bellman operator T IGM D stated in Deﬁnition 2 is a γ -contraction, i.e., the following important property of the standard Bellman optimality operator T will hold for T IGM D .
[Equation]
Proof.
We want to prove
[Equation]
where P is transition function, V Q ( · )
= max a
[Equation]
A Q tot ( · , a ) , and (cid:104)· , ·(cid:105) is inner product.
According to Eq.
(68) and Lemma 1.5 in RL textbook (Agarwal et al., 2019), we can prove that T IGM D is a γ -contraction.
Eq.
(68) indicates that the empirical Bellman error
[Equation]
Let a ∗ , ( t ) =
(cid:104) a ∗ , ( t ) i (cid:105) n i =1 =
arg max a ∈
A y ( t ) ( s, a ) .
Then, ∀ y ( t ) ( s, · ) , we construct Q tot ( s, a ) =
y ( t ) ( s, a ) and its corresponding local action-value functions [ Q i ]
ni =1 satisfying IGM principle:
[Equation]
when a i = a ∗ , ( t
[Equation]
i
[Equation] [Equation]
a i
[Equation]
= a ∗
[Equation]
To avoid the multiple solutions of arg max operator in a ∗ , ( t ) , we consider the lexicographic order of joint actions as the second priority.
Thus, we illustrate the completeness of IGM function class in
MMDP setting from our construction.
Then, Eq.
(68) is held, and T IGM D is a γ -contraction in MMDP framework.
Theorem 3.
FQI-IGM will globally converge to the optimal value function.
Proof.
Let Q ∗ ( s, a ) = max π ∈ Π Q π ( s, a ) where Π is the space of all policies.
According to Lemmaand Theorem 1.4 in RL textbook (Agarwal et al., 2019), we have that
• There exists a stationary and deterministic policy π such that Q π tot =
Q ∗ tot .
• A vector Q tot ∈ R |S|×|A| n is equal to Q ∗ tot if and only if it satisﬁes Q tot = (cid:0) T IGM D
[Equation]
tot .
• ∀ Q (cid:48) tot
[Equation]
Thus, FQI-IGM will globally converge to optimal value function.
E O MITTED P ROOFS OF A PPENDIX C. E. S OME N OTATIONS
In this section, we only consider the data distribution generated by the optimal joint policy π ∗ .
To simplify the notations, we use ε = (cid:15) |A| to reformulate the exploratory policy generated by (cid:15) -greedy exploration as follows
[Equation]
where ˆ ε = ( |A| − 1) ε .
In addition, we use f ( s, · , · ) to denote the corresponding coefﬁcient in the closed-form updating
[Equation]
where ( T Q ) tot = r
( s, a (cid:48) )
+ γV tot ( s (cid:48) ) denote the precise target values derived by Bellman optimality equation.
Formally, according to Eq.
[Equation]
in which
[Equation]
As a reference indicating whether the learned value function produces the optimal policy, we denote
[Equation]
Notice that π ∗ denotes the optimal policy of the given MDP, so E ( Q ) might be negative for a non-optimal or inaccurate value function Q .
E. O MITTED P ROOFS
Lemma 5.
Given a dataset D generated by the optimal policy π ∗ with (cid:15) -greedy
exploration, for any target value function Q ,
[Equation]
we have
[Equation]
(81) where ( T Q ) tot
( s, a ) =
r ( s, a ) + γV tot ( s (cid:48) ) denotes the regression target generated by Q .
Proof.
[Equation]
In the ﬁrst term
[Equation]
∀ s ∈ S
[Equation]
ε
[Equation]
(cid:96)
[Equation]
(cid:33)
[Equation]
ˆ
[Equation] [Equation]
) n
[Equation]
(cid:96)
[Equation] [Equation]
In the second term
[Equation]
∀ s ∈ S ,
[Equation]
n \{ π ∗ ( s ) }
[Table]
2 n
[Table]
1 − ˆ ε ) h π ∗
[Table] [Table]
≤ (cid:88) a (cid:48) ∈A n \{ π ∗ ( s ) } 2 nε ≤ 2 nε |A| n .
Thus
[Equation] [Equation]
Lemma 6.
Given a dataset D generated by the optimal policy π ∗ with (cid:15) -greedy
exploration, for any target value function Q ,
[Equation]
86) we have
[Equation] [Equation]
where V π ∗ tot ( s )
=
Q tot ( s, π ∗ ( s )) .
Proof.
∀ s ∈ S ,
[Equation]
Let δ = 1 − γ 8 nγ E ( Q ∗ ) .
According to Lemma
[Equation]
with the condition
[Equation]
we have
[Equation]
Notice that
[Equation] [Equation]
The overall statement is ∀ 0 < ε ≤
[Equation]
we have ∀ s ∈
[Equation] [Equation]
Lemma 7.
For any value function Q
[Equation]
the corresponding sub-optimality gap satisﬁes
[Equation]
Proof.
With a slight abuse of notation
[Equation]
let s 1 and s 2 denote the next states while taking actions π ∗ ( s ) and a at the state s
[Equation]
respectively.
According to the deﬁnition
[Equation]
Lemma 8.
Given a dataset D generated by the optimal policy π ∗ with (cid:15) -greedy
exploration, for any target value function Q ,
[Equation]
we have ∀ s ∈ S , ∀ a ∈
A n \ { π ∗ ( s ) } ,
[Equation]
98) where ( T Q ) tot ( s, a ) =
r ( s, a ) + γV tot ( s (cid:48) ) denotes the regression target generated by Q .
Proof.
∀ s ∈ S , ∀ a ∈
A n \ { π ∗ ( s ) } ,
[Equation]
In the ﬁrst term
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
In the second term
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
term
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Combining the above terms
[Table]
we can get
[Equation]
+
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
∞ +  
[Equation]

[Equation]
 
[Equation]
  +
[Equation]
in which
[Equation]
Thus ∀ s ∈ S , ∀ a ∈
A n \ { π ∗ ( s ) } ,
[Equation]
According to Lemma
[Equation]
E ( T Q ) ≥ E ( Q ∗ ) − 2 γ (cid:107) V tot − V ∗ (cid:107) ∞ .
So ∀ s ∈ S , ∀
a ∈
A n \ { π ∗ ( s ) }
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
Lemma 9.
Let B denote a subspace of value functions
[Equation]
Given a dataset D generated by the optimal policy π ∗ with (cid:15) -greedy
exploration,
[Equation]
we have ∀ Q ∈ B , T LVD D Q ∈
ˆ B ⊂ B where
[Equation]
Proof.
According to Lemma 5, with the condition
[Equation]
we have ∀ Q ∈ B , ∀
[Equation] [Equation]
[Equation]
which implies ∀ Q ∈ B , ∀ s ∈ S , ( T
[Equation]
D
[Equation]
According to Lemma 8, with the condition
[Equation]
≤
[Equation]
we have ∀ Q ∈ B , ∀ s ∈ S , ∀
a ∈
A n \ { π ∗ ( s ) } ,
[Equation]
which implies E ( T LVD D
Q ) > 0 .
According to Lemma
[Equation]
with the condition
[Equation]
115) we have ∀ Q ∈ B , ∀ s ∈ S ,
[Equation]
[Equation]
Combing Eq.
[Equation]
and (115), the overall condition is
[Equation]
Lemma 3.
There exists a threshold δ > 0 such that the on-policy Bellman operator T LVD (cid:15) is closed in the following subspace B ⊂ Q LVD , when the hyper-parameter (cid:15) is sufﬁciently small.
[Equation]
Formally, ∃ δ > 0 , ∃ (cid:15) > 0 , ∀ Q ∈ B , there must be T LVD (cid:15)
Q ∈ B .
Proof.
It is implied by Lemma 9.
Theorem 4 (Formal version of Theorem 2) .
Besides Lemma 3, Algorithm 1 will have a ﬁxed point value function expressing the optimal policy if the hyper-parameter (cid:15) is sufﬁciently small.
Proof.
Notice that the state value function is sufﬁcient to determine the target values, so the subspace
B deﬁned in Lemma 9 is a compact and convex space in terms of V tot .
The operator T LVD D is a continuous mapping because it only involves elementary functions.
According to Brouwer’s Fixed
Point Theorem (Brouwer, 1911), there exist Q ∈ B satisfying T LVD D
Q ∈ B .
In addition, according to the deﬁnition stated in Eq.
(109), the ﬁxed point must represent the unique optimal policy since it cannot lie on the boundary with E ( Q )
= 0 .
F E XPERIMENT S ETTINGS AND I MPLEMENTATION D ETAILS F. I MPLEMENTATION D ETAILS
We adopt the PyMARL (Samvelyan et al., 2019) implementation with default hyper-parameters to investigate state-of-the-art multi-agent Q-learning algorithms:
VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2018), QTRAN (Son et al., 2019), and QPLEX (Wang et al., 2020a).
The training time of these algorithms on an NVIDIA RTX 2080TI GPU is about 4 hours to 12 hours, which
[Table]
is depended on the number of agents and the episode length limit of each map.
The performance measure of StarCraft II tasks is the percentage of episodes in which RL agents defeat all enemy units within the limited time constraints, called test win rate .
The dataset providing off-policy exploration is constructed by training a behavior policy of VDN and collecting its 20k, 30k or 50k experienced episodes.
The dataset conﬁgurations are shown in Table 2.
We investigate ﬁve multi-agent Q-learning algorithms over 6 random seeds, which includes 3 different datasets and evaluates two seeds on each dataset.
We train 300 epochs to evaluate the learning performance with a given static dataset, of which 32 episodes are trained in each update, and 160k transitions are trained for each epoch totally.
Moreover, the training process of behavior policy is the same as that discussed in PyMARL (Samvelyan et al., 2019), which has collected a total of 2 million timestep data and anneals the hyper-parameter (cid:15) of (cid:15) -greedy exploration strategy linearly from 1.0 to 0.05 over 50k timesteps.
The target network will be updated periodically after training every 200 episodes.
We call this period of 200 episodes an Iteration , which corresponds to an iteration of FQI-LVD (see Deﬁnition 1).
F. T WO -S TATE MMDP
In the two-state MMDP shown in Figure 1a, due to the GRU-based implementation of the ﬁnitehorizon paradigm in the above ﬁve deep multi-agent Q-learning algorithms, we assume that two agents starting from state s 2 have 100 environmental steps executed by a uniform (cid:15) -greedy exploration strategy ( i.e. , (cid:15) = 1 ).
We use this long-term horizon pattern and uniform (cid:15) -greedy
exploration methods to approximate an inﬁnite-horizon MMDP paradigm with uniform data distribution.
We adopt γ = 0 .
9 to implement FQI-LVD and deep MARL algorithms.
In the FQI-LVD framework,
V max = 11 − γ = 100 as shown in Figure 1b.
Figure 1c demonstrates that Optimal line is approximately (cid:80) 99 i =
γ i = 63 .
4 in one episode of 100 timesteps.
F. S TAR C RAFT II
StarCraft II unit micromanagement tasks consider a combat game of two groups of agents, where
StarCraft II takes built-in AI to control enemy units, and MARL algorithms can control each ally unit to ﬁght the enemies.
Units in two groups can contain different types of soldiers, but these soldiers in the same group should belong to the same race.
The action space of each agent includes no-op, move [direction], attack [enemy id], and stop.
At each timestep, agents choose to move or attack in continuous maps.
MARL agents will get a global reward equal to the amount of damage done to enemy units.
Moreover, killing one enemy unit and winning the combat will bring additional bonuses of 10 and 200, respectively.
The maps of SMAC challenges in this paper are introduced in Table 3 in the episodes of 100 timesteps.
[Table]
G D EFERRED T ABLES AND F IGURES IN S ECTIONG. T
[Figure]
OF T ABLE
[Figure] [Figure]
G.
D
EFERRED T ABLES IN
[Figure]
[Table]
H A BLATION S TUDIES ON N ETWORK C APACITY H. A BLATION S TUDIES IN M ATRIX G
[Table]
means the greedy joint action selection from Q tot .
To address the concern that QPLEX naturally uses more hidden parameters than VDN and QMIX, which may also improve its representational capacity.
To demonstrate that the performance gap between QPLEX and other methods does not come from the difference in term of the number of parameters, we increase the number of neurons in VDN and QMIX so that they have comparable number of parameters as QPLEX.
Formally, Large-VDN and Large-QMIX have similar number of parameters as QPLEX.
The experiment results are presented in Table 5, both the “Large-” versions of VDN and QMIX cannot represent an accurate value function in this matrix game.
Increasing the number of parameters cannot address the limitations of VDN and QMIX on representational capacity
[Figure]
H. A BLATION S TUDIES IN S TAR C RAFT II B ENCHMARK T ASKS
[Figure]
In addition to the ablation study in the matrix game, Figure 4 and Figure 5 present the ablation studies in StarCraft II benchmark tasks with ofﬂine data collection.
In comparison to the standard versions of VDN and QMIX, we introduce Large-VDN and Large-QMIX which have similar number of parameters as QPLEX.
As shown in Figure 4, increasing parameters can beneﬁt VDN in some
[Figure]
easy maps such as 2s3z and 2s_vs_1sc, but it cannot provide fundamental improvement in harder tasks.
As shown in Figure 5, the effects of increasing parameters are rather weak for QMIX.
These experiments demonstrate that increasing the number of parameters cannot address the limitations of
VDN and QMIX on representational capacity
[Figure]
I A DDITIONAL E XPERIMENTS ON MMDP E XAMPLE
[Figure]
Remark Assumption 1 on the factorizable dataset does not require the factorizability of the underlying transition and reward functions or the decomposability of the joint action-value function.
On the contrary
[Figure]
all our theorems and examples focus on the situations where the joint Q-function cannot be perfectly factorized.
In addition, Assumption 1 can be naturally satisﬁed when the dataset is collected by decentralized execution of agents’ policies, e.g., an on-policy dataset collected using (cid:15) -greedy exploration policies or an ofﬂine dataset collected by given decentralized policies of agents.
All algorithms discussed in paper, including VDN, QMIX, QTRAN, and QPLEX, learn decentralized policies, which are executed in a decentralized manner.
The theoretical implications derived in this paper are applicable whenever such factorizable data collection procedures are carried out.
To investigate the dependency of our theoretical implications on Assumption 1, we provide an experiment to evaluate the performance of deep multi-agent Q-learning algortihms on unfactorizable datasets.
Figure 6 present the learning curves of VDN, QMIX, QPLEX, and QTRAN in the example
MMDP shown in Figure 1a with an unfactorizable dataset D constructed by a parameter η as follows:
[Equation]
As shown in Figure 6, the choice of parameter η has no impacts on the performance of QPLEX and
QTRAN, which matches the fact that Theorem 3 does not rely on the assumption of factorizable dataset.
As the extension of Proposition 2, VDN and QMIX empirically suffer from unbounded divergence when the dataset is not factorizable.
The only exception is the case of η = 1 , in which the dataset only contains two kinds of joint actions.
In this case, the given MMDP degenerates to a single-agent MDP because agents only perform the same actions in the dataset.
As a result, VDN and
QMIX would not diverge in this special situation.