Original: QPLEX: D UPLEX D UELING M ULTI -A GENT Q-L
Revised: QPLEX: D UPLEX D UELING M ULTI -A GENT Q-L

Original: EARNING
Revised: EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We explore value-based multi-agent reinforcement learning (MARL) in the popular paradigm of centralized training with decentralized execution (CTDE).
Revised: We explore value-based multi-agent reinforcement learning (MARL) in the popular paradigm of centralized training with decentralized execution (CTDE).

Original: CTDE has an important concept, Individual-Global-Max (IGM) principle, which requires the consistency between joint and local action selections to support efﬁcient local decision-making.
Revised: CTDE has an important concept, Individual-Global-Max (IGM) principle, which requires the consistency between joint and local action selections to support efﬁcient local decision-making.

Original: However, in order to achieve scalability, existing MARL methods either limit representation expressiveness of their value function classes or relax the IGM consistency, which may suffer from instability risk or lead to poor performance.
Revised: However, in order to achieve scalability, existing MARL methods either limit representation expressiveness of their value function classes or relax the IGM consistency, which may suffer from instability risk or lead to poor performance.

Original: This paper presents a novel MARL approach, called duPLEX dueling multi-agent Q-learning (QPLEX), which takes a duplex dueling network architecture to factorize the joint value function.
Revised: This paper presents a novel MARL approach, called duPLEX dueling multi-agent Q-learning (QPLEX), which takes a duplex dueling network architecture to factorize the joint value function.

Original: This duplex dueling structure encodes the IGM principle into the neural network architecture and thus enables efﬁcient value function learning.
Revised: This duplex dueling structure encodes the IGM principle into the neural network architecture and thus enables efﬁcient value function learning.

Original: Theoretical analysis shows that QPLEX achieves a complete IGM function class.
Revised: Theoretical analysis shows that QPLEX achieves a complete IGM function class.

Original: Empirical experiments on StarCraft II micromanagement tasks demonstrate that QPLEX signiﬁcantly outperforms state-of-the-art baselines in both online and ofﬂine data collection settings and also reveal that
Revised: Empirical experiments on StarCraft II micromanagement tasks demonstrate that QPLEX signiﬁcantly outperforms state-of-the-art baselines in both online and ofﬂine data collection settings and also reveal that

Original: QPLEX achieves high sample efﬁciency and can beneﬁt from ofﬂine datasets without additional online exploration
Revised: QPLEX achieves high sample efﬁciency and can beneﬁt from ofﬂine datasets without additional online exploration

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Cooperative multi-agent reinforcement learning (MARL) has broad prospects for addressing many complex real-world problems, such as sensor networks (Zhang & Lesser, 2011), coordination of robot swarms (Hüttenrauch et al., 2017), and autonomous cars (Cao et al., 2012).
Revised: Cooperative multi-agent reinforcement learning (MARL) has broad prospects for addressing many complex real-world problems, such as sensor networks (Zhang & Lesser, 2011), coordination of robot swarms (Hüttenrauch et al., 2017), and autonomous cars (Cao et al., 2012).

Original: However, cooperative MARL encounters two major challenges of scalability and partial observability in practical applications.
Revised: However, cooperative MARL encounters two major challenges of scalability and partial observability in practical applications.

Original: The joint state-action space grows exponentially as the number of agents increases.
Revised: The joint state-action space grows exponentially as the number of agents increases.

Original: The partial observability and communication constraints of the environment require each agent to make its individual decisions based on local action-observation histories.
Revised: The partial observability and communication constraints of the environment require each agent to make its individual decisions based on local action-observation histories.

Original: To address these challenges, a popular MARL paradigm, called centralized training with decentralized execution (CTDE) (Oliehoek et al., 2008; Kraemer & Banerjee, 2016), has recently attracted great attention, where agents’ policies are trained with access to global information in a centralized way and executed only based on local histories in a decentralized way.
Revised: To address these challenges, a popular MARL paradigm, called centralized training with decentralized execution (CTDE) (Oliehoek et al., 2008; Kraemer & Banerjee, 2016), has recently attracted great attention, where agents’ policies are trained with access to global information in a centralized way and executed only based on local histories in a decentralized way.

Original: Many CTDE learning approaches have been proposed recently, among which value-based MARL algorithms (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019;
Revised: Many CTDE learning approaches have been proposed recently, among which value-based MARL algorithms (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019;

Original: Yang et al., 2020) have shown state-of-the-art performance on challenging tasks, e.g., unit micromanagement in StarCraft II (Samvelyan et al., 2019).
Revised: Yang et al., 2020) have shown state-of-the-art performance on challenging tasks, e.g., unit micromanagement in StarCraft II (Samvelyan et al., 2019).

Original: To enable effective CTDE for multi-agent Q-learning, it is critical that the joint greedy action should be equivalent to the collection of individual greedy actions of agents, which is called the IGM ( Individual-Global-Max ) principle (Son et al., 2019).
Revised: To enable effective CTDE for multi-agent Q-learning, it is critical that the joint greedy action should be equivalent to the collection of individual greedy actions of agents, which is called the IGM ( Individual-Global-Max ) principle (Son et al., 2019).

Original: This IGM principle provides two advantages: 1) ensuring the policy consistency during centralized training (learning the joint Q-function) and decentralized execution (using individual Q-functions) and 2) enabling scalable centralized training of computing one-step TD target of the joint Q-function (deriving joint greedy action selection from individual Q-functions).
Revised: This IGM principle provides two advantages: 1) ensuring the policy consistency during centralized training (learning the joint Q-function) and decentralized execution (using individual Q-functions) and 2) enabling scalable centralized training of computing one-step TD target of the joint Q-function (deriving joint greedy action selection from individual Q-functions).

Original: To enable this principle, VDN (Sunehag et al., 2018) and QMIX (Rashid et al., 2018) propose two sufﬁcient conditions of IGM to factorize the joint action-value function.
Revised: To enable this principle, VDN (Sunehag et al., 2018) and QMIX (Rashid et al., 2018) propose two sufﬁcient conditions of IGM to factorize the joint action-value function.

Original: However, these two decomposition methods suffer from structural constraints and limit the joint action-value function class they can represent.
Revised: However, these two decomposition methods suffer from structural constraints and limit the joint action-value function class they can represent.

Original: As shown by Wang et al.
Revised: As shown by Wang et al.

Original: (2020), the incompleteness of the joint value function class may lead to poor performance or potential risk of training instability in the ofﬂine setting (Levine et al., 2020).
Revised: (2020), the incompleteness of the joint value function class may lead to poor performance or potential risk of training instability in the ofﬂine setting (Levine et al., 2020).

Original: To address this structural limitation, QTRAN (Son et al., 2019) proposes a factorization method expressing the complete value function space induced by the IGM consistency.
Revised: To address this structural limitation, QTRAN (Son et al., 2019) proposes a factorization method expressing the complete value function space induced by the IGM consistency.

Original: However, its exact implementation is known to be computationally intractable.
Revised: However, its exact implementation is known to be computationally intractable.

Original: The approximate version of QTRAN requires two extra soft regularizations and performs poorly in complex domains with online data collection (Mahajan et al., 2019).
Revised: The approximate version of QTRAN requires two extra soft regularizations and performs poorly in complex domains with online data collection (Mahajan et al., 2019).

Original: Therefore, achieving effective scalability remains an open problem for cooperative MARL.
Revised: Therefore, achieving effective scalability remains an open problem for cooperative MARL.

Original: To address this challenge, this paper presents a novel MARL approach, called duPLEX dueling multiagent Q-learning (QPLEX), that takes a duplex dueling network architecture to factorize the joint action-value function into individual action-value functions.
Revised: To address this challenge, this paper presents a novel MARL approach, called duPLEX dueling multiagent Q-learning (QPLEX), that takes a duplex dueling network architecture to factorize the joint action-value function into individual action-value functions.

Original: QPLEX introduces the dueling structure
Revised: QPLEX introduces the dueling structure

Original: Q = V + A
Revised: Q = V + A

Original: (Wang et al., 2016) for representing both joint and individual (duplex) action-value functions and then reformalizes the IGM principle as an advantage-based IGM .
Revised: (Wang et al., 2016) for representing both joint and individual (duplex) action-value functions and then reformalizes the IGM principle as an advantage-based IGM .

Original: This reformulation transforms the IGM consistency into the constraints on the value range of the advantage functions and thus facilitates the action-value function learning with linear decomposition structure.
Revised: This reformulation transforms the IGM consistency into the constraints on the value range of the advantage functions and thus facilitates the action-value function learning with linear decomposition structure.

Original: Unlike
Revised: Unlike

Original: QTRAN that uses soft constraints and loses the guarantee of exact IGM consistency (Son et al., 2019), QPLEX takes advantage of a duplex dueling architecture to encode it into the neural network structure and provide a guaranteed IGM consistency.
Revised: QTRAN that uses soft constraints and loses the guarantee of exact IGM consistency (Son et al., 2019), QPLEX takes advantage of a duplex dueling architecture to encode it into the neural network structure and provide a guaranteed IGM consistency.

Original: To our best knowledge, QPLEX is the ﬁrst multi-agent Q-learning algorithm that effectively achieves high scalability with a full realization of the IGM principle.
Revised: To our best knowledge, QPLEX is the ﬁrst multi-agent Q-learning algorithm that effectively achieves high scalability with a full realization of the IGM principle.

Original: We evaluate the performance of QPLEX in both didactic problems proposed by prior work (Son et al., 2019; Wang et al., 2020) and a range of unit micromanagement benchmark tasks in StarCraft II (Samvelyan et al., 2019).
Revised: We evaluate the performance of QPLEX in both didactic problems proposed by prior work (Son et al., 2019; Wang et al., 2020) and a range of unit micromanagement benchmark tasks in StarCraft II (Samvelyan et al., 2019).

Original: In these didactic problems, QPLEX demonstrates its full representation expressiveness, thereby learning the optimal policy and avoiding the potential risk of training instability.
Revised: In these didactic problems, QPLEX demonstrates its full representation expressiveness, thereby learning the optimal policy and avoiding the potential risk of training instability.

Original: Empirical results on more challenging StarCraft II tasks show that QPLEX signiﬁcantly outperforms other multi-agent Q-learning baselines in online and ofﬂine data collections.
Revised: Empirical results on more challenging StarCraft II tasks show that QPLEX signiﬁcantly outperforms other multi-agent Q-learning baselines in online and ofﬂine data collections.

Original: It is particularly interesting that QPLEX shows the ability to support ofﬂine training, which is not possessed by other baselines.
Revised: It is particularly interesting that QPLEX shows the ability to support ofﬂine training, which is not possessed by other baselines.

Original: This ability not only provides QPLEX with high stability and sample efﬁciency but also with opportunities to efﬁciently utilize multi-source ofﬂine data without additional online exploration (Fujimoto et al., 2019;
Revised: This ability not only provides QPLEX with high stability and sample efﬁciency but also with opportunities to efﬁciently utilize multi-source ofﬂine data without additional online exploration (Fujimoto et al., 2019;

Original: Fu et al., 2020; Levine et al., 2020;
Revised: Fu et al., 2020; Levine et al., 2020;

Original: Yu et al., 2020).
Revised: Yu et al., 2020).

Original:  P RELIMINARIES 2. D ECENTRALIZED P ARTIALLY O BSERVABLE MDP (D EC -POMDP)
Revised:  P RELIMINARIES 2. D ECENTRALIZED P ARTIALLY O BSERVABLE MDP (D EC -POMDP)

Original: We model a fully cooperative multi-agent task as a Dec-POMDP (Oliehoek et al., 2016) deﬁned by a tuple M = (cid:104)N , S , A , P, Ω, O, r, γ (cid:105) , where N ≡ { 1 , 2 , . . . , n } is a ﬁnite set of agents and s ∈ S is a ﬁnite set of global states.
Revised: We model a fully cooperative multi-agent task as a Dec-POMDP (Oliehoek et al., 2016) deﬁned by a tuple M = (cid:104)N , S , A , P, Ω, O, r, γ (cid:105) , where N ≡ { 1 , 2 , . . . , n } is a ﬁnite set of agents and s ∈ S is a ﬁnite set of global states.

Original: At each time step, every agent i ∈ N chooses an action a i ∈
Revised: At each time step, every agent i ∈ N chooses an action a i ∈

Original: A ≡ {A (1) , . . .
Revised: A ≡ {A (1) , . . .

Original: , A ( |A| ) } on a global state s , which forms a joint action a ≡ [ a i ] ni =1 ∈
Revised: , A ( |A| ) } on a global state s , which forms a joint action a ≡ [ a i ] ni =1 ∈

Original: A ≡ A n .
Revised: A ≡ A n .

Original: It results in a joint reward r ( s, a ) and a transition to the next global state s (cid:48) ∼ P ( ·| s, a ) .
Revised: It results in a joint reward r ( s, a ) and a transition to the next global state s (cid:48) ∼ P ( ·| s, a ) .

Original: γ ∈ [0 , 1) is a discount factor.
Revised: γ ∈ [0 , 1) is a discount factor.

Original: We consider a partially observable setting, where each agent i receives an individual partial observation o i ∈ Ω
Revised: We consider a partially observable setting, where each agent i receives an individual partial observation o i ∈ Ω

Original: according to the observation probability function O
Revised: according to the observation probability function O

Original: ( o i | s, a i ) .
Revised: ( o i | s, a i ) .

Original: Each agent i has an action-observation history τ i ∈ T ≡ ( Ω × A ) ∗ and constructs its individual policy π i ( a | τ i ) to jointly maximize team performance.
Revised: Each agent i has an action-observation history τ i ∈ T ≡ ( Ω × A ) ∗ and constructs its individual policy π i ( a | τ i ) to jointly maximize team performance.

Original: We use τ ∈ T ≡ T n to denote joint action-observation history.
Revised: We use τ ∈ T ≡ T n to denote joint action-observation history.

Original: The formal objective function is to ﬁnd a joint policy
Revised: The formal objective function is to ﬁnd a joint policy

Original: π = (cid:104) π 1 , . . .
Revised: π = (cid:104) π 1 , . . .

Original: , π n (cid:105) that maximizes a joint value function V π ( s ) =
Revised: , π n (cid:105) that maximizes a joint value function V π ( s ) =

Original: E [ (cid:80) ∞ t =0 γ t r t | s 0 = s, π ] .
Revised: E [ (cid:80) ∞ t =0 γ t r t | s 0 = s, π ] .

Original: Another quantity of interest in policy search is the joint action-value function Q π ( s, a ) =
Revised: Another quantity of interest in policy search is the joint action-value function Q π ( s, a ) =

Original: r ( s, a ) + γ E s (cid:48)
Revised: r ( s, a ) + γ E s (cid:48)

Original: [ V π ( s (cid:48) )] .
Revised: [ V π ( s (cid:48) )] .

Original: 2. D EEP M ULTI -A GENT Q-L EARNING IN D EC -POMDP
Revised: 2. D EEP M ULTI -A GENT Q-L EARNING IN D EC -POMDP

Original: Q-learning algorithms is a popular algorithm to ﬁnd the optimal joint action-value function
Revised: Q-learning algorithms is a popular algorithm to ﬁnd the optimal joint action-value function

Original: Q ∗ ( s, a ) =
Revised: Q ∗ ( s, a ) =

Original: r ( s, a )+ γ E s (cid:48)
Revised: r ( s, a )+ γ E s (cid:48)

Original: [max a (cid:48)
Revised: [max a (cid:48)

Original: Q ∗ ( s (cid:48) , a (cid:48) )] .
Revised: Q ∗ ( s (cid:48) , a (cid:48) )] .

Original: Deep Q-learning represents the action-value function with a deep neural network parameterized by θ .
Revised: Deep Q-learning represents the action-value function with a deep neural network parameterized by θ .

Original: Mutli-agent Q-learning algorithms (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019;
Revised: Mutli-agent Q-learning algorithms (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019;

Original: Yang et al., 2020) use a replay memory D to store the transition tuple ( τ , a , r, τ (cid:48) ) , where r is the reward for taking action a at joint action-observation history τ with a transition to τ (cid:48) .
Revised: Yang et al., 2020) use a replay memory D to store the transition tuple ( τ , a , r, τ (cid:48) ) , where r is the reward for taking action a at joint action-observation history τ with a transition to τ (cid:48) .

Original: Due to partial observability, Q ( τ , a ; θ ) is used in place of Q ( s, a ; θ ) .
Revised: Due to partial observability, Q ( τ , a ; θ ) is used in place of Q ( s, a ; θ ) .

Original: Thus, parameters θ are learnt by minimizing the following expected TD error:
Revised: Thus, parameters θ are learnt by minimizing the following expected TD error:

Original: [Equation]
Revised: [Equation]

Original: where V (cid:0) τ (cid:48) ; θ − (cid:1) = max a (cid:48) Q (cid:0) τ (cid:48) , a (cid:48) ; θ − (cid:1) is the one-step expected future return of the TD target and θ − are the parameters of the target network, which will be periodically updated with θ .
Revised: where V (cid:0) τ (cid:48) ; θ − (cid:1) = max a (cid:48) Q (cid:0) τ (cid:48) , a (cid:48) ; θ − (cid:1) is the one-step expected future return of the TD target and θ − are the parameters of the target network, which will be periodically updated with θ .

Original: 2. C ENTRALIZED T RAINING WITH D ECENTRALIZED E XECUTION (CTDE)
Revised: 2. C ENTRALIZED T RAINING WITH D ECENTRALIZED E XECUTION (CTDE)

Original: CTDE is a popular paradigm of cooperative multi-agent deep reinforcement learning (Sunehag et al., 2018;
Revised: CTDE is a popular paradigm of cooperative multi-agent deep reinforcement learning (Sunehag et al., 2018;

Original: Rashid et al., 2018; Son et al., 2019;
Revised: Rashid et al., 2018; Son et al., 2019;

Original: Yang et al., 2020).
Revised: Yang et al., 2020).

Original: Agents are trained in a centralized way and granted access to other agents’ information or the global states during the centralized training process.
Revised: Agents are trained in a centralized way and granted access to other agents’ information or the global states during the centralized training process.

Original: However, due to partial observability and communication constraints, each agent makes its own decision based on its local action-observation history during the decentralized execution phase.
Revised: However, due to partial observability and communication constraints, each agent makes its own decision based on its local action-observation history during the decentralized execution phase.

Original: IGM ( Individual-Global-Max ) (Son et al., 2019) is a popular principle to realize effective value-based
Revised: IGM ( Individual-Global-Max ) (Son et al., 2019) is a popular principle to realize effective value-based

Original: CTDE, which asserts the consistency between joint and local greedy action selections in the joint action-value Q tot ( τ , a ) and individual action-values [ Q i (
Revised: CTDE, which asserts the consistency between joint and local greedy action selections in the joint action-value Q tot ( τ , a ) and individual action-values [ Q i (

Original: [Equation]
Revised: [Equation]

Original: Two factorization structures, additivity and monotonicity , has been proposed by VDN (Sunehag et al., 2018) and QMIX (Rashid et al., 2018), respectively, as shown below:
Revised: Two factorization structures, additivity and monotonicity , has been proposed by VDN (Sunehag et al., 2018) and QMIX (Rashid et al., 2018), respectively, as shown below:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: These two structures are sufﬁcient conditions for the IGM constraint.
Revised: These two structures are sufﬁcient conditions for the IGM constraint.

Original: Qatten (Yang et al., 2020) is a variant of VDN, which supplements global information through a multi-head attention structure.
Revised: Qatten (Yang et al., 2020) is a variant of VDN, which supplements global information through a multi-head attention structure.

Original: However, they are not necessary conditions and limit the representation expressiveness of joint action-value functions (Mahajan et al., 2019).
Revised: However, they are not necessary conditions and limit the representation expressiveness of joint action-value functions (Mahajan et al., 2019).

Original: There exist tasks whose factorizable joint action-value functions can not be represented by these decomposition methods, as shown in Section 4.
Revised: There exist tasks whose factorizable joint action-value functions can not be represented by these decomposition methods, as shown in Section 4.

Original: In contrast,
Revised: In contrast,

Original: QTRAN (Son et al., 2019) transforms IGM into a linear constraint and uses it as soft regularization constrains.
Revised: QTRAN (Son et al., 2019) transforms IGM into a linear constraint and uses it as soft regularization constrains.

Original: However, this relaxation may violate the exact IGM consistency and result in poor performance in complex problems.
Revised: However, this relaxation may violate the exact IGM consistency and result in poor performance in complex problems.

Original:  QPLEX: D UPLEX D UELING M ULTI -A GENT Q-L EARNING
Revised:  QPLEX: D UPLEX D UELING M ULTI -A GENT Q-L EARNING

Original: In this section, we will ﬁrst introduce advantage-based IGM, equivalent to the regular IGM principle, and, with this new deﬁnition, convert the IGM consistency of greedy action selection to simple constraints on advantage functions.
Revised: In this section, we will ﬁrst introduce advantage-based IGM, equivalent to the regular IGM principle, and, with this new deﬁnition, convert the IGM consistency of greedy action selection to simple constraints on advantage functions.

Original: We then present a novel deep MARL model, called duPLEX dueling multi-agent Q-learning algorithm (QPLEX), that directly realizes these constraints by a scalable neural network architecture.
Revised: We then present a novel deep MARL model, called duPLEX dueling multi-agent Q-learning algorithm (QPLEX), that directly realizes these constraints by a scalable neural network architecture.

Original: 3. A DVANTAGE -B ASED IGM
Revised: 3. A DVANTAGE -B ASED IGM

Original: To ensure the consistency of greedy action selection on the joint and local action-value functions, the
Revised: To ensure the consistency of greedy action selection on the joint and local action-value functions, the

Original: IGM principle constrains the relative order of Q-values over actions.
Revised: IGM principle constrains the relative order of Q-values over actions.

Original: From the perspective of dueling decomposition structure Q = V + A proposed by Dueling DQN (Wang et al., 2016), this consistency should only constrain the action-dependent advantage term A and be free of the state-value function
Revised: From the perspective of dueling decomposition structure Q = V + A proposed by Dueling DQN (Wang et al., 2016), this consistency should only constrain the action-dependent advantage term A and be free of the state-value function

Original: V .
Revised: V .

Original: This observation naturally motivates us to reformalize the IGM principle as advantage-based IGM, which transforms the consistency constraint onto advantage functions.
Revised: This observation naturally motivates us to reformalize the IGM principle as advantage-based IGM, which transforms the consistency constraint onto advantage functions.

Original: Deﬁnition 1 (Advantage-based IGM) .
Revised: Deﬁnition 1 (Advantage-based IGM) .

Original: For a joint action-value function Q tot : T × A (cid:55)→ R and individual action-value functions [ Q i : T × A (cid:55)→ R ] ni =1 , where ∀ τ ∈ T , ∀ a ∈ A , ∀ i ∈ N , (Joint Dueling) Q i ) ,
Revised: For a joint action-value function Q tot : T × A (cid:55)→ R and individual action-value functions [ Q i : T × A (cid:55)→ R ] ni =1 , where ∀ τ ∈ T , ∀ a ∈ A , ∀ i ∈ N , (Joint Dueling) Q i ) ,

Original: [Equation]
Revised: [Equation]

Original: such that the following holds
Revised: such that the following holds

Original: [Equation]
Revised: [Equation]

Original: then, we can say that [ Q i ] n i =1 satisﬁes advantage-based IGM for Q tot .
Revised: then, we can say that [ Q i ] n i =1 satisﬁes advantage-based IGM for Q tot .

Original: 𝐴 " 𝝉,𝑎 " "&’( 𝑉 *+* (𝝉) 𝒂 𝑠 * 𝝀 >𝐴 *+* (𝝉,𝒂) 𝑉 " 𝝉 "&’( + 𝑄 *+* (𝝉,𝒂)
Revised: 𝐴 " 𝝉,𝑎 " "&’( 𝑉 *+* (𝝉) 𝒂 𝑠 * 𝝀 >𝐴 *+* (𝝉,𝒂) 𝑉 " 𝝉 "&’( + 𝑄 *+* (𝝉,𝒂)

Original: MLP +
Revised: MLP +

Original: Dot Product (𝑎) (𝑜 "* ,𝑎 "*5’ )
Revised: Dot Product (𝑎) (𝑜 "* ,𝑎 "*5’ )

Original: MLP
Revised: MLP

Original: GRU ℎ "*5’ ℎ "* 𝑄 " (𝜏 " ,𝑎 " )
Revised: GRU ℎ "*5’ ℎ "* 𝑄 " (𝜏 " ,𝑎 " )

Original: MLP 𝑠 * 𝒘 > 𝟎,𝒃 𝑉 " 𝝉 ,𝐴 " (𝝉,𝑎 " )
Revised: MLP 𝑠 * 𝒘 > 𝟎,𝒃 𝑉 " 𝝉 ,𝐴 " (𝝉,𝑎 " )

Original: MLP
Revised: MLP

Original: Weighted Sum (𝑐) 𝑉 " (𝜏 " ),𝐴 " (𝜏 " ,𝑎 " ) ⋯ (𝑜 ’* ,𝑎 ’*5’ )
Revised: Weighted Sum (𝑐) 𝑉 " (𝜏 " ),𝐴 " (𝜏 " ,𝑎 " ) ⋯ (𝑜 ’* ,𝑎 ’*5’ )

Original: (𝑜 (* ,𝑎 (*5’ ) ⋯ 𝑄 ’ (𝜏 ’ ,𝑎 ’ ) 𝑄 *+* (𝝉, 𝒂)
Revised: (𝑜 (* ,𝑎 (*5’ ) ⋯ 𝑄 ’ (𝜏 ’ ,𝑎 ’ ) 𝑄 *+* (𝝉, 𝒂)

Original: Agent
Revised: Agent

Original: Agent 𝑛 (𝑏)
Revised: Agent 𝑛 (𝑏)

Original: TD Loss 𝑄 ( (𝜏 ( ,𝑎 ( ) 𝑉 ( (𝝉),𝐴 ( (𝝉,𝑎 ( ) 𝑉 ’ (𝝉),𝐴 ’ (𝝉,𝑎 ’ ) 𝑠 * ⋯ 𝑠
Revised: TD Loss 𝑄 ( (𝜏 ( ,𝑎 ( ) 𝑉 ( (𝝉),𝐴 ( (𝝉,𝑎 ( ) 𝑉 ’ (𝝉),𝐴 ’ (𝝉,𝑎 ’ ) 𝑠 * ⋯ 𝑠

Original: * 𝑠 * 𝑉 ’ (𝜏 ’ ),𝐴 ’ (𝜏 ’ ,𝑎 ’ )
Revised: * 𝑠 * 𝑉 ’ (𝜏 ’ ),𝐴 ’ (𝜏 ’ ,𝑎 ’ )

Original: 𝑉 ( (𝜏 ( ),𝐴 ( (𝜏 ( ,𝑎 ( )
Revised: 𝑉 ( (𝜏 ( ),𝐴 ( (𝜏 ( ,𝑎 ( )

Original: Duplex Dueling
Revised: Duplex Dueling

Original: Transformation 𝑛 ⋯
Revised: Transformation 𝑛 ⋯

Original: Transformation
Revised: Transformation

Original: Dueling Mixing
Revised: Dueling Mixing

Original: As speciﬁed in Deﬁnition 1, advantage-based IGM takes a duplex dueling architecture, Joint Dueling and Individual Dueling , which induces the joint and local (duplex) advantage functions by A = Q − V .
Revised: As speciﬁed in Deﬁnition 1, advantage-based IGM takes a duplex dueling architecture, Joint Dueling and Individual Dueling , which induces the joint and local (duplex) advantage functions by A = Q − V .

Original: Compared with regular IGM, advantage-based IGM transfers the consistency constraint on actionvalue functions stated in Eq.
Revised: Compared with regular IGM, advantage-based IGM transfers the consistency constraint on actionvalue functions stated in Eq.

Original: (2) to that on advantage functions.
Revised: (2) to that on advantage functions.

Original: This change is an equivalent transformation because the state-value terms V do not affect the action selection, as shown by
Revised: This change is an equivalent transformation because the state-value terms V do not affect the action selection, as shown by

Original: Proposition 1. Proposition 1.
Revised: Proposition 1. Proposition 1.

Original: The advantage-based IGM and IGM function classes are equivalent.
Revised: The advantage-based IGM and IGM function classes are equivalent.

Original: One key beneﬁt of using advantage-based IGM is that its consistency constraint can be directly realized by limiting the value range of advantage functions, as indicated by the following fact.
Revised: One key beneﬁt of using advantage-based IGM is that its consistency constraint can be directly realized by limiting the value range of advantage functions, as indicated by the following fact.

Original: Fact 1.
Revised: Fact 1.

Original: The constraint of advantage-based IGM stated in Eq.
Revised: The constraint of advantage-based IGM stated in Eq.

Original: (6) is equivalent to that when ∀ τ ∈ T , ∀ a ∗ ∈
Revised: (6) is equivalent to that when ∀ τ ∈ T , ∀ a ∗ ∈

Original: A ∗ ( τ ) , ∀ a ∈
Revised: A ∗ ( τ ) , ∀ a ∈

Original: A \ A ∗ ( τ ) , ∀
Revised: A \ A ∗ ( τ ) , ∀

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: 7) where A ∗ ( τ ) =
Revised: 7) where A ∗ ( τ ) =

Original: { a | a ∈ A , Q tot ( τ , a ) =
Revised: { a | a ∈ A , Q tot ( τ , a ) =

Original: V tot ( τ ) } .
Revised: V tot ( τ ) } .

Original: To achieve a full expressiveness power of advantage-based IGM or IGM, Fact 1 enables us to develop an efﬁcient MARL algorithm that allows the joint state-value function learning with any scalable decomposition structure and just imposes simple constraints limiting value ranges of advantage functions.
Revised: To achieve a full expressiveness power of advantage-based IGM or IGM, Fact 1 enables us to develop an efﬁcient MARL algorithm that allows the joint state-value function learning with any scalable decomposition structure and just imposes simple constraints limiting value ranges of advantage functions.

Original: The next subsection will describe such a MARL algorithm.
Revised: The next subsection will describe such a MARL algorithm.

Original: 3. T HE QPLEX A RCHITECTURE
Revised: 3. T HE QPLEX A RCHITECTURE

Original: In this subsection, we present a novel multi-agent Q-learning algorithm with a duplex dueling architecture, called QPLEX, which exploits Fact 1 and realizes the advantage-based IGM constraint.
Revised: In this subsection, we present a novel multi-agent Q-learning algorithm with a duplex dueling architecture, called QPLEX, which exploits Fact 1 and realizes the advantage-based IGM constraint.

Original: The overall architecture of QPLEX is illustrated in Figure 1, which consists of two main components as follows: (i) an Individual Action-Value Function for each agent, and (ii) a Duplex Dueling component that composes individual action-value functions into a joint action-value function under the advantage-based IGM constraint.
Revised: The overall architecture of QPLEX is illustrated in Figure 1, which consists of two main components as follows: (i) an Individual Action-Value Function for each agent, and (ii) a Duplex Dueling component that composes individual action-value functions into a joint action-value function under the advantage-based IGM constraint.

Original: During the centralized training, the whole network is learned in an end-to-end fashion to minimize the TD loss as speciﬁed in Eq.
Revised: During the centralized training, the whole network is learned in an end-to-end fashion to minimize the TD loss as speciﬁed in Eq.

Original: During the decentralized execution, the duplex dueling component will be removed, and each agent will select actions using its individual Q-function based on local action-observation histories.
Revised: During the decentralized execution, the duplex dueling component will be removed, and each agent will select actions using its individual Q-function based on local action-observation histories.

Original: Individual Action-Value Function is represented by a recurrent Q-network for each agent i , which takes previous hidden state h t − 1 i , current local observations o t i , and previous action a t − 1 i as inputs and outputs local Q i ( τ i , a i ) .
Revised: Individual Action-Value Function is represented by a recurrent Q-network for each agent i , which takes previous hidden state h t − 1 i , current local observations o t i , and previous action a t − 1 i as inputs and outputs local Q i ( τ i , a i ) .

Original: Duplex Dueling component connects local and joint action-value functions via two modules: (i) a Transformation network module that incorporates the information of global state or joint history into individual action-value functions during the centralized training process, and (ii) a Dueling
Revised: Duplex Dueling component connects local and joint action-value functions via two modules: (i) a Transformation network module that incorporates the information of global state or joint history into individual action-value functions during the centralized training process, and (ii) a Dueling

Original: Mixing network module that composes separate action-value functions from Transformation into a joint action-value function.
Revised: Mixing network module that composes separate action-value functions from Transformation into a joint action-value function.

Original: Duplex Dueling ﬁrst derives the individual dueling structure for each agent i by computing its value function V i ( τ i )
Revised: Duplex Dueling ﬁrst derives the individual dueling structure for each agent i by computing its value function V i ( τ i )

Original: =
Revised: =

Original: max a i Q i ( τ i , a i ) and its advantage function
Revised: max a i Q i ( τ i , a i ) and its advantage function

Original: A i ( τ i , a i ) =
Revised: A i ( τ i , a i ) =

Original: Q i ( τ i , a i ) − V i ( τ i ) , and then computes the joint dueling structure by using individual dueling structures.
Revised: Q i ( τ i , a i ) − V i ( τ i ) , and then computes the joint dueling structure by using individual dueling structures.

Original: Transformation network module uses the centralized information to transform local dueling structure [ V i ( τ i ) ,
Revised: Transformation network module uses the centralized information to transform local dueling structure [ V i ( τ i ) ,

Original: A i ( τ i , a i )]
Revised: A i ( τ i , a i )]

Original: ni =1 to [ V i ( τ ) , A i ( τ , a i )]
Revised: ni =1 to [ V i ( τ ) , A i ( τ , a i )]

Original: ni =1 conditioned on the joint action-observation history, as shown below, for any agent i , i.e., Q i ( τ , a i ) =
Revised: ni =1 conditioned on the joint action-observation history, as shown below, for any agent i , i.e., Q i ( τ , a i ) =

Original: w i ( τ ) Q i ( τ i , a i )
Revised: w i ( τ ) Q i ( τ i , a i )

Original: + b i ( τ ) , thus,
Revised: + b i ( τ ) , thus,

Original: V i ( τ ) =
Revised: V i ( τ ) =

Original: w i ( τ ) V i ( τ i )
Revised: w i ( τ ) V i ( τ i )

Original: + b i ( τ ) , and A i ( τ , a i ) =
Revised: + b i ( τ ) , and A i ( τ , a i ) =

Original: Q i ( τ , a i ) − V
Revised: Q i ( τ , a i ) − V

Original: i
Revised: i

Original: ( τ ) =
Revised: ( τ ) =

Original: w i ( τ ) A i ( τ i , a i ) , (8) where w i ( τ ) > 0 is a positive weight.
Revised: w i ( τ ) A i ( τ i , a i ) , (8) where w i ( τ ) > 0 is a positive weight.

Original: This positive linear transformation maintains the consistency of the greedy action selection and alleviates partial observability in Dec-POMDP (Son et al., 2019;
Revised: This positive linear transformation maintains the consistency of the greedy action selection and alleviates partial observability in Dec-POMDP (Son et al., 2019;

Original: Yang et al., 2020).
Revised: Yang et al., 2020).

Original: As used by QMIX (Rashid et al., 2018)
Revised: As used by QMIX (Rashid et al., 2018)

Original: , QTRAN (Son et al., 2019), and Qatten (Yang et al., 2020), the centralized information can be the global state s , if available, or the joint action-observation history τ .
Revised: , QTRAN (Son et al., 2019), and Qatten (Yang et al., 2020), the centralized information can be the global state s , if available, or the joint action-observation history τ .

Original: Dueling Mixing network module takes the outputs of the transformation network as input, e.g., [ V i , A i ] ni =1 , and produces the values of joint Q tot , as shown in Figure 1a.
Revised: Dueling Mixing network module takes the outputs of the transformation network as input, e.g., [ V i , A i ] ni =1 , and produces the values of joint Q tot , as shown in Figure 1a.

Original: This dueling mixing network uses individual dueling structure transformed by Transformation to compute the joint value V tot ( τ ) and the joint advantage A tot ( τ , a ) , respectively, and ﬁnally outputs Q tot ( τ , a ) =
Revised: This dueling mixing network uses individual dueling structure transformed by Transformation to compute the joint value V tot ( τ ) and the joint advantage A tot ( τ , a ) , respectively, and ﬁnally outputs Q tot ( τ , a ) =

Original: V tot ( τ ) + A tot ( τ , a ) by using the joint dueling structure.
Revised: V tot ( τ ) + A tot ( τ , a ) by using the joint dueling structure.

Original: Based on Fact 1, the advantage-based IGM principle imposes no constraints on value functions.
Revised: Based on Fact 1, the advantage-based IGM principle imposes no constraints on value functions.

Original: Therefore, to enable efﬁcient learning, we use a simple sum structure to compose the joint value:
Revised: Therefore, to enable efﬁcient learning, we use a simple sum structure to compose the joint value:

Original: [Equation]
Revised: [Equation]

Original: To enforce the IGM consistency of the joint advantage and individual advantages, as speciﬁed by
Revised: To enforce the IGM consistency of the joint advantage and individual advantages, as speciﬁed by

Original: Eq.
Revised: Eq.

Original: (7), QPLEX computes the joint advantage function as follows:
Revised: (7), QPLEX computes the joint advantage function as follows:

Original: [Equation]
Revised: [Equation]

Original: The joint advantage function A tot is the dot product of separate advantage functions [ A i ] ti =1 and positive importance weights [ λ i ] ni =1 .
Revised: The joint advantage function A tot is the dot product of separate advantage functions [ A i ] ti =1 and positive importance weights [ λ i ] ni =1 .

Original: This positivity induced by λ i will continue to maintain the consistency ﬂow of the greedy action selection.
Revised: This positivity induced by λ i will continue to maintain the consistency ﬂow of the greedy action selection.

Original: To enable efﬁcient learning of importance weights λ i with joint history and action, QPLEX uses a scalable multi-head attention module (Vaswani et al., 2017):
Revised: To enable efﬁcient learning of importance weights λ i with joint history and action, QPLEX uses a scalable multi-head attention module (Vaswani et al., 2017):

Original: [Equation]
Revised: [Equation]

Original: where K is the number of attention heads, λ i,k ( τ , a ) and φ i,k ( τ ) are attention weights activated by a sigmoid regularizer, and υ k ( τ ) > 0 is a positive key of each head.
Revised: where K is the number of attention heads, λ i,k ( τ , a ) and φ i,k ( τ ) are attention weights activated by a sigmoid regularizer, and υ k ( τ ) > 0 is a positive key of each head.

Original: This sigmoid activation of λ i brings sparsity to the credit assignment of the joint advantage function to individuals, which enables efﬁcient multi-agent learning (Wang et al., 2019).
Revised: This sigmoid activation of λ i brings sparsity to the credit assignment of the joint advantage function to individuals, which enables efﬁcient multi-agent learning (Wang et al., 2019).

Original: With Eq.
Revised: With Eq.

Original: (9) and (10), the joint action-value function Q tot can be reformulated as follows:
Revised: (9) and (10), the joint action-value function Q tot can be reformulated as follows:

Original: [Equation]
Revised: [Equation]

Original: It can be seen that Q tot consists of two terms.
Revised: It can be seen that Q tot consists of two terms.

Original: The ﬁrst term is the sum of separate action-value functions [ Q i ] ni =1 , which is the joint action-value function Q Qatten tot of Qatten (Yang et al., 2020) (which is the Q tot of VDN (Sunehag et al., 2018) with global information).
Revised: The ﬁrst term is the sum of separate action-value functions [ Q i ] ni =1 , which is the joint action-value function Q Qatten tot of Qatten (Yang et al., 2020) (which is the Q tot of VDN (Sunehag et al., 2018) with global information).

Original: The second term corrects for the discrepancy between the centralized joint action-value function and Q Qatten tot , which is the main contribution of QPLEX to realize the full expressiveness power of value factorization.
Revised: The second term corrects for the discrepancy between the centralized joint action-value function and Q Qatten tot , which is the main contribution of QPLEX to realize the full expressiveness power of value factorization.

Original: Proposition 2.
Revised: Proposition 2.

Original: Given the universal function approximation of neural networks, the action-value function class that QPLEX can realize is equivalent to what is induced by the IGM principle.
Revised: Given the universal function approximation of neural networks, the action-value function class that QPLEX can realize is equivalent to what is induced by the IGM principle.

Original: In practice, QPLEX can utilize common neural network structures (e.g., multi-head attention modules) to achieve superior performance by approximating the universal approximation theorem (Csáji et al., 2001).
Revised: In practice, QPLEX can utilize common neural network structures (e.g., multi-head attention modules) to achieve superior performance by approximating the universal approximation theorem (Csáji et al., 2001).

Original: We will discuss the effects of QPLEX’s duplex dueling network with different conﬁgurations in Section 4.1.
Revised: We will discuss the effects of QPLEX’s duplex dueling network with different conﬁgurations in Section 4.1.

Original: As introduced by Son et al.
Revised: As introduced by Son et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: (2020), the completeness of value factorization is very critical for multi-agent Q-learning and we will illustrate the stability and state-ofthe-art performance of QPLEX in online and ofﬂine data collections in the next section.
Revised: (2020), the completeness of value factorization is very critical for multi-agent Q-learning and we will illustrate the stability and state-ofthe-art performance of QPLEX in online and ofﬂine data collections in the next section.

Original: aa
Revised: aa

Original: A (1)
Revised: A (1)

Original: A (2)
Revised: A (2)

Original: A (3)
Revised: A (3)

Original: A (1) - -
Revised: A (1) - -

Original: A (2) - ( (cid:1) 0)
Revised: A (2) - ( (cid:1) 0)

Original: A (3) -  ( (cid:1) 0)(a) Payoff of a harder matrix game 100 200 300 400 500 Iterations
Revised: A (3) -  ( (cid:1) 0)(a) Payoff of a harder matrix game 100 200 300 400 500 Iterations

Original: M e d i a n
Revised: M e d i a n

Original: T e s t R e t u r n
Revised: T e s t R e t u r n

Original: QPLEXQTRANQMIX
Revised: QPLEXQTRANQMIX

Original: VDNQattenOptimal (b) Deep MARL algorithms     750 Iterations
Revised: VDNQattenOptimal (b) Deep MARL algorithms     750 Iterations

Original: M e d i a n
Revised: M e d i a n

Original: T e s t R e t u r n
Revised: T e s t R e t u r n

Original: QPLEX-3L10HQPLEX-3L4H
Revised: QPLEX-3L10HQPLEX-3L4H

Original: QPLEX-2L10HQPLEX-2L4H
Revised: QPLEX-2L10HQPLEX-2L4H

Original: Optimal (c) Learning curves of ablation study
Revised: Optimal (c) Learning curves of ablation study

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we ﬁrst study didactic examples proposed by prior work (Son et al., 2019; Wang et al., 2020) to investigate the effects of QPLEX’s complete IGM expressiveness on learning optimality and stability.
Revised: In this section, we ﬁrst study didactic examples proposed by prior work (Son et al., 2019; Wang et al., 2020) to investigate the effects of QPLEX’s complete IGM expressiveness on learning optimality and stability.

Original: To demonstrate scalability on complex MARL domains, we also evaluate the performance of QPLEX on a range of StarCraft II benchmark tasks (Samvelyan et al., 2019).
Revised: To demonstrate scalability on complex MARL domains, we also evaluate the performance of QPLEX on a range of StarCraft II benchmark tasks (Samvelyan et al., 2019).

Original: The completeness of the IGM function class can express richer joint action-value function classes induced by large and diverse datasets or training buffers.
Revised: The completeness of the IGM function class can express richer joint action-value function classes induced by large and diverse datasets or training buffers.

Original: This expressiveness can provide QPLEX with higher sample efﬁciency to achieve state-of-the-art performance in online and ofﬂine data collections.
Revised: This expressiveness can provide QPLEX with higher sample efﬁciency to achieve state-of-the-art performance in online and ofﬂine data collections.

Original: We compare
Revised: We compare

Original: QPLEX with state-of-the-art baselines: QTRAN (Son et al., 2019), QMIX (Rashid et al., 2018), VDN (Sunehag et al., 2018), and Qatten (Yang et al., 2020).
Revised: QPLEX with state-of-the-art baselines: QTRAN (Son et al., 2019), QMIX (Rashid et al., 2018), VDN (Sunehag et al., 2018), and Qatten (Yang et al., 2020).

Original: In particular, the second term of Eq.
Revised: In particular, the second term of Eq.

Original: (12) is the main difference between QPLEX and Qatten.
Revised: (12) is the main difference between QPLEX and Qatten.

Original: Thus, Qatten provides a natural ablation baseline of QPLEX to demonstrate the effectiveness of this discrepancy term.
Revised: Thus, Qatten provides a natural ablation baseline of QPLEX to demonstrate the effectiveness of this discrepancy term.

Original: The implementation details of these algorithms and experimental settings are deferred to Appendix B. Towards fair evaluation, all experimental results are illustrated with the median performance and 25-75% percentiles overrandom seeds.
Revised: The implementation details of these algorithms and experimental settings are deferred to Appendix B. Towards fair evaluation, all experimental results are illustrated with the median performance and 25-75% percentiles overrandom seeds.

Original: The videos of our experiments on StarCraft II are available on an anonymous website 1 .
Revised: The videos of our experiments on StarCraft II are available on an anonymous website 1 .

Original: 4. M ATRIX G AMES
Revised: 4. M ATRIX G AMES

Original: QTRAN (Son et al., 2019) proposes a hard matrix game, as shown in Table 4a of Appendix C.
Revised: QTRAN (Son et al., 2019) proposes a hard matrix game, as shown in Table 4a of Appendix C.

Original: In this subsection, we consider a harder matrix game in Table 2a, which also describes a simple cooperative multi-agent task with considerable miscoordination penalties, and its local optimum is more difﬁcult to jump out.
Revised: In this subsection, we consider a harder matrix game in Table 2a, which also describes a simple cooperative multi-agent task with considerable miscoordination penalties, and its local optimum is more difﬁcult to jump out.

Original: The optimal joint strategy of these two games is to perform action A (1) simultaneously.
Revised: The optimal joint strategy of these two games is to perform action A (1) simultaneously.

Original: To ensure sufﬁcient data collection in the joint action space, we adopt uniform data distribution.
Revised: To ensure sufﬁcient data collection in the joint action space, we adopt uniform data distribution.

Original: With this ﬁxed dataset, we can study the optimality of multi-agent Q-learning from an optimization perspective, ignoring the challenge of exploration and sample complexity.
Revised: With this ﬁxed dataset, we can study the optimality of multi-agent Q-learning from an optimization perspective, ignoring the challenge of exploration and sample complexity.

Original: As shown in Figure 2b, QPLEX and QTRAN, which possess a richer expressiveness power of value factorization can achieve optimal performance, while other algorithms with limited expressiveness (e.g., QMIX, VDN, and Qatten) fall into a local optimum induced by miscoordination penalties.
Revised: As shown in Figure 2b, QPLEX and QTRAN, which possess a richer expressiveness power of value factorization can achieve optimal performance, while other algorithms with limited expressiveness (e.g., QMIX, VDN, and Qatten) fall into a local optimum induced by miscoordination penalties.

Original: In the original matrix proposed by QTRAN, QPLEX and QTRAN are still the only two algorithms that can successfully converge to optimal joint action-value functions.
Revised: In the original matrix proposed by QTRAN, QPLEX and QTRAN are still the only two algorithms that can successfully converge to optimal joint action-value functions.

Original: These results are deferred to Appendix C. QTRAN achieves superior performance in the matrix games but suffers from its relaxation of IGM consistency in complex domains (such as StarCraft II) shown in Section 4.3.
Revised: These results are deferred to Appendix C. QTRAN achieves superior performance in the matrix games but suffers from its relaxation of IGM consistency in complex domains (such as StarCraft II) shown in Section 4.3.

Original: In the theoretical analysis of QPLEX, Proposition 2 exploits the universal function approximation of neural networks.
Revised: In the theoretical analysis of QPLEX, Proposition 2 exploits the universal function approximation of neural networks.

Original: QPLEX allows the scalable implementations with various neural network capacities (different layers and heads of attention module) for learning importance weights λ
Revised: QPLEX allows the scalable implementations with various neural network capacities (different layers and heads of attention module) for learning importance weights λ

Original: i (see Eq.
Revised: i (see Eq.

Original: (10) and (11)).
Revised: (10) and (11)).

Original: As shown in Figure 2c, by increasing the neural network size for learning λ
Revised: As shown in Figure 2c, by increasing the neural network size for learning λ

Original: i (e.g.,
Revised: i (e.g.,

Original: QPLEX-3L10H), QPLEX possesses more expressiveness of value factorization and converges faster.
Revised: QPLEX-3L10H), QPLEX possesses more expressiveness of value factorization and converges faster.

Original: However, learning efﬁciency becomes challenging for complex neural networks.
Revised: However, learning efﬁciency becomes challenging for complex neural networks.

Original: To effectively perform StarCraft II tasks ranging from 2 to 27 agents, we use a small multi-head attention module (i.e., QPLEX-1L4H) in complex domains (see Section 4.3).
Revised: To effectively perform StarCraft II tasks ranging from 2 to 27 agents, we use a small multi-head attention module (i.e., QPLEX-1L4H) in complex domains (see Section 4.3).

Original: Please refer to Appendix B for more detailed conﬁgurations.
Revised: Please refer to Appendix B for more detailed conﬁgurations.

Original: [Figure]
Revised: [Figure]

Original: In this subsection, we focus on a Multi-agent Markov
Revised: In this subsection, we focus on a Multi-agent Markov

Original: Decision Process (MMDP) (Boutilier, 1996) which is a fully cooperative multi-agent setting with full observability.
Revised: Decision Process (MMDP) (Boutilier, 1996) which is a fully cooperative multi-agent setting with full observability.

Original: Consider a two-state MMDP proposed by Wang et al.
Revised: Consider a two-state MMDP proposed by Wang et al.

Original: (2020) with two agents, two actions, and a single reward (see Figure 7b of Appendix C).
Revised: (2020) with two agents, two actions, and a single reward (see Figure 7b of Appendix C).

Original: Two agents start at state s 2 and explore extrinsic rewards forenvironment steps.
Revised: Two agents start at state s 2 and explore extrinsic rewards forenvironment steps.

Original: The optimal policy of this MMDP is simply executing the action A (1) at state s 2 , which is the only coordination pattern to obtain the positive reward.
Revised: The optimal policy of this MMDP is simply executing the action A (1) at state s 2 , which is the only coordination pattern to obtain the positive reward.

Original: To approximate the uniform data distribution, we adopt a uniform exploration strategy ( i.e. , (cid:15) -greedy exploration with (cid:15) = 1 ).
Revised: To approximate the uniform data distribution, we adopt a uniform exploration strategy ( i.e. , (cid:15) -greedy exploration with (cid:15) = 1 ).

Original: We consider the training stability of multi-agent Q-learning algorithms with uniform data distribution in this special MMDP task.
Revised: We consider the training stability of multi-agent Q-learning algorithms with uniform data distribution in this special MMDP task.

Original: As shown in Figure 3, the joint state-value function Q tot learned by baseline algorithms using limited function classes, including QMIX, VDN, and Qatten, will diverge.
Revised: As shown in Figure 3, the joint state-value function Q tot learned by baseline algorithms using limited function classes, including QMIX, VDN, and Qatten, will diverge.

Original: This instability phenomenon of VDN has been theoretically investigated by Wang et al.
Revised: This instability phenomenon of VDN has been theoretically investigated by Wang et al.

Original: By utilizing richer function classes, both QPLEX and QTRAN can address this numerical instability issue and converge to the optimal joint state-value function.
Revised: By utilizing richer function classes, both QPLEX and QTRAN can address this numerical instability issue and converge to the optimal joint state-value function.

Original: 4. D ECENTRALIZED S TAR C RAFT II M ICROMANAGEMENT B ENCHMARK
Revised: 4. D ECENTRALIZED S TAR C RAFT II M ICROMANAGEMENT B ENCHMARK

Original: A more challenging set of empirical experiments are based on StarCraft Multi-Agent Challenge (SMAC) benchmark (Samvelyan et al., 2019).
Revised: A more challenging set of empirical experiments are based on StarCraft Multi-Agent Challenge (SMAC) benchmark (Samvelyan et al., 2019).

Original: We ﬁrst investigate empirical performance in a popular experimental setting with (cid:15) -greedy exploration and a limited ﬁrst-in-ﬁrst-out (FIFO) buffer (Samvelyan et al., 2019), named online data collection setting.
Revised: We ﬁrst investigate empirical performance in a popular experimental setting with (cid:15) -greedy exploration and a limited ﬁrst-in-ﬁrst-out (FIFO) buffer (Samvelyan et al., 2019), named online data collection setting.

Original: To demonstrate the ofﬂine training potential of QPLEX, we also adopt the ofﬂine data collection setting proposed by Levine et al.
Revised: To demonstrate the ofﬂine training potential of QPLEX, we also adopt the ofﬂine data collection setting proposed by Levine et al.

Original: (2020), which can be granted access to a given dataset without additional online exploration.
Revised: (2020), which can be granted access to a given dataset without additional online exploration.

Original: [Figure]
Revised: [Figure]

Original: 4.3. T RAINING WITH O NLINE D ATA C OLLECTION
Revised: 4.3. T RAINING WITH O NLINE D ATA C OLLECTION

Original: We evaluate QPLEX in 17 benchmark tasks of StarCraft II, which contains 14 popular tasks proposed by SMAC (Samvelyan et al., 2019) and three new super hard cooperative tasks.
Revised: We evaluate QPLEX in 17 benchmark tasks of StarCraft II, which contains 14 popular tasks proposed by SMAC (Samvelyan et al., 2019) and three new super hard cooperative tasks.

Original: To demonstrate the overall performance of each algorithm, Figure 4 plots the averaged median test win rate across allscenarios and the number of scenarios in which the algorithm outperforms, respectively.
Revised: To demonstrate the overall performance of each algorithm, Figure 4 plots the averaged median test win rate across allscenarios and the number of scenarios in which the algorithm outperforms, respectively.

Original: Figure 4a shows that, compared with other baselines, QPLEX constantly and signiﬁcantly outperforms baselines over the whole training process and exceeds at least 10% median test win rate averaged across allscenarios.
Revised: Figure 4a shows that, compared with other baselines, QPLEX constantly and signiﬁcantly outperforms baselines over the whole training process and exceeds at least 10% median test win rate averaged across allscenarios.

Original: Moreover, Figure 4b illustrates that, among all 17 tasks, QPLEX is the best performer on up to eight tasks, underperforms on just two tasks, and ties for the best performer on the rest tasks.
Revised: Moreover, Figure 4b illustrates that, among all 17 tasks, QPLEX is the best performer on up to eight tasks, underperforms on just two tasks, and ties for the best performer on the rest tasks.

Original: After 0.8M timesteps, the number of tasks that QPLEX achieves the best performance gradually decreases to six, because, in several easy tasks, other baselines also reach almost 100% test win rate as shown in Figure 5.
Revised: After 0.8M timesteps, the number of tasks that QPLEX achieves the best performance gradually decreases to six, because, in several easy tasks, other baselines also reach almost 100% test win rate as shown in Figure 5.

Original: The overall evaluation diagram of the original SMAC benchmark (14 tasks) corresponding to Figure 4 is deferred to Figure 9 in Appendix D.
Revised: The overall evaluation diagram of the original SMAC benchmark (14 tasks) corresponding to Figure 4 is deferred to Figure 9 in Appendix D.

Original: [Figure]
Revised: [Figure]

Original: Figure 5 shows the learning curves on nine tasks in the online data collection setting and the results of other eight maps are deferred to Figure 8 in Appendix D. From Figure 5, we can observe that
Revised: Figure 5 shows the learning curves on nine tasks in the online data collection setting and the results of other eight maps are deferred to Figure 8 in Appendix D. From Figure 5, we can observe that

Original: QPLEX signiﬁcantly outperforms other baselines with higher sample efﬁciency.
Revised: QPLEX signiﬁcantly outperforms other baselines with higher sample efﬁciency.

Original: On the super hard map 5s10z, the performance gap between QPLEX and other baselines exceeds 30% in test win rate, and the visualized strategies of QPLEX and QMIX in this map are deferred to Appendix G. Most multi-agent Q-learning baselines including QMIX, VDN, and Qatten achieve reasonable performance (see Figure 5).
Revised: On the super hard map 5s10z, the performance gap between QPLEX and other baselines exceeds 30% in test win rate, and the visualized strategies of QPLEX and QMIX in this map are deferred to Appendix G. Most multi-agent Q-learning baselines including QMIX, VDN, and Qatten achieve reasonable performance (see Figure 5).

Original: However, as Figure 4 suggests, QTRAN performs the worst in these comparative experiments, even though it performs well in the didactic games.
Revised: However, as Figure 4 suggests, QTRAN performs the worst in these comparative experiments, even though it performs well in the didactic games.

Original: From a theoretical perspective, the online data collection process utilizes an (cid:15) -greedy exploration process, which requires individual greedy action selections to build an effective training buffer.
Revised: From a theoretical perspective, the online data collection process utilizes an (cid:15) -greedy exploration process, which requires individual greedy action selections to build an effective training buffer.

Original: QTRAN may suffer from its relaxation of IGM consistency (soft constraints of IGM) in the online data collection phase, while the duplex dueling architecture of QPLEX (hard constraint of IGM) provides effective individual greedy action selections, making it suitable for data collection with (cid:15) -greedy exploration.
Revised: QTRAN may suffer from its relaxation of IGM consistency (soft constraints of IGM) in the online data collection phase, while the duplex dueling architecture of QPLEX (hard constraint of IGM) provides effective individual greedy action selections, making it suitable for data collection with (cid:15) -greedy exploration.

Original: 4.3. A BLATION S TUDIES WITH O NLINE D ATA C OLLECTION
Revised: 4.3. A BLATION S TUDIES WITH O NLINE D ATA C OLLECTION

Original: In this subsection, we conduct two ablation studies to investigate why QPLEX works, which includes: (i) QPLEX without the multi-head attention structure of dueling architecture, and (ii) QMIX with the same number of parameters as QPLEX.
Revised: In this subsection, we conduct two ablation studies to investigate why QPLEX works, which includes: (i) QPLEX without the multi-head attention structure of dueling architecture, and (ii) QMIX with the same number of parameters as QPLEX.

Original: For these studies, Figure 6 plots the averaged median test win rate % over the tasks of the StarCraft II benchmark mentioned in Section 4.3.1.
Revised: For these studies, Figure 6 plots the averaged median test win rate % over the tasks of the StarCraft II benchmark mentioned in Section 4.3.1.

Original: Detailed learning curves on each task are deferred to E and F, respectively.
Revised: Detailed learning curves on each task are deferred to E and F, respectively.

Original: Multi-head attention structure of importance weights λ
Revised: Multi-head attention structure of importance weights λ

Original: i (see Eq.
Revised: i (see Eq.

Original: (10) and (11)) allows QPLEX to adapt its scalable implementation to different scenarios, e.g., didactic games or StarCraft II benchmark tasks.
Revised: (10) and (11)) allows QPLEX to adapt its scalable implementation to different scenarios, e.g., didactic games or StarCraft II benchmark tasks.

Original: Section 4.1 demonstrates the importance of this attention structure, which can provide QPLEX more expressiveness of value factorization to perform better on the didactic matrix games.
Revised: Section 4.1 demonstrates the importance of this attention structure, which can provide QPLEX more expressiveness of value factorization to perform better on the didactic matrix games.

Original: In this ablation study, we aim to test whether this multi-head attention is necessary for this StarCraft II domain.
Revised: In this ablation study, we aim to test whether this multi-head attention is necessary for this StarCraft II domain.

Original: Speciﬁcally, we use a one-layer forward model instead of this multi-head attention structure in QPLEX, which is called QPLEX-wo-duel-atten .
Revised: Speciﬁcally, we use a one-layer forward model instead of this multi-head attention structure in QPLEX, which is called QPLEX-wo-duel-atten .

Original: Figure 6a shows that QPLEX-wo-duel-atten achieves similar performance as QPLEX, which indicates that the the superiority of QPLEX over
Revised: Figure 6a shows that QPLEX-wo-duel-atten achieves similar performance as QPLEX, which indicates that the the superiority of QPLEX over

Original: [Figure]
Revised: [Figure]

Original: other MARL methods is largely due to the duplex dueling architecture (see Figure 1), instead of the multi-head attention
Revised: other MARL methods is largely due to the duplex dueling architecture (see Figure 1), instead of the multi-head attention

Original: QPLEX uses more parameters in the value factorization architecture of neural networks due to its multi-head attention structure.
Revised: QPLEX uses more parameters in the value factorization architecture of neural networks due to its multi-head attention structure.

Original: We introduce Large QMIX with a similar number of parameters with
Revised: We introduce Large QMIX with a similar number of parameters with

Original: QPLEX, to investigate whether the superiority of QPLEX over QMIX is due to the increase in the number of parameters.
Revised: QPLEX, to investigate whether the superiority of QPLEX over QMIX is due to the increase in the number of parameters.

Original: Figure 6b shows that QMIX with a larger network cannot fundamentally improve its performance, and QPLEX still signiﬁcantly outperforms Large QMIX by a large margin.
Revised: Figure 6b shows that QMIX with a larger network cannot fundamentally improve its performance, and QPLEX still signiﬁcantly outperforms Large QMIX by a large margin.

Original: This result also conﬁrms that the main effect of QPLEX comes from its novel value factorization structure (duplex dueling architecture) rather than the number of parameters.
Revised: This result also conﬁrms that the main effect of QPLEX comes from its novel value factorization structure (duplex dueling architecture) rather than the number of parameters.

Original: 4.3. T RAINING WITH O FFLINE D ATA C OLLECTION
Revised: 4.3. T RAINING WITH O FFLINE D ATA C OLLECTION

Original: Recently, ofﬂine reinforcement learning has been regarded as a key step for real-world RL applications (Dulac-Arnold et al., 2019; Levine et al., 2020).
Revised: Recently, ofﬂine reinforcement learning has been regarded as a key step for real-world RL applications (Dulac-Arnold et al., 2019; Levine et al., 2020).

Original: Agarwal et al.
Revised: Agarwal et al.

Original: (2020) presents an optimistic perspective of ofﬂine Q-learning that DQN and its variants can achieve superior performance in
Revised: (2020) presents an optimistic perspective of ofﬂine Q-learning that DQN and its variants can achieve superior performance in

Original: Atari 2600 games (Bellemare et al., 2013) with sufﬁciently large and diverse datasets.
Revised: Atari 2600 games (Bellemare et al., 2013) with sufﬁciently large and diverse datasets.

Original: In MARL,
Revised: In MARL,

Original: StarCraft II benchmark has the same discrete action space as Atari.
Revised: StarCraft II benchmark has the same discrete action space as Atari.

Original: We conduct a lot of experiments on the StarCraft II benchmark tasks to study ofﬂine multi-agent Q-learning in this subsection.
Revised: We conduct a lot of experiments on the StarCraft II benchmark tasks to study ofﬂine multi-agent Q-learning in this subsection.

Original: We adopt a large and diverse dataset to make the expressiveness power of value factorization become the dominant factor to investigate.
Revised: We adopt a large and diverse dataset to make the expressiveness power of value factorization become the dominant factor to investigate.

Original: We train a behavior policy of QMIX and collect all its experienced transitions throughout the training process (see the details in Appendix C).
Revised: We train a behavior policy of QMIX and collect all its experienced transitions throughout the training process (see the details in Appendix C).

Original: As shown in Figure 13 in Appendix H, QPLEX signiﬁcantly outperforms other multi-agent Q-learning baselines and possesses the state-of-the-art value factorization structure for ofﬂine multi-agent Q-learning.
Revised: As shown in Figure 13 in Appendix H, QPLEX signiﬁcantly outperforms other multi-agent Q-learning baselines and possesses the state-of-the-art value factorization structure for ofﬂine multi-agent Q-learning.

Original: QMIX and Qatten cannot always maintain stable learning performance, and VDN suffers from ofﬂine data collection and leads to weak empirical results.
Revised: QMIX and Qatten cannot always maintain stable learning performance, and VDN suffers from ofﬂine data collection and leads to weak empirical results.

Original: QTRAN may perform well in certain cases when its soft constraints, two (cid:96) 2 -penalty terms, are well minimized.
Revised: QTRAN may perform well in certain cases when its soft constraints, two (cid:96) 2 -penalty terms, are well minimized.

Original: With ofﬂine data collection, individual greedy action selections do not need to build a training buffer, but they still need to compute the one-step TD target for centralized training.
Revised: With ofﬂine data collection, individual greedy action selections do not need to build a training buffer, but they still need to compute the one-step TD target for centralized training.

Original: Therefore, compared with QTRAN, QPLEX still has theoretical advantages regarding the IGM principle in the ofﬂine data collection setting.
Revised: Therefore, compared with QTRAN, QPLEX still has theoretical advantages regarding the IGM principle in the ofﬂine data collection setting.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we introduced QPLEX, a novel multi-agent Q-learning framework that allows centralized end-to-end training and learns to factorize a joint action-value function to enable decentralized execution.
Revised: In this paper, we introduced QPLEX, a novel multi-agent Q-learning framework that allows centralized end-to-end training and learns to factorize a joint action-value function to enable decentralized execution.

Original: QPLEX takes advantage of a duplex dueling architecture that efﬁciently encodes the
Revised: QPLEX takes advantage of a duplex dueling architecture that efﬁciently encodes the

Original: IGM consistency constraint on joint and individual greedy action selections.
Revised: IGM consistency constraint on joint and individual greedy action selections.

Original: Our theoretical analysis shows that QPLEX achieves a complete IGM function class.
Revised: Our theoretical analysis shows that QPLEX achieves a complete IGM function class.

Original: Empirical results demonstrate that it signiﬁcantly outperforms state-of-the-art baselines in both online and ofﬂine data collection settings.
Revised: Empirical results demonstrate that it signiﬁcantly outperforms state-of-the-art baselines in both online and ofﬂine data collection settings.

Original: In particular, QPLEX possesses strong ability of supporting ofﬂine training.
Revised: In particular, QPLEX possesses strong ability of supporting ofﬂine training.

Original: This ability provides
Revised: This ability provides

Original: QPLEX with high sample efﬁciency and opportunities of utilizing ofﬂine multi-source datasets.
Revised: QPLEX with high sample efﬁciency and opportunities of utilizing ofﬂine multi-source datasets.

Original: It will be an interesting and valuable direction to study ofﬂine multi-agent reinforcement learning in continuous action spaces
Revised: It will be an interesting and valuable direction to study ofﬂine multi-agent reinforcement learning in continuous action spaces

Original: A O MITTED P ROOFS IN S ECTION
Revised: A O MITTED P ROOFS IN S ECTION

Original: Deﬁnition 1 (Advantage-based IGM) .
Revised: Deﬁnition 1 (Advantage-based IGM) .

Original: For a joint action-value function Q tot : T × A (cid:55)→ R and individual action-value functions [ Q i : T × A (cid:55)→ R ] ni =1 , where ∀ τ ∈ T , ∀ a ∈ A , ∀ i ∈ N , (Joint Dueling) Q tot ( τ , a ) =
Revised: For a joint action-value function Q tot : T × A (cid:55)→ R and individual action-value functions [ Q i : T × A (cid:55)→ R ] ni =1 , where ∀ τ ∈ T , ∀ a ∈ A , ∀ i ∈ N , (Joint Dueling) Q tot ( τ , a ) =

Original: V tot ( τ ) + A tot ( τ , a ) and V tot ( τ ) = max a (cid:48)
Revised: V tot ( τ ) + A tot ( τ , a ) and V tot ( τ ) = max a (cid:48)

Original: Q tot ( τ , a (cid:48) ) , (4) (Individual Dueling)
Revised: Q tot ( τ , a (cid:48) ) , (4) (Individual Dueling)

Original: Q
Revised: Q

Original: i ( τ i , a i ) =
Revised: i ( τ i , a i ) =

Original: V i (
Revised: V i (

Original: [Equation]
Revised: [Equation]

Original: i )
Revised: i )

Original: +
Revised: +

Original: A i (
Revised: A i (

Original: [Equation]
Revised: [Equation]

Original: i , a i ) and V i ( τ i ) =
Revised: i , a i ) and V i ( τ i ) =

Original: [Equation]
Revised: [Equation]

Original: a (cid:48)
Revised: a (cid:48)

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: i (
Revised: i (

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: such that the following holds
Revised: such that the following holds

Original: [Equation]
Revised: [Equation]

Original: then, we can say that [ Q i ] ni =1 satisﬁes advantage-based IGM for Q tot .
Revised: then, we can say that [ Q i ] ni =1 satisﬁes advantage-based IGM for Q tot .

Original: Let the action-value function class derived from IGM is denoted by
Revised: Let the action-value function class derived from IGM is denoted by

Original: [Equation]
Revised: [Equation]

Original: (13) where (cid:101) Q tot and (cid:104) (cid:101) Q i (cid:105) n i =1 denote the joint and individual action-value functions induced by IGM, respectively.
Revised: (13) where (cid:101) Q tot and (cid:104) (cid:101) Q i (cid:105) n i =1 denote the joint and individual action-value functions induced by IGM, respectively.

Original: Similarly, let
Revised: Similarly, let

Original: [Equation]
Revised: [Equation]

Original: are satisﬁed
Revised: are satisﬁed

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: denote the action-value function class derived from advantage-based IGM.
Revised: denote the action-value function class derived from advantage-based IGM.

Original: (cid:101) V tot and (cid:101)
Revised: (cid:101) V tot and (cid:101)

Original: A tot denote the joint state-value and advantage functions, respectively.
Revised: A tot denote the joint state-value and advantage functions, respectively.

Original: (cid:104)(cid:101) V i (cid:105) n i =1 and (cid:104) (cid:101)
Revised: (cid:104)(cid:101) V i (cid:105) n i =1 and (cid:104) (cid:101)

Original: A i (cid:105) n i =1 denote the individual state-value and advantage functions induced by advantage-IGM, respectively.
Revised: A i (cid:105) n i =1 denote the individual state-value and advantage functions induced by advantage-IGM, respectively.

Original: According to the duplex dueling architecture Q = V + A stated in advantage-based IGM (see Deﬁnition 1), we derive the joint and individual action-value functions as following: ∀ τ ∈ T , ∀ a ∈ A , ∀ i ∈ N ,
Revised: According to the duplex dueling architecture Q = V + A stated in advantage-based IGM (see Deﬁnition 1), we derive the joint and individual action-value functions as following: ∀ τ ∈ T , ∀ a ∈ A , ∀ i ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: Q tot
Revised: Q tot

Original: [Equation]
Revised: [Equation]

Original: τ , a ) = (cid:98) V tot ( τ
Revised: τ , a ) = (cid:98) V tot ( τ

Original: [Equation]
Revised: [Equation]

Original: + (cid:98)
Revised: + (cid:98)

Original: A tot ( τ , a ) and (cid:98)
Revised: A tot ( τ , a ) and (cid:98)

Original: [Equation]
Revised: [Equation]

Original: , a i
Revised: , a i

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: + (cid:98)
Revised: + (cid:98)

Original: A i ( τ i , a i ) .
Revised: A i ( τ i , a i ) .

Original: Proposition 1.
Revised: Proposition 1.

Original: The advantage-based IGM and IGM function classes are equivalent.
Revised: The advantage-based IGM and IGM function classes are equivalent.

Original: Proof.
Revised: Proof.

Original: We will prove (cid:101) Q ≡ (cid:98) Q in the following two directions.
Revised: We will prove (cid:101) Q ≡ (cid:98) Q in the following two directions.

Original: (cid:101) Q ⊆ (cid:98) Q For any (cid:16) (cid:101) Q tot , (cid:104) (cid:101)
Revised: (cid:101) Q ⊆ (cid:98) Q For any (cid:16) (cid:101) Q tot , (cid:104) (cid:101)

Original: Q i (cid:105) n i =1 (cid:17) ∈ (cid:101) Q , we construct (cid:98) Q tot = (cid:101) Q tot and (cid:104) (cid:98) Q i (cid:105) n i =1 = (cid:104) (cid:101) Q i (cid:105) n i =1 .
Revised: Q i (cid:105) n i =1 (cid:17) ∈ (cid:101) Q , we construct (cid:98) Q tot = (cid:101) Q tot and (cid:104) (cid:98) Q i (cid:105) n i =1 = (cid:104) (cid:101) Q i (cid:105) n i =1 .

Original: The joint and individual state-value/advantage functions induced by advantage-IGM (cid:98) V tot ( τ ) = max a (cid:48) (cid:98) Q tot ( τ , a (cid:48) ) and (cid:98)
Revised: The joint and individual state-value/advantage functions induced by advantage-IGM (cid:98) V tot ( τ ) = max a (cid:48) (cid:98) Q tot ( τ , a (cid:48) ) and (cid:98)

Original: A tot ( τ , a ) = (cid:98) Q tot ( τ , a ) − (cid:98) V tot ( τ ) , (16) (cid:98) V i ( τ i ) = max a (cid:48)
Revised: A tot ( τ , a ) = (cid:98) Q tot ( τ , a ) − (cid:98) V tot ( τ ) , (16) (cid:98) V i ( τ i ) = max a (cid:48)

Original: i (cid:98) Q i ( τ i , a (cid:48) i ) and (cid:98) A i ( τ i , a i ) = (cid:98) Q i ( τ i , a (cid:48) i ) − (cid:98) V i ( τ i ) , ∀ i ∈ N , (17) are derived by Eq.
Revised: i (cid:98) Q i ( τ i , a (cid:48) i ) and (cid:98) A i ( τ i , a i ) = (cid:98) Q i ( τ i , a (cid:48) i ) − (cid:98) V i ( τ i ) , ∀ i ∈ N , (17) are derived by Eq.

Original: (4) and Eq.
Revised: (4) and Eq.

Original: (5), respectively.
Revised: (5), respectively.

Original: Because state-value functions do not affect the greedy action selection, ∀ τ ∈ T , ∀ a ∈ A , arg max
Revised: Because state-value functions do not affect the greedy action selection, ∀ τ ∈ T , ∀ a ∈ A , arg max

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ⇒ arg max a ∈ A (cid:98) Q
Revised: ⇒ arg max a ∈ A (cid:98) Q

Original: [Equation]
Revised: [Equation]

Original: = (cid:18)
Revised: = (cid:18)

Original: [Equation]
Revised: [Equation]

Original: (cid:98)
Revised: (cid:98)

Original: [Equation]
Revised: [Equation]

Original: arg
Revised: arg

Original: [Equation]
Revised: [Equation]

Original: 19) ⇒ arg max a ∈ A (cid:16)
Revised: 19) ⇒ arg max a ∈ A (cid:16)

Original: [Equation]
Revised: [Equation]

Original: = (20) (cid:18) arg max a 1 ∈A (cid:16) (cid:98) Q 1 ( τ 1 , a 1 ) − (cid:98) V 1 ( τ 1 ) (cid:17) , . .
Revised: = (20) (cid:18) arg max a 1 ∈A (cid:16) (cid:98) Q 1 ( τ 1 , a 1 ) − (cid:98) V 1 ( τ 1 ) (cid:17) , . .

Original: . , arg max a n ∈A (cid:16) (cid:98) Q n ( τ n , a n ) − (cid:98) V n ( τ n ) (cid:17)(cid:19) (21) ⇒ arg max a ∈ A (cid:98)
Revised: . , arg max a n ∈A (cid:16) (cid:98) Q n ( τ n , a n ) − (cid:98) V n ( τ n ) (cid:17)(cid:19) (21) ⇒ arg max a ∈ A (cid:98)

Original: A tot ( τ , a ) = (cid:18) arg max a 1 ∈A (cid:98)
Revised: A tot ( τ , a ) = (cid:18) arg max a 1 ∈A (cid:98)

Original: A 1 ( τ 1 , a 1 ) , . . .
Revised: A 1 ( τ 1 , a 1 ) , . . .

Original: , arg max a n ∈A (cid:98)
Revised: , arg max a n ∈A (cid:98)

Original: A n ( τ n , a n ) (cid:19) .
Revised: A n ( τ n , a n ) (cid:19) .

Original: Thus, (cid:16) (cid:98) Q tot , (cid:104) (cid:98) Q i (cid:105) n i =1 (cid:17) ∈ (cid:98) Q , which means that (cid:101) Q ⊆ (cid:98) Q .
Revised: Thus, (cid:16) (cid:98) Q tot , (cid:104) (cid:98) Q i (cid:105) n i =1 (cid:17) ∈ (cid:98) Q , which means that (cid:101) Q ⊆ (cid:98) Q .

Original: (cid:98) Q ⊆ (cid:101) Q
Revised: (cid:98) Q ⊆ (cid:101) Q

Original: We will prove this direction in the same way.
Revised: We will prove this direction in the same way.

Original: For any (cid:16) (cid:98) Q tot , (cid:104) (cid:98) Q i (cid:105) n i =1 (cid:17) ∈ (cid:98) Q , we construct (cid:101) Q tot = (cid:98) Q tot and (cid:104) (cid:101) Q i (cid:105) n i =1 = (cid:104) (cid:98) Q i (cid:105) n i =1 .
Revised: For any (cid:16) (cid:98) Q tot , (cid:104) (cid:98) Q i (cid:105) n i =1 (cid:17) ∈ (cid:98) Q , we construct (cid:101) Q tot = (cid:98) Q tot and (cid:104) (cid:101) Q i (cid:105) n i =1 = (cid:104) (cid:98) Q i (cid:105) n i =1 .

Original: Because state-value functions do not affect the greedy action selection,
Revised: Because state-value functions do not affect the greedy action selection,

Original: [Equation]
Revised: [Equation]

Original: ⇒ arg
Revised: ⇒ arg

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: Thus, (cid:16) (cid:101) Q tot , (cid:104) (cid:101) Q i (cid:105) n i =1 (cid:17) ∈ (cid:101) Q , which means that (cid:98) Q ⊆ (cid:101) Q .
Revised: Thus, (cid:16) (cid:101) Q tot , (cid:104) (cid:101) Q i (cid:105) n i =1 (cid:17) ∈ (cid:101) Q , which means that (cid:98) Q ⊆ (cid:101) Q .

Original: The action-value function classes derived from advantage-based IGM and IGM are equivalent.
Revised: The action-value function classes derived from advantage-based IGM and IGM are equivalent.

Original: Fact 1.
Revised: Fact 1.

Original: The constraint of advantage-based IGM stated in Eq.
Revised: The constraint of advantage-based IGM stated in Eq.

Original: (6) is equivalent to that when ∀ τ ∈ T , ∀ a ∗ ∈
Revised: (6) is equivalent to that when ∀ τ ∈ T , ∀ a ∗ ∈

Original: A ∗ ( τ ) , ∀ a ∈
Revised: A ∗ ( τ ) , ∀ a ∈

Original: A \ A ∗ ( τ ) , ∀ i ∈ N ,
Revised: A \ A ∗ ( τ ) , ∀ i ∈ N ,

Original: A tot ( τ , a ∗ ) =
Revised: A tot ( τ , a ∗ ) =

Original: A i ( τ i , a ∗ i ) = 0 and A tot ( τ , a ) < 0 , A i ( τ i , a i ) ≤ 0 , (7) where A ∗ ( τ ) =
Revised: A i ( τ i , a ∗ i ) = 0 and A tot ( τ , a ) < 0 , A i ( τ i , a i ) ≤ 0 , (7) where A ∗ ( τ ) =

Original: { a | a ∈ A , Q tot ( τ , a ) =
Revised: { a | a ∈ A , Q tot ( τ , a ) =

Original: V tot ( τ ) } .
Revised: V tot ( τ ) } .

Original: Proof.
Revised: Proof.

Original: We derive that ∀ τ ∈ T , ∀ a ∈ A , ∀ i ∈ N , (cid:98)
Revised: We derive that ∀ τ ∈ T , ∀ a ∈ A , ∀ i ∈ N , (cid:98)

Original: A tot ( τ , a ) ≤ 0 and (cid:98)
Revised: A tot ( τ , a ) ≤ 0 and (cid:98)

Original: A i ( τ i , a i ) ≤ 0 from Eq.
Revised: A i ( τ i , a i ) ≤ 0 from Eq.

Original: (4) and Eq.
Revised: (4) and Eq.

Original: (5) of Deﬁnition 1, respectively.
Revised: (5) of Deﬁnition 1, respectively.

Original: According to the deﬁnition of arg max operator, Eq.
Revised: According to the deﬁnition of arg max operator, Eq.

Original: (4), and Eq.
Revised: (4), and Eq.

Original: (5), ∀ τ ∈ T , let (cid:98) A ∗ ( τ ) denote arg max a ∈ A (cid:98)
Revised: (5), ∀ τ ∈ T , let (cid:98) A ∗ ( τ ) denote arg max a ∈ A (cid:98)

Original: A tot ( τ , a ) as follows:
Revised: A tot ( τ , a ) as follows:

Original: [Equation]
Revised: [Equation]

Original: Similarly, ∀ τ ∈ T , ∀ i ∈ N , let (cid:98)
Revised: Similarly, ∀ τ ∈ T , ∀ i ∈ N , let (cid:98)

Original: A ∗ i ( τ i ) denote arg max a i ∈A (cid:98)
Revised: A ∗ i ( τ i ) denote arg max a i ∈A (cid:98)

Original: A i ( τ i , a i ) as follows:
Revised: A i ( τ i , a i ) as follows:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Thus, ∀ τ ∈ T , ∀ a ∗ ∈ (cid:98)
Revised: Thus, ∀ τ ∈ T , ∀ a ∗ ∈ (cid:98)

Original: A ∗ ( τ ) , ∀ a ∈
Revised: A ∗ ( τ ) , ∀ a ∈

Original: A \ (cid:98) A ∗ ( τ ) , (cid:98)
Revised: A \ (cid:98) A ∗ ( τ ) , (cid:98)

Original: A tot ( τ , a ∗ ) = 0 and (cid:98)
Revised: A tot ( τ , a ∗ ) = 0 and (cid:98)

Original: A tot ( τ , a ) < 0; (37) ∀ τ ∈ T , ∀ i ∈ N , ∀ a ∗ i ∈ (cid:98)
Revised: A tot ( τ , a ) < 0; (37) ∀ τ ∈ T , ∀ i ∈ N , ∀ a ∗ i ∈ (cid:98)

Original: A ∗ ( τ i ) , ∀ a i ∈
Revised: A ∗ ( τ i ) , ∀ a i ∈

Original: A \ (cid:98)
Revised: A \ (cid:98)

Original: A ∗ ( τ i ) , (cid:98)
Revised: A ∗ ( τ i ) , (cid:98)

Original: A
Revised: A

Original: i ( τ i , a ∗ i ) = 0 and (cid:98) A
Revised: i ( τ i , a ∗ i ) = 0 and (cid:98) A

Original: i ( τ i , a i
Revised: i ( τ i , a i

Original: [Equation]
Revised: [Equation]

Original: Recall the constraint stated in Eq.
Revised: Recall the constraint stated in Eq.

Original: 6, ∀ τ ∈ T ,
Revised: 6, ∀ τ ∈ T ,

Original: [Equation]
Revised: [Equation]

Original: We can rewrite the constraint of advantage-based IGM stated in Eq.
Revised: We can rewrite the constraint of advantage-based IGM stated in Eq.

Original: (6) as ∀ τ ∈ T ,
Revised: (6) as ∀ τ ∈ T ,

Original: [Equation]
Revised: [Equation]

Original: Therefore, combining Eq.
Revised: Therefore, combining Eq.

Original: (37), Eq.
Revised: (37), Eq.

Original: (38), and Eq.
Revised: (38), and Eq.

Original: (40), we can derive ∀ τ ∈ T , ∀ a ∗ ∈ (cid:98)
Revised: (40), we can derive ∀ τ ∈ T , ∀ a ∗ ∈ (cid:98)

Original: A ∗ ( τ ) ,
Revised: A ∗ ( τ ) ,

Original: [Equation]
Revised: [Equation]

Original: (cid:98)
Revised: (cid:98)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original: In another way, combining Eq.
Revised: In another way, combining Eq.

Original: (37), Eq.
Revised: (37), Eq.

Original: (38), and Eq.
Revised: (38), and Eq.

Original: (41), we can derive Eq.
Revised: (41), we can derive Eq.

Original: (40) by the deﬁnition of (cid:98) A ∗ and (cid:104) (cid:98)
Revised: (40) by the deﬁnition of (cid:98) A ∗ and (cid:104) (cid:98)

Original: A ∗ (cid:105) n i =

Revised: A ∗ (cid:105) n i =

Original: (see Eq.
Revised: (see Eq.

Original: (33) and Eq.
Revised: (33) and Eq.

Original: In more detail, the closed set property of Cartesian product of [ a ∗ i ] ni =1 has been encoded into the Eq.
Revised: In more detail, the closed set property of Cartesian product of [ a ∗ i ] ni =1 has been encoded into the Eq.

Original: (40) and Eq.
Revised: (40) and Eq.

Original: (41) simultaneously.
Revised: (41) simultaneously.

Original: Proposition 2.
Revised: Proposition 2.

Original: Given the universal function approximation of neural networks, the action-value function class that QPLEX can realize is equivalent to what is induced by the IGM principle.
Revised: Given the universal function approximation of neural networks, the action-value function class that QPLEX can realize is equivalent to what is induced by the IGM principle.

Original: Proof.
Revised: Proof.

Original: We assume that the neural network of QPLEX can be large enough to achieve the universal function approximation by corresponding theorem (Csáji et al., 2001).
Revised: We assume that the neural network of QPLEX can be large enough to achieve the universal function approximation by corresponding theorem (Csáji et al., 2001).

Original: Let the action-value function class that QPLEX can realize is denoted by
Revised: Let the action-value function class that QPLEX can realize is denoted by

Original: [Equation]
Revised: [Equation]

Original: Q i ∈ R | T ||A| (cid:105) n i = (cid:17) (cid:12)(cid:12)(cid:12) Eq.
Revised: Q i ∈ R | T ||A| (cid:105) n i = (cid:17) (cid:12)(cid:12)(cid:12) Eq.

Original: (8) , (9) , (10) , (11) , (12) are satisﬁed (cid:111) .
Revised: (8) , (9) , (10) , (11) , (12) are satisﬁed (cid:111) .

Original: In addition, Q tot , V tot , A tot , (cid:104) Q (cid:48) i (cid:105) n i =1 , (cid:104) V (cid:48) i (cid:105) n i =1 , (cid:104) A (cid:48) i (cid:105) n i =1 , (cid:2) Q i (cid:3) n i =1 , (cid:2) V i (cid:3) n i =1 , and (cid:2)
Revised: In addition, Q tot , V tot , A tot , (cid:104) Q (cid:48) i (cid:105) n i =1 , (cid:104) V (cid:48) i (cid:105) n i =1 , (cid:104) A (cid:48) i (cid:105) n i =1 , (cid:2) Q i (cid:3) n i =1 , (cid:2) V i (cid:3) n i =1 , and (cid:2)

Original: A i (cid:3) n i =1 denote the corresponding (joint, transformed, and individual) (action-value, state-value, and advantage) functions, respectively.
Revised: A i (cid:3) n i =1 denote the corresponding (joint, transformed, and individual) (action-value, state-value, and advantage) functions, respectively.

Original: In the implementation of QPLEX, we ensure the positivity of important weights of Transformation and joint advantage function, [ w i ] ni =1 and [ λ i ] ni =1 , which maintains the greedy action selection ﬂow and rules out these non-interesting points (zeros) on optimization.
Revised: In the implementation of QPLEX, we ensure the positivity of important weights of Transformation and joint advantage function, [ w i ] ni =1 and [ λ i ] ni =1 , which maintains the greedy action selection ﬂow and rules out these non-interesting points (zeros) on optimization.

Original: We will prove (cid:98) Q ≡ Q in the following two directions.
Revised: We will prove (cid:98) Q ≡ Q in the following two directions.

Original: (cid:98) Q ⊆ Q For any (cid:16) (cid:98) Q tot , (cid:104) (cid:98) Q i (cid:105) n i
Revised: (cid:98) Q ⊆ Q For any (cid:16) (cid:98) Q tot , (cid:104) (cid:98) Q i (cid:105) n i

Original: =1 (cid:17) ∈ (cid:98) Q , we construct Q tot = (cid:98) Q tot and (cid:2) Q i (cid:3) n i
Revised: =1 (cid:17) ∈ (cid:98) Q , we construct Q tot = (cid:98) Q tot and (cid:2) Q i (cid:3) n i

Original: =1 = (cid:104) (cid:98) Q
Revised: =1 = (cid:104) (cid:98) Q

Original: i (cid:105) n i = and derive V tot , A tot , (cid:2) V i (cid:3) n i =1 , and (cid:2)
Revised: i (cid:105) n i = and derive V tot , A tot , (cid:2) V i (cid:3) n i =1 , and (cid:2)

Original: A i (cid:3) n i =1 by Eq.(4) and Eq.
Revised: A i (cid:3) n i =1 by Eq.(4) and Eq.

Original: (5), respectively.
Revised: (5), respectively.

Original: Note that in the construction of QPLEX,
Revised: Note that in the construction of QPLEX,

Original: [Equation]
Revised: [Equation]

Original: ) = w i ( τ ) V i ( τ i ) +
Revised: ) = w i ( τ ) V i ( τ i ) +

Original: b i ( τ ) and A i ( τ , a i ) =
Revised: b i ( τ ) and A i ( τ , a i ) =

Original: w i ( τ ) A i ( τ i , a i ) (43) and
Revised: w i ( τ ) A i ( τ i , a i ) (43) and

Original: Q tot ( τ , a ) =
Revised: Q tot ( τ , a ) =

Original: V tot ( τ ) + A tot ( τ , a ) =
Revised: V tot ( τ ) + A tot ( τ , a ) =

Original: n (cid:88) i =
Revised: n (cid:88) i =

Original: V i ( τ ) + n (cid:88) i
Revised: V i ( τ ) + n (cid:88) i

Original: = λ
Revised: = λ

Original: i ( τ , a ) A i ( τ , a i ) .
Revised: i ( τ , a ) A i ( τ , a i ) .

Original: In addition, we construct transformed functions connecting joint and individual functions as follows: ∀ τ
Revised: In addition, we construct transformed functions connecting joint and individual functions as follows: ∀ τ

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: ∀
Revised: ∀

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original: which means that according to Fact 1, w i
Revised: which means that according to Fact 1, w i

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: when A i ( τ i , a i ) < 0 , 1 , when A i ( τ i , a i ) = 0 .
Revised: when A i ( τ i , a i ) < 0 , 1 , when A i ( τ i , a i ) = 0 .

Original: Thus, (cid:16) Q tot , (cid:2) Q i (cid:3) n i =1 (cid:17) ∈ Q , which means that (cid:98) Q ⊆ Q .
Revised: Thus, (cid:16) Q tot , (cid:2) Q i (cid:3) n i =1 (cid:17) ∈ Q , which means that (cid:98) Q ⊆ Q .

Original: Q ⊆ (cid:98) Q
Revised: Q ⊆ (cid:98) Q

Original: For any (cid:16) Q tot , (cid:2) Q i (cid:3) n i =1 (cid:17) ∈ Q , with the similar discussion of Fact 1, ∀ τ ∈ T , ∀ i ∈ N , let
Revised: For any (cid:16) Q tot , (cid:2) Q i (cid:3) n i =1 (cid:17) ∈ Q , with the similar discussion of Fact 1, ∀ τ ∈ T , ∀ i ∈ N , let

Original: A ∗ i ( τ i ) denote arg max a i ∈A
Revised: A ∗ i ( τ i ) denote arg max a i ∈A

Original: A i ( τ i , a i ) , where
Revised: A i ( τ i , a i ) , where

Original: [Equation]
Revised: [Equation]

Original: Combining the positivity of [ w i ] ni =1 and [ λ i ] ni =1 with Eq.
Revised: Combining the positivity of [ w i ] ni =1 and [ λ i ] ni =1 with Eq.

Original: (8), (9), (10), and (12), we can derive
Revised: (8), (9), (10), and (12), we can derive

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and
Revised: and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ⇒
Revised: ⇒

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where a ∗ = (cid:104) a ∗ 1 , . .
Revised: where a ∗ = (cid:104) a ∗ 1 , . .

Original: . , a ∗ n (cid:105) and a = (cid:104) a 1 , . .
Revised: . , a ∗ n (cid:105) and a = (cid:104) a 1 , . .

Original: . , a n (cid:105) .
Revised: . , a n (cid:105) .

Original: Notably, these a ∗ forms
Revised: Notably, these a ∗ forms

Original: [Equation]
Revised: [Equation]

Original: N (cid:111) (51) which is similar to Eq.
Revised: N (cid:111) (51) which is similar to Eq.

Original: (40) in the proof of Fact 1.
Revised: (40) in the proof of Fact 1.

Original: We construct (cid:98) Q tot = Q tot and (cid:104) (cid:98) Q i (cid:105) n i =

Revised: We construct (cid:98) Q tot = Q tot and (cid:104) (cid:98) Q i (cid:105) n i =

Original: = (cid:2) Q i (cid:3) n i =1 .
Revised: = (cid:2) Q i (cid:3) n i =1 .

Original: According to Eq.
Revised: According to Eq.

Original: (51), the constraints of advantage-based IGM stated in Fact 1 (Eq.
Revised: (51), the constraints of advantage-based IGM stated in Fact 1 (Eq.

Original: (4), Eq.
Revised: (4), Eq.

Original: (5), and
Revised: (5), and

Original: Eq.
Revised: Eq.

Original: (7)) are satisﬁed, which means that (cid:16) (cid:98) Q tot , (cid:104) (cid:98) Q i (cid:105) n i =1 (cid:17) ∈ (cid:98) Q and Q ⊆ (cid:98) Q .
Revised: (7)) are satisﬁed, which means that (cid:16) (cid:98) Q tot , (cid:104) (cid:98) Q i (cid:105) n i =1 (cid:17) ∈ (cid:98) Q and Q ⊆ (cid:98) Q .

Original: Thus, when assuming neural networks provide universal function approximation, the joint action-value function class that QPLEX can realize is equivalent to what is induced by the IGM principle.
Revised: Thus, when assuming neural networks provide universal function approximation, the joint action-value function class that QPLEX can realize is equivalent to what is induced by the IGM principle.

Original: B E XPERIMENT S ETTINGS AND I MPLEMENTATION D ETAILS B. S TAR C RAFT II
Revised: B E XPERIMENT S ETTINGS AND I MPLEMENTATION D ETAILS B. S TAR C RAFT II

Original: We consider the combat scenario of StarCraft II unit micromanagement tasks, where the enemy units are controlled by the built-in AI, and each ally unit is controlled by the reinforcement learning agent.
Revised: We consider the combat scenario of StarCraft II unit micromanagement tasks, where the enemy units are controlled by the built-in AI, and each ally unit is controlled by the reinforcement learning agent.

Original: The units of the two groups can be asymmetric, but the units of each group should belong to the same race.
Revised: The units of the two groups can be asymmetric, but the units of each group should belong to the same race.

Original: At each timestep, every agent takes action from the discrete action space, which includes the following actions: no-op, move [direction], attack [enemy id], and stop.
Revised: At each timestep, every agent takes action from the discrete action space, which includes the following actions: no-op, move [direction], attack [enemy id], and stop.

Original: Under the control of these actions, agents move and attack in continuous maps.
Revised: Under the control of these actions, agents move and attack in continuous maps.

Original: At each time step, MARL agents will get a global reward equal to the total damage done to enemy units.
Revised: At each time step, MARL agents will get a global reward equal to the total damage done to enemy units.

Original: Killing each enemy unit and winning the combat will bring additional bonuses of 10 and 200, respectively.
Revised: Killing each enemy unit and winning the combat will bring additional bonuses of 10 and 200, respectively.

Original: We brieﬂy introduce the SMAC challenges of our paper in Table 1.
Revised: We brieﬂy introduce the SMAC challenges of our paper in Table 1.

Original: B. I MPLEMENTATION D ETAILS
Revised: B. I MPLEMENTATION D ETAILS

Original: We adopt the PyMARL (Samvelyan et al., 2019) implementation of state-of-the-art baselines: QTRAN (Son et al., 2019), QMIX (Rashid et al., 2018), VDN (Sunehag et al., 2018), and
Revised: We adopt the PyMARL (Samvelyan et al., 2019) implementation of state-of-the-art baselines: QTRAN (Son et al., 2019), QMIX (Rashid et al., 2018), VDN (Sunehag et al., 2018), and

Original: Qatten (Yang et al., 2020).
Revised: Qatten (Yang et al., 2020).

Original: The hyper-parameters of these algorithms are the same as that in SMAC (Samvelyan et al., 2019) and referred in their source codes.
Revised: The hyper-parameters of these algorithms are the same as that in SMAC (Samvelyan et al., 2019) and referred in their source codes.

Original: QPLEX is also based on PyMARL, whose special hyperparameters are illustrated in Table 2 and other common hyper-parameters are adopted by the default implementation of PyMARL (Samvelyan et al., 2019).
Revised: QPLEX is also based on PyMARL, whose special hyperparameters are illustrated in Table 2 and other common hyper-parameters are adopted by the default implementation of PyMARL (Samvelyan et al., 2019).

Original: Especially in the online data collection, we take the advanced implementation of Transformation of Qatten in QPLEX.
Revised: Especially in the online data collection, we take the advanced implementation of Transformation of Qatten in QPLEX.

Original: To ensure the positivity of important weights of Transformation and joint advantage function, we add a sufﬁciently small amount (cid:15) (cid:48)
Revised: To ensure the positivity of important weights of Transformation and joint advantage function, we add a sufﬁciently small amount (cid:15) (cid:48)

Original: = 10 − 10 on [ w i ] ni =1 and [ λ i ] ni =1 .
Revised: = 10 − 10 on [ w i ] ni =1 and [ λ i ] ni =1 .

Original: In addition, we stop gradients of local advantage function
Revised: In addition, we stop gradients of local advantage function

Original: A i to increase the optimization stability of the max operator of dueling structure.
Revised: A i to increase the optimization stability of the max operator of dueling structure.

Original: This instability consideration due to max operator has been justiﬁed by Dueling DQN (Wang et al., 2016).
Revised: This instability consideration due to max operator has been justiﬁed by Dueling DQN (Wang et al., 2016).

Original: We approximate the joint action-value function as
Revised: We approximate the joint action-value function as

Original: [Equation]
Revised: [Equation]

Original: where (cid:101)
Revised: where (cid:101)

Original: A i denotes a variant of the local advantage function A i by stoping gradients.
Revised: A i denotes a variant of the local advantage function A i by stoping gradients.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Didactic Examples
Revised: Didactic Examples

Original: StarCraft II
Revised: StarCraft II

Original: The number of layers in w, b, λ, φ, υ 

Revised: The number of layers in w, b, λ, φ, υ

Original: or The number of heads in the attention module 4 or Unit number in middle layers of w, b, λ, φ, υ
Revised: or The number of heads in the attention module 4 or Unit number in middle layers of w, b, λ, φ, υ

Original: ∅
Revised: ∅

Original: Activation in the middle layers of w, υ Relu
Revised: Activation in the middle layers of w, υ Relu

Original: ∅
Revised: ∅

Original: Activation in the last layer of w, υ
Revised: Activation in the last layer of w, υ

Original: [Table]
Revised: [Table]

Original: in the middle layers of b
Revised: in the middle layers of b

Original: [Table]
Revised: [Table]

Original: last layer of b
Revised: last layer of b

Original: [Table]
Revised: [Table]

Original: in the middle layers of λ, φ
Revised: in the middle layers of λ, φ

Original: [Table]
Revised: [Table]

Original: last
Revised: last

Original: [Table]
Revised: [Table]

Original: , φ
Revised: , φ

Original: [Table]
Revised: [Table]

Original: Our training time on an NVIDIA RTX 2080TI GPU of each task is about 6 hours to 20 hours, depending on the agent number and the episode length limit of each map.
Revised: Our training time on an NVIDIA RTX 2080TI GPU of each task is about 6 hours to 20 hours, depending on the agent number and the episode length limit of each map.

Original: The percentage of episodes in which MARL agents defeat all enemy units within the time limit is called test win rate .
Revised: The percentage of episodes in which MARL agents defeat all enemy units within the time limit is called test win rate .

Original: We pause training every 10k timesteps and evaluate 32 episodes with decentralized greedy action selection to measure test win rate of each algorithm.
Revised: We pause training every 10k timesteps and evaluate 32 episodes with decentralized greedy action selection to measure test win rate of each algorithm.

Original: After training every 200 episodes, the target network will be updated once.
Revised: After training every 200 episodes, the target network will be updated once.

Original: We call this update period an Iteration for didactic tasks.
Revised: We call this update period an Iteration for didactic tasks.

Original: In the two-state MMDP,
Revised: In the two-state MMDP,

Original: Optimal line of Figure 3 is approximately (cid:80) 99 i =

Revised: Optimal line of Figure 3 is approximately (cid:80) 99 i =

Original: γ i = 63 .
Revised: γ i = 63 .

Original: 4 in one episode of 100 timesteps.
Revised: 4 in one episode of 100 timesteps.

Original: Training with Online Data Collection We have collected a total of 2 million timestep data for each task and test the model every 10 thousand steps.
Revised: Training with Online Data Collection We have collected a total of 2 million timestep data for each task and test the model every 10 thousand steps.

Original: We use (cid:15) -greedy exploration and a limited ﬁrst-in-ﬁrst-out (FIFO) replay buffer of size 5000 episodes, where (cid:15) is linearly annealed from 1.0 to 0.05 over 50k timesteps and keep it constant for the rest training process.
Revised: We use (cid:15) -greedy exploration and a limited ﬁrst-in-ﬁrst-out (FIFO) replay buffer of size 5000 episodes, where (cid:15) is linearly annealed from 1.0 to 0.05 over 50k timesteps and keep it constant for the rest training process.

Original: To utilize the training buffer more efﬁciently, we perform gradient updates twice with a batch of 32 episodes after collecting every episode for each algorithm.
Revised: To utilize the training buffer more efﬁciently, we perform gradient updates twice with a batch of 32 episodes after collecting every episode for each algorithm.

Original: Training with Ofﬂine Data Collection To construct a diverse dataset, we train a behavior policy of QMIX (Rashid et al., 2018) or VDN (Sunehag et al., 2018) and collect its 20k or 50k experienced episodes throughout the training process.
Revised: Training with Ofﬂine Data Collection To construct a diverse dataset, we train a behavior policy of QMIX (Rashid et al., 2018) or VDN (Sunehag et al., 2018) and collect its 20k or 50k experienced episodes throughout the training process.

Original: The dataset conﬁgurations are shown in Table 3.
Revised: The dataset conﬁgurations are shown in Table 3.

Original: We evaluate QPLEX and four baselines over six random seeds, which includes three different datasets and tests two seeds on each dataset.
Revised: We evaluate QPLEX and four baselines over six random seeds, which includes three different datasets and tests two seeds on each dataset.

Original: We train 300 epochs to demonstrate our learning performance, where each epoch trains 160k transitions with a batch of 32 episodes.
Revised: We train 300 epochs to demonstrate our learning performance, where each epoch trains 160k transitions with a batch of 32 episodes.

Original: Moreover, the training process of behavior policy is the same as that discussed in PyMARL (Samvelyan et al., 2019).
Revised: Moreover, the training process of behavior policy is the same as that discussed in PyMARL (Samvelyan et al., 2019).

Original: [Table]
Revised: [Table]

Original: C D EFERRED F IGURES AND T ABLES IN S ECTION 4.1 AND 4. 
Revised: C D EFERRED F IGURES AND T ABLES IN S ECTION 4.1 AND 4.

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: a
Revised: a

Original: A (1)
Revised: A (1)

Original: A (2)
Revised: A (2)

Original: A (3)
Revised: A (3)

Original: A (1) -6.
Revised: A (1) -6.

Original: [Table]
Revised: [Table]

Original: A (2) -
Revised: A (2) -

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A (3) -
Revised: A (3) -

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: (e) Q tot of VDN
Revised: (e) Q tot of VDN

Original: a
Revised: a

Original: [Table]
Revised: [Table]

Original: a
Revised: a

Original: [Table]
Revised: [Table]

Original: A (
Revised: A (

Original: [Table]
Revised: [Table]

Original: A (
Revised: A (

Original: [Table]
Revised: [Table]

Original: A (
Revised: A (

Original: [Table]
Revised: [Table]

Original: A (1) -
Revised: A (1) -

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A
Revised: A

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A
Revised: A

Original: [Table]
Revised: [Table]

Original: (f) Q tot of Qatten
Revised: (f) Q tot of Qatten

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Large QMIX with data collection.
Revised: Large QMIX with data collection.

Original: G D EFERRED V ISUALIZED S TRATEGIES ON 5 S 10 Z S CENARIO
Revised: G D EFERRED V ISUALIZED S TRATEGIES ON 5 S 10 Z S CENARIO

Original: [Figure]
Revised: [Figure]

Original: As shown in Figure
Revised: As shown in Figure

Original: [Figure]
Revised: [Figure]

Original: both MARL agents and opponents contain 5 ranged soldiers (denoted by a circle) and 10 melee soldiers (denoted by line) on 5s10z map.
Revised: both MARL agents and opponents contain 5 ranged soldiers (denoted by a circle) and 10 melee soldiers (denoted by line) on 5s10z map.

Original: The ranged soldiers have stronger combat capabilities and need to be protected strategically.
Revised: The ranged soldiers have stronger combat capabilities and need to be protected strategically.

Original: QPLEX uses 10 melee soldiers to build lines of defense against the enemy, while QMIX fails to coordinate melee soldiers such that ranged soldiers have to ﬁght against the enemy directly
Revised: QPLEX uses 10 melee soldiers to build lines of defense against the enemy, while QMIX fails to coordinate melee soldiers such that ranged soldiers have to ﬁght against the enemy directly

Original: [Figure]
Revised: [Figure]

Original: H D EFERRED F IGURES OF S TAR C RAFT II WITH O FFLINE D ATA C OLLECTION
Revised: H D EFERRED F IGURES OF S TAR C RAFT II WITH O FFLINE D ATA C OLLECTION

Original: [Figure]
Revised: [Figure]

Original: I A BLATION S TUDY ABOUT QPLEX WITH D IFFERENT N ETWORK C APACITIES IN S TAR C RAFT II
Revised: I A BLATION S TUDY ABOUT QPLEX WITH D IFFERENT N ETWORK C APACITIES IN S TAR C RAFT II

Original: We trade off the expressiveness and learning efﬁciency of the multi-head attention network module for estimating importance weights λ
Revised: We trade off the expressiveness and learning efﬁciency of the multi-head attention network module for estimating importance weights λ

Original: i (see Eq.
Revised: i (see Eq.

Original: (10) and (11) in Section 3.2).
Revised: (10) and (11) in Section 3.2).

Original: It is generally sufﬁcient for
Revised: It is generally sufﬁcient for

Original: QPLEX to use a simple multi-head attention with a small number of heads and layers (as evaluated in
Revised: QPLEX to use a simple multi-head attention with a small number of heads and layers (as evaluated in

Original: StarCraft II tasks) to achieve the state-of-the-art performance shown in Figure 4.
Revised: StarCraft II tasks) to achieve the state-of-the-art performance shown in Figure 4.

Original: However, in some didactic corner-cases with an adequate and uniform dataset, a harder matrix game illustrated in Figure 2a requires very high precision in estimating the action-value function in order to differentiate the optimal solution from the sub-optimal solutions.
Revised: However, in some didactic corner-cases with an adequate and uniform dataset, a harder matrix game illustrated in Figure 2a requires very high precision in estimating the action-value function in order to differentiate the optimal solution from the sub-optimal solutions.

Original: For this matrix game, a multi-head attention structure with more layers and heads has a more representational capacity and results in better performance, as demonstrated by the ablation study illustrated in Figure 2c in Section 4.1.
Revised: For this matrix game, a multi-head attention structure with more layers and heads has a more representational capacity and results in better performance, as demonstrated by the ablation study illustrated in Figure 2c in Section 4.1.

Original: [Figure]
Revised: [Figure]

Original: In contrast, StarCraft II micromanagement benchmark tasks contain much more complicated agents with large state-action spaces and range from 2 to 27 agents.
Revised: In contrast, StarCraft II micromanagement benchmark tasks contain much more complicated agents with large state-action spaces and range from 2 to 27 agents.

Original: To support the superior training scalability of QPLEX, we used a multi-head attention with just one layer and four heads.
Revised: To support the superior training scalability of QPLEX, we used a multi-head attention with just one layer and four heads.

Original: To evaluate the effect of the choice of layer and the number of heads, we conducted an ablation study in
Revised: To evaluate the effect of the choice of layer and the number of heads, we conducted an ablation study in

Original: Starcraft II benchmark tasks.
Revised: Starcraft II benchmark tasks.

Original: For simplicity, we follow the notation of Figure 2, i.e., use QPLEX- a L b H to denote QPLEX with a layers and b heads of importance weights λ i .
Revised: For simplicity, we follow the notation of Figure 2, i.e., use QPLEX- a L b H to denote QPLEX with a layers and b heads of importance weights λ i .

Original: As shown in Figure 14, using more heads,
Revised: As shown in Figure 14, using more heads,

Original: QPLEX-1L10H, does not change the performance, but using more layers, QPLEX-2L4H, may slightly degenerate the learning efﬁciency of QPLEX in this complex domain.
Revised: QPLEX-1L10H, does not change the performance, but using more layers, QPLEX-2L4H, may slightly degenerate the learning efﬁciency of QPLEX in this complex domain.

Original: Detailed learning curves on these tasks are shown in Figure 15.
Revised: Detailed learning curves on these tasks are shown in Figure 15.

Original: This is because using more layers signiﬁcantly increase the number of parameters and requires more samples for learning, which may offset the beneﬁts of higher expressiveness it brings.
Revised: This is because using more layers signiﬁcantly increase the number of parameters and requires more samples for learning, which may offset the beneﬁts of higher expressiveness it brings.

Original: This ablation study shows that a simple attention neural network with just one layer and four heads has enough expressiveness to handle these complex StarCraft II tasks.
Revised: This ablation study shows that a simple attention neural network with just one layer and four heads has enough expressiveness to handle these complex StarCraft II tasks.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: J A BLATION S TUDY ABOUT QTRAN
Revised: J A BLATION S TUDY ABOUT QTRAN

Original: Both QPLEX and QTRAN aim to provide a richer factorized acton-value function class.
Revised: Both QPLEX and QTRAN aim to provide a richer factorized acton-value function class.

Original: The main difference between QPLEX and QTRAN is that QPLEX uses a duplex dueling architecture to realize the IGM principle (a hard constraint).
Revised: The main difference between QPLEX and QTRAN is that QPLEX uses a duplex dueling architecture to realize the IGM principle (a hard constraint).

Original: In contrast, QTRAN uses two penalties as soft constraints to approximate the IGM principle.
Revised: In contrast, QTRAN uses two penalties as soft constraints to approximate the IGM principle.

Original: Moreover, from the perspective of implementation, QTRAN does not have a Transformation module (see Section 3.2) on the individual Q-functions and cannot utilize a multi-head attention module on the joint Q-function directly (because QTRAN does not take the duplex dueling architecture as QPLEX).
Revised: Moreover, from the perspective of implementation, QTRAN does not have a Transformation module (see Section 3.2) on the individual Q-functions and cannot utilize a multi-head attention module on the joint Q-function directly (because QTRAN does not take the duplex dueling architecture as QPLEX).

Original: [Figure]
Revised: [Figure]

Original: To test whether QPLEX outperforms QTRAN because of these factors, we conducted an ablation study by removing the Transformation module and replacing the multi-head attention module with a simple one-layer forward model in the QPLEX’s dueling architecture, which is denoted as QPLEX-wo-trans-atten .
Revised: To test whether QPLEX outperforms QTRAN because of these factors, we conducted an ablation study by removing the Transformation module and replacing the multi-head attention module with a simple one-layer forward model in the QPLEX’s dueling architecture, which is denoted as QPLEX-wo-trans-atten .

Original: In addition, we also introduce a variant of QTRAN, which also uses the Transformation module for individual Qfunctions, denoted as QTRAN-w-trans .
Revised: In addition, we also introduce a variant of QTRAN, which also uses the Transformation module for individual Qfunctions, denoted as QTRAN-w-trans .

Original: Our experiments are evaluated on the tasks of the StarCraft II benchmark mentioned in Section 4.3.1.
Revised: Our experiments are evaluated on the tasks of the StarCraft II benchmark mentioned in Section 4.3.1.

Original: Figure 16 illustrates the averaged median test win rate % over all tested scenarios.
Revised: Figure 16 illustrates the averaged median test win rate % over all tested scenarios.

Original: Detailed learning curves on these tasks are shown in Figure 17.
Revised: Detailed learning curves on these tasks are shown in Figure 17.

Original: These empirical results show that QPLEX-wo-trans-atten signiﬁcantly outperforms QTRAN and QTRAN-w-trans, which implies that the outperformance of QPLEX over QTRAN is largely due to its duplex dueling architecture.
Revised: These empirical results show that QPLEX-wo-trans-atten signiﬁcantly outperforms QTRAN and QTRAN-w-trans, which implies that the outperformance of QPLEX over QTRAN is largely due to its duplex dueling architecture.

Original: It can also be seen that
Revised: It can also be seen that

Original: QTRAN-w-trans cannot signiﬁcantly improve the performance of QTRAN, which implies that
Revised: QTRAN-w-trans cannot signiﬁcantly improve the performance of QTRAN, which implies that

Original: QTRAN cannot beneﬁt a lot from an extra Transformation module empirically.
Revised: QTRAN cannot beneﬁt a lot from an extra Transformation module empirically.

Original: As discussed in Section 4.3.2, Figure 6a can be regarded as an ablation study of QPLEX-wo-atten , which just removes the multi-head attention module from dueling architecture.
Revised: As discussed in Section 4.3.2, Figure 6a can be regarded as an ablation study of QPLEX-wo-atten , which just removes the multi-head attention module from dueling architecture.

Original: That ablation study shows that a simple neural network implementation of QPLEX’s dueling structure has enough expressiveness to handle these StarCraft II tasks in the online data collection setting, even if this structure can provide QPLEX with excellent performance in didactic matrix games using an adequate and uniform dataset (see Section 4.1).
Revised: That ablation study shows that a simple neural network implementation of QPLEX’s dueling structure has enough expressiveness to handle these StarCraft II tasks in the online data collection setting, even if this structure can provide QPLEX with excellent performance in didactic matrix games using an adequate and uniform dataset (see Section 4.1).

Original: Thus, compared with QPLEX-wo-atten in Figure 6a, Figureshows that Transformation (abbreviated as -trans ) is a useful module for QPLEX empirically, which indicates an another QPLEX’s advantage, i.e., QPLEX can equip a Transformation module to improve
Revised: Thus, compared with QPLEX-wo-atten in Figure 6a, Figureshows that Transformation (abbreviated as -trans ) is a useful module for QPLEX empirically, which indicates an another QPLEX’s advantage, i.e., QPLEX can equip a Transformation module to improve

Original: QPLEX’s empirical performance, whereas QTRAN may not by directly using it.
Revised: QPLEX’s empirical performance, whereas QTRAN may not by directly using it.

Original: Moreover, we would like to emphasize that, unlike the multi-head attention module, the Transformation module is actually a necessary module for QPLEX to realize IGM, as shown in Figure 1 and by the proof of
Revised: Moreover, we would like to emphasize that, unlike the multi-head attention module, the Transformation module is actually a necessary module for QPLEX to realize IGM, as shown in Figure 1 and by the proof of

Original: Proposition 2.
Revised: Proposition 2.

Original: Therefore, it is actually fair to evaluate QPLEX with the Transformation module. 
Revised: Therefore, it is actually fair to evaluate QPLEX with the Transformation module. [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Revised: K C OMPARE TO W EIGHTED QMIX

Revised: Weighted QMIX (Rashid et al., 2020) is a recent advanced multi-agent Q-learning algorithm.

Revised: We compare QPLEX with Weighted QMIX in StarCraft II benchmark tasks and a toy game, predatorprey .

Revised: We evaluate these algorithms on 17 scenarios of the StarCraft II benchmark, which contains 14 popular standard maps proposed by SMAC (Samvelyan et al., 2019) and three new super hard cooperative tasks, as mentioned in Section 4.3.1.

Revised: Toy game predator-prey is also considered, which is introduced by Weighted QMIX and aims to test coordination between agents in a partially-observable setting.

Original: [Figure]
Revised: [Figure]

Revised: In this subsection, we conduct an extensive evaluation and compare QPLEX to Weighted QMIX (OW-QMIX and CW-QMIX; Rashid et al., 2020) in the 17 StarCraft II benchmark tasks.

Revised: The implementation of Weighted QMIX are referred to in their available GitHub code.

Revised: Empirical results show that QPLEX signiﬁcantly outperforms OW-QMIX and CW-QMIX.

Revised: As illustrated in Figure 18a,

Revised: QPLEX constantly and signiﬁcantly outperforms Weighted QMIX over the whole training process and exceeds the best baseline with at least 10% median test win rate averaged across all 17 scenarios.

Revised: As shown in Figure 18b, among all 17 tasks, QPLEX is the best performer on up to eight tasks, underperforms on just two tasks, and ties for the best performer with some best baselines on the rest tasks.

Revised: After one million timesteps, the number of tasks that QPLEX achieves the best performance gradually decreases to ﬁve, because, in several easy tasks, other baselines also reach almost 100% test win rate as shown in Figure 19, which shows the individual learning curves on 17 tasks.

Revised: Moreover, although OW-QMIX or CW-QMIX outperforms QMIX in some tasks (illustrated in Figure 19, e.g., 2c_vs_64zg and bane_vs_bane), OW-QMIX and CW-QMIX show very similar overall performance as QMIX across 17 StarCraft II benchmark tasks, as illustrated in Figure 18a.

Revised: In contrast, QPLEX achieves signiﬁcant improvement in convergence performance in a lot of hard and super hard maps and demonstrates high sample efﬁciency across most scenarios (see Figure 19).

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: Predator-prey is a multi-agent coordinated game used by Weighted QMIX (Rashid et al., 2020) with miscoordination penalties.

Revised: In order to collect the experience with the positive reward of agents’ coordinated actions, extensive exploration can beneﬁt multi-agent Q-learning algorithms to solve this kind of tasks.

Revised: Weighted QMIX shapes the data distribution with an importance weight to boost efﬁcient learning, which can also be regarded as a type of biased exploration.

Revised: To support QPLEX with effective exploration, we use an (cid:15) -greedy strategy which is also discussed in the paper of Weighted

Revised: QMIX.

Revised: This strategy’s (cid:15) is linearly annealed from 1 to 0.05 over 1 million timesteps, increased from the 50k used in SMAC (Samvelyan et al., 2019).

Revised: As shown in Figure 20, besides Weighted QMIX and

Revised: QTRAN, QPLEX can solve this task by using the introduced (cid:15) -greedy exploration strategy.

Revised: Moreover,

Revised: QMIX can also solve this task by using the same (cid:15) -greedy strategy as QPLEX, but QPLEX enjoys higher sample efﬁciency.