Original: UTS: W HEN MONOTONIC VALUE FACTORISATION MEETS NON - MONOTONIC AND STOCHASTIC TARGETS
Revised: UTS: W HEN MONOTONIC VALUE FACTORISATION MEETS NON - MONOTONIC AND STOCHASTIC TARGETS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In the paradigm of centralised training with decentralised execution, monotonic value decomposition is one of the most popular methods to guarantee consistency between centralised and decentralised policies.
Revised: In the paradigm of centralised training with decentralised execution, monotonic value decomposition is one of the most popular methods to guarantee consistency between centralised and decentralised policies.

Original: This method always underestimates the value of the optimal joint action and converges to the suboptimal because it can only represent values in the restricted monotonic space.
Revised: This method always underestimates the value of the optimal joint action and converges to the suboptimal because it can only represent values in the restricted monotonic space.

Original: A possible way to rectify this issue is to introduce a weighting function to prioritise the real optimal joint action and learn biased joint action-value functions.
Revised: A possible way to rectify this issue is to introduce a weighting function to prioritise the real optimal joint action and learn biased joint action-value functions.

Original: However, there may not exist an appropriate weight to solve more general tasks with non-monotonic and stochastic target joint action-values.
Revised: However, there may not exist an appropriate weight to solve more general tasks with non-monotonic and stochastic target joint action-values.

Original: To solve this problem, we propose a novel value factorisation method named uncertainty-based target shaping (UTS), which projects the original target to the space that monotonic value factorisation can represent based on its stochasticity.
Revised: To solve this problem, we propose a novel value factorisation method named uncertainty-based target shaping (UTS), which projects the original target to the space that monotonic value factorisation can represent based on its stochasticity.

Original: First, we employ networks to predict the reward and the embedding of the next state, where the prediction error quantifies the stochasticity.
Revised: First, we employ networks to predict the reward and the embedding of the next state, where the prediction error quantifies the stochasticity.

Original: Then, we introduce a target shaping function to replace the targets for deterministic suboptimal with the best per-agent value.
Revised: Then, we introduce a target shaping function to replace the targets for deterministic suboptimal with the best per-agent value.

Original: Since we remain the optimal policy unchanged during shaping, monotonic value decomposition can converge to the real optimal with any original targets.
Revised: Since we remain the optimal policy unchanged during shaping, monotonic value decomposition can converge to the real optimal with any original targets.

Original: Theoretical and empirical results demonstrate the improved performance of UTS in the task with non-monotonic and stochastic target action-value functions.
Revised: Theoretical and empirical results demonstrate the improved performance of UTS in the task with non-monotonic and stochastic target action-value functions.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent progress in cooperative multi-agent reinforcement learning (MARL) has shown attractive prospects for various real-world applications, such as the smart grid management (Aladdin et al., 2020) and autonomous vehicles (Zhou et al., 2021).
Revised: Recent progress in cooperative multi-agent reinforcement learning (MARL) has shown attractive prospects for various real-world applications, such as the smart grid management (Aladdin et al., 2020) and autonomous vehicles (Zhou et al., 2021).

Original: Due to practical communication constraints and intractably large joint action space, decentralised policies are often used in MARL.
Revised: Due to practical communication constraints and intractably large joint action space, decentralised policies are often used in MARL.

Original: It is possible to use extra information from the environment and other agents in a simulated or laboratory setting.
Revised: It is possible to use extra information from the environment and other agents in a simulated or laboratory setting.

Original: Exploiting this information can significantly benefit policy optimisation and improve learning performance (Foerster et al., 2016; 2018; Rashid et al., 2020).
Revised: Exploiting this information can significantly benefit policy optimisation and improve learning performance (Foerster et al., 2016; 2018; Rashid et al., 2020).

Original: In the paradigm of centralised training with decentralised execution (CTDE), agents’ policies are trained with access to global information in a centralised way and executed only based on local histories in a decentralised way (Oliehoek et al., 2008; Kraemer & Banerjee, 2016).
Revised: In the paradigm of centralised training with decentralised execution (CTDE), agents’ policies are trained with access to global information in a centralised way and executed only based on local histories in a decentralised way (Oliehoek et al., 2008; Kraemer & Banerjee, 2016).

Original: One of the most significant challenges is to guarantee the consistency between the individual policies and the centralised policy, i.e., the Individual-Global Max (IGM) principle (Son et al., 2019).
Revised: One of the most significant challenges is to guarantee the consistency between the individual policies and the centralised policy, i.e., the Individual-Global Max (IGM) principle (Son et al., 2019).

Original: In value decomposition methods, QMIX (Rashid et al., 2018) applies a monotonic mixing network to factorise the joint Q -value function, which naturally meets the IGM principle.
Revised: In value decomposition methods, QMIX (Rashid et al., 2018) applies a monotonic mixing network to factorise the joint Q -value function, which naturally meets the IGM principle.

Original: Inspired by QMIX, many algorithms are proposed to improve coordination from different perspectives, e.g., multi-agent exploration (Mahajan et al., 2019), role-based learning (Wang et al., 2020b;c), and policy-based algorithms (Wang et al., 2020d).
Revised: Inspired by QMIX, many algorithms are proposed to improve coordination from different perspectives, e.g., multi-agent exploration (Mahajan et al., 2019), role-based learning (Wang et al., 2020b;c), and policy-based algorithms (Wang et al., 2020d).

Original: However, they can represent the same class of joint Q -values as
Revised: However, they can represent the same class of joint Q -values as

Original: QMIX because they use the same monotonic mixing network.
Revised: QMIX because they use the same monotonic mixing network.

Original: However, since QMIX can only represent values in the restricted monotonic space, there exists a gap between the approximated joint Q -values and the non-monotonic target values Q from the environment.
Revised: However, since QMIX can only represent values in the restricted monotonic space, there exists a gap between the approximated joint Q -values and the non-monotonic target values Q from the environment.

Original: In some special tasks, QMIX can underestimate the value of the real optimal joint action and converge to a suboptimal (Son et al., 2019; Mahajan et al., 2019; Rashid et al., 2020).
Revised: In some special tasks, QMIX can underestimate the value of the real optimal joint action and converge to a suboptimal (Son et al., 2019; Mahajan et al., 2019; Rashid et al., 2020).

Original: Recent works try to solve this representational limitation from two different perspectives.
Revised: Recent works try to solve this representational limitation from two different perspectives.

Original: This first category introduces the joint actions (Wang et al., 2020a; Mahajan et al., 2021) or pairwise interactions (B¨ohmer et al., 2020; Li et al., 2021) into centralised learning to achieve full representational capacity for target Q -values.
Revised: This first category introduces the joint actions (Wang et al., 2020a; Mahajan et al., 2021) or pairwise interactions (B¨ohmer et al., 2020; Li et al., 2021) into centralised learning to achieve full representational capacity for target Q -values.

Original: However, learning such centralised values is difficult due to the large joint action space.
Revised: However, learning such centralised values is difficult due to the large joint action space.

Original: Another category is to prioritise the real optimal joint action and learn biased joint Q -value functions.
Revised: Another category is to prioritise the real optimal joint action and learn biased joint Q -value functions.

Original: WQMIX (Rashid et al., 2020) introduces a weighting function into the projection from the target value functions to the joint Q -values and uses it to down-weight every suboptimal action whose target value is less than the current estimate.
Revised: WQMIX (Rashid et al., 2020) introduces a weighting function into the projection from the target value functions to the joint Q -values and uses it to down-weight every suboptimal action whose target value is less than the current estimate.

Original: However, the poor empirical results on decentralised micromanagement tasks in StarCraft II show that it is difficult to apply the weighting function to more general tasks (Rashid et al., 2020; Wang et al., 2020a).
Revised: However, the poor empirical results on decentralised micromanagement tasks in StarCraft II show that it is difficult to apply the weighting function to more general tasks (Rashid et al., 2020; Wang et al., 2020a).

Original: We prove that the weight for the suboptimal should be small to help QMIX focus on the representation of the optimal joint Q -value and solves the non-monotonic targets.
Revised: We prove that the weight for the suboptimal should be small to help QMIX focus on the representation of the optimal joint Q -value and solves the non-monotonic targets.

Original: In addition, the weight for each action should be uniform to avoid overestimating the suboptimal whose target is large with a low probability.
Revised: In addition, the weight for each action should be uniform to avoid overestimating the suboptimal whose target is large with a low probability.

Original: Due to this contradiction, there may not exist an appropriate weight to recover the optimal policy when the target is non-monotonic and stochastic.
Revised: Due to this contradiction, there may not exist an appropriate weight to recover the optimal policy when the target is non-monotonic and stochastic.

Original: This paper aims to take a step towards the latter category.
Revised: This paper aims to take a step towards the latter category.

Original: We propose a novel value factorisation method named uncertainty-based target shaping (UTS), which projects the original target to the space that monotonic value factorisation can represent based on its stochasticity.
Revised: We propose a novel value factorisation method named uncertainty-based target shaping (UTS), which projects the original target to the space that monotonic value factorisation can represent based on its stochasticity.

Original: First, we formulate two prediction problems and use the prediction error to quantify the stochasticity of the target joint
Revised: First, we formulate two prediction problems and use the prediction error to quantify the stochasticity of the target joint

Original: Q -values.
Revised: Q -values.

Original: We employ a reward predictor and a state predictor to approximate the standard deviation of the reward and the embedding of the following state.
Revised: We employ a reward predictor and a state predictor to approximate the standard deviation of the reward and the embedding of the following state.

Original: The predicted standard deviation of the reward and the error of the state are expected to be significant if the pair leads to stochastic reward and stochastic state transition, respectively.
Revised: The predicted standard deviation of the reward and the error of the state are expected to be significant if the pair leads to stochastic reward and stochastic state transition, respectively.

Original: Then, we introduce a shaping function to project the original targets Q to the monotonic space and keep the optimal policy unchanged.
Revised: Then, we introduce a shaping function to project the original targets Q to the monotonic space and keep the optimal policy unchanged.

Original: In practice, the best action value network is applied to predict the action value when each agent gets the cooperation of others.
Revised: In practice, the best action value network is applied to predict the action value when each agent gets the cooperation of others.

Original: We use the minimal best per-agent values over all agents to replace the suboptimal target.
Revised: We use the minimal best per-agent values over all agents to replace the suboptimal target.

Original: We prove that this shaping can guarantee that all shaped targets are tractable for monotonic value decomposition.
Revised: We prove that this shaping can guarantee that all shaped targets are tractable for monotonic value decomposition.

Original: In addition, the optimal policy is the same for the original and the shaped targets.
Revised: In addition, the optimal policy is the same for the original and the shaped targets.

Original: Therefore, QMIX can achieve full representational capacity for the shaped target Q -values rather than the original ones and converge to the real optimal.
Revised: Therefore, QMIX can achieve full representational capacity for the shaped target Q -values rather than the original ones and converge to the real optimal.

Original: We list our main contributions as follows:
Revised: We list our main contributions as follows:

Original: • We first analyse the limitations of the weighting function in value decomposition methods and show that it cannot guarantee convergence to the optimal when target Q -value functions are non-monotonic and stochastic.
Revised: • We first analyse the limitations of the weighting function in value decomposition methods and show that it cannot guarantee convergence to the optimal when target Q -value functions are non-monotonic and stochastic.

Original: • We introduce a target shaping function to project the original targets to a monotonic space, which ensures that QMIX can converge to the optimal with any original targets.
Revised: • We introduce a target shaping function to project the original targets to a monotonic space, which ensures that QMIX can converge to the optimal with any original targets.

Original: • We propose uncertainty-based target shaping and empirically show its improved performance in practice, especially in tasks with non-monotonic and stochastic targets.
Revised: • We propose uncertainty-based target shaping and empirically show its improved performance in practice, especially in tasks with non-monotonic and stochastic targets.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: A fully cooperative multi-agent task in the partially observable setting can be formulated as a
Revised: A fully cooperative multi-agent task in the partially observable setting can be formulated as a

Original: Decentralised Partially Observable Markov Decision Process (Dec-POMDP) (Oliehoek & Amato, 2016), consisting of a tuple G = ⟨ A, S, Ω , O, U, P, R, n, γ ⟩ , where a ∈ A ≡ { 1 , . . . , n } describes the set of agents, S denotes the set of states, Ω denotes the set of joint observations, and R denotes the set of rewards.
Revised: Decentralised Partially Observable Markov Decision Process (Dec-POMDP) (Oliehoek & Amato, 2016), consisting of a tuple G = ⟨ A, S, Ω , O, U, P, R, n, γ ⟩ , where a ∈ A ≡ { 1 , . . . , n } describes the set of agents, S denotes the set of states, Ω denotes the set of joint observations, and R denotes the set of rewards.

Original: At each time step, an agent obtains its observation o ∈ Ω based on the observation function
Revised: At each time step, an agent obtains its observation o ∈ Ω based on the observation function

Original: O ( s, a ) : S ×
Revised: O ( s, a ) : S ×

Original: A → Ω , and an action-observation history τ a ∈ T ≡ (Ω × U ) ∗ .
Revised: A → Ω , and an action-observation history τ a ∈ T ≡ (Ω × U ) ∗ .

Original: Each agent a chooses an action u a ∈ U by a stochastic policy π a ( u a | τ a ) : T × U → [0 , 1] , forming a joint action u ∈ U , which leads to a transition on the environment through the transition function P ( s ′ , r | s, u ) :
Revised: Each agent a chooses an action u a ∈ U by a stochastic policy π a ( u a | τ a ) : T × U → [0 , 1] , forming a joint action u ∈ U , which leads to a transition on the environment through the transition function P ( s ′ , r | s, u ) :

Original: S × U × S × R → [0 , 1] , where r ∈ R is the team reward.
Revised: S × U × S × R → [0 , 1] , where r ∈ R is the team reward.

Original: The goal of the task is to find the joint policy π which can maximise the joint Q -value function Q π ( s t , u t ) =
Revised: The goal of the task is to find the joint policy π which can maximise the joint Q -value function Q π ( s t , u t ) =

Original: E s t +1: ∞ , u t +1: ∞ [ G t | s t , u ] , where G t = (cid:80) ∞ i =

Revised: E s t +1: ∞ , u t +1: ∞ [ G t | s t , u ] , where G t = (cid:80) ∞ i =

Original: γ i r t
Revised: γ i r t

Original: + i is the discounted return.
Revised: + i is the discounted return.

Original: VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2018), and WQMIX (Rashid et al., 2020) are
Revised: VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2018), and WQMIX (Rashid et al., 2020) are

Original: Q -learning algorithms for the fully cooperative multi-agent tasks, which estimate the joint Q -value function Q ( s, u ) as Q tot with specific forms.
Revised: Q -learning algorithms for the fully cooperative multi-agent tasks, which estimate the joint Q -value function Q ( s, u ) as Q tot with specific forms.

Original: Considering only a fully-observable setting for ease of representation.
Revised: Considering only a fully-observable setting for ease of representation.

Original: VDN factorises Q tot into a sum of individual Q -value functions.
Revised: VDN factorises Q tot into a sum of individual Q -value functions.

Original: By contrast,
Revised: By contrast,

Original: QMIX applies a state-dependent monotonic mixing network to combine per-agent Q -value functions with the joint Q -value function.
Revised: QMIX applies a state-dependent monotonic mixing network to combine per-agent Q -value functions with the joint Q -value function.

Original: The restricted spaces of all Q tot that linear and monotonic value factorisations can represent are:
Revised: The restricted spaces of all Q tot that linear and monotonic value factorisations can represent are:

Original: [Equation]
Revised: [Equation]

Original: where the monotonic mixing function f s is parametrised as a feedforward network, whose nonnegative weights are generated by hypernetworks that take the state as input.
Revised: where the monotonic mixing function f s is parametrised as a feedforward network, whose nonnegative weights are generated by hypernetworks that take the state as input.

Original: WQMIX views QMIX as an operator
Revised: WQMIX views QMIX as an operator

Original: [Equation]
Revised: [Equation]

Original: which can be separated into two parts: the first computes the Q -learning target y ( s, u ) :
Revised: which can be separated into two parts: the first computes the Q -learning target y ( s, u ) :

Original: =
Revised: =

Original: E [ r + γ max u ′ Q tot ( s ′ , u ′ )] , and the second projects the target into Q mvf .
Revised: E [ r + γ max u ′ Q tot ( s ′ , u ′ )] , and the second projects the target into Q mvf .

Original: Then, WQMIX introduces the weighting function into the projection to prioritise the optimal joint Q -value:
Revised: Then, WQMIX introduces the weighting function into the projection to prioritise the optimal joint Q -value:

Original: [Equation]
Revised: [Equation]

Original: Since it is computationally infeasible to obtain the optimal joint Q -value, WQMIX proposes CentrallyWeighted QMIX (CW-QMIX) and
Revised: Since it is computationally infeasible to obtain the optimal joint Q -value, WQMIX proposes CentrallyWeighted QMIX (CW-QMIX) and

Original: Optimistically-Weighted QMIX (OW-QMIX) to place more importance on better joint actions.
Revised: Optimistically-Weighted QMIX (OW-QMIX) to place more importance on better joint actions.

Original: In practice, the weighting function w : S × U → (0 , 1] is: w cw ( s, u ) =
Revised: In practice, the weighting function w : S × U → (0 , 1] is: w cw ( s, u ) =

Original: (cid:40)y ( s, u ) >
Revised: (cid:40)y ( s, u ) >

Original: ˆ Q ∗ ( s, ˆ u ∗ ) or u
Revised: ˆ Q ∗ ( s, ˆ u ∗ ) or u

Original: = ˆ u ∗ α otherwise , w ow ( s, u ) =
Revised: = ˆ u ∗ α otherwise , w ow ( s, u ) =

Original: (cid:26)y ( s, u ) > Q tot ( s, u )
Revised: (cid:26)y ( s, u ) > Q tot ( s, u )

Original: α otherwise , (4) where ˆ u ∗ =
Revised: α otherwise , (4) where ˆ u ∗ =

Original: arg max u Q tot ( s, u ) is the current greedy joint action, ˆ Q ∗ ( s, ˆ u ∗ ) is an approximation of the true optimal value function which is not constrained to be monotonic.
Revised: arg max u Q tot ( s, u ) is the current greedy joint action, ˆ Q ∗ ( s, ˆ u ∗ ) is an approximation of the true optimal value function which is not constrained to be monotonic.

Original:  C ASE S TUDIES
Revised:  C ASE S TUDIES

Original: In this section, we examine the weightings of WQMIX when the Q -learning targets are non-monotonic and stochastic.
Revised: In this section, we examine the weightings of WQMIX when the Q -learning targets are non-monotonic and stochastic.

Original: Since the non-linear mixing network in WQMIX makes many analysis methods inapplicable, we use WVDN to analyse the fundamental limitations of the weighting.
Revised: Since the non-linear mixing network in WQMIX makes many analysis methods inapplicable, we use WVDN to analyse the fundamental limitations of the weighting.

Original: The WVDN operator is defined as Π WVDN Q :=
Revised: The WVDN operator is defined as Π WVDN Q :=

Original: arg min Q tot ∈Q lvf (cid:80) u ∈
Revised: arg min Q tot ∈Q lvf (cid:80) u ∈

Original: U w ow ( s, u )( y ( s, u ) − Q
Revised: U w ow ( s, u )( y ( s, u ) − Q

Original: tot
Revised: tot

Original: ( s, u ))
Revised: ( s, u ))

Original: 2 , where w ow ( s, u ) is the optimistic weighting from (4).
Revised: 2 , where w ow ( s, u ) is the optimistic weighting from (4).

Original: 3. NON - MONOTONIC TARGETS
Revised: 3. NON - MONOTONIC TARGETS

Original: QMIX cannot represent joint Q -value functions that are characterised as non-monotonic.
Revised: QMIX cannot represent joint Q -value functions that are characterised as non-monotonic.

Original: Although this representational limitation will not necessarily result in the convergence to the suboptimal, QMIX fails to recover the optimal policy with some specific non-monotonic targets (Son et al., 2019).
Revised: Although this representational limitation will not necessarily result in the convergence to the suboptimal, QMIX fails to recover the optimal policy with some specific non-monotonic targets (Son et al., 2019).

Original: To accurately represent the optimal joint Q -value rather than the suboptimal ones, Rashid et al.
Revised: To accurately represent the optimal joint Q -value rather than the suboptimal ones, Rashid et al.

Original: (2020) introduce the weighting function in (4) to down-weight every suboptimal action.
Revised: (2020) introduce the weighting function in (4) to down-weight every suboptimal action.

Original: However, the choice of weight α is crucial to ensure that WQMIX can overcome the limitations of QMIX.
Revised: However, the choice of weight α is crucial to ensure that WQMIX can overcome the limitations of QMIX.

Original: The weight α should be small enough to prioritise estimating the real optimal action value.
Revised: The weight α should be small enough to prioritise estimating the real optimal action value.

Original: We start by analysing a matrix game in Figure 1a, where each joint action receives a team reward.
Revised: We start by analysing a matrix game in Figure 1a, where each joint action receives a team reward.

Original: Our study focuses on the greatest value α to converge to the optimal when the policy is initialised to be suboptimal.
Revised: Our study focuses on the greatest value α to converge to the optimal when the policy is initialised to be suboptimal.

Original: Suppose the training dataset has a ϵ -greedy distribution, where the greedy action is ( u 21 , u 22 ) .
Revised: Suppose the training dataset has a ϵ -greedy distribution, where the greedy action is ( u 21 , u 22 ) .

Original: We find that α for WVDN should be smaller than 3 ϵ ( a − c )
Revised: We find that α for WVDN should be smaller than 3 ϵ ( a − c )

Original: + ϵ 2 ( c − b ) ( ϵ − 3) 2 ( c − b ) to recover the optimal joint policy.
Revised: + ϵ 2 ( c − b ) ( ϵ − 3) 2 ( c − b ) to recover the optimal joint policy.

Original: The proof can be found in Appendix A.2.
Revised: The proof can be found in Appendix A.2.

Original: In Figure 1b, We show that α for WQMIX also should be smaller than a specific value to overcome the non-monotonic targets.
Revised: In Figure 1b, We show that α for WQMIX also should be smaller than a specific value to overcome the non-monotonic targets.

Original: The complete results are provided in Appendix B.
Revised: The complete results are provided in Appendix B.

Original: 3. S TOCHASTIC TARGETS
Revised: 3. S TOCHASTIC TARGETS

Original: The stochasticity of the target value functions comes from both rewards and transitions, i.e.,
Revised: The stochasticity of the target value functions comes from both rewards and transitions, i.e.,

Original: P ( s ′ , r | s, u ) is not deterministic.
Revised: P ( s ′ , r | s, u ) is not deterministic.

Original: Traditional Q -learning algorithms solve this problem by calculating the weighted sum of possible rewards and values of the next state multiplied by their probabilities, i.e., Q ( s, u )
Revised: Traditional Q -learning algorithms solve this problem by calculating the weighted sum of possible rewards and values of the next state multiplied by their probabilities, i.e., Q ( s, u )

Original: =
Revised: =

Original: (cid:80) s
Revised: (cid:80) s

Original: ′ (cid:80) r
Revised: ′ (cid:80) r

Original: [ P ( s ′ , r | s, u ) r + γP ( s ′ , r | s, u ) max u ′ Q
Revised: [ P ( s ′ , r | s, u ) r + γP ( s ′ , r | s, u ) max u ′ Q

Original: ( s ′ , u ′ )] .
Revised: ( s ′ , u ′ )] .

Original: By contrast, the weighting function in WQMIX prioritises actions whose targets are greater than the estimated joint Q -value from the current mixing network.
Revised: By contrast, the weighting function in WQMIX prioritises actions whose targets are greater than the estimated joint Q -value from the current mixing network.

Original: This weighting can be seen as a type of distribution shaping, leading to potential risks in policy learning.
Revised: This weighting can be seen as a type of distribution shaping, leading to potential risks in policy learning.

Original: The weight α should be large to anchor down the overestimated value for the suboptimal whose target is huge with a small probability .
Revised: The weight α should be large to anchor down the overestimated value for the suboptimal whose target is huge with a small probability .

Original: Since r ( s, u ) and max u ′ Q ( s ′ , u ′ ) play an equal role in the target value function, we take the matrix game with stochastic rewards as an example.
Revised: Since r ( s, u ) and max u ′ Q ( s ′ , u ′ ) play an equal role in the target value function, we take the matrix game with stochastic rewards as an example.

Original: To get a better understanding of the effect of the weight α in the stochastic environment, we show a numerical example in Figure 1c, where the suboptimal action u s =
Revised: To get a better understanding of the effect of the weight α in the stochastic environment, we show a numerical example in Figure 1c, where the suboptimal action u s =

Original: ( u 21 , u 22 ) receives 12 with probability 0 .and 0 with 0 .
Revised: ( u 21 , u 22 ) receives 12 with probability 0 .and 0 with 0 .

Original: 5 , where the optimal is u ∗ = ( u 11 , u 12 ) .
Revised: 5 , where the optimal is u ∗ = ( u 11 , u 12 ) .

Original: Suppose the training dataset is fixed, where the data has a ϵ -greedy distribution and the greedy action is u s .
Revised: Suppose the training dataset is fixed, where the data has a ϵ -greedy distribution and the greedy action is u s .

Original: Let Q ( u ) denote the true joint Q -value.
Revised: Let Q ( u ) denote the true joint Q -value.

Original: Since this matrix game only involves one state, we omit the state s in the value function for simplicity.
Revised: Since this matrix game only involves one state, we omit the state s in the value function for simplicity.

Original: 𝑢𝑢𝑢𝑢𝑢
Revised: 𝑢𝑢𝑢𝑢𝑢

Original: 𝑢𝑢𝑢𝑢b b c b c b b c b c 𝑢 11 𝑈 1 𝑈𝑢𝑢𝑢𝑢𝑢 a
Revised: 𝑢𝑢𝑢𝑢b b c b c b b c b c 𝑢 11 𝑈 1 𝑈𝑢𝑢𝑢𝑢𝑢 a

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: AlgoAnalysis
Revised: AlgoAnalysis

Original: [Figure]
Revised: [Figure]

Original: Figure 1d-h demonstrate the joint Q -values Q tot ( u ) returned from QMIX and WQMIX with different α .
Revised: Figure 1d-h demonstrate the joint Q -values Q tot ( u ) returned from QMIX and WQMIX with different α .

Original: QMIX is a special case of WQMIX ( α = 1 ) and thus represents the joint Q -value perfectly.
Revised: QMIX is a special case of WQMIX ( α = 1 ) and thus represents the joint Q -value perfectly.

Original: By contrast, WQMIX with a small weight is drawn to the suboptimal u s and thus has an incorrect argmax.
Revised: By contrast, WQMIX with a small weight is drawn to the suboptimal u s and thus has an incorrect argmax.

Original: OW-QMIX returns an overestimated Q tot ( u s )
Revised: OW-QMIX returns an overestimated Q tot ( u s )

Original: =
Revised: =

Original: 10 . 9 > Q tot ( u ∗ ) and thus converge to the suboptimal ˆ u ∗ =
Revised: 10 . 9 > Q tot ( u ∗ ) and thus converge to the suboptimal ˆ u ∗ =

Original: u s .
Revised: u s .

Original: In particular, CW-QMIX places more importance on u s when it is not the greedy action and receives the reward of 12, leading to an overestimated value Q tot ( u s )
Revised: In particular, CW-QMIX places more importance on u s when it is not the greedy action and receives the reward of 12, leading to an overestimated value Q tot ( u s )

Original: > Q ( u s )
Revised: > Q ( u s )

Original: = 6 .
Revised: = 6 .

Original: With the increase of Q tot ( u s ) , u s becomes the greedy action once Q tot ( u s )
Revised: With the increase of Q tot ( u s ) , u s becomes the greedy action once Q tot ( u s )

Original: >
Revised: >

Original: Q tot ( u ∗ ) .
Revised: Q tot ( u ∗ ) .

Original: In this situation,
Revised: In this situation,

Original: CW-QMIX does not place the α for u s when it receives the reward of 0, leading to the decrease of
Revised: CW-QMIX does not place the α for u s when it receives the reward of 0, leading to the decrease of

Original: Q tot ( u s ) .
Revised: Q tot ( u s ) .

Original: As a result, CW-QMIX is stuck in this loop and cannot converge to any policy.
Revised: As a result, CW-QMIX is stuck in this loop and cannot converge to any policy.

Original: Theorem 1 Let Π WVDN Q :=
Revised: Theorem 1 Let Π WVDN Q :=

Original: arg min Q tot ∈Q lvf (cid:80) u ∈
Revised: arg min Q tot ∈Q lvf (cid:80) u ∈

Original: U w ow ( s, u )( Q ( s, u ) − Q
Revised: U w ow ( s, u )( Q ( s, u ) − Q

Original: tot ( s, u )) 2 and w ow ( s, u ) is the optimistic weighting from (4).
Revised: tot ( s, u )) 2 and w ow ( s, u ) is the optimistic weighting from (4).

Original: Then ∃ Q such that arg max Π WVDN Q ̸ = arg max Q for any α ∈ (0 , 1] .
Revised: Then ∃ Q such that arg max Π WVDN Q ̸ = arg max Q for any α ∈ (0 , 1] .

Original: The proof is provided in Appendix A.2.
Revised: The proof is provided in Appendix A.2.

Original: An intuitive understanding is that the weight α in WVDN has an upper bound to deal with non-monotonic Q -learning targets and has a lower bound to anchor down the overestimated suboptimal value due to undesirable weights in the stochastic environment.
Revised: An intuitive understanding is that the weight α in WVDN has an upper bound to deal with non-monotonic Q -learning targets and has a lower bound to anchor down the overestimated suboptimal value due to undesirable weights in the stochastic environment.

Original: The empirical results in Appendix B show that WQMIX also suffers from this contradiction and fails to converge to the optimal.
Revised: The empirical results in Appendix B show that WQMIX also suffers from this contradiction and fails to converge to the optimal.

Original:  M ETHOD
Revised:  M ETHOD

Original: In this section, we introduce a shaping function to generate the shaped target Q f ( s, u ) to replace the original target Q ( s, u ) .
Revised: In this section, we introduce a shaping function to generate the shaped target Q f ( s, u ) to replace the original target Q ( s, u ) .

Original: The shaping function has the following properties: 1) policy invariance , which means the optimal policy remains unchanged during shaping, i.e., arg max
Revised: The shaping function has the following properties: 1) policy invariance , which means the optimal policy remains unchanged during shaping, i.e., arg max

Original: u Q f ( s, u ) = arg max u Q ( s, u ) .
Revised: u Q f ( s, u ) = arg max u Q ( s, u ) .

Original: 2) full representational capacity , i.e., all shaped joint Q -values should belong to a subset of Q mvf .
Revised: 2) full representational capacity , i.e., all shaped joint Q -values should belong to a subset of Q mvf .

Original: Note that we achieve the full representational capacity w.r.t the shaped targets rather than the original ones.
Revised: Note that we achieve the full representational capacity w.r.t the shaped targets rather than the original ones.

Original: QMIX can recover the optimal joint policy with arbitrary original targets based on these properties.
Revised: QMIX can recover the optimal joint policy with arbitrary original targets based on these properties.

Original: To this end, we first predict the standard deviation of the reward and the embedding of the next state.
Revised: To this end, we first predict the standard deviation of the reward and the embedding of the next state.

Original: We then split the stochastic joint Q -values from the sampled data according to the prediction error.
Revised: We then split the stochastic joint Q -values from the sampled data according to the prediction error.

Original: Next, we employ a network that estimates the best action value for each agent.
Revised: Next, we employ a network that estimates the best action value for each agent.

Original: Based on the prediction error and the best action values, we use the shaping function to generate shaped target joint Q -values, which can be represented by monotonic value decomposition.
Revised: Based on the prediction error and the best action values, we use the shaping function to generate shaped target joint Q -values, which can be represented by monotonic value decomposition.

Original: 4. U NCERTAINTY ESTIMATION
Revised: 4. U NCERTAINTY ESTIMATION

Original: The stochasticity of the target joint Q -value comes from the stochastic transition of the reward and the state, i.e., P ( s ′ , r | s, u ) , which can be viewed as a type of the aleatoric uncertainty (Der Kiureghian & Ditlevsen, 2009).
Revised: The stochasticity of the target joint Q -value comes from the stochastic transition of the reward and the state, i.e., P ( s ′ , r | s, u ) , which can be viewed as a type of the aleatoric uncertainty (Der Kiureghian & Ditlevsen, 2009).

Original: We propose a solution to this undesirable stochasticity using two prediction problems, where the output is the standard deviation of the reward and the embedding of the following state of a given state-action pair.
Revised: We propose a solution to this undesirable stochasticity using two prediction problems, where the output is the standard deviation of the reward and the embedding of the following state of a given state-action pair.

Original: This involves three neural networks: a reward predictor network, a state predictor, and a fixed and randomly initialised target network.
Revised: This involves three neural networks: a reward predictor network, a state predictor, and a fixed and randomly initialised target network.

Original: The reward predictor network takes the state and joint action to the reward
Revised: The reward predictor network takes the state and joint action to the reward

Original: ˆ r ( s, u ) : S × U → R and the uncertainty, i.e., the standard deviation ˆ σ ( s, u ) :
Revised: ˆ r ( s, u ) : S × U → R and the uncertainty, i.e., the standard deviation ˆ σ ( s, u ) :

Original: S × U → R .
Revised: S × U → R .

Original: Based on the Bayesian deep learning method (Kendall &
Revised: Based on the Bayesian deep learning method (Kendall &

Original: Gal, 2017), the reward predictor network is trained by minimising the following loss:
Revised: Gal, 2017), the reward predictor network is trained by minimising the following loss:

Original: [Equation]
Revised: [Equation]

Original: where B denotes the batch size of transitions sampled from the replay buffer.
Revised: where B denotes the batch size of transitions sampled from the replay buffer.

Original: The standard deviation is expected to be higher if the state-action pair leads to stochastic rewards.
Revised: The standard deviation is expected to be higher if the state-action pair leads to stochastic rewards.

Original: Inspired by RND (Burda et al., 2018), the target network for the second prediction problem takes a state to an m -dimensional embedding g ( s ) :
Revised: Inspired by RND (Burda et al., 2018), the target network for the second prediction problem takes a state to an m -dimensional embedding g ( s ) :

Original: S → R m , where g ( s ) is a fixed and randomly initialised network.
Revised: S → R m , where g ( s ) is a fixed and randomly initialised network.

Original: The state predictor h ( s, u ; θ h ) : S × U → R m is trained to minimise the prediction error
Revised: The state predictor h ( s, u ; θ h ) : S × U → R m is trained to minimise the prediction error

Original: E ( s, u , s ′ )
Revised: E ( s, u , s ′ )

Original: =
Revised: =

Original: ∥ g ( s ′ )
Revised: ∥ g ( s ′ )

Original: − h ( s, u ) ∥ 2 with respect to its parameters θ h , where s ′ denotes the next state in the sampled trajectory.
Revised: − h ( s, u ) ∥ 2 with respect to its parameters θ h , where s ′ denotes the next state in the sampled trajectory.

Original: The prediction error is expected to be higher if the state-action pair results in stochastic transitions.
Revised: The prediction error is expected to be higher if the state-action pair results in stochastic transitions.

Original: Combining the reward and the state predictors, we define an indicator function
Revised: Combining the reward and the state predictors, we define an indicator function

Original: T ( s, u ) = 1 (ˆ σ (s , u ) > D (cid:83) E(s , u , s ′ ) > E ) , where D and E are hyper-parameters.
Revised: T ( s, u ) = 1 (ˆ σ (s , u ) > D (cid:83) E(s , u , s ′ ) > E ) , where D and E are hyper-parameters.

Original: 4. S HAPING F UNCTION
Revised: 4. S HAPING F UNCTION

Original: QMIX cannot represent the non-monotonic target Q -value because it imposes the monotonic constraint on the relationship between the joint Q -value and the individual Q -values.
Revised: QMIX cannot represent the non-monotonic target Q -value because it imposes the monotonic constraint on the relationship between the joint Q -value and the individual Q -values.

Original: Therefore, we can represent the suboptimality of uncooperative actions rather than their exact values.
Revised: Therefore, we can represent the suboptimality of uncooperative actions rather than their exact values.

Original: To quantify the suboptimality of actions, we define the best action value function q a ( s, τ a , u a ) = max u − a Q ( s, u a , u − a ) as the greatest Q -value of u a when others perform optimally to cooperate with it.
Revised: To quantify the suboptimality of actions, we define the best action value function q a ( s, τ a , u a ) = max u − a Q ( s, u a , u − a ) as the greatest Q -value of u a when others perform optimally to cooperate with it.

Original: q a ( s, τ a , u a ) is less than the optimal Q value if u a is not part of the real optimal.
Revised: q a ( s, τ a , u a ) is less than the optimal Q value if u a is not part of the real optimal.

Original: Inspired by BAIL (Chen et al., 2020), we approximate q a ( s, τ a , u a ) by minimising the following loss:
Revised: Inspired by BAIL (Chen et al., 2020), we approximate q a ( s, τ a , u a ) by minimising the following loss:

Original: [Equation]
Revised: [Equation]

Original: where B denotes the batch size of transitions sampled from the replay buffer, y ba ( s, u a )
Revised: where B denotes the batch size of transitions sampled from the replay buffer, y ba ( s, u a )

Original: =
Revised: =

Original: ˆ r ( s, u ) + γ max u ′ a q a ( s ′ , τ ′ a , u ′ a ) is the target, λ is the regularisation coefficients, K ( s, τ a , u a )
Revised: ˆ r ( s, u ) + γ max u ′ a q a ( s ′ , τ ′ a , u ′ a ) is the target, λ is the regularisation coefficients, K ( s, τ a , u a )

Original: =for the Q -value whose target is stochastic or smaller than the current estimate, i.e., y ba
Revised: =for the Q -value whose target is stochastic or smaller than the current estimate, i.e., y ba

Original: ( s, u a ) <
Revised: ( s, u a ) <

Original: q a ( s, τ a , u a ) (cid:83) T ( s, u ) = 1 , and K ( s, τ a , u a ) =
Revised: q a ( s, τ a , u a ) (cid:83) T ( s, u ) = 1 , and K ( s, τ a , u a ) =

Original: k ≫ 1 otherwise.
Revised: k ≫ 1 otherwise.

Original: Equipped with q a ( s, τ a , u a ) , we can compute the original target y
Revised: Equipped with q a ( s, τ a , u a ) , we can compute the original target y

Original: ( s, τ , u ) for joint Q -value Q tot and projects it into y f ( s, τ , u ) .
Revised: ( s, τ , u ) for joint Q -value Q tot and projects it into y f ( s, τ , u ) .

Original: We define the corresponding shaping function as follows:
Revised: We define the corresponding shaping function as follows:

Original: [Equation]
Revised: [Equation]

Original: where Q tot ( s, τ , u ; θ ) =
Revised: where Q tot ( s, τ , u ; θ ) =

Original: f s ( Q 1 ( τ 1 , u 1 ) , ..., Q n ( τ n , u n )) is the joint Q -values parametrized by θ , ∂f s ∂Q a ⩾ 0 , y ( s, τ , u )
Revised: f s ( Q 1 ( τ 1 , u 1 ) , ..., Q n ( τ n , u n )) is the joint Q -values parametrized by θ , ∂f s ∂Q a ⩾ 0 , y ( s, τ , u )

Original: =
Revised: =

Original: r ( s, u )
Revised: r ( s, u )

Original: [Equation]
Revised: [Equation]

Original: γ max
Revised: γ max

Original: [Equation]
Revised: [Equation]

Original: Q
Revised: Q

Original: [Equation]
Revised: [Equation]

Original: ( s
Revised: ( s

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , u
Revised: , u

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: ) , ˆ u
Revised: ) , ˆ u

Original: [Equation]
Revised: [Equation]

Original: = arg max u Q tot ( s, u ) , and θ ′ denotes parameters θ .
Revised: = arg max u Q tot ( s, u ) , and θ ′ denotes parameters θ .

Original: Then, we introduce a shaping function f
Revised: Then, we introduce a shaping function f

Original: ( s, u ) to project the target Q ( s, τ , u ) into the restricted monotonic space that QMIX can represent.
Revised: ( s, u ) to project the target Q ( s, τ , u ) into the restricted monotonic space that QMIX can represent.

Original: In practice, we replace the target Q ( s, τ , u ) by the minimal value of q a ( s, τ a , u a ) for deterministic and suboptimal joint Q -values.
Revised: In practice, we replace the target Q ( s, τ , u ) by the minimal value of q a ( s, τ a , u a ) for deterministic and suboptimal joint Q -values.

Original: The goal is to ensure that all shaped target value functions at a given state belong to the restricted monotonic space and that the target for the optimal is greater than the suboptimal.
Revised: The goal is to ensure that all shaped target value functions at a given state belong to the restricted monotonic space and that the target for the optimal is greater than the suboptimal.

Original: We do not change the stochastic targets because QMIX can deal with them already.
Revised: We do not change the stochastic targets because QMIX can deal with them already.

Original: To improve learning efficiency, we also remain the target unchanged if it exceeds the current greedy action value estimate and is deterministic.
Revised: To improve learning efficiency, we also remain the target unchanged if it exceeds the current greedy action value estimate and is deterministic.

Original: Q tot ( s, τ , u ; θ ) is trained by minimising the following loss:
Revised: Q tot ( s, τ , u ; θ ) is trained by minimising the following loss:

Original: [Equation]
Revised: [Equation]

Original: Monotonic value factorisation with the original target joint Q -values Q induces representational limitation, but it is unclear if the shaped targets do the same.
Revised: Monotonic value factorisation with the original target joint Q -values Q induces representational limitation, but it is unclear if the shaped targets do the same.

Original: In the following, we give a profound understanding of this.
Revised: In the following, we give a profound understanding of this.

Original: We consider only a fully-observable setting for ease of representation, and our notations do not distinguish the concepts of states and observation-action histories.
Revised: We consider only a fully-observable setting for ease of representation, and our notations do not distinguish the concepts of states and observation-action histories.

Original: Theorem 2 Let Q and Q f be the orignal and the shaped action value based on (7) .
Revised: Theorem 2 Let Q and Q f be the orignal and the shaped action value based on (7) .

Original: Then ∀ s ∈ S and ∀ u ∈ U such that arg max u Q ( s, u ) is unique, arg max
Revised: Then ∀ s ∈ S and ∀ u ∈ U such that arg max u Q ( s, u ) is unique, arg max

Original: u Q f ( s, u ) = arg max u Q ( s, u ) and
Revised: u Q f ( s, u ) = arg max u Q ( s, u ) and

Original: Q f ( s, u ) ∈ Q mvf .
Revised: Q f ( s, u ) ∈ Q mvf .

Original: The proof can be found in Appendix A.3.
Revised: The proof can be found in Appendix A.3.

Original: Since all shaped targets Q f ( s, u ) at a given state s lie in the restricted monotonic space Q mvf , monotonic value factorisation can perfectly represent these values.
Revised: Since all shaped targets Q f ( s, u ) at a given state s lie in the restricted monotonic space Q mvf , monotonic value factorisation can perfectly represent these values.

Original: In addition, the optimal policy remains unchanged during shaping.
Revised: In addition, the optimal policy remains unchanged during shaping.

Original: Therefore, QMIX can now converge to the optimal for any original target Q -value function.
Revised: Therefore, QMIX can now converge to the optimal for any original target Q -value function.

Original: O r i g i n a l t a r g e t v a l u e s
Revised: O r i g i n a l t a r g e t v a l u e s

Original: [Figure]
Revised: [Figure]

Original: In Figure 2, we compare the performance of QMIX with the original and the shaped target joint
Revised: In Figure 2, we compare the performance of QMIX with the original and the shaped target joint

Original: Q -values.
Revised: Q -values.

Original: Consider a 10 × 10 matrix game, where the suboptimal is filled with random numbers generated uniformly between -10 and 9, and the unique optimal value is +10.
Revised: Consider a 10 × 10 matrix game, where the suboptimal is filled with random numbers generated uniformly between -10 and 9, and the unique optimal value is +10.

Original: As shown in Figure 2a-b, the original targets are the rewards from this matrix game, and the shaped targets are obtained based on 7.
Revised: As shown in Figure 2a-b, the original targets are the rewards from this matrix game, and the shaped targets are obtained based on 7.

Original: Figure 2c-d demonstrates that QMIX can perfectly represent all shaped targets and thus converge to the optimal by showing the loss is almost 0 and the test reward is +10 after 400 iterations.
Revised: Figure 2c-d demonstrates that QMIX can perfectly represent all shaped targets and thus converge to the optimal by showing the loss is almost 0 and the test reward is +10 after 400 iterations.

Original: By contrast, QMIX with the original targets in Figure 2a keeps fluctuating during training.
Revised: By contrast, QMIX with the original targets in Figure 2a keeps fluctuating during training.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: In this section, we briefly introduce recent related work on cooperative multi-agent reinforcement learning (MARL) in the paradigm of centralised training with decentralised execution (CTDE).
Revised: In this section, we briefly introduce recent related work on cooperative multi-agent reinforcement learning (MARL) in the paradigm of centralised training with decentralised execution (CTDE).

Original: One of the most significant challenges in CTDE is to ensure the correspondence between the individual
Revised: One of the most significant challenges in CTDE is to ensure the correspondence between the individual

Original: Q -value functions and the joint Q -value function Q tot , i.e., the Individual-Global Max (IGM) principle (Son et al., 2019).
Revised: Q -value functions and the joint Q -value function Q tot , i.e., the Individual-Global Max (IGM) principle (Son et al., 2019).

Original: VDN (Sunehag et al., 2018) and QMIX (Rashid et al., 2018) learn the joint Q -values and factorise them into individual Q -value functions in an additive and a monotonic fashion, respectively.
Revised: VDN (Sunehag et al., 2018) and QMIX (Rashid et al., 2018) learn the joint Q -values and factorise them into individual Q -value functions in an additive and a monotonic fashion, respectively.

Original: Qatten (Yang et al., 2020b) is a variant of QMIX, which applies a multi-head attention structure to the mixing network.
Revised: Qatten (Yang et al., 2020b) is a variant of QMIX, which applies a multi-head attention structure to the mixing network.

Original: QPD (Yang et al., 2020a) utilises integrated gradients to decompose Q tot along trajectory paths.
Revised: QPD (Yang et al., 2020a) utilises integrated gradients to decompose Q tot along trajectory paths.

Original: SMIX( λ ) (Yao et al., 2021) changes the one-step Q -learning target with a SARSA( λ ) target for QMIX.
Revised: SMIX( λ ) (Yao et al., 2021) changes the one-step Q -learning target with a SARSA( λ ) target for QMIX.

Original: RODE (Wang et al., 2020c) decomposes joint action spaces into restricted role action spaces to boost learning efficiency and policy generalisation.
Revised: RODE (Wang et al., 2020c) decomposes joint action spaces into restricted role action spaces to boost learning efficiency and policy generalisation.

Original: These methods apply the same monotonic mixing network and thus can only represent the same class as QMIX.
Revised: These methods apply the same monotonic mixing network and thus can only represent the same class as QMIX.

Original: However, as many previous studies pointed out (Mahajan et al., 2019; Son et al., 2019; Rashid et al., 2020), monotonic value factorisation limits the representational capacity of Q tot , and fail to learn the optimal policy when the target Q -value functions are non-monotonic.
Revised: However, as many previous studies pointed out, monotonic value factorisation limits the representational capacity of Q tot , and fail to learn the optimal policy when the target Q -value functions are non-monotonic (Mahajan et al., 2019; Son et al., 2019; Rashid et al., 2020).

Original: Some recent works try to achieve the full representational capacity of Q tot to solve this problem.
Revised: Some recent works try to achieve the full representational capacity of Q tot to solve this problem.

Original: QPLEX (Wang et al., 2020a) achieves the full representation under the IGM principle theoretically through a duelling mixing network, in which the weights are produced through joint actions.
Revised: QPLEX (Wang et al., 2020a) achieves the full representation under the IGM principle theoretically through a duelling mixing network, in which the weights are produced through joint actions.

Original: However, it also induces the “vanishing gradient” problem, leading to slow convergence and learning instability.
Revised: However, it also induces the “vanishing gradient” problem, leading to slow convergence and learning instability.

Original: More discussions can be found in Appendix B.2.
Revised: More discussions can be found in Appendix B.2.

Original: Deep coordination graph (B¨ohmer et al., 2020) decompose the joint state-action value into individual utilities and payoff contributions based on the actions of the agents connected by the (hyper-)edges.
Revised: Deep coordination graph (B¨ohmer et al., 2020) decompose the joint state-action value into individual utilities and payoff contributions based on the actions of the agents connected by the (hyper-)edges.

Original: Tesseract (Mahajan et al., 2021) decomposes the Q -tensor across agents and utilises low-rank tensor approximations to model agent interactions relevant to the task, and thus learns a compact approximation of the target Q -value function.
Revised: Tesseract (Mahajan et al., 2021) decomposes the Q -tensor across agents and utilises low-rank tensor approximations to model agent interactions relevant to the task, and thus learns a compact approximation of the target Q -value function.

Original: However, since the dimension of the state-action space increases exponentially as the number of agents grows, it is not easy to achieve the full representational capacity in complex MARL tasks.
Revised: However, since the dimension of the state-action space increases exponentially as the number of agents grows, it is not easy to achieve the full representational capacity in complex MARL tasks.

Original: Another solution is to learn a biased Q tot by prioritising the optimal joint action.
Revised: Another solution is to learn a biased Q tot by prioritising the optimal joint action.

Original: QTRAN (Son et al., 2019) uses two soft regularisations to align the greedy action selections between the joint
Revised: QTRAN (Son et al., 2019) uses two soft regularisations to align the greedy action selections between the joint

Original: Q -value and the individual values.
Revised: Q -value and the individual values.

Original: WQMIX (Rashid et al., 2020) introduces a weighting mechanism similar to Hysteretic Q -learning (Matignon et al., 2007), which places more importance on better joint actions.
Revised: WQMIX (Rashid et al., 2020) introduces a weighting mechanism similar to Hysteretic Q -learning (Matignon et al., 2007), which places more importance on better joint actions.

Original: QTRAN can be viewed as a variant of WQMIX, which additionally uses the weightfor the joint Q -value whose target is smaller than the current estimate and the chosen action is not greedy.
Revised: QTRAN can be viewed as a variant of WQMIX, which additionally uses the weightfor the joint Q -value whose target is smaller than the current estimate and the chosen action is not greedy.

Original: QTRAN approximates ˆ Q ∗ as the target instead of the original target y and thus can deal with stochasticity.
Revised: QTRAN approximates ˆ Q ∗ as the target instead of the original target y and thus can deal with stochasticity.

Original: However, due to the 0 weight for overestimated Q-values, QTRAN is empirically hard to scale to more complex tasks (Samvelyan et al., 2019).
Revised: However, due to the 0 weight for overestimated Q-values, QTRAN is empirically hard to scale to more complex tasks (Samvelyan et al., 2019).

Original: We also prove that there may not exist an appropriate weight when the targets are non-monotonic and stochastic.
Revised: We also prove that there may not exist an appropriate weight when the targets are non-monotonic and stochastic.

Original: In addition, there have been many developments in policy-based methods under CTDE settings.
Revised: In addition, there have been many developments in policy-based methods under CTDE settings.

Original: MAPPO (Yu et al., 2021) applies PPO (Schulman et al., 2017) into MARL and shows strong empirical performance.
Revised: MAPPO (Yu et al., 2021) applies PPO (Schulman et al., 2017) into MARL and shows strong empirical performance.

Original: However, Kuba et al.
Revised: However, Kuba et al.

Original: (2021) points out MAPPO suffers from instability arising from the non-stationarity induced by simultaneously learning and exploring agents.
Revised: (2021) points out MAPPO suffers from instability arising from the non-stationarity induced by simultaneously learning and exploring agents.

Original: Therefore, they introduce the multi-agent advantage decomposition lemma and the sequential policy update scheme to achieve monotonic improvement on the joint policy, which can converge to one of the Nash
Revised: Therefore, they introduce the multi-agent advantage decomposition lemma and the sequential policy update scheme to achieve monotonic improvement on the joint policy, which can converge to one of the Nash

Original: Equilibriums.
Revised: Equilibriums.

Original: However, the monotonic target in value decomposition methods can be interpreted as a static multivariable optimisation problem, where there are non-optimal solutions that cannot be improved upon by coordinate descent.
Revised: However, the monotonic target in value decomposition methods can be interpreted as a static multivariable optimisation problem, where there are non-optimal solutions that cannot be improved upon by coordinate descent.

Original: The sequential policy update scheme cannot guarantee convergence to the optimal in this optimisation problem (Bertsekas, 2019).
Revised: The sequential policy update scheme cannot guarantee convergence to the optimal in this optimisation problem (Bertsekas, 2019).

Original: Relationship To Reward Shaping.
Revised: Relationship To Reward Shaping.

Original: Reward shaping is a common technique for improving singleagent learners’ performance by integrating expert knowledge into MDP (Gullapalli & Barto, 1992).
Revised: Reward shaping is a common technique for improving singleagent learners’ performance by integrating expert knowledge into MDP (Gullapalli & Barto, 1992).

Original: It also has been adopted to the MARL setting in recent work (Devlin et al., 2011; Xiao et al., 2021).
Revised: It also has been adopted to the MARL setting in recent work (Devlin et al., 2011; Xiao et al., 2021).

Original: Since it is not always possible to find an expert with complete domain knowledge, it is not easy to set rewards for complex MARL tasks in advance.
Revised: Since it is not always possible to find an expert with complete domain knowledge, it is not easy to set rewards for complex MARL tasks in advance.

Original: Therefore, adaptively shaping the reward is a more attractive way.
Revised: Therefore, adaptively shaping the reward is a more attractive way.

Original: Some reward randomisation methods (Gupta et al., 2021;
Revised: Some reward randomisation methods (Gupta et al., 2021;

Original: Tang et al., 2021) are proposed to convert the original MDP to a new MDP with random rewards through universal successor features.
Revised: Tang et al., 2021) are proposed to convert the original MDP to a new MDP with random rewards through universal successor features.

Original: However, these methods require a massive number of randomisation to find a desirable reward function because they cannot guarantee policy invariance .
Revised: However, these methods require a massive number of randomisation to find a desirable reward function because they cannot guarantee policy invariance .

Original: GVR (Wan et al., 2022) proposes inferior target reshaping and superior experience replay to eliminate the non-optimal self-transition nodes, which is similar to WQMIX and ignores the stochasticity of the transition.
Revised: GVR (Wan et al., 2022) proposes inferior target reshaping and superior experience replay to eliminate the non-optimal self-transition nodes, which is similar to WQMIX and ignores the stochasticity of the transition.

Original: Compared with these methods, UTS is more efficient because it can perfectly represent the expected value for stochastic targets and guarantee policy invariance during target shaping.
Revised: Compared with these methods, UTS is more efficient because it can perfectly represent the expected value for stochastic targets and guarantee policy invariance during target shaping.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we conduct empirical experiments to answer the following questions: (1) Is UTS better than the existing methods when the target is seriously non-monotonic?
Revised: In this section, we conduct empirical experiments to answer the following questions: (1) Is UTS better than the existing methods when the target is seriously non-monotonic?

Original: (2) Can UTS perform well when the target is non-monotonic and stochastic?
Revised: (2) Can UTS perform well when the target is non-monotonic and stochastic?

Original: (3) Can UTS perform efficient coordination in challenging multi-agent tasks?
Revised: (3) Can UTS perform efficient coordination in challenging multi-agent tasks?

Original: We compare UTS with value decomposition baselines, including
Revised: We compare UTS with value decomposition baselines, including

Original: VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2018), QTRAN (Son et al., 2019), QPLEX (Wang et al., 2020a) and WQMIX (Rashid et al., 2020) on a matrix game, a multi-agent Markov Decision Process, predator-prey (Son et al., 2019), predator-stag-hare, and StarCraft II Multi-agent
Revised: VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2018), QTRAN (Son et al., 2019), QPLEX (Wang et al., 2020a) and WQMIX (Rashid et al., 2020) on a matrix game, a multi-agent Markov Decision Process, predator-prey (Son et al., 2019), predator-stag-hare, and StarCraft II Multi-agent

Original: Challenge (Samvelyan et al., 2019).
Revised: Challenge (Samvelyan et al., 2019).

Original: We also conduct ablation studies (Appendix C) and compare
Revised: We also conduct ablation studies (Appendix C) and compare

Original: UTS with GVR (Wan et al., 2022) and WQMIX with different weights (Appendix D), as well as more CTDE-based algorithms (Appendix E).
Revised: UTS with GVR (Wan et al., 2022) and WQMIX with different weights (Appendix D), as well as more CTDE-based algorithms (Appendix E).

Original: More detailed experimental settings are included in
Revised: More detailed experimental settings are included in

Original: Appendix F.
Revised: Appendix F.

Original: 6. M ATRIX GAMES
Revised: 6. M ATRIX GAMES

Original: In this subsection, we consider a matrix game in Figure 3a, where the local optimal is difficult to jump out due to the considerable miscoordination penalties and stochastic reward.
Revised: In this subsection, we consider a matrix game in Figure 3a, where the local optimal is difficult to jump out due to the considerable miscoordination penalties and stochastic reward.

Original: The optimal joint strategy is to perform ( u 11 , u 12 ) simultaneously.
Revised: The optimal joint strategy is to perform ( u 11 , u 12 ) simultaneously.

Original: We adopt a uniform exploration strategy to approximate the uniform data distribution and eliminate the challenge of exploration and sample complexity.
Revised: We adopt a uniform exploration strategy to approximate the uniform data distribution and eliminate the challenge of exploration and sample complexity.

Original: As shown in Figure 3a, UTS and QTRAN can achieve optimal performance, while other algorithms fall into a local optimum.
Revised: As shown in Figure 3a, UTS and QTRAN can achieve optimal performance, while other algorithms fall into a local optimum.

Original: CW-QMIX fluctuates dramatically and cannot converge to any policy, which is consistent with our analysis in Section 3.2.
Revised: CW-QMIX fluctuates dramatically and cannot converge to any policy, which is consistent with our analysis in Section 3.2.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: with stochastic state transition.
Revised: with stochastic state transition.

Original: 𝑠 1 𝑠-12-126 6 - -  𝑢 11 𝑈 1 𝑈 2 𝑢𝑢𝑢𝑢𝑢-12-126 6 - -  𝑢 11 𝑈 1 𝑈 2 𝑢𝑢𝑢𝑢𝑢
Revised: 𝑠 1 𝑠-12-126 6 - -  𝑢 11 𝑈 1 𝑈 2 𝑢𝑢𝑢𝑢𝑢-12-126 6 - -  𝑢 11 𝑈 1 𝑈 2 𝑢𝑢𝑢𝑢𝑢

Original: The payoff matrix at .
Revised: The payoff matrix at .

Original: 𝑠 

Revised: 𝑠

Original: The payoff matrix at .
Revised: The payoff matrix at .

Original: 𝑠-12-126 6 - -  𝑢 11 𝑈 1 𝑈 2 𝑢𝑢𝑢𝑢𝑢
Revised: 𝑠-12-126 6 - -  𝑢 11 𝑈 1 𝑈 2 𝑢𝑢𝑢𝑢𝑢

Original: The payoff matrix at .
Revised: The payoff matrix at .

Original: 𝑠
Revised: 𝑠

Original: We also consider a multi-agent Markov Decision Process (MMDP) with non-monotonic and stochastic targets in Figure 3b.
Revised: We also consider a multi-agent Markov Decision Process (MMDP) with non-monotonic and stochastic targets in Figure 3b.

Original: This MMDP involves two agents, three actions and a team reward.
Revised: This MMDP involves two agents, three actions and a team reward.

Original: Two agents start at state s 1 and explore extrinsic rewards for ten environment steps.
Revised: Two agents start at state s 1 and explore extrinsic rewards for ten environment steps.

Original: The state transits from s 1 to s 2 with probability 50% and s 3 with 50% if agents perform u = ( u 21 , u 22 ) at s 1 , and remains unchanged otherwise.
Revised: The state transits from s 1 to s 2 with probability 50% and s 3 with 50% if agents perform u = ( u 21 , u 22 ) at s 1 , and remains unchanged otherwise.

Original: The team continually obtains a deterministic reward of +10 and -10 at s 2 and s 3 , respectively.
Revised: The team continually obtains a deterministic reward of +10 and -10 at s 2 and s 3 , respectively.

Original: In this task, since VDN, QMIX, and QPLEX estimates the stochastic targets well but cannot deal with the non-monotonicity, they converge to the suboptimal with the return of +60.
Revised: In this task, since VDN, QMIX, and QPLEX estimates the stochastic targets well but cannot deal with the non-monotonicity, they converge to the suboptimal with the return of +60.

Original: The negative results for CW-QMIX and OW-QMIX show that they converge to the stochastic suboptimal due to the undesirable weights.
Revised: The negative results for CW-QMIX and OW-QMIX show that they converge to the stochastic suboptimal due to the undesirable weights.

Original: By contrast, UTS and QTRAN can achieve optimal performance.
Revised: By contrast, UTS and QTRAN can achieve optimal performance.

Original: 6. PREDATOR - PREY
Revised: 6. PREDATOR - PREY

Original: Predator-prey (PP) is a partially observable environment containing eight predators (agents) and eight prey in a 10 × 10 grid world.
Revised: Predator-prey (PP) is a partially observable environment containing eight predators (agents) and eight prey in a 10 × 10 grid world.

Original: Each agent observes a 5 × 5 sub-grid around it and can perform five actions, i.e., up, down, left, right, and catch.
Revised: Each agent observes a 5 × 5 sub-grid around it and can perform five actions, i.e., up, down, left, right, and catch.

Original: The team receives a miscoordination penalty when two agents surround a prey and only one tries to catch it.
Revised: The team receives a miscoordination penalty when two agents surround a prey and only one tries to catch it.

Original: By contrast, they earn a bonus of 10 if they catch the prey simultaneously.
Revised: By contrast, they earn a bonus of 10 if they catch the prey simultaneously.

Original: After a successful catch, the catching predators and prey will be removed from the grid.
Revised: After a successful catch, the catching predators and prey will be removed from the grid.

Original: Figure 4a-b shows that VDN, QMIX, and QPLEX fail to return positive returns with a significant penalty because the penalty makes these methods underestimate the per-agent action value of the real optimal.
Revised: Figure 4a-b shows that VDN, QMIX, and QPLEX fail to return positive returns with a significant penalty because the penalty makes these methods underestimate the per-agent action value of the real optimal.

Original: CW-QMIX and OW-QMIX can achieve positive test returns when the penalty is -2 because they place more weight on better joint actions.
Revised: CW-QMIX and OW-QMIX can achieve positive test returns when the penalty is -2 because they place more weight on better joint actions.

Original: QTRAN can also solve this task through its regularisation terms.
Revised: QTRAN can also solve this task through its regularisation terms.

Original: However, CW-QMIX, OW-QMIX, and QTRAN fail to solve the task when the penalty is -10, suggesting difficulties in choosing the pre-defined weight and the network architecture.
Revised: However, CW-QMIX, OW-QMIX, and QTRAN fail to solve the task when the penalty is -10, suggesting difficulties in choosing the pre-defined weight and the network architecture.

Original: By contrast, UTS solves tasks quickly and steadily in all penalty settings.
Revised: By contrast, UTS solves tasks quickly and steadily in all penalty settings.

Original: We extend predator-prey to a stochastic version named predator-stag-hare (PSH), which contains eight predators, four stags and four hares.
Revised: We extend predator-prey to a stochastic version named predator-stag-hare (PSH), which contains eight predators, four stags and four hares.

Original: The agents receive the same miscoordination penalty in predator-prey when they try to catch a stag.
Revised: The agents receive the same miscoordination penalty in predator-prey when they try to catch a stag.

Original: By contrast, a catch of hare results in a non-deterministic reward, i.e., the probabilities of receiving SR ∈ R + and -SR are both 50%.
Revised: By contrast, a catch of hare results in a non-deterministic reward, i.e., the probabilities of receiving SR ∈ R + and -SR are both 50%.

Original: As shown in Figure 4c-d,
Revised: As shown in Figure 4c-d,

Original: VDN and QMIX can quickly recover the optimal policy when the penalty is 0 because they can accurately evaluate the stochastic reward.
Revised: VDN and QMIX can quickly recover the optimal policy when the penalty is 0 because they can accurately evaluate the stochastic reward.

Original: When the penalty is -2, VDN, QMIX, QPLEX, and
Revised: When the penalty is -2, VDN, QMIX, QPLEX, and

Original: WQMIX fail to solve this task.
Revised: WQMIX fail to solve this task.

Original: QTRAN also struggles with the suboptimal in this setting.
Revised: QTRAN also struggles with the suboptimal in this setting.

Original: By contrast,
Revised: By contrast,

Original: UTS takes advantage of the uncertainty estimation module to avoid overestimating the stochastic action values, and uses the shaping function to eliminate the adverse impact of the miscoordination penalty.
Revised: UTS takes advantage of the uncertainty estimation module to avoid overestimating the stochastic action values, and uses the shaping function to eliminate the adverse impact of the miscoordination penalty.

Original: Therefore, it learns quickly and reliably in these tasks.
Revised: Therefore, it learns quickly and reliably in these tasks.

Original: The detailed comparison between
Revised: The detailed comparison between

Original: UTS and WQMIX with different weights can be found in Appendix D.
Revised: UTS and WQMIX with different weights can be found in Appendix D.

Original: [Figure]
Revised: [Figure]

Original: 6. S TAR C RAFT M ULTI -A GENT C HALLENGE
Revised: 6. S TAR C RAFT M ULTI -A GENT C HALLENGE

Original: We also compare UTS with baselines on eight hard and super hard maps in the decentralised micromanagement tasks in StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019).
Revised: We also compare UTS with baselines on eight hard and super hard maps in the decentralised micromanagement tasks in StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019).

Original: Figure 5 shows the improved performance of UTS, which indicates that the target shaping function can be applied to more general benchmarks
Revised: Figure 5 shows the improved performance of UTS, which indicates that the target shaping function can be applied to more general benchmarks

Original: [Figure]
Revised: [Figure]

Original: C ONCLUSION AND F UTURE W ORK
Revised: C ONCLUSION AND F UTURE W ORK

Original: This paper presents uncertainty-based target shaping (UTS), which was inspired by analysing the limitation of Weighted QMIX.
Revised: This paper presents uncertainty-based target shaping (UTS), which was inspired by analysing the limitation of Weighted QMIX.

Original: We prove that there may not exist an appropriate weighting function for
Revised: We prove that there may not exist an appropriate weighting function for

Original: Weighted QMIX to recover the optimal policy when the targets are stochastic and non-monotonic.
Revised: Weighted QMIX to recover the optimal policy when the targets are stochastic and non-monotonic.

Original: To solve this problem, UTS first identifies whether a target value is stochastic by predicting the standard deviation of the reward and the prediction error between the predicted and the target embedding of the following state.
Revised: To solve this problem, UTS first identifies whether a target value is stochastic by predicting the standard deviation of the reward and the prediction error between the predicted and the target embedding of the following state.

Original: Then we propose a shaping function to project original target values into the monotonic space that QMIX can represent by replacing the deterministic suboptimal target with the minimal best per-agent values.
Revised: Then we propose a shaping function to project original target values into the monotonic space that QMIX can represent by replacing the deterministic suboptimal target with the minimal best per-agent values.

Original: Empirical results demonstrate the improved ability of UTS to deal with the task with non-monotonic and stochastic targets, as well as more complicated tasks.
Revised: Empirical results demonstrate the improved ability of UTS to deal with the task with non-monotonic and stochastic targets, as well as more complicated tasks.

Original: The assumption of “unique optimal policy” is required in our proofs.
Revised: The assumption of “unique optimal policy” is required in our proofs.

Original: In the environment with multiple optimal, the more efficient exploration method could be considered, as opposed to the simplistic ϵ -greedy exploration scheme we used in this paper.
Revised: In the environment with multiple optimal, the more efficient exploration method could be considered, as opposed to the simplistic ϵ -greedy exploration scheme we used in this paper.

Original: A O MITTED P ROOFS A. WVDN AND W EIGHTED L INEAR R EGRESSION P ROBLEM
Revised: A O MITTED P ROOFS A. WVDN AND W EIGHTED L INEAR R EGRESSION P ROBLEM

Original: WVDN operator is defined as
Revised: WVDN operator is defined as

Original: [Equation]
Revised: [Equation]

Original: where Q lvf :=
Revised: where Q lvf :=

Original: { Q tot | Q tot ( s, u ) =
Revised: { Q tot | Q tot ( s, u ) =

Original: (cid:80) na =1 Q a ( s, u a ) , Q a
Revised: (cid:80) na =1 Q a ( s, u a ) , Q a

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: u ) ∈ R } , w ow ( s, u ) is the optimistic weighting:
Revised: u ) ∈ R } , w ow ( s, u ) is the optimistic weighting:

Original: [Equation]
Revised: [Equation]

Original: The optimising process of WVDN can be constructed as a weighted linear regression problem:
Revised: The optimising process of WVDN can be constructed as a weighted linear regression problem:

Original: [Equation]
Revised: [Equation]

Original: where A ∈ R m n × mn , x ∈ R mn , w , p , b ∈ R m n , m , n ∈ Z + .
Revised: where A ∈ R m n × mn , x ∈ R mn , w , p , b ∈ R m n , m , n ∈ Z + .

Original: In particular, m dentoes the action space size for each agent, n denotes the number of agents.
Revised: In particular, m dentoes the action space size for each agent, n denotes the number of agents.

Original: Denote A w = √
Revised: Denote A w = √

Original: [Equation]
Revised: [Equation]

Original: p ⊤ · A , b w = √ w ⊤ · (cid:112) p ⊤ · b .
Revised: p ⊤ · A , b w = √ w ⊤ · (cid:112) p ⊤ · b .

Original: The optimal solution is:
Revised: The optimal solution is:

Original: [Equation]
Revised: [Equation]

Original: where A w, † = V S † U ⊤ is the pseudo-inverse of A w according to the Singular Value Decomposition.
Revised: where A w, † = V S † U ⊤ is the pseudo-inverse of A w according to the Singular Value Decomposition.

Original: U ∈ R m n × m n and V ∈ R mn × mn are real orthogonal matrices, S † ∈ R m n ×
Revised: U ∈ R m n × m n and V ∈ R mn × mn are real orthogonal matrices, S † ∈ R m n ×

Original: mn is a diagonal matric with non-negative real numbers on the diagonal.
Revised: mn is a diagonal matric with non-negative real numbers on the diagonal.

Original: The proof is trivial.
Revised: The proof is trivial.

Original: Consider the full exploration scheme (i.e., ϵ -greedy exploration with ϵ = 1 ), and the non-monotonic game in Figure 6a, where a = 8 , b = − 12 , c = 6 .
Revised: Consider the full exploration scheme (i.e., ϵ -greedy exploration with ϵ = 1 ), and the non-monotonic game in Figure 6a, where a = 8 , b = − 12 , c = 6 .

Original: For VDN (a special case of WVDN where α = 1 ), we have
Revised: For VDN (a special case of WVDN where α = 1 ), we have

Original: [Equation]
Revised: [Equation]

Original:  −
Revised:  −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: According to (12), the optimal solution is
Revised: According to (12), the optimal solution is

Original: [Equation]
Revised: [Equation]

Original: Clearly, Q i ( u 1 i ) < max { Q i ( u 2 i ) , Q 1 ( u 3 i ) } , ∀ i ∈ { 1 , 2 } , which indicates that VDN fails to converge to the optimal.
Revised: Clearly, Q i ( u 1 i ) < max { Q i ( u 2 i ) , Q 1 ( u 3 i ) } , ∀ i ∈ { 1 , 2 } , which indicates that VDN fails to converge to the optimal.

Original: A. O MITTED P ROOFS IN S ECTION
Revised: A. O MITTED P ROOFS IN S ECTION

Original: WVDN operator is defined in Section A.1.
Revised: WVDN operator is defined in Section A.1.

Original: Denote the joint action u
Revised: Denote the joint action u

Original: = [ u 1 , ..., u a , ..., u n ] , where n is the number of agents.
Revised: = [ u 1 , ..., u a , ..., u n ] , where n is the number of agents.

Original: Denote u ia as the i -th action of u a .
Revised: Denote u ia as the i -th action of u a .

Original: Consider matrix games in Figure 6.
Revised: Consider matrix games in Figure 6.

Original: Suppose the training dataset is fixed and the data has a ϵ -greedy distribution, where the greedy action is ( u 21 , u 22 ) and ϵ ∈ (0 , 1] .
Revised: Suppose the training dataset is fixed and the data has a ϵ -greedy distribution, where the greedy action is ( u 21 , u 22 ) and ϵ ∈ (0 , 1] .

Original: Based on the ϵ -greedy distribution, the probabilities of each joint action p ( u i 1 , u j 2 ) is:
Revised: Based on the ϵ -greedy distribution, the probabilities of each joint action p ( u i 1 , u j 2 ) is:

Original: [Equation]
Revised: [Equation]

Original: ) , (3 , 1) , (3 , 3) } , (3 − 2 ϵ ) , ( i, j ) =
Revised: ) , (3 , 1) , (3 , 3) } , (3 − 2 ϵ ) , ( i, j ) =

Original: (2 , 2) , (3 − 2 ϵ ) ϵ , otherwise .
Revised: (2 , 2) , (3 − 2 ϵ ) ϵ , otherwise .

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: 𝑢 11 𝑈 1 𝑈𝑢 2𝑢𝑢𝑢𝑢𝑢 11 𝑈 1 𝑈𝑢 2𝑢𝑢𝑢𝑢𝑢 11 𝑈 1 𝑈𝑢 2𝑢𝑢𝑢𝑢𝑢 11 𝑈 1 𝑈𝑢 2𝑢𝑢𝑢𝑢𝑢 11 𝑈 1 𝑈𝑢𝑢𝑢𝑢𝑢𝑢 11 𝑈 1 𝑈𝑢𝑢𝑢𝑢𝑢𝑢 11 𝑈 1 𝑈𝑢 2𝑢𝑢𝑢𝑢𝑢 11 𝑈 1 𝑈𝑢 2𝑢𝑢𝑢𝑢 is
Revised: 𝑢 11 𝑈 1 𝑈𝑢 2𝑢𝑢𝑢𝑢𝑢 11 𝑈 1 𝑈𝑢 2𝑢𝑢𝑢𝑢𝑢 11 𝑈 1 𝑈𝑢 2𝑢𝑢𝑢𝑢𝑢 11 𝑈 1 𝑈𝑢 2𝑢𝑢𝑢𝑢𝑢 11 𝑈 1 𝑈𝑢𝑢𝑢𝑢𝑢𝑢 11 𝑈 1 𝑈𝑢𝑢𝑢𝑢𝑢𝑢 11 𝑈 1 𝑈𝑢 2𝑢𝑢𝑢𝑢𝑢 11 𝑈 1 𝑈𝑢 2𝑢𝑢𝑢𝑢 is

Original: monotonic
Revised: monotonic

Original: non-monotonic
Revised: non-monotonic

Original: To recover the optimal policy, the weight α for WVDN is bounded by ϵ and the reward space.
Revised: To recover the optimal policy, the weight α for WVDN is bounded by ϵ and the reward space.

Original: We start by analysing the matrix game in Figure 6a.
Revised: We start by analysing the matrix game in Figure 6a.

Original: Since the matrix game only involves one state, we omit s for the input of value functions in the following proofs.
Revised: Since the matrix game only involves one state, we omit s for the input of value functions in the following proofs.

Original: Proposition 1 For WVDN, the weight α should be smaller than 3 ϵ ( a − c )+ ϵ 2 ( c − b ) ( ϵ − 3) 2 ( c − b ) to recover the optimal joint policy on the non-monotonic matrix game in Figure 6a.
Revised: Proposition 1 For WVDN, the weight α should be smaller than 3 ϵ ( a − c )+ ϵ 2 ( c − b ) ( ϵ − 3) 2 ( c − b ) to recover the optimal joint policy on the non-monotonic matrix game in Figure 6a.

Original: Proof.
Revised: Proof.

Original: First compute the gradient for Q ( u 11 ) , Q
Revised: First compute the gradient for Q ( u 11 ) , Q

Original: [Equation]
Revised: [Equation]

Original: ) , Q
Revised: ) , Q

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Let ▽ Q ( u 11 ) =
Revised: Let ▽ Q ( u 11 ) =

Original: ▽ Q ( u 21 ) =
Revised: ▽ Q ( u 21 ) =

Original: ▽ Q ( u 31 ) = 0 :
Revised: ▽ Q ( u 31 ) = 0 :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: To recover the optimal joint policy, the following conditions should be satisfied:
Revised: To recover the optimal joint policy, the following conditions should be satisfied:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Denote t =
Revised: Denote t =

Original: ϵ
Revised: ϵ

Original: + 3 α − ϵα .
Revised: + 3 α − ϵα .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Let Q ( u 11 ) − max { Q ( u 21 ) , Q ( u 31 ) } > 0 and Q ( u 12 ) − max { Q ( u 22 ) , Q ( u 32 ) } > 0 .
Revised: Let Q ( u 11 ) − max { Q ( u 21 ) , Q ( u 31 ) } > 0 and Q ( u 12 ) − max { Q ( u 22 ) , Q ( u 32 ) } > 0 .

Original: Then α < 3 ϵ ( a − c )+ ϵ 2 ( c − b ) ( ϵ − 3) 2 ( c − b ) .
Revised: Then α < 3 ϵ ( a − c )+ ϵ 2 ( c − b ) ( ϵ − 3) 2 ( c − b ) .

Original: □
Revised: □

Original: Proposition 2 For WVDN, the weight α should be large enough to recover the optimal joint policy for the stochastic matrix game in Figure 6b.
Revised: Proposition 2 For WVDN, the weight α should be large enough to recover the optimal joint policy for the stochastic matrix game in Figure 6b.

Original: Theorem 1 Let Π WVDN Q :=
Revised: Theorem 1 Let Π WVDN Q :=

Original: arg min Q tot ∈Q lvf (cid:80) u ∈
Revised: arg min Q tot ∈Q lvf (cid:80) u ∈

Original: U w ow ( s, u )( Q ( s, u ) − Q
Revised: U w ow ( s, u )( Q ( s, u ) − Q

Original: tot ( s, u )) 2 and w ow ( s, u ) is the optimistic weighting from (4).
Revised: tot ( s, u )) 2 and w ow ( s, u ) is the optimistic weighting from (4).

Original: Then ∃ Q such that arg max Π WVDN Q ̸ = arg max Q for any α ∈ (0 , 1] .
Revised: Then ∃ Q such that arg max Π WVDN Q ̸ = arg max Q for any α ∈ (0 , 1] .

Original: Proof.
Revised: Proof.

Original: Since the proof of Proposition 2 and Theorem 1 contains a significant overlap, we will merge them both into a single proof.
Revised: Since the proof of Proposition 2 and Theorem 1 contains a significant overlap, we will merge them both into a single proof.

Original: First compute the gradient for Q ( u 11 ) , Q ( u 21 ) , Q ( u 31 ) :
Revised: First compute the gradient for Q ( u 11 ) , Q ( u 21 ) , Q ( u 31 ) :

Original: [Equation]
Revised: [Equation]

Original: To recover the optimal joint policy
Revised: To recover the optimal joint policy

Original: [Equation]
Revised: [Equation]

Original: the following requirements should be satisfied:
Revised: the following requirements should be satisfied:

Original: [Equation]
Revised: [Equation]

Original: Let ▽ Q ( u 11 ) =
Revised: Let ▽ Q ( u 11 ) =

Original: ▽ Q ( u 21 ) =
Revised: ▽ Q ( u 21 ) =

Original: ▽ Q ( u 31 ) = 0 .
Revised: ▽ Q ( u 31 ) = 0 .

Original: Denote t =
Revised: Denote t =

Original: ϵ
Revised: ϵ

Original: + 3 α − ϵα , m =
Revised: + 3 α − ϵα , m =

Original: (3 − 2 ϵ )(1 − α ) p + 3 α .
Revised: (3 − 2 ϵ )(1 − α ) p + 3 α .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: For the matrix game in Figure 6b, consider the following situations:
Revised: For the matrix game in Figure 6b, consider the following situations:

Original: (1) The target is monotonic but stochastic.
Revised: (1) The target is monotonic but stochastic.

Original: For example, in Figure 6c, a = 8 , b = e = 6 , c
Revised: For example, in Figure 6c, a = 8 , b = e = 6 , c

Original: = 12 , d = 0 , and p = 0 .
Revised: = 12 , d = 0 , and p = 0 .

Original: To recover the optimal joint action, the weight α should be greater than (0 . 434 , 0 . 667 , 0 . 827 , 0 . 932)
Revised: To recover the optimal joint action, the weight α should be greater than (0 . 434 , 0 . 667 , 0 . 827 , 0 . 932)

Original: when ϵ = (1 , 0 .
Revised: when ϵ = (1 , 0 .

Original: 5 , 0 . 25) , respectively.
Revised: 5 , 0 . 25) , respectively.

Original: The target is non-monotonic and stochastic.
Revised: The target is non-monotonic and stochastic.

Original: For example, in Figure 6d, a = 8 , b = − 12 , c = 12 , d = 0 , e = 6 , and p = 0 . 5 .
Revised: For example, in Figure 6d, a = 8 , b = − 12 , c = 12 , d = 0 , e = 6 , and p = 0 . 5 .

Original: When ϵ = (1 , 0 .
Revised: When ϵ = (1 , 0 .

Original: 5 , 0 . 25) , there does not exist the weight α ∈ (0 , 1] making Q
Revised: 5 , 0 . 25) , there does not exist the weight α ∈ (0 , 1] making Q

Original: ( u 11 ) − max
Revised: ( u 11 ) − max

Original: { Q
Revised: { Q

Original: ( u
Revised: ( u

Original: Q
Revised: Q

Original: ( u 31 )
Revised: ( u 31 )

Original: } > 0 , i.e., WVDN cannot converge to the optimal.
Revised: } > 0 , i.e., WVDN cannot converge to the optimal.

Original: Thus, ∃ Q such that arg max Π WVDN Q ̸ = arg max Q for any α ∈ (0 , 1] .
Revised: Thus, ∃ Q such that arg max Π WVDN Q ̸ = arg max Q for any α ∈ (0 , 1] .

Original: □
Revised: □

Original: More experimental results can be found in Appendix B.1.
Revised: More experimental results can be found in Appendix B.1.

Original: A. O MITTED P ROOFS OF R ESHAPING FUNCTION
Revised: A. O MITTED P ROOFS OF R ESHAPING FUNCTION

Original: Theorem 2 Let Q and Q f be the orignal and the shaped action value based on (7) .
Revised: Theorem 2 Let Q and Q f be the orignal and the shaped action value based on (7) .

Original: Then ∀ s ∈ S and ∀ u ∈ U such that arg max u Q ( s, u ) is unique, arg max
Revised: Then ∀ s ∈ S and ∀ u ∈ U such that arg max u Q ( s, u ) is unique, arg max

Original: u Q f ( s, u ) = arg max u Q ( s, u ) and
Revised: u Q f ( s, u ) = arg max u Q ( s, u ) and

Original: Q f ( s, u ) ∈ Q mvf .
Revised: Q f ( s, u ) ∈ Q mvf .

Original: Proof.
Revised: Proof.

Original: Considering only a fully-observable setting for ease of representation.
Revised: Considering only a fully-observable setting for ease of representation.

Original: Thus our notations do not distinguish the concepts of states and observation-action histories.
Revised: Thus our notations do not distinguish the concepts of states and observation-action histories.

Original: In addition, when more than one optimal policy exists, most Q -learning algorithms fail to converge to a stable point (Simchowitz & Jamieson, 2019; Wang et al., 2020a; Rashid et al., 2020).
Revised: In addition, when more than one optimal policy exists, most Q -learning algorithms fail to converge to a stable point (Simchowitz & Jamieson, 2019; Wang et al., 2020a; Rashid et al., 2020).

Original: Thus, consider a s ∈ S , we assume u ∗ Q = arg max u Q ( s, u ) is unique.
Revised: Thus, consider a s ∈ S , we assume u ∗ Q = arg max u Q ( s, u ) is unique.

Original: Assume max u − a Q ( s, u a , u − a ) for all u a are perfectly represented.
Revised: Assume max u − a Q ( s, u a , u − a ) for all u a are perfectly represented.

Original: Then Q ( s, u ) ⩽ min a ∈ A { max u − a Q ( s, u a , u − a ) } .
Revised: Then Q ( s, u ) ⩽ min a ∈ A { max u − a Q ( s, u a , u − a ) } .

Original: By the shaping function from (7):
Revised: By the shaping function from (7):

Original: [Equation]
Revised: [Equation]

Original: Under the assumption of the unique optimal action, we have Q f ( s, u ∗ Q ) >
Revised: Under the assumption of the unique optimal action, we have Q f ( s, u ∗ Q ) >

Original: Q f ( s, u ) for all u ̸ = u ∗ Q .
Revised: Q f ( s, u ) for all u ̸ = u ∗ Q .

Original: Therefore, arg max u Q f ( s, u ) =
Revised: Therefore, arg max u Q f ( s, u ) =

Original: arg max u Q ( s, u ) .
Revised: arg max u Q ( s, u ) .

Original: Suppose ∃ f s ( s, Q 1 ( s, u 1 ) , ..., Q n ( s, u n ))
Revised: Suppose ∃ f s ( s, Q 1 ( s, u 1 ) , ..., Q n ( s, u n ))

Original: =
Revised: =

Original: Q f ( s, u ) , ∀ u
Revised: Q f ( s, u ) , ∀ u

Original: = ( u 1 , ..., u n ) ∈ U .
Revised: = ( u 1 , ..., u n ) ∈ U .

Original: Then  
Revised: Then  

Original: [Equation]
Revised: [Equation]

Original: a ) , Q − a ( u − a ))
Revised: a ) , Q − a ( u − a ))

Original: =
Revised: =

Original: min { max u − a Q ( s, u ′ a , u − a ) , max u − i Q ( s, u i , u − i ) } , ∀ i ∈ − a , (29) where Q − a ( u − a ) =
Revised: min { max u − a Q ( s, u ′ a , u − a ) , max u − i Q ( s, u i , u − i ) } , ∀ i ∈ − a , (29) where Q − a ( u − a ) =

Original: { Q i ( s, u i ) } i ∈− a .
Revised: { Q i ( s, u i ) } i ∈− a .

Original: Let Q a ( s, u a ) ⩾ Q a ( s, u ′ a ) if max u − a Q ( s, u a , u − a ) ⩾ max u − a Q ( s, u ′ a , u − a ) , ∀ a ∈ A .
Revised: Let Q a ( s, u a ) ⩾ Q a ( s, u ′ a ) if max u − a Q ( s, u a , u − a ) ⩾ max u − a Q ( s, u ′ a , u − a ) , ∀ a ∈ A .

Original: Then, according to (29), f s ( s, Q a ( u a ) , Q − a ( u − a )) ⩾ f s ( s, Q a ( u ′ a ) , Q − a ( u − a )) .
Revised: Then, according to (29), f s ( s, Q a ( u a ) , Q − a ( u − a )) ⩾ f s ( s, Q a ( u ′ a ) , Q − a ( u − a )) .

Original: Thus we can find f s ( s, Q 1 ( s, u 1 ) , ..., Q n ( s, u n )) to represent Q f ( s, u ) , ∀ u
Revised: Thus we can find f s ( s, Q 1 ( s, u 1 ) , ..., Q n ( s, u n )) to represent Q f ( s, u ) , ∀ u

Original: = ( u 1 , ..., u n ) ∈ U , where ∂f s ∂Q a ⩾ 0 .
Revised: = ( u 1 , ..., u n ) ∈ U , where ∂f s ∂Q a ⩾ 0 .

Original: And so Q f ( s,
Revised: And so Q f ( s,

Original: u ) ∈
Revised: u ) ∈

Original: Q mvf , ∀ s ∈ S, ∀ u ∈ U .
Revised: Q mvf , ∀ s ∈ S, ∀ u ∈ U .

Original: □
Revised: □

Original: Now we talk about the case of multiple optimal.
Revised: Now we talk about the case of multiple optimal.

Original: When more than one optimal policy exists in the
Revised: When more than one optimal policy exists in the

Original: Markov Decision Process, many value-based algorithms fail to converge on a stable point (Simchowitz & Jamieson, 2019).
Revised: Markov Decision Process, many value-based algorithms fail to converge on a stable point (Simchowitz & Jamieson, 2019).

Original: However, multiple optimal may be very common in a multi-agent setting.
Revised: However, multiple optimal may be very common in a multi-agent setting.

Original: For example, when two agents arrive at a crossroads, the optimal solution is that one of them chooses to move and another chooses to make way.
Revised: For example, when two agents arrive at a crossroads, the optimal solution is that one of them chooses to move and another chooses to make way.

Original: In this situation, the team can solve this simple task no matter which agent chooses to move.
Revised: In this situation, the team can solve this simple task no matter which agent chooses to move.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 𝑢𝑢0.65 0.
Revised: 𝑢𝑢0.65 0.

Original: 𝑢 11 𝑈 1 𝑈𝑢𝑢𝑢0.65 0.
Revised: 𝑢 11 𝑈 1 𝑈𝑢𝑢𝑢0.65 0.

Original: (b) QMIX, 1.
Revised: (b) QMIX, 1.

Original: (c) VDN, 1.
Revised: (c) VDN, 1.

Original: (d) UTS, 1.
Revised: (d) UTS, 1.

Original: (e) UTS, 0.75.
Revised: (e) UTS, 0.75.

Original: (f) UTS, 0.5.
Revised: (f) UTS, 0.5.

Original: (g) UTS, 0.25.
Revised: (g) UTS, 0.25.

Original: (h) UTS, 0.05.
Revised: (h) UTS, 0.05.

Original: 𝜖 = 𝜖 = 𝜖 = 𝜖 = 𝜖 = 𝜖 = 𝜖 = 𝜖 =
Revised: 𝜖 = 𝜖 = 𝜖 = 𝜖 = 𝜖 = 𝜖 = 𝜖 = 𝜖 =

Original: [Figure]
Revised: [Figure]

Original: We take a simple one-step matrix game with multiple optimal as an example.
Revised: We take a simple one-step matrix game with multiple optimal as an example.

Original: The payoff matrix is shown in Figure 7a.
Revised: The payoff matrix is shown in Figure 7a.

Original: We investigate the performance of VDN, QMIX, QPLEX, and UTS on this simple task.
Revised: We investigate the performance of VDN, QMIX, QPLEX, and UTS on this simple task.

Original: Since this matrix game is non-monotonic, the target values cannot be perfectly represented in Q lvf and Q mvf .
Revised: Since this matrix game is non-monotonic, the target values cannot be perfectly represented in Q lvf and Q mvf .

Original: When the full exploration scheme (i.e., ϵ -greedy with ϵ = 1 ) is applied, none of these methods can converge to the optimal.
Revised: When the full exploration scheme (i.e., ϵ -greedy with ϵ = 1 ) is applied, none of these methods can converge to the optimal.

Original: Although QPLEX achieves the full representational capacity (i.e., the joint Q -values Q tot is the same as the payoff), it still cannot converge because all per-agent Q-values are optimal (because of the Individual-Global-Max).
Revised: Although QPLEX achieves the full representational capacity (i.e., the joint Q -values Q tot is the same as the payoff), it still cannot converge because all per-agent Q-values are optimal (because of the Individual-Global-Max).

Original: However, in practice, we can solve this problem by ϵ -greedy with decayed ϵ .
Revised: However, in practice, we can solve this problem by ϵ -greedy with decayed ϵ .

Original: In particular, we show the Q tot returned from UTS to illustrate the effect of the ϵ .
Revised: In particular, we show the Q tot returned from UTS to illustrate the effect of the ϵ .

Original: As shown in Figure 7e-h, we can see the gap between the suboptimal and the optimal increases with the decrease of the ϵ , which improves the learning stability.
Revised: As shown in Figure 7e-h, we can see the gap between the suboptimal and the optimal increases with the decrease of the ϵ , which improves the learning stability.

Original: However, UTS inherits the characteristic of QMIX and thus is not a contraction, i.e., it would return two distinct
Revised: However, UTS inherits the characteristic of QMIX and thus is not a contraction, i.e., it would return two distinct

Original: Q tot and converge to any one of the optimal.
Revised: Q tot and converge to any one of the optimal.

Original: B R ESULTS ON MATRIX GAMES B. WQMIX WITH DIFFERENT WEIGHTS
Revised: B R ESULTS ON MATRIX GAMES B. WQMIX WITH DIFFERENT WEIGHTS

Original: In Section 3.1, we demonstrate the complete data of minimal 1 /α for WVDN and WQMIX to recover the optimal policy, where α denotes the weight on the suboptimal.
Revised: In Section 3.1, we demonstrate the complete data of minimal 1 /α for WVDN and WQMIX to recover the optimal policy, where α denotes the weight on the suboptimal.

Original: The payoff matrix is shown in 6a.
Revised: The payoff matrix is shown in 6a.

Original: The dataset is collected by the ϵ -greedy exploration, where the greedy action is set to u s
Revised: The dataset is collected by the ϵ -greedy exploration, where the greedy action is set to u s

Original: = ( u 21 , u 22 ) .
Revised: = ( u 21 , u 22 ) .

Original: [Figure]
Revised: [Figure]

Original: 𝑄 1 𝑢 1 ∗
Revised: 𝑄 1 𝑢 1 ∗

Original: We select a = 8 , b = − 12 , and (1) three reward assignments, c ∈ {− 3 , 3 , 6 } , ϵ = 0 .
Revised: We select a = 8 , b = − 12 , and (1) three reward assignments, c ∈ {− 3 , 3 , 6 } , ϵ = 0 .

Original: 25 , as well as (2) three exploration rate assignments, ϵ ∈ { 0 . 5 , 0 .
Revised: 25 , as well as (2) three exploration rate assignments, ϵ ∈ { 0 . 5 , 0 .

Original: 75 , 1 } , c = 6 , and run WVDN and WQMIX with five random seeds on a fixed dataset.
Revised: 75 , 1 } , c = 6 , and run WVDN and WQMIX with five random seeds on a fixed dataset.

Original: We show the gap between the approximated real optimal action value ˆ Q 1 ( u ∗ 1 ) and greedy action value
Revised: We show the gap between the approximated real optimal action value ˆ Q 1 ( u ∗ 1 ) and greedy action value

Original: ˆ Q 1 (ˆ u ∗ 1 ) for agent 1, which should be zero if the algorithm converges to the optimal.
Revised: ˆ Q 1 (ˆ u ∗ 1 ) for agent 1, which should be zero if the algorithm converges to the optimal.

Original: Figure 8a-b shows that the gap reduces with the increase of 1 /α , indicating that the weight α should be small enough to deal with non-monotonicity.
Revised: Figure 8a-b shows that the gap reduces with the increase of 1 /α , indicating that the weight α should be small enough to deal with non-monotonicity.

Original: In addition, we can see that α should be small enough to converge to the optimal when the suboptimal value is large or the exploration rate is low, which is consistent with our analysis.
Revised: In addition, we can see that α should be small enough to converge to the optimal when the suboptimal value is large or the exploration rate is low, which is consistent with our analysis.

Original: In Section 3.2 and Section A.2, we prove that the contradiction of the choice of α may result in converging on the suboptimal for WQMIX.
Revised: In Section 3.2 and Section A.2, we prove that the contradiction of the choice of α may result in converging on the suboptimal for WQMIX.

Original: We also conduct experiments to verify our analysis.
Revised: We also conduct experiments to verify our analysis.

Original: We consider the matrix game in Figure 9a, and train WVDN and WQMIX on a fixed dataset.
Revised: We consider the matrix game in Figure 9a, and train WVDN and WQMIX on a fixed dataset.

Original: The dataset is collected by the ϵ -greedy exploration, where the greedy action is set to u s
Revised: The dataset is collected by the ϵ -greedy exploration, where the greedy action is set to u s

Original: = ( u 21 , u 22 ) and ϵ = 0 .
Revised: = ( u 21 , u 22 ) and ϵ = 0 .

Original: We show the gap between the approximated real optimal action value ˆ Q 1 ( u ∗ 1 ) and greedy action value
Revised: We show the gap between the approximated real optimal action value ˆ Q 1 ( u ∗ 1 ) and greedy action value

Original: ˆ Q 1 (ˆ u ∗ 1 ) for agent 1, which should be zero if the algorithm converges to the optimal.
Revised: ˆ Q 1 (ˆ u ∗ 1 ) for agent 1, which should be zero if the algorithm converges to the optimal.

Original: We also show whether the greedy action is optimal by an indicator function 1 ˆQ 1 (u ∗ 1 )
Revised: We also show whether the greedy action is optimal by an indicator function 1 ˆQ 1 (u ∗ 1 )

Original: =
Revised: =

Original: ˆQ 1 (ˆu ∗ 1 ) .
Revised: ˆQ 1 (ˆu ∗ 1 ) .

Original: As shown in Figure 8c-d, WQMIX and WVDN cannot converge to the optimal with different α when the payoff is non-monotonic and stochastic.
Revised: As shown in Figure 8c-d, WQMIX and WVDN cannot converge to the optimal with different α when the payoff is non-monotonic and stochastic.

Original: Since CW-QMIX does not place α on the current greedy joint action, it can get rid of the stochastic suboptimal u s when u s is the current greedy action, i.e., ˆ u ∗ =
Revised: Since CW-QMIX does not place α on the current greedy joint action, it can get rid of the stochastic suboptimal u s when u s is the current greedy action, i.e., ˆ u ∗ =

Original: u s .
Revised: u s .

Original: However, with the decrease of Q tot ( u s ) , CW-QMIX falls into this suboptimal again once ˆ u ∗ ̸ = u s .
Revised: However, with the decrease of Q tot ( u s ) , CW-QMIX falls into this suboptimal again once ˆ u ∗ ̸ = u s .

Original: Therefore, we can see that the gap between ˆ Q 1 ( u ∗ 1 ) and ˆ Q 1 (ˆ u ∗ 1 ) is tiny.
Revised: Therefore, we can see that the gap between ˆ Q 1 ( u ∗ 1 ) and ˆ Q 1 (ˆ u ∗ 1 ) is tiny.

Original: However, as shown in Figure 8d, CW-QMIX cannot converge to the optimal, which is consistent with our analysis.
Revised: However, as shown in Figure 8d, CW-QMIX cannot converge to the optimal, which is consistent with our analysis.

Original: B. T HE APPROXIMATED JOINT ACTION - VALUES
Revised: B. T HE APPROXIMATED JOINT ACTION - VALUES

Original: We also show the joint Q -value returned from our baselines on a matrix game with non-monotonic and stochastic rewards in Figure 9a.
Revised: We also show the joint Q -value returned from our baselines on a matrix game with non-monotonic and stochastic rewards in Figure 9a.

Original: We adopt a uniform exploration strategy to approximate the uniform data distribution (i.e., ϵ -greedy exploration with ϵ = 1 ).
Revised: We adopt a uniform exploration strategy to approximate the uniform data distribution (i.e., ϵ -greedy exploration with ϵ = 1 ).

Original: As shown in Figure 9, UTS and
Revised: As shown in Figure 9, UTS and

Original: QTRAN can recover the optimal policy, while other methods fail to solve this simple task.
Revised: QTRAN can recover the optimal policy, while other methods fail to solve this simple task.

Original: Since we have discussed QMIX and WQMIX in previous sections, here we analyse QPLEX and GVR.
Revised: Since we have discussed QMIX and WQMIX in previous sections, here we analyse QPLEX and GVR.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 𝑢
Revised: 𝑢

Original: [Table]
Revised: [Table]

Original: 𝑢𝑢
Revised: 𝑢𝑢

Original: [Table]
Revised: [Table]

Original: 𝑢𝑢𝑢𝑢𝑢-
Revised: 𝑢𝑢𝑢𝑢𝑢-

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 𝑢𝑢𝑢𝑢𝑢
Revised: 𝑢𝑢𝑢𝑢𝑢

Original: [Table] [Table]
Revised: [Table] [Table]

Original: QMIX
Revised: QMIX

Original: [Table]
Revised: [Table]

Original: w
Revised: w

Original: [Table] [Table]
Revised: [Table] [Table]

Original: e) QMIX
Revised: e) QMIX

Original: [Table]
Revised: [Table]

Original: w
Revised: w

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 𝑢𝑢𝑢𝑢𝑢𝑢
Revised: 𝑢𝑢𝑢𝑢𝑢𝑢

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: 𝑢𝑢
Revised: 𝑢𝑢

Original: [Table] [Table]
Revised: [Table] [Table]

Original: (f) GVR.
Revised: (f) GVR.

Original: In QPLEX, the joint Q -value is:
Revised: In QPLEX, the joint Q -value is:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: (30) where V i ( τ i ) =
Revised: (30) where V i ( τ i ) =

Original: max u i Q
Revised: max u i Q

Original: i ( τ i , u i ) , w i ( τ ) ≥ 0 , λ i ( τ , u ) > 0 .
Revised: i ( τ i , u i ) , w i ( τ ) ≥ 0 , λ i ( τ , u ) > 0 .

Original: Let u ∗ =
Revised: Let u ∗ =

Original: [ u ∗ 1 , ..., u ∗ i , ..., u ∗ n ] , u s =
Revised: [ u ∗ 1 , ..., u ∗ i , ..., u ∗ n ] , u s =

Original: [ u s 1 , ..., u si , ..., u sn ] denote the optimal and one of the suboptimal joint action, where n is the number of agents.
Revised: [ u s 1 , ..., u si , ..., u sn ] denote the optimal and one of the suboptimal joint action, where n is the number of agents.

Original: If QPLEX is initialised to this joint action, i.e.,
Revised: If QPLEX is initialised to this joint action, i.e.,

Original: V tot ( τ ) =
Revised: V tot ( τ ) =

Original: (cid:80) ni
Revised: (cid:80) ni

Original: =1 Q i ( τ i , u si ) , it is difficult to jump out from this suboptimal for QPLEX.
Revised: =1 Q i ( τ i , u si ) , it is difficult to jump out from this suboptimal for QPLEX.

Original: Consider
Revised: Consider

Original: Q ( s, u s ) is the real second greatest action value at a given state, where u si ̸
Revised: Q ( s, u s ) is the real second greatest action value at a given state, where u si ̸

Original: = u ∗ i , ∀
Revised: = u ∗ i , ∀

Original: i ∈ A .
Revised: i ∈ A .

Original: Therefore, joint Q -values whose targets are smaller than Q tot ( τ , u s ) can be perfectly represented by QPLEX.
Revised: Therefore, joint Q -values whose targets are smaller than Q tot ( τ , u s ) can be perfectly represented by QPLEX.

Original: However, when ( τ , u ∗ ) is sampled, QPLEX tries to minimising the following loss:
Revised: However, when ( τ , u ∗ ) is sampled, QPLEX tries to minimising the following loss:

Original: [Equation]
Revised: [Equation]

Original: where Q ( τ , u ∗ ) − V tot ( τ ) > 0 , Q i ( τ , u ∗ i ) − V i ( τ ) ≤ 0 , and λ i ( τ , u ∗ ) > 0 .
Revised: where Q ( τ , u ∗ ) − V tot ( τ ) > 0 , Q i ( τ , u ∗ i ) − V i ( τ ) ≤ 0 , and λ i ( τ , u ∗ ) > 0 .

Original: As a result, λ i ( τ , u ∗ ) decreases to almost zero, and Q i ( τ , u ∗ i ) gradually reaches to V i ( τ ) .
Revised: As a result, λ i ( τ , u ∗ ) decreases to almost zero, and Q i ( τ , u ∗ i ) gradually reaches to V i ( τ ) .

Original: However, the process is very time consuming and Q i ( τ , u ∗ i ) cannot exceed V i ( τ ) in most cases due to the low weight
Revised: However, the process is very time consuming and Q i ( τ , u ∗ i ) cannot exceed V i ( τ ) in most cases due to the low weight

Original: λ
Revised: λ

Original: i ( τ , u ∗ ) , i.e., the vanishing gradient problem.
Revised: i ( τ , u ∗ ) , i.e., the vanishing gradient problem.

Original: Consequently, as shown in Figure 9h, the approximated joint
Revised: Consequently, as shown in Figure 9h, the approximated joint

Original: Q -values for the optimal u ∗ = ( u 11 , u 12 ) and three suboptimal { ( u 21 , u 32 ) , ( u 31 , u 22 )( u 31 , u 32 ) } are 4.
Revised: Q -values for the optimal u ∗ = ( u 11 , u 12 ) and three suboptimal { ( u 21 , u 32 ) , ( u 31 , u 22 )( u 31 , u 32 ) } are 4.

Original: in QPLEX.
Revised: in QPLEX.

Original: GVR uses inferior target shaping and superior experience replay to eliminate the non-optimal selftransition nodes.
Revised: GVR uses inferior target shaping and superior experience replay to eliminate the non-optimal selftransition nodes.

Original: However, it ignores the stochasticity at all as WQMIX, and thus behaves as
Revised: However, it ignores the stochasticity at all as WQMIX, and thus behaves as

Original: CW-QMIX and cannot converge to any joint action (Figure 9f).
Revised: CW-QMIX and cannot converge to any joint action (Figure 9f).

Original: More comparison among GVR,
Revised: More comparison among GVR,

Original: WQMIX, and UTS can be found in Appendix D.
Revised: WQMIX, and UTS can be found in Appendix D.

Original: C A BLATION S TUDIES
Revised: C A BLATION S TUDIES

Original: In this section, we conduct ablation studies to investigate the influence of target shaping in UTS.
Revised: In this section, we conduct ablation studies to investigate the influence of target shaping in UTS.

Original: The following methods are included in the evaluation: 1) QMIX, which is the natural ablation baseline of
Revised: The following methods are included in the evaluation: 1) QMIX, which is the natural ablation baseline of

Original: UTS, 2) UTS without target shaping (denoted by UTS-wo-TS), i.e., agents are trained only according to the best action value function q a ( s, τ a , u a ) , 3) UTS without the reward predictor (denoted by
Revised: UTS, 2) UTS without target shaping (denoted by UTS-wo-TS), i.e., agents are trained only according to the best action value function q a ( s, τ a , u a ) , 3) UTS without the reward predictor (denoted by

Original: UTS-wo-R), 4) UTS without the state predictor (denoted by UTS-wo-S), and 5) the original UTS.
Revised: UTS-wo-R), 4) UTS without the state predictor (denoted by UTS-wo-S), and 5) the original UTS.

Original: C. T HE REWARD AND THE STATE PREDICTORS
Revised: C. T HE REWARD AND THE STATE PREDICTORS

Original: First, we investigate the role of the reward and the state predictors.
Revised: First, we investigate the role of the reward and the state predictors.

Original: We compare UTS-wo-R, UTSwo-S, and UTS on matrix game (stochastic non-monotonic rewards), multi-agent Markov Decision
Revised: We compare UTS-wo-R, UTSwo-S, and UTS on matrix game (stochastic non-monotonic rewards), multi-agent Markov Decision

Original: Process (MMDP) (stochastic state transition and non-monotonic returns), predator-prey (deterministic non-monotonic returns), and predator-stag-hare (stochastic non-monotonic returns).
Revised: Process (MMDP) (stochastic state transition and non-monotonic returns), predator-prey (deterministic non-monotonic returns), and predator-stag-hare (stochastic non-monotonic returns).

Original: As shown in
Revised: As shown in

Original: Figure 10, UTS-wo-R cannot cope with stochastic rewards and thus fail to solve matrix game, MMDP, and predator-stag-hare.
Revised: Figure 10, UTS-wo-R cannot cope with stochastic rewards and thus fail to solve matrix game, MMDP, and predator-stag-hare.

Original: UTS-wo-S cannot identify whether the state-action pair leads to stochastic transition and fail to solve MMDP.
Revised: UTS-wo-S cannot identify whether the state-action pair leads to stochastic transition and fail to solve MMDP.

Original: However, since predator-prey only involves deterministic targets, the target shaping function does not overestimate the suboptimal.
Revised: However, since predator-prey only involves deterministic targets, the target shaping function does not overestimate the suboptimal.

Original: As a result, both UTS-wo-R and
Revised: As a result, both UTS-wo-R and

Original: UTS-wo-S can solve predator-prey, indicating the effectiveness of the target shaping function.
Revised: UTS-wo-S can solve predator-prey, indicating the effectiveness of the target shaping function.

Original: 
Revised: 

Original: N N N N N      0 HD Q  7 H V W  5 H W X U Q
Revised: N N N N N      0 HD Q  7 H V W  5 H W X U Q

Original: D0DWUL[*DPH 
Revised: D0DWUL[*DPH 

Original: N
Revised: N

Original: N N N N 
Revised: N N N N 

Original:     0 HD Q  7 H V W  5 H W X U Q
Revised:     0 HD Q  7 H V W  5 H W X U Q

Original: E00' 
Revised: E00' 

Original: [Figure]
Revised: [Figure]

Original: 
Revised: 

Original: [Figure]
Revised: [Figure]

Original: 
Revised: 

Original: [Figure]
Revised: [Figure]

Original: H
Revised: H

Original: [Figure]
Revised: [Figure]

Original: F
Revised: F

Original: 33SHQDOW\  
Revised: 33SHQDOW\  

Original: [Figure]
Revised: [Figure]

Original:    
Revised:    

Original: [Figure]
Revised: [Figure]

Original: The additional models (the reward and the state predictors) in UTS are introduced to identify stochasticity, which are necessary to cope with any target state-action values.
Revised: The additional models (the reward and the state predictors) in UTS are introduced to identify stochasticity, which are necessary to cope with any target state-action values.

Original: When the task does not involve stochasticity, UTS learns slower because the target shaping function relies on the results from these models.
Revised: When the task does not involve stochasticity, UTS learns slower because the target shaping function relies on the results from these models.

Original: When we have complete domain knowledge of the task, a suitable threshold for the model prediction error can be set to improve the learning speed.
Revised: When we have complete domain knowledge of the task, a suitable threshold for the model prediction error can be set to improve the learning speed.

Original: C. T HE TARGET SHAPING FUNCTION
Revised: C. T HE TARGET SHAPING FUNCTION

Original: Now we show the effectiveness of the target shaping function.
Revised: Now we show the effectiveness of the target shaping function.

Original: We compare QMIX, UTS-wo-TS, and UTS on predator-prey and predator-stag-hare.
Revised: We compare QMIX, UTS-wo-TS, and UTS on predator-prey and predator-stag-hare.

Original: As shown in Figure 11, since the miscoordination penalty is large, QMIX cannot deal with non-monotonic targets and thus fail to solve these tasks.
Revised: As shown in Figure 11, since the miscoordination penalty is large, QMIX cannot deal with non-monotonic targets and thus fail to solve these tasks.

Original: UTS-wo-TS and UTS can learn to catch the prey (stags), whereas UTS learns more quickly.
Revised: UTS-wo-TS and UTS can learn to catch the prey (stags), whereas UTS learns more quickly.

Original: Since the estimated best action value introduces additional estimation error that may cause a significant variance, we also show the standard deviation of per-agent action values Q a .
Revised: Since the estimated best action value introduces additional estimation error that may cause a significant variance, we also show the standard deviation of per-agent action values Q a .

Original: As shown in Figure 11, we can see that UTS has a lower standard deviation than UTS-wo-TS.
Revised: As shown in Figure 11, we can see that UTS has a lower standard deviation than UTS-wo-TS.

Original: To better understand why UTS can achieve better performance, first, we discuss the advantage of value factorisation methods.
Revised: To better understand why UTS can achieve better performance, first, we discuss the advantage of value factorisation methods.

Original: Linear value factorisation implicitly achieves credit assignment using a counterfactual baseline (Wang et al., 2021).
Revised: Linear value factorisation implicitly achieves credit assignment using a counterfactual baseline (Wang et al., 2021).

Original: The individual Q -value function Q i ( s, u i ) is updated by:
Revised: The individual Q -value function Q i ( s, u i ) is updated by:

Original: [Equation]
Revised: [Equation]

Original: where the residue term w ≡ [ w 1 ( s ) , ..., w n ( s )] denotes an arbitrary vector satisfying ∀ s , (cid:80) n 1 w i ( s ) = 0 .
Revised: where the residue term w ≡ [ w 1 ( s ) , ..., w n ( s )] denotes an arbitrary vector satisfying ∀ s , (cid:80) n 1 w i ( s ) = 0 .

Original: As a result, the individual Q -value in linear value factorisation can be viewed as a type of advantage function , where the per-agent baseline reduces variance but does not affect the convergence.
Revised: As a result, the individual Q -value in linear value factorisation can be viewed as a type of advantage function , where the per-agent baseline reduces variance but does not affect the convergence.

Original: Due to the non-linear mixing function in QMIX, a more general theoretical understanding of monotonic value factorisation is still insufficient.
Revised: Due to the non-linear mixing function in QMIX, a more general theoretical understanding of monotonic value factorisation is still insufficient.

Original: However, previous work shows that QMIX also implicitly performs credit assignment and outperforms linear value factorisation (Rashid et al., 2018; 2020;
Revised: However, previous work shows that QMIX also implicitly performs credit assignment and outperforms linear value factorisation (Rashid et al., 2018; 2020;

Original: Zhou et al., 2020).
Revised: Zhou et al., 2020).

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: X P E H U  R I  & D S W X U H G  6 W D J V E36+65 3HQDOW\   0 0 0 0 7LPHVWHSV      6 W D QG D U G  ' H Y L D WL RQ V 87 40,; 876ZR7
Revised: X P E H U  R I  & D S W X U H G  6 W D J V E36+65 3HQDOW\   0 0 0 0 7LPHVWHSV      6 W D QG D U G  ' H Y L D WL RQ V 87 40,; 876ZR7

Original: [Figure]
Revised: [Figure]

Original: -12 612/ 𝑢 11 𝑈 1 𝑈𝑢𝑢 12 𝑢𝑢𝑢𝑢 11 𝑈 1 𝑈𝑢𝑢 12 𝑢𝑢𝑢- - -12 -12 612/ 𝑢 11 𝑈 1 𝑈𝑢𝑢 12 𝑢𝑢𝑢- - - A numerical example is provided in Figure 12.
Revised: -12 612/ 𝑢 11 𝑈 1 𝑈𝑢𝑢 12 𝑢𝑢𝑢𝑢 11 𝑈 1 𝑈𝑢𝑢 12 𝑢𝑢𝑢- - -12 -12 612/ 𝑢 11 𝑈 1 𝑈𝑢𝑢 12 𝑢𝑢𝑢- - - A numerical example is provided in Figure 12.

Original: Figure 12a shows the payoff of a one-step non-monotonic game.
Revised: Figure 12a shows the payoff of a one-step non-monotonic game.

Original: First, we show the standard deviation of per-agent action values Q a .
Revised: First, we show the standard deviation of per-agent action values Q a .

Original: QMIX converges to the suboptimal when it meets non-monotonic targets of the joint Q -value function.
Revised: QMIX converges to the suboptimal when it meets non-monotonic targets of the joint Q -value function.

Original: As shown in Figure 12b, Q QMIX( u 11 ) keeps reducing and is smaller than the values of other actions, which implies that QMIX converges to the suboptimal due to representational limitation.
Revised: As shown in Figure 12b, Q QMIX( u 11 ) keeps reducing and is smaller than the values of other actions, which implies that QMIX converges to the suboptimal due to representational limitation.

Original: Both UTS and UTS-wo-TS can recover the optimal policy.
Revised: Both UTS and UTS-wo-TS can recover the optimal policy.

Original: The action values in UTS-wo-TS show similar trends, and thus UTS-wo-TS can be stable until it perfectly approximates all action values.
Revised: The action values in UTS-wo-TS show similar trends, and thus UTS-wo-TS can be stable until it perfectly approximates all action values.

Original: By contrast, UTS shapes the targets to ensure they can be represented by monotonic value factorisation, where the optimal policy is invariant during this shaping.
Revised: By contrast, UTS shapes the targets to ensure they can be represented by monotonic value factorisation, where the optimal policy is invariant during this shaping.

Original: UTS performs credit assignment with the shaped targets, the action values in UTS can be viewed as a type of advantage and thus are much smaller than UTS-wo-TS.
Revised: UTS performs credit assignment with the shaped targets, the action values in UTS can be viewed as a type of advantage and thus are much smaller than UTS-wo-TS.

Original: The action values in
Revised: The action values in

Original: UTS show two different trends, where Q UTS( u 11 ) is positive, Q UTS( u 21 ) and Q UTS( u 31 ) are negative when the algorithm converges.
Revised: UTS show two different trends, where Q UTS( u 11 ) is positive, Q UTS( u 21 ) and Q UTS( u 31 ) are negative when the algorithm converges.

Original: We also show the standard deviation of per-agent action values from QMIX, UTS, and UTS-wo-TS on a random 10 × 10 matrix game, where the suboptimal is filled with random numbers generated uniformly between -20 and 19, and the unique optimal value is +20.
Revised: We also show the standard deviation of per-agent action values from QMIX, UTS, and UTS-wo-TS on a random 10 × 10 matrix game, where the suboptimal is filled with random numbers generated uniformly between -20 and 19, and the unique optimal value is +20.

Original: As shown in Figure 13, UTS reduces variance compared to UTS-wo-TS, thus improving learning efficiency and stability.
Revised: As shown in Figure 13, UTS reduces variance compared to UTS-wo-TS, thus improving learning efficiency and stability.

Original: D T HE C OMPARISON WITH WQMIX AND GVR
Revised: D T HE C OMPARISON WITH WQMIX AND GVR

Original: In Section 3.1, we prove that WQMIX with a low α can overcome any non-monotonic tasks.
Revised: In Section 3.1, we prove that WQMIX with a low α can overcome any non-monotonic tasks.

Original: However, we also prove that WQMIX is particularly brittle with stochasticity because it overestimates the expected value of the suboptimal whose target is large with a low probability.
Revised: However, we also prove that WQMIX is particularly brittle with stochasticity because it overestimates the expected value of the suboptimal whose target is large with a low probability.

Original: GVR also ignores the stochasticity and cannot accurately identify whether a given action is inferior.
Revised: GVR also ignores the stochasticity and cannot accurately identify whether a given action is inferior.

Original: As a result, WQMIX with fine-tuned α and GVR are able to solve any non-monotonic targets, but cannot deal with stochasticity.
Revised: As a result, WQMIX with fine-tuned α and GVR are able to solve any non-monotonic targets, but cannot deal with stochasticity.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original:      1 X P E H U  R I  & D S W X U H G  6 W D J V H36+3HQDOW\   0 0 0 0 7LPHVWHSV      1 X P E H U  R I  & D S W X U H G  6 W D J V I
Revised:      1 X P E H U  R I  & D S W X U H G  6 W D J V H36+3HQDOW\   0 0 0 0 7LPHVWHSV      1 X P E H U  R I  & D S W X U H G  6 W D J V I

Original: 36+3HQDOW\   0 0 0 0 7LPHVWHSV      1 X P E H U  R I  & D S W X U H G  6 W D J V J36+3HQDOW\
Revised: 36+3HQDOW\   0 0 0 0 7LPHVWHSV      1 X P E H U  R I  & D S W X U H G  6 W D J V J36+3HQDOW\

Original:   0 0 0 0 7LPHVWHSV      1 X P E H U  R I  & D S W X U H G  6 W D J V K36+3HQDOW\  876 &:40,; 
Revised:   0 0 0 0 7LPHVWHSV      1 X P E H U  R I  & D S W X U H G  6 W D J V K36+3HQDOW\  876 &:40,;

Original:  &:40,;   &:40,;   2:40,;   2:40,;   2:40,;   *95 876 &:40,;   &:40,;   &:40,;   2:40,;   2:40,;   2:40,;   *9
Revised:  &:40,;   &:40,;   2:40,;   2:40,;   2:40,;   *95 876 &:40,;   &:40,;   &:40,;   2:40,;   2:40,;   2:40,;   *9

Original: To further show the difference between these methods and UTS, we compare them on predator-prey with different level of miscoordination penalties and stochasticity.
Revised: To further show the difference between these methods and UTS, we compare them on predator-prey with different level of miscoordination penalties and stochasticity.

Original: Figure 14 demonstrates that
Revised: Figure 14 demonstrates that

Original: WQMIX with a very low α and GVR can obtain the positive test return in all penalty settings on predator-prey.
Revised: WQMIX with a very low α and GVR can obtain the positive test return in all penalty settings on predator-prey.

Original: However, they cannot recover the optimal joint policy on stochastic predator-stag-hare.
Revised: However, they cannot recover the optimal joint policy on stochastic predator-stag-hare.

Original: By contrast, UTS outperforms WQMIX and GVR in these tasks.
Revised: By contrast, UTS outperforms WQMIX and GVR in these tasks.

Original: E T HE C OMPARISON WITH MORE ALGORITHMS
Revised: E T HE C OMPARISON WITH MORE ALGORITHMS

Original: We also compare our method with more algorithms to further show our motivation.
Revised: We also compare our method with more algorithms to further show our motivation.

Original: We consider recent multi-agent reinforcement learning works, including RODE (Wang et al., 2020c), MAPPO (Yu et al., 2021), and HAPPO (Kuba et al., 2021).
Revised: We consider recent multi-agent reinforcement learning works, including RODE (Wang et al., 2020c), MAPPO (Yu et al., 2021), and HAPPO (Kuba et al., 2021).

Original: Figure 15a-d shows that RODE, MAPPO, and HAPPO fail to recover the optimal policy in the matrix game and the MMDP with stochastic and non-monotonic targets, where the payoff is shown in Section 6.1.
Revised: Figure 15a-d shows that RODE, MAPPO, and HAPPO fail to recover the optimal policy in the matrix game and the MMDP with stochastic and non-monotonic targets, where the payoff is shown in Section 6.1.

Original: In addition, they also cannot solve predator-prey and predator-stag-hare when the penalty is large.
Revised: In addition, they also cannot solve predator-prey and predator-stag-hare when the penalty is large.

Original: This is because that RODE applies the same monotonic mixing network as QMIX and thus can only represent values in the restricted monotonic space.
Revised: This is because that RODE applies the same monotonic mixing network as QMIX and thus can only represent values in the restricted monotonic space.

Original: This representational limitation makes RODE cannot cope with non-monotonic targets.
Revised: This representational limitation makes RODE cannot cope with non-monotonic targets.

Original: MAPPO is an independent learning approach with many valuable techniques, but it cannot deal with these tasks due to the non-stationarity of the environment.
Revised: MAPPO is an independent learning approach with many valuable techniques, but it cannot deal with these tasks due to the non-stationarity of the environment.

Original: HAPPO applies an agent-by-agent update scheme and improves the performance compared to MAPPO.
Revised: HAPPO applies an agent-by-agent update scheme and improves the performance compared to MAPPO.

Original: This update scheme can only guarantee convergence to one of the Nash
Revised: This update scheme can only guarantee convergence to one of the Nash

Original: Equilibriums in theory (each agent is treated to receive the equal reward under the team reward setting).
Revised: Equilibriums in theory (each agent is treated to receive the equal reward under the team reward setting).

Original: However, the monotonic target in value decomposition methods can be interpreted as a static multivariable optimisation problem, where there are non-optimal solutions that cannot be improved upon by coordinate descent.
Revised: However, the monotonic target in value decomposition methods can be interpreted as a static multivariable optimisation problem, where there are non-optimal solutions that cannot be improved upon by coordinate descent.

Original: The sequential policy update scheme cannot guarantee convergence to the optimal in this optimisation problem (Bertsekas, 2019).
Revised: The sequential policy update scheme cannot guarantee convergence to the optimal in this optimisation problem (Bertsekas, 2019).

Original: Or we can also view the task of predator-prey and predator-stag-hare as a Pareto Selection problem, and HAPPO cannot ensure convergence to the real optimal (the Pareto optimal) and thus fail to solve these tasks
Revised: Or we can also view the task of predator-prey and predator-stag-hare as a Pareto Selection problem, and HAPPO cannot ensure convergence to the real optimal (the Pareto optimal) and thus fail to solve these tasks

Original: [Table]
Revised: [Table]

Original: The representational limitation of monotonic value decomposition will not necessarily result in converging to suboptimal.
Revised: The representational limitation of monotonic value decomposition will not necessarily result in converging to suboptimal.

Original: We can see that RODE outperforms UTS in some SMAC maps.
Revised: We can see that RODE outperforms UTS in some SMAC maps.

Original: However,
Revised: However,

Original: UTS still achieves comparable performance with RODE in many SMAC tasks, indicating that the target shaping function is more general than the weighting function in WQMIX
Revised: UTS still achieves comparable performance with RODE in many SMAC tasks, indicating that the target shaping function is more general than the weighting function in WQMIX

Original: [Table]
Revised: [Table]

Original: 0 0 7LPHVWHSV     
Revised: 0 0 7LPHVWHSV     

Original:  0 HD Q  7 H V W  : L Q   D W H   IFRUULGRU  0 0 0 0 0 7LPHVWHSV 
Revised:  0 HD Q  7 H V W  : L Q   D W H   IFRUULGRU  0 0 0 0 0 7LPHVWHSV 

Original:    
Revised:    

Original:  0 HD Q  7 H V W  : L Q   D W H   JKBYVB]  0 0 0 0 7LPHVWHSV     
Revised:  0 HD Q  7 H V W  : L Q   D W H   JKBYVB]  0 0 0 0 7LPHVWHSV     

Original:  0 HD Q  7 H V W  : L Q   D W H   K
Revised:  0 HD Q  7 H V W  : L Q   D W H   K

Original: PBYVBP 87 0$33 +$33 52'( 87 0$33 +$33 52'(
Revised: PBYVBP 87 0$33 +$33 52'( 87 0$33 +$33 52'(

Original: The sampling efficiency of MAPPO and HAPPO is lower than Q -learning based algorithms because they are on-policy algorithms.
Revised: The sampling efficiency of MAPPO and HAPPO is lower than Q -learning based algorithms because they are on-policy algorithms.

Original: Therefore, Figure 15e-h shows their poor performance compared with
Revised: Therefore, Figure 15e-h shows their poor performance compared with

Original: RODE and UTS with limited environment steps.
Revised: RODE and UTS with limited environment steps.

Original: Not that we use SC2.4.6.2.69232 (the version of StarCraft II which VDN, QMIX, and RODE used in their papers) instead of SC2.4.10 (the version which policy-based algorithm always used).
Revised: Not that we use SC2.4.6.2.69232 (the version of StarCraft II which VDN, QMIX, and RODE used in their papers) instead of SC2.4.10 (the version which policy-based algorithm always used).

Original: Performance is not always comparable between versions.
Revised: Performance is not always comparable between versions.

Original: We emphasise that our work is to provide a concrete analysis of the relationship between the nonmonotonic and stochastic targets and the monotonic value decomposition.
Revised: We emphasise that our work is to provide a concrete analysis of the relationship between the nonmonotonic and stochastic targets and the monotonic value decomposition.

Original: We propose a target shaping function to eliminate the adverse impact from representational limitation, where the approximation of all original targets is not required.
Revised: We propose a target shaping function to eliminate the adverse impact from representational limitation, where the approximation of all original targets is not required.

Original: The shaping function project all original targets to a monotonic space and keeps the optimal policy unchanged.
Revised: The shaping function project all original targets to a monotonic space and keeps the optimal policy unchanged.

Original: Since all shaped targets are tractable to monotonic value decomposition, QMIX can recover the optimal policy with any original target.
Revised: Since all shaped targets are tractable to monotonic value decomposition, QMIX can recover the optimal policy with any original target.

Original: That is to say, we relax the monotonic assumption of the relationship between the joint Q -value and individual
Revised: That is to say, we relax the monotonic assumption of the relationship between the joint Q -value and individual

Original: Q -values.
Revised: Q -values.

Original: F E XPERIMENTAL S ETUP
Revised: F E XPERIMENTAL S ETUP

Original: We adopt the same architecture for VDN, QMIX, QTRAN, QPLEX, and WQMIX as (Samvelyan et al., 2019; Rashid et al., 2020).
Revised: We adopt the same architecture for VDN, QMIX, QTRAN, QPLEX, and WQMIX as (Samvelyan et al., 2019; Rashid et al., 2020).

Original: Each agent independently learns a policy with fully shared parameters between all policies.
Revised: Each agent independently learns a policy with fully shared parameters between all policies.

Original: We used RMSProp with a learning rate of 5 × 10 − 4 and γ = 0 .
Revised: We used RMSProp with a learning rate of 5 × 10 − 4 and γ = 0 .

Original: 99 , buffer size 5000 , mini-batch size 32 for all algorithms.
Revised: 99 , buffer size 5000 , mini-batch size 32 for all algorithms.

Original: We use an ϵ -greedy strategy, and ϵ is linearly annealed from 1 to 0.05 over 50 k timesteps (unless specified otherwise).
Revised: We use an ϵ -greedy strategy, and ϵ is linearly annealed from 1 to 0.05 over 50 k timesteps (unless specified otherwise).

Original: The dimension of each agent’s GRU hidden state is set to 64 .
Revised: The dimension of each agent’s GRU hidden state is set to 64 .

Original: For WQMIX, we use α = 0 .
Revised: For WQMIX, we use α = 0 .

Original: 1 on predator-prey and predator-stag-hare (unless specified otherwise), and α = 0 .
Revised: 1 on predator-prey and predator-stag-hare (unless specified otherwise), and α = 0 .

Original: 5 in SMAC.
Revised: 5 in SMAC.

Original: For UTS, we use the same agent and mixing network in WQMIX as (Rashid et al., 2020).
Revised: For UTS, we use the same agent and mixing network in WQMIX as (Rashid et al., 2020).

Original: For the reward and the state predictors, we use a multilayer perceptron (MLP) with ReLU non-linearities and a gated recurrent unit (GRU) to extract features, where the hidden dim is 128.
Revised: For the reward and the state predictors, we use a multilayer perceptron (MLP) with ReLU non-linearities and a gated recurrent unit (GRU) to extract features, where the hidden dim is 128.

Original: Then we use another MLP to output the predicted reward, the standard deviation and the embedding of the next state.
Revised: Then we use another MLP to output the predicted reward, the standard deviation and the embedding of the next state.

Original: The first and the GRU parameters are shared.
Revised: The first and the GRU parameters are shared.

Original: The dimension of the target embedding for the state m is set to 16.
Revised: The dimension of the target embedding for the state m is set to 16.

Original: We use Adam with a learning rate of 1 × 10 − 3 to train the predictors.
Revised: We use Adam with a learning rate of 1 × 10 − 3 to train the predictors.

Original: For the best Q -value function, we use feed-forward networks with three hidden layers of dim { 64 , 64 } and
Revised: For the best Q -value function, we use feed-forward networks with three hidden layers of dim { 64 , 64 } and

Original: ReLU non-linearities, where the optimiser is RMSProp with a learning rate of 5 ×
Revised: ReLU non-linearities, where the optimiser is RMSProp with a learning rate of 5 ×

Original: 10 − 4 .
Revised: 10 − 4 .

Original: We conduct experiments on an NVIDIA RTX 3090 GPU.
Revised: We conduct experiments on an NVIDIA RTX 3090 GPU.

Original: Each task needs to train for about 5 hours on predator-prey, and about 8 to 16 hours on SMAC, depending on the number of agents and episode length limit of each map.
Revised: Each task needs to train for about 5 hours on predator-prey, and about 8 to 16 hours on SMAC, depending on the number of agents and episode length limit of each map.

Original: We evaluate 32 episodes with decentralised greedy action selection every 10 k timesteps for each algorithm.
Revised: We evaluate 32 episodes with decentralised greedy action selection every 10 k timesteps for each algorithm.

Original: We use an ϵ -greedy policy in which ϵ decreases from 1 to 0.05 over 1 M timesteps for 3s5z vs 3s6z , 6h vs 8z and corridor , and over 50 k timesteps for others.
Revised: We use an ϵ -greedy policy in which ϵ decreases from 1 to 0.05 over 1 M timesteps for 3s5z vs 3s6z , 6h vs 8z and corridor , and over 50 k timesteps for others.

Original: All the learning curves in the experiments are plotted based on five training runs with different random seeds using mean and standard deviation with confidence internal 95%.
Revised: All the learning curves in the experiments are plotted based on five training runs with different random seeds using mean and standard deviation with confidence internal 95%.