Original: B EST P OSSIBLE Q-L EARNING
Revised: B EST P OSSIBLE Q-L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Fully decentralized learning, where the global information, i.e. , the actions of other agents, is inaccessible, is a fundamental challenge in  multi-agent reinforcement learning.
Revised: Fully decentralized learning, where the global information, i.e. , the actions of other agents, is inaccessible, is a fundamental challenge in cooperative multi-agent reinforcement learning.

Original: However, the convergence and optimality of most decentralized algorithms are not theoretically guaranteed, since the transition probabilities are non-stationary as all agents are updating policies simultaneously.
Revised: However, the convergence and optimality of most decentralized algorithms are not theoretically guaranteed, since the transition probabilities are non-stationary as all agents are updating policies simultaneously.

Original: To tackle this challenge, we propose best possible operator , a novel decentralized operator, and prove that the policies of agents will converge to the optimal joint policy if each agent independently updates its individual state-action value by the operator.
Revised: To tackle this challenge, we propose best possible operator , a novel decentralized operator, and prove that the policies of agents will converge to the optimal joint policy if each agent independently updates its individual state-action value by the operator.

Original: Further, to make the update more efficient and practical, we simplify the operator and prove that the convergence and optimality still hold with the simplified one.
Revised: Further, to make the update more efficient and practical, we simplify the operator and prove that the convergence and optimality still hold with the simplified one.

Original: By instantiating the simplified operator, the derived fully decentralized algorithm, best possible Q-learning (BQL), does not suffer from non-stationarity.
Revised: By instantiating the simplified operator, the derived fully decentralized algorithm, best possible Q-learning (BQL), does not suffer from non-stationarity.

Original: Empirically, we show that BQL achieves remarkable improvement over baselines in a variety of cooperative multi-agent tasks.
Revised: Empirically, we show that BQL achieves remarkable improvement over baselines in a variety of cooperative multi-agent tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Multi-agent reinforcement learning (MARL) trains a group of agents to cooperatively maximize the cumulative shared reward, which has great significance for real-world applications, including logistics (Li et al., 2019), traffic signal control (Xu et al., 2021), power dispatch (Wang et al., 2021b), and games (Vinyals et al., 2019).
Revised: Multi-agent reinforcement learning (MARL) trains a group of agents to cooperatively maximize the cumulative shared reward, which has great significance for real-world applications, including logistics (Li et al., 2019), traffic signal control (Xu et al., 2021), power dispatch (Wang et al., 2021b), and games (Vinyals et al., 2019).

Original: Although most existing MARL methods follow the paradigm of centralized training and decentralized execution (CTDE), in many scenarios where the information of all agents is unavailable in the training period, each agent has to learn independently without centralized information.
Revised: Although most existing MARL methods follow the paradigm of centralized training and decentralized execution (CTDE), in many scenarios where the information of all agents is unavailable in the training period, each agent has to learn independently without centralized information.

Original: Thus, fully decentralized learning , where the agents can only use local experiences without the actions of other agents, is highly desirable.
Revised: Thus, fully decentralized learning , where the agents can only use local experiences without the actions of other agents, is highly desirable.

Original: However, in fully decentralized learning, as other agents are treated as a part of the environment and are updating their policies simultaneously, the transition probabilities from the perspective of individual agents will be non-stationary.
Revised: However, in fully decentralized learning, as other agents are treated as a part of the environment and are updating their policies simultaneously, the transition probabilities from the perspective of individual agents will be non-stationary.

Original: Thus, the convergence of most decentralized algorithms, e.g. , independent Q-learning (IQL) (Tan, 1993), is not theoretically guaranteed.
Revised: Thus, the convergence of most decentralized algorithms, e.g. , independent Q-learning (IQL) (Tan, 1993), is not theoretically guaranteed.

Original: Multi-agent alternate Q-learning (MA2QL)
Revised: Multi-agent alternate Q-learning (MA2QL)

Original: (Su et al., 2022) guarantees the convergence to a Nash equilibrium, but the converged equilibrium may not be the optimal one when there are multiple equilibria (Zhang et al., 2021a).
Revised: (Su et al., 2022) guarantees the convergence to a Nash equilibrium, but the converged equilibrium may not be the optimal one when there are multiple equilibria (Zhang et al., 2021a).

Original: Distributed IQL (Lauer & Riedmiller, 2000) can learn the optimal joint policy, yet is limited to deterministic environments.
Revised: Distributed IQL (Lauer & Riedmiller, 2000) can learn the optimal joint policy, yet is limited to deterministic environments.

Original: How to guarantee the convergence of the optimal joint policy in stochastic environments remains open.
Revised: How to guarantee the convergence of the optimal joint policy in stochastic environments remains open.

Original: To tackle this challenge, we propose best possible operator , a novel decentralized operator to update the individual state-action value of each agent, and prove that the policies of agents converge to the optimal joint policy under this operator.
Revised: To tackle this challenge, we propose best possible operator , a novel decentralized operator to update the individual state-action value of each agent, and prove that the policies of agents converge to the optimal joint policy under this operator.

Original: However, it is inefficient and thus impractical to perform best possible operator, because at each update it needs to compute the expected values of all possible transition probabilities and update the state-action value to be the maximal one.
Revised: However, it is inefficient and thus impractical to perform best possible operator, because at each update it needs to compute the expected values of all possible transition probabilities and update the state-action value to be the maximal one.

Original: Therefore, we further propose simplified best possible operator .
Revised: Therefore, we further propose simplified best possible operator .

Original: At each update, the simplified operator only computes the expected value of one of the possible transition probabilities and monotonically updates the state-action value.
Revised: At each update, the simplified operator only computes the expected value of one of the possible transition probabilities and monotonically updates the state-action value.

Original: We prove that the policies of agents also converge to the optimal joint policy under the simplified operator.
Revised: We prove that the policies of agents also converge to the optimal joint policy under the simplified operator.

Original: We respectively instantiate the simplified operator with Q-table for tabular cases and with neural networks for complex environments.
Revised: We respectively instantiate the simplified operator with Q-table for tabular cases and with neural networks for complex environments.

Original: In Q-table instantiation, nonstationarity is instinctively avoided, and in neural network instantiation, non-stationarity in replay buffer is no longer a drawback, but a necessary condition for convergence.
Revised: In Q-table instantiation, nonstationarity is instinctively avoided, and in neural network instantiation, non-stationarity in replay buffer is no longer a drawback, but a necessary condition for convergence.

Original: The proposed algorithm, best possible Q-learning ( BQL ), is fully decentralized, without using the information of other agents.
Revised: The proposed algorithm, best possible Q-learning ( BQL ), is fully decentralized, without using the information of other agents.

Original: We evaluate BQL on a variety of multi-agent cooperative tasks, i.e. , stochastic games, MPE-based differential games (Lowe et al., 2017), multi-agent MuJoCo (de Witt et al., 2020b), and SMAC (Samvelyan et al., 2019), covering fully and partially observable, deterministic and stochastic, discrete and continuous environments.
Revised: We evaluate BQL on a variety of multi-agent cooperative tasks, i.e. , stochastic games, MPE-based differential games (Lowe et al., 2017), multi-agent MuJoCo (de Witt et al., 2020b), and SMAC (Samvelyan et al., 2019), covering fully and partially observable, deterministic and stochastic, discrete and continuous environments.

Original: Empirically, BQL substantially outperforms baselines.
Revised: Empirically, BQL substantially outperforms baselines.

Original: To the best of our knowledge, BQL is the first decentralized algorithm that guarantees the convergence to the global optimum in stochastic environments, and more simplifications and instantiations of best possible operator can be further explored.
Revised: To the best of our knowledge, BQL is the first decentralized algorithm that guarantees the convergence to the global optimum in stochastic environments, and more simplifications and instantiations of best possible operator can be further explored.

Original: We believe BQL will be a new paradigm for fully decentralized learning.
Revised: We believe BQL will be a new paradigm for fully decentralized learning.

Original:  M ETHOD 2. P RELIMINARIES
Revised:  M ETHOD 2. P RELIMINARIES

Original: We consider N -agent MDP M env
Revised: We consider N -agent MDP M env

Original: = < S , O , A , R, P env , γ > with the state space S and the joint action space A .
Revised: = < S , O , A , R, P env , γ > with the state space S and the joint action space A .

Original: At each timestep, each agent i chooses an individual action a i , and the environment transitions to the next state s ′ by taking the joint action a with the transition probabilities
Revised: At each timestep, each agent i chooses an individual action a i , and the environment transitions to the next state s ′ by taking the joint action a with the transition probabilities

Original: P env ( s ′ | s, a ) .
Revised: P env ( s ′ | s, a ) .

Original: For simplicity of theoretical analysis, we assume all agents obtain the state s , though in practice each agent i can make decisions based on local observation o i ∈ O or trajectory.
Revised: For simplicity of theoretical analysis, we assume all agents obtain the state s , though in practice each agent i can make decisions based on local observation o i ∈ O or trajectory.

Original: All agents obtain a shared reward r = R ( s, s ′ ) and learn to maximize the expected return E (cid:80) ∞ t =

Revised: All agents obtain a shared reward r = R ( s, s ′ ) and learn to maximize the expected return E (cid:80) ∞ t =

Original: γ t r t with the discount factor γ .
Revised: γ t r t with the discount factor γ .

Original: In fully decentralized setting, M env is partially observable to each agent, since each agent i only observes its own action a i instead of the joint action a .
Revised: In fully decentralized setting, M env is partially observable to each agent, since each agent i only observes its own action a i instead of the joint action a .

Original: From the perspective of each agent i , there is an MDP M i = < S ,
Revised: From the perspective of each agent i , there is an MDP M i = < S ,

Original: A i , R, P i , γ > with the individual action space
Revised: A i , R, P i , γ > with the individual action space

Original: A i and the transition probabilities
Revised: A i and the transition probabilities

Original: [Equation]
Revised: [Equation]

Original: where π − i denotes the joint policy of all agents except agent i , similarly for a − i .
Revised: where π − i denotes the joint policy of all agents except agent i , similarly for a − i .

Original: According to (1), the transition probabilities P i depend on the policies of other agents π − i .
Revised: According to (1), the transition probabilities P i depend on the policies of other agents π − i .

Original: As other agents are updating their policies continuously, P i becomes non-stationary .
Revised: As other agents are updating their policies continuously, P i becomes non-stationary .

Original: On the non-stationary transition probabilities, the convergence of independent Q-learning
Revised: On the non-stationary transition probabilities, the convergence of independent Q-learning

Original: [Equation]
Revised: [Equation]

Original: is not guaranteed, and how to learn the optimal joint policy in fully decentralized settings is quite a challenge.
Revised: is not guaranteed, and how to learn the optimal joint policy in fully decentralized settings is quite a challenge.

Original: In the next section, we propose best possible operator , a novel fully decentralized operator, which theoretically guarantees the convergence to the optimal joint policy in stochastic environments.
Revised: In the next section, we propose best possible operator , a novel fully decentralized operator, which theoretically guarantees the convergence to the optimal joint policy in stochastic environments.

Original: 2. B EST P OSSIBLE O PERATOR
Revised: 2. B EST P OSSIBLE O PERATOR

Original: First, let us consider the optimal joint Q-value
Revised: First, let us consider the optimal joint Q-value

Original: [Equation]
Revised: [Equation]

Original: which is the expected return of the optimal joint policy π ∗ ( s )
Revised: which is the expected return of the optimal joint policy π ∗ ( s )

Original: = arg max a Q ( s, a ) .
Revised: = arg max a Q ( s, a ) .

Original: Based on the optimal joint Q-value, for each agent i , we define max a − i Q ( s, a i , a − i ) , which follows the fixed point equation:
Revised: Based on the optimal joint Q-value, for each agent i , we define max a − i Q ( s, a i , a − i ) , which follows the fixed point equation:

Original: [Equation]
Revised: [Equation]

Original: where π ∗− i ( s, a i ) =
Revised: where π ∗− i ( s, a i ) =

Original: arg max a − i Q ( s, a i , a − i ) is the optimal conditional joint policy of other agents given a i .
Revised: arg max a − i Q ( s, a i , a − i ) is the optimal conditional joint policy of other agents given a i .

Original: (4) is from taking max a − i on both sides of (3), and (5) is by folding π ∗− i ( s, a i ) into P env .
Revised: (4) is from taking max a − i on both sides of (3), and (5) is by folding π ∗− i ( s, a i ) into P env .

Original: Then we have the following lemma.
Revised: Then we have the following lemma.

Original: Lemma 1.
Revised: Lemma 1.

Original: If each agent i learns the independent value function Q i ( s, a i ) = max a − i Q ( s, a i , a − i ) , and takes actions as arg max a i Q i ( s, a i ) , the agents will obtain the optimal joint policy, when there is only one optimal joint policy 2 .
Revised: If each agent i learns the independent value function Q i ( s, a i ) = max a − i Q ( s, a i , a − i ) , and takes actions as arg max a i Q i ( s, a i ) , the agents will obtain the optimal joint policy, when there is only one optimal joint policy 2 .

Original: Proof.
Revised: Proof.

Original: Since max a i max a − i Q ( s, a i , a − i ) = max a Q ( s, a ) , and there is only one optimal joint policy, arg max a i Q i ( s, a i ) is the action of agent
Revised: Since max a i max a − i Q ( s, a i , a − i ) = max a Q ( s, a ) , and there is only one optimal joint policy, arg max a i Q i ( s, a i ) is the action of agent

Original: i in the optimal joint action a .
Revised: i in the optimal joint action a .

Original: According to Lemma 1, to obtain the optimal joint policy is to let each agent i learn the value function Q i ( s, a i )
Revised: According to Lemma 1, to obtain the optimal joint policy is to let each agent i learn the value function Q i ( s, a i )

Original: =
Revised: =

Original: max a − i Q ( s, a i , a − i ) .
Revised: max a − i Q ( s, a i , a − i ) .

Original: To this end, we propose a new operator to update Q i in a fully decentralized way:
Revised: To this end, we propose a new operator to update Q i in a fully decentralized way:

Original: [Equation]
Revised: [Equation]

Original: Given s and a i , there will be numerous P
Revised: Given s and a i , there will be numerous P

Original: i ( s ′ | s, a i ) due to different other agents’ policies π − i .
Revised: i ( s ′ | s, a i ) due to different other agents’ policies π − i .

Original: To reduces the complexity, we can only consider the deterministic policies,  so the operator (6) takes the maximum only over the transition probabilities
Revised: To reduces the complexity, we can only consider the deterministic policies, because when there is only one optimal joint policy, the optimal joint policy must be deterministic (Puterman, 1994), so the operator (6) takes the maximum only over the transition probabilities

Original: P i ( s ′ | s, a i ) under deterministic π − i .
Revised: P i ( s ′ | s, a i ) under deterministic π − i .

Original: Intuitively, the operator continuously pursues the ‘best possible expected return’, until Q i reaches the optimal expected return max a − i Q ( s, a i , a − i ) , so we name the operator (6) as best possible operator .
Revised: Intuitively, the operator continuously pursues the ‘best possible expected return’, until Q i reaches the optimal expected return max a − i Q ( s, a i , a − i ) , so we name the operator (6) as best possible operator .

Original: In the following, we theoretically prove that Q i ( s, a i ) converges to max a − i Q ( s, a i , a − i ) under best possible operator, thus the agents learn the optimal joint policy.
Revised: In the following, we theoretically prove that Q i ( s, a i ) converges to max a − i Q ( s, a i , a − i ) under best possible operator, thus the agents learn the optimal joint policy.

Original: Let Q ki ( s, a i ) denote the value function in the update k , Q 0 i is initialized to be the minimal return, and
Revised: Let Q ki ( s, a i ) denote the value function in the update k , Q 0 i is initialized to be the minimal return, and

Original: Q i ( s, a i ) :
Revised: Q i ( s, a i ) :

Original: = Q ∞ i ( s, a i ) .
Revised: = Q ∞ i ( s, a i ) .

Original: Then, we have the following Lemma.
Revised: Then, we have the following Lemma.

Original: Lemma 2. max a − i Q ( s, a i , a − i ) ≥ Q ki ( s, a i ) , ∀ s, a i , ∀ k, under best possible operator.
Revised: Lemma 2. max a − i Q ( s, a i , a − i ) ≥ Q ki ( s, a i ) , ∀ s, a i , ∀ k, under best possible operator.

Original: Proof.
Revised: Proof.

Original: We prove the lemma by induction.
Revised: We prove the lemma by induction.

Original: First, as Q 0 i is initialized to be the minimal return, max a − i Q ( s, a i , a − i ) ≥ Q 

Revised: First, as Q 0 i is initialized to be the minimal return, max a − i Q ( s, a i , a − i ) ≥ Q

Original: i ( s, a i ) .
Revised: i ( s, a i ) .

Original: Then, suppose max a − i Q ( s, a i , a − i ) ≥ Q k − 

Revised: Then, suppose max a − i Q ( s, a i , a − i ) ≥ Q k −

Original: i
Revised: i

Original: s, a i ) , ∀ s, a i .
Revised: s, a i ) , ∀ s, a i .

Original: By denoting arg max P i
Revised: By denoting arg max P i

Original: [Equation]
Revised: [Equation]

Original: s ′|
Revised: s ′

Original: [Equation]

Original: ,a i )

Original: [Equation]

Original: i

Original: [Equation]
Revised: [Equation]

Original: |

Original: [Equation]
Revised: [Equation]

Original: ,a i )(cid:2)
Revised: s ′ , a ′ i )

Original: [Equation]
Revised: [Equation]

Original: + γ max a ′ i Q [Equation] ( s ′ , a ′ i ) (cid:3) as P ∗ i ( s ′ | s, a i ) , we have
Revised: as P ∗ i ( s ′ | s, a i ) , we have

Original: [Equation]
Revised: [Equation]

Original: where a ′∗ i = arg max a ′ i Q k − 1 i ( s ′ , a ′ i ) .
Revised: where a ′∗ i = arg max a ′ i Q k − 1 i ( s ′ , a ′ i ) .

Original: Thus, it holds in the update k .
Revised: Thus, it holds in the update k .

Original: By the principle of induction, the lemma holds for all updates.
Revised: By the principle of induction, the lemma holds for all updates.

Original: Intuitively, max a − i Q ( s, a i , a − i ) is the optimal expected return after taking action a i , so it is the upper bound of Q i ( s, a i ) .
Revised: Intuitively, max a − i Q ( s, a i , a − i ) is the optimal expected return after taking action a i , so it is the upper bound of Q i ( s, a i ) .

Original: Further, based on Lemma 2, we prove the following lemma.
Revised: Further, based on Lemma 2, we prove the following lemma.

Original: Lemma 3.
Revised: Lemma 3.

Original: Q i ( s, a i ) converges to max a − i Q ( s, a i , a − i ) under best possible operator.
Revised: Q i ( s, a i ) converges to max a − i Q ( s, a i , a − i ) under best possible operator.

Original: Proof.
Revised: Proof.

Original: From (5) and (6), we have
Revised: From (5) and (6)

Original: [Equation]
Revised: [Equation]

Original: max a − i Q ( s, a i , a − i ) − Q k − 1 i ( s, a
Revised: we have

Original: [Equation]
Revised: [Equation]

Original:  ∞ .
Revised: − i ) − Q k − 1 i ( s ′ , a ′ i ) (cid:33) = γ (cid:13)(cid:13)(cid:13) max a − i Q ( s, a i , a − i ) − Q k − 1 i ( s, a i ) (cid:13)(cid:13)(cid:13) ∞ .

Original: We have (cid:13)(cid:13) max a − i Q ( s, a i , a − i ) − Q ki
Revised: We have (cid:13)(cid:13) max a − i Q ( s, a i , a − i ) − Q ki

Original: ( s, a i ) (cid:13)(cid:13) ∞ ≤ γ k (cid:13)(cid:13) max a − i Q ( s, a i , a − i ) − Q 

Revised: ( s, a i ) (cid:13)(cid:13) ∞ ≤ γ k (cid:13)(cid:13) max a − i Q ( s, a i , a − i ) − Q

Original: i ( s, a i ) (cid:13)(cid:13) ∞ .
Revised: i ( s, a i ) (cid:13)(cid:13) ∞ .

Original: Let k → ∞ , then Q i ( s, a i ) → max a − i Q ( s, a i , a − i ) , thus the lemma holds.
Revised: Let k → ∞ , then Q i ( s, a i ) → max a − i Q ( s, a i , a − i ) , thus the lemma holds.

Original: According to Lemma 1 and 3, we have:
Revised: According to Lemma 1 and 3, we have:

Original: Theorem 1.
Revised: Theorem 1.

Original: The agents learn the optimal joint policy under best possible operator.
Revised: The agents learn the optimal joint policy under best possible operator.

Original: 2. S IMPLIFIED B EST P OSSIBLE O PERATOR
Revised: 2. S IMPLIFIED B EST P OSSIBLE O PERATOR

Original: Best possible operator guarantees the convergence to the optimal joint policy.
Revised: Best possible operator guarantees the convergence to the optimal joint policy.

Original: However, to perform (6), every update, each agent i has to compute the expected values of all possible transition probabilities and update Q i to be the maximal expected value, which is too costly.
Revised: However, to perform (6), every update, each agent i has to compute the expected values of all possible transition probabilities and update Q i to be the maximal expected value, which is too costly.

Original: Therefore, we introduce an auxiliary value function Q e i ( s, a i ) , and simplify (6) into two operators.
Revised: Therefore, we introduce an auxiliary value function Q e i ( s, a i ) , and simplify (6) into two operators.

Original: First, at each update, we randomly select one of possible transition probabilities ˜ P
Revised: First, at each update, we randomly select one of possible transition probabilities ˜ P

Original: i for each ( s, a i ) and update Q e i ( s, a i ) by
Revised: i for each ( s, a i ) and update Q e i ( s, a i ) by

Original: [Equation]
Revised: [Equation]

Original:  Q e i ( s, a i ) represents the expected value of the selected transition probabilities.
Revised: [Equation] Q e i ( s, a i ) represents the expected value of the selected transition probabilities.

Original: Then we monotonically update Q i ( s, a i ) by
Revised: Then we monotonically update Q i ( s, a i ) by

Original: [Equation]
Revised: [Equation]

Original: We define (7) and (8) together as simplified best possible operator .
Revised: We define (7) and (8) together as simplified best possible operator .

Original: By performing simplified best possible operator, Q i ( s, a i ) is efficiently updated towards the maximal expected value.
Revised: By performing simplified best possible operator, Q i ( s, a i ) is efficiently updated towards the maximal expected value.

Original: And we have the following lemma.
Revised: And we have the following lemma.

Original: Lemma 4.
Revised: Lemma 4.

Original: Q i ( s, a i ) converges to max a − i Q ( s, a i , a − i ) under simplified best possible operator.
Revised: Q i ( s, a i ) converges to max a − i Q ( s, a i , a − i ) under simplified best possible operator.

Original: Proof.
Revised: Proof.

Original: According to (8), as Q i ( s, a i ) is monotonically increased, Q ki ( s, a i ) ≥ Q k − 1 i ( s, a i ) in the update k .
Revised: According to (8), as Q i ( s, a i ) is monotonically increased, Q ki ( s, a i ) ≥ Q k − 1 i ( s, a i ) in the update k .

Original: Similar to the proof of Lemma 2, we can easily prove max a − i Q ( s, a i , a − i ) ≥ Q ki ( s, a i ) under (7) and (8).
Revised: Similar to the proof of Lemma 2, we can easily prove max a − i Q ( s, a i , a − i ) ≥ Q ki ( s, a i ) under (7) and (8).

Original: Thus, { Q k i ( s, a i ) } is an increasing sequence and bounded above.
Revised: Thus, { Q ki ( s, a i ) } is an increasing sequence and bounded above.

Original: According to the monotone convergence theorem, { Q ki ( s, a i ) } converges when k → ∞ , and let Q i ( s, a i ) :
Revised: According to the monotone convergence theorem, { Q ki ( s, a i ) } converges when k → ∞ , and let Q i ( s, a i ) :

Original: = Q ∞ i ( s, a i ) .
Revised: = Q ∞ i ( s, a i ) .

Original: Then we prove that the converged value Q i ( s, a i ) is equal to max a − i Q ( s, a i , a − i ) .
Revised: Then we prove that the converged value Q i ( s, a i ) is equal to max a − i Q ( s, a i , a − i ) .

Original: Due to monotonicity and convergence, ∀ ϵ, s, a i , ∃ K, when k > K, Q ki ( s, a i ) − Q k − 1 i ( s, a i ) ≤ ϵ , no matter which ˜ P i is selected in the update k .
Revised: Due to monotonicity and convergence, ∀ ϵ, s, a i , ∃ K, when k > K, Q ki ( s, a i ) − Q k − 1 i ( s, a i ) ≤ ϵ , no matter which ˜ P i is selected in the update k .

Original: Since each ˜ P i is possible to be selected, when selecting
Revised: Since each ˜ P i is possible to be selected, when selecting

Original: [Equation]
Revised: [Equation]

Original: by per [Equation] forming (7) and (8), we have[Equation]
Revised: by performing (7) and (8), we have

Original: k
Revised: Q

Original: [Equation]

Original: 1 i

Original: [Equation]
Revised: [Equation]

Original: According to the proof of Lemma
Revised: According to the proof of Lemma

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: Thus, Q i ( s, a i ) converges to max a − i Q ( s, a i , a − i ) .
Revised: Thus, Q i ( s, a i ) converges to max a − i Q ( s, a i , a − i ) .

Original: According to Lemma 1 and 4, we have:
Revised: According to Lemma 1 and 4, we have:

Original: Theorem 2.
Revised: Theorem 2.

Original: The agents learn the optimal joint policy under simplified best possible operator.
Revised: The agents learn the optimal joint policy under simplified best possible operator.

Original: 2. B EST P OSSIBLE Q-L EARNING
Revised: 2. B EST P OSSIBLE Q-L EARNING

Original: Best possible Q-learning (BQL) is instantiated on simplified best possible operator.
Revised: Best possible Q-learning (BQL) is instantiated on simplified best possible operator.

Original: We first consider learning Q-table for tabular cases.
Revised: We first consider learning Q-table for tabular cases.

Original: The key challenge is how to obtain all possible transition probabilities under deterministic π − i during learning.
Revised: The key challenge is how to obtain all possible transition probabilities under deterministic π − i during learning.

Original: To solve this issue, the whole training process is divided into M epochs.
Revised: To solve this issue, the whole training process is divided into M epochs.

Original: At the epoch m , each agent i randomly and independently initializes a deterministic policy ˆ π mi and selects a subset of states S mi .
Revised: At the epoch m , each agent i randomly and independently initializes a deterministic policy ˆ π mi and selects a subset of states S mi .

Original: Then each agent i interacts with the environment using the deterministic policy
Revised: Then each agent i interacts with the environment using the deterministic policy

Original: [Equation]
Revised: [Equation]

Original: Each agent i stores independent experiences ( s, a i , s ′ , r ) in the replay buffer D mi .
Revised: Each agent i stores independent experiences ( s, a i , s ′ , r ) in the replay buffer D mi .

Original: As P i depends on π − i , and agents act deterministic policies, D mi contains one P i under a deterministic π − i .
Revised: As P i depends on π − i , and agents act deterministic policies, D mi contains one P i under a deterministic π − i .

Original: Since
Revised: Since

Original: P i will change if other agents modify their policies π − i , acting the randomly initialized policy ˆ π mi on S mi in the epoch m not only helps each agent i to explore state-action pairs, but also helps other agents to explore possible transition probabilities.
Revised: P i will change if other agents modify their policies π − i , acting the randomly initialized policy ˆ π mi on S mi in the epoch m not only helps each agent i to explore state-action pairs, but also helps other agents to explore possible transition probabilities.

Original: When M is sufficiently large, given any ( s, a i ) pair, any P i ( s, a i ) can be found in a replay buffer.
Revised: When M is sufficiently large, given any ( s, a i ) pair, any P i ( s, a i ) can be found in a replay buffer.

Original: After interaction of the epoch m , each agent i has a buffer series {D 1 i , · · · , D mi } , each of which has different transition probabilities.
Revised: After interaction of the epoch m , each agent i has a buffer series {D 1 i , · · · , D mi } , each of which has different transition probabilities.

Original: At training period of the epoch m , each agent i randomly selects one replay buffer D ji from {D 1 i , · · · , D mi } , and samples mini-batches { s, a i , s ′ , r } from D ji to update Q-table Q e i ( s, a i ) by (7), and then samples mini-batches from D ji to update Q i ( s, a i ) by (8).
Revised: At training period of the epoch m , each agent i randomly selects one replay buffer D ji from {D 1 i , · · · , D mi } , and samples mini-batches { s, a i , s ′ , r } from D ji to update Q-table Q e i ( s, a i ) by (7), and then samples mini-batches from D ji to update Q i ( s, a i ) by (8).

Original: The Q-table implementation is summarized in Algorithm 1.
Revised: The Q-table implementation is summarized in Algorithm 1.

Original: Then we analyze sample efficiency of collecting the buffer series.
Revised: Then we analyze sample efficiency of collecting the buffer series.

Original: Simplified best possible operator requires that any possible P i ( s, a i ) of ( s, a i ) pair can be found in one buffer, but does not care about the relationship between transition probabilities of different state-action pairs in the same buffer.
Revised: Simplified best possible operator requires that any possible P i ( s, a i ) of ( s, a i ) pair can be found in one buffer, but does not care about the relationship between transition probabilities of different state-action pairs in the same buffer.

Original: So
Revised: So

Original: BQL ideally needs only |A
Revised: BQL ideally needs only |A

Original: i | × |A − i | = |A| small buffers to cover all possible P i for any ( s, a i ) pair, which is very efficient for experience collection.
Revised: i | × |A − i | = |A| small buffers to cover all possible P i for any ( s, a i ) pair, which is very efficient for experience collection.

Original: We give an intuitive illustration for this and analyze that BQL has similar sample complexity to the joint Q-learning (3) in Appendix B.
Revised: We give an intuitive illustration for this and analyze that BQL has similar sample complexity to the joint Q-learning (3) in Appendix B.

Original: In complex environments with large or continuous state-action space, it is inefficient and costly to follow the experience collection in tabular cases, where the agents cannot update their policies during the interaction of each epoch, and each epoch requires adequate samples to accurately estimate the expectation (7).
Revised: In complex environments with large or continuous state-action space, it is inefficient and costly to follow the experience collection in tabular cases, where the agents cannot update their policies during the interaction of each epoch, and each epoch requires adequate samples to accurately estimate the expectation (7).

Original: Thus, in complex environments, same as IQL, each agent i only maintains one replay buffer D i , which contains all historical experiences, and uses the same ϵ -greedy policy as
Revised: Thus, in complex environments, same as IQL, each agent i only maintains one replay buffer D i , which contains all historical experiences, and uses the same ϵ -greedy policy as

Original: IQL without the randomly initialized deterministic policy ˆ π i .
Revised: IQL without the randomly initialized deterministic policy ˆ π i .

Original: Then we instantiate simplified best possible operator with neural networks Q i and Q e i .
Revised: Then we instantiate simplified best possible operator with neural networks Q i and Q e i .

Original: Q e i is updated by minimizing:
Revised: Q e i is updated by minimizing:

Original: [Equation]
Revised: [Equation]

Original: And Q i is updated by minimizing:
Revised: And Q i is updated by minimizing:

Original: [Equation]
Revised: [Equation]

Revised: else

Revised: [Equation]

Original:  ¯ Q e i is the softly updated target network of Q e i .
Revised: (10) ¯ Q e i is the softly updated target network of Q e i .

Original: When λ = 0 , (10) is equivalent to (8).
Revised: When λ = 0 , (10) is equivalent to (8).

Original: However, when λ = 0 , the positive random noise of Q i in the update can be continuously accumulated, which may cause value overestimation.
Revised: However, when λ = 0 , the positive random noise of Q i in the update can be continuously accumulated, which may cause value overestimation.

Original: So we adopt the weighted max in (10) by setting 0 < λ < 1 to offset the positive random noise.
Revised: So we adopt the weighted max in (10) by setting 0 < λ < 1 to offset the positive random noise.

Original: In continuous action space, following DDPG (Lillicrap et al., 2016), we train a policy network π i ( s ) by maximizing Q i ( s, π i ( s )) as a substitute of arg max a i Q i ( s, a i ) .
Revised: In continuous action space, following DDPG (Lillicrap et al., 2016), we train a policy network π i ( s ) by maximizing Q i ( s, π i ( s )) as a substitute of arg max a i Q i ( s, a i ) .

Original: The neural network implementation is summarized in Algorithm 2.
Revised: The neural network implementation is summarized in Algorithm 2.

Original: Simplified best possible operator is meaningful for neural network implementation.
Revised: Simplified best possible operator is meaningful for neural network implementation.

Original: As there is only one buffer D i , we cannot perform (6) but can still perform (7) and (8) on D i .
Revised: As there is only one buffer D i , we cannot perform (6) but can still perform (7) and (8) on D i .

Original: As other agents are updating their policies, the transition probabilities in D i will continuously change.
Revised: As other agents are updating their policies, the transition probabilities in D i will continuously change.

Original: If D i sufficiently goes through all possible transition probabilities, Q i ( s, a i ) converges to max a − i Q ( s, a i , a − i ) and the agents learn the optimal joint policy.
Revised: If D i sufficiently goes through all possible transition probabilities, Q i ( s, a i ) converges to max a − i Q ( s, a i , a − i ) and the agents learn the optimal joint policy.

Original: That is to say, non-stationarity in the replay buffer is no longer a drawback, but a necessary condition for BQL .
Revised: That is to say, non-stationarity in the replay buffer is no longer a drawback, but a necessary condition for BQL .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Most existing MARL methods (Lowe et al., 2017; Iqbal & Sha, 2019; Wang et al., 2020; Zhang et al., 2021b; Su & Lu, 2022; Peng et al., 2021; Li et al., 2022; Sunehag et al., 2018; Rashid et al., 2018;
Revised: Most existing MARL methods (Lowe et al., 2017; Iqbal & Sha, 2019; Wang et al., 2020; Zhang et al., 2021b; Su & Lu, 2022; Peng et al., 2021; Li et al., 2022; Sunehag et al., 2018; Rashid et al., 2018;

Original: Son et al., 2019; Wang et al., 2021a; Rashid et al., 2020) follow the paradigm of centralized training and decentralized execution (CTDE), where the information of all agents can be accessed in a centralized way during training.
Revised: Son et al., 2019; Wang et al., 2021a; Rashid et al., 2020) follow the paradigm of centralized training and decentralized execution (CTDE), where the information of all agents can be accessed in a centralized way during training.

Original: Unlike these methods, we focus on fully decentralized learning where global information is not available.
Revised: Unlike these methods, we focus on fully decentralized learning where global information is not available.

Original: The most straightforward decentralized methods, i.e. , independent Q-learning (Tan, 1993) and independent PPO (IPPO) (de Witt et al., 2020a), cannot guarantee the convergence of the learned policy, because the transition probabilities are non-stationary from
Revised: The most straightforward decentralized methods, i.e. , independent Q-learning (Tan, 1993) and independent PPO (IPPO) (de Witt et al., 2020a), cannot guarantee the convergence of the learned policy, because the transition probabilities are non-stationary from

Original: [Figure]
Revised: [Figure]

Original: the perspective of each agent, as all agents are learning policies simultaneously.
Revised: the perspective of each agent, as all agents are learning policies simultaneously.

Original: Multi-agent alternate Q-learning (MA2QL) (Su et al., 2022) guarantees the convergence to a Nash equilibrium
Revised: Multi-agent alternate Q-learning (MA2QL) (Su et al., 2022) guarantees the convergence to a Nash equilibrium

Original: [Figure]
Revised: [Figure]

Original: but the converged equilibrium may not be the optimal one when there are multiple equilibria
Revised: but the converged equilibrium may not be the optimal one when there are multiple equilibria

Original: [Figure]
Revised: [Figure]

Original: and it has to be trained in an on-policy manner and cannot use replay buffers, which leads to poor sample efficiency.
Revised: and it has to be trained in an on-policy manner and cannot use replay buffers, which leads to poor sample efficiency.

Original: Following the principle of optimistic estimation, Hysteretic IQL (Matignon et al., 2007) sets a slow learning rate to the value punishment.
Revised: Following the principle of optimistic estimation, Hysteretic IQL (Matignon et al., 2007) sets a slow learning rate to the value punishment.

Original: Distributed IQL (Lauer & Riedmiller, 2000), a special case of Hysteretic IQL with the slow learning rate being zero, guarantees the convergence to the optimum but only in deterministic environments.
Revised: Distributed IQL (Lauer & Riedmiller, 2000), a special case of Hysteretic IQL with the slow learning rate being zero, guarantees the convergence to the optimum but only in deterministic environments.

Original: Our BQL is the first fully decentralized algorithm that converges to the optimal joint policy in stochastic environments.
Revised: Our BQL is the first fully decentralized algorithm that converges to the optimal joint policy in stochastic environments.

Original: In the next section, we compare BQL against these Q-learning variants (Distributed IQL is included in Hysteretic IQL).
Revised: In the next section, we compare BQL against these Q-learning variants (Distributed IQL is included in Hysteretic IQL).

Original: Comparing with on-policy algorithms, e.g. , IPPO, that are not sample-efficient especially in fully decentralized settings, is out of focus and thus deferred to Appendix.
Revised: Comparing with on-policy algorithms, e.g. , IPPO, that are not sample-efficient especially in fully decentralized settings, is out of focus and thus deferred to Appendix.

Original: Decentralized methods with communication (Zhang et al., 2018; Konan et al., 2021) allow information sharing with neighboring agents according to a time-varying communication channel, which do not follow the fully decentralized setting and thus are beyond the scope of this paper.
Revised: Decentralized methods with communication (Zhang et al., 2018; Konan et al., 2021) allow information sharing with neighboring agents according to a time-varying communication channel, which do not follow the fully decentralized setting and thus are beyond the scope of this paper.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In experiments, we first test BQL with Q-table on randomly generated cooperative stochastic games to verify its convergence and optimality.
Revised: In experiments, we first test BQL with Q-table on randomly generated cooperative stochastic games to verify its convergence and optimality.

Original: Then, to illustrate its performance on complex tasks, we compare BQL with neural networks against Q-learning variants on MPE-version (Lowe et al., 2017) differential games,
Revised: Then, to illustrate its performance on complex tasks, we compare BQL with neural networks against Q-learning variants on MPE-version (Lowe et al., 2017) differential games,

Original: Multi-Agent MuJoCo
Revised: Multi-Agent MuJoCo

Original: (Peng et al., 2021), and SMAC (Samvelyan et al., 2019).
Revised: (Peng et al., 2021), and SMAC (Samvelyan et al., 2019).

Original: The experiments cover both fully and partially observable, deterministic and stochastic, discrete and continuous environments.
Revised: The experiments cover both fully and partially observable, deterministic and stochastic, discrete and continuous environments.

Original: Since we consider the fully decentralized setting, BQL and the baselines do not use parameter sharing .
Revised: Since we consider the fully decentralized setting, BQL and the baselines do not use parameter sharing .

Original: The results are presented using mean and standard deviation with different random seeds.
Revised: The results are presented using mean and standard deviation with different random seeds.

Original: More details about hyperparameters are available in Appendix E.
Revised: More details about hyperparameters are available in Appendix E.

Original: 4. S TOCHASTIC G AMES
Revised: 4. S TOCHASTIC G AMES

Original: To support the theoretical analysis of BQL, we test Q-table instantiation on stochastic games withagents, 30 states, and infinite horizon.
Revised: To support the theoretical analysis of BQL, we test Q-table instantiation on stochastic games withagents, 30 states, and infinite horizon.

Original: The action space of each agent is 4 , so the joint action space |A| = 256 .
Revised: The action space of each agent is 4 , so the joint action space |A| = 256 .

Original: The distribution of initial states is uniform.
Revised: The distribution of initial states is uniform.

Original: Each state will transition to any state given a joint action according to transition probabilities.
Revised: Each state will transition to any state given a joint action according to transition probabilities.

Original: The transition probabilities and reward function are randomly generated and fixed in each game.
Revised: The transition probabilities and reward function are randomly generated and fixed in each game.

Original: We randomly generate 20 games and train the agents for four different seeds in each game.
Revised: We randomly generate 20 games and train the agents for four different seeds in each game.

Original: The mean normalized return (normalized by the optimal return) and std over the 20 games are shown in Figure 1a.
Revised: The mean normalized return (normalized by the optimal return) and std over the 20 games are shown in Figure 1a.

Original: IQL cannot learn the optimal policies due to non-stationarity.
Revised: IQL cannot learn the optimal policies due to non-stationarity.

Original: Although using the optimistic update to remedy the non-stationarity, Hysteretic IQL (H-IQL) still cannot theoretically solve this problem in stochastic environments and shows similar performance to IQL.
Revised: Although using the optimistic update to remedy the non-stationarity, Hysteretic IQL (H-IQL) still cannot theoretically solve this problem in stochastic environments and shows similar performance to IQL.

Original: In Appendix A, we thoroughly analyze the difference between H-IQL and BQL, and show H-IQL is a special case of
Revised: In Appendix A, we thoroughly analyze the difference between H-IQL and BQL, and show H-IQL is a special case of

Original: BQL in deterministic environments.
Revised: BQL in deterministic environments.

Original: MA2QL guarantees the convergence to a Nash equilibrium, but the converged one may not be the optimal one, thus there is a performance gap between MA2QL and optimal policies.
Revised: MA2QL guarantees the convergence to a Nash equilibrium, but the converged one may not be the optimal one, thus there is a performance gap between MA2QL and optimal policies.

Original: BQL could converge to the optimum, and the tiny gap is caused by the fitting error of the Q-table update, which verifies the theoretical analysis.
Revised: BQL could converge to the optimum, and the tiny gap is caused by the fitting error of the Q-table update, which verifies the theoretical analysis.

Original: Note that, in Q-table instantiations, MA2QL and BQL use different experience collection from IQL, i.e. , exploration strategy and replay buffer.
Revised: Note that, in Q-table instantiations, MA2QL and BQL use different experience collection from IQL, i.e. , exploration strategy and replay buffer.

Original: MA2QL only uses on-policy experiences and BQL collects a series of small buffers.
Revised: MA2QL only uses on-policy experiences and BQL collects a series of small buffers.

Original: However, for sample efficiency, the two methods have to use the same experience collection as IQL
Revised: However, for sample efficiency, the two methods have to use the same experience collection as IQL

Original: [Figure]
Revised: [Figure]

Original: in complex tasks with neural networks.
Revised: in complex tasks with neural networks.

Original: MA2QL- and BQL- respectively denote the two methods with the same experience collection as IQL.
Revised: MA2QL- and BQL- respectively denote the two methods with the same experience collection as IQL.

Original: Trained on off-policy experiences, MA2QL- suffers from non-stationarity and achieves similar performance to IQL.
Revised: Trained on off-policy experiences, MA2QL- suffers from non-stationarity and achieves similar performance to IQL.

Original: Even if using only one buffer, as we have analyzed in Section 2.4, if the non-stationary buffer sufficiently goes through all possible transition probabilities, BQL agents can also converge to the optimum.
Revised: Even if using only one buffer, as we have analyzed in Section 2.4, if the non-stationary buffer sufficiently goes through all possible transition probabilities, BQL agents can also converge to the optimum.

Original: Although going through all possible transition probabilities by one buffer is inefficient, BQL- significantly outperforms IQL, which implies the potential of BQL with one buffer in complex tasks.
Revised: Although going through all possible transition probabilities by one buffer is inefficient, BQL- significantly outperforms IQL, which implies the potential of BQL with one buffer in complex tasks.

Original: Figure 1b shows the effect of the size of buffer D
Revised: Figure 1b shows the effect of the size of buffer D

Original: m i at the epoch m .
Revised: m i at the epoch m .

Original: If |D m i | is too small, i.e. , 200 , the experiences in |D mi | are insufficient to accurately estimate the expected value (7).
Revised: If |D m i | is too small, i.e. , 200 , the experiences in |D mi | are insufficient to accurately estimate the expected value (7).

Original: If |D mi | is too large, i.e. , 10000 , the experiences in |D mi | are redundant, and the buffer series is difficult to cover all possible transition probabilities given total training timesteps.
Revised: If |D mi | is too large, i.e. , 10000 , the experiences in |D mi | are redundant, and the buffer series is difficult to cover all possible transition probabilities given total training timesteps.

Original: Figure 1c shows the effect of the number of states on which the agents perform the randomly initialized deterministic policy ˆ π mi for exploration.
Revised: Figure 1c shows the effect of the number of states on which the agents perform the randomly initialized deterministic policy ˆ π mi for exploration.

Original: The larger | S mi | means a stronger exploration for both state-action pairs and possible transition probabilities, which leads to better performance.
Revised: The larger | S mi | means a stronger exploration for both state-action pairs and possible transition probabilities, which leads to better performance.

Original: We then consider a one-stage game that is wildly adopted in MARL (Son et al., 2019).
Revised: We then consider a one-stage game that is wildly adopted in MARL (Son et al., 2019).

Original: There areagents, and the action space of each agent is 3 .
Revised: There areagents, and the action space of each agent is 3 .

Original: The reward matrix is (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) a 1 /a
Revised: The reward matrix is (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) a 1 /a

Original: A (1)
Revised: A (1)

Original: A (2)
Revised: A (2)

Original: A (3)
Revised: A (3)

Original: A (1) −−
Revised: A (1) −−

Original: A (2) −
Revised: A (2) −

Original: A (3) −  (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) where reward 8 is the global optimum and the reward 0 is the sub-optimal Nash equilibrium.
Revised: A (3) −  (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) where reward 8 is the global optimum and the reward 0 is the sub-optimal Nash equilibrium.

Original: As shown in Figure 1d, MA2QL converges to the sub-optimal Nash equilibrium when the initial policy of the second agent selects A (2) or A (3) .
Revised: As shown in Figure 1d, MA2QL converges to the sub-optimal Nash equilibrium when the initial policy of the second agent selects A (2) or A (3) .

Original: But BQL converges to the global optimum easily.
Revised: But BQL converges to the global optimum easily.

Original: 4. MPE
Revised: 4. MPE

Original: To evaluate the effectiveness of BQL with neural network implementation, we design an MPE-based differential game, where 3 agents can move in the range [ − 1 , 1] .
Revised: To evaluate the effectiveness of BQL with neural network implementation, we design an MPE-based differential game, where 3 agents can move in the range [ − 1 , 1] .

Original: In each timestep, agent i acts the action a i ∈ [ − 1 , 1] , and the position of agent i will be updated as x i = clip ( x i +0 .
Revised: In each timestep, agent i acts the action a i ∈ [ − 1 , 1] , and the position of agent i will be updated as x i = clip ( x i +0 .

Original: 1 × a i , − 1 , 1) ( i.e. , the updated position is clipped to [ − 1 , 1] ) with the probability 1 − β , or will be updated as − x i with the probability β .
Revised: 1 × a i , − 1 , 1) ( i.e. , the updated position is clipped to [ − 1 , 1] ) with the probability 1 − β , or will be updated as − x i with the probability β .

Original: β controls the stochasticity.
Revised: β controls the stochasticity.

Original: The state is the vector of positions { x 1 , x 2 , x 3 } .
Revised: The state is the vector of positions { x 1 , x 2 , x 3 } .

Original: The reward function of each timestep r ( l ) , l = (cid:113) 23 ( x 21 + x 22 + x 23 ) is given in
Revised: The reward function of each timestep r ( l ) , l = (cid:113) 23 ( x 21 + x 22 + x 23 ) is given in

Original: Appendix E.
Revised: Appendix E.

Original: We visualize the relation between r and l in Figure 12.
Revised: We visualize the relation between r and l in Figure 15.

Original: There is only one global optimum ( l = 0 and r = 1 ) but infinite sub-optima ( l = 0 .
Revised: There is only one global optimum ( l = 0 and r = 1 ) but infinite sub-optima ( l = 0 .

Original: 8 and r = 0 . 3 ), and the narrow region with r > 0 .
Revised: 8 and r = 0 . 3 ), and the narrow region with r > 0 .

Original: 3 is surrounded by the region with r = 0 .
Revised: 3 is surrounded by the region with r = 0 .

Original: So it is quite a challenge to learn the optimal policies in a fully decentralized way.
Revised: So it is quite a challenge to learn the optimal policies in a fully decentralized way.

Original: Each episode contains 100 timesteps, and the initial positions follow the uniform distribution.
Revised: Each episode contains 100 timesteps, and the initial positions follow the uniform distribution.

Original: We perform experiments with different stochasticities β , and train the agents for eight seeds with each β .
Revised: We perform experiments with different stochasticities β , and train the agents for eight seeds with each β .

Original: In continuous environments, BQL and baselines are built on DDPG.
Revised: In continuous environments, BQL and baselines are built on DDPG.

Original: As shown in Figure 2, IQL always falls into the local optimum (total reward ≈ 30 ) because of the non-stationary transition probabilities.
Revised: As shown in Figure 2, IQL always falls into the local optimum (total reward ≈ 30 ) because of the non-stationary transition probabilities.

Original: H-IQL only escapes the local optimum in one seed in the setting with β = 0 .
Revised: H-IQL only escapes the local optimum in one seed in the setting with β = 0 .

Original: In neural network implementations, MA2QL and BQL use the same experience collection as IQL, so there is no MA2QL- and BQL-.
Revised: In neural network implementations, MA2QL and BQL use the same experience collection as IQL, so there is no MA2QL- and BQL-.

Original: MA2QL converges to the local optimum because it cannot guarantee that the converged equilibrium is the global optimum, especially trained using off-policy data.
Revised: MA2QL converges to the local optimum because it cannot guarantee that the converged equilibrium is the global optimum, especially trained using off-policy data.

Original: BQL ( λ = 0 . 01 ) can escape from local optimum in more than 4 seeds in all settings, which demonstrates the effectiveness of our optimization objectives (9) and (10).
Revised: BQL ( λ = 0 . 01 ) can escape from local optimum in more than 4 seeds in all settings, which demonstrates the effectiveness of our optimization objectives (9) and (10).

Original: The difference between global optimum (total reward ≈ 100 ) and local optimum is large, which results
Revised: The difference between global optimum (total reward ≈ 100 ) and local optimum is large, which results

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: in the large variance of BQL.
Revised: in the large variance of BQL.

Original: In the objective (10), λ controls the balance between performing best possible operator and offsetting the overestimation caused by the operator.
Revised: In the objective (10), λ controls the balance between performing best possible operator and offsetting the overestimation caused by the operator.

Original: As shown in Figure 2, the large λ , i.e. , 0 . 1 , will weaken the strength of BQL, while too small λ , i.e. , 0 , will cause severe overestimation and destroy the performance.
Revised: As shown in Figure 2, the large λ , i.e. , 0 . 1 , will weaken the strength of BQL, while too small λ , i.e. , 0 , will cause severe overestimation and destroy the performance.

Original: 4. M ULTI -A GENT M U J O C O
Revised: 4. M ULTI -A GENT M U J O C O

Original: To evaluate BQL in partially observable environments, we adopt Multi-Agent MuJoCo
Revised: To evaluate BQL in partially observable environments, we adopt Multi-Agent MuJoCo

Original: (Peng et al., 2021), where each agent independently controls one or some joints of the robot and can only observe the state of its own joints and bodies (with the parameter agent obsk = 0).
Revised: (Peng et al., 2021), where each agent independently controls one or some joints of the robot and can only observe the state of its own joints and bodies (with the parameter agent obsk = 0).

Original: In each task, we test four random seeds and plot the learning curves in Figure 3.
Revised: In each task, we test four random seeds and plot the learning curves in Figure 3.

Original: BQL achieves higher reward or learns faster than the baselines, which verifies that BQL could be applied to partially observable environments.
Revised: BQL achieves higher reward or learns faster than the baselines, which verifies that BQL could be applied to partially observable environments.

Original: Here, we set λ = 0 .
Revised: Here, we set λ = 0 .

Original: 5 and show that BQL is robust to λ in Appendix C.1.
Revised: 5 and show that BQL is robust to λ in Appendix C.1.

Original: In partially observable environments, BQL is performed on transition probabilities of observation P i
Revised: In partially observable environments, BQL is performed on transition probabilities of observation P i

Original: ( o ′ i
Revised: ( o ′ i

Original: | o
Revised: | o

Original: i , a i ) , which also depends on π − i .
Revised: i , a i ) , which also depends on π − i .

Original: The convergence and optimality of BQL can only be guaranteed when one observation o i uniquely corresponds to one state s .
Revised: The convergence and optimality of BQL can only be guaranteed when one observation o i uniquely corresponds to one state s .

Original: It has been proven that the optimality is undecidable in partially observable Markov decision processes (POMDPs) (Madani et al., 1999), so it is not the limitation of BQL.
Revised: It has been proven that the optimality is undecidable in partially observable Markov decision processes (POMDPs) (Madani et al., 1999), so it is not the limitation of BQL.

Original: We only consider two-agent cases in the partially observable setting, because the too limited observation range cannot support strong policies when there are more agents.
Revised: We only consider two-agent cases in the partially observable setting, because the too limited observation range cannot support strong policies when there are more agents.

Original: In
Revised: In

Original: Appendix C.2, we test BQL on 17 -agent
Revised: Appendix C.2, we test BQL on 17 -agent

Original: Humanoid with full observation to verify the scalability.
Revised: Humanoid with full observation to verify the scalability.

Original: 4. SMAC
Revised: 4. SMAC

Original: We also perform experiments on partially observable and stochastic SMAC tasks (Samvelyan et al., 2019) with the version SC2.4.10, including both easy and hard maps (Yu et al., 2021).
Revised: We also perform experiments on partially observable and stochastic SMAC tasks (Samvelyan et al., 2019) with the version SC2.4.10, including both easy and hard maps (Yu et al., 2021).

Original: Agent numbers vary between 2 and 9 .
Revised: Agent numbers vary between 2 and 9 .

Original: We build BQL on the implementation of PyMARL (Samvelyan et al., 2019) and train the agents for four random seeds.
Revised: We build BQL on the implementation of PyMARL (Samvelyan et al., 2019) and train the agents for four random seeds.

Original: The learning curves are shown in Figure 4.
Revised: The learning curves are shown in Figure 4.

Original: BQL also outperforms the baselines, which verifies that BQL can also obtain performance gain in highdimensional complex tasks.
Revised: BQL also outperforms the baselines, which verifies that BQL can also obtain performance gain in highdimensional complex tasks.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose best possible operator and theoretically prove that the policies of agents will converge to the optimal joint policy if each agent independently updates its individual state-action value by the operator.
Revised: We propose best possible operator and theoretically prove that the policies of agents will converge to the optimal joint policy if each agent independently updates its individual state-action value by the operator.

Original: We then simplify the operator and derive BQL, the first decentralized MARL algorithm that guarantees the convergence to the global optimum in stochastic environments.
Revised: We then simplify the operator and derive BQL, the first decentralized MARL algorithm that guarantees the convergence to the global optimum in stochastic environments.

Original: Empirically, BQL outperforms baselines in a variety of multi-agent tasks.
Revised: Empirically, BQL outperforms baselines in a variety of multi-agent tasks.

Original: We believe BQL will be a new paradigm for fully decentralized learning.
Revised: We believe BQL will be a new paradigm for fully decentralized learning.

Original: A C OMPARISON WITH H YSTERETIC IQL
Revised: A C OMPARISON WITH H YSTERETIC IQL

Original: Hysteretic IQL is a special case of BQL when the environment is deterministic.
Revised: Hysteretic IQL is a special case of BQL when the environment is deterministic.

Original: To thoroughly illustrate that, we rewrite the loss function of BQL
Revised: To thoroughly illustrate that, we rewrite the loss function of BQL

Original: [Equation]
Revised: [Equation]

Original: λ else .
Revised: λ else .

Original: If λ = 0 , the update of BQL is
Revised: If λ = 0 , the update of BQL is

Original: [Equation]
Revised: [Equation]

Original: Hysteretic IQL follows the loss function
Revised: Hysteretic IQL follows the loss function

Original: [Equation]
Revised: [Equation]

Original: λ else .
Revised: λ else .

Original: If λ = 0 , Hysteretic IQL degenerates into Distributed IQL (Lauer & Riedmiller, 2000)
Revised: If λ = 0 , Hysteretic IQL degenerates into Distributed IQL (Lauer & Riedmiller, 2000)

Original: [Equation]
Revised: [Equation]

Original: BQL takes the max of the expected target on transition probability ˜ P
Revised: BQL takes the max of the expected target on transition probability ˜ P

Original: i ( s ′ | s, a i ) , while Hysteretic
Revised: i ( s ′ | s, a i ) , while Hysteretic

Original: IQL takes the max of the target on the next state s ′ .
Revised: IQL takes the max of the target on the next state s ′ .

Original: When the environment is deterministic, they are equivalent.
Revised: When the environment is deterministic, they are equivalent.

Original: However, in stochastic environments, Hysteretic IQL cannot guarantee to converge to the global optimum since the environment will not always transition to the same s ′ .
Revised: However, in stochastic environments, Hysteretic IQL cannot guarantee to converge to the global optimum since the environment will not always transition to the same s ′ .

Original: BQL can guarantee the global optimum in both deterministic and stochastic environments.
Revised: BQL can guarantee the global optimum in both deterministic and stochastic environments.

Original: B E FFICIENCY OF BQL
Revised: B E FFICIENCY OF BQL

Original: [Figure]
Revised: [Figure]

Original: We will discuss the efficiency of collecting the replay buffer for BQL.
Revised: We will discuss the efficiency of collecting the replay buffer for BQL.

Original: The space of other agents’ policies π − i given ( s, a i ) pair is a convex polytope.
Revised: The space of other agents’ policies π − i given ( s, a i ) pair is a convex polytope.

Original: For clarity, Figure 5 shows a triangle space.
Revised: For clarity, Figure 5 shows a triangle space.

Original: Each π − i corresponds to a P i ( s ′ | s, a i ) .
Revised: Each π − i corresponds to a P i ( s ′ | s, a i ) .

Original: Deterministic policies π − i locate at the vertexes, while the edges and the inside of the polytope are stochastic π − i , the mix of deterministic ones.
Revised: Deterministic policies π − i locate at the vertexes, while the edges and the inside of the polytope are stochastic π − i , the mix of deterministic ones.

Original: Since
Revised: Since

Original: BQL only considers deterministic policies, the buffer series only needs to cover all the vertexes by acting deterministic policies in the collection of each buffer D mi , which is efficient.
Revised: BQL only considers deterministic policies, the buffer series only needs to cover all the vertexes by acting deterministic policies in the collection of each buffer D mi , which is efficient.

Original: BQL needs only |A
Revised: BQL needs only |A

Original: i | × |A − i | = |A| small buffers, which is irrelevant to state space |S| , to meet the requirement of simplified best possible operator that any one of possible P
Revised: i | × |A − i | = |A| small buffers, which is irrelevant to state space |S| , to meet the requirement of simplified best possible operator that any one of possible P

Original: i ( s ′ | s, a i ) can be found in one (ideally only one) buffer given ( s, a i ) pair.
Revised: i ( s ′ | s, a i ) can be found in one (ideally only one) buffer given ( s, a i ) pair.

Original: More specifically, |A i | buffers are needed to cover action space, and |A − i | buffers are needed to cover transition space for each action.
Revised: More specifically, |A i | buffers are needed to cover action space, and |A − i | buffers are needed to cover transition space for each action.

Original: We intuitively illustrate this in Figure 6.
Revised: We intuitively illustrate this in Figure 6.

Original: Each state in
Revised: Each state in

Original: D mi requires # samples to estimate the expectation in (7), so the sample complexity is O ( |A|| S | #) .
Revised: D mi requires # samples to estimate the expectation in (7), so the sample complexity is O ( |A|| S | #) .

Original: For the joint Q-learning (3), the most efficient known method to guarantee the convergence and optimality in stochastic environments, each state-joint action pair
Revised: For the joint Q-learning (3), the most efficient known method to guarantee the convergence and optimality in stochastic environments, each state-joint action pair

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: BQLJQL
Revised: BQLJQL

Original: ( s, a ) requires # samples to estimate the expectation, so the sample complexity is also O ( |A|| S | #) .
Revised: ( s, a ) requires # samples to estimate the expectation, so the sample complexity is also O ( |A|| S | #) .

Original: Thus, BQL is close to the joint Q-learning in terms of sample complexity, which is empirically verified in Figure 7.
Revised: Thus, BQL is close to the joint Q-learning in terms of sample complexity, which is empirically verified in Figure 7.

Original: One may ask “since you obtain all possible transition probabilities, why not perform IQL on each transition probability and choose the highest value?”
Revised: One may ask “since you obtain all possible transition probabilities, why not perform IQL on each transition probability and choose the highest value?”

Original: Actually, this naive algorithm can also learn the optimal policy, but the buffer collection of the naive algorithm is much more costly than that of BQL.
Revised: Actually, this naive algorithm can also learn the optimal policy, but the buffer collection of the naive algorithm is much more costly than that of BQL.

Original: The naive algorithm requires that any one of possible transition probability functions of the whole state-action space could be found in one buffer, which needs |A − i | |S| buffers.
Revised: The naive algorithm requires that any one of possible transition probability functions of the whole state-action space could be found in one buffer, which needs |A − i | |S| buffers.

Original: And training IQL |A
Revised: And training IQL |A

Original: − i | |S| times is also formidable.
Revised: − i | |S| times is also formidable.

Original: BQL only requires that any one of possible transition probability of any state-action pair could be found in one buffer, which is much more efficient.
Revised: BQL only requires that any one of possible transition probability of any state-action pair could be found in one buffer, which is much more efficient.

Original: However, considering sample efficiency, BQL with neural networks only maintains one replay buffer
Revised: However, considering sample efficiency, BQL with neural networks only maintains one replay buffer

Original: D i containing all historical experiences, which is the same as IQL.
Revised: D i containing all historical experiences, which is the same as IQL.

Original: P i in D
Revised: P i in D

Original: i corresponds to the average of other agents’ historical policies, which is stochastic.
Revised: i corresponds to the average of other agents’ historical policies, which is stochastic.

Original: Therefore, to guarantee the optimality, in theory, BQL with one buffer has to go through almost the whole π − i space, which is costly.
Revised: Therefore, to guarantee the optimality, in theory, BQL with one buffer has to go through almost the whole π − i space, which is costly.

Original: As shown in Figure 1d, BQL- (with one buffer) outperforms IQL but cannot achieve similar results as
Revised: As shown in Figure 1d, BQL- (with one buffer) outperforms IQL but cannot achieve similar results as

Original: BQL (with buffer series), showing that maintaining one buffer is costly but still effective.
Revised: BQL (with buffer series), showing that maintaining one buffer is costly but still effective.

Original: In neural network instantiation, we show the results of BQL with the buffer series in Figure 8.
Revised: In neural network instantiation, we show the results of BQL with the buffer series in Figure 8.

Original: Due to sample efficiency, the buffer series cannot achieve strong performance, and maintaining one buffer like IQL is a better choice in complex environments.
Revised: Due to sample efficiency, the buffer series cannot achieve strong performance, and maintaining one buffer like IQL is a better choice in complex environments.

Original: [Figure]
Revised: [Figure]

Original: C A DDITIONAL R ESULTS C. H YPERPARAMETER λ
Revised: C A DDITIONAL R ESULTS C. H YPERPARAMETER λ

Original: In the objective (10), λ controls the balance between performing best possible operator and offsetting the overestimation caused by the operator.
Revised: In the objective (10) [Figure] λ controls the balance between performing best possible operator and offsetting the overestimation caused by the operator.

Original: We investigate the effectiveness of λ in Figure 9.
Revised: We investigate the effectiveness of λ in Figure 9.

Original: Too large λ will weaken the strength of BQL.
Revised: Too large λ will weaken the strength of BQL.

Original: When λ
Revised: When λ

Original: = 1 . 0 , BQL degenerates into IQL.
Revised: = 1 . 0 , BQL degenerates into IQL.

Original: Too small λ , i.e. , 0 , will cause severe overestimation and destroy the performance.
Revised: Too small λ , i [Figure] e. , 0 , will cause severe overestimation and destroy the performance.

Original: When λ falls within the interval [0 . 2 , 0 . 8] , BQL could obtain performance gain, showing the robustness to λ .
Revised: When λ falls within the interval [0 . 2 , 0 . 8] , BQL could obtain performance gain, showing the robustness to λ .

Original: [Figure]
Revised: [Figure]

Original: C. S CALABILITY
Revised: C. S CALABILITY

Original: We test BQL on 17 -agent
Revised: We test BQL on 17 -agent

Original: Humanoid with full observation, the results are shown in Figure 10.
Revised: Humanoid with full observation, the results are shown in Figure 10.

Original: BQL obtains significant performance gain in this many-agent task, which can be evidence of the good scalability of BQL
Revised: BQL obtains significant performance gain in this many-agent task, which can be evidence of the good scalability of BQL

Original: [Figure]
Revised: [Figure]

Original: C. O THER B ASE A LGORITHMS
Revised: C. O THER B ASE A LGORITHMS

Original: Besides DDPG, BQL could also be built on other variants of Q-learning, e.g. , SAC.
Revised: Besides DDPG, BQL could also be built on other variants of Q-learning, e.g. , SAC.

Original: Figureshows that BQL could also obtain performance gain on independent SAC.
Revised: Figureshows that BQL could also obtain performance gain on independent SAC.

Original: Independent PPO (IPPO) (de Witt et al., 2020a) is an on-policy decentralized MARL baseline.
Revised: Independent PPO (IPPO) (de Witt et al., 2020a) is an on-policy decentralized MARL baseline.

Original: IPPO is not a Q-learning method so it cannot be the base algorithm of BQL.
Revised: IPPO is not a Q-learning method so it cannot be the base algorithm of BQL.

Original: On-policy algorithms do not use old experiences, which makes them weak on sample efficiency (Achiam, 2018) especially in fully decentralized settings as shown in Figure 11.
Revised: On-policy algorithms do not use old experiences, which makes them weak on sample efficiency (Achiam, 2018) especially in fully decentralized settings as shown in Figure 11.

Original: [Figure]
Revised: [Figure]

Revised: BQL can also be naturally and theoretically applied on MA2QL.

Revised: We test BQL+MA2QL on three mujoco tasks used in MA2QL.

Revised: Figure 12 shows BQL could obtain performance gain on MA2QL, especially on HalfCheetah.

Original: [Figure]
Revised: [Figure]

Revised: We also compare BQL against MA2QL in more SMAC tasks.

Revised: As shown in Figure 13, BQL and

Revised: MA2QL achieve similar performance

Revised: [Figure]

Revised: and the converged results are very close to some CTDE methods

Revised: [Figure]

Revised: indicating that non-stationarity is not a severe problem in these two tasks

Revised: [Figure]

Original: D M ULTIPLE O PTIMAL J OINT P OLICIES
Revised: D M ULTIPLE O PTIMAL J OINT P OLICIES

Original: We assume that there is only one optimal joint policy.
Revised: We assume that there is only one optimal joint policy.

Original: With multiple optimal actions (with the max Q i ( s, a i ) ), if each agent arbitrarily selects one of the optimal independent actions, the joint action might not be optimal.
Revised: With multiple optimal actions (with the max Q i ( s, a i ) ), if each agent arbitrarily selects one of the optimal independent actions, the joint action might not be optimal.

Original: To address this, we set a performance tolerance ε and introduce a fixed randomly initialized reward function ˆ r ( s, s ′ ) ∈ (0 , (1 − γ ) ε ] .
Revised: To address this, we set a performance tolerance ε and introduce a fixed randomly initialized reward function ˆ r ( s, s ′ ) ∈ (0 , (1 − γ ) ε ] .

Original: Then all agents perform BQL to learn ˆ Q i ( s, a i ) of the shaped reward r + ˆ r .
Revised: Then all agents perform BQL to learn ˆ Q i ( s, a i ) of the shaped reward r + ˆ r .

Original: Since ˆ r > 0 , ˆ Q i ( s, a i ) > Q i ( s, a i ) .
Revised: Since ˆ r > 0 , ˆ Q i ( s, a i ) > Q i ( s, a i ) .

Original: In ˆ Q i ( s, a i ) , the maximal contribution from ˆ r is (1 − γ ) ε/ (1 − γ ) =
Revised: In ˆ Q i ( s, a i ) , the maximal contribution from ˆ r is (1 − γ ) ε/ (1 − γ ) =

Original: ε , so the minimal contribution from r is ˆQ i ( s, a i ) −
Revised: ε , so the minimal contribution from r is ˆ

Original:  ε > Q i ( s, a i ) − ε , which means that the maximal performance drop is ε when selecting actions according to ˆ Q i .
Revised: Q i ( s, a i ) − ε > Q i ( s, a i ) − ε , which means that the maximal performance drop is ε when selecting actions according to ˆ Q i .

Original: It is a small probability event to find multiple optimal joint policies on the reward function r + ˆ r , because ˆ r ( s, s ′ ) is randomly initialized.
Revised: It is a small probability event to find multiple optimal joint policies on the reward function r + ˆ r , because ˆ r ( s, s ′ ) is randomly initialized.

Original: Therefore, if ε is set to be small enough, BQL could solve the task with multiple optimal joint policies.
Revised: Therefore, if ε is set to be small enough, BQL could solve the task with multiple optimal joint policies.

Original: However, this technique is introduced to only remedy the assumption for our theoretical results.
Revised: However, this technique is introduced to only remedy the assumption for our theoretical results.

Original: Empirically, this is not required, because there is usually only one optimal joint policy in complex environments.
Revised: Empirically, this is not required, because there is usually only one optimal joint policy in complex environments.

Original: In all experiments, we do not use the randomly initialized reward function for BQL and other baselines, thus the comparison is fair.
Revised: In all experiments, we do not use the randomly initialized reward function for BQL and other baselines, thus the comparison is fair.

Revised: We test the randomly initialized reward function on a one-stage matrix game with two optimal joint policies (1 , 2) and (2 , 1) , as shown in Figure 14.

Revised: If the agents independently select actions, they Published as a conference paper at ICLR

Revised: [Figure]

Revised: Proposition 2.

Original: E H YPERPARAMETERS
Revised: Given the universal function approximation of neural networks, the action-value function class that QPLEX can realize is equivalent to what is induced by the IGM principle. In practice, QPLEX can utilize common neural network structures (e.g., multi-head attention modules) to achieve superior performance by approximating the universal approximation theorem (Csáji et al., 2001). We will discuss the effects of QPLEX’s duplex dueling network with different conﬁgurations in Section 4.1. As introduced by Son et al.

Revised: (2019) and Wang et al.

Revised: (2020a), the completeness of value factorization is very critical for multi-agent Q-learning and we will illustrate the stability and state-of-the-art performance of QPLEX in online and ofﬂine data collections in the next section.

Revised: 4 E XPERIMENTS In this section, we ﬁrst study didactic examples proposed by prior work (Son et al., 2019; Wang et al., 2020a) to investigate the effects of QPLEX’s complete IGM expressiveness on learning optimality and stability.

Revised: To demonstrate scalability on complex MARL domains, we also evaluate the performance of QPLEX on a range of StarCraft II benchmark tasks (Samvelyan et al., 2019).

Revised: The completeness of the IGM function class can express richer joint action-value function classes induced by large and diverse datasets or training buffers.

Revised: [Table]

Revised: also successfully converge to optimal joint action-value functions.

Revised: These results are deferred to Appendix C. QTRAN

Revised: [Figure]

Revised: [Figure]

Revised: might choose the miscoordinated joint policies (1 , 1) and (2 , 2) .

Revised: IQL cannot converge, but BQL agents always select coordinated actions, though the value gap between the optimal policy and suboptimal policy is so small, which verifies the effectiveness of the randomly initialized reward.

Revised: E H YPERPARAMETERS

Original: In MPE-based differential games, the reward function is defined as
Revised: In MPE-based differential games, the reward function is defined as

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Original:  And the relationship between r and l is visualized in Figure 12.
Revised: [Figure] [Equation] [Figure] And the relationship between r and l is visualized in Figure 15.

Original: In 2 × 3 Swimmer, there are two agents and each of them controls 3 joints of ManyAgent Swimmer.
Revised: In 2 × 3 Swimmer, there are two agents and each of them controls 3 joints of ManyAgent Swimmer.

Original: In 6 | 2 Ant, there are two agents.
Revised: In 6 | 2 Ant, there are two agents.

Original: One of them controls 6 joints, and one of them controls 2 joints.
Revised: One of them controls 6 joints, and one of them controls 2 joints.

Original: And so on.
Revised: And so on.

Original: In MPE-based differential games and Multi-Agent MuJoCo, we adopt SpinningUp (Achiam, 2018) implementation, the SOTA implementation of DDPG, and follow all hyperparameters in SpinningUp.
Revised: In MPE-based differential games and Multi-Agent MuJoCo, we adopt SpinningUp (Achiam, 2018) implementation, the SOTA implementation of DDPG, and follow all hyperparameters in SpinningUp.

Original: The discount factor γ = 0 .
Revised: The discount factor γ = 0 .

Original: 99 , the learning rate is 0 .
Revised: 99 , the learning rate is 0 .

Original: 001 with Adam optimizer, the batch size is 100 , the replay buffer contains 5 × 10 5 transitions, the hidden units are 256 .
Revised: 001 with Adam optimizer, the batch size is 100 , the replay buffer contains 5 × 10 5 transitions, the hidden units are 256 .

Original: In SMAC, we adopt PyMARL (Samvelyan et al., 2019) implementation and follow all hyperparameters in PyMARL.
Revised: In SMAC, we adopt PyMARL (Samvelyan et al., 2019) implementation and follow all hyperparameters in PyMARL.

Original: The discount factor γ = 0 .
Revised: The discount factor γ = 0 .

Original: 99 , the learning rate is 0 . 0005 with RMSprop optimizer, the batch size is 32 episodes, the replay buffer contains 5000 episodes, the hidden units are 64 .
Revised: 99 , the learning rate is 0 . 0005 with RMSprop optimizer, the batch size is 32 episodes, the replay buffer contains 5000 episodes, the hidden units are 64 .

Original: We adopt the version SC2.4.10 of SMAC.
Revised: We adopt the version SC2.4.10 of SMAC.

Original: In MPE-based differential games, we set λ = 0 .
Revised: In MPE-based differential games, we set λ = 0 .

Original: In Multi-Agent MuJoCo, we set λ = 0 .
Revised: In Multi-Agent MuJoCo, we set λ = 0 .

Original: 5 , and in SMAC, we set λ = 0 .
Revised: 5 , and in SMAC, we set λ = 0 .

Original: The experiments are carried out on Intel i7-8700 CPU and NVIDIA GTX 1080Ti
Revised: The experiments are carried out on Intel i7-8700 CPU and NVIDIA GTX 1080Ti

Original: GPU.
Revised: GPU.

Original: The training of each MPE and Multi-Agent MuJoCo task could be finished in 5 hours, and the training of each
Revised: The training of each MPE and Multi-Agent MuJoCo task could be finished in 5 hours, and the training of each

Original: SMAC task could be finished in 20 hours.
Revised: SMAC task could be finished in 20 hours.