Original: P LAN
Revised: P LAN

Original: B ETTER
Revised: B ETTER

Original: A MID
Revised: A MID

Original: C ONSERVATISM :
Revised: C ONSERVATISM :

Original: O FFLINE
Revised: O FFLINE

Original: M ULTI -A GENT
Revised: M ULTI -A GENT

Original: R EINFORCEMENT
Revised: R EINFORCEMENT

Original: L EARNING
Revised: L EARNING

Original: WITH
Revised: WITH

Original: A CTOR R ECTIFICATION
Revised: A CTOR R ECTIFICATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The idea of conservatism has led to signiﬁcant progress in ofﬂine reinforcement learning (RL) where an agent learns from pre-collected datasets.
Revised: The idea of conservatism has led to signiﬁcant progress in ofﬂine reinforcement learning (RL) where an agent learns from pre-collected datasets.

Original: However, it is still an open question to resolve ofﬂine RL in the more practical multi-agent setting as many real-world scenarios involve interaction among multiple agents.
Revised: However, it is still an open question to resolve ofﬂine RL in the more practical multi-agent setting as many real-world scenarios involve interaction among multiple agents.

Original: Given the recent success of transferring online RL algorithms to the multi-agent setting, one may expect that ofﬂine RL algorithms will also transfer to multi-agent settings directly.
Revised: Given the recent success of transferring online RL algorithms to the multi-agent setting, one may expect that ofﬂine RL algorithms will also transfer to multi-agent settings directly.

Original: Surprisingly, when conservatism-based algorithms are applied to the multi-agent setting, the performance degrades signiﬁcantly with an increasing number of agents.
Revised: Surprisingly, when conservatism-based algorithms are applied to the multi-agent setting, the performance degrades signiﬁcantly with an increasing number of agents.

Original: Towards mitigating the degradation, we identify that a key issue that the landscape of the value function can be non-concave and policy gradient improvements are prone to local optima.
Revised: Towards mitigating the degradation, we identify that a key issue that the landscape of the value function can be non-concave and policy gradient improvements are prone to local optima.

Original: Multiple agents exacerbate the problem since the suboptimal policy by any agent could lead to uncoordinated global failure.
Revised: Multiple agents exacerbate the problem since the suboptimal policy by any agent could lead to uncoordinated global failure.

Original: Following this intuition, we propose a simple yet effective method, O fﬂine
Revised: Following this intuition, we propose a simple yet effective method, O fﬂine

Original: Multi-Agent RL with Actor Rectiﬁcation (OMAR), to tackle this critical challenge via an effective combination of ﬁrst-order policy gradient and zeroth-order optimization methods for the actor to better optimize the conservative value function.
Revised: Multi-Agent RL with Actor Rectiﬁcation (OMAR), to tackle this critical challenge via an effective combination of ﬁrst-order policy gradient and zeroth-order optimization methods for the actor to better optimize the conservative value function.

Original: Despite the simplicity, OMAR signiﬁcantly outperforms strong baselines with state-of-the-art performance in multi-agent continuous control benchmarks.
Revised: Despite the simplicity, OMAR signiﬁcantly outperforms strong baselines with state-of-the-art performance in multi-agent continuous control benchmarks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Ofﬂine reinforcement learning (RL) has shown great potential in advancing the deployment of RL in real-world tasks where interaction with the environment is prohibitive, costly, or risky (Thomas, 2015).
Revised: Ofﬂine reinforcement learning (RL) has shown great potential in advancing the deployment of RL in real-world tasks where interaction with the environment is prohibitive, costly, or risky (Thomas, 2015).

Original: Since an agent has to learn from a given pre-collected dataset in ofﬂine RL, it becomes challenging for regular online RL algorithms such as DDPG (Lillicrap et al., 2016) and TD3 (Fujimoto et al., 2018) due to extrapolation error (Lee et al., 2021).
Revised: Since an agent has to learn from a given pre-collected dataset in ofﬂine RL, it becomes challenging for regular online RL algorithms such as DDPG (Lillicrap et al., 2016) and TD3 (Fujimoto et al., 2018) due to extrapolation error (Lee et al., 2021).

Original: There has been recent progress in tackling the problem based on conservatism.
Revised: There has been recent progress in tackling the problem based on conservatism.

Original: Behavior regularization (Wu et al., 2019; Kumar et al., 2019), e.g. , TD3 with Behavior Cloning (TD3+BC) (Fujimoto & Gu, 2021), compels the learning policy to stay close to the manifold of the datasets.
Revised: Behavior regularization (Wu et al., 2019; Kumar et al., 2019), e.g. , TD3 with Behavior Cloning (TD3+BC) (Fujimoto & Gu, 2021), compels the learning policy to stay close to the manifold of the datasets.

Original: Yet, its performance highly depends on the quality of the dataset.
Revised: Yet, its performance highly depends on the quality of the dataset.

Original: Another line of research investigates incorporating conservatism into the value function by critic regularization (Nachum et al., 2019;
Revised: Another line of research investigates incorporating conservatism into the value function by critic regularization (Nachum et al., 2019;

Original: Kostrikov et al., 2021), e.g. , Conservative Q-Learning (Kumar et al., 2020), which usually learns a conservative estimate of the value function to directly address the extrapolation error.
Revised: Kostrikov et al., 2021), e.g. , Conservative Q-Learning (Kumar et al., 2020), which usually learns a conservative estimate of the value function to directly address the extrapolation error.

Original: However, many practical scenarios involve multiple agents, e.g. , multi-robot control (Amato, 2018), autonomous driving (Pomerleau, 1989; Sadigh et al., 2016).
Revised: However, many practical scenarios involve multiple agents, e.g. , multi-robot control (Amato, 2018), autonomous driving (Pomerleau, 1989; Sadigh et al., 2016).

Original: Therefore, ofﬂine multi-agent reinforcement learning (MARL) (Yang et al., 2021; Jiang & Lu, 2021) is crucial for solving real-world tasks.
Revised: Therefore, ofﬂine multi-agent reinforcement learning (MARL) (Yang et al., 2021; Jiang & Lu, 2021) is crucial for solving real-world tasks.

Original: Observing recent success of Independent PPO (de Witt et al., 2020) and Multi-Agent PPO (Yu et al., 2021), both of which are based on the PPO (Schulman et al., 2017) algorithm, we ﬁnd that online RL algorithms can be transferred to multi-agent scenarios through either decentralized training or a centralized value function without bells and whistles.
Revised: Observing recent success of Independent PPO (de Witt et al., 2020) and Multi-Agent PPO (Yu et al., 2021), both of which are based on the PPO (Schulman et al., 2017) algorithm, we ﬁnd that online RL algorithms can be transferred to multi-agent scenarios through either decentralized training or a centralized value function without bells and whistles.

Original: Hence, we naturally expect that ofﬂine
Revised: Hence, we naturally expect that ofﬂine

Original: RL algorithms would also transfer easily when applied to multi-agent tasks.
Revised: RL algorithms would also transfer easily when applied to multi-agent tasks.

Original: Surprisingly, we observe that the performance of the state-of-the-art conservatism-based CQL (Kumar et al., 2020) algorithm in ofﬂine RL degrades dramatically with an increasing number of agents as shown in Figure 1(c) in our experiments.
Revised: Surprisingly, we observe that the performance of the state-of-the-art conservatism-based CQL (Kumar et al., 2020) algorithm in ofﬂine RL degrades dramatically with an increasing number of agents as shown in Figure 1(c) in our experiments.

Original: Towards mitigating the degradation, we identify a critical issue in CQL: solely regularizing the critic is insufﬁcient for multiple agents to learn good policies for coordination in the ofﬂine setting.
Revised: Towards mitigating the degradation, we identify a critical issue in CQL: solely regularizing the critic is insufﬁcient for multiple agents to learn good policies for coordination in the ofﬂine setting.

Original: The primary cause is that ﬁrst-order policy gradient methods are prone to local optima (Nachum et al., 2016;
Revised: The primary cause is that ﬁrst-order policy gradient methods are prone to local optima (Nachum et al., 2016;

Original: Ge et al., 2017; Safran & Shamir, 2017), saddle points (Vlatakis-Gkaragkounis et al., 2019; Sun et al., 2020), or noisy gradient estimates (Such et al., 2017).
Revised: Ge et al., 2017; Safran & Shamir, 2017), saddle points (Vlatakis-Gkaragkounis et al., 2019; Sun et al., 2020), or noisy gradient estimates (Such et al., 2017).

Original: As a result, this can lead to uncoordinated suboptimal learning behavior because the actor cannot leverage the global information in the critic well.
Revised: As a result, this can lead to uncoordinated suboptimal learning behavior because the actor cannot leverage the global information in the critic well.

Original: The issue is exacerbated more in the multi-agent settings due to the exponentially-sized joint action space (Yang et al., 2021) as well as the nature of the setting that requires each of the agent to learn a good policy for a successful joint policy.
Revised: The issue is exacerbated more in the multi-agent settings due to the exponentially-sized joint action space (Yang et al., 2021) as well as the nature of the setting that requires each of the agent to learn a good policy for a successful joint policy.

Original: For example, in a basketball game, where there are two competing teams each consisting of ﬁve players.
Revised: For example, in a basketball game, where there are two competing teams each consisting of ﬁve players.

Original: When one of the players passes the ball among them, it is important for all teammates to perform their duties well in their roles to win the game.
Revised: When one of the players passes the ball among them, it is important for all teammates to perform their duties well in their roles to win the game.

Original: As a result, if one of the agents in the team fails to learn a good policy, it can fail to cooperate with other agents for coordinated behaviors and lose the ball.
Revised: As a result, if one of the agents in the team fails to learn a good policy, it can fail to cooperate with other agents for coordinated behaviors and lose the ball.

Original: In this paper, we propose a surprisingly simple yet effective method for ofﬂine multi-agent continuous control, Ofﬂine MARL with Actor Rectiﬁcation (OMAR), to better leverage the conservative value function via an effective combination of ﬁrst-order policy gradient and zeroth-order optimization methods.
Revised: In this paper, we propose a surprisingly simple yet effective method for ofﬂine multi-agent continuous control, Ofﬂine MARL with Actor Rectiﬁcation (OMAR), to better leverage the conservative value function via an effective combination of ﬁrst-order policy gradient and zeroth-order optimization methods.

Original: Towards this goal, we add a regularizer to the actor loss, which encourages the actor to mimic actions from the zeroth-order optimizer that maximizes Q-values so that we can combine the best of both ﬁrst-order policy gradient and zeroth-order optimization.
Revised: Towards this goal, we add a regularizer to the actor loss, which encourages the actor to mimic actions from the zeroth-order optimizer that maximizes Q-values so that we can combine the best of both ﬁrst-order policy gradient and zeroth-order optimization.

Original: The sampling mechanism is motivated by evolution strategies (Such et al., 2017; Conti et al., 2017; Mania et al., 2018), which recently emerged as another paradigm for solving sequential decision making tasks (Salimans et al., 2017).
Revised: The sampling mechanism is motivated by evolution strategies (Such et al., 2017; Conti et al., 2017; Mania et al., 2018), which recently emerged as another paradigm for solving sequential decision making tasks (Salimans et al., 2017).

Original: Speciﬁcally, the zeroth-order optimization part maintains an iteratively updated and reﬁned
Revised: Speciﬁcally, the zeroth-order optimization part maintains an iteratively updated and reﬁned

Original: Gaussian distribution to ﬁnd better actions based on Q-values.
Revised: Gaussian distribution to ﬁnd better actions based on Q-values.

Original: Then, we rectify the policy towards this action to better leverage the conservative value function.
Revised: Then, we rectify the policy towards this action to better leverage the conservative value function.

Original: We conduct extensive experiments in standard continuous control multi-agent particle environments and the complex multi-agent locomotion task to demonstrate its effectiveness.
Revised: We conduct extensive experiments in standard continuous control multi-agent particle environments and the complex multi-agent locomotion task to demonstrate its effectiveness.

Original: On all the benchmark tasks, OMAR outperforms the multi-agent version of ofﬂine RL algorithms including CQL (Kumar et al., 2020) and TD3+BC (Fujimoto & Gu, 2021), as well as a recent ofﬂine MARL algorithm MA-ICQ (Yang et al., 2021), and achieves the state-of-the-art performance.
Revised: On all the benchmark tasks, OMAR outperforms the multi-agent version of ofﬂine RL algorithms including CQL (Kumar et al., 2020) and TD3+BC (Fujimoto & Gu, 2021), as well as a recent ofﬂine MARL algorithm MA-ICQ (Yang et al., 2021), and achieves the state-of-the-art performance.

Original: The main contribution of this work can be summarized as follows.
Revised: The main contribution of this work can be summarized as follows.

Original: We propose the OMAR algorithm that effectively leverages both ﬁrst-order and zero-order optimization for solving ofﬂine
Revised: We propose the OMAR algorithm that effectively leverages both ﬁrst-order and zero-order optimization for solving ofﬂine

Original: MARL tasks.
Revised: MARL tasks.

Original: In addition, we theoretically prove that OMAR leads to safe policy improvement.
Revised: In addition, we theoretically prove that OMAR leads to safe policy improvement.

Original: Finally, extensive experimental results demonstrate the effectiveness of OMAR, which signiﬁcantly outperforms strong baseline methods and achieves state-of-the-art performance in datasets with different qualities in both decentralized and centralized learning paradigms.
Revised: Finally, extensive experimental results demonstrate the effectiveness of OMAR, which signiﬁcantly outperforms strong baseline methods and achieves state-of-the-art performance in datasets with different qualities in both decentralized and centralized learning paradigms.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: We consider the framework of partially observable Markov games (POMG) (Littman, 1994; Hu et al., 1998), which extends Markov decision processes to the multi-agent setting.
Revised: We consider the framework of partially observable Markov games (POMG) (Littman, 1994; Hu et al., 1998), which extends Markov decision processes to the multi-agent setting.

Original: A POMG with N agents is deﬁned by a set of global states S , a set of actions
Revised: A POMG with N agents is deﬁned by a set of global states S , a set of actions

Original: A 1 , . . .
Revised: A 1 , . . .

Original: , A N , and a set of observations
Revised: , A N , and a set of observations

Original: O 1 , . . .
Revised: O 1 , . . .

Original: , O N for each agent.
Revised: , O N for each agent.

Original: At each timestep, each agent i receive an observation o i and chooses an action based on its policy
Revised: At each timestep, each agent i receive an observation o i and chooses an action based on its policy

Original: π i .
Revised: π i .

Original: The environment transits to the next state according to the state transition function P : S ×
Revised: The environment transits to the next state according to the state transition function P : S ×

Original: A 1 × . . .
Revised: A 1 × . . .

Original: ×
Revised: ×

Original: A N × S → [0 , 1] .
Revised: A N × S → [0 , 1] .

Original: Each agent receives a reward based on the reward function r i : S
Revised: Each agent receives a reward based on the reward function r i : S

Original: ×
Revised: ×

Original: A 1 . . .
Revised: A 1 . . .

Original: ×
Revised: ×

Original: A N → R and a private observation
Revised: A N → R and a private observation

Original: o i : S
Revised: o i : S

Original: → O i .
Revised: → O i .

Original: The initial state distribution is deﬁned by ρ : S → [0 , 1] .
Revised: The initial state distribution is deﬁned by ρ : S → [0 , 1] .

Original: The goal is to ﬁnd a set of optimal policies
Revised: The goal is to ﬁnd a set of optimal policies

Original: π = { π 1 , . . .
Revised: π = { π 1 , . . .

Original: , π N } , where each agent aims to maximize its own discounted return (cid:80) ∞
Revised: , π N } , where each agent aims to maximize its own discounted return (cid:80) ∞

Original: t =

Revised: t =

Original: γ t r ti with γ denoting the discount factor.
Revised: γ t r ti with γ denoting the discount factor.

Original: In the ofﬂine setting, agents learn from a ﬁxed dataset D generated from the behavior policy π β without interaction with the environments.
Revised: In the ofﬂine setting, agents learn from a ﬁxed dataset D generated from the behavior policy π β without interaction with the environments.

Original: 2. M ULTI -A GENT A CTOR C RITIC
Revised: 2. M ULTI -A GENT A CTOR C RITIC

Original: Centralized critic.
Revised: Centralized critic.

Original: Lowe et al.
Revised: Lowe et al.

Original: (2017) propose Multi-Agent Deep Deterministic Policy Gradients (MADDPG) under the centralized training with decentralized execution (CTDE) paradigm by extending the DDPG algorithm (Lillicrap et al., 2016) to the multi-agent setting.
Revised: (2017) propose Multi-Agent Deep Deterministic Policy Gradients (MADDPG) under the centralized training with decentralized execution (CTDE) paradigm by extending the DDPG algorithm (Lillicrap et al., 2016) to the multi-agent setting.

Original: In CTDE, agents are trained in a centralized way where they can access to extra global information dur-
Revised: In CTDE, agents are trained in a centralized way where they can access to extra global information dur-

Original: ing training while they need to learn decentralized policies in order to act based only on local observations during execution.
Revised: ing training while they need to learn decentralized policies in order to act based only on local observations during execution.

Original: In MADDPG, for an agent i , the centralized critic Q i is parameterized by θ i .
Revised: In MADDPG, for an agent i , the centralized critic Q i is parameterized by θ i .

Original: It takes the global state action joint action as inputs, and aims to minimize the temporal difference error deﬁned by L ( θ i ) =
Revised: It takes the global state action joint action as inputs, and aims to minimize the temporal difference error deﬁned by L ( θ i ) =

Original: E D [( Q i ( s, a 1 , . .
Revised: E D [( Q i ( s, a 1 , . .

Original: . , a n ) − y i ) 2 ] ,
Revised: . , a n ) − y i ) 2 ] ,

Original: where y i =
Revised: where y i =

Original: r i + γ ¯ Q
Revised: r i + γ ¯ Q

Original: i ( s (cid:48) , a (cid:48) 1 , · · · , a (cid:48) n ) | a (cid:48)
Revised: i ( s (cid:48) , a (cid:48) 1 , · · · , a (cid:48) n ) | a (cid:48)

Original: j =¯ π j ( o (cid:48) j ) and ¯ Q i and ¯ π i denote target networks.
Revised: j =¯ π j ( o (cid:48) j ) and ¯ Q i and ¯ π i denote target networks.

Original: To reduce the overestimation problem in MADDPG, MATD3 (Ackermann et al., 2019) estimates the target value using double estimators based on TD3 (Fujimoto et al., 2018), where y i
Revised: To reduce the overestimation problem in MADDPG, MATD3 (Ackermann et al., 2019) estimates the target value using double estimators based on TD3 (Fujimoto et al., 2018), where y i

Original: =
Revised: =

Original: r i + γ min
Revised: r i + γ min

Original: k
Revised: k

Original: =1 , 2 ¯ Q ki ( s (cid:48) , a (cid:48) 1 , · · · , a (cid:48) n ) | a (cid:48)
Revised: =1 , 2 ¯ Q ki ( s (cid:48) , a (cid:48) 1 , · · · , a (cid:48) n ) | a (cid:48)

Original: j =¯ π j ( o (cid:48) j ) .
Revised: j =¯ π j ( o (cid:48) j ) .

Original: Agents learn decentralized policies π i parameterized by φ i , which take only local observations as inputs, and are trained by multi-agent policy gradients according to ∇ φ i J ( π i ) =
Revised: Agents learn decentralized policies π i parameterized by φ i , which take only local observations as inputs, and are trained by multi-agent policy gradients according to ∇ φ i J ( π i ) =

Original: E D (cid:2) ∇ φ i π i ( a i | o i ) ∇ a i Q i ( s, a 1 , . .
Revised: E D (cid:2) ∇ φ i π i ( a i | o i ) ∇ a i Q i ( s, a 1 , . .

Original: . , a n ) | a i =
Revised: . , a n ) | a i =

Original: π i ( o i ) (cid:3) , where a i is predicted from its policy while a − i are sampled from the replay buffer.
Revised: π i ( o i ) (cid:3) , where a i is predicted from its policy while a − i are sampled from the replay buffer.

Original: Decentralized critic.
Revised: Decentralized critic.

Original: Although using centralized critics is widely-adopted in multi-agent actorcritic methods, it introduces scalability issues due to the exponentially sized joint action space w.r.t.
Revised: Although using centralized critics is widely-adopted in multi-agent actorcritic methods, it introduces scalability issues due to the exponentially sized joint action space w.r.t.

Original: the number of agents (Iqbal & Sha, 2019).
Revised: the number of agents (Iqbal & Sha, 2019).

Original: On the other hand, independent learning approaches train decentralized critics that take only the local observation and action as inputs.
Revised: On the other hand, independent learning approaches train decentralized critics that take only the local observation and action as inputs.

Original: It is shown in de Witt et al.
Revised: It is shown in de Witt et al.

Original: ; Lyu et al.
Revised: ; Lyu et al.

Original: (2021) that decentralized value functions can result in more robust performance and be beneﬁcial in practice compared with centralized critic approaches.
Revised: (2021) that decentralized value functions can result in more robust performance and be beneﬁcial in practice compared with centralized critic approaches.

Original: de Witt et al.
Revised: de Witt et al.

Original: (2020) propose Independent Proximal Policy Optimization (IPPO) based on PPO (Schulman et al., 2017), and show that it can match or even outperform CTDE approaches in the challenging discrete control benchmark tasks (Samvelyan et al., 2019).
Revised: (2020) propose Independent Proximal Policy Optimization (IPPO) based on PPO (Schulman et al., 2017), and show that it can match or even outperform CTDE approaches in the challenging discrete control benchmark tasks (Samvelyan et al., 2019).

Original: We can also obtain the Independent TD3 (ITD3) algorithm based on decentralized critics, which is trained to minimize the temporal difference error deﬁned by
Revised: We can also obtain the Independent TD3 (ITD3) algorithm based on decentralized critics, which is trained to minimize the temporal difference error deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where y
Revised: where y

Original: [Equation]
Revised: [Equation]

Original: 2. C ONSERVATIVE Q-L EARNING
Revised: 2. C ONSERVATIVE Q-L EARNING

Original: Conservative Q-Learning (CQL) (Kumar et al., 2020) adds a regularizer to the critic loss to address the extrapolation error and learns lower-bounded Q-values.
Revised: Conservative Q-Learning (CQL) (Kumar et al., 2020) adds a regularizer to the critic loss to address the extrapolation error and learns lower-bounded Q-values.

Original: It penalizes Q-values of state-action pairs sampled from a uniform distribution or a policy while encouraging Q-values for state-action pairs in the dataset to be large.
Revised: It penalizes Q-values of state-action pairs sampled from a uniform distribution or a policy while encouraging Q-values for state-action pairs in the dataset to be large.

Original: Speciﬁcally, when built upon decentralized critic methods in MARL, the critic loss is deﬁned as in Eq.
Revised: Speciﬁcally, when built upon decentralized critic methods in MARL, the critic loss is deﬁned as in Eq.

Original: (1), where α denotes the regularization coefﬁcient and
Revised: (1), where α denotes the regularization coefﬁcient and

Original: ˆ π β
Revised: ˆ π β

Original: i is the empirical behavior policy of agent i .
Revised: i is the empirical behavior policy of agent i .

Original: [Equation]
Revised: [Equation]

Original:  P ROPOSED M ETHOD
Revised:  P ROPOSED M ETHOD

Original: In this section, we ﬁrst provide a motivating example where previous methods, such as CQL (Kumar et al., 2020) and TD3+BC (Fujimoto & Gu, 2021) can be inefﬁcient in the face of the multi-agent setting.
Revised: In this section, we ﬁrst provide a motivating example where previous methods, such as CQL (Kumar et al., 2020) and TD3+BC (Fujimoto & Gu, 2021) can be inefﬁcient in the face of the multi-agent setting.

Original: Then, we propose a method called Ofﬂine Multi-Agent Reinforcement Learning with Actor
Revised: Then, we propose a method called Ofﬂine Multi-Agent Reinforcement Learning with Actor

Original: Rectiﬁcation (OMAR), where we effectively combine ﬁrst-order policy gradients and zeroth-order optimization methods for the actor to better optimize the conservative value function.
Revised: Rectiﬁcation (OMAR), where we effectively combine ﬁrst-order policy gradients and zeroth-order optimization methods for the actor to better optimize the conservative value function.

Original: 3. T HE M OTIVATING E XAMPLE
Revised: 3. T HE M OTIVATING E XAMPLE

Original: We design a Spread environment as shown in Figure 1(a) which involves n agents and n landmarks ( n ≥ 1 ) with 1-dimensional action space to demonstrate the problem and reveal interesting ﬁndings.
Revised: We design a Spread environment as shown in Figure 1(a) which involves n agents and n landmarks ( n ≥ 1 ) with 1-dimensional action space to demonstrate the problem and reveal interesting ﬁndings.

Original: For the multi-agent setting in the Spread task, n agents need to learn how to cooperate to cover all landmarks and avoid colliding with each other or arriving at the same landmark by coordinating their actions.
Revised: For the multi-agent setting in the Spread task, n agents need to learn how to cooperate to cover all landmarks and avoid colliding with each other or arriving at the same landmark by coordinating their actions.

Original: The experimental setup is the same as in Section 4.1.1.
Revised: The experimental setup is the same as in Section 4.1.1.

Original: Figure 1(b) demonstrates the performance of the multi-agent version of TD3+BC (Fujimoto & Gu, 2021), CQL (Kumar et al., 2020), and OMAR based on ITD3 in the medium-replay dataset from the two-agent Spread environment.
Revised: Figure 1(b) demonstrates the performance of the multi-agent version of TD3+BC (Fujimoto & Gu, 2021), CQL (Kumar et al., 2020), and OMAR based on ITD3 in the medium-replay dataset from the two-agent Spread environment.

Original: As MA-TD3+BC is based on behavior regularization that compels the learned policy to stay close to the behavior policy, its performance largely depends on the quality of the dataset.
Revised: As MA-TD3+BC is based on behavior regularization that compels the learned policy to stay close to the behavior policy, its performance largely depends on the quality of the dataset.

Original: Moreover, it can be detrimental to regularize policies to be close to the dataset in multi-agent settings due to decentralized training and the resulting partial observations.
Revised: Moreover, it can be detrimental to regularize policies to be close to the dataset in multi-agent settings due to decentralized training and the resulting partial observations.

Original: MA-CQL outperforms MA-TD3+BC, which pushes down Q-values of state-action pairs that are sampled from a random or the current policy while pushing up Q-values for state-action pairs in the dataset.
Revised: MA-CQL outperforms MA-TD3+BC, which pushes down Q-values of state-action pairs that are sampled from a random or the current policy while pushing up Q-values for state-action pairs in the dataset.

Original: Figure 1(c) demonstrates the performance improvement percentage of MA-CQL over the behavior policy with an increasing number of agents ranging from one to ﬁve.
Revised: Figure 1(c) demonstrates the performance improvement percentage of MA-CQL over the behavior policy with an increasing number of agents ranging from one to ﬁve.

Original: From Figure 1(c), we observe that its performance degrades dramatically as there are more agents.
Revised: From Figure 1(c), we observe that its performance degrades dramatically as there are more agents.

Original: [Figure]
Revised: [Figure]

Original: Towards mitigating the performance degradation, we identify a key issue in MA-CQL that solely regularizing the critic is insufﬁcient for multiple agents to learn good policies for coordination.
Revised: Towards mitigating the performance degradation, we identify a key issue in MA-CQL that solely regularizing the critic is insufﬁcient for multiple agents to learn good policies for coordination.

Original: In
Revised: In

Original: Figure 1(d), we visualize the Q-function landscape of MA-CQL during training for an agent in a timestep, with the red circle corresponding to the predicted action from the actor.
Revised: Figure 1(d), we visualize the Q-function landscape of MA-CQL during training for an agent in a timestep, with the red circle corresponding to the predicted action from the actor.

Original: The green triangle represents the action predicted from the actor after the training step, where the policy gets stuck in a bad local optimum.
Revised: The green triangle represents the action predicted from the actor after the training step, where the policy gets stuck in a bad local optimum.

Original: The ﬁrst-order policy gradient method is prone to local optima (Dauphin et al., 2014; Ahmed et al., 2019), where the agent can fail to globally leverage the conservative value function well and thus leading to suboptimal, uncoordinated learning behavior.
Revised: The ﬁrst-order policy gradient method is prone to local optima (Dauphin et al., 2014; Ahmed et al., 2019), where the agent can fail to globally leverage the conservative value function well and thus leading to suboptimal, uncoordinated learning behavior.

Original: Note that the problem is exacerbated more in the ofﬂine multi-agent setting due to the exponentially sized joint action space w.r.t.
Revised: Note that the problem is exacerbated more in the ofﬂine multi-agent setting due to the exponentially sized joint action space w.r.t.

Original: the number of agents (Yang et al., 2021).
Revised: the number of agents (Yang et al., 2021).

Original: In addition, it usually requires each of the agent to learn a good policy for coordination to solve the task, and the suboptimal policy by any agent could result in uncoordinated global failure.
Revised: In addition, it usually requires each of the agent to learn a good policy for coordination to solve the task, and the suboptimal policy by any agent could result in uncoordinated global failure.

Original: Tables 1 and 2 show the performance of MA-CQL by increasing the learning rate or the number of updates for the actor.
Revised: Tables 1 and 2 show the performance of MA-CQL by increasing the learning rate or the number of updates for the actor.

Original: The results illustrate that, to solve this challenging problem, we need a better solution than blindly tuning hyperparameters.
Revised: The results illustrate that, to solve this challenging problem, we need a better solution than blindly tuning hyperparameters.

Original: In the next section, we introduce how we tackle this problem by combining zeroth-order optimization with current RL algorithms.
Revised: In the next section, we introduce how we tackle this problem by combining zeroth-order optimization with current RL algorithms.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: O FFLINE M ULTI -A GENT R EINFORCEMENT L EARNING WITH A CTOR R ECTIFICATION
Revised: O FFLINE M ULTI -A GENT R EINFORCEMENT L EARNING WITH A CTOR R ECTIFICATION

Original: Our key identiﬁcation as above is that policy gradient improvements are prone to local optima given a bad value function landscape.
Revised: Our key identiﬁcation as above is that policy gradient improvements are prone to local optima given a bad value function landscape.

Original: It is important to note that this presents a particularly critical challenge in the multi-agent setting since it is sensitive to suboptimal actions.
Revised: It is important to note that this presents a particularly critical challenge in the multi-agent setting since it is sensitive to suboptimal actions.

Original: Zeroth-order optimization methods, e.g. , evolution strategies (Rubinstein & Kroese, 2013; Such et al., 2017; Conti et al., 2017;
Revised: Zeroth-order optimization methods, e.g. , evolution strategies (Rubinstein & Kroese, 2013; Such et al., 2017; Conti et al., 2017;

Original: Salimans et al., 2017; Mania et al., 2018), offer an alternative for policy optimization and are also robust to local optima (Rubinstein & Kroese, 2013).
Revised: Salimans et al., 2017; Mania et al., 2018), offer an alternative for policy optimization and are also robust to local optima (Rubinstein & Kroese, 2013).

Original: We propose Ofﬂine Multi-Agent Reinforcement Learning with Actor Rectiﬁcation (OMAR) which incorporates sampled actions based on Q-values to rectify the actor so that it can escape from bad local optima.
Revised: We propose Ofﬂine Multi-Agent Reinforcement Learning with Actor Rectiﬁcation (OMAR) which incorporates sampled actions based on Q-values to rectify the actor so that it can escape from bad local optima.

Original: For simplicity of presentation, we demonstrate our method based on the decentralized training paradigm introduced in Section 2.1.
Revised: For simplicity of presentation, we demonstrate our method based on the decentralized training paradigm introduced in Section 2.1.

Original: Note that it can also be applied to centralized critics, as shown in Section 4.1.4.
Revised: Note that it can also be applied to centralized critics, as shown in Section 4.1.4.

Original: Speciﬁcally, we add a regularizer to the policy objective:
Revised: Speciﬁcally, we add a regularizer to the policy objective:

Original: [Equation]
Revised: [Equation]

Original: where ˆ a
Revised: where ˆ a

Original: i is the action provided by the zeroth-order optimizer and τ ∈ [0 , 1] denotes the regularization coefﬁcient.
Revised: i is the action provided by the zeroth-order optimizer and τ ∈ [0 , 1] denotes the regularization coefﬁcient.

Original: Note that TD3+BC (Fujimoto & Gu, 2021) uses the seen action in the dataset for ˆ a i .
Revised: Note that TD3+BC (Fujimoto & Gu, 2021) uses the seen action in the dataset for ˆ a i .

Original: The distinction between optimized and seen actions enables OMAR to perform well even if the dataset quality is from mediocre to low.
Revised: The distinction between optimized and seen actions enables OMAR to perform well even if the dataset quality is from mediocre to low.

Original: We borrow intuition for sampling actions from recent evolution strategy (ES) algorithms, which show a welcoming avenue towards using zeroth-order method for policy optimization.
Revised: We borrow intuition for sampling actions from recent evolution strategy (ES) algorithms, which show a welcoming avenue towards using zeroth-order method for policy optimization.

Original: For example, the cross-entropy method (CEM) (Rubinstein & Kroese, 2013), a popular ES algorithm, has shown great potential in RL (Lim et al., 2018), especially by sampling in the parameter space of the actor (Pourchot & Sigaud, 2019).
Revised: For example, the cross-entropy method (CEM) (Rubinstein & Kroese, 2013), a popular ES algorithm, has shown great potential in RL (Lim et al., 2018), especially by sampling in the parameter space of the actor (Pourchot & Sigaud, 2019).

Original: However, CEM does not scale to tasks with high-dimensional space well (Nagabandi et al., 2020).
Revised: However, CEM does not scale to tasks with high-dimensional space well (Nagabandi et al., 2020).

Original: We therefore propose to sample actions in a softer way motivated by Williams et al. (2015); Lowrey et al.
Revised: We therefore propose to sample actions in a softer way motivated by Williams et al. (2015); Lowrey et al.

Original: Speciﬁcally, we sample actions according to an iteratively reﬁned Gaussian distribution N ( µ i , σ i ) .
Revised: Speciﬁcally, we sample actions according to an iteratively reﬁned Gaussian distribution N ( µ i , σ i ) .

Original: At each iteration j , we draw K candidate actions by a ji ∼ N ( µ ji , σ ji ) and evaluate their Q-values.
Revised: At each iteration j , we draw K candidate actions by a ji ∼ N ( µ ji , σ ji ) and evaluate their Q-values.

Original: The mean and standard deviation of the sampling distribution is updated and reﬁned by Eq.
Revised: The mean and standard deviation of the sampling distribution is updated and reﬁned by Eq.

Original: (3), which produces a softer update and leverages more samples in the update (Nagabandi et al., 2020).
Revised: (3), which produces a softer update and leverages more samples in the update (Nagabandi et al., 2020).

Original: The OMAR algorithm is shown in Algorithm 1.
Revised: The OMAR algorithm is shown in Algorithm 1.

Original: [Equation]
Revised: [Equation]

Original: Besides the algorithmic design, we also prove that OMAR gives a safe policy improvement guarantee.
Revised: Besides the algorithmic design, we also prove that OMAR gives a safe policy improvement guarantee.

Original: Let J ( π i ) denote the discounted return of a policy
Revised: Let J ( π i ) denote the discounted return of a policy

Original: π i in the empirical MDP ˆ M i which is induced by transitions in the dataset D i , i.e., ˆ M i = {
Revised: π i in the empirical MDP ˆ M i which is induced by transitions in the dataset D i , i.e., ˆ M i = {

Original: ( o i , a i , r i , o (cid:48) i )
Revised: ( o i , a i , r i , o (cid:48) i )

Original: ∈
Revised: ∈

Original: D i } .
Revised: D i } .

Original: In Theorem 1, we give a lower bound on the difference between the policy performance of OMAR over the empirical behavior policy ˆ π β i in the empirical MDP ˆ M i .
Revised: In Theorem 1, we give a lower bound on the difference between the policy performance of OMAR over the empirical behavior policy ˆ π β i in the empirical MDP ˆ M i .

Original: The proof can be found in Appendix A.
Revised: The proof can be found in Appendix A.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let π ∗
Revised: Let π ∗

Original: i be the policy obtained by optimizing Eq.
Revised: i be the policy obtained by optimizing Eq.

Original: Then, we have that
Revised: Then, we have that

Original: [Equation]
Revised: [Equation]

Original: , where D ( π i , ˆ π β i )( o i ) =
Revised: , where D ( π i , ˆ π β i )( o i ) =

Original: [Equation]
Revised: [Equation]

Original: o
Revised: o

Original: [Equation]
Revised: [Equation]

Original: o
Revised: o

Original: [Equation]
Revised: [Equation]

Original: 1 , and d
Revised: 1 , and d

Original: [Equation]
Revised: [Equation]

Original: i ( o i ) is the marginal discounted distribution of observations of policy π i .
Revised: i ( o i ) is the marginal discounted distribution of observations of policy π i .

Original: As shown in Theorem 1, the difference between the second and third terms on the right-hand side is the difference between two expected distances.
Revised: As shown in Theorem 1, the difference between the second and third terms on the right-hand side is the difference between two expected distances.

Original: The former corresponds to the gap between the optimal action and the action from our zeroth-order optimizer, while the latter corresponds to the gap between the action from the behavior policy and the optimized action.
Revised: The former corresponds to the gap between the optimal action and the action from our zeroth-order optimizer, while the latter corresponds to the gap between the action from the behavior policy and the optimized action.

Original: Since both terms can be bounded, we ﬁnd that OMAR gives a safe policy improvement guarantee over ˆ π β i .
Revised: Since both terms can be bounded, we ﬁnd that OMAR gives a safe policy improvement guarantee over ˆ π β i .

Original: 5 Number of agents
Revised: 5 Number of agents

Original: P e r c e n t a g e ( % )
Revised: P e r c e n t a g e ( % )

Original: Discussion of the effect of OMAR in the Spread environment.
Revised: Discussion of the effect of OMAR in the Spread environment.

Original: We now investigate whether OMAR can address the identiﬁed problem and analyze its effect in the Spread environment introduced in Section 3.1.
Revised: We now investigate whether OMAR can address the identiﬁed problem and analyze its effect in the Spread environment introduced in Section 3.1.

Original: In
Revised: In

Original: Figure 1(d), the blue square corresponds to the action from the updated actor using OMAR according to Eq.
Revised: Figure 1(d), the blue square corresponds to the action from the updated actor using OMAR according to Eq.

Original: In contrast to the policy update in MA-CQL, OMAR can better leverage the global information in the critic and help the actor to escape from the bad local optima.
Revised: In contrast to the policy update in MA-CQL, OMAR can better leverage the global information in the critic and help the actor to escape from the bad local optima.

Original: Figure 1(b) further validates that OMAR signiﬁcantly improves MA-CQL in terms of both performance and efﬁciency.
Revised: Figure 1(b) further validates that OMAR signiﬁcantly improves MA-CQL in terms of both performance and efﬁciency.

Original: Figure 2 shows the performance improvement percentage of OMAR over MA-CQL with varying number of agents, where OMAR always outperforms MA-CQL.
Revised: Figure 2 shows the performance improvement percentage of OMAR over MA-CQL with varying number of agents, where OMAR always outperforms MA-CQL.

Original: We also notice that the performance improvement of OMAR over MA-CQL is much more signiﬁcant in the multi-agent setting in the Spread task than the single-agent setting, which echoes with what is discussed above that the problem becomes more critical in scenarios with more agents that requires each of the agents to learn a good policy to cooperate for solving the task.
Revised: We also notice that the performance improvement of OMAR over MA-CQL is much more signiﬁcant in the multi-agent setting in the Spread task than the single-agent setting, which echoes with what is discussed above that the problem becomes more critical in scenarios with more agents that requires each of the agents to learn a good policy to cooperate for solving the task.

Original: +
Revised: +

Original: [Table]
Revised: [Table]

Original: ¯ Q 1 i ( o (cid:48)
Revised: ¯ Q 1 i ( o (cid:48)

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: i ( i +
Revised: i ( i +

Original: ¯ Q 2 i ( o (cid:48)
Revised: ¯ Q 2 i ( o (cid:48)

Original: i π i ( o (cid:48)
Revised: i π i ( o (cid:48)

Original: i +
Revised: i +

Original: ) (cid:1)
Revised: ) (cid:1)

Original: [Table]
Revised: [Table]

Original: a i max
Revised: a i max

Original: [Table]
Revised: [Table]

Original: a i ∈
Revised: a i ∈

Original: ˆ A
Revised: ˆ A

Original: i ∪ π i (
Revised: i ∪ π i (

Original: ) Q 1 i (
Revised: ) Q 1 i (

Original: [Table]
Revised: [Table]

Original: Update
Revised: Update

Original: [Table] [Table]
Revised: [Table] [Table]

Original: S
Revised: S

Original: [Table]
Revised: [Table]

Original: − τ ) Q 1 i ( o i π i ( o i ))
Revised: − τ ) Q 1 i ( o i π i ( o i ))

Original: − τ ( π i ( i ) − ˆ a i
Revised: − τ ( π i ( i ) − ˆ a i

Original: [Table]
Revised: [Table]

Original: ← ρφ
Revised: ← ρφ

Original: +
Revised: +

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we conduct a series of experiments to study the following key questions: i) How does OMAR compare against state-of-the-art ofﬂine RL and ofﬂine MARL methods? ii)
Revised: In this section, we conduct a series of experiments to study the following key questions: i) How does OMAR compare against state-of-the-art ofﬂine RL and ofﬂine MARL methods? ii)

Original: What is the effect of critical hyperparameters and the sampling scheme?
Revised: What is the effect of critical hyperparameters and the sampling scheme?

Original: iii) Does the method help in both centralized training and decentralized training paradigms?
Revised: iii) Does the method help in both centralized training and decentralized training paradigms?

Original: iv) Can OMAR scale to the more complex continuous multi-agent locomotion tasks?
Revised: iv) Can OMAR scale to the more complex continuous multi-agent locomotion tasks?

Original: 4. M ULTI -A GENT P ARTICLE E NVIRONMENTS 4.1. E XPERIMENTAL S ETUP
Revised: 4. M ULTI -A GENT P ARTICLE E NVIRONMENTS 4.1. E XPERIMENTAL S ETUP

Original: We ﬁrst conduct a series of experiments in the widely-adopted multi-agent particle tasks (Lowe et al., 2017) as shown in Figure 5 in Appendix B.1.
Revised: We ﬁrst conduct a series of experiments in the widely-adopted multi-agent particle tasks (Lowe et al., 2017) as shown in Figure 5 in Appendix B.1.

Original: The cooperative navigation task includesagents and 3 landmarks, where agents are rewarded based on the distance to the landmarks and penalized for colliding with each other.
Revised: The cooperative navigation task includesagents and 3 landmarks, where agents are rewarded based on the distance to the landmarks and penalized for colliding with each other.

Original: Thus, it is important for agents to cooperate to cover all landmarks without collision.
Revised: Thus, it is important for agents to cooperate to cover all landmarks without collision.

Original: In predator-prey, 3 predators aim to catch the prey.
Revised: In predator-prey, 3 predators aim to catch the prey.

Original: The predators need to cooperate to surround and catch the prey as the predators are slower than the prey.
Revised: The predators need to cooperate to surround and catch the prey as the predators are slower than the prey.

Original: The world task involves 4 slower cooperating agents that aim to catch 2 faster adversaries, where adversaries desire to eat foods while avoiding being captured.
Revised: The world task involves 4 slower cooperating agents that aim to catch 2 faster adversaries, where adversaries desire to eat foods while avoiding being captured.

Original: We construct a variety of datasets according to behavior policies with different qualities based on adding noises to the MATD3 algorithm to increase diversity following previous work (Fu et al., 2020).
Revised: We construct a variety of datasets according to behavior policies with different qualities based on adding noises to the MATD3 algorithm to increase diversity following previous work (Fu et al., 2020).

Original: The random dataset is generated by rolling out a randomly initialized policy for 1 million (M) steps.
Revised: The random dataset is generated by rolling out a randomly initialized policy for 1 million (M) steps.

Original: We obtain the medium-replay dataset by recording all samples in the experience replay buffer during the training process until the policy reached the medium level of performance.
Revised: We obtain the medium-replay dataset by recording all samples in the experience replay buffer during the training process until the policy reached the medium level of performance.

Original: The medium dataset consists of 1M samples by unrolling a partially-pretrained policy in the online setting whose performance reaches a medium level of the performance.
Revised: The medium dataset consists of 1M samples by unrolling a partially-pretrained policy in the online setting whose performance reaches a medium level of the performance.

Original: The expert dataset is constructed by 1M expert demonstrations from an online fully-trained policy.
Revised: The expert dataset is constructed by 1M expert demonstrations from an online fully-trained policy.

Original: We compare OMAR against state-of-the-art ofﬂine RL algorithms including CQL (Kumar et al., 2020) and TD3+BC (Fujimoto & Gu, 2021).
Revised: We compare OMAR against state-of-the-art ofﬂine RL algorithms including CQL (Kumar et al., 2020) and TD3+BC (Fujimoto & Gu, 2021).

Original: We also compare with a recent ofﬂine MARL algorithm
Revised: We also compare with a recent ofﬂine MARL algorithm

Original: MA-ICQ (Yang et al., 2021).
Revised: MA-ICQ (Yang et al., 2021).

Original: We build all methods on independent TD3 based on decentralized critics following de Witt et al.
Revised: We build all methods on independent TD3 based on decentralized critics following de Witt et al.

Original: (2020), while we also consider centralized critics based on MATD following Yu et al.
Revised: (2020), while we also consider centralized critics based on MATD following Yu et al.

Original: (2021) in Section 4.1.4.
Revised: (2021) in Section 4.1.4.

Original: All baselines are implemented based on the open-source code 1 .
Revised: All baselines are implemented based on the open-source code 1 .

Original: Each algorithm is run for ﬁve random seeds, and we report the mean performance with standard deviation.
Revised: Each algorithm is run for ﬁve random seeds, and we report the mean performance with standard deviation.

Original: A detailed description of the construction of the datasets and hyperparameters can be found in Appendix B.1.
Revised: A detailed description of the construction of the datasets and hyperparameters can be found in Appendix B.1.

Original: 4.1. P ERFORMANCE C OMPARISON
Revised: 4.1. P ERFORMANCE C OMPARISON

Original: Table 3 summarizes the average normalized scores in different datasets in multi-agent particle environments, where the learning curves are shown in Appendix B.2.
Revised: Table 3 summarizes the average normalized scores in different datasets in multi-agent particle environments, where the learning curves are shown in Appendix B.2.

Original: The normalized score is computed as 100 × ( S − S random ) /
Revised: The normalized score is computed as 100 × ( S − S random ) /

Original: ( S expert − S random ) following Fu et al.
Revised: ( S expert − S random ) following Fu et al.

Original: As shown, the performance of MA-TD3+BC highly depends on the quality of the dataset.
Revised: As shown, the performance of MA-TD3+BC highly depends on the quality of the dataset.

Original: As the MA-ICQ method is based on only trusting seen state-action pairs in the dataset, it does not perform well in datasets with more diverse data distribution including random and medium-replay datasets, while generally matches the performance of MA-TD3+BC in datasets with more narrow distribution including medium and expert.
Revised: As the MA-ICQ method is based on only trusting seen state-action pairs in the dataset, it does not perform well in datasets with more diverse data distribution including random and medium-replay datasets, while generally matches the performance of MA-TD3+BC in datasets with more narrow distribution including medium and expert.

Original: MA-CQL matches or outperforms MA-TD3+BC in datasets with lower quality except for the expert dataset, as it does not rely on constraining the learning policy to stay close to the behavior policy.
Revised: MA-CQL matches or outperforms MA-TD3+BC in datasets with lower quality except for the expert dataset, as it does not rely on constraining the learning policy to stay close to the behavior policy.

Original: Our OMAR method signiﬁcantly outperforms all baseline methods and achieves state-of-the-art performance.
Revised: Our OMAR method signiﬁcantly outperforms all baseline methods and achieves state-of-the-art performance.

Original: We attribute the performance gain to the actor rectiﬁcation scheme that is independent of data quality and improves global optimization.
Revised: We attribute the performance gain to the actor rectiﬁcation scheme that is independent of data quality and improves global optimization.

Original: In addition, OMAR does not incur much computation cost and only takes 4 .
Revised: In addition, OMAR does not incur much computation cost and only takes 4 .

Original: 7% more runtime on average compared with that of MA-CQL.
Revised: 7% more runtime on average compared with that of MA-CQL.

Original: [Table]
Revised: [Table]

Original: ndo m
Revised: ndo m

Original: [Table]
Revised: [Table]

Original: M e d i u m -r e p
Revised: M e d i u m -r e p

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A BLATION S TUDY
Revised: A BLATION S TUDY

Original: The effect of the regularization coefﬁcient.
Revised: The effect of the regularization coefﬁcient.

Original: We ﬁrst investigate the effect of the regularization coefﬁcient τ in the actor loss in Eq.
Revised: We ﬁrst investigate the effect of the regularization coefﬁcient τ in the actor loss in Eq.

Original: Figure 3 shows the averaged normalized score of OMAR over different tasks with different values of τ in each kind of dataset.
Revised: Figure 3 shows the averaged normalized score of OMAR over different tasks with different values of τ in each kind of dataset.

Original: As shown, the performance of OMAR is sensitive to this hyperparameter
Revised: As shown, the performance of OMAR is sensitive to this hyperparameter

Original: [Table]
Revised: [Table]

Original: which controls the exploitation level of the critic
Revised: which controls the exploitation level of the critic

Original: We ﬁnd the best value of τ is neither close to 1 nor 0 , showing that it is the combination of both policy gradients and the actor rectiﬁcation that performs well.
Revised: We ﬁnd the best value of τ is neither close to 1 nor 0 , showing that it is the combination of both policy gradients and the actor rectiﬁcation that performs well.

Original: We also notice that the optimal value of τ is smaller for datasets with lower quality and more diverse data distribution including random and medium-replay, but larger for medium and expert datasets.
Revised: We also notice that the optimal value of τ is smaller for datasets with lower quality and more diverse data distribution including random and medium-replay, but larger for medium and expert datasets.

Original: In addition, the performance of OMAR with all values of τ matches or outperforms that of MA-CQL.
Revised: In addition, the performance of OMAR with all values of τ matches or outperforms that of MA-CQL.

Original: This is the only hyperparameter that needs to be tuned in OMAR beyond MA-CQL.
Revised: This is the only hyperparameter that needs to be tuned in OMAR beyond MA-CQL.

Original: [Figure]
Revised: [Figure]

Original: The effect of key hyperparameters in the sampling scheme.
Revised: The effect of key hyperparameters in the sampling scheme.

Original: Core hyperparameters for our sampling mechanism involves the number of iterations, the number of sampled actions, and the initial mean and standard deviation of the Gaussian distribution.
Revised: Core hyperparameters for our sampling mechanism involves the number of iterations, the number of sampled actions, and the initial mean and standard deviation of the Gaussian distribution.

Original: Figures 4(a)-(d) show the performance comparison of OMAR with different values of these hyperparameters in the cooperative navigation task, where the grey dotted line corresponds to the normalized score of MA-CQL.
Revised: Figures 4(a)-(d) show the performance comparison of OMAR with different values of these hyperparameters in the cooperative navigation task, where the grey dotted line corresponds to the normalized score of MA-CQL.

Original: As shown, our sampling mechanism is not sensitive to these hyperparameters, and we ﬁx them to be the set with the best performance.
Revised: As shown, our sampling mechanism is not sensitive to these hyperparameters, and we ﬁx them to be the set with the best performance.

Original: [Figure]
Revised: [Figure]

Original: The effect of the sampling mechanism.
Revised: The effect of the sampling mechanism.

Original: We now analyze the effect of the zeroth-order optimization methods in OMAR, and compare it against random shooting and the cross-entropy method (CEM)
Revised: We now analyze the effect of the zeroth-order optimization methods in OMAR, and compare it against random shooting and the cross-entropy method (CEM)

Original: (De Boer et al., 2005) in the cooperative navigation task.
Revised: (De Boer et al., 2005) in the cooperative navigation task.

Original: As shown in Table 4, our sampling mechanism signiﬁcantly outperforms the random sampling scheme and CEM, with a larger margin in datasets with lower quality including random and medium-replay.
Revised: As shown in Table 4, our sampling mechanism signiﬁcantly outperforms the random sampling scheme and CEM, with a larger margin in datasets with lower quality including random and medium-replay.

Original: The proposed sampling technique incorporates more samples into the distribution updates more effectively.
Revised: The proposed sampling technique incorporates more samples into the distribution updates more effectively.

Original: [Table]
Revised: [Table]

Original: A PPLICABILITY ON C ENTRALIZED T RAINING WITH D ECENTRALIZED E XECUTION
Revised: A PPLICABILITY ON C ENTRALIZED T RAINING WITH D ECENTRALIZED E XECUTION

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we demonstrate the versatility of the method and show that it can also be applied and beneﬁcial to methods based on centralized critics under the CTDE paradigm.
Revised: we demonstrate the versatility of the method and show that it can also be applied and beneﬁcial to methods based on centralized critics under the CTDE paradigm.

Original: Speciﬁcally, all baseline methods are built upon the MATD3 algorithm (Ackermann et al., 2019) using centralized critics as detailed in Section 2.1.
Revised: Speciﬁcally, all baseline methods are built upon the MATD3 algorithm (Ackermann et al., 2019) using centralized critics as detailed in Section 2.1.

Original: Table 5 summarizes the averaged normalized score of different algorithms in each kind of dataset.
Revised: Table 5 summarizes the averaged normalized score of different algorithms in each kind of dataset.

Original: As shown, OMAR (centralized) also signiﬁcantly outperforms
Revised: As shown, OMAR (centralized) also signiﬁcantly outperforms

Original: MA-ICQ (centralized) and MA-CQL (centralized)
Revised: MA-ICQ (centralized) and MA-CQL (centralized)

Original: [Table]
Revised: [Table]

Original: and matches the performance of MA-TD3+BC (centralized) in the expert dataset while outperforming it in other datasets
Revised: and matches the performance of MA-TD3+BC (centralized) in the expert dataset while outperforming it in other datasets

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4. M ULTI -A GENT M U J O C O
Revised: 4. M ULTI -A GENT M U J O C O

Original: In this section, we investigate whether OMAR can scale to more complex continuous control multiagent tasks.
Revised: In this section, we investigate whether OMAR can scale to more complex continuous control multiagent tasks.

Original: Peng et al.
Revised: Peng et al.

Original: (2020) introduce multi-agent locomotion tasks which extends the high- dimensional MuJoCo locomotion tasks in the single-agent setting to the multi-agent case.
Revised: (2020) introduce multi-agent locomotion tasks which extends the high- dimensional MuJoCo locomotion tasks in the single-agent setting to the multi-agent case.

Original: We consider the two-agent HalfCheetah task (Kim et al., 2021) as shown in Appendix B.1, where the ﬁrst and second agents control different parts of joints of the robot.
Revised: We consider the two-agent HalfCheetah task (Kim et al., 2021) as shown in Appendix B.1, where the ﬁrst and second agents control different parts of joints of the robot.

Original: Agents need to cooperate to make the robot run forward by coordinating the actions.
Revised: Agents need to cooperate to make the robot run forward by coordinating the actions.

Original: We also construct different types of datasets following Fu et al.
Revised: We also construct different types of datasets following Fu et al.

Original: (2020) the same as in Section 4.1.1.
Revised: (2020) the same as in Section 4.1.1.

Original: Table 6 summarizes the average normalized scores in each kind of dataset in multi-agent HalfCheetah.
Revised: Table 6 summarizes the average normalized scores in each kind of dataset in multi-agent HalfCheetah.

Original: As shown, OMAR signiﬁcantly outperforms baseline methods in random, medium-replay, and medium datasets
Revised: As shown, OMAR signiﬁcantly outperforms baseline methods in random, medium-replay, and medium datasets

Original: [Table]
Revised: [Table]

Original: and matches the performance of
Revised: and matches the performance of

Original: MA-TD3+BC in expert, demonstrating its effectiveness to scale to more complex control tasks.
Revised: MA-TD3+BC in expert, demonstrating its effectiveness to scale to more complex control tasks.

Original: [Table]
Revised: [Table]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Ofﬂine reinforcement learning.
Revised: Ofﬂine reinforcement learning.

Original: Many recent papers achieve improvements in ofﬂine RL (Wu et al., 2019; Kumar et al., 2019; Yu et al., 2020; Kidambi et al., 2020) that address the extrapolation error.
Revised: Many recent papers achieve improvements in ofﬂine RL (Wu et al., 2019; Kumar et al., 2019; Yu et al., 2020; Kidambi et al., 2020) that address the extrapolation error.

Original: Behavior regularization typically compels the learning policy to stay close to the behavior policy.
Revised: Behavior regularization typically compels the learning policy to stay close to the behavior policy.

Original: Yet, its performance relies heavily on the quality of the dataset.
Revised: Yet, its performance relies heavily on the quality of the dataset.

Original: Critic regularization approaches typically add a regularizer to the critic loss that pushes down Q-values for actions sampled from a given policy (Kumar et al., 2020).
Revised: Critic regularization approaches typically add a regularizer to the critic loss that pushes down Q-values for actions sampled from a given policy (Kumar et al., 2020).

Original: As discussed above, it can be difﬁcult for the actor to best leverage the global information in the critic as policy gradient methods are prone to local optima, which is particularly important in the ofﬂine multi-agent setting.
Revised: As discussed above, it can be difﬁcult for the actor to best leverage the global information in the critic as policy gradient methods are prone to local optima, which is particularly important in the ofﬂine multi-agent setting.

Original: Multi-agent reinforcement learning.
Revised: Multi-agent reinforcement learning.

Original: A number of multi-agent policy gradient algorithms train agents based on centralized value functions (Lowe et al., 2017; Foerster et al., 2018; Yu et al., 2021) while another line of research focuses on decentralized training (de Witt et al., 2020).
Revised: A number of multi-agent policy gradient algorithms train agents based on centralized value functions (Lowe et al., 2017; Foerster et al., 2018; Yu et al., 2021) while another line of research focuses on decentralized training (de Witt et al., 2020).

Original: Yang et al.
Revised: Yang et al.

Original: (2021) show that the extrapolation error in ofﬂine RL can be more severe in the multi-agent setting than the single-agent case due to the exponentially sized joint action space w.r.t.
Revised: (2021) show that the extrapolation error in ofﬂine RL can be more severe in the multi-agent setting than the single-agent case due to the exponentially sized joint action space w.r.t.

Original: the number of agents.
Revised: the number of agents.

Original: In addition, it presents a critical challenge in the decentralized setting when the datasets for each agent only consist of its own action instead of the joint action (Jiang & Lu, 2021).
Revised: In addition, it presents a critical challenge in the decentralized setting when the datasets for each agent only consist of its own action instead of the joint action (Jiang & Lu, 2021).

Original: Jiang &
Revised: Jiang &

Original: Lu (2021) address the challenges based on the behavior regularization BCQ (Fujimoto et al., 2019) algorithm while Yang et al.
Revised: Lu (2021) address the challenges based on the behavior regularization BCQ (Fujimoto et al., 2019) algorithm while Yang et al.

Original: (2021) propose to estimate the target value based on the next action from the dataset.
Revised: (2021) propose to estimate the target value based on the next action from the dataset.

Original: As a result, both methods largely depend on the quality of the dataset.
Revised: As a result, both methods largely depend on the quality of the dataset.

Original: Zeroth-order optimization method.
Revised: Zeroth-order optimization method.

Original: It has been recently shown in (Such et al., 2017; Conti et al., 2017; Mania et al., 2018) that evolutionary strategies (ES) emerge as another paradigm for continuous control.
Revised: It has been recently shown in (Such et al., 2017; Conti et al., 2017; Mania et al., 2018) that evolutionary strategies (ES) emerge as another paradigm for continuous control.

Original: Recent research shows that it is potential to combine RL with ES to reap the best of both worlds (Khadka & Tumer, 2018; Pourchot & Sigaud, 2019) in the high-dimensional parameter space for the actor.
Revised: Recent research shows that it is potential to combine RL with ES to reap the best of both worlds (Khadka & Tumer, 2018; Pourchot & Sigaud, 2019) in the high-dimensional parameter space for the actor.

Original: Sun et al.
Revised: Sun et al.

Original: (2020) replace the policy gradient update via supervised learning based on sampled noises from random shooting.
Revised: (2020) replace the policy gradient update via supervised learning based on sampled noises from random shooting.

Original: Kalashnikov et al.
Revised: Kalashnikov et al.

Original: ; Lim et al.
Revised: ; Lim et al.

Original: Simmons-Edler et al.
Revised: Simmons-Edler et al.

Original: ; Peng et al.
Revised: ; Peng et al.

Original: (2020) extend Q-learning based approaches to handle continuous action space based on the popular cross-entropy method (CEM) in ES.
Revised: (2020) extend Q-learning based approaches to handle continuous action space based on the popular cross-entropy method (CEM) in ES.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we identify the problem that when extending conservatism-based RL algorithms to ofﬂine multi-agent scenarios, the performance degrades along increasing number of agents.
Revised: In this paper, we identify the problem that when extending conservatism-based RL algorithms to ofﬂine multi-agent scenarios, the performance degrades along increasing number of agents.

Original: To tackle this problem, propose Ofﬂine Multi-Agent RL with Actor Rectiﬁcation (OMAR) that combines ﬁrst-order policy gradient with zeroth-order optimization.
Revised: To tackle this problem, propose Ofﬂine Multi-Agent RL with Actor Rectiﬁcation (OMAR) that combines ﬁrst-order policy gradient with zeroth-order optimization.

Original: We ﬁnd that OMAR can successfully help the actor escape from bad local optima and consequently ﬁnd better actions.
Revised: We ﬁnd that OMAR can successfully help the actor escape from bad local optima and consequently ﬁnd better actions.

Original: OMAR achieves state-of-the-art performance on multi-agent continuous control tasks empirically.
Revised: OMAR achieves state-of-the-art performance on multi-agent continuous control tasks empirically.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We include all details for our experiments in Appendix B.1 including open-source implementations of environments and algorithms, with a detailed description for the hyperparameters and network structures.
Revised: We include all details for our experiments in Appendix B.1 including open-source implementations of environments and algorithms, with a detailed description for the hyperparameters and network structures.

Original: For Theorem 1, its proof can be found in Appendix A.
Revised: For Theorem 1, its proof can be found in Appendix A.

Original: The code will be open-sourced upon publication of the work at https://sites.google.com/view/iclr2022omar .
Revised: The code will be open-sourced upon publication of the work at https://sites.google.com/view/iclr2022omar .

Original: A P ROOF OF T HEOREM
Revised: A P ROOF OF T HEOREM

Original: Theorem 1.
Revised: Theorem 1.

Original: Let π ∗ i be the policy obtained by optimizing Eq.
Revised: Let π ∗ i be the policy obtained by optimizing Eq.

Original: Then, we have that J ( π ∗ i ) − J
Revised: Then, we have that J ( π ∗ i ) − J

Original: (ˆ π β i ) ≥
Revised: (ˆ π β i ) ≥

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: − ˆ a i ) 2 (cid:3) , where D ( π i , ˆ π β i  ) 
Revised: − ˆ a [Equation] [Equation] (cid:3) , where D ( π i , ˆ π β i )( o i ) =

Original: ( o i )

Original: =

Original: 1 ˆ π βi ( π i ( o i 
Revised: 1 ˆ π βi ( π i ( o i )

Original: [Equation]
Revised: |

Original: | o i ) − 1 , and d π i ( o i ) is the marginal discounted distribution of observations of policy π i .
Revised: o i ) − 1 , and d π i ( o i ) is the marginal discounted distribution of observations of policy π i .

Original: Proof.
Revised: Proof.

Original: For OMAR, we have the following iterative update for agent i :
Revised: For OMAR, we have the following iterative update for agent i :

Original: [Equation]
Revised: [Equation]

Original: where ˜ π i ( a i | o i )
Revised: where ˜ π i ( a i | o i )

Original: = 

Revised: =

Original: if and only if a i =
Revised: if and only if a i =

Original: π i ( o i ) .
Revised: π i ( o i ) .

Original: Let ˆ Q k
Revised: Let ˆ Q k

Original: +1 i be the ﬁxed point of solving Equation (4) by setting the derivative of Eq.
Revised: +1 i be the ﬁxed point of solving Equation (4) by setting the derivative of Eq.

Original: (4) with respect to Q i to be 0 , then we have that
Revised: (4) with respect to Q i to be 0 , then we have that

Original: [Equation]
Revised: [Equation]

Original: where I is the indicator function.
Revised: where I is the indicator function.

Original: Denote D ( π i , ˆ π β i )( o i ) =
Revised: Denote D ( π i , ˆ π β i )( o i ) =

Original: 1 ˆ π βi ( π i ( o i )
Revised: 1 ˆ π βi ( π i ( o i )

Original: |
Revised: |

Original: o i ) −
Revised: o i ) −

Original: [Equation]
Revised: [Equation]

Original: and we obtain the difference between the value function
Revised: and we obtain the difference between the value function

Original: ˆ V i
Revised: ˆ V i

Original: ( o i ) and the original value function as:
Revised: ( o i ) and the original value function as:

Original: [Equation]
Revised: [Equation]

Original: Then [Equation] the policy that minimizes the loss function deﬁned in Eq.
Revised: Then, the policy that minimizes the loss function deﬁned in Eq.

Original: (2) is equivalently obtained by maximizing
Revised: (2) is equivalently obtained by maximizing

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we obtain that
Revised: we obtain that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then, from Eq.
Revised: Then, from Eq.

Original: (8) we obtain the result.
Revised: (8) we obtain the result.

Original: B M ORE D ETAILS OF THE E XPERIMENTS B. E XPERIMENTAL 
Revised: B M ORE D ETAILS OF THE E XPERIMENTS B. E XPERIMENTAL S ETUP

Original: S ETUP

Original: Tasks.
Revised: Tasks.

Original: We adopt the open-source implementations for multi-agent particle environmentsfrom (Lowe et al., 2017) and Multi-Agent MuJoCo 3 from (Peng et al., 2020).
Revised: We adopt the open-source implementations for multi-agent particle environmentsfrom (Lowe et al., 2017) and Multi-Agent MuJoCo 3 from (Peng et al., 2020).

Original: Figure 5 illustrates the tasks.
Revised: Figure 5 illustrates the tasks.

Revised: The expert and random scores for cooperative navigation, predator-prey, world, and two-agent

Revised: HalfCheetah are { 516 . 8 , 159 . 8 } , { 185 . 6 , − 4 . 1 } , { 79 . 5 , − 6 . 8 } , and { 3568 . 8 , − 284 . 0 } , respectively.

Original: [Figure]
Revised: [Figure]

Original: Baselines.
Revised: Baselines.

Original: All baseline methods are implemented based on an open-source implementation 4 from (Iqbal & Sha, 2019), where we implement MA-TD3+BC 5 , MA-CQL 6 , and MA-ICQ 7 based on authors’ open-source implementations with ﬁne-tuned hyperparameters.
Revised: All baseline methods are implemented based on an open-source implementation 4 from (Iqbal & Sha, 2019), where we implement MA-TD3+BC 5 , MA-CQL 6 , and MA-ICQ 7 based on authors’ open-source implementations with ﬁne-tuned hyperparameters.

Original: For MA-CQL, we tune a best critic regularization coefﬁcient from { 0 . 1 , 0 . 5 , 1 . 0 , 5 . 0 } following (Kumar et al., 2020) for each task.
Revised: For MA-CQL, we tune a best critic regularization coefﬁcient from { 0 . 1 , 0 . 5 , 1 . 0 , 5 . 0 } following (Kumar et al., 2020) for each task.

Original: Speciﬁcally, we use the discount factor γ of 0 .
Revised: Speciﬁcally, we use the discount factor γ of 0 .

Original: We sample a minibatch of 1024 samples from the dataset for updating each agent’s actor and critic using the Adam (Kingma & Ba, 2014) optimizer with the learning rate to be 0 .
Revised: We sample a minibatch of 1024 samples from the dataset for updating each agent’s actor and critic using the Adam (Kingma & Ba, 2014) optimizer with the learning rate to be 0 .

Original: The target networks for the actor and critic are soft updated with the update rate to be 0 .
Revised: The target networks for the actor and critic are soft updated with the update rate to be 0 .

Original: Both the actor and critic networks are feedforward networks consisting of two hidden layers with 64 neurons per layer using ReLU activation.
Revised: Both the actor and critic networks are feedforward networks consisting of two hidden layers with 64 neurons per layer using ReLU activation.

Original: For OMAR, the only hyperparameter that requires tuning is the regularization coefﬁcient λ , where we use a smaller value for datasets with more diverse data distribution in random and medium-replay with a value of 0 .
Revised: For OMAR, the only hyperparameter that requires tuning is the regularization coefﬁcient λ , where we use a smaller value for datasets with more diverse data distribution in random and medium-replay with a value of 0 .

Original: 5 , while we use a larger value for datasets with more narrow data distribution in medium and expert with values of 0 .
Revised: 5 , while we use a larger value for datasets with more narrow data distribution in medium and expert with values of 0 .

Original: 7 and 0 .
Revised: 7 and 0 .

Original: 9 respectively.
Revised: 9 respectively.

Original: As OMAR is insensitive to the hyperparameters of the sampling mechanism, we set them to a ﬁxed set of values for all types of datasets in all tasks, where the number of iteration is 3 , the number of samples is 10 , the mean is 0 . 0 , and the standard deviation is 2 . 0 .
Revised: As OMAR is insensitive to the hyperparameters of the sampling mechanism, we set them to a ﬁxed set of values for all types of datasets in all tasks, where the number of iteration is 3 , the number of samples is 10 , the mean is 0 . 0 , and the standard deviation is 2 . 0 .

Original: The code will be released upon publication of the paper.
Revised: The code will be released upon publication of the paper.

Original: B. L EARNING C URVES
Revised: B. L EARNING C URVES

Original: Figure 6 demonstrates the learning curves of MA-ICQ, MA-TD3+BC, MA-CQL and OMAR in different types of datasets in multi-agent particle environments, where the solid line and shaded region represent mean and standard deviation, respectively. 
Revised: Figure 6 demonstrates the learning curves of MA-ICQ, MA-TD3+BC, MA-CQL and OMAR in different types of datasets in multi-agent particle environments, where the solid line and shaded region represent mean and standard deviation, respectively. [Figure] o r [Figure]

Revised: li z s r

Original: [Figure]
Revised: [Figure]

Revised: r

Revised: [Figure]

Revised: a

Revised: [Figure]

Revised: z

Revised: [Figure]

Revised: s c o

Revised: [Figure]

Revised: B. S

Revised: [Figure]

Revised: C RAFT II M ICROMANAGEMENT B ENCHMARK

Revised: Setup.

Revised: We investigate the effectiveness of OMAR in larger-scale tasks based on the challenging

Revised: StarCraft II micromanagement benchmark (Samvelyan et al., 2019) on maps with an increasing number of agents and difﬁculties including 2s3z, 3s5z, 1c3s5z, and 2c vs 64zg.

Revised: Details for the tasks are shown in Table 7.

Revised: We compare OMAR and MA-CQL based on the evaluation protocol in Kumar et al.

Revised: Agarwal et al.

Revised: ; Gulcehre et al.

Revised: (2020), where datasets are constructed following

Revised: Agarwal et al.

Revised: ; Gulcehre et al.

Revised: (2020) by recording samples observed during training.

Revised: Each dataset consists of 1 million samples.

Revised: Since MATD3 requires differentiable policies yet sampling discrete actions from a categorical distribution is not differentiable (Peng et al., 2020), we use the

Revised: Gumbel-Softmax reparametrization trick (Lowe et al., 2017; Iqbal & Sha, 2019) to enable efﬁcient learning in discrete action space (Jang et al., 2016).

Revised: All implementations are based on open-sourced code 8 and the same experimental setup as in Appendix B.1.

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: Results.

Revised: Figure 7 demonstrates the comparison result in test win rates.

Revised: As shown, OMAR significantly outperforms MA-CQL in performance and learning efﬁciency

Revised: [Figure]

Revised: and the average performance gain of OMAR compared to MA-CQL is 76 .

Revised: 7% in all tested maps.

Revised: [Figure]

Revised: StarCraft II micromanagement tasks.

Revised: B. D ISCUSSION ON OMAR IN O NLINE /O FFLINE , M ULTI - AGENT /S INGLE - AGENT S ETTINGS

Revised: We now investigate the effectiveness of OMAR in the four following settings: (1) online multi-agent setting, (2) online single-agent setting, (3) ofﬂine single-agent setting, and (4) ofﬂine multi-agent setting in the Spread environment shown in Figure 1(a).

Revised: Setup.

Revised: For the online setting, we build our method upon the MATD3 algorithm with our proposed policy objective in Eq.

Revised: (2), and evaluate the performance improvement percentage of our method over MATD3.

Revised: The results for the online setting are shown in the right part in Figure 8, where the x-axis corresponds to the performance improvement percentage and the y-axis corresponds to the number of agents indicating whether its single-agent or multi-agent setting.

Revised: We combine the results in Figure 2 for the ofﬂine setting which shows the performance improvement percentage of OMAR over MA-CQL in the left part in Figure 8 for a better understanding of the effectiveness of our method in different settings.

Revised: Results.

Revised: As shown in Figure 8, our method is generally applicable in all the settings.

Revised: However, the performance improvement is much more signiﬁcant in the ofﬂine setting (left part) than the online case (right part), because the agents cannot explore and interact with the environment.

Revised: Intuitively, in the online setting, if the actor has not well exploited the global information in the value function, it can still interact with the environment to collect better experiences for improving the estimation of the value function and provides a better guidance for the policy.

Revised: However, no exploration and interaction with the environment for new data collection are allowed in the ofﬂine setting.

Revised: Thus, it is much harder for an agent to escape from a bad local optimum and difﬁcult for the actor to best leverage the global information in the critic.

Revised: This presents an even more challenging problem in multi-agent RL because multiple agents result in an exponentially-sized joint action space as well as the nature of the setting that requires a coordinated joint policy.

Revised: As expected, we also ﬁnd that the performance gain is more signiﬁcant in the ofﬂine multi-agent domain, which requires each of the agents to learn a good policy for a successful joint policy for coordination.

Revised: Otherwise, it can lead to an uncoordinated global failure.

Revised: [Figure]

Revised: B. T HE EFFECT OF THE SIZE OF THE DATASET

Revised: In this section, we conduct an ablation study to investigate the effect of the size of the dataset following the experimental protocol in Agarwal et al.

Revised: We ﬁrst generate a full replay dataset by recording all samples in the replay buffer encountered during the training course for 1 million steps.

Revised: Then, we randomly sample N % experiences from the full replay dataset and obtain several smaller datasets with the same data distribution, where N ∈ { 0 . 1 , 1 , 10 , 20 , 50 , 100 } .

Revised: Figure 9 shows that the performance of MA-CQL increases given more data points for N ∈ { 1 , 10 , 20 } .

Revised: However, it does not further increase given an even larger amount of data, which performs much worse than the fully-trained online agents and fails to recover their performance.

Revised: On the contrary, OMAR always outperforms MA-CQL by a large margin when N > 1% , whose performance is much closer to the fully-trained online agents given more data points.

Revised: Therefore, the optimality issue still persists when dataset size becomes larger (e.g., it can take a very long time to escape from them if the objective contains very ﬂat regions (Ahmed et al., 2019)).

Revised: In addition, the zeroth-order optimizer part in OMAR can better guide the actor given a larger amount of data points with a more accurate value function.

Revised: [Figure]

Revised: o r m a li

Revised: z

Revised: e

Revised: d s c o r e

Original:  [Figure]
Revised: MA [Figure]

Revised: B. A DDITIONAL R ESULTS OF OMAR IN S INGLE -A GENT E NVIRONMENTS

Revised: Besides the single-agent setting of the Spread task we have shown in Figure 2, we also evaluate the effectiveness of our method in single-agent tasks by comparing it with CQL in the Maze2D domain from the D4RL benchmark (Fu et al., 2020).

Revised: Table 8 shows the results in an increasing order of complexity of the maze (maze2d-umaze, maze2d-medium, maze2d-large).

Revised: Based on the results in

Revised: Table 8 and Figure 2, we observe that OMAR performs much better than CQL, which indicates that

Revised: OMAR is effective in the ofﬂine single-agent tasks.

Revised: [Table]

Revised: B.

Revised: [Table]

Original: N o r m a li z e d s c o r e
Revised: We consider a non-cooperative version of the Spread task in Figure 1(a) which involves n agents and n landmarks, where each of the agents aims to navigate to its own unique target landmark. In contrast to the Spread task that requires cooperation, the reward function for each agent only depends on its distance to its target landmark. This is a variant of Spread that consists of multiple independent learning agents, and the performance is measured by the average return over all agents.

Original: [Figure]
Revised: [Figure]

Revised: Figure 10 shows the result of the performance improvement percentage of MA-CQL over the behavior policy in the independent Spread task.

Revised: As shown, the performance of CQL does not degrade with an increasing number of agents in this setting that does not require cooperation, unlike a dramatic performance decrease in the cooperative Spread task in Figure 1(c).

Revised: The result further conﬁrms that the issue we discovered is due to the failure of coordination.

Revised: B. D ISCUSSION ABOUT THE CENTRALIZED AND DECENTRALIZED CRITICS IN OFFLINE MULTI - AGENT RL

Revised: We attribute the lower performance in Table 5 (based on a centralized value function) compared to

Revised: Table 3 (based on a decentralized value function) due to the base algorithm, where Table 9 shows the performance comparison of ofﬂine independent TD3 and ofﬂine multi-agent TD3 in different types of dataset in cooperative navigation.

Revised: As shown, utilizing centralized critics underperforms decentralized critics in the ofﬂine setting.

Revised: There has also been recent research (de Witt et al., 2020;

Revised: Lyu et al., 2021)

Revised: showing the beneﬁts of decentralized value functions compared to a centralized one, which leads to a more robust performance.

Revised: We attribute the performance loss of CTDE in the ofﬂine setting due to a more complex and higher-dimensional value function conditioning on all agent’s actions and the global state that is harder to learn well without exploration.

Revised: [Table]