Original: D4RL: D ATASETS FOR
Revised: D4RL: D ATASETS FOR

Original: D EEP D ATA -D RIVEN R
Revised: D EEP D ATA -D RIVEN R

Original: EINFORCEMENT
Revised: EINFORCEMENT

Original: L EARNING
Revised: L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The ofﬂine reinforcement learning (RL) setting (also known as full batch RL), where a policy is learned from a static dataset, is compelling as progress enables
Revised: The ofﬂine reinforcement learning (RL) setting (also known as full batch RL), where a policy is learned from a static dataset, is compelling as progress enables

Original: RL methods to take advantage of large, previously-collected datasets, much like how the rise of large datasets has fueled results in supervised learning.
Revised: RL methods to take advantage of large, previously-collected datasets, much like how the rise of large datasets has fueled results in supervised learning.

Original: However, existing online RL benchmarks are not tailored towards the ofﬂine setting and existing ofﬂine RL benchmarks are restricted to data generated by partially-trained agents, making progress in ofﬂine RL difﬁcult to measure.
Revised: However, existing online RL benchmarks are not tailored towards the ofﬂine setting and existing ofﬂine RL benchmarks are restricted to data generated by partially-trained agents, making progress in ofﬂine RL difﬁcult to measure.

Original: In this work, we introduce benchmarks speciﬁcally designed for the ofﬂine setting, guided by key properties of datasets relevant to real-world applications of ofﬂine RL.
Revised: In this work, we introduce benchmarks speciﬁcally designed for the ofﬂine setting, guided by key properties of datasets relevant to real-world applications of ofﬂine RL.

Original: With a focus on dataset collection, examples of such properties include: datasets generated via hand-designed controllers and human demonstrators, multitask datasets where an agent performs different tasks in the same environment, and datasets collected with mixtures of policies.
Revised: With a focus on dataset collection, examples of such properties include: datasets generated via hand-designed controllers and human demonstrators, multitask datasets where an agent performs different tasks in the same environment, and datasets collected with mixtures of policies.

Original: By moving beyond simple benchmark tasks and data collected by partially-trained RL agents, we reveal important and unappreciated deﬁciencies of existing algorithms.
Revised: By moving beyond simple benchmark tasks and data collected by partially-trained RL agents, we reveal important and unappreciated deﬁciencies of existing algorithms.

Original: To facilitate research, we have released our benchmark tasks and datasets with a comprehensive evaluation of existing algorithms, an evaluation protocol, and open-source examples.
Revised: To facilitate research, we have released our benchmark tasks and datasets with a comprehensive evaluation of existing algorithms, an evaluation protocol, and open-source examples.

Original: This serves as a common starting point for the community to identify shortcomings in existing ofﬂine
Revised: This serves as a common starting point for the community to identify shortcomings in existing ofﬂine

Original: RL methods and a collaborative route for progress in this emerging area.
Revised: RL methods and a collaborative route for progress in this emerging area.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Impressive progress across a range of machine learning applications has been driven by high-capacity neural network models with large, diverse training datasets (Goodfellow et al., 2016).
Revised: Impressive progress across a range of machine learning applications has been driven by high-capacity neural network models with large, diverse training datasets (Goodfellow et al., 2016).

Original: While reinforcement learning (RL) algorithms have also beneﬁted from deep learning (Mnih et al., 2015), active data collection is typically required for these algorithms to succeed, limiting the extent to which large, previously-collected datasets can be leveraged.
Revised: While reinforcement learning (RL) algorithms have also beneﬁted from deep learning (Mnih et al., 2015), active data collection is typically required for these algorithms to succeed, limiting the extent to which large, previously-collected datasets can be leveraged.

Original: Ofﬂine RL (Lange et al., 2012) (also known as full batch RL), where agents learn from previously-collected datasets, provides a bridge between RL and supervised learning.
Revised: Ofﬂine RL (Lange et al., 2012) (also known as full batch RL), where agents learn from previously-collected datasets, provides a bridge between RL and supervised learning.

Original: The promise of ofﬂine RL is leveraging large, previously-collected datasets in the context of sequential decision making, where reward-driven learning can produce policies that reason over temporally extended horizons.
Revised: The promise of ofﬂine RL is leveraging large, previously-collected datasets in the context of sequential decision making, where reward-driven learning can produce policies that reason over temporally extended horizons.

Original: This could have profound implications for a range of application domains, such as robotics, autonomous driving, and healthcare.
Revised: This could have profound implications for a range of application domains, such as robotics, autonomous driving, and healthcare.

Original: Current ofﬂine RL methods have not fulﬁlled this promise yet.
Revised: Current ofﬂine RL methods have not fulﬁlled this promise yet.

Original: While recent work has investigated technical reasons for this (Fujimoto et al., 2018a; Kumar et al., 2019; Wu et al., 2019), a major challenge in addressing these issues has been the lack of standard evaluation benchmarks.
Revised: While recent work has investigated technical reasons for this (Fujimoto et al., 2018a; Kumar et al., 2019; Wu et al., 2019), a major challenge in addressing these issues has been the lack of standard evaluation benchmarks.

Original: Ideally, such a benchmark should: a) be composed of tasks that reﬂect challenges in real-world applications of data-driven RL, b) be widely accessible for researchers and deﬁne clear evaluation protocols for reproducibility, and c) contain a range of difﬁculty to differentiate between algorithms, especially challenges particular to the ofﬂine RL setting.
Revised: Ideally, such a benchmark should: a) be composed of tasks that reﬂect challenges in real-world applications of data-driven RL, b) be widely accessible for researchers and deﬁne clear evaluation protocols for reproducibility, and c) contain a range of difﬁculty to differentiate between algorithms, especially challenges particular to the ofﬂine RL setting.

Original: Most recent works (Fujimoto et al., 2018b; Wu et al., 2019; Kumar et al., 2019; Peng et al., 2019;
Revised: Most recent works (Fujimoto et al., 2018b; Wu et al., 2019; Kumar et al., 2019; Peng et al., 2019;

Original: Agarwal et al., 2019b) use existing online RL benchmark domains and data collected from training runs of online RL methods.
Revised: Agarwal et al., 2019b) use existing online RL benchmark domains and data collected from training runs of online RL methods.

Original: However, these benchmarks were not designed with ofﬂine RL in mind and such datasets do not reﬂect the heterogenous nature of data collected in practice.
Revised: However, these benchmarks were not designed with ofﬂine RL in mind and such datasets do not reﬂect the heterogenous nature of data collected in practice.

Original: Wu et al.
Revised: Wu et al.

Original: ﬁnd that existing benchmark datasets are not sufﬁcient to differentiate between simple baseline approaches and recently proposed algorithms.
Revised: ﬁnd that existing benchmark datasets are not sufﬁcient to differentiate between simple baseline approaches and recently proposed algorithms.

Original: Furthermore, the aforementioned works do not propose a standard evaluation protocol, which makes comparing methods challenging.
Revised: Furthermore, the aforementioned works do not propose a standard evaluation protocol, which makes comparing methods challenging.

Original: Why simulated environments?
Revised: Why simulated environments?

Original: While relying on (existing) real-world datasets is appealing, evaluating a candidate policy is challenging because it weights actions differently than the data collection and may take actions that were not collected.
Revised: While relying on (existing) real-world datasets is appealing, evaluating a candidate policy is challenging because it weights actions differently than the data collection and may take actions that were not collected.

Original: Thus, evaluating a candidate policy requires either collecting additional data from the real-world system, which is hard to standardize and make broadly available, or employing off-policy evaluation, which is not yet reliable enough (e.g., the NeurIPS 2017 Criteo Ad Placement Challenge used off-policy evaluation, however, in spite of an unprecedentedly large dataset because of the variance in the estimator, top entries were not statistically distinguishable from the baseline).
Revised: Thus, evaluating a candidate policy requires either collecting additional data from the real-world system, which is hard to standardize and make broadly available, or employing off-policy evaluation, which is not yet reliable enough (e.g., the NeurIPS 2017 Criteo Ad Placement Challenge used off-policy evaluation, however, in spite of an unprecedentedly large dataset because of the variance in the estimator, top entries were not statistically distinguishable from the baseline).

Original: Both options are at odds with a widely-accessible and reproducible benchmark.
Revised: Both options are at odds with a widely-accessible and reproducible benchmark.

Original: As a compromise, we use high-quality simulations that have been battle-tested in prior domain-speciﬁc work, such as in robotics and autonomous driving.
Revised: As a compromise, we use high-quality simulations that have been battle-tested in prior domain-speciﬁc work, such as in robotics and autonomous driving.

Original: These simulators allow researchers to evaluate candidate policies accurately.
Revised: These simulators allow researchers to evaluate candidate policies accurately.

Original: Our primary contribution is the introduction of Datasets for Deep Data-Driven Reinforcement
Revised: Our primary contribution is the introduction of Datasets for Deep Data-Driven Reinforcement

Original: Learning (D4RL): a suite of tasks and datasets for benchmarking progress in ofﬂine RL.
Revised: Learning (D4RL): a suite of tasks and datasets for benchmarking progress in ofﬂine RL.

Original: We focus our design around tasks and data collection strategies that exercise dimensions of the ofﬂine RL problem likely to occur in practical applications, such as partial observability, passively logged data, or human demonstrations.
Revised: We focus our design around tasks and data collection strategies that exercise dimensions of the ofﬂine RL problem likely to occur in practical applications, such as partial observability, passively logged data, or human demonstrations.

Original: To serve as a reference, we benchmark state-of-theart ofﬂine RL algorithms (Haarnoja et al., 2018b; Kumar et al., 2019; Wu et al., 2019; Agarwal et al., 2019b; Fujimoto et al., 2018a;
Revised: To serve as a reference, we benchmark state-of-theart ofﬂine RL algorithms (Haarnoja et al., 2018b; Kumar et al., 2019; Wu et al., 2019; Agarwal et al., 2019b; Fujimoto et al., 2018a;

Original: Nachum et al., 2019; Peng et al., 2019; Kumar et al., 2020) and provide reference implementations as a starting point for future work.
Revised: Nachum et al., 2019; Peng et al., 2019; Kumar et al., 2020) and provide reference implementations as a starting point for future work.

Original: While previous studies (e.g., (Wu et al., 2019)) found that all methods including simple baselines performed well on the limited set of tasks used in prior work, we ﬁnd that most algorithms struggle to perform well on tasks with properties crucial to real-world applications such as passively logged data, narrow data distributions, and limited human demonstrations.
Revised: While previous studies (e.g., (Wu et al., 2019)) found that all methods including simple baselines performed well on the limited set of tasks used in prior work, we ﬁnd that most algorithms struggle to perform well on tasks with properties crucial to real-world applications such as passively logged data, narrow data distributions, and limited human demonstrations.

Original: By moving beyond simple benchmark tasks and data collected by partially-trained RL agents, we reveal important and unappreciated deﬁciencies of existing algorithms.
Revised: By moving beyond simple benchmark tasks and data collected by partially-trained RL agents, we reveal important and unappreciated deﬁciencies of existing algorithms.

Original: To facilitate adoption, we provide an easy-to-use
Revised: To facilitate adoption, we provide an easy-to-use

Original: API for tasks, datasets, and a collection of benchmark implementations of existing algorithms ( https://sites.google.com/view/d4rl-anonymous/ ).
Revised: API for tasks, datasets, and a collection of benchmark implementations of existing algorithms ( https://sites.google.com/view/d4rl-anonymous/ ).

Original: This is a common starting point for the community to identify shortcomings in existing ofﬂine RL methods, and provides a meaningful metric for progress in this emerging area.
Revised: This is a common starting point for the community to identify shortcomings in existing ofﬂine RL methods, and provides a meaningful metric for progress in this emerging area.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Recent work in ofﬂine RL has primarily used datasets generated by a previously trained behavior policy, ranging from a random initial policy to a near-expert online-trained policy.
Revised: Recent work in ofﬂine RL has primarily used datasets generated by a previously trained behavior policy, ranging from a random initial policy to a near-expert online-trained policy.

Original: This approach has been used for continuous control for robotics (Fujimoto et al., 2018a; Kumar et al., 2019; Wu et al., 2019; Gulcehre et al., 2020), navigation (Laroche et al., 2019), industrial control (Hein et al., 2017), and Atari video games (Agarwal et al., 2019b).
Revised: This approach has been used for continuous control for robotics (Fujimoto et al., 2018a; Kumar et al., 2019; Wu et al., 2019; Gulcehre et al., 2020), navigation (Laroche et al., 2019), industrial control (Hein et al., 2017), and Atari video games (Agarwal et al., 2019b).

Original: To standardize the community around common datasets, several recent works have proposed benchmarks for ofﬂine RL algorithms.
Revised: To standardize the community around common datasets, several recent works have proposed benchmarks for ofﬂine RL algorithms.

Original: Agarwal et al. (2019b); Fujimoto et al.
Revised: Agarwal et al. (2019b); Fujimoto et al.

Original: (2019) propose benchmarking based on the discrete Atari domain.
Revised: (2019) propose benchmarking based on the discrete Atari domain.

Original: Concurrently to our work, Gulcehre et al.
Revised: Concurrently to our work, Gulcehre et al.

Original: (2020) proposed a benchmark 1 based on locomotion and manipulation tasks with perceptually challenging input and partial observability.
Revised: (2020) proposed a benchmark 1 based on locomotion and manipulation tasks with perceptually challenging input and partial observability.

Original: While these are important contributions, both benchmarks suffer from the same shortcomings as prior evaluation protocols: they rely on data collected from online RL training runs.
Revised: While these are important contributions, both benchmarks suffer from the same shortcomings as prior evaluation protocols: they rely on data collected from online RL training runs.

Original: In contrast, with D4RL, in addition to collecting data from online RL training runs, we focus on a range of dataset collection procedures inspired by real-world applications, such as human demonstrations, exploratory agents, and hand-coded controllers.
Revised: In contrast, with D4RL, in addition to collecting data from online RL training runs, we focus on a range of dataset collection procedures inspired by real-world applications, such as human demonstrations, exploratory agents, and hand-coded controllers.

Original: As alluded to by Wu et al.
Revised: As alluded to by Wu et al.

Original: and as we show in our experiments, the performance of current methods depends strongly on the data collection procedure, demonstrating the importance of modeling realistic data collection procedures in a benchmark.
Revised: and as we show in our experiments, the performance of current methods depends strongly on the data collection procedure, demonstrating the importance of modeling realistic data collection procedures in a benchmark.

Original: Ofﬂine RL using large, previously-collect datasets has been successfully applied to real-world systems such as in robotics (Cabi et al., 2019), recommender systems (Li et al., 2010; Strehl et al., 2010;
Revised: Ofﬂine RL using large, previously-collect datasets has been successfully applied to real-world systems such as in robotics (Cabi et al., 2019), recommender systems (Li et al., 2010; Strehl et al., 2010;

Original: Thomas et al., 2017), and dialogue systems (Henderson et al., 2008; Pietquin et al., 2011; Jaques et al., 2019).
Revised: Thomas et al., 2017), and dialogue systems (Henderson et al., 2008; Pietquin et al., 2011; Jaques et al., 2019).

Original: These successes point to the promise of ofﬂine RL, however, they rely on private real-world systems or expensive human labeling for evaluation which is not scalable or accessible for a benchmark.
Revised: These successes point to the promise of ofﬂine RL, however, they rely on private real-world systems or expensive human labeling for evaluation which is not scalable or accessible for a benchmark.

Original: Moreover, signiﬁcant efforts have been made to incorporate large-scale datasets into off-policy RL (Kalashnikov et al., 2018; Mo et al., 2018; Gu et al., 2017), but these works use large numbers of robots to collect online interaction during training.
Revised: Moreover, signiﬁcant efforts have been made to incorporate large-scale datasets into off-policy RL (Kalashnikov et al., 2018; Mo et al., 2018; Gu et al., 2017), but these works use large numbers of robots to collect online interaction during training.

Original: Pure online systems for realworld learning have included model-based approaches (Hester et al., 2011), or approaches which collect large amounts of data in parallel (Gu et al., 2017).
Revised: Pure online systems for realworld learning have included model-based approaches (Hester et al., 2011), or approaches which collect large amounts of data in parallel (Gu et al., 2017).

Original: While we believe these are promising directions for future research, the focus of this work is to provide a varied and accessible platform for developing algorithms.
Revised: While we believe these are promising directions for future research, the focus of this work is to provide a varied and accessible platform for developing algorithms.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: The ofﬂine reinforcement learning problem statement is formalized within a Markov decision process (MDP), deﬁned by a tuple ( S , A , P, R, ρ 0 , γ ) , where S denotes the state space, A denotes the action space, P ( s (cid:48) | s, a ) denotes the transition distribution, ρ 0 ( s ) denotes the initial state distribution, R ( s, a ) denotes the reward function, and γ ∈ (0 , 1) denotes the discount factor.
Revised: The ofﬂine reinforcement learning problem statement is formalized within a Markov decision process (MDP), deﬁned by a tuple ( S , A , P, R, ρ 0 , γ ) , where S denotes the state space, A denotes the action space, P ( s (cid:48) | s, a ) denotes the transition distribution, ρ 0 ( s ) denotes the initial state distribution, R ( s, a ) denotes the reward function, and γ ∈ (0 , 1) denotes the discount factor.

Original: The goal in RL is to ﬁnd a policy π ( a | s ) that maximizes the expected cumulative discounted rewards
Revised: The goal in RL is to ﬁnd a policy π ( a | s ) that maximizes the expected cumulative discounted rewards

Original: J ( π ) =
Revised: J ( π ) =

Original: E π,P,ρ 

Revised: E π,P,ρ

Original: [ (cid:80) ∞ t =

Revised: [ (cid:80) ∞ t =

Original: γ t R ( s t , a t )] , also known as the discounted returns.
Revised: γ t R ( s t , a t )] , also known as the discounted returns.

Original: In episodic RL, the algorithm is given access to the MDP via trajectory samples for arbitrary π of the algorithm’s choosing.
Revised: In episodic RL, the algorithm is given access to the MDP via trajectory samples for arbitrary π of the algorithm’s choosing.

Original: Off-policy methods may use experience replay (Lin, 1992) to store these trajectories in a replay buffer D of transitions ( s t , a t , s t +1 , r t ) , and use an off-policy algorithm such as Q-learning (Watkins & Dayan, 1992) to optimize π .
Revised: Off-policy methods may use experience replay (Lin, 1992) to store these trajectories in a replay buffer D of transitions ( s t , a t , s t +1 , r t ) , and use an off-policy algorithm such as Q-learning (Watkins & Dayan, 1992) to optimize π .

Original: However, these methods still iteratively collect additional data, and omitting this collection step can produce poor results.
Revised: However, these methods still iteratively collect additional data, and omitting this collection step can produce poor results.

Original: For example, running state-of-the-art off-policy RL algorithms on trajectories collected from an expert policy can result in diverging Q-values (Kumar et al., 2019).
Revised: For example, running state-of-the-art off-policy RL algorithms on trajectories collected from an expert policy can result in diverging Q-values (Kumar et al., 2019).

Original: In ofﬂine RL , the algorithm no longer has access to the MDP, and is instead presented with a ﬁxed dataset of transitions D .
Revised: In ofﬂine RL , the algorithm no longer has access to the MDP, and is instead presented with a ﬁxed dataset of transitions D .

Original: The (unknown) policy that generated this data is referred to as a behavior policy π B .
Revised: The (unknown) policy that generated this data is referred to as a behavior policy π B .

Original: Effective ofﬂine RL algorithms must handle distribution shift, as well as data collected via processes that may not be representable by the chosen policy class.
Revised: Effective ofﬂine RL algorithms must handle distribution shift, as well as data collected via processes that may not be representable by the chosen policy class.

Original: Levine et al.
Revised: Levine et al.

Original: (2020) provide a comprehensive discussion of the problems affecting ofﬂine RL.
Revised: (2020) provide a comprehensive discussion of the problems affecting ofﬂine RL.

Original:  T ASK D ESIGN F ACTORS
Revised:  T ASK D ESIGN F ACTORS

Original: In order to design a benchmark that provides a meaningful measure of progress towards realistic applications of ofﬂine RL, we choose datasets and tasks to cover a range of properties designed to challenge existing RL algorithms.
Revised: In order to design a benchmark that provides a meaningful measure of progress towards realistic applications of ofﬂine RL, we choose datasets and tasks to cover a range of properties designed to challenge existing RL algorithms.

Original: We discuss these properties as follows:
Revised: We discuss these properties as follows:

Original: Narrow and biased data distributions , such as those from deterministic policies, are problematic for ofﬂine RL algorithms and may cause divergence both empirically
Revised: Narrow and biased data distributions , such as those from deterministic policies, are problematic for ofﬂine RL algorithms and may cause divergence both empirically

Original: (Fujimoto et al., 2018a; Kumar et al.,
Revised: (Fujimoto et al., 2018a; Kumar et al.,

Original: and theoretically
Revised: and theoretically

Original: (Munos,
Revised: (Munos,

Original: Farahmand et al., 2010; Kumar et al
Revised: Farahmand et al., 2010; Kumar et al

Original: , 2019; Agarwal et al
Revised: , 2019; Agarwal et al

Original: 2019a; Du et al.
Revised: 2019a; Du et al.

Original: Narrow datasets
Revised: Narrow datasets

Original: may arise in human demonstrations, or when using hand-crafted policies.
Revised: may arise in human demonstrations, or when using hand-crafted policies.

Original: An important challenge in ofﬂine RL is to be able to gracefully handle diverse data distributions without algorithms diverging or producing performance worse than the provided behavior.
Revised: An important challenge in ofﬂine RL is to be able to gracefully handle diverse data distributions without algorithms diverging or producing performance worse than the provided behavior.

Original: A common approach for dealing with such data distributions is to adopt a conservative approach which tries to keep the behavior close to the data distribution (Fujimoto et al., 2018a;
Revised: A common approach for dealing with such data distributions is to adopt a conservative approach which tries to keep the behavior close to the data distribution (Fujimoto et al., 2018a;

Original: Kumar et al., 2019; Wu et al., 2019).
Revised: Kumar et al., 2019; Wu et al., 2019).

Original: Undirected and multitask data naturally arises when data is passively logged, such as recording user interactions on the internet or recording videos of a car for autonomous driving.
Revised: Undirected and multitask data naturally arises when data is passively logged, such as recording user interactions on the internet or recording videos of a car for autonomous driving.

Original: This data may not necessarily be directed towards the speciﬁc task one is trying to accomplish.
Revised: This data may not necessarily be directed towards the speciﬁc task one is trying to accomplish.

Original: However, pieces of trajectories can still provide useful information to learn from.
Revised: However, pieces of trajectories can still provide useful information to learn from.

Original: For example, one may be able to combine sub-trajectories to accomplish a task.
Revised: For example, one may be able to combine sub-trajectories to accomplish a task.

Original: In the ﬁgure to the upper-right, if an agent is given trajectories from A-B and B-C in a dataset (left image), it can form a trajectory from AC by combining the corresponding halves of the original trajectories.
Revised: In the ﬁgure to the upper-right, if an agent is given trajectories from A-B and B-C in a dataset (left image), it can form a trajectory from AC by combining the corresponding halves of the original trajectories.

Original: We refer to this property as stitching , since the agent can use portions of existing trajectories in order to solve a task, rather than relying on generalization outside of the dataset.
Revised: We refer to this property as stitching , since the agent can use portions of existing trajectories in order to solve a task, rather than relying on generalization outside of the dataset.

Original: Sparse rewards .
Revised: Sparse rewards .

Original: Sparse reward problems pose challenges to traditional RL methods due to the difﬁculty of credit assignment and exploration.
Revised: Sparse reward problems pose challenges to traditional RL methods due to the difﬁculty of credit assignment and exploration.

Original: Because ofﬂine RL considers ﬁxed datasets without exploration, sparse reward problems provide an unique opportunity to isolate the ability of algorithms to perform credit assignment decoupled from exploration.
Revised: Because ofﬂine RL considers ﬁxed datasets without exploration, sparse reward problems provide an unique opportunity to isolate the ability of algorithms to perform credit assignment decoupled from exploration.

Original: Suboptimal data .
Revised: Suboptimal data .

Original: For tasks with a clear objective, the datasets may not contain behaviors from optimal agents.
Revised: For tasks with a clear objective, the datasets may not contain behaviors from optimal agents.

Original: This represents a challenge for approaches such as imitation learning, which generally require expert demonstrations.
Revised: This represents a challenge for approaches such as imitation learning, which generally require expert demonstrations.

Original: We note prior work (e.g., (Fujimoto et al., 2018a; Kumar et al., 2019; Wu et al., 2019)) predominantly uses data with this property.
Revised: We note prior work (e.g., (Fujimoto et al., 2018a; Kumar et al., 2019; Wu et al., 2019)) predominantly uses data with this property.

Original: Non-representable behavior policies, non-Markovian behavior policies, and partial observability.
Revised: Non-representable behavior policies, non-Markovian behavior policies, and partial observability.

Original: When the dataset is generated from a partially-trained agent, we ensure that the behavior policy can be realized within our model class.
Revised: When the dataset is generated from a partially-trained agent, we ensure that the behavior policy can be realized within our model class.

Original: However, real-life behavior may not originate from a policy within our model class, which can introduce additional representational errors.
Revised: However, real-life behavior may not originate from a policy within our model class, which can introduce additional representational errors.

Original: For example, data generated from human demonstrations or hand-crafted controllers may fall outside of the model class.
Revised: For example, data generated from human demonstrations or hand-crafted controllers may fall outside of the model class.

Original: More generally, non-Markovian policies and tasks with partial observability can introduce additional modeling errors when we estimate action probabilities under the assumption that the data was generated from a Markovian policy.
Revised: More generally, non-Markovian policies and tasks with partial observability can introduce additional modeling errors when we estimate action probabilities under the assumption that the data was generated from a Markovian policy.

Original: These errors can cause additional bias for ofﬂine RL algorithms, especially in methods that assume access to action probabilities from a Markovian policy such as importance weighting (Precup et al., 2000).
Revised: These errors can cause additional bias for ofﬂine RL algorithms, especially in methods that assume access to action probabilities from a Markovian policy such as importance weighting (Precup et al., 2000).

Original: Realistic domains .
Revised: Realistic domains .

Original: As we discussed previously, real-world evaluation is the ideal setting for benchmarking ofﬂine RL, however, it is at odds with a widely-accessible and reproducible benchmark.
Revised: As we discussed previously, real-world evaluation is the ideal setting for benchmarking ofﬂine RL, however, it is at odds with a widely-accessible and reproducible benchmark.

Original: To strike a balance, we opted for simulated environments which have been previously studied and are broadly accepted by the research community.
Revised: To strike a balance, we opted for simulated environments which have been previously studied and are broadly accepted by the research community.

Original: These simulation packages (such as MuJoCo) have been widely used to benchmark online RL methods and are known to ﬁt well into that role.
Revised: These simulation packages (such as MuJoCo, Flow, and CARLA) have been widely used to benchmark online RL methods and are known to ﬁt well into that role.

Revised: Moreover, on several domains we utilize human demonstrations or mathematical models of human behavior in order to provide datasets generated from realistic processes.

Original: This did have the effect of restricting our choice of tasks.
Revised: However, this did have the effect of restricting our choice of tasks.

Original: So while recent progress has been made on simulators for recommender systems (e.g., (Ie et al., 2019)), they use “stylized” user models and they have not been thoroughly evaluated by the community yet.
Revised: While recent progress has been made on simulators for recommender systems (e.g., (Ie et al., 2019)), they use “stylized” user models and they have not been thoroughly evaluated by the community yet.

Original: In the future as simulators mature, we hope to include additional tasks.
Revised: In the future as simulators mature, we hope to include additional tasks.

Original: In addition, we include a variety of qualitatively different tasks to provide broad coverage of the types of domains where ofﬂine RL could be used.
Revised: In addition, we include a variety of qualitatively different tasks to provide broad coverage of the types of domains where ofﬂine RL could be used.

Original: We include locomotion, trafﬁc management, autonomous driving, and robotics tasks.
Revised: We include locomotion, trafﬁc management, autonomous driving, and robotics tasks.

Original: We also provide tasks with a wide range in difﬁculty, from tasks current algorithms can already solve to harder problems that are currently out of reach.
Revised: We also provide tasks with a wide range in difﬁculty, from tasks current algorithms can already solve to harder problems that are currently out of reach.

Original: Finally, for consistency with prior works, we also include the OpenAI Gym robotic locomotion tasks and similar datasets used by Fujimoto et al.
Revised: Finally, for consistency with prior works, we also include the OpenAI Gym robotic locomotion tasks and similar datasets used by Fujimoto et al.

Original: (2018a)
Revised: (2018a)

Original: ; Kumar et al.
Revised: ; Kumar et al.

Original: ; Wu et al.
Revised: ; Wu et al.

Original:  T ASKS AND D ATASETS
Revised:  T ASKS AND D ATASETS

Original: Given the properties outlined in Section 4, we assembled the following tasks and datasets.
Revised: Given the properties outlined in Section 4, we assembled the following tasks and datasets.

Original: All tasks consist of an ofﬂine dataset (typically 10 6 steps) of trajectory samples for training, and a simulator for evaluation.
Revised: All tasks consist of an ofﬂine dataset (typically 10 6 steps) of trajectory samples for training, and a simulator for evaluation.

Original: The mapping is not one-to-one – several tasks use the same simulator with different datasets.
Revised: The mapping is not one-to-one – several tasks use the same simulator with different datasets.

Original: Appendix C lists domains and dataset types along with their sources and Appendix A contains a more comprehensive table of statistics such as size.
Revised: Appendix C lists domains and dataset types along with their sources and Appendix A contains a more comprehensive table of statistics such as size.

Original: Our code and datasetshave been released open-source and are on our website at https://sites.google.com/view/ d4rl-anonymous/ .
Revised: Our code and datasets

Original: Maze2D. (Non-markovian policies, undirected and multitask data)
Revised: Maze2D. (Non-markovian policies, undirected and multitask data)

Original: The Maze2D domain is a navigation task requiring a 2D agent to reach a ﬁxed goal location.
Revised: The Maze2D domain is a navigation task requiring a 2D agent to reach a ﬁxed goal location.

Original: The tasks are designed to provide a simple test of the ability of ofﬂine RL algorithms to stitch together previously collected subtrajectories to ﬁnd the shortest path to the evaluation goal.
Revised: The tasks are designed to provide a simple test of the ability of ofﬂine RL algorithms to stitch together previously collected subtrajectories to ﬁnd the shortest path to the evaluation goal.

Original: Three maze layouts are provided.
Revised: Three maze layouts are provided.

Original: The “umaze” and “medium” mazes are shown to the right, and the “large” maze is shown below.
Revised: The “umaze” and “medium” mazes are shown to the right, and the “large” maze is shown below.

Original: The data is generated by selecting goal locations at random and then using a planner that generates sequences of waypoints that are followed using a PD controller.
Revised: The data is generated by selecting goal locations at random and then using a planner that generates sequences of waypoints that are followed using a PD controller.

Original: In the ﬁgure on the left, the waypoints, represented by circles, are planned from the starting location (1) along the path to a goal (2).
Revised: In the ﬁgure on the left, the waypoints, represented by circles, are planned from the starting location (1) along the path to a goal (2).

Original: Upon reaching a threshold distance to a waypoint, the controller updates its internal state to track the next waypoint along the path to the goal.
Revised: Upon reaching a threshold distance to a waypoint, the controller updates its internal state to track the next waypoint along the path to the goal.

Original: Once a goal is reached, a new goal is selected (3) and the process continues.
Revised: Once a goal is reached, a new goal is selected (3) and the process continues.

Original: The trajectories in the dataset are visualized in Appendix G. Because the controllers memorize the reached waypoints, the data collection policy is non-Markovian.
Revised: The trajectories in the dataset are visualized in Appendix G. Because the controllers memorize the reached waypoints, the data collection policy is non-Markovian.

Original: AntMaze.
Revised: AntMaze.

Original: (Non-markovian policies, sparse rewards, undirected and multitask data)
Revised: (Non-markovian policies, sparse rewards, undirected and multitask data)

Original: The AntMaze domain is a navigation domain that replaces the 2D ball from Maze2D with the more complex 8-DoF “Ant” quadraped robot.
Revised: The AntMaze domain is a navigation domain that replaces the 2D ball from Maze2D with the more complex 8-DoF “Ant” quadraped robot.

Original: We introduce this domain to test the stitching challenge using a morphologically complex robot that could mimic real-world robotic navigation tasks.
Revised: We introduce this domain to test the stitching challenge using a morphologically complex robot that could mimic real-world robotic navigation tasks.

Original: Additionally, for this task we use a sparse 0-1 reward which is activated upon reaching the goal.
Revised: Additionally, for this task we use a sparse 0-1 reward which is activated upon reaching the goal.

Original: The data is generated by training a goal reaching policy and using it in conjunction with the same high-level waypoint generator from Maze2D to provide subgoals that guide the agent to the goal.
Revised: The data is generated by training a goal reaching policy and using it in conjunction with the same high-level waypoint generator from Maze2D to provide subgoals that guide the agent to the goal.

Original: The same 3 maze layouts are used: “umaze”, “medium”, and “large”.
Revised: The same 3 maze layouts are used: “umaze”, “medium”, and “large”.

Original: We introduce three ﬂavors of datasets: 1) the ant is commanded to reach a speciﬁc goal from a ﬁxed start location ( antmaze-umaze-v0 ), 2) in the “diverse” datasets, the ant is commanded to a random goal from a random start location, 3) in the “play” datasets, the ant is commanded to speciﬁc hand-picked locations in the maze (which are not necessarily the goal at evaluation), starting from a different set of hand-picked start locations.
Revised: We introduce three ﬂavors of datasets: 1) the ant is commanded to reach a speciﬁc goal from a ﬁxed start location ( antmaze-umaze-v0 ), 2) in the “diverse” datasets, the ant is commanded to a random goal from a random start location, 3) in the “play” datasets, the ant is commanded to speciﬁc hand-picked locations in the maze (which are not necessarily the goal at evaluation), starting from a different set of hand-picked start locations.

Original: As in Maze2D, the controllers for this task are non-Markovian as they rely on tracking visited waypoints.
Revised: As in Maze2D, the controllers for this task are non-Markovian as they rely on tracking visited waypoints.

Original: Trajectories in the dataset are visualized in Appendix G.
Revised: Trajectories in the dataset are visualized in Appendix G.

Original: Gym-MuJoCo.
Revised: Gym-MuJoCo.

Original: (Suboptimal agents, narrow data distributions)
Revised: (Suboptimal agents, narrow data distributions)

Original: The Gym-MuJoCo tasks (Hopper,
Revised: The Gym-MuJoCo tasks (Hopper,

Original: HalfCheetah, Walker2d) are popular benchmarks used in prior work in ofﬂine deep RL (Fujimoto et al., 2018a; Kumar et al., 2019; Wu et al., 2019).
Revised: HalfCheetah, Walker2d) are popular benchmarks used in prior work in ofﬂine deep RL (Fujimoto et al., 2018a; Kumar et al., 2019; Wu et al., 2019).

Original: For consistency, we provide standardized datasets similar to previous work, and additionally propose mixing datasets to test the impact of heterogenous policy mixtures.
Revised: For consistency, we provide standardized datasets similar to previous work, and additionally propose mixing datasets to test the impact of heterogenous policy mixtures.

Original: We expect that methods that rely on regularizing to the behavior policy may fail when the data contains poorly performing trajectories.
Revised: We expect that methods that rely on regularizing to the behavior policy may fail when the data contains poorly performing trajectories.

Original: The “medium” dataset is generated by ﬁrst training a policy online using Soft Actor-Critic (Haarnoja et al., 2018a), early-stopping the training, and collecting 1M samples from this partially-trained policy.
Revised: The “medium” dataset is generated by ﬁrst training a policy online using Soft Actor-Critic (Haarnoja et al., 2018a), early-stopping the training, and collecting 1M samples from this partially-trained policy.

Original: The “random” datasets are generated by unrolling a randomly initialized policy on these three domains.
Revised: The “random” datasets are generated by unrolling a randomly initialized policy on these three domains.

Original: The “medium-replay” dataset consists of recording all samples in the replay buffer observed during training until the policy reaches the “medium” level of performance.
Revised: The “medium-replay” dataset consists of recording all samples in the replay buffer observed during training until the policy reaches the “medium” level of performance.

Original: Datasets similar to these three have been used in prior work, but in order to evaluate algorithms on mixtures of policies, we further introduce a “medium-expert” dataset by mixing equal amounts of expert demonstrations and suboptimal data, generated via a partially trained policy or by unrolling a uniform-at-random policy.
Revised: Datasets similar to these three have been used in prior work, but in order to evaluate algorithms on mixtures of policies, we further introduce a “medium-expert” dataset by mixing equal amounts of expert demonstrations and suboptimal data, generated via a partially trained policy or by unrolling a uniform-at-random policy.

Original: Adroit.
Revised: Adroit.

Original: (Non-representable policies, narrow data distributions, sparse rewards, realistic)
Revised: (Non-representable policies, narrow data distributions, sparse rewards, realistic)

Original: The
Revised: The

Original: Adroit domain (Rajeswaran et al., 2018) (pictured left) involves controlling a 24-DoF simulated
Revised: Adroit domain (Rajeswaran et al., 2018) (pictured left) involves controlling a 24-DoF simulated

Original: Shadow Hand robot tasked with hammering a nail, opening a door, twirling a pen, or picking up and moving a ball.
Revised: Shadow Hand robot tasked with hammering a nail, opening a door, twirling a pen, or picking up and moving a ball.

Original: This domain was selected to measure the effect of a narrow expert data distributions and human demonstrations on a sparse reward, high-dimensional robotic manipulation task.
Revised: This domain was selected to measure the effect of a narrow expert data distributions and human demonstrations on a sparse reward, high-dimensional robotic manipulation task.

Original: While Rajeswaran et al.
Revised: While Rajeswaran et al.

Original: (2018) propose utilizing human demonstrations, in conjunction with online RL ﬁne-tuning, our benchmark adapts these tasks for evaluating the fully ofﬂine RL setting.
Revised: (2018) propose utilizing human demonstrations, in conjunction with online RL ﬁne-tuning, our benchmark adapts these tasks for evaluating the fully ofﬂine RL setting.

Original: We include three types of datasets for each task, two of which are included from the original paper: a small amount of demonstration data from a human (“human”) (25 trajectories per task) and a large amount of expert data from a ﬁne-tuned RL policy (“expert”).
Revised: We include three types of datasets for each task, two of which are included from the original paper: a small amount of demonstration data from a human (“human”) (25 trajectories per task) and a large amount of expert data from a ﬁne-tuned RL policy (“expert”).

Original: To mimic the use-case where a practitioner collects a small amount of additional data from a policy trained on the demonstrations, we introduce a third dataset generated by training an imitation policy on the demonstrations, running the policy, and mixing data at a 50-50 ratio with the demonstrations, referred to as “cloned.”
Revised: To mimic the use-case where a practitioner collects a small amount of additional data from a policy trained on the demonstrations, we introduce a third dataset generated by training an imitation policy on the demonstrations, running the policy, and mixing data at a 50-50 ratio with the demonstrations, referred to as “cloned.”

Original: The Adroit domain has several unique properties that make it qualitatively different from the Gym MuJoCo tasks.
Revised: The Adroit domain has several unique properties that make it qualitatively different from the Gym MuJoCo tasks.

Original: First, the data is collected in from human demonstrators.
Revised: First, the data is collected in from human demonstrators.

Original: Second, each task is difﬁcult to solve with online RL, due to sparse rewards and exploration challenges, which make cloning and online RL alone insufﬁcient.
Revised: Second, each task is difﬁcult to solve with online RL, due to sparse rewards and exploration challenges, which make cloning and online RL alone insufﬁcient.

Original: Lastly, the tasks are high dimensional, presenting a representation learning challenge.
Revised: Lastly, the tasks are high dimensional, presenting a representation learning challenge.

Original: FrankaKitchen.
Revised: FrankaKitchen.

Original: (Undirected and multitask data, realistic)
Revised: (Undirected and multitask data, realistic)

Original: The
Revised: The

Original: Franka Kitchen domain, ﬁrst proposed by Gupta et al.
Revised: Franka Kitchen domain, ﬁrst proposed by Gupta et al.

Original: (2019), involves controlling a 9-DoF Franka robot in a kitchen environment containing several common household items: a microwave, a kettle, an overhead light, cabinets, and an oven.
Revised: (2019), involves controlling a 9-DoF Franka robot in a kitchen environment containing several common household items: a microwave, a kettle, an overhead light, cabinets, and an oven.

Original: The goal of each task is to interact with the items in order to reach a desired goal conﬁguration.
Revised: The goal of each task is to interact with the items in order to reach a desired goal conﬁguration.

Original: For example, one such state is to have the microwave and sliding cabinet door open with the kettle on the top burner and the overhead light on.
Revised: For example, one such state is to have the microwave and sliding cabinet door open with the kettle on the top burner and the overhead light on.

Original: This domain benchmarks the effect of multitask behavior on a realistic, non-navigation environment in which the “stitching” challenge is non-trivial because the collected trajectories are complex paths through the state space.
Revised: This domain benchmarks the effect of multitask behavior on a realistic, non-navigation environment in which the “stitching” challenge is non-trivial because the collected trajectories are complex paths through the state space.

Original: As a result, algorithms must rely on generalization to unseen states in order to solve the task, rather than relying purely on trajectories seen during training.
Revised: As a result, algorithms must rely on generalization to unseen states in order to solve the task, rather than relying purely on trajectories seen during training.

Original: In order to study the effect of “stitching” and generalization, we use 3 datasets of human demonstrations, originally proposed by Gupta et al.
Revised: In order to study the effect of “stitching” and generalization, we use 3 datasets of human demonstrations, originally proposed by Gupta et al.

Original: The “complete” dataset consists of the robot performing all of the desired tasks in order.
Revised: The “complete” dataset consists of the robot performing all of the desired tasks in order.

Original: This provides data that is easy for an imitation learning method to solve.
Revised: This provides data that is easy for an imitation learning method to solve.

Original: The “partial” and “mixed” datasets consist of undirected data, where the robot performs subtasks that are not necessarily related to the goal conﬁguration.
Revised: The “partial” and “mixed” datasets consist of undirected data, where the robot performs subtasks that are not necessarily related to the goal conﬁguration.

Original: In the “partial” dataset, a subset of the dataset is guaranteed to solve the task, meaning an imitation learning agent may learn by selectively choosing the right subsets of the data.
Revised: In the “partial” dataset, a subset of the dataset is guaranteed to solve the task, meaning an imitation learning agent may learn by selectively choosing the right subsets of the data.

Original: The “mixed” dataset contains no trajectories which solve the task completely, and the RL agent must learn to assemble the relevant sub-trajectories.
Revised: The “mixed” dataset contains no trajectories which solve the task completely, and the RL agent must learn to assemble the relevant sub-trajectories.

Original: This dataset requires the highest degree of generalization in order to succeed.
Revised: This dataset requires the highest degree of generalization in order to succeed.

Original: Flow.
Revised: Flow.

Original: (Non-representable policies, realistic)
Revised: (Non-representable policies, realistic)

Original: The Flow benchmark (Vinitsky et al., 2018) is a framework for studying trafﬁc control using deep reinforcement learning.
Revised: The Flow benchmark (Vinitsky et al., 2018) is a framework for studying trafﬁc control using deep reinforcement learning.

Original: We use two tasks in the Flow benchmark which involve controlling autonomous vehicles to maximize the ﬂow of trafﬁc through a ring or merge road conﬁguration (left).
Revised: We use two tasks in the Flow benchmark which involve controlling autonomous vehicles to maximize the ﬂow of trafﬁc through a ring or merge road conﬁguration (left).

Original: We use the Flow domain in order to provide a task that simulates real-world trafﬁc dynamics.
Revised: We use the Flow domain in order to provide a task that simulates real-world trafﬁc dynamics.

Original: A large challenge in autonomous driving is to be able to directly learn from human behavior.
Revised: A large challenge in autonomous driving is to be able to directly learn from human behavior.

Original: Thus, we include “human” data from agents controlled by the intelligent driver model (IDM) (Treiber et al., 2000), a hand-designed model of human driving behavior.
Revised: Thus, we include “human” data from agents controlled by the intelligent driver model (IDM) (Treiber et al., 2000), a hand-designed model of human driving behavior.

Original: In order to provide data with a wider distribution as a reference, we also include “random” data generated from an agent that commands random vehicle accelerations.
Revised: In order to provide data with a wider distribution as a reference, we also include “random” data generated from an agent that commands random vehicle accelerations.

Original: Ofﬂine CARLA.
Revised: Ofﬂine CARLA.

Original: (Partial observability, non-representable policies, undirected and multitask data, realistic) CARLA (Dosovitskiy et al., 2017) is a high-ﬁdelity autonomous driving simulator that has previously been used with imitation learning approaches (Rhinehart et al., 2018; Codevilla et al., 2018) from large, static datasets.
Revised: (Partial observability, non-representable policies, undirected and multitask data, realistic) CARLA (Dosovitskiy et al., 2017) is a high-ﬁdelity autonomous driving simulator that has previously been used with imitation learning approaches (Rhinehart et al., 2018; Codevilla et al., 2018) from large, static datasets.

Original: The agent controls the throttle (gas pedal), the steering, and the break pedal for the car, and receives 48x48 RGB images from the driver’s perspective as observations.
Revised: The agent controls the throttle (gas pedal), the steering, and the break pedal for the car, and receives 48x48 RGB images from the driver’s perspective as observations.

Original: We propose two tasks for ofﬂine RL: lane following within a ﬁgure eight path (shown to the right, top picture), and navigation within a small town (bottom picture).
Revised: We propose two tasks for ofﬂine RL: lane following within a ﬁgure eight path (shown to the right, top picture), and navigation within a small town (bottom picture).

Original: The principle challenge of the CARLA domain is partial observability and visual complexity, as all observations are provided as ﬁrst-person RGB images.
Revised: The principle challenge of the CARLA domain is partial observability and visual complexity, as all observations are provided as ﬁrst-person RGB images.

Original: The datasets in both tasks are generated via hand-designed controllers meant to emulate human driving - the lane-following task uses simple heuristics to avoid cars and keep the car within lane boundaries, whereas the navigation task layers an additional high-level controller on top that takes turns randomly at intersections.
Revised: The datasets in both tasks are generated via hand-designed controllers meant to emulate human driving - the lane-following task uses simple heuristics to avoid cars and keep the car within lane boundaries, whereas the navigation task layers an additional high-level controller on top that takes turns randomly at intersections.

Original: Similarly to the Maze2D and AntMaze domains, this dataset consists of undirected navigation data in order to test the “stitching” property, however, it is in a more perceptually challenging domain.
Revised: Similarly to the Maze2D and AntMaze domains, this dataset consists of undirected navigation data in order to test the “stitching” property, however, it is in a more perceptually challenging domain.

Original: Evaluation protocol.
Revised: Evaluation protocol.

Original: Previous work tunes hyperparameters with online evaluation inside the simulator, and as Wu et al.
Revised: Previous work tunes hyperparameters with online evaluation inside the simulator, and as Wu et al.

Original: (2019) show, the hyperparameters have a large impact on performance.
Revised: (2019) show, the hyperparameters have a large impact on performance.

Original: Unfortunately, extensive online evaluation is not practical in real-world applications and this leads to over optimistic performance expectations when the system is deployed in a truly ofﬂine setting.
Revised: Unfortunately, extensive online evaluation is not practical in real-world applications and this leads to over optimistic performance expectations when the system is deployed in a truly ofﬂine setting.

Original: To rectify this problem, we designate a subset of tasks in each domain as “training” tasks, where hyperparameter tuning is allowed, and another subset as “evaluation” tasks on which ﬁnal performance is measured (See Appendix D Table 5).
Revised: To rectify this problem, we designate a subset of tasks in each domain as “training” tasks, where hyperparameter tuning is allowed, and another subset as “evaluation” tasks on which ﬁnal performance is measured (See Appendix D Table 5).

Original: To facilitate comparison across tasks, we normalize scores for each environment roughly to the range between 0 and 100, by computing normalized score = 100 ∗
Revised: To facilitate comparison across tasks, we normalize scores for each environment roughly to the range between 0 and 100, by computing normalized score = 100 ∗

Original: [Table]
Revised: [Table]

Original: A normalized score of 0 corresponds to the average returns (over 100 episodes) of an agent taking actions uniformly at random across the action space.
Revised: A normalized score of 0 corresponds to the average returns (over 100 episodes) of an agent taking actions uniformly at random across the action space.

Original: A score of 100 corresponds to the average returns of a domain-speciﬁc expert.
Revised: A score of 100 corresponds to the average returns of a domain-speciﬁc expert.

Original: For Maze2D, and Flow domains, this corresponds to the performance of the hand-designed controller used to collect data.
Revised: For Maze2D, and Flow domains, this corresponds to the performance of the hand-designed controller used to collect data.

Original: For CARLA, AntMaze, and FrankaKitchen, we used an estimate of the maximum score possible.
Revised: For CARLA, AntMaze, and FrankaKitchen, we used an estimate of the maximum score possible.

Original: For Adroit, this corresponds to a policy trained with behavioral cloning on human-demonstrations and ﬁne-tuned with RL.
Revised: For Adroit, this corresponds to a policy trained with behavioral cloning on human-demonstrations and ﬁne-tuned with RL.

Original: For Gym-MuJoCo, this corresponds to a soft-actor critic (Haarnoja et al., 2018b) agent.
Revised: For Gym-MuJoCo, this corresponds to a soft-actor critic (Haarnoja et al., 2018b) agent.

Original:  B ENCHMARKING P RIOR M ETHODS
Revised:  B ENCHMARKING P RIOR M ETHODS

Original: We evaluated recently proposed ofﬂine RL algorithms and several baselines on our ofﬂine RL benchmarks.
Revised: We evaluated recently proposed ofﬂine RL algorithms and several baselines on our ofﬂine RL benchmarks.

Original: This evaluation (1) provides baselines as a reference for future work, and (2) identiﬁes shortcomings in existing ofﬂine RL algorithms in order to guide future research.
Revised: This evaluation (1) provides baselines as a reference for future work, and (2) identiﬁes shortcomings in existing ofﬂine RL algorithms in order to guide future research.

Original: The average normalized performance for all tasks is plotted in the ﬁgure to the right ( ∗ the scores for
Revised: The average normalized performance for all tasks is plotted in the ﬁgure to the right.

Original: CQL on CARLA are not yet available and replaced with zero for the purposes of computing the average).

Original: It is clear that as we move beyond simple tasks and data collection strategies, differences between algorithms are exacerbated and deﬁciencies in all algorithms are revealed.
Revised: It is clear that as we move beyond simple tasks and data collection strategies, differences between algorithms are exacerbated and deﬁciencies in all algorithms are revealed.

Original: See Appendix Table 2 for normalized results for all tasks, Appendix Table 3 for unnormalized scores, and Appendix E for experimental details.
Revised: See Appendix Table 2 for normalized results for all tasks, Appendix Table 3 for unnormalized scores, and Appendix E for experimental details.

Original: We evaluated behavioral cloning (BC), online and ofﬂine soft actor-critic (SAC) (Haarnoja et al., 2018b), bootstrapping error reduction (BEAR) (Kumar et al., 2019), and behavior-regularized actorcritic (BRAC) (Wu et al., 2019), advantage-weighted regression (AWR)
Revised: We evaluated behavioral cloning (BC), online and ofﬂine soft actor-critic (SAC) (Haarnoja et al., 2018b), bootstrapping error reduction (BEAR) (Kumar et al., 2019), and behavior-regularized actorcritic (BRAC) (Wu et al., 2019), advantage-weighted regression (AWR)

Original: (Peng et al., 2019), batchconstrained Q-learning (BCQ) (Fujimoto et al., 2018a), continuous action random ensemble mixtures (cREM) (Agarwal et al., 2019b), and AlgaeDICE (Nachum et al., 2019).
Revised: (Peng et al., 2019), batchconstrained Q-learning (BCQ) (Fujimoto et al., 2018a), continuous action random ensemble mixtures (cREM) (Agarwal et al., 2019b), and AlgaeDICE (Nachum et al., 2019).

Original: We note that REM was originally designed for discrete action spaces, and the continuous action version has not been developed extensively.
Revised: We note that REM was originally designed for discrete action spaces, and the continuous action version has not been developed extensively.

Original: In most domains, we expect online SAC to outperformofﬂine algorithms when given the same amount of data because this baseline is able to collect on-policy data.
Revised: In most domains, we expect online SAC to outperform algorithms when given the same amount of data because this baseline is able to collect on-policy data.

Original: There are a few exceptions, such as for environments where exploration challenges make it difﬁcult to ﬁnd high-reward states, such as the Adroit and maze domains.
Revised: There are a few exceptions, such as for environments where exploration challenges make it difﬁcult to ﬁnd high-reward states, such as the Adroit and maze domains.

Original: Overall, the benchmarked algorithms obtained the most success on datasets generated from an RLtrained policy, such as in the Adroit and Gym-MuJoCo domains.
Revised: Overall, the benchmarked algorithms obtained the most success on datasets generated from an RLtrained policy, such as in the Adroit and Gym-MuJoCo domains.

Original: In these domains, ofﬂine RL algorithms are able to match the behavior policy when given expert data, and outperform when given suboptimal data.
Revised: In these domains, ofﬂine RL algorithms are able to match the behavior policy when given expert data, and outperform when given suboptimal data.

Original: This positive result is expected, as it is the predominant setting in which these prior algorithms have been benchmarked in past work.
Revised: This positive result is expected, as it is the predominant setting in which these prior algorithms have been benchmarked in past work.

Original: Another positive result comes in the form of sparse reward tasks.
Revised: Another positive result comes in the form of sparse reward tasks.

Original: In particular, many methods were able to outperform the baseline online SAC method on the Adroit and AntMaze domains.
Revised: In particular, many methods were able to outperform the baseline online SAC method on the Adroit and AntMaze domains.

Original: This indicates that ofﬂine RL is a promising paradigm for overcoming exploration challenges, which has also been conﬁrmed in recent work (Nair et al., 2020).
Revised: This indicates that ofﬂine RL is a promising paradigm for overcoming exploration challenges, which has also been conﬁrmed in recent work (Nair et al., 2020).

Original: We also ﬁnd that conservative methods that constrain the policy to the dataset, such as BEAR, AWR, CQL, and BCQ, are able to handle biased and narrow data distributions well on domains such as Flow and Gym-MuJoCo.
Revised: We also ﬁnd that conservative methods that constrain the policy to the dataset, such as BEAR, AWR, CQL, and BCQ, are able to handle biased and narrow data distributions well on domains such as Flow and Gym-MuJoCo.

Original: Tasks with undirected data, such as the Maze2D, FrankaKitchen, CARLA and AntMaze domains, are challenging for existing methods.
Revised: Tasks with undirected data, such as the Maze2D, FrankaKitchen, CARLA and AntMaze domains, are challenging for existing methods.

Original: Even in the simpler Maze2D domain, the large maze provides a surprising challenge for most methods.
Revised: Even in the simpler Maze2D domain, the large maze provides a surprising challenge for most methods.

Original: However, the smaller instances of Maze2D and AntMaze are very much within reach of current algorithms.
Revised: However, the smaller instances of Maze2D and AntMaze are very much within reach of current algorithms.

Original: Mixture distributions (a form of non-representable policies) were also challenging for all algorithms we evaluated.
Revised: Mixture distributions (a form of non-representable policies) were also challenging for all algorithms we evaluated.

Original: For MuJoCo, even though the medium-expert data contains expert data, the algorithms performed roughly on-par with medium datasets, except for hopper.
Revised: For MuJoCo, even though the medium-expert data contains expert data, the algorithms performed roughly on-par with medium datasets, except for hopper.

Original: The same pattern was found in the cloned datasets for Adroit, where the algorithms mostly performed on-par with the limited demonstration dataset, even though they had access to additional data.
Revised: The same pattern was found in the cloned datasets for Adroit, where the algorithms mostly performed on-par with the limited demonstration dataset, even though they had access to additional data.

Original: We ﬁnd that many algorithms were able to succeed to some extent on tasks with controller-generated data, such as Flow and carla-lane.
Revised: We ﬁnd that many algorithms were able to succeed to some extent on tasks with controller-generated data, such as Flow and carla-lane.

Original: We also note that tasks with limited data, such as human demonstrations in Adroit and FrankaKitchen, remain challenging.
Revised: We also note that tasks with limited data, such as human demonstrations in Adroit and FrankaKitchen, remain challenging.

Original: This potentially points to the need for more sample-efﬁcient methods, as big datasets may not be always be available.
Revised: This potentially points to the need for more sample-efﬁcient methods, as big datasets may not be always be available.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We have proposed an open-source benchmark for ofﬂine RL.
Revised: We have proposed an open-source benchmark for ofﬂine RL.

Original: The choice of tasks and datasets were motivated by properties reﬂected in real-world applications, such as narrow data distributions and undirected, logged behavior.
Revised: The choice of tasks and datasets were motivated by properties reﬂected in real-world applications, such as narrow data distributions and undirected, logged behavior.

Original: Existing benchmarks have largely concentrated on robotic control using data produced by policies trained with RL (Fujimoto et al., 2018a; Kumar et al., 2019;
Revised: Existing benchmarks have largely concentrated on robotic control using data produced by policies trained with RL (Fujimoto et al., 2018a; Kumar et al., 2019;

Original: Wu et al., 2019; Gulcehre et al., 2020; Dulac-Arnold et al., 2020).
Revised: Wu et al., 2019; Gulcehre et al., 2020; Dulac-Arnold et al., 2020).

Original: This can give a misleading sense of progress, as we have shown in our experiments that many of the more challenging properties that we expect real-world datasets to have appear to result in a substantial challenge for existing methods.
Revised: This can give a misleading sense of progress, as we have shown in our experiments that many of the more challenging properties that we expect real-world datasets to have appear to result in a substantial challenge for existing methods.

Original: We believe that ofﬂine RL holds great promise as a potential paradigm to leverage vast amounts of existing sequential data within the ﬂexible decision making framework of reinforcement learning.
Revised: We believe that ofﬂine RL holds great promise as a potential paradigm to leverage vast amounts of existing sequential data within the ﬂexible decision making framework of reinforcement learning.

Original: We hope that providing a benchmark that is representative of potential problems in ofﬂine RL, but that still can be accessibly evaluated in simulation, will greatly accelerate progress in this ﬁeld and create new opportunities to apply RL in many real-world application areas.
Revised: We hope that providing a benchmark that is representative of potential problems in ofﬂine RL, but that still can be accessibly evaluated in simulation, will greatly accelerate progress in this ﬁeld and create new opportunities to apply RL in many real-world application areas.

Original: There are several important properties exhibited in some real-world applications of RL that are not explored in-depth in our benchmark.
Revised: There are several important properties exhibited in some real-world applications of RL that are not explored in-depth in our benchmark.

Original: One is stochasticity of the environment, which can occur in systems such as ﬁnancial markets, healthcare, or advertisement.
Revised: One is stochasticity of the environment, which can occur in systems such as ﬁnancial markets, healthcare, or advertisement.

Original: And while we explore domains with large observation spaces, some domains such recommender systems can exhibit large action spaces.
Revised: And while we explore domains with large observation spaces, some domains such recommender systems can exhibit large action spaces.

Original: Finally, our tasks are predominantly in the robotics, autonomous driving, and trafﬁc control.
Revised: Finally, our tasks are predominantly in the robotics, autonomous driving, and trafﬁc control.

Original: There are several other areas (e.g. in ﬁnance, industrial, or operations research) which could provide simulated, yet still challenging, benchmarks.
Revised: There are several other areas (e.g. in ﬁnance, industrial, or operations research) which could provide simulated, yet still challenging, benchmarks.

Original: Ultimately, we would like to see ofﬂine RL applications move from simulated domains to real-world domains, using real-world datasets.
Revised: Ultimately, we would like to see ofﬂine RL applications move from simulated domains to real-world domains, using real-world datasets.

Original: This includes exciting areas such as recommender systems, where user behavior can be easily logged, and medicine, where doctors must keep complete medical records for each patient.
Revised: This includes exciting areas such as recommender systems, where user behavior can be easily logged, and medicine, where doctors must keep complete medical records for each patient.

Original: A key challenge in these domains is that reliable evaluation must be done in a real system or using off-policy evaluation (OPE) methods.
Revised: A key challenge in these domains is that reliable evaluation must be done in a real system or using off-policy evaluation (OPE) methods.

Original: We believe that both reliable OPE methods and real-world benchmarks which are standardized across different research groups, will be important to establish for future benchmarks in ofﬂine RL.
Revised: We believe that both reliable OPE methods and real-world benchmarks which are standardized across different research groups, will be important to establish for future benchmarks in ofﬂine RL.

Original: Appendices A T ASK P ROPERTIES
Revised: Appendices A T ASK P ROPERTIES

Original: The following is a full list of task properties and dataset statistics for all tasks in the benchmark.
Revised: The following is a full list of task properties and dataset statistics for all tasks in the benchmark.

Original: Note that the full dataset for “carla-town” requires over 30GB of memory to store, so we also provide a subsampled version of the dataset which we used in our experiments.
Revised: Note that the full dataset for “carla-town” requires over 30GB of memory to store, so we also provide a subsampled version of the dataset which we used in our experiments.

Original: Domain
Revised: Domain

Original: [Table]
Revised: [Table]

Original: Maze2D
Revised: Maze2D

Original: [Table]
Revised: [Table]

Original: diverse
Revised: diverse

Original: Planner
Revised: Planner

Original: [Table]
Revised: [Table]

Original: Planner
Revised: Planner

Original: [Table]
Revised: [Table]

Original: diverse
Revised: diverse

Original: Planner
Revised: Planner

Original: [Table]
Revised: [Table]

Original: antmaze
Revised: antmaze

Original: [Table]
Revised: [Table]

Original: large
Revised: large

Original: [Table]
Revised: [Table]

Original: Planner
Revised: Planner

Original: [Table]
Revised: [Table]

Original: antmaze
Revised: antmaze

Original: [Table]
Revised: [Table]

Original: large
Revised: large

Original: [Table]
Revised: [Table]

Original: diverse
Revised: diverse

Original: Planner
Revised: Planner

Original: [Table]
Revised: [Table]

Original: Gym-MuJoCo hopper-random
Revised: Gym-MuJoCo hopper-random

Original: Policy
Revised: Policy

Original: [Table]
Revised: [Table]

Original: hopper-medium
Revised: hopper-medium

Original: Policy
Revised: Policy

Original: [Table]
Revised: [Table]

Original: hopper
Revised: hopper

Original: [Table]
Revised: [Table]

Original: medium
Revised: medium

Original: [Table]
Revised: [Table]

Original: replay
Revised: replay

Original: Policy hopper
Revised: Policy hopper

Original: [Table]
Revised: [Table]

Original: medium
Revised: medium

Original: [Table]
Revised: [Table]

Original: Policy 2 × 10halfcheetah
Revised: Policy 2 × 10halfcheetah

Original: [Table]
Revised: [Table]

Original: random
Revised: random

Original: Policy 10halfcheetah
Revised: Policy 10halfcheetah

Original: [Table]
Revised: [Table]

Original: medium
Revised: medium

Original: Policy 10halfcheetah
Revised: Policy 10halfcheetah

Original: [Table]
Revised: [Table]

Original: Policy halfcheetah
Revised: Policy halfcheetah

Original: [Table]
Revised: [Table]

Original: Policy
Revised: Policy

Original: [Table]
Revised: [Table]

Original: walker2d
Revised: walker2d

Original: [Table]
Revised: [Table]

Original: Policy 10walker2d-medium
Revised: Policy 10walker2d-medium

Original: Policy 10walker2d
Revised: Policy 10walker2d

Original: [Table]
Revised: [Table]

Original: Policy walker2d-medium-expert
Revised: Policy walker2d-medium-expert

Original: Policy 2 × 

Revised: Policy 2 ×

Original: Adroit pen-human
Revised: Adroit pen-human

Original: Human pen-cloned
Revised: Human pen-cloned

Original: Policy 5 ∗ 10pen-expert
Revised: Policy 5 ∗ 10pen-expert

Original: Policy 5 ∗ 10hammer-human
Revised: Policy 5 ∗ 10hammer-human

Original: Human hammer-cloned
Revised: Human hammer-cloned

Original: Policy 10hammer-expert
Revised: Policy 10hammer-expert

Original: Policy 10door-human
Revised: Policy 10door-human

Original: Human door-cloned
Revised: Human door-cloned

Original: Policy 10door-expert
Revised: Policy 10door-expert

Original: Policy 10relocate-human
Revised: Policy 10relocate-human

Original: Human relocate-cloned
Revised: Human relocate-cloned

Original: Policy 10relocate-expert
Revised: Policy 10relocate-expert

Original: Policy 

Revised: Policy

Original: Flow ﬂow-ring-random
Revised: Flow ﬂow-ring-random

Original: Policy
Revised: Policy

Original: [Table]
Revised: [Table]

Original: B R ESULTS BY D OMAIN
Revised: B R ESULTS BY D OMAIN

Original: The following tables summarize performance for each domain (excluding CARLA), sorted by the best performing algorithm to the worst.
Revised: The following tables summarize performance for each domain (excluding CARLA, due to all algorithms performing poorly), sorted by the best performing algorithm to the worst.

Original: T a s k
Revised: T a s k

Original: N a m e
Revised: N a m e

Original: S AC
Revised: S AC

Original: B C
Revised: B C

Original: S AC - o ff
Revised: S AC - o ff

Original: BE
Revised: BE

Original: AR
Revised: AR

Original: B
Revised: B

Original: RAC - p
Revised: RAC - p

Original: B
Revised: B

Original: RAC - v
Revised: RAC - v

Original: A W R c R E
Revised: A W R c R E

Original: M
Revised: M

Original: B C Q a D
Revised: B C Q a D

Original: I C
Revised: I C

Original: E
Revised: E

Original: C Q L
Revised: C Q L

Original: M aze 2 D m aze 2d - u m aze 62 .
Revised: M aze 2 D m aze 2d - u m aze 62 .

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: m aze 2d - m e d i u m
Revised: m aze 2d - m e d i u m

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: m aze 2d
Revised: m aze 2d

Original: [Table]
Revised: [Table]

Original: r
Revised: r

Original: [Table]
Revised: [Table]

Original: e
Revised: e

Original: [Table]
Revised: [Table]

Original: A
Revised: A

Original: [Table]
Revised: [Table]

Original: M aze
Revised: M aze

Original: [Table]
Revised: [Table]

Original: m aze
Revised: m aze

Original: [Table]
Revised: [Table]

Original: m aze
Revised: m aze

Original: [Table] [Table]
Revised: [Table] [Table]

Original: aze
Revised: aze

Original: [Table]
Revised: [Table]

Original: aze -
Revised: aze -

Original: [Table]
Revised: [Table]

Original: v e r
Revised: v e r

Original: [Table]
Revised: [Table]

Original: e 0 .
Revised: e 0 .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: a n t m aze
Revised: a n t m aze

Original: [Table]
Revised: [Table]

Original: m e d i u m - p l a y 0 .0 .0 .0 .0 .0 .0 .
Revised: m e d i u m - p l a y 0 .0 .0 .0 .0 .0 .0 .

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: a n t m aze
Revised: a n t m aze

Original: [Table]
Revised: [Table]

Original: m e d i u m - d i v e r s e 0 .0 .0 .8 .0 .0 .0 .0 .0 .

Revised: m e d i u m - d i v e r s e 0 .0 .0 .8 .0 .0 .0 .0 .0 .

Original: .53 .a
Revised: .53 .a

Original: n t m aze - l a r g e - p l a y 0 .0 .0 .0 .0 .0 .0 .

Revised: n t m aze - l a r g e - p l a y 0 .0 .0 .0 .0 .0 .0 .

Original: .15 .a
Revised: .15 .a

Original: n t m aze - l a r g e - d i v e r s e 0 .0 .0 .0 .0 .0 .0 .0 .2 .

Revised: n t m aze - l a r g e - d i v e r s e 0 .0 .0 .0 .0 .0 .0 .0 .2 .

Original: G y m h a l f c h ee t a h -r a ndo m 

Revised: G y m h a l f c h ee t a h -r a ndo m

Original: .35 .w
Revised: .35 .w

Original: a l k e r 2d -r a ndo m 100 .1 .4 .7 .- 0 .1 .1 .- 

Revised: a l k e r 2d -r a ndo m 100 .1 .4 .7 .- 0 .1 .1 .-

Original: .hopp
Revised: .hopp

Original: e r-r a ndo m 100 .

Revised: e r-r a ndo m 100 .

Original: .10 .h
Revised: .10 .h

Original: a l f c h ee t a hm e d i u m 100 .36 .-
Revised: a l f c h ee t a hm e d i u m 100 .36 .-

Original: 2 .40 .- 2 .44 .w
Revised: 2 .40 .- 2 .44 .w

Original: a l k e r 2d - m e d i u m 100 .6 .

Revised: a l k e r 2d - m e d i u m 100 .6 .

Original: .hopp e rm e d
Revised: .hopp e rm e d

Original: i
Revised: i

Original: u m
Revised: u m

Original: .58 .h
Revised: .58 .h

Original: a l f c h ee t a hm e d i u m -r e p l a y 100 .38 .-
Revised: a l f c h ee t a hm e d i u m -r e p l a y 100 .38 .-

Original: 2 .46 .w
Revised: 2 .46 .w

Original: a l k e r 2d - m e d i u m -r e p l a y 100 .11 .

Revised: a l k e r 2d - m e d i u m -r e p l a y 100 .11 .

Original: .26 .hopp
Revised: .26 .hopp

Original: e rm e d i u m -r e p l a y 100 .11 .3 .33 .0 .

Revised: e rm e d i u m -r e p l a y 100 .11 .3 .33 .0 .

Original: .48 .h
Revised: .48 .h

Original: a l f c h ee t a hm e d i u m - e x p e r t 100 .35 .

Revised: a l f c h ee t a hm e d i u m - e x p e r t 100 .35 .

Original: .62 .w
Revised: .62 .w

Original: a l k e r 2d - m e d i u m - e xp e r t 100 .6 .- 

Revised: a l k e r 2d - m e d i u m - e xp e r t 100 .6 .-

Original: .hopp e rm e d i u m - e xp e r t 100 .111 .

Revised: .hopp e rm e d i u m - e xp e r t 100 .111 .

Original: A d r o it p e nhu m a n 21 .34 .6 .-
Revised: A d r o it p e nhu m a n 21 .34 .6 .-

Original: 3 .37 .h
Revised: 3 .37 .h

Original: a mm e r- hu
Revised: a mm e r- hu

Original: m a n 0 .1 .0 .0 .

Revised: m a n 0 .1 .0 .0 .

Original: .4 .doo
Revised: .4 .doo

Original: r-
Revised: r-

Original: hu
Revised: hu

Original: m a n - 0 .0 .3 .- 0 .- 0 .- 0 .0 .-
Revised: m a n - 0 .0 .3 .- 0 .- 0 .- 0 .0 .-

Original: 0 .- 0 .- 0 .9 .r
Revised: 0 .- 0 .- 0 .9 .r

Original: e l o ca t ehu m a n - 0 .0 .0 .- 0 .- 0 .- 0 .- 0 .- 0 .- 0 .- 0 .0 .p
Revised: e l o ca t ehu m a n - 0 .0 .0 .- 0 .- 0 .- 0 .- 0 .- 0 .- 0 .- 0 .0 .p

Original: e n - c l on e d 21 .56 .23 .26 .1 .-
Revised: e n - c l on e d 21 .56 .23 .26 .1 .-

Original: 2 .39 .h
Revised: 2 .39 .h

Original: a mm e r-
Revised: a mm e r-

Original: c l on e d 0 .0 .0 .0 .0 .0 .0 .0 .0 .

Revised: c l on e d 0 .0 .0 .0 .0 .0 .0 .0 .0 .

Original: .2 .doo
Revised: .2 .doo

Original: r-
Revised: r-

Original: c l on e d - 0 .- 0 .0 .- 0 .- 0 .- 0 .0 .- 0 .0 .0 .0 .r
Revised: c l on e d - 0 .- 0 .0 .- 0 .- 0 .- 0 .0 .- 0 .0 .0 .0 .r

Original: e l o ca t e - c l on e d - 0 .- 0 .- 0 .- 0 .- 0 .- 0 .- 0 .- 0 .- 0 .- 0 .- 

Revised: e l o ca t e - c l on e d - 0 .- 0 .- 0 .- 0 .- 0 .- 0 .- 0 .- 0 .- 0 .- 0 .-

Original: .p
Revised: .p

Original: e n - e xp e r t 21 .85 .6 .105 .-
Revised: e n - e xp e r t 21 .85 .6 .105 .-

Original: 3 .107 .h
Revised: 3 .107 .h

Original: a mm e r-
Revised: a mm e r-

Original: e xp e r t 0 .125 .25 .127 .0 .

Revised: e xp e r t 0 .125 .25 .127 .0 .

Original: .86 .doo r-
Revised: .86 .doo r-

Original: e xp e r t - 0 .34 .7 .103 .- 0 .- 0 .102 .- 

Revised: e xp e r t - 0 .34 .7 .103 .- 0 .- 0 .102 .-

Original: .99 .0 .101 .r
Revised: .99 .0 .101 .r

Original: e l o ca t e - e xp e r t - 0 .101 .- 

Revised: e l o ca t e - e xp e r t - 0 .101 .-

Original: F l o w ﬂ o w -r i ng - c on t r o ll e r 100 .- 57 .

Revised: F l o w ﬂ o w -r i ng - c on t r o ll e r 100 .- 57 .

Original: .76 .15 .52 .ﬂ
Revised: .76 .15 .52 .ﬂ

Original: o
Revised: o

Original: w -r i
Revised: w -r i

Original: ng -r a ndo m 100 .94 .70 .

Revised: ng -r a ndo m 100 .94 .70 .

Original: 87 .94 .83 .87 .ﬂ
Revised: 87 .94 .83 .87 .ﬂ

Original: o wm e r g e - c on t r o ll e r 121 .114 .111 .150 .129 .143 .152 .183 .114 .

Revised: o wm e r g e - c on t r o ll e r 121 .114 .111 .150 .129 .143 .152 .183 .114 .

Original: .157 .ﬂ
Revised: .157 .ﬂ

Original: o wm e r g e -r a ndo m 121 .- 17 .-
Revised: o wm e r g e -r a ndo m 121 .- 17 .-

Original: F r a nk a K it c h e n k
Revised: F r a nk a K it c h e n k

Original: it c h e n - c
Revised: it c h e n - c

Original: o m p l e t e 0 .

Revised: o m p l e t e 0 .

Original: .43 .k
Revised: .43 .k

Original: it c h e n - p a r ti a l 0 .33 .

Revised: it c h e n - p a r ti a l 0 .33 .

Original: .49 .k
Revised: .49 .k

Original: it c h e n - m i x e d 0 .47 .

Revised: it c h e n - m i x e d 0 .47 .

Original: C A R
Revised: C A R

Original: L A ca r l a - l a n e - 0 .31 .0 .- 

Revised: L A ca r l a - l a n e - 0 .31 .0 .-

Original: ca r l a - t o w n 1 .- 1 .-1 .- 2 .- 4 .- 2 .1 .- 0 .1 .-
Revised: 1 .20 .ca r l a - t o w n 1 .- 1 .-

Original: T a b l e 2 :
Revised: T a b l e 2 :

Original: N o r m a l i ze
Revised: N o r m a l i ze

Original: d r e
Revised: d r e

Original: s u
Revised: s u

Original: lt s c o m p a r i ng on li n e & o f ﬂ i n e S
Revised: lt s c o m p a r i ng on li n e & o f ﬂ i n e S

Original: A C ( S A
Revised: A C ( S A

Original: C , S
Revised: C , S

Original: A C - o ff) , boo t s t r a
Revised: A C - o ff) , boo t s t r a

Original: pp
Revised: pp

Original: i ng e rr o r r e
Revised: i ng e rr o r r e

Original: du
Revised: du

Original: c ti on ( B
Revised: c ti on ( B

Original: E A
Revised: E A

Original: R ) , b
Revised: R ) , b

Original: e h a v i o r-r e gu l a r i ze d ac t
Revised: e h a v i o r-r e gu l a r i ze d ac t

Original: o r c
Revised: o r c

Original: r
Revised: r

Original: iti c w it
Revised: iti c w it

Original: h po li c y ( BR
Revised: h po li c y ( BR

Original: A C - p )
Revised: A C - p )

Original: o r v a l u e (
Revised: o r v a l u e (

Original: BR
Revised: BR

Original: A C - v ) r e gu l a r i za ti on , b e h a v
Revised: A C - v ) r e gu l a r i za ti on , b e h a v

Original: i
Revised: i

Original: o r a l c l on i ng ( BC ) , a dv a n t a g
Revised: o r a l c l on i ng ( BC ) , a dv a n t a g

Original: ew e i gh t e d r e g r e ss i on (
Revised: ew e i gh t e d r e g r e ss i on (

Original: A W
Revised: A W

Original: R ) , b a t c h - c on s t r a
Revised: R ) , b a t c h - c on s t r a

Original: i n e d
Revised: i n e d

Original: Q - l ea r n i ng (
Revised: Q - l ea r n i ng (

Original: BC
Revised: BC

Original: Q )
Revised: Q )

Original: , c on ti nuou s r a ndo m e n s e m b l e
Revised: , c on ti nuou s r a ndo m e n s e m b l e

Original: m i x t u r e s ( c R
Revised: m i x t u r e s ( c R

Original: E M ) , a nd
Revised: E M ) , a nd

Original: A l g ae D
Revised: A l g ae D

Original: I
Revised: I

Original: C E ( a D
Revised: C E ( a D

Original: I C E ) .
Revised: I C E ) .

Original: A v e r a g e r e s u lt s a r e r e po
Revised: A v e r a g e r e s u lt s a r e r e po

Original: r t e d ov e r 3 s ee d s , a nd n o r m a li ze d t o a s c o r e b e t w ee n 0 (r a ndo m ) a nd ( e xp e r t ) .
Revised: r t e d ov e r 3 s ee d s , a nd n o r m a li ze d t o a s c o r e b e t w ee n 0 (r a ndo m ) a nd ( e xp e r t ) .

Original: D o m a i n
Revised: D o m a i n

Original: T a s k
Revised: T a s k

Original: N a m e
Revised: N a m e

Original: S AC
Revised: S AC

Original: B C
Revised: B C

Original: S AC - o ff
Revised: S AC - o ff

Original: BE
Revised: BE

Original: AR
Revised: AR

Original: B
Revised: B

Original: RAC - p
Revised: RAC - p

Original: B
Revised: B

Original: RAC - v
Revised: RAC - v

Original: A W R c R E
Revised: A W R c R E

Original: M
Revised: M

Original: B C Q
Revised: B C Q

Original: A l ga e D I C
Revised: A l ga e D I C

Original: E
Revised: E

Original: C Q L
Revised: C Q L

Original: M aze 2 D m aze 2d - u m aze 110 .29 .

Revised: M aze 2 D m aze 2d - u m aze 110 .29 .

Original: .31 .m
Revised: .31 .m

Original: aze 2d - m e d i u m 69 .

Revised: aze 2d - m e d i u m 69 .

Original: [Table] [Table] [Table] [Table]
Revised: .39 .26 .m

Original: m aze 2d - l a r g e 14 .20 .

Revised: aze 2d - l a r g e 14 .20 .

Original: [Table]

Original: A n t M aze a n t m aze - u m aze 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .a
Revised: A n t M aze a n t m aze - u m aze 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .a

Original: n t m aze - u m aze - d i v e r s e 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .a
Revised: n t m aze - u m aze - d i v e r s e 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .a

Original: n t m aze - m e d i u m - p l a y 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .a n t m aze - m e d i u m - d i v e r s e 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .a
Revised: n t m aze - m e d i u m - p l a y 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .a n t m aze - m e d i u m - d i v e r s e 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .a

Original: n t m aze - l a r g e - p l a y 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .a n t m aze - l a r g e - d i v e r s e 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .
Revised: n t m aze - l a r g e - p l a y 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .a n t m aze - l a r g e - d i v e r s e 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .

Original: G y m h a l f c h ee t a h -r a ndo m .-
Revised: G y m h a l f c h ee t a h -r a ndo m .-

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: w a l k e r 2d -r a ndo m 
Revised: w a l k e r 2d -r a ndo m 4592 .73 .192 .

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: hopp e r-r a ndo m
Revised: hopp e r-r a ndo m

Original: [Table]
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: h a l f c h ee t a hm e d i u m .
Revised: h a l f c h ee t a hm e d i u m .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: w a l k e r 2d - m e d i u m 4592.304 .

Revised: 5232 .w a l k e r 2d - m e d i u m 4592 .

Original: [Table]

Original: .hopp e rm e d i u m 3234 .923 .5 .1674 .1044 .

Revised: .hopp e rm e d i u m 3234 .923 .5 .1674 .1044 .

Original: .17 .2557 .h
Revised: .17 .2557 .h

Original: a l f c h ee t a hm e d i u m -r e p l a y .4492 .-.

Revised: a l f c h ee t a hm e d i u m -r e p l a y .4492 .-.

Original: .-.5455 .w
Revised: .-.5455 .w

Original: a l k e r 2d - m e d i u m -r e p l a y 4592 .

Revised: a l k e r 2d - m e d i u m -r e p l a y 4592 .

Original: .31 .1227 .hopp
Revised: .31 .1227 .hopp

Original: e rm e d i u m -r e p l a y 3234 .364 .93 .1076 .0 .- 0 .904 .

Revised: e rm e d i u m -r e p l a y 3234 .364 .93 .1076 .0 .- 0 .904 .

Original: .14 .1227 .h
Revised: .14 .1227 .h

Original: a l f c h ee t a hm e d i u m - e xp e r t .4169 .-
Revised: a l f c h ee t a hm e d i u m - e xp e r t .4169 .-

Original: .-.7466 .w
Revised: .-.7466 .w

Original: a l k e r 2d - m e d i u m - e xp e r t 4592 .297 .-
Revised: a l k e r 2d - m e d i u m - e xp e r t 4592 .297 .-

Original: .hopp
Revised: .hopp

Original: e rm e d i u m - e xp e r t3234 .
Revised: e rm e d i u m - e xp e r t

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: A d r o it p e nhu m a n

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: e

Original: [Table]
Revised: [Table]

Original: hu
Revised: hu

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: doo r-

Original: hu m

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: r e l o ca t ehu m a n - 13 . [Table] [Table] [Table]
Revised: hu

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: p e n - c l on e d

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: h a mm e r-

Original: c l on e d [Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: doo r-

Original: c l on e d - 61 .- 60 .- 56 .-

Original: [Table]
Revised: [Table]

Original: re l o ca t e - c l on e d - 13 . [Table] [Table] [Table]
Revised: r

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: on

Original: [Table]

Original: [Table]
Revised: [Table]

Original: p e n - e xp e r t

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: r

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: doo r-

Original: e xp e r t

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original:  re l o
Revised: doo r

Original: [Table]
Revised: [Table]

Original: t e- e xp e r t [Table] [Table] [Table]
Revised: on e

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: F l o w ﬂ

Original: [Table]
Revised: [Table]

Original: w -r i ng - c on t r o ll e r

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: w

Original: [Table]
Revised: [Table]

Original: r i ng -r

Original: [Table]
Revised: [Table]

Original: ndo m

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: - 6 .15 .ﬂ
Revised: - 6 .15 .ﬂ

Original: o wm e r g e - c on t r o ll e r 375 .359 .354 .436 .392 .422 .441 .505 .361 .533 .450 .ﬂ
Revised: o wm e r g e - c on t r o ll e r 375 .359 .354 .436 .392 .422 .441 .505 .361 .533 .450 .ﬂ

Original: o wm e
Revised: o wm e

Original: r g e -r a ndo m
Revised: r g e -r a ndo m

Original: F r a nk a K it c h e n k
Revised: F r a nk a K it c h e n k

Original: it c h e n - c
Revised: it c h e n - c

Original: o m p l e t e 0 .1 .0 .0 .0 .0 .0 .0 .0 .0 .

Revised: o m p l e t e 0 .1 .0 .0 .0 .0 .0 .0 .0 .0 .

Original: .k
Revised: .k

Original: it c h e n - p a r ti a l 0 .1 .0 .0 .0 .0 .0 .0 .0 .0 .

Revised: it c h e n - p a r ti a l 0 .1 .0 .0 .0 .0 .0 .0 .0 .0 .

Original: .k
Revised: .k

Original: it c h e n - m i x e d 0 .1 .0 .1 .0 .0 .0 .0 .0 .

Revised: it c h e n - m i x e d 0 .1 .0 .1 .0 .0 .0 .0 .0 .

Original: O f ﬂ i n e C A
Revised: O f ﬂ i n e C A

Original: R L A ca r l a - l a n e - 8 .324 .- 0 .-
Revised: R L A ca r l a - l a n e - 8 .324 .- 0 .-

Revised: 1 .- 13 .213 .ca

Original: ca r l a - t o w n - 79 .-.-.-.-.-.- 65 .-.- 66 .-.-
Revised: r l a - t o w n - 79 .-.-.-.-.-.- 65 .-.- 66 .-.-.

Original: T a b l e
Revised: T a b l e

Original: T h
Revised: T h

Original: e r a w , un - no r m a li
Revised: e r a w , un - no r m a li

Original: ze d s c
Revised: ze d s c

Original: o r e s f o r eac h t
Revised: o r e s f o r eac h t

Original: a s k a nd a l go r
Revised: a s k a nd a l go r

Original: it
Revised: it

Original: h
Revised: h

Original: m a r e r e po r t e d i n t h e t
Revised: m a r e r e po r t e d i n t h e t

Original: a b
Revised: a b

Original: l e b e l o w .
Revised: l e b e l o w .

Original: T
Revised: T

Original: h
Revised: h

Original: e s e s c
Revised: e s e s c

Original: o r e s r e p r e s e n t t
Revised: o r e s r e p r e s e n t t

Original: h e und i s c
Revised: h e und i s c

Original: oun t e d r e t u r n ob t a i n e d fr o m e x ec u ti ng a po li c
Revised: oun t e d r e t u r n ob t a i n e d fr o m e x ec u ti ng a po li c

Original: y i n t
Revised: y i n t

Original: h e s i m u l a t o r , a v e r a g
Revised: h e s i m u l a t o r , a v e r a g

Original: e d ov e r 3 r a ndo m s ee d s .
Revised: e d ov e r 3 r a ndo m s ee d s .

Original: C T ASK AND D ATASETS
Revised: C T ASK AND D ATASETS

Original: The following table lists the tasks and dataset types included in the benchmark, including sources for each.
Revised: The following table lists the tasks and dataset types included in the benchmark, including sources for each.

Original: Domain
Revised: Domain

Original: [Table]
Revised: [Table]

Original: D T RAINING AND E VALUATION T ASK S PLIT
Revised: D T RAINING AND E VALUATION T ASK S PLIT

Original: The following table lists our recommended protocol for hyperparameter tuning.
Revised: The following table lists our recommended protocol for hyperparameter tuning.

Original: Hyperparameters should be tuned on the tasks listed on the left in the “Training” column, and algorithms should be evaluated without tuning on the tasks in the right column labeled “Evaluation”.
Revised: Hyperparameters should be tuned on the tasks listed on the left in the “Training” column, and algorithms should be evaluated without tuning on the tasks in the right column labeled “Evaluation”.

Original: [Table]
Revised: [Table]

Original: E E XPERIMENT D ETAILS
Revised: E E XPERIMENT D ETAILS

Original: For all experiments, we used default hyperparameter settings and minimal modiﬁcations to public implementations wherever possible, using 500K training iterations or gradient steps.
Revised: For all experiments, we used default hyperparameter settings and minimal modiﬁcations to public implementations wherever possible, using 500K training iterations or gradient steps.

Original: The code bases we used for evaluation are listed below.
Revised: The code bases we used for evaluation are listed below.

Original: The most signiﬁcant deviation from original published algorithms was that we used an unofﬁcial continuous-action implementation of REM (Agarwal et al., 2019b), which was originally implemented for discrete action spaces.
Revised: The most signiﬁcant deviation from original published algorithms was that we used an unofﬁcial continuous-action implementation of REM (Agarwal et al., 2019b), which was originally implemented for discrete action spaces.

Original: We ran our experiments using
Revised: We ran our experiments using

Original: Google cloud platform (GCP) on n1-standard-4 machines.
Revised: Google cloud platform (GCP) on n1-standard-4 machines.

Original: • BRAC and AlgaeDICE: https://github.com/google-research/ google-research • AWR:
Revised: • BRAC and AlgaeDICE: https://github.com/google-research/ google-research • AWR:

Original: https://github.com/xbpeng/awr • SAC: https://github.com/vitchyr/rlkit • BEAR: https://github.com/aviralkumar2907/BEAR • Continuous-action REM: https://github.com/theSparta/off_policy_ mujoco • BCQ: https://github.com/sfujim/BCQ • CQL: https://github.com/aviralkumar2907/CQL
Revised: https://github.com/xbpeng/awr • SAC: https://github.com/vitchyr/rlkit • BEAR: https://github.com/aviralkumar2907/BEAR • Continuous-action REM: https://github.com/theSparta/off_policy_ mujoco • BCQ: https://github.com/sfujim/BCQ • CQL: https://github.com/aviralkumar2907/CQL

Original: F A SSESSING THE F EASIBILITY OF H ARD T ASKS
Revised: F A SSESSING THE F EASIBILITY OF H ARD T ASKS

Original: [Figure]
Revised: [Figure]

Original: Few prior methods were able to successfully solve carlatown or the larger AntMaze tasks.
Revised: Few prior methods were able to successfully solve carlatown or the larger AntMaze tasks.

Original: While including tasks that present a challenge for current methods is important to ensure that our benchmark has room for improvement, it is also important to provide some conﬁdence that the tasks are actually solvable.
Revised: While including tasks that present a challenge for current methods is important to ensure that our benchmark has room for improvement, it is also important to provide some conﬁdence that the tasks are actually solvable.

Original: We veriﬁed this in two ways.
Revised: We veriﬁed this in two ways.

Original: First, we ensured that the trajectories observed in these tasks have adequate coverage of the state space.
Revised: First, we ensured that the trajectories observed in these tasks have adequate coverage of the state space.

Original: An illustration of the trajectories in the CARLA and AntMaze tasks are shown below, where trajectories are shown as different colored lines and the goal state is marked with a star.
Revised: An illustration of the trajectories in the CARLA and AntMaze tasks are shown below, where trajectories are shown as different colored lines and the goal state is marked with a star.

Original: We see that in carla-town and AntMaze, each lane or corridor is traversed multiple times by the agent.
Revised: We see that in carla-town and AntMaze, each lane or corridor is traversed multiple times by the agent.

Original: Second, the data in AntMaze was generated by having the ant follow the same high-level planner in the maze as in Maze2D. Because Maze2D is solvable by most methods, we would expect this to mean that AntMaze is potentially solvable as well.
Revised: Second, the data in AntMaze was generated by having the ant follow the same high-level planner in the maze as in Maze2D. Because Maze2D is solvable by most methods, we would expect this to mean that AntMaze is potentially solvable as well.

Original: This While the dynamics of the ant itself are much more complex, its walking gait is a relatively regular periodic motion, and since the high-level waypoints are similar, we would expect the AntMaze data to provide similar coverage as in the 2D mazes, as shown in the ﬁgures on the right.
Revised: This While the dynamics of the ant itself are much more complex, its walking gait is a relatively regular periodic motion, and since the high-level waypoints are similar, we would expect the AntMaze data to provide similar coverage as in the 2D mazes, as shown in the ﬁgures on the right.

Original: While the Ant has more erratic motion, both datasets cover the the majority of the maze thoroughly.
Revised: While the Ant has more erratic motion, both datasets cover the the majority of the maze thoroughly.

Original: A comparison of the state coverage between Maze2D and AntMaze on all tasks is shown in the following Appendix section G.
Revised: A comparison of the state coverage between Maze2D and AntMaze on all tasks is shown in the following Appendix section G.

Original: G M AZE D OMAIN T RAJECTORIES
Revised: G M AZE D OMAIN T RAJECTORIES

Original: In this section, we visualized trajectories for the datasets in the Maze2D and AntMaze domains.
Revised: In this section, we visualized trajectories for the datasets in the Maze2D and AntMaze domains.

Original: Each image plots the states visited along each trajectory as a different colored line, overlaid on top of the maze.
Revised: Each image plots the states visited along each trajectory as a different colored line, overlaid on top of the maze.

Original: The goal state is marked as a white star.
Revised: The goal state is marked as a white star.