Original:  Offline Reinforcement Learningon Real Robot with Realistic Data Sources
Revised: Real World Offline Reinforcement Learning with Realistic Data Source

Original: Abstract
Revised: Abstract

Original: Offline reinforcement learning (ORL) holds great promise for robot learning due toits ability to learn from arbitrary pre-generated experience.
Revised: Offline reinforcement learning (ORL) holds great promise for robot learning due to its ability to learn from arbitrary pre-generated experience.

Original: However, there is a lackof comprehensive benchmarking of ORL algorithms, especially on physical robots.
Revised: However, current

Original: Current ORL benchmarks are almost entirely in simulation and utilize contriveddatasets like replay buffers of online RL agents or sub-optimal trajectories, andthus hold limited relevence for real-world robotics.
Revised: ORL benchmarks are almost entirely in simulation and utilize contrived datasets like replay buffers of online RL agents or sub-optimal trajectories, and thus hold limited relevance for real-world robotics.

Original: In this work, we posit thatnear-optimal and safe demonstrations collected from out-of-domain tasks are morepractical data-sources for real world robot learning.
Revised: In this work (Real-ORL), we posit that data collected from safe operations of closely related tasks are more practical data sources for real-world robot learning.

Original: We perform an extensive (800+hours data, 270+ human hours evaluations, multiple seeds) empirical study ofrepresentative ORL methods on four real world tabletop manipulation tasks.
Revised: Under these settings, we perform an extensive (6500+ trajectories collected over 800+ robot hours and 270+ human labor hour) empirical study evaluating generalization and transfer capabilities of representative ORL methods on four real-world tabletop manipulation tasks.

Original: Ourstudy finds that ORL and imitation learning prefer different action spaces, and that
Revised: Our study finds that ORL and imitation learning prefer different action spaces, and that

Original: ORL algorithms can generalize from leveraging offline heterogeneous data sourcesand outperform imitation learning. 
Revised: ORL algorithms can generalize from leveraging offline heterogeneous data sources and outperform imitation learning. We release our dataset and implementations at

Original: Project URL: https://sites.google.com/view/real-orl-anon
Revised: URL: https://sites.google.com/view/real-orl

Original:  Introduction
Revised:  Introduction

Original: Despite rapid advances the applicability of Deep Reinforcement Learning (DRL) algorithms [1–8] toreal-world robotics tasks is limited due to sample inefficiency and safety considerations.
Revised: Despite rapid advances, the applicability of Deep Reinforcement Learning (DRL) algorithms [1–8] to real-world robotics tasks is limited due to sample inefficiency and safety considerations.

Original: The emergingfield of offline reinforcement learning (ORL) [9, 10] has the potential to overcome these challenges,by learning only from logged or pre-generated offline datasets, thereby circumventing safety andexploration challenges.
Revised: The emerging field of offline reinforcement learning (ORL) [9, 10] has the potential to overcome these challenges, by learning only from logged or pre-generated offline datasets, thereby circumventing safety and exploration challenges.

Original: This makes ORL well suited for applications with large datasets (e g.recommendation systems) or those where online interactions are scarce and expensive (e g. robotics).
Revised: This makes ORL well suited for applications with large datasets (e g. recommendation systems) or those where online interactions are scarce and expensive (e g. robotics).

Original: However, comprehensive benchmarking and empirical evaluation of ORL algorithms is significantlylagging behind the burst of algorithmic progress [11–21].
Revised: However, comprehensive benchmarking and empirical evaluation of ORL algorithms is significantly lagging behind the burst of algorithmic progress [11–21].

Original: Widely used ORL benchmarks [22, 23] areentirely in simulation and use contrived data collection protocols that do not capture fundamentalconsiderations of physical robots.
Revised: Widely used ORL benchmarks [22, 23] are entirely in simulation and use contrived data collection protocols that do not capture fundamental considerations of physical robots.

Original: Our work aims to bridge this gap by outlining practical offlinedataset collection protocols that are representative of real-world robot settings.
Revised: Our work aims to bridge this gap by outlining practical offline dataset collection protocols that are representative of real-world robot settings.

Original: Our work alsoperforms a comprehensive empirical study spanning 6500+ robot trajectories and 270+ humanlabor hours , to benchmark and analyze three representative ORL algorithms thoroughly.
Revised: Our work also performs a comprehensive empirical study spanning 6500+ robot trajectories and 270+ human labor hours , to benchmark and analyze three representative ORL algorithms thoroughly.

Original: We willrelease all the datasets, code, and hardware hooks from this paper.
Revised: We will release all the datasets, code, and hardware hooks from this paper.

Original: In principle, ORL can consume and train policies from arbitrary datasets.
Revised: In principle, ORL can consume and train policies from arbitrary datasets.

Original: This has prompted thedevelopment of simulated ORL benchmarks [22–24] that utilize data sources like expert policiestrained with online RL, exploratory policies, or even replay buffers of online RL agents.
Revised: This has prompted the development of simulated ORL benchmarks [22–24] that utilize data sources like expert policies trained with online RL, exploratory policies, or even replay buffers of online RL agents.

Original: However,simulated dataset may fail to capture the challenges in real world: hardware noises coupled withvarying reset conditions lead to covariate shift and violate the i.i.d.
Revised: However, simulated dataset may fail to capture the challenges in real world: hardware noises coupled with varying reset conditions lead to covariate shift and violate the i.i.d.

Original: assumption about state distributionsbetween train and test time.
Revised: assumption about state distributions between train and test time.

Original: Further, such datasets are not feasible on physical robots and defeat thecore motivation of ORL in robotics – to avoid the use of online RL due to poor sample efficiency andsafety!
Revised: Further, such datasets are not feasible on physical robots and defeat the core motivation of ORL in robotics – to avoid the use of online RL due to poor sample efficiency and safety!

Original: Recent works [24, 25] suggest that dataset composition and distribution dramatically affect the relative performance of algorithms.
Revised: Recent works [24, 25] suggest that dataset composition and distribution dramatically affect the relative performance of algorithms.

Original: In this backdrop, we consider the pertinent question: What isa practical instantiation of the ORL setting for physical robots, and can existing ORL algorithmslearn successful policies in such a setting?
Revised: In this backdrop, we consider the pertinent question: What is a practical instantiation of the ORL setting for physical robots, and can existing ORL algorithms learn successful policies in such a setting?

Original: In this work, we envision practical scenarios to apply ORL for real-world robotics.
Revised: In this work, we envision practical scenarios to apply ORL for real-world robotics.

Original: Towardsthis end, our first insight is that real-world offline datasets are likely to come from well-behavedpolicies that abide by safety and monetary constraints, in sharp contrast to simulator data collectedfrom exploratory or partially trained policies, as used in simulated benchmarks [22–24].
Revised: Towards this end, our first insight is that real-world offline datasets are likely to come from well-behaved policies that abide by safety and monetary constraints, in sharp contrast to simulator data collected from exploratory or partially trained policies, as used in simulated benchmarks [22–24].

Original: Suchtrajectories can be collected by user demonstrations or through hand-scripted policies that arepartially successful but safe.
Revised: Such trajectories can be collected by user demonstrations or through hand-scripted policies that are partially successful but safe.

Original: It is more realistic to collect large volumes of data for real robots usingmultiple successful policies designed under expert supervision for specific tasks than using policiesthat are unsuccessful or without safety guarantees.
Revised: It is more realistic to collect large volumes of data for real robots using multiple successful policies designed under expert supervision for specific tasks than using policies that are unsuccessful or without safety guarantees.

Original: Secondly, the goal of any learning (including
Revised: Secondly, the goal of any learning (including

Original: ORL) is broad generalization and transfer.
Revised: ORL) is broad generalization and transfer.

Original: It is therefore critical to study whether a learning algorithmcan leverage task-agnostic datasets, or datasets intended for a source task, to make progress on a newtarget task.
Revised: It is therefore critical to study whether a learning algorithm can leverage task-agnostic datasets, or datasets intended for a source task, to make progress on a new target task.

Original: In this work, we collect offline datasets consistent with these principles and evaluaterepresentative ORL algorithms on a set of canonical table-top tasks as illustrated in Figure 1.
Revised: In this work, we collect offline datasets consistent with these principles and evaluate representative ORL algorithms on a set of canonical table-top tasks as illustrated in Figure 1.

Original: Evaluation studies on physical robots are sparse in the field due to time and resource constraints,but they are vital to furturing our understanding.
Revised: Evaluation studies on physical robots are sparse in the field due to time and resource constraints, but they are vital to furturing our understanding.

Original: Our real robot results corroborate and validateintuitions from simulated benchmarks [26] but also enable novel discoveries.
Revised: Our real robot results corroborate and validate intuitions from simulated benchmarks [26] but also enable novel discoveries.

Original: We find that (1) evenfor scenarios with sufficiently high-quality data, some ORL algorithms could outperform behaviorcloning (BC) [27] on specific tasks, (2) for scenarios that require generalization or transfer to newtasks with low data support, ORL agents generally outperform BC.
Revised: We find that (1) even for scenarios with sufficiently high-quality data, some ORL algorithms could outperform behavior cloning (BC) [27] on specific tasks, (2) for scenarios that require generalization or transfer to new tasks with low data support, ORL agents generally outperform BC.

Original: (3) in cases with overlappingdata support, ORL algorithms can leverage additional heterogeneous task-agnostic data to improvetheir own performance, and in some cases even surpass the best in-domain agent.
Revised: (3) in cases with overlapping data support, ORL algorithms can leverage additional heterogeneous task-agnostic data to improve their own performance, and in some cases even surpass the best in-domain agent.

Original: Our empirical evaluation is unique as it focuses on ORL algorithms ability to leverage more realistic,multi-task data sources, spans over several tasks that are algorithm-agnostic, trains various ORLalgorithms on the same settings and evaluates them directly in the real world.
Revised: Our empirical evaluation is unique as it focuses on ORL algorithms ability to leverage more realistic, multi-task data sources, spans over several tasks that are algorithm-agnostic, trains various ORL algorithms on the same settings and evaluates them directly in the real world.

Original: In summary, we believeour work establishes the effectiveness of offline RL algorithms in leveraging out of domain highquality heterogeneous data for generalization and transfer in robot-learning, which is representativeof real world applications.
Revised: In summary, we believe our work establishes the effectiveness of offline RL algorithms in leveraging out of domain highquality heterogeneous data for generalization and transfer in robot-learning, which is representative of real world applications.

Original:  Preliminaries and Related Work
Revised:  Preliminaries and Related Work

Original: Offline RL.
Revised: Offline RL.

Original: We consider the ORL framework, which models the environment as a Markov Decision
Revised: We consider the ORL framework, which models the environment as a Markov Decision

Original: Process (MDP): M = ⟨ S, A, R, T, ρ 0 , H ⟩ where S ⊆ R n is the state space, A ⊆ R m is the actionspace, R : S ×
Revised: Process (MDP): M = ⟨ S, A, R, T, ρ 0 , H ⟩ where S ⊆ R n is the state space, A ⊆ R m is the action space, R : S ×

Original: A → R is the reward function, T : S ×
Revised: A → R is the reward function, T : S ×

Original: A × S → R + is the (stochastic) transitiondynamics, ρ 0 : S 
Revised: A × S → R + is the (stochastic) transition dynamics, ρ 0 : S → R

Original: → R + is the initial state distribution, and H is the maximum trajectory horizon.
Revised: + is the initial state distribution, and H is the maximum trajectory horizon.

Original: In the ORL setting, we assume access to the reward function R and a pre-generated dataset of theform: D = { τ 1 , τ 2 , . . .
Revised: In the ORL setting, we assume access to the reward function R and a pre-generated dataset of the form: D = { τ 1 , τ 2 , . . .

Original: τ N } , where each τ i =
Revised: τ N } , where each τ i =

Original: ( s 0 , a 0 , s 1 , a 1 , . . .
Revised: ( s 0 , a 0 , s 1 , a 1 , . . .

Original: s H ) is a trajectory collected using abehavioral policy or a mix of policies π b : S ×
Revised: s H ) is a trajectory collected using a behavioral policy or a mix of policies π b : S ×

Original: A → R + .
Revised: A → R + .

Original: The goal in ORL is to use the offline dataset D to learn a near-optimal policy,
Revised: The goal in ORL is to use the offline dataset D to learn a near-optimal policy,

Original: [Equation]
Revised: [Equation]

Original: In the general case, the optimal policy π ∗ may not be learnable using D due to a lack of sufficientexploration in the dataset.
Revised: In the general case, the optimal policy π ∗ may not be learnable using D due to a lack of sufficient exploration in the dataset.

Original: In this case, we would seek the best policy learnable from the dataset, or,at the very least, a policy improves upon behavioral policy.
Revised: In this case, we would seek the best policy learnable from the dataset, or, at the very least, a policy improves upon behavioral policy.

Original: Offline RL Algorithms.
Revised: Offline RL Algorithms.

Original: Recent years have seen tremendous interests in offline RL and the development of new ORL algorithms.
Revised: Recent years have seen tremendous interests in offline RL and the development of new ORL algorithms.

Original: Most of these algorithms incorporate some form of regularizationor conservatism.
Revised: Most of these algorithms incorporate some form of regularization or conservatism.

Original: This can take many forms, such as regularized policy gradients or actor criticalgorithms [14, 15, 19, 28–30], approximate dynamic programming [11, 13, 17, 18], and modelbased RL [12, 31–33].
Revised: This can take many forms, such as regularized policy gradients or actor critic algorithms [14, 15, 19, 28–30], approximate dynamic programming [11, 13, 17, 18], and modelbased RL [12, 31–33].

Original: We select a representative ORL algorithms from each category: AWAC [19],
Revised: We select a representative ORL algorithms from each category: AWAC [19],

Original: IQL [18] and MOR E L [12].
Revised: IQL [18] and MOR E L [12].

Original: In this work, we do not propose new algorithms for offline RL; rather westudy a spectrum of representative ORL algorithms and evaluate their assumptions and effectivenesson a physical robot under realistic usage scenarios.
Revised: In this work, we do not propose new algorithms for offline RL; rather we study a spectrum of representative ORL algorithms and evaluate their assumptions and effectiveness on a physical robot under realistic usage scenarios.

Original: Offline RL Benchmarks and Evaluation.
Revised: Offline RL Benchmarks and Evaluation.

Original: In conjunction with algorithmic advances, offline RLbenchmarks have also been proposed.
Revised: In conjunction with algorithmic advances, offline RL benchmarks have also been proposed.

Original: However, they are predominantly captured with simulation [22,23, 34] using datasets with idealistic coverage, i.i.d.
Revised: However, they are predominantly captured with simulation [22, 23, 34] using datasets with idealistic coverage, i.i.d.

Original: samples, and synchronous execution.
Revised: samples, and synchronous execution.

Original: Most ofthese assumptions are invalid in real world which is stochastic and has operational delays.
Revised: Most of these assumptions are invalid in real world which is stochastic and has operational delays.

Original: Prior worksinvestigating offline RL for these settings on physical robots are limited.
Revised: Prior works investigating offline RL for these settings on physical robots are limited.

Original: For instance, Kostrikovet al.
Revised: For instance, Kostrikov et al.

Original: [18] did not provide real robot evaluation for IQL, which we conduct in this work; Chebotaret al.
Revised: [18] did not provide real robot evaluation for IQL, which we conduct in this work; Chebotar et al.

Original: [35], Kalashnikov et al.
Revised: [35], Kalashnikov et al.

Original: [36] evaluate performance on a specialized Arm-Farm; Rafailov et al.[37] evaluate on a single drawer closing task; Singh et al.
Revised: [36] evaluate performance on a specialized Arm-Farm; Rafailov et al. [37] evaluate on a single drawer closing task; Singh et al.

Original: [17], Kumar et al.
Revised: [17], Kumar et al.

Original: [38] evaluate only onealgorithm (COG, CQL, respectively).
Revised: [38] evaluate only one algorithm (COG, CQL, respectively).

Original: Mandlekar et al.
Revised: Mandlekar et al.

Original: [39] evaluate BCQ and CQL alongside BC onthree real robotics tasks, but their evaluations consider only in-domain setting: that the agents weretrained only on the specific task data, without giving them access to a pre-generated, offline dataset.
Revised: [39] evaluate BCQ and CQL alongside BC on three real robotics tasks, but their evaluations consider only in-domain setting: that the agents were trained only on the specific task data, without giving them access to a pre-generated, offline dataset.

Original: Thus, it is unclear whether insights from simulated benchmarks or limited hardware evaluation cangeneralize broadly.
Revised: Thus, it is unclear whether insights from simulated benchmarks or limited hardware evaluation can generalize broadly.

Original: Our work aims to bridge this gap by empirically studying representative offline
Revised: Our work aims to bridge this gap by empirically studying representative offline

Original: RL algorithms on a suite of real-world robot learning tasks with an emphasize on transfer learningand out-domain generalization.
Revised: RL algorithms on a suite of real-world robot learning tasks with an emphasize on transfer learning and out-domain generalization.

Original: See Section 3 for detailed discussion.
Revised: See Section 3 for detailed discussion.

Original: Imitation Learning (IL).
Revised: Imitation Learning (IL).

Original: IL [40] is an alternate approach to training control policies for robotics.
Revised: IL [40] is an alternate approach to training control policies for robotics.

Original: Unlike RL, which learns policies by optimizing rewards (or costs), IL (and inverse RL [41–43]) learnsby mimicking expert demonstrations and typically requires no reward function.
Revised: Unlike RL, which learns policies by optimizing rewards (or costs), IL (and inverse RL [41–43]) learns by mimicking expert demonstrations and typically requires no reward function.

Original: IL has been studiedin both the offline setting [44, 45], where the agent learns from a fixed set of expert demonstrations,and the online setting [46, 47], where the agent can perform additional environment interactions.
Revised: IL has been studied in both the offline setting [44, 45], where the agent learns from a fixed set of expert demonstrations, and the online setting [46, 47], where the agent can perform additional environment interactions.

Original: Acombination of RL and IL has also been explored in prior work [48, 49].
Revised: A combination of RL and IL has also been explored in prior work [48, 49].

Original: Our offline dataset consists oftrajectories from a heuristic hand-scripted policy collected under expert supervision, which representsa dataset of reasonably high quality .
Revised: Our offline dataset consists of trajectories from a heuristic hand-scripted policy collected under expert supervision, which represents a dataset of reasonably high quality .

Original: As a result, we consider offline IL and, behavior cloning inparticular, as a baseline algorithm in our empirical evaluation.
Revised: As a result, we consider offline IL and, behavior cloning in particular, as a baseline algorithm in our empirical evaluation.

Original:  Experiment Scope and Setup
Revised:  Experiment Scope and Setup

Original: To investigate the effectiveness of ORL algorithms on real-world robot learning tasks, we adhereto a few guiding principles: (1) we make design choices representing the wider community to theextent possible, (2) we strive to be fair to all baselines by providing them their best chance and workin consultation with their authors; and (3) we prioritize reproducibility and data sharing.
Revised: To investigate the effectiveness of ORL algorithms on real-world robot learning tasks, we adhere to a few guiding principles: (1) we make design choices representing the wider community to the extent possible, (2) we strive to be fair to all baselines by providing them their best chance and work in consultation with their authors; and (3) we prioritize reproducibility and data sharing.

Original: We willopen-source our data, camera images along with our training and evaluation codebase.
Revised: We will open-source our data, camera images along with our training and evaluation codebase.

Original: Hardware Setup.
Revised: Hardware Setup.

Original: Hardware plays a seminal role in robotic capability.
Revised: Hardware plays a seminal role in robotic capability.

Original: For reproducibility andextensibility, we selected a hardware platform that is well-established, non-custom, and commonlyused in the field.
Revised: For reproducibility and extensibility, we selected a hardware platform that is well-established, non-custom, and commonly used in the field.

Original: After an exhaustive literature survey [50–55], we converged on a table-topmanipulation setup, shown in Figure 2.
Revised: After an exhaustive literature survey [50–55], we converged on a table-top manipulation setup, shown in Figure 2.

Original: It consists of a table-mounted Franka panda arm that uses a
Revised: It consists of a table-mounted Franka panda arm that uses a

Original: RobotiQ parallel gripper as its end effector, which is accompanied by two Intel 435 RGBD cameras.
Revised: RobotiQ parallel gripper as its end effector, which is accompanied by two Intel 435 RGBD cameras.

Original: Our robot has 8 DOF, uses factory-supplied default controller gains, accepts position commands at[-
-]
Revised: Our robot has 8 DOF, uses factory-supplied default controller gains, accepts position commands at[-
-]

Original: Hz, and runs a low-level joint position controller at 1000 Hz.
Revised: Hz, and runs a low-level joint position controller at 1000 Hz.

Original: To perceive the object to interact with,we exact the position of the AprilTags attached to the object from RGB images.
Revised: To perceive the object to interact with, we exact the position of the AprilTags attached to the object from RGB images.

Original: Our robot statesconsist of joint positions, joint velocities, and positions of the object to interact with (if applicable).
Revised: Our robot states consist of joint positions, joint velocities, and positions of the object to interact with (if applicable).

Original: Our policies compute actions (desired joint pose) using robot proprioception, tracked object locations,and desired goal location.Realsense
Revised: Our policies compute actions (desired joint pose) using robot proprioception, tracked object locations, and desired goal location.

Original: Canonical Tasks We consider four classic manipulation tasks common in literature: reach , slide ,lift , and pick-n-place (PnP) (see Figure 1).reach requires the robot to move from a randomlysampled configuration in the workspace to anotherconfiguration.
Revised: Canonical Tasks We consider four classic manipulation tasks common in literature: reach , slide , lift , and pick-n-place (PnP) (see Figure 1). reach requires the robot to move from a randomly sampled configuration in the workspace to another configuration.

Original: The other three tasks involve a heavyglass lid with a handle, which is initialized randomlyon
Revised: The other three tasks involve a heavy glass lid with a handle, which is initialized randomly on the table.

Original:  the 
Revised: slide requires the robot to hold and move the lid along the table to a specified goal location. lift requires the robot to grasp and lift the lid 10 cm off the table. PnP requires the robot to grasp, lift, move and place the lid at a designated goal position i.e. the chopping board.

Original:  manipulation challenges: reach focuses on free movements while the other three tasks involveintermittent interaction dynamics between the table, lid, and the parallel grippers.
Revised: The four tasks constitute a representative range of common tabletop manipulation challenges: reach focuses on free movements while the other three tasks involve intermittent interaction dynamics between the table, lid, and the parallel grippers.

Original: We model eachcanonical task as a MDP with an unique reward function.
Revised: We model each canonical task as a MDP with an unique reward function.

Original: Details on our tasks are in Appendix. ?? .
Revised: Details on our tasks are in Appendix. 7.1.

Original: Data Collection.
Revised: Data Collection.

Original: We use a hand-designed, scripted policy developed under expert supervision tocollect (dominantly) successful trajectories for all our canonical tasks.
Revised: We use a hand-designed, scripted policy developed under expert supervision to collect (dominantly) successful trajectories for all our canonical tasks.

Original: To highlight ORL algorithmsability to overcome suboptimal dataset, previous works [22, 34, 39] have crippled expert policieswith noise, use half-trained RL policies or collect human demonstrations with varying qualities tohighlight the performance gain over compromised datasets.
Revised: To highlight ORL algorithms ability to overcome suboptimal dataset, previous works [22, 34, 39] have crippled expert policies with noise, use half-trained RL policies or collect human demonstrations with varying qualities to highlight the performance gain over compromised datasets.

Original: We posit that such data sources are notrepresentative of robotics domains, where noisy or random behaviors are unsafe and detrimental tohardware’s stability.
Revised: We posit that such data sources are not representative of robotics domains, where noisy or random behaviors are unsafe and detrimental to hardware’s stability.

Original: Instead of infusing noise or failure data points to serve as negative examples,we believe that mixing data collected from various tasks offers a more realistic setting in which toapply ORL on real robots for three reasons: (1) collecting such “random/roaming/explorative” dataon a real robot autonomously would require comprehensive safety constraints, expert supervisionand oversight, (2) engaging experts to record such random data in large quantities makes less sensethan utilizing them to collecting meaningful trajectories on a real task, and (3) designing task-specificstrategies and stress testingORL’s ability against such a strong dataset is more viable than using acompromised dataset.
Revised: Instead of infusing noise or failure data points to serve as negative examples, we believe that mixing data collected from various tasks offers a more realistic setting in which to apply ORL on real robots for three reasons: (1) collecting such “random/roaming/explorative” data on a real robot autonomously would require comprehensive safety constraints, expert supervision and oversight, (2) engaging experts to record such random data in large quantities makes less sense than utilizing them to collecting meaningful trajectories on a real task, and (3) designing task-specific strategies and stress testing

Original: We collected offline dataset  usingheuristic strategies designed with reasonableefforts and, to avoid biases favoring task/algorithm, frozen the dataset ahead of time. To create scripted policies for all tasks, we first separate each task into stages with a goal location. We then compute a target robot state in joint angles, where the goal position is reached by the grippervia Mujoco’s IK solver. The scripted policy takes tiny steps toward this goal location until sometask-specific criteria are reached.
Revised: ORL’s ability against such a strong dataset is more viable than using a compromised dataset.

Original: Our heuristic policies didn’t reach the theoretical maximum possiblescores due to controller noises and tracking noises. However, they complete the task at a high successrate and have comparable performance to human demonstration.
Revised: We collected offline dataset using heuristic strategies designed with reasonable efforts and, to avoid biases favoring task/algorithm, frozen the dataset ahead of time.

Original: Dataset.
Revised: Dataset.

Original: In total, we collected a dataset of around 3000 trajectories and the characteristics ofour dataset is shown in Table.
Revised: In total, we collected a dataset of around 3000 trajectories and the characteristics of our dataset is shown in Table.

Original: Our offline dataset is represented as a series of transition tuples{ ( s, a, s ′ ) task }.
Revised: Our offline dataset is represented as a series of transition tuples { ( s, a, s ′ ) task }.

Original: States consist of joint positions, joint velocities, and positions of the object to interactwith (if applicable).
Revised: States consist of joint positions, joint velocities, and positions of the object to interact with (if applicable).

Original: Actions contain target joint positions.
Revised: Actions contain target joint positions.

Original: To perceive the object to interact with,we obtain the position of tracked AprilTags attached to the object from the RGB images of the twocameras.
Revised: To perceive the object to interact with, we obtain the position of tracked AprilTags attached to the object from the RGB images of the two cameras.

Original: More details are available in Appendix ?? .
Revised: More details are available in Appendix 7.2. [Table] reach [Table]

Revised: slide

Original: Experiment Design
Revised: [Table] lift [Table] [Table]

Revised: PnP

Revised: [Table]

Revised: [Table]

Original:  Our experiments aim to answer the following questions.
Revised: [Table] [Table]  Experiment Design Our experiments aim to answer the following questions.

Original: (1) Are ORL algorithms sensitive to, orshow a preference for, any specific state and action space parameterization?
Revised: (1) Are ORL algorithms sensitive to, or show a preference for, any specific state and action space parameterization?

Original: (2) How do they performagainst the standard methods for in-domain tasks?
Revised: (2) How do they perform against the standard methods for in-domain tasks?

Original: (3) How do common methods perform in out-ofdomain tasks requiring (a) generalization, and (b) re-targeting?
Revised: (3) How do common methods perform in out-ofdomain tasks requiring (a) generalization, and (b) re-targeting?

Original: To ensure fair evaluation, we nowoutline our choice of candidate algorithms and performance metrics.
Revised: To ensure fair evaluation, we now outline our choice of candidate algorithms and performance metrics.

Original: Algorithms For all evaluations, we compare four algorithms: Behavior Cloning (BC) [27],
Revised: Algorithms For all evaluations, we compare four algorithms: Behavior Cloning (BC) [27],

Original: Model-based Offline REinforcement Learning (MOR E L) [12] [Table] (AWAC) [19] and Implicit Q-Learning (IQL)[18].
Revised: Model-based Offline REinforcement Learning (MOR E L) [12], Advantage-Weighted Actor Critic (AWAC) [19] and Implicit Q-Learning (IQL)

Original: BC is a model-free IL algorithm that remainsa strong baseline for real robot experiments due to its simplicity and practicality.
Revised: BC is a model-free IL algorithm that remains a strong baseline for real robot experiments due to its simplicity and practicality.

Original: AWAC and
Revised: AWAC and

Original: IQL both train an off-policy value function and then derive a policy to maximize the expected reward.
Revised: IQL both train an off-policy value function and then derive a policy to maximize the expected reward.

Original: AWAC uses KL divergence minimization to constrain the resulting policy to be close to the given policy distribution.
Revised: AWAC uses KL divergence minimization to constrain the resulting policy to be close to the given policy distribution.

Original: In contrast, IQL leverages expectile regression to avoid querying the value functionfor any out-of-distribution query.
Revised: In contrast, IQL leverages expectile regression to avoid querying the value function for any out-of-distribution query.

Original: MOR E L is distinct since it is a model-based approach: it recoversa dynamics model from offline data that allows it directly apply policy gradient RL algorithms.
Revised: MOR E L is distinct since it is a model-based approach: it recovers a dynamics model from offline data that allows it directly apply policy gradient RL algorithms.

Original: Weuse implementations of BC and MOR E L from the MOR E L author implementation.
Revised: We use implementations of BC and MOR E L from the MOR E L author implementation.

Original: For the later,we add a weighted behavior cloning loss to its policy training step to serve as a regularizer, inspiredby [30].
Revised: For the later, we add a weighted behavior cloning loss to its policy training step to serve as a regularizer, inspired by [30].

Original: We use AWAC and IQL implemented in the open sourced d3rlpy library [56].
Revised: We use AWAC and IQL implemented in the open sourced d3rlpy library [56].

Original: Training.
Revised: Training.

Original: Since neural network agents are empirically sensitive to parameters and seeds, we (1) usedthe same fixed random seed ( 123 ) for all our experiments with additional seed sweeping to strengthenthe reproducibility of our results and (2) conducted equal amount of efforts for hyperparameter tuningefforts for all algorithms.
Revised: Since neural network agents are empirically sensitive to parameters and seeds, we (1) used the same fixed random seed ( 123 ) for all our experiments with additional seed sweeping to strengthen the reproducibility of our results and (2) conducted equal amount of efforts for hyperparameter tuning efforts for all algorithms.

Original: Unlike traditional supervised learning, we cannot simply select the agentswith the best validation loss for tuning the hyperparameters, because we cannot know the performanceof an agent unless testing it on a real robot [39].
Revised: Unlike traditional supervised learning, we cannot simply select the agents with the best validation loss for tuning the hyperparameters, because we cannot know the performance of an agent unless testing it on a real robot [39].

Original: We thus keep our tuning simple and fair: startingwith the default parameters and training 5 agents in 3 rounds, trying to make the agent converge.
Revised: We thus keep our tuning simple and fair: starting with the default parameters and training 5 agents in 3 rounds, trying to make the agent converge.

Original: We observe that certain agents cannot converge after exhausting the allocated trials and report theseresults with a (*) marker, signaling the challenge in tuning parameters for such algorithms.
Revised: We observe that certain agents cannot converge after exhausting the allocated trials and report these results with a (*) marker, signaling the challenge in tuning parameters for such algorithms.

Original: Moredetails are available in Appendix.?? .
Revised: More details are available in Appendix.

Original: Evaluation.
Revised: Evaluation.

Original: Real robot evaluations can have high variance due to reset conditions and hardwarenoise.
Revised: Real robot evaluations can have high variance due to reset conditions and hardware noise.

Original: For each agent, we collect 12 trajectories and report their mean and standard deviation ofscores.
Revised: For each agent, we collect 12 trajectories and report their mean and standard deviation of scores.

Original: To confirm the reproducibility of our results and robustness to seed sweeping, for agents thatcontributed to our conclusions (usually the best and the second-best agents) we report performanceswept over three consecutive random seeds ( 122 , 124 in addition to the fixed seed of 123 ) in
Revised: To confirm the reproducibility of our results and robustness to seed sweeping, for agents that contributed to our conclusions (usually the best and the second-best agents) we report performance swept over three consecutive random seeds ( 122 , 124 in addition to the fixed seed of 123 ) in

Original: Appendix ?? .
Revised: Appendix 7.6.

Original: To verify the statistical significance of our results when comparing performancebetween agents, we report the p-value of paired difference tests in Appendix ?? .
Revised: To verify the statistical significance of our results when comparing performance between agents, we report the p-value of paired difference tests in Appendix 7.7.

Original: A. In-domain Ablations.
Revised: A. In-domain Ablations.

Original: We note the distinction between “in-domain” training and “out-domain”training, where the former leverages only data that were collected for the test task and the later allowincorporating heterogeneous data from different tasks.
Revised: We note the distinction between “in-domain” training and “out-domain” training, where the former leverages only data that were collected for the test task and the later allow incorporating heterogeneous data from different tasks.

Original: We first train all agents using in-domain data(i.e. , we train a slide agent by feeding only slide data) to test ORL algorithms’ sensitivity tovarying data representation and inspect: (1) whether it is worth including velocity information inthe state space ( Vel versus NoVel ); for simulator experiments, it is almost always a gain to includevelocity, but velocity sensors on real robots are notoriously noisy; (2) whether to use the policy outputjoint position ( Abs ) vs the change in joint position ( Delta ) as action.
Revised: We first train all agents using in-domain data (i.e., we train a slide agent by feeding only slide data) to test ORL algorithms’ sensitivity to varying data representation and inspect: (1) whether it is worth including velocity information in the state space ( Vel versus NoVel ); for simulator experiments, it is almost always a gain to include velocity, but velocity sensors on real robots are notoriously noisy; (2) whether to use the policy output joint position ( Abs ) vs the change in joint position ( Delta ) as action.

Original: Most BC literature uses theformer, whereas RL prefers the latter.
Revised: Most BC literature uses the former, whereas RL prefers the latter.

Original: 1 We use the outcome of the ablations and the best-performingsetting for each algorithm to study generalization and transfer in the following three scenarios:
Revised: We use the outcome of the ablations and the best-performing setting for each algorithm to study generalization and transfer in the following three scenarios:

Original: B. Generalization: Lacking data support.
Revised: B. Generalization: Lacking data support

Original: The data collection may not cover the task spaceuniformly.
Revised: The data collection may not cover the task space uniformly. For

Original: For example, imagine that a robot trained towipe clean a table but now cleans a biggertable.
Revised: example, imagine that a robot trained to

Revised: wipe clean

Revised: a

Revised: table but now cleans a bigger

Revised: table

Revised: Empirically

Original: Empirically, a policy trained with behavior cloning would have trouble predicting actions forstates when there is less data support.
Revised: a policy trained with behavior cloning would have trouble predicting actions for states

Revised: when

Revised: there

Original:  Can ORL algorithms, by learning a value function or model,generalize to a task space that lacks data support?
Revised: is less data support. Can ORL algorithms, by learning a value function or model,

Original:  To this end, we create a new dataset from our slidetask by dividing the task space to three regions: left, center, right.
Revised: [Table] generalize to a task space that lacks data support? To this end, we create a new dataset from our slide task by dividing the task space to three regions: left, center, right.

Original: We remove any trajectory wherethe object was initially placed in the center region from the collected dataset.
Revised: We remove any trajectory where the object was initially placed in the center region from the collected dataset.

Original: We train all agents andgather evaluation trajectories asking them to slide an object initially placed in the left, center andright regions.
Revised: We train all agents and gather evaluation trajectories asking them to slide an object initially placed in the left, center and right regions.

Original: C. Generalization: Re-targeting data for dynamic tasks.
Revised: C. Generalization: Re-targeting data for dynamic tasks.

Original: For the slide task, our collecteddemonstration has static goal positions.
Revised: For the slide task, our collected demonstration has static goal positions.

Original: We test agents trained using such static data in a dynamicsetting by updating the goal at a fixed frequency, and asking the agents to grasp and slide the lidfollowing some predetermined curves.
Revised: We test agents trained using such static data in a dynamic setting by updating the goal at a fixed frequency, and asking the agents to grasp and slide the lid following some predetermined curves.

Original: We collected the ideal trajectories via human demonstration.
Revised: We collected the ideal trajectories via human demonstration.

Original: This task can be viewed as a simplified version of daily tasks, including drawing, wiping, and cleaning,which require possibly repeated actions and a much longer horizon than usual IL and ORL tasks.
Revised: This task can be viewed as a simplified version of daily tasks, including drawing, wiping, and cleaning, which require possibly repeated actions and a much longer horizon than usual IL and ORL tasks.

Original: We select a variety of trajectories: circle, square, and the numbers 3, 5, 6, 8, which have differentcombinations of smooth curves and corners.
Revised: We select a variety of trajectories: circle, square, and the numbers 3, 5, 6, 8, which have different combinations of smooth curves and corners.

Original: D. Transfer: Reusing data from different tasks.
Revised: D. Transfer: Reusing data from different tasks.

Original: We investigate whether we can reuse hetergenuous data collected from previous tasks to train a policy for a new task.
Revised: We investigate whether we can reuse hetergenuous data collected from previous tasks to train a policy for a new task.

Original: For example, wouldcombining data from two canonical tasks (e.g., slide + lift ) helps the agent perform better oneither of these tasks?
Revised: For example, would combining data from two canonical tasks (e.g., slide + lift ) helps the agent perform better on either of these tasks?

Original: When aggregating data collected for multiple tasks, ORL algorithms can usethe reward function for the test task to relabel the offline dataset.
Revised: When aggregating data collected for multiple tasks, ORL algorithms can use the reward function for the test task to relabel the offline dataset.

Original: Evaluating ORL algorithms onsuch out-domain, transfer-learning settings is practical and relevant: instead of collecting randomexplorative data which demands careful setup of safety constraints on a real robot, we want to leverageoffline datasets collected from different tasks to improve ORL performance.
Revised: Evaluating ORL algorithms on such out-domain, transfer-learning settings is practical and relevant: instead of collecting random explorative data which demands careful setup of safety constraints on a real robot, we want to leverage offline datasets collected from different tasks to improve ORL performance.

Original: We train our algorithmwith different combinations of canonical task demonstrations (“train-data”) and evaluate each agenton each individual task. 
Revised: We train our algorithm with different combinations of canonical task demonstrations (“train-data”) and evaluate each agent on each individual task. [Table]

Original:  Results and Discussion 
Revised:  Results and Discussion 5.1 In-domain Tasks

Original: 5.1 In-domain Tasks

Original: Which agent performs best for in-domain tasks?
Revised: Which agent performs best for in-domain tasks?

Original: Table 2 summarizes agents’ performancefor in-domain tasks, providing empirical insights into agents’ sensitivity to varying representations.
Revised: Table 2 summarizes agents’ performance for in-domain tasks, providing empirical insights into agents’ sensitivity to varying representations.

Original: Interestingly, two of the ORL agent, IQL and AWAC achieved higher mean-scores than BC trainedwith abundant, in-domain real robot demonstrations on 2 out of 4 tasks.
Revised: Interestingly, two of the ORL agent, IQL and AWAC achieved higher mean-scores than BC trained with abundant, in-domain real robot demonstrations on 2 out of 4 tasks.

Original: On the simplest task reach ,the best version of all agents reached comparable performance.
Revised: On the simplest task reach , the best version of all agents reached comparable performance.

Original: On the hardest task PnP , BC outperformed ORL agents.
Revised: On the hardest task PnP , BC outperformed ORL agents.

Original: We would recommend considering IQL as a baseline even for imitation learning,when it is applicable to write a continuous reward function (an additional assumption compared toclassic imitation learning).
Revised: We would recommend considering IQL as a baseline even for imitation learning, when it is applicable to write a continuous reward function (an additional assumption compared to classic imitation learning).

Original: Sensitivity to representation Empirically, BC demonstrated robustness to different state and actionspaces, whereas ORL agents had varied performance.
Revised: Sensitivity to representation Empirically, BC demonstrated robustness to different state and action spaces, whereas ORL agents had varied performance.

Original: In 3 of 4 tasks considered, BCperformed thebest when using absolution joint position as the action space and including velocity in the state space( AbsVel ).
Revised: In 3 of 4 tasks considered, BCperformed the best when using absolution joint position as the action space and including velocity in the state space ( AbsVel ).

Original: On all tasks, ORL agents performed better using delta action space ( Delta ) rather thanjoint position.
Revised: On all tasks, ORL agents performed better using delta action space ( Delta ) rather than joint position.

Original: Intuitively, using the delta action space would be equivalent to restricting the policyto move in a unit ball centered around the current state.
Revised: Intuitively, using the delta action space would be equivalent to restricting the policy to move in a unit ball centered around the current state.

Original: Such constraints could benefit RL policiesthat need exploration and sampling in action space more than it helped BC, which simply learns themapping from states to actions.
Revised: Such constraints could benefit RL policies that need exploration and sampling in action space more than it helped BC, which simply learns the mapping from states to actions.

Original: Additionally, we observed that our best agents all included velocity intheir state space, though they were not guaranteed to have monotonic performance gain with velocityadded.
Revised: Additionally, we observed that our best agents all included velocity in their state space, though they were not guaranteed to have monotonic performance gain with velocity added.

Original: 5.2 Generalization and Transfer
Revised: 5.2 Generalization and Transfer

Original: Generalization to regions that lack data support.
Revised: Generalization to regions that lack data support.

Original: Table 3 trains agent using an carved-out datasetand compares the agents’ performance on regions with more data support versus the region with [Table]
Revised: Table 3 trains agent using an carved-out dataset and compares the agents’ performance on regions with more data support versus the region with less data support ( Center , highlighted in Grey).

Original: [Table] [Table] region.
Revised: For comparison, we also train all agents using full dataset and evaluate them on the Center region.

Original: On  region with abundant data support  we note that BC and IQL performed well and AWAC/MOR E L got lower scores aligning with our previous observation that BC/IQL performed better on in-domain tasks.
Revised: On the region with abundant data support ( Left and Right ), we note that BC and IQL performed well and AWAC/MOR E L got lower scores, aligning with our previous observation that BC/IQL performed better on in-domain tasks.

Original: Inspecting  performance  less , however, we discovered that 
Revised: Inspecting the performance on regions that have less data support ( Center ), however, we discovered that (1)

Original:  MOR  L  of their initial they evaluated on carved-out  performed  worse  them  BC  performed out.
Revised: AWAC and MOR E L could match BC’s performance on low-support region regardless of their initial disadvantage (they had poorer performance on the region with more support) and (2) ORL agents trained with carved-out dataset and evaluated on carved-out region performed no worse than them trained with full dataset, in contrast to BC agent, which performed worse after carving-out.

Original: Generalization to dynamic tasks.
Revised: Generalization to dynamic tasks.

Original: Table 4 lists the ideal curves and the curves traced by each model.
Revised: Table 4 lists the ideal curves and the curves traced by each model.

Original: Each dot represents the location of the lid at a time step.
Revised: Each dot represents the location of the lid at a time step.

Original: BC had the worst performance among allmodels since it failed to complete tracing of the circle, square, and number 8 which requires a largerrange of motion, and the BC agent seemed to get stuck during execution.
Revised: BC had the worst performance among all models since it failed to complete tracing of the circle, square, and number 8 which requires a larger range of motion, and the BC agent seemed to get stuck during execution.

Original: Meanwhile, ORL methodslargely succeeded tracing the entire curve following the time-varying goals, demonstrating strongergeneralizing ability for this dynamic task.
Revised: Meanwhile, ORL methods largely succeeded tracing the entire curve following the time-varying goals, demonstrating stronger generalizing ability for this dynamic task.

Original: [Table]
Revised: [Table]

Original: Transfer learning by leveraging heterogeneous dataset Table 5 evaluates the performance of
Revised: Transfer learning by leveraging heterogeneous dataset Table 5 evaluates the performance of

Original: ORL algorithms when trained with different combinations of datasets from multiple tasks.
Revised: ORL algorithms when trained with different combinations of datasets from multiple tasks.

Original: Weobserve that:
Revised: We observe that:

Original: The performance changes to ORL agents after leveraging offline data from different tasks canvary, due to the characteristics of the algorithm, the nature of the task, design of the reward functionand the data distribution.2.
Revised: The performance changes to ORL agents after leveraging offline data from different tasks can vary, due to the characteristics of the algorithm, the nature of the task, design of the reward function and the data distribution.

Original: We observed all ORL agents could improve their own performance using some task/data combinations.
Revised: We observed all ORL agents could improve their own performance using some task/data combinations.

Original: Noticeably, MOR E L achieved higher or comparable performance on all tasks afterleveraging more offline data.
Revised: Noticeably, MOR E L achieved higher or comparable performance on all tasks after leveraging more offline data.

Original: For instance, its performance on the lift task progressively improved ( 0 . 606 → 0 .
Revised: For instance, its performance on the lift task progressively improved ( 0 . 606 → 0 .

Original: 726 → 0 . 896 )
Revised: 726 → 0 . 896 )

Original: with the inclusion of data from slide and PnP tasks.
Revised: with the inclusion of data from slide and PnP tasks.

Original: Intuitively,MOR E L’s dynamic model training process could benefit from any realistic data, regardless of whetherthe data was in-domain or out-of-domain.3.
Revised: Intuitively, MOR E L’s dynamic model training process could benefit from any realistic data, regardless of whether the data was in-domain or out-of-domain.

Original: Certain task-agnostic data could provide overlapping data support and enable effective transferlearning, allowing some ORL agents to surpass imitation learning and even the best in-domain agents.
Revised: Certain task-agnostic data could provide overlapping data support and enable effective transfer learning, allowing some ORL agents to surpass imitation learning and even the best in-domain agents.

Original: On slide and lift , all ORL algorithms managed to surpass BC.
Revised: On slide and lift , all ORL algorithms managed to surpass BC.

Original: On PnP , AWAC achieved comparable performance as BCbut with a slightly higher mean using a combo of slide and lift data.
Revised: On PnP , AWAC achieved comparable performance as BCbut with a slightly higher mean using a combo of slide and lift data.

Original: Withour extensive ablations, we observe that the final best agent for each task is either an ORL algorithmor a tie between ORL and BC.4.
Revised: With our extensive ablations, we observe that the final best agent for each task is either an ORL algorithm or a tie between ORL and BC. 4.

Original: However, ORL algorithms are not guaranteed to increase performance or guaranteed to surpassthe best in-domain agents.
Revised: However, ORL algorithms are not guaranteed to increase performance or guaranteed to surpass the best in-domain agents.

Original: The performance changes of ORL after leveraging out-domain data arelikely to vary by agents, the task and dataset distribution.
Revised: The performance changes of ORL after leveraging out-domain data are likely to vary by agents, the task and dataset distribution.

Original: For instance, both AWAC and IQL agentshave worse performance for lift , when using slide + lift + PnP than using only slide + lift data( 0 . 

Revised: For instance, both AWAC and IQL agents have worse performance for lift , when using slide + lift + PnP than using only slide + lift data ( 0 .

Original: → 0 .
Revised: → 0 .

Original: → 0 . 
Revised: → 0 . 684 ).

Original: Training IQL for PnP using slide or slide + lift data (withoutusing PnP data), however, yielded even better results than using PnP data.
Revised: Training IQL for PnP using slide or slide + lift data (without using PnP data), however, yielded even better results than using PnP data.

Original: Qualitatively we observethat IQL agents trained with slide data were better at grasping the object than the ones trained with
Revised: Qualitatively we observe that IQL agents trained with slide data were better at grasping the object than the ones trained with

Original: PnP data at lifting the object, completing this first part of the task with more success while claimingdistance-to-goal reward bonus.
Revised: PnP data at lifting the object, completing this first part of the task with more success while claiming distance-to-goal reward bonus.

Original: With our extensive ablations, we observe that the final best agent for each task is either an ORL algorithm or a tie between ORL and BC.
Revised: With our extensive ablations, we observe that the final best agent for each task is either an ORL algorithm or a tie between ORL and BC.

Original: Random Seed Sweeping To improve the statistical significance of our results and to demonstratereproducibility, we conduct additional random seed sweeping (i.e., train an agent with 3 consecutiverandom seeds).
Revised: Random Seed Sweeping To improve the statistical significance of our results and to demonstrate reproducibility, we conduct additional random seed sweeping (i.e., train an agent with 3 consecutive random seeds).

Original: Results from additional agents (Appendix. ?? ) shows that the seed2seed variation ofour experiments are low and provide more statistical significance to some of our observations: ∼ 60%of newly trained agents change score by less than 1%, ∼ 90% of agents change by less than 2%, andthe maximum change was 6% from one agent (whose score change does not affect our conclusion).
Revised: Results from additional agents (Appendix. 7.6) shows that the seed2seed variation of our experiments are low and provide more statistical significance to some of our observations: ∼ 60% of newly trained agents change score by less than 1%, ∼ 90% of agents change by less than 2%, and the maximum change was 6% from one agent (whose score change does not affect our conclusion).

Original: [Table]
Revised: [Table]

Original: Comparison to previous works.
Revised: Comparison to previous works.

Original: Some of our in-domain conclusions are aligned with [40] and [39]:that behavior cloning demonstrates strong robustness to varying representations and tasks, servingas a competitive baseline in all four tasks tested.
Revised: Some of our in-domain conclusions are aligned with [40] and [39]: that behavior cloning demonstrates strong robustness to varying representations and tasks, serving as a competitive baseline in all four tasks tested.

Original: Even when BC is not the best, it has reasonableperformance that is no worse than 85% of the best in-domain agents.
Revised: Even when BC is not the best, it has reasonable performance that is no worse than 85% of the best in-domain agents.

Original: Our findings also provideempirical verification to one of [34]’s observations that ORL could outperform BC for tasks wherethe initial state distributions change during deployment, a common condition for real robotic task, orwhen the environment has a few “critical” states, as seen in our manipulation tasks.
Revised: Our findings also provide empirical verification to one of [34]’s observations that ORL could outperform BC for tasks where the initial state distributions change during deployment, a common condition for real robotic task, or when the environment has a few “critical” states, as seen in our manipulation tasks.

Original: In contrast toprevious works, however, we highlight that (1) IQL can be a competitive baseline for settings thatwere traditionally favoring behavior cloning, as it turns out to be the best in-domain agent on 2 out of4 tasks we tested, despite the lack of real robotic evaluation for IQL [18], (2) our extensive ablationson out-domain transfer learning are unique and allow us to verify several ORL algorithms’ capabilityin generalizing to task region with less data-support (Table. 3) and to dynamic tasks (Figure. 4), (3)we observe that leveraging heterogeneous data has enabled all ORL algorithms to improve their ownperformance on at least one of the tasks, allowing some to even surpass the best in-domain agents,which suggest that ORL can be an interesting paradigm for real-world robotic learning.
Revised: In contrast to previous works, however, we highlight that (1) IQL can be a competitive baseline for settings that were traditionally favoring behavior cloning, as it turns out to be the best in-domain agent on 2 out of 4 tasks we tested, despite the lack of real robotic evaluation for IQL [18], (2) our extensive ablations on out-domain transfer learning are unique and allow us to verify several ORL algorithms’ capability in generalizing to task region with less data-support (Table. 3) and to dynamic tasks (Figure. 4), (3) we observe that leveraging heterogeneous data has enabled all ORL algorithms to improve their own performance on at least one of the tasks, allowing some to even surpass the best in-domain agents, which suggest that ORL can be an interesting paradigm for real-world robotic learning.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we conducted an empirical study of representative ORL algorithms on real-worldrobotic learning tasks.
Revised: In this work, we conducted an empirical study of representative ORL algorithms on real-world robotic learning tasks.

Original: The study encompassed three representative ORL algorithms (along withbehavior cloning), four table-top manipulation tasks with a Franka-Panda robot arm, 3000+ traintrajectories, 3500+ evaluation trajectories, and 270+ human labor hours.
Revised: The study encompassed three representative ORL algorithms (along with behavior cloning), four table-top manipulation tasks with a Franka-Panda robot arm, 3000+ train trajectories, 3500+ evaluation trajectories, and 270+ human labor hours.

Original: Through our extensiveablation studies, we find that (1) even for in-domain tasks with abundant amount of high-quality data,
Revised: Through our extensive ablation studies, we find that (1) even for in-domain tasks with abundant amount of high-quality data,

Original: IQL can be a competitive baseline against the best behavior cloning policy, (2) for out-domain tasks,
Revised: IQL can be a competitive baseline against the best behavior cloning policy, (2) for out-domain tasks,

Original: ORL algorithms were able to generalize to task regions with low data-support and to dynamic tasks,(3) the performance changes of ORL after leveraging heterogeneous data are likely to vary by agents,the design of the task, and the characteristics of the data, (4) certain heterogeneous task-agnostic datacould provide overlapping data support and enable transfer learning, allowing ORL agents to improvetheir own performance and, in some cases, even surpass the best in-domain agents.
Revised: ORL algorithms were able to generalize to task regions with low data-support and to dynamic tasks, (3) the performance changes of ORL after leveraging heterogeneous data are likely to vary by agents, the design of the task, and the characteristics of the data, (4) certain heterogeneous task-agnostic data could provide overlapping data support and enable transfer learning, allowing ORL agents to improve their own performance and, in some cases, even surpass the best in-domain agents.

Original: Overall, (5) thebest agent for each task is either an ORL algorithm or a tie between ORL and BC.
Revised: Overall, (5) the best agent for each task is either an ORL algorithm or a tie between ORL and BC.

Original: Our rigorousempirical evaluations indicate that even in out-of-domain multi-task data regime, (more realistic inreal world setting) offline RL is an effective paradigm to leverage out of domain data.  
Revised: Our rigorous empirical evaluations indicate that even in out-of-domain multi-task data regime, (more realistic in real world setting) offline RL is an effective paradigm to leverage out of domain data.  Appendix 7.1 Canonical Task Setup We considered four canonical tasks: reach , slide , lift and PnP . To apply ORL, each task can be formulated as an MDP.

Revised: The state contains the joint position of the robot, the gripper open position ( R ∼ [0 , 0 . 08] ), (optionally) the velocity of the joints, (optionally) the tracked tag position and a goal position.

Revised: To facilitate RL training we came up with a continuous reward function for each task r : state → R , as shown in Table 6, considering the position of the gripper x , the position of tracked

Revised: AprilTag t (if exists), the position of goal g , the Euclidean distance function dis between two 3D coordinates, a convenient function height to denote the height of a given coordinates.

Revised: While the reward for reach and slide are naturally smaller than 1, we explicitly cap the maximum reward for lift to be 1 since we don’t encourage agents to lift up the lid arbitrarily high.

Revised: We don’t cap the

Revised: PnP reward since we encourage the pick-n-place policy to be distinguished from the policy with a height bonus height ( t ) .

Revised: We used heuristic policies to collect the demonstration data, as described in Sec. 3.

Revised: Our policies have a reasonable success rate accomplishing the task but is not designed to be optimal in solving the MDP.

Revised: To evaluate and compare between agents, we instead report the maximum reward over the trajectory as a proxy of the task completion ("score").

Revised: We report our heuristic policies’ accumulated reward average over trajectories and the score.

Revised: [Table]

Revised: [Table]

Revised: 7.2 Dataset

Revised: In addition to the reward functions and statistics of our dataset, we also attach the score distribution on each task to demonstrate our dataset’s overall quality.

Revised: From Figure 3, we can see that the score distribution for each task skew heavily to the left, which means the datasets are suitable for imitation learning as well.

Revised: 7.3 Open Source Code and Dataset

Revised: To remain anonymity we have only uploaded our collected dataset to here.

Revised: Once accepted, we would share code and instructions on how to process and use our dataset.

Revised: [Figure]

Revised: 7.4 Training Details

Revised: Our code base was built upon the author’s implementation of MOR E L [12] and the D3RLPY [56] library.

Revised: We used the same fixed random seed for all our experiments, unless otherwise specified.

Revised: For hyperparameter tuning, we always started by training using the default hyperparameters.

Revised: If the training loss reported by the agent did not converge, we adjusted the learning rate and retrain, up toagents, till we find a model that converge or have been trained for 5 , 000 , 000 steps using batch size 2048 .

Revised: For model whose training loss exploded (e.g., AWAC), we choose an checkpoint from earlier of the training when the loss were relatively stable for 100 , 000 steps (frequently, this was an agent that finished about half a million to a million training steps).

Revised: Surprisingly, when evaluated on real robot, models that reported convergence did not necessarily perform better than model that did not converge.

Revised: Practicality of Training and Tuning

Revised: BC was the cheapest to train ( ∼ 3min) and easiest to converge (no additional tuning required).

Revised: MOR E L was the second shortest to train ( ∼ 4 hours); most MOR E L agents were able to converge, judged by the reward of trajectories generated by the learned dynamics model.

Revised: AWAC agents took longer to train ( ∼ 12 hours) and had the most trouble converging (8 of the 16 agents in the ablation table could not converge in allocated trials).

Revised: IQL agents took the longest to train (10 ∼ 24 hours) but had more success converging.

Revised: Though loss convergence during training or a good reward estimated by the learned dynamics model or learned value function cannot indicate the agent’s true performance, it is helpful for selecting an agent to test.

Revised: Since some

Revised: AWAC agents had trouble converging, we selected an earlier checkpoint before loss explosion and documented their performance, which, surprisingly, yielded higher reward than some agents that reported convergence.

Revised: We leave it to future work to investigate this phenomenon.

Revised: 7.5 Training Behavior Cloning with Top-K% Trajectories

Revised: To ensure that our dataset contains high quality trajectories that is sufficient to train behavior cloning, we launched new experiments training behavior cloning using only the Top-k % of the best trajectories.

Revised: In Figure.

Revised: 3, we plot the distribution of performance of our data for each task.

Revised: For reach , slide , lift , 90% of trajectories complete the task with good scores (> 0.75).

Revised: For (our most difficult task), 50% of our collected trajectories completed the task (scores > 0.8).

Revised: Thus we train BC for reach , slide , lift on Top-90% of data and train BC for PnP on Top50%,70%,90% of data and observe that, BC in our experiments benefit from using the full dataset.

Revised: [Table]

Revised: 7.6 Sweeping of Random Seeds

Revised: We evaluated an addition of 28 agents for 340 trajectories for a total of 70 hours including training and testing to inspect how the scores for critical agents (i.e., the best agents for a category) would vary by random seeds.

Revised: We now have 3 seeds for each of the following agents:

Revised: The Best Agents for each task in Table2.

Revised: The Second Best Agents for each task in Table3.

Revised: ORL agents with out-domain datasets in in Table

Revised: The original agents are trained with seed 123, we trained the additional agents with seed 122 and seed 124.

Revised: Each seed is evaluated on 12 trajectories.

Revised: The results are listed and we observe that ∼ 60% of newly trained agents change score by less than 1%, ∼ 90% of agents change by less than 2%, and the maximum change was 6% from one agent (whose score change does not affect our conclusion).

Revised: Best Agents

Revised: Seed

Revised: Seed

Revised: Seed

Revised: Means w/

Revised: Mean diff in Table

Revised: [Table]

Revised: AWAC,

Revised: DeltaVel,

Revised: [Table]

Revised: IQL,

Revised: DeltaVel, slide

Revised: [Table]

Revised: IQL,

Revised: DeltaVel, lift

Revised: [Table]

Revised: BC,

Revised: [Table]

Revised: [Table]

Revised: Second Best

Revised: Seed

Revised: [Table]

Revised: [Table]

Revised: Means

Revised: [Table]

Revised: in Table(

Revised: [Table]

Revised: seeds

Revised: MOREL,

Revised: DeltaVel, reach 0.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: , slide

Revised: [Table]

Revised: MOREL,

Revised: DeltaVel,

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: BC,

Revised: [Table]

Revised: , lift

Revised: [Table]

Revised: ORL

Revised: Seed

Revised: Seed

Revised: Seed

Revised: Means w/

Revised: Mean diff in Table(

Revised: [Table]

Revised: seeds

Revised: [Table]

Revised: w/ slide+

Revised: [Table]

Revised: MOREL on lift

Revised: [Table]

Revised: 7.7 Statistical Significance of Conclusions

Revised: In this section we verify the statistical significance of the conclusions we drew from our empirical study.

Revised: To evaluate every trained agent for every task, we collected 12 trajectories and calculated their scores.

Revised: One one hand, the estimated standard deviations of such scores were large, making the comparison between agents challenging (i.e. comparing 0 .

Revised: 818 ± 0 .

Revised: 161 with 0 .

Revised: 884 ± 0 . 120 ).

Revised: On the other hand, the distribution of scores is unknown.

Revised: We cannot exclude the possibility of the distribution being skewed, as the agent could perform better in a certain task region because of the nature of the task.

Revised: Therefore, we conducted both the dependent t-test ( p ) and the Wilcoxon signed

Revised: T-test ( p w ) for paired samples to calculate the p-value to reject or accept this null hypothesis: the two models’ have identical scores.

Revised: We will reject the hypothesis with a small p-value ( p or p w < 0 . 1 ).

Revised: Tasks and application-domains determine the confidence level requirements for any application.

Revised: This often requires domain knowledge and might not transfer between different applications even for the same task.

Revised: For openness and interpretability, we clearly outline our statistical tests and list our p-values, leaving it up to the readers to justify their statistical significance required for their applications.

Revised: We found that:

Revised: On in-domain tasks, we

Revised: initially

Revised: observe that: on reach , BC and the best ORL agent (AWAC) achieved similar performance ( 0 .

Revised: 93 ∼ 0 .

Revised: 93 , p = 0 .

Revised: 953 , p w = 0 . 844 ); on slide , IQL outperform BC ( 0 .

Revised: 76 > 0 .

Revised: 64 , p = 0 .

Revised: 066 , p w = 0 .

Revised: 110 ); on lift , we observe that BC is identical to the best ORL ( 0 .

Revised: 82 ∼ 0 .

Revised: 88 , p = 0 .

Revised: 146 , p w = 0 .

Revised: 110 ); on PnP , we observed that BC outperformed the best ORL agent ( 0 . 90 > 0 .

Revised: 75 , p = 0 .

Revised: 012 , p w = 0 . 016 ).

Revised: After running the best and the second best agents with multiple

Revised: seeds,

Revised: we

Revised: can confirm

Revised: the

Revised: statistical

Revised: significance of

Revised: IQL outperforming BC on lift and

Revised: slide

Revised: 88 >

Revised: p = 0 .

Revised: 084 , p w = 0 .

Revised: 77 > 0 .

Revised: 68 , p = 0 .

Revised: 001 , p w = 0 .

Revised: With such observation, we recommend IQL and BC as a strong baseline for in-domain tasks.

Revised: Testing agent’s ability to generalize to task space lacking data support

Revised: , we verify that MOREL and AWAC achieved comparable performance or better to BC for regions lacking data support (MOREL: 0 .

Revised: 80 ∼ 0 .

Revised: 77 , p = 0 .

Revised: 235 , p w = 0 .

Revised: 500 , AWAC: 0 .

Revised: 82 > 0 .

Revised: 77 , p = 0 .

Revised: 006 , p w = 0 .

Revised: It’s worth noting that MOREL was having an initial disadvantage of having poorer performance on regions that have more data support ( 0 .

Revised: 67 < 0 .

Revised: 78 , p = 0 .

Revised: 050 , p w = 0 . 062 ).

Revised: In terms of leveraging task-agnostic data, MOREL has benefited from inclusion of more data.

Revised: On Slide, the model achieved significantly higher performance when using combined data from three tasks 0 .

Revised: 64 ∼ 0 .

Revised: 72 , p = 0 .

Revised: 113 , p w = 0 . 027 ).

Revised: On Lift, the model achieved significantly higher performance when using combined data from three tasks ( 0 .

Revised: 65 → 0 .

Revised: 91 , p = 0 .

Revised: 000 , p w = 0 . 003 ).

Revised: AWAC and IQL agents, however, had less success achieved higher scores.

Original: {+The only significant improvement is AWAC on Lifting ( 0 . 
+}
Revised: {+The only significant improvement is AWAC on Lifting ( 0 . 
+}

Revised: → 0 . 90 , p = 0 .

Revised: 082 , p w = 0 .

Revised: Otherwise, AWAC agents performed the same irregardless of training data ( p > 0 . 1 ).

Revised: IQL had mostly similar or worse performance leveraging more data (e.g. worse slide performance: 0 .

Revised: 70 → 0 .

Revised: 64 , p = 0 .

Revised: 069 , p w = 0 . 077 ).