Original: AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scale
Revised: AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scale

Original: Abstract: Robotic skills can be learned via imitation learning (IL) using userprovided demonstrations, or via reinforcement learning (RL) using large amountsof autonomously collected experience.
Revised: Abstract: Robotic skills can be learned via imitation learning (IL) using userprovided demonstrations, or via reinforcement learning (RL) using large amounts of autonomously collected experience.

Original: Both methods have complementarystrengths and weaknesses: RL can reach a high level of performance, but requiresexploration, which can be very time consuming and unsafe; IL does not requireexploration, but only learns skills that are as good as the provided demonstrations.
Revised: Both methods have complementary strengths and weaknesses: RL can reach a high level of performance, but requires exploration, which can be very time consuming and unsafe; IL does not require exploration, but only learns skills that are as good as the provided demonstrations.

Original: Can a single method combine the strengths of both approaches?
Revised: Can a single method combine the strengths of both approaches?

Original: A number ofprior methods have aimed to address this question, proposing a variety of techniques that integrate elements of IL and RL.
Revised: A number of prior methods have aimed to address this question, proposing a variety of techniques that integrate elements of IL and RL.

Original: However, scaling up such methodsto complex robotic skills that integrate diverse ofﬂine data and generalize mean- ingfully to real-world scenarios still presents a major challenge.
Revised: However, scaling up such methods to complex robotic skills that integrate diverse ofﬂine data and generalize meaningfully to real-world scenarios still presents a major challenge.

Original: In this paper, our aim is to test the scalability of prior IL + RL algorithms and devise a system basedon detailed empirical experimentation that combines existing components in the most effective and scalable way.
Revised: In this paper, our aim is to test the scalability of prior IL + RL algorithms and devise a system based on detailed empirical experimentation that combines existing components in the most effective and scalable way.

Original: To that end, we present a series of experimentsaimed at understanding the implications of each design decision, so as to develop acombined approach that can utilize demonstrations and heterogeneous prior datato attain the best performance on a range of real-world and realistic simulatedrobotic problems.
Revised: To that end, we present a series of experiments aimed at understanding the implications of each design decision, so as to develop a combined approach that can utilize demonstrations and heterogeneous prior data to attain the best performance on a range of real-world and realistic simulated robotic problems.

Original: Our complete method, which we call AW-Opt, combines elements of advantage-weighted regression [1, 2] and QT-Opt [3], providing a uniﬁedapproach for integrating demonstrations and ofﬂine data for robotic manipulation. 
Revised: Our complete method, which we call AW-Opt, combines elements of advantage-weighted regression [1, 2] and QT-Opt [3], providing a uniﬁed approach for integrating demonstrations and ofﬂine data for robotic manipulation. Please see https://awopt.github.io for more details.

Original:  Introduction
Revised:  Introduction

Original: Learning methods for robotic control are conventionally divided into two groups: methods based onautonomous trial-and-error (reinforcement learning) and methods based on imitating user-provideddemonstrations (imitation learning).
Revised: Learning methods for robotic control are conventionally divided into two groups: methods based on autonomous trial-and-error (reinforcement learning), and methods based on imitating user-provided demonstrations (imitation learning).

Original: These two approaches have complementary strengths andweaknesses.
Revised: These two approaches have complementary strengths and weaknesses.

Original: Reinforcement Learning (RL) enables robots to improve autonomously, but introducesigniﬁcant challenges with exploration and stable learning.
Revised: Reinforcement Learning (RL) enables robots to improve autonomously, but introduce signiﬁcant challenges with exploration and stable learning.

Original: Imitation learning (IL) methods providefor more stable learning from expert demonstrations but cannot improve from failures.
Revised: Imitation learning (IL) methods provide for more stable learning from expert demonstrations, but cannot improve from failures.

Original: This couldpotentially make covering the full distribution of data difﬁcult.
Revised: This could potentially make covering the full distribution of data difﬁcult.

Original: Prior works have sought to combine elements of IL and RL into a single algorithm (IL+RL), bypre-training RL policies with demonstrations [4], incorporating demonstration data into off-policy
Revised: Prior works have sought to combine elements of IL and RL into a single algorithm (IL+RL), by pre-training RL policies with demonstrations [4], incorporating demonstration data into off-policy

Original: RL algorithms [5], and combining IL and RL loss functions into a single algorithm [6].
Revised: RL algorithms [5], and combining IL and RL loss functions into a single algorithm [6].

Original: Whilesuch algorithms have been based on both policy gradients and value-based off-policy methods, thelatter are particularly relevant in robotics settings due to their ability to leverage both demonstrationsand the robot’s own prior experience.
Revised: While such algorithms have been based on both policy gradients and value-based off-policy methods, the latter are particularly relevant in robotics settings due to their ability to leverage both demonstrations and the robot’s own prior experience.

Original: A number of such methods have shown promising results insimulation and in the real world [7], but they often require extensive parameter tuning and, as weshow in our experiments, are difﬁcult to deploy at scale to solve multiple tasks in diverse scenarios.
Revised: A number of such methods have shown promising results in simulation and in the real world [7], but they often require extensive parameter tuning and, as we show in our experiments, are difﬁcult to deploy at scale to solve multiple tasks in diverse scenarios.

Original: In this paper, our goal is to develop, through extensive experimentation, a complete and scalablesystem for integrating IL and RL that enables learning robotic control policies from both demonstration data and suboptimal experience.
Revised: In this paper, our goal is to develop, through extensive experimentation, a complete and scalable system for integrating IL and RL that enables learning robotic control policies from both demonstration data and suboptimal experience.

Original: Our aim is not to propose new principles for algorithmsthat combine IL and RL, but rather to determine how to best combine the components of existingmethods to enable effective large-scale robotic learning.
Revised: Our aim is not to propose new principles for algorithms that combine IL and RL, but rather to determine how to best combine the components of existing methods to enable effective large-scale robotic learning.

Revised: We begin our investigation with two exist5th Conference on Robot Learning (CoRL 2021), London, UK.

Original: We begin our investigation with two existing methods: AWAC [2], which provides an integrated framework for combining IL and RL, and
Revised: methods: AWAC [2] which provides an integrated framework for combining IL and RL, and

Original: QT-Opt [3], which provides a scalable system for robotic reinforcement learning.
Revised: QT-Opt [3], which provides a scalable system for robotic reinforcement learning.

Original: We observe thatneither method by itself provides a complete solution to large-scale RL with demonstrations.
Revised: We observe that neither method by itself provides a complete solution to large-scale RL with demonstrations.

Original: Wethen systematically study the decisions in these methods, identify their shortcomings, and develop ahybrid approach that we call AW-Opt.
Revised: We then systematically study the decisions in these methods, identify their shortcomings and develop a hybrid approach that we call AW-Opt.

Original: The individual design decisions that comprise AW-Opt mayappear minor in isolation, but we show that combining these decisions signiﬁcantly improves on theprior approaches, and scales effectively to learning high-capacity generalizable policies in multiplereal and simulated environments shown in Fig.
Revised: The individual design decisions that comprise AW-Opt may appear minor in isolation but we show that combining these decisions signiﬁcantly improves on the prior approaches and scales effectively to learning high-capacity generalizable policies in multiple real and simulated environments shown in Fig.

Original: 1 (a-d).
Revised: 1 (a-d).

Original: The main contributions of this work consist of a detailed analysis of the design decisions that inﬂuence performance and scalability of IL+RL methods, leading to the development of AW-Opt, ahybrid algorithm that combines insights from multiple prior approaches to achieve scalable and effective robotic learning from demonstrations, suboptimal ofﬂine data, and online exploration.
Revised: The main contributions of this work consist of a detailed analysis of the design decisions that inﬂuence performance and scalability of IL+RL methods, leading to the development of AW-Opt, a hybrid algorithm that combines insights from multiple prior approaches to achieve scalable and effective robotic learning from demonstrations, suboptimal ofﬂine data, and online exploration.

Original: Notonly can AW-Opt be successfully initialized from demonstrations (initial success in Fig. 1(f)), butalso it responds well to RL ﬁne-tuning (ﬁnal success in Fig. 1(f)) while being sample-efﬁcient.
Revised: Not only can AW-Opt be successfully initialized from demonstrations (initial success in Fig. 1(f)), but also it responds well to RL ﬁne-tuning (ﬁnal success in Fig. 1(f)) while being sample-efﬁcient.

Original: In addition to the design of AW-Opt, we hope that the analysis, ablation experiments, and detailedevaluation of individual design decisions that we present in this work would help to guide the development of effective and scalable robotic learning algorithms.
Revised: In addition to the design of AW-Opt, we hope that the analysis, ablation experiments, and detailed evaluation of individual design decisions that we present in this work would help to guide the development of effective and scalable robotic learning algorithms.

Original: Our experimental results evaluateour method, as well as ablations, across a range of simulated and real-world domains, including twodifferent real-world platforms and large datasets with hundreds of thousands of training episodes.
Revised: Our experimental results evaluate our method, as well as ablations, across a range of simulated and real-world domains, including two different real-world platforms and large datasets with hundreds of thousands of training episodes.

Original:  Related Work
Revised:  Related Work

Original: Prior works have proposed a number of methods for learning robotic skills via imitation of experts [8, 9, 10, 11, 12, 13], and via reinforcement learning [14, 15, 16, 17, 18].
Revised: Prior works have proposed a number of methods for learning robotic skills via imitation of experts [8, 9, 10, 11, 12, 13], and via reinforcement learning [14, 15, 16, 17, 18].

Original: In the latter category,off-policy RL methods are especially appealing for large-scale robot learning, since they allow fordata reuse with large, diverse datasets [3, 19, 20].
Revised: In the latter category, off-policy RL methods are especially appealing for large-scale robot learning, since they allow for data reuse with large, diverse datasets [3, 19, 20].

Original: Initially, methods that combined imitation learning (IL) and reinforcement learning (RL) have tended to treat the two data sources differently, using
Revised: Initially, methods that combined imitation learning (IL) and reinforcement learning (RL) have tended to treat the two data sources differently, using

Original: IL demos only at initialization [4, 5, 21], or using different losses for IL demos vs. RL episodes [6].
Revised: IL demos only at initialization [4, 5, 21], or using different losses for IL demos vs. RL episodes [6].

Original: Other methods have used a different approach: combining both expert demos and suboptimal priordata into a single dataset used for off-policy value-based learning [7].
Revised: Other methods have used a different approach: combining both expert demos and suboptimal prior data into a single dataset used for off-policy value-based learning [7].

Original: Ofﬂine RL methods, such as
Revised: Ofﬂine RL methods, such as

Original: CQL [22], QT-Opt [3], and AWR [1] often use this approach, along with constrained optimizationand regularization to encourage staying close to the data distribution [23, 24, 25, 26, 27].
Revised: CQL [22], QT-Opt [3], and AWR [1] often use this approach, along with constrained optimization and regularization to encourage staying close to the data distribution [23, 24, 25, 26, 27].

Original: Our methodbuilds on these approaches, due to their effectiveness, simplicity, and easier data management.
Revised: Our method builds on these approaches, due to their effectiveness, simplicity, and easier data management.

Original: Regardless of whether the data consists of optimal demonstrations, suboptimal human-provided demos,or very suboptimal robot experience, all data can be treated the same way.
Revised: Regardless of whether the data consists of optimal demonstrations, suboptimal human-provided demos, or very suboptimal robot experience, all data can be treated the same way.

Original: Our goal is to scale up IL+RL methods to large datasets with many objects and varied environments.
Revised: Our goal is to scale up IL+RL methods to large datasets with many objects and varied environments.

Original: Standard RL methods have proven difﬁcult to scale up to the same level as modern methods incomputer vision [28, 29] and natural language processing [30, 31, 32], where datasets are diverse andopen-world, and models have millions or billions of parameters.
Revised: Standard RL methods have proven difﬁcult to scale up to the same level as modern methods in computer vision [28, 29] and natural language processing [30, 31, 32], where datasets are diverse and open-world, and models have millions or billions of parameters.

Original: Taking RL to such settings requireslarge, distributed RL systems, which often require careful design decisions in their underlying RLmethods.
Revised: Taking RL to such settings requires large, distributed RL systems, which often require careful design decisions in their underlying RL methods.

Original: While standard methods such as actor-critic have been scaled up in this way [33, 34,35, 36], we show in this work that scaling IL+RL algorithms requires a certain amount of care,especially in robotic settings where collecting real-world experience is costly.
Revised: While standard methods such as actor-critic have been scaled up in this way [33, 34, 35, 36], we show in this work that scaling IL+RL algorithms requires a certain amount of care, especially in robotic settings where collecting real-world experience is costly.

Original: Na¨ıvely incorporatingdemonstrations into an RL system (QT-Opt in our experiments) does not by itself yield good results.
Revised: Na¨ıvely incorporating demonstrations into an RL system (QT-Opt in our experiments) does not by itself yield good results.

Original: Similarly, utilizing prior IL+RL methods (such as AWAC [2] in our experiments) scales poorly.
Revised: Similarly, utilizing prior IL+RL methods (such as AWAC [2] in our experiments) scales poorly.

Original: Weprovide a detailed empirical evaluation of various design decisions underlying IL+RL methods todevelop a hybrid approach, which we call AW-Opt, that combines the scalability of QT-Opt with theability to seamlessly incorporate suboptimal experience and bootstrap from demonstrations.
Revised: We provide a detailed empirical evaluation of various design decisions underlying IL+RL methods to develop a hybrid approach, which we call AW-Opt, that combines the scalability of QT-Opt with the ability to seamlessly incorporate suboptimal experience and bootstrap from demonstrations.

Original:  Preliminaries
Revised:  Preliminaries

Original: Let M =
Revised: Let M =

Original: ( S , A , P, R, p 0 , γ, T ) deﬁne a Markov decision process (MDP), where S and A are stateand action spaces, P : S ×
Revised: ( S , A , P, R, p 0 , γ, T ) deﬁne a Markov decision process (MDP), where S and A are state and action spaces, P : S ×

Original: A × S → R + is a state-transition probability function, R : S ×
Revised: A × S → R + is a state-transition probability function, R : S ×

Original: A → Ris a reward function, p 0 : S → R + is an initial state distribution, γ is a discount factor, and T isthe task horizon.
Revised: A → R is a reward function, p 0 : S → R + is an initial state distribution, γ is a discount factor, and T is the task horizon.

Original: We use τ = ( s 0 , a 0 , . . .
Revised: We use τ = ( s 0 , a 0 , . . .

Original: , s T , a T ) to denote a trajectory of states and actions and
Revised: , s T , a T ) to denote a trajectory of states and actions and

Original: R ( τ ) =
Revised: R ( τ ) =

Original: (cid:80) Tt =

Revised: (cid:80) Tt =

Original: γ t R ( s t , a t ) to denote the trajectory reward.
Revised: γ t R ( s t , a t ) to denote the trajectory reward.

Original: Reinforcement learning (RL) methodsﬁnd a policy π ( a | s ) that maximizes the expected discounted reward over trajectories induced by thepolicy: E π [ R ( τ )] where s 0 ∼ p 0 , s t +1 ∼ P ( s t +1 | s t , a t ) and a t ∼ π ( a t | s t ) .
Revised: Reinforcement learning (RL) methods ﬁnd a policy π ( a | s ) that maximizes the expected discounted reward over trajectories induced by the policy: E π [ R ( τ )] where s 0 ∼ p 0 , s t +1 ∼ P ( s t +1 | s t , a t ) and a t ∼ π ( a t | s t ) .

Original: In imitation learning, demonstrations are provided to facilitate learning an optimal policy, whichcan be produced through teleoperation [37, 38] or kinesthetic teaching [39, 12].
Revised: In imitation learning, demonstrations are provided to facilitate learning an optimal policy, which can be produced through teleoperation [37, 38] or kinesthetic teaching [39, 12].

Original: Given a data set ofdemonstration state-action tuples ( s, a ) ∼ D demo , standard IL methods aim to directly approximatethe expert’s π ∗ ( a | s ) via supervised learning.
Revised: Given a data set of demonstration state-action tuples ( s, a ) ∼ D demo , standard IL methods aim to directly approximate the expert’s π ∗ ( a | s ) via supervised learning.

Original: Another way of learning from prior data has been explored in the ﬁeld of ofﬂine RL [40], where RL methods are used to learn from datasets of previouslycollected data.
Revised: Another way of learning from prior data has been explored in the ﬁeld of ofﬂine RL [40], where RL methods are used to learn from datasets of previously collected data.

Original: This data can include demonstrations, as well as other suboptimal trajectories.
Revised: This data can include demonstrations, as well as other suboptimal trajectories.

Original: Suchofﬂine RL methods are particularly appealing for hybrid IL+RL methods, since they do not requireany explicit distinction between demonstration data and suboptimal experience.
Revised: Such ofﬂine RL methods are particularly appealing for hybrid IL+RL methods, since they do not require any explicit distinction between demonstration data and suboptimal experience.

Original: In this paper, we build on these approaches to develop a scalable algorithm that combines IL and
Revised: In this paper, we build on these approaches to develop a scalable algorithm that combines IL and

Original: RL.
Revised: RL.

Original: Our goal is to smoothly transition between two sources of supervision: the demonstrated behaviors, provided as prior data, and RL exploration, collected autonomously.
Revised: Our goal is to smoothly transition between two sources of supervision: the demonstrated behaviors, provided as prior data, and RL exploration, collected autonomously.

Original: Like many ofﬂine RLalgorithms, our approach is based on Q-learning, which extracts the policy from a Q-function that istrained to satisfy the Bellman equation: Q π ( s t , a t ) =
Revised: Like many ofﬂine RL algorithms, our approach is based on Q-learning, which extracts the policy from a Q-function that is trained to satisfy the Bellman equation: Q π ( s t , a t ) =

Original: R ( s t , a t ) + γ max a [ Q π ( s t +1 , a )] .
Revised: R ( s t , a t ) + γ max a [ Q π ( s t +1 , a )] .

Original: To develop an effective IL-RL algorithm, we start by analyzing various design decisions in two recently proposed ofﬂine RL methods: QT-Opt [3] and Advantage-Weighted Regression (AWR)
Revised: To develop an effective IL-RL algorithm, we start by analyzing various design decisions in two recently proposed ofﬂine RL methods: QT-Opt [3] and Advantage-Weighted Regression (AWR)

Original: [1],with its recent value-based instantiation AWAC [2].
Revised: [1], with its recent value-based instantiation AWAC [2].

Original: QT-Opt is a distributed Q-learning frameworkthat enables learning Q-functions in continuous action spaces by maximizing the Q-function usingthe cross-entropy method (CEM) [41], without an explicit actor model.
Revised: QT-Opt is a distributed Q-learning framework that enables learning Q-functions in continuous action spaces by maximizing the Q-function using the cross-entropy method (CEM) [41], without an explicit actor model.

Original: However, initializing a Qfunction from demonstration data is particularly challenging as it often results in over-optimistic
Revised: However, initializing a Qfunction from demonstration data is particularly challenging as it often results in over-optimistic

Original: Q-values on unseen state-action pairs [42, 24, 22].
Revised: Q-values on unseen state-action pairs [42, 24, 22].

Original: It has been shown that by introducing an explicitactor, one can mitigate this issue by limiting the actor to select actions that are within the (conditional) distribution of actions seen in the data [24, 1].
Revised: It has been shown that by introducing an explicit actor, one can mitigate this issue by limiting the actor to select actions that are within the (conditional) distribution of actions seen in the data [24, 1].

Original: This property is also leveraged by AWAC [2],where the actor is initialized from demonstration data, and then further ﬁne-tuned with RL rewards.
Revised: This property is also leveraged by AWAC [2], where the actor is initialized from demonstration data, and then further ﬁne-tuned with RL rewards.

Original: The AWAC algorithm uses the following update to the actor:
Revised: The AWAC algorithm uses the following update to the actor:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where A ( s, a ) =
Revised: where A ( s, a ) =

Original: Q ( s, a ) − E π ( a | s )
Revised: Q ( s, a ) − E π ( a | s )

Original: [ Q ( s, a )] is the advantage function, Z ( s ) is the normalizingpartition function, λ is a Lagrange multiplier, π θ is the learned policy and π β is the behavior policy,which represents the (unknown) distribution that produces the dataset.
Revised: [ Q ( s, a )] is the advantage function, Z ( s ) is the normalizing partition function, λ is a Lagrange multiplier, π θ is the learned policy and π β is the behavior policy, which represents the (unknown) distribution that produces the dataset.

Original: This corresponds to weightedmaximum likelihood training, using samples from π β ( a | s ) (obtained from the dataset).
Revised: This corresponds to weighted maximum likelihood training, using samples from π β ( a | s ) (obtained from the dataset).

Original: In this paper, we analyze various design decisions in QT-Opt and AWAC, point out their shortcomings in the IL+RL setting, and use these insights to construct an effective hybrid IL+RL method thatwe call AW-Opt.
Revised: In this paper, we analyze various design decisions in QT-Opt and AWAC, point out their shortcomings in the IL+RL setting, and use these insights to construct an effective hybrid IL+RL method that we call AW-Opt.

Original:  Learning via Imitation and Reinforcement with AW-Opt
Revised:  Learning via Imitation and Reinforcement with AW-Opt

Original: In order to derive AW-Opt, we ﬁrst study the performance of QT-Opt [3] and AWAC [2] in the
Revised: In order to derive AW-Opt, we ﬁrst study the performance of QT-Opt [3] and AWAC [2] in the

Original: IL+RL setting, using a complex robotic manipulation task with high-resolution image observationsas our working example.
Revised: IL+RL setting, using a complex robotic manipulation task with high-resolution image observations as our working example.

Original: This setting requires algorithms that are more scalable than those typicallyemployed in standard RL benchmarks.
Revised: This setting requires algorithms that are more scalable than those typically employed in standard RL benchmarks.

Original: We will show that neither QT-Opt nor AWAC effectivelyhandle this setting.
Revised: We will show that neither QT-Opt nor AWAC effectively handle this setting.

Original: We then progressively construct a new algorithm with elements of both approaches, which we call AW-Opt, analyzing each design decision in turn.
Revised: We then progressively construct a new algorithm with elements of both approaches, which we call AW-Opt, analyzing each design decision in turn.

Original: Although each individualdesign decision in AW-Opt may appear relatively simple, we will see that the speciﬁc combinationof these design decisions outperforms both QT-Opt and AWAC by a very large margin, resulting insuccessful learning for tasks where the two prior methods are unable to make meaningful progress.
Revised: Although each individual design decision in AW-Opt may appear relatively simple, we will see that the speciﬁc combination of these design decisions outperforms both QT-Opt and AWAC by a very large margin, resulting in successful learning for tasks where the two prior methods are unable to make meaningful progress.

Original: 4.1 Example Task and Baseline Performance
Revised: 4.1 Example Task and Baseline Performance

Original: As a working example, we use a vision-basedtrash sorting task, where the robot must pickup a speciﬁc type of trash object (e.g., a compostable objects) out of cluttered bins containing diverse object types, including recyclables,landﬁll objects, etc., with only sparse rewardsindicating success once the robot has lifted thecorrect object.
Revised: As a working example, we use a vision-based trash sorting task, where the robot must pick up a speciﬁc type of trash object (e.g., a compostable objects) out of cluttered bins containing diverse object types, including recyclables, landﬁll objects, etc., with only sparse rewards indicating success once the robot has lifted the correct object.

Original: We describe this task in detailin Section 5.
Revised: We describe this task in detail in Section 5.

Original: We will use this environment to illustrate the importance of each component of ourmethod, as we introduce them in this section.
Revised: We will use this environment to illustrate the importance of each component of our method, as we introduce them in this section.

Original: Then, in Section 5, we will evaluate our complete
Revised: Then, in Section 5, we will evaluate our complete

Original: AW-Opt algorithm on a wider range of robotic environments, including two real-world robotic systems and several high-ﬁdelity simulation environments.
Revised: AW-Opt algorithm on a wider range of robotic environments, including two real-world robotic systems and several high-ﬁdelity simulation environments.

Original: Note that unless otherwise speciﬁed, all theplots in this paper is conducted with 3 experiments each.
Revised: Note that unless otherwise speciﬁed, all the plots in this paper is conducted with 3 experiments each.

Original: Each data point is an averaged success rateevaluated with 700 episodes.
Revised: Each data point is an averaged success rate evaluated with 700 episodes.

Original: The shaded region in the plot depicts the 90% conﬁdence interval withthe line being the mean.
Revised: The shaded region in the plot depicts the 90% conﬁdence interval with the line being the mean.

Original: We begin our investigation with the two prior methods: QT-Opt [3], which can scale well to highresolution images and diverse tasks, and AWAC [2], which is designed to integrate demonstrationsand then ﬁne-tune online.
Revised: We begin our investigation with the two prior methods: QT-Opt [3], which can scale well to highresolution images and diverse tasks, and AWAC [2], which is designed to integrate demonstrations and then ﬁne-tune online.

Original: Both algorithms are provided with 300 demonstrations for ofﬂine pretraining, and then switch to on-policy collection after 10k gradient steps to collect and train with200k additional on-policy episodes.
Revised: Both algorithms are provided with 300 demonstrations for ofﬂine pretraining, and then switch to on-policy collection after 10k gradient steps to collect and train with 200k additional on-policy episodes.

Original: In Fig.
Revised: In Fig.

Original: 2, we see that basic QT-Opt fails to make progress onboth the pre-training and online data.
Revised: 2, we see that basic QT-Opt fails to make progress on both the pre-training and online data.

Original: Q-learning methods are typically ineffective when providedwith only successful trials [40], since without both “positive” successful trajectories and “negative”failed trajectories, they cannot determine what not to do.
Revised: Q-learning methods are typically ineffective when provided with only successful trials [40], since without both “positive” successful trajectories and “negative” failed trajectories, they cannot determine what not to do.

Original: Then, during the online phase, any RLmethod that is not pretrained properly is faced with a difﬁcult exploration problem, and cannotmake progress.
Revised: Then, during the online phase, any RL method that is not pretrained properly is faced with a difﬁcult exploration problem, and cannot make progress.

Original: QT-Opt is one of those RL algorithms.
Revised: QT-Opt is one of those RL algorithms.

Original: AWAC [2] does attain a non-zero successrate from the demonstrations, but performance is still poor, and it does not scale successfully tothe high-dimensional and complex task during online ﬁne-tuning.
Revised: AWAC [2] does attain a non-zero success rate from the demonstrations, but performance is still poor, and it does not scale successfully to the high-dimensional and complex task during online ﬁne-tuning.

Original: Next, we will introduce a series of modiﬁcations to AWAC that bring it closer to QT-Opt, while retaining the ability to utilizedemonstrations, culminating in our full AW-Opt algorithm.
Revised: Next, we will introduce a series of modiﬁcations to AWAC that bring it closer to QT-Opt, while retaining the ability to utilize demonstrations, culminating in our full AW-Opt algorithm.

Original: 4.2 Positive Sample Filtering
Revised: 4.2 Positive Sample Filtering

Original: One possible explanation for the poor performance of AWAC in Fig.
Revised: One possible explanation for the poor performance of AWAC in Fig.

Original: 2 is that, with the relatively low success rate (around 4%) after pretraining, the addition of large amounts of additional failed episodes during online explorationdrowns out the initial successful demonstrations and, before the algorithm can learn an effective Q-function to weight the actor updates,results in updates that remove the promisingpre-training initialization.
Revised: 2 is that, with the relatively low success rate (around 4%) after pretraining, the addition of large amounts of additional failed episodes during online exploration drowns out the initial successful demonstrations and, before the algorithm can learn an effective Q-function to weight the actor updates, results in updates that remove the promising pre-training initialization.

Original: To address this issue,we introduce two modiﬁcations: (a) We use aprioritized buffer for the critic , where 50% of the data in each batch comes from successful (positive) episodes that end with a reward of 1 .
Revised: To address this issue, we introduce two modiﬁcations: (a) We use a prioritized buffer for the critic , where 50% of the data in each batch comes from successful (positive) episodes that end with a reward of 1 .

Original: Note that for tasks with non-binary rewards, we wouldneed to introduce an additional hyperparameter for the “success” threshold, which we leave forfuture work.
Revised: Note that for tasks with non-binary rewards, we would need to introduce an additional hyperparameter for the “success” threshold, which we leave for future work.

Original: (b) We use positive ﬁltering for the actor , applying the AWAC actor update in Equation 1 only on samples that pass the success ﬁlter (i.e., receive a reward of 1 ).
Revised: (b) We use positive ﬁltering for the actor , applying the AWAC actor update in Equation 1 only on samples that pass the success ﬁlter (i.e., receive a reward of 1 ).

Original: While modiﬁcation a) ensures that the critic’s training batch is balanced in terms of positive and negative samples, themodiﬁcation in b) plays a subtle but very important role for the actor.
Revised: While modiﬁcation a) ensures that the critic’s training batch is balanced in terms of positive and negative samples, the modiﬁcation in b) plays a subtle but very important role for the actor.

Original: By ﬁltering the data that ispresented to the actor, this modiﬁcation ensures that the actor’s performance should not drop belowbehavioral cloning performance, making it independent of the potentially inaccurate critic duringthe early stages of training.
Revised: By ﬁltering the data that is presented to the actor, this modiﬁcation ensures that the actor’s performance should not drop below behavioral cloning performance, making it independent of the potentially inaccurate critic during the early stages of training.

Original: After applying the above adjustments, AWAC performance no longer collapses after switching toonline ﬁne-tuning, as shown in Fig. 3, but still fails to improve signiﬁcantly, with a ﬁnal successrate of 5%.
Revised: After applying the above adjustments, AWAC performance no longer collapses after switching to online ﬁne-tuning, as shown in Fig. 3, but still fails to improve signiﬁcantly, with a ﬁnal success rate of 5%.

Original: We abbreviate this intermediate method as AWAC P (AWAC + P ositive ﬁltering).
Revised: We abbreviate this intermediate method as AWAC P (AWAC + P ositive ﬁltering).

Original: Toimprove it further, we will discuss how to improve exploration performance in the next subsection.
Revised: To improve it further, we will discuss how to improve exploration performance in the next subsection.

Original: 4.3 Hybrid Actor-Critic Exploration
Revised: 4.3 Hybrid Actor-Critic Exploration

Original: [Figure] As a Q-learning method, QT-Opt does not havean explicit actor.
Revised: As a Q-learning method, QT-Opt does not have an explicit actor.

Original: Since the task involves continuous actions, QT-Opt uses the cross-entropymethod (CEM) [41] to optimize the action using its critic, which can be viewed as an implicitpolicy (CEM policy).
Revised: Since the task involves continuous actions, QT-Opt uses the cross-entropy method (CEM) [41] to optimize the action using its critic, which can be viewed as an implicit policy (CEM policy).

Original: Unlike QT-Opt, AWACis an actor-critic algorithm, which means it hasaccess to both an explicit actor network anda critic network.
Revised: Unlike QT-Opt, AWAC is an actor-critic algorithm, which means it has access to both an explicit actor network and a critic network.

Original: Exploration is performed bysampling actions from this actor.
Revised: Exploration is performed by sampling actions from this actor.

Original: However, thiscan present an obstacle to learning in complexdomains.
Revised: However, this can present an obstacle to learning in complex domains.

Original: First, the commonly used unimodal actor distribution may not capture the multimodal intricacies of the true Q-function, especially early on in training when the optimal action has not beendetermined yet.
Revised: First, the commonly used unimodal actor distribution may not capture the multimodal intricacies of the true Q-function, especially early on in training when the optimal action has not been determined yet.

Original: Second, the bulk of the learning must be done by the critic, since the actor doesn’treason about long-horizon rewards.
Revised: Second, the bulk of the learning must be done by the critic, since the actor doesn’t reason about long-horizon rewards.

Original: Hence, the critic must ﬁrst learn which actions are good, andthen the actor must distill this knowledge into an action distribution, which can introduce delays.
Revised: Hence, the critic must ﬁrst learn which actions are good, and then the actor must distill this knowledge into an action distribution, which can introduce delays.

Original: Here, we aim to address these issues by utilizing both the actor and the critic for exploration.
Revised: Here, we aim to address these issues by utilizing both the actor and the critic for exploration.

Original: Although we are building on AWAC, we can still use the implicit CEM policy from QT-Opt to selectactions via the critic, bypassing the actor.
Revised: Although we are building on AWAC, we can still use the implicit CEM policy from QT-Opt to select actions via the critic, bypassing the actor.

Original: We therefore compare ﬁve exploration strategies: (a)actor-only exploration (b) implicit critic policy (as in QT-Opt, where CEM is used to pick the bestaction according to the critic); (c) episode-level random switcher, which randomly picks a policy atthe beginning of each episode (80% critic policy, 20% actor); and (d) step-level random switcher,which randomly picks a policy at each time step (80% critic policy, 20% actor).
Revised: We therefore compare ﬁve exploration strategies: (a) actor-only exploration (b) implicit critic policy (as in QT-Opt, where CEM is used to pick the best action according to the critic); (c) episode-level random switcher, which randomly picks a policy at the beginning of each episode (80% critic policy, 20% actor); and (d) step-level random switcher, which randomly picks a policy at each time step (80% critic policy, 20% actor).

Original: Fig.
Revised: Fig.

Original: 4 shows the results of this comparison.
Revised: 4 shows the results of this comparison.

Original: The episode level random switcher outperforms theother candidates, achieving more than 10% success within 80k samples.
Revised: The episode level random switcher outperforms the other candidates, achieving more than 10% success within 80k samples.

Original: Adopting the episode-levelrandom switcher, we call this intermediate algorithm AWAC P ELRS.
Revised: Adopting the episode-level random switcher, we call this intermediate algorithm AWAC P ELRS.

Original: Given the observed improvement in the early stage of exploration, we shift our focus to analyzing whether value propagation inthe critic is sufﬁciently able to utilize the data during online ﬁne-tuning.
Revised: Given the observed improvement in the early stage of exploration, we shift our focus to analyzing whether value propagation in the critic is sufﬁciently able to utilize the data during online ﬁne-tuning.

Original: 4.4 Action Selection in the Bellman Update
Revised: 4.4 Action Selection in the Bellman Update

Original: In the previous section, we showed how usingthe critic directly for exploration, rather thanthe actor, can lead to signiﬁcantly better performance.
Revised: In the previous section, we showed how using the critic directly for exploration, rather than the actor, can lead to signiﬁcantly better performance.

Original: We speculate that this is due to thefact that such an exploration strategy providesa degree of “lookahead”: the actor can onlylearn to take good actions after the Q-functionhas learned to assign such actions high values.
Revised: We speculate that this is due to the fact that such an exploration strategy provides a degree of “lookahead”: the actor can only learn to take good actions after the Q-function has learned to assign such actions high values.

Original: This means that by taking actions according tothe Q-function during exploration, we can “getout ahead” of the actor and explore more efﬁciently.
Revised: This means that by taking actions according to the Q-function during exploration, we can “get out ahead” of the actor and explore more efﬁciently.

Original: Can we apply a similar logic during training, and construct target Q-values withsome amount of CEM optimization over actions, rather than simply sampling actions from the actor?
Revised: Can we apply a similar logic during training, and construct target Q-values with some amount of CEM optimization over actions, rather than simply sampling actions from the actor?

Original: Based on this intuition, we hypothesize that, in much the same way that direct maximization overthe Q-function provided better exploration, improving over the actor’s action in the target value calculation will also lead to faster learning.
Revised: Based on this intuition, we hypothesize that, in much the same way that direct maximization over the Q-function provided better exploration, improving over the actor’s action in the target value calculation will also lead to faster learning.

Original: While previous ofﬂine RL techniques [24, 2] makesure to carefully calculate the target value to avoid a potential use of out of distribution actions,we expect this to be less critical for our method, since the positive-ﬁltering strategy in Section 4.2already ensures that the actor is only trained on successful actions, signiﬁcantly limiting the harmcaused by an inaccurate Q-function.
Revised: While previous ofﬂine RL techniques [24, 2] make sure to carefully calculate the target value to avoid a potential use of out of distribution actions, we expect this to be less critical for our method, since the positive-ﬁltering strategy in Section 4. already ensures that the actor is only trained on successful actions, signiﬁcantly limiting the harm caused by an inaccurate Q-function.

Original: To evaluate this idea, we test several ways of selecting actionsin the Bellman backup.
Revised: To evaluate this idea, we test several ways of selecting actions in the Bellman backup.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: In addition to the standard AWAC target value, which is taken as an expectation under the actor, we test: (a) maxQ

Original: In addition to the standard AWAC target value,which is taken as an expectation under the actor, we test: (a) maxQ uses the same targetvalue calculation as QT-Opt, optimizing the action in the target value calculation with CEM(“maxQ”); (b) maxQ + ActorMean combines the actor with CEM by using the actorpredicted action as the initialmean for CEM;(c)
Revised: uses the same target value calculation as QT-Opt, optimizing the action in the target value calculation with CEM (“maxQ”); (b) maxQ + ActorMean combines the actor with CEM by using the actorpredicted action as the initial

Revised: mean for CEM; (

Revised: c

Original:  maxQ
Revised: ) maxQ

Original: + ActorCandidate combines the actor with CEM by using the actor-predicted action as an additional candidate in each round of
Revised: + ActorCandidate combines the actor with CEM by using the actor-predicted action as an additional candidate in each round of

Original: CEM.
Revised: CEM.

Original: These target value calculations, particularly (a) , bring the method signiﬁcantly closerto QT-Opt.
Revised: These target value calculations, particularly (a) , bring the method signiﬁcantly closer to QT-Opt.

Original: In fact, (a) can be equivalently seenas a QT-Opt Q-function learning method, augmented with an additional positive-ﬁltered actor that is used for exploration.
Revised: In fact, (a) can be equivalently seen as a QT-Opt Q-function learning method, augmented with an additional positive-ﬁltered actor that is used for exploration.

Original: However, as wesee in Fig.
Revised: However, as we see in Fig.

Original: 5, the performance of this approachis signiﬁcantly better than both the AWAC-style
Revised: 5, the performance of this approach is signiﬁcantly better than both the AWAC-style

Original: Bellman backup and the standard QT-Opt method, neither of which learn the task successfully.
Revised: Bellman backup and the standard QT-Opt method, neither of which learn the task successfully.

Original: The“maxQ” strategy leads to more than three-fold increase in performance after 200k samples over theprevious best method.
Revised: The “maxQ” strategy leads to more than three-fold increase in performance after 200k samples over the previous best method.

Original: In addition, we notice that (b) is worse than (a) .
Revised: In addition, we notice that (b) is worse than (a) .

Original: Our hypothesis for why”maxQ + ActorMean” is worse is the following: in actor-critic methods, the actor always lags behind the critic.
Revised: Our hypothesis for why ”maxQ + ActorMean” is worse is the following: in actor-critic methods, the actor always lags behind the critic.

Original: Therefore, when the actor is not good enough, it may shift the initial distribution andlead the CEM in the wrong direction.
Revised: Therefore, when the actor is not good enough, it may shift the initial distribution and lead the CEM in the wrong direction.

Original: Fig.
Revised: Fig.

Original: 5 demonstrates the comparison of these techniques.
Revised: 5 demonstrates the comparison of these techniques.

Original: AWAC P ELRS + maxQ is signiﬁcantlybetter than AWAC P ELRS.
Revised: AWAC P ELRS + maxQ is signiﬁcantly better than AWAC P ELRS.

Original: Two variants: AWAC P ELRS + maxQ + ActorMean (using actorpredicted action as the initial CEM mean) and AWAC P ELRS + maxQ + ActorCandidate (usingactor-predicted action as an additional CEM candidate to compare with the best action selected by
Revised: Two variants: AWAC P ELRS + maxQ + ActorMean (using actorpredicted action as the initial CEM mean) and AWAC P ELRS + maxQ + ActorCandidate (using actor-predicted action as an additional CEM candidate to compare with the best action selected by

Original: CEM) also improve over AWAC P ELRS.
Revised: CEM) also improve over AWAC P ELRS.

Original: As we notice that AWAC P ELRS + maxQ + ActorCandidate achieves the best performance, we turn AWAC P ELRS + maxQ
Revised: As we notice that AWAC P ELRS + maxQ + ActorCandidate achieves the best performance, we turn AWAC P ELRS + maxQ

Original: + ActorCandidate into ourﬁnal AW-Opt method, which is shown in Algorithm 1.
Revised: + ActorCandidate into our ﬁnal AW-Opt method, which is shown in Algorithm 1.

Original: 4.5 Ablations
Revised: 4.5 Ablations

Original: After the step-by-step design of the AW-Opt algorithm, we run an additional ablation studyby removing one feature at a time to furtheridentify the importance of each of the designchoices we have made.
Revised: After the step-by-step design of the AW-Opt algorithm, we run an additional ablation study by removing one feature at a time to further identify the importance of each of the design choices we have made.

Original: We compare the following four cases: (a) AW-Opt, (b) AW-Optwithout positive-ﬁltering, (c) AW-Opt without
Revised: We compare the following four cases: (a) AW-Opt, (b) AW-Opt without positive-ﬁltering, (c) AW-Opt without

Original: ActorCandidate, (d) AW-Opt without hybridactor-critic exploration.
Revised: ActorCandidate, (d) AW-Opt without hybrid actor-critic exploration.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 6, each of the introduced components plays a signiﬁcant role in providing aboost in performance.
Revised: 6, each of the introduced components plays a signiﬁcant role in providing a boost in performance.

Original: Among the proposed AW-Opt design decisions, positive ﬁltering is particularly crucial for achieving good results in the online phase.
Revised: Among the proposed AW-Opt design decisions, positive ﬁltering is particularly crucial for achieving good results in the online phase.

Original: Without applying positive ﬁltering, theperformance of AW-Opt collapses to 0% once on-policy training starts, conﬁrming our hypothesisthat positive-ﬁltering of the actor robustiﬁes the actor updates against inaccurate Q-functions in theinitial phases of training.
Revised: Without applying positive ﬁltering, the performance of AW-Opt collapses to 0% once on-policy training starts, conﬁrming our hypothesis that positive-ﬁltering of the actor robustiﬁes the actor updates against inaccurate Q-functions in the initial phases of training.

Original: However, each of the other components is also critical, with the bestperforming ablation (the one that uses the standard AWAC target value calculation) still being aboutthree times worse than full AW-Opt.
Revised: However, each of the other components is also critical, with the bestperforming ablation (the one that uses the standard AWAC target value calculation) still being about three times worse than full AW-Opt.

Original:  Experiments
Revised:  Experiments

Original: In our experimental evaluation, we study the following questions: (1) Can AW-Opt learn from bothexpert demonstrations and suboptimal off-policy data?
Revised: In our experimental evaluation, we study the following questions: (1) Can AW-Opt learn from both expert demonstrations and suboptimal off-policy data?

Original: (2) Is AW-Opt a viable RL ﬁne-tuning algorithm that continues to improve after pre-training on ofﬂine data in diverse real-world and simulatedsettings?
Revised: (2) Is AW-Opt a viable RL ﬁne-tuning algorithm that continues to improve after pre-training on ofﬂine data in diverse real-world and simulated settings?

Original: (3) Is AW-Opt competitive on metrics important for robot learning, such as real-worldperformance and sample efﬁciency?
Revised: (3) Is AW-Opt competitive on metrics important for robot learning, such as real-world performance and sample efﬁciency?

Original: (4) Are the previously mentioned design choices critical for thesuccess of the method on a wide range of tasks?
Revised: (4) Are the previously mentioned design choices critical for the success of the method on a wide range of tasks?

Original: 5.1 Experimental Setup
Revised: 5.1 Experimental Setup

Original: We evaluate our method on simulated and real robotic manipulation tasks (Fig. 1 and Fig. 7).
Revised: We evaluate our method on simulated and real robotic manipulation tasks (Fig. 1 and Fig. 7).

Original: We use two real robotic systems withparallel-jaw grippers and over-the-shoulder cameras: Kuka IIWArobots, and a proprietary 7-DoF arm that we call “Pica” 1 .
Revised: We use two real robotic systems with parallel-jaw grippers and over-the-shoulder cameras: Kuka IIWA robots, and a proprietary 7-DoF arm that we call “Pica” 1 .

Original: We usetwo simulated tasks and three real tasks in our experiments:
Revised: We use two simulated tasks and three real tasks in our experiments:

Original: Task 1: (Sim) Indiscriminate Grasping with Kuka (Fig. 1 (a)).
Revised: Task 1: (Sim) Indiscriminate Grasping with Kuka (Fig. 1 (a)).

Original: Six random procedurally-generated objects are placed in a bin.
Revised: Six random procedurally-generated objects are placed in a bin.

Original: Ifthe robot lifts any of the objects, the episode is successful.
Revised: If the robot lifts any of the objects, the episode is successful.

Original: We utilize around 1000 off-policy simulation trials for this task.
Revised: We utilize around 1000 off-policy simulation trials for this task.

Original: Task 2: (Sim) Green Bowl Grasping with Pica (Fig. 1 (b)).
Revised: Task 2: (Sim) Green Bowl Grasping with Pica (Fig. 1 (b)).

Original: Agreen bowl is randomly placed on a table among other objects.
Revised: A green bowl is randomly placed on a table among other objects.

Original: Ifthe robot lifts the bowl, the episode is considered a success.
Revised: If the robot lifts the bowl, the episode is considered a success.

Original: Weutilize 120 teleoperated simulated demonstrations for this task.
Revised: We utilize 120 teleoperated simulated demonstrations for this task.

Original: Task 3: (Real)
Revised: Task 3: (Real)

Original: Indiscriminate Grasping with Kuka (Fig.
Revised: Indiscriminate Grasping with Kuka (Fig.

Original: 1 (c)).
Revised: 1 (c)).

Original: This is the real-world version of Task 1.
Revised: This is the real-world version of Task 1.

Original: We follow the setup proposed in [3], and utilize off-policy data for this task for pre-training.
Revised: We follow the setup proposed in [3], and utilize off-policy data for this task for pre-training.

Original: Task 4: (Real)
Revised: Task 4: (Real)

Original: Grasping from the Middle Bin with Pica (Fig. 1(d)).
Revised: Grasping from the Middle Bin with Pica (Fig.(d)).

Original: Several types of trash are placed randomly in three bins.
Revised: Several types of trash are placed randomly in three bins.

Original: Theepisode is successful if the robot lifts any object from the middlebin.
Revised: The episode is successful if the robot lifts any object from the middle bin.

Original: We use 1000 demonstrations for this task.
Revised: We use 1000 demonstrations for this task.

Original: Task 5: (Real) Grasping Compostables from the Middle Bin with
Revised: Task 5: (Real) Grasping Compostables from the Middle Bin with

Original: Pica (Fig. 1 (d)).
Revised: Pica (Fig. 1 (d)).

Original: This is a more difﬁcult version of Task 4.
Revised: This is a more difﬁcult version of Task 4.

Original: Whilesetup is equivalent, an episode is successful if a compostable object was lifted.
Revised: While setup is equivalent, an episode is successful if a compostable object was lifted.

Original: We utilize 300demonstrations for this task.
Revised: We utilizedemonstrations for this task.

Original: In the pretraining phase, we utilize the prior data described above and train the network (the samenetwork as QT-Opt [3]) for 5,000 to 10,000 gradient steps.
Revised: In the pretraining phase, we utilize the prior data described above and train the network (the same network as QT-Opt [3]) for 5,000 to 10,000 gradient steps.

Original: Each algorithm then collects 20,000to 200,000 episodes of on-policy data during further training.
Revised: Each algorithm then collects 20, to 200,000 episodes of on-policy data during further training.

Original: For Task 4 and Task 5, ﬁne-tuning isdone in simulation, using RL-CycleGAN [43] to produce realistic images.
Revised: For Task 4 and Task 5, ﬁne-tuning is done in simulation, using RL-CycleGAN [43] to produce realistic images.

Original: This allows us to compareﬁne-tuning performance between QT-Opt, AWAC, and AW-Opt on these tasks, using simulatedevaluations.
Revised: This allows us to compare ﬁne-tuning performance between QT-Opt, AWAC, and AW-Opt on these tasks, using simulated evaluations.

Original: Separately, we conduct real-world evaluations, evaluating the ofﬂine pretraining stageand sim-to-real transfer of ﬁne-tuned policies, since running each of the three algorithms online inthe real world would be too time-consuming.
Revised: Separately, we conduct real-world evaluations, evaluating the ofﬂine pretraining stage and sim-to-real transfer of ﬁne-tuned policies, since running each of the three algorithms online in the real world would be too time-consuming.

Original: This evaluation protocol provides evidence to supportour central claims: the simulated experiments conﬁrms that our method can effectively ﬁnetunefrom online data, while the real-world evaluation conﬁrms that our approach can handle ofﬂinepre-training on real data effectively.
Revised: This evaluation protocol provides evidence to support our central claims: the simulated experiments conﬁrms that our method can effectively ﬁnetune from online data, while the real-world evaluation conﬁrms that our approach can handle ofﬂine pre-training on real data effectively.

Original: 5.2 Learning from Demonstrations and Suboptimal Off-Policy Data
Revised: 5.2 Learning from Demonstrations and Suboptimal Off-Policy Data

Original: To address the question of whether AW-Opt can learn using different types of prior data, we preparedatasets from different sources: simulated off-policy data (Task 1), simulated demonstrations (Task2) and real demonstrations (Task 4 and Task 5).
Revised: To address the question of whether AW-Opt can learn using different types of prior data, we prepare datasets from different sources: simulated off-policy data (Task 1), simulated demonstrations (Task 2) and real demonstrations (Task 4 and Task 5).

Original: We compare the learning performance of QT-Opt,
Revised: We compare the learning performance of QT-Opt,

Original: AWAC, and AW-Opt using these 3 different data sources in Fig.
Revised: AWAC, and AW-Opt using these 3 different data sources in Fig.

Original: 8 (left panels of each subplot).
Revised: 8 (left panels of each subplot).

Original: Both AWAC and AW-Opt can learn from both demonstrations and off-policy data, while QT-Optcannot, since Q-learning requires both successful and failed trials to determine which actions arebetter than others.
Revised: Both AWAC and AW-Opt can learn from both demonstrations and off-policy data, while QT-Opt cannot, since Q-learning requires both successful and failed trials to determine which actions are better than others.

Original: Depending on the difﬁculty of the task, the success rates of the policies learnedby AWAC and AW-Opt vary from 20% to 60% with the exception of the difﬁcult Task 5, whichrequires on-policy ﬁnetuning.
Revised: Depending on the difﬁculty of the task, the success rates of the policies learned by AWAC and AW-Opt vary from 20% to 60% with the exception of the difﬁcult Task 5, which requires on-policy ﬁnetuning.

Original: 5.3 Online Fine-tuning
Revised: 5.3 Online Fine-tuning

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 8 (right panels of each subplot), during online ﬁne-tuning, QT-Opt performs significantly better than AWAC, while the latter struggles to effectively utilize online rollouts.
Revised: 8 (right panels of each subplot) during online ﬁne-tuning QT-Opt performs significantly better than AWAC, while the latter struggles to effectively utilize online rollouts.

Original: However,with the modiﬁcations in AW-Opt, our method is able to attain the best of both worlds, ﬁne-tuningeffectively during the online phase, while utilizing the ofﬂine data to bootstrap this ﬁne-tuning process.
Revised: However, with the modiﬁcations in AW-Opt, our method is able to attain the best of both worlds, ﬁne-tuning effectively during the online phase, while utilizing the ofﬂine data to bootstrap this ﬁne-tuning process.

Original: While AW-Opt is more data-efﬁcient than QT-Opt on Task 1 and Task 2, this difference isparticularly pronounced on Task 4 and Task 5, where the inability of QT-Opt to utilize the prior datasigniﬁcantly hampers its exploration.
Revised: While AW-Opt is more data-efﬁcient than QT-Opt on Task 1 and Task 2, this difference is particularly pronounced on Task 4 and Task 5, where the inability of QT-Opt to utilize the prior data signiﬁcantly hampers its exploration.

Original: 5. Real-World Robotic Manipulation Results
Revised: 5. Real-World Robotic Manipulation Results

Original: [Table]
Revised: [Table]

Original: We evaluate the performance of policies trainedwith QT-Opt, AWAC, and AW-Opt after ofﬂine pretraining for Task 3 and after ﬁnetuning for Task 5.
Revised: We evaluate the performance of policies trained with QT-Opt, AWAC, and AW-Opt after ofﬂine pretraining for Task 3 and after ﬁnetuning for Task 5.

Original: The ﬁne-tuning stage is performed in simulation, asdiscussed in Section 5.1.
Revised: The ﬁne-tuning stage is performed in simulation, as discussed in Section 5.1.

Original: The goal of these experiments is to validate that policies learned with AW-Opt can indeed handle ofﬂine demonstration dataand generalize to real-world settings.
Revised: The goal of these experiments is to validate that policies learned with AW-Opt can indeed handle ofﬂine demonstration data and generalize to real-world settings.

Original: The size and type of the datasets are described in Appendix.
Revised: The size and type of the datasets are described in Appendix.

Original: For Task 3, we ﬁrst train AWAC, QT-Opt, and AW-Opt with only positive data and then train withboth positive and negative data.
Revised: For Task 3, we ﬁrst train AWAC, QT-Opt, and AW-Opt with only positive data and then train with both positive and negative data.

Original: We evaluate each policy with 612 grasps, as shown in Table 1.
Revised: We evaluate each policy with 612 grasps, as shown in Table 1.

Original: When learning from the combined positive and negative data, AWAC performance drops to 0%.
Revised: When learning from the combined positive and negative data, AWAC performance drops to 0%.

Original: When QT-Opt learns from only positive data, it also fails to learn the task.
Revised: When QT-Opt learns from only positive data, it also fails to learn the task.

Original: Only AW-Opt is ableto learn from both positive-only and mixed positive-negative data.
Revised: Only AW-Opt is able to learn from both positive-only and mixed positive-negative data.

Original: For Task 5, we evaluate eachpolicy with 180 grasps, as shown in Table 1.
Revised: For Task 5, we evaluate each policy with 180 grasps, as shown in Table 1.

Original: Only AW-Opt learns this task successfully, achieving a48.89% success rate.
Revised: Only AW-Opt learns this task successfully, achieving a 48.89% success rate.

Original: These results support the conclusions drawn from our simulated experiments.
Revised: These results support the conclusions drawn from our simulated experiments.

Original:  Conclusion
Revised:  Conclusion

Original: We presented AW-Opt, a scalable robotic RL algorithm that incorporates ofﬂine data and performsonline ﬁnetuning.
Revised: We presented AW-Opt, a scalable robotic RL algorithm that incorporates ofﬂine data and performs online ﬁnetuning.

Original: We show that each design decision in AW-Opt leads to signiﬁcant gains in performance, and the ﬁnal method can learn complex image-based tasks starting with either demonstrations or suboptimal ofﬂine data, even in settings where prior methods, such as QT-Opt or AWAC,either fail to leverage prior data or fail to make progress during ﬁnetuning.
Revised: We show that each design decision in AW-Opt leads to signiﬁcant gains in performance, and the ﬁnal method can learn complex image-based tasks starting with either demonstrations or suboptimal ofﬂine data, even in settings where prior methods, such as QT-Opt or AWAC, either fail to leverage prior data or fail to make progress during ﬁnetuning.

Original: Our evaluation does haveseveral limitations.
Revised: Our evaluation does have several limitations.

Original: First, we only evaluate ofﬂine training in the real world, studying ﬁnetuningunder simulated settings where extensive comparisons are feasible.
Revised: First, we only evaluate ofﬂine training in the real world, studying ﬁnetuning under simulated settings where extensive comparisons are feasible.

Original: Our experiments are also allconcerned with sparse, binary reward tasks.
Revised: Our experiments are also all concerned with sparse, binary reward tasks.

Original: Such tasks are common in robotics, and AW-Opt couldbe extended to arbitrary rewards via a threshold for positive ﬁltering, a promising direction for futurework.
Revised: Such tasks are common in robotics, and AW-Opt could be extended to arbitrary rewards via a threshold for positive ﬁltering, a promising direction for future work.

Original: Nonetheless, our results suggest that AW-Opt can be a powerful tool for scaling up IL+RLmethods.
Revised: Nonetheless, our results suggest that AW-Opt can be a powerful tool for scaling up IL+RL methods.

Original: Aside from motivating each design decision, we also hope that the detailed evaluations ofeach component will aid the design of even more effective and scalable RL methods in the future.A.1 Experimental results with navigation task [Figure]
Revised: Aside from motivating each design decision, we also hope that the detailed evaluations of each component will aid the design of even more effective and scalable RL methods in the future.

Original: For an additional test of the proposed algorithm, we compared QTOpt, AWAC and AW-Opt on apoint-to-point LIDAR-based navigation task 9 following the navigation training conﬁguration in [44].

Original: This task is signiﬁcantly simpler thanthe image-based manipulation tasks,since the observation space is lowerdimensional (240 LIDAR points anda 2D goal, rather than 472472 image),and the action space is smaller (a navigation twist with 2 degrees of freedom). Therefore, all threealgorithms attain reasonable performance. However, as expected, AW-Opt and QT-Opt converge tosigniﬁcantly better ﬁnal performance: 90% success rate at reaching the goal versus 70% for AWAC,and AW-Opt attains much better performance after ofﬂine pretraining compared to QT-Opt (20% vs0% success rate), allowing it to learn the task up to the 90% success rate about three times fasterthan QT-Opt during online ﬁnetuning. AWAC requires a similar number of transitions to converge,but reaches a signiﬁcantly worse ﬁnal level of performance.
Revised: Acknowledgments

Original: In Section 4.3, we compared differentexploration strategies and concludedthat episode-level random switcheris the best strategy.

Original: In this section,we further compare the choice ofratio for actor vs. critic-based exploration.

Original: The default ratio is 80%/20%(critic/actor).

Original: Fig.

Original: 11 presents acomparison of different splits, on

Original: Task 1 and Task 3.

Original: Each run wasrepeated three times, with each ofthe following splits: 20%/80%,50%/50%, 60%/40%, 70%/30%,80%/20%, 90%/10%.

Original: For Task 1, wesee similar results for the last threesplits, while for Task 2, we see betterresults with the 30%/70% split.

Original: A.3 Experimentalresults with negative data

Original: [Figure]

Original: In Table 1, we see that including“negatives” in the ofﬂine pretraining data for QT-Opt signiﬁcantlyimproves QT-Opt’s performance on

Original: Task 3 and Task 5, where the datawas collected using a scripted policy.

Original: The negatives and positives are collected using the same scripted policy,with the positives treated as “successful demonstrations.

Original: ” We might wonder if an analogous trend would holdfor other tasks, where human demonstrations are used as positives.

Original: Inthese cases, there are no corresponding negatives, but we can supply additional negatives by using a random policy.

Original: To examine the effect of these synthetic negatives, we evaluate Task 4 witheither 10,000 or 100,000 additional random negatives.

Original: The results are shown in Fig 12.

Original: However,in this case, we see that QT-Opt is still not able to make any progress, remaining at 0% successrate throughout.

Original: This is not surprising.

Original: In the case of Task 3 and Task 5, the randomized scriptedpolicy that collects the data has broad coverage, which means that many different actions are indistribution.

Original: The “negatives” come from the same distribution as the positives (they are collectedby the same scripted policy).

Original: Such a condition is known to be favorable for ofﬂine reinforcementlearning [24], since when the data distribution is broad, fewer actions are out of distribution.

Original: Indeed,this is precisely the distribution used in the original QT-Opt work [3].

Original: However, in the case of Task4, the distribution of positives is very narrow.

Original: We cannot add negatives from the same distribution(since the distribution is over successful behaviors), which forces us to add negatives from a differentdistribution.

Original: While this does provide for broader coverage, it does not provide broader coverage inthe region surrounding the positive examples, and therefore does not relieve the distributional shiftchallenge faced by ofﬂine RL.

Original: A.4 Action space and loss function

Original: For Task 1 and Task 3, the robot is operated in a 7D action space, namely: x, y, z, vertical rotationθ , open gripper, close gripper, terminate episode.

Original: The ﬁrst 4 subactions are continuous while thelast 3 subactions are discrete.

Original: For Task 2, Task 4 and Task 5, the action space is slightly different.

Original: It is operated on a 8D action space, namely: x, y, z angle axis x, angle axis y, angle axis z, gripperclosedness, terminate episode.

Original: The ﬁrst 7 subactions are continuous while the last subaction is dis- crete.

Original: The actor loss function considers both discrete and continuous subactions and also applies aweight on each one of the subaction.

Original: The full equation can be formulated as the following.

Original: [Table]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: where K is the total number of subactions.

Original: w k is the weight to balance between each subaction. w d is the weight for discrete subactions. For Task1, the weights are summarized in Table. For Task 2, Task4, Task 5, the weights are summarized in Table.
Revised: We would like to give special thanks to Dao Tran, Conrad Villondo, Clayton Tan for collecting demonstration data as well as Julian Ibarz and Kanishka Rao for valuable discussions.

Original: A.5 Network architecture

Original: We apply exactly the same network architecture as QT-Opt[3] as thecritic network for all RL algorithms and in all experiments we conducted.

Original: Actor network also uses the same backbone as the critic network.

Original: The difference is that it does not have action input and insteadof outputting a single Q value, it outputs the action distribution (meanand variance of Gaussian distribution for continuous subactions anda one hot vector of the discrete actions).

Original: During the training, critic network and actor network do not shareweights between each other.

Original: They are trained separately.

Original: We plan to open source AW-Opt algorithm once the paper is published.

Original: A.6 Policy Action Selection

Original: [Table]

Original: [Table]

Original: A signiﬁcant limitation of QT-Opt and other methods thatdo not employ an actor in continuous action spaces is thataction-selection at evaluation time requires an optimization with respect to the critic.

Original: In this section, we comparethe run time of action selection for QT-Opt, AWAC, AWOpt on different robots.

Original: Both AWAC and AW-Opt use anactor network after training, while QT-Opt requires optimization over actions with CEM.

Original: As we can see in Table 4,this results in AW-Opt and AWAC selecting actions aboutthree times faster than QT-Opt.

Original: We measure the inference time on a Quadro P1000 GPU.

Original: [Table]

Original: A.7 Datasets

Original: [Table]

Original: In the following we summarize the dataset size and type for different tasks.

Original: [Table]