Original: AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scale
Revised: AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scale

Original: Abstract: Robotic skills can be learned via imitation learning (IL) using userprovided demonstrations, or via reinforcement learning (RL) using large amountsof autonomously collected experience.
Revised: Abstract: Robotic skills can be learned via imitation learning (IL) using userprovided demonstrations, or via reinforcement learning (RL) using large amountsof autonomously collected experience.

Original: Both methods have complementarystrengths and weaknesses: RL can reach a high level of performance, but requiresexploration, which can be very time consuming and unsafe; IL does not requireexploration, but only learns skills that are as good as the provided demonstrations.
Revised: Both methods have complementarystrengths and weaknesses: RL can reach a high level of performance, but requiresexploration, which can be very time consuming and unsafe; IL does not requireexploration, but only learns skills that are as good as the provided demonstrations.

Original: Can a single method combine the strengths of both approaches?
Revised: Can a single method combine the strengths of both approaches?

Original: A number ofprior methods have aimed to address this question, proposing a variety of techniques that integrate elements of IL and RL.
Revised: A number ofprior methods have aimed to address this question, proposing a variety of techniques that integrate elements of IL and RL.

Original: However, scaling up such methodsto complex robotic skills that integrate diverse ofﬂine data and generalize mean- ingfully to real-world scenarios still presents a major challenge.
Revised: However, scaling up such methodsto complex robotic skills that integrate diverse ofﬂine data and generalize mean- ingfully to real-world scenarios still presents a major challenge.

Original: In this paper, our aim is to test the scalability of prior IL + RL algorithms and devise a system basedon detailed empirical experimentation that combines existing components in the most effective and scalable way.
Revised: In this paper, our aim is to test the scalability of prior IL + RL algorithms and devise a system basedon detailed empirical experimentation that combines existing components in the most effective and scalable way.

Original: To that end, we present a series of experimentsaimed at understanding the implications of each design decision, so as to develop acombined approach that can utilize demonstrations and heterogeneous prior datato attain the best performance on a range of real-world and realistic simulatedrobotic problems.
Revised: To that end, we present a series of experimentsaimed at understanding the implications of each design decision, so as to develop acombined approach that can utilize demonstrations and heterogeneous prior datato attain the best performance on a range of real-world and realistic simulatedrobotic problems.

Original: Our complete method, which we call AW-Opt, combines elements of advantage-weighted regression [1, 2] and QT-Opt [3], providing a uniﬁedapproach for integrating demonstrations and ofﬂine data for robotic manipulation.
Revised: Our complete method, which we call AW-Opt, combines elements of advantage-weighted regression [1, 2] and QT-Opt [3], providing a uniﬁedapproach for integrating demonstrations and ofﬂine data for robotic manipulation.

Original:  Introduction
Revised:  Introduction

Original: Learning methods for robotic control are conventionally divided into two groups: methods based onautonomous trial-and-error (reinforcement learning) and methods based on imitating user-provideddemonstrations (imitation learning).
Revised: Learning methods for robotic control are conventionally divided into two groups: methods based onautonomous trial-and-error (reinforcement learning) and methods based on imitating user-provideddemonstrations (imitation learning).

Original: These two approaches have complementary strengths andweaknesses.
Revised: These two approaches have complementary strengths andweaknesses.

Original: Reinforcement Learning (RL) enables robots to improve autonomously, but introducesigniﬁcant challenges with exploration and stable learning.
Revised: Reinforcement Learning (RL) enables robots to improve autonomously, but introducesigniﬁcant challenges with exploration and stable learning.

Original: Imitation learning (IL) methods providefor more stable learning from expert demonstrations but cannot improve from failures.
Revised: Imitation learning (IL) methods providefor more stable learning from expert demonstrations but cannot improve from failures.

Original: This couldpotentially make covering the full distribution of data difﬁcult.
Revised: This couldpotentially make covering the full distribution of data difﬁcult.

Original: Prior works have sought to combine elements of IL and RL into a single algorithm (IL+RL), bypre-training RL policies with demonstrations [4], incorporating demonstration data into off-policy
Revised: Prior works have sought to combine elements of IL and RL into a single algorithm (IL+RL), bypre-training RL policies with demonstrations [4], incorporating demonstration data into off-policy

Original: RL algorithms [5], and combining IL and RL loss functions into a single algorithm [6].
Revised: RL algorithms [5], and combining IL and RL loss functions into a single algorithm [6].

Original: Whilesuch algorithms have been based on both policy gradients and value-based off-policy methods, thelatter are particularly relevant in robotics settings due to their ability to leverage both demonstrationsand the robot’s own prior experience.
Revised: Whilesuch algorithms have been based on both policy gradients and value-based off-policy methods, thelatter are particularly relevant in robotics settings due to their ability to leverage both demonstrationsand the robot’s own prior experience.

Original: A number of such methods have shown promising results insimulation and in the real world [7], but they often require extensive parameter tuning and, as weshow in our experiments, are difﬁcult to deploy at scale to solve multiple tasks in diverse scenarios.
Revised: A number of such methods have shown promising results insimulation and in the real world [7], but they often require extensive parameter tuning and, as weshow in our experiments, are difﬁcult to deploy at scale to solve multiple tasks in diverse scenarios.

Original: In this paper, our goal is to develop, through extensive experimentation, a complete and scalablesystem for integrating IL and RL that enables learning robotic control policies from both demonstration data and suboptimal experience.
Revised: In this paper, our goal is to develop, through extensive experimentation, a complete and scalablesystem for integrating IL and RL that enables learning robotic control policies from both demonstration data and suboptimal experience.

Original: Our aim is not to propose new principles for algorithmsthat combine IL and RL, but rather to determine how to best combine the components of existingmethods to enable effective large-scale robotic learning.
Revised: Our aim is not to propose new principles for algorithmsthat combine IL and RL, but rather to determine how to best combine the components of existingmethods to enable effective large-scale robotic learning.

Original: We begin our investigation with two existing methods: AWAC [2], which provides an integrated framework for combining IL and RL, and
Revised: We begin our investigation with two existing methods: AWAC [2], which provides an integrated framework for combining IL and RL, and

Original: QT-Opt [3], which provides a scalable system for robotic reinforcement learning.
Revised: QT-Opt [3], which provides a scalable system for robotic reinforcement learning.

Original: We observe thatneither method by itself provides a complete solution to large-scale RL with demonstrations.
Revised: We observe thatneither method by itself provides a complete solution to large-scale RL with demonstrations.

Original: Wethen systematically study the decisions in these methods, identify their shortcomings, and develop ahybrid approach that we call AW-Opt.
Revised: Wethen systematically study the decisions in these methods, identify their shortcomings, and develop ahybrid approach that we call AW-Opt.

Original: The individual design decisions that comprise AW-Opt mayappear minor in isolation, but we show that combining these decisions signiﬁcantly improves on theprior approaches, and scales effectively to learning high-capacity generalizable policies in multiplereal and simulated environments shown in Fig.
Revised: The individual design decisions that comprise AW-Opt mayappear minor in isolation, but we show that combining these decisions signiﬁcantly improves on theprior approaches, and scales effectively to learning high-capacity generalizable policies in multiplereal and simulated environments shown in Fig.

Original: 1 (a-d).
Revised: 1 (a-d).

Original: The main contributions of this work consist of a detailed analysis of the design decisions that inﬂuence performance and scalability of IL+RL methods, leading to the development of AW-Opt, ahybrid algorithm that combines insights from multiple prior approaches to achieve scalable and effective robotic learning from demonstrations, suboptimal ofﬂine data, and online exploration.
Revised: The main contributions of this work consist of a detailed analysis of the design decisions that inﬂuence performance and scalability of IL+RL methods, leading to the development of AW-Opt, ahybrid algorithm that combines insights from multiple prior approaches to achieve scalable and effective robotic learning from demonstrations, suboptimal ofﬂine data, and online exploration.

Original: Notonly can AW-Opt be successfully initialized from demonstrations (initial success in Fig. 1(f)), butalso it responds well to RL ﬁne-tuning (ﬁnal success in Fig. 1(f)) while being sample-efﬁcient.
Revised: Notonly can AW-Opt be successfully initialized from demonstrations (initial success in Fig. 1(f)), butalso it responds well to RL ﬁne-tuning (ﬁnal success in Fig. 1(f)) while being sample-efﬁcient.

Original: In addition to the design of AW-Opt, we hope that the analysis, ablation experiments, and detailedevaluation of individual design decisions that we present in this work would help to guide the development of effective and scalable robotic learning algorithms.
Revised: In addition to the design of AW-Opt, we hope that the analysis, ablation experiments, and detailedevaluation of individual design decisions that we present in this work would help to guide the development of effective and scalable robotic learning algorithms.

Original: Our experimental results evaluateour method, as well as ablations, across a range of simulated and real-world domains, including twodifferent real-world platforms and large datasets with hundreds of thousands of training episodes.
Revised: Our experimental results evaluateour method, as well as ablations, across a range of simulated and real-world domains, including twodifferent real-world platforms and large datasets with hundreds of thousands of training episodes.

Original:  Related Work
Revised:  Related Work

Original: Prior works have proposed a number of methods for learning robotic skills via imitation of experts [8, 9, 10, 11, 12, 13], and via reinforcement learning [14, 15, 16, 17, 18].
Revised: Prior works have proposed a number of methods for learning robotic skills via imitation of experts [8, 9, 10, 11, 12, 13], and via reinforcement learning [14, 15, 16, 17, 18].

Original: In the latter category,off-policy RL methods are especially appealing for large-scale robot learning, since they allow fordata reuse with large, diverse datasets [3, 19, 20].
Revised: In the latter category,off-policy RL methods are especially appealing for large-scale robot learning, since they allow fordata reuse with large, diverse datasets [3, 19, 20].

Original: Initially, methods that combined imitation learning (IL) and reinforcement learning (RL) have tended to treat the two data sources differently, using
Revised: Initially, methods that combined imitation learning (IL) and reinforcement learning (RL) have tended to treat the two data sources differently, using

Original: IL demos only at initialization [4, 5, 21], or using different losses for IL demos vs. RL episodes [6].
Revised: IL demos only at initialization [4, 5, 21], or using different losses for IL demos vs. RL episodes [6].

Original: Other methods have used a different approach: combining both expert demos and suboptimal priordata into a single dataset used for off-policy value-based learning [7].
Revised: Other methods have used a different approach: combining both expert demos and suboptimal priordata into a single dataset used for off-policy value-based learning [7].

Original: Ofﬂine RL methods, such as
Revised: Ofﬂine RL methods, such as

Original: CQL [22], QT-Opt [3], and AWR [1] often use this approach, along with constrained optimizationand regularization to encourage staying close to the data distribution [23, 24, 25, 26, 27].
Revised: CQL [22], QT-Opt [3], and AWR [1] often use this approach, along with constrained optimizationand regularization to encourage staying close to the data distribution [23, 24, 25, 26, 27].

Original: Our methodbuilds on these approaches, due to their effectiveness, simplicity, and easier data management.
Revised: Our methodbuilds on these approaches, due to their effectiveness, simplicity, and easier data management.

Original: Regardless of whether the data consists of optimal demonstrations, suboptimal human-provided demos,or very suboptimal robot experience, all data can be treated the same way.
Revised: Regardless of whether the data consists of optimal demonstrations, suboptimal human-provided demos,or very suboptimal robot experience, all data can be treated the same way.

Original: Our goal is to scale up IL+RL methods to large datasets with many objects and varied environments.
Revised: Our goal is to scale up IL+RL methods to large datasets with many objects and varied environments.

Original: Standard RL methods have proven difﬁcult to scale up to the same level as modern methods incomputer vision [28, 29] and natural language processing [30, 31, 32], where datasets are diverse andopen-world, and models have millions or billions of parameters.
Revised: Standard RL methods have proven difﬁcult to scale up to the same level as modern methods incomputer vision [28, 29] and natural language processing [30, 31, 32], where datasets are diverse andopen-world, and models have millions or billions of parameters.

Original: Taking RL to such settings requireslarge, distributed RL systems, which often require careful design decisions in their underlying RLmethods.
Revised: Taking RL to such settings requireslarge, distributed RL systems, which often require careful design decisions in their underlying RLmethods.

Original: While standard methods such as actor-critic have been scaled up in this way [33, 34,35, 36], we show in this work that scaling IL+RL algorithms requires a certain amount of care,especially in robotic settings where collecting real-world experience is costly.
Revised: While standard methods such as actor-critic have been scaled up in this way [33, 34,35, 36], we show in this work that scaling IL+RL algorithms requires a certain amount of care,especially in robotic settings where collecting real-world experience is costly.

Original: Na¨ıvely incorporatingdemonstrations into an RL system (QT-Opt in our experiments) does not by itself yield good results.
Revised: Na¨ıvely incorporatingdemonstrations into an RL system (QT-Opt in our experiments) does not by itself yield good results.

Original: Similarly, utilizing prior IL+RL methods (such as AWAC [2] in our experiments) scales poorly.
Revised: Similarly, utilizing prior IL+RL methods (such as AWAC [2] in our experiments) scales poorly.

Original: Weprovide a detailed empirical evaluation of various design decisions underlying IL+RL methods todevelop a hybrid approach, which we call AW-Opt, that combines the scalability of QT-Opt with theability to seamlessly incorporate suboptimal experience and bootstrap from demonstrations.
Revised: Weprovide a detailed empirical evaluation of various design decisions underlying IL+RL methods todevelop a hybrid approach, which we call AW-Opt, that combines the scalability of QT-Opt with theability to seamlessly incorporate suboptimal experience and bootstrap from demonstrations.

Original:  Preliminaries
Revised:  Preliminaries

Original: Let M =
Revised: Let M =

Original: ( S , A , P, R, p 0 , γ, T ) deﬁne a Markov decision process (MDP), where S and A are stateand action spaces, P : S ×
Revised: ( S , A , P, R, p 0 , γ, T ) deﬁne a Markov decision process (MDP), where S and A are stateand action spaces, P : S ×

Original: A × S → R + is a state-transition probability function, R : S ×
Revised: A × S → R + is a state-transition probability function, R : S ×

Original: A → Ris a reward function, p 0 : S → R + is an initial state distribution, γ is a discount factor, and T isthe task horizon.
Revised: A → Ris a reward function, p 0 : S → R + is an initial state distribution, γ is a discount factor, and T isthe task horizon.

Original: We use τ = ( s 0 , a 0 , . . .
Revised: We use τ = ( s 0 , a 0 , . . .

Original: , s T , a T ) to denote a trajectory of states and actions and
Revised: , s T , a T ) to denote a trajectory of states and actions and

Original: R ( τ ) =
Revised: R ( τ ) =

Original: (cid:80) Tt =

Revised: (cid:80) Tt =

Original: γ t R ( s t , a t ) to denote the trajectory reward.
Revised: γ t R ( s t , a t ) to denote the trajectory reward.

Original: Reinforcement learning (RL) methodsﬁnd a policy π ( a | s ) that maximizes the expected discounted reward over trajectories induced by thepolicy: E π [ R ( τ )] where s 0 ∼ p 0 , s t +1 ∼ P ( s t +1 | s t , a t ) and a t ∼ π ( a t | s t ) .
Revised: Reinforcement learning (RL) methodsﬁnd a policy π ( a | s ) that maximizes the expected discounted reward over trajectories induced by thepolicy: E π [ R ( τ )] where s 0 ∼ p 0 , s t +1 ∼ P ( s t +1 | s t , a t ) and a t ∼ π ( a t | s t ) .

Original: In imitation learning, demonstrations are provided to facilitate learning an optimal policy, whichcan be produced through teleoperation [37, 38] or kinesthetic teaching [39, 12].
Revised: In imitation learning, demonstrations are provided to facilitate learning an optimal policy, whichcan be produced through teleoperation [37, 38] or kinesthetic teaching [39, 12].

Original: Given a data set ofdemonstration state-action tuples ( s, a ) ∼ D demo , standard IL methods aim to directly approximatethe expert’s π ∗ ( a | s ) via supervised learning.
Revised: Given a data set ofdemonstration state-action tuples ( s, a ) ∼ D demo , standard IL methods aim to directly approximatethe expert’s π ∗ ( a | s ) via supervised learning.

Original: Another way of learning from prior data has been explored in the ﬁeld of ofﬂine RL [40], where RL methods are used to learn from datasets of previouslycollected data.
Revised: Another way of learning from prior data has been explored in the ﬁeld of ofﬂine RL [40], where RL methods are used to learn from datasets of previouslycollected data.

Original: This data can include demonstrations, as well as other suboptimal trajectories.
Revised: This data can include demonstrations, as well as other suboptimal trajectories.

Original: Suchofﬂine RL methods are particularly appealing for hybrid IL+RL methods, since they do not requireany explicit distinction between demonstration data and suboptimal experience.
Revised: Suchofﬂine RL methods are particularly appealing for hybrid IL+RL methods, since they do not requireany explicit distinction between demonstration data and suboptimal experience.

Original: In this paper, we build on these approaches to develop a scalable algorithm that combines IL and
Revised: In this paper, we build on these approaches to develop a scalable algorithm that combines IL and

Original: RL.
Revised: RL.

Original: Our goal is to smoothly transition between two sources of supervision: the demonstrated behaviors, provided as prior data, and RL exploration, collected autonomously.
Revised: Our goal is to smoothly transition between two sources of supervision: the demonstrated behaviors, provided as prior data, and RL exploration, collected autonomously.

Original: Like many ofﬂine RLalgorithms, our approach is based on Q-learning, which extracts the policy from a Q-function that istrained to satisfy the Bellman equation: Q π ( s t , a t ) =
Revised: Like many ofﬂine RLalgorithms, our approach is based on Q-learning, which extracts the policy from a Q-function that istrained to satisfy the Bellman equation: Q π ( s t , a t ) =

Original: R ( s t , a t ) + γ max a [ Q π ( s t +1 , a )] .
Revised: R ( s t , a t ) + γ max a [ Q π ( s t +1 , a )] .

Original: To develop an effective IL-RL algorithm, we start by analyzing various design decisions in two recently proposed ofﬂine RL methods: QT-Opt [3] and Advantage-Weighted Regression (AWR)
Revised: To develop an effective IL-RL algorithm, we start by analyzing various design decisions in two recently proposed ofﬂine RL methods: QT-Opt [3] and Advantage-Weighted Regression (AWR)

Original: [1],with its recent value-based instantiation AWAC [2].
Revised: [1],with its recent value-based instantiation AWAC [2].

Original: QT-Opt is a distributed Q-learning frameworkthat enables learning Q-functions in continuous action spaces by maximizing the Q-function usingthe cross-entropy method (CEM)
Revised: QT-Opt is a distributed Q-learning frameworkthat enables learning Q-functions in continuous action spaces by maximizing the Q-function usingthe cross-entropy method (CEM)

Original: [41], without an explicit actor model.
Revised: [41], without an explicit actor model.

Original: However, initializing a Qfunction from demonstration data is particularly challenging as it often results in over-optimistic
Revised: However, initializing a Qfunction from demonstration data is particularly challenging as it often results in over-optimistic

Original: Q-values on unseen state-action pairs [42, 24, 22].
Revised: Q-values on unseen state-action pairs [42, 24, 22].

Original: It has been shown that by introducing an explicitactor, one can mitigate this issue by limiting the actor to select actions that are within the (conditional) distribution of actions seen in the data [24, 1].
Revised: It has been shown that by introducing an explicitactor, one can mitigate this issue by limiting the actor to select actions that are within the (conditional) distribution of actions seen in the data [24, 1].

Original: This property is also leveraged by AWAC [2],where the actor is initialized from demonstration data, and then further ﬁne-tuned with RL rewards.
Revised: This property is also leveraged by AWAC [2],where the actor is initialized from demonstration data, and then further ﬁne-tuned with RL rewards.

Original: The AWAC algorithm uses the following update to the actor:
Revised: The AWAC algorithm uses the following update to the actor:

Original: [Equation]
Revised: [Equation]

Original: where A ( s, a ) =
Revised: where A ( s, a ) =

Original: Q ( s, a ) − E π ( a | s )
Revised: Q ( s, a ) − E π ( a | s )

Original: [ Q ( s, a )] is the advantage function, Z ( s ) is the normalizingpartition function, λ is a Lagrange multiplier, π θ is the learned policy and π β is the behavior policy,which represents the (unknown) distribution that produces the dataset.
Revised: [ Q ( s, a )] is the advantage function, Z ( s ) is the normalizingpartition function, λ is a Lagrange multiplier, π θ is the learned policy and π β is the behavior policy,which represents the (unknown) distribution that produces the dataset.

Original: This corresponds to weightedmaximum likelihood training, using samples from π β ( a | s ) (obtained from the dataset).
Revised: This corresponds to weightedmaximum likelihood training, using samples from π β ( a | s ) (obtained from the dataset).

Original: In this paper, we analyze various design decisions in QT-Opt and AWAC, point out their shortcomings in the IL+RL setting, and use these insights to construct an effective hybrid IL+RL method thatwe call AW-Opt.
Revised: In this paper, we analyze various design decisions in QT-Opt and AWAC, point out their shortcomings in the IL+RL setting, and use these insights to construct an effective hybrid IL+RL method thatwe call AW-Opt.

Original:  Learning via Imitation and Reinforcement with AW-Opt
Revised:  Learning via Imitation and Reinforcement with AW-Opt

Original: In order to derive AW-Opt, we ﬁrst study the performance of QT-Opt [3] and AWAC [2] in the
Revised: In order to derive AW-Opt, we ﬁrst study the performance of QT-Opt [3] and AWAC [2] in the

Original: IL+RL setting, using a complex robotic manipulation task with high-resolution image observationsas our working example.
Revised: IL+RL setting, using a complex robotic manipulation task with high-resolution image observationsas our working example.

Original: This setting requires algorithms that are more scalable than those typicallyemployed in standard RL benchmarks.
Revised: This setting requires algorithms that are more scalable than those typicallyemployed in standard RL benchmarks.

Original: We will show that neither QT-Opt nor AWAC effectivelyhandle this setting.
Revised: We will show that neither QT-Opt nor AWAC effectivelyhandle this setting.

Original: We then progressively construct a new algorithm with elements of both approaches, which we call AW-Opt, analyzing each design decision in turn.
Revised: We then progressively construct a new algorithm with elements of both approaches, which we call AW-Opt, analyzing each design decision in turn.

Original: Although each individualdesign decision in AW-Opt may appear relatively simple, we will see that the speciﬁc combinationof these design decisions outperforms both QT-Opt and AWAC by a very large margin, resulting insuccessful learning for tasks where the two prior methods are unable to make meaningful progress.
Revised: Although each individualdesign decision in AW-Opt may appear relatively simple, we will see that the speciﬁc combinationof these design decisions outperforms both QT-Opt and AWAC by a very large margin, resulting insuccessful learning for tasks where the two prior methods are unable to make meaningful progress.

Original: 4.1 Example Task and Baseline Performance
Revised: 4.1 Example Task and Baseline Performance

Original: [Figure] As a working example, we use a vision-basedtrash sorting task, where the robot must pickup a speciﬁc type of trash object (e.g., a compostable objects) out of cluttered bins containing diverse object types, including recyclables,landﬁll objects, etc., with only sparse rewardsindicating success once the robot has lifted thecorrect object.
Revised: As a working example, we use a vision-basedtrash sorting task, where the robot must pickup a speciﬁc type of trash object (e.g., a compostable objects) out of cluttered bins containing diverse object types, including recyclables,landﬁll objects, etc., with only sparse rewardsindicating success once the robot has lifted thecorrect object.

Original: We describe this task in detailin Section 5.
Revised: We describe this task in detailin Section 5.

Original: We will use this environment to illustrate the importance of each component of ourmethod, as we introduce them in this section.
Revised: We will use this environment to illustrate the importance of each component of ourmethod, as we introduce them in this section.

Original: Then, in Section 5, we will evaluate our complete AWOpt algorithm on a wider range of robotic environments, including two real-world robotic systemsand several high-ﬁdelity simulation environments. 
Revised: Then, in Section 5, we will evaluate our complete AW-Opt algorithm on a wider range of robotic environments, including two real-world robotic systems and several high-ﬁdelity simulation environments. Note that unless otherwise speciﬁed, all theplots in this paper is conducted with 3 experiments each.

Revised: Each data point is an averaged success rateevaluated with 700 episodes.

Revised: The shaded region in the plot depicts the 90% conﬁdence interval withthe line being the mean.

Original: We begin our investigation with the two prior methods: QT-Opt [3], which can scale well to highresolution images and diverse tasks, and AWAC [2], which is designed to integrate demonstrationsand then ﬁne-tune online.
Revised: We begin our investigation with the two prior methods: QT-Opt [3], which can scale well to highresolution images and diverse tasks, and AWAC [2], which is designed to integrate demonstrationsand then ﬁne-tune online.

Original: Both algorithms are provided with 300 demonstrations for ofﬂine pretraining, and then switch to on-policy collection after 10k gradient steps to collect and train with200k additional on-policy episodes.
Revised: Both algorithms are provided with 300 demonstrations for ofﬂine pretraining, and then switch to on-policy collection after 10k gradient steps to collect and train with200k additional on-policy episodes.

Original: In Fig.
Revised: In Fig.

Original: 2, we see that basic QT-Opt fails to make progress onboth the pre-training and online data.
Revised: 2, we see that basic QT-Opt fails to make progress onboth the pre-training and online data.

Original: Q-learning methods are typically ineffective when providedwith only successful trials [40], since without both “positive” successful trajectories and “negative”failed trajectories, they cannot determine what not to do.
Revised: Q-learning methods are typically ineffective when providedwith only successful trials [40], since without both “positive” successful trajectories and “negative”failed trajectories, they cannot determine what not to do.

Original: Then, during the online phase, QT-Optis faced with a difﬁcult exploration problem, and cannot make progress. 
Revised: Then, during the online phase, any RLmethod that is not pretrained properly is faced with a difﬁcult exploration problem, and cannotmake progress. QT-Opt is one of those RL algorithms.

Original: AWAC [2] does attain anon-zero success rate from the demonstrations, but performance is still poor, and it does not scalesuccessfully to the high-dimensional and complex task during online ﬁne-tuning.
Revised: AWAC [2] does attain a non-zero successrate from the demonstrations, but performance is still poor, and it does not scale successfully tothe high-dimensional and complex task during online ﬁne-tuning.

Original: Next, we willintroduce a series of modiﬁcations to AWAC that bring it closer to QT-Opt, while retaining theability to utilize demonstrations, culminating in our full AW-Opt algorithm.
Revised: Next, we will introduce a series of modiﬁcations to AWAC that bring it closer to QT-Opt, while retaining the ability to utilizedemonstrations, culminating in our full AW-Opt algorithm.

Original: 4.2 Positive Sample Filtering
Revised: 4.2 Positive Sample Filtering

Original: One possible explanation for the poor performance of AWAC in Fig.
Revised: One possible explanation for the poor performance of AWAC in Fig.

Original: 2 is that, with the relatively low success rate (around 4%) after pretraining, the addition of large amounts of additional failed episodes during online explorationdrowns out the initial successful demonstrations and, before the algorithm can learn an effective Q-function to weight the actor updates,results in updates that remove the promisingpre-training initialization.
Revised: 2 is that, with the relatively low success rate (around 4%) after pretraining, the addition of large amounts of additional failed episodes during online explorationdrowns out the initial successful demonstrations and, before the algorithm can learn an effective Q-function to weight the actor updates,results in updates that remove the promisingpre-training initialization.

Original: To address this issue,we introduce two modiﬁcations: (a) We use aprioritized buffer for the critic , where 50% of the data in each batch comes from successful (positive) episodes that end with a reward of 1 .
Revised: To address this issue,we introduce two modiﬁcations: (a) We use aprioritized buffer for the critic , where 50% of the data in each batch comes from successful (positive) episodes that end with a reward of 1 .

Original: Note that for tasks with non-binary rewards, we wouldneed to introduce an additional hyperparameter for the “success” threshold, which we leave forfuture work.
Revised: Note that for tasks with non-binary rewards, we wouldneed to introduce an additional hyperparameter for the “success” threshold, which we leave forfuture work.

Original: (b) We use positive ﬁltering for the actor , applying the AWAC actor update in Equation 1 only on samples that pass the success ﬁlter (i.e., receive a reward of 1 ).
Revised: (b) We use positive ﬁltering for the actor , applying the AWAC actor update in Equation 1 only on samples that pass the success ﬁlter (i.e., receive a reward of 1 ).

Original: While modiﬁcationa) ensures that the critic’s training batch is balanced in terms of positive and negative samples, themodiﬁcation in b) plays a subtle but very important role for the actor.
Revised: While modiﬁcation a) ensures that the critic’s training batch is balanced in terms of positive and negative samples, themodiﬁcation in b) plays a subtle but very important role for the actor.

Original: By ﬁltering the data that ispresented to the actor, this modiﬁcation ensures that the actor’s performance should not drop below behavioral cloning performance, making it independent of the potentially inaccurate critic duringthe early stages of training.
Revised: By ﬁltering the data that ispresented to the actor, this modiﬁcation ensures that the actor’s performance should not drop belowbehavioral cloning performance, making it independent of the potentially inaccurate critic duringthe early stages of training.

Original: After applying the above adjustments, AWAC performance no longer collapses after switching toonline ﬁne-tuning, as shown in Fig. 3, but still fails to improve signiﬁcantly, with a ﬁnal successrate of 5%.
Revised: After applying the above adjustments, AWAC performance no longer collapses after switching toonline ﬁne-tuning, as shown in Fig. 3, but still fails to improve signiﬁcantly, with a ﬁnal successrate of 5%.

Original: We abbreviate this intermediate method as AWAC P (AWAC + P ositive ﬁltering).
Revised: We abbreviate this intermediate method as AWAC P (AWAC + P ositive ﬁltering).

Original: Toimprove it further, we will discuss how to improve exploration performance in the next subsection.
Revised: Toimprove it further, we will discuss how to improve exploration performance in the next subsection.

Original: 4.3 Hybrid Actor-Critic Exploration
Revised: 4.3 Hybrid Actor-Critic Exploration

Original: [Figure]
Revised: [Figure]

Original: As a Q-learning method, QT-Opt does not havean explicit actor.
Revised: As a Q-learning method, QT-Opt does not havean explicit actor.

Original: Since the task involves continuous actions, QT-Opt uses the cross-entropymethod (CEM)
Revised: Since the task involves continuous actions, QT-Opt uses the cross-entropymethod (CEM)

Original: [41] to optimize the action using its critic, which can be viewed as an implicitpolicy (CEM policy).
Revised: [41] to optimize the action using its critic, which can be viewed as an implicitpolicy (CEM policy).

Original: Unlike QT-Opt, AWACis an actor-critic algorithm, which means it hasaccess to both an explicit actor network anda critic network.
Revised: Unlike QT-Opt, AWACis an actor-critic algorithm, which means it hasaccess to both an explicit actor network anda critic network.

Original: Exploration is performed bysampling actions from this actor.
Revised: Exploration is performed bysampling actions from this actor.

Original: However, thiscan present an obstacle to learning in complexdomains.
Revised: However, thiscan present an obstacle to learning in complexdomains.

Original: First, the commonly used unimodal actor distribution may not capture the multimodal intricacies of the true Q-function, especially early on in training when the optimal action has not beendetermined yet.
Revised: First, the commonly used unimodal actor distribution may not capture the multimodal intricacies of the true Q-function, especially early on in training when the optimal action has not beendetermined yet.

Original: Second, the bulk of the learning must be done by the critic, since the actor doesn’treason about long-horizon rewards.
Revised: Second, the bulk of the learning must be done by the critic, since the actor doesn’treason about long-horizon rewards.

Original: Hence, the critic must ﬁrst learn which actions are good, andthen the actor must distill this knowledge into an action distribution, which can introduce delays.
Revised: Hence, the critic must ﬁrst learn which actions are good, andthen the actor must distill this knowledge into an action distribution, which can introduce delays.

Original: Here, we aim to address these issues by utilizing both the actor and the critic for exploration.
Revised: Here, we aim to address these issues by utilizing both the actor and the critic for exploration.

Original: Although we are building on AWAC, we can still use the implicit CEM policy from QT-Opt to selectactions via the critic, bypassing the actor.
Revised: Although we are building on AWAC, we can still use the implicit CEM policy from QT-Opt to selectactions via the critic, bypassing the actor.

Original: We therefore compare ﬁve exploration strategies: (a)actor-only exploration  implicit critic policy  QT-Opt, where CEM is used to pick the bestaction  the critic) episode-level random switcher, which randomly picks a policy at of each episode  critic policy 20% actor step- switcher randomly  policy at 
Revised: We therefore compare ﬁve exploration strategies: (a)actor-only exploration (b) implicit critic policy (as in QT-Opt, where CEM is used to pick the bestaction according to the critic); (c) episode-level random switcher, which randomly picks a policy atthe beginning of each episode (80% critic policy, 20% actor); and (d) step-level random switcher,which randomly picks a policy at each time step (80% critic policy, 20% actor).

Original: Fig.
Revised: Fig.

Original: 4 shows the results of this comparison.
Revised: 4 shows the results of this comparison.

Original: The episode level random switcher outperforms theother candidates, achieving more than 10% success within 80k samples.
Revised: The episode level random switcher outperforms theother candidates, achieving more than 10% success within 80k samples.

Original: Adopting the episode-levelrandom switcher, we call this intermediate algorithm AWAC P ELRS.
Revised: Adopting the episode-levelrandom switcher, we call this intermediate algorithm AWAC P ELRS.

Original: Given the observed improvement in the early stage of exploration, we shift our focus to analyzing whether value propagation inthe critic is sufﬁciently able to utilize the data during online ﬁne-tuning.
Revised: Given the observed improvement in the early stage of exploration, we shift our focus to analyzing whether value propagation inthe critic is sufﬁciently able to utilize the data during online ﬁne-tuning.

Original: 4.4 Action Selection in the Bellman Update
Revised: 4.4 Action Selection in the Bellman Update

Original: In the previous section, we showed how usingthe critic directly for exploration, rather thanthe actor, can lead to signiﬁcantly better performance.
Revised: In the previous section, we showed how usingthe critic directly for exploration, rather thanthe actor, can lead to signiﬁcantly better performance.

Original: We speculate that this is due to thefact that such an exploration strategy providesa degree of “lookahead”: the actor can onlylearn to take good actions after the Q-functionhas learned to assign such actions high values.
Revised: We speculate that this is due to thefact that such an exploration strategy providesa degree of “lookahead”: the actor can onlylearn to take good actions after the Q-functionhas learned to assign such actions high values.

Original: This means that by taking actions according tothe Q-function during exploration, we can “getout ahead” of the actor and explore more efﬁciently.
Revised: This means that by taking actions according tothe Q-function during exploration, we can “getout ahead” of the actor and explore more efﬁciently.

Original: Can we apply a similar logic during training, and construct target Q-values withsome amount of CEM optimization over actions, rather than simply sampling actions from the actor?
Revised: Can we apply a similar logic during training, and construct target Q-values withsome amount of CEM optimization over actions, rather than simply sampling actions from the actor?

Original: Based on this intuition, we hypothesize that, in much the same way that direct maximization overthe Q-function provided better exploration, improving over the actor’s action in the target valuecalculation will also lead to faster learning.
Revised: Based on this intuition, we hypothesize that, in much the same way that direct maximization overthe Q-function provided better exploration, improving over the actor’s action in the target value calculation will also lead to faster learning.

Original: While previous ofﬂine RL techniques [24, 2] makesure to carefully calculate the target value to avoid a potential use of out of distribution actions,we expect this to be less critical for our method, since the positive-ﬁltering strategy in Section 4.2 already ensures that the actor is only trained on successful actions, signiﬁcantly limiting the harmcaused by an inaccurate Q-function.
Revised: While previous ofﬂine RL techniques [24, 2] makesure to carefully calculate the target value to avoid a potential use of out of distribution actions,we expect this to be less critical for our method, since the positive-ﬁltering strategy in Section 4.2already ensures that the actor is only trained on successful actions, signiﬁcantly limiting the harmcaused by an inaccurate Q-function.

Original: To evaluate this idea, we test several ways of selecting actionsin the Bellman backup.Algorithm
Revised: To evaluate this idea, we test several ways of selecting actionsin the Bellman backup.

Original: 1: Dataset D =

Original: ( s t , a t , r ( s t , a t ) , s t +1 ) 2: Initialize buffer β

Original: = D 3: Initialize parameter vectors φ, θ, ¯ θ 4: for each iteration iter do 5: if iter > = T then 6: for each environment step do 7: a t ∼ π φ ( a t | s t )) or a t ∼ π CEM ( a t | s t )) 8: s t +1 ∼ p ( s t +1 | s t , a t ) 9: β ← β ∪ ( s t , a t , r ( s t , a t ) , s t +1 ) 10:
Revised: [Table]

Original: Sample batch ( s t , a t , r ( s t , a t ) , s t +1 ) ∼ β 11: for each gradient step do 12: a ∗ = arg max a ( Q θ ( s t +1 , a )) 13: θ i ← θ i − λ Q L Q ( Q θ i ( s t , a t ) , r ( s t , a t )

Original: + γQθ i ( s t +1 , a ∗ ) 14: ¯ θ = τθ + (1 − τ )¯ θ 15: Adv = Q ¯ θ ( s t , a t ) − E s t ∼ β,a t ∼ A φ ( s t ) [ Q ¯ θ ( s t , a t )] 16: φ i ← φ
Revised: [Table]

Original: i − λ A exp( Adv ) L A ( a t , A φ i ( s t )) · 1 success In addition to the standard AWAC target value,which is taken as an expectation under the actor, we test: (a) maxQ uses the same targetvalue calculation as QT-Opt, optimizing the action in the target value calculation with CEM(“maxQ”); (b) maxQ + ActorMean combines the actor with CEM by using the actorpredicted action as the initial mean for CEM;(c)
Revised: In addition to the standard AWAC target value,which is taken as an expectation under the actor, we test: (a) maxQ uses the same targetvalue calculation as QT-Opt, optimizing the action in the target value calculation with CEM(“maxQ”); (b) maxQ + ActorMean combines the actor with CEM by using the actorpredicted action as the initial mean for CEM;(c)

Original: maxQ
Revised: maxQ

Original: + ActorCandidate combines the actor with CEM by using the actor-predicted action as an additional candidate in each round of
Revised: + ActorCandidate combines the actor with CEM by using the actor-predicted action as an additional candidate in each round of

Original: CEM.
Revised: CEM.

Original: These target value calculations, particularly (a) , bring the method signiﬁcantly closerto QT-Opt.
Revised: These target value calculations, particularly (a) , bring the method signiﬁcantly closerto QT-Opt.

Original: In fact, (a) can be equivalentlyseen as a QT-Opt Q-function learning method,augmented with an additional positive-ﬁlteredactor that is used for exploration.
Revised: In fact, (a) can be equivalently seenas a QT-Opt Q-function learning method, augmented with an additional positive-ﬁltered actor that is used for exploration.

Original: However,as we see in Fig.
Revised: However, as wesee in Fig.

Original: 5, the performance of thisapproach is signiﬁcantly better than both the 
Revised: 5, the performance of this approachis signiﬁcantly better than both the AWAC-style

Original: AWAC-style Bellman backup and the standard QT-Opt method, neither of which learn the task successfully.
Revised: Bellman backup and the standard QT-Opt method, neither of which learn the task successfully.

Original: The “maxQ” strategy leads to morethan three-fold increase in performance after 200k samples over the previous best method.
Revised: The“maxQ” strategy leads to more than three-fold increase in performance after 200k samples over theprevious best method.

Revised: In addition, we notice that (b) is worse than (a) .

Revised: Our hypothesis for why”maxQ + ActorMean” is worse is the following: in actor-critic methods, the actor always lags behind the critic.

Revised: Therefore, when the actor is not good enough, it may shift the initial distribution andlead the CEM in the wrong direction.

Original: Fig.
Revised: Fig.

Original: 5 demonstrates the comparison of these techniques.
Revised: 5 demonstrates the comparison of these techniques.

Original: AWAC P ELRS + maxQ is signiﬁcantlybetter than AWAC P ELRS.
Revised: AWAC P ELRS + maxQ is signiﬁcantlybetter than AWAC P ELRS.

Original: Two variants: AWAC P ELRS + maxQ + ActorMean (using actorpredicted action as the initial CEM mean) and AWAC P ELRS + maxQ + ActorCandidate (usingactor-predicted action as an additional CEM candidate to compare with the best action selected by
Revised: Two variants: AWAC P ELRS + maxQ + ActorMean (using actorpredicted action as the initial CEM mean) and AWAC P ELRS + maxQ + ActorCandidate (usingactor-predicted action as an additional CEM candidate to compare with the best action selected by

Original: CEM) also improve over AWAC P ELRS.
Revised: CEM) also improve over AWAC P ELRS.

Original: As we notice that AWAC P ELRS + maxQ + ActorCandidate achieves the best performance, we turn AWAC P ELRS + maxQ
Revised: As we notice that AWAC P ELRS + maxQ + ActorCandidate achieves the best performance, we turn AWAC P ELRS + maxQ

Original: + ActorCandidate into ourﬁnal AW-Opt method, which is shown in Algorithm 1.
Revised: + ActorCandidate into ourﬁnal AW-Opt method, which is shown in Algorithm 1.

Original: 4.5 Ablations
Revised: 4.5 Ablations

Original: [Figure] After the step-by-step design of the AW-Opt algorithm, we run an additional ablation studyby removing one feature at a time to furtheridentify the importance of each of the designchoices we have made.
Revised: After the step-by-step design of the AW-Opt algorithm, we run an additional ablation studyby removing one feature at a time to furtheridentify the importance of each of the designchoices we have made.

Original: We compare the following four cases: (a) AW-Opt, (b) AW-Optwithout positive-ﬁltering, (c) AW-Opt without
Revised: We compare the following four cases: (a) AW-Opt, (b) AW-Optwithout positive-ﬁltering, (c) AW-Opt without

Original: ActorCandidate, (d) AW-Opt without hybridactor-critic exploration.
Revised: ActorCandidate, (d) AW-Opt without hybridactor-critic exploration.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 6, each of the introduced components plays a signiﬁcant role in providing aboost in performance.
Revised: 6, each of the introduced components plays a signiﬁcant role in providing aboost in performance.

Original: Among the proposed AW-Opt design decisions, positive ﬁltering is particularly crucial for achieving good results in the online phase.
Revised: Among the proposed AW-Opt design decisions, positive ﬁltering is particularly crucial for achieving good results in the online phase.

Original: Without applying positive ﬁltering, theperformance of AW-Opt collapses to 0% once on-policy training starts, conﬁrming our hypothesisthat positive-ﬁltering of the actor robustiﬁes the actor updates against inaccurate Q-functions in theinitial phases of training.
Revised: Without applying positive ﬁltering, theperformance of AW-Opt collapses to 0% once on-policy training starts, conﬁrming our hypothesisthat positive-ﬁltering of the actor robustiﬁes the actor updates against inaccurate Q-functions in theinitial phases of training.

Original: However, each of the other components is also critical, with the bestperforming ablation (the one that uses the standard AWAC target value calculation) still being aboutthree times worse than full AW-Opt.
Revised: However, each of the other components is also critical, with the bestperforming ablation (the one that uses the standard AWAC target value calculation) still being aboutthree times worse than full AW-Opt.

Original:  Experiments
Revised:  Experiments

Original: In our experimental evaluation, we study the following questions: (1) Can AW-Opt learn from bothexpert demonstrations and suboptimal off-policy data?
Revised: In our experimental evaluation, we study the following questions: (1) Can AW-Opt learn from bothexpert demonstrations and suboptimal off-policy data?

Original: (2) Is AW-Opt a viable RL ﬁne-tuning algorithm that continues to improve after pre-training on ofﬂine data in diverse real-world and simulatedsettings?
Revised: (2) Is AW-Opt a viable RL ﬁne-tuning algorithm that continues to improve after pre-training on ofﬂine data in diverse real-world and simulatedsettings?

Original: (3) Is AW-Opt competitive on metrics important for robot learning, such as real-world performance and sample efﬁciency?
Revised: (3) Is AW-Opt competitive on metrics important for robot learning, such as real-worldperformance and sample efﬁciency?

Original: (4) Are the previously mentioned design choices critical for thesuccess of the method on a wide range of tasks?
Revised: (4) Are the previously mentioned design choices critical for thesuccess of the method on a wide range of tasks?

Original: 5.1 Experimental Setup
Revised: 5.1 Experimental Setup

Original: [Figure] We evaluate our method on simulated and real robotic manipulation tasks (Fig. 1 and Fig. 7).
Revised: We evaluate our method on simulated and real robotic manipulation tasks (Fig. 1 and Fig. 7).

Original: We use two real robotic systems withparallel-jaw grippers and over-the-shoulder cameras: Kuka IIWArobots, and a proprietary 7-DoF arm that we call “Pica” 1 .
Revised: We use two real robotic systems withparallel-jaw grippers and over-the-shoulder cameras: Kuka IIWArobots, and a proprietary 7-DoF arm that we call “Pica” 1 .

Original: We usetwo simulated tasks and three real tasks in our experiments:
Revised: We usetwo simulated tasks and three real tasks in our experiments:

Original: Task 1: (Sim) Indiscriminate Grasping with Kuka (Fig. 1 (a)).
Revised: Task 1: (Sim) Indiscriminate Grasping with Kuka (Fig. 1 (a)).

Original: Six random procedurally-generated objects are placed in a bin.
Revised: Six random procedurally-generated objects are placed in a bin.

Original: Ifthe robot lifts any of the objects, the episode is successful.
Revised: Ifthe robot lifts any of the objects, the episode is successful.

Original: We utilize around 1000 off-policy simulation trials for this task.
Revised: We utilize around 1000 off-policy simulation trials for this task.

Original: Task 2: (Sim) Green Bowl Grasping with Pica (Fig. 1 (b)).
Revised: Task 2: (Sim) Green Bowl Grasping with Pica (Fig. 1 (b)).

Original: Agreen bowl is randomly placed on a table among other objects.
Revised: Agreen bowl is randomly placed on a table among other objects.

Original: Ifthe robot lifts the bowl, the episode is considered a success.
Revised: Ifthe robot lifts the bowl, the episode is considered a success.

Original: Weutilize 120 teleoperated simulated demonstrations for this task.
Revised: Weutilize 120 teleoperated simulated demonstrations for this task.

Original: Task 3: (Real)
Revised: Task 3: (Real)

Original: Indiscriminate Grasping with Kuka (Fig.
Revised: Indiscriminate Grasping with Kuka (Fig.

Original: 1 (c)).
Revised: 1 (c)).

Original: This is the real-world version of Task 1.
Revised: This is the real-world version of Task 1.

Original: We follow the setup proposed in [3], and utilize off-policy data for this task for pre-training.
Revised: We follow the setup proposed in [3], and utilize off-policy data for this task for pre-training.

Original: Task 4: (Real)
Revised: Task 4: (Real)

Original: Grasping from the Middle Bin with Pica (Fig. 1(d)).
Revised: Grasping from the Middle Bin with Pica (Fig. 1(d)).

Original: Several types of trash are placed randomly in three bins.
Revised: Several types of trash are placed randomly in three bins.

Original: Theepisode is successful if the robot lifts any object from the middlebin.
Revised: Theepisode is successful if the robot lifts any object from the middlebin.

Original: We use 1000 demonstrations for this task.
Revised: We use 1000 demonstrations for this task.

Original: Task 5: (Real) Grasping Compostables from the Middle Bin with
Revised: Task 5: (Real) Grasping Compostables from the Middle Bin with

Original: Pica (Fig. 1 (d)).
Revised: Pica (Fig. 1 (d)).

Original: This is a more difﬁcult version of Task 4.
Revised: This is a more difﬁcult version of Task 4.

Original: Whilesetup is equivalent, an episode is successful if a compostable object was lifted.
Revised: Whilesetup is equivalent, an episode is successful if a compostable object was lifted.

Original: We utilize 300demonstrations for this task.
Revised: We utilize 300demonstrations for this task.

Original: In the pretraining phase, we utilize the prior data described above and train the network (the samenetwork as QT-Opt [3]) for 5,000 to 10,000 gradient steps.
Revised: In the pretraining phase, we utilize the prior data described above and train the network (the samenetwork as QT-Opt [3]) for 5,000 to 10,000 gradient steps.

Original: Each algorithm then collects 20,000to 200,000 episodes of on-policy data during further training.
Revised: Each algorithm then collects 20,000to 200,000 episodes of on-policy data during further training.

Original: For Task 4 and Task 5, ﬁne-tuning isdone in simulation, using RL-CycleGAN [43] to produce realistic images.
Revised: For Task 4 and Task 5, ﬁne-tuning isdone in simulation, using RL-CycleGAN [43] to produce realistic images.

Original: This allows us to compareﬁne-tuning performance between QT-Opt, AWAC, and AW-Opt on these tasks, using simulatedevaluations.
Revised: This allows us to compareﬁne-tuning performance between QT-Opt, AWAC, and AW-Opt on these tasks, using simulatedevaluations.

Original: Separately, we conduct real-world evaluations, evaluating the ofﬂine pretraining stageand sim-to-real transfer of ﬁne-tuned policies, since running each of the three algorithms online inthe real world would be too time-consuming.
Revised: Separately, we conduct real-world evaluations, evaluating the ofﬂine pretraining stageand sim-to-real transfer of ﬁne-tuned policies, since running each of the three algorithms online inthe real world would be too time-consuming.

Original: This evaluation protocol provides evidence to supportour central claims: the simulated experiments conﬁrms that our method can effectively ﬁnetunefrom online data, while the real-world evaluation conﬁrms that our approach can handle ofﬂinepre-training on real data effectively.
Revised: This evaluation protocol provides evidence to supportour central claims: the simulated experiments conﬁrms that our method can effectively ﬁnetunefrom online data, while the real-world evaluation conﬁrms that our approach can handle ofﬂinepre-training on real data effectively.

Original: 5.2 Learning from Demonstrations and Suboptimal Off-Policy Data
Revised: 5.2 Learning from Demonstrations and Suboptimal Off-Policy Data

Original: To address the question of whether AW-Opt can learn using different types of prior data, we preparedatasets from different sources: simulated off-policy data (Task 1), simulated demonstrations (Task2) and real demonstrations (Task 4 and Task 5).
Revised: To address the question of whether AW-Opt can learn using different types of prior data, we preparedatasets from different sources: simulated off-policy data (Task 1), simulated demonstrations (Task2) and real demonstrations (Task 4 and Task 5).

Original: We compare the learning performance of QT-Opt,
Revised: We compare the learning performance of QT-Opt,

Original: AWAC, and AW-Opt using these 3 different data sources in Fig.
Revised: AWAC, and AW-Opt using these 3 different data sources in Fig.

Original: 8 (left panels of each subplot).
Revised: 8 (left panels of each subplot).

Original: Both AWAC and AW-Opt can learn from both demonstrations and off-policy data, while QT-Optcannot, since Q-learning requires both successful and failed trials to determine which actions arebetter than others.
Revised: Both AWAC and AW-Opt can learn from both demonstrations and off-policy data, while QT-Optcannot, since Q-learning requires both successful and failed trials to determine which actions arebetter than others.

Original: Depending on the difﬁculty of the task, the success rates of the policies learnedby AWAC and AW-Opt vary from 20% to 60% with the exception of the difﬁcult Task 5, whichrequires on-policy ﬁnetuning.
Revised: Depending on the difﬁculty of the task, the success rates of the policies learnedby AWAC and AW-Opt vary from 20% to 60% with the exception of the difﬁcult Task 5, whichrequires on-policy ﬁnetuning.

Original: 5.3 Online Fine-tuning
Revised: 5.3 Online Fine-tuning

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 8 (right panels of each subplot), during online ﬁne-tuning, QT-Opt performs significantly better than AWAC, while the latter struggles to effectively utilize online rollouts.
Revised: 8 (right panels of each subplot), during online ﬁne-tuning, QT-Opt performs significantly better than AWAC, while the latter struggles to effectively utilize online rollouts.

Original: However,with the modiﬁcations in AW-Opt, our method is able to attain the best of both worlds, ﬁne-tuningeffectively during the online phase, while utilizing the ofﬂine data to bootstrap this ﬁne-tuning process.
Revised: However,with the modiﬁcations in AW-Opt, our method is able to attain the best of both worlds, ﬁne-tuningeffectively during the online phase, while utilizing the ofﬂine data to bootstrap this ﬁne-tuning process.

Original: While AW-Opt is more data-efﬁcient than QT-Opt on Task 1 and Task 2, this difference is particularly pronounced on Task 4 and Task 5, where the inability of QT-Opt to utilize the prior datasigniﬁcantly hampers its exploration.
Revised: While AW-Opt is more data-efﬁcient than QT-Opt on Task 1 and Task 2, this difference isparticularly pronounced on Task 4 and Task 5, where the inability of QT-Opt to utilize the prior datasigniﬁcantly hampers its exploration.

Original: 5. Real-World Robotic Manipulation Results
Revised: 5. Real-World Robotic Manipulation Results

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: We evaluate the performance of policies trainedwith QT-Opt, AWAC, and AW-Opt after ofﬂine pretraining for Task 3 and after ﬁnetuning for Task 5.
Revised: We evaluate the performance of policies trainedwith QT-Opt, AWAC, and AW-Opt after ofﬂine pretraining for Task 3 and after ﬁnetuning for Task 5.

Original: The ﬁne-tuning stage is performed in simulation, asdiscussed in Section 5.1.
Revised: The ﬁne-tuning stage is performed in simulation, asdiscussed in Section 5.1.

Original: The goal of these experiments is to validate that policies learned with AWOpt can indeed handle ofﬂine demonstration dataand generalize to real-world settings.
Revised: The goal of these experiments is to validate that policies learned with AW-Opt can indeed handle ofﬂine demonstration dataand generalize to real-world settings.

Original: The size andtype of the datasets are described in Appendix.
Revised: The size and type of the datasets are described in Appendix.

Original: For Task 3, we ﬁrst train AWAC, QT-Opt, and AWOpt with only positive data and then train with both positive and negative data.
Revised: For Task 3, we ﬁrst train AWAC, QT-Opt, and AW-Opt with only positive data and then train withboth positive and negative data.

Original: We evaluate eachpolicy with 612 grasps, as shown in Table 1.
Revised: We evaluate each policy with 612 grasps, as shown in Table 1.

Original: When learning from the combined positive and negativedata, AWAC performance drops to 0%.
Revised: When learning from the combined positive and negative data, AWAC performance drops to 0%.

Original: When QT-Opt learns from only positive data, it also fails tolearn the task.
Revised: When QT-Opt learns from only positive data, it also fails to learn the task.

Original: Only AW-Opt is able to learn from both positive-only and mixed positive-negativedata.
Revised: Only AW-Opt is ableto learn from both positive-only and mixed positive-negative data.

Original: For Task 5, we evaluate each policy with 180 grasps, as shown in Table 1.
Revised: For Task 5, we evaluate eachpolicy with 180 grasps, as shown in Table 1.

Original: Only AW-Opt learnsthis task successfully, achieving a 48.89% success rate.
Revised: Only AW-Opt learns this task successfully, achieving a48.89% success rate.

Original: These results support the conclusions drawnfrom our simulated experiments.
Revised: These results support the conclusions drawn from our simulated experiments.

Original:  Conclusion
Revised:  Conclusion

Original: We presented AW-Opt, a scalable robotic RL algorithm that incorporates ofﬂine data and performsonline ﬁnetuning.
Revised: We presented AW-Opt, a scalable robotic RL algorithm that incorporates ofﬂine data and performsonline ﬁnetuning.

Original: We show that each design decision in AW-Opt leads to signiﬁcant gains in performance, and the ﬁnal method can learn complex image-based tasks starting with either demonstrations or suboptimal ofﬂine data, even in settings where prior methods, such as QT-Opt or AWAC,either fail to leverage prior data or fail to make progress during ﬁnetuning.
Revised: We show that each design decision in AW-Opt leads to signiﬁcant gains in performance, and the ﬁnal method can learn complex image-based tasks starting with either demonstrations or suboptimal ofﬂine data, even in settings where prior methods, such as QT-Opt or AWAC,either fail to leverage prior data or fail to make progress during ﬁnetuning.

Original: Our evaluation does haveseveral limitations.
Revised: Our evaluation does haveseveral limitations.

Original: First, we only evaluate ofﬂine training in the real world, studying ﬁnetuningunder simulated settings where extensive comparisons are feasible.
Revised: First, we only evaluate ofﬂine training in the real world, studying ﬁnetuningunder simulated settings where extensive comparisons are feasible.

Original: Our experiments are also allconcerned with sparse, binary reward tasks.
Revised: Our experiments are also allconcerned with sparse, binary reward tasks.

Original: Such tasks are common in robotics, and AW-Opt couldbe extended to arbitrary rewards via a threshold for positive ﬁltering, a promising direction for futurework.
Revised: Such tasks are common in robotics, and AW-Opt couldbe extended to arbitrary rewards via a threshold for positive ﬁltering, a promising direction for futurework.

Original: Nonetheless, our results suggest that AW-Opt can be a powerful tool for scaling up IL+RLmethods.
Revised: Nonetheless, our results suggest that AW-Opt can be a powerful tool for scaling up IL+RLmethods.

Original: Aside from motivating each design decision, we also hope that the detailed evaluations ofeach component will aid the design of even more effective and scalable RL methods in the future. 
Revised: Aside from motivating each design decision, we also hope that the detailed evaluations ofeach component will aid the design of even more effective and scalable RL methods in the future. A Appendix [Figure] A.1 Experimental results with navigation task

Revised: [Figure]

Revised: This task is signiﬁcantly simplerthan the image-based manipulationtasks, since the observation spaceis lower-dimensional (240 LIDARpoints rather than 472 × 472 image),and the action space is smaller.

Revised: Therefore, all three algorithms attainreasonable performance.

Revised: However,

Revised: AW-Opt and QT-Opt converge to signiﬁcantly better ﬁnal performance, asexpected, and AW-Opt attains muchbetter performance after ofﬂine pretraining compared to QT-Opt (0.2 vs 0.0), allowing it to learn thetask up to a 90% success rate about three times faster than QT-Opt during online ﬁnetuning.

Revised: AWACrequires a similar number of transitions to converge, but reaches a signiﬁcantly worse ﬁnal level ofperformance.

Revised: A.2 Ablation study for episode-level random switcher

Revised: In Section 4.3, we compared differentexploration strategies and concludedthat episode-level random switcheris the best strategy.

Revised: In this section,we further compare the choice ofratio for actor vs. critic-based exploration.

Revised: The default ratio is 80%/20%(critic/actor).

Revised: Fig.

Revised: 11 presents acomparison of different splits, on

Revised: Task 1 and Task 3.

Revised: Each run wasrepeated three times, with each ofthe following splits: 20%/80%,50%/50%, 60%/40%, 70%/30%,80%/20%, 90%/10%.

Revised: For Task 1, wesee similar results for the last threesplits, while for Task 2, we see betterresults with the 30%/70% split.

Revised: A.3 Experimental

Revised: [Figure]

Revised: results with negative data

Revised: [Figure]

Revised: In Table 1, we see that including“negatives” in the ofﬂine pretraining data for QT-Opt signiﬁcantlyimproves QT-Opt’s performance on

Revised: Task 3 and Task 5, where the datawas collected using a scripted policy.

Revised: The negatives and positives are collected using the same scripted policy,with the positives treated as “successful demonstrations.

Revised: ” We might wonder if an analogous trend would holdfor other tasks, where human demonstrations are used as positives.

Revised: Inthese cases, there are no corresponding negatives, but we can supply additional negatives by using a random policy.

Revised: To examine theeffect of these synthetic negatives, we evaluate Task 4 with either 10,000 or 100,000 additional random negatives.

Revised: The results are shown in Fig 12.

Revised: However, in this case, we see that QT-Opt is stillnot able to make any progress, remaining at 0% success rate throughout.

Revised: This is not surprising.

Revised: Inthe case of Task 3 and Task 5, the randomized scripted policy that collects the data has broad coverage, which means that many different actions are in-distribution.

Revised: The “negatives” come from thesame distribution as the positives (they are collected by the same scripted policy).

Revised: Such a conditionis known to be favorable for ofﬂine reinforcement learning [24], since when the data distributionis broad, fewer actions are out of distribution.

Revised: Indeed, this is precisely the distribution used in theoriginal QT-Opt work [3].

Revised: However, in the case of Task 4, the distribution of positives is very narrow.

Revised: We cannot add negatives from the same distribution (since the distribution is over successfulbehaviors), which forces us to add negatives from a different distribution.

Revised: While this does providefor broader coverage, it does not provide broader coverage in the region surrounding the positiveexamples, and therefore does not relieve the distributional shift challenge faced by ofﬂine RL.

Revised: A.4 Action space and loss function

Revised: For Task 1 and Task 3, the robot is operated in a 7D action space, namely: x, y, z, vertical rotationθ , open gripper, close gripper, terminate episode.

Revised: The ﬁrst 4 subactions are continuous while thelast 3 subactions are discrete.

Revised: For Task 2, Task 4 and Task 5, the action space is slightly different.

Revised: It is operated on a 8D action space, namely: x, y, z angle axis x, angle axis y, angle axis z, gripperclosedness, terminate episode.

Revised: The ﬁrst 7 subactions are continuous while the last subaction is discrete.

Revised: The actor loss function considers both discrete and continuous subactions and also applies aweight on each one of the subaction.

Revised: The full equation can be formulated as the following

Revised: [Table]

Revised: [Table]

Revised: where K is the total number of subactions.

Revised: w k is the weight to balance between each subaction.

Revised: w d is the weight for discrete subactions.

Revised: For Task1, the weights are summarized in Table.

Revised: For Task 2, Task4, Task 5, the weights are summarized in Table.

Revised: A.5 Network architecture

Revised: We apply exactly the same network architecture as QT-Opt[3] as thecritic network for all RL algorithms and in all experiments we conducted.

Revised: Actor network also uses the same backbone as the critic network.

Revised: The difference is that it does not have action input and insteadof outputting a single Q value, it outputs the action distribution (meanand variance of Gaussian distribution for continuous subactions anda one hot vector of the discrete actions).

Revised: During the training, critic network and actor network do not shareweights between each other.

Revised: They are trained separately.

Revised: We plan to open source AW-Opt algorithm once the paper is published.

Revised: A.6 Policy Action Selection Runtime

Revised: [Table]

Revised: A signiﬁcant limitation of QT-Opt and other methods thatdo not employ an actor in continuous action spaces is thataction-selection at evaluation time requires an optimization with respect to the critic.

Revised: In this section, we comparethe run time of action selection for QT-Opt, AWAC, AWOpt on different robots.

Revised: Both AWAC and AW-Opt use anactor network after training, while QT-Opt requires optimization over actions with CEM.

Revised: As we can see in Table 4,this results in AW-Opt and AWAC selecting actions aboutthree times faster than QT-Opt.

Revised: We measure the inference time on a Quadro P1000 GPU.

Revised: A.7 Datasets

Revised: [Table]

Revised: In the following we summarize the dataset size and type for different tasks.

Revised: [Table]