Original: Distilling Motion-Planner Augmented Policies into Visual Control Policies for Robot ManipulationAnonymous Author(s)
Revised: Distilling Motion Planner Augmented Policies into Visual Control Policies for Robot Manipulation

Original: AfﬁliationAddress email Abstract: We address the problem of learning complex manipulation tasks inobstructed environmentsfrom visual observations. Learning such tasks directly inthe visual domain is  challenging and sample inefﬁcient due to hard exploration inthe presence of obstacles and high-dimensional observation inputs.
Revised: Abstract: Learning complex manipulation tasks in realistic, obstructed environments is a challenging problem due to hard exploration in the presence of obstacles and high-dimensional visual observations. Prior work tackles the exploration problem by integrating motion planning and reinforcement learning.

Original: We proposea framework to learn a visual control policy  in  a two-step procedure: 1) Learn apolicy with long-horizon planning capabilities in state space using motion planningaugmented reinforcement learning; 2) Distilling this policy into the visual domainwhile removing thedependency on motion planner. In this step, we leverage theexploration and experiences accumulated by the state agent from the ﬁrst step in anasymmetric actor-critic framework.
Revised: However, the motion planner augmented policy requires access to state information, which is often not available in the real-world settings. To this end, we propose to distill a state-based motion planner augmented policy to a visual control policy via (1) visual behavioral cloning to remove the motion planner dependency along with its jittery motion, and (2) vision-based reinforcement learning with the guidance of the smoothed trajectories from the behavioral cloning agent.

Original: We evaluate our method on three manipulationtasks in obstructed environments and compare it to various reinforcement andimitation learning baselines.
Revised: We evaluate our method on three manipulation tasks in obstructed environments and compare it against various reinforcement learning and imitation learning baselines.

Original: The results indicate that our framework is highlysample-efﬁcient and outperforms  state-of-the-art algorithms.
Revised: The results demonstrate that our framework is highly sample-efﬁcient and outperforms the state-of-the-art algorithms.

Original: We also demonstrateour approach’s zero-shot domain  transfercapabilities toother environment settingswith unseen distractors by coupling our method with domain randomization.
Revised: Moreover, coupled with domain randomization, our policy is capable of zero-shot transfer to unseen environment settings with distractors.

Revised: Code and videos are available at https://clvrai.com/mopa-pd .

Original:  Introduction
Revised:  Introduction

Original: Efﬁciently learning from visual observations has been a long-standing challenge for Reinforcement Learning (RL) algorithms. More so, it becomes even more difﬁcult when solving complex manipulation tasks in cluttered environments, which require precise object-interactions as well as collision-freemovements around obstacles.
Revised: Solving complex manipulation tasks in obstructed environments is a challenging problem in deep reinforcement learning (RL) since it requires precise object interactions as well as collision-free movement across obstacles.

Original: This problem can be tackled by combining the strengths of Motion Planning (MP) and RL[1, 2, 3] – safe collision-free maneuvers of MP and sophisticated contact-basedinteractions of RL.
Revised: To tackle this problem, prior works [1–3] have proposed to combine the strengths of motion planning (MP) and RL – safe collision-free maneuvers of MP and sophisticated contact-rich interactions of RL, demonstrating promising results.

Original: However, these methods have practical limitations, such as assuming access togeometric state of an environment, necessary to perform collision checks in MP. Another limitation isthe computational cost of MP which makes it difﬁcult to integrate them in  a real-time control loop.
Revised: However, MP requires access to the geometric state of an environment for collision checking, which is often not available in the real world and is also computationally expensive for a real-time control.

Original: In this paper we propose a two-step approach to learn a visuomotor policy thatovercomes thesedrawbacks. We start by training an agent to solve the  task in a simulated environment where accessto privileged state information is available.
Revised: To deploy such agents in realistic settings, we need to resolve the dependency on the state information and costly computation of MP, such that the agent can perform a task in the visual domain.

Original: There, we use Motion Planner Augmented Reinforcement Learning (MoPA-RL) [1] to train a policy equipped with a  motion planner  that can be called toplan collision-free motions and efﬁciently explore theenvironment. In the second step, we transferthis state-based  policy  into  visual domain by distilling thebehavior of motion planner into a directfeedback policy, which is the main focus of this paper.
Revised: To this end, we propose a two-step distillation framework, motion planner augmented policy distillation (MoPA-PD), that transfers the state-based motion planner augmented RL policy (MoPA-RL [1]) into a visual control policy, thereby removing the motion planner and the dependency on the state information.

Original: The resulting policy will not need expensivecalls to motion planner and directly select an action based on visual input, which also makes it suitablefor sim-to-real transfer. Directly learning from raw pixels has been a challenging problem in RL, evenharder for robot learning problems due to the cost and safety concerns with real robots.
Revised: Concretely, our framework consists of two stages: (1) visual behavioral cloning (BC [4]) with trajectories collected using the MoPA-RL policy and (2) vision-based RL training with the guidance of smoothed trajectories from the BC policy.

Original: Therefore,learning transferable policies from simulation to real-world robots has signiﬁcant advantages. Our visual  policy distillation is realized by adopting an asymmetric actor-critic architecture similar to 
Revised: The ﬁrst step, visual BC, removes the dependency on the motion planner and the resulting visual BC policy generates smoother behaviors compared to the motion planner’s jittery behaviors. Then, in the second step, we further improve the visual policy

Original: Pinto et al.
Revised: [Table]

Original: [4] where the value function is learnt using state information as input whereas the actordirectly learns from image observations. This way, our state-based agent’s critic and asymmetriccritic share the same state space, but have different action spaces.
Revised: [Table] [Table] [Table]

Original: Rather than learning from scratch,we leverage the experience of our state agent and critic to accelerate learning a policy in the visual domain.
Revised: [Table]

Original: We achieve this by: 1)
Revised: [Table]

Original: Initializing the critic network of our asymmetric agent with thelearnt weights of the critic network of MoPA-RL. 2) Leveraging optimal expert trajectories fromMoPA-RL and utilizing them for initializing the actor network through pre-training with Behavioral
Revised: using asymmetric actor-critic RL [5] directly from the image observations to overcome sub-optimality in the MoPA-RL policy through self-exploration.

Original: Cloning (BC). While supervised learning with BC does have less training time, it generally failsin near-collision states due to lack of understanding of appropriate collision thresholds because ofunseen environment interactions [5, 6]. We mitigate this issue by training our RL agent on top ofBC. We employ the Soft-Actor Critic (SAC) [7] algorithm for training  the asymmetric agent.
Revised: For efﬁcient vision-based RL training, our method leverages the experience of the BC policy and the state-based critic from MoPA-RL by initializing the critic network with the MoPA-RL critic and the actor network through BC pre-training; BC-trajectory guided RL training using a separate expert replay buffer; tuning the entropy coefﬁcient to encourage exploitation (i.e., maximizing reward) over exploration (i.e., maximizing entropy).

Original: The main contributions of this work are:
Revised: In summary, our contributions are as follows:

Original: A novel framework to learn  visual control policies by learning a state-space policy in simulationthat uses a motion plannerfor efﬁcient exploration in cluttered scenarios and subsequentlytransferring it into the image space while removing the motion planner.2.
Revised: • We propose a novel framework to learn a visual control policy in cluttered scenarios by distilling a state-space policy that uses a motion planner into an image-space policy by removing the motion planner dependency.

Original: An efﬁcient training algorithm for the domain transfer that includes weight initialization, pretraining via BC and ﬁne-tuning with an asymmetric actor-critic algorithm.3.
Revised: • Our asymmetric visual policy leaning ensures high sample-efﬁciency using weight initialization followed by BC trajectory-guided RL with entropy coefﬁcient tuning.

Original:  The evaluation of this approach on three manipulation tasks in obstructed environments and thecomparison against state-of-the-art imitation and reinforcement learning baselines.4. An experiment that demonstrates visualtransfer capabilities of our approach by domain randomizationof textures, colors and lighting during training.
Revised: • The distilled visual policy is capable of zero-shot domain transfer to unseen environments with visual domain randomization during training.

Original:  Our method outperforms  state-of-the-art algorithms in terms of success rate, sample efﬁciency, andaverage pathlength. Interestingly, our method solves tasks much faster, having a signiﬁcantly shorterpath length compared to all other baselines, including the BC expert trajectories as well as ourstate-based MoPA-RL agent. This shows that our asymmetric visual agent optimizes its path beyondthe expert signals it receives by self-exploration.
Revised: • Our method outperforms the state-of-the-art methods in terms of success rate, sample efﬁciency, and path length on three manipulation tasks in obstructed environments.

Original:  Related Work
Revised:  Related Work

Original: Several recent efforts have been made towards distilling motion planning algorithms into neuralmotion planners using learning-based methods [8, 9, 10, 11, 5].
Revised: The objective of our work is to learn complex manipulation tasks in realistic and obstructed environments. Motion planner augmented RL methods [1–3] have shown promising results in solving complex tasks in obstructed environments by combining MP and RL. However, these methods cannot be deployed in real-world settings due to their dependency on state information. To remove this dependency, state-based policies must be transferred to the visual domain.

Original: Most of these efforts can be broadlycategorized into imitation learning (IL) and RL paradigms. Behavioral cloning, being one of the most common frameworks to learn directly from expert demonstrations, has been used to learn neural network based planners [5, 9, 10].
Revised: A typical approach to distill state-based policies into vision-based policies is behavioral cloning (BC [4]). However, BC often suffers when encountering states unseen during training [6]. Therefore, prior works have proposed various methods to improve BC agents using ofﬂine RL [7] and online

Original: The works in [12, 9, 11]also adopt supervised learning over expert demonstrations obtained via sampling-based MP, andmostly focus on learning obstacle representations using sensor-measurements like point-clouds or 2Dlaser scans.
Revised: RL [8].

Original: However the major drawback of supervised learning approaches is that their performance is limited by the demonstrator’s performance itself. Moreover, BC struggles when it encountersdata out of the collected data distribution, due to lack of exploration.
Revised: However, these works do not involve the distillation of motion planners.

Original: A common workaround forimproving BC is to use DAgger [13], which iteratively gathers expert actions for visited states andupdates the state-action pairs in the training data accordingly.
Revised: Several recent efforts have been made towards distilling MP algorithms into neural motion planners using learning-based methods [9–13]. Most of these efforts can be broadly categorized into imitation learning (IL) and RL paradigms.

Original: DAgger hasalso been used in attemptsto distill MP into neural networks [5] and for high-level planning [14].
Revised: For IL, supervised learning on MP trajectories has been used to learn neural network-based planners [10–14].

Original: However, its exploitation ofthe expert for labelling all encountered states can be cumbersome and hard to achieve.
Revised: However, the performance of such supervised learning approaches are limited by the collected dataset and demonstrator’s performance.

Original: To discover better policies using exploration, various off-policy reinforcement learning methodshave been studied for superseding motion planners with neural network policies.
Revised: To discover better policies with additional interactions, various off-policy RL methods have been studied for superseding motion planners with neural network policies [13, 15, 16].

Original: They utilise expertdemonstrations stored in the replay buffer for guided exploration and succeed to learn policies betterthan experts [5, 15, 16]. This technique of using expert replay buffers has been derived from popularimitation learning method [17, 18, 19] which were initially introduced on applications outside MP.
Revised: These approaches utilize expert trajectories stored in the replay buffer for guided exploration, which leads to better policies than experts [17–20].

Original: Prior works combining RL with expert demonstrations are either focused on unobstructed environments [17, 18, 19, 20] or  simpler tasks not involving object manipulation [5].
Revised: However, these works learn either in unobstructed environments [17– 20] or on simpler tasks not involving object manipulation [13].

Original: They also assume fullyobservable environments and learn policies in state space.
Revised: They also assume fully observable environments and learn policies in state space.

Revised: In this work, we focus on complex manipulation tasks in obstructed environments using visual observations.

Original: Moreover, most ofﬂine-RL approaches insparse reward scenarios are coupled with Hindsight Experience Replay (HER) for goal-conditionedpolicies.
Revised: Moreover, most ofﬂine RL approaches [21, 22] in sparse reward scenarios are coupled with hindsight experience replay (HER [23]).

Original: They empirically show that not using HER signiﬁcantly hurts the agent’s performance andmight even completely fail at complex tasks [5, 4, 17, 16].
Revised: They empirically show that not using HER signiﬁcantly hurts the agent’s performance [13, 5, 16, 17].

Original: However,using HER isnot applicable forcomposite tasks involving goals associated with multiple sequential steps. As stated in [21], HERis designed to efﬁciently traina goal-conditioned policy for Multi-goal RL setup, where the policyand critic depend on the state and goal associated with that state. This makes HER not suitable forsequential manipulation tasks, where learning from sparse feedback is challenging without usinggoal-conditioned policies.
Revised: However, HER is designed to efﬁciently train goal-conditioned policies for multi-goal RL, which makes it unsuitable for tasks with multiple sequential steps without explicitly conditioning on goals.

Original: Here, we show that our approach successfully performs composite manipulation tasks in obstructed environments without using HER for tackling sparse rewards.
Revised: In contrast, our method successfully learns composite manipulation tasks in cluttered environments without using HER for tackling sparse rewards.

Original: We also learnour policy in visual domain keeping in mind its applicability for sim-to-sim and sim-to-real transfer.

Original:  Method
Revised:  Method

Original: Our goal is to solve complex manipulation tasks in obstructed environments in the visual domain.
Revised: Our goal is to solve complex manipulation tasks in obstructed environments with visual inputs.

Original: While MP-based methods can efﬁciently solve such tasks, they are restricted to state-space learningwhich is difﬁcult to transfer to real environments.
Revised: While MP-based methods can efﬁciently solve such tasks, they are restricted to learning a state-based policy which is difﬁcult to transfer to real environments.

Original: Moreover, sampling-based motion planning incurssigniﬁcant computational costs that makes it hard to integrate them in real-time controllers.
Revised: Moreover, sampling-based motion planning incurs signiﬁcant computational costs, making it hard to integrate them into real-time controllers.

Original: Wepropose a method which moves from ﬁrst learning a state-based agent using MP-based actions tosubsequently learning a visual policy suitable for zero-shot sim-to-sim and sim-to-real transfers.
Revised: Thus, we propose a method that distills an MP-augmented state-based agent into a visual policy, removing the MP and state dependency.

Original: Ourgeneral approach is illustrated in Figure 1. We accelerate visual policy learning by leveraging ourstate-based agent’s experience and therefore increase its sample-efﬁciency.
Revised: We formally deﬁne our problem and introduce the MP-augmented RL in

Original: 3.1 Problem Formulation
Revised: Section 3.1.

Original: We formulate our problem as a continuous state space Markov Decision Process (MDP) for bothour state-based as well as asymmetric-visual agent.
Revised: Then, we describe our two-step visual distillation approach in Section 3.2.

Original: The MP-augmented MDP ˜ M introduced in[1] can be represented as a tuple ( S , A ∪ ˆ A , ˜ P, ˜ R, ρ 0 , γ ) where S is a set of continuous states, Ais a set of continuous actions in the direct action space and ˆ A is a set of continuous actions in the MP-augmented action space for enlarged actions similar to [1]. For the rest of the paper, we refer tothis joint action space
Revised: 3.1 Preliminaries

Original: A ∪ ˆ A as ˜ A . 1 ˜ P : S × ˜ A × S → R deﬁnes the transition function, ˜ R is thereward function, ρ 0 is the initial state distribution, and γ is the discount factor for ˜ M . Next, we represent our asymmetric-visual agent’s MDP M  by a tuple ( S , O , A , P, R, ρ 0 , γ ) similar to[4].
Revised: Problem formulation We formulate our problem as a Markov Decision Process (MDP) M deﬁned by a tuple ( S , O , A , P, R, ρ 0 , γ ) that consists of a state space S , partial observation space O (visual inputs corresponding to states), action space A , transition function P : S ×

Original: The state space S  , initial state distribution ρ 0 , and discount factor γ are the same as in augmented
Revised: A × S → R , reward function R , initial state distribution ρ 0 , and discount factor γ .

Original: MDP ˜ M .

Original: However, the action space of M is different to ˜ M and only consists of direct action space

Original: A , leading to a different transition function P : S ×A×S → R and reward function R .

Original: Additionally,we assume to have access to a set of continuous partial observations O corresponding to states in S .

Original: Note that the A is a subset of ˜ A – they have the same dimension but the interval bounds are larger for˜ A because it also contains motion planner actions.

Original: Additionally, the reward and transition functionsof both MDPs are the same for direct actions, but only the transition function ˜ P handles MP actions.

Original: At any time step t , the state agent’s action distribution is represented by the policy π φ (˜ a t | s t ) , with itbeing in state s t ∈ S , taking action˜ a t ∈ ˜ A , where φ denotes policy parameters. For our asymmetricvisual agent, the actor takes in apartially observable state o t corresponding to s t and outputs actiona t ∈ A , represented by the visual policy π θ ( a t | o t ) where θ denote the parameters of the visualpolicy network. The goal of this work is to transfer the behavior of the policy π φ (˜ a t | s t ) to the policyπ θ ( a  t | o t ) , which means to transfer a policy into a different state as well as action space. The learnedcritic network Q ψ : S × ˜
Revised: The agent is represented as a policy π ( a t | o t ) , which takes an action a t under a visual observation o t and receives a reward r t =

Original: A → R is shared between the two MDP’s, which is possible since theaction space of the asymmetric agent

Original: A is a subset of the action space of the state-based agent ˜ A .

Original: At every time step t , the agent receives a reward r t =

Original: R ( s t , a t ) , with the state transitioning to s t +1 .
Revised: R ( s t , a t ) , with the state transitioning to s t +1 .

Original: The goal of the agent is to maximize the expected returns where the return is deﬁned as the discountedsum of rewards  (cid:80) T − 1 t =

Revised: The goal of the agent is to maximize the expected discounted sum of rewards E ( s,a ) ∼ π (cid:80) T − 1 t =

Original: γ t R ( s t , a t ) , with T depicting the episode horizon.
Revised: γ t r t , where T is the episode horizon.

Original: 3.2 Motion Planner-Augmented RL Policy as Expert
Revised: Motion Planner-Augmented RL (MoPA-RL) To tackle manipulation tasks in cluttered environments, Yamada et al.

Original: In the ﬁrst step ofour approach, we learn a state-based expert policy using the MoPA-RL framework [1].
Revised: [1] proposed to incorporate MP and RL via an MP-augmented MDP ˜ M = ( S , ˜ A , ˜ P, ˜ R, ρ 0 , γ ) that consists of the state space S , enlarged action space ˜ A augmenting the direct action space A with the MP action space ˆ A , 5 augmented transition function ˜ P : S× ˜ A×S → R , augmented reward function ˜ R ( s, ˜ a ) , initial state distribution ρ 0 , and discount factor γ .

Original: This framework learns a policy π φ (ˆ a t | s t ) on the augmented MDP ˜ M with the Soft Actor
Revised: MoPA-RL [1] learns a policy π φ (˜ a t | s t ) on the augmented MDP ˜ M with an off-policy RL algorithm,

Original: Critic (SAC) algorithm.
Revised: SAC [24].

Original: Precisely,  the policy predicts actions deﬁned as the joint displacements∆ q t for the robot in each state.
Revised: Precisely, given a state s t , the policy predicts an action ˜ a t , which is deﬁned as a robot joint displacement ∆ q t .

Original: Based on the actionmagnitude, if it lies within the original actionspace A , this action is directly executed by the controller, similar to any RL policy. This usuallyaids in performing contact-rich manipulation tasks with high ﬁdelity.
Revised: If the action lies within the direct action space A , it is directly executed by the controller as the agent performs sophisticated and contact-rich manipulations.

Original: When the actions are largerinmagnitude, lying in MP-augmented action space ˜  A , this often increases the probability of collisionsin the presence of obstacles.
Revised: However, when the actions are larger in magnitude (i.e. ˜ a ∈ ˆ A ), the probability of collisions in the presence of obstacles increases.

Original: Therefore, the sampling-based motion planner RRT-Connect [22] iscalled to realize  actionsassociated with large joint displacements, efﬁciently planning and exploringobstructed environments.
Revised: Therefore, a sampling-based motion planner, RRT-Connect [25], is called to realize such actions with large joint displacements by computing collision-free paths.

Original: The results in [1] show that this policy structure leads to a more efﬁcientexploration of the state space and a faster convergence of the learning.
Revised: This allows the agent to efﬁciently explore obstructed environments while avoiding collision [1] 3.2 Motion Planner Augmented Policy Distillation

Original: However, the usage of amotion planner inside an RL policy also has certain weaknesses like high computational costs and thedifﬁculty to transfer it toreal scenarios where the environment state is not fully observable.
Revised: Despite the advantages of MoPA-RL, its applicability to the real world is limited due to the high computational cost of MP and dependency on fully observable environment states.

Original: Therefore,we propose a two-step learning procedure in which we start with MoPA-RL and add a subsequentstep that transfers the learnt policy into the visual domain while removing its dependency on the MP. 3.3 Learning a Direct Policy in the Visual Domain The second step of our approach focuses on moving from a state-based MP-augmented policy to animage-based direct policy (without MP) leveraging demonstrations from our MoPA-RL policy asthe expert.
Revised: To this end, we propose a visual distillation method, motion planner augmented policy distillation (MoPAPD), that learns an image-based control policy (without MP) from a state-based MP-augmented policy by leveraging its rollouts and the learned state-based critic as a guidance.

Original: This stage has two advantages: 1) The learnt policy no longer depends on a samplingbased motion planner for planning collision-free paths in obstructed environments.
Revised: Concretely, our proposed method consists of two stages, as illustrated in Figure 1.

Original: This makesthe policy much faster during learning and inference as MP can be computationally expensive inhigh-dimensional spaces and slow due to performing collision checks. The learnt policy directlyoperates on environment images, not requiring state information and thus making it adequate forsim-to-sim and sim-to-real transfers. We now describe different components of our approach in detail.
Revised: Given a state-based MoPA-RL agent, we ﬁrst use BC to train a vision-based actor with trajectories collected from the MoPA-RL policy (Section 3.2.1), and then we further improve the vision-based agent via BC trajectory-guided asymmetric actor-critic RL [5] with the visual BC actor and MoPA-RL critic (Section 3.2.2).

Original: 3.3.1 Trajectory Smoothing using Behavioral Cloning 
Revised: 3.2.1 Stage 1: Visual Behavioral Cloning and Trajectory Smoothing

Original: Our MoPA-RLexpert policy is learnt with access to complete information about the environmentstates at all times and also utilizes MPcalls for executing large action steps without collisions.
Revised: The MoPA-RL policy is learned with access to complete information about the environment states and also utilizes MP for executing large action steps without collisions.

Original: Weaim to utilise expert demonstrations from this planner-based policy and distill it into RL-based visualpolicy, thereby deducting expensive MP computations and training the actor with only environmentimages.
Revised: In this paper, we aim to utilize demonstrations from this planner-based policy and distill it into a visual control policy, thereby deducting expensive MP computations and training the actor in visual domain.

Original: We collect multiple transitionst i into MoPA-RL demonstration dataset D mp =
Revised: With the learned MoPA-RL policy, we ﬁrst collect multiple transitions

Original:  { t 1 , t 2 , t 3 , ... } ,where t i =
Revised: d i in low-level (direct) action space and store them into the MoPA-RL demonstration dataset D mp = { d 1 , d 2 , d 3 , ... } , where d i =

Original: ( s i , o i , a i , r i , s i +1 ) .
Revised: ( s i , o i , a i , r i , s i +1 ) .

Original: The transitions from MoPA-RL are collected in the low-level (direct)action space. Note that in order to learn a vision-based policy in the subsequent step, we render imageobservations o i corresponding to each environment state s i during ofﬂine data collection. While MP-based methods can be effective for planning collision-free trajectories in obstacle-proneenvironments, their paths can be non-smooth [23].
Revised: Then, we train a visual BC actor π θ ( a t | o t ) using observation-action pairs ( o i , a i ) from the dataset D mp by minimizing the mean squared error. Distilling the MoPA-RL trajectories using BC not only enables the BC actor to work directly on visual inputs but also reduces jerky motion planning behaviors of MoPA-RL, which occur due to motion planner’s priority on obstacle avoidance over trajectory smoothness [26].

Original: This makes it hard for the robot to learn consistent, naturally smooth motion. Therefore, we ﬁrst learn a BC agent using observation-action pairs ( o i , a i )from the dataset D mp by minimizing the mean squared error.
Revised: By removing unnecessary, jerky movements through visual BC, the resulting trajectories become smoother and even shorter, and thus help learning consistent and smooth motions.

Original: The BC agent shares the same networkarchitecture and parameters that we use for our subsequent asymmetric actor π θ ( a t | o t ) . We observethat distilling the MoPA-RL trajectories using BC smooths out the rollouts. We then collect multiplerollouts by interacting with the environment using our BC policy and save the reﬁned transitions( s i , o i , a i , r i , s i +1 ) in an augmented expert replay buffer R e .
Revised: However, the BC actor often fails when it encounters states not seen during training due to covariate shift [6]. To improve the robustness of the policy, we further train the visual BC actor using RL with additional environment interactions.

Original: The main purpose of using R e is topermanently keep hold of expert trajectories during training to facilitate guided exploration. Thisidea was initially proposed in [18] for discrete action space domains. In our experiments in Section 4,we show that trajectory smoothing for R e is crucial for efﬁciently learning a successful policy.
Revised: In other words, the BC actor can be a good starting policy for visual RL training and the trajectories collected from the BC actor can effectively guide exploration.

Original: 3.3.2 Learning a Visual Policy with Asymmetric Actor-Critic
Revised: 3.2.2 Stage 2: Vision-based RL with Asymmetric Actor-Critic

Revised: In the second stage, we further train the visual policy using RL directly on image observations to enhance the robustness of the policy and overcome sub-optimality in the planner-based policy.

Original: To train our RL agent, we adopt an asymmetric actor-critic architecture, similar to [4], whereinputsto the critic are full environment states and the actor is learnt with environment images and robotjoint states.
Revised: For efﬁcient RL training, we adopt an asymmetric actor-critic architecture [5], where the actor acts based on environment images and robot joint states while the critic learns from environment states.

Original: The choice of this method is motivated by learning transferable visual policies, which donot require state information during inference, and also beneﬁt from access to environment states,when learned in simulation.
Revised: This architecture is motivated by learning transferable visual policies, which do not require state information during inference but beneﬁt from them while learning in simulation.

Original: Our training procedure comprises of the following components:
Revised: Our training procedure comprises of the following components:

Original: Initializing actor and critic networks: Initializing network architectures with suitable weightsis a promising method for accelerating RL [24, 25].
Revised: Weight initialization of actor and critic networks Initializing actor and critic networks with suitable weights can accelerate RL training by providing more optimal rollouts from the actor and informative learning signals from the critic compared to the randomly initialized actor and critic [27, 28].

Original: Especially with RL over pixels, meaningfulinitializations can signiﬁcantly improve sample-efﬁciency [20, 26].
Revised: Especially when learning from pixels, suitable initialization can signiﬁcantly improve sample efﬁciency [20, 29].

Original: We propose to utilize our statebased agent’s experience to initialize our asymmetric visual agent’s networks to bring their initialdistributions closer to expert signals.
Revised: To this end, we propose to leverage the state-based agent’s experience by utilizing its weights for initializing our asymmetric visual agent’s networks, thereby bringing their initial distributions closer.

Revised: Thus, the visual actor network π θ ( a t | o t ) is initialized with the weights of the visual BC actor learned in Section 3.2.1.

Original: The critic Q ψ ( s t , a t )is initialized with MoPA-RL’s criticweights.
Revised: We also initialize the critic Q ψ ( s t , a t ) with the critic of MoPA-RL.

Original: Note that the action space of two critics is different, but initialization is still feasible sincethe action space of MoPA-RL is a superset of  direct action space.
Revised: Note that the action spaces of two critic networks are different, but initialization is still feasible since the action space of MoPA-RL is a superset of the direct action space.

Original: For the actor network π θ ( a t | o t ) ,we initialize it with weights of the BC agent learned for trajectory smoothing (Section 3.3.1). Main training phase: After initializing the actor and critic networks, we initiate the agent’s onlineinteractions with the environment. The initial policy rolloutsare collected into the agent’s replaybuffer R π .Further, we adopt a ﬁxed sampling ratio to sample 1:3 transitions from R e and R πrespectively, for each training iteration.
Revised: BC trajectory-guided asymmetric RL After initializing the actor and critic networks, we collect agent rollouts into the agent replay buffer R π .

Original: This ensures that the agent continually improves in an onlinesetting with its own experiences as well as has certain expert transitions to guide exploration.

Original: Fortraining our RL agent, we adopt Soft Actor-Critic (SAC) [7], an off-policy model-free RL algorithmfor continuous control. 
Revised: As an RL algorithm, we adopt Soft Actor-Critic (SAC [24]), an off-policy model-free RL algorithm for continuous control. We optimize our asym- [Figure]

Original: We optimize our asymmetric visual policy π θ by maximizing the  objective:
Revised: metric visual policy π θ by maximizing the following objective:

Original: [Equation]
Revised: [Equation]

Original: where the temperature parameter α is the coefﬁcient of the entropyfunction H which regulates theexploration versus exploitation trade-off during learning.
Revised: where the temperature parameter α balances between exploration and exploitation using entropy H .

Original: The performance of SAC is known to be sensitive with respect to α [27].
Revised: RL training with a BC-initialized policy often quickly deviates from the original policy. To prevent this problem and guide exploration during RL, we update our RL agent not only with agent trajectories but also with BC policy (i.e. expert) trajectories [18]. Thus, we collect smoothed trajectories from our visual BC policy, ( s i , o i , a i , r i , s i +1 ) , in an expert replay buffer R e .

Original: α controls the relativeweight of increasing entropy against reward maximization.
Revised: Then, for each RL training iteration, we sample 1:3 transitions from R e and R π , respectively. A separate expert replay buffer ensures guided exploration from the expert and also circumvents catastrophic forgetting after weight initialization. Note that we use the BC trajectories R e instead of the MoPA-RL data D mp because jittery motion planning paths in the MoPA-RL data make RL training difﬁcult and sub-optimal. Entropy coefﬁcient tuning The performance of SAC is known to be sensitive to α [30].

Original: Since our asymmetric agent already startsfrom experience collected using state-based agent, we initialize α to values lower than the ﬁnal αobtained from our state-based agent.
Revised: Since our asymmetric agent already starts with the well-trained BC actor and MoPA-RL critic, we initialize α to values lower than the ﬁnal α obtained in MoPA-RL.

Original: This is to ensure that with prior knowledge in state-space, theasymmetric visual agent focuses more on maximizing rewards rather than maximizing entropy forexploration.
Revised: This is to ensure that with prior knowledge in the state-based agent, the visual agent focuses more on maximizing rewards over exploration.

Original: We describe the  implementation detailsand choice of hyperparameters in supplementary.
Revised: We describe the hyperparameter choice and implementation details in Section ?? and Section ?? .

Revised: In summary, we propose a two-step visual distillation method for a state-based MoPA-RL agent using visual BC followed by BC trajectory-guided vision-based RL to remove the dependency on MP and environment states.

Revised: For efﬁcient RL training, our method leverages weight initialization of the actor and critic, the expert replay buffer of BC smoothed trajectories, and entropy coefﬁcient tuning.

Original:  Experiments
Revised:  Experiments

Revised: In this paper, we propose to distill a motion planner augmented policy into a visual control policy for complex manipulation tasks in obstructed environments.

Original: Our experiments are motivated to answer the following questions: (1) Does IL efﬁciently learnpolicies for obstructed environments in image space?
Revised: In our experiments, we aim to answer the following questions: (1) Does IL efﬁciently learn policies for obstructed environments in image space?

Original: Moreover, is naively combining IL with RLsufﬁcient for solving complex tasks?
Revised: Moreover, is naively combining IL with RL sufﬁcient for solving complex tasks?

Original: (2) Is  learning a visual policyhard using MP with RL?
Revised: (2) Is distillation better than directly learning a visual policy using MP?

Original: (3) Doesour proposed approach efﬁciently learn a visual policy using prior state-based exploration knowledge?
Revised: (3) Does our approach, MoPA-PD, efﬁciently learn a visual policy using prior state-based exploration knowledge?

Original: Also, is ourlearnt visual policy capable of domain transfer and robust to unseen distractors?
Revised: (4) Is our visual policy capable of domain transfer and robust to unseen distractors?

Original: 4.1 Environments
Revised: 4.1 Environments

Original: We evaluate our approach on three obstructed environments, posing hard exploration challengessimulated using the MuJoCo physics engine [28].
Revised: We evaluate our approach on three obstructed environments (see Figure 2) from Yamada et al. [1], simulated using the MuJoCo physics engine [31].

Original: We use a 32x32 image as a visual observation.
Revised: We use a 32x32 image as a visual observation.

Original: Thestate information for each environment is described in Section C.

Original: [Figure]

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: MoPA-RL [1] is the expert agent trained with state-based policy.
Revised: MoPA-RL [1] is the expert agent trained with state-based policy.

Original: All baselinesbelow horizontal line are trainedwith asymmetric actor-critic for fair comparison.
Revised: All baselines are trained

Original: Maximum horizon is 250 forall tasks. • Sawyer Push: A Rethink Sawyer robot arm with 7 DoF, initially located outside the box needs toreach an object placed inside the box and push it to the goal location within the box.• Sawyer Lift: The agent needs to reach an object placed inside a box, grasp and lift it outside thebox avoiding collisions.
Revised: for • Sawyer Push: A Rethink Sawyer robot arm with 7 DoF, initially located outside the box, needs to reach an object placed inside the box and push it to the goal location within the box. • Sawyer Lift: The Sawyer arm must reach an object placed inside a box, grasp, and lift it outside the box, avoiding collisions.

Original: The Sawyer’s arm is always located outside the box as an initial setup.• Sawyer Assembly:
Revised: The Sawyer arm is always initialized outside the box. • Sawyer Assembly:

Original: Thegoal for the Sawyer agent is to assemble the fourth leg  of a table intothevacant hole in the presence of the other three legs being already assembled, avoiding any collisionswith them. As an initial setup, the leg is already attached to the agent’s gripper.
Revised: The Sawyer arm needs to assemble the fourth leg (already attached to the gripper) of a table into the vacant hole while avoiding collisions to other three legs.

Original: This environmentis built upon the IKEA furniture assembly environment [29].
Revised: This environment is built upon the IKEA furniture assembly environment [32].

Original: Rewards: For Sawyer Push and Assembly environments, we use sparse rewards, where the agentreceives a reward proportional to the distance between the end-effector and object, or the object andgoal position when  distance is less than (cid:15) .
Revised: For Sawyer Push and Sawyer Assembly, the agent receives a reward proportional to the distance between the end-effector and object, or the object and goal position when the distance is less than (cid:15) .

Original: We use (cid:15) = 0 .
Revised: We use (cid:15) = 0 .

Original: 1 for Sawyer Push and (cid:15)
Revised: 1 for Sawyer Push and (cid:15)

Original: = 0 .
Revised: = 0 .

Original: 3 for Sawyer Assemblyenvironment while the distance between initial and goal state is around 1.2.
Revised: 3 for Sawyer Assembly while the distance between initial and goal state is around 1 . 2 .

Original: An episode isconsidered successful when the distance between the cube(Sawyer Push) and goal  or the peg-headand goal (Sawyer Assembly) is less than 0.05.
Revised: An episode is considered successful when the distance between the cube and goal (Sawyer Push) or the peg-head and goal (Sawyer Assembly) is less than 0.05.

Original: For Sawyer Lift task, we use  reward shaping, similarto [30], where a reward is deﬁned for all three intermediate stages of the task: reach , grasp and lift .
Revised: For Sawyer Lift, we use the reward function similar to Fan et al. [33], where a reward is deﬁned for all three intermediate stages of the task: reach , grasp and lift .

Original: A bonus reward signal of (+150) is received upon successful task completion in all the environments.
Revised: A bonus reward signal of 150 is received upon successful task completion in all the environments.

Revised: We refer readers to Section ??

Revised: for more details.

Original: 4.2 Baselines
Revised: 4.2 Baselines

Original: We compare  the followingmethods as baselines to evaluate thesuccess and merits of our approach:
Revised: We compare our method with the following baselines to evaluate the merits of our approach:

Original: [Figure] • MoPA-RL [1]: A MP-augmented policy, for learning large displacement actions with MP andsmaller actions with RL.
Revised: • MoPA-RL [1]: A MP-augmented policy, for learning large displacement actions with MP and smaller actions with RL.

Original: This policy also serves as our state-based expert agent.• Behavioral Cloning (BC-Visual) :
Revised: This policy also serves as our state-based expert agent. • Visual Behavioral Cloning (BC-Visual) :

Original: A behavioral cloning policy trained on the image-action pairscollected fromthe MoPA-RL trajectories using image rendering.
Revised: A behavioral cloning policy trained on the imageaction pairs collected from MoPA-RL trajectories.

Original: • Asym.
Revised: • Asym.

Original: SAC [4]: An asymmetric actor-critic method, learnt using SAC where the critic is learnt inthe state space and the actor is learnt in the image space.
Revised: SAC [5]: An asymmetric actor-critic method, trained using SAC where the critic is learned in the state space and the actor is learned in the image space.

Original: • MoPA Asym.
Revised: • MoPA Asym.

Original: SAC : MoPA-RL policy [1] learnt using the asymmetric framework.
Revised: SAC : MoPA-RL policy [1] learned using the asymmetric framework.

Original: Note that thismethod still uses a motion planner with an augmented action space.
Revised: Note that this method still uses a motion planner with an augmented action space.

Original: This method is a direct attempt at learning a visual policy using [1] without the second step of our proposed approach.
Revised: This method is a direct attempt at learning a visual policy using a MP-augmented framework [1].

Original: • CoL [26]: A policy learnt using the Cycle-of-Learning framework, which is a state-of-the-artalgorithm for learning from demonstrations (LfD).
Revised: • CoL [29]: A policy learned using the Cycle-of-Learning framework, which is the state-of-the-art algorithm for learning from demonstrations (LfD).

Original: • CoL (w BC Smoothing)
Revised: • CoL (w BC Smoothing)

Original: [26]: A policy similar to CoL [26], with BC for trajectory 
Revised: [29]: A policy similar to CoL [29], with BC for trajectory smoothing.

Original: smoothing.• Ours (w/o BC Smoothing) :
Revised: • Ours (w/o BC Smoothing) :

Original: A policy learnt using ourproposed approach described in Section 4without trajectory smoothing using BC,
Revised: A policy learned using our approach described in Section 4 without BC trajectory smoothing,

Original: i
Revised: i

Original: e. we directly using MoPA-RL trajectories in R e .•
Revised: e., we directly use MoPA-RL trajectories in R e .

Original: Ours : A policy learnt using our proposed approach described in Section 4.

Original: 4.3 Evaluation
Revised: 4.3 Evaluation

Original: We compare ourproposed approach against baselines on the following evaluation metrics averagedover ﬁve random seeds and 100 unseen episodes:
Revised: We compare our approach against baselines on the following evaluation metrics averaged over ﬁve random seeds and 100 unseen episodes per seed:

Original:  • Average Success Rate (ASR): For all the baselines, we report the average success rate as the meanof the successful episodes with 100 unseen episodes per random seed.
Revised: [Figure] • Average Success Rate (ASR) is the average number of successful episodes.

Original: • Average Episode Length (AEL): We compare baselines on average length of successful episodes.• Average Discounted Return (ADR):
Revised: • Average Episode Length (AEL) is the average length of successful episodes.

Original: We report average discounted sum of rewards(cid:80) T − 1 t =

Revised: • Average Discounted Return (ADR) is the average discounted sum of rewards (cid:80) T −t =

Original: γ t R ( s t , a t ) , with T being  episode horizon.
Revised: γ t R ( s t , a t ) , with T being the episode horizon.

Original: An episode completed in more time steps haslower discounted reward, due to exponentially decaying weight of the discount factor ( γ = 0 . 99 ).
Revised: An episode completed in more time steps has a lower discounted reward, due to exponentially discounted reward with γ = 0 . 99 .

Original: Comparison with baselines: Figure 3 compares the learning curves of our method against otherbaselines. It summarizes the performance using discounted rewards against the number of trainingsteps.
Revised: Comparisons with baselines Figure 3 illustrates the learning curves of our method and all other baselines using discounted rewards against the number of training steps.

Original: As  the trend shows, our method is far more sample-efﬁcient and outperforms all the baselines.
Revised: As per the trend, our method is far more sample-efﬁcient and outperforms all the baselines.

Original: Directly learning a visual policy using Asym.SAC [4] does not completely learn any task. Although Pinto et al.
Revised: Asym.

Original: [4] proposed their framework on solving manipulation tasks in a controlled environment,it struggles to  learn a visual policy  in obstructed environments, where exploration is hard.
Revised: SAC [5] fails to directly learn a visual policy for Sawyer Lift and Sawyer Assembly in obstructed environments, where exploration is hard.

Original: We alsoshow that MoPA Asym.
Revised: Moreover, MoPA Asym.

Original: SAC,  a direct attempt at learning a visual policy whilestill using MP does notsuccessfully learn to solve the tasks.
Revised: SAC, which is a direct attempt at learning a visual policy while using MP, does not successfully learn to solve the tasks.

Original: CoL [26] which has a straightforward combination ofBC losswith RL  objective for actor optimization receives partial rewards for some tasks after around 1.5-2Msteps, much later than our method’s convergence but does not achieve a high success rate.
Revised: CoL [29] which has a straightforward combination of RL and BC objective for actor optimization receives partial rewards for some tasks at around 1.5-2M steps, much slower than our method’s convergence.

Original: Moreover,our method without BC smoothing has a higher variance across different seeds and converges slower
Revised: As reported in Table 1, BC achieves decent performance for Sawyer Push and Assembly, but not for the Sawyer Lift task. However, it achieves high AEL and low ADR for all tasks, showing its inefﬁciency to solve the task fast.

Original: [Figure]
Revised: This is because BC is bound to perform as good as the expert demonstrations (here from MoPA-RL) at its best.

Original: [Table]
Revised: In contrast, our method optimizes trajectories beyond expert signals and achieves signiﬁcantly lower AEL and higher ADR compared to baselines.

Original: [Figure] [Table] than our method for two tasks.
Revised: Ablation on BC trajectory smoothing Our method without BC smoothing has higher variance across seeds and converges slower than our method for two tasks (see Figure 3).

Original: For Sawyer Assembly it does not learn to solve the task at all.
Revised: For Sawyer Assembly, it does not learn to solve the task at all.

Original: Thisshows the importance of trajectory smoothing in our method. As reported in Table 1, BC achievesdecent performance for Sawyer Push and Assembly but does not work well for Lift task. However,it has signiﬁcantly high AEL and low ADR for all tasks, showing its inefﬁciency to solve the taskfast.
Revised: In short, BC smoothing of the motion planner trajectories is important for our method to work across all environments.

Original: This is becauseBC is bound to perform as good as the demonstrations (here collected using
Revised: This is because the motion planner based trajectories are usually jittery and non-smooth.

Original: MoPA-RL) at its best. In contrast, our method is able to optimize trajectories beyond expert signalsand has signiﬁcantly lower AEL and higher ADR compared to all other baselines for all tasks.
Revised: Using behavioral cloning reﬁnes each transition, thereby making the entire trajectories much smoother and help in efﬁciently training the RL agent.

Original: Ablation study for weight initialization: We compare our method with and without actor-critic initialization  and do not observe any episode success until 1.2M environment steps for the latter.
Revised: Ablation on weight initialization We compare our method with and without actor-critic initialization in appendix, Figure ?? and do not observe any episode success until 1.2M environment steps for the latter.

Original: Thisshows the importance of initializing actor and critic for successful training in the visual domain.
Revised: This shows the importance of our proposed weight initialization for learning in the visual domain.

Revised: We further elaborate the ablation results in Section ?? .

Revised: Ablation on entropy coefﬁcient tuning We examine the effect of different values of α on the trade-off between entropy maximization and reward maximization for the SAC objective as shown in

Revised: Section ?? .

Revised: Compared to higher α values, smaller values of α improve the sample efﬁciency during visual policy learning.

Revised: Since we utilize prior knowledge from the state-based agent, we use a smaller alpha to exploit the previously acquired knowledge instead of exploring the entire state space again.

Original: 4.4 Policy Transfer to Different Domains
Revised: [Figure] [Figure] [Table] 4.5 Policy Transfer to Different Domains

Original: In this experiment, we learn our policyalong with domain randomization (DR) to verify its domaintransfer capabilities.
Revised: In this experiment [Table] we learn our policy with domain randomization (DR) to verify its domain transfer capabilities.

Original: DR is a promising approach for modelling transferable policies using a multitudeof variations in simulation during training [31, 32], which makes the policy invariant to changes thatare insigniﬁcant for the task.
Revised: DR is a promising approach for modelling transferable policies using a multitude of variations in simulation during training [34, 35], which makes the policy invariant to changes that are insigniﬁcant for the task.

Original: For this work, we randomize the simulation environment using differenttextures, colors, and lighting conditions as shown in Figure 2.
Revised: For this work, we randomize the simulation environment using different textures, colors

Original:  We then test our DR-policy on threeunseen scenarios, as illustrated in Figure 4 comprising of 1) the original environment without anyrandomization; 2) modiﬁed environment 1, with real-world like background distractions such asfurniture, walls, and ﬂoor. We also add unseen distractor cube (blue and magenta) near the target(red) cube; 3) modiﬁed environment 2, which is similar to the previous, but has additional changes inthe texture of the table and box, and different size of the table.
Revised: [Table] and lighting conditions for training as shown in Figure 4 and Figure ?? . We then test our DR policy on three unseen scenarios, as illustrated in Figure 4 comprising of (1) the original environment without any randomization; (2) Scenario 1 with realistic background distractions (e.g., furniture, walls, and ﬂoor) and unseen distractor cubes (blue and magenta) near the target cube (red); and (3) Scenario 2 with additional changes in the texture of the table and box, and different size of the table.

Original: As reported in Table 2, we see that ourpolicy performs signiﬁcantly well in all three testing environments.
Revised: Our policy attains more than 96% ASR in all three unseen scenarios for all the tasks (see Table 2).

Original: This illustrates the robustness ofour proposed framework in learning transferable visual policies in a sample-efﬁcient manner.Wefurther discuss about the key learning and limitations of our work in Section D.
Revised: This illustrates the robustness of our proposed framework in learning transferable visual policies, robust to unseen distractors in a sample-efﬁcient manner.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we introduce an efﬁcient method for learning manipulation tasks in obstructed environments in  visual domain.
Revised: In this paper, we introduce a two-step distillation method for learning manipulation tasks in obstructed environments in the visual domain.

Original: We propose a two-step method where in step one, we learn a  policy instate-space using motion planning and RL. In step two, we use an asymmetric actor-critic frameworkto accelerate visual policy learning, which removes the motion planner and beneﬁts from experiencescollected by state-based agent.
Revised: In step one, we use a MP-augmented RL policy as the state-based expert and subsequently learn a visual BC agent from it, removing the motion planner dependency. In step two, we learn a vision-based agent using an asymmetric actor-critic framework.

Revised: This step is further expedited via proper weight initialization, BC trajectory-guided RL training, and entropy coefﬁcient tuning, making our method highly sample-efﬁcient.

Original: We also show that our visual policy combined with domain randomization is robust under different environment settings with unseen distractors and is competent forzero-shot transfers.
Revised: Our visual policy combined with domain randomization demonstrates successful zero-shot transfer to unseen environments with new visual domains and distractors.

Original: This work motivates us to extend our approach on other challenging tasks toensure generalizability work transfer policy  real robots.
Revised: Beyond zero-shot transfer, ﬁne-tuning the policy in the real world by learning a vision-based critic or applying simulation-to-real techniques [36] is our deﬁnite future work to realize simulation-to-real transfer. Acknowledgments This research is supported by the Annenberg Fellowship from USC, NAVER AI Lab, and NSF NRI-2024768.

Revised: We thank our colleagues from the CLVR lab and RESL for the valuable discussions that considerably assisted the research.