Original: Distilling Motion-Planner Augmented Policies into Visual Control Policies for Robot Manipulation
Revised: Distilling Motion-Planner Augmented Policies into Visual Control Policies for Robot Manipulation

Revised: Anonymous Author(s)

Revised: AfﬁliationAddress email

Original: Abstract: We address the problem of learning complex manipulation tasks inobstructed environments from visual observations.
Revised: Abstract: We address the problem of learning complex manipulation tasks inobstructed environments from visual observations.

Original: Learning such tasks directly inthe visual domain is challenging and sample inefﬁcient due to hard exploration inthe presence of obstacles and high-dimensional observation inputs.
Revised: Learning such tasks directly inthe visual domain is challenging and sample inefﬁcient due to hard exploration inthe presence of obstacles and high-dimensional observation inputs.

Original: We proposea framework to learn a visual control policy in a two-step procedure: 1) Learn apolicy with long-horizon planning capabilities in state space using motion planningaugmented reinforcement learning; 2) Distilling this policy into the visual domainwhile removing the dependency on motion planner.
Revised: We proposea framework to learn a visual control policy in a two-step procedure: 1) Learn apolicy with long-horizon planning capabilities in state space using motion planningaugmented reinforcement learning; 2) Distilling this policy into the visual domainwhile removing the dependency on motion planner.

Original: In this step, we leverage theexploration and experiences accumulated by the state agent from the ﬁrst step in anasymmetric actor-critic framework.
Revised: In this step, we leverage theexploration and experiences accumulated by the state agent from the ﬁrst step in anasymmetric actor-critic framework.

Original: We evaluate our method on three manipulationtasks in obstructed environments and compare it to various reinforcement andimitation learning baselines.
Revised: We evaluate our method on three manipulationtasks in obstructed environments and compare it to various reinforcement andimitation learning baselines.

Original: The results indicate that our framework is highlysample-efﬁcient and outperforms state-of-the-art algorithms.
Revised: The results indicate that our framework is highlysample-efﬁcient and outperforms state-of-the-art algorithms.

Original: We also demonstrateour approach’s zero-shot domain transfer capabilities to other environment settingswith unseen distractors by coupling our method with domain randomization.
Revised: We also demonstrateour approach’s zero-shot domain transfer capabilities to other environment settingswith unseen distractors by coupling our method with domain randomization.

Original:  Introduction
Revised:  Introduction

Original: Efﬁciently learning from visual observations has been a long-standing challenge for Reinforcement
Revised: Efﬁciently learning from visual observations has been a long-standing challenge for Reinforcement

Original: Learning (RL) algorithms.
Revised: Learning (RL) algorithms.

Original: More so, it becomes even more difﬁcult when solving complex manipulation tasks in cluttered environments, which require precise object-interactions as well as collision-freemovements around obstacles.
Revised: More so, it becomes even more difﬁcult when solving complex manipulation tasks in cluttered environments, which require precise object-interactions as well as collision-freemovements around obstacles.

Original: This problem can be tackled by combining the strengths of motionplanning (MP) and RL [1, 2, 3] – safe collision-free maneuvers of MP and sophisticated contact-basedinteractions of RL.
Revised: This problem can be tackled by combining the strengths of Motion Planning (MP) and RL [1, 2, 3] – safe collision-free maneuvers of MP and sophisticated contact-basedinteractions of RL.

Original: However, these methods have practical limitations, such as assuming access to thegeometric state of an environment,which is necessary to perform collision checks in MP.
Revised: However, these methods have practical limitations, such as assuming access togeometric state of an environment, necessary to perform collision checks in MP.

Original: Anotherlimitation is the computational cost of motion planners, which makes it difﬁcult to integrate them in areal-time control loop.
Revised: Another limitation isthe computational cost of MP which makes it difﬁcult to integrate them in a real-time control loop.

Original: In this paper we propose a two-step approach to learn a visuomotor policy that overcomes thesedrawbacks.
Revised: In this paper we propose a two-step approach to learn a visuomotor policy that overcomes thesedrawbacks.

Original: We start by training an agent to solve the task in a simulated environment where accessto privileged state information is available.
Revised: We start by training an agent to solve the task in a simulated environment where accessto privileged state information is available.

Original: There, we use Motion Planner Augmented Reinforcement
Revised: There, we use Motion Planner Augmented Reinforcement

Original: Learning (MoPA-RL) [1] to train a policy equipped with a motion planner that can be called to plancollision-free motions and efﬁciently explore the environment.
Revised: Learning (MoPA-RL) [1] to train a policy equipped with a motion planner that can be called toplan collision-free motions and efﬁciently explore the environment.

Original: In the second step, we transfer thisstate-based policy intothe visual domain by distilling the behavior ofthe motion planner into a directfeedback policy, which is the main focus of this paper.
Revised: In the second step, we transferthis state-based policy into visual domain by distilling the behavior of motion planner into a directfeedback policy, which is the main focus of this paper.

Original: The resulting policy will not need expensivecalls tothe motion planner and directly select an action based onthe visual input, which also makesit suitable for sim-to-real transfer.
Revised: The resulting policy will not need expensivecalls to motion planner and directly select an action based on visual input, which also makes it suitablefor sim-to-real transfer.

Original: Directly learningpolicies from raw pixels has been a challengingproblem in RL and even harder for robot learning problems due to the cost and safety concernsinvolved with real robots.
Revised: Directly learning from raw pixels has been a challenging problem in RL, evenharder for robot learning problems due to the cost and safety concerns with real robots.

Original: Therefore, learning transferable policies from simulation to real-worldrobots has signiﬁcant advantages.
Revised: Therefore,learning transferable policies from simulation to real-world robots has signiﬁcant advantages.

Original: Our visual policy distillation is realized by adopting an asymmetric actor-critic architecture similar to
Revised: Our visual policy distillation is realized by adopting an asymmetric actor-critic architecture similar to

Original: Pinto et al.
Revised: Pinto et al.

Original: [4] where the value function is learnt using state information as input whereas the actordirectly learns from image observations.
Revised: [4] where the value function is learnt using state information as input whereas the actordirectly learns from image observations.

Original: This way, our state-based agent’s critic and asymmetriccritic share the same state space, but have different action spaces.
Revised: This way, our state-based agent’s critic and asymmetriccritic share the same state space, but have different action spaces.

Original: Rather than learning from scratch,we leverage the experience of our state agent and critic to accelerate learning a policy in the visualdomain.
Revised: Rather than learning from scratch,we leverage the experience of our state agent and critic to accelerate learning a policy in the visual domain.

Original: We achieve this by: 1)
Revised: We achieve this by: 1)

Original: Initializing the critic network of our asymmetric agent with thelearnt weights of the critic network of MoPA-RL. 2)
Revised: Initializing the critic network of our asymmetric agent with thelearnt weights of the critic network of MoPA-RL. 2)

Original: Leveraging optimal expert trajectories from
Revised: Leveraging optimal expert trajectories from

Original: MoPA-RL and utilizing them for initializing the actor network through pre-training with Behavioral
Revised: MoPA-RL and utilizing them for initializing the actor network through pre-training with Behavioral

Original: Cloning (BC).
Revised: Cloning (BC).

Original: While supervised learning with BC does have less training time, it generally fails in near-collision states due to lack of understanding of appropriate collision thresholds because ofunseen environment interactions [5, 6].
Revised: While supervised learning with BC does have less training time, it generally failsin near-collision states due to lack of understanding of appropriate collision thresholds because ofunseen environment interactions [5, 6].

Original: We mitigate this issue by training our RL agent on top of BC.
Revised: We mitigate this issue by training our RL agent on top of BC.

Original: We employ the Soft-Actor Critic (SAC) [7] algorithm for training the asymmetric agent.
Revised: We employ the Soft-Actor Critic (SAC) [7] algorithm for training the asymmetric agent.

Original: The main contributions of this work are:
Revised: The main contributions of this work are:

Original: A novel framework to learn visual control policies by learning a state-space policy in simulationthat uses a motion planner for efﬁcient exploration in cluttered scenarios and subsequentlytransferring it into the image space while removing the motion planner.2.
Revised: A novel framework to learn visual control policies by learning a state-space policy in simulationthat uses a motion planner for efﬁcient exploration in cluttered scenarios and subsequentlytransferring it into the image space while removing the motion planner.2.

Original: An efﬁcient training algorithm for the domain transfer that includes weight initialization, pretraining via BC and ﬁne-tuning with an asymmetric actor-critic algorithm.3.
Revised: An efﬁcient training algorithm for the domain transfer that includes weight initialization, pretraining via BC and ﬁne-tuning with an asymmetric actor-critic algorithm.3.

Original: The evaluation of this approach on three manipulation tasks in obstructed environments and thecomparison against state-of-the-art imitation and reinforcement learning baselines.4.
Revised: The evaluation of this approach on three manipulation tasks in obstructed environments and thecomparison against state-of-the-art imitation and reinforcement learning baselines.4.

Original: An experiment that demonstrates visual transfer capabilities of our approach by domain randomization of textures, colors and lighting during training.
Revised: An experiment that demonstrates visual transfer capabilities of our approach by domain randomization of textures, colors and lighting during training.

Original: Our method outperforms state-of-the-art algorithms in terms of success rate, sample efﬁciency aswell as average path length for task completion.
Revised: Our method outperforms state-of-the-art algorithms in terms of success rate, sample efﬁciency, andaverage path length.

Original: Interestingly, our method solves tasks much faster,having a signiﬁcantly shorter path length compared to all other baselines, including the BC experttrajectories as well as our state-based MoPA-RL agent.
Revised: Interestingly, our method solves tasks much faster, having a signiﬁcantly shorterpath length compared to all other baselines, including the BC expert trajectories as well as ourstate-based MoPA-RL agent.

Original: This shows that our asymmetric visual agentoptimizes its path beyond the expert signals it receives by self-exploration.
Revised: This shows that our asymmetric visual agent optimizes its path beyondthe expert signals it receives by self-exploration.

Original:  Related Work
Revised:  Related Work

Original: Several recent efforts have been made towards distilling motion planning algorithms into neuralmotion planners using learning-based methods [8, 9, 10, 11, 5].
Revised: Several recent efforts have been made towards distilling motion planning algorithms into neuralmotion planners using learning-based methods [8, 9, 10, 11, 5].

Original: Most of these efforts can be broadlycategorized into imitation learning (IL) and RL paradigms.
Revised: Most of these efforts can be broadlycategorized into imitation learning (IL) and RL paradigms.

Original: Behavioral cloning, being one of the most common frameworks to learn directly from expert demonstrations, has been used to learn neural network based planners [5, 9, 10].
Revised: Behavioral cloning, being one of the most common frameworks to learn directly from expert demonstrations, has been used to learn neural network based planners [5, 9, 10].

Original: The works in [12, 9, 11]also adopt supervised learning over expert demonstrations obtained via sampling-based MP, andmostly focus on learning obstacle representations using sensor-measurements like point-clouds or 2Dlaser scans.
Revised: The works in [12, 9, 11]also adopt supervised learning over expert demonstrations obtained via sampling-based MP, andmostly focus on learning obstacle representations using sensor-measurements like point-clouds or 2Dlaser scans.

Original: However the major drawback of supervised learning approaches is that their performanceis limited by the demonstrator’s performance itself.
Revised: However the major drawback of supervised learning approaches is that their performance is limited by the demonstrator’s performance itself.

Original: Moreover, BC struggles when it encountersdata out of the collected data distribution, due to lack of exploration.
Revised: Moreover, BC struggles when it encountersdata out of the collected data distribution, due to lack of exploration.

Original: A common workaround forimproving BC is to use DAgger [13], which iteratively gathers expert actions for visited states and updates the state-action pairs in the training data accordingly.
Revised: A common workaround forimproving BC is to use DAgger [13], which iteratively gathers expert actions for visited states andupdates the state-action pairs in the training data accordingly.

Original: DAgger has also been used in attemptsto distill MP into neural networks [5] and for high-level planning [14].
Revised: DAgger has also been used in attemptsto distill MP into neural networks [5] and for high-level planning [14].

Original: However, its exploitation ofthe expert for labelling all encountered states can be cumbersome and hard to achieve.
Revised: However, its exploitation ofthe expert for labelling all encountered states can be cumbersome and hard to achieve.

Original: To discover better policies using exploration, various off-policy reinforcement learning methodshave been studied for superseding motion planners with neural network policies.
Revised: To discover better policies using exploration, various off-policy reinforcement learning methodshave been studied for superseding motion planners with neural network policies.

Original: They utilise expertdemonstrations stored in the replay buffer for guided exploration and succeed to learn policies betterthan experts [5, 15, 16].
Revised: They utilise expertdemonstrations stored in the replay buffer for guided exploration and succeed to learn policies betterthan experts [5, 15, 16].

Original: This technique of using expert replay buffers has been derived from popularimitation learning method [17, 18, 19] which were initially introduced on applications outside MP.
Revised: This technique of using expert replay buffers has been derived from popularimitation learning method [17, 18, 19] which were initially introduced on applications outside MP.

Original: Prior works combining RL with expert demonstrations are either focused on unobstructed environments [17, 18, 19, 20] or simpler tasks not involving object manipulation [5].
Revised: Prior works combining RL with expert demonstrations are either focused on unobstructed environments [17, 18, 19, 20] or simpler tasks not involving object manipulation [5].

Original: They also assume fullyobservable environments and learn policies in state space.
Revised: They also assume fullyobservable environments and learn policies in state space.

Original: Moreover, most ofﬂine-RL approaches insparse reward scenarios are coupled with Hindsight Experience Replay (HER) for goal-conditionedpolicies.
Revised: Moreover, most ofﬂine-RL approaches insparse reward scenarios are coupled with Hindsight Experience Replay (HER) for goal-conditionedpolicies.

Original: They empirically show that not using HER signiﬁcantly hurts the agent’s performance andmight even completely fail at complex tasks [5, 4, 17, 16].
Revised: They empirically show that not using HER signiﬁcantly hurts the agent’s performance andmight even completely fail at complex tasks [5, 4, 17, 16].

Original: However, using HER is not applicablefor composite tasks involving goals associated with multiple sequential steps.
Revised: However, using HER is not applicable forcomposite tasks involving goals associated with multiple sequential steps.

Revised: As stated in [21], HERis designed to efﬁciently train a goal-conditioned policy for Multi-goal RL setup, where the policyand critic depend on the state and goal associated with that state.

Original: This makes HERnot suitable for sequential manipulation tasks, where learning from sparse feedback is challenging.
Revised: This makes HER not suitable forsequential manipulation tasks, where learning from sparse feedback is challenging without usinggoal-conditioned policies.

Original: Here, we show that our approach successfully performs composite manipulation tasks in obstructedenvironments withoutexplicitly using HER for tackling sparse rewards.
Revised: Here, we show that our approach successfully performs composite manipulation tasks in obstructed environments without using HER for tackling sparse rewards.

Original: We also learn our policy inthe visual domain keeping in mind its applicability for sim-to-sim and sim-to-real transfer.
Revised: We also learnour policy in visual domain keeping in mind its applicability for sim-to-sim and sim-to-real transfer.

Original:  Method
Revised:  Method

Original: Our goal is to solve complex manipulation tasks in obstructed environments in the visual domain.
Revised: Our goal is to solve complex manipulation tasks in obstructed environments in the visual domain.

Original: While MP-based methods can efﬁciently solve such tasks, they are restricted to state-space learningwhich is difﬁcult to transfer to real environments.
Revised: While MP-based methods can efﬁciently solve such tasks, they are restricted to state-space learningwhich is difﬁcult to transfer to real environments.

Original: Moreover, sampling-based motion planning incurssigniﬁcant computational costs that makes it hard to integrate them in real-time controllers.
Revised: Moreover, sampling-based motion planning incurssigniﬁcant computational costs that makes it hard to integrate them in real-time controllers.

Original: Wepropose a method which moves from ﬁrst learning a state-based agent using MP-based actions tosubsequently learning a visual policy suitable for zero-shot sim-to-sim and sim-to-real transfers.
Revised: Wepropose a method which moves from ﬁrst learning a state-based agent using MP-based actions tosubsequently learning a visual policy suitable for zero-shot sim-to-sim and sim-to-real transfers.

Original: Ourgeneral approach is illustrated in Figure 1.
Revised: Ourgeneral approach is illustrated in Figure 1.

Original: We accelerate visual policy learning by leveraging ourstate-based agent’s experience and therefore increase its sample-efﬁciency.
Revised: We accelerate visual policy learning by leveraging ourstate-based agent’s experience and therefore increase its sample-efﬁciency.

Original: 3.1 Problem Formulation
Revised: 3.1 Problem Formulation

Original: We formulate our problem as a continuous state space Markov Decision Process (MDP) for bothour state-based as well as asymmetric-visual agent.
Revised: We formulate our problem as a continuous state space Markov Decision Process (MDP) for bothour state-based as well as asymmetric-visual agent.

Original: The MP-augmented MDP ˜ M introduced in[1] can be represented as a tuple ( S , A ∪ ˆ A , ˜ P, ˜ R, ρ 0 , γ ) where S is a set of continuous states, Ais a set of continuous actions in the direct action space and
Revised: The MP-augmented MDP ˜ M introduced in[1] can be represented as a tuple ( S , A ∪ ˆ A , ˜ P, ˜ R, ρ 0 , γ ) where S is a set of continuous states, Ais a set of continuous actions in the direct action space and

Original: ˆ A is a set of continuous actions in the
Revised: ˆ A is a set of continuous actions in the

Original: MP-augmented action space for enlarged actions similar to [1].
Revised: MP-augmented action space for enlarged actions similar to [1].

Original: For the rest of the paper, we refer tothis joint action space
Revised: For the rest of the paper, we refer tothis joint action space

Original: A ∪ ˆ A as ˜ A . 1 ˜ P : S × ˜
Revised: A ∪ ˆ A as ˜ A . 1 ˜ P : S × ˜

Original: A × S → R deﬁnes the transition function, ˜ R is thereward function, ρ 0 is the initial state distribution, and γ is the discount factor for ˜ M .
Revised: A × S → R deﬁnes the transition function, ˜ R is thereward function, ρ 0 is the initial state distribution, and γ is the discount factor for ˜ M .

Original: Next, we represent our asymmetric-visual agent’s MDP M by a tuple ( S , O , A , P, R, ρ 0 , γ ) similarto [4].
Revised: Next, we represent our asymmetric-visual agent’s MDP M by a tuple ( S , O , A , P, R, ρ 0 , γ ) similar to[4].

Original: The state space S , initial state distribution ρ 0 , and discount factor γ are the same as in theaugmented MDP ˜ M .
Revised: The state space S , initial state distribution ρ 0 , and discount factor γ are the same as in augmented MDP ˜ M .

Original: However, the action space of M is different to ˜ M and only consists of the directaction space A , which also leads to a different transition function P : S × A × S → R and rewardfunction R .
Revised: However, the action space of M is different to ˜ M and only consists of direct action space A , leading to a different transition function P : S ×A×S → R and reward function R .

Original: Additionally, we assume to have access to a set of continuous partial observations Ocorresponding tothe states in S .
Revised: Additionally,we assume to have access to a set of continuous partial observations O corresponding to states in S .

Revised: Note that the A is a subset of ˜ A – they have the same dimension but the interval bounds are larger for˜ A because it also contains motion planner actions.

Revised: Additionally, the reward and transition functionsof both MDPs are the same for direct actions, but only the transition function ˜ P handles MP actions.

Original: At any time step t , the state agent’s action distribution is represented by the policy π φ (˜ a t | s t ) , with itbeing in state s t ∈ S , taking action ˜ a t ∈ ˜ A , where φ denotes policy parameters.
Revised: At any time step t , the state agent’s action distribution is represented by the policy π φ (˜ a t | s t ) , with itbeing in state s t ∈ S , taking action ˜ a t ∈ ˜ A , where φ denotes policy parameters.

Original: For our asymmetricvisual agent, the actor takes in a partially observable state o t corresponding to s t and outputs actiona t ∈ A , represented by the visual policy π θ ( a t | o t ) where θ denote the parameters of the visualpolicy network.
Revised: For our asymmetricvisual agent, the actor takes in a partially observable state o t corresponding to s t and outputs actiona t ∈ A , represented by the visual policy π θ ( a t | o t ) where θ denote the parameters of the visualpolicy network.

Original: The goal of this work is to transfer the behavior of the policy π φ (˜ a t | s t ) to the policyπ θ ( a t | o t ) , which means to transfer a policy into a different state as well as action space.
Revised: The goal of this work is to transfer the behavior of the policy π φ (˜ a t | s t ) to the policyπ θ ( a t | o t ) , which means to transfer a policy into a different state as well as action space.

Original: The learned critic network Q ψ : S × ˜
Revised: The learnedcritic network Q ψ : S × ˜

Original: A → R is shared between the two MDP’s, which is possible since theaction space of the asymmetric agent
Revised: A → R is shared between the two MDP’s, which is possible since theaction space of the asymmetric agent

Original: A is a subset of the action space of the state-based agent ˜ A .
Revised: A is a subset of the action space of the state-based agent ˜ A .

Original: At every time step t , the agent receives a reward r t =
Revised: At every time step t , the agent receives a reward r t =

Original: R ( s t , a t ) , with the state transitioning to s t +1 .
Revised: R ( s t , a t ) , with the state transitioning to s t +1 .

Original: The goal of the agent is to maximize the expected returns where the return is deﬁned as the discountedsum of rewards (cid:80) T − 1 t =

Revised: The goal of the agent is to maximize the expected returns where the return is deﬁned as the discountedsum of rewards (cid:80) T − 1 t =

Original: γ t R ( s t , a t ) , with T depicting the episode horizon.
Revised: γ t R ( s t , a t ) , with T depicting the episode horizon.

Original: 3.2 Motion Planner-Augmented RL Policy as Expert
Revised: 3.2 Motion Planner-Augmented RL Policy as Expert

Original: In the ﬁrst step of our approach, we learn a state-based expert policy using the MoPA-RL framework [1].
Revised: In the ﬁrst step of our approach, we learn a state-based expert policy using the MoPA-RL framework [1].

Original: This framework learns a policy π φ (ˆ a t | s t ) on the augmented MDP ˜ M with the Soft Actor
Revised: This framework learns a policy π φ (ˆ a t | s t ) on the augmented MDP ˜ M with the Soft Actor

Original: Critic (SAC) algorithm.
Revised: Critic (SAC) algorithm.

Original: Precisely, the policy predicts actions deﬁned as the joint displacements∆ q t for the robot in each state.
Revised: Precisely, the policy predicts actions deﬁned as the joint displacements∆ q t for the robot in each state.

Original: Based on the action magnitude, if it lies within the original actionspace A , this action is directly executed by the controller, similar to any RL policy.
Revised: Based on the action magnitude, if it lies within the original actionspace A , this action is directly executed by the controller, similar to any RL policy.

Original: This usuallyaids in performing contact-rich manipulation tasks with high ﬁdelity.
Revised: This usuallyaids in performing contact-rich manipulation tasks with high ﬁdelity.

Original: When the actions are larger inmagnitude, lying in MP-augmented action space ˜ A , this often increases the probability of collisionsin the presence of obstacles.
Revised: When the actions are larger inmagnitude, lying in MP-augmented action space ˜ A , this often increases the probability of collisionsin the presence of obstacles.

Original: Therefore, the sampling-based motion planner RRT-Connect [21] iscalled to realize actions associated with large joint displacements, efﬁciently planning and exploringobstructed environments.
Revised: Therefore, the sampling-based motion planner RRT-Connect [22] iscalled to realize actions associated with large joint displacements, efﬁciently planning and exploringobstructed environments.

Original: The results in [1] show that this policy structure leads to a more efﬁcientexploration of the state space and a faster convergence of the learning.
Revised: The results in [1] show that this policy structure leads to a more efﬁcientexploration of the state space and a faster convergence of the learning.

Original: However, the usage of amotion planner inside an RL policy also has certain weaknesses like high computational costs and thedifﬁculty to transfer it to real scenarios where the environment state is not fully observable.
Revised: However, the usage of amotion planner inside an RL policy also has certain weaknesses like high computational costs and thedifﬁculty to transfer it to real scenarios where the environment state is not fully observable.

Original: Therefore,we propose a two-step learning procedure in which we start with MoPA-RL and add a subsequentstep that transfers the learnt policy into the visual domain while removing its dependency on the MP.
Revised: Therefore,we propose a two-step learning procedure in which we start with MoPA-RL and add a subsequentstep that transfers the learnt policy into the visual domain while removing its dependency on the MP.

Original: 3.3 Learning a Direct Policy in the Visual Domain
Revised: 3.3 Learning a Direct Policy in the Visual Domain

Original: The second step of our approach focuses on moving from a state-based MP-augmented policy to animage-based direct policy (without MP) leveraging demonstrations from our MoPA-RL policy asthe expert.
Revised: The second step of our approach focuses on moving from a state-based MP-augmented policy to animage-based direct policy (without MP) leveraging demonstrations from our MoPA-RL policy asthe expert.

Original: This stage has two advantages: 1) The learnt policy no longer depends on a samplingbased motion planner for planning collision-free paths in obstructed environments.
Revised: This stage has two advantages: 1) The learnt policy no longer depends on a samplingbased motion planner for planning collision-free paths in obstructed environments.

Original: This makesthe policy much faster during learning and inference as MP can be computationally expensive inhigh-dimensional spaces and slow due to performing collision checks.
Revised: This makesthe policy much faster during learning and inference as MP can be computationally expensive inhigh-dimensional spaces and slow due to performing collision checks.

Original: The learnt policy directlyoperates on environment images, not requiring state information and thus making it adequate forsim-to-sim and sim-to-real transfers.
Revised: The learnt policy directlyoperates on environment images, not requiring state information and thus making it adequate forsim-to-sim and sim-to-real transfers.

Original: We now describe different components of our approach in detail.
Revised: We now describe different components of our approach in detail.

Original: 3.3.1 Trajectory Smoothing using Behavioral Cloning
Revised: 3.3.1 Trajectory Smoothing using Behavioral Cloning

Original: Our MoPA-RL expert policy is learnt with access to complete information about the environmentstates at all times and also utilizes MP calls for executing large action steps without collisions.
Revised: Our MoPA-RL expert policy is learnt with access to complete information about the environmentstates at all times and also utilizes MP calls for executing large action steps without collisions.

Original: Weaim to utilise expert demonstrations from this planner-based policy and distill it into a RL-basedvisual policy, thereby deducting expensive MP computations as well training the actor with onlyenvironment images.
Revised: Weaim to utilise expert demonstrations from this planner-based policy and distill it into RL-based visualpolicy, thereby deducting expensive MP computations and training the actor with only environmentimages.

Original: We collect multiple transitions t i intothe MoPA-RL demonstration dataset D mp =
Revised: We collect multiple transitions t i into MoPA-RL demonstration dataset D mp =

Original: { t 1 , t 2 , t 3 , ... } , where t i =
Revised: { t 1 , t 2 , t 3 , ... } ,where t i =

Original: ( s i , o i , a i , r i , s i +1 ) .
Revised: ( s i , o i , a i , r i , s i +1 ) .

Revised: The transitions from MoPA-RL are collected in the low-level (direct)action space.

Original: Note that in order to learn a vision-basedpolicy in the subsequent step, we render image observations o i corresponding to each environmentstate s iin the simulator during ofﬂine data collection.
Revised: Note that in order to learn a vision-based policy in the subsequent step, we render imageobservations o i corresponding to each environment state s i during ofﬂine data collection.

Original: While MP-based methods can be effective for planning collision-free trajectories in obstacle-proneenvironments, their paths can be non-smooth [22].
Revised: While MP-based methods can be effective for planning collision-free trajectories in obstacle-proneenvironments, their paths can be non-smooth [23].

Original: This makes it hard for the robot to learn consistent,naturally smooth motion.
Revised: This makes it hard for the robot to learn consistent, naturally smooth motion.

Original: Therefore, we ﬁrst learn a BC agent using observation-action pairs ( o i , a i )from the dataset D mp by minimizing the mean squared error.
Revised: Therefore, we ﬁrst learn a BC agent using observation-action pairs ( o i , a i )from the dataset D mp by minimizing the mean squared error.

Original: The BC agent shares the same networkarchitecture and parameters that we use for our subsequent asymmetric actor π θ ( a t | o t ) .
Revised: The BC agent shares the same networkarchitecture and parameters that we use for our subsequent asymmetric actor π θ ( a t | o t ) .

Original: We observethat distilling the MoPA-RL trajectories using BC smooths out the rollouts.
Revised: We observethat distilling the MoPA-RL trajectories using BC smooths out the rollouts.

Original: We then collect multiplerollouts by interacting with the environment using our BC policy and save the reﬁned transitions( s i , o i , a i , r i , s i +1 ) in an augmented expert replay buffer R e .
Revised: We then collect multiplerollouts by interacting with the environment using our BC policy and save the reﬁned transitions( s i , o i , a i , r i , s i +1 ) in an augmented expert replay buffer R e .

Original: The main purpose of using R e is topermanently keep hold of expert trajectories during training to facilitate guided exploration.
Revised: The main purpose of using R e is topermanently keep hold of expert trajectories during training to facilitate guided exploration.

Original: Thisidea was initially proposed in [18] for discrete action space domains.
Revised: Thisidea was initially proposed in [18] for discrete action space domains.

Original: In our experiments in Section 4,we show that trajectory smoothing for R e is crucial for efﬁciently learning a successful policy.
Revised: In our experiments in Section 4,we show that trajectory smoothing for R e is crucial for efﬁciently learning a successful policy.

Original: 3.3.2 Learning a Visual Policy with Asymmetric Actor-Critic
Revised: 3.3.2 Learning a Visual Policy with Asymmetric Actor-Critic

Original: To train our RL agent, we adopt an asymmetric actor-critic architecture, similar to [4], where inputsto the critic are full environment states and the actor is learnt with environment images and robotjoint states.
Revised: To train our RL agent, we adopt an asymmetric actor-critic architecture, similar to [4], where inputsto the critic are full environment states and the actor is learnt with environment images and robotjoint states.

Original: The choice of this method is motivated by learning transferable visual policies, which donot require state information during inference, and also beneﬁt from access to environment states,when learned in simulation.
Revised: The choice of this method is motivated by learning transferable visual policies, which donot require state information during inference, and also beneﬁt from access to environment states,when learned in simulation.

Original: Our training procedure comprises of the following components:
Revised: Our training procedure comprises of the following components:

Original: Initializing actor and critic networks:
Revised: Initializing actor and critic networks:

Original: Initializing network architectures with suitable weights isa promising method for accelerating reinforcement learning [23, 24].
Revised: Initializing network architectures with suitable weightsis a promising method for accelerating RL [24, 25].

Original: Especially, when doing RL overpixels, meaningful initializations can signiﬁcantly reduce sample-efﬁciency [20, 25].
Revised: Especially with RL over pixels, meaningfulinitializations can signiﬁcantly improve sample-efﬁciency [20, 26].

Original: We propose toutilize our state-based agent’s experience for initializing our asymmetric visual agent’s networks tobring their initial distributions closer to expert signals.
Revised: We propose to utilize our statebased agent’s experience to initialize our asymmetric visual agent’s networks to bring their initialdistributions closer to expert signals.

Original: The critic Q ψ ( s t , a t ) is initialized withtheweights of the MoPA-RL’s critic network.
Revised: The critic Q ψ ( s t , a t ) is initialized with MoPA-RL’s criticweights.

Original: Note that the action space ofthe two critics is different,but the initialization is still feasible since the action space of MoPA-RL is a superset of the directaction space.
Revised: Note that the action space of two critics is different, but initialization is still feasible sincethe action space of MoPA-RL is a superset of direct action space.

Original: For the actor network π θ ( a t | o t ) , we initialize it withthe weights of the BC agentthatwe learned for trajectory smoothing (see Section 3.3.1).
Revised: For the actor network π θ ( a t | o t ) ,we initialize it with weights of the BC agent learned for trajectory smoothing (Section 3.3.1).

Original: Main training phase: After initializing the actor and critic networks, we initiate the agent’s onlineinteractions with the environment.
Revised: Main training phase: After initializing the actor and critic networks, we initiate the agent’s onlineinteractions with the environment.

Original: The initial policy rollouts are collected into the agent’s replaybuffer R π .
Revised: The initial policy rollouts are collected into the agent’s replaybuffer R π .

Original: Further, we adopt a ﬁxed sampling ratio to sample 1:3 transitions from R e and R πrespectively, for each training iteration.
Revised: Further, we adopt a ﬁxed sampling ratio to sample 1:3 transitions from R e and R πrespectively, for each training iteration.

Original: This ensures that the agent continually improves in an onlinesetting with its own experiences as well as has certain expert transitions to guide exploration.
Revised: This ensures that the agent continually improves in an onlinesetting with its own experiences as well as has certain expert transitions to guide exploration.

Original: Fortraining our RL agent, we adopt Soft Actor-Critic (SAC) [7], an off-policy model-free RL algorithmfor continuous control.
Revised: Fortraining our RL agent, we adopt Soft Actor-Critic (SAC) [7], an off-policy model-free RL algorithmfor continuous control.

Original: We optimize our asymmetric visual policy π θ by maximizing the objective:
Revised: We optimize our asymmetric visual policy π θ by maximizing the objective:

Original: [Equation]
Revised: [Equation]

Original: where the temperature parameter α is the coefﬁcient of the entropy function H which regulates theexploration versus exploitation trade-off during learning.
Revised: where the temperature parameter α is the coefﬁcient of the entropy function H which regulates theexploration versus exploitation trade-off during learning.

Original: The performance of SAC is known to be sensitive with respect to α [26].
Revised: The performance of SAC is known to be sensitive with respect to α [27].

Original: α controls the relativeweight of increasing entropy against reward maximization objective.
Revised: α controls the relativeweight of increasing entropy against reward maximization.

Original: Since our asymmetric agentalready starts from experience collected using state-based agent for efﬁcient exploration, we initializethe αparameter to values lower than the ﬁnal α value obtained from our state-based agent.
Revised: Since our asymmetric agent already startsfrom experience collected using state-based agent, we initialize α to values lower than the ﬁnal αobtained from our state-based agent.

Original: This isto ensure that withall the prior knowledge inthe state-space, the asymmetric visual agent furtherfocuses more on maximizing rewards rather than maximizing entropy for exploration.
Revised: This is to ensure that with prior knowledge in state-space, theasymmetric visual agent focuses more on maximizing rewards rather than maximizing entropy forexploration.

Original: We describethe implementation details andthe choice of hyperparamters inthe supplementary.
Revised: We describe the implementation details and choice of hyperparameters in supplementary.

Original:  Experiments
Revised:  Experiments

Original: Our experiments are motivated to answer the following questions: (1) Does IL efﬁciently learnpolicies for obstructed environments in image space?
Revised: Our experiments are motivated to answer the following questions: (1) Does IL efﬁciently learnpolicies for obstructed environments in image space?

Original: Moreover, is naively combining IL with RLsufﬁcient for solving complex tasks?
Revised: Moreover, is naively combining IL with RLsufﬁcient for solving complex tasks?

Original: (2) Is learning a visual policy hard using MP with RL?
Revised: (2) Is learning a visual policy hard using MP with RL?

Original: (3) Doesour proposed approach efﬁciently learn a visual policy using prior state-based exploration knowledge?
Revised: (3) Doesour proposed approach efﬁciently learn a visual policy using prior state-based exploration knowledge?

Original: Also, is our learnt visual policy capable of domain transfer and robust to unseen distractors?
Revised: Also, is our learnt visual policy capable of domain transfer and robust to unseen distractors?

Original: 4.1 Environments
Revised: 4.1 Environments

Original: We evaluate our approach on four obstructed environments, posing hard exploration challengessimulated using the MuJoCo physics engine [27].
Revised: We evaluate our approach on three obstructed environments, posing hard exploration challengessimulated using the MuJoCo physics engine [28].

Original: • Sawyer Push: A Rethink Sawyer robot arm with 7 DoF, initially located outside the box needs toreach an object placed inside the box and push it to the goal location within the box. • Sawyer Lift: The agent needs to reach an object placed inside a box, grasp and lift it outside the box avoiding collisions.
Revised: We use a 32x32 image as a visual observation.

Original: The Sawyer’s arm is always located outside the box as an initial setup. • Sawyer Assembly: The goal for the Sawyer agent is to assemble the fourth leg of a table into thevacant hole in the presence of the other three legs being already assembled, avoiding any collisions
Revised: Thestate information for each environment is described in Section C.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: -RL [1] is the expert agent trained witha state-based policy.
Revised: MoPA-RL [1] is the expert agent trained with state-based policy.

Original: All baselines belowthe horizontal line are trained with asymmetric actor-critic for fair comparison.
Revised: All baselines below horizontal line are trained with asymmetric actor-critic for fair comparison.

Original: Maximum horizonfor successful task completion is 250 across all tasks.
Revised: Maximum horizon is 250 for all tasks.

Original:  with  them.
Revised: • Sawyer Push: A Rethink Sawyer robot arm with 7 DoF, initially located outside the box needs toreach an object placed inside the box and push it to the goal location within the box.• Sawyer Lift: The agent needs to reach an object placed inside a box, grasp and lift it outside thebox avoiding collisions. The Sawyer’s arm is always located outside the box as an initial setup.• Sawyer Assembly: The goal for the Sawyer agent is to assemble the fourth leg of a table into thevacant hole in the presence of the other three legs being already assembled, avoiding any collisionswith them.

Original: As an initial setup, the leg is already attached to the agent’s gripper.
Revised: As an initial setup, the leg is already attached to the agent’s gripper.

Original: This environmentis built upon the IKEA furniture assembly environment [28].
Revised: This environmentis built upon the IKEA furniture assembly environment [29].

Original: Rewards: For Sawyer Push andSawyer Assembly environments, we use sparse rewards, where theagent receives a reward proportional to the distance between the end-effector andthe object, ortheobject and the goal position when distance is less than (cid:15) . 
Revised: Rewards: For Sawyer Push and Assembly environments, we use sparse rewards, where the agentreceives a reward proportional to the distance between the end-effector and object, or the object andgoal position when distance is less than (cid:15) . We use (cid:15) = 0 . 1 for Sawyer Push and (cid:15) = 0 .

Original: For the  Sawyer Lift tasks, we use rewardshaping, similar to [29], where a reward is deﬁned for all three intermediate stages of the task: reach ,grasp and lift, based on the distance threshold (cid:15) .
Revised: 3 for Sawyer Assembly environment while the distance between initial and goal state is around 1.2. An episode isconsidered successful when the distance between the cube (Sawyer Push) and goal or the peg-headand goal (Sawyer Assembly) is less than 0.05. For Sawyer Lift task, we use reward shaping, similarto [30], where a reward is deﬁned for all three intermediate stages of the task: reach , grasp and lift .

Original: A bonus reward signal of (+150) is received uponsuccessful completion of the task  in all the environments.
Revised: A bonus reward signal of (+150) is received upon successful task completion in all the environments.

Original: 4.2 Baselines
Revised: 4.2 Baselines

Original: We compare the following methods as baselines to evaluate the success and merits of our approach:
Revised: We compare the following methods as baselines to evaluate the success and merits of our approach:

Original: • MoPA-RL [1]: A MP-augmented policy, for learning large displacement actions with MP andsmaller actions with RL.

Original: This policy also serves as our state-based expert agent.

Original: Note that this is

Original: the only method in our baselines which learns in the state space.

Original: [Figure]
Revised: [Figure]

Original: d
Revised: • MoPA-RL [1]: A MP-augmented policy, for learning large displacement actions with MP andsmaller actions with RL.

Original: [Figure] w a r d [Figure] • Behavioral Cloning (BC-Visual) :
Revised: This policy also serves as our state-based expert agent.• Behavioral Cloning (BC-Visual) :

Original: A behavioral cloning policy trained on the image-actionpairs collected from the MoPA-RL trajectories using image rendering.
Revised: A behavioral cloning policy trained on the image-action pairscollected from the MoPA-RL trajectories using image rendering.

Original: • Asym.
Revised: • Asym.

Original: SAC [4]: An asymmetric actor-critic method, learnt using SAC where the critic is learntin the state space and the actor is learnt in the image space.• MoPA Asym.
Revised: SAC [4]: An asymmetric actor-critic method, learnt using SAC where the critic is learnt inthe state space and the actor is learnt in the image space. • MoPA Asym.

Original: SAC : MoPA-RL policy [1] learnt using the asymmetric framework.
Revised: SAC : MoPA-RL policy [1] learnt using the asymmetric framework.

Original: Note thatthis method still uses a motion planner with an augmented action space.
Revised: Note that thismethod still uses a motion planner with an augmented action space.

Original: This method is a directattempt at learning a visual policy using [1] without the second step of our proposed approach.• CoL [25]: A policy learnt using the Cycle-of-Learning framework, which is a state-of-the-artalgorithm for learning from demonstrations (LfD).• CoL (w BC Smoothing)
Revised: This method is a direct attempt at learning a visual policy using [1] without the second step of our proposed approach. • CoL [26]: A policy learnt using the Cycle-of-Learning framework, which is a state-of-the-artalgorithm for learning from demonstrations (LfD). • CoL (w BC Smoothing)

Original: [25]: A policy similar to CoL [25], with BC for trajectory smoothing.• Ours (w/o BC Smoothing) :
Revised: [26]: A policy similar to CoL [26], with BC for trajectory smoothing.• Ours (w/o BC Smoothing) :

Original: A policy learnt using our proposed approach described in Section4 without trajectory smoothing using BC,
Revised: A policy learnt using our proposed approach described in Section 4without trajectory smoothing using BC,

Original: i
Revised: i

Original: e. we directly using MoPA-RL trajectories in R e .•
Revised: e. we directly using MoPA-RL trajectories in R e .•

Original: Ours : A policy learnt using our proposed approach described in Section 4.
Revised: Ours : A policy learnt using our proposed approach described in Section 4.

Original: 4.3 Evaluation
Revised: 4.3 Evaluation

Original: We compare our proposed approach against baselines on the following evaluation metrics over threedifferent random seeds and 100 unseen episodes:
Revised: We compare our proposed approach against baselines on the following evaluation metrics averagedover ﬁve random seeds and 100 unseen episodes:

Original: • Average Success Rate (ASR): For all the baselines, we report the average success rate as themean of the successful episodesevaluated over 3 different seeds, with 100 unseen episodes each.• Average Episode Length (AEL): Wealso compareour baselines on the basis of the averagepath length ofthe successful trajectories.• Average Discounted Return (ADR):
Revised: • Average Success Rate (ASR): For all the baselines, we report the average success rate as the meanof the successful episodes with 100 unseen episodes per random seed. • Average Episode Length (AEL): We compare baselines on average length of successful episodes.• Average Discounted Return (ADR):

Original: We reportthe average discounted sum of rewards(cid:80) T − 1 t =

Revised: We report average discounted sum of rewards(cid:80) T − 1 t =

Original: γ t R ( s t , a t ) , with T beingthe episode horizon.
Revised: γ t R ( s t , a t ) , with T being episode horizon.

Original: An episode completed in more numberof steps shall have lower discounted reward, due tothe exponentially decaying weight of thediscount factor  γ with increasing time steps.
Revised: An episode completed in more time steps haslower discounted reward, due to exponentially decaying weight of the discount factor ( γ = 0 . 99 ).

Original: Comparison with baselines: Figure 3 compares the learning curves of our method against otherbaselines.
Revised: Comparison with baselines: Figure 3 compares the learning curves of our method against otherbaselines.

Original: It summarizes the performance using discounted rewards against the number of trainingsteps.
Revised: It summarizes the performance using discounted rewards against the number of trainingsteps.

Original: As the trend shows, our method is extremely sample-efﬁcient and outperforms all the baselines.
Revised: As the trend shows, our method is far more sample-efﬁcient and outperforms all the baselines.

Original: Directly learning a visual policy using Asym.
Revised: Directly learning a visual policy using Asym.

Original: SAC [4] does not begin to learn any of the three tasks.
Revised: SAC [4] does not completely learn any task.

Original: Although Pinto et al.
Revised: Although Pinto et al.

Original: [4] proposed their framework on solving manipulation tasks in a controlledenvironment, it struggles to learn a visual policy in obstructed environments, where exploration ishard.
Revised: [4] proposed their framework on solving manipulation tasks in a controlled environment,it struggles to learn a visual policy in obstructed environments, where exploration is hard.

Original: We also show that MoPA Asym.
Revised: We alsoshow that MoPA Asym.

Original: SAC which is a direct attempt at learning a visual policy whilestill using MP does not successfully learn to solve the tasks.
Revised: SAC, a direct attempt at learning a visual policy while still using MP does notsuccessfully learn to solve the tasks.

Original: CoL [25] which has a straightforwardcombination of BC loss with RL objective foroptimizing the actor starts receiving partial rewardsafter around 1M training steps, much later than our method’s convergence but does not achieve a highsuccess rate.
Revised: CoL [26] which has a straightforward combination of BC losswith RL objective for actor optimization receives partial rewards for some tasks after around 1.5-2Msteps, much later than our method’s convergence but does not achieve a high success rate.

Original: Moreover, our method without BC smoothing has a higher variance across different 
Revised: Moreover,our method without BC smoothing has a higher variance across different seeds and converges slower [Figure] [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: seeds and converges much slower than our method for two tasks.
Revised: than our method for two tasks.

Original: For Sawyer Assembly it does notlearn to solve the task at all.
Revised: For Sawyer Assembly it does not learn to solve the task at all.

Original: This shows the importance of trajectory smoothing in our approach.
Revised: Thisshows the importance of trajectory smoothing in our method.

Original: As reported in Table 1, BC can achieve decent performance for Sawyer Push and Sawyer Assemblybut does not work well forSawyer Lift environment.
Revised: As reported in Table 1, BC achievesdecent performance for Sawyer Push and Assembly but does not work well for Lift task.

Original: However, it has signiﬁcantly high AEL andlow ADR for all tasks, showing its inefﬁciency to solve the task fast.
Revised: However,it has signiﬁcantly high AEL and low ADR for all tasks, showing its inefﬁciency to solve the taskfast.

Original: This is because BC is bound toperform as good as the demonstrations (here collected using MoPA-RL) at its best.
Revised: This is because BC is bound to perform as good as the demonstrations (here collected using MoPA-RL) at its best.

Original: In contrast, ourmethod is able to optimizethe trajectories beyond expert signals and has signiﬁcantly lower AEL andhigher ADR compared to MoPA-RL, BC and other baselines across all tasks.
Revised: In contrast, our method is able to optimize trajectories beyond expert signalsand has signiﬁcantly lower AEL and higher ADR compared to all other baselines for all tasks.

Original: Ablation study for weight initialization: We compare our method with and without actor-criticinitialization and do not observe any episode successup until 1.2M environment steps for the variantwithout actor-critic initialization.
Revised: Ablation study for weight initialization: We compare our method with and without actor-critic initialization and do not observe any episode success until 1.2M environment steps for the latter.

Original: This shows the importance of initializingthe actor as well as the training in the visual domain
Revised: Thisshows the importance of initializing actor and critic for successful training in the visual domain.

Original: 4.4 Policy Transfer to Different Domains
Revised: 4.4 Policy Transfer to Different Domains

Original: environment for evaluating domain transfer with object distractors such as walls, ﬂoor, furniture for

Original: Sawyer Push task.

Original: We also add 2 distractor cubes (blue, magenta) near the target cube (red) which were not seen during training.

Original: In this experiment, we learn our policy along with domain randomization (DR) to verify its domain transfer capabilities.
Revised: In this experiment, we learn our policy along with domain randomization (DR) to verify its domaintransfer capabilities.

Original: DR is apromising approach for modelling transferable policies using a multitude of variations in the simulated images during training [30, 31],which makes the policy invariant to changes in different domainsthat are insigniﬁcant for the task.
Revised: DR is a promising approach for modelling transferable policies using a multitudeof variations in simulation during training [31, 32], which makes the policy invariant to changes thatare insigniﬁcant for the task.

Original: For this work, we randomize thesimulation environment using different textures, colors, and lightingconditions as shown in Figure 2.
Revised: For this work, we randomize the simulation environment using differenttextures, colors, and lighting conditions as shown in Figure 2.

Original: We then test our DR-policy onthe original environment(see Figure 2) without any randomization,which was never seen by the agent during training. We also simulate a different environment setting with real-world like backgrounddistractions such as furniture, walls, ﬂoor as illustrated in Figure 4for evaluating our policy.
Revised: We then test our DR-policy on threeunseen scenarios, as illustrated in Figure 4 comprising of 1) the original environment without anyrandomization; 2) modiﬁed environment 1, with real-world like background distractions such asfurniture, walls, and ﬂoor.

Original: Apart from the red target cube, which ouragent sees during training, we also add two additional cubes (blueand magenta) close to it toevaluate our agent’s robustness in solvingthe task in the presence of distracting objects.
Revised: We also add unseen distractor cube (blue and magenta) near the target(red) cube; 3) modiﬁed environment 2, which is similar to the previous, but has additional changes inthe texture of the table and box, and different size of the table.

Original: As reported in Table 2, we see that our policy performs signiﬁcantlywell in both testing environments.
Revised: As reported in Table 2, we see that ourpolicy performs signiﬁcantly well in all three testing environments.

Original: This illustrates the robustness ofour proposed framework in learning transferable visual policies ina sample-efﬁcient manner.
Revised: This illustrates the robustness ofour proposed framework in learning transferable visual policies in a sample-efﬁcient manner.

Revised: Wefurther discuss about the key learning and limitations of our work in Section D.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we introduce an efﬁcient method for learning manipulation tasks in obstructed environments inthe visual domain.
Revised: In this work, we introduce an efﬁcient method for learning manipulation tasks in obstructed environments in visual domain.

Original: We propose a two-step methodto achieve this where in the ﬁrst step, welearn a policy in the state-space using motion planning with RL.
Revised: We propose a two-step method where in step one, we learn a policy instate-space using motion planning and RL.

Original: In step two, we use an asymmetricactor-critic framework to accelerate visual policy learning, which removes the motion planner andbeneﬁts from the experiences collected bythe state-based agent.
Revised: In step two, we use an asymmetric actor-critic frameworkto accelerate visual policy learning, which removes the motion planner and beneﬁts from experiencescollected by state-based agent.

Original: Since our asymmetric actor does notrequire state-information, we show that our learnt policy combined with domain randomization isrobust under different environment settings with unseen distractors and is competent for zero-shotdomain transfers.
Revised: We also show that our visual policy combined with domain randomization is robust under different environment settings with unseen distractors and is competent forzero-shot transfers.

Original: This workfurther motivates us to extend ourtwo-step approachfrom state to visualpolicy learning on other challengingrobotics and RL tasks to ensure its generalizability. We also
Revised: This work motivates us to extend our approach on other challenging tasks toensure generalizability work transfer policy real robots.