Original: Distilling Motion Planner Augmented Policies into Visual Control Policies for Robot Manipulation
Revised: Distilling Motion Planner Augmented Policies into Visual Control Policies for Robot Manipulation

Original: Abstract: Learning complex manipulation tasks in realistic, obstructed environments is a challenging problem due to hard exploration in the presence of obstacles and high-dimensional visual observations.
Revised: Abstract: Learning complex manipulation tasks in realistic, obstructed environments is a challenging problem due to hard exploration in the presence of obstacles and high-dimensional visual observations.

Original: Prior work tackles the exploration problem by integrating motion planning and reinforcement learning.
Revised: Prior work tackles the exploration problem by integrating motion planning and reinforcement learning.

Original: However, the motion planner augmented policy requires access to state information, which is often not available in the real-world settings.
Revised: However, the motion planner augmented policy requires access to state information, which is often not available in the real-world settings.

Original: To this end, we propose to distill a state-based motion planner augmented policy to a visual control policy via (1) visual behavioral cloning to remove the motion planner dependency along with its jittery motion, and (2) vision-based reinforcement learning with the guidance of the smoothed trajectories from the behavioral cloning agent.
Revised: To this end, we propose to distill a state-based motion planner augmented policy to a visual control policy via (1) visual behavioral cloning to remove the motion planner dependency along with its jittery motion, and (2) vision-based reinforcement learning with the guidance of the smoothed trajectories from the behavioral cloning agent.

Original: We evaluate our method on three manipulation tasks in obstructed environments and compare it against various reinforcement learning and imitation learning baselines.
Revised: We evaluate our method on three manipulation tasks in obstructed environments and compare it against various reinforcement learning and imitation learning baselines.

Original: The results demonstrate that our framework is highly sample-efﬁcient and outperforms the state-of-the-art algorithms.
Revised: The results demonstrate that our framework is highly sample-efﬁcient and outperforms the state-of-the-art algorithms.

Original: Moreover, coupled with domain randomization, our policy is capable of zero-shot transfer to unseen environment settings with distractors.
Revised: Moreover, coupled with domain randomization, our policy is capable of zero-shot transfer to unseen environment settings with distractors.

Original: Code and videos are available at https://clvrai.com/mopa-pd .
Revised: Code and videos are available at https://clvrai.com/mopa-pd .

Original:  Introduction
Revised:  Introduction

Original: Solving complex manipulation tasks in obstructed environments is a challenging problem in deep reinforcement learning (RL) since it requires precise object interactions as well as collision-free movement across obstacles.
Revised: Solving complex manipulation tasks in obstructed environments is a challenging problem in deep reinforcement learning (RL) since it requires precise object interactions as well as collision-free movement across obstacles.

Original: To tackle this problem, prior works [1–3] have proposed to combine the strengths of motion planning (MP) and RL – safe collision-free maneuvers of MP and sophisticated contact-rich interactions of RL, demonstrating promising results.
Revised: To tackle this problem, prior works [1–3] have proposed to combine the strengths of motion planning (MP) and RL – safe collision-free maneuvers of MP and sophisticated contact-rich interactions of RL, demonstrating promising results.

Original: However, MP requires access to the geometric state of an environment for collision checking, which is often not available in the real world and is also computationally expensive for a real-time control.
Revised: However, MP requires access to the geometric state of an environment for collision checking, which is often not available in the real world and is also computationally expensive for a real-time control.

Original: To deploy such agents in realistic settings, we need to resolve the dependency on the state information and costly computation of MP, such that the agent can perform a task in the visual domain.
Revised: To deploy such agents in realistic settings, we need to resolve the dependency on the state information and costly computation of MP, such that the agent can perform a task in the visual domain.

Original: To this end, we propose a two-step distillation framework, motion planner augmented policy distillation (MoPA-PD), that transfers the state-based motion planner augmented RL policy (MoPA-RL [1]) into a visual control policy, thereby removing the motion planner and the dependency on the state information.
Revised: To this end, we propose a two-step distillation framework, motion planner augmented policy distillation (MoPA-PD), that transfers the state-based motion planner augmented RL policy (MoPA-RL [1]) into a visual control policy, thereby removing the motion planner and the dependency on the state information.

Original: Concretely, our framework consists of two stages: (1) visual behavioral cloning (BC [4]) with trajectories collected using the MoPA-RL policy and (2) vision-based RL training with the guidance of smoothed trajectories from the BC policy.
Revised: Concretely, our framework consists of two stages: (1) visual behavioral cloning (BC [4]) with trajectories collected using the MoPA-RL policy and (2) vision-based RL training with the guidance of smoothed trajectories from the BC policy.

Original: The ﬁrst step, visual BC, removes the dependency on the motion planner and the resulting visual BC policy generates smoother behaviors compared to the motion planner’s jittery behaviors.Then, in the second step, we further improve the visual policy [Table] [Table]
Revised: The ﬁrst step, visual BC, removes the dependency on the motion planner and the resulting visual BC policy generates smoother behaviors compared to the motion planner’s jittery behaviors.

Original: [Table]

Original: [Table] [Table] [Table] using asymmetric actor-critic RL [5] directly from the image observations to overcome sub-optimality in the MoPA-RL policy through self-exploration.
Revised: Then, in the second step, we further improve the visual policy using asymmetric actor-critic RL [5] directly from the image observations to overcome sub-optimality in the MoPA-RL policy through self-exploration.

Original: For efﬁcient vision-based RL training, our method leverages the experience of the BC policy and the state-based critic from MoPA-RL by initializing the critic network with the MoPA-RL critic and the actor network through BC pre-training; BC-trajectory guided RL training using a separate expert replay buffer; tuning the entropy coefﬁcient to encourage exploitation (i.e., maximizing reward) over exploration (i.e., maximizing entropy).
Revised: For efﬁcient vision-based RL training, our method leverages the experience of the BC policy and the state-based critic from MoPA-RL by initializing the critic network with the MoPA-RL critic and the actor network through BC pre-training; BC-trajectory guided RL training using a separate expert replay buffer; tuning the entropy coefﬁcient to encourage exploitation (i.e., maximizing reward) over exploration (i.e., maximizing entropy).

Original: In summary, our contributions are as follows:
Revised: In summary, our contributions are as follows:

Original: • We propose a novel framework to learn a visual control policy in cluttered scenarios by distilling a state-space policy that uses a motion planner into an image-space policy by removing the motion planner dependency.
Revised: • We propose a novel framework to learn a visual control policy in cluttered scenarios by distilling a state-space policy that uses a motion planner into an image-space policy by removing the motion planner dependency.

Original: • Our asymmetric visual policy leaning ensures high sample-efﬁciency using weight initialization followed by BC trajectory-guided RL with entropy coefﬁcient tuning.
Revised: • Our asymmetric visual policy leaning ensures high sample-efﬁciency using weight initialization followed by BC trajectory-guided RL with entropy coefﬁcient tuning.

Original: • The distilled visual policy is capable of zero-shot domain transfer to unseen environments with visual domain randomization during training.
Revised: • The distilled visual policy is capable of zero-shot domain transfer to unseen environments with visual domain randomization during training.

Original: • Our method outperforms the state-of-the-art methods in terms of success rate, sample efﬁciency, and path length on three manipulation tasks in obstructed environments.
Revised: • Our method outperforms the state-of-the-art methods in terms of success rate, sample efﬁciency, and path length on three manipulation tasks in obstructed environments.

Original:  Related Work
Revised:  Related Work

Original: The objective of our work is to learn complex manipulation tasks in realistic and obstructed environments.
Revised: The objective of our work is to learn complex manipulation tasks in realistic and obstructed environments.

Original: Motion planner augmented RL methods [1–3] have shown promising results in solving complex tasks in obstructed environments by combining MP and RL.
Revised: Motion planner augmented RL methods [1–3] have shown promising results in solving complex tasks in obstructed environments by combining MP and RL.

Original: However, these methods cannot be deployed in real-world settings due to their dependency on state information.
Revised: However, these methods cannot be deployed in real-world settings due to their dependency on state information.

Original: To remove this dependency, state-based policies must be transferred to the visual domain.
Revised: To remove this dependency, state-based policies must be transferred to the visual domain.

Original: A typical approach to distill state-based policies into vision-based policies is behavioral cloning (BC [4]).
Revised: A typical approach to distill state-based policies into vision-based policies is behavioral cloning (BC [4]).

Original: However, BC often suffers when encountering states unseen during training [6].
Revised: However, BC often suffers when encountering states unseen during training [6].

Original: Therefore, prior works have proposed various methods to improve BC agents using ofﬂine RL [7] and online
Revised: Therefore, prior works have proposed various methods to improve BC agents using ofﬂine RL [7] and online

Original: RL [8].
Revised: RL [8].

Original: However, these works do not involve the distillation of motion planners.
Revised: However, these works do not involve the distillation of motion planners.

Original: Several recent efforts have been made towards distilling MP algorithms into neural motion planners using learning-based methods [9–13].
Revised: Several recent efforts have been made towards distilling MP algorithms into neural motion planners using learning-based methods [9–13].

Original: Most of these efforts can be broadly categorized into imitation learning (IL) and RL paradigms.
Revised: Most of these efforts can be broadly categorized into imitation learning (IL) and RL paradigms.

Original: For IL, supervised learning on MP trajectories has been used to learn neural network-based planners [10–14].
Revised: For IL, supervised learning on MP trajectories has been used to learn neural network-based planners [10–14].

Original: However, the performance of such supervised learning approaches are limited by the collected dataset and demonstrator’s performance.
Revised: However, the performance of such supervised learning approaches are limited by the collected dataset and demonstrator’s performance.

Original: To discover better policies with additional interactions, various off-policy RL methods have been studied for superseding motion planners with neural network policies [13, 15, 16].
Revised: To discover better policies with additional interactions, various off-policy RL methods have been studied for superseding motion planners with neural network policies [13, 15, 16].

Original: These approaches utilize expert trajectories stored in the replay buffer for guided exploration, which leads to better policies than experts [17–20].
Revised: These approaches utilize expert trajectories stored in the replay buffer for guided exploration, which leads to better policies than experts [17–20].

Original: However, these works learn either in unobstructed environments [17– 20] or on simpler tasks not involving object manipulation [13].
Revised: However, these works learn either in unobstructed environments [17– 20] or on simpler tasks not involving object manipulation [13].

Original: They also assume fully observable environments and learn policies in state space.
Revised: They also assume fully observable environments and learn policies in state space.

Original: In this work, we focus on complex manipulation tasks in obstructed environments using visual observations.
Revised: In this work, we focus on complex manipulation tasks in obstructed environments using visual observations.

Original: Moreover, most ofﬂine RL approaches [21, 22] in sparse reward scenarios are coupled with hindsight experience replay (HER [23]).
Revised: Moreover, most ofﬂine RL approaches [21, 22] in sparse reward scenarios are coupled with hindsight experience replay (HER [23]).

Original: They empirically show that not using HER signiﬁcantly hurts the agent’s performance [13, 5, 16, 17].
Revised: They empirically show that not using HER signiﬁcantly hurts the agent’s performance [13, 5, 16, 17].

Original: However, HER is designed to efﬁciently train goal-conditioned policies for multi-goal RL, which makes it unsuitable for tasks with multiple sequential steps without explicitly conditioning on goals.
Revised: However, HER is designed to efﬁciently train goal-conditioned policies for multi-goal RL, which makes it unsuitable for tasks with multiple sequential steps without explicitly conditioning on goals.

Original: In contrast, our method successfully learns composite manipulation tasks in cluttered environments without using HER for tackling sparse rewards.
Revised: In contrast, our method successfully learns composite manipulation tasks in cluttered environments without using HER for tackling sparse rewards.

Original:  Method
Revised:  Method

Original: Our goal is to solve complex manipulation tasks in obstructed environments with visual inputs.
Revised: Our goal is to solve complex manipulation tasks in obstructed environments with visual inputs.

Original: While
Revised: While

Original: MP-based methods can efﬁciently solve such tasks, they are restricted to learning a state-based policy which is difﬁcult to transfer to real environments.
Revised: MP-based methods can efﬁciently solve such tasks, they are restricted to learning a state-based policy which is difﬁcult to transfer to real environments.

Original: Moreover, sampling-based motion planning incurs signiﬁcant computational costs, making it hard to integrate them into real-time controllers.
Revised: Moreover, sampling-based motion planning incurs signiﬁcant computational costs, making it hard to integrate them into real-time controllers.

Original: Thus, we propose a method that distills an MP-augmented state-based agent into a visual policy, removing the
Revised: Thus, we propose a method that distills an MP-augmented state-based agent into a visual policy, removing the

Original: MP and state dependency.
Revised: MP and state dependency.

Original: We formally deﬁne our problem and introduce the MP-augmented RL in
Revised: We formally deﬁne our problem and introduce the MP-augmented RL in

Original: Section 3.1.
Revised: Section 3.1.

Original: Then, we describe our two-step visual distillation approach in Section 3.2.
Revised: Then, we describe our two-step visual distillation approach in Section 3.2.

Original: 3.1 Preliminaries
Revised: 3.1 Preliminaries

Original: Problem formulation We formulate our problem as a Markov Decision Process (MDP) M deﬁned by a tuple ( S , O , A , P, R, ρ 0 , γ ) that consists of a state space S , partial observation space O (visual inputs corresponding to states), action space A , transition function P : S ×
Revised: Problem formulation We formulate our problem as a Markov Decision Process (MDP) M deﬁned by a tuple ( S , O , A , P, R, ⇢ 0 ,  ) that consists of a state space S , partial observation space O (visual inputs corresponding to states), action space A , transition function P : S ⇥

Original: A × S → R , reward function R , initial state distribution ρ 0 , and discount factor γ .
Revised: A ⇥ S ! R , reward function R , initial state distribution ⇢ 0 , and discount factor  .

Original: The agent is represented as a policy π ( a t | o t ) , which takes an action a t under a visual observation o t and receives a reward r t =
Revised: The agent is represented as a policy ⇡ ( a t | o t ) , which takes an action a t under a visual observation o t and receives a reward r t =

Original: R ( s t , a t ) , with the state transitioning to s t +1 .
Revised: R ( s t , a t ) , with the state transitioning to s t +1 .

Original: The goal of the agent is to maximize the expected discounted sum of rewards E ( s,a ) ∼ π (cid:80) T − 1 t =

Revised: The goal of the agent is to maximize the expected discounted sum of rewards E ( s,a ) ⇠ ⇡ P T  1 t =

Original: γ t r t , where T is the episode horizon.
Revised:  t r t , where T is the episode horizon.

Original: Motion Planner-Augmented RL (MoPA-RL) To tackle manipulation tasks in cluttered environments, Yamada et al.
Revised: Motion Planner-Augmented RL (MoPA-RL) To tackle manipulation tasks in cluttered environments, Yamada et al.

Original: [1] proposed to incorporate MP and RL via an MP-augmented MDP ˜
Revised: [1] proposed to incorporate MP and RL via an MP-augmented MDP ˜

Original: M = ( S , ˜ A , ˜ P, ˜ R, ρ 0 , γ ) that consists of the state space S , enlarged action space ˜
Revised: M = ( S , ˜ A , ˜ P, ˜ R, ⇢ 0 ,  ) that consists of the state space S , enlarged action space ˜

Original: A augmenting the direct action space A with the MP action space ˆ A , 5 augmented transition function ˜ P : S× ˜ A×S
Revised: A augmenting the direct action space A with the MP action space ˆ A , 5 augmented transition function ˜ P : S ⇥ ˜ A ⇥ S !

Original: → R , augmented reward function ˜ R ( s, ˜ a ) , initial state distribution ρ 0 , and discount factor γ .
Revised: R , augmented reward function ˜ R ( s, ˜ a ) , initial state distribution ⇢ 0 , and discount factor  .

Original: MoPA-RL [1] learns a policy π φ (˜ a t | s t ) on the augmented MDP ˜ M with an off-policy RL algorithm,
Revised: MoPA-RL [1] learns a policy ⇡  (˜ a t | s t ) on the augmented MDP ˜ M with an off-policy RL algorithm,

Original: SAC [24].
Revised: SAC [24].

Original: Precisely, given a state s t , the policy predicts an action ˜ a t , which is deﬁned as a robot joint displacement ∆ q t .
Revised: Precisely, given a state s t , the policy predicts an action ˜ a t , which is deﬁned as a robot joint displacement  q t .

Original: If the action lies within the direct action space A , it is directly executed by the controller as the agent performs sophisticated and contact-rich manipulations.
Revised: If the action lies within the direct action space A , it is directly executed by the controller as the agent performs sophisticated and contact-rich manipulations.

Original: However, when the actions are larger in magnitude (i.e. ˜ a ∈ ˆ A ), the probability of collisions in the presence of obstacles increases.
Revised: However, when the actions are larger in magnitude (i.e. ˜ a 2 ˆ A ), the probability of collisions in the presence of obstacles increases.

Original: Therefore, a sampling-based motion planner, RRT-Connect [25], is called to realize such actions with large joint displacements by computing collision-free paths.
Revised: Therefore, a sampling-based motion planner, RRT-Connect [25], is called to realize such actions with large joint displacements by computing collision-free paths.

Original: This allows the agent to efﬁciently explore obstructed environments while avoiding collision [1]
Revised: This allows the agent to efﬁciently explore obstructed environments while avoiding collision [1].

Original: 3.2 Motion Planner Augmented Policy Distillation
Revised: 3.2 Motion Planner Augmented Policy Distillation

Original: Despite the advantages of MoPA-RL, its applicability to the real world is limited due to the high computational cost of MP and dependency on fully observable environment states.
Revised: Despite the advantages of MoPA-RL, its applicability to the real world is limited due to the high computational cost of MP and dependency on fully observable environment states.

Original: To this end, we propose a visual distillation method, motion planner augmented policy distillation (MoPAPD), that learns an image-based control policy (without MP) from a state-based MP-augmented policy by leveraging its rollouts and the learned state-based critic as a guidance.
Revised: To this end, we propose a visual distillation method, motion planner augmented policy distillation (MoPAPD), that learns an image-based control policy (without MP) from a state-based MP-augmented policy by leveraging its rollouts and the learned state-based critic as a guidance.

Original: Concretely, our proposed method consists of two stages, as illustrated in Figure 1.
Revised: Concretely, our proposed method consists of two stages, as illustrated in Figure 1.

Original: Given a state-based MoPA-RL agent, we ﬁrst use BC to train a vision-based actor with trajectories collected from the MoPA-RL policy (Section 3.2.1), and then we further improve the vision-based agent via BC trajectory-guided asymmetric actor-critic RL [5] with the visual BC actor and MoPA-RL critic (Section 3.2.2).
Revised: Given a state-based MoPA-RL agent, we ﬁrst use BC to train a vision-based actor with trajectories collected from the MoPA-RL policy (Section 3.2.1), and then we further improve the vision-based agent via BC trajectory-guided asymmetric actor-critic RL [5] with the visual BC actor and MoPA-RL critic (Section 3.2.2).

Original: 3.2.1 Stage 1: Visual Behavioral Cloning and Trajectory Smoothing
Revised: 3.2.1 Stage 1: Visual Behavioral Cloning and Trajectory Smoothing

Original: The MoPA-RL policy is learned with access to complete information about the environment states and also utilizes MP for executing large action steps without collisions.
Revised: The MoPA-RL policy is learned with access to complete information about the environment states and also utilizes MP for executing large action steps without collisions.

Original: In this paper, we aim to utilize demonstrations from this planner-based policy and distill it into a visual control policy, thereby deducting expensive MP computations and training the actor in visual domain.
Revised: In this paper, we aim to utilize demonstrations from this planner-based policy and distill it into a visual control policy, thereby deducting expensive MP computations and training the actor in visual domain.

Original: With the learned MoPA-RL policy, we ﬁrst collect multiple transitions
Revised: With the learned MoPA-RL policy, we ﬁrst collect multiple transitions

Original: d i in low-level (direct) action space and store them into the MoPA-RL demonstration dataset D mp = { d 1 , d 2 , d 3 , ... } , where d i =
Revised: d i in low-level (direct) action space and store them into the MoPA-RL demonstration dataset D mp = { d 1 , d 2 , d 3 , ... } , where d i =

Original: ( s i , o i , a i , r i , s i +1 ) .
Revised: ( s i , o i , a i , r i , s i +1 ) .

Original: Then, we train a visual BC actor π θ ( a t | o t ) using observation-action pairs ( o i , a i ) from the dataset D mp by minimizing the mean squared error.
Revised: Then, we train a visual BC actor ⇡ ✓ ( a t | o t ) using observation-action pairs ( o i , a i ) from the dataset D mp by minimizing the mean squared error.

Original: Distilling the MoPA-RL trajectories using BC not only enables the BC actor to work directly on visual inputs but also reduces jerky motion planning behaviors of MoPA-RL, which occur due to motion planner’s priority on obstacle avoidance over trajectory smoothness [26].
Revised: Distilling the MoPA-RL trajectories using BC not only enables the BC actor to work directly on visual inputs but also reduces jerky motion planning behaviors of MoPA-RL, which occur due to motion planner’s priority on obstacle avoidance over trajectory smoothness [26].

Original: By removing unnecessary, jerky movements through visual BC, the resulting trajectories become smoother and even shorter, and thus help learning consistent and smooth motions.
Revised: By removing unnecessary, jerky movements through visual BC, the resulting trajectories become smoother and even shorter, and thus help learning consistent and smooth motions.

Original: However, the BC actor often fails when it encounters states not seen during training due to covariate shift [6].
Revised: However, the BC actor often fails when it encounters states not seen during training due to covariate shift [6].

Original: To improve the robustness of the policy, we further train the visual BC actor using RL with additional environment interactions.
Revised: To improve the robustness of the policy, we further train the visual BC actor using RL with additional environment interactions.

Original: In other words, the BC actor can be a good starting policy for visual RL training and the trajectories collected from the BC actor can effectively guide exploration.
Revised: In other words, the BC actor can be a good starting policy for visual RL training and the trajectories collected from the BC actor can effectively guide exploration.

Original: 3.2.2 Stage 2: Vision-based RL with Asymmetric Actor-Critic
Revised: 3.2.2 Stage 2: Vision-based RL with Asymmetric Actor-Critic

Original: In the second stage, we further train the visual policy using RL directly on image observations to enhance the robustness of the policy and overcome sub-optimality in the planner-based policy.
Revised: In the second stage, we further train the visual policy using RL directly on image observations to enhance the robustness of the policy and overcome sub-optimality in the planner-based policy.

Original: For efﬁcient RL training, we adopt an asymmetric actor-critic architecture [5], where the actor acts based on environment images and robot joint states while the critic learns from environment states.
Revised: For efﬁcient RL training, we adopt an asymmetric actor-critic architecture [5], where the actor acts based on environment images and robot joint states while the critic learns from environment states.

Original: This architecture is motivated by learning transferable visual policies, which do not require state information during inference but beneﬁt from them while learning in simulation.
Revised: This architecture is motivated by learning transferable visual policies, which do not require state information during inference but beneﬁt from them while learning in simulation.

Original: Our training procedure comprises of the following components:
Revised: Our training procedure comprises of the following components:

Original: Weight initialization of actor and critic networks Initializing actor and critic networks with suitable weights can accelerate RL training by providing more optimal rollouts from the actor and informative learning signals from the critic compared to the randomly initialized actor and critic [27, 28].
Revised: Weight initialization of actor and critic networks Initializing actor and critic networks with suitable weights can accelerate RL training by providing more optimal rollouts from the actor and informative learning signals from the critic compared to the randomly initialized actor and critic [27, 28].

Original: Especially when learning from pixels, suitable initialization can signiﬁcantly improve sample efﬁciency [20, 29].
Revised: Especially when learning from pixels, suitable initialization can signiﬁcantly improve sample efﬁciency [20, 29].

Original: To this end, we propose to leverage the state-based agent’s experience by utilizing its weights for initializing our asymmetric visual agent’s networks, thereby bringing their initial distributions closer.
Revised: To this end, we propose to leverage the state-based agent’s experience by utilizing its weights for initializing our asymmetric visual agent’s networks, thereby bringing their initial distributions closer.

Original: Thus, the visual actor network π θ ( a t | o t ) is initialized with the weights of the visual BC actor learned in Section 3.2.1.
Revised: Thus, the visual actor network ⇡ ✓ ( a t | o t ) is initialized with the weights of the visual BC actor learned in Section 3.2.1.

Original: We also initialize the critic Qψ ( s t , a t ) with the critic of
Revised: We also initialize the critic Q ( s t , a t ) with the critic of

Original: MoPA-RL.
Revised: MoPA-RL.

Original: Note that the action spaces of two critic networks are different, but initialization is still feasible since the action space of MoPA-RL is a superset of the direct action space.
Revised: Note that the action spaces of two critic networks are different, but initialization is still feasible since the action space of MoPA-RL is a superset of the direct action space.

Original: BC trajectory-guided asymmetric RL After initializing the actor and critic networks, we collect agent rollouts into the agent replay buffer R π .
Revised: BC trajectory-guided asymmetric RL After initializing the actor and critic networks, we collect agent rollouts into the agent replay buffer R ⇡ .

Original: As an RL algorithm, we adopt Soft Actor-Critic (SAC [24]), an off-policy model-free RL algorithm for continuous control.
Revised: As an RL algorithm, we adopt Soft Actor-Critic (SAC [24]), an off-policy model-free RL algorithm for continuous control.

Original: We optimize our asym-
Revised: We optimize our asym-

Original: [Figure]
Revised: [Figure]

Original: metric visual policy π θ by maximizing the following objective:
Revised: metric visual policy ⇡ ✓ by maximizing the following objective:

Original: [Equation]
Revised: [Equation]

Original: where the temperature parameter α balances between exploration and exploitation using entropy H .
Revised: where the temperature parameter ↵ balances between exploration and exploitation using entropy H .

Original: RL training with a BC-initialized policy often quickly deviates from the original policy.
Revised: RL training with a BC-initialized policy often quickly deviates from the original policy.

Original: To prevent this problem and guide exploration during RL, we update our RL agent not only with agent trajectories but also with BC policy (i.e. expert) trajectories [18].
Revised: To prevent this problem and guide exploration during RL, we update our RL agent not only with agent trajectories but also with BC policy (i.e. expert) trajectories [18].

Original: Thus, we collect smoothed trajectories from our visual BC policy, ( s i , o i , a i , r i , s i +1 ) , in an expert replay buffer R e .
Revised: Thus, we collect smoothed trajectories from our visual BC policy, ( s i , o i , a i , r i , s i +1 ) , in an expert replay buffer R e .

Original: Then, for each RL training iteration, we sample 1:3 transitions from R e and R π , respectively.
Revised: Then, for each RL training iteration, we sample 1:3 transitions from R e and R ⇡ , respectively.

Original: A separate expert replay buffer ensures guided exploration from the expert and also circumvents catastrophic forgetting after weight initialization.
Revised: A separate expert replay buffer ensures guided exploration from the expert and also circumvents catastrophic forgetting after weight initialization.

Original: Note that we use the BC trajectories R e instead of the MoPA-RL data D mp because jittery motion planning paths in the MoPA-RL data make RL training difﬁcult and sub-optimal.
Revised: Note that we use the BC trajectories R e instead of the MoPA-RL data D mp because jittery motion planning paths in the MoPA-RL data make RL training difﬁcult and sub-optimal.

Original: Entropy coefﬁcient tuning The performance of SAC is known to be sensitive to α [30].
Revised: Entropy coefﬁcient tuning The performance of SAC is known to be sensitive to ↵ [30].

Original: Since our asymmetric agent already starts with the well-trained BC actor and MoPA-RL critic, we initialize α to values lower than the ﬁnal α obtained in MoPA-RL.
Revised: Since our asymmetric agent already starts with the well-trained BC actor and MoPA-RL critic, we initialize ↵ to values lower than the ﬁnal ↵ obtained in MoPA-RL.

Original: This is to ensure that with prior knowledge in the state-based agent, the visual agent focuses more on maximizing rewards over exploration.
Revised: This is to ensure that with prior knowledge in the state-based agent, the visual agent focuses more on maximizing rewards over exploration.

Original: We describe the hyperparameter choice and implementation details in Section ?? and Section ?? .
Revised: We describe the hyperparameter choice and implementation details in Section A.1 and Section B.

Original: In summary, we propose a two-step visual distillation method for a state-based MoPA-RL agent using visual BC followed by BC trajectory-guided vision-based RL to remove the dependency on MP and environment states.
Revised: In summary, we propose a two-step visual distillation method for a state-based MoPA-RL agent using visual BC followed by BC trajectory-guided vision-based RL to remove the dependency on MP and environment states.

Original: For efﬁcient RL training, our method leverages weight initialization of the actor and critic, the expert replay buffer of BC smoothed trajectories, and entropy coefﬁcient tuning.
Revised: For efﬁcient RL training, our method leverages weight initialization of the actor and critic, the expert replay buffer of BC smoothed trajectories, and entropy coefﬁcient tuning.

Original:  Experiments
Revised:  Experiments

Original: In this paper, we propose to distill a motion planner augmented policy into a visual control policy for complex manipulation tasks in obstructed environments.
Revised: In this paper, we propose to distill a motion planner augmented policy into a visual control policy for complex manipulation tasks in obstructed environments.

Original: In our experiments, we aim to answer the following questions: (1) Does IL efﬁciently learn policies for obstructed environments in image space?
Revised: In our experiments, we aim to answer the following questions: (1) Does IL efﬁciently learn policies for obstructed environments in image space?

Original: Moreover, is naively combining IL with RL sufﬁcient for solving complex tasks?
Revised: Moreover, is naively combining IL with RL sufﬁcient for solving complex tasks?

Original: (2) Is distillation better than directly learning a visual policy using MP?
Revised: (2) Is distillation better than directly learning a visual policy using MP?

Original: (3) Does our approach, MoPA-PD, efﬁciently learn a visual policy using prior state-based exploration knowledge?
Revised: (3) Does our approach, MoPA-PD, efﬁciently learn a visual policy using prior state-based exploration knowledge?

Original: (4) Is our visual policy capable of domain transfer and robust to unseen distractors?
Revised: (4) Is our visual policy capable of domain transfer and robust to unseen distractors?

Original: 4.1 Environments
Revised: 4.1 Environments

Original: We evaluate our approach on three obstructed environments (see Figure 2) from Yamada et al.
Revised: We evaluate our approach on three obstructed environments (see Figure 2) from Yamada et al.

Original: [1], simulated using the MuJoCo physics engine [31].
Revised: [1], simulated using the MuJoCo physics engine [31].

Original: We use a 32x32 image as a visual observation.
Revised: We use a 32x32 image as a visual observation.

Original: [Table]
Revised: [Table]

Original: MoPA-RL [1] is the expert agent trained with state-based policy.
Revised: MoPA-RL [1] is the expert agent trained with state-based policy.

Original: All baselines
Revised: All baselines

Original:  are trained 
Revised: horizontal line are trained with asymmetric

Original: for
Revised: for

Original:  • Sawyer Push: A Rethink Sawyer robot arm with 7 DoF, initially located outside the box, needs to reach an object placed inside the box and push it to the goal location within the box.
Revised: comparison horizon is 250 for • Sawyer Push: A Rethink Sawyer robot arm with 7 DoF, initially located outside the box, needs to reach an object placed inside the box and push it to the goal location within the box.

Original: • Sawyer Lift: The Sawyer arm must reach an object placed inside a box, grasp, and lift it outside the box, avoiding collisions.
Revised: • Sawyer Lift: The Sawyer arm must reach an object placed inside a box, grasp, and lift it outside the box, avoiding collisions.

Original: The Sawyer arm is always initialized outside the box.
Revised: The Sawyer arm is always initialized outside the box.

Original: • Sawyer Assembly:
Revised: • Sawyer Assembly:

Original: The Sawyer arm needs to assemble the fourth leg (already attached to the gripper) of a table into the vacant hole while avoiding collisions to other three legs.
Revised: The Sawyer arm needs to assemble the fourth leg (already attached to the gripper) of a table into the vacant hole while avoiding collisions to other three legs.

Original: This environment is built upon the IKEA furniture assembly environment [32].
Revised: This environment is built upon the IKEA furniture assembly environment [32].

Original: For Sawyer Push and Sawyer Assembly, the agent receives a reward proportional to the distance between the end-effector and object, or the object and goal position when the distance is less than (cid:15) .
Revised: For Sawyer Push and Sawyer Assembly, the agent receives a reward proportional to the distance between the end-effector and object, or the object and goal position when the distance is less than ✏ .

Original: We use (cid:15) = 0 .
Revised: We use ✏ = 0 .

Original: 1 for Sawyer Push and (cid:15) = 0 .
Revised: 1 for Sawyer Push and ✏ = 0 .

Original: 3 for Sawyer Assembly while the distance between initial and goal state is around 1 . 2 .
Revised: 3 for Sawyer Assembly while the distance between initial and goal state is around 1 . 2 .

Original: An episode is considered successful when the distance between the cube and goal (Sawyer Push) or the peg-head and goal (Sawyer Assembly) is less than 0.05.
Revised: An episode is considered successful when the distance between the cube and goal (Sawyer Push) or the peg-head and goal (Sawyer Assembly) is less than 0.05.

Original: For Sawyer
Revised: For Sawyer

Original: Lift, we use the reward function similar to Fan et al.
Revised: Lift, we use the reward function similar to Fan et al.

Original: [33], where a reward is deﬁned for all three intermediate stages of the task: reach , grasp and lift .
Revised: [33], where a reward is deﬁned for all three intermediate stages of the task: reach , grasp and lift .

Original: A bonus reward signal of 150 is received upon successful task completion in all the environments.
Revised: A bonus reward signal of 150 is received upon successful task completion in all the environments.

Original: We refer readers to Section ?? for more details.
Revised: We refer readers to Section D for more details.

Original: 4.2 Baselines
Revised: 4.2 Baselines

Original: We compare our method with the following baselines to evaluate the merits of our approach:
Revised: We compare our method with the following baselines to evaluate the merits of our approach:

Original: • MoPA-RL [1]: A MP-augmented policy, for learning large displacement actions with MP and smaller actions with RL.
Revised: • MoPA-RL [1]: A MP-augmented policy, for learning large displacement actions with MP and smaller actions with RL.

Original: This policy also serves as our state-based expert agent.
Revised: This policy also serves as our state-based expert agent.

Original: • Visual Behavioral Cloning (BC-Visual) :
Revised: • Visual Behavioral Cloning (BC-Visual) :

Original: A behavioral cloning policy trained on the imageaction pairs collected from MoPA-RL trajectories.
Revised: A behavioral cloning policy trained on the imageaction pairs collected from MoPA-RL trajectories.

Original: • Asym.
Revised: • Asym.

Original: SAC [5]: An asymmetric actor-critic method, trained using SAC where the critic is learned in the state space and the actor is learned in the image space.
Revised: SAC [5]: An asymmetric actor-critic method, trained using SAC where the critic is learned in the state space and the actor is learned in the image space.

Original: • MoPA Asym.
Revised: • MoPA Asym.

Original: SAC : MoPA-RL policy [1] learned using the asymmetric framework.
Revised: SAC : MoPA-RL policy [1] learned using the asymmetric framework.

Original: Note that this method still uses a motion planner with an augmented action space.
Revised: Note that this method still uses a motion planner with an augmented action space.

Original: This method is a direct attempt at learning a visual policy using a MP-augmented framework [1].
Revised: This method is a direct attempt at learning a visual policy using a MP-augmented framework [1].

Original: • CoL [29]: A policy learned using the Cycle-of-Learning framework, which is the state-of-the-art algorithm for learning from demonstrations (LfD).
Revised: • CoL [29]: A policy learned using the Cycle-of-Learning framework, which is the state-of-the-art algorithm for learning from demonstrations (LfD).

Original: • CoL (w BC Smoothing)
Revised: • CoL (w BC Smoothing)

Original: [29]: A policy similar to CoL [29], with BC for trajectory smoothing.
Revised: [29]: A policy similar to CoL [29], with BC for trajectory smoothing.

Original: • Ours (w/o BC Smoothing) :
Revised: • Ours (w/o BC Smoothing) :

Original: A policy learned using our approach described in Section 4 without BC trajectory smoothing,
Revised: A policy learned using our approach described in Section 4 without BC trajectory smoothing,

Original: i
Revised: i

Original: e., we directly use MoPA-RL trajectories in R e .
Revised: e., we directly use MoPA-RL trajectories in R e .

Original: 4.3 Evaluation
Revised: 4.3 Evaluation

Original: We compare our approach against baselines on the following evaluation metrics averaged over ﬁve random seeds and 100 unseen episodes per seed: 
Revised: We compare our approach against baselines on the following evaluation metrics averaged over ﬁve random seeds and 100 unseen episodes per seed: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: • Average Success Rate (ASR) is the average number of successful episodes.
Revised: • Average Success Rate (ASR) is the average number of successful episodes.

Original: • Average Episode Length (AEL) is the average length of successful episodes.
Revised: • Average Episode Length (AEL) is the average length of successful episodes.

Original: • Average Discounted Return (ADR) is the average discounted sum of rewards (cid:80) T −t =

Revised: • Average Discounted Return (ADR) is the average discounted sum of rewards P T  1 t =

Original: γ t R ( s t , a t ) , with T being the episode horizon.
Revised:  t R ( s t , a t ) , with T being the episode horizon.

Original: An episode completed in more time steps has a lower discounted reward, due to exponentially discounted reward with γ = 0 . 99 .
Revised: An episode completed in more time steps has a lower discounted reward, due to exponentially discounted reward with  = 0 . 99 .

Original: 4.4 Results
Revised: 4.4 Results

Original: Comparisons with baselines Figure 3 illustrates the learning curves of our method and all other baselines using discounted rewards against the number of training steps.
Revised: Comparisons with baselines Figure 3 illustrates the learning curves of our method and all other baselines using discounted rewards against the number of training steps.

Original: As per the trend, our method is far more sample-efﬁcient and outperforms all the baselines.
Revised: As per the trend, our method is far more sample-efﬁcient and outperforms all the baselines.

Original: Asym.
Revised: Asym.

Original: SAC [5] fails to directly learn a visual policy for Sawyer Lift and Sawyer Assembly in obstructed environments, where exploration is hard.
Revised: SAC [5] fails to directly learn a visual policy for Sawyer Lift and Sawyer Assembly in obstructed environments, where exploration is hard.

Original: Moreover, MoPA Asym.
Revised: Moreover, MoPA Asym.

Original: SAC, which is a direct attempt at learning a visual policy while using
Revised: SAC, which is a direct attempt at learning a visual policy while using

Original: MP, does not successfully learn to solve the tasks.
Revised: MP, does not successfully learn to solve the tasks.

Original: CoL [29] which has a straightforward combination of RL and BC objective for actor optimization receives partial rewards for some tasks at around 1.5-2M steps, much slower than our method’s convergence.
Revised: CoL [29] which has a straightforward combination of RL and BC objective for actor optimization receives partial rewards for some tasks at around 1.5-2M steps, much slower than our method’s convergence.

Original: As reported in Table 1, BC achieves decent performance for Sawyer Push and Assembly, but not for the Sawyer Lift task.
Revised: As reported in Table 1, BC achieves decent performance for Sawyer Push and Assembly, but not for the Sawyer Lift task.

Original: However, it achieves high AEL and low ADR for all tasks, showing its inefﬁciency to solve the task fast.
Revised: However, it achieves high AEL and low ADR for all tasks, showing its inefﬁciency to solve the task fast.

Original: This is because BC is bound to perform as good as the expert demonstrations (here from MoPA-RL) at its best.
Revised: This is because BC is bound to perform as good as the expert demonstrations (here from MoPA-RL) at its best.

Original: In contrast, our method optimizes trajectories beyond expert signals and achieves signiﬁcantly lower AEL and higher ADR compared to baselines.
Revised: In contrast, our method optimizes trajectories beyond expert signals and achieves signiﬁcantly lower AEL and higher ADR compared to baselines.

Original: Ablation on BC trajectory smoothing Our method without BC smoothing has higher variance across seeds and converges slower than our method for two tasks (see Figure 3).
Revised: Ablation on BC trajectory smoothing Our method without BC smoothing has higher variance across seeds and converges slower than our method for two tasks (see Figure 3).

Original: For Sawyer Assembly, it does not learn to solve the task at all.
Revised: For Sawyer Assembly, it does not learn to solve the task at all.

Original: In short, BC smoothing of the motion planner trajectories is important for our method to work across all environments.
Revised: In short, BC smoothing of the motion planner trajectories is important for our method to work across all environments.

Original: This is because the motion planner based trajectories are usually jittery and non-smooth.
Revised: This is because the motion planner based trajectories are usually jittery and non-smooth.

Original: Using behavioral cloning reﬁnes each transition, thereby making the entire trajectories much smoother and help in efﬁciently training the RL agent.
Revised: Using behavioral cloning reﬁnes each transition, thereby making the entire trajectories much smoother and help in efﬁciently training the RL agent.

Original: Ablation on weight initialization We compare our method with and without actor-critic initialization in appendix, Figure ?? and do not observe any episode success until 1.2M environment steps for the latter.
Revised: Ablation on weight initialization We compare our method with and without actor-critic initialization in appendix, Figure 7 and do not observe any episode success until 1.2M environment steps for the latter.

Original: This shows the importance of our proposed weight initialization for learning in the visual domain.
Revised: This shows the importance of our proposed weight initialization for learning in the visual domain.

Original: We further elaborate the ablation results in Section ?? .
Revised: We further elaborate the ablation results in Section A.2.

Original: Ablation on entropy coefﬁcient tuning We examine the effect of different values of α on the trade-off between entropy maximization and reward maximization for the SAC objective as shown in
Revised: Ablation on entropy coefﬁcient tuning We examine the effect of different values of ↵ on the trade-off between entropy maximization and reward maximization for the SAC objective as shown in

Original: Section ?? .
Revised: Section A.1.

Original: Compared to higher α values, smaller values of α improve the sample efﬁciency during visual policy learning.
Revised: Compared to higher ↵ values, smaller values of ↵ improve the sample efﬁciency during visual policy learning.

Original: Since we utilize prior knowledge from the state-based agent, we use a smaller alpha to exploit the previously acquired knowledge instead of exploring the entire state space again.
Revised: Since we utilize prior knowledge from the state-based agent, we use a smaller alpha to exploit the previously acquired knowledge instead of exploring the entire state space again.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: 4.5 Policy Transfer to Different Domains
Revised: 4.5 Policy Transfer to Different Domains

Original: In this experiment
Revised: In this experiment

Original: [Table]
Revised: [Table]

Original: we learn our policy with domain randomization (DR) to verify its domain transfer capabilities.
Revised: we learn our policy with domain randomization (DR) to verify its domain transfer capabilities.

Original: DR is a promising approach for modelling transferable policies using a multitude of variations in simulation during training [34, 35], which makes the policy invariant to changes that are insigniﬁcant for the task.
Revised: DR is a promising approach for modelling transferable policies using a multitude of variations in simulation during training [34, 35], which makes the policy invariant to changes that are insigniﬁcant for the task.

Original: For this work, we randomize the simulation environment using different textures, colors
Revised: For this work, we randomize the simulation environment using different textures, colors

Original: [Table]
Revised: [Table]

Original: and lighting conditions for training as shown in Figure 4 and Figure ?? .
Revised: and lighting conditions for training as shown in Figure 4 and Figure 10.

Original: We then test our DR policy on three unseen scenarios, as illustrated in Figure 4 comprising of (1) the original environment without any randomization; (2) Scenario 1 with realistic background distractions (e.g., furniture, walls, and ﬂoor) and unseen distractor cubes (blue and magenta) near the target cube (red); and (3) Scenario 2 with additional changes in the texture of the table and box, and different size of the table.
Revised: We then test our DR policy on three unseen scenarios, as illustrated in Figure 4 comprising of (1) the original environment without any randomization; (2) Scenario 1 with realistic background distractions (e.g., furniture, walls, and ﬂoor) and unseen distractor cubes (blue and magenta) near the target cube (red); and (3) Scenario 2 with additional changes in the texture of the table and box, and different size of the table.

Original: Our policy attains more than 96% ASR in all three unseen scenarios for all the tasks (see
Revised: Our policy attains more than 96% ASR in all three unseen scenarios for all the tasks (see

Original: Table 2).
Revised: Table 2).

Original: This illustrates the robustness of our proposed framework in learning transferable visual policies, robust to unseen distractors in a sample-efﬁcient manner.
Revised: This illustrates the robustness of our proposed framework in learning transferable visual policies, robust to unseen distractors in a sample-efﬁcient manner.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we introduce a two-step distillation method for learning manipulation tasks in obstructed environments in the visual domain.
Revised: In this paper, we introduce a two-step distillation method for learning manipulation tasks in obstructed environments in the visual domain.

Original: In step one, we use a MP-augmented RL policy as the state-based expert and subsequently learn a visual BC agent from it, removing the motion planner dependency.
Revised: In step one, we use a MP-augmented RL policy as the state-based expert and subsequently learn a visual BC agent from it, removing the motion planner dependency.

Original: In step two, we learn a vision-based agent using an asymmetric actor-critic framework.
Revised: In step two, we learn a vision-based agent using an asymmetric actor-critic framework.

Original: This step is further expedited via proper weight initialization, BC trajectory-guided RL training, and entropy coefﬁcient tuning, making our method highly sample-efﬁcient.
Revised: This step is further expedited via proper weight initialization, BC trajectory-guided RL training, and entropy coefﬁcient tuning, making our method highly sample-efﬁcient.

Original: Our visual policy combined with domain randomization demonstrates successful zero-shot transfer to unseen environments with new visual domains and distractors.
Revised: Our visual policy combined with domain randomization demonstrates successful zero-shot transfer to unseen environments with new visual domains and distractors.

Original: Beyond zero-shot transfer, ﬁne-tuning the policy in the real world by learning a vision-based critic or applying simulation-to-real techniques [36] is our deﬁnite future work to realize simulation-to-real transfer.
Revised: Beyond zero-shot transfer, ﬁne-tuning the policy in the real world by learning a vision-based critic or applying simulation-to-real techniques [36] is our deﬁnite future work to realize simulation-to-real transfer.

Original: Acknowledgments
Revised: Acknowledgments

Original: This research is supported by the Annenberg Fellowship from USC, NAVER AI Lab, and NSF
Revised: This research is supported by the Annenberg Fellowship from USC, NAVER AI Lab, and NSF

Original: NRI-2024768.
Revised: NRI-2024768.

Original: We thank our colleagues from the CLVR lab and RESL for the valuable discussions that considerably assisted the research.
Revised: We thank our colleagues from the CLVR lab and RESL for the valuable discussions that considerably assisted the research.