Original: Follow the Object: Curriculum Learning for Manipulation Tasks with Imagined Goals
Revised: Follow the Object: Curriculum Learning for Manipulation Tasks with Imagined Goals

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Learning robot manipulation through deep reinforcement learning in environmentswith sparse rewards is a challenging task.
Revised: Learning robot manipulation through deep reinforcement learning in environments with sparse rewards is a challenging task.

Original: In this paper we address this problemby introducing a notion of imaginary object goals.
Revised: In this paper we address this problem by introducing a notion of imaginary object goals.

Original: For a given manipulation task,the object of interest is ﬁrst trained to reach a desired target position on its own,without being manipulated, through physically realistic simulations.
Revised: For a given manipulation task, the object of interest is ﬁrst trained to reach a desired target position on its own, without being manipulated, through physically realistic simulations.

Original: The objectpolicy is then leveraged to build a predictive model of plausible object trajectoriesproviding the robot with a curriculum of incrementally more difﬁcult object goalsto reach during training.
Revised: The object policy is then leveraged to build a predictive model of plausible object trajectories providing the robot with a curriculum of incrementally more difﬁcult object goals to reach during training.

Original: The proposed algorithm, Follow the Object (FO), has beenevaluated on 7 MuJoCo environments requiring increasing degree of exploration,and has achieved higher success rates compared to alternative algorithms.
Revised: The proposed algorithm, Follow the Object (FO), has been evaluated on 7 MuJoCo environments requiring increasing degree of exploration, and has achieved higher success rates compared to alternative algorithms.

Original: Inparticularly challenging learning scenarios, e g. where the object’s initial and targetpositions are far apart, our approach can still learn a policy whereas competingmethods currently fail.
Revised: In particularly challenging learning scenarios, e.g. where the object’s initial and target positions are far apart, our approach can still learn a policy whereas competing methods currently fail.

Original:  Introduction
Revised:  Introduction

Original: Reinforcement Learning (RL) aims to solve sequential decision making problems where the decisionmaker seeks to ﬁnd the optimal actions that maximise a feedback signal [1].
Revised: Reinforcement Learning (RL) aims to solve sequential decision making problems where the decision maker seeks to ﬁnd the optimal actions that maximise a feedback signal [1].

Original: Robotic problems arecharacterised by the need to make sequential decisions, and recent advances in this domain havebeen made through deep RL.
Revised: Robotic problems are characterised by the need to make sequential decisions, and recent advances in this domain have been made through deep RL.

Original: Typical applications involve locomotion [2–4] and manipulation tasks[5, 6], such as grasping [7, 8], stacking [9] and dexterous hand manipulation [10, 11].
Revised: Typical applications involve locomotion [2–4] and manipulation tasks [5, 6], such as grasping [7, 8], stacking [9] and dexterous hand manipulation [10, 11].

Original: Since the largemajority of today’s RL algorithms are still sample inefﬁcient, learning from trial and error in the realworld is often unfeasible, and it is commonplace to train RL policies for continuous control using asimulated environment, e.g. through a physics engine such as MuJoCo [12].
Revised: Since the large majority of today’s RL algorithms are still sample inefﬁcient, learning from trial and error in the real world is often unfeasible and it is commonplace to train RL policies for continuous control using a simulated environment, e.g. through a physics engine such as MuJoCo [12].

Original: Traditional RL approaches for robotics have required manually designed and problem-speciﬁc rewardfunctions providing a smoothly varying feedback signal for every visited state [13, 14, 8, 6].
Revised: Traditional RL approaches for robotics have required manually designed and problem-speciﬁc reward functions providing a smoothly varying feedback signal for every visited state [13, 14, 8, 6].

Original: Recently,there has been increasing evidence that similar or even superior policies for robotics tasks can beobtained using sparse rewards, i.e. when non-zero rewards are given to the agent only when thetask is successfully completed [15].
Revised: Recently, there has been increasing evidence that similar or even superior policies for robotics tasks can be obtained using sparse rewards, i.e. when non-zero rewards are given to the agent only when the task is successfully completed [15].

Original: Although exploration becomes much harder in this setting, oneparticular approach - Hindsight Experience Replay (HER) [16] - has been proven to enable learning.
Revised: Although exploration becomes much harder in this setting, one particular approach - Hindsight Experience Replay (HER) [16] - has been proven to enable learning.

Original: Despite such progress, contradicting claims have been reported in the literature about the empiricalperformance of HER even for simple robotics tasks.
Revised: Despite such progress, contradicting claims have been reported in the literature about the empirical performance of HER even for simple robotics tasks.

Original: For instance, in PickAndPlace , HER has beenfound to fail in some cases [9] or to succeed in other cases, provided that some of the training episodesare started from ‘easy’ states, e.g. when the object has already been grasped by the robot [16].
Revised: For instance, in PickAndPlace , HER has been found to fail in some cases [9] or to succeed in other cases, provided that some of the training episodes are started from ‘easy’ states, e.g. when the object has already been grasped by the robot [16].

Original: Otherstudies have indicated that initial grasping is not a prerequisite to success as long as the object’starget position is ‘easy’, e.g. the target position is on the table in at least some training episodes [17].
Revised: Other studies have indicated that initial grasping is not a prerequisite to success as long as the object’s target position is ‘easy’, e.g. the target position is on the table in at least some training episodes [17].

Original: Overall, the currently available experimental evidence conﬁrms what intuition alone would suggest
Revised: Overall, the currently available experimental evidence conﬁrms what intuition alone would suggest:

Original: [Figure]

Original:  obo t i c
Revised: R obo t i c

Original: M a n i p u l a t i o n
Revised: M a n i p u l a t i o n

Original: [Figure]
Revised: [Figure]

Original: Inspired by these ﬁndings, we set out to investigate mechanisms to further improve upon the performance of HER on manipulation tasks that are currently difﬁcult to learn, e.g. when the object’s targetposition is far apart from its starting position.
Revised: Inspired by these ﬁndings, we set out to investigate mechanisms to further improve upon the performance of HER on manipulation tasks that are currently difﬁcult to learn, e.g. when the object’s target position is far apart from its starting position.

Original: These situations require a signiﬁcantly higher degree ofexploration which makes learning with only sparse rewards particularly challenging.
Revised: These situations require a signiﬁcantly higher degree of exploration which makes learning with only sparse rewards particularly challenging.

Original: The approachwe propose stems from recognising the importance of learning over a curriculum of progressivelymore difﬁcult goals.
Revised: The approach we propose stems from recognising the importance of learning over a curriculum of progressively more difﬁcult goals.

Original: Curriculum learning is based upon the principle that mastering simple skills ﬁrstcan help master harder ones later [18].
Revised: Curriculum learning is based upon the principle that mastering simple skills ﬁrst can help master harder ones later [18].

Original: However, how to implement it on manipulation tasks withonly sparse reward signals is not obvious.
Revised: However, how to implement it on manipulation tasks with only sparse reward signals is not obvious.

Original: The approach proposed here relies on a mechanism to imagine object positions that are progressivelymore difﬁcult for the object to reach during training.
Revised: The approach proposed here relies on a mechanism to imagine object positions that are progressively more difﬁcult for the object to reach during training.

Original: These intermediary goals guide the progressionof mastering simple skills ﬁrst (e.g. pushing an object to a nearby target) followed by more difﬁcultones.
Revised: These intermediary goals guide the progression of mastering simple skills ﬁrst (e.g. pushing an object to a nearby target) followed by more difﬁcult ones.

Original: Our developments in this direction are inspired by the notion of object locomotion policies [19],where the objects involved in a manipulation task are modelled as independent agents that must learna locomotion policy (i.e. how to move from any initial position to a ﬁnal one).
Revised: Our developments in this direction are inspired by the notion of object locomotion policies [19], where the objects involved in a manipulation task are modelled as independent agents that must learn a locomotion policy (i.e. how to move from any initial position to a ﬁnal one).

Original: The authors used theobject policies to deﬁne auxiliary rewards for the main robot manipulation task at hand.
Revised: The authors used the object policies to deﬁne auxiliary rewards for the main robot manipulation task at hand.

Original: We initially follow a similar approach as in [19], and obtain an object locomotion policy for the giventask.
Revised: We initially follow a similar approach as in [19], and obtain an object locomotion policy for the given task.

Original: Learning such policies successfully with only sparse rewards can be accomplished using aphysics engine such as MuJoCo where objects are allowed to move freely in space, on their own,without being manipulated.
Revised: Learning such policies successfully with only sparse rewards can be accomplished using a physics engine such as MuJoCo where objects are allowed to move freely in space, on their own, without being manipulated.

Original: Our key contribution here is to leverage the object policy to implementa curriculum learning strategy for the manipulation task.
Revised: Our key contribution here is to leverage the object policy to implement a curriculum learning strategy for the manipulation task.

Original: As the robot learns the task, imaginedobject positions are produced, each one conditional on the object’s initial and target position.
Revised: As the robot learns the task, imagined object positions are produced, each one conditional on the object’s initial and target position.

Original: Theseimagined positions act as intermediary goals of increasing complexity.
Revised: These imagined positions act as intermediary goals of increasing complexity.

Original: Since this complexity can becontrolled precisely, the resulting learning strategy facilitates mastering simple tasks ﬁrst followed bymore difﬁcult ones, and results in higher success rate overall.
Revised: Since this complexity can be controlled precisely, the resulting learning strategy facilitates mastering simple tasks ﬁrst followed by more difﬁcult ones, and results in higher success rate overall.

Original: The proposed approach, Follow the
Revised: The proposed approach, Follow the

Original: Object (FO), is illustrated in Figure 1 and described in Section 4.
Revised: Object (FO), is illustrated in Figure 1 and described in Section 4.

Original: Its performance has been testedon 7 MuJoCo environments using a Fetch robotic arm with different degrees of task complexity anddifferent starting conﬁgurations.
Revised: Its performance has been tested on 7 MuJoCo environments using a Fetch robotic arm with different degrees of task complexity and different starting conﬁgurations.

Original: Our empirical results and comparisons to related algorithms arepresented in Section 5, and indicate that FO is able to learn in particularly challenging scenarioswhereas competing methods currently fail.
Revised: Our empirical results and comparisons to related algorithms are presented in Section 5, and indicate that FO is able to learn in particularly challenging scenarios whereas competing methods currently fail.

Original:  Related Work
Revised:  Related Work

Original: Imitation learning.
Revised: Imitation learning.

Original: Imitation Learning (IL) approaches have been used to facilitate reinforcementlearning of robotic manipulation tasks with sparse rewards [15, 20, 9].
Revised: Imitation Learning (IL) approaches have been used to facilitate reinforcement learning of robotic manipulation tasks with sparse rewards [15, 20, 9].

Original: IL casts the sequential decision making problem as a supervised learning problem requiring either pre-collected observational data[21, 22] or an interactive demonstrator [23, 24].
Revised: IL casts the sequential decision making problem as a supervised learning problem requiring either pre-collected observational data [21, 22] or an interactive demonstrator [23, 24].

Original: Demonstrations can also be used to learn a rewardfunction [25–27].
Revised: Demonstrations can also be used to learn a reward function [25–27].

Original: In general, collecting demonstrations for robotics tasks can be time consuming andrequires a dedicated data collection setup, e.g. virtual reality or tele-operation facility.
Revised: In general, collecting demonstrations for robotics tasks can be time consuming and requires a dedicated data collection setup, e.g. virtual reality or tele-operation facility.

Original: Goal conditional policies.
Revised: Goal conditional policies.

Original: Recently, the advantages of goal-conditioned policies for robotics problems have been discussed [28].
Revised: Recently, the advantages of goal-conditioned policies for robotics problems have been discussed [28].

Original: Different approaches have investigated how to automatically generatesuch goals.
Revised: Different approaches have investigated how to automatically generate such goals.

Original: For instance, [29] follow a self-play approach on reversible or resettable environments,[30] employ adversarial training for robotic locomotion tasks, [31] use variational autoencodersfor visual robotics tasks and [16] propose an experience sampling approach, Hindsight Experience
Revised: For instance, [29] follow a self-play approach on reversible or resettable environments, [30] employ adversarial training for robotic locomotion tasks, [31] use variational autoencoders for visual robotics tasks and [16] propose an experience sampling approach, Hindsight Experience

Original: Replay (HER), to randomly draw synthetic goals from previously experienced trajectories.
Revised: Replay (HER), to randomly draw synthetic goals from previously experienced trajectories.

Original: HER initself may be seen as a form of implicit curriculum learning.
Revised: HER in itself may be seen as a form of implicit curriculum learning.

Original: The effectiveness of this approach hasbeen demonstrated on a suite of challenging continuous control environments [17].
Revised: The effectiveness of this approach has been demonstrated on a suite of challenging continuous control environments [17].

Original: Curiosity.
Revised: Curiosity.

Original: In these approaches, the notion of novelty of a state provides the RL agent with anadditional reward and motivation for exploring less novel states [32, 33].
Revised: In these approaches, the notion of novelty of a state provides the RL agent with an additional reward and motivation for exploring less novel states [32, 33].

Original: Curiosity has been deﬁnedin many ways, e.g. as the error in predicting the RL agent’s actions by an inverse dynamics model[34], as the error in predicting the output of a ﬁxed randomly initialised neural network [35], or byevaluating the novelty of a recently visited state according to its reachability from those stored in amemory [36].
Revised: Curiosity has been deﬁned in many ways, e.g. as the error in predicting the RL agent’s actions by an inverse dynamics model [34], as the error in predicting the output of a ﬁxed randomly initialised neural network [35], or by evaluating the novelty of a recently visited state according to its reachability from those stored in a memory [36].

Original: The large majority of these works describe gaming or maze environments with discreteaction spaces, but a locomotion task with continuous actions has also been considered [36].
Revised: The large majority of these works describe gaming or maze environments with discrete action spaces, but a locomotion task with continuous actions has also been considered [36].

Original: Curriculum learning.
Revised: Curriculum learning.

Original: Curriculum learning has been widely adopted into the RL framework andmany researchers have sought to ﬁnd a solution to the problem of how to automatically generate a wayto order the goals according to difﬁculty.
Revised: Curriculum learning has been widely adopted into the RL framework and many researchers have sought to ﬁnd a solution to the problem of how to automatically generate a way to order the goals according to difﬁculty.

Original: For example, [37] and [38] propose to automatically adjustthe difﬁculty of the gaming environment, [39] use the policies of the intermediate agents obtainedduring the training, [40] evaluate the occurrence frequencies of pre-deﬁned events.
Revised: For example, [37] and [38] propose to automatically adjust the difﬁculty of the gaming environment, [39] use the policies of the intermediate agents obtained during the training, [40] evaluate the occurrence frequencies of pre-deﬁned events.

Original: In robotics, [41]propose to schedule the initial positions, [42] create easier sub-goals spanning over one dimension ofthe target position, and [43] select experiences adaptively according to diversity-based curiosity.
Revised: In robotics, [41] propose to schedule the initial positions, [42] create easier sub-goals spanning over one dimension of the target position, and [43] select experiences adaptively according to diversity-based curiosity.

Original: Object motion prediction.
Revised: Object motion prediction.

Original: A body of works exists on predicting how objects behave under manipulative actions.
Revised: A body of works exists on predicting how objects behave under manipulative actions.

Original: For example, [44] learn a probabilistic model using a real system where a roboticarm applies random pushes to various objects, and [45] propose a simulation-based framework.
Revised: For example, [44] learn a probabilistic model using a real system where a robotic arm applies random pushes to various objects, and [45] propose a simulation-based framework.

Original: Acombination of these methods has also been proposed [46].
Revised: A combination of these methods has also been proposed [46].

Original: Some efforts have also been madeto model the dynamics of complex scenes involving non-rigid bodies, e.g. using a particle-basedsimulator [47].
Revised: Some efforts have also been made to model the dynamics of complex scenes involving non-rigid bodies, e.g. using a particle-based simulator [47].

Original: Similarly, the object locomotion policies we use here can predict the most likelytrajectory that an object has to travel to reach a given target.
Revised: Similarly, the object locomotion policies we use here can predict the most likely trajectory that an object has to travel to reach a given target.

Original: Unlike these works, our policies arelearned irrespective of the manipulative actions exercised by the robot.
Revised: Unlike these works, our policies are learned irrespective of the manipulative actions exercised by the robot.

Original: Advantages and disadvantagesof this approach are discussed in Section 5.3.
Revised: Advantages and disadvantages of this approach are discussed in Section 5.3.

Original:  Problem Deﬁnition and Background 
Revised:  Problem Deﬁnition and Background 3.1 Multi-goal RL for Robotic Manipulation

Original: 3.1 Multi-goal RL for Robotic Manipulation

Original: We are concerned with solving a manipulation task: an object is presented to the robot, and has to bemanipulated so as to reach a target position.
Revised: We are concerned with solving a manipulation task: an object is presented to the robot, and has to be manipulated so as to reach a target position.

Original: In the tasks we consider, the target goal is speciﬁed bythe object location and orientation, and the robot is rewarded only when it reaches its goal.
Revised: In the tasks we consider, the target goal is speciﬁed by the object location and orientation, and the robot is rewarded only when it reaches its goal.

Original: We modelthe robot’s sequential decision process as a Markov Decision Process (MDP) deﬁned by a tuple,
Revised: We model the robot’s sequential decision process as a Markov Decision Process (MDP) deﬁned by a tuple,

Original: M = (cid:104)S , G , A , T , R , γ (cid:105) , where S is the set of states, G is the set of goals, A is the set of actions, Tis the state transition function, R is the reward function and γ ∈ [0 , 1) is the discounting factor.
Revised: M = (cid:104)S , G , A , T , R , γ (cid:105) , where S is the set of states, G is the set of goals, A is the set of actions, T is the state transition function, R is the reward function and γ ∈ [0 , 1) is the discounting factor.

Original: Atthe beginning of an episode, the environment samples a goal g ∈ G .
Revised: At the beginning of an episode, the environment samples a goal g ∈ G .

Original: The position of the object attime t is denoted by o t and the state of the environment is s t ∈ S , which includes o t .
Revised: The position of the object at time t is denoted by o t and the state of the environment is s t ∈ S , which includes o t .

Original: We assume that,given s t , we can recover o t through a known mapping, i.e. o t = m ( s t ) .
Revised: We assume that, given s t , we can recover o t through a known mapping, i.e. o t = m ( s t ) .

Original: A robot’s action is controlledby a deterministic policy, i.e. a t = µ θ ( s t , g ) : S × G → A , parameterised by θ .
Revised: A robot’s action is controlled by a deterministic policy, i.e. a t = µ θ ( s t , g ) : S × G → A , parameterised by θ .

Original: The environmentmoves to its next state through its state transition function, i.e. s t +1 =
Revised: The environment moves to its next state through its state transition function, i.e. s t +1 =

Original: T ( s t , a t ) : S ×
Revised: T ( s t , a t ) : S ×

Original: A → S , andprovides an immediate and sparse reward r t , deﬁned as
Revised: A → S , and provides an immediate and sparse reward r t , deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) is a pre-deﬁned threshold.
Revised: where (cid:15) is a pre-deﬁned threshold.

Original: Following its policy, the robot interacts with the environmentuntil the episode terminates after T steps.
Revised: Following its policy, the robot interacts with the environment until the episode terminates after T steps.

Original: The interaction between the robot and the environment generates a trajectory, τ = ( g, s 1 , a 1 , r 1 , . . .
Revised: The interaction between the robot and the environment generates a trajectory, τ = ( g, s 1 , a 1 , r 1 , . . .

Original: , s T , a T , r T , s T +1 ) .
Revised: , s T , a T , r T , s T +1 ) .

Original: The ultimate learning objective isto ﬁnd the optimal policy that maximises the expected sum of the discounted rewards over the timehorizon T , i.e.
Revised: The ultimate learning objective is to ﬁnd the optimal policy that maximises the expected sum of the discounted rewards over the time horizon T , i.e.

Original: J ( µ θ ) =
Revised: J [Equation]

Original: E τ ∼ P ( τ | µ θ )

Original: [ R ( τ ) =

Original: T (cid:88) i = γ i − 1 r
Revised: T (cid:88) i = γ i − 1 r

Original: i ] (2) where γ is the discount factor.
Revised: i ] (2) where γ is the discount factor.

Original: 3.2 Deep Deterministic Policy Gradient Algorithm
Revised: 3.2 Deep Deterministic Policy Gradient Algorithm

Original: Deep Deterministic Policy Gradient (DDPG) [2] is adopted here as our main learning algorithm,however any other off-policy algorithm that operates on continuous action domains could be equallyused.
Revised: Deep Deterministic Policy Gradient (DDPG) [2] is adopted here as our main learning algorithm, however any other off-policy algorithm that operates on continuous action domains could be equally used.

Original: DDPG integrates deterministic policy functions [48] with non-linear function approximatorssuch as deep neural networks, and maintains a policy (actor) network µ θ ( s t , g ) and an action-value(critic) network Q µ ( s t , a t , g ) .
Revised: DDPG integrates deterministic policy functions [48] with non-linear function approximators such as deep neural networks, and maintains a policy (actor) network µ θ ( s t , g ) and an action-value (critic) network Q µ ( s t , a t , g ) .

Original: Further details about our implementation are provided in the Appendix.
Revised: Further details about our implementation are provided in the Appendix.

Original: 3.3 Hindsight Experience Replay
Revised: 3.3 Hindsight Experience Replay

Original: Suppose we are given an observed trajectory,
Revised: Suppose we are given an observed trajectory,

Original: τ = ( g, s 1 , a 1 , r 1 , . .
Revised: τ = ( g, s 1 , a 1 , r 1 , . .

Original: , s T , a T , r T , s T +1 ) .
Revised: , s T , a T , r T , s T +1 ) .

Original: Since o tcan be obtained from s t using a ﬁxed and known mapping, the path that was followed by theobject during the trajectory, i.e. o 1 , . . .
Revised: Since o t can be obtained from s t using a ﬁxed and known mapping, the path that was followed by the object during the trajectory, i.e. o 1 , . . .

Original: , o T +1 , can be easily extracted.
Revised: , o T +1 , can be easily extracted.

Original: HER samples a new goalfrom this path, i.e. ˜ g ∼ { o 1 , . . .
Revised: HER samples a new goal from this path, i.e. ˜ g ∼ { o 1 , . . .

Original: , o T } , and the rewards are recomputed with respect to ˜ g , i.e.˜
Revised: , o T } , and the rewards are recomputed with respect to ˜ g , i.e.

Original:  r t =
Revised: ˜ r t =

Original: R ( o t +1 , ˜ g ) .
Revised: R ( o t +1 , ˜ g ) .

Original: Using these rewards and ˜ g , a new trajectory is created implicitly, i.e. ˜ τ =(˜
Revised: Using these rewards and ˜ g , a new trajectory is created implicitly, i.e. ˜ τ =

Original:  g, s 1 , a 1 , ˜ r 1 , . ..
Revised: (˜ g, s 1 , a 1 , ˜ r 1 , . .

Original: , s T , a T , ˜ r T , s T +1 ) .
Revised: , s T , a T , ˜ r T , s T +1 ) .

Original: These HER trajectories ˜ τ are used to train the policy parameterstogether with the original trajectories.
Revised: These HER trajectories ˜ τ are used to train the policy parameters together with the original trajectories.

Original:  Methodology
Revised:  Methodology

Original: In this section, we explain the steps involved in our proposed procedure, i.e. (a) how object locomotionpolicies are learned, (b) how the novel curriculum learning model is deﬁned upon them, and (c) howthe curriculum is implemented for robotic manipulation.
Revised: In this section, we explain the steps involved in our proposed procedure, i.e. (a) how object locomotion policies are learned, (b) how the novel curriculum learning model is deﬁned upon them, and (c) how the curriculum is implemented for robotic manipulation.

Original: 4.1 Learning Object Locomotion Policies
Revised: 4.1 Learning Object Locomotion Policies

Original: The object involved in the manipulation task is initially modelled as an agent capable of independentdecision making abilities, and its decision process is modelled by a separate MDP deﬁned by a tuple
Revised: The object involved in the manipulation task is initially modelled as an agent capable of independent decision making abilities, and its decision process is modelled by a separate MDP deﬁned by a tuple

Original: L = (cid:104)Z , G , U , Y , R , γ (cid:105) .
Revised: L = (cid:104)Z , G , U , Y , R , γ (cid:105) .

Original: Here, Z is the set of states, G is the set of goals, U is the set of actions, Y isthe state transition function, R is the reward function and γ ∈ [0 , 1) is the discounting factor.
Revised: Here, Z is the set of states, G is the set of goals, U is the set of actions, Y is the state transition function, R is the reward function and γ ∈ [0 , 1) is the discounting factor.

Original: Thesame goal space, G , is used as in the robotic manipulation.
Revised: The same goal space, G , is used as in the robotic manipulation.

Original: z t ∈ Z is a reduced version of s t that onlyinvolves object-related features including the position of the object, i.e. o t ⊂ z t .
Revised: z t ∈ Z is a reduced version of s t that only involves object-related features including the position of the object, i.e. o t ⊂ z t .

Original: The object’s actionspace explicitly controls the pose of the object, and these actions are controlled by a deterministicpolicy, i.e. u t
Revised: The object’s action space explicitly controls the pose of the object, and these actions are controlled by a deterministic policy, i.e. u t

Original: = ν θ ( z t , g ) : Z × G → U .
Revised: = ν θ ( z t , g ) : Z × G → U .

Original: The state transition is now deﬁned on different spaces thanrobotic manipulation, i.e. Y : Z × U
Revised: The state transition is now deﬁned on different spaces than robotic manipulation, i.e. Y : Z × U

Original: → Z ; however, the same sparse reward function is used here asbefore.
Revised: → Z ; however, the same sparse reward function is used here as before.

Original: Figure 2a illustrates the training procedure used in this context and based on DDPG with HER.
Revised: Figure 2a illustrates the training procedure used in this context and based on DDPG with HER.

Original: The optimal object policy ν θ maximises the expected return J ( ν θ ) =
Revised: The optimal object policy ν θ maximises the expected return J ( ν θ ) =

Original: E g,z t ∼D [ (cid:80) Ti =1 γ
Revised: E g,z t ∼D [ (cid:80) Ti =1 γ

Original: i − 1 r i ] where
Revised: i − 1 r i ] where

Original: D denotes the replay buffer containing the trajectories, indicated by η , obtained by ν θ throughouttraining.
Revised: D denotes the replay buffer containing the trajectories, indicated by η , obtained by ν θ throughout training.

Original: 4.2 Learning to Imagine Goals
Revised: 4.2 Learning to Imagine Goals

Original: Upon training, the object locomotion policy ν θ can be used to generate trajectories, i.e.( g, z 1 , u 1 , r 1 , . . 
Revised: Upon training, the object locomotion policy ν θ can be used to generate trajectories, i.e. ( g, z 1 , u 1 , r 1 , . . .

Original: , z T , u T , r T , z T +1 ) ∼ P ( η | ν θ ) .
Revised: , z T , u T , r T , z T +1 ) ∼ P ( η | ν θ ) .

Original: Since o t ⊂ z t , an object path originating at o 1and moving all the way to the ﬁnal target g , i.e. ( o 1 , . . .
Revised: Since o t ⊂ z t , an object path originating at oand moving all the way to the ﬁnal target g , i.e. ( o 1 , . . .

Original: , o T +1 ) , can be obtained from ( z 1 , . . .
Revised: , o T +1 ) , can be obtained from ( z 1 , . . .

Original: , z T +1 ) .
Revised: , z T +1 ) .

Original: We use these trajectories to train a model, F φ , to predict the object’s position k -steps ahead withina trajectory.
Revised: We use these trajectories to train a model, F φ , to predict the object’s position k -steps ahead within a trajectory.

Original: The model takes o 1 and g together with an integer k ∈ [1 , T ] as input, and outputs theprediction for the object position at time-step k + 1 , i.e. ˆ o k +

Revised: The model takes o 1 and g together with an integer k ∈ [1 , T ] as input, and outputs the prediction for the object position at time-step k + 1 , i.e. ˆ o k +

Original: =
Revised: =

Original: F φ ( o 1 , g, k ) .
Revised: F φ ( o 1 , g, k ) .

Original: The model parameters
Revised: The model parameters

Original: [Figure]
Revised: [Figure]

Original: φ are found such that (see also Figure 2b):
Revised: φ are found such that (see also Figure 2b):

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: 4.3 Learning the Robot Policy for Manipulation with Imagined Goals
Revised: 4.3 Learning the Robot Policy for Manipulation with Imagined Goals

Original: In this ﬁnal step, the predictive model F φ is leveraged to generate a sequence of imagined goals thatcan facilitate the learning of the original manipulation policy.
Revised: In this ﬁnal step, the predictive model F φ is leveraged to generate a sequence of imagined goals that can facilitate the learning of the original manipulation policy.

Original: For every new episode characterisedby o 1 and g , a predicted object position at time-step k + 1 , i.e. h = F φ ( o 1 , g, k ) , is used as anintermediate goal to train the robot.
Revised: For every new episode characterised by o 1 and g , a predicted object position at time-step k + 1 , i.e. h = F φ ( o 1 , g, k ) , is used as an intermediate goal to train the robot.

Original: By adjusting k throughout the training, a curriculum is formed.
Revised: By adjusting k throughout the training, a curriculum is formed.

Original: Adjusting the curriculum difﬁculty: The value of k controls the difﬁculty of an imagined goal h .
Revised: Adjusting the curriculum difﬁculty: The value of k controls the difﬁculty of an imagined goal h .

Original: Starting from k = 1 and all the way up to k = T , one could simply increase k by one every time therobot masters the current level.
Revised: Starting from k = 1 and all the way up to k = T , one could simply increase k by one every time the robot masters the current level.

Original: However, we have observed that this simple strategy can underperformdue to some form of forgetting; this happens when the robot fails to master the current level, andthe replay buffer starts to be ﬁlled with failing trajectories.
Revised: However, we have observed that this simple strategy can underperform due to some form of forgetting; this happens when the robot fails to master the current level, and the replay buffer starts to be ﬁlled with failing trajectories.

Original: Instead, k is sampled from a uniformdistribution, U (1 , k max ) , and k max is increased upon reaching ﬂuency at the current difﬁculty level.
Revised: Instead, k is sampled from a uniform distribution, U (1 , k max ) , and k max is increased upon reaching ﬂuency at the current difﬁculty level.

Original: This remedy resolves the forgetting issue since training at a given difﬁculty level now allows forgoals at lower levels to be randomly introduced; this forces the robot to occasionally ‘practice’ onsimpler problems rather than always being confronted with more difﬁcult ones.
Revised: This remedy resolves the forgetting issue since training at a given difﬁculty level now allows for goals at lower levels to be randomly introduced; this forces the robot to occasionally ‘practice’ on simpler problems rather than always being confronted with more difﬁcult ones.

Original: In order to decidewhether the robot is ﬂuent enough with the current difﬁculty level, we evaluate its performance fork =
Revised: In order to decide whether the robot is ﬂuent enough with the current difﬁculty level, we evaluate its performance for k = k max , i.e. h max =

Original: k max , i.e. h max = F φ ( o 1 , g, k max ) .
Revised: F φ ( o 1 , g, k max ) .

Original: When the robot achieves a success rate higher than somethreshold (e.g. 0 . 2 ) for a given h max , we increase k max by one, and keep increasing it until T isreached.
Revised: When the robot achieves a success rate higher than some threshold (e.g. 0 . 2 ) for a given h max , we increase k max by one, and keep increasing it until T is reached.

Original: This scheme is somewhat inspired by the boundary-sampling idea used in the Automatic
Revised: This scheme is somewhat inspired by the boundary-sampling idea used in the Automatic

Original: Domain Randomization (ADR) algorithm [49].
Revised: Domain Randomization (ADR) algorithm [49].

Original: Learning the Robot Policy: The policy µ θ is learned by DDPG as in Section 3.2.
Revised: Learning the Robot Policy: The policy µ θ is learned by DDPG as in Section 3.2.

Original: We ensure thatboth original goals g and imagined goals h are used during training.
Revised: We ensure that both original goals g and imagined goals h are used during training.

Original: That is, goal ˆ g where ˆ g = gwith probability p , and otherwise ˆ g = h .
Revised: That is, goal ˆ g where ˆ g = g with probability p , and otherwise ˆ g = h .

Original: Introducing g early on in the training allows for fasterlearning as the policy may become capable to accomplish the original goals before the curriculumprovided through h is completed at k = T .
Revised: Introducing g early on in the training allows for faster learning as the policy may become capable to accomplish the original goals before the curriculum provided through h is completed at k = T .

Original: Using a mixture of g and h , the robot interacts with theenvironment and obtains trajectories τ
Revised: Using a mixture of g and h , the robot interacts with the environment and obtains trajectories τ

Original: = (ˆ g, s 1 , a 1 , ˆ r 1 , . . .
Revised: = (ˆ g, s 1 , a 1 , ˆ r 1 , . . .

Original: , s T , a T , ˆ r T , s T +1 ) , where rewards arecalculated with respect to ˆ g , i.e. ˆ r t
Revised: , s T , a T , ˆ r T , s T +1 ) , where rewards are calculated with respect to ˆ g , i.e. ˆ r t

Original: =
Revised: =

Original: R ( o t , ˆ g ) .
Revised: R ( o t , ˆ g ) .

Original: Experienced trajectories are stored into the replaybuffer D .
Revised: Experienced trajectories are stored into the replay buffer D .

Original: The parameters of µ θ are updated to maximise Eq.
Revised: The parameters of µ θ are updated to maximise Eq.

Original: (2) using the trajectories sampled from
Revised: (2) using the trajectories sampled from

Original: D .
Revised: D .

Original: Figure 2d illustrates this procedure, and Algorithm 1 provides the pseudo code.
Revised: Figure 2d illustrates this procedure, and Algorithm 1 provides the pseudo code.

Original: Given : Robotic Manipulation MDP M = (cid:104)S , G , A , T , R ,γ (cid:105) , Imagined goal generator F φ ( o 1 ,g,k ) , Random process N for exploration, Fixed and known mapping function m : S → O
Revised: Given : Robotic Manipulation MDP M = (cid:104)S , G , A , T , R ,γ (cid:105) , Imagined goal generator F φ ( o 1 ,g,k ) , Random process N for exploration, Fixed and known mapping function m : S → O

Original: Initialise: Parameters θ for robot policy µ θ , Experience replay buffer D k max = 2 for episode = 1 to N do
Revised: Initialise: Parameters θ for robot policy µ θ , Experience replay buffer D k max = 2 for episode = 1 to N do

Original: Receive initial state s 1 and g , o 

Revised: Receive initial state s 1 and g , o

Original: = m ( s 1 )
Revised: = m ( s 1 )

Original: Sample k ∼ U (1 ,k max ) where U is uniform distribution h
Revised: Sample k ∼ U (1 ,k max ) where U is uniform distribution h

Original: = F φ ( o 1 ,g,k )
Revised: = F φ ( o 1 ,g,k )

Original: ˆ g = (cid:40) g with prob.
Revised: ˆ g = (cid:40) g with prob.

Original: p h with prob.
Revised: p h with prob.

Original: 1 − p for t = 1 ,T do
Revised: 1 − p for t = 1 ,T do

Original: Sample an action: a t = µ θ (cid:0) s t , ˆ g )
Revised: Sample an action: a t = µ θ (cid:0) s t , ˆ g )

Original: + N Execute the action: s t
Revised: + N Execute the action: s t

Original: +1 =
Revised: +1 =

Original: T ( s t ,a t ) and r t =
Revised: T ( s t ,a t ) and r t =

Original: R ( o t +1 , ˆ g )
Revised: R ( o t +1 , ˆ g )

Original: Store τ =
Revised: Store τ =

Original: (ˆ g,s 1 ,a 1 , ˆ r 1 ,...,s T ,a T , ˆ r T ,s T +1 ) in memory buffer D Update µ θ with trajectories drawn from D if k max ≤ T then
Revised: (ˆ g,s 1 ,a 1 , ˆ r 1 ,...,s T ,a T , ˆ r T ,s T +1 ) in memory buffer D Update µ θ with trajectories drawn from D if k max ≤ T then

Original: Test µ θ for h max = F φ ( o 1 ,g,k max ) if success ≥ threshold then k max = k max +
Revised: Test µ θ for h max = F φ ( o 1 ,g,k max ) if success ≥ threshold then k max = k max +

Original:  Experiments 
Revised:  Experiments 5.1 Environments

Original: 5.1 Environments

Original: We evaluate our method on 7 simulated MuJoCo environments that use a 7-DoF Fetch robotics arm.
Revised: We evaluate our method on 7 simulated MuJoCo environments that use a 7-DoF Fetch robotics arm.

Original: We build these environments upon the Push and PickAndPlace environments from [17].
Revised: We build these environments upon the Push and PickAndPlace environments from [17].

Original: As visualisedin Figure 3, the environments differ in terms of initial and target object positions, as well as theobstacles between these two.
Revised: As visualised in Figure 3, the environments differ in terms of initial and target object positions, as well as the obstacles between these two.

Original: In all the environments, the target indicates the desired 3D position ofthe object.
Revised: In all the environments, the target indicates the desired 3D position of the object.

Original: A reward of 0 is provided to the agent if the object is within 5-cm range to the target, and− 1 otherwise.
Revised: A reward of 0 is provided to the agent if the object is within 5-cm range to the target, and − 1 otherwise.

Original: Robot actions are 4-dimensional: 3D to specify the desired arm movement in Cartesiancoordinates and 1D to control the opening of the gripper.
Revised: Robot actions are 4-dimensional: 3D to specify the desired arm movement in Cartesian coordinates and 1D to control the opening of the gripper.

Original: In Push tasks, the robot is not allowed tocontrol the opening of the gripper to prevent grasping.
Revised: In Push tasks, the robot is not allowed to control the opening of the gripper to prevent grasping.

Original: The observations include the positions andlinear velocities of the robot arm and the gripper, as well as the object’s position, rotation and angularvelocity, and its relative position and linear velocity to the gripper.
Revised: The observations include the positions and linear velocities of the robot arm and the gripper, as well as the object’s position, rotation and angular velocity, and its relative position and linear velocity to the gripper.

Original: An episode terminates after 50time-steps, except for Push-DoubleObstacles which terminates after 80 time-steps.
Revised: An episode terminates aftertime-steps, except for Push-DoubleObstacles which terminates after 80 time-steps.

Original: In the auxiliaryobject locomotion tasks, on the other hand, observations include the object’s position, rotation andangular velocity.
Revised: In the auxiliary object locomotion tasks, on the other hand, observations include the object’s position, rotation and angular velocity.

Original: The object actions are 7 dimensional (3D for translation and 4D for rotation).
Revised: The object actions are 7 dimensional (3D for translation and 4D for rotation).

Original: Inpractice, we deﬁne the object as a mocap MuJoCo entity which enables us to control its 7D pose.
Revised: In practice, we deﬁne the object as a mocap MuJoCo entity which enables us to control its 7D pose.

Original: Inall cases, the object actions correspond to the desired relative change in the object pose between twoconsecutive time-steps; however, their realisation depends on the dynamics executed by the simulator.
Revised: In all cases, the object actions correspond to the desired relative change in the object pose between two consecutive time-steps; however, their realisation depends on the dynamics executed by the simulator.

Original: The same reward function and termination criteria are used as in manipulation.
Revised: The same reward function and termination criteria are used as in manipulation.

Original: 5.2 Implementation and Training Process
Revised: 5.2 Implementation and Training Process

Original: We use three-layer neural networks with ReLU activations for all models, and optimise them using
Revised: We use three-layer neural networks with ReLU activations for all models, and optimise them using

Original: Adam [50].
Revised: Adam [50].

Original: We adopt the optimised hyperparameter values for HER (the baseline approach) from
Revised: We adopt the optimised hyperparameter values for HER (the baseline approach) from

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Revised: d i a n t e s t

Original: [Figure]
Revised: [Figure]

Original: d i
Revised: u cc e

Original: [Figure]
Revised: [Figure]

Original:  t 
Revised: a t e

Original: [Figure]
Revised: [Figure]

Original: FOSLDRHERShapedRNDCHER
Revised: FOSLDRHERShapedRNDCHER

Original: [Figure]
Revised: [Figure]

Original: FOSLDRHERShapedRNDCHER
Revised: FOSLDRHERShapedRNDCHER

Original: [17], and use them for  models.
Revised: [17], and use them for all models.

Original: We train the models on a GPU enabled machine using its CPU cores to generate experiences and GPU to optimise the parameters.
Revised: We train the models on a GPU enabled machine using its CPU cores to generate experiences and GPU to optimise the parameters.

Original: At each epoch of the training, we generate 38 ∗ 50 = 1900 full episodes and update the parameters 40 times using batches of size 
Revised: At each epoch of the training, we generate 38 ∗ 50 = 1900 full episodes and update the parameters 40 times using batches of size 4864 .

Original: We observed that the performance of the proposed approach is robust with respect to the modelspeciﬁc hyperparameters and report the results by choosing p = 0 .
Revised: We observed that the performance of the proposed approach is robust with respect to the model speciﬁc hyperparameters and report the results by choosing p = 0 .

Original: 2 and a success threshold of 0 .
Revised: 2 and a success threshold of 0 .

Original: An ablation study on the effects of these two hyperparameters can be found in Appendix, where wealso describe all the remaining hyperparameters in greater detail.
Revised: An ablation study on the effects of these two hyperparameters can be found in Appendix, where we also describe all the remaining hyperparameters in greater detail.

Original: 5.3 Comparison and Performance Evaluation
Revised: 5.3 Comparison and Performance Evaluation

Original: We consider 6 alternative methods for comparison.
Revised: We consider 6 alternative methods for comparison.

Original: All methods are built upon DDPG [2] andimplement HER [16] as standard.
Revised: All methods are built upon DDPG [2] and implement HER [16] as standard.

Original: Differences are as follows: HER is the baseline as described in[16] using sparse rewards.
Revised: Differences are as follows: HER is the baseline as described in [16] using sparse rewards.

Original: Shaped uses distance-based shaped rewards instead of sparse rewards.
Revised: Shaped uses distance-based shaped rewards instead of sparse rewards.

Original: SLDR uses auxiliary rewards introduced by [19] together with sparse rewards.
Revised: SLDR uses auxiliary rewards introduced by [19] together with sparse rewards.

Original: In SLDR , the optimalobject locomotion actions are compared with those caused by the robot in terms of the action-values,and the difference is introduced as an auxiliary reward.
Revised: In SLDR , the optimal object locomotion actions are compared with those caused by the robot in terms of the action-values, and the difference is introduced as an auxiliary reward.

Original: RND uses curiosity-based explorationbonuses together with sparse rewards.
Revised: RND uses curiosity-based exploration bonuses together with sparse rewards.

Original: Exploration bonuses are obtained by adopting Random
Revised: Exploration bonuses are obtained by adopting Random

Original: Network Distillation (RND) proposed in [35].
Revised: Network Distillation (RND) proposed in [35].

Original: Curriculum-guided HER ( CHER )
Revised: Curriculum-guided HER ( CHER )

Original: [43] brings adaptiveexperience selection upon baseline HER.
Revised: [43] brings adaptive experience selection upon baseline HER.

Original: FO refers to the approach introduced in this paper.
Revised: FO refers to the approach introduced in this paper.

Original: Wealso combine our approach with SLDR and refer this combination as FO+SLDR .
Revised: We also combine our approach with SLDR and refer this combination as FO+SLDR .

Original: Following [17],we evaluate the performances after each training epoch by computing the test success rate averagedacross 380 deterministic rollouts with original goals g .
Revised: Following [17], we evaluate the performances after each training epoch by computing the test success rate averaged across 380 deterministic rollouts with original goals g .

Original: In all cases, we repeat an experiment with5 different random seeds and report results by computing the median test success rate with theinterquartile range.
Revised: In all cases, we repeat an experiment with 5 different random seeds and report results by computing the median test success rate with the interquartile range.

Original: Demonstrating the Problem on Simple PickAndPlace : To investigate how initial and target objectpositions affect learning, we deﬁne 2 different versions of PickAndPlace ( PnP ).
Revised: Demonstrating the Problem on Simple PickAndPlace : To investigate how initial and target object positions affect learning, we deﬁne 2 different versions of PickAndPlace ( PnP ).

Original: PnP-Simple-v1corresponds to the version mentioned in [17].
Revised: PnP-Simple-v corresponds to the version mentioned in [17].

Original: In PnP-Simple-v2 , we simply remove two constraintsthat condition the initial and target object positions, i.e. unlike [17] we do not require the positionsto be sampled from within a smaller square centred around the gripper, and we do not enforce thetargets to be on the table level in some of the training episodes.
Revised: In PnP-Simple-v2 , we simply remove two constraints that condition the initial and target object positions, i.e. unlike [17] we do not require the positions to be sampled from within a smaller square centred around the gripper, and we do not enforce the targets to be on the table level in some of the training episodes.

Original: We adopt the baseline HER from[16] to train two policies, i.e. HER-a on PnP-Simple-v1 and HER-b on PnP-Simple-v2 , followingidentical learning procedures.
Revised: We adopt the baseline HER from [16] to train two policies, i.e. HER-a on PnP-Simple-v1 and HER-b on PnP-Simple-v2 , following identical learning procedures.

Original: Throughout the training, we test their performances on both versionsas presented in Figure 4.
Revised: Throughout the training, we test their performances on both versions as presented in Figure 4.

Original: HER-a achieves PnP-Simple-v1 with 1 . 0 success rate, and PnP-Simple-v2with 0 . 9 .
Revised: HER-a achieves PnP-Simple-v1 with 1 . 0 success rate, and PnP-Simple-v with 0 . 9 .

Original: This result indicates that the basic characteristics of the required robotic behaviours aresimilar in both versions, and PnP-Simple-v2 is not drastically harder to perform than PnP-Simple-v1 .
Revised: This result indicates that the basic characteristics of the required robotic behaviours are similar in both versions, and PnP-Simple-v2 is not drastically harder to perform than PnP-Simple-v1 .

Original: However, HER-b fails on both versions suggesting that PnP-Simple-v2 is harder to learn for HER.
Revised: However, HER-b fails on both versions suggesting that PnP-Simple-v2 is harder to learn for HER.

Original: This is explained by the fact that easy-to-solve episodes are seen less frequently in PnP-Simple-v2than in PnP-Simple-v1 .
Revised: This is explained by the fact that easy-to-solve episodes are seen less frequently in PnP-Simple-v than in PnP-Simple-v1 .

Original: Figure 4 also demonstrates the performances of all 7 approaches on bothversions.
Revised: Figure 4 also demonstrates the performances of all 7 approaches on both versions.

Original: It can be noted that our approach is robust to such differences in the environments at a smallcost of a slightly slower learning rate.
Revised: It can be noted that our approach is robust to such differences in the environments at a small cost of a slightly slower learning rate.

Original: We observe that this trade-off is generally compensated whencombining FO with SLDR , as reported in the following section.
Revised: We observe that this trade-off is generally compensated when combining FO with SLDR , as reported in the following section.

Original: Performance Evaluation on Other Environments: Before evaluating the performances, it is worthnoting a difference between Push and PnP tasks in terms of exploration requirements.
Revised: Performance Evaluation on Other Environments: Before evaluating the performances, it is worth noting a difference between Push and PnP tasks in terms of exploration requirements.

Original: Push variationsonly require horizontal exploration in the space of object positions.
Revised: Push variations only require horizontal exploration in the space of object positions.

Original: Even a random robot policymay change the object position in the x - and y - axes, and contribute to horizontal exploration.
Revised: Even a random robot policy may change the object position in the x - and y - axes, and contribute to horizontal exploration.

Original: PnP
Revised: PnP

Original: [Figure]
Revised: [Figure]

Original: variations, on the other hand, additionally require vertical exploration, which becomes possible onlyafter the robot has learned to grasp the object.
Revised: variations, on the other hand, additionally require vertical exploration, which becomes possible only after the robot has learned to grasp the object.

Original: In general, the PnP variations are associated withgreater exploration demands.
Revised: In general, the PnP variations are associated with greater exploration demands.

Original: However, PnP-Insertion proves to be easier than PnP-Obstacle and
Revised: However, PnP-Insertion proves to be easier than PnP-Obstacle and

Original: PnP-Shelf ; unlike the other 5 environments, in this task the initial and target object positions aresampled from intersecting sets (see Figure 3 for a visualisation.)
Revised: PnP-Shelf ; unlike the other 5 environments, in this task the initial and target object positions are sampled from intersecting sets (see Figure 3 for a visualisation.)

Original: The HER baseline fails in most cases as the exploration becomes harder.
Revised: The HER baseline fails in most cases as the exploration becomes harder.

Original: Interestingly, in somecases, it is outperformed by Shaped , which uses distance-based rewards as the only difference.
Revised: Interestingly, in some cases, it is outperformed by Shaped , which uses distance-based rewards as the only difference.

Original: RNDdemonstrates that curiosity-based exploration bonuses can improve the performance beyond HERas the exploration demand increases.
Revised: RND demonstrates that curiosity-based exploration bonuses can improve the performance beyond HER as the exploration demand increases.

Original: However, it fails in more difﬁcult cases such as PnP-Obstacleand PnP-Shelf .
Revised: However, it fails in more difﬁcult cases such as PnP-Obstacle and PnP-Shelf .

Original: One can also notice the ﬂuctuations in the learning curves of RND .
Revised: One can also notice the ﬂuctuations in the learning curves of RND .

Original: In terms ofcuriosity, the robot here would be rewarded when, for example, it throws the object out of the tablefor exploration.
Revised: In terms of curiosity, the robot here would be rewarded when, for example, it throws the object out of the table for exploration.

Original: However, these rewards can jeopardise the learning process towards the main task.
Revised: However, these rewards can jeopardise the learning process towards the main task.

Original: Although SLDR ’s manipulation-speciﬁc auxiliary rewards may seem as providing an advantage over
Revised: Although SLDR ’s manipulation-speciﬁc auxiliary rewards may seem as providing an advantage over

Original: RND , SLDR can also fail in more difﬁcult environments, and in general its performance is poorer than
Revised: RND , SLDR can also fail in more difﬁcult environments, and in general its performance is poorer than

Original: RND .
Revised: RND .

Original: We also observe that CHER fails to learn our manipulation tasks.
Revised: We also observe that CHER fails to learn our manipulation tasks.

Original: Overall, FO outperformsthese 5 approaches and achieves encouraging learning rates in all 6 environments.
Revised: Overall, FO outperforms these 5 approaches and achieves encouraging learning rates in all 6 environments.

Original: Moreover, thecombined approach FO+SLDR results in faster learning and has been proved to be the best approachoverall on these environments.
Revised: Moreover, the combined approach FO+SLDR results in faster learning and has been proved to be the best approach overall on these environments.

Original: This is explained by the fact that both approaches, individually,motivate the robot to follow the object policy.
Revised: This is explained by the fact that both approaches, individually, motivate the robot to follow the object policy.

Original: A potential limitation of the proposed approach is that not all the trajectories generated by the objectlocomotion policy are always guaranteed to be attainable by the robot.
Revised: A potential limitation of the proposed approach is that not all the trajectories generated by the object locomotion policy are always guaranteed to be attainable by the robot.

Original: For example, in Push tasks,occasionally the simulated object trajectories may pass over the obstacles, which clearly are notfeasible for the robot.
Revised: For example, in Push tasks, occasionally the simulated object trajectories may pass over the obstacles, which clearly are not feasible for the robot.

Original: Our investigations have indicated that including unattainable imaginary goalsdo not hinder the performance of the proposed methodology in any signiﬁcant way, and furtherdiscussions of these ﬁndings can be found in the Appendix.
Revised: Our investigations have indicated that including unattainable imaginary goals do not hinder the performance of the proposed methodology in any signiﬁcant way, and further discussions of these ﬁndings can be found in the Appendix.

Original:  Conclusions
Revised:  Conclusions

Original: We have proposed Follow the Object ( FO ), a new curriculum generation framework for learningrobotic manipulation tasks with only sparse rewards.
Revised: We have proposed Follow the Object ( FO ), a new curriculum generation framework for learning robotic manipulation tasks with only sparse rewards.

Original: Using an object locomotion policy, a predictivemodel provides a curriculum of intermediary imagined goals resulting in high learning success rate.
Revised: Using an object locomotion policy, a predictive model provides a curriculum of intermediary imagined goals resulting in high learning success rate.

Original: The object policies can be learned easily and realistically with a physics simulation engine.
Revised: The object policies can be learned easily and realistically with a physics simulation engine.

Original: Wehave demonstrated that FO is particularly beneﬁcial in difﬁcult manipulation problems requiringsubstantial exploration and can improve upon the performance obtained by existing algorithms.
Revised: We have demonstrated that FO is particularly beneﬁcial in difﬁcult manipulation problems requiring substantial exploration and can improve upon the performance obtained by existing algorithms.

Original: Infuture work, this framework can be extended to include more complex objects (e.g. ﬂexible objects)and manipulation tasks, and account for uncertainty in the imagined goals, e.g. by using a distributionof predicted goals.
Revised: In future work, this framework can be extended to include more complex objects (e.g. ﬂexible objects) and manipulation tasks, and account for uncertainty in the imagined goals, e.g. by using a distribution of predicted goals.