Original: S OLVING C OMPOSITIONAL R EINFORCEMENT L EARNING P ROBLEMS VIA T ASK R EDUCTION∗
Revised: S OLVING C OMPOSITIONAL R EINFORCEMENT L EARNING P ROBLEMS VIA T ASK R EDUCTION

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a novel learning paradigm, Self-Imitation via Reduction (SIR) , for solving compositional reinforcement learning problems.
Revised: We propose a novel learning paradigm, Self-Imitation via Reduction (SIR) , for solving compositional reinforcement learning problems.

Original: SIR is based on two core ideas: task reduction and self-imitation .
Revised: SIR is based on two core ideas: task reduction and self-imitation .

Original: Task reduction tackles a hard-to-solve task by actively reducing it to an easier task whose solution is known by the RL agent.
Revised: Task reduction tackles a hard-to-solve task by actively reducing it to an easier task whose solution is known by the RL agent.

Original: Once the original hard task is successfully solved by task reduction, the agent naturally obtains a self-generated solution trajectory to imitate.
Revised: Once the original hard task is successfully solved by task reduction, the agent naturally obtains a self-generated solution trajectory to imitate.

Original: By continuously collecting and imitating such demonstrations, the agent is able to progressively expand the solved subspace in the entire task space.
Revised: By continuously collecting and imitating such demonstrations, the agent is able to progressively expand the solved subspace in the entire task space.

Original: Experiment results show that SIR can signiﬁcantly accelerate and improve learning on a variety of challenging sparse-reward continuous-control problems with compositional structures.
Revised: Experiment results show that SIR can signiﬁcantly accelerate and improve learning on a variety of challenging sparse-reward continuous-control problems with compositional structures.

Revised: Code and videos are available at https://sites.google.com/ view/sir-compositional .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A large part of everyday human activities involves sequential tasks that have compositional structure, i.e., complex tasks which are built up in a systematic way from simpler tasks (Singh, 1991).
Revised: A large part of everyday human activities involves sequential tasks that have compositional structure, i.e., complex tasks which are built up in a systematic way from simpler tasks (Singh, 1991).

Original: Even in some very simple scenarios such as lifting an object, opening a door, sitting down, or driving a car, we usually decompose an activity into multiple steps and take multiple physical actions to ﬁnish each of the steps in our mind.
Revised: Even in some very simple scenarios such as lifting an object, opening a door, sitting down or driving a car, we usually decompose an activity into multiple steps and take multiple physical actions to ﬁnish each of the steps in our mind.

Original: Building an intelligent agent that can solve a wide range of compositional decision making problems as such remains a long-standing challenge in AI.
Revised: Building an intelligent agent that can solve a wide range of compositional decision making problems as such remains a long-standing challenge in AI.

Original: Deep reinforcement learning (RL) has recently shown promising capabilities for solving complex decision making problems.
Revised: Deep reinforcement learning (RL) has recently shown promising capabilities for solving complex decision making problems.

Original: But most approaches for these compositional challenges either rely on carefully designed reward function (Warde-Farley et al., 2019;
Revised: But most approaches for these compositional challenges either rely on carefully designed reward function (Warde-Farley et al., 2019;

Original: Yu et al., 2019; Li et al., 2020; OpenAI et al., 2019), which is typically subtle to derive and requires strong domain knowledge, or utilize a hierarchical policy structure (Kulkarni et al., 2016; Bacon et al., 2017; Nachum et al., 2018; Lynch et al., 2019; Bagaria & Konidaris, 2020), which typically assumes a set of low-level policies for skills and a high-level policy for choosing the next skill to use.
Revised: Yu et al., 2019; Li et al., 2020; OpenAI et al., 2019), which is typically subtle to derive and requires strong domain knowledge, or utilize a hierarchical policy structure (Kulkarni et al., 2016; Bacon et al., 2017; Nachum et al., 2018; Lynch et al., 2019; Bagaria & Konidaris, 2020), which typically assumes a set of low-level policies for skills and a high-level policy for choosing the next skill to use.

Original: Although the policy hierarchy introduces structural inductive bias into RL, effective low-level skills can be non-trivial to obtain and the bi-level policy structure often causes additional optimization difﬁculties in practice.
Revised: Although the policy hierarchy introduces structural inductive bias into RL, effective low-level skills can be non-trivial to obtain and the bi-level policy structure often causes additional optimization difﬁculties in practice.

Original: In this paper, we propose a novel RL paradigm, Self-Imitation via Reduction (SIR) , which (1) naturally works on sparse-reward problems with compositional structures, and (2) does not impose any structural requirement on policy representation.
Revised: In this paper, we propose a novel RL paradigm, Self-Imitation via Reduction (SIR) , which (1) naturally works on sparse-reward problems with compositional structures, and (2) does not impose any structural requirement on policy representation.

Original: SIR has two critical components, task reduction and imitation learning .
Revised: SIR has two critical components, task reduction and imitation learning .

Original: Task reduction leverages the compositionality in a parameterized task space and tackles an unsolved hard task by actively “ simplifying ” it to an easier one whose solution is known already.
Revised: Task reduction leverages the compositionality in a parameterized task space and tackles an unsolved hard task by actively “ simplifying ” it to an easier one whose solution is known already.

Original: When a hard task is successfully accomplished via task reduction, we can further run imitation learning on the obtained solution trajectories to signiﬁcantly accelerate training.
Revised: When a hard task is successfully accomplished via task reduction, we can further run imitation learning on the obtained solution trajectories to signiﬁcantly accelerate training.

Original: Fig.
Revised: Fig.

Original: 1 illustrates task reduction in a pushing scenario with an elongated box (brown) and a cubic box (blue).
Revised: 1 illustrates task reduction in a pushing scenario with an elongated box (brown) and a cubic box (blue).

Original: Consider pushing the cubic box to the goal position (red).
Revised: Consider pushing the cubic box to the goal position (red).

Original: The difﬁculty comes with the wall on the table which contains a small door in the middle.
Revised: The difﬁculty comes with the wall on the table which contains a small door in the middle.

Original: When the door is clear (Fig. 1a), the task is easy and straightforward to solve.
Revised: When the door is clear (Fig. 1a), the task is easy and straightforward to solve.

Original: However, when the door is blocked by the elongated box (Fig. 1b), the task becomes signiﬁcantly harder: the robot needs to clear the door before it can start pushing the
Revised: However, when the door is blocked by the elongated box (Fig. 1b), the task becomes signiﬁcantly harder: the robot needs to clear the door before it can start pushing the

Original: [Figure]
Revised: [Figure]

Original: cubic box.
Revised: cubic box.

Original: Such a compositional strategy is non-trivial to discover by directly running standard RL.
Revised: Such a compositional strategy is non-trivial to discover by directly running standard RL.

Original: By contrast, this solution is much easier to derive via task reduction: starting from an initial state with the door blocked (Fig. 1b), the agent can ﬁrst imagine a simpliﬁed task that has the same goal but starts from a different initial state with a clear door (Fig. 1a); then the agent can convert to this simpliﬁed task by physically perturbing the elongate box to reach the desired initial state (Fig. 1c); ﬁnally, a straightforward push completes the solution to the original hard task (Fig. 1b).
Revised: By contrast, this solution is much easier to derive via task reduction: starting from an initial state with the door blocked (Fig. 1b), the agent can ﬁrst imagine a simpliﬁed task that has the same goal but starts from a different initial state with a clear door (Fig. 1a); then the agent can convert to this simpliﬁed task by physically perturbing the elongate box to reach the desired initial state (Fig. 1c); ﬁnally, a straightforward push completes the solution to the original hard task (Fig. 1b).

Original: Notably, by running imitation learning on the composite reduction trajectories, we effectively incorporate the inductive bias of compositionality into the learned policy without the need of explicitly specifying any low-level skills or options, which signiﬁcantly simpliﬁes policy optimization.
Revised: Notably, by running imitation learning on the composite reduction trajectories, we effectively incorporate the inductive bias of compositionality into the learned policy without the need of explicitly specifying any low-level skills or options, which signiﬁcantly simpliﬁes policy optimization.

Original: Moreover, although task reduction only performs 1-step planning from a planning perspective, SIR still retains the capability of learning an arbitrarily complex policy by alternating between imitation and reduction: as more tasks are solved, these learned tasks can recursively further serve as new reduction targets for unsolved ones in the task space, while the policy gradually generalizes to increasingly harder tasks by imitating solution trajectories with growing complexity.
Revised: Moreover, although task reduction only performs 1-step planning from a planning perspective, SIR still retains the capability of learning an arbitrarily complex policy by alternating between imitation and reduction: as more tasks are solved, these learned tasks can recursively further serve as new reduction targets for unsolved ones in the task space, while the policy gradually generalizes to increasingly harder tasks by imitating solution trajectories with growing complexity.

Original: We implement SIR by jointly learning a goal-conditioned policy and a universal value function (Schaul et al., 2015), so that task reduction can be accomplished via state search over the value function and policy execution with different goals.
Revised: We implement SIR by jointly learning a goal-conditioned policy and a universal value function (Schaul et al., 2015), so that task reduction can be accomplished via state search over the value function and policy execution with different goals.

Original: Empirical results show that SIR can signiﬁcantly accelerate and improve policy learning on challenging sparse-reward continuous-control problems, including robotics pushing, stacking and maze navigation, with both object-based and visual state space.
Revised: Empirical results show that SIR can signiﬁcantly accelerate and improve policy learning on challenging sparse-reward continuous-control problems, including robotics pushing, stacking and maze navigation, with both object-based and visual state space.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Hierarchical RL (HRL) (Barto & Mahadevan, 2003) is perhaps the most popular approach for solving compositional RL problems, which assumes a low-level policy for primitive skills and a high-level policy for sequentially proposing subgoals.
Revised: Hierarchical RL (HRL) (Barto & Mahadevan, 2003) is perhaps the most popular approach for solving compositional RL problems, which assumes a low-level policy for primitive skills and a high-level policy for sequentially proposing subgoals.

Original: Such a bi-level policy structure reduces the planning horizon for each policy but raises optimization challenges.
Revised: Such a bi-level policy structure reduces the planning horizon for each policy but raises optimization challenges.

Original: For example, many HRL works require pretraining low-level skills (Singh, 1991;
Revised: For example, many HRL works require pretraining low-level skills (Singh, 1991;

Original: Kulkarni et al., 2016; Florensa et al., 2017;
Revised: Kulkarni et al., 2016; Florensa et al., 2017;

Original: Riedmiller et al., 2018; Haarnoja et al., 2018a) or non-trivial algorithmic modiﬁcations to learn both modules jointly (Dayan & Hinton, 1993; Silver & Ciosek, 2012; Bacon et al., 2017; Vezhnevets et al., 2017; Nachum et al., 2018).
Revised: Riedmiller et al., 2018; Haarnoja et al., 2018a) or non-trivial algorithmic modiﬁcations to learn both modules jointly (Dayan & Hinton, 1993; Silver & Ciosek, 2012; Bacon et al., 2017; Vezhnevets et al., 2017; Nachum et al., 2018).

Original: On the contrary, our paradigm works with any policy architecture of enough representation power by imitating composite trajectories.
Revised: On the contrary, our paradigm works with any policy architecture of enough representation power by imitating composite trajectories.

Original: Notably, such an idea of exploiting regularities on behavior instead of constraints on policy structures can be traced back to the “ Hierarchies of Machines ” model in 1998 (Parr & Russell, 1998).
Revised: Notably, such an idea of exploiting regularities on behavior instead of constraints on policy structures can be traced back to the “ Hierarchies of Machines ” model in 1998 (Parr & Russell, 1998).

Original: Our approach is also related to the options framework (Sutton et al., 1999; Precup, 2000), which explicitly decomposes a complex task into a sequence of temporal abstractions, i.e., options.
Revised: Our approach is also related to the options framework (Sutton et al., 1999; Precup, 2000), which explicitly decomposes a complex task into a sequence of temporal abstractions, i.e., options.

Original: The options framework also assumes a hierarchical representation with an inter-option controller and a discrete set of reusable intra-option policies, which are typically non-trivial to obtain (Konidaris &
Revised: The options framework also assumes a hierarchical representation with an inter-option controller and a discrete set of reusable intra-option policies, which are typically non-trivial to obtain (Konidaris &

Original: Barto, 2009; Konidaris et al., 2012; Bagaria & Konidaris, 2020; Wulfmeier et al., 2020).
Revised: Barto, 2009; Konidaris et al., 2012; Bagaria & Konidaris, 2020; Wulfmeier et al., 2020).

Original: SIR implicitly reuses previously learned “skills” via task reduction and distills all the composite strategies into a single policy.
Revised: SIR implicitly reuses previously learned “skills” via task reduction and distills all the composite strategies into a single policy.

Original: Besides, there are other works tackling different problems with conceptually similar composition techniques.
Revised: Besides, there are other works tackling different problems with conceptually similar composition techniques.

Original: Probabilistic policy reuse (Fern´andez-Rebollo & Veloso, 2006; Fern´andezRebollo et al., 2010) executes a discrete set of previously learned policies for structural exploration.
Revised: Probabilistic policy reuse (Fern´andez-Rebollo & Veloso, 2006; Fern´andezRebollo et al., 2010) executes a discrete set of previously learned policies for structural exploration.

Original: Fang et al.
Revised: Fang et al.

Original: (2019) consider a dynamic goal setting without manipulable objects and compose an agent trajectory and a goal trajectory that collide as supervision.
Revised: (2019) consider a dynamic goal setting without manipulable objects and compose an agent trajectory and a goal trajectory that collide as supervision.

Original: Dhiman et al.
Revised: Dhiman et al.

Original: (2018) transfer previous knowledge to new goals in tabular cases by constraining value functions.
Revised: (2018) transfer previous knowledge to new goals in tabular cases by constraining value functions.

Original: Task reduction produces an expanding solved task space, which is conceptually related to life-long learning (Tessler et al., 2017) and open-ended learning (Wang et al., 2019).
Revised: Task reduction produces an expanding solved task space, which is conceptually related to life-long learning (Tessler et al., 2017) and open-ended learning (Wang et al., 2019).

Original: Evidently, many recent advances have shown that with generic architectures, e.g., convolution (LeCun et al., 1995) and self-attention (Vaswani et al., 2017), simply training on massive data yields surprisingly intelligent agents for image generation (Chen et al., 2020), language modeling (Brown et al., 2020), dialogue (Adiwardana et al., 2020) and gaming (Schrittwieser et al., 2019; Baker et al., 2020).
Revised: Evidently, many recent advances have shown that with generic architectures, e.g., convolution (LeCun et al., 1995) and self-attention (Vaswani et al., 2017), simply training on massive data yields surprisingly intelligent agents for image generation (Chen et al., 2020), language modeling (Brown et al., 2020), dialogue (Adiwardana et al., 2020) and gaming (Schrittwieser et al., 2019; Baker et al., 2020).

Original: Hence, we suggest building intelligent RL agents via a similar principle, i.e., design a diverse task spectrum and then perform supervised learning on complex strategies.
Revised: Hence, we suggest building intelligent RL agents via a similar principle, i.e., design a diverse task spectrum and then perform supervised learning on complex strategies.

Original: The technical foundation of SIR is to use a universal value function and a goal conditioned policy (Kaelbling, 1993; Schaul et al., 2015), which has been successfully applied in various domains, including tackling sparse rewards (Andrychowicz et al., 2017), visual navigation (Zhu et al., 2017), planning (Srinivas et al., 2018), imitation learning (Ding et al., 2019) and continuous control (Nair et al., 2018).
Revised: The technical foundation of SIR is to use a universal value function and a goal conditioned policy (Kaelbling, 1993; Schaul et al., 2015), which has been successfully applied in various domains, including tackling sparse rewards (Andrychowicz et al., 2017), visual navigation (Zhu et al., 2017), planning (Srinivas et al., 2018), imitation learning (Ding et al., 2019) and continuous control (Nair et al., 2018).

Original: Besides, task reduction requires state-space search/planning over an approximate value function.
Revised: Besides, task reduction requires state-space search/planning over an approximate value function.

Original: Similar ideas of approximate state search have been also explored (Pathak et al., 2018;
Revised: Similar ideas of approximate state search have been also explored (Pathak et al., 2018;

Original: Eysenbach et al., 2019; Nasiriany et al., 2019;
Revised: Eysenbach et al., 2019; Nasiriany et al., 2019;

Original: Ren et al., 2019; Nair & Finn, 2020).
Revised: Ren et al., 2019; Nair & Finn, 2020).

Original: SIR accelerates learning by imitating its own behavior, which is typically called self-imitation (Oh et al., 2018;
Revised: SIR accelerates learning by imitating its own behavior, which is typically called self-imitation (Oh et al., 2018;

Original: Ding et al., 2019; Lynch et al., 2019) or self-supervision (Nair et al., 2018; 2019; Pong et al., 2019; Ghosh et al., 2019).
Revised: Ding et al., 2019; Lynch et al., 2019) or self-supervision (Nair et al., 2018; 2019; Pong et al., 2019; Ghosh et al., 2019).

Original: In this paper, we use techniques similar to Rajeswaran et al.
Revised: In this paper, we use techniques similar to Rajeswaran et al.

Original: and Oh et al.
Revised: and Oh et al.

Original: (2018), but
Revised: (2018), but

Original: in general, SIR does not rely on any particular imitation learning algorithm.
Revised: in general, SIR does not rely on any particular imitation learning algorithm.

Original: There are also related works on compositional problems of orthogonal interests, such as learning to smoothly compose skills (Lee et al., 2019), composing skills via embedding arithmetic (Sahni et al., 2017; Devin et al., 2019), composing skills in real-time (Peng et al., 2019), following instructions (Andreas et al., 2017; Jiang et al., 2019) and manually constructing “iterative-reﬁnement” strategies to address supervised learning tasks (Nguyen et al., 2017; Hjelm et al., 2016; Chang et al., 2019).
Revised: There are also related works on compositional problems of orthogonal interests, such as learning to smoothly compose skills (Lee et al., 2019), composing skills via embedding arithmetic (Sahni et al., 2017; Devin et al., 2019), composing skills in real-time (Peng et al., 2019), following instructions (Andreas et al., 2017; Jiang et al., 2019) and manually constructing “iterative-reﬁnement” strategies to address supervised learning tasks (Nguyen et al., 2017; Hjelm et al., 2016; Chang et al., 2019).

Original: Conceptually, our method is also related to curriculum learning (CL).
Revised: Conceptually, our method is also related to curriculum learning (CL).

Original: Many automatic CL methods (Graves et al., 2017; Florensa et al., 2018; Matiisen et al., 2019; Portelas et al., 2019;
Revised: Many automatic CL methods (Graves et al., 2017; Florensa et al., 2018; Matiisen et al., 2019; Portelas et al., 2019;

Original: Racaniere et al., 2020) also assume a task space and explicitly construct task curricula from easy cases to hard ones over the space.
Revised: Racaniere et al., 2020) also assume a task space and explicitly construct task curricula from easy cases to hard ones over the space.

Original: In our paradigm, task reduction reduces a hard task to a simpler solved task, so the reduction trace over the solved tasks can be viewed as an implicit task curriculum.
Revised: In our paradigm, task reduction reduces a hard task to a simpler solved task, so the reduction trace over the solved tasks can be viewed as an implicit task curriculum.

Original: Such a phenomenon of an implicitly ever-expanding solved subspace has been also reported in other works for tackling goal-conditioned RL problems (Warde-Farley et al., 2019; Lynch et al., 2019).
Revised: Such a phenomenon of an implicitly ever-expanding solved subspace has been also reported in other works for tackling goal-conditioned RL problems (Warde-Farley et al., 2019; Lynch et al., 2019).

Original: Despite the conceptual similarity, there are major differences between SIR and CL.
Revised: Despite the conceptual similarity, there are major differences between SIR and CL.

Original: SIR assumes a given task sampler and primarily focuses on efﬁciently ﬁnding a policy to a given task via task decomposition.
Revised: SIR assumes a given task sampler and primarily focuses on efﬁciently ﬁnding a policy to a given task via task decomposition.

Original: By contrast, CL focuses on task generation and typically treats policy learning as a black box.
Revised: By contrast, CL focuses on task generation and typically treats policy learning as a black box.

Original: We also show in the experiment section that SIR is complementary to CL training and can solve a challenging sparse-reward stacking task (Sec. 5.2) that was unsolved by curriculum learning alone.
Revised: We also show in the experiment section that SIR is complementary to CL training and can solve a challenging sparse-reward stacking task (Sec. 5.2) that was unsolved by curriculum learning alone.

Original: We remark that it is also possible to extend SIR to a full CL method and leave it as future work.
Revised: We remark that it is also possible to extend SIR to a full CL method and leave it as future work.

Original:  P RELIMINARY AND N OTATIONS
Revised:  P RELIMINARY AND N OTATIONS

Original: We consider a multi-task reinforcement learning setting over a parameterized task space T .
Revised: We consider a multi-task reinforcement learning setting over a parameterized task space T .

Original: Each task T ∈ T is represented as a goal-conditioned Markov Decision Process (MDP),
Revised: Each task T ∈ T is represented as a goal-conditioned Markov Decision Process (MDP),

Original: T = ( S , A , s T , g T , r ( s, a ; g T ) , P ( s (cid:48) | s, a ; g T )) , where S is the state space,
Revised: T = ( S , A , s T , g T , r ( s, a ; g T ) , P ( s (cid:48) | s, a ; g T )) , where S is the state space,

Original: A is the action space, s T , g T ∈ S are the initial state and the goal to reach, r ( s, a ; g T ) is the reward function of task T , and P ( s (cid:48) | s, a ; g T ) is the transition probability.
Revised: A is the action space, s T , g T ∈ S are the initial state and the goal to reach, r ( s, a ; g T ) is the reward function of task T , and P ( s (cid:48) | s, a ; g T ) is the transition probability.

Original: Particularly, we consider a sparse reward setting, i.e., r ( s, a ; g T ) =
Revised: Particularly, we consider a sparse reward setting, i.e., r ( s, a ; g T ) =

Original: I [ d ( s, g T ) ≤ δ ] , where the agent only receives a non-zero reward when the goal is achieved w.r.t.
Revised: I [ d ( s, g T ) ≤ δ ] , where the agent only receives a non-zero reward when the goal is achieved w.r.t.

Original: some distance metric d ( · , · ) .
Revised: some distance metric d ( · , · ) .

Original: We assume that the state space and the action space are shared across tasks, and different tasks share the same physical dynamics (i.e., g T only inﬂuences the termination condition of the transition probability P ), so a task T is determined by its initial state s T and goal g T .
Revised: We assume that the state space and the action space are shared across tasks, and different tasks share the same physical dynamics (i.e., g T only inﬂuences the termination condition of the transition probability P ), so a task T is determined by its initial state s T and goal g T .

Original: We also assume a task sampler C , which samples training tasks from T .
Revised: We also assume a task sampler C , which samples training tasks from T .

Original: C can be simply a uniform sampler or something more intelligent.
Revised: C can be simply a uniform sampler or something more intelligent.

Original: For evaluation, we are interested in the most challenging tasks in T .
Revised: For evaluation, we are interested in the most challenging tasks in T .

Original: The learning agent is represented as a stochastic policy π θ ( a | s, g T ) conditioning on state s and goal g T and parameterized by θ .
Revised: The learning agent is represented as a stochastic policy π θ ( a | s, g T ) conditioning on state s and goal g T and parameterized by θ .

Original: We aim to ﬁnd the optimal parameter θ (cid:63) such that policy π θ (cid:63) maximizes the expected accumulative reward across every task T ∈ T with a discounted factor γ , i.e., θ (cid:63) =
Revised: We aim to ﬁnd the optimal parameter θ (cid:63) such that policy π θ (cid:63) maximizes the expected accumulative reward across every task T ∈ T with a discounted factor γ , i.e., θ (cid:63) =

Original: arg max θ E [ (cid:80) t γ t r ( s t , a t ; g T )] where a t ∼ π θ ( a | s t , g T ) .
Revised: arg max θ E [ (cid:80) t γ t r ( s t , a t ; g T )] where a t ∼ π θ ( a | s t , g T ) .

Original: Correspondingly, we also learn a universal value function V ψ ( s t , g T ) for state s t and goal g T parameterized by ψ w.r.t.
Revised: Correspondingly, we also learn a universal value function V ψ ( s t , g T ) for state s t and goal g T parameterized by ψ w.r.t.

Original: π θ .
Revised: π θ .

Original: The subscript θ in π θ and ψ in V ψ will be omitted for simplicity in the following content.
Revised: The subscript θ in π θ and ψ in V ψ will be omitted for simplicity in the following content.

Original:  M ETHOD 4. T ASK R EDUCTION
Revised:  M ETHOD 4. T ASK R EDUCTION

Original: The core idea of task reduction is to convert a hard-to-solve task A ∈ T to some task B ∈ T that the agent has learned to solve, so that the original task A can be accomplished via composing two simpler tasks, i.e., (1) reducing task A to B and (2) solving the reduced task B .
Revised: The core idea of task reduction is to convert a hard-to-solve task A ∈ T to some task B ∈ T that the agent has learned to solve, so that the original task A can be accomplished via composing two simpler tasks, i.e., (1) reducing task A to B and (2) solving the reduced task B .

Original: More formally, for a task
Revised: More formally, for a task

Original: A with initial state s A and goal g A that the current policy π cannot solve, we want to ﬁnd a reduction target B with initial state s B and goal g B such that (1) the task to reach state s B from s A is easy, and (2) B has the same goal as A (i.e., g B = g A ) and solving task B is easy.
Revised: A with initial state s A and goal g A that the current policy π cannot solve, we want to ﬁnd a reduction target B with initial state s B and goal g B such that (1) the task to reach state s B from s A is easy, and (2) B has the same goal as A (i.e., g B = g A ) and solving task B is easy.

Original: A natural measure of how easy a task is under π is to evaluate the universal value function V ( · ) .
Revised: A natural measure of how easy a task is under π is to evaluate the universal value function V ( · ) .

Original: Hence, task reduction aims to ﬁnd the best target state s (cid:63)B , which is the initial state of some task B , such that
Revised: Hence, task reduction aims to ﬁnd the best target state s (cid:63)B , which is the initial state of some task B , such that

Original: [Equation]
Revised: [Equation]

Original: , (1) where ⊕ is some composition operator and V ( s A , s B , g A ) denotes the composite value w.r.t a particular target state s B .
Revised: , (1) where ⊕ is some composition operator and V ( s A , s B , g A ) denotes the composite value w.r.t a particular target state s B .

Original: There are multiple choices for ⊕ , including minimum value, average or product (due to 0/1 sparse rewards).
Revised: There are multiple choices for ⊕ , including minimum value, average or product (due to 0/1 sparse rewards).

Original: Empirically, taking the product of two values leads to the best practical performance.
Revised: Empirically, taking the product of two values leads to the best practical performance.

Original: Thus, we assume V ( s A , s B , g A ) :
Revised: Thus, we assume V ( s A , s B , g A ) :

Original: =
Revised: =

Original: V ( s A , s B ) · V ( s B , g A ) in this paper.
Revised: V ( s A , s B ) · V ( s B , g A ) in this paper.

Original: With the optimal target state s (cid:63)B obtained, the agent can perform task reduction by executing the policy π ( a | s A , s (cid:63)B ) from the initial state s A .
Revised: With the optimal target state s (cid:63)B obtained, the agent can perform task reduction by executing the policy π ( a | s A , s (cid:63)B ) from the initial state s A .

Original: If this policy execution leads to some successful state s (cid:48)
Revised: If this policy execution leads to some successful state s (cid:48)

Original: B (i.e., within a threshold to s (cid:63)B ), the agent can continue executing π ( a | s (cid:48) B , g A ) from state s (cid:48)
Revised: B (i.e., within a threshold to s (cid:63)B ), the agent can continue executing π ( a | s (cid:48) B , g A ) from state s (cid:48)

Original: B towards the ultimate goal g A .
Revised: B towards the ultimate goal g A .

Original: Note that it may not always be possible to apply task reduction successfully since either of these two policy executions can be a failure.
Revised: Note that it may not always be possible to apply task reduction successfully since either of these two policy executions can be a failure.

Original: There are two causes of failure: (i) the approximation error of the value function; (ii) task
Revised: There are two causes of failure: (i) the approximation error of the value function; (ii) task

Original: A is merely too hard for the current policy.
Revised: A is merely too hard for the current policy.

Original: For cause (i), the error is intrinsic and cannot be even realized until we actually execute the policy in the environment; while for cause (ii), we are able to improve sample efﬁciency by only performing task reduction when it is likely to succeed, e.g., when V ( s A , s (cid:63)B , g A ) exceeds some threshold σ .
Revised: For cause (i), the error is intrinsic and cannot be even realized until we actually execute the policy in the environment; while for cause (ii), we are able to improve sample efﬁciency by only performing task reduction when it is likely to succeed, e.g., when V ( s A , s (cid:63)B , g A ) exceeds some threshold σ .

Original: 4. S ELF -I MITATION
Revised: 4. S ELF -I MITATION

Original: Once task reduction is successfully applied to a hard task A via a target state s (cid:63)B , we obtain two trajectories, one for reducing A to B with ﬁnal state s (cid:48) B , and the other for the reduced task B starting at s (cid:48) B .
Revised: Once task reduction is successfully applied to a hard task A via a target state s (cid:63)B , we obtain two trajectories, one for reducing A to B with ﬁnal state s (cid:48) B , and the other for the reduced task B starting at s (cid:48) B .

Original: Concatenating these two trajectories yields a successful demonstration for the original task
Revised: Concatenating these two trajectories yields a successful demonstration for the original task

Original: A .
Revised: A .

Original: Therefore, by continuously collecting successful trajectories produced by task reduction, we can perform imitation learning on these self-demonstrations.
Revised: Therefore, by continuously collecting successful trajectories produced by task reduction, we can perform imitation learning on these self-demonstrations.

Original: Suppose demonstration dataset is denoted by D il = { ( s, a, g ) } .
Revised: Suppose demonstration dataset is denoted by D il = { ( s, a, g ) } .

Original: We utilize a weighted imitation learning objective (Oh et al., 2018) as follows:
Revised: We utilize a weighted imitation learning objective (Oh et al., 2018) as follows:

Original: [Equation]
Revised: [Equation]

Original: where A ( s, g ) + denotes the clipped advantage, i.e., max(0 , R − V ( s, g )) .
Revised: where A ( s, g ) + denotes the clipped advantage, i.e., max(0 , R − V ( s, g )) .

Original: Self-imitation allows the agent to more efﬁciently solve harder tasks by explicitly using past experiences on easy tasks.
Revised: Self-imitation allows the agent to more efﬁciently solve harder tasks by explicitly using past experiences on easy tasks.

Original: 4. SIR: S ELF -I MITATION VIA R EDUCTION
Revised: 4. SIR: S ELF -I MITATION VIA R EDUCTION

Original: We present SIR by combining deep RL and self-imitation on reduction demonstrations.
Revised: We present SIR by combining deep RL and self-imitation on reduction demonstrations.

Original: SIR is compatible with both off-policy and on-policy RL.
Revised: SIR is compatible with both off-policy and on-policy RL.

Original: We use soft actor-critic (Haarnoja et al., 2018b) with hindsight experience replay (Andrychowicz et al., 2017) (SAC) and proximal policy gradient (PPO)
Revised: We use soft actor-critic (Haarnoja et al., 2018b) with hindsight experience replay (Andrychowicz et al., 2017) (SAC) and proximal policy gradient (PPO)

Original: (Schulman et al., 2017) respectively in this paper.
Revised: (Schulman et al., 2017) respectively in this paper.

Original: In general, an RL algorithm trains an actor and a critic with rollout data D rl from the environment or the replay buffer by optimizing algorithm-speciﬁc actor loss L actor ( θ ; D rl ) and critic loss L critic ( ψ ; D rl ) .
Revised: In general, an RL algorithm trains an actor and a critic with rollout data D rl from the environment or the replay buffer by optimizing algorithm-speciﬁc actor loss L actor ( θ ; D rl ) and critic loss L critic ( ψ ; D rl ) .

Original: For policy learning, since our reduction demonstrations D il are largely off-policy, it is straightforward to incorporate D il in the off-policy setting: we can simply aggregate the demonstrations into the experience replay and jointly optimize the self-imitation loss and the policy learning loss by
Revised: For policy learning, since our reduction demonstrations D il are largely off-policy, it is straightforward to incorporate D il in the off-policy setting: we can simply aggregate the demonstrations into the experience replay and jointly optimize the self-imitation loss and the policy learning loss by

Original: [Equation]
Revised: [Equation]

Original: While in the on-policy setting, we only perform imitation on demonstrations.
Revised: While in the on-policy setting, we only perform imitation on demonstrations.

Original: Since the size of D il is typically much smaller than D rl , we utilize the following per-instance loss to prevent overﬁt on D il :
Revised: Since the size of D il is typically much smaller than D rl , we utilize the following per-instance loss to prevent overﬁt on D il :

Original: [Equation]
Revised: [Equation]

Original: For critic learning, since the objective is consistent with both data sources, we can simply optimize
Revised: For critic learning, since the objective is consistent with both data sources, we can simply optimize

Original: L critic ( ψ ; D rl ∪ D il ) .
Revised: L critic ( ψ ; D rl ∪ D il ) .

Original: The proposed algorithm is summarized in Algo.
Revised: The proposed algorithm is summarized in Algo.

Original: 4. I MPLEMENTATION AND E XTENSIONS
Revised: 4. I MPLEMENTATION AND E XTENSIONS

Original: Algorithm initialize π θ , V ψ ; D rl ← ∅ , D il ← ∅ ; repeat τ ← trajectory by π θ in task A from
Revised: Algorithm initialize π θ , V ψ ; D rl ← ∅ , D il ← ∅ ; repeat τ ← trajectory by π θ in task A from

Original: C ;
Revised: C ;

Original: D rl ← D rl + { τ } ; if τ is a failure then ﬁnd target state s (cid:63)B w.r.t.
Revised: D rl ← D rl + { τ } ; if τ is a failure then ﬁnd target state s (cid:63)B w.r.t.

Original: Eq.
Revised: Eq.

Original: (1); if V ( s A , s (cid:63)B , g A ) >
Revised: (1); if V ( s A , s (cid:63)B , g A ) >

Original: σ then τ (cid:48) ← trajectory by reduction; if τ (cid:48) is a success then
Revised: σ then τ (cid:48) ← trajectory by reduction; if τ (cid:48) is a success then

Original: D il ← D il + { τ (cid:48) } ; if enough new data collected then θ ← θ + α ∇ L SIR ( θ ; D rl , D il ) ; ψ ← ψ + α ∇ L critic ( ψ ; D rl ∪D il ) ; until enough iterations ; return π θ , V ψ ;
Revised: D il ← D il + { τ (cid:48) } ; if enough new data collected then θ ← θ + α ∇ L SIR ( θ ; D rl , D il ) ; ψ ← ψ + α ∇ L critic ( ψ ; D rl ∪D il ) ; until enough iterations ; return π θ , V ψ ;

Original: Task Space: We primarily consider object-based state space, where a goal state is determined by whether an object reaches some desired location.
Revised: Task Space: We primarily consider object-based state space, where a goal state is determined by whether an object reaches some desired location.

Original: We also extend our method to image input by learning a low-dimensional latent representation (Higgins et al., 2017b) and then perform task reduction over the latent states (see Sec. 5.3).
Revised: We also extend our method to image input by learning a low-dimensional latent representation (Higgins et al., 2017b) and then perform task reduction over the latent states (see Sec. 5.3).

Original: We do not consider strong partial-observability (Pinto et al., 2018) in this paper and leave it as future work.
Revised: We do not consider strong partial-observability (Pinto et al., 2018) in this paper and leave it as future work.

Original: Reduction-Target Search:
Revised: Reduction-Target Search:

Original: In our implementation, we simply perform a random search to ﬁnd the topcandidates for s (cid:63)B in Eq.
Revised: In our implementation, we simply perform a random search to ﬁnd the topcandidates for s (cid:63)B in Eq.

Original: (1) and try both of them.
Revised: (1) and try both of them.

Original: We also use the cross-entropy method (De Boer et al., 2005) for efﬁcient planning over the latent space with image input (Sec. 5.3).
Revised: We also use the cross-entropy method (De Boer et al., 2005) for efﬁcient planning over the latent space with image input (Sec. 5.3).

Original: Alternative planning methods can be also applied, such as Bayesian optimization (Snoek et al., 2012; Swersky et al., 2014), gradient descent (Srinivas et al., 2018), or learning a neural planner (Wang et al., 2018; Nair et al., 2019).
Revised: Alternative planning methods can be also applied, such as Bayesian optimization (Snoek et al., 2012; Swersky et al., 2014), gradient descent (Srinivas et al., 2018), or learning a neural planner (Wang et al., 2018; Nair et al., 2019).

Original: We emphasize that the reduction search is performed completely on the learned value function, which does not require any environment interactions .
Revised: We emphasize that the reduction search is performed completely on the learned value function, which does not require any environment interactions .

Original: Particularly in the object-centric state space, an extremely effective strategy for reducing the search space is to constrain the target s B to differ from the current state s A by just a few objects (e.g., 1 object in our applications).
Revised: Particularly in the object-centric state space, an extremely effective strategy for reducing the search space is to constrain the target s B to differ from the current state s A by just a few objects (e.g., 1 object in our applications).

Original: Due to the compositionality over objects, the agent can still learn arbitrarily complex behavior recursively throughout the training.
Revised: Due to the compositionality over objects, the agent can still learn arbitrarily complex behavior recursively throughout the training.

Original: When performing task reduction in training, SIR explicitly resets the initial state and goal of the current task, which is a very common setting in curriculum learning and goal-conditioned policy learning.
Revised: When performing task reduction in training, SIR explicitly resets the initial state and goal of the current task, which is a very common setting in curriculum learning and goal-conditioned policy learning.

Original: It is possible to avoid resets by archiving solutions to solved tasks to approximate a reduction trajectory.
Revised: It is possible to avoid resets by archiving solutions to solved tasks to approximate a reduction trajectory.

Original: We leave it as future work.
Revised: We leave it as future work.

Original: Estimating σ : We use a static σ for SAC.
Revised: Estimating σ : We use a static σ for SAC.

Original: For PPO, we empirically observe that, as training proceeds, the output scale of the learned value function keeps increasing.
Revised: For PPO, we empirically observe that, as training proceeds, the output scale of the learned value function keeps increasing.

Original: Hence, we maintain an online estimate for σ , i.e., a running average of the composite values over all the selected reduction targets.
Revised: Hence, we maintain an online estimate for σ , i.e., a running average of the composite values over all the selected reduction targets.

Original: Imitation Learning: Note that there can be multiple feasible reduction targets for a particular task.
Revised: Imitation Learning: Note that there can be multiple feasible reduction targets for a particular task.

Original: Hence, the self-demonstrations can be highly multi-modal.
Revised: Hence, the self-demonstrations can be highly multi-modal.

Original: This issue is out of the scope of this paper but it can be possibly addressed through some recent advances in imitation learning (Ho & Ermon, 2016;
Revised: This issue is out of the scope of this paper but it can be possibly addressed through some recent advances in imitation learning (Ho & Ermon, 2016;

Original: Guo et al., 2019; Lynch et al., 2019).
Revised: Guo et al., 2019; Lynch et al., 2019).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We compare SIR with baselines without task reduction.
Revised: We compare SIR with baselines without task reduction.

Original: Experiments are presented in 3 different environments simulated in MuJoCo (Todorov et al., 2012) engine: a robotic-hand pushing scenario with a wall opening (denoted by “ Push ”), a robotic-gripper stacking scenario (denoted by “ Stack ”), and a 2D particle-based maze scenario with a much larger environment space (denoted by “ Maze ”).
Revised: Experiments are presented in 3 different environments simulated in MuJoCo (Todorov et al., 2012) engine: a robotic-hand pushing scenario with a wall opening (denoted by “ Push ”), a robotic-gripper stacking scenario (denoted by “ Stack ”), and a 2D particle-based maze scenario with a much larger environment space (denoted by “ Maze ”).

Original: All of the scenarios allow random initial and goal states so as to form a continuously parameterized task space.
Revised: All of the scenarios allow random initial and goal states so as to form a continuously parameterized task space.

Original: All of the problems are fully-observable with 0/1 sparse rewards and are able to restart from a given state when performing task reduction.
Revised: All of the problems are fully-observable with 0/1 sparse rewards and are able to restart from a given state when performing task reduction.

Original: All the training results are repeated overrandom seeds.
Revised: All the training results are repeated overrandom seeds.

Original: We empirically notice that SAC has superior performance in robotics tasks but takes signiﬁcantly longer wall-clock time than PPO in the maze navigation tasks (see Fig. 17 in Appx.
Revised: We empirically notice that SAC has superior performance in robotics tasks but takes signiﬁcantly longer wall-clock time than PPO in the maze navigation tasks (see Fig. 17 in Appx.

Original: C).
Revised: C).

Original: Hence, we use SAC in Push and Stack while PPO in Maze for RL training in all the algorithms if not otherwise stated.
Revised: Hence, we use SAC in Push and Stack while PPO in Maze for RL training in all the algorithms if not otherwise stated.

Original: We remark that we also additionally report SAC results for a simpliﬁed Maze task in Appx.
Revised: We remark that we also additionally report SAC results for a simpliﬁed Maze task in Appx.

Original: C (see Fig. 18).
Revised: C (see Fig. 18).

Original: We primarily consider state-based observations and additionally explore the extension to image input in a Maze scenario.
Revised: We primarily consider state-based observations and additionally explore the extension to image input in a Maze scenario.

Original: More details and results can be found in the appendices.
Revised: More details and results can be found in the appendices.

Original: 5. Push S CENARIO
Revised: 5. Push S CENARIO

Original: [Figure]
Revised: [Figure]

Original: Hard
Revised: Hard

Original: Rand.
Revised: Rand.

Original: SAC only 0.
Revised: SAC only 0.

Original: SAC + TR 0.
Revised: SAC + TR 0.

Original: We evaluate the success rates on test tasks when additional TR steps are allowed in the execution of the SAC-trained policy.
Revised: We evaluate the success rates on test tasks when additional TR steps are allowed in the execution of the SAC-trained policy.

Original: [Figure]
Revised: [Figure]

Original: r a t
Revised: r a t

Original: [Figure]
Revised: [Figure]

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1 (Sec. 1), there are two objects on the table, a cubic box (blue) and an elongated box (brown).
Revised: 1 (Sec. 1), there are two objects on the table, a cubic box (blue) and an elongated box (brown).

Original: The robot needs to push a speciﬁed object to a goal location (red).
Revised: The robot needs to push a speciﬁed object to a goal location (red).

Original: The table is separated into two connected regions by a ﬁxed wall with an open door in the middle.
Revised: The table is separated into two connected regions by a ﬁxed wall with an open door in the middle.

Original: The initial object positions, goal location, robot hand location, and which object to push are randomly reset per episode.
Revised: The initial object positions, goal location, robot hand location, and which object to push are randomly reset per episode.

Original: Learned Value Function:
Revised: Learned Value Function:

Original: We retrieve a partially trained value function during learning and illustrate how task reduction is performed over it in Fig.
Revised: We retrieve a partially trained value function during learning and illustrate how task reduction is performed over it in Fig.

Original: Consider a hard task shown in Fig.
Revised: Consider a hard task shown in Fig.

Original: 2a where the door is blocked by the elongated box and the agent needs to move the cubic box to the goal on the other side.
Revised: 2a where the door is blocked by the elongated box and the agent needs to move the cubic box to the goal on the other side.

Original: A possible search space is the easier tasks with the position of the elongated box perturbed.
Revised: A possible search space is the easier tasks with the position of the elongated box perturbed.

Original: We demonstrate the learned values for moving the elongated box to different goal positions in Fig.
Revised: We demonstrate the learned values for moving the elongated box to different goal positions in Fig.

Original: 2b, where we observe consistently lower values for faraway goals.
Revised: 2b, where we observe consistently lower values for faraway goals.

Original: Similarly, Fig.
Revised: Similarly, Fig.

Original: 2c visualizes the values for all possible reduced tasks, i.e., moving the cubic box to the goal with the elongated box placed in different places.
Revised: 2c visualizes the values for all possible reduced tasks, i.e., moving the cubic box to the goal with the elongated box placed in different places.

Original: We can observe higher values when the elongated box is away from the door.
Revised: We can observe higher values when the elongated box is away from the door.

Original: Fig.
Revised: Fig.

Original: 2d shows the composite values w.r.t.
Revised: 2d shows the composite values w.r.t.

Original: different target states, where the reduction search is actually performed.
Revised: different target states, where the reduction search is actually performed.

Original: We can clearly see two modes on both sides of the door.
Revised: We can clearly see two modes on both sides of the door.

Original: Task Reduction as a Planner: Task reduction can serve as a test-time planner by searching for a reduction target that possibly yields a higher composite value than direct policy execution.
Revised: Task Reduction as a Planner: Task reduction can serve as a test-time planner by searching for a reduction target that possibly yields a higher composite value than direct policy execution.

Original: We use a half-trained policy and value function by SAC and test in a ﬁxed set of 1000 random tasks.
Revised: We use a half-trained policy and value function by SAC and test in a ﬁxed set of 1000 random tasks.

Original: Since random sampling rarely produces hard cases, we also evaluate in 1000 hard tasks where the elongated box blocks the door and separates the cubic box and goal.
Revised: Since random sampling rarely produces hard cases, we also evaluate in 1000 hard tasks where the elongated box blocks the door and separates the cubic box and goal.

Original: The results are in Table.
Revised: The results are in Table.

Original: 1, where task reduction effectively boosts performances in hard cases, which require effective planning.
Revised: 1, where task reduction effectively boosts performances in hard cases, which require effective planning.

Original: Performances: SIR is compared with three different baselines: SAC; SAC with self-imitation learning (SIL), which does not run task reduction and only imitates past successful trajectories directly generated by policy execution; SAC with a dense reward (DS), i.e., a potential-based distance reward from object to the goal.
Revised: Performances: SIR is compared with three different baselines: SAC; SAC with self-imitation learning (SIL), which does not run task reduction and only imitates past successful trajectories directly generated by policy execution; SAC with a dense reward (DS), i.e., a potential-based distance reward from object to the goal.

Original: We evaluate the average success rate over the entire task distribution
Revised: We evaluate the average success rate over the entire task distribution

Original: 8 samples 1e 0.
Revised: 8 samples 1e 0.

Original: a v g .
Revised: a v g .

Original: s u cc .
Revised: s u cc .

Original: r a t e   8 samples 1e 0.
Revised: r a t e   8 samples 1e 0.

Original: h a r d s u cc .
Revised: h a r d s u cc .

Original: r a t e
Revised: r a t e

Original: SAC SIR SIL
Revised: SAC SIR SIL

Original: DS
Revised: DS

Original: [Table]
Revised: [Table]

Original: as well as the hard-case success rate (i.e., with a blocked door) in Fig.
Revised: as well as the hard-case success rate (i.e., with a blocked door) in Fig.

Original: 3, where SIR achieves the best overall performances, particularly in hard cases.
Revised: 3, where SIR achieves the best overall performances, particularly in hard cases.

Original: We emphasize that for SIR, we include all the discarded samples due to failed task reductions in the plots.
Revised: We emphasize that for SIR, we include all the discarded samples due to failed task reductions in the plots.

Original: Since the only difference between
Revised: Since the only difference between

Original: SIR and SIL is the use of task reduction, our improvement over SIL suggests that task reduction is effective for those compositional challenges while burdened with minimal overall overhead.
Revised: SIR and SIL is the use of task reduction, our improvement over SIL suggests that task reduction is effective for those compositional challenges while burdened with minimal overall overhead.

Original: 5. Stack S CENARIO
Revised: 5. Stack S CENARIO

Original: Next, we consider a substantially more challenging task in a similar environment to Push : there is a robotics gripper and at most 3 cubic boxes with different colors on the table.
Revised: Next, we consider a substantially more challenging task in a similar environment to Push : there is a robotics gripper and at most 3 cubic boxes with different colors on the table.

Original: The goal is to stack a tower using these boxes such that a speciﬁed cubic box remains stable at a desired goal position even with gripper ﬁngers left apart.
Revised: The goal is to stack a tower using these boxes such that a speciﬁed cubic box remains stable at a desired goal position even with gripper ﬁngers left apart.

Original: Therefore, if the goal is in the air, the agent must learn to ﬁrst stack non-target boxes in the bottom, to which there are even multiple valid solutions, before it can ﬁnally put the target box on the top.
Revised: Therefore, if the goal is in the air, the agent must learn to ﬁrst stack non-target boxes in the bottom, to which there are even multiple valid solutions, before it can ﬁnally put the target box on the top.

Original: See Fig.
Revised: See Fig.

Original: 4 for an example of stacking the yellow box towards the goal position (i.e, the yellow spot in the air).
Revised: 4 for an example of stacking the yellow box towards the goal position (i.e, the yellow spot in the air).

Original: Each episode starts with a random number of boxes with random initial locations and a colored goal that is viable to reach via stacking.
Revised: Each episode starts with a random number of boxes with random initial locations and a colored goal that is viable to reach via stacking.

Original: This task is extremely challenging, so we (1) introduce an auxiliary training task, pick-and-place , whose goal is simply relocating a speciﬁed box to the goal position, and (2) carefully design a training curriculum for all methods by gradually decreasing the ratio of pick-and-place tasks from 70% to 30% throughout the training.
Revised: This task is extremely challenging, so we (1) introduce an auxiliary training task, pick-and-place , whose goal is simply relocating a speciﬁed box to the goal position, and (2) carefully design a training curriculum for all methods by gradually decreasing the ratio of pick-and-place tasks from 70% to 30% throughout the training.

Original: For evaluation, a task is considered as a hard case if all the 3 boxes are required to be stacked together.
Revised: For evaluation, a task is considered as a hard case if all the 3 boxes are required to be stacked together.

Original: We show the training curves as well as the hard-case success rate for SIR and other methods in Fig.
Revised: We show the training curves as well as the hard-case success rate for SIR and other methods in Fig.

Original: SIR signiﬁcantly outperforms all the baselines, especially in the hard cases, i.e., stacking all theboxes.
Revised: SIR signiﬁcantly outperforms all the baselines, especially in the hard cases, i.e., stacking all theboxes.

Original: Table.
Revised: Table.

Original: 2 illustrates the detailed success rate of different methods for stacking 1, 2 and 3 boxes.
Revised: 2 illustrates the detailed success rate of different methods for stacking 1, 2 and 3 boxes.

Original: SIR reliably solves all the cases while other methods without task reduction struggle to discover the desired compositional strategy to solve the hardest cases.
Revised: SIR reliably solves all the cases while other methods without task reduction struggle to discover the desired compositional strategy to solve the hardest cases.

Original: Fig.
Revised: Fig.

Original: 4 visualizes a successful trajectory by
Revised: 4 visualizes a successful trajectory by

Original: SIR for stacking 3 boxes from scratch, where the goal is to stack the yellow box to reach the goal spot in the air.
Revised: SIR for stacking 3 boxes from scratch, where the goal is to stack the yellow box to reach the goal spot in the air.

Original: Interestingly, the agent even reliably discovers a non-trivial strategy: it ﬁrst puts the yellow box on the top of the green box to form a tower of two stories, and then picks these two boxes jointly, and ﬁnally places this tower on top of the blue box to ﬁnish the task.
Revised: Interestingly, the agent even reliably discovers a non-trivial strategy: it ﬁrst puts the yellow box on the top of the green box to form a tower of two stories, and then picks these two boxes jointly, and ﬁnally places this tower on top of the blue box to ﬁnish the task.

Original: 5. Maze S CENARIO
Revised: 5. Maze S CENARIO

Original: In this scenario, we focus on a navigation problem with a particle-based agent in a large 2D maze, named “ 4-Room maze ”.
Revised: In this scenario, we focus on a navigation problem with a particle-based agent in a large 2D maze, named “ 4-Room maze ”.

Original: The maze is separated into 4 connected rooms by three ﬁxed walls and each wall has an open door in the middle.
Revised: The maze is separated into 4 connected rooms by three ﬁxed walls and each wall has an open door in the middle.

Original: There are 4 objects in the maze, including 3 elongated boxes
Revised: There are 4 objects in the maze, including 3 elongated boxes

Original: [Figure]
Revised: [Figure]

Original: a v g .
Revised: a v g .

Original: s u cc .
Revised: s u cc .

Original: r a t e
Revised: r a t e

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Figure 6 visualizes a particularly challenging situation: the agent needs to move the cubic box to the goal but all three doors are precisely blocked by the elongated boxes.
Revised: Figure 6 visualizes a particularly challenging situation: the agent needs to move the cubic box to the goal but all three doors are precisely blocked by the elongated boxes.

Original: To solve this task, the agent must perturb all the elongated boxes to clear the doors before moving the cubic box.
Revised: To solve this task, the agent must perturb all the elongated boxes to clear the doors before moving the cubic box.

Original: As is shown in
Revised: As is shown in

Original: Fig.
Revised: Fig.

Original: 6, the agent trained by SIR successfully learns this non-trivial strategy and ﬁnishes the task.
Revised: 6, the agent trained by SIR successfully learns this non-trivial strategy and ﬁnishes the task.

Original: We adopt PPO for training here and compare SIR to baselines in terms of average success rate over both the training task distribution and the hard tasks only.
Revised: We adopt PPO for training here and compare SIR to baselines in terms of average success rate over both the training task distribution and the hard tasks only.

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: SIR outperforms both PPO and SIL baselines with a clear margin, especially when evaluated on hard cases.
Revised: SIR outperforms both PPO and SIL baselines with a clear margin, especially when evaluated on hard cases.

Original: Note that although SIL learns slightly more efﬁciently at the beginning of training, it gets stuck later and does not make sufﬁcient progress towards the most challenging cases requiring complex compositional strategies to solve.
Revised: Note that although SIL learns slightly more efﬁciently at the beginning of training, it gets stuck later and does not make sufﬁcient progress towards the most challenging cases requiring complex compositional strategies to solve.

Original: Extension to visual domain: We consider extending SIR to the setting of image input by ﬁrst pretraining a β -VAE (Higgins et al., 2017a) to project the high-dimensional image to a low-dimensional latent vector and then perform task reduction over the latent space.
Revised: Extension to visual domain: We consider extending SIR to the setting of image input by ﬁrst pretraining a β -VAE (Higgins et al., 2017a) to project the high-dimensional image to a low-dimensional latent vector and then perform task reduction over the latent space.

Original: We train the β -VAE using random samples from the environment and apply the cross-entropy method (CEM) (De Boer et al., 2005) to search for the reduction targets.
Revised: We train the β -VAE using random samples from the environment and apply the cross-entropy method (CEM) (De Boer et al., 2005) to search for the reduction targets.

Original: We experiment in a sparse-reward variant of the U-Wall maze task adapted from Nasiriany et al.
Revised: We experiment in a sparse-reward variant of the U-Wall maze task adapted from Nasiriany et al.

Original: (2019), where the agent (blue) needs to navigate to the goal (green) by turning around the wall (boundary plotted in black).
Revised: (2019), where the agent (blue) needs to navigate to the goal (green) by turning around the wall (boundary plotted in black).

Original: Task visualization and training results are shown in Fig.
Revised: Task visualization and training results are shown in Fig.

Original: 8, where SIR still achieves the best performance compared with PPO and SIL.
Revised: 8, where SIR still achieves the best performance compared with PPO and SIL.

Original: Comparison with HRL: We also compare SIR with two baselines with hierarchical policy structures,
Revised: Comparison with HRL: We also compare SIR with two baselines with hierarchical policy structures,

Original: i.e., an off-policy HRL algorithm HIRO (Nachum et al., 2018) and an option discovery algorithm
Revised: i.e., an off-policy HRL algorithm HIRO (Nachum et al., 2018) and an option discovery algorithm

Original: DSC (Bagaria & Konidaris, 2020).
Revised: DSC (Bagaria & Konidaris, 2020).

Original: Since these two HRL baselines are both based on off-policy learning, we use SAC as our training algorithm in SIR for a fair comparison.
Revised: Since these two HRL baselines are both based on off-policy learning, we use SAC as our training algorithm in SIR for a fair comparison.

Original: In addition, since both
Revised: In addition, since both

Original: [Figure]
Revised: [Figure]

Original: two baselines also assume a ﬁxed goal position, we experiment in two ﬁxed-goal Maze scenarios.
Revised: two baselines also assume a ﬁxed goal position, we experiment in two ﬁxed-goal Maze scenarios.

Original: We ﬁrst conduct a sanity check in a simple U-Shape maze without any boxes (Fig. 9a), where the agent (yellow) is expected to propel itself towards the goal (red).
Revised: We ﬁrst conduct a sanity check in a simple U-Shape maze without any boxes (Fig. 9a), where the agent (yellow) is expected to propel itself towards the goal (red).

Original: Then we experiment in a 3-Room maze (Fig. 9b), which is a simpliﬁed version of the previously described 4-Room maze .
Revised: Then we experiment in a 3-Room maze (Fig. 9b), which is a simpliﬁed version of the previously described 4-Room maze .

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: Although HIRO and DSC learn to solve the easy U-shape maze , they hardly make any progress in the more difﬁcult 3-Room maze , which requires a non-trivial compositional strategy.
Revised: Although HIRO and DSC learn to solve the easy U-shape maze , they hardly make any progress in the more difﬁcult 3-Room maze , which requires a non-trivial compositional strategy.

Original: Note that DSC is extremely slow in 3-Room maze , so we terminate it after 5 days of training.
Revised: Note that DSC is extremely slow in 3-Room maze , so we terminate it after 5 days of training.

Original: In contrast, even without any policy hierarchies, SIR learns signiﬁcantly faster in both two scenarios.
Revised: In contrast, even without any policy hierarchies, SIR learns signiﬁcantly faster in both two scenarios.

Original: [Figure]
Revised: [Figure]

Original: Comparison with automatic curriculum learning:
Revised: Comparison with automatic curriculum learning:

Original: As discussed at the end of Sec. 2, SIR implicitly constructs a task curriculum by ﬁrst solving easy tasks and then reducing a hard task to a solved one.
Revised: As discussed at the end of Sec. 2, SIR implicitly constructs a task curriculum by ﬁrst solving easy tasks and then reducing a hard task to a solved one.

Original: Hence, we also conduct experiments with curriculum learning (CL) methods.
Revised: Hence, we also conduct experiments with curriculum learning (CL) methods.

Original: We compare SIR using a uniform task sampler with standard PPO learning with GoalGAN (Florensa et al., 2018), an automatic CL approach, as well as a manually designed training curriculum, which gradually increases the distance between the goal and the start position.
Revised: We compare SIR using a uniform task sampler with standard PPO learning with GoalGAN (Florensa et al., 2018), an automatic CL approach, as well as a manually designed training curriculum, which gradually increases the distance between the goal and the start position.

Original: Since GoalGAN assumes a ﬁxed initial position for the agent while only speciﬁes the goal position, we consider the unmodiﬁed navigation problem used in Florensa et al.
Revised: Since GoalGAN assumes a ﬁxed initial position for the agent while only speciﬁes the goal position, we consider the unmodiﬁed navigation problem used in Florensa et al.

Original: (2018) for a fair comparison.
Revised: (2018) for a fair comparison.

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: Even with a uniform task sampler,
Revised: Even with a uniform task sampler,

Original: SIR signiﬁcantly outperforms the CL baselines that do not leverage the task reduction technique.
Revised: SIR signiﬁcantly outperforms the CL baselines that do not leverage the task reduction technique.

Original: We want to remark that GoalGAN eventually solves the task with 10 6 samples since it takes a large number of samples to train an effective GAN for task sampling.
Revised: We want to remark that GoalGAN eventually solves the task with 10 6 samples since it takes a large number of samples to train an effective GAN for task sampling.

Original: By contrast, SIR directly works with a uniform task sampler and implicitly learns from easy to hard in a much lightweight fashion.
Revised: By contrast, SIR directly works with a uniform task sampler and implicitly learns from easy to hard in a much lightweight fashion.

Original: Finally, we emphasize that SIR is also complementary to curriculum learning methods.
Revised: Finally, we emphasize that SIR is also complementary to curriculum learning methods.

Original: As we previously show in Sec. 5.2, combining SIR and a manually-designed training curriculum solves a challenging stacking problem, which is unsolved by solely running curriculum learning.
Revised: As we previously show in Sec. 5.2, combining SIR and a manually-designed training curriculum solves a challenging stacking problem, which is unsolved by solely running curriculum learning.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper presents a novel learning paradigm SIR for solving complex RL problems.
Revised: This paper presents a novel learning paradigm SIR for solving complex RL problems.

Original: SIR utilizes task reduction to leverage compositional structure in the task space and performs imitation learning to effectively tackle successively more challenging tasks.
Revised: SIR utilizes task reduction to leverage compositional structure in the task space and performs imitation learning to effectively tackle successively more challenging tasks.

Original: We show that an RL agent trained by SIR can efﬁciently learn sophisticated strategies in challenging sparse-reward continuous-control problems.
Revised: We show that an RL agent trained by SIR can efﬁciently learn sophisticated strategies in challenging sparse-reward continuous-control problems.

Original: Task reduction suggests a simple but effective way, i.e., supervised learning on composite trajectories with a generic policy representation, to incorporate inductive bias into policy learning, which, we hope, could bring valuable insights to the community towards a wider range of challenges.
Revised: Task reduction suggests a simple but effective way, i.e., supervised learning on composite trajectories with a generic policy representation, to incorporate inductive bias into policy learning, which, we hope, could bring valuable insights to the community towards a wider range of challenges.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: Co-author Wang was supported, in part, by gifts from Qualcomm and TuSimple.
Revised: Co-author Wang was supported, in part, by gifts from Qualcomm and TuSimple.

Original: A E NVIRONMENT D ETAILS A. P USH S CENARIO
Revised: A E NVIRONMENT D ETAILS A. P USH S CENARIO

Original: The robot is placed beside a 50cm × 70cm table with an elongated box and a cubic box on it.
Revised: The robot is placed beside a 50cm × 70cm table with an elongated box and a cubic box on it.

Original: The initial position of the two boxes, the projection of the initial gripper position onto the table plane, and the goal position are sampled from a 30cm × 30cm square on the table.
Revised: The initial position of the two boxes, the projection of the initial gripper position onto the table plane, and the goal position are sampled from a 30cm × 30cm square on the table.

Original: The gripper is initialized at a ﬁxed height above the table.
Revised: The gripper is initialized at a ﬁxed height above the table.

Original: Observation space The observation vector is a concatenation of state , achieved goal and desired goal .
Revised: Observation space The observation vector is a concatenation of state , achieved goal and desired goal .

Original: State consists of the position and velocity of the gripper, the cubic box and the elongated box, as well as the state and velocity of the gripper ﬁngers.
Revised: State consists of the position and velocity of the gripper, the cubic box and the elongated box, as well as the state and velocity of the gripper ﬁngers.

Original: Desired goal vector consists of a position coordinate and a one-hot vector indicating the identity of the target object.
Revised: Desired goal vector consists of a position coordinate and a one-hot vector indicating the identity of the target object.

Original: Achieved goal contains the current position of the target object and its corresponding one-hot index vector.
Revised: Achieved goal contains the current position of the target object and its corresponding one-hot index vector.

Original: Action space
Revised: Action space

Original: The action is 4-dimensional, with the ﬁrst 3 elements indicating the desired position shift of the end-effector and the last element controlling the gripper ﬁngers (locked in this scenario).
Revised: The action is 4-dimensional, with the ﬁrst 3 elements indicating the desired position shift of the end-effector and the last element controlling the gripper ﬁngers (locked in this scenario).

Original: Hard case conﬁguration Our training distribution is a mixture of easy and hard cases.
Revised: Hard case conﬁguration Our training distribution is a mixture of easy and hard cases.

Original: In easy cases, positions of the gripper, the boxes and the goal are all uniformly sampled.
Revised: In easy cases, positions of the gripper, the boxes and the goal are all uniformly sampled.

Original: The index of the target object is also uniformly sampled from total number of objects.
Revised: The index of the target object is also uniformly sampled from total number of objects.

Original: In hard cases, the position of the gripper, the position of the cubic box, and the target object index are still uniformly sampled, but the elongated box is initialized at a ﬁxed location that blocks the door open of the wall.
Revised: In hard cases, the position of the gripper, the position of the cubic box, and the target object index are still uniformly sampled, but the elongated box is initialized at a ﬁxed location that blocks the door open of the wall.

Original: The goal is on the opposite side of the wall if the target object is the cubic box.
Revised: The goal is on the opposite side of the wall if the target object is the cubic box.

Original: A. S TACK S CENARIO
Revised: A. S TACK S CENARIO

Original: The physical setup of this scenario is much the same as that in Push scenario, except that there is no wall on the table.
Revised: The physical setup of this scenario is much the same as that in Push scenario, except that there is no wall on the table.

Original: At the beginning of each episode, the number of boxes on the table, the goal position and the target object index are sampled from a designed task curriculum.
Revised: At the beginning of each episode, the number of boxes on the table, the goal position and the target object index are sampled from a designed task curriculum.

Original: As explained in the main paper, there are two types of tasks, stacking and pick-and-place in the curriculum, and the type of task is also reset in each episode.
Revised: As explained in the main paper, there are two types of tasks, stacking and pick-and-place in the curriculum, and the type of task is also reset in each episode.

Original: The ( x, y ) coordinate of the goal is uniformly sampled from a 30cm × 30cm square on the table, while goal height g z is initialized depending on the type of task of the current episode: for pick-and-place tasks, g z ∼ table height + U (0 , 0 . 45) and for stacking tasks, g z = table height + N · box height, where N ∈ [1 , N max ] denotes the number of boxes.
Revised: The ( x, y ) coordinate of the goal is uniformly sampled from a 30cm × 30cm square on the table, while goal height g z is initialized depending on the type of task of the current episode: for pick-and-place tasks, g z ∼ table height + U (0 , 0 . 45) and for stacking tasks, g z = table height + N · box height, where N ∈ [1 , N max ] denotes the number of boxes.

Original: Observation space The observation vector shares the same structure with Push scenario, but includes another one-hot vector indicating the type of the task.
Revised: Observation space The observation vector shares the same structure with Push scenario, but includes another one-hot vector indicating the type of the task.

Original: Action space
Revised: Action space

Original: The action vector is the same as that in Push scenario.
Revised: The action vector is the same as that in Push scenario.

Original: Hard case conﬁguration The hard cases we are most interested in are stacking N max boxes so that the target box reaches the goal position with the gripper ﬁngers left apart.
Revised: Hard case conﬁguration The hard cases we are most interested in are stacking N max boxes so that the target box reaches the goal position with the gripper ﬁngers left apart.

Original: A. M AZE S CENARIO
Revised: A. M AZE S CENARIO

Original: This scenario deals with a 2D maze.
Revised: This scenario deals with a 2D maze.

Original: The agent (green) can only apply force to itself in the directions parallel to the ﬂoor plane.
Revised: The agent (green) can only apply force to itself in the directions parallel to the ﬂoor plane.

Original: The position of the agent and the goal are both uniformly sampled from the collision-free space in the maze.
Revised: The position of the agent and the goal are both uniformly sampled from the collision-free space in the maze.

Original: Observation space
Revised: Observation space

Original: The structure of observation space is generally the same as the Push scenario except that the state and velocity of gripper ﬁngers are not included in the state vector since there is no gripper in Maze scenarios.
Revised: The structure of observation space is generally the same as the Push scenario except that the state and velocity of gripper ﬁngers are not included in the state vector since there is no gripper in Maze scenarios.

Original: In U-Shape Maze , the one-hot target object indicator is omitted since there is no object except the agent itself.
Revised: In U-Shape Maze , the one-hot target object indicator is omitted since there is no object except the agent itself.

Original: For experiments in the visual domain, RGB images of shape 48 × 48 are directly used as observations.
Revised: For experiments in the visual domain, RGB images of shape 48 × 48 are directly used as observations.

Original: [Figure]
Revised: [Figure]

Original: Action space
Revised: Action space

Original: The action is a 2-dimensional vector representing the force applied to the agent.
Revised: The action is a 2-dimensional vector representing the force applied to the agent.

Original: Hard case conﬁguration There is no manually designed hard case for U-Shape Maze and U-Wall
Revised: Hard case conﬁguration There is no manually designed hard case for U-Shape Maze and U-Wall

Original: Maze .
Revised: Maze .

Original: In hard cases of 4-Room maze , a random number of elongated boxes block the doors on.
Revised: In hard cases of 4-Room maze , a random number of elongated boxes block the doors on.

Original: The goal is forced to initialize in a different room from that the cubic box is in.
Revised: The goal is forced to initialize in a different room from that the cubic box is in.

Original: For hard cases of 3-Room
Revised: For hard cases of 3-Room

Original: Maze , each elongated box blocks a door.
Revised: Maze , each elongated box blocks a door.

Original: More precisely, the rooms that elongated boxes are in are determined by the agent’s position: if the agent is in the leftmost room, the elongated boxes are placed to the left of each door; if the agent is in the middle room, the elongated boxes are also placed in the middle room and each blocks a door; if the agent is in the rightmost room, the elongated boxes are placed to the right of each door.
Revised: More precisely, the rooms that elongated boxes are in are determined by the agent’s position: if the agent is in the leftmost room, the elongated boxes are placed to the left of each door; if the agent is in the middle room, the elongated boxes are also placed in the middle room and each blocks a door; if the agent is in the rightmost room, the elongated boxes are placed to the right of each door.

Original: Fixed goal location In the ﬁxed goal version of 3-Room maze , the goal is located at the same spot in the rightmost room for each episode.
Revised: Fixed goal location In the ﬁxed goal version of 3-Room maze , the goal is located at the same spot in the rightmost room for each episode.

Original: In U-Shape Maze , the goal is always at the end of the maze.
Revised: In U-Shape Maze , the goal is always at the end of the maze.

Original: B T RAINING D ETAILS
Revised: B T RAINING D ETAILS

Original: Composition operator As is mentioned in Sec. 4.1, there are multiple choices for the composition operator given two values (Eq.
Revised: Composition operator As is mentioned in Sec. 4.1, there are multiple choices for the composition operator given two values (Eq.

Original: In addition to the one we used in SIR, i.e., product operator, we also visualize the composite values derived by other operators, including minimum (i.e., “min”, taking the smaller value of the two) and average (i.e., take the mean value) in Fig.
Revised: In addition to the one we used in SIR, i.e., product operator, we also visualize the composite values derived by other operators, including minimum (i.e., “min”, taking the smaller value of the two) and average (i.e., take the mean value) in Fig.

Original: Besides, note that due to value approximation error, even if the reward is 0/1, the output of a learned value function may not perfectly lie between 0 and 1.
Revised: Besides, note that due to value approximation error, even if the reward is 0/1, the output of a learned value function may not perfectly lie between 0 and 1.

Original: So, in order to apply the “product” operator well in practice, we normalize the value functions before they are actually composed.
Revised: So, in order to apply the “product” operator well in practice, we normalize the value functions before they are actually composed.

Original: Speciﬁcally, the composite values are computed as follow:
Revised: Speciﬁcally, the composite values are computed as follow:

Original: [Equation]
Revised: [Equation]

Original: Empirically, the “product” operator is the best among all candidates, since it most clearly demonstrates two high-value modes on both sides of the door and low composite values at the door.
Revised: Empirically, the “product” operator is the best among all candidates, since it most clearly demonstrates two high-value modes on both sides of the door and low composite values at the door.

Original: Estimating σ
Revised: Estimating σ

Original: When combining SIR with SAC baseline, the output of value function converges to a stable range after the initial warm-up phase (see Fig. 12a).
Revised: When combining SIR with SAC baseline, the output of value function converges to a stable range after the initial warm-up phase (see Fig. 12a).

Original: Therefore it sufﬁces to use a static high threshold (set to 0.9 in Push scenario with mixed task sampler and 1 in all other cases) to
Revised: Therefore it sufﬁces to use a static high threshold (set to 0.9 in Push scenario with mixed task sampler and 1 in all other cases) to

Original: [Figure]
Revised: [Figure]

Original: discard reduction targets in the initial value overestimation phase and a static low threshold σ to ﬁlter out reduction targets with low values that are not likely to succeed.
Revised: discard reduction targets in the initial value overestimation phase and a static low threshold σ to ﬁlter out reduction targets with low values that are not likely to succeed.

Original: When training SIR with
Revised: When training SIR with

Original: PPO baseline, We empirically observe that the scale of value function keeps increasing as training proceeds, therefore a static ﬁltering threshold is not appropriate here.
Revised: PPO baseline, We empirically observe that the scale of value function keeps increasing as training proceeds, therefore a static ﬁltering threshold is not appropriate here.

Original: Hence we maintain an online estimation of the ﬁltering threshold by keeping a running average of composite values of all the selected reduction targets.
Revised: Hence we maintain an online estimation of the ﬁltering threshold by keeping a running average of composite values of all the selected reduction targets.

Original: As is shown in Fig.
Revised: As is shown in Fig.

Original: 12b, the successful reduction targets typically have higher values than average.
Revised: 12b, the successful reduction targets typically have higher values than average.

Original: Designed task curriculum in Stack scenario
Revised: Designed task curriculum in Stack scenario

Original: The initial positions of all the boxes are generated as follows: select M ( 0 ≤ M ≤ N − 1 ) boxes from the total N boxes, and stack them under the goal, then initialize other boxes randomly over a square on the table.
Revised: The initial positions of all the boxes are generated as follows: select M ( 0 ≤ M ≤ N − 1 ) boxes from the total N boxes, and stack them under the goal, then initialize other boxes randomly over a square on the table.

Original: The ratio of stacking and pick-and-place tasks is adapted according to the agent’s learning progress.
Revised: The ratio of stacking and pick-and-place tasks is adapted according to the agent’s learning progress.

Original: We keep 70% chance of sampling pick-and-place tasks until the agent achieves a success rate of 0.8 when evaluating on pick-and-place tasks, then gradually decrease the ratio to 30%.
Revised: We keep 70% chance of sampling pick-and-place tasks until the agent achieves a success rate of 0.8 when evaluating on pick-and-place tasks, then gradually decrease the ratio to 30%.

Original: Note that such task curriculum is applied for all the experimented algorithms in Stack scenario.
Revised: Note that such task curriculum is applied for all the experimented algorithms in Stack scenario.

Original: Baseline implementation We utilize prioritized experience replay when working with off-policy
Revised: Baseline implementation We utilize prioritized experience replay when working with off-policy

Original: RL algorithms.
Revised: RL algorithms.

Original: Two separate replay buffers D rl , D il are prepared for rollouts generated by direct policy execution and composite task reduction.
Revised: Two separate replay buffers D rl , D il are prepared for rollouts generated by direct policy execution and composite task reduction.

Original: We compute priority as p = | r + γV − Q | for each instance when storing transitions into the buffers.
Revised: We compute priority as p = | r + γV − Q | for each instance when storing transitions into the buffers.

Original: When sampling from D rl ∪ D il , each element can be sampled with probability proportional to p α .
Revised: When sampling from D rl ∪ D il , each element can be sampled with probability proportional to p α .

Original: The replay buffers maintain a ﬁxed ﬁnite number of samples and data are ﬂushed following the ﬁrst-in-ﬁrst-out manner.
Revised: The replay buffers maintain a ﬁxed ﬁnite number of samples and data are ﬂushed following the ﬁrst-in-ﬁrst-out manner.

Original: As for on-policy baselines, data in D rl are discarded after every iteration.
Revised: As for on-policy baselines, data in D rl are discarded after every iteration.

Original: D il keeps track of a recent history of task reduction demonstration.
Revised: D il keeps track of a recent history of task reduction demonstration.

Original: As the training proceeds, we get better policies and new demonstrations with higher quality.
Revised: As the training proceeds, we get better policies and new demonstrations with higher quality.

Original: Some stale demonstrations may not be beneﬁcial anymore (e.g., early accidental successes by random exploration).
Revised: Some stale demonstrations may not be beneﬁcial anymore (e.g., early accidental successes by random exploration).

Original: However, the advantage clipping scheme in
Revised: However, the advantage clipping scheme in

Original: Eq.
Revised: Eq.

Original: (2) may not effectively eliminate undesired stale data: in the sparse reward setting, since all the demonstrations are successful trajectories, the advantage will be hardly negative 1 .
Revised: (2) may not effectively eliminate undesired stale data: in the sparse reward setting, since all the demonstrations are successful trajectories, the advantage will be hardly negative 1 .

Original: Thus, we maintain the dataset D il as a buffer which keeps every trajectory for a ﬁxed number of iterations after it was generated.
Revised: Thus, we maintain the dataset D il as a buffer which keeps every trajectory for a ﬁxed number of iterations after it was generated.

Original: Network architecture In Push scenario, we simply use MLPs with 256 hidden units per layer to map the input observations to actions or values.
Revised: Network architecture In Push scenario, we simply use MLPs with 256 hidden units per layer to map the input observations to actions or values.

Original: In Stack and Maze scenario, since there are more objects in observations, we adopt an attention-based feature extractor to get representation invariant to the number of objects.
Revised: In Stack and Maze scenario, since there are more objects in observations, we adopt an attention-based feature extractor to get representation invariant to the number of objects.

Original: The original observation vector can be rewritten as [ o self , o 1 , o 2 , · · · , o N ] , where o self contains the state of the agent (robot or particle) and the position of the desired goal, o i (1 ≤ i ≤ N ) represents the state of each object.
Revised: The original observation vector can be rewritten as [ o self , o 1 , o 2 , · · · , o N ] , where o self contains the state of the agent (robot or particle) and the position of the desired goal, o i (1 ≤ i ≤ N ) represents the state of each object.

Original: We then apply MLP to get observation embeddings f ( o self ) , g ( o i , 1 ≤ i ≤ N ) .
Revised: We then apply MLP to get observation embeddings f ( o self ) , g ( o i , 1 ≤ i ≤ N ) .

Original: The attention embedding e of objects is then computed by attending f ( o self ) over g
Revised: The attention embedding e of objects is then computed by attending f ( o self ) over g

Original: ( o i ) :
Revised: ( o i ) :

Original: [Equation]
Revised: [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Finally e is concatenated with f ( o self ) and fed into another MLP to produce actions or values.
Revised: Finally e is concatenated with f ( o self ) and fed into another MLP to produce actions or values.

Original: Hyperparameters We summarize hyperparameters of off-policy and on-policy algorithms in each scenario in Table 3 and Table 4.
Revised: Hyperparameters We summarize hyperparameters of off-policy and on-policy algorithms in each scenario in Table 3 and Table 4.

Original: C M ORE R ESULTS
Revised: C M ORE R ESULTS

Original: Success rate statistics The success rates over the training task distribution and on hard cases evaluated at the end of training for all the experimented algorithms are listed in Table 5 and Table 6.
Revised: Success rate statistics The success rates over the training task distribution and on hard cases evaluated at the end of training for all the experimented algorithms are listed in Table 5 and Table 6.

Original: All the numbers are averaged over 3 random seeds.
Revised: All the numbers are averaged over 3 random seeds.

Original: The proposed SIR generally achieves the best result across the board.
Revised: The proposed SIR generally achieves the best result across the board.

Original: Particularly in Stack scenario with 3 boxes, we provide more detailed hard-case success rates from all kinds of initial conﬁgurations for stacking 3 boxes (explained in Sec.
Revised: Particularly in Stack scenario with 3 boxes, we provide more detailed hard-case success rates from all kinds of initial conﬁgurations for stacking 3 boxes (explained in Sec.

Original: B ) in Table.
Revised: B ) in Table.

Original: Although all the algorithms succeed from the initial states with 2 boxes already stacked by simply stacking the target box on top of the existing tower, only SIR can solve more compositional situations withor no box stacked in the beginning, which require strategic manipulation of other non-target boxes before ﬁnishing the tasks.
Revised: Although all the algorithms succeed from the initial states with 2 boxes already stacked by simply stacking the target box on top of the existing tower, only SIR can solve more compositional situations withor no box stacked in the beginning, which require strategic manipulation of other non-target boxes before ﬁnishing the tasks.

Original: Additional results in Push scenario
Revised: Additional results in Push scenario

Original: In addition to the results presented in the main paper, which is trained on a mixture of 30% hard cases and 70% uniformly sampled cases, we also show the training curves with a uniform task sampler in Fig. 13.
Revised: In addition to the results presented in the main paper, which is trained on a mixture of 30% hard cases and 70% uniformly sampled cases, we also show the training curves with a uniform task sampler in Fig. 13.

Original: Furthermore, we repeat these experiments
Revised: Furthermore, we repeat these experiments

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: with PPO-based algorithms as shown in Fig. 14.
Revised: with PPO-based algorithms as shown in Fig. 14.

Original: SIR achieves generally the best result compared with other baselines in all the settings, especially in hard cases which require compositional policies.
Revised: SIR achieves generally the best result compared with other baselines in all the settings, especially in hard cases which require compositional policies.

Original: It also illustrates that SIR is agnostic to baseline RL algorithms.
Revised: It also illustrates that SIR is agnostic to baseline RL algorithms.

Original: Additional results in Stack scenario
Revised: Additional results in Stack scenario

Original: In Stack scenario, we also provide results trained with a maximum of two boxes.
Revised: In Stack scenario, we also provide results trained with a maximum of two boxes.

Original: SIR is compared with other three baselines in Fig.
Revised: SIR is compared with other three baselines in Fig.

Original: Our method is signiﬁcantly more sample efﬁcient especially when learning to stack.
Revised: Our method is signiﬁcantly more sample efﬁcient especially when learning to stack.

Original: Additional results in Maze scenario We empirically ﬁnd that SAC runs a magnitude slower in wall-clock time than PPO in n-Room maze as shown in Figure 17, therefore we use PPO-based algorithms in 4-Room maze experiments in the main paper.
Revised: Additional results in Maze scenario We empirically ﬁnd that SAC runs a magnitude slower in wall-clock time than PPO in n-Room maze as shown in Figure 17, therefore we use PPO-based algorithms in 4-Room maze experiments in the main paper.

Original: The result with SAC-based algorithms in 3-Room maze scenario is shown in Figure 18.
Revised: The result with SAC-based algorithms in 3-Room maze scenario is shown in Figure 18.

Original: SIR still outperforms SAC and SIL baselines in terms of average success rate over both the training task distribution and the hard tasks only.
Revised: SIR still outperforms SAC and SIL baselines in terms of average success rate over both the training task distribution and the hard tasks only.

Original: We also include the DS reward explained in Push scenario as a baseline for comparison in Figure 19.
Revised: We also include the DS reward explained in Push scenario as a baseline for comparison in Figure 19.

Original: Interestingly, DS surpasses all other methods when evaluated in hard cases only in the later training phase since it discovers how to navigate around the walls and push the blocking box away at the same time and remembers this policy.
Revised: Interestingly, DS surpasses all other methods when evaluated in hard cases only in the later training phase since it discovers how to navigate around the walls and push the blocking box away at the same time and remembers this policy.

Original: Episode length We are also interested in if SIR-learned policy can efﬁciently solve tasks besides the success rate reported in the main paper.
Revised: Episode length We are also interested in if SIR-learned policy can efﬁciently solve tasks besides the success rate reported in the main paper.

Original: Hence we test trained policies with different algorithms
Revised: Hence we test trained policies with different algorithms

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: on a ﬁxed set of 500 hard tasks in each scenario and report the average length of successful episodes in Table 8.
Revised: on a ﬁxed set of 500 hard tasks in each scenario and report the average length of successful episodes in Table 8.

Original: In Push and Maze scenarios, these hard tasks are directly sampled from corresponding hard case conﬁgurations, while in Stack scenario, we only sample the most difﬁcult subset of hard case conﬁgurations, i.e. stacking from scratch.
Revised: In Push and Maze scenarios, these hard tasks are directly sampled from corresponding hard case conﬁgurations, while in Stack scenario, we only sample the most difﬁcult subset of hard case conﬁgurations, i.e. stacking from scratch.

Original: SIR achieves comparable mean successful episode length with other algorithms in Push and Maze scenarios, and is signiﬁcantly more effective in Stack scenario where all other methods fail or struggle to solve.
Revised: SIR achieves comparable mean successful episode length with other algorithms in Push and Maze scenarios, and is signiﬁcantly more effective in Stack scenario where all other methods fail or struggle to solve.

Original: Pure RL
Revised: Pure RL

Original: SIL
Revised: SIL

Original: DS
Revised: DS

Original: SIR
Revised: SIR

Original: Push Unif.
Revised: Push Unif.

Original: [Table]
Revised: [Table]

Original: Push
Revised: Push

Original: [Table]
Revised: [Table]

Original: Stack (
Revised: Stack (

Original: [Table]
Revised: [Table]

Original: boxes from
Revised: boxes from

Original: [Table]
Revised: [Table]

Original: Stack (3 boxes
Revised: Stack (3 boxes

Original: [Table]
Revised: [Table]

Original: N/A N/A
Revised: N/A N/A

Original: N/
Revised: N/

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]