Original: Demonstration-Guided Reinforcement Learning with Learned Skills
Revised: Demonstration-Guided Reinforcement Learning with Learned Skills

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract: Demonstration-guided reinforcement learning (RL) is a promising approach for learning complex behaviors by leveraging both reward feedback and aset of target task demonstrations.
Revised: Abstract: Demonstration-guided reinforcement learning (RL) is a promising approach for learning complex behaviors by leveraging both reward feedback anda set of target task demonstrations.

Original: Prior approaches for demonstration-guided RLtreat every new task as an independent learning problem and attempt to follow theprovided demonstrations step-by-step, akin to a human trying to imitate a completely unseen behavior by following the demonstrator’s exact muscle movements.
Revised: Prior approaches for demonstration-guided RL treat every new task as an independent learning problem and attempt to followthe provided demonstrations step-by-step, akin to a human trying to imitate acompletely unseen behavior by following the demonstrator’s exact muscle movements.

Original: Naturally, such learning will be slow, but often new behaviors are not completelyunseen: they share subtasks with behaviors we have previously learned.
Revised: Naturally, such learning will be slow, but often new behaviors are notcompletely unseen: they share subtasks with behaviors we have previously learned.

Original: In thiswork, we aim to exploit this shared subtask structure to increase the efﬁciencyof demonstration-guided RL.
Revised: In this work, we aim to exploit this shared subtask structure to increase the efﬁciency of demonstration-guided RL.

Original: We ﬁrst learn a set of reusable skills from largeofﬂine datasets of prior experience collected across many tasks.
Revised: We ﬁrst learn a set of reusable skills fromlarge ofﬂine datasets of prior experience collected across many tasks.

Original: We then proposean algorithm for demonstration-guided RL that efﬁciently leverages the provideddemonstrations by following the demonstrated skills instead of the primitive actions,resulting in substantial performance improvements over prior demonstration-guided RL approaches.
Revised: We thenpropose Ski ll-based L earning with D emonstrations ( SkiLD ), an algorithm fordemonstration-guided RL that efﬁciently leverages the provided demonstrationsby following the demonstrated skills instead of the primitive actions, resultingin substantial performance improvements over prior demonstration-guided RLapproaches.

Original: We validate the effectiveness of our approach on long-horizonmaze navigation and complex robot manipulation tasks.
Revised: We validate the effectiveness of our approach on long-horizon mazenavigation and complex robot manipulation tasks.

Original:  Introduction
Revised:  Introduction

Original: [Figure]
Revised: [Figure]

Original: Policy
Revised: Policy

Revised: Environment

Original: [Figure]
Revised: [Figure]

Original: many different tasks efﬁcient demonstration-guided reinforce ment learning by (1) acquiring a rich motor skill repertoire from such ofﬂine data and (2) understanding and imitating the demonstrations based on the skill repertoire.

Original: Humans are remarkably efﬁcient at acquiringnew skills from demonstrations: often a singledemonstration of the desired behavior and a fewtrials of the task are sufﬁcient to master it [1, 2,3].
Revised: Humans are remarkably efﬁcient at acquiringnew skills from demonstrations: often a singledemonstration of the desired behavior and a fewtrials of the task are sufﬁcient to master it [1, 2,3].

Original: To allow for such efﬁcient learning, we canleverage a large number of previously learnedbehaviors [2, 3].
Revised: To allow for such efﬁcient learning, we canleverage a large number of previously learnedbehaviors [2, 3].

Original: Instead of imitating preciselyeach of the demonstrated muscle movements,humans can extract the performed skills and usethe rich repertoire of already acquired skills toefﬁciently reproduce the desired behavior.
Revised: Instead of imitating preciselyeach of the demonstrated muscle movements,humans can extract the performed skills and usethe rich repertoire of already acquired skills toefﬁciently reproduce the desired behavior.

Original: Demonstrations are also commonly used in reinforcement learning (RL) to guide explorationand improve sample efﬁciency [4, 5, 6, 7, 8].
Revised: Demonstrations are also commonly used in reinforcement learning (RL) to guide explorationand improve sample efﬁciency [4, 5, 6, 7, 8].

Original: However, such demonstration-guided RL approaches attempt to learn tasks from scratch :analogous to a human trying to imitate a completely unseen behavior by following everydemonstrated muscle movement, they try to imitate the primitive actions performed in the provided demonstrations.
Revised: However, such demonstration-guided RL approaches attempt to learn tasks from scratch :analogous to a human trying to imitate a completely unseen behavior by following everydemonstrated muscle movement, they try to imitate the primitive actions performed in the provided demonstrations.

Original: As with humans, such step-by-step imitation leads to brittle policies [9], andthus these approaches require many demonstrations and environment interactions to learn a new task.
Revised: As with humans, such step-by-step imitation leads to brittle policies [9], andthus these approaches require many demonstrations and environment interactions to learn a new task.

Original: We propose to improve the efﬁciency of demonstration-guided RL by leveraging prior experiencein the form of an ofﬂine “task-agnostic“ experience dataset, collected not on one but across manytasks (see Figure 1).
Revised: We propose to improve the efﬁciency of demonstration-guided RL by leveraging prior experiencein the form of an ofﬂine “task-agnostic“ experience dataset, collected not on one but across manytasks (see Figure 1).

Original: Given such a dataset, our approach extracts reusable skills: robust short-horizonbehaviors that can be recombined to learn new tasks.
Revised: Given such a dataset, our approach extracts reusable skills: robust short-horizonbehaviors that can be recombined to learn new tasks.

Original: Like a human imitating complex behaviors viathe chaining of known skills, we can use this repertoire of skills for efﬁcient demonstration-guided
Revised: Like a human imitating complex behaviors viathe chaining of known skills, we can use this repertoire of skills for efﬁcient demonstration-guided

Original: RL on a new task by guiding the policy using the demonstrated skills instead of the primitive actions.
Revised: RL on a new task by guiding the policy using the demonstrated skills instead of the primitive actions.

Original: Concretely, we propose  a demonstration-guided RL algorithm that builds on recent works in skillbased RL [10, 11] in order to learn low-dimensional representations of short-horizon skills fromofﬂine datasets. We then learn new tasks efﬁciently by leveraging these skills to follow a givenset of demonstrations.
Revised: Concretely, we propose Ski ll-based L earning with D emonstrations ( SkiLD ), a demonstration-guided RL algorithm that learns short-horizon skills from ofﬂine datasets and then learns new tasks efﬁcientlyby leveraging these skills to follow a given set of demonstrations.

Original: Across challenging navigation and robotic manipulation tasks our approachsigniﬁcantly improves the learning efﬁciency over prior demonstration-guided RL approaches.
Revised: Across challenging navigationand robotic manipulation tasks our approach signiﬁcantly improves the learning efﬁciency over priordemonstration-guided RL approaches.

Original: In summary, the contributions of our work are threefold: (1) we introduce the problem of leveragingtask-agnostic ofﬂine datasets for accelerating demonstration-guided RL on unseen tasks, (2) we propose a skill-based algorithm for efﬁcient demonstration-guided RL and (3) we show the effectivenessof our approach on a maze navigation and two complex robotic manipulation tasks.
Revised: In summary, the contributions of our work are threefold: (1) we introduce the problem of leveragingtask-agnostic ofﬂine datasets for accelerating demonstration-guided RL on unseen tasks, (2) wepropose SkiLD, a skill-based algorithm for efﬁcient demonstration-guided RL and (3) we show theeffectiveness of our approach on a maze navigation and two complex robotic manipulation tasks.

Original:  Related Work
Revised:  Related Work

Original: Imitation learning.
Revised: Imitation learning.

Original: Learning from Demonstration, also known as imitation learning [12], is acommon approach for learning complex behaviors by leveraging a set of demonstrations.
Revised: Learning from Demonstration, also known as imitation learning [10], is acommon approach for learning complex behaviors by leveraging a set of demonstrations.

Original: Mostprior approaches for imitation learning are either based on behavioral cloning (BC, [13]), whichuses supervised learning to mimic the demonstrated actions, or inverse reinforcement learning (IRL,[14, 15]), which infers a reward from the demonstrations and then trains a policy to optimize it.
Revised: Mostprior approaches for imitation learning are either based on behavioral cloning (BC, [11]), whichuses supervised learning to mimic the demonstrated actions, or inverse reinforcement learning (IRL,[12, 13]), which infers a reward from the demonstrations and then trains a policy to optimize it.

Original: However, BC commonly suffers from distribution shift and struggles to learn robust policies [9],while IRL’s joint optimization of reward and policy can result in unstable training.
Revised: However, BC commonly suffers from distribution shift and struggles to learn robust policies [9],while IRL’s joint optimization of reward and policy can result in unstable training.

Original: Demonstration-guided RL.
Revised: Demonstration-guided RL.

Original: A number of prior works aim to mitigate these problems by combiningreinforcement learning with imitation learning.
Revised: A number of prior works aim to mitigate these problems by combiningreinforcement learning with imitation learning.

Original: This allows the agent to leverage demonstrations forovercoming exploration challenges in RL while using RL to increase robustness and performance ofthe imitation learning policies.
Revised: This allows the agent to leverage demonstrations forovercoming exploration challenges in RL while using RL to increase robustness and performance ofthe imitation learning policies.

Original: Prior work on demonstration-guided RL can be categorized into threegroups: (1) approaches that use BC to initialize and regularize policies during RL training [6, 7],(2) approaches that place the demonstrations in the replay buffer of an off-policy RL algorithm [4,5 ], and (3) approaches that augment the environment rewards with rewards extracted from thedemonstrations [8, 16, 17].
Revised: Prior work on demonstration-guided RL can be categorized into threegroups: (1) approaches that use BC to initialize and regularize policies during RL training [6, 7],(2) approaches that place the demonstrations in the replay buffer of an off-policy RL algorithm [4,5 ], and (3) approaches that augment the environment rewards with rewards extracted from thedemonstrations [8, 14, 15].

Original: While these approaches improve the efﬁciency of RL, they treat eachnew task as an independent learning problem, i.e., attempt to learn policies without taking any priorexperience into account.
Revised: While these approaches improve the efﬁciency of RL, they treat eachnew task as an independent learning problem, i.e., attempt to learn policies without taking any priorexperience into account.

Original: As a result, they require many demonstrations to learn effectively, which isespecially expensive since a new set of demonstrations needs to be collected for every new task.
Revised: As a result, they require many demonstrations to learn effectively, which isespecially expensive since a new set of demonstrations needs to be collected for every new task.

Original: Online RL with ofﬂine datasets.
Revised: Online RL with ofﬂine datasets.

Original: As an alternative to expensive task-speciﬁc demonstrations,multiple recent works have proposed to accelerate reinforcement learning by leveraging task-agnosticexperience in the form of large datasets collected across many tasks [10, 18, 19,11, 20, 21].
Revised: As an alternative to expensive task-speciﬁc demonstrations,multiple recent works have proposed to accelerate reinforcement learning by leveraging task-agnosticexperience in the form of large datasets collected across many tasks [16, 17, 18, 19, 20, 21].

Original: Incontrast to demonstrations, such task-agnostic datasets can be collected cheaply from a variety ofsources like autonomous exploration [22, 23] or human tele-operation [24, 25, 26], but will lead toslower learning than demonstrations since the data is not speciﬁc to the downstream task.
Revised: Incontrast to demonstrations, such task-agnostic datasets can be collected cheaply from a variety ofsources like autonomous exploration [22, 23] or human tele-operation [24, 25, 26], but will lead toslower learning than demonstrations since the data is not speciﬁc to the downstream task.

Original: Skill-based RL.
Revised: Skill-based RL.

Original: One class of approaches for leveraging such ofﬂine datasets that is particularlysuited for learning long-horizon behaviors is skill-based RL [22, 27, 28, 29, 30, 31, 24, 32, 26, 33, 10].
Revised: One class of approaches for leveraging such ofﬂine datasets that is particularlysuited for learning long-horizon behaviors is skill-based RL [22, 27, 28, 29, 30, 31, 24, 32, 26, 33, 16].

Original: These methods extract reusable skills from task-agnostic datasets and learn new tasks by recombiningthem.
Revised: These methods extract reusable skills from task-agnostic datasets and learn new tasks by recombiningthem.

Original: Yet, such approaches perform reinforcement learning over the set of extracted skills to learnthe downstream task.
Revised: Yet, such approaches perform reinforcement learning over the set of extracted skills to learnthe downstream task.

Original: Although being more efﬁcient than RL over primitive actions, they still requiremany environment interactions to learn long-horizon tasks.
Revised: Although being more efﬁcient than RL over primitive actions, they still requiremany environment interactions to learn long-horizon tasks.

Original: In our work we combine the best of bothworlds: by using large, task-agnostic datasets and a small number of task-speciﬁc demonstrations, weaccelerate the learning of long-horizon tasks while reducing the number of required demonstrations.
Revised: In our work we combine the best of bothworlds: by using large, task-agnostic datasets and a small number of task-speciﬁc demonstrations, weaccelerate the learning of long-horizon tasks while reducing the number of required demonstrations.

Original:  Approach
Revised:  Approach

Original: Our goal is to use skills extracted from task-agnostic prior experience data to improve the efﬁciencyof demonstration-guided RL on a new task.
Revised: Our goal is to use skills extracted from task-agnostic prior experience data to improve the efﬁciencyof demonstration-guided RL on a new task.

Original: We aim to leverage a set of provided demonstrations sssssaaaaq ω ( z | s , a ) π ϕ ( a t | s t , z ) s t π θ ( z | s t ) π ϕ
Revised: We aim to leverage a set of provided demonstrations sssssaaaaq ω ( z | s , a ) π ϕ ( a t | s t , z ) s t π θ ( z | s t ) π ϕ

Original: ( a t | s t , z )
Revised: ( a t | s t , z )

Original: Environment z a t
Revised: Environment z a t

Original: Pre-Trained q ζ ( z | s 0 ) q ω ( z | s , a )
Revised: Pre-Trained q ζ ( z | s 0 ) q ω ( z | s , a )

Original: Regularization
Revised: Regularization

Original: Skill Extraction
Revised: Skill Extraction

Original: Prior & Posterior Training
Revised: Prior & Posterior Training

Original: Downstream RLs 2 ssssp a ( z | s 0 ) q ω ( z | s , a )
Revised: Downstream RLs 2 ssssp a ( z | s 0 ) q ω ( z | s , a )

Original: D ( s )
Revised: D ( s )

Original: Posterior Prior by following the performed skills as opposed to the primitive actions.
Revised: Posterior Prior by following the performed skills as opposed to the primitive actions.

Original: Therefore, we need a modelthat can (1) leverage prior data to learn a rich set of skills and (2) identify the skills performed in thedemonstrations in order to follow them.
Revised: Therefore, we need a modelthat can (1) leverage prior data to learn a rich set of skills and (2) identify the skills performed in thedemonstrations in order to follow them.

Original: Next, we formally deﬁne our problem, summarize relevantprior work on RL with learned skills and then describe our demonstration-guided RL approach.
Revised: Next, we formally deﬁne our problem, summarize relevantprior work on RL with learned skills and then describe our demonstration-guided RL approach.

Original: 3.1 Preliminaries
Revised: 3.1 Preliminaries

Original: Problem Formulation We assume access to two types of datasets: a large task-agnostic ofﬂinedataset and a small task-speciﬁc demonstration dataset.
Revised: Problem Formulation We assume access to two types of datasets: a large task-agnostic ofﬂinedataset and a small task-speciﬁc demonstration dataset.

Original: The task-agnostic dataset D = { s t , a t , ... } 99 consists of trajectories of meaningful agent behaviors, but includes no demonstrations of the targettask.
Revised: The task-agnostic dataset D = { s t , a t , ... } 98 consists of trajectories of meaningful agent behaviors, but includes no demonstrations of the targettask.

Original: We only assume that its trajectories contain short-horizon behaviors that can be reused to solvethe target task.
Revised: We only assume that its trajectories contain short-horizon behaviors that can be reused to solvethe target task.

Original: Such data can be collected without a particular task in mind using a mix of sources,e.g., via human teleoperation, autonomous exploration, or through policies trained for other tasks.
Revised: Such data can be collected without a particular task in mind using a mix of sources,e.g., via human teleoperation, autonomous exploration, or through policies trained for other tasks.

Original: Since it can be used to accelerate many downstream task that utilize similar short-term behaviorswe call it task-agnostic .
Revised: Since it can be used to accelerate many downstream task that utilize similar short-term behaviorswe call it task-agnostic .

Original: In contrast, the task-speciﬁc data is a much smaller set of demonstrationtrajectories D demo =
Revised: In contrast, the task-speciﬁc data is a much smaller set of demonstrationtrajectories D demo =

Original: { s dt , a dt , ... } that are speciﬁc to a single target task.
Revised: { s dt , a dt , ... } that are speciﬁc to a single target task.

Original: The downstream learning problem is formulated as a Markov decision process (MDP) deﬁned by atuple ( S , A , T , R, ρ, γ ) of states, actions, transition probabilities, rewards, initial state distribution, 108 and discount factor.
Revised: The downstream learning problem is formulated as a Markov decision process (MDP) deﬁned by atuple ( S , A , T , R, ρ, γ ) of states, actions, transition probabilities, rewards, initial state distribution, 107 and discount factor.

Original: We aim to learn a policy π θ ( a | s ) with parameters θ that maximizes the discountedsum of rewards J ( θ )=
Revised: We aim to learn a policy π θ ( a | s ) with parameters θ that maximizes the discountedsum of rewards J ( θ )

Original: E π (cid:2)

Original: [Equation]
Revised: [Equation]

Original: , where T is the episode horizon.
Revised: , where T is the episode horizon.

Original: Skill Prior RL Our goal is to extract skills from task-agnostic experience data and reuse them fordemonstration-guided RL.
Revised: Skill Prior RL Our goal is to extract skills from task-agnostic experience data and reuse them fordemonstration-guided RL.

Original: Prior work has investigated the reuse of learned skills for accelerating
Revised: Prior work has investigated the reuse of learned skills for accelerating

Original: RL [10].
Revised: RL [16].

Original: In this section, we will brieﬂy summarize their proposed approach Skill Prior RL (SPiRL)and then describe how our approach improves upon it in the demonstration-guided RL setting.
Revised: In this section, we will brieﬂy summarize their proposed approach Skill Prior RL (SPiRL)and then describe how our approach improves upon it in the demonstration-guided RL setting.

Original: SPiRL deﬁnes a skill as a sequence of H consecutive actions a = { a t , . .
Revised: SPiRL deﬁnes a skill as a sequence of H consecutive actions a = { a t , . .

Original: . , a t + H − 1 } , where the skill 115 horizon H is a hyperparameter.
Revised: . , a t + H − 1 } , where the skill 114 horizon H is a hyperparameter.

Original: It uses the task-agnostic data to jointly learn (1) a generative modelof skills p ( a | z ) , that decodes latent skill embeddings z into executable action sequences a , and (2) astate-conditioned prior distribution p ( z | s ) over skill embeddings.
Revised: It uses the task-agnostic data to jointly learn (1) a generative modelof skills p ( a | z ) , that decodes latent skill embeddings z into executable action sequences a , and (2) astate-conditioned prior distribution p ( z | s ) over skill embeddings.

Original: For learning a new downstreamtask, SPiRL trains a high-level skill policy π  ( z | s ) whose outputs get decoded into executable actionsusing the pre-trained skill decoder.
Revised: For learning a new downstreamtask, SPiRL trains a high-level skill policy π θ ( z | s ) whose outputs get decoded into executable actionsusing the pre-trained skill decoder.

Original: Crucially, the learned skill prior is used to guide the policy duringdownstream RL by maximizing the following divergence-regularized RL objective:
Revised: Crucially, the learned skill prior is used to guide the policy duringdownstream RL by maximizing the following divergence-regularized RL objective:

Original: [Equation]
Revised: [Equation]

Original: Here, the KL-divergence term ensures that the policy remains close to the learned skill prior, guidingexploration during RL.
Revised: Here, the KL-divergence term ensures that the policy remains close to the learned skill prior, guidingexploration during RL.

Original: By combining this guided exploration with temporal abstraction via thelearned skills, SPiRL substantially improves the efﬁciency of RL on long-horizon tasks.
Revised: By combining this guided exploration with temporal abstraction via thelearned skills, SPiRL substantially improves the efﬁciency of RL on long-horizon tasks.

Original: 3.2 Skill Representation Learning
Revised: 3.2 Skill Representation Learning

Original: We leverage SPiRL’s skill embedding model for learning our skill representation.
Revised: We leverage SPiRL’s skill embedding model for learning our skill representation.

Original: We follow priorwork on skill-based RL [26, 11] and increase the expressiveness of the skill representation byreplacing SPiRL’s low-level skill decoder p ( a | z ) with a closed-loop skill policy π  ( a | s, z )  that isconditioned on the current environment state.
Revised: We follow priorwork on skill-based RL [26, 19] and increase the expressiveness of the skill representation byreplacing SPiRL’s low-level skill decoder p ( a | z ) with a closed-loop skill policy π φ ( a | s, z ) withparameters φ that is conditioned on the current environment state.

Original: In our experiments we found this closed-loop decoderto improve performance (see Section C for an empirical comparison).
Revised: In our experiments we found thisclosed-loop decoder to improve performance (see Section C for an empirical comparison).

Original: Figure 2 (left) summarizes our skill learning model.
Revised: Figure 2 (left) summarizes our skill learning model.

Original: It consists of two parts: the skill inferencenetwork q ω ( z | s 0: H − 1 , a 0: H − 2 )  and the closed-loop skill policy π φ ( a t | s t , z t ) .
Revised: It consists of two parts: the skill inferencenetwork q ω ( z | s 0: H − 1 , a 0: H − 2 ) with parameters ω and the closed-loop skill policy π φ ( a t | s t , z t ) .

Original:  Note that in contrast to132 SPiRL the skill inference network is state-conditioned to account for the state-conditioned low-levelpolicy.
Revised: 131 Note that in contrast to SPiRL the skill inference network is state-conditioned to account for thestate-conditioned low-level policy.

Original: During training we randomly sample an H -step state-action trajectory from the task-agnosticdataset and pass it to the skill inference network, which predicts the low-dimensional skill embeddingz .
Revised: During training we randomly sample an H -step state-actiontrajectory from the task-agnostic dataset and pass it to the skill inference network, which predictsthe low-dimensional skill embedding z .

Original: This skill embedding is then input into the low-level policy π φ ( a t
Revised: This skill embedding is then input into the low-level policyπ φ ( a t

Original: | s t , z ) for every input state.
Revised: | s t , z ) for every input state.

Original: Thepolicy is trained to imitate the given action sequence, thereby learning to reproduce the behaviorsencoded by the skill embedding z .
Revised: The policy is trained to imitate the given action sequence, therebylearning to reproduce the behaviors encoded by the skill embedding z .

Original: The latent skill representation is optimized using variational inference, which leads to the full skilllearning objective:
Revised: The latent skill representation is optimized using variational inference, which leads to the full skilllearning objective:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: We use a unit Gaussian prior p ( z ) and weight the embedding regularization term with a factor β [34]
Revised: We use a unit Gaussian prior p ( z ) and weight the embedding regularization term with a factor β [34]

Original: [Table]
Revised: [Table]

Original: 3.3 Demonstration-Guided RL with Learned Skills
Revised: 3.3 Demonstration-Guided RL with Learned Skills

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: =</latexit> <latexit sha1_base64="MUjWve21UASCHuJ02UPoxZNYlZE=">AAACDnicbVBNS8NAEN3Ur1q/qh69BEvBU0lE0GNRDx4r2A9oStlsJ+3STTbsTsQS+gu8+Fe8eFDEq2dv/hu3aQ7a+mDg8d4MM/P8WHCNjvNtFVZW19Y3ipulre2d3b3y/kFLy0QxaDIppOr4VIPgETSRo4BOrICGvoC2P76a+e17UJrL6A4nMfRCOox4wBlFI/XLVU8EQkrlhRRHjIr0etr3EB4wHUAop57K3H654tScDPYycXNSITka/fKXN5AsCSFCJqjWXdeJsZdShZwJmJa8RENM2ZgOoWtoREPQvTR7Z2pXjTKwA6lMRWhn6u+JlIZaT0LfdM6u1oveTPzP6yYYXPRSHsUJQsTmi4JE2CjtWTb2gCtgKCaGUKa4udVmI6ooQ5NgyYTgLr68TFqnNdepubdnlfplHkeRHJFjckJcck7q5IY0SJMw8kieySt5s56sF+vd+pi3Fqx85pD8gfX5A06ZnYY
Revised: =</latexit> <latexit sha1_base64="MUjWve21UASCHuJ02UPoxZNYlZE=">AAACDnicbVBNS8NAEN3Ur1q/qh69BEvBU0lE0GNRDx4r2A9oStlsJ+3STTbsTsQS+gu8+Fe8eFDEq2dv/hu3aQ7a+mDg8d4MM/P8WHCNjvNtFVZW19Y3ipulre2d3b3y/kFLy0QxaDIppOr4VIPgETSRo4BOrICGvoC2P76a+e17UJrL6A4nMfRCOox4wBlFI/XLVU8EQkrlhRRHjIr0etr3EB4wHUAop57K3H654tScDPYycXNSITka/fKXN5AsCSFCJqjWXdeJsZdShZwJmJa8RENM2ZgOoWtoREPQvTR7Z2pXjTKwA6lMRWhn6u+JlIZaT0LfdM6u1oveTPzP6yYYXPRSHsUJQsTmi4JE2CjtWTb2gCtgKCaGUKa4udVmI6ooQ5NgyYTgLr68TFqnNdepubdnlfplHkeRHJFjckJcck7q5IY0SJMw8kieySt5s56sF+vd+pi3Fqx85pD8gfX5A06ZnYY

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: To leverage the learned skills for acceleratingdemonstration-guided RL on a new task, we use ahierarchical policy learning scheme (see Figure 2, right): ahigh-level policy π θ ( z | s ) outputs latent skill embeddingsz that get decoded into actions using the pre-trainedlow-level skill policy.
Revised: To leverage the learned skills for acceleratingdemonstration-guided RL on a new task, we use ahierarchical policy learning scheme (see Figure 2, right): ahigh-level policy π θ ( z | s ) outputs latent skill embeddingsz that get decoded into actions using the pre-trainedlow-level skill policy.

Original: We freeze the weights of the skillpolicy during downstream training for simplicity.
Revised: We freeze the weights of the skillpolicy during downstream training for simplicity.

Original: Our goal is to leverage the task-speciﬁc demonstrationsto guide learning of the high-level policy on the new task.
Revised: Our goal is to leverage the task-speciﬁc demonstrationsto guide learning of the high-level policy on the new task.

Original: In Section 3.1, we showed how SPiRL [10] leverages alearned skill prior pa ( z | s ) to guide exploration.
Revised: In Section 3.1, we showed how SPiRL [16] leverages alearned skill prior p ( z | s ) to guide exploration.

Original: However,this prior is task-agnostic, i.e., it encourages exploration ofall skills that are meaningful to be explored, independentof which task the agent is trying to solve.
Revised: However,this prior is task-agnostic, i.e., it encourages exploration ofall skills that are meaningful to be explored, independentof which task the agent is trying to solve.

Original: Even though
Revised: Even though

Original: SPiRL’s objective makes learning with a large number ofskills more efﬁcient, it encourages the policy to exploremany skills that are not relevant to the downstream task.
Revised: SPiRL’s objective makes learning with a large number ofskills more efﬁcient, it encourages the policy to exploremany skills that are not relevant to the downstream task.

Original: In this work, we propose to extend the skill prior guidedapproach and leverage target task demonstrations to additionally learn a task-speciﬁc skill distribution, which we call skill posterior q ζ ( z | s )  (in contrast tothe skill prior it is conditioned on the target task, hence “posterior”).
Revised: In this work, we propose to extend the skill prior guidedapproach and leverage target task demonstrations to additionally learn a task-speciﬁc skill distribution, which we call skill posterior q ζ ( z | s ) with parametersζ (in contrast to the skill prior it is conditioned on the target task, hence “posterior”).

Original: We train this skill posterior byusing the pre-trained skill inference model q ω ( z | s 0: H − 1 , a 0: H − 2 ) to extract the embeddings for the skills performed in the demonstration sequences (see Figure 2, middle):
Revised: We train thisskill posterior by using the pre-trained skill inference model q ω ( z | s 0: H − 1 , a 0: H − 2 ) to extract the embeddings for the skills performed in the demonstration sequences (see Figure 2, middle):

Original: [Equation]
Revised: [Equation]

Original: where D KL denotes the Kullback-Leibler divergence.
Revised: where D KL denotes the Kullback-Leibler divergence.

Original: A naive approach for leveraging the skill posterior is to simply use it to replace the skill prior in
Revised: A naive approach for leveraging the skill posterior is to simply use it to replace the skill prior in

Original: Equation 1, i.e., to regularize the policy to stay close to the skill posterior in every state.
Revised: Equation 1, i.e., to regularize the policy to stay close to the skill posterior in every state.

Original: However,the trained skill posterior is only accurate within the support of the demonstration dataset (cid:98)D demo (cid:99) .
Revised: However,the trained skill posterior is only accurate within the support of the demonstration dataset (cid:98)D demo (cid:99) , 168 because by deﬁnition it was only trained on demonstration sequences.

Original: Since |D demo | (cid:28) |D| , this  support will only be a small subset of all states (see Figure 3) and thus the170 skill posterior will often provide incorrect guidance in states outside the demonstrations’ support.
Revised: Since |D demo | (cid:28) |D| , this 169 support will only be a small subset of all states (see Figure 3) and thus the skill posterior will oftenprovide incorrect guidance in states outside the demonstrations’ support.

Original: Instead, we propose to use a three-part objective to guide the policy during downstream learning.
Revised: Instead, we propose to use a three-part objective to guide the policy during downstream learning.

Original: Our goal in formulating this objective is to (1) follow the skill posterior within the support of thedemonstrations, (2) follow the skill prior outside the demonstration support, and (3) encouragethe policy to reach states within the demonstration support.
Revised: Our goal in formulating this objective is to (1) follow the skill posterior within the support of thedemonstrations, (2) follow the skill prior outside the demonstration support, and (3) encouragethe policy to reach states within the demonstration support.

Original: Crucial for all three components is todetermine whether a given state is within the support of the demonstration data.
Revised: Crucial for all three components is todetermine whether a given state is within the support of the demonstration data.

Original: We propose to usea learned discriminator D ( s ) to answer this question.
Revised: We propose to usea learned discriminator D ( s ) to answer this question.

Original: D ( s ) is a binary classiﬁer that distinguishesdemonstration and non-demonstration states and it is trained using samples from the demonstrationand task-agnostic datasets, respectively.
Revised: D ( s ) is a binary classiﬁer that distinguishesdemonstration and non-demonstration states and it is trained using samples from the demonstrationand task-agnostic datasets, respectively.

Revised: In summary, our algorithm pre-trains the following components:

Revised: low-level skill policy: π φ ( a | s, z ) task-agnostic skill prior: p ( z | s ) task-speciﬁc skill posterior: q ζ ( z | s ) demo vs non-demo discriminator: D ( s )

Revised: Note that the ﬁrst two components can be reused for arbitrary target tasks.

Revised: Thus, only the lattertwo components need to be re-trained for a new target task using the demonstration data.

Revised: Since allpre-training is done ofﬂine without the need for e.g. adversarial training or training in the RL loop,we ﬁnd that pre-training converges stably across a wide range of architecture and hyperparameterchoices and results can be separately visualized and debugged.

Original: Once trained, we use its output to weight terms in ourobjective that regularize the  policy  towards the skill prior or posterior.
Revised: Once all components are pre-trained, we use the discriminator’s output to weight terms in our objectivethat regularize the high-level policy π θ ( z | s ) towards the skill prior or posterior.

Original: Additionally, we provide areward bonus for reaching states which the discriminator classiﬁes as being within the demonstrationsupport.
Revised: Additionally, weprovide a reward bonus for reaching states which the discriminator classiﬁes as being within thedemonstration support.

Original: This results in the following term J t for our full RL objective:
Revised: This results in the following term J t for SkiLD’s full RL objective:

Original: [Equation]
Revised: [Equation]

Revised: prior regularization , with ˜ r ( s t , z t ) =

Original: (cid:123)(cid:122) (cid:125) [Table] − αD KL (
Revised: (1 − κ ) · r ( s t , z t ) + κ · (cid:2) log D (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation] [Equation]
Revised: log (cid:0) 1 − D (

Original: [Equation]
Revised: [Equation]

Original: t )) (cid:124)(cid:123)(cid:122) (cid:125) prior regularization , with ˜ r
Revised: t ) (cid:1)(cid:3) (cid:124)

Original: ( s t , z t ) =
Revised: [Equation]

Original: (1 − κ ) · r ( s t , z t ) + κ · (cid:2) log D ( s t ) − log (cid:0) 1 − D ( s t ) (cid:1)(cid:3) (cid:124) (cid:123)(cid:122) (cid:125) discriminator reward .
Revised: (cid:125) discriminator reward .

Original: The weighting factor κ is a hyperparameter; α and α q are either constant or tuned automaticallyvia dual gradient descent [36].
Revised: The weighting factor κ is a hyperparameter; α and α q are either constant or tuned automaticallyvia dual gradient descent [36].

Original: The discriminator reward follows common formulations used inadversarial imitation learning [37, 38, 8, 39].
Revised: The discriminator reward follows common formulations used inadversarial imitation learning [37, 38, 8, 39].

Original: Our formulation combines IRL-like and BC-likeobjectives by using learned rewards and trying to match the demonstration’s skill distribution.
Revised: Our formulation combines IRL-like and BC-likeobjectives by using learned rewards and trying to match the demonstration’s skill distribution.

Original: For policy optimization, we use a modiﬁed version of the SPiRL algorithm [10], which itself is basedon Soft Actor-Critic [40].
Revised: For policy optimization, we use a modiﬁed version of the SPiRL algorithm [16], which itself is basedon Soft Actor-Critic [40].

Original: Concretely, we replace the environment reward with the discriminatoraugmented reward and all prior divergence terms with our new, weighted prior-posterior-divergenceterms from equation 4 (for the full algorithm see appendix, Section A).
Revised: Concretely, we replace the environment reward with the discriminatoraugmented reward and all prior divergence terms with our new, weighted prior-posterior-divergenceterms from equation 4 (for the full algorithm see appendix, Section A).

Original:  Experiments
Revised:  Experiments

Original: In this paper, we propose to leverage a large ofﬂine experience dataset for efﬁcient demonstrationguided RL.
Revised: In this paper, we propose to leverage a large ofﬂine experience dataset for efﬁcient demonstrationguided RL.

Original: We aim to answer the following questions: (1) Can the use of task-agnostic priorexperience improve the efﬁciency of demonstration-guided RL?
Revised: We aim to answer the following questions: (1) Can the use of task-agnostic prior experience improve the efﬁciency of demonstration-guided RL?

Original: (2) Does the reuse of pre-trainedskills reduce the number of required target-speciﬁc demonstrations?
Revised: (2) Does the reuse of pre-trainedskills reduce the number of required target-speciﬁc demonstrations?

Original: In what scenarios does thecombination of prior experience and demonstrations lead to the largest efﬁciency gains?
Revised: In what scenarios does thecombination of prior experience and demonstrations lead to the largest efﬁciency gains?

Original: 4.1 Experimental Setup and Comparisons
Revised: 4.1 Experimental Setup and Comparisons

Original: To evaluate whether our method  can efﬁciently usethe task-agnostic data, we compare it to priordemonstration-guided RL approaches on three complex, long-horizon tasks: a 2D maze navigationtask, a robotic kitchen manipulation task and a robotic ofﬁce cleaning task (see Figure 4, left).
Revised: To evaluate whether our method SkiLD can efﬁciently use task-agnostic data, we compare it to priordemonstration-guided RL approaches on three complex, long-horizon tasks: a 2D maze navigationtask, a robotic kitchen manipulation task and a robotic ofﬁce cleaning task (see Figure 4, left).

Original: Maze Navigation.
Revised: Maze Navigation.

Original: We adapt the maze navigation task from Pertsch et al.
Revised: We adapt the maze navigation task from Pertsch et al.

Original: [10] and increase taskcomplexity by adding randomness to the agent’s initial position.
Revised: [16] and increase taskcomplexity by adding randomness to the agent’s initial position.

Original: The agent needs to navigatethrough the maze to a ﬁxed goal position using planar velocity commands.
Revised: The agent needs to navigate throughthe maze to a ﬁxed goal position using planar velocity commands.

Original: It only receives a binaryreward upon reaching the goal.
Revised: It only receives a binary rewardupon reaching the goal and its observations consist of its 2D position and velocity.

Original: This environment is challenging for prior demonstration-guidedpre stable it RL approaches since demonstrations of the task span hundreds of time steps, making step-by-stepimitation of primitive actions inefﬁcient.
Revised: This environmentis challenging for prior demonstration-guided RL approaches since demonstrations of the task spanhundreds of time steps, making step-by-step imitation of primitive actions inefﬁcient.

Original: We collect a task-agnostic ofﬂine experience dataset with3000 sequences using a motion planner to ﬁnd paths between randomly sampled start-goal pairs.
Revised: We collect atask-agnostic ofﬂine experience dataset with 3000 sequences using a motion planner to ﬁnd pathsbetween randomly sampled start-goal pairs.

Original: Thisdata can be used to extract relevant short-horizon skills like navigating hallways or passing throughnarrow doors.
Revised: This data can be used to extract relevant short-horizonskills like navigating hallways or passing through narrow doors.

Original: For the target task we sample an unseen start-goal pair and collect 5 demonstrationsfor reaching the goal position from states sampled around the start position.
Revised: For the target task we sample anunseen start-goal pair and collect 5 demonstrations for reaching the goal position from states sampledaround the start position.

Original: [Figure]
Revised: [Figure]

Original: demon improve  has access 
Revised: improve efﬁciency even if the agent has access to large amounts of prior experience.

Original: Robot Kitchen Environment.
Revised: Robot Kitchen Environment.

Original: We use the environment of Gupta et al.
Revised: We use the environment of Gupta et al.

Original: [24] in which a 7DOFrobot arm needs to perform a sequence of foursubtasks, such as opening the microwave orswitching on the light, in the correct order.
Revised: [24] in which a 7DOFrobot arm needs to perform a sequence of foursubtasks, such as opening the microwave orswitching on the light, in the correct order.

Original: Theagent receives a binary reward upon completionof each consecutive subtask. 
Revised: Theagent receives a binary reward upon completion of each consecutive subtask. Following

Revised: Gupta et al.

Revised: [24] the agent’s 30-dimensional observation consists of its joint velocities as wellas low-dimensional representations of state ofmovable objects, like their positions or openingangles.

Revised: It outputs 9-dimensional action commands, including two continuous closing actionsfor the gripper ﬁngers.

Original: In addition to thelong task horizon, this environment requires precise control of a high-DOF manipulator, testingthe scalability of our approach.
Revised: In addition to the longtask horizon, this environment requires precisecontrol of a high-DOF manipulator, testing thescalability of our approach.

Original: We use 603 teleoperated sequences performing various subtaskcombinations (from Gupta et al.
Revised: We use 603 teleoperated sequences performing various subtaskcombinations (from Gupta et al.

Original: [24]) as ourtask-agnostic experience dataset D and separate 224 a set of 20 demonstrations for one particularsequence of subtasks, which we deﬁne as ourtarget task (see Figure 4, middle).
Revised: [24]) as ourtask-agnostic experience dataset D and separate 239 a set of 20 demonstrations for one particular sequence of subtasks, which we deﬁne as our target task (see Figure 4, middle).

Revised: From the ofﬂinedata the agent learns basic kitchen manipulationskills like opening the microwave or turning onthe stove, which need to be recombined in novelways to solve the target task.

Original: Robot Ofﬁce Environment.
Revised: Robot Ofﬁce Environment.

Original: A 5 DOF robotarm needs to clean an ofﬁce environment byplacing objects in their target bins or puttingthem in a drawer.
Revised: A 5 DOF robotarm needs to clean an ofﬁce environment byplacing objects in their target bins or puttingthem in a drawer.

Original: It receives binary rewards forthe completion of each subtask.
Revised: It receives binary rewardsfor the completion of each subtask.

Revised: The 97dimensional observation consists of the positionand orientation of the agent’s end-effector andall movable objects in the scene.

Revised: The agent controls an 8-dimensional action space of end-effector position and orientation and a continuous gripper

Revised: [Table]

Revised: KL  policy || prior

Original:  In addition tothe challenges of the kitchen environment, thistask tests the ability of our approach to learnlong-horizon behaviors with freely manipulatable objects.
Revised: [Table] [Figure] closing action. In addition to the challenges of the kitchen environment, this task tests the ability ofour approach to learn long-horizon behaviors with freely manipulatable objects.

Original: We collect 2400 training trajectories by placing the objects at randomized positions in the environment and performing randomsubtasks using scripted policies.
Revised: We collect 2400training trajectories by placing the objects at randomized positions in the environment and performingrandom subtasks using scripted policies.

Revised: From this, the agent learns general object manipulationskills like picking, placing and opening or closing drawers.

Original: We also collect 50 demonstrations for the unseen target taskwith new object locations and subtask sequence.
Revised: We also collect 50 demonstrations for theunseen target task with new object locations and subtask sequence.

Original: We compare our approach to multiple priordemonstration-guided RL approaches that represent the different classes of existing algorithmsintroduced in Section 2.
Revised: We compare our approach to multiple prior demonstration-guided RL approaches that representthe different classes of existing algorithms introduced in Section 2.

Original: In contrast to ourmethod, these approaches are not designed toleverage task-agnostic prior experience: BC + RL initializes a policy with behavioral cloning of thedemonstrations, then continues to apply BC loss while ﬁnetuning the policy with Soft Actor-Critic(SAC, [40]), representative of [6, 7].
Revised: In contrast to SkiLD, theseapproaches are not designed to leverage task-agnostic prior experience: BC + RL initializes a policywith behavioral cloning of the demonstrations, then continues to apply BC loss while ﬁnetuning thepolicy with Soft Actor-Critic (SAC, [40]), representative of [6, 7].

Original: GAIL + RL [8] combines rewards from the environment andadversarial imitation learning (GAIL, [15]), and optimizes the policy using PPO [41].
Revised: GAIL + RL [8] combines rewardsfrom the environment and adversarial imitation learning (GAIL, [13]), and optimizes the policyusing PPO [41].

Original: Demo Replayinitializes the replay buffer of an SAC agent with the demonstrations and uses them with prioritizedreplay during updates, representative of [4].
Revised: Demo Replay initializes the replay buffer of an SAC agent with the demonstrationsand uses them with prioritized replay during updates, representative of [4].

Original: We also compare our approach to RL-only methods toshow the beneﬁt of using demonstration data: SAC [40] is a state-of-the-art model-free RL algorithm,it neither uses ofﬂine experience nor demonstrations.
Revised: We also compare ourapproach to RL-only methods to show the beneﬁt of using demonstration data: SAC [40] is a stateof-the-art model-free RL algorithm, it neither uses ofﬂine experience nor demonstrations.

Original: SPiRL [10] extracts skills from task-agnosticexperience and performs prior-guided RL on the target task (see Section 3.1) 3 .
Revised: SPiRL[16] extracts skills from task-agnostic experience and performs prior-guided RL on the target task(see Section 3.1) 3 .

Original: Finally, we compare toa baseline that combines skills learned from task-agnostic data with target task demonstrations:Skill
Revised: Finally, we compare to a baseline that combines skills learned from task-agnosticdata with target task demonstrations:

Original:  BC+RL encodes the demonstrations with the pre-trained skill encoder and runs BC for the high-levelskill policy, then ﬁnetunes on the target task using SAC.
Revised: Skill BC+RL encodes the demonstrations with the pre-trainedskill encoder and runs BC for the high-level skill policy, then ﬁnetunes on the target task using SAC.

Original: For further details on the environments, datacollection, and implementation, [Table] KL  policy || prior  [Table]
Revised: For further details on the environments, data collection, and implementation, see appendix Section B.

Original: [Figure]

Original: 4.2 Demonstration-Guided RL with Learned Skills
Revised: 4.2 Demonstration-Guided RL with Learned Skills

Original: Maze Navigation.
Revised: Maze Navigation.

Original: We compare the downstream task performance of the tested methods on themaze navigation task in Figure 4 (right).
Revised: We compare the downstream task performance of the tested methods on themaze navigation task in Figure 4 (right).

Original: Prior approaches for demonstration-guided RL struggle tolearn the task since task-rewards are sparse and only ﬁve demonstrations are provided.
Revised: Prior approaches for demonstration-guided RL struggle tolearn the task since task-rewards are sparse and only ﬁve demonstrations are provided.

Original: With suchsmall coverage, behavioral cloning of the demonstrations’ primitive actions leads to brittle policieswhich are hard to ﬁnetune (for an analysis of the inﬂuence of the number of demonstrations, see
Revised: With suchsmall coverage, behavioral cloning of the demonstrations’ primitive actions leads to brittle policieswhich are hard to ﬁnetune (for an analysis of the inﬂuence of the number of demonstrations, see

Original: Section 4.3).
Revised: Section 4.3).

Original: The Replay agent improves over SAC without demonstrations and partly succeeds atthe task, but learning is slow.
Revised: The Replay agent improves over SAC without demonstrations and partly succeeds at thetask, but learning is slow.

Original: The GAIL+RL approach is able to follow part of the demonstratedbehavior, but fails to reach the ﬁnal goal and as a result does not receive the sparse environmentreward (see Figure 8 for qualitative results).
Revised: The GAIL+RL approach is able to follow part of the demonstrated behavior,but fails to reach the ﬁnal goal and as a result does not receive the sparse environment reward (see Figure 8 for qualitative results).

Original: SPiRL and Skill BC+RL, in contrast, are able to leverageofﬂine, task-agnostic experience to learn to occasionally solve the task, but require a substantialamount of environment interactions: SPiRL’s learned, task-agnostic skill prior and Skill BC+RL’suniform skill prior during SAC ﬁnetuning encourage the policy to try many skills before convergingto the ones that solve the downstream task 4 .
Revised: SPiRL and Skill BC+RL, in contrast, are able to leverage ofﬂine,task-agnostic experience to learn to occasionally solve the task, but require a substantial amount ofenvironment interactions: SPiRL’s learned, task-agnostic skill prior and Skill BC+RL’s uniform skill prior during SAC ﬁnetuning encourage the policy to try many skills before converging to the onesthat solve the downstream task 4 .

Original: In contrast, our approach  leverages the task-speciﬁc skillposterior to quickly explore the relevant skills, leading to signiﬁcant efﬁciency gains (see Figure 9 fora comparison of the exploration of our method vs. SPiRL).
Revised: In contrast, our approach SkiLD leverages the task-speciﬁc skillposterior to quickly explore the relevant skills, leading to signiﬁcant efﬁciency gains (see Figure 9 fora comparison of the exploration of SkiLD vs. SPiRL).

Original: We qualitatively analyze our approach in Figure 5: we visualize the output of the discriminator
Revised: We qualitatively analyze our approach in Figure 5: we visualize the output of the discriminator

Original: D ( s ) , and the divergences between policy and skill prior and posterior.
Revised: D ( s ) , and the divergences between policy and skill prior and posterior.

Original: The discriminator accuratelyestimates the demonstration support, providing a good weighting for prior and posterior regularization,as well as a dense reward bonus.
Revised: The discriminator accuratelyestimates the demonstration support, providing a good weighting for prior and posterior regularization,as well as a dense reward bonus.

Original: The policy successfully minimizes divergence to the task-speciﬁcskill posterior only within the demonstration support and follows the skill prior otherwise.
Revised: The policy successfully minimizes divergence to the task-speciﬁcskill posterior only within the demonstration support and follows the skill prior otherwise.

Original: Robotic Manipulation.
Revised: Robotic Manipulation.

Original: We show the performance comparison on the robotic manipulation tasksin Figure 4 (right) 5 .
Revised: We show the performance comparison on the robotic manipulation tasksin Figure 4 (right) 5 .

Original: Both tasks are more challenging since they require precise control of a high-DOFmanipulator.
Revised: Both tasks are more challenging since they require precise control of a high-DOFmanipulator.

Original: We ﬁnd that approaches for demonstration-guided RL that do not leverage task-agnosticexperience struggle to learn either of the tasks since following the demonstrations step-by-step isinefﬁcient and prone to accumulating errors.
Revised: We ﬁnd that approaches for demonstration-guided RL that do not leverage task-agnosticexperience struggle to learn either of the tasks since following the demonstrations step-by-step isinefﬁcient and prone to accumulating errors.

Original: SPiRL, in contrast, is able to learn meaningful skillsfrom the ofﬂine datasets, but struggles to explore the task-relevant skills and therefore learns slowly.
Revised: SPiRL, in contrast, is able to learn meaningful skillsfrom the ofﬂine datasets, but struggles to explore the task-relevant skills and therefore learns slowly.

Original: Worse yet, the uniform skill prior used in Skill BC+RL’s SAC ﬁnetuning is even less suited for thetarget task and prevents the  agent from learning the task altogether.
Revised: Worse yet, the uniform skill prior used in Skill BC+RL’s SAC ﬁnetuning is even less suited for thetarget task and leads the policy to deviate from the BC initialization early on in training, preventingthe agent from learning the task altogether (for pure BC performance, see appendix, Figure 13).

Original: Our approach, however, usesthe learned skill posterior to guide the chaining of the extracted skills and thereby learns to solvethe tasks efﬁciently, showing how to effectively combine task-agnostic and task-speciﬁc data fordemonstration-guided RL.
Revised: Ourapproach, however, uses the learned skill posterior to guide the chaining of the extracted skills andthereby learns to solve the tasks efﬁciently, showing how SkiLD effectively combines task-agnosticand task-speciﬁc data for demonstration-guided RL.

Original: 4.3 Ablation Studies
Revised: 4.3 Ablation Studies

Original: [Figure]
Revised: [Figure]

Revised: demonstration dataset |D demo | on the maze navigation task ( left ) and ablate the components of our objective on the kitchen manipulation task ( right ).

Original: In Figure 6 (left) we test the robustness of our approach to the number of demonstrations in themaze navigation task and compare to BC+RL,which we found to work best across differentdemonstration set sizes.
Revised: In Figure 6 (left) we test the robustness of our approach to the number of demonstrations in themaze navigation task and compare to BC+RL,which we found to work best across differentdemonstration set sizes.

Original: Both approaches beneﬁt from more demonstrations, but our approach is able to learn with much fewer demonstrations by using prior experience.
Revised: Both approaches beneﬁt from more demonstrations, but our approachis able to learn with much fewer demonstrationsby using prior experience.

Original: While BC+RL learns each low-level action from the demonstrations, our approach merely learns to recombine skills it has already mastered using the ofﬂine data, thus requiring less dense supervision and fewer demonstrations.
Revised: While BC+RL learnseach low-level action from the demonstrations, SkiLD merely learns to recombine skills it hasalready mastered using the ofﬂine data, thus requiring less dense supervision and fewer demonstrations.

Original: We also ablate the components of our RL objective on the kitchen task in Figure 6 (right).
Revised: We also ablate the components ofour RL objective on the kitchen task in Figure 6 (right).

Original: Removing the discriminator rewardbonus (" no-GAIL ") slows convergence since the agent lacks a dense reward signal.
Revised: Removing the discriminator reward bonus(" no-GAIL ") slows convergence since the agent lacks a dense reward signal.

Original: Naively replacingthe skill prior in the SPiRL objective of Equation 1 with the learned skill posterior (" post-only ")fails since the agent follows the skill posterior outside its support.
Revised: Naively replacing theskill prior in the SPiRL objective of Equation 1 with the learned skill posterior (" post-only ") failssince the agent follows the skill posterior outside its support.

Original: Removing the skill posterior andoptimizing a discriminator bonus augmented reward using SPiRL (" no-post ") fails because the agentcannot efﬁciently explore the rich skill space.
Revised: Removing the skill posterior andoptimizing a discriminator bonus augmented reward using SPiRL (" no-post ") fails because the agentcannot efﬁciently explore the rich skill space.

Original: Finally, we show the efﬁcacy of our approach in thepure imitation setting, without environment rewards, in appendix, Section E.
Revised: Finally, we show the efﬁcacy of our approach in thepure imitation setting, without environment rewards, in appendix, Section E.

Original: 4.4 Data Alignment Analysis
Revised: 4.4 Data Alignment Analysis

Original: [Figure]

Original: SPiRL ence

Original: We aim to analyze in what scenarios the use of demonstrations in addition to task-agnostic experience is mostbeneﬁcial.
Revised: We aim to analyze in what scenarios the use of demonstrations in addition to task-agnostic experienceis most beneﬁcial.

Original: In particular, we evaluate how the alignmentbetween the distribution of observed behaviors in the taskagnostic dataset and the target behaviors inﬂuences learning efﬁciency.
Revised: In particular, we evaluate how the alignment between the distribution of observedbehaviors in the task-agnostic dataset and the target behaviors inﬂuences learning efﬁciency.

Original: We choose two different target tasks inthe kitchen environment, one with good and one with badalignment between the behavior distributions, and compare our method, which uses demonstrations, to SPiRL,which only relies on the task-agnostic data 6 . 
Revised: Wechoose two different target tasks in the kitchen environment, one with good and one with badalignment between the behavior distributions, and compare our method, which uses demonstrations,to SPiRL, which only relies on the task-agnostic data 6 . [Figure]

Original:  In the well-aligned case (Figure 7, solid lines), we ﬁndthat both approaches learn the task efﬁciently.
Revised: [Figure] In the well-aligned case (Figure 7, solid lines), we ﬁndthat both approaches learn the task efﬁciently.

Original: Since theskill prior encourages effective exploration on the downstream task, the beneﬁt of the additional demonstrationsleveraged in our method is marginal.
Revised: Since theskill prior encourages effective exploration on the downstream task, the beneﬁt of the additional demonstrationsleveraged in our method is marginal.

Original: In contrast, if taskagnostic data and downstream task are not well-aligned(Figure 7, dashed), SPiRL struggles to learn the task sinceit cannot maximize task reward and minimize divergencefrom the mis-aligned skill prior at the same time.
Revised: In contrast, if taskagnostic data and downstream task are not well-aligned(Figure 7, dashed), SPiRL struggles to learn the task sinceit cannot maximize task reward and minimize divergencefrom the mis-aligned skill prior at the same time.

Original: Ourapproachlearns more reliably by encouraging thepolicy to reach demonstration-like states  and thenfollow the skill posterior, which by design is well-aligned with the target task.
Revised: Ourapproach more reliably by encouraging the to like states In summary, our analysis ﬁnds that approaches whichleverage both task-agnostic data and demonstrations, improve over methods that use either of the data sourcesalone across all tested tasks. We ﬁnd that combining thedata sources is particularly beneﬁcial in two cases:

Original: Thus, the gains fromour approach are largest when it (a) focuses a skill priorthat explores a too wide range of skills ontask-relevant skills or (b) compensates for mis-alignment between task-agnostic data and target task.
Revised: • Diverse Task-Agnostic Data. Demonstrations can focus exploration on task-relevant skills if the task-agnostic skill prior explores a too large set of skills (see Section 4.2). • Mis-Aligned Task-Agnostic Data.

Revised: Demonstrations can compensate mis-alignment be-

Revised: tween task-agnostic data and target task by guiding exploration with the skill posterior

Revised: instead of the mis-aligned prior.

Revised: Both scenarios are important since it is challenging to control the quality of the prior experiencedataset, particularly if collected at scale from multiple sources.

Revised: Our experiments show that thecombination of prior experience and demonstrations can make the learner more robust to ﬂuctuationsin the characteristics of the prior experience dataset.

Original:  Conclusion
Revised:  Conclusion

Original: We have proposed  a novel approach for demonstration-guided RL that is able to jointly leveragetask-agnostic experience datasets and task-speciﬁc demonstrations for accelerated learning of unseentasks.
Revised: We have proposed SkiLD, a novel approach for demonstration-guided RL that is able to jointlyleverage task-agnostic experience datasets and task-speciﬁc demonstrations for accelerated learningof unseen tasks.

Original: In three challenging environments our approach learns unseen tasks more efﬁciently than both,prior demonstration-guided RL approaches that are not able to leverage task-agnostic experience, aswell as skill-based RL methods that cannot effectively incorporate demonstrations.
Revised: In three challenging environments our approach learns unseen tasks more efﬁcientlythan both, prior demonstration-guided RL approaches that are not able to leverage task-agnosticexperience, as well as skill-based RL methods that cannot effectively incorporate demonstrations.