Original: M ULTITASK S OFT O
Revised: M ULTITASK S OFT O

Original: PTION L EARNING
Revised: PTION L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present Multitask Soft Option Learning ( MSOL ), a hierarchical multitask framework based on Planning as Inference.
Revised: We present Multitask Soft Option Learning ( MSOL ), a hierarchical multitask framework based on Planning as Inference.

Original: MSOL extends the concept of options, using separate variational posteriors for each task, regularized by a shared prior.
Revised: MSOL extends the concept of options, using separate variational posteriors for each task, regularized by a shared prior.

Original: This allows ﬁne-tuning of options for new tasks without forgetting their learned policies, leading to faster training without reducing the expressiveness of the hierarchical policy.
Revised: This allows ﬁne-tuning of options for new tasks without forgetting their learned policies, leading to faster training without reducing the expressiveness of the hierarchical policy.

Original: MSOL avoids several instabilities during training in a multitask setting and provides a natural way to learn both intra-option policies and their terminations.
Revised: MSOL avoids several instabilities during training in a multitask setting and provides a natural way to learn both intra-option policies and their terminations.

Original: We demonstrate empirically that MSOL signiﬁcantly outperforms both hierarchical and ﬂat transfer-learning baselines in challenging multi-task environments.
Revised: We demonstrate empirically that MSOL signiﬁcantly outperforms both hierarchical and ﬂat transfer-learning baselines in challenging multi-task environments.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A key challenge in Reinforcement Learning ( RL ) is to scale current approaches to higher complexity tasks without requiring a prohibitive number of environmental interactions.
Revised: A key challenge in Reinforcement Learning ( RL ) is to scale current approaches to higher complexity tasks without requiring a prohibitive number of environmental interactions.

Original: However, for many tasks, it is possible to construct or learn efﬁcient exploration priors that allow to focus on more relevant parts of the state-action space, reducing the number of required interactions.
Revised: However, for many tasks, it is possible to construct or learn efﬁcient exploration priors that allow to focus on more relevant parts of the state-action space, reducing the number of required interactions.

Original: These include, for example, reward shaping (Ng et al., 1999; Konidaris & Barto, 2006), curriculum learning (Bengio et al., 2009), some meta-learning algorithms (Wang et al., 2016; Duan et al., 2016; Gupta et al., 2018;
Revised: These include, for example, reward shaping (Ng et al., 1999; Konidaris & Barto, 2006), curriculum learning (Bengio et al., 2009), some meta-learning algorithms (Wang et al., 2016; Duan et al., 2016; Gupta et al., 2018;

Original: Houthooft et al., 2018; Xu et al., 2018), and transfer learning (Caruana, 1997; Taylor & Stone, 2011;
Revised: Houthooft et al., 2018; Xu et al., 2018), and transfer learning (Caruana, 1997; Taylor & Stone, 2011;

Original: Bengio, 2012; Parisotto et al., 2015; Rusu et al., 2015; Teh et al., 2017).
Revised: Bengio, 2012; Parisotto et al., 2015; Rusu et al., 2015; Teh et al., 2017).

Original: One promising way to capture prior knowledge is to decompose policies into a hierarchy of subpolicies (or skills) that can be reused and combined in novel ways to solve new tasks (Dayan &
Revised: One promising way to capture prior knowledge is to decompose policies into a hierarchy of subpolicies (or skills) that can be reused and combined in novel ways to solve new tasks (Dayan &

Original: Hinton, 1993; Thrun & Schwartz, 1995; Parr & Russell, 1998; Sutton et al., 1999; Barto & Mahadevan, 2003).
Revised: Hinton, 1993; Thrun & Schwartz, 1995; Parr & Russell, 1998; Sutton et al., 1999; Barto & Mahadevan, 2003).

Original: The idea of Hierarchical RL ( HRL ) is also supported by ﬁndings that humans appear to employ a hierarchical mental structure when solving tasks (Botvinick et al., 2009; Collins & Frank, 2016).
Revised: The idea of Hierarchical RL ( HRL ) is also supported by ﬁndings that humans appear to employ a hierarchical mental structure when solving tasks (Botvinick et al., 2009; Collins & Frank, 2016).

Original: In such a hierarchical RL policy, lower-level, temporally extended skills yield directed behavior over multiple time steps.
Revised: In such a hierarchical RL policy, lower-level, temporally extended skills yield directed behavior over multiple time steps.

Original: This has two advantages: i) it allows efﬁcient exploration, as the target states of skills can be reached without having to explore much of the state space in between, and ii) directed behavior also reduces the variance of the future reward, which accelerates convergence of estimates thereof.
Revised: This has two advantages: i) it allows efﬁcient exploration, as the target states of skills can be reached without having to explore much of the state space in between, and ii) directed behavior also reduces the variance of the future reward, which accelerates convergence of estimates thereof.

Original: On the other hand, while a hierarchical approach can therefore signiﬁcantly speed up exploration and training, it can also severely limit the expressiveness of the ﬁnal policy and lead to suboptimal performance when the temporally extended skills are not able to express the required policy for the task at hand (Mankowitz et al., 2014).
Revised: On the other hand, while a hierarchical approach can therefore signiﬁcantly speed up exploration and training, it can also severely limit the expressiveness of the ﬁnal policy and lead to suboptimal performance when the temporally extended skills are not able to express the required policy for the task at hand (Mankowitz et al., 2014).

Original: Many methods exist for constructing and/or learning skills for particular tasks (Dayan & Hinton, 1993;
Revised: Many methods exist for constructing and/or learning skills for particular tasks (Dayan & Hinton, 1993;

Original: Sutton et al., 1999; McGovern & Barto, 2001; Menache et al., 2002; S¸ims¸ek & Barto, 2009;
Revised: Sutton et al., 1999; McGovern & Barto, 2001; Menache et al., 2002; S¸ims¸ek & Barto, 2009;

Original: Gregor et al., 2016; Kulkarni et al., 2016; Bacon et al., 2017; Nachum et al., 2018a).
Revised: Gregor et al., 2016; Kulkarni et al., 2016; Bacon et al., 2017; Nachum et al., 2018a).

Original: Training on multiple tasks simultaneously is one promising approach to learn skills that are both relevant and generalise across tasks (Thrun & Schwartz, 1995; Pickett & Barto, 2002;
Revised: Training on multiple tasks simultaneously is one promising approach to learn skills that are both relevant and generalise across tasks (Thrun & Schwartz, 1995; Pickett & Barto, 2002;

Original: Fox et al., 2016; Andreas et al., 2017; Frans et al., 2018).
Revised: Fox et al., 2016; Andreas et al., 2017; Frans et al., 2018).

Original: Ideally, the entire hierarchy can be trained end-to-end on the obtained return, obviating the need to specify proxy rewards for skills (Frans et al., 2018).
Revised: Ideally, the entire hierarchy can be trained end-to-end on the obtained return, obviating the need to specify proxy rewards for skills (Frans et al., 2018).

Original: However, learning hierarchical policies end-to-end in a multitask setting poses two major challenges: i) because skills optimize environmental rewards directly, correctly updating them relies on already (nearly) converged master policies that use them similarly across all tasks, requiring complex training schedules (Frans et al., 2018), and ii) the end-to-end optimization is prone to local minima in which multiple skills have learned similar behavior.
Revised: However, learning hierarchical policies end-to-end in a multitask setting poses two major challenges: i) because skills optimize environmental rewards directly, correctly updating them relies on already (nearly) converged master policies that use them similarly across all tasks, requiring complex training schedules (Frans et al., 2018), and ii) the end-to-end optimization is prone to local minima in which multiple skills have learned similar behavior.

Original: This second points is explained in more detail in Appendix A.
Revised: This second points is explained in more detail in Appendix A.

Original: In this paper, we propose Multitask Soft Option Learning ( MSOL ), a novel approach to learning hierarchical policies in a multi-task setting that extends Options (Sutton et al., 1999), a common deﬁnition for skills, and casts the concept into the Planning as Inference ( PAI ) framework (see, e.g.,
Revised: In this paper, we propose Multitask Soft Option Learning ( MSOL ), a novel approach to learning hierarchical policies in a multi-task setting that extends Options (Sutton et al., 1999), a common deﬁnition for skills, and casts the concept into the Planning as Inference ( PAI ) framework (see, e.g.,

Original: Levine, 2018, for a review).
Revised: Levine, 2018, for a review).

Original: MSOL brings multiple advantages: i) it stabilizes end-to-end multitask training, removing the need for complex training schedules like in Frans et al.
Revised: MSOL brings multiple advantages: i) it stabilizes end-to-end multitask training, removing the need for complex training schedules like in Frans et al.

Original: (2018), ii) it gives rise to coordination between master policies, avoiding local minima of the type described in Appendix A, iii)
Revised: (2018), ii) it gives rise to coordination between master policies, avoiding local minima of the type described in Appendix A, iii)

Original: it allows ﬁne-tuning of options, i.e. adapting them to new tasks at test-time without the risk of unlearning previously acquired useful behavior, thereby avoiding suboptimal performance due to restricted expressiveness, iv) and lastly, we show how the soft option framework gives rise to a natural solution to the challenging task of learning option-termination policies.
Revised: it allows ﬁne-tuning of options, i.e. adapting them to new tasks at test-time without the risk of unlearning previously acquired useful behavior, thereby avoiding suboptimal performance due to restricted expressiveness, iv) and lastly, we show how the soft option framework gives rise to a natural solution to the challenging task of learning option-termination policies.

Original: MSOL differentiates between a prior policy for each option, shared across all tasks, and a ﬂexible task-speciﬁc posterior policy.
Revised: MSOL differentiates between a prior policy for each option, shared across all tasks, and a ﬂexible task-speciﬁc posterior policy.

Original: The option prior can be ﬁxed once it is fully trained, preventing unlearning of useful behavior even when the posteriors are updated.
Revised: The option prior can be ﬁxed once it is fully trained, preventing unlearning of useful behavior even when the posteriors are updated.

Original: On new tasks, the option posteriors are initialized to the priors and regularized towards them, but are still adaptable to the speciﬁc task.
Revised: On new tasks, the option posteriors are initialized to the priors and regularized towards them, but are still adaptable to the speciﬁc task.

Original: This allows the same accelerated training as with ‘hard’ options, but can solve more tasks due to the adjustable posteriors.
Revised: This allows the same accelerated training as with ‘hard’ options, but can solve more tasks due to the adjustable posteriors.

Original: Furthermore, during option learning, we can train prior and posterior policies simultaneously (Teh et al., 2017), all without the need for complex training schedules (Frans et al., 2018): training is stabilized because only the priors are shared across tasks.
Revised: Furthermore, during option learning, we can train prior and posterior policies simultaneously (Teh et al., 2017), all without the need for complex training schedules (Frans et al., 2018): training is stabilized because only the priors are shared across tasks.

Original: Our experiments demonstrate that MSOL outperforms previous hierarchical and transfer learning algorithms during transfer tasks in a multitask setting.
Revised: Our experiments demonstrate that MSOL outperforms previous hierarchical and transfer learning algorithms during transfer tasks in a multitask setting.

Original: M SOL only modiﬁes the regularized reward and loss function, but does not require any specialized architecture.
Revised: M SOL only modiﬁes the regularized reward and loss function, but does not require any specialized architecture.

Original: In particular, it also does not require artiﬁcial
Revised: In particular, it also does not require artiﬁcial

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: An agent’s task is formalized as a MDP ( S , A , ρ, P, r ) , consisting of the state space S , the action space A , the initial state distribution ρ , the transition probability P ( s t +1 | s t , a t ) of reaching state s t +1 by executing action a t in state s t , and the reward r ( s t , a t ) an agent receives for this transition.
Revised: An agent’s task is formalized as a MDP ( S , A , ρ, P, r ) , consisting of the state space S , the action space A , the initial state distribution ρ , the transition probability P ( s t +1 | s t , a t ) of reaching state s t +1 by executing action a t in state s t , and the reward r ( s t , a t ) an agent receives for this transition.

Original: 2. P LANNING AS INFERENCE
Revised: 2. P LANNING AS INFERENCE

Original: Planning as inference ( PAI ) (Todorov, 2008; Toussaint, 2009; Kappen et al., 2012) frames RL as a probabilistic-inference problem (Levine, 2018).
Revised: Planning as inference ( PAI ) (Todorov, 2008; Toussaint, 2009; Kappen et al., 2012) frames RL as a probabilistic-inference problem (Levine, 2018).

Original: The agent learns a distribution q φ ( a | s ) over actions a given states s , i.e., a policy, parameterized by φ , which induces a distribution over trajectories τ of length T , i.e., τ = ( s 1 , a 1 , s 2 , . .
Revised: The agent learns a distribution q φ ( a | s ) over actions a given states s , i.e., a policy, parameterized by φ , which induces a distribution over trajectories τ of length T , i.e., τ = ( s 1 , a 1 , s 2 , . .

Original: . , a T , s T +1 ) :
Revised: . , a T , s T +1 ) :

Original: [Equation]
Revised: [Equation]

Original: This can be seen as a structured variational approximation of the optimal trajectory distribution.
Revised: This can be seen as a structured variational approximation of the optimal trajectory distribution.

Original: Note that the true initial state probability ρ ( s 1 ) and transition probability P ( s t +1 | s t , a t ) are used in the variational posterior, as we can only control the policy, not the environment.
Revised: Note that the true initial state probability ρ ( s 1 ) and transition probability P ( s t +1 | s t , a t ) are used in the variational posterior, as we can only control the policy, not the environment.

Original: An advantage of this formulation is that we can incorporate information both from prior knowledge, in the form of a prior policy distribution, and the task at hand through a likelihood function that is deﬁned in terms of the achieved reward.
Revised: An advantage of this formulation is that we can incorporate information both from prior knowledge, in the form of a prior policy distribution, and the task at hand through a likelihood function that is deﬁned in terms of the achieved reward.

Original: The prior policy p ( a t | s t ) can be speciﬁed by hand or, as in our case, learned (see Section 3).
Revised: The prior policy p ( a t | s t ) can be speciﬁed by hand or, as in our case, learned (see Section 3).

Original: To incorporate the reward, we introduce a binary optimality variable O t (Levine, 2018), whose likelihood is highest along the optimal trajectory that maximizes return: p ( O t = 1 | s t , a t ) =
Revised: To incorporate the reward, we introduce a binary optimality variable O t (Levine, 2018), whose likelihood is highest along the optimal trajectory that maximizes return: p ( O t = 1 | s t , a t ) =

Original: exp (cid:0) r
Revised: exp (cid:0) r

Original: ( s t , a t ) /β (cid:1) .
Revised: ( s t , a t ) /β (cid:1) .

Original: The constraint r ∈ ( −∞ , 0] can be relaxed without changing the inference procedure (Levine, 2018).
Revised: The constraint r ∈ ( −∞ , 0] can be relaxed without changing the inference procedure (Levine, 2018).

Original: For brevity, we denote O t = 1 as O t ≡ ( O t = 1) .
Revised: For brevity, we denote O t = 1 as O t ≡ ( O t = 1) .

Original: If a given prior policy p ( a t | s t ) explores the state-action space sufﬁciently, then p ( τ, O 1: T ) is the distribution of desirable trajectories.
Revised: If a given prior policy p ( a t | s t ) explores the state-action space sufﬁciently, then p ( τ, O 1: T ) is the distribution of desirable trajectories.

Original: PAI aims to ﬁnd a policy such that the variational posterior in (1) approximates this distribution by minimizing the Kullback-Leibler ( KL ) divergence
Revised: PAI aims to ﬁnd a policy such that the variational posterior in (1) approximates this distribution by minimizing the Kullback-Leibler ( KL ) divergence

Original: L ( φ ) =
Revised: L ( φ ) =

Original: D KL ( q φ ( τ ) (cid:107) p ( τ, O 1: T )) , with p ( τ, O 1: T ) =
Revised: D KL ( q φ ( τ ) (cid:107) p ( τ, O 1: T )) , with p ( τ, O 1: T ) =

Original: ρ ( s 1 ) T (cid:81) t =1 p ( a t | s t ) P ( s t +1 | s t , a t ) p ( O t | s t , a t ) .
Revised: ρ ( s 1 ) T (cid:81) t =1 p ( a t | s t ) P ( s t +1 | s t , a t ) p ( O t | s t , a t ) .

Original: 2. M ULTI - TASK LEARNING
Revised: 2. M ULTI - TASK LEARNING

Original: In a multi-task setting, we have a set of different tasks i ∈ T , drawn from a task distribution with probability ξ ( i ) .
Revised: In a multi-task setting, we have a set of different tasks i ∈ T , drawn from a task distribution with probability ξ ( i ) .

Original: All tasks share state space S and action space A , but each task has its own initialstate distribution
Revised: All tasks share state space S and action space A , but each task has its own initialstate distribution

Original: ρ i , transition probability P i ( s t +1 | s t , a t ) , and reward function r i .
Revised: ρ i , transition probability P i ( s t +1 | s t , a t ) , and reward function r i .

Original: Our goal is to learn n tasks concurrently, distilling common information that can be leveraged to learn faster on new tasks from T .
Revised: Our goal is to learn n tasks concurrently, distilling common information that can be leveraged to learn faster on new tasks from T .

Original: In this setting, the prior policy p θ ( a t | s t ) can be learned jointly with the taskspeciﬁc posterior policies q φ i ( a t | s t ) (Teh et al., 2017).
Revised: In this setting, the prior policy p θ ( a t | s t ) can be learned jointly with the taskspeciﬁc posterior policies q φ i ( a t | s t ) (Teh et al., 2017).

Original: To do so, we simply extend (2) to
Revised: To do so, we simply extend (2) to

Original: L ( {

Original: [Equation]

Original: , θ )

Original: [Equation]
Revised: [Equation]

Original: where R reg i,t :=
Revised: where R reg i,t :=

Original: [Equation]
Revised: [Equation]

Original: is the regularised reward.
Revised: is the regularised reward.

Original: Minimizing the loss in (3) is equivalent to maximizing the regularized reward R reg i,t .
Revised: Minimizing the loss in (3) is equivalent to maximizing the regularized reward R reg i,t .

Original: Moreover, minimizing the term E τ ∼ q (cid:2) ln q φi ( a t | s t ) p θ ( a t | s t ) (cid:3) implicitly minimizes the expected KL -divergence
Revised: Moreover, minimizing the term E τ ∼ q (cid:2) ln q φi ( a t | s t ) p θ ( a t | s t ) (cid:3) implicitly minimizes the expected KL -divergence

Original: E s t ∼ q (cid:2) D KL [ q φ i ( ·| s t ) (cid:107) p θ ( ·| s t )] (cid:3) .
Revised: E s t ∼ q (cid:2) D KL [ q φ i ( ·| s t ) (cid:107) p θ ( ·| s t )] (cid:3) .

Original: In practise (see Appendix C.1)
Revised: In practise (see Appendix C.1)

Original: we will also make use of a discount factor γ ∈ [0 , 1] .
Revised: we will also make use of a discount factor γ ∈ [0 , 1] .

Original: For details on how γ arises in the PAI framework we refer to Levine (2018).
Revised: For details on how γ arises in the PAI framework we refer to Levine (2018).

Original: 2. O PTIONS
Revised: 2. O PTIONS

Original: Options (Sutton et al., 1999) are skills that generalize primitive actions and consist of three components: i) an intra-option policy p ( a t | s t , z t ) selecting primitive actions according to the currently active option z t , ii) a probability p ( b t | s t , z t − 1 ) of terminating the previously active option z t − 1 , and iii) an initiation set I ⊆ S , which we simply assume to be S .
Revised: Options (Sutton et al., 1999) are skills that generalize primitive actions and consist of three components: i) an intra-option policy p ( a t | s t , z t ) selecting primitive actions according to the currently active option z t , ii) a probability p ( b t | s t , z t − 1 ) of terminating the previously active option z t − 1 , and iii) an initiation set I ⊆ S , which we simply assume to be S .

Original: Note that by construction, the higher- level (or master-) policy p ( z t | z t − 1 , s t , b t ) can only select a new option z t if the previous option z t − 1 has terminated.
Revised: Note that by construction, the higher- level (or master-) policy p ( z t | z t − 1 , s t , b t ) can only select a new option z t if the previous option z t − 1 has terminated.

Original:  M ETHOD
Revised:  M ETHOD

Original: s 1 t T ASKq Tφz 1 t −b 1 t q Hφz 1 t q Lφa 1 t s 2 t T ASKq Tφz 2 t −b 2 t q Hφz 2 t q Lφa 2 t p
Revised: s 1 t T ASKq Tφz 1 t −b 1 t q Hφz 1 t q Lφa 1 t s 2 t T ASKq Tφz 2 t −b 2 t q Hφz 2 t q Lφa 2 t p

Original: Tθ
Revised: Tθ

Original: P RIORS p H
Revised: P RIORS p H

Original: We aim to learn a reusable set of options that allow for faster training on new tasks from a given distribution.
Revised: We aim to learn a reusable set of options that allow for faster training on new tasks from a given distribution.

Original: Unlike prior work (Frans et al., 2018), we learn both intraoption and termination policies, while preventing multiple options from learning the same behavior.
Revised: We learn both intra-option and termination policies, while preventing multiple options from learning the same behavior.

Original: To differentiate ourselves from classical ‘hard’ options, which, once learned, do not change during new tasks, we call our novel approach soft-options (this is further discussed in Appendix B).
Revised: To differentiate ourselves from classical ‘hard’ options, which, once learned, do not change during new tasks, we call our novel approach soft-options (this is further discussed in Appendix B).

Original: Each soft-option consists of an option prior , which is shared across all tasks, and a taskspeciﬁc option posterior .
Revised: Each soft-option consists of an option prior , which is shared across all tasks, and a taskspeciﬁc option posterior .

Original: The priors of both the intraoption policy and the termination policy capture how an option typically behaves and remain ﬁxed once they are fully learned.
Revised: The priors of both the intraoption policy and the termination policy capture how an option typically behaves and remain ﬁxed once they are fully learned.

Original: At the beginning of training on a new task, they are used to initialize the task-speciﬁc posterior distribution.
Revised: At the beginning of training on a new task, they are used to initialize the task-speciﬁc posterior distribution.

Original: During training, the posterior is then regularized against the prior to prevent inadvertent unlearning.
Revised: During training, the posterior is then regularized against the prior to prevent inadvertent unlearning.

Original: However, if maximizing the reward on certain tasks is not achievable with the prior policy, the posterior is free to deviate from it.
Revised: However, if maximizing the reward on certain tasks is not achievable with the prior policy, the posterior is free to deviate from it.

Original: We can thus speed up training using options, while remaining ﬂexible enough to solve any task.
Revised: We can thus speed up training using options, while remaining ﬂexible enough to solve any task.

Original: Additionally, this soft option framework also allows for learning good priors in a multitask setting while avoiding local minima in which several options learn the same behavior.
Revised: Additionally, this soft option framework also allows for learning good priors in a multitask setting while avoiding local minima in which several options learn the same behavior.

Original: See Figure 1 for an overview over the hierarchical prior-posterior architecture that we explain further below.
Revised: See Figure 1 for an overview over the hierarchical prior-posterior architecture that we explain further below.

Original: 3. H IERARCHICAL POSTERIOR POLICIES
Revised: 3. H IERARCHICAL POSTERIOR POLICIES

Original: To express options in the PAI framework, we introduce two additional variables at each time step t : option selections z t , representing the currently selected option, and decisions b t to terminate them and allow the higher-level (master) policy to choose a new option.
Revised: To express options in the PAI framework, we introduce two additional variables at each time step t : option selections z t , representing the currently selected option, and decisions b t to terminate them and allow the higher-level (master) policy to choose a new option.

Original: The agent’s behavior depends on the currently selected option z t , by drawing actions a t from the intra-option posterior policy
Revised: The agent’s behavior depends on the currently selected option z t , by drawing actions a t from the intra-option posterior policy

Original: q Lφ i ( a t | s t , z t ) .
Revised: q Lφ i ( a t | s t , z t ) .

Original: The selection z t itself is drawn from a master policy q Hφ i
Revised: The selection z t itself is drawn from a master policy q Hφ i

Original: ( z t | s t , z t − 1 , b t ) =
Revised: ( z t | s t , z t − 1 , b t ) =

Original: (1 − b t ) δ ( z t − z t − 1 ) + b t q Hφ
Revised: (1 − b t ) δ ( z t − z t − 1 ) + b t q Hφ

Original: i ( z t | s t ) , which conditions on b t ∈ { 0 , 1 } , drawn by the termination posterior policy
Revised: i ( z t | s t ) , which conditions on b t ∈ { 0 , 1 } , drawn by the termination posterior policy

Original: q Tφ i ( b t | s t , z t − 1 ) .
Revised: q Tφ i ( b t | s t , z t − 1 ) .

Original: The master policy either continues with the previous z t − 1 or draws a new option, where we set b 1 = 1 at the beginning of each episode.
Revised: The master policy either continues with the previous z t − 1 or draws a new option, where we set b 1 = 1 at the beginning of each episode.

Revised: We slightly abuse notation by referring by δ ( z t − z t − 1 ) to the Kronecker delta δ z t ,z t − 1 for discrete and the Dirac delta distribution for continuous z t .

Original: The joint posterior policy is
Revised: The joint posterior policy is

Original: [Equation]
Revised: [Equation]

Original: While z t can be a continuous variable, we consider only z t ∈ { 1 . . .
Revised: While z t can be a continuous variable, we consider only z t ∈ { 1 . . .

Original: m } , where m is the number of available options.
Revised: m } , where m is the number of available options.

Original: The induced distribution q φ i ( τ ) over trajectories of task i , τ = ( s 1 , b 1 , z 1 , a 1 , s 2 , . .
Revised: The induced distribution q φ i ( τ ) over trajectories of task i , τ = ( s 1 , b 1 , z 1 , a 1 , s 2 , . .

Original: , s T , b T , z T , a T , s T +1 ) , is then
Revised: , s T , b T , z T , a T , s T +1 ) , is then

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 3. H IERARCHICAL PRIOR POLICY
Revised: 3. H IERARCHICAL PRIOR POLICY

Original: Our framework transfers knowledge between tasks by a shared prior p θ ( a t , z t , b t | s t , z t − 1 ) over all joint policies (4): p θ ( a t , z t , b t | s t , z t − 1 ) =
Revised: Our framework transfers knowledge between tasks by a shared prior p θ ( a t , z t , b t | s t , z t − 1 ) over all joint policies (4): p θ ( a t , z t , b t | s t , z t − 1 ) =

Original: p Tθ ( b t | s t , z t − 1 ) p H ( z t | z t − 1 , b t ) p Lθ ( a t | s t , z t ) .
Revised: p Tθ ( b t | s t , z t − 1 ) p H ( z t | z t − 1 , b t ) p Lθ ( a t | s t , z t ) .

Original: By choosing p Tθ , p H , and p Lθ correctly, we can learn useful temporally extended options.
Revised: By choosing p Tθ , p H , and p Lθ correctly, we can learn useful temporally extended options.

Original: The parameterized priors p Tθ ( b t | s t , z t − 1 ) and p Lθ ( a t | s t , z t ) are structurally equivalent to the posterior policies
Revised: The parameterized priors p Tθ ( b t | s t , z t − 1 ) and p Lθ ( a t | s t , z t ) are structurally equivalent to the posterior policies

Original: q
Revised: q

Original: Tφ i and q Lφ i so that they can be used as initialization for the latter.
Revised: Tφ i and q Lφ i so that they can be used as initialization for the latter.

Original: Optimizing the regularized return (see next section) w.r.t.
Revised: Optimizing the regularized return (see next section) w.r.t.

Original: θ distills the common behavior into the prior policy and softly enforces similarity across posterior distributions of each option amongst all tasks i .
Revised: θ distills the common behavior into the prior policy and softly enforces similarity across posterior distributions of each option amongst all tasks i .

Original: The prior p H ( z t | z t − 1 , b t ) =
Revised: The prior p H ( z t | z t − 1 , b t ) =

Original: (1 − b t ) δ ( z t − z t − 1 ) + b t 1 m selects the previous option z t − 1 if b t = 0 , and otherwise draws options uniformly to ensure exploration.
Revised: (1 − b t ) δ ( z t − z t − 1 ) + b t 1 m selects the previous option z t − 1 if b t = 0 , and otherwise draws options uniformly to ensure exploration.

Original: Because the posterior master policy is different on each task, there is no need to distill common behavior into a joint prior.
Revised: Because the posterior master policy is different on each task, there is no need to distill common behavior into a joint prior.

Original: 3. O BJECTIVE
Revised: 3. O BJECTIVE

Original: We extend the multitask objective in (3) by substituting p θ ( τ, O 1: T ) and p φ i ( τ ) with those induced by our hierarchical posterior policy in (4) and the corresponding prior.
Revised: We extend the multitask objective in (3) by substituting p θ ( τ, O 1: T ) and p φ i ( τ ) with those induced by our hierarchical posterior policy in (4) and the corresponding prior.

Original: The resulting objective has the same form but with a new regularized reward that is maximized:
Revised: The resulting objective has the same form but with a new regularized reward that is maximized:

Original: [Equation]
Revised: [Equation]

Original: Lθ ( a t | s t ,z t ) (cid:124) (cid:123)(cid:122) (cid:125)− β ln q Tφi ( b t | s t ,z t − 1 )

Original: p Tθ ( b t | s t ,z t − 1 ) (cid:124) (cid:123)(cid:122) (cid:125).

Original: As we maximize E q [ R reg i,t ] , this corresponds to maximizing the expectation over r
Revised: As we maximize E q [ R reg i,t ] , this corresponds to maximizing the expectation over r

Original: i ( s t ,a t ) − β (cid:2) D KL ( q Hφ i (cid:107) p H )
Revised: i ( s t ,a t ) − β (cid:2) D KL ( q Hφ i (cid:107) p H )

Original: + D KL ( q Lφ i (cid:107) p Lθ )
Revised: + D KL ( q Lφ i (cid:107) p Lθ )

Original: + D KL ( q Tφ i (cid:107)
Revised: + D KL ( q Tφ i (cid:107)

Original: p Tθ ) (cid:3) , along the on-policy trajectories drawn from q φ
Revised: p Tθ ) (cid:3) , along the on-policy trajectories drawn from q φ

Original: i ( τ ) .
Revised: i ( τ ) .

Original: Term 1 of the regularization encourages exploration in the space of options.
Revised: Term 1 of the regularization encourages exploration in the space of options.

Original: It can also be seen as a form of deliberation cost (Harb et al., 2017) as it is only nonzero whenever we terminate an option and the master policy needs to select another to execute.
Revised: It can also be seen as a form of deliberation cost (Harb et al., 2017) as it is only nonzero whenever we terminate an option and the master policy needs to select another to execute.

Original: Term 2 softly enforces similarity between option posteriors across tasks and updates the prior towards the ‘average’ posterior.
Revised: Term 2 softly enforces similarity between option posteriors across tasks and updates the prior towards the ‘average’ posterior.

Original: It also encourages the master policy to pick the most specialized option whose posteriors across all tasks are most similar.
Revised: It also encourages the master policy to pick the most specialized option whose posteriors across all tasks are most similar.

Original: In other words, the master policy q Hφ i is encouraged to pick option z t which maximizes r i , but minimizes term 2 by picking the option z t for which prior p L θ and posterior q L φ i are the most similar.
Revised: In other words, the master policy q H φ i is encouraged to pick option z t which maximizes r i , but minimizes term 2 by picking the option z t for which prior p Lθ and posterior q Lφ i are the most similar.

Original: Because the prior is the average posterior, this rewards the master policy to pick the most specialized option (that still achieves high reward).
Revised: Because the prior is the average posterior, this rewards the master policy to pick the most specialized option (that still achieves high reward).

Original: As discussed in more detail in Appendix A, this allows us to escape the local optimization minima that hard options face in multitask learning, while still having fully specialized options after training.
Revised: As discussed in more detail in Appendix A, this allows us to escape the local optimization minima that hard options face in multitask learning, while still having fully specialized options after training.

Original: Lastly, we can use 3 to also encourage temporal abstraction of options.
Revised: Lastly, we can use 3 to also encourage temporal abstraction of options.

Original: To do so, during option learning , we ﬁx the termination prior p T to a Bernoulli distribution p T ( b ) =
Revised: To do so, during option learning , we ﬁx the termination prior p T to a Bernoulli distribution p T ( b ) =

Original: (1 − α )
Revised: (1 − α )

Original: b α 1 − b .
Revised: b α 1 − b .

Original: Choosing a large α encourages prolonged execution of one option, but allows switching whenever necessary.
Revised: Choosing a large α encourages prolonged execution of one option, but allows switching whenever necessary.

Original: This is also similar to deliberation costs (Harb et al., 2017) but with a more ﬂexible cost model.
Revised: This is also similar to deliberation costs (Harb et al., 2017) but with a more ﬂexible cost model.

Original: Additionally, we can still distill a termination prior p Tθ which can be used on future tasks.
Revised: Additionally, we can still distill a termination prior p Tθ which can be used on future tasks.

Original: Instead of learning p Tθ by minimizing the KL against the posterior termination policies, we can get more decisive terminations by minimizing
Revised: Instead of learning p Tθ by minimizing the KL against the posterior termination policies, we can get more decisive terminations by minimizing

Original: [Equation]
Revised: [Equation]

Original:  and ˆ q φ i ( b = 1 | s t , z t − 1 ) = (cid:80) 
Revised: (7) and ˆ q φ i ( b = 1 | s t , z t − 1 ) = (cid:80) z t (cid:54) = z

Original: [Equation]
Revised: [Equation]

Original: (cid:54) =

Original: z t
Revised: −

Original: [Equation]
Revised: [Equation]

Original: q Hφ i ( z t | s t , z t − 1 , b t = 1) i.e., the learned termination prior distills the probability that the tasks’ master policies would change the active option if they had the opportunity.
Revised: q Hφ i ( z t | s t , z t − 1 , b t = 1) i.e., the learned termination prior distills the probability that the tasks’ master policies would change the active option if they had the opportunity.

Original: Details on how we optimiszed the MSOL objective are given in Appendix C.
Revised: Details on how we optimized the MSOL objective are given in Appendix C.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Most hierarchical approaches rely on proxy rewards to train the lower level components.
Revised: Most hierarchical approaches rely on proxy rewards to train the lower level components and their terminations.

Original: Some of them aim to reach pre-speciﬁed subgoals (Sutton et al., 1999; Kulkarni et al., 2016), which are often found by analyzing the structure of the MDP (McGovern & Barto, 2001;Menache et al., 2002;
Revised: Some of them aim to reach pre-speciﬁed subgoals (Sutton et al., 1999; Kulkarni et al., 2016), which are often found by analyzing the structure of the MDP (McGovern & Barto, 2001;

Original:  S¸ims¸ek et al., 2005; S¸ims¸ek & Barto, 2009) or previously learned policies (Goel & Huber, 2003).
Revised: Menache et al., 2002; S¸ims¸ek et al., 2005; S¸ims¸ek & Barto, 2009), previously learned policies (Goel & Huber, 2003; Tessler et al., 2017) or predictability (Harutyunyan et al., 2019).

Original: Those methods typically require knowledge, or a sufﬁcient approximation, of the transition model, both of which are often infeasible.
Revised: Those methods typically require knowledge, or a sufﬁcient approximation, of the transition model, both of which are often infeasible.

Original: Recently, several authors have proposed unsupervised training objectives for learning diverse skills based on their distinctiveness (Gregor et al., 2016;
Revised: Recently, several authors have proposed unsupervised training objectives for learning diverse skills based on their distinctiveness (Gregor et al., 2016;

Original: Florensa et al., 2017; Achiam et al., 2018;
Revised: Florensa et al., 2017; Achiam et al., 2018;

Original: Eysenbach et al., 2019).
Revised: Eysenbach et al., 2019).

Original: However, those approaches don’t learn termination functions and cannot guarantee that the required behavior on the downstream task is included in the set of learned skills.
Revised: However, those approaches don’t learn termination functions and cannot guarantee that the required behavior on the downstream task is included in the set of learned skills.

Original: Hausman et al.
Revised: Hausman et al.

Original: (2018) also incorporate reward information, but do not learn termination policies and are therefore restricted to learning multiple solutions to the provided task instead of learning a decomposition of the task solutions which can be re-composed to solve new tasks.
Revised: (2018) also incorporate reward information, but do not learn termination policies and are therefore restricted to learning multiple solutions to the provided task instead of learning a decomposition of the task solutions which can be re-composed to solve new tasks.

Original: A third usage of proxy rewards is by training lower level policies to move towards goals deﬁned by the higher levels.
Revised: A third usage of proxy rewards is by training lower level policies to move towards goals deﬁned by the higher levels.

Original: When those goals are set in the original state space (Nachum et al., 2018a), this approach has difﬁculty scaling to high dimensional state spaces like images.
Revised: When those goals are set in the original state space (Nachum et al., 2018a), this approach has difﬁculty scaling to high dimensional state spaces like images.

Original: Setting the goals in a learned embedding space (Dayan & Hinton, 1993; Vezhnevets et al., 2017; Nachum et al., 2018b) can be difﬁcult to train, though.
Revised: Setting the goals in a learned embedding space (Dayan & Hinton, 1993; Vezhnevets et al., 2017; Nachum et al., 2018b) can be difﬁcult to train, though.

Original: In both cases, the temporal extension of the learned skills are set manually.
Revised: In both cases, the temporal extension of the learned skills are set manually.

Original: On the other hand, Goyal et al.
Revised: On the other hand, Goyal et al.

Original: (2019) also learn a hierarchical agent, but not to transfer skills, but to ﬁnd decisions states based on how much information is encoded in the latent layer.
Revised: (2019) also learn a hierarchical agent, but not to transfer skills, but to ﬁnd decisions states based on how much information is encoded in the latent layer.

Original: HiREPS Daniel et al.
Revised: HiREPS Daniel et al.

Original: (2012) also take an inference motivated approach to learning options.
Revised: (2012) also take an inference motivated approach to learning options.

Original: In particular Daniel et al.
Revised: In particular Daniel et al.

Original: (2016) propose a similarly structured hierarchical policy, albeit in a single task setting.
Revised: (2016) propose a similarly structured hierarchical policy, albeit in a single task setting.

Original: However, they do not utilize learned prior and posterior distributions, but instead use expectation maximization to iteratively infer a hierarchical policy to explain the current rewardweighted trajectory distribution.
Revised: However, they do not utilize learned prior and posterior distributions, but instead use expectation maximization to iteratively infer a hierarchical policy to explain the current rewardweighted trajectory distribution.

Original: Several previous works try to overcome the restrictive nature of options that can lead to sub-optimal solutions by allowing the higher-level actions to modulate the behavior of the lower-level policies
Revised: Several previous works try to overcome the restrictive nature of options that can lead to sub-optimal solutions by allowing the higher-level actions to modulate the behavior of the lower-level policies

Original: Schaul et al. (2015); Heess et al.
Revised: Schaul et al. (2015); Heess et al.

Original: (2016); Haarnoja et al.
Revised: (2016); Haarnoja et al.

Original: However, this signiﬁcantly increases the required complexity of the higher-level policy and therefore the learning time.
Revised: However, this signiﬁcantly increases the required complexity of the higher-level policy and therefore the learning time.

Original: The multitask- and transfer-learning setup used in this work is inspired by Thrun & Schwartz (1995) and Pickett & Barto (2002) who suggest extracting options by using commonalities between solutions to multiple tasks.
Revised: The multitask- and transfer-learning setup used in this work is inspired by Thrun & Schwartz (1995) and Pickett & Barto (2002) who suggest extracting options by using commonalities between solutions to multiple tasks.

Original: Andreas et al. (2017) extend this to deep policies but require additional human supervision in the form of policy sketches.
Revised: Prior multitask approaches often rely on additional human supervision like policy sketches (Andreas et al., 2017) or desirable sub-goals (Tessler et al., 2017; Konidaris & Barto, 2007; Mann et al., 2015) in order to learn skills which transfer well between tasks.

Original: In contrast to our work, which focusses on learning transferrable option priors, Tirumala et al.
Revised: In contrast, our work aims at ﬁnding good termination states without such supervision. Tirumala et al.

Original: (2019) investigate the use of different priors for the higher-level policy.
Revised: (2019) investigate the use of different priors for the higher-level policy while we are focussing on learning transferrable option priors.

Original: Closest to our work is Meta Learning of Shared Hierarchies ( MLSH ) (Frans et al., 2018) which, however, shares the lower-level policies across all tasks without distinguishing between prior and posterior.
Revised: Closest to our work is Meta Learning of Shared Hierarchies ( MLSH ) (Frans et al., 2018) which, however, shares the lower-level policies across all tasks without distinguishing between prior and posterior and does not learn termination policies.

Original: As discussed, this leads to local minima and insufﬁcient diversity in the learned options.
Revised: As discussed, this leads to local minima and insufﬁcient diversity in the learned options.

Original: Similarly to us, Fox et al.
Revised: Similarly to us, Fox et al.

Original: (2016) differentiate between prior and posterior policies on multiple tasks and utilize a KL-divergence between them for training.
Revised: (2016) differentiate between prior and posterior policies on multiple tasks and utilize a KLdivergence between them for training.

Original: However, they do not consider termination probabilities and instead only choose one option per task.
Revised: However, they do not consider termination probabilities and instead only choose one option per task.

Revised: Instead of transferring option policies between tasks, Ammar et al.

Revised: (2014) aim to share behavior through a latent embedding.

Revised: Another interesting approach to multitask learning is (Mankowitz et al., 2016) which learns decision regions that are linear in the state instead of learning nonlinear master- and termination policies.

Original: Our approach is closely related to DISTRAL (Teh et al., 2017) with which we share the multitask learning of prior and posterior policies.
Revised: Our approach is closely related to DISTRAL (Teh et al., 2017) with which we share the multitask learning of prior and posterior policies.

Original: However, DISTRAL has no hierarchical structure and applies the same prior distribution over primitive actions, independent of the task.
Revised: However, DISTRAL has no hierarchical structure and applies the same prior distribution over primitive actions, independent of the task.

Original: As a necessary hierarchical heuristic, the authors propose to also condition on the last primitive action taken.
Revised: As a necessary hierarchical heuristic, the authors propose to also condition on the last primitive action taken.

Original: This works well when the last action is indicative of future behavior; however, in Section 5 we show several failure cases where a learned hierarchy is needed.
Revised: This works well when the last action is indicative of future behavior; however, in Section 5 we show several failure cases where a learned hierarchy is needed.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conduct a series of experiments to show: i) when learning hierarchies in a multitask setting,
Revised: We conduct a series of experiments to show: i) when learning hierarchies in a multitask setting,

Original: MSOL successfully overcomes the local minimum of insufﬁcient option diversity, as described in
Revised: MSOL successfully overcomes the local minimum of insufﬁcient option diversity, as described in

Original: Appendix A; ii) MSOL can learn useful termination policies; iii) MSOL is equally applicable to discrete as well as continuous domains; and iv) using soft options yields fast transfer learning while still reaching optimal performance, even on new, out-of-distribution tasks.
Revised: Appendix A; ii) MSOL can learn useful termination policies; iii) MSOL is equally applicable to discrete as well as continuous domains; and iv) using soft options yields fast transfer learning while still reaching optimal performance, even on new, out-of-distribution tasks.

Original: All architectural details and hyper-parameters can be found in the appendix.
Revised: All architectural details and hyper-parameters can be found in the appendix.

Original: For all experiments, we ﬁrst train the exploration priors and options on n tasks from the available task distribution T (training phase is plotted in Appendix E).
Revised: For all experiments, we ﬁrst train the exploration priors and options on n tasks from the available task distribution T (training phase is plotted in Appendix E).

Original: Subsequently, we test how quickly we can learn new tasks from T (or another distribution T (cid:48) ).
Revised: Subsequently, we test how quickly we can learn new tasks from T (or another distribution T (cid:48) ).

Original: We compare the following algorithms: MSOL is our proposed method that utilizes soft options both during option learning and transfer.
Revised: We compare the following algorithms: MSOL is our proposed method that utilizes soft options both during option learning and transfer.

Original: MSOL (frozen) uses the soft options framework during learning to ﬁnd more diverse skills, but does not allow ﬁne-tuning the posterior sub-policies after transfer.
Revised: MSOL (frozen) uses the soft options framework during learning to ﬁnd more diverse skills, but does not allow ﬁne-tuning the posterior sub-policies after transfer.

Original: DISTRAL (Teh et al., 2017) is a strong non-hierarchical transfer learning algorithm that also utilizes prior and posterior distributions.
Revised: DISTRAL (Teh et al., 2017) is a strong non-hierarchical transfer learning algorithm that also utilizes prior and posterior distributions.

Original: DISTRAL (+action) utilizes the last action as option-heuristic which works well in some tasks but fails when the last action is not sufﬁciently informative.
Revised: DISTRAL (+action) utilizes the last action as option-heuristic which works well in some tasks but fails when the last action is not sufﬁciently informative.

Original: MLSH (Frans et al., 2018) is a multitask option learning algorithm like MSOL , but utilizes ‘hard’ options for both learning and transfer, i.e., sub-policies that are shared exactly across tasks.
Revised: MLSH (Frans et al., 2018) is a multitask option learning algorithm like MSOL , but utilizes ‘hard’ options for both learning and transfer, i.e., sub-policies that are shared exactly across tasks.

Original: It also relies on ﬁxed option durations and requires a complex training schedule between master and intra-option policies to stabilize training.
Revised: It also relies on ﬁxed option durations and requires a complex training schedule between master and intra-option policies to stabilize training.

Original: We use the author’s MLSH implementation.
Revised: We use the author’s MLSH implementation.

Original: Lastly, we compare against Option
Revised: Lastly, we compare against Option

Original: Critic ( OC ) (Bacon et al., 2017), which takes the task-id as additional input in order to apply it to a multitask setting.
Revised: Critic ( OC ) (Bacon et al., 2017), which takes the task-id as additional input in order to apply it to a multitask setting.

Original: 5. M OVING B ANDITS
Revised: 5. M OVING B ANDITS

Original: We start with the 2D Moving Bandits environment proposed and implemented by Frans et al.
Revised: We start with the 2D Moving Bandits environment proposed and implemented by Frans et al.

Original: (2018), which is similar to the example in Appendix A.
Revised: (2018), which is similar to the example in Appendix A.

Original: In each episode, the agent receives a reward offor each time step it is sufﬁciently close to one of two randomly sampled, distinguishable, marked positions in the environment.
Revised: In each episode, the agent receives a reward offor each time step it is sufﬁciently close to one of two randomly sampled, distinguishable, marked positions in the environment.

Original: The agent can take actions that move it in one of the four cardinal directions.
Revised: The agent can take actions that move it in one of the four cardinal directions.

Original: Which position is not signaled in the observation.
Revised: Which position is not signaled in the observation.

Original: Each episode lasts 50 time steps.
Revised: Each episode lasts 50 time steps.

Original: We compare against MLSH and DISTRAL to highlight challenges that arise in multitask training.
Revised: We compare against MLSH and DISTRAL to highlight challenges that arise in multitask training.

Original: We allow MLSH and MSOL to learn two options.
Revised: We allow MLSH and MSOL to learn two options.

Original: During transfer, optimal performance can only be achieved when both options successfully learned to reach different marked locations, i.e., when they are diverse.
Revised: During transfer, optimal performance can only be achieved when both options successfully learned to reach different marked locations, i.e., when they are diverse.

Original: In Figure 2(a) we can see that MSOL is able to do so but the options learned by
Revised: In Figure 2(a) we can see that MSOL is able to do so but the options learned by

Original: MLSH are not sufﬁciently diverse, for the reason explain in Appendix A. DISTRAL , even with the last action provided as additional input, is not able to quickly utilize the prior knowledge.
Revised: MLSH are not sufﬁciently diverse, for the reason explain in Appendix A. DISTRAL , even with the last action provided as additional input, is not able to quickly utilize the prior knowledge.

Original: The last action only conveys meaningful information when taking the goal locations into account:
Revised: The last action only conveys meaningful information when taking the goal locations into account:

Original: DISTRAL agents need to infer the intention based on the last action and the relative goal positions.
Revised: DISTRAL agents need to infer the intention based on the last action and the relative goal positions.

Original: While this is possible, in practice the agent was not able to do so, even with a much larger network.
Revised: While this is possible, in practice the agent was not able to do so, even with a much larger network.

Original: However, longer training allows DISTRAL to perform as well as MSOL , since its posterior is ﬂexible, denoted  by “ DISTRAL (+action) limit”.
Revised: However, longer training allows DISTRAL to perform as well as MSOL , since its posterior is ﬂexible, denoted [Figure] by “ DISTRAL (+action) limit”.

Original: Lastly, MSOL (frozen) also outperforms DISTRAL (+action) and MLSH , but performs worse that MSOL .
Revised: Lastly, MSOL (frozen) also outperforms DISTRAL (+action) and MLSH , but performs worse that MSOL .

Original: This highlights the utility of making options soft, i.e. adaptable.
Revised: This highlights the utility of making options soft, i.e. adaptable.

Original: 5. T AXI
Revised: 5. T AXI

Original: Next, we use a slightly modiﬁed version of the original Taxi domain (Dietterich, 1998) to show learning of termination functions as well as transfer- and generalization capabilities.
Revised: Next, we use a slightly modiﬁed version of the original Taxi domain (Dietterich, 1998) to show learning of termination functions as well as transfer- and generalization capabilities.

Original: To solve the task, the agent must pick up a passenger on one of four possible locations by moving to their location[Figure] and executing a special ‘pickup/drop-off’ action.
Revised: To solve the task, the agent must pick up a passenger on one of four possible locations by moving to their location and executing a special ‘pickup/drop-off’ action.

Original: Then, the passenger must be dropped off at one of the other three locations, again using the same action executed at the corresponding location.
Revised: Then, the passenger must be dropped off at one of the other three locations, again using the same action executed at the corresponding location.

Original: The domain has a discrete state space with 30 locations arranged on a grid and a ﬂag indicating whether the passenger was already picked up.
Revised: The domain has a discrete state space with 30 locations arranged on a grid and a ﬂag indicating whether the passenger was already picked up.

Original: The observation is a one-hot encoding of the discrete state, excluding passenger- and goal location.
Revised: The observation is a one-hot encoding of the discrete state, excluding passenger- and goal location.

Original: This introduces an information-asymmetry between the task-speciﬁc master policy, and the shared options, allowing them to generalize well (Galashov et al., 2018).
Revised: This introduces an information-asymmetry between the task-speciﬁc master policy, and the shared options, allowing them to generalize well (Galashov et al., 2018).

Original: Walls (see Figure 3) limit the movement of the agent and invalid actions.
Revised: Walls (see Figure 3) limit the movement of the agent and invalid actions.

Original: We investigate two versions of Taxi.
Revised: We investigate two versions of Taxi.

Original: In the original (Dietterich, 1998, just called Taxi ), the action space consists of one no-op, one ‘pickup/drop-off’ action and four actions to move in all cardinal directions.
Revised: In the original (Dietterich, 1998, just called Taxi ), the action space consists of one no-op, one ‘pickup/drop-off’ action and four actions to move in all cardinal directions.

Original: In Directional Taxi , we extend this setup: the agent faces in one of the cardinal directions and the available movements are to move forward or rotate either clockwise or counter-clockwise.
Revised: In Directional Taxi , we extend this setup: the agent faces in one of the cardinal directions and the available movements are to move forward or rotate either clockwise or counter-clockwise.

Original: In both environments the set of tasks T are the 12 different combinations of pickup/drop-off locations.
Revised: In both environments the set of tasks T are the 12 different combinations of pickup/drop-off locations.

Original: Episodes last at most 50 steps and there is a reward of 2 for delivering the passenger to its goal and a penalty of -0.1 for each time step.
Revised: Episodes last at most 50 steps and there is a reward of 2 for delivering the passenger to its goal and a penalty of -0.1 for each time step.

Original: During training, the agent is initialized to any valid state.
Revised: During training, the agent is initialized to any valid state.

Original: During testing, the agent is always initialized without the passenger on board.
Revised: During testing, the agent is always initialized without the passenger on board.

Original: We allow four learnable options in MLSH and MSOL .
Revised: We allow four learnable options in MLSH and MSOL .

Original: This necessitates the options to be diverse, i.e., one option to reach each of the four pickup/drop-off locations.
Revised: This necessitates the options to be diverse, i.e., one option to reach each of the four pickup/drop-off locations.

Original: Importantly, it also requires the options to learn to terminate when a passenger is picked up.
Revised: Importantly, it also requires the options to learn to terminate when a passenger is picked up.

Original: As one can see in Figure 2(b), MLSH struggles both with option-diversity and due to its ﬁxed option duration which is not ﬂexible enough for this environment.
Revised: As one can see in Figure 2(b), MLSH struggles both with option-diversity and due to its ﬁxed option duration which is not ﬂexible enough for this environment.

Original: DISTRAL (+action) performs well in the original Taxi environment, as seen in Figure 2(b), since here the last action is a good indicator for the agent’s intention.
Revised: DISTRAL (+action) performs well in the original Taxi environment, as seen in Figure 2(b), since here the last action is a good indicator for the agent’s intention.

Original: However, in the directional case shown in Figure 2(c), the actions are less informative and make it much harder for DISTRAL to use prior knowledge.
Revised: However, in the directional case shown in Figure 2(c), the actions are less informative and make it much harder for DISTRAL to use prior knowledge.

Original: By contrast, MSOL performs well in both taxi environments.
Revised: By contrast, MSOL performs well in both taxi environments.

Original: Comparing its performance with MSOL (frozen) shows the utility of adaptable soft options during transfer.
Revised: Comparing its performance with MSOL (frozen) shows the utility of adaptable soft options during transfer.

Original: Figure 3, which visualizes the options learned by MSOL , shows that it successfully learns useful movement primitives and termination functions.
Revised: Figure 3, which visualizes the options learned by MSOL , shows that it successfully learns useful movement primitives and termination functions.

Original: The same soft option represents different behavior depending on whether it already picked up the passenger.
Revised: The same soft option represents different behavior depending on whether it already picked up the passenger.

Original: This is expected as this behavior does not need to terminate the current option on three of the 12 tasks.
Revised: This is expected as this behavior does not need to terminate the current option on three of the 12 tasks.

Original: Next we show how learning information-asymmetric soft options can help with transfer to unseen tasks.
Revised: Next we show how learning information-asymmetric soft options can help with transfer to unseen tasks.

Original: In Figure 2(d) we show learning on four tasks from T using options that were trained on the remaining eight, comparing against A2C and OC .
Revised: In Figure 2(d) we show learning on four tasks from T using options that were trained on the remaining eight, comparing against A2C and OC .

Original: Note that in OC , there is no informationasymmetry: We share the same networks across all tasks and provide the task-id as additional input, including to the option-policies.
Revised: Note that in OC , there is no informationasymmetry: We share the same networks across all tasks and provide the task-id as additional input, including to the option-policies.

Original: This prevents them from generalizing well to unseen tasks.
Revised: This prevents them from generalizing well to unseen tasks.

Original: On the other hand, withholding the task-information from them would be similar to MLSH , which we already showed to struggle with local minima.
Revised: On the other hand, withholding the task-information from them would be similar to MLSH , which we already showed to struggle with local minima.

Original: The strong performance of MSOL on this task shows that we need soft options to be able to train information-asymmetric options that generalize well.
Revised: The strong performance of MSOL on this task shows that we need soft options to be able to train information-asymmetric options that generalize well.

Original: We also investigate the utility of ﬂexible soft options: In Figure 2(e) we show learning performance on twelve changed tasks in which the pickup/dropoff locations where moved by one cell while the options were trained with the original locations.
Revised: We also investigate the utility of ﬂexible soft options: In Figure 2(e) we show learning performance on twelve changed tasks in which the pickup/dropoff locations where moved by one cell while the options were trained with the original locations.

Original: As expected, hard options are not able to solve this tasks.
Revised: As expected, hard options are not able to solve this tasks.

Original: Even with additional access to primitive actions, exploration is inefﬁcient (Jong et al., 2008).
Revised: Even with additional access to primitive actions, exploration is inefﬁcient (Jong et al., 2008).

Original: On the other hand, MSOL is able to quickly learn this new task by adapting the previously learned options, outperforming hard options and ﬂat policies learned from scratch.
Revised: On the other hand, MSOL is able to quickly learn this new task by adapting the previously learned options, outperforming hard options and ﬂat policies learned from scratch.

Original: 5. S WIMMER
Revised: 5. S WIMMER

Original: Lastly, we show that MSOL can also be applied to continuous multitask domains.
Revised: Lastly, we show that MSOL can also be applied to continuous multitask domains.

Original: In particular, we investigate the MuJoCo environment ‘Swimmer’ (Todorov et al., 2012; Brockman et al., 2016).
Revised: In particular, we investigate the MuJoCo environment ‘Swimmer’ (Todorov et al., 2012; Brockman et al., 2016).

Original: Instead of rewarding forward movement as in the original implementation, now the rewarded movement direction depends on the task from T
Revised: Instead of rewarding forward movement as in the original implementation, now the rewarded movement direction depends on the task from T

Original: = { up, down, left, right } .
Revised: = { up, down, left, right } .

Original: We also include a small amount of additive action noise (details in the Appendix).
Revised: We also include a small amount of additive action noise (details in the Appendix).

Original: We show that MSOL performs competitive even in the absence of known failure cases of DISTRAL (see Figure 2(f)).
Revised: We show that MSOL performs competitive even in the absence of known failure cases of DISTRAL (see Figure 2(f)).

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Multitask Soft Option Learning ( MSOL ) proposes reformulating options using the perspective of prior and posterior distributions.
Revised: Multitask Soft Option Learning ( MSOL ) proposes reformulating options using the perspective of prior and posterior distributions.

Original: This offers several key advantages.
Revised: This offers several key advantages.

Original: First, during transfer, it allows us to distinguish between ﬁxed, and therefore knowledge-preserving option priors , and ﬂexible option posteriors that can adjust to the reward structure of the task at hand.
Revised: First, during transfer, it allows us to distinguish between ﬁxed, and therefore knowledge-preserving option priors , and ﬂexible option posteriors that can adjust to the reward structure of the task at hand.

Original: This effects a similar speed-up in learning as the original options framework, while avoiding sub-optimal performance when the available options are not perfectly aligned to the task.
Revised: This effects a similar speed-up in learning as the original options framework, while avoiding sub-optimal performance when the available options are not perfectly aligned to the task.

Original: Second, utilizing this ‘soft’ version of options in a multitask learning setup increases optimization stability and removes the need for complex training schedules between master and lower level policies.
Revised: Second, utilizing this ‘soft’ version of options in a multitask learning setup increases optimization stability and removes the need for complex training schedules between master and lower level policies.

Original: Furthermore, this framework naturally allows master policies to coordinate across tasks and avoid local minima of insufﬁcient option diversity.
Revised: Furthermore, this framework naturally allows master policies to coordinate across tasks and avoid local minima of insufﬁcient option diversity.

Original: It also allows for autonomously learning option-termination policies, a very challenging task which is often avoided by ﬁxing option durations manually.
Revised: It also allows for autonomously learning option-termination policies, a very challenging task which is often avoided by ﬁxing option durations manually.

Original: Lastly, using this formulation also allows inclusion of prior information in a principled manner without imposing too rigid a structure on the resulting hierarchy.
Revised: Lastly, using this formulation also allows inclusion of prior information in a principled manner without imposing too rigid a structure on the resulting hierarchy.

Original: We utilize this advantage to explicitly incorporate the bias that good options should be temporally extended.
Revised: We utilize this advantage to explicitly incorporate the bias that good options should be temporally extended.

Original: In future research, other types of information can be explored.
Revised: In future research, other types of information can be explored.

Original: As an example, one could investigate sets of tasks which would beneﬁt from a learned master prior, like walking on different types of terrain.
Revised: As an example, one could investigate sets of tasks which would beneﬁt from a learned master prior, like walking on different types of terrain.

Original: Appendix A L OCAL M INIMA IN M ULTITASK O PTION L EARNING
Revised: Appendix A L OCAL M INIMA IN M ULTITASK O PTION L EARNING

Original: Here our aim is to provide an intuitive explanation of why learning hard options in a multitask setting can lead to local minima.
Revised: Here our aim is to provide an intuitive explanation of why learning hard options in a multitask setting can lead to local minima.

Original: In this local minimum, multiple options have learned the same behavior and are unable to change it, even if doing so would ultimately lead to a higher reward.
Revised: In this local minimum, multiple options have learned the same behavior and are unable to change it, even if doing so would ultimately lead to a higher reward.

Original: We use the
Revised: We use the

Original: Moving Bandits experiment schematically depicted in Figure 4 as example: The agent (black dot) observes two target locations A and B but does not know which one is the correct one that has to be reached in order to generate a reward.
Revised: Moving Bandits experiment schematically depicted in Figure 4 as example: The agent (black dot) observes two target locations A and B but does not know which one is the correct one that has to be reached in order to generate a reward.

Original: The state- and action-spaces are continuous, requiring multiple actions to reach either A or B from the starting position.
Revised: The state- and action-spaces are continuous, requiring multiple actions to reach either A or B from the starting position.

Original: Consequently, having access to two options, one for each location, can accelerate learning.
Revised: Consequently, having access to two options, one for each location, can accelerate learning.

Original: We denote the options by z 1 and z 2 .
Revised: We denote the options by z 1 and z 2 .

Original: Assume that due to random initialization or later discovery of target B , both skills currently reach
Revised: Assume that due to random initialization or later discovery of target B , both skills currently reach

Original: A .
Revised: A .

Original: Furthermore, assume that we are training on multiple task simultaneously as often the case in RL .
Revised: Furthermore, assume that we are training on multiple task simultaneously as often the case in RL .

Original: In this situation, the master policies on tasks in which the correct goal is A are indifferent between using z 1 and z 2 and will consequently use both with high probability.
Revised: In this situation, the master policies on tasks in which the correct goal is A are indifferent between using z 1 and z 2 and will consequently use both with high probability.

Original: Now, changing one skill, e.g. z 2 , towards B in order to solve tasks in which B is the correct target, decreases the performance on all tasks that currently use z 2 to reach target A because for hard options the skills are shared exactly across tasks.
Revised: Now, changing one skill, e.g. z 2 , towards B in order to solve tasks in which B is the correct target, decreases the performance on all tasks that currently use z 2 to reach target A because for hard options the skills are shared exactly across tasks.

Original: Due to small, stochastic gradient updates changing z 2 towards B only slightly increases the probability of reaching B but signiﬁcantly decreases the probability of reaching A (because now the option
Revised: Due to small, stochastic gradient updates changing z 2 towards B only slightly increases the probability of reaching B but signiﬁcantly decreases the probability of reaching A (because now the option

Original: z 2 will at ﬁrst point to neither A or B ).
Revised: z 2 will at ﬁrst point to neither A or B ).

Original: Averaged across all tasks, this would at ﬁrst decrease the average reward, consequently preventing any option from changing away from A , leaving B unreachable.
Revised: Averaged across all tasks, this would at ﬁrst decrease the average reward, consequently preventing any option from changing away from A , leaving B unreachable.

Original: To “free up” z 2 (or any other option) and learn a new skill, all master policies would need to coordinate and refrain from using it to reach A and instead use the equally useful skill z 1 exclusively.
Revised: To “free up” z 2 (or any other option) and learn a new skill, all master policies would need to coordinate and refrain from using it to reach A and instead use the equally useful skill z 1 exclusively.

Original: Importantly, using soft options makes this possible.
Revised: Importantly, using soft options makes this possible.

Original: In Figures 4(b) and 4(c) we depict this schematically: The key difference is that in MSOL we have separate task-speciﬁc posteriors
Revised: In Figures 4(b) and 4(c) we depict this schematically: The key difference is that in MSOL we have separate task-speciﬁc posteriors

Original: z
Revised: z

Original: ( a ) i and z ( b ) i for tasks
Revised: ( a ) i and z ( b ) i for tasks

Original: a and b and soft options i ∈ { 1 , 2 } (for simplicity, we assume that the correct target is A for task a and B for task b ).
Revised: a and b and soft options i ∈ { 1 , 2 } (for simplicity, we assume that the correct target is A for task a and B for task b ).

Original: This allows us, in a ﬁrst step, to solve all tasks (Figure 4(b)).
Revised: This allows us, in a ﬁrst step, to solve all tasks (Figure 4(b)).

Original: Later, as discussed in Section 3.3, maximizing the regularized reward (6) will lead to specialized skills, i.e. option priors (Figure 4(c)).
Revised: Later, as discussed in Section 3.3, maximizing the regularized reward (6) will lead to specialized skills, i.e. option priors (Figure 4(c)).

Original: The corresponding shared priors ¯ z i (one per option) are updated to effectively yield the ’average’ (as measured by the KL-divergence) between the task-speciﬁc posteriors
Revised: The corresponding shared priors ¯ z i (one per option) are updated to effectively yield the ’average’ (as measured by the KL-divergence) between the task-speciﬁc posteriors

Original: z ( a ) i and z ( b ) i .
Revised: z ( a ) i and z ( b ) i .

Original: Allowing the posteriors of an option, for example z ( a ) 2 and z ( b ) 2 , to deviate from the prior ¯
Revised: Allowing the posteriors of an option, for example z ( a ) 2 and z ( b ) 2 , to deviate from the prior ¯

Original: z 2 allows that option, in a ﬁrst step, to solve both tasks, irrespective of the misaligned priors.
Revised: z 2 allows that option, in a ﬁrst step, to solve both tasks, irrespective of the misaligned priors.

Original: This is shown in Figure 4(b) where all posteriors of z 1 are moving to A , but for z 2 , the posteriors on different tasks a and b move to different targets.
Revised: This is shown in Figure 4(b) where all posteriors of z 1 are moving to A , but for z 2 , the posteriors on different tasks a and b move to different targets.

Original: Crucially, to maximize the regularized reward (6), the KL-divergences between priors and posteriors should be minimized along they trajectories, i.e. weighted by how likely they are to occur .
Revised: Crucially, to maximize the regularized reward (6), the KL-divergences between priors and posteriors should be minimized along they trajectories, i.e. weighted by how likely they are to occur .

Original: Consequently, the higher D KL [ z ( a ) 2 (cid:107) ¯ z 2 ] > 0 can be avoided on task a by picking the more specialized option z 1 for which D KL [ z ( a ) 1 (cid:107) ¯ z 1 ] = 0 .
Revised: Consequently, the higher D KL [ z ( a ) 2 (cid:107) ¯ z 2 ] > 0 can be avoided on task a by picking the more specialized option z 1 for which D KL [ z ( a ) 1 (cid:107) ¯ z 1 ] = 0 .

Original: This, in turn, “frees up” z 2 to now also specialize to B as shown in
Revised: This, in turn, “frees up” z 2 to now also specialize to B as shown in

Original: Figure 4(c).
Revised: Figure 4(c).

Original: After this implicit coordination, z ( a ) 1 is always used to reach A and z 2 ( b ) is always used to reach B , allowing both ¯ z 1 and ¯ z 2 to also lead to A and B respectively.
Revised: After this implicit coordination, z ( a ) 1 is always used to reach A and z 2 ( b ) is always used to reach B , allowing both ¯ z 1 and ¯ z 2 to also lead to A and B respectively.

Original: B R ELATIONSHIP BETWEEN MSOL AND CLASSICAL OPTIONS
Revised: B R ELATIONSHIP BETWEEN MSOL AND CLASSICAL OPTIONS

Original: Assume we are faced with a new task and are given some prior knowledge in the form of a set of skills that we can use.
Revised: Assume we are faced with a new task and are given some prior knowledge in the form of a set of skills that we can use.

Original: Using those skills and their termination probabilities as prior policies p T and p L in the soft option framework, we can see β as a temperature parameter determining how closely we are restricted to following them.
Revised: Using those skills and their termination probabilities as prior policies p T and p L in the soft option framework, we can see β as a temperature parameter determining how closely we are restricted to following them.

Original: For β → ∞ we recover the classical ‘hard’ option case and our posterior option policies are restricted to the prior.
Revised: For β → ∞ we recover the classical ‘hard’ option case and our posterior option policies are restricted to the prior.

Original: 1 For β = 

Revised: 1 For β =

Original: the priors only initialize the otherwise unconstrained policy, quickly unlearning behavior that may be useful down the line.
Revised: the priors only initialize the otherwise unconstrained policy, quickly unlearning behavior that may be useful down the line.

Original: Lastly, for 0 < β < ∞ we use the prior information to guide exploration but are only softly restricted to the given skills and can also explore and use policies ‘close’ to them.
Revised: Lastly, for 0 < β < ∞ we use the prior information to guide exploration but are only softly restricted to the given skills and can also explore and use policies ‘close’ to them.

Original: C MSOL TRAINING DETAILS C. O PTIMIZATION
Revised: C MSOL TRAINING DETAILS C. O PTIMIZATION

Original: Even though R reg i depends on φ i , its gradient w.r.t.
Revised: Even though R reg i depends on φ i , its gradient w.r.t.

Original: φ i vanishes.
Revised: φ i vanishes.

Original: Consequently, we can treat the regularized reward as a classical RL reward and use any RL algorithm to ﬁnd the optimal hierarchical policy parameters φ i .
Revised: Consequently, we can treat the regularized reward as a classical RL reward and use any RL algorithm to ﬁnd the optimal hierarchical policy parameters φ i .

Original: In the following, we explain how to adapt A2C (Mnih et al., 2016) to soft options.
Revised: In the following, we explain how to adapt A2C (Mnih et al., 2016) to soft options.

Original: The extension to PPO (Schulman et al., 2017) is straightforward.
Revised: The extension to PPO (Schulman et al., 2017) is straightforward.

Original: The joint posterior policy in (4) depends on the current state s t and the previously selected option z t − 1 .
Revised: The joint posterior policy in (4) depends on the current state s t and the previously selected option z t − 1 .

Original: The expected sum of regularized future rewards of task i , the value function V i , must therefore also condition on this pair:
Revised: The expected sum of regularized future rewards of task i , the value function V i , must therefore also condition on this pair:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: As V i ( s t , z t − 1 ) cannot be directly observed, we approximate it with a parametrized model
Revised: As V i ( s t , z t − 1 ) cannot be directly observed, we approximate it with a parametrized model

Original: V φ i ( s t , z t − 1 ) .
Revised: V φ i ( s t , z t − 1 ) .

Original: The k -step advantage estimation at time t of trajectory τ is given by
Revised: The k -step advantage estimation at time t of trajectory τ is given by

Original: [Equation]
Revised: [Equation]

Original: where the superscript ‘ − ’ indicates treating the term as a constant.
Revised: where the superscript ‘ − ’ indicates treating the term as a constant.

Original: The approximate value function V φ i can be optimized towards its bootstrapped k -step target by minimizing L V ( φ i , τ 1: T ) :
Revised: The approximate value function V φ i can be optimized towards its bootstrapped k -step target by minimizing L V ( φ i , τ 1: T ) :

Original: = (cid:80)
Revised: = (cid:80)

Original: Tt =

Revised: Tt =

Original: ( A φ i ( τ t :( t + k ) )) 2 .
Revised: ( A φ i ( τ t :( t + k ) )) 2 .

Original: As per A2C, k ∈ [1 . . .
Revised: As per A2C, k ∈ [1 . . .

Original: n s ] depending on the state (Mnih et al., 2016).
Revised: n s ] depending on the state (Mnih et al., 2016).

Original: The corresponding policy gradient loss is
Revised: The corresponding policy gradient loss is

Original: [Equation]
Revised: [Equation]

Original: The gradient w.r.t.
Revised: The gradient w.r.t.

Original: the prior parameters θ is
Revised: the prior parameters θ is

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ˜ b t = δ z t − 1 ( z (cid:48) t ) and z (cid:48) t ∼ q H ( z (cid:48)
Revised: where ˜ b t = δ z t − 1 ( z (cid:48) t ) and z (cid:48) t ∼ q H ( z (cid:48)

Original: t | s t , z t − 1 , b t = 1) .
Revised: t | s t , z t − 1 , b t = 1) .

Original: To encourage exploration in all policies of the hierarchy, we also include an entropy maximization loss:
Revised: To encourage exploration in all policies of the hierarchy, we also include an entropy maximization loss:

Original: [Equation]
Revised: [Equation]

Original: Note that term 1 in (6) already encourages maximizing L H ( φ i , τ ) for the master policy, since we chose a uniform prior p H ( z t | b t = 1) .
Revised: Note that term 1 in (6) already encourages maximizing L H ( φ i , τ ) for the master policy, since we chose a uniform prior p H ( z t | b t = 1) .

Original: As both terms serve the same purpose, we are free to drop either one of them.
Revised: As both terms serve the same purpose, we are free to drop either one of them.

Original: In our experiments, we chose to drop the term for q H in R reg t , which proved slightly more stable to optimize that the alternative.
Revised: In our experiments, we chose to drop the term for q H in R reg t , which proved slightly more stable to optimize that the alternative.

Original: We can optimize all parameters jointly with a combined loss over all tasks i , based on sampled trajectories τ i :=
Revised: We can optimize all parameters jointly with a combined loss over all tasks i , based on sampled trajectories τ i :=

Original: τ i 1: T ∼ q φ i and corresponding sampled values of ˜ b
Revised: τ i 1: T ∼ q φ i and corresponding sampled values of ˜ b

Original: i :=
Revised: i :=

Original: ˜ b i 1: T :
Revised: ˜ b i 1: T :

Original: L ( {
Revised: L ( {

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: i } , { ˜ b i } ) =
Revised: i } , { ˜ b i } ) =

Original: [Equation]
Revised: [Equation]

Original: C. T RAINING SCHEDULE
Revised: C. T RAINING SCHEDULE

Original: For faster training, it is important to prevent the master policies q H from converging too quickly to allow sufﬁcient updating of all options.
Revised: For faster training, it is important to prevent the master policies q H from converging too quickly to allow sufﬁcient updating of all options.

Original: On the other hand, a lower exploration rate leads to more clearly deﬁned options.
Revised: On the other hand, a lower exploration rate leads to more clearly deﬁned options.

Original: We consequently anneal the exploration bonus λ H with a linear schedule during training.
Revised: We consequently anneal the exploration bonus λ H with a linear schedule during training.

Original: Similarly, a high value of β leads to better options but can prevent ﬁnding the extrinsic reward r i ( s t , a t ) early on in training.
Revised: Similarly, a high value of β leads to better options but can prevent ﬁnding the extrinsic reward r i ( s t , a t ) early on in training.

Original: Consequently, we increase β over the course of training, also using a linear schedule.
Revised: Consequently, we increase β over the course of training, also using a linear schedule.

Original: D A RCHITECTURE
Revised: D A RCHITECTURE

Original: All policies and value functions share the same encoder network with two fully connected hidden layers of size 64 for the Moving Bandits environment and three hidden layers of sizes 512, 256, and 512 for the Taxi environments.
Revised: All policies and value functions share the same encoder network with two fully connected hidden layers of size 64 for the Moving Bandits environment and three hidden layers of sizes 512, 256, and 512 for the Taxi environments.

Original: Distral was tested with both model sizes on the Moving Bandits task to make sure that limited capacity is not the problem.
Revised: Distral was tested with both model sizes on the Moving Bandits task to make sure that limited capacity is not the problem.

Original: Both models resulted in similar performance, the results shown in the paper are for the larger model.
Revised: Both models resulted in similar performance, the results shown in the paper are for the larger model.

Original: On swimmer the encoder model size is 1024 × 256 × 512 .
Revised: On swimmer the encoder model size is 1024 × 256 × 512 .

Original: Master-policies, as well as all prior- and posterior policies and value functions consist of only one layer which takes the latent embedding produced by the encoder as input.
Revised: Master-policies, as well as all prior- and posterior policies and value functions consist of only one layer which takes the latent embedding produced by the encoder as input.

Original: Furthermore, the encoder is shared across tasks, allowing for much faster training since observations can be batched together.
Revised: Furthermore, the encoder is shared across tasks, allowing for much faster training since observations can be batched together.

Original: Options are speciﬁed as an additional one-hot encoded input to the corresponding network that is passed through a single 128 dimensional fully connected layer and concatenated to the state embedding before the last hidden layer.
Revised: Options are speciﬁed as an additional one-hot encoded input to the corresponding network that is passed through a single 128 dimensional fully connected layer and concatenated to the state embedding before the last hidden layer.

Original: We implement the single-column architecture of Distral as a hierarchical policy with just one option and with a modiﬁed loss function that does not include terms for the master and termination policies.
Revised: We implement the single-column architecture of Distral as a hierarchical policy with just one option and with a modiﬁed loss function that does not include terms for the master and termination policies.

Original: Our implementation builds on the A2C/PPO implementation by Kostrikov (2018), and we use the implementation for MLSH that is provided by the authors ( https://github.com/openai/mlsh ).
Revised: Our implementation builds on the A2C/PPO implementation by Kostrikov (2018), and we use the implementation for MLSH that is provided by the authors ( https://github.com/openai/mlsh ).

Original: E H YPER - PARAMETERS AND ADDITIONAL ENVIRONMENT DETAILS
Revised: E H YPER - PARAMETERS AND ADDITIONAL ENVIRONMENT DETAILS

Original: We use 2 λ V =
Revised: We use 2 λ V =

Original: λ
Revised: λ

Original: A = λ P = 1 in all experiments.
Revised: A = λ P = 1 in all experiments.

Original: Furthermore, we train on all tasks from the task distribution, regularly resetting individual tasks by resetting the corresponding master and re-initializing the posterior policies.
Revised: Furthermore, we train on all tasks from the task distribution, regularly resetting individual tasks by resetting the corresponding master and re-initializing the posterior policies.

Original: Optimizing β for MSOL and Distral was done over { 0 . 01 , 0 . 02 , 0 . 04 , 0 . 1 , 0 . 2 , 0 . 4 } .
Revised: Optimizing β for MSOL and Distral was done over { 0 . 01 , 0 . 02 , 0 . 04 , 0 . 1 , 0 . 2 , 0 . 4 } .

Original: We use γ = 0 .
Revised: We use γ = 0 .

Original: 95 for Moving Bandits and Taxi and γ = 0 .for
Revised: 95 for Moving Bandits and Taxi and γ = 0 .for

Original: Swimmer.
Revised: Swimmer.

Original: E. M OVING B ANDITS
Revised: E. M OVING B ANDITS

Original: For MLSH , we use the original hyper-parameters (Frans et al., 2018).
Revised: For MLSH , we use the original hyper-parameters (Frans et al., 2018).

Original: The duration of each option is ﬁxed to 10.
Revised: The duration of each option is ﬁxed to 10.

Original: The required warm-up duration is set to 9 and the training duration set to 1.
Revised: The required warm-up duration is set to 9 and the training duration set to 1.

Original: We also use 30 parallel environments split between 10 tasks.
Revised: We also use 30 parallel environments split between 10 tasks.

Original: This and the training duration are the main differences to the original paper.
Revised: This and the training duration are the main differences to the original paper.

Original: Originally, MLSH was trained on 120 parallel environments which we were unable to do due to hardware constraints.
Revised: Originally, MLSH was trained on 120 parallel environments which we were unable to do due to hardware constraints.

Original: Training is done over 6 million frames per task.
Revised: Training is done over 6 million frames per task.

Original: For MSOL and Distral we use the same number of 10 tasks and 30 processes.
Revised: For MSOL and Distral we use the same number of 10 tasks and 30 processes.

Original: The duration of options are learned and we do not require a warm-up period.
Revised: The duration of options are learned and we do not require a warm-up period.

Original: We set the learning rate to 0 .
Revised: We set the learning rate to 0 .

Original: 01 and β = 0 .
Revised: 01 and β = 0 .

Original: 2 , α = 0 .
Revised: 2 , α = 0 .

Original: 95 , λ H = 0 .
Revised: 95 , λ H = 0 .

Original: Training is done over 0.6 million frames per task.
Revised: Training is done over 0.6 million frames per task.

Original: For Distral we use β = 0 .
Revised: For Distral we use β = 0 .

Original: 04 , λ H = 0 .
Revised: 04 , λ H = 0 .

Original: 05 and also 0.6 million frames per task.
Revised: 05 and also 0.6 million frames per task.

Original: E. T AXI
Revised: E. T AXI

Original: For MSOL we anneal β from 0.02 to 0.1 and λ H from 0.1 to 0.05.
Revised: For MSOL we anneal β from 0.02 to 0.1 and λ H from 0.1 to 0.05.

Original: For Distral we use β = 0 .
Revised: For Distral we use β = 0 .

Original: We use 3 processes per task to collect experience for a batch size of 15 per task.
Revised: We use 3 processes per task to collect experience for a batch size of 15 per task.

Original: Training is done over 1.4 million frames per task for Taxi and 4 million frames per task for Directional Taxi .
Revised: Training is done over 1.4 million frames per task for Taxi and 4 million frames per task for Directional Taxi .

Original: MLSH was trained on 0.6 million frames for Taxi as due to it’s long runtime of several days, using more frames was infeasible.
Revised: MLSH was trained on 0.6 million frames for Taxi as due to it’s long runtime of several days, using more frames was infeasible.

Original: Training was already converged.
Revised: Training was already converged.

Original: E. T AXI : A DAPTATION TO CHANGED TASKS
Revised: E. T AXI : A DAPTATION TO CHANGED TASKS

Original: As shown in the main experiments, soft options can still be useful, even when the task distribution changes.
Revised: As shown in the main experiments, soft options can still be useful, even when the task distribution changes.

Original: It is unsurprising that hard options are not able to solve the task.
Revised: It is unsurprising that hard options are not able to solve the task.

Original: However, interestingly, providing hard options and primitive actions to the master policy performs much worse than just learning from scratch.
Revised: However, interestingly, providing hard options and primitive actions to the master policy performs much worse than just learning from scratch.

Original: This phenomenon was investigated previously by Jong et al.
Revised: This phenomenon was investigated previously by Jong et al.

Original: (2008) and further justiﬁes using soft options for transfer to out-of-distribution tasks.
Revised: (2008) and further justiﬁes using soft options for transfer to out-of-distribution tasks.

Original: Whether training from scratch or re-using misspeciﬁed options that were trained on a different set of tasks is learning faster mainly depends on i) how strongly the options are misspeciﬁed and ii) how difﬁcult the exploration problem is in the environment.
Revised: Whether training from scratch or re-using misspeciﬁed options that were trained on a different set of tasks is learning faster mainly depends on i) how strongly the options are misspeciﬁed and ii) how difﬁcult the exploration problem is in the environment.

Original: This tradeoff is shown in Figure 6:
Revised: This tradeoff is shown in Figure 6:

Original: On a smaller 8x8 grid (left), learning from scratch performs competitively because exploration is sufﬁciently simple.
Revised: On a smaller 8x8 grid (left), learning from scratch performs competitively because exploration is sufﬁciently simple.

Original: On the other hand, on a 10x10 grid (right, results from the main paper), exploration is harder and soft options allow for signiﬁcantly faster learning because they can guide the exploration in a helpful way.
Revised: On the other hand, on a 10x10 grid (right, results from the main paper), exploration is harder and soft options allow for signiﬁcantly faster learning because they can guide the exploration in a helpful way.

Original: E. S WIMMER
Revised: E. S WIMMER

Original: For training Distral and MSOL we use PPO instead of A2C as it generally achieves better performance on continuous tasks.
Revised: For training Distral and MSOL we use PPO instead of A2C as it generally achieves better performance on continuous tasks.

Original: We have λ H = 0 .
Revised: We have λ H = 0 .

Original: for both MSOL and Distral for primitive actions and λ (cid:48) H = 0 .
Revised: for both MSOL and Distral for primitive actions and λ (cid:48) H = 0 .

Original: 1 for the master- and termination policies in MSOL .
Revised: 1 for the master- and termination policies in MSOL .

Original: We use a learning rate of 0 . 0002 ,
Revised: We use a learning rate of 0 . 0002 ,

Original: GAE (Schulman et al., 2015) with τ = 0 .
Revised: GAE (Schulman et al., 2015) with τ = 0 .

Original: We collect 2000 steps in parallel on 6 processes per task, resulting in a batchsize of 12 , 000 per task.
Revised: We collect 2000 steps in parallel on 6 processes per task, resulting in a batchsize of 12 , 000 per task.

Original: Training is done over 6 million frames with a linearly scheduled increase of β from 0 to 0 .
Revised: Training is done over 6 million frames with a linearly scheduled increase of β from 0 to 0 .

Original: 04 for MSOL and 0 .
Revised: 04 for MSOL and 0 .

Original: 01 for Distral.
Revised: 01 for Distral.

Original: We set α = 0 . 98 .
Revised: We set α = 0 . 98 .

Original: [Figure]
Revised: [Figure]

Original: ) share the same training as they only differ during testing.
Revised: ) share the same training as they only differ during testing.

Original: Further, note that the highest achievable performance for
Revised: Further, note that the highest achievable performance for

Original: Taxi and Directional Taxi is higher during training as they can be initialized closer to the ﬁnal goal (i.e. with the passenger on board).
Revised: Taxi and Directional Taxi is higher during training as they can be initialized closer to the ﬁnal goal (i.e. with the passenger on board).

Original: 5 Steps × 10−−−−−
Revised: 5 Steps × 10−−−−−

Original: E x p ec t e d
Revised: E x p ec t e d

Original: R e t u r n (
Revised: R e t u r n (

Original: [Figure]
Revised: [Figure]

Original: 8x 0 .0 .0 .0 .0 .1 .
Revised: 8x 0 .0 .0 .0 .0 .1 .

Original: Steps × 10−−−−−
Revised: Steps × 10−−−−−

Original: E x p ec t e d
Revised: E x p ec t e d

Original: R e t u r n
Revised: R e t u r n

Original: [Figure]
Revised: [Figure]