Original: H I T-MDP: L EARNING THE SMDP OPTION FRAMEWORK ON MDP S WITH H IDDEN T EMPORAL V ARIABLES
Revised: H I T-MDP: L EARNING THE SMDP OPTION FRAMEWORK ON MDP S WITH H IDDEN T EMPORAL V ARIABLES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The standard option framework is developed on the Semi-Markov Decision Process (SMDP) which is unstable to optimize and sample inefficient.
Revised: The standard option framework is developed on the Semi-Markov Decision Process (SMDP) which is unstable to optimize and sample inefficient.

Original: To this end, we propose a novel Markov Decision Process (MDP), the Hidden Temporal MDP (HiTMDP), and prove that the option-induced HiT-MDP is homomorphic equivalent to the option-induced SMDP.
Revised: To this end, we propose a novel Markov Decision Process (MDP), the Hidden Temporal MDP (HiTMDP), and prove that the option-induced HiT-MDP is homomorphic equivalent to the option-induced SMDP.

Original: We also derive a sample efficient structured variational inference-based algorithm which leads to a novel stable option discovering method under the maximum-entropy reinforcement learning framework.
Revised: We also derive a sample efficient structured variational inference-based algorithm which leads to a novel stable option discovering method under the maximum-entropy reinforcement learning framework.

Original: Extensive experiments on challenging Mujoco environments demonstrate HiT-MDP’s efficiency and effectiveness: under widely used configurations, HiT-MDP achieves competitive, if not better, performance compared to the state-of-the-art baselines on all finite horizon and transfer learning environments.
Revised: Extensive experiments on challenging Mujoco environments demonstrate HiT-MDP’s efficiency and effectiveness: under widely used configurations, HiT-MDP achieves competitive, if not better, performance compared to the state-of-the-art baselines on all finite horizon and transfer learning environments.

Original: Moreover, HiT-MDP significantly outperforms all baselines on infinite horizon environments while exhibiting smaller variance, faster convergence, and better interpretability.
Revised: Moreover, HiT-MDP significantly outperforms all baselines on infinite horizon environments while exhibiting smaller variance, faster convergence, and better interpretability.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The option framework (Sutton et al., 1999) is one of the most promising frameworks to enable RL methods to conduct lifelong learning (Mankowitz et al., 2016) and has proven benefits in speeding learning (Bacon, 2018), improving exploration (Harb et al., 2018), and facilitating transfer learning (Zhang & Whiteson, 2019).
Revised: The option framework (Sutton et al., 1999) is one of the most promising frameworks to enable RL methods to conduct lifelong learning (Mankowitz et al., 2016) and has proven benefits in speeding learning (Bacon, 2018), improving exploration (Harb et al., 2018), and facilitating transfer learning (Zhang & Whiteson, 2019).

Original: Standard option framework is developed on Semi-Markov Decision
Revised: Standard option framework is developed on Semi-Markov Decision

Original: Process, we refer to this as SMDP-Option.
Revised: Process, we refer to this as SMDP-Option.

Original: In SMDP-Option, an option is a temporally abstracted action whose execution cross a variable amount of time steps.
Revised: In SMDP-Option, an option is a temporally abstracted action whose execution cross a variable amount of time steps.

Original: A master policy is employed to compose these options and determine which option should be executed and stopped.
Revised: A master policy is employed to compose these options and determine which option should be executed and stopped.

Original: The SMDP formulation has two deficiencies that severely impair options’ applicability in a broader context (Jong et al., 2008).
Revised: The SMDP formulation has two deficiencies that severely impair options’ applicability in a broader context (Jong et al., 2008).

Original: The first deficiency is sample inefficiency.
Revised: The first deficiency is sample inefficiency.

Original: Since the execution of an option persists over multiple time steps, one update of the master policy consumes various steps of samples and thus is sample inefficient (Levy & Shimkin, 2011; Daniel et al., 2016; Bacon et al., 2017).
Revised: Since the execution of an option persists over multiple time steps, one update of the master policy consumes various steps of samples and thus is sample inefficient (Levy & Shimkin, 2011; Daniel et al., 2016; Bacon et al., 2017).

Original: The second deficiency is unstable optimizing algorithms.
Revised: The second deficiency is unstable optimizing algorithms.

Original: SMDP-based optimization algorithms are notoriously sensitive to hyperparameters.
Revised: SMDP-based optimization algorithms are notoriously sensitive to hyperparameters.

Original: Therefore, they often exhibit large variance (Wulfmeier et al., 2020) and encounter convergence issues (Klissarov et al., 2017).
Revised: Therefore, they often exhibit large variance (Wulfmeier et al., 2020) and encounter convergence issues (Klissarov et al., 2017).

Original: Extensive research has tried to tackle these issues from aspects such as improving the policy iteration procedure (Sutton et al., 1999; Daniel et al., 2016; Bacon et al., 2017) and adding extra constraints on option discovering objectives (Khetarpal et al., 2020; Wulfmeier et al., 2020; Hyun et al., 2019).
Revised: Extensive research has tried to tackle these issues from aspects such as improving the policy iteration procedure (Sutton et al., 1999; Daniel et al., 2016; Bacon et al., 2017) and adding extra constraints on option discovering objectives (Khetarpal et al., 2020; Wulfmeier et al., 2020; Hyun et al., 2019).

Original: However, rare works (Levy & Shimkin, 2011; Smith et al., 2018; Zhang & Whiteson, 2019) explore from the perspective of improving the underlying Decision Process.
Revised: However, rare works (Levy & Shimkin, 2011; Smith et al., 2018; Zhang & Whiteson, 2019) explore from the perspective of improving the underlying Decision Process.

Original: Our work is largely different from literatures above and more details are discussed in Section 6.
Revised: Our work is largely different from literatures above and more details are discussed in Section 6.

Original: In this work, we present a counterintuitive finding: the SMDP formulated option framework has an MDP equivalence which is still able to temporally extending the execution of abstracted actions.
Revised: In this work, we present a counterintuitive finding: the SMDP formulated option framework has an MDP equivalence which is still able to temporally extending the execution of abstracted actions.

Original: MDP-based options can address SMDP-based ones’ deficiencies from two aspects: (1) sample efficient, i.e. , MDPs policies can be optimized at every sampling step (Bacon, 2018); and (2) more stable to optimize, i.e. , convergence of MDPs algorithms are well theoretically justified and have smaller variance (Schulman et al., 2015).
Revised: MDP-based options can address SMDP-based ones’ deficiencies from two aspects: (1) sample efficient, i.e. , MDPs policies can be optimized at every sampling step (Bacon, 2018); and (2) more stable to optimize, i.e. , convergence of MDPs algorithms are well theoretically justified and have smaller variance (Schulman et al., 2015).

Original: In this paper, we propose the Hidden Temporal MDP (HiT-MDP) and theoretically prove the equivalence to the SMDP-Option.
Revised: In this paper, we propose the Hidden Temporal MDP (HiT-MDP) and theoretically prove the equivalence to the SMDP-Option.

Original: We first formulate HiTMDP as an HMM-like PGM and introduce temporally dependent latent variables into the MDP to preserve temporal abstractions.
Revised: We first formulate HiTMDP as an HMM-like PGM and introduce temporally dependent latent variables into the MDP to preserve temporal abstractions.

Original: By exploiting conditional independencies in PGM, we prove that the HiT-MDP is homomorphic equivalent (Ravindran, 2003) to SMDP-Option.
Revised: By exploiting conditional independencies in PGM, we prove that the HiT-MDP is homomorphic equivalent (Ravindran, 2003) to SMDP-Option.

Original: To the best of our knowledge, this is the first work proposing an MDP equivalence of the standard option framework.
Revised: To the best of our knowledge, this is the first work proposing an MDP equivalence of the standard option framework.

Original: In order to solve optimal values of the HiT-MDPs, we devise a Markovian Option-Value Function ¯
Revised: In order to solve optimal values of the HiT-MDPs, we devise a Markovian Option-Value Function ¯

Original: V [ s t , o t − 1 ] and prove that it is an unbiased estimation of the standard value function V [ s t ] .
Revised: V [ s t , o t − 1 ] and prove that it is an unbiased estimation of the standard value function V [ s t ] .

Original: We further develop the Hidden Temporal Bellman Equation in order to derive the policy evaluation theorem for
Revised: We further develop the Hidden Temporal Bellman Equation in order to derive the policy evaluation theorem for

Original: HiT-MDPs.
Revised: HiT-MDPs.

Original: We also show that the Markovian Option-Value Function has a variance reduction effect.
Revised: We also show that the Markovian Option-Value Function has a variance reduction effect.

Original: As a result, HiT-MDP is a general-purpose MDP that can be updated at every sampling step, and thus naturally address the sample inefficiency issue.
Revised: As a result, HiT-MDP is a general-purpose MDP that can be updated at every sampling step, and thus naturally address the sample inefficiency issue.

Original: We solve the learning problem by deriving a stable on-policy policy gradient method under the maximum entropy reinforcement learning framework.
Revised: We solve the learning problem by deriving a stable on-policy policy gradient method under the maximum entropy reinforcement learning framework.

Original: One difficulty of learning standard option frameworks is that they do not have any constraint on qualities of options (Harb et al., 2018).
Revised: One difficulty of learning standard option frameworks is that they do not have any constraint on qualities of options (Harb et al., 2018).

Original: Standard option frameworks have tendencies to learn either degenerate options (Harb et al., 2018)
Revised: Standard option frameworks have tendencies to learn either degenerate options (Harb et al., 2018)

Original: (short execution time) that switching back-and-forth frequently, or dominant options (Zhang &
Revised: (short execution time) that switching back-and-forth frequently, or dominant options (Zhang &

Original: Whiteson, 2019) (long execution time) that executing through the whole episode.
Revised: Whiteson, 2019) (long execution time) that executing through the whole episode.

Original: We tackle this problem by proposing the Maximum entropy Options Policy Gradient (MOPG) algorithm.
Revised: We tackle this problem by proposing the Maximum entropy Options Policy Gradient (MOPG) algorithm.

Original: MOPG includes an information-theoretical intrinsic reward to encourage consecutive executions of options and entropy terms to encourage explorations of options.
Revised: MOPG includes an information-theoretical intrinsic reward to encourage consecutive executions of options and entropy terms to encourage explorations of options.

Original: The whole algorithm can be solved in an end-to-end manner under the structional variational inference framework.
Revised: The whole algorithm can be solved in an end-to-end manner under the structional variational inference framework.

Original: We theoretically prove that optimizing through MOPG converges to the optimal trajectory.
Revised: We theoretically prove that optimizing through MOPG converges to the optimal trajectory.

Original: We conduct experiments on challenging Mujoco (Todorov et al., 2012; Brockman et al., 2016b;
Revised: We conduct experiments on challenging Mujoco (Todorov et al., 2012; Brockman et al., 2016b;

Original: Tunyasuvunakool et al., 2020) environments.
Revised: Tunyasuvunakool et al., 2020) environments.

Original: Thorough empirical results demonstrate that under widely used configurations, HiT-MDP achieves competitive, if not better, performance compared to the state-of-the-art baselines on all finite horizon and transfer learning environments.
Revised: Thorough empirical results demonstrate that under widely used configurations, HiT-MDP achieves competitive, if not better, performance compared to the state-of-the-art baselines on all finite horizon and transfer learning environments.

Original: Moreover,
Revised: Moreover,

Original: HiT-MDP significantly outperforms all baselines on infinite horizon environments while exhibiting smaller variance, faster convergence, and interpretability.
Revised: HiT-MDP significantly outperforms all baselines on infinite horizon environments while exhibiting smaller variance, faster convergence, and interpretability.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Markov Decision Process: A Markov Decision Process (Puterman, 1994)
Revised: Markov Decision Process: A Markov Decision Process (Puterman, 1994)

Original: M = { S , A , r, P, γ } consists of a state space S , an action space A , a state transition function P ( s t +1 | s t , a t ) :
Revised: M = { S , A , r, P, γ } consists of a state space S , an action space A , a state transition function P ( s t +1 | s t , a t ) :

Original: S ×
Revised: S ×

Original: A → S , a discount factor γ ∈ R , and a reward function r ( s , a ) =
Revised: A → S , a discount factor γ ∈ R , and a reward function r ( s , a ) =

Original: E [ r | s , a ] : S ×
Revised: E [ r | s , a ] : S ×

Original: A → R which is the expectation of the reward r t +1 ∈ R received from the environment after executing action a t at state s t .
Revised: A → R which is the expectation of the reward r t +1 ∈ R received from the environment after executing action a t at state s t .

Original: A policy π = P ( a | s ) :
Revised: A policy π = P ( a | s ) :

Original: A × S → [0 , 1] is a probability distribution defined over actions conditioning on states.
Revised: A × S → [0 , 1] is a probability distribution defined over actions conditioning on states.

Original: A discounted return is defined as G t = (cid:80)
Revised: A discounted return is defined as G t = (cid:80)

Original: Nk γ k r t + k +1 , where γ ∈ (0 , 1) is a discounting factor.
Revised: Nk γ k r t + k +1 , where γ ∈ (0 , 1) is a discounting factor.

Original: The value function V [ s t ] =
Revised: The value function V [ s t ] =

Original: E τ ∼ π
Revised: E τ ∼ π

Original: [ G t | s t ] is the expected return starting at state s t and the trajectory τ = { s t , a t , r t +1 , s t +1 , . . .
Revised: [ G t | s t ] is the expected return starting at state s t and the trajectory τ = { s t , a t , r t +1 , s t +1 , . . .

Original: } follows policy π thereafter.
Revised: } follows policy π thereafter.

Original: The action-value function is defined as Q [ s t , a t ] =
Revised: The action-value function is defined as Q [ s t , a t ] =

Original: E τ ∼ π
Revised: E τ ∼ π

Original: [ G t | s t , a t ] .
Revised: [ G t | s t , a t ] .

Original: Homomorphic Equivalence: The equivalence relation between MDPs is defined by Givan et al.
Revised: Homomorphic Equivalence: The equivalence relation between MDPs is defined by Givan et al.

Original: (2003) as symmetric equivalence (bisimulation relation).
Revised: (2003) as symmetric equivalence (bisimulation relation).

Original: Ravindran (2003) extends their work to homomorphic equivalence that allows defining symmetries between an MDP and SMDP.
Revised: Ravindran (2003) extends their work to homomorphic equivalence that allows defining symmetries between an MDP and SMDP.

Original: Given two processes, an MDP M = { S , A , R, P, γ } with the trajectory τ and an SMDP ˜ M = { ˜ S , A , ˜ R, ˜ P, ˜ γ } with the trajectory ˜ τ .
Revised: Given two processes, an MDP M = { S , A , R, P, γ } with the trajectory τ and an SMDP ˜ M = { ˜ S , A , ˜ R, ˜ P, ˜ γ } with the trajectory ˜ τ .

Original: Assume both M and ˜ M share the same action space A .
Revised: Assume both M and ˜ M share the same action space A .

Original: An homomorphism ˜
Revised: An homomorphism ˜

Original: B is a tuple of surjection partition functions, M and ˜ M is Homomorphic Equivalence if 1) for all state-action pairs { s , a } , there exists a many-to-one correspondence equivalent state-action pair { ˜ s , ˜ a } that { s , a } / ˜ B
Revised: B is a tuple of surjection partition functions, M and ˜ M is Homomorphic Equivalence if 1) for all state-action pairs { s , a } , there exists a many-to-one correspondence equivalent state-action pair { ˜ s , ˜ a } that { s , a } / ˜ B

Original: =
Revised: =

Original: { ˜ s , ˜ a } / ˜ B , or denoted as ˜ B ( { ˜ s , ˜ a } )
Revised: { ˜ s , ˜ a } / ˜ B , or denoted as ˜ B ( { ˜ s , ˜ a } )

Original: = { s , a } , 2) and following conditions hold:
Revised: = { s , a } , 2) and following conditions hold:

Original: P ( τ/ ˜ B ) ≡ P (˜ τ/ ˜ B ) , and ˜ B is a surjection , 2.
Revised: P ( τ/ ˜ B ) ≡ P (˜ τ/ ˜ B ) , and ˜ B is a surjection , 2.

Original: r ( τ/ ˜ B ) ≡ r (˜ τ/ ˜ B ) ,
Revised: [Equation]

Original: The SMDP-based Option Framework : In SMDP-Option (Sutton et al., 1999; Bacon, 2018), an option is a triple ( I o , π o , β o ) ∈
Revised: The SMDP-based Option Framework : In SMDP-Option (Sutton et al., 1999; Bacon, 2018), an option is a triple ( I o , π o , β o ) ∈

Original: O , where O denotes the option set; the subscript o ∈ O = { 1 , 2 , . .
Revised: O , where O denotes the option set; the subscript o ∈ O = { 1 , 2 , . .

Original: , K } is a positive integer index which denotes the o -th triple where K is the number of options; I o is an initiation set indicating where the option can be initiated; π o = P o ( a | s ) :
Revised: , K } is a positive integer index which denotes the o -th triple where K is the number of options; I o is an initiation set indicating where the option can be initiated; π o = P o ( a | s ) :

Original: A × S → [0 , 1] is the action policy of the o th option; β o =
Revised: A × S → [0 , 1] is the action policy of the o th option; β o =

Original: P o ( b = 1 | s ) :
Revised: P o ( b = 1 | s ) :

Original: S → [0 , 1] where b ∈ { 0 , 1 } is a termination function .
Revised: S → [0 , 1] where b ∈ { 0 , 1 } is a termination function .

Original: For clarity, we use P o ( b = 1 | s ) instead of β o which is widely used in previous option literatures (e.g., Sutton et al.
Revised: For clarity, we use P o ( b = 1 | s ) instead of β o which is widely used in previous option literatures (e.g., Sutton et al.

Original: ; Bacon et al.
Revised: ; Bacon et al.

Original: A master policy π ( o | s ) =
Revised: A master policy π ( o | s ) =

Original: P ( o | s ) where o ∈ O is used to sample which option will be executed.
Revised: P ( o | s ) where o ∈ O is used to sample which option will be executed.

Original: Note that we use the bold-case o to denote unrealized random variables and the light-italic-case o to denote a realized instantiation. Conventionally, the execution ofan option employs thecall-and-return model (Sutton et al., 1999): at time step t , an agent either continues the previously executed option [-o t − 
 = o with probability P o ( b = 0 | s ) and sets o t = o t − 
-]
Revised: Therefore, the dynamics (stochastic process) of the option [-o t − 
 = o with probability P o ( b = 0 | s ) and sets o t = o t − 
-]framework is written as:

Original: = o , or terminates o with probability P o ( b = 1 | s ) and samples a new option o t from the master policy P ( o t | s t ) . Therefore, the dynamics (stochastic process) of the option framework is written as:
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where τ = { s 0 , o 0 , a 0 , s 1 , o 1 , a 1 , . . 
Revised: ) where τ = { s 0 , o 0 , a 0 , s 1 , o 1 , a 1 , . . .

Original: . } denotes the trajectory of the option framework.
Revised: } denotes the trajectory of the option framework.

Original: 1 is an indicator function and is only true when o t =
Revised: 1 is an indicator function and is only true when o t =

Original: o t − 1 (notice that o t − 1 is the realization at o t − 1 ).
Revised: o t − 1 (notice that o t − 1 is the realization at o t − 1 ).

Original: Therefore, under this formulation the option framework is defined as a Semi-Markov process since the dependency on an activated option o can cross a variable amount of time (Sutton et al., 1999).
Revised: Therefore, under this formulation the option framework is defined as a Semi-Markov process since the dependency on an activated option o can cross a variable amount of time (Sutton et al., 1999).

Original:  A N MDP E QUIVALENCE OF THE SMDP- BASED O PTION F RAMEWORK
Revised:  A N MDP E QUIVALENCE OF THE SMDP- BASED O PTION F RAMEWORK

Original: In this section, we propose the Hidden Temporal MDP (HiT-MDP).
Revised: In this section, we propose the Hidden Temporal MDP (HiT-MDP).

Original: We first reformulate the SMDPOption as an HMM-like Probabilistic Graphical Model (PGM).
Revised: We first reformulate the SMDPOption as an HMM-like Probabilistic Graphical Model (PGM).

Original: We then propose the HiT-MDP as the marginalized variation of the PGM and prove that the HiT-MDP is homomorphic equivalent (Ravindran, 2003) to the SMDP-Option .
Revised: We then propose the HiT-MDP as the marginalized variation of the PGM and prove that the HiT-MDP is homomorphic equivalent (Ravindran, 2003) to the SMDP-Option .

Original: We also derive the HiT-MDP’s Bellman Equation and prove its convergence by presenting the policy evaluation theorem.
Revised: We also derive the HiT-MDP’s Bellman Equation and prove its convergence by presenting the policy evaluation theorem.

Original: As a result, HiT-MDP is a general-purpose MDP that can be combined with any policy optimization algorithm off-the-shelf.
Revised: As a result, HiT-MDP is a general-purpose MDP that can be combined with any policy optimization algorithm off-the-shelf.

Original: We propose an efficient learning algorithm and complete the proof of the policy iteration theorem in
Revised: We propose an efficient learning algorithm and complete the proof of the policy iteration theorem in

Original: Section 4.
Revised: Section 4.

Original: 3. A N MDP F ORMULATION OF THE O PTION F RAMEWORK
Revised: 3. A N MDP F ORMULATION OF THE O PTION F RAMEWORK

Original: Following Bishop (2006)’s formulation of mixture distributions, we redefine the option random variable o ∈ O
Revised: Following Bishop (2006)’s formulation of mixture distributions, we redefine the option random variable o ∈ O

Original: = { 1 , 2 , . .
Revised: = { 1 , 2 , . .

Original: , K } , which was originally defined as an integer index, but now as a
Revised: , K } , which was originally defined as an integer index, but now as a

Original: K -dimensional one-hot vector ¯ o ∈ ¯ O = { 0 , 1 } K where K is the number of options.
Revised: K -dimensional one-hot vector ¯ o ∈ ¯ O = { 0 , 1 } K where K is the number of options.

Original: We further employ the one-hot vector to reformulate the termination function and action function of each option into two mixture distributions by introducing extra dependencies on ¯ o :
Revised: We further employ the one-hot vector to reformulate the termination function and action function of each option into two mixture distributions by introducing extra dependencies on ¯ o :

Original: [Equation]
Revised: [Equation]

Original: Since the option random variable ¯ o is now a one-hot vector, an instantiation ¯ o ≜ k denotes the activation of the option k , and by definition only the k -th entry of ¯
Revised: Since the option random variable ¯ o is now a one-hot vector, an instantiation ¯ o ≜ k denotes the activation of the option k , and by definition only the k -th entry of ¯

Original: o t is 1 and all the other entries are 0 .
Revised: o t is 1 and all the other entries are 0 .

Original: Therefore, we have P o t ( a t
Revised: Therefore, we have P o t ( a t

Original: | s t ) =
Revised: | s t ) =

Original: P ( a t | s t , ¯ o t = ¯ o t ) and
Revised: P ( a t | s t , ¯ o t = ¯ o t ) and

Original: β o t − 1 = P o t − 1 ( b t = 1 | s t )
Revised: β o t − 1 = P o t [Equation] 1 ( b t = 1 | s t )

Original: = P ( b t = 1 | s t , ¯ o t − 1 =
Revised: = P ( b t = 1 | s t , ¯ o t − 1 =

Original: ¯ o t − 1 ) .
Revised: ¯ o t − 1 ) .

Original: The third reformulation is that we propose a novel MDP mixture master policy P (¯ o t | s t , b t , ¯ o t − 1 ) , which is a mixture distribution containing the SMDP master policy P (¯ o t | s t ) and a degenerate probability as mixture components by adding two extra dependencies on b t and ¯ o t − 1 :
Revised: The third reformulation is that we propose a novel MDP mixture master policy P (¯ o t | s t , b t , ¯ o t − 1 ) , which is a mixture distribution containing the SMDP master policy P (¯ o t | s t ) and a degenerate probability as mixture components by adding two extra dependencies on b t and ¯ o t − 1 :

Original: [Equation]
Revised: [Equation]

Original: where the indicator function 1 o t =
Revised: where the indicator function 1 o t =

Original: o t − 1 used in Eq.
Revised: o t − 1 used in Eq.

Original: 1 is now redefined as a degenerate probability distribution (Puterman, 1994):
Revised: 1 is now redefined as a degenerate probability distribution (Puterman, 1994):

Original: [Equation]
Revised: [Equation]

Original: and the joint distribution can be written as:
Revised: and the joint distribution can be written as:

Original: [Equation]
Revised: [Equation]

Original: Although the mixture master policy in Eq. 4 is MDPformulated, as a mixture component within it, the master policy P (¯ o t | s t ) is still SMDP-formulated and hence cannot be updated by MDP-based algorithms.
Revised: Although the mixture master policy in Eq. 4 is MDP-formulated, as a mixture component within it, the master policy P (¯ o t | s t ) is still SMDP-formulated and hence cannot be updated by MDP-based algorithms.

Original: By marginalizing over the termination variable b t in Eq. 4:
Revised: By marginalizing over the termination variable b t in Eq. 4:

Original: [Equation]
Revised: [Equation]

Original: b t P

Original: [Equation]

Original: t

Original: [Equation]

Original: t ,

Original: [Equation]

Original: 1 ) P (¯ o t | b t , s t , ¯ o t

Original: [Equation]

Original: 1 ) , we propose the
Revised: we propose the

Original: Markovian master policy P (¯ o t | s t , ¯ o t − 1 ) to model this marginal distribution explicitly:
Revised: Markovian master policy P ( ¯ o t | s t , ¯ o t − 1 ) to model this marginal distribution explicitly:

Original: [Equation]
Revised: [Equation]

Original: where Eq.
Revised: where Eq.

Original: 5 denotes the joint distribution of the PGM.
Revised: 5 denotes the joint distribution of the PGM.

Original: In this formulation, P (¯ τ ) is actually an HMM with s t , a t as observable random variables and ¯ o t as latent variables.
Revised: In this formulation, P (¯ τ ) is actually an HMM with s t , a t as observable random variables and ¯ o t as latent variables.

Original: We use π O ( s t , ¯ o t − 1 ) =
Revised: We use π O ( s t , ¯ o t − 1 ) =

Original: P (¯ o t | s t , ¯ o t − 1 ) to denote the master policy and π A ( s t , o t ) =
Revised: P (¯ o t | s t , ¯ o t − 1 ) to denote the master policy and π A ( s t , o t ) =

Original: P ( a t | s t , ¯ o t ) to denote the action policy.
Revised: P ( a t | s t , ¯ o t ) to denote the action policy.

Original: Figure 1 shows the PGM (Eq. 5).
Revised: Figure 1 shows the PGM (Eq. 5).

Original: 3. T HE H IDDEN T EMPORAL MDP S (H I T-MDP S )
Revised: 3. T HE H IDDEN T EMPORAL MDP S (H I T-MDP S )

Original: Given the PGM (Eq. 5, Figure 1), the Hidden Temporal MDPs (HiT-MDPs) family can be described by a tuple M = { ¯ S , ¯ A , r, P, ϕ, γ } where ¯ S .
Revised: Given the PGM (Eq. 5, Figure 1), the Hidden Temporal MDPs (HiT-MDPs) family can be described by a tuple M = { ¯ S , ¯ A , r, P, ϕ, γ } where ¯ S .

Original: = S
Revised: = S

Original: × ¯ O is an augmented state space, ¯ A .
Revised: × ¯ O is an augmented state space, ¯ A .

Original: =
Revised: =

Original: A × ¯ O is an augmented action space, and
Revised: A × ¯ O is an augmented action space, and

Original: ϕ = P (¯ o t | ¯ s t )
Revised: ϕ = P (¯ o t | ¯ s t )

Original: =
Revised: =

Original: P (¯ o t | s t , ¯ o t − 1 ) is the emit function for hidden variables.
Revised: P (¯ o t | s t , ¯ o t − 1 ) is the emit function for hidden variables.

Original: The joint distribution of HiT-MDPs is factorized as Eq. 5 (derivations see Appendix C).
Revised: The joint distribution of HiT-MDPs is factorized as Eq.(derivations see Appendix C).

Original: We define a partition function ¯ B (¯ s , ¯ a ) =
Revised: We define a partition function ¯ B (¯ s , ¯ a ) =

Original: ¯ B ( { ¯ o t − 1 , a t , ¯ o t , s t } ) =
Revised: ¯ B ( { ¯ o t − 1 , a t , ¯ o t , s t } ) =

Original: ( o t − 1 , a t , o t , s t ) (proofs in Appendix C), which maps ¯ τ to τ , where ¯ τ = { s 0 , ¯ o 0 , a 0 , s 1 , ¯ o 1 , a 1 , . . .
Revised: ( o t − 1 , a t , o t , s t ) (proofs in Appendix C), which maps ¯ τ to τ , where ¯ τ = { s 0 , ¯ o 0 , a 0 , s 1 , ¯ o 1 , a 1 , . . .

Original: } is the trajectory of the HiT-MDP .
Revised: } is the trajectory of the HiT-MDP .

Original: Therefore, by following the Partition Function ¯ B , the dynamics of the SMDP-Option in Eq. 1 under the Surjection ¯ B is equivalent to HiT-MDP P ( τ/ ¯ B ) =
Revised: Therefore, by following the Partition Function ¯ B , the dynamics of the SMDP-Option in Eq. 1 under the Surjection ¯ B is equivalent to HiT-MDP P ( τ/ ¯ B ) =

Original: P (¯ τ/ ¯ B ) .
Revised: P (¯ τ/ ¯ B ) .

Original: With P ( τ/ ¯ B ) =
Revised: With P ( τ/ ¯ B ) =

Original: P (¯ τ/ ¯ B ) in hand, to prove the equivalence between the SMDP-Option and HiT-MDP , we move on to prove both of them share the same expected reward.
Revised: P (¯ τ/ ¯ B ) in hand, to prove the equivalence between the SMDP-Option and HiT-MDP , we move on to prove both of them share the same expected reward.

Original: This is non-trivial since compared to the SMDP-Option , the MDP formulation introduces extra dependencies on ¯ o .
Revised: This is non-trivial since compared to the SMDP-Option , the MDP formulation introduces extra dependencies on ¯ o .

Original: However, in Appendix C, by exploiting conditional independencies we prove that they share the same expected return under the Surjection ¯ B .
Revised: However, in Appendix C, by exploiting conditional independencies we prove that they share the same expected return under the Surjection ¯ B .

Original: Therefore, the SMDP-based option framework has an MDP-based equivalence:
Revised: Therefore, the SMDP-based option framework has an MDP-based equivalence:

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: By the definition of Bisimulation Relation, the SMDP-based option framework, which employs Markovian options, has an underlying MDP equivalence because:
Revised: By the definition of Bisimulation Relation, the SMDP-based option framework, which employs Markovian options, has an underlying MDP equivalence because:

Original: P ( τ/ ¯ B ) = P (¯ τ/ ¯ B ) and ¯ B is a Surjection.
Revised: P ( τ/ ¯ B ) = P (¯ τ/ ¯ B ) and ¯ B is a Surjection.

Original: r ( τ/ ¯ B ) =
Revised: r ( τ/ ¯ B ) =

Original: r (¯ τ/ ¯ B )
Revised: r (¯ τ/ ¯ B )

Original: Proof.
Revised: Proof.

Original: See Appendix C.
Revised: See Appendix C.

Original: Since the master policy P (¯ o t | s t , ¯ o t − 1 ) introduces one extra dependency on ¯ o t − 1 , conventional
Revised: Since the master policy P (¯ o t | s t , ¯ o t − 1 ) introduces one extra dependency on ¯ o t − 1 , conventional

Original: Bellman equation which is derived by following the conventional value function V [ s t ] no longer applies to HiT-MDP (see Appendix D).
Revised: Bellman equation which is derived by following the conventional value function V [ s t ] no longer applies to HiT-MDP (see Appendix D).

Original: In order to derive the Bellman equation of HiT-MDP , we propose the novel Markovian option-value function , value functions with Markov dependencies (such as ¯ o t − 1 ).
Revised: In order to derive the Bellman equation of HiT-MDP , we propose the novel Markovian option-value function , value functions with Markov dependencies (such as ¯ o t − 1 ).

Original: Specifically, rather than use the conventional value function V [ s t ] , let the Markovian option-value function ¯ V [ s t , ¯ o t − 1 ] and the option-action value function Q A [ s t , ¯ o t , a t ] be defined by:
Revised: Specifically, rather than use the conventional value function V [ s t ] , let the Markovian option-value function ¯ V [ s t , ¯ o t − 1 ] and the option-action value function Q A [ s t , ¯ o t , a t ] be defined by:

Original: [Equation]
Revised: [Equation]

Original: As with the standard Q-function and value function, we can relate the Q-function to the Markovian option-value function at a future state via a Hidden Temporal Bellman Operator T H .
Revised: As with the standard Q-function and value function, we can relate the Q-function to the Markovian option-value function at a future state via a Hidden Temporal Bellman Operator T H .

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: The option-action value function Eq.
Revised: The option-action value function Eq.

Original: 7 satisfies the Bellman Operator T H
Revised: 7 satisfies the Bellman Operator T H

Original: [Equation]
Revised: [Equation]

Original: where the Markovian option-value function given by Eq. 6.
Revised: where the Markovian option-value function given by Eq. 6.

Original: Proof.
Revised: Proof.

Original: See Appendix D.
Revised: See Appendix D.

Original: We can obtain the option-action value function for any policy by repeatedly applying T H and the sequence converges to the optimal value function:
Revised: We can obtain the option-action value function for any policy by repeatedly applying T H and the sequence converges to the optimal value function:

Original: Theorem 3.3.
Revised: Theorem 3.3.

Original: (Markovian Option Policy Evaluation Theorem).
Revised: (Markovian Option Policy Evaluation Theorem).

Original: Assume that throughout our computation the Q A [ · , · ] and ¯ V [ · ] are bounded and A < ∞ , the sequence Q kA defined by Q k +1 A
Revised: Assume that throughout our computation the Q A [ · , · ] and ¯ V [ · ] are bounded and A < ∞ , the sequence Q kA defined by Q k +1 A

Original: = T H Q kA will converge to the option-action value function Q π A A as k → ∞ .
Revised: = T H Q kA will converge to the option-action value function Q π A A as k → ∞ .

Original: Proof.
Revised: Proof.

Original: As with the standard convergence results for policy evaluation (Sutton & Barto, 2018), by the definition of T H (Eq.
Revised: As with the standard convergence results for policy evaluation (Sutton & Barto, 2018), by the definition of T H (Eq.

Original: 8) the option-action value function Q π A A is a fixed point.
Revised: 8) the option-action value function Q π A A is a fixed point.

Original: In Appendix D we show that T H is a contraction, and then Theorem 3.3 follows immediately.
Revised: In Appendix D we show that T H is a contraction, and then Theorem 3.3 follows immediately.

Original: In Appendix D, we further prove that ¯ V [ s t , ¯ o t − 1 ] is an unbiased estimation of V [ s t ] and has a variance-reduction effect compared to the conventional value function.
Revised: In Appendix D, we further prove that ¯ V [ s t , ¯ o t − 1 ] is an unbiased estimation of V [ s t ] and has a variance-reduction effect compared to the conventional value function.

Original: This property is empirically witnessed and further discussed in experiments (Section 5.1).
Revised: This property is empirically witnessed and further discussed in experiments (Section 5.1).

Original: Proposition 3.4.
Revised: Proposition 3.4.

Original: ¯ V [ s t , ¯ o t − 1 ] is an unbiased estimation of V [ s t ] .
Revised: ¯ V [ s t , ¯ o t − 1 ] is an unbiased estimation of V [ s t ] .

Original: Proposition 3.5.
Revised: Proposition 3.5.

Original: The variance of ¯ V [ s t , ¯ o t − 1 ] is up-bounded by V [ s t ] .
Revised: The variance of ¯ V [ s t , ¯ o t − 1 ] is up-bounded by V [ s t ] .

Original: Proof.
Revised: Proof.

Original: See both Proposition 3.4 and 3.5’s proof in Appendix D
Revised: See both Proposition 3.4 and 3.5’s proof in Appendix D

Original: The theoretical analysis above presents a counterintuitive fact: the SMDP formulated option framework has an MDP formulated homomorphic equivalence (HiT-MDPs), and they both converge to the same optimal value function.
Revised: The theoretical analysis above presents a counterintuitive fact: the SMDP formulated option framework has an MDP formulated homomorphic equivalence (HiT-MDPs), and they both converge to the same optimal value function.

Original: One natural question to ask is that how could an MDP temporally extend executions of options?
Revised: One natural question to ask is that how could an MDP temporally extend executions of options?

Original: Note that the Markovian master policy P (¯ o t | s t , ¯ o t − 1 ) is a result of the marginalization over the termination variable b t (See Eq.
Revised: Note that the Markovian master policy P (¯ o t | s t , ¯ o t − 1 ) is a result of the marginalization over the termination variable b t (See Eq.

Original: 5 and Eq. 1) and has an extra dependency on ¯ o t − 1 ) .
Revised: 5 and Eq. 1) and has an extra dependency on ¯ o t − 1 ) .

Original: Therefore, the Markovian master policy acts like a distance measure.
Revised: Therefore, the Markovian master policy acts like a distance measure.

Original: The decision of selecting an option ¯ o t can be made by simply comparing which ¯ o ∈ ¯ o is closest to the vector [ s t , ¯ o t − 1 ] .
Revised: The decision of selecting an option ¯ o t can be made by simply comparing which ¯ o ∈ ¯ o is closest to the vector [ s t , ¯ o t − 1 ] .

Original: Because a vector is closest to itself, this mechanism has a natural tendency to assign o t to o t − 1 , and thus extends ¯ o t − 1 ’s execution.
Revised: Because a vector is closest to itself, this mechanism has a natural tendency to assign o t to o t − 1 , and thus extends ¯ o t − 1 ’s execution.

Original: On the other hand, a significantly different state s t will pull the distance far enough from o t − 1 and result in other options being assigned.
Revised: On the other hand, a significantly different state s t will pull the distance far enough from o t − 1 and result in other options being assigned.

Original: The effectiveness of the
Revised: The effectiveness of the

Original: Markovian master policy is also empirically addressed in Section 5.4.
Revised: Markovian master policy is also empirically addressed in Section 5.4.

Original:  L EARNING H I T-MDP S UNDER THE M AXIMUM E NTROPY F RAMEWORK
Revised:  L EARNING H I T-MDP S UNDER THE M AXIMUM E NTROPY F RAMEWORK

Original: In this section, we propose a stable and sample efficient
Revised: In this section, we propose a stable and sample efficient

Original: Maximum entropy Option Policy Gradient (MOPG) algorithm to learn HiT-MDPs.
Revised: Maximum entropy Option Policy Gradient (MOPG) algorithm to learn HiT-MDPs.

Original: Learning temporal abstraction like options has been a long standing challenge (Sutton et al., 1999; Givan et al., 2003; Kolobov et al., 2012; Bacon et al., 2017).
Revised: Learning temporal abstraction like options has been a long standing challenge (Sutton et al., 1999; Givan et al., 2003; Kolobov et al., 2012; Bacon et al., 2017).

Original: In theory (Sutton et al., 1999), options are not generally necessary for learning optimal policies since MDPs are sufficient enough.
Revised: In theory (Sutton et al., 1999), options are not generally necessary for learning optimal policies since MDPs are sufficient enough.

Original: Therefore, standard option learning algorithms optimizing Eq.
Revised: Therefore, standard option learning algorithms optimizing Eq.

Original: 1 often leads to sub-optimal results: either degenerate options (Harb et al., 2018)
Revised: 1 often leads to sub-optimal results: either degenerate options (Harb et al., 2018)

Original: (short execution time) switching back-and-forth frequently, or dominant options (Zhang & Whiteson, 2019)
Revised: (short execution time) switching back-and-forth frequently, or dominant options (Zhang & Whiteson, 2019)

Original: (long execution time) executing through the whole episode.
Revised: (long execution time) executing through the whole episode.

Original: Although there are various attempts (Harb et al., 2018; Hyun et al., 2019; Smith et al., 2018) tackling this issue, yet they often introduce bias which accumulates along the length of the trajectory and result in sub-optimal solutions.
Revised: Although there are various attempts (Harb et al., 2018; Hyun et al., 2019; Smith et al., 2018) tackling this issue, yet they often introduce bias which accumulates along the length of the trajectory and result in sub-optimal solutions.

Original: Recently maximum entropy reinforcement learning framework (Todorov, 2006; Ziebart et al., 2010;
Revised: Recently maximum entropy reinforcement learning framework (Todorov, 2006; Ziebart et al., 2010;

Original: Haarnoja et al., 2017; 2018) has been witnessed success in encouraging exploration and improving sample complexity.
Revised: Haarnoja et al., 2017; 2018) has been witnessed success in encouraging exploration and improving sample complexity.

Original: With the PGM of the HiT-MDP in hand, we tackle the above issues by deriving the Maximum entropy Option Policy Gradient (MOPG), a stable algorithm for learning diversified options and preventing degeneracy.
Revised: With the PGM of the HiT-MDP in hand, we tackle the above issues by deriving the Maximum entropy Option Policy Gradient (MOPG), a stable algorithm for learning diversified options and preventing degeneracy.

Original: Following the control as inference framework (Levine, 2018;
Revised: Following the control as inference framework (Levine, 2018;

Original: Haarnoja et al., 2018), we introduce the concept of “Optimality” (Todorov, 2006) into the HiT-MDP and formulate the conventional RL problems Eq.
Revised: Haarnoja et al., 2018), we introduce the concept of “Optimality” (Todorov, 2006) into the HiT-MDP and formulate the conventional RL problems Eq.

Original: 5 as a probabilistic inference problem (Kappen et al., 2012).
Revised: 5 as a probabilistic inference problem (Kappen et al., 2012).

Original: Specifically, we follow Levine (2018); Koller & Friedman (2009) and factorize Eq.into another PGM (as shown in Figure 2): q (¯ τ, e A 1: T , e O 1: T ) =
Revised: Specifically, we follow Levine (2018); Koller & Friedman (2009) and factorize Eq.into another PGM (as shown in Figure 2): [Equation]

Original: P ( s 0 ) q (¯ o 0 ) T (cid:89) t = q ( e At | s t , a t ) q ( e Ot | s t , a t , ¯ o t , ¯ o t − 1 ) P ( s t +1 | s t , a t ) q (¯ o t ) q ( a t ) ∝ P ( s 0 ) T (cid:89) t = P ( s t +1 | s t , a t ) (cid:124) (cid:123)(cid:122) (cid:125) Original Dynamics
Revised: Original Dynamics

Original: T (cid:89) t = q ( e At | s t , a t ) q ( e Ot | s t , a t , ¯ o t , ¯ o t − 1 ) (cid:124) (cid:123)(cid:122) (cid:125)
Revised: [Equation]

Original: Variational Terms , (9) where e ∈ { 0 , 1 } are observable binary “optimal random variables” (Levine, 2018).
Revised: Variational Terms [Equation] where e ∈ { 0 , 1 } are observable binary “optimal random variables” (Levine, 2018).

Original: The agent is optimal at time step t when q ( e At = 

Revised: The agent is optimal at time step t when q ( e At =

Original: | s t , a t ) and q ( e Ot = 1 | s t , a t , ¯ o t , ¯ o t − 1 ) (to keep notations uncluttered we use e t to denote e t = 1 .
Revised: | s t , a t ) and q ( e Ot = 1 | s t , a t , ¯ o t , ¯ o t − 1 ) (to keep notations uncluttered we use e t to denote e t = 1 .

Original: To simplify the derivation, priors q (¯ o ) and q ( a ) can be assumed to be uniform distributions without loss of generality (Levine, 2018).
Revised: To simplify the derivation, priors q (¯ o ) and q ( a ) can be assumed to be uniform distributions without loss of generality (Levine, 2018).

Original: Note that Eq.
Revised: Note that Eq.

Original: 9 shares the same state-action dynamics with Eq.
Revised: 9 shares the same state-action dynamics with Eq.

Original: With the optimal random variables e O and e A , the conditional probability of a state-action { s t , a t } pair that is optimal is defined as:
Revised: With the optimal random variables e O and e A , the conditional probability of a state-action { s t , a t } pair that is optimal is defined as:

Original: [Equation]
Revised: [Equation]

Original: which is an energy function that follows the Boltzmann distribution (Levine, 2018).
Revised: which is an energy function that follows the Boltzmann distribution (Levine, 2018).

Original: This specific design facilitates recovering the value function at the latter structural variational inference stage.
Revised: This specific design facilitates recovering the value function at the latter structural variational inference stage.

Original: Based on the same motivation, the conditional probability of an option-state-action [Equation] o t , s t , a t , o t − 1 } pair that is optimal is defined as,
Revised: Based on the same motivation, the conditional probability of an option-state-action { o t , s t , a t , o t − 1 } pair that is optimal is defined as,

Original: [Equation]
Revised: [Equation]

Original: where the mutual-information I [¯ o t | s t , a t , ¯ o t − 1 ] is chosen to be the energy function follows the
Revised: where the mutual-information I [¯ o t | s t , a t , ¯ o t − 1 ] is chosen to be the energy function follows the

Original: Boltzmann distribution.
Revised: Boltzmann distribution.

Original: This design choice arises from a fact that when the uniform prior assumption of q ( o ) is relaxed the optimization introduces a mutual-information as a regularizer in the Evidence Lower BOund (ELBO) (Proof in Appendix E).
Revised: This design choice arises from a fact that when the uniform prior assumption of q ( o ) is relaxed the optimization introduces a mutual-information as a regularizer in the Evidence Lower BOund (ELBO) (Proof in Appendix E).

Original: When optimal random variables are observed, substituting Eq. 10 & 11 into Eq.
Revised: When optimal random variables are observed, substituting Eq. 10 & 11 into Eq.

Original: 9, the conditional probability of any feasible trajectory ¯ τ is:
Revised: 9, the conditional probability of any feasible trajectory ¯ τ is:

Original: [Equation]
Revised: [Equation]

Original: where optimal random variables e 1: T are treated as observed random variables (evidences) and the trajectory ¯ τ is treated as latent variables.
Revised: where optimal random variables e 1: T are treated as observed random variables (evidences) and the trajectory ¯ τ is treated as latent variables.

Original: Therefore, the problem of fitting the optimal trajectory is equivalent to maximizing ELBO: log q [Equation] [ P [Equation]
Revised: Therefore, the problem of fitting the optimal trajectory is equivalent to maximizing ELBO: log q ( e 1: T ) ≥ − D KL [ P (¯ τ ) ||

Original: ¯ τ [Equation] || q 
Revised: q (¯ τ | e 1: T )]

Original: [Equation]

Original: ¯ τ | e 1: T )]

Original: (Proof in Appendix E).
Revised: (Proof in Appendix E).

Original: This observation immediately gives rise to a structural variational inference solution as keeping the system dynamix P ( s 0 ) (cid:81)
Revised: This observation immediately gives rise to a structural variational inference solution as keeping the system dynamix P ( s 0 ) (cid:81)

Original: Tt =1 P ( s t
Revised: Tt =1 P ( s t

Original: +1 | s t , a t ) in both Eq.
Revised: +1 | s t , a t ) in both Eq.

Original: 12 and Eq.
Revised: 12 and Eq.

Original: 5 fixed while optimizing the action and master policies in the variational distribution of Eq.
Revised: 5 fixed while optimizing the action and master policies in the variational distribution of Eq.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: (Markovian Option Policy Improvement Theorem).
Revised: (Markovian Option Policy Improvement Theorem).

Original: The problem of learning optimal action and master policies can be simplified as shrinking the KL-Divergence: D KL [ P (¯ τ ) ||
Revised: The problem of learning optimal action and master policies can be simplified as shrinking the KL-Divergence: D KL [ P (¯ τ ) ||

Original: q [Equation] T [Equation]
Revised: q (¯ τ | e 1: T )]

Original: [Equation]
Revised: [Equation]

Original: where H ( · ) denotes the entropy term.
Revised: where H ( · ) denotes the entropy term.

Original: Proof.
Revised: Proof.

Original: See Appendix E.
Revised: See Appendix E.

Revised: With policy evaluation and improvement theorems, we further prove the convergence of MOPG by proposing the Markovian Option Policy Iteration Theorem (Appendix E).

Original: Standard option frameworks (as shown in Eq.
Revised: Standard option frameworks (as shown in Eq.

Original: 1 and Eq. 5) only consider maximizing the value function and have no constraints on qualities of options, e.g., what good options should behave (Harb et al., 2018).
Revised: 1 and Eq. 5) only consider maximizing the value function and have no constraints on qualities of options, e.g., what good options should behave (Harb et al., 2018).

Original: As a result, previous researches often lead to suboptimal results.
Revised: As a result, previous researches often lead to suboptimal results.

Original: In Eq.
Revised: In Eq.

Original: 13, the mutual-information is introduced as an intrinsic reward to encourage consecutive executions of options and thus prevent degenerate options, while entropy terms encourage explorations of options thus prevent dominant options.
Revised: 44, the mutual-information is introduced as an intrinsic reward to encourage consecutive executions of options and thus prevent degenerate options, while entropy terms encourage explorations of options thus prevent dominant options.

Original: Following value functions from Section 3.2, we directly optimize ELBO with respect to the variational distribution and derive the Maximum entropy Options Policy Gradients (MOPG) theorems:
Revised: Following value functions from Section 3.2, we directly optimize ELBO with respect to the variational distribution and derive the Maximum entropy Options Policy Gradients (MOPG) theorems:

Original: Theorem 4.2.
Revised: Theorem 4.2.

Original: Master Policy Gradient Theorem:
Revised: Master Policy Gradient Theorem:

Original: Given a stochastic master policy which is differentiable w.r.t.
Revised: Given a stochastic master policy which is differentiable w.r.t.

Original: its parameter vector θ ¯ o , the gradient of the expected discounted return w.r.t.
Revised: its parameter vector θ ¯ o , the gradient of the expected discounted return w.r.t.

Original: θ ¯ o is:
Revised: θ ¯ o is:

Original: [Equation]
Revised: [Equation]

Original: 14) where Q O [ s t , ¯ o t ] =
Revised: ) where Q O [ s t , ¯ o t ] =

Original: E [ G t | s t , ¯ o t ] (Appendix D Eq. 28).
Revised: E [ G t | s t , ¯ o t ] (Appendix D Eq. 28).

Original: Proof.
Revised: Proof.

Original: See Appendix E
Revised: See Appendix E

Original: Theorem 4.3.
Revised: Theorem 4.3.

Original: Action Policy Gradient Theorem:
Revised: Action Policy Gradient Theorem:

Original: Given a stochastic action policy which is differentiable w.r.t.
Revised: Given a stochastic action policy which is differentiable w.r.t.

Original: its parameter vector θ a , the gradient of the expected discounted return w.r.t.
Revised: its parameter vector θ a , the gradient of the expected discounted return w.r.t.

Original: θ a is:
Revised: θ a is:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Appendix E.
Revised: See Appendix E.

Original: To keep notations uncluttered, we use θ ¯ o to denote master policy’s parameters
Revised: To keep notations uncluttered, we use θ ¯ o to denote master policy’s parameters

Original: π Oθ ¯ o
Revised: π O θ ¯ o

Original: and θ a to denote action policy’s parameters π Aθ a .
Revised: and θ a to denote action policy’s parameters π A θ a .

Original: The algorithm is summarized in Appendix F.
Revised: The algorithm is summarized in Appendix F.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we design experiments to answer five questions: (Q1)
Revised: In this section, we design experiments to answer five questions: (Q1)

Original: Whether MOPG can achieve better performance than other option variants and non-option baselines?
Revised: Whether MOPG can achieve better performance than other option variants and non-option baselines?

Original: (Q2)
Revised: (Q2)

Original: Does MOPG have a performance boost over other option variants in transfer learning settings?
Revised: Does MOPG have a performance boost over other option variants in transfer learning settings?

Original: (Q3)
Revised: (Q3)

Original: Is HiT-MDP interpretable?
Revised: Is HiT-MDP interpretable?

Original: (Q4)
Revised: (Q4)

Original: Whether MOPG can encourage consecutive executions of options while preventing dominant options problem?
Revised: Whether MOPG can encourage consecutive executions of options while preventing dominant options problem?

Original: For baselines, we follow DAC (Zhang & Whiteson, 2019)’s open source implementations and compare our algorithm with six baselines, five of which are option variants, i.e. , DAC+PPO, AHP+PPO (Levy & Shimkin, 2011), IOPG (Smith et al., 2018), PPOC (Klissarov et al., 2017), and OC (Bacon et al., 2017).
Revised: For baselines, we follow DAC (Zhang & Whiteson, 2019)’s open source implementations and compare our algorithm with six baselines, five of which are option variants, i.e. , DAC+PPO, AHP+PPO (Levy & Shimkin, 2011), IOPG (Smith et al., 2018), PPOC (Klissarov et al., 2017), and OC (Bacon et al., 2017).

Original: The non-option baseline is PPO (Schulman et al., 2017).
Revised: The non-option baseline is PPO (Schulman et al., 2017).

Original: All baselines’ parameters used by
Revised: All baselines’ parameters used by

Original: DAC remain unchanged other than the maximum number of training steps: MOPG only needsmillion steps to converge rather than the 2 million used in DAC.
Revised: DAC remain unchanged other than the maximum number of training steps: MOPG only needsmillion steps to converge rather than the 2 million used in DAC.

Original: For single task learning, experiments are conducted on all OpenAI Gym MuJoCo environments (10 environments) (Brockman et al., 2016a).
Revised: For single task learning, experiments are conducted on all OpenAI Gym MuJoCo environments (10 environments) (Brockman et al., 2016a).

Original: For transfer learning, we follow DAC and run 6 pairs of transfer learning tasks based on DeepMind
Revised: For transfer learning, we follow DAC and run 6 pairs of transfer learning tasks based on DeepMind

Original: Control Suite (Tassa et al., 2020) are plotted standard deviations are 
Revised: Control Suite (Tassa et al., 2020). Figures are plotted following DAC’s protocol: curves are averaged over 10 independent runs and smoothed by a sliding window of size 20. Shaded regions indicate standard deviations.

Original:  Core™ i 9900X
Revised: All experiments are run on an Intel® Core™ i9-9900X CPU @

Original: G
Revised: 3.50GHz with a single thread and process. Our implementation details are summarized in Appendix G. For a fair comparison, we follow DAC and use four options in all implementations. Our code is available in supplemental materials.

Original: 5. S INGLE -T ASK L EARNING (Q1)
Revised: 5. S INGLE -T ASK L EARNING (Q1)

Original: To answer Q1, we compare MOPG against five different option variants ( i.e. , DAC+PPO (Zhang &
Revised: To answer Q1, we compare MOPG against five different option variants ( i.e. , DAC+PPO (Zhang &

Original: Whiteson, 2019), AHP+PPO (Levy & Shimkin, 2011), IOPG (Smith et al., 2018), PPOC (Klissarov et al., 2017) and OC (Bacon et al., 2017)) and PPO (Schulman et al., 2017).
Revised: Whiteson, 2019), AHP+PPO (Levy & Shimkin, 2011), IOPG (Smith et al., 2018), PPOC (Klissarov et al., 2017) and OC (Bacon et al., 2017)) and PPO (Schulman et al., 2017).

Original: We observe that MOPG exhibits two different behaviors on infinite (Figure 3a) and finite (Figure 3b) horizon environments.
Revised: We observe that MOPG exhibits two different behaviors on infinite (Figure 3a) and finite (Figure 3b) horizon environments.

Original: Previous literatures (Klissarov et al., 2017; Smith et al., 2018; Harb et al., 2018; Zhang & Whiteson, 2019) find that option-based algorithms do not have advantages over hierarchy-free algorithms on single-task environments.
Revised: Previous literatures (Klissarov et al., 2017; Smith et al., 2018; Harb et al., 2018; Zhang & Whiteson, 2019) find that option-based algorithms do not have advantages over hierarchy-free algorithms on single-task environments.

Original: Despite this, MOPG can still achieve comparable performance with state of the arts (Figure 3b).
Revised: Despite this, MOPG can still achieve comparable performance with state of the arts (Figure 3b).

Original: More importantly, on infinite horizon environments (Figure 3a), MOPG’s performance significantly outperforms all baselines with respect to episodic return, convergence speed, variance between steps, and variance between 10 runs (Proposition 3.5).
Revised: More importantly, on infinite horizon environments (Figure 3a), MOPG’s performance significantly outperforms all baselines with respect to episodic return, convergence speed, variance between steps, and variance between 10 runs (Proposition 3.5).

Original: Since infinite and finite horizon environments are theoretically identical (Sutton & Barto, 2018), we do not have a theoretical explanation for this phenomenon.
Revised: Since infinite and finite horizon environments are theoretically identical (Sutton & Barto, 2018), we do not have a theoretical explanation for this phenomenon.

Original: In Appendix A, we conceptually explain that this might be because conventional value functions are insufficient to approximate environments in which hidden variables o only affect rewards but not states.
Revised: In Appendix A, we conceptually explain that this might be because conventional value functions are insufficient to approximate environments in which hidden variables o only affect rewards but not states.

Original: In conclusion, our experiment results show that MOPG is at least as effective as other option variants, but is significantly more sample efficient.
Revised: In conclusion, our experiment results show that MOPG is at least as effective as other option variants, but is significantly more sample efficient.

Original: Furthermore, it has significant advantages in infinite horizon environments.
Revised: Furthermore, it has significant advantages in infinite horizon environments.

Original: [Figure]

Original: 5. T RANSFER L EARNING (Q2)
Revised: 5. T RANSFER L EARNING (Q2)

Original: We run 6 pairs of transfer learning tasks based on DeepMind Control Suite (Tassa et al., 2020).
Revised: We run 6 pairs of transfer learning tasks based on DeepMind Control Suite (Tassa et al., 2020).

Original: Each pair contains two different tasks.
Revised: Each pair contains two different tasks.

Original: Following the setting in DAC (Zhang & Whiteson, 2019), we train all models one million steps on the first task and then switch to the second task to further train another one million steps.
Revised: Following the setting in DAC (Zhang & Whiteson, 2019), we train all models one million steps on the first task and then switch to the second task to further train another one million steps.

Original: Results are reported in Figure 4.
Revised: Results are reported in Figure 4.

Original: On the transfer learning task, MOPG’s  performance ranks the first in 5 out of 6 environments.
Revised: On the transfer learning task, MOPG’s [Figure] performance ranks the first in 5 out of 6 environments.

Original: This demonstrate MOPG’s advantages in knowledge reuse and its performance is at least comparable with other option variants.
Revised: This demonstrate MOPG’s advantages in knowledge reuse and its performance is at least comparable with other option variants.

Original: 5. I NTERPRETATION OF O PTION E MBEDDINGS (Q3)
Revised: 5. I NTERPRETATION OF O PTION E MBEDDINGS (Q3)

Original: Interpretability is a key property to apply RL agents in real-world applications. 
Revised: Interpretability is a key property to apply RL agents in real-world applications. Attention Option

Revised: Critic (AOC) (Chunduru & Precup, 2020) first introduces the attention mechanism into options.

Revised: In AOC, each option has a unique attention mask over state ¨ s =

Revised: ¨ W o ⊙ s and enables learning interpretable options with unique activation field (e.g., initiation set) over the observation space.

Revised: Unlike AOC, our implementation follows the Skill-Action (SA) architecture (Li et al., 2020) and the attention is introduced on the action space as a distance measure to learn option embeddings.

Original: One unique advantage of HiT-MDP is that hidden variables ¯ o are interpretable.
Revised: One unique advantage of HiT-MDP is that hidden variables ¯ o are interpretable.

Original: In the implementation, the master policy is defined as π O = P (¯ o t | s t , ¯ o t − 1 ; W o ) where W o =
Revised: In the implementation, the master policy is defined as π O = P (¯ o t | s t , ¯ o t − 1 ; W o ) where W o =

Original: [ˆ o 1 , ..., ˆ o K ] is an embedding matrix, ˆ o ∈ R
Revised: [ˆ o 1 , ..., ˆ o K ] is an embedding matrix, ˆ o ∈ R

Original: N is the embedding vector for an option with dimension N , and K is the total number of options ( N =and K = 4 in figure 5(a)).
Revised: N is the embedding vector for an option with dimension N , and K is the total number of options ( N = 40 and K = 4 in figure 5(a)).

Original: [-Since ¯ o is a one-hot vector, the embedding vector of an option is obtained via ˆ o t − 
-]
Revised: [-Since ¯ o is a one-hot vector, the embedding vector of an option is obtained via ˆ o t − 
-]

Original: =

Original: W o ¯ o t − 1 . More implementation details are described inAppendix G.
Revised: More implementation details are described in

Original:  Similar to word embeddings (Vaswani et al., 2017), option embeddings learn a semantic space of options with each dimension encodes a particular property and can be interpreted explicitly.
Revised: Appendix G. Similar to word embeddings (Vaswani et al., 2017), option embeddings learn a semantic space of options with each dimension encodes a particular property and can be interpreted explicitly.

Original: As in the capsule network (Sabour et al., 2017), we first infer each embedding dimension’s semantic by adding perturbations on to it ˆ o perturb [ i,j ] =
Revised: As in the capsule network (Sabour et al., 2017), we first infer each embedding dimension’s semantic by adding perturbations on to it ˆ o perturb [ i,j ] =

Original: ϵ
Revised: ϵ

Original: [ i ] + ˆ o [: ,j ] , where ϵ i ∈ [ − 0 .
Revised: [ i ] + ˆ o [: ,j ] , where ϵ i ∈ [ − 0 .

Original: 1 , − 0 .
Revised: 1 , − 0 .

Original: 09 , ..., 0 . 09] is an array ranging from − 0 .
Revised: 09 , ..., 0 . 09] is an array ranging from − 0 .

Original: 1 to 0 .
Revised: 1 to 0 .

Original: with an interval of 0 .
Revised: with an interval of 0 .

Original: 01 and j ∈ N is the j th dimension of the option embedding.
Revised: 01 and j ∈ N is the j th dimension of the option embedding.

Original: We then sample primary actions a [: ,j ] ∼ π A ( s t , o perturb [ i,j ] ) from perturbed options to observe how actions change along with perturbations.
Revised: We then sample primary actions a [: ,j ] ∼ π A ( s t , o perturb [ i,j ] ) from perturbed options to observe how actions change along with perturbations.

Original: For instance, Figure 5(b) visualizes how perturbations on dimension 0 of the option embedding affect the primary action.
Revised: For instance, Figure 5(b) visualizes how perturbations on dimension 0 of the option embedding affect the primary action.

Original: The Y-axis denotes actions defined in the HalfCheetah environment.
Revised: The Y-axis denotes actions defined by the HalfCheetah environment.

Original: It is clear that changes on Dim 0 have an opposite effect on the back leg and front leg: a larger valueon Dim 0 will assign the back leg a larger torque while  the front leg a smaller one, and vice versa.
Revised: The magnitude of Dim 0 have an opposite effect on the back leg and front leg: a larger value will increase the back leg’s torque while decrease the front leg’s, and vice versa.

Original: This means Dim 0 has a “focus point” property: it focuses torque on only one leg.
Revised: This means Dim 0 has a “focus point” property: it focuses torque on only one leg.

Original: Other dimensions in Figure 5(c) can be interpreted in the same manner.
Revised: Other dimensions in Figure 5(c) can be interpreted in the same manner.

Original: Once each dimension is understood, embeddings become straightforward to interpret by simply inspecting on which dimensions each embedding ˆ o (Figure 5(a)) has significant weight.
Revised: Once each dimension is understood, embeddings become straightforward to interpret by simply inspecting on which dimensions each embedding ˆ o (Figure 5(a)) has significant weight.

Original: Due to space limitations, more details and GIFs are provided in Appendix B.3.
Revised: Due to space limitations, more details and GIFs are provided in Appendix B.3.

Original: [Figure]
Revised: [Figure]

Original: 5. T EMPORAL E XTENSION (Q4)
Revised: 5. T EMPORAL E XTENSION (Q4)

Original: Because of the intrinsic reward and entropy terms in Eq. 13, one significant advantage ofthe MOPG over other algorithms is that the objective function automatically learns to balance between degenerate and dominant options.
Revised: One significant advantage of MOPG over other algorithms is that the intrinsic reward and entropy terms (Eq. 44) enable MOPG automatically learns to balance between degenerate and dominant options.

Original: In this subsection, we empirically demonstrate this in Figure(experiment results from HalfCheetah, more details in Appendix B.2).
Revised: In this subsection, we empirically demonstrate this in Figure 6 (experiment results from HalfCheetah, more details in Appendix B.2).

Original: At the start of training, all options’ durations are short, while Option 3’s duration quickly grows.
Revised: At the start of training, all options’ durations are short, while Option 3’s duration quickly grows.

Original: This proves that MOPG can indeed temporally extend an option.
Revised: This proves that MOPG can indeed temporally extend an option.

Original: More than temporal extension, as shown in the video 1 , MOPG learns to compose distinguishable options.
Revised: More than temporal extension, as shown in the video 1 , MOPG learns to compose distinguishable options.

Original: Option 3 (green background in the video) is a running forward skill thus it is executed most of the time.
Revised: Option 3 (green background in the video) is a running forward skill thus it is executed most of the time.

Original: Option 2 (blue background) is mainly used to recover from falling down thus its duration decreases with training.
Revised: Option 2 (blue background) is mainly used to recover from falling down thus its duration decreases with training.

Original: Therefore, the empirical study also shows that MOPG can compose disentangled options while preventing the dominant option problem.
Revised: Therefore, the empirical study also shows that MOPG can compose disentangled options while preventing the dominant option problem.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: To discover options automatically, Sutton et al.
Revised: To discover options automatically, Sutton et al.

Original: (1999) proposed Intra-option Q-learning to update the master Q value function at every time step.
Revised: (1999) proposed Intra-option Q-learning to update the master Q value function at every time step.

Original: However, all policies under this formulation are approximated implicitly using the Q-learning method.
Revised: However, all policies under this formulation are approximated implicitly using the Q-learning method.

Original: AHP (Levy & Shimkin, 2011) is proposed to unify the Semi-Markov process into an augmented Markov process and explicitly learn an “overall policy” by applying MDP-based policy gradient algorithms.
Revised: AHP (Levy & Shimkin, 2011) is proposed to unify the Semi-Markov process into an augmented Markov process and explicitly learn an “overall policy” by applying MDP-based policy gradient algorithms.

Original: However, their method for updating the master policy is still SMDP-style thus sample inefficient (Zhang & Whiteson, 2019).
Revised: However, their method for updating the master policy is still SMDP-style thus sample inefficient (Zhang & Whiteson, 2019).

Original: OC (Bacon et al., 2017) proposes a policy gradient based framework for explicitly learning intra-option policies and termination function s in an intra-option manner.
Revised: OC (Bacon et al., 2017) proposes a policy gradient based framework for explicitly learning intra-option policies and termination function s in an intra-option manner.

Original: However, for the master policy’s policy gradients learning, OC remains SMDP-style.
Revised: However, for the master policy’s policy gradients learning, OC remains SMDP-style.

Original: One closely related work is DAC (Zhang & Whiteson, 2019) which reformulates the option framework into two augmented MDPs.
Revised: One closely related work is DAC (Zhang & Whiteson, 2019) which reformulates the option framework into two augmented MDPs.

Original: Under this formulation, all policies can be modeled explicitly and learned in MDP-style.
Revised: Under this formulation, all policies can be modeled explicitly and learned in MDP-style.

Original: However, DAC is not theoretically proven to be homomorphic equivalent to the SMDP-Option.
Revised: However, DAC is not theoretically proven to be homomorphic equivalent to the SMDP-Option.

Original: Moreover, the reward function r ( τ ) used in Lemma 1 & 2 (Zhang & Whiteson, 2019) is undefined and makes the equivalence in proposition 1 (Zhang & Whiteson, 2019) does not hold.
Revised: Moreover, the reward function r ( τ ) used in Lemma 1 & 2 (Zhang & Whiteson, 2019) is undefined and makes the equivalence in proposition 1 (Zhang & Whiteson, 2019) does not hold.

Original: To the best of our knowledge, this is the first work proving
Revised: To the best of our knowledge, this is the first work proving

Original: MDPs’ equivalence to option-induced SMDPs under the homomorphic equivalence framework.
Revised: MDPs’ equivalence to option-induced SMDPs under the homomorphic equivalence framework.

Revised: Regarding network architectures, Attention Option Critic (AOC) (Chunduru & Precup, 2020) first introduces the attention mechanism into options.

Revised: Unlike AOC attends to the observation space, the attention mechanism employed in Section 5.3 attends to the action space.

Original: Regarding optimizing algorithms, Zhang & Whiteson (2019) pointed out that a large margin of the performance boost of DAC comes from the Proximal Policy Optimization (Schulman et al., 2017)
Revised: Regarding optimizing algorithms, Zhang & Whiteson (2019) pointed out that a large margin of the performance boost of DAC comes from the Proximal Policy Optimization (Schulman et al., 2017)

Original: (PPO).
Revised: (PPO).

Original: However, recent advances in probability inference based RL methods have also shown great success.
Revised: However, recent advances in probability inference based RL methods have also shown great success.

Original: Haarnoja et al.
Revised: Haarnoja et al.

Original: combined the idea of soft optimality (Haarnoja et al., 2017) from earlier control as inference frameworks (Todorov, 2006; Ziebart et al., 2010; Kappen et al., 2012) with the actor-critic architecture (Konda & Tsitsiklis, 2000) and proposed the state-of-the-art off-policy baseline SAC.
Revised: combined the idea of soft optimality (Haarnoja et al., 2017) from earlier control as inference frameworks (Todorov, 2006; Ziebart et al., 2010; Kappen et al., 2012) with the actor-critic architecture (Konda & Tsitsiklis, 2000) and proposed the state-of-the-art off-policy baseline SAC.

Original: Recent works show that the option framework trained under soft-optimality off-policy algorithms outperform on-policy methods.
Revised: Recent works show that the option framework trained under soft-optimality off-policy algorithms outperform on-policy methods (Wulfmeier et al., 2020).

Original: For instance, HO2 (Wulfmeier et al., 2020) employs a trust-region constrained off-policy algorithm and shows that it exhibits significant advantages over on-policy methods in both sample efficiency and performance.

Original: In this paper, we propose HiT-MDPs as a general MDPs which can be trained by both on-policy and off-policy algorithms. Although we propose MOPG as an on-policy policy gradient algorithm, the ELBO proposed in Section 4 can easily be extended to a SAC-like algorithm and remains to the future work.
Revised: HiT-MDPs are general MDPs which can be trained by both on-policy and off-policy algorithms: the ELBO proposed in Section 4 can easily be extended to a SAC-like algorithm and remains to the future work.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we propose the Hidden Temporal MDP (HiT-MDP), to the best of our knowledge, the first work proving option-induced MDP’ is homomorphic equivalent to the standard SMDP Option framework.
Revised: In this paper, we propose the Hidden Temporal MDP (HiT-MDP), to the best of our knowledge, the first work proving option-induced MDP’ is homomorphic equivalent to the standard SMDP Option framework.

Original: As for the learning algorithm, we propose the Maximum entropy Option Policy Gradients (MOPG) algorithm.
Revised: As for the learning algorithm, we propose the Maximum entropy Option Policy Gradients (MOPG) algorithm.

Original: Unlike conventional option learning algorithms that only maximize the value function without any constraint on the qualities of options, MOPG optimizes the Evidence Lower
Revised: Unlike conventional option learning algorithms that only maximize the value function without any constraint on the qualities of options, MOPG optimizes the Evidence Lower

Original: BOund (ELBO) with an intrinsic reward to prevent degenerate options and entropy terms to prevent dominant options.
Revised: BOund (ELBO) with an intrinsic reward to prevent degenerate options and entropy terms to prevent dominant options.

Original: It is also theoretically proven fitting optimal trajectories without adding any bias.
Revised: It is also theoretically proven fitting optimal trajectories without adding any bias.

Original: On challenging Mujoco (Todorov et al., 2012; Brockman et al., 2016b; Tunyasuvunakool et al., 2020) environments, thorough empirical results demonstrate that under widely used configurations, HiTMDP achieves competitive, if not better, performance compared to the state-of-the-art baselines on all finite horizon and transfer learning environments.
Revised: On challenging Mujoco (Todorov et al., 2012; Brockman et al., 2016b; Tunyasuvunakool et al., 2020) environments, thorough empirical results demonstrate that under widely used configurations, HiTMDP achieves competitive, if not better, performance compared to the state-of-the-art baselines on all finite horizon and transfer learning environments.

Original: Moreover, HiT-MDP significantly outperforms all baselines on infinite horizon environments while exhibiting smaller variance, faster convergence.
Revised: Moreover, HiT-MDP significantly outperforms all baselines on infinite horizon environments while exhibiting smaller variance, faster convergence.

Original: We also demonstrate hidden temporal variables in HiT-MDPs are interpretable, which is a key property for applying RL agents to real-world applications.
Revised: We also demonstrate hidden temporal variables in HiT-MDPs are interpretable, which is a key property for applying RL agents to real-world applications.

Original: A L EARNING O PTIONS AT M ULTI - LEVELS OF G RANULARITY
Revised: A L EARNING O PTIONS AT M ULTI - LEVELS OF G RANULARITY

Original: Implementations of the option framework share some common limitations.
Revised: Implementations of the option framework share some common limitations.

Original: When proposing the option framework, Sutton et al.
Revised: When proposing the option framework, Sutton et al.

Original: (1999) expected that learning at multi-level of temporal abstraction should be in favor of faster convergence and better exploration.
Revised: (1999) expected that learning at multi-level of temporal abstraction should be in favor of faster convergence and better exploration.

Original: On the contrary, significant improvements on single task environments have not been witnessed in most option implementations (Klissarov et al., 2017; Smith et al., 2018;
Revised: On the contrary, significant improvements on single task environments have not been witnessed in most option implementations (Klissarov et al., 2017; Smith et al., 2018;

Original: Harb et al., 2018; Zhang & Whiteson, 2019).
Revised: Harb et al., 2018; Zhang & Whiteson, 2019).

Original: To the best of our knowledge,
Revised: To the best of our knowledge,

Original: MOPG is the first option implementation in which these properties are significantly witnessed but only on infinite horizon environments.
Revised: MOPG is the first option implementation in which these properties are significantly witnessed but only on infinite horizon environments.

Original: In this section, we study this problem by explaning an intuition of why the value function is the main reason for this deficiency and how deep wide value functions could solve this problem.
Revised: In this section, we study this problem by explaning an intuition of why the value function is the main reason for this deficiency and how deep wide value functions could solve this problem.

Original: The expectation of improvements of the option framework on single task environment builds on an assumption that, by exploiting hierarchical action and state space, an agent’s searching space can be greatly reduced thus accelerates learning and improving exploration.
Revised: The expectation of improvements of the option framework on single task environment builds on an assumption that, by exploiting hierarchical action and state space, an agent’s searching space can be greatly reduced thus accelerates learning and improving exploration.

Original: However, as reported in section 5.4, most option frameworks suffer from “the dominant option problem” (Zhang & Whiteson, 2019) which prevents option frameworks from effectively learning hierarchy in action and state space as well as coordinating between options.
Revised: However, as reported in section 5.4, most option frameworks suffer from “the dominant option problem” (Zhang & Whiteson, 2019) which prevents option frameworks from effectively learning hierarchy in action and state space as well as coordinating between options.

Original: One intuition behind this problem is that conventional value functions V [ S t ] and Q [ S t , O t , A t ] make values depend on temporal latent variables indistinguishable (i.e. Although different options oand o 2 results to different values, such as V [ S t , O t − 1 = o 1 ] = 10 and V [ S t , O t − 1 =
Revised: One intuition behind this problem is that conventional value functions V [ S t ] and Q [ S t , O t , A t ] make values depend on temporal latent variables indistinguishable (i.e. Although different options oand o 2 results to different values, such as V [ S t , O t − 1 = o 1 ] = 10 and V [ S t , O t − 1 =

Original: o 2 ] = − 10 .
Revised: o 2 ] = − 10 .

Original: Because they arrive at the same state S t , they have identical values under conventional value function
Revised: Because they arrive at the same state S t , they have identical values under conventional value function

Original: V [ S t ] = 0 ).
Revised: V [ S t ] = 0 ).

Original: This deficiency makes option frameworks can only learn options at very coarse level thus fail to exploit hierarchical information.
Revised: This deficiency makes option frameworks can only learn options at very coarse level thus fail to exploit hierarchical information.

Original: The solution might be using a deep wide value function : enabling the framework to learn fine-grained options at mutli-levels of granularity (deep) and making value functions depend on latent variables with longer (wide) dependencies (e.g. V [ S t , O t − 1 ] and
Revised: The solution might be using a deep wide value function : enabling the framework to learn fine-grained options at mutli-levels of granularity (deep) and making value functions depend on latent variables with longer (wide) dependencies (e.g. V [ S t , O t − 1 ] and

Original: Q [ S t , O t , A t , O t − 1 ] ).
Revised: Q [ S t , O t , A t , O t − 1 ] ).

Original: To have a better understanding the importance of the deep wide value function, let us consider a simple environment which can be easily solved by Q [ s t , a t , a t − 1 ] but not Q
Revised: To have a better understanding the importance of the deep wide value function, let us consider a simple environment which can be easily solved by Q [ s t , a t , a t − 1 ] but not Q

Original: [ s t , a t ] .
Revised: [ s t , a t ] .

Original: Suppose we are training a robot which only has a camera sensor to cook thanksgiving turkey.
Revised: Suppose we are training a robot which only has a camera sensor to cook thanksgiving turkey.

Original: In this setting there are only two states: S = { Raw Turkey Image , Cooked Turkey Image } .
Revised: In this setting there are only two states: S = { Raw Turkey Image , Cooked Turkey Image } .

Original: The robot’s action space only consists of two actions: A = { Stuff turkey , Roast turkey } .
Revised: The robot’s action space only consists of two actions: A = { Stuff turkey , Roast turkey } .

Original: As for reward, if the robot roasted a stuffed turkey, then the reward is 10 .
Revised: As for reward, if the robot roasted a stuffed turkey, then the reward is 10 .

Original: However, if the robot roasted an un-stuffed turkey, then the reward is − 10 .
Revised: However, if the robot roasted an un-stuffed turkey, then the reward is − 10 .

Original: The stuff turkey action receives 0 reward.
Revised: The stuff turkey action receives 0 reward.

Original: The difficulty in this environment is, since the robot only has a camera to capture an image of the turkey, it can only observes either { Raw Turkey Image } or { Cooked Turkey Image } .
Revised: The difficulty in this environment is, since the robot only has a camera to capture an image of the turkey, it can only observes either { Raw Turkey Image } or { Cooked Turkey Image } .

Original: There is no way to look inside the turkey and see if the turkey is stuffed.
Revised: There is no way to look inside the turkey and see if the turkey is stuffed.

Original: Under this setting, a robot can never learn to first stuff a turkey and then roast it because Q [ Raw Turkey Image , Stuff Turkey ] =
Revised: Under this setting, a robot can never learn to first stuff a turkey and then roast it because Q [ Raw Turkey Image , Stuff Turkey ] =

Original: Q [ Raw Turkey Image , Roast Turkey ] = 0 .
Revised: Q [ Raw Turkey Image , Roast Turkey ] = 0 .

Original: Therefore, the robot can only randomly cook a turkey.
Revised: Therefore, the robot can only randomly cook a turkey.

Original: However, this problem can be easily solved by using a deep wide value function Q [ S t , A t , A t − 1 ] .
Revised: However, this problem can be easily solved by using a deep wide value function Q [ S t , A t , A t − 1 ] .

Original: The core problem in this setting is, action has no effect on states, it only affects rewards.
Revised: The core problem in this setting is, action has no effect on states, it only affects rewards.

Original: At the first glance this is a Partially Observed MDP (POMDP) problem since the state of whether the turkey is stuffed is un-observed.
Revised: At the first glance this is a Partially Observed MDP (POMDP) problem since the state of whether the turkey is stuffed is un-observed.

Original: This is true in all reinforcement learning settings without dependencies on latent variables.
Revised: This is true in all reinforcement learning settings without dependencies on latent variables.

Original: However, it goes much deeper in HRL settings.
Revised: However, it goes much deeper in HRL settings.

Original: In HRL, a common formulation is to estimate a latent variable O to encode hierarchical information and makes the policy depends on it P
Revised: In HRL, a common formulation is to estimate a latent variable O to encode hierarchical information and makes the policy depends on it P

Original: ( A t | S t , O t ) .
Revised: ( A t | S t , O t ) .

Original: Since O is a latent variable, it is highly likely that at state S t , different latent variable P ( A t | S t , O t
Revised: Since O is a latent variable, it is highly likely that at state S t , different latent variable P ( A t | S t , O t

Original: = o x ) and P ( A t | S t , O t
Revised: = o x ) and P ( A t | S t , O t

Original: = o y ) emits the same action A t =
Revised: = o y ) emits the same action A t =

Original: A 1 , and thus makes the conventional value function indistinguishable between o x and o y .
Revised: A 1 , and thus makes the conventional value function indistinguishable between o x and o y .

Original: This phenomenon is especially common around the switching time step of two options: around switching point, states are usually compatible with both old and new options.
Revised: This phenomenon is especially common around the switching time step of two options: around switching point, states are usually compatible with both old and new options.

Original: Conventional value functions will be especially confused at those moments.
Revised: Conventional value functions will be especially confused at those moments.

Original: This is exactly what we observed in Figure 5.3: overall, option 3 is executed consistently.
Revised: This is exactly what we observed in Figure 5.3: overall, option 3 is executed consistently.

Original: However, there are some random switches to option 2.
Revised: However, there are some random switches to option 2.

Original: And the randomization is increased between around switching time steps.
Revised: And the randomization is increased between around switching time steps.

Original: To explicitly show this, we visualized “Run4” into a video: https://www.youtube.com/shorts/M06BPqit7l4?
Revised: To explicitly show this, we visualized “Run4” into a video: https://www.youtube.com/shorts/M06BPqit7l4?

Original: feature=share .
Revised: feature=share .

Original: The option selection is very random at the beginning of the episode as well as around the switching point (the 16th second).
Revised: The option selection is very random at the beginning of the episode as well as around the switching point (the 16th second).

Original: These are exactly the most confusing moments of conventional value functions.
Revised: These are exactly the most confusing moments of conventional value functions.

Original: Thanks to proposition 3.4 and 3.5, one unbiased solution might be employing the deep wide value function: the higher the order of the MDPs, the smaller the variance will be.
Revised: Thanks to proposition 3.4 and 3.5, one unbiased solution might be employing the deep wide value function: the higher the order of the MDPs, the smaller the variance will be.

Original: We will explore this direction in our future work
Revised: We will explore this direction in our future work.

Original: [Table]

Original: B E XPERIMENTS R ESULTS B. P ERFORMANCE
Revised: B E XPERIMENTS R ESULTS B. P ERFORMANCE

Original: In this section we provide results for all ten OpenAI Gym Mujoco Environments (Brockman et al., 2016b; Todorov et al., 2012).
Revised: In this section we provide results for all ten OpenAI Gym Mujoco Environments (Brockman et al., 2016b; Todorov et al., 2012).

Original: Those environments can be classified into two categories: infinite horizon environments (i
Revised: Those environments can be classified into two categories: infinite horizon environments (i.e., HalfCheetah, Swimmer, HumanoidStandup and Reacher) and finite horizon environments (the other).

Original: [Table]
Revised: [Table]

Original: e., HalfCheetah, Swimmer, HumanoidStandup and Reacher) and finite horizon environments (the other).
Revised: HumanoidStandup Reacher PPO 2143. DAC+PPO 1830.

Original: [Table]
Revised: AHP+PPO 1701. PPOC 1441. OC 832. MOPG 3446.

Original: [Table]
Revised: [Table]

Original: B. T EMPORAL E XTENSION
Revised: T EMPORAL E XTENSION

Original: In Figure 6, we plot the average duration of each option during 430 training episodes (each episode contains a trajectory of 512 time steps) of the HalfCheetah environment.
Revised: In Figure 6, we plot the average duration of each option during 430 training episodes (each episode contains a trajectory of 512 time steps) of the HalfCheetah environment.

Original: In this environment, the agent learns to run half of a Cheetah by controlling 6 joints: back thigh, back shin, back foot, front thigh, front shin, and front foot.
Revised: In this environment, the agent learns to run half of a Cheetah by controlling 6 joints: back thigh, back shin, back foot, front thigh, front shin, and front foot.

Original: The faster the Cheetah runs forward, the higher return it gets from the environment.
Revised: The faster the Cheetah runs forward, the higher return it gets from the environment.

Original: At the start of training, all options’ durations are short.
Revised: At the start of training, all options’ durations are short.

Original: After the 100 -th episode,
Revised: After the 100 -th episode,

Original: Option 3’s duration quickly grows.
Revised: Option 3’s duration quickly grows.

Original: This answers the question Q4 in the main text that MOPG is able to temporally extending the execution of an option in the abscence of the termination function.
Revised: This answers the question Q4 in the main text that MOPG is able to temporally extending the execution of an option in the abscence of the termination function.

Original: [Figure]
Revised: [Figure]

Original: To illustrate how MOPG composes options, we take the HalfCheetah model trained after 1 million steps and independently run it 4 times (4 episodes.
Revised: To illustrate how MOPG composes options, we take the HalfCheetah model trained after 1 million steps and independently run it 4 times (4 episodes.

Original: each episode contains 512 time steps).
Revised: each episode contains 512 time steps).

Original: Option activation sequences of 4 runs are then plotted in Figure 7. 
Revised: Option activation sequences of 4 runs are then plotted in Figure 7. Figure

Original: As we can see that there are some common patterns between all 4 independent runs.
Revised: As we can see that there are some common patterns between all 4 independent runs.

Original: For example, all runs start with Option 1 and use Optionat the early stage.
Revised: For example, all runs start with Option 1 and use Option 2 at the early stage.

Original: After executing Option 2 for a short period, they all switch to Option 3 which has longest durations in all 4 runs.
Revised: After executing Option 2 for a short period, they all switch to Option 3 which has longest durations in all 4 runs.

Original: From time to time they will fall back to Option 2 for short periods and quickly switch to Option 3 again.
Revised: From time to time they will fall back to Option 2 for short periods and quickly switch to Option 3 again.

Original: This pattern of coordination indicates that because of the intrinsic reward and entropy terms in Eq. 13, one significant advantages of the MOPG over other algorithms is that the ELBO objective function automatically learns to balance between degenerate options and dominant options.
Revised: This pattern of coordination indicates that because of the intrinsic reward and entropy terms in Eq. 44, one significant advantages of the MOPG over other algorithms is that the ELBO objective function automatically learns to balance between degenerate options and dominant options.

Original: As shown in the video 2 , Option 3 (green background in the video) is a running forward option thus it is executed most of the time.
Revised: As shown in the video 2 , Option 3 (green background in the video) is a running forward option thus it is executed most of the time.

Original: Option 2 (blue background) is mainly used to recover from falling down thus its duration decreases with training.
Revised: Option 2 (blue background) is mainly used to recover from falling down thus its duration decreases with training.

Original: Option 2 and Optionhave completely different functionality.
Revised: Option 2 and Option 3 have completely different functionality.

Original: Therefore, empircal study also shows that MOPG is able to compose disentangled options while preventing dominant options problem.
Revised: Therefore, empircal study also shows that MOPG is able to compose disentangled options while preventing dominant options problem.

Revised: One interesting question to ask is whether the performance boost of MOPG comes from increasing or decreasing of option frequencies compared to other option variants.

Revised: In Figure 8 we plot one run of all option variants’ option duration.

Revised: As shown in Figure 8, it appears to be that there is not a significant difference in terms of option duration between our method and other option variants.

Revised: All of these methods are able to learn options which execute across various amount of time as well as composing those options.

Original: B. I NTERPRETATION OF O PTION C ONTEXT V ECTORS
Revised: It is also worth to mention that comparing with Figure 3, there is not an apparent monotonically relation between option changing frequency and performance. Therefore, on simple environments like HalfCheetah, main performance difference comes with sample efficiency of training intra-option policies. On the contrary, in harder environment like HumanoidStandupV2, there is a significant difference in terms of option composition between our method and other variants. As shown in Figure 3, all option variants except ours have a rather low score in HumanoidStandupV2.

Revised: As shown in Figure 9, all of these option variants tends to quickly and randomly switching between options.

Revised: This means that they failed to learn useful options which focus on specific sub-tasks with longer durations.

Revised: This is because that standard option frameworks have tendencies to learn either degenerate (Harb et al., 2018) options

Revised: (short execution time) that switching back-and-forth too frequently, or dominant options (Zhang &

Revised: Whiteson, 2019) (long execution time) that executing through the whole episode.

Revised: Both cases severely impair performance.

Revised: Figure 10 shows our method’s options composition in HumanoidStandupV2.

Revised: As shown in the figure that our method employs more options as the environment gets harder: it successfully learns 4 useful optons and has a clear options composition schedule between stages of standing up.

Revised: For option (a), the humanoid is lying around and trying to sit up.

Revised: For option (b) and option (c), these two options are more interchangeably used: option (b) is more sitting and swirling while option (c) is more trying to stand up and recover from failed trials.

Revised: After the humanoid stands up, the agent constantly call option (d) which is running around and trying to balance itself.

Revised: In conclusion, our method can learn distinguishable options and composition schedule in harder environment.

Revised: Because of MOPG’s maximum entropy framework, our method is able to adapt to environment’s complexity in an end-to-end manner and is more robust to degenerate and dominant options.

Revised: One last thing worth to mention is that the number of options used is not only related to complexity of the environment, but also the coefficient of entropy terms

Revised: Figure 11 shows a histogram of the magnitude of entropy weights and how many options are used in total 20 runs.

Revised: As we can see that as entropy weight increasing, more options are to be used since there is more randomness in option policy: when entropy weight is 0 .

Revised: 01 , all 20 runs only use 2 options; when entropy weight is 0 . 2 , 9 of 20 runs use 4 options while only 7 runs use 2 options.

Revised: Therefore, having an annealing training schedule of entropy weights might significantly boost performance of MOPG.

Revised: However, to demonstrate the robustness of our method we fixed the entropy weight to 0 .

Original:  In this section we continue with the HalfCheetah model used in Section B.2 and demonstrate how to interpret option context vectors as well as option activation sequences (Figure 7).
Revised: 01 in all experiments. B. I NTERPRETATION OF O PTION C ONTEXT V ECTORS In this section we continue with the HalfCheetah model used in Section B.2 and demonstrate how to interpret option context vectors as well as option activation sequences (Figure 7).

Original: In HalfCheetah, the agent learns to run half of a Cheetah by controlling 6 joints: back thigh, back shin, back foot, front thigh, front shin, and front foot.
Revised: In HalfCheetah, the agent learns to run half of a Cheetah by controlling 6 joints: back thigh, back shin, back foot, front thigh, front shin, and front foot.

Original: The faster the Cheetah runs forward, the higher return it gets from the environment.
Revised: The faster the Cheetah runs forward, the higher return it gets from the environment.

Original: We interpret option context vectors and activation patterns by first inspecting what property each dimension of the option context vector encodes (Figure 9).
Revised: We interpret option context vectors and activation patterns by first inspecting what property each dimension of the option context vector encodes (Figure 13).

Original: Once each dimension is understood, options (Figure 8) become straight forward to interpret by simply inspecting on which dimension (property) they have the most significant weights (Figure 10).
Revised: Once each dimension is understood, options (Figure 12) become straight forward to interpret by simply inspecting on which dimension (property) they have the most significant weights (Figure 14).

Original: These interpretations can further be taken to explain option activation patterns in Figure 7.
Revised: These interpretations can further be taken to explain option activation patterns in Figure 7.

Original: As the first step, we follow Sabour et al.
Revised: As the first step, we follow Sabour et al.

Original: (2017) to interpret what property each dimension of the option context vector in Figure 8 encodes by perturbing each dimension and decode perturbed option context vectors into primary actions.
Revised: (2017) to interpret what property each dimension of the option context vector in Figure 12 encodes by perturbing each dimension and decode perturbed option context vectors into primary actions.

Original: Specifically, we perturb one dimension by adding a range of perturbations [ − 0 .
Revised: Specifically, we perturb one dimension by adding a range of perturbations [ − 0 .

Original: 1 , 0 . 09] by intervals of 0 .
Revised: 1 , 0 . 09] by intervals of 0 .

Original: 01 onto it while keep the other dimensions fixed.
Revised: 01 onto it while keep the other dimensions fixed.

Original: After perturbation, each option context vector dimension has 20 perturbed vectors.
Revised: After perturbation, each option context vector dimension has 20 perturbed vectors.

Original: We then use the action policy decoder to decode all those vectors into primary actions and see how the perturbation affects the primary action.
Revised: We then use the action policy decoder to decode all those vectors into primary actions and see how the perturbation affects the primary action.

Original: As an illustration, we plot Dimension 0’s all 20 perturbed results in Figure 9.
Revised: As an illustration, we plot Dimension 4 ’s all 20 perturbed results in Figure 13.

Original: With visualization of perturbation results in hand, we can interpret what property each dimension encode by inspecting relationships between perturbations and primary actions.
Revised: With visualization of perturbation results in hand, we can interpret what property each dimension encode by inspecting relationships between perturbations and primary actions.

Original: In Figure 9, as an example, it is clear that changes on Dim 0 hasopposite effect on the back leg and front leg: a larger value on Dim 0 will assign the back leg a larger torque while the front leg a smaller one, and vice versa.
Revised: In Figure 13, as an example, it is clear that changes on Dim 4 has the same direction: as the magnitude of Dim 4 increase, all actuators move towards the same direction.

Original: Thismeans Dim 0 is has a focus point property: it focuses torque on only one leg.
Revised: This Dim can be seen as having an acceleration of running forward effect.

Original: Once we know how to interpret one dimension, we can move on to interpret the whole option context vector.
Revised: Once we know how to interpret one dimension, we can move on to interpret the whole option context vector.

Original: Since Option 1 and Option 2 are two main options employed in Figure 7, here we provide an example of how to interpret them.
Revised: Since Option 1 and Option 2 are two main options employed in Figure 7, here we provide an example of how to interpret them.

Original: Figure 8 shows that Option 1 has significant values on dimension 11 , 15 and 22 .
Revised: Figure 12 shows that Option 1 has significant values on dimension 11 , 15 and 22 .

Original: Option 2 is significant on dimension 2 , 5 and 36 .
Revised: Option 2 is significant on dimension 2 , 5 and 36 .

Original: We demonstrate these dimensions in the same manner as Figure 9 below:
Revised: We demonstrate these dimensions in the same manner as Figure 13 below:

Original: Subfigures in Figure 10 can be interpreted in the same manner as Figure 9.
Revised: Subfigures in Figure 14 can be interpreted in the same manner as Figure 13.

Original: As an example, from
Revised: As an example, from

Original: Figure 8 we can see that Option 1 has a significant small value on Dim 11 .
Revised: Figure 12 we can see that Option 1 has a significant small value on Dim 11 .

Original: In Figure 10, it shows that a smaller Dim 11 will twist the front leg forward and back foot forward while twist back thigh, back shin backward.
Revised: In Figure 14, it shows that a smaller Dim 11 will twist the front leg forward and back foot forward while twist back thigh, back shin backward.

Original: Composition of these movements is a back leg landing property.
Revised: Composition of these movements is a back leg landing property.

Original: Similarly, we can interpret that Dim 15 is a front leg landing property and Dim 22 is a balancing property.
Revised: Similarly, we can interpret that Dim 15 is a front leg landing property and Dim 22 is a balancing property.

Original: Therefore, Option 1 is focusing on landing from all positions.
Revised: Therefore, Option 1 is focusing on landing from all positions.

Original: Unlike other option context vectors which have apparent focusing dimensions, Option 2 has a rather balanced option context vector.
Revised: Unlike other option context vectors which have apparent focusing dimensions, Option 2 has a rather balanced option context vector.

Original: It has no apparently dominant dimension.
Revised: It has no apparently dominant dimension.

Original: It only has slightly more significant values on Dim 2 , 5 , 36 , which are focusing on jumping and running properties.
Revised: It only has slightly more significant values on Dim 2 , 5 , 36 , which are focusing on jumping and running properties.

Original: Therefore,
Revised: Therefore,

Original: Option 2 is more like an “all-weather” option: it is a option having very balanced properties with a slightly demonstration on running and jumping.
Revised: Option 2 is more like an “all-weather” option: it is a option having very balanced properties with a slightly demonstration on running and jumping.

Original: Interpretations of Option 1 and 2 above can then be taken to understand option activation patterns in Figure 7: as an all-weather option, Option 2 is the most frequently executed one and has the longest duration.
Revised: Interpretations of Option 1 and 2 above can then be taken to understand option activation patterns in Figure 7: as an all-weather option, Option 2 is the most frequently executed one and has the longest duration.

Original: From time to time, when the Cheetah needs to land and balance itself, Optionwill be executed.
Revised: From time to time, when the Cheetah needs to land and balance itself, Optionwill be executed.

Original: However, since landing option does not provide power of moving forward and thus has lower returns to continue, once the body is balanced the Cheetah will quickly stop Option 1 ’s execution and keep running with Option 2 .
Revised: However, since landing option does not provide power of moving forward and thus has lower returns to continue, once the body is balanced the Cheetah will quickly stop Option 1 ’s execution and keep running with Option 2 .

Original: B. T RANSFER L EARNING R ESULTS
Revised: B. T RANSFER L EARNING R ESULTS

Original: Below are statistics for Deepmind Control Suite (Tunyasuvunakool et al., 2020) transfer learning environments.
Revised: Below are statistics for Deepmind Control Suite (Tunyasuvunakool et al., 2020) transfer learning environments.

Original: [Table]
Revised: [Table]

Original: [Table] [Table] AHP+PPO 966.
Revised: PPO

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table] PPOC 942.
Revised: DAC+PPO

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: OC 106.

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: C A N MDP E QUIVALENCE TO THE SMDP O PTION F RAMEWORK
Revised: C A N MDP E QUIVALENCE TO THE SMDP O PTION F RAMEWORK

Original: In this section, we show that the the conventional Semi-Markov Decision Problem (SMDP) option framework which employs Markovian options has an option-induced MDP equivalence.
Revised: In this section, we show that the the conventional Semi-Markov Decision Problem (SMDP) option framework which employs Markovian options has an option-induced MDP equivalence.

Original: We first introduce the conventional SMDP formulated option framework (SMDP-Option).
Revised: We first introduce the conventional SMDP formulated option framework (SMDP-Option).

Original: In Section C. we follow Bishop (2006)’s method and formulate the dynamics of the SMDP-Option framework as an Hidden Markov Model (HMM) style Probability Graphical Model (PGM) (Koller & Friedman, 2009).
Revised: In Section C. we follow Bishop (2006)’s method and formulate the dynamics of the SMDP-Option framework as an Hidden Markov Model (HMM) style Probability Graphical Model (PGM) (Koller & Friedman, 2009).

Original: With the PGM and its conditional independence relationships (Chapter 8.2.1 (Bishop, 2006)) in hand, in Section C.3 we move on to propose the Hidden Temporal MDPs (HiT-MDP) and prove its equivalence to the SMDP-Option under the definition of homomorphic equivalence (Ravindran, 2003).
Revised: With the PGM and its conditional independence relationships (Chapter 8.2.1 (Bishop, 2006)) in hand, in Section C.3 we move on to propose the Hidden Temporal MDPs (HiT-MDP) and prove its equivalence to the SMDP-Option under the definition of homomorphic equivalence (Ravindran, 2003).

Original: To the best of our knowledge, this is the first work discovering the option framework’s MDP equivalence.
Revised: To the best of our knowledge, this is the first work discovering the option framework’s MDP equivalence.

Original: Following Bishop (2006)’s notation, we use A , B and C to denote three non-overlapping sets of arbitrarily many random variables.
Revised: Following Bishop (2006)’s notation, we use A , B and C to denote three non-overlapping sets of arbitrarily many random variables.

Original: Sets A and B are conditional independent on set C if
Revised: Sets A and B are conditional independent on set C if

Original: P ( A, B | C ) =
Revised: P ( A, B | C ) =

Original: P ( A | C ) P ( B | C ) , denoted as A ⊥⊥ B | C .
Revised: P ( A | C ) P ( B | C ) , denoted as A ⊥⊥ B | C .

Original: We mainly use head-to-tail conditional independence properties (Chapter 8.2.1 (Bishop, 2006)) in this section.
Revised: We mainly use head-to-tail conditional independence properties (Chapter 8.2.1 (Bishop, 2006)) in this section.

Original: C. T HE SMDP FORMULATED O PTION F RAMEWORK
Revised: C. T HE SMDP FORMULATED O PTION F RAMEWORK

Original: Sutton et al.
Revised: Sutton et al.

Original: (1999) proposed the option framework to demonstrate the temporal abstraction problem.
Revised: (1999) proposed the option framework to demonstrate the temporal abstraction problem.

Original: Following Bishop (2006)’s notation, we use bolded letter s ∈ S to denote a random variable and normal letter s to denote its realization.
Revised: Following Bishop (2006)’s notation, we use bolded letter s ∈ S to denote a random variable and normal letter s to denote its realization.

Original: Without special clarification, a random vector can have either a vector of continuous or discrete entries.
Revised: Without special clarification, a random vector can have either a vector of continuous or discrete entries.

Original: A scalar o ∈ Z denotes the index of an option where O ⊆ { 1 , 2 , . . .
Revised: A scalar o ∈ Z denotes the index of an option where O ⊆ { 1 , 2 , . . .

Original: , M } and M is the number of options.
Revised: , M } and M is the number of options.

Original: An Markovian option is a triple ( I o , P o ( a | s ) , P o ( b | s ))
Revised: An Markovian option is a triple ( I o , P o ( a | s ) , P o ( b | s ))

Original: in which I o ⊆
Revised: in which I o ⊆

Original: S is an initiation set where the option o can be initiated.
Revised: S is an initiation set where the option o can be initiated.

Original: P o ( a | s ) : S
Revised: P o ( a | s ) : S

Original: → A is the intra-option policy which maps environment states s ∈ S to an action vector a ∈ A .
Revised: → A is the intra-option policy which maps environment states s ∈ S to an action vector a ∈ A .

Original: P o ( b | s ) :
Revised: P o ( b | s ) :

Original: S → Z 2 is a termination function where b is a binary random variable.
Revised: S → Z 2 is a termination function where b is a binary random variable.

Original: It is used to determine whether to terminate (b = 1 ) the policy P
Revised: It is used to determine whether to terminate (b = 1 ) the policy P

Original: o ( a | s ) or not (b = 0 ).
Revised: o ( a | s ) or not (b = 0 ).

Original: Conventionally, β o = P o ( b = 1 | s ) .
Revised: Conventionally, β o = P o ( b = 1 | s ) .

Original: Since an option’s execution may persist over a variable period of time, a set of options’ execution together with its value functions constitutes a Semi-Markov Decision Problem (SMDP) (Puterman, 1994).
Revised: Since an option’s execution may persist over a variable period of time, a set of options’ execution together with its value functions constitutes a Semi-Markov Decision Problem (SMDP) (Puterman, 1994).

Original: When an old option is terminated, a new option will be sampled from the master policy (policy-over-options) o ∼ P ( o t +1 | s t +1 ) :
Revised: When an old option is terminated, a new option will be sampled from the master policy (policy-over-options) o ∼ P ( o t +1 | s t +1 ) :

Original: S → O .
Revised: S → O .

Original: A master policy π ( o | s ) =
Revised: A master policy π ( o | s ) =

Original: P ( o | s ) where o ∈ O is used to sample which option will be executed.
Revised: P ( o | s ) where o ∈ O is used to sample which option will be executed.

Original: Note that we use the bold-case o to denote unrealized random variables and the light-italic-case o to denote a realized instantiation.
Revised: Note that we use the bold-case o to denote unrealized random variables and the light-italic-case o to denote a realized instantiation.

Original: Conventionally, the execution of an option employs the call-and-return model (Sutton et al., 1999): at time step t , an agent either continues the previously executed option o t − 

Revised: Conventionally, the execution of an option employs the call-and-return model (Sutton et al., 1999): at time step t , an agent either continues the previously executed option o t −

Original: = o with probability P o ( b = 0 | s ) and sets o t =
Revised: = o with probability P o ( b = 0 | s ) and sets o t =

Original: o t − 

Revised: o t −

Original: = o , or terminates o with probability
Revised: = o , or terminates o with probability

Original: P o ( b = 1 | s ) and samples a new option o t from the master policy P ( o t
Revised: P o ( b = 1 | s ) and samples a new option o t from the master policy P ( o t

Original: | s t ) .
Revised: | s t ) .

Original: Therefore, the dynamics (stochastic process) of the option framework is written as:
Revised: Therefore, the dynamics (stochastic process) of the option framework is written as:

Original: [Equation]
Revised: [Equation]

Original: where τ = { s 0 , o 0 , a 0 , s 1 , o 1 , a 1 , . .
Revised: where τ = { s 0 , o 0 , a 0 , s 1 , o 1 , a 1 , . .

Original: . } denotes the trajectory of the option framework.
Revised: . } denotes the trajectory of the option framework.

Original: 1 is an indicator function and is only true when o t =
Revised: 1 is an indicator function and is only true when o t =

Original: o t − 1 (notice that o t − 1 is the realization at o t − 1 ).
Revised: o t − 1 (notice that o t − 1 is the realization at o t − 1 ).

Original: For clarity reasons, we use P o ( b = 1 | s ) instead of β o which is widely used in previous option literatures (e.g. (Sutton et al., 1999; Bacon et al., 2017)).
Revised: For clarity reasons, we use P o ( b = 1 | s ) instead of β o which is widely used in previous option literatures (e.g. (Sutton et al., 1999; Bacon et al., 2017)).

Original: Therefore, under this formulation the option framework is defined as a Semi-Markov process since the dependency on an activated option o can cross a variable amount of time (Sutton et al., 1999).
Revised: Therefore, under this formulation the option framework is defined as a Semi-Markov process since the dependency on an activated option o can cross a variable amount of time (Sutton et al., 1999).

Original: C. A N MDP FORMULATION OF THE O PTION F RAMEWORK
Revised: C. A N MDP FORMULATION OF THE O PTION F RAMEWORK

Original: We follow Bishop (2006)’s formulation of mixture distribution and Probabilistic Graphical Models (PGMs).
Revised: We follow Bishop (2006)’s formulation of mixture distribution and Probabilistic Graphical Models (PGMs).

Original: By introducing option variables as latent variables and adding extra dependencies into the termination function and master policy, we show that the conventional SMDP version of the option framework (Bacon et al., 2017; Sutton & Barto, 2018; Sutton et al., 1999; Harb et al., 2018; Zhang &
Revised: By introducing option variables as latent variables and adding extra dependencies into the termination function and master policy, we show that the conventional SMDP version of the option framework (Bacon et al., 2017; Sutton & Barto, 2018; Sutton et al., 1999; Harb et al., 2018; Zhang &

Original: Whiteson, 2019) can be re-formulated into an MDP formulation.
Revised: Whiteson, 2019) can be re-formulated into an MDP formulation.

Original: We first follow Bishop (2006)’s formulation of mixture distributions and redefine the option random variable o ∈ O = { 1 , 2 , . .
Revised: We first follow Bishop (2006)’s formulation of mixture distributions and redefine the option random variable o ∈ O = { 1 , 2 , . .

Original: , K } , which was originally defined as an integer index, but now as a K -dimensional one-hot vector ¯ o ∈ ¯ O = { 0 , 1 } K where K is the number of options, and each entry o ∈ { 0 , 1 } is a binary random variable.
Revised: , K } , which was originally defined as an integer index, but now as a K -dimensional one-hot vector ¯ o ∈ ¯ O = { 0 , 1 } K where K is the number of options, and each entry o ∈ { 0 , 1 } is a binary random variable.

Original: P (¯ o t | s t ) denotes the probability distribution over one-hot vector ¯ o at time step t conditioned on state s t .
Revised: P (¯ o t | s t ) denotes the probability distribution over one-hot vector ¯ o at time step t conditioned on state s t .

Original: P (¯ o t =
Revised: P (¯ o t =

Original: o t | s t ) denotes a probability entry (a scalar value) of the random variable
Revised: o t | s t ) denotes a probability entry (a scalar value) of the random variable

Original: ¯ o t with a realization at time step t where o t = 1 and o ∈ ¯ o t / o t = 0 .
Revised: ¯ o t with a realization at time step t where o t = 1 and o ∈ ¯ o t / o t = 0 .

Original: In figure 12, s ∈ S , ¯ o ∈ ¯ O K , b ∈ B K and a ∈ A , denote the state, option, termination and action random variable respectively.
Revised: In figure 16, s ∈ S , ¯ o ∈ ¯ O K , b ∈ B K and a ∈ A , denote the state, option, termination and action random variable respectively.

Original: ¯ o is an K -dimensional one-hot vector and b is an K -dimensional binary vector where each entry b ∈ { 0 , 1 } .
Revised: ¯ o is an K -dimensional one-hot vector and b is an K -dimensional binary vector where each entry b ∈ { 0 , 1 } .

Original: K is the number of options.
Revised: K is the number of options.

Original: R t +1 is the actual reward received from the environment after executing action a t in state s t .
Revised: R t +1 is the actual reward received from the environment after executing action a t in state s t .

Original: G t =
Revised: G t =

Original: R t +1 + γR t +2 + γ 2 R t +3 · · · is the discounted expected return where γ ∈ R is a discount factor.
Revised: R t +1 + γR t +2 + γ 2 R t +3 · · · is the discounted expected return where γ ∈ R is a discount factor.

Original: The termination policy distribution P ( b t | s t , ¯ o t − 1 ) : S × ¯ O → B can be formulated as a mixture distribution 3 conditioned on option vector (the one-hot vector)
Revised: The termination policy distribution P ( b t | s t , ¯ o t − 1 ) : S × ¯ O → B can be formulated as a mixture distribution 3 conditioned on option vector (the one-hot vector)

Original: ¯ o t − 1 and state s t .
Revised: ¯ o t − 1 and state s t .

Original: [Equation]
Revised: [Equation]

Original: Because each option has its own termination policy P o ( b | s ) , with a slightly abuse of notation, in equation (17) we use P ( b t | s t , ¯ o t − 1 ) to denote the termination policy activated at time step t by previous chosen option ¯ o t − 1 .
Revised: Because each option has its own termination policy P o ( b | s ) , with a slightly abuse of notation, in equation (17) we use P ( b t | s t , ¯ o t − 1 ) to denote the termination policy activated at time step t by previous chosen option ¯ o t − 1 .

Original: To keep notation uncluttered, we use β t =
Revised: To keep notation uncluttered, we use β t =

Original: P ( b t = 1 | s t , ¯ o t − 1 ) to denote the probability of option ¯ o t − 1 terminates at time step t and (1 − β t ) =
Revised: P ( b t = 1 | s t , ¯ o t − 1 ) to denote the probability of option ¯ o t − 1 terminates at time step t and (1 − β t ) =

Original: P ( b t = 

Revised: P ( b t =

Original: | s t , ¯ o t − 1 ) to denote the probability of continuation.
Revised: | s t , ¯ o t − 1 ) to denote the probability of continuation.

Original: Conventionally, master policy (Zhang & Whiteson, 2019) (also called “policy-over-options” (Sutton et al., 1999; Bacon et al., 2017))) is defined as:
Revised: Conventionally, master policy (Zhang & Whiteson, 2019) (also called “policy-over-options” (Sutton et al., 1999; Bacon et al., 2017))) is defined as:

Original: [Equation]
Revised: [Equation]

Original: Similarly, we propose a novel mixture master policy as a mixture distribution 4 :
Revised: Similarly, we propose a novel mixture master policy as a mixture distribution 4 :

Original: [Equation]
Revised: [Equation]

Original: where P (¯ o t | ¯ o t − 1 ) is a degenerated probability distribution (Puterman, 1994)
Revised: where P (¯ o t | ¯ o t − 1 ) is a degenerated probability distribution (Puterman, 1994)

Original: [Equation]
Revised: [Equation]

Original: As shown in equation (19), the master policy only exists when b t = 1 the option terminates.
Revised: As shown in equation (19), the master policy only exists when b t = 1 the option terminates.

Original: Therefore,
Revised: Therefore,

Original: PPOC (Klissarov et al., 2017) uses inaccurate gradients for updating the master policy during an option’s execution.
Revised: PPOC (Klissarov et al., 2017) uses inaccurate gradients for updating the master policy during an option’s execution.

Original: According to the conditional dependency relationships in PGM (figure 12), the joint probability distribution of ¯ o t and b t can be written as:
Revised: According to the conditional dependency relationships in PGM (figure 16), the joint probability distribution of ¯ o t and b t can be written as:

Original: [Equation]
Revised: [Equation]

Original: and the marginal probability distribution can be written as:
Revised: and the marginal probability distribution can be written as:

Original: [Equation]
Revised: [Equation]

Original: The intra-option (action) policy distribution can also be formulated as a mixture distribution
Revised: The intra-option (action) policy distribution can also be formulated as a mixture distribution

Original: [Equation]
Revised: [Equation]

Original: Therefore, the dynamics of the PGM in figure 12 can be written as:
Revised: Therefore, the dynamics of the PGM in figure 16 can be written as:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where P (¯ τ ) =
Revised: where P (¯ τ ) =

Original: P ( s 0 , ¯ o 0 , a 0 , s 1 , b 1 , ¯ o 1 , a 1 , . . . )
Revised: P ( s 0 , ¯ o 0 , a 0 , s 1 , b 1 , ¯ o 1 , a 1 , . . . )

Original: denotes the joint distribution of the PGM.
Revised: denotes the joint distribution of the PGM.

Original: Notice that under this formulation, P ( τ ) is actually an HMM with s t , a t as observable random variables and b t , ¯ o t as latent variables.
Revised: Notice that under this formulation, P ( τ ) is actually an HMM with s t , a t as observable random variables and b t , ¯ o t as latent variables.

Original: It is worth to mention that equation (20) is essentially the indicator function 1 ¯ o t =
Revised: It is worth to mention that equation (20) is essentially the indicator function 1 ¯ o t =

Original: ¯ o t − 1 used in conventional SMDP option framework papers and the last line in equation (22) is identical to transitional probability distribution in their formulation.
Revised: ¯ o t − 1 used in conventional SMDP option framework papers and the last line in equation (22) is identical to transitional probability distribution in their formulation.

Original: However, as we show in this section, by adding latent variables ¯ o t − 1 and introducing the dependency between ¯ o t and b t , our formulation is essentially an HMM.
Revised: However, as we show in this section, by adding latent variables ¯ o t − 1 and introducing the dependency between ¯ o t and b t , our formulation is essentially an HMM.

Original: It opens the door to introduce many well developed PGM algorithms such as message passing (Forney, 1973) and variational inference (Hoffman et al., 2013) to the reinforcement learning framework.
Revised: It opens the door to introduce many well developed PGM algorithms such as message passing (Forney, 1973) and variational inference (Hoffman et al., 2013) to the reinforcement learning framework.

Original: As we show below, the nice conditional independence relationships enjoyed by this model also enable us to prove the equivalence between the option framework’s SMDP and MDP formulation.
Revised: As we show below, the nice conditional independence relationships enjoyed by this model also enable us to prove the equivalence between the option framework’s SMDP and MDP formulation.

Original: C. H IDDEN T EMPORAL MDP S (H I T-MDP S )
Revised: C. H IDDEN T EMPORAL MDP S (H I T-MDP S )

Original: In this section we propose the Hidden Temporal MDPs (HiT-MDPs) and prove its equivalence under the definition of homomorphic equivalence.
Revised: In this section we propose the Hidden Temporal MDPs (HiT-MDPs) and prove its equivalence under the definition of homomorphic equivalence.

Original: Following notations from Section C.2, the
Revised: Following notations from Section C.2, the

Original: Hidden Temporal MDPs (HiT-MDPs) family can be described by a tuple ¯ M = { ¯ S , ¯ A , r, P, ϕ, γ } where ¯ S .
Revised: Hidden Temporal MDPs (HiT-MDPs) family can be described by a tuple ¯ M = { ¯ S , ¯ A , r, P, ϕ, γ } where ¯ S .

Original: = S
Revised: = S

Original: × ¯ O is an augmented state space, ¯ A .
Revised: × ¯ O is an augmented state space, ¯ A .

Original: =
Revised: =

Original: A × ¯ O is an augmented action space, and
Revised: A × ¯ O is an augmented action space, and

Original: ϕ = P (¯ o t | ¯ s t )
Revised: ϕ = P (¯ o t | ¯ s t )

Original: =
Revised: =

Original: P (¯ o t | s t , ¯ o t − 1 ) is the emit function for hidden variables.
Revised: P (¯ o t | s t , ¯ o t − 1 ) is the emit function for hidden variables.

Original: The joint distribution of
Revised: The joint distribution of

Original: HiT-MDPs is:
Revised: HiT-MDPs is:

Original: [Equation]
Revised: [Equation]

Original: Notice that the Markovian master policy P (¯ o t | s t , ¯ o t − 1 ) is actually the marginalization over the termination variable b t in Eq.
Revised: Notice that the Markovian master policy P (¯ o t | s t , ¯ o t − 1 ) is actually the marginalization over the termination variable b t in Eq.

Original: 24: P (¯ o t | s t , ¯ o t − 1 ) = (cid:80) b t P ( b t | s t , ¯ o t − 1 ) P (¯ o t | b t , s t , ¯ o t − 1 ) .
Revised: 24: P (¯ o t | s t , ¯ o t − 1 ) = (cid:80) b t P ( b t | s t , ¯ o t − 1 ) P (¯ o t | b t , s t , ¯ o t − 1 ) .

Original: The joint distribution is factorized from Eq.
Revised: The joint distribution is factorized from Eq.

Original: 25 to Eq.
Revised: 25 to Eq.

Original: by following conditional independences in the
Revised: by following conditional independences in the

Original: PGM (Figure 13):
Revised: PGM (Figure 17):

Original: We define a linear function ¯ f (¯ o ) =
Revised: We define a linear function ¯ f (¯ o ) =

Original: ¯ o · d T : ¯ O → O which maps ¯ o to o , where d = [1 , 2 , . . .
Revised: ¯ o · d T : ¯ O → O which maps ¯ o to o , where d = [1 , 2 , . . .

Original: , K ] T is a K -dimensional constant integer vector, and hence ¯ f (¯ o )
Revised: , K ] T is a K -dimensional constant integer vector, and hence ¯ f (¯ o )

Original: = o .
Revised: = o .

Original: Note that ¯ f is a Bijection since it is a linear function defined on a finite integer space.
Revised: Note that ¯ f is a Bijection since it is a linear function defined on a finite integer space.

Original: We define a tuple of partition function ¯
Revised: We define a tuple of partition function ¯

Original: B = < ¯ f, ¯ I, ¯ I, ¯ f > where ¯ I is the identical function and is also a Bijection.
Revised: B = < ¯ f, ¯ I, ¯ I, ¯ f > where ¯ I is the identical function and is also a Bijection.

Original: Therefore, the partition function ¯ B is a tuple of Bijection functions ¯ B (¯ s , ¯ a ) =
Revised: Therefore, the partition function ¯ B is a tuple of Bijection functions ¯ B (¯ s , ¯ a ) =

Original: ¯ B ( { ¯ o t − 1 , s t , a t , ¯ o t } ) =
Revised: ¯ B ( { ¯ o t − 1 , s t , a t , ¯ o t } ) =

Original: { o t − 1 , s t , a t , o t } , which maps ¯ τ to τ , where ¯ τ = { s 0 , ¯ o 0 , a 0 , s 1 , ¯ o 1 , a 1 , . . .
Revised: { o t − 1 , s t , a t , o t } , which maps ¯ τ to τ , where ¯ τ = { s 0 , ¯ o 0 , a 0 , s 1 , ¯ o 1 , a 1 , . . .

Original: } is the trajectory of the HiT-MDP .
Revised: } is the trajectory of the HiT-MDP .

Original: Therefore, by following the Partition Function ¯ B , the dynamics P ( τ ) of the SMDP-Option in Eq. 24 under the
Revised: Therefore, by following the Partition Function ¯ B , the dynamics P ( τ ) of the SMDP-Option in Eq. 24 under the

Original: Bijection ¯ B is equivalent to the dynamics P (¯ τ ) of the HiT-MDP P ( τ/ ¯ B ) ≡ P (¯ τ/ ¯ B ) .
Revised: Bijection ¯ B is equivalent to the dynamics P (¯ τ ) of the HiT-MDP P ( τ/ ¯ B ) ≡ P (¯ τ/ ¯ B ) .

Original: As for the reward function r , notice that from the PGM we have the conditional independence r t +1 ⊥⊥ ¯ o t | a t .
Revised: As for the reward function r , notice that from the PGM we have the conditional independence r t +1 ⊥⊥ ¯ o t | a t .

Original: Therefore, SMDP-Option and the HiT-MDP also share the same expected reward function r t +1 ( s t , a t , o t ) =
Revised: Therefore, SMDP-Option and the HiT-MDP also share the same expected reward function r t +1 ( s t , a t , o t ) =

Original: E [ r t +1 | s t , a t , o t ] = E [ r t +1 | s t , a t ] =
Revised: E [ r t +1 | s t , a t , o t ] = E [ r t +1 | s t , a t ] =

Original: r t +1 ( s t , a t ) .
Revised: r t +1 ( s t , a t ) .

Original: Therefore, the
Revised: Therefore, the

Original: SMDP-based option framework has an MDP-based equivalence:
Revised: SMDP-based option framework has an MDP-based equivalence:

Original: Theorem C.1.
Revised: Theorem C.1.

Original: By the definition of Bisimulation Relation, the SMDP-based option framework, which employs Markovian options, has an underlying MDP equivalence because:
Revised: By the definition of Bisimulation Relation, the SMDP-based option framework, which employs Markovian options, has an underlying MDP equivalence because:

Original: P ( τ/ ¯ B ) ≡ P (¯ τ/ ¯ B ) and ¯ B is a Bijection.
Revised: P ( τ/ ¯ B ) ≡ P (¯ τ/ ¯ B ) and ¯ B is a Bijection.

Original: r ( τ/ ¯ B ) ≡ r (¯ τ/ ¯ B ) .
Revised: r ( τ/ ¯ B ) ≡ r (¯ τ/ ¯ B ) .

Original: Although the reward function r is equivalent, in Section D we identify a problem that under the configuration of HiT-MDPs, the conventional value function V [ s t ] does not yield a Bellman equation.
Revised: Although the reward function r is equivalent, in Section D we identify a problem that under the configuration of HiT-MDPs, the conventional value function V [ s t ] does not yield a Bellman equation.

Original: We have to propose a Markovian Option-Value Function V
Revised: We have to propose a Markovian Option-Value Function V

Original: [ s t , ¯ o t − 1 ] to solve this problem and prove that it is equivalent to V [ s t ] .
Revised: [ s t , ¯ o t − 1 ] to solve this problem and prove that it is equivalent to V [ s t ] .

Original: This is non-trivial since compared to the SMDP-Option , the
Revised: This is non-trivial since compared to the SMDP-Option , the

Original: MDP formulation introduces extra dependencies on ¯ o .
Revised: MDP formulation introduces extra dependencies on ¯ o .

Original: In Section D.2, by exploiting conditional independencies we prove that they do have the same expected return under the Bijection ¯ B .
Revised: In Section D.2, by exploiting conditional independencies we prove that they do have the same expected return under the Bijection ¯ B .

Original: D P ROOF OF THE O PTIMAL V ALUE E QUIVALENCE
Revised: D P ROOF OF THE O PTIMAL V ALUE E QUIVALENCE

Original: In this Section we derive value functions of HiT-MDPs and prove its optimal value equivalence to the option framework.
Revised: In this Section we derive value functions of HiT-MDPs and prove its optimal value equivalence to the option framework.

Original: Following the structure of Section C, we first reformulate value functions of
Revised: Following the structure of Section C, we first reformulate value functions of

Original: SMDP-Option into an MDP formulation by following the dynamics in Section C.2.
Revised: SMDP-Option into an MDP formulation by following the dynamics in Section C.2.

Original: We then following the HiT-MDPs’ dynamics proposed in Section C.3 and derive a Markovian Option-Value function of
Revised: We then following the HiT-MDPs’ dynamics proposed in Section C.3 and derive a Markovian Option-Value function of

Original: HiT-MDPs, and prove it converges to the optimal value.
Revised: HiT-MDPs, and prove it converges to the optimal value.

Original: In the end we prove that MDP-Option is optimal value equivalent to the SMDP-Option.
Revised: In the end we prove that MDP-Option is optimal value equivalent to the SMDP-Option.

Original: D. T HE MDP F ORMULATED O PTION F RAMEWORK ’ S V ALUE F UNCTIONS
Revised: D. T HE MDP F ORMULATED O PTION F RAMEWORK ’ S V ALUE F UNCTIONS

Original: With dynamics proposed in C.2, we now derive the MDP formulated value functions for the option framework (Bacon et al., 2017; Sutton et al., 1999).
Revised: With dynamics proposed in C.2, we now derive the MDP formulated value functions for the option framework (Bacon et al., 2017; Sutton et al., 1999).

Original: We follow Sutton & Barto (2018)’s notation in this section and write value functions for MDP below:
Revised: We follow Sutton & Barto (2018)’s notation in this section and write value functions for MDP below:

Original: [Equation]
Revised: [Equation]

Original: where V [ s t ] is the state value function(Sutton & Barto, 2018) and Q O [ o t , s t ] is the option value function(Bacon et al., 2017; Sutton et al., 1999), G t = (cid:80) t r t +1 is the return.
Revised: where V [ s t ] is the state value function(Sutton & Barto, 2018) and Q O [ o t , s t ] is the option value function(Bacon et al., 2017; Sutton et al., 1999), G t = (cid:80) t r t +1 is the return.

Original: Note that in deriving equation (27) we only use summation rule and production rule, the conditional dependency relationships in PGM (figure 12) are not used.
Revised: Note that in deriving equation (27) we only use summation rule and production rule, the conditional dependency relationships in PGM (figure 16) are not used.

Original: The option value function Q O [ o t , s t ] can be further expanded as:
Revised: The option value function Q O [ o t , s t ] can be further expanded as:

Original: [Equation]
Revised: [Equation]

Original: where Q U [ o t , s t , a t ] is the option-action value function.
Revised: where Q U [ o t , s t , a t ] is the option-action value function.

Original: Proposition D.1.1.
Revised: Proposition D.1.1.

Original: MDP formulation has identical state value function V [ s t ] and option value function Q O [ o t , s t ] to SMDP formulations
Revised: MDP formulation has identical state value function V [ s t ] and option value function Q O [ o t , s t ] to SMDP formulations

Original: Proof.
Revised: Proof.

Original: Note that in derivations above we only use summation and production rules.
Revised: Note that in derivations above we only use summation and production rules.

Original: Both equation (27) and (28) are identical to the conventional SMDP option framework.
Revised: Both equation (27) and (28) are identical to the conventional SMDP option framework.

Original: From now on, we will continue derivations with conditonal independence relationships encoded in
Revised: From now on, we will continue derivations with conditonal independence relationships encoded in

Original: PGM (Chapter 8.2.1 (Bishop, 2006)).
Revised: PGM (Chapter 8.2.1 (Bishop, 2006)).

Original: We have following conditional independence relationships from PGM (figure 12):
Revised: We have following conditional independence relationships from PGM (figure 16):

Original: [Equation]
Revised: [Equation]

Original: With above conditional independence relationships in hand, we now show that the MDP formulation has identical value functions to the conventional SMDP formulation(Sutton et al., 1999; Bacon et al., 2017).
Revised: With above conditional independence relationships in hand, we now show that the MDP formulation has identical value functions to the conventional SMDP formulation(Sutton et al., 1999; Bacon et al., 2017).

Original: Proposition D.1.2.
Revised: Proposition D.1.2.

Original: MDP formulation has identical option-action value function Q U [ o t , s t , a t ] to
Revised: MDP formulation has identical option-action value function Q U [ o t , s t , a t ] to

Original: SMDP formulations
Revised: SMDP formulations

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: by definition of G
Revised: by definition of G

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: use eq 30 31 and
Revised: use eq 30 31 and

Original: [Equation]
Revised: [Equation]

Original: However, unlike conventional value functions V [ s t ] , Q O [ o t , s t ] , Q U [ o t , s t , a t ] of the option framework derived in Section D.1, the value function U [ s t +1 , ¯ o t ] has an extra dependency on the hidden variable ¯
Revised: However, unlike conventional value functions V [ s t ] , Q O [ o t , s t ] , Q U [ o t , s t , a t ] of the option framework derived in Section D.1, the value function U [ s t +1 , ¯ o t ] has an extra dependency on the hidden variable ¯

Original: o t .
Revised: o t .

Original: Therefore, conventional value function V [ s t ] does not yield a Bellman equation under the configuration of HiT-MDPs.
Revised: Therefore, conventional value function V [ s t ] does not yield a Bellman equation under the configuration of HiT-MDPs.

Original: To tackle this issue, we propose the Markovian Option-Value Function ¯
Revised: To tackle this issue, we propose the Markovian Option-Value Function ¯

Original: V (derivations of Eq.
Revised: V (derivations of Eq.

Original: (6) in the main text):
Revised: (6) in the main text):

Original: [Equation]
Revised: [Equation]

Original: where from line 2 to line 3 we use the conditional independence property in PGM that G t ⊥⊥ ¯ o t − 1 |{ s t , ¯ o t } .
Revised: where from line 2 to line 3 we use the conditional independence property in PGM that G t ⊥⊥ ¯ o t − 1 |{ s t , ¯ o t } .

Original: Proposition D.1.3.
Revised: Proposition D.1.3.

Original: MDP formulation has identical option-value function upon arrival U
Revised: MDP formulation has identical option-value function upon arrival U

Original: [ s t
Revised: [ s t

Original: +1 , o t ] to SMDP formulations
Revised: +1 , o t ] to SMDP formulations

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: from line 3 to line 4 use equation (29) and (32).
Revised: from line 3 to line 4 use equation (29) and (32).

Original: From line 4 to line 5 use equation (22) and definition of Q O .
Revised: From line 4 to line 5 use equation (22) and definition of Q O .

Original: The second last line use equation (27).
Revised: The second last line use equation (27).

Original: The last line use the definition of advantage function
Revised: The last line use the definition of advantage function

Original: A .
Revised: A .

Original: Under our MDP formulation, we also propose proposition D.1.4.
Revised: Under our MDP formulation, we also propose proposition D.1.4.

Original: We derive our gradient theorems based on equation (38) in section E. This important relationship largely simplify derivations than the original paper (Bacon et al., 2017) as well as give rise to the the HiT-MDP.
Revised: We derive our gradient theorems based on equation (38) in section E. This important relationship largely simplify derivations than the original paper (Bacon et al., 2017) as well as give rise to the the HiT-MDP.

Original: Proposition D.1.4.
Revised: Proposition D.1.4.

Original: The option-value function upon arrival U [ s t +1 , o t ] is an expectation over option value function Q O [ o t +1 , s t +1 ] conditioned on previous option
Revised: The option-value function upon arrival U [ s t +1 , o t ] is an expectation over option value function Q O [ o t +1 , s t +1 ] conditioned on previous option

Original: O t
Revised: O t

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Following proof of proposition D.1.3,
Revised: Following proof of proposition D.1.3,

Original: U [ s t +1 , o t ] = (cid:88)
Revised: U [ s t +1 , o t ] = (cid:88)

Original: [Equation]
Revised: [Equation]

Original: D.
Revised: D.

Original: T HE O PTIMAL V ALUE E QUIVALENCE BETWEEN H I T-MDP AND O PTION F RAMEWORKS
Revised: T HE O PTIMAL V ALUE E QUIVALENCE BETWEEN H I T-MDP AND O PTION F RAMEWORKS

Original: Theorem D.1.
Revised: Theorem D.1.

Original: The option-action value function Eq.
Revised: The option-action value function Eq.

Original: 7 satisfies the Bellman Operator T H
Revised: 7 satisfies the Bellman Operator T H

Original: T H Q A [ s t , ¯ o t , a t ] =
Revised: T H Q A [ s t , ¯ o t , a t ] =

Original: E [ G t | s t , ¯ o t , a t ] =
Revised: E [ G t | s t , ¯ o t , a t ] =

Original: r ( s, a ) + γ (cid:88) s t +
Revised: r ( s, a ) + γ (cid:88) s t +

Original: P ( s
Revised: P ( s

Original: [Equation]
Revised: [Equation]

Original: +1 | s t
Revised: +1 | s t

Original: [Equation]
Revised: [Equation]

Original: where the Markovian option-value function given by Eq. 6.
Revised: where the Markovian option-value function given by Eq. 6.

Original: Proof.
Revised: Proof.

Original: Following dynamics derived in Section C.3, we can define value functions on HiT-MDPs as (derivations of Eq.
Revised: Following dynamics derived in Section C.3, we can define value functions on HiT-MDPs as (derivations of Eq.

Original: (8) in the main text):
Revised: (8) in the main text):

Original: [Equation]
Revised: [Equation]

Original: where from line 2 to line 3 we use the conditional independence property in PGM that R t +1 ⊥⊥ ¯ o t | a t ,
Revised: where from line 2 to line 3 we use the conditional independence property in PGM that R t +1 ⊥⊥ ¯ o t | a t ,

Original: G t +1 ⊥⊥ s t |{ s t +1 , ¯ o t } and G t +1 ⊥⊥ a t | s t +1 .
Revised: G t +1 ⊥⊥ s t |{ s t +1 , ¯ o t } and G t +1 ⊥⊥ a t | s t +1 .

Original: γ ∈ R is a discounting factor.
Revised: γ ∈ R is a discounting factor.

Original: The last line uses the definition of the Markovian option value function (Eq. 35).
Revised: The last line uses the definition of the Markovian option value function (Eq. 35).

Original: Theorem D.2.
Revised: Theorem D.2.

Original: (Markovian Option Policy Evaluation Theorem).
Revised: (Markovian Option Policy Evaluation Theorem).

Original: Assume that throughout our computation the Q A [ · , · ] and ¯ V [ · ] are bounded and A < ∞ , the sequence Q kA defined by Q k
Revised: Assume that throughout our computation the Q A [ · , · ] and ¯ V [ · ] are bounded and A < ∞ , the sequence Q kA defined by Q k

Original: +1 A
Revised: +1 A

Original: [Equation]
Revised: [Equation]

Original: H Q kA will converge to the option-action value function Q π A A as k → ∞ .
Revised: H Q kA will converge to the option-action value function Q π A A as k → ∞ .

Original: Proof.
Revised: Proof.

Original: As with the standard convergence results for policy evaluation (Sutton & Barto, 2018), by the definition of T H (Eq.
Revised: As with the standard convergence results for policy evaluation (Sutton & Barto, 2018), by the definition of T H (Eq.

Original: 8) the option-action value function Q π A A is a fixed point.
Revised: 8) the option-action value function Q π A A is a fixed point.

Original: To prove the T H is a contraction, define a norm on V -values functions V and U
Revised: To prove the T H is a contraction, define a norm on V -values functions V and U

Original: [Equation]
Revised: [Equation]

Original: where ¯ s = { s, o } .
Revised: where ¯ s = { s, o } .

Original: By recurssively apply the Hidden Temporal Bellman Operator T H
Revised: By recurssively apply the Hidden Temporal Bellman Operator T H

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: by applying Eq.
Revised: by applying Eq.

Original: 41 to V
Revised: 41 to V

Original: and U we have:
Revised: and U we have:

Original: [Equation]
Revised: [Equation]

Original: Therefore, T H is a contraction and by the fixed point theorem, Theorem
Revised: Therefore, T H is a contraction and by the fixed point theorem, Theorem

Original: [Equation]
Revised: [Equation]

Original: 3 follows immediately
Revised: 3 follows immediately

Original: [Equation]
Revised: [Equation]

Original: Proposition D.2.1.
Revised: Proposition D.2.1.

Original: The option-induced Markovian Option-Value Function ¯ V is equivalent to the conventional value function V
Revised: The option-induced Markovian Option-Value Function ¯ V is equivalent to the conventional value function V

Original: Proof.
Revised: Proof.

Original: for Proposition 3.4: By law of total expectation:
Revised: for Proposition 3.4: By law of total expectation:

Original: [Equation]
Revised: [Equation]

Original: thus V [ s t , ¯ o t − 1 ] is an unbiased estimator of V [ s t ] , with conditional independences defined in PGM 13.
Revised: thus V [ s t , ¯ o t − 1 ] is an unbiased estimator of V [ s t ] , with conditional independences defined in PGM 17.

Original: Proposition D.2.2.
Revised: Proposition D.2.2.

Original: The option-induced Markovian Option-Value Function ¯ V has smaller variance than the conventional value function V
Revised: The option-induced Markovian Option-Value Function ¯ V has smaller variance than the conventional value function V

Original: Proof.
Revised: Proof.

Original: for Proposition 3.5: By law of total conditional variance:
Revised: for Proposition 3.5: By law of total conditional variance:

Original: [Equation]
Revised: [Equation]

Original: with conditional independences defined in PGM
Revised: with conditional independences defined in PGM

Original: E M AX - ENTROPY OPTION POLICY GRADIENTS
Revised: E M AX - ENTROPY OPTION POLICY GRADIENTS

Original: In the main text Section 4 we solve the HiT-MDPs under the maximum entropy framework (Levine, 2018).
Revised: In the main text Section 4 we solve the HiT-MDPs under the maximum entropy framework (Levine, 2018).

Original: Here we proof the Evidence Lower BOund (ELBO) proposed in the main text and derive the
Revised: Here we proof the Evidence Lower BOund (ELBO) proposed in the main text and derive the

Original: Maximum entropy Option Policy Gradient (MOPG) theorems.
Revised: Maximum entropy Option Policy Gradient (MOPG) theorems.

Original: E. T HE M AXIMUM E NTROPY O BJECTIVE F UNCTION
Revised: E. T HE M AXIMUM E NTROPY O BJECTIVE F UNCTION

Original: Following notations defined in the main text Section 4, optimality variables e 1: T ( e denotes the realization of e when e = 1 ) are treated as observed variables while random variables from the trajectory ¯ τ = { s , a , ¯ o , ... } are treated as latent variables.
Revised: Following notations defined in the main text Section 4, optimality variables e 1: T ( e denotes the realization of e when e = 1 ) are treated as observed variables while random variables from the trajectory ¯ τ = { s , a , ¯ o , ... } are treated as latent variables.

Original: The variational Evidence Lower BOund (ELBO) is given by:
Revised: The variational Evidence Lower BOund (ELBO) is given by:

Original: Theorem E.1.
Revised: Theorem E.1.

Original: The problem of learning optimal action and master policies can be simplified as shrinking the KL-Divergence: D KL [ P (¯ τ [Equation] ||q (¯
Revised: The problem of learning optimal action and master policies can be simplified as shrinking the KL-Divergence: D KL [ P (¯ τ ) ||

Original: [Equation] | e 1: T )]
Revised: q (¯ τ | e 1: T )]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: where third last line to the second last line follows from the Jensen’s inequality and P (¯ τ ) is the dynamics of the HiT-MDPs defined in Eq. 
Revised: where third last line to the second last line follows from the Jensen’s inequality and P (¯ τ ) is the dynamics of the HiT-MDPs defined in Eq. Theorem E.2.

Revised: (Markovian Option Policy Iteration Theorem).

Revised: Repeated application of Markovian Option evaluation and improvement to any π O,A ∈ (cid:81) converges to a policy π ∗ O,A such that

Revised: Q π ∗ O,A ( ¯ s t , ¯ a t ) ≥ Q π O,A ( ¯ s t , ¯ a t ) for all π O,A ∈ (cid:81) and ¯ s t , ¯ a t ∈ ¯ S × ¯ A , assuming | ¯ A | < ∞ .

Original: E. P ROOF FOR THE MASTER POLICY GRADIENT THEOREM
Revised: Proof. In this proof we use π A for illustration, the proof of π O follows the same derivation. From Theorem 4.1 we have that:

Revised: [Equation]

Revised: Therefore, it must be the case that E a t ∼ π newA [ r

Revised: [Equation]

Revised: s

Revised: [Equation]

Revised: + I

Revised: [Equation]

Revised: + H

Revised: [Equation]

Revised: π oldA

Revised: [Equation]

Revised: ] ≥ E a

Revised: [Equation]

Revised: ∼ π oldA [ r ( s t , a t

Revised: [Equation]

Revised: +

Revised: I [ o t ] + H ( π oldA )] =

Revised: ¯ V π oldA [ s , ¯ o ] , since we can always choose π newA = π oldA ∈ (cid:81) .

Revised: Substituting this inequality into Theorem 3.2 leads to:

Revised: [Equation]

Revised: where the convergence to Q π newA A follows from the Markovian Option Policy Evaluation Theorem D.2.

Revised: Therefore, the iteration of the sequence Q π iA A is monotonically increasing and we get Q π ∗ A A > Q π A A for all ( s , o , a ) ∈ S

Revised: × O × A .

Revised: E. P ROOF FOR THE MASTER POLICY GRADIENT THEOREM

Original: Proof.
Revised: Proof.

Revised: [Equation]

Revised: ] ∂θ

Revised: [Equation]

Revised: o

Revised: [Equation]

Revised: [Equation]

Revised: o

Revised: [Equation]

Revised: o

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: o

Revised: [Equation]

Revised: [Equation]

Revised: t ,

Revised: [Equation]

Revised: o t

Revised: [Equation]

Revised: 1 ] ∂θ o =

Revised: [Equation]

Revised: ¯ o

Revised: [Equation]

Revised: o

Revised: [Equation]

Revised: o

Revised: [Equation]

Revised: ∂θ o

Revised: Q O [

Revised: [Equation]

Revised: t ,

Revised: [Equation]

Revised: o t ]

Revised: [Equation]

Revised: [Equation]

Revised: o ′ |

Revised: [Equation]

Revised: , a

Revised: [Equation]

Revised: o )

Revised: [Equation]

Revised: H (

Revised: [Equation]

Revised: Oθ ¯ o )

Revised: [Equation]

Revised: ¯ o t

Revised: P (¯ o t |

Revised: [Equation]

Revised: t , ¯ o t

Revised: [Equation]

Revised: [Equation]

Revised: O [

Revised: [Equation]

Revised: t , ¯ o t ] ∂θ o

Revised: [Equation]

Revised: ¯ o t

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: o t

Revised: [Equation]

Revised: 1 ) ∂θ o

Revised: Q O [

Revised: [Equation]

Revised: t ,

Revised: [Equation]

Revised: o t ]

Revised: [Equation]

Revised: |

Revised: [Equation]

Revised: a

Revised: [Equation]

Revised: o )

Revised: [Equation]

Revised: [Equation]

Revised: ¯ o )

Revised: [Equation]

Revised: s t +1 , ¯ o t

Revised: [Equation]

Revised: [Equation]

Revised: o

Revised: [Equation]

Revised: |

Revised: [Equation]

Revised: [Equation]

Revised: o

Revised: [Equation]

Revised: 1 ) ∂V [

Revised: [Equation]

Revised: 1 , ¯ o

Revised: [Equation]

Revised: ] ∂θ o = −

Revised: [Equation]

Revised: (cid:88)

Revised: [Equation]

Revised: [Equation]

Revised: s t

Revised: [Equation]

Revised: k ,

Revised: [Equation]

Revised: o t

Revised: [Equation]

Revised: k −

Revised: P

Revised: [Equation]

Revised: [Equation]

Revised: o

Revised: [Equation]

Revised: o t

Revised: [Equation]

Revised: [Equation]

Revised: ¯ o t + k ∂P

Original: [Equation]
Revised: [Equation]

Original: E. P ROOF FOR THE ACTION POLICY GRADIENT THEOREM
Revised: E. P ROOF FOR THE ACTION POLICY GRADIENT THEOREM

Original: Proof.
Revised: Proof.

Original: Similar to the first equation above [Equation] continue expanding gradients of ∂Q O ∂θ a by equations (6) (28) and (8):
Revised: Similar to the first equation above, continue expanding gradients of ∂Q O ∂θ a by equations (6) (28) and (8):

Original: [Equation]
Revised: [Equation]

Original: F L EARNING A LGORITHM FOR MOPG
Revised: F L EARNING A LGORITHM FOR MOPG

Original: 1 Initialize the option embedding matrix W S 2 Assign Initial State: s t ← s3 Assign Initial Option: ˆ o t − 1 ← ˆ o 5 while Converge do # Rollout trajectories and store in replay buffer repeat
Revised: 1 Initialize the option embedding matrix W S 2 Assign Initial State: s t ← s3 Assign Initial Option: ˆ o t − 1 ← ˆ o 5 while Converge do # Rollout trajectories and store in replay buffer repeat

Original: Retrieve the option context vector ˆ o t − 1 = W TS · ˆ o t −
Revised: Retrieve the option context vector ˆ o t − 1 = W TS · ˆ o t −

Original: Sample ˆ o t ∼ P (ˆ o t | s t , ˆ o t − 1 )
Revised: Sample ˆ o t ∼ P (ˆ o t | s t , ˆ o t − 1 )

Original: Retrieve the option context vector ˆ o t = W TS · ˆ o t
Revised: Retrieve the option context vector ˆ o t = W TS · ˆ o t

Original: Sample a t ∼ P ( a t | s t , ˆ o t )
Revised: Sample a t ∼ P ( a t | s t , ˆ o t )

Original: Compute Q O [ s t , ˆ o t ] and V [ s t , ˆ o t − 1 ]
Revised: Compute Q O [ s t , ˆ o t ] and V [ s t , ˆ o t − 1 ]

Original: Take action a t in s t , observe new state s t +1 and reward R t +  until Rollout Length Reached  # Compute Advantages for option & action policies
Revised: Take action a t in s t , observe new state s t +1 and reward R t +  until Rollout Length Reached  # Compute Advantages for option & action policies

Original: Assign t reversely, from RolloutLength − 1 to repeat
Revised: Assign t reversely, from RolloutLength − 1 to repeat

Original: Compute option Advantage
Revised: Compute option Advantage

Original: A Ot = R t +1 + γ ( V [ s t
Revised: A Ot = R t +1 + γ ( V [ s t

Original: +1 , ˆ o t ] − V [ s t , ˆ o t − 1 ] + ∆ I + ∆ H O ) + γλA
Revised: +1 , ˆ o t ] − V [ s t , ˆ o t − 1 ] + ∆ I + ∆ H O ) + γλA

Original: Ot +
Revised: Ot +

Original: Compute action Advantage
Revised: Compute action Advantage

Original: A At = R t +1 + γ ( Q O
Revised: A At = R t +1 + γ ( Q O

Original: [ s t +1 , ˆ o t +1 ]
Revised: [ s t +1 , ˆ o t +1 ]

Original: − Q O [ s t , ˆ o t ] + ∆ H A ) + γλA
Revised: − Q O [ s t , ˆ o t ] + ∆ H A ) + γλA

Original: At +  until Rollout Length Reached  # λ is the GAE coefficient used in PPO.
Revised: At +  until Rollout Length Reached  # λ is the GAE coefficient used in PPO.

Original: # Optimize PPO Obj while i < PPO Optimization Epochs do θ o ← PPO ( ∂P ( o ′ | s ′ , o )
Revised: # Optimize PPO Obj while i < PPO Optimization Epochs do θ o ← PPO ( ∂P ( o ′ | s ′ , o )

Original: ∂θ o , A O ) θ a ← PPO ( ∂P ( a | s , o )
Revised: ∂θ o , A O ) θ a ← PPO ( ∂P ( a | s , o )

Original: ∂θ a , A A ) end 29 end
Revised: ∂θ a , A A ) end 29 end

Original: G I MPLEMENTATION D ETAILS G. 
Revised: G I MPLEMENTATION D ETAILS G.

Original: N EURAL N ETWORK
Revised: N EURAL N ETWORK

Original: A RCHITECTURE
Revised: A RCHITECTURE

Original: [Figure]
Revised: Interpretability is a key property to apply RL agents in real-world applications.

Original: 𝑾 𝑶 𝑶# 𝒕#𝟏 C 𝑪𝒐𝒏𝒄𝒂𝒕[𝑺 𝒕 ,𝑶# 𝒕#𝟏 ] [Figure] 𝑺 𝒕
Revised: Attention Option Critic (AOC) (Chunduru & Precup, 2020) first introduces the attention mechanism into options.

Original: [Figure] Query Key [Figure] 𝑾 𝑶
Revised: In

Original: Add & [Figure] Forward Add & Norm
Revised: AOC, each option has a unique attention mask over state ¨ s = ¨ W o ⊙ s .

Original: Linear CategoricalDistribution 𝑷(𝑶’ 𝒕 |𝑺 𝒕 ,𝑶’ 𝒕"𝟏 ) Option Policy 𝑶# 𝒕 C 𝑪𝒐𝒏𝒄𝒂𝒕[𝑺 𝒕 ,𝑶# 𝒕 ]
Revised: Therefore, each option has a unique activation field (e.g., initiation set) over the observation space.

Original: State 𝑺 𝒕 Feed Forward GaussianDistribution 𝑷(𝑨 𝒕 |𝑺 𝒕 ,𝑶( 𝒕 ) Action Policy
Revised: Options learned by AOC are also interpretable since each option attends to different context in the state vector.

Original: [Figure]
Revised: [Figure]

Original: 𝑸 𝑶
Revised: Our methods contributes to the option framework from a different aspect.

Original: [𝑺 𝒕 ,𝑶# 𝒕 ] Critic Feed Forward 𝑨 𝒕 𝑶0 𝒕 𝑶0 𝒕#
Revised: In AOC, an option is still a tuple of ( I , β, π A ) while in our work an option is an embedding vector ˆ o .

Original: Clustering
Revised: The attention mechanism is used as a distance measure to compare which option is closer to the concatenated state-option pair (as shown in Eq. 48).

Original: As explained in the main text Section 5.3, options in HiTMDP can be learned as option embeddings.
Revised: As explained in the main text Section 5.3, options in HiT-MDP can be learned as option embeddings.

Original: Inspired by the Transformer (Vaswani et al., 2017), in this section we implement the HiT-MDP as a simple yet effective Multi-Head Attention (MHA) (Vaswani et al., 2017) based Encoder-Decoder architecture as shown in Figure 14.
Revised: Inspired by the Transformer (Vaswani et al., 2017), in this section we implement the HiT-MDP as a simple yet effective Multi-Head Attention (MHA) (Vaswani et al., 2017) based Encoder-Decoder architecture as shown in Figure 18.

Original: Specifically, an attention mechanism is described as the mapping from a query q ∈ R E and a set of key-value pairs, i.e., K ∈ R M × E and V ∈ R M × E ( M and E are total number of options and embedding dimensions), to an output:
Revised: Specifically, an attention mechanism is described as the mapping from a query q ∈ R E and a set of key-value pairs, i.e., K ∈ R M × E and V ∈ R M × E ( M and E are total number of options and embedding dimensions), to an output:

Original: [Equation]
Revised: [Equation]

Original: A Multi-Head Attention MHA ( q , K , V ) is a linear projection of h (number of heads) concatenated linearly projected Attention outputs:
Revised: A Multi-Head Attention MHA ( q , K , V ) is a linear projection of h (number of heads) concatenated linearly projected Attention outputs:

Original: [Equation]
Revised: [Equation]

Original: where head i =
Revised: where head i =

Original: Attention ( q W qi , KW Ki , V W Vi ) where projections are parameter matrices W qi ∈ R E 
Revised: Attention [Equation] where projections are parameter matrices W qi ∈ R E × E , W Ki ∈ R E

Original: × E , WKi ∈ R E × E , W Vi ∈ R E × E , W Oi ∈
Revised: × E , W Vi ∈ R E × E , W Oi ∈

Original: R hE × E .
Revised: R hE × E .

Original: In this paper we use MHA as one building block as illustrated in Figure 14.
Revised: In this paper we use MHA as one building block as illustrated in Figure 18.

Original: We implement the option policy P (¯ o t | s t , ¯ o t − 1 ; W o ) as the encoder and treat the option embedding matrix W o as encoder’s parameters.
Revised: We implement the option policy P (¯ o t | s t , ¯ o t − 1 ; W o ) as the encoder and treat the option embedding matrix W o as encoder’s parameters.

Original: Specifically, we define the option encoder as:
Revised: Specifically, we define the option encoder as:

Original: [Equation]
Revised: [Equation]

Original: where Categorical ( · ) is a K -dimensional categorical distribution, K is the number of options, and distances between the pair [ s t , ˆ o t − 1 ] (where ˆ
Revised: where Categorical ( · ) is a K -dimensional categorical distribution, K is the number of options, and distances between the pair [ s t , ˆ o t − 1 ] (where ˆ

Original: o t − 1 = W o ¯
Revised: o t − 1 = W o ¯

Original: o t − 1 ).Under
Revised: o t − 1 ).Under

Original: this configuration, options can be seen as clustering centroids.
Revised: this configuration, options can be seen as clustering centroids.

Original: The problem of selecting the next option
Revised: The problem of selecting the next option

Original: o t is equivalent to calculate which clustering centroid ˆ o in embedding matrix W o =
Revised: o t is equivalent to calculate which clustering centroid ˆ o in embedding matrix W o =

Original: [ˆ o 1 , ..., ˆ o K ] is closest to the projected state-option pair FFN ([ s t , ˆ o t − 1 ]) by an efficient MHA-based clustering module : 
Revised: [ˆ o 1 , ..., ˆ o K ] is closest to the projected state-option pair FFN ([ s t , ˆ o t − 1 ]) by an efficient MHA-based clustering module : Clustering ( s t , ¯ o t − 1 , W [Equation] ) =

Original: [Equation]
Revised: [Equation]

Original:  The action policy can be simply implemented as one decoder, which learns to decode ˆ o t and s t into primary actions a t .
Revised: MHA [Equation] [Equation] The action policy can be simply implemented as one decoder, which learns to decode ˆ o t and s t into primary actions a t .

Original: [Equation]
Revised: [Equation]

Original: Because of the Markovian option-value function ¯ V [ s t +1 , ¯ o t ] is an expectation of the option-value function Q O
Revised: Because of the Markovian option-value function ¯ V [ s t +1 , ¯ o t ] is an expectation of the option-value function Q O

Original: [ s t +1 , ¯ o t +1 ] in Eq.
Revised: [ s t +1 , ¯ o t +1 ] in Eq.

Original: (6), we only need to model only one critic function: Q O = FFN ( s t , ¯ o t ) , where Q O is also a decoder of s t and ¯ o t .
Revised: (6), we only need to model only one critic function: Q O = FFN ( s t , ¯ o t ) , where Q O is also a decoder of s t and ¯ o t .

Original: We summarize the detailed algorithm in Appendix F and upload our code in supplemental materials.
Revised: We summarize the detailed algorithm in Appendix F and upload our code in supplemental materials.

Original: G. H YPERPARAMETERS
Revised: G. H YPERPARAMETERS

Original: In this section we summarize our implementation details.
Revised: In this section we summarize our implementation details.

Original: For a fair comparison, all baselines:
Revised: For a fair comparison, all baselines:

Original: DAC+PPO (Zhang & Whiteson, 2019), AHP+PPO (Levy & Shimkin, 2011), PPOC (Klissarov et al., 2017), OC (Bacon et al., 2017) and PPO (Schulman et al., 2017) are from DAC’s open source Github repo: https://github.com/ShangtongZhang/DeepRL/tree/DAC.
Revised: DAC+PPO (Zhang & Whiteson, 2019), AHP+PPO (Levy & Shimkin, 2011), PPOC (Klissarov et al., 2017), OC (Bacon et al., 2017) and PPO (Schulman et al., 2017) are from DAC’s open source Github repo: https://github.com/ShangtongZhang/DeepRL/tree/DAC.

Original: Hyper-parameters used in DAC (Zhang & Whiteson, 2019) for all these baselines are kept unchanged.
Revised: Hyper-parameters used in DAC (Zhang & Whiteson, 2019) for all these baselines are kept unchanged.

Original: MOPG Architecture:
Revised: MOPG Architecture:

Original: For all experiments, our implementation of MOPG is exactly the same as
Revised: For all experiments, our implementation of MOPG is exactly the same as

Original: Figure 14 (b).
Revised: Figure 18 (b).

Original: We use Pytorch to build neural networks.
Revised: We use Pytorch to build neural networks.

Original: Specifically, for skill policy module, we use a skill context matrix W S ∈ R 4 × 40 which has 4 skills ( 4 rows) and an embedding size of 40 (columns).
Revised: Specifically, for skill policy module, we use a skill context matrix W S ∈ R 4 × 40 which has 4 skills ( 4 rows) and an embedding size of 40 (columns).

Original: For Multi-Head Attention, we use Pytorch’s built-in MultiheadAttention function 6 with num _ heads = 1 and embed _ dim = 40 .
Revised: For Multi-Head Attention, we use Pytorch’s built-in MultiheadAttention function 6 with num _ heads = 1 and embed _ dim = 40 .

Original: For layer normalization we use Pytorch’s built-in function
Revised: For layer normalization we use Pytorch’s built-in function

Original: LayerNorm 7 .
Revised: LayerNorm 7 .

Original: For Feed Forward Networks (FNN), we use a 2 layer FNN with ReLu function as activation function with input size of 40 , hidden size of 64 , and output size of 64 neurons.
Revised: For Feed Forward Networks (FNN), we use a 2 layer FNN with ReLu function as activation function with input size of 40 , hidden size of 64 , and output size of 64 neurons.

Original: For Linear layer, we use built-in Linear function 8 to map FFN’s outputs to 4 dimension.
Revised: For Linear layer, we use built-in Linear function 8 to map FFN’s outputs to 4 dimension.

Original: Each dimension actslike a logit for each skill and is used as density in Categorical distribution 9 . For both action policy and critic module, FFNs are of the same size as the one used in the skill policy.
Revised: Each dimension acts

Original: Preprocessing: States are normalized by a running estimation of mean and std.
Revised: Preprocessing: States are normalized by a running estimation of mean and std.

Original: Hyperparameters of PPO:
Revised: Hyperparameters of PPO:

Original: For a fair comparison, we use exactly the same parameters of PPO as
Revised: For a fair comparison, we use exactly the same parameters of PPO as

Original: DAC.
Revised: DAC.

Original: Specifically:
Revised: Specifically:

Original: • Optimizer: Adam with ϵ = 10 − 5 and an initial learning rate 3 × 10 −• Discount ratio γ : 0 .•
Revised: • Optimizer: Adam with ϵ = 10 − 5 and an initial learning rate 3 × 10 −• Discount ratio γ : 0 .•

Original: GAE coefficient: 0 .
Revised: GAE coefficient: 0 .•

Original: • Gradient clip by norm: 0 .•
Revised: Gradient clip by norm: 0 .•

Original: Rollout length: 2048 environment steps • Optimization epochs:• Optimization batch size:• Action probability ratio clip: 0 .
Revised: Rollout length: 2048 environment steps • Optimization epochs:• Optimization batch size:• Action probability ratio clip: 0 .

Original: Computing Infrastructure: We conducted our experiments on an Intel® Core™ i9-9900X CPU @
Revised: Computing Infrastructure: We conducted our experiments on an Intel® Core™ i9-9900X CPU @

Original: 3.50GHz with a single thread and process with PyTorch.
Revised: 3.50GHz with a single thread and process with PyTorch.