Original: H IERARCHICAL R EINFORCEMENT L EARNING VIA A DVANTAGE -W EIGHTED I NFORMATION M AXIMIZATION
Revised: H IERARCHICAL R EINFORCEMENT L EARNING VIA A DVANTAGE -W EIGHTED I NFORMATION M AXIMIZATION

Original: A 
Revised: A BSTRACT

Original: BSTRACT

Original: Real-world tasks are often highly structured.
Revised: Real-world tasks are often highly structured.

Original: Hierarchical reinforcement learning (HRL) has attracted research interest as an approach for leveraging the hierarchical structure of a given task in reinforcement learning (RL).
Revised: Hierarchical reinforcement learning (HRL) has attracted research interest as an approach for leveraging the hierarchical structure of a given task in reinforcement learning (RL).

Original: However, identifying the hierarchical policy structure that enhances the performance of RL is not a trivial task.
Revised: However, identifying the hierarchical policy structure that enhances the performance of RL is not a trivial task.

Original: In this paper, we propose an HRL method that learns a latent variable of a hierarchical policy using mutual information maximization.
Revised: In this paper, we propose an HRL method that learns a latent variable of a hierarchical policy using mutual information maximization.

Original: Our approach can be interpreted as a way to learn a discrete and latent representation of the state-action space.
Revised: Our approach can be interpreted as a way to learn a discrete and latent representation of the state-action space.

Original: To learn option policies that correspond to modes of the advantage function, we introduce advantage-weighted importance sampling .
Revised: To learn option policies that correspond to modes of the advantage function, we introduce advantage-weighted importance sampling .

Original: In our HRL method, the gating policy learns to select option policies based on an option-value function, and these option policies are optimized based on the deterministic policy gradient method.
Revised: In our HRL method, the gating policy learns to select option policies based on an option-value function, and these option policies are optimized based on the deterministic policy gradient method.

Original: This framework is derived by leveraging the analogy between a monolithic policy in standard RL and a hierarchical policy in
Revised: This framework is derived by leveraging the analogy between a monolithic policy in standard RL and a hierarchical policy in

Original: HRL by using a deterministic option policy.
Revised: HRL by using a deterministic option policy.

Original: Experimental results indicate that our HRL approach can learn a diversity of options and that it can enhance the performance of RL in continuous control tasks.
Revised: Experimental results indicate that our HRL approach can learn a diversity of options and that it can enhance the performance of RL in continuous control tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) has been successfully applied to a variety of tasks, including board games (Silver et al., 2016), robotic manipulation tasks (Levine et al., 2016), and video games (Mnih et al., 2015).
Revised: Reinforcement learning (RL) has been successfully applied to a variety of tasks, including board games (Silver et al., 2016), robotic manipulation tasks (Levine et al., 2016), and video games (Mnih et al., 2015).

Original: Hierarchical reinforcement learning (HRL) is a type of RL that leverages the hierarchical structure of a given task by learning a hierarchical policy (Sutton et al., 1999; Dietterich, 2000).
Revised: Hierarchical reinforcement learning (HRL) is a type of RL that leverages the hierarchical structure of a given task by learning a hierarchical policy (Sutton et al., 1999; Dietterich, 2000).

Original: Past studies in this ﬁeld have shown that HRL can solve challenging tasks in the video game domain (Vezhnevets et al., 2017; Bacon et al., 2017) and robotic manipulation (Daniel et al., 2016;
Revised: Past studies in this ﬁeld have shown that HRL can solve challenging tasks in the video game domain (Vezhnevets et al., 2017; Bacon et al., 2017) and robotic manipulation (Daniel et al., 2016;

Original: Osa et al., 2018b).
Revised: Osa et al., 2018b).

Original: In HRL, lower-level policies, which are often referred to as option policies , learn different behavior/control patterns, and the upper-level policy, which is often referred to as the gating policy , learns to select option policies.
Revised: In HRL, lower-level policies, which are often referred to as option policies , learn different behavior/control patterns, and the upper-level policy, which is often referred to as the gating policy , learns to select option policies.

Original: Recent studies have developed HRL methods using deep learning (Goodfellow et al., 2016) and have shown that HRL can yield impressive performance for complex tasks (Bacon et al., 2017; Frans et al., 2018;
Revised: Recent studies have developed HRL methods using deep learning (Goodfellow et al., 2016) and have shown that HRL can yield impressive performance for complex tasks (Bacon et al., 2017; Frans et al., 2018;

Original: Vezhnevets et al., 2017; Haarnoja et al., 2018a).
Revised: Vezhnevets et al., 2017; Haarnoja et al., 2018a).

Original: However, identifying the hierarchical policy structure that yields efﬁcient learning is not a trivial task, since the problem involves learning a sufﬁcient variety of types of behavior to solve a given task.
Revised: However, identifying the hierarchical policy structure that yields efﬁcient learning is not a trivial task, since the problem involves learning a sufﬁcient variety of types of behavior to solve a given task.

Original: In this study, we present an HRL method via the mutual information (MI) maximization with advantage-weighted importance, which we refer to as adInfoHRL.
Revised: In this study, we present an HRL method via the mutual information (MI) maximization with advantage-weighted importance, which we refer to as adInfoHRL.

Original: We formulate the problem of learning a latent variable in a hierarchical policy as one of learning discrete and interpretable repre- sentations of states and actions.
Revised: We formulate the problem of learning a latent variable in a hierarchical policy as one of learning discrete and interpretable repre- sentations of states and actions.

Original: Ideally, each option policy should be located at separate modes of the advantage function.
Revised: Ideally, each option policy should be located at separate modes of the advantage function.

Original: To estimate the latent variable that corresponds to modes of the advantage function, we introduce advantage-weighted importance weights.
Revised: To estimate the latent variable that corresponds to modes of the advantage function, we introduce advantage-weighted importance weights.

Original: Our approach can be considered to divide the state-action space based on an information maximization criterion, and it learns option policies corresponding to each region of the state-action space.
Revised: Our approach can be considered to divide the state-action space based on an information maximization criterion, and it learns option policies corresponding to each region of the state-action space.

Original: We derive adInfoHRL as an HRL method based on deterministic option policies that are trained based on an extension of the deterministic policy gradient (Silver et al., 2014; Fujimoto et al., 2018).
Revised: We derive adInfoHRL as an HRL method based on deterministic option policies that are trained based on an extension of the deterministic policy gradient (Silver et al., 2014; Fujimoto et al., 2018).

Original: The contributions of this paper are twofold:
Revised: The contributions of this paper are twofold:

Original: We propose the learning of a latent variable of a hierarchical policy as a discrete and hidden representation of the state-action space.
Revised: We propose the learning of a latent variable of a hierarchical policy as a discrete and hidden representation of the state-action space.

Original: To learn option policies that correspond to the modes of the advantage function, we introduce advantage-weighted importance.
Revised: To learn option policies that correspond to the modes of the advantage function, we introduce advantage-weighted importance.

Original: We propose an HRL method, where the option policies are optimized based on the deterministic policy gradient and the gating policy selects the option that maximizes the expected return.
Revised: We propose an HRL method, where the option policies are optimized based on the deterministic policy gradient and the gating policy selects the option that maximizes the expected return.

Original: The experimental results show that our proposed method adInfoHRL can learn a diversity of options on continuous control tasks.
Revised: The experimental results show that our proposed method adInfoHRL can learn a diversity of options on continuous control tasks.

Original: Moreover, our approach can improve the performance of TD3 on such tasks as the Walker2d and Ant tasks in OpenAI Gym with MuJoco simulator.
Revised: Moreover, our approach can improve the performance of TD3 on such tasks as the Walker2d and Ant tasks in OpenAI Gym with MuJoco simulator.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: In this section, we formulate the problem of HRL in this paper and describe methods related to our proposal.
Revised: In this section, we formulate the problem of HRL in this paper and describe methods related to our proposal.

Original: 2. H IERARCHICAL R EINFORCEMENT L EARNING
Revised: 2. H IERARCHICAL R EINFORCEMENT L EARNING

Original: We consider tasks that can be modeled as a Markov decision process (MDP), consisting of a state space S , an action space A , a reward function r : S × A (cid:55)→ R , an initial state distribution ρ ( s 0 ) , and a transition probability p ( s t +1 | s t , a t ) that deﬁnes the probability of transitioning from state s t and action a t at time t to next state s t +1 .
Revised: We consider tasks that can be modeled as a Markov decision process (MDP), consisting of a state space S , an action space A , a reward function r : S × A (cid:55)→ R , an initial state distribution ρ ( s 0 ) , and a transition probability p ( s t +1 | s t , a t ) that deﬁnes the probability of transitioning from state s t and action a t at time t to next state s t +1 .

Original: The return is deﬁned as R t =
Revised: The return is deﬁned as R t =

Original: (cid:80)
Revised: (cid:80)

Original: Ti = t γ i − t r
Revised: Ti = t γ i − t r

Original: ( s i , a i ) , where γ is a discount factor, and policy π ( a | s ) is deﬁned as the density of action a given state s .
Revised: ( s i , a i ) , where γ is a discount factor, and policy π ( a | s ) is deﬁned as the density of action a given state s .

Original: Let d π ( s ) =
Revised: Let d π ( s ) =

Original: (cid:80) Tt =

Revised: (cid:80) Tt =

Original: γ t p ( s t = s ) denote the discounted visitation frequency induced by the policy π .
Revised: γ t p ( s t = s ) denote the discounted visitation frequency induced by the policy π .

Original: The goal of reinforcement learning is to learn a policy that maximizes the expected return
Revised: The goal of reinforcement learning is to learn a policy that maximizes the expected return

Original: J ( π ) =
Revised: J ( π ) =

Original: E s 0 , a 0 ,...
Revised: E s 0 , a 0 ,...

Original: [ R 0 ] where s 0 ∼ ρ ( s 0 ) , a ∼ π and s t +1 ∼ p ( s t +1 | s t , a t ) .
Revised: [ R 0 ] where s 0 ∼ ρ ( s 0 ) , a ∼ π and s t +1 ∼ p ( s t +1 | s t , a t ) .

Original: By deﬁning the
Revised: By deﬁning the

Original: Q-function as Q π ( s , a ) =
Revised: Q-function as Q π ( s , a ) =

Original: E s 0 , a 0 ,...
Revised: E s 0 , a 0 ,...

Original: [ R t | s t = s , a t = a ] , the objective function of reinforcement learning can be rewritten as follows:
Revised: [ R t | s t = s , a t = a ] , the objective function of reinforcement learning can be rewritten as follows:

Original: [Equation]
Revised: [Equation]

Original: Herein, we consider hierarchical policy π ( a | s )
Revised: Herein, we consider hierarchical policy π ( a | s )

Original: =
Revised: =

Original: (cid:80) o ∈O π
Revised: (cid:80) o ∈O π

Original: ( o | s ) π ( a | s , o ) , where o is the latent variable and O is the set of possible values of o .
Revised: ( o | s ) π ( a | s , o ) , where o is the latent variable and O is the set of possible values of o .

Original: Many existing HRL methods employ a policy structure of this form (Frans et al., 2018;
Revised: Many existing HRL methods employ a policy structure of this form (Frans et al., 2018;

Original: Vezhnevets et al., 2017; Bacon et al., 2017; Florensa et al., 2017;
Revised: Vezhnevets et al., 2017; Bacon et al., 2017; Florensa et al., 2017;

Original: Daniel et al., 2016).
Revised: Daniel et al., 2016).

Original: In general, latent variable o can be discrete (Frans et al., 2018; Bacon et al., 2017; Florensa et al., 2017; Daniel et al., 2016; Osa & Sugiyama, 2018) or continuous (Vezhnevets et al., 2017).
Revised: In general, latent variable o can be discrete (Frans et al., 2018; Bacon et al., 2017; Florensa et al., 2017; Daniel et al., 2016; Osa & Sugiyama, 2018) or continuous (Vezhnevets et al., 2017).

Original: π ( o | s ) is often referred to as a gating policy (Daniel et al., 2016; Osa & Sugiyama, 2018), policy over options (Bacon et al., 2017), or manager (Vezhnevets et al., 2017).
Revised: π ( o | s ) is often referred to as a gating policy (Daniel et al., 2016; Osa & Sugiyama, 2018), policy over options (Bacon et al., 2017), or manager (Vezhnevets et al., 2017).

Original: Likewise, π ( a | s , o ) is often referred to as an option policy (Osa & Sugiyama, 2018), sub-policy (Daniel et al., 2016), or worker (Vezhnevets et al., 2017).
Revised: Likewise, π ( a | s , o ) is often referred to as an option policy (Osa & Sugiyama, 2018), sub-policy (Daniel et al., 2016), or worker (Vezhnevets et al., 2017).

Original: In HRL, the objective function is given by
Revised: In HRL, the objective function is given by

Original: [Equation]
Revised: [Equation]

Original: As discussed in the literature on inverse RL (Ziebart, 2010), multiple policies can yield equivalent expected returns.
Revised: As discussed in the literature on inverse RL (Ziebart, 2010), multiple policies can yield equivalent expected returns.

Original: This indicates that there exist multiple solutions to latent variable o that maximizes the expected return.
Revised: This indicates that there exist multiple solutions to latent variable o that maximizes the expected return.

Original: To obtain the preferable solution for o , we need to impose additional constraints in HRL.
Revised: To obtain the preferable solution for o , we need to impose additional constraints in HRL.

Original: Although prior work has employed regularizers (Bacon et al., 2017) and constraints (Daniel et al., 2016) to obtain various option policies, the method of learning a good latent variable o that improves sample-efﬁciency of the learning process remains unclear.
Revised: Although prior work has employed regularizers (Bacon et al., 2017) and constraints (Daniel et al., 2016) to obtain various option policies, the method of learning a good latent variable o that improves sample-efﬁciency of the learning process remains unclear.

Original: In this study we propose the learning of the latent variable by maximizing MI between latent variables and state-action pairs.
Revised: In this study we propose the learning of the latent variable by maximizing MI between latent variables and state-action pairs.

Original: 2. D ETERMINISTIC P OLICY G RADIENT
Revised: 2. D ETERMINISTIC P OLICY G RADIENT

Original: The deterministic policy gradient (DPG) algorithm was developed for learning a monolithic deterministic policy µ θ ( s ) :
Revised: The deterministic policy gradient (DPG) algorithm was developed for learning a monolithic deterministic policy µ θ ( s ) :

Original: S (cid:55)→ A by Silver et al.
Revised: S (cid:55)→ A by Silver et al.

Original: In off-policy RL, the objective is to maximize the expectation of the return, averaged over the state distribution induced by a behavior policy β ( a | s ) :
Revised: In off-policy RL, the objective is to maximize the expectation of the return, averaged over the state distribution induced by a behavior policy β ( a | s ) :

Original: [Equation]
Revised: [Equation]

Original: When a policy is deterministic, the objective becomes J ( π ) =
Revised: When a policy is deterministic, the objective becomes J ( π ) =

Original: (cid:82) d β ( s ) Q
Revised: (cid:82) d β ( s ) Q

Original: π
Revised: π

Original: (cid:0) s , µ θ ( s ) (cid:1)
Revised: (cid:0) s , µ θ ( s ) (cid:1)

Original: d s .
Revised: d s .

Original: Silver et al.
Revised: Silver et al.

Original: (2014) have shown that the gradient of a deterministic policy is given by
Revised: (2014) have shown that the gradient of a deterministic policy is given by

Original: [Equation]
Revised: [Equation]

Original: The DPG algorithm has been extended to the deep deterministic policy gradient (DDPG) for continuous control problems that require neural network policies (Lillicrap et al., 2016).
Revised: The DPG algorithm has been extended to the deep deterministic policy gradient (DDPG) for continuous control problems that require neural network policies (Lillicrap et al., 2016).

Original: Twin Delayed
Revised: Twin Delayed

Original: Deep Deterministic policy gradient algorithm (TD3) proposed by Fujimoto et al.
Revised: Deep Deterministic policy gradient algorithm (TD3) proposed by Fujimoto et al.

Original: (2018) is a variant of DDPG that outperforms the state-of-the-art on-policy methods such as TRPO (Schulman et al., 2017a) and PPO (Schulman et al., 2017b) in certain domains.
Revised: (2018) is a variant of DDPG that outperforms the state-of-the-art on-policy methods such as TRPO (Schulman et al., 2017a) and PPO (Schulman et al., 2017b) in certain domains.

Original: We extend this deterministic policy gradient to learn a hierarchical policy.
Revised: We extend this deterministic policy gradient to learn a hierarchical policy.

Original: 2. R EPRESENTATION L EARNING VIA I NFORMATION M AXIMIZATION
Revised: 2. R EPRESENTATION L EARNING VIA I NFORMATION M AXIMIZATION

Original: Recent studies such as those by Chen et al.
Revised: Recent studies such as those by Chen et al.

Original: ; Hu et al.
Revised: ; Hu et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: (2017) have shown that an interpretable representation can be learned by maximizing MI.
Revised: (2017) have shown that an interpretable representation can be learned by maximizing MI.

Original: Given a dataset X =
Revised: Given a dataset X =

Original: ( x 1 , ..., x n ) , regularized information maximization (RIM) proposed by Gomes et al.
Revised: ( x 1 , ..., x n ) , regularized information maximization (RIM) proposed by Gomes et al.

Original: (2010) involves learning a conditional model ˆ p ( y | x ; η ) with parameter vector η that predicts a label y .
Revised: (2010) involves learning a conditional model ˆ p ( y | x ; η ) with parameter vector η that predicts a label y .

Original: The objective of RIM is to minimize
Revised: The objective of RIM is to minimize

Original: [Equation]
Revised: [Equation]

Original: where (cid:96) ( η ) is the regularization term,
Revised: where (cid:96) ( η ) is the regularization term,

Original: I η ( x , y ) is MI, and λ is a coefﬁcient.
Revised: I η ( x , y ) is MI, and λ is a coefﬁcient.

Original: MI can be decomposed as I η ( x , y )
Revised: MI can be decomposed as I η ( x , y )

Original: =
Revised: =

Original: H ( y ) − H
Revised: H ( y ) − H

Original: ( y | x ) where H ( y ) is entropy and H ( y | x ) the conditional entropy.
Revised: ( y | x ) where H ( y ) is entropy and H ( y | x ) the conditional entropy.

Original: Increasing H ( y ) conduces the label to be uniformly distributed, and decreasing H ( y | x ) conduces to clear cluster assignments.
Revised: Increasing H ( y ) conduces the label to be uniformly distributed, and decreasing H ( y | x ) conduces to clear cluster assignments.

Original: Although RIM was originally developed for unsupervised clustering problems, the concept is applicable to various problems that require learning a hidden discrete representation.
Revised: Although RIM was originally developed for unsupervised clustering problems, the concept is applicable to various problems that require learning a hidden discrete representation.

Original: In this study, we formulate the problem of learning the latent variable o of a hierarchical policy as one of learning a latent representation of the state-action space.
Revised: In this study, we formulate the problem of learning the latent variable o of a hierarchical policy as one of learning a latent representation of the state-action space.

Original:  L EARNING O PTIONS VIA A DVANTAGE -W EIGHTED I NFORMATION M AXIMIZATION
Revised:  L EARNING O PTIONS VIA A DVANTAGE -W EIGHTED I NFORMATION M AXIMIZATION

Original: In this section, we propose a novel HRL method based on advantage-weighted information maximization.
Revised: In this section, we propose a novel HRL method based on advantage-weighted information maximization.

Original: We ﬁrst introduce the latent representation learning via advantage-weighted information maximization, and we then describe the HRL framework based on deterministic option policies.
Revised: We ﬁrst introduce the latent representation learning via advantage-weighted information maximization, and we then describe the HRL framework based on deterministic option policies.

Original: 3. L ATENT R EPRESENTATION L EARNING VIA A DVANTAGE -W EIGHTED I NFORMATION M AXIMIZATION
Revised: 3. L ATENT R EPRESENTATION L EARNING VIA A DVANTAGE -W EIGHTED I NFORMATION M AXIMIZATION

Original: Although prior work has often considered H ( o | s ) or I ( s , o ) , which results in a division of the state space, we are interested in using I (cid:0) ( s , a ) , o (cid:1) for dividing the state-action space instead.
Revised: Although prior work has often considered H ( o | s ) or I ( s , o ) , which results in a division of the state space, we are interested in using I (cid:0) ( s , a ) , o (cid:1) for dividing the state-action space instead.

Original: A schematic sketch of our approach is shown in Figure 1.
Revised: A schematic sketch of our approach is shown in Figure 1.

Original: As shown in the left side of Figure 1, the advantage function often has multiple modes.
Revised: As shown in the left side of Figure 1, the advantage function often has multiple modes.

Original: Ideally, each option policies should correspond to separate modes of the advantage function.
Revised: Ideally, each option policies should correspond to separate modes of the advantage function.

Original: However, it is non-trivial to ﬁnd the modes of the advantage function in practice.
Revised: However, it is non-trivial to ﬁnd the modes of the advantage function in practice.

Original: For this purpose, we reduce the problem of ﬁnding modes of the advantage function to that of ﬁnding the modes of the probability density of state action pairs.
Revised: For this purpose, we reduce the problem of ﬁnding modes of the advantage function to that of ﬁnding the modes of the probability density of state action pairs.

Original: We consider a policy based on the advantage function of the form
Revised: We consider a policy based on the advantage function of the form

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: where A π ( s , a ) =
Revised: where A π ( s , a ) =

Original: Q π ( s , a ) − V π ( s ) is the advantage function, V π ( s ) is the state value function, and Z is the partition function.
Revised: Q π ( s , a ) − V π ( s ) is the advantage function, V π ( s ) is the state value function, and Z is the partition function.

Original: f ( · ) is a functional , which is a function of a function.
Revised: f ( · ) is a functional , which is a function of a function.

Original: f ( · ) is a monotonically increasing function with respect to the input variable and always satisﬁes f ( · ) > 0 .
Revised: f ( · ) is a monotonically increasing function with respect to the input variable and always satisﬁes f ( · ) > 0 .

Original: In our implementation we used the exponential function f ( · ) = exp( · ) .
Revised: In our implementation we used the exponential function f ( · ) = exp( · ) .

Original: When following such a policy, an action with the larger advantage is drawn with a higher probability.
Revised: When following such a policy, an action with the larger advantage is drawn with a higher probability.

Original: Under this assumption, ﬁnding the modes of the advantage function is equivalent to ﬁnding modes of the density induced by π Ad .
Revised: Under this assumption, ﬁnding the modes of the advantage function is equivalent to ﬁnding modes of the density induced by π Ad .

Original: Thus, ﬁnding the modes of the advantage function can be reduced to the problem of clustering samples induced by π Ad .
Revised: Thus, ﬁnding the modes of the advantage function can be reduced to the problem of clustering samples induced by π Ad .

Original: Following the formulation of RIM introduced in Section 2.3, we formulate the problem of clustering samples induced by π Ad as the learning of discrete representations via MI maximization.
Revised: Following the formulation of RIM introduced in Section 2.3, we formulate the problem of clustering samples induced by π Ad as the learning of discrete representations via MI maximization.

Original: For this purpose, we consider a neural network that estimates p ( o | s , a ; η ) parameterized with vector η , which we refer to as the option network .
Revised: For this purpose, we consider a neural network that estimates p ( o | s , a ; η ) parameterized with vector η , which we refer to as the option network .

Original: We formulate the learning of the latent variable o as minimizing
Revised: We formulate the learning of the latent variable o as minimizing

Original: [Equation]
Revised: [Equation]

Original: where I ( o, ( s , a ))
Revised: where I ( o, ( s , a ))

Original: =
Revised: =

Original: ˆ H ( o | s , a ; η ) − ˆ H ( o ; η ) , and (cid:96) ( η ) is the regularization term.
Revised: ˆ H ( o | s , a ; η ) − ˆ H ( o ; η ) , and (cid:96) ( η ) is the regularization term.

Original: In practice, we need to approximate the advantage function, and we learn the discrete variable o that corresponds to the modes of the current estimate of the advantage function.
Revised: In practice, we need to approximate the advantage function, and we learn the discrete variable o that corresponds to the modes of the current estimate of the advantage function.

Original: For regularization, we used a simpliﬁed version of virtual adversarial training (VAT) proposed by Miyato et al.
Revised: For regularization, we used a simpliﬁed version of virtual adversarial training (VAT) proposed by Miyato et al.

Original: Namely, we set (cid:96) ( η ) =
Revised: Namely, we set (cid:96) ( η ) =

Original: D KL (cid:0) p ( o | s noise , a noise ; η ) ||
Revised: D KL (cid:0) p ( o | s noise , a noise ; η ) ||

Original: p
Revised: p

Original: ( o | s , a ; η ) (cid:1) where s noise =
Revised: ( o | s , a ; η ) (cid:1) where s noise =

Original: s + (cid:15) s , a noise =
Revised: s + (cid:15) s , a noise =

Original: a + (cid:15) a , (cid:15) s and (cid:15) a denote white noise.
Revised: a + (cid:15) a , (cid:15) s and (cid:15) a denote white noise.

Original: This regularization term penalizes dissimilarity between an original state-action pair and a perturbed one, and Hu et al.
Revised: This regularization term penalizes dissimilarity between an original state-action pair and a perturbed one, and Hu et al.

Original: (2017) empirically show that this regularization improves the performance of learning latent discrete representations.
Revised: (2017) empirically show that this regularization improves the performance of learning latent discrete representations.

Original: When computing MI, we need to compute p ( o ) and H ( o | s , a ) given by
Revised: When computing MI, we need to compute p ( o ) and H ( o | s , a ) given by

Original: [Equation]
Revised: [Equation]

Original: Thus, the probability density of ( s , a ) induced by π Ad is necessary for computing MI for our purpose.
Revised: Thus, the probability density of ( s , a ) induced by π Ad is necessary for computing MI for our purpose.

Original: To estimate the probability density of ( s , a ) induced by π Ad , we introduce the advantageweighted importance in the next section.
Revised: To estimate the probability density of ( s , a ) induced by π Ad , we introduce the advantageweighted importance in the next section.

Original: 3. I MPORTANCE W EIGHTS FOR M UTUAL I NFORMATION E STIMATION
Revised: 3. I MPORTANCE W EIGHTS FOR M UTUAL I NFORMATION E STIMATION

Original: Although we show that the problem of ﬁnding the modes of the advantage function can be reduced to MI maximization with respect to the samples induced by π Ad , samples induced by π Ad are not available in practice.
Revised: Although we show that the problem of ﬁnding the modes of the advantage function can be reduced to MI maximization with respect to the samples induced by π Ad , samples induced by π Ad are not available in practice.

Original: While those induced during the learning process are available, a discrete representation obtained from such samples does not correspond to the modes of the advantage function.
Revised: While those induced during the learning process are available, a discrete representation obtained from such samples does not correspond to the modes of the advantage function.

Original: To estimate the density induced by π Ad , we employ an importance sampling approach.
Revised: To estimate the density induced by π Ad , we employ an importance sampling approach.

Original: We assume that the change of the state distribution induced by the policy update is sufﬁciently small, namely, d π Ad ( s ) ≈
Revised: We assume that the change of the state distribution induced by the policy update is sufﬁciently small, namely, d π Ad ( s ) ≈

Original: d β ( s ) .
Revised: d β ( s ) .

Original: Then, the importance weight can be approximated as
Revised: Then, the importance weight can be approximated as

Original: [Equation]
Revised: [Equation]

Original: and the normalized importance weight is given gy
Revised: and the normalized importance weight is given gy

Original: [Equation]
Revised: [Equation]

Original: As the partition function Z is canceled, we do not need to compute Z when computing the importance weight in practice.
Revised: As the partition function Z is canceled, we do not need to compute Z when computing the importance weight in practice.

Original: We call this importance weight W the advantage-weighted importance and employ it to compute the objective function used to estimate the latent variable.
Revised: We call this importance weight W the advantage-weighted importance and employ it to compute the objective function used to estimate the latent variable.

Original: This advantage-weighted importance is used to compute the entropy terms for computing MI in
Revised: This advantage-weighted importance is used to compute the entropy terms for computing MI in

Original: Equation (7).
Revised: Equation (7).

Original: The empirical estimate of the entropy H ( o ) is given by
Revised: The empirical estimate of the entropy H ( o ) is given by

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the samples ( s i , a i ) are drawn from p β ( s , a ) induced by a behavior policy β ( a | s ) .
Revised: where the samples ( s i , a i ) are drawn from p β ( s , a ) induced by a behavior policy β ( a | s ) .

Original: Likewise, the empirical estimate of the conditional entropy H
Revised: Likewise, the empirical estimate of the conditional entropy H

Original: [Equation]
Revised: [Equation]

Original: o | s , a ) is given by
Revised: o | s , a ) is given by

Original: [Equation]
Revised: [Equation]

Original: The derivations of Equations (12) and (13) are provided in Appendix A. To train the option network, we store the samples collected by the M most recent behavior policies, to which we refer as onpolicy buffer D on .
Revised: The derivations of Equations (12) and (13) are provided in Appendix A. To train the option network, we store the samples collected by the M most recent behavior policies, to which we refer as onpolicy buffer D on .

Original: Although the algorithm works with entire samples stored in the replay buffer, we observe that the use of the on-policy buffer for latent representation learning exhibits better performance.
Revised: Although the algorithm works with entire samples stored in the replay buffer, we observe that the use of the on-policy buffer for latent representation learning exhibits better performance.

Original: For this reason, we decided to use the on-policy buffer in our implementation.
Revised: For this reason, we decided to use the on-policy buffer in our implementation.

Original: Therefore, while the algorithm is off-policy in the sense that the option is learned from samples collected by behavior policies, our implementation is “semi”on-policy in the sense that we use samples collected by the most recent behavior policies.
Revised: Therefore, while the algorithm is off-policy in the sense that the option is learned from samples collected by behavior policies, our implementation is “semi”on-policy in the sense that we use samples collected by the most recent behavior policies.

Original:  HRL O BJECTIVE WITH D ETERMINISTIC O PTION P OLICIES
Revised:  HRL O BJECTIVE WITH D ETERMINISTIC O PTION P OLICIES

Original: Instead of stochastic option policies, we consider deterministic option policies and model them using separate neural networks.
Revised: Instead of stochastic option policies, we consider deterministic option policies and model them using separate neural networks.

Original: We denote by π ( a | s , o )
Revised: We denote by π ( a | s , o )

Original: =
Revised: =

Original: µ o θ ( s ) deterministic option policies parameterized by vector θ .
Revised: µ o θ ( s ) deterministic option policies parameterized by vector θ .

Original: The objective function of off-policy HRL with deterministic option policies can then be obtained by replacing π ( a | s ) with (cid:80) o ∈O π
Revised: The objective function of off-policy HRL with deterministic option policies can then be obtained by replacing π ( a | s ) with (cid:80) o ∈O π

Original: ( o | s ) π ( a | s , o ) in Equation (3):
Revised: ( o | s ) π ( a | s , o ) in Equation (3):

Original: [Equation]
Revised: [Equation]

Original: (14) where Q π ( s , a ; w ) is an approximated Q-function parameterized using vector w .
Revised: (14) where Q π ( s , a ; w ) is an approximated Q-function parameterized using vector w .

Original: This form of the objective function is analogous to Equation (3).
Revised: This form of the objective function is analogous to Equation (3).

Original: Thus, we can extend standard RL techniques to the learning of the gating policy π
Revised: Thus, we can extend standard RL techniques to the learning of the gating policy π

Original: ( o | s ) in HRL with deterministic option policies.
Revised: ( o | s ) in HRL with deterministic option policies.

Original: In HRL, the goal of the gating policy is to generate a value of o that maximizes the conditional expectation of the return:
Revised: In HRL, the goal of the gating policy is to generate a value of o that maximizes the conditional expectation of the return:

Original: [Equation]
Revised: [Equation]

Original: which is often referred to as the option-value function (Sutton et al., 1999).
Revised: which is often referred to as the option-value function (Sutton et al., 1999).

Original: When option policies are stochastic, it is often necessary to approximate the option-value function
Revised: When option policies are stochastic, it is often necessary to approximate the option-value function

Original: Q π Ω ( s , o ) in addition to the action-value function Q π ( s , a ) .
Revised: Q π Ω ( s , o ) in addition to the action-value function Q π ( s , a ) .

Original: However, in our case, the option-value function for deterministic option policies is given by
Revised: However, in our case, the option-value function for deterministic option policies is given by

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Input: Number of options O , size of on-policy buffer
Revised: Input: Number of options O , size of on-policy buffer

Original: Initialize: Replay buffer D R , on-policy buffer D on , network parameters η , θ , w , θ target , w target repeatfor t = 0 to t =
Revised: Initialize: Replay buffer D R , on-policy buffer D on , network parameters η , θ , w , θ target , w target repeatfor t = 0 to t =

Original: T do
Revised: T do

Original: Draw an option for a given s by following Equation 17: o ∼ π ( o | s )
Revised: Draw an option for a given s by following Equation 17: o ∼ π ( o | s )

Original: Draw an action a ∼ β ( a | s , o ) = µ o θ ( s ) + (cid:15)
Revised: Draw an action a ∼ β ( a | s , o ) = µ o θ ( s ) + (cid:15)

Original: Record a data sample ( s , a , r, s (cid:48) )
Revised: Record a data sample ( s , a , r, s (cid:48) )

Original: Aggregate the data in D R and D on if the on-policy buffer is full then
Revised: Aggregate the data in D R and D on if the on-policy buffer is full then

Original: Update the option network by minimizing Equation (7) for samples in D on
Revised: Update the option network by minimizing Equation (7) for samples in D on

Original: Clear the on-policy buffer D on end if
Revised: Clear the on-policy buffer D on end if

Original: Sample a batch D batch ∈ D R
Revised: Sample a batch D batch ∈ D R

Original: Update the Q network parameter w if t mod d then
Revised: Update the Q network parameter w if t mod d then

Original: Estimate p ( o | s i , a i ) for ( s i , a
Revised: Estimate p ( o | s i , a i ) for ( s i , a

Original: i ) ∈ D batch using the option network
Revised: i ) ∈ D batch using the option network

Original: Assign samples ( s i , a
Revised: Assign samples ( s i , a

Original: i ) ∈
Revised: i ) ∈

Original: D batch to the option o ∗ = arg max p
Revised: D batch to the option o ∗ = arg max p

Original: o | s i , a i )
Revised: o | s i , a i )

Original: Update the option policy networks µ o θ ( s ) for o = 1 , ..., O with Equation (19)
Revised: Update the option policy networks µ o θ ( s ) for o = 1 , ..., O with Equation (19)

Original: Update the target networks: w target ← τ w +(1 − τ ) w target , θ target ← τ θ +(1 − τ ) θ target end if end for until the convergence return θ which we can estimate using the deterministic option policy µ o θ ( s ) and the approximated actionvalue function Q π ( s , a ; w ) .
Revised: Update the target networks: w target ← τ w +(1 − τ ) w target , θ target ← τ θ +(1 − τ ) θ target end if end for until the convergence return θ which we can estimate using the deterministic option policy µ o θ ( s ) and the approximated actionvalue function Q π ( s , a ; w ) .

Original: In this work we employ the softmax gating policy of the form
Revised: In this work we employ the softmax gating policy of the form

Original: [Equation]
Revised: [Equation]

Original: which encodes the exploration in its form (Daniel et al., 2016)
Revised: which encodes the exploration in its form (Daniel et al., 2016)

Original: The state value function is given as
Revised: The state value function is given as

Original: [Equation]
Revised: [Equation]

Original: which can be computed using Equation (17).
Revised: which can be computed using Equation (17).

Original: We use this state-value function when computing the advantage-weighted importance as A ( s , a ) =
Revised: We use this state-value function when computing the advantage-weighted importance as A ( s , a ) =

Original: Q ( s , a ) − V ( s ) .
Revised: Q ( s , a ) − V ( s ) .

Original: In this study, the Q-function is trained in a manner proposed by Fujimoto et al.
Revised: In this study, the Q-function is trained in a manner proposed by Fujimoto et al.

Original: Two neural networks ( Q π w 1 , Q π w 2 ) are trained to estimate the Q-function, and the target value of the Q-function is computed as y i
Revised: Two neural networks ( Q π w 1 , Q π w 2 ) are trained to estimate the Q-function, and the target value of the Q-function is computed as y i

Original: =
Revised: =

Original: r i + γ min 1 , 2 Q ( s i , a i ) for sample ( s i , a i , a (cid:48)
Revised: r i + γ min 1 , 2 Q ( s i , a i ) for sample ( s i , a i , a (cid:48)

Original: i , r i ) in a batch sampled from a replay buffer, where r i = r
Revised: i , r i ) in a batch sampled from a replay buffer, where r i = r

Original: ( s i , a i ) .
Revised: ( s i , a i ) .

Original: In this study, the gating policy determines the option once every N time steps, i.e., t = 0 , N, 2 N, . . .
Revised: In this study, the gating policy determines the option once every N time steps, i.e., t = 0 , N, 2 N, . . .

Original: Neural networks that model µ o θ ( a | s ) for o = 1 , ..., O , which we refer to as option-policy networks , are trained separately for each option.
Revised: Neural networks that model µ o θ ( a | s ) for o = 1 , ..., O , which we refer to as option-policy networks , are trained separately for each option.

Original: In the learning phase, p ( o | s , a ) is estimated by the option network.
Revised: In the learning phase, p ( o | s , a ) is estimated by the option network.

Original: Then, samples are assigned to option o ∗ = arg max o p ( o | s , a ; η ) and are used to update the option-policy network that corresponds to o ∗ .
Revised: Then, samples are assigned to option o ∗ = arg max o p ( o | s , a ; η ) and are used to update the option-policy network that corresponds to o ∗ .

Original: When performing a rollout, o is drawn by following the gating policy in Equation (17), and an action is generated by the selected option-policy network.
Revised: When performing a rollout, o is drawn by following the gating policy in Equation (17), and an action is generated by the selected option-policy network.

Original: Differentiating the objective function in Equation (14), we obtain the deterministic policy gradient of our option-policy µ o θ ( s ) given by
Revised: Differentiating the objective function in Equation (14), we obtain the deterministic policy gradient of our option-policy µ o θ ( s ) given by

Original: [Equation]
Revised: [Equation]

Original: The procedure of adInfoHRL is summarized by Algorithm 1.
Revised: The procedure of adInfoHRL is summarized by Algorithm 1.

Original: As in TD3 (Fujimoto et al., 2018), we employed the soft update using a target value network and a target policy network.
Revised: As in TD3 (Fujimoto et al., 2018), we employed the soft update using a target value network and a target policy network.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluated the proposed algorithm adInfoHRL on the OpenAI Gym platform (Brockman et al., 2016) with the MuJoCo Physics simulator (Todorov et al., 2012).
Revised: We evaluated the proposed algorithm adInfoHRL on the OpenAI Gym platform (Brockman et al., 2016) with the MuJoCo Physics simulator (Todorov et al., 2012).

Original: We compared its performance with that of PPO implemented in OpenAI baselines (Dhariwal et al., 2017) and TD3.
Revised: We compared its performance with that of PPO implemented in OpenAI baselines (Dhariwal et al., 2017) and TD3.

Original: Henderson et al.
Revised: Henderson et al.

Original: (2018) have recently claimed that algorithm performance varies across environment, there is thus no clearly best method for all benchmark environments, and off-policy and on-policy methods have advantages in different problem domains.
Revised: (2018) have recently claimed that algorithm performance varies across environment, there is thus no clearly best method for all benchmark environments, and off-policy and on-policy methods have advantages in different problem domains.

Original: To analyze the performance of adInfoHRL, we compared it with state-of-the-art algorithms for both on-policy and off-policy methods, although we focused on the comparison with TD3, as our implementation of adInfoHRL is based on it.
Revised: To analyze the performance of adInfoHRL, we compared it with state-of-the-art algorithms for both on-policy and off-policy methods, although we focused on the comparison with TD3, as our implementation of adInfoHRL is based on it.

Original: To determine the effect of learning the latent variable via information maximization, we used the same network architectures for the actor and critic in adInfoHRL and TD3.
Revised: To determine the effect of learning the latent variable via information maximization, we used the same network architectures for the actor and critic in adInfoHRL and TD3.

Original: In addition, to evaluate the beneﬁt of the advantage-weighted importance, we evaluated a variant of adInfoHRL, which does not use the advantage-weighted importance for computing mutual information.
Revised: In addition, to evaluate the beneﬁt of the advantage-weighted importance, we evaluated a variant of adInfoHRL, which does not use the advantage-weighted importance for computing mutual information.

Original: We refer to this variant of adInfoHRL as infoHRL.
Revised: We refer to this variant of adInfoHRL as infoHRL.

Original: The gating policy updated variable o once every three time steps.
Revised: The gating policy updated variable o once every three time steps.

Original: We tested the performance of adInfoHRL with two and four options.
Revised: We tested the performance of adInfoHRL with two and four options.

Original: The activation of options over time and snapshots of the learned option policies on the Walker2d task are shown in Figure 2, which visualizes the result from adInfoHRL with four options.
Revised: The activation of options over time and snapshots of the learned option policies on the Walker2d task are shown in Figure 2, which visualizes the result from adInfoHRL with four options.

Original: One can see that the option policies are activated in different phases of locomotion.
Revised: One can see that the option policies are activated in different phases of locomotion.

Original: While the option indicated by yellow in Figure 2 corresponds to the phase for kicking the ﬂoor, the option indicated by blue corresponds to the phase when the agent was on the ﬂy.
Revised: While the option indicated by yellow in Figure 2 corresponds to the phase for kicking the ﬂoor, the option indicated by blue corresponds to the phase when the agent was on the ﬂy.

Original: Visualization of the options learned on the
Revised: Visualization of the options learned on the

Original: HalfCheetah and Ant tasks are shown in Appendix D.
Revised: HalfCheetah and Ant tasks are shown in Appendix D.

Original: The averaged return of ﬁve trials is reported in Figure 3(a)-(d).
Revised: The averaged return of ﬁve trials is reported in Figure 3(a)-(d).

Original: AdIfoHRL yields the best performance on Ant 1 and Walker2d, whereas the performance of TD3 and adInfoHRL was comparable on HalfCheetah and Hopper, and PPO outperformed the other methods on Hopper.
Revised: AdIfoHRL yields the best performance on Ant 1 and Walker2d, whereas the performance of TD3 and adInfoHRL was comparable on HalfCheetah and Hopper, and PPO outperformed the other methods on Hopper.

Original: Henderson et al.
Revised: Henderson et al.

Original: (2018) claimed that on-policy methods show their superiority on tasks with unstable dynamics, and our experimental results are in line with such previous studies.
Revised: (2018) claimed that on-policy methods show their superiority on tasks with unstable dynamics, and our experimental results are in line with such previous studies.

Original: AdinfoHRL outperformed infoHRL, which isthe variant of adInfoHRL without the advantage-weighted importance on all the tasks.
Revised: AdinfoHRL outperformed infoHRL, which isthe variant of adInfoHRL without the advantage-weighted importance on all the tasks.

Original: This result shows that the adavatage-weighted importance enhanced the performance of learning options.
Revised: This result shows that the adavatage-weighted importance enhanced the performance of learning options.

Original: AdInfoHRL exhibited the sample efﬁciency on Ant and Walker2d in the sense that it required fewer samples than TD3 to achieve comparable performance on those tasks.
Revised: AdInfoHRL exhibited the sample efﬁciency on Ant and Walker2d in the sense that it required fewer samples than TD3 to achieve comparable performance on those tasks.

Original: The concept underlying adInfoHRL is to divide the state-action space to deal with the multi-modal advantage function and learn option policies corresponding to separate modes of the advantage function.
Revised: The concept underlying adInfoHRL is to divide the state-action space to deal with the multi-modal advantage function and learn option policies corresponding to separate modes of the advantage function.

Original: Therefore, adInfoHRL shows its superiority on tasks with the multi-modal advantage function and not on tasks with a simple advantage function.
Revised: Therefore, adInfoHRL shows its superiority on tasks with the multi-modal advantage function and not on tasks with a simple advantage function.

Original: Thus, it is natural that the beneﬁt of adInfoHRL is dependent on the characteristics of the task.
Revised: Thus, it is natural that the beneﬁt of adInfoHRL is dependent on the characteristics of the task.

Original: The outputs of the option network and the activation of options on Walker2d are shown in Figure 3(e)-(f), which visualize the result from adInfoHRL with four options.
Revised: The outputs of the option network and the activation of options on Walker2d are shown in Figure 3(e)-(f), which visualize the result from adInfoHRL with four options.

Original: For visualization, the dimensionality was reduced using t-SNE (van der Maaten & Hinton, 2008).
Revised: For visualization, the dimensionality was reduced using t-SNE (van der Maaten & Hinton, 2008).

Original: The state-action space
Revised: The state-action space

Original:       7LPHVWHSV[       $ Y H U DJHG  U H W X U Q :DONHUGY
Revised:       7LPHVWHSV[       $ Y H U DJHG  U H W X U Q :DONHUGY

Original: [Figure]
Revised: [Figure]

Original:       7LPHVWHSV[      $ Y H U DJHG  U H W X U Q $QWBUOODE 0HWKRG DG,QIR+5/RSWLRQQXP
Revised:       7LPHVWHSV[      $ Y H U DJHG  U H W X U Q $QWBUOODE 0HWKRG DG,QIR+5/RSWLRQQXP

Original: DG,QIR+5/RSWLRQQXPLQIR+5/QR,:
Revised: DG,QIR+5/RSWLRQQXPLQIR+5/QR,:

Original: RSWLRQQXP
Revised: RSWLRQQXP

Original: 7'33
Revised: 7'33

Original: [Figure]
Revised: [Figure]

Original:      
Revised:      

Original: [Figure]
Revised: [Figure]

Original:  
Revised:  

Original:    $ Y H U DJHG  U H W X U Q +DOI&KHHWDKY
Revised:    $ Y H U DJHG  U H W X U Q +DOI&KHHWDKY

Original: [Figure]
Revised: [Figure]

Original:      
Revised:      

Original: [Figure]
Revised: [Figure]

Original:         $ Y H U DJHG  U H W X U Q +RSSHUY
Revised:         $ Y H U DJHG  U H W X U Q +RSSHUY

Original: [Figure]
Revised: [Figure]

Original: is clearly divided into separate domains in Figure 3(e).
Revised: is clearly divided into separate domains in Figure 3(e).

Original: As shown in Figure 3(f), the options are activated in different domains of the state space, which indicates that diverse options are learned by adInfoHRL.
Revised: As shown in Figure 3(f), the options are activated in different domains of the state space, which indicates that diverse options are learned by adInfoHRL.

Original:  R ELATED W ORK AND D ISCUSSION
Revised:  R ELATED W ORK AND D ISCUSSION

Original: Past studies have proposed several ways to deal with the latent variable in HRL.
Revised: Past studies have proposed several ways to deal with the latent variable in HRL.

Original: The recent work by Smith et al.
Revised: The recent work by Smith et al.

Original: (2018) proposed inferred option policy gradients (IOPG), which is derived as an extension of policy gradient to the option framework.
Revised: (2018) proposed inferred option policy gradients (IOPG), which is derived as an extension of policy gradient to the option framework.

Original: Nachum et al.
Revised: Nachum et al.

Original: (2018) recently proposed off-policy target correction for HRL on goal-oriented tasks, where a higher-level policy instructs a lower-level policy by generating the goal signal instead of an inferred latent variable.
Revised: (2018) recently proposed off-policy target correction for HRL on goal-oriented tasks, where a higher-level policy instructs a lower-level policy by generating the goal signal instead of an inferred latent variable.

Original: A popular approach for learning the latent variable in HRL is the variational approach.
Revised: A popular approach for learning the latent variable in HRL is the variational approach.

Original: The recent work by Haarnoja et al.
Revised: The recent work by Haarnoja et al.

Original: (2018a) is based on soft actor critic (Haarnoja et al., 2018b), and the latent variable is inferred using the variational approach.
Revised: (2018a) is based on soft actor critic (Haarnoja et al., 2018b), and the latent variable is inferred using the variational approach.

Original: The work by Hausman et al.
Revised: The work by Hausman et al.

Original: (2018) is also closely related to the variational approach, and they proposed a method for learning a latent variable of a hierarchical policy via a variational bound.
Revised: (2018) is also closely related to the variational approach, and they proposed a method for learning a latent variable of a hierarchical policy via a variational bound.

Original: On the contrary, our method learns the latent variable by maximizing MI with advantage-weighted importance.
Revised: On the contrary, our method learns the latent variable by maximizing MI with advantage-weighted importance.

Original: Recent studies by Gregor et al.
Revised: Recent studies by Gregor et al.

Original: Florensa et al.
Revised: Florensa et al.

Original: Eysenbach et al.
Revised: Eysenbach et al.

Original: (2018) also considered the MI in their formulation.
Revised: (2018) also considered the MI in their formulation.

Original: In these methods, MI between the state and the latent variable is considered so as to obtain diverse behaviors.
Revised: In these methods, MI between the state and the latent variable is considered so as to obtain diverse behaviors.

Original: Our approach is different from the previous studies in the sense that we employ MI between the latent variable and the state-action pairs, which leads to the division of the state-action space instead of considering only the state space.
Revised: Our approach is different from the previous studies in the sense that we employ MI between the latent variable and the state-action pairs, which leads to the division of the state-action space instead of considering only the state space.

Original: We think that dividing the state-action space is an efﬁcient approach when the advantage function is multi-modal, as depicted in Figure 1.
Revised: We think that dividing the state-action space is an efﬁcient approach when the advantage function is multi-modal, as depicted in Figure 1.

Original: InfoGAIL proposed by Li et al.
Revised: InfoGAIL proposed by Li et al.

Original: (2017) learns the interpretable representation of the state-action space via MI maximization.
Revised: (2017) learns the interpretable representation of the state-action space via MI maximization.

Original: InfoGAIL can be interpreted as a method that divides the state-action space based on the density induced by an expert’s policy by maximizing the regularized MI objective.
Revised: InfoGAIL can be interpreted as a method that divides the state-action space based on the density induced by an expert’s policy by maximizing the regularized MI objective.

Original: In this sense, it is closely related to our method, although their problem setting is imitation learning (Osa et al., 2018a), which is different from our HRL problem setting.
Revised: In this sense, it is closely related to our method, although their problem setting is imitation learning (Osa et al., 2018a), which is different from our HRL problem setting.

Original: Our method can be interpreted as an approach that divides the state-action space based on the MI criterion.
Revised: Our method can be interpreted as an approach that divides the state-action space based on the MI criterion.

Original: This concept is related to that of Divide and Conquer (DnC) proposed by Ghosh et al.
Revised: This concept is related to that of Divide and Conquer (DnC) proposed by Ghosh et al.

Original: (2018), although DnC clusters the initial states and does not consider switching between option policies during the execution of a single trajectory.
Revised: (2018), although DnC clusters the initial states and does not consider switching between option policies during the execution of a single trajectory.

Original: In this study we developed adInfoHRL based on deterministic option policies.
Revised: In this study we developed adInfoHRL based on deterministic option policies.

Original: However, the concept of dividing the state-action space via advantage-weighted importance can be applied to stochastic policy gradients as well.
Revised: However, the concept of dividing the state-action space via advantage-weighted importance can be applied to stochastic policy gradients as well.

Original: Further investigation in this direction is necessary in future work.
Revised: Further investigation in this direction is necessary in future work.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We proposed a novel HRL method, hierarchical reinforcement learning via advantage-weighted information maximization.
Revised: We proposed a novel HRL method, hierarchical reinforcement learning via advantage-weighted information maximization.

Original: In our framework, the latent variable of a hierarchical policy is learned as a discrete latent representation of the state-action space.
Revised: In our framework, the latent variable of a hierarchical policy is learned as a discrete latent representation of the state-action space.

Original: Our HRL framework is derived by considering deterministic option policies and by leveraging the analogy between the gating policy for HRL and a monolithic policy for the standard RL.
Revised: Our HRL framework is derived by considering deterministic option policies and by leveraging the analogy between the gating policy for HRL and a monolithic policy for the standard RL.

Original: The results of the experiments indicate that adInfoHRL can learn diverse options on continuous control tasks.
Revised: The results of the experiments indicate that adInfoHRL can learn diverse options on continuous control tasks.

Original: Our results also suggested that our approach can improve the performance of TD3 in certain problem domains
Revised: Our results also suggested that our approach can improve the performance of TD3 in certain problem domains

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: MS was partially supported by KAKENHI 17H00757.
Revised: MS was partially supported by KAKENHI 17H00757.

Original: A M UTUAL I NFORMATION WITH A DVANTAGE -W EIGHTED I MPORTANCE
Revised: A M UTUAL I NFORMATION WITH A DVANTAGE -W EIGHTED I MPORTANCE

Original: The mutual information (MI) between the latent variable o and the state action pair ( s , a ) is deﬁned as
Revised: The mutual information (MI) between the latent variable o and the state action pair ( s , a ) is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where H ( o ) =
Revised: where H ( o ) =

Original: (cid:82) p ( o ) log p ( o )
Revised: (cid:82) p ( o ) log p ( o )

Original: d o and H
Revised: d o and H

Original: ( o | s , a ) =
Revised: ( o | s , a ) =

Original: (cid:82) p ( o | s , a ) log p ( o | s , a ) d o .
Revised: (cid:82) p ( o | s , a ) log p ( o | s , a ) d o .

Original: We make the empirical estimate of MI employed by Gomes et al.
Revised: We make the empirical estimate of MI employed by Gomes et al.

Original: ; Hu et al.
Revised: ; Hu et al.

Original: and modify it to employ the importance weight.
Revised: and modify it to employ the importance weight.

Original: The empirical estimate of MI with respect to the density induced by a policy π is given by
Revised: The empirical estimate of MI with respect to the density induced by a policy π is given by

Original: [Equation]
Revised: [Equation]

Original: We consider the case where we have samples collected by a behavior policy
Revised: We consider the case where we have samples collected by a behavior policy

Original: β
Revised: β

Original: ( s | a ) and need to estimate MI with respect to the density induced by policy π .
Revised: ( s | a ) and need to estimate MI with respect to the density induced by policy π .

Original: Given a model p ( o | s , a ; η ) parameterized by vector η , p ( o ) can be rewritten as
Revised: Given a model p ( o | s , a ; η ) parameterized by vector η , p ( o ) can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: where W ( s , a ) =
Revised: where W ( s , a ) =

Original: p π ( s , a ) p β ( s , a ) is the importance weight.
Revised: p π ( s , a ) p β ( s , a ) is the importance weight.

Original: Therefore, the empirical estimate of p ( o ) with respect to the density induced by a policy π is given by
Revised: Therefore, the empirical estimate of p ( o ) with respect to the density induced by a policy π is given by

Original: [Equation]
Revised: [Equation]

Original: where ˜ W ( s , a )
Revised: where ˜ W ( s , a )

Original: [Equation]
Revised: [Equation]

Original: is the normalized importance weight.
Revised: is the normalized importance weight.

Original: Likewise, the conditional entropy with respect to the density induced by a policy π is given by
Revised: Likewise, the conditional entropy with respect to the density induced by a policy π is given by

Original: [Equation]
Revised: [Equation]

Original: Therefore, the empirical estimate of the conditional entropy with respect to the density induced by a policy π is given by
Revised: Therefore, the empirical estimate of the conditional entropy with respect to the density induced by a policy π is given by

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Thus, the empirical estimates of MI can be computed by Equations (21), (23) and (27).
Revised: Thus, the empirical estimates of MI can be computed by Equations (21), (23) and (27).

Original: B D ERIVATION OF THE STATE - VALUE FUNCTION
Revised: B D ERIVATION OF THE STATE - VALUE FUNCTION

Original: In HRL, the value function is given by
Revised: In HRL, the value function is given by

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since option policies are deterministic given by µ o θ ( s ) , the state-value function is given by
Revised: Since option policies are deterministic given by µ o θ ( s ) , the state-value function is given by

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: C E XPERIMENTAL D ETAILS
Revised: C E XPERIMENTAL D ETAILS

Original: We performed evaluations using benchmark tasks in the OpenAI Gym platform (Brockman et al., 2016) with Mujoco physics simulator (Todorov et al., 2012).
Revised: We performed evaluations using benchmark tasks in the OpenAI Gym platform (Brockman et al., 2016) with Mujoco physics simulator (Todorov et al., 2012).

Original: Hyperparameters of reinforcement learning methods used in the experiment are shown in Tables 1-3.
Revised: Hyperparameters of reinforcement learning methods used in the experiment are shown in Tables 1-3.

Original: For exploration, both adInfoHRL and TD3 used the clipped noise drawn from the normal distribution as (cid:15) ∼ clip (cid:0) N (0 , σ ) , − c, c (cid:1) , where σ = 0 .
Revised: For exploration, both adInfoHRL and TD3 used the clipped noise drawn from the normal distribution as (cid:15) ∼ clip (cid:0) N (0 , σ ) , − c, c (cid:1) , where σ = 0 .

Original: 2 and c = 0 . For hyperparameters of PPO, we used the default values in OpenAI baselines (Dhariwal et al., 2017).
Revised: 2 and c = 0 . For hyperparameters of PPO, we used the default values in OpenAI baselines (Dhariwal et al., 2017).

Original: For the Walker2d, HalfCheetah, and Hopper tasks, we used the
Revised: For the Walker2d, HalfCheetah, and Hopper tasks, we used the

Original: Walker2d-v1, HalfCHeetah-v1, and Hopper-v1 in the OpenAI Gym, respectively.
Revised: Walker2d-v1, HalfCHeetah-v1, and Hopper-v1 in the OpenAI Gym, respectively.

Original: For the Ant task, we used the AntEnv implemented in the rllab (Duan et al., 2016).
Revised: For the Ant task, we used the AntEnv implemented in the rllab (Duan et al., 2016).

Original: When training a policy with
Revised: When training a policy with

Original: AdInfoHRL, infoHRL, and TD3, critics are trained once per time step, and actors are trained once every after two updates of the critics.
Revised: AdInfoHRL, infoHRL, and TD3, critics are trained once per time step, and actors are trained once every after two updates of the critics.

Original: The source code is available at https://github.com/
Revised: The source code is available at https://github.com/

Original: TakaOsa/adInfoHRL .
Revised: TakaOsa/adInfoHRL .

Original: We performed the experiments ﬁve times with different seeds, and reported the averaged test return where the test return was computed once every 5000 time steps by executing 10 episodes without exploration.
Revised: We performed the experiments ﬁve times with different seeds, and reported the averaged test return where the test return was computed once every 5000 time steps by executing 10 episodes without exploration.

Original: When performing the learned policy without exploration, the option was drawn as
Revised: When performing the learned policy without exploration, the option was drawn as

Original: [Equation]
Revised: [Equation]

Original: instead of following the stochastic gating policy in Equations (17).
Revised: instead of following the stochastic gating policy in Equations (17).

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: D A DDITIONAL I NFORMATION ON E XPERIMENTAL R ESULTS
Revised: D A DDITIONAL I NFORMATION ON E XPERIMENTAL R ESULTS

Original: On the HalfCheetah task
Revised: On the HalfCheetah task

Original: [Figure]
Revised: [Figure]

Original: adInfoHRL delivered the best performance with two options.
Revised: adInfoHRL delivered the best performance with two options.

Original: The distribution of options on HalfCheetah0v1 after one million steps is shown in Figure 4.
Revised: The distribution of options on HalfCheetah0v1 after one million steps is shown in Figure 4.

Original: Although the state-action space is evenly divided
Revised: Although the state-action space is evenly divided

Original: [Figure]
Revised: [Figure]

Original: the options are not evenly activated.
Revised: the options are not evenly activated.

Original: This behavior can occur because the state-action space is divided based on the density induced by the behavior policy while the activation of options is determined based on the quality of the option policies in a given state.
Revised: This behavior can occur because the state-action space is divided based on the density induced by the behavior policy while the activation of options is determined based on the quality of the option policies in a given state.

Original: Moreover, an even division in the action-state space is not necessarily the even division in the state space.
Revised: Moreover, an even division in the action-state space is not necessarily the even division in the state space.

Original: The activation of the options over time is shown in Figure 5.
Revised: The activation of the options over time is shown in Figure 5.

Original: It is clear that one of the option corresponds to the stable running phase and the other corresponds to the phase for recovering from unstable states.
Revised: It is clear that one of the option corresponds to the stable running phase and the other corresponds to the phase for recovering from unstable states.

Original: [Figure]
Revised: [Figure]

Original: The distribution of four options on the Ant-rllab task after one million steps is shown in Figure 6.
Revised: The distribution of four options on the Ant-rllab task after one million steps is shown in Figure 6.

Original: Four options are activated in the different domains of the state space.
Revised: Four options are activated in the different domains of the state space.

Original: The activation of the options over time on the Ant-rllab task is shown in Figure 7.
Revised: The activation of the options over time on the Ant-rllab task is shown in Figure 7.

Original: While four options are actively used in the beginning of the episode, two (blue and yellow) options are mainly activated during the stable locomotion.
Revised: While four options are actively used in the beginning of the episode, two (blue and yellow) options are mainly activated during the stable locomotion.

Original: Since the Ant task implemented in rllab is known to be harder than the Ant-v1 implemented in the
Revised: Since the Ant task implemented in rllab is known to be harder than the Ant-v1 implemented in the

Original: OpenAI gym, we reported the result of the Ant task in rllab in the main manuscript.
Revised: OpenAI gym, we reported the result of the Ant task in rllab in the main manuscript.

Original: Here, we report the result of the Ant-v1 task implemented in the OpenAI gym.
Revised: Here, we report the result of the Ant-v1 task implemented in the OpenAI gym.

Original: On the Ant-v1 task, adInfoHRL yielded the best performance with two options.
Revised: On the Ant-v1 task, adInfoHRL yielded the best performance with two options.

Original: The performance of adInfoHRL with two options is comparable to that of TD3 on Ant-v1.
Revised: The performance of adInfoHRL with two options is comparable to that of TD3 on Ant-v1.

Original: This result indicates that the Ant-v1 task does not require a hierarchical policy structure, while a hierarchical policy improves the performance of learning on
Revised: This result indicates that the Ant-v1 task does not require a hierarchical policy structure, while a hierarchical policy improves the performance of learning on

Original: Ant-rllab.
Revised: Ant-rllab.

Original: The distribution of options on Ant-v1 task after one million steps is shown in Figure 8.
Revised: The distribution of options on Ant-v1 task after one million steps is shown in Figure 8.

Original: The activation of the options over time is shown in Figure 9.
Revised: The activation of the options over time is shown in Figure 9.

Original: It is evident that two option policies on the Ant-v1 task corresponded to different postures of the agent.
Revised: It is evident that two option policies on the Ant-v1 task corresponded to different postures of the agent.

Original: A recent study on HRL by Smith et al.
Revised: A recent study on HRL by Smith et al.

Original: (2018) reported the performance of IOPG on Walker2d-v1,
Revised: (2018) reported the performance of IOPG on Walker2d-v1,

Original: Hopper-v1, and HalfCheetah-v1.
Revised: Hopper-v1, and HalfCheetah-v1.

Original: The study by Haarnoja et al.
Revised: The study by Haarnoja et al.

Original: (2018a) reported the performance of SAC-LSP on Walker2d-v1, Hopper-v1, HalfCheetah-v1, and Ant-rllab.
Revised: (2018a) reported the performance of SAC-LSP on Walker2d-v1, Hopper-v1, HalfCheetah-v1, and Ant-rllab.

Original: A comparison of performance between our method, IOPG, and SAC-LSP is summarized in Table 4.
Revised: A comparison of performance between our method, IOPG, and SAC-LSP is summarized in Table 4.

Original: We report the performance after 1 million steps.
Revised: We report the performance after 1 million steps.

Original: It is worth noting that adInfoHRL outperformed IOPG on these tasks in terms of the achieved return, although we are aware that the qualitative performance is also important in HRL.
Revised: It is worth noting that adInfoHRL outperformed IOPG on these tasks in terms of the achieved return, although we are aware that the qualitative performance is also important in HRL.

Original: AdInfoHRL outperformed SAC-LSP on Walker2d-v1 and Ant-rllab, and SAC-LSP shows its superiority on HalfCheetah-v1 and Hopper-v1.
Revised: AdInfoHRL outperformed SAC-LSP on Walker2d-v1 and Ant-rllab, and SAC-LSP shows its superiority on HalfCheetah-v1 and Hopper-v1.

Original: However, the results of SAC-LSP were obtained by using reward scaling, which was not used in the evaluation of adInfoHRL.
Revised: However, the results of SAC-LSP were obtained by using reward scaling, which was not used in the evaluation of adInfoHRL.

Original: Therefore, further experiments are necessary for fair comparison under the same condition.
Revised: Therefore, further experiments are necessary for fair comparison under the same condition.

Original:    
Revised:    

Original: [Figure]
Revised: [Figure]

Original: 7LPHVWHSV[     $ Y H U DJHG  U H W X U Q $QWY 0HWKRG DG,QIR+5/RSWLRQQXP
Revised: 7LPHVWHSV[     $ Y H U DJHG  U H W X U Q $QWY 0HWKRG DG,QIR+5/RSWLRQQXP

Original: DG,QIR+5/RSWLRQQXPLQIR+5/QR,:
Revised: DG,QIR+5/RSWLRQQXPLQIR+5/QR,:

Original: RSWLRQQXP
Revised: RSWLRQQXP

Original: 7'33
Revised: 7'33

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: – ≈
Revised: – ≈