Original: Disentangling Controlled Effects for Hierarchical Reinforcement Learning
Revised: Disentangling Controlled Effects for Hierarchical Reinforcement Learning

Original: Author names withheld
Revised: Bernhard Sch¨olkopf

Original: Editor: Under Review for CLeaR
Revised: Caroline Uhler Zhang

Original: Abstract
Revised: Abstract

Original: Exploration and credit assignment are still challenging problems for RL agents under sparse rewards.
Revised: Exploration and credit assignment are still challenging problems for RL agents under sparse rewards.

Original: We argue that these challenges arise partly due to the intrinsic rigidity of operating at the level of actions.
Revised: We argue that these challenges arise partly due to the intrinsic rigidity of operating at the level of actions.

Original: Actions can precisely deﬁne how to perform an activity but are ill-suited to describe what activity to perform.
Revised: Actions can precisely deﬁne how to perform an activity but are ill-suited to describe what activity to perform.

Original: Instead, controlled effects describe transformations in the environment caused by the agent.
Revised: Instead, controlled effects describe transformations in the environment caused by the agent.

Original: These transformations are inherently composable and temporally abstract, making them ideal for descriptive tasks.
Revised: These transformations are inherently composable and temporally abstract, making them ideal for descriptive tasks.

Original: This work introduces CEHRL 1 , a hierarchical method leveraging the compositional nature of controlled effects to expedite the learning of task-speciﬁc behavior and aid exploration.
Revised: This work introduces CEHRL 1 , a hierarchical method leveraging the compositional nature of controlled effects to expedite the learning of task-speciﬁc behavior and aid exploration.

Original: Borrowing counterfactual and normality measures from causal literature, CEHRL learns an implicit hierarchy of transformations an agent can perform on the environment.
Revised: Borrowing counterfactual and normality measures from causal literature, CEHRL learns an implicit hierarchy of transformations an agent can perform on the environment.

Original: This hierarchy allows a high-level policy to set temporally abstract goals and, by doing so, long-horizon credit assignment.
Revised: This hierarchy allows a high-level policy to set temporally abstract goals and, by doing so, long-horizon credit assignment.

Original: Experimental results show that using effects instead of actions provides a more efﬁcient exploration mechanism.
Revised: Experimental results show that using effects instead of actions provides a more efﬁcient exploration mechanism.

Original: Moreover, by leveraging prior knowledge in the hierarchy, CEHRL assigns credit to few effects instead of many actions and consequently learns tasks more rapidly.
Revised: Moreover, by leveraging prior knowledge in the hierarchy, CEHRL assigns credit to few effects instead of many actions and consequently learns tasks more rapidly.

Original: 1. Introduction
Revised: 1. Introduction

Original: Value-based methods for reinforcement learning (RL) (Sutton and Barto, 1998) have achieved impressive results in environments with dense rewards (Mnih et al., 2013).
Revised: Value-based methods for reinforcement learning (RL) (Sutton and Barto, 1998) have achieved impressive results in environments with dense rewards (Mnih et al., 2013).

Original: These methods learn by estimating the causal effect of actions on the reward function.
Revised: These methods learn by estimating the causal effect of actions on the reward function.

Original: However, this type of learning is particularly ineffective in environments with sparse rewards where no rewards are given for long periods of time, thus requiring to collect vast amounts of experience, each providing little to no learning.
Revised: However, this type of learning is particularly ineffective in environments with sparse rewards where no rewards are given for long periods of time, thus requiring to collect vast amounts of experience, each providing little to no learning.

Original: A promising solution is to use hierarchical RL (Sutton et al., 1999) to learn reusable skills.
Revised: A promising solution is to use hierarchical RL (Sutton et al., 1999) to learn reusable skills.

Original: Skill discovery research has focused on methods based on information theory (Florensa et al., 2017;
Revised: Skill discovery research has focused on methods based on information theory (Florensa et al., 2017;

Original: Eysenbach et al., 2018; Sharma et al., 2019) or intrinsic motivators (Kulkarni et al., 2016; Nachum et al., 2018).
Revised: Eysenbach et al., 2018; Sharma et al., 2019) or intrinsic motivators (Kulkarni et al., 2016; Nachum et al., 2018).

Original: In contrast, we are motivated by studies in the ﬁeld of developmental psychology, indicating that children use the consequences of their actions to learn how to control their environment (Goodman et al., 2007; Buchsbaum et al., 2012, 2015).
Revised: In contrast, we are motivated by studies in the ﬁeld of developmental psychology, indicating that children use the consequences of their actions to learn how to control their environment (Goodman et al., 2007; Buchsbaum et al., 2012, 2015).

Original: We conceptualize skills as useful changes in the environment; thus, we propose agents that learn by estimating the causal effect of an action on the state.
Revised: We conceptualize skills as useful changes in the environment; thus, we propose agents that learn by estimating the causal effect of an action on the state.

Original: Intuitively, these effects describe ways in which the agent can change its environment.
Revised: Intuitively, these effects describe ways in which the agent can change its environment.

Original: Since the agent does not control every change, we adopt counterfactual measures from causal literature (Pearl, 2009; Halpern, 2016) to disentangle effects caused by the agent from effects caused by other dynamics, e.g., other agents, wind, etc.
Revised: Since the agent does not control every change, we adopt counterfactual measures from causal literature (Pearl, 2009; Halpern, 2016) to disentangle effects caused by the agent from effects caused by other dynamics, e.g., other agents, wind, etc.

Original: A key aspect of controlled effects is their compositionality, i.e., to perform a complex effect, an agent needs to combine simpler effects, which are also composed of more simple effects, all the way

Original: [Figure]
Revised: [Figure]

Original:  down to actions.
Revised: A key aspect of controlled effects is their compositionality, i.e., to perform a complex effect, an agent needs to combine simpler effects, which are also composed of more simple effects, all the way down to actions.

Original: Figure 1( a ) illustrates the compositional nature of controlled effects, making them temporally abstract.
Revised: Figure 1( a ) illustrates the compositional nature of controlled effects, making them temporally abstract.

Original: In other words, effects may take several, possibly variable, number of actions to be performed (Sutton et al., 1999).
Revised: In other words, effects may take several, possibly variable, number of actions to be performed (Sutton et al., 1999).

Original: Temporal abstraction promises to ease the credit assignment problem by decoupling the agent from a ﬁne-grained time scale.
Revised: Temporal abstraction promises to ease the credit assignment problem by decoupling the agent from a ﬁne-grained time scale.

Original: To this end, we introduce CEHRL (Controlled Effects for Hierarchical
Revised: To this end, we introduce CEHRL (Controlled Effects for Hierarchical

Original: RL), a method that builds an implicit hierarchy of controlled effects serving a twofold purpose.
Revised: RL), a method that builds an implicit hierarchy of controlled effects serving a twofold purpose.

Original: First, instead of exploring the environment via random actions, CEHRL relies on random effect exploration.
Revised: First, instead of exploring the environment via random actions, CEHRL relies on random effect exploration.

Original: Consider the scenario presented in Figure 1( b ), where an agent has already learned some basic effects like moving the agent and the boxes.
Revised: Consider the scenario presented in Figure 1( b ), where an agent has already learned some basic effects like moving the agent and the boxes.

Original: Turning on the light is more likely to happen the more boxes are moved.
Revised: Turning on the light is more likely to happen the more boxes are moved.

Original: Consequently, random effect exploration is based on the idea that the agent can discover and learn more complex effects by combining basic effects, continuously enriching the hierarchy in an unsupervised manner.
Revised: Consequently, random effect exploration is based on the idea that the agent can discover and learn more complex effects by combining basic effects, continuously enriching the hierarchy in an unsupervised manner.

Original: Second, a rich hierarchy facilitates the learning of task-speciﬁc behavior efﬁciently.
Revised: Second, a rich hierarchy facilitates the learning of task-speciﬁc behavior efﬁciently.

Original: By reusing prior knowledge, the agent quickly discovers and learns from reward.
Revised: By reusing prior knowledge, the agent quickly discovers and learns from reward.

Original: CEHRL relies on counterfactuals to identify the causal effect of an agent on the environment.
Revised: CEHRL relies on counterfactuals to identify the causal effect of an agent on the environment.

Original: Then, it uses a Variational Autoencoder (VAE) (Kingma and Welling, 2013) to represent the hierarchy of transformations an agent can cause on the environment.
Revised: Then, it uses a Variational Autoencoder (VAE) (Kingma and Welling, 2013) to represent the hierarchy of transformations an agent can cause on the environment.

Original: A goal-conditioned policy translates controlled effects from the VAE into a sequence of actions.
Revised: A goal-conditioned policy translates controlled effects from the VAE into a sequence of actions.

Original: The VAE and policy work in conjunction to explore the environment by 1) randomly selecting effects from the hierarchy as goals for the policy and 2) using newly discovered controlled effects as learning signal to the VAE.
Revised: The VAE and policy work in conjunction to explore the environment by 1) randomly selecting effects from the hierarchy as goals for the policy and 2) using newly discovered controlled effects as learning signal to the VAE.

Original: Finally, to learn a particular task, CEHRL trains a state-effect value function to score how valuable a given controlled effect is to maximize the environment’s reward.
Revised: Finally, to learn a particular task, CEHRL trains a state-effect value function to score how valuable a given controlled effect is to maximize the environment’s reward.

Original: Our main contributions are:
Revised: Our main contributions are:

Original: • We establish a link between the advantage function and causality.
Revised: • We establish a link between the advantage function and causality.

Original: • A novel method based on counterfactuals to identify effects controlled by the agent.
Revised: • A novel method based on counterfactuals to identify effects controlled by the agent.

Original: • An unsupervised approach to continually learna hierarchy of controlled effects.
Revised: unsupervised approach to continually learn hierarchy of controlled effects

Original: • An algorithm that uses this hierarchy to rapidly learn task-speciﬁc behavior from rewards.
Revised: algorithm that uses this hierarchy to rapidly learn task-speciﬁc behavior from rewards.

Original: 2. Disentangling Controlled Effects
Revised: 2. Disentangling Controlled Effects

Original: Pearl et al.
Revised: Pearl et al.

Original: (2016) provide an intuitive deﬁnition of cause-effect relations: “ A variable X is a cause of a variable Y if Y, in any way, relies on X for its value ”.
Revised: (2016) provide an intuitive deﬁnition of cause-effect relations: “ A variable X is a cause of a variable Y if Y, in any way, relies on X for its value ”.

Original: For example, if the life of an agent relies on eating food, eating food has a causal effect on the agent’s life.
Revised: For example, if the life of an agent relies on eating food, eating food has a causal effect on the agent’s life.

Original: Actual causality proposed in
Revised: Actual causality proposed in

Original: Halpern (2016) studies causal relations between individual events of X and Y .
Revised: Halpern (2016) studies causal relations between individual events of X and Y .

Original: For example, would taking action a have an effect on the agent’s life at a particular state?
Revised: For example, would taking action a have an effect on the agent’s life at a particular state?

Original: Figure 3 shows the causal graph of a typical RL setting, where a state s has a causal effect on both the agent’s choice of action and the next state s (cid:48) .
Revised: Figure 3 shows the causal graph of a typical RL setting, where a state s has a causal effect on both the agent’s choice of action and the next state s (cid:48) .

Original: Similarly, an action has an effect on the next state.
Revised: Similarly, an action has an effect on the next state.

Revised: In our experiments, we assume deterministic transitions, discrete actions and s ∈ N p is a p -dimensional vector.

Original: We deﬁne the total effect e t as the change in the environment’s state when transitioning from s to s (cid:48) due to taking action a i.e. e t ( s, a ) ≡ s (cid:48) − s .
Revised: We deﬁne the total effect e t as the change in the environment’s state when transitioning from s to s (cid:48) due to taking action a i.e. e t ( s, a ) ≡ s (cid:48) − s .

Original: This section aims to disentangle controlled effects e c caused by the agent’s action from total effects.
Revised: This section aims to disentangle controlled effects e c caused by the agent’s action from total effects.

Original: The individual causal effect (ICE) of a variable X on another variable Y can be measured by comparing counterfactual worlds (Pearl, 2009)
Revised: The individual causal effect (ICE) of a variable X on another variable Y can be measured by comparing counterfactual worlds (Pearl, 2009)

Original: [Equation]
Revised: [Equation]

Original: (1) where Y xi reads as ”what would the value of Y i be if X had taken value x ”.
Revised: ) where Y xi reads as ”what would the value of Y i be if X had taken value x ”.

Original: Similarly, Y ˜ xi describes the value of Y i when X does not take the value of x .
Revised: Similarly, Y ˜ xi describes the value of Y i when X does not take the value of x .

Original: Intuitively, Eq. 1 compares the world where the event x happened against an alternative world where event x had not happened.
Revised: Intuitively, Eq. 1 compares the world where the event x happened against an alternative world where event x had not happened.

Original: The fundamental problem of causal inference (Holland, 1986) states that this last world cannot be observed, therefore needs to be imagined.
Revised: The fundamental problem of causal inference (Holland, 1986) states that this last world cannot be observed, therefore needs to be imagined.

Original: Halpern and Hitchcock (2011, 2014) propose to compare what happened with what normally would happen by constructing a normative world
Revised: Halpern and Hitchcock (2011, 2014) propose to compare what happened with what normally would happen by constructing a normative world

Original: [Equation]
Revised: [Equation]

Original: where β Y i is the value Y i would normally take.
Revised: where β Y i is the value Y i would normally take.

Original: Of course, such value is contingent to the notion of normality used, which is for us to deﬁne.
Revised: Of course, such value is contingent to the notion of normality used, which is for us to deﬁne.

Original: Note that contrary to Eq.
Revised: Note that contrary to Eq.

Original: 1, this formulation uses the magnitude and direction of the effect, thus computing the difference between worlds.
Revised: 1, this formulation uses the magnitude and direction of the effect, thus computing the difference between worlds.

Original: A common use in reinforcement learning of this formulation is to compute the causal effect of an action on the return G
Revised: A common use in reinforcement learning of this formulation is to compute the causal effect of an action on the return G

Original: [Equation]
Revised: [Equation]

Original: G a is the return the agent would get if action a were to be taken.
Revised: G a is the return the agent would get if action a were to be taken.

Original: Usually, G a is estimated using a state-action value function Q ( s, a ) and the choice of normality for G ˜ a is the expected return estimated with the state-value function V ( s ) .
Revised: Usually, G a is estimated using a state-action value function Q ( s, a ) and the choice of normality for G ˜ a is the expected return estimated with the state-value function V ( s ) .

Original: This choice gives the advantage function A ( s, a ) which estimates the causal effect of an action on the return.
Revised: This choice gives the advantage function A ( s, a ) which estimates the causal effect of an action on the return.

Original: As described in Richard S. and Barto (2018), General Value Functions include general knowledge of the world, leaving the return as special case.
Revised: As described in Richard S. and Barto (2018), General Value Functions include general knowledge of the world, leaving the return as special case.

Original: Here, we follow the same idea and use Eq. 2 to identify the causal effect of an action on the state
Revised: Here, we follow the same idea and use Eq. 2 to identify the causal effect of an action on the state

Original: [Equation]
Revised: [Equation]

Original: where as before, e t ( s, ˜ a ) needs to be imagined.
Revised: where as before, e t ( s, ˜ a ) needs to be imagined.

Original: Note that deﬁning ˜ a as a special do-nothing action would not work since an agent can still have an effect on the environment by doing nothing.
Revised: Note that deﬁning ˜ a as a special do-nothing action would not work since an agent can still have an effect on the environment by doing nothing.

Original: For example, an agent doing nothing when an enemy is moving towards it would have a causal effect on its life.
Revised: For example, an agent doing nothing when an enemy is moving towards it would have a causal effect on its life.

Original: Our choice of normality is to compute e t ( s, ˜ a ) as the most common world among every possible world, i.e. we compute e t ( s, ˜ a ) as the mode over all actions and consequently e c
Revised: Our choice of normality is to compute e t ( s, ˜ a ) as the most common world among every possible world, i.e. we compute e t ( s, ˜ a ) as the mode over all actions and consequently e c

Original: ( s, a ) as
Revised: ( s, a ) as

Original: [Equation]
Revised: [Equation]

Original: In practice we do not have access to every world and cannot compute Eq.
Revised: In practice we do not have access to every world and cannot compute Eq.

Original: directly.
Revised: directly.

Original: We approximate e t ( s, a ) using a neural network ˆ e t ( s [Equation] a ) trained using experiences (  s [Equation]
Revised: We approximate e t ( s, a ) using a neural network ˆ e t ( s, a ) trained using experiences ( s, a, s (cid:48) ) and loss

Original: a, s

Original: [Equation]

Original: and loss

Original: [Equation]
Revised: [Equation]

Original: In our experiments, we assume discrete actions and s ∈ N p is a p -dimensional vector.

Original: We discretize total effects to its nearest integer when computing Eq.
Revised: We discretize total effects to its nearest integer when computing Eq.

Original: 5 but train ˆ e t with its continuous output.
Revised: 5 but train ˆ e t with its continuous output.

Original: We provide more details of the training in section 3.
Revised: We provide more details of the training in section 3.

Original: 3. CEHRL
Revised: 3. CEHRL

Original: This section explores how controlled effects identiﬁed as in Eq.
Revised: This section explores how controlled effects identiﬁed as in Eq.

Original: 5 can be used for temporal abstraction.
Revised: 5 can be used for temporal abstraction.

Original: Knowing the changes an agent can perform to the environment is valuable information invariant to the environment’s time scale.
Revised: Knowing the changes an agent can perform to the environment is valuable information invariant to the environment’s time scale.

Original: When operating at the level of actions, reward needs to propagate through many time steps.
Revised: When operating at the level of actions, reward needs to propagate through many time steps.

Original: In the example from Fig.
Revised: In the example from Fig.

Original: 1( a ), the reward gotten when the light turns on would need to travel through the whole bottom of the hierarchy, i.e., actions.
Revised: 1( a ), the reward gotten when the light turns on would need to travel through the whole bottom of the hierarchy, i.e., actions.

Original: Instead, we aim to propagate reward through higher levels of the hierarchy, e.g., turn red and blue switches on.
Revised: Instead, we aim to propagate reward through higher levels of the hierarchy, e.g., turn red and blue switches on.

Original: By doing so, we would be able to reuse prior knowledge and learn from few examples.
Revised: By doing so, we would be able to reuse prior knowledge and learn from few examples.

Original: The following describes CEHRL (see Figure 2), an agent that uses controlled effects to learn temporally abstract effects to operate on an extended time horizon.
Revised: The following describes CEHRL (see Figure 2), an agent that uses controlled effects to learn temporally abstract effects to operate on an extended time horizon.

Original: CEHRL is a hierarchical method composed of a generative model E to provide the different ways an agent could change the environment; a task policy π t ( e goal c | s, e c ) that decides what effect to perform next; and an effectconditioned policy π e ( a | s, e goal c ) that learns how to translate an effect into actions.
Revised: CEHRL is a hierarchical method composed of a generative model E to provide the different ways an agent could change the environment; a task policy π t ( e goal c | s, e c ) that decides what effect to perform next; and an effectconditioned policy π e ( a | s, e goal c ) that learns how to translate an effect into actions.

Original: CEHRL is trained in two stages, 1) the generative model and effect-conditioned policy are trained by exploring the environment to enrich the hierarchy; 2) the task-policy is trained to maximize the environment’s reward by reusing prior knowledge from the hierarchy.
Revised: CEHRL is trained in two stages, 1) the generative model and effect-conditioned policy are trained by exploring the environment to enrich the hierarchy; 2) the task-policy is trained to maximize the environment’s reward by reusing prior knowledge from the hierarchy.

Original: 3.1. Controlled Effects for Unsupervised Exploration
Revised: 3.1. Controlled Effects for Unsupervised Exploration

Original: CEHRL uses random effect exploration to continuously discover novel controlled effects.
Revised: CEHRL uses random effect exploration to continuously discover novel controlled effects.

Original: Random effect exploration is based on the idea that by combining controlled effects, an agent can discover novel and more complex effects.
Revised: Random effect exploration is based on the idea that by combining controlled effects, an agent can discover novel and more complex effects.

Original: In other words, learned effects are stepping stones that facilitate the discovery of novel effects.
Revised: In other words, learned effects are stepping stones that facilitate the discovery of novel effects.

Original: Following Fig.
Revised: Following Fig.

Original: 1( b ), the agent would explore more efﬁciently by reusing previously learned effects which would ease the learning of more complex effects like turning a switch on.
Revised: 1( b ), the agent would explore more efﬁciently by reusing previously learned effects which would ease the learning of more complex effects like turning a switch on.

Original: Random effect exploration samples effects from the learned distribution and sets these effects as goals for the effect-conditioned policy.
Revised: Random effect exploration samples effects from the learned distribution and sets these effects as goals for the effect-conditioned policy.

Original: Furthermore, the resulting controlled effects are used to train this distribution further, creating a continuous learning cycle.
Revised: Furthermore, the resulting controlled effects are used to train this distribution further, creating a continuous learning cycle.

Original: Acting: to perform random effect exploration, we ﬁrst sample N candidate effects e c
Revised: Acting: to perform random effect exploration, we ﬁrst sample N candidate effects e c

Original: = ( e 1 c , . . .
Revised: = ( e 1 c , . . .

Original: , e Nc ) from the distribution of effects E ; then the task policy selects one of these effects as goal e goal c ; and ﬁnally, we unroll the effect-conditioned policy until either the controlled effect for the current step matches the goal or more than K actions have been performed.
Revised: , e Nc ) from the distribution of effects E ; then the task policy selects one of these effects as goal e goal c ; and ﬁnally, we unroll the effect-conditioned policy until either the controlled effect for the current step matches the goal or more than K actions have been performed.

Original: We store experiences of the form ( s, e goal c , a, s (cid:48) , e c ( s, a ) , r (cid:48) , d (cid:48) ) in a prioritized replay memory D (Schaul et al., 
Revised: We store experiences of the form ( s, e goal c , a, s (cid:48) , e c ( s, a ) , r (cid:48) , d (cid:48) ) in a prioritized replay memory D (Schaul et al., 2016).

Original: Instead of using the environment’s terminal state d , we terminate the episode if either the environment’s episode ends, the number of steps trying the goal exceeds a limit K or the goal is reached.
Revised: Instead of using the environment’s terminal state d , we terminate the episode if either the environment’s episode ends, the number of steps trying the goal exceeds a limit K or the goal is reached.

Original: We ignore the extrinsic reward provided by the environment and give a reward of 1 if the goal is reached or a punishment P otherwise.
Revised: We ignore the extrinsic reward provided by the environment and give a reward of 1 if the goal is reached or a punishment P otherwise.

Original: Experiences are augmented with Hindsight Experience Replay (Andrychowicz et al., Additionally,
Revised: Experiences are augmented with Hindsight Experience Replay (Andrychowicz et al., 2017).

Original:  we deal with the intrinsic imbalance between basic and complex effects by using a function g ( e c ) to compute the rarity of an effect and set it as initial priority 
Revised: Additionally, we deal with the intrinsic imbalance between basic and complex effects by using a function g ( e c ) to compute the rarity of an effect and set it as initial priority to each experience.

Original: to each experience. Learning: here we describe CEHRL’s components, the data they require and their training.
Revised: Learning: here we describe CEHRL’s components, the data they require and their training.

Original: • Total effects model: to be able to compute e c ( s, a ) from Eq.
Revised: • Total effects model: to be able to compute e c ( s, a ) from Eq.

Original: 5, we train the network ˆ e t ( s, a ) with the loss deﬁned in Eq.
Revised: 5, we train the network ˆ e t ( s, a ) with the loss deﬁned in Eq.

Original: 6 using samples ( s, a, s (cid:48) ) from the replay memory.
Revised: 6 using samples ( s, a, s (cid:48) ) from the replay memory.

Original: • Distribution of effects: we train a Variational Autoencoder (Kingma and Welling, 2013) to approximate the distribution of controlled effects present in the replay memory.
Revised: • Distribution of effects: we train a Variational Autoencoder (Kingma and Welling, 2013) to approximate the distribution of controlled effects present in the replay memory.

Original: This model is not conditioned on state nor actions, making effects reachable from any state.
Revised: This model is not conditioned on state nor actions, making effects reachable from any state.

Original: • Effect-conditioned policy: we train a neural network Q e ( s, e goal c , a ) to learn the state-effect-action value function using DQN (Mnih et al., 2013) with prioritized experience replay (Schaul et al., 2015), Double Q-learning (van Hasselt et al., 2015) and Dueling Networks (Wang et al., 2015); and use this network as effect-conditioned policy
Revised: • Effect-conditioned policy: we train a neural network Q e ( s, e goal c , a ) to learn the state-effect-action value function using DQN (Mnih et al., 2013) with prioritized experience replay (Schaul et al., 2015), Double Q-learning (van Hasselt et al., 2015) and Dueling Networks (Wang et al., 2015); and use this network as effect-conditioned policy

Original: [Equation]
Revised: [Equation]

Original: • Task policy: to bias exploration as little as possible, our choice of task policy is to choose effects following a uniform distribution
Revised: • Task policy: to bias exploration as little as possible, our choice of task policy is to choose effects following a uniform distribution

Original: [Equation]
Revised: [Equation]

Revised: We deem a goal achieved using the function has achieved ( e step c , e goal c

Revised: [Equation]

Revised: | < T , where

Revised: T is a threshold ﬁxed before training.

Original: In our experiments, we start by training the total effects model only, then we incorporate the distribution of effects to the training, and ﬁnally the effect-conditioned policy.
Revised: In our experiments, we start by training the total effects model only, then we incorporate the distribution of effects to the training, and ﬁnally the effectconditioned policy.

Original: See algorithmand the supplementary materials for more details on the training, hyperparameters and network architectures used in our experiments.
Revised: See algorithm 1 and the supplementary materials for more details on the training, hyperparameters and network architectures used in our experiments.

Original: 3.2. Controlled Effects for Learning from Reward
Revised: 3.2. Controlled Effects for Learning from Reward

Original: In the previous section we chose to implement the task policy as a uniform sampling over effects so our exploration is as unbiased as possible.
Revised: In the previous section we chose to implement the task policy as a uniform sampling over effects so our exploration is as unbiased as possible.

Original: Here, we want to learn how to bias the agent so as to maximize a reward function.
Revised: Here, we want to learn how to bias the agent so as to maximize a reward function.

Original: We do this by learning a Q-value function that estimates the value of state-effect pairs, thus the task policy is implemented as
Revised: We do this by learning a Q-value function that estimates the value of state-effect pairs, thus the task policy is implemented as

Original: [Equation]
Revised: [Equation]

Original: We train Q t using DQN and a prioritized replay buffer with experiences ( s, e goal c
Revised: We train Q t using DQN and a prioritized replay buffer with experiences ( s, e goal c

Original: [Equation]
Revised: [Equation]

Original: s

Original: [Equation]

Original: , e [Equation] c , r (cid:48) , d ) . Here, d is the terminal state provided by the environment and e (cid:48)
Revised: Here, d is the terminal state provided by the environment and e (cid:48)

Original: c are the candidate effects produced by the effect distribution for the next state.
Revised: c are the candidate effects produced by the effect distribution for the next state.

Original: The effect-conditioned policy and task policy work at different time scales, i.e. an “effect step” is composed of multiple “environment steps”, therefore states s and s (cid:48) are consecutive effect steps but may not be consecutive environment steps.
Revised: The effect-conditioned policy and task policy work at different time scales, i.e. an “effect step” is composed of multiple “environment steps”, therefore states s and s (cid:48) are consecutive effect steps but may not be consecutive environment steps.

Original: Since rewards are provided at every environment step, we accumulate rewards given between effect steps into r (cid:48) .
Revised: Since rewards are provided at every environment step, we accumulate rewards given between effect steps into r (cid:48) .

Original: In this phase, we use pre-trained frozen models ˆ e t , E and π e learned in the exploration phase.
Revised: In this phase, we use pre-trained frozen models ˆ e t , E and π e learned in the exploration phase.

Original: See algorithm 2 and supplementary materials for more details.
Revised: See algorithm 2 and supplementary materials for more details.

Original: Require: K ; D Initialize ˆ e t ; E ; π e ; s = s 0 ; d (cid:48)
Revised: Require: K ; D Initialize ˆ e t ; E ; π e ; s = s 0 ; d (cid:48)

Original: =while  training do  E  )  while not d (cid:48) do a ∼ π e ( s,e goal c ) s (cid:48) ,d = step env ( a ) e step c = e c ( s,a ) h = has achieved ( e step c ,e goal c ) r (cid:48)
Revised: =while keep training do e c ∼ E e goal c = π t ( s, e c ) # π t samples candidates uniformly while not d (cid:48) do a ∼ π e ( s,e goal c ) s (cid:48) , d = step env ( a ) e step c = e c ( s,a ) h = has achieved ( e step c ,e goal c ) r (cid:48)

Original: =
Revised: =

Original: compute reward ( h ) d (cid:48)
Revised: compute reward ( h ) d (cid:48)

Original: =
Revised: =

Original: ( d or ( t > K ) or h ) add with her( D , g ( e step c ) , ( s,e goal c ,a,e step c ,s (cid:48) ,r (cid:48) ,d (cid:48) ) )
Revised: ( d or ( t > K ) or h ) add with her( D , g ( e step c ) , ( s,e goal c ,a,e step c ,s (cid:48) ,r (cid:48) ,d (cid:48) ) )

Original: s = s (cid:48) train if needed ( ˆ e t , D ) train if needed ( E, D ) train if needed ( π e , D ) end while end while
Revised: s = s (cid:48) train if needed ( ˆ e t , D ) train if needed ( E, D ) train if needed ( π e , D ) end while end while

Original: Require: K ; D Load pre-trained ˆ e t ; E and π e
Revised: Require: K ; D Load pre-trained ˆ e t ; E and π e

Original: Initialize π t ; s = s 0 ; d (cid:48)
Revised: Initialize π t ; s = s 0 ; d (cid:48)

Original: = 0 while keep training do e c ∼ E e goal c
Revised: = 0 while keep training do e c ∼ E e goal c

Original: =
Revised: =

Original: π t ( s, e c ) s g
Revised: π t ( s, e c ) s g

Original: = s while not d (cid:48) do a
Revised: = s while not d (cid:48) do a

Original: = π e ( s,e goal c ) s (cid:48) ,r,d = step ( a ) e step c = e c ( s,a ) h = has achieved ( e step c ,e goal c ) r (cid:48)
Revised: = π e ( s,e goal c ) s (cid:48) ,r,d = step ( a ) e step c = e c ( s,a ) h = has achieved ( e step c ,e goal c ) r (cid:48)

Original: =
Revised: =

Original: r (cid:48)
Revised: r (cid:48)

Original: + r d (cid:48) =
Revised: + r d (cid:48) =

Original: ( d or ( t > K ) or h ) s
Revised: ( d or ( t > K ) or h ) s

Original: = s (cid:48) end while if h then add( D , ( s g ,e goal c ,e step c ,s, e c ,r (cid:48) ,d ) ) end

Original: if r (cid:48) = 
Revised: = s (cid:48)

Original: r (cid:48) ∗ (1 − d ) train if needed ( π t , D ) end while
Revised: [Table]

Original: 4. Experiments
Revised: 4. Experiments

Original: The following experiments provide empirical answers to the following questions: a) credit assignment - can controlled effects ease the task of credit assignment?
Revised: The following experiments provide empirical answers to the following questions: a) credit assignment - can controlled effects ease the task of credit assignment?

Original: b) exploration - How does exploration with random effects differ from using random actions?
Revised: b) exploration - How does exploration with random effects differ from using random actions?

Original: c) cost - how expensive is to operate with effects?
Revised: c) cost - how expensive is to operate with effects?

Original: d) disentanglement - Is there any beneﬁt to use controlled effects?
Revised: d) disentanglement - Is there any beneﬁt to use controlled effects?

Original: First, we show how using a hierarchy of controlled effects can help with the credit assignment and exploration problems.
Revised: First, we show how using a hierarchy of controlled effects can help with the credit assignment and exploration problems.

Original: Secondly, we show how the cost of building this hierarchy is amortized when the number of tasks or their complexity increases.
Revised: Secondly, we show how the cost of building this hierarchy is amortized when the number of tasks or their complexity increases.

Original: We use a Grid World environment that includes multiple objects with different properties and ways of interaction.
Revised: We use a Grid World environment that includes multiple objects with different properties and ways of interaction.

Original: These objects are: a ball that can be picked and dropped, a chest where the agent can store the ball and a special target location where the agent can step into.
Revised: These objects are: a ball that can be picked and dropped, a chest where the agent can store the ball and a special target location where the agent can step into.

Original: Additionally, we introduce a demon that has its own dynamics to help us evaluate if CEHRL can disentangle controlled effects.
Revised: Additionally, we introduce a demon that has its own dynamics to help us evaluate if CEHRL can disentangle controlled effects.

Original: We provide three tasks where reaching the special target location becomes increasingly difﬁcult:
Revised: We provide three tasks where reaching the special target location becomes increasingly difﬁcult:

Original: • T: go to the target location.
Revised: • T: go to the target location.

Original: • BT: go to the target location while carrying a ball • CBT: pick ball, put it in the chest, and go to the target.
Revised: • BT: go to the target location while carrying a ball • CBT: pick ball, put it in the chest, and go to the target.

Original:  To implement this environment we use MiniGrid 2D (Chevalier-Boisvert et al., 2018).
Revised: [Figure] To implement this environment we use MiniGrid 2D (Chevalier-Boisvert et al., 2018).

Original: Although these kind of environments may seem simple, they are known to be hard to explore due to the combinatorial explosion of possible effects.
Revised: Although these kind of environments may seem simple, they are known to be hard to explore due to the combinatorial explosion of possible effects.

Original: In other words, it is not enough to move forward to explore new regions, the agent needs to learn to combine effects.
Revised: In other words, it is not enough to move forward to explore new regions, the agent needs to learn to combine effects.

Original: Moreover, to separate the problems of representation learning and exploration, agents are provided with an entity-centric representation of the world as in Baker et al.
Revised: Moreover, to separate the problems of representation learning and exploration, agents are provided with an entity-centric representation of the world as in Baker et al.

Original: Campero et al.
Revised: Campero et al.

Original: A reward of one is given discounted by the time it took to solve the task, see supplementary material for more details.
Revised: A reward of one is given discounted by the time it took to solve the task, see supplementary material for more details.

Original: Although we list the set of possible effects in our results, they are not given to the agent; the agent has to learn them.
Revised: Although we list the set of possible effects in our results, they are not given to the agent; the agent has to learn them.

Revised: We evaluate how CEHRL scales with the complexity of the task by comparing it to a DQN baseline.

Original: Note that the duration of a training step is different for each of the components in CEHRL and the baseline.
Revised: Note that the duration of a training step is different for each of the components in CEHRL and the baseline.

Original: To be fair to the baseline, we report training time.
Revised: To be fair to the baseline, we report training time.

Original: 4.1. Can controlled effects ease the task of credit assignment?
Revised: 4.1. Can controlled effects ease the task of credit assignment?

Original: This experiment evaluates if CEHRL can ease the credit assignment problem by using a few effects instead of numerous actions.
Revised: This experiment evaluates if CEHRL can ease the credit assignment problem by using a few effects instead of numerous actions.

Original: We train the task policy on the three variants of the environment using the pre-trained models ˆ e t , E and π e from experiment 4.3.
Revised: We train the task policy on the three variants of the environment using the pre-trained models ˆ e t , E and π e from experiment 4.3.

Original: We evaluate how CEHRL scales with the complexity of the task by comparing it to a DQN baseline.

Original: Figure 4( a ) shows the baseline achieving high reward in the ﬁrst two tasks but cannot complete the more complex task.
Revised: Figure 5( a ) shows the baseline achieving high reward in the ﬁrst two tasks but cannot complete the more complex task.

Original: Moreover, it requires increasingly more time to complete each task, not scaling well with the task’s complexity.
Revised: Moreover, it requires increasingly more time to complete each task, not scaling well with the task’s complexity.

Original: On the other hand, CEHRL achieves high reward in the three tasks and needs 30 minutes to complete each task, scaling better with the task’s complexity (Fig.
Revised: On the other hand, CEHRL achieves high reward in the three tasks and needs 30 minutes to complete each task, scaling better with the task’s complexity (Fig.

Original: 4( b )).
Revised: 5( b )).

Original: We conjecture that learning the value of individual actions at every state requires a large number of samples, where using effects as abstractions reduces the space of values to model.
Revised: We conjecture that learning the value of individual actions at every state requires a large number of samples, where using effects as abstractions reduces the space of values to model.

Original: Note that since we do not ﬁne-tune π e further, CEHRL does not achieve an optimal reward.
Revised: Note that since we do not ﬁne-tune π e further, CEHRL does not achieve an optimal reward.

Original: [Figure] These results suggest that using a hierarchy of effects simpliﬁes the credit assignment problem and enables solving long-horizon tasks more efﬁciently.
Revised: These results suggest that using a hierarchy of effects simpliﬁes the credit assignment problem and enables solving long-horizon tasks more efﬁciently.

Original: This is possible due to reusing prior knowledge acquired during exploration.
Revised: This is possible due to reusing prior knowledge acquired during exploration.

Original: Experiment 4.3 analyses the cost of learning these effects.
Revised: Experiment 4.3 analyses the cost of learning these effects.

Revised: [Figure]

Original: 4.2. How does random effect exploration differ from using random actions?
Revised: 4.2. How does random effect exploration differ from using random actions?

Original: To analyze random effect exploration, we record the effects performed by each method during 500K steps.
Revised: To analyze random effect exploration, we record the effects performed by each method during 500K steps.

Original: We do not perform any training and use pre-trained models as in the previous experiment.
Revised: We do not perform any training and use pre-trained models as in the previous experiment.

Original: Note that some effects are more difﬁcult to perform than others, e.g. putting the ball in the chest vs moving forward.
Revised: Note that some effects are more difﬁcult to perform than others, e.g. putting the ball in the chest vs moving forward.

Original: We use the following categorization: basic effects require usually one or two actions to be performed (these are usually effects on the agent), simple effects require more than one basic effect and complex effects require multiple simple effects.
Revised: We use the following categorization: basic effects require usually one or two actions to be performed (these are usually effects on the agent), simple effects require more than one basic effect and complex effects require multiple simple effects.

Original: This choice is arbitrary but helps us better understand what each method is exploring.
Revised: This choice is arbitrary but helps us better understand what each method is exploring.

Original: An unbiased exploration method would perform effects as uniformly as possible.
Revised: An unbiased exploration method would perform effects as uniformly as possible.

Original: Note that a perfectly uniform exploration is not possible due to the compositional nature of effects i.e. to perform a complex effect the agent needs to perform multiple simple or basic effects.
Revised: Note that a perfectly uniform exploration is not possible due to the compositional nature of effects i.e. to perform a complex effect the agent needs to perform multiple simple or basic effects.

Original: The results in Figure 5( a ) show that random effect exploration performs simple and complex effects almost three times more often than with random actions.
Revised: The results in Figure 6( a ) show that random effect exploration performs simple and complex effects almost three times more often than with random actions.

Original: The effect ”nothing” is not needed to perform other effects; therefore, it happens less often.
Revised: The effect ”nothing” is not needed to perform other effects; therefore, it happens less often.

Original: In contrast, other basic effects are central to simpler effects and consequently happening more often.
Revised: In contrast, other basic effects are central to simpler effects and consequently happening more often.

Original: 4.3. How expensive is to operate with effects?
Revised: 4.3. How expensive is to operate with effects?

Original: We have shown that exploration and credit assignment beneﬁt from operating with effects.
Revised: We have shown that exploration and credit assignment beneﬁt from operating with effects.

Original: Here, we study the cost of creating the hierarchy of effects using the method described in section 3.1.
Revised: Here, we study the cost of creating the hierarchy of effects using the method described in section 3.1.

Original: For this, we train models ˆ e t , E and π e and record the reward achieved by each individual effect.
Revised: For this, we train models ˆ e t , E and π e and record the reward achieved by each individual effect.

Original: Figure 5( b ) shows that CEHRL can perform every basic effect (reward higher than 0 . 9 ) after three hours of training and that simple and complex effects take ﬁve and six hours respectively.
Revised: Figure 6( b ) shows that CEHRL can perform every basic effect (reward higher than 0 . 9 ) after three hours of training and that simple and complex effects take ﬁve and six hours respectively.

Original: Comparing these results with experiment 4.1, we can see that CEHRL learns to perform every effect in six hours but takes ﬁve hours for the baseline to learn tasks T and BT , and cannot learn task CBT .
Revised: Comparing these results with experiment 4.1, we can see that CEHRL learns to perform every effect in six hours but takes ﬁve hours for the baseline to learn tasks T and BT , and cannot learn task CBT .

Original: These results indicate that the cost of learning a hierarchy of effects is quickly amortized when the number of tasks[Figure] or their complexity increases.
Revised: These results indicate that the cost of learning a hierarchy of effects is quickly amortized when the number of tasks or their complexity increases.

Original: Moreover, using effects enables the learning of tasks that the baseline struggles to learn.
Revised: Moreover, using effects enables the learning of tasks that the baseline struggles to learn.

Original: We conjecture that this is due to the implicit curriculum of learning causal effects, i.e. the learning of simpler effects makes easier to learn complex effects.
Revised: We conjecture that this is due to the implicit curriculum of learning causal effects, i.e. the learning of simpler effects makes easier to learn complex effects.

Original: 4.4. Is there any beneﬁt to use controlled effects?
Revised: 4.4. Is there any beneﬁt to use controlled effects?

Original: Throughout this work, we have proposed to disentangle controlled effects from total effects, i.e., we deliberately ignore uncontrolled effects since the agent cannot cause these by deﬁnition.
Revised: Throughout this work, we have proposed to disentangle controlled effects from total effects, i.e., we deliberately ignore uncontrolled effects since the agent cannot cause these by deﬁnition.

Original: The intuition behind this choice is that controlled effects reduce the effect space to model.
Revised: The intuition behind this choice is that controlled effects reduce the effect space to model.

Original: Disentangling effects is costly; therefore it must be worth doing.
Revised: Disentangling effects is costly; therefore it must be worth doing.

Original: This experiment compares how controlled effects differ from total effects when learning to pick up a ball.
Revised: This experiment compares how controlled effects differ from total effects when learning to pick up a ball.

Original: The main difference between total and controlled effects is the uncontrolled changes in the environment.
Revised: The main difference between total and controlled effects is the uncontrolled changes in the environment.

Original: Thus, for this experiment we increase the number of demons and the complexity of their dynamics to measure if controlled effects provide a beneﬁt by ignoring the uncontrolled part of the state.
Revised: Thus, for this experiment we increase the number of demons and the complexity of their dynamics to measure if controlled effects provide a beneﬁt by ignoring the uncontrolled part of the state.

Original: We deﬁne three different dynamics: static where the demon does not move; horizontal where demons only move in a horizontal line; and circular where the demon describes a circular movement in the arena.
Revised: We deﬁne three different dynamics: static where the demon does not move; horizontal where demons only move in a horizontal line; and circular where the demon describes a circular movement in the arena.

Original: Additionally, we do not train the total effects model when using total effects.
Revised: Additionally, we do not train the total effects model when using total effects.

Original: We report the average number of actions taken to pick the ball.
Revised: We report the average number of actions taken to pick the ball.

Original: The ﬁrst row in Figure 4.4 shows that in the absence of dynamics (static demon), total effects learn more efﬁciently than controlled effects.
Revised: The ﬁrst row in Figure 4.4 shows that in the absence of dynamics (static demon), total effects learn more efﬁciently than controlled effects.

Original: This is expected since there is an overhead to train a forward model; moreover the model will not be perfect either.
Revised: This is expected since there is an overhead to train a forward model; moreover the model will not be perfect either.

Original: By comparing the use of controlled against total effects column wise, we can see that the performance when using total effects decreases the more demons there are.
Revised: By comparing the use of controlled against total effects column wise, we can see that the performance when using total effects decreases the more demons there are.

Original: On the other hand, the agent using controlled effects has similar performance no matter the number of demons used.
Revised: On the other hand, the agent using controlled effects has similar performance no matter the number of demons used.

Original: Furthermore, the agent using total effects does not learn how to pick the ball in the horizontal and circular variants with neither two and three demons.
Revised: Furthermore, the agent using total effects does not learn how to pick the ball in the horizontal and circular variants with neither two and three demons.

Original: 4.5. How does the task policy use the hierarchy of effects?
Revised: 4.5. How does the task policy use the hierarchy of effects?

Original: This experiment evaluates how the task policy uses the hierarchy of effects i.e. does it use abstract effects or does it use effects closer to actions?
Revised: This experiment evaluates how the task policy uses the hierarchy of effects i.e. does it use abstract effects or does it use effects closer to actions?

Original: For this, we use the variant ”T” of the environment where the agent needs to go to a target location.
Revised: For this, we use the variant ”T” of the environment where the agent needs to go to a target location.

Original: This variant can be solved in multiple ways, for example, the task policy could have learned to use only basic effects to change the x or y coordinates of the agent one step at a time; it could use more complex effects like go-to-target; or a mix of these two.
Revised: This variant can be solved in multiple ways, for example, the task policy could have learned to use only basic effects to change the x or y coordinates of the agent one step at a time; it could use more complex effects like go-to-target; or a mix of these two.

Original: Ideally, the use of higher levels of the hierarchy is preferred since these allow for better credit assignment.
Revised: Ideally, the use of higher levels of the hierarchy is preferred since these allow for better credit assignment.

Original: The results in Figure 7 show how the policy learns to alternate between the two high-level effects that lead to complete the task.
Revised: The results in Figure 8 show how the policy learns to alternate between the two high-level effects that lead to complete the task.

Original: By working at a higher time scale, together with the environment’s reward function, the agent is motivated to use more abstract effects.
Revised: By working at a higher time scale, together with the environment’s reward function, the agent is motivated to use more abstract effects.

Original: 5. Related Work
Revised: 5. Related Work

Original: Intrinsic rewards: a popular solution to the exploration and credit assignment problems is to use intrinsic rewards (Singh et al., 2005; Pathak et al., 2017; Burda et al., 2018; Song et al., 2019; Choi et al., 2019; Badia et al., 2020).
Revised: Intrinsic rewards: a popular solution to the exploration and credit assignment problems is to use intrinsic rewards (Singh et al., 2005; Pathak et al., 2017; Burda et al., 2018; Song et al., 2019; Choi et al., 2019; Badia et al., 2020).

Original: This approach promotes exploratory behavior by rewarding curiosity.
Revised: This approach promotes exploratory behavior by rewarding curiosity.

Original: Burda et al.
Revised: Burda et al.

Original: (2018) use an untrained neural network to estimate surprise and reward for it.
Revised: (2018) use an untrained neural network to estimate surprise and reward for it.

Original: Choi et al.
Revised: Choi et al.

Original: (2019) and Badia et al.
Revised: (2019) and Badia et al.

Original: (2020) reward for the discovery of states controlled by the agent.
Revised: (2020) reward for the discovery of states controlled by the agent.

Original: We consider these methods complementary to our work and can be incorporated to CEHRL for better exploration.
Revised: We consider these methods complementary to our work and can be incorporated to CEHRL for better exploration.

Original: Hierarchical RL: Dayan and Hinton (1993) proposed Feudal RL where a hierarchy of managers work at different granularity by controlling the information and rewards transferred to lower levels of the hierarchy.
Revised: Hierarchical RL: Dayan and Hinton (1993) proposed Feudal RL where a hierarchy of managers work at different granularity by controlling the information and rewards transferred to lower levels of the hierarchy.

Original: Vezhnevets et al.
Revised: Vezhnevets et al.

Original: (2017) gave a deep learning implementation of this framework where a worker performs actions to accomplish the goal set by a manager.
Revised: (2017) gave a deep learning implementation of this framework where a worker performs actions to accomplish the goal set by a manager.

Original: Sukhbaatar et al.
Revised: Sukhbaatar et al.

Original: (2018) on the other hand, divided the training of the manager and worker into exploration and task-dependent stages.
Revised: (2018) on the other hand, divided the training of the manager and worker into exploration and task-dependent stages.

Original: These methods rely on extrinsic rewards to learn representations between levels making them ill-suited for environments with sparse rewards due to their high exploratory demands.
Revised: These methods rely on extrinsic rewards to learn representations between levels making them ill-suited for environments with sparse rewards due to their high exploratory demands.

Original: The options framework (Sutton et al., 1999) adopts a more decentralized approach to temporal abstraction.
Revised: The options framework (Sutton et al., 1999) adopts a more decentralized approach to temporal abstraction.

Original: Each option represents a skill that decides if it should be started or ﬁnished based on the current state.
Revised: Each option represents a skill that decides if it should be started or ﬁnished based on the current state.

Original: A well-known problem is to discover useful skills (Bacon et al., 2016; Florensa et al., 
Revised: A well-known problem is to discover useful skills (Bacon et al., 2016; Florensa et al., 2017;

Original: Eysenbach et al., 2018; Nair et al., 2018; Held et al., 2018;
Revised: Eysenbach et al., 2018; Nair et al., 2018; Held et al., 2018;

Original: Jegorova et al., 2018; Sharma et al., 2019).
Revised: Jegorova et al., 2018; Sharma et al., 2019).

Original: A common practice is to optimize the mutual information between skills to make them easily distinguishable.
Revised: A common practice is to optimize the mutual information between skills to make them easily distinguishable.

Original: Instead, we consider every way an agent can modify the state as a potential skill.
Revised: Instead, we consider every way an agent can modify the state as a potential skill.

Original: Generative RL: generative models like Variational Autoencoders (VAE) or Generative Adversarial
Revised: Generative RL: generative models like Variational Autoencoders (VAE) or Generative Adversarial

Original: Networks (GAN) have proven to be extremely useful in image or audio generation.
Revised: Networks (GAN) have proven to be extremely useful in image or audio generation.

Original: Nonetheless, these methods are gaining popularity in RL (Held et al., 2018; Nair et al., 2018; Nair and Finn, 2019).
Revised: Nonetheless, these methods are gaining popularity in RL (Held et al., 2018; Nair et al., 2018; Nair and Finn, 2019).

Original: Nair and Finn (2019) used a VAE to produce intermediate goal-images that facilitate the planning of actions to achieve a ﬁnal goal.
Revised: Nair and Finn (2019) used a VAE to produce intermediate goal-images that facilitate the planning of actions to achieve a ﬁnal goal.

Original: Using images as goals in environments with dynamics makes difﬁcult to decide whether a goal has been achieved.
Revised: Using images as goals in environments with dynamics makes difﬁcult to decide whether a goal has been achieved.

Original: Instead, CEHRL removes dynamic effects, only using controlled effects.
Revised: Instead, CEHRL removes dynamic effects, only using controlled effects.

Original: Additionally, using effects instead of states reduce the size of the distribution to model, e.g. in a 1D inﬁnite environment a generator that works with states would need to produce inﬁnite states to achieve the effect of moving forward.
Revised: Additionally, using effects instead of states reduce the size of the distribution to model, e.g. in a 1D inﬁnite environment a generator that works with states would need to produce inﬁnite states to achieve the effect of moving forward.

Original: In contrast, CEHRL would generate the same effect over and over.
Revised: In contrast, CEHRL would generate the same effect over and over.

Original: Similarly, Held et al.
Revised: Similarly, Held et al.

Original: (2018) and Campero et al.
Revised: (2018) and Campero et al.

Original: (2021) used GANs to produce a curriculum of goals to ease the learning of a speciﬁc task.
Revised: (2021) used GANs to produce a curriculum of goals to ease the learning of a speciﬁc task.

Original: Causality: Incorporating concepts from causality to deep learning in general (Bengio et al., 2019;
Revised: Causality: Incorporating concepts from causality to deep learning in general (Bengio et al., 2019;

Original: Chattopadhyay et al., 2019; Ke et al., 2019) and reinforcement learning in particular (Buesing et al., 2018; Jaques et al., 2018; Dasgupta et al., 2019; Goyal et al., 2019; Nair et al., 2019) has shown to be an important research avenue that can beneﬁt a wide range of tasks.
Revised: Chattopadhyay et al., 2019; Ke et al., 2019) and reinforcement learning in particular (Buesing et al., 2018; Jaques et al., 2018; Dasgupta et al., 2019; Goyal et al., 2019; Nair et al., 2019) has shown to be an important research avenue that can beneﬁt a wide range of tasks.

Original: Badia et al.
Revised: Badia et al.

Original: (2020) use an inverse model to identify controlled aspects of the environment.
Revised: (2020) use an inverse model to identify controlled aspects of the environment.

Original: Although efﬁcient, this approach is local to the agent; ignoring changes faraway from the agent.
Revised: Although efﬁcient, this approach is local to the agent; ignoring changes faraway from the agent.

Original: By using counterfactual measures we identify a broad set of controlled effects.
Revised: By using counterfactual measures we identify a broad set of controlled effects.

Original: 6. Conclusion and future work
Revised: 6. Conclusion and future work

Original: We presented CEHRL, a hierarchical method that leverages causal tools to continuously learn a hierarchy of controlled effects.
Revised: We presented CEHRL, a hierarchical method that leverages causal tools to continuously learn a hierarchy of controlled effects.

Original: We showed that this hierarchy can be used to efﬁciently learn taskspeciﬁc behavior in an scalable and efﬁcient manner.
Revised: We showed that this hierarchy can be used to efﬁciently learn taskspeciﬁc behavior in an scalable and efﬁcient manner.

Original: Our experiments also show that the cost of building this hierarchy is quickly amortized when the number of tasks or their complexity increases.
Revised: Our experiments also show that the cost of building this hierarchy is quickly amortized when the number of tasks or their complexity increases.

Original: To avoid confounding our experiments, CEHRL has been applied to entity-centric representations.
Revised: To avoid confounding our experiments, CEHRL has been applied to entity-centric representations.

Original: In future work we want to incorporate methods for unsupervised state representation learning (Burgess et al., 2019;
Revised: In future work, we want to incorporate methods for unsupervised state representation learning (Burgess et al., 2019;

Original: Anand et al., 2019)
Revised: Anand et al., 2019)

Original: so CEHRL can learn from observations.
Revised: so CEHRL can learn from observations.

Original: Additionally, to avoid biasing exploration, effects are sampled uniformly.
Revised: Additionally, to avoid biasing exploration, effects are sampled uniformly.

Original: Instead, we could choose goals from the hierarchy that optimize the learning of novel effects.
Revised: Instead, we could choose goals from the hierarchy that optimize the learning of novel effects.

Original: Another interesting avenue is to use the hierarchy of effects as a temporally abstract model of the world by conditioning it to the state, so only effects achievable in certain number of steps are selected.
Revised: Another interesting avenue is to use the hierarchy of effects as a temporally abstract model of the world by conditioning it to the state, so only effects achievable in certain number of steps are selected.

Original: The possibility of creating social norms ought to be explored using counterfactuals and measures normality in a multi-agent setting. 
Revised: The possibility of creating social norms ought to be explored using counterfactuals and measures normality in a multi-agent setting. Limitations

Revised: Deterministic transitions: this work, more speciﬁcally Eq.

Revised: 5, considers a deterministic setting.

Revised: The mode can be approximated for stochastic domains or domain with large number of actions using the mode-collapse problem/feature in GANs.

Revised: Representation learning: CEHRL works best when the state representation is disentangled and/or compositional.

Original: Appendix A. Implementation
Revised: In many cases access to this kind (or any) of state is impossible. Representation learning is an important area of research that we hope to incorporate into CEHRL in the future. Normality : although we propose a measure of normality in Eq. 5, this is far from ideal.

Revised: We believe the way humans see normality is context dependent and should be learned instead of a ﬁxed function.

Revised: This is an active research area by psychologist looking to underpin human causal judgment.

Revised: Uncontrolled effects : there may be uncontrolled events required so the agent can fulﬁll more complex controlled effects, e g. moving a heavy box with two agents.

Revised: The high-level policy and prior model do not take this kind of effects into account.

Revised: Appendix A. Implementation

Original: Here we describe more details related to the training and random effect exploration of CEHRL and how they are used for task-agnostic and task-speciﬁc learning.
Revised: Here we describe more details related to the training and random effect exploration of CEHRL and how they are used for task-agnostic and task-speciﬁc learning.

Original: A.1. Implementation details of task-agnostic learning
Revised: A.1. Implementation details of task-agnostic learning

Original: In this section, we provide additional details to CEHRL’s training for task-agnostic learning.
Revised: In this section, we provide additional details to CEHRL’s training for task-agnostic learning.

Original: To decide when a goal is reached, we deﬁne a function h ( e c , e goal c ) that compares if two controlled effects match at each dimension
Revised: To decide when a goal is reached, we deﬁne a function h ( e c , e goal c ) that compares if two controlled effects match at each dimension

Original: [Equation]
Revised: [Equation]

Original: Dealing with effect imbalance: The replay memory is dominated by basic effects e.g. moving forward happens more often than turning the light on.
Revised: Dealing with effect imbalance: The replay memory is dominated by basic effects e.g. moving forward happens more often than turning the light on.

Original: This makes rare effects difﬁcult to learn.
Revised: This makes rare effects difﬁcult to learn.

Original: To alleviate this imbalance, each experience is added to the buffer with an initial priority proportional to how rare an effect is
Revised: To alleviate this imbalance, each experience is added to the buffer with an initial priority proportional to how rare an effect is

Original: [Equation]
Revised: [Equation]

Original: where p ( e c |D ) is approximated by a running average of the seen effects, and the ﬁnal value is capped.
Revised: where p ( e c |D ) is approximated by a running average of the seen effects, and the ﬁnal value is capped.

Original: Shared data, individual priorities: Every component shares the same experiences stored in the replay memory D
Revised: Shared data, individual priorities: Every component shares the same experiences stored in the replay memory D

Original: but we use an dedicated priority queue for each component.
Revised: but we use an dedicated priority queue for each component.

Original: We use the same initial priority but each component updates its priorities based on their individual errors.
Revised: We use the same initial priority but each component updates its priorities based on their individual errors.

Original: Fixed (cid:15) -greedy exploration: The effect-conditioned policy produces a sequence of actions to perform an effect on the environment.
Revised: Fixed (cid:15) -greedy exploration: The effect-conditioned policy produces a sequence of actions to perform an effect on the environment.

Original: This policy relies on (cid:15) -greedy exploration to ﬁnd the right action sequence.
Revised: This policy relies on (cid:15) -greedy exploration to ﬁnd the right action sequence.

Original: Typically, the exploration rate (cid:15) decays over time.
Revised: Typically, the exploration rate (cid:15) decays over time.

Original: Instead, we adopt an approach similar to Ape-X
Revised: Instead, we adopt an approach similar to Ape-X

Original: (Horgan et al., 2018) where this value is ﬁxed throughout the training.
Revised: (Horgan et al., 2018) where this value is ﬁxed throughout the training.

Original: Since CEHRL does not work in a distributed manner, we deﬁne a set of candidate epsilons C
Revised: Since CEHRL does not work in a distributed manner, we deﬁne a set of candidate epsilons C

Original: = { (cid:15) 1 , . . .
Revised: = { (cid:15) 1 , . . .

Original: , (cid:15) L } and ﬁx one randomly for each episode.
Revised: , (cid:15) L } and ﬁx one randomly for each episode.

Original: Additionally, we use a warmup period where C = { 1 } .
Revised: Additionally, we use a warmup period where C = { 1 } .

Original: This period is used during the initial training of the total effects model and the distribution of effects, see
Revised: This period is used during the initial training of the total effects model and the distribution of effects, see

Original: Table 1 for more details.
Revised: Table 1 for more details.

Original: We use a function f C : t, M → C that decides which epsilons to use based on the current training step and the set of models to be trained.
Revised: We use a function f C : t, M → C that decides which epsilons to use based on the current training step and the set of models to be trained.

Original: Hindsight learning: To speed up the learning of effects, we augment the collected data in a similar way to Hindsight Experience Replay (HER) by Andrychowicz et al.
Revised: Hindsight learning: To speed up the learning of effects, we augment the collected data in a similar way to Hindsight Experience Replay (HER) by Andrychowicz et al.

Original: Every experience ( s, e goal c , a, s (cid:48) , e c ( s, a ) , r (cid:48) , d (cid:48) ) is augmented with an additional experience where the goal effect has been replaced with the reached controlled effect i.e. ( s, e c ( s, a ) , a, s (cid:48) , e c ( s, a ) , r (cid:48)
Revised: Every experience ( s, e goal c , a, s (cid:48) , e c ( s, a ) , r (cid:48) , d (cid:48) ) is augmented with an additional experience where the goal effect has been replaced with the reached controlled effect i.e. ( s, e c ( s, a ) , a, s (cid:48) , e c ( s, a ) , r (cid:48)

Original: = 1 , d (cid:48)
Revised: = 1 , d (cid:48)

Original: = 1) .
Revised: = 1) .

Original: A.2. Implementation details of task-speciﬁc learning
Revised: A.2. Implementation details of task-speciﬁc learning

Original: The task policy is trained by performing random effect exploration.
Revised: The task policy is trained by performing random effect exploration.

Original: This method sets controlled effects as goals where an effect-conditioned policy tries to perform them on the environment.
Revised: This method sets controlled effects as goals where an effect-conditioned policy tries to perform them on the environment.

Original: Unfortunately, this last policy will not be perfect and may fail to perform some goals, thus we do not add them to the replay buffer for training.
Revised: Unfortunately, this last policy will not be perfect and may fail to perform some goals, thus we do not add them to the replay buffer for training.

Original: In contrast to ﬁxed (cid:15) -greedy, we use the usual (cid:15) -greedy decay to explore different goal sequences before relying only on the learned value function.
Revised: In contrast to ﬁxed (cid:15) -greedy, we use the usual (cid:15) -greedy decay to explore different goal sequences before relying only on the learned value function.

Original: Note that even with (cid:15) = 

Revised: Note that even with (cid:15) =

Original: CEHRL still explores the environment, this is possible thanks to random effect exploration.
Revised: CEHRL still explores the environment, this is possible thanks to random effect exploration.

Original: Appendix B. Experimental setup
Revised: Appendix B. Experimental setup

Original: Here we specify the architectures used for CEHRL and the baseline.
Revised: Here we specify the architectures used for CEHRL and the baseline.

Original: We also provide the hyperparameters used in our experiments and the search range of hyperparameters done.
Revised: We also provide the hyperparameters used in our experiments and the search range of hyperparameters done.

Original: Furthermore, we provide more speciﬁcs on the environment used.
Revised: Furthermore, we provide more speciﬁcs on the environment used.

Original: B.1. Environment
Revised: B.1. Environment

Original: The environment is implemented using MiniGrid 2D by Chevalier-Boisvert et al.
Revised: The environment is implemented using MiniGrid 2D by Chevalier-Boisvert et al.

Original: The agent can move forward, turn left or right, pick up different objects, and put objects into boxes.
Revised: The agent can move forward, turn left or right, pick up different objects, and put objects into boxes.

Original: The variants
Revised: The variants

Original: T, BT, and CBT of the environment have all the same number of objects but the task rewarded for is different.
Revised: T, BT, and CBT of the environment have all the same number of objects but the task rewarded for is different.

Original: Agents are provided with a discrete, entity-centric representation of the world as in Baker et al.
Revised: Agents are provided with a discrete, entity-centric representation of the world as in Baker et al.

Original: This representation is a J × I matrix with the J objects in the arena (including the agent) and the following I attributes per object: type of object, x and y coordinates, color for objects or direction for the agent, and the carrying object.
Revised: This representation is a J × I matrix with the J objects in the arena (including the agent) and the following I attributes per object: type of object, x and y coordinates, color for objects or direction for the agent, and the carrying object.

Original: If the agent achieves a task, the environment gives a reward using the following function
Revised: If the agent achieves a task, the environment gives a reward using the following function

Original: [Equation]
Revised: [Equation]

Original: solved otherwise (12) where t is the current time step and T is the maximum time steps per episode.
Revised: solved [Equation] (12) where t is the current time step and T is the maximum time steps per episode.

Original: B.2. Architecture
Revised: B.2. Architecture

Original: We implement every network as an MLP with fully connected layers and ReLU activation functions.
Revised: We implement every network as an MLP with fully connected layers and ReLU activation functions.

Original: State encoder:
Revised: State encoder:

Original: We use a network common to other components to encode the state.
Revised: We use a network common to other components to encode the state.

Original: Each attribute type in the state is encoded into the same embedding space.
Revised: Each attribute type in the state is encoded into the same embedding space.

Original: We concatenate the resulting embeddings, and process them with a two fully connected layers to create a reduced latent space.
Revised: We concatenate the resulting embeddings, and process them with a two fully connected layers to create a reduced latent space.

Original: See Figure 8.
Revised: See Figure 9.

Original: Effect encoder:
Revised: Effect encoder:

Original: Similarly, the effect encoder creates a low-dimensional latent space by using the effect as input of two fully connected layers.
Revised: Similarly, the effect encoder creates a low-dimensional latent space by using the effect as input of two fully connected layers.

Original: See Figure 9.
Revised: See Figure 10.

Original: Total effects model: To predict the dynamics of the environment, we provide this component with two consecutive states.
Revised: Total effects model: To predict the dynamics of the environment, we provide this component with two consecutive states.

Original: We implement this component with two state encoders to encode each consecutive state into a latent vector.
Revised: We implement this component with two state encoders to encode each consecutive state into a latent vector.

Original: These two latent vectors are then processed by four fully connected layers to output the predicted total effects for each of the actions.
Revised: These two latent vectors are then processed by four fully connected layers to output the predicted total effects for each of the actions.

Original: See Figure 10.
Revised: See Figure 11.

Original: Distribution of effects: This component is implemented as a VAE with four fully connected layers to encode controlled effects into a latent space and three fully connected layers to decode latent vectors to controlled effects.
Revised: Distribution of effects: This component is implemented as a VAE with four fully connected layers to encode controlled effects into a latent space and three fully connected layers to decode latent vectors to controlled effects.

Original: Note that the last layer of the decoder has a Sigmoid activation function so as to output a binary vector.
Revised: Note that the last layer of the decoder has a Sigmoid activation function so as to output a binary vector.

Original: Consequently, we replace the typical mean-squared-error reconstruction loss of the VAE with a binary cross-entropy loss.
Revised: Consequently, we replace the typical mean-squared-error reconstruction loss of the VAE with a binary cross-entropy loss.

Original: See Figure 11.
Revised: See Figure 12.

Original: Effect-conditioned Q-value function
Revised: Effect-conditioned Q-value function

Original: : This component uses state and effect encoders to create a latent representation of its input.
Revised: : This component uses state and effect encoders to create a latent representation of its input.

Original: This latent representation is then passed to a four fully connected layer network with an additional Siamese layer to compute state-value and advantage functions, as used in dueling networks.
Revised: This latent representation is then passed to a four fully connected layer network with an additional Siamese layer to compute state-value and advantage functions, as used in dueling networks.

Original: See Figure 12.
Revised: See Figure 13.

Original: Task Q-value function: This is the same network as the effect-conditioned Q-value function but without the Siamese network since we do not have a limited set of controlled effects.
Revised: Task Q-value function: This is the same network as the effect-conditioned Q-value function but without the Siamese network since we do not have a limited set of controlled effects.

Original: See Figure 13.
Revised: See Figure 14.

Original: Baseline : We use the same network as the effect-conditioned Q-value network but without the effect encoder, and train it using DQN with prioritized experience replay, double DQN and dueling networks.
Revised: Baseline : We use the same network as the effect-conditioned Q-value network but without the effect encoder, and train it using DQN with prioritized experience replay, double DQN and dueling networks.

Original: This baseline uses the ﬁxed (cid:15) -greedy exploration described above.
Revised: This baseline uses the ﬁxed (cid:15) -greedy exploration described above.

Original: Since the baseline is composed of only a policy, we do not do any warmup phase.
Revised: Since the baseline is composed of only a policy, we do not do any warmup phase.

Original: B.3. Hyperparameters
Revised: B.3. Hyperparameters

Original: Table 1 provides the training schedule used to implement function f M and f C .
Revised: Table 1 provides the training schedule used to implement function f M and f C .

Original: Hyperparameters common to every training and models are shown in Table 2.
Revised: Hyperparameters common to every training and models are shown in Table 2.

Original: Hyperparameters for task-agnostic learning are shown in Table 3.
Revised: Hyperparameters for task-agnostic learning are shown in Table 3.

Original: Hyperparameters for task-speciﬁc learning are shown in Table 4.
Revised: Hyperparameters for task-speciﬁc learning are shown in Table 4.

Original: Hyperparameters for the baseline are shown in Table 5.
Revised: Hyperparameters for the baseline are shown in Table 5.

Original: Note that each table also provides the search range we used to ﬁx each hyperparameter, if applicable.
Revised: Note that each table also provides the search range we used to ﬁx each hyperparameter, if applicable.

Original: [Table]
Revised: Models

Original: Steps
Revised: Steps

Original: Use warmup C { ˆ e t } [90000, 0]
Revised: Use warmup C { ˆ e t } [90000, 0]

Original: True { ˆ e t , E } [30000, 0]
Revised: [Table] { ˆ e t , E } [ [Table]

Original: True { ˆ e t , π e } [30000, 0] 
Revised: [Table] { ˆ e t , π e } [30000, 0] [Table]

Original: False { ˆ e t , E } [10000]
Revised: { ˆ [Table] t , E } [10000]

Original: [Table]
Revised: [Table]

Original: { π e } [60000]
Revised: { π e } [60000]

Original: False
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ,10K,15K,20K)
Revised: 15K (1K,5K,10K,15K,20K)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: units (64-128-256) (512,256,128) E latent (4,8,16,32)
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: (5e-3,1e-3,5e-4,1e-4,5e-5)
Revised: (5e-3,1e-3,5e-4,1e-4,5e-5)

Original: [Table]
Revised: [Table]

Original: (1e-3,5e-4,1e-4)
Revised: (1e-3,5e-4,1e-4)

Original: [Table]
Revised: [Table]

Original: Hyperparameter
Revised: Hyperparameter

Original: Value
Revised: Value

Original: Search range
Revised: Search range

Original: Task-speciﬁc replay capacity 100K
Revised: Task-speciﬁc replay capacity 100K

Original: N/A
Revised: N/A

Original: Q t units (8,16,32,64,128,256)
Revised: Q t units (8,16,32,64,128,256)

Original: Q t learning rate 0.
Revised: Q t learning rate 0.

Original: (1e-3,1e-4,5e-4,5e-5)
Revised: (1e-3,1e-4,5e-4,5e-5)

Original: Q t target update 2K (1K,2K,5K,10K,15K)
Revised: Q t target update 2K (1K,2K,5K,10K,15K)

Original: Q t epsilon start 1.
Revised: Q t epsilon start 1.

Original: N/A
Revised: N/A

Original: Q t epsilon end 0.
Revised: Q t epsilon end 0.

Original: N/A
Revised: N/A

Original: Q t epsilon steps 50K
Revised: Q t epsilon steps 50K

Original: N/A
Revised: N/A

Original: [Table]
Revised: Hyperparameter Value Search range Replay capacity 500K

Original: N/A
Revised: N/A

Original: C { 0.8, 0.6, 0.4, 0.2, 0.1, 0.01 }
Revised: C { 0.8, 0.6, 0.4, 0.2, 0.1, 0.01 }

Original: N/A
Revised: N/A

Original: [Table] units
Revised: State encoder units

Original: N/A
Revised: N/A

Original: [Table] encoder [Table]
Revised: State encoder latent

Original: N/A
Revised: N/A

Original: Effect encoder units
Revised: Effect encoder units

Original: N/A
Revised: N/A

Original: Effect encoder latent
Revised: Effect encoder latent

Original: N/A
Revised: N/A

Original: Q b units (32,64,128,256,512)
Revised: Q b units (32,64,128,256,512)

Original: [Table] rate 0.
Revised: Q b learning rate 0.

Original: (1e-3,1e-4,5e-4,5e-5)
Revised: (1e-3,1e-4,5e-4,5e-5)

Original: [Table] (1K,5K,10K,15K,20K)
Revised: Q b target update (1K,5K,10K,15K,20K)