Original: D ID I DO THAT ? B LAME AS A MEANS TO IDENTIFY CONTROLLED EFFECTS IN REINFORCEMENT LEARNING
Revised: D ID I DO THAT ? B LAME AS A MEANS TO IDENTIFY CONTROLLED EFFECTS IN REINFORCEMENT LEARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Identifying controllable aspects of the environment has proven to be an extraordinary intrinsic motivator to reinforcement learning agents.
Revised: Identifying controllable aspects of the environment has proven to be an extraor dinary intrinsic motivator to reinforcement learning agents

Original: Despite repeatedly achieving State-of-the-Art results, this approach has only been studied as a proxy to a reward-based task and has not yet been evaluated on its own.
Revised: Despite repeatedly achieving State-of- the -Art results, this approach has only been studied as a proxy to a reward-based task and has not yet been evaluated on its own

Original: We show that solutions relying on action-prediction fail to model critical controlled events.
Revised: Current methods are based on action-prediction.

Original: Humans, on the other hand, assign blame to their actions to decide what they controlled.
Revised: Humans, on the other hand, assign blame to their actions to decide what they controlled.

Original: This work proposes Controlled Effect Network (CEN), an unsupervised method based on counterfactual measures of blame to identify effects on the environment controlled by the agent.
Revised: This work proposes Controlled Effect Net work (CEN) an unsupervised method based on counterfactual measures of blame to identify effects on the environment controlled by the agent

Original: CEN is evaluated in a wide range of environments showing that it can accurately identify controlled effects.
Revised: CEN is evaluated in a wide range of environments showing that it can accurately identify controlled effects

Original: Moreover, we demonstrate CEN’s capabilities as intrinsic motivator by integrating it in the state-of-the-art exploration method, achieving substantially better performance than action-prediction models
Revised: Moreover, we demonstrate CEN’s capabilities as intrinsic motivator by integrating it in the state-of-the-art exploration method, achieving substantially better performance than action-prediction models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The recent success of reinforcement learning (RL) methods in difﬁcult environments such as Hide & Seek (Baker et al., 2019), StarCraft II (Vinyals et al., 2019), or Dota2 (OpenAI et al., 2019) has shown the potential of RL to learn complex behavior.
Revised: The recent success of reinforcement learning (RL) methods in difﬁcult environments such as Hide & Seek (Baker et al., 2019), StarCraft II (Vinyals et al., 2019), or Dota2 (OpenAI et al., 2019) has shown the potential of RL to learn complex behavior.

Original: Unfortunately, these methods also show RL’s inefﬁciency to learn (Espeholt et al., 2018; Kapturowski et al., 2019; Gulcehre et al., 2020), requiring a vast amount of interactions with the environment before meaningful learning occurs.
Revised: Unfortunately, these methods also show RL’s inefﬁciency to learn (Espeholt et al., 2018; Kapturowski et al., 2019; Gulcehre et al., 2020), requiring a vast amount of interactions with the environment before meaningful learning occurs.

Original: Consequently, environments with sparse rewards are known to be extremely difﬁcult making imperative a good exploration strategy.
Revised: Consequently, environments with sparse rewards are known to be extremely difﬁcult making imperative a good exploration strategy.

Original: A popular approach to exploration is to introduce behavioral biases in the form of intrinsic motivators (Chentanez et al., 2005; Mohamed & Rezende, 2015).
Revised: A popular approach to exploration is to introduce behavioral biases in the form of intrinsic motivators (Chentanez et al., 2005; Mohamed & Rezende, 2015).

Original: This technique aims to facilitate the learning of task-agnostic behavior by producing dense rewards, driving the agent to discover novel states and by doing so increase the chance of discovering the environment’s reward.
Revised: This technique aims to facilitate the learning of task-agnostic behavior by producing dense rewards, driving the agent to discover novel states and by doing so increase the chance of discovering the environment’s reward.

Original: Numerous motivators have been developed taking inspiration from humans, e.g. curiosity or control (Bellemare et al., 2012b; Pathak et al., 2017; Burda et al., 2018; Choi et al., 2019; Badia et al., 2020b).
Revised: Numerous motivators have been developed taking inspiration from humans, e.g. curiosity or control (Bellemare et al., 2012b; Pathak et al., 2017; Burda et al., 2018; Choi et al., 2019; Badia et al., 2020b).

Original: Recent work (Choi et al., 2019; Song et al., 2019; Badia et al., 2020a;b) has achieved State-of-the-Art on the Atari benchmark (Bellemare et al., 2012a) by rewarding agents for the discovery of novel ways to control their environment.
Revised: Recent work (Choi et al., 2019; Song et al., 2019; Badia et al., 2020a;b) has achieved State-of-the-Art on the Atari benchmark (Bellemare et al., 2012a) by rewarding agents for the discovery of novel ways to control their environment.

Original: A common design principle among these methods is the use of an inverse model to predict the chosen action from two consecutive observations.
Revised: A common design principle among these methods is the use of an inverse model to predict the chosen action from two consecutive observations.

Original: The hope is that the latent representation learnedby these models encloses aspects of the environment controlled by the agent.
Revised: The hope is that the latent representation learned encloses aspects of the environment controlled by the agent.

Original: We hypothesize that these methods are ill-suited to model anything other than the agent, limiting their applicability.

Original: Fig.

Original: 1 (left) shows an scenario where an agent moves a box by moving left; an inverse model would only need to represent the agent to predict the action but not the box.

Original: A more causal approach is to compare counterfactual worlds (Pearl, 2009), i.e., an effect is controllable if the effect would have been different had the agent taken another action.
Revised: A more causal approach is to compare counterfactual worlds (Pearl, 2009), i.e., an effect is controllable if the effect would have been different had the agent taken another action.

Original: A caveat of this approach is that things become trivially controllable.
Revised: A caveat of this approach is that things become trivially controllable.

Original: Following the previous example, a box becomes controllable even when the agent performs the action ”do-nothing” since there is an action, ”move-left”, that would move the box.
Revised: Fig. 1 (left) shows an scenario where an agent moves a box by moving left. Here a box becomes controllable even when the agent performs the action ”do-nothing” since there is an action, ”move-left”, that would move the box.

Original: Contrarily, it is believed that humans identify controlled effects by assigning a degree of blame to their actions.
Revised: Contrarily, it is believed that humans identify controlled effects by assigning a degree of blame to their actions.

Original: In particular, humans compare what happened to a normative world (Halpern, 2016; Morris et al., 2018; Langenhoff et al., 2019; Grinfeld et al., 2020).
Revised: In particular, humans compare what happened to a normative world (Halpern, 2016; Morris et al., 2018; Langenhoff et al., 2019; Grinfeld et al., 2020).

Original: If what happened is normal, humans would not assign blame to their actions, e.g. when performing ”do-nothing”, the box’s effect would not be controlled since normally the box would not move.
Revised: If what happened is normal, humans would not assign blame to their actions, e.g. when performing ”do-nothing”, the box’s effect would not be controlled since normally the box would not move.

Original: However, it would be considered controlled when performing ”move-left” since its normative state is to not move.
Revised: However, it would be considered controlled when performing ”move-left” since its normative state is to not move.

Original: This work proposes Controlled Effects Network (CEN), a completely unsupervised approach to identify controlled aspects of the environment based on the human notion of blame.
Revised: This work proposes Controlled Effects Network (CEN), a completely unsupervised approach to identify controlled aspects of the environment based on the human notion of blame.

Original: In contrast to models based on action prediction, CEN is composed of a two-branch forward model that creates a normal and controlled view of the world.
Revised: In contrast to models based on action prediction, CEN is composed of a two-branch forward model that creates a normal and controlled view of the world.

Original: Our experiments show that CEN can disentangle effects precisely, outperforming state-of-the-art approaches to detect controlled effects.
Revised: Our experiments show that CEN can disentangle effects precisely, outperforming state-of-the-art approaches to detect controlled effects.

Original: Additionally, we evaluate CEN as an intrinsic motivator by replacing the action-prediction model in Never Give Up’s (Badia et al., 2020b) episodic reward with CEN, leading to a substantial gain in performance.
Revised: Additionally, we evaluate CEN as an intrinsic motivator by replacing the action-prediction model in Never Give Up’s (Badia et al., 2020b) episodic reward with CEN, leading to a substantial gain in performance.

Original:  I DENTIFYING CONTROLLED EFFECTS USING BLAME
Revised:  I DENTIFYING CONTROLLED EFFECTS USING BLAME

Original: Our goal is to identify changes in the environment that were controlled by the agent.
Revised: Our goal is to identify changes in the environment that were controlled by the agent.

Original: This section introduces Individual Causal Effect (ICE), a fundamental measure in causal literature, and frames it in the context of RL.
Revised: This section introduces Individual Causal Effect (ICE), a fundamental measure in causal literature, and frames it in the context of RL.

Original: We show how this measure can be used to identify controllable effects but argue that these are not suitable for RL.
Revised: We show how this measure can be used to identify controllable effects but argue that these are not suitable for RL.

Original: In contrast, the human perception of causality is associated with the concept of blame (Gerstenberg & Lagnado, 2014).
Revised: In contrast, the human perception of causality is associated with the concept of blame (Gerstenberg & Lagnado, 2014).

Original: For example, if lightning hits a forest tree and starts a ﬁre, humans would point to the lightning as the cause of the ﬁre, not the oxygen or wood since they are normally present in the forest.
Revised: For example, if lightning hits a forest tree and starts a ﬁre, humans would point to the lightning as the cause of the ﬁre, not the oxygen or wood since they are normally present in the forest.

Original: Consequently, we expand the idea of blame to identify controlled effects by using measures of normality and counterfactuals.
Revised: Consequently, we expand the idea of blame to identify controlled effects by using measures of normality and counterfactuals.

Original: 2. C ONTROLLABLE EFFECTS
Revised: 2. C ONTROLLABLE EFFECTS

Original: What does it mean to cause something?
Revised: What does it mean to cause something?

Original: Pearl et al.
Revised: Pearl et al.

Original: (2016) provide an intuitive deﬁnition of causeeffect relations: “A variable X is a cause of a variable Y if Y, in any way, relies on X for its value”.
Revised: (2016) provide an intuitive deﬁnition of causeeffect relations: “A variable X is a cause of a variable Y if Y, in any way, relies on X for its value”.

Original: This kind of formulation tries to answer questions like “does smoking causes cancer?”.
Revised: This kind of formulation tries to answer questions like “does smoking causes cancer?”.

Original: Actual causality, proposed in Halpern (2016), studies causal relations between individual events of X and Y .
Revised: Actual causality, proposed in Halpern (2016), studies causal relations between individual events of X and Y .

Original: It aims to answer questions like, “did smoking for 30 years caused David’s cancer?”.
Revised: It aims to answer questions like, “did smoking for 30 years caused David’s cancer?”.

Original: In the following, we introduce the concept of causal effect to then deﬁne controllable effect in the context of RL.
Revised: In the following, we introduce the concept of causal effect to then deﬁne controllable effect in the context of RL.

Original: The individual causal effect (ICE) of an event X
Revised: The individual causal effect (ICE) of an event X

Original: = x on a variable Y i can be measured by comparing counterfactual worlds
Revised: = x on a variable Y i can be measured by comparing counterfactual worlds

Original: [Equation]
Revised: [Equation]

Original: where Y xi reads as “what would the value of an individual Y i be if X is forced to be x ”.
Revised: where Y xi reads as “what would the value of an individual Y i be if X is forced to be x ”.

Original: Similarly,
Revised: Similarly,

Original: Y ˜ xi describes the value of Y i when X is forced to not be x .
Revised: Y ˜ xi describes the value of Y i when X is forced to not be x .

Original: Note that the sub-index i refers to an individual, and hence in the following, we use Y xi and Y x interchangeably.
Revised: Note that the sub-index i refers to an individual, and hence in the following, we use Y xi and Y x interchangeably.

Original: The fundamental problem of causal inference states that we can only observe one of these counterfactual worlds and the other needs to be imagined.
Revised: The fundamental problem of causal inference states that we can only observe one of these counterfactual worlds and the other needs to be imagined.

Original: Intuitively, Eq. 1 compares the world where the event x happened to an alternative world where event x had not happened.
Revised: Intuitively, Eq. 1 compares the world where the event x happened to an alternative world where event x had not happened.

Original: Consequently, we say that x has a causal effect on Y if there is an ˜ x ∈ X that satisﬁes Eq. 1.
Revised: Consequently, we say that x has a causal effect on Y if there is an ˜ x ∈ X that satisﬁes Eq. 1.

Original: In the context of RL, X and Y take the form of actions, states and observations.
Revised: In the context of RL, X and Y take the form of actions, states and observations.

Original: Fig.
Revised: Fig.

Original: 1 (right) illustrates the causal relations present in a typical RL setting, where a state s has an effect on both the next state s (cid:48) and the produced observation o which, in turn, has an effect on the agent’s choice of action a ∈ A .
Revised: 1 (right) illustrates the causal relations present in a typical RL setting, where a state s has an effect on both the next state s (cid:48) and the produced observation o which, in turn, has an effect on the agent’s choice of action a ∈ A .

Original: Similarly, an action has an effect on the next state.
Revised: Similarly, an action has an effect on the next state.

Original: Since states are typically not accessible by the agent, we do not use states as variables; nevertheless the same principle can be applied if these are accessible.
Revised: Since states are typically not accessible by the agent, we do not use states as variables; nevertheless the same principle can be applied if these are accessible.

Original: We deﬁne the perceived effect e ap as the difference between consecutive observations when taking action a , i.e. e ap ≡ o (cid:48) − o .
Revised: We deﬁne the perceived effect e ap as the difference between consecutive observations when taking action a , i.e. e ap ≡ o (cid:48) − o .

Original: As in Eq. 1, we say that a perceived effect was controllable by the agent’s action when
Revised: As in Eq. 1, we say that a perceived effect was controllable by the agent’s action when

Original: [Equation]
Revised: [Equation]

Original: Since we want to know what elements of the perceived effect are controllable, the inequality is an element-wise operation.
Revised: Since we want to know what elements of the perceived effect are controllable, the inequality is an element-wise operation.

Original: It is important to notice that Eq. 2 has far-reaching consequences, for example, an agent next to a box would have a causal effect on it even when not moved since there is a counterfactual world where that box would have moved.
Revised: It is important to notice that Eq. 2 has far-reaching consequences, for example, an agent next to a box would have a causal effect on it even when not moved since there is a counterfactual world where that box would have moved.

Original: Using Eq.
Revised: Using Eq.

Original: 2 as reward
Revised: 2 as reward

Original: , the agent would be rewarded for almost every action at every state!
Revised: , the agent would be rewarded for almost every action at every state!

Original: Note that taking ˜ a as a special ”do-nothing” action would not work since even doing nothing does something, e.g., Fig.
Revised: Note that taking ˜ a as a special ”do-nothing” action would not work since even doing nothing does something, e.g., Fig.

Original: 1 (middle) shows a scenario where doing nothing has an effect on the agent’s health.
Revised: 1 (middle) shows a scenario where doing nothing has an effect on the agent’s health.

Original: Taking do-nothing as ˜ a would not attribute the effect to the agent.
Revised: Taking do-nothing as ˜ a would not attribute the effect to the agent.

Original: Instead, we would want a more human-like deﬁnition of what is controlled where an agent controls a box if moved or its life if a bomb could have been easily avoided.
Revised: Instead, we would want a more human-like deﬁnition of what is controlled where an agent controls a box if moved or its life if a bomb could have been easily avoided.

Original: 2. B LAME
Revised: 2. B LAME

Original: It has been shown that human notion of causality is affected by what is normal (Kahneman & Miller, 1986; Cushman et al., 2008; Knobe & Fraser, 2008; Hitchcock & Knobe, 2009).
Revised: It has been shown that human notion of causality is affected by what is normal (Kahneman & Miller, 1986; Cushman et al., 2008; Knobe & Fraser, 2008; Hitchcock & Knobe, 2009).

Original: Here, we resort to concepts of normality from actual causality to ﬁnd if the agent’s action is to blame for what happened.
Revised: Here, we resort to concepts of normality from actual causality to ﬁnd if the agent’s action is to blame for what happened.

Original: Halpern & Hitchcock (2014) propose to compare what actually happened with what normally would happen.
Revised: Halpern & Hitchcock (2014) propose to compare what actually happened with what normally would happen.

Original: Following this idea we use a normative world in replacement to Y ˜ x
Revised: Following this idea we use a normative world in replacement to Y ˜ x

Original: [Equation]
Revised: [Equation]

Original: where β Y is the value Y would normally take.
Revised: where β Y is the value Y would normally take.

Original: Such a value is of course contingent to the notion of normality used, which is to us to deﬁne.
Revised: Such a value is of course contingent to the notion of normality used, which is to us to deﬁne.

Original: Note that since we are interested in the magnitude and direction of the effect, Eq. 3 uses the difference rather than the less speciﬁc inequality used in Eq.
Revised: Note that since we are interested in the magnitude and direction of the effect, Eq. 3 uses the difference rather than the less speciﬁc inequality used in Eq.

Original: Advantage function as Blame: A typical use of this formulation is to compute the causal effect of an action on the return G as
Revised: Advantage function as Blame: A typical use of this formulation is to compute the causal effect of an action on the return G as

Original: [Equation]
Revised: [Equation]

Original: G a ( s ) is the return the agent would get if action a were to be taken at state s and is typically estimated using a state-action value function Q ( s, a ) .
Revised: G a ( s ) is the return the agent would get if action a were to be taken at state s and is typically estimated using a state-action value function Q ( s, a ) .

Original: The choice of normality for β G ( s ) is to estimate the expected return with the state-value function V ( s ) , giving us the advantage function A ( s, a ) .
Revised: The choice of normality for β G ( s ) is to estimate the expected return with the state-value function V ( s ) , giving us the advantage function A ( s, a ) .

Original: As described in Sutton & Barto (2018), Generalized Value Functions aim to integrate general knowledge of the world; leaving return as special case.
Revised: As described in Sutton & Barto (2018), Generalized Value Functions aim to integrate general knowledge of the world; leaving return as special case.

Original: Following the same idea, we can reformulate
Revised: Following the same idea, we can reformulate

Original: Eq.
Revised: Eq.

Original: 3 to compute the controlled effect of an action as
Revised: 3 to compute the controlled effect of an action as

Original: [Equation]
Revised: [Equation]

Revised: Note that in stochastic environments there may be multiple next observations o (cid:48) for each action ˜ a ∈ A .

Original: To simplify notation, the following sections use controlled effect as e ac =
Revised: To simplify notation, the following sections use controlled effect as e ac =

Original: ICE ae p and normal effect e n = β e p .
Revised: ICE ae p and normal effect e n = β e p .

Original: Intuitively, Eq. 5 builds a normal world by observing every alternative e p produced by each action creating an average perceived effect.
Revised: Intuitively, Eq. 5 builds a normal world by observing every alternative e p produced by each action creating an average perceived effect.

Original: Consider the example in Fig.
Revised: Consider the example in Fig.

Original: 1 (middle), moving left or doing nothing would make the agent’s health decrease.
Revised: 1 (middle), moving left or doing nothing would make the agent’s health decrease.

Original: Eq.
Revised: Eq.

Original: 5 would indicate that no changes to the health bar are normal; thus, the loss of health when moving left or staying would be attributed to the agent.
Revised: 5 would indicate that no changes to the health bar are normal; thus, the loss of health when moving left or staying would be attributed to the agent.

Original: On the other hand, moving right would only attribute the change in the agent’s position as controlled.
Revised: On the other hand, moving right would only attribute the change in the agent’s position as controlled.

Original: Note that the explosion would never be credited to the agent.
Revised: Note that the explosion would never be credited to the agent.

Original: Special care needs to be taken when constructing the normal world β e p for continuous action spaces.
Revised: Special care needs to be taken when constructing the normal world β e p for continuous action spaces.

Original: Computing counterfactuals on an inﬁnite number of possibilities cannot be done and some approximation needs to be implemented.
Revised: Computing counterfactuals on an inﬁnite number of possibilities cannot be done and some approximation needs to be implemented.

Original: Although our experiments use discrete actions, the proposed method in the following section is equipped to handle continuous action spaces since it does not compute counterfactuals for each possible action but approximates the normal world directly.
Revised: Although our experiments use discrete actions, the proposed method in the following section is equipped to handle continuous action spaces since it does not compute counterfactuals for each possible action but approximates the normal world directly.

Original: It is also important to notice that the controlled effects Eq.
Revised: It is also important to notice that the controlled effects Eq.

Original: 5 can identify in a partially observable setting ( o (cid:54) =
Revised: 5 can identify in a partially observable setting ( o (cid:54) =

Original: s ), are constrained to those observed by the agent.
Revised: s ), are constrained to those observed by the agent.

Original: Nevertheless, humans cannot perceive every change in state but can identify relevant controlled effects for their survival and joy.
Revised: Nevertheless, humans cannot perceive every change in state but can identify relevant controlled effects for their survival and joy.

Original:  U NSUPERVISED LEARNING OF CONTROLLED EFFECTS
Revised:  U NSUPERVISED LEARNING OF CONTROLLED EFFECTS

Original: In practice, we do not have access to every world and cannot compute Eq.
Revised: In practice, we do not have access to every world and cannot compute Eq.

Original: directly.
Revised: directly.

Original: We propose an unsupervised method that disentangles controlled and normal effects only using perceived effects as a self-supervised training signal.
Revised: We propose an unsupervised method that disentangles controlled and normal effects only using perceived effects as a self-supervised training signal.

Original: Here, we introduce Controlled Effects Network (CEN) , depicted in Fig.
Revised: Here, we introduce Controlled Effects Network (CEN) , depicted in Fig.

Original: CEN is based on a forward model, where observation and action are used to predict the outcome of performing such action on the environment.
Revised: CEN is based on a forward model, where observation and action are used to predict the outcome of performing such action on the environment.

Original: In contrast to conventional forward models, CEN divides its latent space into controlled and normal representations; similarly to Dueling Networks (Wang et al., 2015).
Revised: In contrast to conventional forward models, CEN divides its latent space into controlled and normal representations; similarly to Dueling Networks (Wang et al., 2015).

Original: These two representations approximate the controlled and normal effects in latent space.
Revised: These two representations approximate the controlled and normal effects in latent space.

Original: A decoder converts these latent representations into pixel space allowing to estimate e ac + e n = e ap as in Eq.
Revised: A decoder converts these latent representations into pixel space allowing to estimate e ac + e n = e ap as in Eq.

Original: The controlled branch has privileged access to the action; having only this branch would make CEN a regular forward model, i.e., the controlled branch alone can predict the perceived effect resulting from the action.
Revised: The controlled branch has privileged access to the action; having only this branch would make CEN a regular forward model, i.e., the controlled branch alone can predict the perceived effect resulting from the action.

Original: Then, why do we need the normal branch ?
Revised: Then, why do we need the normal branch ?

Original: The role of the normal branch is to force the controlled branch to predict only what is not predictable from the observation alone and hence, modeling what is controlled by the agent.
Revised: The role of the normal branch is to force the controlled branch to predict only what is not predictable from the observation alone and hence, modeling what is controlled by the agent.

Original: In a way, the normal branch acts as a distillation mechanism where only what can be controlled will be represented by the controlled branch.
Revised: In a way, the normal branch acts as a distillation mechanism where only what can be controlled will be represented by the controlled branch.

Original: To promote the controlled branch to model only controlled effects, we use the following loss
Revised: To promote the controlled branch to model only controlled effects, we use the following loss

Original: [Equation]
Revised: [Equation]

Original: (6) where the ﬁrst part is the reconstruction loss in which the predicted target ˆ e ap =
Revised: (6) where the ﬁrst part is the reconstruction loss in which the predicted target ˆ e ap =

Original: ˆ e ac + ˆ e n is compared to the perceived effects provided by the environment.
Revised: ˆ e ac + ˆ e n is compared to the perceived effects provided by the environment.

Original: The second part of the loss enforces the network to use the normal branch as much as possible to model the world.
Revised: The second part of the loss enforces the network to use the normal branch as much as possible to model the world.

Original: Since this branch cannot predict everything without the action the model will converge to the expected effect due to the MSE loss.
Revised: Since this branch cannot predict everything without the action the model will converge to the expected effect due to the MSE loss.

Original: Additionally, a hyperparameter α regulates how much the normal branch should model the environment.
Revised: Additionally, a hyperparameter α regulates how much the normal branch should model the environment.

Original: In practice, we found that this hyperparameter creates an agreement between branches on uncertain futures which seemed to be critical in environments with stochastic entities.
Revised: In practice, we found that this hyperparameter creates an agreement between branches on uncertain futures which seemed to be critical in environments with stochastic entities.

Original: Let us look again at the example in Fig.
Revised: Let us look again at the example in Fig.

Original: 1 (middle) and assume the agent picks the do-nothing action.
Revised: 1 (middle) and assume the agent picks the do-nothing action.

Original: The normal branch is encouraged to model the bomb since it does not depend on the action.
Revised: The normal branch is encouraged to model the bomb since it does not depend on the action.

Original: Furthermore, it should not predict any change in health since what is normal is for health to not change.
Revised: Furthermore, it should not predict any change in health since what is normal is for health to not change.

Original: Thus, the controlled branch must model the change in health.
Revised: Thus, the controlled branch must model the change in health.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: This section evaluates CEN 1 on three main questions: 1) Can CEN identify controlled effects at pixel-level?
Revised: This section evaluates CEN 1 on three main questions: 1) Can CEN identify controlled effects at pixel-level?

Original: i.e. can it produce an accurate segmentation mask?
Revised: i.e. can it produce an accurate segmentation mask?

Original: Some applications may not require pixel-level precision; we asses CEN on predicting controlled effects at attribute-level from both pixel masks and latent representations.
Revised: Some applications may not require pixel-level precision; we asses CEN on predicting controlled effects at attribute-level from both pixel masks and latent representations.

Original: 3) Can RL agents beneﬁt from using CEN as intrinsic motivator?
Revised: 3) Can RL agents beneﬁt from using CEN as intrinsic motivator?

Original: We use different baselines to compare CEN against.

Original: We use Attentive Dynamics Model (ADM)

Original: (Choi et al., 2019) in 1) and 2).

Original: For 2) and 3) we rely on Never Give Up (NGU) (Badia et al., 2020b), the current state-of-the-art exploration method.

Original: Environments: we use multiple environments (Fig. 3) to answer the above questions, each showcasing a different aspect of what can be controlled.
Revised: Environments: we use multiple environments (Fig. 3) to answer the above questions, each showcasing a different aspect of what can be controlled.

Original: These environments are based on Griddly (Bamford et al., 2020) and Atari ALE (Bellemare et al., 2012a), both using the Gym interface (Brockman et al., 2016).
Revised: These environments are based on Griddly (Bamford et al., 2020) and Atari ALE (Bellemare et al., 2012a), both using the Gym interface (Brockman et al., 2016).

Original: More details of these environments are given in each experiment and appendix.
Revised: More details of these environments are given in each experiment and appendix.

Original: Implementation: CEN isimplemented as an encoder-decoder architecture with 2D convolutional layers and ReLU activation functions; the normal and controlled branches are implemented with linear layers.
Revised: Baselines: We use Attentive Dynamics Model (ADM) (Choi et al., 2019) in experiments 1) and 2). ADM is an action-prediction model based on an spatial attention mechanism. It works by predicting the action performed on individual image patches.

Revised: Then, a spatial attention mechanism selects a sparse set of patches to use when making a ﬁnal prediction of the action.

Revised: The masks produced by the attention mechanism are considered controlled aspects of the environment.

Revised: Although ADM does not produce pixel level masks, only at patch level, to our knowledge ADM is the most competitive method to provide pixel-level information about controlled aspects.

Revised: For 2) and 3) we rely on Never

Revised: Give Up (NGU) (Badia et al., 2020b), the current SOTA for exploration, and its inverse model.

Revised: Implementation: CEN is implemented as an encoder-decoder architecture with 2D convolutional layers and ReLU activation functions; the normal and controlled branches are implemented with linear layers.

Revised: Additionally decoder weights are shared.

Original: Throughout the experiments we use the same neural networks and hyperparameters unless speciﬁed otherwise.
Revised: Throughout the experiments we use the same neural networks and hyperparameters unless speciﬁed otherwise.

Original: Our implementation of ADM uses the same architecture and hyperparameters proposed in Choi et al.
Revised: Our implementation of ADM uses the same architecture and hyperparameters proposed in Choi et al.

Original: See appendix for more details on the architecture and hyperparameters.
Revised: See appendix for more details on the architecture and hyperparameters.

Original: 4. C ONTROLLED EFFECTS AT PIXEL - LEVEL
Revised: 4. C ONTROLLED EFFECTS AT PIXEL - LEVEL

Original: This set of experiments explores CEN’s ability to identify pixels corresponding to controlled entities.
Revised: This set of experiments explores CEN’s ability to identify pixels corresponding to controlled entities.

Original: Although Eq. computes the magnitude and direction of the effects, we create a binary mask by setting a threshold for the predicted controlled effects (see exact details in the appendix).
Revised: Although CEN computes the magnitude and direction of the effects, we create a binary mask by setting a threshold for the predicted controlled effects (see exact details in the appendix).

Original: We report pixel F1 scores between ground truth and predicted binary masks.
Revised: We report pixel F1 scores between ground truth and predicted binary masks.

Original: The network is trained to minimize
Revised: The network is trained to minimize

Original: Eq.
Revised: Eq.

Original: 6 using the ADAM optimizer (Kingma & Ba, 2015) and 300K samples of the form ( o, a, e ap ) collected using a random policy.
Revised: 6 using the ADAM optimizer (Kingma & Ba, 2015) and 300K samples of the form ( o, a, e ap ) collected using a random policy.

Original: The evaluation of the model is done using an environment with different seed to the one used for training for 5K steps.

Original: To generate the ground truth masks for Griddly environments, we use the internal environment state and transform object coordinates onto pixel coordinates.

Original: In experiments using the Atari environment the ground truth is collected by computing

Original: Eq.

Original: 5 using the ALE’s special calls cloneSystemState and restoreSystemState .

Original: ADM produces an attention mask for pixels controlled by the agent therefore it is used as baseline in these experiments.

Original: 4.1. C ONTROLLED VS UNCONTROLLED EFFECTS
Revised: 4.1. C ONTROLLED VS UNCONTROLLED EFFECTS

Original: Here we use the Spiders environment to evaluate CEN’s ability to disentangle controlled from uncontrolled effects.
Revised: Here we use the Spiders environment to evaluate CEN’s ability to disentangle controlled from uncontrolled effects.

Original: This environment has two main entities, the agent and a spider.
Revised: This environment has two main entities, the agent and a spider.

Original: The controlled masks must only focus on the agent and ignore the spider.
Revised: The controlled masks must only focus on the agent and ignore the spider.

Original: Fig.
Revised: Fig.

Original: 4a shows the pixel F1 score for our model and the baseline.
Revised: 4a shows the pixel F1 score for our model and the baseline.

Original: CEN is able to correctly disentangle controlled effects and can produce accurate masks.
Revised: CEN is able to correctly disentangle controlled effects and can produce accurate masks.

Original: Although our implementation of ADM can predict[Figure] [Figure] the agent’s action with 88% accuracy, it is not capable of modeling the agent’s controlled pixels.
Revised: Although our implementation of ADM can predict the agent’s action with 88% accuracy, it is not capable of modeling the agent’s controlled pixels.

Original: We conjecture that this is due to ADM’s sparse softmax mechanism; nonetheless this behavior persisted when increasing its entropy weight which should produce more dense masks.
Revised: We conjecture that this is due to ADM’s sparse softmax mechanism; nonetheless this behavior persisted when increasing its entropy weight which should produce more dense masks.

Original: 4.1. N EARBY CONTROLLED EFFECTS
Revised: 4.1. N EARBY CONTROLLED EFFECTS

Original: Models based on action prediction are expected to work well on aspects related to the agent.
Revised: Models based on action prediction are expected to work well on aspects related to the agent.

Original: For example, if an agent moves a box due to moving right; the box’s movement is also controlled.
Revised: For example, if an agent moves a box due to moving right; the box’s movement is also controlled.

Original: It is unclear why these models would pay attention to the box since just knowing where the agent is, sufﬁces to predict the chosen action.
Revised: It is unclear why these models would pay attention to the box since just knowing where the agent is, sufﬁces to predict the chosen action.

Original: CEN’s controlled branch, on the other hand, is motivated to model the box’s effect since the normal branch would predict that the box stays where it is.
Revised: CEN’s controlled branch, on the other hand, is motivated to model the box’s effect since the normal branch would predict that the box stays where it is.

Original: We call ”nearby” controlled effect to an effect that happens adjacent the agent, like the box’s movement.
Revised: We call “nearby” controlled effect to an effect that happens adjacent the agent, like the box’s movement.

Original: To evaluate CEN on this kind of effects we use the Clusters environment where an agent needs to move colored boxes to their corresponding ﬁxed colored blocks.
Revised: To evaluate CEN on this kind of effects we use the Clusters environment where an agent needs to move colored boxes to their corresponding ﬁxed colored blocks.

Original: Fig.
Revised: Fig.

Original: 4b shows that CEN can precisely model effects on the agent and boxes.
Revised: 4b shows that CEN can precisely model effects on the agent and boxes.

Original: We breakdown individual effects to account for the class  imbalance between the agent’s movement and the boxes.
Revised: We breakdown individual effects to account for the class [Figure] imbalance between the agent’s movement and the boxes.

Original: CEN seems to make more mistakes with boxes than the agent but nonetheless, it can consistently model both.
Revised: CEN seems to make more mistakes with boxes than the agent but nonetheless, it can consistently model both.

Original: 4.1. F ARAWAY CONTROLLED EFFECTS
Revised: 4.1. F ARAWAY CONTROLLED EFFECTS

Original: In contrast to the previous experiment, we want to evaluate if CEN can model distant effects, i.e. effects that are reasonably far away from the agent’s location.
Revised: In contrast to the previous experiment, we want to evaluate if CEN can model distant effects, i.e. effects that are reasonably far away from the agent’s location.

Original: In this case, we use the Lights environment.
Revised: In this case, we use the Lights environment.

Original: Here the environment presents two buttons of different color that, when pressed, turn on their corresponding light.
Revised: Here the environment presents two buttons of different color that, when pressed, turn on their corresponding light.

Original: Lights are relatively far away from their corresponding buttons thus making it difﬁcult to model them.
Revised: Lights are relatively far away from their corresponding buttons thus making it difﬁcult to model them.

Original: As show in Fig.
Revised: As show in Fig.

Original: 5a, CEN is able to model this kind ofcontrolled effects.
Revised: 5a, CEN is able to model this kind of effects.

Original: We believe the decrease in F1 score is due to the complex shape of lights and buttons.

Original: 4.1. C ONTROLLED EFFECTS IN M ONTEZUMA ’ S R EVENGE
Revised: 4.1. C ONTROLLED EFFECTS IN M ONTEZUMA ’ S R EVENGE

Original: This last pixel-level experiment evaluates CEN on Montezuma’s Revenge  environment.
Revised: This last pixel-level experiment evaluates CEN on Montezuma’s Revenge (MZR) environment.

Original: Although agents in Atari environments have limited control over the environment, the relatively complex dynamics of Montezuma’s Revenge makes it a challenging test-bed.
Revised: Although agents in Atari environments have limited control over the environment, the relatively complex dynamics of Montezuma’s Revenge makes it a challenging test-bed.

Original: Results shown in Fig.
Revised: Results shown in Fig.

Original: 8 indicate that CEN can also model controlled effects but that the increase in complexity affects its ability to predict accurate masks.
Revised: 5b indicate that CEN can also model controlled effects.

Original: Similarly to  the Lights environment the shape of some elements is more complex which we conjecture is  the cause of the model decrease in performance.
Revised: Unlike previous environments, CEN’s F1 score behaves differently due to two main reasons: 1) F decreases over time: the F1 score is extremely sensitive for empty ground truth, a single pixel marked as controlled takes the score from 1 to 0. At the beginning of training, CEN produces empty masks correctly but performs poorly at controlled effects, which makes the F1 score artiﬁcially high. Over time it improves at controlled effects but starts marking very few pixels as controlled for effects where nothing is controlled. 2) F1 is lower: the sensitivity of the F1 score also affects CEN’s performance on controlled effects since pixels outside the ground truth are penalized the same no matter how close to the ground truth they are.

Revised: Appendix A.2 includes additional masks showing that CEN fails by pixels close to the controlled object instead of identifying the wrong object as controlled.

Original: 4. C ONTROLLED EFFECTS AT ATTRIBUTE - LEVEL
Revised: 4. C ONTROLLED EFFECTS AT ATTRIBUTE - LEVEL

Original: In some cases requiring pixel-level precision can be excessive.
Revised: In some cases requiring pixel-level precision can be excessive.

Original: The following experiments analyze how different representations can predict effects on attributes from the environment’s state, e.g. changes on the agent’s ( x, y ) location or if a light turned on.
Revised: The following experiments analyze how different representations can predict effects on attributes from the environment’s state, e.g. changes on the agent’s ( x, y ) location or if a light turned on.

Original: To this end, we use a probing technique similar to the one described in Anand et al.
Revised: To this end, we use a probing technique similar to the one described in Anand et al.

Original: This approach trains a classiﬁer per each attribute of interest in the environment’s state using frozen versions of trained networks to produce the classiﬁers inputs.
Revised: This approach trains a classiﬁer per each attribute of interest in the environment’s state using frozen versions of trained networks to produce the classiﬁers inputs.

Original: More speciﬁcally we use two different sources for the probing classiﬁers, pixel-level masks (as in the previous section) or the model’s latent representation.
Revised: More speciﬁcally we use two different sources for the probing classiﬁers, pixel-level masks (as in the previous section) or the model’s latent representation.

Original: For the ﬁrst case, we produce a binary mask, as in the previous experiments, to occlude perceived effects and use these to train each probing classiﬁer.
Revised: For the ﬁrst case, we produce a binary mask, as in the previous experiments, to occlude perceived effects and use these to train each probing classiﬁer.

Original: Classiﬁers have to predict if there was a positive, negative or none effect.
Revised: Classiﬁers have to predict if there was a positive, negative or none effect.

Original: Note that the classiﬁers need to predict any effect, not just controlled.
Revised: Note that the classiﬁers need to predict any effect, not just controlled.

Original: Thus, the classiﬁer should only be able to predict accurately controlled attributes such as the agent’s position but should fail at predicting uncontrollable effects like the location of the spider or skull.
Revised: Thus, the classiﬁer should only be able to predict accurately controlled attributes such as the agent’s position but should fail at predicting uncontrollable effects like the location of the spider or skull.

Original: We use  a random policy to collect a dataset of 35K samples of the form ( m ∗ e p , y ) where m is the mask produced by the model and y is the ground truth class.
Revised: We use [Figure] a random policy to collect a dataset of 35K samples of the form ( m ∗ e p , y ) where m is the mask produced by the model and y is the ground truth class.

Original: We ensure that each class is relatively balanced, allowing a 20% class imbalance. Each dataset is split into a typical 70/20/10, we report F1 score of each attribute on the test set.
Revised: Each dataset is split into a typical 70/20/10, allowing a 20% class imbalance. We report F1 score of each attribute on the test set.

Original: 4.2. L ATENT PROBING FROM PIXELS
Revised: 4.2. L ATENT PROBING FROM PIXELS

Original: The results in Table 1(ﬁrst block) indicate that CEN signiﬁcantly outperforms the baseline when predicting controlled effects for state’s attributes, and thus modeling controlled effects accurately.
Revised: The results in Table 1(ﬁrst block) indicate that CEN signiﬁcantly outperforms the baseline when predicting controlled effects for state’s attributes, and thus modeling controlled effects accurately.

Original: Furthermore, for both Spiders and Montezuma’s Revenge environments the model cannot predict the uncontrolled effects, as expected.
Revised: Furthermore, for both Spiders and Montezuma’s Revenge environments the model cannot predict the uncontrolled effects, as expected.

Original: Even though ADM’s action prediction accuracy was high ( ∼ 88% ) on every environment, it is not able to consistently predict controlled effects atthe attribute-level.
Revised: Even though ADM’s action prediction accuracy was high ( ∼ 88% ) on every environment, it is not able to consistently predict controlled effects at attribute-level.

Original: 4.2. L ATENT PROBING FROM LATENT REPRESENTATIONS
Revised: 4.2. L ATENT PROBING FROM LATENT REPRESENTATIONS

Original: In this case, we train classiﬁers using a latent representation instead of pixels.
Revised: In this case, we train classiﬁers using a latent representation instead of pixels.

Original: We use the latent representation from CEN’s controlled branch ( h c ).
Revised: We use the latent representation from CEN’s controlled branch ( h c ).

Original: It is unclear how to create a latent representation in ADM’s architecture, thus, we use another extremely popular action-prediction model. In this experiment, we evaluate the inverse model as in Badia et al. (2020b).
Revised: It is unclear how to create a latent representation from ADM, so we use an inverse model.

Original: The features of current and next observations are concatenated to create a latent representation of what is controlled.
Revised: The features of current and next observations are concatenated to create a latent representation of what is controlled.

Original: As before, we train a linear probe to predict controlled attributes from the latent space of these models.
Revised: As before, we train a linear probe to predict controlled attributes from the latent space of these models.

Original: The probe predicts changes on the x, y and direction (if applies) for agent, spider, skull and boxes; and on/off for lights and buttons.
Revised: The probe predicts changes on the x, y and direction (if applies) for agent, spider, skull and boxes; and on/off for lights and buttons.

Original: Table 1 (second block) presents the results of this experiment, showing that CEN improves on the baseline’s performance.
Revised: Table 1 (second block) show that CEN improves on the baseline’s performance.

Original: Although the inverse model is closer to CEN’s performance than ADM, it still has difﬁculties predicting the agent and box changes in location.
Revised: Although the inverse model is closer to CEN’s performance than ADM, it still has difﬁculties predicting the agent and box changes in location.

Original: We hypothesize that  when probing from latent space instead of masked effects, correlations between attributes can be more easily exploited.
Revised: The reason of having high score in Skull is that the Skull is still only when the agent dies, making it easy to predict from the agent’s position. Removing this event leads to a score of ∼ 0 .

Original: For example, in the lights environment the representation of pushing a button may be enough to predict the button and light turning on. In the case of pixels, predicting the light turning on from the pixels representing the agent becomes a harder task.
Revised: 35 for both models.

Original: [Table]
Revised: [Table]

Original: 4. CEN AS INTRINSIC MOTIVATOR
Revised: 4. CEN AS INTRINSIC MOTIVATOR

Original: We have shown that CEN can learn controlled effects in an unsupervised manner.
Revised: We have shown that CEN can learn controlled effects in an unsupervised manner.

Original: Here we showcase the use of this ability as an intrinsic motivator of a reinforcement learning agent.
Revised: Here we showcase the use of this ability as an intrinsic motivator of a reinforcement learning agent.

Original: We consider two tasks, an empty environment without any extrinsic reward where the agent can only control itself and
Revised: We consider two tasks, an empty environment without any extrinsic reward where the agent can only control itself and

Original: Clusters.
Revised: Clusters.

Original: The RL agent is implemented using PPO (vanilla).
Revised: The RL agent is implemented using PPO (vanilla).

Original: Additionally, PPO is augmented with the exploration bonus proposed in Never Give Up (NGU + Inverse)
Revised: Additionally, PPO is augmented with the exploration bonus proposed in Never Give Up (NGU + Inverse)

Original: (Badia et al., 2020b).
Revised: (Badia et al., 2020b).

Original: NGU is composed of two modules for computing episodic and life-long rewards.
Revised: NGU is composed of two modules for computing episodic and life-long rewards.

Original: For simplicity the following experiments only use the episodic module which in NGU consists of a count-based method using an episodic memory to approximate the number of times an agent visited each state, and an inverse model to identify controlled states.
Revised: For simplicity the following experiments only use the episodic module which in NGU consists of a count-based method using an episodic memory to approximate the number of times an agent visited each state, and an inverse model to identify controlled states.

Original: We replace the inverse model with CEN (NGU + CEN) and use the latent representation of the controlled branch to compute NGU’s episodic reward.
Revised: We replace the inverse model with CEN (NGU + CEN) and use the latent representation of the controlled branch to compute NGU’s episodic reward.

Revised: In this experiments, CEN is trained along the PPO policy using experiences collected with the same policy.

Original: Empty environment: the goal of an empty environment is to showcase how each NGU variant rewardseach controlled event.
Revised: Empty environment: the goal of this environment is to showcase how each NGU variant rewards controlled events.

Original: In this environment, only the agent’s movement is controlled; thus, rewarding for controlling the agent location should promote a uniform exploration of the environment since once a location is visited, it should not be rewarded as much as visiting a new location.
Revised: In this environment, only the agent’s movement is controlled; thus, rewarding for controlling the agent’s location should promote a uniform exploration of the environment since once a location is visited, it should not be rewarded as much as visiting a new location.

Original: We hypothesize that an inverse model will create similar representations for the same action disregarding the location where it was taken.
Revised: We hypothesize that an inverse model will create similar representations for the same action disregarding the location where it was taken.

Original: This should impair exploration since the reward will be similar regardless of where the action is performed.
Revised: This should impair exploration since the reward will be similar regardless of where the action is performed.

Original: Fig.
Revised: Fig.

Original: 7a shows how an inverse model does not. Furthermore, since it is hard to predict the action near a wall or corner, these states have higher value, and consequently, the agent tends to hog them.
Revised: 7a shows that agent trained with the inverse model hogs walls and corners. This is because is hard to predict the action near unmovable objects, leading to higher value.

Original: Contrarily, CEN promotes different locations equally, and the agent learns to explore them more uniformly.
Revised: Contrarily, CEN promotes different locations more uniformly and the agent learns to explore better the environment.

Original: Clusters environment: A more challenging environment is Clusters, where the agent needs to move colored boxes to their respective colored blocks.
Revised: Clusters environment: A more challenging environment is Clusters, where the agent needs to move colored boxes to their respective colored blocks.

Original: This environment provides a reward at the end of the episode corresponding to the total number of boxes correctly placed.
Revised: This environment provides a reward at the end of the episode corresponding to the total number of boxes correctly placed.

Original: Results  provided in
Revised: Results are provided in

Original: Fig.
Revised: Fig.

Original: 7b.
Revised: 7b.

Original: Due to the sparsity of the reward PPO does not learn a correct behavior in the given time.
Revised: Due to the sparsity of the reward PPO does not learn a correct behavior in the given time.

Original: Similarly, NGU + Inverse learns to place one box but fails to learn a general behavior to solve the task.
Revised: Similarly, NGU + Inverse learns to place one box but fails to learn a general behavior to solve the task.

Original: Conversely, NGU + CEN quickly learns to move boxes leading to a high extrinsic reward.
Revised: Conversely, NGU + CEN quickly learns to move boxes leading to a high extrinsic reward.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Intrinsic motivators:
Revised: Intrinsic motivators:

Original: A popular way of introducing behavioral biases in RL agents is the use of intrinsic motivators (Singh et al., 2005; Mohamed & Rezende, 2015).
Revised: A popular way of introducing behavioral biases in RL agents is the use of intrinsic motivators (Singh et al., 2005; Mohamed & Rezende, 2015).

Original: These motivators can promote different types of exploration, from observational surprise (Burda et al., 2018) to control seeking agents (Pathak et al., 2017; Choi et al., 2019).
Revised: These motivators can promote different types of exploration, from observational surprise (Burda et al., 2018) to control seeking agents (Pathak et al., 2017; Choi et al., 2019).

Original: Methods in the latter category have shown extremely good results achieving State-of-the-Art in important benchmarks.
Revised: Methods in the latter category have shown extremely good results achieving SOTA in important benchmarks.

Original: Choi et al.
Revised: Choi et al.

Original: (2019) proposed
Revised: (2019) proposed

Original: Attentive Dynamics Model (ADM), an attention based method to discover controlled elements in

Original: [Figure]
Revised: [Figure]

Original:  discovering  method  showed State of the in Montezuma’s Revenge extended in Song et al (2019) to incorporate multi-step controlled effects as part of the intrinsic reward, again achieving human-level performance just with intrinsic rewards.
Revised: Attentive Dynamics Model (ADM), an attention based method that discovers controlled elements in the environment and rewards the agent for discovering them. This method and its extension (Song et al., 2019) showed SOTA in Montezuma’s Revenge.

Original: Badia et al.
Revised: Badia et al.

Original: (2020b)
Revised: (2020b)

Original: combined control and observational surprisemotivators to create a so called ”reward soup” where multiple rewards were combined into one
Revised: combined control and observational surprise to promote exploration.

Original: Their method uses an episodic memory in combination to an inverse model to promote the discovery of controlled effectsin a single episode and Random Network Distillation (Burda et al., 2018) to promote long term progress through the game This method again achieves State the Art in Atari’s hard environments
Revised: Their method uses an episodic memory with an inverse model to promote the discovery of controlled effects and Random Network Distillation (Burda et al., 2018) to promote long term progress; again achieving SOTA in Atari’s hard exploration environments.

Original: One thing in common among these methods is the use of an inverse model to model controlled aspects of the environment.

Original: Our work instead uses a two-branch forward model showing a better coverage of these effects.

Original: These methods show the importance of identifying what an agent can control, an avenue that deserves to be explored in depth.
Revised: These methods show the importance of identifying what an agent can control.

Original: Causality in deep reinforcement learning: Causality is central to humans; we think in terms of cause-effect.
Revised: Causality in deep reinforcement learning: Causality is central to humans; we think in terms of cause-effect.

Original: A similar method  was proposed in Chattopadhyay et al.
Revised: A similar method to Blame was proposed in Chattopadhyay et al.

Original: (2019), where they use causal attribution methods to analyzed the effect of inputs on a neural network’s outputs via causal attribution.
Revised: (2019), where they use causal attribution methods to analyzed the effect of inputs on a neural network’s outputs.

Original: Recent work has introduced causality into deep reinforcement learning (Foerster et al., 2018; Buesing et al., 2018;
Revised: Recent work has introduced causality into deep reinforcement learning (Foerster et al., 2018; Buesing et al., 2018;

Original: Jaques et al., 2018; Dasgupta et al., 2019; Goyal et al., 2019; Nair et al., 2019; Madumal et al., 2020) showing that this is a promising avenue for the training of agents.
Revised: Jaques et al., 2018; Dasgupta et al., 2019; Goyal et al., 2019; Nair et al., 2019; Madumal et al., 2020) showing that this is a promising avenue for the training of agents.

Original: Corcoll & Vicente (2020) proposed an attribution method to learn temporal abstractions for object-centric hierarchical RL.
Revised: Corcoll & Vicente (2020) proposed an attribution method to learn temporal abstractions for object-centric hierarchical RL.

Original: Bellemare et al.
Revised: Bellemare et al.

Original: (2012b) compute controllable aspects of the environment by generating a mask with all possible controllable areas of an image and uses it as part of the policy’s input.
Revised: (2012b) compute controllable aspects of the environment by generating a mask with all possible controllable areas of an image and uses it as part of the policy’s input.

Original: In this work, we identify the controlled effects of individual actions using causal concepts of normality and blame.
Revised: In this work, we identify the controlled effects of individual actions using causal concepts of normality and blame.

Original:  C ONCLUSIONS 
Revised:  C ONCLUSIONS AND LIMITATIONS

Original: The identiﬁcation of controlled effects is a central task in RL.

Original: This work proposes a fully unsupervised approach to this problem named Controlled Effect Network (CEN).
Revised: This work proposes a fully unsupervised approach to this problem named Controlled Effect Network (CEN).

Original: CEN creates a normative world using counterfactuals and compares what actually happened with this normative world to attribute changes on the environment to the agent.
Revised: CEN creates a normative world using counterfactuals and compares what actually happened with what normally would happen to attribute changes on the environment to the agent.

Original: The presented experiments show that, despite being unsupervised, this method precisely identiﬁes controlled effects.
Revised: The presented experiments show that, despite being unsupervised, this method precisely identiﬁes controlled effects.

Original: Furthermore, we show that popular methods based on action prediction do not model controlled effects besides the agent.

Original:  CEN is showcased as intrinsic motivator for RL agents where results suggest that a more targeted exploration leads to substantially better policies.
Revised: Furthermore, CEN is showcased as intrinsic motivator for RL agents where results suggest that a more targeted exploration leads to substantially better policies.

Original: Understanding the world in terms of what can be controlled is  a promising avenue.
Revised: Normality: although we propose a measure of normality in Eq. 5, this is far from ideal. We believe the way humans see normality is context dependent and should be learned instead of a ﬁxed function.

Original: In this direction, more sophisticated modeling of normality, e.g., social norms or time-based norms, can provide a measure of relevance or importance, useful to prioritize the learning of effects.
Revised: This is an active research area by psychologist looking to underpin human causal judgment. State instead of observations: our current formulation of ICE uses perceived effects as opposed to states. When CEN indicates that some change is controlled this is not (necessarily) equivalent to stating that the objects represented by those pixels are controlled. An exciting avenue to explore is combining CEN with a state representation model (e.g. LSTM or RSSM by Hafner et al.

Original: Additionally, we assumed that agents only control effects happening immediately after performing an action.
Revised: where changes would happen at state level. Reliance on policy: since Eq. 5 does not depend on the policy, CEN needs diverse data for each action to approximate it, ideally from a random policy.

Original: A valuable extension to this work is  to identify the consequences of an action multiple steps ahead.
Revised: This important problem is not speciﬁc to CEN, any forward model needs a policy that explores multiple actions to provide accurate predictions.

Original:  R EPRODUCIBILITY
Revised:  R EPRODUCIBILITY

Original: To ensure reproducibility we have included the CEN architecture in Fig.2 while discussing the details and intuition behind the design choices in section 3.
Revised: To ensure reproducibility we have included the CEN architecture in Fig.2 while discussing the details and intuition behind the design choices in section 3.

Original: We also included all the hyperparameters used in the appendix B.3.
Revised: We also included all the hyperparameters used in the appendix C.3.

Original: For the RL experiments, we based our implementation on PFRL (Fujita et al., 2021), we included changes in the default hyperparameters in appendix B.3.
Revised: For the RL experiments, we based our implementation on PFRL (Fujita et al., 2021), we included changes in the default hyperparameters in appendix C.3.

Original: Finally we used the
Revised: Finally we used the

Original: NGU episodic memory described in Badia et al. (2020b).
Revised: NGU episodic memory described in Badia et al. (2020b).

Original: We used the default conﬁguration of Griddly environments except for Clusters environment in our RL experiments, where intermediate rewards were removed to make the environment sparse, details are discussed in appendix A.1.
Revised: We used the default conﬁguration of Griddly environments except for Clusters environment in our RL experiments, where intermediate rewards were removed to make the environment sparse, details are discussed in appendix B.1.

Original: Currently, our implementation depends on internal dependencies, making it impractical to share.
Revised: Currently, our implementation depends on internal dependencies, making it impractical to share.

Original: We will decouple the implementation from internal dependencies and make it publicly accessible upon acceptance
Revised: We will decouple the implementation from internal dependencies and make it publicly accessible upon acceptance

Original: Appendices A E NVIRONMENTS A. G RIDDLY
Revised: Appendices A A DDITIONAL EXPERIMENTS A. E FFECT OF A LPHA ON CEN

Revised: Here we study how α affects the loss in Eq. 6. This experiment uses Montezuma’s Revenge and the same setup as in Experiment 4.1. We analyze the effect of alpha on CEN using values ranging from 0.01 to 20.

Revised: As can be seen in Fig.

Revised: 8, the different alphas do not impact the performance when the ground truth or controlled effects are not empty.

Revised: On the other hand, the performance degrades the higher the alpha.

Revised: We hypothesize that this behavior happens when the normal branch is forced to model controlled effects too strongly and the controlled branch needs to counter those bad predictions.

Revised: In this case, the controlled branch will produce wrong masks, especially for empty ground truth.

Revised: [Figure]

Revised: ASKS

Revised: Here we provide masks for both CEN and ADM.

Revised: Mask are based on effects and extend over two frames, for visualization purposes here we only show the next observation with the full mask.

Revised: It can be seen that CEN fails for a few pixels in MZR, which may explain the low F1 score for empty GT.

Revised: B E NVIRONMENTS B. G RIDDLY

Original: Griddly (Bamford et al., 2020) is a highly optimized grid-world based suite of environments.
Revised: Griddly (Bamford et al., 2020) is a highly optimized grid-world based suite of environments.

Original: Environments used in this work based on Griddly generate 64 × 64 pixel observations, although the size of the grid-world may vary.
Revised: Environments used in this work based on Griddly generate 64 × 64 pixel observations, although the size of the grid-world may vary.

Original: Griddly supports multiple rendering formats, this work uses the 2D rendering of sprites.
Revised: Griddly supports multiple rendering formats, this work uses the 2D rendering of sprites.

Original: Spiders: is a 6x arena where a Gnome (the agent) has 
Revised: Spiders: is a 6x6 arena where a Gnome (the agent) has to grab a Gem without being killed by a

Original: to

Original: grab a Gem without

Original: being

Original: killed

Original: by a

Original: Spider.
Revised: Spider.

Original: The agent dies if it collides with the spider.
Revised: The agent dies if it collides with the spider.

Original: In this environment, the agent can move left , right , up , down or stay .
Revised: In this environment, the agent can move left , right , up , down or stay .

Original: The spider takes an action randomly from the following: rotate left, rotate right or move forward.
Revised: The spider takes an action randomly from the following: rotate left, rotate right or move forward.

Original: This environment’s controlled entities are: Agent.
Revised: This environment’s controlled entities are: Agent.

Original: Clusters: is a 13x10 arena where a Knight has to move boxes of the same color to their corresponding colored-block without touching the spikes.
Revised: Clusters: is a 13x10 arena where a Knight has to move boxes of the same color to their corresponding colored-block without touching the spikes.

Original: There are two different colors, blue and red.
Revised: There are two different colors, blue and red.

Original: The agent is rewarded with +1 whenever a box is pushed towards a similar colored block.
Revised: The agent is rewarded with +1 whenever a box is pushed towards a similar colored block.

Original: The agent dies if it collides with spikes or if a box is destroyed by spikes.
Revised: The agent dies if it collides with spikes or if a box is destroyed by spikes.

Original: The agent can move left , right , up , down or stay .
Revised: The agent can move left , right , up , down or stay .

Original: This environment’s controlled entities are: Agent and Boxes.
Revised: This environment’s controlled entities are: Agent and Boxes.

Original: For RL experiments, we made the environment more sparse by removing all intermediate reward and only rewarded the agent after all the boxes of the same color are pushed to blocks.
Revised: For RL experiments, we made the environment more sparse by removing all intermediate reward and only rewarded the agent after all the boxes of the same color are pushed to blocks.

Original: Since the agent can not get any reward whenever a box is stuck to a wall.
Revised: Since the agent can not get any reward whenever a box is stuck to a wall.

Original: We removed boxes that touches the wall and punished the agent with -0.01.
Revised: We removed boxes that touches the wall and punished the agent with -0.01.

Original: We then scaled the reward of solving a color with the number of boxes pushed to the block.
Revised: We then scaled the reward of solving a color with the number of boxes pushed to the block.

Original: This modiﬁcations encourage the agent to solve the environment by pushing the maximum number of boxes into blocks while preventing it from getting deprived from reward by accidentally pushing boxes to walls.
Revised: This modiﬁcations encourage the agent to solve the environment by pushing the maximum number of boxes into blocks while preventing it from getting deprived from reward by accidentally pushing boxes to walls.

Original: Lights: is a 11x8 arena where a Ghost (the agent) has to turn all the lights on by pressing each button.
Revised: Lights: is a 11x8 arena where a Ghost (the agent) has to turn all the lights on by pressing each button.

Original: Buttons and lights are colored either blue or green.
Revised: Buttons and lights are colored either blue or green.

Original: Pressing a button of one color turns the light of the same color on.
Revised: Pressing a button of one color turns the light of the same color on.

Original: The agent can move left , right , up , down or stay .
Revised: The agent can move left , right , up , down or stay .

Original: This environment’s controlled entities are : Agent, Buttons and Lights
Revised: This environment’s controlled entities are: Agent, Buttons and Lights.

Original: Empty: this environment is a copy of the clusters environment where all 
Revised: Empty: this environment is a copy of the clusters environment where all the boxes, blocks, spikes and rewards were removed.

Original: the

Original: boxes

Original: , blocks, spikes and rewards

Original: were removed. Ground truth: Griddly provides access to each entity’s state (x, y, light is on/off, etc); a binary mask is produced for each controlled entity with any of its attributes changed when transitioning between two time steps.
Revised: Ground truth: Griddly provides access to each entity’s state (x, y, light is on/off, etc); a binary mask is produced for each controlled entity with any of its attributes changed when transitioning between two time steps.

Original: x and y coordinates are projected into pixel space and a bounding box is generated using the size of that entity.
Revised: x and y coordinates are projected into pixel space and a bounding box is generated using the size of that entity.

Original: Note that the coordinates may be different between steps thus the generated mask may enclose multiple locations.
Revised: Note that the coordinates may be different between steps thus the generated mask may enclose multiple locations.

Original: The resulting masks for each entity are combined into a single mask m by taking the maximum value among them.
Revised: The resulting masks for each entity are combined into a single mask m by taking the maximum value among them.

Original: Since we want to know what pixels were actually controlled, the ﬁnal ground truth mask is produced as: m · e pa .
Revised: Since we want to know what pixels were actually controlled, the ﬁnal ground truth mask is produced as: m · e pa .

Original: [Equation]
Revised: [Equation]

Original: The ALE (Bellemare et al., 2012a) provides access to Atari 2600 games to learning methods like
Revised: The ALE (Bellemare et al., 2012a) provides access to Atari 2600 games to learning methods like

Original: RL.
Revised: RL.

Original: As it has been a popular choice by methods using inverse models, in this work we use the game Montezuma’s Revenge to evaluate CEN.
Revised: As it has been a popular choice by methods using inverse models, in this work we use the game Montezuma’s Revenge to evaluate CEN.

Original: This environment provides uncontrolled as well as controlled effects with more complex entities.
Revised: This environment provides uncontrolled as well as controlled effects with more complex entities.

Original: The environment typically generates observations of 210x160 pixels which we downscale to 64x64 pixels.
Revised: The environment typically generates observations of 210x160 pixels which we downscale to 64x64 pixels.

Original: Additionally the action space is of size 10.
Revised: Additionally the action space is of size 10.

Original: Ground truth: in contrast to Griddly, we can actually compute counterfactual worlds by saving and loading the state of the game (RAM) multiple times when taking different actions.
Revised: Ground truth: in contrast to Griddly, we can actually compute counterfactual worlds by saving and loading the state of the game (RAM) multiple times when taking different actions.

Original: For this, we directly compute Eq.
Revised: For this, we directly compute Eq.

Original: 5 using ALE’s special calls cloneSystemState and restoreSystemState .
Revised: 5 using ALE’s special calls cloneSystemState and restoreSystemState .

Original: More precisely, we compute every possible perceived effect reachable from the current state and build a normal effect using the mode over all possible effects.
Revised: More precisely, we compute every possible perceived effect reachable from the current state and build a normal effect using the mode over all possible effects.

Original: Then, we compare the perceived effect for the agent’s chosen action against the normal effect.
Revised: Then, we compare the perceived effect for the agent’s chosen action against the normal effect.

Original: The ground truth mask will have 1s where these two effects are different.
Revised: The ground truth mask will have 1s where these two effects are different.

Original: B T RAINING B. A RCHITECTURE
Revised: C T RAINING C. A RCHITECTURE

Original: Encoder: is composed of two 2D convolutional layers with 4x4 kernels, stride 2 and padding of 1.
Revised: Encoder: is composed of two 2D convolutional layers with 4x4 kernels, stride 2 and padding of 1.

Original: Additionally, we have 2 residual blocks each with two 2D convolutional layers with stride 1 and padding of 1.
Revised: Additionally, we have 2 residual blocks each with two 2D convolutional layers with stride 1 and padding of 1.

Original: The ﬁrst layer has a kernel of 3x3 and the second layer of 1x1.
Revised: The ﬁrst layer has a kernel of 3x3 and the second layer of 1x1.

Original: ReLU is used as activation function throughout the network; BatchNorm is used between each layer; and 64 channels on every convolutional layer.
Revised: ReLU is used as activation function throughout the network; BatchNorm is used between each layer; and 64 channels on every convolutional layer.

Original: We project the resulting maps into a ﬂatten vector of size 32 using a linear layer with ReLU activation function.
Revised: We project the resulting maps into a ﬂatten vector of size 32 using a linear layer with ReLU activation function.

Original: Decoder: this module is composed of six 2D transposed convolutional layers all having 4x4 kernels, stride 2 and padding of 1.
Revised: Decoder: this module is composed of six 2D transposed convolutional layers all having 4x4 kernels, stride 2 and padding of 1.

Original: Each layer uses ReLU as activation function but the output layer which uses Tanh activation.
Revised: Each layer uses ReLU as activation function but the output layer which uses Tanh activation.

Original: Every layer uses 64 channels with the exception of the last layer which outputs a 1 channel prediction of the perceived effects.
Revised: Every layer uses 64 channels with the exception of the last layer which outputs a 1 channel prediction of the perceived effects.

Original: Parameters are shared among the controlled and normal branch decoders.
Revised: Parameters are shared among the controlled and normal branch decoders.

Original: Controlled and normal modules: both modules are composed of three linear layers with 32 hidden units, each with a ReLU as activation function.
Revised: Controlled and normal modules: both modules are composed of three linear layers with 32 hidden units, each with a ReLU as activation function.

Original: The input to the controlled branch are the encoded observation and an embedding of size 8 of the chosen action.
Revised: The input to the controlled branch are the encoded observation and an embedding of size 8 of the chosen action.

Original: PPO Agent: uses an encoder consisting of 3 convolutional layers with (channels, padding, strides) equal to (32, 8, 4), (64, 4, 2), (64, 3, 1) respectively.
Revised: PPO Agent: uses an encoder consisting of 3 convolutional layers with (channels, padding, strides) equal to (32, 8, 4), (64, 4, 2), (64, 3, 1) respectively.

Original: The encoder is followed by two linear layers of sizes 512 and number of actions respectively, to transform the feature map to the environment’s number of actions.
Revised: The encoder is followed by two linear layers of sizes 512 and number of actions respectively, to transform the feature map to the environment’s number of actions.

Original: B. M ASK GENERATION
Revised: C. M ASK GENERATION

Original: CEN’s controlled masks are generated using the encoder, controlled branch and decoder.
Revised: CEN’s controlled masks are generated using the encoder, controlled branch and decoder.

Original: The predicted controlled effect is binarized using a threshold as m CEN =
Revised: The predicted controlled effect is binarized using a threshold as m CEN =

Original: ( − T <
Revised: ( − T <

Original: ˆ e c ) | ( ˆ e c > T ) .
Revised: ˆ e c ) | ( ˆ e c > T ) .

Original: In the case of ADM, its attention mask is thresholded in the same way and resized to the size of the effect
Revised: In the case of ADM, its attention mask is thresholded in the same way and resized to the size of the effect.

Original: [Table]
Revised: [Table]

Original: B. H YPERPARAMETERS

Original: [Table]
Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]