Original: DS-VIC: U NSUPERVISED D ISCOVERY OF D ECISION S TATES FOR T RANSFER IN RL
Revised: DS-VIC: U NSUPERVISED D ISCOVERY OF D ECISION S TATES FOR T RANSFER IN RL

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We learn to identify ‘decision states’, namely the parsimonious set of states where decisions meaningfully affect the future states an agent can reach in an environment.
Revised: We learn to identify ‘decision states’, namely the parsimonious set of states where decisions meaningfully affect the future states an agent can reach in an environment.

Original: We utilize the VIC framework (Gregor et al., 2016), which maximizes an agent’s ‘empowerment’, i.e . the ability to reliably reach a diverse set of states – and formulate a sandwich bound on the empowerment objective that allows identiﬁcation of decision states.
Revised: We utilize the VIC framework (Gregor et al., 2016), which maximizes an agent’s ‘empowerment’, i.e . the ability to reliably reach a diverse set of states – and formulate a sandwich bound on the empowerment objective that allows identiﬁcation of decision states.

Original: Unlike previous work (Goyal et al., 2019), our decision states are discovered without extrinsic rewards – simply by interacting with the world.
Revised: Unlike previous work (Goyal et al., 2019), our decision states are discovered without extrinsic rewards – simply by interacting with the world.

Original: Our results show that our decision states are: 1) often interpretable, and 2) lead to better exploration on downstream goal-driven tasks in partially observable environments.
Revised: Our results show that our decision states are: 1) often interpretable, and 2) lead to better exploration on downstream goal-driven tasks in partially observable environments.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Not all states in a decision making process are created equal.
Revised: Not all states in a decision making process are created equal.

Original: Consider the middle illustration in Figure 1, where a robot has four ‘options’, each representing a goal shape it can potentially reach.
Revised: Consider the middle illustration in Figure 1, where a robot has four ‘options’, each representing a goal shape it can potentially reach.

Original: Given its spawn location, it can initially proceed straight regardless of the option it chooses until the intersection, at which point it needs to utilise 1 bit of (Shannon) information from the option variable to inform the choice of whether to turn left or right.
Revised: Given its spawn location, it can initially proceed straight regardless of the option it chooses until the intersection, at which point it needs to utilise 1 bit of (Shannon) information from the option variable to inform the choice of whether to turn left or right.

Original: However, right after it takes the left turn (say), it again does not need the goal information when choosing actions and can follow ‘default’ or ‘option-independent’ behaviour.
Revised: However, right after it takes the left turn (say), it again does not need the goal information when choosing actions and can follow ‘default’ or ‘option-independent’ behaviour.

Original: Thus, there is a natural difference in the minimum amount of necessary goal information or ‘relevant information’ (Polani et al., 2006) needed at different states.
Revised: Thus, there is a natural difference in the minimum amount of necessary goal information or ‘relevant information’ (Polani et al., 2006) needed at different states.

Original: Identifying these ‘decision states’, i.e . states in the environment where a high amount of relevant goal information is needed, leads to better understanding of the environment structure, which has the potential to enable better transfer to novel environments and tasks.
Revised: Identifying these ‘decision states’, i.e . states in the environment where a high amount of relevant goal information is needed, leads to better understanding of the environment structure, which has the potential to enable better transfer to novel environments and tasks.

Original: This has previously been shown by Goyal et al.
Revised: This has previously been shown by Goyal et al.

Original: in partially-observable, goal-driven ( i.e . with extrinsic rewards) 2 D navigation settings, where using decision states to guide exploration enables faster learning.
Revised: in partially-observable, goal-driven ( i.e . with extrinsic rewards) 2 D navigation settings, where using decision states to guide exploration enables faster learning.

Original: Goal-independent Decision States .
Revised: Goal-independent Decision States .

Original: Our key intuition is that decision states exist in an environment independent of extrinsic goals.
Revised: Our key intuition is that decision states exist in an environment independent of extrinsic goals.

Original: For example, one can imagine the (middle) red tile at the intersection in Figure 1 being a useful decision state – even across all possible navigation goals the agent could have.
Revised: For example, one can imagine the (middle) red tile at the intersection in Figure 1 being a useful decision state – even across all possible navigation goals the agent could have.

Original: Ofcourse, not all intersections are decision states.
Revised: Ofcourse, not all intersections are decision states.

Original: For e.g ., if the left room was full of lava, where the robot would die, thinking of the middle red tile as a decision state would have limited utility, since the relevant goal information (across all reachable goals) in this case would be 0 bits.
Revised: For e.g ., if the left room was full of lava, where the robot would die, thinking of the middle red tile as a decision state would have limited utility, since the relevant goal information (across all reachable goals) in this case would be 0 bits.

Original: Identifying ‘unsupervised’ (or task-agnostic) decision states is advantageous in scenarios where: 1) rewards are sparse or absent (Pathak et al., 2017), 2) for an agent to learn meaningful behaviour, proxy goals and rewards need to be hand engineered (making it hard to scale), and 3) the notion of a goal might not even be obvious, e.g . in continuous control tasks (Lillicrap et al., 2015).
Revised: Identifying ‘unsupervised’ (or task-agnostic) decision states is advantageous in scenarios where: 1) rewards are sparse or absent (Pathak et al., 2017), 2) for an agent to learn meaningful behaviour, proxy goals and rewards need to be hand engineered (making it hard to scale), and 3) the notion of a goal might not even be obvious, e.g . in continuous control tasks (Lillicrap et al., 2015).

Original: Our DS-VIC method identiﬁes decision states without extrinsic rewards, solely by interacting with partially-observable environments.
Revised: Our DS-VIC method identiﬁes decision states without extrinsic rewards, solely by interacting with partially-observable environments.

Original: These decision states are a function of the environment as well as the states an agent can reliably reach.
Revised: These decision states are a function of the environment as well as the states an agent can reliably reach.

Original: We demonstrate that our decision states generalise to novel environments and tasks, leading to comparable (or better) exploration and performance on downstream tasks compared to Goyal et al.
Revised: We demonstrate that our decision states generalise to novel environments and tasks, leading to comparable (or better) exploration and performance on downstream tasks compared to Goyal et al.

Original: that identiﬁes goal-driven decision states.
Revised: that identiﬁes goal-driven decision states.

Original: Decision States in Empowered Agents.
Revised: Decision States in Empowered Agents.

Original: We build on the VIC framework (Gregor et al., 2016)
Revised: We build on the VIC framework (Gregor et al., 2016)

Original: (Figure 1), where the core idea is to maximize the empowerment of an agent.
Revised: (Figure 1), where the core idea is to maximize the empowerment of an agent.

Original: In simple terms, an empowered agent is able to reliably reach a diverse set of states in the environment, while avoiding states that are difﬁcult to reach reliably.
Revised: In simple terms, an empowered agent is able to reliably reach a diverse set of states in the environment, while avoiding states that are difﬁcult to reach reliably.

Original: We use VIC’s formulation to learn options Ω that maximize (a lower bound J V IC on) the mutual information I ( S f , Ω) where S f is the ﬁnal state in a trajectory.
Revised: We use VIC’s formulation to learn options Ω that maximize (a lower bound J V IC on) the mutual information I ( S f , Ω) where S f is the ﬁnal state in a trajectory.

Original: We use latent options since we operate in a setting without external goals.
Revised: We use latent options since we operate in a setting without external goals.

Original: To see why this maximizes empowerment, notice that I ( S f , Ω)
Revised: To see why this maximizes empowerment, notice that I ( S f , Ω)

Original: =
Revised: =

Original: H ( S f ) − H
Revised: H ( S f ) − H

Original: ( S f | Ω) , where H ( . ) denotes entropy.
Revised: ( S f | Ω) , where H ( . ) denotes entropy.

Original: Thus, empowerment maximizes the diversity in ﬁnal states S f while learning options highly predictive of S f (Salge et al., 2013).
Revised: Thus, empowerment maximizes the diversity in ﬁnal states S f while learning options highly predictive of S f (Salge et al., 2013).

Original: Using VIC, we identify decision states by attempting to hit the relevant information required at a particular state by minimizing the mutual information
Revised: Using VIC, we identify decision states by attempting to hit the relevant information required at a particular state by minimizing the mutual information

Original: I (Ω , A t | S t ) (for every state S t , action A t ).
Revised: I (Ω , A t | S t ) (for every state S t , action A t ).

Original: This is similar in spirit to Goyal et al.
Revised: This is similar in spirit to Goyal et al.

Original: ; Polani et al.
Revised: ; Polani et al.

Original: (2006), with the key difference that we use latent options Ω instead of external goals.
Revised: (2006), with the key difference that we use latent options Ω instead of external goals.

Original: Interestingly, we prove that  t I (Ω , A t | S t ) is an upper bound on empowerment, which we minimize, in addition to maximizing the VIC lower bound J V IC .
Revised: Interestingly, we prove that (cid:80) t I (Ω , A t | S t ) is an upper bound on empowerment, which we minimize, in addition to maximizing the VIC lower bound J V IC .

Original: This shines a connection to VIC– while they optimize a lower bound, our work optimizes a sandwich bound on the empowerment 1 .
Revised: This shines a connection to VIC– while they optimize a lower bound, our work optimizes a sandwich bound on the empowerment 1 .

Original: To summarize our contributions,
Revised: To summarize our contributions,

Original: • We propose DS-VIC, a novel framework to identify decision states in a task-agnostic manner.
Revised: • We propose DS-VIC, a novel framework to identify decision states in a task-agnostic manner.

Original: These decisionstates
Revised: These decision

Original:  align with our intuitive assessments in various partially
Revised: states align with our intuitive assessments in various partially

Original: observed scenarios
Revised: -observed scenarios

Original: • We provide an understanding of our proposed objective, proving that our framework optimizes a sandwich bound on the empowerment
Revised: • We provide an understanding of our proposed objective, proving that our framework optimizes a sandwich bound on the empowerment

Original: I (Ω , S f ) .
Revised: I (Ω , S f ) .

Original: • We show that our mechanism to identify decision states is transferable and leads to improved sample-efﬁciency on goal-driven tasks in novel environments.
Revised: • We show that our mechanism to identify decision states is transferable and leads to improved sample-efﬁciency on goal-driven tasks in novel environments.

Original: On a challenging grid-world navigation task, our method outperforms (a re-implementation of) Goyal
Revised: On a challenging grid-world navigation task, our method outperforms (a re-implementation of) Goyal

Original: et al.
Revised: et al.

Original:  M ETHODS 2. N OTATION
Revised:  M ETHODS 2. N OTATION

Original: We consider a Partially Observable Markov Decision Process (POMDP), deﬁned by the tuple ( S , A , P , r ) , where the (discrete or continuous) state-space S is a set of underlying but unobserved states s ∈ S and A denotes a discrete action space.
Revised: We consider a Partially Observable Markov Decision Process (POMDP), deﬁned by the tuple ( S , A , P , r ) , where the (discrete or continuous) state-space S is a set of underlying but unobserved states s ∈ S and A denotes a discrete action space.

Original: P : S × S ×
Revised: P : S × S ×

Original: A denotes an unknown transition function, representing p ( s t + | s t , a t ) : s t , s t +1 ∈ S , A t ∈ A .
Revised: A denotes an unknown transition function, representing p ( s t + | s t , a t ) : s t , s t +1 ∈ S , A t ∈ A .

Original: Both VIC and DS-VIC train an option ( Ω ) conditioned policy π ( a t | ω, s t ) , where ω ∈ { 1 , · · · , | Ω |} .
Revised: Both VIC and DS-VIC train an option ( Ω ) conditioned policy π ( a t | ω, s t ) , where ω ∈ { 1 , · · · , | Ω |} .

Original: Following standard practice (Cover & Thomas, 1991), we denote random variables in uppercase ( Ω ), and items from the sample space of random variables in lowercase ( ω ).
Revised: Following standard practice (Cover & Thomas, 1991), we denote random variables in uppercase ( Ω ), and items from the sample space of random variables in lowercase ( ω ).

Original: 2. V ARIATIONAL I NTRINSIC C ONTROL (VIC)
Revised: 2. V ARIATIONAL I NTRINSIC C ONTROL (VIC)

Original: The option Ω in VIC is a global latent variable underlying a trajectory τ = { S 0 , A 0 , · · · , S f } (Figure 2).
Revised: The option Ω in VIC is a global latent variable underlying a trajectory τ = { S 0 , A 0 , · · · , S f } (Figure 2).

Original: To encourage the agent to reach a diverse set of states reliably, VIC proposes maximizing the mutual information (Cover & Thomas, 1991) between Ω and ﬁnal state S f given s 0 ,
Revised: To encourage the agent to reach a diverse set of states reliably, VIC proposes maximizing the mutual information (Cover & Thomas, 1991) between Ω and ﬁnal state S f given s 0 ,

Original: i.e . I
Revised: i.e . I

Original: ( S f , Ω |
Revised: ( S f , Ω |

Original: S 

Revised: S

Original: = s 0 ) .
Revised: = s 0 ) .

Original: Informally, this maximizes the empowerment for an agent, i.e . its internal options Ω have a high degree of correspondence to the states of the world S
Revised: Informally, this maximizes the empowerment for an agent, i.e . its internal options Ω have a high degree of correspondence to the states of the world S

Original: f
Revised: f

Original: that it can reach.
Revised: that it can reach.

Original: VIC formulates a variational lower bound on this mutual information.
Revised: VIC formulates a variational lower bound on this mutual information.

Original: Speciﬁcally, let p ( ω | s 0 )
Revised: Speciﬁcally, let p ( ω | s 0 )

Revised: = p ( ω ) be a prior on options (we keep the prior ﬁxed as per Eysenbach et al.

Original: = p ( ω ) be a prior on options (assumed to be same for all s 0 ), p J ( s f | ω, s 0 ) denote the (unknown) terminal state distribution achieved when executing the policy π ( a t | ω, s t ) , and q ν ( ω | s f , s 0 ) denote a 
Revised: p J ( s f | ω, s 0 ) is deﬁned as the (unknown) terminal state distribution achieved when executing the policy π ( a t | ω, s t ) , and q ν ( ω | s f , s 0 ) denote a (parameterized) variational approximation to the true posterior on options

Original: [Table]
Revised: [Table]

Original: /x7UyiTFhA7ioqFQwFF2DWFcCbMr5R2mGEcTasGE4IyfPEku96rOftU+Pygdn4ziWCBbZJuUiUMOyTE5I3XSIJw8kmfySt6sJ+vFerf636VT1qhnk/yB9fkFICiiPA==</latexit>
Revised: /x7UyiTFhA7ioqFQwFF2DWFcCbMr5R2mGEcTasGE4IyfPEku96rOftU+Pygdn4ziWCBbZJuUiUMOyTE5I3XSIJw8kmfySt6sJ+vFerf636VT1qhnk/yB9fkFICiiPA==</latexit>

Original: VIC:
Revised: VIC:

Original: DS-VIC: Policy Parameterization + VIC
Revised: DS-VIC: Policy Parameterization + VIC

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: (parameterized) variational approximation to the true posterior on options given S f and S 0 .
Revised: given S f and S 0 .

Original: Then:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: 2. D ECISON S TATE VIC (DS-VIC)
Revised: 2. D ECISON S TATE VIC (DS-VIC)

Original: We identify decision states by employing an information regularizer that ﬁnds states with minimum relevant option information.
Revised: We identify decision states by employing an information regularizer that ﬁnds states with minimum relevant option information.

Original: Formally, this means that at every timestep t in the trajectory, we minimize the mutual information
Revised: Formally, this means that at every timestep t in the trajectory, we minimize the mutual information

Original: I (Ω , A t | S t , S 

Revised: I (Ω , A t | S t , S

Original: = s ) .
Revised: = s ) and the resulting states where this mutual information remains high despite the minimization are identiﬁed as decision states.

Original: Intuitively, this means that on average (across different options), a decision state is a state with higher relevant option information than other states ( e.g . the red regions in Figure 1). Overall, our objective is to maximize:
Revised: Intuitively, this means that on average (across different options), a decision state is a state with higher relevant option information than other states ( e.g . the red regions in Figure 1) Overall, our objective is to maximize:

Original: [Equation]
Revised: [Equation]

Original: where β > 0 is a trade-off parameter.
Revised: where β > 0 is a trade-off parameter.

Original: Thus, this is saying that one wants options Ω which allow the agent to have a high empowerment, while utilizing the least relevant information at each step.
Revised: Thus, this is saying that one wants options Ω which allow the agent to have a high empowerment, while utilizing the least relevant information at each step.

Original: Interestingly, Equation 2 has a clear, principled interpretation in terms of the empowerment
Revised: Interestingly, Equation 2 has a clear, principled interpretation in terms of the empowerment

Original: I (Ω , S f | S 0 ) from the VIC model.
Revised: I (Ω , S f | S 0 ) from the VIC model.

Original: We state the following lemma (full proof in Appendix):
Revised: We state the following lemma (full proof in Appendix):

Original: Lemma 2.1 Let A t be the action random variable at timestep t and state S t following an optionconditioned policy π ( a t | s t , ω ) .
Revised: Lemma 2.1 Let A t be the action random variable at timestep t and state S t following an optionconditioned policy π ( a t | s t , ω ) .

Original: Then, I (Ω , A t | S t , S 0 ) i.e .
Revised: Then, I (Ω , A t | S t , S 0 ) i.e .

Original: the conditional mutual information between the option Ω and action A t when summed over all timesteps in the trajectory, upper bounds the conditional mutual information I (Ω , S f | S 0 ) between Ω and the ﬁnal state S f – namely the empowerment as deﬁned by Gregor et al.
Revised: the conditional mutual information between the option Ω and action A t when summed over all timesteps in the trajectory, upper bounds the conditional mutual information I (Ω , S f | S 0 ) between Ω and the ﬁnal state S f – namely the empowerment as deﬁned by Gregor et al.

Original: [Equation]
Revised: [Equation]

Original: Implications : With this lens, one can view the optimization problem in Equation 2 as a Lagrangian relaxation of the following constrained optimization problem:
Revised: Implications : With this lens, one can view the optimization problem in Equation 2 as a Lagrangian relaxation of the following constrained optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where R > 0 is a constant.
Revised: where R > 0 is a constant.

Original: Thus, we can essentially target values of R we have in mind by maximizing the lower bound and minimizing the upper bound, by (in practice) sweeping different values of β in eq.
Revised: Thus, we can essentially target values of R we have in mind by maximizing the lower bound and minimizing the upper bound, by (in practice) sweeping different values of β in eq.

Original: This is a result potentially of interest beyond the speciﬁc context of decision states (Achiam et al., 2018; Gregor et al., 2016).
Revised: This is a result potentially of interest beyond the speciﬁc context of decision states (Achiam et al., 2018; Gregor et al., 2016).

Original: Speciﬁcally, while the VIC objective only allows us to maximize the empowerment
Revised: Speciﬁcally, while the VIC objective only allows us to maximize the empowerment

Original: I (Ω , S f ) =
Revised: I (Ω , S f ) =

Original: H ( S f ) − H
Revised: H ( S f ) − H

Original: ( S f | Ω) , imposing an upper bound can help control how ‘abstract’ the latent option representation is relative to the states S f , by ﬁnding solutions that have say a higher entropy H ( S f | Ω) on states given options.
Revised: ( S f | Ω) , imposing an upper bound can help control how ‘abstract’ the latent option representation is relative to the states S f , by ﬁnding solutions that have say a higher entropy H ( S f | Ω) on states given options.

Original: This might lead to more abstract options that achieve better generalisation depending on downstream tasks.
Revised: This might lead to more abstract options that achieve better generalisation depending on downstream tasks.

Original: Note that most approaches currently limit the abstraction by constraining the number of discrete options, which (usually) imposes a (tighter) upper bound onthe mutual information of H (Ω) , since H ≥ 0 in the discrete case.
Revised: Note that most approaches currently limit the abstraction by constraining the number of discrete options, which (usually) imposes a (tighter) upper bound on H (Ω) , since H ≥ 0 in the discrete case.

Original: However, this does not hold for the continuous case, where the bound might be more useful.
Revised: However, this does not hold for the continuous case, where the bound might be more useful.

Original: Investigating this is beyond the scope of this current paper, however, as our central aim is to identify decision states, and not to scale the VIC framework to continuous options.
Revised: Investigating this is beyond the scope of this current paper, however, as our central aim is to identify decision states, and not to scale the VIC framework to continuous options.

Original: 2. A LGORITHMIC D ETAILS
Revised: 2. A LGORITHMIC D ETAILS

Original: Upper Bounds for I (Ω , A t | S t , S 0 ) .
Revised: Upper Bounds for I (Ω , A t | S t , S 0 ) .

Original: Inspired by InfoBot (Goyal et al., 2019), we bottleneck the information in a statistic Z t of the state S t and option Ω used to parameterize the policy π
Revised: Inspired by InfoBot (Goyal et al., 2019), we bottleneck the information in a statistic Z t of the state S t and option Ω used to parameterize the policy π

Original: ( A t | Ω , S t ) (ﬁg. 2 right).
Revised: ( A t | Ω , S t ) (ﬁg. 2 right).

Original: This is justiﬁed by the the data-processing inequality (DPI) (Cover & Thomas, 1991) for the markov chain Ω , S t ↔ Z t ↔ A t , which implies I (Ω , A t | S t , S 0 ) ≤
Revised: This is justiﬁed by the the data-processing inequality (DPI) (Cover & Thomas, 1991) for the markov chain Ω , S t ↔ Z t ↔ A t , which implies I (Ω , A t | S t , S 0 ) ≤

Original: I (Ω , Z t | S t , S 0 ) .
Revised: I (Ω , Z t | S t , S 0 ) .

Original: We can then obtain the following upper bound on I (Ω , Z t
Revised: We can then obtain the following upper bound on I (Ω , Z t

Original: [Equation]
Revised: [Equation]

Original: t , S

Original: [Equation]

Original: ) (see appendix for derivation):
Revised: ) (see appendix for derivation):

Original: [Equation]
Revised: [Equation]

Original: where q ( z t ) is a ﬁxed variational approximation (set to N (0 , I ) as in InfoBot), and p φ ( z t | ω, s t ) is a parameterized encoder.
Revised: where q ( z t ) is a ﬁxed variational approximation (set to N (0 , I ) as in InfoBot), and p φ ( z t | ω, s t ) is a parameterized encoder.

Original: As explained in section 1 the key difference between eq.
Revised: As explained in section 1 the key difference between eq.

Original: (5) and InfoBot is that they construct upper bounds on I ( G, A t | S t , S 0 ) , while we bottleneck the option-information.
Revised: (5) and InfoBot is that they construct upper bounds on I ( G, A t | S t , S 0 ) , while we bottleneck the option-information.

Original: We can compute a Monte Carlo estimate of Equation 5 by ﬁrst sampling an option ω at s 0 and then keeping track of all states visited in trajectory τ .
Revised: We can compute a Monte Carlo estimate of Equation 5 by ﬁrst sampling an option ω at s 0 and then keeping track of all states visited in trajectory τ .

Original: In addition to the VIC term and our bottleneck regularizer, we also include the entropy of the policy over the actions (maximum-entropy RL (Ziebart, 2010)) as a bonus to encourage sufﬁcient exploration.
Revised: In addition to the VIC term and our bottleneck regularizer, we also include the entropy of the policy over the actions (maximum-entropy RL (Ziebart, 2010)) as a bonus to encourage sufﬁcient exploration.

Original: Overall, the DS-VIC objective is:
Revised: Overall, the DS-VIC objective is:

Original: [Equation]
Revised: [Equation]

Original: where θ, φ and ν are the parameters of the policy, latent variable decoder and the option inference network respectively.
Revised: where θ, φ and ν are the parameters of the policy, latent variable decoder and the option inference network respectively.

Original: Note that we abuse the notation slightly in Equation 6 by writing τ ∼ p J ( · | ω, s 0 ) to denote a trajectory sampled by following π ( · | ω, s 0 ) .
Revised: Note that we abuse the notation slightly in Equation 6 by writing τ ∼ p J ( · | ω, s 0 ) to denote a trajectory sampled by following π ( · | ω, s 0 ) .

Original: This is a sequential process where
Revised: This is a sequential process where

Original: Z t ∼ p φ ( z t | s t , ω ) , A t ∼ π θ ( a t | s t , z t ) and S t +1 ∼ p ( s t +1 | s t , a t ) (state transition function).
Revised: Z t ∼ p φ ( z t | s t , ω ) , A t ∼ π θ ( a t | s t , z t ) and S t +1 ∼ p ( s t +1 | s t , a t ) (state transition function).

Original: The ﬁrst term in the objective ensures reliable control in terms of visitation over a diverse set of ﬁnal states in the environment while learning options; the second term ensures minimality in using the options sampled to make decisions (taking actions) and the third provides an incentive for exploration .
Revised: The ﬁrst term in the objective ensures reliable control in terms of visitation over a diverse set of ﬁnal states in the environment while learning options; the second term ensures minimality in using the options sampled to make decisions (taking actions) and the third provides an incentive for exploration .

Original: See algorithm 1 in appendix for more details on the entire training pipeline.
Revised: See algorithm 1 in appendix for more details on the entire training pipeline.

Original: Handling Partial Observability.
Revised: Handling Partial Observability.

Original: We are generally interested in identifying decision states in partially observable environments (similar to InfoBot Goyal et al.
Revised: We are generally interested in identifying decision states in partially observable environments (similar to InfoBot Goyal et al.

Original: To formalise how we handle this, let o t be the (partial) observation made by the agent at time t .
Revised: To formalise how we handle this, let o t be the (partial) observation made by the agent at time t .

Original: We set the state s t = g
Revised: We set the state s t = g

Original: ( o 0 , · · · , o t ) in all the equations above, where g is a recurrent neural network (RNN) (Hochreiter & Schmidhuber, 1997) encoding partial observations.
Revised: ( o 0 , · · · , o t ) in all the equations above, where g is a recurrent neural network (RNN) (Hochreiter & Schmidhuber, 1997) encoding partial observations.

Original: We found it important to parameterize the policy (Figure 2, right) as: π ( a t | ω, s t ) =
Revised: We found it important to parameterize the policy (Figure 2, right) as: π ( a t | ω, s t ) =

Original: (cid:82) dz p ( z | s t , ω ) p ( a t | o t , z ) , i.e . where action distribution p ( a t | o t , z ) are modelled as ‘reactive’ (conditioned only on the current observation) but z distributions p ( z t | s t , ω ) are ‘recurrent‘ (conditioning on the entire history of observations via g ( · ) ).
Revised: (cid:82) dz p ( z | s t , ω ) p ( a t | o t , z ) , i.e . where action distribution p ( a t | o t , z ) are modelled as ‘reactive’ (conditioned only on the current observation) but z distributions p ( z t | s t , ω ) are ‘recurrent‘ (conditioning on the entire history of observations via g ( · ) ).

Original: This is because the sequence of observations { o 1 , · · · , o t } might be informative of the option Ω being followed i.e . conditioning p ( a t | · ) on S t in addition to z could ‘leak’ information about the underlying option to the actions, making it the bottleneck penalty on z potentially ineffective.
Revised: This is because the sequence of observations { o 1 , · · · , o t } might be informative of the option Ω being followed i.e . conditioning p ( a t | · ) on S t in addition to z could ‘leak’ information about the underlying option to the actions, making it the bottleneck penalty on z potentially ineffective.

Original: Finally, when maximizing the likelihood of option Ω given the ﬁnal state S f
Revised: Finally, when maximizing the likelihood of option Ω given the ﬁnal state S f

Original: in learning q ( ω | s f ) in VIC (the option inference network), we use the global ( x, y ) coordinates of the ﬁnal  state of the agent.
Revised: in learning q ( ω | s f ) in VIC (the option inference network), we use the global ( x, y ) coordinates of the ﬁnal and initial state of the agent.

Original: Note that the agent’s policy continues to be based only on observations o t , and it is only at the end of the trajectory that we use the global coordinates to compute intrinsic reward.
Revised: Note that the agent’s policy continues to be based only on observations o t , and it is only at the end of the trajectory that we use the global coordinates to compute intrinsic reward.

Original: Please refer to Sec.
Revised: Please refer to Sec.

Original: A.6 for a discussion of the issues when training the model with partial observations in the option inference network.
Revised: A.6 for a discussion of the issues when training the model with partial observations in the option inference network.

Original: Related Objectives.
Revised: Related Objectives.

Original: Eysenbach et al.
Revised: Eysenbach et al.

Original: (2018) (DIAYN) attempts to learn skills (similar to options) which can control the states visited by agents while ensuring that states, not actions, are used to distinguish skills.
Revised: (2018) (DIAYN) attempts to learn skills (similar to options) which can control the states visited by agents while ensuring that states, not actions, are used to distinguish skills.

Original: Thus, for an option Ω and every state S t in a trajectory, they maximize (cid:80) t
Revised: Thus, for an option Ω and every state S t in a trajectory, they maximize (cid:80) t

Original: I (Ω , S t ) − I ( A t , Ω | S t )
Revised: I (Ω , S t ) − I ( A t , Ω | S t )

Original: + H
Revised: + H

Original: ( A t | S t ) , as opposed to I (Ω , S f ) − β (cid:80) t I ( A t , Ω | S t )+ H ( A t | S t ) in our objective.
Revised: ( A t | S t ) , as opposed to I (Ω , S f ) − β (cid:80) t I ( A t , Ω | S t )+ H ( A t | S t ) in our objective.

Original: With the sum over all timesteps for I (Ω , S t ) , the bound in theorem 2.1 no longer holds true, which also means that there is no principled reason (unlike our model) to scale the second term with β – which we ﬁnd to be crucial for identifying decision states (similar to ﬁndings in InfoBot).
Revised: With the sum over all timesteps for I (Ω , S t ) , the bound in theorem 2.1 no longer holds true, which also means that there is no principled reason (unlike our model) to scale the second term with β – which we ﬁnd to be crucial for identifying decision states (similar to ﬁndings in InfoBot).

Original: Experimentally, we show that the decision states with DIAYN are no longer as interpretable, since options in this case are more low level and 
Revised: Experimentally, we show that the decision states with DIAYN are no longer as interpretable, since options in this case are more low level and correspond to trajectories instead of goals.

Original: The most closely related work to ours is InfoBot, which maximizes [Equation] t R ( t ) − βI ( A t , G | S t ) for a goal ( G ) conditioned policy π ( a t | S t , G ) .
Revised: The most closely related work to ours is InfoBot, which maximizes (cid:80) t R ( t ) − βI ( A t , G | S t ) for a goal ( G ) conditioned policy π ( a t | S t , G ) .

Original: The key difference is that InfoBot requires extrinsic rewards, while our work is strictly more general and scales even in the absence of extrinsic rewards.
Revised: The key difference is that InfoBot requires extrinsic rewards, while our work is strictly more general and scales even in the absence of extrinsic rewards.

Original: Further, in context of both these works, our work provides a principled connection between actionoption information reqularization
Revised: Further, in context of both these works, our work provides a principled connection between actionoption information reqularization

Original: I ( A t , Ω | S t ) and empowerment of an agent.
Revised: I ( A t , Ω | S t ) and empowerment of an agent.

Original: The tools from Theorem 2.1 might be useful for analysing these previous objectives which both employ this technique.
Revised: The tools from Theorem 2.1 might be useful for analysing these previous objectives which both employ this technique.

Original:  E XPERIMENTS 3. T RANSFER TO G OAL -D RIVEN T ASKS
Revised:  E XPERIMENTS 3. T RANSFER TO G OAL -D RIVEN T ASKS

Original: Do the unsupervised decision states meaningfully transfer to novel tasks?
Revised: Do the unsupervised decision states meaningfully transfer to novel tasks?

Original: To address this, we replicate the InfoBot experimental setup, replacingthe decision state identiﬁcation part with DS-VIC.
Revised: To address this, we replicate the InfoBot experimental setup, replacing decision state identiﬁcation (pre-training) with DS-VIC.

Original: Exploration-based Transfer .
Revised: Exploration-based Transfer .

Original: Goyal et al.
Revised: Goyal et al.

Original: (2019) pretrain their model to identify decision states, and then study if decision states improve exploration when training a new policy π γ ( a | s, g ) from scratch.
Revised: (2019) pretrain their model to identify decision states, and then study if decision states improve exploration when training a new policy π γ ( a | s, g ) from scratch.

Original: Given an environment with reward R e ( t ) , goal G , κ > 0 , and state visitation count c ( S t ) , their reward is:
Revised: Given an environment with reward R e ( t ) , goal G , κ > 0 , and state visitation count c ( S t ) , their reward is:

Original: [Equation]
Revised: [Equation]

Original: The count-based reward 2 decays with square root of c ( S t ) to encourage the model to explore novel states, and the mutual information between goal G and bottleneck variable Z t identiﬁes decision states, and is multiplied with the exploration bonus to encourage visitation of decision states.
Revised: The count-based reward 2 decays with square root of c ( S t ) to encourage the model to explore novel states, and the mutual information between goal G and bottleneck variable Z t identiﬁes decision states, and is multiplied with the exploration bonus to encourage visitation of decision states.

Original: [Figure]
Revised: [Figure]

Original: [Equation] 
Revised: [Equation] [Figure] [Equation] [Table]

Original: [Table]
Revised: "b/RawkXkc2cxCmi+lkBqg/YJl [Figure] [Figure] AAAB9HicbVDLSgNBEJyNrxhfUY9eBoPgadmNgh6DXjxGMA9IljA76U2GzD6c6Q2GJd/hxYMiXv0Yb/6Nk2QPmljQUFR1093lJ1JodJxvq7C2vrG5Vdwu7ezu7R+UD4+aOk4VhwaPZazaPtMgRQQNFCihnShgoS+h5Y9uZ35rDEqLOHrASQJeyAaRCARnaCSvi/CEiFnVrlanvXLFsZ056Cpxc1IhOeq98le3H

Original: [Table]
Revised: [Figure]

Original: We use an almost identical setup, replacing their decision-state term from supervised pretraining with DS-VIC intrinsic reward pretraining:
Revised: We use an almost identical setup, replacing their decision-state term from supervised pretraining with DS-VIC intrinsic reward pretraining:

Original: [Equation]
Revised: [Equation]

Original: I ( · ) is computed with eq.
Revised: I ( · ) is computed with eq.

Original: (5) with a parameterized p ( z t | ω, s t ) encoder frozen during transfer.
Revised: (5) with a parameterized p ( z t | ω, s t ) encoder frozen during transfer.

Original: See Algorithm 1 in appendix for more details.
Revised: See Algorithm 1 in appendix for more details.

Original: Environments.
Revised: Environments.

Original: We pre-train and test on gridworlds from the MiniGrid (Chevalier-Boisvert et al., 2018) environments.
Revised: We pre-train and test on gridworlds from the MiniGrid (Chevalier-Boisvert et al., 2018) environments.

Original: We ﬁrst consider a set of simple environments – 4-Room and Maze (see Fig. 3) followed by the MultiRoomNXSY also used by Goyal et al.
Revised: We ﬁrst consider a set of simple environments – 4-Room and Maze (see Fig. 3) followed by the MultiRoomNXSY also used by Goyal et al.

Original: The MultiRoomNXSY environments consist of X rooms of size Y, connected in random orientations across multiple runs.
Revised: The MultiRoomNXSY environments consist of X rooms of size Y, connected in random orientations across multiple runs.

Original: We also conduct experiments on Mountain-Car (Brockman et al., 2016), a continuous state-space environment with discrete actions.
Revised: We also conduct experiments on Mountain-Car (Brockman et al., 2016), a continuous state-space environment with discrete actions.

Revised: In all pre-training environments, options are executed for a ﬁxed number of steps before terminating.

Original: We use Advantage Actor-Critic (A2C) for all experiments.
Revised: We use Advantage Actor-Critic (A2C) for all experiments.

Original: Since code for InfoBot (Goyal et al., 2019) was not public, we report numbers based on a re-implementation of InfoBot, ensuring consistency with their architectural and hyperparameter choices.
Revised: Since code for InfoBot (Goyal et al., 2019) was not public, we report numbers based on a re-implementation of InfoBot, ensuring consistency with their architectural and hyperparameter choices.

Original: We will release our code upon publication.
Revised: We will release our code upon publication.

Original:  R ESULTS
Revised:  R ESULTS

Original: Baselines.
Revised: Baselines.

Original: We evaluate the following on quality of exploration and transfer to downstream goal-driven tasks with sparse rewards: 1) InfoBot (our implementation) – which does goal-driven extraction of decision states, 2) DIAYN – whose focus is unsupervised skill acquisition (and not decision states) 3 , but has an I ( A t , Ω | S t ) term which can be used for the bonus in Equation 8, 3) count-based exploration which uses visitation counts as exploration incentive (this corresponds to replacing
Revised: We evaluate the following on quality of exploration and transfer to downstream goal-driven tasks with sparse rewards: 1) InfoBot (our implementation) – which does goal-driven extraction of decision states, 2) DIAYN – whose focus is unsupervised skill acquisition (and not decision states) 3 , but has an I ( A t , Ω | S t ) term which can be used for the bonus in Equation 8, 3) count-based exploration which uses visitation counts as exploration incentive (this corresponds to replacing

Original: I (Ω , Z t | S t , S 0 ) with 1 in Equation 8), 4) a randomly initialized encoder p ( z t | ω, s t ) to check if learning in DS-VIC or InfoBot models improves over random initialization, 5) how different values of β affect performance, and 6) a heuristic baseline that looks for local features like corners and doorways and uses every such occurrence as a decision state.
Revised: I (Ω , Z t | S t , S 0 ) with 1 in Equation 8), 4) a randomly initialized encoder p ( z t | ω, s t ) to check if learning in DS-VIC or InfoBot models improves over random initialization, 5) how different values of β affect performance, and 6) a heuristic baseline that looks for local features like corners and doorways and uses every such occurrence as a decision state.

Original: This validates the extent to which relevant goal information is useful in identifying decision states, vs .
Revised: This validates the extent to which relevant goal information is useful in identifying decision states, vs .

Original: local, feature-based heuristics.
Revised: local, feature-based heuristics.

Original: 4. Q UALITATIVE R ESULTS
Revised: 4. Q UALITATIVE R ESULTS

Original: Grid Worlds: Figure 3 shows results on 4-Room and maze .
Revised: Grid Worlds: Figure 3 shows results on 4-Room and maze .

Original: We notice that when β
Revised: We notice that when β

Original: = 1 , DS-VIC collapses to identifying every state as a decision state and gets poor empowerment: 0 .
Revised: = 1 , DS-VIC collapses to identifying every state as a decision state and gets poor empowerment: 0 .

Original: 51 (row 1, column 3 in Figure 3), indicating that β is too large and the upper bound on the empowerment too tight.
Revised: 51 (row 1, column 3 in Figure 3), indicating that the regularization strength is too high and hinders empowerment maximization.

Original: At lower values of β = 1e-3 , we get more discernible decision states.
Revised: At lower values of β = 1e-3 , we get more discernible decision states.

Original: Across both environments, we notice that some corners and intersections are identiﬁed as decision states.
Revised: Across both environments, we notice that some corners and intersections are identiﬁed as decision states.

Original: In general, for DS-VIC on maze with β = 1e-3 (when it gets an empowerment of 2 . 61 ), we notice that if an intersection is a decision state, and there is a corner next to it, it is often not a decision  state since it is likely one is following default behaviour having already made the decision.
Revised: In general, for DS-VIC on maze with β = 1e-3 (when it gets an empowerment of 2 . 61 ), we notice that if an intersection is a decision state, and there is a corner next to it, it is often not a decision state. That is, not every state which looks like an corner needs to be a decision state, if for example the part of the state space in question is only spanned by one option.

Original: Finally, for maze we see that for a similar value of empowerment 4 , DS-VIC leads to a sparser decision state distribution than DIAYN.
Revised: Finally, for maze we see that for a similar value of empowerment 4 , DS-VIC leads to a sparser decision state distribution than DIAYN.

Original: This makes sense, since intuitively, one expects to have to make more decisions when following a particular trajectory (as in DIAYN), as opposed to a terminal state (as in DS-VIC).
Revised: This makes sense, since intuitively, one expects to have to make more decisions when following a particular trajectory (as in DIAYN), as opposed to a terminal state (as in DS-VIC).

Original: [Figure]
Revised: [Figure]

Original: Mountain Car Environment: We also use DS-VIC to identify decision states in the Mountain
Revised: Mountain Car Environment: We also use DS-VIC to identify decision states in the Mountain

Original: Car (Brockman et al., 2016) environment – a continuous state-space (position and velocity of the car) environment with three discrete actions { +1( right ) , 0 , − 1( left ) } , which correspond to the direction of the force being applied to the car (see Fig. 4(a)).
Revised: Car (Brockman et al., 2016) environment – a continuous state-space (position and velocity of the car) environment with three discrete actions { +1( right ) , 0 , − 1( left ) } , which correspond to the direction of the force being applied to the car (see Fig. 4(a)).

Original: Fig. 4(b) shows the trajectories and Fig. 4(c) shows the ﬁnal state(s) corresponding to 3 speciﬁc options (out of 8) overlaid on a position-velocity plane.
Revised: Fig. 4(b) shows the trajectories and Fig. 4(c) shows the ﬁnal state(s) corresponding to 3 speciﬁc options (out of 8) overlaid on a position-velocity plane.

Original: Upon observing decision states in the same position-velocity plane (see
Revised: Upon observing decision states in the same position-velocity plane (see

Original: Fig. 4(d), point-radius indicating I (Ω , Z t | s t , s 0 ) , we ﬁnd that decision states occur at points in the state-space where trajectories associated with different options intersect.
Revised: Fig. 4(d), point-radius proportional to I (Ω , Z t | s t , s 0 ) , we ﬁnd that decision states occur at points in the state-space where trajectories associated with different options intersect.

Original: Intuitively, these are the states where the agent can switch from one option (mode of behavior) to another and therefore, needs to make a decision.
Revised: Intuitively, these are the states where the agent can switch from one option (mode of behavior) to another and therefore, needs to make a decision.

Original: Interestingly, there is a dominance of decision states along the velocity = 0 line, which indicates that the model identiﬁes decision states to be points where the cart has velocity =
Revised: Interestingly, there is a dominance of decision states along the velocity = 0 line, which indicates that the model identiﬁes decision states to be points where the cart has velocity =

Original: at some position (speciﬁc to the option).
Revised: at some position (speciﬁc to the option).

Original: Finally, decision states often occur, for say, the orange option ( Ω 1 ) in the right half of the position-velocity plane and never in the bottom-left quadrant, where it is the only option covering that part of the state-space, and thus default behavior can be practiced. 
Revised: Finally, decision states often occur, for say, the orange option ( Ω 1 ) in the right half of the position-velocity plane and never in the bottom-left quadrant, where it is the only option covering that part of the state-space, and thus default behavior can be practiced. [Table]

Original: 4. Q UANTITATIVE R ESULTS
Revised: 4. Q UANTITATIVE R ESULTS

Original: Transfer to Goal-Driven Tasks .
Revised: Transfer to Goal-Driven Tasks .

Original: Next, we evaluate Equation 8, i.e . whether incentivizing the agent to visit decision states in addition to extrinsic reward can aid in transfer to goal-driven tasks in different environments (see Sec. 3.1).
Revised: Next, we evaluate Equation 8, i.e . whether incentivizing the agent to visit decision states in addition to extrinsic reward can aid in transfer to goal-driven tasks in different environments (see Sec. 3.1).

Original: We restrict ourselves to the point-navigation task (Goyal et al., 2019) transfer in the MultiRoomNXSY set of partially-observable environments.
Revised: We restrict ourselves to the point-navigation task (Goyal et al., 2019) transfer in the MultiRoomNXSY set of partially-observable environments.

Original: In this task, the agent is always spawned in the ﬁrst room, and has to go to a randomly sampled goal location in the last room, and is rewarded only when it reaches the correct cell.
Revised: In this task, the agent is always spawned in the ﬁrst room, and has to go to a randomly sampled goal location in the last room, and is rewarded only when it reaches the correct cell.

Original: Goyal et al.
Revised: Goyal et al.

Original: (2019) test the efﬁcacy of different[Table] exploration objectives 5 and show that this is a hard setting where efﬁcient exploration is necessary.
Revised: (2019) test the efﬁcacy of different exploration objectives 5 and show that this is a hard setting where efﬁcient exploration is necessary.

Original: They show that InfoBot outperforms several state-of-the art exploration methods in this environment.
Revised: They show that InfoBot outperforms several state-of-the art exploration methods in this environment.

Original: Concretely, we 1) train DS-VIC to identify decision states (Equation 2) on MultiRoomN2S6 and transfer to a goal-driven task on MultiRoomN3S and MultiRoomN5S4 (similar to Goyal et al.
Revised: Concretely, we 1) train DS-VIC to identify decision states (Equation 2) on MultiRoomN2S6 and transfer to a goal-driven task on MultiRoomN3S and MultiRoomN5S4 (similar to Goyal et al.

Original: (2019)), and 2) train on MultiRoomN2S10 and transfer to
Revised: (2019)), and 2) train on MultiRoomN2S10 and transfer to

Original: MultiRoomN6S25 , which is a more challenging transfer task i.e .
Revised: MultiRoomN6S25 , which is a more challenging transfer task i.e .

Original: it has larger rooms ( 25x25 ) making efﬁcient exploration critical to ﬁnd doors quickly.
Revised: it has larger rooms ( 25x25 ) making efﬁcient exploration critical to ﬁnd doors quickly.

Revised: For DSVIC and DIAYN, we train for 6 million episodes in the decision state identiﬁcation phase (pre-training) and pick the checkpoints with highest empowerment values across training.

Original: Overall Trends.
Revised: Overall Trends.

Original: Table.
Revised: Table.

Original: 1 reports success rate – the % of times the agent reaches the goal on validation.
Revised: 1 reports success rate – the % of times the agent reaches the goal on validation.

Original: First, our implementation of InfoBot is competitive with Goyal et al.
Revised: First, our implementation of InfoBot is competitive with Goyal et al.

Original: For the MultiRoomN2S to N5S4 transfer (middle column), methods which identify decision states outperform count-based exploration.
Revised: For the MultiRoomN2S6 to N5S4 transfer (middle column), methods which identify decision states outperform count-based exploration.

Original: Interestingly, in this setup, a randomly initialized encoder is competitive with both InfoBot and DS-VIC, and serves as an important baseline.
Revised: Interestingly, in this setup, a randomly initialized encoder is competitive with both InfoBot and DS-VIC, and serves as an important baseline.

Original: In MultiRoomN2S10 to N6S25 transfer, which has a large state space, we ﬁnd that DS-VIC (at β = 10 − 2 ) achieves the best performance, followed by count-based exploration and InfoBot (that are both within standard-error of each other).
Revised: In MultiRoomN2S to N6S25 transfer, which has a large state space, we ﬁnd that DS-VIC (at β = 10 − 2 ) achieves the best performance, followed by count-based exploration and InfoBot (that are both within standard-error of each other).

Original: This shows that decision states identiﬁed by DS-VIC effectively guide exploration in sparse-reward settings.
Revised: This shows that decision states identiﬁed by DS-VIC effectively guide exploration in sparse-reward settings.

Original: More importantly, our model beats a strong heuristic baseline which treats every corner and doorway as a decision state (more details on this are in the appendix).
Revised: More importantly, our model beats a strong heuristic baseline which treats every corner and doorway as a decision state (more details on this are in the appendix).

Original: Finally, the randomly initialized encoder as well as DIAYN generalize much worse in this transfer task.
Revised: Finally, the randomly initialized encoder as well as DIAYN generalize much worse in this transfer task.

Original: β sensitivity.
Revised: β sensitivity.

Original: We sweep over β in log-scale from { 10 − 1 , · · · , 10 − 6 } , as shown in Figure 5 (except β = 10 − 1 which does not converge to ≥ 0 empowerment).
Revised: We sweep over β in log-scale from { 10 − 1 , · · · , 10 − 6 } , as shown in Figure 5 (except β = 10 − 1 which does not converge to > 0 empowerment) and also report β = 0 which recovers a no information regularization baseline.

Original: We ﬁnd that 10 − 2 works best, followed by 10 − 5 and 10 − 3 .
Revised: We ﬁnd that 10 − 2 works best, followed by 10 − 5 and 10 − 3 .

Original: In general, performance is robust, i.e . across β , we  reach ≈ 90% success 7 .
Revised: In general, performance is robust, i.e . across β , we seem to reach ≈ 90% success for most values.

Original: [Figure]
Revised: [Figure]

Original: More results on N6S25.
Revised: More results on N6S25.

Original: To better understand Equation 8 compared to other exploration incentives, in addition to success rate and sample-efﬁciency (Figure 6), we also consider another metric where we count the number of unique rooms reached (other than the starting room 0 ) as training progresses (Figure 6 (c)).
Revised: To better understand Equation 8 compared to other exploration incentives, in addition to success rate and sample-efﬁciency (Figure 6), we also consider another metric where we count the number of unique rooms reached (other than the starting room 0 ) as training progresses (Figure 6 (c)).

Original: Across metrics, DS-VIC performs the best on the challenging N6S25 transfer.
Revised: Across metrics, DS-VIC performs the best on the challenging N6S25 transfer.

Original: Furthermore, the unique room metric reveals that DIAYN gets stuck after exploring the ﬁrst room.
Revised: Furthermore, the unique room metric reveals that DIAYN gets stuck after exploring the ﬁrst room.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Intrinsic Control and Intrinsic Motivation.
Revised: Intrinsic Control and Intrinsic Motivation.

Original: Learning how to explore without extrinsic rewards is a foundational problem in Reinforcement Learning (Machado et al., 2017; Pathak et al., 2017; Gregor et al., 2016; Strehl & Littman, 2008; Schmidhuber, 1990).
Revised: Learning how to explore without extrinsic rewards is a foundational problem in Reinforcement Learning (Machado et al., 2017; Pathak et al., 2017; Gregor et al., 2016; Strehl & Littman, 2008; Schmidhuber, 1990).

Original: Typical curiosity-driven approaches attempt to visit states that maximize the surprise of an agent (Pathak et al., 2017) or improvement in predictions from a dynamics model (Stadie et al., 2015; Lopes et al., 2012).
Revised: Typical curiosity-driven approaches attempt to visit states that maximize the surprise of an agent (Pathak et al., 2017) or improvement in predictions from a dynamics model (Stadie et al., 2015; Lopes et al., 2012).

Original: While curiosity-driven approaches seek out and explore novel states, they typically do not measure how reliably the agent can reach them.
Revised: While curiosity-driven approaches seek out and explore novel states, they typically do not measure how reliably the agent can reach them.

Original: In contrast, approaches for intrinsic control (Eysenbach et al., 2018; Achiam et al., 2018; Gregor et al., 2016) explore novel states while ensuring those states are reliably reachable.
Revised: In contrast, approaches for intrinsic control (Eysenbach et al., 2018; Achiam et al., 2018; Gregor et al., 2016) explore novel states while ensuring those states are reliably reachable.

Original: Gregor et al.
Revised: Gregor et al.

Original: (2016) maximize the number of ﬁnal states that can be reliably reached by the policy, while Eysenbach et al.
Revised: (2016) maximize the number of ﬁnal states that can be reliably reached by the policy, while Eysenbach et al.

Original: (2018) distinguish an option at every state along the trajectory, and Achiam et al.
Revised: (2018) distinguish an option at every state along the trajectory, and Achiam et al.

Original: (2018) learn options for entire trajectories by encoding the sequence of states at regular intervals.
Revised: (2018) learn options for entire trajectories by encoding the sequence of states at regular intervals.

Original: Since decision states are more related to reliably acting in an environment rather than just visiting novel states (without an estimate of reachability), we formulate our regularizer in the intrinsic control framework, speciﬁcally building on the work of Gregor et al.
Revised: Since decision states are more related to reliably acting in an environment rather than just visiting novel states (without an estimate of reachability), we formulate our regularizer in the intrinsic control framework, speciﬁcally building on the work of Gregor et al.

Original: Default Behavior and Decision States.
Revised: Default Behavior and Decision States.

Original: Recent work in policy compression has focused on learning a default policy when training on a family of tasks, to be able to re-use behavior across tasks.
Revised: Recent work in policy compression has focused on learning a default policy when training on a family of tasks, to be able to re-use behavior across tasks.

Original: In (Galashov et al., 2018; Teh et al., 2017), default behavior is learnt using a set of task-speciﬁc policies which then regularizes each policy, while Goyal et al.
Revised: In (Galashov et al., 2018; Teh et al., 2017), default behavior is learnt using a set of task-speciﬁc policies which then regularizes each policy, while Goyal et al.

Original: (2019) learn a default policy using an information bottleneck on task information and a latent variable the policy conditions on.
Revised: (2019) learn a default policy using an information bottleneck on task information and a latent variable the policy conditions on.

Original: We devise a similar information regularization objective that learns default behavior shared by all intrinsic options without external rewards so as to reduce learning pressure on option-conditioned policies.
Revised: We devise a similar information regularization objective that learns default behavior shared by all intrinsic options without external rewards so as to reduce learning pressure on option-conditioned policies.

Original: Bottleneck states in MDPs.
Revised: Bottleneck states in MDPs.

Original: There is rich literature on identiﬁcation of bottleneck states in MDPs.
Revised: There is rich literature on identiﬁcation of bottleneck states in MDPs.

Original: The core idea is to either identify all the states that are common to multiple goals in an environment (McGovern & Barto, 2001) or use a diffusion model built using an MDP’s transition matrix (Machado et al., 2017; ¸Sim¸sek & Barto, 2004; Theocharous & Mahadevan, 2002).
Revised: The core idea is to either identify all the states that are common to multiple goals in an environment (McGovern & Barto, 2001) or use a diffusion model built using an MDP’s transition matrix (Machado et al., 2017; ¸Sim¸sek & Barto, 2004; Theocharous & Mahadevan, 2002).

Original: The key distinction between bottleneck states and decision states is that the latter are more closely tied to the information available to the agent and what it can act upon, whereas bottleneck states are more tied to the connectivity structure of an MDP and intrinsic to the environment, representing states which when visited allow access to a novel set of states (Goyal et al., 2019).
Revised: The key distinction between bottleneck states and decision states is that the latter are more closely tied to the information available to the agent and what it can act upon, whereas bottleneck states are more tied to the connectivity structure of an MDP and intrinsic to the environment, representing states which when visited allow access to a novel set of states (Goyal et al., 2019).

Original: Concretely, while a corner in a room need not be a bottleneck state, since visiting a corner does not ‘open up’ new states (the way a doorway would), it is still a useful state for a goal-driven agent with partial observation to visit (since it is a distinct landmark in the state space where meaningful decisions can be made).
Revised: Concretely, while a corner in a room need not be a bottleneck state, since visiting a corner does not ‘open up’ new states (the way a doorway would), it is still a useful state for a goal-driven agent with partial observation to visit (since it is a distinct landmark in the state space where meaningful decisions can be made).

Original: Information Bottleneck in Machine Learning.
Revised: Information Bottleneck in Machine Learning.

Original: Since the foundational work of Tishby et al.
Revised: Since the foundational work of Tishby et al.

Original: Chechik et al.
Revised: Chechik et al.

Original: (2005), there has been a lot of interest in making use of ideas from information bottleneck (IB) for various tasks such as clustering (Strouse & Schwab, 2017; Still et al., 2004), sparse coding (Chalk et al., 2016), classiﬁcation using deep learning (Alemi et al., 2016; Achille & Soatto, 2016), cognitive science and language (Zaslavsky et al., 2018) and reinforcement learning (Goyal et al., 2019; Galashov et al., 2018; Strouse et al., 2018).
Revised: (2005), there has been a lot of interest in making use of ideas from information bottleneck (IB) for various tasks such as clustering (Strouse & Schwab, 2017; Still et al., 2004), sparse coding (Chalk et al., 2016), classiﬁcation using deep learning (Alemi et al., 2016; Achille & Soatto, 2016), cognitive science and language (Zaslavsky et al., 2018) and reinforcement learning (Goyal et al., 2019; Galashov et al., 2018; Strouse et al., 2018).

Original: We apply an information regularizer to an
Revised: We apply an information regularizer to an

Original: RL agent that must learn without extrinsic rewards to identify decision states.
Revised: RL agent that must learn without extrinsic rewards to identify decision states.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We devise a principled approach to identify decision states in an environment without any extrinsic reward supervision using a sandwich bound on the empowerment of Gregor et al.
Revised: We devise a principled approach to identify decision states in an environment without any extrinsic reward supervision using a sandwich bound on the empowerment of Gregor et al.

Original: Our approach yields decision states that align with human intuition across environments, and aid directed exploration on external-reward tasks and subsequently lead to better success rate and sample complexity in novel environments (competitive to supervised decision states of Goyal et al.
Revised: Our approach yields decision states that align with human intuition across environments, and aid directed exploration on external-reward tasks and subsequently lead to better success rate and sample complexity in novel environments (competitive to supervised decision states of Goyal et al.

Original: All our code and environments will be made publicly available.
Revised: All our code and environments will be made publicly available

Original: A A PPENDIX A. A LGORITHM
Revised: A A PPENDIX A. A LGORITHM

Original: We present the algorithm summarizing our proposed approach DS-VIC– (1) unsupervised discovery of decision states and (2) transfer to goal-driven tasks.
Revised: We present the algorithm summarizing our proposed approach DS-VIC– (1) unsupervised discovery of decision states and (2) transfer to goal-driven tasks.

Original: Algorithm 1 DS-VIC
Revised: Algorithm 1 DS-VIC

Original: Require:
Revised: Require:

Original: A parameterized encoder p φ ( z t | ω, s t ) , policy π ( a t | ω, s t ) Require:
Revised: A parameterized encoder p φ ( z t | ω, s t ) , policy π ( a t | ω, s t ) Require:

Original: A parameterized option inference network q ν ( ω | s 0 , s f ) Require:
Revised: A parameterized option inference network q ν ( ω | s 0 , s f ) Require:

Original: A parameterized goal-conditioned policy π γ ( a t | s t , g )
Revised: A parameterized goal-conditioned policy π γ ( a t | s t , g )

Original: Require:
Revised: Require:

Original: A prior on discrete options p ( ω )  Require:
Revised: A prior on discrete options p ( ω ) and integer H - the length of each option trajectory. Require:

Original: A variational approximation of the option-marginalized encoder q ( z t ) Require:
Revised: A variational approximation of the option-marginalized encoder q ( z t ) Require:

Original: A regularization weight β and max-ent coefﬁcient α Require:
Revised: A regularization weight β and max-ent coefﬁcient α Require:

Original: A set of training environments p train ( E ) and transfer environments p transfer ( E ) Unsupervised Discovery
Revised: A set of training environments p train ( E ) and transfer environments p transfer ( E ) Unsupervised Discovery

Original: Sample training environment E train ∼ p train ( E ) for episodes = 1 to max − episodes do
Revised: Sample training environment E train ∼ p train ( E ) for episodes = 1 to max − episodes do

Original: Sample a spawn location S 0 ∼ p ( s 0 | E train ) and an option Ω ∼ p ( ω ) Unroll a state-action trajectory τ under π θ ( a t | s t , z t )  with reparametrized Z t ∼ p φ ( z t | s t , ω )
Revised: Sample a spawn location S 0 ∼ p ( s 0 | E train ) and an option Ω ∼ p ( ω ) Unroll a state-action trajectory τ under π θ ( a t | s t , z t ) for H steps with reparametrized Z t ∼ p φ ( z t | s t , ω )

Original: Infer Ω from q ν ( ω | s o , s f ) Update the parameters θ, ν and φ based on Eqn.end for Transfer to Goal-Driven Tasks
Revised: Infer Ω from q ν ( ω | s o , s f ) Update the parameters θ, ν and φ based on Eqn.end for Transfer to Goal-Driven Tasks

Original: Sample transfer environment E transfer ∼ p transfer ( E ) for episodes = 1 to max − episodes do
Revised: Sample transfer environment E transfer ∼ p transfer ( E ) for episodes = 1 to max − episodes do

Original: Sample a goal G ∼ p ( g | E transfer ) Unroll a state-action trajectory under the goal-conditioned policy π γ ( a t | s t , g )
Revised: Sample a goal G ∼ p ( g | E transfer ) Unroll a state-action trajectory under the goal-conditioned policy π γ ( a t | s t , g )

Original: Update policy parameters γ to maximize the reward given by Eqn.
Revised: Update policy parameters γ to maximize the reward given by Eqn.

Original: 8 end for
Revised: 8 end for

Original: A.
Revised: A.

Original: L EMMA P ROOF
Revised: L EMMA P ROOF

Original: We state a proof of the Lemma 2.1 stating that our proposed regularizer (cid:80) ft =

Revised: We state a proof of the Lemma 2.1 stating that our proposed regularizer (cid:80) ft =

Original: I
Revised: I

Original: (Ω , A t | S t , S 0 ) forms an bound on empowerment
Revised: (Ω , A t | S t , S 0 ) forms an bound on empowerment

Original: I (Ω , S f | S 0 ) from the main paper.
Revised: I (Ω , S f | S 0 ) from the main paper.

Original: This, combined with the lower-bound presented in VIC (Gregor et al., 2016), forms a sandwich bound on I (Ω , S f | S 0 ) .
Revised: This, combined with the lower-bound presented in VIC (Gregor et al., 2016), forms a sandwich bound on I (Ω , S f | S 0 ) .

Original: Lemma 2.1 Let A t be the action random variable at timestep t and state S t following an optionconditioned policy π ( a t | s t , ω ) .
Revised: Lemma 2.1 Let A t be the action random variable at timestep t and state S t following an optionconditioned policy π ( a t | s t , ω ) .

Original: Then, I (Ω , A t | S t , S 0 ) i.e .
Revised: Then, I (Ω , A t | S t , S 0 ) i.e .

Original: the conditional mutual information be- tween the option Ω and action A t when summed over all timesteps in the trajectory, upper bounds the conditional mutual information I (Ω , S f | S 0 ) between Ω and the ﬁnal state S f – namely the empowerment as deﬁned by Gregor et al.
Revised: the conditional mutual information be- tween the option Ω and action A t when summed over all timesteps in the trajectory, upper bounds the conditional mutual information I (Ω , S f | S 0 ) between Ω and the ﬁnal state S f – namely the empowerment as deﬁned by Gregor et al.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: To begin, observe that the graphical model presented in Fig.
Revised: To begin, observe that the graphical model presented in Fig.

Original: 2 satisﬁes the markov chain Ω ↔ { s 1 , a 1 } f − 1 t =1 ↔ S f
Revised: 2 satisﬁes the markov chain Ω ↔ { s 1 , a 1 } f − 1 t =1 ↔ S f

Original: (assuming every node is conditioned on the intitial state S 0 ).
Revised: (assuming every node is conditioned on the intitial state S 0 ).

Original: Therefore, the data-processing inequality (DPI) Cover & Thomas (1991) implies: 
Revised: Therefore, the data-processing inequality (DPI) Cover & Thomas (1991) implies: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Furthermore, using the chain rule of mutual information Cover & Thomas (1991), we can write:
Revised: Furthermore, using the chain rule of mutual information Cover & Thomas (1991), we can write:

Original: [Equation]
Revised: [Equation]

Original: Repeating the same set of steps recursively gives us:
Revised: Repeating the same set of steps recursively gives us:

Original: [Equation]
Revised: [Equation]

Original: Chain rule of MI
Revised: Chain rule of MI

Original: [Equation]
Revised: [Equation]

Original: Note that the graphical model presented in Fig.
Revised: Note that the graphical model presented in Fig.

Original: 2 implies that Ω ⊥⊥ S 0 and hence,
Revised: 2 implies that Ω ⊥⊥ S 0 and hence,

Original: [Equation]
Revised: [Equation]

Original: A. U PPER BOUND ON I ( A t , Ω | S t , S 0 )
Revised: A. U PPER BOUND ON I ( A t , Ω | S t , S 0 )

Original: We explain the steps to derive Eqn.
Revised: We explain the steps to derive Eqn.

Original: (4) in the main paper, as an upper bound on I ( A t , Ω | S t , S 0 ) .
Revised: (4) in the main paper, as an upper bound on I ( A t , Ω | S t , S 0 ) .

Original: By the data processing inequality (Cover & Thomas, 1991)
Revised: By the data processing inequality (Cover & Thomas, 1991)

Original: I ( A t , Ω | S t , S 0 ) ≤ I ( Z t , Ω | S t , S 0 ) for the graphical model in Fig 2.
Revised: I ( A t , Ω | S t , S 0 ) ≤ I ( Z t , Ω | S t , S 0 ) for the graphical model in Fig 2.

Original: So we will next derive an upper bound on I ( Z t , Ω | S t , S 0 ) .
Revised: So we will next derive an upper bound on I ( Z t , Ω | S t , S 0 ) .

Original: We write I (Ω , Z t | S t , S 

Revised: We write I (Ω , Z t | S t , S

Original: = s ) , given a start state S 

Revised: = s ) , given a start state S

Original: = s as:
Revised: = s as:

Original: [Equation]
Revised: [Equation]

Original: The key difference here is that our objective here uses the options Ω internal to the agent as opposed to Goyal et al.
Revised: The key difference here is that our objective here uses the options Ω internal to the agent as opposed to Goyal et al.

Original: (2019), who use external goal speciﬁcations g provided to the agent.
Revised: (2019), who use external goal speciﬁcations g provided to the agent.

Original: Similar to VIC, p J here denotes the (unkown) state distribution at time t from which we can draw samples when we execute a policy.
Revised: Similar to VIC, p J here denotes the (unkown) state distribution at time t from which we can draw samples when we execute a policy.

Original: We then assume a variational approximation q ( z t ) 8 for p ( z t | S t ) , and using the fact that
Revised: We then assume a variational approximation q ( z t ) 7 for p ( z t | S t ) , and using the fact that

Original: D KL [ p ( z t | s t )
Revised: D KL [ p ( z t | s t )

Original: || q ( z t )] ≥ 0 , we get a the following lower bound:
Revised: || q ( z t )] ≥ 0 , we get a the following lower bound:

Original: [Equation]
Revised: [Equation]

Original: A. D ECISION S TATE I DENTIFICATION – O N POLICY WITH O PTIONS .
Revised: A. D ECISION S TATE D ISCOVERY – O N POLICY WITH O PTIONS .

Original: We use Eqn. 16 to identify and visualize the decision states learned in an environment, augmented with random sampling of the start state S 0 .
Revised: We use Eqn. 16 to discover and visualize the decision states learned in an environment, augmented with random sampling of the start state S 0 .

Original: Thus, we compute our decision states in an on-policy manner.
Revised: Thus, we compute our decision states in an on-policy manner.

Original: Mathematically, we can write this as:
Revised: Mathematically, we can write this as:

Original: [Equation]
Revised: [Equation]

Original: where S 0 is a random spawn location uniformly chosen from the set of states in the environment and Ω is a random option chosen at each of the spawn locations.
Revised: where S 0 is a random spawn location uniformly chosen from the set of states in the environment and Ω is a random option chosen at each of the spawn locations.

Original: Thus, for each state S t in the environment, we look at the aggregate of all the trajectories that pass through it and compute the values of log p ( z t | ω,s t ) q ( z t ) to identify / visualize decision states.
Revised: Thus, for each state S t in the environment, we look at the aggregate of all the trajectories that pass through it and compute the values of log p ( z t | ω,s t ) q ( z t ) to identify / visualize decision states.

Original: In addition to being principled, this methodology also precisely captures our intuition that it is possible to identify decision states which are common to, or frequented across multiple options.
Revised: In addition to being principled, this methodology also precisely captures our intuition that it is possible to identify decision states which are common to, or frequented across multiple options.

Original: Our results section 4 show that the identiﬁed decision states match our assessments of decision states corresponding to some structural regularities in the environment.
Revised: Our results section 4 show that the identiﬁed decision states match our assessments of decision states corresponding to some structural regularities in the environment.

Original: A. I DENTIFICATION OF D ECISION S TATES DURING T RANSFER
Revised: A. I DENTIFICATION OF D ECISION S TATES DURING T RANSFER

Original: As mentioned in the main paper, we would like to compute I (Ω , Z t | S t , S 0 ) to identify decision states and given a (potentially) novel environment and a novel goal-conditioned task, to provide it an exploration bonus.
Revised: As mentioned in the main paper, we would like to compute I (Ω , Z t | S t , S 0 ) to identify decision states and given a (potentially) novel environment and a novel goal-conditioned task, to provide it an exploration bonus.

Original: Given a state s (cid:48) , that we would like to compute I (Ω , Z t | S t = s (cid:48) , S 0 ) , we can write:
Revised: Given a state s (cid:48) , that we would like to compute I (Ω , Z t | S t = s (cid:48) , S 0 ) , we can write:

Original: [Equation]
Revised: [Equation]

Original: However, this cannot be computed in this form in a transfer task, since a goal driven agent is not following on-policy actions for an option Ω that would allow us to draw samples from p J ( ·| S 0 , Ω) above (in order to do a monte-carlo estimate of the integral above).
Revised: However, this cannot be computed in this form in a transfer task, since a goal driven agent is not following on-policy actions for an option Ω that would allow us to draw samples from p J ( ·| S 0 , Ω) above (in order to do a monte-carlo estimate of the integral above).

Original: Thus instead, we propose to compute the mutual information as follows:
Revised: Thus instead, we propose to compute the mutual information as follows:

Original: [Equation]
Revised: [Equation]

Original: Now, given a state S t = s (cid:48) , this requires us to draw samples from p ( s 0 , ω | S t = s (cid:48) ) , which in general is intractable (since this requires us to know p J ( s t | Ω) , which is not available in closed form).
Revised: Now, given a state S t = s (cid:48) , this requires us to draw samples from p ( s 0 , ω | S t = s (cid:48) ) , which in general is intractable (since this requires us to know p J ( s t | Ω) , which is not available in closed form).

Original: In order to compute the above equation, we make the assumption that p ( s 0 , ω | S t = s (cid:48) )
Revised: In order to compute the above equation, we make the assumption that p ( s 0 , ω | S t = s (cid:48) )

Original: =
Revised: =

Original: p ( s 0 ) p ( ω ) .
Revised: p ( s 0 ) p ( ω ) .

Original: Breaking it down, this makes two assumptions: ﬁrstly, p ( ω | S t = s (cid:48) , s 0 )
Revised: Breaking it down, this makes two assumptions: ﬁrstly, p ( ω | S t = s (cid:48) , s 0 )

Original: = p ( ω | s 0 ) .
Revised: = p ( ω | s 0 ) .

Original: This means that all options have equal probability of passing through a state s (cid:48) at time t , which is not true in general.
Revised: This means that all options have equal probability of passing through a state s (cid:48) at time t , which is not true in general.

Original: The second assumption is the same as VIC, namely that p ( ω | s 0 )
Revised: The second assumption is the same as VIC, namely that p ( ω | s 0 )

Original: =
Revised: =

Original: p ( ω ) .
Revised: p ( ω ) .

Original: Instead of making this assumption, one could also ﬁt a parameterized variational approximation to p ( ω | S t = s (cid:48) , s 0 ) and train it in conjunction with VIC.
Revised: Instead of making this assumption, one could also ﬁt a parameterized variational approximation to p ( ω | S t = s (cid:48) , s 0 ) and train it in conjunction with VIC.

Original: However, we found our simple scheme to work well in practice, and thus avoid ﬁtting an additional variational approximation.
Revised: However, we found our simple scheme to work well in practice, and thus avoid ﬁtting an additional variational approximation.

Original: A. D ECISION STATES FROM LOCAL OBSERVATIONS
Revised: A. D ECISION STATES FROM LOCAL OBSERVATIONS

Original: In the main paper, our results assumed that one had access to a good SLAM routine that did mapping and localization from partial observations { u 0 , · · · , u t } .
Revised: In the main paper, our results assumed that one had access to a good SLAM routine that did mapping and localization from partial observations { u 0 , · · · , u t } .

Original: In general, as we discuss in the main paper, this is not a bad assumption since our model is always trained on a single environment, and thus it is reasonable for us to expect it to have a sense of the (x, y) coordinates in its internal representation / weights as training progresses.
Revised: In general, as we discuss in the main paper, this is not a bad assumption since our model is always trained on a single environment, and thus it is reasonable for us to expect it to have a sense of the (x, y) coordinates in its internal representation / weights as training progresses.

Original: In this section, we detail some of the pathologies that we observed when kicking off training, if we took a naive approach to SLAM, estimating a function f ( { u 0 , · · · , u t } ) to directly regress to Ω from partial observations (Algorithm 1).
Revised: In this section, we detail some of the pathologies that we observed when kicking off training, if we took a naive approach to SLAM, estimating a function f ( { u 0 , · · · , u t } ) to directly regress to Ω from partial observations (Algorithm 1).

Original: We note that since our particular choice of environment is a discrete gridworld, several partial observations (especially for small agent view sizes like the 3x3 window) look the same to the agent and the agent has a tendency to converge to the most trivial optima for learning options – one which learns to end at an easily obtainable partial observation given an option.
Revised: We note that since our particular choice of environment is a discrete gridworld, several partial observations (especially for small agent view sizes like the 3x3 window) look the same to the agent and the agent has a tendency to converge to the most trivial optima for learning options – one which learns to end at an easily obtainable partial observation given an option.

Original: For instance, while training DS-VIC, one particular policy simply learned 4 options corresponding to the four cardinal directions the agent can face, and each can be achieved by left or right turns, without actually requiring any movement by the agent.
Revised: For instance, while training DS-VIC, one particular policy simply learned 4 options corresponding to the four cardinal directions the agent can face, and each can be achieved by left or right turns, without actually requiring any movement by the agent.

Original: Since the agent has a compass which tells it the direction it is facing, it can ignore the partial observation and just use the direction vector to predict which of the 4 options was sampled.
Revised: Since the agent has a compass which tells it the direction it is facing, it can ignore the partial observation and just use the direction vector to predict which of the 4 options was sampled.

Original: A. B ASELINES
Revised: A. B ASELINES

Original: We use an exploration bonus coefﬁcient of κ = 0 .
Revised: We use an exploration bonus coefﬁcient of κ = 0 .

Original: 1 for the count-based exploration bonus baseline.
Revised: 1 for the count-based exploration bonus baseline.

Original: The heuristic baseline identiﬁes all occurrences of the following types of states in the MultiRoom environments – (1) corners of the room, (2) doorways, and gives them a slightly higher coefﬁcient of exploration bonus than the κ used for the count-based bonus.
Revised: The heuristic baseline identiﬁes all occurrences of the following types of states in the MultiRoom environments – (1) corners of the room, (2) doorways, and gives them a slightly higher coefﬁcient of exploration bonus than the κ used for the count-based bonus.

Original: We ran a sweep for the values of this higher coefﬁcient and found that 0 .
Revised: We ran a sweep for the values of this higher coefﬁcient and found that 0 .

Original: 105 (i.e. a +5% increase) gave best results.
Revised: 105 (i.e. a +5% increase) gave best results.

Original: A. I MPLEMENTATION D ETAILS
Revised: A. I MPLEMENTATION D ETAILS

Original: Network Architecture: We use a 3 layered convolutional neural network with kernels of size 3x3, 2x2 and 2x2 in the 3 layers respectively to process the agent’s egocentric observation.
Revised: Network Architecture: We use a 3 layered convolutional neural network with kernels of size 3x3, 2x2 and 2x2 in the 3 layers respectively to process the agent’s egocentric observation.

Original: We use
Revised: We use

Original: ReLU as the non-linear activation function after each convolutional layer.
Revised: ReLU as the non-linear activation function after each convolutional layer.

Original: The output of the CNN is then concatenated with the agent’s direction vector (compass) and the option (or goal encoding).
Revised: The output of the CNN is then concatenated with the agent’s direction vector (compass) and the option (or goal encoding).

Original: The concatenated features are then passed through a linear layer with hidden size 64 to produce the ﬁnal features used by option encoder p φ ( z t | s t , ω ) head and the policy head π θ ( a t
Revised: The concatenated features are then passed through a linear layer with hidden size 64 to produce the ﬁnal features used by option encoder p φ ( z t | s t , ω ) head and the policy head π θ ( a t

Original: | s t , z t ) .
Revised: | s t , z t ) .

Original: We use the (x, y) coordinates of the ﬁnal state (embedded through a linear layer of hidden size 64) to regress to the option via q ν ( ω | s 0 , s f ) .
Revised: We use the (x, y) coordinates of the ﬁnal state (embedded through a linear layer of hidden size 64) to regress to the option via q ν ( ω | s 0 , s f ) .

Original: Furthermore, our parameterized policy is a reactive one and the encoder p φ ( z t | s t , ω ) is recurrent over the sequence of states encountered in the episode.
Revised: Furthermore, our parameterized policy is a reactive one and the encoder p φ ( z t | s t , ω ) is recurrent over the sequence of states encountered in the episode.

Original: The bottleneck random variable Z t is sampled from the parameterized gaussian p φ ( z t | s t , ω ) and is made a differentiable stochastic node using the re-parmaterization trick for gaussian random variables.
Revised: The bottleneck random variable Z t is sampled from the parameterized gaussian p φ ( z t | s t , ω ) and is made a differentiable stochastic node using the re-parmaterization trick for gaussian random variables.

Original: Training Details: We use Advantage Actor-Critic (A2C) (open-sourced implementation by Kostrikov (2018)) for all our experiments.
Revised: Training Details: We use Advantage Actor-Critic (A2C) (open-sourced implementation by Kostrikov (2018)) for all our experiments.

Original: We use RMSprop as the optimizer for all our experiments.
Revised: We use RMSprop as the optimizer for all our experiments.

Original: For the partially-observable grid world settings, the agent receives an egocentric view of it’s surroundings as input, encoded as an occupancy grid where the channel dimension speciﬁes whether the agent or an obstacle is present at an ( x, y ) location.
Revised: For the partially-observable grid world settings, the agent receives an egocentric view of it’s surroundings as input, encoded as an occupancy grid where the channel dimension speciﬁes whether the agent or an obstacle is present at an ( x, y ) location.

Original: We set the coefﬁcient α in Eqn.
Revised: We set the coefﬁcient α in Eqn.

Original: 6 to be 10 − 3 for all our experiments based on sweeps conducted across multiple values.
Revised: 6 to be 10 − 3 for all our experiments based on sweeps conducted across multiple values.

Original: In practice, we found it to be difﬁcult to optimize our unsupervised objective in absence of an entropy bonus – the parameterized policy collapses to a deterministic one and no options are learned in addition to inefﬁcient exploration of the state-space.
Revised: In practice, we found it to be difﬁcult to optimize our unsupervised objective in absence of an entropy bonus – the parameterized policy collapses to a deterministic one and no options are learned in addition to inefﬁcient exploration of the state-space.

Original: Moreover, we found that it was hard to obtain reasonable results by optimizing for both the terms in the objective from scratch and therefore, we optimize the intrinsic objective itself for ∼ 8 k episodes (i.e., we set β = 0 ) after which we turn on the regularization term and let β grow linearly for another ∼ 8 k episodes to get feasible outcomes at convergence.
Revised: Moreover, we found that it was hard to obtain reasonable results by optimizing for both the terms in the objective from scratch and therefore, we optimize the intrinsic objective itself for ∼ 8 k episodes (i.e., we set β = 0 ) after which we turn on the regularization term and let β grow linearly for another ∼ 8 k episodes to get feasible outcomes at convergence.

Original: We experiment with a vocabulary of 2, 4 and 32 options (imitating undercomplete and overcomplete settings) for all the environments.
Revised: We experiment with a vocabulary of 2, 4 and 32 options (imitating undercomplete and overcomplete settings) for all the environments.

Original: For all the exploration incentives presented in Table 1, we ﬁrst picked a value of κ in Eqn.
Revised: For all the exploration incentives presented in Table 1, we ﬁrst picked a value of κ in Eqn.

Original: 7 (decides how to weigh the bonus with respect to the reward) from { 10 , 1 , 0 . 1 , 0 . 01 , 0 . 001 , 0 . 0001 } based on the best sample-complexity on goal-driven transfer.
Revised: 7 (decides how to weigh the bonus with respect to the reward) from { 10 , 1 , 0 . 1 , 0 . 01 , 0 . 001 , 0 . 0001 } based on the best sample-complexity on goal-driven transfer.

Original: InfoBot Implementation: Since code to reproduce InfoBot (Goyal et al., 2019) was not publiclt available, we implemented InfoBot ourselves while making sure that we are consistent with the architectural and hyper-parameter choices adopted by InfoBot (as per the A2C implementation by (Chevalier-Boisvert et al., 2018)).
Revised: InfoBot Implementation: Since code to reproduce InfoBot (Goyal et al., 2019) was not publiclt available, we implemented InfoBot ourselves while making sure that we are consistent with the architectural and hyper-parameter choices adopted by InfoBot (as per the A2C implementation by (Chevalier-Boisvert et al., 2018)).

Original: (i)
Revised: (i)

Original: We replace the convolution layers with fully connected layers to process the observations, and (ii) we use the layer sizes as mentioned by (Goyal et al., 2019) in the appendix.
Revised: We replace the convolution layers with fully connected layers to process the observations, and (ii) we use the layer sizes as mentioned by (Goyal et al., 2019) in the appendix.

Original: The checkpoint used to report the ﬁnal performance for N3S4 and N5S4 were picked by doing validation on the transfer task based on the success metric.
Revised: The checkpoint used to report the ﬁnal performance for N3S4 and N5S4 were picked by doing validation on the transfer task based on the success metric.

Original: However, this procedure was infeasible for the bigger N6S25 environment and we chose the last checkpoint from the training for this environment.
Revised: However, this procedure was infeasible for the bigger N6S25 environment and we chose the last checkpoint from the training for this environment.

Original: Convergence Criterion.
Revised: Convergence Criterion.

Original: In practice, we observe that it is hard to learn a large number of discrete options using the unsupervised objective.
Revised: In practice, we observe that it is hard to learn a large number of discrete options using the unsupervised objective.

Original: From the entire option vocabulary, the agent only learns a few number of options discriminatively (as identiﬁed by the option termination state), with the rest collapsing into the same set of ﬁnal states.
Revised: From the entire option vocabulary, the agent only learns a few number of options discriminatively (as identiﬁed by the option termination state), with the rest collapsing into the same set of ﬁnal states.

Original: To pick a suitable checkpoint for the transfer experiments, we pick the ones which have learned the maximum number of options reliably – as measured by the likelihood of the correct option from the ﬁnal state log q ν ( ω | s 0 , s f ) .
Revised: To pick a suitable checkpoint for the transfer experiments, we pick the ones which have learned the maximum number of options reliably – as measured by the likelihood of the correct option from the ﬁnal state log q ν ( ω | s 0 , s f ) .

Original: Option Curriculum It is standard to have a ﬁxed-sized discrete option space Ω with a uniform pior (Gregor et al., 2016).
Revised: Option Curriculum It is standard to have a ﬁxed-sized discrete option space Ω with a uniform pior (Gregor et al., 2016).

Original: However, learning a meaningful option space with larger option vocabulary size | Ω | = K has been reported to be difﬁcult Achiam et al.
Revised: However, learning a meaningful option space with larger option vocabulary size | Ω | = K has been reported to be difﬁcult Achiam et al.

Original: We adopt a curriculum based approach proposed in Achiam et al.
Revised: We adopt a curriculum based approach proposed in Achiam et al.

Original: (2018) where the option vocabulary size is gradually increased as the option decoder q ν ( ω | s f ) becomes more conﬁdent in mapping back the ﬁnal state back to the corresponding option sampled at the beginning of the episode.
Revised: (2018) where the option vocabulary size is gradually increased as the option decoder q ν ( ω | s f ) becomes more conﬁdent in mapping back the ﬁnal state back to the corresponding option sampled at the beginning of the episode.

Original: More concretely, whenever q ν ( ω | s 0 , s f ) > 0 .
Revised: More concretely, whenever q ν ( ω | s 0 , s f ) > 0 .

Original: 75 (threshold chosen via hyperparameter tuning), the option vocabulary size increases according to
Revised: 75 (threshold chosen via hyperparameter tuning), the option vocabulary size increases according to

Original: [Equation]
Revised: [Equation]

Original: For our experiments, we start from K = 2 and terminate the curriculum when K = K max = 32 .
Revised: For our experiments, we start from K = 2 and terminate the curriculum when K = K max = 32 .

Original: We run each of our experiments on a single GTX Titan-X GPU, and use no more than 1 GB of GPU memory per experimental setting.
Revised: We run each of our experiments on a single GTX Titan-X GPU, and use no more than 1 GB of GPU memory per experimental setting.

Original: A. A DDITIONAL R ESULTS
Revised: A. A DDITIONAL R ESULTS

Original: In Fig.
Revised: In Fig.

Original: 7 and Fig.
Revised: 7 and Fig.

Original: 8, we show further results on the MultiRoomN3S4 and MultiRoomN5S4 environments to compare DS-VIC with other exploration incentives in terms of sample-complexity and the quality of the exploration incentive.
Revised: 8, we show further results on the MultiRoomN3S4 and MultiRoomN5S4 environments to compare DS-VIC with other exploration incentives in terms of sample-complexity and the quality of the exploration incentive.

Original: As seen in the ﬁgures, DS-VIC is either competitive with or better than the presented baselines.
Revised: As seen in the ﬁgures, DS-VIC is either competitive with or better than the presented baselines.

Original: As highlighted before, MultiRoomN3S4 and MultiRoomN5S
Revised: As highlighted before, MultiRoomN3S4 and MultiRoomN5S

Original: [Table]
Revised: [Table]

Original: are relatively simpler environments where simpler baselines like count based exploration incentives also perform quite well due to the small size of rooms.
Revised: are relatively simpler environments where simpler baselines like count based exploration incentives also perform quite well due to the small size of rooms.

Original: In Table.
Revised: In Table.

Original: we report broader comparisons with other exploration baselines InfoBot (Goyal et al., 2019) compare with.
Revised: we report broader comparisons with other exploration baselines InfoBot (Goyal et al., 2019) compare with.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]