Original: S UCCESSOR O PTIONS : A N O PTION D ISCOVERY A LGORITHM FOR R EINFORCEMENT L EARNING
Revised: S UCCESSOR O PTIONS : A N O PTION D ISCOVERY A LGORITHM FOR R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Hierarchical Reinforcement Learning is a popular method to exploit temporal abstractions in order to tackle the curse of dimensionality.
Revised: Hierarchical Reinforcement Learning is a popular method to exploit temporal abstractions in order to tackle the curse of dimensionality.

Original: The options framework is one such hierarchical framework that models the notion of skills or options.
Revised: The options framework is one such hierarchical framework that models the notion of skills or options.

Original: However, learning a collection of task-agnostic transferable skills is a challenging task.
Revised: However, learning a collection of task-agnostic transferable skills is a challenging task.

Original: Option discovery typically entails using heuristics, the majority of which revolve around discovering bottleneck states.
Revised: Option discovery typically entails using heuristics, the majority of which revolve around discovering bottleneck states.

Original: In this work, we adopt a method complementary to the idea of discovering bottlenecks.
Revised: In this work, we adopt a method complementary to the idea of discovering bottlenecks.

Original: Instead, we attempt to discover “landmark” sub-goals which are prototypical states of well connected regions.
Revised: Instead, we attempt to discover “landmark” sub-goals which are prototypical states of well connected regions.

Original: These sub-goals are points from which  densely connected set of states are easily accessible.
Revised: These sub-goals are points from which a densely connected set of states are easily accessible.

Original: We propose a new model called Successor options that leverages Successor Representations to achieve the same.
Revised: We propose a new model called Successor options that leverages Successor representations to achieve the same.

Original: We also design a novel pseudoreward for learning the intra-option policies.
Revised: We also design a novel pseudo-reward for learning the intra-option policies.

Original: Additionally, we describe an Incremental Successor options model that iteratively builds options and explores in environments where exploration through primitive actions is inadequate to form the Successor Representations.
Revised: Additionally, we describe an Incremental Successor options model that iteratively builds options and explores in environments where exploration through primitive actions is inadequate to form the Successor representations.

Original: Finally, we demonstrate the efﬁcacy of our approach on a collection of grid worlds and on complex high dimensional environments like
Revised: Finally, we demonstrate the efﬁcacy of our approach on a collection of grid worlds and on complex high dimensional environments like

Original: Deepmind-Lab.
Revised: Deepmind-Lab.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement Learning (RL) (Sutton & Barto, 1998) has garnered signiﬁcant attention recently due to its success in challenging high-dimensional tasks (Mnih et al., 2015; 2016; Silver et al., 2016;
Revised: Reinforcement Learning (RL) (Sutton & Barto, 1998) has garnered signiﬁcant attention recently due to its success in challenging high-dimensional tasks (Mnih et al., 2015; 2016; Silver et al., 2016;

Original: Hessel et al., 2017; Lillicrap et al., 2015;
Revised: Hessel et al., 2017; Lillicrap et al., 2015;

Original: Oh et al., 2016; Schulman et al., 2015).
Revised: Oh et al., 2016; Schulman et al., 2015).

Original: Deep Learning has had a major role in the achievements of RL by enabling generalization across a large number of states using powerful function approximators.
Revised: Deep Learning has had a major role in the achievements of RL by enabling generalization across a large number of states using powerful function approximators.

Original: Nevertheless, there exists vast room for improvement in terms of exploration and sample complexity that cannot be addressed solely using Deep Learning.
Revised: Nevertheless, there exists vast room for improvement in terms of exploration and sample complexity that cannot be addressed solely using Deep Learning.

Original: Hierarchical Reinforcement Learning(HRL) is one potential strategy that mitigates the curse of dimensionality using temporal abstraction.
Revised: Hierarchical Reinforcement Learning (HRL) is one potential strategy that mitigates the curse of dimensionality using temporal abstraction.

Original: Recent work (Vezhnevets et al., 2017; Kulkarni et al., 2016a; Bacon et al., 2017) has attempted to use a hierarchy of controllers operating in different time-scales, in order to search large state spaces rapidly.
Revised: Recent work (Vezhnevets et al., 2017; Kulkarni et al., 2016a; Bacon et al., 2017) has attempted to use a hierarchy of controllers operating in different time-scales, in order to search large state spaces rapidly.

Original: The options framework (Sutton et al., 1999) is an example of a hierarchical approach that models temporally extended actions or skills .
Revised: The options framework (Sutton et al., 1999) is an example of a hierarchical approach that models temporally extended actions or skills .

Original: The discovery of options has not been effectively addressed and is known to be a hard task.
Revised: The discovery of options has not been effectively addressed and is known to be a hard task.

Original: Meticulous design of options are often required to reap the beneﬁts of the options framework.
Revised: Meticulously designing the options is often required to reap the beneﬁts of the options framework.

Original: While there are a number of approaches to this problem, a large fraction of literature revolves around discovering options that navigate to bottleneck states (McGovern & Barto, 2001; S¸ims¸ek & Barto, 2004; S¸ims¸ek et al., 2005; Menache et al., 2002).
Revised: While there are a number of approaches to this problem, a large fraction of literature revolves around discovering options that navigate to bottleneck states (McGovern & Barto, 2001; S¸ims¸ek & Barto, 2004; S¸ims¸ek et al., 2005; Menache et al., 2002).

Original: In this work, we adopt a paradigm that fundamentally differs from the idea of identifying bottleneck states as subgoals.
Revised: In this work, we adopt a paradigm that fundamentally differs from the idea of identifying bottleneck states as sub-goals.

Original: Instead, we attempt to discover landmark or prototypical states of well connected regions while ensuring that they are also spread apart.
Revised: Instead, we attempt to discover landmark or prototypical states of well connected regions while ensuring that they are also spread apart.

Original: Building on this intuition, we propose Successor
Revised: Building on this intuition, we propose Successor

Original: Options , a sub-goal discovery and intra-option policy learning framework.
Revised: Options , a sub-goal discovery and intra-option policy learning framework.

Original: While bottlenecks have their utility in many scenarios(discovering keys or doors), Successor options discover options that are likely to be better for exploration.
Revised: While bottlenecks have their utility in many scenarios (discovering keys or doors), Successor options discover options that are likely to be better for exploration.

Original: The sub-goals provide greater accessibility to a larger number of states within a well connected region allowing an agent to effectively explore within that region.
Revised: The sub-goals provide greater accessibility to a larger number of states within a well connected region allowing an agent to effectively explore within that region.

Original: We validate this claim on a number of grid world tasks and observe faster learning in all of the tasks.
Revised: We validate this claim on a number of grid world tasks and observe faster learning in all of the tasks.

Original: To discover the options, we leverage Successor representations(SR)
Revised: To discover the options, we leverage Successor representations

Original:  (Dayan, 1993) to learn the sub-goals and the intra-option policy.
Revised: (SR) (Dayan, 1993) to learn the sub-goals and the intra-option policy.

Original: Successor representation is a vector encoding the expected discounted visitation counts of all states in the future.
Revised: Successor representation is a vector encoding the expected discounted visitation counts of all states in the future.

Original: The SR varies with the policy since the visitation counts of the states depend on the policy being executed to estimate these counts.
Revised: The SR varies with the policy since the visitation counts of the states depend on the policy being executed to estimate these counts.

Original: The ﬁrst step in our option discovery method involves collecting the SRs of all states.
Revised: The ﬁrst step in our option discovery method involves collecting the SRs of all states.

Original: The sub-goals are then identiﬁed by clustering a large sample(or all) of the SR vectors and assigning the cluster centers as the various sub-goals.
Revised: The sub-goals are then identiﬁed by clustering a large sample (or all) of the SR vectors and assigning the cluster centers as the various sub-goals.

Original: This steps discovers a collection of goals which are well separated in the Successor representation space.
Revised: This steps discovers a collection of goals which are well separated in the Successor representation space.

Original: Since the SR encodes information about the consequently occurring states, the cluster centers translate to sub-goals that have vastly different successor states, meaning different sub-goals provide access to a different region in state-space.
Revised: Since the SR encodes information about the consequently occurring states, the cluster centers translate to sub-goals that have vastly different successor states, meaning different sub-goals provide access to a different region in state-space.

Original: The overview of our option discovery procedure is summarized in Figure 1.
Revised: The overview of our option discovery procedure is summarized in Figure 1.

Original: Building the successor representation however relies on the usage of primitive actions to navigate the state space(required to estimate discounted visitation counts).
Revised: Building the Successor representation however relies on the usage of primitive actions to navigate the state space (required to estimate discounted visitation counts).

Original: To mitigate the same, we propose an Incremental Successor Option algorithm.
Revised: To mitigate the same, we propose an Incremental Successor Option algorithm.

Original: This method works in an iterative fashion where each iteration involves an option discovery step and an SR update step.
Revised: This method works in an iterative fashion where each iteration involves an option discovery step and an SR update step.

Original: The updated SR is used to form a more accurate set of options.
Revised: The updated SR is used to form a more accurate set of options.

Original: These newly formed options improve the SR updates in the consequent iteration.
Revised: These newly formed options improve the SR updates in the consequent iteration.

Original: This synergy enables the rapid learning  Successor representations by improving sample efﬁciency.
Revised: This synergy enables the rapid learning of Successor representations by improving sample efﬁciency.

Original: The Successor Options offer a number of advantages.
Revised: Successor Options offer a number of advantages.

Original: While an intermediate clustering step segments the algorithm into distinct stages(non-differentiability introduced), the step is critical in many aspects.
Revised: While an intermediate clustering step segments the algorithm into distinct stages (non-differentiability introduced), the step is critical in many aspects.

Original: Firstly, the number of options k , is speciﬁed beforehand which allows the model to adapt by ﬁnding k most suited sub-goals.
Revised: Firstly, the number of options k , is speciﬁed beforehand which allows the model to adapt by ﬁnding k most suited sub-goals.

Original: The algorithm does not require the pruning of options from a very large set unlike other works (McGovern & Barto, 2001; S¸ims¸ek & Barto, 2009; Machado et al., 2017).
Revised: The algorithm does not require the pruning of options from a very large set unlike other works (McGovern & Barto, 2001; S¸ims¸ek & Barto, 2009; Machado et al., 2017).

Original: The clustering also ensures robust options since the cluster centroids are an average of many successor representations.
Revised: The clustering also ensures robust options since the cluster centroids are an average of many Successor representations.

Original: Furthermore, the discovered options are reward agnostic and are easily transferable across multiple tasks.
Revised: Furthermore, the discovered options are reward agnostic and are easily transferable across multiple tasks.

Original: This enables solving a plethora of tasks varying solely in reward structure, in a rapid fashion.
Revised: This enables solving a plethora of tasks varying solely in reward structure, in a rapid fashion.

Original: The sub-goals are relevant even in cases where the environment has no natural bottlenecks and hence has wider applicability.
Revised: The sub-goals are relevant even in cases where the environment has no natural bottlenecks and hence has wider applicability.

Original: Lastly, the algorithm is easily extended to function approximators since Successor representations have a suitable neural network formulation (Kulkarni et al., 2016b; Barreto et al., 2017).
Revised: Lastly, the algorithm is easily extended to function approximators since Successor representations have a suitable neural network formulation (Kulkarni et al., 2016b; Barreto et al., 2017).

Original: The principle contributions of this paper are as follows : (i)
Revised: The principle contributions of this paper are as follows : (i)

Original: An automatic option discovery mechanism revolving around identifying landmark states, (ii) A novel pseudo reward for learning the intra-option policies that extends to function approximators (iii)
Revised: An automatic option discovery mechanism revolving around identifying landmark states, (ii) A novel pseudo reward for learning the intra-option policies that extends to function approximators (iii)

Original: An incremental approach that alternates between exploration and option construction to navigate the state space in tasks with a ﬁxed horizon setup where primitive actions fail to explore fully.
Revised: An incremental approach that alternates between exploration and option construction to navigate the state space in tasks with a ﬁxed horizon setup where primitive actions fail to explore fully.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Reinforcement Learning deals with sequential decision making tasks and considers the interaction of an agent with an environment.
Revised: Reinforcement Learning deals with sequential decision making tasks and considers the interaction of an agent with an environment.

Original: It is traditionally modeled by a Markov Decision Process (MDP)(Puterman, 1994), deﬁned by the tuple (cid:104)S , A , P , ρ (cid:48) , r, γ (cid:105) , where S deﬁnes the set of states, A the set of actions, P : S ×
Revised: It is traditionally modeled by a Markov Decision Process (MDP) (Puterman, 1994), deﬁned by the tuple (cid:104)S , A , P , ρ (cid:48) , r, γ (cid:105) , where S deﬁnes the set of states, A the set of actions, P : S ×

Original: A → S the transition function, ρ (cid:48)
Revised: A → S the transition function, ρ (cid:48)

Original: the probability distribution over initial states, r : S × S (cid:48) ×
Revised: the probability distribution over initial states, r : S × S (cid:48) ×

Original: A → R the reward function and γ the discount factor.
Revised: A → R the reward function and γ the discount factor.

Original: A policy dictates the behaviour of the agent in an environment.
Revised: A policy dictates the behaviour of the agent in an environment.

Original: More formally, a policy is denoted by π ( s ): S → P ( A ) , where P ( A ) deﬁnes a probability distribution over actions
Revised: More formally, a policy is denoted by π ( s ): S → P ( A ) , where P ( A ) deﬁnes a probability distribution over actions

Original: a (cid:15) A in a state s (cid:15) S .
Revised: a (cid:15) A in a state s (cid:15) S .

Original: In the context of optimal control, the objective is to learn a policy that maximizes the discounted return R t = (cid:80)
Revised: In the context of optimal control, the objective is to learn a policy that maximizes the discounted return R t = (cid:80)

Original: Ti = t γ ( i − t ) r ( s i , s i +1 , a i ) , where r ( s i , s i +1 , a i ) is the reward function.
Revised: Ti = t γ ( i − t ) r ( s i , s i +1 , a i ) , where r ( s i , s i +1 , a i ) is the reward function.

Original: Deep Q-learning: Q-learning (Watkins & Dayan, 1992) attempts to estimate the optimal actionvalue function Q ∗ ( s, a ) .
Revised: Deep Q-learning: Q-learning (Watkins & Dayan, 1992) attempts to estimate the optimal actionvalue function Q ∗ ( s, a ) .

Original: It exploits the Bellman optimality equation, the repeated application of which leads to convergence to Q ∗ ( s, a ) .
Revised: It exploits the Bellman optimality equation, the repeated application of which leads to convergence to Q ∗ ( s, a ) .

Original: The optimal value function can be used to behave optimally by selecting action a in every state such that a ∈ argmax(cid:48) a  Q ( s, a (cid:48) )
Revised: The optimal value function can be used to behave optimally by selecting action a in every state such that a ∈ argmax a [Equation] Q [Equation]

Revised: s

Original: [Equation]
Revised: [Equation]

Revised: a

Revised: [Equation]

Original: Mnih et al.
Revised: Mnih et al.

Original: (2015) introduce Deep Q-learning, that extends Q-learning to high dimensional spaces by using a neural network to model Q
Revised: (2015) introduce Deep Q-learning, that extends Q-learning to high dimensional spaces by using a neural network to model Q

Original: θ ( s, a ) .
Revised: θ ( s, a ) .

Original: The authors use the TD-error to back-propagate the gradients which are used to learn the parameters θ .
Revised: The authors use the TD-error to back-propagate the gradients which are used to learn the parameters θ .

Original: A delayed target-network and an experience replay (Mnih et al., 2015) are typically utilized to stabilize the training and reduce the correlation between samples in a mini-batch.
Revised: A delayed target-network and an experience replay (Mnih et al., 2015) are typically utilized to stabilize the training and reduce the correlation between samples in a mini-batch.

Original: Options and Semi-Markov Decision Processes: Options (Sutton et al., 1999) provide a framework to model temporally extended actions.
Revised: Options and Semi-Markov Decision Processes: Options (Sutton et al., 1999) provide a framework to model temporally extended actions.

Original: Formally, an option is deﬁned using the 3-state tuple : (cid:104)I , β, π (cid:105) , where I ⊆ S is the initiation set, β : S → [0 , 1] the termination probabilities for each state and π : S → P ( A ) the intra-option policy.
Revised: Formally, an option is deﬁned using the 3-state tuple : (cid:104)I , β, π (cid:105) , where I ⊆ S is the initiation set, β : S → [0 , 1] the termination probabilities for each state and π : S → P ( A ) the intra-option policy.

Original: In this work, we assume that the intra-option policies satisfy the Markov assumption.
Revised: In this work, we assume that the intra-option policies satisfy the Markov assumption.

Original: The options can be understood as a sequence of actions and hence naturally ﬁt into the Semi-Markov decision process framework (Puterman, 1994).
Revised: The options can be understood as a sequence of actions and hence naturally ﬁt into the Semi-Markov decision process framework (Puterman, 1994).

Original: The Markov nature of intra-option policies can be exploited using intra-option value function updates (Sutton et al., 1999).
Revised: The Markov nature of intra-option policies can be exploited using intra-option value function updates (Sutton et al., 1999).

Original: This enables learning the value function associated with one option while executing another.
Revised: This enables learning the value function associated with one option while executing another.

Original: Successor Representation: The Successor Representation (SR) (Dayan, 1993) represents a state s in terms of its successors.
Revised: Successor Representation: The Successor Representation (SR) (Dayan, 1993) represents a state s in terms of its successors.

Original: The SR for s is deﬁned as a vector of size |S| with the i th index equal to the discounted future occupancy for state s i given the agent starts from s .
Revised: The SR for s is deﬁned as a vector of size |S| with the i th index equal to the discounted future occupancy for state s i given the agent starts from s .

Original: Since the SR captures the visitation of successor states, it is directly dependent on the policy π
Revised: Since the SR captures the visitation of successor states, it is directly dependent on the policy π

Original: and the transition dynamics p ( s t +1 | s t , a t ) .
Revised: and the transition dynamics p ( s t +1 | s t , a t ) .

Original: More concretely, the SR can be written as follows:
Revised: More concretely, the SR can be written as follows:

Original: [Equation]
Revised: [Equation]

Original: where, I ( . ) is 1 if its argument is true, else 0(indicator{+
 (indicator+} function).
Revised: where, I ( . ) is 1 if its argument is true, else {+
 (indicator+} function).

Original: The SR can be learned in a temporal difference(TD) like fashion by writing it in terms of the SR of the next state.
Revised: The SR can be learnt in a temporal difference (TD) like fashion by writing it in terms of the SR of the next state.

Original: [Equation]
Revised: [Equation]

Original: The above equation is for state samples s, s (cid:48) , where ˆ ψ is the estimate of SR being learned and 1 s is a one-hot vector with all zeros except a 1 at the s th position.
Revised: The above equation is for state samples s, s (cid:48) , where ˆ ψ is the estimate of SR being learnt and 1 s is a one-hot vector with all zeros except a 1 at the s th position.

Original: Successor Representations can be naturally extended to the deep setting (Kulkarni et al., 2016b; Barreto et al., 2017) as follows (note φ ( s ) is feature representation of s ) :[Equation]
Revised: Successor Representations can be naturally extended to the deep setting (Kulkarni et al., 2016b; Barreto et al., 2017) as follows (note φ ( s ) is feature representation of s ) :

Original: [Equation]
Revised: [Equation]

Original:  P ROPOSED M ETHOD
Revised:  P ROPOSED M ETHOD

Original: Option discovery has typically involved some type of Bottleneck discovery in many prior works.
Revised: Option discovery has typically involved some form of bottleneck discovery in many prior works.

Original: Bottleneck states can be discovered in a myriad of ways, such as ﬁnding states that are frequently visited while executing successful trajectories (McGovern & Barto, 2001), or by identifying states that connect different densely connected regions of the state space (Menache et al., 2002).
Revised: Bottleneck states can be discovered in a myriad of ways, such as ﬁnding states that are frequently visited while executing successful trajectories (McGovern & Barto, 2001), or by identifying states that connect different densely connected regions of the state space (Menache et al., 2002).

Original: Learning options that go to such bottleneck states have been shown to accelerate learning and improve exploration.
Revised: Learning options that go to such bottleneck states have been shown to accelerate learning and improve exploration.

Original: Successor Options(SR-options) adopts an approach which attempts to discover sub-goals that are representative of well connected regions.
Revised: Successor Options (SR-options) adopts an approach which attempts to discover sub-goals that are representative of well connected regions.

Original: This technique exhibits two advantages, (i) of learning useful options without extrinsic reward, i.e. latent learning scenarios and (ii) of learning to explore the state space incrementally in scenarios where primitive actions are unable to facilitate full exploration.
Revised: This technique exhibits two advantages, (i) of learning useful options without extrinsic reward, i.e. latent learning scenarios and (ii) of learning to explore the state space incrementally in scenarios where primitive actions are unable to facilitate full exploration.

Original: 3. S UCCESSOR O PTIONS
Revised: 3. S UCCESSOR O PTIONS

Original: In learning Successor Options, the ﬁrst step involves learning the SR.
Revised: In learning Successor Options, the ﬁrst step involves learning the SR.

Original: This is followed by clustering states based on the learnt SR values(We utilize K-means++ (Arthur & Vassilvitskii, 2007)).
Revised: This is followed by clustering states based on the learnt SR (We utilize K-means++ (Arthur & Vassilvitskii, 2007)).

Original: Since the SR captures temporally close-by states efﬁciently, we are able to generate clusters which are spread across the state space, with each cluster assigned to  states that are densely connected.
Revised: Since the SR captures temporally close-by states efﬁciently, we are able to generate clusters which are spread across the state space, with each cluster assigned to a set of states that are densely connected.

Original: The
Revised: The

Original: SR can be understood to encode the connectivity of the underlying graph of the environment as evident in Figure 2.
Revised: SR can be understood to encode the connectivity of the underlying graph of the environment as evident in Figure 2.

Original: We wish to learn options over a given clustering since the cluster centers act as landmark states or sub-goals for these options.
Revised: We wish to learn options over a given clustering, since the cluster centers act as landmark states or sub-goals for these options.

Original: Hence, a sub-goal is that state whose SR has the smallest cosine similarity to a corresponding cluster center.
Revised: Hence, a sub-goal is that state whose SR has the largest cosine similarity to a corresponding cluster center.

Original: The sub-goal discovery step is followed by learning the intra-option policies corresponding to every sub-goal.
Revised: The sub-goal discovery step is followed by learning the intra-option policies corresponding to every sub-goal.

Original: Latent Learning:
Revised: Latent Learning:

Original: In order to learn option policies, we attempt to ascend the SRvalue of the cluster center.
Revised: In order to learn option policies, we attempt to ascend the SR of the cluster center.

Original: This intuition is translated into an appropriate reward over which the policy is learnt.
Revised: This intuition is translated into an appropriate reward over which the policy is learnt.

Original: Figureis the Successor Representation of a state learnt using the uniformly random policy which highlights why climbing the SR of the sub-goal is a valid choice.
Revised: Figure 2 is the Successor Representation of a state learnt using the uniformly random policy which highlights why climbing the SR of the sub-goal is a valid choice.

Original: The reward function used, is formalized in equation 5 ( ψ s is the SR of state s ) .
Revised: The reward function used, is formalized in equation 5 ( ψ s is the SR of state s ) .

Original: This provides a dense reward which ensures that steps toward the sub-goal are positively rewarded while moving away from the cluster center is penalized.
Revised: This provides a dense reward which ensures that steps toward the sub-goal are positively rewarded while moving away from the cluster center is penalized.

Original: Note that an approximately developed SR is often a sufﬁcient signal for learning optimal options.
Revised: Note that an approximately developed SR is often a sufﬁcient signal for learning optimal options.

Original: [Equation]
Revised: [Equation]

Original: Every option has every state in it’s initiation set.
Revised: Every option has every state in it’s initiation set.

Original: The termination set of every option is the set of all states satsifying Q ( s, a ) ≤ 0 .
Revised: The termination set of every option is the set of all states satisfying Q ( s, a ) ≤ 0 .

Original: The intra-option policy, is that learnt from the pseudo reward.
Revised: The intra-option policy, is that learnt from the pseudo reward.

Original: Now that the options are obtained, the only remaining step is to use SMDP Q-learning to learn a policy over the discovered options.
Revised: Now that the options are obtained, the only remaining step is to use SMDP Q-learning to learn a policy over the discovered options.

Original: 3. I NCREMENTAL S UCCESSOR O PTIONS
Revised: 3. I NCREMENTAL S UCCESSOR O PTIONS

Original: Here we consider the case where the agent is unable to explore the whole state space with only primitive actions in ﬁnite time.
Revised: Here we consider the case where the agent is unable to explore the whole state space with only primitive actions in ﬁnite time.

Original: This hinders the learning of usable Successor representations, critical to the SR options algorithm.
Revised: This hinders the learning of usable Successor representations, critical to the SR options algorithm.

Original: We wish to explore incrementally and learn options which cover all parts of the state space in order to facilitate learning for reward based tasks.
Revised: We wish to explore incrementally and learn options which cover all parts of the state space in order to facilitate learning for reward based tasks.

Original: This is especially important in sparse reward long horizon tasks where exploration can be facilitated using reward independent options.
Revised: This is especially important in sparse reward long horizon tasks where exploration can be facilitated using reward independent options.

Original: While incrementally exploring and learning options, no extrinsic reward is considered.
Revised: While incrementally exploring and learning options, no extrinsic reward is considered.

Original: We propose iteratively learning the SR and options using the above mentioned clustering scheme.
Revised: We propose iteratively learning the SR and options using the above mentioned clustering scheme.

Original: The policy used for learning the SR is augmented with the previously learnt options to allow traversing between distinct clusters in the state space.
Revised: The policy used for learning the SR is augmented with the previously learnt options to allow traversing between distinct clusters in the state space.

Original: However, since the SR directly depends on the policy, we do not update the SR while executing an option.
Revised: However, since the SR directly depends on the policy, we do not update the SR while executing an option.

Original: This is desirable as we want to learn SR for a random policy and not for an option biased policy.
Revised: This is desirable as we want to learn SR for a random policy and not for an option biased policy.

Original: Executing the options allows the agent to land in well spread parts of the state space around which learning takes place only through random actions.
Revised: Executing the options allows the agent to land in well spread parts of the state space around which learning takes place only through random actions.

Original: Therefore, the augmented policy only uses options to quickly reach dissimilar parts of the state space and not for learning the SR itself.
Revised: Therefore, the augmented policy only uses options to quickly reach dissimilar parts of the state space and not for learning the SR itself.

Original: Algorithm
Revised: Algorithm

Original: A ← { a 0 , a 1 , a 2 , ..., } , a i (cid:15) A , O ← {} // Starting with only primitive actions for i = 1 , ..., N iterations do ψ ← LearnSR(O + A) //
Revised: A ← { a 0 , a 1 , a 2 , ..., } , a i (cid:15) A , O ← {} // Starting with only primitive actions for i = 1 , ..., N iterations do ψ ← LearnSR(O + A) //

Original: Eq.
Revised: Eq.

Original: C ← GetCandidateSubgoals()
Revised: C ← GetCandidateSubgoals()

Original: S ← ClusterSR( C , n)
Revised: S ← ClusterSR( C , n)

Original: Empty option set O ← {} // Relearning n new options every step for each subgoal s (cid:15) S do o ← LearnOption( ψ ( s ) ) //
Revised: Empty option set O ← {} // Relearning n new options every step for each sub-goal s (cid:15) S do o ← LearnOption( ψ ( s ) ) //

Original: Q learning with SR pseudo-reward
Revised: Q learning with SR pseudo-reward

Original: Store o in O end for end for return S , O
Revised: Store o in O end for end for return S , O

Original: In the above described algorithm, candidate subgoals are a fraction of reached states.
Revised: In the above described algorithm, candidate sub-goals are a fraction of reached states.

Original: Formally, a state s is a candidate subgoal if SR min < (cid:80) s (cid:48) ψ ( s, :) <
Revised: Formally, a state s is a candidate sub-goal if SR min < (cid:80) s (cid:48) ψ ( s, :) <

Original: SR max , where SR min and SR max are the ﬁrst and third quartile values of SR sums of all reached states respectively.
Revised: SR max , where SR min and SR max are the ﬁrst and third quartile values of SR sums of all reached states respectively.

Original: Such a condition ensures that all candidate subgoal states have an SRvalue that is neither fully developed nor extremely sparse or under developed, thus providing a pseudo reward which is easy to learn over.
Revised: Such a condition ensures that all candidate sub-goal states have an SR that is neither fully developed nor extremely sparse or under developed, thus providing a pseudo reward which is easy to learn over.

Original: This also allows candidate states to be states which lead to more exploration since these are not visited as frequently as some of the others.
Revised: This also allows candidate states to be states which lead to more exploration since these are not visited as frequently as some of the others.

Original: 3. S UCCESSOR O PTIONS IN THE F UNCTION A PPROXIMATION S ETTING
Revised: 3. S UCCESSOR O PTIONS IN THE F UNCTION A PPROXIMATION S ETTING

Original: In this section we extend our formulation for successor options to real world tasks with large number of achievable states.
Revised: In this section we extend our formulation for successor options to real world tasks with large number of achievable states.

Original: Kulkarni et al.
Revised: Kulkarni et al.

Original: (2016b)
Revised: (2016b)

Original: ; Barreto et al.
Revised: ; Barreto et al.

Original: (2017) propose an architecture for generating the SR in the function approximation case or Successor Feature(SF) by learning from three signal branches, i.e. the reward prediction error, image reconstruction error and the TD error for learning Successor Features.
Revised: (2017) propose an architecture for generating the SR in the function approximation case or Successor Feature (SF) by learning from three signal branches, i.e. the reward prediction error, image reconstruction error and the TD error for learning Successor Features.

Original: All three branches share the same base representation φ ( s ) for learning here.
Revised: All three branches share the same base representation φ ( s ) for learning here.

Original: The reward prediction is required in order to compute the Q values as the dot product of the SF ψ ( s ) and reward weight vector w .
Revised: The reward prediction is required in order to compute the Q values as the dot product of the SF ψ ( s ) and reward weight vector w .

Original: Since we do not learn for control in our case, we do not include the reward layer in our architecture.
Revised: Since we do not learn for control in our case, we do not include the reward layer in our architecture.

Original: Unlike other works (Machado et al., 2017; Lakshminarayanan et al., 2016)
Revised: Unlike other works (Machado et al., 2017; Lakshminarayanan et al., 2016)

Original: we do not need to approximate or compute a graph Laplacian and the formulation can be naturally extended to neural networks.
Revised: we do not need to approximate or compute a graph Laplacian and the formulation can be naturally extended to neural networks.

Original: The architecture used, is presented in
Revised: The architecture used, is presented in

Original: Figure 4.
Revised: Figure 4.

Original: Generally, an image reconstruction task acts as an auxiliary task to stabilize training and avoid learning a trivial SF representation.
Revised: Generally, an image reconstruction task acts as an auxiliary task to stabilize training and avoid learning a trivial SF representation.

Original: We use a similar task of reconstructing an intermediate representation instead of predicting the input image itself.
Revised: We use a similar task of reconstructing an intermediate representation instead of predicting the input image itself.

Original: The auxiliary loss hence takes the form
Revised: The auxiliary loss hence takes the form

Original: L auxiliary =
Revised: L auxiliary =

Original: Error in Reconstruction of layer.
Revised: Error in Reconstruction of layer.

Original: Following the training of SF, we collect sufﬁcient samples of the SF vectors and stack them to form our sample SF matrix.
Revised: Following the training of SF, we collect sufﬁcient samples of the SF vectors and stack them to form our sample SF matrix.

Original: Similar to the tabular case, this matrix is clustered to produce SF cluster center.
Revised: Similar to the tabular case, this matrix is clustered to produce SF cluster center.

Original: For learning corresponding options, we use the dense reward given by the cluster center
Revised: For learning corresponding options, we use the dense reward given by the cluster center

Original: SF( M s ) value as the reward in a standard DQN training procedure.
Revised: SF ( ψ s ) value as the reward in a standard DQN training procedure.

Original: The weights for the shared representation are frozen when learning option policies.
Revised: The weights for the shared representation are frozen when learning option policies.

Original: This is required since the rewards depend on φ ( s ) and hence should not change for a particular state.
Revised: This is required since the rewards depend on φ ( s ) and hence should not change for a particular state.

Original: Equation 6 gives the reward function( M s is  one of the cluster centers) used to train the intra-option policy.[Equation]
Revised: Equation 6 gives the reward function( ψ s is the SR of one of the cluster centers) used to train the intra-option policy.

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS AND R ESULTS
Revised:  E XPERIMENTS AND R ESULTS

Original: We demonstrate our method on four different grid world tasks for the tabular setting and on the
Revised: We demonstrate our method on four different grid world tasks for the tabular setting and on the

Original: Deepmind-Lab Suite tasks for the function approximation case.
Revised: Deepmind-Lab Suite tasks for the function approximation case.

Original: The grid world tasks (Figure 3) are designed with increasing complexity both in terms of the number of states available and the structure of rooms.
Revised: The grid world tasks (Figure 3) are designed with increasing complexity both in terms of the number of states available and the structure of rooms.

Original: This is done so as to test the generality of Successor options in learning well spread out options, each being representative of states enclosed in close-by rooms.
Revised: This is done so as to test the generality of Successor options in learning well spread out options, each being representative of states enclosed in close-by rooms.

Original: [Equation]
Revised: [Equation]

Original: Visualizing Successor Option Subgoals
Revised: Visualizing Successor Option Sub-goals

Original: The sub-goals obtained using our clustering scheme are more spread out as compared to those obtained using the eigen-option discovery framework (Machado et al., 2017) as evident from Figure 5.
Revised: The sub-goals obtained using our clustering scheme are more spread out as compared to those obtained using the eigen-option discovery framework (Machado et al., 2017) as evident from Figure 5.

Original: Moreover, it can be easily seen that each sub-goal works as a good landmark state for all states very close to it, i.e. landing in any of the sub-goals allows efﬁciently exploring other states around them when learning for goal directed behavior.
Revised: Moreover, it can be easily seen that each sub-goal works as a good landmark state for all states very close to it, i.e. landing in any of the sub-goals allows efﬁciently exploring other states around them when learning for goal directed behavior.

Original: This is not the  sub
Revised: This is not the case sub eigen- around corners with multiple sub

Original: Successor Options vs Eigen-Options We compare successor options with eigen-options with respect to performance in learning extrinsic reward based tasks(see Figure 6).
Revised: Successor Options vs Eigen-Options We compare successor options with eigen-options with respect to performance in learning extrinsic reward based tasks (see Figure 6).

Original: During SMDP Qlearning, a uniformly random policy among the options and actions (typically used in exploration) will result in the agent spending a large fraction of it’s time near these sub-goals.
Revised: During SMDP Qlearning, a uniformly random policy among the options and actions (typically used in exploration) will result in the agent spending a large fraction of it’s time near these sub-goals.

Original: Machado et al.
Revised: Machado et al.

Original: (2017) also observe this behaviour when working with eigen-options.
Revised: (2017) also observe this behaviour when working with eigen-options.

Original: Hence, in order to effectively explore the environment using the exploration policy, it is important to sample actions and options non-uniformly.
Revised: Hence, in order to effectively explore the environment using the exploration policy, it is important to sample actions and options non-uniformly.

Original: Therefore, we utilize variants of both successor and eigen-options called successor non-uniform and eigen non-uniform options which refer to sampling primitive actions at a higher probability than the options(sampling in the exploratory policy).
Revised: Therefore, we utilize variants of both successor and eigen-options called successor non-uniform and eigen non-uniform options which refer to sampling primitive actions at a higher probability than the options(sampling in the exploratory policy).

Original: In all our experiments, we ﬁx the ratio of sampling an option to an action as 1:19.
Revised: In all our experiments, we ﬁx the ratio of sampling an option to an action as 1:19.

Original: The task used to compare the algorithms was a binary reward task where an agent was expected to navigate to the goal(+1 on reaching goal and +0 otherwise).
Revised: The task used to compare the algorithms was a binary reward task where an agent was expected to navigate to the goal(+1 on reaching goal and +0 otherwise).

Original: We use (4, 5, 10, 10) options in each of the grid worlds.
Revised: We use (4, 5, 10, 10) options in each of the grid worlds.

Original: 500 different instances of the task were run, each with a random start and goal state and the reported plots were averaged over these 500 instances.
Revised: 500 different instances of the task were run, each with a random start and goal state and the reported plots were averaged over these 500 instances.

Original: In order to gauge the speed at which each algorithm learns, we use the area under the return curve(AUC).
Revised: In order to gauge the speed at which each algorithm learns, we use the area under the return curve (AUC).

Original: Let N be the total number of steps(primitive actions) taken in the environment during training.
Revised: Let N be the total number of steps (primitive actions) taken in the environment during training.

Original: After every K steps, we test the performance of the agent and note the return.
Revised: After every K steps, we test the performance of the agent and note the return.

Original: The sum of the (cid:98) NK (cid:99) different evaluations(each occurring at an interval of K steps) is deﬁned as the Area under the Return Curve .
Revised: The sum of the (cid:98) NK (cid:99) different evaluations (each occurring at an interval of K steps) is deﬁned as the Area under the Return Curve .

Original: Incremental Successor Options We consider scenarios where learning from only primitive actions is insufﬁcient.
Revised: Incremental Successor Options We consider scenarios where learning from only primitive actions is insufﬁcient.

Original: Such a case resembles many real world tasks and therefore is important to show the generality of our method.
Revised: Such a case resembles many real world tasks and therefore is important to show the generality of our method.

Original: For instance, in a robotic manipulation task of pushing a block, given enough freedom in the setup, the agent fails to hit the block with random primitive actions in given time steps.
Revised: For instance, in a robotic manipulation task of pushing a block, given enough freedom in the setup, the agent fails to hit the block with random primitive actions in given time steps.

Original: To realize this distinction, we consider working strictly in a ﬁnite horizon setting even when learning without an extrinsic reward, i.e. the agent restarts from a single state after every N number of steps.
Revised: To realize this distinction, we consider working strictly in a ﬁnite horizon setting even when learning without an extrinsic reward, i.e. the agent restarts from a single state after every N number of steps.

Original: Since a random policy of primitive actions is not able to reach all states inside the horizon, adding such a constraint now makes learning the SR extremely hard for the grid world tasks we consider.
Revised: Since a random policy of primitive actions is not able to reach all states inside the horizon, adding such a constraint now makes learning the SR extremely hard for the grid world tasks we consider.

Original: We show how using the Incremental Successor options, we are able to explore intelligently, gradually learning the SR and ﬁnding sub-goals that slowly spread out across the whole state space (see Figure 7).
Revised: We show how using the Incremental Successor options, we are able to explore intelligently, gradually learning the SR and ﬁnding sub-goals that slowly spread out across the whole state space (see Figure 7).

Original: Deep Successor Options We show how clustering the successor feature vectors after learning the deep successor representation can allow learning useful options in the maze task of the DeepmindLab Suite (Figure 8).
Revised: Deep Successor Options We show how clustering the successor feature vectors after learning the deep successor representation can allow learning useful options in the maze task of the DeepmindLab Suite (Figure 8).

Original: For learning option policies, we use standard DQN with the reward given as in equation 6.
Revised: For learning option policies, we use standard DQN with the reward given as in equation 6.

Original: We train a total of 8 options.
Revised: We train a total of 8 options.

Original: We noted that one option crosses the doorway to get to the other room.
Revised: We noted that one option crosses the doorway to get to the other room.

Original: Another option learns to walk to the end of a corridor from the central part of the same.
Revised: Another option learns to walk to the end of a corridor from the central part of the same.

Original: The t-SNE visualization presented in the same ﬁgure indicates the learnt SR is able to categorize the states into discrete categories.
Revised: The t-SNE visualization presented in the same ﬁgure indicates the learnt SR is able to categorize the states into discrete categories.

Original: This is evident from the separation among groups of data-points.
Revised: This is evident from the separation among groups of data-points.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Moore et al.
Revised: Moore et al.

Original: (1999) introduce airport hierarchies which assign different states as airports or landmarks with various levels deﬁned on the basis of seniority.
Revised: (1999) introduce airport hierarchies which assign different states as airports or landmarks with various levels deﬁned on the basis of seniority.

Original: Each state is assigned to be a landmark only if it is reachable from a threshold number of states.
Revised: Each state is assigned to be a landmark only if it is reachable from a threshold number of states.

Original: The airport analogy is similar to the spread
Revised: The airport analogy is similar to the spread

Original: [Figure]
Revised: [Figure]

Original: of clusters
Revised: of clusters

Original: that we obtain for discovering options, since each airport also represents a group of similar states.
Revised: that we obtain for discovering options, since each airport also represents a group of similar states.

Original: Moreover, while learning iteratively, since we only use states with an adequately learnt
Revised: Moreover, while learning iteratively, since we only use states with an adequately learnt

Original: SR value, the candidate cluster centers are always able to leverage the SR as a dense reward and in turn learn successful option policies.
Revised: SR , the candidate cluster centers are always able to leverage the SR as a dense reward and in turn learn successful option policies.

Original: The overwhelming majority of literature on option discovery use techniques that revolve around discovering bottleneck states.
Revised: The overwhelming majority of literature on option discovery employ techniques that revolve around discovering bottleneck states.

Original: McGovern & Barto (2001) describe a diverse density based solution that casts this problem as a multiple-instance learning task.
Revised: McGovern & Barto (2001) describe a diverse density based solution that casts this problem as a multiple-instance learning task.

Original: The discovered solutions are bottlenecks since they are present in a larger fraction of positive bags.
Revised: The discovered solutions are bottlenecks since they are present in a larger fraction of positive bags.

Original: S¸ims¸ek & Barto (2009) describe a betweenness centrality based approach which also naturally leads to bottleneck based options.
Revised: S¸ims¸ek & Barto (2009) describe a betweenness centrality based approach which also naturally lead to bottleneck based options.

Original: Sub-goals based on relative novelty (S¸ims¸ek & Barto, 2004) identify states that could lead to vastly different states consequently which is closely tied to the notion of bottleneck states.
Revised: Subgoals based on relative novelty (S¸ims¸ek & Barto, 2004) identify states that could lead to vastly different states consequently which is closely tied to the notion of bottleneck states.

Original: Graph partitioning methods have also been employed to ﬁnd options (S¸ims¸ek et al., 2005; Lakshminarayanan et al., 2016; Menache et al., 2002).
Revised: Graph partitioning methods have also been employed to ﬁnd options (S¸ims¸ek et al., 2005; Lakshminarayanan et al., 2016; Menache et al., 2002).

Original: These methods design options that transition from one well connected region to another.
Revised: These methods design options that transition from one well connected region to another.

Original: Since the sub-goals are the boundaries between two well connected regions, these method also typically identify bottlenecks as sub-goals.
Revised: Since the sub-goals are the boundaries between two well connected regions, these method also typically identify bottlenecks as sub-goals.

Original: In contrast, our work attempts to discover sub-goals that are representative of a well-connected region and are well separated in addition.
Revised: In contrast, our work attempts to discover sub-goals that are representative of a well-connected region and are well separated in addition.

Original: Option Critic (Bacon et al., 2017) is an end-to-end differentiable model that learns options on a single task.
Revised: Option Critic (Bacon et al., 2017) is an end-to-end differentiable model that learns options on a single task.

Original: However, this method is forced to specialize to a single task and the learnt options are not easily transferable.
Revised: However, this method is forced to specialize to a single task and the learnt options are not easily transferable.

Original: Eigen-options ?
Revised: Eigen-options Machado et al.

Original: grpmachado2017laplacian uses the eigen-vectors of the Laplacian as rewards to learn intra-option policies.
Revised: (2017) use the eigen-vectors of the Laplacian as rewards to learn intra-option policies.

Original: This method however lacks a variety of subgoals since ascending the different eigen-vectors often correspond to reaching the same sub-goal.
Revised: This method however lacks a variety of sub-goals since ascending the different eigen-vectors often correspond to reaching the same sub-goal.

Original: Our proposed model uses a clustering stage to ensure that the sub-goals are sufﬁciently varied in nature.
Revised: Our proposed model uses a clustering stage to ensure that the sub-goals are sufﬁciently varied in nature.

Original: The clustering step also provides ﬂexibility regarding the number of options required, which is absent in the case of eigen-options.
Revised: The clustering step also provides ﬂexibility regarding the number of options required, which is absent in the case of eigen-options.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose a scheme for discovering options based on clusteringusing the Successor Representation.
Revised: We propose a scheme for discovering options based on clustering the Successor representations.

Original: We demonstrate that this method produces well separated and suitable located sub-goals which allows faster exploration when learning for extrinsic reward based tasks.
Revised: We demonstrate that this method produces well separated and suitably located sub-goals which allows faster exploration when learning for extrinsic reward based tasks.

Original: We also propose an iterative approach to this setting where primitive actions are unsuitable for navigating the state space.
Revised: We also propose an iterative approach to this setting where primitive actions are unsuitable for navigating the state space.

Original: We show how using such an approach in latent learning scenarios allows exploring the environment incrementally, ending up with useful options that can be reused for reward based tasks as well.
Revised: We show how using such an approach in latent learning scenarios allows exploring the environment incrementally and yields useful options that can be reused for reward based tasks as well.

Original: As future work we aim to use Deep Successor options to achieve optimal control on sparse reward tasks.
Revised: As future work, we aim to use Deep Successor options to achieve optimal control on sparse reward tasks.

Original: This work assumes that the initiation set is the set of all states.
Revised: This work assumes that the initiation set is the set of all states.

Original: We plan on looking into ways in which we can exploit the connectivity information of the SR to determine the initiation set.
Revised: We plan on looking into ways in which we can exploit the connectivity information of the SR to determine the initiation set.

Original: Currently, our setup uses very strict termination conditions which we could potentially turn into a soft probabilistic approach.
Revised: Currently, our setup uses very strict termination conditions which we could potentially turn into a soft probabilistic approach.

Original: Our current approach does not use the rewards to shape the nature of the options.
Revised: Our current approach does not use the rewards to shape the nature of the options.

Original: We plan on looking into techniques that alter the policy used to generate the successor representation based on extrinsic rewards to build SRs on relevant parts of the state space.
Revised: We plan on looking into techniques that alter the policy used to generate the Successor representation based on extrinsic rewards to build SRs on relevant parts of the state space.

Original: B G RID WORLD D ESCRIPTION
Revised: B G RID WORLD D ESCRIPTION

Original: We have 4 grid worlds in total.
Revised: We have 4 grid worlds in total.

Original: The dimensions of the grid worlds are 13 × 10 , 10 × 10 , 30 × 23 and 26 × 30 (left to right in Fig 3).
Revised: The dimensions of the grid worlds are 13 × 10 , 10 × 10 , 30 × 23 and 26 × 30 (left to right in Fig 3).

Original: The ﬁrst two grid worlds are the standard two-room and four-room domains.
Revised: The ﬁrst two grid worlds are the standard two-room and four-room domains.

Original: The tasks used for evaluation are navigation tasks where the aim is to reach a certain randomly selected goal from a random starting location.
Revised: The tasks used for evaluation are navigation tasks where the aim is to reach a certain randomly selected goal from a random starting location (500 such randomly chosen scenarios were considered).

Original: The value of γ = 0 .
Revised: The value of γ = 0 .

Original: 95 for each of these tasks so that the agent is encouraged to ﬁnish the task in least possible time.
Revised: 95 for each of these tasks so that the agent is encouraged to ﬁnish the task in least possible time.

Original: The agents haveactions from every state which include Left, right, forward, backward and No-op.
Revised: The agents have 5 actions from every state which include Left, right, forward, backward and No-op.

Revised: The task has a reward of 0 for every transition barring the transitions into the goal state which have a reward of +1.

Original: C D EEPMIND -L AB D OMAIN
Revised: C D EEPMIND -L AB D OMAIN

Original: The Deepmind-Lab task is a partially observable MDP and is a maze navigation task.
Revised: The Deepmind-Lab task is a partially observable MDP and is a maze navigation task.

Original: We designed our own custom maps to test the Deep Successor options Algorithm.
Revised: We designed our own custom maps to test the Deep Successor options Algorithm.

Original: The 4 available actions are rotate left, rotate right, move forward or move backward.
Revised: The 4 available actions are rotate left, rotate right, move forward or move backward.

Original: The input image is of size 320 ×pixels(RGB) and γ = 0 . 99 .
Revised: The input image is of size 320 ×pixels(RGB) and γ = 0 . 99 .

Original: We designed a task presented in Table 1.
Revised: We designed a task presented in Table 1.

Original: * indicates walls, P indicates potential player spawn point,
Revised: * indicates walls, P indicates potential player spawn point,

Original: G indicates the Goal (+10 reward) and A indicates Apple (+1) reward.
Revised: G indicates the Goal (+10 reward) and A indicates Apple (+1) reward.

Original: (notation consistent with
Revised: (notation consistent with

Original: Deepmind-Lab lua Map API)
Revised: Deepmind-Lab lua Map API)

Original: [Equation]
Revised: [Equation]

Original: P P
Revised: P P

Original: P P 
Revised: P P P

Original: P
Revised: P

Original: P * * * * * * *
Revised: P * * P A P

Original:  P* * * * * * * *
Revised: A P P P

Original: P P
Revised: P P

Original: P 
Revised: P P P * * * * * * * P * * * * * * * *

Original: P P 
Revised: P P P

Original: P P 
Revised: P P P

Original: P P 
Revised: P P P

Original: A
Revised: A

Original: P
Revised: P

Original: A * *
Revised: A * *

Original: A A 
Revised: A A P

Original: P P 
Revised: P P P

Original: PP P
Revised: P

Original: A
Revised: A

Original: P P
Revised: P P

Original: A
Revised: A

Original: P * * * * * * * * * * * * * * *
Revised: P * * * * * * * * * * * * * * *

Original: D S UCCESSOR R EPRESENTATION AND S TATE G RAPHS
Revised: D S UCCESSOR R EPRESENTATION AND S TATE G RAPHS

Original: Successor Representations encode the information regarding the state connectivity without explicitly modelling the same.
Revised: Successor Representations encode the information regarding the state connectivity without explicitly modelling the same.

Original: This is advantageous over other methods that work with the graph Laplacian since it becomes intractable in very large state spaces.
Revised: This is advantageous over other methods that work with the graph Laplacian since building graphs become intractable in very large state spaces.

Original: To understand the same, we analyze the 2dimensional projections of SRs obtained from grid worlds using t-SNE (Maaten & Hinton, 2008).
Revised: To understand the same, we analyze the 2-dimensional projections of SRs obtained from grid worlds using t-SNE (Maaten & Hinton, 2008).

Original: Figure 9 clearly indicates that the adjacency information of the states are present in the SR matrix.
Revised: Figure 9 clearly indicates that the adjacency information of the states are present in the SR matrix.

Original: The number of distinct clusters are highly indicative of the number distinct well connected regions present in the environment.
Revised: The number of distinct clusters are highly indicative of the number distinct well connected regions present in the environment.

Original: E C OMPARING C LUSTERING T ECHNIQUES AND M ETHODS
Revised: E C OMPARING C LUSTERING T ECHNIQUES AND M ETHODS

Original: The clustering of successor representations involves a number of different decisions.
Revised: The clustering of Successor representations involve a number of different decisions.

Original: The clustering algorithm is one such choice we experimented with.
Revised: The clustering algorithm is one such choice we experimented with.

Original: We looked at three algorithms, namely K- means++ (Arthur & Vassilvitskii, 2007), K-medoids (Park & Jun, 2009) and a greedy approach.
Revised: We looked at three algorithms, namely Kmeans++ (Arthur & Vassilvitskii, 2007), K-medoids (Park & Jun, 2009) and a greedy approach.

Original: The greedy approach is an iterative approach where each iteration adds a new sub-goal to an existing list.
Revised: The greedy approach is an iterative approach where each iteration adds a new sub-goal to an existing list.

Original: The added sub-goal is selected based on the criteria of having the maximum value for the minimum distance to all other existing list of sub-goals.
Revised: The added sub-goal is selected based on the criteria of having the maximum value for the minimum distance to all other existing list of sub-goals.

Original: The algorithm is a simple O ( n 2 ) routine.
Revised: The algorithm is a simple O ( n 2 ) routine.

Original: We use the same evaluation procedure as that used Section 4, in order to compare eigen-options and
Revised: We use the same evaluation procedure as that used Section 4, in order to compare eigen-options and

Original: Successor options.
Revised: Successor options.

Original: We also run a uniform and non-uniform option-action sampling procedure for each clustering method.
Revised: We also run a uniform and non-uniform option-action sampling procedure for each clustering method.

Original: The performances are shown in Figure 10.
Revised: The performances are shown in Figure 10.

Original: The plots indicate that using the K-means++ routine to obtain centroids (which is followed by choosing states closest to each of these centroids as sub-goals) yields the lhighest performance.
Revised: The plots indicate that using the K-means++ routine to obtain centroids (which is followed by choosing states closest to each of these centroids as sub-goals) yields the highest performance.

Original: Euclidean distance based clustering algorithms like K-means also rely on normalizing(normalization occurs across features) the various data-points across dimensions.
Revised: Euclidean distance based clustering algorithms like K-means also rely on normalizing (normalization occurs across features) the various data-points across dimensions.

Original: We look at the effect of the same on performance using a setup identical to that in Section 4,  used to compare eigen-options and Successor options.
Revised: We look at the effect of the same on performance, using a setup identical to that in Section 4, which was used to compare eigenoptions and Successor options.

Original: From Figure 11, the unnormalized version of the SR has a superior performance.
Revised: From Figure 11, the unnormalized version of the SR has a superior performance.

Original: We attribute this to the fact that the L 1 norm of the SR is identical for every state on convergence.
Revised: We attribute this to the fact that the L 1 norm of the SR is identical for every state on convergence.

Original: Hence, there is no explicit need for a normalization step.
Revised: Hence, there is no explicit need for a normalization step.

Original: The equation below gives the expression for the L 1 norm of the SR.
Revised: The equation below gives the expression for the L 1 norm of the SR.

Original: It is in-fact dependent only on γ as evident from the equation.[Equation] =
Revised: It is in-fact dependent only on γ as evident from the equation.

Original: [Equation]
Revised: [Equation]

Original: F I NCREMENTAL S UCCESSOR O PTIONS
Revised: F I NCREMENTAL S UCCESSOR O PTIONS

Original: In Figure 12, we showplots for how  increasingly spread out sub-goal options are learnt iteratively using incremental successor options.
Revised: In Figure 12, we show how the sub-goals become increasingly spread out as they are learnt iteratively using Incremental Successor options.

Original: We use 5 options in both grid worlds and the horizon is set to 100 time steps.
Revised: We use 5 options in both grid worlds and the horizon is set to 100 time steps.

Original: We observe that learning naively by selecting the rarest states as sub-goals does not help in exploration.
Revised: We observe that learning naively by selecting the rarest states as sub-goals does not help in exploration.

Original: This is because the SR for such states is extremely sparse and therefore does not provide a rich signal to learn options from.
Revised: This is because the SR for such states is extremely sparse and therefore does not provide a rich signal to learn options from.

Original: Moreover, since such least visited states are present close-by the sub-goals selected are not evenly spread out.
Revised: Moreover, since such least visited states are present close-by, the sub-goals selected are not evenly spread out.

Original: G E XPERIMENTAL D ETAILS - G RID W ORLD
Revised: G E XPERIMENTAL D ETAILS - G RID W ORLD

Original: The grid world tasks learnt (4, 5, 10, 10) options on the corresponding environment.
Revised: The grid world tasks learnt (4, 5, 10, 10) options on the corresponding environment.

Original: For the case of Eigen-options, an adjacency graph over the states of the environment was built by the agent.
Revised: For the case of eigen-options, an adjacency graph over the states of the environment was built by the agent.

Original: The eigen-vectors of the graph Laplacian are then obtained.
Revised: The eigen-vectors of the graph Laplacian are then obtained.

Original: These eigen-vectors are used as rewards for learning the eigen-options(as dictated by Machado et al.
Revised: These eigen-vectors are used as rewards for learning the eigen-options(as dictated by Machado et al.

Original: The positive and the negative of the same eigen-vector were used to generate options and were counted as separate options.
Revised: The positive and the negative of the same eigen-vector were used to generate options and were counted as separate options.

Original: The successor representations were collected using the uniformly random policy run for 5 × 10 6 steps in the environment.
Revised: The Successor representations were collected using the uniformly random policy run for 5 × 10 6 steps in the environment.

Original: The SRs were then clustered using the K-means algorithm following which a
Revised: The SRs were then clustered using the K-means algorithm following which a

Original: Q-learner learns the option policy in 1 × 10 6 steps.
Revised: Q-learner learns the option policy in 1 × 10 6 steps.

Original: Once both the Successor options and Eigen-options for an environment were discovered, they were used in an SMDP Q-learning framework to learn an option policy.
Revised: Once both the Successor options and Eigen-options for an environment were discovered, they were used in an SMDP Q-learning framework to learn an option policy.

Original: A non-uniform sampling of option to actions(1:19) was attempted instead of the uniformly random sampling for the exploratory action.
Revised: A non-uniform sampling of option to actions (1:19) was attempted instead of the uniformly random sampling for the exploratory action.

Original: The algorithms were trained for 5 × 10 5 steps(primitive actions) in the environment.
Revised: The algorithms were trained for 5 × 10 5 steps (primitive actions) in the environment.

Original: A γ = 0 .
Revised: A γ = 0 .

Original: 99 was used for this task.
Revised: 99 was used for this task.

Original: The SMDP Q-learning updates were augmented with intra-option action-Value function updates presented by Sutton et al.
Revised: The SMDP Q-learning updates were augmented with intra-option action-Value function updates presented by Sutton et al.

Original: These updates exploit the fact that intra-option policies could take the same action from the same state.
Revised: These updates exploit the fact that intra-option policies could take the same action from the same state.

Original: In such a scenario, the value functions of both options can be updated.
Revised: In such a scenario, the value functions of both options can be updated.

Original: H E XPERIMENTAL D ETAILS - D EEPMIND -L AB
Revised: H E XPERIMENTAL D ETAILS - D EEPMIND -L AB

Original: The Deepmind Lab was trained using DQN(with experience replay and target network).
Revised: The Deepmind-Lab task was trained using DQN (with experience replay and target network).

Original: Double Qlearning was also used to stabilize training.
Revised: Double Q-learning was also used to stabilize training.

Original: The value of (cid:15) was annealed from 1.0 to 0.1 in1 Million steps.
Revised: The value of (cid:15) was annealed from 1.0 to 0.1 in Million steps.

Original: A batch size of 32 was used with γ = 0 .
Revised: A batch size of 32 was used with γ = 0 .

Original: The same learning rate of 2 . 5 × 10 − 4 was used across all stages.
Revised: The same learning rate of 2 . 5 × 10 −was used across all stages.

Original: The training of the SR occurred for 5 Million steps and each option policy was trained for 2 Million steps.
Revised: The training of the SR occurred for 5 Million steps and each option policy was trained for 2 Million steps.

Original: A memory of size 100,000 was used.
Revised: A memory of size 100,000 was used.

Original: The target network was updated every 10,000 steps.
Revised: The target network was updated every 10,000 steps.

Original: RMS-prop was used as the gradient optimizer technique.
Revised: RMS-prop was used as the gradient optimizer technique.

Revised: 8 x 8 conv, 8 x 8 conv, 4 x 4 conv, 4 x 4 conv, 3 x 3 conv, 512 fully connected, 128 fully connected

Revised: Reconstruction branch 128, 128 fully connected

Revised: SR branch 64, 64, 128 fully connected

Revised: I P ERFORMANCE C URVES

Original: [Table]
Revised: We present the performance curves as a function of steps in the environment in Figure 13. These are the same curves whose area is reported in Figure 6. It is clear to see that SR-options result in better returns throughout training. The plot clearly highlights the gulf in performance between the various methods in comparison to our method SR-options.

Revised: J D ENSITY PLOT FOR O PTIONS

Revised: In order to understand the necessity of the non-uniform exploration strategy, we look at the visitation count of various states when exploring with different policies (see Figure 14.

Revised: Using only the actions result in low visitation counts at far-away states.

Revised: However, the non-uniform strategy results in well distributed visitation counts across the state space.

Revised: A possible optimization would be to explore in proportion to the size of the cluster, after selecting the option corresponding to that cluster.

Revised: This pattern in visitation counts is indicative of the fact that the non-uniform exploration strategy is likely to be more effective.

Revised: The policy is executed for primitive actions in the environment.

Revised: Another point to note is that a higher ratio seems to perform less effectively.

Revised: Hence, the options ensure that a key sequence of actions are performed to navigate to a different region in state space, which is consequently explored only if primitive actions are taken.

Revised: Sampling options in a frequent manner results in the agent spending a large fraction of its time near the termination states of these options.

Revised: We also plot the density maps for the ﬁnite horizon case (Figure 15.

Revised: The agent is allowed to explore the environment for only 100 steps after which it is reset back to its start state ([1,1] in this experiment).

Revised: The plots are visitation counts of the agent (for 100 primitive actions in the environment) averaged over 1000 runs.

Revised: K I NTRA - OPTION P OLICY VISUALIZATION

Revised: The intra-option policies are visualized for both eigen-options (Figure 16 and SR-options (Figure 17).

Revised: The green states are states, where the option terminates in a deterministic fashion.

Revised: SR-options are run for 4 sub-goals while eigen-options uses the ﬁrst 2 eigen vectors (4 options considering positive and negative version of each eigen-vectors to obtain eigen-options).

Revised: L V ARYING NUMBER OF O PTIONS

Revised: This plots (Figure 19) qualitatively demonstrate that the pre-determined number of options, changes the nature of the ﬁnally achieved sub-goals.

Revised: The sub-goals are well separated and not redundant in nature.

Revised: The method is hence expected to be fairly robust to the number of sub-goals.

Revised: 19 shows the performance of SR-options, with a varying number of sub-goals.

Revised: A small number of sub-goals may not easily reach all parts of the state space and a large number of sub-goals will result in poorer exploration (since there are more options to choose from).

Revised: Hence, we see thatsub-goals are optimal for this problem.

Revised: The plots are for the same grid-world as that in Figure 18.

Revised: The setup is identical to that used in the earlier evaluations.

Revised: After learning the options, 100 different scenarios of start and end states were used.

Revised: For each scenario the training was done for 5 × 10 5 steps (primitive actions) in the environment and the evaluation of return was done 200 times in equally spaced intervals.

Revised: Figure 19 is a result of averaging the return curves for the 100 different scenarios of start and end states.

Revised: M V ARYING TRADE - OFF BETWEEN A CTIONS AND O PTIONS IN E XPLORATION

Revised: This section varies the ratio of sampling action and options in the non-uniform exploration scheme.

Revised: The performance is optimal for a large exploration ratio of 1:999 which is not necessarily surprising, since Figure 14 indicates that this scheme covers the state-space most effectively.

Revised: The experimental conditions are same as that described in Appendix L (similar to main experiments for evaluating different methods).

Revised: N C HOICE OF POLICY FOR LEARNING SR

Revised: We varied the policies used to collect the Successor representations to visualize the nature of the sub-goals.

Revised: This policy can be understood as a prior over the state space, when deciding sub-goals.

Revised: Figure 21 highlights that a policy biased towards a region in state space (say top room in 2-room domain) results in more sub-goals present in that region in state space.

Revised: This is a result of the fact that the SRs are more detailed in these regions resulting in a ﬁne-grained clustering of that region.

Revised: Hence, if we have a policy that operates on an “important” part of the state space, SR-options automatically deﬁnes sub-goals in only these regions in state space.

Revised: O D ENSE NATURE OF INTRINSIC REWARD

Revised: While the Successor representations may not have a large magnitude in all parts of the state space, the same is not required to quickly learn a policy.

Revised: The cardinal requirement of a good intrinsic reward is to ensure that the largest reward, points the agent in the right direction, which is the case here.

Revised: For example, in Figure 22, the difference in values correspond to the reward, which although small guides the agent in the right direction.

Revised: This is a good enough signal to determine the right action to take from every state with ease.