Original: E XPLORATION IN R EINFORCEMENT L EARNING WITH D EEP C OVERING O PTIONS
Revised: E XPLORATION IN R EINFORCEMENT L EARNING WITH D EEP C OVERING O PTIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: While many option discovery methods have been proposed to accelerate exploration in reinforcement learning, they are often heuristic.
Revised: While many option discovery methods have been proposed to accelerate exploration in reinforcement learning, they are often heuristic.

Original: Recently, covering options was proposed to discover a set of options that provably reduce the upper bound of the environment’s cover time, a measure of the difﬁculty of exploration.
Revised: Recently, covering options was proposed to discover a set of options that provably reduce the upper bound of the environment’s cover time, a measure of the difﬁculty of exploration.

Original: However, they are constrained to tabular tasks and are not applicable to tasks with large or continuous state-spaces.
Revised: However, they are constrained to tabular tasks and are not applicable to tasks with large or continuous state-spaces.

Original: We introduce deep covering options , an online method that extends covering options to large state spaces, automatically discovering taskagnostic options that encourage exploration.
Revised: We introduce deep covering options , an online method that extends covering options to large state spaces, automatically discovering taskagnostic options that encourage exploration.

Original: We evaluate our method in several challenging sparse-reward domains and we show that our approach identiﬁes less explored regions of the state-space and successfully generates options to visit these regions, substantially improving both the exploration and the total accumulated reward.
Revised: We evaluate our method in several challenging sparse-reward domains and we show that our approach identiﬁes less explored regions of the state-space and successfully generates options to visit these regions, substantially improving both the exploration and the total accumulated reward.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Temporal abstraction, often formalized via the options framework (Sutton et al., 1999), has the potential to greatly improve the performance of reinforcement learning (RL) agents by representing actions at different time scales.
Revised: Temporal abstraction, often formalized via the options framework (Sutton et al., 1999), has the potential to greatly improve the performance of reinforcement learning (RL) agents by representing actions at different time scales.

Original: However, the question of which options an agent should construct, and the related question of what objective function that option construction process should be optimizing, remain open.
Revised: However, the question of which options an agent should construct, and the related question of what objective function that option construction process should be optimizing, remain open.

Original: One recent approach is to construct options that aid exploration by providing agents with more decisive behavior than the dithering common to random exploration (e.g., Menache et al., 2002; Stolle and Precup, 2002; ¸Sim¸sek and Barto, 2004; ¸Sim¸sek et al., 2005; ¸Sim¸sek and Barto, 2009;
Revised: One recent approach is to construct options that aid exploration by providing agents with more decisive behavior than the dithering common to random exploration (e.g., Menache et al., 2002; Stolle and Precup, 2002; ¸Sim¸sek and Barto, 2004; ¸Sim¸sek et al., 2005; ¸Sim¸sek and Barto, 2009;

Original: Machado et al., 2017; Eysenbach et al., 2019).
Revised: Machado et al., 2017; Eysenbach et al., 2019).

Original: The Laplacian (Chung, 1996), the matrix extracted from the graph induced by the agent’s policy and the dynamics of the environment, is often used when discovering options for exploration (e.g., Machado and Bowling, 2016;
Revised: The Laplacian (Chung, 1996), the matrix extracted from the graph induced by the agent’s policy and the dynamics of the environment, is often used when discovering options for exploration (e.g., Machado and Bowling, 2016;

Original: Machado et al., 2017; 2018;
Revised: Machado et al., 2017; 2018;

Original: Jinnai et al., 2019b).
Revised: Jinnai et al., 2019b).

Original: The options discovered with such an approach encourage agents to navigate to parts of the state space that are infrequently visited.
Revised: The options discovered with such an approach encourage agents to navigate to parts of the state space that are infrequently visited.

Original: However, the existing methods either lack a principled way of constraining the number of discovered options (e.g., Machado and Bowling, 2016;
Revised: However, the existing methods either lack a principled way of constraining the number of discovered options (e.g., Machado and Bowling, 2016;

Original: Machado et al., 2017; 2018) or are limited to the tabular setting (e.g., Jinnai et al., 2019b).
Revised: Machado et al., 2017; 2018) or are limited to the tabular setting (e.g., Jinnai et al., 2019b).

Original: In this paper we show how recent developments in eigenfunction estimation of the Laplacian (Wu et al., 2019) can be used to extend a theoretically principled approach for option discovery (Jinnai et al., 2019b) to the non-linear function approximation case.
Revised: In this paper we show how recent developments in eigenfunction estimation of the Laplacian (Wu et al., 2019) can be used to extend a theoretically principled approach for option discovery (Jinnai et al., 2019b) to the non-linear function approximation case.

Original: This new algorithm for option discovery, deep covering options , is computationally tractable and it is applicable to environments with large (or continuous) state-spaces.
Revised: This new algorithm for option discovery, deep covering options , is computationally tractable and it is applicable to environments with large (or continuous) state-spaces.

Original: Despite methods that learn representations generally being more ﬂexible, more scalable, and often leading to better performance, before this paper, covering options could not be easily combined with modern representation learning techniques.
Revised: Despite methods that learn representations generally being more ﬂexible, more scalable, and often leading to better performance, before this paper, covering options could not be easily combined with modern representation learning techniques.

Original: Deep covering options discovers a small set of options that encourage exploration by minimizing the agent’s expected cover time —the expected number of steps required to visit every state in the environment (Broder and Karlin, 1989).
Revised: Deep covering options discovers a small set of options that encourage exploration by minimizing the agent’s expected cover time —the expected number of steps required to visit every state in the environment (Broder and Karlin, 1989).

Original: Moreover, unlike most previous approaches to discovering options for exploration, it can be applied to both settings where a pretraining (unsupervised) phase is available (e.g., Eysenbach et al., 2019) and to the traditional, fully online, setting.
Revised: Moreover, unlike most previous approaches to discovering options for exploration, it can be applied to both settings where a pretraining (unsupervised) phase is available (e.g., Eysenbach et al., 2019) and to the traditional, fully online, setting.

Original: We evaluate our method, in both settings, in three different platforms to demonstrate its applicability in a wide range of domains.
Revised: We evaluate our method, in both settings, in three different platforms to demonstrate its applicability in a wide range of domains.

Original: First, we apply it to the Pinball domain (Konidaris and Barto, 2009), which has a discrete action-space and a continuous state-space.
Revised: First, we apply it to the Pinball domain (Konidaris and Barto, 2009), which has a discrete action-space and a continuous state-space.

Original: Second, we apply it to three MuJoCo control tasks (Todorov et al., 2012), which are continuous state- and action-space domains.
Revised: Second, we apply it to three MuJoCo control tasks (Todorov et al., 2012), which are continuous state- and action-space domains.

Original: In all of these domains, our method improves over the baseline.
Revised: In all of these domains, our method improves over the baseline.

Original: Finally, we perform a qualitative analysis of our method in three Atari 2600 games (Bellemare et al., 2013) to demonstrate its potential in domains with very large state-spaces.
Revised: Finally, we perform a qualitative analysis of our method in three Atari 2600 games (Bellemare et al., 2013) to demonstrate its potential in domains with very large state-spaces.

Original: Deep covering options successfully ﬁnds under-explored regions of the state space and builds options to target those regions.
Revised: Deep covering options successfully ﬁnds under-explored regions of the state space and builds options to target those regions.

Original:  B ACKGROUND AND R ELATED W ORK
Revised:  B ACKGROUND AND R ELATED W ORK

Original: We assume the standard reinforcement learning setting (Sutton and Barto, 1998), where the environment is modeled as a Markov Decision Process (MDP), ( S , A , T, R, γ ) , where S is the set of states,
Revised: We assume the standard reinforcement learning setting (Sutton and Barto, 1998), where the environment is modeled as a Markov Decision Process (MDP), ( S , A , T, R, γ ) , where S is the set of states,

Original: A is the set of actions, T : S ×
Revised: A is the set of actions, T : S ×

Original: A × S → [0 , 1] is the state transition function, R : S ×
Revised: A × S → [0 , 1] is the state transition function, R : S ×

Original: A → R is the reward function, and≤ γ ≤ 1 is the discount factor.
Revised: A → R is the reward function, and≤ γ ≤ 1 is the discount factor.

Original: We use the options framework (Sutton et al., 1999) to represent temporally extended actions.
Revised: We use the options framework (Sutton et al., 1999) to represent temporally extended actions.

Original: It deﬁnes an option as a triple ( I , π, β ) , where I ⊆ S is the set of states in which the option can initiate, π :
Revised: It deﬁnes an option as a triple ( I , π, β ) , where I ⊆ S is the set of states in which the option can initiate, π :

Original: S → Pr( A ) is the policy the agent follows when that option is being executed, and β : S → [0 , 1] , is the termination condition.
Revised: S → Pr( A ) is the policy the agent follows when that option is being executed, and β : S → [0 , 1] , is the termination condition.

Original: We refer to a set of states in which β ( s ) = 1 as a termination set.
Revised: We refer to a set of states in which β ( s ) = 1 as a termination set.

Original: 2. R ELATED W ORK
Revised: 2. R ELATED W ORK

Original: Many option discovery algorithms are based on the reward signals generated by the environment and are thus task dependent.
Revised: Many option discovery algorithms are based on the reward signals generated by the environment and are thus task dependent.

Original: These methods often decompose the trajectories reaching the rewarding states into options.
Revised: These methods often decompose the trajectories reaching the rewarding states into options.

Original: Several papers have proposed generating options from trajectories reaching these rewarding states (e.g., McGovern and Barto, 2001; Menache et al., 2002; Konidaris and Barto, 2009), while other approaches use the observed rewards to generate options with gradient descent (e.g.,
Revised: Several papers have proposed generating options from trajectories reaching these rewarding states (e.g., McGovern and Barto, 2001; Menache et al., 2002; Konidaris and Barto, 2009), while other approaches use the observed rewards to generate options with gradient descent (e.g.,

Original: Mankowitz et al., 2016; Bacon et al., 2017;
Revised: Mankowitz et al., 2016; Bacon et al., 2017;

Original: Harb et al., 2018; Tiwari and Thomas, 2019).
Revised: Harb et al., 2018; Tiwari and Thomas, 2019).

Original: These approaches are often ineffective in sparse reward problems, where only a few state-action pairs lead to a positive reward.
Revised: These approaches are often ineffective in sparse reward problems, where only a few state-action pairs lead to a positive reward.

Original: Fewer papers have tackled the problem of option discovery for exploration without using reward signals.
Revised: Fewer papers have tackled the problem of option discovery for exploration without using reward signals.

Original: Eysenbach et al.
Revised: Eysenbach et al.

Original: (2019) proposed to generate options maximizing an information theoretic objective so that each option generates diverse behavior.
Revised: (2019) proposed to generate options maximizing an information theoretic objective so that each option generates diverse behavior.

Original: While many option discovery methods are limited to discrete state and action space tasks, their method can generate options that solve many continuous control tasks, even when ignoring the environment’s reward function.
Revised: While many option discovery methods are limited to discrete state and action space tasks, their method can generate options that solve many continuous control tasks, even when ignoring the environment’s reward function.

Original: Machado et al. Machado et al.
Revised: Machado et al. Machado et al.

Original: (2017; 2018) proposed eigenoptions, a method to generate options using the Laplacian eigenvectors (Chung, 1996).
Revised: (2017; 2018) proposed eigenoptions, a method to generate options using the Laplacian eigenvectors (Chung, 1996).

Original: Their approach is similar to covering options but requires the set of options to be orthogonal to each other and introduces a prohibitively large number of options at each iteration.
Revised: Their approach is similar to covering options but requires the set of options to be orthogonal to each other and introduces a prohibitively large number of options at each iteration.

Original: Several papers have proposed identifying subgoal states without reward information through graph concepts such as clustering (Menache et al., 2002; ¸Sim¸sek et al., 2005), visitation statistics (¸Sim¸sek and Barto, 2004; Stolle and Precup, 2002), and betweenness centrality (¸Sim¸sek and Barto, 2009).
Revised: Several papers have proposed identifying subgoal states without reward information through graph concepts such as clustering (Menache et al., 2002; ¸Sim¸sek et al., 2005), visitation statistics (¸Sim¸sek and Barto, 2004; Stolle and Precup, 2002), and betweenness centrality (¸Sim¸sek and Barto, 2009).

Original: As they use graph algorithms to discover subgoals, their scope is often limited to tabular domains.
Revised: As they use graph algorithms to discover subgoals, their scope is often limited to tabular domains.

Original: 2. C OVERING O PTIONS
Revised: 2. C OVERING O PTIONS

Original: Covering options (Jinnai et al., 2019b) is an approach that minimizes the expected cover time of a uniformly random policy by augmenting the agent’s action set with options obtained from the eigenvector associated with the second smallest eigenvalue of the Laplacian.
Revised: Covering options (Jinnai et al., 2019b) is an approach that minimizes the expected cover time of a uniformly random policy by augmenting the agent’s action set with options obtained from the eigenvector associated with the second smallest eigenvalue of the Laplacian.

Original: Covering options can be seen as increasing the likelihood that a random walk is going to lead to a rewarding state since the expected cover time is the time required for a random walk to visit all the vertices in a graph (Broder and Karlin, 1989).
Revised: Covering options can be seen as increasing the likelihood that a random walk is going to lead to a rewarding state since the expected cover time is the time required for a random walk to visit all the vertices in a graph (Broder and Karlin, 1989).

Original: Covering options achieves such an objective by minimizing the upper bound of the expected cover time, E [ C ( G )] , which is given by the second smallest eigenvalue of the normalized Laplacian, λ 2 , also known as the algebraic connectivity (Fiedler, 1973):
Revised: Covering options achieves such an objective by minimizing the upper bound of the expected cover time, E [ C ( G )] , which is given by the second smallest eigenvalue of the normalized Laplacian, λ 2 , also known as the algebraic connectivity (Fiedler, 1973):

Original: [Equation]
Revised: [Equation]

Original: where n is the number of vertices of the graph.
Revised: where n is the number of vertices of the graph.

Original: Equation 1 shows that the larger the algebraic connectivity, the smaller the upper bound of the expected cover time.
Revised: Equation 1 shows that the larger the algebraic connectivity, the smaller the upper bound of the expected cover time.

Original: Intuitively, algebraic connectivity represents how densely the graph is connected.
Revised: Intuitively, algebraic connectivity represents how densely the graph is connected.

Original: The eigenvector f corresponding to λ 2 is an embedding of a graph to a one-dimensional interval where nodesThe second eigenvector captures the connectivity of the graph, so distances reﬂect path lengths in the graph; the pair of nodes with the maximum and minimum values are the farthest apart. connected by an edge tend to be placed nearby (see Figure 1).
Revised: The eigenvector f corresponding to λ 2 is an embedding of a graph to a one-dimensional interval where nodes connected by an edge tend to be placed nearby (see Figure 1, adapted from Jinnai et al., 2019b).

Original: A pair of nodes with the maximum and minimum value in f are the most distant nodes in the embedding space.
Revised: A pair of nodes with the maximum and minimum value in f are the most distant nodes in the embedding space.

Original: Connecting these two nodes greedily maximizes the algebraic connectivity to a ﬁrst order approximation (Ghosh and Boyd, 2006).
Revised: Connecting these two nodes greedily maximizes the algebraic connectivity to a ﬁrst order approximation (Ghosh and Boyd, 2006).

Original: Covering options works as follows:
Revised: Covering options works as follows:

Original: Compute the second smallest eigenvalue and the corresponding eigenvector f of the Laplacian exactly by solving the
Revised: Compute the second smallest eigenvalue and the corresponding eigenvector f of the Laplacian exactly by solving the

Original: following
Revised: following

Original: constraint optimization problem:
Revised: constraint optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where A is the adjacency matrix of the state-space graph where the entry at ( s, s (cid:48) ) is 1 if s and s (cid:48) are adjacent and 0 otherwise.
Revised: where A is the adjacency matrix of the state-space graph where the entry at ( s, s (cid:48) ) is 1 if s and s (cid:48) are adjacent and 0 otherwise.

Original: Let v i and v j be the state with largest and smallest value in the eigenvector respectively.
Revised: Let v i and v j be the state with largest and smallest value in the eigenvector respectively.

Original: Generate two options; one with I = { v i } and β = { v j } and the other one with I = { v j } and β = { v i } .
Revised: Generate two options; one with I = { v i } and β = { v j } and the other one with I = { v j } and β = { v i } .

Original: Each option policy is the optimal path from the initial state to the termination state.
Revised: Each option policy is the optimal path from the initial state to the termination state.

Original: Set G ← G ∪ { ( v i , v j ) } and repeat the process until the number of options reaches a threshold.
Revised: Set G ← G ∪ { ( v i , v j ) } and repeat the process until the number of options reaches a threshold.

Original: While this method is an efﬁcient algorithm with performance guarantees, it is limited to small discrete
Revised: While this method is an efﬁcient algorithm with performance guarantees, it is limited to small discrete

Original: MDPs as it requires a state-space graph.
Revised: MDPs as it requires a state-space graph.

Original: Moreover, explicitly computing the matrix that encodes the environment’s adjacency matrix is unrealistic beyond small problems.
Revised: Moreover, explicitly computing the matrix that encodes the environment’s adjacency matrix is unrealistic beyond small problems.

Original: Finally, the method is constrained to point options where both the initiation and termination sets consist of a single state (Jinnai et al., 2019a).
Revised: Finally, the method is constrained to point options where both the initiation and termination sets consist of a single state (Jinnai et al., 2019a).

Original: Options generated by this method are therefore only executable at a single state.
Revised: Options generated by this method are therefore only executable at a single state.

Original: This is not useful for tasks with large (or continuous) state-spaces as the probability of visiting the state in the initiation set of the option tends to zero.
Revised: This is not useful for tasks with large (or continuous) state-spaces as the probability of visiting the state in the initiation set of the option tends to zero.

Original: Even if the agent visits the state in the option’s initiation set and starts following the corresponding option’s policy, the probability of reaching the state in the termination set is also small (see Figure 2).
Revised: Even if the agent visits the state in the option’s initiation set and starts following the corresponding option’s policy, the probability of reaching the state in the termination set is also small (see Figure 2).

Original: In the next section we introduce an approach that addresses these limitations.
Revised: In the next section we introduce an approach that addresses these limitations.

Original:  D EEP C OVERING O PTIONS
Revised:  D EEP C OVERING O PTIONS

Original: We propose deep covering options , a new algorithm that ﬁnds options that speed-up exploration in domains with large (or continuous) state-spaces.
Revised: We propose deep covering options , a new algorithm that ﬁnds options that speed-up exploration in domains with large (or continuous) state-spaces.

Original: It directly seeks to optimize an objective for exploration.
Revised: It directly seeks to optimize an objective for exploration.

Original: If the objective function is optimized, the options generated by the algorithm greedily maximize the algebraic connectivity of the underlying state-space graph to a ﬁrst order approximation (Ghosh and Boyd, 2006), which in turn minimize the upper bound on the expected cover time.
Revised: If the objective function is optimized, the options generated by the algorithm greedily maximize the algebraic connectivity of the underlying state-space graph to a ﬁrst order approximation (Ghosh and Boyd, 2006), which in turn minimize the upper bound on the expected cover time.

Original: Deep covering options consists of four steps (see Algorithm 1):
Revised: Deep covering options consists of four steps (see Algorithm 1):

Original: an
Revised: an

Original: of the Laplacian of the state-space
Revised: of the Laplacian of the state-space

Original: in the state-space using the eigenfunctions
Revised: in the state-space using the eigenfunctions

Original: line
Revised: line

Original: , 3. set the under-explored region as the termination set and set the compliment of it as the initiation set (line 4,
Revised: , 3. set the under-explored region as the termination set and set the compliment of it as the initiation set (line 4,

Original: , 4. train a policy of the option using the pseudo-reward induced by the eigenfunctions (line 6)
Revised: , 4. train a policy of the option using the pseudo-reward induced by the eigenfunctions (line 6)

Original: There are two problems in Equation 2 that prevent its applicability to non-tabular domains.
Revised: There are two problems in Equation 2 that prevent its applicability to non-tabular domains.

Original: First, the equation requires the adjacency matrix A as input.
Revised: First, the equation requires the adjacency matrix A as input.

Original: Second, a constrained optimization problem is hard to solve using gradient-based methods.
Revised: Second, a constrained optimization problem is hard to solve using gradient-based methods.

Original: We address these issues by approximating the computation of the Laplacian with the following objective (Wu et al., 2019, Equation 6):
Revised: We address these issues by approximating the computation of the Laplacian with the following objective (Wu et al., 2019, Equation 6):

Original: [Equation]
Revised: [Equation]

Original: where H is the set of sampled state-transitions, ρ is a distribution of states in the dataset ( ρ ( s ) is the number of occurrence of s in H divided by the size of H ), η is the Lagrange multiplier, and δ jk isif j (cid:54) =
Revised: where H is the set of sampled state-transitions, ρ is a distribution of states in the dataset ( ρ ( s ) is the number of occurrence of s in H divided by the size of H ), η is the Lagrange multiplier, and δ jk isif j (cid:54) =

Original: k and 0 otherwise.
Revised: k and 0 otherwise.

Original: Such an expression, inspired by spectral graph drawing theory, uses the repulsive term (the summation multiplied by η ) to ensure the functions f 1 , ..., f d are orthogonal to each other.
Revised: Such an expression, inspired by spectral graph drawing theory, uses the repulsive term (the summation multiplied by η ) to ensure the functions f 1 , ..., f d are orthogonal to each other.

Original: Unlike G , ˜ G is a constraint-free objective to compute the eigenfunction, only requiring trajectories instead of the state-space graph.
Revised: Unlike G , ˜ G is a constraint-free objective to compute the eigenfunction, only requiring trajectories instead of the state-space graph.

Original: As we only require the second eigenfunction (unlike eigenoptions), we can simplify the objective function to take only two arguments:
Revised: As we only require the second eigenfunction (unlike eigenoptions), we can simplify the objective function to take only two arguments:

Original: [Equation]
Revised: [Equation]

Original: Assume G ( f 1 ) ≤
Revised: Assume G ( f 1 ) ≤

Original: G ( f 2 ) without loss of generality.
Revised: G ( f 2 ) without loss of generality.

Original: G ( f 1 ) = 

Revised: G ( f 1 ) =

Original: and f 1 is a constant function because the ﬁrst eigenvalue of the Laplacian matrix is zero.
Revised: and f 1 is a constant function because the ﬁrst eigenvalue of the Laplacian matrix is zero.

Original: To simplify the equation, we assume f 1 =without loss of generality.
Revised: To simplify the equation, we assume f 1 =without loss of generality.

Original: Then:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: Deep covering options compute the second eigenfunction f by minimizing ˜ G ( f ) instead of G ( f ) (see Algorithm 1).
Revised: Deep covering options compute the second eigenfunction f by minimizing ˜ G ( f ) instead of G ( f ) (see Algorithm 1).

Original: Our objective function only needs sampled state-transitions H instead of a complete state-space graph.
Revised: Our objective function only needs sampled state-transitions H instead of a complete state-space graph.

Original: As it is an unconstrained optimization problem, we can optimize by simple gradient-based methods.
Revised: As it is an unconstrained optimization problem, we can optimize by simple gradient-based methods.

Original: The objective function is essentially the same as the objective function of covering options which has theoretical guarantee on the expected cover time but computed approximately so that it scales to large or inﬁnite state-space domains.
Revised: The objective function is essentially the same as the objective function of covering options which has theoretical guarantee on the expected cover time but computed approximately so that it scales to large or inﬁnite state-space domains.

Original: While covering options is constrained to options with the initiation set consisting of a single state, we set the termination set as a set of states with f value smaller than its k -th percentile.
Revised: While covering options is constrained to options with the initiation set consisting of a single state, we set the termination set as a set of states with f value smaller than its k -th percentile.

Original: As proposed by
Revised: As proposed by

Original: Machado et al.
Revised: Machado et al.

Original: (2017; 2018), we deﬁne the initiation set to be the complement of the termination set.
Revised: (2017; 2018), we deﬁne the initiation set to be the complement of the termination set.

Original: We train the option policy off-policy, maximizing the total pseudo-reward r o = f ( s ) − f ( s (cid:48) ) so that it learns to reach the termination set (i.e., the set of states with f ( s ) < β (cid:48) ).
Revised: We train the option policy off-policy, maximizing the total pseudo-reward r o = f ( s ) − f ( s (cid:48) ) so that it learns to reach the termination set (i.e., the set of states with f ( s ) < β (cid:48) ).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate our method in both the online setting and the setting in which a pretraining phase is available.
Revised: We evaluate our method in both the online setting and the setting in which a pretraining phase is available.

Original: We use three different platforms: the Pinball domain (Konidaris and Barto, 2009), three
Revised: We use three different platforms: the Pinball domain (Konidaris and Barto, 2009), three

Original: MuJoCo control tasks (Todorov et al., 2012), and three Atari games (Bellemare et al., 2013).
Revised: MuJoCo control tasks (Todorov et al., 2012), and three Atari games (Bellemare et al., 2013).

Original: See the
Revised: See the

Original: Appendix for the experimental details.
Revised: Appendix for the experimental details.

Original: 4. O FFLINE O PTION D ISCOVERY
Revised: 4. O FFLINE O PTION D ISCOVERY

Original: We ﬁrst consider the setting in which the agent collects samples in the environment for a given number of time steps before being given a reward signal to maximize.
Revised: We ﬁrst consider the setting in which the agent collects samples in the environment for a given number of time steps before being given a reward signal to maximize.

Original: Pinball In the Pinball domain the goal is to maneuver a small ball from a start state to a goal state (Figure 3a; Konidaris and Barto, 2009).
Revised: Pinball In the Pinball domain the goal is to maneuver a small ball from a start state to a goal state (Figure 3a; Konidaris and Barto, 2009).

Original: The state-space consists of four continuous variables, the coordinates of the ball position ( x, y ) and the velocity ( ˙ x, ˙ y ) .
Revised: The state-space consists of four continuous variables, the coordinates of the ball position ( x, y ) and the velocity ( ˙ x, ˙ y ) .

Original: There are ﬁve primitive actions: incrementing or decrementing ˙ x or ˙ y by a small amount or leaving them unchanged.
Revised: There are ﬁve primitive actions: incrementing or decrementing ˙ x or ˙ y by a small amount or leaving them unchanged.

Original: The ball bounces on colliding with obstacles.
Revised: The ball bounces on colliding with obstacles.

Original: In order to reach the goal (red cross) from the initial position (purple circle), the agent must get through one of the narrow passages while taking the bounce into consideration.
Revised: In order to reach the goal (red cross) from the initial position (purple circle), the agent must get through one of the narrow passages while taking the bounce into consideration.

Original: The agent receives a reward of 10 upon arrival at the goal and of -0.001 in each other time step.
Revised: The agent receives a reward of 10 upon arrival at the goal and of -0.001 in each other time step.

Original: The start state is ﬁxed throughout the training.
Revised: The start state is ﬁxed throughout the training.

Original: To generate an option, we sampled 100 trajectories of 1000 time steps in which the agent selects between the available actions and options uniformly at random.
Revised: To generate an option, we sampled 100 trajectories of 1000 time steps in which the agent selects between the available actions and options uniformly at random.

Original: We trained a neural network to learn the eigenfunction by minimizing ˜ G using the sampled state-transitions (see Equation 5).
Revised: We trained a neural network to learn the eigenfunction by minimizing ˜ G using the sampled state-transitions (see Equation 5).

Original: We set the threshold percentile k = 30 , and used Q-learning (Watkins and Dayan, 1992) with Fourier basis linear function approximation (Konidaris et al., 2011) to train the option policy off-policy using the sampled trajectories but using the pseudo-reward r o (see Algorithm 1).
Revised: We set the threshold percentile k = 30 , and used Q-learning (Watkins and Dayan, 1992) with Fourier basis linear function approximation (Konidaris et al., 2011) to train the option policy off-policy using the sampled trajectories but using the pseudo-reward r o (see Algorithm 1).

Original: We repeat this process with the generated option added to the agent’s action set.
Revised: We repeat this process with the generated option added to the agent’s action set.

Original: We evaluate the performance of the agent with access to the discovered options to evaluate the claim that these options do indeed allow the agent to collect more rewards by making it more capable of navigating through the state space.
Revised: We evaluate the performance of the agent with access to the discovered options to evaluate the claim that these options do indeed allow the agent to collect more rewards by making it more capable of navigating through the state space.

Original: The agent has access to both these computed options and primitive actions, and uses Q-learning with the Fourier basis to train the high-level policy.
Revised: The agent has access to both these computed options and primitive actions, and uses Q-learning with the Fourier basis to train the high-level policy.

Original: Figure 3e depicts the agent’s performance with a varying number of options.
Revised: Figure 3e depicts the agent’s performance with a varying number of options.

Original: The proposed algorithm signiﬁcantly outperforms the ﬂat baseline.
Revised: The proposed algorithm signiﬁcantly outperforms the ﬂat baseline.

Original: We also evaluated the performance of ﬂat Q-learning pretrained with reward signals for the same number of episodes the hierarchical agents were (basepretrained).
Revised: We also evaluated the performance of ﬂat Q-learning pretrained with reward signals for the same number of episodes the hierarchical agents were (basepretrained).

Original: While the options are generated without reward information, the performance of the agent with the option set is close to the performance of the agent trained with reward information, showing that such an approach does not hinder performance even in a single task setting.
Revised: While the options are generated without reward information, the performance of the agent with the option set is close to the performance of the agent trained with reward information, showing that such an approach does not hinder performance even in a single task setting.

Original: The termination set generated by deep covering options tend to be larger than options which seek to minimize the size of the termination set (e.g. Harutyunyan et al., 2019).
Revised: The termination set generated by deep covering options tend to be larger than options which seek to minimize the size of the termination set (e.g. Harutyunyan et al., 2019).

Original: The results indicate that interpretable options are not necessarily efﬁcient for reducing cover time.
Revised: The results indicate that interpretable options are not necessarily efﬁcient for reducing cover time.

Original: This is a known behavior of option-discovery algorithms based on spectral methods such as eigenoptions (Machado et al., 2017).
Revised: This is a known behavior of option-discovery algorithms based on spectral methods such as eigenoptions (Machado et al., 2017).

Original: We also compared our approach to Diversity Is All You Need (DIAYN)
Revised: We also compared our approach to Diversity Is All You Need (DIAYN)

Original: (Eysenbach et al., 2019).
Revised: (Eysenbach et al., 2019).

Original: Like our method, DIAYN was recently proposed to generate exploratory options without using reward signals.
Revised: Like our method, DIAYN was recently proposed to generate exploratory options without using reward signals.

Original: While many option discovery methods are limited to discrete state-space tasks, DIAYN can operate in continuous control tasks.
Revised: While many option discovery methods are limited to discrete state-space tasks, DIAYN can operate in continuous control tasks.

Original: We trained DIAYN for the same length of pretraining steps ( episodes each one being 1000 steps long) to generate a set of options.
Revised: We trained DIAYN for the same length of pretraining steps ( episodes each one being 1000 steps long) to generate a set of options.

Original: See the appendix for details of the agent.
Revised: See the appendix for details of the agent.

Original: While Eysenbach et al.
Revised: While Eysenbach et al.

Original: (2019, Section 5.1) assumed that one can pick the option (i.e. skill) with highest reward for the task and trained the agent starting from that single option, we use the more realistic assumption that the agent has no prior information of which option is most useful for a given task.
Revised: (2019, Section 5.1) assumed that one can pick the option (i.e. skill) with highest reward for the task and trained the agent starting from that single option, we use the more realistic assumption that the agent has no prior information of which option is most useful for a given task.

Original: Based on this assumption, we evaluated an agent equipped with all the generated options
Revised: Based on this assumption, we evaluated an agent equipped with all the generated options

Original: Domains
Revised: Domains

Original: [Figure]
Revised: [Figure]

Original: and the primitive actions so that the agent must learn which option is most useful for the given task by itself.
Revised: and the primitive actions so that the agent must learn which option is most useful for the given task by itself.

Original: We used Q-learning with Fourier basis to train the high-level policy.
Revised: We used Q-learning with Fourier basis to train the high-level policy.

Original: As the termination condition of DIAYN is not deﬁned by Eysenbach et al.
Revised: As the termination condition of DIAYN is not deﬁned by Eysenbach et al.

Original: (2019), we tested the termination probability of 0.0, 0.01, 0.1, and 0.5 for any state, and picked 0.1 as it performed the best.
Revised: (2019), we tested the termination probability of 0.0, 0.01, 0.1, and 0.5 for any state, and picked 0.1 as it performed the best.

Original: We set the initiation set to be the whole state space, and evaluated the performance of DIAYN for up to three options.
Revised: We set the initiation set to be the whole state space, and evaluated the performance of DIAYN for up to three options.

Original: DIAYN outperforms the baseline that consists of only primitive actions.
Revised: DIAYN outperforms the baseline that consists of only primitive actions.

Original: While DIAYN generates a diverse set of options by maximizing the mutual information between states and options, it does not consider state connectivity.
Revised: While DIAYN generates a diverse set of options by maximizing the mutual information between states and options, it does not consider state connectivity.

Original: As our algorithm takes into account the connectivity of the states to generate diverse set of options, it successfully ﬁnds an option which leads to a state close (in terms of the number of steps to reach) to the goal state with high probability, resulting in better performance than DIAYN.
Revised: As our algorithm takes into account the connectivity of the states to generate diverse set of options, it successfully ﬁnds an option which leads to a state close (in terms of the number of steps to reach) to the goal state with high probability, resulting in better performance than DIAYN.

Original: MuJoCo Next, we evaluated our method in three simulated continuous control tasks introduced by Nachum et al.
Revised: MuJoCo Next, we evaluated our method in three simulated continuous control tasks introduced by Nachum et al.

Original: (2018): PointFall, PointMaze, and PointPush.
Revised: (2018): PointFall, PointMaze, and PointPush.

Original: The agent receives a reward signal of value 8000 when it reaches the goal and a reward signal of value − (L2 distance to the goal) /
Revised: The agent receives a reward signal of value 8000 when it reaches the goal and a reward signal of value − (L2 distance to the goal) /

Original: (maximum possible L2 distance to the goal) otherwise.
Revised: (maximum possible L2 distance to the goal) otherwise.

Original: The agent cannot reach to goal state just by maximizing the immediate reward given by the L2 distance to the goal (see Figure 3).
Revised: The agent cannot reach to goal state just by maximizing the immediate reward given by the L2 distance to the goal (see Figure 3).

Original: PointFall is difﬁcult for a plain agent because if it just follows the immediate reward it falls off the cliff and can never reach the goal whereas in PointMaze and PointPush are relatively easy for a plain agent as it can eventually reach the goal.
Revised: PointFall is difﬁcult for a plain agent because if it just follows the immediate reward it falls off the cliff and can never reach the goal whereas in PointMaze and PointPush are relatively easy for a plain agent as it can eventually reach the goal.

Original: The start state is ﬁxed to the same position throughout the training but the initial rotation of the agent is set randomly.
Revised: The start state is ﬁxed to the same position throughout the training but the initial rotation of the agent is set randomly.

Original: We sampled 200 episodes of length 2000 with a uniform random policy to generate each option.
Revised: We sampled 200 episodes of length 2000 with a uniform random policy to generate each option.

Original: We trained the option’s policy using deep deterministic policy gradient (DDPG Lillicrap et al., 2016), used the same hyperparameters as Wu et al.
Revised: We trained the option’s policy using deep deterministic policy gradient (DDPG Lillicrap et al., 2016), used the same hyperparameters as Wu et al.

Original: (2019) for DDPG and for learning the eigenfunction f (Wu et al., 2019,
Revised: (2019) for DDPG and for learning the eigenfunction f (Wu et al., 2019,

Original: Appendix D2.2), and set the threshold percentile k = 10 .
Revised: Appendix D2.2), and set the threshold percentile k = 10 .

Original: The high-level policy chooses options with Double Deep Q-learning (van Hasselt et al., 2016).
Revised: The high-level policy chooses options with Double Deep Q-learning (van Hasselt et al., 2016).

Original: We train the agents for 100 episodes, each 2000 time steps long.
Revised: We train the agents for 100 episodes, each 2000 time steps long.

Original: Figure 3f, 3g, and 3h show the performance with varying number of options.
Revised: Figure 3f, 3g, and 3h show the performance with varying number of options.

Original: While the performance improvement is small in PointPush and PointMaze, where even a ﬂat agent can easily reach the goal, it is signiﬁcantly improved in PointFall, which is hard to solve without an efﬁcient exploration strategy.
Revised: While the performance improvement is small in PointPush and PointMaze, where even a ﬂat agent can easily reach the goal, it is signiﬁcantly improved in PointFall, which is hard to solve without an efﬁcient exploration strategy.

Original: Our method sometimes even outperforms the agent pretrained with reward signal available (base-pretrained).
Revised: Our method sometimes even outperforms the agent pretrained with reward signal available (base-pretrained).

Original: We trained DIAYN for continuous control tasks too, but it did not outperform the baseline in PointFall,
Revised: We trained DIAYN for continuous control tasks too, but it did not outperform the baseline in PointFall,

Original: PointMaze, and PointPush.
Revised: PointMaze, and PointPush.

Original: See the Appendix for experimental details.
Revised: See the Appendix for experimental details.

Original: 4. O NLINE O PTION G ENERATION
Revised: 4. O NLINE O PTION G ENERATION

Original: In the previous section we evaluated option discovery methods assuming that the agent can collect samples by interacting with the environment prior to solving the task itself.
Revised: In the previous section we evaluated option discovery methods assuming that the agent can collect samples by interacting with the environment prior to solving the task itself.

Original: We now evaluate the proposed algorithm in the online setting, where the agent generates options using trajectories sampled during the learning phase.
Revised: We now evaluate the proposed algorithm in the online setting, where the agent generates options using trajectories sampled during the learning phase.

Original: At the beginning of training the agent only has access to primitive actions; it then generates one new option every 2000 time steps using the observed data until the number of options reaches a pre-speciﬁed threshold.
Revised: At the beginning of training the agent only has access to primitive actions; it then generates one new option every 2000 time steps using the observed data until the number of options reaches a pre-speciﬁed threshold.

Original: The option policy is trained off-policy using the trajectories sampled while training.
Revised: The option policy is trained off-policy using the trajectories sampled while training.

Original: Thus, our method does not require any extra samples.
Revised: Thus, our method does not require any extra samples.

Original: We use the same learning algorithms and hyperparameters as the ofﬂine experiments.
Revised: We use the same learning algorithms and hyperparameters as the ofﬂine experiments.

Original: These experiments aim to evaluate whether the cost of learning the options when a task is given is prohibitive.
Revised: These experiments aim to evaluate whether the cost of learning the options when a task is given is prohibitive.

Original: We train the agents for 1000 episodes, each 500 time steps long for Pinball.
Revised: We train the agents for 1000 episodes, each 500 time steps long for Pinball.

Original: In the continuous control tasks we train the agents for 200 episodes, each 1000 time steps long.
Revised: In the continuous control tasks we train the agents for 200 episodes, each 1000 time steps long.

Original: Figure 3i, 3j, 3k, and 3l (right most column) depict the agent’s performance with a varying maximum number of options.
Revised: Figure 3i, 3j, 3k, and 3l (right most column) depict the agent’s performance with a varying maximum number of options.

Original: Overall, the proposed algorithm signiﬁcantly improved performance compared to the baseline in most tasks (i.e., Pinball, PointMaze, and PointFall).
Revised: Overall, the proposed algorithm signiﬁcantly improved performance compared to the baseline in most tasks (i.e., Pinball, PointMaze, and PointFall).

Original: As in the previous section, we did not see a major improvement in PointPush where the agent can easily discover near-optimal policies using only primitive actions.
Revised: As in the previous section, we did not see a major improvement in PointPush where the agent can easily discover near-optimal policies using only primitive actions.

Original: These results suggest that the proposed method is not only useful for pretraining, but can also discover useful options during training and successfully speed up learning without additional samples.
Revised: These results suggest that the proposed method is not only useful for pretraining, but can also discover useful options during training and successfully speed up learning without additional samples.

Original: [Figure]
Revised: [Figure]

Original: 4. Q UALITATIVE E VALUATION
Revised: 4. Q UALITATIVE E VALUATION

Original: We now show how the options discovered in the pretraining phase (see Section 4.1) improve the agent’s exploration capabilities.
Revised: We now show how the options discovered in the pretraining phase (see Section 4.1) improve the agent’s exploration capabilities.

Original: Figure 4a depicts the ( x, y ) positions visited by the agent in the 10 trajectories generated by a random walk when using only primitive actions.
Revised: Figure 4a depicts the ( x, y ) positions visited by the agent in the 10 trajectories generated by a random walk when using only primitive actions.

Original: Note that the agent rarely gets through the corridor.
Revised: Note that the agent rarely gets through the corridor.

Original: Figure 4b visualizes the termination set and one of the trajectories generated by the ﬁrst option discovered by the algorithm.
Revised: Figure 4b visualizes the termination set and one of the trajectories generated by the ﬁrst option discovered by the algorithm.

Original: The shaded region indicates the option termination set.
Revised: The shaded region indicates the option termination set.

Original: Notice that the algorithm successfully discovers the region under-explored by the agent (Figure 4a).
Revised: Notice that the algorithm successfully discovers the region under-explored by the agent (Figure 4a).

Original: Figure 4c shows the ( x, y ) positions of the states visited by 10 trajectories in total, with 5 trajectories only using primitive actions and 5 trajectories with the ﬁrst option available to the agent.
Revised: Figure 4c shows the ( x, y ) positions of the states visited by 10 trajectories in total, with 5 trajectories only using primitive actions and 5 trajectories with the ﬁrst option available to the agent.

Original: The agent now consistently gets through the narrow passages.
Revised: The agent now consistently gets through the narrow passages.

Original: Figure 4d shows one of the trajectories generated by the second discovered option.
Revised: Figure 4d shows one of the trajectories generated by the second discovered option.

Original: The same process can be observed when a second and third options are added to the action set.
Revised: The same process can be observed when a second and third options are added to the action set.

Original: They keep identifying under-explored regions of the state space and further narrow down the termination set to visit these regions (Figures 4d).
Revised: They keep identifying under-explored regions of the state space and further narrow down the termination set to visit these regions (Figures 4d).

Original: These results suggest that the proposed method successfully extends the frontier of the exploration by discovering options incrementally.
Revised: These results suggest that the proposed method successfully extends the frontier of the exploration by discovering options incrementally.

Original: The same intuition holds for continuous control tasks.
Revised: The same intuition holds for continuous control tasks.

Original: This can be seen in Figures 4e and 4f.
Revised: This can be seen in Figures 4e and 4f.

Original: Figure 4e shows the states visited by 10 trajectories generated by a random walk using primitive actions in
Revised: Figure 4e shows the states visited by 10 trajectories generated by a random walk using primitive actions in

Original: PointMaze; the agent does not deviate far from its start state.
Revised: PointMaze; the agent does not deviate far from its start state.

Original: Figure 4f shows the state visited by 10 trajectories in total, 5 trajectories with primitive actions and 5 with the ﬁrst option available to the agent; the agent is now able to explore further along the corridor.
Revised: Figure 4f shows the state visited by 10 trajectories in total, 5 trajectories with primitive actions and 5 with the ﬁrst option available to the agent; the agent is now able to explore further along the corridor.

Original: By incrementally discovering options our method generates options to navigate through the maze without any reward information.
Revised: By incrementally discovering options our method generates options to navigate through the maze without any reward information.

Original: This becomes evident in Figure 4g, which depicts the trajectory followed by the ﬁrst (green), second (yellow), and third (purple) options.
Revised: This becomes evident in Figure 4g, which depicts the trajectory followed by the ﬁrst (green), second (yellow), and third (purple) options.

Original: Each option explores more deeply into the state space.
Revised: Each option explores more deeply into the state space.

Original: To demonstrate the potential of the proposed method in a domain with a very large discrete state-space, we visualize the termination set of options generated in three Atari games from the Arcade Learning
Revised: To demonstrate the potential of the proposed method in a domain with a very large discrete state-space, we visualize the termination set of options generated in three Atari games from the Arcade Learning

Original: Environment (Bellemare et al., 2013): Montezuma’s Revenge, MsPacman, and Amidar.
Revised: Environment (Bellemare et al., 2013): Montezuma’s Revenge, MsPacman, and Amidar.

Original: See the
Revised: See the

Original: Appendix for the experimental details.
Revised: Appendix for the experimental details.

Original: The ﬁgures suggest that the options aim to visit different regions of the state space, promoting exploration in these games as well.
Revised: The ﬁgures suggest that the options aim to visit different regions of the state space, promoting exploration in these games as well.

Original: Importantly, unlike other approaches evaluated in Atari games (e.g., Machado et al., 2017; 2018), the options our method generates need not to be curated by an expert, who ﬁlters out non-meaningful options.
Revised: Importantly, unlike other approaches evaluated in Atari games (e.g., Machado et al., 2017; 2018), the options our method generates need not to be curated by an expert, who ﬁlters out non-meaningful options.

Original: Nevertheless, further analyses down the eigenspectrum are required for a better understanding of the diversity and utility of the discovered options.
Revised: Nevertheless, further analyses down the eigenspectrum are required for a better understanding of the diversity and utility of the discovered options.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Deep covering options is a new method for learning options to explore the state-space efﬁciently in a task-agnostic way.
Revised: Deep covering options is a new method for learning options to explore the state-space efﬁciently in a task-agnostic way.

Original: By minimizing expected cover time, it automatically discovers less-explored regions of the state-space and generates options to reach those regions.
Revised: By minimizing expected cover time, it automatically discovers less-explored regions of the state-space and generates options to reach those regions.

Original: Our algorithm is inspired by strong theoretical results in the tabular case while being computationally practical in large domains.
Revised: Our algorithm is inspired by strong theoretical results in the tabular case while being computationally practical in large domains.

Original: We demonstrated the use of our method in a pretraining setting as well as for the traditional online setting.
Revised: We demonstrated the use of our method in a pretraining setting as well as for the traditional online setting.

Original: In pretraining experiments we showed that the method is able to generate task-agnostic options which expand the frontier of the known regions of the state space without any reward information and successfully improves the performance of the agent in continuous control tasks.
Revised: In pretraining experiments we showed that the method is able to generate task-agnostic options which expand the frontier of the known regions of the state space without any reward information and successfully improves the performance of the agent in continuous control tasks.

Original: In online experiments we showed that the proposed method can also discover useful options during training and successfully speeds up learning without additional sampling.
Revised: In online experiments we showed that the proposed method can also discover useful options during training and successfully speeds up learning without additional sampling.

Original: A E XPERIMENTAL S ETUP A. P INBALL
Revised: A E XPERIMENTAL S ETUP A. P INBALL

Original: We used Q-learning ( α = 0 .
Revised: We used Q-learning ( α = 0 .

Original: 1 , γ = 0 .
Revised: 1 , γ = 0 .

Original: 99 , (cid:15) = 0 . 05 )
Revised: 99 , (cid:15) = 0 . 05 )

Original: with linear function approximation with a 3rd-order Fourier basis Konidaris et al.
Revised: with linear function approximation with a 3rd-order Fourier basis Konidaris et al.

Original: (2011) to train the policy of the option off-policy using the sampled trajectories but using the intrinsic reward r o (Algorithm 1).
Revised: (2011) to train the policy of the option off-policy using the sampled trajectories but using the intrinsic reward r o (Algorithm 1).

Original: We set the percentile to k = 30 .
Revised: We set the percentile to k = 30 .

Original: We set the Lagrange multiplier η to 1.0.
Revised: We set the Lagrange multiplier η to 1.0.

Original: We used Q-learning ( α = 0 .
Revised: We used Q-learning ( α = 0 .

Original: 1 , γ = 0 .
Revised: 1 , γ = 0 .

Original: 99 , (cid:15) = 0 . 05 )
Revised: 99 , (cid:15) = 0 . 05 )

Original: with a 3rd-order Fourier basis to train the high-level policy.
Revised: with a 3rd-order Fourier basis to train the high-level policy.

Original: The option’s policy is obtained with Q-learning ( α = 0 .
Revised: The option’s policy is obtained with Q-learning ( α = 0 .

Original: 1 , γ = 0 .
Revised: 1 , γ = 0 .

Original: 99 , (cid:15) = 0 . 05 )
Revised: 99 , (cid:15) = 0 . 05 )

Original: with linear function approximation with a 3rd-order Fourier basis.
Revised: with linear function approximation with a 3rd-order Fourier basis.

Original: We used the same setup for both ofﬂine and online option discovery.
Revised: We used the same setup for both ofﬂine and online option discovery.

Original: A.1. DIAYN
Revised: A.1. DIAYN

Original: The actor and the critic are implemented with 3 hidden layers with 256 units each followed by a
Revised: The actor and the critic are implemented with 3 hidden layers with 256 units each followed by a

Original: ReLU activation function with Adam optimizer with a step size of 0.005.
Revised: ReLU activation function with Adam optimizer with a step size of 0.005.

Original: We used neural networks for the actor and the critic as it outperformed an agent with the actor and the critic implemented by a linear approximator using 3rd order
Revised: We used neural networks for the actor and the critic as it outperformed an agent with the actor and the critic implemented by a linear approximator using 3rd order

Original: Fourier basis Konidaris et al.
Revised: Fourier basis Konidaris et al.

Original: Our discriminator network consists of 2 hidden layers with 256 units each followed by a ReLU activation function.
Revised: Our discriminator network consists of 2 hidden layers with 256 units each followed by a ReLU activation function.

Original: We trained the discriminator with Adam optimizer with a step size of 0.001.
Revised: We trained the discriminator with Adam optimizer with a step size of 0.001.

Original: We used Q-learning ( α = 0 .
Revised: We used Q-learning ( α = 0 .

Original: 1 , γ = 0 .
Revised: 1 , γ = 0 .

Original: 99 , (cid:15) = 0 . 05 )
Revised: 99 , (cid:15) = 0 . 05 )

Original: with linear function approximation with 3rd-order
Revised: with linear function approximation with 3rd-order

Original: Fourier basis to train the high-level policy.
Revised: Fourier basis to train the high-level policy.

Original: As the termination condition of DIAYN is not deﬁned in the work by Eysenbach et al.
Revised: As the termination condition of DIAYN is not deﬁned in the work by Eysenbach et al.

Original: (2019), we tested the termination probability of 0.0, 0.01, and 0.1, and 0.5 for any states.
Revised: (2019), we tested the termination probability of 0.0, 0.01, and 0.1, and 0.5 for any states.

Original: We picked 0.1 as it performed the best.
Revised: We picked 0.1 as it performed the best.

Original: [Equation]
Revised: [Equation]

Original: We trained the option’s policy using deep deterministic policy gradient Lillicrap et al.
Revised: We trained the option’s policy using deep deterministic policy gradient Lillicrap et al.

Original: We used the same hyperparameters as Wu et al.
Revised: We used the same hyperparameters as Wu et al.

Original: (2019) for DDPG and the eigenfunction f (see Appendix
Revised: (2019) for DDPG and the eigenfunction f (see Appendix

Original: D.2.2 in Wu et al.
Revised: D.2.2 in Wu et al.

Original: We set the threshold percentile k = 10 .
Revised: We set the threshold percentile k = 10 .

Original: We set the Lagrange multiplier η to 1.0.
Revised: We set the Lagrange multiplier η to 1.0.

Original: The high-level policy chooses options with double deep Q-learning van Hasselt et al.
Revised: The high-level policy chooses options with double deep Q-learning van Hasselt et al.

Original: The
Revised: The

Original: Q-network consists of two fully connected layers with 400 units with a batch normalization and a
Revised: Q-network consists of two fully connected layers with 400 units with a batch normalization and a

Original: ReLU in between.
Revised: ReLU in between.

Original: We trained it with the Adam optimizer using a step size of 0.0001 and a batch size of 64.
Revised: We trained it with the Adam optimizer using a step size of 0.0001 and a batch size of 64.

Original: We updated a target policy every step by update rate of 0.001.
Revised: We updated a target policy every step by update rate of 0.001.

Original: We set (cid:15) to 0.05.
Revised: We set (cid:15) to 0.05.

Original: We used the same setup for both ofﬂine and online option discovery.
Revised: We used the same setup for both ofﬂine and online option discovery.

Original: We evaluated DIAYN for the continuous control tasks.
Revised: We evaluated DIAYN for the continuous control tasks.

Original: The actor and the critic consist of three hidden layers with 256 units each followed by a ReLU activation function with Adam optimizer with a step size of 0.005.
Revised: The actor and the critic consist of three hidden layers with 256 units each followed by a ReLU activation function with Adam optimizer with a step size of 0.005.

Original: Our discriminator network consists of two hidden layers with 256 units each followed by a ReLU activation function.
Revised: Our discriminator network consists of two hidden layers with 256 units each followed by a ReLU activation function.

Original: We evaluated Adam optimizer with a step size of 0.001 and 0.005 for the discriminator.
Revised: We evaluated Adam optimizer with a step size of 0.001 and 0.005 for the discriminator.

Original: We used double DQN van Hasselt et al.
Revised: We used double DQN van Hasselt et al.

Original: (2016) to implement the high-level policy.
Revised: (2016) to implement the high-level policy.

Original: The Q-network consists of three fully connected layers with 400 units with a batch normalization and a ReLU in between.
Revised: The Q-network consists of three fully connected layers with 400 units with a batch normalization and a ReLU in between.

Original: We also evaluated the Q-network with two fully connected layers.
Revised: We also evaluated the Q-network with two fully connected layers.

Original: We trained it with the Adam optimizer with a step size of 0.001 and a batch size of 64.
Revised: We trained it with the Adam optimizer with a step size of 0.001 and a batch size of 64.

Original: We updated a target policy every step by update rate of 0.001.
Revised: We updated a target policy every step by update rate of 0.001.

Original: We also tried update rate of 0.01.
Revised: We also tried update rate of 0.01.

Original: For the setup we have tried,
Revised: For the setup we have tried,

Original: DIAYN did not outperform the baseline in all PointMaze, PointPush, and PointFall.
Revised: DIAYN did not outperform the baseline in all PointMaze, PointPush, and PointFall.

Original: A. A RCADE L EARNING E NVIRONMENT
Revised: A. A RCADE L EARNING E NVIRONMENT

Original: We use the screen image as the observation and train f using the sampled pixel images.
Revised: We use the screen image as the observation and train f using the sampled pixel images.

Original: We use a transformation from the original (210, 160, 3) dimensional RGB image to a (105, 80) grayscale image.
Revised: We use a transformation from the original (210, 160, 3) dimensional RGB image to a (105, 80) grayscale image.

Original: We set the threshold percentile k = 4 .
Revised: We set the threshold percentile k = 4 .

Original: We set the Lagrange multiplier η to 1.0.
Revised: We set the Lagrange multiplier η to 1.0.

Original: We sampled 200 trajectories each being 2000 steps long.
Revised: We sampled 200 trajectories each being 2000 steps long.

Original: The eigenfunction is learned with a convolutional neural network with 2 convolution layers (32 8x8 ﬁlters with stride 4 and 64 4x4 ﬁlters with stride 2) and a fully-connected hidden layer with 400 units and ReLU in between.
Revised: The eigenfunction is learned with a convolutional neural network with 2 convolution layers (32 8x8 ﬁlters with stride 4 and 64 4x4 ﬁlters with stride 2) and a fully-connected hidden layer with 400 units and ReLU in between.

Original: We trained the policy of the option with double deep Q-learning van Hasselt et al.
Revised: We trained the policy of the option with double deep Q-learning van Hasselt et al.

Original: The Q-network consists of 2 convolution layers (32 8x8 ﬁlters with stride 4 and 64 4x4 ﬁlters with stride 2) and a fully-connected hidden layer with 400 units and ReLU in between.
Revised: The Q-network consists of 2 convolution layers (32 8x8 ﬁlters with stride 4 and 64 4x4 ﬁlters with stride 2) and a fully-connected hidden layer with 400 units and ReLU in between.

Original: We trained it with Adam optimizer with a step size of 0.0001 and a batch size of 64.
Revised: We trained it with Adam optimizer with a step size of 0.0001 and a batch size of 64.

Original: We updated a target policy every step by update rate of 0.001.
Revised: We updated a target policy every step by update rate of 0.001.

Original: We set (cid:15) to 0.05.
Revised: We set (cid:15) to 0.05.

Original: We sampled 50 trajectories and plotted the positions of the player agent when the option terminated.
Revised: We sampled 50 trajectories and plotted the positions of the player agent when the option terminated.

Original: The plots depict the termination set of the generated options.
Revised: The plots depict the termination set of the generated options.

Original: The ﬁgures suggest that different options aim at visiting different regions of the state space, promoting exploration in these games as well.
Revised: The ﬁgures suggest that different options aim at visiting different regions of the state space, promoting exploration in these games as well.

Original: Importantly, differently from other approaches evaluated in Atari games (e.g.,
Revised: Importantly, differently from other approaches evaluated in Atari games (e.g.,

Original: Machado et al., 2017; 2018), the options our method generates do not need to be curated by an expert, with the ﬁrst eigenfunctions being able to generate meaningful options.
Revised: Machado et al., 2017; 2018), the options our method generates do not need to be curated by an expert, with the ﬁrst eigenfunctions being able to generate meaningful options.

Original: Nevertheless, further analyses down the eigenspectrum are required for a better understanding of the diversity and utility of the discovered options.
Revised: Nevertheless, further analyses down the eigenspectrum are required for a better understanding of the diversity and utility of the discovered options.