Original: B ETTER STATE EXPLORATION USING ACTION SEQUENCE EQUIVALENCE
Revised: B ETTER STATE EXPLORATION USING ACTION SEQUENCE EQUIVALENCE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Incorporating prior knowledge in reinforcement learning algorithms is mainly an open question.
Revised: Incorporating prior knowledge in reinforcement learning algorithms is mainly an open question.

Original: Even when insights about the environment dynamics are available, reinforcement learning is traditionally used in a tabula rasa setting and must explore and learn everything from scratch.
Revised: Even when insights about the environment dynamics are available, reinforcement learning is traditionally used in a tabula rasa setting and must explore and learn everything from scratch.

Original: In this paper, we consider the problem of exploiting priors about action sequence equivalence: that is, when different sequences of actions produce the same effect.
Revised: In this paper, we consider the problem of exploiting priors about action sequence equivalence: that is, when different sequences of actions produce the same effect.

Original: We propose a new local exploration strategy calibrated to minimize collisions and maximize new state visitations.
Revised: We propose a new local exploration strategy calibrated to minimize collisions and maximize new state visitations.

Original: We show that this strategy can be computed at little cost, by solving a convex optimization problem.
Revised: We show that this strategy can be computed at little cost, by solving a convex optimization problem.

Original: By replacing the usual (cid:15) -greedy strategy in a DQN, we demonstrate its potential in several environments with various dynamic structures.
Revised: By replacing the usual (cid:15) -greedy strategy in a DQN, we demonstrate its potential in several environments with various dynamic structures.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Despite the rapidly improving performance of Reinforcement Learning (RL) agents on a variety of tasks (Mnih et al., 2015; Silver et al., 2016), they remain largely sample-inefﬁcient learners compared to humans (Toromanoff et al., 2019).
Revised: Despite the rapidly improving performance of Reinforcement Learning (RL) agents on a variety of tasks (Mnih et al., 2015; Silver et al., 2016), they remain largely sample-inefﬁcient learners compared to humans (Toromanoff et al., 2019).

Original: Contributing to this is the vast amount of prior knowledge humans bring to the table before their ﬁrst interaction with a new task, including an understanding of physics, semantics, and affordances (Dubey et al., 2018).
Revised: Contributing to this is the vast amount of prior knowledge humans bring to the table before their ﬁrst interaction with a new task, including an understanding of physics, semantics, and affordances (Dubey et al., 2018).

Original: The considerable quantity of data necessary to train agents is becoming more problematic as RL is applied to ever more challenging and complex tasks.
Revised: The considerable quantity of data necessary to train agents is becoming more problematic as RL is applied to ever more challenging and complex tasks.

Original: Much research aims at tackling this issue, for example through transfer learning (Rusu et al., 2016), meta learning, and hierarchical learning, where agents are encouraged to use what they learn in one environment to solve a new task more quickly.
Revised: Much research aims at tackling this issue, for example through transfer learning (Rusu et al., 2016), meta learning, and hierarchical learning, where agents are encouraged to use what they learn in one environment to solve a new task more quickly.

Original: Other approaches attempt to use the structure of Markov Decision Processes (MDP) to accelerate learning without resorting to pretraining.
Revised: Other approaches attempt to use the structure of Markov Decision Processes (MDP) to accelerate learning without resorting to pretraining.

Original: Mahajan & Tulabandhula (2017) and Biza & Jr. (2019) learn simpler representations of MDPs that exhibit symmetrical structure, while van der Pol et al.
Revised: Mahajan & Tulabandhula (2017) and Biza & Jr. (2019) learn simpler representations of MDPs that exhibit symmetrical structure, while van der Pol et al.

Original: (2020) show that environment invariances can be hard-coded into equivariant neural networks.
Revised: (2020) show that environment invariances can be hard-coded into equivariant neural networks.

Original: A fundamental challenge standing in the way of improved sample efﬁciency is exploration.
Revised: A fundamental challenge standing in the way of improved sample efﬁciency is exploration.

Original: We consider a situation where the exact transition function of a Markov Decision Process is unknown, but some knowledge of its local dynamics is available under the form of a prior expectation that given sequences of actions have identical results.
Revised: We consider a situation where the exact transition function of a Markov Decision Process is unknown, but some knowledge of its local dynamics is available under the form of a prior expectation that given sequences of actions have identical results.

Original: This way of encoding prior knowledge is sufﬁciently ﬂexible to describe many useful environment structures, particularly when actions correspond to agent movement.
Revised: This way of encoding prior knowledge is sufﬁciently ﬂexible to describe many useful environment structures, particularly when actions correspond to agent movement.

Original: For example, in a gridworld (called RotationGrid hereafter) where the agent can move forward ( ↑ ) and rotate 90 ◦ to the left ( (cid:120) ) or to the right ( (cid:121) ), the latter two actions are the inverse of each other, in that performing one undoes the effect of the other.
Revised: For example, in a gridworld (called RotationGrid hereafter) where the agent can move forward ( ↑ ) and rotate 90 ◦ to the left ( (cid:120) ) or to the right ( (cid:121) ), the latter two actions are the inverse of each other, in that performing one undoes the effect of the other.

Original: During exploration, to encourage the visitation of not yet seen states, it is natural to simply ban sequences of actions that revert to previously visited states, following the reasoning of Tabu search (Glover, 1986).
Revised: During exploration, to encourage the visitation of not yet seen states, it is natural to simply ban sequences of actions that revert to previously visited states, following the reasoning of Tabu search (Glover, 1986).

Original: We observe further that (cid:121)(cid:121) and (cid:120)(cid:120) both lead to the same state (represented as state 4 in Figure 1).
Revised: We observe further that (cid:121)(cid:121) and (cid:120)(cid:120) both lead to the same state (represented as state 4 in Figure 1).

Original: If actions were uniformly sampled, the chances of visiting this state would be much higher than any of the others.
Revised: If actions were uniformly sampled, the chances of visiting this state would be much higher than any of the others.

Original: Based on these observations, we introduce a new method taking advantage of Equivalent
Revised: Based on these observations, we introduce a new method taking advantage of Equivalent

Original: Action SEquences for Exploration (EASEE), an overview of which can be found in Figure 1.
Revised: Action SEquences for Exploration (EASEE), an overview of which can be found in Figure 1.

Original: EASEE looks ahead several steps and calculates action sampling probabilities to explore as uniformly as possible new states conditionally on the action sequence equivalences given to it.
Revised: EASEE looks ahead several steps and calculates action sampling probabilities to explore as uniformly as possible new states conditionally on the action sequence equivalences given to it.

Original: It constructs a partial MDP which corresponds to a local representation of the true MDP around the current state.
Revised: It constructs a partial MDP which corresponds to a local representation of the true MDP around the current state.

Original: We then formulate the problem of determining the best distribution over action sequences as a linearly constrained convex optimization problem.
Revised: We then formulate the problem of determining the best distribution over action sequences as a linearly constrained convex optimization problem.

Original: Solving this optimization problem is computationally
Revised: Solving this optimization problem is computationally

Original: [Figure]
Revised: [Figure]

Original: inexpensive and can be done once and for all before learning begins, providing a principled and tractable exploration policy that takes into account environment structure.
Revised: inexpensive and can be done once and for all before learning begins, providing a principled and tractable exploration policy that takes into account environment structure.

Original: This policy can easily be injected into existing reinforcement learning algorithms as a substitute for (cid:15) -greedy exploration.
Revised: This policy can easily be injected into existing reinforcement learning algorithms as a substitute for (cid:15) -greedy exploration.

Original: Our contribution is threefold.
Revised: Our contribution is threefold.

Original: First, we formally introduce the notion of equivalent action sequences, a novel type of structure in Markov Decision Processes.
Revised: First, we formally introduce the notion of equivalent action sequences, a novel type of structure in Markov Decision Processes.

Original: Then, we show that priors on this type of structure can easily be exploited during ofﬂine exploration by solving a convex optimization problem.
Revised: Then, we show that priors on this type of structure can easily be exploited during ofﬂine exploration by solving a convex optimization problem.

Original: Finally, we provide experimental insights and show that incorporating EASEE into a DQN (Mnih et al., 2015) improves agent performance in several environments with various structures.
Revised: Finally, we provide experimental insights and show that incorporating EASEE into a DQN (Mnih et al., 2015) improves agent performance in several environments with various structures.

Revised: Overview We assume that we have sets of equivalent action-sequences for the environment.

Revised: Equivalent action sequences are sequences that lead to the same state.

Revised: These sequences are used to build a

Revised: DAG that models where the agent will end up after any sequence of actions of length d .

Original:   R ELATED W ORK
Revised: Because some sequences are equivalent, several parent nodes may share a child node. A naive exploration scheme like (cid:15) -greedy would waste resources by over exploring such child nodes. Instead, we leverage this information using the DAG constructed above; our method executes an exploratory action that maximizes the entropy of the future visited states.  R ELATED W ORK

Original: Curiosity-driven Exploration
Revised: Improved Exploration

Original: The problem of ensuring that agents see sufﬁciently diverse states has received a lot of attention from the RL community.
Revised: The problem of ensuring that agents see sufﬁciently diverse states has received a lot of attention from the RL community.

Original: Many methods rely on intrinsic rewards (Schmidhuber, 1991; Chentanez et al., 2005; ¸Sim¸sek & Barto, 2006; Lopes et al., 2012;
Revised: Many methods rely on intrinsic rewards (Schmidhuber, 1991; Chentanez et al., 2005; ¸Sim¸sek & Barto, 2006; Lopes et al., 2012;

Original: Bellemare et al., 2016; Ostrovski et al., 2017; Pathak et al., 2017) to entice agents to unseen or misunderstood areas.
Revised: Bellemare et al., 2016; Ostrovski et al., 2017; Pathak et al., 2017) to entice agents to unseen or misunderstood areas.

Original: In the tabular setting, these take the form of count-based exploration bonuses which guide the agent toward poorly visited states ( e.g. Strehl & Littman (2008)).
Revised: In the tabular setting, these take the form of count-based exploration bonuses which guide the agent toward poorly visited states ( e.g. Strehl & Littman (2008)).

Original: Scaling this method requires the use of function approximators (Burda et al., 2019; Badia et al., 2020; Flet-Berliac et al., 2021).
Revised: Scaling this method requires the use of function approximators (Burda et al., 2019; Badia et al., 2020; Flet-Berliac et al., 2021).

Original: Unlike EASEE, these methods necessitate the computation of non-stationary and vanishing novelty estimates, which require careful tuning to balance learning stability and exploration incentives.
Revised: Unlike EASEE, these methods necessitate the computation of non-stationary and vanishing novelty estimates, which require careful tuning to balance learning stability and exploration incentives.

Original: Moreover, because these bonuses are learned, and do not allow for the use of prior structure knowledge, they constitute an orthogonal approach to ours.
Revised: Moreover, because these bonuses are learned, and do not allow for the use of prior structure knowledge, they constitute an orthogonal approach to ours.

Revised: In Gupta et al.

Revised: (2018) exploration strategies are learned from prior experience.

Revised: Unlike EASEE this requires meta-training over a distribution of tasks.

Original: Redundancies in Trajectories The idea that different trajectories can overlap and induce redundancies in state visitation is used in Leurent & Maillard (2020) and Czech et al.
Revised: Redundancies in Trajectories The idea that different trajectories can overlap and induce redundancies in state visitation is used in Leurent & Maillard (2020) and Czech et al.

Original: (2020) in the case of Monte-Carlo tree search.
Revised: (2020) in the case of Monte-Carlo tree search.

Original: However, they require a generative model, and propose a new Bellman operator to update node values according to newly uncovered transitions rather than modifying exploration.
Revised: However, they require a generative model, and propose a new Bellman operator to update node values according to newly uncovered transitions rather than modifying exploration.

Original: Closer to our work, Caselles-Dupré et al.
Revised: Closer to our work, Caselles-Dupré et al.

Original: (2020) study structure in action sequences, but restrict themselves to commutative properties.
Revised: (2020) study structure in action sequences, but restrict themselves to commutative properties.

Revised: Grinsztajn et al.

Revised: (2021) quantiﬁes the probability of cycling back to a previously visited state, motivated by the analysis of reversible actions.

Original: Tabu search (Glover, 1986) is a meta-heuristic which uses knowledge of the past to escape local optima.
Revised: Tabu search (Glover, 1986) is a meta-heuristic which uses knowledge of the past to escape local optima.

Original: It is popular for combinatorial optimization (Hertz & Werra, 2005).
Revised: It is popular for combinatorial optimization (Hertz & Werra, 2005).

Original: Like our approach, it relies on a local structure: actions which are known to cancel out recent moves are deemed tabu , and are forbidden for a short period of time.
Revised: Like our approach, it relies on a local structure: actions which are known to cancel out recent moves are deemed tabu , and are forbidden for a short period of time.

Original: This prevents cycling around already found solutions, and thus encourages exploration.
Revised: This prevents cycling around already found solutions, and thus encourages exploration.

Original: In Abramson & Wechsler (2003), tabu search is combined with reinforcement learning, using action priors.
Revised: In Abramson & Wechsler (2003), tabu search is combined with reinforcement learning, using action priors.

Original: However, their method cannot make use of more complex action-sequence structure.
Revised: However, their method cannot make use of more complex action-sequence structure.

Original: Maximum State-Visitation Entropy Our goal to explore as uniformly as possible every nearby state can be seen as a local version of the Maximum State-Visitation Entropy problem (MSVE) (de Farias & Van Roy, 2003; Hazan et al., 2019; Lee et al., 2019; Guo et al., 2021).
Revised: Maximum State-Visitation Entropy Our goal to explore as uniformly as possible every nearby state can be seen as a local version of the Maximum State-Visitation Entropy problem (MSVE) (de Farias & Van Roy, 2003; Hazan et al., 2019; Lee et al., 2019; Guo et al., 2021).

Original: MSVE formulates exploration as a policy optimization problem whose solution maximizes the entropy of the distribution of visited states.
Revised: MSVE formulates exploration as a policy optimization problem whose solution maximizes the entropy of the distribution of visited states.

Original: Although some of these works (Hazan et al., 2019; Lee et al., 2019;
Revised: Although some of these works (Hazan et al., 2019; Lee et al., 2019;

Original: Guo et al., 2021) can make use of priors about state similarities, they learn a global policy and cannot exploit structure in action sequences.
Revised: Guo et al., 2021) can make use of priors about state similarities, they learn a global policy and cannot exploit structure in action sequences.

Original: Action Space Structure
Revised: Action Space Structure

Original: The idea of exploiting structure in action spaces is not new.
Revised: The idea of exploiting structure in action spaces is not new.

Original: Large discrete action spaces may be embedded in continuous action spaces either by leveraging prior information (Dulac-Arnold et al., 2016) or learning representations (Chandak et al., 2019).
Revised: Large discrete action spaces may be embedded in continuous action spaces either by leveraging prior information (Dulac-Arnold et al., 2016) or learning representations (Chandak et al., 2019).

Original: Tavakoli et al.
Revised: Tavakoli et al.

Original: (2018) manage high-dimensional action spaces by assuming a degree of independence between each dimension.
Revised: (2018) manage high-dimensional action spaces by assuming a degree of independence between each dimension.

Original: These methods aim to improve the generalization of policies to unseen actions rather than enhancing exploration.

Original: Tennenholtz & Mannor (2019) provide an understanding of actions through their context in demonstrations.

Original: Farquhar et al.
Revised: Farquhar et al.

Original: (2020) introduce a curriculum of progressively growing action spaces to accelerate learning.
Revised: (2020) introduce a curriculum of progressively growing action spaces to accelerate learning.

Original: Certain characteristics of actions, such as reversibility, can be learned through training, as in Grinsztajn et al.
Revised: These methods aim to improve the generalization of policies to unseen actions in large action spaces rather than enhancing exploration. Leveraging previous trajectories to extract prior knowledge, Tennenholtz & Mannor (2019) provide an understanding of actions through their context in demonstrations.

Original:  F ORMALISM 3. E QUIVALENCE OVER A CTION S EQUENCES
Revised:  F ORMALISM 3. E QUIVALENCE OVER A CTION S EQUENCES

Original: We consider a Markov Decision Process (MDP) deﬁned as a 5-tuple M = ( S , A , T, R, γ ) , with S the set of states, A the action set, T the transition function, R the reward function and the discount factor γ .
Revised: We consider a Markov Decision Process (MDP) deﬁned as a 5-tuple M = ( S , A , T, R, γ ) , with S the set of states, A the action set, T the transition function, R the reward function and the discount factor γ .

Original: The set of actions is assumed to be ﬁnite |A| < ∞ .
Revised: The set of actions is assumed to be ﬁnite |A| < ∞ .

Original: We restrict ourselves to deterministic
Revised: We restrict ourselves to deterministic

Original: MDPs.
Revised: MDPs.

Original: A possible extension to MDPs with stochastic dynamics is discussed in Appendix A.6.
Revised: A possible extension to MDPs with stochastic dynamics is discussed in Appendix A.6.

Original: In the following, the notations are borrowed from formal language theory.
Revised: In the following, the notations are borrowed from formal language theory.

Original: Sequences of actions are analogous to strings over the set of symbols A (possible actions).
Revised: Sequences of actions are analogous to strings over the set of symbols A (possible actions).

Original: The set of all possible sequences of actions is denoted A (cid:63) =
Revised: The set of all possible sequences of actions is denoted A (cid:63) =

Original: (cid:83) ∞
Revised: (cid:83) ∞

Original: k
Revised: k

Original: =
Revised: =

Original: A k where A k is the set of all sequences of length k and A 0 contains as single element the empty sequence Λ .
Revised: A k where A k is the set of all sequences of length k and A 0 contains as single element the empty sequence Λ .

Original: We use .
Revised: We use .

Original: for the concatenation operator, such that for v 1 ∈
Revised: for the concatenation operator, such that for v 1 ∈

Original: A h 1 , v 2 ∈ A h 2 , v 1 .v 2 ∈
Revised: A h 1 , v 2 ∈ A h 2 , v 1 .v 2 ∈

Original: A h 1 + h 2 .
Revised: A h 1 + h 2 .

Original: The transition function T : S ×
Revised: The transition function T : S ×

Original: A → S gives the next state s (cid:48) when action a is taken in state s : T ( s, a )
Revised: A → S gives the next state s (cid:48) when action a is taken in state s : T ( s, a )

Original: = s (cid:48) .
Revised: = s (cid:48) .

Original: We recursively extend this operator to action sequences T : S × A (cid:63) → S such that, ∀ s ∈ S , ∀ a ∈ A ,
Revised: We recursively extend this operator to action sequences T : S × A (cid:63) → S such that, ∀ s ∈ S , ∀ a ∈ A ,

Original: ∀ w ∈ A (cid:63) :
Revised: ∀ w ∈ A (cid:63) :

Original: [Equation]
Revised: [Equation]

Original: Intuitively, this operator gives the new state of the MDP after a sequence of actions is performed from state s .
Revised: Intuitively, this operator gives the new state of the MDP after a sequence of actions is performed from state s .

Original: Deﬁnition 1 (Equivalent sequences) .
Revised: Deﬁnition 1 (Equivalent sequences) .

Original: We say that two action sequences a 1 . . .
Revised: We say that two action sequences a 1 . . .

Original: a n and a (cid:48) 1 . . .
Revised: a n and a (cid:48) 1 . . .

Original: a (cid:48) m ∈ A (cid:63) are equivalent at state s ∈ S if
Revised: a (cid:48) m ∈ A (cid:63) are equivalent at state s ∈ S if

Original: [Equation]
Revised: [Equation]

Original: Two sequences of actions are equivalent over M if they are equivalent at state s for all s in S .
Revised: Two sequences of actions are equivalent over M if they are equivalent at state s for all s in S .

Original: This is written:
Revised: This is written:

Original: [Equation]
Revised: [Equation]

Original: This means that we consider two sequences of actions to be equivalent when following one or the other will always lead to the same state.
Revised: This means that we consider two sequences of actions to be equivalent when following one or the other will always lead to the same state.

Original: When the considered MDP M is unambiguous, we simplify the notation by writing ∼ instead of ∼ M .
Revised: When the considered MDP M is unambiguous, we simplify the notation by writing ∼ instead of ∼ M .

Original: We argue that some priors about the environments can be easily encoded as a small set of action sequence equivalences.
Revised: We argue that some priors about the environments can be easily encoded as a small set of action sequence equivalences.

Original: For example, we may know that going left then right is the same thing as going right then left, that rotating two times to the left is the same thing as rotating two times to the right, or that opening a door twice is the same thing as opening the door once.
Revised: For example, we may know that going left then right is the same thing as going right then left, that rotating two times to the left is the same thing as rotating two times to the right, or that opening a door twice is the same thing as opening the door once.

Original: All these priors can be encoded as a set of equivalences:
Revised: All these priors can be encoded as a set of equivalences:

Original: Deﬁnition 2 (Equivalence set) .
Revised: Deﬁnition 2 (Equivalence set) .

Original: Given a MDP M and several equivalent sequence pairs v 1 ∼ w 1 , v 2 ∼ w 2 , . . .
Revised: Given a MDP M and several equivalent sequence pairs v 1 ∼ w 1 , v 2 ∼ w 2 , . . .

Original: , v n ∼ w n , we say that Ω
Revised: , v n ∼ w n , we say that Ω

Original: = {{ v 1 , w 1 } , { v 2 , w 2 } , . . .
Revised: = {{ v 1 , w 1 } , { v 2 , w 2 } , . . .

Original: { v n , w n }} is an equivalence set over M .
Revised: { v n , w n }} is an equivalence set over M .

Original: Formally, Ω is a set of pairs of elements of A (cid:63) , such that Ω ⊂ ( A (cid:63) ) 2 .
Revised: Formally, Ω is a set of pairs of elements of A (cid:63) , such that Ω ⊂ ( A (cid:63) ) 2 .

Original: By abuse of notation, we write v ∼ w ∈ Ω if { v, w } ∈ Ω .
Revised: By abuse of notation, we write v ∼ w ∈ Ω if { v, w } ∈ Ω .

Original: Intuitively, it is clear that action sequence equivalences can be combined to form new, longer equivalences.
Revised: Intuitively, it is clear that action sequence equivalences can be combined to form new, longer equivalences.

Original: For example, knowing that going left then right is the same thing as going right then left, we can deduce that going two times left then two times right is the same thing as going two times right then two times left.
Revised: For example, knowing that going left then right is the same thing as going right then left, we can deduce that going two times left then two times right is the same thing as going two times right then two times left.

Original: In the same fashion, if opening a door twice produces the same effect as opening it once, opening three times the door does the same.
Revised: In the same fashion, if opening a door twice produces the same effect as opening it once, opening three times the door does the same.

Original: We formalize these notions in what follows.
Revised: We formalize these notions in what follows.

Original: First, we note that equivalent sequences can be concatenated.
Revised: First, we note that equivalent sequences can be concatenated.

Original: Proposition 1.
Revised: Proposition 1.

Original: If we have two pairs of equivalent sequences over M , i.e. w 1 , w 2 , w 3 , w 4 ∈ A (cid:63) such that
Revised: If we have two pairs of equivalent sequences over M , i.e. w 1 , w 2 , w 3 , w 4 ∈ A (cid:63) such that

Original: [Equation]
Revised: [Equation]

Original: then the concatenation of the sequences are also equivalent sequences:
Revised: then the concatenation of the sequences are also equivalent sequences:

Original: [Equation]
Revised: [Equation]

Original: The proof is given in Appendix A.1.
Revised: The proof is given in Appendix A.1.

Original: We are now going to deﬁne formally the fact that the equivalence of two sequences can be deduced from an equivalence set Ω .
Revised: We are now going to deﬁne formally the fact that the equivalence of two sequences can be deduced from an equivalence set Ω .

Original: We ﬁrst consider the previous example where an action a has the effect of opening a door, such that a.a ∼ a .
Revised: We ﬁrst consider the previous example where an action a has the effect of opening a door, such that a.a ∼ a .

Original: We can then write a.a.a ∼ ( a.a ) .a
Revised: We can then write a.a.a ∼ ( a.a ) .a

Original: ∼ ( a ) .a
Revised: ∼ ( a ) .a

Original: ∼ a.a ∼ a by applying two times the equivalence a.a ∼ a and rearranging the parentheses.
Revised: ∼ a.a ∼ a by applying two times the equivalence a.a ∼ a and rearranging the parentheses.

Original: More generally and intuitively, the equivalence of two action sequences v and w can be deduced from Ω , which we denote v ∼ Ω w , if v can be changed into w iteratively, chaining equivalences of Ω .
Revised: More generally and intuitively, the equivalence of two action sequences v and w can be deduced from Ω , which we denote v ∼ Ω w , if v can be changed into w iteratively, chaining equivalences of Ω .

Original: More formally, we write v ∼ 1Ω w if v can be changed to w in one steps, meaning: ∃ u 1 , u 2 , v 1 , w 1 ∈ A (cid:63) such that
Revised: More formally, we write v ∼ 1Ω w if v can be changed to w in one steps, meaning: [Equation] such that

Original: [Equation]
Revised: [Equation]

Original: For n ≥ 2 , we say that v can be changed into w in n steps if there is a sequence v 1 , . .
Revised: For n ≥ 2 , we say that v can be changed into w in n steps if there is a sequence v 1 , . .

Original: , v n ∈ A (cid:63)
Revised: , v n ∈ A (cid:63)

Original: such that v ∼ 1Ω v 1 ∼ 1Ω · · · ∼ 1Ω v n = w .
Revised: such that v ∼ 1Ω v 1 ∼ 1Ω · · · ∼ 1Ω v n = w .

Original: Finally, we say that v ∼ Ω w if there is n ∈ N such that v can be changed into w in n steps.
Revised: Finally, we say that v ∼ Ω w if there is n ∈ N such that v can be changed into w in n steps.

Original: The relation ∼ Ω is thus a formal way of extending equivalences from a ﬁxed equivalence set Ω , and at ﬁrst glance not connected with ∼ , which deals with the equivalences of the MDP dynamics.
Revised: The relation ∼ Ω is thus a formal way of extending equivalences from a ﬁxed equivalence set Ω , and at ﬁrst glance not connected with ∼ , which deals with the equivalences of the MDP dynamics.

Original: We now show a connection between the two notions.
Revised: We now show a connection between the two notions.

Original: Theorem 1.
Revised: Theorem 1.

Original: Given an equivalence set Ω , ∼ Ω is an equivalence relationship.
Revised: Given an equivalence set Ω , ∼ Ω is an equivalence relationship.

Original: Furthermore, for v, w ∈ A (cid:63) , v ∼ Ω w ⇒ v ∼ w .
Revised: Furthermore, for v, w ∈ A (cid:63) , v ∼ Ω w ⇒ v ∼ w .

Original: The proof is given in Appendix A.2.
Revised: The proof is given in Appendix A.2.

Original: Given this relation between ∼ and ∼ Ω , we will simplify the notation in what follows by writing ∼ instead of ∼ Ω when the equivalence set considered is unambiguous.
Revised: Given this relation between ∼ and ∼ Ω , we will simplify the notation in what follows by writing ∼ instead of ∼ Ω when the equivalence set considered is unambiguous.

Original: As ∼ Ω is an equivalence relationship, it provides a partition over action sequences: two action sequences in the same set lead to the same ﬁnalstate from any given state.
Revised: As ∼ Ω is an equivalence relationship, it provides a partition over action sequences: two action sequences in the same set lead to the same ﬁnal

Original: 3. L OCAL - DYNAMICS G RAPH
Revised: 3. L OCAL - DYNAMICS G RAPH

Original: We leverage the equivalences deﬁned above to determine a model of the MDP up to a few timesteps.
Revised: We leverage the equivalences deﬁned above to determine a model of the MDP up to a few timesteps.

Original: As traditionally done in Monte-Carlo Tree Search (Coulom, 2007), an MDP ( S , A , T, R, γ ) with deterministic dynamics can be locally unrolled to produce a tree, where a node of depth h represents a sequence of actions v ∈ A h , and the edges represent transitions between such sequences.
Revised: As traditionally done in Monte-Carlo Tree Search (Coulom, 2007), an MDP ( S , A , T, R, γ ) with deterministic dynamics can be locally unrolled to produce a tree, where a node of depth h represents a sequence of actions v ∈ A h , and the edges represent transitions between such sequences.

Original: The root of the tree corresponds to the empty action sequence Λ .
Revised: The root of the tree corresponds to the empty action sequence Λ .

Original: Here we adopt the same formalism, except that equivalent sequences will point to the same node.
Revised: Here we adopt the same formalism, except that equivalent sequences will point to the same node.

Original: Given a tree T of depth d ∈ N corresponding to a partial unrolling of sequences in A (cid:63) , and an equivalence set Ω , we call local-dynamics graph of depth d under equivalence Ω the graph G =
Revised: Given a tree T of depth d ∈ N corresponding to a partial unrolling of sequences in A (cid:63) , and an equivalence set Ω , we call local-dynamics graph of depth d under equivalence Ω the graph G =

Original: ( V, E ) corresponding to the tree T where nodes are quotiented with the equivalence relation ∼ Ω .
Revised: ( V, E ) corresponding to the tree T where nodes are quotiented with the equivalence relation ∼ Ω .

Original: Intuitively, it means that nodes corresponding to equivalent action sequences are merged.
Revised: Intuitively, it means that nodes corresponding to equivalent action sequences are merged.

Original: In this case, the resulting graph is not necessarily a tree.
Revised: In this case, the resulting graph is not necessarily a tree.

Original: In the following, unless the distinction is necessary, we identify action sequences with their equivalence classes.
Revised: In the following, unless the distinction is necessary, we identify action sequences with their equivalence classes.

Original: The graph G gives rise to a new, smaller MDP resulting from M : the state space V is the set of action sequences smaller than d quotiented by the equivalence relation ∼ Ω , the action space A is untouched.
Revised: The graph G gives rise to a new, smaller MDP resulting from M : the state space V is the set of action sequences smaller than d quotiented by the equivalence relation ∼ Ω , the action space A is untouched.

Original: Given a node n corresponding to a sequence w ∈ A (cid:63) , and an action a ∈ A , T ( n, a ) is the node representing the sequence w.a ∈ A (cid:63) .
Revised: Given a node n corresponding to a sequence w ∈ A (cid:63) , and an action a ∈ A , T ( n, a ) is the node representing the sequence w.a ∈ A (cid:63) .

Original: Nodes representing sequences of length exactly d are ﬁnal states .
Revised: Nodes representing sequences of length exactly d are ﬁnal states .

Original: The initial state v 0 is the empty sequence Λ .
Revised: The initial state v 0 is the empty sequence Λ .

Original: This MDP represents the local dynamics induced by ∼ Ω from a given root state.
Revised: This MDP represents the local dynamics induced by ∼ Ω from a given root state.

Original: We detail in the next section how to construct such graphs in practice, and how to use these sub-MDPs for a better exploration.
Revised: We detail in the next section how to construct such graphs in practice, and how to use these sub-MDPs for a better exploration.

Original:  E QUIVALENT A CTION SE QUENCES FOR E XPLORATION (EASEE) 4. F ROM E QUIVALENT A CTIONS TO L OCAL - DYNAMICS G RAPH
Revised:  E QUIVALENT A CTION SE QUENCES FOR E XPLORATION (EASEE) 4. F ROM E QUIVALENT A CTIONS TO L OCAL - DYNAMICS G RAPH

Original: Producing the local-dynamics graph involves considering all possible action sequences and merging those that are equivalent.
Revised: Producing the local-dynamics graph involves considering all possible action sequences and merging those that are equivalent.

Original: Figure 2 illustrates the construction of a local-dynamics graph, given
Revised: Figure 2 illustrates the construction of a local-dynamics graph, given

Original: A = { a 1 , a 2 } and Ω = { a 1 a 1 ∼ Λ , a 2 a 1 ∼ a 1 a 2 } .
Revised: A = { a 1 , a 2 } and Ω = { a 1 a 1 ∼ Λ , a 2 a 1 ∼ a 1 a 2 } .

Original: Starting from the root node 

Revised: Starting from the root node

Original: (ﬁrst step), we iteratively expand the graph by unrolling the nodes at the edges of the graph.
Revised: (ﬁrst step), we iteratively expand the graph by unrolling the nodes at the edges of the graph.

Original: Steps 2 and 3 create nodes 1 and 2 corresponding to action sequences a 1 and a 2 respectively.
Revised: Steps 2 and 3 create nodes 1 and 2 corresponding to action sequences a 1 and a 2 respectively.

Original: In a tree, the expansion of a node corresponding to a sequence w ∈
Revised: In a tree, the expansion of a node corresponding to a sequence w ∈

Original: A h with the action a ∈
Revised: A h with the action a ∈

Original: A always leads to the creation of a new leaf that results from the sequence of actions w.a ∈ A h +1 .
Revised: A always leads to the creation of a new leaf that results from the sequence of actions w.a ∈ A h +1 .

Original: However, in a local-dynamics graph the node representing w.a might already be present, in which case we add an edge from w without creating a new node.
Revised: However, in a local-dynamics graph the node representing w.a might already be present, in which case we add an edge from w without creating a new node.

Original: In Figure 2, this case occurs at the 4 th and the 6 th construction steps.

Original: The ﬁrst case corresponds to adding the action a 1 to the node 1 , which represents the action sequence a 1 .a 1 .

Original: Since a 1 .a 1 ∼ Λ ∈ Ω , we simply add an edge from 1 to 0 .

Original: The second case occurs when expanding node 2 with the action a 1 , leading to the action sequence a 2 .a 1 ∼ Ω a 1 .a 2 .

Original: Since node 3 already represents a 1 .a 2 , we simply add an edge from node 2 to node 3 .

Original: As a ﬁnal construction step, we prune edges which go backward in the local-dynamics graph, like (1, 0) in Fig.
Revised: As a ﬁnal construction step, we prune edges which go backward in the local-dynamics graph, like (1, 0) in Fig.

Original: 2, such that the resulting graph is a DAG.
Revised: 2, such that the resulting graph is a DAG.

Original: This is motivated by the fact that we are interested in ﬁnding a good exploration policy: an action which takes us back to a previously visited state should be ignored.
Revised: This is motivated by the fact that we are interested in ﬁnding a good exploration policy: an action which takes us back to a previously visited state should be ignored.

Original: From a practical point of view, the graph construction algorithm takes as input the action set A , the sequence equivalence set Ω , and the desired depth d , and outputs a DAG.
Revised: From a practical point of view, the graph construction algorithm takes as input the action set A , the sequence equivalence set Ω , and the desired depth d , and outputs a DAG.

Original: Informally, it starts from a graph G =
Revised: Informally, it starts from a graph G =

Original: ( V, E ) reduced to a root state { 0 } and iteratively expands G until a distance d to the root is reached.
Revised: ( V, E ) reduced to a root state { 0 } and iteratively expands G until a distance d to the root is reached.

Original: We store in each node every action sequence which allows to reach it from any parent nodes.
Revised: We store in each node every action sequence which allows to reach it from any parent nodes.

Original: For example, in Fig.2, the node 3 can be reached from the node 0 with sequences { a 1 .a 2 , a 2 .a 1 } , from node 1 with sequence { a 2 } and from node 2 with sequence { a 1 } , thus the set of sequences stored in node 3 would be { a 1 , a 2 , a 1 .a 2 , a 2 .a 1 } .

Original: When expanding a node n with an action a ∈ A , we check every sequence w stored in n if w.a appears in Ω , and if a node corresponding to an equivalent sequence of w.a is already in V .
Revised: When expanding a node n with an action a ∈ A , we check every sequence w stored in n if w.a appears in Ω , and if a node corresponding to an equivalent sequence of w.a is already in V .

Original: If it is the case, we simply add an edge from n to this node, otherwise we create a new node representing w.a .
Revised: If it is the case, we simply add an edge from n to this node, otherwise we create a new node representing w.a .

Original: In practice, this algorithm can be reﬁned: in each node, we only store sequences which are subsequences of sequences in Ω .

Original: We provide a more detailed implementation of this algorithm in Appendix A.3.
Revised: We provide a more detailed implementation of this algorithm in Appendix A.3.

Original: Proposition 2.
Revised: Proposition 2.

Original: The complexity of this graph construction algorithm is upper bounded by
Revised: The complexity of this graph construction algorithm is upper bounded by

Original: O (cid:0) |A| 2 d | Ω | d (cid:1) .
Revised: O (cid:0) |A| 2 d | Ω | d (cid:1) .

Original: The proof is given in Appendix A.4.
Revised: The proof is given in Appendix A.4.

Original: It is to be noted that this upper bound is in general far larger than the actual number of operations.
Revised: It is to be noted that this upper bound is in general far larger than the actual number of operations.

Original: Indeed, it supposes that the number of nodes in the graph is |A| d , although it can be much smaller thanks to the redundancies induced by Ω .
Revised: Indeed, it supposes that the number of nodes in the graph is |A| d , although it can be much smaller thanks to the redundancies induced by Ω .

Original: A more precise formula is
Revised: A more precise formula is

Original: O (cid:0) |
Revised: O (cid:0) |

Original: V ||A| d | Ω | (cid:1) , where | V | is the number of nodes in the ﬁnal graph and depends on the structure of Ω .
Revised: V ||A| d | Ω | (cid:1) , where | V | is the number of nodes in the ﬁnal graph and depends on the structure of Ω .

Original: Despite this exponential theoretical complexity, the goal is to use this algorithm locally, thus for small depths.
Revised: Despite this exponential theoretical complexity, the goal is to use this algorithm locally, thus for small depths.

Original: In practice we found that local-dynamics graphs could be computed within a few seconds on a standard laptop.
Revised: In practice we found that local-dynamics graphs could be computed within a few seconds on a standard laptop.

Original: 4. F ROM L OCAL - DYNAMICS G RAPH TO L OCAL E XPLORATION P OLICY
Revised: 4. F ROM L OCAL - DYNAMICS G RAPH TO L OCAL E XPLORATION P OLICY

Original: Once the local-dynamics graph ( V, E ) has been constructed, our goal is to ﬁnd a good local exploration policy in the resulting MDP as deﬁned in Section 3.2.
Revised: Once the local-dynamics graph ( V, E ) has been constructed, our goal is to ﬁnd a good local exploration policy in the resulting MDP as deﬁned in Section 3.2.

Original: We recall that its set of states is V , and its actions dynamics are given by the edges E .
Revised: We recall that its set of states is V , and its actions dynamics are given by the edges E .

Original: Ideally, we would want to ﬁnd a policy π such that all nodes in the local-dynamics graph are visited equally often.
Revised: Ideally, we would want to ﬁnd a policy π such that all nodes in the local-dynamics graph are visited equally often.

Original: Given a policy π , a state v ∈ V and an action a ∈ A , we denote p π,t ( v ) and p π,t ( v, a ) the t -steps state distribution and state-action distribution respectively.
Revised: Given a policy π , a state v ∈ V and an action a ∈ A , we denote p π,t ( v ) and p π,t ( v, a ) the t -steps state distribution and state-action distribution respectively.

Original: Formally, p π,t ( v ) =
Revised: Formally, p π,t ( v ) =

Original: P π ( v t = v ) and p π,t ( v, a )
Revised: P π ( v t = v ) and p π,t ( v, a )

Original: =
Revised: =

Original: P π ( v t
Revised: P π ( v t

Original: = v, a t = a ) .
Revised: = v, a t = a ) .

Original: Ideally we would like each t -step state distribution to be uniform.
Revised: Ideally we would like each t -step state distribution to be uniform.

Original: However, depending on the exact local-dynamics graph this may or may not be possible.
Revised: However, depending on the exact local-dynamics graph this may or may not be possible (see Figure 1 for an example where obtaining a uniform distribution is impossible).

Original: Instead, following the principle of maximum entropy (Jaynes, 1957), we frame the objective of balancing the state distribution at step t as maximizing H ([ p π,t ( v 0 ) , p π,t ( v 1 ) , . . .
Revised: Instead, following the principle of maximum entropy (Jaynes, 1957), we frame the objective of balancing the state distribution at step t as maximizing H ([ p π,t ( v 0 ) , p π,t ( v 1 ) , . . .

Original: , p π,t ( v | V |− 1 )]
Revised: , p π,t ( v | V |− 1 )]

Original: = H ( p π,t ) , where H is the Shannon entropy.
Revised: = H ( p π,t ) , where H is the Shannon entropy.

Original: For a local-dynamics graph of depth d ∈ N , we deﬁne our global objective as maximizing
Revised: For a local-dynamics graph of depth d ∈ N , we deﬁne our global objective as maximizing

Original: J ( π ) =
Revised: J ( π ) =

Original: ˜ J ( p π, 1 , . .
Revised: ˜ J ( p π, 1 , . .

Original: . , p π,d )
Revised: . , p π,d )

Original: = 1 d (cid:80) dt
Revised: = 1 d (cid:80) dt

Original: =1 H ( p π,t ) .
Revised: =1 H ( p π,t ) .

Original: Other global objectives are possible, for example optimizing entropy over only the ﬁnal states, or some other weighted mixture.
Revised: Other global objectives are possible, for example optimizing entropy over only the ﬁnal states, or some other weighted mixture.

Original: In practice, over simple experiments, we observed that changes in the entropy mixture hardly induced any variation in the computed policies and agent behavior.
Revised: In practice, over simple experiments, we observed that changes in the entropy mixture hardly induced any variation in the computed policies and agent behavior.

Original: Informally, our objective can be understood as maximizing state diversity locally, for every timestep smaller than d .
Revised: Informally, our objective can be understood as maximizing state diversity locally, for every timestep smaller than d .

Original: For environments where additional priors about state interests are available, one could adapt the quantity J to compute the entropy on a subset of the most interesting states, therefore biasing exploration toward promising areas.
Revised: For environments where additional priors about state interests are available, one could adapt the quantity J to compute the entropy on a subset of the most interesting states, therefore biasing exploration toward promising areas.

Original: We consider K , the set of joint distributions ( p0 , p 1 , . . .
Revised: We consider K , the set of joint distributions ( p , p 1 , . . .

Original: p d ) which veriﬁes the following properties:
Revised: p d ) which veriﬁes the following properties:

Original: • ∀ t ≤ d, p t ( v, a ) ≥• ∀ v ∈ V, (cid:80) a ∈A p 0 ( v, a ) = p 0 ( v ) = 1 v 0 ( v ) • ∀ t < d, (cid:80) a ∈A p t +1 ( v, a ) =
Revised: • ∀ t ≤ d, p t ( v, a ) ≥• ∀ v ∈ V, (cid:80) a ∈A p 0 ( v, a ) = p 0 ( v ) = 1 v [Equation] v ) • ∀ t < d, (cid:80) a ∈A p t +1 ( v, a ) =

Original: (cid:80) v (cid:48) ∈ V,a ∈A p t ( v (cid:48) , a ) P ( v | v (cid:48) , a )
Revised: (cid:80) v (cid:48) ∈ V,a ∈A p t ( v (cid:48) , a ) P ( v | v (cid:48) , a )

Original: We denote D ( A ) the set of distributions over A .
Revised: We denote D ( A ) the set of distributions over A .

Original: From any ( p 0 , p 1 , . . .
Revised: From any ( p 0 , p 1 , . . .

Original: , p d ) ∈ K , it is possible to ﬁnd a time-dependent policy π : V ×{ 0 , . . .
Revised: , p d ) ∈ K , it is possible to ﬁnd a time-dependent policy π : V ×{ 0 , . . .

Original: , d } → D ( A ) such that p 

Revised: , d } → D ( A ) such that p

Original: = p π, 0 , p 1 = p π, 1 , . . .
Revised: = p π, 0 , p 1 = p π, 1 , . . .

Original: , p d = p π,d , and for any policy π we have ( p π, 0 , p π, 1 , . . .
Revised: , p d = p π,d , and for any policy π we have ( p π, 0 , p π, 1 , . . .

Original: , p π,t ) ∈ K (Puterman, 2014).
Revised: , p π,t ) ∈ K (Puterman, 2014).

Original: As the entropy H is concave, the function ˜
Revised: As the entropy H is concave, the function ˜

Original: J is a concave function over K .
Revised: J is a concave function over K .

Original: Moreover, the constraints deﬁning K are linear.
Revised: Moreover, the constraints deﬁning K are linear.

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: can be solved efﬁciently using any convex solver.
Revised: can be solved efﬁciently using any convex solver.

Original: In our implementation, we use CVXPY (Diamond & Boyd, 2016; Agrawal et al., 2018).
Revised: In our implementation, we use CVXPY (Diamond & Boyd, 2016; Agrawal et al., 2018).

Original: Once ( p (cid:63) 1 , . . .
Revised: Once ( p (cid:63) 1 , . . .

Original: , p (cid:63)d ) =
Revised: , p (cid:63)d ) =

Original: arg max K ˜ J is computed, we can immediately calculate a time-dependent policy π (cid:63) from such a distribution (Puterman, 2014) with:[Equation]
Revised: arg max K ˜ J is computed, we can immediately calculate a time-dependent policy π (cid:63) from such a distribution (Puterman, 2014) with:

Original: [Equation]
Revised: [Equation]

Original: As the local-dynamics graph ( V, E ) is a DAG, the set of nodes V 0 , V 1 , . .
Revised: As the local-dynamics graph ( V, E ) is a DAG, the set of nodes V 0 , V 1 , . .

Original: , V d which can be reached respectively at timesteps t = 0 , t = 1 , . .
Revised: , V d which can be reached respectively at timesteps t = 0 , t = 1 , . .

Original: t = d are disjoint.
Revised: t = d are disjoint.

Original: Therefore any time-dependent policy deﬁned on V can be framed as a stationary policy.
Revised: Therefore any time-dependent policy deﬁned on V can be framed as a stationary policy.

Original: Considering for example π (cid:63) , we can write π (cid:63) ( v, · ) =
Revised: Considering for example π (cid:63) , we can write π (cid:63) ( v, · ) =

Original: π (cid:63) 

Revised: π (cid:63)

Original: ( v, · ) if v ∈ V 0 , π (cid:63) ( v, · ) =
Revised: ( v, · ) if v ∈ V 0 , π (cid:63) ( v, · ) =

Original: π (cid:63) 1 ( v, · ) if v ∈ V 1 , . . .
Revised: π (cid:63) 1 ( v, · ) if v ∈ V 1 , . . .

Original: and π (cid:63) ( v, · ) =
Revised: and π (cid:63) ( v, · ) =

Original: π (cid:63)d ( v, · ) if v ∈ V d .
Revised: π (cid:63)d ( v, · ) if v ∈ V d .

Original: 4. F ROM L OCAL E XPLORATION TO G LOBAL P OLICY
Revised: 4. F ROM L OCAL E XPLORATION TO G LOBAL P OLICY

Original: The optimal π (cid:63) determined in the previous section can then be used to guide exploration.
Revised: The optimal π (cid:63) determined in the previous section can then be used to guide exploration.

Original: With an (cid:15) -greedy policy, each step has a probability (cid:15) of being an exploration step, where an action is sampled uniformly.
Revised: With an (cid:15) -greedy policy, each step has a probability (cid:15) of being an exploration step, where an action is sampled uniformly.

Original: Instead, we keep in memory the local-dynamics graph, and initialize the current state at v = Λ .
Revised: Instead, we keep in memory the local-dynamics graph, and initialize the current state at v = Λ .

Original: Everytime an action a is performed, v is updated such that v ← v.a , and reinitialized to Λ after a sequence of length d .
Revised: Everytime an action a is performed, v is updated such that v ← v.a , and reinitialized to Λ after a sequence of length d .

Original: At each exploration step, instead of sampling a uniformly, EASEE samples a according to the distribution π (cid:63) ( v, · ) .
Revised: At each exploration step, instead of sampling a uniformly, EASEE samples a according to the distribution π (cid:63) ( v, · ) .

Original: Pseudocode for this process can be found in Appendix A.5.
Revised: Pseudocode for this process can be found in Appendix A.5.

Original:  R ESULTS
Revised:  R ESULTS

Original: For every experiments, additional details about environments and hyperparameters are given in
Revised: For every experiments, additional details about environments and hyperparameters are given in

Original: Appendix B.
Revised: Appendix B.

Original: 5. P URE E XPLORATION
Revised: 5. P URE E XPLORATION

Original: To get a better understanding of EASEE, we consider two simple gridworld environments with different structures: CardinalGrid and RotationGrid.
Revised: To get a better understanding of EASEE, we consider two simple gridworld environments with different structures: CardinalGrid and RotationGrid.

Original: These environments are both 100 ×gridworlds, but with different action structures.
Revised: These environments are both 100 ×gridworlds, but with different action structures.

Original: In CardinalGrid, the agent can move one square in the four cardinal directions ( → , ← , ↑ , ↓ ), whereas in RotationGrid, the agent 
Revised: In CardinalGrid, the agent can move one square in the four cardinal directions ( → , ← , ↑ , ↓ ), whereas in RotationGrid, the agent can move either forward one square ( ↑ ), or rotate 90 ◦ on the spot to the left ( (cid:120) ) or to the right ( (cid:121) ).

Revised: The agent starts in the middle of the grid and can explore for 100 timesteps, after which the environment is reset.

Original: In CardinalGrid, we consider the 4 equivalence sets:
Revised: In CardinalGrid, we consider the 4 equivalence sets:

Original: • {→←∼←→} (“ → and ← commute ”) • {→←∼←→ , ↑↓∼↓↑} (“all actions commute”)
Revised: • {→←∼←→} (“ → and ← commute ”) • {→←∼←→ , ↑↓∼↓↑} (“all actions commute”)

Original: • {→←∼←→ , ↑↓∼↓↑ , →←∼ Λ } (“all actions commute and →←∼ Λ ”) • {→←∼←→ , ↑↓∼↓↑ , →←∼ Λ , ↑↓∼ Λ } (“ all actions commute and →←∼↑↓∼ Λ ”),
Revised: • {→←∼←→ , ↑↓∼↓↑ , →←∼ Λ } (“all actions commute and →←∼ Λ ”) • {→←∼←→ , ↑↓∼↓↑ , →←∼ Λ , ↑↓∼ Λ } (“ all actions commute and →←∼↑↓∼ Λ ”),

Original: while in RotationGrid, we consider the three equivalence sets:
Revised: while in RotationGrid, we consider the three equivalence sets:

Original: • { (cid:121)(cid:120) ∼ Λ } • { (cid:121)(cid:120) ∼ Λ , (cid:120)(cid:121) ∼ Λ } • { (cid:121)(cid:120) ∼ Λ , (cid:120)(cid:121) ∼ Λ , (cid:121)(cid:121) ∼ (cid:120)(cid:120) } .
Revised: • { (cid:121)(cid:120) ∼ Λ } • { (cid:121)(cid:120) ∼ Λ , (cid:120)(cid:121) ∼ Λ } • { (cid:121)(cid:120) ∼ Λ , (cid:120)(cid:121) ∼ Λ , (cid:121)(cid:121) ∼ (cid:120)(cid:120) } .

Original: Fig.
Revised: Fig.

Original: 3 shows the beneﬁts of exploiting the structure of the action space for exploration.
Revised: 3 shows the beneﬁts of exploiting the structure of the action space for exploration.

Original: Figures 3a, 3b show the ratio of the number of unique states visited using EASEE over a standard uniform exploration policy.
Revised: Figures 3a, 3b show the ratio of the number of unique states visited using EASEE over a standard uniform exploration policy.

Original: For both environments, a greater equivalence set leads to a more efﬁcient exploration.
Revised: For both environments, a greater equivalence set leads to a more efﬁcient exploration.

Original: In the environmentCardinalGrid for example, for a ﬁxed depth of 6, adding the information that → and ← commute and that every actions commute allow to reach respectively 10% and 60% more states in 100 episodes.
Revised: In the environmentCardinalGrid for example, for a ﬁxed depth of 6, adding the information that → and ← commute and that every actions commute allow to reach respectively 10% and 60% more states in 100 episodes.

Original: Furthermore, extra equivalences encoding that → is the inverse of ← , and ↑ the inverse of ↓ increase the number of new states encountered threefold.
Revised: Furthermore, extra equivalences encoding that → is the inverse of ← , and ↑ the inverse of ↓ increase the number of new states encountered threefold.

Original: It can also be seen that deeper graphs provide better exploration, which is expected: using deeper graphs results in exploiting equivalence priors over longer action sequences.
Revised: It can also be seen that deeper graphs provide better exploration, which is expected: using deeper graphs results in exploiting equivalence priors over longer action sequences.

Original: Figures 3c, 3d show the number of unique states visited with respect to the total number of episodes of exploration.
Revised: Figures 3c, 3d show the number of unique states visited with respect to the total number of episodes of exploration.

Original: We see that EASEE beneﬁts exploration in all conﬁgurations considered: it allows the agent to visit more states within a single trajectory, and as well as across a thousand.
Revised: We see that EASEE beneﬁts exploration in all conﬁgurations considered: it allows the agent to visit more states within a single trajectory, and as well as across a thousand.

Original: It gives insight about the sample-efﬁciency gain which can be achieved using EASEE over a standard random policy.
Revised: It gives insight about the sample-efﬁciency gain which can be achieved using EASEE over a standard random policy.

Original: In the CardinalGrid setting, EASEE visits more unique states over 100 episodes than uniform exploration over 1000.
Revised: In the CardinalGrid setting, EASEE visits more unique states over 100 episodes than uniform exploration over 1000.

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: u
Revised: u

Original: [Figure]
Revised: [Figure]

Original: v is i t e d
Revised: i

Original: [Figure]
Revised: [Figure]

Original: t a t
Revised: i t

Original: [Figure]
Revised: [Figure]

Original: s : r a t

Original: i
Revised: t

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: e li n
Revised: t i o o v

Original: [Figure]
Revised: [Figure]

Revised: -greedy policy.

Original: 5. M INIGRID
Revised: 5. M INIGRID

Original: The Minimalistic Gridworld Environment (MiniGrid) is a suite of environments that test diverse capabilities in RL agents (Chevalier-Boisvert et al., 2018).
Revised: The Minimalistic Gridworld Environment (MiniGrid) is a suite of environments that test diverse capabilities in RL agents (Chevalier-Boisvert et al., 2018).

Original: We evaluated the inﬂuence of adding
Revised: We evaluated the inﬂuence of adding

Original: EASEE to Q-learning on the DoorKey task.
Revised: EASEE to Q-learning on the DoorKey task.

Original: The environment is a gridworld split into two rooms separated by a locked door.
Revised: The environment is a gridworld split into two rooms separated by a locked door.

Original: The agent must collect a key to get to the objective in the other room.
Revised: The agent must collect a key to get to the objective in the other room.

Original: The dynamics of the environment are those of RotationGrid with two extra actions: the agent
Revised: The dynamics of the environment are those of RotationGrid with two extra actions: the agent

Original: may
Revised: may

Original: PICKUP the key when facing it and OPEN the door when carrying the key.
Revised: PICKUP the key when facing it and OPEN the door when carrying the key.

Original: The EASEE version of the Q-learning assumes the following action sequence equivalences:
Revised: The EASEE version of the Q-learning assumes the following action sequence equivalences:

Original: [Equation]
Revised: [Equation]

Original: OPEN ∼ OPEN · OPEN

Original: PICKUP ∼ PICKUP · PICKUP

Original: The reward over this training is presented in Figure 4a.
Revised: The reward over this training is presented in Figure 4a.

Original: Using a depth of 6 , the EASEE augmented version outperforms classic Q-learning.
Revised: Using a depth of 6 , the EASEE augmented version outperforms classic Q-learning.

Original: 5. C ATCHER
Revised: 5. C ATCHER

Original: We test EASEE on a game of Catcher, where the agent must catch a ball falling vertically with a paddle that can move left and right.
Revised: We test EASEE on a game of Catcher, where the agent must catch a ball falling vertically with a paddle that can move left and right.

Original: It receives a reward of +1 when the ball is caught and − 1 when it is missed.
Revised: It receives a reward of +1 when the ball is caught and − 1 when it is missed.

Original: The prior we incorporate into the exploration is that the actions commute i.e. ←→∼→← .
Revised: The prior we incorporate into the exploration is that the actions commute i [Figure] e. ←→∼→← .

Original: For faster learning we restrict each episode to a single ball drop, with the agent starting in the middle of the environment.
Revised: For faster learning we restrict each episode to a single ball drop, with the agent starting in the middle of the environment.

Original: We choose a depth of 30 for EASEE.
Revised: We choose a depth of 30 for EASEE.

Original: This is also the length of a single episode.
Revised: This is also the length of a single episode.

Original: The mean reward over training is plotted in Figure 4b.
Revised: The mean reward over training is plotted in Figure 4b.

Original: [Figure]
Revised: [Figure]

Original: 5. F REEWAY
Revised: 5. F REEWAY

Original: We test our method on the Atari 2600 game Freeway (Bellemare et al., 2013).
Revised: We test our method on the Atari 2600 game Freeway (Bellemare et al., 2013).

Revised: To illustrate EASEE’s performance in a non-deterministic setting, we add stochasticity to the dynamics of the game using sticky actions (Machado et al., 2018).

Original: The agent has to cross a road with multiple lanes without getting hit by the cars, and receives a reward when it reaches safety on the other side.
Revised: The agent has to cross a road with multiple lanes without getting hit by the cars [Figure] and receives a reward when it reaches safety on the other side.

Original: The action space is composed of 3 actions : moving forward of 1 lane ( ↑ ), moving backward of 1 lane ( ↓ ), and passing (–).
Revised: The action space is composed of 3 actions : moving forward of 1 lane ( ↑ ), moving backward of 1 lane ( ↓ ), and passing (–).

Original: As cars arrive randomly, it is not easy to ﬁnd priors on action equivalences in this environment.
Revised: As cars arrive randomly, it is not easy to ﬁnd priors on action equivalences in this environment.

Original: Since passing and moving backwards can sometimes be useful to avoid cars we cannot forbid these actions.
Revised: Since passing and moving backwards can sometimes be useful to avoid cars we cannot forbid these actions.

Original: However, we have prior knowledge that performing these two actions does not lead to visiting new lanes.
Revised: However, we have prior knowledge that performing these two actions does not lead to visiting new lanes.

Original: We restrict the use of these actions with Ω = {↓∼↓↓ , – ↓∼↓ – } , which has the effect of removing every node which is reached by chaining two ↓ actions without moving forward, and compute the exploration policy on the remaining nodes.
Revised: We restrict the use of these actions with Ω = {↓∼↓↓ , – ↓∼↓ – } , which has the effect of removing every node which is reached by chaining two ↓ actions without moving forward, and compute the exploration policy on the remaining nodes.

Original: Results can be seen in Fig.
Revised: Results can be seen in Fig.

Original: 4c.
Revised: 4c.

Original: Interestingly, incorporating such a prior does not lead to better sample-efﬁciency, as in the previous environments, but to a better ﬁnal policy.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We assume that implementers of reinforcement learning agents can provide insights about the environment, despite not knowing its precise dynamics or optimal policy.
Revised: We assume that implementers of reinforcement learning agents can provide insights about the environment, despite not knowing its precise dynamics or optimal policy.

Original: In this work, we argue that some of these insights can be efﬁciently represented using the notion of action-sequence equivalence, which we formalize.
Revised: In this work, we argue that some of these insights can be efﬁciently represented using the notion of action-sequence equivalence, which we formalize.

Original: We propose a method to incorporate such priors in classic Q-learning algorithms and demonstrate empirically its ability to improve sample efﬁciency and performance.
Revised: We propose a method to incorporate such priors in classic Q-learning algorithms and demonstrate empirically its ability to improve sample efﬁciency and performance.

Original: More precisely, our approach can be divided into two steps: ﬁrst, the construction of a graph representing the local dynamics, and then the resolution of a convex optimization problem aiming to balance node visitation.
Revised: More precisely, our approach can be divided into two steps: ﬁrst, the construction of a graph representing the local dynamics, and then the resolution of a convex optimization problem aiming to balance node visitation.

Original: We show that incorporating such prior knowledge can replace standard (cid:15) -greedy and improve at little cost RL algorithms, which traditionally start from a tabula rasa setting, learning everything from scratch.
Revised: We show that incorporating such prior knowledge can replace standard (cid:15) -greedy and improve at little cost RL algorithms, which traditionally start from a tabula rasa setting, learning everything from scratch.

Original: We expect EASEE to be robust to slight errors in the action sequence equivalence set.
Revised: We expect EASEE to be robust to slight errors in the action sequence equivalence set.

Original: It may be that the states at the end of two sequences are not exactly the same but very similar, or that an action-sequence equivalence is veriﬁed at all but a few states.
Revised: It may be that the states at the end of two sequences are not exactly the same but very similar, or that an action-sequence equivalence is veriﬁed at all but a few states.

Original: In such cases, the exploration policy we determine is not optimal, but should be much closer to optimality than uniformly sampling actions.
Revised: In such cases, the exploration policy we determine is not optimal, but should be much closer to optimality than uniformly sampling actions.

Original: We additionally experimented with EASEE on two other Atari games, where the improvements are less pronounced: the details are given in Appendix B.4, as well as possible explanations.
Revised: We additionally experimented with EASEE on two other Atari games, where the improvements are less pronounced: the details are given in Appendix B.4, as well as possible explanations.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: In an effort to help with reproducibility, we provide a light-weight implementation of the experiments discussed in sections 5.1 and 5.2.
Revised: In an effort to help with reproducibility, we provide a light-weight implementation of the experiments discussed in sections 5.1 and 5.2.

Original: For Catcher and Atari environments, all details of the environments, as well as the preprocessing steps and the hyperparameters we chose are described in Appendix B. and Appendix B.3 respectively.
Revised: For Catcher and Atari environments, all details of the environments, as well as the preprocessing steps and the hyperparameters we chose are described in Appendix B. and Appendix B.3 respectively.

Original: Additionally, the pseudo-code for the proposed methods is given in
Revised: Additionally, the pseudo-code for the proposed methods is given in

Original: Appendix A.3.
Revised: Appendix A.3.

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: A direct application of this paper is to make reinforcement learning techniques in general, and deep
Revised: A direct application of this paper is to make reinforcement learning techniques in general, and deep

Original: Q networks in particular, more efﬁcient at solving Markov decision processes by making use of prior knowledge about action sequence equivalences.
Revised: Q networks in particular, more efﬁcient at solving Markov decision processes by making use of prior knowledge about action sequence equivalences.

Original: On its own, we do not expect EASEE to have immediate societal risks.
Revised: On its own, we do not expect EASEE to have immediate societal risks.

Original: Moreover, approaches incorporating prior knowledge in reinforcement learning algorithms often increase sample efﬁciency at little cost, thereby leading to less expensive and more environmentally-friendly methods.
Revised: Moreover, approaches incorporating prior knowledge in reinforcement learning algorithms often increase sample efﬁciency at little cost, thereby leading to less expensive and more environmentally-friendly methods.

Original: A T ECHNICAL E LEMENTS AND P ROOFS A. P ROOF OF P ROPOSITION
Revised: A T ECHNICAL E LEMENTS AND P ROOFS A. P ROOF OF P ROPOSITION

Original: Proposition 1.
Revised: Proposition 1.

Original: If we have two pairs of equivalent sequences over M , i.e. w 1 , w 2 , w 3 , w 4 ∈ A (cid:63) such that
Revised: If we have two pairs of equivalent sequences over M , i.e. w 1 , w 2 , w 3 , w 4 ∈ A (cid:63) such that

Original: [Equation]
Revised: [Equation]

Original: then the concatenation of the sequences are also equivalent sequences:
Revised: then the concatenation of the sequences are also equivalent sequences:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For any s ∈ S , we have T ( s, w 1 ) =
Revised: For any s ∈ S , we have T ( s, w 1 ) =

Original: T ( s, w 2 ) as w 1 ∼ w 2 .
Revised: T ( s, w 2 ) as w 1 ∼ w 2 .

Original: We apply the same property for w 3 and w 4 on the state T ( s, w 1 ) :
Revised: We apply the same property for w 3 and w 4 on the state T ( s, w 1 ) :

Original: [Equation]
Revised: [Equation]

Original: Therefore w 1 .w 3 ∼ w 2 .w 4 .
Revised: Therefore w 1 .w 3 ∼ w 2 .w 4 .

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Theorem 1.
Revised: Theorem 1.

Original: Given an equivalence set Ω , ∼ Ω is an equivalence relationship.
Revised: Given an equivalence set Ω , ∼ Ω is an equivalence relationship.

Original: Furthermore, for v, w ∈ A (cid:63) , v ∼ Ω w ⇒ v ∼ w .
Revised: Furthermore, for v, w ∈ A (cid:63) , v ∼ Ω w ⇒ v ∼ w .

Original: ∼ Ω is an equivalence relation Let u, v, w ∈ A (cid:63) .
Revised: ∼ Ω is an equivalence relation Let u, v, w ∈ A (cid:63) .

Original: Proof.
Revised: Proof.

Original: • We immediately have v ∼ 1Ω v by choosing v 1 = Λ in equation 3, and therefore v ∼ Ω v , thus ∼ Ω is reﬂexive.
Revised: • We immediately have v ∼ 1Ω v by choosing v 1 = Λ in equation 3, and therefore v ∼ Ω v , thus ∼ Ω is reﬂexive.

Original: • It is clear from its deﬁnition that ∼ 1Ω is symmetric, as ∼ is symmetric.
Revised: • It is clear from its deﬁnition that ∼ 1Ω is symmetric, as ∼ is symmetric.

Original: Then, we suppose that v ∼ Ω w .
Revised: Then, we suppose that v ∼ Ω w .

Original: We have n ∈ N and v 1 , . .
Revised: We have n ∈ N and v 1 , . .

Original: , v n ∈ A (cid:63)
Revised: , v n ∈ A (cid:63)

Original: such that v ∼ 1Ω v 1 ∼ 1Ω · · · ∼ 1Ω v n ∼ 1Ω w , therefore w ∼ 1Ω v n ∼ 1Ω · · · ∼ 1Ω v 1 ∼ 1Ω v , thus w ∼ Ω v .
Revised: such that v ∼ 1Ω v 1 ∼ 1Ω · · · ∼ 1Ω v n ∼ 1Ω w , therefore w ∼ 1Ω v n ∼ 1Ω · · · ∼ 1Ω v 1 ∼ 1Ω v , thus w ∼ Ω v .

Original: Hence ∼ Ω is symmetric.
Revised: Hence ∼ Ω is symmetric.

Original: • If u ∼ Ω v and v ∼ Ω w , we have n 1 , n 2 ∈ N , and u 1 , . . .
Revised: • If u ∼ Ω v and v ∼ Ω w , we have n 1 , n 2 ∈ N , and u 1 , . . .

Original: , u n 1 ∈ A (cid:63) , v 1 , . .
Revised: , u n 1 ∈ A (cid:63) , v 1 , . .

Original: , v n 2 ∈ A (cid:63) , such that u ∼ 1Ω u 1 ∼ 1Ω · · · ∼ 1Ω u n 1 ∼ 1Ω v and v ∼ 1Ω v 1 ∼ 1Ω · · · ∼ 1Ω v
Revised: , v n 2 ∈ A (cid:63) , such that u ∼ 1Ω u 1 ∼ 1Ω · · · ∼ 1Ω u n 1 ∼ 1Ω v and v ∼ 1Ω v 1 ∼ 1Ω · · · ∼ 1Ω v

Original: n 2 ∼ 1Ω w .
Revised: n 2 ∼ 1Ω w .

Original: It is then clear that u ∼ 1Ω u 1 ∼ 1Ω · · · ∼ 1Ω u n 1 ∼ 1Ω v ∼ 1Ω v 1 ∼ 1Ω · · · ∼ 1Ω v n 2 ∼ 1Ω w , and thus u ∼ Ω w .
Revised: It is then clear that u ∼ 1Ω u 1 ∼ 1Ω · · · ∼ 1Ω u n 1 ∼ 1Ω v ∼ 1Ω v 1 ∼ 1Ω · · · ∼ 1Ω v n 2 ∼ 1Ω w , and thus u ∼ Ω w .

Original: Therefore ∼ Ω is transitive.
Revised: Therefore ∼ Ω is transitive.

Original: The relation ∼ Ω is reﬂexive, symmetric and transitive.
Revised: The relation ∼ Ω is reﬂexive, symmetric and transitive.

Original: Therefore it is an equivalence relation.
Revised: Therefore it is an equivalence relation.

Original: ∼ Ω implies ∼
Revised: ∼ Ω implies ∼

Original: Proof.
Revised: Proof.

Original: Let v, w ∈ A (cid:63) .
Revised: Let v, w ∈ A (cid:63) .

Original: From Proposition 1, we immediately get v ∼ 1Ω w ⇒ v ∼ M w .
Revised: From Proposition 1, we immediately get v ∼ 1Ω w ⇒ v ∼ M w .

Original: Then we can prove by immediate induction that ∀ n ∈ N , v 1 , . .
Revised: Then we can prove by immediate induction that ∀ n ∈ N , v 1 , . .

Original: , v n ∈ A (cid:63) , v ∼ 1Ω v 1 ∼ 1Ω · · · ∼ 1Ω v n ∼ 1Ω w ⇒ v ∼ w , from which we deduce v ∼ Ω w implies v ∼ w .
Revised: , v n ∈ A (cid:63) , v ∼ 1Ω v 1 ∼ 1Ω · · · ∼ 1Ω v n ∼ 1Ω w ⇒ v ∼ w , from which we deduce v ∼ Ω w implies v ∼ w .

Original: /* create a node
Revised: /* create a node

Original: ; if
Revised: ; if

Original: /* Because of sequence redundancies, the state may already appear in the graph.
Revised: /* Because of sequence redundancies, the state may already appear in the graph.

Original: /* Update the equivalences E (newState) to account for the new ways of reaching newState */
Revised: /* Update the equivalences E (newState) to account for the new ways of reaching newState */

Original: E ← UpdateDic (newState, E , Ω ) ; end end
Revised: E ← UpdateDic (newState, E , Ω ) ; end end

Original: l ← l + 1 ;
Revised: l ← l + 1 ;

Original: S ← newStates ; end /
Revised: S ← newStates ; end /

Original: * Prune edges such that the resulting graph is a DAG.
Revised: * Prune edges such that the resulting graph is a DAG.

Original: T = GraphToDAG ( G ) ;
Revised: T = GraphToDAG ( G ) ;

Original: Output DAG G ;
Revised: Output DAG G ;

Original: A. G RAPH C ONSTRUCTION A LGORITHM
Revised: A. G RAPH C ONSTRUCTION A LGORITHM

Original: We present in Algorithm 1 an overview of the graph construction algorithm.
Revised: We present in Algorithm 1 an overview of the graph construction algorithm.

Original: It takes as input the action set A , the sequence equivalence set Ω , and the desired depth d , and outputs a DAG.
Revised: It takes as input the action set A , the sequence equivalence set Ω , and the desired depth d , and outputs a DAG.

Original: Informally, it starts from a graph G =
Revised: Informally, it starts from a graph G =

Original: ( V, E ) reduced to a root state { 0 } and iteratively expands G until a distance
Revised: ( V, E ) reduced to a root state { 0 } and iteratively expands G until a distance

Original: L to the root is reached.
Revised: L to the root is reached.

Original: For a node n ∈ V we store in E ( v ) sequences which reach v , and are preﬁxes of sequences of Ω .
Revised: For a node n ∈ V we store in E ( v ) sequences which reach v , and are preﬁxes of sequences of Ω .

Original: When expanding a state v ∈ V using an action a ∈ A (Line. 12), we look at every partial sequence s ∈ E ( v ) .
Revised: When expanding a state v ∈ V using an action a ∈ A (Line. 12), we look at every partial sequence s ∈ E ( v ) .

Original: If s.a is in Ω , it means that we have found a redundant sequence.
Revised: If s.a is in Ω , it means that we have found a redundant sequence.

Original: If the equivalent sequence has already been computed, it means that a node u representing
Revised: If the equivalent sequence has already been computed, it means that a node u representing

Original: T ( v, s.a ) has previously been added in G .
Revised: T ( v, s.a ) has previously been added in G .

Original: Otherwise, we add a new node u .
Revised: Otherwise, we add a new node u .

Original: In both case, we update the equivalences E ( u ) to account for the new ways of reaching u (Line. 18).
Revised: In both case, we update the equivalences E ( u ) to account for the new ways of reaching u (Line. 18).

Original: A. G RAPH C ONSTRUCTION C OMPLEXITY
Revised: A. G RAPH C ONSTRUCTION C OMPLEXITY

Original: As shown in Section 3.2, constructing the graph necessitates three intricate loops: The ﬁrst one goes over every internal node n ∈ V , the second one loops over the set of actions A , and the last one loops over every partial sequence which allows to reach v from a parent node.
Revised: As shown in Section 3.2, constructing the graph necessitates three intricate loops: The ﬁrst one goes over every internal node n ∈ V , the second one loops over the set of actions A , and the last one loops over every partial sequence which allows to reach v from a parent node.

Original: Inside these three loops, one has to compare the partial sequence with every sequence of Ω .
Revised: Inside these three loops, one has to compare the partial sequence with every sequence of Ω .

Original: As sequence length in Ω can be bounded by d , the complexity cost inside the tree loops is bounded by O ( | Ω | d ) .
Revised: As sequence length in Ω can be bounded by d , the complexity cost inside the tree loops is bounded by O ( | Ω | d ) .

Original: The total complexity is therefore lower than O ( | V ||A||A| d − 1 | Ω | d ) =
Revised: The total complexity is therefore lower than O ( | V ||A||A| d − 1 | Ω | d ) =

Original: O ( | V ||A| d | Ω | d ) .
Revised: O ( | V ||A| d | Ω | d ) .

Original: As | V | ≤ |A| d , the complexity can also be bounded by O ( |A| 2 d | Ω | d ) .
Revised: As | V | ≤ |A| d , the complexity can also be bounded by O ( |A| 2 d | Ω | d ) .

Original: A. M ODIFIED DQN
Revised: A. M ODIFIED DQN

Original: Our modiﬁed version of the DQN algorithm can be found at Algorithm 2.
Revised: Our modiﬁed version of the DQN algorithm can be found at Algorithm 2.

Original: Algorithm 2: Modiﬁed DQN
Revised: Algorithm 2: Modiﬁed DQN

Original: Initialize replay
Revised: Initialize replay

Original: A. P OSSIBLE E XTENSION TO THE S TOCHASTIC C ASE
Revised: A. P OSSIBLE E XTENSION TO THE S TOCHASTIC C ASE

Original: In this section we discuss the possibility of extending EASEE to the case of MDPs with stochastic transitions.
Revised: In this section we discuss the possibility of extending EASEE to the case of MDPs with stochastic transitions.

Original: EASEE relies on three components: the formalization of action sequence equivalences (Def. 1), the construction of a local-dynamics graph (Section 3.2), and the construction of a local exploration policy by solving a convex problem (Section 4.2).
Revised: EASEE relies on three components: the formalization of action sequence equivalences (Def. 1), the construction of a local-dynamics graph (Section 3.2), and the construction of a local exploration policy by solving a convex problem (Section 4.2).

Original: We now detail for each step the necessary changes to adapt EASEE to M =
Revised: We now detail for each step the necessary changes to adapt EASEE to M =

Original: ( S , A , T, R, γ ) , a MDP with stochastic dynamics.
Revised: ( S , A , T, R, γ ) , a MDP with stochastic dynamics.

Original: • Action Sequence equivalences : the difference with the deterministic case here is that given an action a ∈ A and a state s ∈ S , T ( s, a ) is not a state but a distribution over the set of states S .
Revised: • Action Sequence equivalences : the difference with the deterministic case here is that given an action a ∈ A and a state s ∈ S , T ( s, a ) is not a state but a distribution over the set of states S .

Original: Therefore every equality considered in Section 3.1 has now to be understood not as an equality between two states but between two distributions.
Revised: Therefore every equality considered in Section 3.1 has now to be understood not as an equality between two states but between two distributions.

Original: Other than this the formalism can be kept identical.
Revised: Other than this the formalism can be kept identical.

Original: Intuitively, two sequences of actions are equivalent if they lead to the same state distribution from any given state, i.e. if they produce the same effect everywhere.
Revised: Intuitively, two sequences of actions are equivalent if they lead to the same state distribution from any given state, i.e. if they produce the same effect everywhere.

Original: • Local-Dynamics Graph : Here, the formalism can again be kept identical.
Revised: • Local-Dynamics Graph : Here, the formalism can again be kept identical.

Original: A node in the local-dynamics graph will not represent a state anymore, but rather a distribution over S .
Revised: A node in the local-dynamics graph will not represent a state anymore, but rather a distribution over S .

Original: • Local Exploration Policy : Solving directly the objective given in equation 4 would lead to maximize the diversity among state distributions encountered.
Revised: • Local Exploration Policy : Solving directly the objective given in equation 4 would lead to maximize the diversity among state distributions encountered.

Original: As is, it would not necessarily lead to a better diversity among states, as two different distributions can have an almost similar support.
Revised: As is, it would not necessarily lead to a better diversity among states, as two different distributions can have an almost similar support.

Original: Therefore, adapting EASEE to a stochastic setting would require encoding additional priors about the distributions represented by the nodes of the local-dynamics graph, which we leave for future work.
Revised: Therefore, adapting EASEE to a stochastic setting would require encoding additional priors about the distributions represented by the nodes of the local-dynamics graph, which we leave for future work.

Original: If we suppose that the distributions encountered have disjoint supports, and that their entropy is the same, EASEE can be applied without modiﬁcation.
Revised: If we suppose that the distributions encountered have disjoint supports, and that their entropy is the same, EASEE can be applied without modiﬁcation.

Original: B E XPERIMENTAL D ETAILS B. G RIDWORLDS
Revised: B E XPERIMENTAL D ETAILS B. G RIDWORLDS

Original: We tested EASEE on the DoorKey task.
Revised: We tested EASEE on the DoorKey task.

Original: An illustration of the initial state is given in Fig.
Revised: An illustration of the initial state is given in Fig.

Original: The agent is represented by the red triangle.
Revised: The agent is represented by the red triangle.

Original: The yellow key is necessary to open the yellow door.
Revised: The yellow key is necessary to open the yellow door.

Original: The two room are respectively 12 × 17 and 4 × 17 grids.
Revised: The two room are respectively 12 × 17 and 4 × 17 grids.

Original: The agent has 3249 timesteps to reach the goal and receive a reward of 1 before the environment is reset.
Revised: The agent has 3249 timesteps to reach the goal and receive a reward of 1 before the environment is reset.

Original: [Figure]
Revised: [Figure]

Original: The paddle is 1 block wide.
Revised: The paddle is 1 block wide.

Original: The environment is 60 blocks wide and 30 blocks high.
Revised: The environment is 60 blocks wide and 30 blocks high.

Original: The ball and the paddle both move at a rate of 1 block per timestep, so each episode lasts 30 timesteps.
Revised: The ball and the paddle both move at a rate of 1 block per timestep, so each episode lasts 30 timesteps.

Original: We use the same architecture for the DQN with and without EASEE.
Revised: We use the same architecture for the DQN with and without EASEE.

Original: Each observation is a 60 ×image.
Revised: Each observation is a 60 ×image.

Original: The feature extractor network is a CNN composed of 3 convolution layers with kernel sizefollowed by ReLU activation.
Revised: The feature extractor network is a CNN composed of 3 convolution layers with kernel sizefollowed by ReLU activation.

Original: In both cases, we update the online network every 4 timesteps, and the target network every 10 3 timesteps.
Revised: In both cases, we update the online network every 4 timesteps, and the target network every 10 3 timesteps.

Original: We use a replay buffer of size 10 4 , and sample batches of size 32.
Revised: We use a replay buffer of size 10 4 , and sample batches of size 32.

Original: We use the Adam optimizer with a learning rate of 10 − 4 .
Revised: We use the Adam optimizer with a learning rate of 10 − 4 .

Original: We train for 3 .
Revised: We train for 3 .

Original: 10 5 timesteps.
Revised: 10 5 timesteps.

Original: The exploration parameter (cid:15) is linearly annealed from 1 down to 0.
Revised: The exploration parameter (cid:15) is linearly annealed from 1 down to 0.

Original: over 20% of the training period.
Revised: over 20% of the training period.

Original: Other DQN hyperparameters were defaults in Rafﬁn et al.
Revised: Other DQN hyperparameters were defaults in Rafﬁn et al.

Original: [Figure]
Revised: [Figure]

Original: . F REEWAY

Original: Environment In Freeway, the agent has to cross a road with multiple lanes without getting hit by the cars.
Revised: Environment In Freeway, the agent has to cross a road with multiple lanes without getting hit by the cars.

Original: It only receives a reward when it safely reaches the other side of the road.
Revised: It only receives a reward when it safely reaches the other side of the road.

Original: An illustration is given in Fig.
Revised: An illustration is given in Fig.

Original: The agent is represented by the yellow chicken.
Revised: The agent is represented by the yellow chicken.

Original: We use the default preprocessing in Rafﬁn et al.
Revised: To add stochasticity to the dynamic of the environment, we use sticky actions as proposed in Machado et al.

Original: (2019), which follows guidelines of Bellemare et al.
Revised: (2018), with a stickiness parameter of 0.25.

Original: More precisely, the environment is initialized with a random number of up to 30 no-op actions.
Revised: More precisely, the environment has 0.25 probability of executing the previous action again instead of the current desired action.

Original: The frame is recast as a 84 × 84 × 3 image, and the number of frames to skip between each observation is set to 4.
Revised: The frame is recast as a 84 × 84 × 3 image, and the number of frames to skip between each observation is set to 4.

Original: The reward is scaled to [-1, 1].
Revised: The reward is scaled to [-1, 1].

Original: An observation corresponds to 4 stacked game frames.
Revised: An observation corresponds to 4 stacked game frames.

Original: Architecture and hyperparameters We use the same architecture for the DQN with and without
Revised: Architecture and hyperparameters We use the same architecture for the DQN with and without

Original: EASEE.
Revised: EASEE.

Original: Input images ﬁrst go through a convolutional neural network, with the same architecture as in Mnih et al.
Revised: Input images ﬁrst go through a convolutional neural network, with the same architecture as in Mnih et al.

Original: We update the online network every 4 timesteps, and the target network every 10 3 timesteps.
Revised: We update the online network every 4 timesteps, and the target network every 10 3 timesteps.

Original: We use a replay buffer of size 10 5 , and sample batches of size 32.
Revised: We use a replay buffer of size 10 5 , and sample batches of size 32.

Original: We use the Adam optimizer with a learning rate of 10 − 4 .
Revised: We use the Adam optimizer with a learning rate of 10 − 4 .

Original: We train for 10 7 timesteps.
Revised: We train for 10 7 timesteps.

Original: The exploration parameter (cid:15) is linearly annealed from 1 down to 0.
Revised: The exploration parameter (cid:15) is linearly annealed from 1 down to 0.

Original: over 10% of the training period, which are the default in Rafﬁn (2018) for Atari games.
Revised: over 10% of the training period, which are the default in Rafﬁn (2018) for Atari games.

Original: Other DQN hyperparameters were defaults in Rafﬁn et al.
Revised: Other DQN hyperparameters were defaults in Rafﬁn et al.

Original: B. A DDITIONAL E XPERIMENTS
Revised: B. A DDITIONAL E XPERIMENTS

Original: Environments We experimented EASEE on two other Atari environments, where the action sequence structures are less straight-forward.
Revised: Environments We experimented EASEE on two other Atari environments, where the action sequence structures are less straight-forward.

Original: The three environments are preprocessed as explained in AppendixB.3.
Revised: The three environments are preprocessed as explained in AppendixB

Original: • Boxing : This game shows a top-down view of two boxers.
Revised: • Boxing : This game shows a top-down view of two boxers.

Original: The player can move in all four directions, and punch his opponent (pressing the “ FIRE ” button).
Revised: The player can move in all four directions, and punch his opponent (pressing the “ FIRE ” button).

Original: The action space is composed of 18 actions : NOOP , FIRE , UP , RIGHT, LEFT , DOWN , UPRIGHT , UPLEFT , DOWNRIGHT , DOWNLEFT , UPFIRE , RIGHTFIRE , LEFTFIRE , DOWNFIRE , UPRIGHTFIRE , UPLEFTFIRE , DOWNRIGHTFIRE , DOWNLEFTFIRE . We incorporated priors by decomposing actions, in the form of UPRIGHT ∼ UP . RIGHT , UPLEFT ∼ UP .
Revised: The action space is composed of 18 actions : NOOP , FIRE , UP , RIGHT

Original: LEFT , UPRIGHTFIRE
Revised: LEFT DOWN

Original: UPRIGHT
Revised: UPRIGHT

Original: FIRE , UPLEFTFIRE ∼ UPLEFT .
Revised: RIGHTFIRE We incorporated priors by

Original: FIRE , etc ... • Carnival : The goal of the game is to shoot at targets, which include rabbits, ducks, owls, scroll across the screen in alternating directions, and sometimes come at the player.
Revised: • Carnival : The goal of the game is to shoot at targets, which include rabbits, ducks, owls, scroll across the screen in alternating directions, and sometimes come at the player.

Original: The player can only move in 1 direction, such that the action space is composed of 6 actions: [ NOOP , FIRE , RIGHT , LEFT , RIGHTFIRE , LEFTFIRE ].
Revised: The player can only move in 1 direction, such that the action space is composed of 6 actions: [ NOOP , FIRE , RIGHT , LEFT , RIGHTFIRE , LEFTFIRE ].

Original: As NOOP is not an useful action, EASEE could get an edge simply by adding the equivalence NOOP ∼ Λ .
Revised: As NOOP is not an useful action, EASEE could get an edge simply by adding the equivalence NOOP ∼ Λ .

Original: For a fair
Revised: For a fair

Original: [Figure]
Revised: [Figure]

Original: comparison, we restricted the action space to meaningful actions by removing for both and the baseline.
Revised: comparison, we restricted the action space to meaningful actions by removing for both and the baseline.

Original: We limited ourselves to the commutative property of
Revised: We limited ourselves to the commutative property of

Original: Architecture and hyperparameters We use the same architecture and the exact same DQN parameters as in AppendixB.3.
Revised: Architecture and hyperparameters We use the same architecture and the exact same DQN parameters as in AppendixB.3.

Original: In all three environments, EASEE is used with a depth of 4.
Revised: In all three environments, EASEE is used with a depth of 4.

Original: Results We can see the results on Fig.7.
Revised: Results We can see the results on Fig.7.

Original: We can see a slight gain for Boxing, and a marginal improvement for Carnival.
Revised: We can see a slight gain for Boxing, and a marginal improvement for Carnival.

Original: This can come from various factors:
Revised: This can come from various factors:

Original: • When the number of action sequence equivalences considered is small compared to the number of actions, as it is the case for Carnival, the exploration policy computed with EASEE is very much like a uniform policy.
Revised: • When the number of action sequence equivalences considered is small compared to the number of actions, as it is the case for Carnival, the exploration policy computed with EASEE is very much like a uniform policy.

Original: It logically makes
Revised: It logically makes

Original: its
Revised: its

Original: performances converge toward those of a standard DQN.
Revised: performances converge toward those of a standard DQN.

Original: • The action sequence equivalences considered here are only approximately true.
Revised: • The action sequence equivalences considered here are only approximately true.

Original: In boxing, it is only approximately true that UPRIGHT ∼ UP .
Revised: In boxing, it is only approximately true that UPRIGHT ∼ UP .

Original: RIGHT for example.
Revised: RIGHT for example.

Original: In Carnival, RIGHT and LEFT commute as long as the player is not at the edges of the screen, in which case RIGHT or LEFT could have no effect.
Revised: In Carnival, RIGHT and LEFT commute as long as the player is not at the edges of the screen, in which case RIGHT or LEFT could have no effect.

Original: In both cases, this induces a bias that may harm performance.
Revised: In both cases, this induces a bias that may harm performance.