Original: The Importance of Non-Markovianity in Maximum State Entropy Exploration
Revised: The Importance of Non-Markovianity in Maximum State Entropy Exploration

Original: Abstract
Revised: Abstract

Original: In the maximum state entropy exploration framework, an agent interacts with a reward-free environment to learn a policy that maximizes the entropy of the expected state visitations it is inducing.
Revised: In the maximum state entropy exploration framework, an agent interacts with a reward-free environment to learn a policy that maximizes the entropy of the expected state visitations it is inducing.

Original: (Hazan et al., 2019) noted that the class of
Revised: (Hazan et al., 2019) noted that the class of

Original: Markovian stochastic policies is sufﬁcient for the maximum state entropy objective, and exploiting non-Markovianity is generally considered pointless in this setting.
Revised: Markovian stochastic policies is sufﬁcient for the maximum state entropy objective, and exploiting non-Markovianity is generally considered pointless in this setting.

Original: In this paper, we argue that non-Markovianity is instead paramount for maximum state entropy exploration in a ﬁnite-sample regime.
Revised: In this paper, we argue that non-Markovianity is instead paramount for maximum state entropy exploration in a ﬁnite-sample regime.

Original: Especially, we recast the objective to target the expected entropy of the induced state visitations in a single trial.
Revised: Especially, we recast the objective to target the expected entropy of the induced state visitations in a single trial.

Original: Then, we show that the class of nonMarkovian deterministic policies is sufﬁcient for the introduced objective, while Markovian policies suffer non-zero regret in general.
Revised: Then, we show that the class of nonMarkovian deterministic policies is sufﬁcient for the introduced objective, while Markovian policies suffer non-zero regret in general.

Original: However, we prove that the problem of ﬁnding an optimal non-Markovian policy is NP-hard.
Revised: However, we prove that the problem of ﬁnding an optimal non-Markovian policy is NP-hard.

Original: Despite this negative result, we discuss avenues to address the problem in a tractable way and how nonMarkovian exploration could beneﬁt the sample efﬁciency of online reinforcement learning in future works.
Revised: Despite this negative result, we discuss avenues to address the problem in a tractable way and how nonMarkovian exploration could beneﬁt the sample efﬁciency of online reinforcement learning in future works.

Original: 1. Introduction
Revised: 1. Introduction

Original: Several recent works have addressed Maximum State Entropy (MSE) exploration (Hazan et al., 2019; Tarbouriech & Lazaric, 2019; Lee et al., 2019; Mutti & Restelli, 2020;
Revised: Several recent works have addressed Maximum State Entropy (MSE) exploration (Hazan et al., 2019; Tarbouriech & Lazaric, 2019; Lee et al., 2019; Mutti & Restelli, 2020;

Original: Mutti et al., 2020; Zhang et al., 2020a; Guo et al., 2021;
Revised: Mutti et al. , 2021b;a; Zhang et al., 2020a; Guo et al., 2021;

Original: Liu & Abbeel, 2021; Seo et al.,  2021) as a pre-training objective for online Reinforcement Learning (RL) (Sutton & Barto, 2018).
Revised: Liu & Abbeel, 2021b;a; Seo et al., 2021; Yarats et al., 2021) as a pre-training objective for online Reinforcement Learning (RL) (Sutton & Barto, 2018).

Original: In this line of work, an agent interacts with a reward-free environment (Jin et al., 2020) in order to learn a general exploration strategy.
Revised: In this line of work, an agent interacts with a reward-free environment (Jin et al., Milan, Italy 2 Università di Bologna, Bologna, Italy 3 ETH Zurich, Zurich, Switzerland.

Revised: Correspondence to:

Original:  The aim of this strategy is to improve the sample efﬁciency of any RL task that could be speciﬁed over the same environment afterwards, serving as an exploratory initialization to standard learning techniques, such as Q-learning (Watkins & Dayan, 1992) or policy gradient (Peters & Schaal, 2008).
Revised: Mirco Mutti < mirco.mutti@polimi.it >. Unsupervised Reinforcement Learning Workshop at ICML 2021. 2020) in order to learn a general exploration strategy. The aim of this strategy is to improve the sample efﬁciency of any RL task that could be speciﬁed over the same environment afterwards, serving as an exploratory initialization to standard learning techniques, such as Q-learning (Watkins & Dayan, 1992) or policy gradient (Peters & Schaal, 2008).

Original: To learn this strategy, the agent maximizes an entropic measure of the state distribution induced by its behavior over the environment, effectively targeting a uniform exploration of the state space.
Revised: To learn this strategy, the agent maximizes an entropic measure of the state distribution induced by its behavior over the environment, effectively targeting a uniform exploration of the state space.

Original: In tabular domains, optimizing this kind of MSE objective is known to be provably efﬁcient (Hazan et al., 2019; Zhang et al., 2020b), whereas the obtained exploratory strategy lead to outstanding empirical results in continuous and highdimensional domains as well, (e.g., Mutti et al., 2020; Liu & Abbeel, 2021), especially w.r.t.
Revised: In tabular domains, optimizing this kind of MSE objective is known to be provably efﬁcient (Hazan et al., 2019; Zhang et al., 2020b), whereas the obtained exploratory strategy lead to outstanding empirical results in continuous and high-dimensional domains as well, (e.g., Mutti et al., 2021b; Liu & Abbeel, 2021b), especially w.r.t.

Original: RL from scratch.
Revised: RL from scratch.

Original: All of the existing works pursuing a MSE objective solely focus on Markovian exploration strategies, in which each decision is conditioned on the current state of the environment rather than the full history of interactions.
Revised: All of the existing works pursuing a MSE objective solely focus on Markovian exploration strategies, in which each decision is conditioned on the current state of the environment rather than the full history of interactions.

Original: This choice is common in RL, as it is well-known that an optimal deterministic Markovian strategy maximizes the usual cumulative sum of rewards objective (Puterman, 2014).
Revised: This choice is common in RL, as it is well-known that an optimal deterministic Markovian strategy maximizes the usual cumulative sum of rewards objective (Puterman, 2014).

Original: Similarly, (Hazan et al., 2019, Lemma 3.3) note that the class of
Revised: Similarly, (Hazan et al., 2019, Lemma 3.3) note that the class of

Original: Markovian strategies is sufﬁcient for the standard MSE objective.
Revised: Markovian strategies is sufﬁcient for the standard MSE objective.

Original: Indeed, a carefully constructed Markovian strategy is able to induce the same state distribution of any historybased (non-Markovian) one by exploiting randomization.
Revised: Indeed, a carefully constructed Markovian strategy is able to induce the same state distribution of any historybased (non-Markovian) one by exploiting randomization.

Original: Crucially, this result does not hold only for asymptotic state distributions (Puterman, 2014), but also for state distributions that are marginalized over a ﬁnite horizon.
Revised: Crucially, this result does not hold only for asymptotic state distributions (Puterman, 2014), but also for state distributions that are marginalized over a ﬁnite horizon.

Original: As a matter of fact, there is little incentive to consider more complicated strategies as they are not providing any beneﬁt on the value of the entropy objective.
Revised: As a matter of fact, there is little incentive to consider more complicated strategies as they are not providing any beneﬁt on the value of the entropy objective.

Original: However, the intuition suggests that exploiting the history of the interactions is useful when the agent’s goal is to uniformly explore the environment: If you know what you have visited already, you can take decisions accordingly.
Revised: However, the intuition suggests that exploiting the history of the interactions is useful when the agent’s goal is to uniformly explore the environment: If you know what you have visited already, you can take decisions accordingly.

Original: To this point, let us consider an illustrative example in which the agent ﬁnds itself in the middle of a two-rooms domain (as depicted in Figure 1), having a budget of interactions that is just enough to visit every state within a single episode.
Revised: To this point, let us consider an illustrative example in which the agent ﬁnds itself in the middle of a two-rooms domain (as depicted in Figure 1), having a budget of interactions that is just enough to visit every state within a single episode.

Original: It is easy to see that an optimal Markovian strategy for the MSE objective would randomize between going left and right in the initial position, and then would follow the optimal route within a room, ﬁnally ending in the initial position again.
Revised: It is easy to see that an optimal Markovian strategy for the MSE objective would randomize between going left and right in the initial position, and then would follow the optimal route within a room, ﬁnally ending in the initial position again.

Original: An episode either results in visiting the left room twice, or the right room twice, or each room once, and all of this outcomes have the same probability.
Revised: An episode either results in visiting the left room twice, or the right room twice, or each room once, and all of this outcomes have the same probability.

Original: Thus, the agent might explore poorly when considering a single episode, but the exploration is uniform in the average of inﬁnite trials .
Revised: Thus, the agent might explore poorly when considering a single episode, but the exploration is uniform in the average of inﬁnite trials .

Original: Arguably, this is quite different from how a human being would tackle this problem, i.e., taking intentional decisions in the middle position to visit a room before going to the other.
Revised: Arguably, this is quite different from how a human being would tackle this problem, i.e., taking intentional decisions in the middle position to visit a room before going to the other.

Original: This strategy leads to uniform exploration of the environment in any trial , but it is inherently non-Markovian.
Revised: This strategy leads to uniform exploration of the environment in any trial , but it is inherently non-Markovian.

Original: Backed by this intuition, we argue that prior work does not recognize the importance of non-Markovianity in MSE exploration due to an hidden inﬁnite-samples assumption in the objective formulation.
Revised: Backed by this intuition, we argue that prior work does not recognize the importance of non-Markovianity in MSE exploration due to an hidden inﬁnite-samples assumption in the objective formulation.

Original: In this paper, we introduce a new ﬁnite-sample MSE objective, which targets the expected entropy of the state visitation frequency induced within an episode instead of the entropy of the expected state visitation frequency over inﬁnite samples.
Revised: In this paper, we introduce a new ﬁnite-sample MSE objective, which targets the expected entropy of the state visitation frequency induced within an episode instead of the entropy of the expected state visitation frequency over inﬁnite samples.

Original: In this ﬁnite-sample formulation non-Markovian strategies are crucial, and we believe they can beneﬁt a signiﬁcant range of relevant applications.
Revised: In this ﬁnite-sample formulation non-Markovian strategies are crucial, and we believe they can beneﬁt a signiﬁcant range of relevant applications.

Original: For example, collecting task-speciﬁc samples might be costly in some real-world domains, and a pretrained non-Markovian strategy is essential to guarantee quality exploration even in a single-trial setting.
Revised: For example, collecting task-speciﬁc samples might be costly in some real-world domains, and a pretrained non-Markovian strategy is essential to guarantee quality exploration even in a single-trial setting.

Original: In another instance, one might aim to pre-train an exploration strategy for a class of multiple environments instead of a single one.
Revised: In another instance, one might aim to pre-train an exploration strategy for a class of multiple environments instead of a single one.

Original: A non-Markovian strategy could exploit the history of interactions to swiftly identify the structure of the environment, then employing the environment-speciﬁc optimal strategy thereafter.
Revised: A non-Markovian strategy could exploit the history of interactions to swiftly identify the structure of the environment, then employing the environment-speciﬁc optimal strategy thereafter.

Original: The aim of this paper is to highlight the importance of non-Markovinaity to fulﬁll the promises of maximum state entropy exploration.
Revised: The aim of this paper is to highlight the importance of non-Markovinaity to fulﬁll the promises of maximum state entropy exploration.

Original: The contributions are organized as follows.
Revised: The contributions are organized as follows.

Original: First, in Section 3, we extend known results (Puterman, 2014) to show that the class of Markovian strategies is sufﬁcient for any inﬁnite-samples MSE objective, including the entropy of the induced marginal state distributions in episodic settings.
Revised: First, in Section 3, we extend known results (Puterman, 2014) to show that the class of Markovian strategies is sufﬁcient for any inﬁnite-samples MSE objective, including the entropy of the induced marginal state distributions in episodic settings.

Original: Then, in Section 4, we propose a novel ﬁnite-sample MSE objective and a corresponding regret formulation.
Revised: Then, in Section 4, we propose a novel ﬁnite-sample MSE objective and a corresponding regret formulation.

Original: Especially, we prove that the class of non-Markovian strategies is sufﬁcient for the introduced objective, whereas the optimal
Revised: Especially, we prove that the class of non-Markovian strategies is sufﬁcient for the introduced objective, whereas the optimal

Original: Markovian strategy suffers a non-zero regret, for which we provide lower and upper bounds.
Revised: Markovian strategy suffers a non-zero regret, for which we provide lower and upper bounds.

Original: However, in Section 5, we show that the problem of ﬁnding an optimal non-Markovian strategy for the ﬁnite-sample MSE objective is NP-hard in general.
Revised: However, in Section 5, we show that the problem of ﬁnding an optimal non-Markovian strategy for the ﬁnite-sample MSE objective is NP-hard in general.

Original: Despite the hardness result, we provide a numerical validation of the theory (Section 6), and we comment some potential options to address the problem in a tractable way (Section 7).
Revised: Despite the hardness result, we provide a numerical validation of the theory (Section 6), and we comment some potential options to address the problem in a tractable way (Section 7).

Original: In Appendix A, we discuss related work, while the missing proofs can be found in Appendix B. Finally, we provide some additional remarks on the role of non-stationarity and state-action entropy objectives (Appendix C).
Revised: In Appendix A, we discuss related work, while the missing proofs can be found in Appendix B. Finally, we provide some additional remarks on the role of non-stationarity and state-action entropy objectives (Appendix C).

Original: 2. Preliminaries
Revised: 2. Preliminaries

Original: In this section, we report the notation and the basic background notions we will make use of.
Revised: In this section, we report the notation and the basic background notions we will make use of.

Original: We will denote with ∆( X ) a distribution over the space X , and with [ T ] the set of integers { 0 , . . .
Revised: We will denote with ∆( X ) a distribution over the space X , and with [ T ] the set of integers { 0 , . . .

Original: , T − 1 } .
Revised: , T − 1 } .

Original: Controlled Markov Processes A Controlled Markov
Revised: Controlled Markov Processes A Controlled Markov

Original: Process (CMP) is a tuple M :=
Revised: Process (CMP) is a tuple M :=

Original: ( S , A , P, µ ) , where S is a ﬁnite state space ( |S| = S ), A is a ﬁnite action space ( |A| =
Revised: ( S , A , P, µ ) , where S is a ﬁnite state space ( |S| = S ), A is a ﬁnite action space ( |A| =

Original: A ), P : S ×
Revised: A ), P : S ×

Original: A → ∆( S ) is the transition model, such that P ( s (cid:48)
Revised: A → ∆( S ) is the transition model, such that P ( s (cid:48)

Original: | a, s ) denotes the conditional probability of reaching state s (cid:48) ∈ S when selecting action a ∈ A in state s ∈ S , and µ : ∆( S ) is the initial state distribution.
Revised: | a, s ) denotes the conditional probability of reaching state s (cid:48) ∈ S when selecting action a ∈ A in state s ∈ S , and µ : ∆( S ) is the initial state distribution.

Original: Policies A policy π deﬁnes the behavior of an agent interacting with an environment modelled by a CMP.
Revised: Policies A policy π deﬁnes the behavior of an agent interacting with an environment modelled by a CMP.

Original: It consists of a sequence of decision rules π :=
Revised: It consists of a sequence of decision rules π :=

Original: ( π 1 , π 2 , . . .
Revised: ( π 1 , π 2 , . . .

Original: , π t , . . . ) .
Revised: , π t , . . . ) .

Original: Each of them is a map between histories h :
Revised: Each of them is a map between histories h :

Original: = ( s 0 , a 0 , . .
Revised: = ( s 0 , a 0 , . .

Original: . , a t − 1 , s t ) ∈
Revised: . , a t − 1 , s t ) ∈

Original: H t and actions π t :
Revised: H t and actions π t :

Original: H t → ∆( A ) , such that π t ( a | h ) deﬁnes the conditional probability of taking action
Revised: H t → ∆( A ) , such that π t ( a | h ) deﬁnes the conditional probability of taking action

Original: a ∈ A having experienced the history h ∈ H t .
Revised: a ∈ A having experienced the history h ∈ H t .

Original: We denote as H the space of the histories of arbitrary length.
Revised: We denote as H the space of the histories of arbitrary length.

Original: We denote as Π the set of all policies, and as Π D the set of deterministic policies
Revised: We denote as Π the set of all policies, and as Π D the set of deterministic policies

Original: π = ( π t ) ∞ t = such that π t : H t
Revised: π = ( π t ) ∞ t = such that π t : H t

Original: →
Revised: →

Original: A .
Revised: A .

Original: We further deﬁne relevant subsets of Π :
Revised: We further deﬁne relevant subsets of Π :

Original: • Non-Markovian (NM) policies Π NM , where each π ∈ Π NM collapses to a single time-invariant decision rule
Revised: • Non-Markovian (NM) policies Π NM , where each π ∈ Π NM collapses to a single time-invariant decision rule

Original: π = ( π, π, . . . )
Revised: π = ( π, π, . . . )

Original: such that π : H → ∆( A ) ; • Non-Stationary (NS) policies Π NS , where each π ∈ Π NS is deﬁned through a sequence of Markovian decision rules
Revised: such that π : H → ∆( A ) ; • Non-Stationary (NS) policies Π NS , where each π ∈ Π NS is deﬁned through a sequence of Markovian decision rules

Original: π = ( π 1 , π 2 , . . .
Revised: π = ( π 1 , π 2 , . . .

Original: , π t , . . . )
Revised: , π t , . . . )

Original: such that π t : S → ∆( A ) ; • Markovian (M) policies Π M , where each π ∈ Π M collapses to a single, time-invariant, Markovian decision rule π = ( π, π, . . . )
Revised: such that π t : S → ∆( A ) ; • Markovian (M) policies Π M , where each π ∈ Π M collapses to a single, time-invariant, Markovian decision rule π = ( π, π, . . . )

Original: such that π : S → ∆( A ) .
Revised: such that π : S → ∆( A ) .

Original: State Distributions and Visitation Frequency 
Revised: State Distributions and Visitation Frequency A policy

Original: A policy π ∈ Π interacting with a CMP induces a t -step state distribution d πt ( s ) :
Revised: π ∈ Π interacting with a CMP induces a t -step state distribution d πt ( s ) :

Original: = Pr ( s t = s | π ) over S (Puterman, 2014).
Revised: = Pr ( s t = s | π ) over S (Puterman, 2014).

Original: This distribution is described by the temporal relation d πt ( s ) =
Revised: This distribution is described by the temporal relation d πt ( s ) =

Original: (cid:82) S (cid:82)
Revised: (cid:82) S (cid:82)

Original: A d πt − 1 ( s (cid:48) , a (cid:48) )
Revised: A d πt − 1 ( s (cid:48) , a (cid:48) )

Original: P ( s | s (cid:48) , a (cid:48) )
Revised: P ( s | s (cid:48) , a (cid:48) )

Original: d s (cid:48)
Revised: d s (cid:48)

Original: d a (cid:48) , where d πt ( · , · ) : ∆( S × A ) is the t -step state-action distribution.
Revised: d a (cid:48) , where d πt ( · , · ) : ∆( S × A ) is the t -step state-action distribution.

Original: We call the asymptotic ﬁxed point of this temporal relation the stationary state distribution d π ∞ ( s ) :
Revised: We call the asymptotic ﬁxed point of this temporal relation the stationary state distribution d π ∞ ( s ) :

Original: =
Revised: =

Original: lim t →∞ d πt ( s ) , and we denote as d πγ ( s ) :
Revised: lim t →∞ d πt ( s ) , and we denote as d πγ ( s ) :

Original: = (1 − γ ) (cid:80) ∞
Revised: = (1 − γ ) (cid:80) ∞

Original: t =

Revised: t =

Original: γ t d πt ( s ) its γdiscounted counterpart ( γ ∈ (0 , 1) is the discount factor).
Revised: γ t d πt ( s ) its γdiscounted counterpart ( γ ∈ (0 , 1) is the discount factor).

Original: A marginalization of the t -step state distribution over a ﬁnite horizon T , i.e., d πT ( s ) := 1 T (cid:80) t ∈
Revised: A marginalization of the t -step state distribution over a ﬁnite horizon T , i.e., d πT ( s ) := 1 T (cid:80) t ∈

Original: [ T ] d πt ( s ) , is called the marginal state distribution .
Revised: [ T ] d πt ( s ) , is called the marginal state distribution .

Original: The state visitation frequency d h ( s ) =
Revised: The state visitation frequency d h ( s ) =

Original: 1 T (cid:80) t ∈
Revised: 1 T (cid:80) t ∈

Original: [ T ] 1 ( s t = s | h ) is a realization of the marginal state distribution, such that E h ∼ p
Revised: [ T ] 1 ( s t = s | h ) is a realization of the marginal state distribution, such that E h ∼ p

Original: π (cid:2)
Revised: π (cid:2)

Original: d h ( s ) (cid:3) =
Revised: d h ( s ) (cid:3) =

Original: d πT ( s ) , where the distribution over histories p π : ∆( H ) is deﬁned as p 
Revised: d πT ( s ) , where the distribution over histories p π : ∆( H ) is deﬁned as p πT ( h ) = µ ( s 0 ) (cid:81) t ∈

Original: [Equation]
Revised: [ T − 1] π ( a t | h t ) P ( s t +1 | a t , s t ) .

Original: Markov Decision Processes A CMP M paired with a reward function R : S ×
Revised: Markov Decision Processes A CMP M paired with a reward function R : S ×

Original: A → R is called a Markov Decision Process (MDP) (Puterman, 2014)
Revised: A → R is called a Markov Decision Process (MDP) (Puterman, 2014)

Original: M R :=
Revised: M R :=

Original: M ∪ R .
Revised: M ∪ R .

Original: We denote with R ( s, a ) the expected immediate reward when taking action a ∈ A in s ∈ S , and with R ( h )
Revised: We denote with R ( s, a ) the expected immediate reward when taking action a ∈ A in s ∈ S , and with R ( h )

Original: = (cid:80) t ∈ [ T ] R ( s t , a t ) the return over the horizon T .
Revised: = (cid:80) t ∈ [ T ] R ( s t , a t ) the return over the horizon T .

Original: The performance of a policy π over the MDP M R is deﬁned as the average return J M R ( π ) =
Revised: The performance of a policy π over the MDP M R is deﬁned as the average return J M R ( π ) =

Original: E h ∼ p πT
Revised: E h ∼ p πT

Original: [ R ( h )] , and π ∗J ∈ arg max π ∈ Π J M R
Revised: [ R ( h )] , and π ∗J ∈ arg max π ∈ Π J M R

Original: ( π ) is called an optimal policy.
Revised: ( π ) is called an optimal policy.

Original: For any
Revised: For any

Original: MDP M R , there always exists a deterministic Markovian policy π ∈ Π DM that is optimal.
Revised: MDP M R , there always exists a deterministic Markovian policy π ∈ Π DM that is optimal.

Original: Extended MDP
Revised: Extended MDP

Original: The problem of ﬁnding an optimal nonMarkovian policy with history-length T in an MDP M R , i.e., π ∗ NM ∈ arg max π ∈ Π NM J M R ( π ) , can be reformulated as the one of ﬁnding an optimal Markovian policy π ∗ M ∈ arg max π ∈ Π M J (cid:102)
Revised: The problem of ﬁnding an optimal nonMarkovian policy with history-length T in an MDP M R , i.e., π ∗ NM ∈ arg max π ∈ Π NM J M R ( π ) , can be reformulated as the one of ﬁnding an optimal Markovian policy π ∗ M ∈ arg max π ∈ Π M J (cid:102)

Original: M RT ( π ) in an extended MDP (cid:102) M RT .
Revised: M RT ( π ) in an extended MDP (cid:102) M RT .

Original: The extended MDP is deﬁned as (cid:102) M RT :
Revised: The extended MDP is deﬁned as (cid:102) M RT :

Original: = ( (cid:101) S , (cid:101) A , (cid:101) P, (cid:101) R, (cid:101) µ ) ,
Revised: = ( (cid:101) S , (cid:101) A , (cid:101) P, (cid:101) R, (cid:101) µ ) ,

Original: in which (cid:101) S ⊆ H [ T ] =
Revised: in which (cid:101) S ⊆ H [ T ] =

Original: H 1 ∪ . . .
Revised: H 1 ∪ . . .

Original: ∪ H T , and (cid:101) s :=
Revised: ∪ H T , and (cid:101) s :=

Original: ( (cid:101) s 0 , . .
Revised: ( (cid:101) s 0 , . .

Original: , (cid:101) s − 1 ) corresponds to a history in M R of length
Revised: , (cid:101) s − 1 ) corresponds to a history in M R of length

Original: | (cid:101) s | , (cid:101)
Revised: | (cid:101) s | , (cid:101)

Original: A =
Revised: A =

Original: A , (cid:101) P ( (cid:101) s (cid:48) | (cid:101) s, (cid:101) a ) =
Revised: A , (cid:101) P ( (cid:101) s (cid:48) | (cid:101) s, (cid:101) a ) =

Original: P ( s (cid:48)
Revised: P ( s (cid:48)

Original: = (cid:101) s (
Revised: = (cid:101) s (

Original: cid:48)− 1 | s = (cid:101) s − 1 , a = (cid:101) a ) , (cid:101) R ( (cid:101) s, (cid:101) a ) =
Revised: cid:48)− 1 | s = (cid:101) s − 1 , a = (cid:101) a ) , (cid:101) R ( (cid:101) s, (cid:101) a ) =

Original: R ( s = (cid:101) s − 1 , a = (cid:101) a ) , and (cid:101) µ ( (cid:101) s )
Revised: R ( s = (cid:101) s − 1 , a = (cid:101) a ) , and (cid:101) µ ( (cid:101) s )

Original: = µ ( s = (cid:101) s ) for any (cid:101) s ∈ (cid:101) S of unit length.
Revised: = µ ( s = (cid:101) s ) for any (cid:101) s ∈ (cid:101) S of unit length.

Original: Partially Observable MDP A Partially Observable
Revised: Partially Observable MDP A Partially Observable

Original: Markov Decision Process (POMDP) (Astrom, 
Revised: Markov Decision Process (POMDP) (Astrom, 1965;

Original: Kaelbling et al., 1998) is described by M R Ω :=
Revised: Kaelbling et al., 1998) is described by M R Ω :=

Original: ( S , A , P, R, µ, Ω , O ) , where S , A , P, R, µ are deﬁned as in an MDP, Ω is a ﬁnite observation space, and O : S
Revised: ( S , A , P, R, µ, Ω , O ) , where S , A , P, R, µ are deﬁned as in an MDP, Ω is a ﬁnite observation space, and O : S

Original: ×
Revised: ×

Original: A → ∆(Ω) is the observation function, such that O ( o | s (cid:48) , a ) denotes the conditional probability of the observation o ∈ Ω when selecting action a ∈ A in state s ∈ S .
Revised: A → ∆(Ω) is the observation function, such that O ( o | s (cid:48) , a ) denotes the conditional probability of the observation o ∈ Ω when selecting action a ∈ A in state s ∈ S .

Original: Crucially, while interacting with a POMDP the agent cannot observe the state s ∈ S , but just the observation o ∈ Ω .
Revised: Crucially, while interacting with a POMDP the agent cannot observe the state s ∈ S , but just the observation o ∈ Ω .

Original: The performance of a policy π is deﬁned as in an MDP.
Revised: The performance of a policy π is deﬁned as in an MDP.

Original: 3. Inﬁnite Samples: Non-Markovianity Does Not Matter
Revised: 3. Inﬁnite Samples: Non-Markovianity Does Not Matter

Original: Previous works pursuing maximum state entropy exploration of a CMP consider an objective function of the kind:
Revised: Previous works pursuing maximum state entropy exploration of a CMP consider an objective function of the kind:

Original: [Equation]
Revised: [Equation]

Original: where d π ( · ) is either a stationary state distribution (Mutti & Restelli, 2020), a discounted state distribution (Hazan et al., 2019; Tarbouriech & Lazaric, 2019), or a marginal state distribution (Lee et al., 2019; Mutti et al., 2020).
Revised: where d π ( · ) is either a stationary state distribution (Mutti & Restelli, 2020), a discounted state distribution (Hazan et al., 2019; Tarbouriech & Lazaric, 2019), or a marginal state distribution (Lee et al., 2019; Mutti et al., 2021b).

Original: While it is well-known (Puterman, 2014) that there exists an optimal deterministic policy π ∗ ∈ Π DM for the common average return objective J M R , it is not pointless to wonder whether the reward-free objective in (1) would require a more powerful policy class than Π M .
Revised: While it is well-known (Puterman, 2014) that there exists an optimal deterministic policy π ∗ ∈ Π DM for the common average return objective J M R , it is not pointless to wonder whether the reward-free objective in (1) would require a more powerful policy class than Π M .

Original: Unsurprisingly, Hazan et al.
Revised: Unsurprisingly, Hazan et al.

Original: (Hazan et al., 2019, Lemma 3.3) conﬁrm that the set of (randomized) Markovian policies Π M is indeed sufﬁcient for E ∞ deﬁned over asymptotic (either stationary or discounted) state distributions.
Revised: (Hazan et al., 2019, Lemma 3.3) conﬁrm that the set of (randomized) Markovian policies Π M is indeed sufﬁcient for E ∞ deﬁned over asymptotic (either stationary or discounted) state distributions.

Original: In the following theorem and corollary, we build upon common MDP results (Puterman, 2014) to show that Π M sufﬁces for E ∞ deﬁned over (non-asymptotic) marginal state distributions as well.
Revised: In the following theorem and corollary, we build upon common MDP results (Puterman, 2014) to show that Π M sufﬁces for E ∞ deﬁned over (non-asymptotic) marginal state distributions as well.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Let x ∈ {∞ , γ, T } , and let D x NM = { d πx ( · ) : π ∈ Π NM } , D x NS = { d πx ( · ) : π ∈ Π NS } , D x M = { d πx ( · ) : π ∈ Π M } the corresponding sets of distributions.
Revised: Let x ∈ {∞ , γ, T } , and let D x NM = { d πx ( · ) : π ∈ Π NM } , D x NS = { d πx ( · ) : π ∈ Π NS } , D x M = { d πx ( · ) : π ∈ Π M } the corresponding sets of distributions.

Original: We can prove that:
Revised: We can prove that:

Original: (i)
Revised: (i)

Original: The sets of stationary state distributions are equivalent D ∞ NM ≡ D ∞ NS ≡ D ∞ M ; (ii) The sets of discounted state distributions are equivalent D γ NM ≡ D γ NS ≡ D γ M for any γ ; (iii) The sets of marginal state distributions are equivalent D T NM ≡ D T NS ≡ D T M for any T .
Revised: The sets of stationary state distributions are equivalent D ∞ NM ≡ D ∞ NS ≡ D ∞ M ; (ii) The sets of discounted state distributions are equivalent D γ NM ≡ D γ NS ≡ D γ M for any γ ; (iii) The sets of marginal state distributions are equivalent D T NM ≡ D T NS ≡ D T M for any T .

Original: Proof Sketch.
Revised: Proof Sketch.

Original: The results (i), (ii) are a consequence of (Hazan et al., 2019, Lemma 3.3), and we refer to Appendix B.1 for a complete proof.
Revised: The results (i), (ii) are a consequence of (Hazan et al., 2019, Lemma 3.3), and we refer to Appendix B.1 for a complete proof.

Original: Here we focus on the result (iii) .
Revised: Here we focus on the result (iii) .

Original: From (Puterman, 2014, Theorem 5.5.1) we know that, for any π ∈ Π NM , we can build a π (cid:

Revised: From (Puterman, 2014, Theorem 5.5.1) we know that, for any π ∈ Π NM , we can build a π (cid:

Original: ) ∈
Revised: ) ∈

Original: Π NS having d π (cid:48) t ( s ) =
Revised: Π NS having d π (cid:48) t ( s ) =

Original: d πt ( s ) for every t ≥ 0 and s ∈ S , which implies
Revised: d πt ( s ) for every t ≥ 0 and s ∈ S , which implies

Original: D T NM ≡ D T NS .
Revised: D T NM ≡ D T NS .

Original: Thus, it is sufﬁcient to show D T NS ≡ D T M .
Revised: Thus, it is sufﬁcient to show D T NS ≡ D T M .

Original: The key point of the proof is that, for any π ∈ Π NS , we can build a policy π (cid:

Revised: The key point of the proof is that, for any π ∈ Π NS , we can build a policy π (cid:

Original: ) ∈
Revised: ) ∈

Original: Π M inducing d π (cid:48) T ( · ) =
Revised: Π M inducing d π (cid:48) T ( · ) =

Original: d πT ( · ) by marginalizing π over the time-steps t ∈ [ T − 1] , i.e., π (cid:48)
Revised: d πT ( · ) by marginalizing π over the time-steps t ∈ [ T − 1] , i.e., π (cid:48)

Original: ( a | s ) = 
Revised: ( a | s ) = (cid:80) t ∈

Original: [Equation]
Revised: [ T − 1] d πt ( s ) π t ( a | s ) (cid:14) (cid:80) t ∈ [ T − 1] d πt ( s ) , ∀ s ∈

Original: S , ∀ a ∈ A .
Revised: S , ∀ a ∈ A .

Original: This gives D T NS ⊆
Revised: This gives D T NS ⊆

Original: D T M
Revised: D T M

Original: and concludes the proof.
Revised: and concludes the proof.

Original: From the equivalence of the sets of induced distributions, it is straightforward to derive the following corollary on the optimality of Markovian policies for objective (1).
Revised: From the equivalence of the sets of induced distributions, it is straightforward to derive the following corollary on the optimality of Markovian policies for objective (1).

Original: Corollary 3.2.
Revised: Corollary 3.2.

Original: For every CMP M , there exists a Markovian policy π ∗ ∈ Π M such that π ∗ ∈ arg max π ∈ Π E ∞ ( π ) .
Revised: For every CMP M , there exists a Markovian policy π ∗ ∈ Π M such that π ∗ ∈ arg max π ∈ Π E ∞ ( π ) .

Original: As a consequence of Corollary 3.2, there is little incentive to consider non-Markovian (or non-stationary) policies when optimizing objective (1), since there is no clear advantage to make up for the additional complexity of the policy.
Revised: As a consequence of Corollary 3.2, there is little incentive to consider non-Markovian (or non-stationary) policies when optimizing objective (1), since there is no clear advantage to make up for the additional complexity of the policy.

Original: This result might be unsurprising when considering asymptotic distributions, as one can expect a carefully constructed Markovian policy to be able to tie the distribution induced by a non-Markovian (or non-stationary) policy in the limit of the interaction steps.
Revised: This result might be unsurprising when considering asymptotic distributions, as one can expect a carefully constructed Markovian policy to be able to tie the distribution induced by a non-Markovian (or non-stationary) policy in the limit of the interaction steps.

Original: However, it is less evident that a similar property holds for the expectation of ﬁnallength interactions alike.
Revised: However, it is less evident that a similar property holds for the expectation of ﬁnallength interactions alike.

Original: Yet, we were able to prove that a Markovian policy that properly exploits randomization can always achieve equivalent state distributions w.r.t.
Revised: Yet, we were able to prove that a Markovian policy that properly exploits randomization can always achieve equivalent state distributions w.r.t.

Original: nonMarkovian (or non-stationary) counterparts.
Revised: nonMarkovian (or non-stationary) counterparts.

Original: Note that state distributions are actually expected state visitation frequency, and the expectation practically implies an inﬁnite number of realizations.
Revised: Note that state distributions are actually expected state visitation frequency, and the expectation practically implies an inﬁnite number of realizations.

Original: In this paper, we show that this underlying inﬁnite-sample regime is the reason why the beneﬁt of non-Markovianity, albeit backed up by intuition, does not matter.
Revised: In this paper, we show that this underlying inﬁnite-sample regime is the reason why the beneﬁt of non-Markovianity, albeit backed up by intuition, does not matter.

Original: Instead, we propose a relevant ﬁnite-sample entropy objective in which non-Markovianity is crucial.
Revised: Instead, we propose a relevant ﬁnite-sample entropy objective in which non-Markovianity is crucial.

Original: 4. Finite Samples: Non-Markovianity Matters
Revised: 4. Finite Samples: Non-Markovianity Matters

Original: In this section, we reformulate the typical maximum state entropy exploration objective of a CMP (1) to account for a ﬁnite-sample regime.
Revised: In this section, we reformulate the typical maximum state entropy exploration objective of a CMP (1) to account for a ﬁnite-sample regime.

Original: Crucially, we consider the expected entropy of the state visitation frequency rather than the entropy of the expected state visitation frequency, which results in
Revised: Crucially, we consider the expected entropy of the state visitation frequency rather than the entropy of the expected state visitation frequency, which results in

Original: [Equation]
Revised: [Equation]

Original: We note that E ( π ) ≤ E ∞ ( π ) for any π ∈ Π , which is trivial by the concavity of the entropy function and the Jensen’s inequality.
Revised: We note that E ( π ) ≤ E ∞ ( π ) for any π ∈ Π , which is trivial by the concavity of the entropy function and the Jensen’s inequality.

Original: Whereas (3) is ultimately an expectation as it is (1), the entropy is not computed over the inﬁnite-sample state distribution d πT ( · ) but its ﬁnite-sample realization d h ( · ) .
Revised: Whereas (3) is ultimately an expectation as it is (1), the entropy is not computed over the inﬁnite-sample state distribution d πT ( · ) but its ﬁnite-sample realization d h ( · ) .

Original: Thus, to maximize E ( π ) we have to ﬁnd a policy inducing high-entropy state visits within a single trajectory rather than high-entropy state visits over inﬁnitely many trajectories.
Revised: Thus, to maximize E ( π ) we have to ﬁnd a policy inducing high-entropy state visits within a single trajectory rather than high-entropy state visits over inﬁnitely many trajectories.

Original: Crucially, while Markovian policies are as powerful as any other policy class in terms of induced state distributions (Theorem 3.1), this is no longer true when looking at induced trajectory distributions p πT .
Revised: Crucially, while Markovian policies are as powerful as any other policy class in terms of induced state distributions (Theorem 3.1), this is no longer true when looking at induced trajectory distributions p πT .

Original: Indeed, we show in this section that non-Markovianity provides a superior policy class for objective (3).
Revised: Indeed, we show in this section that non-Markovianity provides a superior policy class for objective (3).

Original: First, we deﬁne a performance measure to formally assess this beneﬁt, which we call regret-to-go .
Revised: First, we deﬁne a performance measure to formally assess this beneﬁt, which we call regret-to-go .

Original: Deﬁnition 4.1 (Expected Regret-to-go) .
Revised: Deﬁnition 4.1 (Expected Regret-to-go) .

Original: Consider a policy π ∈ Π interacting with a CMP over T steps.
Revised: Consider a policy π ∈ Π interacting with a CMP over T steps.

Original: We deﬁne the expected regret-to-go R T − t at step t (i.e., from step t onwards) of π as
Revised: We deﬁne the expected regret-to-go R T − t at step t (i.e., from step t onwards) of π as

Original: [Equation]
Revised: [Equation]

Original: where h ∗ T =
Revised: where h ∗ T =

Original: ( h ∗ t , h ∗ T − t ) , h T =
Revised: ( h ∗ t , h ∗ T − t ) , h T =

Original: ( h ∗ t , h T − t ) are concatenations of the ( T − t ) -step trajectories h ∗ T − t , h T − t (starting from the state s t,h ∗ t ) and the t -step optimal trajectory h ∗ t ∼ p π ∗ respectively.
Revised: ( h ∗ t , h T − t ) are concatenations of the ( T − t ) -step trajectories h ∗ T − t , h T − t (starting from the state s t,h ∗ t ) and the t -step optimal trajectory h ∗ t ∼ p π ∗ respectively.

Original: The term R T ( π ) denotes the expected regret-to-go of a T -step trajectory h T starting from s ∼ µ .
Revised: The term R T ( π ) denotes the expected regret-to-go of a T -step trajectory h T starting from s ∼ µ .

Original: The intuition behind the regret-to-go is quite simple.
Revised: The intuition behind the regret-to-go is quite simple.

Original: Suppose to have drawn a zero-regret trajectory h ∗ t upon step t .
Revised: Suppose to have drawn a zero-regret trajectory h ∗ t upon step t .

Original: If we take the subsequent action with the (possibly suboptimal) policy π , by how much would we decrease (in expectation) the entropy of the state visits Entropy ( d h T ( · )) w.r.t.
Revised: If we take the subsequent action with the (possibly suboptimal) policy π , by how much would we decrease (in expectation) the entropy of the state visits Entropy ( d h T ( · )) w.r.t.

Original: an optimal policy π ∗ ?
Revised: an optimal policy π ∗ ?

Original: In particular, we would like to know how limiting the policy π to a speciﬁc policy class would affect the expected regret-to-go and the value of E ( π ) we could achieve.
Revised: In particular, we would like to know how limiting the policy π to a speciﬁc policy class would affect the expected regret-to-go and the value of E ( π ) we could achieve.

Original: The following lemma shows that an optimal non-Markovian policy suffers zero expected regretto-go.
Revised: The following lemma shows that an optimal non-Markovian policy suffers zero expected regretto-go.

Original: Lemma 4.2.
Revised: Lemma 4.2.

Original: For every CMP M , there exists a deterministic non-Markovian policy π NM ∈ Π DNM such that π NM ∈ arg max π ∈ Π NM E ( π ) , which suffers expected regretto-go R T − t ( π NM )
Revised: For every CMP M , there exists a deterministic non-Markovian policy π NM ∈ Π DNM such that π NM ∈ arg max π ∈ Π NM E ( π ) , which suffers expected regretto-go R T − t ( π NM )

Original: = 0 , ∀ t ∈ [ T ] .
Revised: = 0 , ∀ t ∈ [ T ] .

Original: Proof.
Revised: Proof.

Original: The result R T − t ( π NM )
Revised: The result R T − t ( π NM )

Original: = 0 is straightforward by noting that the set of non-Markovian policies Π NM with arbitrary history-length is as powerful as the general set of policies Π .
Revised: = 0 is straightforward by noting that the set of non-Markovian policies Π NM with arbitrary history-length is as powerful as the general set of policies Π .

Original: To show that there exists a deterministic π NM , we consider the extended MDP (cid:102)
Revised: To show that there exists a deterministic π NM , we consider the extended MDP (cid:102)

Original: M RT obtained from the CMP M as in Section 2, in which the extended reward function is (
Revised: M RT obtained from the CMP M as in Section 2, in which the extended reward function is (

Original: cid:101) R ( (cid:101) s, (cid:101) a ) =
Revised: cid:101) R ( (cid:101) s, (cid:101) a ) =

Original: Entropy ( d (cid:101) s ( · )) for every (cid:101) a ∈ (cid:101) A and every (cid:101)
Revised: Entropy ( d (cid:101) s ( · )) for every (cid:101) a ∈ (cid:101) A and every (cid:101)

Original: s ∈ (cid:101) S such that | (cid:101) s |
Revised: s ∈ (cid:101) S such that | (cid:101) s |

Original: = T , and (cid:101) R ( (cid:101) s, (cid:101) a )
Revised: = T , and (cid:101) R ( (cid:101) s, (cid:101) a )

Original: =otherwise.
Revised: =otherwise.

Original: Since a Markovian policy (cid:101) π M ∈ Π DM on (cid:102)
Revised: Since a Markovian policy (cid:101) π M ∈ Π DM on (cid:102)

Original: M RT can be mapped to a non-Markovian policy π NM ∈ Π DNM on M , and it is well-known (Puterman, 2014)
Revised: M RT can be mapped to a non-Markovian policy π NM ∈ Π DNM on M , and it is well-known (Puterman, 2014)

Original: that for any
Revised: that for any

Original: MDP there exists an optimal deterministic Markovian policy, we have that (cid:101) πM ∈ arg max π ∈ Π M J (cid:102)
Revised: MDP there exists an optimal deterministic Markovian policy, we have that (cid:101) π

Original: M RT ( π ) implies π NM ∈ arg max π ∈ Π NM E ( π ) .
Revised: [Equation]

Original: Whereas Lemma 4.2 ensures that the set of non-Markovian policies Π NM is sufﬁcient for the objective (3), we would like to know if it is also necessary.
Revised: Whereas Lemma 4.2 ensures that the set of non-Markovian policies Π NM is sufﬁcient for the objective (3), we would like to know if it is also necessary.

Original: Especially, we aim to assess whether there exist CMPs in which a Markovian policy π ∈ Π M would suffer non-zero regret-to-go.
Revised: Especially, we aim to assess whether there exist CMPs in which a Markovian policy π ∈ Π M would suffer non-zero regret-to-go.

Original: First, it is worth showing that Markovian policies can rely on randomization to optimize objective (3).
Revised: First, it is worth showing that Markovian policies can rely on randomization to optimize objective (3).

Original: Lemma 4.3.
Revised: Lemma 4.3.

Original: Let π NM ∈ Π DNM be a non-Markovian policy such that π NM ∈ arg max π ∈ Π E
Revised: Let π NM ∈ Π DNM be a non-Markovian policy such that π NM ∈ arg max π ∈ Π E

Original: ( π ) on a CMP
Revised: ( π ) on a CMP

Original: M .
Revised: M .

Original: The variance of an optimal Markovian policy π M ∈ arg max π ∈ Π M E ( π ) is given byV ar a ∼ π M ( s ) (cid:2) a [Equation] =
Revised: The variance of an optimal Markovian policy π M ∈ arg max π ∈ Π M E ( π ) is given by

Original: [Equation]

Original: ar

Original: [Equation]

Original: (cid:2) π NM ( hs ) (cid:3) , ∀

Original: [Equation]

Original: ∈ S

Original: [Equation]
Revised: [Equation]

Original: where hs is any history hs ∈ H
Revised: where hs is any history hs ∈ H

Original: [ T ] such that the ﬁnal state is s .
Revised: [ T ] such that the ﬁnal state is s .

Original: Proof Sketch.
Revised: Proof Sketch.

Original: We can prove the result through the Law of
Revised: We can prove the result through the Law of

Original: Total Variance (LoTV) (Bertsekas & Tsitsiklis, 2002), which gives V ar a
Revised: Total Variance (LoTV) (Bertsekas & Tsitsiklis, 2002), which gives V ar a

Original: [Equation]
Revised: [Equation]

Original: (cid:2) E a ∼
Revised: V ar hs ∼

Original: [Equation]
Revised: [Equation]

Original:  NM ( hs ) [ a ] (cid:3) , ∀ s ∈ S .
Revised: (cid:2) E a ∼ π NM ( hs ) [ a ] (cid:3) , ∀ s ∈ S .

Original: Then, exploiting the determinism of π NM (Lemma 4.2), we have
Revised: Then, exploiting the determinism of π NM (through Lemma 4.2), it is straightforward to see that E hs ∼ p π NM T

Original: [Equation]
Revised: [Equation]

Original: and V
Revised: and that

Original: [Equation]
Revised: [Equation]

Original: (cid:2) π NM ( hs ) (cid:3) , which concludes the proof.
Revised: , which concludes the proof.

Original: Especially, Lemma 4.3 shows that, whenever the optimal strategy for objective (3) (i.e., the non-Markovian π NM ) requires to adapt its decision in a state s according to the history that led to it ( hs ), an optimal Markovian policy for the same objective (i.e., π M ) must necessarily be randomized.
Revised: Especially, Lemma 4.3 shows that, whenever the optimal strategy for objective (3) (i.e., the non-Markovian π NM ) requires to adapt its decision in a state s according to the history that led to it ( hs ), an optimal Markovian policy for the same objective (i.e., π M ) must necessarily be randomized.

Original: This is crucial to prove the main result of this section, which establishes a lower bound R T − t and an upper bound
Revised: This is crucial to prove the main result of this section, which establishes a lower bound R T − t and an upper bound

Original: R T − t to the expected regret-to-go of any Markovian policy that optimizes objective (3).
Revised: R T − t to the expected regret-to-go of any Markovian policy that optimizes objective (3).

Original: Theorem 4.4.
Revised: Theorem 4.4.

Original: Let π M ∈
Revised: Let π M ∈

Original: Π M be a Markovian policy such that π M ∈ arg max π ∈ Π M E ( π ) on a CMP M .
Revised: Π M be a Markovian policy such that π M ∈ arg max π ∈ Π M E ( π ) on a CMP M .

Original: Then, for any t ∈ [ T ] , it holds R T − t ( π M ) ≤ R T − t
Revised: Then, for any t ∈ [ T ] , it holds R T − t ( π M ) ≤ R T − t

Original: ( π M ) ≤ R T − t ( π M ) such that
Revised: ( π M ) ≤ R T − t ( π M ) such that

Original: [Equation]
Revised: [Equation]

Original: where π NM ∈ arg max π ∈ Π DNM E ( π ) , a NM = π NM ( h ∗ t ) is the unique optimal action in s t , and E max , E max, 2 , E min,t are given by
Revised: where π NM ∈ arg max π ∈ Π DNM E ( π ) , a NM = π NM ( h ∗ t ) is the unique optimal action in s t , and E max , E max, 2 , E min,t are given by

Original: [Equation]
Revised: [Equation]

Original: Proof Sketch.
Revised: Proof Sketch.

Original: The crucial idea to derive lower and upper bounds to the regret-to-go is to consider the impact of a suboptimal action in the best-case and the worst-case CMP respectively (see Lemma B.1, B.2).
Revised: The crucial idea to derive lower and upper bounds to the regret-to-go is to consider the impact of a suboptimal action in the best-case and the worst-case CMP respectively (see Lemma B.1, B.2).

Original: This gives R T − t ( π M ) ≥
Revised: This gives R T − t ( π M ) ≥

Original: [Equation]
Revised: [Equation]

Original: ( a NM | s t ) (cid:1) E max,and R T − t ( π M ) ≤
Revised: and R [Equation] |

Original: E max − π M ( a NM | s t ) E max − (cid:0) 1 − π M ( a NM | s t ) (cid:1) E min,t .
Revised: [Equation] ) (cid:1) E [Equation] ,t .

Original: Then, by combining the variance of the Bernoulli distribution that controls the event of taking a sub-optimal action together with Lemma 4.3, we get V ar a ∼ π NM
Revised: Then, by combining the variance of the Bernoulli distribution that controls the event of taking a sub-optimal action together with Lemma 4.3, we get V ar a ∼ π NM

Original: ( s t ) [ a ] = π M ( a NM | s t ) (cid:0) 1 − π M ( a NM | s t ) (cid:1) =
Revised: ( s t ) [ a ] = π M ( a NM | s t ) (cid:0) 1 − π M ( a NM | s t ) (cid:1) =

Original: V ar hs t ∼ p π NM T (cid:2)
Revised: V ar hs t ∼ p π NM T (cid:2)

Original: π NM ( hs t ) (cid:3) , which concludes the proof.
Revised: π NM ( hs t ) (cid:3) , which concludes the proof.

Original: Theorem 4.4 basically states that, whenever the Markovian policy ( π M ) has to randomize its strategy at the step t , it will suffer non-zero regret-to-go under the assumption that
Revised: Theorem 4.4 basically states that, whenever the Markovian policy ( π M ) has to randomize its strategy at the step t , it will suffer non-zero regret-to-go under the assumption that

Original: M admits a unique optimal action a NM in s t .
Revised: M admits a unique optimal action a NM in s t .

Original: Clearly, the value of this regret is related to the probability of taking a sub-optimal action (through the factor 1 /π M ( a NM | s t ) ) and the variance of the optimal strategy in s t over the possible histories (through the factor V ar hs t ∼ p π NM T [ π NM ( hs t )] ).
Revised: Clearly, the value of this regret is related to the probability of taking a sub-optimal action (through the factor 1 /π M ( a NM | s t ) ) and the variance of the optimal strategy in s t over the possible histories (through the factor V ar hs t ∼ p π NM T [ π NM ( hs t )] ).

Original: To compute the regret-to-go exactly, one should have access to the full structure of the CMP M and its transition dynamics P .
Revised: To compute the regret-to-go exactly, one should have access to the full structure of the CMP M and its transition dynamics P .

Original: Instead, we provide an upper bound and a lower bound to the regret that only depends on the length of the remaining interaction T − t .
Revised: Instead, we provide an upper bound and a lower bound to the regret that only depends on the length of the remaining interaction T − t .

Original: While at the ﬁnal step t = T a policy cannot suffer any regret, the earlier it pulls a sub-optimal action in the interaction process, the more it might incur in a bad entropy over this trajectory.
Revised: While at the ﬁnal step t = T a policy cannot suffer any regret, the earlier it pulls a sub-optimal action in the interaction process, the more it might incur in a bad entropy over this trajectory.

Original: The factor
Revised: The factor

Original: E max − E min,t in the upper bound quantiﬁes how badly it can go in the worst possible CMP, in which the agent never[-sub − opt ( s t ) , such that A opt ( s t ) are optimal actions and A sub − opt ( s t ) are sub-optimal, and substituting the term 
 /π M ( a NM | s t ) with 1 / (cid:80) a ∈A opt ( s t ) π M ( a | s t ) in the regret bounds.-] recovers from a sub-optimal action.
Revised: E max − E min,t in the upper bound quantiﬁes how badly it can go in the worst possible CMP, in which the agent never[-sub − opt ( s t ) , such that A opt ( s t ) are optimal actions and A sub − opt ( s t ) are sub-optimal, and substituting the term 
 /π M ( a NM | s t ) with 1 / (cid:80) a ∈A opt ( s t ) π M ( a | s t ) in the regret bounds.-] recovers from a sub-optimal action.

Original: Instead, E max − E max,in the lower bound quantiﬁes the regret caused by a suboptimal action in the best-case CMP, in which the negative impact of the sub-optimal action is minimized.
Revised: Instead, E max − E max,in the lower bound quantiﬁes the regret caused by a suboptimal action in the best-case CMP, in which the negative impact of the sub-optimal action is minimized.

Original: Note that, whenever the optimal decision in s t does not depend on the history that lead to it ( hs t ), the policy π M can act deterministically ( π M ( a NM | s t ) = 1 ) and the regret-to-go, its lower bound, and its upper bound are simultaneously zero (through V ar hs t ∼ p π NM T [ π NM ( hs t )]
Revised: Note that, whenever the optimal decision in s t does not depend on the history that lead to it ( hs t ), the policy π M can act deterministically ( π M ( a NM | s t ) = 1 ) and the regret-to-go, its lower bound, and its upper bound are simultaneously zero (through V ar hs t ∼ p π NM T [ π NM ( hs t )]

Original: = 0 ).
Revised: = 0 ).

Original: Finally, although the objective (3) is non-additive across time steps, we can still deﬁne a notion of pseudoinstantaneous regret by comparing the regret-to-go of two subsequent time steps.
Revised: Finally, although the objective (3) is non-additive across time steps, we can still deﬁne a notion of pseudoinstantaneous regret by comparing the regret-to-go of two subsequent time steps.

Original: In the following, we provide the definition of this expected pseudo-instantaneous regret along with lower and upper bounds to the regret suffered by an optimal Markovian policy.
Revised: In the following, we provide the definition of this expected pseudo-instantaneous regret along with lower and upper bounds to the regret suffered by an optimal Markovian policy.

Original: Deﬁnition 4.5 (Expected Pseudo-Instantaneous Regret) .
Revised: Deﬁnition 4.5 (Expected Pseudo-Instantaneous Regret) .

Original: Consider a policy π ∈ Π interacting with a CMP over
Revised: Consider a policy π ∈ Π interacting with a CMP over

Original: T steps.
Revised: T steps.

Original: We deﬁne the expected pseudo-instantaneous regret of π at step t as r t ( π ) :
Revised: We deﬁne the expected pseudo-instantaneous regret of π at step t as r t ( π ) :

Original: = max (cid:0) 0 , R T − t ( π ) −R T − t − 1 ( π ) (cid:1) .
Revised: = max (cid:0) 0 , R T − t ( π ) −R T − t − 1 ( π ) (cid:1) .

Original: Corollary 4.6.
Revised: Corollary 4.6.

Original: Let π M ∈
Revised: Let π M ∈

Original: Π M be a Markovian policy such that π M ∈ arg max π ∈ Π M E ( π ) on a CMP M .
Revised: Π M be a Markovian policy such that π M ∈ arg max π ∈ Π M E ( π ) on a CMP M .

Original: Then, for any t ∈ [ T ] , it holds r t ( π M ) ≤ r t ( π M ) ≤ r t ( π M ) such that r t ( π M ) = max (cid:16) 0 , E max (cid:0) V t ( π M ) − V t +1 ( π M ) (cid:1) − E max, 2 V t ( π M ) + E min,t +1 V t 
Revised: Then, for any t ∈ [ T ] , it holds r t ( π M ) ≤ r t ( π M ) ≤ r t ( π M ) such that r t ( π M ) = max (cid:16) 0 , E max (cid:0) V t ( π M ) − V t +1 ( π M ) (cid:1) − E max, 2 V t ( π M ) + E min,t +1 V t +1 ( π M ) (cid:17) , r t ( π M ) =

Original: +1 ( π M ) (cid:17) , r t ( π M ) = max (cid:16) 0 , E max (cid:0) V t ( π M ) − V t +1 ( π M ) (cid:1) − E min,t V t ( π M ) + E max, 2 V t +1 ( π M ) (cid:17) , where
Revised: [Equation]

Original:  Vt ( π M ) :
Revised: where V

Original: [Equation]
Revised: [Equation]

Original: 5. Complexity Analysis
Revised: 5. Complexity Analysis

Original: Having established the importance of non-Markovianity in dealing with MSE exploration in a ﬁnite-sample regime, it is worth considering how hard it is to optimize the objectivewithin the class of non-Markovian policies.
Revised: Having established the importance of non-Markovianity in dealing with MSE exploration in a ﬁnite-sample regime, it is worth considering how hard it is to optimize the objectivewithin the class of non-Markovian policies.

Original: Especially, we aim at characterizing the complexity of the problem:
Revised: Especially, we aim at characterizing the complexity of the problem:

Original: [Equation]
Revised: [Equation]

Original: deﬁned over a CMP M .
Revised: deﬁned over a CMP M .

Original: First, we recall that Ψ 0 can be rewritten as the problem of ﬁnding a reward-maximizing
Revised: First, we recall that Ψ 0 can be rewritten as the problem of ﬁnding a reward-maximizing

Original: Markovian policy, i.e., (cid:101) π M ∈ arg max π ∈ Π M J (cid:102)
Revised: Markovian policy, i.e., (cid:101) π M ∈ arg max π ∈ Π M J (cid:102)

Original: M RT ( π ) , over a convenient extended MDP (cid:102)
Revised: M RT ( π ) , over a convenient extended MDP (cid:102)

Original: M RT obtained from CMP M (see the proof of Lemma 4.2 for further details).
Revised: M RT obtained from CMP M (see the proof of Lemma 4.2 for further details).

Original: We call this problem (cid:101) Ψ 

Revised: We call this problem (cid:101) Ψ

Original: and we note that (cid:101) Ψ 0 ∈ P , as the problem of ﬁnding a reward-maximizing Markovian policy is wellknown to be in P for any MDP (Papadimitriou & Tsitsiklis, 1987).
Revised: and we note that (cid:101) Ψ 0 ∈ P , as the problem of ﬁnding a reward-maximizing Markovian policy is wellknown to be in P for any MDP (Papadimitriou & Tsitsiklis, 1987).

Original: However, the following lemma shows that it does not exist a many-to-one reduction from Ψ 0 to (cid:101) Ψ 0 .
Revised: However, the following lemma shows that it does not exist a many-to-one reduction from Ψ 0 to (cid:101) Ψ 0 .

Original: Lemma 5.1.
Revised: Lemma 5.1.

Original: A reduction Ψ 0 ≤ m (cid:101) Ψ 0 does not exist.
Revised: A reduction Ψ 0 ≤ m (cid:101) Ψ 0 does not exist.

Original: Proof.
Revised: Proof.

Original: In the general case, coding any instance of Ψ 0 in the representation required by (cid:101) Ψ 0 holds exponential complexity w.r.t.
Revised: In the general case, coding any instance of Ψ 0 in the representation required by (cid:101) Ψ 0 holds exponential complexity w.r.t.

Original: the input of the initial instance of Ψ 0 .
Revised: the input of the initial instance of Ψ 0 .

Original: The latter result informally suggests that Ψ / ∈ P .
Revised: The latter result informally suggests that Ψ / ∈ P .

Original: Indeed, we can show that Ψ / ∈ NP .
Revised: Indeed, we can show that Ψ / ∈ NP .

Original: Lemma 5.2.
Revised: Lemma 5.2.

Original: Ψ 0 / ∈ NP .
Revised: Ψ 0 / ∈ NP .

Original: Proof.
Revised: Proof.

Original: This proof is based on the veriﬁer-based deﬁnition of
Revised: This proof is based on the veriﬁer-based deﬁnition of

Original: NP.
Revised: NP.

Original: According to this deﬁnition, given any instance I ∈ I
Revised: According to this deﬁnition, given any instance I ∈ I

Original: Ψof problem Ψ 0 and a candidate solution π ∈ Π NM , if there exists an algorithm Λ that can verify the optimality of π in polynomial time, then Ψ 0 ∈ NP .
Revised: Ψof problem Ψ 0 and a candidate solution π ∈ Π NM , if there exists an algorithm Λ that can verify the optimality of π in polynomial time, then Ψ 0 ∈ NP .

Original: We prove that Ψ 0 / ∈ NP by showing that a polynomial time veriﬁer does not exist for any instance of the problem.
Revised: We prove that Ψ 0 / ∈ NP by showing that a polynomial time veriﬁer does not exist for any instance of the problem.

Original: It sufﬁces to note that given a non-Markovian policy π ∈ Π NM , i.e., a candidate solution of an instance of the problem Ψ 0 , a polynomial veriﬁer Λ does not exist, since any possible veriﬁer has to compute the objective function for every possible well-deﬁned nonMarkovian policy in order to determine whether π is optimal according to Ψ 0 , but this operation is exponential w.r.t.
Revised: It sufﬁces to note that given a non-Markovian policy π ∈ Π NM , i.e., a candidate solution of an instance of the problem Ψ 0 , a polynomial veriﬁer Λ does not exist, since any possible veriﬁer has to compute the objective function for every possible well-deﬁned nonMarkovian policy in order to determine whether π is optimal according to Ψ 0 , but this operation is exponential w.r.t.

Original: the input of Ψ 0 .
Revised: the input of Ψ 0 .

Original: As a direct consequence, Ψ 0 / ∈ NP -complete.
Revised: As a direct consequence, Ψ 0 / ∈ NP -complete.

Original: We can now prove the main theorem of this section, which shows that Ψ 0 is NP -hard under the common assumption that P (cid:54) = NP .
Revised: We can now prove the main theorem of this section, which shows that Ψ 0 is NP -hard under the common assumption that P (cid:54) = NP .

Original: Theorem 5.3.
Revised: Theorem 5.3.

Original: Ψ 0 ∈ NP -hard.
Revised: Ψ 0 ∈ NP -hard.

Original: Proof Sketch.
Revised: Proof Sketch.

Original: To prove that Ψ 0 ∈ NP -hard, it is sufﬁcient to show that there exists a problem Ψ c ∈ NP -complete
Revised: To prove that Ψ 0 ∈ NP -hard, it is sufﬁcient to show that there exists a problem Ψ c ∈ NP -complete

Original: so that Ψ c ≤ p Ψ 0 .
Revised: so that Ψ c ≤ p Ψ 0 .

Original: We show this by reducing 3SAT, which is a well-known NP -complete problem, to Ψ 0 .
Revised: We show this by reducing 3SAT, which is a well-known NP -complete problem, to Ψ 0 .

Original: To derive the reduction we consider two intermediate problems, namely Ψ 1 and Ψ 2 .
Revised: To derive the reduction we consider two intermediate problems, namely Ψ 1 and Ψ 2 .

Original: Especially, we aim to show that the following chain of reductions holds
Revised: Especially, we aim to show that the following chain of reductions holds

Original: [Equation]
Revised: [Equation]

Original: p Ψ 2 ≥ p 3SAT .

Original: First, we deﬁne Ψ 1 and we prove that Ψ 0 ≥ m Ψ 1 .
Revised: First, we deﬁne Ψ 1 and we prove that Ψ 0 ≥ m Ψ 1 .

Original: Informally, Ψ 1 is the problem of ﬁnding a reward-maximizing
Revised: Informally, Ψ 1 is the problem of ﬁnding a reward-maximizing

Original: Markovian policy π M ∈ Π M w.r.t.
Revised: Markovian policy π M ∈ Π M w.r.t.

Original: the entropy objective (3) encoded through a reward function in a convenient POMDP (cid:102) M R Ω .
Revised: the entropy objective (3) encoded through a reward function in a convenient POMDP (cid:102) M R Ω .

Original: We can build (cid:102) M R Ω from the CMP M similarly as the extended MDP (cid:102) M RT (see Section 2 and the proof of
Revised: We can build (cid:102) M R Ω from the CMP M similarly as the extended MDP (cid:102) M RT (see Section 2 and the proof of

Original: Lemma 4.2 for details), except that the agent only access the observation space (cid:101) Ω instead of the extended state space (cid:101) S .
Revised: Lemma 4.2 for details), except that the agent only access the observation space (cid:101) Ω instead of the extended state space (cid:101) S .

Original: In particular, we deﬁne (cid:101) Ω = S
Revised: In particular, we deﬁne (cid:101) Ω = S

Original: (note that S is the state space of the original CMP M ), and (cid:101) O
Revised: (note that S is the state space of the original CMP M ), and (cid:101) O

Original: ( (cid:101)
Revised: ( (cid:101)

Original: o | (cid:101) s ) =
Revised: o | (cid:101) s ) =

Original: (cid:101) s − 1 .
Revised: (cid:101) s − 1 .

Original: Then, the reduction Ψ 0 ≥ m Ψ 1 works as follows.
Revised: Then, the reduction Ψ 0 ≥ m Ψ 1 works as follows.

Original: We denote as I Ψ
Revised: We denote as I Ψ

Original: i the set of possible instances of problem Ψ i .
Revised: i the set of possible instances of problem Ψ i .

Original: We show that Ψ 0 is harder than Ψ 1 by deﬁning the polynomialtime functions ψ and φ such that any instance of Ψ 1 can be rewritten through ψ as an instance of Ψ 0 , and a solution π ∗ NM ∈ Π NM for Ψ 0 can be converted through φ into a solution π ∗ M ∈ Π M for the original instance of Ψ 1 .
Revised: We show that Ψ 0 is harder than Ψ 1 by deﬁning the polynomialtime functions ψ and φ such that any instance of Ψ 1 can be rewritten through ψ as an instance of Ψ 0 , and a solution π ∗ NM ∈ Π NM for Ψ 0 can be converted through φ into a solution π ∗ M ∈ Π M for the original instance of Ψ 1 .

Original: The function ψ sets S = (cid:101) Ω and derives the transition model of M from the one of (cid:102) M R Ω , while φ converts the optimal solution of Ψby computing π ∗ M = (cid:80)
Revised: The function ψ sets S = (cid:101) Ω and derives the transition model of M from the one of (cid:102) M R Ω , while φ converts the optimal solution of Ψby computing π ∗ M = (cid:80)

Original: ho ∈H o p π ∗ NM ( ho )
Revised: ho ∈H o p π ∗ NM ( ho )

Original: π ∗ NM ( a | ho ) , where
Revised: π ∗ NM ( a | ho ) , where

Original: H o stands for the set of histories
Revised: H o stands for the set of histories

Original: h ∈ H
Revised: h ∈ H

Original: [ T ] ending in the observation o ∈ Ω .
Revised: [ T ] ending in the observation o ∈ Ω .

Original: Thus, we have that Ψ 0 ≥ m Ψ 1 holds.
Revised: Thus, we have that Ψ 0 ≥ m Ψ 1 holds.

Original: We now deﬁne Ψ 2 as the policy existence problem w.r.t.
Revised: We now deﬁne Ψ 2 as the policy existence problem w.r.t.

Original: the problem statement of Ψ 1 .
Revised: the problem statement of Ψ 1 .

Original: Hence, Ψ 2 is the problem of determining whether the value of a reward-maximizing
Revised: Hence, Ψ 2 is the problem of determining whether the value of a reward-maximizing

Original: Markovian policy π ∗ M ∈ arg max π ∈ Π M J (cid:102)
Revised: Markovian policy π ∗ M ∈ arg max π ∈ Π M J (cid:102)

Original: M R Ω ( π ) is greater than 0.
Revised: M R Ω ( π ) is greater than 0.

Original: Since computing an optimal policy in POMDPs is in general harder than the relative policy existence problem (Lusena et al., 2001, Section 3), we have that Ψ 1 ≥ p Ψ 2 .
Revised: Since computing an optimal policy in POMDPs is in general harder than the relative policy existence problem (Lusena et al., 2001, Section 3), we have that Ψ 1 ≥ p Ψ 2 .

Original: For the last reduction, i.e., Ψ 2 ≥ p 3SAT, we extend the proof of Theorem 4.13 in (Mundhenk et al., 2000), which states that the policy existence problem for POMDPs is
Revised: For the last reduction, i.e., Ψ 2 ≥ p 3SAT, we extend the proof of Theorem 4.13 in (Mundhenk et al., 2000), which states that the policy existence problem for POMDPs is

Original: NP -complete.
Revised: NP -complete.

Original: In particular, we show that this holds within the restricted class of POMDPs deﬁned in Ψ 1 .
Revised: In particular, we show that this holds within the restricted class of POMDPs deﬁned in Ψ 1 .

Original: Since the chain Ψ 0 ≥ m Ψ 1 ≥ p Ψ 2 ≥ p 3SAT holds, we have that Ψ 0 ≥ p 3SAT.
Revised: Since the chain Ψ 0 ≥ m Ψ 1 ≥ p Ψ 2 ≥ p 3SAT holds, we have that Ψ 0 ≥ p 3SAT.

Original: Moreover, since 3SAT ∈ NP -complete and Ψ 0 / ∈ NP (thanks to Lemma 5.2), we conclude that Ψ 0 ∈ NP -hard.
Revised: Moreover, since 3SAT ∈ NP -complete and Ψ 0 / ∈ NP (thanks to Lemma 5.2), we conclude that Ψ 0 ∈ NP -hard.

Original: 6. Numerical Validation
Revised: 6. Numerical Validation

Original: Despite the hardness result of Theorem 5.3, we provide a brief numerical validation around the potential of nonMarkovianity in MSE exploration.
Revised: Despite the hardness result of Theorem 5.3, we provide a brief numerical validation around the potential of nonMarkovianity in MSE exploration.

Original: Crucially, the reported analysis is limited to simple domains and short time horizons, and it has to be intended as an illustration of the theoretical claims reported in previous sections.
Revised: Crucially, the reported analysis is limited to simple domains and short time horizons, and it has to be intended as an illustration of the theoretical claims reported in previous sections.

Original: Whereas a comprehensive evaluation of the practical beneﬁts of nonMarkovianity in MSE exploration is left as future work, we discuss in Section 7 why we believe that the development of scalable methods is not hopeless even in this challenging setting.
Revised: Whereas a comprehensive evaluation of the practical beneﬁts of nonMarkovianity in MSE exploration is left as future work, we discuss in Section 7 why we believe that the development of scalable methods is not hopeless even in this challenging setting.

Original: In this section, we consider a 3State ( S = 3 , A = 2 , T = 9 ), which is a simple abstraction of the two-rooms in Figure 1, and a River Swim (Strehl & Littman, 2008)
Revised: In this section, we consider a 3State ( S = 3 , A = 2 , T = 9 ), which is a simple abstraction of the two-rooms in Figure 1, and a River Swim (Strehl & Littman, 2008)

Original: ( S = 3 , A = 2 , T = 10 ) that are depicted in Figure 2a, 2d respectively.
Revised: ( S = 3 , A = 2 , T = 10 ) that are depicted in Figure 2a, 2d respectively.

Original: Especially, we compare the expected entropy (3) achieved by an optimal non-Markovian policy π NM ∈ arg max π ∈ Π NM E ( π ) , which is obtained by solving the extended MDP as described in the proof of Lemma 4.2, against an optimal Markovian policy π M ∈ argmax
Revised: Especially, we compare the expected entropy (3) achieved by an optimal non-Markovian policy π NM ∈ arg max π ∈ Π NM E ( π ) , which is obtained by solving the extended MDP as described in the proof of Lemma 4.2, against an optimal Markovian policy π M [Equation] arg

Original: [Equation]
Revised: [Equation]

Original: ( π ) , which is obtained from π NM through a marginalization over histories (as mentioned in the proof sketch of Theorem 3.1).
Revised: which is obtained from π NM through a marginalization over histories (as mentioned in the proof sketch of Theorem 3.1).

Original: In conﬁrmation of the result in Theorem 4.4, π M cannot match the performance of π NM (see Figure 2b, 2e).
Revised: In conﬁrmation of the result in Theorem 4.4, π M cannot match the performance of π NM (see Figure 2b, 2e).

Original: In 3State , an optimal strategy requires going left when arriving in state 0 from state 2 and vice versa.
Revised: In 3State , an optimal strategy requires going left when arriving in state 0 from state 2 and vice versa.

Original: The policy π NM is able to do that, and it always realizes the optimal trajectory (Figure 2c).
Revised: The policy π NM is able to do that, and it always realizes the optimal trajectory (Figure 2c).

Original: Instead, π M is uniform in 0 and it often runs into sub-optimal trajectories.
Revised: Instead, π M is uniform in 0 and it often runs into sub-optimal trajectories.

Original: In the River Swim , the main hurdle is to reach state 2 from the initial one.
Revised: In the River Swim , the main hurdle is to reach state 2 from the initial one.

Original: Whereas π M and π NM are equivalently good in doing so, as reported in
Revised: Whereas π M and π NM are equivalently good in doing so, as reported in

Original: Figure 2f, only the non-Markovian strategy is able to balance the visitations in the previous states when it eventually reaches 2 .
Revised: Figure 2f, only the non-Markovian strategy is able to balance the visitations in the previous states when it eventually reaches 2 .

Original: The difference is already noticeable with a short horizon and it would further increase with a longer T .
Revised: The difference is already noticeable with a short horizon and it would further increase with a longer T .

Original: 7. Discussion and Conclusion
Revised: 7. Discussion and Conclusion

Original: In the previous sections, we detailed the importance of nonMarkovianity when optimizing a ﬁnite-sample MSE objective, but we also proved that the corresponding optimization problem is NP-hard in its general formulation.
Revised: In the previous sections, we detailed the importance of nonMarkovianity when optimizing a ﬁnite-sample MSE objective, but we also proved that the corresponding optimization problem is NP-hard in its general formulation.

Original: Despite the hardness result, we believe that it is not hopeless to learn exploration policies with some form of non-Markovianity, while still preserving an edge over Markovian strategies.
Revised: Despite the hardness result, we believe that it is not hopeless to learn exploration policies with some form of non-Markovianity, while still preserving an edge over Markovian strategies.

Original: In the following paragraphs, we discuss potential avenues to derive practical methods for relevant relaxations to the general class of non-Markovian policies.
Revised: In the following paragraphs, we discuss potential avenues to derive practical methods for relevant relaxations to the general class of non-Markovian policies.

Original: Finite-Length Histories Throughout the paper
Revised: Finite-Length Histories Throughout the paper

Original: , we considered non-Markovian policies that condition their decisions on histories of arbitrary length, i.e., π : H → ∆( A ) .
Revised: , we considered non-Markovian policies that condition their decisions on histories of arbitrary length, i.e., π : H → ∆( A ) .

Original: However, the complexity of optimizing such policies grows exponentially with the length of the history.
Revised: However, the complexity of optimizing such policies grows exponentially with the length of the history.

Original: To avoid this exponential blowup, one can deﬁne a class of non-Markovian policies π : H H → ∆( A ) in which the decisions are conditioned on histories of a ﬁnite length H > 1 that are obtained from a sliding window on the full history.
Revised: To avoid this exponential blowup, one can deﬁne a class of non-Markovian policies π : H H → ∆( A ) in which the decisions are conditioned on histories of a ﬁnite length H > 1 that are obtained from a sliding window on the full history.

Original: The optimal policy within this class would still retain better regret guarantees than an optimal Markovian policy, but it would not achieve zero regret in general.
Revised: The optimal policy within this class would still retain better regret guarantees than an optimal Markovian policy, but it would not achieve zero regret in general.

Original: With the length parameter H one can trade-off the learning complexity with the regret according to the structure of the domain.
Revised: With the length parameter H one can trade-off the learning complexity with the regret according to the structure of the domain.

Original: For instance, H =would be sufﬁcient to achieve zero regret in the 3State domain, whereas in the River Swim domain any H < T would cause some positive regret.
Revised: For instance, H =would be sufﬁcient to achieve zero regret in the 3State domain, whereas in the River Swim domain any H < T would cause some positive regret.

Original: Compact Representations of the History Instead of setting a ﬁnite length H , one can choose to perform function approximation on the full history to obtain a class of policies π : f ( H ) → ∆( A ) , where f is a function that maps an history h to some compact representation.
Revised: Compact Representations of the History Instead of setting a ﬁnite length H , one can choose to perform function approximation on the full history to obtain a class of policies π : f ( H ) → ∆( A ) , where f is a function that maps an history h to some compact representation.

Original: An interesting option is to use the notion of eligibility traces (Sutton &
Revised: An interesting option is to use the notion of eligibility traces (Sutton &

Original: Barto, 2018) to encode the information of h in a vector of
Revised: Barto, 2018) to encode the information of h in a vector of

Revised: [Figure]

Revised: E ( π )

Original: [Figure]
Revised: [Figure]

Original: {+Entropy 0 .0 .0 .
+}
Revised: {+Entropy 0 .0 .0 .
+}

Revised: Entropy ( d τ ( · ))

Revised: N ( τ ) π M π NM (c) Entropy Frequency

Original:  length S , which is updated as z t +1 ← λ z t + 1 s t , where λ ∈ (0 , 1) is a discount factor, 1 s t is a vector with a unit entry at the index s t , and z 0 = 0 .
Revised: [Figure] [Figure] length S , which is updated as z t +1 ← λ z t + 1 s t , where λ ∈ (0 , 1) is a discount factor, 1 s t is a vector with a unit entry at the index s t , and z 0 = 0 .

Original: The discount factor λ acts as a smoothed version of the length parameter H , and it can be dynamically adapted while learning.
Revised: The discount factor λ acts as a smoothed version of the length parameter H , and it can be dynamically adapted while learning.

Original: Indeed, this eligibility traces representation is particularly convenient for policy optimization (Deisenroth et al., 2013), in which we could optimize in turn a parametric policy over actions π θ ( ·| z , λ ) and a parametric policy over the discount π ν ( λ ) .
Revised: Indeed, this eligibility traces representation is particularly convenient for policy optimization (Deisenroth et al., 2013), in which we could optimize in turn a parametric policy over actions π θ ( ·| z , λ ) and a parametric policy over the discount π ν ( λ ) .

Original: To avoid a direct dependence on S , one can deﬁne the vector z over a discretization of the state space.
Revised: To avoid a direct dependence on S , one can deﬁne the vector z over a discretization of the state space.

Original: Deep Recurrent Policies
Revised: Deep Recurrent Policies

Original: Another noteworthy way to do function approximation on the history is to employ recurrent neural networks (Williams & Zipser, 1989;
Revised: Another noteworthy way to do function approximation on the history is to employ recurrent neural networks (Williams & Zipser, 1989;

Original: Hochreiter &
Revised: Hochreiter &

Original: Schmidhuber, 1997) to represent the non-Markovian policy.
Revised: Schmidhuber, 1997) to represent the non-Markovian policy.

Original: This kind of recurrent architecture is already popular in
Revised: This kind of recurrent architecture is already popular in

Original: RL.
Revised: RL.

Original: In this paper we are providing the theoretical ground to motivate the use of deep recurrent policies to address maximum state entropy exploration.
Revised: In this paper we are providing the theoretical ground to motivate the use of deep recurrent policies to address maximum state entropy exploration.

Original: Non-Markovian Control with Tree Search
Revised: Non-Markovian Control with Tree Search

Original: In principle, one can get a realization of actions from the optimal non-Markovian policy without ever computing it, e.g., by employing a Monte-Carlo Tree Search (MCTS)
Revised: In principle, one can get a realization of actions from the optimal non-Markovian policy without ever computing it, e.g., by employing a Monte-Carlo Tree Search (MCTS)

Original: (Kocsis &
Revised: (Kocsis &

Original: Szepesvári, 2006) approach to select the next action to take.
Revised: Szepesvári, 2006) approach to select the next action to take.

Original: Given the current state s t as a root, we can build the tree of trajectories from the root through repeated simulations of potential action sequences.
Revised: Given the current state s t as a root, we can build the tree of trajectories from the root through repeated simulations of potential action sequences.

Original: With a sufﬁcient number of simulations and a sufﬁciently deep tree, we are guaranteed to select the optimal action at the root.
Revised: With a sufﬁcient number of simulations and a sufﬁciently deep tree, we are guaranteed to select the optimal action at the root.

Original: If the horizon is too long, we can still cut the tree at any depth and approximately evaluate a leaf node with the entropy induced by the path from the root to the leaf.
Revised: If the horizon is too long, we can still cut the tree at any depth and approximately evaluate a leaf node with the entropy induced by the path from the root to the leaf.

Original: The drawback of this MCTS procedure is that we would require to have access to a simulator with reset (or a reliable estimate of the transition model) to actually build the tree.
Revised: The drawback of this MCTS procedure is that we would require to have access to a simulator with reset (or a reliable estimate of the transition model) to actually build the tree.

Original: Having reported interesting directions to learn nonMarkovian exploration policies in practice, we would like to mention some relevant online RL settings that might beneﬁt from such exploration policies.
Revised: Having reported interesting directions to learn nonMarkovian exploration policies in practice, we would like to mention some relevant online RL settings that might beneﬁt from such exploration policies.

Original: We leave as future work a more formal deﬁnition of the settings and a thorough empirical study.
Revised: We leave as future work a more formal deﬁnition of the settings and a thorough empirical study.

Original: Single-Trial RL
Revised: Single-Trial RL

Original: In many relevant real-world scenarios, where data collection might be costly or non-episodic in nature, we cannot afford multiple trials to achieve the desired exploration of the environment.
Revised: In many relevant real-world scenarios, where data collection might be costly or non-episodic in nature, we cannot afford multiple trials to achieve the desired exploration of the environment.

Original: Non-Markovian exploration policies guarantee a good coverage of the environment in a single trial and they are particularly suitable for online learning processes.
Revised: Non-Markovian exploration policies guarantee a good coverage of the environment in a single trial and they are particularly suitable for online learning processes.

Original: Learning in Latent MDPs In a latent MDP scenario (Hallak et al., 2015; Kwon et al., 2021) an agent interacts with an (unknown) environment drawn from a class of MDPs to solve an online RL task.
Revised: Learning in Latent MDPs In a latent MDP scenario (Hallak et al., 2015; Kwon et al., 2021) an agent interacts with an (unknown) environment drawn from a class of MDPs to solve an online RL task.

Original: A non-Markovian exploration policy pre-trained on the whole class could exploit the memory to perform a fast identiﬁcation of the speciﬁc context that has been drawn, quickly adapting to the optimal environmentspeciﬁc policy.
Revised: A non-Markovian exploration policy pre-trained on the whole class could exploit the memory to perform a fast identiﬁcation of the speciﬁc context that has been drawn, quickly adapting to the optimal environmentspeciﬁc policy.

Original: To conclude, we believe that this work sheds some light on the, previously neglected, importance of non-Markovianity to address maximum state entropy exploration, and we believe it can provide inspiration for future empirical and theoretical contributions on the matter.
Revised: To conclude, we believe that this work sheds some light on the, previously neglected, importance of non-Markovianity to address maximum state entropy exploration, and we believe it can provide inspiration for future empirical and theoretical contributions on the matter.

Original: A. Related Work
Revised: A. Related Work

Original: Hazan et al.
Revised: Hazan et al.

Original: (Hazan et al., 2019) were the ﬁrst to consider an entropic measure over the state distribution as a sensible learning objective for an agent interacting with a reward-free environment (Jin et al., 2020).
Revised: (Hazan et al., 2019) were the ﬁrst to consider an entropic measure over the state distribution as a sensible learning objective for an agent interacting with a reward-free environment (Jin et al., 2020).

Original: Especially, they propose an algorithm, called MaxEnt, that learns a mixture of policies that collectively maximize the Shannon entropy, i.e., (1), of the discounted state distribution.
Revised: Especially, they propose an algorithm, called MaxEnt, that learns a mixture of policies that collectively maximize the Shannon entropy, i.e., (1), of the discounted state distribution.

Original: The ﬁnal mixture is learned through a conditional gradient method, in which the algorithm iteratively estimates the state distribution of the current mixture to deﬁne an intrinsic reward function, and then identiﬁes the next policy to be added by solving a speciﬁc RL sub-problem with this reward.
Revised: The ﬁnal mixture is learned through a conditional gradient method, in which the algorithm iteratively estimates the state distribution of the current mixture to deﬁne an intrinsic reward function, and then identiﬁes the next policy to be added by solving a speciﬁc RL sub-problem with this reward.

Original: A similar methodology has been obtained by
Revised: A similar methodology has been obtained by

Original: Lee et al.
Revised: Lee et al.

Original: (Lee et al., 2019) from a game-theoretic perspective on the MSE exploration problem.
Revised: (Lee et al., 2019) from a game-theoretic perspective on the MSE exploration problem.

Original: Their algorithms, called
Revised: Their algorithms, called

Original: SMM, targets the Shannon entropy of the marginal state distribution instead of the discounted distribution of MaxEnt.
Revised: SMM, targets the Shannon entropy of the marginal state distribution instead of the discounted distribution of MaxEnt.

Original: Another approach based on the conditional gradient method is FW-AME (Tarbouriech & Lazaric, 2019), which learns a mixture of policies to maximize the entropy of the stationary state-action distribution.
Revised: Another approach based on the conditional gradient method is FW-AME (Tarbouriech & Lazaric, 2019), which learns a mixture of policies to maximize the entropy of the stationary state-action distribution.

Original: As noted in (Tarbouriech & Lazaric, 2019), the mixture of policies might suffer a slow mixing to the asymptotic distribution for which the entropy is maximized.
Revised: As noted in (Tarbouriech & Lazaric, 2019), the mixture of policies might suffer a slow mixing to the asymptotic distribution for which the entropy is maximized.

Original: In (Mutti & Restelli, 2020), the authors present a method (IDE 3 AL) to learn a single exploration policy that simultaneously accounts for the entropy of the stationary state-action distribution and the mixing time.
Revised: In (Mutti & Restelli, 2020), the authors present a method (IDE 3 AL) to learn a single exploration policy that simultaneously accounts for the entropy of the stationary state-action distribution and the mixing time.

Original: Even if they are sometimes evaluated on continuous domains (especially (Hazan et al., 2019; Lee et al., 2019)), the methods we mentioned require an accurate estimate of either the state distribution (Hazan et al., 2019; Lee et al., 2019) or the transition model (Tarbouriech & Lazaric, 2019; Mutti & Restelli, 2020), which hardly scales to high-dimensional domains.
Revised: Even if they are sometimes evaluated on continuous domains (especially (Hazan et al., 2019; Lee et al., 2019)), the methods we mentioned require an accurate estimate of either the state distribution (Hazan et al., 2019; Lee et al., 2019) or the transition model (Tarbouriech & Lazaric, 2019; Mutti & Restelli, 2020), which hardly scales to high-dimensional domains.

Original: A subsequent work (Mutti et al., 2020) proposes an approach to estimate the entropy of the state distribution through a non-parametric method, and then to directly optimize the estimated entropy via policy optimization.
Revised: A subsequent work (Mutti et al., 2021b) proposes an approach to estimate the entropy of the state distribution through a non-parametric method, and then to directly optimize the estimated entropy via policy optimization.

Original: Their algorithm, called MEPOL, is able to learn a single exploration policy that maximizes the entropy of the marginal state distribution in challenging continuous control domains.
Revised: Their algorithm, called MEPOL, is able to learn a single exploration policy that maximizes the entropy of the marginal state distribution in challenging continuous control domains.

Original: Liu and Abbeel (Liu & Abbeel, 2021) combine non-parametric entropy estimation with learned state representations into an algorithm, called APT, that successfully addresses MSE exploration problems in visual-inputs domains.
Revised: Liu and Abbeel (Liu & Abbeel, 2021b) combine non-parametric entropy estimation with learned state representations into an algorithm, called APT, that successfully addresses MSE exploration problems in visual-inputs domains.

Original: Seo et al.
Revised: Seo et al.

Original: (Seo et al., 2021) shows that even random state representations are sufﬁcient to learn MSE exploration policies from visual inputs.
Revised: (Seo et al., 2021) shows that even random state representations are sufﬁcient to learn MSE exploration policies from visual inputs.

Original: Whereas all of the previous approaches accounts for the Shannon entropy in their objectives, recent works (Zhang et al., 2020a; Guo et al., 2021) consider alternative formulations.
Revised: Whereas all of the previous approaches accounts for the Shannon entropy in their objectives, recent works (Zhang et al., 2020a; Guo et al., 2021) consider alternative formulations.

Original: Especially, Zhang et al.
Revised: Especially, Zhang et al.

Original: (Zhang et al., 2020a) argues that the
Revised: (Zhang et al., 2020a) argues that the

Original: Rényi entropy provides a superior incentive to cover all of the corresponding space than the Shannon entropy, and they propose a method to optimize the Rényi of the state-action distribution via gradient ascent (MaxRényi).
Revised: Rényi entropy provides a superior incentive to cover all of the corresponding space than the Shannon entropy, and they propose a method to optimize the Rényi of the state-action distribution via gradient ascent (MaxRényi).

Original: On an orthogonal direction, the authors of (Guo et al., 2021) consider a reformulation of the entropy function that accounts for the underlying geometry of the space.
Revised: On an orthogonal direction, the authors of (Guo et al., 2021) consider a reformulation of the entropy function that accounts for the underlying geometry of the space.

Original: They present a method, called GEM, to learn an optimal policy for the geometry-aware entropy objective.
Revised: They present a method, called GEM, to learn an optimal policy for the geometry-aware entropy objective.

Original: -parametric entropy estimation (Non-Parametric)

Original: [Table]
Revised: [Table]

Original: B. Missing Proofs B.1. Proofs of Section
Revised: B. Missing Proofs B.1. Proofs of Section

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Let x ∈ {∞ , γ, T } , and let D x NM = { d πx ( · ) : π ∈ Π NM } , D x NS = { d πx ( · ) : π ∈ Π NS } , D x M = { d πx ( · ) : π ∈ Π M } the corresponding sets of distributions.
Revised: Let x ∈ {∞ , γ, T } , and let D x NM = { d πx ( · ) : π ∈ Π NM } , D x NS = { d πx ( · ) : π ∈ Π NS } , D x M = { d πx ( · ) : π ∈ Π M } the corresponding sets of distributions.

Original: We can prove that:
Revised: We can prove that:

Original: (i)
Revised: (i)

Original: The sets of stationary state distributions are equivalent D ∞ NM ≡ D ∞ NS ≡ D ∞ M ; (ii) The sets of discounted state distributions are equivalent D γ NM ≡ D γ NS ≡ D γ M for any γ ; (iii) The sets of marginal state distributions are equivalent D T NM ≡ D T NS ≡ D T M for any T .
Revised: The sets of stationary state distributions are equivalent D ∞ NM ≡ D ∞ NS ≡ D ∞ M ; (ii) The sets of discounted state distributions are equivalent D γ NM ≡ D γ NS ≡ D γ M for any γ ; (iii) The sets of marginal state distributions are equivalent D T NM ≡ D T NS ≡ D T M for any T .

Original: Proof.
Revised: Proof.

Original: First, note that a non-Markovian policy π ∈ Π NM can always reduce to a non-stationary policy π ∈ Π NS by conditioning the decision rules on the history length, or to a Markovian policy π ∈ Π M by conditioning the decision rules on the last history entry.
Revised: First, note that a non-Markovian policy π ∈ Π NM can always reduce to a non-stationary policy π ∈ Π NS by conditioning the decision rules on the history length, or to a Markovian policy π ∈ Π M by conditioning the decision rules on the last history entry.

Original: Thus, D x NM ⊇ D x NS ⊇ D x M is straightforward for any x ∈ {∞ , γ, T } .
Revised: Thus, D x NM ⊇ D x NS ⊇ D x M is straightforward for any x ∈ {∞ , γ, T } .

Original: From the derivations in (Puterman, 2014, Theorem 5.5.1), we have that D x NS ⊇ D x NM as well.
Revised: From the derivations in (Puterman, 2014, Theorem 5.5.1), we have that D x NS ⊇ D x NM as well.

Original: Indeed, for any non-Markovian policy π ∈ Π NM , we can build a non-stationary policy π (cid:

Revised: Indeed, for any non-Markovian policy π ∈ Π NM , we can build a non-stationary policy π (cid:

Original: ) ∈
Revised: ) ∈

Original: Π NS as π (cid:48)
Revised: Π NS as [Equation]

Original: = (cid:0) π (cid:48) 1 , π (cid:48) 2 , . . , π (cid:48) t , . . . (cid:1) , such thatπ (cid:48)
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: For t = 0 , we have that d π 0 ( · ) =
Revised: For t = 0 , we have that d π 0 ( · ) =

Original: d π (cid:48) 

Revised: d π (cid:48)

Original: ( · ) = µ ( · ) , which is the initial state distribution.
Revised: ( · ) = µ ( · ) , which is the initial state distribution.

Original: We proceed by induction to show that if d πt − 1 ( · ) =
Revised: We proceed by induction to show that if d πt − 1 ( · ) =

Original: d π (cid:48)
Revised: d π (cid:48)

Original: t − 1 ( · ) , then we have
Revised: t − 1 ( · ) , then we have

Original: [Equation]
Revised: [Equation]

Original: Since d πt ( s ) =
Revised: Since d πt ( s ) =

Original: d π (cid:48)
Revised: d π (cid:48)

Original: t ( s ) holds for any t ≥ 0 and ∀ s ∈ S , we have d π ∞ ( · ) = 
Revised: t ( s ) holds for any t ≥ 0 and ∀ s ∈ S , we have d π ∞ ( · ) = d π

Original: d π (cid:48) ∞ ( · ) , d πγ ( · ) = 
Revised: [Equation] ∞ ( · ) , d πγ ( · ) = d π

Original: d π (cid:48) γ ( · ) , d πT ( · ) =
Revised: [Equation] γ ( · ) , d πT ( · ) =

Original: d π (cid:48) T ( · ) , and thus
Revised: d π (cid:48) T ( · ) , and thus

Original: D x NS ⊇ D x NM .
Revised: D x NS ⊇ D x NM .

Original: Then, D x NM ≡ D x NS follows.
Revised: Then, D x NM ≡ D x NS follows.

Original: [Equation]
Revised: [Equation]

Original: This result can be obtained with the same construction as before.
Revised: This result can be obtained with the same construction as before.

Original: Let us consider a non-Markovian policy π ∈ Π NM having mixing time t mix := { t ∈ N : sup s ∈S | d π ∞ ( s ) − d πt ( s ) | ≤ (cid:15) } for some mixing threshold (cid:15) .
Revised: Let us consider a non-Markovian policy π ∈ Π NM having mixing time t mix := { t ∈ N : sup s ∈S | d π ∞ ( s ) − d πt ( s ) | ≤ (cid:15) } for some mixing threshold (cid:15) .

Original: Then, we can deﬁne a Markovian policy π M ∈ Π M as π M ( a | s ) =
Revised: Then, we can deﬁne a Markovian policy π M ∈ Π M as π M ( a | s ) =

Original: d π t mix ( s, a ) (cid:14) d π t mix ( s ) , ∀ s ∈ S , ∀ a ∈ A .
Revised: d π t mix ( s, a ) (cid:14) d π t mix ( s ) , ∀ s ∈ S , ∀ a ∈ A .

Original: If we construct the same non-stationary policy π (cid:48) t = ( π (cid:48) 1 , π (cid:48) 2 , . .
Revised: If we construct the same non-stationary policy π (cid:48) t = ( π (cid:48) 1 , π (cid:48) 2 , . .

Original: , π (cid:48) t , . . . )
Revised: , π (cid:48) t , . . . )

Original: ∈
Revised: ∈

Original: Π NS as before, i.e., π (cid:48)
Revised: Π NS as before, i.e., π (cid:48)

Original: ( a | s ) =
Revised: ( a | s ) =

Original: d πt ( s, a ) (cid:14) d πt ( s ) , ∀ s ∈ S , ∀ a ∈ A ,we have that d π (cid:48) t ( · ) =
Revised: d πt ( s, a ) (cid:14) d πt ( s ) , ∀ s ∈ S , ∀ a ∈ A ,we have that d π (cid:48) t ( · ) =

Original: d πt ( · ) for any t ≥ 0 , and thus also for t ≥ t mix .
Revised: d πt ( · ) for any t ≥ 0 , and thus also for t ≥ t mix .

Original: This implies that π, π (cid:48) , π M all converges to the same stationary state distribution d π ∞ ( · ) =
Revised: This implies that π, π (cid:48) , π M all converges to the same stationary state distribution d π ∞ ( · ) =

Original: lim t →∞ d πt ( · ) if we take the limit (cid:15) → 0 on the mixing threshold.
Revised: lim t →∞ d πt ( · ) if we take the limit (cid:15) → 0 on the mixing threshold.

Original: Then, we have D ∞ NM ⊇ D ∞ NS ⊇ D ∞ M and
Revised: Then, we have D ∞ NM ⊇ D ∞ NS ⊇ D ∞ M and

Original: D ∞ NM ≡ D ∞ NS ≡ D ∞ M .
Revised: D ∞ NM ≡ D ∞ NS ≡ D ∞ M .

Original: [Equation]
Revised: [Equation]

Original: We can prove the statement by showing that D γ M ⊇ D γ NS .
Revised: We can prove the statement by showing that D γ M ⊇ D γ NS .

Original: To this purpose, let us deﬁne the T -bounded discounted state distribution as d πγ,T ( · ) :
Revised: To this purpose, let us deﬁne the T -bounded discounted state distribution as d πγ,T ( · ) :

Original: = (1 − γ ) (1 − γ T ) (cid:80) t ∈
Revised: = (1 − γ ) (1 − γ T ) (cid:80) t ∈

Original: [ T ] γ t d πt ( · ) , such that lim T →∞ d πγ,T ( · ) =
Revised: [ T ] γ t d πt ( · ) , such that lim T →∞ d πγ,T ( · ) =

Original: d πγ ( · ) .
Revised: d πγ ( · ) .

Original: Then, We consider a general non-stationary policy π =
Revised: Then, We consider a general non-stationary policy π =

Original: ( π 0 , π 1 , . . .
Revised: ( π 0 , π 1 , . . .

Original: , π t , . . . )
Revised: , π t , . . . )

Original: ∈
Revised: ∈

Original: Π NS .
Revised: Π NS .

Original: We can build a Markovian policy π (cid:

Revised: We can build a Markovian policy π (cid:

Original: ) ∈
Revised: ) ∈

Original: Π M as follows
Revised: Π M as follows

Original: [Equation]
Revised: [Equation]

Original: We prove by induction that such a policy π (cid:48) induces the same T -bounded discounted state distribution of π for every T > 0 .
Revised: We prove by induction that such a policy π (cid:48) induces the same T -bounded discounted state distribution of π for every T > 0 .

Original: For T = 1 we have d π(cid:48) γ,T ( · ) = d πγ,T ( · ) = µ ( · ) . Then, if d π (cid:48) γ,T
Revised: For T = 1 we have d π

Original: [Equation]
Revised: [Equation]

Original:  1 ( · ) = d πγ,T − 1 ( · ) , for every s ∈ S it follows
Revised: γ,T ( · ) = d πγ,T ( · ) = µ ( · ) . Then, if d π (cid:48) γ,T − 1 ( · ) = d πγ,T − 1 ( · ) , for every s ∈ S it follows

Original: [Equation]
Revised: [Equation]

Original: (cid:80)
Revised: [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: (cid:80)
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: t = γ t d
Revised: [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: (cid:48) )
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: ( a (cid:48)
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: s (cid:48)
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Since the relation holds for any T > 0 , we can take the limit for T → ∞ to have d π (cid:48) γ ( · ) =
Revised: Since the relation holds for any T > 0 , we can take the limit for T → ∞ to have d π (cid:48) γ ( · ) =

Original: d πγ ( · ) , which gives D γ M ⊇ D γ NS and then D γ NM ≡ D γ NS ≡ D γ M .
Revised: d πγ ( · ) , which gives D γ M ⊇ D γ NS and then D γ NM ≡ D γ NS ≡ D γ M .

Original: [Equation]
Revised: (iii) D T NM ≡ D T NS ≡ D T M

Original: We can prove the statement by showing that D T M ⊇ D T NS .
Revised: We can prove the statement by showing that D T M ⊇ D T NS .

Original: We consider a general non-stationary policy π = ( π 0 , π 1 , . . .
Revised: We consider a general non-stationary policy π = ( π 0 , π 1 , . . .

Original: , π t , . . . )
Revised: , π t , . . . )

Original: ∈
Revised: ∈

Original: Π NS .
Revised: Π NS .

Original: We can build a Markovian policy π (cid:

Revised: We can build a Markovian policy π (cid:

Original: ) ∈
Revised: ) ∈

Original: Π M which marginalizes π over the time steps, such that
Revised: Π M which marginalizes π over the time steps, such that

Original: [Equation]
Revised: [Equation]

Original: a

Original: [Equation]

Original: ∀

Original: [Equation]

Original: ∀

Original: [Equation]

Original: ∈ A .

Original: We prove the statement by induction.
Revised: We prove the statement by induction.

Original: For T = 1 we have d π (cid:48) T ( · ) =
Revised: For T = 1 we have d π (cid:48) T ( · ) =

Original: d πT ( · ) = µ ( · ) .
Revised: d πT ( · ) = µ ( · ) .

Original: Then, we can show that if d π (cid:48)
Revised: Then, we can show that if d π (cid:48)

Original: T − 1 ( · ) =
Revised: T − 1 ( · ) =

Original: d πT − 1 ( · ) , it follows d π (cid:48) T ( · ) =
Revised: d πT − 1 ( · ) , it follows d π (cid:48) T ( · ) =

Original: d πT ( · ) .
Revised: d πT ( · ) .

Original: Especially,
Revised: Especially,

Original: [Equation]
Revised: [Equation]

Original: | s (cid:48) )
Revised: [Table] [Equation] [Equation] [Table]

Original: P ( s | s (cid:48) , a (cid:48) )
Revised: [Equation] [Table] [Equation] [Table]

Original: = 1 T
Revised: [Equation]

Original: T − 2 (cid:88) t = (cid:88) s (cid:48) ∈S d πt ( s (cid:48) )
Revised: [Table] [Table] [Equation]

Original: (cid:88) a (cid:48) ∈A π t ( a (cid:48) | s (cid:48) )
Revised: [Table] [Equation] [Table] = 1 T

Original: P ( s | s (cid:48) , a (cid:48) )
Revised: [Table]

Original: = 1 T
Revised: = 1 T

Original: T − 1 (cid:88) t = d πt ( s ) =
Revised: T − 1 (cid:88) t = d πt ( s ) =

Original: d πT ( s ) , holds for any s ∈ S , thus D T M ⊇ D T NS and D T NM ≡ D T NS ≡ D T M follows.
Revised: d πT ( s ) , holds for any s ∈ S , thus D T M ⊇ D T NS and D T NM ≡ D T NS ≡ D T M follows.

Original: Corollary 3.2.
Revised: Corollary 3.2.

Original: For every CMP M , there exists a Markovian policy π ∗ ∈ Π M such that π ∗ ∈ arg max π ∈ Π E ∞ ( π ) .
Revised: For every CMP M , there exists a Markovian policy π ∗ ∈ Π M such that π ∗ ∈ arg max π ∈ Π E ∞ ( π ) .

Original: Proof.
Revised: Proof.

Original: The result is straightforward from Theorem 3.1 and noting that the set of non-Markovian policies Π NM with arbitrary history-length is as powerful as the general set of policies Π .
Revised: The result is straightforward from Theorem 3.1 and noting that the set of non-Markovian policies Π NM with arbitrary history-length is as powerful as the general set of policies Π .

Original: Thus, for every policy π ∈ Π there exists a (possibly randomized) policy π (cid:

Revised: Thus, for every policy π ∈ Π there exists a (possibly randomized) policy π (cid:

Original: ) ∈
Revised: ) ∈

Original: Π M inducing the same (stationary, discounted or marginal) state distribution of π , i.e., d π ( · ) =
Revised: Π M inducing the same (stationary, discounted or marginal) state distribution of π , i.e., d π ( · ) =

Original: d π (cid:48) ( · ) , which implies Entropy (cid:0) d π ( · ) (cid:1) =
Revised: d π (cid:48) ( · ) , which implies Entropy (cid:0) d π ( · ) (cid:1) =

Original: Entropy (cid:0) d π (cid:48) ( · ) (cid:1) .
Revised: Entropy (cid:0) d π (cid:48) ( · ) (cid:1) .

Original: If it holds for any π ∈ Π , then it holds for π ∗ ∈ arg max
Revised: If it holds for any π ∈ Π , then it holds for π ∗ ∈ arg max

Original: π ∈ Π Entropy (cid:0) d π ( · ) (cid:1) .
Revised: π ∈ Π Entropy (cid:0) d π ( · ) (cid:1) .

Original: B.2. Proofs of Section
Revised: B.2. Proofs of Section

Original: Lemma 4.3.
Revised: Lemma 4.3.

Original: Let π NM ∈ Π DNM be a non-Markovian policy such that π NM ∈ arg max π ∈ Π E
Revised: Let π NM ∈ Π DNM be a non-Markovian policy such that π NM ∈ arg max π ∈ Π E

Original: ( π ) on a CMP M .
Revised: ( π ) on a CMP M .

Original: The variance of an optimal Markovian policy π M ∈ arg max π ∈ Π M E ( π ) is given by
Revised: The variance of an optimal Markovian policy π M ∈ arg max π ∈ Π M E ( π ) is given by

Original: [Equation]
Revised: [Equation]

Original: where hs is any history hs ∈ H
Revised: where hs is any history hs ∈ H

Original: [ T ] such that the ﬁnal state is s .
Revised: [ T ] such that the ﬁnal state is s .

Original: Proof.
Revised: Proof.

Original: Let us derive the variance of a policy π ∈ Π in state s ∈ S through the law of total variance (Bertsekas & Tsitsiklis, 2002).
Revised: Let us derive the variance of a policy π ∈ Π in state s ∈ S through the law of total variance (Bertsekas & Tsitsiklis, 2002).

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Let h ∼ p πT , so that the variable a | s, h becomes a | hs where hs = ( s 0 ,hs , a 0 ,hs , s 1 ,hs , . . .
Revised: Let h ∼ p πT , so that the variable a | s, h becomes a | hs where hs = ( s 0 ,hs , a 0 ,hs , s 1 ,hs , . . .

Original: , s t,
Revised: , s t,

Original: hs = s ) ∈ H [ T ] , to obtain
Revised: hs = s ) ∈ H [ T ] , to obtain

Original: [Equation]
Revised: [Equation]

Original: If we take the policy π that optimizes the objective (3), we have that the action a on the left-hand side of equation (4) is distributed according to π M arg max π ∈ Π M E ( π ) , while on the right-hand side is distributed according to π NM arg max π ∈ Π NM E ( π ) , i.e.,
Revised: If we take the policy π that optimizes the objective (3), we have that the action a on the left-hand side of equation (4) is distributed according to π M arg max π ∈ Π M E ( π ) , while on the right-hand side is distributed according to π NM arg max π ∈ Π NM E ( π ) , i.e.,

Original: [Equation]
Revised: [Equation]

Original: From Lemma 4.2, we know that there exists a deterministic optimal non-Markovian policy π NM ∈ Π DNM , which gives E hs ∼ p π NM T (cid:2) V ar a ∼ π NM ( hs ) [ a ] (cid:3) = 0 in (5) and concludes the proof.
Revised: From Lemma 4.2, we know that there exists a deterministic optimal non-Markovian policy π NM ∈ Π DNM , which gives E hs ∼ p π NM T (cid:2) V ar a ∼ π NM ( hs ) [ a ] (cid:3) = 0 in (5) and concludes the proof.

Original: Lemma B.1 (Best-Case CMP) .
Revised: Lemma B.1 (Best-Case CMP) .

Original: Let h ∗ t be a zero-regret trajectory of t -steps.
Revised: Let h ∗ t be a zero-regret trajectory of t -steps.

Original: Taking a sub-optimal action a t ∈
Revised: Taking a sub-optimal action a t ∈

Original: A \ π NM ( h ∗ t ) at step t in the best-case CMP M gives a ﬁnal entropy
Revised: A \ π NM ( h ∗ t ) at step t in the best-case CMP M gives a ﬁnal entropy

Original: [Equation]
Revised: [Equation]

Original: where the maximum is attained by
Revised: where the maximum is attained by

Original: [Equation]
Revised: [Equation]

Original: in which s max, 2 is any state that is the second-closest to a uniform entry in d ( h ∗ t ,h T − t ) .
Revised: in which s max, 2 is any state that is the second-closest to a uniform entry in d ( h ∗ t ,h T − t ) .

Original: Proof.
Revised: Proof.

Original: The best-case CMP M is designed such that taking a sub-optimal action a t ∈
Revised: The best-case CMP M is designed such that taking a sub-optimal action a t ∈

Original: A \ π NM ( h ∗ t ) minimally decrease the ﬁnal entropy.
Revised: A \ π NM ( h ∗ t ) minimally decrease the ﬁnal entropy.

Original: Especially, instead of reaching at step t + 1 an optimal state s max , i.e., a state that maximally balances the state visits of the ﬁnal trajectory, the agent is drawn to the second-to-optimal state s max, 2 , from which it gets back on track on the optimal trajectory for the remaining steps.
Revised: Especially, instead of reaching at step t + 1 an optimal state s max , i.e., a state that maximally balances the state visits of the ﬁnal trajectory, the agent is drawn to the second-to-optimal state s max, 2 , from which it gets back on track on the optimal trajectory for the remaining steps.

Original: Note that visiting s max, 2 cannot lead to the optimal ﬁnal entropy, achieved when s max is visited at step t + 
Revised: Note that visiting s max, 2 cannot lead to the optimal ﬁnal entropy, achieved when s max is visited at step t + 1 , due to the sub-optimality of action a t .

Original: Lemma B.2 (Worst-Case CMP) .
Revised: Lemma B.2 (Worst-Case CMP) .

Original: Let h ∗ t be a zero-regret trajectory of t -steps.
Revised: Let h ∗ t be a zero-regret trajectory of t -steps.

Original: Taking a sub-optimal action a t ∈ A\ π NM ( h ∗ t ) at step t in the worst-case CMP M gives a ﬁnal entropy
Revised: Taking a sub-optimal action a t ∈ A\ π NM ( h ∗ t ) at step t in the worst-case CMP M gives a ﬁnal entropy

Original: [Equation]
Revised: [Equation]

Original: where the minimum is attained by
Revised: where the minimum is attained by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The worst-case CMP M is designed such that the agent cannot recover from a sub-optimal action a t ∈
Revised: The worst-case CMP M is designed such that the agent cannot recover from a sub-optimal action a t ∈

Original: A \ π NM ( h ∗ t ) as it is absorbed by a worst-case state given the trajectory h ∗ t .
Revised: A \ π NM ( h ∗ t ) as it is absorbed by a worst-case state given the trajectory h ∗ t .

Original: A worst-case state is one that maximizes the visitation frequency in h ∗ t , i.e., s ∈ arg max s ∈S d h ∗ t ( s ) , so that the visitation frequency becomes increasingly unbalanced.
Revised: A worst-case state is one that maximizes the visitation frequency in h ∗ t , i.e., s ∈ arg max s ∈S d h ∗ t ( s ) , so that the visitation frequency becomes increasingly unbalanced.

Original: A sub-optimal action at the ﬁrst step in M leads to T − 1 visits to the initial state s 0 ∼ µ , and the ﬁnal entropy is zero.
Revised: A sub-optimal action at the ﬁrst step in M leads to T − 1 visits to the initial state s 0 ∼ µ , and the ﬁnal entropy is zero.

Original: Theorem 4.4.
Revised: Theorem 4.4.

Original: Let π M ∈
Revised: Let π M ∈

Original: Π M be a Markovian policy such that π M ∈ arg max π ∈ Π M E ( π ) on a CMP M .
Revised: Π M be a Markovian policy such that π M ∈ arg max π ∈ Π M E ( π ) on a CMP M .

Original: Then, for any t ∈ [ T ] , it holds R T − t ( π M ) ≤ R T − t 
Revised: Then, for any t ∈ [ T ] , it holds R T − t ( π M ) ≤ R T − t ( π M

Revised: [Equation]

Original: ( π M ) ≤ R T − t ( π M ) such that
Revised: ≤ R T − t ( π M ) such that

Original: [Equation]
Revised: [Equation]

Original: where π NM ∈ arg max π ∈ Π DNM E ( π ) , a NM = π NM ( h ∗ t ) is the unique optimal action in s t , and E max , E max, 2 , E min,t are given by
Revised: where π NM ∈ arg max π ∈ Π DNM E ( π ) , a NM = π NM ( h ∗ t ) is the unique optimal action in s t , and E max , E max, 2 , E min,t are given by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From the deﬁnition of the regret-to-go (Deﬁnition 4.1), we have that
Revised: From the deﬁnition of the regret-to-go (Deﬁnition 4.1), we have that

Original: [Equation]
Revised: [Equation]

Original: in which we substitute E max = max [Equation]
Revised: in which we substitute E max = max π ∗ ∈ Π E h ∗ T − t ∼ p π ∗ (cid:2)

Original: Entropy (cid:0) d h ∗ T ( · ) (cid:1)(cid:3) .
Revised: Entropy (cid:0) d h ∗ T ( · ) (cid:1)(cid:3) .

Original: To derive a lower bound and an upper bound to
Revised: To derive a lower bound and an upper bound to

Original: R T − t ( π ) we consider the impact that taking a sub-optimal action
Revised: R T − t ( π ) we consider the impact that taking a sub-optimal action

Original: a ∈
Revised: a ∈

Original: A \ { a NM } in state s t would have in a best-case and a worst-case CMP respectively, which is detailed in Lemma B.1 and Lemma B.2.
Revised: A \ { a NM } in state s t would have in a best-case and a worst-case CMP respectively, which is detailed in Lemma B.1 and Lemma B.2.

Original: Especially, we can write
Revised: Especially, we can write

Original: [Equation]
Revised: [Equation]

Original:  and
Revised: = E max − E h T − t ∼ p π M T (cid:2) Entropy (cid:0) d h T ( · ) ( cid:1)(cid:3) ≥ E max − π M ( a NM | s t ) E max − (cid:0) 1 − π M ( a NM | s t ) (cid:1) E max,= ( E max − E max, 2 ) (cid:0) 1 − π NM ( a NM | s t ) (cid:1) and

Original: [Equation]
Revised: R T − t ( π M ) = E max − E h T − t ∼ p π M T (cid:2) Entropy (cid:0) d h T ( · ) (cid:1)(cid:3) ≤ E max − π M ( a NM | s t )

Revised: E max − (cid:0) 1 − π M ( a NM | s t ) (cid:1) E min,t = ( E max − E min,t ) (cid:0) 1 − π NM ( a NM | s t ) (cid:1) .

Original: Then, we note that the event of taking a sub-optimal action a ∈
Revised: Then, we note that the event of taking a sub-optimal action a ∈

Original: A \ { a NM } with policy π M can be modelled by a Bernoulli distribution with parameter (cid:0) 1 − π NM ( a NM | s t ) (cid:1) .
Revised: A \ { a NM } with policy π M can be modelled by a Bernoulli distribution with parameter (cid:0) 1 − π NM ( a NM | s t ) (cid:1) .

Original: By combining the equation of the variance of a Bernoulli distribution and
Revised: By combining the equation of the variance of a Bernoulli distribution and

Original: Lemma 4.3 we obtain
Revised: Lemma 4.3 we obtain

Original: [Equation]
Revised: [Equation]

Original: which gives
Revised: which gives

Original: [Equation]
Revised: [Equation]

Original: Corollary 4.6.
Revised: Corollary [Equation] [Equation]

Original: Let π M ∈
Revised: Let π M ∈

Original: Π M be a Markovian policy such that π M ∈ arg max π ∈ Π M E ( π ) on a CMP M .
Revised: Π M be a Markovian policy such that π M ∈ arg max π ∈ Π M E ( π ) on a CMP M .

Original: Then, for any t ∈ [ T ], it holds r t ( π M ) ≤ r t ( π M ) ≤ r t ( π M ) such that
Revised: Then, for any t ∈ [ T ]

Original: [Equation]
Revised: [Equation]

Revised: it holds r t ( π M ) ≤ r t ( π M ) ≤ r t ( π M ) such that

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From Deﬁnition 4. [Equation] we have that r t ( π M ) =
Revised: From Deﬁnition 4.5, we have that r t ( π M ) =

Original: R T − t − R T − t − 1 .
Revised: R T − t − R T − t − 1 .

Original: Recall that
Revised: Recall that

Original: [Equation]
Revised: [Equation]

Original: from Theorem 
Revised: from Theorem 4.4.

Original: [Equation]

Original: We can write
Revised: We can write

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: B.3. Proofs of Section
Revised: B.3. Proofs of Section

Original: Theorem 5.3.
Revised: Theorem 5.3.

Original: Ψ 0 ∈ NP -hard.
Revised: Ψ 0 ∈ NP -hard.

Original: Proof.
Revised: Proof.

Original: To prove that Ψ 0 ∈ NP-hard, it is sufﬁcient to show that there exists a problem Ψ c ∈ NP-complete so that Ψ c ≤ p Ψ 0 .
Revised: To prove that Ψ 0 ∈ NP-hard, it is sufﬁcient to show that there exists a problem Ψ c ∈ NP-complete so that Ψ c ≤ p Ψ 0 .

Original: We show this by reducing 3SAT, a well-known NP-complete problem, to Ψ 0 .
Revised: We show this by reducing 3SAT, a well-known NP-complete problem, to Ψ 0 .

Original: To derive the reduction we consider two intermediate problems, namely Ψ 1 and Ψ 2 .
Revised: To derive the reduction we consider two intermediate problems, namely Ψ 1 and Ψ 2 .

Original: Especially, we aim to show that the following chain of reductions hold:
Revised: Especially, we aim to show that the following chain of reductions hold:

Original: [Equation]
Revised: [Equation]

Original: First, we deﬁne Ψ 1 and we prove that Ψ 0 ≥ m Ψ 1 .
Revised: First, we deﬁne Ψ 1 and we prove that Ψ 0 ≥ m Ψ 1 .

Original: Informally, Ψ 1 is the problem of ﬁnding a reward-maximizing Markovian policy π M ∈ Π M w.r.t.
Revised: Informally, Ψ 1 is the problem of ﬁnding a reward-maximizing Markovian policy π M ∈ Π M w.r.t.

Original: the entropy objective (3) encoded through a reward function in a convenient POMDP (cid:102) M R Ω .
Revised: the entropy objective (3) encoded through a reward function in a convenient POMDP (cid:102) M R Ω .

Original: We can build (cid:102) M R Ω from the CMP M similarly as the extended MDP (cid:102) M RT (see Section 2 and the proof of Lemma 4.2 for details), except that the agent only access the observation space (cid:101) Ω instead of the extended state space (cid:101) S .
Revised: We can build (cid:102) M R Ω from the CMP M similarly as the extended MDP (cid:102) M RT (see Section 2 and the proof of Lemma 4.2 for details), except that the agent only access the observation space (cid:101) Ω instead of the extended state space (cid:101) S .

Original: In particular, we deﬁne (cid:101) Ω = S
Revised: In particular, we deﬁne (cid:101) Ω = S

Original: (note that S is the state space of the original CMP M ), (cid:101) O
Revised: (note that S is the state space of the original CMP M ), (cid:101) O

Original: ( (cid:101)
Revised: ( (cid:101)

Original: o | (cid:101) s ) =
Revised: o | (cid:101) s ) =

Original: (cid:101) s − 1 , and the reward function (cid:101) R assigns value 0 to all states (cid:101) s
Revised: (cid:101) s − 1 , and the reward function (cid:101) R assigns value 0 to all states (cid:101) s

Original: ∈ (cid:101) S such that | (cid:101) s | (cid:54)
Revised: ∈ (cid:101) S such that | (cid:101) s | (cid:54)

Original: =
Revised: =

Original: T , otherwise (if | (cid:101) s | = T ) the reward corresponds to the entropy value of the state visitation frequences induced by the trajectory codiﬁed through (cid:101) s .
Revised: T , otherwise (if | (cid:101) s | = T ) the reward corresponds to the entropy value of the state visitation frequences induced by the trajectory codiﬁed through (cid:101) s .

Original: Then, the reduction Ψ 0 ≥ m Ψ 1 works as follows.
Revised: Then, the reduction Ψ 0 ≥ m Ψ 1 works as follows.

Original: We denote as I Ψ
Revised: We denote as I Ψ

Original: i the set of possible instances of problem Ψ i .
Revised: i the set of possible instances of problem Ψ i .

Original: We show that Ψ 0 is harder than Ψ 1 by deﬁning the polynomial-time functions ψ and φ such that any instance of Ψ 1 can be rewritten through ψ as an instance of Ψ 0 , and a solution π ∗ NM ∈ Π NM for Ψ 0 can be converted through φ into a solution π ∗ M ∈ Π M for the original instance of Ψ 1 .
Revised: We show that Ψ 0 is harder than Ψ 1 by deﬁning the polynomial-time functions ψ and φ such that any instance of Ψ 1 can be rewritten through ψ as an instance of Ψ 0 , and a solution π ∗ NM ∈ Π NM for Ψ 0 can be converted through φ into a solution π ∗ M ∈ Π M for the original instance of Ψ 1 .

Original: [Equation]
Revised: [Equation]

Original: The function ψ sets S = (cid:101) Ω and derives the transition model of M from the one of (cid:102) M R Ω , while φ converts the optimal solution of Ψ 0 by computing
Revised: The function ψ sets S = (cid:101) Ω and derives the transition model of M from the one of (cid:102) M R Ω , while φ converts the optimal solution of Ψ 0 by computing

Original: [Equation]
Revised: [Equation]

Original:  where H o stands for the set of histories h ∈ H [ T ] ending in the observation o ∈ Ω .
Revised: 6) where H o stands for the set of histories h ∈ H [ T ] ending in the observation o ∈ Ω .

Original: Thus, we have that Ψ 0 ≥ m Ψ 1 holds.
Revised: Thus, we have that Ψ 0 ≥ m Ψ 1 holds.

Original: We now deﬁne Ψ 2 as the policy existence problem w.r.t.
Revised: We now deﬁne Ψ 2 as the policy existence problem w.r.t.

Original: the problem statement of Ψ 1 .
Revised: the problem statement of Ψ 1 .

Original: Hence, Ψ 2 is the problem of determining whether the value of a reward-maximizing Markovian policy π ∗ M ∈ arg max π ∈ Π M J (cid:102) M R Ω ( π ) is greater than 0.
Revised: Hence, Ψ 2 is the problem of determining whether the value of a reward-maximizing Markovian policy π ∗ M ∈ arg max π ∈ Π M J (cid:102) M R Ω ( π ) is greater than 0.

Original: Since computing an optimal policy in POMDPs is in general harder than the relative policy existence problem (Lusena et al., 2001, Section 3), we have that Ψ 1 ≥ p Ψ 2 .
Revised: Since computing an optimal policy in POMDPs is in general harder than the relative policy existence problem (Lusena et al., 2001, Section 3), we have that Ψ 1 ≥ p Ψ 2 .

Original: For the last reduction, i.e., Ψ 2 ≥ p 3SAT, we extend the proof of Theorem 4.13 in (Mundhenk et al., 2000), which states that the policy existence problem for POMDPs is NP -complete.
Revised: For the last reduction, i.e., Ψ 2 ≥ p 3SAT, we extend the proof of Theorem 4.13 in (Mundhenk et al., 2000), which states that the policy existence problem for POMDPs is NP -complete.

Original: In particular, we show that this holds within the restricted class of POMDPs deﬁned in Ψ 1 .
Revised: In particular, we show that this holds within the restricted class of POMDPs deﬁned in Ψ 1 .

Original: The restrictions on the POMDPs class are the following:
Revised: The restrictions on the POMDPs class are the following:

Original: The reward function R ( s ) ≥ 0 only in the subset of states reachable in T steps, otherwise R ( s )
Revised: The reward function R ( s ) ≥ 0 only in the subset of states reachable in T steps, otherwise R ( s )

Original: =2.
Revised: =2.

Original: | (cid:101) S| = (cid:101) S =
Revised: | (cid:101) S| = (cid:101) S =

Original: | (cid:101) Ω | T
Revised: | (cid:101) Ω | T

Original: Both limitations can be overcome in the following ways:
Revised: Both limitations can be overcome in the following ways:

Original: It sufﬁces to add states with deterministic transitions so that T = m · n can be deﬁned a priori, where T is the number of steps needed to reach the state with positive reward through every possible path.
Revised: It sufﬁces to add states with deterministic transitions so that T = m · n can be deﬁned a priori, where T is the number of steps needed to reach the state with positive reward through every possible path.

Original: Here m is the number of clauses, and n is the number of variables in the 3SAT instance, as deﬁned in (Mundhenk et al., 2000).
Revised: Here m is the number of clauses, and n is the number of variables in the 3SAT instance, as deﬁned in (Mundhenk et al., 2000).

Original: The POMDPs class deﬁned by Ψ 1 is such that (cid:101) S =
Revised: The POMDPs class deﬁned by Ψ 1 is such that (cid:101) S =

Original: | (cid:101) Ω | T .
Revised: | (cid:101) Ω | T .

Original: Noticing that the set of observations corresponds with the set of variables and that from the previous point T = m · n , we have that | (cid:101) Ω | T = n m · n , while the POMDPs class used by the proof hereinabove has (cid:101) S = m · n 2 .
Revised: Noticing that the set of observations corresponds with the set of variables and that from the previous point T = m · n , we have that | (cid:101) Ω | T = n m · n , while the POMDPs class used by the proof hereinabove has (cid:101) S = m · n 2 .

Original: Notice that n ≥ 2 and m ≥ 1 implies that n m · n ≥ m · n 2 .
Revised: Notice that n ≥ 2 and m ≥ 1 implies that n m · n ≥ m · n 2 .

Original: Moreover, notice that every instance of 3SAT has m ≥ 1 and n ≥ 3 .
Revised: Moreover, notice that every instance of 3SAT has m ≥ 1 and n ≥ 3 .

Original: Hence, to extend the proof to the POMDPs class deﬁned by Ψ 1 it sufﬁces to add a set of states (cid:101)
Revised: Hence, to extend the proof to the POMDPs class deﬁned by Ψ 1 it sufﬁces to add a set of states (cid:101)

Original: S p s
Revised: S p s.t.

Original: t. R ( s ) = 0 ∀ s{+
+}
Revised: R ( s ) = {+
+}

Original:  ∈ (cid:101) S p .
Revised: ∀ s ∈ (cid:101) S p .

Original: Since the chain Ψ 0 ≥ m Ψ 1 ≥ p Ψ 2 ≥ p 3SAT holds, we have that Ψ 0 ≥ p 3SAT.
Revised: Since the chain Ψ 0 ≥ m Ψ 1 ≥ p Ψ 2 ≥ p 3SAT holds, we have that Ψ 0 ≥ p 3SAT.

Original: Moreover, since 3SAT ∈ NP -complete and Ψ 0 / ∈ NP (thanks to Lemma 5.2), we conclude that Ψ 0 ∈ NP -hard.
Revised: Moreover, since 3SAT ∈ NP -complete and Ψ 0 / ∈ NP (thanks to Lemma 5.2), we conclude that Ψ 0 ∈ NP -hard.

Original: C. Non-Stationary Policies and State-Action Distributions
Revised: C. Non-Stationary Policies and State-Action Distributions

Original: Throughout the paper, we mostly considered the class of non-Markovian policies against the class of Markovian policies.
Revised: Throughout the paper, we mostly considered the class of non-Markovian policies against the class of Markovian policies.

Original: However, one could wonder how non-stationary policies fare in dealing with the ﬁnite-sample MSE objective (3).
Revised: However, one could wonder how non-stationary policies fare in dealing with the ﬁnite-sample MSE objective (3).

Original: Here we would like to report some informal results on this point.
Revised: Here we would like to report some informal results on this point.

Original: On the one hand, we can prove that there exists a deterministic non-stationary policy π NS ∈ Π DNS such that π NS ∈ arg max π ∈ Π NS E ( π ) through a standard backward-induction argument.
Revised: On the one hand, we can prove that there exists a deterministic non-stationary policy π NS ∈ Π DNS such that π NS ∈ arg max π ∈ Π NS E ( π ) through a standard backward-induction argument.

Original: Nonetheless, differently from the result in Lemma 4.2 for non-Markovian policies, π NS is not guaranteed to suffer zero regret.
Revised: Nonetheless, differently from the result in Lemma 4.2 for non-Markovian policies, π NS is not guaranteed to suffer zero regret.

Original: Especially, it might happen that π NS ( s t ) (cid:54) = a NM as the non-stationary policy is uncertain about the past.
Revised: Especially, it might happen that π NS ( s t ) (cid:54) = a NM as the non-stationary policy is uncertain about the past.

Original: One should account for the probability of this event to extend the regret bounds of Theorem 4.4 to non-stationary policies.
Revised: One should account for the probability of this event to extend the regret bounds of Theorem 4.4 to non-stationary policies.

Original: Whereas π NS is not zero-regret in general, it can suffer a lower regret than an optimal Markovian policy.
Revised: Whereas π NS is not zero-regret in general, it can suffer a lower regret than an optimal Markovian policy.

Original: Formally establishing this regret gap might be an interesting direction for future works.
Revised: Formally establishing this regret gap might be an interesting direction for future works.

Original: To assess the importance of non-Markovianity in MSE exploration, we adopted the most common state distribution formulation.
Revised: To assess the importance of non-Markovianity in MSE exploration, we adopted the most common state distribution formulation.

Original: As we mentioned in Appendix A, other works in the MSE literature considered the entropy of the state-action distribution in the objective function.
Revised: As we mentioned in Appendix A, other works in the MSE literature considered the entropy of the state-action distribution in the objective function.

Original: Analogously, we can recast the ﬁnite-sample objective as E h ∼ p πT
Revised: Analogously, we can recast the ﬁnite-sample objective as E h ∼ p πT

Original: [ Entropy ( d h ( · , · ))] and replicate most of the results that we reported in the paper.
Revised: [ Entropy ( d h ( · , · ))] and replicate most of the results that we reported in the paper.

Original: In this alternative formulation, the class of non-Markovian policies would still preserve an edge over the classes of non-stationary and Markovian policies.
Revised: In this alternative formulation, the class of non-Markovian policies would still preserve an edge over the classes of non-stationary and Markovian policies.

Original: Accounting for the entropy of the state-action distribution might be crucial in the settings where the exploration over the action space cannot be overlooked (e.g., a single-trial setting).
Revised: Accounting for the entropy of the state-action distribution might be crucial in the settings where the exploration over the action space cannot be overlooked (e.g., a single-trial setting).