Original: P ARTIALLY O BSERVABLE RL WITH B-S TABILITY : U NIFIED S TRUCTURAL C ONDITION AND S HARP S AMPLE -E FFICIENT A LGORITHMS
Revised: P ARTIALLY O BSERVABLE RL WITH B-S TABILITY : U NIFIED S TRUCTURAL C ONDITION AND S HARP S AMPLE -E FFICIENT A LGORITHMS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Partial Observability—where agents can only observe partial information about the true underlying state of the system—is ubiquitous in real-world applications of Reinforcement Learning (RL).
Revised: Partial Observability—where agents can only observe partial information about the true underlying state of the system—is ubiquitous in real-world applications of Reinforcement Learning (RL).

Original: Theoretically, learning a near-optimal policy under partial observability is known to be hard in the worst case due to an exponential sample complexity lower bound.
Revised: Theoretically, learning a near-optimal policy under partial observability is known to be hard in the worst case due to an exponential sample complexity lower bound.

Original: Recent work has identified several tractable subclasses that are learnable with polynomial samples, such as Partially Observable Markov Decision Processes (POMDPs) with certain revealing or decodability conditions.
Revised: Recent work has identified several tractable subclasses that are learnable with polynomial samples, such as Partially Observable Markov Decision Processes (POMDPs) with certain revealing or decodability conditions.

Original: However, this line of research is still in its infancy, where (1) unified structural conditions enabling sample-efficient learning are lacking; (2) existing sample complexities for known tractable subclasses are far from sharp; and (3) fewer sample-efficient algorithms are available than in fully observable RL.
Revised: However, this line of research is still in its infancy, where (1) unified structural conditions enabling sample-efficient learning are lacking; (2) existing sample complexities for known tractable subclasses are far from sharp; and (3) fewer sample-efficient algorithms are available than in fully observable RL.

Original: This paper advances all three aspects above for Partially Observable RL in the general setting of Predictive State Representations (PSRs).
Revised: This paper advances all three aspects above for Partially Observable RL in the general setting of Predictive State Representations (PSRs).

Original: First, we propose a natural and unified structural condition for PSRs called B-stability .
Revised: First, we propose a natural and unified structural condition for PSRs called B-stability .

Original: B-stable PSRs encompasses the vast majority of known tractable subclasses such as weakly revealing POMDPs, low-rank future-sufficient POMDPs, decodable POMDPs, and regular PSRs.
Revised: B-stable PSRs encompasses the vast majority of known tractable subclasses such as weakly revealing POMDPs, low-rank future-sufficient POMDPs, decodable POMDPs, and regular PSRs.

Original: Next, we show that any B-stable PSR can be learned with polynomial samples in relevant problem parameters.
Revised: Next, we show that any B-stable PSR can be learned with polynomial samples in relevant problem parameters.

Original: When instantiated in the aforementioned subclasses, our sample complexities improve substantially over the current best ones.
Revised: When instantiated in the aforementioned subclasses, our sample complexities improve substantially over the current best ones.

Original: Finally, our results are achieved by three algorithms simultaneously: Optimistic Maximum Likelihood Estimation, Estimation-to-Decisions, and Model-Based Optimistic Posterior Sampling.
Revised: Finally, our results are achieved by three algorithms simultaneously: Optimistic Maximum Likelihood Estimation, Estimation-to-Decisions, and Model-Based Optimistic Posterior Sampling.

Original: The latter two algorithms are new for sample-efficient learning of POMDPs/PSRs.
Revised: The latter two algorithms are new for sample -efficient learning of POMDPs/PSRs.

Revised: We additionally design a variant of the Estimation-to-Decisions algorithm to perform sample-efficient all-policy model estimation for B-stable PSRs, which also yields guarantees for reward-free learning as an implication.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Partially Observable Reinforcement Learning (RL)—where agents can only observe partial information about the true underlying state of the system—is ubiquitous in real-world applications of
Revised: Partially Observable Reinforcement Learning (RL)—where agents can only observe partial information about the true underlying state of the system—is ubiquitous in real-world applications of

Original: RL such as robotics (Akkaya et al., 2019), strategic games (Brown & Sandholm, 2018; Vinyals et al., 2019; Berner et al., 2019), economic simulation (Zheng et al., 2020), and so on.
Revised: RL such as robotics (Akkaya et al., 2019), strategic games (Brown & Sandholm, 2018; Vinyals et al., 2019; Berner et al., 2019), economic simulation (Zheng et al., 2020), and so on.

Original: Partially observable RL defies standard efficient approaches for learning and planning in the fully observable case (e.g. those based on dynamical programming) due to the non-Markovian nature of the observations (Jaakkola et al., 1994), and has been a hard challenge for RL research.
Revised: Partially observable RL defies standard efficient approaches for learning and planning in the fully observable case (e.g. those based on dynamical programming) due to the non-Markovian nature of the observations (Jaakkola et al., 1994), and has been a hard challenge for RL research.

Original: Theoretically, it is well-established that learning in partial observable RL is statistically hard in the worst case—In the standard setting of Partially Observable Markov Decision Processes (POMDPs), learning a near-optimal policy has an exponential sample complexity lower bound in the horizon length (Mossel & Roch, 2005; Krishnamurthy et al., 2016), which in stark contrast to fully observable MDPs where polynomial sample complexity is possible (Kearns & Singh, 2002; Jaksch et al.,  2010; Azar et al., 2017).
Revised: Theoretically, it is well-established that learning in partial observable RL is statistically hard in the worst case—In the standard setting of Partially Observable Markov Decision Processes (POMDPs), learning a near-optimal policy has an exponential sample complexity lower bound in the horizon length (Mossel & Roch, 2005; Krishnamurthy et al., 2016), which in stark contrast to fully observable MDPs where polynomial sample complexity is possible (Kearns & Singh, 2002; Jaksch et al., ˚ [Table] 2010; Azar et al., 2017).

Original: A later line of work identifies various additional structural conditions or alternative learning goals that enable sample-efficient learning, such as reactiveness (Jiang et al., 2017), revealing conditions (Jin et al., 2020a;
Revised: A later line of work identifies various additional structural conditions or alternative learning goals that enable sample-efficient learning, such as reactiveness (Jiang et al., 2017), revealing conditions (Jin et al., 2020a;

Original: Liu et al., 2022b;
Revised: Liu et al., 2022c;

Original: Cai et al., 2022; Wang et al., 2022), decodability (Du et al., 2019; Efroni et al., 2022), and learning memoryless or short-memory policies (Azizzadenesheli et al., 2018; Uehara et al., 2022b).[Table] Despite these progresses [Table]
Revised: Cai et al., 2022; Wang et al., 2022), decodability (Du et al., 2019; Efroni et al., 2022), and learning memoryless or short-memory policies (Azizzadenesheli et al., 2018; Uehara et al., 2022b).

Original:  research on sample-efficient partially observable RL is still at an early stage [Table] with several important questions remaining open.First
Revised: Despite these progresses, research on sample-efficient partially observable RL is still at an early stage, with several important questions remaining open.

Original: [Table] to a large extent [Table] existing tractable structural conditions are mostly identified and analyzed in a case-by-case manner and lack a more unified understanding.
Revised: First, to a large extent, existing tractable structural conditions are mostly identified and analyzed in a case-by-case manner and lack a more unified understanding.

Original: This question has just started to be tackled in the very recent work of Zhan et al.
Revised: This question has just started to be tackled in the very recent work of Zhan et al.

Original: (2022), who show that sample-efficient learning is possible in the more general setting of
Revised: (2022), who show that sample-efficient learning is possible in the more general setting of

Original: Predictive State Representations (PSRs) (Littman & Sutton, 2001)—which include POMDPs as a special case—with a certain regularity condition.
Revised: Predictive State Representations (PSRs) (Littman & Sutton, 2001)—which include POMDPs as a special case—with a certain regularity condition.

Original: However, their regularity condition is defined in terms of additional quantities (such as “core matrices”) not directly encoded in the definition of
Revised: However, their regularity condition is defined in terms of additional quantities (such as “core matrices”) not directly encoded in the definition of

Original: PSRs, which makes it unnatural in many known examples and unable to subsume important tractable problems such as decodable POMDPs.
Revised: PSRs, which makes it unnatural in many known examples and unable to subsume important tractable problems such as decodable POMDPs.

Original: Second, even in known sample-efficient problems such as revealing POMDPs (Jin et al., 2020c; Liu et al., 2022a), existing sample complexities involve large polynomial factors of relevant problem parameters that are likely far from sharp.
Revised: Second, even in known sample-efficient problems such as revealing POMDPs (Jin et al., 2020c; Liu et al., 2022a), existing sample complexities involve large polynomial factors of relevant problem parameters that are likely far from sharp.

Original: Third, relatively few principles are known for designing sample-efficient algorithms in POMDPs/PSRs, such as spectral or tensor-based approaches (Hsu et al., 2012;
Revised: Third, relatively few principles are known for designing sample-efficient algorithms in POMDPs/PSRs, such as spectral or tensor-based approaches (Hsu et al., 2012;

Original: Azizzadenesheli et al., 2016; Jin et al., 2020c), maximum likelihood or density estimation (Liu et al., 2022a; Wang et al., 2022; Zhan et al., 2022), or learning short-memory policies (Efroni et al., 2022; Uehara et al., 2022b).
Revised: Azizzadenesheli et al., 2016; Jin et al., 2020c), maximum likelihood or density estimation (Liu et al., 2022a; Wang et al., 2022; Zhan et al., 2022), or learning short-memory policies (Efroni et al., 2022; Uehara et al., 2022b).

Original: This contrasts with fully observable RL where the space of sample-efficient algorithms is much more diverse (Agarwal et al., 2019).
Revised: This contrasts with fully observable RL where the space of sample-efficient algorithms is much more diverse (Agarwal et al., 2019).

Original: It is an important question whether we can expand the space of algorithms for partially observable RL.
Revised: It is an important question whether we can expand the space of algorithms for partially observable RL.

Original: This paper advances all three aspects above for partially observable RL.
Revised: This paper advances all three aspects above for partially observable RL.

Original: We define B-stablility , a natural and general structural condition for PSRs, and design sharp algorithms for learning any
Revised: We define B-stablility , a natural and general structural condition for PSRs and design sharp algorithms for learning any

Original: B-stable PSR sample-efficiently.
Revised: B-stable PSR sample-efficiently.

Original: Our contributions can be summarized as follows.
Revised: Our contributions can be summarized as follows.

Original: • We identify a new structural condition for PSRs termed B-stability , which simply requires its B-representation (or observable operators)  be bounded in a suitable operator norm (Section 3.1).
Revised: • We identify a new structural condition for PSRs termed B-stability , which simply requires its Brepresentation (or observable operators) to be bounded in a suitable operator norm (Section 3.1).

Original: B-stable PSRs subsume most known tractable subclasses such as revealing POMDPs, decodable POMDPs, low-rank future-sufficient POMDPs, and regular PSRs (Section 3.2).
Revised: B-stable PSRs subsume most known tractable subclasses such as revealing POMDPs, decodable POMDPs, low-rank future-sufficient POMDPs, and regular PSRs (Section 3.2).

Original: • We show that B-stable PSRs can be learned sample-efficiently by three algorithms simultaneously with sharp sample complexities (Section 4): Optimistic Maximum Likelihood Estimation (OMLE), Explorative Estimation-to-Decisions (E XPLORATIVE E2D), and Model-based Optimistic Posterior Sampling (MOPS).
Revised: • We show that B-stable PSRs can be learned sample-efficiently by three algorithms simultaneously with sharp sample complexities (Section 4): Optimistic Maximum Likelihood Estimation (OMLE), Explorative Estimation-to-Decisions (E XPLORATIVE E2D), and Model-based Optimistic Posterior Sampling (MOPS).

Original: To our best knowledge, the latter two algorithms are first shown to be sample-efficient in partially observable RL.
Revised: To our best knowledge, the latter two algorithms are first shown to be sample-efficient in partially observable RL.

Original: • Our sample complexities improve substantially over the current best when instantiated in both regular PSRs (Section 4.1) and known tractable subclasses of POMDPs (Section 5).
Revised: • Our sample complexities improve substantially over the current best when instantiated in both regular PSRs (Section 4.1) and known tractable subclasses of POMDPs (Section 5).

Original: For example, for m -step α rev -revealing POMDPs with S latent states, our algorithms find an ε nearoptimal policy within r O ` S 2 A
Revised: For example, for m -step α rev -revealing POMDPs with S latent states, our algorithms find an ε nearoptimal policy within r O ` S 2 A

Original: m log N {p α 2 rev ε 2 q ˘ episodes of play (with S 2 { α 2 rev replaced by S Λ 2 B if measured in B-stability), which improves significantly over the current best result of r O ` [Equation] N
Revised: m log N {p α 2 rev ε 2 q ˘ episodes of play (with S 2 { α 2 rev replaced by 1 For ν -future-sufficient POMDPs, Wang et al.

Original: [Equation]
Revised: (2022)’s

Original: p
Revised: sample complexity depends on γ , which is an additional l -step past-sufficiency parameter that they require. S Λ 2 B if measured in B-stability), which improves significantly over the current best result of r O `

Original: [Equation]
Revised: [Equation]

Original: q ˘ .

Original: A summary of such comparisons is presented in Table 1.
Revised: A summary of such comparisons is presented in Table 1.

Revised: • As a variant of the E2D algorithm, we design the A LL -P OLICY M ODEL -E STIMATION E2D

Revised: algorithm that achieves sample-efficient all-policy model estimation —and as an application, rewardfree learning—for B-stable PSRs (Section 4.2 & Appendix H.2).

Original: • Technically, our three algorithms rely on a unified sharp analysis of B-stable PSRs that involves a careful error decomposition in terms of its B-representation, along with a new generalized ℓ 2 -type Eluder argument, which may be of future interest (Appendix B).
Revised: • Technically, our three algorithms rely on a unified sharp analysis of B-stable PSRs that involves a careful error decomposition in terms of its B-representation, along with a new generalized ℓ 2 -type Eluder argument, which may be of future interest (Appendix B).

Original: Related work Our work is closely related to the long lines of work on sample-efficient learning of fully/partially observable RL (with/without function approximation), especially the lines of work on POMDPs and PSRs.
Revised: Related work Our work is closely related to the long lines of work on sample-efficient learning of fully/partially observable RL (with/without function approximation), especially the lines of work on POMDPs and PSRs.

Original: We review these related works in Appendix A due to the space limit.
Revised: We review these related works in Appendix A due to the space limit

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Sequential decision processes with observations An episodic sequential decision process is specified by a tuple␣
Revised: Sequential decision processes with observations An episodic sequential decision process is specified by a tuple

Original: H, O , A , P , t r h u
Revised: [Equation]

Original: Hh “( , where H P Z ě 1 is the horizon length; O is the observation space with | O | “ O ; A is the action space with | A | “ A ; P specifies the transition dynamics, such that the initial observation follows o 1 „ P 

Revised: ( , where H P Z ě 1 is the horizon length; O is the observation space with | O | “ O ; A is the action space with | A | “ A ; P specifies the transition dynamics, such that the initial observation follows o 1 „ P

Original: p¨q P ∆ p O q , and given the history τ h : “ p o 1 , a 1 , ¨ ¨ ¨ , o h , a h q up to step h , the observation follows o h ` 1 „ P p¨| τ h q ; r h : O ˆ A Ñ r 0 , 1 s is the reward function at h -th step, which we assume is a known deterministic function of p o h , a h q .
Revised: p¨q P ∆ p O q , and given the history τ h : “ p o 1 , a 1 , ¨ ¨ ¨ , o h , a h q up to step h , the observation follows o h ` 1 „ P p¨| τ h q ; r h : O ˆ A Ñ r 0 , 1 s is the reward function at h -th step, which we assume is a known deterministic function of p o h , a h q .

Original: A policy[-π “ t π h : p O ˆ A q h ´ 
-]
Revised: A policy[-π “ t π h : p O ˆ A q h ´ 
-]

Original: ˆ O Ñ ∆ p
Revised: [Equation] p

Original: A qu Hh “ 1 is a collection of H functions.
Revised: A qu Hh “ 1 is a collection of H functions.

Original: At step h P r H s , an agent running policy π observes the observation o h and takes action a h „ π h p¨|
Revised: At step h P r H s , an agent running policy π observes the observation o h and takes action a h „ π h p¨|

Original: τ h ´ 1 , o h q P ∆ p
Revised: τ h ´ 1 , o h q P ∆ p

Original: A q based on the history p
Revised: A q based on the history p

Original: τ h ´ 1 , o h q “ p o 1 , a 1 , . .
Revised: τ h ´ 1 , o h q “ p o 1 , a 1 , . .

Original: , o h ´ 1 , a h ´ 1 , o h q .
Revised: , o h ´ 1 , a h ´ 1 , o h q .

Original: The agent then receives their reward r h p o h , a h q , and the environment generates the next observation o h ` 1 „ P p¨|
Revised: The agent then receives their reward r h p o h , a h q , and the environment generates the next observation o h ` 1 „ P p¨|

Original: τ h q based on τ h “ p o 1 , a 1 , ¨ ¨ ¨ , o h , a h q .
Revised: τ h q based on τ h “ p o 1 , a 1 , ¨ ¨ ¨ , o h , a h q .

Original: The episode terminates immediately after the dummy observation o H ` 1 “ o dum is generated.
Revised: The episode terminates immediately after the dummy observation o H ` 1 “ o dum is generated.

Original: We use Π to denote the set of all deterministic policies, and identify ∆ p Π q as both the set of all policies and all distributions over deterministic policies interchangeably.
Revised: We use Π to denote the set of all deterministic policies, and identify ∆ p Π q as both the set of all policies and all distributions over deterministic policies interchangeably.

Original: For any p h, τ h q , let P p τ h
Revised: For any p h, τ h q , let P p τ h

Original: q
Revised: q

Original: “ ś h 1 ď h P p
Revised: “ ś h 1 ď h P p

Original: o h 1 | τ h 1 ´ 1 q , π p τ h q : “ ś h 1 ď h
Revised: o h 1 | τ h 1 ´ 1 q , π p τ h q : “ ś h 1 ď h

Original: π h 1 p a h 1 | τ h 1 ´
Revised: π h 1 p a h 1 | τ h 1 ´

Original: , o h 1 q , and let P π p τ h q : “ P p τ h q ˆ π p τ h q denote the probability of observing τ h (for the first h steps) when executing π .
Revised: , o h 1 q , and let P π p τ h q : “ P p τ h q ˆ π p τ h q denote the probability of observing τ h (for the first h steps) when executing π .

Original: The value of a policy π is defined as the expected cumulative reward V p π q : “ E π r ř
Revised: The value of a policy π is defined as the expected cumulative reward V p π q : “ E π r ř

Original: Hh “ 1 r h p o h , a h qs .
Revised: Hh “ 1 r h p o h , a h qs .

Original: We assume that ř Hh “ 1 r h p o h , a h q ď 1 almost surely for any policy π .
Revised: We assume that ř Hh “ 1 r h p o h , a h q ď 1 almost surely for any policy π .

Original: POMDPs
Revised: POMDPs

Original: A Partially Observable Markov Decision Process (POMDP) is a special sequential decision process whose transition dynamics are governed by latent states .
Revised: A Partially Observable Markov Decision Process (POMDP) is a special sequential decision process whose transition dynamics are governed by latent states .

Original: An episodic POMDP is specified by a tuple t H, S , O , A , t T
Revised: An episodic POMDP is specified by a tuple t H, S , O , A , t T

Original: h u
Revised: h u

Original: Hh “ 1 , t O h u Hh “ 1 , t r h u
Revised: Hh “ 1 , t O h u Hh “ 1 , t r h u

Original: Hh “ 1 , µ 1 u , where S is the latent state space with | S | “ S ,
Revised: Hh “ 1 , µ 1 u , where S is the latent state space with | S | “ S ,

Original: O h p¨|¨q : S Ñ ∆ p
Revised: O h p¨|¨q : S Ñ ∆ p

Original: O q is the emission dynamics at step h (which we identify as an emission matrix O h P R O ˆ S ), T h p¨|¨ , ¨q : S ˆ A Ñ  S  is the transition dynamics over the latent states (which we identify as transition matrices T h p¨|¨ , a q P R S ˆ S for each a P A ), and µ 1 P ∆ p S q specifies the distribution of initial state.
Revised: O q is the emission dynamics at step h (which we identify as an emission matrix O h P R O ˆ S ), T h p¨|¨ , ¨q : S ˆ A Ñ ∆ p S q is the transition dynamics over the latent states (which we identify as transition matrices T h p¨|¨ , a q P R S ˆ S for each a P A ), and µ 1 P ∆ p S q specifies the distribution of initial state.

Original: At each step h , given latent state s h (which the agent cannot observe), the system emits observation o h „
Revised: At each step h , given latent state s h (which the agent cannot observe), the system emits observation o h „

Original: O h p¨| s h q , receives action a h P A from the agent, emits the reward r h p o h , a h q , and then transits to the next latent state s h ` 1 „ T h
Revised: O h p¨| s h q , receives action a h P A from the agent, emits the reward r h p o h , a h q , and then transits to the next latent state s h ` 1 „ T h

Original: p¨|
Revised: p¨|

Original: s
Revised: s

Original: h , a h q in a Markov fashion.
Revised: h , a h q in a Markov fashion.

Original: Note that(with known rewards) a POMDP can be fully described by the parameter θ : “ p T , O , µ 1 q .
Revised: Note that a POMDP can be fully described by the parameter θ : “ p T , O , µ 1 q .

Original: 2. P REDICTIVE S TATE R EPRESENTATIONS
Revised: 2. P REDICTIVE S TATE R EPRESENTATIONS

Original: We consider Predictive State Representations (PSRs) (Littman & Sutton, 2001), a broader class of sequential decision processes that generalize POMDPs by removing the explicit assumption of latent states, but still requiring the system dynamics to be described succinctly by a core test set.
Revised: We consider Predictive State Representations (PSRs) (Littman & Sutton, 2001), a broader class of sequential decision processes that generalize POMDPs by removing the explicit assumption of latent states, but still requiring the system dynamics to be described succinctly by a core test set.

Original: PSR, core test sets, and predictive states A test t is a sequence of future observations and actions (i.e. t P T : “ Ť W P Z ě 1 O W ˆ A W ´ 1 ).
Revised: PSR, core test sets, and predictive states A test t is a sequence of future observations and actions (i.e. t P T : “ Ť W P Z ě 1 O W ˆ A W ´ 1 ).

Original: For some test t h “ p o h : h ` W ´ 1 , a h : h ` W ´ 2 q with length
Revised: For some test t h “ p o h : h ` W ´ 1 , a h : h ` W ´ 2 q with length

Original: W ě 1 , we define the probability of test t h being successful conditioned on (reachable) history
Revised: W ě 1 , we define the probability of test t h being successful conditioned on (reachable) history

Original: τ h ´
Revised: τ h ´

Original: as P p t h | τ h ´ 1 q : “ P p
Revised: as P p t h | τ h ´ 1 q : “ P p

Original: o h : h ` W ´ 1 |
Revised: o h : h ` W ´ 1 |

Original: τ h ´ 1 ; do p a h : h ` W ´ 2 qq , i.e., the probability of observing o h : h ` W ´ 1 if the agent deterministically executes actions a h : h ` W ´ 2 , conditioned on history τ h ´ 1 .
Revised: τ h ´ 1 ; do p a h : h ` W ´ 2 qq , i.e., the probability of observing o h : h ` W ´ 1 if the agent deterministically executes actions a h : h ` W ´ 2 , conditioned on history τ h ´ 1 .

Original: We follow the convention that, if P π p τ h ´ 1 q “ 0 for any π, then P p t | τ h ´ 1 q “ 0 .
Revised: We follow the convention that, if P π p τ h ´ 1 q “ 0 for any π

Original: Definition 1 (PSR, core test sets, and predictive states) .
Revised: Definition 1 (PSR, core test sets, and predictive states) .

Original: For any h P r H s , we say a set U h Ă T is a core test set at step h if the following holds: For any W P Z ě 1 , any possible future (i.e., test) t h “ p o h : h ` W ´ 1 , a h : h ` W ´ 2 q P
Revised: For any h P r H s , we say a set U h Ă T is a core test set at step h if the following holds: For any W P Z ě 1 , any possible future (i.e., test) t h “ p o h : h ` W ´ 1 , a h : h ` W ´ 2 q P

Original: O W ˆ A W ´ 1 , there exists a vector b t h ,h P R U h
Revised: O W ˆ A W ´ 1 , there exists a vector b t h ,h P R U h

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: We refer to the vector q p τ h ´ 
Revised: We refer to the vector q p τ h ´ 1 q : “ r P p t | τ h ´ 1 qs t P U h

Revised: as the predictive state at step h (with convention

Revised: q p τ h ´ 1 q “ 0 if τ h ´ 1 is not reachable), and q 0 : “ r P p t qs t P U 1 as the initial predictive state.

Revised: A (linear)

Revised: PSR is a sequential decision process equipped with a core test set t

Original:  The predictive state q p τ h ´ 1 q P R U h in a PSR acts like a “latent state” that governs the transition P p¨| τ h ´ 1 q through the linear structure (1).
Revised: U h u h Pr H s . The predictive state q p τ h ´ 1 q P R U h in a PSR acts like a “latent state” that governs the transition P p¨| τ h ´ 1 q through the linear structure (1).

Original: We define U A,h : “ t a : p o , a q P U h for some o P Ť
Revised: We define U A,h : “ t a : p o , a q P U h for some o P Ť

Original: W P N ` O W u as the set of action sequences (possibly including an empty sequence) in U h , with
Revised: W P N ` O W u as the set of action sequences (possibly including an empty sequence) in U h , with

Original: U A : “ max h Pr H s | U A,h | .
Revised: U A : “ max h Pr H s | U A,h | .

Original: Further define U H ` 1 : “ t o dum u for notational simplicity.
Revised: Further define U H ` 1 : “ t o dum u for notational simplicity.

Original: Throughout the paper, we assume the core test sets p U
Revised: Throughout the paper, we assume the core test sets p U

Original: h
Revised: h

Original: q h
Revised: q h

Original: Pr H s are known and the same within the PSR model class.
Revised: Pr H s are known and the same within the PSR model class.

Original: B-representation We define the B-representation of a PSR, a standard notion for PSRs (also known as the observable operators (Jaeger, 2000)).
Revised: B-representation We define the B-representation of a PSR, a standard notion for PSRs (also known as the observable operators (Jaeger, 2000)).

Original: Definition 2 (B-representation) .
Revised: Definition 2 (B-representation) .

Original: A B-representation of a PSR with core test set p U
Revised: A B-representation of a PSR with core test set p U

Original: h
Revised: h

Original: q h
Revised: q h

Original: Pr H s is a set of matrices 2 tp B h p o h , a h q P R U
Revised: Pr H s is a set of matrices 2 tp B h p o h , a h q P R U

Original: h
Revised: h

Original: ` 1 ˆ U h q h,o h ,a h , q 0 P R U 1 u such that for any 0 ď h ď H , policy π , history τ h “ p o 1: h , a 1: h q P T h , and core test t h ` 1 “ p o h ` 1: h ` W , a h ` 1: h ` W ´ 1 q P
Revised: ` 1 ˆ U h q h,o h ,a h , q 0 P R U 1 u such that for any 0 ď h ď H , policy π , history τ h “ p o 1: h , a 1: h q P T h , and core test t h ` 1 “ p o h ` 1: h ` W , a h ` 1: h ` W ´ 1 q P

Original: U
Revised: U

Original: h
Revised: h

Original: ` 1 , the quantity P p τ h , t h ` 1 q , i.e. the probability of observing o 1: h ` W upon taking actions a 1: h ` W ´ 1 , admits the decomposition
Revised: ` 1 , the quantity P p τ h , t h ` 1 q , i.e. the probability of observing o 1: h ` W upon taking actions a 1: h ` W ´ 1 , admits the decomposition

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where e t h ` 1 P R U h
Revised: where e t h ` 1 P R U h

Original: ` 1 is the indicator vector of t h ` 1 P U h ` 1 , and
Revised: ` 1 is the indicator vector of t h ` 1 P U h ` 1 , and

Original: [Equation]
Revised: [Equation]

Original: It is a standard result (see e.g. Thon & Jaeger (2015)) that any PSR admits a B-representation, and the converse also holds—any sequential decision process admitting a B-representation on test sets p U
Revised: It is a standard result (see e.g. Thon & Jaeger (2015)) that any PSR admits a B-representation, and the converse also holds—any sequential decision process admitting a B-representation on test sets p U

Original: h
Revised: h

Original: q h
Revised: q h

Original: Pr H s is a PSR with core test set p U
Revised: Pr H s is a PSR with core test set p U

Original: h
Revised: h

Original: q h
Revised: q h

Original: Pr H s (Proposition D.1).
Revised: Pr H s (Proposition D.1).

Original: However, the B-representation of a given PSR may not be unique.
Revised: However, the B-representation of a given PSR may not be unique.

Original: We also remark that the B-representation is used in the structural conditions and theoretical analyses only, and will not be explicitly used in our algorithms.
Revised: We also remark that the B-representation is used in the structural conditions and theoretical analyses only, and will not be explicitly used in our algorithms.

Original: Rank An important complexity measure of a PSR is its PSR rank (henceforth also “rank”).
Revised: Rank An important complexity measure of a PSR is its PSR rank (henceforth also “rank”).

Original: Definition 3 (PSR rank) .
Revised: Definition 3 (PSR rank) .

Original: Given a PSR, its PSR rank is defined as d PSR : “ max h Pr H s rank p D h q , where D h : “ r q p τ h qs τ h P T
Revised: Given a PSR, its PSR rank is defined as d PSR : “ max h Pr H s rank p D h q , where D h : “ r q p τ h qs τ h P T

Original: h P R U
Revised: h P R U

Original: h ` 1 ˆ T h is the matrix formed by predictive states at step
Revised: h ` 1 ˆ T h is the matrix formed by predictive states at step

Original: h P r H s .
Revised: h P r H s .

Original: The PSR rank measures the inherent dimension 3 of the space of predictive state vectors, which always admits the upper bound
Revised: The PSR rank measures the inherent dimension 3 of the space of predictive state vectors, which always admits the upper bound

Original: d PSR ď max h
Revised: d PSR ď max h

Original: Pr H s | U h | , but may in addition be much smaller.
Revised: Pr H s | U h | , but may in addition be much smaller.

Original: POMDPs as low-rank PSRs As a primary example, all POMDPs are PSRs with rank at most S (Zhan et al., 2022, Lemma 2).
Revised: As a primary example, all POMDPs are PSRs with rank at most S (Zhan et al., 2022, Lemma 2).

Original: First, Definition 1 can be satisfied trivially by choosing U h “ Ť 1 ď W ď H
Revised: First, Definition 1 can be satisfied trivially by choosing U h “ Ť 1 ď W ď H

Original: ´ h ` 1 tp o h , a h , . . .
Revised: ´ h ` 1 tp o h , a h , . . .

Original: , o h ` W ´ 1 qu as the set of all possible tests, and b t h ,h “ e t h P R U h as indicator vectors.
Revised: , o h ` W ´ 1 qu as the set of all possible tests, and b t h ,h “ e t h P R U h as indicator vectors.

Original: For concrete subclasses of POMDPs, we will consider alternative choices of p U
Revised: For concrete subclasses of POMDPs, we will consider alternative choices of p U

Original: h
Revised: h

Original: q h
Revised: q h

Original: Pr H s with much smaller cardinalities than this default choice.
Revised: Pr H s with much smaller cardinalities than this default choice.

Original: Second, to compute the rank (Definition 3), note that by the latent state structure of POMDPs, we have P p t h ` 1 | τ h q “ ř s h ` 1 P p t h ` 1 | s h ` 1 q P p s h ` 1 | τ h q for any p h, τ h , t h ` 1 q .
Revised: Second, to compute the rank (Definition 3), note that by the latent state structure of POMDPs, we have P p t h ` 1 | τ h q “ ř s h ` 1 P p t h ` 1 | s h ` 1 q P p s h ` 1 | τ h q for any p h, τ h , t h ` 1 q .

Original: Therefore, the associated matrix D h “ r P p t h ` 1 | τ h qs p t
Revised: Therefore, the associated matrix D h “ r P p t h ` 1 | τ h qs p t

Original: h
Revised: h

Original: ` 1 ,τ h qP U
Revised: ` 1 ,τ h qP U

Original: h
Revised: h

Original: ` 1 ˆ T h always has the following decomposition:
Revised: ` 1 ˆ T h always has the following decomposition:

Original: [Equation]
Revised: [Equation]

Original:  which implies that d PSR “ max h Pr H s rank p D h q ď S .
Revised: , which implies that d PSR “ max h Pr H s rank p D h q ď S .

Original: Learning goal We consider the standard PAC learning setting, where we are given a model class of PSRs Θ and interact with a ground truth model θ ‹ P Θ .
Revised: Learning goal We consider the standard PAC learning setting, where we are given a model class of PSRs Θ and interact with a ground truth model θ ‹ P Θ .

Original: Note that, as we do not put further restrictions on the parametrization, this setting allows anygeneral function approximation for the model class.
Revised: Note that, as we do not put further restrictions on the parametrization, this setting allows any function approximation for the model class.

Original: For any model class Θ , we define its (optimistic) covering number N Θ p ρ q for ρ ą 0 in Definition C.4.
Revised: For any model class Θ , we define its (optimistic) covering number N Θ p ρ q for ρ ą 0 in Definition C.4.

Original: Let V θ p π q denote the value function of policy π under model θ , and π θ : “ arg max π P Π V θ p π q denote the optimal policy of model θ .
Revised: Let V θ p π q denote the value function of policy π under model θ , and π θ : “ arg max π P Π V θ p π q denote the optimal policy of model θ .

Original: The goal is to learn a policy p π that achieves small suboptimality V ‹ ´ V θ ‹ p p
Revised: The goal is to learn a policy p π that achieves small suboptimality V ‹ ´ V θ ‹ p p

Original: π q within as few episodes of play as possible, where
Revised: π q within as few episodes of play as possible, where

Original: V ‹ : “ V θ ‹ p π θ ‹ q .
Revised: V ‹ : “ V θ ‹ p π θ ‹ q .

Original: We refer to an algorithm as sample-efficient if it finds an ε -near optimal policy within poly p relevant problem parameters , 1 { ε q 4 episodes of play.
Revised: We refer to an algorithm as sample-efficient if it finds an ε -near optimal policy within poly p relevant problem parameters , 1 { ε q 4 episodes of play.

Original:  PSR S WITH B- STABILITY
Revised:  PSR S WITH B- STABILITY

Original: We begin by proposing a natural and general structural condition for PSR called B-stability (or also stability ).
Revised: We begin by proposing a natural and general structural condition for PSR called B-stability (or also stability ).

Original: We show that B-stable PSRs encompass and generalize a variety of existing tractable
Revised: We show that B-stable PSRs encompass and generalize a variety of existing tractable

Original: POMDPs and PSRs, and can be learned sample-efficiently as we show in the sequel.
Revised: POMDPs and PSRs, and can be learned sample-efficiently as we show in the sequel.

Original: 3. T HE B- STABILITY CONDITION
Revised: 3. T HE B- STABILITY CONDITION

Original: For any PSR with an associated B-representation, we define its B -operators t B H :
Revised: For any PSR with an associated B-representation, we define its B -operators t B H :

Original: h u h
Revised: h u h

Original: Pr H s as
Revised: Pr H s as

Original: [Equation]
Revised: [Equation]

Original: h [Equation] Operator B H : h maps any predictive state q “ q p τ h ´ 1 q at step h to the vector B H : h q “ p P p τ h : H | τ h
Revised: Operator B H : h maps any predictive state q “ q p τ h ´ 1 q at step h to the vector B H : h q “ p P p τ h : H | τ h

Original: ´ 1 qq τ h : H which governs the probability of transitioning to all possible futures, by properties of the B-representation (cf.
Revised: ´ 1 qq τ h : H which governs the probability of transitioning to all possible futures, by properties of the B-representation (cf.

Original: (18) & Corollary D.2).
Revised: (17) & Corollary D.2).

Original: For each h P r H s , we equip the image space of B H : h with the Π -norm : For a vector b indexed by τ h : H P
Revised: For each h P r H s , we equip the image space of B H : h with the Π -norm : For a vector b indexed by τ h : H P

Original: p
Revised: p

Original: O ˆ
Revised: O ˆ

Original: A q H ´
Revised: A q H ´

Original: h ` 1 , we define
Revised: h ` 1 , we define

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the maximization is over all policies ¯ π starting from step h (ignoring the history τ h ´ 1 ) and ¯ π p τ h : H q “ śh ď h 1 ď H ¯ π h 1 p a h 1 | o h
Revised: where the maximization is over all policies ¯ π starting from step h (ignoring the history τ h ´ 1 ) and ¯ π p τ h : H q “ ś

Original: [Equation]
Revised: [Equation]

Original: τ h :

Original: [Equation]

Original: [Equation]

Original: 1 q .
Revised: q .

Original: We further equip the domain R U h with a fused-norm } ¨ } ˚ , which is defined as the maximum of p 1 , 2 q -norm and Π 1 -norm 5 :
Revised: We further equip the domain R U h with a fused-norm } ¨ } ˚ , which is defined as the maximum of p 1 , 2 q -norm and Π 1 -norm 5 :

Original: [Equation]
Revised: [Equation]

Original: where U h : “ t t P U h : E t 1 P U h such that t is a prefix of t 1 u .
Revised: where U h : “ t t P U h : E t 1 P U h such that t is a prefix of t 1 u .

Original: We now define the B-stability condition, which simply requires the B -operators t B H :
Revised: We now define the B-stability condition, which simply requires the B -operators t B H :

Original: h u h
Revised: h u h

Original: Pr H s to have bounded operator norms from the fused-norm to the Π -norm.
Revised: Pr H s to have bounded operator norms from the fused-norm to the Π -norm.

Original: Definition 4 (B-stability) .
Revised: Definition 4 (B-stability) .

Original: A PSR is B-stable with parameter Λ B ě 1 (henceforth also Λ B -stable ) if it admits a B-representation with associated B -operators t B H :
Revised: A PSR is B-stable with parameter Λ B ě 1 (henceforth also Λ B -stable ) if it admits a B-representation with associated B -operators t B H :

Original: h u h
Revised: h u h

Original: Pr H s such that
Revised: Pr H s such that

Original: [Equation]
Revised: [Equation]

Original: When using the B-stability condition, we will often take q “ q 1 p τ h ´ 1 q ´ q 2 p τ h ´ 1 q to be the difference between two predictive states at step h .
Revised: When using the B-stability condition, we will often take q “ q 1 p τ h ´ 1 q ´ q 2 p τ h ´ 1 q to be the difference between two predictive states at step h .

Original: Intuitively, Definition 4 requires that the propagated Π -norm error } B H : h p q 1 ´ q 2 q} Π to be controlled by the original fused-norm error } q 1 ´ q 2 } ˚ .
Revised: Intuitively, Definition 4 requires that the propagated Π -norm error } B H : h p q 1 ´ q 2 q} Π to be controlled by the original fused-norm error } q 1 ´ q 2 } ˚ .

Original: The fused-norm }¨} ˚ is equivalent to the vector 1 -norm up to a | U A,h | 1 { 2 -factor (despite its seemingly involved form): We have } q } ˚ ď } q } 1 ď | U A,h | 1 { 2 } q } ˚ (Lemma D.6), and thus assuming a relaxed condition max } q } 1 “ 1 } B H : h } Π ď Λ will also enable sample-efficient learning of PSRs.
Revised: The fused-norm }¨} ˚ is equivalent to the vector 1 -norm up to a | U A,h | 1 { 2 -factor (despite its seemingly involved form): We have } q } ˚ ď } q } 1 ď | U A,h | 1 { 2 } q } ˚ (Lemma D.6), and thus assuming a relaxed condition max } q } 1 “ 1 } B H : h } Π ď Λ will also enable sample-efficient learning of PSRs.

Original: However, we consider the fused-norm in order to obtain the sharpest possible sample complexity guarantees.
Revised: However, we consider the fused-norm in order to obtain the sharpest possible sample complexity guarantees.

Original: Finally, all of our theoretical results still hold under a more relaxed (though less intuitive) weak
Revised: Finally, all of our theoretical results still hold under a more relaxed (though less intuitive) weak

Original: B-stability condition (Definition D.4), with the same sample complexity guarantees.
Revised: B-stability condition (Definition D.4), with the same sample complexity guarantees.

Original: (See also the additional discussions in Appendix D.2.)
Revised: (See also the additional discussions in Appendix D.2.)

Original: 3. R ELATION WITH KNOWN SAMPLE - EFFICIENT SUBCLASSES
Revised: 3. R ELATION WITH KNOWN SAMPLE - EFFICIENT SUBCLASSES

Original: We show that the B-stability condition encompasses many known structural conditions of PSRs and
Revised: We show that the B-stability condition encompasses many known structural conditions of PSRs and

Original: POMDPs that enable sample-efficient learning.
Revised: POMDPs that enable sample-efficient learning.

Original: Throughout, for a matrix A P R m ˆ n , we define its operator norm } A } p Ñ q : “ max } x } p ď 1 } Ax } q , and use } A } p : “ } A } p Ñ p for shorthand.
Revised: Throughout, for a matrix A P R m ˆ n , we define its operator norm } A } p Ñ q : “ max } x } p ď 1 } Ax } q , and use } A } p : “ } A } p Ñ p for shorthand.

Original: Weakly revealing POMDPs (Jin et al., 2020a; Liu et al., 2022a) is a subclass of POMDPs that assumes the current latent state can be probabilistically inferred from the next m emissions.
Revised: Weakly revealing POMDPs (Jin et al., 2020a; Liu et al., 2022a) is a subclass of POMDPs that assumes the current latent state can be probabilistically inferred from the next m emissions.

Original: Example 5 (Multi-step weakly revealing POMDPs) .
Revised: Example 5 (Multi-step weakly revealing POMDPs) .

Original: A POMDP is called m -step α rev -weakly revealing (henceforth also “ α rev -revealing”) with α rev ď 1 if max h Pr H
Revised: A POMDP is called m -step α rev -weakly revealing (henceforth also “ α rev -revealing”) with α rev ď 1 if max h Pr H

Original: ´ m ` 1 s } M : h } 2 Ñ 2 ď α ´ 1 rev , where t M h P R 
Revised: ´ m ` 1 s } M : h } 2 Ñ 2 ď α ´ 1 rev , where for h P r H ´ m ` 1 s , M h P R O m A m

Revised: ´ 1 ˆ S is the m -step emission-action matrix at step h , defined as r M

Original: [Equation]
Revised: [Equation]

Original: S u h
Revised: s p o , a q ,s : “ P p

Original: [Equation]
Revised: [Equation]

Original: m ` 1 s are the m -step emission-action matrices defined as
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: We show that any m -step α rev -weakly revealing POMDP is a Λ B -stable PSR with core test sets
Revised: We show that any m -step α rev -weakly revealing POMDP is a Λ B -stable PSR with core test sets

Original: U h
Revised: U h

Original: “ p O ˆ A q min t m ´ 1 ,H ´ h u ˆ O , and Λ B ď ?
Revised: “ p O ˆ A q min t m ´ 1 ,H ´ h u ˆ O , and Λ B ď ?

Original: Sα ´ 1 rev (Proposition D.7).
Revised: Sα ´ 1 rev (Proposition D.7).

Original: ♢
Revised: ♢

Original: When the transition matrix T h of the POMDP has a low rank structure, Wang et al.
Revised: When the transition matrix T h of the POMDP has a low rank structure, Wang et al.

Original: (2022) show that a ℓ 1 -variant of the revealing condition—the future-sufficiency condition —enables sample-efficient learning of POMDPs with large state/observation spaces ( S and O may be infinite).
Revised: (2022) show that a subspace-aware generalization of the ℓ 1 -revealing condition—the future-sufficiency condition — enables sample-efficient learning of POMDPs with large state/observation spaces ( S and O may be infinite).

Original: Such a condition is also assumed by Cai et al.
Revised: Such a condition is also assumed by Cai et al.

Original: for efficient learning of linear POMDPs.
Revised: for efficient learning of linear POMDPs.

Revised: We consider the following generalized version of the future-sufficiency condition.

Original: Example 6 (Low-rank future-sufficient POMDPs) .
Revised: Example 6 (Low-rank future-sufficient POMDPs) .

Original: We say a POMDP has transition rank d trans if for each h P r H ´ 1 s , the transition kernel of the POMDP has factorization [Equation]
Revised: We say a POMDP has transition rank d trans if for each h P r H ´ 1 s , the transition kernel of the POMDP has rank at most d trans (i.e. max h rank p T h q ď d trans ).

Original: where p Ψ h , Φ h q h Pr H s satisfy standard normalization assumptions (25).
Revised: It is clear that low-rank POMDPs with transition rank d trans has PSR rank d PSR ď d trans .

Original: A transition rank-
Revised: A transition rank-

Original: d trans (henceforth rank- d trans ) POMDP is called m -step ν
Revised: d trans (henceforth rank- d trans ) POMDP is called m -step ν

Original: -future-sufficient with ν ě 1 , if for h P r H ´ 1 s , 
Revised: -future-sufficient with ν ě 1 , if for h P r H ´ 1 s , there exists

Original: M  h Ψ h ´ 1has full row rank, and } M 6 h } 1 Ñ 1 ď ν , where M h is the m -step emission-action matrix defined in [-(7), and M 6 h : “ Ψ h ´ 1 p M h Ψ h ´ 
-]
Revised: M 6 h P R S ˆ U h such that M 6 h M h T h ´ 1 “ T h ´ 1 and } M 6 h } 1 Ñ 1 ď ν , where M h is the m -step emission-action matrix defined in [-(7), and M 6 h : “ Ψ h ´ 1 p M h Ψ h ´ 
-](7). We show that any m -step ν

Original: q : P R S ˆ U h .
Revised: -future sufficient

Original: We show that any m -step ν -weakly revealing rank- d trans POMDP is a B-stable PSR with core test sets U [Equation] “ p O ˆ A q 
Revised: rank- d trans POMDP is a B-stable PSR with core test sets U h “ p O ˆ A q min t m ´ 1 ,

Original: [Equation] m [Equation]
Revised: H ´ h

Original: 1 ,H ´ h u ˆ O , d PSR ď d trans , and Λ B ď ?U
Revised: u ˆ O , d PSR ď d trans , and Λ B ď ?

Original: A  ν (Proposition D.13).
Revised: A m ´ 1 ν (Proposition D.12).

Original: ♢
Revised: ♢

Original: Decodable POMDPs (Efroni et al., 2022), as a multi-step generalization of Block MDPs (Du et al., 2019), assumes the current latent state can be perfectly decoded from the recent m observations.
Revised: Decodable POMDPs (Efroni et al., 2022), as a multi-step generalization of Block MDPs (Du et al., 2019), assumes the current latent state can be perfectly decoded from the recent m observations.

Original: Example 7 (Multi-step decodable POMDPs) .
Revised: Example 7 (Multi-step decodable POMDPs) .

Original: A POMDP is called m -step decodable if there exists (unknown) decoders ϕ ‹ “ t ϕ ‹ h u
Revised: A POMDP is called m -step decodable if there exists (unknown) decoders ϕ ‹ “ t ϕ ‹ h u

Original: h Pr H s , such that for every reachable trajectory p s 1 , o 1 , a 1 , ¨ ¨ ¨ , s h , o
Revised: h Pr H s , such that for every reachable trajectory p s 1 , o 1 , a 1 , ¨ ¨ ¨ , s h , o

Original: h q
Revised: h q

Original: we have s h “ ϕ ‹ h
Revised: we have s h “ ϕ ‹ h

Original: p z h q , where z h “ p o m p h q , a m p h q , ¨ ¨ ¨ , o h q
Revised: p z h q , where z h “ p o m p h q , a m p h q , ¨ ¨ ¨ , o h q

Original: and m p h q “ max t h
Revised: and m p h q “ max t h

Original: ´ m ` 1 , 1 u .
Revised: ´ m ` 1 , 1 u .

Original: We show that any m -step decodable POMDP is a B-stable PSR with core test sets U h “ p O ˆ A q min t m ´ 1 ,
Revised: We show that any m -step decodable POMDP is a B-stable PSR with core test sets U h “ p O ˆ A q min t m ´ 1 ,

Original: H
Revised: H

Original: ´ h
Revised: ´ h

Original: u ˆ O and Λ B “ 1 (Proposition D.18).
Revised: u ˆ O and Λ B “ 1 (Proposition D.17).

Original: ♢
Revised: ♢

Original: Finally, Zhan et al. (2022) define the following regularity condition for general PSRs .
Revised: Finally, Zhan et al. (2022) define the following regularity condition for general PSRs .

Original: Example 8 (Regular PSRs) .
Revised: Example 8 (Regular PSRs) .

Original: A PSR is called α psr -regular if for all h P r H s there exists a core matrix K h P R U
Revised: A PSR is called α psr -regular if for all h P r H s there exists a core matrix K h P R U

Original: h
Revised: h

Original: ` 1 ˆ rank p D h q , which is a column-wise sub-matrix of D h such that rank p K h q “ rank p D h q and max h Pr H s } K : h } 1 Ñ 1 ď α ´ 1 psr .
Revised: ` 1 ˆ rank p D h q , which is a column-wise sub-matrix of D h such that rank p K h q “ rank p D h q and max h Pr H s } K : h } 1 Ñ 1 ď α ´ 1 psr .

Original: We show that any α psr -regular PSR is Λ B -stable with Λ B ď ?
Revised: We show that any α psr -regular PSR is Λ B -stable with Λ B ď ?

Original: U
Revised: U

Original: A α ´ 1 psr (Proposition D.19).
Revised: A α ´ 1 psr (Proposition D.18).

Original: ♢
Revised: ♢

Original: We emphasize that B-stability not only encompasses α psr -regularity, but is also strictly more expressive.
Revised: We emphasize that B-stability not only encompasses α psr -regularity, but is also strictly more expressive.

Original: For example, decodable POMDPs are not α psr
Revised: For example, decodable POMDPs are not α psr

Original: -regular
Revised: -regular

Original: unless with additional assumptions on
Revised: unless with additional assumptions on

Original: K : h (Zhan et al., 2022, Section 6.5), whereas they are B-stable with Λ B “ 1 (Example 7).
Revised: K : h (Zhan et al., 2022, Section 6.5), whereas they are B-stable with Λ B “ 1 (Example 7).

Original: Also, any α rev -revealing POMDP is α psr -regular with some α ´ 1 psr ă 8 , but with α ´ 1 psr potentially not polynomially bounded by α ´ 1 rev (and other problem parameters) due to the restriction of K h being a column-wise sub-matrix of D h ; By contrast it is B-stable with Λ B ď ?
Revised: Also, any α rev -revealing POMDP is α psr -regular with some α ´ 1 psr ă 8 , but with α ´ 1 psr potentially not polynomially bounded by α ´ 1 rev (and other problem parameters) due to the restriction of K h being a column-wise sub-matrix of D h ; By contrast it is B-stable with Λ B ď ?

Original: Sα ´ 1 rev (Example 5).
Revised: Sα ´ 1 rev (Example 5).

Original:  L EARNING B- STABLE PSR S
Revised:  L EARNING B- STABLE PSR S

Original: In this section, we show that B-stable PSRs can be learned sample-efficiently, achieved by three model-based algorithms simultaneously.
Revised: In this section, we show that B-stable PSRs can be learned sample-efficiently, achieved by three model-based algorithms simultaneously.

Original: We instantiate our results to POMDPs in Section 5. 
Revised: We instantiate our results to POMDPs in Section 5. do 6:

Revised: Set set

Revised: [Equation]

Revised: Output

Original: 4. O PTIMISTIC M AXIMUM L IKELIHOOD E STIMATION (OMLE)
Revised: O PTIMISTIC M AXIMUM L IKELIHOOD E STIMATION (OMLE)

Original: The OMLE algorithm is proposed by Liu et al.
Revised: The OMLE algorithm is proposed by Liu et al.

Original: (2022a)
Revised: (2022a)

Original: for learning revealing POMDPs and adapted 7 by Zhan et al.
Revised: for learning revealing POMDPs and adapted 7 by Zhan et al.

Original: for learning regular PSRs, achieving polynomial sample complexity (in relevant problem parameters) in both cases.
Revised: for learning regular PSRs, achieving polynomial sample complexity (in relevant problem parameters) in both cases.

Original: We show that OMLE works under the broader condition of B-stability, with significantly improved sample complexities.
Revised: We show that OMLE works under the broader condition of B-stability, with significantly improved sample complexities.

Original: Algorithm and theoretical guarantee
Revised: Algorithm and theoretical guarantee

Original: The OMLE algorithm (described in Algorithm 1) takes in a class of PSRs Θ , and performs two main steps in each iteration k P r K s :θ 1 do 6:
Revised: The OMLE algorithm (described in Algorithm 1) takes in a class of PSRs Θ , and performs two main steps in each iteration k P r K s :

Original: Set exploration policy π kh exp : “ π k ˝ h Unif p

Original: A q ˝ h ` 1 Unif p U A,h ` 1 q .

Original: Execute π kh, exp to collect a trajectory τ k,h , and add p π kh, exp , τ k,h q into D .

Original: Update confidence set

Original: [Equation]

Original: Output: p π out : “ Unif p ␣ π k ( k Pr K s q .

Original: (Optimism
Revised: (Optimism)

Original: ) Construct a confidence set Θ kĎ Θ ,
Revised: Construct a confidence set Θ k

Original:  which is a superlevel set of the log-likelihood of all trajectories within dataset D (Line 8)
Revised: Ď Θ , which is a superlevel set of the log-likelihood of all trajectories within dataset D (Line 8).

Original: The policy πk is then chosen as the greedy policy with respect to the most optimistic model within Θ
Revised: The policy π

Original: k
Revised: k

Original: Line
Revised: is

Original: (Data collection) Execute exploration policies p π kh, exp q 0 ď h ď H ´
Revised: then chosen

Original: where each
Revised: as the

Original: π kh exp is defined via the ˝ h notation as follows: Follow
Revised: greedy policy

Original: π k for
Revised: with respect to the

Original: the first
Revised: most optimistic model within

Original:  h 
Revised: Θ k (Line 4) (Data collection) Execute exploration policies p π kh, exp q 0 ď h ď H ´ 1 , where each π kh, exp is defined via the ˝ h notation as follows:

Original:  ´ 1 steps, take a uniform action
Revised: Follow π k for the first h ´ 1 steps, take a uniform action

Original: Unif p
Revised: Unif p

Original: A q at step h , take an action sequence sampled from Unif p U A,h ` 1 q at step h ` 1 , and behave arbitrarily afterwards (Line 6).
Revised: A q at step h , take an action sequence sampled from Unif p U A,h ` 1 q at step h ` 1 , and behave arbitrarily afterwards (Line 6).

Original: All collected trajectories are then added into D (Line 7).
Revised: All collected trajectories are then added into D (Line 7).

Original: Intuitively, the concatenation of the current policy π k with Unif p
Revised: Intuitively, the concatenation of the current policy π k with Unif p

Original: A q and Unif p U A,h ` 1 q in Stepabove is designed according to the structure of PSRs to foster exploration.
Revised: A q and Unif p U A,h ` 1 q in Stepabove is designed according to the structure of PSRs to foster exploration.

Original: Theorem 9 (Guarantee of OMLE) .
Revised: Theorem 9 (Guarantee of OMLE) .

Original: Suppose every θ P Θ is Λ B -stable
Revised: Suppose every θ P Θ is Λ B -stable

Original: (Definition 4) and the true model θ ‹ P Θ has rank d PSR ď d .
Revised: (Definition 4) and the true model θ ‹ P Θ has rank d PSR ď d .

Original: Then, choosing β “ C log p N Θ p 1 { KH q{ δ q for some absolute constant C ą 0 , with probability at least 1 ´ δ , Algorithm 1 outputs a policy p π out P ∆ p Π q such that
Revised: Then, choosing β “ C log p N Θ p 1 { KH q{ δ q for some absolute constant C ą 0 , with probability at least 1 ´ δ , Algorithm 1 outputs a policy p π out P ∆ p Π q such that

Original: V ‹ ´ V θ ‹ p p π out q ď ε , as long as the number of episodes
Revised: V ‹ ´ V θ ‹ p p π out q ď ε , as long as the number of episodes

Original: [Equation]
Revised: [Equation]

Original: where ι : “ log p 1 ` Kd Λ B R Bκ d q , with R B : “ max h t 1 , max } v } 1 “ř o,a } B h p o, a q v } 1 u and
Revised: where ι : “ log p 1 ` KdU A Λ B R B q , with R B : “ max h t 1 , max } v } 1 “

Original: [Equation]
Revised: [Equation]

Original: Theorem 9 shows that OMLE is sample-efficient for any B-stable PSRs—a broader class than in existing results for the same algorithm (Liu et al., 2022a; Zhan et al., 2022)—with much sharper sample complexities than existing work when instantiated to their settings.
Revised: Theorem 9 shows that OMLE is sample-efficient for any B-stable PSRs—a broader class than in existing results for the same algorithm (Liu et al., 2022a; Zhan et al., 2022)—with much sharper sample complexities than existing work when instantiated to their settings.

Original: Importantly, we achieve the first polynomial sample complexity that scales with Λ 2 B dependence B-stability parameter (or regularity parameters alike 8 ).
Revised: Importantly, we achieve the first polynomial sample complexity that scales with Λ 2 B dependence B-stability parameter (or regularity parameters alike 8 ).

Original: Instantiating to α psr -regular PSRs, using Λ B ď ?
Revised: Instantiating to α psr -regular PSRs, using Λ B ď ?

Original: U
Revised: U

Original: A α ´ 1 psr (Example 8), our result implies a r O p dAU 2 A log N Θ {p α 2 psr ε 2 qq sample complexity (ignoring H and ι 9 ).
Revised: A α ´ 1 psr (Example 8), our result implies a r O p dAU 2 A log N Θ {p α 2 psr ε 2 qq sample complexity (ignoring H and ι 9 ).

Original: This improves significantly over the r O p d 4 A 4 U 9 A log p N Θ O q{p α 6 psr ε 2 qq result of Zhan et al.
Revised: This improves significantly over the r O p d 4 A 4 U 9 A log p N Θ O q{p α 6 psr ε 2 qq result of Zhan et al.

Original:  G) builds upon a sharp analysis for B-stable PSRs: 1) We use a more delicate choice of norm for bounding the errors (in the
Revised: Overview of techniques The proof of Theorem 9 (deferred to Appendix G) builds upon a sharp analysis for B-stable PSRs: 1) We use a more delicate choice of norm for bounding the errors (in the

Original: B operators) yielded from performance difference arguments; 2) We develop a generalized ℓ 2 -type
Revised: B operators) yielded from performance difference arguments; 2) We develop a generalized ℓ 2 -type

Original: Eluder argument that is sharper than the ℓ 1 -Eluder argument of Liu et al.
Revised: Eluder argument that is sharper than the ℓ 1 -Eluder argument of Liu et al.

Original: (2022a)
Revised: (2022a)

Original: ; Zhan et al.
Revised: ; Zhan et al.

Original: A more detailed overview of techniques is presented in Appendix B.
Revised: A more detailed overview of techniques is presented in Appendix B.

Original: 4. E XPLORATIVE E STIMATION -T O -D ECISIONS (E XPLORATIVE E2D)
Revised: E XPLORATIVE E STIMATION -T O -D ECISIONS (E XPLORATIVE E2D)

Original: Estimation-To-Decisions (E2D) is a general model-based algorithm that is sample-efficient for any interactive decision making problem (including MDPs) with a bounded Decision-Estimation Coefficient (DEC), as established in the DEC framework by Foster et al.
Revised: Estimation-To-Decisions (E2D) is a general model-based algorithm that is sample-efficient for any interactive decision making problem (including MDPs) with a bounded Decision-Estimation Coefficient (DEC), as established in the DEC framework by Foster et al.

Original: However, the E2D algorithm has not been instantiated on POMDPs/PSRs.
Revised: However, the E2D algorithm has not been instantiated on POMDPs/PSRs.

Original: We show that B-stable PSRs admit a sharp
Revised: We show that B-stable PSRs admit a sharp

Original: DEC bound, and thus can be learned sample-efficiently by a suitable E2D algorithm.
Revised: DEC bound, and thus can be learned sample-efficiently by a suitable E2D algorithm.

Original: EDEC & E XPLORATIVE E2D
Revised: EDEC & E XPLORATIVE E2D

Original: algorithm
Revised: algorithm

Original: We consider the Explorative DEC (EDEC) proposed in the recent work of Chen et al.
Revised: We consider the Explorative DEC (EDEC) proposed in the recent work of Chen et al.

Original: (2022), which for a PSR class Θ is defined as
Revised: (2022), which for a PSR class Θ is defined as

Original: [Equation]
Revised: [Equation]

Original: where D 2H p P πθ , P π ¯ θ q : “ ř τ H p P
Revised: where D 2H p P πθ , P π ¯ θ q : “ ř τ H p P

Original: πθ p
Revised: πθ p

Original: τ H q 1 { 2 ´ P π ¯ θ p τ H q 1 { 2 q 2 denotes the squared Hellinger distance between P πθ and P π ¯ θ .
Revised: τ H q 1 { 2 ´ P π ¯ θ p τ H q 1 { 2 q 2 denotes the squared Hellinger distance between P πθ and P π ¯ θ .

Original: Intuitively, the EDEC measures the optimal trade-off on model class Θ between gaining information by an “exploration policy” π „ p exp and achieving near-optimality by an “output policy” π „ p out .
Revised: Intuitively, the EDEC measures the optimal trade-off on model class Θ between gaining information by an “exploration policy” π „ p exp and achieving near-optimality by an “output policy” π „ p out .

Original: Chen et al.
Revised: Chen et al.

Original: (2022) further design the E XPLORATIVE E2D
Revised: (2022) further design the E XPLORATIVE E2D

Original: algorithm, a general model-based RL algorithm with sample complexity scaling with the EDEC.
Revised: algorithm, a general model-based RL algorithm with sample complexity scaling with the EDEC.

Original: We sketch the E XPLORATIVE E2D
Revised: We sketch the E XPLORATIVE E2D

Original: algorithm for a PSR class Θ as follows (full description in Algorithm 2):
Revised: algorithm for a PSR class Θ as follows (full description in Algorithm 2):

Original: In each episode t P r T s , we maintain a distribution µ t P ∆ p Θ 0 q over an optimistic cover p r P , Θ 0 q of Θ with radius 1 { T (cf.
Revised: In each episode t P r T s , we maintain a distribution µ t P ∆ p Θ 0 q over an optimistic cover p r P , Θ 0 q of Θ with radius 1 { T (cf.

Original: Definition C.4), which we use to compute two policy distributions p p t exp , p t out q by minimizing the following risk:
Revised: Definition C.4), which we use to compute two policy distributions p p t exp , p t out q by minimizing the following risk:

Original: [Equation]
Revised: [Equation]

Original: Then, we sample policy π t „ p t exp , execute π t and collect trajectory τ t , and update the model distribution  using a Tempered Aggregation scheme, which performs a Hedge update with initialization µ 1 “ Unif p Θ 0 q , the log-likelihood loss with r P π t θ p¨q denoting the optimistic likelihood associated with model θ P Θ 0 and policy π t (cf.
Revised: Then, we sample policy π t „ p t exp , execute π t and collect trajectory τ t , and update the model distribution µ t using a Tempered Aggregation scheme, which performs a Hedge update with initialization µ 1 “ Unif p Θ 0 q , the log-likelihood loss with r P π t θ p¨q denoting the optimistic likelihood associated with model θ P Θ 0 and policy π t (cf.

Original: Definition C.4), and learning rate η ď 1 { 2 :
Revised: Definition C.4), and learning rate η ď 1 { 2 :

Original: [Equation]
Revised: [Equation]

Original: After T episodes, we output the average policy p π out : “ 
Revised: After T episodes, we output the average policy p π out : “ 1 T ř

Original: [Equation]
Revised: [Equation]

Original: ř Tt
Revised: p

Original: [Equation]
Revised: [Equation]

Original: out . Theoretical guarantee We provide a sharp bound on the EDEC for B-stable PSRs, which implies that E XPLORATIVE E2D can also learn them sample-efficient efficiently.
Revised: Theoretical guarantee We provide a sharp bound on the EDEC for B-stable PSRs, which implies that E XPLORATIVE E2D can also learn them sample-efficient efficiently.

Original: Theorem 10 (Bound on EDEC & Guarantee of E XPLORATIVE E2D) .
Revised: Theorem 10 (Bound on EDEC & Guarantee of E XPLORATIVE E2D) .

Original: Suppose Θ is a PSR class with the same core test sets t
Revised: Suppose Θ is a PSR class with the same core test sets t

Original: U
Revised: U

Original: h u h
Revised: h u h

Original: Pr H s , and
Revised: Pr H s , and

Original: each θ P Θ admits a B-representation that is Λ B -stable and has PSR rank at most d .
Revised: each θ P Θ admits a B-representation that is Λ B -stable and has PSR rank at most d .

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: As a corollary, with probability at least 1 ´ δ , Algorithm 2 outputs a policy p π out P ∆ p Π q such that
Revised: As a corollary, with probability at least 1 ´ δ , Algorithm 2 outputs a policy p π out P ∆ p Π q such that

Original: V ‹ ´ V θ ‹ p p π out q ď ε , as long as the number of episodesT ě O
Revised: V ‹ ´ V θ ‹ p p π out q ď ε , as long as the number of episodes

Original: [Equation]
Revised: [Equation]

Original: The sample complexity (10) matches OMLE (Theorem 9) and has a slight advantage in avoiding the log factor ι therein.
Revised: The sample complexity (9) matches OMLE (Theorem 9) and has a slight advantage in avoiding the log factor ι therein.

Original: In return, the d in Theorem 10 needs to upper bound the PSR rank of all models in Θ , whereas the d in Theorem 9 only needs to upper bound the rank of the true model θ ‹ .
Revised: In return, the d in Theorem 10 needs to upper bound the PSR rank of all models in Θ , whereas the d in Theorem 9 only needs to upper bound the rank of the true model θ ‹ .

Original: We also remark that E XPLORATIVE E2D explicitly requires an optimistic covering of Θ as an input to the algorithm, which may be another disadvantage compared to OMLE (which uses optimistic covering implicitly in the analyses only).
Revised: We also remark that E XPLORATIVE E2D explicitly requires an optimistic covering of Θ as an input to the algorithm, which may be another disadvantage compared to OMLE (which uses optimistic covering implicitly in the analyses only).

Original: The proof of Theorem 10 (in Appendix I.2) relies on mostly the same key steps as for analyzing the OMLE algorithm (overview in Appendix B).
Revised: The proof of Theorem 10 (in Appendix I.2) relies on mostly the same key steps as for analyzing the OMLE algorithm (overview in Appendix B).

Original: Extension: Reward-free learning  Chen et al.
Revised: Extension: Reward-free learning & All-policy model estimation Chen et al.

Original: (2022) also propose the R EWARD -F REE E2D
Revised: (2022) also design the A LL -P OLICY M ODEL -E STIMATION E2D

Original: algorithm for reward-free RL, which achieves sample complexity scaling with the Reward-Free DEC (RFDEC).
Revised: algorithm for reward-free RL (Jin et al., 2020b) and (a harder related task) all-policy model estimation , with sample complexity scaling with the All-policy Model-estimation DEC (AMDEC) of the model class.

Original: We show that for B-stable PSRs, the RFDEC (44) can be upper bounded similar to the 
Revised: We show that for B-stable PSRs, the AMDEC (43) can be upper bounded similar to the EDEC, and

Original: EDEC, and thus R EWARD -F REE E2D (Algorithm 3) can be used to learn stable PSRs in a rewardfree manner (Theorem H.4 & Appendix H.2).
Revised: thus A LL -P OLICY M ODELE STIMATION E2D (Algorithm 3) can be used to learn stable PSRs in a reward-free manner (Theorem H.4 & Appendix H.2).

Original: 4. M ODEL - BASED O PTIMISTIC P OSTERIOR S AMPLING (MOPS)
Revised: 4. M ODEL - BASED O PTIMISTIC P OSTERIOR S AMPLING (MOPS)

Original: Finally, we show that MOPS—a general model-based algorithm originally proposed for MDPs by Agarwal & Zhang (2022)—can learn B-stable PSRs with the same sample complexity as
Revised: Finally, we show that MOPS—a general model-based algorithm originally proposed for MDPs by Agarwal & Zhang (2022)—can learn B-stable PSRs with the same sample complexity as

Original: OMLE and E XPLORATIVE E2D
Revised: OMLE and E XPLORATIVE E2D

Original: modulo minor differences (Theorem H.6 & Appendix H.3).
Revised: modulo minor differences (Theorem H.6 & Appendix H.3).

Original: The analysis is parallel to that of E XPLORATIVE E2D, building on insights from Chen et al.
Revised: The analysis is parallel to that of E XPLORATIVE E2D, building on insights from Chen et al.

Original:  E XAMPLES : S AMPLE COMPLEXITY OF LEARNING POMDP S
Revised:  E XAMPLES : S AMPLE COMPLEXITY OF LEARNING POMDP S

Original: We illustrate the sample complexity of OMLE and E XPLORATIVE E2D
Revised: We illustrate the sample complexity of OMLE and E XPLORATIVE E2D

Original: given in Theorem 9 &(with MOPS giving similar results) for learning an ε near-optimal policy in the tractable POMDP subclasses presented in Section 3.2, and compare with existing results.
Revised: given in Theorem 9 &(with MOPS giving similar results) for learning an ε near-optimal policy in the tractable POMDP subclasses presented in Section 3.2, and compare with existing results.

Revised: Besides the following, our results can be further instantiated to latent MDPs (Kwon et al.

Revised: (2021), as a special case of revealing

Revised: POMDPs) and linear POMDPs (Cai et al., 2022) and improve over existing results, which we present in Appendix D.3.2 & D.3.4.

Original: Weakly revealing tabular POMDPs m -step α rev -weakly revealing tabular POMDPs are B-stable
Revised: Weakly revealing tabular POMDPs m -step α rev -weakly revealing tabular POMDPs are B-stable

Original: PSRs with Λ B ď ?
Revised: PSRs with Λ B ď ?

Original: Sα ´ 1 rev , d PSR ď S , and U A ď A m ´ 1 (Example 5).
Revised: Sα ´ 1 rev , d PSR ď S , and U A “ A m ´ 1 (Example 5).

Original: Further, the log-factor ι in Theorem 9 satisfies ι ď O p log p

Original: AU

Original: A α ´ 1 rev qq “ r O p 1 q (Appendix D.3.1).

Original: Therefore, both Theorem 9 &achieve sample complexity 
Revised: Therefore, both Theorem 9 & 10 achieve sample complexity r O ` S 2 A

Original: [Equation]
Revised: m H 2 log N Θ {p α 2 rev ε 2 q ˘ .

Original: This improves substantially over the current best result r O p S 4 A 6 m ´ 4 H 6 log N Θ {p α 4 rev ε 2 qq of Liu et al.
Revised: This improves substantially over the current best result r O p S 4 A 6 m ´ 4 H 6 log N Θ {p α 4 rev ε 2 qq of Liu et al.

Original: (2022a, Theorem 24).
Revised: (2022a, Theorem 24).

Original: For tabular POMDPs, we further have log N Θ ď r O p H p S
Revised: For tabular POMDPs, we further have log N Θ ď r O p H p S

Original: 2 A ` SO qq .
Revised: 2 A ` SO qq .

Original: Low-rank future-sufficient POMDPs m -step ν
Revised: Low-rank future-sufficient POMDPs m -step ν

Original: -future-sufficient rank- d trans POMDPs are Bstable PSRs with Λ B ď ?
Revised: -future-sufficient rank- d trans POMDPs are B-stable PSRs with Λ B ď ?

Original: U
Revised: U

Original: A ν , d PSR ď d trans , and U A ď A m ´ 1 (Example 6).
Revised: A ν , d PSR ď d trans , and U A “ A m ´ 1 (Example 6).

Original: Further, the logfactor ι in Theorem 9 satisfies ι ď O p log p

Original: d trans AU A ν qq “ r O p 1 q (Appendix D.3.3). Therefore, Theorem 9 & 10 achieve sample complexity [Equation]
Revised: Therefore, Theorem 9 &achieve sample complexity r O ` d trans A 2 m ´ 1 H 2 log N Θ ¨ ν 2 { ε 2 ˘ .

Original: This improves substantially over the r O p d 2 trans
Revised: This improves substantially over the r O p d 2 trans

Original: A 5 m ` [Equation] 1 H 2 p log N Θ q [Equation]
Revised: A 5 m ` 3 l ` 1 H 2 p log N Θ q 2 ¨ ν 4 γ 2 { ε 2 q achieved by Wang et al.

Original: ν 4 γ 2 { ε 2 q achieved by Wang et al. (2022), which requires an extra l -step γ -past-sufficiency assumption that we do not require.
Revised: (2022), which requires an extra l -step γ -past-sufficiency assumption that we do not require.

Original: Decodable low-rank POMDPs m -step decodable POMDPs with transition rank d trans are Bstable PSRs with Λ B “ 1 , d PSR ď d transď S , and U A ď A m ´ 1 (Example 7).
Revised: Decodable low-rank POMDPs m -step decodable POMDPs with transition rank d trans are B-stable PSRs with Λ B “ 1 , d PSR ď d trans , and U A “ A m ´ 1 (Example 7).

Original: Further, the log-factor ι in Theorem 9 satisfies ι ď O p log p

Original: d trans AU A qq “ r O p 1 q (Appendix D.3.5). Therefore, Theorem 9 & 10 achieve sample complexity 
Revised: Therefore, Theorem 9 &achieve sample complexity r O ` d trans A m H 2 log N Θ { ε 2 ˘ .

Original: [Equation] Compared with the  r O p d trans A m H 2 log N G { ε 2 qresult of Efroni et al.
Revised: Compared with the sample complexity upper bound r O p d trans A m H 2 log N G { ε 2 q of Efroni et al.

Original: (2022), the only difference is that their covering number N G is for the value class while N Θ is for the model class.
Revised: (2022), the only difference is that their covering number N G is for the value class while N Θ is for the model class.

Original: However, this difference is nontrivial if the model class admits a much smaller covering number than the value class required for a concrete problem.
Revised: However, this difference is nontrivial if the model class admits a much smaller covering number than the value class required for a concrete problem.

Original: For example, for tabular decodable POMDPs, using d trans ď S and log N Θ ď r O p H p S 2 A ` SO qq , we achieve the first r O p
Revised: For example, for tabular decodable POMDPs, using d trans ď S and log N Θ ď r O p H p S 2 A ` SO qq , we achieve the first r O p

Original: A m poly p H, S, O, A q{ ε 2 q sample complexity, which resolves the open question of Efroni et al.
Revised: A m poly p H, S, O, A q{ ε 2 q sample complexity, which resolves the open question of Efroni et al.

Original: Additional examples Besides the above, our results can be further instantiated to latent MDPs (Kwon et al.

Original: (2021), as a special case of revealing POMDPs) and linear POMDPs

Original: (Cai et al., 2022) and improve over existing results, which we present in Appendix D.3.2 & D.3.4.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper proposes B-stability—a new structural condition for PSRs that encompasses most of the known tractable partially observable RL problems—and designs algorithms for learning B-stable
Revised: This paper proposes B-stability—a new structural condition for PSRs that encompasses most of the known tractable partially observable RL problems—and designs algorithms for learning B-stable

Original: PSRs with sharp sample complexities.
Revised: PSRs with sharp sample complexities.

Original: We believe our work opens up many interesting questions, such as the computational efficiency of our algorithms, alternative (e.g. model-free) approaches for learning B-stable PSRs, or extensions to multi-agent settings.
Revised: We believe our work opens up many interesting questions, such as the computational efficiency of our algorithms, alternative (e.g. model-free) approaches for learning B-stable PSRs, or extensions to multi-agent settings.

Original: A R ELATED WORK
Revised: A R ELATED WORK

Original: Learning POMDPs Due to the non-Markovian nature of observations, policies in POMDPs in general depend on the full history of observations and thus are much harder to learn than in fully observable MDPs.
Revised: Learning POMDPs Due to the non-Markovian nature of observations, policies in POMDPs in general depend on the full history of observations, and thus are much harder to learn than in fully observable MDPs.

Original: It is well-established that learning a near-optimal policy in POMDPs is indeed statistically hard in the worst-case due to a sample complexity lower bound that is exponential in the horizon (Mossel & Roch, 2005; Krishnamurthy et al., 2016).
Revised: It is well-established that learning a near-optimal policy in POMDPs is indeed statistically hard in the worst-case, due to a sample complexity lower bound that is exponential in the horizon (Mossel & Roch, 2005; Krishnamurthy et al., 2016).

Original: Algorithms achieving such upper bounds are developed in (Kearns et al., 1999; Even-Dar et al., 2005).
Revised: Algorithms achieving such upper bounds are developed in (Kearns et al., 1999; Even-Dar et al., 2005).

Original: Poupart & Vlassis (2008); Ross et al.
Revised: Poupart & Vlassis (2008); Ross et al.

Original: (2007) develop Bayesian methods to learn POMDPs while Azizzadenesheli et al.
Revised: (2007) develop Bayesian methods to learn POMDPs, while Azizzadenesheli et al.

Original: (2018) consider learning the optimal memoryless policies with policy gradient methods.
Revised: (2018) consider learning the optimal memoryless policies with policy gradient methods.

Original: Sample-efficient algorithms for learning POMDPs have also been developed in Hsu et al.
Revised: Sample-efficient algorithms for learning POMDPs have also been developed in Hsu et al.

Original: Azizzadenesheli et al.
Revised: Azizzadenesheli et al.

Original: ; Guo et al.
Revised: ; Guo et al.

Original: ; Xiong et al.
Revised: ; Xiong et al.

Original: ; Jahromi et al.
Revised: ; Jahromi et al.

Original: ; These works assume exploratory data or reachability assumptions, and thus do not address the challenge of exploration.
Revised: ; These works assume exploratory data or reachability assumptions, and thus do not address the challenge of exploration.

Original: For learning POMDPs in the online (exploration) setting, sample-efficient algorithms have been proposed under various structural conditions, including reactiveness (Jiang et al., 2017), revealing conditions (Jin et al., 2020a;
Revised: For learning POMDPs in the online (exploration) setting, sample-efficient algorithms have been proposed under various structural conditions, including reactiveness (Jiang et al., 2017), revealing conditions (Jin et al., 2020a;

Original: Liu et al., 2022a;b), revealing (future/past-sufficiency) and low rank (Cai et al., 2022; Wang et al., 2022), decodablity (Efroni et al., 2022), latent MDP
Revised: Liu et al., 2022a;c), revealing (future/past-sufficiency) and low rank (Cai et al., 2022; Wang et al., 2022), decodablity (Efroni et al., 2022), latent MDP

Original: (Kwon et al., 2021), learning short-memory policies (Uehara et al., 2022b), and deterministic transitions (Uehara et al., 2022a).
Revised: (Kwon et al., 2021), learning short-memory policies (Uehara et al., 2022b), and deterministic transitions (Uehara et al., 2022a).

Original: Our B-stability condition encompasses most of these structural conditions, through which we provide a unified analysis with significantly sharper sample complexities (cf.
Revised: Our B-stability condition encompasses most of these structural conditions, through which we provide a unified analysis with significantly sharper sample complexities (cf.

Original: Section 3 & 5).
Revised: Section 3 & 5).

Original: For the computational aspect, planning in POMDPs is known to be PSPACE-compete (Papadimitriou & Tsitsiklis, 1987; Littman, 1994; Burago et al., 1996; Lusena et al., 2001).
Revised: For the computational aspect, planning in POMDPs is known to be PSPACE-compete (Papadimitriou & Tsitsiklis, 1987; Littman, 1994; Burago et al., 1996; Lusena et al., 2001).

Original: The recent work of Golowich et al.
Revised: The recent work of Golowich et al.

Original: (2022b;a) establishes the belief contraction property in revealing POMDPs, which leads to algorithms with quasi-polynomial statistical and computational efficiency.
Revised: (2022b;a) establishes the belief contraction property in revealing POMDPs, which leads to algorithms with quasi-polynomial statistical and computational efficiency.

Original: Uehara et al.
Revised: Uehara et al.

Original: (2022a) design computationally efficient algorithms under the deterministic latent transition assumption.
Revised: (2022a) design computationally efficient algorithms under the deterministic latent transition assumption.

Original: We remark that computational efficiency is beyond the scope of this paper, but is an important direction for future work.
Revised: We remark that computational efficiency is beyond the scope of this paper, but is an important direction for future work.

Original: Extensive-Form Games with Imperfect Information (EFGs; (Kuhn, 1953)) is an alternative formulation of partial observability in sequential decision-making.
Revised: Extensive-Form Games with Imperfect Information (EFGs; (Kuhn, 1953)) is an alternative formulation of partial observability in sequential decision-making.

Original: EFGs can be formulated as Partially
Revised: EFGs can be formulated as Partially

Original: Observable Markov Games (the multi-agent version of POMDPs (Liu et al., 2022b)) with a treestructure.
Revised: Observable Markov Games (the multi-agent version of POMDPs (Liu et al., 2022c)) with a treestructure.

Original: Learning from bandit feedback in EFGs has been recently studied in Farina et al.
Revised: Learning from bandit feedback in EFGs has been recently studied in Farina et al.

Original: Kozuno et al.
Revised: Kozuno et al.

Original: ; Bai et al.
Revised: ; Bai et al.

Original: (2022a;b); Song et al.
Revised: (2022a;b); Song et al.

Original: (2022), where the sample complexity scales polynomially in the size of the game tree (typically exponential in the horizon).
Revised: (2022), where the sample complexity scales polynomially in the size of the game tree (typically exponential in the horizon).

Original: This line of results is in general incomparable to ours as their tree structure assumption is different from B-stability.
Revised: This line of results is in general incomparable to ours as their tree structure assumption is different from B-stability.

Original: Learning PSRs PSRs is proposed in Littman & Sutton (2001); Singh et al.
Revised: Learning PSRs PSRs is proposed in Littman & Sutton (2001); Singh et al.

Original: ; Rosencrantz et al.
Revised: ; Rosencrantz et al.

Original: ; Boots et al.
Revised: ; Boots et al.

Original: as a general formulation of partially observable systems, following the idea of Observable Operator Models (Jaeger, 2000).
Revised: as a general formulation of partially observable systems, following the idea of Observable Operator Models (Jaeger, 2000).

Original: POMDPs can be seen as a special case of PSRs (Littman & Sutton, 2001).
Revised: POMDPs can be seen as a special case of PSRs (Littman & Sutton, 2001).

Original: Algorithms for learning PSRs have been designed assuming reachability or exploratory data, including spectral algorithms (Boots et al., 2011; Zhang et al., 2021;
Revised: Algorithms for learning PSRs have been designed assuming reachability or exploratory data, including spectral algorithms (Boots et al., 2011; Zhang et al., 2021;

Original: Jiang et al., 2018), supervised learning (Hefny et al., 2015), and others (Hamilton et al., 2014;
Revised: Jiang et al., 2018), supervised learning (Hefny et al., 2015), and others (Hamilton et al., 2014;

Original: Thon & Jaeger, 2015; Grinberg et al., 2018).
Revised: Thon & Jaeger, 2015; Grinberg et al., 2018).

Original: Closely related to us, the very recent work of Zhan et al.
Revised: Closely related to us, the very recent work of Zhan et al.

Original: (2022) develops the first sample-efficient algorithm for learning PSRs in the online setting assuming under a regularity condition.
Revised: (2022) develops the first sample-efficient algorithm for learning PSRs in the online setting assuming under a regularity condition.

Original: Our work provides three algorithms with sharper sample complexities for learning PSRs, under the more general condition of B-stability.
Revised: Our work provides three algorithms with sharper sample complexities for learning PSRs, under the more general condition of B-stability.

Revised: A concurrent work by Liu et al. (2022b) (released on the same day as this work) also identifies a general class of “well-conditioned” PSRs that can be learned sample-efficiently by the OMLE algorithm

Revised: (Liu et al., 2022a).

Revised: Our B-stability condition encompasses and is slightly more relaxed than their condition (consisting of two parts), whose part one is similar to the operator norm requirement in B-stability with a different choice of input norm, and which requires an additional second part.

Original: RL{+Next, our sample complexity is much tighter than that of Liu et al. (2022b), on both general wellconditioned/B-stable PSRs and the specific examples encompassed (such as revealing POMDPs). For example, for the general class of “ γ well-conditioned PSRs” considered in their work, our results imply a r O ` dAU 2 A H 2 log N Θ {p γ 2 ε 2 q ˘ sample complexity, whereas their result scales as r O ` d 2 A 5 U 3 A H 4 log N Θ {p γ 
 ε 2 q ˘ (extracted from their proofs, cf.+}
Revised: {+Next, our sample complexity is much tighter than that of Liu et al. (2022b), on both general wellconditioned/B-stable PSRs and the specific examples encompassed (such as revealing POMDPs). For example, for the general class of “ γ well-conditioned PSRs” considered in their work, our results imply a r O ` dAU 2 A H 2 log N Θ {p γ 2 ε 2 q ˘ sample complexity, whereas their result scales as r O ` d 2 A 5 U 3 A H 4 log N Θ {p γ 
 ε 2 q ˘ (extracted from their proofs, cf.+}

Revised: Appendix D.4).

Revised: This originates from several differences between our techniques: First, Liu et al.

Revised: (2022b)’s analysis of the OMLE algorithm is based on an ℓ 1 -type operator error bound for PSRs, combined with an ℓ 1 -Eluder argument, whereas our analysis is based on a new stronger ℓ 2 -type operator error bound for PSRs (Proposition F.2) combined with a new generalized ℓ 2 -Eluder argument (Proposition E.1), which together results in a sharper rate.

Revised: Besides, our ℓ 2 -Eluder argument also admits an in-expectation decoupling form as a variant (Proposition E.6) that is necessary for bounding the EDEC (and hence the sample complexity of the E XPLORATIVE E2D algorithm) for B-stable PSRs; it is unclear whether their ℓ 1 -Eluder argument can give the same results.

Revised: Another difference is that our performance decomposition and Eluder argument are done on a slightly difference choice of vectors from Liu et al.

Revised: (2022b), which is the main reason for our better 1 { γ dependency (or Λ B dependency for B-stable

Revised: PSRs)

Revised: ; See Appendix B for a detailed overview of our technique.

Revised: Further, in terms of algorithms, Liu et al.

Revised: (2022b) only study the OMLE algorithm, whereas we study both OMLE and two alternative algorithms Explorative E2D & MOPS in addition, which enjoy similar guarantees (with minor differences) as OMLE.

Revised: In summary, Liu et al.

Revised: (2022b) do not overlap with our contributions (2) and (3) highlighted in our abstract.

Revised: Finally, complementary to our work, Liu et al.

Revised: (2022b) identify new concrete problems such as observable POMDPs with continuous observations, and develop new techniques to show that they fall into both of our general PSR frameworks, and thus tractable to sample-efficient learning.

Revised: In particular, their result implies that this class is contained in (an extension of) the low-rank future-sufficient

Revised: POMDPs defined in Definition D.11, if we suitably extend the formulation in Definition D.11 to the continuous observation setting by replacing vectors with L 1 -integrable functions and matrices with linear operators.

Original:  with function approximation (Fully observable) RL with general function approximation has been extensively studied in a recent line of work (Jiang et al., 2017; Sun et al., 2019; Du et al., 2021;
Revised: RL with function approximation (Fully observable) RL with general function approximation has been extensively studied in a recent line of work (Jiang et al., 2017; Sun et al., 2019; Du et al., 2021;

Original: Jin et al., 2021; Foster et al., 2021; Agarwal & Zhang, 2022; Chen et al., 2022), where sampleefficient algorithms are constructed for problems admitting bounds in certain general complexity measures.
Revised: Jin et al., 2021; Foster et al., 2021; Agarwal & Zhang, 2022; Chen et al., 2022), where sampleefficient algorithms are constructed for problems admitting bounds in certain general complexity measures.

Original: While POMDPs/PSRs can be cast into their settings by treating the history p τ h ´ 1 , o h q as the state, prior to our work, it was highly unclear whether any sample-efficient learning results can be deduced from their results due to challenges in bounding the complexity measures (Liu et al., 2022a).
Revised: While POMDPs/PSRs can be cast into their settings by treating the history p τ h ´ 1 , o h q as the state, prior to our work, it was highly unclear whether any sample-efficient learning results can be deduced from their results due to challenges in bounding the complexity measures (Liu et al., 2022a).

Original: Our work answers this positively by showing that the Decision-Estimation Coefficient (DEC; Foster et al.
Revised: Our work answers this positively by showing that the Decision-Estimation Coefficient (DEC; Foster et al.

Original: for B-stable PSRs is bounded, using an explorative variant of the DEC defined by Chen et al.
Revised: for B-stable PSRs is bounded, using an explorative variant of the DEC defined by Chen et al.

Original: (2022), thereby showing that their E XPLORATIVE E2D
Revised: (2022), thereby showing that their E XPLORATIVE E2D

Original: algorithm and the closely related MOPS algorithm (Agarwal & Zhang, 2022) are both sample-efficient for B-stable
Revised: algorithm and the closely related MOPS algorithm (Agarwal & Zhang, 2022) are both sample-efficient for B-stable

Original: PSRs.
Revised: PSRs.

Original: Our work further corroborates the connections between E2D, MOPS, and OMLE identified in (Chen et al., 2022) in the setting of partially observable RL.
Revised: Our work further corroborates the connections between E2D, MOPS, and OMLE identified in (Chen et al., 2022) in the setting of partially observable RL.

Original: B O VERVIEW OF TECHNIQUES
Revised: B O VERVIEW OF TECHNIQUES

Original: The proof of Theorem 9 consists of three main steps: a careful performance decomposition into certain B-errors, bounding the squared B-errors by squared Hellinger distances, and a generalized ℓ 2 -Eluder argument.
Revised: The proof of Theorem 9 consists of three main steps: a careful performance decomposition into certain B-errors, bounding the squared B-errors by squared Hellinger distances, and a generalized ℓ 2 -Eluder argument.

Original: The proof of (the EDEC bound in)
Revised: The proof of (the EDEC bound in)

Original: Theorem 10 follows similar steps except for replacing the final Eluder argument with a decoupling argument (Proposition E.4).
Revised: Theorem 10 follows similar steps except for replacing the final Eluder argument with a decoupling argument (Proposition E.6).

Original: Step 1: Performance decomposition By the standard excess risk guarantee for MLE, our choice of β “ O p log p N Θ p 1 { T q{ δ qq guarantees with probability at least 1 ´ δ that θ ‹ P Θ k for all k P r K s (Proposition G.2(a)).
Revised: Step 1: Performance decomposition By the standard excess risk guarantee for MLE, our choice of β “ O p log p N Θ p 1 { T q{ δ qq guarantees with probability at least 1 ´ δ that θ ‹ P Θ k for all k P r K s (Proposition G.2(a)).

Original: Thus, the greedy step (Line 4 in Algorithm 1) implies valid optimism: V ‹ ď
Revised: Thus, the greedy step (Line 4 in Algorithm 1) implies valid optimism: V ‹ ď

Original: V θ k p π k q .
Revised: V θ k p π k q .

Original: We then perform an error decomposition (Proposition F.1):
Revised: We then perform an error decomposition (Proposition F.1):

Original: [Equation]
Revised: [Equation]

Original: where E ‹ k, 0 : “ 12 ›› B kH : ` q k 0 ´
Revised: where E ‹ k, 0 : “ 12 ›› B kH : ` q k 0 ´

Original: q ‹ 0 ˘›› Π , and
Revised: q ‹ 0 ˘›› Π , and

Original: E
Revised: E

Original: [Equation][Equation]
Revised: [Equation]

Original: ˘

Original: [Equation]
Revised: [Equation]

Original: where for the ground truth PSR θ ‹ and the OMLE estimates θ k from Algorithm 1, we have defined respectively t B ‹ h , q ‹ 0 u and t B kh , q k 0 u as their B-representations, and t B ‹ H : h u and t B kH : h u as the corresponding B -operators.
Revised: where for the ground truth PSR θ ‹ and the OMLE estimates θ k from Algorithm 1, we have defined respectively t B ‹ h , q ‹ 0 u and t B kh , q k 0 u as their B-representations, and t B ‹ H : h u and t B kH : h u as the corresponding B -operators.

Original: (11) follows by expanding the P π k θ k p τ q and P
Revised: (10) follows by expanding the P π k θ k p τ q and P

Original: π k θ ‹ p τ q (within the TV distance) using the B-representation and telescoping (Proposition F.1).
Revised: π k θ ‹ p τ q (within the TV distance) using the B-representation and telescoping (Proposition F.1).

Original: This decomposition is similar as the ones in Liu et al.
Revised: This decomposition is similar as the ones in Liu et al.

Original: (2022a)
Revised: (2022a)

Original: ; Zhan et al.
Revised: ; Zhan et al.

Original: (2022), and more refined by keeping the B kH : h ` 1 term in (12) (instead of bounding it right away),
Revised: (2022), and more refined by keeping the B kH : h ` 1 term in (11) (instead of bounding it right away),

Original: and using the Π -norm (3) instead of the ℓ 1 -norm as the error metric.
Revised: and using the Π -norm (3) instead of the ℓ 1 -norm as the error metric.

Original: Step 2:
Revised: Step 2:

Original: Bounding the squared B-errors By again the standard fast-rate guarantee of MLE in squared Hellinger distance (Proposition G.2(b)), we have ř k ´ 1 t “1 ř Hh “ 0 D 2H p P π th, exp θ k , P π th [Equation] exp
Revised: Bounding the squared B-errors By again the standard fast-rate guarantee of MLE in squared Hellinger distance (Proposition G.2(b)), we have ř k ´ 1 t “

Original: [Equation]
Revised: [Equation]

Original: for all k P r K s .
Revised: for all k P r K s .

Original: Next, using the B-stability of the PSR, we have for any 1 ď t ă k ď K that (Proposition F.2)
Revised: Next, using the B-stability of the PSR, we have for any 1 ď t ă k ď K that (Proposition F.2)

Original: [Equation]
Revised: [Equation]

Original: Plugging the MLE guarantee into (13) and summing over t P r k ´ 1 s yields that for all k P r K s , ř 
Revised: Plugging the MLE guarantee into (12) and summing over t P r k ´ 1 s yields that for all k P r K s , ř k

Original: [Equation]
Revised: [Equation]

Original: 1 t

Original: [Equation]
Revised: [Equation]

Revised: ı ď O ` Λ

Original: [Equation]
Revised: [Equation]

Revised: AU A β ˘ .

Original: (14) is more refined than e.g. Liu et al.
Revised: (13) (13) is more refined than e.g. Liu et al.

Original: (2022a, Lemma 11), as (14) controls the second moment of
Revised: (2022a, Lemma 11), as (13) controls the second moment of

Original: E θ,h , whereas their result only controls the first moment of a similar error.
Revised: E θ,h , whereas their result only controls the first moment of a similar error.

Original: Step 3: Generalized ℓ 2 -Eluder argument We now have (14) as a precondition and bounding (11) as our target.
Revised: Step 3: Generalized ℓ 2 -Eluder argument We now have (13) as a precondition and bounding (10) as our target.

Original: The only remaining difference is that (14) controls the error E ‹ k with respect to t π t u t ď k ´ 1 , whereas (11) requires controlling the error E ‹ k with respect to π k .
Revised: The only remaining difference is that (13) controls the error E ‹ k with respect to t π t u t ď k ´ 1 , whereas (10) requires controlling the error E ‹ k with respect to π k .

Original: To this end, we perform a generalized ℓ 2 -Eluder dimension argument adapted to the structure of the function E ‹ k ’s (Proposition E.1), which implies that when d PSR ď d , ˜ k 
Revised: To this end, we perform a generalized ℓ 2 -Eluder dimension argument adapted to the structure of the function E ‹ k ’s (Proposition E.1), which implies that when d PSR ď d , ˜ k ÿ

Original: [Equation]
Revised: [Equation]

Original: Note that such an ℓ 2 -type Eluder argument is allowed precisely as our precondition (14) is in ℓwhereas our target (11) only requires an ℓ 1 bound.
Revised: Note that such an ℓ 2 -type Eluder argument is allowed precisely as our precondition (13) is in ℓwhereas our target (10) only requires an ℓ 1 bound.

Original: In comparison, Liu et al.
Revised: In comparison, Liu et al.

Original: (2022a)
Revised: (2022a)

Original: ; Zhan et al.
Revised: ; Zhan et al.

Original: (2022) only obtain a precondition in ℓ 1 , and thus has to perform an ℓ 1 -Eluder argument which results in an additional d factor in the final sample complexity.
Revised: (2022) only obtain a precondition in ℓ 1 , and thus has to perform an ℓ 1 -Eluder argument which results in an additional d factor in the final sample complexity.

Original: Combining (11), (14) (summed over k P r K s ) and (15) completes the proof of Theorem 9.
Revised: Combining (10), (13) (summed over k P r K s ) and (14) completes the proof of Theorem 9.

Original: C T ECHNICAL TOOLS C. T ECHNICAL TOOLS
Revised: C T ECHNICAL TOOLS C. T ECHNICAL TOOLS

Original: Lemma C.1 (Hellinger conditioning lemma (Chen et al., 2022, Lemma A.4)) .
Revised: Lemma C.1 (Hellinger conditioning lemma (Chen et al., 2022, Lemma A.4)) .

Original: For any pair of random variable p X, Y q , it holds that
Revised: For any pair of random variable p X, Y q , it holds that

Original: [Equation]
Revised: [Equation]

Original: The following strong duality of (generalized) bilinear function is standard, e.g. it follows from the proof of Foster et al.
Revised: The following strong duality of (generalized) bilinear function is standard, e.g. it follows from the proof of Foster et al.

Original: (2021, Proposition 4.2).
Revised: (2021, Proposition 4.2).

Original: Theorem C.2 (Strong duality) .
Revised: Theorem C.2 (Strong duality) .

Original: Suppose that X , Y are two topological spaces, such that X is discrete and Y is finite (with discrete topology).
Revised: Suppose that X , Y are two topological spaces, such that X is discrete and Y is finite (with discrete topology).

Original: Then for a function f : X ˆ Y Ñ R that is uniformly bounded, it holds that
Revised: Then for a function f : X ˆ Y Ñ R that is uniformly bounded, it holds that

Original: [Equation]
Revised: [Equation]

Original: where ∆ 0 p X q stands for space of the finitely supported distribution on X .
Revised: where ∆ 0 p X q stands for space of the finitely supported distribution on X .

Original: We will also use the following standard concentration inequality (see e.g. Foster et al.
Revised: We will also use the following standard concentration inequality (see e.g. Foster et al.

Original: (2021, Lemma
Revised: (2021, Lemma

Original: A.4)) when analyzing algorithm OMLE.
Revised: A.4)) when analyzing algorithm OMLE.

Original: Lemma C.3.
Revised: Lemma C.3.

Original: For a sequence of real-valued random variables p X t q t ď T adapted to a filtration p F t q t ď T , the following holds with probability at least 1 ´ δ :
Revised: For a sequence of real-valued random variables p X t q t ď T adapted to a filtration p F t q t ď T , the following holds with probability at least 1 ´ δ :

Original: [Equation]
Revised: [Equation]

Original: C. C OVERING NUMBER
Revised: C. C OVERING NUMBER

Original: In this section, we present the definition of the optimistic covering number N Θ .
Revised: In this section, we present the definition of the optimistic covering number N Θ .

Original: Suppose that we have a model class Θ , such that each θ P Θ parameterizes a sequential decision process.
Revised: Suppose that we have a model class Θ , such that each θ P Θ parameterizes a sequential decision process.

Original: The ρ -optimistic covering number of Θ is defined as follows.
Revised: The ρ -optimistic covering number of Θ is defined as follows.

Original: Definition C.4 (Optimistic cover) .
Revised: Definition C.4 (Optimistic cover) .

Original: Suppose that there is a context space X .
Revised: Suppose that there is a context space X .

Original: An optimistic ρ -cover of Θ is a tuple p r P , Θ 0 q , where Θ 0 Ă Θ is a finite set, r P “
Revised: An optimistic ρ -cover of Θ is a tuple p r P , Θ 0 q , where Θ 0 Ă Θ is a finite set, r P “

Original: [Equation]
Revised: [Equation]

Original: Π specifies a optimistic likelihood function for each θ P Θ 0 , such that:
Revised: specifies a optimistic likelihood function for each θ P Θ 0 , such that:

Original: (1) For θ P Θ , there exists a θ 0 P Θ 0 satisfying: for all τ P T H and π , it holds that r P πθ 

Revised: (1) For θ P Θ , there exists a θ 0 P Θ 0 satisfying: for all τ P T H and π , it holds that r P πθ

Original: p τ q ě P πθ p τ q .
Revised: p τ q ě P πθ p τ q .

Original: For θ P Θ 0 , max π ››› P
Revised: For θ P Θ 0 , max π ››› P

Original: [Equation]
Revised: πθ p τ H “ ¨q ´ r P πθ p τ H “ ¨q ››› 1 ď ρ 2 .

Original: The optimistic covering number N Θ p ρ q is defined as the minimal cardinality of Θ 0 such that there exists r P such that p r P , M 0 q is an optimistic ρ -cover of Θ .
Revised: The optimistic covering number N Θ p ρ q is defined as the minimal cardinality of Θ 0 such that there exists r P such that p r P , M 0 q is an optimistic ρ -cover of Θ .

Original: The above definition is taken from Chen et al.
Revised: The above definition is taken from Chen et al.

Original: (2022); the covering argument in Liu et al.
Revised: (2022); the covering argument in Liu et al.

Original: (2022a) essentially uses the above notion of covering number.
Revised: (2022a) essentially uses the above notion of covering number.

Original: Besides, the optimistic covering number can be upper bounded by the bracketing number adopted by Zhan et al.
Revised: Besides, the optimistic covering number can be upper bounded by the bracketing number adopted by Zhan et al.

Original: By an explicit construction, Liu et al.
Revised: By an explicit construction, Liu et al.

Original: (2022a) show that there is a universal constant C such that for any model class Θ of tabular POMDPs, it holds that
Revised: (2022a) show that there is a universal constant C such that for any model class Θ of tabular POMDPs, it holds that

Original: [Equation]
Revised: [Equation]

Original: D P ROOFS FOR S ECTIOND. B ASIC PROPERTY OF B- REPRESENTATION
Revised: D P ROOFS FOR S ECTIOND. B ASIC PROPERTY OF B- REPRESENTATION

Original: Proposition D.1 (Equivalence between PSR definition and B-representation) .
Revised: Proposition D.1 (Equivalence between PSR definition and B-representation) .

Original: A sequential decision process is a PSR with core test sets p U
Revised: A sequential decision process is a PSR with core test sets p U

Original: h
Revised: h

Original: q h
Revised: q h

Original: Pr H s (in the sense of Definition 1)
Revised: Pr H s (in the sense of Definition 1)

Original: if and only if it admits a
Revised: if and only if it admits a

Original: B-representation with respect to p U
Revised: B-representation with respect to p U

Original: h
Revised: h

Original: q h
Revised: q h

Original: Pr H s (in the sense of Definition 2).
Revised: Pr H s (in the sense of Definition 2).

Original: Proof of Proposition D.1.
Revised: Proof of Proposition D.1.

Original: We first show that a PSR admits a B-representation.
Revised: We first show that a PSR admits a B-representation.

Original: Suppose we have a PSR with core test sets p U
Revised: Suppose we have a PSR with core test sets p U

Original: h
Revised: h

Original: q h
Revised: q h

Original: Pr H s satisfying Definition 1, with associated vectors t b t h ,h P R U
Revised: Pr H s satisfying Definition 1, with associated vectors t b t h ,h P R U

Original: h u h
Revised: h u h

Original: Pr H s ,t h P T given by (1).
Revised: Pr H s ,t h P T given by (1).

Original: Then, define
Revised: Then, define

Original: [Equation]
Revised: [Equation]

Original: We show that this gives a B-representation of the PSR.
Revised: We show that this gives a B-representation of the PSR.

Original: By (1), we have for all p h, τ h ´ 1 , o, a q that
Revised: By (1), we have for all p h, τ h ´ 1 , o, a q that

Original: [Equation]
Revised: [Equation]

Original: Applying this formula recursively, we obtain
Revised: Applying this formula recursively, we obtain

Original: [Equation]
Revised: [Equation]

Original: which completes the verification of (2) in Definition 2.
Revised: which completes the verification of (2) in Definition 2.

Original: We next show that a process admitting a B-representation is a PSR.
Revised: We next show that a process admitting a B-representation is a PSR.

Original: Suppose we have a sequential decision process that admits a B-representation with respect to p U
Revised: Suppose we have a sequential decision process that admits a B-representation with respect to p U

Original: h
Revised: h

Original: q h
Revised: q h

Original: Pr H s as in Definition 2.
Revised: Pr H s as in Definition 2.

Original: Fix h P r H s .
Revised: Fix h P r H s .

Original: We first claim that, to construct vectors p b t h ,h q t h P R U h
Revised: We first claim that, to construct vectors p b t h ,h q t h P R U h

Original: such that P p t h | τ h ´ 1 q “ x b t,h , q p τ h ´ 1 qy for all test t h and history τ h ´ 1 (Definition 1), we only need to construct such vectors for full-length tests t h “ p o h : H ` 1 , a h : H q .
Revised: such that P p t h | τ h ´ 1 q “ x b t,h , q p τ h ´ 1 qy for all test t h and history τ h ´ 1 (Definition 1), we only need to construct such vectors for full-length tests t h “ p o h : H ` 1 , a h : H q .

Original: This is because, suppose we have assigned b t h ,h P R U h for all full-length t h ’s. Then for any other t h “ p o h : h ` W ´ 1 , a h : h ` W ´ 2 q with h ` W ´ 1 ă H ` 1 (non-full-length), takeb [Equation]
Revised: This is because, suppose we have assigned b t h ,h P R U h for all full-length t h ’s. Then for any other t h “ p o h : h ` W ´ 1 , a h : h ` W ´ 2 q with h ` W ´ 1 ă H ` 1 (non-full-length), take

Original: [Equation]
Revised: [Equation]

Original: where a 1 h ` W ´ 1: H P
Revised: where a 1 h ` W ´ 1: H P

Original: A H ´ h ´ W
Revised: A H ´ h ´ W

Original: ` 2 is an arbitrary and fixed action sequence.
Revised: ` 2 is an arbitrary and fixed action sequence.

Original: For this choice we have
Revised: For this choice we have

Original: [Equation]
Revised: [Equation]

Original: h ´ 1 qy “ ÿ o h ` W : H `

Original: A b t h , p o h ` W : H ` 1 ,a 1 h ` W ´ 1: H q ,h , q p τ h ´ 1 q

Original: E “

Original: [Equation]

Original: o h ` W : H `

Original: [Equation]

Original: o

Original: [Equation]

Original: ` W : H `

Original: [Equation]

Original: 1 h ` W ´ 1:

Original: [Equation]

Original: ´

Original: [Equation]

Original: as desired.
Revised: as desired.

Original: It remains to construct b t h ,h for all full-length tests.
Revised: It remains to construct b t h ,h for all full-length tests.

Original: For any full-length test t h “ p o h : H ` 1 , a h : H q , take b t h ,h P R U h with[Equation] h
Revised: For any full-length test t h “ p o h : H ` 1 , a h : H q , take b t h ,h P R U h with

Original: [Equation]
Revised: [Equation]

Original: By definition of the B-representation, for any history τ h “ p o 1 , a 1 , ¨ ¨ ¨ , o h , a h q , and any test t h ` 1 P
Revised: By definition of the B-representation, for any history τ h “ p o 1 , a 1 , ¨ ¨ ¨ , o h , a h q , and any test t h ` 1 P

Original: U h ` 1 , we have
Revised: U h ` 1 , we have

Original: [Equation]
Revised: [Equation]

Original: or in vector form,
Revised: or in vector form,

Original: P

Original: [Equation]
Revised: [Equation]

Original: (16) where we recall P p
Revised: where we recall P p

Original: τ h q “ P p o 1 , ¨ ¨ ¨ , o h | do p a 1 , ¨ ¨ ¨ , a h qq .
Revised: τ h q “ P p o 1 , ¨ ¨ ¨ , o h | do p a 1 , ¨ ¨ ¨ , a h qq .

Original: Therefore, for the particular full history τ H “ p τ h ´ 1 , t h q , we have by applying (16) twice (for steps H and h ´ 1 )
Revised: Therefore, for the particular full history τ H “ p τ h ´ 1 , t h q , we have by applying (15) twice (for steps H and h ´ 1 )

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: Dividing both sides by P p τ h ´ 1 q (when it is nonzero), we get
Revised: Dividing both sides by P p τ h ´ 1 q (when it is nonzero), we get

Original: [Equation]

Original: P p

Original: [Equation]
Revised: [Equation]

Original: This verifies (1) for all τ h ´ 1 that are reachable.
Revised: This verifies (1) for all τ h ´ 1 that are reachable.

Original: For τ h ´ 1 that are not reachable, (17) also holds as both sides equal zero by our convention.
Revised: For τ h ´ 1 that are not reachable, (16) also holds as both sides equal zero by our convention.

Original: This completes the verification of (1) in Definition 1.
Revised: This completes the verification of (1) in Definition 1.

Original: From the proof above, we can extract the following basic property of B-representation.
Revised: From the proof above, we can extract the following basic property of B-representation.

Original: Corollary D.2.
Revised: Corollary D.2.

Original: Consider a PSR model with B-representation tt B h p o h , a h qu h,o h ,a h , q 0 u .
Revised: Consider a PSR model with B-representation tt B h p o h , a h qu h,o h ,a h , q 0 u .

Original: For 0 ď h ď H ´ 1 , it holds that
Revised: For 0 ď h ď H ´ 1 , it holds that

Original: [Equation]
Revised: [Equation]

Original: Furthermore, it holds that
Revised: Furthermore, it holds that

Original: [Equation]
Revised: [Equation]

Original: D. W EAK B- STABILITY CONDITION
Revised: D. W EAK B- STABILITY CONDITION

Original: In this section, we define a weaker structural condition on PSRs, named the weak B-stability condition.
Revised: In this section, we define a weaker structural condition on PSRs, named the weak B-stability condition.

Original: In the remaining appendices, the proofs of our main sample complexity guarantees (Theorem 9, 10, H.4, H.6) will then assume the less-stringent weak B-stability condition of PSRs.
Revised: In the remaining appendices, the proofs of our main sample complexity guarantees (Theorem 9, 10, H.4, H.6) will then assume the less-stringent weak B-stability condition of PSRs.

Original: Therefore, these main results will hold under both Λ B -stablility (Definition 4) and weak Λ B -stablility
Revised: Therefore, these main results will hold under both Λ B -stablility (Definition 4) and weak Λ B -stablility

Original: (Definition D.4) simultaneously.
Revised: (Definition D.4) simultaneously.

Original: To define weak B-stability, we first extend our definition of Π -norm to R T for any set T of tests.
Revised: To define weak B-stability, we first extend our definition of Π -norm to R T for any set T of tests.

Original: Recall that in (3), we have defined Π -norm on R T with T “ p O ˆ A q H ´ h (and in (5), the Π 1 -norm for T “ U h ).
Revised: Recall that in (3), we have defined Π -norm on R T with T “ p O ˆ A q H ´ h (and in (5), the Π 1 -norm for T “ U h ).

Original: Definition D.3 ( Π -norm for general test set) .
Revised: Definition D.3 ( Π -norm for general test set) .

Original: For T Ă T , we equip R T with }¨} Π defined by
Revised: For T Ă T , we equip R T with }¨} Π defined by

Original: [Equation]
Revised: [Equation]

Original: where max T 1 Ă T is taken over all subsets T 1 of T such that T 1 satisfies the prefix condition: there is no two t ‰ t 1 P T 1 such that t is a prefix of t 1 .
Revised: where max T 1 Ă T is taken over all subsets T 1 of T such that T 1 satisfies the prefix condition: there is no two t ‰ t 1 P T 1 such that t is a prefix of t 1 .

Original: It is straightforward to see that, for any v P R U h , we have } v } 1 ě } v } Π ě } v } Π
Revised: It is straightforward to see that, for any v P R U h , we have } v } 1 ě } v } Π ě } v } Π

Original: Definition D.4 (Weak B-stability) .
Revised: Definition D.4 (Weak B-stability) .

Original: A PSR is weakly B-stable with parameter Λ B ě 1 (henceforth weakly Λ B -stable) if it admits a B-representation and associated B -operators t B H :
Revised: A PSR is weakly B-stable with parameter Λ B ě 1 (henceforth weakly Λ B -stable) if it admits a B-representation and associated B -operators t B H :

Original: h u h
Revised: h u h

Original: Pr H s such that, for any h P r H s and p , q P R U h ě 0 , we have
Revised: Pr H s such that, for any h P r H s and p , q P R U h ě 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Despite the seemingly different form, we can show that the weak B-stability condition is indeed weaker than the B-stability condition.
Revised: Despite the seemingly different form, we can show that the weak B-stability condition is indeed weaker than the B-stability condition.

Original: Furthermore, the converse also holds: the B-stability can be implied by the weak B-stability condition, if we are willing to pay a ? 2 U
Revised: Furthermore, the converse also holds: the B-stability can be implied by the weak B-stability condition, if we are willing to pay a ? 2 U

Original: A factor.
Revised: A factor.

Original: This is given by the proposition below.
Revised: This is given by the proposition below.

Original: Proposition D.5.
Revised: Proposition D.5.

Original: If a PSR is B-stable with parameter Λ B , then it is weakly B-stable with the same parameter Λ B .
Revised: If a PSR is B-stable with parameter Λ B , then it is weakly B-stable with the same parameter Λ B .

Original: Conversely, if a PSR is weakly B-stable with parameter Λ B (cf.
Revised: Conversely, if a PSR is weakly B-stable with parameter Λ B (cf.

Original: Definition D.4)
Revised: Definition D.4)

Original: , then it is B-stable with parameter ?
Revised: , then it is B-stable with parameter ?

Original: 2 U A Λ B .
Revised: 2 U A Λ B .

Original: Proof of Proposition D.5.
Revised: Proof of Proposition D.5.

Original: We first show that B-stability implies weak B-stability.
Revised: We first show that B-stability implies weak B-stability.

Original: Fix a h P r H s .
Revised: Fix a h P r H s .

Original: We only need to show that, for p , q P R U h ě 0 , we have
Revised: We only need to show that, for p , q P R U h ě 0 , we have

Original: [Equation]
Revised: [Equation]

Original: We show this inequality by showing the bound for the p 1 , 2 q -norm and the Π 1 -norm separately.
Revised: We show this inequality by showing the bound for the p 1 , 2 q -norm and the Π 1 -norm separately.

Original: First, we have
Revised: First, we have

Original: [Equation]
Revised: [Equation]

Original: where the first inequality is due to the Cauchy-Schwarz inequality; the second inequality is due to
Revised: where the first inequality is due to the Cauchy-Schwarz inequality; the second inequality is due to

Original: AM-GM inequality; the last inequality is because max a P U A,hř
Revised: AM-GM inequality; the last inequality is because max a P U A,h

Original: [Equation] p o , a qP U 
Revised: ř o : p o , a qP U h

Original: [Equation]

Original:  p o , a q ď } v } Π .
Revised: v p o , a q ď } v } Π .

Original: Next, we have
Revised: Next, we have

Original: [Equation]
Revised: [Equation]

Original: Combining these two inequalities completes the proof of Eq.
Revised: Combining these two inequalities completes the proof of Eq.

Original: (20), which gives the first claim of
Revised: (19), which gives the first claim of

Original: Proposition D.5.
Revised: Proposition D.5.

Original: Next, we show that weak B-stability implies B-stability up to a ? 2 U
Revised: Next, we show that weak B-stability implies B-stability up to a ? 2 U

Original: A factor.
Revised: A factor.

Original: Fix a h P r H s .
Revised: Fix a h P r H s .

Original: For x P R U h , we take p “ r x s ` , q “ r x s ´ , then it suffices to show that b
Revised: For x P R U h , we take p “ r x s ` , q “ r x s ´ , then it suffices to show that b

Original: [Equation]
Revised: [Equation]

Original: Indeed, we have
Revised: Indeed, we have

Original: [Equation]
Revised: [Equation]

Original: This implies that
Revised: This implies that

Original: [Equation]
Revised: [Equation]

Original: Applying Lemma D.6 completes the proof of Eq.
Revised: Applying Lemma D.6 completes the proof of Eq.

Original: (21), and hence proves the second claim of Proposition D.5.
Revised: (20), and hence proves the second claim of Proposition D.5.

Original: Lemma D.6.
Revised: Lemma D.6.

Original: Consider the fused-norm as defined in Eq.
Revised: Consider the fused-norm as defined in Eq.

Original: For any q P R U h , we have
Revised: For any q P R U h , we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma D.6.
Revised: Proof of Lemma D.6.

Original: By definition, we clearly have } q } 1 , 2 ď } q } 1 and } q } Π 1 ď } q } 1 .
Revised: By definition, we clearly have } q } 1 , 2 ď } q } 1 and } q } Π 1 ď } q } 1 .

Original: On the other hand, by Cauchy-Schwarz inequality,
Revised: On the other hand, by Cauchy-Schwarz inequality,

Original: [Equation]
Revised: [Equation]

Original: Combining the inequalities above completes the proof
Revised: Combining the inequalities above completes the proof.

Original: [Equation]

Original: D. P ROOFS FOR S ECTION 3. D.3. R EVEALING POMDP S
Revised: D. P ROOFS FOR S ECTION 3. D.3. R EVEALING POMDP S

Original: We consider Example 5, and show that any m -step revealing POMDP admit a B-representation that is B-stable.
Revised: We consider Example 5, and show that any m -step revealing POMDP admit a B-representation that is B-stable.

Original: By definition, the initial predictive state is given by q 0 “ M 1 µ 1 .
Revised: By definition, the initial predictive state is given by q 0 “ M 1 µ 1 .

Original: For h ď H
Revised: For h ď H

Original: ´ m , we take
Revised: ´ m , we take

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where T h,a : “ T h p¨|¨ , a q P R S ˆ S is the transition matrix of action a P A  .
Revised: where T h,a : “ T h p¨|¨ , a q P R S ˆ S is the transition matrix of action a P A , and M ` h is any left inverse of M h .

Original: When h ą H ´ m , we only need to take
Revised: When h ą H ´ m , we only need to take

Original: [Equation]
Revised: [Equation]

Original: (24) where 1 p t h “ p o h , a h , t h ` 1 qq is 
Revised: [Equation] where 1 p t h “ p o h , a h , t h ` 1 qq is 1 if t h equals to p o h , a h , t h ` 1 q , and 0 otherwise.

Original: Proposition D.7 (Weakly revealing POMDPs are B-stable) .
Revised: Proposition D.7 (Weakly revealing POMDPs are B-stable) .

Original: For m -step  α rev -weakly revealing
Revised: For m -step revealing POMDP, (22) and (23) indeed give a B-representation, which is B-stable with Λ B ď max h ›› M ` h ›› ˚Ñ 1 , where [Equation] Therefore, any m -step α rev -weakly revaling POMDP is B-stable with Λ B ď ? Sα ´ 1 rev (by taking ` “ : , using }¨} 2 ď }¨} ˚ , and }¨} 1 ď ?

Original: POMDP, (23) and (24) indeed give a B-representation, which is B-stable with Λ B ď ?
Revised: S }¨} 2 ). Similarly, any m -step α rev ,ℓ 1 ℓ 1 -revealing POMDP is B-stable with Λ B ď ?

Original: Sα ´ 1 rev .
Revised: A m ´ 1 α ´ 1 rev ,ℓ 1 (using }¨} 1 ď ? U A }¨} ˚ with U A “ A m ´ 1 ).

Original: For succinctness, we  provide the proof of a more general result (Proposition D.13) that implies this result.
Revised: For succinctness, we only provide the proof of a more general result (Proposition D.12).

Original: Besides, by a similar argument, we can also show that the parameter  B that appears in Theorem 9 can be bounded by R B ď α ´ 1 rev A m (see e.g. Lemma D.14). Finally, we have κ S ď  A m by Lemma D.20.
Revised: Besides, by a similar argument, we can also show that the parameter R B that appears in Theorem 9 can be bounded by R B ď α ´ 1 rev A m (for α rev -weakly revealing POMDP) or R B ď α ´ 1 rev A m (for α rev ,ℓℓ 1 -revealing POMDP, see e.g. Lemma D.13).

Original: D.3. L ATENT MDP S
Revised: D.3. L ATENT MDP S

Original: In this section, we follow Kwon et al.
Revised: In this section, we follow Kwon et al.

Original: (2021) to show that latent MDPs as a sub-class of POMDPs, and then obtain the sample complexity for learning latent MDPs of our algorithms.
Revised: (2021) to show that latent MDPs as a sub-class of POMDPs, and then obtain the sample complexity for learning latent MDPs of our algorithms.

Original: Example D.8 (Latent MDP) .
Revised: Example D.8 (Latent MDP) .

Original: A latent MDP M is specified by a tuple
Revised: A latent MDP M is specified by a tuple

Original: [Equation]
Revised: [Equation]

Original:  where M 1 , ¨ ¨ ¨ , M N are N MDPs with joint state space S , joint action space A , horizon H , and ν P ∆ pr N sq is the mixing distribution over M 1 , ¨ ¨ ¨ , M N .
Revised: [Equation] where M 1 , ¨ ¨ ¨ , M N are N MDPs with joint state space S , joint action space A , horizon H , and ν P ∆ pr N sq is the mixing distribution over M 1 , ¨ ¨ ¨ , M N .

Original: For m P r N s , the transition dynamic of
Revised: For m P r N s , the transition dynamic of

Original: M m is specified by p T h,m :
Revised: M m is specified by p T h,m :

Original: S ˆ A Ñ ∆ p S qq Hh “ 1 along with the initial state distribution µ m , and at step h the binary random reward r h,m is generated according to probability R h,m : S ˆ A Ñ r 0 , 1 s .
Revised: S ˆ A Ñ ∆ p S qq Hh “ 1 along with the initial state distribution µ m , and at step h the binary random reward r h,m is generated according to probability R h,m : S ˆ A Ñ r 0 , 1 s .

Original: Clearly, M can be casted into a POMDP M 1 with state space S
Revised: Clearly, M can be casted into a POMDP M 1 with state space S

Original: “ r N sˆ S ˆt 0 , 1 u and observation space
Revised: “ r N sˆ S ˆt 0 , 1 u and observation space

Original: O “ S ˆ t 0 , 1 u by considering the latent state being ¯ s h “ p s h , r h , m q P S and observation being o h “ p s h , r h ´ 1 q P O .
Revised: O “ S ˆ t 0 , 1 u by considering the latent state being ¯ s h “ p s h , r h , m q P S and observation being o h “ p s h , r h ´ 1 q P O .

Original: More specifically, at the start of each episode, the environment generates a m „ ν and a state s „ µ m , then the initial latent state is ¯ s 1 “ p m, s, 0 q and o 1 “ p s, 0 q ; at each step h , the agent takes a h after receiving o h , then the environment generates r h P t 0 , 1 u , ¯ s h `and o h ` 1 according to p ¯ s h , a h q : r h “ 1 with probability R h,m p s h , a h q 11 , s h ` 1 „ T h,m p¨| s h , a h q , ¯ s h ` 1 “ p m, s h ` 1 , r h q and o h ` 1 “ p s h ` 1 , r h q .
Revised: More specifically, at the start of each episode, the environment generates a m „ ν and a state s „ µ m , then the initial latent state is ¯ s 1 “ p m, s, 0 q and o 1 “ p s, 0 q ; at each step h , the agent takes a h after receiving o h , then the environment generates r h P t 0 , 1 u , ¯ s h ` and o h ` 1 according to p ¯ s h , a h q : r h “ 1 with probability R h,m p s h , a h q 11 , s h ` 1 „ T h,m p¨| s h , a h q , ¯ s h ` 1 “ p m, s h ` 1 , r h q and o h ` 1 “ p s h ` 1 , r h q .

Original: In a latent MDP, we denote T h to be the set of all possible sequences of the form p a h , r h , s h ` 1 , ¨ ¨ ¨ , a h ` l ´ 1 , r h ` l ´ 1 , s h ` l q (called a test in (Kwon et al., 2021)).
Revised: In a latent MDP, we denote T h to be the set of all possible sequences of the form p a h , r h , s h ` 1 , ¨ ¨ ¨ , a h ` l ´ 1 , r h ` l ´ 1 , s h ` l q (called a test in (Kwon et al., 2021)).

Original: For h ď H ´ l ` 1 , t “ p a h , r h , s h ` 1 , ¨ ¨ ¨ , a h ` l ´ 1 , r h ` l ´ 1 , s h ` l q P T h and s P S , we can defineP [Equation] [Equation]
Revised: For h ď H ´ l ` 1 , t “ p a h , r h , s h ` 1 , ¨ ¨ ¨ , a h ` l ´ 1 , r h ` l ´ 1 , s h ` l q P T h and s P S , we can define

Original: [Equation]

Original: [Equation] , r [Equation] ´ 1 , s h ` l | s h “ s, do p a h , ¨ ¨ ¨ , a h ` l ´ 1 qq , where P m stands for the probability distribution under MDP M m .
Revised: P h,m p t | s q “ P m p r h , s h ` 1 , ¨ ¨ ¨ , r h ` l ´ 1 , s h ` l | s h “ s, do p a h , ¨ ¨ ¨ , a h ` l ´ 1 qq , where P m stands for the probability distribution under MDP M m .

Original: Definition D.9 (Sufficient tests for latent MDP) .
Revised: Definition D.9 (Sufficient tests for latent MDP) .

Original: A latent MDP M is said to be l -step test-sufficient, if for h P r H ´ l ` 1 s and s P S , the matrix L h p s q given by
Revised: A latent MDP M is said to be l -step test-sufficient, if for h P r H ´ l ` 1 s and s P S , the matrix L h p s q given by

Original: [Equation]
Revised: [Equation]

Original: has rank N .
Revised: has rank N .

Original: M is l -step σ -test-sufficient if σ N p
Revised: M is l -step σ -test-sufficient if σ N p

Original: L h p s qq ě σ for all h P r H ´ l ` 1 s and s P S .
Revised: L h p s qq ě σ for all h P r H ´ l ` 1 s and s P S .

Original: Under test sufficiency, the latent MDP is an p l ` 1 q -step σ -weakly revealing POMDP, as shown in (Zhan et al., 2022, Lemma 12).
Revised: Under test sufficiency, the latent MDP is an p l ` 1 q -step σ -weakly revealing POMDP, as shown in (Zhan et al., 2022, Lemma 12).

Original: Hence, as a corollary of Proposition D.7, using the fact that ˇˇ
Revised: Hence, as a corollary of Proposition D.7, using the fact that ˇˇ

Original: S ˇˇ “ 2 SN , we have the following result.
Revised: S ˇˇ “ 2 SN , we have the following result.

Original: Proposition D.10 (Latent MDPs are B-stable) .
Revised: Proposition D.10 (Latent MDPs are B-stable) .

Original: For an l -step σ -test-sufficient latent MDP M , its equivalent POMDP M 1 is p l ` 1 q -step σ -weakly revealing, and thus B-stable with Λ B ď ? 2 SNσ ´ 1 .
Revised: For an l -step σ -test-sufficient latent MDP M , its equivalent POMDP M 1 is p l ` 1 q -step σ -weakly revealing, and thus B-stable with Λ B ď ? 2 SNσ ´ 1 .

Original: Therefore, by a similar reasoning to m -step revealing POMDPs in Section 5 (and Appendix D.3.1), our algorithms OMLE/E XPLORATIVE E2D/MOPS can achieve a sample complexity of
Revised: Therefore, by a similar reasoning to m -step revealing POMDPs in Section 5 (and Appendix D.3.1), our algorithms OMLE/E XPLORATIVE E2D/MOPS can achieve a sample complexity of

Original: [Equation]
Revised: [Equation]

Original: for learning ε -optimal policy, where Θ is the class of all such latent MDPs.
Revised: for learning ε -optimal policy, where Θ is the class of all such latent MDPs.

Original: Further, the optimistic covering number of Θ can be bounded as (similar as (Liu et al., 2022a, Appendix B) and
Revised: Further, the optimistic covering number of Θ can be bounded as (similar as (Liu et al., 2022a, Appendix B) and

Original: Appendix D.3.4)
Revised: Appendix D.3.4)

Original: [Equation]
Revised: [Equation]

Original: Thus, we achieve a r O ` S 4 N 3 A l ` 3 H 3 σ ´ 2 ε ´ 2 ˘ sample complexity.
Revised: Thus, we achieve a r O ` S 4 N 3 A l ` 2 H 3 σ ´ 2 ε ´ 2 ˘ sample complexity.

Original: This improves over the result of Kwon et al.
Revised: This improves over the result of Kwon et al.

Original: (2021) who requires extra assumptions including  a gap between the N MDP transitions,reachability, and full rank condition of histories (Kwon et al., 2021, Condition 2.2).
Revised: (2021) who requires extra assumptions including reachability, a gap between the N MDP transitions, and full rank condition of histories (Kwon et al., 2021, Condition 2.2).

Original: Besides, our result does not require extra assumptions on core histories—which is needed for deriving sample complexities from the α psr -regularity of (Zhan et al., 2022)—which could be rather unnatural for latent MDPs.
Revised: Besides, our result does not require extra assumptions on core histories—which is needed for deriving sample complexities from the α psr -regularity of (Zhan et al., 2022)—which could be rather unnatural for latent MDPs.

Original: We remark that the argument above can be generalized to low-rank latent MDPs 13 straightforwardly, achieving a sample complexity of r O ` d 2 trans N 2 A l ` 2 H 2 log N Θ { σ 2 ε 2 ˘ .
Revised: We remark that the argument above can be generalized to low-rank latent MDPs 13 straightforwardly, achieving a sample complexity of r O ` d 2 trans N 2 A l ` 2 H 2 log N Θ { σ 2 ε 2 ˘ .

Original: For more details, see Appendix D.3.3.
Revised: For more details, see Appendix D.3.3.

Original: Proof of Proposition D.10.
Revised: Proof of Proposition D.10.

Original: As is pointed out by Zhan et al.
Revised: As is pointed out by Zhan et al.

Original: , the p l ` 1 q -step emission matrix of M 1 has a relatively simple form: notice that for h P r H ´ l ` 1 s , ¯ s “ p m, s, r q P S and t h “ p o h , a h , ¨ ¨ ¨ , o h ` l q P U h (with o h ` 1 “ p s h ` 1 , r h q , ¨ ¨ ¨ , o h ` l “ p s h ` l , r h ` l ´ 1 q ), we have
Revised: , the p l ` 1 q -step emission matrix of M 1 has a relatively simple form: notice that for h P r H ´ l ` 1 s , ¯ s “ p m, s, r q P S and t h “ p o h , a h , ¨ ¨ ¨ , o h ` l q P U h (with o h ` 1 “ p s h ` 1 , r h q , ¨ ¨ ¨ , o h ` l “ p s h ` l , r h ` l ´ 1 q ), we have

Original: M h p t, s q “ 1 p o h “ p s h , r h ´ 1 qq P m p r h , s h ` 1 , ¨ ¨ ¨ , r h ` l ´ 1 , s h ` l | s h “ s, do p a h , ¨ ¨ ¨ , a h ` l ´ 1 qq , where 1 p o h “ p s h , r h ´ 1 qq is 1 when o h “ p s h , r h ´ 1 q and 0 otherwise.
Revised: M h p t, s q “ 1 p o h “ p s h , r h ´ 1 qq P m p r h , s h ` 1 , ¨ ¨ ¨ , r h ` l ´ 1 , s h ` l | s h “ s, do p a h , ¨ ¨ ¨ , a h ` l ´ 1 qq , where 1 p o h “ p s h , r h ´ 1 qq is 1 when o h “ p s h , r h ´ 1 q and 0 otherwise.

Original: Therefore, up to some permutation, M h has the form
Revised: Therefore, up to some permutation, M h has the form

Original: [Equation]
Revised: [Equation]

Original:  , where␣
Revised: fi ffiffiffiffiffiffiffiffiffi fl , where

Original: [Equation]
Revised: [Equation]

Original: an ordering of S .

Original: Therefore, it follows from definition that} M : [Equation] Ñ [Equation]
Revised: Therefore, it follows from definition that

Original: max

Original: [Equation]

Original: L h p s q : }

Original: [Equation]

Original: Ñ

Original: [Equation]
Revised: [Equation]

Original: ď σ ´ 1 .

Original: Applying Proposition D.7 completes the proof.
Revised: Applying Proposition D.7 completes the proof.

Original: D.3. L OW - RANK POMDP S WITH FUTURE SUFFICIENCY
Revised: D.3. L OW - RANK POMDP S WITH FUTURE SUFFICIENCY

Original: In this section, we provide a detailed discussion of low-rank POMDPs and m -step future sufficiency condition mentioned in Example 6.
Revised: In this section, we provide a detailed discussion of low-rank POMDPs and m -step future sufficiency condition mentioned in Example 6.

Original: We present a slightly generalized version of the m -step future sufficiency condition defined in (Wang et al., 2022); see also (Cai et al., 2022).

Original: Definition D.11 (Low-rank POMDP) .

Original: A POMDP is called low-rank with transition rank d trans if its latent transitions admit the following factorization for all h P r H ´ 1 s :

Original: [Equation]

Original: where Ψ h , Φ h are unknown matrices.

Original: For notational simplicity, we also assume that there is a factorization µ 1 “ Ψ 0 Φ 0 , where Ψ 0 P R S

Original: ˆ d trans , Φ 0 P R d trans . We further make a standard normalization assumption (e.g. Jin et al. (2020c)): [Equation]
Revised: We present a slightly generalized version of the m -step future sufficiency condition defined in (Wang et al., 2022); see also (Cai et al., 2022).

Original: It is clear that low-rank POMDPs with transition rank d trans has PSR rank d PSR ď d trans .

Original: For low-rank POMDPs, we now state a slightly more relaxed version of the future-sufficiency condition defined in (Wang et al., 2022).
Revised: For low-rank POMDPs, we now state a slightly more relaxed version of the future-sufficiency condition defined in (Wang et al., 2022).

Original: Recall the m -step emission-action matrices M h P R U
Revised: Recall the m -step emission-action matrices M h P R U

Original: h ˆ S defined in (7).
Revised: h ˆ S defined in (7).

Original: Consider matrices of the form

Original: [Equation]

Original: (26) where M ` denotes any left inverse of a full-rank matrix M (i.e. M ` M “ I ).

Original: that existence p q

Original: Definition D.12 ( m -step ν -future-sufficient POMDP) .
Revised: Definition D.11 ( m -step ν -future-sufficient POMDP) .

Original: We say a low-rank POMDP is m -step νfuture-sufficient if for h P r H s , min M 6 h } M 6 h } 1 Ñ 1 ď ν , where min M 6 h is taken over all possible M 6 h ’s of the form (26) .
Revised: We say a low-rank POMDP is m -step νfuture-sufficient if for h P r H s , min M 6 h } M 6 h } 1 Ñ 1 ď ν , where min M 6 h is taken over all possible M 6 h ’s such that M 6 h M h T h ´ 1 “ T h ´ 1 .

Original: Restricting M ` “  M  : recovers the definition of Wang et al.{+any general left inverse). It is straightforward to check that this choice indeed satisfies M 
+}
Revised: Wang et al. (2022) consider a factorization of the latent transition: T h “ Ψ h Φ h with Ψ h P R S ˆ d trans , Φ h P R d trans ˆp S ˆ A q for h P r H s , and assumes that } M 6 h } 1 Ñ 1 ď ν with the specific choice M 6 h “ Ψ h ´ 1 p M h Ψ h ´ 1 q : (note that it is taking an exact pseudo-inverse instead of {+any general left inverse). It is straightforward to check that this choice indeed satisfies M 
+}

Original: Taking M ` to be any arbitrary left inverse of M (i.e. M ` M  “ I ) encompasses the setting of Cai et al. 
Revised: h M h T h “ T h , using which Definition D.11 recovers the definition of Wang et al. It also encompasses the setting of Cai et al. ( m “ 1 ).

Original: We show that the following (along with (24)) gives a B-representation for the POMDP:
Revised: We show that the following (along with (23)) gives a B-representation for the POMDP:

Original: [Equation]
Revised: [Equation]

Original: This generalizes the choice of B-representation in (23) for (tabular) revealing POMDPs, as the matrix
Revised: This generalizes the choice of B-representation in (22) for (tabular) revealing POMDPs, as the matrix

Original: M 6 h can be thought of as a “generalized pseudo-inverse” of M h that is aware of the subspace spanned by Ψ h ´ 1 .
Revised: M 6 h can be thought of as a “generalized pseudo-inverse” of M h that is aware of the subspace spanned by T h ´ 1 .

Original: This choice is more suitable when S or O are extremely large, in which case the vanilla pseudo-inverse M : h may not be bounded in }¨} 1 Ñ 1 norm.
Revised: This choice is more suitable when S or O are extremely large, in which case the vanilla pseudo-inverse M : h may not be bounded in }¨} 1 Ñ 1 norm.

Original: In the tabular case where d trans “ S and Ψ h “ I S , (27) recovers (23).
Revised: In the tabular case, setting 6 “ : in (24) recovers (22).

Original: Proposition D.13 (Future-sufficient low-rank POMDPs are B-stable) .
Revised: Proposition D.12 (Future-sufficient low-rank POMDPs are B-stable) .

Original: The operators p B h p o, a qq h,o,a given by (27) (with the case h ą H ´ m given by (24) )
Revised: The operators p B h p o, a qq h,o,a given by (24) (with the case h ą H ´ m given by (23) )

Original: is indeed a B-representation, and it is B-stable with Λ B ď  max h } M 6 h } 1 .
Revised: is indeed a B-representation, and it is B-stable with Λ B ď ? A m ´ 1 max h } M 6 h } 1 .

Original: As a corollary, any m -step ν -future-sufficient low-rank POMDP admits a B-representation with Λ B ď ?
Revised: As a corollary, any m -step ν -future-sufficient low-rank POMDP admits a B-representation with Λ B ď ?

Original: A m ´ 1 ν (and also
Revised: A m ´ 1 ν (and also

Original: R B ď A m ν ).
Revised: R B ď A m ν ).

Original: Combining Proposition D.13 and Algorithm 2 gives the sample complexity guarantee of Algorithmfor future sufficient POMDP.
Revised: Combining Proposition D.12 and Algorithm 2 gives the sample complexity guarantee of Algorithmfor future sufficient POMDP.

Original: For Algorithm OMLE, under assumption (25) with d trans ď d , we have κ d ď dA m ´ 1 by Lemma D.21.

Original: Combining this fact with R B ď A m ν  establishes the sample complexity of OMLE, as claimed in Section 5.
Revised: For Algorithm OMLE, combining R B ď A m ν with Theorem 9 establishes the sample complexity of OMLE, as claimed in Section 5.

Original: Proof of Proposition D.13.
Revised: Proof of Proposition D.12.

Original: We first show that, for h P r H s , it holds M 6 h M h T h

Original: ´ 1 “ T h

Original: [Equation]

Original: Note that

Original: [Equation]

Original: where we use the fact that p M h Ψ h ´ 1 q ` M h Ψ h

Original: ´ 1 “ I , which is due to the definition of left inverse.

Original: This gives M 6 h M h T h ´ 1 “ T h ´ 1 directly.

Original: Next, we verify (2) for 0 ď h ď H ´ m .
Revised: First, we verify (2) for 0 ď h ď H ´ m .

Original: In this case, for t h ` 1 P U h ` 1 , we have
Revised: In this case, for t h ` 1 P U h ` 1 , we have

Original: [Equation]
Revised: [Equation]

Original: ) where (i) is due to M 6 l M l T l “ T l for 1 ď l ď h , in (ii) we use the definition (7) to deduce that the p t h ` 1 , s h ` 1 q -entry of M h ` 1 is P p t
Revised: where (i) is due to M 6 l M l T l “ T l for 1 ď l ď h , in (ii) we use the definition (7) to deduce that the p t h ` 1 , s h ` 1 q -entry of M h ` 1 is P p t

Original: h ` 1 | s h ` 1 q .
Revised: h ` 1 | s h ` 1 q .

Original: Finally, we verify (2) for
Revised: Finally, we verify (2) for

Original: H ´ m ă h ă H .
Revised: H ´ m ă h ă H .

Original: In this case, U h ` 1 “ O H
Revised: In this case, U h ` 1 “ O H

Original: ´ h
Revised: ´ h

Original: ˆ A H
Revised: ˆ A H

Original: ´ h ´ 1 , and hence for τ h “ p o 1 , a 1 , ¨ ¨ ¨ , o h , a h q , t h 
Revised: ´ h ´ 1 , and hence for τ h “ p o 1 , a 1 , ¨ ¨ ¨ , o h , a h q , t h ` 1 “ p o h ` 1 , a h ` 1 , ¨ ¨ ¨ , o H q P U

Original: [Equation]
Revised: h ` 1 , we consider t H ´ m ` 1 “ p o H

Original:  1 “ p o h
Revised: ´ m ` 1 , a H ´ m ` 1 , ¨ , o H q :

Original: [Equation]
Revised: [Equation]

Original: 1 , a h` 1 , ¨ ¨ ¨ , o H q P U h ` 1 , we consider t H ´ m ` 1 “ p o H ´ m ` 1 , a H ´ m ` 1 , ¨ , o H q :
Revised: h

Original: [Equation]
Revised: [Equation]

Original: It remains to verify that the B-representation is Λ B -stable with Λ B ď ?
Revised: It remains to verify that the B-representation is Λ B -stable with Λ B ď ?

Original: A m ´ 1 ν and R B
Revised: A m ´ 1 ν and R B

Original: ď A
Revised: ď A

Original: m ν , we invoke the following lemma.
Revised: m ν , we invoke the following lemma.

Original: Lemma D.14.
Revised: Lemma D.13.

Original: For 1 ď h ď H , x P R | U h | , it holds that
Revised: For 1 ď h ď H , x P R | U h | , it holds that

Original: [Equation]
Revised: [Equation]

Original: Similarly, we haveř o,a } B h p o, a q v } 1 ď max
Revised: Similarly, we have

Original: [Equation]
Revised: [Equation]

Original: For the clarity of presentation, this section we adopt the following notation: for operator p L n q n P N , we write ś

Original: By Lemma D.14, it holds that
Revised: By Lemma D.13, it holds that

Original: [Equation]
Revised: [Equation]

Original: where the second inequality is because } x } Π ď } x } 1 and ν ě 1 , and the third inequality is due to
Revised: where the second inequality is because } x } Π ď } x } 1 and ν ě 1 , and the third inequality is due to

Original: Lemma D.6 and } x } ˚ ě } x } Π 1 by definition.
Revised: Lemma D.6 and } x } ˚ ě } x } Π 1 by definition.

Original: Similarly, we have R B ď A m ν .
Revised: Similarly, we have R B ď A m ν .

Original: This concludes the proof of Proposition D.13.
Revised: This concludes the proof of Proposition D.12.

Original: Proof of Lemma D.14.
Revised: Proof of Lemma D.13.

Original: We first consider the case h ą H
Revised: We first consider the case h ą H

Original: ´ m .
Revised: ´ m .

Original: Then for each h ď l ď H , B l is given by (24), and hence for trajectory τ h : H “ p o h , a h , ¨ ¨ ¨ , o H , a H q and x P R U h , it holds that 
Revised: Then for each h ď l ď H , B l is given by (23), and hence for trajectory τ h : H “ p o h , a h , ¨ ¨ ¨ , o H , a H q and x P R U h , it holds that B h : H p

Original: [Equation]
Revised: [Equation]

Original: This implies that } B H : h x } Π “ } x } Π and ř o
Revised: h : H q

Original: [Equation]
Revised: [Equation]

Original: B [Equation] p [Equation]
Revised: “ x p o h , a h , ¨ ¨ ¨ , o H q .

Original: [Equation] } x [Equation] 1 directly.
Revised: This implies that } B H : h x } Π “ } x } Π and ř o,a } B h p o, a q x } 1 “ A } x } 1 directly.

Original: We next consider the case h ď H
Revised: We next consider the case h ď H

Original: ´ m .
Revised: ´ m .

Original: Note that for τ h : H “ p o h , a h , ¨ ¨ ¨ , o H ´ m , a H ´ m , ¨ ¨ ¨ , o H q , we can denote t H
Revised: Note that for τ h : H “ p o h , a h , ¨ ¨ ¨ , o H ´ m , a H ´ m , ¨ ¨ ¨ , o H q , we can denote t H

Original: ´ m ` 1 “ p o H ´ m ` 1 , a H ´ m ` 1 , ¨ ¨ ¨ , o H q , then similar to (28) we have
Revised: ´ m ` 1 “ p o H ´ m ` 1 , a H ´ m ` 1 , ¨ ¨ ¨ , o H q , then similar to (25) we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, for policy π and trajectory τ h : H , it holds that
Revised: Therefore, for policy π and trajectory τ h : H , it holds that

Original: [Equation]
Revised: [Equation]

Original: and this gives } B H : h x } Π ď ››› M 6 h x ››› 1 directly.
Revised: and this gives } [Equation] 1 directly.

Original: Besides, we similarly have
Revised: Besides, we similarly have

Original: [Equation]
Revised: [Equation]

Original: The proof is completed by combining the two cases above.
Revised: The proof is completed by combining the two cases above.

Original: D.3. L INEAR POMDP S
Revised: D.3. L INEAR POMDP S

Original: Linear POMDPs (Zhan et al., 2022) is a subclass of low-rank POMDPs where the latent transition and emission dynamics are linear in certain known feature maps.
Revised: Linear POMDPs (Zhan et al., 2022) is a subclass of low-rank POMDPs where the latent transition and emission dynamics are linear in certain known feature maps.

Original: In the following, we present a slightly more general version of the linear POMDP definition in Zhan et al.
Revised: In the following, we present a slightly more general version of the linear POMDP definition in Zhan et al.

Original: (2022, Definition 5).
Revised: (2022, Definition 5).

Original: Definition D.15 (Linear POMDP) .
Revised: Definition D.14 (Linear POMDP) .

Original: A POMDP is linear with respect to the given set Ψ of feature maps p ψ h : S Ñ R d s, 1 , ψ h : S
Revised: A POMDP is linear with respect to the given set Ψ of feature maps p ψ h : S Ñ R d s, 1 , ψ h : S

Original: ˆ A Ñ R d s, 2 , φ h :
Revised: ˆ A Ñ R d s, 2 , φ h :

Original: O ˆ S Ñ R d o q h if there exists
Revised: O ˆ S Ñ R d o q h if there exists

Original: A h P R d s, 1 ˆ d s, 2 , u h P R d , v P R d s, 1 such that
Revised: A h P R d s, 1 ˆ d s, 2 , u h P R d , v P R d s, 1 such that

Original: [Equation]
Revised: [Equation]

Original: We further assume a standard normalization condition:
Revised: We further assume a standard normalization condition:

Original: For R : “ max t d s, 1 , d s, 2 , 
Revised: For R : “ max t d s, 1 , d s, 2 , d o u ,

Original: [Equation]
Revised: [Equation]

Original: Proposition D.16.
Revised: Proposition D.15.

Original: Suppose that Θ is the set of models that are linear with respect to a given Ψ and have parameters bounded by R .
Revised: Suppose that Θ is the set of models that are linear with respect to a given Ψ and have parameters bounded by R .

Original: Then log N Θ p ρ q “ O pp d s ` d o q H log p d s d o H { ρ qq , where we denote d s : “ d s, 1 d s, 2 .
Revised: Then log N Θ p ρ q “ O pp d s ` d o q H log p d s d o H { ρ qq , where we denote d s : “ d s, 1 d s, 2 .

Original: It is direct to check that any linear POMDP is a low-rank POMDP (cf.
Revised: It is direct to check that any linear POMDP is a low-rank POMDP (cf.

Original: Definition D.11) with d PSR ď d trans ď min t d s, 1 , d s, 2 u .
Revised: Example 6) with d PSR ď d trans ď min t d s, 1 , d s, 2 u .

Original: Therefore, by a similar reasoning to Appendix D.3.3, Theorem& 10 both achieve a sample complexity of r O ` min t d s, 1 , d s, 2 up d s, 1 d s, 2 ` d o q AU
Revised: Therefore, by a similar reasoning to Appendix D.3.3, Theorem 9 &both achieve a sample complexity of r O ` min t d s, 1 , d s, 2 up d s, 1 d s, 2 ` d o q AU

Original: A H 3 Λ 2 B ε ´˘ for learning an ε -optimal policy in Λ B -stable linear POMDPs (which include e.g. revealing and decodable linear POMDPs).
Revised: A H 3 Λ 2 B ε ´˘ for learning an ε -optimal policy in Λ B -stable linear POMDPs (which include e.g. revealing and decodable linear POMDPs).

Original: This result significantly improves over the result extracted from (Zhan et al., 2022, Corollary 6.5):
Revised: This result significantly improves over the result extracted from (Zhan et al., 2022, Corollary 6.5):

Original: Assuming their α psr -regularity, we have Λ B ď ?
Revised: Assuming their α psr -regularity, we have Λ B ď ?

Original: U
Revised: U

Original: A α ´ 1 psr (Example 8) and thus obtain a sample complexity of
Revised: A α ´ 1 psr (Example 8) and thus obtain a sample complexity of

Original: [Equation]
Revised: [Equation]

Original: This only scales with d 3 AU 2 A (where d ě max t d s, 1 , d s, 2 , d o u ), whereas their results involve much larger polynomial factors of all three parameters.
Revised: This only scales with d 3 AU 2 A (where d ě max t d s, 1 , d s, 2 , d o u ), whereas their results involve much larger polynomial factors of all three parameters.

Original: Further, apart from the dimension-dependence, their covering number scales with an additional log O (and thus their result does not handle extremely large observation spaces).
Revised: Further, apart from the dimension-dependence, their covering number scales with an additional log O (and thus their result does not handle extremely large observation spaces).

Original: Proof of Proposition D.16.
Revised: Proof of Proposition D.15.

Original: In the following, we generalize the construction of optimistic covering of Θ using the optimistic covering of ␣ O θh ( θ P Θ and ␣ T θh ( θ P Θ as in Liu et al.
Revised: In the following, we generalize the construction of optimistic covering of Θ using the optimistic covering of [Equation] ␣ T θh ( θ P Θ as in Liu et al.

Original: (2022a, Appendix B).
Revised: (2022a, Appendix B).

Original: Lemma D.17 (Bounding optimistic covering number for POMDPs) .
Revised: Lemma D.16 (Bounding optimistic covering number for POMDPs) .

Original: For Θ a class of POMDPs, let us denote Θ h ; o “ ␣ O θh ( θ P Θ and Θ h ; o “ ␣ T θh ( θ P Θ 16 .
Revised: For Θ a class of POMDPs, let us denote Θ h ; o “ ␣ O θh ( θ P Θ and Θ h ; o “ ␣ T θh ( θ P Θ 16 .

Original: Then it holds that for ρ P p 0 , 1 s ,
Revised: Then it holds that for ρ P p 0 , 1 s ,

Original: [Equation]
Revised: [Equation]

Original: By Lemma D.17, we only need to verify that for all h P r H s ,
Revised: By Lemma D.16, we only need to verify that for all h P r H s ,

Original: [Equation]
Revised: [Equation]

Original: We demonstrate how to construct a ρ -optimistic covering for Θ h ; o ; the construction for Θ h ; t is essentially the same.
Revised: We demonstrate how to construct a ρ -optimistic covering for Θ h ; o ; the construction for Θ h ; t is essentially the same.

Original: In the following, we follow the idea of (Chen et al., 2022, Proposition H.15).
Revised: In the following, we follow the idea of (Chen et al., 2022, Proposition H.15).

Original: Fix a h P r H s and set N “ r R { ρ s .
Revised: Fix a h P r H s and set N “ r R { ρ s .

Original: Let R 1 “ Nρ , for u P r´ R 1 , R 1 s d o , we define the ρ -neighborhood of u as B 8 p u, ρ q : “ ρ t u { ρ u ` r 0 , ρ s d , and let
Revised: Let R 1 “ Nρ , for u P r´ R 1 , R 1 s d o , we define the ρ -neighborhood of u as B 8 p u, ρ q : “ ρ t u { ρ u ` r 0 , ρ s d , and let

Original: [Equation]
Revised: [Equation]

Original: Then, if u induces a emission dynamic O h ; v , then r O h ; u p o | s q ě O h ; u p o | s q , and ÿ
Revised: Then, if u induces a emission dynamic O h ; v , then r O h ; u p o | s q ě O h ; u p o | s q , and ÿ

Original: [Equation]
Revised: [Equation]

Original: Therefore, we can pick each r O h ; u p¨|¨q a representative u such that u induce a lawful emission dynamic; there are at most p 2 N q d o many elements in the set 
Revised: Therefore, we can pick each r O h ; u p¨|¨q a representative u such that u induce a lawful emission dynamic; there are at most p 2 N q d o many elements in the set ! r O

Original: [Equation]
Revised: [Equation]

Original:  do , and hence by doing this, we obtain a Rρ -optimistic covering p r O , Θ 1 h ; o q of Θ h ; o such thatˇˇˇ Θ 1 h ; o ˇˇˇ ď p 2 r R
Revised: u p¨ [Equation] 1 ,R 1 s do , and hence by doing this, we obtain a Rρ -optimistic covering p r O , Θ 1 h ; o q of Θ h ; o such that

Original: [Equation]
Revised: [Equation]

Original: This proves Proposition D.16.
Revised: This proves Proposition D.15.

Original: Proof of Lemma D.17.
Revised: Proof of Lemma D.16.

Original: Fix a ρ 1 P p 0 , 1 s and let ρ “ ρ 1 { 3 H .
Revised: Fix a ρ 1 P p 0 , 1 s and let ρ “ ρ 1 { 3 H .

Original: Note that given a tuple of parameters p r µ 1 , r T , r O q (not necessarily induce a POMDP model), we can define r P as
Revised: Note that given a tuple of parameters p r µ 1 , r T , r O q (not necessarily induce a POMDP model) [Equation] we can define r P as

Original: [Equation]
Revised: [Equation]

Original: and r P π p τ H q “ π p τ H q ˆ r P p τ H q .
Revised: and r P π p τ H q “ π p τ H q ˆ r P p τ H q .

Original: Then for a tuple of parameters p µ 1 , T , O q that induce a POMDP such that
Revised: Then for a tuple of parameters p µ 1 , T , O q that induce a POMDP such that

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: it holds that 
Revised: it holds that ›››r

Original: [Equation]
Revised: [Equation]

Original: where (*) is because ř s h
Revised: where (*) is because ř s h

Original: ` 1 r T h p s h ` 1 | s h , a h q ď 1 ` ρ 2 and ř  o h 
Revised: ` 1 r T h p s h ` 1 | s h , a h q ď 1 ` ρ 2 and ř [Equation] O h p o h [Equation]

Original: O h p o h | s h q ď 1 ` ρ 2 for all h, s h , a h .
Revised: s [Equation] [Equation] ρ 2 for all h, s h , a h .

Original: Therefore, suppose that for each h , p r T h , Θ 1 h ; t q is a ρ -optimistic covering of Θ h ; t , and p r O h , Θ 1 h ; o q is a ρ -optimistic covering of Θ h ; o , then we can obtain a ρ 1 -optimistic covering p r P , Θ 1 q of Θ , where
Revised: Therefore, suppose that for each h , p r T h , Θ 1 h ; t q is a ρ -optimistic covering of Θ h ; t , and p r O h , Θ 1 h ; o q is a ρ -optimistic covering of Θ h ; o , then we can obtain a ρ 1 -optimistic covering p r P , Θ 1 q of Θ , where

Original: [Equation]
Revised: [Equation]

Original: Θ 1 H ; o .

Original: This completes the proof.
Revised: This completes the proof.

Original: D.3. D ECODABLE POMDP S
Revised: D.3. D ECODABLE POMDP S

Original: To construct a B-representation for the decodable POMDP, we introduce the following notation.
Revised: To construct a B-representation for the decodable POMDP, we introduce the following notation.

Original: For h ď H ´ m , we consider t h “ p o h , a h , ¨ ¨ ¨ , o h ` m ´
Revised: For h ď H ´ m , we consider t h “ p o h , a h , ¨ ¨ ¨ , o h ` m ´

Original: 1 q P U h , t h ` 1 “ p o 1 h ` 1 , a 1 h ` 1 , ¨ ¨ ¨ , o 1 h ` m q P U h ` 1 , and define 
Revised: 1 q P U h , t h ` 1 “ p o 1 h ` 1 , a 1 h ` 1 , ¨ ¨ ¨ , o 1 h ` m q P U h ` 1 , and define P

Original: [Equation]
Revised: [Equation]

Revised: |

Original: and a h ` 1: h ` m ´ 2 “ a 1 h ` 1: h ` m ´ 2 , 
Revised: [Equation] h q “ [Equation] , if

Original: [Equation]
Revised: [Equation]

Original: otherwise ,
Revised: and a h ` 1: h ` m ´ 2 “ a 1 h ` 1: h ` m ´

Original: [Equation]
Revised: [Equation]

Original: 29) where ϕ h ` m ´ 1 is the decoder function that maps t h to a latent state s h ` m ´ 1 .
Revised: 0 , otherwise , (26) where ϕ h ` m ´ 1 is the decoder function that maps t h to a latent state s h ` m ´ 1 .

Original: Similarly, for h ą H ´ m , t h P
Revised: Similarly, for h ą H ´ m , t h P

Original: U
Revised: U

Original: h , t h ` 1 P U h ` 1 , we let P h p t
Revised: h , t h ` 1 P U h ` 1 , we let P h p t

Original: h
Revised: h

Original: ` 1 | t
Revised: ` 1 | t

Original: h q be 1 if t h ends with t h ` 1 , and 0 otherwise.
Revised: h q be 1 if t h ends with t h ` 1 , and 0 otherwise.

Original: Under such definition, for all h P r H s , t h P
Revised: Under such definition, for all h P r H s , t h P

Original: U
Revised: U

Original: h , t h ` 1 P U h ` 1 , it is clear that
Revised: h , t h ` 1 P U h ` 1 , it is clear that

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: for any reachable p τ h ´ 1 , t h q , because of decodability.
Revised: for any reachable p τ h ´ 1 , t h q , because of decodability.

Original: Hence, we can interpret P h p t h ` 1 | t h q as the probability of observing t h ` 1 conditional on observing t h on step h .
Revised: Hence, we can interpret P h p t h ` 1 | t h q as the probability of observing t h ` 1 conditional on observing t h on step h .

Original: Then, for h P r H s , we can take
Revised: Then, for h P r H s , we can take

Original: [Equation]
Revised: [Equation]

Original: (31) where 1 pp o, a q Ñ t h q is 1 if t h starts with p o, a q and 0 otherwise 19 .
Revised: [Equation] where 1 pp o, a q Ñ t h q is 1 if t h starts with p o, a q and 0 otherwise 19 .

Original: We verify that (31) indeed gives a B-representation for decodable POMDPs:
Revised: We verify that (28) indeed gives a B-representation for decodable POMDPs:

Original: Proposition D.18 (Decodable POMDPs are B-stable) .
Revised: Proposition D.17 (Decodable POMDPs are B-stable) .

Original: (31) gives a B-stable B-representation of the m -step decodable POMDP, with Λ B “ 1 .
Revised: (28) gives a B-stable B-representation of the m -step decodable POMDP, with Λ B “ 1 .

Original: The results above already guarantee the sample complexity of E XPLORATIVE E2D for decodable
Revised: The results above already guarantee the sample complexity of E XPLORATIVE E2D for decodable

Original: POMDPs.
Revised: POMDPs.

Original: For OMLE, we can similarly obtain that ř o,a } B h p o, a q x } 1 “ A } x } 1 , and thus we can take R B “ A .
Revised: For OMLE, we can similarly obtain that ř o,a } B h p o, a q x } 1 “ A } x } 1 , and thus we can take R B “ A .

Original: Furthermore, when we consider tabular decodable POMDP, κ S ď A m ´ 1 by

Original: Lemma D.20; for the transition

Original: rank- d decodable POMDP (Definition D.11 with normalization (25)), κ d ď dA m ´ 1 by Lemma D.21.

Original: Combining these facts establishes the sample complexity of OMLE as claimed in Section 5.
Revised: Combining this fact with Theorem 9 establishes the sample complexity of OMLE as claimed in Section 5.

Original: Proof of Proposition D.18.
Revised: Proof of Proposition D.17.

Original: We verify that (31) gives a B-representation for decodable POMDP:
Revised: We verify that (28) gives a B-representation for decodable POMDP:

Original: Note that for h P r H ´ 1 s , p o h , a h q P O ˆ A , t h ` 1 P U h ` 1 , there is a unique element t h P U h
Revised: Note that for h P r H ´ 1 s , p o h , a h q P O ˆ A , t h ` 1 P U h ` 1 , there is a unique element t h P U h

Original: such that t h is the prefix of the trajectory p o h , a h , t h ` 1 q , and it holds that
Revised: such that t h is the prefix of the trajectory p o h , a h , t h ` 1 q , and it holds that

Original: [Equation]
Revised: [Equation]

Original: Applying this equality recursively, we obtain the following fact: For trajectory τ h 1 : h and t h ` 1 P
Revised: Applying this equality recursively, we obtain the following fact: For trajectory τ h 1 : h and t h ` 1 P

Original: U h ` 1 , p τ h 1 : h , t h ` 1 q has a prefix t h 1 P U h 1 , and
Revised: U h ` 1 , p τ h 1 : h , t h ` 1 q has a prefix t h 1 P U h 1 , and

Original: [Equation]
Revised: [Equation]

Original: [Equation] where P p τ h 1 : h , t h ` 1 | t h 1 q stands for the probability of observing p τ h 1 : h , t h ` 1 q conditional on observing t h 1 at step h 1 , which is well-defined due to decodability (similar to (30)).
Revised: (29) where P p τ h 1 : h , t h ` 1 | t h 1 q stands for the probability of observing p τ h 1 : h , t h ` 1 q conditional on observing t h 1 at step h 1 , which is well-defined due to decodability (similar to (27)).

Original: Taking h 1 “ 1 and x “ q 0 in (32), we have for any history τ h and t h ` 1 P U h ` 1 thatP
Revised: Taking h 1 “ 1 and x “ q 0 in (29), we have for any history τ h and t h ` 1 P U h ` 1 that

Original: [Equation]
Revised: [Equation]

Original: Therefore, (31) indeed gives a B-representation of the decodable POMDP.
Revised: Therefore, (28) indeed gives a B-representation of the decodable POMDP.

Original: Furthermore, we can take h “ H in (32) to obtain that:
Revised: Furthermore, we can take h “ H in (29) to obtain that:

Original: For any trajectory τ h : H
Revised: For any trajectory τ h : H

Original: [Equation]
Revised: [Equation]

Original: p o h , a h , ¨ ¨ ¨ , o H , a H q , it has a prefix t
Revised: p o h , a h , ¨ ¨ ¨ , o H , a H q , it has a prefix t

Original: h P
Revised: h P

Original: U h , and
Revised: U h , and

Original: [Equation]
Revised: [Equation]

Original: Hence, for any policy π , it holds that ÿ
Revised: Hence, for any policy π , it holds that ÿ

Original: [Equation]
Revised: [Equation]

Original: Therefore, } B H : h x } Π ď } x } Π always.
Revised: Therefore, } B H : h x } Π ď } x } Π always.

Original: This completes the proof of Proposition D.18.
Revised: This completes the proof of Proposition D.17.

Original: D.3. R EGULAR PSR S
Revised: D.3. R EGULAR PSR S

Original: Proposition D.19 (Regular PSRs are B-stable) .
Revised: Proposition D.18 (Regular PSRs are B-stable) .

Original: Any α psr -regular PSR admits a Brepresentation p B q such that for all 1 ď h ď H , } B H : h x } Π ď } K : h x } 1 , where K h is any core matrix of D h (cf.
Revised: Any α psr -regular PSR admits a Brepresentation p B q such that for all 1 ď h ď H , } B H : h x } Π ď } K : h x } 1 , where K h is any core matrix of D h (cf.

Original: Example 8).
Revised: Example 8).

Original: Hence, any α psr -regular PSR is B-stable with Λ B ď ?
Revised: Hence, any α psr -regular PSR is B-stable with Λ B ď ?

Original: U
Revised: U

Original: A α ´ 1 psr .As
Revised: A α ´ 1 psr .As

Original: a byproduct, we show that the B-representation also has R B ď
Revised: a byproduct, we show that the B-representation also has R B ď

Original: α ´ 1 psr AU A .
Revised: α ´ 1 psr AU A .

Original: Proof of Proposition D.19.
Revised: Proof of Proposition D.18.

Original: By (Zhan et al., 2022, Lemma 6), the PSR admits a Brepresentation such that rowspan p B h p
Revised: By (Zhan et al., 2022, Lemma 6), the PSR admits a Brepresentation such that rowspan p B h p

Original: o, a qq Ă colspan p D h ´ 1 q .
Revised: o, a qq Ă colspan p D h ´ 1 q .

Original: In the following, we show that such a B-representation is indeed what we want.
Revised: In the following, we show that such a B-representation is indeed what we want.

Original: Fix a core matrix K h ´ 1 of D h ´ 1 , and suppose that K h ´ 1 “ 
Revised: Fix a core matrix K h ´ 1 of D h ´ 1 , and suppose that K h ´ 1 “ “ q p τ 1 h ´ 1 q , ¨ ¨ ¨ , q p τ dh ´

Original: [Equation]
Revised: [Equation]

Original: ¨ ¨ ¨ , q p τ dh ´ 1 q ‰ with d “ rank p D h q .
Revised: q ‰ with d “ rank p D h q .

Original: Then it holds that
Revised: Then it holds that

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Notice that B H : h p τ h : H q q p τ jh ´ 1 q “ P p τ h : H | τ jh ´ 1 q by Corollary D.2, and hence for any policy π , we have ÿ
Revised: Notice that B H : h p τ h : H q q p τ jh ´ 1 q “ P p τ h : H | τ jh ´ 1 q by Corollary D.2, and hence for any policy π , we have ÿ

Original: [Equation]
Revised: [Equation]

Original: Therefore, it holds that } B H : h x } Π ď ››› K [Equation] h ´ 1 x ››› 1 for h P r H s and any core matrix K h ´ 1 of D h ´ 1 .
Revised: Therefore, it holds that } B H : h x } Π ď ››› K : h ´ 1 x ››› 1 for h P r H s and any core matrix K h ´ 1 of D h ´ 1 .

Original: Similarly, we can pick a core matrix K h ´ 1 such that } K : h ´ 1 } 1 ď α ´ 1 psr , then ÿo,a
Revised: Similarly, we can pick a core matrix K h ´ 1 such that } K : h ´ 1 } 1 ď α ´ 1 psr , then ÿ

Original: [Equation]
Revised: [Equation]

Original: This completes the proof.
Revised: This completes the proof.

Original: D.3. U PPER BOUND OF LOGARITHM FACTOR IN T HEOREM
Revised: D. C OMPARISON WITH WELL - CONDITIONED PSR S

Original: Recall the definition of κ d in Theorem 9: [Equation] Lemma D.20 (Upper bound on κ d for tabular POMDPs) .
Revised: Concurrent work by Liu et al.

Original: For any (tabular) POMDP, we have κ S ď U A .
Revised: (2022b) defines the following class of well-conditioned PSRs.

Original: Proof.
Revised: Definition D.19.

Original: For a POMDP with any choice of p U h q h Pr H s , fix h P r H s , we have [Equation] For d “ S , we can take
Revised: A PSR is γ -well-conditioned if it admits a B-representation such that for all h P r H s , policy π (that starts at step h ), vector x P R U h , the following holds:

Original: [Equation]
Revised: [Equation]

Original: then } F 1 } 1 Ñ 1 ď U A because each column of F 1 has 1-norm bounded by | U A,h ` 1 | , and  } F 2 } 1Ñ 1 ď 1 each column of F 1 has 1-norm bounded by 1 .
Revised: By (30) and the inequality } x } 1 ď ?

Original: Therefore, κ S ď U A . Lemma D.21 (Upper bound on κ d for low-rank POMDPs) .
Revised: U A } x } ˚ (Lemma D.6), any γ -well-conditioned PSR is a B-stable PSR with Λ B ď ?

Original: For any low-rank POMDP with transition rank d trans , we have κ d trans ď U A max h κ d trans p T h q . In particular, assuming (25) , it holds that κ d trans ď UA d trans .
Revised: U

Original: Proof.
Revised: A γ ´ 1 .

Original: In a low-rank POMDP, fix h P r H s , for any factorization T h “ Ψ h Φ h , it holds that
Revised: Plugging this into our main results shows that, for wellconditioned PSRs, OMLE, E XPLORATIVE E2D and MOPS all achieve sample complexity

Original: [Equation]
Revised: [Equation]

Original: For d “ d trans , we can take
Revised: which is better than the sample complexity 20 r O ` d 2 A

Original: [Equation]
Revised: [Equation]

Original: then } F 1 } 1 Ñ 1 ď U A } Ψ h } 1 Ñ 1 , } F 2 } 1 Ñ 1 ď } Φ h } 1 Ñ 1 .

Original: Therefore, κ d trans p D h q ď U

Original: A κ d trans p T h q , and hence κ d trans ď U A max h κ d trans p T h q for low-rank POMDPs.

Original: Lemma D.22 (Upper bound on κ d for regular PSRs) .

Original: For any α psr -regular PSR with rank d PSR , we have κ d PSR ď U 2 A α ´ 1 psr . Proof. For any regular PSR, fix h P r H s , we have D h “ K h p K : h D h q , where K h P R Uh ` 1 ˆ d h with d h “ rank p D h q is any core matrix of D h as defined in Example 8.
Revised: U

Original: [-Therefore, we can pick a core matrix K h such that } K : h } 
-]
Revised: [-Therefore, we can pick a core matrix K h such that } K : h } 
-]

Original: ď α ´ 1 psr , then

Original: [Equation]
Revised: [Equation]

Original: where we have used } K h } 1 Ñ 1 ď U A and } D h } 1 Ñ 1 ď U A .
Revised: log N Θ { γ 4 ε 2 ˘ achieved by the analysis of OMLE in Liu et al. (2022b).

Original: [-Thus, κ d PSR ď U 
 A α ´ 1 psr for PSRs.-]
Revised: [-Thus, κ d PSR ď U 
 A α ´ 1 psr for PSRs.-]Also, being well-conditioned imposes the extra restriction (31) on the structure of the PSR, while our B-stability condition does not.

Original: E D ECORRELATION ARGUMENTS
Revised: E D ECORRELATION ARGUMENTS

Original: In this section, we present two decorrelation propositions: the generalized ℓ 2 -Eluder argument (Proposition E.1), and the decoupling argument (Proposition E.4).
Revised: In this section, we present two decorrelation propositions: the generalized ℓ 2 -Eluder argument (Proposition E.1), and the decoupling argument (Proposition E.6).

Original: These two propositions are important steps in the proof of main theorems (Theorem 9, 10, H.4, H.6).
Revised: These two propositions are important steps in the proof of main theorems (Theorem 9, 10, H.4, H.6).

Original: These two Propositions are parallel: Proposition E.1 is the triangular-to-diagonal version of the decorrelation used in the proof of Theorem 9 (see Appendix G for its proof), whereas Proposition E.4 is the expectation-toexpectation version of the decorrelation used in the proof of Theorem 10 (see Appendix I for its proof).
Revised: These two Propositions are parallel: Proposition E.1 is the triangular-to-diagonal version of the decorrelation used in the proof of Theorem 9 (see Appendix G for its proof), whereas Proposition E.6 is the expectation-toexpectation version of the decorrelation used in the proof of Theorem 10 (see Appendix I for its proof).

Original: E. G ENERALIZED ℓ 2 -E LUDER ARGUMENT
Revised: E. G ENERALIZED ℓ 2 -E LUDER ARGUMENT

Original: We first present the triangular-to-diagonal version of the decorrelation argument, the generalized ℓ 2 -Eluder argument.
Revised: We first present the triangular-to-diagonal version of the decorrelation argument, the generalized ℓ 2 -Eluder argument.

Original: Proposition E.1 (Generalized ℓ 2 -Eluder argument) .
Revised: Proposition E.1 (Generalized ℓ 2 -Eluder argument) .

Original: Suppose we have sequences of vectors
Revised: Suppose we have sequences of vectors

Original: [Equation]
Revised: [Equation]

Original: where I , R are arbitrary (abstract) index sets.
Revised: where I , R are arbitrary (abstract) index sets.

Original: Consider functions t f k : R d Ñ R u k Pr Ks
Revised: Consider functions t f k : R d Ñ R u k Pr K

Original: [Equation]
Revised: [Equation]

Original: Assume that the following condition holds:
Revised: Assume that the following condition holds:

Original: [Equation]
Revised: [Equation]

Original: where p q k P ∆ p
Revised: where p q k P ∆ p

Original: I qq k Pr K s is a family of distributions over I .
Revised: I qq k Pr K s is a family of distributions over I .

Original: Then for any M ą 0 , it holds that
Revised: Then for any M ą 0 , it holds that

Original: [Equation]
Revised: [Equation]

Original: where R 2 x “ max k E i „
Revised: where R 2 x “ max k E i „

Original: q k r} x k,i } 22 s , R y “ max k,r ř

Original: [Equation]
Revised: [Equation]

Original: y k
Revised: k

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: j
Revised: k

Original: [Equation]
Revised: [Equation]

Original: r } 2 .
Revised: ř

Original:  We call this proposition “generalized ℓ 2 -Eluder argument” because, when I is a single element set and β k “ β , the result reduces to if 
Revised: [Equation] We call this proposition “generalized ℓ 2 -Eluder argument” because, when I is a single element set and β k “ β , the result reduces to if ÿ

Original: [Equation]
Revised: [Equation]

Original:  all k P 
Revised: β, for all k P r K s , then k ÿ

Original: [Equation]
Revised: [Equation]

Original: then

Original: [Equation]
Revised: [Equation]

Original:  as long as max t | f t p x t q| ď 1 , which implies that the function class t f t u t has Eluder dimension r
Revised: (32) as long as max t | f t p x t q| ď 1 , which implies that the function class t f t u t has Eluder dimension r

Original: O p d q .
Revised: O p d q .

Original: In particular, when t f k u k Pr K s is given by f k p x q “ |x y k , x y| , (33) is equivalent to the standard ℓ 2 -Eluder argument for linear functions, which can be proved using the elliptical potential lemma (Lattimore & Szepesv´ari, 2020, Lemma 19.4).
Revised: In particular, when t f k u k Pr K s is given by f k p x q “ |x y k , x y| , (32) is equivalent to the standard ℓ 2 -Eluder argument for linear functions, which can be proved using the elliptical potential lemma (Lattimore & Szepesv ´ ari, 2020, Lemma 19.4).

Original: In the following, we present a corollary of Proposition E.1 that is more suitable for our applications.
Revised: In the following, we present a corollary of Proposition E.1 that is more suitable for our applications.

Original: Corollary E.2.
Revised: Corollary E.2.

Original: Suppose we have a sequence of functions t f k : R n Ñ R u k Pr K s :
Revised: Suppose we have a sequence of functions t f k : R n Ñ R u k Pr K s :

Original: [Equation]
Revised: [Equation]

Original: which is given by the family of vectors t y k,j,r u p k,j,r qPr K sˆr J sˆ R Ă R n .
Revised: which is given by the family of vectors t y k,j,r u p k,j,r qPr K sˆr J sˆ R Ă R n .

Original: Further assume that there exists L ą 0 such that f k p x q ď L } x } 1 .
Revised: Further assume that there exists L ą 0 such that f k p x q ď L } x } 1 .

Original: Consider further a sequence of vector p x i q i P I , satisfying the following condition
Revised: Consider further a sequence of vector p x i q i P I , satisfying the following condition

Original: [Equation]
Revised: [Equation]

Original: and the matrix X “ r x i s i 
Revised: and the subspace spanned by p x i q i P I has dimension at most d . Then it holds that k

Original: P
Revised: [Equation]

Original: I P R n ˆ I has rank at most d . Then it holds that 
Revised: We prove Proposition E.1 and Corollary E.2 in the following subsections. Remark E.3. In the initial version of this paper, the statement of Corollary E.2 was slightly different from above, which states that under the same precondition,

Original: [Equation]
Revised: [Equation]

Original: where 
Revised: where matrix X : “ r x i s i P I P R n ˆ I and

Original: [Equation]
Revised: [Equation]

Original: We prove Proposition E.1 and Corollary E.2 in the following.
Revised: After our initial version, we noted the concurrent work Liu et al. (2022b, Lemma G.3) which essentially shows that κ d p X q ď d max i } x i } 1 by an elegant argument using the Barycentric spanner. For the sake of simplicity, we have applied their result (cf. Lemma E.5) to make Corollary E.2 slightly more convenient to use.

Revised: We also note that, in the initial version of this paper, in the statement of Theorem 9, the sample complexity involved a log factor ι : “ log p 1 ` Kd Λ B R B κ d q , where κ d : “ max h κ d p D h q , which we then tightly bounded for all concrete problem classes in terms of the corresponding problem parameter.

Revised: The above change makes the statement slightly cleaner (though the result slightly looser) by always using the bound κ d ď dU A .

Revised: The effect on the final result is however minor, as the sample complexity of OMLE only depends on κ d logarithmically through ι , and the sample complexity of

Revised: MOPS or E XPLORATIVE E2D does not involve this factor.

Original: E.1. P ROOF OF P ROPOSITION E. 
Revised: E.1. P ROOF OF P ROPOSITION E.

Original: To prove this proposition, we first show that the proposition can be reduced to the case when n “ 1 , extending the idea of the proof of (Liu et al., 2022a, Proposition 22).
Revised: To prove this proposition, we first show that the proposition can be reduced to the case when n “ 1 , extending the idea of the proof of (Liu et al., 2022a, Proposition 22).

Original: After that, we invoke a certain variant of the elliptical potential lemma to derive the desired inequality.
Revised: After that, we invoke a certain variant of the elliptical potential lemma to derive the desired inequality.

Original: We first transform and reduce the problem.
Revised: We first transform and reduce the problem.

Original: For every pair of p k, i q P r K s ˆ
Revised: For every pair of p k, i q P r K s ˆ

Original: I , we take r ˚ p
Revised: I , we take r ˚ p

Original: k, i q : “ arg max r 
Revised: k, i q : “ arg max r ř j |x x k,i , y k,j,r y| , and consider r y k,i,j : “ y k,j,r ˚ p k,i q @p k, i, j q P r K s ˆ I ˆ r n s . We then define r y k,i : “ ÿ

Original: [Equation]
Revised: [Equation]

Original: k
Revised: ,i

Original: [Equation]
Revised: [Equation]

Original: i , yk,j,r y| , and consider
Revised: y

Original: [Equation]
Revised: [Equation]

Original: We then define
Revised: ,i,

Original: [Equation]
Revised: [Equation]

Original:  Under such a transformation, it holds that for all t, k, i, i 1 ,
Revised: I . Under such a transformation, it holds that for all t, k, i, i 1 ,

Original: [Equation]
Revised: [Equation]

Revised: y

Revised: [Equation]

Original: Therefore, it remains to bound ř kt
Revised: Therefore, it remains to bound ř kt

Original: “ 1 M ^ E i „
Revised: “ 1 M ^ E i „

Original: q t |x x t,i , r y t,i y| , under the condition
Revised: q t |x x t,i , r y t,i y| , under the condition

Original: that for all k P r K s , ř t
Revised: that for all k P r K s , ř t

Original: ă k E i „
Revised: ă k E i „

Original: q t r max i 1 |x x t,i , r y k,i 1 y| 2 s ď β k .
Revised: q t r max i 1 |x x t,i , r y k,i 1 y| 2 s ď β k .

Original: To show this, we define Φ t : “ E i „ q t “ x t,i x J t,i ‰ , and take λ 0 “ M 2 R 2 y , V k : “ λ 0 I ` ř t ă k Φ t .
Revised: To show this, we define Φ t : “ E i „ q t “ x t,i x J t,i ‰ , and take λ 0 “ M 2 R 2 y , V k : “ λ 0 I ` ř t ă k Φ t .

Original: Then k ÿ t “
Revised: Then k ÿ t “

Original: [Equation]
Revised: [Equation]

Original:  ď 
Revised: “min ! M, E i „ q t ” } x t,i } V ´ 1 t } r y t,i } V t ı) ď k ÿ t “min "

Original: [Equation]
Revised: M, c p M 2 ` β t q E i „ q t ” } x t,i } 2 V ´ 1 t ı* ď k ÿ t “c p M 2 ` β t q min !

Original:  where the second inequality is due to the fact that for all p t, 
Revised: 1 , E i „ q t ” } x t,i } 2 V ´ 1 t ı) ď ´ kM 2 ` k ÿ t “β t ¯ 12 ´ k ÿ t “min ! 1 , E i „ q t ” } x t,i } 2 V ´ 1 t ı)¯, where the second inequality is due to the fact that for all p t, i q ,

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: In order to bound the term ř kt “ 1 min t 1 , tr p V ´ 1 { 2 k Φ k V ´ 1 { 2 k qu , we invoke the following standard lemma, which generalizes Lattimore & Szepesv´ari (2020, Lemma 19.4).
Revised: In order to bound the term ř kt “ 1 min t 1 , tr p V ´ 1 { 2 k [Equation] qu , we invoke the following standard lemma, which generalizes Lattimore & Szepesv´ari (2020, Lemma 19.4).

Original: Lemma E.3 (Generalized elliptical potential lemma) .
Revised: Lemma E.4 (Generalized elliptical potential lemma) .

Original: Let t Φ k P R d ˆ d u k Pr K
Revised: Let t Φ k P R d ˆ d u k Pr K

Original: s be a sequence of symmetric semi-positive definite matrix, and V k : “ λ 0 I ` ř t
Revised: s be a sequence of symmetric semi-positive definite matrix, and V k : “ λ 0 I ` ř t

Original: ă k Φ t , where λ 0 ą 0 is a fixed real.
Revised: ă k Φ t , where λ 0 ą 0 is a fixed real.

Original: Then it holds that
Revised: Then it holds that

Revised: K ÿ

Original: [Equation]
Revised: [Equation]

Original: Applying Lemma E.3 and noticing tr p Φ t q “ E i „ q t r} x t,i } 22 s ď R 2 x , the proof of Proposition E.1 is completed.
Revised: Applying Lemma E.4 and noticing tr p Φ t q “ E i „ q t r} x t,i } 22 s ď R 2 x , the proof of Proposition E.1 is completed.

Original: Proof of Lemma E.3.
Revised: Proof of Lemma E.4.

Original: By definition and by linear algebra, we have
Revised: By definition and by linear algebra, we have

Original: [Equation]
Revised: [Equation]

Original: and hence det p V k ` 1 q “ det p V k q det p
Revised: and hence det p V k ` 1 q “ det p V k q det p

Original: I ` V ´ 12 k
Revised: I ` V ´ 12 k

Original: [Equation] q .
Revised: Φ k V ´ 12 k q .

Original: Therefore, we have
Revised: Therefore, we have

Original: K 
Revised: K ÿ

Original: [Equation]
Revised: [Equation]

Original: where the first inequality is due to the fact that min t 1 , u u ď 2 log p 1 ` u q , @ u ě 0 , and the second inequality is because for any positive semi-definite matrix X , it holds det p
Revised: where the first inequality is due to the fact that min t 1 , u u ď 2 log p 1 ` u q , @ u ě 0 , and the second inequality is because for any positive semi-definite matrix X , it holds det p

Original: I ` X q ě 1 ` tr p X q .
Revised: I ` X q ě 1 ` tr p X q .

Original: Now, we have
Revised: Now, we have

Original: [Equation]
Revised: [Equation]

Original: ˙ d “ d log ˜ λ 0 ` ř Kk “ 1 tr p Φ k q d ¸ , which completes the proof of Lemma E.3.
Revised: which completes the proof of Lemma E.4.

Original: E.1. P ROOF OF C OROLLARY E. 
Revised: E.1. P ROOF OF C OROLLARY E.

Original: Let us take a pair p F 1 , F 2 q such that F 1 P R n ˆ d , F 2 P R d ˆ I , X “ F 1 F 2 and } F 1 } 1 } F 2 } 1 “ κ d p X q .

Original: We write F 2 “ r v i s i P
Revised: Let us take a decomposition x i “ Fv i @

Original: I , where v i P R d . Then for all i P I , x i “ F  1 v i .
Revised: i P I , such that } v i } 8 ď 1 and } F } 1 Ñ 1 ď max i } x i }(the existence of such a decomposition is guaranteed by Lemma E.5).

Original: We define r f k : R d Ñ R as follows:
Revised: We define r f k : R d Ñ R as follows:

Original: [Equation]
Revised: [Equation]

Original: By definition, r f k p v i q “ f k p x i q , and hence our condition becomes ÿ
Revised: By definition, r f k p v i q “ f k p x i q , and hence our condition becomes ÿ

Original: [Equation]
Revised: [Equation]

Original: Then applying Proposition E.1 gives for all k P r K s , k ÿ 
Revised: Then applying Proposition E.1 gives for all k P r K s , k ÿ t

Original: [Equation]
Revised: [Equation]

Original: where R 2“ max i } v i } 2 ď max i } v i }1 “ } F 2 } 1 , and R 1 “ max k [Equation]
Revised: where R 2 ď max i } v i } 2 ď ?

Original: r ÿ
Revised: d , and

Original: [Equation]
Revised: [Equation]

Original: ÿ

Original: [Equation]

Original: [Equation]

Original: Therefore, we have 
Revised: Therefore, we have log

Original: [Equation]
Revised: [Equation]

Original: which completes the proof of Corollary E.2.
Revised: which completes the proof of Corollary E.2.

Revised: The following lemma is an immediate consequence of Liu et al.

Original: E. D ECOUPLING ARGUMENT
Revised: (2022b, Lemma G.3). Lemma E.5. Assume that a sequence of vectors t x i u i P

Revised: I Ă R n satisfies that span p x

Revised: i : i P I q has dimension at most d and R “ max i } x } 1 ă 8 .

Revised: Then, there exists a sequence of vectors t v i u i P

Revised: I Ă R d and a matrix F P R n ˆ d , such that x i “ Fv i @

Revised: i P I , and } v i } 8 ď 1 , } F } 1 Ñ 1 ď R.

Revised: Proof.

Revised: Without loss of generality, we assume that X “ span p x

Revised: i : i P I q has dimension at most d .

Revised: Then X is a d -dimensional compact subset of R n , and we take a Barycentric spanner of X to be t w 1 , ¨ ¨ ¨ , w d u .

Revised: By definition, for each i P I , there exists weights p α ij q 1 ď j ď d such that α ij P r´ 1 , 1 s and x i “ ř dj “ 1 α ij w j .

Revised: Therefore, we can take v i “ r α ij s J 1 ď j ď d P R d and F “ r w 1 , ¨ ¨ ¨ , w d s P

Revised: R n ˆ d , and they clearly fulfill the statement of Lemma E.5.

Revised: E. D ECOUPLING ARGUMENT

Original: Proposition E.1 can be regarded a triangular-to-diagonal decorrelation result.
Revised: Proposition E.1 can be regarded a triangular-to-diagonal decorrelation result.

Original: In this section, we present its expectation-to-expectation analog, which is central for bounding Explorative DEC.
Revised: In this section, we present its expectation-to-expectation analog, which is central for bounding Explorative DEC.

Original: Proposition E.4 (Decoupling argument) .
Revised: Proposition E.6 (Decoupling argument) .

Original: Suppose we have vectors and functions
Revised: Suppose we have vectors and functions

Original: [Equation]
Revised: [Equation]

Original: where Θ , I are arbitrary abstract index sets, with functions f θ given by
Revised: where Θ , I are arbitrary abstract index sets, with functions f θ given by

Original: [Equation]
Revised: [Equation]

Original: where t y θ,j,r u p θ,j,r qP Θ ˆr J sˆ R Ă R n is a family of bounded vectors in R n .
Revised: where t y θ,j,r u p θ,j,r qP Θ ˆr J sˆ R Ă R n is a family of bounded vectors in R n .

Original: Then for any distribution µ over Θ and probability family t q θ u θ P Θ Ă ∆ p I q 
Revised: Then for any distribution µ over Θ and probability family t q θ u θ P Θ Ă ∆ p I q ,

Original: [Equation]
Revised: [Equation]

Original: where d X is the dimension of the subspace of R n spanned by p x i q i P I .
Revised: where d X is the dimension of the subspace of R n spanned by p x i q i P I .

Original: Proof of Proposition E.4.
Revised: Proof of Proposition E.6.

Original: By the assumption that t y θ,j,r u p θ,j,r q is a family of bounded vectors in
Revised: By the assumption that t y θ,j,r u p θ,j,r q is a family of bounded vectors in

Original: R d , there exists R y ă 8 such that sup θ,r [Equation] r } ď R y .
Revised: R d , there exists R y ă 8 such that sup θ,r ř n j “ 1 } y θ,j,r } ď R y .

Original: We follow the same two steps as the proof of Proposition E.1.
Revised: We follow the same two steps as the proof of Proposition E.1.

Original: First, we reduce the problem.
Revised: First, we reduce the problem.

Original: We consider r ˚ p θ, i q “ arg max r P R ř j |x x
Revised: We consider r ˚ p θ, i q “ arg max r P R ř j |x x

Original: i , y θ,j,r y| , and define the vectors
Revised: i , y θ,j,r y| , and define the vectors

Original: [Equation]
Revised: [Equation]

Original: sign

Original: [Equation]
Revised: [Equation]

Original: Then for all i P I , θ P Θ ,
Revised: Then for all i P I , θ P Θ ,

Original: [Equation]
Revised: [Equation]

Original: Dˇˇ ď
Revised: and } r y θ,i }

Original: [Equation]
Revised: [Equation]

Original: p
Revised: y θ

Original: [Equation]
Revised: [Equation]

Original: q
Revised: r ˚ p θ,

Original: [Equation]
Revised: [Equation]

Original: (34) and } r y θ,i } 2 ď ř j ›› y θ,j,r ˚p θ,i q ›› 2 ď R y .

Original:  Therefore, it suffices to bound E θ „ µ E i „
Revised: [Equation] ď R y . Therefore, it suffices to bound E θ „ µ E i „

Original: q θ r|x x i , r y θ,i y|s .
Revised: q θ r|x x i , r y θ,i y|s .

Original: Next, we define Φ λ : “ λ ` E θ „ µ E i „
Revised: Next, we define Φ λ : “ λ ` E θ „ µ E i „

Original: q θ “ x i x J i ‰ with λ ą 0 .
Revised: q θ “ x i x J i ‰ with λ ą 0 .

Original: Then we can bound the target as
Revised: Then we can bound the target as

Revised: E

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The first term can be rewritten as
Revised: The first term can be rewritten as

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: ¯

Original: [Equation]
Revised: [Equation]

Original: The second term can be bounded as
Revised: The second term can be bounded as

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to (34).
Revised: where the last inequality is due to (33).

Original: Letting λ Ñ 0 ` completes the proof of Proposition E.4.
Revised: Letting λ Ñ 0 ` completes the proof of Proposition E.6.

Original: F S TRUCTURAL PROPERTIES OF B- STABLE PSR S
Revised: F S TRUCTURAL PROPERTIES OF B- STABLE PSR S

Original: In this section, we present two important propositions that are used in the proofs of all the main theorems (Theorem 9, 10, H.4, H.6).
Revised: In this section, we present two important propositions that are used in the proofs of all the main theorems (Theorem 9, 10, H.4, H.6).

Original: The first proposition bounds the performance difference of two PSR models by B-errors.
Revised: The first proposition bounds the performance difference of two PSR models by B-errors.

Original: The second proposition bounds the squared B-errors by the Hellinger distance of observation probabilities between two models.
Revised: The second proposition bounds the squared B-errors by the Hellinger distance of observation probabilities between two models.

Original: F. P ERFORMANCE DECOMPOSITION
Revised: F. P ERFORMANCE DECOMPOSITION

Original: We first present the performance decomposition proposition.
Revised: We first present the performance decomposition proposition.

Original: Proposition F.1 (Performance decomposition) .
Revised: Proposition F.1 (Performance decomposition) .

Original: Suppose that two PSR models θ, ¯ θ admit tt B θh p o h , a h qu h,o h ,a h , q θ 0 u and tt B ¯ θh p o h , a h qu h,o h ,a h , q ¯ θ 0 u as B-representation respectively, and suppose that t B θH : h u h Pr H s and t B ¯ θH :
Revised: Suppose that two PSR models θ, ¯ θ admit tt B θh p o h , a h qu h,o h ,a h , q θ 0 u and tt B ¯ θh p o h , a h qu h,o h ,a h , q ¯ θ 0 u as B-representation respectively, and suppose that t B θH : h u h Pr H s and t B ¯ θH :

Original: h u h
Revised: h u h

Original: Pr H s are the associated B -operators respectively.
Revised: Pr H s are the associated B -operators respectively.

Original: Define
Revised: Define

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then it holds that
Revised: Then it holds that

Original: [Equation]

Original: ı

Original: [Equation]
Revised: [Equation]

Original: where for h P r H s , the expectation E ¯ θ,π is taking over τ h ´ 1 under model ¯ θ and policy π 
Revised: where for h P r H s , the expectation E ¯ θ,π is taking over τ h ´ 1 under model ¯ θ and policy π .

Original: [Equation]

Original: Proof of Proposition F.1.
Revised: Proof of Proposition F.1.

Original: By the definition of B-representation
Revised: By the definition of B-representation

Original: [Equation]
Revised: [Equation]

Original: we have P πθ p
Revised: we have P πθ p

Original: τ H q “ π p τ H q ˆ
Revised: τ H q “ π p τ H q ˆ

Original: B θH :1 p
Revised: B θH :1 p

Original: τ H
Revised: τ H

Original: q q θ 0 for PSR model θ .
Revised: q q θ 0 for PSR model θ .

Original: Then for two different PSR models θ, ¯ θ , we have
Revised: Then for two different PSR models θ, ¯ θ , we have

Original: [Equation]
Revised: [Equation]

Original: where the last equality is due to the definition of B-representation (see e.g. (16)).
Revised: where the last equality is due to the definition of B-representation (see e [Equation] g. (15)).

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to the definition of E ¯ θ,π and E ¯ θθ,h p τ h ´ 1 q .
Revised: where the last inequality is due to the definition of E ¯ θ,π and E ¯ θθ,h p τ h ´ 1 q .

Original: F. B OUNDING THE SQUARED B- ERRORS BY H ELLINGER DISTANCE
Revised: F. B OUNDING THE SQUARED B- ERRORS BY H ELLINGER DISTANCE

Original: In the following proposition [Equation] we show that under B-stability or weak B-stability, the squared Berrors can be bounded by the Hellinger distance between P π h, exp θ and P π h, exp ¯ θ .Here
Revised: In the following proposition, we show that under B-stability or weak B-stability, the squared Berrors can be bounded by the Hellinger distance between P π h, exp θ and P π h, exp ¯ θ .

Original: [Equation]

Original:  for a policy π P Π and h P r H s , π h, exp is defined as[Equation] Unif
Revised: Here, for a policy π P Π and h P r H s , π h, exp is defined as

Original: [Equation]
Revised: [Equation]

Original: which is the policy that follows π for the first h ´ 1 steps, takes Unif p
Revised: which is the policy that follows π for the first h ´ 1 steps, takes Unif p

Original: A q at step h , takes an action sequence sampled from Unif p U A,h ` 1 q at step h ` 1 , and behaves arbitrarily afterwards.
Revised: A q at step h , takes an action sequence sampled from Unif p U A,h ` 1 q at step h ` 1 , and behaves arbitrarily afterwards.

Original: This notation is consistent with the exploration policy in the OMLE
Revised: This notation is consistent with the exploration policy in the OMLE

Original: algorithm (Algorithm 1).
Revised: algorithm (Algorithm 1).

Original: Proposition F.2 (Bounding squared B-errors by squared Hellinger distance) .
Revised: Proposition F.2 (Bounding squared B-errors by squared Hellinger distance) .

Original: Suppose that the Brepresentation of θ is Λ B -stable
Revised: Suppose that the Brepresentation of θ is Λ B -stable

Original: (cf.
Revised: (cf.

Original: Definition 4) or weakly Λ B -stable (cf.
Revised: Definition 4) or weakly Λ B -stable (cf.

Original: Definition D.4), then we have for h P r H
Revised: Definition D.4), then we have for h P r H

Original: ´ 1 s
Revised: ´ 1 s

Revised: E ¯ θ,π ”

Revised: E ¯ θθ,h p τ h ´ 1 qı ď

Original: [Equation]
Revised: [Equation]

Original: and
Revised: AU A “

Revised: D 2H ` P

Original: [Equation]
Revised: [Equation]

Original: where E ¯ θθ,h p τ h ´ 1 q and E ¯ θθ, 0 are as defined in Proposition F.
Revised: h, exp θ , P π h, exp ¯ θ ˘ ` D 2H ` P π h ´ 1 , exp θ , P π h ´ 1 , exp ¯ θ ˘‰ , and E ¯ [Equation] ˘

Original: [Equation]
Revised: [Equation]

Revised: where E ¯ θ θ,h p τ h ´ 1 q and E ¯ θ θ, 0 are as defined in Proposition F.1.

Original: Proof of Proposition F.2.
Revised: Proof of Proposition F.2.

Original: We first deal with the case h P r H s .
Revised: We first deal with the case h P r H s .

Original: By taking the difference, we have2 E ¯
Revised: By taking the difference, we have

Original: [Equation]
Revised: [Equation]

Original: We now introduce several notations for the convenience of the proof.
Revised: We now introduce several notations for the convenience of the proof.

Original: For an action sequence a of length 
Revised: For an action sequence a of length l p a q , P p¨|

Original: l

Original: p a

Original: q , P p¨| τ h ´ 1 , do

Original:  p a qq stands for the distribution of o h : h ` l p a q conditional on τ h ´ 1 and taking action a for step h to step h ` l p a q ´ 1 . 2.
Revised: τ h ´ 1 , do p a qq stands for the distribution of o h : h ` l p a q conditional on τ h ´ 1 and taking action a for step h to step h ` l p a q ´ 1 . 2.

Original: Given a set A of action sequences possibly of different length
Revised: Given a set A of action sequences (possibly of different length), P Unif p

Original: P Unif p A q 
Revised: A q p¨| τ h ´ 1 q stands for the distribution of observation generated by: conditional on τ h ´ 1 , first sample a a „ Unif

Original: p¨ | τ h ´ 1 q stands for the distribution of observation generated by: conditional on τ h ´ 1 , first sample a a „ Unif p U A,h q , then take a and then observe o (of length l p a q ` 1 ).
Revised: p U A,h q , then take a and then observe o (of length l p a q ` 1 ) [Equation]

Original: By the definition of Hellinger distances and by the notations above, we have
Revised: By the definition of Hellinger distances and by the notations above [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: Next, we present two lemmas whose proof will be deferred after the proof of the proposition.
Revised: Next [Equation] we present two lemmas whose proof will be deferred after the proof of the proposition.

Original: Lemma F.3.
Revised: Lemma F.3.

Original: Suppose that B is weakly Λ B -stable
Revised: Suppose that B is weakly Λ B -stable

Original: ( Λ B -stable is a sufficient condition), then it holds that›
Revised: ( Λ B -stable is a sufficient condition) [Equation] then it holds that

Original: [Equation]
Revised: [Equation]

Original: Lemma F.4.
Revised: Lemma F.4.

Original: Suppose that B is weakly Λ B -stable
Revised: Suppose that B is weakly Λ B -stable

Original: ( Λ B -stable is a sufficient condition), then it holds that
Revised: ( Λ B -stable is a sufficient condition), then it holds that

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we first consider the case h P r H ´ 1 s .
Revised: we first consider the case h P r H ´ 1 s .

Original: Applying Lemma F.3 and taking expectation with respect to τ h ´ 1 , we obtain
Revised: Applying Lemma F.3 and taking expectation with respect to τ h ´ 1 , we obtain

Original: [Equation]
Revised: [Equation]

Original: where the second inequality is due to Lemma C.
Revised: where the second inequality is due to Lemma C.

Original: [Equation]
Revised: [Equation]

Original: and the last inequality is due to importance sampling.
Revised: and the last inequality is due to importance sampling.

Original: Similarly, applying Lemma F.4 and taking expectation with respect to τ h ´ 1 , we have E ¯ θ,π [Equation] ¸
Revised: Similarly, applying Lemma F

Original: [Equation]
Revised: [Equation]

Original: fi fl [Equation]
Revised: 4 and taking expectation with respect to τ h ´ 1 , we have

Original:  ¯ 
Revised: E ¯ θ,π

Original: [Equation]
Revised: [Equation]

Original: P

Original: [Equation]
Revised: [Equation]

Original: ˘ . The proof for h P r H ´ 1 s is completed by noting that p x ` y q 2 ď 2 x 2 ` 2 y 2 and U A “ max h | U A,h | .
Revised: The proof for h P r H ´ 1 s is completed by noting that p x ` y q 2 ď 2 x 2 ` 2 y 2 and U A “ max h | U A,h | .

Original: For the case h “ H , note that by Corollary D.2,max π
Revised: For the case h “ H , note that by Corollary D.2,

Original: [Equation]
Revised: [Equation]

Original: a
Revised: π

Original: [Equation]
Revised: [Equation]

Original: and by Lemma F.3 it holds that
Revised: and by Lemma F

Original: [Equation]
Revised: [Equation]

Revised: 3 it holds that

Original: [Equation]
Revised: [Equation]

Original: where the equality is due to U A,H only containing the null action sequence.
Revised: where the equality is due to U A,H only containing the null action sequence.

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: and applying Lemma C.1 completes the proof of the case h “ H .
Revised: and applying Lemma C.1 completes the proof of the case h “ H .

Original: The case h “ 0 is directly implied by Lemma F.3:
Revised: The case h “ 0 is directly implied by Lemma F.3:

Original: [Equation]
Revised: [Equation]

Original: Combining all these cases finishes the proof of Proposition F.2.
Revised: Combining all these cases finishes the proof of Proposition F.2.

Original: We next prove Lemma F.3 and F.4 that were used in the proof of Proposition F.2.
Revised: We next prove Lemma F.3 and F.4 that were used in the proof of Proposition F.2.

Original: Proof of Lemma F.3.
Revised: Proof of Lemma F.3.

Original: By the weak B-stability as in Definition D.

Revised: By the weak B-stability as in Definition D.

Original: (B-stability is also sufficient, see
Revised: (B-stability is also sufficient, see

Original: Eq.
Revised: Eq.

Original: (20)), we have
Revised: [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: where }¨} Π is defined in Definition D.3.
Revised: where }¨} Π is defined in Definition D

Revised: [Equation]

Original: By the definition of q θ p
Revised: By the definition of q θ p

Original: τ
Revised: τ

Original: h
Revised: h

Original: ´ 1 q , for t h “ p o , a q P U h , we have
Revised: ´ 1 q , for t h “ p o , a q P U h , we have

Original: [Equation]
Revised: [Equation]

Original: Hence, we have
Revised: Hence [Equation] we have

Original: [Equation]
Revised: [Equation]

Original:  where P πθ p T 1 | τ h ´ 1 q stands for the probability that some test in T 1 is observed under ¯ θ, π conditional on τ h ´ 1 .
Revised: ÿ [Equation] where P πθ p T 1 | τ h ´ 1 q stands for the probability that some test in T 1 is observed under ¯ θ, π conditional on τ h ´ 1 .

Original: Similarly, we have ››› q ¯ θ p τ h ´ 1 q ››› Π ď 1 .
Revised: Similarly, we have ››› q ¯ θ p τ h ´ 1 q ››› Π ď 1 .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original:  where in (i) we include those o such that p o , a q may not belong to U h ` 1 into summation, (ii) is due to the definition of P p¨|
Revised: [Equation] [Equation] where in (i) we include those o such that p o , a q may not belong to U h ` 1 into summation, (ii) is due to the definition of P p¨|

Original: τ h ´ 1 , do p a qq , and (iii) follows from importance sampling (36).
Revised: τ h ´ 1 , do p a qq , and (iii) follows from importance sampling (35).

Original: This completes the proof of Lemma F.3.
Revised: This completes the proof of Lemma F.3.

Original: Proof of Lemma F.4.
Revised: Proof of Lemma F.4.

Original: Similar to the proof of Lemma F.3, we only need to work under the weak
Revised: Similar to the proof of Lemma F.3, we only need to work under the weak

Original: B-stability condition.
Revised: B-stability condition.

Original: By Corollary D.2, for t h ` 1 “ p o , a q P U h ` 1 , it holds that“
Revised: By Corollary D.2, for t h ` 1 “ p o , a q P U h ` 1 , it holds that

Original: [Equation]
Revised: [Equation]

Original:  and hence 
Revised: q , and hence ›

Original: [Equation]
Revised: [Equation]

Original: where P πθ p T 1 | τ h ´ 1 , o, a q stands for the probability that some test in T 1 is observed under θ, π conditional on observing τ h “ p τ h ´ 1 , o, a q .
Revised: where P πθ p T 1 | τ h ´ 1 , o, a q stands for the probability that some test in T 1 is observed under θ, π conditional on observing τ h “ p τ h ´ 1 , o, a q .

Original: Similarly, we have 
Revised: Similarly, we have ››› B ¯ θh p o, a q q

Original: [Equation]
Revised: [Equation]

Original:  Therefore, by the weak B-stability as in Definition D.4 and combining with the inequalities above, it holds that
Revised: q . Therefore, by the weak B-stability as in Definition D.4 and combining with the inequalities above

Original: [Equation]
Revised: [Equation]

Original: Hence, we have
Revised: it holds that

Original: [Equation]
Revised: [Equation]

Original: ÿ
Revised: Hence, we have

Original: [Equation]
Revised: [Equation]

Original: ÿ
Revised: p¨

Original: [Equation]
Revised: [Equation]

Original: where (i) is due to the fact that max π P ∆ p A q ř [Equation] p a
Revised: τ

Original: [Equation]
Revised: [Equation]

Original: ď
Revised: ¯

Original: [Equation]
Revised: [Equation]

Original: ř a [Equation]
Revised: where (i) is due to the fact that max π P ∆ p

Original: p a q 
Revised: A q ř a P

Original: [Equation] ˘ 1 { 2 , (ii) is due to CauchySchwarz inequality, in (iii) we include those o such that p o , a q may not belong to U h ` 1 into summation, (iv) is due to (36):
Revised: A π p a q x p a q ď `ř a x p a q˘ 1 { 2 , (ii) is due to CauchySchwarz inequality, in (iii) we include those o such that p o , a q may not belong to U h ` 1 into summation, (iv) is due to (35):

Original: Unif p
Revised: Unif p