Original: A q ˝ Unif p U
Revised: A q ˝ Unif p U

Original: A,h ` 1 q is simply the uniform policy over A ˆ U A,h ` 1 .
Revised: A,h ` 1 q is simply the uniform policy over A ˆ U A,h ` 1 .

Original: This concludes the proof of Lemma F.4.
Revised: This concludes the proof of Lemma F.4.

Original: G P ROOF OF T HEOREM
Revised: G P ROOF OF T HEOREM

Original: We first restate Theorem 9 as follows in terms of the (more relaxed) weak B-stability condition.
Revised: We first restate Theorem 9 as follows in terms of the (more relaxed) weak B-stability condition.

Original: Theorem G.1 (Restatement of Theorem 9) .
Revised: Theorem G.1 (Restatement of Theorem 9) .

Original: Suppose every θ P Θ is Λ B -stable
Revised: Suppose every θ P Θ is Λ B -stable

Original: (Definition 4) or weakly Λ B -stable
Revised: (Definition 4) or weakly Λ B -stable

Original: (Definition D.4), and the true model θ ‹ P Θ with rank d PSR ď d .
Revised: (Definition D.4), and the true model θ ‹ P Θ with rank d PSR ď d .

Original: Then, choosing β “ C log p N Θ p 1 { KH qq{ δ q for some absolute constant C ą 0 , with probability at least 1 ´ δ , Algorithm 1 outputs a policy p π out P ∆ p Π q such that V ‹ ´ V θ ‹ p p π out q ď ε , as long as the number of episodes
Revised: Then, choosing β “ C log p N Θ p 1 { KH qq{ δ q for some absolute constant C ą 0 , with probability at least 1 ´ δ , Algorithm 1 outputs a policy p π out P ∆ p Π q such that V ‹ ´ V θ ‹ p p π out q ď ε , as long as the number of episodes

Original: [Equation]
Revised: [Equation]

Original: where ι : “ log p 1 ` Kd Λ B R Bκ d q with R B : “ max h t 1 , max } v } 1“ř o,a } B h p o, a q v } 1 u and κ d : “max h
Revised: where ι : “ log p 1 ` KdU A Λ B R B q with R B : “ max h t 1 , max } v } 1 “

Original: Pr H s min t} F 1 } 1 } F 2 } 1 : D h “ F 1 F 2 , F 1 P R U h ˆ d , F 2 P R d ˆ T h
Revised: [Equation]

Original: ´ 1 u . The proof of Theorem G.1 uses the following fast rate guarantee for the OMLE algorithm, which is standard (e.g. Van de Geer (2000); Agarwal et al.
Revised: The proof of Theorem G.1 uses the following fast rate guarantee for the OMLE algorithm, which is standard (e.g. Van de Geer (2000); Agarwal et al.

Original: For completeness, we present its proof in
Revised: For completeness, we present its proof in

Original: Appendix G.1.
Revised: Appendix G.1.

Original: Proposition G.2 (Guarantee of MLE) .
Revised: Proposition G.2 (Guarantee of MLE) .

Original: Suppose that we choose β ě 2 log N Θ p 1 { T q` 2 log p 1 { δ q`in Algorithm 1.
Revised: Suppose that we choose β ě 2 log N Θ p 1 { T q` 2 log p 1 { δ q`in Algorithm 1.

Original: Then with probability at least 1 ´ δ , the following holds:
Revised: Then with probability at least 1 ´ δ , the following holds:

Original: (a) For all k P r K s , θ ‹ P Θ k ; (b) For all k P r K s and any θ P Θ k , it holds that
Revised: (a) For all k P r K s , θ ‹ P Θ k ; (b) For all k P r K s and any θ P Θ k , it holds that

Original: [Equation]
Revised: [Equation]

Original: We next prove Theorem G.1.
Revised: We next prove Theorem G.1.

Original: We adopt the definitions of E ¯ θθ,h p τ h ´ 1 q as in Proposition F.1 and abbreviate E ‹ k,h “ E θ ‹ θ k ,h .
Revised: We adopt the definitions of E ¯ θθ,h p τ h ´ 1 q as in Proposition F.1 and abbreviate E ‹ k,h “ E θ ‹ θ k ,h .

Original: We also condition on the success of the event in Proposition G.2.
Revised: We also condition on the success of the event in Proposition G.2.

Original: Step 1.
Revised: Step 1.

Original: By Proposition G.2, it holds that θ ‹ P Θ .
Revised: By Proposition G.2, it holds that θ ‹ P Θ .

Original: Therefore, V θ k p π k q ě V ‹ , and by Proposition F.1, we havek ÿ
Revised: Therefore, V θ k p π k q ě V ‹ , and by Proposition F.1, we have

Original: [Equation]
Revised: [Equation]

Original: On the other hand, by Proposition F.2, we have
Revised: On the other hand, by Proposition F.2, we have

Original: [Equation]
Revised: [Equation]

Original: Furthermore, by Proposition G.2 we have
Revised: Furthermore, by Proposition G.2 we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, defining β k,h : “ ř t ă k E π t r E ‹ k,h p τ h ´ 1 q 2 s , combining the two equations above gives
Revised: Therefore, defining β k,h : “ ř t ă k E π t r E ‹ k,h p τ h ´ 1 q 2 s , combining the two equations above gives

Original: [Equation]
Revised: [Equation]

Original: Step 2.
Revised: Step 2.

Original: We would like to bridge the performance decomposition (39) and the squared B-errors bound (40) using the generalized ℓ 2 -Eluder argument.
Revised: We would like to bridge the performance decomposition (38) and the squared B-errors bound (39) using the generalized ℓ 2 -Eluder argument.

Original: We consider separately the case for h “and h P r H s .
Revised: We consider separately the case for h “and h P r H s .

Original: Case 1: h “ 0 .
Revised: Case 1: h “ 0 .

Original: This case follows directly from Cauchy-Schwarz inequality:
Revised: This case follows directly from Cauchy-Schwarz inequality:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Case 2: h P r H s .
Revised: Case 2: h P r H s .

Original: We invoke the generalized ℓ 2 -Eluder argument (actually, its corollary) as in
Revised: We invoke the generalized ℓ 2 -Eluder argument (actually, its corollary) as in

Original: Appendix E.1, restated as follows for convenience.
Revised: Appendix E.1, restated as follows for convenience.

Original: Corollary E.2.
Revised: Corollary E.2.

Original: Suppose we have a sequence of functionst f k : R n Ñ R u k Pr K s :
Revised: Suppose we have a sequence of functions

Original: [Equation]
Revised: [Equation]

Original: which is given by the family of vectors t y k,j,r u p k,j,r qPr K sˆr J sˆ R Ă R n .
Revised: which is given by the family of vectors t y k,j,r u p k,j,r qPr K sˆr J sˆ R Ă R n .

Original: Further assume that there exists L ą 0 such that f k p x q ď L } x } 1 .
Revised: Further assume that there exists L ą 0 such that f k p x q ď L } x } 1 .

Original: Consider further a sequence of vector p x i q i P I , satisfying the following condition
Revised: Consider further a sequence of vector p x i q i P I , satisfying the following condition

Original: [Equation]
Revised: [Equation]

Original: and the matrix X “ r x i s i P IP R n ˆ I has rank at most d .
Revised: and the subspace spanned by p x i q i P I has dimension at most d .

Original: Then it holds that 
Revised: Then it holds that k ÿ

Original: [Equation]
Revised: [Equation]

Original: where

Original: [Equation]

Original: We have the following three preparation steps to apply Corollary E.2.
Revised: We have the following three preparation steps to apply Corollary E.2.

Original: Recall the definition of E ‹ t,h p τ h ´ 1 q as in Proposition F.1 (in short E ‹ k,h p
Revised: Recall the definition of E ‹ t,h p τ h ´ 1 q as in Proposition F.1 (in short E ‹ k,h p

Original: τ h ´ 1 q : “ E θ ‹ θ k ,h p τ h ´ 1 q ), E ‹ k [Equation] τ 
Revised: τ h ´ 1 q : “ E θ ‹ θ k ,h p τ h ´ 1 q ), E ‹ k,h p τ h ´ 1 q : “ 1 2 max π ÿ τ h : H

Original: [Equation]
Revised: [Equation]

Original: where we replace superscript θ k of B by k for simplicity.
Revised: where we replace superscript θ k of B by k for simplicity.

Original: Let us define 
Revised: Let us define y

Original: [Equation]
Revised: [Equation]

Original: o jh , a jh q ¯ı J P R | U h | , where t τ jh : H “ p o h , a h , ¨ ¨ ¨ , o H , a H qu nj “ 1 is an ordering of all possible τ h : H (and hence n “ p OA q H ´ h ` 1 ), π is any policy that starts at step h .
Revised: where t τ jh : H “ p o h , a h , ¨ ¨ ¨ , o H , a H qu nj “ 1 is an ordering of all possible τ h : H (and hence n “ p OA q H ´ h ` 1 ), π is any policy that starts at step h .

Original: We then define f k p x q “ max π ÿ j |x y k,j,π , x y| , x P R U h .
Revised: We then define [Equation]

Original: It follows from definition that E ‹ k,h p τ h ´ 1 q “ f k p q ‹ p τ h ´ 1 qq .
Revised: It follows from definition that E ‹ k,h p τ h ´ 1 q “ f k p q ‹ p τ h ´ 1 qq .

Original: We define x i “ q ‹ p τ ih ´
Revised: We define x i “ q ‹ p τ ih ´

Original: 1 q P R | U h | , where t τ ih ´
Revised: 1 q P R | U h | , where t τ ih ´

Original: 1 u i is an ordering of all possible τ h ´ 1 P p
Revised: 1 u i is an ordering of all possible τ h ´ 1 P p

Original: O ˆ
Revised: O ˆ

Original: A q h ´ 1 .
Revised: A q h ´ 1 .

Original: Thenfor the matrix X “ r x i s i , by the assumption that θ ‹ has PSR rank less than or equal to d , we have rank p X q “ rank p D h ´ 1 q ď d .
Revised: Then by the assumption that θ ‹ has PSR rank less than or equal to d , we have dim span p x i : i P I q ď d .

Original: Furthermore, we have κ d p X q “ κ d p D h ´ q ď κ d by assumption.
Revised: Furthermore, we have } x i } 1 ď U A by definition.

Original: It remains to verify that f k is Lipschitz with respect to 1 -norm.
Revised: It remains to verify that f k is Lipschitz with respect to 1 -norm.

Original: We only need to verify it under the weak Λ B -stability condition.
Revised: We only need to verify it under the weak Λ B -stability condition.

Original: We have
Revised: We have

Revised: [Equation]

Revised: π ÿ o,a π p a | o

Original: [Equation]
Revised: [Equation]

Revised: } B ‹ h p o, a

Revised: [Equation]

Revised: }ď 2Λ B } q } 1 ` 2Λ B ÿ o,a }

Revised: [Equation]

Revised: [Equation]

Revised: a

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: ď 2Λ

Revised: [Equation]

Original:  where the first inequality follows the same argument as (36); the second inequality is due to Bstability (or weak B-stability and (22)); the last inequality is due to the definition of B .
Revised: p [Equation] ` 1 q } q } 1 , where the first inequality follows the same argument as (35) ; the second inequality is due to Bstability (or weak B-stability and (21)); the last inequality is due to the definition of B .

Original: Hence we can take L “ 2Λ B p R B ` 1 q to ensure that f k p x q ď L } x } 1 .
Revised: Hence we can take L “ 2Λ B p R B ` 1 q to ensure that f k p x q ď L } x } 1 .

Original: Therefore, applying Corollary E.2 yields
Revised: Therefore, applying Corollary E.2 yields

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ι “ log p 1 ` 2 kdκ Λ B p R B ` 1 qq .
Revised: where ι “ log p 1 ` 2 kdU A Λ B p R B ` 1 qq .

Original: This completes case 2.
Revised: This completes case 2.

Original: Combining these two cases, we obtain
Revised: Combining these two cases, we obtain

Original: [Equation]
Revised: [Equation]

Original: where (i) used (39); (ii) used the above two cases (41) and (42); (iii) used (40).
Revised: where (i) used (38); (ii) used the above two cases (40) and (41); (iii) used (39).

Original: As a consequence, whenever k ě O p Λ 2 B dAU A H ¨ βι { ε 2 q , we have 1 k ř k t “ 
Revised: As a consequence, whenever k ě O p Λ 2 B dAU A H ¨ βι { ε 2 q , we have 1 k ř kt “ 1 p V ‹ ´

Original: [Equation] ¯ ď ε .
Revised: V θ ‹ p π t qq ď ε .

Original: This completes the proof of Theorem G.1 (and hence Theorem 9).
Revised: This completes the proof of Theorem G.1 (and hence Theorem 9).

Original: G. P ROOF OF P ROPOSITION G.
Revised: G. P ROOF OF P ROPOSITION G

Original: For the simplicity of presentation, we consider the following general interaction process: For t “ 1 , ¨ ¨ ¨ , T , the learner determines a ¯ π t , then executes ¯ π t and collects a trajectory ¯ τ t „ P ¯ π t θ ‹ p¨q .
Revised: For the simplicity of presentation, we consider the following general interaction process: For t “ 1 , ¨ ¨ ¨ , T , the learner determines a ¯ π t , then executes ¯ π t and collects a trajectory ¯ τ t „ P ¯ π t θ ‹ p¨q .

Original: We show that, with probability at least 1 ´ δ , the following holds for all t P r T s and θ P Θ :
Revised: We show that, with probability at least 1 ´ δ , the following holds for all t P r T s and θ P Θ :

Original: [Equation]
Revised: [Equation]

Original: 43) where L t is the total log-likelihood (at step t ) defined as
Revised: where L t is the total log-likelihood (at step t ) defined as

Original: [Equation]
Revised: [Equation]

Original: Proposition G.2 is implied by (43) directly: Suppose we choose β ě 2 log N Θ p 1 { T q` 2 log p 1 { δ q` 2 .
Revised: Proposition G.2 is implied by (42) directly: Suppose we choose β ě 2 log N Θ p 1 { T q` 2 log p 1 { δ q` 2 .

Original: On the event (43), by the non-negativity of squared Hellinger distances, we have for all k P r K s and θ P Θ that 
Revised: On the event (42), by the non-negativity of squared Hellinger distances [Equation] we have for all k P r K s and θ P Θ that ÿ

Original: [Equation]
Revised: [Equation]

Original: where D k is the dataset of all histories before the outer loop of Algorithm 1 enters step k .
Revised: where D k is the dataset of all histories before the outer loop of Algorithm 1 enters step k .

Original: Taking max over θ P Θ on the right-hand side, we obtain θ ‹ P Θ k , which gives Proposition G.2(1).
Revised: Taking max over θ P Θ on the right-hand side, we obtain θ ‹ P Θ k , which gives Proposition G.2(1).

Original: Furthermore, for k P r K s and θ P Θ k , (43) implies that
Revised: Furthermore, for k P r K s and θ P Θ k , (42) implies that

Original: [Equation]
Revised: [Equation]

Original: which gives Proposition G.2(2)
Revised: which gives Proposition G.2(2)

Original: In the following, we establish (43).
Revised: In the following, we establish (42).

Original: Let us fix a 1 { T -optimistic covering p r P , Θ 0 q of Θ , such that n : “ | Θ 0 | “ N Θ p 1 { T q .
Revised: Let us fix a 1 { T -optimistic covering p r P , Θ 0 q of Θ , such that n : “ | Θ 0 | “ N Θ p 1 { T q .

Original: We label p r P θ 0 q θ 0 P Θ 0 by r P 1 , ¨ ¨ ¨ , r P n .
Revised: We label p r P θ 0 q θ 0 P Θ 0 by r P 1 , ¨ ¨ ¨ , r P n .

Original: By the definition of optimistic covering, it is clear that for any θ P Θ , there exists i P r n s such that for all π , τ , it holds that r
Revised: By the definition of optimistic covering, it is clear that for any θ P Θ , there exists i P r n s such that for all π , τ , it holds that r

Original: P
Revised: P

Original: [Equation]

Original:  ě P πθ p τ q and } r P πi p¨q ´P
Revised: πi p τ q ě P πθ p τ q and } r P πi p¨q ´

Original:  πθ p¨q} 1 ď 1 { T 2 .
Revised: P πθ p¨q} 1 ď 1 { T 2 .

Original: We say θ is covered by this i P r n s .
Revised: We say θ is covered by this i P r n s .

Original: Then, we consider
Revised: Then, we consider

Original: [Equation]
Revised: [Equation]

Original: By Lemma C.3, the following holds with probability at least 1 ´ δ : for all t P r T s , i P r n s ,t ´ 1 ÿ
Revised: By Lemma C.3, the following holds with probability at least 1 ´ δ : for all t P r T s , i P r n s ,

Original: [Equation]
Revised: [Equation]

Original: “ℓ si ` log p n { δ q ě t ´ 1 ÿ s “´ E s „ exp ˆ ´

Original: [Equation]

Original: ℓ si ˙ȷ , where E s denotes the conditional expectation over all randomness after ¯ π s has been determined.
Revised: where E s denotes the conditional expectation over all randomness after ¯ π s has been determined.

Original: By definition,
Revised: By definition,

Original: [Equation]
Revised: [Equation]

Original: Therefore, for any θ P Θ that is covered by i P r n s , we have ´ 
Revised: Therefore, for any θ P Θ that is covered by i P r n s , we have ´ log E t

Original: [Equation]
Revised: [Equation]

Original: E t
Revised: ˜ÿ

Original: [Equation]
Revised: [Equation]

Original: where the first inequality is due to ´ log x ě 1 ´ x ; in the second inequality we use the definition of
Revised: where the first inequality is due to ´ log x ě 1 ´ x ; in the second inequality we use the definition of

Original: Hellinger distance and Cauchy inequality; the third inequality is because p?
Revised: Hellinger distance and Cauchy inequality; the third inequality is because p?

Original: x ´?
Revised: x ´?

Original: y q 2 ď | x ´ y | for all x, y P R ě 0 ; the last inequality is due to our assumption that θ is covered by i .
Revised: y q 2 ď | x ´ y | for all x, y P R ě 0 ; the last inequality is due to our assumption that θ is covered by i .

Original: Notice that every θ P Θ is covered by some i P r n s , and for such i , ř t ´ 1 s “ 1 ℓ si ď L t p θ ‹ q ´ L t p θ q ; therefore, it holds with probability 1 ´ δ that, for all θ P Θ , t P r T s ,
Revised: Notice that every θ P Θ is covered by some i P r n s , and for such i , ř t ´ 1 s “ 1 ℓ si ď L t p θ ‹ q ´ L t p θ q ; therefore, it holds with probability 1 ´ δ that, for all θ P Θ , t P r T s ,

Original: [Equation]
Revised: [Equation]

Original: Plugging in n “ N Θ p 1 { T q and scaling the above inequality by 2 gives (43).
Revised: Plugging in n “ N Θ p 1 { T q and scaling the above inequality by 2 gives (42).

Original: H E XPLORATIVE E2D, R EWARD -F REE E2D, AND MOPS
Revised: H [Equation] E2D, AND MOPS

Original: In this section, we present the detailed algorithms of E XPLORATIVE E2D, R EWARD -F REE E2D, and MOPS introduced in Section 4.
Revised: In this section, we present the detailed algorithms of E XPLORATIVE E2D, A LL -P OLICY M ODELE STIMATION E2D, and MOPS introduced in Section 4.

Original: We also state the theorems for their sample complexity bounds of learning ε -optimal policy of B-stable PSRs.
Revised: We also state the theorems for their sample complexity bounds of learning ε -optimal policy of B-stable PSRs.

Original: H. E XPLORATIVE E2D ALGORITHM
Revised: H. E XPLORATIVE E2D ALGORITHM

Original: In this section, we provide more details about the E XPLORATIVE E2D algorithm as discussed in
Revised: In this section, we provide more details about the E XPLORATIVE E2D algorithm as discussed in

Original: Section 4.2.
Revised: Section 4.2.

Original: The full algorithm of E XPLORATIVE E2D is given in Algorithm 2, equivalent to Chen et al.
Revised: The full algorithm of E XPLORATIVE E2D is given in Algorithm 2, equivalent to Chen [Equation] [Equation] p

Original:  (2022, Algorithm 2) in the known reward setting ( D 2RL becomes D 2 H since we assumed that the reward is deterministic and known, so that the contribution from reward distance in D 2RL becomes 0 ).
Revised: [Equation] [Equation] et al. (2022, Algorithm 2) in the known reward setting ( D 2RL becomes D 2 H since we assumed that the reward is deterministic and known, so that the contribution from reward distance in D 2RL becomes 0 ).

Original: Chen et al.
Revised: Chen et al.

Original: (2022, Theorem F.1) showed that E XPLORATIVE E2D achieves the following estimation bound.
Revised: (2022, Theorem F.1) showed that E XPLORATIVE E2D achieves the following estimation bound.

Original: Theorem H.1 (Chen et al.
Revised: Theorem H.1 (Chen et al.

Original: (2022), Theorem F.1) .
Revised: (2022), Theorem F.1) .

Original: Given an 1 { T -optimistic cover p r P , Θ 0 q (c.f. Definition C.4) of the model class Θ , Algorithm 2 with η “ 1 { 3 achieves the following with probability at least 1 ´ δ :
Revised: Given an 1 { T -optimistic cover p r P , Θ 0 q (c.f. Definition C.4) of the model class Θ , Algorithm 2 with η “ 1 { 3 achieves the following with probability at least 1 ´ δ :

Original: [Equation]
Revised: [Equation]

Original: where edec γ is the Explorative DEC as defined in (9). [Equation] [Equation] [Equation]
Revised: where edec γ is the Explorative DEC as defined in Section 4.2.

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation] [Equation] [Equation] As we can see from the theorem above, as long as we can bound edec γ p Θ q , we can get a sample complexity bound for the E XPLORATIVE E2D
Revised: As we can see from the theorem above, as long as we can bound edec γ p Θ q , we can get a sample complexity bound for the E XPLORATIVE E2D

Original: algorithm.
Revised: algorithm.

Original: This gives Theorem 10 in the main text, which we restate as below.
Revised: This gives Theorem 10 in the main text, which we restate as below.

Original: Theorem H.2 (Restatement of Theorem 10) .
Revised: Theorem H.2 (Restatement of Theorem 10) .

Original: Suppose Θ is a PSR class with the same core test sets t
Revised: Suppose Θ is a PSR class with the same core test sets t

Original: U
Revised: U

Original: h u h
Revised: h u h

Original: Pr H s , and
Revised: Pr H s , and

Original: each θ P Θ admits a B-representation that is Λ B -stable
Revised: each θ P Θ admits a B-representation that is Λ B -stable

Original: (c.f. Definition 4) or weakly Λ B -stable (c.f.
Revised: (c.f. Definition 4) or weakly Λ B -stable (c.f.

Original: Definition D.4), and has PSR rank d PSR ď d .
Revised: Definition D.4), and has PSR rank d PSR ď d .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Therefore, we can choose a suitable parameter γ and an 1 { T -optimistic cover p r P , Θ 0 q , such that with probability at least 1 ´ δ , Algorithm 2 outputs a policy p π out P ∆ p Π q such that V ‹ ´ V θ ‹ p p π out q ď ε , as long as the number of episodes
Revised: Therefore, we can choose a suitable parameter γ and an 1 { T -optimistic cover p r P , Θ 0 q , such that with probability at least 1 ´ δ , Algorithm 2 outputs a policy p π out P ∆ p Π q such that V ‹ ´ V θ ‹ p p π out q ď ε , as long as the number of episodes

Original: [Equation]
Revised: [Equation]

Original: The proof of Theorem H.2 and hence Theorem 10 is contained in Appendix I.2.
Revised: The proof of Theorem H.2 and hence Theorem 10 is contained in Appendix I.2.

Original: H. R EWARD -F REE E2D FOR REWARD - FREE LEARNING
Revised: H. A LL -P OLICY M ODEL -E STIMATION E2D FOR MODEL - ESTIMATION

Original: In this section, we provide more details about reward-free learning in PSRs as discussed in Section 4.2.
Revised: In this section, we provide more details about model-estimation learning in PSRs as discussed in Section 4.2.

Original: In reward-free RL (Jin et al., 2020b), the goal is to optimally explore the environment without observing reward information, so that after the exploration phase, a near-optimal policy of any given reward can be computed using the collected trajectory data alone without further interacting with the environment.
Revised: In reward-free RL (Jin et al., 2020b), the goal is to optimally explore the environment without observing reward information, so that after the exploration phase, a near-optimal policy of any given reward can be computed using the collected trajectory data alone without further interacting with the environment.

Original: Chen et al.
Revised: Chen et al.

Original: (2022) developed R EWARD -F REE E2D as a unified algorithm for reward-free learning in RL, and showed that its sample complexity scales with a complexity measure named Reward-Free
Revised: (2022) developed A LL -P OLICY M ODEL -E STIMATION E2D as a unified algorithm for reward-free/model-estimation learning in RL, and showed that its sample complexity scales with a complexity measure named All-policy Model-Estimation DEC (AMDEC).

Original: DEC (RFDEC).
Revised: The AMDEC is defined

Original: To define RFDEC, we still use Θ to denote the model class of transition dynamics, and use R to denote the class of (deterministic) reward functions. For θ P Θ and R P R , we denote V θ,R : ∆ pΠ q Ñ R to be the value function of p θ, R q , and π θ,R P ∆ p Π q to be the optimal policy under the model p θ, R q . Then the RFDEC rfdec γ is defined as
Revised: Set p p t exp , µ

Original: [Equation]
Revised: [Equation]

Original: [Equation] The R EWARD -F REE E2D algorithm (Algorithm 3) for a PSR class Θ is given as follows: In each episode t P r T s , we maintain a distribution µ t P ∆ p Θ 0 q over an 1 { T -optimistic cover p r P , Θ 0 q of Θ (c.f. Definition C.4), which we use to compute an exploration policy distribution p t exp by minimizing the following risk:
Revised: out q “ arg

Original: [Equation]
Revised: [Equation]

Original: 4: for do 5: Set p t exp “ arg min
Revised: , µ

Original: [Equation]
Revised: [Equation]

Original: P ∆ p Π q sup R P R p V µ t rf ,γ p p exp

Original: R q , where
Revised: q , where

Original: [Equation]
Revised: [Equation]

Original: D 2H ´
Revised: D 2H ´

Original: [Equation]
Revised: [Equation]

Original: ¯ı .
Revised: ¯ı .

Original: Computeµ
Revised: Compute

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: ´
Revised: ´

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: ¯ .
Revised: ¯ .

Original: Planning phase 9: Input: R P R 10: for t “ do 11:
Revised: 7: Compute µ out “ 1 T ř Tt [Equation] [Table] [Equation]

Original: Compute p
Revised: [Table] P

Original: [Equation]
Revised: [Table]

Original: out p 
Revised: p Θ q .

Revised: 8: Output: p θ P Θ sup π P Π E ¯ θ „ µ out “

Original: [Equation]
Revised: [Equation]

Original:  : “ arg min
Revised: as amdec γ p Θ q : “ sup p µ P ∆ p Θ q amdec γ p Θ , p µ q , where amdec γ p Θ , p µ q : “ inf p exp P

Original: [Equation]
Revised: [Equation]

Original: 12: Output: p π out p R q“
Revised: p Π q

Original: [Equation]
Revised: [Equation]

Revised: µ out P ∆ p

Revised: [Equation]

Original: T ř Tt “ 1 p t out p R q .
Revised: q sup [Equation] P [Equation]

Revised: sup

Revised: [Equation]

Revised: P

Revised: [Equation]

Revised: E

Revised: [Equation]

Revised: out “

Revised: [Equation]

Revised: D

Revised: [Equation]

Revised: ¯ı .

Revised: The A LL -P OLICY M ODEL -E STIMATION E2D algorithm

Revised: (Algorithm 3) for a PSR class Θ is given as follows:

Revised: In each episode t P r T s , we maintain a distribution µ t P ∆ p Θ 0 q over an 1 { T -optimistic cover p r P , Θ 0 q of Θ (c.f. Definition C.4), which we use to compute an exploration policy distribution p t exp by minimizing the following risk:

Revised: [Equation]

Original: Then, we execute policy π t „ p t exp , collect trajectory τ t , and update the model distribution using the same Tempered Aggregation scheme as in E XPLORATIVE E2D.
Revised: Then, we execute policy π t „ p t exp , collect trajectory τ t , and update the model distribution using the same Tempered Aggregation scheme as in E XPLORATIVE E2D.

Original: After T episodes, wereceive the reward R and output the average policy p π outp R q : “ 1 T ř Tt “ 1 p t out  p R q where p t out p R q solves
Revised: After T episodes, we output the emipirical model p θ by computing µ out “ 1 T ř Tt “ 1 µ t out P ∆ p Θ q and then projecting it into Θ , i.e.

Original: [Equation]
Revised: [Equation]

Original: Chen et al.
Revised: Chen et al.

Original: (2022, Theorem G.1) show that the output policy p π out p R q of R EWARD -F REE E2D has a sub-optimality gap that scales as rfdec γ .
Revised: (2022, Theorem H.2) show that the output model p θ of A LL -P OLICY M ODELE STIMATION E2D has an estimation error (measured in terms of the TV distance) that scales as amdec γ .

Original: Theorem H.3.
Revised: Theorem H.3.

Original: Given an 1 { T -optimistic cover p r P , Θ 0 q (c.f. Definition C.4) of the class of transition dynamics Θ , Algorithm 3 with η “ 1 { 2 achieves the following with probability at least 1 ´ δ :
Revised: Given an 1 { T -optimistic cover p r P , Θ 0 q (c.f. Definition C.4) of the class of transition dynamics Θ , Algorithm 3 with η “ 1 { 2 achieves the following with probability at least 1 ´ δ :

Original: [Equation]
Revised: [Equation]

Original: where rfdec γ is the Reward-Free DEC as defined in (44).
Revised: where amdec γ is the All-policy Model-Estimation DEC as defined in (43).

Original: We provide a sharp bound on the RFDEC for B-stable PSRs, which implies that R EWARD -F REE
Revised: We provide a sharp bound on the AMEDEC for B-stable PSRs, which implies that A LL -P OLICY

Original:  E2D can also learn them sample-efficient efficiently in a reward-free manner.
Revised: M ODEL -E STIMATION E2D can also learn them sample-efficient efficiently in a model-estimation manner.

Original: Theorem H.4.
Revised: Theorem H.4.

Original: Suppose Θ is a PSR class with the same core test sets t
Revised: Suppose Θ is a PSR class with the same core test sets t

Original: U
Revised: U

Original: h u h
Revised: h u h

Original: Pr H s , and
Revised: Pr H s , and

Original: each θ P Θ admits a B-representation that is Λ B -stable
Revised: each θ P Θ admits a B-representation that is Λ B -stable

Original: (c.f. Definition 4) or weakly Λ B -stable (c.f.
Revised: (c.f. Definition 4) or weakly Λ B -stable (c.f.

Original: Definition D.4), and has PSR rank d PSR ď d .
Revised: Definition D.4), and has PSR rank d PSR ď d .

Original: Then
Revised: Then

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore, we can choose a suitable parameter γ and an 1 { T -optimistic cover p r P , Θ 0 q , such that with probability at least 1 ´ δ , Algorithm 3 outputs a policy p π out P ∆ p Π q such that supR t V θ ‹ ,R p π θ ‹ ,R q ´ V θ ‹ ,R p p π out p R qqu ď ε , as long as the number of episodes
Revised: Therefore, we can choose a suitable parameter γ and an 1 { T -optimistic cover p r P , Θ 0 q , such that with probability at least 1 ´ δ , Algorithm 3 outputs a model p θ P Θ such that sup π D TV [Equation] as long as the number of episodes

Original: [Equation]
Revised: [Equation]

Original: The proof of Theorem H.4 is contained in Appendix I.3.
Revised: The proof of Theorem H.4 is contained in Appendix I.3.

Original: H. M ODEL - BASED OPTIMISTIC POSTERIOR SAMPLING (MOPS)
Revised: H. M ODEL - BASED OPTIMISTIC POSTERIOR SAMPLING (MOPS)

Original: In this section, we provide more details about the MOPS algorithm as discussed in Section 4.3.
Revised: In this section, we provide more details about the MOPS algorithm as discussed in Section 4.3.

Original: We consider the following version of the MOPS algorithm of Agarwal & Zhang (2022);
Revised: We consider the following version of the MOPS algorithm of Agarwal & Zhang (2022);

Original: Chen et al.
Revised: Chen et al.

Original: Similar to E XPLORATIVE E2D, MOPS also maintains a posterior µ t P ∆ p Θ 0 q over an 1 { T optimistic cover p r P , Θ 0 q , initialized at a suitable prior µ 1 .
Revised: Similar to E XPLORATIVE E2D, MOPS also maintains a posterior µ t P ∆ p Θ 0 q over an 1 { T optimistic cover p r P , Θ 0 q , initialized at a suitable prior µ 1 .

Original: The exploration policy in the t -th episode is obtained by posterior sampling: π t “ π θ t ˝
Revised: The exploration policy in the t -th episode is obtained by posterior sampling: π t “ π θ t ˝

Original: h t Unif p
Revised: h t Unif p

Original: A q ˝ h t ` 1 Unif p U A,h t ` 1 q , where θ t „ µ t and h t „ Unif pt 0 , 1 , . .
Revised: A q ˝ h t ` 1 Unif p U A,h t ` 1 q , where θ t „ µ t and h t „ Unif pt 0 , 1 , . .

Original: , H ´ 1 uq .
Revised: , H ´ 1 uq .

Original: After executing π t and observing τ t , the algorithm updates the posterior as
Revised: After executing π t and observing τ t , the algorithm updates the posterior as

Original: [Equation]
Revised: [Equation]

Original: Finally, the algorithm output p π out : “ 1 T ř Tt “ 1 p out p µ t q , where p out p µ t q P ∆ p Π q is defined as
Revised: Finally, the algorithm output p π out : “ 1 T ř Tt “ 1 p out p µ t q , where p out p µ t q P ∆ p Π q is defined as

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: We further consider the following Explorative PSC (EPSC), which is a modification of the PSC proposed in Chen et al.
Revised: We further consider the following Explorative PSC (EPSC), which is a modification of the PSC proposed in Chen et al.

Original: (2022, Definition 4):
Revised: (2022, Definition 4):

Original: [Equation]
Revised: [Equation]

Original: where ∆ 0 p Θ q is the set of all finitely supported distributions on Θ , π exp is defined as
Revised: where ∆ 0 p Θ q is the set of all finitely supported distributions on Θ , π exp is defined as

Original: [Equation]
Revised: [Equation]

Original: and we abbreviate π „ p out p µ q to π „ µ .
Revised: and we abbreviate π „ p out p µ q to π „ µ .

Original: Adapting the proof for the MOPS algorithm in Chen et al.
Revised: Adapting the proof for the MOPS algorithm in Chen et al.

Original: (2022, Corollary D.3 & Theorem D.1) to the explorative version, we can show that the output policy p π out of MOPS has a sub-optimality gap that scales as psc est .
Revised: (2022, Corollary D.3 & Theorem D.1) to the explorative version, we can show that the output policy p π out of MOPS has a sub-optimality gap that scales as psc est .

Original: Theorem H.5.
Revised: Theorem H.5.

Original: Given an 1 { T -optimistic cover p r P , Θ 0 q (c.f. Definition C.4) of the class of PSR models Θ , Algorithm 4 with η “ 1 { 6 and γ ě 1 achieves the following with probability at least 1 ´ δ :
Revised: Given an 1 { T -optimistic cover p r P , Θ 0 q (c.f. Definition C.4) of the class of PSR models Θ , Algorithm 4 with η “ 1 { 6 and γ ě 1 achieves the following with probability at least 1 ´ δ :

Original: [Equation]
Revised: [Equation]

Original: where psc est γ is the Explorative PSC as defined in (47).
Revised: where psc est γ is the Explorative PSC as defined in (46).

Original: We provide a sharp bound on the EPSC for B-stable PSRs, which implies that MOPS can also learn them sample-efficient efficiently.
Revised: We provide a sharp bound on the EPSC for B-stable PSRs, which implies that MOPS can also learn them sample-efficient efficiently.

Original: Theorem H.6.
Revised: Theorem H.6.

Original: Suppose Θ is a PSR class with the same core test sets t
Revised: Suppose Θ is a PSR class with the same core test sets t

Original: U
Revised: U

Original: h u h
Revised: h u h

Original: Pr H s , and
Revised: Pr H s , and

Original: each θ P Θ admits a B-representation that is Λ B -stable
Revised: each θ P Θ admits a B-representation that is Λ B -stable

Original: (c.f. Definition 4) or weakly Λ B -stable (c.f.
Revised: (c.f. Definition 4) or weakly Λ B -stable (c.f.

Original: Definition D.4), and the ground truth model θ ‹ has PSR rank at most d .
Revised: Definition D.4), and the ground truth model θ ‹ has PSR rank at most d .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original:  Therefore, we can choose a suitable parameter γ and an 1 { T -optimistic cover p r P , Θ 0 q , such that with probability at least 1 ´ δ , Algorithm 4 outputs a policy p π out P ∆ p Π q such thatsup R t Vθ ‹,R p π θ ‹ ,R q ´ V θ ‹,R p p π out p R qqu ď ε , as long as the number of episodes
Revised: A H 2 { γ. Therefore, we can choose a suitable parameter γ and an 1 { T -optimistic cover p r P , Θ 0 q , such that with probability at least 1 ´ δ , Algorithm 4 outputs a policy p π out P ∆ p Π q such that V ‹ ´ V θ ‹ p p π out q ď ε , as long as the number of episodes

Original: [Equation]
Revised: [Equation]

Original: The proof of Theorem H.6 is contained in Appendix I.2.
Revised: The proof of Theorem H.6 is contained in Appendix I.2.

Original: We remark here that EPSC is an upper bound of EDEC (c.f. Eq.
Revised: We remark here that EPSC provides an upper bound of EDEC (c.f. Eq.

Original: So Theorem H.2 (and hence Theorem 10) directly follows from Theorem H.6.` θ q
Revised: So Theorem H.2 (and hence Theorem 10) directly follows from Theorem H.6.

Original: I P ROOFS FOR A PPENDIX H
Revised: I P ROOFS FOR A PPENDIX H

Original: For the clarity of discussion, we introduce the following notation in this section: for policy π , we denote φ h to be a policy modification such that
Revised: For the clarity of discussion, we introduce the following notation in this section: for policy π , we denote φ h to be a policy modification such that

Original: [Equation]
Revised: [Equation]

Original: Again, here φ h ˛ π means that we follow π for the first h ´ 1 steps, takes Unif p
Revised: Again, here φ h ˛ π means that we follow π for the first h ´ 1 steps, takes Unif p

Original: A q at step h , takes an action sequence sampled from Unif p U A,h ` 1 q at step h ` 1 , and behaves arbitrarily afterwards.
Revised: A q at step h , takes an action sequence sampled from Unif p U A,h ` 1 q at step h ` 1 , and behaves arbitrarily afterwards.

Original: Such definition agrees with (48).
Revised: Such definition agrees with (47).

Original: We further define the φ policy modification as
Revised: We further define the φ policy modification as

Original: [Equation]
Revised: [Equation]

Original: We call φ ˛ π the exploration policy of π .
Revised: We call φ ˛ π the exploration policy of π .

Original: [Equation]
Revised: I. P ROOF OF T HEOREM

Original: H.
Revised: H.

Original: To prove Theorem H.6, due to Theorem H.5, we only need to bound the coefficients psc est γ p Θ , θ ‹ q .
Revised: To prove Theorem H.6, due to Theorem H.5, we only need to bound the coefficients psc est γ p Θ , θ ‹ q .

Original: By its definition, we have
Revised: By its definition, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: We then invoke the following error decorrelation result, which follows from the decoupling argument in Appendix E.2 and Proposition F.1.
Revised: We then invoke the following error decorrelation result, which follows from the decoupling argument in Appendix E.2 and Proposition F.1.

Original: Proposition I.1 (Error decorrelation) .
Revised: Proposition I.1 (Error decorrelation) .

Original: Under the condition of Theorem H.2 (the same condition as
Revised: Under the condition of Theorem H.2 (the same condition as

Original: Theorem H.6) , for any µ P ∆ 0 p Θ q and any reference model ¯ θ  , we have
Revised: Theorem H.6), for any µ P ∆ 0 p Θ q and any reference model ¯ θ P Θ , we have

Original: [Equation]
Revised: [Equation]

Original: where  φ ˛ π defined in (48) is the exploration policy of π .
Revised: where d ¯ θ is the PSR rank of ¯ θ , φ ˛ π defined in (47) is the exploration policy of π .

Original: Combining Proposition I.1 with (49) immediately gives the desired upper bound of psc est γ p Θ , θ ‹ q , and thus completes the proof of Theorem H.6.
Revised: Combining Proposition I.1 with (48) immediately gives the desired upper bound of psc est γ p Θ , θ ‹ q , and thus completes the proof of Theorem H.6.

Original: We next turn to prove the Proposition I.1 above.
Revised: We next turn to prove the Proposition I.1 above.

Revised: We consider the following generalized version of

Original: Proof of Proposition I.1.
Revised: Proposition I.1.

Revised: Proposition I.2 (Generalized error decorrelation) .

Original: {+Under the condition of Theorem H.4, for any ¯ θ P Θ ν P ∆ 
+}
Revised: {+Under the condition of Theorem H.4, for any ¯ θ P Θ ν P ∆ 
+}

Revised: p Θ ˆ Π q , we have

Revised: [Equation]

Revised: where φ ˛ π defined in (47) is the exploration policy of π .

Revised: Proof of Proposition I.2.

Original: In the following, we fix a ¯ θ P Θ and abbreviate E “ E ¯ θ , q “ q ¯ θ .
Revised: In the following, we fix a ¯ θ P Θ and abbreviate E “ E ¯ θ , q “ q ¯ θ .

Original: Then, by
Revised: Then, by

Original: Proposition F.1, we have
Revised: Proposition F.1, we have

Original: [Equation]
Revised: [Equation]

Original: Note that for the term E θ „ µ “E
Revised: Note that for the term E θ „ µ “

Original: [Equation]

Original: ‰ , we have
Revised: E θ,‰ , we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We next consider the case for h P r H s , and upper bound the corresponding terms in the righthand-side of (50) using the decoupling argument introduced in Appendix E.2, restated as follows for convenience.
Revised: We next consider the case for h P r H s , and upper bound the corresponding terms in the righthand-side of (49) using the decoupling argument introduced in Appendix E.2, restated as follows for convenience.

Original: Proposition E.4 (Decoupling argument) .
Revised: Proposition E.6 (Decoupling argument) .

Original: Suppose we have vectors and functions
Revised: Suppose we have vectors and functions

Original: [Equation]
Revised: [Equation]

Original: where Θ , I are arbitrary abstract index sets, with functions f θ given by
Revised: where Θ , I are arbitrary abstract index sets, with functions f θ given by

Original: [Equation]
Revised: [Equation]

Original: where t y θ,j,r u p θ,j,r qP Θ ˆr J sˆ R Ă R n is a family of bounded vectors in R n .
Revised: where t y θ,j,r u p θ,j,r qP Θ ˆr J sˆ R Ă R n is a family of bounded vectors in R n .

Original: Then for any distribution µ over Θ and probability family t q θ u θ P Θ Ă ∆ p I q ,
Revised: Then for any distribution µ over Θ and probability family t q θ u θ P Θ Ă ∆ p I q ,

Original: [Equation]
Revised: [Equation]

Original: where d X is the dimension of the subspace of R n spanned by p x i q i P I .
Revised: where d X is the dimension of the subspace of R n spanned by p x i q i P I .

Original: We have the following three preparation steps to apply Proposition E.4: 1.
Revised: We have the following three preparation steps to apply Proposition E.6: 1.

Original: Recall that E “ E ¯ θ is defined in Proposition F.1.
Revised: Recall that E “ E ¯ θ is defined in Proposition F.1.

Original: Let us define
Revised: Let us define

Original: [Equation]
Revised: [Equation]

Original: where t τ jh : H “ p o h , a h , ¨ ¨ ¨ , o H , a H qu n y j “ 1 is an ordering of all possible τ h : H (and hence n y “ p OA q H ´ h ` 1 ), π is any policy (that starts at step h ).
Revised: where t τ jh : H “ p o jh , a jh , ¨ ¨ ¨ , o jH , a jH qu n y j “ 1 is an ordering of all possible τ h : H (and hence n y “ p OA q H ´ h ` 1 ), π is any policy (that starts at step h ).

Original: We then definef
Revised: We then define

Original: [Equation]

Original: p

Original: [Equation]
Revised: [Equation]

Original: Then it follows from definition (c.f. Proposition F.1)
Revised: Then it follows from definition (c.f. Proposition F.1)

Original: that E θ,h p τ h ´ 1 q “ f θ p q p τ h ´ 1 qq .
Revised: that E θ,h p τ h ´ 1 q “ f θ p q p τ h ´ 1 qq .

Original: We define x i “ q ‹ p τ ih ´
Revised: We define x i “ q ‹ p τ ih ´

Original: 1 q P R | U h | for i P I “ p O ˆ A q h ´ 1 where t τ ih ´
Revised: 1 q P R | U h | for i P I “ p O ˆ A q h ´ 1 where t τ ih ´

Original: 1 u i P
Revised: 1 u i P

Original: I is an ordering of all possible τ h ´ 1 P p
Revised: I is an ordering of all possible τ h ´ 1 P p

Original: O ˆ
Revised: O ˆ

Original: A q h ´ 1 .
Revised: A q h ´ 1 .

Original: Then by our definition of PSR rank (c.f. Definition 3), the subspace of R | U h | spanned by t x i u i P I has dimension less than or equal to d ¯ θ .
Revised: Then by our definition of PSR rank (c.f. Definition 3), the subspace of R | U h | spanned by t x i u i P I has dimension less than or equal to d ¯ θ .

Original: We take q θ P ∆ p
Revised: We take q θ P ∆ p

Original: I q as
Revised: I q as

Original: [Equation]
Revised: [Equation]

Original: Therefore, applying Proposition E.4 to function family t f θ u θ P Θ , vector family t x i u i P I , and distribution family t q θ u θ P Θ gives
Revised: Therefore, applying Proposition E.6 to function family t f θ u θ P Θ , vector family t x i u i P I , and distribution family t q θ u θ P Θ gives

Original: [Equation]
Revised: [Equation]

Original: Combining Eq.
Revised: Combining Eq.

Original: (51), (53), and (50) yields
Revised: (50), (52), and (49) yields

Original: [Equation] ıı ď gffe p 
Revised: E p θ,π q„ µ “

Original: Hd ¯ θ ` 1 q ˜
Revised: D TV ` P πθ , P

Original: E θ „ µ ” E 2 θ,ı `
Revised: [Table]

Original: H ÿ h “
Revised: ÿ [Table]

Original: E θ,θ 1 „ µ ”
Revised: [Table]

Original: E ¯ θ,π θ” E 2 θ,h p τ h ´ 1 q ıı¸ ď gffe p
Revised: ‰ ď c [Table] [Table]

Original: Hd ¯ θ ` 1 q ˜
Revised: ı [Table] c [Table]

Original: E θ,θ 1 „ µ « H ´ 1 ÿ h “12Λ 2 B AU
Revised: [Table] [Table] [Table] [Table]

Original: A ¨ D 2H ` P φ h ˛ π θ 1 θ , P φ h ˛ π θ 1 ¯ θ ˘ff¸ “ b 12 p Hd ¯ θ ` 1 q H ¨ Λ 2 B AU
Revised: ıı¸ ď

Original: A E θ,θ 1 „ µ “
Revised: [Table]

Original: D 2H ` P φ
Revised: ˘ff¸ “ b

Original: [Table]
Revised: [Table]

Original: π θ 1 θ , P
Revised: Hd

Original: [Table]
Revised: [Table]

Original: ˛ π θ 1 ¯ θ ˘‰ , where the third inequality is due to Cauchy-Schwarz inequality, and the fourth inequality is due to
Revised: , where the third inequality is due to Cauchy-Schwarz inequality, and the fourth inequality is due to

Original: Proposition F.2.
Revised: Proposition F.2.

Original: This completes the proof of Proposition I.1.
Revised: This completes the proof of Proposition I.1.

Original: I. P ROOF OF T HEOREM H.2 (T HEOREM 10)
Revised: I. P ROOF OF T HEOREM H.2 (T HEOREM 10)

Original: According to Theorem H.1, in order to prove Theorem H.2 (Theorem 10), we only need to bound the coefficients edec γ p Θ q for γ ą 0 .
Revised: According to Theorem H.1, in order to prove Theorem H.2 (Theorem 10), we only need to bound the coefficients edec γ p Θ q for γ ą 0 .

Original: In the following, we bound edec by psc est using the idea of Chen et al.
Revised: In the following, we bound edec by psc est using the idea of Chen et al.

Original: (2022, Proposition 6).
Revised: (2022, Proposition 6).

Original: Recall that edec is defined in (9).
Revised: Recall that edec is defined in (4.2).

Original: By strong duality (c.f. Theorem C.2), we have
Revised: By strong duality (c.f. Theorem C.2), we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that | V θ pπ q ´ V ¯ θ p π q| ď D TV [Equation] D [Equation]
Revised: Note that | V θ p

Original: P

Original: [Equation]
Revised: [Equation]

Original: , P
Revised: , P

Original: [Equation]
Revised: [Equation]

Original: θ ˘ .
Revised: θ ˘ .

Original: Therefore, we can take p out “ p µ , where p µ is defined as p µ p π q “ µ pt θ : π θ “ π uq .
Revised: Therefore, we can take p out “ p µ , where p µ is defined as p µ p π q “ µ pt θ : π θ “ π uq .

Original: Then for a fixed α P p 0 , 1 q , we have
Revised: Then for a fixed α P p [Equation] 1 q [Equation]

Revised: we have

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: where the equality is due to our choice of p µ :
Revised: where the equality is due to our choice of p µ :

Original: [Equation]
Revised: [Equation]

Original: and the last inequality is due to AM-GM inequality.

Original: Therefore, we can take p exp “ αp µ ` p 1 ´ α q p e P ∆ p
Revised: Therefore, we can take p exp “ αp µ ` p 1 ´ α q p e P ∆ p

Original: Π q , where p e is given by p e p π q “ µ pt θ : φ ˛ π θ “ π uq , 21 and using this choice of p exp and p out in Eq.
Revised: Π q , where p e is given by p e p π q “ µ pt θ : φ ˛ π θ “ π uq , 22 and using this choice of p exp and p out in Eq.

Original: (54) and using Eq.
Revised: (53) and using Eq.

Original: (55), we get
Revised: , we get

Original: [Equation]
Revised: [Equation]

Original: Recall that psc est γ has been bounded in Theorem H.6.
Revised: Recall that psc est γ has been bounded in Theorem H.6.

Original: Taking α “ 3 { 4 yields edec γ p Θ , µ q ď p 8Λ 2 B dAU A H 2 ` 1 q{ γ .
Revised: Taking α “ 3 { 4 yields edec γ p Θ , µ q ď p 8Λ 2 B dAU A H 2 ` 1 q{ γ .

Original: This completes the proof of Theorem H.2.
Revised: This completes the proof of Theorem H.2.

Original: I. P ROOF OF T HEOREM H. 
Revised: I. P ROOF OF T HEOREM H.

Original: To prove Theorem H.4, we need to use the following generalized version of Proposition I.1.

Original: Proposition I.2 (Generalized error decorrelation) .

Original: Under the condition of Theorem H.4, for any ν P ∆ 0 p Θ ˆ Π q , we have

Original: [Equation]

Original: where φ ˛ π defined in (48) is the exploration policy of π .

Original: The generalization is that, in Proposition I.2 the expectation is with respect to p θ, π q „ ν for general correlated distribution ν , whereas in Proposition I.1 the expectation is with respect to θ „ µ and π “ π θ .

Original: The proof of Proposition I.2 is almost the same as the proof of Proposition I.1: we only need to change the distribution family in (52) to be q θ

Original: [Equation]

Original: E π „ ν p¨|

Original: [Equation]

Original: q [Equation] Hence, we omit the proof of Proposition I.2. To prove Theorem H.4, by Theorem H.5, we only need to bound rfdec γ p Θ  q for γ ą 0 .
Revised: To prove Theorem H.4, due to Theorem H.3, we only need to bound the coefficients amdec γ p Θ , p µ q for all p µ P ∆ p Θ q .

Original: By strong duality (c.f. Theorem C.2), we have [Equation]
Revised: By strong duality (c.f. Theorem C.2), we have amdec

Original: Note that if we take p o “ p µ P ∆ p Π q where

Original: [Equation]

Original: then we have

Original: [Equation]
Revised: [Equation]

Original: where the second equality is due to that π „ p µ  and  πθ,R with θ „ µ follow the same distribution. Therefore [Equation] we have
Revised: where the first inequality is because we take µ out “ p µ in inf µ out , and the second inequality is because we can take p exp P ∆ p Π q corresponds to φ ˛ π with π „ ν .

Original: [Equation]

Original: Next

Original: [Equation]

Original: we define

Original: [Equation]

Original: [Equation]

Original: so that rfdec γ p Θ q “ sup µ P ∆ p Θ q rfdec γ p Θ , µ q .

Original: By the upper bound of p V µ rf ,γ and by strong duality (c.f. Theorem C.2)

Original: [Equation]

Original: we have

Original: [Equation] Applying Proposition I.2 givesthat, for all ν P ∆ 0 p Θ ˆ Π q and ¯ θ P Θ : [Equation] Therefore
Revised: Applying Proposition I.2 gives

Original: [Equation]

Original: we can upper bound rfdec γ as

Original: [Equation]
Revised: [Equation]

Original: where the second inequality is because we can take p

Original: [Equation]

Original: P ∆ p

Original: Π q

Original: [Equation]

Original: p

Original: [Equation]

Original: p π q

Original: [Equation]

Original: This completes the proof of Theorem H.4.
Revised: and thus the proof of Theorem H.4.