Original: [Equation]
Revised: [Equation]

Original: π

Original: [Equation]

Original: and by Lemma F [Equation] 3 it holds that 
Revised: and by Lemma F.3 it holds that ›››

Original: [Equation]
Revised: [Equation]

Original: where the equality is due to U A,H only containing the null action sequence.
Revised: where the equality is due to U A,H only containing the null action sequence.

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: and applying Lemma C.1 completes the proof of the case h “ H .
Revised: and applying Lemma C.1 completes the proof of the case h “ H .

Original: The case h “ 0 is directly implied by Lemma F.3:
Revised: The case h “ 0 is directly implied by Lemma F.3:

Original: [Equation]
Revised: [Equation]

Original: Combining all these cases finishes the proof of Proposition F.2.
Revised: Combining all these cases finishes the proof of Proposition F.2.

Original: We next prove Lemma F.3 and F.4 that were used in the proof of Proposition F.2.
Revised: We next prove Lemma F.3 and F.4 that were used in the proof of Proposition F.2.

Original: Proof of Lemma F.3.
Revised: Proof of Lemma F.3.

Original: By the weak B-stability as in Definition D.

Revised: By the weak B-stability as in Definition D.

Original: (B-stability is also sufficient, see
Revised: (B-stability is also sufficient, see

Original: Eq.
Revised: Eq.

Original: [Equation] we have
Revised: (19)), we have

Original: [Equation]
Revised: [Equation]

Original: where }¨} Π is defined in Definition D
Revised: where }¨} Π is defined in Definition D.3.

Original: [Equation]

Original: By the definition of q θ p
Revised: By the definition of q θ p

Original: τ
Revised: τ

Original: h
Revised: h

Original: ´ 1 q , for t h “ p o , a q P U h , we have
Revised: ´ 1 q , for t h “ p o , a q P U h , we have

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence, we have

Original: [Equation]
Revised: [Equation]

Original:  we have
Revised: where P πθ p T 1 | τ h ´ 1 q stands for the probability that some test in T 1 is observed under ¯ θ, π conditional on τ h ´ 1 . Similarly, we have › ›› q ¯ θ p τ h ´ 1 q › ›› Π ď 1 . Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: ÿ
Revised: ´ b

Original: [Equation]
Revised: [Equation]

Original: where P πθ pT 1 | τ h ´ 1 q stands for the probability that some test in T 1 is observed under ¯ θ, π conditional on τ h ´ 1 . Similarly, we have ››› q ¯ θ p τ h ´ 1 q ››› Π ď 1 . Therefore, we have
Revised: p

Original: [Equation] 
Revised: [Equation] [Equation] [Equation] [Equation]

Revised: ˇˇˇ r

Revised: [Equation]

Revised: o |

Revised: [Equation]

Revised: p

Original: [Equation]
Revised: [Equation]

Revised: p

Original: [Equation]
Revised: [Equation]

Revised: a

Original:  where in (i) we include those o such that p o , a q may not belong to U h ` 1 into summation, (ii) is due to the definition of P p¨|
Revised: [Equation] ˇˇˇ [Equation] where in (i) we include those o such that p o , a q may not belong to U h ` 1 into summation, (ii) is due to the definition of P p¨|

Original: τ h ´ 1 , do p a qq , and (iii) follows from importance sampling (35).
Revised: τ h ´ 1 , do p a qq , and (iii) follows from importance sampling (35).

Original: This completes the proof of Lemma F.3.
Revised: This completes the proof of Lemma F.3.

Original: Proof of Lemma F.4.
Revised: Proof of Lemma F.4.

Original: Similar to the proof of Lemma F.3, we only need to work under the weak
Revised: Similar to the proof of Lemma F.3, we only need to work under the weak

Original: B-stability condition.
Revised: B-stability condition.

Original: By Corollary D.2, for t h ` 1 “ p o , a q P U h ` 1 , it holds that 
Revised: By Corollary D.2, for t h ` 1 “ p o , a q P U h ` 1 , it holds that “

Original: [Equation]
Revised: [Equation]

Original: q , and hence ›
Revised: and hence ››

Original: [Equation]
Revised: [Equation]

Original: where P πθ p T 1 | τ h ´ 1 , o, a q stands for the probability that some test in T 1 is observed under θ, π conditional on observing τ h “ p τ h ´ 1 , o, a q .
Revised: where P πθ p T 1 | τ h ´ 1 , o, a q stands for the probability that some test in T 1 is observed under θ, π conditional on observing τ h “ p τ h ´ 1 , o, a q .

Original: Similarly, we have ››› B ¯ θh po, a q q
Revised: Similarly, we have ››› B ¯ θh p

Original: [Equation]
Revised: [Equation]

Original: q .

Original: Therefore, by the weak B-stability as in Definition D.4 and combining with the inequalities above
Revised: Therefore, by the weak B-stability as in Definition D.4 and combining with the inequalities above

Original: [Equation]
Revised: [Equation]

Original: it holds that
Revised: it holds that

Original: [Equation]
Revised: [Equation]

Original: Hence, we have
Revised: Hence, we have

Original: [Equation]
Revised: [Equation]

Original: p¨
Revised: [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: τ
Revised: [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: ¯
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original:  where (i) is due to the fact that max π P ∆ p
Revised: P Unif p A q˝ Unif p U A,h ` 1 q ¯ θ p¨| τ h ´ 1 q ¯ , where (i) is due to the fact that max π P ∆ p

Original: A q ř a P
Revised: A q ř a P

Original: A π p a q x p a q ď `ř a x p a q˘ 1 { 2 , (ii) is due to CauchySchwarz inequality, in (iii) we include those o such that p o , a q may not belong to U h ` 1 into summation, (iv) is due to (35):
Revised: A π p a q x p a q ď `ř a x p a q˘ 1 { 2 , (ii) is due to CauchySchwarz inequality, in (iii) we include those o such that p o , a q may not belong to U h ` 1 into summation, (iv) is due to (35):

Original: Unif p
Revised: Unif p

Original: A q ˝ Unif p U
Revised: A q ˝ Unif p U

Original: A,h ` 1 q is simply the uniform policy over A ˆ U A,h ` 1 .
Revised: A,h ` 1 q is simply the uniform policy over A ˆ U A,h ` 1 .

Original: This concludes the proof of Lemma F.4.
Revised: This concludes the proof of Lemma F.4.

Original: G P ROOF OF T HEOREM
Revised: G P ROOF OF T HEOREM

Original: We first restate Theorem 9 as follows in terms of the (more relaxed) weak B-stability condition.
Revised: We first restate Theorem 9 as follows in terms of the (more relaxed) weak B-stability condition.

Original: Theorem G.1 (Restatement of Theorem 9) .
Revised: Theorem G.1 (Restatement of Theorem 9) .

Original: Suppose every θ P Θ is Λ B -stable
Revised: Suppose every θ P Θ is Λ B -stable

Original: (Definition 4) or weakly Λ B -stable
Revised: (Definition 4) or weakly Λ B -stable

Original: (Definition D.4), and the true model θ ‹ P Θ with rank d PSR ď d .
Revised: (Definition D.4), and the true model θ ‹ P Θ with rank d PSR ď d .

Original: Then, choosing β “ C log p N Θ p 1 { KH qq{ δ q for some absolute constant C ą 0 , with probability at least 1 ´ δ , Algorithm 1 outputs a policy p π out P ∆ p Π q such that V ‹ ´ V θ ‹ p p π out q ď ε , as long as the number of episodes
Revised: Then, choosing β “ C log p N Θ p 1 { KH qq{ δ q for some absolute constant C ą 0 , with probability at least 1 ´ δ , Algorithm 1 outputs a policy p π out P ∆ p Π q such that V ‹ ´ V θ ‹ p p π out q ď ε , as long as the number of episodes

Original: [Equation]
Revised: [Equation]

Original: where ι : “ log p 1 ` KdU A Λ B R B q with R B : “ max h t 1 , max } v } 1 “
Revised: where ι : “ log p 1 ` KdU A Λ B R B q with R B : “ max h t 1 , max } v } 1 “

Original: [Equation]
Revised: [Equation]

Original: The proof of Theorem G.1 uses the following fast rate guarantee for the OMLE algorithm, which is standard (e.g. Van de Geer (2000); Agarwal et al.
Revised: The proof of Theorem G.1 uses the following fast rate guarantee for the OMLE algorithm, which is standard (e.g. Van de Geer (2000); Agarwal et al.

Original: For completeness, we present its proof in
Revised: For completeness, we present its proof in

Original: Appendix G.1.
Revised: Appendix G.1.

Original: Proposition G.2 (Guarantee of MLE) .
Revised: Proposition G.2 (Guarantee of MLE) .

Original: Suppose that we choose β ě 2 log N Θ p 1 { T q` 2 log p 1 { δ q`in Algorithm 1.
Revised: Suppose that we choose β ě 2 log N Θ p 1 { T q` 2 log p 1 { δ q`in Algorithm 1.

Original: Then with probability at least 1 ´ δ , the following holds:
Revised: Then with probability at least 1 ´ δ , the following holds:

Original: (a) For all k P r K s , θ ‹ P Θ k ; (b) For all k P r K s and any θ P Θ k , it holds that
Revised: (a) For all k P r K s , θ ‹ P Θ k ; (b) For all k P r K s and any θ P Θ k , it holds that

Original: [Equation]
Revised: [Equation]

Original: We next prove Theorem G.1.
Revised: We next prove Theorem G.1.

Original: We adopt the definitions of E ¯ θθ,h p τ h ´ 1 q as in Proposition F.1 and abbreviate E ‹ k,h “ E θ ‹ θ k ,h .
Revised: We adopt the definitions of E ¯ θθ,h p τ h ´ 1 q as in Proposition F.1 and abbreviate E ‹ k,h “ E θ ‹ θ k ,h .

Original: We also condition on the success of the event in Proposition G.2.
Revised: We also condition on the success of the event in Proposition G.2.

Original: Step 1.
Revised: Step 1.

Original: By Proposition G.2, it holds that θ ‹ P Θ .
Revised: By Proposition G.2, it holds that θ ‹ P Θ .

Original: Therefore, V θ k p π k q ě V ‹ , and by Proposition F.1, we have 
Revised: Therefore, V θ k p π k q ě V ‹ , and by Proposition F.1, we have k

Original: [Equation]
Revised: [Equation]

Original: On the other hand, by Proposition F.2, we have
Revised: On the other hand [Equation] by Proposition F.

Revised: [Equation]

Revised: we have

Revised: [Equation]

Revised: Furthermore

Original: [Equation]
Revised: [Equation]

Original: Furthermore, by Proposition G.2 we have
Revised: by Proposition G [Equation] 2 we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, defining β k,h : “ ř t ă k E π t r E ‹ k,h p τ h ´ 1 q 2 s , combining the two equations above gives
Revised: Therefore, defining β k,h : “ ř t ă k E π t r E ‹ k,h p τ h ´ 1 q 2 s , combining the two equations above gives

Original: [Equation]
Revised: [Equation]

Original: Step 2.
Revised: Step 2.

Original: We would like to bridge the performance decomposition (38) and the squared B-errors bound (39) using the generalized ℓ 2 -Eluder argument.
Revised: We would like to bridge the performance decomposition (38) and the squared B-errors bound (39) using the generalized ℓ 2 -Eluder argument.

Original: We consider separately the case for h “and h P r H s .
Revised: We consider separately the case for h “and h P r H s .

Original: Case 1: h “ 0 .
Revised: Case 1: h “ 0 .

Original: This case follows directly from Cauchy-Schwarz inequality:
Revised: This case follows directly from Cauchy-Schwarz inequality:

Original: [Equation]
Revised: [Equation]

Original: Case 2: h P r H s .
Revised: Case 2: h P r H s .

Original: We invoke the generalized ℓ 2 -Eluder argument (actually, its corollary) as in
Revised: We invoke the generalized ℓ 2 -Eluder argument (actually, its corollary) as in

Original: Appendix E.1, restated as follows for convenience.
Revised: Appendix E.1, restated as follows for convenience.

Original: Corollary E.2.
Revised: Corollary E.2.

Original: Suppose we have a sequence of functions 
Revised: Suppose we have a sequence of functions t f k : R n Ñ R u k Pr K s :

Original: [Equation]
Revised: [Equation]

Original: which is given by the family of vectors t y k,j,r u p k,j,r qPr K sˆr J sˆ R Ă R n .
Revised: which is given by the family of vectors t y k,j,r u p k,j,r qPr K sˆr J sˆ R Ă R n .

Original: Further assume that there exists L ą 0 such that f k p x q ď L } x } 1 .
Revised: Further assume that there exists L ą 0 such that f k p x q ď L } x } 1 .

Original: Consider further a sequence of vector p x i q i P I , satisfying the following condition
Revised: Consider further a sequence of vector p x i q i P I , satisfying the following condition

Original: [Equation]
Revised: [Equation]

Original: and the subspace spanned by p x i q i P I has dimension at most d .
Revised: and the subspace spanned by p x i q i P I has dimension at most d .

Original: Then it holds that kÿ
Revised: Then it holds that k

Original: [Equation]
Revised: [Equation]

Original: We have the following three preparation steps to apply Corollary E.2.
Revised: We have the following three preparation steps to apply Corollary E.2.

Original: Recall the definition of E ‹ t,h p τ h ´ 1 q as in Proposition F.1 (in short E ‹ k,h p
Revised: Recall the definition of E ‹ t,h p τ h ´ 1 q as in Proposition F.1 (in short E ‹ k,h p

Original: τ h ´ 1 q : “ E θ ‹ θ k ,h p τ h ´ 1 q ), E ‹ k,h p τ h ´1 q : “ 1 2 max π ÿ τ h : H
Revised: τ h ´ 1 q : “ E θ ‹ θ k ,h p τ h ´ 1 q ), E ‹ k,h p [Equation] ´

Revised: [Equation]

Revised: q

Original: [Equation]
Revised: [Equation]

Original: where we replace superscript θ k of B by k for simplicity.
Revised: where we replace superscript θ k of B by k for simplicity.

Original: Let us definey
Revised: Let us define

Original: [Equation]
Revised: [Equation]

Original: where t τ jh : H “ p o h , a h , ¨ ¨ ¨ , o H , a H qu nj “ 1 is an ordering of all possible τ h : H (and hence n “ p OA q H ´ h ` 1 ), π is any policy that starts at step h .
Revised: where t τ jh : H “ p o h , a h , ¨ ¨ ¨ , o H , a H qu nj “ 1 is an ordering of all possible τ h : H (and hence n “ p OA q H ´ h ` 1 ), π is any policy that starts at step h .

Original: We then define
Revised: We then define

Original: [Equation]
Revised: [Equation]

Revised: x y

Revised: [Equation]

Revised: , x y

Revised: [Equation]

Revised: x P

Revised: [Equation]

Original: It follows from definition that E ‹ k,h p τ h ´ 1 q “ f k p q ‹ p τ h ´ 1 qq .
Revised: It follows from definition that E ‹ k,h p τ h ´ 1 q “ f k p q ‹ p τ h ´ 1 qq .

Original: We define x i “ q ‹ p τ ih ´
Revised: We define x i “ q ‹ p τ ih ´

Original: 1 q P R | U h | , where t τ ih ´
Revised: 1 q P R | U h | , where t τ ih ´

Original: 1 u i is an ordering of all possible τ h ´ 1 P p
Revised: 1 u i is an ordering of all possible τ h ´ 1 P p

Original: O ˆ
Revised: O ˆ

Original: A q h ´ 1 .
Revised: A q h ´ 1 .

Original: Then by the assumption that θ ‹ has PSR rank less than or equal to d , we have dim span p x
Revised: Then by the assumption that θ ‹ has PSR rank less than or equal to d , we have dim span p x

Original: i : i P I q ď d .
Revised: i : i P I q ď d .

Original: Furthermore, we have } x i } 1 ď U A by definition.
Revised: Furthermore, we have } x i } 1 ď U A by definition.

Original: It remains to verify that f k is Lipschitz with respect to 1 -norm.
Revised: It remains to verify that f k is Lipschitz with respect to 1 -norm.

Original: We only need to verify it under the weak Λ B -stability condition.
Revised: We only need to verify it under the weak Λ B -stability condition.

Original: We have
Revised: We have

Original: [Equation]

Original: π ÿ o,a π p a | o

Original: [Equation]
Revised: [Equation]

Original: } B ‹ h p o, a

Original: [Equation]

Original: }ď 2Λ B } q } 1 ` 2Λ B ÿ o,a }

Original: [Equation]

Original: [Equation]

Original: a

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: ď 2Λ

Original: [Equation]

Original: p [Equation] ` 1 q } q } 1 , where the first inequality follows the same argument as (35) ; the second inequality is due to Bstability (or weak B-stability and (21)); the last inequality is due to the definition of B .
Revised: where the first inequality follows the same argument as (35); the second inequality is due to Bstability (or weak B-stability and (21)); the last inequality is due to the definition of B .

Original: Hence we can take L “ 2Λ B p R B ` 1 q to ensure that f k p x q ď L } x } 1 .
Revised: Hence we can take L “ 2Λ B p R B ` 1 q to ensure that f k p x q ď L } x } 1 .

Original: Therefore, applying Corollary E.2 yields
Revised: Therefore, applying Corollary E.2 yields

Original: [Equation]
Revised: [Equation]

Original: where ι “ log p 1 ` 2 kdU A Λ B p R B ` 1 qq .
Revised: where ι “ log p 1 ` 2 kdU A Λ B p R B ` 1 qq .

Original: This completes case 2.
Revised: This completes case 2.

Original: Combining these two cases, we obtain 
Revised: Combining these two cases [Equation] we obtain [Equation]

Revised: ÿ t “

Revised: H ÿ h “β t,h ¯ p iii q

Original: [Equation]
Revised: [Equation]

Original:  where (i) used (38); (ii) used the above two cases (40) and (41); (iii) used (39).
Revised: O ˆ [Equation] kβι ˙ . where (i) used (38); (ii) used the above two cases (40) and (41); (iii) used (39).

Original: As a consequence, whenever k ě O p Λ 2 B dAU A H ¨ βι { ε 2 q , we have 1 k ř kt “ 1 p V ‹ ´
Revised: As a consequence, whenever k ě O p Λ 2 B dAU A H ¨ βι { ε 2 q , we have 1 k ř kt “ 1 p V ‹ ´

Original: V θ ‹ p π t qq ď ε .
Revised: V θ ‹ p π t qq ď ε .

Original: This completes the proof of Theorem G.1 (and hence Theorem 9).
Revised: This completes the proof of Theorem G.1 (and hence Theorem 9).

Original: G. P ROOF OF P ROPOSITION G 
Revised: G. P ROOF OF P ROPOSITION G.

Original: For the simplicity of presentation, we consider the following general interaction process: For t “ 1 , ¨ ¨ ¨ , T , the learner determines a ¯ π t , then executes ¯ π t and collects a trajectory ¯ τ t „ P ¯ π t θ ‹ p¨q .
Revised: For the simplicity of presentation, we consider the following general interaction process: For t “ 1 , ¨ ¨ ¨ , T , the learner determines a ¯ π t , then executes ¯ π t and collects a trajectory ¯ τ t „ P ¯ π t θ ‹ p¨q .

Original: We show that, with probability at least 1 ´ δ , the following holds for all t P r T s and θ P Θ : 
Revised: We show that, with probability at least 1 ´ δ , the following holds for all t P r T s and θ P Θ : t ´ 1 ÿ

Original: [Equation]
Revised: [Equation]

Original: where L t is the total log-likelihood (at step t ) defined as
Revised: where L t is the total log-likelihood (at step t ) defined as

Original: [Equation]
Revised: [Equation]

Original: Proposition G.2 is implied by (42) directly: Suppose we choose β ě 2 log N Θ p 1 { T q` 2 log p 1 { δ q` 2 .
Revised: Proposition G.2 is implied by (42) directly: Suppose we choose β ě 2 log N Θ p 1 { T q` 2 log p 1 { δ q` 2 .

Original: On the event (42), by the non-negativity of squared Hellinger distances [Equation] we have for all k P r K s and θ P Θ that ÿ
Revised: On the event (42), by the non-negativity of squared Hellinger distances, we have for all k P r K s and θ P Θ that ÿ

Original: [Equation]
Revised: [Equation]

Original: where D k is the dataset of all histories before the outer loop of Algorithm 1 enters step k .
Revised: where D k is the dataset of all histories before the outer loop of Algorithm 1 enters step k .

Original: Taking max over θ P Θ on the right-hand side, we obtain θ ‹ P Θ k , which gives Proposition G.2(1).
Revised: Taking max over θ P Θ on the right-hand side, we obtain θ ‹ P Θ k , which gives Proposition G.2(1).

Original: Furthermore, for k P r K s and θ P Θ k , (42) implies that
Revised: Furthermore, for k P r K s and θ P Θ k , (42) implies that

Original: [Equation]
Revised: [Equation]

Original: which gives Proposition G.2(2)
Revised: which gives Proposition G.2(2)

Original: In the following, we establish (42).
Revised: In the following, we establish (42).

Original: Let us fix a 1 { T -optimistic covering p r P , Θ 0 q of Θ , such that n : “ | Θ 0 | “ N Θ p 1 { T q .
Revised: Let us fix a 1 { T -optimistic covering p r P , Θ 0 q of Θ , such that n : “ | Θ 0 | “ N Θ p 1 { T q .

Original: We label p r P θ 0 q θ 0 P Θ 0 by r P 1 , ¨ ¨ ¨ , r P n .
Revised: We label p r P θ 0 q θ 0 P Θ 0 by r P 1 , ¨ ¨ ¨ , r P n .

Original: By the definition of optimistic covering, it is clear that for any θ P Θ , there exists i P r n s such that for all π , τ , it holds that r
Revised: By the definition of optimistic covering, it is clear that for any θ P Θ , there exists i P r n s such that for all π , τ , it holds that r

Original: P
Revised: P

Original: πi p τ q ě P πθ p τ q and } r P πi p¨q ´
Revised: πi p τ q ě P πθ p τ q and } r P πi p¨q ´

Original: P πθ p¨q} 1 ď 1 { T 2 .
Revised: P πθ p¨q} 1 ď 1 { T 2 .

Original: We say θ is covered by this i P r n s .
Revised: We say θ is covered by this i P r n s .

Original: Then, we consider
Revised: Then, we consider

Original: [Equation]
Revised: [Equation]

Original: By Lemma C.3, the following holds with probability at least 1 ´ δ : for all t P r T s , i P r n s ,
Revised: By Lemma C.3, the following holds with probability at least 1 ´ δ : for all t P r T s , i P r n s ,

Original: [Equation]
Revised: [Equation]

Original: where E s denotes the conditional expectation over all randomness after ¯ π s has been determined.
Revised: where E s denotes the conditional expectation over all randomness after ¯ π s has been determined.

Original: By definition,
Revised: By definition,

Original: [Equation]
Revised: [Equation]

Original: Therefore, for any θ P Θ that is covered by i P r n s , we have ´ log E t
Revised: Therefore, for any θ P Θ that is covered by i P r n s , we have ´ log E t

Original: [Equation]
Revised: [Equation]

Revised: ˙

Revised: Input: Model class Θ , parameters γ ą 0 , η P p 0 , 1 { 2 q .

Revised: An 1 { T -optimistic cover p r P , Θ 0 q .

Revised: 1: Initialize µ 1 “ Unif p Θ 0 q .

Revised: 2: for t “ 1 , . . .

Revised: , T do 3:

Revised: Set p p t exp , p t out q “ arg min p p exp ,p

Revised: [Equation]

Revised: p

Revised: [Equation]

Revised: p p exp , p out q , where p V µ t γ is defined by

Revised: [Equation]

Revised: [Equation]

Revised: Sample π t „ p t exp .

Revised: Execute π t and observe τ t .

Revised: Compute µ t ` 1 P ∆ p Θ 0 q by

Revised: [Equation]

Original: ˜ÿ
Revised: ¯ .

Revised: Output: Policy p π out : “ 1 T ř Tt “ 1 p t out .

Original: [Equation]
Revised: [Equation]

Original: where the first inequality is due to ´ log x ě 1 ´ x ; in the second inequality we use the definition of
Revised: where the first inequality is due to ´ log x ě 1 ´ x ; in the second inequality we use the definition of

Original: Hellinger distance and Cauchy inequality; the third inequality is because p?
Revised: Hellinger distance and Cauchy inequality; the third inequality is because p?

Original: x ´?
Revised: x ´?

Original: y q 2 ď | x ´ y | for all x, y P R ě 0 ; the last inequality is due to our assumption that θ is covered by i .
Revised: y q 2 ď | x ´ y | for all x, y P R ě 0 ; the last inequality is due to our assumption that θ is covered by i .

Original: Notice that every θ P Θ is covered by some i P r n s , and for such i , ř t ´ 1 s “ 1 ℓ si ď L t p θ ‹ q ´ L t p θ q ; therefore, it holds with probability 1 ´ δ that, for all θ P Θ , t P r T s ,
Revised: Notice that every θ P Θ is covered by some i P r n s , and for such i , ř t ´ 1 s “ 1 ℓ si ď L t p θ ‹ q ´ L t p θ q ; therefore, it holds with probability 1 ´ δ that, for all θ P Θ , t P r T s ,

Original: [Equation]
Revised: [Equation]

Original: Plugging in n “ N Θ p 1 { T q and scaling the above inequality by 2 gives (42).
Revised: Plugging in n “ N Θ p 1 { T q and scaling the above inequality by 2 gives (42).

Original: H [Equation] E2D, AND MOPS
Revised: H E XPLORATIVE E2D, A LL -P OLICY M ODEL -E STIMATION E2D, AND MOPS

Original: In this section, we present the detailed algorithms of E XPLORATIVE E2D, A LL -P OLICY M ODELE STIMATION E2D, and MOPS introduced in Section 4.
Revised: In this section, we present the detailed algorithms of E XPLORATIVE E2D, A LL P OLICY M ODEL E STIMATION E2D, and MOPS introduced in Section 4.

Original: We also state the theorems for their sample complexity bounds of learning ε -optimal policy of B-stable PSRs.
Revised: We also state the theorems for their sample complexity bounds of learning ε -optimal policy of B-stable PSRs.

Original: H. E XPLORATIVE E2D ALGORITHM
Revised: H. E XPLORATIVE E2D ALGORITHM

Original: In this section, we provide more details about the E XPLORATIVE E2D algorithm as discussed in
Revised: In this section, we provide more details about the E XPLORATIVE E2D algorithm as discussed in

Original: Section 4.2.
Revised: Section 4.2.

Original: The full algorithm of E XPLORATIVE E2D is given in Algorithm 2, equivalent to Chen [Equation] [Equation] p
Revised: The full algorithm of E XPLORATIVE E2D is given in Algorithm 2, equivalent to Chen et al.

Original: [Equation] [Equation] et al. (2022, Algorithm 2) in the known reward setting ( D 2RL becomes D 2 H since we assumed that the reward is deterministic and known, so that the contribution from reward distance in D 2RL becomes 0 ).
Revised: (2022, Algorithm 2) in the known reward setting ( D 2RL becomes D 2 H since we assumed that the reward is deterministic and known, so that the contribution from reward distance in D 2RL becomes 0 ).

Original: Chen et al.
Revised: Chen et al.

Original: (2022, Theorem F.1) showed that E XPLORATIVE E2D achieves the following estimation bound.
Revised: (2022, Theorem F.1) showed that E XPLORATIVE E2D achieves the following estimation bound.

Original: Theorem H.1 (Chen et al.
Revised: Theorem H.1 (Chen et al.

Original: (2022), Theorem F.1) .
Revised: (2022), Theorem F.1) .

Original: Given an 1 { T -optimistic cover p r P , Θ 0 q (c.f. Definition C.4) of the model class Θ , Algorithm 2 with η “ 1 { 3 achieves the following with probability at least 1 ´ δ :
Revised: Given an 1 { T -optimistic cover p r P , Θ 0 q (c.f. Definition C.4) of the model class Θ , Algorithm 2 with η “ 1 { 3 achieves the following with probability at least 1 ´ δ :

Original: [Equation]
Revised: [Equation]

Original: where edec γ is the Explorative DEC as defined in Section 4.2. 
Revised: where edec γ is the Explorative DEC as defined in Section 4.2. [Table] [Table] [Table]

Revised: [Table]

Original: As we can see from the theorem above, as long as we can bound edec γ p Θ q , we can get a sample complexity bound for the E XPLORATIVE E2D
Revised: [Table] [Table] 2H ´ [Table]

Revised: ¯ı

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: As we can see from the theorem above, as long as we can bound edec γ p Θ q , we can get a sample complexity bound for the E XPLORATIVE E2D

Original: algorithm.
Revised: algorithm.

Original: This gives Theorem 10 in the main text, which we restate as below.
Revised: This gives Theorem 10 in the main text, which we restate as below.

Original: Theorem H.2 (Restatement of Theorem 10) .
Revised: Theorem H.2 (Restatement of Theorem 10) .

Original: Suppose Θ is a PSR class with the same core test sets t
Revised: Suppose Θ is a PSR class with the same core test sets t

Original: U
Revised: U

Original: h u h
Revised: h u h

Original: Pr H s , and
Revised: Pr H s , and

Original: each θ P Θ admits a B-representation that is Λ B -stable
Revised: each θ P Θ admits a B-representation that is Λ B -stable

Original: (c.f. Definition 4) or weakly Λ B -stable (c.f.
Revised: (c.f. Definition 4) or weakly Λ B -stable (c.f.

Original: Definition D.4), and has PSR rank d PSR ď d .
Revised: Definition D.4), and has PSR rank d PSR ď d .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Therefore, we can choose a suitable parameter γ and an 1 { T -optimistic cover p r P , Θ 0 q , such that with probability at least 1 ´ δ , Algorithm 2 outputs a policy p π out P ∆ p Π q such that V ‹ ´ V θ ‹ p p π out q ď ε , as long as the number of episodes 
Revised: Therefore, we can choose a suitable parameter γ and an 1 { T -optimistic cover p r P , Θ 0 q , such that with probability at least 1 ´ δ , Algorithm 2 outputs a policy p π out P ∆ p Π q such that V ‹ ´ V θ ‹ p p π out q ď ε , as long as the number of episodes T ě O

Original: [Equation]
Revised: [Equation]

Original: The proof of Theorem H.2 and hence Theorem 10 is contained in Appendix I.2.
Revised: The proof of Theorem H.2 and hence Theorem 10 is contained in Appendix I.2.

Original: H. A LL -P OLICY M ODEL -E STIMATION E2D FOR MODEL - ESTIMATION
Revised: H. A LL -P OLICY M ODEL -E STIMATION E2D FOR MODEL - ESTIMATION

Original: In this section, we provide more details about model-estimation learning in PSRs as discussed in
Revised: In this section, we provide more details about model-estimation learning in PSRs as discussed in

Original: Section 4.2.
Revised: Section 4.2.

Original: In reward-free RL (Jin et al., 2020b), the goal is to optimally explore the environment without observing reward information, so that after the exploration phase, a near-optimal policy of any given reward can be computed using the collected trajectory data alone without further interacting with the environment.
Revised: In reward-free RL (Jin et al., 2020b), the goal is to optimally explore the environment without observing reward information, so that after the exploration phase, a near-optimal policy of any given reward can be computed using the collected trajectory data alone without further interacting with the environment.

Original: Chen et al.
Revised: Chen et al.

Original: (2022) developed A LL -P OLICY M ODEL -E STIMATION E2D as a unified algorithm for reward-free/model-estimation learning in RL, and showed that its sample complexity scales with a complexity measure named All-policy Model-Estimation DEC (AMDEC).The AMDEC is defined Set p p t exp , µ [Equation]
Revised: (2022) developed A LL -P OLICY M ODEL -E STIMATION E2D as a unified algorithm for reward-free/model-estimation learning in RL, and showed that its sample complexity scales with a complexity measure named All-policy Model-Estimation DEC (AMDEC).

Original: out q “ arg

Original: [Equation]

Original: , µ

Original: [Equation]

Original: q , where

Original: [Equation]

Original: D 2H ´

Original: [Equation]

Original: ¯ı .

Original: Compute

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: ´

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: ¯ .

Original: 7: Compute µ out “ 1 T ř Tt

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: P

Original: [Table]

Original: p Θ q .

Original: 8: Output: p θ P Θ sup π P Π E ¯ θ „ µ out “

Original: [Equation]

Original:  as amdec γ p Θ q : “ sup p µ P ∆ p Θ q amdec γ p Θ , p µ q , where amdecγ p Θ , p µ q : “ inf p exp P [Equation] p Π q [Equation]
Revised: The AMDEC is defined as amdec γ p Θ q : “ sup p µ P ∆ p Θ q amdec γ p Θ , p µ q , where amdec

Original: µ out P ∆ p

Original: [Equation]
Revised: [Equation]

Original: q sup

Original: [Equation]

Original: P

Original: [Equation]

Original: sup

Original: [Equation]

Original: P

Original: [Equation]

Original: E

Original: [Equation]

Original: out “

Original: [Equation]

Original: D

Original: [Equation]

Original: ¯ı .
Revised: ¯ı .

Original: The A LL -P OLICY M ODEL -E STIMATION E2D algorithm
Revised: The A LL -P OLICY M ODEL -E STIMATION E2D algorithm

Original: (Algorithm 3) for a PSR class Θ is given as follows:
Revised: (Algorithm 3) for a PSR class Θ is given as follows:

Original: In each episode t P r T s , we maintain a distribution µ t P ∆ p Θ 0 q over an 1 { T -optimistic cover p r P , Θ 0 q of Θ (c.f. Definition C.4), which we use to compute an exploration policy distribution p t exp by minimizing the following risk:
Revised: In each episode t P r T s , we maintain a distribution µ t P ∆ p Θ 0 q over an 1 { T -optimistic cover p r P , Θ 0 q of Θ (c.f. Definition C.4), which we use to compute an exploration policy distribution p t exp by minimizing the following risk:

Original: [Equation]
Revised: [Equation]

Original: Then, we execute policy π t „ p t exp , collect trajectory τ t , and update the model distribution using the same Tempered Aggregation scheme as in E XPLORATIVE E2D.
Revised: Then, we execute policy π t „ p t exp , collect trajectory τ t , and update the model distribution using the same Tempered Aggregation scheme as in E XPLORATIVE E2D.

Original: After T episodes, we output the emipirical model p θ by computing µ out “ 1 T
Revised: After T episodes, we output the emipirical model p θ by computing µ out “ 1 T

Original: ř Tt “ 1 µ t out P ∆ p Θ q
Revised: ř Tt “ 1 µ t out P ∆ p Θ q

Original: and then projecting it into Θ , i.e.
Revised: and then projecting it into Θ , i.e.

Original: [Equation]
Revised: [Equation]

Original: Chen et al.
Revised: Chen et al.

Original: (2022, Theorem H.2) show that the output model p θ of A LL -P OLICY M ODELE STIMATION E2D has an estimation error (measured in terms of the TV distance) that scales as amdec γ .
Revised: (2022, Theorem H.2) show that the output model p θ of A LL -P OLICY M ODELE STIMATION E2D has an estimation error (measured in terms of the TV distance) that scales as amdec γ .

Original: Theorem H.3.
Revised: Theorem H.3.

Original: Given an 1 { T -optimistic cover p r P , Θ 0 q (c.f. Definition C.4) of the class of transition dynamics Θ , Algorithm 3 with η “ 1 { 2 achieves the following with probability at least 1 ´ δ :
Revised: Given an 1 { T -optimistic cover p r P , Θ 0 q (c.f. Definition C.4) of the class of transition dynamics Θ , Algorithm 3 with η “ 1 { 2 achieves the following with probability at least 1 ´ δ :

Original: [Equation]
Revised: [Equation]

Original: where amdec γ is the All-policy Model-Estimation DEC as defined in (43).
Revised: where amdec γ is the All-policy Model-Estimation DEC as defined in (43).

Original: We provide a sharp bound on the AMEDEC for B-stable PSRs, which implies that A LL -P OLICY
Revised: We provide a sharp bound on the AMEDEC for B-stable PSRs, which implies that A LL -P OLICY

Original: M ODEL -E STIMATION E2D can also learn them sample-efficient efficiently in a model-estimation manner.
Revised: M ODEL -E STIMATION E2D can also learn them sample-efficient efficiently in a model-estimation manner.

Original: Theorem H.4.
Revised: Theorem H.4.

Original: Suppose Θ is a PSR class with the same core test sets t
Revised: Suppose Θ is a PSR class with the same core test sets t

Original: U
Revised: U

Original: h u h
Revised: h u h

Original: Pr H s , and
Revised: Pr H s , and

Original: each θ P Θ admits a B-representation that is Λ B -stable
Revised: each θ P Θ admits a B-representation that is Λ B -stable

Original: (c.f. Definition 4) or weakly Λ B -stable (c.f.
Revised: (c.f. Definition 4) or weakly Λ B -stable (c.f.

Original: Definition D.4), and has PSR rank d PSR ď d .
Revised: Definition D.4), and has PSR rank d PSR ď d .

Original: Then
Revised: Then

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Therefore, we can choose a suitable parameter γ and an 1 { T -optimistic cover p r P , Θ 0 q , such that with probability at least 1 ´ δ , Algorithm 3 outputs a model p θ P Θ such that sup π D TV
Revised: Therefore, we can choose a suitable parameter γ and an 1 { T -optimistic cover p r P , Θ 0 q , such that with probability at least 1 ´ δ , Algorithm 3 outputs a model p θ P Θ such that sup π D TV

Original: [Equation]
Revised: [Equation]

Original: as long as the number of episodes
Revised: as long as the number of episodes

Original: [Equation]
Revised: [Equation]

Original: The proof of Theorem H.4 is contained in Appendix I.3.
Revised: The proof of Theorem H.4 is contained in Appendix I.3.

Original: H. M ODEL - BASED OPTIMISTIC POSTERIOR SAMPLING (MOPS)
Revised: H. M ODEL - BASED OPTIMISTIC POSTERIOR SAMPLING (MOPS)

Original: In this section, we provide more details about the MOPS algorithm as discussed in Section 4.3.
Revised: In this section, we provide more details about the MOPS algorithm as discussed in Section 4.3.

Original: We consider the following version of the MOPS algorithm of Agarwal & Zhang (2022);
Revised: We consider the following version of the MOPS algorithm of Agarwal & Zhang (2022);

Original: Chen et al.
Revised: Chen et al.

Original: Similar to E XPLORATIVE E2D, MOPS also maintains a posterior µ t P ∆ p Θ 0 q over an 1 { T optimistic cover p r P , Θ 0 q , initialized at a suitable prior µ 1 .
Revised: Similar to E XPLORATIVE E2D, MOPS also maintains a posterior µ t P ∆ p Θ 0 q over an 1 { T optimistic cover p r P , Θ 0 q , initialized at a suitable prior µ 1 .

Original: The exploration policy in the t -th episode is obtained by posterior sampling: π t “ π θ t ˝
Revised: The exploration policy in the t -th episode is obtained by posterior sampling: π t “ π θ t ˝

Original: h t Unif p
Revised: h t Unif p

Original: A q ˝ h t ` 1 Unif p U A,h t ` 1 q , where θ t „ µ t and h t „ Unif pt 0 , 1 , . .
Revised: A q ˝ h t ` 1 Unif p U A,h t ` 1 q , where θ t „ µ t and h t „ Unif pt 0 , 1 , . .

Original: , H ´ 1 uq .
Revised: , H ´ 1 uq .

Original: After executing π t and observing τ t , the algorithm updates the posterior as
Revised: After executing π t and observing τ t , the algorithm updates the posterior as

Original: [Equation]
Revised: [Equation]

Revised: ˘¯

Revised: [Equation]

Original: Finally, the algorithm output p π out : “ 1 T ř Tt “ 1 p out p µ t q , where p out p µ t q P ∆ p Π q is defined as
Revised: Finally, the algorithm output p π out : “ 1 T ř Tt “ 1 p out p µ t q , where p out p µ t q P ∆ p Π q is defined as

Original: [Equation]
Revised: [Equation]

Original: We further consider the following Explorative PSC (EPSC), which is a modification of the PSC proposed in Chen et al.
Revised: We further consider the following Explorative PSC (EPSC), which is a modification of the PSC proposed in Chen et al.

Original: (2022, Definition 4):
Revised: (2022, Definition 4):

Original: [Equation]
Revised: [Equation]

Original: where ∆ 0 p Θ q is the set of all finitely supported distributions on Θ , π exp is defined as
Revised: where ∆ 0 p Θ q is the set of all finitely supported distributions on Θ , π exp is defined as

Original: [Equation]
Revised: [Equation]

Original: and we abbreviate π „ p out p µ q to π „ µ .
Revised: and we abbreviate π „ p out p µ q to π „ µ .

Original: Adapting the proof for the MOPS algorithm in Chen et al.
Revised: Adapting the proof for the MOPS algorithm in Chen et al.

Original: (2022, Corollary D.3 & Theorem D.1) to the explorative version, we can show that the output policy p π out of MOPS has a sub-optimality gap that scales as psc est .
Revised: (2022, Corollary D.3 & Theorem D.1) to the explorative version, we can show that the output policy p π out of MOPS has a sub-optimality gap that scales as psc est .

Original: Theorem H.5.
Revised: Theorem H.5.

Original: Given an 1 { T -optimistic cover p r P , Θ 0 q (c.f. Definition C.4) of the class of PSR models Θ , Algorithm 4 with η “ 1 { 6 and γ ě 1 achieves the following with probability at least 1 ´ δ :
Revised: Given an 1 { T -optimistic cover p r P , Θ 0 q (c.f. Definition C.4) of the class of PSR models Θ , Algorithm 4 with η “ 1 { 6 and γ ě 1 achieves the following with probability at least 1 ´ δ :

Original: [Equation]
Revised: [Equation]

Original: where psc est γ is the Explorative PSC as defined in (46).
Revised: where psc est γ is the Explorative PSC as defined in (46).

Original: We provide a sharp bound on the EPSC for B-stable PSRs, which implies that MOPS can also learn them sample-efficient efficiently.
Revised: We provide a sharp bound on the EPSC for B-stable PSRs, which implies that MOPS can also learn them sample-efficient efficiently.

Original: Theorem H.6.
Revised: Theorem H.6.

Original: Suppose Θ is a PSR class with the same core test sets t
Revised: Suppose Θ is a PSR class with the same core test sets t

Original: U
Revised: U

Original: h u h
Revised: h u h

Original: Pr H s , and
Revised: Pr H s , and

Original: each θ P Θ admits a B-representation that is Λ B -stable
Revised: each θ P Θ admits a B-representation that is Λ B -stable

Original: (c.f. Definition 4) or weakly Λ B -stable (c.f.
Revised: (c.f. Definition 4) or weakly Λ B -stable (c.f.

Original: Definition D.4), and the ground truth model θ ‹ has PSR rank at most d .
Revised: Definition D.4), and the ground truth model θ ‹ has PSR rank at most d .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: A H 2 { γ. Therefore, we can choose a suitable parameter γ and an 1 { T -optimistic cover p r P , Θ 0 q , such that with probability at least 1 ´ δ , Algorithm 4 outputs a policy p π out P ∆ p Π q such that V ‹ ´ V θ ‹ p p π out q ď ε , as long as the number of episodes
Revised: Therefore, we can choose a suitable parameter γ and an 1 { T -optimistic cover p r P , Θ 0 q , such that with probability at least 1 ´ δ , Algorithm 4 outputs a policy p π out P ∆ p Π q such that V ‹ ´ V θ ‹ p p π out q ď ε , as long as the number of episodes

Original: [Equation]
Revised: [Equation]

Original: The proof of Theorem H.6 is contained in Appendix I.2.
Revised: The proof of Theorem H.6 is contained in Appendix I.2.

Original: We remark here that EPSC provides an upper bound of EDEC (c.f. Eq.
Revised: We remark here that EPSC provides an upper bound of EDEC (c.f. Eq.

Original: So Theorem H.2 (and hence Theorem 10) directly follows from Theorem H.6.
Revised: So Theorem H.2 (and hence Theorem 10) directly follows from Theorem H.6.

Original: I P ROOFS FOR A PPENDIX H
Revised: I P ROOFS FOR A PPENDIX H

Original: For the clarity of discussion, we introduce the following notation in this section: for policy π , we denote φ h to be a policy modification such that
Revised: For the clarity of discussion, we introduce the following notation in this section: for policy π , we denote φ h to be a policy modification such that

Original: [Equation]
Revised: [Equation]

Original: Again, here φ h ˛ π means that we follow π for the first h ´ 1 steps, takes Unif p
Revised: Again, here φ h ˛ π means that we follow π for the first h ´ 1 steps, takes Unif p

Original: A q at step h , takes an action sequence sampled from Unif p U A,h ` 1 q at step h ` 1 , and behaves arbitrarily afterwards.
Revised: A q at step h , takes an action sequence sampled from Unif p U A,h ` 1 q at step h ` 1 , and behaves arbitrarily afterwards.

Original: Such definition agrees with (47).
Revised: Such definition agrees with (47).

Original: We further define the φ policy modification as
Revised: We further define the φ policy modification as

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We call φ ˛ π the exploration policy of π .
Revised: We call φ ˛ π the exploration policy of π .

Original: I. P ROOF OF T HEOREM  
Revised: I. P ROOF OF T HEOREM H.

Original: H.

Original: To prove Theorem H.6, due to Theorem H.5, we only need to bound the coefficients psc est γ p Θ , θ ‹ q .
Revised: To prove Theorem H.6, due to Theorem H.5, we only need to bound the coefficients psc est γ p Θ , θ ‹ q .

Original: By its definition, we have
Revised: By its definition, we have

Original: [Equation]
Revised: [Equation]

Original: We then invoke the following error decorrelation result, which follows from the decoupling argument in Appendix E.2 and Proposition F.1.
Revised: We then invoke the following error decorrelation result, which follows from the decoupling argument in Appendix E.2 and Proposition F.1.

Original: Proposition I.1 (Error decorrelation) .
Revised: Proposition I.1 (Error decorrelation) .

Original: Under the condition of Theorem H.2 (the same condition as
Revised: Under the condition of Theorem H.2 (the same condition as

Original: Theorem H.6), for any µ P ∆ 0 p Θ q and any reference model ¯ θ P Θ , we have
Revised: Theorem H.6), for any µ P ∆ 0 p Θ q and any reference model ¯ θ P Θ , we have

Original: [Equation]
Revised: [Equation]

Original: where d ¯ θ is the PSR rank of ¯ θ , φ ˛ π defined in (47) is the exploration policy of π .
Revised: where d ¯ θ is the PSR rank of ¯ θ , φ ˛ π defined in (47) is the exploration policy of π .

Original: Combining Proposition I.1 with (48) immediately gives the desired upper bound of psc est γ p Θ , θ ‹ q , and thus completes the proof of Theorem H.6.
Revised: Combining Proposition I.1 with (48) immediately gives the desired upper bound of psc est γ p Θ , θ ‹ q , and thus completes the proof of Theorem H.6.

Original: We next turn to prove the Proposition I.1 above.
Revised: We next turn to prove the Proposition I.1 above.

Original: We consider the following generalized version of
Revised: We consider the following generalized version of

Original: Proposition I.1.
Revised: Proposition I.1.

Original: Proposition I.2 (Generalized error decorrelation) .
Revised: Proposition I.2 (Generalized error decorrelation) .

Original: Under the condition of Theorem H.4, for any ¯ θ P Θ ν P ∆ 

Revised: Under the condition of Theorem H.4, for any ¯ θ P Θ ν P ∆

Original: p Θ ˆ Π q , we have
Revised: p Θ ˆ Π q , we have

Original: [Equation]
Revised: [Equation]

Original: where φ ˛ π defined in (47) is the exploration policy of π .
Revised: where φ ˛ π defined in (47) is the exploration policy of π .

Original: Proof of Proposition I.2.
Revised: Proof of Proposition I.2.

Original: In the following, we fix a ¯ θ P Θ and abbreviate E “ E ¯ θ , q “ q ¯ θ .
Revised: In the following, we fix a ¯ θ P Θ and abbreviate E “ E ¯ θ , q “ q ¯ θ .

Original: Then, by
Revised: Then, by

Original: Proposition F.1, we have
Revised: Proposition F.1, we have

Original: [Equation]
Revised: [Equation]

Original: Note that for the term E θ „ µ “
Revised: Note that for the term E θ „ µ “

Original: E θ,‰ , we have
Revised: E θ,‰ , we have

Original: [Equation]
Revised: [Equation]

Original: We next consider the case for h P r H s , and upper bound the corresponding terms in the righthand-side of (49) using the decoupling argument introduced in Appendix E.2, restated as follows for convenience.
Revised: We next consider the case for h P r H s , and upper bound the corresponding terms in the righthand-side of (49) using the decoupling argument introduced in Appendix E.2, restated as follows for convenience.

Original: Proposition E.6 (Decoupling argument) .
Revised: Proposition E.6 (Decoupling argument) .

Original: Suppose we have vectors and functions
Revised: Suppose we have vectors and functions

Original: [Equation]
Revised: [Equation]

Original: where Θ , I are arbitrary abstract index sets, with functions f θ given by
Revised: where Θ , I are arbitrary abstract index sets, with functions f θ given by

Original: [Equation]
Revised: [Equation]

Original: where t y θ,j,r u p θ,j,r qP Θ ˆr J sˆ R Ă R n is a family of bounded vectors in R n .
Revised: where t y θ,j,r u p θ,j,r qP Θ ˆr J sˆ R Ă R n is a family of bounded vectors in R n .

Original: Then for any distribution µ over Θ and probability family t q θ u θ P Θ Ă ∆ p I q ,
Revised: Then for any distribution µ over Θ and probability family t q θ u θ P Θ Ă ∆ p I q ,

Original: [Equation]
Revised: [Equation]

Original: where d X is the dimension of the subspace of R n spanned by p x i q i P I .
Revised: where d X is the dimension of the subspace of R n spanned by p x i q i P I .

Original: We have the following three preparation steps to apply Proposition E.6: 1.
Revised: We have the following three preparation steps to apply Proposition E.6: 1.

Original: Recall that E “ E ¯ θ is defined in Proposition F.1.
Revised: Recall that E “ E ¯ θ is defined in Proposition F.1.

Original: Let us define
Revised: Let us define

Original: [Equation]
Revised: [Equation]

Original: where t τ jh : H “ p o jh , a jh , ¨ ¨ ¨ , o jH , a jH qu n y j “ 1 is an ordering of all possible τ h : H (and hence n y “ p OA q H ´ h ` 1 ), π is any policy (that starts at step h ).
Revised: where t τ jh : H “ p o jh , a jh , ¨ ¨ ¨ , o jH , a jH qu n y j “ 1 is an ordering of all possible τ h : H (and hence n y “ p OA q H ´ h ` 1 ), π is any policy (that starts at step h ).

Original: We then define
Revised: We then define

Original: [Equation]
Revised: [Equation]

Original: Then it follows from definition (c.f. Proposition F.1)
Revised: Then it follows from definition (c.f. Proposition F.1)

Original: that E θ,h p τ h ´ 1 q “ f θ p q p τ h ´ 1 qq .
Revised: that E θ,h p τ h ´ 1 q “ f θ p q p τ h ´ 1 qq .

Original: We define x i “ q ‹ p τ ih ´
Revised: We define x i “ q ‹ p τ ih ´

Original: 1 q P R | U h | for i P I “ p O ˆ A q h ´ 1 where t τ ih ´
Revised: 1 q P R | U h | for i P I “ p O ˆ A q h ´ 1 where t τ ih ´

Original: 1 u i P
Revised: 1 u i P

Original: I is an ordering of all possible τ h ´ 1 P p
Revised: I is an ordering of all possible τ h ´ 1 P p

Original: O ˆ
Revised: O ˆ

Original: A q h ´ 1 .
Revised: A q h ´ 1 .

Original: Then by our definition of PSR rank (c.f. Definition 3), the subspace of R | U h | spanned by t x i u i P I has dimension less than or equal to d ¯ θ .
Revised: Then by our definition of PSR rank (c.f. Definition 3), the subspace of R | U h | spanned by t x i u i P I has dimension less than or equal to d ¯ θ .

Original: We take q θ P ∆ p
Revised: We take q θ P ∆ p

Original: I q as
Revised: I q as

Original: [Equation]
Revised: [Equation]

Original: Therefore, applying Proposition E.6 to function family t f θ u θ P Θ , vector family t x i u i P I , and distribution family t q θ u θ P Θ gives
Revised: Therefore, applying Proposition E.6 to function family t f θ u θ P Θ , vector family t x i u i P I , and distribution family t q θ u θ P Θ gives

Original: [Equation]
Revised: [Equation]

Original: Combining Eq. (50), (52), and (49) yields
Revised: ,π q„ µ

Original: E p θ,π q„ µ “ D TV ` P πθ , P [Table]
Revised: [Equation]

Original: ÿ
Revised: E

Original: [Table] [Table]
Revised: [Equation] [Equation]

Original: ‰ ď c [Table] [Table]
Revised: Combining Eq.

Original: ı
Revised: (50), (52), and (49) yields

Original: [Table]
Revised: [Equation]

Original: c

Original: [Table] [Table] [Table] [Table]
Revised: [Equation]

Original: [Table]
Revised: [Equation]

Original: ıı¸ď
Revised: ıı¸

Original: [Table]
Revised: [Equation]

Original: ˘ff¸“ b
Revised: ˘ff¸

Original: [Table]

Original: Hd

Original: [Table]
Revised: [Equation]

Original: , where the third inequality is due to Cauchy-Schwarz inequality, and the fourth inequality is due to
Revised: where the third inequality is due to Cauchy-Schwarz inequality, and the fourth inequality is due to

Original: Proposition F.2.
Revised: Proposition F.2.

Original: This completes the proof of Proposition I.1.
Revised: This completes the proof of Proposition I.1.

Original: I. P ROOF OF T HEOREM H.2 (T HEOREM 10)
Revised: I. P ROOF OF T HEOREM H.2 (T HEOREM 10)

Original: According to Theorem H.1, in order to prove Theorem H.2 (Theorem 10), we only need to bound the coefficients edec γ p Θ q for γ ą 0 .
Revised: According to Theorem H.1, in order to prove Theorem H.2 (Theorem 10), we only need to bound the coefficients edec γ p Θ q for γ ą 0 .

Original: In the following, we bound edec by psc est using the idea of Chen et al.
Revised: In the following, we bound edec by psc est using the idea of Chen et al.

Original: (2022, Proposition 6).
Revised: (2022, Proposition 6).

Original: Recall that edec is defined in (4.2).
Revised: Recall that edec is defined in (4.2).

Original: By strong duality (c.f. Theorem C.2), we have 
Revised: By strong duality (c.f. Theorem C.2), we have edec γ p Θ , µ q

Original: [Equation]
Revised: [Equation]

Original: Note that |V θ p
Revised: Note that |

Original: [Equation]
Revised: [Equation]

Original: , P

Original: [Equation]

Original: θ ˘ .
Revised: ˘ .

Original: Therefore, we can take p out “ p µ , where p µ is defined as p µ p π q “ µ pt θ : π θ “ π uq .
Revised: Therefore, we can take p out “ p µ , where p µ is defined as p µ p π q “ µ pt θ : π θ “ π uq .

Original: Then for a fixed α P p
Revised: Then for a fixed α P p

Original: [Equation]
Revised: [Equation]

Original: 1 q
Revised: 1 q

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: where the equality is due to our choice of pµ :
Revised: where the equality is due to our choice of p

Original: [Equation]
Revised: [Equation]

Revised: and the last inequality is due to AM-GM inequality.

Original: Therefore, we can take p exp “ αp µ ` p 1 ´ α q p e P ∆ p
Revised: Therefore, we can take p exp “ αp µ ` p 1 ´ α q p e P ∆ p

Original: Π q , where p e is given by p e p π q “ µ pt θ : φ ˛ π θ “ π uq , 22 and using this choice of p exp and p out in Eq.
Revised: Π q , where p e is given by p e p π q “ µ pt θ : φ ˛ π θ “ π uq , 22 and using this choice of p exp and p out in Eq.

Original: (53) and using Eq.
Revised: (53) and using Eq.

Original: , we get
Revised: , we get

Original: [Equation]
Revised: [Equation]

Original: Recall that psc est γ has been bounded in Theorem H.6.
Revised: Recall that psc est γ has been bounded in Theorem H.6.

Original: Taking α “ 3 { 4 yields edec γ p Θ , µ q ď p 8Λ 2 B dAU A H 2 ` 1 q{ γ .
Revised: Taking α “ 3 { 4 yields edec γ p Θ , µ q ď p 8Λ 2 B dAU A H 2 ` 1 q{ γ .

Original: This completes the proof of Theorem H.2.
Revised: This completes the proof of Theorem H.2.

Original: I. P ROOF OF T HEOREM H. 
Revised: I. P ROOF OF T HEOREM H.

Original: To prove Theorem H.4, due to Theorem H.3, we only need to bound the coefficients amdec γ p Θ , p µ q for all p µ P ∆ p Θ q .
Revised: To prove Theorem H.4, due to Theorem H.3, we only need to bound the coefficients amdec γ p Θ , p µ q for all p µ P ∆ p Θ q .

Original: By strong duality (c.f. Theorem C.2), we have amdec 
Revised: By strong duality (c.f. Theorem C.2), we have amdec γ p Θ , p µ q “

Original: [Equation]
Revised: [Equation]

Original: where the first inequality is because we take µ out “ p µ in inf µ out , and the second inequality is because we can take p exp P ∆ p Π q corresponds to φ ˛ π with π „ ν .
Revised: where the first inequality is because we take µ out “ p µ in inf µ out [Equation] and the second inequality is because we can take p exp P ∆ p Π q corresponds to φ ˛ π with π „ ν .

Original: Applying Proposition I.2 gives
Revised: Applying Proposition I.2 gives

Original: [Equation]
Revised: [Equation]

Original: and thus the proof of Theorem H.4.
Revised: and thus the proof of Theorem H.4.