Original: Theorem 5 (Robust version of Theorem 3) Assume concentrability of an optimal policy
Revised: Theorem 5 (Robust version of Theorem 3) Assume concentrability of an optimal policy

Original: π ⋆ (Definition 1) and let w ⋆ ( s, a ) =
Revised: π ⋆ (Definition 1) and let w ⋆ ( s, a ) =

Original: d π ⋆ ( s, a ) /µ
Revised: d π ⋆ ( s, a ) /µ

Original: ( s, a ) , v ⋆ =
Revised: ( s, a ) , v ⋆ =

Original: J ( π ⋆ ) .
Revised: J ( π ⋆ ) .

Original: Assume that | v ( s ) |≤ B v for v ∈ V and 0 ≤ w ( s, a ) ≤ B w for w ∈ W .
Revised: Assume that | v ( s ) |≤ B v for v ∈ V and 0 ≤ w ( s, a ) ≤ B w for w ∈ W .

Original: Moreover, assume (75) and (76) hold.
Revised: Moreover, assume (75) and (76) hold.

Original: Then for any fixed δ > 0 , policy ˆ π returned by Algorithm 2 (where ( ˆ w, ˆ v ) satisfies (76) with ˆ L =
Revised: Then for any fixed δ > 0 , policy ˆ π returned by Algorithm 2 (where ( ˆ w, ˆ v ) satisfies (76) with ˆ L =

Original: ˆ L CBAL instead of the exact max-min solution as in (8) ) achieves
Revised: ˆ L CBAL instead of the exact max-min solution as in (8) ) achieves

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ , where ϵ opt =
Revised: with probability at least 1 − δ , where ϵ opt =

Original: ϵ o,w + ϵ o,v and ϵ mis =
Revised: ϵ o,w + ϵ o,v and ϵ mis =

Original: ( B w + B v + 3) ϵ r,w,w ⋆ + ( B w + 1) ϵ r,v .
Revised: ( B w + B v + 3) ϵ r,w,w ⋆ + ( B w + 1) ϵ r,v .

Original: Note that we only present the robustness result for contextual bandit settings for conciseness.
Revised: Note that we only present the robustness result for contextual bandit settings for conciseness.

Original: Similar results also hold for MAB, model-based MDP, and model-free MDP settings.
Revised: Similar results also hold for MAB, model-based MDP, and model-free MDP settings.

Original: Proof.
Revised: Proof.

Original: Note that the proof is almost the same as Appendix D.3, except for part (II) of Lemma 6.
Revised: Note that the proof is almost the same as Appendix D.3, except for part (II) of Lemma 6.

Original: Part (I) of Lemma 6 directly holds in model misspecification and optimization error by the same proof.
Revised: Part (I) of Lemma 6 directly holds in model misspecification and optimization error by the same proof.

Original: Now we show part (II) of Lemma 6.
Revised: Now we show part (II) of Lemma 6.

Original: For convenience, we use L , ˆ L to represent L CBAL , ˆ L CBAL respectively, and let ϵ stat = 

Revised: For convenience, we use L , ˆ L to represent L CBAL , ˆ L CBAL respectively, and let ϵ stat =

Original: ( B w + 1) 2 ( B v + 1) (cid:113) log( |W||V| /δ )
Revised: ( B w + 1) 2 ( B v + 1) (cid:113) log( |W||V| /δ )

Original: N .
Revised: N .

Original: Also, let v
Revised: Also, let v

Original: [Equation]
Revised: [Equation]

Original: By the same argument as in Appendix D.3, ( w ⋆ , v ⋆ ) ∈ argmax w ≥ 0 argmin v L ( w, v ) .
Revised: By the same argument as in Appendix D.3, ( w ⋆ , v ⋆ ) ∈ argmax w ≥ 0 argmin v L ( w, v ) .

Original: Also, we can decompose L ( w ⋆ , v ⋆ ) − L ( ˆ w, v ⋆ ) according to
Revised: Also, we can decompose L ( w ⋆ , v ⋆ ) − L ( ˆ w, v ⋆ ) according to

Original: [Equation]
Revised: [Equation]

Original: where ˆ v ( w ) =
Revised: where ˆ v ( w ) =

Original: arg min v ∈V ˆ L ( w, v ) .
Revised: arg min v ∈V ˆ L ( w, v ) .

Original: Each term is bounded as follows:
Revised: Each term is bounded as follows:

Original: • T 1 = 0 because w ⋆ satisfies the optimization constraints.
Revised: • T 1 = 0 because w ⋆ satisfies the optimization constraints.

Original: • T 2 ≤ ( B w + B v + 3) ϵ r,w,w ⋆ by Lemma 12.
Revised: • T 2 ≤ ( B w + B v + 3) ϵ r,w,w ⋆ by Lemma 12.

Original: • T 3 ≤ ϵ stat due to part (I) of Lemma 6.
Revised: • T 3 ≤ ϵ stat due to part (I) of Lemma 6.

Original: • T 4 ≤ ϵ o,w because max w ∈W min v ∈V ˆ L ( w, v ) − min v ∈V ˆ L ( ˆ w, v ) ≤ ϵ o,w and w ⋆ W ∈ W .
Revised: • T 4 ≤ ϵ o,w because max w ∈W min v ∈V ˆ L ( w, v ) − min v ∈V ˆ L ( ˆ w, v ) ≤ ϵ o,w and w ⋆ W ∈ W .

Original: • T 5 ≤ ϵ o,v because ˆ L ( ˆ w, ˆ v ) − min v ∈V ˆ L ( ˆ w, v ) ≤ ϵ o,v and v ⋆ V ∈ V .
Revised: • T 5 ≤ ϵ o,v because ˆ L ( ˆ w, ˆ v ) − min v ∈V ˆ L ( ˆ w, v ) ≤ ϵ o,v and v ⋆ V ∈ V .

Original: • T 6 ≤ ϵ stat due to part (I) of Lemma 6.
Revised: • T 6 ≤ ϵ stat due to part (I) of Lemma 6.

Original: • T 7 ≤ ( B w + 1) ϵ r,v by Lemma 12.
Revised: • T 7 ≤ ( B w + 1) ϵ r,v by Lemma 12.

Original: Summing up the bounds on each term, we have
Revised: Summing up the bounds on each term, we have

Original: [Equation]
Revised: [Equation]

Original: The remaining steps are the same as Appendix D.3, and we can finally obtain that
Revised: The remaining steps are the same as Appendix D.3, and we can finally obtain that

Original: [Equation]
Revised: [Equation]

Original: Remark 1 Note that the suboptimality caused by model misspecification and optimization error in our algorithm is of order
Revised: Remark 1 Note that the suboptimality caused by model misspecification and optimization error in our algorithm is of order

Original: O ( ϵ opt + ϵ mis ) .
Revised: O ( ϵ opt + ϵ mis ) .

Original: This is much better than the result of Zhan et al.
Revised: This is much better than the result of Zhan et al.

Original: (2022) where the suboptimality caused by model misspecification and optimization error is of order
Revised: (2022) where the suboptimality caused by model misspecification and optimization error is of order

Original: O ( (cid:112) ( ϵ opt + ϵ mis ) /α ) .
Revised: O ( (cid:112) ( ϵ opt + ϵ mis ) /α ) .

Original: Finally, we show and prove the following lemma which is key to the proof of our main theorem (Theorem 5) in this section.
Revised: Finally, we show and prove the following lemma which is key to the proof of our main theorem (Theorem 5) in this section.

Original: This is also similar to Zhan et al.
Revised: This is also similar to Zhan et al.

Original: Lemma 

Revised: Lemma

Original: Under the same setting as in Theorem 5, for any v ∈ V and any w 1 , w 2 ∈ W , it holds that
Revised: Under the same setting as in Theorem 5, for any v ∈ V and any w 1 , w 2 ∈ W , it holds that

Original: [Equation]
Revised: [Equation]

Original: v 2 ∥ 1 ,µ ′ ) .
Revised: v 2 ∥ 1 ,µ ′ ) .

Original: Also, for any v 1 , v 2 ∈ V and any w ∈ W , it holds that
Revised: Also, for any v 1 , v 2 ∈ V and any w ∈ W , it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Recall that
Revised: Recall that

Original: [Equation]
Revised: [Equation]

Original: s, a )] − 1) 2 ] , and in contextual bandits we have ρ = µ .
Revised: s, a )] − 1) 2 ] , and in contextual bandits we have ρ = µ .

Original: Therefore, by definition,
Revised: Therefore, by definition,

Original: [Equation]
Revised: [Equation]

Original: Similarly, we have
Revised: Similarly, we have

Original: [Equation]
Revised: [Equation]

Original: □
Revised: □

Original: G A UXILIARY RESULTS
Revised: G A UXILIARY RESULTS

Original: Theorem 6 (Convergence of MLE for learning transitions (Van de Geer, 2000))
Revised: Theorem 6 (Convergence of MLE for learning transitions (Van de Geer, 2000))

Original: Given a realizable model class P = { P : ( S , A ) → ∆( S ) } that contains the true model P ⋆ and a dataset
Revised: Given a realizable model class P = { P : ( S , A ) → ∆( S ) } that contains the true model P ⋆ and a dataset

Original: D m = { ( s i , a i , s ′ i ) } with ( s i , a i ) iid ∼ µ, s ′ i ∼ P
Revised: D m = { ( s i , a i , s ′ i ) } with ( s i , a i ) iid ∼ µ, s ′ i ∼ P

Original: ⋆ ( ·| s i , a i ) , let ˆ P be
Revised: ⋆ ( ·| s i , a i ) , let ˆ P be

Original: [Equation]
Revised: [Equation]

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original: i = ln P ( s ′ i | s i , a i ) .
Revised: i = ln P ( s ′ i | s i , a i ) .

Original: Fix the failure probability δ > 0 .
Revised: Fix the failure probability δ > 0 .

Original: Then, with probability at least 1 − δ , we have the following concentration on the squared Hellinger distance between ˆ P and P ⋆ :
Revised: Then, with probability at least 1 − δ , we have the following concentration on the squared Hellinger distance between ˆ P and P ⋆ :

Original: [Equation]
Revised: [Equation]

Original: Lemma 

Revised: Lemma

Original: For any 0 ≤ b i ≤ B and x i , y i ≥ 0 for i ∈ { 0 , . . . , n } , the following holds
Revised: For any 0 ≤ b i ≤ B and x i , y i ≥ 0 for i ∈ { 0 , . . . , n } , the following holds

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We expend the left-hand side of (77), use Cauchy-Schwarz inequality
Revised: We expend the left-hand side of (77), use Cauchy-Schwarz inequality

Original: [Equation]
Revised: [Equation]

Original: and then complete the square:
Revised: and then complete the square:

Original: [Equation]
Revised: [Equation]

Original: □
Revised: □

Original: Lemma 

Revised: Lemma

Original: For any two arbitrary sets X , Y , let f ( · , · ) :
Revised: For any two arbitrary sets X , Y , let f ( · , · ) :

Original: X × Y → R be an arbitrary function.
Revised: X × Y → R be an arbitrary function.

Original: Let X 

Revised: Let X

Original: = { x ∈ X | inf y ∈Y f
Revised: = { x ∈ X | inf y ∈Y f

Original: ( x, y ) >
Revised: ( x, y ) >

Original: −∞} and assume X 0 is non-empty.
Revised: −∞} and assume X 0 is non-empty.

Original: For any x ∈ X 0 , assume there exists y ∗ ( x )
Revised: For any x ∈ X 0 , assume there exists y ∗ ( x )

Original: ∈ Y s.t.
Revised: ∈ Y s.t.

Original: f ( x, y ∗ ( x ))
Revised: f ( x, y ∗ ( x ))

Original: =
Revised: =

Original: min
Revised: min

Original: y ∈Y f ( x, y ) .
Revised: y ∈Y f ( x, y ) .

Original: Also, let X ∗ p = { x ∈ X 0 | x ∈ arg max x ∈X 0 f
Revised: Also, let X ∗ p = { x ∈ X 0 | x ∈ arg max x ∈X 0 f

Original: ( x, y ⋆ ( x )) } and assume X ∗ p is non-empty.
Revised: ( x, y ⋆ ( x )) } and assume X ∗ p is non-empty.

Original: For a nonnegative function A ( · ) on X , let
Revised: For a nonnegative function A ( · ) on X , let

Original: X ∗ = { x ∈ X ∗ p | A ( x ) = 0 } and assume X ∗ is non-empty.
Revised: X ∗ = { x ∈ X ∗ p | A ( x ) = 0 } and assume X ∗ is non-empty.

Original: Define f AL ( x, y )
Revised: Define f AL ( x, y )

Original: =
Revised: =

Original: f ( x, y ) − A ( x ) .
Revised: f ( x, y ) − A ( x ) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: and for any x ∈ X 0 ,
Revised: and for any x ∈ X 0 ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Note that for any fixed x , f AL ( x, y ) is a constant shift of f ( x, y ) , which implies that inf y ∈Y f ( x, y ) >
Revised: Note that for any fixed x , f AL ( x, y ) is a constant shift of f ( x, y ) , which implies that inf y ∈Y f ( x, y ) >

Original: −∞ ⇐⇒ inf y ∈Y f AL ( x, y ) >
Revised: −∞ ⇐⇒ inf y ∈Y f AL ( x, y ) >

Original: −∞ .
Revised: −∞ .

Original: This also implies that for any x ∈ X 0 , arg min y ∈Y f ( x, y ) =
Revised: This also implies that for any x ∈ X 0 , arg min y ∈Y f ( x, y ) =

Original: arg min y ∈Y f AL ( x, y ) .
Revised: arg min y ∈Y f AL ( x, y ) .

Original: For any x ∈ X 0 , let y ∗ ( x ) denote any one of y ∈ Y s.t.
Revised: For any x ∈ X 0 , let y ∗ ( x ) denote any one of y ∈ Y s.t.

Original: f ( x, y ∗ ( x ))
Revised: f ( x, y ∗ ( x ))

Original: =
Revised: =

Original: min
Revised: min

Original: y ∈Y f ( x, y ) .
Revised: y ∈Y f ( x, y ) .

Original: Now for any x ∗ ∈ X ∗ , we have
Revised: Now for any x ∗ ∈ X ∗ , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: For the other direction, given any x 0 ∈ arg max x ∈X 0 min y ∈Y f AL ( x, y ) , we have
Revised: For the other direction, given any x 0 ∈ arg max x ∈X 0 min y ∈Y f AL ( x, y ) , we have

Original: [Equation]
Revised: [Equation]

Original: Fix any x ∗ ∈
Revised: Fix any x ∗ ∈

Original: X ∗ ⊆
Revised: X ∗ ⊆

Original: X 0 , we have f ( x ∗ , y ∗ ( x ∗ )) ≥ f ( x 0 , y ∗ ( x 0 )) and − A ( x ∗ ) ≥ − A ( x 0 ) by definition.
Revised: X 0 , we have f ( x ∗ , y ∗ ( x ∗ )) ≥ f ( x 0 , y ∗ ( x 0 )) and − A ( x ∗ ) ≥ − A ( x 0 ) by definition.

Original: Now assume x 0 / ∈ X ∗ .
Revised: Now assume x 0 / ∈ X ∗ .

Original: Then either f ( x ∗ , y ∗ ( x ∗ )) >
Revised: Then either f ( x ∗ , y ∗ ( x ∗ )) >

Original: f ( x 0 , y ∗ ( x 0 ))
Revised: f ( x 0 , y ∗ ( x 0 ))

Original: if x 0 / ∈
Revised: if x 0 / ∈

Original: X ∗ p , or − A ( x ∗ ) > − A ( x 0 ) if x 0 ∈ X ∗ p \X ∗ .
Revised: X ∗ p , or − A ( x ∗ ) > − A ( x 0 ) if x 0 ∈ X ∗ p \X ∗ .

Original: Either one of the above two conditions implies that f ( x ∗ , y ∗ ( x ∗ ))
Revised: Either one of the above two conditions implies that f ( x ∗ , y ∗ ( x ∗ ))

Original: [Equation]
Revised: [Equation]

Original: A ( x ∗ ) >
Revised: A ( x ∗ ) >

Original: f ( x 0 , y ∗ ( x 0 ))
Revised: f ( x 0 , y ∗ ( x 0 ))

Original: − A ( x 0 ) =
Revised: − A ( x 0 ) =

Original: ⇒
Revised: ⇒

Original: f AL ( x ∗ , y ∗ ( x ∗ )) >
Revised: f AL ( x ∗ , y ∗ ( x ∗ )) >

Original: f AL ( x 0 , y ∗ ( x 0 )) , which contradicts with (78).
Revised: f AL ( x 0 , y ∗ ( x 0 )) , which contradicts with (78).

Original: Therefore, x 0 ∈ X ∗ .
Revised: Therefore, x 0 ∈ X ∗ .

Original: □
Revised: □