Original: O N THE I NTERPLAY B ETWEEN M ISSPECIFICATION AND S UB - OPTIMALITY G AP : F ROM L INEAR C ONTEXTUAL B ANDITS TO L INEAR MDP S
Revised: O N THE I NTERPLAY B ETWEEN M ISSPECIFICATION AND S UB - OPTIMALITY G AP : F ROM L INEAR C ONTEXTUAL B ANDITS TO L INEAR MDP S

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We study linear contextual bandits in the misspecified setting, where the expected reward function can be approximated by a linear function class up to a bounded misspecification level ζ > 0 .
Revised: We study linear contextual bandits in the misspecified setting, where the expected reward function can be approximated by a linear function class up to a bounded misspecification level ζ > 0 .

Original: We propose an algorithm based on a novel data selection scheme, which only selects the contextual vectors with large uncertainty for online regression.
Revised: We propose an algorithm based on a novel data selection scheme, which only selects the contextual vectors with large uncertainty for online regression.

Original: We show that, when the misspecification level ζ is dominated by (cid:101) O (∆ / √ d ) with ∆ being the minimal sub-optimality gap and d being the dimension of the contextual vectors, our algorithm enjoys the same gap-dependent regret bound (cid:101)
Revised: We show that, when the misspecification level ζ is dominated by (cid:101) O (∆ / √ d ) with ∆ being the minimal sub-optimality gap and d being the dimension of the contextual vectors, our algorithm enjoys the same gap-dependent regret bound (cid:101)

Original: O ( d 2 / ∆)
Revised: O ( d 2 / ∆)

Original: as in the well-specified setting up to logarithmic factors.
Revised: as in the well-specified setting up to logarithmic factors.

Original: Together with a lower bound adapted from Du et al.
Revised: Together with a lower bound adapted from Du et al.

Original: Lattimore et al.
Revised: Lattimore et al.

Original: (2020), our result suggests an interplay between misspecification level and the sub-optimality gap: (1) the linear contextual bandit model is efficiently learnable when ζ ≤ (cid:101) O
Revised: (2020), our result suggests an interplay between misspecification level and the sub-optimality gap: (1) the linear contextual bandit model is efficiently learnable when ζ ≤ (cid:101) O

Original: (∆ / √ d ) ; and (2) it is not efficiently learnable when ζ ≥ (cid:101) Ω(∆ / √ d ) .
Revised: (∆ / √ d ) ; and (2) it is not efficiently learnable when ζ ≥ (cid:101) Ω(∆ / √ d ) .

Original: We also extend our algorithm to reinforcement learning with linear Markov decision processes (linear MDPs), and obtain a parallel result of gap-dependent regret.
Revised: We also extend our algorithm to reinforcement learning with linear Markov decision processes (linear MDPs), and obtain a parallel result of gap-dependent regret.

Original: Experiments on both synthetic and real-world datasets corroborate our theoretical results.
Revised: Experiments on both synthetic and real-world datasets corroborate our theoretical results.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Linear contextual bandits (Li et al., 2010; Chu et al., 2011; Abbasi-Yadkori et al., 2011; Agrawal &
Revised: Linear contextual bandits (Li et al., 2010; Chu et al., 2011; Abbasi-Yadkori et al., 2011; Agrawal &

Original: Goyal, 2013) have been extensively studied when the reward function can be represented as a linear function of the contextual vectors.
Revised: Goyal, 2013) have been extensively studied when the reward function can be represented as a linear function of the contextual vectors.

Original: However, such a well-specified linear model assumption sometimes does not hold in practice.
Revised: However, such a well-specified linear model assumption sometimes does not hold in practice.

Original: This motivates the study of misspecified linear models.
Revised: This motivates the study of misspecified linear models.

Original: In particular, we only assume that the reward function can be approximated by a linear function up to some worstcase error ζ called misspecification level .
Revised: In particular, we only assume that the reward function can be approximated by a linear function up to some worstcase error ζ called misspecification level .

Original: Existing algorithms for misspecified linear contextual bandits (Lattimore et al., 2020; Foster et al., 2020) can only achieve an (cid:101)
Revised: Existing algorithms for misspecified linear contextual bandits (Lattimore et al., 2020; Foster et al., 2020) can only achieve an (cid:101)

Original: O
Revised: O

Original: ( d √ K + ζK √ d log K ) regret bound, where K is the total number of rounds and d is the dimension of the contextual vector.
Revised: ( d √ K + ζK √ d log K ) regret bound, where K is the total number of rounds and d is the dimension of the contextual vector.

Original: Such a regret, however, suggests that the performance of these algorithms will degenerate to be linear in K when K is sufficiently large.
Revised: Such a regret, however, suggests that the performance of these algorithms will degenerate to be linear in K when K is sufficiently large.

Original: The reason for this performance degeneration is because existing algorithms, such as OFUL (Abbasi-Yadkori et al., 2011) and linear Thompson sampling (Agrawal & Goyal, 2013), utilize all the collected data without selection.
Revised: The reason for this performance degeneration is because existing algorithms, such as OFUL (Abbasi-Yadkori et al., 2011) and linear Thompson sampling (Agrawal & Goyal, 2013), utilize all the collected data without selection.

Original: This makes these algorithms vulnerable to “outliers” caused by the misspecified model.
Revised: This makes these algorithms vulnerable to “outliers” caused by the misspecified model.

Original: Meanwhile, the aforementioned results do not consider the sub-optimality gap in the expected reward between the best arm and the second best arm.
Revised: Meanwhile, the aforementioned results do not consider the sub-optimality gap in the expected reward between the best arm and the second best arm.

Original: Intuitively speaking, if the sub-optimality gap is smaller than the misspecification level, there is no hope to obtain a sublinear regret.
Revised: Intuitively speaking, if the sub-optimality gap is smaller than the misspecification level, there is no hope to obtain a sublinear regret.

Original: Therefore, it is sensible to take into account the sub-optimality gap in the misspecified setting, and pursue a gap-dependent regret bound.
Revised: Therefore, it is sensible to take into account the sub-optimality gap in the misspecified setting, and pursue a gap-dependent regret bound.

Original: The same misspecification issue also appears in reinforcement learning with linear function approximation, when a linear function cannot exactly represent the transition kernel or value function of the underlying MDP.
Revised: The same misspecification issue also appears in reinforcement learning with linear function approximation, when a linear function cannot exactly represent the transition kernel or value function of the underlying MDP.

Original: In this case, Du et al.
Revised: In this case, Du et al.

Original: (2019) provided a negative result showing that if the misspecification level is larger than a certain threshold, any RL algorithm will suffer from an exponentially large sample complexity.
Revised: (2019) provided a negative result showing that if the misspecification level is larger than a certain threshold, any RL algorithm will suffer from an exponentially large sample complexity.

Original: This result was later revisited in the stochastic linear bandit setting by Lattimore et al.
Revised: This result was later revisited in the stochastic linear bandit setting by Lattimore et al.

Original: (2020), which shows that a large misspecification error will make the bandit model not efficiently learnable.
Revised: (2020), which shows that a large misspecification error will make the bandit model not efficiently learnable.

Original: However, these results cannot well explain the tremendous success of deep reinforcement learning on various tasks (Mnih et al., 2013; Schulman et al., 2015;
Revised: However, these results cannot well explain the tremendous success of deep reinforcement learning on various tasks (Mnih et al., 2013; Schulman et al., 2015;

Original: 2017), where the
Revised: 2017), where the

Original: deep
Revised: deep

Original: neural networks
Revised: neural networks

Original: are used as function approximators with misspecification error
Revised: are used as function approximators with misspecification error

Original: In this paper, we aim to understand the role of model misspecification in linear contextual bandits through the lens of sub-optimality gap.
Revised: In this paper, we aim to understand the role of model misspecification in linear contextual bandits through the lens of sub-optimality gap.

Original: By proposing a new algorithm with data selection, we can achieve a constant regret bound for such a problem.
Revised: By proposing a new algorithm with data selection, we can achieve a constant regret bound for such a problem.

Original: We also extend our algorithm to the linear Markov decision processes (Jin et al., 2020) and obtain a regret bound of similar flavor.
Revised: We also extend our algorithm to the linear Markov decision processes (Jin et al., 2020) and obtain a regret bound of similar flavor.

Original: Our contributions are highlighted as follows:
Revised: Our contributions are highlighted as follows:

Original: • We propose a new algorithm called DS-OFUL (Data Selection OFUL).
Revised: • We propose a new algorithm called DS-OFUL (Data Selection OFUL).

Original: DS-OFUL only learns from the data with large uncertainty.
Revised: DS-OFUL only learns from the data with large uncertainty.

Original: We prove an (cid:101) O
Revised: We prove an (cid:101) O

Original: ( d 2 ∆ − 1 ) gap-dependent regret 1 bound when the misspecification level is small (i.e., ζ = (cid:101)
Revised: ( d 2 ∆ − 1 ) gap-dependent regret 1 bound when the misspecification level is small (i.e., ζ = (cid:101)

Original: O (∆ / √ d ) ) and the minimal sub-optimality gap ∆ is known.
Revised: O (∆ / √ d ) ) and the minimal sub-optimality gap ∆ is known.

Original: Our regret bound improves upon the gap-dependent regret in the well-specified setting (Abbasi-Yadkori et al., 2011) by a logarithmic factor.
Revised: Our regret bound improves upon the gap-dependent regret in the well-specified setting (Abbasi-Yadkori et al., 2011) by a logarithmic factor.

Original: To the best of our knowledge, this is the first constant gap-dependant regret bound for misspecified linear contextual bandits, even assuming a known minimal sub-optimality gap.
Revised: To the best of our knowledge, this is the first constant gap-dependant regret bound for misspecified linear contextual bandits, even assuming a known minimal sub-optimality gap.

Original: • We also prove a gap-dependent lower bound following the lower bound proof technique in Du et al.
Revised: • We also prove a gap-dependent lower bound following the lower bound proof technique in Du et al.

Original: Lattimore et al.
Revised: Lattimore et al.

Original: This together with the upper bound suggests an interplay between the misspecification level and the sub-optimality gap: the linear contextual bandit is efficiently learnable if ζ ≤ (cid:101) O (∆ / √ d ) while it is not efficiently learnable if ζ ≥ (cid:101) Ω(∆ / √ d ) .
Revised: This together with the upper bound suggests an interplay between the misspecification level and the sub-optimality gap: the linear contextual bandit is efficiently learnable if ζ ≤ (cid:101) O (∆ / √ d ) while it is not efficiently learnable if ζ ≥ (cid:101) Ω(∆ / √ d ) .

Original: • We extend the same idea to the misspecified linear MDP, and propose an algorithm called DSLSVI (Data-Selection LSVI).
Revised: • We extend the same idea to the misspecified linear MDP, and propose an algorithm called DSLSVI (Data-Selection LSVI).

Original: DS-LSVI enjoys a gap-dependent regret bound, which suggests a similar interplay between the misspecification level and sub-optimality gap in episodic MDPs to achieve a logarithmic regret bound (cid:101)
Revised: DS-LSVI enjoys a gap-dependent regret bound, which suggests a similar interplay between the misspecification level and sub-optimality gap in episodic MDPs to achieve a logarithmic regret bound (cid:101)

Original: O ( H 5 d 3 ∆ − 1 log( K )) • Finally, we conduct experiments on the linear contextual bandit with both synthetic and real datasets, and demonstrate the superior performance of DS-OFUL algorithm.
Revised: O ( H 5 d 3 ∆ − 1 log( K )) • Finally, we conduct experiments on the linear contextual bandit with both synthetic and real datasets, and demonstrate the superior performance of DS-OFUL algorithm.

Original: This corroborates our theoretical results.
Revised: This corroborates our theoretical results.

Original: Notation.
Revised: Notation.

Original: Scalars and constants are denoted by lower and upper case letters, respectively.
Revised: Scalars and constants are denoted by lower and upper case letters, respectively.

Original: Vectors are denoted by lower case bold face letters x , and matrices by upper case bold face letters A .
Revised: Vectors are denoted by lower case bold face letters x , and matrices by upper case bold face letters A .

Original: We denote by [ k ] the set { 1 , 2 , · · · , k } for positive integers k .
Revised: We denote by [ k ] the set { 1 , 2 , · · · , k } for positive integers k .

Original: For two non-negative sequence { a n } , { b n } , a n = O ( b n ) means that there exists a positive constant C such that a n ≤ Cb n , and we use (cid:101) O ( · ) to hide the log factor in O ( · ) other than number of rounds T or episode K ; a n = Ω( b n ) means that there exists a positive constant C such that a n ≥ Cb n , and we use (cid:101) Ω( · ) to hide the log factor.
Revised: For two non-negative sequence { a n } , { b n } , a n = O ( b n ) means that there exists a positive constant C such that a n ≤ Cb n , and we use (cid:101) O ( · ) to hide the log factor in O ( · ) other than number of rounds T or episode K ; a n = Ω( b n ) means that there exists a positive constant C such that a n ≥ Cb n , and we use (cid:101) Ω( · ) to hide the log factor.

Original: For a vector x ∈ R d and a positive semi-definite matrix A ∈ R d × d , we define ∥ x ∥ 2 A =
Revised: For a vector x ∈ R d and a positive semi-definite matrix A ∈ R d × d , we define ∥ x ∥ 2 A =

Original: x ⊤ Ax .
Revised: x ⊤ Ax .

Original: For any set C , we use |C| to denote its cardinality.
Revised: For any set C , we use |C| to denote its cardinality.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In this section, we review the related work for misspecified linear bandits and misspecified reinforcement learning.
Revised: In this section, we review the related work for misspecified linear bandits and misspecified reinforcement learning.

Original: We defer more related work on the function approximation in bandits and RL to Appendix A.
Revised: We defer more related work on the function approximation in bandits and RL to Appendix A.

Original: Misspecified Linear Bandits.
Revised: Misspecified Linear Bandits.

Original: Ghosh et al.
Revised: Ghosh et al.

Original: (2017) is probably the first work considering the misspecified linear bandits, which shows that the OFUL (Abbasi-Yadkori et al., 2011) algorithm cannot achieve a sublinear regret in the presence of misspecification.
Revised: (2017) is probably the first work considering the misspecified linear bandits, which shows that the OFUL (Abbasi-Yadkori et al., 2011) algorithm cannot achieve a sublinear regret in the presence of misspecification.

Original: They, therefore, proposed a new algorithm with a hypothesis testing module for linearity to determine whether to use OFUL (AbbasiYadkori et al., 2011) or the multi-armed UCB algorithm.
Revised: They, therefore, proposed a new algorithm with a hypothesis testing module for linearity to determine whether to use OFUL (AbbasiYadkori et al., 2011) or the multi-armed UCB algorithm.

Original: Their algorithm enjoys the same performance guarantee as OFUL in the well-specified setting and can avoid the linear regret under certain misspecification setting.
Revised: Their algorithm enjoys the same performance guarantee as OFUL in the well-specified setting and can avoid the linear regret under certain misspecification setting.

Original: Lattimore et al.
Revised: Lattimore et al.

Original: (2020) proposed a phase-elimination algorithm for misspecified stochastic linear bandits, which achieves (cid:101) O ( √ dK + ζK √ d ) regret bound.
Revised: (2020) proposed a phase-elimination algorithm for misspecified stochastic linear bandits, which achieves (cid:101) O ( √ dK + ζK √ d ) regret bound.

Original: For contextual linear bandits, both Lattimore et al.
Revised: For contextual linear bandits, both Lattimore et al.

Original: (2020) and Foster et al.
Revised: (2020) and Foster et al.

Original: (2020) proved a (cid:101) O
Revised: (2020) proved a (cid:101) O

Original: ( d √ K + ζK √ d ) regret bound.
Revised: ( d √ K + ζK √ d ) regret bound.

Original: Takemura et al.
Revised: Takemura et al.

Original: Vial et al.
Revised: Vial et al.

Original: (2022) also provide a similar regret bound without the knowledge of the misspecification level.
Revised: (2022) also provide a similar regret bound without the knowledge of the misspecification level.

Original: Van Roy & Dong (2019) proved a lower bound of sample complexity, which suggests when ζ √ d ≥ (cid:112) 8 log |D| , any best arm identification algorithm will suffer a Ω(2 d ) sample complexity, where D is the decision set.
Revised: Van Roy & Dong (2019) proved a lower bound of sample complexity, which suggests when ζ √ d ≥ (cid:112) 8 log |D| , any best arm identification algorithm will suffer a Ω(2 d ) sample complexity, where D is the decision set.

Original: When the outcome is deterministic and does not contain noise, they provided an algorithm using (cid:101)
Revised: When the outcome is deterministic and does not contain noise, they provided an algorithm using (cid:101)

Original: O ( d ) sample complexity to identify a ∆ -optimal arm when ζ ≤ ∆ / √ d .
Revised: O ( d ) sample complexity to identify a ∆ -optimal arm when ζ ≤ ∆ / √ d .

Original: Lattimore et al.
Revised: Lattimore et al.

Original: (2020) also mentioned that if ζ √ d ≤ ∆ , there exists a best arm identification algorithm can only use (cid:101)
Revised: (2020) also mentioned that if ζ √ d ≤ ∆ , there exists a best arm identification algorithm can only use (cid:101)

Original: O ( d ) arms to find a ∆ -optimal arm with the knowledge of ζ .
Revised: O ( d ) arms to find a ∆ -optimal arm with the knowledge of ζ .

Original: Note that although the exponential sample complexity lower bound for best arm identification can be translated into a regret lower bound in linear contextual bandits, the algorithms for best-arm identification and the corresponding upper bounds cannot be easily extended to linear contextual bandits.
Revised: Note that although the exponential sample complexity lower bound for best arm identification can be translated into a regret lower bound in linear contextual bandits, the algorithms for best-arm identification and the corresponding upper bounds cannot be easily extended to linear contextual bandits.

Original: Besides these works on misspecification, He et al.
Revised: Besides these works on misspecification, He et al.

Original: (2022) studies the linear contextual bandits with adversarial corruptions, which can be considered as a similar setting of misspecification.
Revised: (2022) studies the linear contextual bandits with adversarial corruptions, which can be considered as a similar setting of misspecification.

Original: They assume the summation of the approximation error over all K rounds is bounded by the corruption level C .
Revised: They assume the summation of the approximation error over all K rounds is bounded by the corruption level C .

Original: They proposed an algorithm achieving (cid:101) O
Revised: They proposed an algorithm achieving (cid:101) O

Original: ( d √ K + dC ) regret bound.
Revised: ( d √ K + dC ) regret bound.

Original: However, their result in adversarial corrupted bandits cannot be directly translated to the misspecification setting since letting C =
Revised: However, their result in adversarial corrupted bandits cannot be directly translated to the misspecification setting since letting C =

Original: Kζ will lead to a O ( d √ K + dKζ ) linear regret.
Revised: Kζ will lead to a O ( d √ K + dKζ ) linear regret.

Original: Besides these series of work, Camilleri et al.
Revised: Besides these series of work, Camilleri et al.

Original: (2021) also studies the robustness of kernel bandit algorithms on the misspecification.
Revised: (2021) also studies the robustness of kernel bandit algorithms on the misspecification.

Original: Misspecification in Reinforcement Learning Du et al.
Revised: Misspecification in Reinforcement Learning Du et al.

Original: (2019) showed that having a good representation is insufficient for efficient reinforcement learning unless the approximation error (i.e., misspecification level) by the representation is small enough.
Revised: (2019) showed that having a good representation is insufficient for efficient reinforcement learning unless the approximation error (i.e., misspecification level) by the representation is small enough.

Original: In particular, Du et al.
Revised: In particular, Du et al.

Original: (2019) suggested that a (cid:101) Ω( (cid:112) H/d ) misspecification will lead to Ω(2 H ) sample complexity for RL to find the optimal policy, even with a generative model.
Revised: (2019) suggested that a (cid:101) Ω( (cid:112) H/d ) misspecification will lead to Ω(2 H ) sample complexity for RL to find the optimal policy, even with a generative model.

Original: On the other hand, a series of work (Jin et al., 2020;
Revised: On the other hand, a series of work (Jin et al., 2020;

Original: Zanette et al., 2020b; Foster & Rakhlin, 2020) provided (cid:101) O ( √ T + ζT ) -type regret bound for RL in various settings, ignoring the dependence on the dimension of the feature representation d and the planing horizon H .
Revised: Zanette et al., 2020b; Foster & Rakhlin, 2020) provided (cid:101) O ( √ T + ζT ) -type regret bound for RL in various settings, ignoring the dependence on the dimension of the feature representation d and the planing horizon H .

Original: This suggests that the performance of RL will degenerate as the total number of interactions with the environment T increases.
Revised: This suggests that the performance of RL will degenerate as the total number of interactions with the environment T increases.

Original: Also, these results do not consider the minimal sub-optimality gap of the action-value function.
Revised: Also, these results do not consider the minimal sub-optimality gap of the action-value function.

Original: Du et al.
Revised: Du et al.

Original: (2020) considered the agnostic Q-learning with misspecified linear function approximation.
Revised: (2020) considered the agnostic Q-learning with misspecified linear function approximation.

Original: They proposed an algorithm with the access to a generative model and showed that if ζ ≤ (cid:101) O (∆ / √ d ) , one can find the optimal policy using O ( d ) trajectories.
Revised: They proposed an algorithm with the access to a generative model and showed that if ζ ≤ (cid:101) O (∆ / √ d ) , one can find the optimal policy using O ( d ) trajectories.

Original: Together with the lower bound provided in Du et al.
Revised: Together with the lower bound provided in Du et al.

Original: , it suggests that ζ = (cid:101) O (∆ / √ d ) is a sufficient and necessary condition to achieve a polynominal sample complexity given the access to the generative model.
Revised: , it suggests that ζ = (cid:101) O (∆ / √ d ) is a sufficient and necessary condition to achieve a polynominal sample complexity given the access to the generative model.

Original:  P RELIMINARIES OF L INEAR C ONTEXTUAL B ANDITS
Revised:  P RELIMINARIES OF L INEAR C ONTEXTUAL B ANDITS

Original: We consider a linear contextual bandit problem.
Revised: We consider a linear contextual bandit problem.

Original: In round k ∈ [ K ] , the agent receives a decision set D k ⊂ R d and selects an arm x k ∈ D
Revised: In round k ∈ [ K ] , the agent receives a decision set D k ⊂ R d and selects an arm x k ∈ D

Original: k then observes the reward r k = r ( x k ) + ε k , where r ( · ) : R d (cid:55)→ [0 , 1] is a deterministic expected reward function and ε k is a zero-mean R -sub-Gaussian random noise.
Revised: k then observes the reward r k = r ( x k ) + ε k , where r ( · ) : R d (cid:55)→ [0 , 1] is a deterministic expected reward function and ε k is a zero-mean R -sub-Gaussian random noise.

Original: i.e., E [e λε k | x 1: k , ε 1: k − 1 ] ≤ exp( λ 2 R 2 / 2) , ∀ k ∈
Revised: i.e., E [e λε k | x 1: k , ε 1: k − 1 ] ≤ exp( λ 2 R 2 / 2) , ∀ k ∈

Original: [ K ] , λ ∈ R .
Revised: [ K ] , λ ∈ R .

Original: In this work, we assume that all contextual vector x ∈ D k satisfies ∥ x ∥ 2 ≤ L and the reward function r ( · ) : R d → [0 , 1] can be approximated by a linear function r ( x ) =
Revised: In this work, we assume that all contextual vector x ∈ D k satisfies ∥ x ∥ 2 ≤ L and the reward function r ( · ) : R d → [0 , 1] can be approximated by a linear function r ( x ) =

Original: x ⊤ θ ∗ + η ( x ) , where η ( · ) : R d (cid:55)→ [ − ζ, ζ ] is the unknown misspecification error function.
Revised: x ⊤ θ ∗ + η ( x ) , where η ( · ) : R d (cid:55)→ [ − ζ, ζ ] is the unknown misspecification error function.

Original: We further assume ∥ θ ∗ ∥ 2 ≤ B and for simplicity, we assume B, L ≥ 1 .
Revised: We further assume ∥ θ ∗ ∥ 2 ≤ B and for simplicity, we assume B, L ≥ 1 .

Original: We denote the optimal reward at round k as r ∗ k = max x ∈D k r ( x ) and the optimal arm x ∗ k = argmax x ∈D k r ( x ) .
Revised: We denote the optimal reward at round k as r ∗ k = max x ∈D k r ( x ) and the optimal arm x ∗ k = argmax x ∈D k r ( x ) .

Original: Our goal is to minimize the regret defined by Regret ( K ) :
Revised: Our goal is to minimize the regret defined by Regret ( K ) :

Original: = (cid:80) Kk =1 r ∗
Revised: = (cid:80) Kk =1 r ∗

Original: k − r ( x k ) .
Revised: k − r ( x k ) .

Original: In this paper, we focus on the minimal sub-optimality gap condition.
Revised: In this paper, we focus on the minimal sub-optimality gap condition.

Original: Definition 3.1 (Minimal sub-optimality gap) .
Revised: Definition 3.1 (Minimal sub-optimality gap) .

Original: For each x ∈ D k , the sub-optimality gap ∆ k
Revised: For each x ∈ D k , the sub-optimality gap ∆ k

Original: ( x ) is defined by ∆ k ( x ) :
Revised: ( x ) is defined by ∆ k ( x ) :

Original: = r ∗ k − r ( x ) and the minimal sub-optimality gap ∆ is defined by ∆ := min k ∈ [ K ] , x ∈D k { ∆ k
Revised: = r ∗ k − r ( x ) and the minimal sub-optimality gap ∆ is defined by ∆ := min k ∈ [ K ] , x ∈D k { ∆ k

Original: ( x ) : ∆ k ( x ) > 0 } .
Revised: ( x ) : ∆ k ( x ) > 0 } .

Original: Then we further assume this minimal sub-optimality gap is strictly positive, i.e., ∆ > 0 .
Revised: Then we further assume this minimal sub-optimality gap is strictly positive, i.e., ∆ > 0 .

Original:  P ROPOSED A LGORITHM
Revised:  P ROPOSED A LGORITHM

Original: In this section, we propose our algorithm, DS-OFUL, in Algorithm 1.
Revised: In this section, we propose our algorithm, DS-OFUL, in Algorithm 1.

Original: The algorithm runs for K rounds.
Revised: The algorithm runs for K rounds.

Original: In each round, the algorithm first estimates the underlying parameter θ ∗ by solving the following ridge regression problem in Line
Revised: In each round, the algorithm first estimates the underlying parameter θ ∗ by solving the following ridge regression problem in Line

Original: [Equation]
Revised: [Equation]

Original: where C k − 1 is the index set of the selected contextual vectors for regression and is initialized as an empty set at the beginning.
Revised: where C k − 1 is the index set of the selected contextual vectors for regression and is initialized as an empty set at the beginning.

Original: After receiving the contextual vectors set D k , the algorithm selects an arm from the optimistic estimation powered by the Upper Confidence Bound (UCB) bonus in
Revised: After receiving the contextual vectors set D k , the algorithm selects an arm from the optimistic estimation powered by the Upper Confidence Bound (UCB) bonus in

Original: Line 4.
Revised: Line 4.

Original: In line 5, the algorithm adds the index of current round into C k if the UCB bonus of the chosen arm x k , denoted by ∥ x k ∥ U − 1 k , is greater than the threshold Γ .
Revised: In line 5, the algorithm adds the index of current round into C k if the UCB bonus of the chosen arm x k , denoted by ∥ x k ∥ U − 1 k , is greater than the threshold Γ .

Original: Intuitively speaking, since the
Revised: Intuitively speaking, since the

Original: UCB bonus reflects the uncertainty of the model about the given arm x , Line 5 discards the data that brings little uncertainty ( ∥ x ∥ U − 1 k ) to the model.
Revised: UCB bonus reflects the uncertainty of the model about the given arm x , Line 5 discards the data that brings little uncertainty ( ∥ x ∥ U − 1 k ) to the model.

Original: Finally we denote the total number of selected data in Line 5 by |C K | .
Revised: Finally we denote the total number of selected data in Line 5 by |C K | .

Original: We will declare the choices of the parameter Γ , β and λ in the next section.
Revised: We will declare the choices of the parameter Γ , β and λ in the next section.

Original: Input: Threshold Γ , radius β and regularizor λ 1: Initialize C 0 = ∅ , U 

Revised: Input: Threshold Γ , radius β and regularizor λ 1: Initialize C 0 = ∅ , U

Original: =
Revised: =

Original: λ
Revised: λ

Original: I , θ 0 =2: for k = 1 , . . .
Revised: I , θ 0 =2: for k = 1 , . . .

Original: , K do 3:
Revised: , K do 3:

Original: Set U k
Revised: Set U k

Original: = λ
Revised: = λ

Original: I + (cid:80) i ∈C k − 1 x i x ⊤ i , θ k
Revised: I + (cid:80) i ∈C k − 1 x i x ⊤ i , θ k

Original: = U − 1 k (cid:80) i ∈C k − 1 r i x i .
Revised: = U − 1 k (cid:80) i ∈C k − 1 r i x i .

Original: Receive decision set D k , select x k = argmax x ∈D k (cid:8) x
Revised: Receive decision set D k , select x k = argmax x ∈D k (cid:8) x

Original: ⊤ θ k + β
Revised: ⊤ θ k + β

Original: ∥
Revised: ∥

Original: x
Revised: x

Original: ∥ U − 1 k (cid:9) , receive reward r k 5: if ∥ x k ∥ U − 1 k ≥ Γ then
Revised: ∥ U − 1 k (cid:9) , receive reward r k 5: if ∥ x k ∥ U − 1 k ≥ Γ then

Original: C k = C k − 1 ∪ { k } else
Revised: C k = C k − 1 ∪ { k } else

Original: C k = C k −6: end for
Revised: C k = C k −6: end for

Original:  R EGRET A NALYSIS
Revised:  R EGRET A NALYSIS

Original: In this section, we provide the regret upper bound of Algorithm 1 and the regret lower bound for learning the misspecified linear contextual bandit.
Revised: In this section, we provide the regret upper bound of Algorithm 1 and the regret lower bound for learning the misspecified linear contextual bandit.

Original: Theorem 5.1 (Upper Bound) .
Revised: Theorem 5.1 (Upper Bound) .

Original: For any 0 < δ < 1 , let λ
Revised: For any 0 < δ < 1 , let λ

Original: =
Revised: =

Original: B − 2 and Γ = ∆ / (2 √ dι 1 ) where ι 1 = (24 + 18 R ) log((72 + 54 R ) LB √ d ∆ − 1 ) + (cid:112)
Revised: B − 2 and Γ = ∆ / (2 √ dι 1 ) where ι 1 = (24 + 18 R ) log((72 + 54 R ) LB √ d ∆ − 1 ) + (cid:112)

Original: 8 R 2 log(1 /δ ) .
Revised: 8 R 2 log(1 /δ ) .

Original: Set β
Revised: Set β

Original: = 1 + 4 √
Revised: = 1 + 4 √

Original: dι 2 + R √ 2 dιwhere ι 

Revised: dι 2 + R √ 2 dιwhere ι

Original: = log(3 LB Γ − 1 ) , ι 3 = log((

Revised: = log(3 LB Γ − 1 ) , ι 3 = log((

Original: + 16 L 2 B 2 Γ − 2 ι 2 ) /δ ) .
Revised: + 16 L 2 B 2 Γ − 2 ι 2 ) /δ ) .

Original: If the misspecification level is bounded by 2 √ dζι 1 ≤ ∆ , then with probability at least 1 − δ , the cumulative regret of Algorithmis bounded by
Revised: If the misspecification level is bounded by 2 √ dζι 1 ≤ ∆ , then with probability at least 1 − δ , the cumulative regret of Algorithmis bounded by

Original: [Equation]
Revised: [Equation]

Original: Remark 5.2.
Revised: Remark 5.2.

Original: Since β = (cid:101) O ( √ d ) , Theorem 5.1 suggests an (cid:101) O
Revised: Since β = (cid:101) O ( √ d ) , Theorem 5.1 suggests an (cid:101) O

Original: ( d 2 ∆ − 1 ) constant regret bound independent of the total number of rounds K when ζ ≤ (cid:101) O (∆ / √ d ) .
Revised: ( d 2 ∆ − 1 ) constant regret bound independent of the total number of rounds K when ζ ≤ (cid:101) O (∆ / √ d ) .

Original: This suggests an (cid:101) O
Revised: This suggests an (cid:101) O

Original: ( d 2 ∆ − 1 ) constant regret bound if the misspecification level is reasonably small, which improves the logarithmic regret (cid:101)
Revised: ( d 2 ∆ − 1 ) constant regret bound if the misspecification level is reasonably small, which improves the logarithmic regret (cid:101)

Original: O ( d 2 ∆ − 1 log( K ) in Abbasi-Yadkori et al.
Revised: O ( d 2 ∆ − 1 log( K ) in Abbasi-Yadkori et al.

Original: (2011) to a constant regret 2 .
Revised: (2011) to a constant regret 2 .

Original: Note that our constant regret bound relies on the knowledge of the minimal sub-optimality gap ∆ , while the
Revised: Note that our constant regret bound relies on the knowledge of the minimal sub-optimality gap ∆ , while the

Original: OFUL algorithm in Abbasi-Yadkori et al.
Revised: OFUL algorithm in Abbasi-Yadkori et al.

Original: (2011) does not need prior knowledge about the minimal sub-optimality gap ∆ .
Revised: (2011) does not need prior knowledge about the minimal sub-optimality gap ∆ .

Original: Remark 5.3.
Revised: Remark 5.3.

Original: Our high probability constant regret bound does not violate the lower bound proved in Hao et al.
Revised: Our high probability constant regret bound does not violate the lower bound proved in Hao et al.

Original: (2020), which says that certain diversity condition on the contexts is necessary to achieve an expected constant regret bound (Papini et al., 2021).
Revised: (2020), which says that certain diversity condition on the contexts is necessary to achieve an expected constant regret bound (Papini et al., 2021).

Original: In contrast, we only provide a highprobability constant regret bound.
Revised: In contrast, we only provide a highprobability constant regret bound.

Original: When extending this high probability constant regret bound to expected regret bound, we have
Revised: When extending this high probability constant regret bound to expected regret bound, we have

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: which depends on K .
Revised: which depends on K .

Original: To obtain a sub-linear expected regret, we can set δ = 1 /K which yields a logarithmic regret (cid:101) O
Revised: To obtain a sub-linear expected regret, we can set δ = 1 /K which yields a logarithmic regret (cid:101) O

Original: ( d 2 ∆ − 1 log( K )) and does not violate the lower bound in Hao et al.
Revised: ( d 2 ∆ − 1 log( K )) and does not violate the lower bound in Hao et al.

Original: Furthermore, following the similar idea in Lattimore et al.
Revised: Furthermore, following the similar idea in Lattimore et al.

Original: , we can prove a gap-dependent lower bound for misspecified stochastic linear bandits.
Revised: , we can prove a gap-dependent lower bound for misspecified stochastic linear bandits.

Original: Note that stochastic linear bandit can be seen as a special case of linear contextual bandits with a fixed decision set
Revised: Note that stochastic linear bandit can be seen as a special case of linear contextual bandits with a fixed decision set

Original: D k
Revised: D k

Original: = D across all round k ∈ [ K ] .
Revised: = D across all round k ∈ [ K ] .

Original: Similar result and proof can be found in Du et al.
Revised: Similar result and proof can be found in Du et al.

Original: (2019) for episodic reinforcement learning.
Revised: (2019) for episodic reinforcement learning.

Original: Theorem 5.4 (Lower Bound) .
Revised: Theorem 5.4 (Lower Bound) .

Original: Given the dimension d and the number of arms |D| , for any ∆ ≤and ζ ≥ 3∆ (cid:112) 8 log( |D| ) / ( d − 1) , there exists a set of stochastic linear bandit problems Θ with minimal sub-optimality gap ∆ and misspecification error level ζ , such that for any algorithm that has a sublinear expected regret bound for all θ ∈ Θ , i.e., E
Revised: Given the dimension d and the number of arms |D| , for any ∆ ≤and ζ ≥ 3∆ (cid:112) 8 log( |D| ) / ( d − 1) , there exists a set of stochastic linear bandit problems Θ with minimal sub-optimality gap ∆ and misspecification error level ζ , such that for any algorithm that has a sublinear expected regret bound for all θ ∈ Θ , i.e., E

Original: [ Regret θ ( K )] ≤ CK α with C > 0 and 0 ≤ α < 1 , we have
Revised: [ Regret θ ( K )] ≤ CK α with C > 0 and 0 ≤ α < 1 , we have

Original: • When K ≤ O ( |D| ) , the expected regret is lower bounded by E θ ∼ Unif.
Revised: • When K ≤ O ( |D| ) , the expected regret is lower bounded by E θ ∼ Unif.

Original: ( Θ )
Revised: ( Θ )

Original: [ Regret θ ( K )] ≥ K ∆ .
Revised: [ Regret θ ( K )] ≥ K ∆ .

Original: • When K ≥ Ω( |D| ) , the expected regret is lower bounded by sup θ ∈ Θ E [ Regret θ ( K )] ≥ (cid:101) Ω( |D| log( K )∆ − 1 ) .
Revised: • When K ≥ Ω( |D| ) , the expected regret is lower bounded by sup θ ∈ Θ E [ Regret θ ( K )] ≥ (cid:101) Ω( |D| log( K )∆ − 1 ) .

Original: Remark 5.5.
Revised: Remark 5.5.

Original: Theorem 5.4 shows two regimes under the case ζ ≥ (cid:101) Ω(∆ / √ d ) .
Revised: Theorem 5.4 shows two regimes under the case ζ ≥ (cid:101) Ω(∆ / √ d ) .

Original: In the first regime
Revised: In the first regime

Original: K ≤ O ( |D| ) where the decision set is large (e.g., |D| = d 100 ), any algorithm will suffer from a linear regret (cid:101) O (∆ K ) , which suggests that the regime cannot be efficiently learnable.
Revised: K ≤ O ( |D| ) where the decision set is large (e.g., |D| = d 100 ), any algorithm will suffer from a linear regret (cid:101) O (∆ K ) , which suggests that the regime cannot be efficiently learnable.

Original: In the second regime K ≥ O ( |D| ) , Theorem 5.4 suggests a (cid:101) Ω( |D| ∆ − 1 log( K )) regret lower bound, which is matched by the multi-armed bandit algorithm with an upper bound (cid:101)
Revised: In the second regime K ≥ O ( |D| ) , Theorem 5.4 suggests a (cid:101) Ω( |D| ∆ − 1 log( K )) regret lower bound, which is matched by the multi-armed bandit algorithm with an upper bound (cid:101)

Original: O ( |D| ∆ − 1 log( K ))
Revised: O ( |D| ∆ − 1 log( K ))

Original: (Lattimore & Szepesv ´ ari, 2020).
Revised: (Lattimore & Szepesv ´ ari, 2020).

Original: Therefore, in this easier regime, linear function approximation cannot provide any performance improvement and one can simply adopt the multi-armed bandit algorithm to learn the bandit model.
Revised: Therefore, in this easier regime, linear function approximation cannot provide any performance improvement and one can simply adopt the multi-armed bandit algorithm to learn the bandit model.

Original: Remark 5.6.
Revised: Remark 5.6.

Original: Theorems 5.1 and 5.4 provide a holistic picture about the role of misspecification in linear contextual bandits.
Revised: Theorems 5.1 and 5.4 provide a holistic picture about the role of misspecification in linear contextual bandits.

Original: Here we focus on the more difficult regime K ≤ |D| .
Revised: Here we focus on the more difficult regime K ≤ |D| .

Original: In the regime K ≤ |D| , when ζ ≤ (cid:101) O (∆ / √ d ) , Theorem 5.1 suggests that the bandit problem is efficiently learnable, and our algorithm DS-OFUL can achieve a constant regret, which improves upon the logarithmic regret bound in the well-specified setting (Abbasi-Yadkori et al., 2011).
Revised: In the regime K ≤ |D| , when ζ ≤ (cid:101) O (∆ / √ d ) , Theorem 5.1 suggests that the bandit problem is efficiently learnable, and our algorithm DS-OFUL can achieve a constant regret, which improves upon the logarithmic regret bound in the well-specified setting (Abbasi-Yadkori et al., 2011).

Original: On the other hand, when ζ ≥ (cid:101) Ω(∆ / √ d ) , Theorem 5.4 provides a linear regret lower bound suggesting that the bandit model can not be efficiently learned.
Revised: On the other hand, when ζ ≥ (cid:101) Ω(∆ / √ d ) , Theorem 5.4 provides a linear regret lower bound suggesting that the bandit model can not be efficiently learned.

Original:  P ROOF S KETCH OF T HEOREM 5. 
Revised:  P ROOF S KETCH OF T HEOREM 5.

Original: In this section, we give an overview of the main technical difficulty and our proof technique to derive
Revised: In this section, we give an overview of the main technical difficulty and our proof technique to derive

Original: Theorem 5.1.
Revised: Theorem 5.1.

Original: The detailed proof is deferred to Appendix C.
Revised: The detailed proof is deferred to Appendix C.

Original: First, we aim at controlling the number of rounds in the index set C K .
Revised: First, we aim at controlling the number of rounds in the index set C K .

Original: Since we only select the data with ∥ x k ∥ U − 1 k ≥ Γ for ridge regression, we can lower bound the summation of the selected UCB terms as (cid:80) k ∈C K ∥ x k ∥ U − 1 k ≥ |C K | Γ .
Revised: Since we only select the data with ∥ x k ∥ U − 1 k ≥ Γ for ridge regression, we can lower bound the summation of the selected UCB terms as (cid:80) k ∈C K ∥ x k ∥ U − 1 k ≥ |C K | Γ .

Original: On the other hand, noticing that U k
Revised: On the other hand, noticing that U k

Original: = (cid:80) i ∈C k − 1 x k x ⊤ k , we can upper bound the summation of UCB terms by using the elliptical potential lemma from AbbasiYadkori et al.
Revised: = (cid:80) i ∈C k − 1 x k x ⊤ k , we can upper bound the summation of UCB terms by using the elliptical potential lemma from AbbasiYadkori et al.

Original: (2011) as (cid:80) k ∈C K ∥ x
Revised: (2011) as (cid:80) k ∈C K ∥ x

Original: k ∥ U − 1 k ≤ (cid:101) O ( (cid:112) d |C K | ) .
Revised: k ∥ U − 1 k ≤ (cid:101) O ( (cid:112) d |C K | ) .

Original: Combining the upper bound and lower bound together we can bound the total number of the selected data |C K | as Γ |C K | ≤ (cid:101) O
Revised: Combining the upper bound and lower bound together we can bound the total number of the selected data |C K | as Γ |C K | ≤ (cid:101) O

Original: ( (cid:112) d |C K | ) , which suggests that |C K | ≤ (cid:101) O
Revised: ( (cid:112) d |C K | ) , which suggests that |C K | ≤ (cid:101) O

Original: ( d Γ − 2 ) which is irreverent with the total number of rounds K .
Revised: ( d Γ − 2 ) which is irreverent with the total number of rounds K .

Original: Second, we control the fluctuations in the regression with misspecification error by | x ⊤ ( θ k − θ ∗ ) | ≤ (cid:101) O
Revised: Second, we control the fluctuations in the regression with misspecification error by | x ⊤ ( θ k − θ ∗ ) | ≤ (cid:101) O

Original: ( R √ d + ζ (cid:112) |C K | )
Revised: ( R √ d + ζ (cid:112) |C K | )

Original: ∥ x
Revised: ∥ x

Original: ∥ U − 1 k .
Revised: ∥ U − 1 k .

Original: Compare this result with the original result | x ⊤ ( θ k − θ ∗ ) | ≤ (cid:101)
Revised: Compare this result with the original result | x ⊤ ( θ k − θ ∗ ) | ≤ (cid:101)

Original: O
Revised: O

Original: ( R √ d + ζ √ dK )
Revised: ( R √ d + ζ √ dK )

Original: ∥ x
Revised: ∥ x

Original: ∥ U − 1 k in Jin et al.
Revised: ∥ U − 1 k in Jin et al.

Original: (2020), our confidence radius (cid:101) O
Revised: (2020), our confidence radius (cid:101) O

Original: ( R √ d + ζ (cid:112) |C K | ) does not grow with the total number of rounds K .
Revised: ( R √ d + ζ (cid:112) |C K | ) does not grow with the total number of rounds K .

Original: In fact, directly use the result in Jin et al.
Revised: In fact, directly use the result in Jin et al.

Original: and follow the proof outline in Abbasi-Yadkori et al.
Revised: and follow the proof outline in Abbasi-Yadkori et al.

Original: (2011) will lead to the following regret bound:
Revised: (2011) will lead to the following regret bound:

Original: [Equation]
Revised: [Equation]

Original: which suggests a linear regret bound.
Revised: which suggests a linear regret bound.

Original: As a comparison, with the help of data selection rule in our work, the regression set C K is finite and we can avoid the linear regret when using all data into regression.
Revised: As a comparison, with the help of data selection rule in our work, the regression set C K is finite and we can avoid the linear regret when using all data into regression.

Original: In addition, our result provides a √ d tighter bound compared with Jin et al.
Revised: In addition, our result provides a √ d tighter bound compared with Jin et al.

Original: (2020) by using the result provided in Zanette et al.
Revised: (2020) by using the result provided in Zanette et al.

Original: (2020c)
Revised: (2020c)

Original: Based on these two key observations, we overcome the linear regret bound by partitioning the total round K into two different sets.
Revised: Based on these two key observations, we overcome the linear regret bound by partitioning the total round K into two different sets.

Original: The first set contains all non-selected round, i.e. [ K ] \ C K .
Revised: The first set contains all non-selected round, i.e. [ K ] \ C K .

Original: In this situation, the uncertainty satisfies ∥ x k ∥ U − 1 k < Γ
Revised: In this situation, the uncertainty satisfies ∥ x k ∥ U − 1 k < Γ

Original: and we can prove that when ζ ≤ (cid:101) O (∆ / √ d ) , the instantaneous regret for the un-selected round is bounded by
Revised: and we can prove that when ζ ≤ (cid:101) O (∆ / √ d ) , the instantaneous regret for the un-selected round is bounded by

Original: [Equation]
Revised: [Equation]

Original: which suggests that the non-selected data is optimal and incur no regret.
Revised: which suggests that the non-selected data is optimal and incur no regret.

Original: For the data in the finite set C K , we follows the gap-dependent regret bound in Abbasi-Yadkori et al.
Revised: For the data in the finite set C K , we follows the gap-dependent regret bound in Abbasi-Yadkori et al.

Original: (2011) to show that (cid:80)
Revised: (2011) to show that (cid:80)

Original: k ∈G r ∗ k − r ( x k ) ≤ (cid:101) O
Revised: k ∈G r ∗ k − r ( x k ) ≤ (cid:101) O

Original: ( d 2 ∆ − 1 ) log( |C K | ) .
Revised: ( d 2 ∆ − 1 ) log( |C K | ) .

Original: As a result, by partition the set [ K ] into two subsets
Revised: As a result, by partition the set [ K ] into two subsets

Original: [ K ] \ C K , C K , we get the claimed cumulative regret bound by
Revised: [ K ] \ C K , C K , we get the claimed cumulative regret bound by

Original: [Equation]
Revised: [Equation]

Original: where we denote Reg ( k ) as the instantaneous regret in round k (i.e. r ∗ k − r ( x k ) ).
Revised: where we denote Reg ( k ) as the instantaneous regret in round k (i.e. r ∗ k − r ( x k ) ).

Original:  M ISSPECIFIED L INEAR MDP S 7. P RELIMINARIES OF L INEAR MDP S
Revised:  M ISSPECIFIED L INEAR MDP S 7. P RELIMINARIES OF L INEAR MDP S

Original: We consider the episodic Markov Decision Process (MDP).
Revised: We consider the episodic Markov Decision Process (MDP).

Original: Each episodic MDP is defined by a tuple
Revised: Each episodic MDP is defined by a tuple

Original: M ( S , A , H, { r h } Hh =1 , { P h } Hh =1 ) where S is the state space,
Revised: M ( S , A , H, { r h } Hh =1 , { P h } Hh =1 ) where S is the state space,

Original: A is the action space, H is the length of each episode and r h :
Revised: A is the action space, H is the length of each episode and r h :

Original: S × A
Revised: S × A

Original: (cid:55)→ [0 , 1] is the reward function at stage h .
Revised: (cid:55)→ [0 , 1] is the reward function at stage h .

Original: P h is the transition kernel where P h ( s ′ | s, a ) denotes the transition probability from state s to s ′ with action a at stage h .
Revised: P h is the transition kernel where P h ( s ′ | s, a ) denotes the transition probability from state s to s ′ with action a at stage h .

Original: At the beginning of each episode, the agent determines a policy π :=
Revised: At the beginning of each episode, the agent determines a policy π :=

Original: { π h } Hh =1 where π h : S (cid:55)→ A .
Revised: { π h } Hh =1 where π h : S (cid:55)→ A .

Original: Then from stage h = 1 to h
Revised: Then from stage h = 1 to h

Original: = H , the agent repeatedly receives state s h , takes the action a h
Revised: = H , the agent repeatedly receives state s h , takes the action a h

Original: = π h
Revised: = π h

Original: ( s h ) , receives the reward r h ( s h , a h ) and
Revised: ( s h ) , receives the reward r h ( s h , a h ) and

Original: the next state s h
Revised: the next state s h

Original: +1 .
Revised: +1 .

Original: For any policy π , the value function and the
Revised: For any policy π , the value function and the

Original: Q -function at stage h is defined by
Revised: Q -function at stage h is defined by

Original: [Equation]
Revised: [Equation]

Original: It’s obvious that for all policy π , for all h ∈ [ H ] , s ∈ S and a ∈ A , the value function and the
Revised: It’s obvious that for all policy π , for all h ∈ [ H ] , s ∈ S and a ∈ A , the value function and the

Original: Q -function is bounded by 0 ≤ V πh ( s ) ≤ H , 0 ≤ Q πh ( s, a ) ≤ H since r h ( s, a ) ∈ [0 , 1] .
Revised: Q -function is bounded by 0 ≤ V πh ( s ) ≤ H , 0 ≤ Q πh ( s, a ) ≤ H since r h ( s, a ) ∈ [0 , 1] .

Original: We further define the optimal value function and the optimal Q -function as
Revised: We further define the optimal value function and the optimal Q -function as

Original: [Equation]
Revised: [Equation]

Original: For simplicity, we denote [ P h V ]( s, a ) =
Revised: For simplicity, we denote [ P h V ]( s, a ) =

Original: E s ′ ∼ P h ( ·| s,a )
Revised: E s ′ ∼ P h ( ·| s,a )

Original: [ V ( s ′ )]
Revised: [ V ( s ′ )]

Original: and we have the Bellman equation along with the Bellman optimality equation as
Revised: and we have the Bellman equation along with the Bellman optimality equation as

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We consider the ζ -approximate linear MDP setting (Jin et al., 2020) in this work to study the impact of misspecification on function approximations in reinforcement learning.
Revised: We consider the ζ -approximate linear MDP setting (Jin et al., 2020) in this work to study the impact of misspecification on function approximations in reinforcement learning.

Original: Definition 7.1 ( ζ -approximate linear MDP, Jin et al. 2020) .
Revised: Definition 7.1 ( ζ -approximate linear MDP, Jin et al. 2020) .

Original: For any ζ ≤ 1 , we say that MDP
Revised: For any ζ ≤ 1 , we say that MDP

Original: M ( S , A , H, { r h } , { P h } ) is a ζ -approximate linear MDP with feature map ϕ : S × A (cid:55)→ R d , if for any h ∈ [ H ] , there exists d unknown (signed) measures µ h = (cid:16) µ (1) h , · · · , µ ( d ) h (cid:17) over S and an unknown vector θ ∗ h ∈ R d such that for any ( s, a ) ∈ S × A ,
Revised: M ( S , A , H, { r h } , { P h } ) is a ζ -approximate linear MDP with feature map ϕ : S × A (cid:55)→ R d , if for any h ∈ [ H ] , there exists d unknown (signed) measures µ h = (cid:16) µ (1) h , · · · , µ ( d ) h (cid:17) over S and an unknown vector θ ∗ h ∈ R d such that for any ( s, a ) ∈ S × A ,

Original: [Equation]
Revised: [Equation]

Original: w.l.o.g.
Revised: w.l.o.g.

Original: we assume ∥
Revised: we assume ∥

Original: ϕ
Revised: ϕ

Original: ( s, a ) ∥ 2 ≤ 1 for all ( s, a ) ∈ S × A and max {∥ µ h ( S ) ∥ 2 , ∥
Revised: ( s, a ) ∥ 2 ≤ 1 for all ( s, a ) ∈ S × A and max {∥ µ h ( S ) ∥ 2 , ∥

Original: θ ∗ h ∥ 2 } ≤ √ d for all h ∈
Revised: θ ∗ h ∥ 2 } ≤ √ d for all h ∈

Original: [ H ] .
Revised: [ H ] .

Original: Under Definition 7.1, it is easy to show that the Q -function under a certain policy π is close to a linear function of the feature map ϕ .
Revised: Under Definition 7.1, it is easy to show that the Q -function under a certain policy π is close to a linear function of the feature map ϕ .

Original: Lemma 7.2 (Lemma C.1, Lemma C.2, Jin et al. 2020) .
Revised: Lemma 7.2 (Lemma C.1, Lemma C.2, Jin et al. 2020) .

Original: For a ζ -approximate linear MDP, for any policy π , there exists a corresponding { w πh } h such that for any ( s, a, h ) ∈ S
Revised: For a ζ -approximate linear MDP, for any policy π , there exists a corresponding { w πh } h such that for any ( s, a, h ) ∈ S

Original: ×
Revised: ×

Original: A ×
Revised: A ×

Original: [ H ] :
Revised: [ H ] :

Original: [Equation]
Revised: [Equation]

Original: We are concerning about minimizing the cumulative regret defined by Regret ( K )
Revised: We are concerning about minimizing the cumulative regret defined by Regret ( K )

Original: =
Revised: =

Original: (cid:80)
Revised: (cid:80)

Original: Kk =1 V ∗ 1 ( s k 1 ) − V π k 1 ( s k 1 ) , where π k is the policy used in the k -th episode.
Revised: Kk =1 V ∗ 1 ( s k 1 ) − V π k 1 ( s k 1 ) , where π k is the policy used in the k -th episode.

Original: Similar to linear contextual bandits, we introduce the minimal sub-optimality gap ∆ originally defined in He et al.
Revised: Similar to linear contextual bandits, we introduce the minimal sub-optimality gap ∆ originally defined in He et al.

Original: (2021a)
Revised: (2021a)

Original: Definition 7.3 (Minimal sub-optimality gap, He et al. 2021a) .
Revised: Definition 7.3 (Minimal sub-optimality gap, He et al. 2021a) .

Original: For each ( s, a, h ) ∈ S ×
Revised: For each ( s, a, h ) ∈ S ×

Original: A ×
Revised: A ×

Original: [ H ] , the sub-optimality gap ∆ h ( s, a ) is defined as ∆ h ( s, a ) :
Revised: [ H ] , the sub-optimality gap ∆ h ( s, a ) is defined as ∆ h ( s, a ) :

Original: = V ∗ h ( s ) − Q ∗ h ( s, a ) and the minimal suboptimality gap is defined as ∆ = min h,s,a { ∆ h ( s, a ) :
Revised: = V ∗ h ( s ) − Q ∗ h ( s, a ) and the minimal suboptimality gap is defined as ∆ = min h,s,a { ∆ h ( s, a ) :

Original: ∆ h ( s, a ) > 0 } .
Revised: ∆ h ( s, a ) > 0 } .

Original: We further assume this minimal sub-optimality gap is strictly positive, i.e., ∆ > 0 .
Revised: We further assume this minimal sub-optimality gap is strictly positive, i.e., ∆ > 0 .

Original: P ROPOSED A LGORITHM
Revised: P ROPOSED A LGORITHM

Original: We propose our algorithm, DS-LSVI, for misspecified linear MDP in Algorithm 2.
Revised: We propose our algorithm, DS-LSVI, for misspecified linear MDP in Algorithm 2.

Original: It applies the idea of DS-OFUL to the LSVI-UCB (Jin et al., 2020).
Revised: It applies the idea of DS-OFUL to the LSVI-UCB (Jin et al., 2020).

Original: For simplicity, we denote ϕ
Revised: For simplicity, we denote ϕ

Original: kh
Revised: kh

Original: =
Revised: =

Original: ϕ
Revised: ϕ

Original: ( s kh , a kh ) , r kh = r h
Revised: ( s kh , a kh ) , r kh = r h

Original: ( s k h , a k h ) for short when there is no confusion.
Revised: ( s k h , a k h ) for short when there is no confusion.

Original: The algorithm runs for K episodes.
Revised: The algorithm runs for K episodes.

Original: In the kth episode, the algorithm estimates the optimal Q -function using a linear function as indicated by
Revised: In the kth episode, the algorithm estimates the optimal Q -function using a linear function as indicated by

Original: Lemma 7.2.
Revised: Lemma 7.2.

Original: In detail, at each stage h , after acquiring the estimated value function V kh +1 ( · ) at state h
Revised: In detail, at each stage h , after acquiring the estimated value function V kh +1 ( · ) at state h

Original: + 1 , in Line 4, the algorithm solves the following ridge regression problem
Revised: + 1 , in Line 4, the algorithm solves the following ridge regression problem

Original: [Equation]
Revised: [Equation]

Original: where C k − 1 contains the indices of episodes selected for regression and r ih
Revised: where C k − 1 contains the indices of episodes selected for regression and r ih

Original: + V kh +

Revised: + V kh +

Original: ( s ih +1 ) is the estimated Q -function by Bellman optimality equation (7.1).
Revised: ( s ih +1 ) is the estimated Q -function by Bellman optimality equation (7.1).

Original: Then the algorithm takes the greedy policy based on the estimated Q -function and receives the full episode.
Revised: Then the algorithm takes the greedy policy based on the estimated Q -function and receives the full episode.

Original: In Line 12, the algorithm adds the episode k into the regression index set C k if the data on k -th episode provides more uncertainty (i.e.
Revised: In Line 12, the algorithm adds the episode k into the regression index set C k if the data on k -th episode provides more uncertainty (i.e.

Original: ∥ ϕ ∥ U − 1 ≥ Γ ) at any stage h ∈ [ H ] .
Revised: ∥ ϕ ∥ U − 1 ≥ Γ ) at any stage h ∈ [ H ] .

Original: The intuition behind this selection is the same as Line 5 in
Revised: The intuition behind this selection is the same as Line 5 in

Original: Algorithm 1 when dealing with linear bandits: when one episode provide a data sample with large uncertainty at any stage, we add it to the regression.
Revised: Algorithm 1 when dealing with linear bandits: when one episode provide a data sample with large uncertainty at any stage, we add it to the regression.

Original: Otherwise, the episode will be ignored if the whole episode provide few uncertainty.
Revised: Otherwise, the episode will be ignored if the whole episode provide few uncertainty.

Original: Input: Threshold Γ , radius β
Revised: Input: Threshold Γ , radius β

Original: U
Revised: U

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ϕ ( · , · ) , w kh (cid:11) + β ∥ ϕ ( · , · ) ∥
Revised: ϕ ( · , · ) , w kh (cid:11) + β ∥ ϕ ( · , · ) ∥

Original: ( U k h ) −6:
Revised: ( U k h ) −6:

Original: V kh ( · ) = min { max a { Q kh ( · , a ) } , H } , π kh ( · ) = argmax a { Q kh ( · , a ) } 7: end for 8:
Revised: V kh ( · ) = min { max a { Q kh ( · , a ) } , H } , π kh ( · ) = argmax a { Q kh ( · , a ) } 7: end for 8:

Original: Receive initial state s k9: for stage h = 1 , . . .
Revised: Receive initial state s k9: for stage h = 1 , . . .

Original: , H do 10:
Revised: , H do 10:

Original: Take action a kh = π kh ( s kh ) and receive reward r kh and next state s kh + 11: end for 12:
Revised: Take action a kh = π kh ( s kh ) and receive reward r kh and next state s kh + 11: end for 12:

Original: C k = C k − 1 ∪ { k }
Revised: C k = C k − 1 ∪ { k }

Original: if ∃ h ∈ [ H ] , ∥ ϕ kh ∥
Revised: if ∃ h ∈ [ H ] , ∥ ϕ kh ∥

Original: ( U kh ) − 1 ≥ Γ else =
Revised: ( U kh ) − 1 ≥ Γ else =

Original: C k −13: end for
Revised: C k −13: end for

Original: 7. R EGRET A NALYSIS
Revised: 7. R EGRET A NALYSIS

Original: We provide the regret upper bound of Algorithm 2 for the ζ -approximate linear MDP.
Revised: We provide the regret upper bound of Algorithm 2 for the ζ -approximate linear MDP.

Original: The proof is deferred to Appendix F.
Revised: The proof is deferred to Appendix F.

Original: Theorem 7.4 (Upper Bound) .
Revised: Theorem 7.4 (Upper Bound) .

Original: Let Γ =
Revised: Let Γ =

Original: (cid:101) Θ(∆ d − 1 H − 2 ) ,
Revised: (cid:101) Θ(∆ d − 1 H − 2 ) ,

Original: β = (cid:101) O ( Hd ) , with probability at least 1 − δ , if ζ = (cid:101) O (∆ d − 0 .
Revised: β = (cid:101) O ( Hd ) , with probability at least 1 − δ , if ζ = (cid:101) O (∆ d − 0 .

Original: 5 H − 2 . 5 ) , the cumulative regret of Algorithm 2 is bounded by Regret ( K ) ≤ (cid:101)
Revised: 5 H − 2 . 5 ) , the cumulative regret of Algorithm 2 is bounded by Regret ( K ) ≤ (cid:101)

Original: O ( H 5 d 3 ∆ − 1 log( K )) .
Revised: O ( H 5 d 3 ∆ − 1 log( K )) .

Original: Remark 7.5.
Revised: Remark 7.5.

Original: Theorem 7.4 suggests that if the misspecification level ζ is upper bounded by
Revised: Theorem 7.4 suggests that if the misspecification level ζ is upper bounded by

Original: O (∆ d − 0 .
Revised: O (∆ d − 0 .

Original: 5 H − 2 . 5 ) , we can achieve the same logarithmic regret bound (cid:101)
Revised: 5 H − 2 . 5 ) , we can achieve the same logarithmic regret bound (cid:101)

Original: O
Revised: O

Original: ( H 5 d 3 ∆ − 1 ) as the wellspecified setting (He et al., 2021a).
Revised: ( H 5 d 3 ∆ − 1 ) as the wellspecified setting (He et al., 2021a).

Original: This result indicates that a reasonably small misspecification will not deteriorate the performance.
Revised: This result indicates that a reasonably small misspecification will not deteriorate the performance.

Original: Our result improves the original regret bound (cid:101) O ( √ d 3 H 4 K + ζdH 2 K ) for the misspecified linear MDP provided in Jin et al.
Revised: Our result improves the original regret bound (cid:101) O ( √ d 3 H 4 K + ζdH 2 K ) for the misspecified linear MDP provided in Jin et al.

Original: Remark 7.6.
Revised: Remark 7.6.

Original: Compared with the linear contextual bandit results, our result in the linear MDP setting also suggests the same relationship among ζ , ∆ and dimension d , ignoring the horizon factor
Revised: Compared with the linear contextual bandit results, our result in the linear MDP setting also suggests the same relationship among ζ , ∆ and dimension d , ignoring the horizon factor

Original: H .
Revised: H .

Original: Du et al.
Revised: Du et al.

Original: (2019) showed that when ζ ≥ (cid:101) Ω(∆ / √ d ) , any reinforcement learning algorithm suffers an O (2 H ) sample complexity.
Revised: (2019) showed that when ζ ≥ (cid:101) Ω(∆ / √ d ) , any reinforcement learning algorithm suffers an O (2 H ) sample complexity.

Original: In addition, Du et al.
Revised: In addition, Du et al.

Original: (2020) provided algorithm for agnostic
Revised: (2020) provided algorithm for agnostic

Original: Q-learning which takes (cid:101)
Revised: Q-learning which takes (cid:101)

Original: O ( d ) trajectories to find the optimal policy when ζ < ∆ / √ d .
Revised: O ( d ) trajectories to find the optimal policy when ζ < ∆ / √ d .

Original: However, their algorithm relies on a generative model which takes multiple actions a for the same state s at the same time.
Revised: However, their algorithm relies on a generative model which takes multiple actions a for the same state s at the same time.

Original: In contrast, Theorem 7.4 suggests that an (cid:101) O (∆ d − 0 .
Revised: In contrast, Theorem 7.4 suggests that an (cid:101) O (∆ d − 0 .

Original: 5 H − 2 . 5 ) misspecification level can lead to a logarithmic regret without accessing the generative model.
Revised: 5 H − 2 . 5 ) misspecification level can lead to a logarithmic regret without accessing the generative model.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: [Table]
Revised: [Table]

Original: To verify the performance improvement by data selection using the UCB bonus in Algorithm 1, we conduct experiments for bandit tasks on both synthetic and real-world datasets, which we will describe in detail below.
Revised: To verify the performance improvement by data selection using the UCB bonus in Algorithm 1, we conduct experiments for bandit tasks on both synthetic and real-world datasets, which we will describe in detail below.

Original: We also carry out experiments for linear MDPs on a synthetic dataset in
Revised: We also carry out experiments for linear MDPs on a synthetic dataset in

Original: Appendix B.4.
Revised: Appendix B.4.

Original: 8. S YNTHETIC D ATASET
Revised: 8. S YNTHETIC D ATASET

Original: The synthetic dataset is composed as follows: we set d = 16 and generate parameter θ ∗ ∼ N ( 0 , I d ) and contextual vectors { x i } Ni =1 ∼ N ( 0 , I d ) where
Revised: The synthetic dataset is composed as follows: we set d = 16 and generate parameter θ ∗ ∼ N ( 0 , I d ) and contextual vectors { x i } Ni =1 ∼ N ( 0 , I d ) where

Original: N = 100 .
Revised: N = 100 .

Original: The generated parameter and vectors are later normalized to be ∥ θ ∗ ∥ 2 = ∥ x i ∥ 2 = 1 .
Revised: The generated parameter and vectors are later normalized to be ∥ θ ∗ ∥ 2 = ∥ x i ∥ 2 = 1 .

Original: The reward function is calculated by r i = ⟨ θ ∗ , x i ⟩ + η i where η i ∼ Unif {− ζ, ζ } .
Revised: The reward function is calculated by r i = ⟨ θ ∗ , x i ⟩ + η i where η i ∼ Unif {− ζ, ζ } .

Original: The contextual vectors and reward function is fixed after generated.
Revised: The contextual vectors and reward function is fixed after generated.

Original: The random noise on the receiving rewards ε t are sampled from the standard normal distribution.
Revised: The random noise on the receiving rewards ε t are sampled from the standard normal distribution.

Original: We set the misspecification level ζ = 0 .
Revised: We set the misspecification level ζ = 0 .

Original: 02 and verified that the sub-optimality gap over the N contextual vectors ∆ ≈ 0 .
Revised: 02 and verified that the sub-optimality gap over the N contextual vectors ∆ ≈ 0 .

Original: We do a grid search for β = { 1 , 3 , 10 } , λ = { 1 , 3 , 10 } and report the cumulative regret of Algorithm 1 with different parameter Γ = { 0 , 0 .
Revised: We do a grid search for β = { 1 , 3 , 10 } , λ = { 1 , 3 , 10 } and report the cumulative regret of Algorithm 1 with different parameter Γ = { 0 , 0 .

Original: 08 , 0 . 2 } over 32 independent trials with total rounds K = 2000 .
Revised: 08 , 0 . 2 } over 32 independent trials with total rounds K = 2000 .

Original: It’s obvious that when Γ = 0 , our algorithm degrades to the standard OFUL algorithm (Abbasi-Yadkori et al., 2011) which uses data from all rounds into regression.
Revised: It’s obvious that when Γ = 0 , our algorithm degrades to the standard OFUL algorithm (Abbasi-Yadkori et al., 2011) which uses data from all rounds into regression.

Original: The result is shown in Figure 1(b) and the average cumulative regret on the last round is reported in
Revised: The result is shown in Figure 1(b) and the average cumulative regret on the last round is reported in

Original: Table 1 with its variance over 32 trials.
Revised: Table 1 with its variance over 32 trials.

Original: We can see that by setting Γ ≈ ∆ / √ d ≈ 0 .
Revised: We can see that by setting Γ ≈ ∆ / √ d ≈ 0 .

Original: 18 / √ 

Revised: 18 / √

Original: ≈ 0 .
Revised: ≈ 0 .

Original: Algorithm 1 can achieve less cumulative regret compared with OFUL ( Γ = 0 ).
Revised: Algorithm 1 can achieve less cumulative regret compared with OFUL ( Γ = 0 ).

Original: The algorithm with a proper choice of Γ also convergences to zero instantaneous regret faster than OFUL.
Revised: The algorithm with a proper choice of Γ also convergences to zero instantaneous regret faster than OFUL.

Original: It is also evident that a slightly larger Γ = 0 .
Revised: It is also evident that a slightly larger Γ = 0 .

Original: 08 will not affect the performance but a too large Γ = 0 .
Revised: 08 will not affect the performance but a too large Γ = 0 .

Original: 20 ≥ ∆ will cause the algorithm to fail to learn the contextual vectors and induce a linear regret.
Revised: 20 ≥ ∆ will cause the algorithm to fail to learn the contextual vectors and induce a linear regret.

Original: Also, our algorithm shows that using a larger Γ can significantly boost the speed of the algorithm by reducing the number of regressions needed in the algorithm.
Revised: Also, our algorithm shows that using a larger Γ can significantly boost the speed of the algorithm by reducing the number of regressions needed in the algorithm.

Original: We also compare with the algorithm (LSW) in Equation (6) of Lattimore et al.
Revised: We also compare with the algorithm (LSW) in Equation (6) of Lattimore et al.

Original: (2020) and the
Revised: (2020) and the

Original: RLB in Ghosh et al.
Revised: RLB in Ghosh et al.

Original: (2017) in Figure 1(b) and Table 1.
Revised: (2017) in Figure 1(b) and Table 1.

Original: For Lattimore et al.
Revised: For Lattimore et al.

Original: (2020), the estimated reward is updated by r ( x ) =
Revised: (2020), the estimated reward is updated by r ( x ) =

Original: x
Revised: x

Original: ⊤ θ k + β ∥ x
Revised: ⊤ θ k + β ∥ x

Original: ∥
Revised: ∥

Original: U − 1 k + ε (cid:80) ks =1 | x ⊤ U − 1 k x − 1 s | .
Revised: U − 1 k + ε (cid:80) ks =1 | x ⊤ U − 1 k x − 1 s | .

Original: However, since the term ε (cid:80) ks =1 | x ⊤ U − 1 k x − 1 s | is hard to be updated incrementally w.r.t.
Revised: However, since the term ε (cid:80) ks =1 | x ⊤ U − 1 k x − 1 s | is hard to be updated incrementally w.r.t.

Original: k , this algorithm is less efficient than OFUL Abbasi-Yadkori et al.
Revised: k , this algorithm is less efficient than OFUL Abbasi-Yadkori et al.

Original: (2011) as well as our algorithm.
Revised: (2011) as well as our algorithm.

Original: For the RLB algorithm in Ghosh et al.
Revised: For the RLB algorithm in Ghosh et al.

Original: , we did the hypothesis test for k = 10 rounds and then decided whether to use OFUL or multi-armed UCB.
Revised: , we did the hypothesis test for k = 10 rounds and then decided whether to use OFUL or multi-armed UCB.

Original: The results show that both LSW and RLB achieve a worse regret than OFUL since in our setting ζ is relatively small.
Revised: The results show that both LSW and RLB achieve a worse regret than OFUL since in our setting ζ is relatively small.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: 8. R EAL - WORLD D ATASET
Revised: 8. R EAL - WORLD D ATASET

Original: To demonstrate that the proposed algorithm can be easily applied to modern machine learning tasks, we carried out experiments on the Asirra dataset (Elson et al., 2007).
Revised: To demonstrate that the proposed algorithm can be easily applied to modern machine learning tasks, we carried out experiments on the Asirra dataset (Elson et al., 2007).

Original: The task of agent is to distinguish the image of cats from the image of dogs.
Revised: The task of agent is to distinguish the image of cats from the image of dogs.

Original: At each round k , the agent receives the feature vector ϕ 1 ,k ∈ R 512 of a cat image and another feature vector ϕ 2 ,k ∈ R 512 of a dog image.
Revised: At each round k , the agent receives the feature vector ϕ 1 ,k ∈ R 512 of a cat image and another feature vector ϕ 2 ,k ∈ R 512 of a dog image.

Original: Both feature vectors are generated using ResNet-18 (He et al., 2016) pretrained on ImageNet (Deng et al., 2009).
Revised: Both feature vectors are generated using ResNet-18 (He et al., 2016) pretrained on ImageNet (Deng et al., 2009).

Original: We normalize ∥ ϕ 1 ,k ∥ 2 = ∥ ϕ 2 ,k ∥ 2 = 1 .
Revised: We normalize ∥ ϕ 1 ,k ∥ 2 = ∥ ϕ 2 ,k ∥ 2 = 1 .

Original: The agent is required to select the cat from these two vectors.
Revised: The agent is required to select the cat from these two vectors.

Original: It receives reward r t = 1 if it selects the correct feature vector, and receives r t
Revised: It receives reward r t = 1 if it selects the correct feature vector, and receives r t

Original: =otherwise.
Revised: =otherwise.

Original: It is trivial that the sub-optimality gap of this task is ∆ = 1 .
Revised: It is trivial that the sub-optimality gap of this task is ∆ = 1 .

Original: To better demonstrate the influence of misspecification on the performance of the algorithm, we only select the data with | ϕ ⊤ i θ ∗ − r i | ≤ ζ with r i = 1 if it is a cat and r i = 0 otherwise.
Revised: To better demonstrate the influence of misspecification on the performance of the algorithm, we only select the data with | ϕ ⊤ i θ ∗ − r i | ≤ ζ with r i = 1 if it is a cat and r i = 0 otherwise.

Original: θ ∗ is a pretrained parameter on the whole dataset using linear regression θ ∗ =
Revised: θ ∗ is a pretrained parameter on the whole dataset using linear regression θ ∗ =

Original: argmin θ (cid:80) Ni =1 ( ϕ ⊤ i θ
Revised: argmin θ (cid:80) Ni =1 ( ϕ ⊤ i θ

Original: − r i ) 2 , which the agent does not know.
Revised: − r i ) 2 , which the agent does not know.

Original: For hyper-parameter tuning, we select β = { 1 , 0 . 3 , 0 . 1 } and λ = { 1 , 3 , 10 } by doing a grid search and repeat the experiments for 8 times over 1M rounds for each parameter configuration.
Revised: For hyper-parameter tuning, we select β = { 1 , 0 . 3 , 0 . 1 } and λ = { 1 , 3 , 10 } by doing a grid search and repeat the experiments for 8 times over 1M rounds for each parameter configuration.

Original: As shown in Figure 1(a), when ζ = 0 .
Revised: As shown in Figure 1(a), when ζ = 0 .

Original: 01 , though the OFUL algorithm (setting Γ = 0 ) will have a better performance at the very beginning, setting Γ = 0 .
Revised: 01 , though the OFUL algorithm (setting Γ = 0 ) will have a better performance at the very beginning, setting Γ = 0 .

Original: ≈ ∆ / √ d = 1 / √ 512 will eventually improve the performance of the algorithm.
Revised: ≈ ∆ / √ d = 1 / √ 512 will eventually improve the performance of the algorithm.

Original: As a sensitivity analysis, we also set ζ = { 0 . 5 , 0 . 1 , 0 . 05 } to test the impact of misspecification on the performance of algorithm choices of Γ .
Revised: As a sensitivity analysis, we also set ζ = { 0 . 5 , 0 . 1 , 0 . 05 } to test the impact of misspecification on the performance of algorithm choices of Γ .

Original: More experiment configurations and results are deferred to Appendix B.
Revised: More experiment configurations and results are deferred to Appendix B.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: We study the misspecified linear contextual bandit from a gap-dependent perceptive.
Revised: We study the misspecified linear contextual bandit from a gap-dependent perceptive.

Original: We propose an algorithm and show that if the misspecification level ζ ≤ (cid:101) O (∆ / √ d ) , the proposed algorithm can achieve the same gap-dependent regret bound as in the well-specified case.
Revised: We propose an algorithm and show that if the misspecification level ζ ≤ (cid:101) O (∆ / √ d ) , the proposed algorithm can achieve the same gap-dependent regret bound as in the well-specified case.

Original: Along with Lattimore et al.
Revised: Along with Lattimore et al.

Original: Du et al.
Revised: Du et al.

Original: (2019), we provide a complete picture on the interplay between misspecification and sub-optimality gap, in which ∆ / √ d plays an important role on the phase transition of ζ to decide if the bandit model can be efficiently learned.
Revised: (2019), we provide a complete picture on the interplay between misspecification and sub-optimality gap, in which ∆ / √ d plays an important role on the phase transition of ζ to decide if the bandit model can be efficiently learned.

Original: The algorithm and analysis have been extended to linear Markov decision processes and verified via experiments as well.
Revised: The algorithm and analysis have been extended to linear Markov decision processes and verified via experiments as well.

Original: The promising result suggests a few interesting directions for future research.
Revised: The promising result suggests a few interesting directions for future research.

Original: For example, it remains unclear if we can get rid of the prior knowledge of the minimum sub-optimality gap to achieve similar regret guarantees.
Revised: For example, it remains unclear if we can get rid of the prior knowledge of the minimum sub-optimality gap to achieve similar regret guarantees.

Original: It would also be interesting to incorporate the Lipschitz continuity or smoothness properties of the reward function to derive fine-grained results.
Revised: It would also be interesting to incorporate the Lipschitz continuity or smoothness properties of the reward function to derive fine-grained results.

Original: A A DDITIONAL R ELATED WORK
Revised: A A DDITIONAL R ELATED WORK

Original: Linear Contextual Bandits.
Revised: Linear Contextual Bandits.

Original: There is a large body of literature on linear contextual bandits.
Revised: There is a large body of literature on linear contextual bandits.

Original: For example, Auer (2002); Chu et al.
Revised: For example, Auer (2002); Chu et al.

Original: ; Agrawal & Goyal (2013) studied linear contextual bandits when the number of arms is finite.
Revised: ; Agrawal & Goyal (2013) studied linear contextual bandits when the number of arms is finite.

Original: Abbasi-Yadkori et al.
Revised: Abbasi-Yadkori et al.

Original: (2011) proposed an algorithm called OFUL to deal with the infinite arm set.
Revised: (2011) proposed an algorithm called OFUL to deal with the infinite arm set.

Original: All these works come with an (cid:101) O ( √ K ) problem-independent regret bound, and an O ( d 2 ∆ − 1 log( K )) gap-dependent regret bound is also given by Abbasi-Yadkori et al.
Revised: All these works come with an (cid:101) O ( √ K ) problem-independent regret bound, and an O ( d 2 ∆ − 1 log( K )) gap-dependent regret bound is also given by Abbasi-Yadkori et al.

Original: RL with Linear Function Approximation.
Revised: RL with Linear Function Approximation.

Original: To tackle RL tasks in large state space, a line of work on RL with linear function approximation has emerged in the past years.
Revised: To tackle RL tasks in large state space, a line of work on RL with linear function approximation has emerged in the past years.

Original: For example, linear
Revised: For example, linear

Original: MDPs (Yang & Wang, 2019; Jin et al., 2020) is probably one of the most widely studied models where both the transition kernel and the reward function are linear functions of a known feature mapping.
Revised: MDPs (Yang & Wang, 2019; Jin et al., 2020) is probably one of the most widely studied models where both the transition kernel and the reward function are linear functions of a known feature mapping.

Original: Typical algorithms in this setting include LSVI-UCB (Jin et al., 2020) and randomized
Revised: Typical algorithms in this setting include LSVI-UCB (Jin et al., 2020) and randomized

Original: LSVI (Zanette et al., 2020a), both of which can achieve a sublinear regret.
Revised: LSVI (Zanette et al., 2020a), both of which can achieve a sublinear regret.

Original: Besides, linear mixture/kernel MDPs (Modi et al., 2020; Jia et al., 2020; Ayoub et al., 2020; Zhou et al., 2021) has emerged as a popular model for model-based RL with linear function approximation, in which the transition kernel is defined as a mixture of feature mappings defined on the triplet of state, action, and next state.
Revised: Besides, linear mixture/kernel MDPs (Modi et al., 2020; Jia et al., 2020; Ayoub et al., 2020; Zhou et al., 2021) has emerged as a popular model for model-based RL with linear function approximation, in which the transition kernel is defined as a mixture of feature mappings defined on the triplet of state, action, and next state.

Original: In this setting, nearly minimax optimal regret has been attained for both finite-horizon episodic MDPs and infinite-horizon discounted MDPs (Zhou et al., 2021).
Revised: In this setting, nearly minimax optimal regret has been attained for both finite-horizon episodic MDPs and infinite-horizon discounted MDPs (Zhou et al., 2021).

Original: The aforementioned works are focused on the problem-independent regret bound, while He et al.
Revised: The aforementioned works are focused on the problem-independent regret bound, while He et al.

Original: (2021a) provided a gap-dependent regret bound for both linear MDPs (i.e., (cid:101)
Revised: (2021a) provided a gap-dependent regret bound for both linear MDPs (i.e., (cid:101)

Original: O ( d 3 H 5 ∆ − 1 ) ) and linear mixture MDPs (i.e., (cid:101) O
Revised: O ( d 3 H 5 ∆ − 1 ) ) and linear mixture MDPs (i.e., (cid:101) O

Original: ( d 2 H 5 ∆ − 1 ) ).
Revised: ( d 2 H 5 ∆ − 1 ) ).

Original: B E XPERIMENT D ETAILS AND A DDITIONAL R ESULTS B. E XPERIMENT C ONFIGURATION
Revised: B E XPERIMENT D ETAILS AND A DDITIONAL R ESULTS B. E XPERIMENT C ONFIGURATION

Original: The experiment on synthetic dataset is conducted on Google Colab with a 2-core Intel ® Xeon ® CPU @
Revised: The experiment on synthetic dataset is conducted on Google Colab with a 2-core Intel ® Xeon ® CPU @

Original: 2.20GHz.
Revised: 2.20GHz.

Original: The experiment on the real-world Asirra dataset (Elson et al., 2007) is conducted on an AWS p2-xlarge instance.
Revised: The experiment on the real-world Asirra dataset (Elson et al., 2007) is conducted on an AWS p2-xlarge instance.

Original: B. D ATA P REPROCESSING FOR THE A SIRRA D ATASET
Revised: B. D ATA P REPROCESSING FOR THE A SIRRA D ATASET

Original: [Table]
Revised: [Table]

Original: To demonstrate how our algorithm can deal with different levels of misspecification, we do data preprocessing before feeding the data into the agent.
Revised: To demonstrate how our algorithm can deal with different levels of misspecification, we do data preprocessing before feeding the data into the agent.

Original: As described in Section 8.2, the remaining data with expected misspecification level ζ are shown in Table 2.
Revised: As described in Section 8.2, the remaining data with expected misspecification level ζ are shown in Table 2.

Original: It can be verified that even with the smallest misspecification level, there are still more than 10% of the data is selected.
Revised: It can be verified that even with the smallest misspecification level, there are still more than 10% of the data is selected.

Original: B. A DDITIONAL R ESULT ON THE A SIRRA D ATASET
Revised: B. A DDITIONAL R ESULT ON THE A SIRRA D ATASET

Original: As a sensitivity analysis, we change the misspecification level in the preprocessing part in the Asirra dataset.
Revised: As a sensitivity analysis, we change the misspecification level in the preprocessing part in the Asirra dataset.

Original: The result is shown in Figure 2.
Revised: The result is shown in Figure 2.

Original: This result suggests that when the misspecification is small enough, setting Γ = ∆ / √ d can deliver a reasonable result.
Revised: This result suggests that when the misspecification is small enough, setting Γ = ∆ / √ d can deliver a reasonable result.

Original: It is aligned with the parameter setting in our theorem.
Revised: It is aligned with the parameter setting in our theorem.

Original: Meanwhile, we found that when ζ = 0 . 5 , which means it is strictly larger than the threshold ∆ / √ d , the algorithm cannot achieve a similar performance with of ζ < 0 .
Revised: Meanwhile, we found that when ζ = 0 . 5 , which means it is strictly larger than the threshold ∆ / √ d , the algorithm cannot achieve a similar performance with of ζ < 0 .

Original: 1 , regardless of the setting of parameter Γ .
Revised: 1 , regardless of the setting of parameter Γ .

Original: This also verifies the theoretical understanding of how a large misspecification level will harm the performance of the algorithm.
Revised: This also verifies the theoretical understanding of how a large misspecification level will harm the performance of the algorithm.

Original: 1.0 # of rounds 1e 

Revised: 1.0 # of rounds 1e

Original: C u m u l a t
Revised: C u m u l a t

Original: i v e
Revised: i v e

Original: R e g r e t =0 (OFUL) =0.05=0.
Revised: R e g r e t =0 (OFUL) =0.05=0.

Original: (a) ζ = 0 .0.
Revised: (a) ζ = 0 .0.

Original: 1.0 # of rounds 1e
Revised: 1.0 # of rounds 1e

Original: C u m u l a t
Revised: C u m u l a t

Original: i v e
Revised: i v e

Original: R e g r e t =0 (OFUL) =0.05=0.
Revised: R e g r e t =0 (OFUL) =0.05=0.

Original: (b) ζ = 0 .0.
Revised: (b) ζ = 0 .0.

Original: 1.0 # of rounds 1e
Revised: 1.0 # of rounds 1e

Original: C u m u l a t
Revised: C u m u l a t

Original: i v e
Revised: i v e

Original: R e g r e t =0 (OFUL) =0.05=0.
Revised: R e g r e t =0 (OFUL) =0.05=0.

Original: (c) ζ = 0 .
Revised: (c) ζ = 0 .

Original: B. A DDITIONAL E XPERIMENTS ON MISSPECIFIED LINEAR MDP S
Revised: B. A DDITIONAL E XPERIMENTS ON MISSPECIFIED LINEAR MDP S

Original: To further verify the performance of our algorithm in misspecified linear MDPs, we generate a synthetic MDP as follows.
Revised: To further verify the performance of our algorithm in misspecified linear MDPs, we generate a synthetic MDP as follows.

Original: We select S = 6 , A = 4 , H = 2 and generate ϕ ( s, a ) and µ ( s ′ ) ∼
Revised: We select S = 6 , A = 4 , H = 2 and generate ϕ ( s, a ) and µ ( s ′ ) ∼

Original: Unif.
Revised: Unif.

Original: ( 0 d , 1 d ) where d = 3 respectively.
Revised: ( 0 d , 1 d ) where d = 3 respectively.

Original: Since H = 2 , we define the first-stage transition kernel as
Revised: Since H = 2 , we define the first-stage transition kernel as

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: s ′ | s, a ) =
Revised: s ′ | s, a ) =

Original: [Equation]
Revised: [Equation]

Original: s, a ) , µ h ( s ′ ) ⟩ + η ( s ′ ) /S with normalization ϕ ( s, a ) =
Revised: s, a ) , µ h ( s ′ ) ⟩ + η ( s ′ ) /S with normalization ϕ ( s, a ) =

Original: ϕ ( s, a ) / (cid:80) s ∈ [ S ] ⟨ ϕ ( s, a ) , µ h ( s ′ ) ⟩ .
Revised: ϕ ( s, a ) / (cid:80) s ∈ [ S ] ⟨ ϕ ( s, a ) , µ h ( s ′ ) ⟩ .

Original: The misspecification error η ( s ′ ) satisfies that | η ( s ′ )
Revised: The misspecification error η ( s ′ ) satisfies that | η ( s ′ )

Original: | ≤ 2 × 10 − 3 and (cid:80) s ′ ∈
Revised: | ≤ 2 × 10 − 3 and (cid:80) s ′ ∈

Original: [ S ] η ( s ′ )
Revised: [ S ] η ( s ′ )

Original: = 0 .
Revised: = 0 .

Original: The reward is generated by r ( s, a ) = ⟨ ϕ ( s, a ) , θ ⟩ + η ( s, a ) with | η ( s, a ) | ≤ 5 × 10 − 3 .
Revised: The reward is generated by r ( s, a ) = ⟨ ϕ ( s, a ) , θ ⟩ + η ( s, a ) with | η ( s, a ) | ≤ 5 × 10 − 3 .

Original: It is easy to verify that the generated MDP is a ζ -approximate linear MDP with ζ = 5 × 10 − 3 according to Definition 7.1.
Revised: It is easy to verify that the generated MDP is a ζ -approximate linear MDP with ζ = 5 × 10 − 3 according to Definition 7.1.

Original: Then with the true transition kernel and the reward function, we can calculate that the minimal sub-optimality gap for the generated MDP is ∆ ≈ 0 . 0101 .
Revised: Then with the true transition kernel and the reward function, we can calculate that the minimal sub-optimality gap for the generated MDP is ∆ ≈ 0 . 0101 .

Original: According to our theory, we choose β = { 1 , 3 , 10 } and report the cumulative regret with different choices of Γ .
Revised: According to our theory, we choose β = { 1 , 3 , 10 } and report the cumulative regret with different choices of Γ .

Original: The results are shown in Table 3 and Figure 3.
Revised: The results are shown in Table 3 and Figure 3.

Original: Three key observations can be revealed from the experiment result in Table 3.
Revised: Three key observations can be revealed from the experiment result in Table 3.

Original: First, choosing Γ = 0 .
Revised: First, choosing Γ = 0 .

Original: 01 can achieve the best performance (lowest cumulative regret).
Revised: 01 can achieve the best performance (lowest cumulative regret).

Original: Second, choosing 0 .
Revised: Second, choosing 0 .

Original: 005 ≤ Γ ≤ 0 .
Revised: 005 ≤ Γ ≤ 0 .

Original: 01 can also lead to a comparable constant regret, according to Table 3, but smaller Γ may not lead to zero instantaneous regret within 200K rounds.
Revised: 01 can also lead to a comparable constant regret, according to Table 3, but smaller Γ may not lead to zero instantaneous regret within 200K rounds.

Original: Third, setting Γ > ∆ will lead to a linear regret.
Revised: Third, setting Γ > ∆ will lead to a linear regret.

Original: In addition, our algorithm degrades to LSVI-UCB algorithm Jin et al.
Revised: In addition, our algorithm degrades to LSVI-UCB algorithm Jin et al.

Original: (2020) by setting Γ = 0 .
Revised: (2020) by setting Γ = 0 .

Original: For misspecified linear MDPs, the algorithm studied in Theorem 3.2 of Jin et al.
Revised: For misspecified linear MDPs, the algorithm studied in Theorem 3.2 of Jin et al.

Original: (2020) is still
Revised: (2020) is still

Original: LSVI-UCB with a different choices of confidence radius β .
Revised: LSVI-UCB with a different choices of confidence radius β .

Original: In our experiments the β is tuned as a hyper-parameter.
Revised: In our experiments the β is tuned as a hyper-parameter.

Original: Therefore, the experiment results suggest that our algorithm outperforms the
Revised: Therefore, the experiment results suggest that our algorithm outperforms the

Original: LSVI-UCB algorithm in the misspecified setting.
Revised: LSVI-UCB algorithm in the misspecified setting.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: C D ETAILED PROOF OF T HEOREM 5. 
Revised: C D ETAILED PROOF OF T HEOREM 5.

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we provide the detailed proof for Theorem 5.1.
Revised: we provide the detailed proof for Theorem 5.1.

Original: First, we present a technical lemma to bound the total number of data used in the online linear regression in Algorithm 1.
Revised: First, we present a technical lemma to bound the total number of data used in the online linear regression in Algorithm 1.

Original: Lemma C.1.
Revised: Lemma C.1.

Original: Given 0 < Γ ≤ 1 , set λ
Revised: Given 0 < Γ ≤ 1 , set λ

Original: = B − 2 .
Revised: = B − 2 .

Original: For any k ∈ [ K ] , |C k | ≤ 16 d Γ − 2 log(3 LB Γ − 1 ) .
Revised: For any k ∈ [ K ] , |C k | ≤ 16 d Γ − 2 log(3 LB Γ − 1 ) .

Original: Lemma C.1 suggests that up to (cid:101) O
Revised: Lemma C.1 suggests that up to (cid:101) O

Original: ( d Γ − 2 ) contextual vectors has a UCB bonus greater than Γ .
Revised: ( d Γ − 2 ) contextual vectors has a UCB bonus greater than Γ .

Original: A similar result is also provided in He et al.
Revised: A similar result is also provided in He et al.

Original: (2021b), suggesting a (cid:101)
Revised: (2021b), suggesting a (cid:101)

Original: O (Γ − 2 ) Uniform-PAC sample complexity.
Revised: O (Γ − 2 ) Uniform-PAC sample complexity.

Original: Lemma C.1 also suggests that the numbers of data added in regression set C is finite, thus the regression procedure is not affected critically by the noise and the misspecification error.
Revised: Lemma C.1 also suggests that the numbers of data added in regression set C is finite, thus the regression procedure is not affected critically by the noise and the misspecification error.

Original: For a linear regression with up to |C k | data, the next lemma is crucial in controlling the fluctuations with misspecification error.
Revised: For a linear regression with up to |C k | data, the next lemma is crucial in controlling the fluctuations with misspecification error.

Original: Lemma C.2.
Revised: Lemma C.2.

Original: Let λ =
Revised: Let λ =

Original: B − 2 .
Revised: B − 2 .

Original: For all δ > 0 , with probability at least 1 − δ , for all x ∈ R d , k ∈ [ K ] , the estimation error is bounded by:
Revised: For all δ > 0 , with probability at least 1 − δ , for all x ∈ R d , k ∈ [ K ] , the estimation error is bounded by:

Original: [Equation]
Revised: [Equation]

Original: where ι = log(( d + |C k | L 2 B 2 ) / ( dδ )) and |C k | is the total number of data used in regression at k -th round.
Revised: where ι = log(( d + |C k | L 2 B 2 ) / ( dδ )) and |C k | is the total number of data used in regression at k -th round.

Original: Lemma C.2 provides a similar decomposition as the well-specified linear contextual bandits algorithms like OFUL (Abbasi-Yadkori et al., 2011).
Revised: Lemma C.2 provides a similar decomposition as the well-specified linear contextual bandits algorithms like OFUL (Abbasi-Yadkori et al., 2011).

Original: However, comparing the confidence radius here (cid:101) O
Revised: However, comparing the confidence radius here (cid:101) O

Original: ( R √ d + ζ (cid:112)
Revised: ( R √ d + ζ (cid:112)

Original: |C k − 1 | ) with the conventional radius in OFUL (cid:101)
Revised: |C k − 1 | ) with the conventional radius in OFUL (cid:101)

Original: O
Revised: O

Original: ( R √ d ) , one can find that the misspecification term will affect the radius in an (cid:112)
Revised: ( R √ d ) , one can find that the misspecification term will affect the radius in an (cid:112)

Original: |C K | order.
Revised: |C K | order.

Original: If we directly use all data to do the regression, the confidence radius will be in the order of (cid:101) O ( √ K ) and therefore would lead to a
Revised: If we directly use all data to do the regression, the confidence radius will be in the order of (cid:101) O ( √ K ) and therefore would lead to a

Original: O ( K √ log K ) regret bound (see Lemma 11 in Abbasi-Yadkori et al.
Revised: O ( K √ log K ) regret bound (see Lemma 11 in Abbasi-Yadkori et al.

Original: This makes the regret bound trivial since it goes beyond the trivial regret upper bound O ( K ) when K grows larger.
Revised: This makes the regret bound trivial since it goes beyond the trivial regret upper bound O ( K ) when K grows larger.

Original: In ours, however, the confidence radius is only (cid:112) |C K | where |C K | is finite given Lemma C.1.
Revised: In ours, however, the confidence radius is only (cid:112) |C K | where |C K | is finite given Lemma C.1.

Original: As a result, our regret bound will not grow with K as OFUL, and will have a more stable prediction.
Revised: As a result, our regret bound will not grow with K as OFUL, and will have a more stable prediction.

Original: When the misspecification level is well bounded by ζ = (cid:101) O (∆ / √ d ) , the following corollary is a direct result of Lemmas C.2 by replacing the term |C K | with its upper bound provided in Lemma C.1.
Revised: When the misspecification level is well bounded by ζ = (cid:101) O (∆ / √ d ) , the following corollary is a direct result of Lemmas C.2 by replacing the term |C K | with its upper bound provided in Lemma C.1.

Original: Corollary C.3.
Revised: Corollary C.3.

Original: Suppose 2 √ dζι 1 ≤ ∆ , let λ
Revised: Suppose 2 √ dζι 1 ≤ ∆ , let λ

Original: =
Revised: =

Original: B − 2 and 0 < Γ ≤ 1 .
Revised: B − 2 and 0 < Γ ≤ 1 .

Original: Let β = 1+2∆Γ − 1 √ ι 2 /ι 1 +
Revised: Let β = 1+2∆Γ − 1 √ ι 2 /ι 1 +

Original: R √ 2 dι 3 where ι 2 = log(3 LB Γ − 1 ) , ι 3 = log((

Revised: R √ 2 dι 3 where ι 2 = log(3 LB Γ − 1 ) , ι 3 = log((

Original: +16 L 2 B 2 Γ − 2 ι 2 ) /δ ) , then with probability at least
Revised: +16 L 2 B 2 Γ − 2 ι 2 ) /δ ) , then with probability at least

Original: 1 − δ , for all x ∈ R d , k ∈
Revised: 1 − δ , for all x ∈ R d , k ∈

Original: [ K ] , the estimation error at round k ∈ [ K ] is bounded by: | x ⊤ ( θ k − θ ∗ ) | ≤ β ∥ x
Revised: [ K ] , the estimation error at round k ∈ [ K ] is bounded by: | x ⊤ ( θ k − θ ∗ ) | ≤ β ∥ x

Original: ∥
Revised: ∥

Original: U k − 1 .
Revised: U k − 1 .

Original: Proof.
Revised: Proof.

Original: By Lemma C.1, replacing |C K | with its upper bound yields
Revised: By Lemma C.1, replacing |C K | with its upper bound yields

Original: [Equation]
Revised: [Equation]

Original: where the second inequality is due to the condition 2 √ dζ ≤ ∆ /ι 1 .
Revised: where the second inequality is due to the condition 2 √ dζ ≤ ∆ /ι 1 .

Original: Next we introduce an auxiliary lemma controlling the instantaneous regret bound using the UCB bonus and the misspecification level.
Revised: Next we introduce an auxiliary lemma controlling the instantaneous regret bound using the UCB bonus and the misspecification level.

Original: Lemma C.4.
Revised: Lemma C.4.

Original: Suppose Corollary C.3 holds, for all k ∈ [ K ] , the instantaneous regret at round k is bounded by
Revised: Suppose Corollary C.3 holds, for all k ∈ [ K ] , the instantaneous regret at round k is bounded by

Original: [Equation]
Revised: [Equation]

Original: The next auxiliary lemma from He et al.
Revised: The next auxiliary lemma from He et al.

Original: (2021a) bounds the summation of a subset of the selfnormalized vectors.
Revised: (2021a) bounds the summation of a subset of the selfnormalized vectors.

Original: Lemma C.5 (Lemma 6.6, He et al. 2021a) .
Revised: Lemma C.5 (Lemma 6.6, He et al. 2021a) .

Original: For any subset G = { c 1 , · · · , c i } ⊆
Revised: For any subset G = { c 1 , · · · , c i } ⊆

Original: C K , we have (cid:88)
Revised: C K , we have (cid:88)

Original: [Equation]
Revised: [Equation]

Original: The next two technical algebra lemma is used to control the dominated terms
Revised: The next two technical algebra lemma is used to control the dominated terms

Original: Lemma C.6.
Revised: Lemma C.6.

Original: Let ι 1 = (24+18 R ) log((72+54 R ) LB √ d ∆ − 1 )+ (cid:112) 8 R 2 log(1 /δ ) , Γ = ∆ / (2 √ dι 1 ) , ι 2 = log(3 LB Γ − 1 ) , ι 3 = log((

Revised: Let ι 1 = (24+18 R ) log((72+54 R ) LB √ d ∆ − 1 )+ (cid:112) 8 R 2 log(1 /δ ) , Γ = ∆ / (2 √ dι 1 ) , ι 2 = log(3 LB Γ − 1 ) , ι 3 = log((

Original: + 16 L 2 B 2 Γ − 2 ι 2 ) /δ ) , we have ι 1 > 2 + 4 √ ι 2 + R √ 2 ι 3 .
Revised: + 16 L 2 B 2 Γ − 2 ι 2 ) /δ ) , we have ι 1 > 2 + 4 √ ι 2 + R √ 2 ι 3 .

Original: Equipped with these lemmas, we can start the proof of Theorem 5.1.
Revised: Equipped with these lemmas, we can start the proof of Theorem 5.1.

Original: Proof of Theorem 5.1.
Revised: Proof of Theorem 5.1.

Original: First, it worth mentioning that by setting Γ = ∆ / (2 √ dι 1 ) , the confidence radius β becomes 1 + 4 √
Revised: First, it worth mentioning that by setting Γ = ∆ / (2 √ dι 1 ) , the confidence radius β becomes 1 + 4 √

Original: dι 2 + R √ 

Revised: dι 2 + R √

Original: dι 3 .
Revised: dι 3 .

Original: Then our proof starts with assuming that Corollary C. holds with probability at least 1 − δ .
Revised: Then our proof starts with assuming that Corollary C. holds with probability at least 1 − δ .

Original: We decompose the index set
Revised: We decompose the index set

Original: [ K ] into two subsets.
Revised: [ K ] into two subsets.

Original: The first set is [ K ] \ C K indicating the non-selected data, the second set is the selected set C K .
Revised: The first set is [ K ] \ C K indicating the non-selected data, the second set is the selected set C K .

Original: We will bound the cumulative regret within two set separately.
Revised: We will bound the cumulative regret within two set separately.

Original: First, for those non-selected data k / ∈ C k , i.e. ∥ x k ∥ U − 1 k < Γ , combining Lemma C.4 with Corollary C.3 yields
Revised: First, for those non-selected data k / ∈ C k , i.e. ∥ x k ∥ U − 1 k < Γ , combining Lemma C.4 with Corollary C.3 yields

Original: [Equation]
Revised: [Equation]

Original: where ι 1 , ι 2 , ι 3 are the same as Theorem 5.1, and the second equation is from Γ =
Revised: where ι 1 , ι 2 , ι 3 are the same as Theorem 5.1, and the second equation is from Γ =

Original: ∆ / (2 √ dι 1 ) .
Revised: ∆ / (2 √ dι 1 ) .

Original: When misspecification condition 2 √ dζ ≤ ∆ /ι 3 holds, (C.1) suggests that
Revised: When misspecification condition 2 √ dζ ≤ ∆ /ι 3 holds, (C.1) suggests that

Original: [Equation]
Revised: [Equation]

Original: Lemma C.6 suggests that when ι 1 = (24 + 18 R ) log((72 + 54 R ) LB √ d ∆ − 1 ) + (cid:112) 8 R 2 log(1 /δ )
Revised: Lemma C.6 suggests that when ι 1 = (24 + 18 R ) log((72 + 54 R ) LB √ d ∆ − 1 ) + (cid:112) 8 R 2 log(1 /δ )

Original: ι 1 > 2 + 4 √
Revised: ι 1 > 2 + 4 √

Original: ι 2 + R √ 2 ι 3 , (C.2) yields that the instantaneous regret r ∗ k − r ( x k ) <
Revised: ι 2 + R √ 2 ι 3 , (C.2) yields that the instantaneous regret r ∗ k − r ( x k ) <

Original: ∆ at round k .
Revised: ∆ at round k .

Original: By
Revised: By

Original: Definition 3.1, the instantaneous regret is zero for all k / ∈ C k , indicating the non-selected data can incur zero instantaneous regret.
Revised: Definition 3.1, the instantaneous regret is zero for all k / ∈ C k , indicating the non-selected data can incur zero instantaneous regret.

Original: Then Lemma C.4 suggests that the instantaneous regret for those k ∈ C K is bounded by
Revised: Then Lemma C.4 suggests that the instantaneous regret for those k ∈ C K is bounded by

Original: [Equation]
Revised: [Equation]

Original: where the second inequality follows the C.S. inequality, the third one yields from Lemma C.5 while the fourth utilize the fact that Γ = ∆ / (2 √ dι 1 ) and ζ ≤ ∆ / (2 √ dι 1 ) .
Revised: where the second inequality follows the C.S. inequality, the third one yields from Lemma C.5 while the fourth utilize the fact that Γ = ∆ / (2 √ dι 1 ) and ζ ≤ ∆ / (2 √ dι 1 ) .

Original: The last one is due to the fact that the second term in the forth inequality is dominated by the first one.
Revised: The last one is due to the fact that the second term in the forth inequality is dominated by the first one.

Original: To warp up, the cumulative regret can be decomposed by
Revised: To warp up, the cumulative regret can be decomposed by

Original: Regret ( K ) =
Revised: Regret ( K ) =

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: where the first two zeros are given by the fact that for k / ∈ C K , we have r ∗ k − r ( x k )
Revised: where the first two zeros are given by the fact that for k / ∈ C K , we have r ∗ k − r ( x k )

Original: = 0 .
Revised: = 0 .

Original: the regret bound for k ∈ G is given by (C.3).
Revised: the regret bound for k ∈ G is given by (C.3).

Original: D P ROOF OF T ECHNICAL L EMMAS IN A PPENDIX C D. P ROOF OF L EMMA C. 
Revised: D P ROOF OF T ECHNICAL L EMMAS IN A PPENDIX C D. P ROOF OF L EMMA C.

Original: To prove this lemma, we introduce the well known elliptical potential lemma by Abbasi-Yadkori et al.
Revised: To prove this lemma, we introduce the well known elliptical potential lemma by Abbasi-Yadkori et al.

Original: Lemma D.1 (Lemma 11, Abbasi-Yadkori et al. 2011) .
Revised: Lemma D.1 (Lemma 11, Abbasi-Yadkori et al. 2011) .

Original: Let { ϕ i } Ii =1 be a sequence in R d , define
Revised: Let { ϕ i } Ii =1 be a sequence in R d , define

Original: U i
Revised: U i

Original: =
Revised: =

Original: λ
Revised: λ

Original: I + (cid:80) ij =1 ϕ j ϕ ⊤ j , then
Revised: I + (cid:80) ij =1 ϕ j ϕ ⊤ j , then

Original: [Equation]
Revised: [Equation]

Original: Then the following auxiliary lemma and its corollary are useful
Revised: Then the following auxiliary lemma and its corollary are useful

Original: Lemma D.2 (Lemma A.2, Shalev-Shwartz & Ben-David 2014) .
Revised: Lemma D.2 (Lemma A.2, Shalev-Shwartz & Ben-David 2014) .

Original: Let a ≥ 1 and b > 0 .
Revised: Let a ≥ 1 and b > 0 .

Original: Then x ≥ 4 a log(2 a ) + 2 b yields x ≥ a log( x ) + b .
Revised: Then x ≥ 4 a log(2 a ) + 2 b yields x ≥ a log( x ) + b .

Original: Lemma D.2 can easily indicate the following lemma.
Revised: Lemma D.2 can easily indicate the following lemma.

Original: Lemma D.3.
Revised: Lemma D.3.

Original: Let a ≥ 1 .
Revised: Let a ≥ 1 .

Original: Then x ≥ 4 log(2 a ) + a − 1 yields x ≥ log(1 + ax ) .
Revised: Then x ≥ 4 log(2 a ) + a − 1 yields x ≥ log(1 + ax ) .

Original: Proof.
Revised: Proof.

Original: Let y = 1 + ax, x =
Revised: Let y = 1 + ax, x =

Original: ( y − 1)
Revised: ( y − 1)

Original: /a .
Revised: /a .

Original: Then x ≥ 4 log(2 a ) + a − 1 is equivalent with y ≥ 4 a log(2 a )+2 .
Revised: Then x ≥ 4 log(2 a ) + a − 1 is equivalent with y ≥ 4 a log(2 a )+2 .

Original: By Lemma D.2, this implies y ≥ a log( y )+1 which is exactly x ≥ log(1+ ax ) .
Revised: By Lemma D.2, this implies y ≥ a log( y )+1 which is exactly x ≥ log(1+ ax ) .

Original: Equipped with these technical lemmas, we can start our proof.
Revised: Equipped with these technical lemmas, we can start our proof.

Original: Proof of Lemma C.1.
Revised: Proof of Lemma C.1.

Original: Since the cardinality of set C k is monotonically increasing w.r.t.
Revised: Since the cardinality of set C k is monotonically increasing w.r.t.

Original: k , we fix k to be K in the proof and only provide the bound of C K .
Revised: k , we fix k to be K in the proof and only provide the bound of C K .

Original: For all selected data k ∈ C K , we have ∥ ϕ k ∥
Revised: For all selected data k ∈ C K , we have ∥ ϕ k ∥

Original: U − 1 k ≥ Γ .
Revised: U − 1 k ≥ Γ .

Original: Therefore when Γ ≤ 1 , the summation of the UCB bonus over data k ∈ C K is lower bounded by (cid:88)
Revised: Therefore when Γ ≤ 1 , the summation of the UCB bonus over data k ∈ C K is lower bounded by (cid:88)

Original: [Equation]
Revised: [Equation]

Original: On the other hand, Lemma D.1 implies (cid:88)
Revised: On the other hand, Lemma D.1 implies (cid:88)

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Combining (D.2) and (D.1), the total number of the selected samples |C K | is bounded by
Revised: Combining (D.2) and (D.1), the total number of the selected samples |C K | is bounded by

Original: [Equation]
Revised: [Equation]

Original: This result can be re-organized as
Revised: This result can be re-organized as

Original: [Equation]
Revised: [Equation]

Original: Let λ =
Revised: Let λ =

Original: B − 2 and since 2 L 2 B 2 ≥ 2 ≥ Γ 2 , Lemma D.3 suggests that if Γ
Revised: B − 2 and since 2 L 2 B 2 ≥ 2 ≥ Γ 2 , Lemma D.3 suggests that if Γ

Original: [Equation]
Revised: [Equation]

Original: |C K
Revised: |C K

Original: [Equation]
Revised: [Equation]

Original: then (D.3) will not hold.
Revised: then (D.3) will not hold.

Original: Thus the necessary condition for (D.3) is Γ 2 |C K |
Revised: Thus the necessary condition for (D.3) is Γ 2 |C K |

Original: [Equation]
Revised: [Equation]

Original: By basic calculus we get the claimed bound for |C K | and complete the proof of Lemma C.1.
Revised: By basic calculus we get the claimed bound for |C K | and complete the proof of Lemma C.1.

Original: D.
Revised: D.

Original: P ROOF OF L EMMA C. 
Revised: P ROOF OF L EMMA C.

Original: The proof follows the standard technique for linear bandits, we first introduce the self-normalized bound for vector-valued martingales from Abbasi-Yadkori et al.
Revised: The proof follows the standard technique for linear bandits, we first introduce the self-normalized bound for vector-valued martingales from Abbasi-Yadkori et al.

Original: Lemma D.4 (Theorem 1, Abbasi-Yadkori et al. 2011) .
Revised: Lemma D.4 (Theorem 1, Abbasi-Yadkori et al. 2011) .

Original: Let {F t } ∞ t
Revised: Let {F t } ∞ t

Original: =
Revised: =

Original: be a filtration.
Revised: be a filtration.

Original: Let { ε t } ∞ t =1 be a real-valued stochastic process such that ε t is F t -measurable and ε t is conditionally R -sub-Gaussian for some R ≥ 0 .
Revised: Let { ε t } ∞ t =1 be a real-valued stochastic process such that ε t is F t -measurable and ε t is conditionally R -sub-Gaussian for some R ≥ 0 .

Original: Let { ϕ t } ∞ t =1 be an R d -valued stochastic process such that ϕ t is F t − 1 measurable and ∥ ϕ ∥ 2 ≤ L for all t .
Revised: Let { ϕ t } ∞ t =1 be an R d -valued stochastic process such that ϕ t is F t − 1 measurable and ∥ ϕ ∥ 2 ≤ L for all t .

Original: For any t ≥ 0 , define U t
Revised: For any t ≥ 0 , define U t

Original: = λ
Revised: = λ

Original: I + (cid:80) tk =1 ϕ k ϕ k .
Revised: I + (cid:80) tk =1 ϕ k ϕ k .

Original: Then for any δ > 0 , with probability at least 1 − δ , for all t ≥
Revised: Then for any δ > 0 , with probability at least 1 − δ , for all t ≥

Original: [Equation]
Revised: [Equation]

Original: Lemma D.5 (Lemma 8, Zanette et al.
Revised: Lemma D.5 (Lemma 8, Zanette et al.

Original: (2020c)) .
Revised: (2020c)) .

Original: Let { a i } di =1 be any sequence of vectors in R d and { b i } di =1 be any sequence of scalars such that | b i | ≤ ϵ .
Revised: Let { a i } di =1 be any sequence of vectors in R d and { b i } di =1 be any sequence of scalars such that | b i | ≤ ϵ .

Original: For any λ > 0 :
Revised: For any λ > 0 :

Original: [Equation]
Revised: [Equation]

Original: The next lemma is to bound the perturbation of the misspecification
Revised: The next lemma is to bound the perturbation of the misspecification

Original: Lemma D.6.
Revised: Lemma D.6.

Original: Let { η k } k be any sequence of scalars such that | η k | ≤ ζ for any k ∈ [ K ] .
Revised: Let { η k } k be any sequence of scalars such that | η k | ≤ ζ for any k ∈ [ K ] .

Original: For any index subset C ⊆
Revised: For any index subset C ⊆

Original: [ K ] , define U
Revised: [ K ] , define U

Original: =
Revised: =

Original: λ I + (cid:80) k ∈C x k x ⊤ k , then for any x ∈ R d , we have
Revised: λ I + (cid:80) k ∈C x k x ⊤ k , then for any x ∈ R d , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By Cauchy-Schwartz inequality we have
Revised: By Cauchy-Schwartz inequality we have

Original: [Equation]
Revised: [Equation]

Original: where the second inequality dues to lemma D.5.
Revised: where the second inequality dues to lemma D.5.

Original: The next lemma provides the Determinant-Trace inequality.
Revised: The next lemma provides the Determinant-Trace inequality.

Original: Lemma D.7.
Revised: Lemma D.7.

Original: Suppose sequence { x k } Kk =1 ⊂ R d and for any k ∈ [ K ] , ∥ x k ∥ 2 ≤ L .
Revised: Suppose sequence { x k } Kk =1 ⊂ R d and for any k ∈ [ K ] , ∥ x k ∥ 2 ≤ L .

Original: For any index subset C ⊆
Revised: For any index subset C ⊆

Original: [ K ] , define U
Revised: [ K ] , define U

Original: =
Revised: =

Original: λ
Revised: λ

Original: I + (cid:80) k ∈C x k x ⊤ k for some λ > 0 , then det( U ) ≤ ( λ + |C| L 2 /d )
Revised: I + (cid:80) k ∈C x k x ⊤ k for some λ > 0 , then det( U ) ≤ ( λ + |C| L 2 /d )

Original: d .
Revised: d .

Original: Proof.
Revised: Proof.

Original: The proof of this lemma is almost the same with Lemma 10 in Abbasi-Yadkori et al.
Revised: The proof of this lemma is almost the same with Lemma 10 in Abbasi-Yadkori et al.

Original: (2011) by replacing the index set [ K ] with any subset C .
Revised: (2011) by replacing the index set [ K ] with any subset C .

Original: We refer the readers to check Abbasi-Yadkori et al.
Revised: We refer the readers to check Abbasi-Yadkori et al.

Original: (2011) for details.
Revised: (2011) for details.

Original: Equipped with these lemmas, we can start our proof.
Revised: Equipped with these lemmas, we can start our proof.

Original: Proof of Lemma C.2.
Revised: Proof of Lemma C.2.

Original: For any k ∈ [ K ] , considering the data samples k ′ ∈
Revised: For any k ∈ [ K ] , considering the data samples k ′ ∈

Original: C k − 1 used for regression at round k .
Revised: C k − 1 used for regression at round k .

Original: Following the update rule of U k and θ k yields
Revised: Following the update rule of U k and θ k yields

Original: [Equation]
Revised: [Equation]

Original: where the first equation is due to the fact that U
Revised: where the first equation is due to the fact that U

Original: k
Revised: k

Original: = λ
Revised: = λ

Original: I + (cid:80) k
Revised: I + (cid:80) k

Original: [Equation]
Revised: [Equation]

Original: ∈C k − 1 x k x ⊤ k and θ
Revised: ∈C k − 1 x k x ⊤ k and θ

Original: [Equation]
Revised: [Equation]

Original: The last equation follows the fact that r k ′ is generated from r k ′ =
Revised: The last equation follows the fact that r k ′ is generated from r k ′ =

Original: r ( x k ′ )
Revised: r ( x k ′ )

Original: + ε k ′ =
Revised: + ε k ′ =

Original: x ⊤ k ′ θ ∗ + η ( x k ′ ) + ε k ′ , where we denote η ( x k ′ ) as η k ′ for the model misspecification error and ε k ′ is the random noise.
Revised: x ⊤ k ′ θ ∗ + η ( x k ′ ) + ε k ′ , where we denote η ( x k ′ ) as η k ′ for the model misspecification error and ε k ′ is the random noise.

Original: Therefore, consider any contextual vector x ∈ R d , we have
Revised: Therefore, consider any contextual vector x ∈ R d , we have

Original: [Equation]
Revised: [Equation]

Original: where the inequality is due to the triangles inequality.
Revised: where the inequality is due to the triangles inequality.

Original: Lemma D.6 yields q 3 ≤ ζ (cid:112) |C k − 1 |∥ x ∥ U − 1 k .
Revised: Lemma D.6 yields q 3 ≤ ζ (cid:112) |C k − 1 |∥ x ∥ U − 1 k .

Original: From the fact that | x ⊤ Ay | ≤ ∥ x
Revised: From the fact that | x ⊤ Ay | ≤ ∥ x

Original: ∥
Revised: ∥

Original: A ∥ y ∥ A , we can bound term q 1 by
Revised: A ∥ y ∥ A , we can bound term q 1 by

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to the fact that U − 1 i ⪯ λ − 1 I .
Revised: where the last inequality is due to the fact that U − 1 i ⪯ λ − 1 I .

Original: Term q 2 is also bounded as
Revised: Term q 2 is also bounded as

Original: [Equation]
Revised: [Equation]

Original: where the second equation uses the indicator to replace the summation over subset C k − 1 .
Revised: where the second equation uses the indicator to replace the summation over subset C k − 1 .

Original: Denoting y
Revised: Denoting y

Original: k
Revised: k

Original: =
Revised: =

Original: [ k ′ ∈
Revised: [ k ′ ∈

Original: C k − 1 ] x k ′ , noticing that ∥ y k ∥ 2 ≤ ∥ x k ∥ 2 ≤ L and
Revised: C k − 1 ] x k ′ , noticing that ∥ y k ∥ 2 ≤ ∥ x k ∥ 2 ≤ L and

Original: U k
Revised: U k

Original: =
Revised: =

Original: (cid:88) k ′ ∈C
Revised: (cid:88) k ′ ∈C

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: 1 [ k
Revised: 1 [ k

Original: [Equation]
Revised: [Equation]

Original: ∈ C k −
Revised: ∈ C k −

Original: [Equation]
Revised: [Equation]

Original: x k ′ x ⊤ k ′
Revised: x k ′ x ⊤ k ′

Original: [Equation]
Revised: [Equation]

Original: K (cid:88) k ′
Revised: K (cid:88) k ′

Original: [Equation]
Revised: [Equation]

Original: y k
Revised: y k

Original: [Equation]
Revised: [Equation]

Original: y ⊤ k
Revised: y ⊤ k

Original: [Equation]
Revised: [Equation]

Original: by Lemma D.4, I 1 can be further bounded by
Revised: by Lemma D.4, I 1 can be further bounded by

Original: I
Revised: I

Original: [Equation]
Revised: [Equation]

Original: (cid:115) 2 log (cid:18)
Revised: (cid:115) 2 log (cid:18)

Original: [Equation]
Revised: [Equation]

Original: U k )
Revised: U k )

Original: [Equation]
Revised: [Equation]

Original: where the second inequality follows the fact that det( U k ) ≥ det( U 0 ) =
Revised: where the second inequality follows the fact that det( U k ) ≥ det( U 0 ) =

Original: λ d .
Revised: λ d .

Original: Notice that U k
Revised: Notice that U k

Original: = λ
Revised: = λ

Original: I + (cid:80) k ′ ∈C k − 1 x k ′ x ⊤ k ′ , lemma D.7 suggests that det( U k ) ≤ ( λ + |C k − 1 | L 2 /d )
Revised: I + (cid:80) k ′ ∈C k − 1 x k ′ x ⊤ k ′ , lemma D.7 suggests that det( U k ) ≤ ( λ + |C k − 1 | L 2 /d )

Original: d , plugging this into (D.6) we have I 1 can be finally bounded by
Revised: d , plugging this into (D.6) we have I 1 can be finally bounded by

Original: I
Revised: I

Original: [Equation]
Revised: [Equation]

Original: Plugging the bound of I 1 into (D.5) and combining with (D.4) and Lemma D.6 together, replacing |C k − 1 | with its upper bound |C K | we have with probability at least 1 − δ , for all k ∈ [ K ] , x ∈ R d ,
Revised: Plugging the bound of I 1 into (D.5) and combining with (D.4) and Lemma D.6 together, replacing |C k − 1 | with its upper bound |C K | we have with probability at least 1 − δ , for all k ∈ [ K ] , x ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: Letting λ
Revised: Letting λ

Original: = B − 2 we get the claimed results.
Revised: = B − 2 we get the claimed results.

Original: D. P ROOF OF L EMMA C. 
Revised: D. P ROOF OF L EMMA C.

Original: Proof.
Revised: Proof.

Original: According to the definition of expected reward function r ( x ) , we have for all k ∈
Revised: According to the definition of expected reward function r ( x ) , we have for all k ∈

Original: [ K ] , suppose the condition in Lemma C.2 holds, then
Revised: [ K ] , suppose the condition in Lemma C.2 holds, then

Original: [Equation]
Revised: [Equation]

Original: where the second inequality utilize the fact that | η ( x ) | ≤ ζ for all x ∈ D k .
Revised: where the second inequality utilize the fact that | η ( x ) | ≤ ζ for all x ∈ D k .

Original: The inequality on the forth line follows Corollary C.3.
Revised: The inequality on the forth line follows Corollary C.3.

Original: The inequality on the fifth line is due to the fact that x k = argmax x ∈D k x ⊤ θ k + β
Revised: The inequality on the fifth line is due to the fact that x k = argmax x ∈D k x ⊤ θ k + β

Original: ∥
Revised: ∥

Original: x
Revised: x

Original: ∥ U − 1 k , which is executed in Line 4 in Algorithm 1.
Revised: ∥ U − 1 k , which is executed in Line 4 in Algorithm 1.

Original: D. P ROOF OF L EMMA C. 
Revised: D. P ROOF OF L EMMA C.

Original: Proof.
Revised: Proof.

Original: First it is clear to see that √ 2 ι 3 = (cid:112) 2 log(1 + 16 L 2 B 2 Γ − 2 ι 2 ) + 2 log(1 /δ ) .
Revised: First it is clear to see that √ 2 ι 3 = (cid:112) 2 log(1 + 16 L 2 B 2 Γ − 2 ι 2 ) + 2 log(1 /δ ) .

Original: Using the fact that √ a + b ≤ √ a + √ b , it can be further bounded by
Revised: Using the fact that √ a + b ≤ √ a + √ b , it can be further bounded by

Original: [Equation]
Revised: [Equation]

Original: Assuming L ≥ 1 , B ≥ 1 , Γ = ∆ / (2 √ dι 1 ) ≤ 1 yields LB Γ − 1 ≥ 1 , then by basic calculus one can verify that
Revised: Assuming L ≥ 1 , B ≥ 1 , Γ = ∆ / (2 √ dι 1 ) ≤ 1 yields LB Γ − 1 ≥ 1 , then by basic calculus one can verify that

Original: [Equation]
Revised: [Equation]

Original: therefore we have that
Revised: therefore we have that

Original: [Equation]
Revised: [Equation]

Original: where the last equality is from the fact that Γ = ∆ / (2 √ dι 1 ) .
Revised: where the last equality is from the fact that Γ = ∆ / (2 √ dι 1 ) .

Original: Lemma D.2 suggests that the necessary condition for
Revised: Lemma D.2 suggests that the necessary condition for

Original: [Equation]
Revised: [Equation]

Original: is that
Revised: is that

Original: [Equation]
Revised: [Equation]

Original: which suggests that setting
Revised: which suggests that setting

Original: [Equation]
Revised: [Equation]

Original: implies the fact that ι 1 ≥ 2 + 4 √ ι 2 + R √ 2 ι
Revised: implies the fact that ι 1 ≥ 2 + 4 √ ι 2 + R √ 2 ι

Original: E P ROOF OF T HEOREM 5. 
Revised: E P ROOF OF T HEOREM 5.

Original: To begin with, we introduce the lemma providing a sparse vector set in R d .
Revised: To begin with, we introduce the lemma providing a sparse vector set in R d .

Original: Lemma E.1 (Lemma 3.1, Lattimore et al. 2020) .
Revised: Lemma E.1 (Lemma 3.1, Lattimore et al. 2020) .

Original: For any ε > 0 and d < [ |D| ] such that d ≥ ⌈ 8 log( |D| ) ε − 2 ⌉ , there exists a vector set D ⊂ R d
Revised: For any ε > 0 and d < [ |D| ] such that d ≥ ⌈ 8 log( |D| ) ε − 2 ⌉ , there exists a vector set D ⊂ R d

Original: such that ∥ x ∥ 2 = 1 for all x ∈ D and | ⟨ x , y ⟩ | ≤ ε for all x , y ∈ D and x ̸ = y .
Revised: such that ∥ x ∥ 2 = 1 for all x ∈ D and | ⟨ x , y ⟩ | ≤ ε for all x , y ∈ D and x ̸ = y .

Original: Next, we present the Bretagnolle–Huber inequality providing the lower bound to distinguish a system.
Revised: Next, we present the Bretagnolle–Huber inequality providing the lower bound to distinguish a system.

Original: Lemma E.2 (Bretagnolle–Huber inequality) .
Revised: Lemma E.2 (Bretagnolle–Huber inequality) .

Original: Let P and Q be probability measures on the same measurable space (Ω , F ) , let A ∈ F be an arbitary event.
Revised: Let P and Q be probability measures on the same measurable space (Ω , F ) , let A ∈ F be an arbitary event.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: For stochastic linear bandit problem with finite arm, we can denote T i ( k ) as the number of rounds the algorithm visit the i -th arm over total k rounds.
Revised: For stochastic linear bandit problem with finite arm, we can denote T i ( k ) as the number of rounds the algorithm visit the i -th arm over total k rounds.

Original: Then We have the KL-divergence decomposition lemma.
Revised: Then We have the KL-divergence decomposition lemma.

Original: Lemma E.3 (Lemma 15.1, Lattimore & Szepesv ´ ari (2020)) .
Revised: Lemma E.3 (Lemma 15.1, Lattimore & Szepesv ´ ari (2020)) .

Original: Let ν =
Revised: Let ν =

Original: ( P 1 , · · · , P n ) be the reward distributions associated with one n -armed bandit and let ν ′ =
Revised: ( P 1 , · · · , P n ) be the reward distributions associated with one n -armed bandit and let ν ′ =

Original: ( P ′ 1 , · · · , P ′ n ) be another n -armed bandit.
Revised: ( P ′ 1 , · · · , P ′ n ) be another n -armed bandit.

Original: Fix some algorithm π and let P ν
Revised: Fix some algorithm π and let P ν

Original: = P νπ , P ν ′ =
Revised: = P νπ , P ν ′ =

Original: P ν ′ ,
Revised: P ν ′ ,

Original: π be the probability measures on the canonical bandit model induced by the k -round interconnection of π and ν (respectively, π and ν ′ ).
Revised: π be the probability measures on the canonical bandit model induced by the k -round interconnection of π and ν (respectively, π and ν ′ ).

Original: Then KL( P ν , P ν ′ ) =
Revised: Then KL( P ν , P ν ′ ) =

Original: (cid:80) ni
Revised: (cid:80) ni

Original: =1 E ν
Revised: =1 E ν

Original: [ T i ( n )]
Revised: [ T i ( n )]

Original: KL( P i , P ′ i )
Revised: KL( P i , P ′ i )

Original: Proof of Theorem 5.4.
Revised: Proof of Theorem 5.4.

Original: The proof starts from inheriting the idea from Lattimore et al.
Revised: The proof starts from inheriting the idea from Lattimore et al.

Original: Given dimension d and the number of arms |D| , setting ε = (cid:112) 8 log( |D| ) / ( d − 1) , we can provide the contextual vector set D such that
Revised: Given dimension d and the number of arms |D| , setting ε = (cid:112) 8 log( |D| ) / ( d − 1) , we can provide the contextual vector set D such that

Original: [Equation]
Revised: [Equation]

Original: For simplicity, we index the decision set as x 1 , · · · , x |D| .
Revised: For simplicity, we index the decision set as x 1 , · · · , x |D| .

Original: Given the minimal sub-optimality gap ∆ , we provide the parameter set Θ as follows:
Revised: Given the minimal sub-optimality gap ∆ , we provide the parameter set Θ as follows:

Original: [Equation]
Revised: [Equation]

Original: It can be verified that Θ contains two kinds of θ .
Revised: It can be verified that Θ contains two kinds of θ .

Original: The first one θ ( i,j ) is a mixture of two different contexts x i , x j with different strength ∆ and 2∆ .
Revised: The first one θ ( i,j ) is a mixture of two different contexts x i , x j with different strength ∆ and 2∆ .

Original: The second one is θ i which only contains features from one context x i .
Revised: The second one is θ i which only contains features from one context x i .

Original: We can further verify that the size of | Θ | = |D| 2 and ∥ θ ∥ 2 ≤ √ 5∆ for θ ∈ Θ .
Revised: We can further verify that the size of | Θ | = |D| 2 and ∥ θ ∥ 2 ≤ √ 5∆ for θ ∈ Θ .

Original: For different parameter θ , the reward function is sampled from a Gaussian distribution N ( r θ ( x ) , 1) , where the expected reward function is defined as
Revised: For different parameter θ , the reward function is sampled from a Gaussian distribution N ( r θ ( x ) , 1) , where the expected reward function is defined as

Original: [Equation]
Revised: [Equation]

Original: ∆ if x = x i 0 otherwise .
Revised: ∆ if x = x i 0 otherwise .

Original: We can verify that the minimal sub-optimality of all these bandit problem is ∆ .
Revised: We can verify that the minimal sub-optimality of all these bandit problem is ∆ .

Original: For different parameter θ and input x , by utilizing the sparsity of the set D (i.e. | x ⊤ y | ≤ ε if x ̸ = y ), we can verify the misspecification level as
Revised: For different parameter θ and input x , by utilizing the sparsity of the set D (i.e. | x ⊤ y | ≤ ε if x ̸ = y ), we can verify the misspecification level as

Original: [Equation]
Revised: [Equation]

Original: Therefore we have verified that the misspecification level is bounded by ζ = 3∆ ε .
Revised: Therefore we have verified that the misspecification level is bounded by ζ = 3∆ ε .

Original: The provided bandit structure is hard for any linear algorithm to learn since any algorithm cannot get any information before it encounters non-zero expected rewards, even regardless of the noise of the rewards.
Revised: The provided bandit structure is hard for any linear algorithm to learn since any algorithm cannot get any information before it encounters non-zero expected rewards, even regardless of the noise of the rewards.

Original: We following the same method in Lattimore & Szepesv´ari (2020).
Revised: We following the same method in Lattimore & Szepesv´ari (2020).

Original: If the algorithm choose arm i at the first round, there would be |D| parameters (i.e. θ i , θ ( i, · ) receiving a non-zero expected reward.
Revised: If the algorithm choose arm i at the first round, there would be |D| parameters (i.e. θ i , θ ( i, · ) receiving a non-zero expected reward.

Original: On the second round if the algorithm choose a different arm j , there would be |D| parameters (i.e. θ j , θ ( j,k : k ̸
Revised: On the second round if the algorithm choose a different arm j , there would be |D| parameters (i.e. θ j , θ ( j,k : k ̸

Original: = i ) receiving a non-zero expected reward.
Revised: = i ) receiving a non-zero expected reward.

Original: Therefore the average time of receiving zero expected reward should be
Revised: Therefore the average time of receiving zero expected reward should be

Original: [Equation]
Revised: [Equation]

Original: where the third equation is from the fact that (cid:80) ni
Revised: where the third equation is from the fact that (cid:80) ni

Original: =

Revised: =

Original: i = n ( n + 1) / 2 and (cid:80) ni
Revised: i = n ( n + 1) / 2 and (cid:80) ni

Original: =

Revised: =

Original: i 2 = n ( n + 1)(2 n + 1) / 6 .
Revised: i 2 = n ( n + 1)(2 n + 1) / 6 .

Original: The last inequality is from the fact that 2 |D| − 1) / (3 |D| ) ≤ 2 / 3 .
Revised: The last inequality is from the fact that 2 |D| − 1) / (3 |D| ) ≤ 2 / 3 .

Original: Therefore, even without of the random noise, any algorithm is expected to receive min { K, ( |D| − 1) / 6 } uninformative data with expected reward to be zero.
Revised: Therefore, even without of the random noise, any algorithm is expected to receive min { K, ( |D| − 1) / 6 } uninformative data with expected reward to be zero.

Original: Therefore any algorithm will receive a ∆ min { K, ( |D| − 1) / 6 } regret considers the suboptimality as ∆ .
Revised: Therefore any algorithm will receive a ∆ min { K, ( |D| − 1) / 6 } regret considers the suboptimality as ∆ .

Original: Next, we consider the effect of random noise.
Revised: Next, we consider the effect of random noise.

Original: For any algorithm running on this parameter set Θ , we find two parameter θ
Revised: For any algorithm running on this parameter set Θ , we find two parameter θ

Original: i and θ i,j
Revised: i and θ i,j

Original: where j ̸ = i .
Revised: where j ̸ = i .

Original: Define the event as A =
Revised: Define the event as A =

Original: { T j ( k ) ≥ k/ 2 }
Revised: { T j ( k ) ≥ k/ 2 }

Original: and
Revised: and

Original: A c = { T j ( k ) < k/ 2 } .
Revised: A c = { T j ( k ) < k/ 2 } .

Original: By Lemma E.2 and Lemma E.3,
Revised: By Lemma E.2 and Lemma E.3,

Original: P θ i
Revised: P θ i

Original: [Equation]
Revised: [Equation]

Original: Noticing the minimal sub-optimality gap is ∆ .
Revised: Noticing the minimal sub-optimality gap is ∆ .

Original: Also the j -th arm is the sub-optimality arm for parameter θ i .
Revised: Also the j -th arm is the sub-optimal arm for parameter θ i .

Original: Therefore, once T j ( k ) ≥ k/ 2 , the algorithm will at least suffer from ∆ k/ 2 regret for parameter θ i .
Revised: Therefore, once T j ( k ) ≥ k/ 2 , the algorithm will at least suffer from ∆ k/ 2 regret for parameter θ i .

Original: Also, since the j -th arm is the optimal arm for bandit
Revised: Also, since the j -th arm is the optimal arm for bandit

Original: θ
Revised: θ

Original: ( i,j ) .
Revised: ( i,j ) .

Original: If T j ( k ) < k/ 2 , the algorithm will also at least suffer from ∆ k/ 2 regret for θ ( i,j ) .
Revised: If T j ( k ) < k/ 2 , the algorithm will also at least suffer from ∆ k/ 2 regret for θ ( i,j ) .

Original: Denoting R θ ( k ) as the expected cumulative regret over k rounds, that is to say
Revised: Denoting R θ ( k ) as the expected cumulative regret over k rounds, that is to say

Original: [Equation]
Revised: [Equation]

Original: On the other hand since the bandit using θ i and θ j only differ in the j -th arm.
Revised: On the other hand since the bandit using θ i and θ j only differ in the j -th arm.

Original: Since standard
Revised: Since standard

Original: Gaussian noise is adapted, KL( P θ i ,n , P θ ( i,j ) ,n ) =
Revised: Gaussian noise is adapted, KL( P θ i ,n , P θ ( i,j ) ,n ) =

Original: ∆ 2 1 [ n = j ] / 2 .
Revised: ∆ 2 1 [ n = j ] / 2 .

Original: Combining this with (E.2), (E.1) suggests that
Revised: Combining this with (E.2), (E.1) suggests that

Original: [Equation]
Revised: [Equation]

Original: which suggests that
Revised: which suggests that

Original: [Equation]
Revised: [Equation]

Original: For any algorithm seeking to get a sublinear expected regret bound of R θ ( k ) ≤ Ck α with C > 0 , 0 ≤ α < 1 for all θ ∈ Θ , (E.3) becomes
Revised: For any algorithm seeking to get a sublinear expected regret bound of R θ ( k ) ≤ Ck α with C > 0 , 0 ≤ α < 1 for all θ ∈ Θ , (E.3) becomes

Original: [Equation]
Revised: [Equation]

Original: Since that the regret on θ i can be decomposed by
Revised: Since that the regret on θ i can be decomposed by

Original: R θ i ( k
Revised: R θ i ( k

Original: [Equation]
Revised: [Equation]

Original: D|
Revised: D|

Original: [Equation]
Revised: [Equation]

Original: combining (E.5) with (E.4) yields
Revised: combining (E.5) with (E.4) yields

Original: [Equation]
Revised: [Equation]

Original: where the max operator is trivially taken for R θ ( k ) ≥ 0 .
Revised: where the max operator is trivially taken for R θ ( k ) ≥ 0 .

Original: F P ROOF OF T HEOREM 7. 
Revised: F P ROOF OF T HEOREM 7.

Original: In this section we provide the proof of Theorem 7.4, we start from the technical lemmas for the proof.
Revised: In this section we provide the proof of Theorem 7.4, we start from the technical lemmas for the proof.

Original: The first lemma is similar with Lemma C.1 by setting λ
Revised: The first lemma is similar with Lemma C.1 by setting λ

Original: = L = 1 as in Definition 7.1 and taking union for H possible cases
Revised: = L = 1 as in Definition 7.1 and taking union for H possible cases

Original: Lemma F.1.
Revised: Lemma F.1.

Original: Given 0 < Γ ≤ 1 .
Revised: Given 0 < Γ ≤ 1 .

Original: For any k ∈ [ K ] , |C k | ≤ 8 Hd Γ − 2 log(6 H Γ − 2 ) .
Revised: For any k ∈ [ K ] , |C k | ≤ 8 Hd Γ − 2 log(6 H Γ − 2 ) .

Original: Then the next lemma extends Lemma C.2 and Lemma C.4 to H > 1 setting and is similar with
Revised: Then the next lemma extends Lemma C.2 and Lemma C.4 to H > 1 setting and is similar with

Original: Lemma C.5 in Jin et al.
Revised: Lemma C.5 in Jin et al.

Original: (2020), by replacing the total number K with |C K | used in regression
Revised: (2020), by replacing the total number K with |C K | used in regression

Original: Lemma F.2 (Lemma C.5, Jin et al. 2020) .
Revised: Lemma F.2 (Lemma C.5, Jin et al. 2020) .

Original: Let (cid:101) β
Revised: Let (cid:101) β

Original: = c β H ( d √ ι 2 + ζ Γ − 1 √ 8 Hdι 1 ) , where ι 1 = log(6 H Γ − 2 ) , ι 2 = log((

Revised: = c β H ( d √ ι 2 + ζ Γ − 1 √ 8 Hdι 1 ) , where ι 1 = log(6 H Γ − 2 ) , ι 2 = log((

Original: H 2 d 2 Γ − 2 ι 1 ) /δ )
Revised: H 2 d 2 Γ − 2 ι 1 ) /δ )

Original: and c β is an absolute constant.
Revised: and c β is an absolute constant.

Original: With probability at least 1 − δ , for any fixed policy π
Revised: With probability at least 1 − δ , for any fixed policy π

Original: and ( s, a, h, k ) ∈ S ×
Revised: and ( s, a, h, k ) ∈ S ×

Original: A ×
Revised: A ×

Original: [ H ] ×
Revised: [ H ] ×

Original: [ K ] ,
Revised: [ K ] ,

Original: [Equation]
Revised: [Equation]

Original: where | ρ kh ( s, a ) | ≤ (cid:101) β
Revised: where | ρ kh ( s, a ) | ≤ (cid:101) β

Original: ∥ ϕ
Revised: ∥ ϕ

Original: ( s, a ) ∥
Revised: ( s, a ) ∥

Original: ( U kh ) − 1 + 4 Hζ .
Revised: ( U kh ) − 1 + 4 Hζ .

Original: Given Lemma F.2 we can provide the rate of confidence radius β by adapting the condition and parameters setting: on the misspecification level and the parameter setting
Revised: Given Lemma F.2 we can provide the rate of confidence radius β by adapting the condition and parameters setting: on the misspecification level and the parameter setting

Original: Corollary F.3.
Revised: Corollary F.3.

Original: Let the misspecification level be bounded by ζ = (cid:101)
Revised: Let the misspecification level be bounded by ζ = (cid:101)

Original: O (∆ d − 0 .
Revised: O (∆ d − 0 .

Original: 5 H − 2 . 5 ) and choose the threshold parameter to be Γ = (cid:101) Θ(∆ d − 1 H − 2 ) , one can verify that the (cid:101) β in Lemma F.2 can be bounded by (cid:101) β ≤ β
Revised: 5 H − 2 . 5 ) and choose the threshold parameter to be Γ = (cid:101) Θ(∆ d − 1 H − 2 ) , one can verify that the (cid:101) β in Lemma F.2 can be bounded by (cid:101) β ≤ β

Original: = (cid:101) O
Revised: = (cid:101) O

Original: ( Hd ) .
Revised: ( Hd ) .

Original: Given this corollary, we show that the sub-optimality gap is controlled by three parts: summation of the UCB bonus, misspecification level and the noise induced by the probability transition kernel.
Revised: Given this corollary, we show that the sub-optimality gap is controlled by three parts: summation of the UCB bonus, misspecification level and the noise induced by the probability transition kernel.

Original: Lemma F.4.
Revised: Lemma F.4.

Original: Suppose Corollary F.3 and Lemma F.2 holds, for any subset K ⊆
Revised: Suppose Corollary F.3 and Lemma F.2 holds, for any subset K ⊆

Original: [ K ] and h ∈ [ H ] we have
Revised: [ K ] and h ∈ [ H ] we have

Original: [Equation]
Revised: [Equation]

Original: where ε kh :
Revised: where ε kh :

Original: =
Revised: =

Original: [ P h ( V kh +1 − V π k h +1 )]
Revised: [ P h ( V kh +1 − V π k h +1 )]

Original: ( s kh , a kh ) − ( V kh +1 ( s kh +1 )
Revised: ( s kh , a kh ) − ( V kh +1 ( s kh +1 )

Original: − V π k h +

Revised: − V π k h +

Original: ( s kh +1 )) is a zero-mean random variable conditioned on all randomness before episode k .
Revised: ( s kh +1 )) is a zero-mean random variable conditioned on all randomness before episode k .

Original: Then the next lemma suggests that the cumulative regret can be bounded by the summation of suboptimality gap at all stage, with high probability
Revised: Then the next lemma suggests that the cumulative regret can be bounded by the summation of suboptimality gap at all stage, with high probability

Original: Lemma F.5 (Lemma 6.1, He et al. 2021a) .
Revised: Lemma F.5 (Lemma 6.1, He et al. 2021a) .

Original: For each MDP M ( S , A , H, { r h } , { P h } ) , with probability at least 1 − δ , the cumulative regret over K episode is bounded by
Revised: For each MDP M ( S , A , H, { r h } , { P h } ) , with probability at least 1 − δ , the cumulative regret over K episode is bounded by

Original: [Equation]
Revised: [Equation]

Original: Equipped with these lemmas, we can start our proof.
Revised: Equipped with these lemmas, we can start our proof.

Original: Proof of Theorem 7.4.
Revised: Proof of Theorem 7.4.

Original: For simplicity, we denote the non-selected episode set as (cid:101) C K :=
Revised: For simplicity, we denote the non-selected episode set as (cid:101) C K :=

Original: [ K ] \ C K .
Revised: [ K ] \ C K .

Original: We first assume Lemma F.2 holds.
Revised: We first assume Lemma F.2 holds.

Original: We fix stage h and then define the following sequence to apply the peeling technique which is also used in He et al.
Revised: We fix stage h and then define the following sequence to apply the peeling technique which is also used in He et al.

Original: (2021a).
Revised: (2021a).

Original: For 0 ≤ l ≤ log( H/ ∆) / log(2) , l ∈ N , let k l 0 = 

Revised: For 0 ≤ l ≤ log( H/ ∆) / log(2) , l ∈ N , let k l 0 =

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Since ∆ h ( s, a ) ≤ H , there exists 1 + log( H/ ∆) / log(2) levels.
Revised: Since ∆ h ( s, a ) ≤ H , there exists 1 + log( H/ ∆) / log(2) levels.

Original: We further denote K l = { k l 1 , · · · , k l |K l | } to be the set of the sequence { k li } i .
Revised: We further denote K l = { k l 1 , · · · , k l |K l | } to be the set of the sequence { k li } i .

Original: We fix one level l in the following proof.
Revised: We fix one level l in the following proof.

Original: On the one hand, due to the fact that ∆ h ( s kh , a kh ) is lower bounded, we have
Revised: On the one hand, due to the fact that ∆ h ( s kh , a kh ) is lower bounded, we have

Original: [Equation]
Revised: [Equation]

Original: On the other hand, Lemma F.4 yields
Revised: On the other hand, Lemma F.4 yields

Original: [Equation]
Revised: [Equation]

Original: By Azuma-Hoeffding’s inequality, with probability at least 1 − δ/H , I l 2 is bounded by
Revised: By Azuma-Hoeffding’s inequality, with probability at least 1 − δ/H , I l 2 is bounded by

Original: [Equation]
Revised: [Equation]

Original: δ
Revised: δ

Original: [Equation]
Revised: [Equation]

Original: Since [ K ] = (cid:101) C K ∪ C K , term I l 1 is be decomposed as
Revised: Since [ K ] = (cid:101) C K ∪ C K , term I l 1 is be decomposed as

Original: [Equation]
Revised: [Equation]

Original: F.
Revised: F.

Original: [Equation]
Revised: [Equation]

Original: where the first term in the first inequality utilizes that ∥ ϕ kh ∥ ≤ Γ for all h if k ∈ (cid:101) C K and the second term is from triangle’s inequality.
Revised: where the first term in the first inequality utilizes that ∥ ϕ kh ∥ ≤ Γ for all h if k ∈ (cid:101) C K and the second term is from triangle’s inequality.

Original: The second inequality utilizes Lemma C.5.
Revised: The second inequality utilizes Lemma C.5.

Original: The third inequality is because both |K l ∩ C K | and |K l ∩ (cid:101)
Revised: The third inequality is because both |K l ∩ C K | and |K l ∩ (cid:101)

Original: C K | is smaller than |K l | as well as K .
Revised: C K | is smaller than |K l | as well as K .

Original: Plugging (F.3) and (F.4) into (F.2) then combining (F.2) with (F.1) yields
Revised: Plugging (F.3) and (F.4) into (F.2) then combining (F.2) with (F.1) yields

Original: [Equation]
Revised: [Equation]

Original: Recall the parameter setting β
Revised: Recall the parameter setting β

Original: =
Revised: =

Original: (cid:101) O ( Hd ) , Γ = (cid:101) O (∆ d − 0 .
Revised: (cid:101) O ( Hd ) , Γ = (cid:101) O (∆ d − 0 .

Original: 5 H − 2 ) , as long as ζ = (cid:101) O (∆ d − 1 H − 2 . 5 ) , it can be guaranteed that I 3 ≤ ∆ / 2 ≤ 2 l ∆ / 2 .
Revised: 5 H − 2 ) , as long as ζ = (cid:101) O (∆ d − 1 H − 2 . 5 ) , it can be guaranteed that I 3 ≤ ∆ / 2 ≤ 2 l ∆ / 2 .

Original: The calculation of the logarithmic terms can follow the proof of Theorem 5.1 where we ignore it for simplicity.
Revised: The calculation of the logarithmic terms can follow the proof of Theorem 5.1 where we ignore it for simplicity.

Original: Plugging this into (F.5) we have that when the condition on ζ is satisfied,
Revised: Plugging this into (F.5) we have that when the condition on ζ is satisfied,

Original: [Equation]
Revised: [Equation]

Original: by the fact that ( √ a + √ b ) 2 ≤ 2 a + 2 b , this implies
Revised: by the fact that ( √ a + √ b ) 2 ≤ 2 a + 2 b , this implies

Original: [Equation]
Revised: [Equation]

Original: This suggests that |K l | ≤ 4 − l ∆ − 2 I 4 with probability at least 1 − δ/H .
Revised: This suggests that |K l | ≤ 4 − l ∆ − 2 I 4 with probability at least 1 − δ/H .

Original: Since for all k ∈ K l , ∆ h ( s kh , a kh ) < 2 l +1 ∆ , the cumulative sub-optimality gap in set K l is bounded by
Revised: Since for all k ∈ K l , ∆ h ( s kh , a kh ) < 2 l +1 ∆ , the cumulative sub-optimality gap in set K l is bounded by

Original: [Equation]
Revised: [Equation]

Original: Replacing δ with δ/ (1 + log( H ∆ − 1 ) / log(2)) , we have that with probability at least 1 − δ/H , (F.7) holds for all 

Revised: Replacing δ with δ/ (1 + log( H ∆ − 1 ) / log(2)) , we have that with probability at least 1 − δ/H , (F.7) holds for all

Original: ≤ l ≤ log( H ∆ − 1 ) / log(2) , l ∈ N by union bound.
Revised: ≤ l ≤ log( H ∆ − 1 ) / log(2) , l ∈ N by union bound.

Original: Therefore we have that
Revised: Therefore we have that

Original: [Equation]
Revised: [Equation]

Original: By union bound we have with probability at least 1 − δ , (F.8) holds for all h ∈
Revised: By union bound we have with probability at least 1 − δ , (F.8) holds for all h ∈

Original: [ H ] thus
Revised: [ H ] thus

Original: [Equation]
Revised: [Equation]

Original: By Lemma F.5, plugging the value of I 4 in (F.6) back to (F.9) yields
Revised: By Lemma F.5, plugging the value of I 4 in (F.6) back to (F.9) yields

Original: Regret
Revised: Regret

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − 3 δ by union bound over the probability event in Lemma F.2, Lemma F. and (F.9).
Revised: with probability at least 1 − 3 δ by union bound over the probability event in Lemma F.2, Lemma F. and (F.9).

Original: By Corollary F.3, β = (cid:101) O
Revised: By Corollary F.3, β = (cid:101) O

Original: ( Hd + √ H 3 dζ Γ − 1 ) =
Revised: ( Hd + √ H 3 dζ Γ − 1 ) =

Original: (cid:101) O ( Hd ) , replace δ with δ/ 3 , we have the regret is bounded by
Revised: (cid:101) O ( Hd ) , replace δ with δ/ 3 , we have the regret is bounded by

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ .
Revised: with probability at least 1 − δ .

Original: G P ROOF OF L EMMAS IN A PPENDIX F G. P ROOF OF L EMMA F. 
Revised: G P ROOF OF L EMMAS IN A PPENDIX F G. P ROOF OF L EMMA F.

Original: The proof technique is similar with Lemma C.1 equipped with Lemma D.1 and the Lemma D.3.
Revised: The proof technique is similar with Lemma C.1 equipped with Lemma D.1 and the Lemma D.3.

Original: Proof of Lemma F.1.
Revised: Proof of Lemma F.1.

Original: Since the selected data samples follows that there exists an h ∈
Revised: Since the selected data samples follows that there exists an h ∈

Original: [ H ] such that ∥ ϕ kh ∥
Revised: [ H ] such that ∥ ϕ kh ∥

Original: ( U kh )
Revised: ( U kh )

Original: [Equation]
Revised: [Equation]

Original: 1 ≥ Γ , therefore the summation of the data is lower bounded by
Revised: 1 ≥ Γ , therefore the summation of the data is lower bounded by

Original: [Equation]
Revised: [Equation]

Original: 1) on the other hand, Lemma D.1 yields
Revised: 1) on the other hand, Lemma D.1 yields

Original: [Equation]
Revised: [Equation]

Original: where λ =
Revised: where λ =

Original: L = 1 .
Revised: L = 1 .

Original: Combining (G.1) with (G.2) yields
Revised: Combining (G.1) with (G.2) yields

Original: [Equation]
Revised: [Equation]

Original: then following the same rule as the proof of Lemma C.1 we have the necessary condition for (G.3) is
Revised: then following the same rule as the proof of Lemma C.1 we have the necessary condition for (G.3) is

Original: [Equation]
Revised: [Equation]

Original: G. P ROOF OF L EMMA F. 
Revised: G. P ROOF OF L EMMA F.

Original: Proof of Lemma F.2.
Revised: Proof of Lemma F.2.

Original: The proof of this lemma follows the proof of Lemma C.5 in Jin et al.
Revised: The proof of this lemma follows the proof of Lemma C.5 in Jin et al.

Original: (2020), from which we have
Revised: (2020), from which we have

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) P ( ·| s, a ) is the well-specified transition kernel defined by (cid:101) P ( ·| s, a )
Revised: where (cid:101) P ( ·| s, a ) is the well-specified transition kernel defined by (cid:101) P ( ·| s, a )

Original: = ⟨ µ ( · ) , ϕ ( s, a ) ⟩ .
Revised: = ⟨ µ ( · ) , ϕ ( s, a ) ⟩ .

Original: From the proof in Jin et al.
Revised: From the proof in Jin et al.

Original: we have that for any ( s, a ) ∈ S × A ,
Revised: we have that for any ( s, a ) ∈ S × A ,

Original: [Equation]
Revised: [Equation]

Original: For the fourth term, by Lemma D.6, which improves the Lemma C.4 in Jin et al.
Revised: For the fourth term, by Lemma D.6, which improves the Lemma C.4 in Jin et al.

Original: (2020) by a factor of √ d , we have
Revised: (2020) by a factor of √ d , we have

Original: [Equation]
Revised: [Equation]

Original: Finally, since (cid:10) ϕ ( s, a ) , w kh − w πh (cid:11) =
Revised: Finally, since (cid:10) ϕ ( s, a ) , w kh − w πh (cid:11) =

Original: ⟨ ϕ ( s, a ) , q 1 + q 2 + q 3 + q 4 ⟩ we have that
Revised: ⟨ ϕ ( s, a ) , q 1 + q 2 + q 3 + q 4 ⟩ we have that

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: plugging the bound of |C k − 1 | in Lemma F.1 back we will have the claimed results.
Revised: plugging the bound of |C k − 1 | in Lemma F.1 back we will have the claimed results.

Original: G. P ROOF OF L EMMA F. 
Revised: G. P ROOF OF L EMMA F.

Original: The proof of Lemma F.4 follows the same technique from Jin et al.
Revised: The proof of Lemma F.4 follows the same technique from Jin et al.

Original: and we warp it here for completeness.
Revised: and we warp it here for completeness.

Original: The first lemma shows that the estimation of Q function is still optimistic regardless the misspecification.
Revised: The first lemma shows that the estimation of Q function is still optimistic regardless the misspecification.

Original: Lemma G.1 (Lemma C.5, Jin et al. 2020) .
Revised: Lemma G.1 (Lemma C.5, Jin et al. 2020) .

Original: Under the parameter setting in Theorem 7.4 and
Revised: Under the parameter setting in Theorem 7.4 and

Original: Lemma F.2 holds, we have Q kh ( s, a ) ≥ Q ∗ h ( s, a )
Revised: Lemma F.2 holds, we have Q kh ( s, a ) ≥ Q ∗ h ( s, a )

Original: − 4 H ( H + 1 − h ) ζ for all ( s, a, h, k ) ∈
Revised: − 4 H ( H + 1 − h ) ζ for all ( s, a, h, k ) ∈

Original: S ×
Revised: S ×

Original: A ×
Revised: A ×

Original: [ H ] ×
Revised: [ H ] ×

Original: [ K ] .
Revised: [ K ] .

Original: Next lemma provides the recursive error bound for the estimated value function V kh ( · )
Revised: Next lemma provides the recursive error bound for the estimated value function V kh ( · )

Original: Lemma G.2 (Lemma C.6, Jin et al. 2020) .
Revised: Lemma G.2 (Lemma C.6, Jin et al. 2020) .

Original: Suppose Lemma F.2 holds, we have for all ( h, k ) ∈
Revised: Suppose Lemma F.2 holds, we have for all ( h, k ) ∈

Original: [ H ] ×
Revised: [ H ] ×

Original: [ K ] ,
Revised: [ K ] ,

Original: [Equation]
Revised: [Equation]

Original: where V h +1 ( · ) = 0 .
Revised: where V h +1 ( · ) = 0 .

Original: Proof of Lemma F.4.
Revised: Proof of Lemma F.4.

Original: We first denote ε kh as [ P ( V kh +

Revised: We first denote ε kh as [ P ( V kh +

Original: − V π k h +1 )]
Revised: − V π k h +1 )]

Original: ( s kh , a kh ) , it’s easy to verify that ε kh is a zero-mean random variable conditioned on all randomness before k -th episode.
Revised: ( s kh , a kh ) , it’s easy to verify that ε kh is a zero-mean random variable conditioned on all randomness before k -th episode.

Original: By Lemma G.1, for all ( s, a, h, k ) ∈ S ×
Revised: By Lemma G.1, for all ( s, a, h, k ) ∈ S ×

Original: A ×
Revised: A ×

Original: [ H ] ×
Revised: [ H ] ×

Original: [ K ] , we have Q kh ( s, a ) ≥ Q ∗ h ( s, a ) − 4 H 2 ζ .
Revised: [ K ] , we have Q kh ( s, a ) ≥ Q ∗ h ( s, a ) − 4 H 2 ζ .

Original: Also, by the definition of minimal sub-optimality gap in Definition 7.3, we have
Revised: Also, by the definition of minimal sub-optimality gap in Definition 7.3, we have

Original: [Equation]
Revised: [Equation]

Original: Since Algorithm 2 is taking the greedy policy a kh = argmax a Q kh ( s kh , a ) , the sub-optimality gap is bounded by
Revised: Since Algorithm 2 is taking the greedy policy a kh = argmax a Q kh ( s kh , a ) , the sub-optimality gap is bounded by

Original: [Equation]
Revised: [Equation]

Original: since V π k h ( s kh ) ≤ V ∗ h ( s kh ) .
Revised: since V π k h ( s kh ) ≤ V ∗ h ( s kh ) .

Original: Following Lemma G.2 by telescoping we have
Revised: Following Lemma G.2 by telescoping we have

Original: [Equation]
Revised: [Equation]

Original: Plugging (G.6) back into (G.5) we will have the result as
Revised: Plugging (G.6) back into (G.5) we will have the result as

Original: [Equation]
Revised: [Equation]

Original: Since it holds for all k ∈ [ K ] , we can take an additional summation over k ∈ K ⊆
Revised: Since it holds for all k ∈ [ K ] , we can take an additional summation over k ∈ K ⊆

Original: [ K ] to get the claimed result.
Revised: [ K ] to get the claimed result.